vCenter 6.7 vPostgres Database Tips

While digging into some validation tests and orphaned hosts I got a chance to learn how to do more things in the vPostgres database of a vCenter. For reference an orphaned host is one that exists in the DB but is no longer present in vCenter. Running a test against a vCenter may return failing results against a host that no long exists.

Accessing your vPostgres DB:

*Please do not run these commands without understanding exactly what they do and have a valid backup.*

Login to the vCenter via SSH, access the shell and run the following command(Note the -d VCDB portion refers to the default embedded DB instance name and your name may change)

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Once logged into the vCenter DB the prompt will be similar to VCDB# and commands against the DB can be run.

*Lots of disclaimers here since running commands against a live database can cause massive damage – please be responsible*

Basic commands:

\l+ shows a list of databases

\dt  shows a list of relations/tables

How to run select commands(Notice the ; at the end of each line – this is required to complete the command):

Below are examples of commands to return different values

The below command will show you the vCenter version

 SELECT * FROM VPX_VERSION;

This command will list all entities within the DB, which could include hosts, resource pools, vApps, virtual wires etc..

SELECT * FROM VPX_ENTITY;

This command proved very useful for me and it allows you to look up if a host(or other object) is actually still in the vC DB:

SELECT * FROM VPX_ENTITY WHERE NAME ='your-hostname-or-object-name';

Depending on the object you may get a return value or nothing like I did below:

id | name | type_id | parent_id
----+------+---------+-----------
(0 rows)

Once the work has been completed in the vC DB this is how you exit

\q

Hopefully this quick reference is helpful and please make sure you take backups before working on any DB

Advertisements

vCloud Availability for Cloud-to-Cloud DR aka vCAV Install and Config part 2

Building off of Part 1(what is vCAV, the architecture and how to deploy it), I’ll detail the configuration process of the Manager and Replication appliances.

Now that you’ve deployed vApp Replication Manager(s)/Replication Manager(s) and Replicators, it is time to configure them! I’ll start off with configuring a replicator appliance in a similar fashion as the documentation for vCloud availability for Cloud-to-Cloud DR.

How does one configure a vCloud availability for Cloud-to-Cloud DR Replicator appliance?

First, login to the deployed replicator by going to the IP address or FQDN set during deploy with :8043/ui/admin at the end.

My example is: https://replicator.vcavlab.com:8043/ui/admin

*Make sure you specify https otherwise your connection will fail*

Login with the set password during deployment of the OVA. Once logged in, a prompt will appear to change the password.

Create a secured password with a minimum of eight characters and at least one of each of the following criteria:

• Lowercase: a b c

• Uppercase: A B C

• Numeric: 1 2 3

• Special: & # %

Update the password, click apply and the main page loads where you can move on to the next configuration item!

Next up, click on “Register in Lookup Service”. Enter a valid lookup service address and click Apply. Enter the lookup service address in the following format https://Lookup-Service-IP-address: 443/lookupservice/sdk.

My example https://vCenter-PSC.vcavlab.com:443/lookupservice/sdk

After clicking apply for the Lookup Service Details, a prompt will show to accept a certificate. If the details look correct, click accept.

The System Monitoring page follows, which will show the details and health of the current deployment. Quick and easy, that is all there is to configuring a vCAV Replicator.

How to configure a vCloud availability for Cloud-to-Cloud DR Replication Manager appliance?

Much like the above replicator appliance initial configuration, login to the web portal of the replication manager appliance you have deployed. Using the IP address or FQDN, login to the replication manager by added :8044/ui/admin to the end

My example is: https://manager.vcavlab.com:8044/ui/admin

*Make sure you specify https otherwise your connection will fail

Another new password.  The same rules apply as above:

Create a secured password with a minimum of eight characters and containing at least one of each of the following criteria:

• Lowercase: a b c

• Uppercase: A B C

• Numeric: 1 2 3

• Special: & # %

Once the password is updated, it’ll redirect to the System Monitoring page which will have a red triangle stating “Missing LookupService settings.” Click on “Configuration” on the left hand side.

On the Configuration page, there should be nothing listed under “Service endpoints –> Lookup service address”. Click Edit on the right which will allow the details to be entered into the Lookup service information.

To finish configuring the Lookup service, accept the certificate.

Once accepted, the Replication Manager configuration is finished. These steps can also be found here. Next up we will cover the configuration of the vApp Replication Manager.

How to configure a vCloud availability for Cloud-to-Cloud DR vApp Replication Manager appliance?

Following the same trend as above, login to the IP address or FQDN with /ui/admin on the end.

My example is: https://manager.vcavlab.com/ui/admin

*Make sure you specify https otherwise your connection will fail

Once logged in with the root account, it will redirect to the Getting Started page which has the Run initial setup wizard link. Click that to begin the configuration.

First, enter a Site name and a Description. Once entered you cannot change your Site name later! Enter something memorable or descriptive like your cloud location or something that makes it easy to identify later on and then click next.

Step 2 asks one more time to enter in the Lookup Service information like the above portals and accept that certificate to proceed.

Step 3 in this wizard asks to configure vCD or discover it automatically. The differences are listed below. Even if the vCloud Director instance is federated with a previously specified lookup service you can select to enter the details manually. Both options work in that case!

If selecting the “Enter details for the vCloud Director Service manually” option you will see the below screen where the following items need to be entered:

•  vCloud Director URL *Make sure you add it with https in the front and :443/api on the end
• vCloud Director Username – make sure the @system portion of the username is the organization the user is part of in vCloud Director. If the user is a system administration user @system
• vCloud Director Password – the password of the user account above

Step 4 is Licensing. Enter license key here and click next.

Step 5 asks if you would like to join the VMware Customer Experience Improvement Program. Please join if you can, the feedback is valuable to VMware.

Step 6 is the moment of truth where you have a chance to verify all of the information just entered in. After validating the details, simply click finish.

It then redirects to the portal login page where you can login with a username from vCD.

My example vcavlabuser@system

These steps can also be found in the official documentation here.

The configuration of the appliances has been completed! However, there are a couple more steps needed before we can start replicating content between sites.

How to add a configured Replicator to a configured Replication Manager in the same site?

Login to the Replication Manager management portal by using the IP address or FQDN configured upon deploy adding :8044/ui/admin to the end.

My example is: https://manager.vcavlab.com:8044/ui/admin

Login with the configured root account to get to the System Monitoring page. Click on Replicators. Click new to add replicator(s).

Enter in the following information for each added replicator.

*If you entered in the API URL as a FQDN the Replication management interface will show only the IP after you add the replicator. 

Accept the presented certificate and the replicator will be added!

Next we will pair cloud sites. Pairing cloud sites creates a trust between instances which allows protection, scheduling, fail over and/or migration workloads between sites.

How to pair clouds sites in a vCloud availability for Cloud-to-Cloud DR vApp Replication Manager appliance?

Login to the IP address or FQDN of the Replication Manager by either the default URL or by adding /ui/admin to the end

My example is: https://manager.vcavlab.com/ui/admin (Login as root)

or

My example is: https://manager.vcavlab.com (Login with a user account from vCloud Director)

Click Sites on the left hand side and then New Cloud Site:

After clicking on New Cloud Site, answer the following:

• Name the site
• Enter in the URL of the remote site Replication Manager *Be sure to use https and :8046
• Description, often overlooked but very useful for others that may use the environment
• Remote appliance credentials
  • Username/password not required but you cannot use the new site until these are entered. This will be the root user and password you changed upon login of that site

Click pair and accept the certificate that pops up. Repeat these steps for each additional site.

[/End post]

In Part 3 I will detail the steps to configure policies, protect VMs/vApps, do a migration and cleanup migrated VMs/vApps that no longer need to be protected.

 

vCloud Availability for Cloud-to-Cloud DR aka vCAV Install and Config part 1

Recently I have had the pleasure of deploying and working extensively with vCloud Availability for Cloud-to-Cloud DR  product (and migrations!) aka vCAV. In my position, I get to dedicate some of my time to vCAV. I figured I would breakdown what vCAV actually is, how to install, configure and use it.

*Disclaimer – I’m writing this for the 3.0 Beta version and features or the process outlined below may change

What is vCloud Availability Cloud-to-Cloud DR? 

“The vCloud Availability Cloud-to-Cloud DR solution provides replication and failover capabilities for vCloud Director workloads at both VM and vApp level.”

The above is taken from here where you can also find the vCAV documentation. This product allows you to protect VMs or vApps for any tenant in a multiple vCloud Director instances for the purpose of Disaster Recovery or Migrations… and it does it really well!

What does vCloud Availability Cloud-to-Cloud DR offer for features?

Tons! Too many to list here, but I will call out some of the important ones from the documentation here. One of the features is how easy it is to deploy the appliances from either the ova in vCenter or via the OVFTOOL. In the below features, notice the Multi-tenant support. This is critical for Service Providers and allows flexibility when migrating multiple tenants.

• Single installation package as a Photon-based virtual appliance.
• The capability of each deployment to serve as both source and recovery vCloud Director instance. There are no dedicated source and destination sites.
• Symmetrical replication flow that can be started from either the source or the recovery vCloud Director site.
• Replication and recovery of vApps and VMs between vCloud Director sites.
• Multi-tenant support.

What is the Cloud Deployment Architecture?

Deploying vCAV into a test or development environment would look something like the below screenshot. You can deploy, configure and be protecting workloads quickly as all three services(vApp Replication Manager/Replication Manager/Replicators) are deployed in a single appliance.

Deploying vCAV into a production environment would look something like the below screenshot. You have a vCloud Availability vApp Replication Manager/Replication Manager which is one appliance and then one or more vCloud Availability Replicators (Scale out depending on workload) in each vCloud Director site.

The above information and screenshots are taken from the product documentation.

How to deploy vCloud Availability Cloud-to-Cloud DR?

There are two easy ways to deploy the vCAV appliances. First is by using vSphere and the “Deploy OVF template” feature. This will allow you to select the ova/ovf you wish to deploy like in the screenshot below:

*There is a 10 minute timer it seems so be sure to complete the “Deploy OVF Template” wizard in 10 minutes or less otherwise your deploy will timeout. 

Walking through the menu you will select the virtual machine name and folder, then select the compute resource (host/cluster/vApp/resource pool).

After you click next you will see a few more items appear on the left (10 total). Clicking next brings you to the Review details screen to verify the size and version of the install:

Clicking next takes you to a License agreement, after you thoroughly read and accept that, clicking next takes you to items 6. Configuration! This screen allows you to select which type of deployment/appliance you want to deploy. All of the roles for vCAV are contained in the same OVA. A breakdown of the appliances can be found here and in the screenshots below. For the purpose of this install and my use case we deploy Cloud Replicator(s) and Cloud Replication Management appliances.

Next is storage. Select the virtual disk format, VM storage policy and datastore/datastore cluster.

The 8th item is networking, select the network you wish to connect the appliance to and the IP protocol(IPv4/IPv6). Clicking next you will see three Application properties:

• Root password
• Enable SSH
• NTP server(s)

You will also see six Networking Properties. You may want to have these figured out before deployment as there is a 10 minute timer on the wizard.

• Default Gateway
• Domain Name
• Domain Search Path
• Domain Name Servers(DNS)
• VM Network IP Address
• VM Network Netmask

Once you have filled these out click next to review your overall configuration.

*Note the Root password gets changed upon the first login

Finally you are ready to complete. This page gives you a full overview of the settings you have configured and selected. Once you are satisfied with the settings, click Finish and you will see your deployment going!

The second way to deploy the appliance is with the OVFTOOL. Below you can see an example from the vCAV documentation. It is easy to customize to your own needs and perhaps add a few more variables.

# echo $VMNAME 

# ./ovftool/ovftool --name"${VMNAME};" --datastore="${VSPHERE_DATASTORE}" --acceptAllEulas --powerOn --X:enableHiddenProperties --X:injectOvfEnv --X:waitForIp --ipAllocationPolicy=fixedPolicy --deploymentOption=combined --machineOutput --noSSLVerify --overwrite --powerOffTarget --prop:guestinfo.cis.appliance.root.password="Your-Root-Password" --prop:guestinfo.cis.appliance.ssh.enabled=True --prop:guestinfo.cis.appliance.net.ntp="your-ntp-server-ip-address" --prop:vami.DNS.VMware_vCloud_Availability_for_Cloud-to-Cloud_DR="Your-DNS-Server-Address" --prop:vami.domain.VMware_vCloud_Availability_for_Cloud-to-Cloud_DR=Your-Domain-Name --prop:vami.gateway.VMware_vCloud_Availability_for_Cloud-to-Cloud_DR="Your-Gateway-IP-Address" --prop:vami.ip0.VMware_vCloud_Availability_for_Cloud-to-Cloud_DR="IP-to-be-Assigned-to-the-Appliance" --prop:vami.netmask0.VMware_vCloud_Availability_for_Cloud-to-Cloud_DR="Your-Netmask-Address" --prop:vami.searchpath.VMware_vCloud_Availability_for_Cloud-to-Cloud_DR="Your-Search-Path-Address" "--net:VM Network=${VSPHERE_NETWORK} --diskMode=thin ${OVA} ${VCENTER_SERVER_USER} VCENTER_SERVER_USER_PASSWORD=${VCENTER_SERVER_ADDRESS}${VSPHERE_LOCATOR}

In Part 2 I’ll detail the configuration of the appliances. Thanks for reading, hopefully this helps!

vCD Cell reboots with service check

In the past, I have had a need to reboot vCloud Director cells and verify that they come up properly. Now that might mean something different depending on your circumstances but for me it mean the following:

1. Quiesce the cell before rebooting
2. Reboot the cell
3. Validate the vmware vcd service started properly
4. Make sure the transfer directory mounted without issue
5. Maybe validate the NTP sync status

Now I have a long list of things I would love to check and be able to do. Sadly I haven’t had the time to make this fully featured. I currently have steps 2/3/4 highlighted in green above completed and tested. If you have a need for some other feature I would be happy to try and add it in, just message me.

The file can be found here  !

vExpert 2019 award announcement

Yesterday the announcement was made for who was accepted into the vExpert program for 2019. Thankfully I made the cut again this year along with my colleague @virtsouthwest. I’m very honored to be part of this program and the community. You can read the full announcement here. Thank you to the entire team that put so much time and effort into reviewing all of the 2000+applications!

The program will open up for new applications sometime around June I think when the 2nd half of the year voting will happen. There is never a bad time to apply and if you have questions about the path you should take or what you should include in your application, reach out to the community or the “vExpert Pro” group that can be found here. I’m always happy to answer questions as well 🙂

Now to keep blogging!

vCenter HTML client critical bug with “Remove from inventory” feature

A new bug has been discovered in the HTML5 vCenter interface:

If you try to remove a VM from inventory in vCenter with the HTML client it will actually delete it from disk. This is only from the HTML client NOT the FLEX client and NOT from the ESXi HTML client as within ESXi you can only register VMs and not “remove from inventory”.

https://kb.vmware.com/s/article/65207

This is only against vCenter 6.5 and 6.7 less than U1b.

Please be careful if you do plan on removing anything from inventory in those versions of vCenter. This is fixed in vCenter 6.7 U1b

Error caused by file /vmfs/volumes Copy/Clone/migration error

If you have ever tried to Copy, Clone or Migrate a VM or VDMK and run into an issue where it fails with an error similar to:

Error caused by file /vmfs/volumes/some-path/*.vmdk

*Note this is just for VMFS not NFS storage

This could be caused by a few different reasons. One of the more well documented reasons is found in a VMware KB. The issue is related to the block size of the destination datastore. The KB notes a simple fix which would be:

ensure that the destination datastore is formatted with a block size that is large enough to support the VMDK file of the source machine

You can read more about block size limitations for different VMFS version here

A cause/solution I’ve read about is deleting or repairing snapshots attached to the VM you are trying to copy/clone/migrate. This article on Ather Beg’s blog details how to repair snapshots or corrupted disks by SSH’ing into the ESXi host and running some vmkfstools commands to hopefully fix the disks.

Another creative solution is where an “export” or convert using Converter can be done . This blog has an excellent breakdown of the installation and step-by-step use! Basically a “convert” of the VM from the source vCenter to the destination using the tool. I’ve read a few posts that this has worked around the same error this blog post is about.

Using internal storage that might not be an option so be creative using vmkfstools commands to manually create new partitions. Be careful that if you do partition an existing datastore or create a new one, you may wipe out existing data.

As noted in this VMware communities thread  the only way to increase the blocksize is to recreate the datastore. By doing so you could choose the block size to match the source thus letting the copy/clone/migration complete without issue.