vCloud Director: A general system error occurred: Exceeded the maximum number of permitted snapshots while capturing vApp to a catalog

During some troubleshooting this past week, the following error came up while trying to capture a vApp to a catalog within VCD:

An error occurred while taking a snapshot: Exceeded the maximum number of permitted snapshots.

Along with a more detailed version:

[ 108e6549-9f26-8e11-a55a-584120dd8ead ] “Cannot create snapshot of VM [vcId=UUID-HERE, moref=vm-6].”
 - Underlying system error:
An error occurred while taking a snapshot: Exceeded the maximum number of permitted snapshots.
vCenter Server task (moref: task-143) failed in vCenter Server ‘bblab-vc1’ (UUID-HERE).
 - An error occurred while taking a snapshot: Exceeded the maximum number of permitted snapshots.

This specific issue centered around a NSX VM that was being used within Cloud Director.
Before checking on that specific VM I found the following KB which confirmed my suspicions that a hard limit has been set on the VM itself.

To confirm this, login to the vCenter where this VM exists. Edit the settings of the VM, browse to “VM Options”, searching the advanced configuration parameters the snapshot.maxSnpshots was present and set to 0 which disables snapshots.

This was updated to a value that the user can decide on(more than 0 to allow snapshots) and from there the content was able to be captured once again

Directly from the KB(please note the CAUTION line and use at your own risk):

To resolve this issue:

1. Power down the virtual machine.
2. Right-click the virtual machine in the Web Client and click Edit Settings.
3. Click the VM Options tab, under Advanced, select the Edit Configuration option.
4. Search for a string name called snapshot.maxSnapshots and increase its value.
5. Click OK to save the changes.
6. Power on the virtual machine and create the required snapshot.

Caution: Only modify the snapshot.maxSnapshots parameter with the guidance of VMware technical support. Increasing the number of allowed snapshots beyond the maximum value of 32 is not supported.
Notes:

• If the snapshot.maxSnapshots value is not present, you may manually enter this configuration setting. In vSphere 6.7.x and 7.0.x, the setting is not listed by default.
• To disallow snapshots, set the snapshot.maxSnapshots value to 0.

vCloud Director: How to bypass SAML authentication for a tenant Org in the H5 UI

In the past tenants with Orgs in vCloud Director that were configured with SAML authentication were able to bypass the SAML authentication redirect by adding /login.jsp to the end of the Org URL. This would then allow a tenant to login with a local or LDAP account.

For example: https://bblab-vcd.com/cloud/org/testing-org/ would be changed to https://bblab-vcd.com/cloud/org/testing-org/login.jsp to allow local or LDAP authentication

Fast forward, there is no more FLEX/Flash UI and I could not find anything documented for the HTML5/H5 UI on how to do this. Trying to login to the Default organization URL(for example: https://bblab-vcd.com/tenant/testing-org/) would fail and could result in the following screen:

In the H5 UI, if the need arises to bypass the SAML authentication; /login could be added to the Default organization URL: https://bblab-vcd.com/tenant/testing-org/login which should redirect to something similar to this: https://bblab-vcd.com/login/login.jsp?service=tenant:testing-org&redirectTo=%2Ftenant%2Ftesting-org .Resulting in a much more pleasant page prompting for a User name/Password:

vCloud Director: HTML5 UI Login fails – “Failed to Start” error and fix

After upgrading a vCloud Director instance to 10.x, I’ve seen many logins fail with the following “Failed to Start” error message in the HTML5 UI only(**Note the FLEX UI is gone in 10.1 versions) This is covered in the following VMware KB as well: https://kb.vmware.com/s/article/75305

**Note this is found on vCD instances where the cells have been installed on linux, I’ve yet to test the appliance for the same**

Digging into this more, this error is caused by something called CORS filter not being updated with all values required. What is CORS and what values should be in CORS?

CORS stands for Cross-Origin Resource Sharing and is used by the H5 UI NOT Flex(flash) UI. The filter limits how each vCD endpoint can be accessed. In the configuration of the cells, if the CORS filter list doesn’t not contain the DNS, IP address, and shortname of each cell the login fails.

The workaround is a pretty quick and painless one. Run the following command on any cell in the vCD instance that is giving this error to retrieve the current CORS configuration:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n webapp.allowed.origins -l

The above will return something similar to this and in my case this is missing all 3 cell instances:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n webapp.allowed.origins -l ",10.1.1.1,http://10.1.1.3,http://10.1.1.2,http://10.1.1.1,https://10.1.1.2,https://10.1.1.1,https://10.1.1.3,https://vcd-bblab.com,http://vcd.bblab.com”

This list is comma separated, so simply adding in the missing values and making one small change to edit the values on the cell instead of listing them is all that is needed. To do that change the webapp.allowed.origins -l to a webapp.allowed.origins -v:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n webapp.allowed.origins -v "10.1.1.1,http://10.1.1.3,http://10.1.1.2,http://10.1.1.1,https://10.1.1.2,https://10.1.1.1,https://10.1.1.3,https://vcd-bblab.com,http://vcd.bblab.com,cell1.bblab.com,cell2.bblab.com,cell3.bblab.com,http://cell1.bblab.com,https://cell1.bblab.com,http://cell2.bblab.com,https://cell2.bblab.com,http://cell3.bblab.com,https://cell3.bblab.com”

Once the CORS values have been updated with the required information, logins will work once again without the need of restarting vCD cell services.

**Not supported, us at your own risk!!!**

These values are all stored in the vCloud Director database under the “config” table under “webapp.allowed.origins”

ESXi 7.0 Upgrade and Lifecycle Manager/VUM VIB dependencies

In my current position I’m working with upgrades at scale, and using  VUM/Lifecycle Manager (new to 7.0). Recently, I went to start the upgrade process on a number of hosts but ran into an issue pretty quick while trying to complete remediation. The exact error was this:

The upgrade has VIBs that are missing dependencies:

(List of 10 VIBs)

Remove the VIBs or use Image Builder to create a custom upgrade ISO image that contains the missing dependencies, and try to upgrade again.

It should be easy to search a way to massively remove these VIBs and continue the upgrade… nope. Enter this blog post and the need to remove VIBs at scale. Two scripts were created using get-esxcli and get-esxcli -V2!

Features of the scripts:

vCenter all host vib remove using get-esxcl V1 <–this was built on get-esxcli v1:

• It loops through hosts found in a given vCenter and removes the VIB(s) 1 by 1
• Allows the user to specify the VIB(s) by name and adds to a remove array
• Prompts for vCenter connection info

vCenter all host vib remove using get-esxcl -V2  <–This is the newer fully supported version built on get-esxcli -V2

• get-esxcli -V2 is the supported versions which allows “arguments” and a user to break down all of the variables
• This version prompts and asks for the following variable/arguments Dryrun, Maintenancemode, NoLiveInstall, and Force for force removal

Hopefully this can help you in mass removal of VIBs. Please reach out if you have questions!

vCloud Director: How to consolidate a VM with PowerCLI

With changes in the UI, features moving around or coming in later releases I was working on a consolidation script for VMs in vCloud Director.

Working with the Flash UI it was easy to see the “Chain Length” and consolidation in the VM menu:

 

Depending on versions of vCloud Director this may not be something exposed at the moment in the version being used. Even if it was exposed it might be nice to script the consolidation when needed.

The script will check if the VM that is to be consolidated is powered off and has a chain length longer than 1. If either of those checks fail it will return an error message. The script can be found in my gitlab blogpost files. Here is the output of it in action reporting my VM is powered on!

****************************************************************
Input the Org name: bblab
Input the vApp name: bbvapp
Input the VM name: bb

****************************************************************

****************************************************************

Could not consolidate VM bb within vApp bbvapp since it is PoweredOn

Please power off this VM and run script again
****************************************************************

Here is the expected return when it works!

****************************************************************
Input the Org name: bblab
Input the vApp name: bbvapp
Input the VM name: chain-test

****************************************************************
Processing VM: chain-test
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 100 / State: Success

 

Hopefully this is helpful!

vCloud Director: How to return default session(s) in powershell

While looking up something for a script idea in the works, google did not return any useful references for “How to return vCloud Director connected session in powershell/powercli” Thought I’d put this here as a reminder and resource in case others are looking for the same information!

$global:DefaultCIServers  # returns all sessions you are connected to, if scripting make sure you disconnect from sessions before scripting against an assumed session [0]

$global:DefaultCIServers[0]  # returns the first session you are connected to, does not appear to be alphabetical 

 $global:DefaultCIServers[0].Name  # can be used for variables or anything 

 $global:DefaultCIServers[0].ExtensionData  # tons of information if you dig in here 

Short, simple and hopefully helpful for others looking to return default sessions….and by “others” I also mean me in about 6 months when I forget how to do this again 😀  I had no clue how to do this but started tinkering around and found it.

vCloud Director: New/Set Catalog PowerShell Modules

A couple years ago I asked the #vExpert community for help to figure out how to create a new catalog within vCloud Director with powershell. There was no option to do so, and Jon Waite was amazing when he created the new-catalog module. The module works great to create a new catalog, but I wanted to take it another step to add in the publish and subscribe to external catalogs ability. I modified the module from Jon to add in these features. I am not 100% sure how to modify the module and check it into his original module so I did it in my blogpost files.

Giving full credit to Jon Waite for making this module. I have added the following to new-catalog.psm1:

• Ability to create a new catalog as published externally
• Script returns the published catalogs URL
• The above URL can be captured as a variable for use in other scripts (see below)

Set-cicatalog was created to change an existing catalog into an externally published catalog:

• Requires the catalog to already be created
• Script returns the published catalogs URL
• The above URL can be captured as a variable for use in other scripts (see below)

New-subscribedcatalog was created as I could not figure out how to put all of these features into 1 single module/function, which I might revisit:

• Ability to create a new catalog within vCD that is subscribed to an “external” feed
• Does minor error reporting if the catalog is created with a bad password and fails to sync
• Does minor status reporting and will return status of sync
• Allows variables to be passed in the script – meaning you can use the URL captured if the above modules 🙂

Hope this helps people in the future and please let me know if there are features I missed or something I can change to make it more user friendly!

vCloud Director: How to unsubscribe a subscribed catalog using PostgreSQL

Something that has perplexing for some time in vCloud Director is how to unsubscribed a catalog once it has been subscribed to an external feed. This is something that could be done to retain content without having to copy it to a local catalog. vCloud Director does not allow this to be done within the UI but it is a pretty quick change in the database

**Please note that these scripts are just for reference – use at your own risk** – Standard disclaimer 🙂 and take backups!!

Using either a SSH session or pgAdmin

The SSH session will be covered first(hopefully adding pgAdmin steps later):

Once a SSH session has been established with the PSQL DB machine, use the following commands to access the specific vCloud Director Database instance:

[root@bblab-vcddb ~]# su - postgres

Last login: Sat Jan 25 21:51:55 UTC 2020 on pts/0

-bash-4.2$ psql -d bblab-vcddb

psql (10.11)

Type "help" for help.

bblab-vcddb=#

Having accessed the vCloud Director database, it is time to track down the catalog in question. A safe way that I have found is to use the org_id to ensure you have tracked down the correct catalog within the given org that is being targeted. Copy the below and change the org name as needed:

select org_id from organization WHERE name = 'bblab-testorg1';

This will return the org_id from the database that is needed to ensure the proper catalog is selected and modified to no longer be subscribed.

select org_id from organization WHERE name = 'bblab-testorg1';

org_id

--------------------------------------

6a8b1172-7c13-4111-bb76-33a1b9471147

(1 row)

From there, take the org_id and catalog name in that needs to no longer be subscribed. Input those into the below statement and run!(please take a backup first – safety)

UPDATE catalog SET subscribed_to_ext_feeds = 'f' WHERE name = 'bblab-catalog1' AND org_id = '6a8b1172-7c13-4111-bb76-33a1b9471147';

Once the above has been ran, only 1 row will be updated. Checking the vCD UI, the catalog that was changed in the DB will no longer show subscribed and the items that had been synced will remain!

Hope this helps and I hope to post pgAdmin steps/screenshots in the near future!

vCloud Director: Update PVDC supported hardware version via PowerCLI

This post servers as a reminder for me but also hopefully helpful to others that want to “explore” ExtensionData with powercli. This post shows how to declare a variable and navigate to the extension data for a Provider Virtual DataCenter within vCloud Director and adjust the Highest Supported Hardware Version.

**Please note that these scripts are just for reference – use at your own risk** – Standard disclaimer 🙂

Connect to a vCloud Director Instance:

Connect-CIServer bblab-vcd

Get a list of PVDCs within the vCD instance:

Get-ProviderVdc

Name Status Enabled CpuUsedGhz MemoryUsedGB StorageUsedGB
---- ------ ------- ---------- ------------ -------------
bblab-pvdc Ready True 8.43 (14.3%) 9.464 (13.0%)

Selecting one of the returned PVDCs above declare it as a variable:

 $hwversionchange = Get-ProviderVdc -name bblab-pvdc

Using the declared variable “pipe” “|” it to Format-List or FL to show more details(Note there is a HighestSupportedHardwareVersion here showing Unknown(possible bug):

 $hwversionchange | FL
Href : https://bblab-vcd/api/admin/extension/providervdc/uuid-here
StorageUsedGB :
StorageOverheadGB :
StorageTotalGB : 0
StorageAllocatedGB :
Status : Ready
MemoryUsedGB : 49.3623046875
MemoryOverheadGB : 1.73046875
MemoryTotalGB : 30.9296875
MemoryAllocatedGB : 14
Enabled : True
HighestSupportedHardwareVersion : Unknown
CpuUsedGHz : 17.164
CpuOverheadGHz : 0.384
CpuTotalGHz : 5.104
CpuAllocatedGHz : 4
ExtensionData : VMware.VimAutomation.Cloud.Views.VMWProviderVdc
StorageProfiles : {bblabcompute}
Description :
Name : bblab-pvdc

ExtensionData is one of the listed items above, adding a . after the variable will expose usable options. Type the PVDC variable and add .ExtensionData on the end to see a full list(Note as in above examples you can add any of these to the end of the variable.extensiondata to see more):

$hwversionchange.ExtensionData
HighestSupportedHardwareVersion : vmx-13
ResourcePoolRefs : {VMware.VimAutomation.Cloud.Views.VimObjectRef1}
VimServer : {bblab-vc1}
DataStoreRefs : {VMware.VimAutomation.Cloud.Views.VimObjectRef1, VMware.VimAutomation.Cloud.Views.VimObjectRef1, VMware.VimAutomation.Cloud.Views.VimObjectRef1,
VMware.VimAutomation.Cloud.Views.VimObjectRef1...}
HostReferences : VMware.VimAutomation.Cloud.Views.VMWHostReferences
NsxTManagerReference :
Status : 1
StorageCapacity :
ComputeCapacity : VMware.VimAutomation.Cloud.Views.RootComputeCapacity
IsEnabled : True
AvailableNetworks : VMware.VimAutomation.Cloud.Views.AvailableNetworks
Vdcs :
Capabilities :
StorageProfiles : VMware.VimAutomation.Cloud.Views.ProviderVdcStorageProfiles
NetworkPoolReferences : VMware.VimAutomation.Cloud.Views.NetworkPoolReferences
Tasks :
Description :
OperationKey :
Client : VMware.VimAutomation.Cloud.Views.CloudClient
Type : application/vnd.vmware.admin.vmwprovidervdc+xml
Link : {, , , ...}
AnyAttr :
VCloudExtension

Simplifying things the HighestSupportedHardwareVersion  has been added to the command:

$hwversionchange.ExtensionData.HighestSupportedHardwareVersion

vmx-13

Keeping the format from above I know I can increase the value to vmx-14 based on the vCloud Director Admin Guide. Set the value with the below command:

$hwversionchange.ExtensionData.HighestSupportedHardwareVersion = 'vmx-14'

The value needs to be “committed” by using the “UpdateServerData()” command which will push the updated value to the vCD instance(Note – make sure you have a backup of your vCD Database in case something goes wrong):

 $hwversionchange.ExtensionData.UpdateServerData()

Once the above command has finished it will return all of the ExtensionData including the newly changed first line 

HighestSupportedHardwareVersion : vmx-14

There is plenty to explore with ExtensionData in vSphere and vCloud Director – explore safely……… with backups 😉

vCloud Availability for Cloud-to-Cloud DR aka vCAV Install and Config part 3

Building off of Part 1 & Part 2 which covers “what is vCAV and how to use vCAV”, I’ll detail in this post, the configurations of policies, protect VMs/vApps, do a migration and cleanup migrated VMs/vApps that no longer need to be protected.

*Once again disclaimer – I’m writing this for the 3.0 Beta version and features. The process outlined below may change – 3.0.1 is GA as of the 23rd of May 2019 release notes found here

Before protecting a VM or vApp to be migrated to a new cloud site the policies will need to be configured on each site. Policies can be created for use by a single Org or can be assigned any combination of Orgs per policy. Each policy can be configured to allow outgoing or incoming replications, to limit the number of replications and snapshots per replications, and the minimum allowed RPO. A fantastic explanation for RPO can be found here. The RPO value can be changed per workload protection but this setting is for the minimum allowed value.

If a new policy is not created, the following error will appear when attempting to protect a new workload:

To create a new policy, navigate to Policies on the left hand navigate pane and click NEW(The default policy can be edited to allow replications and by default all discovered orgs are attached to this policy):

The New Policy window will pop up and each name and setting can be adjusted for each policy created:

Once created, click the radio button next to the policy to assign a new org or orgs and click the ASSIGN button above the polices:

Once the policies have been assigned to the orgs required, vApps can now be protected. Navigate to the left hand side and click from Cloud or to Cloud depending on the direction of the migration:

For this example to Cloud was selected under Outgoing Replications – the on the top menu click NEW to begin protecting a vApp.

The New Outgoing Replication menu appears and allows to sort by the vApp/VM or OrgVDC(VDC aka OVDC) to find the items that need to be protected:

***Note the yellow warning banner notes the VM(s) are powered off and a seed will not be created on the destination side unless done manually or the VM(s) becomes powered on***

After clicking NEXT, the Target Site needs to be selected and authenticated to along with the Target VDC(OVDC). Once these have been configured the next menu will show the Protection Settings:. Below for each migration there are a number of values that can be configured based on the policy settings created above. RPO, Storage policies, does the VM support quiesceing, and compress replication traffic. Each of these is configurable for each protection.

The next menu is Scheduling, which is a fantastic add in 3.x versions! This allow the synchronization of the selected workloads to begin as a specified time or done immediately:

Lastly a summary page Ready To Complete appears detailing the protection settings for the selected workloads. Clicking on FINISH will complete the wizard and file the job within the vCAV appliances!

What happens when vApps/VMs are migrated?

When a vApp or VM is migrated with vCAV, all of the data is synced over to the destination side, the source VM is powered off gracefully and the destination becomes ready in vCD(depending on the power state selected during migration, the power state may be on/off).

What happens when vApps/VMs are “failed-over”?

This assume the source side is down and is more of a disaster recovery option. The Source side is not touched in this case, and the destination side will have the last synchronized data based on the selected RPO.

*Please note the below feature was available in 1.5.

How to report on usage:

Login to the vCloud Availability vApp Replication Manager as root with the following command

c4 loginroot C4-Root-Password-Here

Generate the vCloud Availability for Cloud-to-Cloud DR usage report(report_summary and report_details can be changed to any report name)

usage-report --output /tmp/report_summary.tsv --details /tmp/report_details.tsv

Download the vCloud Availability for Cloud-to-Cloud DR usage report locally.

scp /tmp/report_summary.tsv /tmp/report_details.tsv user@your-host:/download-target-location

(Optional) Remove the generated reports from the vCloud Availability vApp Replication Manager appliance.

rm /tmp/report_summary.tsv /tmp/report_details.tsv

Notes for the above usage report can be found here:

https://docs.vmware.com/en/VMware-vCloud-Availability-for-Cloud-to-Cloud-DR/1.5/com.vmware.c2c.install.config.upgrade.doc/GUID-F36C3CDD-2477-48DE-9883-333D1828A969.html

A HUGE thanks to the vCAV product team that I’ve had the pleasure of working closely with in my current role. They are striving to get to increase scale and get new features into the product quickly! This product just keeps getting better with every release.

Please let me know if you have feedback or question – I’ll do my best to answer them!