VMware Cloud Director – Failed to delete folder from vCenter Server. Could not delete folder “vApp-bblab” because it contains: child “vm-12345”

When attempting to delete vApps (possibly in bulk) there are occasions where some remain in an unresolved state with no Virtual Machines present within the VCD UI:

From the VCD UI under Recent Tasks or the Monitor tab, this type of message appears:

Looking into the VCD Debug logs:

From Debug logs: 
 activity=(com.vmware.vcloud.vdc.impl.DeleteVappActivity,urn:uuid:8c6b91ce-72a4-4eb9-b1f7-31951ede930b)
2021-09-23 15:35:12,102 | ERROR    | Backend-activity-pool-4546 | DeleteVappActivity             | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.vcloud.vdc.impl.DeleteVappActivity$FabricsCleanedPhase@6b7b7ba7 - Handle: urn:uuid:, Current Phase: DeleteVappActivity$FabricsCleanedPhase | requestId=,request=DELETE https://sc1-vcd02.oc.vmware.com/api/vApp/vapp-94a35eb5-7762-46d1-95c0-7d77c089a6d4,requestTime=1632411311263,remoteAddress=10.10.10.10:55491,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 37.0.0-alpha vcd=2f57d2ff-c33a-4499-9ac5-b5f1b0e2d4c2,task=91234567-c0ad-4829-9fe7-d81123456789 activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:12345678-c1ad-1234-1fe7-d81123456789) activity=(com.vmware.vcloud.vdc.impl.DeleteVappActivity,urn:uuid:8c6b91ce-72a4-4eb9-b1f7-11951ede930b)java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.ssdc.util.LMException: Failed to delete folder [vcId=259e22a3-eda1-4e34-2b1e-55a57dc236d8, moref=group-v789012] from vCenter Server.
        at com.vmware.vcloud.val.purger.impl.FolderPurgeUtil.purge(FolderPurgeUtil.java:58)
        at com.vmware.vcloud.val.purger.impl.PurgeServiceImpl.purgeFolder(PurgeServiceImpl.java:49)
        at com.vmware.vcloud.vdc.impl.DeleteVappActivity$FabricsCleanedPhase.invoke(DeleteVappActivity.java:200)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        ... 5 more
Caused by: com.vmware.ssdc.library.exceptions.FolderNotEmptyException: Could not delete folder "vApp-bblab (22a28eb9-1234-33d1-95c0-7d77c089a6d4)" because it contains:
child "vm-123456"
child "vm-789012"
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.DeleteFolder(VC20VirtualEngine.java:2551)
        at com.vmware.vcloud.val.purger.impl.FolderPurgeUtil.purge(FolderPurgeUtil.java:50)
        ... 9 more

The errors note there are still child VMs within the vApp. Knowing that the vApp itself is empty within VCD, moving down the stack the VMs can be found within vCenter still. The “vm-#####” can be found within vCenter either by using the ID listed in the VM or searching the unique ID of the vApp to ensure the correct VMs are found. Once found, these VMs can be deleted from disk:

Once they are deleted in vCenter, the vApp “folder” can be deleted within VCD without issue.

***As with any blog post, please be aware that once these VMs are deleted from disk they are actually deleted. Meaning they are not coming back 🙂 please be cautious and ensure the VMs are no longer required***

NSX-T: Upgrade from 3.0.2 to 3.1.1 UI Errors “Failed to get *”

A fantastic colleague of mine was working on upgrades of NSX-T earlier this year. During the upgrade while checking the cluster status the UI began displaying errors:

The status is miss leading as the upgrade was progressing as expected. There is no need to panic and reboot manager nodes! After waiting 7-10 minutes and checking the cluster status, the errors were gone as the upgrade continued. This does not appear to happen after NSX-T 3.1.1.

vCloud Director: Post upgrade to 10.2.2 Cell or Cells are inactive

During a normal upgrade to 10.2.2 that has been completed many times without issue, a colleague of mine noticed a single cell in an “Inactive” state in the UI:

This was not apparent post upgrade, until the service of that single cell were restarted. Attempting to fix this cell, the service was restarted which made no difference(failed to start), also rebooting the cell did not fix the issue. *Rebooting additional cells resulted in those cells failing to start and being Inactive as well

Next step was to dive into the logs and look to see why the service was not starting and the cell was showing inactive but nothing jumped out as “the problem”. Working with a very diligent support engineer, the following was found:

Caused by: java.util.concurrent.TimeoutException: Timed out waiting for service: 'filterConfigurationService', objectClasses='[interface com.vmware.vcloud.security.filters.FilterConfigurationService]', filter='(objectClass=com.vmware.vcloud.security.filters.FilterConfigurationService)'
       at com.vmware.vcloud.common.service.OsgiServiceReferenceFactoryBean.getObject(OsgiServiceReferenceFactoryBean.java:234)
       at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)

This was leading down a path of an issue that had been seen before where the “jms.user.system.password” was blank within the database. To validate, run the following:

select * from config where name = 'jms.user.system.password';

If the above does not return any value, the password is not set.

To fix this issue, create a new temporary database. First create a database template

#Create a template based on your existing DB
CREATE DATABASE dbname TEMPLATE template0;

#Create a temp DB based on template:
createdb -U test -T template0 vcddb-temp

#Restore DB backup pre upgrade to temp DB created above:
psql -U test vcddb-temp -f /path/to/vcddb-backup.dump.out

Run the above command to select the password, using that password run the following insert command:

insert into config (config_id,catvalue,name,value,sortorder) values ('588','vcloud','jms.user.system.password','xxxxxxxxxx','0'0);

Restart the cells and they should come back up as expected!

VMware Cloud Director: ExtraConfig Keys and NSX Unified Appliance

Have you encountered the error “Validation failed for the OVF file you provided: The OVF uses vmw:ExtraConfig elements which are not allowed by vCloud Director“? You might have noticed it prevents content from being uploaded or imported. Different appliances and a wide range of content require “ExtraConfig” keys in a VCD database (or imported via cell-management-tool). Reference to the commands for the cell-management-tool can be found here. Below, I’ve posted my team’s workaround and detailed some NSX related ExtraConfig keys.

**Disclaimer #1: use these DB modifications at your own risk and please take backups**

**Disclaimer #2: This workaround is highly unsupported**

Where do you find these settings? Opening the OVF file, a simple search for “ExtraConfig” will show what elements are being used. Example:

      <vmw:Config ovf:required="false" vmw:key="tools.syncTimeWithHost" vmw:value="false"/>
      <vmw:ExtraConfig vmw:key="time.synchronize.continue" vmw:value="false"/>
      <vmw:ExtraConfig vmw:key="time.synchronize.restore" vmw:value="false"/>
      <vmw:ExtraConfig vmw:key="time.synchronize.resume.disk" vmw:value="false"/>
      <vmw:ExtraConfig vmw:key="time.synchronize.shrink" vmw:value="false"/>
      <vmw:ExtraConfig vmw:key="time.synchronize.tools.startup" vmw:value="false"/>
      <vmw:ExtraConfig vmw:key="ethernet0.rxDataRingEnabled" vmw:value="0"/>
      <vmw:ExtraConfig vmw:key="ethernet1.rxDataRingEnabled" vmw:value="0"/>
      <vmw:ExtraConfig vmw:key="ethernet2.rxDataRingEnabled" vmw:value="0"/>
      <vmw:ExtraConfig vmw:key="ethernet3.rxDataRingEnabled" vmw:value="0"/>
      <vmw:ExtraConfig vmw:key="isolation.tools.diskWiper.disable" vmw:value="true"/>
      <vmw:ExtraConfig vmw:key="isolation.tools.memSchedFakeSampleStats.disable" vmw:value="true"/>
      <vmw:ExtraConfig vmw:key="isolation.tools.diskShrink.disable" vmw:value="true"/>
      <vmw:ExtraConfig vmw:key="isolation.tools.vmxDnDVersionGet.disable" vmw:value="true"/>
      <vmw:ExtraConfig vmw:key="isolation.tools.unityActive.disable" vmw:value="true"/>
      <vmw:ExtraConfig vmw:key="isolation.tools.guestDnDVersionSet.disable" vmw:value="true"/>
      <vmw:ExtraConfig vmw:key="snapshot.maxSnapshots" vmw:value="0"/>
      <vmw:ExtraConfig vmw:key="RemoteDisplay.maxConnections" vmw:value="1"/>

Below is a small sample of how to insert these into the DB. A larger list can be found here. Please do research before blindly allowing every ExtraConfig key to be imported into Cloud Director.

INSERT into config (config_id, cat, name, value)
values

    --ExtraConfigs for nsx-unified-appliance
    (NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'ethernet0.rxDataRingEnabled', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'ethernet1.rxDataRingEnabled', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'ethernet2.rxDataRingEnabled', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'ethernet3.rxDataRingEnabled', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'isolation.tools.diskWiper.disable', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'isolation.tools.memSchedFakeSampleStats.disable', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'isolation.tools.diskShrink.disable', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'isolation.tools.vmxDnDVersionGet.disable', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'isolation.tools.unityActive.disable', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'isolation.tools.guestDnDVersionSet.disable', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'sched.mem.pin', 'true')
    ,(NEXTVAL('seq_config'), 'ExtraConfigWhitelist', 'svga.autodetect', 'true')
  ON CONFLICT (cat, name) DO UPDATE
  SET value = EXCLUDED.value;

NSX Unified Appliances within Cloud Director allow 0 snapshots for the NSX VM by default. Taking a snapshot will fail if this value is not changed:

<vmw:ExtraConfig vmw:key="snapshot.maxSnapshots" vmw:value="0"/>

The failure (and where to update the value):

VMware Cloud Director: How to consolidate VMs in a vApp Template

As a follow up to this post: vCloud Director: How to consolidate a VM with PowerCLI, I’m creating this one which is almost identical for consolidating all VMs in a vApp templates or individual VMs within a vApp Template with VMware Cloud Director aka VCD or vCD 😉

The feature to consolidate in the UI is exposed in 10.2 versions of VCD but not in earlier versions it seems.

The script found here vCD-consolidate-vapptemplate-vm.ps1 and takes the required input of Org name, Catalog name, vApp Template name. Optionally VM name. The VM name can be left blank and it will consolidate all VMs within the vApp Template.

The same process is followed for checking power state, as a powered on vApp Template will cause errors while attempting to consolidate.

Good luck and hopefully this is useful somewhere along the way!

vCloud Director: A general system error occurred: Exceeded the maximum number of permitted snapshots while capturing vApp to a catalog

During some troubleshooting this past week, the following error came up while trying to capture a vApp to a catalog within VCD:

An error occurred while taking a snapshot: Exceeded the maximum number of permitted snapshots.

Along with a more detailed version:

[ 108e6549-9f26-8e11-a55a-584120dd8ead ] “Cannot create snapshot of VM [vcId=UUID-HERE, moref=vm-6].”
 - Underlying system error:
An error occurred while taking a snapshot: Exceeded the maximum number of permitted snapshots.
vCenter Server task (moref: task-143) failed in vCenter Server ‘bblab-vc1’ (UUID-HERE).
 - An error occurred while taking a snapshot: Exceeded the maximum number of permitted snapshots.

This specific issue centered around a NSX VM that was being used within Cloud Director.
Before checking on that specific VM I found the following KB which confirmed my suspicions that a hard limit has been set on the VM itself.

To confirm this, login to the vCenter where this VM exists. Edit the settings of the VM, browse to “VM Options”, searching the advanced configuration parameters the snapshot.maxSnpshots was present and set to 0 which disables snapshots.

This was updated to a value that the user can decide on(more than 0 to allow snapshots) and from there the content was able to be captured once again

Directly from the KB(please note the CAUTION line and use at your own risk):

To resolve this issue:

1. Power down the virtual machine.
2. Right-click the virtual machine in the Web Client and click Edit Settings.
3. Click the VM Options tab, under Advanced, select the Edit Configuration option.
4. Search for a string name called snapshot.maxSnapshots and increase its value.
5. Click OK to save the changes.
6. Power on the virtual machine and create the required snapshot.

Caution: Only modify the snapshot.maxSnapshots parameter with the guidance of VMware technical support. Increasing the number of allowed snapshots beyond the maximum value of 32 is not supported.
Notes:

• If the snapshot.maxSnapshots value is not present, you may manually enter this configuration setting. In vSphere 6.7.x and 7.0.x, the setting is not listed by default.
• To disallow snapshots, set the snapshot.maxSnapshots value to 0.

vCloud Director: How to bypass SAML authentication for a tenant Org in the H5 UI

In the past tenants with Orgs in vCloud Director that were configured with SAML authentication were able to bypass the SAML authentication redirect by adding /login.jsp to the end of the Org URL. This would then allow a tenant to login with a local or LDAP account.

For example: https://bblab-vcd.com/cloud/org/testing-org/ would be changed to https://bblab-vcd.com/cloud/org/testing-org/login.jsp to allow local or LDAP authentication

Fast forward, there is no more FLEX/Flash UI and I could not find anything documented for the HTML5/H5 UI on how to do this. Trying to login to the Default organization URL(for example: https://bblab-vcd.com/tenant/testing-org/) would fail and could result in the following screen:

In the H5 UI, if the need arises to bypass the SAML authentication; /login could be added to the Default organization URL: https://bblab-vcd.com/tenant/testing-org/login which should redirect to something similar to this: https://bblab-vcd.com/login/login.jsp?service=tenant:testing-org&redirectTo=%2Ftenant%2Ftesting-org .Resulting in a much more pleasant page prompting for a User name/Password:

vCloud Director: HTML5 UI Login fails – “Failed to Start” error and fix

After upgrading a vCloud Director instance to 10.x, I’ve seen many logins fail with the following “Failed to Start” error message in the HTML5 UI only(**Note the FLEX UI is gone in 10.1 versions) This is covered in the following VMware KB as well: https://kb.vmware.com/s/article/75305

**Note this is found on vCD instances where the cells have been installed on linux, I’ve yet to test the appliance for the same**

Digging into this more, this error is caused by something called CORS filter not being updated with all values required. What is CORS and what values should be in CORS?

CORS stands for Cross-Origin Resource Sharing and is used by the H5 UI NOT Flex(flash) UI. The filter limits how each vCD endpoint can be accessed. In the configuration of the cells, if the CORS filter list doesn’t not contain the DNS, IP address, and shortname of each cell the login fails.

The workaround is a pretty quick and painless one. Run the following command on any cell in the vCD instance that is giving this error to retrieve the current CORS configuration:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n webapp.allowed.origins -l

The above will return something similar to this and in my case this is missing all 3 cell instances:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n webapp.allowed.origins -l ",10.1.1.1,http://10.1.1.3,http://10.1.1.2,http://10.1.1.1,https://10.1.1.2,https://10.1.1.1,https://10.1.1.3,https://vcd-bblab.com,http://vcd.bblab.com”

This list is comma separated, so simply adding in the missing values and making one small change to edit the values on the cell instead of listing them is all that is needed. To do that change the webapp.allowed.origins -l to a webapp.allowed.origins -v:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n webapp.allowed.origins -v "10.1.1.1,http://10.1.1.3,http://10.1.1.2,http://10.1.1.1,https://10.1.1.2,https://10.1.1.1,https://10.1.1.3,https://vcd-bblab.com,http://vcd.bblab.com,cell1.bblab.com,cell2.bblab.com,cell3.bblab.com,http://cell1.bblab.com,https://cell1.bblab.com,http://cell2.bblab.com,https://cell2.bblab.com,http://cell3.bblab.com,https://cell3.bblab.com”

Once the CORS values have been updated with the required information, logins will work once again without the need of restarting vCD cell services.

**Not supported, us at your own risk!!!**

These values are all stored in the vCloud Director database under the “config” table under “webapp.allowed.origins”

ESXi 7.0 Upgrade and Lifecycle Manager/VUM VIB dependencies

In my current position I’m working with upgrades at scale, and using  VUM/Lifecycle Manager (new to 7.0). Recently, I went to start the upgrade process on a number of hosts but ran into an issue pretty quick while trying to complete remediation. The exact error was this:

The upgrade has VIBs that are missing dependencies:

(List of 10 VIBs)

Remove the VIBs or use Image Builder to create a custom upgrade ISO image that contains the missing dependencies, and try to upgrade again.

It should be easy to search a way to massively remove these VIBs and continue the upgrade… nope. Enter this blog post and the need to remove VIBs at scale. Two scripts were created using get-esxcli and get-esxcli -V2!

Features of the scripts:

vCenter all host vib remove using get-esxcl V1 <–this was built on get-esxcli v1:

• It loops through hosts found in a given vCenter and removes the VIB(s) 1 by 1
• Allows the user to specify the VIB(s) by name and adds to a remove array
• Prompts for vCenter connection info

vCenter all host vib remove using get-esxcl -V2  <–This is the newer fully supported version built on get-esxcli -V2

• get-esxcli -V2 is the supported versions which allows “arguments” and a user to break down all of the variables
• This version prompts and asks for the following variable/arguments Dryrun, Maintenancemode, NoLiveInstall, and Force for force removal

Hopefully this can help you in mass removal of VIBs. Please reach out if you have questions!

vCloud Director: How to consolidate a VM with PowerCLI

With changes in the UI, features moving around or coming in later releases I was working on a consolidation script for VMs in vCloud Director.

Working with the Flash UI it was easy to see the “Chain Length” and consolidation in the VM menu:

 

Depending on versions of vCloud Director this may not be something exposed at the moment in the version being used. Even if it was exposed it might be nice to script the consolidation when needed.

The script will check if the VM that is to be consolidated is powered off and has a chain length longer than 1. If either of those checks fail it will return an error message. The script can be found in my gitlab blogpost files. Here is the output of it in action reporting my VM is powered on!

****************************************************************
Input the Org name: bblab
Input the vApp name: bbvapp
Input the VM name: bb

****************************************************************

****************************************************************

Could not consolidate VM bb within vApp bbvapp since it is PoweredOn

Please power off this VM and run script again
****************************************************************

Here is the expected return when it works!

****************************************************************
Input the Org name: bblab
Input the vApp name: bbvapp
Input the VM name: chain-test

****************************************************************
Processing VM: chain-test
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 0 / State: Running
chain-test - Percent: 100 / State: Success

 

Hopefully this is helpful!