During a normal upgrade to 10.2.2 that has been completed many times without issue, a colleague of mine noticed a single cell in an “Inactive” state in the UI:
This was not apparent post upgrade, until the service of that single cell were restarted. Attempting to fix this cell, the service was restarted which made no difference(failed to start), also rebooting the cell did not fix the issue. *Rebooting additional cells resulted in those cells failing to start and being Inactive as well
Next step was to dive into the logs and look to see why the service was not starting and the cell was showing inactive but nothing jumped out as “the problem”. Working with a very diligent support engineer, the following was found:
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for service: 'filterConfigurationService', objectClasses='[interface com.vmware.vcloud.security.filters.FilterConfigurationService]', filter='(objectClass=com.vmware.vcloud.security.filters.FilterConfigurationService)'
at com.vmware.vcloud.common.service.OsgiServiceReferenceFactoryBean.getObject(OsgiServiceReferenceFactoryBean.java:234)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
This was leading down a path of an issue that had been seen before where the “jms.user.system.password” was blank within the database. To validate, run the following:
select * from config where name = 'jms.user.system.password';
If the above does not return any value, the password is not set.
To fix this issue, create a new temporary database. First create a database template
#Create a template based on your existing DB
CREATE DATABASE dbname TEMPLATE template0;
#Create a temp DB based on template:
createdb -U test -T template0 vcddb-temp
#Restore DB backup pre upgrade to temp DB created above:
psql -U test vcddb-temp -f /path/to/vcddb-backup.dump.out
Run the above command to select the password, using that password run the following insert command:
insert into config (config_id,catvalue,name,value,sortorder) values ('588','vcloud','jms.user.system.password','xxxxxxxxxx','0'0);
Restart the cells and they should come back up as expected!
