Ohio VMUG UserCon presentation: Upgrade to 6.5 with Zero Downtime

A little while ago I was able to fill in for a colleague of mine(@TomRalph) and present on his original VMworld session “Upgrading to vSphere 6.5 with Zero Downtime” at the Ohio VMUG/UserConference. It was a fantastic experience and my first time presenting at a VMUG/UserConference.


Being my first time presenting at a VMUG I was a bit nervous. I had to present twice that day and was lucky enough that another colleague of mine, Simon was presenting there as well and gave me plenty of pointers before and after my presentation, for which I am incredibly grateful!

Overall it was an amazing experience and if you haven’t had a chance to present and share your knowledge it is well worth it. Everyone involved was amazing to work with and encouraging to give back to the vcommunity.  Everyone wants you to succeed up there and to hear your experiences and story!


vCenter Host Profile compliance iSCSI Configuration error “Could not select an iSCSI HBA for the profile instance”

Today I ran into an issue that I could only find a partial answer to on the internet. My question was how to “remediate” this host profile failure in vCenter:

“Could not select an iSCSI HBA for the profile instance: 08756bda103420a7485c6cb8c2cb79b4cacbbab40be1830e9e06bfefaef25cc3. One of following may be true: (1) Some of the required user-input host customization settings are missing for the host (2) The system does not have a matching hardware for the given ‘Initiator Selection Policy'”

I attempted to un-check these adapters causing the host profile compliance failures. That did not actually help. The compliance failure was still there.

I decided to take a backup of the host profile config just in case something went wrong and delete these from the profile. These are located in the host profile under “Storage configuration –> iSCSI Initiator Configuration –> Dependent Hardware iSCSI Adapters”. For my issue I had two that were not in use. I deleted the adapters from the profile, checked compliance and all was green!!



NSX Backups failing “HTTP request time out”

“Server failed to respond. HTTP request time out.”

I’ve been seeing this more frequently and thought I’d post something about how to fix this error that may happen when you click on the “Backup & Restore” button on the home page of NSX. It can affect the list of backups from loading and the backup jobs from running altogether. As you can see below NSX failed to load all of the backups in the history:


I have seen in the past a limit of 100 backups listed in history before performance of NSX listing those can be affected. I will try to link official documentation as soon as I find it.

I went and checked how many backups I had and found….. 830!! Just slightly over the (assumed) recommended amount of 100…

[root@ifitisnotbroken-file1 NSX1]# ls -lath | wc -l
[root@ifitisnotbroken-file1 NSX1]#

To clean this up, I ran the following command which was take from here:

find /ifitisnotbrokenbackups/NSX/NSX1-* -mtime +20 -exec rm {} \;

It finds any file that is over 20 days old(the +20 part) and will remove them. After I ran that command I had 82 files left:

[root@ifitisnotbroken-file1 NSX1]# ls -lath | wc -l
[root@ifitisnotbroken-file1 NSX1]#

I wanted to lower the number of files/backups so I didn’t have to do the cleanup process as often. I ran the above command but changed the days to 10 and I was left with 43 files. I refreshed the backup page in NSX and it took less than 17 seconds to load and my backups started working without issue:


Hope this helps and please let me know if you have any questions!


Why get VMware Certified and keep it current?

Today I decided to take a step back and think why I got certified in the first place(way back in 2009). People have very different motivations for getting certified, mine was not related to monetary gain or advancement at my workplace when I first (started using haha – they are a bit like an addiction) took a VMware exam.  I worked for a very small company and I wanted a way to prove to myself that I knew what I was actually doing; also prove to my employer that I knew what I was doing!

Enter the VCP-4 exam, I felt this was the perfect way to show that I did know all of the features, such as how to configure ESX(i)/vCenter, which license level gave you which specific features and most importantly what all of those cool features did/could do.  Working for a small company I did not have the luxury of getting exams paid for sadly, so when I decided to take my exam I needed to make sure I was ready, had put the time in to study and find every resource possible to learn all that I could.

My first attempt I was very nervous and unsure even though I had spent 2 months going over things I thought I would need to know. I failed, I not only failed I bombed it. I did not let it get my down for long and after that I decided to start fresh and go over this “blueprint” thing that I had downloaded but did not pay (enough)attention to. I began reviewing it and looking over various breakdowns of the blueprint on blogs like this one from Simon Long. I took another 3 months to study, really take time to go over the blueprint, and feel comfortable with all topics. During that 3 months I spent time reading over VMware documentation that was relevant to the blueprint and learning everything I could.

I lined up another exam date and was ready to give it a go after months of study time. I was still nervous but felt much better about taking it this time. I had gained more confidence after putting in the study time and felt like I could really do this(now that did not mean that I didn’t get a little flustered whilst taking the exam). I passed this time with flying colors. I felt on top of the world after that.

Moving on from the first pass:

I really started reading more and more after passing that first exam. I setup a home lab and found many answers to my questions on the VMware community page, and on Twitter. The community that was around at that time was already really amazing as people were so helpful and eager to answer questions.

Did this actually help you or your career and why keep taking them?

So, why take these exams and put all of this time in? Well, again I started this to justify my knowledge to myself really. I kept taking these over the years as the versions changed and new exams came out; I wanted to push myself to learn more/new technologies and show it by passing these exams. By passing these exams I got an opportunity to interview for an amazing position. Part of why I got the position is I took the time to pursue these certs on my own and kept learning. It made a huge difference in my career passing these certs and moving into a role where I have had and still have tons of room to grow.

Looking back at this first exam I really learned what VMware was expecting and that you can’t ignore the blueprint!! The issues I have/had thinking about this one is that you had to memorize so many config min/max settings that many people found to be useless. Moving on from version 4 I feel that VMware has correct this issue and is testing for much more “real life” applicable knowledge from people attempting these tests. I also felt that it was a huge benefit to setup a home lab going forward with my career and future exams. Having taken many more exams since the VCP4 I really have a good process for prepping for the exam which includes reviewing the full blueprint. A fantastic example of covering the blueprint is what Mike Preston did on his blog  when he covered his 8 weeks of VCAP which I used for my first VCAP!


Never stop learning!


Photon OS vCenter 6.5 deleting EAM folders in /tmp

To anyone that runs into an issue where hosts fail to get prepared with VXLAN by NSX, hopefully this post will help you out. This specific issue a very wise colleague(Mr.Sage) found, is when EAM(ESX Agent Manager) folders get deleted within the /tmp directory in the Photon OS 6.5 vCenter and that causes your hosts to not get prepared by NSX with VXLAN until a workaround is put in place or your restart EAM.

The good news is that there is a workaround(Please note I’m not expert on this and implementing this is done at your own risk

  1. First, as noted above you can simply restart EAM. Seems easy enough but how often do you reboot a host and how often do you really want to restart EAM?
    1. If you do want to restart EAM you can simply use the following command to check the status/stop/start vmware-eam

# Use this to check the status of EAM and simply change the "--status" to "--start" or "--stop"

service-control --status vmware-eam

  1. The other work around is to create a new file under this directory: /usr/lib/tmpfiles.d named tmp-eam.conf with the following contents:

# Exclude the following for EAM service
x /tmp/eam*

The above would allow the eam files to stay around until the system is rebooted. Once the vCenter is rebooted EAM would be restarted anyway and the files would be recreated.

Hope this helps and ping me with any feedback or questions

NSX API tips and guides

Lately I’ve been working more with the API for multiple versions of NSX with my colleague @VirtSouthWest. Here are a couple of of API calls that we have been using which are something I’d like to keep track of for future configurations and hope they help someone else:

To get started you need a REST API Client/plugin, here is one I use that works with FireFox – RESTCLIENT

Once you have that installed you are ready connect to your NSX manager. If you have a self signed cert you may need to go to the NSX Manager and accept the “not secure connection”. That is something good to check if you get a response like the one below:


Once you accept the security warning you and ensure you have the correct Authorization and Header in place you should get a 200 OK response as shown below:


Here is a sample configuration of what you would send to a NSX manager API to configure Syslog. Make sure to specify the protocol TCP/UDP) and which port you have your syslog configured on, the standard being 514.

<port>514</port> - Port Configured on your Syslog Server
<protocol>UDP</protocol> - TCP/UDP

Here is a sample configuration of what you would send to a NSX manager API to configure NTP, you can configure 2 NTP servers using IP or FQDN which is great for redundancy.

<string>NTPServer-IP1</string><string>time1.google.com</string> - You can configure 2 NTP Servers

From the limited experince I have the backups are small ranging from 10-40MB.

Please note that once they reach their destination you configure they stay there and NSX does not currently clean up the backups. Meaning if you configure a backup job to run daily, after 1 year you will have 365 backups. This can take a while to load on the backup/restore screen. Please configure a job on the destination end to cleanup the backup jobs as needed. NSX will reflect these backups being gone and the list will be come shorter/load faster.

Here is a sample configuration of what you would send to a NSX manager API to configure scheduled Backups. In the example I have the time scheduled for 19:50 and for each manager you can configure the backup time. I have mine set to be staggered every 5 minutes.

Replace the following fields(Snip from the API Guide 6.2 below):

transferProtocol: FTP, SFTP

frequency: weekly, daily, hourly

dayOfWeek: SUNDAY, MONDAY, …., SATURDAY(Not in my example below)

Hour of Day: [0 ‐ 24 [  Minute of hour: [0 ‐ 60 [


The tables specified in the excludeTables parameter are not backed up.

<passPhrase>passPhrase</passPhrase> - For the backup file to restore

There are many of other things you can do via the NSX API and the above are just some some calls to get started. You can create controllers, controller backups, edges, etc.. Please review the guides below for the version you have.

API Guides link for different versions:

NSX 6.0.4 Guide

NSX 6.2 Guide

vExpert 2016


As you could probably guess I am a vExpert 2016! I’m very thankful and honored to be part of this group. Already getting emails/info/perks from the community and Corey Romero is pretty amazing. One of the best perks being a FREE year of PluralSight access.

I can’t thank Corey and everyone else involved enough for all of the hard work they put in to make this program happen. Congrats to all of the other 2016 vExperts!! The full directory can be found here https://communities.vmware.com/vexpert.jspa and I can’t help but link to my own listing as I’m very proud of this – Me!