VCAP 6 – Datacenter Virtualization Deployment

After passing the 3V0-624 (VCAP 6.5 Design) I want to pursuit the VCIX6-DCV certification, so the next exam I need to take is the 3V0-623 VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam. Since the design exam has been upgraded to vSphere 6.5, I went to the VMware Education booth at VMworld to ask if they knew if the deploy exam is stil based on 6.0. And it should still be. VMware Education could not tell me if it will be upgraded, and if so, when. But it shouldn’t happen anytime soon. So I need to prepare with the 6.0 version of vSphere. Sadly no HTML 5 client…

I did get some pointers on the exam:

  • It is based on the hands on labs, but the Control, Alt and Backspace keys are disabled. This means that you can’t use the MKS client, since you can’t release the focus of the window with CTRL+ALT, use the web console!
  • You can go back and forth between questions.
  • Even questions that you can’t complete, will be scored on the parts you did manage.
  • Scoring is done by a script that simply checks if settings are as desired, a sort of desired state configuration check.
  • Deploying the lab environment will take some time, use this time to setup the screen size of the controls.
  • vCenter will take even longer to come online, use this time to enable SSH on the ESXi hosts.
  • It’s a single lab environment during the entire exam, so if you come up with an answer to a previous question, you didn’t manage earlier, you are always able to answer it.
  • The allotted time still is an issue, there is simply not enough time to comfortable answer each question.
  • Screwing up the ESXi host networking is less likely, since the automatic revert of settings if the host becomes isolated. I’ve read that this occurred to some in the 5.x edition of the exam.
  • PowerCli should be available, for me this is important since I use it allot.
  • Use Hands on Labs. Not only to get a feel for the exam environment, but to get proficient with the tasks that you may not do on a daily basis.
    • I don’t have a list of HoLs just yet. I haven’t really started with my studies, but wanted to document the points from VMware Education, before I forget them.

Read the Exam Guide and the Platform Interface guide.

There still is no official Certification Guide from VMware Press like there was for 5.x, nor will there be anytime soon. But there is an unofficial ‘VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam Preparation Guide’ written by Ramy Mahmoud. I haven’t been able to read it, but judging by the first few pages, it should be awesome.

Schermafbeelding 2018-11-09 om 22.13.16


3V0-624: VCAP 6.5 Datacenter Virtualization Design my exam experience

VMworld Barcelona 2018 started for me with a personal victory, I passed the 3V0-624 exam!

Schermafbeelding 2018-11-09 om 21.23.25

My exam experience was great, started my exam at 9 AM, (my appointment was a half hour later, but since there were plenty of seats available, I could start early. 30 minutes less to be a worried nervous wreck. I feel that I didn’t give myself enough time to properly prepare for this exam, but then again, isn’t that always the case? Just schedule an exam date far away, to use the date as the motivator to at least try to put in some effort… But with VMworld exams rescheduling is not really possible, so I just went for it.

The exam consisted of 60 questions, either multiple choice or drag and drop, no Visio style infrastructure quizlets. The time allotted was fine for me, I had ample time to read and re-read all questions. Yes it will involve some reading, since lots of (background) information is presented about the design in question.

Many questions weren’t so much technical in nature, but ask you the define a statement to be a functional or non-functional requirement. Or if something should be classified as being a risk, constraint or an assumption. Some questions ask to sort requirements by stakeholder. So know how to identify stakeholders, and have an idea about the common tasks or interests are for C-level executives, such as CEO, CIO, CISO and my wife CFO.

Oh and the rule that with a 4 answer multiple choice question, 2 choices are evidently ridiculous, will not apply in this exam. Many of the answers seem valid! Make sure to re-read the question to see, if something that seems valid enough may be invalidated by some of the wording in the question. For every answer you give, don’t just click it because it’s the first thing that comes to mind, but try to defend you answer against the question. Think “Why is answer X wrong?” or “Why does answer X fit better than Y?”

“Dear algebra, please stop asking us to find your X, she’s never coming back and we don’t know Y” – From somewhere on the internet.

In this exam you must know why. To get a taste of it, visit vMusketeers (See below)

But to prepare for the exam, a lot of reading is required anyways…

What I did to prepare for this exam:

  • Read books
    • VMware vSphere 6.x Datacenter Design Cookbook Second Edition by Hersey Cartwright ISBN: 9781785283468
    • Essential vSAN 6.2 by Duncan Epping and Cormac Hogan. An updated version may come soon, Duncan tweeted about it a few days back!
    • Host Resources Deep Dive 6.5 by Frank Denneman and Niels Hagoort. ISBN: 9781540873064 or get a free digital copy from Rubrik
    • vSphere HA DeepDive 6.0u1 by Duncan Epping which isn’t available anymore, but the content is updated and combined with the updated Host Resource Deep Dive 6.5 book, in the VMware vSphere Clustering Deep Dive 6.7 book. I haven’t read it yet. But picked up a signed copy at VMworld, again sponsored by Rubrik, if history truly repeats itself, they may provide a free ebook version soon, so check the Rubrik site regularly!
  • The VMware Exam Guide, which contains links to allot of (VMware) Documentation, which may not be the most fun to read, but as always in a VMware exam, know the configuration maximums, limits, product compatibility and caveats for each listed product. Knowing which operating systems can or can’t be converted on which platform using VMware Converter could win you a couple of points. Know the product suite on a high level, have intimate knowledge on all things vSphere. The guide contains 10 practice questions, which are somewhat easier than the real exam, at least in my experience. All are solely of the multiple choice variety. Beware: Even if you ace these, don’t think you’ll automatically ace the exam.
  • Watched a few recordings of the VCAP6.x Design sessions from vBrownbag. These are awesome, not only because of the technical content, but especially for the explanations of the differences between Functional and Non-Functional requirements, Risks, Assumptions, Constraints, which are very important in the exam. (I really loved the videos on Section 2.1 and 3.1
  • Tested my understanding with the VCAP 6.x DCD mock exam from vMusketeers. The score of this mock was surprising accurate compared to my actual exam score, and the type of drag ‘n drop questions are quite similar to those of the exam. I didn’t see any re-order questions in my exam, but that doesn’t necessarily means that there aren’t any.
  • Blogs of each of the gentlemen reverenced above. Don’t remember the exact blog posts, but many.

Thanks to all those people that make such an effort writing books, blogs and hosting/presenting these sessions, I wouldn’t know how I could have accumulated this amount of knowledge in such a short time without all your hard work!

New VMware Skill, VMware Practitioner – NFV Operations, and you can get it for free!

A few days ago I saw a tweet from Eric Sloof about a new VMware Skill badge one could earn for free.

So I decided to go for it. Followed the also free online course VMware vCloud NFV Foundations with vCloud Director [V8.x]

And sat the VMware vCloud NFV with vCloud Director [V8.x] Test, I failed it the first time, you need to score at least 80%. But you do have 3 attempts, so the next day after reviewing my mistakes, I tried again, and I passed!

I must say the test is more trying than the course made me expect, the course doesn’t tell you allot, you need to have some knowledge of vCloud Director, vSAN, NSX and of course vSphere. Resource management and update/migration of (vCloud) infrastructures are very important topics in the tests I’ve done, however you will need solid knowledge of NSX and vSphere/vSAN! Allot of detail questions, the course focuses on NFV stuff that I really couldn’t match with the questions in the test. There is a module in the course outlining some basics about vSphere, NSX and vSAN, but not nearly detailed enough to pass the test.

For those of you who have done VTSP tests, this is much the same, it is using the same test engine, and you cannot go back to a previous question. With the score report, you will get a list of questions you got wrong, with your answers, but only those! There are some multi select questions which won’t tell you how many ‘right’ answers you need to pick from the list. So you won’t know if the question is wrong because of a wrong answer, or if it is due to many or to few selected options… Only that you didn’t receive it as 100% correct…

For some reason I wasn’t issued the badge, contacting VMware Education support didn’t really help, however they pointed me to Acclaim’s support, who seemed very willing to find out what went wrong. Today I finally received notice of the badge being issued.

vmware_Skill_Practitioner_NFV (1)


Daisy Chaining VMware UMDS

For a design question I was wondering if one could daisy chain multiple VMware Update Manager Download Service (UMDS) appliances. The documentation doesn’t say a word about it. The only thing I found Googling this was one blog that say’s it can’t be done. But that blog was from 2014, now, 2018, let’s see…

I started with a Ubuntu 16.04 LTS server and used William Lam’s script to install. It needed some more config:

(I tested this in Fusion virtual machines using vSphere 6.5 Update 1 (5969303)

Open the console of the first UMDS

sudo -i
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
apt-get install openssh-server -y

This gives the IP address (Ubuntu in Fusion creates ens33 interface)

Using an SSH (Windows: PuTTY / Mac: Termius) client, connect to the UMDS:

ssh vmninja@

In the SSH session

sudo -i
chmod +x
./ /mnt/cdrom/umds/VMware-UMDS-6.5.0-5939545.tar.gz UMDSDB UMDS_DSN umdsuser VMware1!
/usr/local/vmware-umds/bin/vmware-umds -v
/usr/local/vmware-umds/bin/vmware-umds -G
/usr/local/vmware-umds/bin/vmware-umds -S --add-url --enable-host --url-type HOST
/usr/local/vmware-umds/bin/vmware-umds -D
(In the example above I’ve added the HPE VibsDepot, to see if non-firstparty updates will get downloaded.
It will now start downloading… It takes some time, at the time of writing it was about 65 GB
Per William’s suggestion for the test I used Pythons buildin webserver:
apt-get install python-minimal -y
cd /var/lib/vmware-umds
python -m SimpleHTTPServer 80
Using this as a foreground task, it shows all HTTP requests being received: - - [18/Feb/2018 05:28:19] "GET /hostupdate/HPQ/ HTTP/1.1" 200 - - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 - - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/ HTTP/1.1" 200 - - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 - - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/ HTTP/1.1" 200 - - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/ HTTP/1.1" 200 - - - [18/Feb/2018 05:28:20] "GET /hostupdate/vmw/ HTTP/1.1" 200 - - - [18/Feb/2018 05:28:20] "GET /vaupgrade/bootstrap_index.xml HTTP/1.1" 200 - - - [18/Feb/2018 05:28:20] "GET /vaupgrade/__valm-consolidated-index__.xml HTTP/1.1" 200 -

But first I needed to build a second UMDS, mostly the same as above, but:

  1. I didn’t add the HPE VibsDepot
  2. I pointed and to localhost, by editing the /etc/hosts file to prevent it to try to download anything directly from’s website.
    • For some reason it is not possible to remove these entries from UMDS’s config, or to remove the older versions… I don’t need updates for any ESXi prior to 6.5….
    vmninja@ubuntu:~$ cat /etc/hosts       localhost       ubuntu
    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
  3. Instead of the HPE VibsDepot, I added the first UMDB as source:
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url --enable-host --url-type HOST
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url --enable-va --url-type VA
  4. Start the download task:

    /usr/local/vmware-umds/bin/vmware-umds -D

After downloading was complete, I added the second UMDS to vCenter Update Manager:


After clicking download now, the Non VMware Patches are visible in vCenter:


So it seams to be possible to daisy chain UMDS. I have no idea about the supportability of this configuration, and if using something in production, use a real webserver!

How to reinstall VMware Tools on a ParaVirtual VM

At a customer location I was asked to reinstall VMware Tools on machines, normally the customer would do this their selves, but after uninstall of the VMware Tools, the VM wouldn’t boot. They needed to reinstall the VMware Tools because of the issue encountered described in  VMware KB2063887

The issue is quite simple, it’s a chicken-egg kind of thing. VMware Tools includes the device driver for the PVSCSI Controller, without VMware Tools, Windows lost its driver for the PVSCSI controller and can’t boot from it.

So I created this how to for the customer, and thought it might be useful for others.

So what’s the trick. Actually if you ever changed a VM to use the ParaVirtual SCSI Controller, you boot the VM using it’s ‘normal’ SCSI controller, you add the PVSCSI and a (temporary) hard disk to it, wait for Windows to install all drivers and ‘see’ the hard disk connected to the PVSCSI. You then shutdown Windows, remove any temporary hard disk, remove the extra PVSCSI controller and change the type of the original controller to PVSCSI.  This is mostly doing the opposite:

This is done on Windows Server 2012R2, the process should be much the same on other versions.


Device Manager showing VMware hardware devices in default view.


Device Manager showing VMware hardware devices in Device by connection view.

  1. Add Temporary Hardware
    1. While the Virtual Machine is running, using the vSphere Client, select the Virtual Machine and open Edit Settings…20171018_003
    2. From the Pull-Down Menu New Device, select SCSI Controller
    3. Click Add, a new LSI Logic SAS Controller will be added.
    4. From the Pull-Down Menu New Device, select New Hard Disk
    5. Click Add, a new 40 GB Hard Disk will be added.
    6. 20171018_004

    7. From the Virtual Device Node Pull-Down Menu select New SCSI controller
    8. (Optional) Adjust the size of the new Hard Disk from 40 GB to 1 GB
    9. (Optional) From the Disk Provisioning Pull-Down Menu select Thin provision
    10. Click OK
    11. Almost instantaneous the added hardware is visible in Windows:


      Device Manager showing the newly added LSI Adapter, SAS 3000 series, 8-port with 1068 SCSI controller and the added VMware Virtual disk SCSI Disk Device hard disk connected to it. The original two disks are still connected to the VMware PVSCSI Controller


      Disk Manager showing the added Hard Disk. It does not have to be partitioned or formatted.

  2. Reboot
    1. Reboot the Virtual Machine.
    2. When the Virtual Machine starts, Windows will commit some changes to the device drivers.
    3. 20171018_008

      Windows is installing drivers for the added hard disk.

      When you are quick enough you might see the Plug and Play Device Setup running installing the hardware. (On my example server it took only a few seconds to install.)

      For those interested, the Kernel-PnP log shows what happened:


      GridView showing Kernel Plug & Play events created by the following PowerShell command:

      Get-WinEvent -LogName Microsoft-Windows-Kernel-PnP/Configuration |
      Where-Object { $_.TimeCreated -ge ( (Get-Date).AddHours(-1) ) } |
      Select-Object TimeCreated,Message | Out-GridView
    4. Shut down the Virtual Machine
    5. TL;DR
    6. Important: If you should skip the reboot step and just shut down the Virtual Machine and start modifying the Virtual Hard Disk Configuration, you’ll be presented with a BSoD boot loop. “Your PC ran into a problem and needs to restart. We’re just collecting some error info, and then we’ll restart for you.”

  3. Modify Virtual Hard Disk Configuration
    1. While the Virtual Machine is shut down, using the vSphere Client, select the Virtual Machine and open Edit Settings…
    2. In my example the Virtual Machine started with 2 original Hard Disks before the third one was added. Since only the first two disks are showed, to remove the third Hard Disk first we need to click on Manage other disks.
    3. In the Pop-Up window, hover above line Hard disk 3, and a gray icon with a white X is shown. Click this icon.
    4. The line changes and shows a checkbox Delete files from datastore. As long as you are sure that you are removing the newly added disk, check the checkbox.
    5. 20171018_011

    6. Click Close
    7. Back in the Edit Settings window, click OK
    8. Important: Only after closing Edit Settings with OK, the changes will be applied. If not applied the following steps cannot continue.

    9. Open Edit Settings
    10. Click the little triangle shaped toggle in front of each Hard disk to open the advanced settings.
    11. From the Virtual Device Node Pull-Down Menu select SCSI controller 1
    12. Important: The Logical Unit Number should stay the same, only the adapter number may change:

      • SCSI(0:0) must become SCSI(1:0).
      • SCSI(0:1) must become SCSI(1:1).
      • Etc.
    13. Click OK
    14. Start the Virtual Machine.
    15. Check if all disks are only connected to the LSI Adapter using Device Manager:
    16. 20171018_012

  4. Remove VMware Tools
    1. Click Start -> Control Panel -> Program and Features
    2. Select VMware Tools and click Uninstall.
    3. Click Yes
    4. When the Uninstall is finished, reboot the Virtual Machine20171018_014
    5.  When the Virtual Machine is restarted, the Device Drivers for the VMware Paravirtual devices are missing (Unknown devices):


  5. Install VMware Tools
    1. While the Virtual Machine is running, using the vSphere Client, select the Virtual Machine and click Guest OS -> Install VMware Tools…
    2. 20171018_016

    3. If the VMware Tools Setup is not started automatically, click Choose what to do with this disc Pop-Up.
    4. Click Run setup64.exe
    5. 20171018_017

    6. Install the VMware Tools using all defaults, unless you have a reason to change them.
    7. Click Next >, Next >, Install, Finish
    8. 20171018_018

    9. After the installation is finished, reboot the Virtual Machine
    10. 20171018_019

    11. After the reboot, shut down the Virtual Machine
  6. Reconfigure Virtual Hardware (Change back to Paravirtual SCSI Controller)
    1. While the Virtual Machine is shut down, using the vSphere Client, select the Virtual Machine and open Edit Settings…
    2. Click the little triangle shaped toggle in front of each Hard disk to open the advanced settings.
    3. From the Virtual Device Node Pull-Down Menu select SCSI controller 0
    4. Important: The Logical Unit Number should stay the same, only the adapter number may change:

      • SCSI(1:0) must become SCSI(0:0).
      • SCSI(1:1) must become SCSI(0:1).
      • Etc.


    5. Click OK
    6. Important: Only after closing Edit Settings with OK, the changes will be applied. If not applied the following steps cannot continue.
    7. Open Edit Settings
    8. Hover above line SCSI Controller 1 LSI Logic SAS, and a gray icon with a white X is shown. Click this icon.
    9. 20171018_021

    10. The line changes and shows Device will be removed.
    11. 20171018_022

    12. Click OK
    13. Start the Virtual Machine. 

VMworld Certifications

Each year at VMworld there are 50% discounts on exams. And since my VCP DCV 6.0 certification was about to expire… End of May the news came that I failed the 6.5 beta exam in januari, so needed to sit the exam again. But at least the results came quicker than the results of the 6.0 beta exam, which took almost 8 and a half months to get.)

So with the discount I decided to retry the exam at VMworld. I figured monday (Partner day) would be a good time to take it, so I scheduled the exam at 11:00. I wasn’t the only one who figured this, and the examroom was full. Which actually was a blessing, but more on that later.

My VCP 6.5 (2V0-622) experience.


The exam consisted of 70 multiple choice questions. You have the ability to go back to any previous question. Which can be helpful sometimes the answer of a question is given in another question so going back and forth gives you an advantage. There is no calculator available, and some questions require some calculus. Don’t worry too much, the questions I got were quite easy to solve with the use of the pen and paper. But I can imagine that when one is very nervous, this may add to the pressure.

Next to this, the number of typo’s, especially the lack of spaces was really bad. This is something that shouldn’t happen. Simply because one could loose points if a question is regarding an exact command line command, and the spaces, or lack there of, will invalidate the command on the commandline itself. So if one is too critical the correct answer is not listed… However in most cases choosing the most likely should be ok. I got 450 point out of 500, so approx 90% correct. (I know the scoring is calculated by a bunch of witches flying on broomsticks and it involves in which house the moon and Mars are present.) but it worked for me.

You are able to comment on each question. I’ve done so for a few.

Blessing in disguise

I promised to get back to the blessing in disguise of the full exam room. It allowed me the time to talk to other people waiting for their spot. I ended up talking to @eelkehagen who was waiting for his chance on the 2VB-601 (vSAN Specialist 2017) exam. His enthousiasme and the fact that when you take the exam at VMworld you get a free t-shirt made me want to do the exam as wel.


(image courtesy of VMware)

So I decided to read the Exam Preparation Guide and it contains 10 mock questions which I tried and scored over 70% correct. At VMworld the new vSAN 6.6 Hands on Labs were available so I sat the HOL-1808 and just went for it. The worst that could happen is that I failed the exam, so why not…

My vSAN Specialist 2017 (2VB-601) experience.


This exam is like any other, 60 multiple choice questions. You are able to go back and forth between questions, again no calculator available, but also no comment option. I never sat a VMware exam before that didn’t include the comment option. Maybe something went wrong, maybe it just doesn’t include the feature.

The exam doesn’t extend the validity of the VCP status, however it does not expire, and doesn’t feature an certifcate, only the badge. (Avaliable thru

However the exam is quite doable when you have:

I actually never used vSAN in an production environment, only in my homelab and HoL.

I had one question in the exam that reverencing a product of a third party, which a had to guess since I never heard of the product mentioned. There were quite a few questions about designs if they are valid or not. So having a good idea about all the configuration maximums is a must. Anyways: I received my T-shirt!


Me and Peter van de Bree (@PvdBree) showing off my T-Shirt @VMware Education booth




Homelab upgrade to 10 Gbps for less than $1000 /

I’ve finally installed my new Layer 3 10Gbps switch. An Ubiquiti EdgeSwitch 16-XG. This switch with 12 SFP+ ports and 4 RJ45 ports and is capable of Layer 3 routing. This device is available around 550 – 600 euro (roughly 650-700 US dollar)



I bought some second hand Mellanox  ConnectX-2 10Gbps SFP+ cards on Ebay for around   $20 each and also found Cisco H10GB CU3M 3meter Twinax SFP+ cables costing around $7 a pop on there as well.

So what does this bring, firstly the Mellanox cards are listed as compatible with ESXi 6.5 GA and U1 so after installation, they got recognized instantly.

After configuring a VMkernel port on it to test a vMotion from a single SATA SSD from Host01 to the same SSD in Host02 I saw this troughput:

WhatsApp Image 2017-08-29 at 08.16.00

It’s maxing out at 249 Megabyte per second transfer speed.

Mind you this is done with Intel Avoton servers, and some Kingston SSDNow V300 SV300S37A/60G SSDs. I was impressed!

And as an added benifit I could say goodbye to my trusty old Vyatta VM acting as router. The Ubiquiti switch is doing the routing now.

Few remarks: The switch can out of the box only be managed using the webbrowser to its default IP address ( even the console port is disabled out of the box. But since there is no console cable included it only bothers those idiots that have Cisco DB9-RJ45 console cables laying around…

But if you are accustomed to Cisco CLI, the CLI is compatible. No webadmin, CLI FTW.

I’ve broken the LACP LAG between my two Zyxel GS1910-24 switches, and connected each to the Ubiquiti using LC-LC Fiber and 1GBps SFPs. I’m geeking out with this new setup!


When I get the time I’m gonna try to setup LACP LAG between the Ubiquiti and each of the Zyxels to get 2 GBps of uplink bandwidth to the Zyxel switches, and maybe do some cable management 🙂


I know, it should be much neater, but to be perfectly honest I didn’t even got to bolt down the Ubiquiti (this is an older picture, before I got the Ubiquiti.)