RBD Health Alarm on vCenter 6.7

01a

vCenter greeted me this morning with a RBD Health Alarm. Not really sure what the issue could be, I just googled it. Sure enough, VMPros already blogged about it. It is about Auto Deploy. Something that is not used by the customer at this point, but is nevertheless enabled, and encountered some sort of error. The blog post points to VMware KB2000988. But it states:

“This article provides troubleshooting guidance on VMware Auto Deploy 5.x. This article does not apply to vSphere 6.0”

Continue reading

Advertisements

Remove Inaccessible datastore from inventory

DatastoreError

Datastore ‘{name}’ is not accessible. “No connected and accessible host is attached to the datastore.”

At a client’s we encountered a rather classical error, a dismounted datastore that couldn’t be removed/distroyed. So the customer just deleted the LUN from the storagebox expecting the datastore to disappear from the inventory. That didn’t happen. Neither did the datastore get removed after a reboot of the VCSA.

The customer was adament that everything from the datastore was moved to other datastores prior to the unmounting. The filesystem seems empty (except for the default folders (.naa.XXXXXXXXXX and .ssd.sf) And all of the VMs were XvMotioned to other datastores.

Two other datastores were assigned to be used by vSphere HA as Heartbeat Datastores

So why couldn’t it be removed? Continue reading

VCAP 6 – Datacenter Virtualization Deployment

After passing the 3V0-624 (VCAP 6.5 Design) I want to pursuit the VCIX6-DCV certification, so the next exam I need to take is the 3V0-623 VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam. Since the design exam has been upgraded to vSphere 6.5, I went to the VMware Education booth at VMworld to ask if they knew if the deploy exam is stil based on 6.0. And it should still be. VMware Education could not tell me if it will be upgraded, and if so, when. But it shouldn’t happen anytime soon. So I need to prepare with the 6.0 version of vSphere. Sadly no HTML 5 client…

I did get some pointers on the exam:

  • It is based on the hands on labs, but the Control, Alt and Backspace keys are disabled. This means that you can’t use the MKS client, since you can’t release the focus of the window with CTRL+ALT, use the web console!
  • You can go back and forth between questions.
  • Even questions that you can’t complete, will be scored on the parts you did manage.
  • Scoring is done by a script that simply checks if settings are as desired, a sort of desired state configuration check.
  • Deploying the lab environment will take some time, use this time to setup the screen size of the controls.
  • vCenter will take even longer to come online, use this time to enable SSH on the ESXi hosts.
  • It’s a single lab environment during the entire exam, so if you come up with an answer to a previous question, you didn’t manage earlier, you are always able to answer it.
  • The allotted time still is an issue, there is simply not enough time to comfortable answer each question.
  • Screwing up the ESXi host networking is less likely, since the automatic revert of settings if the host becomes isolated. I’ve read that this occurred to some in the 5.x edition of the exam.
  • PowerCli should be available, for me this is important since I use it allot.
  • Use Hands on Labs. Not only to get a feel for the exam environment, but to get proficient with the tasks that you may not do on a daily basis.
    • I don’t have a list of HoLs just yet. I haven’t really started with my studies, but wanted to document the points from VMware Education, before I forget them.

Read the Exam Guide and the Platform Interface guide.

There still is no official Certification Guide from VMware Press like there was for 5.x, nor will there be anytime soon. But there is an unofficial ‘VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam Preparation Guide’ written by Ramy Mahmoud. I haven’t been able to read it, but judging by the first few pages, it should be awesome.

Schermafbeelding 2018-11-09 om 22.13.16

3V0-624: VCAP 6.5 Datacenter Virtualization Design my exam experience

VMworld Barcelona 2018 started for me with a personal victory, I passed the 3V0-624 exam!

Schermafbeelding 2018-11-09 om 21.23.25

My exam experience was great, started my exam at 9 AM, (my appointment was a half hour later, but since there were plenty of seats available, I could start early. 30 minutes less to be a worried nervous wreck. I feel that I didn’t give myself enough time to properly prepare for this exam, but then again, isn’t that always the case? Just schedule an exam date far away, to use the date as the motivator to at least try to put in some effort… But with VMworld exams rescheduling is not really possible, so I just went for it.

The exam consisted of 60 questions, either multiple choice or drag and drop, no Visio style infrastructure quizlets. The time allotted was fine for me, I had ample time to read and re-read all questions. Yes it will involve some reading, since lots of (background) information is presented about the design in question.

Many questions weren’t so much technical in nature, but ask you the define a statement to be a functional or non-functional requirement. Or if something should be classified as being a risk, constraint or an assumption. Some questions ask to sort requirements by stakeholder. So know how to identify stakeholders, and have an idea about the common tasks or interests are for C-level executives, such as CEO, CIO, CISO and my wife CFO.

Oh and the rule that with a 4 answer multiple choice question, 2 choices are evidently ridiculous, will not apply in this exam. Many of the answers seem valid! Make sure to re-read the question to see, if something that seems valid enough may be invalidated by some of the wording in the question. For every answer you give, don’t just click it because it’s the first thing that comes to mind, but try to defend you answer against the question. Think “Why is answer X wrong?” or “Why does answer X fit better than Y?”

“Dear algebra, please stop asking us to find your X, she’s never coming back and we don’t know Y” – From somewhere on the internet.

In this exam you must know why. To get a taste of it, visit vMusketeers (See below)

But to prepare for the exam, a lot of reading is required anyways…

What I did to prepare for this exam:

  • Read books
    • VMware vSphere 6.x Datacenter Design Cookbook Second Edition by Hersey Cartwright ISBN: 9781785283468
    • Essential vSAN 6.2 by Duncan Epping and Cormac Hogan. An updated version may come soon, Duncan tweeted about it a few days back!
    • Host Resources Deep Dive 6.5 by Frank Denneman and Niels Hagoort. ISBN: 9781540873064 or get a free digital copy from Rubrik
    • vSphere HA DeepDive 6.0u1 by Duncan Epping which isn’t available anymore, but the content is updated and combined with the updated Host Resource Deep Dive 6.5 book, in the VMware vSphere Clustering Deep Dive 6.7 book. I haven’t read it yet. But picked up a signed copy at VMworld, again sponsored by Rubrik, if history truly repeats itself, they may provide a free ebook version soon, so check the Rubrik site regularly!
  • The VMware Exam Guide, which contains links to allot of (VMware) Documentation, which may not be the most fun to read, but as always in a VMware exam, know the configuration maximums, limits, product compatibility and caveats for each listed product. Knowing which operating systems can or can’t be converted on which platform using VMware Converter could win you a couple of points. Know the product suite on a high level, have intimate knowledge on all things vSphere. The guide contains 10 practice questions, which are somewhat easier than the real exam, at least in my experience. All are solely of the multiple choice variety. Beware: Even if you ace these, don’t think you’ll automatically ace the exam.
  • Watched a few recordings of the VCAP6.x Design sessions from vBrownbag. These are awesome, not only because of the technical content, but especially for the explanations of the differences between Functional and Non-Functional requirements, Risks, Assumptions, Constraints, which are very important in the exam. (I really loved the videos on Section 2.1 and 3.1
  • Tested my understanding with the VCAP 6.x DCD mock exam from vMusketeers. The score of this mock was surprising accurate compared to my actual exam score, and the type of drag ‘n drop questions are quite similar to those of the exam. I didn’t see any re-order questions in my exam, but that doesn’t necessarily means that there aren’t any.
  • Blogs of each of the gentlemen reverenced above. Don’t remember the exact blog posts, but many.

Thanks to all those people that make such an effort writing books, blogs and hosting/presenting these sessions, I wouldn’t know how I could have accumulated this amount of knowledge in such a short time without all your hard work!

New VMware Skill, VMware Practitioner – NFV Operations, and you can get it for free!

A few days ago I saw a tweet from Eric Sloof about a new VMware Skill badge one could earn for free.

So I decided to go for it. Followed the also free online course VMware vCloud NFV Foundations with vCloud Director [V8.x]

And sat the VMware vCloud NFV with vCloud Director [V8.x] Test, I failed it the first time, you need to score at least 80%. But you do have 3 attempts, so the next day after reviewing my mistakes, I tried again, and I passed!

I must say the test is more trying than the course made me expect, the course doesn’t tell you allot, you need to have some knowledge of vCloud Director, vSAN, NSX and of course vSphere. Resource management and update/migration of (vCloud) infrastructures are very important topics in the tests I’ve done, however you will need solid knowledge of NSX and vSphere/vSAN! Allot of detail questions, the course focuses on NFV stuff that I really couldn’t match with the questions in the test. There is a module in the course outlining some basics about vSphere, NSX and vSAN, but not nearly detailed enough to pass the test.

For those of you who have done VTSP tests, this is much the same, it is using the same test engine, and you cannot go back to a previous question. With the score report, you will get a list of questions you got wrong, with your answers, but only those! There are some multi select questions which won’t tell you how many ‘right’ answers you need to pick from the list. So you won’t know if the question is wrong because of a wrong answer, or if it is due to many or to few selected options… Only that you didn’t receive it as 100% correct…

For some reason I wasn’t issued the badge, contacting VMware Education support didn’t really help, however they pointed me to Acclaim’s support, who seemed very willing to find out what went wrong. Today I finally received notice of the badge being issued.

vmware_Skill_Practitioner_NFV (1)

 

Daisy Chaining VMware UMDS

For a design question I was wondering if one could daisy chain multiple VMware Update Manager Download Service (UMDS) appliances. The documentation doesn’t say a word about it. The only thing I found Googling this was one blog that say’s it can’t be done. But that blog was from 2014, now, 2018, let’s see…

I started with a Ubuntu 16.04 LTS server and used William Lam’s script to install. It needed some more config:

(I tested this in Fusion virtual machines using vSphere 6.5 Update 1 (5969303)

Open the console of the first UMDS

sudo -i
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
apt-get install openssh-server -y
ifconfig

This gives the IP address (Ubuntu in Fusion creates ens33 interface)

Using an SSH (Windows: PuTTY / Mac: Termius) client, connect to the UMDS:

ssh vmninja@172.16.250.129

In the SSH session

sudo -i
wget https://github.com/lamw/vghetto-scripts/raw/master/shell/install_umds65.sh
chmod +x install_umds65.sh
./install_umds65.sh /mnt/cdrom/umds/VMware-UMDS-6.5.0-5939545.tar.gz UMDSDB UMDS_DSN umdsuser VMware1!
/usr/local/vmware-umds/bin/vmware-umds -v
/usr/local/vmware-umds/bin/vmware-umds -G
/usr/local/vmware-umds/bin/vmware-umds -S --add-url https://vibsdepot.hpe.com/index.xml --enable-host --url-type HOST
/usr/local/vmware-umds/bin/vmware-umds -D
(In the example above I’ve added the HPE VibsDepot, to see if non-firstparty updates will get downloaded.
It will now start downloading… It takes some time, at the time of writing it was about 65 GB
Per William’s suggestion for the test I used Pythons buildin webserver:
apt-get install python-minimal -y
cd /var/lib/vmware-umds
python -m SimpleHTTPServer 80
Using this as a foreground task, it shows all HTTP requests being received:
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/HPQ/metadata-hpnmi-vmware55-bundle-2.3-6.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/csco-VEM-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/bootstrap_index.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/__valm-consolidated-index__.xml HTTP/1.1" 200 -

But first I needed to build a second UMDS, mostly the same as above, but:

  1. I didn’t add the HPE VibsDepot
  2. I pointed hostupdate.vmware.com and vapp-updates.vmware.com to localhost, by editing the /etc/hosts file to prevent it to try to download anything directly from VMware.com’s website.
    • For some reason it is not possible to remove these entries from UMDS’s config, or to remove the older versions… I don’t need updates for any ESXi prior to 6.5….
    vmninja@ubuntu:~$ cat /etc/hosts
    127.0.0.1       localhost
    127.0.1.1       ubuntu
    127.0.0.1       hostupdate.vmware.com
    127.0.0.1       vapp-updates.vmware.com
    
    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
  3. Instead of the HPE VibsDepot, I added the first UMDB as source:
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/hostupdate/__hostupdate20-consolidated-index__.xml --enable-host --url-type HOST
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/vaupgrade/__valm-consolidated-index__.xml --enable-va --url-type VA
  4. Start the download task:

    /usr/local/vmware-umds/bin/vmware-umds -D

After downloading was complete, I added the second UMDS to vCenter Update Manager:

UpdateManagerConfig

After clicking download now, the Non VMware Patches are visible in vCenter:

PatchRepository

So it seams to be possible to daisy chain UMDS. I have no idea about the supportability of this configuration, and if using something in production, use a real webserver!

How to reinstall VMware Tools on a ParaVirtual VM

At a customer location I was asked to reinstall VMware Tools on machines, normally the customer would do this their selves, but after uninstall of the VMware Tools, the VM wouldn’t boot. They needed to reinstall the VMware Tools because of the issue encountered described in  VMware KB2063887

The issue is quite simple, it’s a chicken-egg kind of thing. VMware Tools includes the device driver for the PVSCSI Controller, without VMware Tools, Windows lost its driver for the PVSCSI controller and can’t boot from it.

So I created this how to for the customer, and thought it might be useful for others.

So what’s the trick. Actually if you ever changed a VM to use the ParaVirtual SCSI Controller, you boot the VM using it’s ‘normal’ SCSI controller, you add the PVSCSI and a (temporary) hard disk to it, wait for Windows to install all drivers and ‘see’ the hard disk connected to the PVSCSI. You then shutdown Windows, remove any temporary hard disk, remove the extra PVSCSI controller and change the type of the original controller to PVSCSI.  This is mostly doing the opposite:

This is done on Windows Server 2012R2, the process should be much the same on other versions.

20171018_001

Device Manager showing VMware hardware devices in default view.

20171018_002

Device Manager showing VMware hardware devices in Device by connection view.

  1. Add Temporary Hardware
    1. While the Virtual Machine is running, using the vSphere Client, select the Virtual Machine and open Edit Settings…20171018_003
    2. From the Pull-Down Menu New Device, select SCSI Controller
    3. Click Add, a new LSI Logic SAS Controller will be added.
    4. From the Pull-Down Menu New Device, select New Hard Disk
    5. Click Add, a new 40 GB Hard Disk will be added.
    6. 20171018_004

    7. From the Virtual Device Node Pull-Down Menu select New SCSI controller
    8. (Optional) Adjust the size of the new Hard Disk from 40 GB to 1 GB
    9. (Optional) From the Disk Provisioning Pull-Down Menu select Thin provision
    10. Click OK
    11. Almost instantaneous the added hardware is visible in Windows:

      20171018_005

      Device Manager showing the newly added LSI Adapter, SAS 3000 series, 8-port with 1068 SCSI controller and the added VMware Virtual disk SCSI Disk Device hard disk connected to it. The original two disks are still connected to the VMware PVSCSI Controller

      20171018_006

      Disk Manager showing the added Hard Disk. It does not have to be partitioned or formatted.

  2. Reboot
    1. Reboot the Virtual Machine.
    2. When the Virtual Machine starts, Windows will commit some changes to the device drivers.
    3. 20171018_008

      Windows is installing drivers for the added hard disk.

      When you are quick enough you might see the Plug and Play Device Setup running installing the hardware. (On my example server it took only a few seconds to install.)

      For those interested, the Kernel-PnP log shows what happened:

      20171018_009

      GridView showing Kernel Plug & Play events created by the following PowerShell command:

      Get-WinEvent -LogName Microsoft-Windows-Kernel-PnP/Configuration |
      Where-Object { $_.TimeCreated -ge ( (Get-Date).AddHours(-1) ) } |
      Select-Object TimeCreated,Message | Out-GridView
      
    4. Shut down the Virtual Machine
    5. TL;DR
    6. Important: If you should skip the reboot step and just shut down the Virtual Machine and start modifying the Virtual Hard Disk Configuration, you’ll be presented with a BSoD boot loop. “Your PC ran into a problem and needs to restart. We’re just collecting some error info, and then we’ll restart for you.”
      20171018_010

  3. Modify Virtual Hard Disk Configuration
    1. While the Virtual Machine is shut down, using the vSphere Client, select the Virtual Machine and open Edit Settings…
    2. In my example the Virtual Machine started with 2 original Hard Disks before the third one was added. Since only the first two disks are showed, to remove the third Hard Disk first we need to click on Manage other disks.
    3. In the Pop-Up window, hover above line Hard disk 3, and a gray icon with a white X is shown. Click this icon.
    4. The line changes and shows a checkbox Delete files from datastore. As long as you are sure that you are removing the newly added disk, check the checkbox.
    5. 20171018_011

    6. Click Close
    7. Back in the Edit Settings window, click OK
    8. Important: Only after closing Edit Settings with OK, the changes will be applied. If not applied the following steps cannot continue.

    9. Open Edit Settings
    10. Click the little triangle shaped toggle in front of each Hard disk to open the advanced settings.
    11. From the Virtual Device Node Pull-Down Menu select SCSI controller 1
    12. Important: The Logical Unit Number should stay the same, only the adapter number may change:

      • SCSI(0:0) must become SCSI(1:0).
      • SCSI(0:1) must become SCSI(1:1).
      • Etc.
    13. Click OK
    14. Start the Virtual Machine.
    15. Check if all disks are only connected to the LSI Adapter using Device Manager:
    16. 20171018_012

  4. Remove VMware Tools
    1. Click Start -> Control Panel -> Program and Features
    2. Select VMware Tools and click Uninstall.
    3. Click Yes
      20171018_013
    4. When the Uninstall is finished, reboot the Virtual Machine20171018_014
    5.  When the Virtual Machine is restarted, the Device Drivers for the VMware Paravirtual devices are missing (Unknown devices):

      20171018_015

  5. Install VMware Tools
    1. While the Virtual Machine is running, using the vSphere Client, select the Virtual Machine and click Guest OS -> Install VMware Tools…
    2. 20171018_016

    3. If the VMware Tools Setup is not started automatically, click Choose what to do with this disc Pop-Up.
    4. Click Run setup64.exe
    5. 20171018_017

    6. Install the VMware Tools using all defaults, unless you have a reason to change them.
    7. Click Next >, Next >, Install, Finish
    8. 20171018_018

    9. After the installation is finished, reboot the Virtual Machine
    10. 20171018_019

    11. After the reboot, shut down the Virtual Machine
  6. Reconfigure Virtual Hardware (Change back to Paravirtual SCSI Controller)
    1. While the Virtual Machine is shut down, using the vSphere Client, select the Virtual Machine and open Edit Settings…
    2. Click the little triangle shaped toggle in front of each Hard disk to open the advanced settings.
    3. From the Virtual Device Node Pull-Down Menu select SCSI controller 0
    4. Important: The Logical Unit Number should stay the same, only the adapter number may change:

      • SCSI(1:0) must become SCSI(0:0).
      • SCSI(1:1) must become SCSI(0:1).
      • Etc.

      20171018_020

    5. Click OK
    6. Important: Only after closing Edit Settings with OK, the changes will be applied. If not applied the following steps cannot continue.
    7. Open Edit Settings
    8. Hover above line SCSI Controller 1 LSI Logic SAS, and a gray icon with a white X is shown. Click this icon.
    9. 20171018_021

    10. The line changes and shows Device will be removed.
    11. 20171018_022

    12. Click OK
    13. Start the Virtual Machine.