The Quest for VMware Horizon View Client with PC over IP on Raspberry Pi

In this blog I’m discussing multiple options to use an Raspberry Pi 4 as a VMware Horizon View 8 Thin Client. What works, what doesn’t.

Due to the COVID restrictions, and the push for working from home, I’ve been at home for the last 18 months. The customer for whom I work at the moment uses a VMware Horizon VDI infrastructure, so working from home is basically the same as working from the office, bar the co-workers.

But why? Don’t you have a PC or laptop for that?

Sure I do, but my primary PC has some issue when running the Horizon View and the Microsoft Teams clients, it gets unstable. Don’t know why, haven’t found anything, PC just freezes, leaving no trace of any issue in any log/eventlog. It just locks up (I’ve got alot of stuff, including VM’s running on the background, so probably my own fault). But that is not the topic of this post.

I needed to find a reasonable alternative, I don’t want to have yet another PC on my desk only for some remote desktop stuff. I do have a bunch of Raspberry Pi 4’s stacked, which I use for either Kubernetes or ESXi on ARM. I decided that I could use one of them to test if using a Raspberry Pi 4B (4GB) for work was feasible.

Short story short: Yes the Horizon Client works on a Raspberry Pi! But not how you may think. It will not do PC over IP on a Raspberry Pi running Linux. It does when running Windows or Android on the Raspberry Pi.

To keep the story as short as needed, use these links to jump to each section:

Setup and criteria

My setup consists of 2 LG 27UL500-W 27″ 4K screens I just bought from Amazon, they weren’t that expensive to begin with and by choosing the returned items (both were advertised with having minor scratches on the front bezels (rated ‘good’ by Amazon)). But I haven’t yet found any! (Your experience may vary) but for now, I’ve gotten an 15% discount on them just because they were returned. Since I’m located in the EU, they are covered with a 2 year warranty, and I’m able to return them without question within two weeks and just get my money back if the scratches are really an eyesore. But as stated earlier, I don’t see any! I’ve got an Larmtek 1080p USB webcam attached, also from Amazon, keyboard and mouse are an old Logitech wireless desktop set, connected over the included USB dongle. My headset is a Jabra Evolve2 headset connected over USB-A. All of this connected to an Raspberry Pi 4B 4 GB (with the older BCM2711 B0-stepping). I need to acquire a 8GB model with the C0 stepping, to see if that feels like it runs better with Windows or at higher clock speeds. But that’s for later. When do I consider the setup feasible? When I could recommend it to my co-workers. Who may have reservations about using their gaming rig for the desktop work, consuming some hundreds of Watts just for doing their work, especially now with the incredible rise off energy prices. So the setup must be able to run VMware Horizon View Client 8.0 (2006) or higher and run Microsoft Teams outside of the VDI. An added requirement is that Horizon works using Teradici’s PC over IP protocol. Since that is what my current customer uses. No Blast sadly.

PC over IP support is sparse!

The Linux (armhf/32-bit) version of the client does not support PC over IP (PCoIP), and that is the only protocol the customer allows, since they use Teradici hardware based Zero Clients at the office. And this is partly the reason for this post and probably why this has come up in your Google search. It cannot do PCoIP, and wont give any helpful error either: (from /tmp/vmware/vmware-mks-XXXX.log)

blastSocket SSL: syscall error 104: Connection reset by peer
blastSocket SOCKET 6 (31) Could not negotiate SSL
blastSocket
blastSocket SOCKET 6 (31) Cannot verify target host.

It seems to point to a certificate / SSL issue, especially when you Google error 104, it’s an error raised by OpenSSL. But it appears that it is raised when the connection couldn’t be made because the display protocol can’t connect. All messages are prefixed with blastSocket indicating that it is the Blast protocol connection that is attempted to be established.

Only by resorting to ThinLinx TLXOS a warning was shown, prompting me to change my Google searches. Still I couldn’t find a definitive VMware documentation source that states that PCoIP is not possible on the Linux ARM Client. Only that connecting to a Linux Agent requires Blast. And PCoIP should just work on the x86_64/AMD64 Linux clients.

ThinLinx TLXOS: Horizon Client for ARM Linux does not support the PCoIP and RDP protocols. Your session will use the BLAST protocol instead.

There is a script included in the Horizon Client bundle, vmware-view-lib-scan, that raised an error on the PCoIP installation, it’s missing a file vmware-remotemks-container which is not included in any of the armhf archives. I even checked the Android 32-bit ARM files, but those didn’t contain the file either. I saw some posts dating back to version 4, and something about an HP ThinClient update package that should contain a working PCoIP connection. But since that version is just ancient, I didn’t explore that path.

However if your setup does allows for Blast, and you want to use a standard Raspberry Pi 4B to use as a thin client, I’ve written this how to, since (and this is for me a common experience with anything Linux related) guides are not easy to come by or don’t elaborate enough and maybe the most common, they assume a lot of background knowledge of the reader. I try to explain the steps taken as much as I can, to hopefully make it easy to follow, and if anything changes in the future, and they will (!), maybe make it understandable how to update your steps.

Because both the Microsoft Teams app (3rd party webapp wrapper app) and the Horizon View client are only available for armhf and not ARM64 or aarch64, I used Raspberry OS in this example. I’ve tried the 64bit version of Ubuntu 21.04, and couldn’t install the Teams app, which for my use case/experiment was a requirement. Sadly Microsoft doesn’t supply a 64bit ARM version of Teams nor VMware an ARM64 version of Horizon View. Both have 64bit Linux support (at the time of writing) but only x86_64 / AMD64. The armhf Teams app is from a 3rd party developer who sadly stopped development. So no ARM64 version of that is to be expected. The last resort for Teams is the webapp, and the same goes for Horizon, but the HTML5 version still requires Blast. To at least test if Blast is working, I build a basic Horizon View 8.3 setup in my homelab.

The servers in my homelab do not have any hardware graphics acceleration. To test if Blast H.264/HVEC decoding works, I tested Big Buck Bunny 4K over YouTube, and 360p video in the VDI is choppy. Without the Blast acceleration, even 360p was unwatchable, chucks missing or no change between keyframes. The same video directly in Chromium on the Raspberry allowed for a choppy but watchable 720p resolution.

I did not try any of the central management options of ThinLinx or StratoDesk This was not the aim of this test. Both come with an free (!) management appliance. For more information on these appliances, check the websites of each vendor..

Conclusion

If PC-over-IP is a requirement, the clear winner is Android. It’s quick, has great hardware and app support, sadly no dual screen. If using it for Blast, the best hardware support comes with Raspberry Pi OS, StatoDesk NoTouch OS comes with a nicer interface (Windows inspired) and ThinLinx TLXOS sports a clean minimalistic approach. Windows 11, to be able to run x86_64 apps on a Raspberry Pi is just awesome. Most of all I just love the fact that almost any major OS runs on a such an underpowered cheap device as a Raspberry and quite workable at that. As far as I know it is not yet possible to get MacOS running on a Raspberry Pi, but when some one manages to do so, Apple will still catagoricaly refuse to allow MacOS to run on any non Apple hardware. It most likely will allways be a EULA violation. That said, Windows has not been officially released for the Raspberry Pi, and as far as I know the ARM version is OEM only. But that may change in the future. (Bootcamp on M1 Macs? More ARM SoCs (other than Qualcomm) in the laptop/desktop space, who knows)

(*) I briefly tested Ubuntu, but I couldn’t get snap to install armhf stuff, so no Teams. I did add the armhf microarchitecture using dpkg --add-architecture armhf but couldn’t find a way to configure snap.
(!)Max Resolution of the Raspberry Pi 4B/400 is 2x 3840×2160 (4K) @ 30Hz
(%)ThinLinx TLXOS only allows a single app/connection at a time. So no browser and View Client at the same time.
(^) Only PowerPoint, Pictures or Video’s and some apps can be shared.
(a) Horizon View armhf is only supported by VMware on StratoDesk and ThinLinx.
(b)Windows on ARM is only supported on selected devices, sadly the RPi is not one of them.

kubernetes “volume X already bound to a different claim”

volume X already bound to a different claim when recreating Persistent Volume Claim

I recently encountered an issue in Kubernetes (Azure Kubernetes Service v1.19.7 to be exact) in which a Persistent Volume Claim (PVC) kept giving the error “volume “myvolume” already bound to a different claim.” I do not know what caused this, since there were no other claims to the volume. The PVC kept status Pending and the Persistent Volume (Azure-File) kept the status Released

My Google-Fu could not help me, I found an question on stackoverflow, but there the error was caused by sizing differences in the PV and PVC (a PVC may not claim a larger size than the PV provides). But that wasn’t the case in my environment.

Eventually I found the sourcecode for the Persistent Volume Controller which throws the error and it shows that the error should not pop up if the volume.Spec.ClaimRef == nil (line 416 at the time of writing). So I decided to see if I could find this spec.claimref in the properties of the PV. I use Lens IDE which allows you to simply edit the configuration in a GUI, but kubectl edit should do the same)

kubectl edit pv <pv-name>
(Replace the values between <> with the correct values)

I tried to remove everything under spec.claimRef by simply replacing all text with “{}” but after saving the file, it still retained the original values.

So when playing around with the values, I saw that if I editted the spec.claimRef.name to some other value, that it would be applied immediately, so changes do get saved.

At this point I tried changing the spec.claimRef.uid to the value that I found in the properties of the PVC. And that resulted in a Bound PV and PVC.

To find the uid of the PVC:

kubectl get pvc <pvc-name> --namespace <namespace-name> --output yaml |grep uid
(Replace the values between <> with the correct values)

So for some reason this value is not overwritten with the newly created PVC’s value. (Line 83 in the screenshot) But doing by hand fixed the issue. Atleast for now.

apiVersion: v1
kind: PersistentVolume
spec:
  claimRef:
    kind: PersistentVolumeClaim
    namespace: <namespace-name>
    name: <pvc-name>
    uid: 12345678-1234-abcd-defg-1234567890ab <- The uid of the PVC
    apiVersion: v1
    resourceVersion: '14061061'

Configured LAG not showing on Juniper EX switch.

Building an aggregated ethernet interface with LACP didn’t work on a switch that was but back to factory defaults (load factory-defaults).

Creating an LAG with LACP between switches is quite easy, at least I thought it was… I followed the steps in: Configuring Link Aggregation Control Protocol

root@switch# set interfaces ae0 aggregated-ether-options lacp active
root@switch# set interfaces ae0 aggregated-ether-options lacp periodic fast
root@switch# set interfaces ae0 unit 0 description "LAG to Upstream Switch"
root@switch# set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
root@switch# set interfaces ae0 unit 0 family ethernet-switching vlan members all

{master:0}[edit]
root@switch# set interfaces ge-0/0/22 ether-options 802.3ad ae0

{master:0}[edit]
root@switch# set interfaces ge-0/0/23 ether-options 802.3ad ae0

{master:0}[edit]
root@switch# commit check
configuration check succeeds
fpc1:
configuration check succeeds

{master:0}[edit]
root@switch# commit
configuration check succeeds
fpc1:
commit complete
commit complete

{master:0}[edit]
root@switch# run show interfaces terse |match ae0
ge-0/0/22.0             up    up   aenet    --> ae0.0
ge-0/0/23.0             up    up   aenet    --> ae0.0

{master:0}[edit]

I was expecting an ae0 interface with up/up state, but no interface is listed?

Something like this:

root@switch# run show interfaces terse |match ae0
ge-0/0/22.0 up up aenet --> ae0.0
ge-0/0/23.0 up up aenet --> ae0.0
ae0         up up
ae0.0       up up eth-switch

{master:0}[edit]

I guess that anyone with a decent Juniper background already guessed it, but apparently I followed an incomplete guide, since my factory default switch was missing a statement telling the switch how many LAGs are defined on the switch. I assumed (and assumptions are the mother of all ….) that if you don’t set a limit there won’t be any limit. Since this isn’t a limit, it is simply the number of (virtual) interfaces that the switch (pre-)creates. A LAG (ae) interface needs to exist before it can be configured. My bad. As stated in (Configuring Aggregated Ethernet Links) I needed to set the ‘aggregated-devices ethernet device-count’:

{master:0}[edit]
root@switch# set chassis aggregated-devices ethernet device-count 1

Now the LAG is allowed, and operational:

root@switch# run show interfaces terse |match ae0
ge-0/0/22.0 up up aenet --> ae0.0
ge-0/0/23.0 up up aenet --> ae0.0
ae0 up up
ae0.0 up up eth-switch

More information on aggregated devices can be found here

RBD Health Alarm on vCenter 6.7

01a

vCenter greeted me this morning with a RBD Health Alarm. Not really sure what the issue could be, I just googled it. Sure enough, VMPros already blogged about it. It is about Auto Deploy. Something that is not used by the customer at this point, but is nevertheless enabled, and encountered some sort of error. The blog post points to VMware KB2000988. But it states:

“This article provides troubleshooting guidance on VMware Auto Deploy 5.x. This article does not apply to vSphere 6.0”

Continue reading

Remove Inaccessible datastore from inventory

DatastoreError

Datastore ‘{name}’ is not accessible. “No connected and accessible host is attached to the datastore.”

At a client’s we encountered a rather classical error, a dismounted datastore that couldn’t be removed/distroyed. So the customer just deleted the LUN from the storagebox expecting the datastore to disappear from the inventory. That didn’t happen. Neither did the datastore get removed after a reboot of the VCSA.

The customer was adament that everything from the datastore was moved to other datastores prior to the unmounting. The filesystem seems empty (except for the default folders (.naa.XXXXXXXXXX and .ssd.sf) And all of the VMs were XvMotioned to other datastores.

Two other datastores were assigned to be used by vSphere HA as Heartbeat Datastores

So why couldn’t it be removed? Continue reading

VCAP 6 – Datacenter Virtualization Deployment

After passing the 3V0-624 (VCAP 6.5 Design) I want to pursuit the VCIX6-DCV certification, so the next exam I need to take is the 3V0-623 VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam. Since the design exam has been upgraded to vSphere 6.5, I went to the VMware Education booth at VMworld to ask if they knew if the deploy exam is stil based on 6.0. And it should still be. VMware Education could not tell me if it will be upgraded, and if so, when. But it shouldn’t happen anytime soon. So I need to prepare with the 6.0 version of vSphere. Sadly no HTML 5 client…

I did get some pointers on the exam:

  • It is based on the hands on labs, but the Control, Alt and Backspace keys are disabled. This means that you can’t use the MKS client, since you can’t release the focus of the window with CTRL+ALT, use the web console!
  • You can go back and forth between questions.
  • Even questions that you can’t complete, will be scored on the parts you did manage.
  • Scoring is done by a script that simply checks if settings are as desired, a sort of desired state configuration check.
  • Deploying the lab environment will take some time, use this time to setup the screen size of the controls.
  • vCenter will take even longer to come online, use this time to enable SSH on the ESXi hosts.
  • It’s a single lab environment during the entire exam, so if you come up with an answer to a previous question, you didn’t manage earlier, you are always able to answer it.
  • The allotted time still is an issue, there is simply not enough time to comfortable answer each question.
  • Screwing up the ESXi host networking is less likely, since the automatic revert of settings if the host becomes isolated. I’ve read that this occurred to some in the 5.x edition of the exam.
  • PowerCli should be available, for me this is important since I use it allot.
  • Use Hands on Labs. Not only to get a feel for the exam environment, but to get proficient with the tasks that you may not do on a daily basis.
    • I don’t have a list of HoLs just yet. I haven’t really started with my studies, but wanted to document the points from VMware Education, before I forget them.

Read the Exam Guide and the Platform Interface guide.

There still is no official Certification Guide from VMware Press like there was for 5.x, nor will there be anytime soon. But there is an unofficial ‘VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam Preparation Guide’ written by Ramy Mahmoud. I haven’t been able to read it, but judging by the first few pages, it should be awesome.

Schermafbeelding 2018-11-09 om 22.13.16

3V0-624: VCAP 6.5 Datacenter Virtualization Design my exam experience

VMworld Barcelona 2018 started for me with a personal victory, I passed the 3V0-624 exam!

Schermafbeelding 2018-11-09 om 21.23.25

My exam experience was great, started my exam at 9 AM, (my appointment was a half hour later, but since there were plenty of seats available, I could start early. 30 minutes less to be a worried nervous wreck. I feel that I didn’t give myself enough time to properly prepare for this exam, but then again, isn’t that always the case? Just schedule an exam date far away, to use the date as the motivator to at least try to put in some effort… But with VMworld exams rescheduling is not really possible, so I just went for it.

The exam consisted of 60 questions, either multiple choice or drag and drop, no Visio style infrastructure quizlets. The time allotted was fine for me, I had ample time to read and re-read all questions. Yes it will involve some reading, since lots of (background) information is presented about the design in question.

Many questions weren’t so much technical in nature, but ask you the define a statement to be a functional or non-functional requirement. Or if something should be classified as being a risk, constraint or an assumption. Some questions ask to sort requirements by stakeholder. So know how to identify stakeholders, and have an idea about the common tasks or interests are for C-level executives, such as CEO, CIO, CISO and my wife CFO.

Oh and the rule that with a 4 answer multiple choice question, 2 choices are evidently ridiculous, will not apply in this exam. Many of the answers seem valid! Make sure to re-read the question to see, if something that seems valid enough may be invalidated by some of the wording in the question. For every answer you give, don’t just click it because it’s the first thing that comes to mind, but try to defend you answer against the question. Think “Why is answer X wrong?” or “Why does answer X fit better than Y?”

“Dear algebra, please stop asking us to find your X, she’s never coming back and we don’t know Y” – From somewhere on the internet.

In this exam you must know why. To get a taste of it, visit vMusketeers (See below)

But to prepare for the exam, a lot of reading is required anyways…

What I did to prepare for this exam:

  • Read books
    • VMware vSphere 6.x Datacenter Design Cookbook Second Edition by Hersey Cartwright ISBN: 9781785283468
    • Essential vSAN 6.2 by Duncan Epping and Cormac Hogan. An updated version may come soon, Duncan tweeted about it a few days back!
    • Host Resources Deep Dive 6.5 by Frank Denneman and Niels Hagoort. ISBN: 9781540873064 or get a free digital copy from Rubrik
    • vSphere HA DeepDive 6.0u1 by Duncan Epping which isn’t available anymore, but the content is updated and combined with the updated Host Resource Deep Dive 6.5 book, in the VMware vSphere Clustering Deep Dive 6.7 book. I haven’t read it yet. But picked up a signed copy at VMworld, again sponsored by Rubrik, if history truly repeats itself, they may provide a free ebook version soon, so check the Rubrik site regularly!
  • The VMware Exam Guide, which contains links to allot of (VMware) Documentation, which may not be the most fun to read, but as always in a VMware exam, know the configuration maximums, limits, product compatibility and caveats for each listed product. Knowing which operating systems can or can’t be converted on which platform using VMware Converter could win you a couple of points. Know the product suite on a high level, have intimate knowledge on all things vSphere. The guide contains 10 practice questions, which are somewhat easier than the real exam, at least in my experience. All are solely of the multiple choice variety. Beware: Even if you ace these, don’t think you’ll automatically ace the exam.
  • Watched a few recordings of the VCAP6.x Design sessions from vBrownbag. These are awesome, not only because of the technical content, but especially for the explanations of the differences between Functional and Non-Functional requirements, Risks, Assumptions, Constraints, which are very important in the exam. (I really loved the videos on Section 2.1 and 3.1
  • Tested my understanding with the VCAP 6.x DCD mock exam from vMusketeers. The score of this mock was surprising accurate compared to my actual exam score, and the type of drag ‘n drop questions are quite similar to those of the exam. I didn’t see any re-order questions in my exam, but that doesn’t necessarily means that there aren’t any.
  • Blogs of each of the gentlemen reverenced above. Don’t remember the exact blog posts, but many.

Thanks to all those people that make such an effort writing books, blogs and hosting/presenting these sessions, I wouldn’t know how I could have accumulated this amount of knowledge in such a short time without all your hard work!

New VMware Skill, VMware Practitioner – NFV Operations, and you can get it for free!

A few days ago I saw a tweet from Eric Sloof about a new VMware Skill badge one could earn for free.

So I decided to go for it. Followed the also free online course VMware vCloud NFV Foundations with vCloud Director [V8.x]

And sat the VMware vCloud NFV with vCloud Director [V8.x] Test, I failed it the first time, you need to score at least 80%. But you do have 3 attempts, so the next day after reviewing my mistakes, I tried again, and I passed!

I must say the test is more trying than the course made me expect, the course doesn’t tell you allot, you need to have some knowledge of vCloud Director, vSAN, NSX and of course vSphere. Resource management and update/migration of (vCloud) infrastructures are very important topics in the tests I’ve done, however you will need solid knowledge of NSX and vSphere/vSAN! Allot of detail questions, the course focuses on NFV stuff that I really couldn’t match with the questions in the test. There is a module in the course outlining some basics about vSphere, NSX and vSAN, but not nearly detailed enough to pass the test.

For those of you who have done VTSP tests, this is much the same, it is using the same test engine, and you cannot go back to a previous question. With the score report, you will get a list of questions you got wrong, with your answers, but only those! There are some multi select questions which won’t tell you how many ‘right’ answers you need to pick from the list. So you won’t know if the question is wrong because of a wrong answer, or if it is due to many or to few selected options… Only that you didn’t receive it as 100% correct…

For some reason I wasn’t issued the badge, contacting VMware Education support didn’t really help, however they pointed me to Acclaim’s support, who seemed very willing to find out what went wrong. Today I finally received notice of the badge being issued.

vmware_Skill_Practitioner_NFV (1)

 

Daisy Chaining VMware UMDS

For a design question I was wondering if one could daisy chain multiple VMware Update Manager Download Service (UMDS) appliances. The documentation doesn’t say a word about it. The only thing I found Googling this was one blog that say’s it can’t be done. But that blog was from 2014, now, 2018, let’s see…

I started with a Ubuntu 16.04 LTS server and used William Lam’s script to install. It needed some more config:

(I tested this in Fusion virtual machines using vSphere 6.5 Update 1 (5969303)

Open the console of the first UMDS

sudo -i
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
apt-get install openssh-server -y
ifconfig

This gives the IP address (Ubuntu in Fusion creates ens33 interface)

Using an SSH (Windows: PuTTY / Mac: Termius) client, connect to the UMDS:

ssh vmninja@172.16.250.129

In the SSH session

sudo -i
wget https://github.com/lamw/vghetto-scripts/raw/master/shell/install_umds65.sh
chmod +x install_umds65.sh
./install_umds65.sh /mnt/cdrom/umds/VMware-UMDS-6.5.0-5939545.tar.gz UMDSDB UMDS_DSN umdsuser VMware1!
/usr/local/vmware-umds/bin/vmware-umds -v
/usr/local/vmware-umds/bin/vmware-umds -G
/usr/local/vmware-umds/bin/vmware-umds -S --add-url https://vibsdepot.hpe.com/index.xml --enable-host --url-type HOST
/usr/local/vmware-umds/bin/vmware-umds -D
(In the example above I’ve added the HPE VibsDepot, to see if non-firstparty updates will get downloaded.
It will now start downloading… It takes some time, at the time of writing it was about 65 GB
Per William’s suggestion for the test I used Pythons buildin webserver:
apt-get install python-minimal -y
cd /var/lib/vmware-umds
python -m SimpleHTTPServer 80
Using this as a foreground task, it shows all HTTP requests being received:
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/HPQ/metadata-hpnmi-vmware55-bundle-2.3-6.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/csco-VEM-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/bootstrap_index.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/__valm-consolidated-index__.xml HTTP/1.1" 200 -

But first I needed to build a second UMDS, mostly the same as above, but:

  1. I didn’t add the HPE VibsDepot
  2. I pointed hostupdate.vmware.com and vapp-updates.vmware.com to localhost, by editing the /etc/hosts file to prevent it to try to download anything directly from VMware.com’s website.
    • For some reason it is not possible to remove these entries from UMDS’s config, or to remove the older versions… I don’t need updates for any ESXi prior to 6.5….
    vmninja@ubuntu:~$ cat /etc/hosts
    127.0.0.1       localhost
    127.0.1.1       ubuntu
    127.0.0.1       hostupdate.vmware.com
    127.0.0.1       vapp-updates.vmware.com
    
    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
  3. Instead of the HPE VibsDepot, I added the first UMDB as source:
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/hostupdate/__hostupdate20-consolidated-index__.xml --enable-host --url-type HOST
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/vaupgrade/__valm-consolidated-index__.xml --enable-va --url-type VA
  4. Start the download task:

    /usr/local/vmware-umds/bin/vmware-umds -D

After downloading was complete, I added the second UMDS to vCenter Update Manager:

UpdateManagerConfig

After clicking download now, the Non VMware Patches are visible in vCenter:

PatchRepository

So it seams to be possible to daisy chain UMDS. I have no idea about the supportability of this configuration, and if using something in production, use a real webserver!

How to reinstall VMware Tools on a ParaVirtual VM

At a customer location I was asked to reinstall VMware Tools on machines, normally the customer would do this their selves, but after uninstall of the VMware Tools, the VM wouldn’t boot. They needed to reinstall the VMware Tools because of the issue encountered described in  VMware KB2063887

The issue is quite simple, it’s a chicken-egg kind of thing. VMware Tools includes the device driver for the PVSCSI Controller, without VMware Tools, Windows lost its driver for the PVSCSI controller and can’t boot from it.

So I created this how to for the customer, and thought it might be useful for others.

So what’s the trick. Actually if you ever changed a VM to use the ParaVirtual SCSI Controller, you boot the VM using it’s ‘normal’ SCSI controller, you add the PVSCSI and a (temporary) hard disk to it, wait for Windows to install all drivers and ‘see’ the hard disk connected to the PVSCSI. You then shutdown Windows, remove any temporary hard disk, remove the extra PVSCSI controller and change the type of the original controller to PVSCSI.  This is mostly doing the opposite:

This is done on Windows Server 2012R2, the process should be much the same on other versions.

20171018_001

Device Manager showing VMware hardware devices in default view.

20171018_002

Device Manager showing VMware hardware devices in Device by connection view.

  1. Add Temporary Hardware
    1. While the Virtual Machine is running, using the vSphere Client, select the Virtual Machine and open Edit Settings…20171018_003
    2. From the Pull-Down Menu New Device, select SCSI Controller
    3. Click Add, a new LSI Logic SAS Controller will be added.
    4. From the Pull-Down Menu New Device, select New Hard Disk
    5. Click Add, a new 40 GB Hard Disk will be added.
    6. 20171018_004

    7. From the Virtual Device Node Pull-Down Menu select New SCSI controller
    8. (Optional) Adjust the size of the new Hard Disk from 40 GB to 1 GB
    9. (Optional) From the Disk Provisioning Pull-Down Menu select Thin provision
    10. Click OK
    11. Almost instantaneous the added hardware is visible in Windows:

      20171018_005

      Device Manager showing the newly added LSI Adapter, SAS 3000 series, 8-port with 1068 SCSI controller and the added VMware Virtual disk SCSI Disk Device hard disk connected to it. The original two disks are still connected to the VMware PVSCSI Controller

      20171018_006

      Disk Manager showing the added Hard Disk. It does not have to be partitioned or formatted.

  2. Reboot
    1. Reboot the Virtual Machine.
    2. When the Virtual Machine starts, Windows will commit some changes to the device drivers.
    3. 20171018_008

      Windows is installing drivers for the added hard disk.

      When you are quick enough you might see the Plug and Play Device Setup running installing the hardware. (On my example server it took only a few seconds to install.)

      For those interested, the Kernel-PnP log shows what happened:

      20171018_009

      GridView showing Kernel Plug & Play events created by the following PowerShell command:

      Get-WinEvent -LogName Microsoft-Windows-Kernel-PnP/Configuration |
      Where-Object { $_.TimeCreated -ge ( (Get-Date).AddHours(-1) ) } |
      Select-Object TimeCreated,Message | Out-GridView
      
    4. Shut down the Virtual Machine
    5. TL;DR
    6. Important: If you should skip the reboot step and just shut down the Virtual Machine and start modifying the Virtual Hard Disk Configuration, you’ll be presented with a BSoD boot loop. “Your PC ran into a problem and needs to restart. We’re just collecting some error info, and then we’ll restart for you.”
      20171018_010

  3. Modify Virtual Hard Disk Configuration
    1. While the Virtual Machine is shut down, using the vSphere Client, select the Virtual Machine and open Edit Settings…
    2. In my example the Virtual Machine started with 2 original Hard Disks before the third one was added. Since only the first two disks are showed, to remove the third Hard Disk first we need to click on Manage other disks.
    3. In the Pop-Up window, hover above line Hard disk 3, and a gray icon with a white X is shown. Click this icon.
    4. The line changes and shows a checkbox Delete files from datastore. As long as you are sure that you are removing the newly added disk, check the checkbox.
    5. 20171018_011

    6. Click Close
    7. Back in the Edit Settings window, click OK
    8. Important: Only after closing Edit Settings with OK, the changes will be applied. If not applied the following steps cannot continue.

    9. Open Edit Settings
    10. Click the little triangle shaped toggle in front of each Hard disk to open the advanced settings.
    11. From the Virtual Device Node Pull-Down Menu select SCSI controller 1
    12. Important: The Logical Unit Number should stay the same, only the adapter number may change:

      • SCSI(0:0) must become SCSI(1:0).
      • SCSI(0:1) must become SCSI(1:1).
      • Etc.
    13. Click OK
    14. Start the Virtual Machine.
    15. Check if all disks are only connected to the LSI Adapter using Device Manager:
    16. 20171018_012

  4. Remove VMware Tools
    1. Click Start -> Control Panel -> Program and Features
    2. Select VMware Tools and click Uninstall.
    3. Click Yes
      20171018_013
    4. When the Uninstall is finished, reboot the Virtual Machine20171018_014
    5.  When the Virtual Machine is restarted, the Device Drivers for the VMware Paravirtual devices are missing (Unknown devices):

      20171018_015

  5. Install VMware Tools
    1. While the Virtual Machine is running, using the vSphere Client, select the Virtual Machine and click Guest OS -> Install VMware Tools…
    2. 20171018_016

    3. If the VMware Tools Setup is not started automatically, click Choose what to do with this disc Pop-Up.
    4. Click Run setup64.exe
    5. 20171018_017

    6. Install the VMware Tools using all defaults, unless you have a reason to change them.
    7. Click Next >, Next >, Install, Finish
    8. 20171018_018

    9. After the installation is finished, reboot the Virtual Machine
    10. 20171018_019

    11. After the reboot, shut down the Virtual Machine
  6. Reconfigure Virtual Hardware (Change back to Paravirtual SCSI Controller)
    1. While the Virtual Machine is shut down, using the vSphere Client, select the Virtual Machine and open Edit Settings…
    2. Click the little triangle shaped toggle in front of each Hard disk to open the advanced settings.
    3. From the Virtual Device Node Pull-Down Menu select SCSI controller 0
    4. Important: The Logical Unit Number should stay the same, only the adapter number may change:

      • SCSI(1:0) must become SCSI(0:0).
      • SCSI(1:1) must become SCSI(0:1).
      • Etc.

      20171018_020

    5. Click OK
    6. Important: Only after closing Edit Settings with OK, the changes will be applied. If not applied the following steps cannot continue.
    7. Open Edit Settings
    8. Hover above line SCSI Controller 1 LSI Logic SAS, and a gray icon with a white X is shown. Click this icon.
    9. 20171018_021

    10. The line changes and shows Device will be removed.
    11. 20171018_022

    12. Click OK
    13. Start the Virtual Machine.