Configured LAG not showing on Juniper EX switch.

Building an aggregated ethernet interface with LACP didn’t work on a switch that was but back to factory defaults (load factory-defaults).

Creating an LAG with LACP between switches is quite easy, at least I thought it was… I followed the steps in: Configuring Link Aggregation Control Protocol

root@switch# set interfaces ae0 aggregated-ether-options lacp active
root@switch# set interfaces ae0 aggregated-ether-options lacp periodic fast
root@switch# set interfaces ae0 unit 0 description "LAG to Upstream Switch"
root@switch# set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
root@switch# set interfaces ae0 unit 0 family ethernet-switching vlan members all

{master:0}[edit]
root@switch# set interfaces ge-0/0/22 ether-options 802.3ad ae0

{master:0}[edit]
root@switch# set interfaces ge-0/0/23 ether-options 802.3ad ae0

{master:0}[edit]
root@switch# commit check
configuration check succeeds
fpc1:
configuration check succeeds

{master:0}[edit]
root@switch# commit
configuration check succeeds
fpc1:
commit complete
commit complete

{master:0}[edit]
root@switch# run show interfaces terse |match ae0
ge-0/0/22.0             up    up   aenet    --> ae0.0
ge-0/0/23.0             up    up   aenet    --> ae0.0

{master:0}[edit]

I was expecting an ae0 interface with up/up state, but no interface is listed?

Something like this:

root@switch# run show interfaces terse |match ae0
ge-0/0/22.0 up up aenet --> ae0.0
ge-0/0/23.0 up up aenet --> ae0.0
ae0         up up
ae0.0       up up eth-switch

{master:0}[edit]

I guess that anyone with a decent Juniper background already guessed it, but apparently I followed an incomplete guide, since my factory default switch was missing a statement telling the switch how many LAGs are defined on the switch. I assumed (and assumptions are the mother of all ….) that if you don’t set a limit there won’t be any limit. Since this isn’t a limit, it is simply the number of (virtual) interfaces that the switch (pre-)creates. A LAG (ae) interface needs to exist before it can be configured. My bad. As stated in (Configuring Aggregated Ethernet Links) I needed to set the ‘aggregated-devices ethernet device-count’:

{master:0}[edit]
root@switch# set chassis aggregated-devices ethernet device-count 1

Now the LAG is allowed, and operational:

root@switch# run show interfaces terse |match ae0
ge-0/0/22.0 up up aenet --> ae0.0
ge-0/0/23.0 up up aenet --> ae0.0
ae0 up up
ae0.0 up up eth-switch

More information on aggregated devices can be found here

RBD Health Alarm on vCenter 6.7

01a

vCenter greeted me this morning with a RBD Health Alarm. Not really sure what the issue could be, I just googled it. Sure enough, VMPros already blogged about it. It is about Auto Deploy. Something that is not used by the customer at this point, but is nevertheless enabled, and encountered some sort of error. The blog post points to VMware KB2000988. But it states:

“This article provides troubleshooting guidance on VMware Auto Deploy 5.x. This article does not apply to vSphere 6.0”

Continue reading

Remove Inaccessible datastore from inventory

DatastoreError

Datastore ‘{name}’ is not accessible. “No connected and accessible host is attached to the datastore.”

At a client’s we encountered a rather classical error, a dismounted datastore that couldn’t be removed/distroyed. So the customer just deleted the LUN from the storagebox expecting the datastore to disappear from the inventory. That didn’t happen. Neither did the datastore get removed after a reboot of the VCSA.

The customer was adament that everything from the datastore was moved to other datastores prior to the unmounting. The filesystem seems empty (except for the default folders (.naa.XXXXXXXXXX and .ssd.sf) And all of the VMs were XvMotioned to other datastores.

Two other datastores were assigned to be used by vSphere HA as Heartbeat Datastores

So why couldn’t it be removed? Continue reading

VCAP 6 – Datacenter Virtualization Deployment

After passing the 3V0-624 (VCAP 6.5 Design) I want to pursuit the VCIX6-DCV certification, so the next exam I need to take is the 3V0-623 VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam. Since the design exam has been upgraded to vSphere 6.5, I went to the VMware Education booth at VMworld to ask if they knew if the deploy exam is stil based on 6.0. And it should still be. VMware Education could not tell me if it will be upgraded, and if so, when. But it shouldn’t happen anytime soon. So I need to prepare with the 6.0 version of vSphere. Sadly no HTML 5 client…

I did get some pointers on the exam:

  • It is based on the hands on labs, but the Control, Alt and Backspace keys are disabled. This means that you can’t use the MKS client, since you can’t release the focus of the window with CTRL+ALT, use the web console!
  • You can go back and forth between questions.
  • Even questions that you can’t complete, will be scored on the parts you did manage.
  • Scoring is done by a script that simply checks if settings are as desired, a sort of desired state configuration check.
  • Deploying the lab environment will take some time, use this time to setup the screen size of the controls.
  • vCenter will take even longer to come online, use this time to enable SSH on the ESXi hosts.
  • It’s a single lab environment during the entire exam, so if you come up with an answer to a previous question, you didn’t manage earlier, you are always able to answer it.
  • The allotted time still is an issue, there is simply not enough time to comfortable answer each question.
  • Screwing up the ESXi host networking is less likely, since the automatic revert of settings if the host becomes isolated. I’ve read that this occurred to some in the 5.x edition of the exam.
  • PowerCli should be available, for me this is important since I use it allot.
  • Use Hands on Labs. Not only to get a feel for the exam environment, but to get proficient with the tasks that you may not do on a daily basis.
    • I don’t have a list of HoLs just yet. I haven’t really started with my studies, but wanted to document the points from VMware Education, before I forget them.

Read the Exam Guide and the Platform Interface guide.

There still is no official Certification Guide from VMware Press like there was for 5.x, nor will there be anytime soon. But there is an unofficial ‘VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment Exam Preparation Guide’ written by Ramy Mahmoud. I haven’t been able to read it, but judging by the first few pages, it should be awesome.

Schermafbeelding 2018-11-09 om 22.13.16

3V0-624: VCAP 6.5 Datacenter Virtualization Design my exam experience

VMworld Barcelona 2018 started for me with a personal victory, I passed the 3V0-624 exam!

Schermafbeelding 2018-11-09 om 21.23.25

My exam experience was great, started my exam at 9 AM, (my appointment was a half hour later, but since there were plenty of seats available, I could start early. 30 minutes less to be a worried nervous wreck. I feel that I didn’t give myself enough time to properly prepare for this exam, but then again, isn’t that always the case? Just schedule an exam date far away, to use the date as the motivator to at least try to put in some effort… But with VMworld exams rescheduling is not really possible, so I just went for it.

The exam consisted of 60 questions, either multiple choice or drag and drop, no Visio style infrastructure quizlets. The time allotted was fine for me, I had ample time to read and re-read all questions. Yes it will involve some reading, since lots of (background) information is presented about the design in question.

Many questions weren’t so much technical in nature, but ask you the define a statement to be a functional or non-functional requirement. Or if something should be classified as being a risk, constraint or an assumption. Some questions ask to sort requirements by stakeholder. So know how to identify stakeholders, and have an idea about the common tasks or interests are for C-level executives, such as CEO, CIO, CISO and my wife CFO.

Oh and the rule that with a 4 answer multiple choice question, 2 choices are evidently ridiculous, will not apply in this exam. Many of the answers seem valid! Make sure to re-read the question to see, if something that seems valid enough may be invalidated by some of the wording in the question. For every answer you give, don’t just click it because it’s the first thing that comes to mind, but try to defend you answer against the question. Think “Why is answer X wrong?” or “Why does answer X fit better than Y?”

“Dear algebra, please stop asking us to find your X, she’s never coming back and we don’t know Y” – From somewhere on the internet.

In this exam you must know why. To get a taste of it, visit vMusketeers (See below)

But to prepare for the exam, a lot of reading is required anyways…

What I did to prepare for this exam:

  • Read books
    • VMware vSphere 6.x Datacenter Design Cookbook Second Edition by Hersey Cartwright ISBN: 9781785283468
    • Essential vSAN 6.2 by Duncan Epping and Cormac Hogan. An updated version may come soon, Duncan tweeted about it a few days back!
    • Host Resources Deep Dive 6.5 by Frank Denneman and Niels Hagoort. ISBN: 9781540873064 or get a free digital copy from Rubrik
    • vSphere HA DeepDive 6.0u1 by Duncan Epping which isn’t available anymore, but the content is updated and combined with the updated Host Resource Deep Dive 6.5 book, in the VMware vSphere Clustering Deep Dive 6.7 book. I haven’t read it yet. But picked up a signed copy at VMworld, again sponsored by Rubrik, if history truly repeats itself, they may provide a free ebook version soon, so check the Rubrik site regularly!
  • The VMware Exam Guide, which contains links to allot of (VMware) Documentation, which may not be the most fun to read, but as always in a VMware exam, know the configuration maximums, limits, product compatibility and caveats for each listed product. Knowing which operating systems can or can’t be converted on which platform using VMware Converter could win you a couple of points. Know the product suite on a high level, have intimate knowledge on all things vSphere. The guide contains 10 practice questions, which are somewhat easier than the real exam, at least in my experience. All are solely of the multiple choice variety. Beware: Even if you ace these, don’t think you’ll automatically ace the exam.
  • Watched a few recordings of the VCAP6.x Design sessions from vBrownbag. These are awesome, not only because of the technical content, but especially for the explanations of the differences between Functional and Non-Functional requirements, Risks, Assumptions, Constraints, which are very important in the exam. (I really loved the videos on Section 2.1 and 3.1
  • Tested my understanding with the VCAP 6.x DCD mock exam from vMusketeers. The score of this mock was surprising accurate compared to my actual exam score, and the type of drag ‘n drop questions are quite similar to those of the exam. I didn’t see any re-order questions in my exam, but that doesn’t necessarily means that there aren’t any.
  • Blogs of each of the gentlemen reverenced above. Don’t remember the exact blog posts, but many.

Thanks to all those people that make such an effort writing books, blogs and hosting/presenting these sessions, I wouldn’t know how I could have accumulated this amount of knowledge in such a short time without all your hard work!

New VMware Skill, VMware Practitioner – NFV Operations, and you can get it for free!

A few days ago I saw a tweet from Eric Sloof about a new VMware Skill badge one could earn for free.

So I decided to go for it. Followed the also free online course VMware vCloud NFV Foundations with vCloud Director [V8.x]

And sat the VMware vCloud NFV with vCloud Director [V8.x] Test, I failed it the first time, you need to score at least 80%. But you do have 3 attempts, so the next day after reviewing my mistakes, I tried again, and I passed!

I must say the test is more trying than the course made me expect, the course doesn’t tell you allot, you need to have some knowledge of vCloud Director, vSAN, NSX and of course vSphere. Resource management and update/migration of (vCloud) infrastructures are very important topics in the tests I’ve done, however you will need solid knowledge of NSX and vSphere/vSAN! Allot of detail questions, the course focuses on NFV stuff that I really couldn’t match with the questions in the test. There is a module in the course outlining some basics about vSphere, NSX and vSAN, but not nearly detailed enough to pass the test.

For those of you who have done VTSP tests, this is much the same, it is using the same test engine, and you cannot go back to a previous question. With the score report, you will get a list of questions you got wrong, with your answers, but only those! There are some multi select questions which won’t tell you how many ‘right’ answers you need to pick from the list. So you won’t know if the question is wrong because of a wrong answer, or if it is due to many or to few selected options… Only that you didn’t receive it as 100% correct…

For some reason I wasn’t issued the badge, contacting VMware Education support didn’t really help, however they pointed me to Acclaim’s support, who seemed very willing to find out what went wrong. Today I finally received notice of the badge being issued.

vmware_Skill_Practitioner_NFV (1)

 

Daisy Chaining VMware UMDS

For a design question I was wondering if one could daisy chain multiple VMware Update Manager Download Service (UMDS) appliances. The documentation doesn’t say a word about it. The only thing I found Googling this was one blog that say’s it can’t be done. But that blog was from 2014, now, 2018, let’s see…

I started with a Ubuntu 16.04 LTS server and used William Lam’s script to install. It needed some more config:

(I tested this in Fusion virtual machines using vSphere 6.5 Update 1 (5969303)

Open the console of the first UMDS

sudo -i
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
apt-get install openssh-server -y
ifconfig

This gives the IP address (Ubuntu in Fusion creates ens33 interface)

Using an SSH (Windows: PuTTY / Mac: Termius) client, connect to the UMDS:

ssh vmninja@172.16.250.129

In the SSH session

sudo -i
wget https://github.com/lamw/vghetto-scripts/raw/master/shell/install_umds65.sh
chmod +x install_umds65.sh
./install_umds65.sh /mnt/cdrom/umds/VMware-UMDS-6.5.0-5939545.tar.gz UMDSDB UMDS_DSN umdsuser VMware1!
/usr/local/vmware-umds/bin/vmware-umds -v
/usr/local/vmware-umds/bin/vmware-umds -G
/usr/local/vmware-umds/bin/vmware-umds -S --add-url https://vibsdepot.hpe.com/index.xml --enable-host --url-type HOST
/usr/local/vmware-umds/bin/vmware-umds -D
(In the example above I’ve added the HPE VibsDepot, to see if non-firstparty updates will get downloaded.
It will now start downloading… It takes some time, at the time of writing it was about 65 GB
Per William’s suggestion for the test I used Pythons buildin webserver:
apt-get install python-minimal -y
cd /var/lib/vmware-umds
python -m SimpleHTTPServer 80
Using this as a foreground task, it shows all HTTP requests being received:
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/HPQ/metadata-hpnmi-vmware55-bundle-2.3-6.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/csco-VEM-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/bootstrap_index.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/__valm-consolidated-index__.xml HTTP/1.1" 200 -

But first I needed to build a second UMDS, mostly the same as above, but:

  1. I didn’t add the HPE VibsDepot
  2. I pointed hostupdate.vmware.com and vapp-updates.vmware.com to localhost, by editing the /etc/hosts file to prevent it to try to download anything directly from VMware.com’s website.
    • For some reason it is not possible to remove these entries from UMDS’s config, or to remove the older versions… I don’t need updates for any ESXi prior to 6.5….
    vmninja@ubuntu:~$ cat /etc/hosts
    127.0.0.1       localhost
    127.0.1.1       ubuntu
    127.0.0.1       hostupdate.vmware.com
    127.0.0.1       vapp-updates.vmware.com
    
    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
  3. Instead of the HPE VibsDepot, I added the first UMDB as source:
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/hostupdate/__hostupdate20-consolidated-index__.xml --enable-host --url-type HOST
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/vaupgrade/__valm-consolidated-index__.xml --enable-va --url-type VA
  4. Start the download task:

    /usr/local/vmware-umds/bin/vmware-umds -D

After downloading was complete, I added the second UMDS to vCenter Update Manager:

UpdateManagerConfig

After clicking download now, the Non VMware Patches are visible in vCenter:

PatchRepository

So it seams to be possible to daisy chain UMDS. I have no idea about the supportability of this configuration, and if using something in production, use a real webserver!