Set-CDDrive -Connected:$false may result in paused Linux VMs (RHEL 6.4)

I found lots of connected iso images connected to VMs and wanted to release the iso images. So I used Get-VM | Get-CDDrive | Set-CDDrive -Connected:$false

That seemed to work until the command seemed to halt. I killed the PowerShell session and wanted to retry later. One of my co-workers came to my office to tell me about a bunch of servers that had become non responsive.

After some searching it appeared that some VM’s asked a Virtual Machine question. During the time that the question is unawnsered the virtual machine is paused.

I didn’t think removing the CD images from VMs could result in downtime, but apparantely it does, atleast with RHEL 6 VMs. I didn’t see this behaviour with Windows VMs.

I found KB2000110 which lists the symptoms, but talks about an issue when using Update Manager with the option “Disable any removable media devices connected to the virtual machines on the host” selected during remediation. I guess that it uses the same API/method to eject the CD-ROM.

The guest operating system has locked the CD-ROM door and is probably using the CD-ROM, which can prevent the guest from recognizing media changes. If possible, eject the CD-ROM from inside the guest before disconnecting. Disconnect anyway and override the lock? [Yes/No]

The guest operating system has locked the CD-ROM door and is probably using the CD-ROM, which can prevent the guest from recognizing media changes. If possible, eject the CD-ROM from inside the guest before disconnecting. Disconnect anyway and override the lock? [Yes/No]

So after some clicking all was ok again. But it did disconnected a bunch of end-user sessions to the (web)servers resulting in failed attempts to commit data.

Maybe next time I try to disconnect ISO’s by adding

Get-VMQuestion | Set-VMQuestion -Option 'No' -Confirm:$false
Advertisement

Get NetApp LUN Serial from VMware naa.60a98000*

Today I needed to find the LUN ID of a NetApp iSCSI LUN. VMware shows it as naa.xxxxxx I found this forumpost in which it’s explaned that the LUN serial number is shown in hexadecimal within the naa. I wrote this script to give me all RDMs in and transcode the LUN serial number from the naa (Network Adress Authority) name.

Function Get-NetAppLunSerial ($naa) {
#Take VMware's naa indentifier as input

	#Prepare array to contain all separate ASCII characters
	$LunSerialArray=@()
	#Check if the naa has the NetApp prefix of "60a98000" and cut it. 
	If (($NAA.Split(".")[1].Substring(0,8)) -eq "60a98000") {
		Write-Host "NetApp LUN"
		#Cut the prefix
		$WWN=($NAA.Split(".")[1]).Substring(8,24)
		Write-Host $WWN
		# Cut the string in to one byte blocks
		$WWN -split "(\w{2})"|ForEach-Object {
			If ($_ -ne "") {
				$ThisHex=$_
				#Convert Hex block to Decimal Number
				$ThisDec=[Convert]::ToInt16($ThisHex,16)
				#Convert Decimal Number to ASCII Character
				$ThisChar=[CHAR][BYTE]$ThisDec
				$LunSerialArray+=$ThisChar
				}
			}
		}
	#Read the LunSerialArray as a continues string
	$LunSerial=$LunSerialArray -join ''
	#Expose the serial string outside the function
	$LunSerial
}

Get-VM | Get-HardDisk -DiskType "RawPhysical","RawVirtual" | Select Parent,Name,DiskType,ScsiCanonicalName,DeviceName,@{"Name"="NetAppLunSerial";"Expression"={Get-NetAppLunSerial $_.ScsiCanonicalName}}
# $NetAppLunSerial=Get-NetAppLunSerial naa.60a9800057322d6a5aad437267302658

This translates the naa.60a9800057322d6a5aad437267302658 to W2-jZ­Crg0&X (I’ve typed some random numbers but it’s not strange to find somewhat special characters in the Serial Number.)

All logic about the hexASCII conversion I got from here

HP’s VIBsDepot local repository without internet access

In an environment without internet access one could install an UMDS server to allow Update Manager to download patches. But when even UMDS is unavailable adding patches is limited to manual downloading and adding patches on a one-at-a-time base. Since HP has a browsable vib repository for their updates and patches, I thought I could cheat my way around clicking a trillion times to get all HP updates into Update Manager.

So I started with a download tool to download the entire vibsdepot.hp.com website. Because of quite old 5.1 GA hosts I wasn’t sure if only downloading the bundles would give me everything, I wanted to download the entire repository. Installed HTTrack Website Copier (available here) Followed the wizard, and let it do it’s business. I couldn’t get it to download faster than 25 KB/s so downloading approx 800 MB took a while… But it eventually got there and I managed to get the downloaded files through all security layers in to the infrastucture.
On a Windows 2008 R2 server with the IIS roles I created a new website and copied the contents of my download in to the website.

Update Manager could find the repository after I added a Download Source, but only over http, https, I couldn’t get to work, while browsing the website went ok, maybe it has something to do with the self-signed certificate I used. Even after I added the certificate to the Trusted Root Certificate store on the Update Manager and the vCenter server… Secure it wouldn’t validate the URL. Only unsecure. Since the enviroment I was playing with is almost completly air-gapped I didn’t really mind. So I went with the http option.

So I clicked download now… And it failed. Enventually Google found Jase McCarty’s excellent blog posting about UMDS on IIS. It stated:

Properly configured MIME-types settings are important, because the .vib & .sig files cannot be downloaded from the UMDS server without them. To add them, appcmd.exe will be used again. Note: Application/Octet-Stream is required, per the vSphere 5 Documentation.

For .vib files
c:\windows\system32\inetsrv\appcmd set config /section:staticContent /+”[fileExtension=’.vib’,mimeType=’application/octet-stream’]“
For .sig files
c:\windows\system32\inetsrv\appcmd set config /section:staticContent /+”[fileExtension=’.sig’,mimeType=’application/octet-stream’]“

Should’ve payed more attention to the VMware documentation…

PowerCli: Get VMware Licenses from an array of vCenter Servers

VMware’s licensing is quite flexible, if you have, say 1 10 socket license, you are able to split the license in to blocks so each vSphere instance get’s it’s own dedicated license key. But sometimes you encounter a situation where the people before you just reused licenses, and the vCenters aren’t linked. Because one could’t login to the license portal, and “yeah, I know we bought extra licenses, but I already know this license key, so why bother with another..” As long as you’ve bought all the license keys you need it shouldn’t be to big of a deal. <- This is my guess, I must admit that I've never looked in to the details of the licensing agreements! But in my case I found an enviroment where keys where used in multiple instances. I needed a way to quickly find all keys used, and split them in to groups of the same kind (Enterprise, Enterprise Plus, etc) and calculate if we have over- or under-utilized them.

I've written a script to help me find out. The examples shown are old values, all systems are now fully licensed.

Output of the script: (Summary.txt)

LicenseKey                    Available Used
----------                    --------- ----
AAAAA-AAAAA-AAAAA-AAAAA-AAAAA        48   48
BBBBB-BBBBB-BBBBB-BBBBB-BBBBB         1    1
CCCCC-CCCCC-CCCCC-CCCCC-CCCCC       225  279
00000-00000-00000-00000-00000         0    4
DDDDD-DDDDD-DDDDD-DDDDD-DDDDD         1    1
EEEEE-EEEEE-EEEEE-EEEEE-EEEEE         1    2
FFFFF-FFFFF-FFFFF-FFFFF-FFFFF         1    2
GGGGG-GGGGG-GGGGG-GGGGG-GGGGG        16    8
HHHHH-HHHHH-HHHHH-HHHHH-HHHHH         8    8
IIIII-IIIII-IIIII-IIIII-IIIII        12   46
JJJJJ-JJJJJ-JJJJJ-JJJJJ-JJJJJ     10000    0
KKKKK-KKKKK-KKKKK-KKKKK-KKKKK         4    4
LLLLL-LLLLL-LLLLL-LLLLL-LLLLL         1    1
MMMMM-MMMMM-MMMMM-MMMMM-MMMMM        12   12
NNNNN-NNNNN-NNNNN-NNNNN-NNNNN       175  168
OOOOO-OOOOO-OOOOO-OOOOO-OOOOO         1    1
PPPPP-PPPPP-PPPPP-PPPPP-PPPPP        20    0
QQQQQ-QQQQQ-QQQQQ-QQQQQ-QQQQQ         1    1

Product                                                    Available Used Adheres
-------                                                    --------- ---- -------
vCenter Operations Management Suite Enterprise Edition 5.6     10175  168    True
vCenter Server 5 Standard                                          7    9   False
VMware vSphere 5 Enterprise Plus                                 120  126   False
Product Evaluation                                                 0    4   False
vCenter Operations Management Suite Standard Edition 5.6         225  279   False

The fun starts with the IIIII-IIIII-IIIII-IIIII-IIIII license. It was used on 46 sockets, while it only allows for 12. But when we looked at the type of license it’s only over utilized by 6 sockets.

The complete list of licenses added to each vCenter: (VMwareLicenses.csv)

VC,Name,Key,Total,Used,ExpirationDate,Information
10.18.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,0,,
10.10.1.1,vCenter Server 5 Standard,AAAAA-AAAAA-AAAAA-AAAAA-AAAAA,1,1,,
10.18.1.1,vCenter Server 5 Standard,BBBBB-BBBBB-BBBBB-BBBBB-BBBBB,1,1,,
10.18.1.1,VMware vSphere 5 Enterprise Plus,CCCCC-CCCCC-CCCCC-CCCCC-CCCCC,12,10,,
10.11.1.1,vCenter Operations Management Suite Standard Edition 5.6,DDDDD-DDDDD-DDDDD-DDDDD-DDDDD,225,57,,vCOPS Standard
10.11.1.1,vCenter Server 5 Standard,EEEEE-EEEEE-EEEEE-EEEEE-EEEEE,1,1,,
10.10.1.1,VMware vSphere 5 Enterprise Plus,CCCCC-CCCCC-CCCCC-CCCCC-CCCCC,12,12,,
10.10.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,1,,
10.12.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,0,,
10.13.1.1,VMware vSphere 5 Enterprise Plus,FFFFF-FFFFF-FFFFF-FFFFF-FFFFF,12,12,,vSphere 5 Enterprise Plus
10.12.1.1,VMware vSphere 5 Enterprise Plus,CCCCC-CCCCC-CCCCC-CCCCC-CCCCC,12,6,,
10.12.1.1,vCenter Server 5 Standard,EEEEE-EEEEE-EEEEE-EEEEE-EEEEE,1,1,,
10.18.1.1,vCenter Operations Management Suite Standard Edition 5.6,DDDDD-DDDDD-DDDDD-DDDDD-DDDDD,225,222,,vCops Standard
10.18.1.1,VMware vSphere 5 Enterprise Plus,GGGGG-GGGGG-GGGGG-GGGGG-GGGGG,16,8,,
10.13.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,1,,
10.13.1.1,vCenter Server 5 Standard,HHHHH-HHHHH-HHHHH-HHHHH-HHHHH,1,1,,
10.11.1.1,VMware vSphere 5 Enterprise Plus,IIIII-IIIII-IIIII-IIIII-IIIII,8,8,,
10.14.1.1,vCenter Operations Management Suite Enterprise Edition 5.6,JJJJJ-JJJJJ-JJJJJ-JJJJJ-JJJJJ,10000,0,VMware.Vim.KeyAnyValue,
10.14.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,1,,
10.14.1.1,VMware vSphere 5 Enterprise Plus,KKKKK-KKKKK-KKKKK-KKKKK-KKKKK,48,48,,
10.14.1.1,vCenter Operations Management Suite Enterprise Edition 5.6,LLLLL-LLLLL-LLLLL-LLLLL-LLLLL,175,168,,
10.15.1.1,vCenter Server 5 Standard,MMMMM-MMMMM-MMMMM-MMMMM-MMMMM,1,1,,
10.15.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,0,,
10.14.1.1,vCenter Server 5 Standard,NNNNN-NNNNN-NNNNN-NNNNN-NNNNN,1,1,,
10.15.1.1,VMware vSphere 5 Enterprise Plus,OOOOO-OOOOO-OOOOO-OOOOO-OOOOO,4,4,,
10.16.1.1,VMware vSphere 5 Enterprise Plus,CCCCC-CCCCC-CCCCC-CCCCC-CCCCC,12,6,,
10.16.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,0,,
10.11.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,0,,
10.16.1.1,vCenter Server 5 Standard,PPPPP-PPPPP-PPPPP-PPPPP-PPPPP,1,1,,
10.17.1.1,Product Evaluation,00000-00000-00000-00000-00000,0,1,,
10.14.1.1,VMware vSphere 5 Enterprise Plus,QQQQQ-QQQQQ-QQQQQ-QQQQQ-QQQQQ,20,0,,
10.17.1.1,VMware vSphere 5 Enterprise Plus,CCCCC-CCCCC-CCCCC-CCCCC-CCCCC,12,12,,
10.17.1.1,vCenter Server 5 Standard,PPPPP-PPPPP-PPPPP-PPPPP-PPPPP,1,1,,

And finally the script:

Function Get-vCenterCredential ($vCenterDomain, $UserName) {
	$CredentialsFile = $vCenterDomain+".cred"
	$UPN = $UserName+"@"+$vCenterDomain
	$Password = Get-Content C:\Scripts\Creds\$CredentialsFile | ConvertTo-SecureString
	$Creds = New-Object System.Management.Automation.PSCredential($UPN,$Password)
	$Creds
}

Function Make-Dir ($Directory) {
	If (!(Test-Path $OutputPath)) {
		New-Item $OutputPath -ItemType Directory | Out-Null
	}
}

$UserName = "vmninja"
$Creds = $Null
$AllVMs=@{}
Add-PSSnapin VMware.VimAutomation.Core
$vCenters=Import-Csv C:\Scripts\vCenters.csv -Delimiter ";"
$Date=Get-Date -Format yyyyMMdd
$vCenters|ForEach-Object {
	$Domain = $_.UserDomain
	$vCenterServer = $_.vCenterIP
	$OutputPath = "C:\Scripts\Output\LicenseInfo\"+$Date.ToString()
	Make-Dir $OutputPath
	$Creds = Get-vCenterCredential -vCenterDomain $Domain -UserName $UserName
	$VIConnection = Connect-VIServer $vCenterServer -Credential $Creds
}

$vSphereLicInfo = @()
$ServiceInstance = Get-View ServiceInstance
Foreach ($LicenseMan in Get-View ($ServiceInstance | Select-Object -First 1).Content.LicenseManager) {
	Foreach ($License in ($LicenseMan | Select -ExpandProperty Licenses)) {
		$Details = "" | Select VC, Name, Key, Total, Used, ExpirationDate, Information
		$Details.VC = ([Uri]$LicenseMan.Client.ServiceUrl).Host
		$Details.Name = $License.Name
		$Details.Key = $License.LicenseKey
		$Details.Total = $License.Total
		$Details.Used = $License.Used
		$Details.Information = $License.Labels | Select -ExpandProperty Value
		$Details.ExpirationDate = $License.Properties | Where-Object {$_.Key -eq "expirationDate"} | Select-Object -ExpandProperty Value
		$vSphereLicInfo += $Details
		}
	}

$vSphereLicInfo | Sort-Object | Export-Csv -NoTypeInformation -Path $OutputPath\VMwareLicenses.csv	
$vSphereLicInfo | Format-Table -AutoSize

$AvailableLicenses=$vSphereLicInfo | Select-Object Key, Total -Unique
$UsedLicenses=$vSphereLicInfo | Select-Object Key, Used 

$DataSet=$UsedLicenses|Group-Object -Property key -AsHashTable
$tab=@{}
$DataSet.Keys | % {$tab += @{$_ = ($DataSet[$_]|Measure-Object -Property Used -Sum)}}
$Licenses = @()
$tab.Keys | % {
	[string]$CurrentKey=$_
	$Avail=$AvailableLicenses|Where-Object {$_.key -match $CurrentKey}|Select-Object $_.Total
	$Lic = "" | Select LicenseKey, Available, Used
	$Lic.LicenseKey = $CurrentKey
	$Lic.Available = $Avail.Total
	$Lic.Used = $tab[$CurrentKey].sum
	$Licenses += $Lic
}
$Licenses | ft -AutoSize | Out-File $OutputPath\Summary.txt

$AvailableLicenses=$vSphereLicInfo | Select-Object Name, Key, Total -Unique
$UsedLicenses=$vSphereLicInfo | Select-Object Name, Key, Used 

$DataSetAvail=$AvailableLicenses|Group-Object -Property Name -AsHashTable
$tabAvail=@{}
$DataSetAvail.Keys | % {$tabAvail += @{$_ = ($DataSetAvail[$_]|Measure-Object -Property Total -Sum)}}

$DataSetUsed=$UsedLicenses|Group-Object -Property Name -AsHashTable
$tabUsed=@{}
$DataSetUsed.Keys | % {$tabUsed += @{$_ = ($DataSetUsed[$_]|Measure-Object -Property Used -Sum)}}
$Licenses = @()
$LicInUse = @()
$LicInStock = @()

$tabUsed.Keys | % {
	$Product=$_
	$LicUsed=$tabUsed[$Product].sum
	$LU = "" | Select Product, UsedLicenses
	$LU.Product = $Product
	$LU.UsedLicenses = $LicUsed
	$LicInUse += $LU
	Write-Host Used: $Product`t $LicUsed
}	

$tabAvail.Keys | % {
	$Product=$_
	$LicAvail=$tabAvail[$Product].sum
	$LA = "" | Select Product, AvailableLicenses
	$LA.Product = $Product
	$LA.AvailableLicenses = $LicAvail
	$LicInStock += $LA
}	

$LicenseOverview=@()
$LicInUse | ForEach-Object {
	[string]$Product = $_.Product
	[int]$Used = $_.UsedLicenses
	$Available = $LicInStock | Where-Object {$_.Product -eq $Product} | Select-Object -ExpandProperty AvailableLicenses
	If ($Used -le $Available){
		$Adheres=$true
		}
	Else {
		$Adheres=$false
	}
	$LO = "" | Select Product, Available, Used, Adheres
	$LO.Product = $Product
	$LO.Available = $Available
	$LO.Used = $Used
	$LO.Adheres = $Adheres
	$LicenseOverview += $LO
}
$LicenseOverview | ft -AutoSize | Out-File $OutputPath\Summary.txt -Append

To save the credentials used by the script, I’ve used the following script:
The user that saves the credentials is the only one who can decrypt, so make sure you run the script with the correct credentials.

$file=$null
$vCenterDomain=Read-Host "Please enter the userdomain: " 
$File="C:\Scripts\Creds\"+$vCenterDomain.ToUpper()+".cred"
$Credential = Get-Credential
$Credential.Password | ConvertFrom-SecureString | Set-Content $File