vCenter greeted me this morning with a RBD Health Alarm. Not really sure what the issue could be, I just googled it. Sure enough, VMPros already blogged about it. It is about Auto Deploy. Something that is not used by the customer at this point, but is nevertheless enabled, and encountered some sort of error. The blog post points to VMware KB2000988. But it states:
“This article provides troubleshooting guidance on VMware Auto Deploy 5.x. This article does not apply to vSphere 6.0”
So first check the status of Auto Deploy (vSphere H5 client -> Menu -> Auto Deploy), and there we have an error message:
“Managing depots is temporary unavailable. The Image Builder service is down. Try restarting the service to fix the issue.”
That’s quite a helpful message, it even tells me what to try next.
So I connected to the VAMI (https://vcenter:5480) and navigated to Services.
Image Builder is stopped, but since it is a Manual service, it shouldn’t be much of an issue, but the Auto Deploy service seemed to be in a degraded state.
I just selected the Auto Deploy service and clicked Restart.
I was in luck, the simplest solution worked for me, just restarting the service, now the service is healthy again, but the alert remained. The error message was gone after manually starting Image Builder service.
Although, they explicitly state that it is not meant for vCenter 6.x, it speaks of the vCenter Appliance (both Auto Deploy and the Appliance were introduced with vSphere 5.0 afaik), one needs to start somewhere.
The paths are still valid:
Configuration files for the vCenter Server Appliance:
- Configuration files: /etc/vmware-rbd
- Runtime state files: /var/lib/rbd
Database files for the vCenter Server Appliance:
- /var/lib/rbd/db*
Cache for the vCenter Server Appliance:
- /var/lib/rbd/cache
Logfiles:
- /var/log/vmware/rbd
/etc/vmware-rbd/httpd/logs<- This one does not exist anymore.
The autodeploy-register still exists, and is located in:
- /usr/bin/autodeploy-register
The instructions on which files to edit, seem to be the same:
/etc/vmware-rbd/autodeploy-setup.xml
/etc/vmware/rbd/logging.conf
/etc/vmware-rbd/autodeploy-setup.xml & /etc/vmware/rbd/logging.conf both still exist and show content that seems to be consistent with the instructions per KB.
Re-registering Auto Deploy seems to still work:
root@vcenter [ ~ ]# autodeploy-register -R -a 127.0.0.1 -u administrator@vsphere.local -p 80 -s /etc/vmware/rbd/autodeploy-setup.xml -m Password: Succesfully registered autodeploy with vCenter 127.0.0.1
However it did not remove the alarm. This alarm, on what does it trigger, and how are the current conditions? This is the alarm definition:
So I started searching the vCenter database for a rbd component and it’s current status.
VCDB=# SELECT * FROM vpxv_alarms WHERE name LIKE '%RBD%'; alarmid | name | enabled ---------+----------------------+--------- 612 | alarm.RBDHealthAlarm | 1 (1 row)
Only one alarm definition found with RDB in its name, so check the details. And yes, it’s the same as the screenshot:
VCDB=# SELECT * FROM vpx_alarm_expr_comp WHERE alarm_id = 612 ; alarm_id | expression_id | attribute_name | operator_type | attribute_value | surr_key ---------+---------------+----------------+---------------+-----------------+---------- 612 | 1 | componentId | equals | rbd | 162 612 | 1 | newStatus | equals | green | 163 612 | 2 | componentId | equals | rbd | 164 612 | 2 | newStatus | equals | yellow | 165 612 | 3 | componentId | equals | rbd | 166 612 | 3 | newStatus | equals | orange | 167 612 | 4 | componentId | equals | rbd | 168 612 | 4 | newStatus | equals | red | 169 (8 rows)
I found that the vpx_ext table contains Rule-Based Deployment (com.vmware.rbd) and it lists the health URL for the component/extension:
VCDB=# SELECT ext_id,health_url FROM vpx_ext; ext_id | health_url ---------------------------------+---------------------------------------------------------------------- com.vmware.vcops | com.vmware.rbd | https://vcenter:6502/vmw/rbd/health-info com.vmware.vcIntegrity | http://vcenter:9084/vci/downloads/health.xml com.vmware.vsphere.client.h5vsan | http://vcenter:-80/vsanHealth/health com.vmware.vim.eam | https://*:443/eam/eamService-web/health.xml com.vmware.vrops.install | http://vcenter:-80/vsanHealth/health com.vmware.imagebuilder | com.vmware.vmcam | com.vmware.vsan.dp | com.vmware.vsan.health | http://vcenter:-80/vsanHealth/health com.vmware.vcenter.iso | com.vmware.ovf | com.vmware.cl | com.vmware.vcenter.vmtx | (14 rows)
So lets check https://<vcenter>:6502/vmw/rbd/health-info :
root@vcenter [ ~ ]# wget https://vcenter:6502/vmw/rbd/health-info --2019-04-11 10:07:01-- https://vcenter:6502/vmw/rbd/health-info Resolving vcenter... 127.0.0.1 Connecting to vcenter|127.0.0.1|:6502... connected. HTTP request sent, awaiting response... 200 OK Length: 294 Saving to: ‘health-info’ health-info 100%[===================>] 294 --.-KB/s in 0s 2019-04-11 10:07:01 (74.4 MB/s) - ‘health-info’ saved [294/294] root@vcenter [ ~ ]# cat health-info <vimhealth xmlns="http://www.vmware.com/vi/healthservice" schemaVersion="1.0"> <health id="com.vmware.rbd"> <name>vSphere AutoDeploy</name> <status>green</status> <message id='com.vmware.autodeploy.ok' level='info' time='2019-04-11T07:23:42'>OK</message> </health> </vimhealth>
And its status is currently green. I don’t know why vCenter didn’t read the current status and removed the alert, but hey, according to this info, RBD is OK. I ended up just resetting the alert to green.
vmninja 