At 1049pm a Xen host at MCNC became unresponsive and unexpectedly rebooted.  All VMs running on this host appear to have restarted automatically after the host resumed.  PCS staff are investigating and will provide more information as it becomes available.

Update 10:51pm

All OIT virtual hosting services restored.

Comtech engineers repaired a faulty link

Update 9:17pm

Services continue to be impacted. Comtech engineers have a support case open with the vendor.

Update 6:58pm

vcenter(vcenter.ncsu.edu, corevc) and Xen (prodxo and corexo) are still unavailable or sporadic connections. Comtech engineers continue to work on the issue.  

Some or most of the VM backup jobs will not complete successfully tonight due to the management network/context being affected, PCS team will restart the backup jobs as needed.

Update 4:31pm:

At this time we feel all VMware and Xen VMs are running normally.

4:20pm

We are experiencing connectivity issues for OIT VMware vcenters and Xen platforms, this includes vcenter.ncsu.edu, corevc, prodxo.oit.ncsu.edu . Platform engineers are working Network engineers to troubleshoot and find a resolution.

Update: A hardware fault has been diagnosed and the vendor is dispatching a replacement.  The host will remain offline until the replacement is installed.

The following VMs were automatically restarted during this event.

At 923pm a host at MCNC became unresponsive due to a hardware fault.  All running VMs were automatically failed over to alternate hosts and restarted.  More information will be provided as it becomes available.

Update 9:00am

All VMs have networking now and should be recovered or recovering. 

The issue appeared to have started around 2:15am, due to the zabbix monitoring server being impacted it was unknown at the time.

A PCS rack switch became unresponsive and a restart resolved the issue. A vendor support request is being opened to determine the root cause.

---

Starting at 6:15am this morning, various VMs in the MCNC general cluster lost networking affecting about half of the servers. Engineers are troubleshooting.

Update ---

DIT is resolving the temperature issue, according to operations it should return to normal over the next hour.  At this time no additional VMware host have failed, however the Platform team is proactively taking measures to prevent any additional failures. 

Below is the list of affected VMs from the initial host outage:

cgiplprdww1.oit.ncsu.edu

cs927lt-aa9.acs.ncsu.edu

cs927lt-ww9.acs.ncsu.edu

cs927prdwi3.acs.ncsu.edu

cs927upg-v1.acs.ncsu.edu

cs928prd-v1.acs.ncsu.edu

edccorefcs.oit.ncsu.edu

em13cprdaa1.oit.ncsu.edu

fo927dv1ww1.acs.ncsu.edu

fo927lt-ww1.acs.ncsu.edu

fo927preww1.acs.ncsu.edu

fo927tst-v1.acs.ncsu.edu

fsc92pumvm1.acs.ncsu.edu

hc927lt--v1.acs.ncsu.edu

hc927lt-aa1.acs.ncsu.edu

hc927rptaa1.acs.ncsu.edu

iws95prdaa1.acs.ncsu.edu

jvwebdevww1.oit.ncsu.edu

mfc06dev-v1.oit.ncsu.edu

omr100.oit.ncsu.edu

onb16devbc1.oit.ncsu.edu

onb16prdax1.oit.ncsu.edu

onb20devaa1.oit.ncsu.edu

onb20prdaa5.oit.ncsu.edu

psattprdaa1.acs.ncsu.edu

sva75sysll1.oit.ncsu.edu

sva75sysww1.oit.ncsu.edu

vya35pocva1.oit.ncsu.edu

wwphpdevww1.oit.ncsu.edu

----------

An OIT VMware hosts are failing this morning at Eastern DataCenter, due to high temperatures. Operations team is investigating with DIT staff. 

OIT Platform team will restart services for vcenter.ncsu.edu and other core VMware appliances to address security vulnerability.  VMware vcenter services will be unavailable at times during the scheduled maintenance window. There will be no impact to any virtual machines during the maintenance.

In the OIT VMware cluster. A VMware Host hardware failure occurred at 5:49am causing an HA (High Availability) event. All VMs residing on this host were automatically restarted on alternate hosts in the related cluster.  This is the second occurrence and not related to the event on 5/7. 

The host failure is suspected to be a failed NIC mezzanine and the host is taken out of service.

A list of VM's impacted:

dpadb.oit.ncsu.edu, engr22lic.eos.ncsu.edu, esftp200.oit.ncsu.edu, idm-prd-con-214.oit.ncsu.edu, im122qp200.oit.ncsu.edu, iws-wpctrl-200.oit.ncsu.edu, iws-wsproxy-200.oit.ncsu.edu, iws204mr.oit.ncsu.edu, iws206mr.oit.ncsu.edu, linux12tst.unity.ncsu.edu, natira.oit.ncsu.edu, oit208lic.oit.ncsu.edu, pciforeman-d1.oit.ncsu.edu, proxyav6.oit.ncsu.edu, ps290cups.oit.ncsu.edu, puppet-ca.oit.ncsu.edu, snowprodvam.oit.ncsu.edu, spl-d-icm-200.oit.ncsu.edu, uni200c.unity.ncsu.edu, wt-cmdp-200.wolftech.ad.ncsu.edu, wt-cmmp-200.wolftech.ad.ncsu.edu

In the OIT VMware cluster. A VMware Host fault occurred at 4:39AM causing an HA (High Availability) event. All VMs residing on this host were automatically restarted on alternate hosts in the cluster.

The host crashed with PSOD and team is investigating the root cause. A list of VM's impacted: 

advweb03.oit.ncsu.edu ece-cron-04.ece.ncsu.edu engr-ws-201.eos.ncsu.edu engr97lic.eos.ncsu.edu hsjj3.oit.ncsu.edu im122do201oit.ncsu.edu iris200.oit.ncsu.edu iso-appchk8.oit.ncsu.edu itreweb.oit.ncsu.edu iws-dbms-201.oit.ncsu.edu jhperry-vm oirp-vm04.oitclients.ncsu.edu oirp-vmtest.oitclients.ncsu.edu oit200mfa.oit.ncsu.edu oit212sccm-rp.oit.ncsu.edu oittss-89e992.oitclients.ncsu.edu orchard.oit.ncsu.edu pcitest1.oit.ncsu.edu ps100sql.oit.ncsu.edu ps280w.oit.nsu.edu regais-02.oit.ncsu.edu regais-10.oit.ncsu.edu regcsi-02.oit.ncsu.edu regcsi-08.oit.ncsu.edu regcsi-test.oit.ncsu.edu regeas-04.oit.ncsu.edu regeas-06.oit.ncsu.edu regeas-10.oit.ncsu.edu renv-rds-210.oit.ncsu.edu rhn201cap.unity.ncsu.edu rman200db.oit.ncsu.edu sec-nse-203.oit.ncsu.edu sec-s-200.oit.ncsu.edu simphony1.oit.ncsu.edu simphony2.oit.ncsu.edu spl-p-ds-200.oit.ncsu.edu tex-manager.tx.ncsu.edu titan.ced.ncsu.edu topazapp.oit.ncsu.edu uni02db.unity.ncsu.edu uni200tb.unity.ncsu.edu vrops.oit.ncsu.edu watoga.cnr.ncsu.edu web02rest.unity.ncsu.edu web14cp.unity.ncsu.edu web201home.oit.ncsu.edu

Update completed successfully.

----

The OIT production vCenter portal https://vcenter.ncsu.edu will be down during the scheduled time 3/2 5-6pm to apply updates. During this maintenance only the portal will be unavailable, all OIT hosted virtual machines will continue to operate normally.

If you experience a problem with a VM after the update that requires intervention, OIT_Platform staff will be on call to provide assistance.  

Some virtual machines in DC2-Main0 lost network connectivity during a migration. Scope assessment underway.

All workload has been migrated back to regular hosts. The impacted host is being reviewed.

The following machines lost network connectivity from approximately 11:35 to 11:42 am.

That host has been returned to ma maintenance state and network configurations as well as validation processes are being reviewed.

A host in DC2 crashed, vms were restarted on other hosts in the cluster. The other nodes are being reviewed currently.

Impacted guests may include:

Update: The impacted ESXi host network connectivity has been restored and back in production.

Update: Comtech engineers are en route to replace the failed switch.

Impacted vms migrated to alternate hosts in the cluster are as follows. Service owners, please check to confirm the vm restarted correctly.

Update: Two additional ESXi hosts have been impacted and included in affected list.

At 4:40pm 2 ESXi hosts became unresponsive causing VMs to restart on other ESXi hosts in the cluster. Platform engineers are working with Comtech to determine the root cause.  A list of impacted VM's will be provided shortly.