Curent Status - Backup & Recovery Services
Planned Maintenance - Backup & Recovery Services
Backup & Recovery Services (Last 90 days)
System History - Backup & Recovery Services
Outage - Backup & Recovery Services
**Update: 2/21/2025 13:40:00**
The LDAP authentication for Avamar continues to fail. The support case has been escalated to tier3 support. Backups are completing successfully. If any restores are needed during this period, send a request to oit_platform@help.ncsu.edu
**Update: 2/20/2025**
The Avamar upgrade has been completed. However, an issue with LDAP is currently preventing domain users from logging in. We are actively collaborating with support to resolve this issue as soon as possible.
**Update: 3/18/2025**
The LDAP login issues have been resolved.
The OIT Platform team will be upgrading the Avamar server and its components to version 19.12.0 on Wednesday, February 19th, commencing at 10:00 AM and concluding at 3:00 PM.
Outage - Backup & Recovery Services
The OIT Platform team will be upgrading both production Data Domains to address security vulnerabilities. Please suspend all ongoing backup jobs during the upgrade period.
Outage - Backup & Recovery Services
The OIT Platform group will be installing the 2024 R2 OS Security Update on all Avamar production servers and components to address various vulnerabilities
The 2024 R2 OS Security Update for Avamar production servers and its components has been installed successfully.
Planned maintenance - Backup & Recovery Services
ESX hosts having firmware and hypervisor software update to current levels. Avamar and DataDomain software is not a part of this upgrade just the underlying ESX hosts.
Planned maintenance - Backup & Recovery Services
The production Avamar server (avprod) will be upgraded on February 23rd to address critical vulnerabilities outlined by EMC. The server will be down during the upgrade, so if a restore is needed during the upgrade period, please use the replication server.
Planned maintenance - Backup & Recovery Services
The Avamar replication server (avrep) will be upgraded on February 20th to address a critical vulnerability outlined by EMC.
Planned maintenance - Backup & Recovery Services
The OIT Platform group will upgrade the production Data Domains to version 7.13 to address multiple security vulnerabilities that malicious users could exploit.
The upgrade is scheduled to begin at 10 am and end at 5 pm. During that time, we ask that all backups be paused.
Planned maintenance - Backup & Recovery Services
The DC2 Data Domain located at EDC will be down from 9 am until 5 pm today as the EMC team performs a head swap. Please pause any backups that run during the day until further notice.
Thanks
The EDC Data Domain controller upgrade has been completed and backup services are back online.
Degradation - Backup & Recovery Services
A portion of the enterprise backups for VMs failed 10/19 between the hours of 7:00pm and 10:20pm due to the network event. Staff are in the process of cleaning up. Backups after 10:20pm completed successfully. The failed VM backups will resume during tonights scheduled backups.
Planned maintenance - Backup & Recovery Services
Patching ESX host firmware and OS
Outage - Backup & Recovery Services
The Avamar version upgrade will begin on July 12th at 10 am beginning with the production replication server followed by the production source server on July 13th at 10 am.
Outage - Backup & Recovery Services
The Avamar version upgrade will begin on July 12th at 10 am beginning with the production replication server followed by the production source server on July 13th at 10 am.
Degradation - Backup & Recovery Services
Due to a backup policy synchronization issue between the Avamar service and OIT's vcenter a majority of the VM image level backups failed to run last night. The backups polices have been re-synced and the image level backups are expected to run tonight.
Planned maintenance - Backup & Recovery Services
The faulty DIMM module has been replaced successfully and the EDC Data Domain is back online.
EMC will be replacing a faulty DIMM module in the EDC Data Domain tomorrow 9/29 starting at 10:30 am to 12:00 pm during this time the system will need to be powered down. If you have any backups running during that time please pause them until the work has been completed.
Degradation - Backup & Recovery Services
EMC will be replacing a faulty DIMM module in the EDC Data Domain on 9/29 starting at 10:30 am to 12:00 pm during this time the system will need to be powered down. If you have any backups running during that time please pause them until the work has been completed.
Degradation - Backup & Recovery Services
Update 9/27 15:48: NAS backups completed successfully overnight. The issue was/is related to the lack of enough memory on the storage nodes used for backups. As a workaround, backup jobs are assigned to the front facing nodes which have sufficient memory. Additional memory is being ordered to permanently address the root cause.
Initial post:OIT NAS NDMP backups have experienced partial failures for both the backups and restores since late last week. All local attempts to repair have not succeeded. A severity 1 support case has been opened and is active with vendor for resolution.
Until resolved the existing NAS snapshots have been extended to 28 days.
Avamar VM image level, Avamar agent backups, RMAN/Oracle backups, and AFS backups are not impacted.
Outage - Backup & Recovery Services
Update 9/2 00:05 OIT Avamar backups issue has been resolved and are actively running. OIT staff will closely monitor and address any issues if they occur.
Update 15:37: OIT Avamar backups are still down and engineers continue to work with the vendor under a sev1 (mission critical) support case. Additional information will be provided as engineers work toward resolution.
The majority of OIT VM image level backups failed overnight with ~10% succeeding. Staff is investigating.
Outage - Backup & Recovery Services
All Avamar backups failed to complete last night after Incommon certificate install. The failures are due to a SSL communication issue between Avamar and Data Domain. OIT is currently working to resolve the issue before tonights scheduled backups.
Degradation - Backup & Recovery Services
Backup issue has been resolved and operational now.
-------
OIT Virtual Machine image level backups failed overnight. Backup engineers have a support case open with vendor to resolve as quickly as possible. Network Attached Storage, Oracle, and agent level backups are not impacted.
Degradation - Backup & Recovery Services
The OIT Avamar Image level backups failed overnight. The issue impacted the VMware image level backups and was resolved this morning. Backups are expected to resume tonight after normal business hours.
Degradation - Backup & Recovery Services
Avamar nightly backups failed to run due to backup scheduler being offline, the scheduler has been restarted and nightly backups will resume as scheduled tonight.
Planned maintenance - Backup & Recovery Services
On 3/31/2021 OIT-Platform team will be relocating the primary storage array supporting OIT backup services for campus. The work will occur during business hours after the nightly general backups have completed and planned to be completed by the end of the day on 3/31 before nightly backups start.
Self-service restores will not be available during this time. Submit emergency restore requests to oit_platform@help.ncsu.edu, which will be processed as soon as feasible.
Planned maintenance - Backup & Recovery Services
On Thursday 11/12 9:00am-12:00pm the Data Domain storage arrays operating system will be upgraded. Data Domain provides the storage for all OIT general backup services including, Avamar, AFS, NAS, RMAN(oracle DB). During this time the arrays will be rebooted and will impact any active backup or restore activity.