Curent Status - Load Balancers
Planned Maintenance - Load Balancers
Load Balancers (Last 90 days)
System History - Load Balancers
Planned maintenance - Load Balancers
-Upgrade complete.
-Upgrade is underway.
The FortiADC Load Balancers will be upgraded to ver 7.6.1
to address the RADIUS vulnerability described in CVE-2024-3596
Planned maintenance - Load Balancers
-Firmware upgrade complete
-The Campus, Comtech, and Primary DNS Load Balancers will be upgraded to ver 7.4.5 build 0364
Planned maintenance - Load Balancers
The Campus, Comtech, and Primary DNS Load Balancers will be upgraded to ver 7.4.5 build 0364
Planned maintenance - Load Balancers
Upgrade Complete.
The Campus, Comtech, and Primary DNS Load Balancers will be upgraded to ver 7.4.5
Planned maintenance - Load Balancers
Hardware replacement complete.
The CMDF and SMDF campus load balancer will be replaced with a newer model 2000F chassis.
Planned maintenance - Load Balancers
-Upgrade complete. Certificate upload and Virtual Server Availability has been verified.
-The Campus, Comtech, and Primary DNS Load Balancers will be upgraded to ver 7.4.3
Planned maintenance - Load Balancers
Upgrade is Complete.
The Campus, Primary DNS, and Comtech Load Balancers will be upgraded to ver. 7.4.1.
Planned maintenance - Load Balancers
Upgrade complete.
The Campus, Comtech, and Primary DNS Load Balancers will be upgraded to ver 7.2.3.
Planned maintenance - Load Balancers
-Hardware has been replaced. All services online.
-Replacing FortiADC 200F HA Pair with newer models to address memory issues.
No outages are anticipated with this change.
Planned maintenance - Load Balancers
-Upgrade complete.
-Firmware upgrade is in progress.
-The Firmware on the DDI Load Balancer will be upgraded from 6.2.1 to 7.2.1
Planned maintenance - Load Balancers
--Comtech load balancer upgrade complete.
--Campus load balancer upgrade complete.
--The Comtech and Campus Load Balancers will be upgraded to firmware version 7.2.1.
Planned maintenance - Load Balancers
-Change complete
-The admin password will be change on the Campus, Comtech, and DDI FortiADC Load balancers.
Planned maintenance - Load Balancers
-Change complete.
-Enabling Duo 2FA on Campus Load Balancer.
Planned maintenance - Load Balancers
-Reboot complete. All connected systems and resources online.
-Maintenance reboot to optimize WAF performance.
Planned maintenance - Load Balancers
-Upgrade is complete for both the Comtech and Campus Load Balancers.
-Upgrade Comtech and Campus Load Balancers from ver 6.2.5 to ver 7.1.1
Planned maintenance - Load Balancers
-Upgrade of Comtech Load Balancers complete. This concludes the planned maintenance. All connected systems are online.
-Upgrade of Campus Load Balancer is complete.
-Upgrading Firmware to address an issue with VDOM access and certificate install.
Degradation - Load Balancers
-Reboot is complete and services are back online.
Emergency reboot of campus load balancers to address issues with VIPs
causing intermittent delays in hosted sites.
Outage - Load Balancers
A work around was put in place at 3:30pm yesterday. There is still an ongoing permissions issue that needs to be resolved, but all system impact is currently mitigated.
-------
Beginning about 1:30pm a handful of university websites began failing certificate validation due to an error updating certificates in the Load Balancer. We are working with Comtech and the vendor on a solution.
bmig.ncsu.edu
privatewell.ncsu.edu
email.web.ncsu.edu
cvm.ncsu.edu
internal.cvm.ncsu.edu
docs.cvm.ncsu.edu
hospital.cvm.ncsu.edu
mbs.cvm.ncsu.edu
news.cvm.ncsu.edu
php.cvm.ncsu.edu
staging.cvm.ncsu.edu
staging.hr.ncsu.edu
pace.ncsu.edu
google-reports.oit.ncsu.edu
wp-updates.oit.ncsu.edu
Planned maintenance - Load Balancers
-Upgrade complete
Upgrading from version 6.1.6 to 6.2.4
Planned maintenance - Load Balancers
-Upgrade Complete.
-Upgrading to ver. 6.2.4 as recommended by support.
Planned maintenance - Load Balancers
-Firmware upgrade complete.
-Upgrade Firmware: balance.ddi.ncsu
Planned maintenance - Load Balancers
-Reload complete.
-The Campus Load Balancer will be reloaded to address an issue with applying WAF profiles
Planned maintenance - Load Balancers
-Upgrade complete.
Upgrading from version 6.1.5 to 6.1.6
Outage - Load Balancers
The load balancer VIP for cdn.ncsu.edu on-campus was down for about 10 minutes when the service broke during a certificate update. This service provides support content for the homepage. Off-campus users are directed to cdn.ncsu.edu in AWS, so they were not affected.
Planned maintenance - Load Balancers
-Upgrade complete
-Upgrading from version 5.3.5 to 6.1.5
Degradation - Load Balancers
Fortinet developers have identified a large amount of historical data causing the WebGUI to become unresponsive. Once this data was removed the WebGUI loaded normally.
Planned maintenance - Load Balancers
-The HA pair became inaccessible after the upgrade to 5.4.5. Firmware has been reverted back to version 5.3.5. All connected systems are back online.
Fortinet support has been contacted to debug issues with the upgrade. Upgrade to be rescheduled for a later date.
-Upgrading from version 5.3.5 to 5.4.5
Outage - Load Balancers
All traffic to/from the "web" vDom in the Load Balancer went down from 9:17-9:30am this morning. This would have affected most WordPress sites, Go Links, Web Registry, the Campus Directory, and a number of smaller sites. Service is restored and we're waiting on a root cause.
Planned maintenance - Load Balancers
-All Firmware Upgrades have completed. Current version 6.2.1.
- A system reboot was required for the initial Firmware upgrade. Continuing to upgrade Firmware versions according to upgrade path.
Upgrading Firmware for functionality with Kubernetes.
Outage - Load Balancers
-The HA cluster and previous settings have been restored.
While troubleshooting an issue with the HA cluster on balance.ncsu.edu some configuration settings were lost. Working to restore HA members from backup to retrieve settings.