---Followup Post ---

CLS Foreman infrastructure has been successfully upgraded to Foreman 3.12.

--- Initial Post ---

On January 15th beginning at 11:00 AM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version v3.12: https://docs.theforeman.org/3.12/Release_Notes/index-foreman-el.html

During the outage no one will be able to login or interact with the service in any way:

No new configuration changes may be deployed
No new hosts will be able to be provisioned
Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

---Followup Post ---

CLS Foreman infrastructure has been successfully upgraded to Foreman 3.11.

--- Initial Post ---

On October 7th beginning at 1:00 PM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version v3.11: https://docs.theforeman.org/3.11/Release_Notes/index-foreman-el.html

During the outage no one will be able to login or interact with the service in any way:

No new configuration changes may be deployed
No new hosts will be able to be provisioned
Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

---Followup Post ---

CLS Foreman infrastructure has been successfully upgraded to Foreman 3.10.

--- Initial Post ---

On Aug 7th beginning at 1:00 PM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version 3.10: https://www.theforeman.org/manuals/3.10/index.html#Releasenotesfor3.10

During the outage no one will be able to login or interact with the service in any way:

• No new configuration changes may be deployed
• No new hosts will be able to be provisioned
• Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

---Followup Post ---

CLS Foreman infrastructure has been successfully upgraded to Foreman 3.9.

--- Initial Post ---

On June 12th beginning at 11:00 AM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version v3.9: https://docs.theforeman.org/3.9/Release_Notes/index-foreman-el.html

During the outage no one will be able to login or interact with the service in any way:

No new configuration changes may be deployed
No new hosts will be able to be provisioned
Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

OIT Central Services (CSI) will be upgrading Enterprise Resource Planning File Transfer service during the specified time frame.  Servers will be taken offline, upgraded, and reinstated in serial fashion.  The service continues to operate without interruption throughout this entire upgrade. 

In the background on exadata compute nodes, this change provides graceful handling of password expiration/change when accounts are ldap.   The update happens without disruption to users.

---Followup Post ---

CLS Foreman infrastructure has been successfully upgraded to Foreman 3.8.

--- Initial Post ---

On March 11th beginning at 1:00 PM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version v3.8: https://docs.theforeman.org/3.8/Release_Notes/index-foreman-el.html

During the outage no one will be able to login or interact with the service in any way:

• No new configuration changes may be deployed
• No new hosts will be able to be provisioned
• Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

DBA group will be synced over from IPA for the exadata compute production nodes.

--- Followup Post ---

Upgrade has been completed successfully! We are now running Foreman 3.6!

--- Original Post ---

On Aug 10th beginning at 10:00 AM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version 3.6.x: https://www.theforeman.org/manuals/3.6/index.html#Releasenotesfor3.6

During the outage no one will be able to login or interact with the service in any way:

• No new configuration changes may be deployed
• No new hosts will be able to be provisioned
• Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

--- Followup Post ---

Upgrade complete! We are now running Foreman v3.5!

--- Original Post ---

On April 12th beginning at 3:00 PM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version 3.5.x: https://theforeman.org/manuals/3.5/index.html#Releasenotesfor3.5

During the outage no one will be able to login or interact with the service in any way:

• No new configuration changes may be deployed
• No new hosts will be able to be provisioned
• Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

Operating system updates & patches will be applied to the omr20 server (aka "Oracle Enterprise Manager", OEM) at the shown time.  Full hostname: omr20.oit.ncsu.edu. 

--- Followup Post ---

Maintenance is complete. CLS Foreman is running version 3.4.

--- Initial Post ---

On January 4th beginning at 10:00 AM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version 3.4: https://theforeman.org/manuals/3.4/#Releasenotesfor3.4

During the outage no one will be able to login or interact with the service in any way:

• No new configuration changes may be deployed
• No new hosts will be able to be provisioned
• Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

An update to the Splunk Forwarder package is applied at shown time for the new exadata x9 production servers.  The update includes updates for maintaining compliance within scope of PCI.  The servers remain up and active.

Information-only notice of Tenable software installation

Tenable's Falcon-Sensor will be installed at the shown time for specific subset of servers.  The change is non-disruptive and occurs in the background.

Apply changes to SFTP servers for ERP.   This includes patches and various changes to configuration for file transfer (SFTP and FTP processes) and the Linux Operating System.

--- Followup Post ---

Upgrade Complete! We are now running Foreman 3.2!

--- Original Post ---

On Aug 8th beginning at 2:00 PM EST, we will be taking the CLS Foreman service offline for an upgrade.

We will be upgrading the service to version 3.2: https://theforeman.org/manuals/3.2/index.html#Releasenotesfor3.2

During the outage no one will be able to login or interact with the service in any way:

• No new configuration changes may be deployed
• No new hosts will be able to be provisioned
• Puppet agents will continue to apply their previously cached catalogs, but will not send their reports to the server

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

--- Followup Post ---

The upgrade is now complete, and we are running Foreman v3.1. 

Foreman and Puppet services should be fully online and operating as normal, and provisioning services should be functional again. 

--- Original Post ---

The CLS Foreman and Puppet services will be unavailable on May 9th between 2:00PM and 4:00PM. 

We will be upgrading the Foreman server to v3.1. 

Puppet agents will not be able to fetch new catalogs during this time; however, they will apply their previously cached catalog. 

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this upgrade.

--- Followup Post ---

Puppet 7 upgrade complete! We are seeing agents are successfully upgrading to 7 as well. 

Note that any agents running versions older than 6 will not automatically upgrade, and must be upgraded manually. 

Also note that any unsupported operating systems will likely not upgrade successfully, and should be upgraded manually. 

--- Original Post ---

On March 15th, beginning at 10am, the CLS Puppet Servers will be upgraded from Puppet 6 to Puppet 7. 

During the upgrade, Puppet services will be unavailable and agents will use a cached catalog. 

At the conclusion of the upgrade, all agents using the CLS Config Management profile will automatically upgrade their agent to the latest Puppet 7 version on their next puppet run. 

Tenants who wish to prevent this behavior and stay on Puppet 6 can set "profile::config_management::puppet::manage_agent_version: false" in hiera to disable the automated upgrade. 

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns.

2022-01-25 3:19 PM

We finished the Foreman upgrade. The service is running normally.

The Foreman Server used by the CLS tenants will be upgraded on Jan 25th at 2:00 pm. 

During the upgrade, no users will be able to access the Foreman interface, and Puppet execution by all agents will use their last cached catalog. 

We will be upgrading the Foreman version from 2.4.1 to 2.5.4. 

Please contact group-campus-linux@ncsu.edu with any questions or concerns. 

--- Followup Post 2021-09-30 ---

Maintenance completed! Provisioning via PXE will not work until the IPV4 filters are updated in InfoBlox soon. 

--- Followup Post 2021-09-21 ---

The upgrade to RHEL 8 has been postponed until 9/30/21 at 10 AM.

--- Original Post ---

On Sept 21, beginning at 2:00PM, the CLS Foreman service will be migrated to a new host running on RHEL 8. 

During the migration, the Foreman service will be taken down, and no clients will be able to interact with the Foreman service in any way. 

Additionally, puppet runs during this time will not be able to generate a catalog and will fail. 

After this migration, any client using the linux-for-200.eos.ncsu.edu hostname will no longer function. 

Please ensure that all client configuration and scripts refer to the correct VIP instead: foreman.linux.ncsu.edu, puppet.linux.ncsu.edu, puppet-ca.linux.ncsu.edu

--- Followup Post ---

The upgrade has been completed. We are now running Foreman 2.4. All Foreman and Puppet operations should be functioning as normal. 

--- Original Post ---

The CLS Foreman server will be unavailable for maintenance on Tuesday, July 20th from 1PM until 3PM. 

During the maintenance, no users will be able to interact with the Foreman server in any way until the maintenance is complete. 

We will be upgrading the Foreman server from 2.3.x to 2.4.x. https://theforeman.org/manuals/2.4/index.html#Releasenotesfor2.4

Puppet agents will NOT be able to execute catalogs or apply configuration during the maintenance period. 

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this change. 

-- Followup Post --

The new permissions model has been applied. Tenants should see little change in the Foreman interface, and should be able to access all their resources as normal. 

Users with membership in multiple tenants will need to switch between organization in the top left corner, or select "Any Organization" to see hosts in all their tenants.

Please contact group-campus-linux@ncsu.edu if you experience any issues accessing resources in Foreman, or ask in the #cls channel in Slack. 

-- Original Post --

The CLS Foreman server will be partially unavailable for maintenance on Tuesday, July 13th from 1PM until 3PM. 

During the maintenance, no users will be able to view or modify hosts in Foreman until the maintenance is complete. 

We will be modifying the Foreman permissions model to utilize organizations instead of hostgroups. 

After the maintenance, users should be able to use Foreman as normal, with only some minor UI changes related to organizations. 

Puppet agents will be able to execute catalogs and apply configuration as normal during the maintenance period. 

Please contact group-campus-linux@ncsu.edu if you have any questions or concerns regarding this change. 

--- Followup Post 2021-05-19 --

The Shared Puppetfile has been removed from the CLS Puppet servers. 

The following environments still retain a reference to the Shared Puppetfile, and will no longer pull new code as of today until the line is removed:

cnr_master
cosbrc_master
cosmath_Offline_Credentials
csc_development
cuacs_feature_os_auto_updates
cuacs_production
ece_production
ise_development
ise_production
ise_puppetfile_updates
lib_staging
oitiws_dbms_improvements
oitiws_feature_im122dc200
oitiws_feature_im122dg
oitiws_feature_reorganize_idm
oitlab_hiera
oitlab_production
oitlab_test
oitsecurity_nse
oitsecurity_production
oitsecurity_samplicator
sys_master
wolfprint_tftp

--- Original Post ---

The Shared Puppefile will be removed from the CLS Puppet servers on May 19th.

Any control repositories that still reference the Shared Puppetfile at that time will fail to pull new code onto the Puppet servers. 

Tenants should be sure to remove the reference from the Puppetfile in their control repository by May 18th to prevent issues. 

Additionally, the Shared Puppetfile will be truncated on April 21st. This will force all tenants into using the new Global Modules functionality which replaces the Shared Puppetfile. 

Tenants are highly encouraged to test their machines by removing the reference to the Shared Puppetfile before April 21st. 

See the following documentation for more details: https://docs.google.com/document/d/1KugXKMgFtPzDRdljRD_gyUCKepacznKLIEQQXEvqsfA/edit

Please email group-campus-linux@ncsu.edu if you have any questions or concerns. 

The VIP puppet4.linux.ncsu.edu has been replaced with puppet.linux.ncsu.edu, and so will be removed on May 19th. 

Any client configured to use the puppet4.linux.ncsu.edu VIP will cease to communicate with the Puppet servers, and will need to be manually fixed. 

Most machines have already been converted to use the puppet.linux.ncsu.edu VIP, and do not need any changes. 

For more information, including a list of affected machines, see here: https://github.ncsu.edu/ncstate-linux/infrastructure/issues/850

Please contact group-campus-linux@ncsu.edu with any questions or concerns.

The Foreman Server used by the CLS tenants will be upgraded on May 13th at 10:00 am. 

During the upgrade, no users will be able to access the Foreman interface, and Puppet execution by all agents will fail. 

We will be upgrading the Foreman version from 1.24 to 2.3. 

Please contact group-campus-linux@ncsu.edu with any questions or concerns. 

RHEL security updates are being applied to renv-IPA servers.  

• Information only  
• Continuous operation to clients will be provided  
  
• No special action needed 

Click here to view linked Change Request: CHG0030775

--- Followup Post ---

Upgrade complete! Server infrastructure is now running Puppet 6. Clients will be updated via configuration management sometime in the near future once all the details have been worked out. 

--- Followup Post ---

We have identified the source of the issue with the previous upgrade attempt, and have implemented a workaround. We will be upgrading the Foreman server to Puppet 6 on Wednesday Feb 3 at 10:00 AM. Puppet services will be briefly unavailable during the upgrade. 

--- Followup Post ---

The puppetserver processes were having consistent issues compiling catalogs for clients after the upgrade to Puppet 6, even on Puppet 6 agents. Therefore, we have reverted back to Puppet 5 for the time being. We will be investigating this issue further in a dev environment, and rescheduling the Puppet 6 upgrade at a later date. 

--- Original Post ---

The CLS infrastructure will be upgraded to Puppet 6 on Wednesday January 27th at 10:00 AM.

The puppetserver processes will be offline during the upgrade, and so agents will be unable to fetch new catalogs during the upgrade. The Foreman interface will be available, and changes made via Foreman will be applied when the puppetservers are brought back online. New machines will not be able to be provisioned during the upgrade. 

Puppet agents do not need to upgrade at this time, since Puppet 5 agent can successfully communicate with a Puppet 6 server.