Discovery - Daily - Check the "Newly Discovered Items" in the Dashboard

Make sure that all the components of the newly discovered CIs are discovered

• Go to Discovery dashboard , select "Newly discovered devices"
• Refer to CSI-ServiceNow Discovery Configuration for Linux machines for detail on the probe commands that collect the serial number, etc
• For each device in the list
  • Support group should be populated
  • Manufacturer, Model and Serial number may not be populated if permissions were not adequate to set the values (may need to check sudoers file and rerun discovery)

 

Discovery - Daily - Check scheduled discovery jobs status executed within the last 24 hours

When checking the status, compare the run time of the discovery job to how long the discovery job took in the past. You are looking for anomalies from the norm. For example a "probe skipped" error indicates a CI that is failing.

• CSI Dashboard contains "PCS-VMI's Request for Discovery"
• On hosts where CI is empty or have a bad config, do Quick Discovery. 
• On hosts where IPs fail, re-run discovery on individual failed IPs (if there are not that many). If there are many failures, this should be worked out with the customer…probably credential problems)
  • Select report in "Discovery Status" (Discovery -> Status)
  • Change to "Devices" tab
  • For Hosts that have no name  in CMDB CI column,  copy the IP
  • Go to "Discovery Schedules"
  • Run "Quick Discovery" on empty IPs
    • Enter IP
    • Select "snowdismidprod" as the mid server for prod  (or snowdismiddev for dev)
  • Open discovery status after it completes
• Switch to Devices tab
• Confirm success
  • If still failing, confirm that the Puppet and Foreman config are using the current CLS Foreman (foreman.linux.ncsu.edu) and Puppet control repo that is managed by a CLS tenant
•  If CMDB CI created
  • Add IP(s) to Discovery IP Ranges  for that particular organization 
  • (Discovery Schedules > organization)
• If CMDB CI still empty
  • Contact owner of host to fix host to allow discovery 
• If the IP has been set, it can be added to the appropriate Discovery schedule

 

Daily - Discovery - Run quick discovery as needed on failed hosts (listed in Discovery Status)

If hosts have credentials but discovery has failed for some other reason

Report errors to customers to get a resolution

 

Daily - Check ServiceNow Store for CMDB related module updates

In the ServiceNow store,  view the CMDB related modules that NCSU is currently using. Subscribing to the modules in the store provides notifications

If any modules have been updated in the ServiceNow Store, that information needs to be shared with the ServiceNow team. Usually this information would be added to the meeting notes for the Wed. and Fri. ServiceNow Peer Review

The CMDB team is currently responsible for this monitoring which typically includes : 

• Discovery
• Service Mapping
• Event Management
• MID Server

 

Daily - Troubleshooting - Update CI data, Find CIs with bad/missing creds or config, Retire CIs older than 60 days (Friday)

KB0020912 - How to update CI Status. This process is the same as "Retiring CIs" below.

 

Daily - Check De-Duplication Tasks 

• Open "De-Duplication" (use the search)
• All of the duplicate tasks opened by the MID server should be in State = Closed Complete
• Run any open de-duplication task opened by the MID server 
• Ignore the ones that have a specific person's name attached to it.

**Known Issues (Duplicate records found in 2 class: Storage Node Element,Disk Shelf Chassis)

 

Daily - Confirm that all server components of each CI is discovered

Note : When this CI is scheduled to be retired in the future, all of the related CIs must also be retired. This retirement process is still in development

If there are many, pick a few at random and check them. There will generally not be adequate time to manually check all CIs for completeness. As we are still developing the process, the manual checking is only temporary.

A typical server CI is represented by a group of component CIs.

A minimal server CI consists of multiple CIs in Classes

• VMWare virtual instance (created when VM is created)
• VMWare network adapter (created when VM is created)
• Network adapter (created during Discovery)
• Server (created during Discovery)
• IP address (created during Discovery)
• Router interface (created during Discovery)
• DNS name (created during Discovery)
• Zabbix (if zabbix agent has been added)

 

Start at Discovery Dashboard to view the recently discovered VMs and servers

• Search for recently discovered CI’s for new VM’s and Servers discovered (last 7 days), choose one and copy the IP address
• Open the CI Class Manager and add "IP Address" to the list of fields being shown
  • Sort by most recent discovery date to see if there are any orphaned components and retire them (old CIs that were not properly retired)
  • Search on the IP address just copied
  • If ServiceNow discovered VMI’s and servers are matched, you are done.  If it does not match, run Discovery and help the CI owner get the servers discovered.
  • Every server includes a minimum of 7 CI's

Friday - Retiring CIs

Process retired VMI CIs from PCS - captured by vCenter Event Collector. 

Eventually to be processed by CMDB data manager as manual processing can take significant time to complete. Currently, vCenter Event Collector only retires the VMI component of the VM. All other component parts of the CI to be retired remain in an active state

CMDB team will mark the component CIs to “Retired” status (in the future this manual process should only be for rare circumstances)

Note:  This process repeats for all the classes (except the classes "Computer" and "Handheld Computing". John C handles those)

Note : Looking into CMDB Data Manager to automatically retire all the related parts…a process which is now manual

Start at Discovery Dashboard

• Select "Dashboards"
• Find "Discovery Dashboard" in the Dashboards screen. It's probably under "Shared with me" and will contain the necessary information
• Open "NCSU unrefreshed devices more than 60 days old" (this will probably be a custom dashboard) including Applications.

Open each class - it may be helpful to include "Most Recent Discovery" in the list of fields that are being shown

• View items in each class (wireless access point, printer, etc)
• Double Click the CI Class to open.
• Check the "Most Recent discovery" if more than 60 days
• Use KB0020912 for process
• For each class of CI
  • Applications - If there are many, just check a few random devices to confirm that the application has been retired
  • VMIs - For VMIs, only the VMI status is changed the CIs Status(install_status) to “Retired”
  • VMIs - MUST NOT BE IN "NONE" (needs to be fixed by PCS)
  • IP Switches - Retire all CI components. Sort by IP and include "Status" column in list to find all components
• On the Status Column> “Hold Ctrl+drag cursor down” to select multiple items to mark, then double click to select Retired.

*Verified CI's that are really a "test" server/device can be excluded.  

**Excluding Computer Class from retiring, John is still working with endpoints CI owners.

If we find situations where VMIs don't retire properly (are turned off),  communicate with PCS to have VMI status updated