When Major League Baseball runs its nightly configuration backup for the 1,800 network devices in its infrastructure, it must ensure that it is collecting accurate and current data from various systems.
To implement data consistency between its infrastructure systems, MLB’s network engineering team has launched an initiative to improve its network reliability engineering. This meant better integration between its monitoring system, help desk, server system, discovery and assessment system, GitHub repository and more, said Jeremy Schulman, senior network automation engineer at MLB, at a recent virtual conference hosted by NetBox, an open source project that provides a network management web application. To achieve the best results, the MLB team needed to integrate more network automation into their infrastructure.
“For me, network automation is about integrating systems together to create reliable results,” said Schulman. “If we build systems that are unreliable, then people don’t trust them. And, if people don’t trust them, they don’t want to use them.”
In order for MLB’s disparate systems to exchange data, the systems had to be in sync with each other, Schulman said. This meant the team needed a source of inventory truth – a record that would collect and track data across different infrastructure systems, storing information about the condition, configurations, and maintenance of devices. This source of truth would allow network automation workflows to provide scripts with the data they need to run. It was then that MLB turned to NetBox.
NetBox as a source of truth
NetBox would establish two-way communication with the various systems in the MLB environment, Schulman said, and automatically update its inventory information using data from those systems. For example, NetBox could communicate with MLB’s Simple Network Management Protocol monitoring system to collect and track device data.
If we build systems that are unreliable then people don’t trust them. And, if people don’t trust them, they don’t want to use them.
Jeremy schulmanSenior Network Automation Engineer, MLB
The team’s first use case for NetBox automation was to integrate it with MLB’s ChatOps systems, which developers use to communicate with applications, projects, and infrastructure through a central collaboration platform. . The team set up NetBox as the inventory system for all of MLB’s stadium infrastructure, Schulman said.
“When our ChatOps system presented a series of drop-down lists and widgets on devices and interfaces to select from, it pulled them out of NetBox,” he said.
NetBox for configuration backups
Eventually, the MLB team extended their NetBox use case to automate their nightly configuration backups. NetBox detects and reports which network devices are active and which are offline or disabled, separating which devices should be backed up from those which should be left alone. One of the ways NetBox identifies these devices is tagging.
For example, NetBox compiles information from the entire MLB infrastructure, so its inventory includes non-network devices the team might not want to back up, such as console servers, Schulman said. In response, the team marks these devices as no backup.
MLB uses NetBox markup to specify which devices should not be backed up.
Schulman also created custom dynamic inventory scripts that pull information from NetBox and other MLB systems. The scripts communicate with NetBox to request device configuration information, while they collect broader information about the network environment. Schulman said he initially configured the dynamic inventory script – versus a more generally available plugin – because MLB network engineers were using Ansible Tower to run workflows and playbooks. As a result, the NetBox inventory had to integrate into Ansible Tower so that network engineers could access it.
“If someone wanted to run or write their own playbooks and not run them through Tower, they’ll be using the exact same code in both environments, and it’s important to have that consistency to ensure that the same set of features is available for inventory, ”he said.
The backup process
The MLB team runs Ansible playbooks to perform automated configuration backups. The backup process uses an authorized jump server – a server used to access devices in separate security zones – with a cron job that’s scheduled to run a Bash script, which then runs a series of playbooks, Schulman said. For 1,800 devices, the process can take up to an hour, he added.
An example MLB cron job script
MLB has a GitHub storage repository that it uses to store current network configurations and backups. NetBox also enables features – such as custom links – that make it easier for teams to integrate into GitHub to better track and access network configurations. For example, Schulman inserted a custom link to MLB’s GitHub environment, where users can directly access device configurations and “reduce friction between one system to another,” he said.
More Stories
Itential “Modernizes” the Management of Legacy Network Configurations with a Low-Code Automation Platform | VanillaMore