When Major League Baseball runs its nightly configuration backup for the 1,800 network devices in its infrastructure, it must ensure that it collects accurate and up-to-date data from various systems.
To implement data consistency across its infrastructure systems, MLB’s network engineering team launched an initiative to improve the reliability engineering of its network. This meant better integration between its monitoring system, help desk, server system, discovery and assessment system, GitHub repository and more, said Jeremy Schulman, principal network automation engineer at MLB, during a recent virtual conference hosted by NetBox, an open-source project that provides a web-based network management application. To get the best results, the MLB team needed to build more network automation into their infrastructure.
“For me, network automation is about integrating systems together to create reliable results,” Schulman said. “If we build systems that aren’t reliable, people don’t trust them. And, if people don’t trust them, they don’t want to use them.”
For MLB’s disparate systems to be able to exchange data, the systems had to be in sync with each other, Schulman said. This meant the team needed a source of truth inventory – a record that would collect and track data across different infrastructure systems, storing information about device status, configurations, and maintenance. This source of truth would enable network automation workflows, providing scripts with the data they need to run. That’s when MLB turned to NetBox.
NetBox as a source of truth
NetBox would establish two-way communication with the various systems in MLB’s environment, Schulman said, and automatically update its inventory information using data from those systems. For example, NetBox could communicate with MLB’s Simple Network Management Protocol monitoring system to collect and track device data.
Jeremy SchulmanSenior Network Automation Engineer, MLB
The team’s first use case for NetBox automation was to integrate it with MLB’s ChatOps systems, which developers use to communicate with apps, projects, and infrastructure through a central collaboration platform. . The team implemented NetBox as an inventory system for all of MLB’s baseball stadium infrastructure, Schulman said.
“When our ChatOps system presented a series of dropdowns and widgets on devices and interfaces to select, it pulled them from NetBox,” he said.
NetBox for configuration backups
Eventually, the MLB team extended its NetBox use case to automate its nightly configuration backups. NetBox detects and reports which network devices are active and which are offline or down, separating devices that need to be backed up from those that need to be left alone. One of the ways NetBox identifies these devices is through tagging.
For example, NetBox compiles information from all of MLB infrastructure, so its inventory includes non-network devices the team might not want to back up, such as console servers, Schulman said. In response, the team marks these devices as no backup.
Schulman also created custom dynamic inventory scripts that pull information from NetBox and other MLB systems. The scripts communicate with NetBox to request information about the device’s configuration, while gathering broader information about the network environment. Schulman said he originally set up the dynamic inventory script — over a more generally available plugin — because MLB network engineers used Ansible Tower to run workflows and playbooks. As a result, NetBox inventory needed to integrate with Ansible Tower so that network engineers could access it.
“If someone wanted to run or write their own playbooks and not run them through Tower, they would be using the exact same code in both environments, and it’s important to have that consistency to ensure that the same feature set is available for inventory,” he said.
The backup process
The MLB team runs Ansible playbooks to perform automated configuration backups. The backup process uses an authorized jump server — a server used to access devices in separate security zones — with a cron job scheduled to run a Bash script, which then runs a series of playbooks, Schulman said. For 1,800 devices, the process can take up to an hour, he added.
MLB has a GitHub storage repository which it uses to store current network configurations and backups. NetBox also enables features, such as custom links, that allow teams to more easily integrate with GitHub to better track and access network configurations. For example, Schulman inserted a custom link to MLB’s GitHub environment, where users can directly access device configurations and “reduce the friction of going from one system to another,” he said. he declares.