Archive Production Configurations in a GitHub repository
Getting an always up to date archive of production configurations into a shared git repository, is the first step in your network automation journey. You can use any git based system – GitHub, GitLab, Bitbucket, etc… to manage this repository. In this document, we are going to use Github.
There are multiple open-source tools available to you to get these configurations into git. To name a few:
All of these systems ask you for an inventory and access credentials and then use SSH under the covers to connect to the devices and retrieve the running configuration. There is no wrong tool to get started with, and in the future you can easily replace it, so pick the one you and your team are most comfortable with.
In this post, I am going to show you how to use Netcfgbu to archive production configurations into a GitHub managed repository.
GitHub setup
First let’s get the GitHub repository setup. Follow these instructions on how to Create a repo.
Note: Make sure you set this to be private, so that only users in your organization have access to it.
You will need to store the URL for Git repo in the environment variable GIT_REPO. You can retrieve the URL by clicking on the green Code button on the top right corner of the page and selecting Clone with SSH.
Next you will need to use a Deploy Key to access that repository. Follow the instructions here to create a new SSH key and add it to the repository as a Deploy Key. Make sure you give that key write access. Store the path to the private key in the environment variable GIT_DEPLOY_KEY.
Netcfgbu setup
Netcfgbu aka Network Configuration Backup is an open-source utility created by Jeremy Schulman. The GitHub repository is jeremyschulman/netcfgbu: Network Configuration Backup, but you can just install it from PyPi by running python3 -m pip install netcfgbu.
Note: Before you get started, remember to create a python virtual environment which you will use for this project. You really don’t want to install modules in your default system python. The rest of this doc assumes you are using pyenv to manage your virtual environments.
Note: Checkout this article to learn how to setup a python virtual environment using pyenv
Note: Make sure your virtual environment is using Python 3.8 or newer
First thing that you need to get started is your inventory. Currently, this must be a .csv file. The example below is from one of our demo lab networks.
ipaddr,host,os_name 192.168.123.22,exitgw,ios 192.168.123.1,leaf01,cumulus 192.168.123.2,leaf02,cumulus 192.168.123.3,leaf03,cumulus 192.168.123.4,leaf04,cumulus 192.168.123.5,leaf10,cumulus 192.168.123.6,leaf11,cumulus 192.168.123.7,leaf12,cumulus 192.168.123.8,leaf13,cumulus 192.168.123.9,leaf14,cumulus 192.168.123.10,leaf15,cumulus 192.168.123.11,leaf16,cumulus 192.168.123.12,leaf17,cumulus 192.168.123.13,leaf18,cumulus 192.168.123.14,leaf19,cumulus 192.168.123.15,spine01,cumulus 192.168.123.16,spine02,cumulus 192.168.123.17,spine03,cumulus 192.168.123.18,spine04,cumulus 192.168.123.19,bl01,cumulus 192.168.123.20,bl02,cumulus 192.168.123.21,fwl01,junos 192.168.123.23,bor01,junos 192.168.123.24,bor02,junos |
Note: The first row defines the structure of the CSV file.
Note: If the hostname can be resolved via DNS (entries are present in the /etc/hosts file on the machine, or they are in your actual DNS system) you do not need the ipaddr column in the inventory file.
Credentials are not stored in the inventory file in our setup. The lab environment uses a single username and password combination for each OS type, so it is simpler to store that information in the TOML configuration file. If your environment isn’t that uniform, we recommend that you store access credentials in the inventory file. See the netcfgbu inventory README for details on the format.
Below is the configuration file used for our lab environment. See the configuration README for details on the various options.
# ----------------------------------------------------------------------------- # # Network Configuration Backup # Configuration File # ----------------------------------------------------------------------------- # ----------------------------------------------------------------------------- # Default Settings # ----------------------------------------------------------------------------- [defaults] inventory = "$CFG_BU_DIR/inventory.csv" configs_dir = "$CFG_BU_DIR/configs" # credentials below are never used credentials.username = "dummy" credentials.password = "dummy" # ----------------------------------------------------------------------------- # # Version Control System(s) # # ----------------------------------------------------------------------------- [[git]] # the first entry does not require a name and it will be treated # as a default; i.e. when the --name option is omitted. repo = "$GIT_REPO" # make sure you are using the git@ URL, not HTTPS deploy_key = "$GIT_DEPLOY_KEY" # SSH key file # ----------------------------------------------------------------------------- # Global SSH Configs # ----------------------------------------------------------------------------- [ssh_configs] kex_algs = [ 'ecdh-sha2-nistp256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1', 'diffie-hellman-group-exchange-sha1' ] # ----------------------------------------------------------------------------- # Jumphost # ----------------------------------------------------------------------------- [[jumphost]] proxy = "$JUMPHOST" include = ['host=.*'] # ----------------------------------------------------------------------------- # # Network OS Specifications # # ----------------------------------------------------------------------------- # ----------------------------------------------------------------------------- # Cisco switches, routers, and firewalls # ----------------------------------------------------------------------------- [os_name.ios] pre_get_config = "terminal length 0" linter = "ios" [[os_name.ios.credentials]] username='$CISCO_USER' password='$CISCO_PASSWD' [os_name.iosxe] linter = 'ios' [[os_name.iosxe.credentials]] username='$CISCO_USER' password='$CISCO_PASSWD' # ----------------------------------------------------------------------------- # Cumulus Linux # ----------------------------------------------------------------------------- [os_name.cumulus] # NOTE: make sure that the user has password-less sudo access, otherwise the # pre_get_config execution will fail. There is no workaround for this. # # Do not change the order of the cat commands either. This ensures the final # file format is recognized correctly by Batfish # (https://github.com/batfish/batfish) pre_get_config = [ "cat -u /etc/hostname > /tmp/running.cfg", "cat -u /etc/network/interfaces >> /tmp/running.cfg", "cat -u /etc/cumulus/ports.conf >> /tmp/running.cfg", "sudo cat -u /etc/frr/frr.conf >> /tmp/running.cfg" ] get_config = "cat /tmp/running.cfg" prompt_pattern = '[a-z0-9.\-@:~]{10,65}\s*[#$]' [[os_name.cumulus.credentials]] username='$CUMULUS_USER' password='$CUMULUS_PASSWD' # ----------------------------------------------------------------------------- # Juniper JUNOS routers, switches and firewalls # ----------------------------------------------------------------------------- [os_name.junos] # NOTE: Do not login as the root user. This will require you to enter CLI mode # prior to getting the configuration, which currently does not work for Juniper # devices. pre_get_config = [ "set cli screen-length 0" ] get_config = "show configuration | display set" [[os_name.junos.credentials]] username='$JNPR_USER' password='$JNPR_PASSWD' # ----------------------------------------------------------------------------- # Linters # ----------------------------------------------------------------------------- [linters.iosxr] config_starts_after = 'Building configuration' [linters.ios] config_starts_after = 'Current configuration' # ----------------------------------------------------------------------------- # Logging - follows Python format as described # https://docs.python.org/3/library/logging.config.html # ----------------------------------------------------------------------------- [logging.loggers.netcfgbu] handlers = ["console", "file"] level = "DEBUG" [logging.loggers.asyncssh] # set the level to warning by default. If you want to enable debugging # use the '--debug-ssh' option to set the debug level from [1-3]; # https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.set_debug_level handlers = ["console"] level = "WARNING" [logging.handlers.console] class = "logging.StreamHandler" formatter = "basic" stream = "ext://sys.stdout" [logging.handlers.file] class = "logging.FileHandler" formatter = "basic" filename = "netcfgbu.log" [logging.formatters.basic] format = "%(asctime)s %(levelname)s: %(message)s" |
NOTE: This configuration file uses Deploy Keys to interact with the Github repo. So make sure the $GIT_REPO variable is using the git protocol, not HTTPS. The URL should look like this: git@github.com:{org}/{repo}.git
NOTE: Netcfgbu supports other platforms not shown in this configuration file. For our lab environment, only the device and OS types that are in use are specified in the configuration file.
NOTE: Device access credentials are stored as environment variables. That is not required, you can inline them in the TOML file.
Now you just need to set up the local git clone. Detailed instructions can be found here, but all you need to do is run netcfgbu vcs prepare In the directory where you have stored the netcfgbu configuration TOML file.
All of your basic setup has now been completed and you are ready to do your first configuration backup. It is highly recommended that you first ensure that there are no issues accessing the devices in your inventory. To do that, run netcfgbu backup login. If there is an issue with your jumphost configuration, or if the device is down, or if your access credentials are incorrect you will be able to discover that and rectify it before attempting your first backup.
Once you have verified that you can log into all of the devices, simply run these commands to retrieve the configurations and commit them to your GitHub repository.
netcfgbu backup netcfgbu vcs save |
If you go to the GitHub repository in your browser you will see all of your running configurations in that repository. To automate this process, set up a cronjob to run these commands at the desired interval. We recommend that you start out with running this every night. This post by DigitalOcean – How To Use Cron to Automate Tasks on Ubuntu 18.04 covers the necessary steps.
Congratulations, you now have a nightly archive of production configurations for your network in a GitHub repository. You are ready to take the next step in network automation – validating network policies with Batfish Enterprise.
