Overview & Some Back Story

The team at Unif.io was tasked with eliminating the last piece of on-premises hardware in a client’s infrastructure. This client had a pair of on-premises Cisco ISRs (Integrated Services Routers) that connected their AWS VPC (Amazon Web Services Virtual Private Cloud) of backend service nodes to client devices on a private wireless M2M (machine-to-machine) network.

Existing Topology

To generalize the infrastructure, they needed to connect a private carrier network to their AWS private cloud network. Now, if you are familiar with AWS VPC’s you might be wondering why would you need dedicated on premise Cisco ISRs to accomplish this. Well the answer to that is GRE (Generic Routing Encapsulation) which AWS did not support on their VPC VPN connections.

So what were our Options?

Replacing the Cisco ISRs with a solution in the clients AWS environment required an EC2 (Elastic Cloud Compute) instance that could provide the needed routing capabilities previously handled by the Cisco ISRs. The search for a viable option in the AWS Marketplace resulted in a few different AMIs (Amazon Machine Images):

The Cisco CSR 1000v was certainly not the least expensive of the options, but it would allow migrating the existing Cisco ISR configuration with few modifications. Additionally, many carriers, service providers, and network technicians are familiar with Cisco appliances making maintenance and support less of a frustration. Yes, we probably could have gone with VyOS or pfSense but this would introduce a greater level of complexity to migrating the configuration from the Cisco ISRs and few carriers and network engineers would be familiar with the new configurations. In light of the aforementioned points we selected the Cisco CSR 1000v as it made the most sense for this particular client and infrastructure.

The Continuous Delivery Challenge

Now sometimes a Hybrid cloud solution makes sense, or there is no getting around having some infrastructure components on-premises (or in some type of dedicated DC (Data Center)), but this was the clients last hold out on codifying their entire infrastructure. CD (Continuous Delivery) can be challenging when it comes to infrastructure but with the proper tools, environment and deployment pipeline its fully attainable. This particular client had already invested early on developing a comprehensive pipeline for their AWS infrastructure deployment. With valuable feedback loops already in place, getting the Cisco CSR1000v AMI into their existing Deployment Pipeline was the goal.

Before we configured the Cisco CSR1000v we needed to get it into the existing deployment pipeline as an immutable artifact. Rather than spinning up AWS AMI instances and configuring them at launch in production, the goal was to provision, configure and snapshot an AWS AMI, creating an immutable artifact/machine image. Creating immutable artifacts in the AWS environment in an automated, codified, and versioned fashion can be simplified using a tool such as Packer, which we decided to use with this project. Packer is an open source HashiCorp tool that automates and simplifies the process of creating machine images for multiple platforms, including AWS. The impetus behind this choice was to help meet the goals of having a completely immutable infrastructure with every artifact a snapshot ready to deploy with easy rollback, versioning, and guaranteed parity across environments.

Creating Immutable CSR 1000V Artifact

Once we had our plan in place for generating the Immutable CSR 1000v artifact, we needed to automate the configuration of the instance prior to snapshotting it as the immutable artifact with Packer. To create the Packer AMI configuration files for the Cisco CSR1000v, we needed to determine how to bootstrap its configuration. The CSR 1000V AMI did not come with EC2 or AWS API tools available at boot, nor could they be installed since it was not a standard Linux distribution. This made provisioning post boot via execution of shell scripts very messy via SSH since none of the standard Linux shell utilities were present either. In order to provide a configuration made available at boot, we instead used the User Data field, which needed to be populated with a series of number indexed commands to be later parsed at boot and loaded into the running-configuration of the CSR 1000V. Below is an example of the syntax needed in the User Data provided.

ios-config-1=”username cisco priv 15 pass ciscoxyz”

ios-config-2=”ip scp server enable”

ios-config-3=”ip domain lookup”

ios-config-4=”ip domain name cisco.com”

Bootstrapping the needed configuration in this manner presented a few challenges. There were passwords and other secret data that should not be left in source control, so a means of obtaining that data securely was needed. Also, the User Data configuration syntax was non-standard so it needed to be auto generated from a standard Cisco configuration file to ease readability as well as modification and maintenance for administrators. Lastly, the Cisco CSR needed two separate immutable artifacts created: A Primary and a Secondary version that had slightly different configurations. There were minor differences between the Primary and Secondary configurations, so unifying the commonality between them in one standard template configuration file would ease future maintenance and updates to the configuration for both Cisco CSR artifacts. Our typical approach would have been to create just one immutable artifact reconciling the configuration differences and dependencies via User Data in the launch configuration and for sensitive data via consul-template and Vault/Consul. Unfortunately, the Cisco CSR AMI utilized User Data for configuration in such a way that would have left sensitive data inside of the launch configuration which is considered bad practice. As a result, we opted for separate immutable artifacts which sped up the boot time of the CSR AMI artifact and avoided exposure of sensitive data inside of the instance User Data.

To address these challenges the following files are created in the packer AMI cisco directory:

cisco.json — The Packer AMI configuration file specifying all the Packer required values for build.

— The Packer AMI configuration file specifying all the Packer required values for build. cisco-config.conf — A standard Cisco configuration file to be shared for both primary and secondary configuration using template variables for unique values between the two.

— A standard Cisco configuration file to be shared for both primary and secondary configuration using template variables for unique values between the two. cisco-userdata.json — A JSON formatted file containing sections for the Primary and Secondary configuration values. The key value pairs used in the JSON file correspond to the Template variables used in the cisco-config.conf file. Since passwords are stored in this file the file will be encrypted, hosted in a secure S3 bucket and fetched by Packer during build for parsing.

— A JSON formatted file containing sections for the Primary and Secondary configuration values. The key value pairs used in the JSON file correspond to the Template variables used in the cisco-config.conf file. Since passwords are stored in this file the file will be encrypted, hosted in a secure S3 bucket and fetched by Packer during build for parsing. provision-csr.sh — A shell script that creates two separate base64 encoded User Data files required for the bootstrapping by Packer. While creating each file the script will operate on each line from the cisco-config.conf, reformat the line to the User Data syntax and replace any template variables with their corresponding key values found in the cisco-userdata.json file.

At this point, we had come up with a deployment plan that allowed for the generation of immutable artifacts during the build phase using Packer. The deployment plan provided secure easy to maintain configuration files versioned in source control, immutable Cisco CSR artifacts, and a solid means for integration into the clients automated deployment pipeline. The client’s AWS infrastructure components, such as their VPC, VPG (Virtual Private Gateway), etc., were all managed using another HashiCorp tool: Terraform. Terraform maintains state information regarding the infrastructure and was being used to store versioned artifacts of the AWS infrastructure state, facilitating easy rollbacks and deployments in the client’s pipeline.

Cisco Configuration Challenges

After we had the immutable artifact plan in place and in the client’s deployment pipeline, we needed to dial in the configuration changes required for the Cisco CSR 1000v for it to actually connect to the M2M VPN network. First thing we needed to do was create the new terraform configurations for the AWS infrastructure components the Cisco CSR depended on listed below:

The VPC

The VPG . Which in this case is the Cisco CSR instead of the usual on-premises Cisco ISR.

The Launch Configurations and Autoscaling Group which will maintain the desired number of Healthy Cisco CSR instances available. If one fails a new one will be spawned from the saved immutable AMI in the appropriate availability zone.

The VPN Tunnels to the VPG redundant active/active connections from the Cisco CSR to the AWS VPC of backend service nodes.

The Security Groups/Network ACLs, DHCP Options, Internet Gateway and other VPC related resources for the Cisco CSRs.

Overview of Planned Infrastructure:

In a follow-up post, I will go into detail regarding the AWS infrastructure deployment and Terraform configurations used to deploy it, but want to first jump ahead to the Cisco configuration details and caveats we encountered. Let’s just assume the entire AWS infrastructure is all setup and was codified with Terraform. The next thing we had to do was get VPN information for the Cisco CSR configuration. The configuration files for the VPN connections needed to be downloaded in Cisco ISR 12 format from AWS Web UI. The corresponding VPN files would contain the needed values for connectivity from each CSR to the VPG. Once the VPN configurations were downloaded from AWS, all future deployments would be completely automated since the values for those connections would remain static.

Since the Cisco CSR would be behind an Internet Gateway in its VPC it would have no interface with its static EIP (Elastic IP) so the interface name must be used instead. When the CSR static Elastic IP is referenced inside of the AWS VPN configuration file, it would instead be replaced by “GigabitEthernet1”. This was reflected in the cisco-config.conf where {AWSLocalAddress} and {M2MTunnelSource} were used. In order to keep the Primary Cisco CSR as the preferred path for BGP, the ASN would be prepended but only on the Secondary CSR. To facilitate this, the route-map for the prepend was excluded from the key value pairs in the cisco-userdata.json file.

There were a couple filters that needed to be added to the Cisco CSR BGP configurations to avoid announcing unneeded networks to the Backend Services VPC and to the M2M wireless network. Both VPN connections used the same BGP ASN so they would pass all routes in both directions. To avoid BGP selecting the M2M tunnel as its best path to the M2M tunnel interface we needed to prevent the Cisco CSR from announcing its default route using a prefix-list. Additionally, the M2M connection announced several networks via BGP that were not needed in the Backend Services VPC routes. To avoid unwanted networks being propagated up to the Backend Services VPC, another prefix list was added. The additional prefix list permitted only the required networks for the Backend services VPC to communicate with the clients on the M2M private wireless network. Below are snippets from the cisco-config.conf where this was applied.

prefix-list usage

address-family ipv4 unicast

! add each of the lan ips on m2m as /16s

network 10.0.0.0 mask 255.255.0.0

! add each of the aws neighbors again

! aws 1

neighbor {AWSVPGPInsideIP1} activate

neighbor {AWSVPGPInsideIP1} soft-reconfiguration inbound

neighbor {AWSVPGPInsideIP1} prefix-list deny-default in

neighbor {AWSVPGPInsideIP1} prefix-list route-filter out

! aws 2

neighbor {AWSVPGPInsideIP2} activate

neighbor {AWSVPGPInsideIP2} soft-reconfiguration inbound

neighbor {AWSVPGPInsideIP2} prefix-list deny-default in

neighbor {AWSVPGPInsideIP2} prefix-list route-filter out

! add the m2m again

neighbor {M2MTunnelRemoteIp} activate

neighbor {M2MTunnelRemoteIp} {M2MTunnelDefaultOriginate}

neighbor {M2MTunnelRemoteIp} prefix-list deny-default in

exit

exit

prefix-list declarations

! create prefix lists to prevent default route from m2m and tunnel destinations

ip prefix-list deny-default seq 5 deny 0.0.0.0/0

ip prefix-list deny-default seq 6 deny {AWSVPGPOutsideIP1}/32

ip prefix-list deny-default seq 7 deny {AWSVPGPOutsideIP2}/32

ip prefix-list deny-default seq 8 deny {M2MEndPointIP}/32

ip prefix-list deny-default seq 10 permit 0.0.0.0/0 le 32

! create prefix lists to prevent un-needed routes from getting to aws

ip prefix-list route-filter seq 5 permit 10.0.0.0/16

Since the deployment was automated, the DHCP address assigned to the CSR instances would be unknown and was the only interface available on the routers. The problem with this was that we needed to specify a static default route and could not use the interface name since it wasn’t a point to point interface. To specify the default route in the absence of a known interface IP address we used dhcp as the route.

ip route 0.0.0.0 0.0.0.0 dhcp

An access-list was created to restrict the M2M Crypto map to just the subnet assigned to the Cisco DHCP interface.

crypto-map match address

crypto map {M2MCryptoMapName} 10 ipsec-isakmp

set peer {M2MEndPointIP}

set transform-set {M2MTransformSetName}

match address {M2MGREAclName}

Since there were no other networks, an access-list would be specified to simply “permit any” but only be applied when sent to the M2M VPN end point address.

access-list definition

access-list {M2MGREAclName} permit gre any host {M2MEndPointIP}

applied to GigabitEthernet1 interface

! set the crypto map for the main wan interface

interface {M2MTunnelSource}

! set the MTU size to 9216 for jumbo frames

mtu 9216

ip nat outside

crypto map {M2MCryptoMapName}

So that the M2M private network clients were able to also reach networks outside of the AWS VPC, Network Address Translations were needed. The outside NAT declaration is shown in the above snippet.

nat inside on m2m tunnel interface

! create the tunnel interface for m2m

interface {M2MTunnelInterface}

ip address {M2MTunnelLocalIp} {M2MTunnelLocalIpMask}

ip nat inside

tunnel source {M2MTunnelSource}

tunnel destination {M2MEndPointIP}

no shutdown

exit

nat access-list

ip nat inside source list 90 interface {M2MTunnelSource} overload

To allow for use of the IOS RestAPI a user should be created and the scp server enabled as well.

! turn on scp

ip scp server enable

! now lets create the restapi user just in case

username m2mrapi privilege 15 password {RestAPIUserPass}

For monitoring we needed to integrate the CSR instances with the client’s Logentries service. At the time of this writing, Logentries documentation indicated the Cisco Configuration requires the use of separate port numbers for each client with no security or user verification for logs. To work around this limitation and utilize the Token Based Input, the token was specified inside of a session-id string on the Cisco instances. This caused the token to be present in the string which is all that was required according to the Logentries documentation.

logging to logentries using token string verified work around:

! set the syslog server connection settings

logging trap {SysLogServerTrapLevel}

logging facility {SysLogServerFacilityType}

logging host {SysLogServerHost} transport {SysLogServerProtocol} port \

{SysLogServerProtocolPort} session-id string {SysLogServerTokenString} \

sequence-num-session

Autoscaling and Deployment Automation

To keep the Cisco instance stack flexible it was placed inside a separate VPC, allowing/requiring it to be connected to the Backend Services stack via VPN Gateway. Since each instance had a static elastic IP, it could be connected to any AWS VPC regardless of region. The Cisco stack Autoscaling group was configured to deliver an SNS notification when they are spawned that would be handled by a Lambda function stack. The Ciscos Autoscaling group was configured to ensure the Primary and Secondary Cisco instances were never on the same Availability Zone. At the time of writing, there were four availability zones in us-east-1 A,C,D,E the Primary and Secondary Cisco instances were restricted to two of the four availability zones respectively. With two availability zones for each Cisco instance we achieved fault tolerance against instance health and availability zone outages.

Part 1 — Recap

At this point, we have tested the configuration and initial deployment of Cisco CSR 1000V. The configuration has been kept as immutable as possible and codified for automated deployment using Packer and Terraform storing snapshots of the configured CSR’s as AMI Artifacts.

The CSR 1000V configuration and deployment was only one part of the project. I covered the gotchas but left the Packer and Terraform details for a follow up.

Here is a list of the planned follow-ups to this post:

Cisco CSR 1000V Part 1 — Configuration Overview & Creation of an Immutable Artifact

Automated configuration of a pair of Cisco CSR 1000V AMIs for VPN connectivity between AWS VPCS and a Private Network using GRE/IPSEC/BGP. Creating an immutable artifact and incorporating into an existing deployment pipeline.

Cisco CSR 1000V Part 2 — Terraform, Packer and Atlas

Detailing the immutable deployment strategy, configuration, and framework used for automation and high availability.

Conclusion Part 3 — Tying it all together

Tying together the immutable framework solution used for automated deployment.

References

Unif.io Opensource AWS Terraform Modules

terraform-aws-vpc Module stack that supports full AWS VPC deployment. Users can provision a basic VPC with optional support for Availability Zone (AV) provisioning w/configurable subnet and NAT configuration, DHCP Options Set and Virtual Private Gateway creation.

terraform-aws-asg Module stack supporting multiple deployment scenarios of an Auto Scaling Group to an AWS VPC. Each of the generic Unif.io Terraform modules were nested inside of the projects stack modules as applicable.

HashiCorp Tools

CI Tools and Services

Monitoring Tools Services

Daniel Isaac, Senior Consultant

Unif.io, Inc