How to deploy Cisco SD-WAN Controllers.

Overview

This article is going to illustrate the process of how I deployed the Cisco Catalyst SD-WAN control components in Cisco Modelling Labs 2.8. The configuration should also work if you want to deploy them on a hypervisor of your choice.

Prerequisites

You have valid cisco.com credentials.
You have a valid Cisco Smart Account with access to the Plug and Play portal. You can see how to set one up here.
You have a valid Cisco Modelling Labs subscription or access to control component images.

Loading images to Cisco Modelling Labs

If you’re running the control components on a hypervisor you can skip this step.

The Cisco Catalyst SD-WAN control components and routers are made available through a separate refplat file from other. Use the recommended release if you’d like, I was curious to check out the later versions of the Manager so went with 2.8.

Extract the .iso file from the zip file and attach it to a DVD drive of your CML VM. Here is a Hyper-V example.

Browse to the IP address of your CML VM on port 9090 and login using the sysadmin credentials.

Expand the ‘Copy Refplat ISO’ and click the ‘Copy Refplat ISO’ button.

At the end of the copy you’ll see the SD-WAN nodes available in your labs.

Generating the Controller Profile

The controller profile is what defines your unique SD-WAN deployment across all Cisco customers. Even though you may just be creating a lab environment, you still need one as it’s also used to generate serial numbers and OTP tokens for virtual devices (inside CML, EVE-NG or production cloud environments).

To begin log into your Cisco Smart account and find the ‘Plug and Play Connect’ portal. Inside there, add a new Controller Profile as per below.

Select ‘VBOND’ as the Controller Type and click ‘Next’.

Fill out the Profiler Settings page with values applicable to you. FYI, the organization name has to be globally unique. It identifies your unique instance of the SD-WAN fabric. You’ll notice if you try to use cml.lab it’ll say its taken so you’ll need to find your own. In the Primary Controller field enter the unique IP or FQDN of your controller.

Click ‘Next’, review the settings and press ‘Submit’.

Generate software devices

This section only applies if you’re going to be running virtual C8K routers in your labs or in a cloud environment like AWS or Azure.

Back in the Plug and Play Connect portal go to the ‘Devices’ tab and click ‘Add Software Devices’.

Fill out the form as per below. Quantity is how many serial numbers and token will be generated, aka, how many virtual devices you can have online at any time. You can create more later if needed.

Click ‘Next’ and ‘Submit’. You’ll now see 20 devices under the ‘Devices’ tab.

Whilst we’re here lets also download the version appropriate list of devices which will later be uploaded to the Manager.

Topology

The topology I’m using here is very simple. I have Eth0 on all three nodes connecting to my management switch which connect to my Hyper-V switch giving my server connectivity to all three nodes. Then I have Eth1 on the Manager and Controller as well as Ge0/0 on the Validator connected to a flat switch for any to any connectivity. That’s the interfaces the control components will use to communicate with each other. Notice how Validator uses Ge0/0 instead of Eth1.

The ccloud-sw-1 switch is connected to a router which acts as a default gateway through which all the routers will reach the control components so that router will be our default gateway.

The subnet the control components will talk to each other on is 203.0.0.0/24.

Default gateway: 203.0.0.1
Validator: 203.0.0.10
Controller: 203.0.0.11
Manager: 203.0.0.12

Subnet 172.16.200.0/24 will be used for system IP addressing. This range doesn’t need to be routable, its there to act as a pool for unique identifiers of devices. Similar to BGP router ID.

Change admin password

This step is for Cisco Modelling Labs only. For some reason the nodes boot up with default credentials being cisco/cisco. I want to write this article so it can also apply to non CML deployments so I changed admin credentials. I recommend you do the same as you may run into issues later on when we go to generate some certificates. Login with cisco/cisco and change admin on all 3 nodes:

config
 system
  aaa
   user admin password admin
    commit
    end

Configuration

I have an article here if you’re unsure about what the system paraments mean. <INSERT GUIDE HERE>

You’ll notice I move my Eth0 (my management interface) from VPN0 (default) to VPN512 as that’s the dedicated management VPN. You don’t have to use management interfaces if you’re happy to SSH to your controllers on the interface which will be used for communicating with each other and the routers. I wouldn’t recommend that as best practice though.

Console into the control components using either admin/admin or cisco/cisco and enter the following.

Validator:

Make sure you commit after setting the vbond otherwise the ‘encapsulation ipsec’ command under ge0/0 will not be recognised.

config
 system
  host-name validator01
  system-ip 172.16.200.10
  site-id 200
  organization-name cml.lab
  vbond 203.0.0.10 local
  commit
  exit
 vpn 0
  no int eth0
  exit
 vpn 512
  int eth0
   ip address 192.168.0.10/24
   no shut
   exit
  exit
 vpn 0
  interface ge0/0 
   tunnel-interface 
    encapsulation ipsec
    allow-service all
    exit
  ip address 203.0.0.10/24
  no shut
  exit
 ip route 0.0.0.0/0 203.0.0.1
 commit
 end

Controller:

Notice how we don’t need to specify ‘encapsulation ipsec’ here or on the Manager. This is valid configuration.

config
 system
  host-name controller01
  system-ip 172.16.200.11
  site-id 200
  organization-name cml.lab
  vbond 203.0.0.10
  exit
 vpn 0
  no int eth0
  exit
 vpn 512
  int eth0
   ip address 192.168.0.11/24
   no shut
   exit
  exit
 vpn 0
  int eth1
   ip address 203.0.0.11/24
   tunnel-interface
    allow-service all
    exit
   no shut
   exit
  ip route 0.0.0.0/0 203.0.0.1
  commit
  end

Manager:

config
 system
  host-name manager01
  system-ip 172.16.200.12
  site-id 200
  organization-name cml.lab
  vbond 203.0.0.10
  exit
 vpn 0
  no int eth0
  exit
 vpn 512
  int eth0
   ip address 192.168.0.12/24
   no shut
   exit
  exit
 vpn 0
  int eth1
   ip address 203.0.0.12/24
   tunnel-interface
    allow-service all
    exit
   no shut
   exit
  ip route 0.0.0.0/0 203.0.0.1
  commit
  end

Use this command to confirm interface config

show interface | tab

All of your control components should now be able to ping each other

Root CA

In this example I will generate a new root CA on the Manager which will act as my signing server. First lets generate an RSA key.

vshell
openssl genrsa -out root-ca.key 2048

Now lets use this key to create a root CA. Fill out the requested fields and ensure that the Organization Name, Organization Unit Name and Common Name are all the same as the organization name used when creating the controller profile at the start.

openssl req -new -x509 -days 356 -key root-ca.key -out root-ca.crt

Confirm the key and certificate have been created.

ls -l

Install the certificate on the manager by exiting vshell and running this command. If so far you’ve followed along logged in as the ‘cisco’ user, change ‘/home/admin/’ to ‘/home/cisco/’.

exit
request root-cert-chain install /home/admin/root-ca.crt

Now for the Validator and Controller we will use SCP to install the file directly from the Manager. Again, replace ‘admin’ with ‘cisco’ if required. Repeat this on both nodes.

request root-cert-chain install scp://[email protected]:/home/admin/root-ca.crt vpn 0

Browse to the IP address of the Manager and log in. If this is the first time, you’ll be asked to change the password. Note that I’m running one of the latest releases so my GUI might look slightly different but should be similar enough.

Once logged in, open a new tab in your browser and browse to this URL. Replace the IP with your own. This will finish syncing the root CA on the Manager node.

https://203.0.0.12/dataservice/system/device/sync/rootcertchain

Back in the Manager GUI, go to Settings > Organization Name and enter the same organization name you’ve been using so far. In the newer versions of SD-WAN you will see this filled in already from the CLI config.

Now navigate to the ‘Validator’ page and enter the IP of your Validator.

Back in the Manager CLI, go back into the vshell and view the root-ca.crt.

vshell
cat root-ca.crt

Copy the output including the BEGIN and END certificate lines. In your Manager GUI go to Settings > Controller Certificate > Authorization and select the ‘Enterprise Root Certificate’ option. Paste the root CA in there.

Add Control Components

In the Manager GUI browse to Configuration > Devices > Control Components and click ‘Add Validator’. Fill out the form and add.

Repeat for the Controller

At this point the control components will try to establish DTLS tunnels between each other. Although we’ve installed the root CA on all 3 we still need to generate their individual certificates as under the hood it’s still a TLS connection which needs to be secure.

Generate certificates

Navigate to Configuration > Certificates > Control Components and generate a CSR for each node.

Go to your Manager CLI and download the CSR from each node (including itself). This is where you may see an issue if you’ve been using the ‘cisco’ username as I believe they are by default put into the admin directory but not 100% sure.

vshell
scp [email protected]:/home/admin/vbond_csr validator.csr
scp [email protected]:/home/admin/vsmart_csr controller.csr
scp [email protected]:/home/admin/vmanage_csr manager.csr

You should now see them in your directory.

Sign each CSR using the root CA and key.

openssl x509 -req -in validator.csr -CA root-ca.crt -CAkey root-ca.key -CAcreateserial -out validator.crt -days 365 -sha256
openssl x509 -req -in controller.csr -CA root-ca.crt -CAkey root-ca.key -CAcreateserial -out controller.crt -days 365 -sha256
openssl x509 -req -in manager.csr -CA root-ca.crt -CAkey root-ca.key -CAcreateserial -out manager.crt -days 365 -sha256

View the generated certificates.

cat validator.crt
cat manager.crt
cat controller.crt

Go to the Manager GUI Configuration > Control Components > Install Certificate. Copy the cert from the CLI and install one at a time.

If successful, you’ll eventually see all of your control components on the Monitor > Overview dashboard with 0 invalid certs.

Controller device template

At this point, all control components are still managed individually. In order to be able to configure policies on the Manager and push them to the Controller we will need to first make it controller managed. This is achieved by creating a device template and attaching the Controller to it. Go to Configuration, Templates, Device Templates. Click ‘Create Template’ and choose ‘CLI Template’.

Click the ‘Load Running config from reachable device’ and select the Controller. That should load it’s running config into the device template. Click ‘Add’.

Back in the device templates screen, click the ‘…’ icon to the right of the newly created device template and click ‘Attach Devices’. Move the controller from the left section to the right and click ‘Attach’. Click the controller on the left and click ‘Config Diff’. There should be no highlighted sections as we just copied the running config. Click ‘Configure Devices’.

The task should finish with a ‘Success’ status meaning the Controller is now in Manager Mode. You do not need to repeat this for the Validator as the only bit of config it gets from the Manager is the valid device list (see below).

Upload device list

Currently your control components don’t have a list of valid WAN edge devices so if you jump to onboarding routers they will get an invalid serial number error when trying to establish DTLS. We need to take the provisioning file we generated near the start and upload it to the control components.

Go to Configuration > Devices > WAN Edge List and click ‘Upload WAN Edge List’.

Tick the ‘Upload file with serial numbers’ option and browse to the file we downloaded. At the end the Configuration > Devices > WAN Edge List will show all the software routers we generated.

On my version of SD-WAN the list automatically propagates to other control components but to be sure click the ‘Send to Controllers’ button as well.

Verification

Back in the Manager CLI you can run this command to verify the control connections to the Validator and Controller. You’ll notice that even though we only have one Controller we have 6 control connections to it. That’s because there is one connection from each CPU the Manager has. If there were 2 Controllers there would be 12 connections, 6 to each.

show control connection

Similarly you’ll a connection from the Validator the Controller per CPU as well.

And finally the Validator which uses a a slightly different command than the other two but adheres to the per CPU connection rule.

show orchestrator connections

Here is a detailed view of a control connection, in this case on the Manager to the Controller. You’ll notice they use the certificates we generated earlier to establish a DTLS tunnel. Its worth pointing out there is no OMP involved between the controllers. OMP only runs between Controllers and routers as well as Controllers themselves if you have multiple.

show control connections 0 detail

Some other commands you may find interesting.

show control local-properties
show control protocol-stats peer layer 3
show control statistics
show control summary
show control valid-vsmarts
show control valid-vedges