Blog by Dave Rose

This article tries to explain how to set up VPN connectivity from your network to the Amazon VPC (Virtual Private Cloud). The Amazon Web Services cloud offering is really quite sophisticated, and it enables businesses to extend their traditional IT services to an environment with little to no financial commitment. So like most IT departments, I decided to get my feet wet in this new boom town, and I quickly realized that it would be so much nicer to manage the virtual servers and services much like I do today with all our other physical gear. The first time I tried this was almost a year ago and it really just wasn’t ready for prime time, and the options for connectivity were limited. They are still slim, but now there is a serious low cost method for extending your network into the cloud. Initially I wanted to extend connectivity to the AWS cloud using our Cisco ASA, however Amazon requires that when using that style of connectivity, that you need to exchange routes using BGP, which unfortunately wasn’t available as an option on our Cisco ASA5520. Digging around the net like usual pointed to many people that had some sort of solution working, but yet not enough information to get it working. I spent some time researching the options available, and landed on using a low cost solution using the community version of Vyatta.

What I used:

• VMware ESXi
• Vyatta Community Edition
• Amazon AWS account with VPC enabled

I’m going to make the assumption here, that you have your Amazon AWS account created and enabled and you also have a working VMware ESXi environment.

Some Background

Amazon Virtual Private Cloud (Amazon VPC)

Get Vyatta Up and Running

You need to download the Vyatta Community edition of the software, at the time of this writing Vyatta Core 6.3 was the most recent version. It comes in ISO format, just download it and save it to your local hard disk. From there you’ll want to upload it into your VMware ESXi hosts local disk store using the Datastore Browser .

From here, you’ll want to create a simple VM of type Other 2.6 Linux (32 bit), which has 512MB RAM, 2 network interface cards, and a 4GB hard disk. One network card should be on your outside or Internet VLAN, and the inside interface card should be on your internal network. Mount the ISO that you recently uploaded to your ESXi host’s datastore and then boot up the VM. The setup should look like this:

Once the ISO boots, it becomes ready, but is in the “live CD” mode and isn’t installed on the local disk. Log in with the userid vyatta and password vyatta.

Once logged in, type the following:

install-image

For the most part, all you need to do is accept the default answers and it will install it to the local hard disk you created.

Type:

shutdown

Shut down the VM, unmount the ISO from the the vSphere Client and boot it back up.

Now you’ll want to configure the most basic network configuration so that you will be able to SSH into it and manage it outside of the vSphere Client.

Log in again using vyatta/vyatta, in my case eth0 is my external interface on the Internet, and eth1 is my internal interface. The following will be different for your situation, but here is what you need to do to get basic connectivity going:

configure

set interfaces ethernet eth0 address 207.182.253.60/27

set interfaces ethernet eth1 address 10.1.12.200/24

set protocols static route 0.0.0.0/0 next-hop 207.182.253.33

set protocols static route 10.1.0.0/16 next-hop 10.1.12.1

set service ssh

commit

save

exit

What I’ve done here is assigned the IP addresses to both the internal and external interfaces, set the default route to be our Internet router, and then routed our internal private address space at our internal router. I’ve also enabled the SSH service so you can now access it from something like a Putty SSH client.

Ok, so now SSH into it, your route table should look something like:

# ssh vyatta@10.1.12.200
Welcome to Vyatta
vyatta@10.1.12.200's password:
Linux vyatta 2.6.37-1-586-vyatta-virt #1 SMP Thu Jul 7 22:30:24 PDT 2011 i686
Welcome to Vyatta.
This system is open-source software. The exact distribution terms for
each module comprising the full system are described in the individual
files in /usr/share/doc/*/copyright.
Last login: Tue Jan 17 22:23:18 2012 from 10.1.12.30
vyatta@vyatta:~$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         207.182.253.33  0.0.0.0         UG        0 0          0 eth0
10.1.0.0        10.1.12.1       255.255.0.0     UG        0 0          0 eth1
10.1.12.0       0.0.0.0         255.255.255.0   U         0 0          0 eth1
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
207.182.253.32  0.0.0.0         255.255.255.224 U         0 0          0 eth0
vyatta@vyatta:~$

So at this point, the basics are there, and we can move onto the Amazon AWS VPC side of things.

Get VPC Up and Running

Before we start, here is a very simple network diagram to help explain things a little:

The next step is to get the Amazon side of things going. Log into your AWS Management Console and select the VPC tab. I’m going to keep things super simple here, I’ll be creating a VPC with Private Subnet Only and Hardware VPN Access.

I used these values:

Customer gateway, use the Internet or public IP address that you assigned to eth0, in my case 207.182.253.60

One VPC section I am using: 10.20.0.0/16

One Subnet section I am using 10.20.1.0/24

It should look like this:

Hit the Create VPC button.

When completed, it will prompt you to Download Configuration, click the button.

For the Vendor, select Generic

This should save a text file to your computer using a filename something like vpn-0f90c14a.txt. You will need this to convert it to Vyatta configuration format. To do the conversion, I found a great little website on the net, and it was just awesome for doing all the heavy lifting, here is the URL you need to navigate to:

http://gen-vyatta-conf.fluxflex.com/en/index.html

Once there, for the IPSEC interface value, use eth0.

For Your Private network, use the network you chose for your internal network on eth1, in my case it was 10.1.12.0/24

For the VPC IP CIDR block, you use the /16 that was defined when you created the VPC in the AWS Management Console. In my case 10.20.0.0/16

Open up the text file you saved, and cut and paste all the text into the section that asks for the Configuration of Generic Customer Gateway, and hit the Generate button.

You will then need to save the Configuration for Vyatta information. I just copied it into my clipboard, and created a file using vi in the Vyatta SSH session.

I saved my file as /home/vyatta/vpc_config.txt and you can get it here: vpc_config.txt.

Now from the SSH prompt type the following:

configure
merge /home/vyatta/vpc_config.txt
commit
save
exit

 

Hopefully at this point, the Vyatta VPN subsystem will start up the connection to the Amazon VPC. You can validate PHASE_1 or IKE on the Vyatta system, by doing the following:

show vpn ike sa

This should show you if you have IKE security associations:

 

vyatta@vyatta:~$ show vpn ike sa
Peer ID / IP                            Local ID / IP
------------                            -------------
204.246.163.76                          207.182.253.60

    State  Encrypt  Hash  D-H Grp  NAT-T  A-Time  L-Time
    -----  -------  ----  -------  -----  ------  ------
    up     aes128   sha1  2        no     1015    28800
    up     aes128   sha1  2        no     284     28800

Peer ID / IP                            Local ID / IP
------------                            -------------
204.246.163.77                          207.182.253.60

    State  Encrypt  Hash  D-H Grp  NAT-T  A-Time  L-Time
    -----  -------  ----  -------  -----  ------  ------
    up     aes128   sha1  2        no     992     28800
    up     aes128   sha1  2        no     283     28800

vyatta@vyatta:~$

So now we have our PHASE_1 IKE SA’s, lets check for the IPSEC SA’s now

show vpn ipsec sa

It should look something like:

vyatta@vyatta:~$ show vpn ipsec sa
Peer ID / IP                            Local ID / IP
------------                            -------------
204.246.163.76                          207.182.253.60

    Tunnel  State  Bytes Out/In   Encrypt  Hash  NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----  -----  ------  ------  -----
    1       up     1.9K/1.9K      aes128   sha1  no     1059    3600    all
    2       up     0.0/0.0        aes128   sha1  no     849     3600    all

Peer ID / IP                            Local ID / IP
------------                            -------------
204.246.163.77                          207.182.253.60

    Tunnel  State  Bytes Out/In   Encrypt  Hash  NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----  -----  ------  ------  -----
    1       up     1.7K/1.6K      aes128   sha1  no     1129    3600    all
    2       down   n/a            n/a      n/a   no     0       3600    all

vyatta@vyatta:~$

Notice that the second tunnel on peer 2 is down, this is ok and expected, Amazon has you create two VPN connections to them for when they do their system maintenace.

Ok, lets make sure our BGP peer is also established:

show ip bgp summary

It should look something like:

vyatta@vyatta:~$ show ip bgp summary
BGP router identifier 207.182.253.60, local AS number 65000
RIB entries 1, using 64 bytes of memory
Peers 2, using 5048 bytes of memory

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.253.17  4  7224     143     144        0    0    0 00:02:23        0
169.254.253.21  4  7224     142     143        0    0    0 00:02:20        0

Total number of neighbors 2

Fix the Gotcha

Before you go and spin up a server instance in your newly configured VPC, you will have to resolve a small little gotcha. It took me a while, as when I first did this, I could NOT talk to the other side. The Amazon VPC is configured for tunnel mode and will only encrypt packets for networks that it learns from the BGP route exchange. In this above default case, your internal network subnet of 10.1.0.0/16 is not allowed to speak to 10.20.0/16 yet because you haven’t announced that route via BGP. You can see this by typing the following:

show ip bgp neighbors 169.254.253.17

It returns the following information:

vyatta@vyatta:~$ show ip bgp neighbors 169.254.253.17 advertised-routes
BGP table version is 0, local router ID is 207.182.253.60
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 0.0.0.0          169.254.253.18           0         32768 i

Total number of prefixes 1
vyatta@vyatta:~$

You will notice that we are only announcing a default route, which in our case won’t allow any VPC instance to talk back to us on the internal network of 10.1.0.0/16 and in our very specific case 10.1.12.0/24. To correct this, you will need to do the following:

configure
set protocols bgp 65000 network 10.1.0.0/16
commit
save
exit

Now when we have a look at the advertised routes, we should see our internal network being advertised to the Amazon VPC VPN.

show ip bgp neighbors 169.254.253.17 advertised-routes

It returns the following information:

vyatta@vyatta:~$ show ip bgp neighbors 169.254.253.17 advertised-routes
BGP table version is 0, local router ID is 207.182.253.60
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 0.0.0.0          169.254.253.18           0         32768 i
*> 10.1.0.0/16      169.254.253.18           0         32768 i

Total number of prefixes 2
vyatta@vyatta:~$

From the Amazon Management Console, if you select the VPC tab an then click the VPN Connections link on the left it should look something like:

Spin up your first instance in your VPC

Now it’s time to get something running on the other side, this blog posting assumes you already know how to do this, so I’ll show you what to expect. Select the EC2 tab in your AWS Management Console and click the Launch Instance button:

I’m just going to pick something simple to demonstrate connectivity:

We’ll just use a 32bit basic Amazon Linux instance:

Amazon makes a “small” instance the minimum for launching in a VPC, so you’ll need to use at least that size, also you will need to select the VPC radio button and choose the subnet to be used for the instance.

The defaults are fine for the next screen as we are just using this for demonstration purposes. If you leave the IP address blank, Amazon provides the instance with an IP from that subnet via DHCP.

Just click continue on the next screen:

And finally choose your Key Pair and click continue:

For this example, the Security Group defaults are fine as well, so click continue:

Ok, you should be ready to Launch:

Once the instance has spun up, you should be able to see that it is inside your new VPC and has been assigned an IP address out of the VPC subnet you previously defined.

Now you should be able to connect to it via SSH:

vyatta@vyatta:~$ ssh -i vantagemedia-west.pem ec2-user@10.20.1.244

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
[ec2-user@ip-10-20-1-244 ~]$

Adjusting the Security Group

One thing you may choose to do, is enable PING to work, and to do so, you would need to modify the Security Group. Go to your EC2 tab, select Security Groups on the left, and highlight the quick-start-1 security group that you created above. Down below, you should select the Inbound tab, and add a Custom ICMP rule, where the Type is Echo Request and the Source is 10.1.0.0/16. Click Add Rule, and click Apply Rule Changes:

You should now be able to ping the instance.

vyatta@vyatta:~$ ping 10.20.1.244
PING 10.20.1.244 (10.20.1.244) 56(84) bytes of data.
64 bytes from 10.20.1.244: icmp_req=1 ttl=62 time=10.7 ms
64 bytes from 10.20.1.244: icmp_req=2 ttl=62 time=10.7 ms
64 bytes from 10.20.1.244: icmp_req=3 ttl=62 time=10.7 ms
^C
--- 10.20.1.244 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 10.710/10.743/10.783/0.123 ms
vyatta@vyatta:~$

The goal of this article is to capture some of the technical details around getting a VMware virtual machine to boot up using PXEBOOT and then have the CentOS 5 operating system installed via HTTP using Kickstart.

Prerequisites

This article makes some assumptions that you have a TFTP and DHCP server installed, I am using the ones that come with CentOS 5.

You will also need to have the CentOS 5 content to use in the Kickstart portion, and I suggest having it local although you don’t have to. I wrote a previous article about How to mirror CentOS 5 and use it as a local YUM repository which should show you how to set things up.

This article does not go into how to set up a PXEBOOT server, however I found this article on the net which should provide a decent starting point.

DHCP is necessary for diskless network booting, and again, I am not going into detail about how to install or configure a DHCP server, nor how to get your network to send those requests to your server, but here is a sample of my relevant DHCP config to allow PXEBOOTing.

/etc/dhcpd.conf

allow bootp;
allow booting;
#
# Virtual Servers Subnet
#
subnet 10.1.16.0 netmask 255.255.255.0 {
  option routers 10.1.16.1;
  next-server 10.1.12.30; filename "pxelinux.0";

  range dynamic-bootp 10.1.16.240 10.1.16.250;
  authoritative;
}

First, we need to get the stock PXEBOOT linux images from our local mirror, if you are looking for information on how to set a local CentOS 5 mirror, you can reference another one of my previous posts.

First we need to make a destination directory and copy the images in:

mkdir /tftpboot/centos/centos5.5/x86_64
cp /usr/local/src/repos/centos/5.5/os/x86_64/images/pxeboot/* /tftpboot/centos/centos5.5/x86_64

Now, the images that come with CentOS 5 , do not contain drivers for the VMXNET2 and VMXNET3, however the Intel E1000 does work fine. I wrote a previous article titled VMware VMXNET driver support when using PXEBOOT and Kickstart that explains how to fix this problem, so you’ll want to read that article, and rebuild your images to include the drivers.
 

I’ve created a template configuration file and then call a wrapper script when I am making new hosts, my template configuration file is as follows:

/tftpboot/pxelinux.cfg/centos5.5.x86_64.template

prompt 1
say ######################################
say HITTING ENTER HERE WILL NUKE THIS BOX!
say YOU HAVE BEEN WARNED!
say ######################################
say Type 0 to boot from HD
label 0
  localboot 0x80
label newinstall
  default centos/centos5.5/x86_64/vmlinuz initrd=centos/centos5.5/x86_64/initrd.img text ks=http://yourserver/centos5.5/x86_64/newhost.cfg

I then call a wrapper script which will be used to create the PXEBOOT and Kickstart configuration files:
 
newpxe_5.5.sh <hostname> <IP 4th octet>

#!/bin/bash

#
# arg 1 is hostname
# arg 2 is IP (last octect)
#

if [ "$1" = "" ];
then
  echo "$0  ";
  exit;
fi

cd /tftpboot/pxelinux.cfg
cp centos5.5.x86_64.template centos5.5.x86_64.$1

sed "s/newhost/$1/g" centos5.5.x86_64.$1 > $1.tmp
mv $1.tmp centos5.5.x86_64.$1

cd /usr/local/src/centos5.5/x86_64
cp template.cfg $1.cfg

sed "s/newhost/$1/g" $1.cfg > $1.tmp
mv $1.tmp $1.cfg

if [ $2 ] ; then
  sed "s/NEWIP/$2/g" $1.cfg > $1.tmp
  mv $1.tmp $1.cfg
fi

As you will notice in the above script, I have a location on disk that I use to store my Kickstart configurations. The above wrapper copies a Kickstart template script called template.cfg which is located on my system in /usr/local/src/centos5.5/x86_64 and edits a couple key items such as the hostname and IP.

I use Apache to serve out this configuration file via HTTP and it is necessary as the PXEBOOT configuration file requests it using the ks= directive.

I’ll include the httpd.conf configuration snippet here:

Alias "/centos5.5/x86_64/" "/usr/local/src/centos5.5/x86_64/"

<Directory "/usr/local/src/centos5.5/x86_64">
    Options Indexes MultiViews FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

Here is the Kickstart template file that you will need to copy into /usr/local/src/centos5.5/x86_64:

/usr/local/src/centos5.5/x86_64/template.cfg

 

You’ll probably also notice that inside the Kickstart file, there is a wget of the VMware Tools, since the goal of this article was to be able to get CentOS 5 on a VMware VM, you really want to have the VMware Tools on your VM. You can edit the template.cfg file accordingly to point to the server that can serve you the file.

We’re now at the point where we would create the VM and start the boot process. We are missing one final crucial piece and that is the MAC address of the VM so that we can link it to the PXEBOOT configuration file that we created above.

Create a VM the way you would normally, and if you built the PXEBOOT images to include the VMXNET3 drivers you can configure your VM to use that network adapter. I’ve decided to do this step in a video, as I figured it would be easier to show people.

This article was recently updated using CentOS 5.5

I know VMware would probably ask why I don’t just clone a reference image when creating additional VM’s, but I find that I like to be able to create a system from scratch and apply my cfengine against it.  Also, contrary to what VMware would like, not every server I bring up is virtual, and I’d like to use the same mechanism for creating physical systems as I do for virtual systems.

My main issue was, that when I used PXEBOOT and Kickstart, I could not get Linux to recognize the vmxnet2 or vmxnet3 NIC.  What I ended up having to do, was to install a E1000 NIC first, go through the PXEBOOT and Kickstart method, install the VMware Tools, and  then shut the VM down, remove the E1000 NIC, boot the VM back up, re-configure IP settings, run vmware-config-tools.pl and finally reboot. Talk about a PIA. I thought, wouldn’t it be nice for me to be able to use the vmxnet2 or vmxnet3  NIC right from the start. It took some time and a bit of Googling, but I was able to piece together a recipe for doing  just that. It may not be perfect, but it works for me. If you can see some room for improvement, please comment.

The operating system I am currently using is CentOS 5.5

Prerequisites:

A VM with the vmxnet2 or vmxnet3 NIC installed and VMware tools configured. You will need this so that you can grab the kernel drivers. This blog assumes you already know or have a working PXEBOOT and Kickstart installation set up.

Ok, lets begin…first you will need to grab the initrd.img from the PXEBOOT directory, originally I got it from the images/pxeboot directory on the CentOS 5.5 DVD

mkdir /tmp/work
cd /tmp/work
cp /mnt/dvd/images/pxeboot/initrd.img .
mkdir initrd

Now we need to unpack the initial ramdisk, on a 2.6 kernel version it is a gzipped cpio archive.

cd initrd
zcat ../initrd.img | cpio -id

This will essentially explode the contents of the initrd.img disk so that we can modify things.

Once you have things exploded, you will need to gather some information for use later, so on your reference VM system perform the following

lspci

 

Now that we know what number we are looking for, we actually need some hex numbers, which you can get by typing:

lscpci -n

We now need to explode the modules cpio archive from within the initrd subdirectory

mkdir /tmp/work/modules
cd /tmp/work/modules
zcat ../initrd/modules/modules.cgz  | cpio -id

In my case the PXEBOOT initrd.img modules.cgz contains x86_64 2.6.18-194.el5 modules.

We want to now copy in the modules from your reference VM system, in my case the modules were slightly newer and located in /lib/modules/2.6.18-194.11.3.el5/misc. These would be here if you have the VMware Tools installed on your reference VM and you have run vmware-config-tools.pl script.

cd /tmp/work/modules/2.6.18-194.el5/x86_64
cp /lib/modules/2.6.18-194.11.3.el5/misc/vmxnet*.ko .
chmod 744 vmxnet*

I recently learned that modules ending in .o are from 2.4 kernels and the .ko are those from a 2.6 kernel. Since CentOS 5.5 is 2.6 based, I grabbed the vmxnet.ko and vmxnet3.ko modules.

Time to pack the modules back up again

cd /tmp/work/modules
find . | cpio -o -H crc | gzip -9 > /tmp/work/initrd/modules/modules.cgz

This will essentially create a new cpio archive and replace the modules.cgz from the original initrd.img. Notice in this case the cpio format type is crc.

Now we need to modify a few more files to get this to work

cd /tmp/work/initrd/modules
vi pci.ids

You will have to search for VMware, and as you will notice there is a 15ad number which came from the above 2 you were supposed to remember.

Now you can put the folowing line in

07b0  VMware Adapter

 

Now you want to add the following to the file module-info

vmxnet
      eth
      "VMware vmxnet ethernet driver"

vmxnet3
      eth
      "VMware vmxnet3 ethernet driver"

Next you will need to get the vmxnet entries for the modules.alias file, on my reference VM system, it was in /lib/modules/2.6.18-92.1.22.el5/modules.alias

cd /lib/modules/2.6.18-194.11.3.el5
grep vmxnet modules.alias  >> /tmp/work/initrd/modules/modules.alias

The contents we are looking for was as follows:

alias pci:v000015ADd00000720sv*sd*bc*sc*i* vmxnet
alias pci:v00001022d00002000sv*sd*bc*sc*i* vmxnet
alias pci:v000015ADd000007B0sv*sd*bc*sc*i* vmxnet3

Ok, now its time to package the initrd.img back up

cd /tmp/work/initrd
find . | cpio -o -H newc | gzip -9 > /tmp/work/initrd.img.vmxnet

Notice in this case the cpio format type is newc.

Now copy that new initrd into your pxeboot environment:

cp /tmp/work/initrd.img.vmxnet /tftpboot/centos/centos5.5/x86_64

Be aware that this just allows the install of the OS via PXEBOOT and Kickstart to happen with a vmxnet NIC, but you will also need to have the VMware Tools installed for the VM as well. Prior to ESXi 4, the VMware Tools came in an RPM format, but apparently VMware had many issues with it and now have gone with the tar.gz install method. Personally the RPM always worked for me. I did find it on their VMware Tools CD image for Linux Guest OSes iso, and in my Kickstart configuration I have a custom YUM repo that I have put the RPM into. So during my Kickstart, the VMwareTools-4.0.0-164009.i386.rpm is installed, and I modified the rc.local to make sure to run the vmware-config-tools.pl script

if [ ! -e /etc/vmware-config.ok ]; then
  /usr/bin/vmware-config-tools.pl --default
  /bin/touch /etc/vmware-config.ok
  /sbin/reboot
fi

At the end of all this, I am able to create a 64bit CentOS 5.5 VM with a vmxnet3 NIC and have it installed using PXEBOOT and Kickstart and come up with VMware Tools installed and active.