Posted on

Build Vyatta AMI

Overview

The following are the major steps involved:

  1. Build 64bit Vyatta ISO image
  2. Install the ISO image to a virtual machine and obtain the root FS image
  3. Modify the above root fs image to adapt to AWS EC2 environment (dark magic, haha)
  4. Create EBS-based AMI from the above modified root fs image

Build 64bit Vyatta ISO image

My starting points are two places:

  1. The README file from vyatta repository. It gives very good instructions on building the vyatta.
  2. This site has developed a script that builds vyatta from source, basically following the above instructions. The only difference is that they were building for 32bit while we are building for 64bit.

This script is my final script used to build the ISO image. A couple of points noted below:

  1. You need a debian 64bit build machine. I used debian 6.0.9. See my debian setup at the end of this page.
  2. I used vyatta daisy branch, which is v6.6r1. See here for more vesioning details
  3. The 64-bit version already has a kernel configured for running on virtual machines (specifically Xen, which is same as AWS EC2). However, for 32-bit versions, one needs to modify build-iso/pkgs/linux-image/debian/rules.gen to use i386/config.586-vyatta-virt instead of i386/config.586-vyatta. The config file appears three times (for setup, build, binary). I don’t understand, just modify them all.

Install ISO

This is relatively straightforward. To save space, I installed the ISO on a 2GB hard disk with VMWare Workstation.

  1. I boot up the VM with liveCD
  2. Log in with “vyatta”/”vyatta”.
  3. Type in “install system”. This will install Vyatta into the hard disk
  4. I boot up the virtual machine with a liveCD (actually the ISO itself) again. Then do something like

    dd if=/dev/sda of=<my root fs image> bs=1M

Modify the root fs image

A lot of try-n-error and dark magic happen in this step. I will try to cover as much as possible.

Mount the root fs image under /mnt/ec2-image

mount -oloop /opt/ec2/images/vyatta-64bit.img /mnt/ec2-image/

Create menu.lst for pv-grub

Create /mnt/ec2-image/boot/grub/menu.lst (Note the suffix is “L”-st, not “One”-st)

default=0
timeout=0
title Jun Vyatta 64bit
root (hd0)
kernel /boot/vmlinuz-3.3.8-1-amd64-vyatta ro root=/dev/xvda1 console=hvc0 rd_NO_PLYMOUTH
initrd /boot/initrd.img-3.3.8-1-amd64-vyatta

Fix /etc/fstab

vi /mnt/ec2-image/etc/fstab

/dev/xvda1  /           ext4         noatime           0    1
/dev/xvda3  swap        swap         defaults          0    0

Fix vyatta config

vi /mnt/ec2-image/opt/vyatta/etc/config/config.boot. Main changes are a) remove eth0 MAC address, b) add sshd service, c) change console

nterfaces {
    ethernet eth0 {
        address dhcp
    }
    loopback lo {
    }
}
service {
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    console {
        device hvc0 {
            speed 9600
        }
    }
    host-name vyatta-64bit
    login {
        user vyatta {
            authentication {
                encrypted-password $1$2LZU31YS$ShE9ovJPjaJGZDCw9iLW20
            }
            level admin
        }
    }
    ntp {
        server 0.vyatta.pool.ntp.org {
....

Set up ssh key access for ‘vyatta’ user

  1. Copy/install /mnt/ec2-image/etc/init.d/ec2-ssh-key
  2. Copy/install /mnt/ec2-image/opt/vyatta/sbin/vyatta_config_ssh
  3. ln -s ../init.d/ec2-ssh-key /mnt/ec2-image/etc/{rc2.d,rc3.d,rc4.d,rc5.d}/S05ec2-ssh-key
  4. Change “CONCURRENCY” to “none” in /mnt/ec2-image/etc/rc file.

Also remove default password for ‘vyatta’ user in /mnt/ec2-image/etc/passwd file.

vyatta:x:1000:100::/home/vyatta:/bin/vbash

Create initial host SSH key

Add the following to the /mnt/ec2-image/etc/rc.local file:

#
# [jsun] generate host key if not available
#
if [ ! -f /etc/ssh/ssh_host_key ]; then
        dpkg-reconfigure openssh-server
fi

/sbin/ifconfig

exit 0

Don’t remember hw-id for eth0

This feature screws up AMI instace each time it is stop and re-started, because the eth0 hw-id will be different. A simple solution is to not remember it at all.

--- backup/opt/vyatta/sbin/vyatta_interface_rescan	2014-03-25 16:53:02.000000000 -0700
+++ /mnt/ec2-image/opt/vyatta/sbin/vyatta_interface_rescan	2014-03-25 15:41:02.505167405 -0700
@@ -132,7 +132,8 @@
 	my $ifpath = interface_type($ifname) . " $ifname";
 
 	syslog(LOG_INFO, "add config for %s hw-id %s", $ifname, $hwaddr);
-	$xcp->create_node(['interfaces',$ifpath,"hw-id $hwaddr"]);
+	#$xcp->create_node(['interfaces',$ifpath,"hw-id $hwaddr"]);
+	$xcp->create_node(['interfaces',$ifpath,"address dhcp"]);
 
 	# Add existing phy entry for wireless
 	if ($ifname =~ /^wlan/) {

Create AMI images

Even though we got a perfect FS image above for AMI, there is no easy way to create one. *sigh*

The theory

Vyatta is using its own Linux kernel. We are relying on a AWS EC2 feature, called, PV-GRUB. We use the AKI for 64bit and partionless disk (hd0 version).

We rely on a AMI build host in AWS that helps us to create a volumn with all the root fs image content. We then create an snapshot from the volumn, and then create AMI from thsnapshot. We then copy the AMI to the targeted regions, if they are different from the build host.

Note that we will ssh into the build host. It is much more convenient if we set up key-based access to the build host. Also the login user must be in “disk” group.

We use the ec2 tool PHP SDK. Please install that first.

The script

The PHP script used to create the AMI is listed here.

Run this script from local host (e.g., the debian 6 vyatta build host). The usage is pretty simple

  1. ./create-ebs-image.php –delete-image : will delete AMI image in the target regions.
  2. ./create-ebs-image.php –delete-snapshot: will delete both AMI image and snapshots in the target regions.
  3. ./create-ebs-image.php –run : will create and deloy AMI images from the local root fs image

How to update a single package

One does not need to go through the above steps if he/she wants to change some source code of a package. Instead, if you already have build vyatta ISO environment and an EC2 instance running the vyatta AMI, use the following procedure.

  1. modify the source code of a pkg, e.g., vyatta-bash. (BTW, all pkgs are under build-iso/vyatta/build-iso/pkgs.)
  2. clean the pkg : tools/submod-clean -d vyatta-bash
  3. re-build .deb package file: tools/submod-mk vyatta-bash
  4. upload pkg to EC2 instance: scp -i <ssh pem> vyatta-bash_999.dev_amd64.deb vyatta@<ec2 ip address>
  5. install the package: sudo dpkg -i vyatta-bash_999.dev_amd64.deb

TODO

  1. It would be good we can modify the vyatta build process and obtain the installed root fs image directly, instead of going through installing ISO and copy root disk steps.

Appendix

Set up Debian 6 build host

- debian 6 net iso install VM;
	system utilities
	ssh server
	graphic install

- install kernel header 
	sudo apt-get install linux-headers-$(uname -r)

- install vmware-tools

- install build packages
apt-get install       ssh build-essential sudo bzip2 curl autoconf git devscripts \
      debhelper autotools-dev automake libtool bison flex lintian \
      libglib2.0-dev libapt-pkg-dev libboost-filesystem1.42-dev \
      libncurses5-dev libdb-dev libssl-dev cdbs libmozjs-dev \
      libreadline5-dev libpam0g-dev libcap-dev libsnmp-dev gawk unzip \
      kernel-package libatm1-dev git-buildpackage libnfnetlink-dev \
      libnetfilter-conntrack-dev libattr1-dev rsync libxml2-dev \
      libedit-dev libpcap0.8-dev libpci-dev lsb-release quilt ruby \
      genisoimage liblzo2-dev unifont libpopt-dev libgmp3-dev \
      libcurl4-openssl-dev libopensc2-dev libldap2-dev libkrb5-dev \
      hardening-wrapper libgcrypt11-dev libpcre3-dev libprelude-dev \
      libgnutls-dev libperl-dev python-all-dev python-setuptools \
      live-helper syslinux libsort-versions-perl libexpat1-dev \
      libfile-sync-perl gcc-multilib libfreetype6-dev libusb-dev \
      libdevmapper-dev libmysqlclient-dev autogen libdumbnet-dev