Port-xen archive

Instructions for running NetBSD on EC2

To : port-xen%netbsd.org@localhost

: Subject : Instructions for running NetBSD on EC2

: From : Alistair Crooks <agc%pkgsrc.org@localhost>

: Date: Wed, 26 Jan 2011 02:46:08 +0100

Hi, I've been asked to distribute the notes and instructions I received about running NetBSD/EC2 to the port-xen list. These are from some time ago (June 2010, I believe), but should be useful to people. I can't remember whether anonymity was requested in this (it's not obvious from my notes of the phone call), I don't think so, but just in case, I'll remove identifying traces. If I've got it wrong, then I'll re-post with the correct attributions. Please advise. Please feel free to contact me if anything is unclear. Thanks, Alistair > Here's my NetBSD-AMI-building notes. It's more or less in a "copy and paste > into console" format. As written it creates an i386 AMI; creating an amd64 > AMI needs a global s/i386/amd64/ plus some adjustment to the ec2-register > command. It also creates the AMI in Amazon's US-East region; creating an AMI > in a different region is just a matter of adjusting the ec2-* command lines. > > The basic approach here is > 1. Build NetBSD locally with some minor changes (the XEN3PAE_DOMU > configuration > is adjusted to find the root disk; root's password is starred out; root logins > via ssh are enabled; dhcp+ssh are turned on; and an extra rc.d script fetches > root's public ssh key the first time the instance boots). > 2. Create a Linux EC2 instance, upload the NetBSD bits to it, and write them > out to EC2 disks. > 3. Snapshot those disks and register a new AMI using them. > > Let me know if you have any problems. #!/bin/sh # PROVIDE: ec2_firstboot # REQUIRE: NETWORKING # BEFORE: LOGIN . /etc/rc.subr name="ec2_firstboot" start_cmd="ec2_firstboot_run" stop_cmd=":" SSHKEYURL="http://169.254.169.254/1.0/meta-data/public-keys/0/openssh-key" SSHKEYFILE="/root/.ssh/authorized_keys" ec2_firstboot_run() { if ! [ -f /etc/ec2_firstboot_done ]; then # Grab the provided SSH public key and authorize it to log in # as root. (Note that /etc/ssh/sshd_config has been edited # to allow root to login directly.) mkdir -p `dirname ${SSHKEYFILE}` chmod 700 `dirname ${SSHKEYFILE}` ftp -o ${SSHKEYFILE}.ec2 -a ${SSHKEYURL} if [ -f ${SSHKEYFILE}.ec2 ]; then touch ${SSHKEYFILE} sort -u ${SSHKEYFILE} ${SSHKEYFILE}.ec2 \ > ${SSHKEYFILE}.tmp mv ${SSHKEYFILE}.tmp ${SSHKEYFILE} rm ${SSHKEYFILE}.ec2 fi # Mark that we've done the first boot. echo "DO NOT REMOVE THIS FILE" > /etc/ec2_firstboot_done fi } load_rc_config $name run_rc_command "$1" ### NetBSD AMI building instructions ### ================================ ### Build EC2ized NetBSD bits ### ------------------------- # Fetch src.tgz, syssrc.tgz, sharesrc.tgz, and gnusrc.tgz into the cwd. # TODO # Extract source tree tar -xf gnusrc.tgz tar -xf sharesrc.tgz tar -xf syssrc.tgz tar -xf src.tgz # Patch source tree for EC2 cat root-starpass.patch | (cd usr && patch -p0) cp usr/src/sys/arch/i386/conf/XEN3PAE_DOMU usr/src/sys/arch/i386/conf/EC2 echo 'config netbsd root on xbd1 type ffs dumps on none' >> usr/src/sys/arch/i386/conf/EC2 echo 'options DDB_COMMANDONENTER=bt' >> usr/src/sys/arch/i386/conf/EC2 sed -EI '' -e 's/^#*config.*//' usr/src/sys/arch/i386/conf/XEN2_DOMU # Build NetBSD rm -rf DEST OBJ TOOLS mkdir DEST OBJ TOOLS export DIRS="-D ../../DEST -O ../../OBJ -T ../../TOOLS" cd usr/src sh build.sh $DIRS -m i386 -N 0 -U tools kernel=EC2 distribution cd ../.. # Create tarball of grub boot filesystem contents mkdir GRUB GRUB/boot GRUB/boot/grub cp OBJ/sys/arch/i386/compile/EC2/netbsd GRUB/boot/kern.netbsd cat > GRUB/boot/grub/menu.lst <<- EOF default=0 timeout=0 hiddenmenu title NetBSD AMI root (hd0) kernel /boot/kern.netbsd root=xbd1 EOF tar -cf GRUB.tar -C GRUB . rm -rf GRUB # Add fstab file cat >DEST/etc/fstab <<EOF /dev/xbd1 / ffs rw 1 1 /dev/xbd0 /grub ext2 rw 2 2 kernfs /kern kernfs rw ptyfs /dev/pts ptyfs rw procfs /proc procfs rw EOF # Add EC2 first-boot configuration script cp ec2_firstboot DEST/etc/rc.d # Tell NetBSD to go ahead and boot properly; to use dhclient to set up its # network; to enable sshd; and to use a hostname of NetBSD-EC2. sed -I '' -e 's/rc_configured=NO/rc_configured=YES/' DEST/etc/rc.conf cat >>DEST/etc/rc.conf <<EOF dhclient="YES" dhclient_flags="xennet0" hostname="NetBSD-EC2" sshd="YES" EOF # Tell sshd to allow root logins (standard for EC2 instances, and necessary # since there are no other users yet). sed -I '' -e 's/.*PermitRootLogin.*/PermitRootLogin yes/' DEST/etc/ssh/sshd_config # Remove some lines from METALOG which have (now incorrect) size values. cat DEST/METALOG | grep -vE '^./etc/rc.conf ' | grep -vE '^./etc/ssh/sshd_config ' > DEST/METALOG.tmp mv DEST/METALOG.tmp DEST/METALOG # Add some necessary bits to METALOG, including replacing the lines we # removed above. cat >> DEST/METALOG <<EOF ./etc/fstab type=file uname=root gname=wheel mode=644 ./kern type=dir uname=root gname=wheel mode=0755 ./proc type=dir uname=root gname=wheel mode=0755 ./grub type=dir uname=root gname=wheel mode=0755 ./etc/rc.d/ec2_firstboot type=file uname=root gname=wheel mode=0555 ./etc/rc.conf type=file uname=root gname=wheel mode=0644 ./etc/ssh/sshd_config type=file uname=root gname=wheel mode=0444 EOF # Create the world disk image ./TOOLS/bin/nbmakefs -B le -F DEST/METALOG -s 10g -N DEST/etc/ -o density=32k -x NetBSD-i386.disk DEST gzip -f NetBSD-i386.disk ### Upload NetBSD bits to EC2 ### ------------------------- # Download and extract the latest EC2 API tools. # Create an AWS x.509 private key & certificate (pk-XXX.pem and cert-XXX.pem). # Create an SSH keypair via EC2. # Set shell variables: # EC2_HOME=/path/to/ec2-api-tools # JAVA_HOME=/path/to/jdk # EC2_PRIVATE_KEY=/path/to/pk-XXX.pem # EC2_CERT=/path/to/cert-XXX.pem # EC2_SSH_KEY=/path/to/ssh_private_key # EC2_SSH_KEYNAME=ec2_ssh_keyname # We will use a linux AMI as a staging point AMI_LINUX=ami-3ac33653 # Create an Amazon Linux instance ${EC2_HOME}/bin/ec2-run-instances ${AMI_LINUX} -z us-east-1b -t t1.micro \ -k ${EC2_SSH_KEYNAME} | tee tmp.run-instances export I_LINUX=`cat tmp.run-instances | grep ^INSTANCE | cut -f 2` while ${EC2_HOME}/bin/ec2-describe-instances ${I_LINUX} | grep -q pending; do sleep 1; done # Get IP address of Linux instance ${EC2_HOME}/bin/ec2-describe-instances ${I_LINUX} | tee tmp.describe-instance export IP_LINUX=`cat tmp.describe-instance | grep ^INSTANCE | cut -f 17` # Create and attach a 1G EBS volume for the grub boot filesystem ${EC2_HOME}/bin/ec2-create-volume -s 1 -z us-east-1b | tee tmp.create-volume export VOL_KERN=`cat tmp.create-volume | grep ^VOLUME | cut -f 2` ${EC2_HOME}/bin/ec2-attach-volume ${VOL_KERN} -i ${I_LINUX} -d "/dev/sdf" # Copy grub boot filesystem contents across scp -i ${EC2_SSH_KEY} GRUB.tar ec2-user@${IP_LINUX}: # Create ext2 filesystem for grub boot ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo mke2fs /dev/xvdf # Mount filesystem ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo mount /dev/xvdf /mnt # Populate filesystem ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo tar -xvf GRUB.tar -C /mnt # Unmount filesystem ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo umount /mnt # Detach the volume and create a snapshot ${EC2_HOME}/bin/ec2-detach-volume ${VOL_KERN} ${EC2_HOME}/bin/ec2-create-snapshot ${VOL_KERN} | tee tmp.create-snapshot export SNAP_KERN=`cat tmp.create-snapshot | grep ^SNAPSHOT | cut -f 2` while ${EC2_HOME}/bin/ec2-describe-snapshots ${SNAP_KERN} | grep -q pending; do sleep 1; done ${EC2_HOME}/bin/ec2-delete-volume ${VOL_KERN} # Create and attach a 10G EBS volume for the world filesystem ${EC2_HOME}/bin/ec2-create-volume -s 10 -z us-east-1b | tee tmp.create-volume export VOL_WORLD=`cat tmp.create-volume | grep ^VOLUME | cut -f 2` ${EC2_HOME}/bin/ec2-attach-volume ${VOL_WORLD} -i ${I_LINUX} -d "/dev/sdg" # Copy world filesystem across scp -i ${EC2_SSH_KEY} NetBSD-i386.disk.gz ec2-user@${IP_LINUX}: # Write filesystem to EBS volume ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} \ 'gunzip < NetBSD-i386.disk.gz | sudo dd of=/dev/xvdg bs=128k' # Detach the volume and create a snapshot ${EC2_HOME}/bin/ec2-detach-volume ${VOL_WORLD} ${EC2_HOME}/bin/ec2-create-snapshot ${VOL_WORLD} | tee tmp.create-snapshot export SNAP_WORLD=`cat tmp.create-snapshot | grep ^SNAPSHOT | cut -f 2` while ${EC2_HOME}/bin/ec2-describe-snapshots ${SNAP_WORLD} | grep -q pending; do sleep 1; done ${EC2_HOME}/bin/ec2-delete-volume ${VOL_WORLD} # Kill Linux instance ${EC2_HOME}/bin/ec2-terminate-instances ${I_LINUX} # Register an AMI ${EC2_HOME}/bin/ec2-register -a i386 --kernel aki-407d9529 \ -b "/dev/sda1=${SNAP_KERN}" -b "/dev/sda2=${SNAP_WORLD}" \ -n "NetBSD `date | tr : -`" | tee tmp.register export AMI_NETBSD=`cat tmp.register | grep ^IMAGE | cut -f 2` ### Test the AMI we just created ### ---------------------------- # Launch a NetBSD instance ${EC2_HOME}/bin/ec2-run-instances ${AMI_NETBSD} -z us-east-1b -t t1.micro \ -k ${EC2_SSH_KEYNAME} | tee tmp.run-instances export I_NETBSD=`cat tmp.run-instances | grep ^INSTANCE | cut -f 2` while ${EC2_HOME}/bin/ec2-describe-instances ${I_NETBSD} | grep -q pending; do sleep 1; done # Read console output (EC2 only polls the instances for output occasionally, # so there will be some delay before the output is available) sleep 240 && ${EC2_HOME}/bin/ec2-get-console-output ${I_NETBSD} # Kill NetBSD instance ${EC2_HOME}/bin/ec2-terminate-instances ${I_NETBSD} ----- End forwarded message -----