Spherical Chicken

Quick note on how to use chef 0.9.12′s knife command to provision EC2 instances from the command line using the Amazon CentOS-based AMI. I’m assuming you already have your ~/.chef/knife.rb setup to talk to either the chef platform or some chef servers that you’ve built and that you can run commands like ‘knife node list’ successfully and have your chef client_key setup. Setting up knife to talk to your chef server is outside the scope of this post.

NOTE: It turns out that the Amazon AMI looks like its based on RHEL6, with some upgraded RPMs. The centos5-based chef install seems to work fine though.

First, of all the server you run knife on needs some extra gems in order to talk to EC2:

sudo gem install net-ssh net-ssh-multi fog highline

Second, you need to setup your Amazon AWS keys in your ~/.chef/knife.rb file. These are *not* your ssh keys associated with your EC2 account. These are the key strings that you use to authenticate to AWS, which you should be able to find here. Add these two lines to your knife.rb:

knife[:aws_access_key_id] = "<your access key goes here>"
knife[:aws_secret_access_key] =  "<your secret key goes here>"

Then you need to download your EC2 ssh private key. If you use multiple regions (us-west and us-east for example) you will need multiple ssh key pairs. I put mine in ~/.ssh/ec2-west.pem and ~/.ssh/ec2-east.pem and I’ve named the keys in the UI “ec2-west” and “ec2-east”.

One you’ve done that, bootstrapping is just one command line, although there is considerable amounts of magic:

knife ec2 server create --region us-west-1 -Z us-west-1b -i ami-d40e5e91 -f t1.micro -G server -I ~/.ssh/ec2-west.pem -S ec2-west --ssh-user ec2-user -d centos5-gems

The breakdown of what the arguments do:

• –region us-west-1: This is the EC2 region east-coast/west-coast/EU/etc.
• -Z us-west-1b: This is the availability zone, the default is an us-east-1 zone, so you need to specify this to make knife work in other regions.
• -i ami-d40e5e91: This is the 32-bit Amazon CentOS-based AMI for the us-west region. If you use an image sized larger than t1.micro you can use the 64-bit version, and if you change the region then you will need to chose the appropriate AMI in that region.
• -f t1.micro: This is the micro/cheap EC2 instance that only supports 32-bit.
• -G server: This is your security group. I’ve created a security group named “server” which opens up DNS and HTTP in addition to SSH. If you have not created a security group try “-G default” here.
• -I ~/.ssh/ec2-west.pem: This is the path to the ssh secret key that I downloaded off my EC2 dashboard — this is also region specific.
• -S ec2-west: This is the name of the ssh secret key pair that I setup in the EC2 dashboard, this is also region specific. The -l argument tells chef what to use to ssh in, the -S argument tells EC2 what to setup on the server. They must obviously agree.
• –ssh-user ec2-user: Use the ec2-user to ssh into the Amazon AMI.
• -d centos5-gems: This overrides the default ubuntu-centric knife install of chef on the target host with a CentOS-5 RPM-based install.

You should then see your instance successfully start up and register with the chef platform. After a bunch of output it should look something like:

ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:33 +0000] INFO: Client key /etc/chef/client.pem is not present - registering
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:37 +0000] WARN: HTTP Request Returned 404 Not Found: Cannot load node i-32ae3376
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:39 +0000] INFO: Setting the run_list to [] from JSON
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:40 +0000] INFO: Starting Chef Run (Version 0.9.12)
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:40 +0000] WARN: Node i-32ae3376 has an empty run list.
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: Chef Run complete in 1.13335 seconds
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: cleaning the checksum cache
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: Running report handlers
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: Report handlers complete

Instance ID: i-32ae3376
Flavor: t1.micro
Image: ami-d40e5e91
Availability Zone: us-west-1b
Security Groups: server
SSH Key: ec2-west
Public DNS Name: ec2-50-18-12-5.us-west-1.compute.amazonaws.com
Public IP Address: 50.18.12.5
Private DNS Name: ip-10-162-215-71.us-west-1.compute.internal
Private IP Address: 10.162.215.71
Run List:

You should be able to use ‘knife node list’ to see the new instance, and you should be able to ssh into the instance using your ssh key.

> knife node list
[
  "i-32ae3376"
]
> ssh -i ~/.ssh/ec2-west.pem ec2-50-18-12-5.us-west-1.compute.amazonaws.com -l ec2-user
Last login: Fri Dec 17 06:32:10 2010 from 

       __|  __|_  )  Amazon Linux AMI
       _|  (     /     Beta
      ___|\___|___|

See /etc/image-release-notes for latest release notes. :-)
[ec2-user@ip-10-162-215-71 ~]$