Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available.

Well, one of my micro instances has looked like this with 98-100% steal cycles for hours:

08:59:32          CPU     %user     %nice   %system   %iowait    %steal     %idle
09:00:01          all      5.51      0.00      0.00      0.00     94.49      0.00
09:01:09          all      1.04      0.00      0.01      0.00     98.85      0.09
09:02:02          all      3.41      0.00      0.00      0.00     96.59      0.00
09:03:01          all      1.09      0.00      0.02      0.00     98.84      0.05
09:04:02          all      1.74      0.00      0.00      0.00     98.24      0.02
09:05:02          all     10.15      0.00      1.46      1.74     69.08     17.56
09:06:03          all      4.66      0.00      0.03      0.31     94.75      0.25
09:07:05          all      1.46      0.00      0.00      0.00     98.54      0.00
09:08:02          all      2.98      0.00      0.00      0.00     97.00      0.02
09:09:04          all      3.40      0.00      0.02      0.00     96.28      0.30
09:10:04          all      1.59      0.00      0.00      0.02     98.40      0.00
09:11:15          all     10.16      0.00      0.79      0.83     80.32      7.91
09:12:02          all      2.06      0.00      0.02      0.00     97.88      0.04
09:13:03          all      4.32      0.00      0.00      0.00     95.61      0.07
09:14:01          all      0.00      0.00      0.00      0.00    100.00      0.00
09:15:01          all      3.99      0.00      0.00      0.00     95.99      0.02
09:16:01          all      2.35      0.00      0.00      0.48     94.37      2.80
09:17:01          all     16.01      0.00      0.81      0.61     76.10      6.46
09:18:01          all      0.00      0.00      0.00      0.00    100.00      0.00
09:19:02          all      0.79      0.00      0.00      0.00     99.19      0.02
09:20:02          all      4.56      0.00      0.02      0.00     95.43      0.00
09:21:03          all      0.00      0.00      0.00      0.00    100.00      0.00
09:22:15          all     10.84      0.00      0.76      0.61     83.74      4.04
09:23:01          all      3.70      0.00      0.00      0.00     96.27      0.02
09:24:05          all      5.87      0.00      0.00      0.00     94.08      0.05
09:25:02          all      0.00      0.00      0.00      0.00    100.00      0.00
09:26:01          all      2.02      0.00      0.00      0.03     97.95      0.00
09:27:01          all      0.00      0.00      0.00      0.00    100.00      0.00
09:28:02          all     11.65      0.00      1.09      1.06     77.05      9.16
09:29:09          all      3.24      0.00      0.00      0.00     94.92      1.84
09:30:13          all      2.13      0.00      0.00      0.00     97.87      0.00
09:31:01          all      1.99      0.00      0.00      0.00     97.98      0.02
09:32:02          all      3.43      0.00      0.02      0.00     96.56      0.00
09:33:01          all      0.22      0.00      0.05      0.25     96.12      3.36
09:34:02          all     14.96      0.00      1.16      1.29     70.74     11.84
09:35:01          all      0.95      0.00      0.00      0.00     99.05      0.00
09:36:17          all      5.62      0.00      0.03      0.00     94.36      0.00
09:37:02          all      0.00      0.00      0.00      0.00    100.00      0.00
09:38:02          all      1.84      0.00      0.00      0.00     98.13      0.03
09:39:01          all      1.92      0.00      0.27      0.71     87.51      9.59
09:40:14          all      8.11      0.00      0.43      0.35     87.92      3.19
09:41:01          all      2.46      0.00      0.02      0.00     97.50      0.02
09:42:02          all      2.22      0.00      0.00      0.00     97.78      0.00
09:43:02          all      2.00      0.00      0.00      0.00     98.00      0.00

Note that sar is taking up to 14 seconds at times (e.g. 09:40:14) in order to gather statistics, and it is quite light weight and doesn’t do much other than read a bit out of /proc.

This is so bad that the instance is having 600+ms ping times:

64 bytes from 50.18.x.y: icmp_seq=0 ttl=53 time=609.608 ms
64 bytes from 50.18.x.y: icmp_seq=1 ttl=53 time=1107.454 ms
64 bytes from 50.18.x.y: icmp_seq=2 ttl=53 time=107.230 ms
64 bytes from 50.18.x.y: icmp_seq=3 ttl=53 time=605.416 ms
64 bytes from 50.18.x.y: icmp_seq=4 ttl=53 time=1104.728 ms
64 bytes from 50.18.x.y: icmp_seq=5 ttl=53 time=104.510 ms
64 bytes from 50.18.x.y: icmp_seq=6 ttl=53 time=633.574 ms
64 bytes from 50.18.x.y: icmp_seq=7 ttl=53 time=1133.371 ms
64 bytes from 50.18.x.y: icmp_seq=8 ttl=53 time=133.149 ms
64 bytes from 50.18.x.y: icmp_seq=9 ttl=53 time=631.604 ms

That means that interrupt context and enough horsepower to respond to an ICMP ping is not able to run at all for 600+ ms.

And the problem is definitely on the EC2 side, I’m getting clean pings to the internet:

 1  a.b.c.d  102.019 ms  15.208 ms  13.735 ms
 2  69.17.83.233  11.109 ms  10.859 ms  10.363 ms
 3  209.247.91.169  10.861 ms  12.741 ms  12.855 ms
 4  4.68.105.30  11.484 ms  16.482 ms  17.860 ms
 5  4.69.132.49  27.975 ms  28.601 ms  30.103 ms
 6  4.69.153.18  30.226 ms  28.726 ms  28.478 ms
 7  4.69.152.16  28.352 ms  61.582 ms  31.225 ms
 8  4.53.208.22  31.100 ms  30.775 ms  30.788 ms
 9  72.21.222.208  32.222 ms  32.219 ms  30.851 ms
10  72.21.222.255  33.224 ms  31.350 ms  31.226 ms
11  * * *
12  * * *
13  * * *
14  50.18.x.y  393.064 ms  1500.394 ms  989.786 ms

This isn’t “small amounts of consistent CPU”, this is useless. They’re clearly prioritizing micro instances down to the point where if the server is otherwise being utilized 100% the micro instances get completely queue starved.

First of all, outbound ACLs create constant rollout failures of software. Unless you’ve got really excellent process around replicating the ACLs in pre-production and tracking the ACLs as part of rollout procedures you will inevitably wind up with rollout/deployment failures in production. I’ve been on way too many post mortems of software deployments where the failure to track production outbound ACLs has been the cause. I’ll say what doesn’t get said at these meetings, which is that the outbound ACLs themselves are the cause of the problem.

Second of all, with all the cloud services now the outbound ACLs are only going to cause more and more problems. All my servers now talk to chef servers “in the cloud” run externally by opscode. All my Dell servers pull OMSA and firmware directly from Dell. The software dev teams are using github more and more and pulling directly from those servers “in the cloud”. As more and more external platform dependencies are built it becomes impossible to deal with them all, particularly as the platform vendors change their own IPs around on their own schedule. Opscode migrated from Amazon EC2 to rackspace and I never knew about that — if we had outbound ACLs everything would have broken, or it would have been a huge firedrill. In the past 9 months I’ve gone from having basically no external dependencies on “the cloud” and burning tons of time carefully mirroring all my CentOS and RHEL RPMs, to having 3 major external dependencies to “the cloud”, and I’ve gotten more done and slept fine at night.

Sure, you can say that we should minimize all of this. Replicate the dell stuff with in-house mirrors, stop using github and run our internal git servers, setup chef servers internally and host it ourselves. But you just cut three big checks on our time, and we’re simply not staffed to handle that. Even if we were staffed to handle it, if you stack rank what I care about none of that comes close to the top of the list of stuff that I would do if we had 2 more headcount. I’ve given up thinking that its an appropriate job for a system admin to in-house everything that can just be outsourced to “the cloud” in order to remove external dependencies of the enterprise.

Sure this increases risk, but the job of the system admin is not to decrease every single risk that they find. Everything needs to be stack ranked and needs to be assessed in terms of its importance. I still have huge issues with account management and our deployment process is crap and we need to patch our servers and I don’t have enough time to do any of that. How much time do I have to worry that my servers point directly at Dell’s repo for OMSA? I don’t have time to care about that. There’s much riskier things in my infrastructure that if I ignore them I will get hacked or will definitely cause me downtime.

And sure, outbound ACLs help after you do get hacked. I’d argue, however, that the cost of the outbound ACLs is starting to greatly exceed the level that they help. I’d also argue that you need to focus on preventing the attacks from being successful in the first place. The use of outbound ACLs is lazy security engineering and damages the rest of the business. Become pro-active and scan your network for open proxies and close them, don’t rely on outbound ACLs so that you can have open proxies available to the internet, but its okay (because nobody can use them to bounce into the internet and suck up your bandwidth — they can only scan your whole internal network — its okay!). Tighten up your border security and if you must install “detect” controls instead of “prevent” controls. Log when you see anomalous behavior.

There also will still be PCI-DSS to deal with, and every security best practices document out there probably start approaching the network from the perspective of locking everything down and then poking tiny little holes in it — writing checks on your time to manage all of that. The people who wrote those docs don’t have to manage your network and it only took them a few minutes to draft that policy, while you’re stuck with the fallout from the policies for the rest of your career. You don’t have to follow them if they’re no longer working. Except for VISA, you have to follow that, but VISA would prefer that your servers all crash rather than having you violate their standard. So, wall off the PCI-DSS environment and get used to managing the outbound firewall ACLs there and suck it up as a cost to the business. But do not make the mistake of thinking that what you do for PCI-DSS is necessarily a best practice and that it should be applied to the rest of your Enterprise.

Also, certain outbound ACLs are fine. None of your production servers should probably be making IRC connections. By all means block that port and have any outbound connection to IRC servers set off security alarms that wake up your security engineers and put them to work. Similarly, you can probably engineer your SMTP sending infrastructure so that most leaf nodes forward to dedicated SMTP internal relays and only those servers are allowed to send outgoing SMTP. On a case-by-case basis that is fine. But port 80 and port 443 to the internet are going to need to be wide open in the future.

Get used to it. At some point, you will not be able to keep up and “the cloud” will solve too many problems to be able to continue to burn time replicating internet resources in-house, and the ACL management will become a nightmare and more of a risk to uptime to manage badly than to simply open it up.

Its time to stop considering outbound ACLs a “best practice” and start considering it something that bad SEs and NEs do who are just control freaks that haven’t evolved out of the late 90s best practices. We have passed an inflection point where this practice is no longer net positive to the Enterprise and it is net negative in terms of management and availability. Stop thinking that its part of doing a good job to block everything from being able to talk to the Internet. You are in the way of the business. You are an impediment to getting things done. Servers, both inside your company and outside, desperately want to talk to each other to get work done, stop getting in the way.

FAIL.

The stated goal of Amazon’s AMI is to provide essentially a RHEL6 kernel on a RHEL5 ABI. However, if you really try to use RHEL5 RPMs you’ll run into things like curl being upgraded to:

curl-7.19.7-15.16.amzn1.x86_64

Which is basically the RHEL6 version:

curl-7.19.7-6.el6.x86_64.rpm

Not the RHEL5 version:

curl-7.15.5-9.el5.x86_64.rpm

That breaks all kinds of RHEL5 RPMs, while its true that the rest of the O/S isn’t RHEL6 enough to install RHEL6 RPMs successfully. The result is a painful inability to consistently install either RHEL5 or RHEL6 RPMs.

Its possible that the Amazon AMI might collect itself a following of people who build up public repos with all the RPMs that you can get from DAG, EPEL, ELFF, Jpackage, etc for the Amazon AMI. However, right now its basically its own beast — not RHEL5 or RHEL6 compatible — with no publicly-available prebuilt package repos for it. Since I have better things to do than recompile packages for it, its kinda useless to me, so I’d recommend the Rightscale CentOS 5.4 EBS-based images (to use on the t1.micro instance) instead.

NOTE: It turns out that the Amazon AMI looks like its based on RHEL6, with some upgraded RPMs. The centos5-based chef install seems to work fine though.

First, of all the server you run knife on needs some extra gems in order to talk to EC2:

sudo gem install net-ssh net-ssh-multi fog highline

Second, you need to setup your Amazon AWS keys in your ~/.chef/knife.rb file. These are *not* your ssh keys associated with your EC2 account. These are the key strings that you use to authenticate to AWS, which you should be able to find here. Add these two lines to your knife.rb:

knife[:aws_access_key_id] = "<your access key goes here>"
knife[:aws_secret_access_key] =  "<your secret key goes here>"

Then you need to download your EC2 ssh private key. If you use multiple regions (us-west and us-east for example) you will need multiple ssh key pairs. I put mine in ~/.ssh/ec2-west.pem and ~/.ssh/ec2-east.pem and I’ve named the keys in the UI “ec2-west” and “ec2-east”.

One you’ve done that, bootstrapping is just one command line, although there is considerable amounts of magic:

knife ec2 server create --region us-west-1 -Z us-west-1b -i ami-d40e5e91 -f t1.micro -G server -I ~/.ssh/ec2-west.pem -S ec2-west --ssh-user ec2-user -d centos5-gems

The breakdown of what the arguments do:

• –region us-west-1: This is the EC2 region east-coast/west-coast/EU/etc.
• -Z us-west-1b: This is the availability zone, the default is an us-east-1 zone, so you need to specify this to make knife work in other regions.
• -i ami-d40e5e91: This is the 32-bit Amazon CentOS-based AMI for the us-west region. If you use an image sized larger than t1.micro you can use the 64-bit version, and if you change the region then you will need to chose the appropriate AMI in that region.
• -f t1.micro: This is the micro/cheap EC2 instance that only supports 32-bit.
• -G server: This is your security group. I’ve created a security group named “server” which opens up DNS and HTTP in addition to SSH. If you have not created a security group try “-G default” here.
• -I ~/.ssh/ec2-west.pem: This is the path to the ssh secret key that I downloaded off my EC2 dashboard — this is also region specific.
• -S ec2-west: This is the name of the ssh secret key pair that I setup in the EC2 dashboard, this is also region specific. The -l argument tells chef what to use to ssh in, the -S argument tells EC2 what to setup on the server. They must obviously agree.
• –ssh-user ec2-user: Use the ec2-user to ssh into the Amazon AMI.
• -d centos5-gems: This overrides the default ubuntu-centric knife install of chef on the target host with a CentOS-5 RPM-based install.

You should then see your instance successfully start up and register with the chef platform. After a bunch of output it should look something like:

ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:33 +0000] INFO: Client key /etc/chef/client.pem is not present - registering
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:37 +0000] WARN: HTTP Request Returned 404 Not Found: Cannot load node i-32ae3376
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:39 +0000] INFO: Setting the run_list to [] from JSON
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:40 +0000] INFO: Starting Chef Run (Version 0.9.12)
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:40 +0000] WARN: Node i-32ae3376 has an empty run list.
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: Chef Run complete in 1.13335 seconds
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: cleaning the checksum cache
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: Running report handlers
ec2-50-18-12-5.us-west-1.compute.amazonaws.com [Sat, 18 Dec 2010 17:29:41 +0000] INFO: Report handlers complete

Instance ID: i-32ae3376
Flavor: t1.micro
Image: ami-d40e5e91
Availability Zone: us-west-1b
Security Groups: server
SSH Key: ec2-west
Public DNS Name: ec2-50-18-12-5.us-west-1.compute.amazonaws.com
Public IP Address: 50.18.12.5
Private DNS Name: ip-10-162-215-71.us-west-1.compute.internal
Private IP Address: 10.162.215.71
Run List:

You should be able to use ‘knife node list’ to see the new instance, and you should be able to ssh into the instance using your ssh key.

> knife node list
[
  "i-32ae3376"
]
> ssh -i ~/.ssh/ec2-west.pem ec2-50-18-12-5.us-west-1.compute.amazonaws.com -l ec2-user
Last login: Fri Dec 17 06:32:10 2010 from 

       __|  __|_  )  Amazon Linux AMI
       _|  (     /     Beta
      ___|\___|___|

See /etc/image-release-notes for latest release notes. :-)
[ec2-user@ip-10-162-215-71 ~]$

It makes it easy to install some useful utility stuff like lsof and strace. However, if you mistakenly run ‘yum -y upgrade’ on a XenServer which is pointed at an enabled CentOS 5 repo, it will upgrade the gnupg RPM which nukes some special sauce that Citrix has put in there. This also updates your /etc/redhat-release file to suggest that your XenServer is now Citrix which can break cfengine, chef, etc which all look in that file to determine what flavor of redhat-ish server you are running and will now treat your box just like any other CentOS box.

One solution to this is simply to do something like on a sane XenServer:

tar -cvzf /tmp/gnupg.tar.gz `rpm -q -l gnupg`

Then copy that to a broken XenServer and something like:

cd / && tar -xvzpf /path/to/gnupg.tar.gz

That’ll screw up your RPM database a little bit, but it’ll repair the problem.

If you can get the gnupg RPM out of the base distro via yum and install that, it would probably be better to repair the system that way.

I also have a suspicion that its possible to do an ‘upgrade’ of the XenServer dom0 from the busted 5.6 to a vanilla 5.6 which should do a wipe-and-reinstall of the dom0 while preserving all the domUs on the server (I discovered doing upgrades from 5.5 to 5.6 that this is what actually happens with a XenServer ‘upgrade’ — slightly surprising, but possibly useful in this kind of a situation). This would probably be the best solution since it would back out any other problems caused by the errant ‘yum upgrade’. I haven’t tried this yet, but its on my list of things to explore.

I think that best practice with XenServer for “third-party” RPMs should probably be to setup a blank repo and copy only the RPMs that you want from CentOS5 into that repo. Either that or to have the CentOS5 repo be disabled in your yum config and only –enablerepo it when you want to install a few utilities like strace and lsof. That will keep admins from accidentally causing quite a bit of chaos to the distro.

Of course I think this only points out how KVM is going to ultimately be much more preferable to XenServer. Since KVM just runs on a vanilla CentOS image, a ‘yum upgrade’ is something that you really want to be doing on that image. You want to manage it more-or-less like all the rest of your CentOS distributions. XenServer will undoubtedly wind up going towards a more closed VMWare-ESXi-like model where the hypervisor is a thinner shim that no longer is a linux O/S image that you can’t log into and cause this kind of chaos. After I hosed my XenServers this way and knew it was my own damn fault, I had the good sense not to bother Citrix customer support with this stupid problem, but I’m sure that others have engaged Citrix on this, and those people (legitimately) don’t want to see this kind of issue — this will be yet another point that drives them towards a more closed platform — and will drive me towards KVM.

Well, I surfed around through the options available in the /etc/chef/client.rb file and wound up frustrated because nobody had spec’d out the obvious use case of needing to set environment variables. I considered submitting a bug for this. I wrote out the bug in my head, and it was going to suggest either a specific “path=” directive to set the path, or some more complicated directive, ideally, to support setting arbitrary environment variables.

Then I pondered what Adam had told me a couple days earlier to just drop code in there. In this case what we were discussing was setting the node_name off of `/bin/hostname`. I momentarily meditated on the fact that it really was a client.*RB* file and not client.*CONF*. Could it be this easy:

# cat /etc/chef/client.rb
log_level        :info
log_location     STDOUT
chef_server_url  'https://api.opscode.com/organizations/CENSORED'
validation_client_name 'CENSORED-validator'

Ohai::Config[:plugin_path] << '/etc/ohai/plugins'

ENV['PATH'] = "/usr/local/ruby-1.8.7/bin:/usr/local/perl-5.10.1/bin:/usr/local/sbin:/usr/local/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/sbin:/usr/bin:/sbin:/bin"

And, yes, apparently it just is that easy. Nobody at opscode had to think about supporting setting ENV variables in the conf file, because they just executed the conf file so I can just embed ruby in there and change the behavior in ways that the authors possibly never considered.

Code-as-configuration FTW.

DevOps is a developing meme which seems to be attacking the wall between ‘development’ and ‘operations’, incorporating agile methodologies in ‘operations’ and expressing ‘infrastructure as code’. This is a very good thing, and its about damn time. But I’ve been thinking about these things for a long time, and I’m turning into a cantankerous old fart at this point, so here’s my critique of the memes that I’m seeing so far…

My Background

I’ve been a Unix power user since 1988 on the Eskimo North BBS running Xenix and teaching myself C. I’ve been a Unix SA since around 1996 (UW HITLab, UW MBT, various dotcoms). From 2001-2006 I worked at [A large Internet Retailer] on the CFengine infrastructure there. I CFengine-ized the 400 servers we mainted in 2001 on the front end of the website. By 2006 I was pushing out CFengine code to 30,000 servers, and was by far the most prolific configuration management engineer at [Internet Retailer] (80 commits to what was the ‘base role’ in that system vs. the next runner up having 5). Since then I’ve worked at Real Networks (ask me about that after some drinks) and now the Rhapsody spin-off managing 2,000+ servers.

Issue #1: the name “Operations”

I find myself having radically more in common with software developers than with operations people. I actually have a substantially worse time dealing with bureaucratic process than most software developers do. The job that I do requires pushing out to every server in the organization. The common notification and approval processes in Enterprise organizations tend to focus on operations fixing individual issues on individual servers, or code being deployed to a given set of servers with a very finite list of users that need to be notified of the changes. The case of having to make changes on every server, potentially affecting every user is not a ‘use case’ considered by the average policy.

When it does get handled, often it gets handled as a months-long operational fire-drill to get something simple done. A case in point would be the 2007 Daylight Savings Time issue which at the organization I was at required all-hands-on-deck for 2 months, nearly killed one SA, and burned up massive amounts of Program Management time doing the notification and approvals to reboot every service in the company. That kind of process is unsustainable when your basic job is to repeatedly do changes potentially affecting every service in the company. If every change like that is going to take 2 months to do, that means that I would only be able to do 6 things a year.

Obviously there is a clear need for agile methodologies here. But I find myself more and more being part of “Operational” departments and yet being substantially more at odd with the “Operational” mindset that most any software development team. I have quit jobs because it was clear to me that I would not get a change control policy that were sufficient agile that I could do my job and have adequate air cover to deal with inevitable mistakes. Through working on massive environments and doing repeated global pushes of config changes, I’m very good at mitigating and assessing risk, but I’m not perfect. I could do the math and see that inevitably I’d make a mistake trying to do my job, and simply get fired, while no manager was interested in trying to solve my unique change management issues. Every time I bring up the problem, some massively bureaucratic and heavy process designed to cover my manager’s butt was the response, which would not allow me to do my job.

I believe so strongly that what I primarily do is not “operations” that I got into an argument with my present manager that our group should be named “infrastructure” but was overridden and wound up in yet another “operations” group.

I’d like to see more discussion of “Agile Infrastructure”, “Infrastructure Engineering”, “Infrastructure Architecture”, etc and less of an assumption that just because I know how to setup a DNS server that I’m “Operational”. For awhile on Orkut (blast from the past, anyone?) there was an “Infrastructure Architects” group, which didn’t get a whole lot of traffic, but that’s what I’ve thought of as my job, even as I’ve suffered under being “Operational”.

Issue #2: The Whole Organization Needs to be Operational

In web-centric service-oriented-architecture agile shops (buzzword bingo!) the domain experts of the software are the software developers. I’ve watched “Operational” managers attempt and fail to execute on the strict playbook of having software developers write code and operations run the code. This has extended to the ludicrous extreme of having a requirement for massively scalable real-time logfile publishing so that software developers could still debug production issues while having their accounts removed from the app servers that their code ran on. I’ve also seen policy decisions made that “operations runs Java” which leads to fairly ludicrous behavior where SAs that couldn’t explain the first thing about how Java garbage collection runs are in change of all the GC options to Java (because they are defined as the experts, without actually being the experts).

Software developers in this world are the domain experts. They don’t write shrink-wrapped software and they need to develop expertise in tuning their apps, debugging their apps and they need access to their software and arguably even need access to bounce servers to debug simple production issues. The circus that ensues when software development orgs are not trusted to ‘touch’ production and they have to use the ‘remote hands’ of operations, when the SAs involved aren’t qualified to understand what the software team is telling them to do is amusing at first and then quickly gets tiresome.

SAs are not genetically born understanding “operations” and here’s nothing preventing software teams from developing good operational practices and being held accountable by their management chain. Having software teams only responsible for writing software, while “operations” is entirely the responsibility of the operations teams doesn’t work well when you need to get operational work out of the software teams. I would argue that placing operational responsibility for the software back onto the software teams will produce better uptime and operations and better code.

The point of all this, tough, is to attempt to break down the dichotomy between operations people vs. everything else. We need Operations in Infrastructure teams, in Security teams, and in Development teams. It may be that you have dedicated people to Operations in those teams, but Operations-as-a-discipline needs to be smeared out across the Enterprise and needs to not be just the sole responsibility of one department.

This probably aligns strongly with DevOps, but I haven’t seen anyone get hardcore enough about this problem. To paraphrase Slim Shady: “Guess there’s a little Operations in all of us. Fuck it, lets all stand up.”

Issue #3: Operations is not Already Agile

This isn’t so much a criticism as it is a rallying cry. There’s an idea that operations is already inherently more Agile than development teams already because they’ve got ticketing systems and they work in a bit of an ADHD manner on simple problems and knock them out all the time. I’d like to pop that bubble and point out really how truly horrible operations is at doing agile methodologies.

First of all, I repeatedly see scrum used by PMs in operations as an excuse to have a daily meeting to go over action items every day and report progress back to the PMs. If you go through any Agile documentation on how not do do scrum, that’s the only way that I’ve ever seen it done in Operations. Somewhere there’s a huge disconnect between software development managers and PMs and ‘Operations’ managers and PMs.

I also repeatedly see project work in ‘Operations’ being done waterfall and that being presented as a Best Practice. I tend to believe that there’s a psychological reason behind this in that because operations tends to be highly ADHD and interrupt driven all the time, and that tends to produce poorly designed and thought out work, that operational managers are natural driven towards wanting the complete other end of the pendulum — a massive process where everything is designed and documented ahead of time, with an absolutely unassailable list of requirements and the perfect implementation that will never go out of date, and we’ll never wind up looking back at it 5 years from now wanting to scrap the whole thing. Focusing on just a few different major business objectives, getting the big issues right from the start, and then scrumming in tight repetitive cycles is something that feels like its just more of the poor practices that operations does all the time — the virtues of dealing with uncertainty and complexity in an Agile methodology are completely missed. I would kill to work for a manager that just understood YAGNI as a means of reducing complexity.

Operational people are also plagued by the ‘bikeshed’ problem. The IT infrastructure needed to build a company is very complicated. I’ve found that in large Enterprise institutions that it is very, very difficult to meet even the simplest of requirements, so I aggressively whittle down what want to accomplish to what I can accomplish. This is almost universally met in meetings by people who want to be intelligent and develop lots of smart requirements about what the perfect solution would be in an ideal world with an infinite amount of time and resources to throw at the problem, or what could be accomplished in a smaller environment. I’m not dumb, and you’re not proving how smart you are to me by making the problem as complicated as possible and solving all the problems of the world. If I’m trying to accomplish something very stupid and simple it is because getting that simple problem solved across thousands of servers and hundreds of different types of apps is actually very hard.

A case in point would be centralized logfile collection. After probably a decade of having no universally centralized logfile processing in one particular company (but lots of one-offs scattered throughout the Enterprise), I simply wanted to setup a syslog-ng server and point all system syslog traffic from all servers at it to get centralized logs for security. Throw away all the requirements about application logs, centralized logs, GUIs for novice users, AAA to protect certain logs, etc. I just wanted security-related logs to be centralized and to be able to have clueful SAs use grep on the logs. Eminently achievable and basically out of the playbook of System Admin 101, but insufficiently clever and lazy and not taking the requirements of the rest of the business into consideration.

Anyway, the problem is bad. Very bad. This is not going to be an easy fight. Far from finding a fertile field of people who are used to doing quick iterative processes, Operations is dominantly made up of people who really believe they’d be doing a better job if they were in charge of really massive projects, with huge requirements and piles of resources and planning that would make any nuclear plant manager proud.

Issue #4: Is ITIL actually the Enemy?

Honestly, my knowledge of ITIL is largely limited to interacting with people who are ITIL-certified (certifiable?) and that leaves me not being very interested in learning much more about it. My understanding is that ITIL codifies the wall between development and operations and strictly defines them as two entirely different things.

I know that I consistently see this diagram of ITIL-compliant escalation which involves issues going through tier 1, then often a tier 2, then kicked up to systems/network/database administrators at tier 3, then software developers are tier 4. That just annoys me because I think the responsibility should be up to 24/7 staffed tier 1+2 to properly determine how to route the problem and should route *most* problems to software operations in tier 3, bypassing systems and networking folks entirely. The bulk of issues in a well-designed systems/network infrastructure lie either with application code or performance issues with SQL databases, and I don’t need to get woken up at 3AM in the morning to rule out systems and networks in order to kick it over to software. I’m also not interested in being woken up at 3AM in the morning to kick over services repeatedly because of a software bug which isn’t a high priority of the software development team, because features are more important. I, personally, am a bit of an outright primadonna because I’m usually the tier 5 person who can deal with having multiple app departments, along with systems, networking and databases all doing a circular firing squad blaming each other for an outage and come into that kind of a problem and correctly determine the root cause and get the appropriate department working on fixing whatever is broken. Still, I don’t see why systems engineers wind up being the goto-people at 3AM in the morning to figure out why its all broken — we do have lives, many of us have kids, and it gets really old really fast. Operational SAs really need an actual Union and we need to go out on strike over how we’re expected to do oncall — the way that the 24/7 workplace has crept into all of our lives and just become a natural part of it that we have to do or else we’re not doing it right, is utterly appalling from a labor-relations perspective.

And if the other teams don’t have access to figure out issues they need it. Network engineers need to be able to login to servers and tcpdump and traceroute. Software devs need access to each others servers and logs, not just their own servers and logs. Separation of debugging responsibility just leads to the circular firing squad. One reason why I can be the Tier 5 goto is because I have root on all the servers, and often I’ve managed to get myself enable on the switches, routers and load balancers (and can we get a job title for this kind of person? I’m tired of having my networking access yanked because I’m not a network engineer — I’ve never, ever done something nasty with enable access, but sometimes it is highly useful and prevents me from wasting NEs time on tickets that I cut just to ask stupid questions that I could login and determine myself — I’ll get CCNA/CCDP certified if i have to).

Anyway, my understanding of ITIL is that its rigid and inflexible in its definition of roles and responsibilities around operations vs. development and systems vs. network vs. software vs. databases, etc. It seems to me to be the opposite of Agile. If someone who really understands both ITIL and Agile can explain to me how ITIL can be Agile, by all means I’ll listen…

On the other hand…

Way to go. I’ve obviously been thinking about the problems for a long time and just getting old and grouchy about it. Hopefully DevOps or some intellectual descendant can catch on and start getting me excited… Its about time someone got a meme started on what I’m really trying to do with my job…

I/O is a pretty large deal with virtualized hosts. If you have 10:1 or 35:1 VM compression (guest-to-physical ratio) then any useless I/O which your servers are doing is going to cost you big in the overall I/O load on your physical servers.

Many O/S images, however, have a lot of useless I/O which goes on, which can be a factor in scaling VMs and pushing towards expensive SAN/NAS centralized storage solutions to deal just with all the useless I/O. I’ve been successful in identifying some fairly severe I/O issues from default RedHat installs which allow me to run a lot of services off of internal storage and don’t require centralized storage.

Useless I/O Sucking Cronjobs

On RHEL5 there is a pile of cronjobs that run periodically and like to suck up all the I/O on a server. These cronjobs do try to do something useful, but the I/O load is counterproductive. A classical example is the slocate cronjob that runs every night. This cronjob indexes your entire disk so walks the entire inode tree every night. If you have 35 virtual images on the same physical server, this will crush your I/O every single night. While having the ability to easily locate files from the commandline sounds useful, the nightly cost of generating this index is too expensive and this cronjob must be turned off for servers — more so than ever in the virtualized-era of computing.

There are also cronjobs that generate the indexes for man pages that run every night. While this produces less I/O than walking the entire inode tree, it’ll still produce synchronized I/O every night as all your VMs spin up at the same time to perform this task. What you lose is the ability to do ‘man -k’ or ‘apropos’ on your servers (since every image should be the same you can have a bastion host with this cronjob enabled so you can look things up when you need to).

Here is chef recipe for RH7.3 through RHEL5 for reducing cronjob I/O load on servers:

#
# remove default cronjobs that suck up I/O
#

file "/etc/cron.daily/mlocate.cron" do
        action :delete
end

file "/etc/cron.daily/slocate.cron" do
        action :delete
end

file "/etc/cron.daily/makewhatis.cron" do
        action :delete
end

file "/etc/cron.weekly/makewhatis.cron" do
        action :delete
end

file "/etc/cron.daily/00-makewhatis.cron" do
        action :delete
end

file "/etc/cron.weekly/00-makewhatis.cron" do
        action :delete
end

file "/etc/cron.daily/00-logwatch" do
        action :delete
end

file "/etc/cron.daily/0logwatch" do
        action :delete
end

Turning off fsync on syslog

The fsync() or fdatasync() system calls force writes to be sent to the disk, which is something that databases need to do for ACID-compliant reasons and which reduces throughput. For some reason the default install of /etc/syslog.conf in RedHat has syslogd treat the syslog files as databases and calls fsync() after every syslog line is written to the disk. If your servers do any syslog at all, or you enable debug logging for some syslog functionality this can crush your I/O if multiple virts are syslog’ing at the same time (particularly out of cron).

Another admin at work tipped me off by asking me about this backtrace showing syslog stuck in fsync() [also note that the hung_task messages from newer kernels can be quite useful]:

echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslogd       D 0007f2bb68a22287     0 28885      1         28888 28872 (NOTLB)
 ffff8801e3e7fd88  0000000000000282  0000000000000000  0000000000000001
 000000000000000a  ffff8801d8f4a080  ffff880000d32080  000000000000d03d
 ffff8801d8f4a268  0000000000000000

Call Trace:
 [] :jbd:log_wait_commit+0xa3/0xf5
 [] autoremove_wake_function+0x0/0x2e
 [] :jbd:journal_stop+0x1cf/0x1ff
 [] __writeback_single_inode+0x1e9/0x328
 [] do_readv_writev+0x26e/0x291
 [] sync_inode+0x24/0x33
 [] :ext3:ext3_sync_file+0xc9/0xdc
 [] do_fsync+0x52/0xa4
 [] __do_fsync+0x23/0x36
 [] tracesys+0xab/0xb6

Syslog, in my opinion, is no where near important enough to be calling fsync() on everything it does, so I hit the man pages to determine how to disable this:

You may prefix each entry with the minus ”-” sign to omit syncing the file after every logging. Note that you might
lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some perfor-
mance, especially if you run programs that use logging in a very verbose manner.

And all I did was to add the ‘-’ sign to all the logfiles that syslog.conf is configured to write to:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                -/var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              -/var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  -/var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          -/var/log/spooler

# Save boot messages also to boot.log
local7.*                                                -/var/log/boot.log

*.*                                                     @SYSLOG-SERVER.example.com

Note that every file is preceeded by a ‘-’ which is what does the magic.

Summary

If you are using Ubuntu, I expect that there’s a pile of useless cronjobs that you can also track down, but I haven’t spent the time to do so (use RHEL rather than Ubuntu at work). I’d also be a little surprised if Ubuntu did not have the same behavior with syslog, and I expect the syntax is similar (although its always possible that Ubuntu got this behavior correct right out of the gate…)

The JAVA httpclient package is used by many software devs in SOA architecture shops to make back end connections from service to service. The apache developers who wrote the httpclient library clearly were considering the use of the httpclient library to make web browsers. As such they included a parameter, MaxConnectionsPerHost which would limit the number of simultaneous requests a browser could make to a website to 2, in order to avoid overloading the site. This made more sense back in the 90s when RFC 2068 was written with this recommendation, and firefox now has upped the default limit to 15, and I believe that IE has raised the default limit to 8.

My contention is that in SOA shops where servers are calling services that this connection limit is useless and should be disabled or raised to a very, very large value (10,000+).

The Problem

If you have a bank of servers calling a bank of other servers, in an SOA environment, you can wind up hitting an artificial limit caused by this httpclient limit which is nearly identical in behavior to exhaustion of database connection pools. It should be noted that database connection pooling, however, is necessary in order to reduce the cost of expensive database connection establishment and to reduce the memory impact on the database server of large amounts of database connections (particularly with Oracle, less so with a thinner database like MySQL).

What this appears like is that you have a bank of java (tomcat or whatever) servers, which are idle but periodically spiking to insane latency and timeouts. There’s no maxing out of CPU, I/O, network bandwidth or any other server resources. Similarly, the back end service that these servers are trying to contact is also scaled out adequately for the load and there’s no obvious performance issues that it is hitting, but response times in getting back a reply from the service is very, very slow as measured by the java making the httpclient call. This problem can masquerade as networking or load balancer issues and can drive network engineers nuts trying to track down why “the network is slow”.

What is observed, however, on the java app in thread dumps is potentially hundreds of threads stuck in doGetConnection:

"XXX THREAD NAME CHANGED TO PROTECT THE GUILTY XXX" daemon prio=10 tid=0x0000002ddb745800 nid=0x5edd in Object.wait() [0x000000005587f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:509)
        - locked <0x0000002aa37a9208> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:394)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:152)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
[...etc...]

So, we have idle service on one end, idle service on the other end, everything is working but the service is effectively crashed because of this artificial limit. This happens because in an SOA environment, having a limit of 2 simultaneous connections from server to service VIP is way, way too low. In a bank of 8 servers that means you can only have 16 simultaneous connections and if you’re doing in aggregate 100 tps it only takes a few slow, expensive calls to the service to “clog up the pipes”.

Why raising the limit to 10 or 20 or 30 is bad

The obviously prudent thing to do is to raise this limit up to some limit which is reasonable but still “protects” the back end service. We wouldn’t want to disable the limit entirely because… well, something bad would clearly happen and someone put that limit in there for a reason.

I’m going to try to argue that the reason that limit was put in there had nothing to do with your SOA environment, and that I’ve worked in truly massive non-java SOA environments that didn’t have this kind of limit and never saw an issue, and that there’s much better ways to deal with scaling limits and brown-outs and that this limit is always the behavior you don’t want.

I just diagnosed this problem, again, for the hundredth time in a situation where the MaxConnectionsPerHost limit had been raised to 10 on a bank of 8 servers. This had been running fine for a long time, but 5 of the servers crashed at once due to a memory exhaustion issue. That was bad, but the set of servers is so overscaled that there was still 60% idle cpu cycles available on the clients. The problem was that the farm went from having a limit of 80 simultaneous connections down to having a limit of 30 simultaneous connections. That was the only thing that caused the entire farm to fail (due to timeouts).

Granted, having 5 of 8 servers out of rotation is a bad thing, but the farm actually could have taken the load and this would have been an “oops, we had 5 servers out, damn we’re overscaled, good no customers were impacted” problem, but the “prudent” limit of 10 resulted in an outage. I’d rather jack the limit up to something very, very large and make this problem simply go away and stop encountering it. It didn’t do any good, and just caused us another outage.

Effect of Removing the Limit Completely

I worked in a very large Seattle-based Internet Retailor for 5 years as one of the “Tier 3 or 4″ Senior SEs who would see any kind of crazy infrastructure problem like this bubble up to us. We were not java-based at the time and were instead using process-based clients that simply had no concept of this kind of connection pooling to back end SOA services. Any server could open up as many connections to a back end service as it liked, each process could open up as many back end connection as it liked, and the processes did not share any state to know in aggregate how many connections were open to any back end server. With 30,000 servers and thousands of different deployed applications (literally) we never encountered any issues that the maxrequestsperhost limit would have solved. In my opinion, in an SOA shop this is a solution which is looking for a problem. I lived for 5 years in a massive environment and never once saw some kind of issue which made me think to utilize something like this limit.

And I would argue that this is because HTTP connections are *massively* cheap compared to Oracle connections. Sure you need to use a little bit of TCP/IP to get it going, but modern processors can do many more of those connection opens per second than your servers are ever going to want to be submitting (100 tps coming from a typical java tomcat is going to be impressive programming, but the TCP/IP stack won’t break a sweat). Trying to do some kind of HTTP/1.1-based connection pooling with a finite limit on it (which in my experience is *not* what is typically going on when I see the httpclient bug — most of the time these connections are not being reused at all) is a premature optimization in the Knuth-sense.

Poor Behavior on Surge Traffic

A common thought is that this “protects” your back-end services from surges. But the infinite-queuing behavior of the httpclient is precisely what you don’t want. As soon as the client overall starts to require more simultaneous connections than it can submit to the back end service the queue will attempt to grow infinitely long, creating infinite latency. What effectively happens is that every single request will take as long as the timeout period of whatever meta-client is calling the java service that uses httpclient.

In brown-out loading what you want is to start aggressively dropping connections to take load off, but you want to do that based on real brown-out of your back end service. SDEs are terrible at estimating what level of simultaneous connections would actually result in a real brown-out of the back end service. Nobody measures this in Q/A or load, or looks at it in production, and it would probably take a team of people in a large site to keep measuring and tweaking all the clients in production. The only way to reliable tell you really are in a brown-out situation is by wrapping your back-end calls with timeouts — and not queueing.

Retries are also poor behavior as well, unless you have exponential backoff like TCP/IP uses — otherwise you ensure that a momentary brown-out produces an permanent overload of the back-end service.

You could still use a simultaneous connection limit if you must, but you must not queue, you must drop. If you queue, in an overload you will build latency without limit once the pipes are filled up, causing every request to timeout which results in a 100% outage anyway. If you immediately drop requests over the limit then there is the possibility that you could hit a situation where dropping 10% of the requests allows 90% of the other requests to succeed in a timely manner. However, again, this requires being able to accurately measure exactly what the simultaneous connection threshold should be. Set it too low and you start to deny requests before you have overloaded your backend service. Set it too high and you overload your back-end service anyway — you will never manage to set it correctly and budget adequate time to maintain it as the software changes, so its effectively useless go down that road. Anyway, the httpclient blocks requests in doGetConnection when all the connections are being used and does not drop them, so the httpclient does not implement this kind of behavior.

MaxConnectionsPerHost recommendation

Find some way to disable it, or set it to something “insane” like 10,000. All it does, in an SOA environment, is cause problems without usefully solving any problem.

You will then never again see the problem where an idle java app is having problems talking to an idle back end service when everything in the network checks out fine — an outage due simply to configuration.

You can then focus on scalability and brownouts using timeouts, exponential backoff, or simply elastic or “cloudy” scaling of services in response to demand.

And in general, you should not try to “protect” back end services with any kind of artificial limits. Invariably this result in exercising the Law of Unintended Consequences when those limits are accidentally hit when all the services are fine. It will not ‘save’ you but will cause you problems. Of course, when you encounter a problem like Oracle’s expensive connections you need to manage that resource and establish limits, but still those limits should be pushed to the point where you are using the limits to protect Oracle’s memory and not in an attempt to prevent apps from delivery ‘too many’ requests to Oracle. If your database melts down under load you need to address the problem well upstream with whatever is misbehaving — or just buy a bigger database machine, or use some more horizontally scalable NoSQL solution.

Summary

The MaxConnectionsPerHost parameter is a throwback to RFC 2068, written in 1997, before anyone dreamed up the term “Service Oriented Architecture”. In an SOA shop, this behavior is worse than useless and wastes time and causes needless outages without solving any problems.

The apache httpclient authors probably will not by default remove this limit, since they have to consider that their code could also be used to write web browsers, where some kind of finite limit is certainly prudent. But in any SOA shop this limit should be disabled or raised to something like 10,000 in all service-calling code — effectively making it unlimited and removing it.

First, you can signup for the free hosting service for 5 servers, which eliminates the need to screw around setting up the server infrastructure:

https://cookbooks.opscode.com/users/new

Second, go through the five step tutorial here:

http://help.opscode.com/faqs/start/how-to-get-started

Configuring ~/.chef

The opscode instructions have you create ~/chef-repo/.chef and put your knife.rb and .pem file in there.

I prefer to move those files into ~/.chef so that I don’t have to cd ~/chef-repo in order to use knife. You can also push out thse config files in ~/.chef just like ~/.bashrc files so that you can use knife on different nodes. Of course a hacker who gains root on one of the servers where your knife credentials are can impersonate you to the chef server, so it may be wise to restrict your knife config to a set of bastion servers. Obviously, chef can help you manage this config.

My ~/.chef/knife.rb looks like:

log_level                :info
log_location             STDOUT
node_name                "USERNAME"
client_key               "#{ENV['HOME']}/.chef/USERNAME.pem"
validation_client_name   "ORGANIZATION-validator"
validation_key           "#{ENV['HOME']}/.chef/ORGANIZATION-validator.pem"
chef_server_url          "https://api.opscode.com/organizations/ORGANIZATION"
cache_type               'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{ENV['HOME']}/chef-repo/site-cookbooks"]

Here, USERNAME and ORGANIZATION should be replaced by your chef username and your organization’s name. You will also need to put your credentials in ~/.chef/USERNAME.pem (replace USERNAME by your username).

You have now configured knife to know who you are, where your credentials are, and configured some paths in your environment.

You should now be able to run comands like knife node list in any directory on the host that you have configured.

I’ve also setup my cookbook_path to ~/chef-repo/site-cookbooks so that from any directory, knife can find my cookbooks in order to upload them — you should modify that to wherever you have your site’s cookbooks.

Adding New Nodes

Now, when you’re done with that, to add a new node, you can deploy these two files to the new node:

/etc/chef/client.rb
- this points chef at the server
/etc/chef/validation.pem
- your organization key to authenticate first time to the server

Then become root and run your chef-client to add the node to the server and create the client.pem

Now you can run knife node list to see the newly added node.

You should protect /etc/chef by making it mode 0700 or something similar. Once the /etc/chef/client.pem file has been bootstrapped down from the server, you (or a chef recipe) can delete the /etc/chef/validation.pem file.

Summary

So, now you should have setup an account at opscode, successfully setup your organization, configured knife to be easy to use on your host, and bootstrapped a few hosts to use chef. Now you can move on to getting chef to do useful things.