Spherical Chicken

Amazon claims that their EC2 micro instance provides a “small amount of consistent CPU resources”:

Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available.

Well, one of my micro instances has looked like this with 98-100% steal cycles for hours:

08:59:32          CPU     %user     %nice   %system   %iowait    %steal     %idle
09:00:01          all      5.51      0.00      0.00      0.00     94.49      0.00
09:01:09          all      1.04      0.00      0.01      0.00     98.85      0.09
09:02:02          all      3.41      0.00      0.00      0.00     96.59      0.00
09:03:01          all      1.09      0.00      0.02      0.00     98.84      0.05
09:04:02          all      1.74      0.00      0.00      0.00     98.24      0.02
09:05:02          all     10.15      0.00      1.46      1.74     69.08     17.56
09:06:03          all      4.66      0.00      0.03      0.31     94.75      0.25
09:07:05          all      1.46      0.00      0.00      0.00     98.54      0.00
09:08:02          all      2.98      0.00      0.00      0.00     97.00      0.02
09:09:04          all      3.40      0.00      0.02      0.00     96.28      0.30
09:10:04          all      1.59      0.00      0.00      0.02     98.40      0.00
09:11:15          all     10.16      0.00      0.79      0.83     80.32      7.91
09:12:02          all      2.06      0.00      0.02      0.00     97.88      0.04
09:13:03          all      4.32      0.00      0.00      0.00     95.61      0.07
09:14:01          all      0.00      0.00      0.00      0.00    100.00      0.00
09:15:01          all      3.99      0.00      0.00      0.00     95.99      0.02
09:16:01          all      2.35      0.00      0.00      0.48     94.37      2.80
09:17:01          all     16.01      0.00      0.81      0.61     76.10      6.46
09:18:01          all      0.00      0.00      0.00      0.00    100.00      0.00
09:19:02          all      0.79      0.00      0.00      0.00     99.19      0.02
09:20:02          all      4.56      0.00      0.02      0.00     95.43      0.00
09:21:03          all      0.00      0.00      0.00      0.00    100.00      0.00
09:22:15          all     10.84      0.00      0.76      0.61     83.74      4.04
09:23:01          all      3.70      0.00      0.00      0.00     96.27      0.02
09:24:05          all      5.87      0.00      0.00      0.00     94.08      0.05
09:25:02          all      0.00      0.00      0.00      0.00    100.00      0.00
09:26:01          all      2.02      0.00      0.00      0.03     97.95      0.00
09:27:01          all      0.00      0.00      0.00      0.00    100.00      0.00
09:28:02          all     11.65      0.00      1.09      1.06     77.05      9.16
09:29:09          all      3.24      0.00      0.00      0.00     94.92      1.84
09:30:13          all      2.13      0.00      0.00      0.00     97.87      0.00
09:31:01          all      1.99      0.00      0.00      0.00     97.98      0.02
09:32:02          all      3.43      0.00      0.02      0.00     96.56      0.00
09:33:01          all      0.22      0.00      0.05      0.25     96.12      3.36
09:34:02          all     14.96      0.00      1.16      1.29     70.74     11.84
09:35:01          all      0.95      0.00      0.00      0.00     99.05      0.00
09:36:17          all      5.62      0.00      0.03      0.00     94.36      0.00
09:37:02          all      0.00      0.00      0.00      0.00    100.00      0.00
09:38:02          all      1.84      0.00      0.00      0.00     98.13      0.03
09:39:01          all      1.92      0.00      0.27      0.71     87.51      9.59
09:40:14          all      8.11      0.00      0.43      0.35     87.92      3.19
09:41:01          all      2.46      0.00      0.02      0.00     97.50      0.02
09:42:02          all      2.22      0.00      0.00      0.00     97.78      0.00
09:43:02          all      2.00      0.00      0.00      0.00     98.00      0.00

Note that sar is taking up to 14 seconds at times (e.g. 09:40:14) in order to gather statistics, and it is quite light weight and doesn’t do much other than read a bit out of /proc.

This is so bad that the instance is having 600+ms ping times:

64 bytes from 50.18.x.y: icmp_seq=0 ttl=53 time=609.608 ms
64 bytes from 50.18.x.y: icmp_seq=1 ttl=53 time=1107.454 ms
64 bytes from 50.18.x.y: icmp_seq=2 ttl=53 time=107.230 ms
64 bytes from 50.18.x.y: icmp_seq=3 ttl=53 time=605.416 ms
64 bytes from 50.18.x.y: icmp_seq=4 ttl=53 time=1104.728 ms
64 bytes from 50.18.x.y: icmp_seq=5 ttl=53 time=104.510 ms
64 bytes from 50.18.x.y: icmp_seq=6 ttl=53 time=633.574 ms
64 bytes from 50.18.x.y: icmp_seq=7 ttl=53 time=1133.371 ms
64 bytes from 50.18.x.y: icmp_seq=8 ttl=53 time=133.149 ms
64 bytes from 50.18.x.y: icmp_seq=9 ttl=53 time=631.604 ms

That means that interrupt context and enough horsepower to respond to an ICMP ping is not able to run at all for 600+ ms.

And the problem is definitely on the EC2 side, I’m getting clean pings to the internet:

 1  a.b.c.d  102.019 ms  15.208 ms  13.735 ms
 2  69.17.83.233  11.109 ms  10.859 ms  10.363 ms
 3  209.247.91.169  10.861 ms  12.741 ms  12.855 ms
 4  4.68.105.30  11.484 ms  16.482 ms  17.860 ms
 5  4.69.132.49  27.975 ms  28.601 ms  30.103 ms
 6  4.69.153.18  30.226 ms  28.726 ms  28.478 ms
 7  4.69.152.16  28.352 ms  61.582 ms  31.225 ms
 8  4.53.208.22  31.100 ms  30.775 ms  30.788 ms
 9  72.21.222.208  32.222 ms  32.219 ms  30.851 ms
10  72.21.222.255  33.224 ms  31.350 ms  31.226 ms
11  * * *
12  * * *
13  * * *
14  50.18.x.y  393.064 ms  1500.394 ms  989.786 ms

This isn’t “small amounts of consistent CPU”, this is useless. They’re clearly prioritizing micro instances down to the point where if the server is otherwise being utilized 100% the micro instances get completely queue starved.

Its time. The idea that all your servers need to be by default ACL’d so that they can’t talk to the internet is an idea that is dying and needs to be put out of its misery.

Continue reading ‘The Heretical Sysadmin: Outbound ACLs need to die’ »

I posted some instructions here on using knife to bootstrap EC2 instances. I used Amazon’s AMIs thinking that they’d be more appropriate for EC2 because I’m an EC2 newbie.

FAIL.

The stated goal of Amazon’s AMI is to provide essentially a RHEL6 kernel on a RHEL5 ABI. However, if you really try to use RHEL5 RPMs you’ll run into things like curl being upgraded to:

curl-7.19.7-15.16.amzn1.x86_64

Which is basically the RHEL6 version:

curl-7.19.7-6.el6.x86_64.rpm

Not the RHEL5 version:

curl-7.15.5-9.el5.x86_64.rpm

That breaks all kinds of RHEL5 RPMs, while its true that the rest of the O/S isn’t RHEL6 enough to install RHEL6 RPMs successfully. The result is a painful inability to consistently install either RHEL5 or RHEL6 RPMs.

Its possible that the Amazon AMI might collect itself a following of people who build up public repos with all the RPMs that you can get from DAG, EPEL, ELFF, Jpackage, etc for the Amazon AMI. However, right now its basically its own beast — not RHEL5 or RHEL6 compatible — with no publicly-available prebuilt package repos for it. Since I have better things to do than recompile packages for it, its kinda useless to me, so I’d recommend the Rightscale CentOS 5.4 EBS-based images (to use on the t1.micro instance) instead.

Quick note on how to use chef 0.9.12′s knife command to provision EC2 instances from the command line using the Amazon CentOS-based AMI. I’m assuming you already have your ~/.chef/knife.rb setup to talk to either the chef platform or some chef servers that you’ve built and that you can run commands like ‘knife node list’ successfully and have your chef client_key setup. Setting up knife to talk to your chef server is outside the scope of this post.

NOTE: It turns out that the Amazon AMI looks like its based on RHEL6, with some upgraded RPMs. The centos5-based chef install seems to work fine though.

Continue reading ‘Bootstrapping Amazon’s CentOS-based AMI in EC2 using knife’ »

Turns out its fairly bad practice to take a XenServer instance and point your yum repos at a CentOS 5 yum repository.

It makes it easy to install some useful utility stuff like lsof and strace. However, if you mistakenly run ‘yum -y upgrade’ on a XenServer which is pointed at an enabled CentOS 5 repo, it will upgrade the gnupg RPM which nukes some special sauce that Citrix has put in there. This also updates your /etc/redhat-release file to suggest that your XenServer is now Citrix which can break cfengine, chef, etc which all look in that file to determine what flavor of redhat-ish server you are running and will now treat your box just like any other CentOS box.

One solution to this is simply to do something like on a sane XenServer:

tar -cvzf /tmp/gnupg.tar.gz `rpm -q -l gnupg`

Then copy that to a broken XenServer and something like:

cd / && tar -xvzpf /path/to/gnupg.tar.gz

That’ll screw up your RPM database a little bit, but it’ll repair the problem.

Continue reading ‘XenServer, yum and “Error processing your License File”’ »

One interesting thing to wrap my head around in the cfengine-vs-chef world is code being used all over the place as configuration. I ran into an issue where I needed to set the PATH environment that chef executes in order to fix ohai to be able to realiably find dmidecode in /usr/sbin (among other things). The way that I would normally expect to do this is to look into an configuration file in /etc and find some documentation that someone supported a “PATH” keyword or some kind of keyword in order to pass environment variables into the code in order to do what I want.

Well, I surfed around through the options available in the /etc/chef/client.rb file and wound up frustrated because nobody had spec’d out the obvious use case of needing to set environment variables. I considered submitting a bug for this. I wrote out the bug in my head, and it was going to suggest either a specific “path=” directive to set the path, or some more complicated directive, ideally, to support setting arbitrary environment variables.

Then I pondered what Adam had told me a couple days earlier to just drop code in there. In this case what we were discussing was setting the node_name off of `/bin/hostname`. I momentarily meditated on the fact that it really was a client.*RB* file and not client.*CONF*. Could it be this easy:

# cat /etc/chef/client.rb
log_level        :info
log_location     STDOUT
chef_server_url  'https://api.opscode.com/organizations/CENSORED'
validation_client_name 'CENSORED-validator'

Ohai::Config[:plugin_path] << '/etc/ohai/plugins'

ENV['PATH'] = "/usr/local/ruby-1.8.7/bin:/usr/local/perl-5.10.1/bin:/usr/local/sbin:/usr/local/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/sbin:/usr/bin:/sbin:/bin"

And, yes, apparently it just is that easy. Nobody at opscode had to think about supporting setting ENV variables in the conf file, because they just executed the conf file so I can just embed ruby in there and change the behavior in ways that the authors possibly never considered.

Code-as-configuration FTW.

DevOps background

DevOps is a developing meme which seems to be attacking the wall between ‘development’ and ‘operations’, incorporating agile methodologies in ‘operations’ and expressing ‘infrastructure as code’. This is a very good thing, and its about damn time. But I’ve been thinking about these things for a long time, and I’m turning into a cantankerous old fart at this point, so here’s my critique of the memes that I’m seeing so far…

Continue reading ‘Issues with DevOps’ »

The Problem

I/O is a pretty large deal with virtualized hosts. If you have 10:1 or 35:1 VM compression (guest-to-physical ratio) then any useless I/O which your servers are doing is going to cost you big in the overall I/O load on your physical servers.

Many O/S images, however, have a lot of useless I/O which goes on, which can be a factor in scaling VMs and pushing towards expensive SAN/NAS centralized storage solutions to deal just with all the useless I/O. I’ve been successful in identifying some fairly severe I/O issues from default RedHat installs which allow me to run a lot of services off of internal storage and don’t require centralized storage.

Useless I/O Sucking Cronjobs

On RHEL5 there is a pile of cronjobs that run periodically and like to suck up all the I/O on a server. These cronjobs do try to do something useful, but the I/O load is counterproductive. A classical example is the slocate cronjob that runs every night. This cronjob indexes your entire disk so walks the entire inode tree every night. If you have 35 virtual images on the same physical server, this will crush your I/O every single night. While having the ability to easily locate files from the commandline sounds useful, the nightly cost of generating this index is too expensive and this cronjob must be turned off for servers — more so than ever in the virtualized-era of computing.

There are also cronjobs that generate the indexes for man pages that run every night. While this produces less I/O than walking the entire inode tree, it’ll still produce synchronized I/O every night as all your VMs spin up at the same time to perform this task. What you lose is the ability to do ‘man -k’ or ‘apropos’ on your servers (since every image should be the same you can have a bastion host with this cronjob enabled so you can look things up when you need to).

Here is chef recipe for RH7.3 through RHEL5 for reducing cronjob I/O load on servers:

#
# remove default cronjobs that suck up I/O
#

file "/etc/cron.daily/mlocate.cron" do
        action :delete
end

file "/etc/cron.daily/slocate.cron" do
        action :delete
end

file "/etc/cron.daily/makewhatis.cron" do
        action :delete
end

file "/etc/cron.weekly/makewhatis.cron" do
        action :delete
end

file "/etc/cron.daily/00-makewhatis.cron" do
        action :delete
end

file "/etc/cron.weekly/00-makewhatis.cron" do
        action :delete
end

file "/etc/cron.daily/00-logwatch" do
        action :delete
end

file "/etc/cron.daily/0logwatch" do
        action :delete
end

Turning off fsync on syslog

The fsync() or fdatasync() system calls force writes to be sent to the disk, which is something that databases need to do for ACID-compliant reasons and which reduces throughput. For some reason the default install of /etc/syslog.conf in RedHat has syslogd treat the syslog files as databases and calls fsync() after every syslog line is written to the disk. If your servers do any syslog at all, or you enable debug logging for some syslog functionality this can crush your I/O if multiple virts are syslog’ing at the same time (particularly out of cron).

Another admin at work tipped me off by asking me about this backtrace showing syslog stuck in fsync() [also note that the hung_task messages from newer kernels can be quite useful]:

echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslogd       D 0007f2bb68a22287     0 28885      1         28888 28872 (NOTLB)
 ffff8801e3e7fd88  0000000000000282  0000000000000000  0000000000000001
 000000000000000a  ffff8801d8f4a080  ffff880000d32080  000000000000d03d
 ffff8801d8f4a268  0000000000000000

Call Trace:
 [] :jbd:log_wait_commit+0xa3/0xf5
 [] autoremove_wake_function+0x0/0x2e
 [] :jbd:journal_stop+0x1cf/0x1ff
 [] __writeback_single_inode+0x1e9/0x328
 [] do_readv_writev+0x26e/0x291
 [] sync_inode+0x24/0x33
 [] :ext3:ext3_sync_file+0xc9/0xdc
 [] do_fsync+0x52/0xa4
 [] __do_fsync+0x23/0x36
 [] tracesys+0xab/0xb6

Syslog, in my opinion, is no where near important enough to be calling fsync() on everything it does, so I hit the man pages to determine how to disable this:

You may prefix each entry with the minus ”-” sign to omit syncing the file after every logging. Note that you might
lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some perfor-
mance, especially if you run programs that use logging in a very verbose manner.

And all I did was to add the ‘-’ sign to all the logfiles that syslog.conf is configured to write to:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                -/var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              -/var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  -/var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          -/var/log/spooler

# Save boot messages also to boot.log
local7.*                                                -/var/log/boot.log

*.*                                                     @SYSLOG-SERVER.example.com

Note that every file is preceeded by a ‘-’ which is what does the magic.

Summary

If you are using Ubuntu, I expect that there’s a pile of useless cronjobs that you can also track down, but I haven’t spent the time to do so (use RHEL rather than Ubuntu at work). I’d also be a little surprised if Ubuntu did not have the same behavior with syslog, and I expect the syntax is similar (although its always possible that Ubuntu got this behavior correct right out of the gate…)

Background

The JAVA httpclient package is used by many software devs in SOA architecture shops to make back end connections from service to service. The apache developers who wrote the httpclient library clearly were considering the use of the httpclient library to make web browsers. As such they included a parameter, MaxConnectionsPerHost which would limit the number of simultaneous requests a browser could make to a website to 2, in order to avoid overloading the site. This made more sense back in the 90s when RFC 2068 was written with this recommendation, and firefox now has upped the default limit to 15, and I believe that IE has raised the default limit to 8.

My contention is that in SOA shops where servers are calling services that this connection limit is useless and should be disabled or raised to a very, very large value (10,000+).

The Problem

If you have a bank of servers calling a bank of other servers, in an SOA environment, you can wind up hitting an artificial limit caused by this httpclient limit which is nearly identical in behavior to exhaustion of database connection pools. It should be noted that database connection pooling, however, is necessary in order to reduce the cost of expensive database connection establishment and to reduce the memory impact on the database server of large amounts of database connections (particularly with Oracle, less so with a thinner database like MySQL).

What this appears like is that you have a bank of java (tomcat or whatever) servers, which are idle but periodically spiking to insane latency and timeouts. There’s no maxing out of CPU, I/O, network bandwidth or any other server resources. Similarly, the back end service that these servers are trying to contact is also scaled out adequately for the load and there’s no obvious performance issues that it is hitting, but response times in getting back a reply from the service is very, very slow as measured by the java making the httpclient call. This problem can masquerade as networking or load balancer issues and can drive network engineers nuts trying to track down why “the network is slow”.

What is observed, however, on the java app in thread dumps is potentially hundreds of threads stuck in doGetConnection:

"XXX THREAD NAME CHANGED TO PROTECT THE GUILTY XXX" daemon prio=10 tid=0x0000002ddb745800 nid=0x5edd in Object.wait() [0x000000005587f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:509)
        - locked <0x0000002aa37a9208> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:394)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:152)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
[...etc...]

So, we have idle service on one end, idle service on the other end, everything is working but the service is effectively crashed because of this artificial limit. This happens because in an SOA environment, having a limit of 2 simultaneous connections from server to service VIP is way, way too low. In a bank of 8 servers that means you can only have 16 simultaneous connections and if you’re doing in aggregate 100 tps it only takes a few slow, expensive calls to the service to “clog up the pipes”.

Why raising the limit to 10 or 20 or 30 is bad

The obviously prudent thing to do is to raise this limit up to some limit which is reasonable but still “protects” the back end service. We wouldn’t want to disable the limit entirely because… well, something bad would clearly happen and someone put that limit in there for a reason.

I’m going to try to argue that the reason that limit was put in there had nothing to do with your SOA environment, and that I’ve worked in truly massive non-java SOA environments that didn’t have this kind of limit and never saw an issue, and that there’s much better ways to deal with scaling limits and brown-outs and that this limit is always the behavior you don’t want.

I just diagnosed this problem, again, for the hundredth time in a situation where the MaxConnectionsPerHost limit had been raised to 10 on a bank of 8 servers. This had been running fine for a long time, but 5 of the servers crashed at once due to a memory exhaustion issue. That was bad, but the set of servers is so overscaled that there was still 60% idle cpu cycles available on the clients. The problem was that the farm went from having a limit of 80 simultaneous connections down to having a limit of 30 simultaneous connections. That was the only thing that caused the entire farm to fail (due to timeouts).

Granted, having 5 of 8 servers out of rotation is a bad thing, but the farm actually could have taken the load and this would have been an “oops, we had 5 servers out, damn we’re overscaled, good no customers were impacted” problem, but the “prudent” limit of 10 resulted in an outage. I’d rather jack the limit up to something very, very large and make this problem simply go away and stop encountering it. It didn’t do any good, and just caused us another outage.

Effect of Removing the Limit Completely

I worked in a very large Seattle-based Internet Retailor for 5 years as one of the “Tier 3 or 4″ Senior SEs who would see any kind of crazy infrastructure problem like this bubble up to us. We were not java-based at the time and were instead using process-based clients that simply had no concept of this kind of connection pooling to back end SOA services. Any server could open up as many connections to a back end service as it liked, each process could open up as many back end connection as it liked, and the processes did not share any state to know in aggregate how many connections were open to any back end server. With 30,000 servers and thousands of different deployed applications (literally) we never encountered any issues that the maxrequestsperhost limit would have solved. In my opinion, in an SOA shop this is a solution which is looking for a problem. I lived for 5 years in a massive environment and never once saw some kind of issue which made me think to utilize something like this limit.

And I would argue that this is because HTTP connections are *massively* cheap compared to Oracle connections. Sure you need to use a little bit of TCP/IP to get it going, but modern processors can do many more of those connection opens per second than your servers are ever going to want to be submitting (100 tps coming from a typical java tomcat is going to be impressive programming, but the TCP/IP stack won’t break a sweat). Trying to do some kind of HTTP/1.1-based connection pooling with a finite limit on it (which in my experience is *not* what is typically going on when I see the httpclient bug — most of the time these connections are not being reused at all) is a premature optimization in the Knuth-sense.

Poor Behavior on Surge Traffic

A common thought is that this “protects” your back-end services from surges. But the infinite-queuing behavior of the httpclient is precisely what you don’t want. As soon as the client overall starts to require more simultaneous connections than it can submit to the back end service the queue will attempt to grow infinitely long, creating infinite latency. What effectively happens is that every single request will take as long as the timeout period of whatever meta-client is calling the java service that uses httpclient.

In brown-out loading what you want is to start aggressively dropping connections to take load off, but you want to do that based on real brown-out of your back end service. SDEs are terrible at estimating what level of simultaneous connections would actually result in a real brown-out of the back end service. Nobody measures this in Q/A or load, or looks at it in production, and it would probably take a team of people in a large site to keep measuring and tweaking all the clients in production. The only way to reliable tell you really are in a brown-out situation is by wrapping your back-end calls with timeouts — and not queueing.

Retries are also poor behavior as well, unless you have exponential backoff like TCP/IP uses — otherwise you ensure that a momentary brown-out produces an permanent overload of the back-end service.

You could still use a simultaneous connection limit if you must, but you must not queue, you must drop. If you queue, in an overload you will build latency without limit once the pipes are filled up, causing every request to timeout which results in a 100% outage anyway. If you immediately drop requests over the limit then there is the possibility that you could hit a situation where dropping 10% of the requests allows 90% of the other requests to succeed in a timely manner. However, again, this requires being able to accurately measure exactly what the simultaneous connection threshold should be. Set it too low and you start to deny requests before you have overloaded your backend service. Set it too high and you overload your back-end service anyway — you will never manage to set it correctly and budget adequate time to maintain it as the software changes, so its effectively useless go down that road. Anyway, the httpclient blocks requests in doGetConnection when all the connections are being used and does not drop them, so the httpclient does not implement this kind of behavior.

MaxConnectionsPerHost recommendation

Find some way to disable it, or set it to something “insane” like 10,000. All it does, in an SOA environment, is cause problems without usefully solving any problem.

You will then never again see the problem where an idle java app is having problems talking to an idle back end service when everything in the network checks out fine — an outage due simply to configuration.

You can then focus on scalability and brownouts using timeouts, exponential backoff, or simply elastic or “cloudy” scaling of services in response to demand.

And in general, you should not try to “protect” back end services with any kind of artificial limits. Invariably this result in exercising the Law of Unintended Consequences when those limits are accidentally hit when all the services are fine. It will not ‘save’ you but will cause you problems. Of course, when you encounter a problem like Oracle’s expensive connections you need to manage that resource and establish limits, but still those limits should be pushed to the point where you are using the limits to protect Oracle’s memory and not in an attempt to prevent apps from delivery ‘too many’ requests to Oracle. If your database melts down under load you need to address the problem well upstream with whatever is misbehaving — or just buy a bigger database machine, or use some more horizontally scalable NoSQL solution.

Summary

The MaxConnectionsPerHost parameter is a throwback to RFC 2068, written in 1997, before anyone dreamed up the term “Service Oriented Architecture”. In an SOA shop, this behavior is worse than useless and wastes time and causes needless outages without solving any problems.

The apache httpclient authors probably will not by default remove this limit, since they have to consider that their code could also be used to write web browsers, where some kind of finite limit is certainly prudent. But in any SOA shop this limit should be disabled or raised to something like 10,000 in all service-calling code — effectively making it unlimited and removing it.

Setup

First, you can signup for the free hosting service for 5 servers, which eliminates the need to screw around setting up the server infrastructure:

https://cookbooks.opscode.com/users/new

Second, go through the five step tutorial here:

http://help.opscode.com/faqs/start/how-to-get-started

Configuring ~/.chef

The opscode instructions have you create ~/chef-repo/.chef and put your knife.rb and .pem file in there.

I prefer to move those files into ~/.chef so that I don’t have to cd ~/chef-repo in order to use knife. You can also push out thse config files in ~/.chef just like ~/.bashrc files so that you can use knife on different nodes. Of course a hacker who gains root on one of the servers where your knife credentials are can impersonate you to the chef server, so it may be wise to restrict your knife config to a set of bastion servers. Obviously, chef can help you manage this config.

My ~/.chef/knife.rb looks like:

log_level                :info
log_location             STDOUT
node_name                "USERNAME"
client_key               "#{ENV['HOME']}/.chef/USERNAME.pem"
validation_client_name   "ORGANIZATION-validator"
validation_key           "#{ENV['HOME']}/.chef/ORGANIZATION-validator.pem"
chef_server_url          "https://api.opscode.com/organizations/ORGANIZATION"
cache_type               'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{ENV['HOME']}/chef-repo/site-cookbooks"]

Here, USERNAME and ORGANIZATION should be replaced by your chef username and your organization’s name. You will also need to put your credentials in ~/.chef/USERNAME.pem (replace USERNAME by your username).

You have now configured knife to know who you are, where your credentials are, and configured some paths in your environment.

You should now be able to run comands like knife node list in any directory on the host that you have configured.

I’ve also setup my cookbook_path to ~/chef-repo/site-cookbooks so that from any directory, knife can find my cookbooks in order to upload them — you should modify that to wherever you have your site’s cookbooks.

Adding New Nodes

Now, when you’re done with that, to add a new node, you can deploy these two files to the new node:

/etc/chef/client.rb
- this points chef at the server
/etc/chef/validation.pem
- your organization key to authenticate first time to the server

Then become root and run your chef-client to add the node to the server and create the client.pem

Now you can run knife node list to see the newly added node.

You should protect /etc/chef by making it mode 0700 or something similar. Once the /etc/chef/client.pem file has been bootstrapped down from the server, you (or a chef recipe) can delete the /etc/chef/validation.pem file.

Summary

So, now you should have setup an account at opscode, successfully setup your organization, configured knife to be easy to use on your host, and bootstrapped a few hosts to use chef. Now you can move on to getting chef to do useful things.