First of all, outbound ACLs create constant rollout failures of software. Unless you’ve got really excellent process around replicating the ACLs in pre-production and tracking the ACLs as part of rollout procedures you will inevitably wind up with rollout/deployment failures in production. I’ve been on way too many post mortems of software deployments where the failure to track production outbound ACLs has been the cause. I’ll say what doesn’t get said at these meetings, which is that the outbound ACLs themselves are the cause of the problem.

Second of all, with all the cloud services now the outbound ACLs are only going to cause more and more problems. All my servers now talk to chef servers “in the cloud” run externally by opscode. All my Dell servers pull OMSA and firmware directly from Dell. The software dev teams are using github more and more and pulling directly from those servers “in the cloud”. As more and more external platform dependencies are built it becomes impossible to deal with them all, particularly as the platform vendors change their own IPs around on their own schedule. Opscode migrated from Amazon EC2 to rackspace and I never knew about that — if we had outbound ACLs everything would have broken, or it would have been a huge firedrill. In the past 9 months I’ve gone from having basically no external dependencies on “the cloud” and burning tons of time carefully mirroring all my CentOS and RHEL RPMs, to having 3 major external dependencies to “the cloud”, and I’ve gotten more done and slept fine at night.

Sure, you can say that we should minimize all of this. Replicate the dell stuff with in-house mirrors, stop using github and run our internal git servers, setup chef servers internally and host it ourselves. But you just cut three big checks on our time, and we’re simply not staffed to handle that. Even if we were staffed to handle it, if you stack rank what I care about none of that comes close to the top of the list of stuff that I would do if we had 2 more headcount. I’ve given up thinking that its an appropriate job for a system admin to in-house everything that can just be outsourced to “the cloud” in order to remove external dependencies of the enterprise.

Sure this increases risk, but the job of the system admin is not to decrease every single risk that they find. Everything needs to be stack ranked and needs to be assessed in terms of its importance. I still have huge issues with account management and our deployment process is crap and we need to patch our servers and I don’t have enough time to do any of that. How much time do I have to worry that my servers point directly at Dell’s repo for OMSA? I don’t have time to care about that. There’s much riskier things in my infrastructure that if I ignore them I will get hacked or will definitely cause me downtime.

And sure, outbound ACLs help after you do get hacked. I’d argue, however, that the cost of the outbound ACLs is starting to greatly exceed the level that they help. I’d also argue that you need to focus on preventing the attacks from being successful in the first place. The use of outbound ACLs is lazy security engineering and damages the rest of the business. Become pro-active and scan your network for open proxies and close them, don’t rely on outbound ACLs so that you can have open proxies available to the internet, but its okay (because nobody can use them to bounce into the internet and suck up your bandwidth — they can only scan your whole internal network — its okay!). Tighten up your border security and if you must install “detect” controls instead of “prevent” controls. Log when you see anomalous behavior.

There also will still be PCI-DSS to deal with, and every security best practices document out there probably start approaching the network from the perspective of locking everything down and then poking tiny little holes in it — writing checks on your time to manage all of that. The people who wrote those docs don’t have to manage your network and it only took them a few minutes to draft that policy, while you’re stuck with the fallout from the policies for the rest of your career. You don’t have to follow them if they’re no longer working. Except for VISA, you have to follow that, but VISA would prefer that your servers all crash rather than having you violate their standard. So, wall off the PCI-DSS environment and get used to managing the outbound firewall ACLs there and suck it up as a cost to the business. But do not make the mistake of thinking that what you do for PCI-DSS is necessarily a best practice and that it should be applied to the rest of your Enterprise.

Also, certain outbound ACLs are fine. None of your production servers should probably be making IRC connections. By all means block that port and have any outbound connection to IRC servers set off security alarms that wake up your security engineers and put them to work. Similarly, you can probably engineer your SMTP sending infrastructure so that most leaf nodes forward to dedicated SMTP internal relays and only those servers are allowed to send outgoing SMTP. On a case-by-case basis that is fine. But port 80 and port 443 to the internet are going to need to be wide open in the future.

Get used to it. At some point, you will not be able to keep up and “the cloud” will solve too many problems to be able to continue to burn time replicating internet resources in-house, and the ACL management will become a nightmare and more of a risk to uptime to manage badly than to simply open it up.

Its time to stop considering outbound ACLs a “best practice” and start considering it something that bad SEs and NEs do who are just control freaks that haven’t evolved out of the late 90s best practices. We have passed an inflection point where this practice is no longer net positive to the Enterprise and it is net negative in terms of management and availability. Stop thinking that its part of doing a good job to block everything from being able to talk to the Internet. You are in the way of the business. You are an impediment to getting things done. Servers, both inside your company and outside, desperately want to talk to each other to get work done, stop getting in the way.

This is going to be a growing list of all the global configuration commands that I come up with that are useful for setting up a router/switch first-time (or for enforcing policy on all routers/switches). It is going to start out fairly sparse.

Basic

hostname <routername>
ip domain-name <dns name>

Sets the hostname and domainname.

Convenience

line console 0
  logging synchronous

Sets synchronous output on the console.

Security

enable password foo
enable secret bar

Sets the enable password, only “enable secret” should be used since it encrypts the password in the config.

service password-encryption

Sets up weak password encryption to obscure passwords in router config.

line vty 0 4
  login
  password foo
  logging synchronous

Set synchronous output on the first 5 telnet vtys and sets a login password for the terminal.

banner login #

    Authorized uses only.  All activity may be monitored and reported.

#

Set a multi-line banner displayed before the password prompt for telnet.

banner motd #

    Authorized uses only.  All activity may be monitored and reported.

#

Set a multi-line banner displayed before the password prompt for telnet *and* on console login (better).

Logging

archive
  log config
    logging enable
    logging size 200
    notify syslog contenttype plaintext
    hidekeys

Sets an archive history of router configuration commands

Time

clock timezone UTC 0

Set the timezone of the router manually.

clock set 02:11:25 Feb 15 2010
clock update-calendar

This is not entered in configuration mode, and sets the software clock and then writes to the hardware clock.

ntp server 10.1.1.1
ntp server 10.1.1.2 prefer
ntp server 10.1.1.3
ntp update-calendar

Set the router to be an NTP client, and use NTP to sync the hardware clock.

DNS

ip nameserver 10.1.1.1
ip nameserver 10.1.1.2

Sets nameservers for DNS queries

ip domain-lookup

Enable DNS lookups. This may be disabled by NEs to avoid command typos from being looked up in DNS, but it globally disables DNS lookups inside commands as well.

Spanning Tree

spanning-tree mode rapid-pvst

Use Rapid-PVST by default everywhere.

SNMP
TACACS

MISC

ip subnet-zero

Allow subnet zero ip addresses.

system mtu jumbo 9000

Set jumbo frames on 3750/3560/49xx switches.