Affordable Bandwidth Management for Organizations

Broadband Internet access at speeds of 3 to 10 Megabits/sec is generally available at most business locations in the US, and t1 (1.5 Mbits/sec) server is available everywhere (for a higehr price of course). Managing how this internet bandwidth is used though is more important than ever before as many companies are pushing business-critical traffic such as LAN to LAN VPNs to other offices and outsourced applications, both web and non-web based. End users that run Peer2Peer (P2P) file sharing, streaming media, and large web searches/requests like Craigslist, Ebay, and fan site message boards can easily render a corporate network unusable, unstable, or simply very slow and unresponsive for periods ranging from a few seconds to minutes or more. Content filtering devices such as WebSense, SurfControl, and Barracuda Networks Web Filter are certainly options for some organizations but these solutions tend to work best when the internet access is centralized at one common choke point, such as in front of an internet-facing firewall. What about networks that are de-centralized, though, where multiple locations of operation are spread across a WAN or VPN cloud? It certainly is less than optimal in terms of bandwidth usage and performance to backhaul internet access to a common choke point where a single appliance can manage the internet access policy. Licensing for these commercial appliances is also not cheap.

OpenDNS is a service that you can sign up and test for free at http://www.opendns.com. Commercial use does come with an access charge, and there are additional benefits that come with the paid-for service. Richweb sets up an openbsd appliance on a 1u rack mount server (or desktop tower for lower budgets) and configures a DNS cache on the appliance (we refer to these appliances as an sgw box or secure gateway). The sgw box has the local DNS cache set to forward all queries to the OpenDNS servers. Additionally, squid proxy cache is installed on the sgw box. The local GPO (windows domain group policies) can be set to require that all outbound http traffic uses the proxy server on a per user, per group policy level.

Once you have an account created with OpenDNS, you can automatically add the ip address of the proxy server into the managed Domain List with OpenDNS. You can then set controls over what types of content to allow via the OpenDNS web console. It acts very much like a typical web-content filter, except that the filtering is done at the DNS level, not at the http level.

Since the squid proxy cache is caching both DNS responses as well as fetched http data, the cache will speed up web access and save bandwidth by usually 30 to 50% if not more. If you have multiple internet connections, such as a private t1 or vpn connection back to a head quarters link, and a broadband (FIOS or Comcast) internet connection, you can start load balancing traffic across the broadband connection by pointing the proxy server out that internet pipe. This leaves the other connection free for handling business applications, vpns, etc that will run better without being clogged with internet traffic.

Be sure to review the information in this link below for an interaction between OpenDNS and your local Email server that can cause problems:

http://www.richweb.com/opendns_comcast_email

Firewall cabling

Richweb sgw boxes are similar to Cisco Pixes in that eth0 is ALWAYS the uplink or WAN or PUBLIC facing connection and eth1 is ALWAYS the internal network or a DMZ subnet.

Firewall change process

For an incoming service (such as a port that needs to be opened from the outside world on a server) to be configured you must provide the following information so that Richweb can execute the changes. An incoming service is defined as traffic flow that originates from OUTSIDE the firewall and is destined for an internal server on the client network behind the firewall.

1. The internal, private IP address of the server
2. The protocol type (usually tcp,udp; may also be gre/pptp and/or ipsec)
3. The port number (for tcp and udp services)

Optional:

4. If you only want the service open to selected external IPs (which must be static, non-changing ips or ip blocks) you can provide this as well.

Richweb will then create a NAT translation on the firewall appliance so that traffic is forwarded to the desired internal service.

Sometimes, you will have a situation where you have a resource that needs a public IP on it (perhaps its an SSL protected appliance or web server) that must be reachable via internal and external clients on the same dns name. This can happen with non MS Exchange mail servers as well. Richweb will often push or route a static IP THRU the firewall, configure the packet inspection on the firewall, and bind the static public ip to the inside server as an alias. In this case Richweb will NOT be natting the IP on the firewall and will NOT be forwarding any ports, but will be restricting the traffic using access control lists.

ISPs that Richweb supports will often use this passthru configuration on Cisco PIXes were the PIX provides the firewalling benefit without the complexity of NAT and dual DNS (inside and public IPs).

For OUTGOING services the basic outbound NAT confguration should allow the packets to pass without any modification on the firewall unless the firewall has been locked down with a higher level of inspection or access control, which sometimes will be the case.

The basic way to test that a tcp port is open for OUTBOUND connectivity is to use the telnet program.

Example:

telnet mail.richweb.com 25
Trying 63.90.9.3...
Connected to ford.
Escape character is '^]'.
220 mail.richweb.com ESMTP Postfix

That shows that port 25 (smtp mail) is open on the outside server mail.richweb.com.

If a vendor is requesting that a "port be opened" in order to reach a service that the vendor may host, the first rule of thumb is to check that the port is tcp. In most cases it will be, and you can test for yourself that it does or does not allow a connection.

If the connection does NOT work, your telnet prompt will hang and be unresponsive. In that case try testing the connection again from a different network (such as a home broadband dsl or fios service) before assuming that the firewall is blocking OUTBOUND connections. In most cases it (the firewall) will NOT be blocking outbound traffic.

ICMP Filtering on a firewall

A. These icmp types are nice to allow:

# echo reply inbound:
# echo request:

B. You really need to allow these or else your network is not rfc-compliant and is quite broken and prone to mysterious problems that manifest themselves in strange, hard to troubleshoot ways like path mtu discovery. As a side note if you have hanging TCP connections, the old cultprit used to be ICMP filtering and path mtu blackholes, but unrelated problems withtcp window scaling and firewalls that filter these options on ip packets with tcp connection setup may be at fault.

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080742d6e.shtml

From the doc:

dropped TCP connections [are] caused by some versions of PIX software not
supporting the TCP Window Scaling option. This causes it to have a much
smaller TCP window than the endpoints actually have. This causes the Cisco
PIX to drop packets that it believes are outside the TCP window, but which
really are not.

# Inbound can't fragment, among other messages:
# ttl exceeded:
# general parameter problem:

C. Here is the ruleset Richweb uses for icmp which works quite well. You could translate this to whatever firewall OS you use.

# echo reply inbound:
print qx{$FW -t filter -A INPUT -p icmp --icmp-type 0 -j ACCEPT};

# Inbound can't fragment, among other messages:
print qx{$FW -t filter -A INPUT -p icmp --icmp-type 3 -j ACCEPT};

# Inbound Sourch quench - no longer needed 2007/12/31:
# print qx{$FW -t filter -A INPUT -p icmp --icmp-type 4 -j ACCEPT};

# echo request:
print qx{$FW -t filter -A INPUT -p icmp --icmp-type 8 -j ACCEPT};

# ttl exceeded:
print qx{$FW -t filter -A INPUT -p icmp --icmp-type 11 -j ACCEPT};

# general parameter problem:
print qx{$FW -t filter -A INPUT -p icmp --icmp-type 12 -j ACCEPT};

# drop all other icmp:
print qx{$FW -t filter -A INPUT -p icmp -j DROP};

Here is a proper ruleset and config for a Cisco ASA firewall:

access-list inet_in extended permit icmp any any time-exceeded
access-list inet_in extended permit icmp any any unreachable
access-list inet_in extended permit icmp any any echo-reply
access-list inet_in extended permit icmp any any echo

policy-map global_policy
 class inspection_default
  inspect icmp

access-group inet_in in interface outside

Richweb SSH Customizations

By default ssh binds to ALL ip addresses on a host. For application servers that run 1 or more vservers Richweb changes the sshd config on the main host to listen only on the main host ip address.

Example /etc/ssh/sshd_config entry:
ListenAddress feller.richweb.com

Where feller.richweb.com is a valid /etc/hosts entry.

When changing the ip address of an appliance or app server it is important to correct /etc/hosts and restart sshd.

Richweb will often change the default listen port as well:

Example:
Port 822

Other typical sshd settings:

TCPKeepAlive yes

X11Forwarding no
ClientAliveInterval 120
ClientAliveCountMax 3
UseDNS no

PermitEmptyPasswords no

Richweb sgw (Secure GateWay) firewalls

sgw = Secure GateWay; a Richweb security appliance that is based upon the OpenBSD operating system (http://openbsd.org/) which provides industry leading out of the box security and firewall feature sets. A low end sgw box is a desktop class Intel PC with a single hard disk and 1 or more network interfaces (NICs) and 512MB of RAM or more. Higher end sgw boxes are typically 1u IBM servers with hardware raid, RSA cards for remote management, and 3 Gigabit network cards and 2GB of RAM.

Richweb has a standard set of hardening techniques that are applied to the sgw boxes and a regular set of vulnerability scans and intrusion detection tests are performed against both test and live sgw instances to ensure that the sgw appliance is the most secure device you will have on your network.

sgw boxes all have a complete set of firewalling functionalty that is managed via a the /etc/pf.conf configuration file. Richweb sgw boxes are deployed as network firewalls, network packet shapers, tcp or http proxy servers, spam filtering firewalls, intrusion detection servers (running snort for example), dns caches / dns servers, and mail bastion hosts (secure smtp relay). Only the software that is needed to perform the specific job functions is loaded onto the appliance.

In many cases Richweb will install an sgw box for monitoring purposes; one or more serial ports (RS232) will be connected to Cisco router or firewall consoles or other serial devices. The sgw box is aslo very useful as a TCP proxy; it can terminate inbound TCP connections to an internal server via a backup internet connection even when the primary connection is up and traffic by default is flowing out the primary connection. This is very useful for load balancing and disaster recovery situations with smtp and rdp, where the limitations of source and destination NAT can be painful.

All sgw configurations are backed up via subversion repositories so that if a disk or raid array is lost, full functionality can be recovered by retoring the configuration files from svn.

 

The cabinets which house the servers are locked; the servers are locked at the BIOS level. Only authorized (by Richweb) data center personnel have access to the servers (as directed by Richweb).

Our colocation facilities have SAS70 type II compliant access control implemented. There is 2 factor authentication: ID card acquired in exchange for valid US Govt. ID as well a biometric reader that blocks unauthorized access at the man trap. Only pre-registered users that have entries in the biometric scanner database are allowed into the data center. Each user type (employee, contractor, vendor, customer) is categorized for the purposes of ID generation. All movement through the facility is monitored and tracked.

Richweb sgw development and maintenance is isolated to Richweb's corporate headquarters where the senior developers (a small group of 3 to 4 analysts) have access to the code base. All source code, server configuraton files and server system files that detail setup are checked into and managed with the industry leading open source code management software: Subversion. Richweb uses Subversion over SSL protected HTTPS sessions so no source code or config files are pulled via clear text protocols at any time.

 

Richweb does not maintain or create accounts for Richweb management purposes to client sgw instances unless authorized and instructed to do so by an authorized client administrator with the client in question. This authorization should be given via the telephone.

Some clients have made arrangements with Richweb whereby certain actions can be taken via emails from the client administrator but usually these directives are issued via telephone conferences or conversations. A written
email will then be sent for confirmation of any action that is to be taken.

This provides an audit trail for the access that was granted. Richweb logs all time spent supporting a client in our EMS project management system. The EMS system project logs will contain these notes and actions taken.

Actual access to the appliance takes place via the ssh2 protocol using ssh keys (password authentication can be disabled for additional security measures).

TCP Port 22 is open inbound to the firewall from Richweb Network Operations. This port allows authorized Richweb’s system administrators to access the system to perform maintenance operations. Backups also travel through this port. All communications across this port use strong encryption for SSH access (SSH = secure shell, an industry standard remote access protocol for secured hardware).

SSH access to the sgw boxes is restricted at the firewall level to these 2 ip ranges. In some cases the internal (private) ip ranges of a customer's own network are allowed to ssh into the sgw appliance if a customer has 1 or
more resources that are trained in the basic sgw command set, such as examining bandwidth usage.

 

Because there is virtually no "attack surface vector" for an sgw box setup as a firewall for example, updates are really not required as frequently as with Windows bases systems. The "attack surface vector" is kept to a minimum by virtue of having all incoming ports filtered, and all programs not needed shut off and un-installed. OpenBSD patches come out on occasion; Richweb tests and implements patches as needed once they are (A) verified not to cause any other problems and (B) relevant and needed by the sgw in question.

All package updates are crypto signed and installed from openbsd sources or added via the official pkg_add utility that comes with the system.

 

OpenBSD uses a monolithic kernel approach; a "root kit" would require the kernel to be patched, recompiled, and the system rebooted, which would be rather obvious. The whole point of a root kit is to be able to slip a kernel loadable module undetected into a running kernel, which is not going to happen.

 

OpenBSD is used in many secure network appliances. Refer to http://www.openbsd.org/ for more information.

Debian Linux (used by Richweb on certain app server sgw boxes) has achieved Carrier Grade Linux status.

 

TCP Window Scaling Problems with broken Firewalls

This issue was seen by Richweb recently when mail from a 3rd party network was not being delivered. Initially the Mail Filter Richweb operates was suspected, as some mails (short text) would get through but longer text emails and html emails would not.

Richweb started a tcpdump (packet trace) on the firewall and observed that the mail flow would start, and then would hang during the SMTP DATA phase.

1. We found out that the 3rd party network:

A. did NOT have the infamous smtp fixup protocol Cisco PIX problem
B. was running a non-cisco firewall that blocked ALL ICMP traffic.

Filtering ALL ICMP of course is a bad idea. We referred the admin to this resource initially:
http://blogs.richweb.com/icmp_filter

C. had domains with incorrect SPF records

We asked the admin to fix the SPF records, and we whitelisted the domains in the meantime on our MailFoundry appliance.

2. We then confirmed that the mail foundry does indeed have tcp window scaling turned on (all OSes built after 2004 or 2005 or so that are trying to do better tcp bulk data transfer will have this option turned on by default, Solaris, Linux, BSDs, Windows, etc.).

3. We believe that the remote firewall in use was stripping the TCP window scaling options which was causing the issue.

What did we learn ?

Make sure this option is not set in your firewall (Cisco ASA Pixes)
tcp-options window-scale clear

http://www.cisco.com/en/US/docs/security/asa/asa70/configuration/guide/ids.html

Clears the selective-ack, timestamps, or window-scale TCP options, or drops
a range of TCP options by number. The default is to allow packets with
specified options, or to clear the options within the range, so use this
command to clear, allow, or drop them.

hostname(config-tcp-map)# tcp-options {selective-ack | timestamp |
window-scale}
{allow | clear}

For older 6.x pixes:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080742d6e.shtml

From the doc:

dropped TCP connections [are] caused by some versions of PIX software not
supporting the TCP Window Scaling option. This causes it to have a much
smaller TCP window than the endpoints actually have. This causes the Cisco
PIX to drop packets that it believes are outside the TCP window, but which
really are not.

This bug: CSCsg00748

Was resolved in pix 7.2(2)
Clear window-scale sack option in non-syn packets instead of dropping it

More information - excerpted from:
http://lwn.net/Articles/92727/

TCP window scaling and broken routers
[Posted July 7, 2004 by corbet]

Every TCP packet includes, in the header, a "window" field which specifies how much data the system which sent the packet is willing and able to receive from the other end. The window is the flow control mechanism used by TCP; it controls the maximum amount of data which can be "in flight" between two communicating systems and keeps one side from overwhelming the other with data.

In the early days of TCP, windows tended to be relatively small. The computers of that age did not have huge amounts of memory to dedicate toward buffering network data, and the available networking technology was not fast enough to make use of a larger window in any case. Modern network interfaces can handle larger packets and keep more of them in flight at any given time; they will perform better with a larger window. Some kinds of high-speed long-haul links can have very high bandwidth, but also high latency. Keeping that sort of pipe filled can require a very large window; if a sending system cannot have a large number of packets in transit at any given time, it will not be able to make use of the bandwidth available. For these reasons, good performance can often require very large windows.

The TCP window field, however, is only 16 bits wide, allowing for a maximum window size of 64KB. The TCP designers must have thought that nobody would ever need a larger window than that. But 64KB is not even close to what is needed in many situations today. The solution to this problem is called "window scaling." It is not new; window scaling was codified in RFC 1323 back in 1992. It is also not complicated: a system wanting to use window scaling sets a TCP option containing an eight-bit scale factor. All window values used by that system thereafter should be left-shifted by that scale factor; a window scale of zero, thus, implies no scaling at all, while a scale factor of five implies that window sizes should be shifted five bits, or multiplied by 32. With this scheme, a 128KB window could be expressed by setting the scale factor to five and putting 4096 in the window field.

To keep from breaking TCP on systems which do not understand window scaling, the TCP option can only be provided in the initial SYN packet which initiates the connection, and scaling can only be used if the SYN+ACK packet sent in response also contains that option. The scale factor is thus set as part of the setup handshake, and cannot be changed thereafter.

The details are still being figured out, but it would appear that some routers on the net are rewriting the window scale TCP option on SYN packets as they pass through. In particular, they seem to be setting the scale factor to zero, but leaving the option in place. The receiving side sees the option, and responds with a window scale factor of its own. At this point, the initiating system believes that its scale factor has been accepted, and scales its windows accordingly. The other end, however, believes that the scale factor is zero. The result is a misunderstanding over the real size of the receive window, with the system behind the firewall believing it to be much smaller than it really is. If the expected scale factor (and thus the discrepancy) is large, the result is, at best, very slow communication. In many cases, the small window can cause no packets to be transmitted at all, breaking TCP between the two affected systems entirely.

In the 2.6.7 kernel, the default scale factor is zero; in Linus's BitKeeper tree and the 2.6.7-mm kernels, instead, it has been increased to seven. This change has brought the broken router behavior to light; suddenly people running current kernels are finding that they cannot talk to a number of systems out there. One of the higher-profile affected sites is packages.gentoo.org. Gentoo users are, unsurprisingly, not pleased.

As a way of making things work, Stephen Hemminger has proposed a patch which adds a calculation to select the smallest scale factor which covers the largest possible window size. The result on most systems is that the scale factor gets set to two. This factor will still be corrupted by broken routers, but the resulting window size (¼ of what it should be) is still large enough to allow communication to happen.

The patch makes networking with systems behind broken routers work again, but it has been rejected anyway. The networking maintainers (and David Miller in particular) believe that the patch simply papers over a problem, and that adding hacks to the Linux network stack to accommodate broken routers is a mistake. If, instead, the situation is left as it is, pressure on the router manufacturers should get the problem fixed relatively quickly. It has been a few years, now, that Linux has a strong enough presence in the networking world that it can get away with taking this sort of position.

In the mean time, anybody running a current kernel who is having trouble connecting to a needed site can work around the problem with a command like:

echo 0 > /proc/sys/net/ipv4/tcp_default_win_scale

or by adding a line like:

net.ipv4.tcp_default_win_scale = 0

to /etc/sysctl.conf