The trouble with in-line content filtering devices

Content guard / filtering devices like Surf Control, Websense, and now the Barracuda filter are often placed inline with the network, between the firewall and the local LAN. In this mode they act like a transparent (but intelligent) bridge or switch. They can block packets such as icmp echo requests (pings) and tcp/http gets, and smtp connections if they find that the destination ipv4 address matches an entry in the block or drop list that the devices download from their parent database. These drops can be very difficult to troubleshoot, especially if the firewall in question has limited packet debugging capabilities.

In a recent troubleshooting session after several hours spent trying to figure out why seemingly random destination hosts could not be contacted via SMTP or icmp echo, we finally figured out the culprit was an inline content filter. We could ping the hosts in question from the firewall itself, but hosts inside the firewall could not. Packet capture on the Cisco PIX firewall showed that the packets were not even reaching the INSIDE interface of the PIX. We had been thinking the problem was with either the output filters on the PIX, the stateful inspection, or the nat rules. But it ended up being a device that was silently dropping the packets before they even reached the PIX.

Lesson learned - a better way perhaps to use content filters (if they are http only) would be via the url server component on the firewall itself. Or perhaps OopenDNS ... block the dns requests before an HTTP connection can even be made.