Month: October 2009

Living with ISA 2006 and the ISA Firewall client

 

One of my big projects in 2008 was making the transition from my old firewall to a new solution.  I’ve had 18 months or so to work with ISA and the workstations running the Firewall Client software, and thought I’d share my experiences.

First, a little background.  The network I inherited long ago was protected by a Watchguard Firewall.  At the time, it was a moderately capable stateful packet inspection (SPI) unit that performed what was asked of it;  ingress filtering with a little protection from a few application layer proxies.  But times had changed and communication sessions had become more sophisticated.  Exploits were getting more creative and difficult to defend against because they were occurring high up at the application layer.  Like many SPI firewalls, it’s ability to intelligently control outbound traffic was limited.

My acceptance criteria included better protection at the application layer, as well as close integration with my Active Directory based infrastructure.  I also needed a firewall that would help me get a handle on outbound traffic.  ISA 2006 was the answer.  I chose a Celestix MsA4000i appliance running ISA to simplify the hardware procurement and deployment process.

During my implementation planning, I had the opportunity to talk at length with Richard Hicks, a Senior Engineer for Celestix Networks.  Celestix makes a fine product line of security solution appliances running ISA, and Richard (a recent MVP award winner) had excellent insight into ISA implementations, large and small.  I give him credit for helping me translate the functional requirements I was used to with my old firewall, while giving practical recommendations on how ISA performs those same functions, and policy design and implementation.

One of the unique traits of ISA is the various methods it allows internal clients to communicate with. 

  • SecureNAT.  The most basic of the three, and uses ISA as the gateway/router for traditional perimeter based protection.  Used when a default gateway is assigned to the client.
  • Web Proxy Client.  Generally called upon when there are web based requests such as HTTP and FTP calls, etc. 
  • Firewall client.  An optional piece of the ISA solution that runs on Windows clients, and extends the functionality of ISA in ways that cannot be matched by other solutions.

None of these are mutually exclusive, and can be run all at the same time.  Unfortunately, this flexibility can hinder your intentions.  If you want to restrict outbound communication to authenticated access only, running SecureNAT will compromise that ability.  The solution?  Run all non server systems without a default gateway, to force the client to use the web proxy client, or firewall client.  In the event that the target is beyond your LAN, the firewall client will handle all routing.

The easiest transition would have been using SecureNAT for the initial deployment, but there was an opportunity for monumental improvements if I attempted to go without it.  Am I glad I took this extra step?  Yes!  Some of the highlights have been:

  • Outbound connections limited to authenticated users only.  If an outbound connection is made,  I could see what user is requesting it.  Logging provides meaningful data now.
  • True egress control.  Connections initiated from the inside can finally be controlled.  Once everything was up and running, it was fascinating to see what was initiating outbound connections.
  • Forces compliance of application related restrictions.  IM and P2P applications specialize in working their way around firewalls.  The combination of the web proxy, and the firewall client with no SecureNAT helps achieve this.
  • Suppression of malware.   The combination of allowing only authenticated outbound access, along with utilizing an automated malware blacklist database helped control users who had a knack of making a mess out of their PCs.

The results of the improved security stance was impressive.  So was the amount of complaining from end users.  They were furious.  I had angry developers shutting off the firewall client software on their PC.  It made them feel good until they realized shutting down the firewall client gave them less access, not more.  They made claims that BitTorrent was a necessary part of their job, and found it insulting that outbound SSH sessions were not allowed to any host on the Internet.  They didn’t like that their non-domain joined test machines (or unapproved personal laptop) would require a username and password before they could access the Internet.  Their complaints went straight to the top of the organization, as did my explanations.  Security won out, and policies stood without change.

There were some hiccups along the way.  Most deployment related problems were fixed, while others forced some changes in how we worked.  The ISA community is an active one, but with the move of using workstations running the ISA firewall client without a default gateway, it made finding out answers much more difficult.  Some of the obstacles I ran into were:

  • Lack of support for CIFS traversing across network segments.  The firewall client cannot handle this alone, and needs a default gateway.
  • Vista and later workstations need a static route added for remote targets that were not web based.  This can be added via DHCP (option 121, but don’t try to add it via the DHCP snap-in in Vista, otherwise it won’t work).  Thanks to some assistance from Richard Hicks and Microsoft for ultimately explaining the reason behind the inconsistent behavior between XP and Vista.  More info can be found here: http://tmgblog.richardhicks.com/2009/01/10/dns-resolver-behavior-in-windows-vista/ 
  • Building up a healthy list of domains that will be allowed to have anonymous outbound access.  OS and application update domains and mirrors are good examples of this.
  • Older Outlook Clients (2003) wouldn’t talk to the internal Exchange Server using it’s MAPI connection until the following tweak was made:  http://www.isaserver.org/articles/2004olpop3smtp.html
  • Web services that use SSL, but do not run over port 443 had to be accommodated for.  http://www.isaserver.org/articles/2004tunnelportrange.html
  • Browser proxy configurations in *nix workstations may not be enough.  For those workstations, leave a default gateway.

As you can see from the links I provide, I found www.isaserver.org invaluable during my implementation.  It attracts some of the brightest and the best in the security world who contribute articles, and to community forums.  It’s a great resource for any ISA administrator. 

My biggest annoyances in using the firewall client are small, but still worth mentioning.

  • The virtual black hole that the occurs on socket of the workstation running the firewall client.  Trying to debug via traditional methods is nearly impossible.  It simplifies the number of connections from the client, but it’s hard to tell the contents of the connection.
  • The name.  “Firewall Client” implies that it is some application that protects a workstation like ZoneAlarm, Norton, or the Windows Firewall.  A simple name change would eliminate this confusion to newer users, and some IT guys not familiar with ISA.

If I were to do it over again, I would have given more notice on what changes would be occurring, and why.  I had previous verbal green lights from management to restrict thing things like P2P and IM sessions, and our written IT policies had already reflected these restrictions.  I just never had the capability to do so.  I warned staff, but apparently not enough.  I had to do a healthy amount of explaining, which was fine because I had the technical reasons, and the business case on my side. 

I look forward to the next version of ISA (Threat Management Gateway, or TMG) and the steps it takes to improve upon the Firewall Client component.  Recommended reading on using the Firewall Client in ISA 2004 and 2006 can be found below.

Firewall Client
http://www.isaserver.org/tutorials/Understanding-ISA-Firewall-Client-Part1.html

http://www.isaserver.org/articles/2004firewallclient.html

http://www.isaserver.org/tutorials/Understanding_and_installing_ISA_Firewall_Clients.html

http://www.isaserver.org/tutorials/ISA_Clients__Part_2_SecureNAT_and_Web_Proxy_Client.html

Database of malware domains that can be imported directly into ISA
http://www.malwaredomains.com/

A special thanks to Richard Hicks from Celestix, and my good friend Glenn Barnas from Inno-Tech, who provided invaluable information when I needed it most.

Exchange 2007… Better late than never

 

Count me in as one of the many Administrators that finally got around to moving from Exchange 2003 to Exchange 2007.   Yes, it’s almost 2010, and probably just a few months away from the release of Exchange 2010.  I never disagreed with Microsoft’s decision to release Exchange 2007 as a 64 bit only application.  It’s just that it made it incredibly difficult to find a way to transition when your infrastructure is full of physical 32 bit servers.  I was constantly reminded that my Exchange 2003 server was brittle, a resource hog, and lacked many of the abilities that the end users and applications were demanding.  Twenty-five minute reboots, and inexplicable behaviors did not instill confidence.    It was time to make the move.

My timing couldn’t have been better.  I have a new virtualized infrastructure powered by some Dell blades, running VMWare ESX 3.5 using a Dell/ Equallogic PS5000 SAN to help with this transition.  It didn’t take away from the normal shrewd planning necessary on a project of this nature.  It just made it easier.  The other benefit to deploying 3 year old software is that issues, workarounds, and fixes to Exchange, as well as how to make it play nicely with other components (ISA, CRM, etc.) were well documented. 

With respect to running Exchange 2007 in a virtual environment, I noticed information on sizing is difficult to find.  Capacity planning guidelines still reflected deployments with physical servers (read: more is better), and white papers on virtualizing Exchange appeared to have the intended audience of large enterprise environments only.  My environment is relatively small, and a single server was going to be handling all Exchange roles.  Based on my environment of about 50 users and 100 mailboxes, I sifted through all the material I could find, threw in a few wild guesses, and decided with the following configuration.

  • 1 vCPU
  • 2.5GB of RAM
  • Primary OS resides in VMFS volume on the SAN
  • Using guest OS based iSCSI initiator with MPIO enabled, dedicated NTFS volume for Exchange database. 
  • Using guest OS based iSCSI initiator with MPIO enabled, dedicated NTFS volume for Exchange transaction logs.  

The transition occurred over one weekend, while the remainder of the outstanding issues were cleaned up throughout the week.  A moment of embarrassment occurred when I realized my thorough planning never took into consideration that the “move mailbox” function would hit the transaction logs so much.   I didn’t see it until it was too late.  The partition for the transaction logs filled up and the services shut down.  But, thanks to the ease of handling storage with the SAN, I was able to create a new LUN, initialize it in the OS, and change a couple of drive letters.  Ran a backup to commit the transaction logs to the database, and I was back in business.  The only puzzle that took me far too long to figure out was getting the AutoDiscover function in Exchange 2007 to work as intended.  It is worthy of it’s own post at another time.

As far as performance goes, I couldn’t be happier with it running as a VM.  I truly didn’t know how it would react under the settings I established, and was ready to make whatever changes necessary, but it has been performing very well.  Below are some very simple utilization numbers to give you an idea.

CPU:  Hovers around 50% utilization during the day, and around 15% utilization during off hours.  CPU Ready values range anywhere from 20 to 200 milliseconds.  This VM has just 1 vCPU assigned to it.  I wanted to keep it this way if possible, so that it could use the “Fault Tolerant” feature of vSphere when we upgrade.  Looks like I get my wish.

RAM:  Typically runs about 50% of the 2.5GB assigned.  No higher than 75% utilization during the busiest time of the day, and 15% utilization during off hours

Network:  Runs about 1.2 MBps.  Spikes of 40 MBps occur only because of some on-host backups occasionally occurring.  Bandwidth utilization is imperceptible during off hours.

Disk I/O:  Hard to gauge, but mostly because the need seems to be so low.  The OS partition coming from the VMFS volume might show 200KBps, but has bounced up to 7MBps on occasion.  No performance data yet on the drives connected via the guest iSCSI initiator.  I think they qualify as “fast” until I find anything that suggests otherwise.

Perception:  You won’t find this on any ESX or OS performance monitor.  But it’s importance cannot be underestimated.  I had about half of the staff comment about how much snappier their Outlook clients and OWA was working for them.  I never have staff stop by and offer random comments telling me how fast something is.  It’s a nice compliment to the improvements of Exchange 2007, and what it is running on.

No more painfully long restart times for me now.  It’s one of the more overlooked benefits I’ve noticed while moving my systems over to my new infrastructure.  Planned server restarts are a part of responsible management of IT systems.  Anything that can be done to reduce any interruption is appreciated.  A two minute restart is always welcome

It probably goes without saying that I took great joy decommissioning the old server.  Hopefully this will be me in a good position when Exchange 2010 comes off the presses.