Firewall adventures: Transitioning from ISA 2006 to TMG

 

One of the key parts of my seemingly never-ending Offsite Replication project was to build out a second location to replicate my data to.  Before I could do this, some prep work to my network was in order.  It was a great opportunity for me to replace my existing firewall running Microsoft’s ISA 2006 server, to their newest edition, named ForeFront Threat Management Gateway, or TMG. 

My new TMG system is running on a 1u appliance provided by Celestix Networks, Inc.  Introduced to the Celestix line of appliances back in 2007, I’ve been very happy with the great turn-key solutions they provide.  Its great for those who want to run ISA/TMG, but do not want to build up their own unit, and do not want to handle licensing of the OS or TMG.  The lineup they offer ranges anywhere from branch office solutions to backbone class systems  Some really nice abilities are built right into the unit, such as web based management, and updating the unit to a new build by booting to PXE.  It also offers a “Last Good Version” (LGV) that will reimage the disk the the state it was saved, in the event of a configuration change going terribly wrong.  Definitely peace of mind for those critical upgrades.  The nature of the image creation and restore is such that it requires the system to be offline.  I hope that in the future, Celestix can perhaps partner with Acronis, or some other disk imaging solution to make this process a little more convenient.  It still works pretty well though.  Anyway, onto the transition.

 

Upgrade, or transition?

This seems to be one of those ubiquitous IT related questions to almost any enterprise solution that is being run in a production environment.  Should you do an in place upgrade, or should you transition to a pristine installation?  In this particular case, this was already answered for me, as my old appliance ran a 32bit version of Windows Server 2003, and could not be upgraded due to system requirements.  That was okay with me.  A true upgrade fell out of favor with me years ago; there are just too many unknowns introduced, which can make post deployment issues extremely difficult to diagnose.  I’ve also sensed that the true upgrade has fallen out of favor with software manufacturers as well.  Whether it’s Exchange, SQL, or a server OS, the recommended way these days seems to be transitioning to a pristine installation.

 

The new box

For the new environment I was building, I chose two Celestix MSA5200i units; one for the primary facility, and one for the CoLocation.  These particular units run TMG Standard, on top of Windows Server 2008R2. It would have been nice to go with a unit running the Enterprise Edition of TMG (that offers the ability to create a redundant array of servers), but I had to cut costs, and going with the Standard Edition was the easiest way to do this.

With the new unit sitting in front of me, I decided to build it up in its entirety offline, and wait for a weekend to cut it over.  ISA has the ability to dump out all, or parts of the old configuration in XML, so my early (albeit naive) visions had me thinking that my transition steps would simply be exporting the configuration running on the ISA 2006 box, and importing it to the TMG box.  Well, the devil is in the details, and while this could work for certain scenarios, it didn’t work for me on the first a few tries.  I had a choice.  Continue chasing down the reason why it wasn’t importing (an unknown time limit), or pound out a new configuration in a few days (a known time limit).  No time to complain – just do it and get it over with.  Good documentation in OneNote, and the ability to RDP into your existing ISA installation is key to this being a successful way to build a new configuration from scratch.  To minimize typos and other fat fingering, I did export custom sets and protocols at the very granular level.  Sure, I could type them out easy enough, but it was more reliable to export at the very small item level.

A properly configured TMG box is almost always joined to Active Directory, and there are some steps that you just have to wait to get to on the day of transition.  This is reasonable, but it does have to be planned for.  Things like using Kerberos Constrained Delegation in publishing rules, can only be configured after it’s joined.  It’s also worth making sure you know all AD related settings (Delegation, OU location, GPO overrides, etc.) for the existing Firewall that you will be decommissioning.  Nothing like a oversight here to mess you up.

 

Post installation surprises

The abilities of TMG make it far more than a simple edge security device.  It is what truly separates it from the competition.  Since it is integrated into the operation of so many functions up and down the protocol stack, transition like this can be a bit disruptive.  I’m happy to say that considering the type of change, I didn’t run into too many troubles. I had prepared a checklist of basic functions and services I could run over to quickly validate a successful transition.  This made validation easy, and prevented most Monday morning surprises. 

After about 20 minutes, I had the old ISA box removed from the domain, and the new one added and configured.  The rest of the time was spent confirming functionality, and resolving a few issues.  Here were some of the minor ones:

  • ARP caching.  This isn’t the first time this has bitten me.  I forgot that the ARP cache on the connecting devices needed to be flushed.  Silly mistake, but the nice part is, that it eventually corrects itself. (I wish I had a few more of those kinds of problems).
  • Publishing rules and Listeners.  After you join the box to the domain, you will want to check these, and recreate if necessary.  I had a few publishing rules that I had to recreate.  Not a big deal.  They looked okay, but just didn’t work.
  • I have several publicly registered IP addresses bound to the external (WAN) interface.  Windows  2008 and TMG didn’t bind to the IP address I was thinking it was going to bind to (or at least the way Win2003 and ISA did).  A quick fix in the TMG configuration resolved this.   Look to this TechNet Article on why the behavior is different.

The final issue was a little trickier to fix.  The symptoms were that web browsing was working, but it just took a while to connect.  After looking at the logging, (and being tipped off on a thread on isaserver.org’s community forum), I noticed that the web proxy was attempting to use one of the RRAS adapters as the default gateway.  It was being caused by web proxy clients getting confused when reading WPAD for automatic browser/proxy configuration.  The slow browsing would go away as soon as the web browser’s proxy settings were manually configured.  Apparently this behavior wasn’t unique to TMG (others on ISA 2006 have experienced similar behavior), but this was the first time I’ve ever seen it. 

There was a .vbs script that supposedly fixed the issue.  The purpose of the .vbs script was to insert the FQDN of the TMG unit into WPAD.  While the script ran successfully, it didn’t change the behavior for me.  At this point, a little bit of panic set in.  I thought it best to tap into the expertise of my good friend, and TMG superstar Richard Hicks.  Richard is a Microsoft MVP, and has a great blog that should be in everyone’s RSS feed list.  After briefing him on the scenario, he provided me with another script (courtesy of Technet) that would attempt to achieve the same results as the failed script.

http://blogs.technet.com/isablog/archive/2008/06/26/understanding-by-design-behavior-of-isa-server-2006-using-kerberos-authentication-for-web-proxy-requests-on-isa-server-2006-with-nlb.aspx

Option Explicit

Const fpcCarpNameSystem_DNS = 0
Const fpcCarpNameSystem_WINS = 1
Const fpcCarpNameSystem_IP = 2

Dim Root, Array, WebProxy

Set Root = CreateObject("FPC.Root")
Set Array = Root.GetContainingArray
Set WebProxy = Array.ArrayPolicy.WebProxy

If fpcCarpNameSystem_DNS = WebProxy.CarpNameSystem Then

  MsgBox "ISA is already configured to provide DNS names in the WPAD script.", vbInformation
  WScript.Quit

End If

WebProxy.CarpNameSystem = fpcCarpNameSystem_DNS
WebProxy.Save true

MsgBox "ISA was configured to provide DNS names in the WPAD script.", vbInformation

Set WebProxy = Nothing
Set Array = Nothing
Set Root = Nothing

After I applied the .vbs script above, the issue has seemed to resolve itself, and now it’s all running smooth. 

 

Observations

During my initial build of the new TMG unit, the first thing I noticed was the apparent efforts the TMG Team took to maintain the same look and feel as the previous version.  I had seen screenshots of TMG, but that doesn’t give a good feel for UI interaction.  Aside from the new features, it was quiet a relief to feel instantly comfortable with the UI.  What a welcome relief to the overworked IT guy.

The next step was to give myself a refresher on what was new with TMG, and digest how that was going to influence my configuration after the cutover was complete.  The improvements really do read like a wish list for the seasoned ISA 2006 user.  Sometimes the Value Proposition for a software manufacturer, and their customers don’t match up.  The result is this odd rollout of new features that the customer never asked for, and ignoring what the customer wants.  That doesn’t seem to be the case at all with this product. 

For my transition, it was most prudent for me to delay taking advantage of some of these features, just to reduce all variables, but will definitely be exploring the great features of of TMG in the coming weeks and months.  The top priority right now is getting my second TMG unit built and configured for my CoLo facility, and test my replication.  That’s what a deadline does for you.  It ruins all the fun.

Once again, a big thanks to ISAserver.org for being a great resource for the ISA/TMG user community, as well as the folks at Microsoft, Rich, and the others at Celestix for making a quality product.

Discovering AutoDiscover in Exchange 2007

 

In my post “Exchange 2007… Better Later than Never” I mentioned one of the post-deployment difficulties I faced was getting the "AutoDiscover” function to behave the way it was designed.  For those unfamiliar with the feature, it allows for automated discovery and configuration of various connectivity methods to an Exchange Server.  Exchange MAPI clients, Exchange HTTP/RPC clients, and mobile devices using ActiveSync all can use AutoDiscover in some form or another.

While it wasn’t critical for the transition itself, AutoDiscover was vital for our future deployments of “Outlook Anywhere” and “ActiveSync.”  I figured skimming over a few TechNet articles and blog postings, and I’d be quickly onto the next project.  That began my long ugly journey getting AutoDiscover to work.

It became clear that the ingredients for AutoDiscover to work correctly was a properly configured ISA Server, SSL certificates, namespace/DNS accommodations, and of course, Exchange.  What was really interesting about this particular project was that I was dealing with very mature products, yet, I never ran across so much contradicting information on how to make it work.  Perhaps some of that stems from so many valid topologies and configurations, or possibly big changes between the RTM versions of Exchange and ISA and their first service packs.  Still, it seemed odd.  I sifted through postings from desperate IT Administrators in similar situations who had no more hair to pull out.  You could sense the defeat in their words.  Now I understand.

One guideline mentioned quite often was the need for a special SSL certificate that allowed for more than one FQDN to be assigned to it.  You’ll see it referred to as a Unified Communication Certificate (UCC or UC) or a Subject Alternative Name (SAN) certificate.  The purpose is the same, but the names and the references are different.  While UC certificates are not technically a requirement, it is best to think of it that way.  For AutoDiscover, the names needed on a UC cert would look something like:

mypubliccompanyname.com
autodiscover.mypubliccompanyname.com
mail.mypubliccompanyname.com
internalmailservername
internalmailservername.myprivatelanname.lan

I went with a UC cert from DigiCert, but any of the larger commercial CA’s should work.  However, a word of warning.  Exchange doesn’t like self signed certificates, and many mobile phones have troubles with private certificates as well as those from smaller commercial CAs.  You should be fine if you run Certificate Services internally (or so I’m told), and your namespace checks out okay.  Don’t forget look at your ISA server and make sure you are running SP1 or later, due to limitations on how the RTM version handled UC certificates.

Speaking of namespaces, time for a thorn in my side to come back and sting me.  My internal namespace is not a name that we own (a legacy issue I should have taken care of long ago).  Certificate Authorities will not issue standard or UC SSL certificates to names you do not own for obvious reasons, even if the references are private.  Fortunately, I was able to work around this by making absolutely sure the simple name was used in any Exchange configuration settings that usually accepted the internal FQDN.  Disaster averted.

Now for the dirt on how I was able to make it work.  My as-built  design is modeled somewhat after Jason Jones’ method of Publishing Exchange 2007 Services with ISA 2006.  Following the construct of:

  • Not using the existing listener created for OWA, and creating a separate listener for Outlook Anywhere (OA)/Autodiscover, and binding UC cert to that listener. Using HTTP authentication with Integrated/Windows Auth (aka NTLM). This would provides HTTP/Integrated auth from the client to the FW, then basic auth from the FW to the Exchange server.
  • Allowing the ISA server to utilize Kerberos constrained delegation (KCD) by way of changes in AD.
  • Creating a single Publishing rule for OA , where KCD is used.
  • Setting internal and external URL’s to their respective internal and external locations (internalmailservername and autodiscover.mycompanyname.com)

After configuring it as above, AutoDiscover worked internally, but not externally.  Continually getting failures with the /rpc directory when testing internally (via test-outlookwebservices) and externally (via testexchangeconnecivity.com).  I found a post that gave the missing piece of the puzzle, and modified my configuration per the recommendations  http://forums.isaserver.org/m_2002041377/mpage_2/key_/tm.htm:

  • Create a 2nd Publishing rule for OA, sitting on top of primary OA publishing rule.
    • Only /rpc/* is published
    • Auth Delegation is set to "No Delegation, but client may authenticate directly"
    • Set to "all users" instead of "authenticated users"
    • Changing "EXPR" Outlookprovider to msstd:mail.mycompanyname.com so that the certificate mutual authentication test passes.

Under the conditions described above, Outlook Anywhere with Autodiscover functions as desired.

As Jason Jones put it best, “The reason for the need of a separate listener is that Windows Authentication (NTLM) and Forms Based Authentication (FBA) are mutually exclusive. It is not possible to use a single web listener for all Exchange 2007 publishing and achieve transparent authentication within Outlook anywhere.”  Thus the need to create a dedicated listener to be used exclusively for Outlook Anywhere and associated services.

I did have to make one other adjustment that is rarely brought up in the AutoDiscover deployment scenarios.  We know that AutoDiscover wants to look at your TLD name (e.g. yourpubliccompanyname.com) when doing it’s discovery process.  However, you may have simply had an “A” record of “yourpubliccompanyname.com” pointing to your web server to catch those users who forget to type in “www” before your domain name.  It’s also not a far fetch to assume you had an SSL certificate bound to that web server as well.  This is exactly what I had, so I had to make the following changes:

1.  Have our ISP (or whomever has authoritative control on the DNS zone file for “mypubliccompanyname.com”) change the “A” record from my public web server IP address, to my autodiscover address.

2.  In ISA, add a new “DENY (REDIRECT)” rule for mypubliccompanyname.com that does, well, a deny, and a redirect www.mypubliccompanyname.com.  This sits right above the web publishing rule for www.mypubliccompanyname.com

The original setup was a carryover from an earlier time.  The configuration above is the way I should have set it up.  Nice to do a little cleanup along the way.

I can’t tell you how relieved I was in getting this to work, no matter how many hoops I had to jump through.  I also have a complete set of as-built notes in case I need to recreate or debug the existing configuration.  It’s been stable since, but I have a feeling I’ll be looking at this again as soon as we transition to Exchange 2010. 

Other helpful links:

Microsoft Exchange Remote Connectivity Analyzer
https://www.testexchangeconnectivity.com/

Publishing Exchange 2007 Services with ISA Server 2006…
http://blog.msfirewall.org.uk/2008/07/publishing-exchange-2007-services-with.html

Technet white paper:  Exchange 2007 Autodiscover service:
http://technet.microsoft.com/en-us/library/bb332063.aspx

Generating SSL certificates for Exchange 2007 and ISA 2006:
http://www.isaserver.org/tutorials/Generating-SSL-Certificates-Exchange-2007-ISA-Server-2006.html 

Dr. Tom Shinder’s guides on Publishing Exchange 2007 OWA, activeSync, and RPC/HTTP using ISA 2006:
http://www.isaserver.org/tutorials/Publishing-Exchange-2007-OWA-Exchange-ActiveSync-RPCHTTP-using-2006-ISA-Firewall-Part1.html

A bulk discount on Tylenol.  …You’ll need it.
http://www.costco.com

Living with ISA 2006 and the ISA Firewall client

 

One of my big projects in 2008 was making the transition from my old firewall to a new solution.  I’ve had 18 months or so to work with ISA and the workstations running the Firewall Client software, and thought I’d share my experiences.

First, a little background.  The network I inherited long ago was protected by a Watchguard Firewall.  At the time, it was a moderately capable stateful packet inspection (SPI) unit that performed what was asked of it;  ingress filtering with a little protection from a few application layer proxies.  But times had changed and communication sessions had become more sophisticated.  Exploits were getting more creative and difficult to defend against because they were occurring high up at the application layer.  Like many SPI firewalls, it’s ability to intelligently control outbound traffic was limited.

My acceptance criteria included better protection at the application layer, as well as close integration with my Active Directory based infrastructure.  I also needed a firewall that would help me get a handle on outbound traffic.  ISA 2006 was the answer.  I chose a Celestix MsA4000i appliance running ISA to simplify the hardware procurement and deployment process.

During my implementation planning, I had the opportunity to talk at length with Richard Hicks, a Senior Engineer for Celestix Networks.  Celestix makes a fine product line of security solution appliances running ISA, and Richard (a recent MVP award winner) had excellent insight into ISA implementations, large and small.  I give him credit for helping me translate the functional requirements I was used to with my old firewall, while giving practical recommendations on how ISA performs those same functions, and policy design and implementation.

One of the unique traits of ISA is the various methods it allows internal clients to communicate with. 

  • SecureNAT.  The most basic of the three, and uses ISA as the gateway/router for traditional perimeter based protection.  Used when a default gateway is assigned to the client.
  • Web Proxy Client.  Generally called upon when there are web based requests such as HTTP and FTP calls, etc. 
  • Firewall client.  An optional piece of the ISA solution that runs on Windows clients, and extends the functionality of ISA in ways that cannot be matched by other solutions.

None of these are mutually exclusive, and can be run all at the same time.  Unfortunately, this flexibility can hinder your intentions.  If you want to restrict outbound communication to authenticated access only, running SecureNAT will compromise that ability.  The solution?  Run all non server systems without a default gateway, to force the client to use the web proxy client, or firewall client.  In the event that the target is beyond your LAN, the firewall client will handle all routing.

The easiest transition would have been using SecureNAT for the initial deployment, but there was an opportunity for monumental improvements if I attempted to go without it.  Am I glad I took this extra step?  Yes!  Some of the highlights have been:

  • Outbound connections limited to authenticated users only.  If an outbound connection is made,  I could see what user is requesting it.  Logging provides meaningful data now.
  • True egress control.  Connections initiated from the inside can finally be controlled.  Once everything was up and running, it was fascinating to see what was initiating outbound connections.
  • Forces compliance of application related restrictions.  IM and P2P applications specialize in working their way around firewalls.  The combination of the web proxy, and the firewall client with no SecureNAT helps achieve this.
  • Suppression of malware.   The combination of allowing only authenticated outbound access, along with utilizing an automated malware blacklist database helped control users who had a knack of making a mess out of their PCs.

The results of the improved security stance was impressive.  So was the amount of complaining from end users.  They were furious.  I had angry developers shutting off the firewall client software on their PC.  It made them feel good until they realized shutting down the firewall client gave them less access, not more.  They made claims that BitTorrent was a necessary part of their job, and found it insulting that outbound SSH sessions were not allowed to any host on the Internet.  They didn’t like that their non-domain joined test machines (or unapproved personal laptop) would require a username and password before they could access the Internet.  Their complaints went straight to the top of the organization, as did my explanations.  Security won out, and policies stood without change.

There were some hiccups along the way.  Most deployment related problems were fixed, while others forced some changes in how we worked.  The ISA community is an active one, but with the move of using workstations running the ISA firewall client without a default gateway, it made finding out answers much more difficult.  Some of the obstacles I ran into were:

  • Lack of support for CIFS traversing across network segments.  The firewall client cannot handle this alone, and needs a default gateway.
  • Vista and later workstations need a static route added for remote targets that were not web based.  This can be added via DHCP (option 121, but don’t try to add it via the DHCP snap-in in Vista, otherwise it won’t work).  Thanks to some assistance from Richard Hicks and Microsoft for ultimately explaining the reason behind the inconsistent behavior between XP and Vista.  More info can be found here: http://tmgblog.richardhicks.com/2009/01/10/dns-resolver-behavior-in-windows-vista/ 
  • Building up a healthy list of domains that will be allowed to have anonymous outbound access.  OS and application update domains and mirrors are good examples of this.
  • Older Outlook Clients (2003) wouldn’t talk to the internal Exchange Server using it’s MAPI connection until the following tweak was made:  http://www.isaserver.org/articles/2004olpop3smtp.html
  • Web services that use SSL, but do not run over port 443 had to be accommodated for.  http://www.isaserver.org/articles/2004tunnelportrange.html
  • Browser proxy configurations in *nix workstations may not be enough.  For those workstations, leave a default gateway.

As you can see from the links I provide, I found www.isaserver.org invaluable during my implementation.  It attracts some of the brightest and the best in the security world who contribute articles, and to community forums.  It’s a great resource for any ISA administrator. 

My biggest annoyances in using the firewall client are small, but still worth mentioning.

  • The virtual black hole that the occurs on socket of the workstation running the firewall client.  Trying to debug via traditional methods is nearly impossible.  It simplifies the number of connections from the client, but it’s hard to tell the contents of the connection.
  • The name.  “Firewall Client” implies that it is some application that protects a workstation like ZoneAlarm, Norton, or the Windows Firewall.  A simple name change would eliminate this confusion to newer users, and some IT guys not familiar with ISA.

If I were to do it over again, I would have given more notice on what changes would be occurring, and why.  I had previous verbal green lights from management to restrict thing things like P2P and IM sessions, and our written IT policies had already reflected these restrictions.  I just never had the capability to do so.  I warned staff, but apparently not enough.  I had to do a healthy amount of explaining, which was fine because I had the technical reasons, and the business case on my side. 

I look forward to the next version of ISA (Threat Management Gateway, or TMG) and the steps it takes to improve upon the Firewall Client component.  Recommended reading on using the Firewall Client in ISA 2004 and 2006 can be found below.

Firewall Client
http://www.isaserver.org/tutorials/Understanding-ISA-Firewall-Client-Part1.html

http://www.isaserver.org/articles/2004firewallclient.html

http://www.isaserver.org/tutorials/Understanding_and_installing_ISA_Firewall_Clients.html

http://www.isaserver.org/tutorials/ISA_Clients__Part_2_SecureNAT_and_Web_Proxy_Client.html

Database of malware domains that can be imported directly into ISA
http://www.malwaredomains.com/

A special thanks to Richard Hicks from Celestix, and my good friend Glenn Barnas from Inno-Tech, who provided invaluable information when I needed it most.

Follow

Get every new post delivered to your Inbox.

Join 869 other followers