Exchange

Exchange 2007 on a VM, and the case of the mysterious Isapi deadlock detected error.

Are you running Exchange 2007 on a VM? Are you experiencing odd warning events in the event log that look something like this?

Event Type: Warning
Event Source: W3SVC-WP
Event Category: None
Event ID: 2262
Date:
Time: 12:28:18 PM
User: N/A
Computer: [yourserver]
Description:
ISAPI ‘c:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected’.

If you’ve answered yes to these questions, you’ve most certainly looked for the fix, and found other users in the same boat. They try and try to fix the issue with adjustments in official documentation or otherwise, with no results.

That was me. …until I ran across this link.

So, as suggested, I added a 2nd vCPU to my exchange server (running on Windows Server 2008 x64, in a vSphere cluster), and started it up. These specific warning messages in my event log went away completely. Okay, after several weeks of monitoring, I may have had a couple of warnings here and there. But that’s it. No longer the hundreds of warnings every day.

As for the official explanation, I don’t have one. Adding vCPU’s to fix problems is not something I want to get in the habit of, but it was an interesting problem, with an interesting solution that was worth sharing.

Helpful links:

Microsoft’s most closely related KB article on the issue (that didn’t fix anything for me):
http://support.microsoft.com/kb/821268

Application pool recycling:
http://technet.microsoft.com/en-us/library/cc735314(WS.10).aspx

Discovering AutoDiscover in Exchange 2007

In my post “Exchange 2007… Better Later than Never” I mentioned one of the post-deployment difficulties I faced was getting the "AutoDiscover” function to behave the way it was designed. For those unfamiliar with the feature, it allows for automated discovery and configuration of various connectivity methods to an Exchange Server. Exchange MAPI clients, Exchange HTTP/RPC clients, and mobile devices using ActiveSync all can use AutoDiscover in some form or another.

While it wasn’t critical for the transition itself, AutoDiscover was vital for our future deployments of “Outlook Anywhere” and “ActiveSync.” I figured skimming over a few TechNet articles and blog postings, and I’d be quickly onto the next project. That began my long ugly journey getting AutoDiscover to work.

It became clear that the ingredients for AutoDiscover to work correctly was a properly configured ISA Server, SSL certificates, namespace/DNS accommodations, and of course, Exchange. What was really interesting about this particular project was that I was dealing with very mature products, yet, I never ran across so much contradicting information on how to make it work. Perhaps some of that stems from so many valid topologies and configurations, or possibly big changes between the RTM versions of Exchange and ISA and their first service packs. Still, it seemed odd. I sifted through postings from desperate IT Administrators in similar situations who had no more hair to pull out. You could sense the defeat in their words. Now I understand.

One guideline mentioned quite often was the need for a special SSL certificate that allowed for more than one FQDN to be assigned to it. You’ll see it referred to as a Unified Communication Certificate (UCC or UC) or a Subject Alternative Name (SAN) certificate. The purpose is the same, but the names and the references are different. While UC certificates are not technically a requirement, it is best to think of it that way. For AutoDiscover, the names needed on a UC cert would look something like:

mypubliccompanyname.com
autodiscover.mypubliccompanyname.com
mail.mypubliccompanyname.com
internalmailservername
internalmailservername.myprivatelanname.lan

I went with a UC cert from DigiCert, but any of the larger commercial CA’s should work. However, a word of warning. Exchange doesn’t like self signed certificates, and many mobile phones have troubles with private certificates as well as those from smaller commercial CAs. You should be fine if you run Certificate Services internally (or so I’m told), and your namespace checks out okay. Don’t forget look at your ISA server and make sure you are running SP1 or later, due to limitations on how the RTM version handled UC certificates.

Speaking of namespaces, time for a thorn in my side to come back and sting me. My internal namespace is not a name that we own (a legacy issue I should have taken care of long ago). Certificate Authorities will not issue standard or UC SSL certificates to names you do not own for obvious reasons, even if the references are private. Fortunately, I was able to work around this by making absolutely sure the simple name was used in any Exchange configuration settings that usually accepted the internal FQDN. Disaster averted.

Now for the dirt on how I was able to make it work. My as-built design is modeled somewhat after Jason Jones’ method of Publishing Exchange 2007 Services with ISA 2006. Following the construct of:

Not using the existing listener created for OWA, and creating a separate listener for Outlook Anywhere (OA)/Autodiscover, and binding UC cert to that listener. Using HTTP authentication with Integrated/Windows Auth (aka NTLM). This would provides HTTP/Integrated auth from the client to the FW, then basic auth from the FW to the Exchange server.
Allowing the ISA server to utilize Kerberos constrained delegation (KCD) by way of changes in AD.
Creating a single Publishing rule for OA , where KCD is used.
Setting internal and external URL’s to their respective internal and external locations (internalmailservername and autodiscover.mycompanyname.com)

After configuring it as above, AutoDiscover worked internally, but not externally. Continually getting failures with the /rpc directory when testing internally (via test-outlookwebservices) and externally (via testexchangeconnecivity.com). I found a post that gave the missing piece of the puzzle, and modified my configuration per the recommendations http://forums.isaserver.org/m_2002041377/mpage_2/key_/tm.htm:

Create a 2nd Publishing rule for OA, sitting on top of primary OA publishing rule.
- Only /rpc/* is published
- Auth Delegation is set to "No Delegation, but client may authenticate directly"
- Set to "all users" instead of "authenticated users"
- Changing "EXPR" Outlookprovider to msstd:mail.mycompanyname.com so that the certificate mutual authentication test passes.

Under the conditions described above, Outlook Anywhere with Autodiscover functions as desired.

As Jason Jones put it best, “The reason for the need of a separate listener is that Windows Authentication (NTLM) and Forms Based Authentication (FBA) are mutually exclusive. It is not possible to use a single web listener for all Exchange 2007 publishing and achieve transparent authentication within Outlook anywhere.” Thus the need to create a dedicated listener to be used exclusively for Outlook Anywhere and associated services.

I did have to make one other adjustment that is rarely brought up in the AutoDiscover deployment scenarios. We know that AutoDiscover wants to look at your TLD name (e.g. yourpubliccompanyname.com) when doing it’s discovery process. However, you may have simply had an “A” record of “yourpubliccompanyname.com” pointing to your web server to catch those users who forget to type in “www” before your domain name. It’s also not a far fetch to assume you had an SSL certificate bound to that web server as well. This is exactly what I had, so I had to make the following changes:

1. Have our ISP (or whomever has authoritative control on the DNS zone file for “mypubliccompanyname.com”) change the “A” record from my public web server IP address, to my autodiscover address.

2. In ISA, add a new “DENY (REDIRECT)” rule for mypubliccompanyname.com that does, well, a deny, and a redirect www.mypubliccompanyname.com. This sits right above the web publishing rule for www.mypubliccompanyname.com

The original setup was a carryover from an earlier time. The configuration above is the way I should have set it up. Nice to do a little cleanup along the way.

I can’t tell you how relieved I was in getting this to work, no matter how many hoops I had to jump through. I also have a complete set of as-built notes in case I need to recreate or debug the existing configuration. It’s been stable since, but I have a feeling I’ll be looking at this again as soon as we transition to Exchange 2010.

Exchange 2007… Better late than never

Count me in as one of the many Administrators that finally got around to moving from Exchange 2003 to Exchange 2007. Yes, it’s almost 2010, and probably just a few months away from the release of Exchange 2010. I never disagreed with Microsoft’s decision to release Exchange 2007 as a 64 bit only application. It’s just that it made it incredibly difficult to find a way to transition when your infrastructure is full of physical 32 bit servers. I was constantly reminded that my Exchange 2003 server was brittle, a resource hog, and lacked many of the abilities that the end users and applications were demanding. Twenty-five minute reboots, and inexplicable behaviors did not instill confidence. It was time to make the move.

My timing couldn’t have been better. I have a new virtualized infrastructure powered by some Dell blades, running VMWare ESX 3.5 using a Dell/ Equallogic PS5000 SAN to help with this transition. It didn’t take away from the normal shrewd planning necessary on a project of this nature. It just made it easier. The other benefit to deploying 3 year old software is that issues, workarounds, and fixes to Exchange, as well as how to make it play nicely with other components (ISA, CRM, etc.) were well documented.

With respect to running Exchange 2007 in a virtual environment, I noticed information on sizing is difficult to find. Capacity planning guidelines still reflected deployments with physical servers (read: more is better), and white papers on virtualizing Exchange appeared to have the intended audience of large enterprise environments only. My environment is relatively small, and a single server was going to be handling all Exchange roles. Based on my environment of about 50 users and 100 mailboxes, I sifted through all the material I could find, threw in a few wild guesses, and decided with the following configuration.

1 vCPU
2.5GB of RAM
Primary OS resides in VMFS volume on the SAN
Using guest OS based iSCSI initiator with MPIO enabled, dedicated NTFS volume for Exchange database.
Using guest OS based iSCSI initiator with MPIO enabled, dedicated NTFS volume for Exchange transaction logs.

The transition occurred over one weekend, while the remainder of the outstanding issues were cleaned up throughout the week. A moment of embarrassment occurred when I realized my thorough planning never took into consideration that the “move mailbox” function would hit the transaction logs so much. I didn’t see it until it was too late. The partition for the transaction logs filled up and the services shut down. But, thanks to the ease of handling storage with the SAN, I was able to create a new LUN, initialize it in the OS, and change a couple of drive letters. Ran a backup to commit the transaction logs to the database, and I was back in business. The only puzzle that took me far too long to figure out was getting the AutoDiscover function in Exchange 2007 to work as intended. It is worthy of it’s own post at another time.

As far as performance goes, I couldn’t be happier with it running as a VM. I truly didn’t know how it would react under the settings I established, and was ready to make whatever changes necessary, but it has been performing very well. Below are some very simple utilization numbers to give you an idea.

CPU: Hovers around 50% utilization during the day, and around 15% utilization during off hours. CPU Ready values range anywhere from 20 to 200 milliseconds. This VM has just 1 vCPU assigned to it. I wanted to keep it this way if possible, so that it could use the “Fault Tolerant” feature of vSphere when we upgrade. Looks like I get my wish.

RAM: Typically runs about 50% of the 2.5GB assigned. No higher than 75% utilization during the busiest time of the day, and 15% utilization during off hours

Network: Runs about 1.2 MBps. Spikes of 40 MBps occur only because of some on-host backups occasionally occurring. Bandwidth utilization is imperceptible during off hours.

Disk I/O: Hard to gauge, but mostly because the need seems to be so low. The OS partition coming from the VMFS volume might show 200KBps, but has bounced up to 7MBps on occasion. No performance data yet on the drives connected via the guest iSCSI initiator. I think they qualify as “fast” until I find anything that suggests otherwise.

Perception: You won’t find this on any ESX or OS performance monitor. But it’s importance cannot be underestimated. I had about half of the staff comment about how much snappier their Outlook clients and OWA was working for them. I never have staff stop by and offer random comments telling me how fast something is. It’s a nice compliment to the improvements of Exchange 2007, and what it is running on.

No more painfully long restart times for me now. It’s one of the more overlooked benefits I’ve noticed while moving my systems over to my new infrastructure. Planned server restarts are a part of responsible management of IT systems. Anything that can be done to reduce any interruption is appreciated. A two minute restart is always welcome

It probably goes without saying that I took great joy decommissioning the old server. Hopefully this will be me in a good position when Exchange 2010 comes off the presses.