Finally. A practical solution to protecting Active Directory

Active Directory.  It is the brains of most modern-day IT infrastructures, providing just about every conceivable control of how users, computers and information will interact with each other.  Authentication, user, group and computer access control, all help provide logical barriers that allow for secure access, but a seamless user experience with single sign-on access to resources.  While it has the ability to improve and integrate critical services such as DNS, DHCP, and NTP, in many ways those services become dependent on Active Directory.  These days, Active Directory controls more than just pure Windows environments.  Integration with non Microsoft Operating systems like Ubuntu, Suse, and VMWare’s vSphere is becoming more common thanks to products such as LikeWise.  The environment that I manage has Windows Servers and clients, most distributions of Linux, Macs, a few flavors of Unix, VMware, and iPhones.  All of them rely on Active Directory.  You quickly learn that if Active Directory goes down, so does your job security.

Active Directory will run happily even under less than ideal circumstances.  It is incredibly resilient, and somehow can put up with server crashes, power outages, and all sorts of debauchery.  But neglect is not a required ingredient for things to go wrong.  When it does, the results can be devastating.  AD problems can be difficult to track down, and it’s tentacles will affect services you never considered.  A corrupt Active Directory, or the Controllers it runs on, can make your Exchange and SQL servers crumble around you.  I lived through this experience (barely) a while back, and even though my preparation for such scenarios looked very good on paper, I spent a healthy amount of time licking my wounds, and reassessing my backup strategy of Active Directory.  I never want to put myself in that position again.

As important as Active Directory is, it can be quiet challenging to protect.  Why?  I believe the answer can be boiled down into two main factors; it’s distributed, and it’s transaction based.  In other words, the two traits that makes it robust also makes it difficult to protect.  Large enterprises usually have a well architected AD infrastructure, and at least understand the complexities of protecting their AD environment.  Many others are left with pondering the various ways to protect it.

• File based backups using traditional backup methods.  This has never been enough, but my bet is that you’d find a number of smaller environments do this – if they do anything at all.  It has worked for them only because they’ve never had a failure of any sort.
• AD backup agents that are a part of a commercial backup application.  Some applications like Symantec Backup Exec (what I previously relied on) seem like a good idea, but show their true colors when you actually try to use it for recovery.  While the agents should be extending the functionality of the backup software, they just add to an already complex solution that feels like a monstrosity geared for other purposes.
• Exporting AD on Windows 2008 based Domain controllers by using NTDSUTIL and the like.  This is difficult at best, arguably incomplete, and if you have a mix of Windows 2008 and Windows 2003 DC’s, won’t work.
• Those who have virtualized their domain controllers often think that the well timed independent snapshot or VCB backup will protect them.  This is not true either.  You will have a VM consistent backup of the VM itself, but it does nothing to coordinate the application with the other Domain Controllers and the integrity of it’s contents.  In theory, they could be backed up properly if every single DC was shut down at the same time, but most of us know that would not be a solution at all.
• Dedicated Solutions exist to protect Active Directory, but can be overly complex, and outrageously expensive.  I’m sure they do their job well, but I couldn’t get the line item past our budget line owner to find out.

The result can be a desire to want to protect AD, but uncertainty on what “protect” really means.  Is protecting the server good enough?  Is protecting AD itself enough?  Does one need both, and if so, how does one go about doing that?  Without fully understanding the answers to those questions, something inevitably goes wrong, and the Administrator is frantically flipping through the latest TechNet Article on Authoritative Restores, while attempting to figure out their backup software.  It’s particularly painful to the Administrator, who had the impression that they were protecting their Organization (and themselves) when in fact, they were not. 

In my opinion, protecting the domain should occur at two different levels.

• Application layer.  This is critical.  Among other things, the backup will coordinate Active Directory so that all of it’s Update Sequence Numbers (USN’s) are at an agreed upon state.  This will avoid USN’s that are out of sync, which can be the trouble of so many AD related problems.  Application layer protection should also honor these AD specific attributes so that granular recovery of individual objects is possible.  Good backup software should leverage API’s that take advantage of Volume Shadow Copy Services (VSS).
• Physical layer.  This protects the system that the services may be running on.  If it’s a physical server, it could be using some disk imaging software such as Acronis, or Backup Exec System Recovery.  If it’s virtualized, an independent backup of the VM will do.  Some might suggest that protecting the actual machine isn’t technically required.  The idea behind that reasoning is that if there is a problem with the physical machine, or the OS, one can quickly decommission and commission another DC with “dcpromo.”  While protecting the system that AD runs on may not be required, it may help speed up your ability (in conjunction with Application layer protection) to correct issues from a previously known working state.

I was introduced to CionSystems by a colleague of mine who suggested their “Active Directory Self-Service” product to help us with another need of ours.  Along the way, I couldn’t help but notice their AD backup offering.  Aptly named, “Active Directory Recovery” is a complete application layer solution.  I tried it out, and was sold.  It allows for a simple, coordinated backup and recovery of Active Directory.  A recovery can be either a complete point-in-time, or a granular restore of an object.  It is agentless, meaning that you don’t have to install software on the DCs.  The first impression after working with it is that it was designed for one purpose; to backup Active Directory.  It does it, and does it well.

The solution will run on any spare machine running IIS and SQL.  Once installed, configuring it is just a matter of pointing it to your Domain Controller that runs the PDC Emulator role.  After a few configuration entries are made, the Administration console can be accessed with your web browser from anywhere on your network.

The next step is to set up a backup job, and let it run.  That’s it.  Fast, simple, and complete.  From the home page, there are a few different ways you can look at objects that you want to recover.

If it’s a deleted object, you can click on the “Deleted Objects” section.  Objects with a backup to restore from will show up in green, and present itself below each object.  Below you will see a deleted computer object, and the backups that it can be restored from.

The “List Backups” simply shows the backups created in chronological order.  From there you can do full restores, or restore an individual object that still exists in AD.  Unlike authoritative restores, you do not have to do any system restarts.

During the restore process, “Active Directory Recovery” will expose individual attributes of the object that you want to restore – if you wish for the restore to be that granular.  If it’s restorable, there is a checkbox next to it.  Non-modifiable objects will not have a checkbox next to it.

One of my favorite features is that it provides a way for a true, portable backup.  One can export the backup to a single file (a proprietary .bin file) that is your entire AD backup, and save it onto a CD, or to a remote location.  This is a wish list item I’ve had for about as long as AD has been around.    There are many other nice features, such as email notifications, filtering and comparison tools, as well as backup retention settings. 

I use this product to compliment my existing strategy for protecting my AD infrastructure.  While my virtualized Domain Controllers are replicated to a remote site (the physical protection, so to speak), I protect my AD environment at the application level with this product.  The server that “Active Directory Recovery” runs on is also replicated, but to be extra safe, I create a portable/exported backup that is also shipped off to the offsite location.  This way I have a fully independent backup of AD.  If I’m doing some critical updates to my Domain Controllers, I first make a backup using Active Directory Recovery, then make my snapshots on my virtualized DC’s  That way, I have a way to roll back the changes that are truly application consistent.

After using the product for a while, I can appreciate that I don’t have to invest much time to keep my backups up and running.  I previously used Symantec’s Backup Exec to protect AD, but grew tired of agent issues, licensing problems, and the endless backup failure messages.  I lost confidence in its ability to protect AD, and am not interested in going back. 

Hopefully this gives you a little food for thought on how you are protecting your Active Directory environment.  Good luck!