How I use Dell/EqualLogic’s SANHQ in my environment


One of the benefits of investing in Dell/EqualLogic’s SAN solutions are the number of great tools included with the product, at no extra charge.  I’ve written in the past about leveraging their AutoSnapshot Manager for VM and application consistent snapshots and replicas.  Another tool that deserves a few words is SAN HeadQuarters (SANHQ). 

SANHQ allows for real-time and historical analysis of your EqualLogic arrays.  Many EqualLogic users are well versed with this tool, and may not find anything here that they didn’t already know.  But I’m surprised to hear that many are not.  So, what better way to help those unfamiliar with SANHQ than to describe how it helps me with my environment.

While the tool itself is “optional” in the sense that you don’t need to deploy it to use the EqualLogic arrays, it is an easy (and free) way to expose the powers of your storage infrastructure.  If you want to see what your storage infrastructure is doing, do yourself a favor and run SANHQ.   

Starting up the application, you might find something like this:


You’ll find an interesting assortment of graphs, and charts that help you decipher what is going on with your storage.  Take a few minutes and do a little digging.  There are various ways that it can help you do your job better.



Sometimes good monitoring is downright annoying.  It’s like your alarm clock next to the bed; it’s difficult to overlook, but that’s the point.  SANHQ has proven to be an effective tool for proactive monitoring and alerting of my arrays.  While some of these warnings are never fun, it’s biggest value is that it can help prevent those larger, much more serious problems, which always seem to be a series of small issues thrown together.  Here are some examples of how it has acted as the canary in the coalmine for me in my environment.

  • When I had a high number of TCP retransmits after changing out my SAN Switchgear, it was SANHQ that told me something was wrong.  EqualLogic Support helped me determine that my new switchgear wasn’t handling jumbo frames correctly. 
  • When I had a network port go down on the SAN, it was SANHQ that alerted me via email.  A replacement network cable fixed the problem, and the alarm went away.
  • If replication across groups is unable to occur, you’ll get notified right away that replication isn’t running.  The reasons for this can be many, but SANHQ usually gives you the first sign that something is up.  This works across physical topologies where your target my be at another site.
  • Under maintenance scenarios, you might find the need to pause replication on a volume, or on the entire group.  SANHQ will do a nice job of reminding you that it’s still not replicating, and will bug you at a regular interval that it’s still not running.


Analysis and Planning

SANHQ will allow you to see performance data at the group level, by storage pools, volumes, or volume collections.  One of the first things I do when spinning up a VM that uses guest attached volumes, is to jump into SANHQ, and see how those guest attached volumes are running.  How are the average IOPS? What about Latencies and Queue depth?  All of those can be found  easily in SANHQ, and can help put your mind at ease if you are concerned about your new virtualized Exchange or SQL servers.  Here is a screenshot of a 7 day history for SQL server with guest attached volumes, driving our SharePoint backend services.


The same can be done of course for VMFS volumes.  This information will compliment existing data one gathers from vCenter to understand if there are performance issues with a particular VMFS volume.

Often times monitoring and analysis isn’t about absolute numbers, but rather, allowing the user to see changes relative to previous conditions.  This is especially important for the IT generalist who doesn’t have time or the know-how for deep dive storage analysis, or have a dedicated Storage Administrator to analyze the data.  This is where the tool really shines.  For whatever type of data you are looking at, you can easily choose a timeline by the last hour, 8 hours, 1 day, 7 days, 30 days, etc.  The anomalies, if there are any, will stand out. 


Simply click on the Timeline that you want, and the historical data of the Group, member, volume, etc will show up below.


I find analyzing individual volumes (when they are VMFS volumes) and volume collections (when they are guest attached volumes) the most helpful in making sure that there are not any hotspots in I/O.  It can help you determine if a VM might be better served in a VMFS volume that hasn’t been demanding as much I/O as the one it’s currently in.

It can also play a role in future procurement.  Those 15k SAS drives may sound like a neat idea, but does your environment really need that when you decide to add storage?  Thinking about VDI?  It can be used to help determine I/O requirements.  Recently, I was on the phone with a friend of mine, Tim Antonowicz.  Tim is a Senior Solutions Architect from Mosaic Technology who has done a number of successful VDI deployments (and who recently started a new blog).  We were discussing the possibility of VDI in my environment, and one of the first things he asked of me was to pull various reports from SANHQ so that he could understand our existing I/O patterns.  It wasn’t until then that I noticed all of the great storage analysis offerings that any geek would love.  There are a number of canned reports that can be saved out as a pdf, html, csv, or other format to your liking.


Replication Monitoring

The value of SANHQ went way up for me when I started replication.  It will give you summaries of the each volume replicated.


If you click on an individual volume, it will help you see transfer sizes and replication times of the most recent replicas.  It also separates inbound replica data from outbound replica data.


While the times and the transfer rates will be skewed somewhat if you have multiple replica’s running (as I do), it is a great example on how you can understand patterns in changed data on a specific volume.  The volume captured above represents where one of my Domain Controllers lives.  As you can see, it’s pretty consistent, and doesn’t change much, as one would expect (probably not much more than the swap file inside the VM, but that’s another story).  Other kinds of data replicated will fluctuate more widely.  This is your way to see it.


Running SANHQ

SANHQ will live happily on a stand alone VM.  It doesn’t require much, but does need direct access to your SAN, and uses SNMP.  Once installed, SANHQ can be run directly on that VM, or the client-only application can be installed on your workstation for a little more convenience.  If you are replicating data, you will want SANHQ to connect to the source site, and the target site, for most effective use of the tool.

Improvements?  Sure, there are a number of things that I’d love to see.  Setting alarms for performance thresholds.  Threshold templates that you could apply to a volume (VMFS or native) that would help you understand the numbers (green = good.  Red = bad).  The ability to schedule reports, and define how and where they are posted.  Free pool space activity warnings (important if you choose to keep replica reserves low and leverage free pool space).  Array diagnostics dumps directly from SANHQ.  Programmatic access for scripting.  Improvements like these could make a useful product become indispensible in a production environment.

Replication with an EqualLogic SAN; Part 5


Well, I’m happy to say that replication to my offsite facility is finally up and running now.  Let me share with you the final steps to get this project wrapped up. 

You might recall that in my previous offsite replication posts, I had a few extra challenges.  We were a single site organization, so in order to get replication up and running, an infrastructure at a second site needed to be designed and in place.  My topology still reflects what I described in the first installment, but simple pictures don’t describe the work getting this set up.  It was certainly a good exercise in keeping my networking skills sharp.  My appreciation for the folks who specialize in complex network configurations, and address management has been renewed.  They probably seldom hear words of thanks for say, that well designed sub netting strategy.  They are an underappreciated bunch for sure.

My replication has been running for some time now, but this was all within the same internal SAN network.  While other projects prevented me from completing this sooner, it gave me a good opportunity to observe how replication works.

Here is the way my topology looks fully deployed.


Most Collocations or Datacenters give you about 2 square feet to move around, (only a slight exaggeration on the truth) so it’s not the place you want to be contemplating reasons why something isn’t working.  It’s also no fun realizing you don’t have the remote access you need to make the necessary modifications, and you don’t, or can’t drive to the CoLo.  My plan for getting this second site running was simple.  Build up everything locally (switchgear, firewalls, SAN, etc.) and change my topology at my primary site to emulate my the 2nd site.

Here is the way it was running while I worked out the kinks.


All replication traffic occurs over TCP port 3260.  Both locations had to have accommodations for this.  I also had to ensure I could manage the array living offsite.  Testing this out with the modified infrastructure at my primary site allowed me to verify traffic was flowing correctly.

The steps taken to get two SAN replication partners transitioned from a single network to two networks (onsite) were:

  1. Verify that all replication is running correctly when the two replication partners are in the same SAN Network
  2. You will need a way to split the feed from your ISP, so if you don’t have one already, place a temporary switch at the primary site on the outside of your existing firewall.  This will allow you to emulate the physical topology of the real site, while having the convenience of all of the equipment located at the primary site. 
  3. After the 2nd firewall (destined for the CoLo) is built and configured, place it on that temporary switch at the primary site.
  4. Place something (a spare computer perhaps) on the SAN segment of the 2nd firewall so you can test basic connectivity (to ensure routing is functioning, etc) between the two SAN networks. 
  5. Pause replication on both ends, take the target array and it’s switchgear offline. 
  6. Plug the target array’s Ethernet ports to the SAN switchgear for the second site, then change the IP addressing of the array/group so that it’s running under the correct net block.
  7. Re-enable replication and run test replicas.  Starting out with the Group Manager.  Then to ASM/VE, then onto ASM/ME.

It would be crazy not to take one step at a time on this, as you learn a little on each step, and can identify issues more easily.  Step 3 introduced the most problems, because traffic has to traverse routers that also are secure gateways.  Not only does one have to consider a couple of firewalls, you now run into other considerations that may be undocumented.  For instance.

  • ASM/VE replication occurs courtesy of vCenter.  But ASM/ME replication is configured inside the VM.  Sure, it’s obvious, but so obvious it’s easy to overlook.  That means any topology changes will require adjustments in each VM that utilize guest attached volumes.  You will need to re-run the “Remote Setup Wizard” to adjust the IP address of the target group that you will be replicating to.
  • ASM/ME also uses a VSS control channel to talk with the array.  If you changed the target array’s group and interface IP addresses, you will probably need to adjust what IP range will be allowed for VSS control.
  • Not so fast though.  VM’s that use guest iSCSI initiated volumes typically have those iSCSi dedicated virtual network cards set with no default gateway.  You never want to enter more than one default gateway on this sort of situation.  The proper way to do this will be to add a persistent static route.  This needs to be done before you run the remote Setup Wizard above.  Fortunately the method to do this hasn’t changed for at least a decade.  Just type in

route –p add [destinationnetwork] [subnetmask] [gateway] [metric]

  • Certain kinds of traffic that passes almost without a trace across a layer 2 segment shows up right away when being pushed through very sophisticated firewalls who’s default stances are deny all unless explicitly allowed.  Fortunately, Dell puts out a nice document on their EqualLogic arrays.
  • If possible, it will be easiest to configure your firewalls with route relationships between the source SAN and the target SAN.  It may complicate your rulesets (NAT relationships are a little more intelligent when it comes to rulesets in TMG), but it simplifies how each node is seeing each other.  This is not to say that NAT won’t work, but it might introduce some issues that wouldn’t be documented.

Step 7 exposed an unexpected issue; terribly slow replicas.  Slow even though it wasn’t even going across a WAN link.  We’re talking VERY slow, as in 1/300th the speed I was expecting.  The good news is that this problem had nothing to do with the EqualLogic arrays.  It was an upstream switch that I was using to split my feed from my ISP.  The temporary switch was not negotiating correctly, and causing packet fragmentation.  Once that switch was replaced, all was good.

The other strange issue was that even though replication was running great in this test environment, I was getting errors with VSS.  ASM/ME at startup would indicate “No control volume detected.”  Even though replicas were running, the replica’s can’t be accessed, used, or managed in any way.  After a significant amount of experimentation, I eventually opened up a case with Dell Support.  Running out of time to troubleshoot, I decided to move the equipment offsite so that I could meet my deadline.  Well, when I came back to the office, VSS control magically worked.  I suspect that the array simply needed to be restarted after I had changed the IP addressing assigned to it. 

My CoLo facility is an impressive site.  Located in the Westin Building in Seattle, it is also where the Seattle Internet Exchange (SIX) is located.  Some might think of it as another insignificant building in Seattle’s skyline, but it plays an important part in efficient peering for major Service Providers.  Much of the building has been converted from a hotel to a top tier, highly secure datacenter and a location in which ISP’s get to bridge over to other ISP’s without hitting the backbone.  Dedicated water and power supplies, full facility fail-over, and elevator shafts that have been remodeled to provide nothing but risers for all of the cabling.  Having a CoLo facility that is also an Internet Exchange Point for your ISP is a nice combination.

Since I emulated the offsite topology internally, I was able to simply plug in the equipment, and turn it on, with the confidence that it will work.  It did. 

My early measurements on my feed to the CoLo are quite good.  Since the replication times include buildup and teardown of the sessions, one might get a more accurate measurement on sustained throughput on larger replicas.  The early numbers show that my 30mbps circuit is translating to replication rates that range in the neighborhood of 10 to 12GB per hour (205MB per min, or 3.4MB per sec.).  If multiple jobs are running at the same time, the rate will be affected by the other replication jobs, but the overall throughput appears to be about the same.  Also affecting speeds will be other traffic coming to and from our site.

There is still a bit of work to do.  I will monitor the resources, and tweak the scheduling to minimize the overlap on the replication jobs.  In past posts, I’ve mentioned that I’ve been considering the idea of separating the guest OS swap files from the VM’s, in an effort to reduce the replication size.  Apparently I’m not the only one thinking about this, as I stumbled upon this article.  It’s interesting, but a nice amount of work.  Not sure if I want to go down that road yet.

I hope this series helped someone with their plans to deploy replication.  Not only was it fun, but it is a relief to know that my data, and the VM’s that serve up that data, are being automatically replicated to an offsite location.

Replication with an EqualLogic SAN; Part 3


In parts one and two of my journey in deploying replication between two EqualLogic PS arrays, I described some of the factors that came into play on how my topology would be designed, and the preparation that needed to occur to get to the point of testing the replication functions. 

Since my primary objective of this project was to provide offsite protection of my VMs and data in the event of a disaster at my primary facility,  I’ve limited my tests to validating that the data is recoverable from or at the remote site.   The logistics of failing over to a remote site (via tools like Site Recovery Manager) is way outside the scope of what I’m attempting to accomplish right now.  That will certainly be a fun project to work on some day, but for now, I’ll be content with knowing my data is replicating offsite successfully.

With that out of the way, let the testing begin…


Replication using Group Manager 

Just like snapshots, replication using the EqualLogic Group Manager is pretty straight forward.  However, in my case, using this mechanism would not produce snapshots or replicas that are file-system consistent of VM datastores, and would only be reliable for data that was not being accessed, or VM’s that were turned off.  So for the sake of brevity, I’m going to skip these tests.


ASM/ME Replica creation.

My ASM/ME replication tests will simulate how I plan on replicating the guest attached volumes within VMs.  Remember, these are replicas of the guest attached volumes  only – not of the VM. 

On each VM where I have guest attached volumes and the HITKit installed (Exchange, SQL, file servers, etc.) I launched ASM/ME to configure and create the new replicas.  I’ve scheduled them to occur at a time separate from the daily snapshots.


As you can see, there are two different icons used; one represents snapshots, and the other representing replicas.  Each snapshot and replica will show that the guest attached volumes (in this case, “E:\” and “F:\” )  have been protected using the Exchange VSS writer.  The two drives are being captured because I created the job from a “Collection” which makes most sense for Exchange and SQL systems that have DB files and transaction log data that you’d want to capture at the exact same time.  For the time being, I’m just letting them run once a day to collect some data on replication sizes.  ASM/ME is where recovery tasks would be performed on the guest attached volumes.

A tip for those who are running ASM/ME for Smartcopy snapshots or replication.  Define in your schedules a “keep count” number of snapshots or replicas that fall within the amount of snapshot reserve you have for that volume.  Otherwise, ASM/ME may take a very long time to start  the console and reconcile the existing smart copies, and you will also find those old snapshots in the “broken” container of ASM/ME.    The startup delay can be so long, it almost looks as if the application has hung, but it has not, so be patient.  (By the way, ASM/VE version 2.0, which should be used to protect your VMs, does not have any sort of “keep count” mechanism.  Lets keep our fingers crossed for that feature in version 3.0)


ASM/ME Replica restores

Working with replicas using ASM/ME is about as easy as it gets.  Just highlight the replica, and click on “Mount as read-only.”  Unlike a snapshot, you do not have the option to “restore” over the existing volume when its a replica.


ASM/ME will ask for a drive letter to assign that cloned replica to.  Once it’s mounted, you may do with the data as you wish.  Note that it will be in a read only state.  This can be changed later if needed.

When you are finished with the replica, you can click on the “Unmount and Resume Replication…”


ASM/ME will ask you if you want to keep the replica around after you unmount it.  To keep it, uncheck the box next to “Delete snapshot from the PS Series group…”


ASM/VE replica creation

ASM/VE replication, which will be the tool I use to protect my VMs, took a bit more time to set up correctly due to the way that ASM/VE likes to work.  I somehow missed the fact that one needed a second ASM/VE server to run at the target/offsite location for the ASM/VE server at the primary site to communicate with.  ASM/VE also seems to be hyper-sensitive to the version of Java installed on the ASM/VE servers.  Don’t get too anxious on updating to the latest version of Java.   Stick with a version recommended by EqualLogic.  I’m not sure what that officially would be, but I have been told by Tech Support that version 1.6 Update 18 is safe.

Unlike creating Smartcopy snapshots in ASM/VE, you cannot use the “Virtual Machines” view in ASM/VE to create Smartcopy replicas.  Only Datastores, Datacenters, and Clusters support replicas.  In my case, I will click  “Datastores” view to create Replicas.  Since I made the adjustments to where my VM’s were placed in the datastores, (see part 2, under “Preparing VMs for Replication”) it will still be clear as to which VMs will be replicated. 


After creating a Smartcopy replica of one of the datastores, I went to see how it looked.  In ASM/VE it appeared to complete successfully, and in SANHQ it also seemed to indicate a successful replica.  ASM/VE then gave a message of “contacting ASM peer” in the “replica status” column.  I’ve seen this occur right after I kicked off a replication job, but on successful jobs, it will disappear shortly.  If it doesn’t disappear, this can be a configuration issue (user accounts used to establish the connection due to known issues with ASM/VE 2.0), or caused by Java.


ASM/VE replica restores

At first, ASM/VE Smartcopy replicas didn’t make much sense to me, especially when it came to restores.  Perhaps I was attempting to think of them as a long distance snapshot, or that they might behave in the same way as ASM/ME replicas.  They work a bit  differently than that.  It’s not complicated, just different.

To work with the Smartcopy replica, you must first log into the ASM/VE server at the remote site.  From there, click on “Replication” > “Inbound Replicas” highlighting the replica from the datastore you are interested in.  Then it will present you with the options of “Failover from replica” and “clone from replica”  If you attempt to do this from the ASM/VE server from the primary site, these options never present themselves.  It makes sense to me after the fact, but took me a few tries to figure that out.  For my testing purposes, I’m focusing exclusively on “clone from replica.”  The EqualLogic documentation has good information on when each option can be used.

When choosing “Clone from Replica” it will have a checkbox for “Register new virtual machines.”  In my case, I uncheck this box, as my remote site will have just a few hosts running ESXi, and will not have a vCenter server to contact.



Once it is complete, access will need to be granted for the remote host in which you will want to try to mount the volume.  This can be accomplished by logging into the Group Manager of the target/offsite SAN group, selecting the cloned volume, and entering CHAP credentials, the IP address of the remote host, or the iSCSI initiator name. 



Jump right on over to the vSphere client for the remote host, and under “Configuration” > “Storage Adapters”  right click on your iSCSI software adapter, and select “Rescan”  When complete, go to “Configuration” > “Storage” and you will notice that it the volume does NOT show up.  Click “Add Storage” > “Disk/LUN”



When a datastore is recognized as a snapshot, it will present you with the following options.  See for more information on which option to choose.



Once completed, the datastore that was replicated to the remote site and cloned so that it can be made available to the remote ESX/i host, should now be visible in “Datastores.” 


From there just browse the Datastore, drilling down to the folder of the VM you wish to turn up, highlight and right click the .vmx file, and select “Add to inventory.”  Your replicated VM should now be ready for you to power up.

If you are going to be cloning a VM replica living on the target array to a datastore, you will need to do one additional step if any of the VM’s have guest attached volumes using the guest iSCSI initiator.  At the target location, open up Group Manager, and drill down to “Replication Partners” > “[partnername]” and highlight the “Inbound” tab.  Expand the volume(s) that are associated with that VM.  Highlight the replica that you want, then click on “Clone replica”


This will allow you to reattach a guest attached volume to that VM.  Remember that I’m using the cloning feature simply to verify that my VM’s and data are replicating as they should.  Turning up systems for offsite use is a completely different ballgame, and not my goal – for right now anyway.

Depending on how you have your security and topology set up, and how connected your ESX host is offsite, your test VM you just turned up at the remote site may have the ability to contact Active Directory at your primary site, or guest attached volumes at your primary site.  This can cause problems for obvious reasons, so be careful to not let either one of those happen.  



While demonstrating some of these capabilities recently to the company, the audience (Developers, Managers, etc.) was very impressed with the demonstration, but their questions reminded me of just how little they understood the new model of virtualization, and shared storage.  This can be especially frustrating for Software Developers, who generally consider that there isn’t anything in IT that they don’t understand or know about.  They walked away impressed, and confused.  Mission accomplished.

Now that I’ve confirmed that my data and VM’s are replicating correctly, I’ll be building up some of my physical topology so that the offsite equipment has something to hook up to.  That will give me a chance to collect some some statistics on replication, which I will share on the next post.

Replication with an EqualLogic SAN; Part 2


In part 1 of this series, I outlined the decisions made in order to build a replicated environment.  On to the next step.  Racking up the equipment, migrating my data, and laying some groundwork for testing replication.

While waiting for the new equipment to arrive, I wanted to take care of a few things first:

1.  Update my existing PS5000E array up to the latest firmware.  This has never been a problem, other than the times that I’ve forgotten to log in as the default  ‘grpadmin’ account (the only account allowed to do firmware updates).  The process is slick, with no perceived interruption.

2.  Map out how my connections should be hooked up on the switches.  Redundant switches can only be redundant if you plug everything in the correct way.

3.  IP addressing.  It’s all too easy just to randomly assign IP addresses to a SAN.  It may be it’s own isolated network, but in the spirit of “design as if you know its going to change”  it might just be worth observing good addressing practices.  My SAN is on a /24 net block.  But I configure my IP addresses to respect potential address boundaries within that address range.  This is so that I can subnet or VLAN them down (e.g. /28)  later on, as well as helping to simplify rule sets on my ISA server that are based on address boundaries, and not a scattering of addresses.

Preparing the new array

Once the equipment arrived, it made most sense to get the latest firmware on the new array.  The quickest way is to set it up temporarily using the “initialize PS series  array” feature in the “Remote Setup Wizard” of the EqualLogic HITKit on a machine that can access the array.  Make it it’s own group, update the firmware, then reset the array to the factory defaults.  After completing the update and  typing “reset”  up comes the most interesting confirmation prompt you’ll ever see.  Instead of “Reset this array to factory defaults?”  [Y/N]”  where a “Y” or “N” is required, the prompt is “Reset this array to factory defaults? [n/DeleteAllMyDataNow]”  You can’t say that isn’t clear.  I applaud EqualLogic for making this very clear.  Wiping a SAN array clean is serious stuff, and definitely should be harder than typing a “Y” after the word “reset.” 

After the unit was reset, I was ready to join it to the existing group temporarily so that I could evacuate all of the data from the old array, and have it placed on the new array.  I plugged all of the array ports into the SAN switches, and turned it on.  Using the Remote Setup Wizard, I initialized the array, joined it to the group, then assigned and activated the rest of the NICs.   To migrate all of the data from one array to another, highlight the member with the data on it, then  click on “Delete Member”  Perhaps EqualLogic will revisit this term.  “Delete” just implies way too many things that doesn’t relate to this task.

The process of migrating data chugs along nicely.  VM’s and end users are none-the-wiser.  Once it is complete, the old array will remove itself from the group, and reset itself to the factory defaults.  It’s really impressive.  Actually, the speed and simplicity of the process gave me confidence when we need to add additional storage.

When the old array was back to it’s factory defaults,  I went back to initialize the array, and set it up as a new member in a new group.  This would be my new group that would be used for some preliminary replication testing, and will eventually live at the offsite location.

As for how this process compares with competing products, I’m the wrong guy to ask.  I’ve had zero experience with Fiber Channel SANs, and iSCSI SANs from other vendors.  But what I can say is that it was easy, and fast.

After configuring the replication between the two group, which consisted of configuring a few shared passwords between the the two groups, and configuring replication to occur on each volume, I was ready to try it out  …Almost.


Snapshots, and replication.

It’s worth taking a step back to review a few things on snapshots and how the EqualLogic handles them.  Replicas appear to work in a similar (but not exact) manner to snapshots, so many of the same principals apply.  Remember that snapshots can be made in several ways.

1.  The most basic are snapshots created in the EqualLogic group Manager.  These do exactly as they say, making a snapshot of the volume.  The problem is that they are not file-system consistent of VM datastores, and would only  be suitable for datastores in which all of the VM’s were turned off at the time the snapshot was made.

2.  To protect VM’s, “Autosnapshot manager VMware Edition” (ASM/VE) provides and ability to create a point-in-time snapshot, leveraging vCenter through VMware’s API, then does some nice tricks to make this an independent snapshot (well, of the datastore anyway) that you see in the EqualLogic group manager, under each respective volume.

3.  For VM’s with guest iscsi attached drives, there is “Autosnapshot Manager Microsoft Edition” (ASM/ME).  This great tool is installed with the Host Integration Toolkit (HITkit).  This makes application aware snapshots by taking advantage of the Microsoft Volume Shadow Copy Service Provider.  This is key for protecting SQL databases, Exchange databases, and even flat-file storage residing on guest attached drives.  It insures that all I/O is flushed when the snapshot is created.  I’ve grown quite partial to this type of snapshot, as its nearly instant, no interruption to the end users or services, and provides easy recoverability.  The downside is that it can only protect data on iscsi attached drives within the VM’s guest iscsi initiator, and must have a VSS writer specific to an application (e.g. Exchange, SQL) in order for it to talk correctly.  You cannot protect the VM itself with this type of snapshot.  Also, vCenter is generally unaware of these types of guest attached drives, so VCB backups and other apps that rely on vCenter won’t include these types of volumes.

So just as I use ASM/ME for smartcopy snapshots of my guest attached drives, and ASM/VE for my VM snapshots, I will use these tools in the similar way to create VM and application aware replica’s of the VM’s and the data.

ASM/VE tip:  Smartcopy snapshots using ASM/VE give the option to “Include PS series volumes accessed by guest iSCSI initiators.”  I do not use this option for a few very good reasons, and rely completely on ASM/ME for properly capturing guest attached volumes. 

Default replication settings in EqualLogic Group Manager

When one first configures a volume for replication, some of the EqualLogic defaults are set very generous.  The two settings to look out for are the “Total replica reserve” and the “Local replication reserve.”  The result is that these very conservative settings can chew up a lot of your free space on your SAN.  Assuming you have a decent amount of free space in your storage pool, and you choose to stagger some of your replication to occur at various times of the day, you can reduce the “Local replication reserve” down to it’s minimum, then click the checkbox for “allow temporary use of free pool space.”  This will minimize the impact of enabling replication on your array.


Preparing VM’s for replication

There were a few things I needed to do to prepare my VM’s to be replicated.  I wasn’t going to tackle all optimization techniques at this time, but thought it be best to get some of the easy things out of the way first.

1.  Reconfigure VM’s so that swap file is NOT in the same directory as the other VM files.  (This is the swap file for the VM at the hypervisor level; not to be confused with the guest OS swap file.)  First I created a volume in the EqualLogic group manager that would be dedicated for VM swap files, then made sure it was visible to each ESX host.  Then, simply configure the swap location at the cluster level in vCenter, followed by changing the setting on each ESX host.  The final step will be to power off and power on of each VM.  (A restart/reboot will not work for this step).  Once this is completed, you’ve eliminated a sizeable amount of data that doesn’t need to be replicated.

2.  Revamp datastores to reflect good practices with ASM/VE.  (I’d say “best practices” but I’m not sure if they exist, or if these qualify as such).  This is a step that takes into consideration how ASM/VE works, and how I use ASM/VE.   I’ve chosen to make my datastores reflect how my VM’s are arranged in vCenter.    Below is a screenshot in vCenter of the folders that contain all of my VMs.


Each folder has VMs in it that reside in just one particular datastore.  So for instance, the “Prodsystems-Dev” has a half dozen VM’s exclusively for our Development team.  These all reside in one datastore called VMFS05DS.  When a scheduled snapshot of a vcenter folder (e.g. “Prodsystems-Dev”) using ASM/VE, it will only hit those VM’s in that vcenter folder, and the single datastore that they reside on.  If it is not done this way, an ASM/VE snapshot of a folder containing VM’s that reside in different datastores will generate snapshots in each datastore.  This becomes terribly confusing to administer, especially when trying to recover a VM.

Since I recreated many of my volumes and datastores, I also jumped on the opportunity to make these new datastores with a 4MB block size instead of the the default 1MB block size.  Not really necessary in my situation, but based on the link here, it seems like a a good idea.

Once the volumes and the datastores were created and sized the way I desired, I used the storage vmotion function in vCenter to move each VM into the appropriate datastore to mimic my arrangement of folders in vCenter.  Because I’m sizing my datastores for a functional purpose, I have a mix of large and small datastores.  I probably would have made these the same size if it weren’t for how ASM/VE works.

The datastores are in place, and now mimic the arrangement of folders of VM’s in vCenter.  Now I’m ready to do a little test replication.  I’ll save that for the next post.

Suggested reading

Michael Ellerbeck has some great posts on his experiences with EqualLogic, replication, Dell switches, and optimization.    A lot of good links within the posts.

The Dell/EqualLogic Document Center has some good overview documents on how these components work together.  Lots of pretty pictures.

Replication with an EqualLogic SAN; Part 1


Behind every great virtualized infrastructure is a great SAN to serve everything up.  I’ve had the opportunity to work with the Dell/EqualLogic iSCSI array for a while now, taking advantage of all of the benefits that the iSCSI based SAN array offers.  One feature that I haven’t been able to use is the built in replication feature.  Why?  I only had one array, and I didn’t have a location offsite to replicate to.

I suppose the real “part 1” of my replication project was selling the idea to the Management Team.  When it came to protecting our data and the systems that help generate that data, it didn’t take long for them to realize it wasn’t a matter of what we could afford, but how much we could afford to lose.  Having a building less than a mile away burn to the ground also helped the proposal.  On to the fun part; figuring out how to make all of this stuff work.

Of the many forms of replication out there, the most obvious one for me to start with is native SAN to SAN replication.  Why?  Well, it’s built right into the EqualLogic PS arrays, with no additional components to purchase, or license keys or fees to unlock features.  Other solutions exist, but it was best for me to start with the one I already had.

For companies with multiple sites, replication using EqualLogic arrays seems pretty straight forward.  For a company with nothing more than a single site, there are a few more steps that need to occur before the chance to start replicating data can happen.


Decision:  Colocation, or hosting provider

One of the first decisions that had to be made was if we wanted our data to be replicated to a Colocation (CoLo) with equipment that we owned and controlled, or with a hosting provider that can provide native PS array space and replication abilities.  Most hosting providers use a mixed variety of metering of data replicated to charge.  Accurately estimating your replication costs assumes you have a really good understanding of how much data will be replicated.  Unfortunately, this is difficult to know until you start replicating.  The pricing models of these hosting providers reminded me too much of a cab fare; never knowing what you are going to pay until you get the big bill when you are finished.    A CoLo with equipment that we owned fit with our current and future objectives much better.  We wanted fixed costs, and the ability to eventually do some hosting of critical services at the CoLo (web, ftp, mail relay, etc.), so it was an easy decision for us.

Our decision was to go with a CoLo facility located in the Westin Building in downtown Seattle.  Commonly known as the Seattle Internet Exchange (SIX), this is an impressive facility not only in it’s physical infrastructure, but how it provides peered interconnects directly from one ISP to another.  Our ISP uses this facility, so it worked out well to have our CoLo there as well


Decision:  Bandwidth

Bandwidth requirements for our replication was, and is still unknown, but I knew our bonded T1’s probably weren’t going to be enough, so I started exploring other options for higher speed access.  The first thing to check was to see if we qualified for a Metro-E or “Ethernet over Copper” (award winner for the dumbest name ever).  Metro-E removes the element of T-carrier lines along with any proprietary signaling, and provides internet access of point-to-point connections at Layer 2, instead of Layer 3.  We were not close enough to the carriers central office to get adequate bandwidth, and even if we were, it probably wouldn’t scale up to our future needs.

Enter QMOE, or Qwest Metro Optical Ethernet.  This solution feeds Layer 2 Ethernet to our building via fiber, offering the benefit of high bandwidth, low latency, that can be scaled easily.

Our first foray using QMOE is running a 30mbps point-to-point feed to our CoLo, and uplinked to the Internet.  If we need more later, there is no need to add or change equipment.  Just have them turn up the dial, and bill you accordingly.


Decision:  Topology

Topology planning has been interesting to say the least.  The best decision here depends on the use-case, and lets not forget, what’s left in the budget. 

Two options immediately presented themselves.

1.  Replication data from our internal SAN would be routed (Layer 3) to the SAN at the CoLo.

2.  Replication data  from our internal SAN would travel by way of a VLAN to the SAN at the CoLo.

If my need was only to send replication data to the CoLo, one could take advantage of that layer 2 connection, and send replication data directly to the CoLo, without it being routed.  This would mean that it would have to bypass any routers/firewalls in place, and have to be running to the CoLo on it’s own VLAN.

The QMOE network is built off of Cisco Equipment, so in order to utilize any VLANing from the CoLo to the primary facility, you must have Cisco switches that will support their VLAN trunking protocol (VTP).  I don’t have the proper equipment for that right now.

In my case, here is a very simplified illustration as to how the two topologies would look:

Routed Topology



Topology using VLANs


One may introduce more overhead and less effective throughput when the traffic becomes routed.  This is where a WAN optimization solution could come into play.  These solutions (SilverPeak, Riverbed, etc.) appear to be extremely good at improving effective throughput across many types of WAN connections.  These of course must sit at the correct spot in the path to the destination.  The units are often priced on bandwidth speed, and while they are very effective, are also quite an investment.  But they work at layer 3, and must in between the source and a router at both ends of the communication path; something that wouldn’t exist on a Metro-E circuit where VLANing was used to transmit replicated data.

The result is that for right now, I have chosen to go with a routed arrangement with no WAN optimization.  This does not differ too much from a traditional WAN circuit, other than my latencies should be much better.  The next step if our needs are not sufficiently met would be to invest in a couple of Cisco switches, then send replication data over it’s own VLAN to the CoLo, similar to the illustration above.


The equipment

My original SAN array is an EqualLogic PS5000e connected to a couple of Dell PowerConnect 5424 switches.  My new equipment closely mirrors this, but is slightly better;  An EqualLogic PS6000e and two PowerConnect 6224 switches.  Since both items will scale a bit better, I’ve decided to change out the existing array and switches with the new equipment.


Some Lessons learned so far

If you are changing ISPs, and your old ISP has authoritative control of your DNS zone files, make sure your new ISP has the zone file EXACTLY the way you need it.  Then confirm it one more time.  Spelling errors and omissions in DNS zone files doesn’t work out very well, especially when you factor in the time it takes for the corrections to propagate through the net.  (Usually up to 72 hours, but can feel like a lifetime when your customers can’t get to your website) 

If you are going to go with a QMOE or Metro-E circuit, be mindful that you might have to force the external interface on your outermost equipment (in our case, the firewall/router, but could be a managed switch as well) to negotiate to 100mbps full duplex.  Auto negotiation apparently doesn’t work to well on many Metro-E implementations, and can cause fragmentation that will reduce your effective throughput by quite a bit.  This is exactly what we saw.  Fortunately it was an easy fix.


Stay tuned for what’s next…


Get every new post delivered to your Inbox.

Join 641 other followers