Dell EqualLogic’s newest Host Integration Tools for Linux (v1.1)

 

It was just last September that I wrote about Using the Dell EqualLogic HIT for Linux (HIT/LE) Version 1.0.  At the time, the HIT/LE was beginning to play an important role in how we housed large volumes of data, and I wanted to share with others what I learned in the process.  While it has been running well in our environment, it was definitely a 1.0 product when it came to features and configuration, so I was anxious to see what was in store for the next version.  Version 1.1 was released in April of 2012, and it addressed some of the observations I had about HIT/LE 1.0.  Here are a few highlights.

  • Better distribution support.  CentOS, the binary compatible/clone to RHEL is now supported.  Versions 5.7 through 6.2 of CentOS are now supported.  According to the documentation, RHEL 5.5 is no longer supported, which is a change from the previous edition.  Suse Enterprise Linux is also supported.
  • Auto Snapshot Manager, Linux Edition. (ASM/LE).  A new feature that will allow you to create, manage, and schedule volume snapshots (Smart Copies), clones, and replicas from inside of the guest.  This is huge improvement. 
  • A new installer and configuration process. 
  • Better documentation.  This wasn’t listed in the release notes, but was immediately a noticeable improvement.

Version 1.0 did a good job applying the benefits of guest volumes to Linux based Operating Systems.  The problem was that it left out key abilities that prevented an automated way to manage those snapshots for specific purposes.  The biggest challenge I had was finding an automated way to take snapshots of these Linux guest attached volumes, and mount them to a Linux media server so that the data could be archived onto tape.  No amount of glue or duct tape helped in bridging the functions needed with snapshot manipulation inside the guests.

Configuring and Connecting
The configuration and connection of volumes seems to be greatly simplified.  Below demonstrates a simplified method of connecting an existing volume to a VM running the new HIT/LE 1.1.  Compare this to the instructions I provided on my post about HIT/LE 1.0, and you’ll see quite a difference.

  1. Add access to a PS Series Group called "MYEQLGRP”
    rswcli –add-group-access –gn MYEQLGRP –gip 10.10.10.100

    VERIFICATION: List the group added above
    rswcli –l

  2. Discover iSCSI targets
    iscsiadm -m discoverydb

    VERIFICATION: Confirm by viewing current list of discovered targets
    iscsiadm -m node | sort –u

    (returns the iqn needed in the next step)

  3. Log into a volume name and automatically connect at boot:
    ehcmcli login –target iqn.2001-05.com.equallogic:0-8a0906-3a7da1609-e720013e5c54e679-nfs100 –login-at-boot

    (returns the new device bound to a subdirectory below /dev/eql)

    VERIFICATION: Confirm device connection:
    ehcmcli status

  4. Mount it (and add to fstab for automatic mounting if desired)
    mount /dev/eql/nfs100 /mnt/myexport

In-Guest Volume Snapshots (Smart Copies)
The old version of HIT/LE didn’t offer any way of creating a snapshot inside the guest.  One could create volume snapshots from the Group Manager GUI, and even schedule them.  However, when it came to manipulating that snapshot from a guest, such as turning it online, or connecting to it, there was no way to do so.  Since the snapshots generate their own unique IQN, one needed a way to query for, and pass these variables as parameters. 

The new version offers a complete command set that fills the void.  At the root of the new found intelligence is the “asmcli” command.  The asmcli help command will provide you with a complete listing of options.  I’m not going to dive into each option, but rather, provide a simple example of how one can create a smart copy, and mount it if needed.

Before you get started, you may wish to choose or create a dedicated account on your PS Group that has volume administrator privileges.  Each system that has ASM/LE installed needs an account to interact with the volumes, and this offers the least privilege necessary to interact with the Smart Copies. The example below uses an account named “asmleadmin”

  1. Create PS group access (one time configuration step)
    asmcli create group-access –-name MYEQLGRP –-ip-address 10.10.10.100 –-user-name asmleadmin 

    VERIFICATION:  Confirm group access is set the way you want it.
    asmcli list group-access

  2. Create Smart Copy of the guest attached volume mounted to /mnt/myexport
    asmcli create smart-copy –-source /mnt/myexport

    VERIFICATION:  List all available Smart Copies
    asmcli list smart-copy –verbose 2

    (this will provide the object ID used in the next step)

  3. Mount a Smart Copy to a temporary location of /mnt/smartcopy
    asmcli mount smart-copy –-source /mnt/myexport –-object \f-f6a7e0-234b7ce30-d9c3f81bedbb96ba –-destination /mnt/smartcopy

  4. Unmount a Smart Copy mounted in the previous step
    asmcli unmount smart-copy –-object f-f6a7e0-234b7ce30-d9c3f81bedbb96ba –source /mnt/smartcopy

When documentation becomes a feature
The combination of a refined product, and improved documentation allowed for complete configuration and operation by just reading the manual.  It contained real examples of commands and actions, and even a few best practices.  No fumbling around due to an absence of detail or accuracy.  No need to search the net or call Technical Support this time.  Installation and configuration procedures reflected exactly what I experienced when testing out the new version.  What a nice surprise.  I wish this was more common.

More Tips for using the HIT/LE
Since my initial deployment of the HIT/LE, I had to do a fair amount of testing with these Linux systems running guest attached volumes to make sure they were satisfying performance needs; in particular, file I/O.  From that testing, and observations of the systems in production, here are a few things worth noting.

  • Getting data that lives on guest attached volumes onto traditional backup media does require extra thought and consideration, as traditional backup solutions that use the vCenter API can’t see these volumes. Take this into consideration when deciding use cases for guest attached volumes.
  • Don’t skimp on Linux VM memory.  Linux file I/O can be really impressive, but only if it has enough RAM.  If you have a lot of file I/O, linux will need the RAM.  I found going with anything less that 2GB of RAM had a pretty big impact on performance.
  • Review the role of the Linux VM so that it can be right sized.  I ran into a case where I was replacing a very important physical server with a Linux VM for our Development group, but unbeknownst to me, it was performing duties I was not aware of.
  • Make sure there aren’t traditional routines that unnecessarily manipulate that data on the guest volume.  This is reflected as changed block data, and could dramatically reduce the number of snaphots or replicas you can retain at any given time.
  • Take a quick look at your vSwitch and port group configuration in vSphere for your guest attached volumes to make sure you are getting the most out of MPIO.  Will Urban has written a great post Data Drives in VMware which addresses this topic. 

In summary, the newest edition of the HIT/LE is definitely new. In fact, it feels like a complete re-write, and leaves me baffled as to why it didn’t warrant a 2.0 version designator. Nevertheless, the specific features added allow for real protection workflows to be achieved. I need to spend some more time with it to incorporate many of the new features into our environment.  If you were interested in guest attached volumes in Linux, but were intimidated by the complexity of the old version, give HIT/LE 1.1 a try.

Upgrading to vSphere 5.0 by starting from scratch. …well, sort of.

It is never any fun getting left behind in IT.  Major upgrades every year or two might not be a big deal if you only had to deal with one piece of software, but take a look at most software inventories, and you’ll see possibly dozens of enterprise level applications and supporting services that all contribute to the chaos.  It can be overwhelming for just one person to handle.  While you may be perfectly justified in holding off on specific upgrades, there still seems to be a bit of guilt around doing so.  You might have ample business and technical factors to support such decisions, and a well crafted message providing clear reasons to stakeholders.  The business and political pressures ultimately win out, and you find yourself addressing the more customer/user facing application upgrades before the behind-the-scenes tools that power it all.

That is pretty much where I stood with my virtualized infrastructure.  My last major upgrade was to vSphere 4.0.  Sure, I had visions of keeping up with every update and patch, but a little time passed, and several hundred distractions later, I found myself left behind.  When vSphere 4.1 came out, I also had every intention of upgrading.  However, I was one of the legions of users who had a vCenter server running on a 32bit OS, and that complicated matters a little bit.  I looked at the various publications and posts on the upgrade paths and experiences.  Nothing seemed quite as easy as I was hoping for, so I did what came easiest to my already packed schedule; nothing.  I wondered just how many Administrators found themselves in the same predicament; not touching an aging, albeit perfectly fine running system.  

My ESX 4.0 cluster served my organization well, but times change, and so do needs.  A few things come up to kick-start the desire to upgrade.

  • I needed to deploy a pilot VDI project, fast.  (more about this in later posts)
  • We were a victim of our own success with virtualization, and I needed to squeeze even more power and efficiency out of our investment in our infrastructure.

Both are pretty good reasons to upgrade, and while I would have loved to do my typical due diligence on every possible option, I needed a fast track.  My move to vSphere 5.0 was really just a prerequisite of sorts to my work with VDI. 

But how should I go about an upgrade?

Do I update my 4.0 hosts to the latest update that would be eligible for an upgrade path to 5.0, and if so, how much work would that be?  Should I transition to a new vCenter server, migrating the database, then run a mixed environment of ESX hosts running with different versions?  What sort of problems would that introduce?  After conferring with a trusted colleague of mine who always seems to have pragmatic sensibilities when it comes to virtualization, I decided which option was going to be the best for me.  I opted not to do any upgrade, and simply transition to a pristine new cluster.  It looked something like this:

  • Take a host (either new, or by removing an existing one from the cluster), and build it up with ESXi 5.0.
  • Build up a new 64bit VM for running a brand new vCenter, and configure as needed.
  • Remove one VM at a time from the old cluster by powering them down, remove from inventory, add to the new cluster.
  • Once enough VM’s have been removed, take another host, remove from the old cluster, rebuild as ESXi 5.0, and add to the new cluster.
  • Repeat until finished.

For me, the decision to start from scratch won out.  Why?

  • I could build up a pristine vCenter server, with a database that wasn’t going to carry over any unwanted artifacts of my previous installation.
  • I could easily set up the new vCenter to emulate my old settings.  Folders, EVC settings, resource pools, etc.
  • I could transition or build up my supporting VM’s or appliances to my new infrastructure to make sure they worked before committing to the transition.
  • I could afford a simple restart of each VM as I transitioned it to a new cluster.  I used this as an opportunity to update the VMware Tools when added to the new inventory.
  • I was willing to give up historical data in my old vSphere 4.0 cluster for the sake of simplicity of the plan and cleanliness of the configuration.
  • Predictability.  I didn’t have to read a single white paper or discussion thread on database migrations or troubles with DSNs.
  • I have a well documented ESX host configuration that is not terribly complex, and easy to recreate across 6 hosts.
  • I just happened to have purchased an additional blade and license of ESX, so it was an ideal time to introduce it to my environment.
  • I could get my entire setup working, then get my licensing figured out after it’s all complete.

You’ll notice that one option similar to this approach would have been to simply remove a host of running VM’s out of the existing cluster, and add it to the new cluster.  This may have been just as good of a plan, as it would have avoided the need to manually shut down and remove each VM one at a time during the transition.  However, I would have needed to run a mix of ESX 4.0 and 5.0 hosts in the new cluster.  I didn’t want to carry anything over from the old setup.  I would have needed to upgrade or rebuild the host anyway, and I had to restart each VM to make sure it was running the latest tools.  If for nothing other than clarity of mind, my approach seemed best for me.

Prior to beginning the transition, I needed to update my Dell EqualLogic firmware to 5.1.2.  A collection of very nice improvements made this a nice upgrade, but a requirement for what I wanted to do.  While the upgrade itself went smoothly, it did re-introduce an issue or two.  The folks at Dell EqualLogic are aware of this, and are working to address it hopefully in their next release.  The combination of the firmware upgrade, and vSphere 5 allowed me to use the latest and greatest tools from EqualLogic, primarily the Host Integration Tools VMWare Edition (HIT/VE) and the storage integration in vSphere thanks to VASA.  Although, as of this writing, EqualLogic does not have a full production release of their MultiPathing Extension Module (MEM) for vSphere 5.0.  The EPA version was just released, but I’ll probably wait for the full release of MEM to come out before I apply it to the hosts in the cluster.

While I was eager to finish the transition, I didn’t want to prematurely create any problems.  I took a page from my own lessons learned during my upgrade to ESX 4.0, and exercised some restraint when it came to updating my Virtual Hardware for each VM to version 8.  My last update of Virtual Hardware levels in each VM caused some unexpected results, as I shared in “Side effects of upgrading VM’s to Virtual Hardware 7 in vSphere”   Apparently, I wasn’t the only one who ran into issues, because that post has statistically been my all time most popular post.  The abilities of Virtual Hardware 8 powered VMs are pretty neat, but I’m in no rush to make any virtual hardware changes to some of my key production systems, especially those noted. 

So, how did it work out?  The actual process completed without a single major hang-up, and am thrilled with the result.  The irony here is that even though vSphere provides most of the intelligence behind my entire infrastructure, and does things that are mind bogglingly cool, it was so much easier to upgrade than say, SharePoint, AD, Exchange, or some other enterprise software.  Great technologies are great because they work like you think they should.  No exception here.  If you are considering a move to vSphere 5.0, and are a little behind on your old infrastructure, this upgrade approach might be worth considering.

Now, onto that little VDI project…

Resources

A great resource on setting up SQL 2008 R2 for vCenter
How to Install Microsoft SQL Server 2008 R2 for VMware vCenter 5

Installing vCenter 5 Best Practices
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003790

A little VMFS 5.0 info
http://www.yellow-bricks.com/2011/07/13/vsphere-5-0-what-has-changed-for-vmfs/

Information on the EqualLogic Multipathing Extension Module (MEM), and if you are an EqualLogic customer, why you should care.
https://whiteboardninja.wordpress.com/2011/02/01/equallogic-mem-and-vstorage-apis/