Dell EqualLogic’s newest Host Integration Tools for Linux (v1.1)

 

It was just last September that I wrote about Using the Dell EqualLogic HIT for Linux (HIT/LE) Version 1.0.  At the time, the HIT/LE was beginning to play an important role in how we housed large volumes of data, and I wanted to share with others what I learned in the process.  While it has been running well in our environment, it was definitely a 1.0 product when it came to features and configuration, so I was anxious to see what was in store for the next version.  Version 1.1 was released in April of 2012, and it addressed some of the observations I had about HIT/LE 1.0.  Here are a few highlights.

  • Better distribution support.  CentOS, the binary compatible/clone to RHEL is now supported.  Versions 5.7 through 6.2 of CentOS are now supported.  According to the documentation, RHEL 5.5 is no longer supported, which is a change from the previous edition.  Suse Enterprise Linux is also supported.
  • Auto Snapshot Manager, Linux Edition. (ASM/LE).  A new feature that will allow you to create, manage, and schedule volume snapshots (Smart Copies), clones, and replicas from inside of the guest.  This is huge improvement. 
  • A new installer and configuration process. 
  • Better documentation.  This wasn’t listed in the release notes, but was immediately a noticeable improvement.

Version 1.0 did a good job applying the benefits of guest volumes to Linux based Operating Systems.  The problem was that it left out key abilities that prevented an automated way to manage those snapshots for specific purposes.  The biggest challenge I had was finding an automated way to take snapshots of these Linux guest attached volumes, and mount them to a Linux media server so that the data could be archived onto tape.  No amount of glue or duct tape helped in bridging the functions needed with snapshot manipulation inside the guests.

Configuring and Connecting
The configuration and connection of volumes seems to be greatly simplified.  Below demonstrates a simplified method of connecting an existing volume to a VM running the new HIT/LE 1.1.  Compare this to the instructions I provided on my post about HIT/LE 1.0, and you’ll see quite a difference.

  1. Add access to a PS Series Group called "MYEQLGRP”
    rswcli –add-group-access –gn MYEQLGRP –gip 10.10.10.100

    VERIFICATION: List the group added above
    rswcli –l

  2. Discover iSCSI targets
    iscsiadm -m discoverydb

    VERIFICATION: Confirm by viewing current list of discovered targets
    iscsiadm -m node | sort –u

    (returns the iqn needed in the next step)

  3. Log into a volume name and automatically connect at boot:
    ehcmcli login –target iqn.2001-05.com.equallogic:0-8a0906-3a7da1609-e720013e5c54e679-nfs100 –login-at-boot

    (returns the new device bound to a subdirectory below /dev/eql)

    VERIFICATION: Confirm device connection:
    ehcmcli status

  4. Mount it (and add to fstab for automatic mounting if desired)
    mount /dev/eql/nfs100 /mnt/myexport

In-Guest Volume Snapshots (Smart Copies)
The old version of HIT/LE didn’t offer any way of creating a snapshot inside the guest.  One could create volume snapshots from the Group Manager GUI, and even schedule them.  However, when it came to manipulating that snapshot from a guest, such as turning it online, or connecting to it, there was no way to do so.  Since the snapshots generate their own unique IQN, one needed a way to query for, and pass these variables as parameters. 

The new version offers a complete command set that fills the void.  At the root of the new found intelligence is the “asmcli” command.  The asmcli help command will provide you with a complete listing of options.  I’m not going to dive into each option, but rather, provide a simple example of how one can create a smart copy, and mount it if needed.

Before you get started, you may wish to choose or create a dedicated account on your PS Group that has volume administrator privileges.  Each system that has ASM/LE installed needs an account to interact with the volumes, and this offers the least privilege necessary to interact with the Smart Copies. The example below uses an account named “asmleadmin”

  1. Create PS group access (one time configuration step)
    asmcli create group-access –-name MYEQLGRP –-ip-address 10.10.10.100 –-user-name asmleadmin 

    VERIFICATION:  Confirm group access is set the way you want it.
    asmcli list group-access

  2. Create Smart Copy of the guest attached volume mounted to /mnt/myexport
    asmcli create smart-copy –-source /mnt/myexport

    VERIFICATION:  List all available Smart Copies
    asmcli list smart-copy –verbose 2

    (this will provide the object ID used in the next step)

  3. Mount a Smart Copy to a temporary location of /mnt/smartcopy
    asmcli mount smart-copy –-source /mnt/myexport –-object \f-f6a7e0-234b7ce30-d9c3f81bedbb96ba –-destination /mnt/smartcopy

  4. Unmount a Smart Copy mounted in the previous step
    asmcli unmount smart-copy –-object f-f6a7e0-234b7ce30-d9c3f81bedbb96ba –source /mnt/smartcopy

When documentation becomes a feature
The combination of a refined product, and improved documentation allowed for complete configuration and operation by just reading the manual.  It contained real examples of commands and actions, and even a few best practices.  No fumbling around due to an absence of detail or accuracy.  No need to search the net or call Technical Support this time.  Installation and configuration procedures reflected exactly what I experienced when testing out the new version.  What a nice surprise.  I wish this was more common.

More Tips for using the HIT/LE
Since my initial deployment of the HIT/LE, I had to do a fair amount of testing with these Linux systems running guest attached volumes to make sure they were satisfying performance needs; in particular, file I/O.  From that testing, and observations of the systems in production, here are a few things worth noting.

  • Getting data that lives on guest attached volumes onto traditional backup media does require extra thought and consideration, as traditional backup solutions that use the vCenter API can’t see these volumes. Take this into consideration when deciding use cases for guest attached volumes.
  • Don’t skimp on Linux VM memory.  Linux file I/O can be really impressive, but only if it has enough RAM.  If you have a lot of file I/O, linux will need the RAM.  I found going with anything less that 2GB of RAM had a pretty big impact on performance.
  • Review the role of the Linux VM so that it can be right sized.  I ran into a case where I was replacing a very important physical server with a Linux VM for our Development group, but unbeknownst to me, it was performing duties I was not aware of.
  • Make sure there aren’t traditional routines that unnecessarily manipulate that data on the guest volume.  This is reflected as changed block data, and could dramatically reduce the number of snaphots or replicas you can retain at any given time.
  • Take a quick look at your vSwitch and port group configuration in vSphere for your guest attached volumes to make sure you are getting the most out of MPIO.  Will Urban has written a great post Data Drives in VMware which addresses this topic. 

In summary, the newest edition of the HIT/LE is definitely new. In fact, it feels like a complete re-write, and leaves me baffled as to why it didn’t warrant a 2.0 version designator. Nevertheless, the specific features added allow for real protection workflows to be achieved. I need to spend some more time with it to incorporate many of the new features into our environment.  If you were interested in guest attached volumes in Linux, but were intimidated by the complexity of the old version, give HIT/LE 1.1 a try.

Using the Dell EqualLogic HIT for Linux

 

I’ve been a big fan of Dell EqualLogic Host Integration Tools for Microsoft (HIT/ME), so I was looking forward to seeing how the newly released HIT for Linux (HIT/LE) was going to pan out.  The HIT/ME and HIT/LE offer unique features when using guest attached volumes in your VM’s.  What’s the big deal about guest attached volumes?  Well, here is why I like them.

  • It keeps the footprint of the VM really small.  The VM can easily fit in your standard VMFS volumes.
  • Portable/replaceable.  Often times, systems serving up large volumes of unstructured data are hard to update.  Having the data as guest attached means that you can easily prepare a new VM presenting the data (via NFS, Samba, etc.), and cut it over without anyone knowing – especially when you are using DNS aliasing.
  • Easy and fast data recovery.  My “in the trenches” experience with the guest attached volumes in VM’s running Microsoft OS’s (and EqualLogic’s HIT/ME) have proven that recovering data off of guest attached volumes is just easier – whether you recover it from snapshot or replica, clone it for analysis, etc. 
  • Better visibility of performance. Thanks to the independent volume(s), one can easily see with SANHQ what the requirements of that data volume is. 
  • More flexible protection.  With guest attached volumes, it’s easy to crank up the frequency of snapshot and replica protection on just the data, without interfering with the VM that is serving up the data.
  • Efficient, tunable MPIO. 
  • Better utilization of space.  If you wanted to serve up a 2TB volume of storage using a VMDK, more than likely you’d have a 2TB VMFS volume, and something like a 1.6TB VMDK file to accommodate hypervisor snapshots.  With a native volume, you would be able to use the entire 2TB of space. 

The one “gotcha” about guest attached volumes is that they aren’t visible by the vCenter API, so commercial backup applications that rely on the visibility of these volumes via vCenter won’t be able to back them up.  If you use these commercial applications for protection, you may want to determine if guest attached volumes are a good fit, and if so, find alternate ways of protecting the volumes.    Others might contend that because the volumes aren’t seen by vCenter, one is making things more complex, not less.  I understand the reason for thinking this way, but my experience with them have proven quite the contrary.

Motive
I wasn’t trying out the HIT/LE because I ran out of things to do.  I needed it to solve a problem.  I had to serve up a large amount (several Terabytes) of flat file storage for our Software Development Team.  In fact, this was just the first of several large pools of storage that I need to serve up.  It would have been simple enough to deploy a typical VM with a second large VMDK, but managing such an arrangement would be more difficult.  If you are ever contemplating deployment decisions, remember that simplicity and flexibility of management should trump simplicity of deployment if it’s a close call.  Guest attached volumes align well with the “design as if you know it’s going to change” concept.  I knew from my experience with working with guest attached volumes for Windows VM’s, that they were very agile, and offered a tremendous amount of flexibility.

But wait… you might be asking, “If I’m doing nothing but presenting large amounts of raw storage, why not skip all of this and use Dell’s new EqualLogic FS7500 Multi-Protocol NAS solution?”  Great question!  I had the opportunity to see the FS7500 NAS head unit at this year’s Dell Storage Forum.  The FS7500 turns the EqualLogic block based storage accessible only on your SAN network into CIFS/NFS storage presentable to your LAN.  It is impressive.  It is also expensive.  Right now, using VM’s to present storage data is the solution that fits within my budget.  There are some downfalls (Samba not supporting SMB2), but for the most part, it falls in the “good enough” category.

I had visions of this post focusing on the performance tweaks and the unique abilities of the HIT/LE.  After implementing it, I was reminded that it was indeed a 1.0  product.  There were enough gaps in deployment information that I felt it necessary to provide information on exactly how I actually made the HIT for Linux work.  IT Generalists who I suspect make up a significant amount of the Dell EqualLogic customer base have learned to appreciate their philosophy of “if you can’t make it easy, don’t add the feature.”   Not everything can be made intuitive however, especially the first time around.

Deployment Assumptions 
The scenario and instructions are for a single VM that will be used to serve up a single large volume for storage. It could serve up many guest attached volumes, but for the sake of simplicity, we’ll just be connecting to a single volume.

  • VM with 3 total vNICs.  One used for LAN traffic, and the other two, used exclusively for SAN traffic.  The vNIC’s for the SAN will be assigned to the proper vswitch and portgroup, and will have static IP addresses.  The VM name in this example is “testvm”
  • A single data volume in your EqualLogic PS group, with an ACL that allows for the guest VM to connect to the volume using CHAP, IQN, or IP addresses.  (It may be easiest to first restrict it by IP address, as you won’t be able to determine your IQN until the HIT is installed).  The native volume name in this example is “nfs001” and the group IP address is 10.1.0.10
  • Guest attached volume will be automatically connected at boot, and will be accessible via NFS export.  In this example I will be configuring the system so that the volume is available via the “/data1” directory.
  • OS used will be RedHat Enterprise Linux (RHEL) 5.5. 
  • EqualLogic’s HIT 1.0

Each step below that starts with word “VERIFICATION” is not a necessary step, but it helps you understand the process, and will validate your findings.  For brevity, I’ve omitted some of the output of these commands.

Deploying and configuring the HIT for Linux
Here we go…

Prepping for Installation

1.     Verify installation of EqualLogic prerequisites (via rpm -q [pkgname]).  If not installed, run yum install [pkgname]

openssl                    (0.9.8e for RHEL 5.5)

libpcap                    (0.9.4 for RHEL 5.5)

iscsi-initiator-utils      (6.2.0.871 for RHEL 5.5)

device-mapper-multipath    (0.4.7 for RHEL 5.5)

python                                          (2.4 for RHEL 5.5.) 

dkms                       (1.9.5 for RHEL 5.5)

 

(dkms is not part of RedHat repo.  Need to download from http://linux.dell.com/dkms/ or via the "Extra Packages for LInux" epel repository.  I chose Dell website location because it was a newer version.  Simply download and execute RPM.). 

 

2.     Snapshot Linux machine so that if things go terribly wrong, it can be reversed

 

3.     Shutdown VM, and add NIC’s for guest access

Make sure to choose iSCSI network when adding to VM configuration

After startup, manually specify Static IP addresses and subnet mask for both.  (No default gateway!)

Activate NIC’s, and reboot

 

4.     Power up, then add the following lines to /etc/sysctl.conf  (for RHEL 5.5)

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

 

5.     Establish NFS and related daemons to automatically boot

chkconfig portmap on

chkconfig nfs on

chkconfig nfslock on

 

6.     Establish directory which will ultimately be used to export for mounting.  In this example, the iSCSI device will mount to a directory called “eql2tbnfs001” in the /mnt directory. 

mkdir /mnt/eql2tbnfs001

 

7.     Make symbolic link called “data1” in the root of the file system.

ln -s /mnt/eql2tbnfs001 /data1 

 

Installation and configuration of the HIT

8.     Verify that the latest HIT Kit for Linux is being used for installation.  (V1.0.0 as of 9/2011)

 

9.     Import public key

      Download the public key from eql support site under HIT for Linux, and place in /tmp/ )

Add key:

rpm –import RPM-GPG-KEY-DELLEQL (docs show lower case, but file is upper case)

 

10.  Run installation

yum localinstall equallogic-host-tools-1.0.0-1.e15.x86_64.rpm

 

Note:  After HIT is installed, you may get the IQN for use of restricting volume access in the EqualLogic group manager by typing the following:

cat /etc/iscsi/initiatorname.iscsi.

 

11.  Run eqltune (verbose).  (Tip.  You may want to capture results to file for future reference and analysis)

            eqltune -v

 

12.  Make adjustments based on eqltune results.  (Items listed below were mine.  Yours may be different)

 

            NIC Settings

   Flow Control. 

ethtool -A eth1 autoneg off rx on tx on

ethtool -A eth2 autoneg off rx on tx on

 

(add the above lines to /etc/rc.d/rc.local to make persistent)

 

There may be a suggestion to use jumbo frames by increasing the MTU size from 1500 to 9000.  This has been omitted from the instructions, as it requires proper configuration of jumbos from end to end.  If you are uncertain, keep standard frames for the initial deployment.

 

   iSCSI Settings

   (make backup of /etc/iscsi/iscid.conf before changes)

 

      Change node.startup to manual.

   node.startup = manual

 

      Change FastAbort to the following:

   node.session.iscsi.FastAbort = No

 

      Change initial_login_retry to the following:

   node.session.initial_login_retry_max = 12

 

      Change number of queued iSCSI commands per session

   node.session.cmds_max = 1024

 

      Change device queue depth

   node.session.queue_depth = 128

 

13.  Re-run Eqltune -v to see if changes took affect

All changes took effect, minus the NIC settings added to the rc.local file.  Looks to be a syntax error from Eql documentation provided.  It has been corrected in the documentation above.

 

14.  Run command to view and modify MPIO settings

rswcli –mpio-parameters

 

This returns the results of:  (seems to be good for now)

Processing mpio-parameters command…

MPIO Parameters:

Max sessions per volume slice:: 2

Max sessions per entire volume:: 6

Minimum adapter speed:: 1000

Default load balancing policy configuration: Round Robin (RR)

IOs Per Path: 16

Use MPIO for snapshots: Yes

Internet Protocol: IPv4

The mpio-parameters command succeeded.

 

15.  Restrict MPIO to just the SAN interfaces

Exclude LAN traffic

            rswcli -E -network 192.168.0.0 -mask 255.255.255.0

 

VERIFICATION:  List status of includes/excludes to verify changes

            rswcli –L

 

VERIFICATION:  Verify Host connection Mgr is managing just two interfaces

      ehcmcli –d

 

16.  Discover targets

iscsiadm -m discovery -t st -p 10.1.0.10

(Make sure no unexpected volumes connect.  But note the IQN name presented.  You’ll need it for later.)

 

VERIFICATION:  shows iface

[root@testvm ~]# iscsiadm -m iface | sort

default tcp,<empty>,<empty>,<empty>,<empty>

eql.eth1_0 tcp,00:50:56:8B:1F:71,<empty>,<empty>,<empty>

eql.eth1_1 tcp,00:50:56:8B:1F:71,<empty>,<empty>,<empty>

eql.eth2_0 tcp,00:50:56:8B:57:97,<empty>,<empty>,<empty>

eql.eth2_1 tcp,00:50:56:8B:57:97,<empty>,<empty>,<empty>

iser iser,<empty>,<empty>,<empty>,<empty>

 

VERIFICATION:  Check connection sessions via iscsiadm -m session to show that no connections exist

[root@testvm ~]# iscsiadm -m session

iscsiadm: No active sessions.

 

VERIFICATION:  Check connection sessions via /dev/mapper to show that no connections exist

[root@testvm ~]# ls -la /dev/mapper

total 0

drwxr-xr-x  2 root root     60 Aug 26 09:59 .

drwxr-xr-x 10 root root   3740 Aug 26 10:01 ..

crw——-  1 root root 10, 63 Aug 26 09:59 control

 

VERIFICATION:  Check connection sessions via ehcmcli -d to show that no connections exist

[root@testvm ~]# ehcmcli -d

 

17.  Login just one of the iface paths of your liking (shown in red here).  Replace the IQN here (shown in green) with yours. The HIT will take care of the rest.

iscsiadm -m node -T iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001 -I eql.eth1_0 -l

 

This returned:

[root@testvm ~]# iscsiadm -m node -T iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001 -I eql.eth1_0 -l

Logging in to [iface: eql.eth1_0, target: iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001, portal: 10.1.0.10,3260]

Login to [iface: eql.eth1_0, target: iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001, portal: 10.1.0.10,3260] successful.

 

VERIFICATION:  Check connection sessions via iscsiadm -m session

[root@testvm ~]# iscsiadm -m session

tcp: [1] 10.1.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001

tcp: [2] 10.1.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001

 

VERIFICATION:  Check connection sessions via /dev/mapper.  This is going to give you the string you will need to use making and mounting the filesystem.

[root@testvm ~]# ls -la /dev/mapper

 

 

VERIFICATION:  Check connection sessions via ehcmcli -d

[root@testvm ~]# ehcmcli -d

 

18.  Make new file system from the dm-switch name.  Replace the IQN here (shown in green) with yours.  If this is an existing volume that has been used before (from a snapshot, or another machine) there is no need to perform this step.  Documentation will show this step without the “-j” switch, which will format it as a non-journaled ext2 file system.  The –j switch will format it as an ext3 file system.

mke2fs -j -v /dev/mapper/eql-0-8a0906-451da1609-2660013c7c34e45d-nfs001

 

19.  Mount the device to a directory

[root@testvm mnt]# mount /dev/mapper/eql-0-8a0906-451da1609-2660013c7c34e45d-nfs001 /mnt/eql2tbnfs001

 

20.  Establish iSCSI connection automatically

[root@testvm ~]# iscsiadm -m node -T iqn.2001-05.com.equallogic:0-8a0906-451da1609-2660013c7c34e45d-nfs001 -I eql.eth1_0 -o update -n node.startup -v automatic

 

21.  Mount volume automatically

Change /etc/fstab, adding the following:

/dev/mapper/eql-0-8a0906-451da1609-2660013c7c34e45d-nfs001 /mnt/eql2tbnfs001 ext3 _netdev  0 0

Restart system to verify automatic connection and mounting.

 

Working with guest attached volumes
After you have things configured and operational, you’ll see how flexible guest iSCSI volumes are to work with.

  • Do you want to temporarily mount a snapshot to this same VM or another VM? Just turn the snapshot online, and make a connection inside the VM.
  • Do you need to archive your data volume to tape, but do not want to interfere with your production system? Mount a recent snapshot of the volume to another system, and perform the backup there.
  • Do you want to do a major update to that front end server presenting the data? Just build up a new VM, connect the new VM to that existing data volume, and change your DNS aliasing, (which you really should be using) and you’re done.
  • Do you need to analyze the I/O of the guest attached volumes? Just use SANHQ. You can easily see if that data should be living on some super fast pool of SAS drives, or a pool of PS4000e arrays.  You’ll be able to make better purchasing decisions because of this.

So, how did it measure up?

The good…
Right out of the gate, I noticed a few really great things about the HIT for Linux.

  • The prerequisites and installation.  No compiling or other unnecessary steps.  The installation package installed clean with no fuss.  That doesn’t happen every day.
  • Eqltune.  This little utility is magic.  Talk about reducing overhead in preparing a system for MPIO and all things related to guest based iSCSI volumes.  It gave me a complete set of adjustments to make, divided into 3 simple categories.  After I made the adjustments, I re-ran the utility, everything checked out okay.  Actually, all of the command line tools were extremely helpful.  Bravo!
  • One really impressive trait of the HIT/LE is how it handles the iSCSI sessions for you. Session build up and teardown is all taken care of by the HIT for Linux.

The not so good…
Almost as fast as the good shows up, you’ll notice a few limitations

  • Version 1.0 is only officially supported on RedHat Enterprise Linux (RHEL) 5.5 and 6.0 (no 6.1 as of this writing).  This might be news to Dell, but Debian based systems like Ubuntu are running in enterprises everywhere for it’s cost, solid behavior, and minimalist approach.  RedHat clones dominate much of the market; some commercial, and some free.  Personally, upstream Distributions such as Fedora are sketchy, and prone to breakage with each release (Note to Dell, I don’t blame you for not supporting these.  I wouldn’t either).  Other distributions are quirky for their own reasons of “improvement” and I can understand why these weren’t initially supported either.  A safer approach for Dell (and the more flexible approach for the customer) would be to 1.) Get out a version for Ubuntu as fast as possible, and 2.)  Extend the support of this version to RedHat’s, downstream, 100% binary compatible, very conservative distribution, CentOS.  For you Linux newbies, think of CentOS as being the RedHat installation but with the proprietary components stripped out, and nothing else added.  While my first production Linux server running the HIT is RedHat 5.5, all of my testing and early deployment occurred on a CentOS 5.5 Distribution, and it worked perfectly. 
  • No AutoSnapshot Manager (ASM) or equivalent.  I rely on ASM/ME on my Windows VM’s with guest attached volumes to provide me with a few key capabilities.  1.)  A mechanism to protect the volumes via snaphots and replicas.  2.)  Coordinating applications and I/O so that I/O is flushed properly.  Now, Linux does not have any built-in facility like Microsoft’s Volume Shadow Copy Services (VSS), so Dell can’t do much about that.  But perhaps some simple script templates might give the users ideas on how to flush and pause I/O of the guest attached volumes for snapshots.  Just having a utility to create Smart copies or mount them would be pretty nice. 

The forgotten…
A few things overlooked?  Yep.

  • I was initially encouraged by the looks of the documentation.  However, In order to come up with the above, I had to piece together information from a number of different resources.   Syntax and capitalization errors will kill you in a Linux shell environment.  Some of those inconsistencies and omissions showed up.  With a little triangulation, I was able to get things running correctly, but it quickly became a frustrating, time consuming exercise that I felt like I’ve been through before.  Hopefully the information provided here will help.
  • Somewhat related to the documentation issue is something that has come up with a few of the other EqualLogic tools;  Customers often don’t understand WHY one might want to use the tool.  Same thing goes with the HIT for Linux.  Nobody even gets to the “how” if they don’t understand the “why”.  But, I’m encouraged by the great work the Dell TechCenter has been doing with their white papers and videos.  It has become a great source for current information, and are moving in the right direction of customer education.   

Summary
I’m generally encouraged by what I see, and am hoping that Dell EqualLogic takes on the design queues of the HIT/ME to employ features like AutoSnapshot Manager, and an equivalent to eqlxcp (EqualLogic’s offloaded file copy command in Windows).  The HIT for Linux  helped me achieve exactly what I was trying to accomplish.  The foundation for another easy to use tool in the EqualLogic line up is certainly there, and I’m looking forward to how this can improve.

Helpful resources
Configuring and Deploying the Dell EqualLogic Host Integration Toolkit for Linux
http://en.community.dell.com/dell-groups/dtcmedia/m/mediagallery/19861419/download.aspx

Host Integration Tools for Linux – Installation and User Guide
https://www.equallogic.com/support/download_file.aspx?id=1046 (login required)

Getting more IOPS on workloads running RHEL and EQL HIT for Linux
http://en.community.dell.com/dell-blogs/enterprise/b/tech-center/archive/2011/08/17/getting-more-iops-on-your-oracle-workloads-running-on-red-hat-enterprise-linux-and-dell-equallogic-with-eql-hitkit.aspx 

RHEL5.x iSCSI configuration (Not originally authored by Dell, nor specific to EqualLogic)
http://www.equallogic.com/resourcecenter/assetview.aspx?id=8727 

User’s experience trying to use the HIT on RHEL 6.1, along with some other follies
http://www.linux.com/community/blogs/configuring-dell-equallogic-ps6500-array-to-work-with-redhat-linux-6-el.html 

Dell TechCenter website
http://DellTechCenter.com/ 

Dell TechCenter twitter handle
@DellTechCenter

Reworking my PowerConnect 6200 switches for my iSCSI SAN

It sure is easy these days to get spoiled with the flexibility of virtualization and shared storage.  Optimization, maintenance, fail-over, and other adjustments are so much easier than they used to be.  However, there is an occasional reminder that some things are still difficult to change.  For me, that reminder was my switches I use for my SAN.

One of the many themes I kept hearing at this year’s Dell Storage Forum (a great experience I must say) throughout several of the breakout sessions I went to was “get your SAN switches configured correctly.”  A nice reminder to something I was all too aware of already; my Dell PowerConnect 6224 switches were not configured correctly since the day they replaced my slightly less capable (but rock solid) PowerConnect 5424’s.  I returned from the forum committed to getting my switchgear updated and configured the correct way.  Now for the tough parts…  What does “correct” really mean when it comes to the 6200 series switches?  And why didn’t I take care of this a long time ago?  Here are just a few excuses reasons. 

  • At the time of initial deployment, I had difficulty tracking down documentation written specifically for the 6224’s to be configured with iSCSI.  Eventually, I did my best to interpret the configuration settings of the 5424’s, and apply the same principals to the 6224’s.  Unfortunately, the 6224’s are a different animal than the 5424’s, and that showed up after I placed them into production – a task that I regretfully rushed.
  • When I deployed them into production, the current firmware was the 2.x generation.  It was my understanding after the deployment that the 2.x firmware on the 6200 series definitely had growing pains.  I also had the unfortunate timing that the next major revision came out shortly after I put them into production.
  • I had two stacked 6224 switches running my production SAN environment (a setup that was quite common for those I asked at the Dell Storage Forum). While experimenting with settings might be fun in a lab, it is no fun, and serious business when they are running a production environment. I wanted to make adjustments just once, but had difficulty confirming settings.
  • When firmware needs to be updated (a conclusion to an issue I was reporting to Technical Support), it is going to take down the entire stack.  This means that you’d better have everything that uses the SAN off unless you like living dangerously.  Major firmware updates will also require the boot code in each switch to be updated.  A true “lights out” maintenance window that required everything to be shut down.  The humble little 5424’s LAGd together didn’t have that problem.
  • The 2.x to 3.x firmware update also required the boot code to be updated.  However, you simply couldn’t run an “update bootcode” command.  The documentation made this very clear.  The PowerConnect Technical Support Team indicated that the two versions ran different algorithms to unpack the contents, which was the reason for yet another exception to the upgrade process. 

One of the many best practices recommended at the Forum was to stack the switches instead of LAGing them.  Stack, stack, stack was drilled into everyone’s head.  The reasons are very good, and make a lot of sense.

  • Stacking modules in many ways extend the circuiting of a single switch, thus the stacking module doesn’t have to honor or be limited by traditional Ethernet.
  • Managing one switch manages them all.
  • Better, more scalable bandwidth between switches
  • No messing around with LAG’s

But here lays the conundrum of many Administrators who are responsible for production environments.  While stacked 6224’s offer redundancy against hardware failure, they offer no redundancy when it comes to maintenance.  These stacked switches are seen as one logical unit, and may be your weakest link when it comes to maintenance of your virtualized infrastructure.  Interestingly enough, when inquiring further on effective strategies for updating under this topology, I observed a few things;  many other users who were stuck with this very same dilemma, and the answers provided weren’t too exciting.  There were generally three answers I heard from this design decision:

  • Plan for a “lights out” maintenance window.
  • Buy another set of two switches, stack those, then trunk the two together via 10Gbe,
  • Buy better switches. 

See why I wasn’t too excited about my options?

Decision time.  I knew I’d suffer a bit of downtime updating the firmware and revamping the configuration no matter what I did.  Do I stack them as recommended, only to be faced with the same dilemma on the next firmware upgrade?  Or do I LAG the switches together so that I avoid this upgrade fiasco in the future?  LAG’ing is not perfect either, and the more arrays I add (as well as the inter-array traffic increasing with new array features), the more it might compound some of the limitations of LAGs. 

What option won out?  I decided to give stacking ONE more try.  I had to keep the eye on my primary objective; correcting my configuration by way of firmware upgrade and build up a simple, pristine configuration from scratch.  The idea was that the configuration would initially contain the minimum set of modifications to get them working according to best practices.  Then, I could build off of the configuration in the future.  Also influencing my decision was finding out that recommended settings with LAGs apparently change frequently.  For instance, just recently, the recommended setting for flow control for the port channel in a LAG was changed.  These are the types of things I wanted to stay away from.  But with that said, I will continue to keep the option open to LAGing them, for the sole reason that it offers the flexibility for maintenance without shutting down your entire cluster.

So here was my minimum desired results for the switch stack after the upgrade and reconfiguration.  Pretty straight forward. 

  • Management traffic on another VLAN (VLAN 10) on port 1 (for uplinking) and port 2 (for local access).
  • iSCSI traffic on it’s own VLAN (VLAN 100), on all ports not including the management ports.
  • Essentially no traffic on the Default VLAN
  • Recommended global and port specific settings (flow control, spanning tree, jumbo frames, etc.) for iSCSI traffic endpoint connections
  • iSCSI traffic that was available to be routed through my firewall (for replication).

My configuration rework assumed the successful boot code and firmware upgrade to version 3.2.1.3.  I pondered a few different ways to speed this process up, but ultimately just followed the very good steps provided with the documentation for the firmware.  They were clear, and accurate.

By the way, on June 20th, 2011, Dell released their very latest firmware update (thank you RSS feed) to 3.2.1.3 A23.  This now includes their “Auto Detection” of ports for iSCSI traffic.  Even though the name implies a feature that might be helpful, the documentation did not provide enough information needed, and I decided to manually configure as originally planned.

For those who might be in the same boat as I was, here were the exact steps I did for building up a pristine configuration after updating the firmware and boot code.  The configuration below was definitely a combined effort by the folks from the EqualLogic and PowerConnect Teams, and me pouring over a healthy amount of documentation.  It was my hope that this combined effort would eliminate some of the contradictory information I found in previous best practices articles, forum threads, and KB articles that assumed earlier firmware.  I’d like to thank them for being tolerant of my attention to detail, and to get this right the first time.  You’ll see that the rebuild steps are very simple.  Getting confirmation on this was not.

Step 1:  Reset the switch to defaults (make a backup of your old config, just in case)
enable
delete startup-config
reload

 
Step 2:  When prompted, follow the setup wizard in order to establish your management IP, etc. 
 
Step 3:  Put the switch into admin and configuration mode.
enable
configure

 
Step 4:  Establish Management Settings
hostname [yourstackhostname]
enable password [yourenablepassword]
spanning-tree mode rstp
flowcontrol

 
Step 5: Add the appropriate VLAN IDs to the database and setup interfaces.
vlan database
vlan 10
vlan 100
exit
interface vlan 1
exit
interface vlan 10
name Management
exit
interface vlan 100
name iSCSI
exit
ip address vlan 10
 
Step 6: Create an Etherchannel Group for Management Uplink
interface port-channel 1
switchport mode access
switchport access vlan 10
exit
NOTE: Because the switches are stacked, port one on each switch will be configured in this channel-group which can then be connected to their core switch or intermediate switch for management access. Port two on each switch can be used if they need to plug a laptop into the management VLAN, etc.
 
Step 7: Configure/assign Port 1 as part of the management channel-group:
interface ethernet 1/g1
switchport access vlan 10
channel-group 1 mode auto
exit
interface ethernet 2/g1
switchport access vlan 10
channel-group 1 mode auto
exit
 
Step 8: Configure Port 2 as Management Access Switchports (not part of the channel-group):
interface ethernet 1/g2
switchport access vlan 10
exit
interface ethernet 2/g2
switchport access vlan 10
exit
 
Step 9: Configure Ports 3-24 as iSCSI access Switchports
interface range ethernet 1/g3-1/g24
switchport access vlan 100
no storm-control unicast
spanning-tree portfast
mtu 9216
exit
interface range ethernet 2/g3-2/g24
switchport access vlan 100
no storm-control unicast
spanning-tree portfast
mtu 9216
exit
NOTE:  Binding the xg1 and xg2 interfaces into a port-channel is not required for stacking. 
 
Step 10: Exit from Configuration Mode
exit
 
Step 11: Save the configuration!
copy running-config startup-config

Step 12: Back up the configuration
console#copy startup-config tftp://[yourTFTPip]/conf.cfg

In hindsight, the most time consuming aspect of all of this was trying to confirm the exact settings for the 6224’s in an iSCSI SAN.  Running in second was shutting down all of my VMs, ESX hosts, and anything else that connected to the SAN switchgear.  The upgrade and the rebuild was relatively quick and trouble-free.  I’m thrilled to have this behind me now, and I hope that by passing this information along, you too will have a very simple working example to build your configuration off of.  As for the 6224’s, they are working fine now.  I will continue to keep my fingers crossed that Dell will eventually provide a way to update firmware to a stacked set of 6200 series switches without a lights out maintenance window.

How I use Dell/EqualLogic’s SANHQ in my environment

 

One of the benefits of investing in Dell/EqualLogic’s SAN solutions are the number of great tools included with the product, at no extra charge.  I’ve written in the past about leveraging their AutoSnapshot Manager for VM and application consistent snapshots and replicas.  Another tool that deserves a few words is SAN HeadQuarters (SANHQ). 

SANHQ allows for real-time and historical analysis of your EqualLogic arrays.  Many EqualLogic users are well versed with this tool, and may not find anything here that they didn’t already know.  But I’m surprised to hear that many are not.  So, what better way to help those unfamiliar with SANHQ than to describe how it helps me with my environment.

While the tool itself is “optional” in the sense that you don’t need to deploy it to use the EqualLogic arrays, it is an easy (and free) way to expose the powers of your storage infrastructure.  If you want to see what your storage infrastructure is doing, do yourself a favor and run SANHQ.   

Starting up the application, you might find something like this:

image

You’ll find an interesting assortment of graphs, and charts that help you decipher what is going on with your storage.  Take a few minutes and do a little digging.  There are various ways that it can help you do your job better.

 

Monitoring

Sometimes good monitoring is downright annoying.  It’s like your alarm clock next to the bed; it’s difficult to overlook, but that’s the point.  SANHQ has proven to be an effective tool for proactive monitoring and alerting of my arrays.  While some of these warnings are never fun, it’s biggest value is that it can help prevent those larger, much more serious problems, which always seem to be a series of small issues thrown together.  Here are some examples of how it has acted as the canary in the coalmine for me in my environment.

  • When I had a high number of TCP retransmits after changing out my SAN Switchgear, it was SANHQ that told me something was wrong.  EqualLogic Support helped me determine that my new switchgear wasn’t handling jumbo frames correctly. 
  • When I had a network port go down on the SAN, it was SANHQ that alerted me via email.  A replacement network cable fixed the problem, and the alarm went away.
  • If replication across groups is unable to occur, you’ll get notified right away that replication isn’t running.  The reasons for this can be many, but SANHQ usually gives you the first sign that something is up.  This works across physical topologies where your target my be at another site.
  • Under maintenance scenarios, you might find the need to pause replication on a volume, or on the entire group.  SANHQ will do a nice job of reminding you that it’s still not replicating, and will bug you at a regular interval that it’s still not running.

 

Analysis and Planning

SANHQ will allow you to see performance data at the group level, by storage pools, volumes, or volume collections.  One of the first things I do when spinning up a VM that uses guest attached volumes, is to jump into SANHQ, and see how those guest attached volumes are running.  How are the average IOPS? What about Latencies and Queue depth?  All of those can be found  easily in SANHQ, and can help put your mind at ease if you are concerned about your new virtualized Exchange or SQL servers.  Here is a screenshot of a 7 day history for SQL server with guest attached volumes, driving our SharePoint backend services.

image

The same can be done of course for VMFS volumes.  This information will compliment existing data one gathers from vCenter to understand if there are performance issues with a particular VMFS volume.

Often times monitoring and analysis isn’t about absolute numbers, but rather, allowing the user to see changes relative to previous conditions.  This is especially important for the IT generalist who doesn’t have time or the know-how for deep dive storage analysis, or have a dedicated Storage Administrator to analyze the data.  This is where the tool really shines.  For whatever type of data you are looking at, you can easily choose a timeline by the last hour, 8 hours, 1 day, 7 days, 30 days, etc.  The anomalies, if there are any, will stand out. 

image

Simply click on the Timeline that you want, and the historical data of the Group, member, volume, etc will show up below.

image

I find analyzing individual volumes (when they are VMFS volumes) and volume collections (when they are guest attached volumes) the most helpful in making sure that there are not any hotspots in I/O.  It can help you determine if a VM might be better served in a VMFS volume that hasn’t been demanding as much I/O as the one it’s currently in.

It can also play a role in future procurement.  Those 15k SAS drives may sound like a neat idea, but does your environment really need that when you decide to add storage?  Thinking about VDI?  It can be used to help determine I/O requirements.  Recently, I was on the phone with a friend of mine, Tim Antonowicz.  Tim is a Senior Solutions Architect from Mosaic Technology who has done a number of successful VDI deployments (and who recently started a new blog).  We were discussing the possibility of VDI in my environment, and one of the first things he asked of me was to pull various reports from SANHQ so that he could understand our existing I/O patterns.  It wasn’t until then that I noticed all of the great storage analysis offerings that any geek would love.  There are a number of canned reports that can be saved out as a pdf, html, csv, or other format to your liking.

image

Replication Monitoring

The value of SANHQ went way up for me when I started replication.  It will give you summaries of the each volume replicated.

image

If you click on an individual volume, it will help you see transfer sizes and replication times of the most recent replicas.  It also separates inbound replica data from outbound replica data.

image

While the times and the transfer rates will be skewed somewhat if you have multiple replica’s running (as I do), it is a great example on how you can understand patterns in changed data on a specific volume.  The volume captured above represents where one of my Domain Controllers lives.  As you can see, it’s pretty consistent, and doesn’t change much, as one would expect (probably not much more than the swap file inside the VM, but that’s another story).  Other kinds of data replicated will fluctuate more widely.  This is your way to see it.

 

Running SANHQ

SANHQ will live happily on a stand alone VM.  It doesn’t require much, but does need direct access to your SAN, and uses SNMP.  Once installed, SANHQ can be run directly on that VM, or the client-only application can be installed on your workstation for a little more convenience.  If you are replicating data, you will want SANHQ to connect to the source site, and the target site, for most effective use of the tool.

Improvements?  Sure, there are a number of things that I’d love to see.  Setting alarms for performance thresholds.  Threshold templates that you could apply to a volume (VMFS or native) that would help you understand the numbers (green = good.  Red = bad).  The ability to schedule reports, and define how and where they are posted.  Free pool space activity warnings (important if you choose to keep replica reserves low and leverage free pool space).  Array diagnostics dumps directly from SANHQ.  Programmatic access for scripting.  Improvements like these could make a useful product become indispensible in a production environment.

Replication with an EqualLogic SAN; Part 5

 

Well, I’m happy to say that replication to my offsite facility is finally up and running now.  Let me share with you the final steps to get this project wrapped up. 

You might recall that in my previous offsite replication posts, I had a few extra challenges.  We were a single site organization, so in order to get replication up and running, an infrastructure at a second site needed to be designed and in place.  My topology still reflects what I described in the first installment, but simple pictures don’t describe the work getting this set up.  It was certainly a good exercise in keeping my networking skills sharp.  My appreciation for the folks who specialize in complex network configurations, and address management has been renewed.  They probably seldom hear words of thanks for say, that well designed sub netting strategy.  They are an underappreciated bunch for sure.

My replication has been running for some time now, but this was all within the same internal SAN network.  While other projects prevented me from completing this sooner, it gave me a good opportunity to observe how replication works.

Here is the way my topology looks fully deployed.

image

Most Collocations or Datacenters give you about 2 square feet to move around, (only a slight exaggeration on the truth) so it’s not the place you want to be contemplating reasons why something isn’t working.  It’s also no fun realizing you don’t have the remote access you need to make the necessary modifications, and you don’t, or can’t drive to the CoLo.  My plan for getting this second site running was simple.  Build up everything locally (switchgear, firewalls, SAN, etc.) and change my topology at my primary site to emulate my the 2nd site.

Here is the way it was running while I worked out the kinks.

image

All replication traffic occurs over TCP port 3260.  Both locations had to have accommodations for this.  I also had to ensure I could manage the array living offsite.  Testing this out with the modified infrastructure at my primary site allowed me to verify traffic was flowing correctly.

The steps taken to get two SAN replication partners transitioned from a single network to two networks (onsite) were:

  1. Verify that all replication is running correctly when the two replication partners are in the same SAN Network
  2. You will need a way to split the feed from your ISP, so if you don’t have one already, place a temporary switch at the primary site on the outside of your existing firewall.  This will allow you to emulate the physical topology of the real site, while having the convenience of all of the equipment located at the primary site. 
  3. After the 2nd firewall (destined for the CoLo) is built and configured, place it on that temporary switch at the primary site.
  4. Place something (a spare computer perhaps) on the SAN segment of the 2nd firewall so you can test basic connectivity (to ensure routing is functioning, etc) between the two SAN networks. 
  5. Pause replication on both ends, take the target array and it’s switchgear offline. 
  6. Plug the target array’s Ethernet ports to the SAN switchgear for the second site, then change the IP addressing of the array/group so that it’s running under the correct net block.
  7. Re-enable replication and run test replicas.  Starting out with the Group Manager.  Then to ASM/VE, then onto ASM/ME.

It would be crazy not to take one step at a time on this, as you learn a little on each step, and can identify issues more easily.  Step 3 introduced the most problems, because traffic has to traverse routers that also are secure gateways.  Not only does one have to consider a couple of firewalls, you now run into other considerations that may be undocumented.  For instance.

  • ASM/VE replication occurs courtesy of vCenter.  But ASM/ME replication is configured inside the VM.  Sure, it’s obvious, but so obvious it’s easy to overlook.  That means any topology changes will require adjustments in each VM that utilize guest attached volumes.  You will need to re-run the “Remote Setup Wizard” to adjust the IP address of the target group that you will be replicating to.
  • ASM/ME also uses a VSS control channel to talk with the array.  If you changed the target array’s group and interface IP addresses, you will probably need to adjust what IP range will be allowed for VSS control.
  • Not so fast though.  VM’s that use guest iSCSI initiated volumes typically have those iSCSi dedicated virtual network cards set with no default gateway.  You never want to enter more than one default gateway on this sort of situation.  The proper way to do this will be to add a persistent static route.  This needs to be done before you run the remote Setup Wizard above.  Fortunately the method to do this hasn’t changed for at least a decade.  Just type in

route –p add [destinationnetwork] [subnetmask] [gateway] [metric]

  • Certain kinds of traffic that passes almost without a trace across a layer 2 segment shows up right away when being pushed through very sophisticated firewalls who’s default stances are deny all unless explicitly allowed.  Fortunately, Dell puts out a nice document on their EqualLogic arrays.
  • If possible, it will be easiest to configure your firewalls with route relationships between the source SAN and the target SAN.  It may complicate your rulesets (NAT relationships are a little more intelligent when it comes to rulesets in TMG), but it simplifies how each node is seeing each other.  This is not to say that NAT won’t work, but it might introduce some issues that wouldn’t be documented.

Step 7 exposed an unexpected issue; terribly slow replicas.  Slow even though it wasn’t even going across a WAN link.  We’re talking VERY slow, as in 1/300th the speed I was expecting.  The good news is that this problem had nothing to do with the EqualLogic arrays.  It was an upstream switch that I was using to split my feed from my ISP.  The temporary switch was not negotiating correctly, and causing packet fragmentation.  Once that switch was replaced, all was good.

The other strange issue was that even though replication was running great in this test environment, I was getting errors with VSS.  ASM/ME at startup would indicate “No control volume detected.”  Even though replicas were running, the replica’s can’t be accessed, used, or managed in any way.  After a significant amount of experimentation, I eventually opened up a case with Dell Support.  Running out of time to troubleshoot, I decided to move the equipment offsite so that I could meet my deadline.  Well, when I came back to the office, VSS control magically worked.  I suspect that the array simply needed to be restarted after I had changed the IP addressing assigned to it. 

My CoLo facility is an impressive site.  Located in the Westin Building in Seattle, it is also where the Seattle Internet Exchange (SIX) is located.  Some might think of it as another insignificant building in Seattle’s skyline, but it plays an important part in efficient peering for major Service Providers.  Much of the building has been converted from a hotel to a top tier, highly secure datacenter and a location in which ISP’s get to bridge over to other ISP’s without hitting the backbone.  Dedicated water and power supplies, full facility fail-over, and elevator shafts that have been remodeled to provide nothing but risers for all of the cabling.  Having a CoLo facility that is also an Internet Exchange Point for your ISP is a nice combination.

Since I emulated the offsite topology internally, I was able to simply plug in the equipment, and turn it on, with the confidence that it will work.  It did. 

My early measurements on my feed to the CoLo are quite good.  Since the replication times include buildup and teardown of the sessions, one might get a more accurate measurement on sustained throughput on larger replicas.  The early numbers show that my 30mbps circuit is translating to replication rates that range in the neighborhood of 10 to 12GB per hour (205MB per min, or 3.4MB per sec.).  If multiple jobs are running at the same time, the rate will be affected by the other replication jobs, but the overall throughput appears to be about the same.  Also affecting speeds will be other traffic coming to and from our site.

There is still a bit of work to do.  I will monitor the resources, and tweak the scheduling to minimize the overlap on the replication jobs.  In past posts, I’ve mentioned that I’ve been considering the idea of separating the guest OS swap files from the VM’s, in an effort to reduce the replication size.  Apparently I’m not the only one thinking about this, as I stumbled upon this article.  It’s interesting, but a nice amount of work.  Not sure if I want to go down that road yet.

I hope this series helped someone with their plans to deploy replication.  Not only was it fun, but it is a relief to know that my data, and the VM’s that serve up that data, are being automatically replicated to an offsite location.

Restoring an Exchange 2007 mailbox using EqualLogic’s ASM/ME

 

 

Ask most IT Administrators in small to medium sized organizations to recover an Exchange mailbox, and you’ll get responses like, “how important is it to recover it?”  and “How much of it is gone?”  You might even get the slightly patronizing “Oh, well you were close to your storage quota anyway”   This is IT-speak for “I don’t want to recover it” (labor intensive), “I’m not sure if I can recover it” (it didn’t work the last time they tried), or “I can’t recover it, and don’t really want to tell you that.”  Trust me, I’ve been there.

The recovery process has ranged anywhere from non-existent (Exchange 4.0) to  supposedly easy (according to the glossy ad of the 3rd party solution you might have purchased, but could never get to work correctly), to cautiously doable, but an incomplete solution, with later versions of Exchange.  As each year passes, I’m always hoping that technology can make the process easier, without pushing through another big purchase.

I got my wish.  Technology has indeed improved the process.  I recently had a user mailbox’s contents vanish.  Who knows what happened, but I had to get the mailbox back fast, so I had to familiarize myself with the process again.  This time, since my Exchange 2007 Server is virtualized, and the Exchange databases and logs reside on guest attached volumes, I was able to take advantage of EqualLogic’s “AutoSnapshot Manager Microsoft Edition” or ASM/ME. 

ASM/ME allowed me to easily recover an Exchange Storage Group, then mount it as a Recovery Storage Group (RSG).  From that point I could restore just that single mailbox on top of the existing mailbox.  What a refreshing discovery to see how simple the process has become.  And that, it just worked.  No weird errors to investigate, no tapes to fiddle with.  It was a complete solution for a mailbox recovery scenario.  The best part of all, was that its a function available free of charge to any EqualLogic user who is using the Host Integration Toolkit (HITKit) on their Exchange server. 

Here is how you do it.

1.  On the Exchange Server, open up ASM/ME, highlight the smart copy collection that you’d like to recover.  I say “Collection” because I want to recover the volume that has the DB on it and the volume that has the Transaction Logs on it at the same time.

2.  Select option to “Create Recover Storage Group”

3.  Select the desired Storage Group

4.  It will prompt for two drive letters not being used.  This will represent the location of the restored volumes.  So if the Exchange databases are on E: and the Transaction Logs are on F:, it might prompt you to used “G:” and “H:” respectively.

5.  It will complete with the following message
image

6.  Close out of ASM/ME, and launch the “Database Recovery Management” in the Toolbox section of the Exchange Management Console.  This leads to the “Exchange Troubleshooting Assistant in the Exchange Management Console (EMC).

7.  Run through the restoration process.  It will restore the selected mailbox on top of the existing mailbox.

8.  Once it is complete, as the dialog above instructs, you will need to dismount and logoff the smart copy collection set with ASM/ME after the RSG is removed.

The process was fast, and worked the very first time without error.  I’d still prefer to never have to recover a mailbox, but it is nice to know that now, thanks to ASM/ME and the Exchange Database Recovery Management tool, that its really easy to do.

 

Helpful Links

A more detailed guide on using RSGs in Exchange:
http://www.msexchange.org/tutorials/Working-Recovery-Storage-Groups-Exchange-2007.html 

Working with a Recovery Storage group in the Exchange Management Shell (EMS) instead of the EMC.
http://technet.microsoft.com/en-us/library/bb125197(EXCHG.80).aspx

Replication with an EqualLogic SAN; Part 4

 

If you had asked me 6+ weeks ago how far along my replication project would be on this date, I would have thought I’d be basking in the glory of success, and admiring my accomplishments.

…I should have known better.

Nothing like several IT emergencies unrelated to this project to turn one’s itinerary into garbage.  A failed server (an old physical storage server that I don’t have room on my SAN for), a tape backup autoloader that tanked, some Exchange Server and Domain Controller problems, and a host of other odd things that I don’t even want to think about.  It’s overlooked how much work it takes to keep an IT infrastructure from not losing any ground from the day before.  At times, it can make you wonder how any progress is made on anything.

Enough complaining for now.  Lets get back to it.  

 

Replication Frequency

For my testing, all of my replication is set to occur just once a day.  This is to keep it simple, and to help me understand what needs to be adjusted when my offsite replication is finally turned up at the remote site.

I’m not overly anxious to turn up the frequency even if the situation allows.  Some pretty strong opinions exist on how best to configure the frequency of the replicas.  Do a little bit with a high frequency, or a lot with a low frequency.  What I do know is this.  It is a terrible feeling to lose data, and one of the more overlooked ways to lose data is for bad data to overwrite your good data on the backups before you catch it in time to stop it.  Tapes, disk, simple file cloning, or fancy replication; the principal is the same, and so is the result.   Since the big variable is retention period, I want to see how much room I have to play with before I decide on frequency.  My purpose of offsite replication is disaster recovery.  …not to make a disaster bigger.

 

Replication Sizes

The million dollar question has always been how much changed data, as perceived from the SAN will occur for a given period of time, on typical production servers.  It is nearly impossible to know this until one is actually able to run real replication tests.  I certainly had no idea.  This would be a great feature for Dell/EqualLogic to add to their solution suite.  Have a way for a storage group to run in a simulated replication where it simply collects statistics that would accurately reflect the amount of data that would be replicate during the test period.  What a great feature for those looking into SAN to SAN replication.

Below are my replication statistics for a 30 day period, where the replicas were created once per day, after the initial seed replica was created.

Average data per day per VM

  • 2 GB for general servers (service based)
  • 3 GB for servers with guest iSCSI attached volumes.
  • 5.2 GB for code compiling machines

Average data per day for guest iSCSI attached data volumes

  • 11.2 GB for Exchange DB and Transaction logs (for a 50GB database)
  • 200 MB for a SQL Server DB and Transaction logs
  • 2 GB for SharePoint DB and Transaction logs

The replica sizes for the VM’s were surprisingly consistent.  Our code compiling machines had larger replica sizes, as they write some data temporarily to the VM’s during their build processes.

The guest iSCSI attached data volumes naturally varied more from day-to-day activities.  Weekdays had larger amounts of replicated data than weekends.  This was expected.

Some servers, and how they generate data may stick out like sore thumbs.  For instance, our source code control server uses a crude (but important) way of an application layer backup.  The result is that for 75 GB worth of repositories, it would generate 100+ GB of changed data that it would want to replicate.  If the backup mechanism (which is a glorified file copy and package dump) is turned off, the amount of changed data is down to a very reasonable 200 MB per day.  This is a good example of how we will have to change our practices to accommodate replication.

 

Decreasing the amount of replicated data

Up to this point, the only step to reduce the amount of data replication is the adjustment made in vCenter to move the VM’s swap files off onto another VMFS volume that will not be replicated.  That of course only affects the VM’s paging files – not the guest VM’s paging files that are controlled by the OS.  I suspect that a healthy amount of changed data on the VMs are the paging files for the OS.  The amount of changed data on those VM’s looked suspiciously similar to the amount of RAM assigned to the VM.  There typically is some correlation to how much RAM an OS has to run with, and the size of the page file.  This is pure speculation at this point, but certainly worth looking into.

The next logical step would be to figure out what could be done to reconfigure VM’s to perhaps place their paging/swap files in a different, non-replicated location.   Two issues come to mind when I think about this step. 

1.)  This adds an unknown amount of complexity (for deploying, and restoring) to the systems running.  You’d have to be confident in the behavior of each OS type when it comes to restoring from a replica where it expects to see a page file in a certain location, but does not.  How scalable this approach is would also need to be asked.  It might be okay for a few machines, but how about a few hundred?  I don’t know.

2.)  It is unknown as to how much of a payoff there will be.  If the amount of data per VM gets reduced by say, 80%, then that would be pretty good incentive.  If it’s more like 10%, then not so much.  It’s disappointing that there seems to be only marginal documentation on making such changes.  I will look to test this when I have some time, and report anything interesting that I find along the way.

 

The fires… unrelated, and related

One of the first problems to surface recently were issues with my 6224 switches.  These were the switches that I put in place of our 5424 switches to provide better expandability.  Well, something wasn’t configured correctly, because the retransmit ratio was high enough that SANHQ actually notified me of the issue.  I wasn’t about to overlook this, and reported it to the EqualLogic Support Team immediately.

I was able to get these numbers under control by reconfiguring the NIC’s on my ESX hosts to talk to the SAN with standard frames.  Not a long term fix, but for the sake of the stability of the network, the most prudent step for now.

After working with the 6224’s, they do seem to behave noticeably different than the 5242’s.  They are more difficult to configure, and the suggested configurations from the Dell documentation seem were more convoluted and contradictory.  Multiple documents and deployment guides had inconsistent information.  Technical Support from Dell/EqualLogic has been great in helping me determine what the issue is.  Unfortunately some of the potential fixes can be very difficult to execute.  Firmware updates on a stacked set of 6224’s will result in the ENTIRE stack rebooting, so you have to shut down virtually everything if you want to update the firmware.  The ultimate fix for this would be a revamp of the deployment guides (or lets try just one deployment guide) for the 6224’s that nullifies any previous documentation.  By way of comparison, the 5424 switches were, and are very easy to deploy. 

The other issue that came up was some unexpected behavior regarding replication, and it’s use of free pool space.  I don’t have any empirical evidence to tie these two together, but this is what I had observed.

During this past month in which I had an old physical storage server fail on me, there was a moment where I had to provision what was going to be a replacement for this box, as I wasn’t even sure if the old physical server was going to be recoverable.  Unfortunately, I didn’t have a whole lot of free pool space on my array, so I had to trim things up a bit, to get it to squeeze on there.  Once I did, I noticed all sorts of weird behavior.

1.  Since my replication jobs (with ASM/ME and ASM/VE) leverage the free pool space for the creation of temporary replica/snap that is created on the source array, this caused problems.  The biggest one was that my Exchange server would completely freeze during it’s ASM/ME snapshot process.  Perhaps I had this coming to me, because I deliberately configured it to use free pool space (as opposed to a replica reserve) for it’s replication.  How it behaved caught me off guard, and made it interesting enough for me to never want to cut it close on free pool space again.

2.  ASM/VE replica jobs also seems to behave odd with very little free pool space.  Again, this was self inflicted because of my configuration settings.  It left me desiring a feature that would allow you to set a threshold so that in the event of x amount of free pool space remaining, replication jobs would simply not run.  This goes for ASM/VE and ASM/ME.

Once I recovered that failed physical system, I was able to remove that VM I set aside for emergency turn up.  That increased my free pool space back up over 1TB, and all worked well from that point on. 

 

Timing

Lastly, one subject matter came up that doesn’t show up in any deployment guide I’ve seen.  The timing of all this protection shouldn’t be overlooked.  One wouldn’t want to stack several replication jobs on top of each other that use the same free pool space, but haven’t had the time to replicate.  Other snapshot jobs, replicas, consistency checks, traditional backups, etc should be well coordinated to keep overlap to a minimum.  If you are limited on resources, you may also be able to use timing to your advantage.  For instance, set your daily replica of your Exchange database to occur at 5:00am, and your daily snapshot to occur at 5:00pm.  That way, you have reduced your maximum loss period from 24 hours to 12 hours, just by offsetting the times.

Follow

Get every new post delivered to your Inbox.

Join 813 other followers