Well, I’m happy to say that replication to my offsite facility is finally up and running now. Let me share with you the final steps to get this project wrapped up.
You might recall that in my previous offsite replication posts, I had a few extra challenges. We were a single site organization, so in order to get replication up and running, an infrastructure at a second site needed to be designed and in place. My topology still reflects what I described in the first installment, but simple pictures don’t describe the work getting this set up. It was certainly a good exercise in keeping my networking skills sharp. My appreciation for the folks who specialize in complex network configurations, and address management has been renewed. They probably seldom hear words of thanks for say, that well designed sub netting strategy. They are an underappreciated bunch for sure.
My replication has been running for some time now, but this was all within the same internal SAN network. While other projects prevented me from completing this sooner, it gave me a good opportunity to observe how replication works.
Here is the way my topology looks fully deployed.
Most Collocations or Datacenters give you about 2 square feet to move around, (only a slight exaggeration on the truth) so it’s not the place you want to be contemplating reasons why something isn’t working. It’s also no fun realizing you don’t have the remote access you need to make the necessary modifications, and you don’t, or can’t drive to the CoLo. My plan for getting this second site running was simple. Build up everything locally (switchgear, firewalls, SAN, etc.) and change my topology at my primary site to emulate my the 2nd site.
Here is the way it was running while I worked out the kinks.
All replication traffic occurs over TCP port 3260. Both locations had to have accommodations for this. I also had to ensure I could manage the array living offsite. Testing this out with the modified infrastructure at my primary site allowed me to verify traffic was flowing correctly.
The steps taken to get two SAN replication partners transitioned from a single network to two networks (onsite) were:
- Verify that all replication is running correctly when the two replication partners are in the same SAN Network
- You will need a way to split the feed from your ISP, so if you don’t have one already, place a temporary switch at the primary site on the outside of your existing firewall. This will allow you to emulate the physical topology of the real site, while having the convenience of all of the equipment located at the primary site.
- After the 2nd firewall (destined for the CoLo) is built and configured, place it on that temporary switch at the primary site.
- Place something (a spare computer perhaps) on the SAN segment of the 2nd firewall so you can test basic connectivity (to ensure routing is functioning, etc) between the two SAN networks.
- Pause replication on both ends, take the target array and it’s switchgear offline.
- Plug the target array’s Ethernet ports to the SAN switchgear for the second site, then change the IP addressing of the array/group so that it’s running under the correct net block.
- Re-enable replication and run test replicas. Starting out with the Group Manager. Then to ASM/VE, then onto ASM/ME.
It would be crazy not to take one step at a time on this, as you learn a little on each step, and can identify issues more easily. Step 3 introduced the most problems, because traffic has to traverse routers that also are secure gateways. Not only does one have to consider a couple of firewalls, you now run into other considerations that may be undocumented. For instance.
- ASM/VE replication occurs courtesy of vCenter. But ASM/ME replication is configured inside the VM. Sure, it’s obvious, but so obvious it’s easy to overlook. That means any topology changes will require adjustments in each VM that utilize guest attached volumes. You will need to re-run the “Remote Setup Wizard” to adjust the IP address of the target group that you will be replicating to.
- ASM/ME also uses a VSS control channel to talk with the array. If you changed the target array’s group and interface IP addresses, you will probably need to adjust what IP range will be allowed for VSS control.
- Not so fast though. VM’s that use guest iSCSI initiated volumes typically have those iSCSi dedicated virtual network cards set with no default gateway. You never want to enter more than one default gateway on this sort of situation. The proper way to do this will be to add a persistent static route. This needs to be done before you run the remote Setup Wizard above. Fortunately the method to do this hasn’t changed for at least a decade. Just type in
route –p add [destinationnetwork] [subnetmask] [gateway] [metric]
- Certain kinds of traffic that passes almost without a trace across a layer 2 segment shows up right away when being pushed through very sophisticated firewalls who’s default stances are deny all unless explicitly allowed. Fortunately, Dell puts out a nice document on their EqualLogic arrays.
- If possible, it will be easiest to configure your firewalls with route relationships between the source SAN and the target SAN. It may complicate your rulesets (NAT relationships are a little more intelligent when it comes to rulesets in TMG), but it simplifies how each node is seeing each other. This is not to say that NAT won’t work, but it might introduce some issues that wouldn’t be documented.
Step 7 exposed an unexpected issue; terribly slow replicas. Slow even though it wasn’t even going across a WAN link. We’re talking VERY slow, as in 1/300th the speed I was expecting. The good news is that this problem had nothing to do with the EqualLogic arrays. It was an upstream switch that I was using to split my feed from my ISP. The temporary switch was not negotiating correctly, and causing packet fragmentation. Once that switch was replaced, all was good.
The other strange issue was that even though replication was running great in this test environment, I was getting errors with VSS. ASM/ME at startup would indicate “No control volume detected.” Even though replicas were running, the replica’s can’t be accessed, used, or managed in any way. After a significant amount of experimentation, I eventually opened up a case with Dell Support. Running out of time to troubleshoot, I decided to move the equipment offsite so that I could meet my deadline. Well, when I came back to the office, VSS control magically worked. I suspect that the array simply needed to be restarted after I had changed the IP addressing assigned to it.
My CoLo facility is an impressive site. Located in the Westin Building in Seattle, it is also where the Seattle Internet Exchange (SIX) is located. Some might think of it as another insignificant building in Seattle’s skyline, but it plays an important part in efficient peering for major Service Providers. Much of the building has been converted from a hotel to a top tier, highly secure datacenter and a location in which ISP’s get to bridge over to other ISP’s without hitting the backbone. Dedicated water and power supplies, full facility fail-over, and elevator shafts that have been remodeled to provide nothing but risers for all of the cabling. Having a CoLo facility that is also an Internet Exchange Point for your ISP is a nice combination.
Since I emulated the offsite topology internally, I was able to simply plug in the equipment, and turn it on, with the confidence that it will work. It did.
My early measurements on my feed to the CoLo are quite good. Since the replication times include buildup and teardown of the sessions, one might get a more accurate measurement on sustained throughput on larger replicas. The early numbers show that my 30mbps circuit is translating to replication rates that range in the neighborhood of 10 to 12GB per hour (205MB per min, or 3.4MB per sec.). If multiple jobs are running at the same time, the rate will be affected by the other replication jobs, but the overall throughput appears to be about the same. Also affecting speeds will be other traffic coming to and from our site.
There is still a bit of work to do. I will monitor the resources, and tweak the scheduling to minimize the overlap on the replication jobs. In past posts, I’ve mentioned that I’ve been considering the idea of separating the guest OS swap files from the VM’s, in an effort to reduce the replication size. Apparently I’m not the only one thinking about this, as I stumbled upon this article. It’s interesting, but a nice amount of work. Not sure if I want to go down that road yet.
I hope this series helped someone with their plans to deploy replication. Not only was it fun, but it is a relief to know that my data, and the VM’s that serve up that data, are being automatically replicated to an offsite location.
vmpete.com 
Hi there,
Your 5 part series on replication is excellent, and really helped me along. I’m having some issues getting replication going across a VPN tunnel between two Cisco ASA devices, and was wondering if I could get your advice.
Feel free to send me an email and I will pass along the details.
Thanks and keep up the good writing.
Mike
Thanks for the kind words Mike. Sure, I’ll be happy to offer whatever input I can. I’ll send you a note offline.
Great series.. Very good stuff!!
Got a question for you though. For your W2K8R2 servers, how are you handling Windows OS pagefiles??? Do you have any tips on what you do?
I started creating a separate datastore on a different volume for the pagefile, but the issues is, ASM/VE reports multiple datastores and no smart copies can be made. So, I’m back to putting the pagefile on the C: volume, it’s not a huge deal but it just takes up more SAN space.
I’d love to hear how your setup is.. thanks!
Bill
Hi Bill,
Great questions! Well, right now I’m still running the pagefile inside the local OS drive (C:\). I had not split them out for a couple of different reasons. 1.) No time yet to optimize. 2.) No time to test spinning them up at a remote location with the non replicated pagefile. There are several things to consider, (as mentioned in this article: http://vmguy.com/wordpress/index.php/archives/1525 ), and I wanted to be sure that I had all my bases covered before I put the practice into production use. The other thing that showed up for me was my ROI. I had several data volumes that if I had put in some work on those, the return on replication improvements (or lack of replicating unneeded data) was a WAY better pay-off for me. But with that said, I definately would like to post about the topic of splitting the guest VM swap/page file to a non replicating volume. It’s just not covered as much as it should be.
You have also noticed a few shortcomings of ASM/VE. Thats the bad news. The good news is that there is ample room for improvement, and that is exactly what Dell EqualLogic is doing. They are in tune with some of the features that would make it more flexible, and I believe are working hard to improve it. The app is still a little rough around the edges, but definately a good starting point, with a lot of exciting changes coming.
Thanks for reading, and please let me know if there is anything I can do for you.
Thanks for your reply. I hear you on ROI.. I have 4 Equallogic shelves and am configuring vSphere 4.1 with a PS3700X and a PS6000XV. I plan to replicate(haven’t gotten there yet) to a PS6500, so your testing and documentation has been great!
With ASM/VE one thing I can not figure out is why 2 of my datastores will not allow me to create Smart Copies. In ASM/VE when I select these volumes , it reports Smart Copies ‘Unsupported’ / Volume: unsupported storage!
Have you ran into this? I have other datastores that are supported and work find, but the last 2 volumes I’ve created show unsupported, my search has come up with nothing!
Thanks again and please continue to share with us as you go down this journey !!
Wow… great project you have underway. Keep me posted on how things go, and if wish, we can exchange emails for a little easier communication.
I have not run into the scenario you described with two of your datastores not allowing smart copies. The “unsupported storage” message is very peculiar. My only thought is that it’s leveraging vCenters API, so perhaps you have a volume in there that isn’t currently seen by vCenter? Or something to that effect? Perhaps create a new VMFS volume and do a storage vmotion into there, then try it? (I’ve ended up recreating my VMFS volumes for various reasons and have resorted to do this a few times.
Anyway, let me know how it goes.
Yes, this is odd. It could be I’m forgetting to do something during the configuration, but for the life of me I can’t figure out what. I’d be glad to converse offline.. Can you see my email address from my post? If so, shoot me an email.. maybe just bouncing things around I can see what the issue is. Then post about it for others!
Brilliant set of articles around EQL replication in virtualized environments.
Got a couple of questions:
1- Could this concept of spreading vmdks of a VM in different datastores (thus reducing replication size), be applied to File Server VMs? Idea would be giving different replication policies to volumes with temp, old or active data.
2- Way volumes are presented to VMs has an impact in both Replication and local Backup. Sometimes what simplify things in one area, make things more complicated in the other. Any lessons learned backing up your enviromnet?
Thanks !!
Thank you so much Luis for the compliments. I’ve been caught off guard a bit as to how much these particular posts have helped people. It’s great to hear! As to your questions…
1. Possibly. With ASM/VE 2.x, one was limited to protecting VM’s that had all of their vmdks in the same datastore. The newly released ASM/VE 3.0 does away with that requirement, so if offers up a number of new options. What you might want to consider for your file server VM’s is running the volumes that house the files as guest attached volumes. Then, you could use ASM/ME to schedule protecting the data at a frequency of your liking, while you use ASM/VE to protect the VM serving up that data at more infrequent intervals. I’m a big fan of guest attached volumes coupled with the use of ASM/ME, but understand that it may not align well if an organization has already invested money in one of the great commercial backup packages out there (e.g. Veeam) where it requires all volumes to be visible in vcenter (guest attached volumes are not).
2. You bring up a great point here. Simplifying on thing can come at a price elsewhere. Wow… I could go on and on about this topic (I think you’ve helped me decide on the subject of my next post). Yes, some lessons learned, now that the dust has settled a bit.
a.) When looking to tune your replication (to minimize the amount replicated), look at what other things might be sabotaging your efforts. Are there other scripts or backup technologies that are staging backup data to disk? Are you defragging your data? If so, that is probably generating a huge amount of replication traffic. For me, this was a much larger issue than replication of each VM’s guest swap partition.
b.) Let go of the term “backup” and focus in on “protection.” This sounds like semantics, but really allows you to re-evaluate what you need to protect, why you need to protect it, and how it should be protected. This will help clear up how one might want to used smartcopy snapshots for short term protection, replicas for offsite and intermediate term protection, and tapes for long term archiving.
c.). Do your own audit of scenarios for the protection of these systems. What happens if server x fails for some reason. What are you going to do? How are you going to handle if there is a problem with the local smartcopy? etc.
d.). The smartcopy snapshot and replication tools are great, but you don’t necessarily have to use them for recovery. If you need to be careful on a particular recovery, just clone that snapshot or replica, turn it online, and pull up the VM (or the files) in an isolated environment, and cherry pick the files you need.
e.) Don’t try to build the perfect system right from the start. It won’t happen, and will just overwhelm you. Take iterative steps at accomplishing the task. That way you can provide, and demonstrate real value right away to the key stakeholders. Then work out the smaller problems later. In my particular case, I had to keep reminding myself that I was building up an environment to replicate to. The scope didn’t include the ability for a full on hot-cutover to a new site (e.g. VMWare’s Site Recovery Manager).
Hope this helps!
– Pete
Thanks for posting your experiences on this. How did you come up with your bandwidth requirements for the replication? Was 30 megabits just a guess, or did you use info from SAN HQ for this?
Hi Tim,
Much of it was a rough guess early on, but used SANHQ once I was doing my test replicas locally. If had to do it over again, or nail down the requirements more, I would have approached the estimate in the following ways.
1. Look at the volumes you want to replicate, and look at the patterns for the amount of space say a daily snapshot takes. It won’t be an exact translation, but it will be pretty darned close.
2. If you have the opportunities to run replicas locally first (which I highly encourage!), I would do so, then use SANHQ to monitor the change rate patters from there.
Does that answer your question?
Yeah, that’s what I was thinking. It would be convenient to get a good idea of the bandwidth needs before getting the replica target box, that way we could shop bandwidth at the same time, but this should work for me.
I have already started to provision machines with dedicated swap/pagefile volumes, so this should help too.
Thanks!
I am surprised that your SANs are in a different subnet. Is it not recommended to have both SANs in the same subnet. I am trying to do the exactly same thing. I guess, I will need your help when I get to the advanced stage.
Actually, you might be confusing a few things here. You are absolutely correct in saying that you will want your SAN environment to be a flat, single netblock design. But this pertains to the hosts and the SAN arrays that are all connected by the concentrators (the SAN switch stack). That is exactly what I have. In cases where you want to replicate the data, across different geological boundaries, it is almost a necessity to have them on different networks. Just like I have, one would have two different network peers, and you’d be letting that replication traffic traverse across the network to the remote location. It is perfectly fine to route SAN replica traffic onto different networks.
Does that answer your question?
First of all, thank you for your prompt reponse. At the moment, both SANs are in the primary site and replication is enabled using the same switch and the same subnet which is 10.10.2.x/24. My plan is that my promary site SAN network will be on 10.10.2.x/24 and my recovery site SAN network will also be on 10.10.2.x/24. They will be both connected by the site to site VPN though firewall (they are in different geological boundaries). I thought, this should work. But off course my knowledge is limited in this.
You are welcome. Yes, much of my early testing was under that exact same arrangement. The source and the target living on the same network at the primary site. However, once you move the target array to the remote location, you will really want to consider a traditional routed arrangement (No NAT, and no VPN). The former won’t work, and the latter you will run into difficulties.
Do does it mean that I will have to run the leased line between the two sites as appose to site to site VPN? That would be quite expensive Arghhh!!!!. But at the same time, you must have saved my a lot of testing time using the site to site VPN.
Does it mean that I will have to run the leased line between the two sites as appose to site to site VPN? That would be quite expensive Arghhh!!!!. But at the same time, you must have saved me a lot of testing time using the site to site VPN.
You have a variety of options in front of you. The main point being that you will want a traditional routed path. Something like PrimarySiteSAN–>PublicInterface–>AcrossTheNet–>PublicInterface—>TargetSite SAN. I use a pair of TMG 2010 based firewall appliances to secure the traffic. Works great!
In my case, I will be using two SonicWall firewall. I have TMG 2010 but this is being used for the clients gateway. Before seeing your post, I was thinking to use the site to site VPN between the two sonicwalls and I had a discussion with Dell about this scenerio and they said that it will be fine. Now I will see what leased line costs us.
I know that I bombarded with lot of questions, could I ask you one more thing? you mentioned that you used temporary switch to emulate the other site. For instance, if I pu the temporary switch on my ISP router and we have a block of IP 62.133.9.240/30 than primary SAN will be using 62.133.9.242 with the gateway of 62.133.9.241 and the secondary SAN will be using 62.133.9.243 with the same gateway.
You may want to go ahead and go with a site to site VPN for now. It may not be the ideal scenario, and it may introduce some other problems, but it may be the right solution for you. You’ll have to determine that.
Unfortunately, you would need a larger netblock to test. A /30 subnet yields only two usable addresses; the interface for the ISP’s router, and the interface for your router. You can’t include the net ID nor the broadcast address in the count of available addresses.
No doubt, it was a great help. As it is in the testing phase so I have a liberty to test both options. I will get back to you with the update once I tested these options.