IOPS

Most know Iometer as the go-to synthetic I/O measuring tool used to simulate real workload conditions. Well, somewhere, somehow, someone forgot the latter part of that sentence, which is why it ends up being so misused and abused. How many of us have seen a storage solution delivering 6 figure IOPS using Iometer, only to find that they are running a 100% read, 512 byte 100% sequential access workload simulation. Perfect for the two people on the planet that those specifications might apply to. For the rest of us, it doesn’t help much. So why would they bother running that sort of unrealistic test? Pure, unapologetic number chasing.

The unfortunate part is that sometimes this leads many to simply dismiss Iometer results. That is a shame really, as it can provide really good data if used in the correct way. Observing real world data will tell you a lot of things, but the sporadic nature of real workloads make it difficult to use for empirical measurement – hence the need for simulation.

So, what are the correct settings to use in Iometer? The answer is completely dependent on what you are trying to accomplish. The race for a million IOPS by your favorite storage vendor really means nothing if their is no correlation between their simulated workload, and your real workload. Maybe IOPS isn’t even an issue for you. Perhaps your applications are struggling with poor latency. The challenge is to emulate your environment with a synthetic workload that helps you understand how a potential upgrade, new array, or optimization might be of benefit.

The mysteries of workloads
Creating a synthetic workload representing your real workload assumes one thing; that you know what your real workload really is. This can be more challenging that one might think, as many storage monitoring tools do not help you understand the subtleties of patterns to the data that is being read or written.

Most monitoring tools tend to treat all I/O equally. By that I mean, if over a given period of time, say you have 10 million I/Os occur. Let’s say your monitoring tells you that you average 60% reads and 40% writes. What is not clear is how many of those reads are multiple reads of the same data or completely different, untouched data. It also doesn’t tell you if the writes are overwriting existing blocks (which might be read again shortly thereafter) or generating new data. As more and more tiered storage mechanisms comes into play, understanding this aspect of your workload is becoming extremely important. You may be treating your I/Os equally, but the tiered storage system using sophisticated caching algorithms certainly do not.

How can you gain more insight? Use every tool at your disposal. Get to know your applications, and the duty cycles around them. What are your peak hours? Are they in the middle of the day, or in the middle of the night when backups are running?

Suggestions on Iometer settings
You may find that the settings you choose for Iometer yields results from your shared storage that isn’t nearly as good as you thought. But does it matter? If it is an accurate representation of your real workload, not really. What matters is if are you able to deliver the payload from point a to point b to meet your acceptance criteria (such as latency, throughput, etc.). The goal would be to represent that in a synthetic workload for accurate measurement and comparison.

With that in mind, here are some suggestions for the next time you set up those Iometer runs.

1. Read/write ratio. Choose a realistic read/write ratio representing your workload. With writes, RAID penalties can hurt your effective performance by quite a bit, so if you don’t have an idea of what this ratio currently is, it’s time for you to find out.

2. Transfer request size. Is your payload the size of a ball bearing, or a bowling ball? Applications and operating systems vary on what size is used. Use your monitoring systems to best determine what your environment consists of.

3. Disk size. Use the "maximum disk size" in multiples of 1048576, which is a 1GB file. Throwing a bunch of zeros in there might fill up your disk with Iometer’s test file. Depending on your needs, a setting of 2 to 20 GB might be a good range to work with.

4. Number of outstanding I/Os. This needs to be high enough so that the test can keep sending I/O requests to it as the storage is fulfilling requests to it. A setting of 32 is pretty common.

5. Alignment of I/O. Many of the standard Iometer ICF files you find were built for physical drives. It has the "Align I/Os on:" setting to "Sector boundaries" When running tests on a storage array, this can lead to inconsistent results, so it is best to align on 4K or 512 bytes.

6. Ramp up time. Offer at least a minute of ramp up time.

7. Run time. Some might suggest running simulations long enough to exhaust all caching, so that you can see "real" throughput. While I understand the underlying reason for this statement, I believe this is missing the point. Caching is there in the first place to take advantage of a working set of warm and cold data, bursts, etc. If you have a storage solution that satisfies the duty cycles that exists in your environment, that is the most important part.

8. Number of workers. Let this spawn automatically to the number of logical processors in your VM. It might be overkill in many cases because of terrible multithreading abilities of most applications, but its a pretty conventional practice.

9. Multiple Iometer instances. Not really a setting, but more of a practice. I’ve found running multiple tests a way to better understand how a storage solution will react under load as opposed to on it’s own. It is shared storage after all.

Disclaimer
If you were looking for this to be the definitive post on Iometer, that isn’t what I was shooting for. There are many others who are much more qualified to speak to the nuances of Iometer than me. What I hope to do is to offer a little practical perspective on it’s use, and how it can help you. So next time you run Iometer, think about what you are trying to accomplish, and let go of the number chasing. Understand your workloads, and use the tool to help you improve your environment.

One of the benefits of investing in Dell/EqualLogic’s SAN solutions are the number of great tools included with the product, at no extra charge. I’ve written in the past about leveraging their AutoSnapshot Manager for VM and application consistent snapshots and replicas. Another tool that deserves a few words is SAN HeadQuarters (SANHQ).

SANHQ allows for real-time and historical analysis of your EqualLogic arrays. Many EqualLogic users are well versed with this tool, and may not find anything here that they didn’t already know. But I’m surprised to hear that many are not. So, what better way to help those unfamiliar with SANHQ than to describe how it helps me with my environment.

While the tool itself is “optional” in the sense that you don’t need to deploy it to use the EqualLogic arrays, it is an easy (and free) way to expose the powers of your storage infrastructure. If you want to see what your storage infrastructure is doing, do yourself a favor and run SANHQ.

Starting up the application, you might find something like this:

You’ll find an interesting assortment of graphs, and charts that help you decipher what is going on with your storage. Take a few minutes and do a little digging. There are various ways that it can help you do your job better.

Monitoring

Sometimes good monitoring is downright annoying. It’s like your alarm clock next to the bed; it’s difficult to overlook, but that’s the point. SANHQ has proven to be an effective tool for proactive monitoring and alerting of my arrays. While some of these warnings are never fun, it’s biggest value is that it can help prevent those larger, much more serious problems, which always seem to be a series of small issues thrown together. Here are some examples of how it has acted as the canary in the coalmine for me in my environment.

When I had a high number of TCP retransmits after changing out my SAN Switchgear, it was SANHQ that told me something was wrong. EqualLogic Support helped me determine that my new switchgear wasn’t handling jumbo frames correctly.
When I had a network port go down on the SAN, it was SANHQ that alerted me via email. A replacement network cable fixed the problem, and the alarm went away.
If replication across groups is unable to occur, you’ll get notified right away that replication isn’t running. The reasons for this can be many, but SANHQ usually gives you the first sign that something is up. This works across physical topologies where your target my be at another site.
Under maintenance scenarios, you might find the need to pause replication on a volume, or on the entire group. SANHQ will do a nice job of reminding you that it’s still not replicating, and will bug you at a regular interval that it’s still not running.

Analysis and Planning

SANHQ will allow you to see performance data at the group level, by storage pools, volumes, or volume collections. One of the first things I do when spinning up a VM that uses guest attached volumes, is to jump into SANHQ, and see how those guest attached volumes are running. How are the average IOPS? What about Latencies and Queue depth? All of those can be found easily in SANHQ, and can help put your mind at ease if you are concerned about your new virtualized Exchange or SQL servers. Here is a screenshot of a 7 day history for SQL server with guest attached volumes, driving our SharePoint backend services.

The same can be done of course for VMFS volumes. This information will compliment existing data one gathers from vCenter to understand if there are performance issues with a particular VMFS volume.

Often times monitoring and analysis isn’t about absolute numbers, but rather, allowing the user to see changes relative to previous conditions. This is especially important for the IT generalist who doesn’t have time or the know-how for deep dive storage analysis, or have a dedicated Storage Administrator to analyze the data. This is where the tool really shines. For whatever type of data you are looking at, you can easily choose a timeline by the last hour, 8 hours, 1 day, 7 days, 30 days, etc. The anomalies, if there are any, will stand out.

Simply click on the Timeline that you want, and the historical data of the Group, member, volume, etc will show up below.

I find analyzing individual volumes (when they are VMFS volumes) and volume collections (when they are guest attached volumes) the most helpful in making sure that there are not any hotspots in I/O. It can help you determine if a VM might be better served in a VMFS volume that hasn’t been demanding as much I/O as the one it’s currently in.

It can also play a role in future procurement. Those 15k SAS drives may sound like a neat idea, but does your environment really need that when you decide to add storage? Thinking about VDI? It can be used to help determine I/O requirements. Recently, I was on the phone with a friend of mine, Tim Antonowicz. Tim is a Senior Solutions Architect from Mosaic Technology who has done a number of successful VDI deployments (and who recently started a new blog). We were discussing the possibility of VDI in my environment, and one of the first things he asked of me was to pull various reports from SANHQ so that he could understand our existing I/O patterns. It wasn’t until then that I noticed all of the great storage analysis offerings that any geek would love. There are a number of canned reports that can be saved out as a pdf, html, csv, or other format to your liking.

Replication Monitoring

The value of SANHQ went way up for me when I started replication. It will give you summaries of the each volume replicated.

If you click on an individual volume, it will help you see transfer sizes and replication times of the most recent replicas. It also separates inbound replica data from outbound replica data.

While the times and the transfer rates will be skewed somewhat if you have multiple replica’s running (as I do), it is a great example on how you can understand patterns in changed data on a specific volume. The volume captured above represents where one of my Domain Controllers lives. As you can see, it’s pretty consistent, and doesn’t change much, as one would expect (probably not much more than the swap file inside the VM, but that’s another story). Other kinds of data replicated will fluctuate more widely. This is your way to see it.

Running SANHQ

SANHQ will live happily on a stand alone VM. It doesn’t require much, but does need direct access to your SAN, and uses SNMP. Once installed, SANHQ can be run directly on that VM, or the client-only application can be installed on your workstation for a little more convenience. If you are replicating data, you will want SANHQ to connect to the source site, and the target site, for most effective use of the tool.

Improvements? Sure, there are a number of things that I’d love to see. Setting alarms for performance thresholds. Threshold templates that you could apply to a volume (VMFS or native) that would help you understand the numbers (green = good. Red = bad). The ability to schedule reports, and define how and where they are posted. Free pool space activity warnings (important if you choose to keep replica reserves low and leverage free pool space). Array diagnostics dumps directly from SANHQ. Programmatic access for scripting. Improvements like these could make a useful product become indispensible in a production environment.

Tag: IOPS

Iometer. As good as you want to make it.

How I use Dell/EqualLogic’s SANHQ in my environment