Performance Testing with Apdex

Often when performance testing in a black box environment, you are left with the onerous responsibility to report against response time performance.

A typical approach by performance testers is to rely on 95th percentiles, which is effectively a Service Level Agreement (SLA) saying that 95 percent of all my samples have a response time below 5 seconds. This is often specified as a Non Functional Requirement (NFR).

But so what? What does this really tell us? More importantly, if the SLA fails, you’re probably left hanging in the wind trying to explain how “bad” it failed to a (now interested) Project Manager. On the flip side, if everything is green, how close were you to failing? Are you an n’th degree away from failing? What about comparing two different application’s response time performance when they have different NFRs to begin with? Getting confused?

Enter the Apdex performance index. Apdex is a numerical measure of user satisfaction that can be built from metrics expressed via more traditional SLAs and/or NFRs. The fundamental objective of Apdex is to simplify the reporting of application response time measurements by making it possible to represent any such measurement using a common metric. This can be reported on by extracting data from Load Runner / JMeter and analysing within Excel as demonstrated below.

Read on to find out more about these scores and how they are calculated.
Read more »

Posted in LoadRunner, Analysis, Tools, JMeter | 1 Comment »

Throughput vs. Latency

When/if I am asked to conduct interviews again for potential performance test analysts, I think I will include this question in my repertoire to suss out those in the know, and those just, um, pretending …

Here is a great analogy of throughput and latency. I don’t pretend to understand queuing theory (just yet), but am working on it

Posted in Tuning | No Comments »

FireWatir 1.2.1 is released

A new version of FireWatir has been released. This includes some fixes related to Firefox 3 but more importantly, it marks the merger of FireWatir and Watir projects, allowing for tighter coupling of methods and functionality between the two, allowing you to write less code =)

To use FireWatir, you will have to install a Firefox plugin that enables use of a JavaScript Shell (jssh). FireWatir works on Windows, Mac and Linux.
Install FireWatir: [sudo] gem install FireWatir
Install Plugin: http://wiki.openqa.org/display/WTR/FireWatir

I will be using FireWatir on my mac for impromptu testing, since more recently I’ve found it beneficial to write automated test cases to provide consistent input when recording load test scenarios in JMeter or LoadRunner. For those interested in even more x-browser support, check out SafariWatir as well: http://safariwatir.rubyforge.org/

Posted in Ruby, watir | No Comments »

A New Project: just_add_watir

In an effort to get more actively involved with the open source community, I’ve recently starting working with another colleague on a new site called justaddwatir.com. This is an exciting collaboration of examples for web application testing in ruby (watir - pronounced “water”).

Watir is a simple open-source library for automating web browsers. It allows you to write tests that are easy to read and easy to maintain. It is optimized for simplicity and flexibility. Watir drives browsers the same way people do. It clicks links, fills in forms, presses buttons. Watir also checks results, such as whether expected text appears on the page.

Its main purpose will be to serve as a working repository of examples that answer questions newcomers have when scripting in watir. Its secondary purpose is to act as an extended regression suite of more applied test cases, which with the blessing of the lead developers of watir (Brett Pettichord) will help test future releases of watir. A working code base of unit tests is being maintained at http://code.google.com/p/justaddwatir/

If you’d like to get involved in the project or you have a blog about watir which you’d like to be aggregated, please send an email to tim.koops at gmail.com

Posted in Ruby, watir | No Comments »

Why do I get an “Error 501/505 - Not implemented or not supported” in JMeter?

This is most likely because the JMeter recording proxy does not support the https protocol. If it’s a screen you can live without, you can just stop/start the proxy recorder around that page (I use the Firefox plugin called FoxyProxy to quickly switch between different proxy configurations).

However, if it is vital to your script (for example, a login page) then you can use an alternative recording tool called Badboy Software. I frequently use this to record HTTPS then export the test plan to JMeter. Alternatively you can manually create the HTTPS request sample yourself.

JMeter can run HTTPS request samplers, it just can’t record them using the native proxy recorder.

Posted in JMeter | No Comments »

How to Generate Random Alphas in LoadRunner

Just a quick post for myself so I don’t forget… I needed to generate a random alpha for use in a LoadRunner web vuser script. The native LoadRunner parameters can do random numerics and date/times but I couldn’t find a way to generate only characters, as one might need to generate a random password for example.

So a quick hack to do it in C is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

random_alpha(char* param_name, int length) {
  char buff[32] = "";
  int r,i;
  char c;
  srand((unsigned int)time(0)); //Seed number for rand()
  for (i = 0; i < length; i++) {
	// A-Z = 65-90 = rand() % 25 + 65
    r = rand() % 25 + 65;
    c = (char)r;
		buff[i] = c;
    printf("%c", c);
  }
  lr_save_string(buff, param_name);
  return 0;
}

I keep that as part of my functions library (referenced in globals.h) and can call on it in a script using the following syntax.
random_alpha(char* param_name, int length)

Example:
random_alpha("advertiser_name_suffix", 4);

Returns:
Notify: Saving Parameter "advertiser_name_suffix = SDNG"

Posted in LoadRunner | No Comments »

JMeter Tips and Tricks - Correlating with Regular Expressions

Another post in this current theme of JMeter tips and tricks… You will hear LoadRunner consultants preaching to the converted about the requirement to correlate data accurately. What they’re referring to, is making sure that dynamic data received by the client from the server (typically in a response body, header or url), which changes from request to request, is accurately correlated. If you don’t, you will probably ‘break’ the application logic with most obvious signs during replay being things like HTTP Server 500 errors.

In practice, correlation requirements relate to things like session ids, viewstate, cookies, date time stamps and other transactional information. So more often than not you will see this data changing following a post back or response from the server. With the introduction of AJAX calls, correlation requirements probably increase (per page) as the number of transactional requests probably increases as well.

In LoadRunner, you typically correlate data manually using the web_reg_save_param function. This in effect is a prepared statement, placed before the request that will return the data in the response you are looking for. For example, your function will specify a left and right boundary of the data that it is looking for before you make the request. Then the request is made and if a match is found, then the data is saved in a parameter which you can later call as a variable in subsequent requests. LoadRunner also has a feature called Correlation Studio which takes affect during recording of your script, using a similar approach (left and right boundaries) and then dynamically swapping corresponding matches with the variabilzed data. All this is quite painful though, and whenever I’m specifying primitive left and right boundaries or using the flaky Correlation Studio (I can explain if you like), I’m left wishing I could just use regular expressions …

The trick to correlating data easily with JMeter is to first know what data it is that needs correlating! (duh…) Experience will make this easier as a lot of common web apps have common dynamic data requirements as already mentioned. Another trick is to always record the script twice (and diff to see differences between unique user sessions), then more often than not record 2+ iterations of an action within a single script (and diff to see differences between user transactions/actions). If you do this before anything else, and diff the results, you will save bucketloads of heartache in getting your script harnesses running, no matter which tool you choose.

But back to JMeter, once you do know what data you need to correlate, simply create a Post Processor -> Regular Expression Extractor to ‘variablize’ that data. Huh? Read on for a more detailed explanation of this.
Read more »

Posted in JMeter | 1 Comment »

JMeter Tips and Tricks - Better (More Pretty) Graphs

This plugin has been around for a while and it’s definitely worth including in your JMeter library if you want to go for a glossy finish! The StatAggVisualizer will transform your graphs
from this to this

If you roll-up your transactions into separate transaction controllers as I demonstrated in a previous post, you will get useful summary views for all your transactions:

Alternatively, drop a stat agg visualizer listener under a single controller to get a graph for a single transaction, useful if clients are only interested in certain transactions.

Posted in JMeter | No Comments »

HttpWatch, a Free Alternative Using Ruby

When performance testing I often need to spy on TCP (http) traffic between the client and server on a frequent basis. This is pretty much a staple of any performance test script development effort.

In the past I’ve relied heavily on trace options within LoadRunner and the like, but often find this a tad cumbersome. LiveHTTPHeaders is a great plugin for Firefox but it only captures header information. WireShark is great if you need to ‘go deep’, but often equates to opening a walnut with a sledgehammer. FireBug and IE Developer Toolbar are great to see what’s happening on a page-by-page basis. TCPdump is great if I’m on a Mac or Linux box, but it can be tricky parsing binary data. In any case most client sites are windows based for me.

So that’s a great list of tools to help you correlate and sniff out dynamic data, but the closest I’ve found for IE to doing what I want, that is, capturing all HTTP data going back and forth with the ability to filter for specific information from a console/command line is HttpWatch. A $295USD price tag / crippled basic (free) edition really forced me down this path…to roll my own HttpWatch using Ruby and the WEBBrick proxy.

Read on for a description of the code.
Read more »

Posted in OSX, Ruby, Tools, Productivity, Windows | 4 Comments »

NumbrCrunchr - Performance Monitoring Using Sparklines

I’ve been working on a light weight performance monitoring solution which can be hosted on a LAMP (or WAMP, or OSX =) installation and uses the Google chart API to present data. Data is populated by custom Ruby scripts that either use ssh or perfmon (typeperf) to collect performance metrics from remote Unix and Windows machines.

The intent is to keep the data collection process detailed enough but simple. Presentation of the collected data then uses the API to present charts in a sparkline format.

Sparkline is a type of information graphics, that presents in a simple and condensed way trends and variation, associated with a measurement such as average temperature or stock market activity. These are often used as elements of a small multiple with several lines used together.

The Sparkline is proposed by Edward Tufte for “small, high resolution graphics embedded in a context of words, numbers, images” Tufte describes sparklines as “data-intense, design-simple, word-sized graphics”. Sparklines are intended to be succinct, memorable, and located where they are discussed.

The finished product looks like this:

Other functionality includes the ability to automatically ‘band’ expected data ranges for each metric collected, using a 30 day (customizable) sample that shows the 20 - 80th percentile. This should offer some insight into trends, and leverage off the sparkline format in being able to quickly assess a large range of metrics or performance data.

There’s also a SiteScope plugin, or in fact an emulator, so that if you’re still using LoadRunner you will be able to dip into the same metrics collected by NumbrCrunchr in a similar fashion to SiteScope with minimal configuration changes.

I’ve decided to go open source with this project in the interests of improving and continually updating its capabilities. If you’d like to contribute to the source please go to the project for more information. If you’d just like to see a quick demo, please go to NumbrCrunchr.com.

Posted in Statistics, Analysis, Tuning, SiteScope | No Comments »

Job Interview Question #11 - Summing Numbers

I’ve recently started following a blog which sets weekly programming challenges. I’m getting sick of Nintendo DS brain training and thought I’d try something more relevant to my line of work. Also, this gives me a good opportunity to learn more about my favoured scripting language at present, which is Ruby. So this week’s challenge states:

Given a list of n integers and another integer called m, determine (true / false) if there exist 2 numbers in that list which sum up to m.
Example: 2,6,4,9,1,12,7 and m=14 -> 2 and 12 sum up to 14, so the answer is true.
Provide the best algorithm in both manners: performance and memory to solve this puzzle. Don’t forget to mention the complexity of your solution!

Read on for my thoughts in solving this challenge.
Read more »

Posted in Ruby, Programming | No Comments »

JMeter Tips and Tricks - Application Simulation Models

I’ve been working with JMeter of late and after speaking with some people at conferences and the like, I thought it would be good to share what I’ve learned along the way in building, structuring and executing JMeter test plans.

One of the first things I like to do when performance testing is establish in reasonable detail what the Application Simulation Model (ASM) will entail. You can create an ASM visually using something like UCML. This is a good starting point for performance testers looking to structure their test planning efforts.

UCML™ is a set of symbols that can be used to create visual system usage models and depict associated parameters. When applied to Performance Testing these symbols can serve to represent the workload distributions, operational profiles, pivot tables, matrixes, and Markov chains that performance testers often employ to determine what activities are to be included in a test and with what frequency they’ll occur.

More often than not though, I create an ASM using a spreadsheet format as I find this easiest to communicate with people from the business teams. If people are interested I will share some of the spreadsheet templates. The following screenshot gives you an idea of what I’m talking about:

The pertinent points for an ASM, is that it should:

e.g. “we have 3 user groups with a total of 150 users, 80% of them are helpdesk support, 10% are team leads and 10% are management”

e.g. “we’re aiming for around 240 trans/hour, of which 50% are searches, 25% are updates and 25% are reports”

Read on to discover how to turn ASMs into working JMeter test plans.
Read more »

Posted in Testing, JMeter | 17 Comments »

Just Another Framework - for developing watir test cases

Recently I attended Railscamp 08 in Sydney Australia for a weekend of code, beer and bzflag. For my ‘project’ I decided on brushing up my my Ruby skills and constructing a simple framework from which I could structure watir test cases.

Specifically I wanted to achieve the following:

To that end I was reasonably successful. The only limitation was that I only had my macbookpro with me and probably would have been shunned even if I attempted to load a windows XP virtual machine at a rails camp full of mac and linux geeks. So I chose FireWatir as my test platform running on my Mac, but am reasonably comfortable you can easily port this framework for use with Watir.

Now on with the framework contents …
Read more »

Posted in Ruby, Automation, watir | No Comments »

Establishing Mysql DB Connectivity from Ruby

If you are a rails fan you will probably recommend using active record instead, but if you just want to hack away at your mysql db from within your Ruby code, you may find this handy…

I’m currently using a wamp box to serve up content for performance metrics. What I needed was a Ruby script which could interrogate data on this installation.

You can download the binary required for this from here. After you install the gem (I do this off-line as my command prompt doesn’t allow me internet access here at work) you may find that you get an error similar to the following:
"NameError: uninitialized constant Mysql"

See what’s happening?
Read more »

Posted in Ruby, Windows | 2 Comments »

CITCON 2008 - Melbourne

I recently attended the 2008 conference for Continuous Integration in Melbourne, and mixed with like minded professionals involved with all aspects of CI and testing in general.

In short this is a great (free) opportunity to attend a conference using an open session format. By that I mean the conference is run/organized like a wiki, where the attendees nominate topics they’d like to discuss or facilitate and then a user vote system organizes and makes it happen.

I avoided the performance testing oriented sessions and went for sessions that spoke about exploratory testing, defect management, benefits of test driven development, skills required for test automation and the last one titled ‘do we still need testers?’ …

They all generated a healthy amount of discussion and it was great to hear other people’s experienced opinions on the subject matter. It was also a great opportunity to network, find out who’s working on similar problems in the local industry and from afar. There was a healthy interest in using JMeter as a performance test tool to support CI, so I’ll be posting some more targeted blogs about this in the near future, as well as some consolidated info on performance metrics collection (without SiteScope) which a few people were also interested in.

Anyway, if you haven’t heard about the conference, keep an eye out for next years version. It’s well worth the zero dollar commitment, free beer and t-shirt. All you have to do is ‘use your feet’.
=)

Posted in General | No Comments »

Diff’ing Made Easier

If diff -u or wdiff isn’t cutting it for your correlating requirements when scripting load harnesses, you might want to try daisydiff as an alternative …

* Works with badly formed HTML that can be found “in the wild”.
* The diffing is more specialized in HTML than XML tree differs. Changing part of a text node will not cause the entire node to be changed.
* In addition to the default visual diff, HTML source can be diffed coherently.
* Provides easy to understand descriptions of the changes.
* Allow easy browsing of the modifications through keyboard shortcuts.

Of late have been working with pesky AJAX controls inside a web-app, and one of my gripes with more traditional tools such as wdiff or diff, is that they don’t tell you *where* in the line the change has occurred. Read on for some hacky ways I’ve implemented daisydiff with LoadRunner and other open source tools in general …
Read more »

Posted in Tools, Productivity, Testing | 1 Comment »

RailsCamp 08

Went to the third RailsCamp held in a scout hall near Gosford NSW last weekend; my brain has just about got back to normal after 3 solid days of drinking and coding =)

RailsCamp is a get together of mostly Rails developers, but also a fairly eclectic mix of Ruby hackers and geeks in general. 3 nights of drinking, camping and coding with NO internet. Possibly the hardest habit to kick is to stop checking for emails every 5 minutes.

For those with harder to kick habits such as Twittering, there was a ‘Twetter’ clone amongst other cool apps such as the ‘Duke’. The Duke was a user vote playlist driver for winamp (I think) and I got to observe just how close the geek community is when it comes to music tastes. Daft Punk was on fairly high rotation. Of mention was the first time we got Rick Rolled, which then led to a flurry of hack attempts at the guy hosting it on his Mac.

Guitar Hero was in force, as was Urban Terror and a touch of BZFlag. Nice.

Accommodation, setup and food was awesome, the organisers really did do a good job in getting this together. Big thanks to those guys! Perhaps the best part was the ability to network with like-minded people and discuss all things related to Ruby or Rails in a collaborative manner. I managed to get a little work done relating to a test framework for use with FireWatir which I’ll post later.

IF you didn’t or couldn’t make it, keep an eye out for the next one to occur. Definitely worth the $100 to nerd on for a full weekend …

Posted in General, Ruby, Programming | No Comments »

Installing Firewatir on Mac OSX

By now you’ve probably heard of watir or Web Application Testing In Ruby, a great automation framework which supports Microsoft Internet Explorer, and recently came in at number 3 as a popular oss tool on this blog.

I’ve been using this tool mostly from a windows box, and knew of the firefox equivalent, aptly named ‘Firewatir’ which is currently a Google code project. The beauty of this version, is that it allows you as a tester to explore functionality on your favourite platform for development, as I like to leave windows at work, and take OSX with me on the train.

Read on for some simple instructions in getting this running on your Macbook Pro …
Read more »

Posted in Ruby, Automation, watir | 2 Comments »

Using SFTP with LoadRunner Java Vusers

Recently had a requirement to construct a LoadRunner harness that could sftp files (over ssh) to and from remote servers. As some of the harnesses were already written in Java (for loading of JMS queues) it made sense to use a Java Vuser to achieve the result required.

A work colleague found a knowledge base reference that recommended using an RTE Vuser for secure FTP recording and replay. Document ID 41820 refers. I don’t know, but that seemed like a kludge to me, and I wasn’t keen on a multi-protocol harness, so we went searching on Google for a Java alternative.
Read more »

Posted in LoadRunner, Java, Security | No Comments »

Capturing Screenshots in Watir

Recently I noticed some discussion in the watir user group about trying to capture screenshots when running automated tests.

For such a simple requirement, it’s frustrating that solutions available either cost money (such as the purchase of SnagIt) or are a tad complicated in implementation requiring the installation of some out of date gems and dependencies. I’ve also come across this requirement in other automation tools such as QTP, so I thought it’s about time I rolled up my sleeves and hack together a simple solution with C#.NET 2.0.
Read more »

Posted in Ruby, Tools, Programming, watir | 3 Comments »