I’ve been thinking about some promotional material for the company I work for. What inspired me was seeing a vi coffee mug… Unfortunately only available in the US. I’ve been endeavouring to wean myself off textmate and the like so thought this was a brilliant idea. But with limited space, what essential commands should go on a vim cheat sheet? What other cheat sheets would you like to see in this format?
“Sparklines: Intense, Simple, Word-Sized Graphics”
If you’ve been reading this blog for sometime, you will know I’ve got a certain penchant for sparklines, a term coined by Edward Tufte and referred to in his book titled “Beautiful Evidence”
True sparklines are meant to be word-like graphics, such that they fit inline with the body of the text, however I’ve had difficulty implementing this with tools such as Excel or custom chart libraries like JFreeChart. The Google Chart API makes this relatively simple.
To that end, my choice of implementation is to present inline with a paragraph. Say for example I wanted to display the amount of jobs advertised over 4 months, then my version of a sparkline would look like this:
Sure, it’s pixelated, the colours are a little hard to differentiate, and the vertical axis is lacking a scale or unit of measurement. However this simple column chart does convey quite succinctly, when jobs are advertised, whether it is the start, middle of end of a month, and how many jobs are available per month.
So what makes a perfect sparkline? To be honest, I don’t ever think sparklines should be used as a supplement for detail. I believe sparklines are best used in situations where you want to convey a lot of data in a small space. They really should be thought of as triggers, not from which you conduct a detailed analysis, but something which triggers the thought to pursue more information in the first place.
Here’s another example recently used to show size of allocation failures on an IBM JVM heap.
Cool? Go get em: http://en.wikipedia.org/wiki/Sparkline
Well, as a performance test lead I’ve been getting more involved in the hiring process.
To cut through the chaff, these are some of the questions I like to ask potential candidates. What’s your favourite questions? How might you answer these?
I also like to follow up these questions with a half hour practical scripting in VuGen (or your tool of choice). You can learn alot about someone by the way they write their scripts.
The point? Determine if the person has recently used the tool (you’d be surprised) and suss out general knowledge of performance testing…
If you’re testing HTTPS with JMeter 2.3.2 and a current version of Java greater than 1.5 e.g.
java -version
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)
Then you are likely to encounter this error when using the standard HTTP Request sampler:
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Unconnected sockets not implemented
If you have the opportunity, use the HTTP Request HTTPClient sampler instead and this problem should be resolved. Otherwise use Java 1.5 instead. If you’ve already recorded your test plan in JMeter or other tools (like BadBoy) then some simple regular expressions on the jmx file will fix the problem manually.
Windows:
ruby -pi.bak -e "gsub(/<HTTPSampler guiclass="\""HttpTestSampleGui"\"" testclass="\""HTTPSampler"\""/, '<HTTPSampler2 guiclass="\""HttpTestSampleGui2"\"" testclass="\""HTTPSampler2"\""')" *.jmx
ruby -pi.bak -e "gsub(/HTTPSampler>/, 'HTTPSampler2>')" *.jmx
Mac:
ruby -pi.bak -e "gsub(/<HTTPSampler guiclass=\"HttpTestSampleGui\" testclass=\"HTTPSampler\"/, '<HTTPSampler2 guiclass=\"HttpTestSampleGui2\" testclass=\"HTTPSampler2\"')" *.jmx
ruby -pi.bak -e "gsub(/HTTPSampler>/, 'HTTPSampler2>')" *.jmx
Previously I identified how to setup pattern matching with Regex in LoadRunner.
Here’s how you implement search and replace functionality with the same POSIX libraries used in the previous example.
Create a replace function in your favourite headers file or wherever you keep your framework’y stuff… You also need the pcre3.dll and pcreposix.h header added as files to your script.
replace(const char *string, char *pattern, char *replace, char *match) { int length; int status; int eflag; char buf[1024] = ""; char out[1024] = ""; regex_t re; regmatch_t pmatch[128]; lr_load_dll("pcre3.dll"); if((status = regcomp(&re, pattern, REG_EXTENDED)) != 0){ regerror(status, &re, buf, 120); lr_output_message("Match PCRE Exit 2"); return 2; } while(status = regexec( &re, string, 1, pmatch, eflag)== 0){ strncat(out, string, pmatch[0].rm_so); strcat(out, replace); string += pmatch[0].rm_eo; eflag = REG_NOTBOL; } strcat(out, string); lr_save_string(out, match); }
Then anywhere in your actions you can call on this function, passing it the string buffer you wish to operate on, a search value, a replace value and the name of the LoadRunner parameter you want the result saved into.
lr_save_string("FOO%20BAR%20DING", "buffer"); replace(lr_eval_string("{buffer}"),"%20"," ", "custom");
What you’ll end up with is something like this:
Home.c(6): Notify: Saving Parameter "buffer = FOO%20BAR%20DING"
Home.c(7): Notify: Parameter Substitution: parameter "buffer" = "FOO%20BAR%20DING"
framework.h(135): Notify: Saving Parameter "custom = FOO BAR DING"
Enjoy.
Previously I identified a way in which to test SPNEGO or Kerberos authentication with LoadRunner. However this implementation was buggy in the sense that if you ran your load tests under reasonable load with the WinInet replay engine (instead of sockets) you were likely to encounter the following error:
Error -27492: "HttpSendRequest" failed, Windows error code=12057 (certificate revoked) and retry limit (0) exceeded for URL="
https://someplacesecure.com.au/secure.html", Snapshot Info [MSH 1 2]
This error occurs when using WinInet replay instead of sockets with Integrated Authentication enabled in run-time settings. The purpose of this was to allow vusers to use SSO with SPNEGO authentication in an IBM WebSEAL environment.
After spending some time with the mystical HP level 3 support, they identified an undocumented flag which helps out significantly in this. So, instead of using the WinInet replay engine (which is not encouraged by HP) you should do something similar to the following.
vuser_init() { // Preferred run-time settings // Browser -> Browser Emulation // [ ] Simulate a new user on each iteration // Preferences -> Options // Enable Integration Authentication [Yes] web_set_sockets_option("INITIAL_BASIC_AUTH","1"); web_set_user("DOMAIN.LOCAL\\username", "password", "someplacesecure.com.au:443"); web_url("myportal", "URL=https://someplacesecure.com.au/wps", "Resource=0", "Referer=", "Mode=HTML", LAST); return 0; }
The magic is in the web_set_sockets_option("INITIAL_BASIC_AUTH","1") flag. Set that and you can then use LoadRunner in Sockets mode which as it turns out, is much more stable.
Enjoy.
By this I mean I wanted to determine how much network and server time was spent processing a request initiated by a Watir script. Server time as defined by time to first buffer and network time as time to last buffer (minus the first).
I couldn’t really do this from Ruby itself, so I installed Fiddler. Fiddler lets you capture traffic in much the same way as ethereal / wireshark but with a nice GUI. There are some Fiddler customization scripts which let you display this info in the Custom column i.e.
oSession["ui-customcolumn"] = "FB: " + oSession["X-TTFB"] + "; LB: " + oSession["X-TTLB"];
But if you’re a neat freak like me and want them in separate columns then do this:
public static BindUIColumn("Time")
function CalcMethodCol1(oS: Session){
if (null != oS.oRequest) return DateTime.Now.ToString("dd/MM/yy HH:mm:ss.ffff");
else return String.Empty;
}
public static BindUIColumn("AbbreviatedUrl")
function CalcMethodCol2(oS: Session){
if (null != oS.oRequest) return oS.url.substring(oS.url.lastIndexOf("/"));
else return String.Empty;
}
public static BindUIColumn("TTFBuffer")
function CalcMethodCol3(oS: Session){
if (null != oS.oResponse) return oS["X-TTFB"];
else return String.Empty;
}
public static BindUIColumn("TTLBuffer")
function CalcMethodCol4(oS: Session){
if (null != oS.oResponse) return oS["X-TTLB"];
else return String.Empty;
}
You place them before you declare the static function OnBeforeRequest. The time stamp used above ends up being the ‘end’ timestamp, not the ’start’ so you just need to subtract time to last buffer to determine that. You end up with this in your summary view:
Once this is done you can pretty up the results by copying the ‘Full Summary’ for your selected sessions in Fiddler and pasting them into Excel. This was the main reason for doing this. If you’re happy with the Fiddler ‘timeline’ view then you don’t need to bother. I just wanted the data so I could merge/correlate with other data sources.
Would be interested to know if anyone has achieved this in ruby itself. I was thinking of using the webrick/httpproxy but would have come short if testing a HTTPS web app.
This might help you out when analyzing time series data.
In this particular situation I’m looking at a bunch of SOA web services that get called synchronously in the backend infrastructure. The logs have entries with a unix timestamp, service description and duration. What I want to do is view the data in a Gantt Chart format. This link from Microsoft put me on the right track.
Here’s how to do it in Open Office.
Read more »
I hate to say it… but Microsoft really have produced an excellent tool for sysadmins and performance testers alike. That tool is called LogParser. I’ve mentioned it before but that was more IIS specific. LogParser can be pointed at just about any type of log file. So with a bit of Perl (or equivalent) to cleanup, and LogParser I was able to analyze Gigabytes of IBM WebSEAL access logs on a production system.
When running multiple vuser groups with the same script i.e.
If that script references a data file (for parameters) then that data file is shared by *all* vuser groups.
e.g. my parameter called {values} has
When running, vuser group_b will error with these settings e.g.
i.e.
So 2 separate vuser groups still equals one shared parameter file (and settings).
Watch out for this!
NOTE: This doesn’t generate a timestamp accurate to milliseconds. It just stores it in two fields. AFAIK there is no way to generate such a thing. It is accurately described here. Sorry for the confusion =)
Bug #8523 details MySQL’s deficiency in its ability to store and retrieve datetime values with millisecond or microsecond precision. When performance testing and perhaps storing trace logs or audit information in a MySQL table for later analysis, this shortfall is quite frustrating.
My hacky way to deal with this is to store the millisecond component as its own integer. Say for example you are interested in the difference between a Put datetime and Get datetime for MQ, your table might look like this:
CREATE TABLE `msecs` ( `put_datetime` datetime DEFAULT NULL, `put_msecs` int(11) DEFAULT NULL, `get_datetime` datetime DEFAULT NULL, `get_msecs` int(11) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
So in this table I am storing the datetime component in its native MySQL format and the millisecond component as an integer. Here are some examples:
Now to retrieve and compare the values, I get the difference using TIME_TO_SEC for the datetime component, multiply it by 1000 to convert to milliseconds then add to the difference of the milliseconds component using basic maths:
SELECT ((TIME_TO_SEC(get_datetime) - TIME_TO_SEC(put_datetime))*1000) + (get_msecs-put_msecs) AS diff_msecs FROM msecs
That gives you difference in milliseconds for the demo table along these lines:
2300
1700
300
600700
700
Hopefully that makes sense. Pipe up if you have an easier solution!
I’ve recently been playing around with AutoIt in an effort to steer clear of QTP and the like. AutoIt is a simple but effective way in which to automate pretty much any Windows GUI application. To that end I needed a simple data driven framework to include in the script. AutoIt comes with some handy Excel user defined functions which allow you to do the same.
For example, the following spreadsheet:
Can be read into an array and looped through as follows:
#include <Excel.au3> #include <Array.au3> $file = @ScriptDir & "\excel.xls" $oExcel = _ExcelBookOpen($file, 0) ; open excel in the background $aArray = _ExcelReadSheetToArray($oExcel, 2) ; ignoring header row _ArrayDisplay($aArray, "Array using Default Parameters") For $i = 1 to UBound($aArray)-1 $name = $aArray[$i][1] $surname = $aArray[$i][2] ; do application stuff MsgBox(0,'debug', $name &' '& $surname ) ; write result to the book _ExcelWriteCell($oExcel, "PASS", $i+1, 5) Next _ExcelBookClose($oExcel) ; close excel
Pretty simple stuff!
I recently had a conversation with a colleague about how one might determine the concurrency for a given page or transaction at any point in time. The system under test was a typical web server and we had access to the web logs for analysis.
That is, we wanted to know how many virtual users were hitting a web page at the same time without using a sledgehammer to open the walnut so to speak… To that end we fell back to simple unix commands like awk, sort and uniq.
The typical web log entry we were looking for looked like this:
10.1.20.10 - Unauth [16/Dec/2008:11:59:16 +1100] "GET HTTP://system.under.test/secure.html HTTP/1.1" 200 692
On windows we would use a command similar to this to generate our report:
gawk "{print $4,$7}" web01-request.log | grep "\/secure" | uniq -c | sort -T C:\temp | gawk "{print $1}" | uniq -c
The first awk prints the date time column and url column which we then grep for any entries containing our target word ’secure’ (we could have done this in the awk statement too).
Then we use uniq -c to generate a unique count of rows that have the same date time, sort that (using a temp file), awk again on the first column and uniq -c again. The output looks like this:
2218 1
1483 2
263 3
258 4
38 5
36 6
2 7
5 8
1 12
So we can then determine for example that there were 1,483 occurrences where 2 virtual users hit the target at the same time. There were 258 occurrences where 4 virtual users hit the target at the same time and so on.
You can then graph this using the google chart API. No excel required =)
No this is not a post on how to get free licenses…
There should be no doubt that purchasing LoadRunner licenses is expensive. How Mercury/HP has achieved this stranglehold on the market is strange to behold but it is the way it is.
Needless to say, in another purchasing license cycle I jokingly said to a colleague that it would probably be cheaper to fly to India, hire some office space and PCs, give some training to 100 production workers and pay them 3 months wage to facilitate your whole test effort using real people and PCs. You’d still probably come out ahead of a typical yearly budget for HP products… Or would you?
Read more »
Alas, it can’t be done in JMeter. So this is how it works in LoadRunner …
This challenge came up recently and we were able to figure out how to test SPNEGO or Kerberos using Integrated Windows Authentication with LoadRunner.
Read more »
The wheel has turned full circle and I’m back in MQ land… So here is a a simple Java vuser harness which you can use in LoadRunner to put or get messages from Websphere MQ. This version doesn’t require any JNDI bindings (or access to JMS). Instead it just uses a standard server connection channel over TCP to get and put messages.
You will need to get the following jars from your Websphere MQ installation:
1. com.ibm.mq.jar
2. connector.jar
3. jta.jar
Make sure you include them in your run-time settings (F4) for your script:
Read more »
This is a relatively new example of how to use JMeter with WebSphere MQ. This approach is a clever way of inserting messages into MQ using a JMS sampler (point to point) with a JNDI binding within MQ.
I had not heard of this approach before and am sure it will be useful in future, provided you have sufficient access to create the bindings in the first place.
Often when performance testing in a black box environment, you are left with the onerous responsibility to report against response time performance.
A typical approach by performance testers is to rely on 95th percentiles, which is effectively a Service Level Agreement (SLA) saying that 95 percent of all my samples have a response time below 5 seconds. This is often specified as a Non Functional Requirement (NFR).
But so what? What does this really tell us? More importantly, if the SLA fails, you’re probably left hanging in the wind trying to explain how “bad” it failed to a (now interested) Project Manager. On the flip side, if everything is green, how close were you to failing? Are you an n’th degree away from failing? What about comparing two different application’s response time performance when they have different NFRs to begin with? Getting confused?
Enter the Apdex performance index. Apdex is a numerical measure of user satisfaction that can be built from metrics expressed via more traditional SLAs and/or NFRs. The fundamental objective of Apdex is to simplify the reporting of application response time measurements by making it possible to represent any such measurement using a common metric. This can be reported on by extracting data from Load Runner / JMeter and analysing within Excel as demonstrated below.
Read on to find out more about these scores and how they are calculated.
Read more »
When/if I am asked to conduct interviews again for potential performance test analysts, I think I will include this question in my repertoire to suss out those in the know, and those just, um, pretending …
Here is a great analogy of throughput and latency. I don’t pretend to understand queuing theory (just yet), but am working on it 
A new version of FireWatir has been released. This includes some fixes related to Firefox 3 but more importantly, it marks the merger of FireWatir and Watir projects, allowing for tighter coupling of methods and functionality between the two, allowing you to write less code =)
To use FireWatir, you will have to install a Firefox plugin that enables use of a JavaScript Shell (jssh). FireWatir works on Windows, Mac and Linux.
Install FireWatir: [sudo] gem install FireWatir
Install Plugin: http://wiki.openqa.org/display/WTR/FireWatir
I will be using FireWatir on my mac for impromptu testing, since more recently I’ve found it beneficial to write automated test cases to provide consistent input when recording load test scenarios in JMeter or LoadRunner. For those interested in even more x-browser support, check out SafariWatir as well: http://safariwatir.rubyforge.org/
