Promiscuity and the tester

Last week I had the great fortune of attending the London Tester Gathering, where first time speaker Mark Winteringham was leading the discussion with a great talk about mental models around testing tools. A write up and copy of his slides are here: What’s So Great about Webdriver

He was talking about how using Webdriver to automate (testing/checking…I’m not getting into that argument now) has helped him in his work as a test consultant.

However he also referred to the fact that when we use tools, whatever they are, they shape the way we think, how we behave and interact with software, solve problems and communicate. Essentially, through the use of the tool, they end up defining how we test…if we let them.

Referring to one of the highlights of Test Bash 3 – Iain McCowatt’s talk Automation: Time to change our models; (watch the video here) Mark raises the issue that we should be wary of tools, how we select them, use them to solve problems and achieve our end goals.

During the talk, Mark dares us to “Be Promiscuous” with our use of tools, to shop around, not to limit the way we work through limited use of tools, because ultimately that then leads to limited thinking. Whilst the analogy he used sparked a few laughs, he’s dead right.

Mark examined data on the number of conference talks about automation, and the number of automated testing jobs out there, and a large proportion of those are linked to specific tools – mostly Webdriver.

In itself this isn’t too worrying. Webdriver is clearly a popular tool, works well for most people and organisations that use it, and solves a lot of their testing problems. I asked Mark whether the ubiquity of Webdriver presented any dangers to us as testers, and his response was (If you’ll pardon me paraphrasing, it was a noisy room and I had drunk a couple of beers by then) if you define a tool, and eventually it will define you through its use. Essentially, that we should not let the industry or the ubiquity of a specific set of tools define us as testers, nor define our testing.

Whilst I am not a webdriver user, I do use other tools to solve some of the security testing problems that I encounter. In recent months, whilst I have been using tools such as Zed Attack Proxy and BurpSuite, I have found that my approach to testing has been limited by my ability to use these tools, rather than looking around the tool, or using them in different ways to solve problems.

Essentially, the tool was beginning to define how I tested…something to be avoided I feel. The tools that I have mentioned are great, and have a lot of useful features. However through using them, I focussed on what feedback they were providing me, rather than focussing on what was important – what I needed to do to identify vulnerabilities, understanding the underlying functionality and security of the application under test, and by trying to think like someone who wanted to undermine that application in order to protect it.

If that’s not clear, then perhaps this analogy would help – when we learn to drive for example, we are with an instructor, or a parent, in a car with a steering wheel, handbrake, accelerator, brake pedal, clutch pedal etc. It’s usually a small car, on urban roads with a bit of traffic. We build up a mental model of how to drive in our mind based on the tool we are using; the car, and the environment we are in, our local area.

Lets say then, you switch from one car to another – this one might have an automatic gear box and no clutch pedal. This then removes the need for the driver to make their own judgements about when to change gear, as the car will do it for you.

Lets take this a step further…the car has parking assistance technology, or anti collision or adaptive cruise control. These driver aids might then reduce our need to focus on the important skills of parking, or safe driving distances, maintaining a decent speed etc. We become reliant on the tool to do the job for us, rather than using our own mental models for a task to do it. Is these breeding better drivers? I’m not sure that it is.

At some point I will try to take this discussion a bit further and apply some of this learning to the security testing I have been doing. Last year, when I ran the talk ‘New Adventures in Security Testing’ I came up with this mnemonic: Cartoon Tester: EXTERMINATE (Thanks Andy for the great cartoon)

This is my personal developing mental model for security testing, and in the very near future I will work on challenging and modifying this model. Where appropriate I’ll work on sharing and discussing it.

Mark’s talk (and Iain’s) has really inspired me to think again about how I use tools effectively to solve testing problems, but also to remember that a tool, like any device, is only as good as the person controlling it.