Thursday, November 06, 2014 22:01 PM
At Oredev 2014 I presented "Confessions of an Accidental Security Tester".
The slides are on slideshare, the video below and on vimeo:
"Alan Richardson does not describe himself as a security tester. He's read the books so he knows enough to know he doesn't know or do that stuff properly. But he has found security issues, on projects, and on live sites that he depends on for his business.
You want to know user details? Yup, found those. You want to download the paid for assets from the site without paying for them? Yup, can do. You want to see the payment details for other people? OK, here they are. All of this, and more, as Alan stumbled, shocked, from one security issue to the next,
In this session Alan describes examples of security issues, and how he found them: the tools he used, why he used them, what he observed and what that triggered in his thought processes.
Perhaps most shocking, is not that the issues were live, and relatively easy to find and exploit. But that the companies were so uninterested in them. So this talk also covers how to 'advocate' for these issues. It also warns you not to expect rewards and gratitude. Companies with these type of issues typically do not have bug bounty schemes.
Nowadays, many of the tools you need to find and exploit these issues are built in to the browser. Anyone could find them. But testers have a head start. So in this session Alan shows how you can build on the knowledge and thought processes you already have, to find these types of issues.
This is a talk about pushing your functional testing further, deeper, and with more technical observation, so you too can 'accidentally' discover security issues."
CONFESSIONS OF AN ACCIDENTAL SECURITY TESTER - "I DIDN'T BREAK IN, YOU LEFT THE DOOR OPEN" from Øredev Conference on Vimeo.
Friday, September 26, 2014 14:05 PM
or "Why I explored Taskwarrior the way I did".
In a previous post I discussed the tooling environment that I wanted to support my testing of Taskwarrior for the Black Ops Testing webinar of 22nd September 2014.
In this post, I'll discuss the 'actual testing' that I performed, and why I think I performed it the way I did. At the end of the post I have summarised some 'principles' that I have drawn from my notes.
I didn't intend to write such a long post, but I've added sections to break it up. Best of luck if you make it all the way through.
First Some ContextFirst, some context:
- 4 of us, are involved in the testing for the webinar
- I want the webinars to have entertainment and educational value
- I work fast to try and find 'talking points' to support the webinar
- I test to 'find bugs fast' because they are 'talking points' in the webinar
- I explore different ways of testing because we learn from that and can hopefully talk about things that are 'new' for us, and possibly also new for some people on the webinar
And then I started Learning
I spent a quick 20 minutes learning the basics of the application using the 30 second guide. At this point I knew 'all you need to know' to use the application and create tasks, delete tasks, complete tasks and view upcoming tasks.
Because of the tooling I set up, I can now see that there are 4 files used in the application to store data (in version 2.2)
- the format of the data
- items move from one file to another
- undo.data has a 'before' and 'after' state
- history would normally look 'empty' when viewing the file, but because I'm tailing it, I can see data added and then removed, so it acts as a temporary buffer for data
- there might be risks moving data from one file to another if a file was locked etc.
- text files can be amended,
- does the app handle malformed files? truncated files? erroneous input in files?
- state is represented in the files, so if the app crashed midway then the files might be out of sync
- undo would act on a task action by task action
- as a user, the text files make it simpler for me to recover from any application errors and increase the flexibility open to me as a technical user
- as a tester, I could create test data pretty easily by creating files from scratch or amending the files
- as a tester, I can put the application into the state I want by amending the files and bypassing the GUI
- I have a basic understanding of the application,
- I've read minimal documentation,
- I've had hands on experience with the application,
- I have a working model of the application storage mechanism
- I have a model of some risks to pursue
I take my initial learning and build a 'plan'
- What if I do the minimal commands incorrectly?
- if I give a wrong command?
- if I miss out an id?
- if I repeat a command?
- if a task id does not exist?
- if I use a priority that does not exist?
- if I use an attribute that does not exist?
I start to follow the plan
- the message by the application suggests that 'app' == 'append'
- I didn't know there was an append command, I didn't spot that. A quick 'task help | grep append' tells me that there is an append command. I try to grep for 'app' but I don't see that 'app' is a synonym for append. Perhaps commands can be truncated? (A question about truncation to investigate for later)
- I also didn't expect to see 3 tasks listed as being appended to, since I only have one pending task, one completed task, and one deleted task. (Is that normal? A question to investigate in the future)
I break and reflect
- how can I have the system 'show' me the data? i.e. deleted data
- how can I add a task with a 'due' date to explore more functionality?
I start testing modify
- reporting - actually does a data clean up, prior to the report and moves deleted tasks and hanging actions so that the pending file is 'clean'
- 'modify' bypasses some of the top level application controls so might be a way of stressing the functionality a little
I start modifying calculated fields
I start exploring the 'entry' value
- Should I be able to amend it?
- What happens if I can?
- if the task doesn't exist yet, should priority matter?
- Can I have a due date before the entry exists?
- If only there were a better way of tracking the testing.
- Questions can drive testing...
- You don't need the answers, you just need to know how to convert the question into a form the application can understand. The application knows all the answers, and the answers it gives you are always correct. You have to decide if those 'correct' answers were the ones you, or the spec, or the user, or the data, or etc. etc., actually wanted.
- The answers you receive from your questions will drive your testing
- Answers lead to more questions. More questions drive more testing.
- The observations you make as you ask questions and review your answers, will drive your testing.
- The level to which you can observe, will determine how far you can take this. If you only observe at the GUI layer then you are limited to a surface examination. I was able to observe at the storage layer, so I was able to pursue different approaches than someone working at the GUI.
- Experiment with internal consistency
- Entity level with consistency between attributes
- Referential consistency between entities
- Consistency between internal storage and persistent storage
- Debriefs are important to create new plans of attack
- personal debriefs allow you to learn from yourself, identify gaps and new approaches
- your notes have to support your debriefs otherwise you will work from memory and don't give yourself all the tools you need to do a proper debrief.
- Switch between 'serious' testing and 'playful' testing
- It gives your brain a rest
- It keeps you energised
- You'll find different things.
Friday, September 26, 2014 14:26 PM
For the Black Ops Testing Webinar on 22nd September 2014 we were testing Taskwarrior, a CLI application.
I test a lot of Web software, hence the "Technical Web Testing 101" course, but this was a CLI application so I needed to get my Unix skills up to date again, and figure out what supporting infrastructure I needed.
By the way, you can see the full list of notes I made at github. In this post I'm going to explain the thought process a little more.
Before I started testing I wanted to make sure I could meet my 'technical testing' needs:
- Observe the System
- Control the Environment
- Restore the App to known states
- Take time stamp logs of my testing
- Backup my data and logs from the VM to my main machine
You will also find here a real life exploratory testing log that I wrote as I tested.
Observe the SystemPart of my Technical Testing approach requires me to have the ability to Observe the system that I test.
With the web this is easy, I use proxies and developer tools.
How to do this with a CLI app? Well in this case Taskwarrior is file based. So I hunted around for some file monitoring tools.
I already knew about Multitail, so that was my default. 'Tail' allows you to monitor the changes to the end of a file. Multitail allows me to 'tail' multiple files in the same window.
I looked around for other file monitoring tools, but couldn't really find any.
With Multitail I was able to start it with a single command that was 'monitor all files in this directory'
- multitail -Q 1 ~/.task/*
I knew that the files would grow larger than tail could display, so I really wanted a way to view the files.
James Lyndsay used Textmate to see the files changing. I didn't have time to look around for an editor that would do that (and I didn't know James was using Textmate because we test in isolation and debrief later so we can all learn from each other and ask questions).
So I used gedit. The out of the box editing tool on Linux Ubuntu. This will reload the file if it has changed on the disk when I change tabs. And since I was monitoring the files using Multitail, I knew when to change tabs.
OK, so I'm 'good' on the monitoring front.
Control the EnvironmentNext thing I want the ability to do? Reset my environment to a clean state.
With Taskwarrior that simply involves deleting the data files:
- rm ~/.task/*
Restore the App to known statesOK. So now I want the ability to backup my data, and restore it.
I found a backup link on the Taskwarrior site. But I wanted to zip up the files rather than tar them, simply because it made it easier to work cross platform.
- zip -r ~/Dropbox/shared/taskData.zip ~/.task
Same as the Multitail command, which will automatically add any new files it finds in the directory, so it added a history file after I started monitoring.
Backup my data and logs from the VM to my main machineI would do most of my testing in a VM, and I really wanted the files to exist on my main machine because I have all the tools I need to edit and process them there. And rather than messing about with a version control system or shared folders between VM etc.
I decided the fastest way of connecting my machines was by using Dropbox. So I backup the data and my logs to Dropbox on the VM and it automatically syncs to all my other machines, including my desktop.
Take time stamp logs of my testingI had in the back of my mind that I could probably use the 'history' function in Bash to help me track my testing.
Every command you type into the Bash shell is recorded in the history log. And you can see the commands you type if you use the 'history' command. You can even repeat them if you use '!' i.e. '!12' repeats the 12th command in the history.
I wanted to use the log as my 'these are the actual commands I typed in over this period of time.
So I had to figure out how to add time stamps to the history log
- export HISTTIMEFORMAT='%F %T '
I looked for simple way to extract items from the history log to a text file, but couldn't find anything in time so I eventually settled on a simple redirect e.g. redirect the output from the history command to a text file (in dropbox of course to automatically sync it)
- history > ~/Dropbox/shared/task20140922_1125_installAndStartMultiTail.txt
And since I did not add the HISTTIMEFORMAT to anything global I tagged it on the front of my history command
- export HISTTIMEFORMAT='%F %T '; history > ~/Dropbox/shared/task20140922_1125_installAndStartMultiTail.txt
I added 'comments' to the history using the 'echo' command, which displays the text on screen and appears in the history as an obviously non-testing command.
I also found with history:
- add a ' ' space before the command and it doesn't appear in the history - great for 'non core' actions
- you can delete 'oops' actions from the history with 'history -d #number'
And that was my basic test environment setup. If I tracked this as session based testing then this would have been a test environment setup session.
All my notes are in the full testing notes I made.
I could have gone further:
- ideally I wanted to automatically log the output of commands as well as the command input
- I could have figured out how to make the HISTTIMEFORMAT stick
- I could have figured out how to have the dropbox daemon run automatically rather than running in a command window
- I could have found a tool to automatically refresh a view of the full file (i.e. as James did with textmate)
But I had done enough to meet my basic needs for Observation, Manipulation and Reporting.
Oh, and I decided on a Google doc for my test notes very early on, because, again, that would be real time accessible from all the machines I was working from. I exported a pdf of the docs to the gihub repo.
And what about the testing?Yeah, I did some testing as well. You can see the notes I made for the testing in the github repo.
So how did the tools help?
As I was testing, I was able to observe the system as I entered commands.
So I 'task add' a new task and I could see an entry added to the pending.data, and the history.data, and the undo.data.
As I explored, I was able to see 'strange' data in the undo.data file that I would have missed had I not had visibility into the system.
You are able to view the data if you want, because I backed up up as I went along, and it automatically saved to dropbox, which I later committed to github.
What else?At the end of the day, I spent a little time on the additional tooling tasks I had noted earlier.
I looked at the 'script' command which allows you to record your input and output to a file, but since it had all the display esc characters in it, it wasn't particularly useful for reviewing.
Then I thought I'd see what alternative terminal tools might be useful.
I initially thought of Terminator, since I've used it before. And when I tried it, I noticed the logger plug-in, which would have been perfect, had I had that at the start of my testing since the output is human readable and captures the command input and the system output as you test. You can see an example of the output in the github repo.
I would want to change the bash prompt so that it had the date time in it though so the log was more self documenting.
You can see the change in my environment in the two screenshots I captured.
- The initial setup was just a bunch of terminals
- The final setup has Terminator for the execution, with gedit for viewing the changed files, and multitail on the right showing me the changes in the system after each command
- Good enough tooling
- Ideally I'd love a fully featured environment and great tools, but so long as my basic needs are met, I can spend time on testing. There is no point building a great environment and not testing. But similarly, if I don' t have the tools, I can't see and do certain things that I need in my testing.
- Learn to use the default built in tools
- Gedit, Bash, History - none of that was new. By using the defaults I have knowledge I can transfer between environments. And can try and use the tools in more creative ways e.g. 'echo' statements in the history
- Automated Logging doesn't trump note-taking
- Even if I had Terminator at the start of my testing, and a full log. I'd still take notes in my exploratory log. I need to write down my questions, conclusions, thoughts, research links, etc. etc.
- If I only had one, I'd take my exploratory log above any automated log taking tool.
- Write the log to be read
- When you read the exploratory log, I think it is fairly readable (you might disagree). But I did not go back and edit the content. That is all first draft writing.
- What do I add in the edit? I answer any questions that I spot that I asked and answered. I change TODOs that I did. etc. I try not to change the substance of the text, even if my 'conclusions' as to how something are working are wrong at the start, but change at the end, I leave it as is, so I have a log of my changing model.
Why didn't you use tool X?If you have particular tools that you prefer to use when testing CLI applications then please leave a comment so I can learn from your experience.
Thursday, September 11, 2014 11:58 AM
You can find wireshark on line - it is a free tool.
Note that you may not be able to capture the mobile traffic on Windows because of WinPCap limitations. You may need to buy an additional adapter to do this. I'm using Mac to show you this functionality.
What is it?Wireshark is a tool for monitoring network traffic. Unlike an HTTP proxy server where you have to configure your machine to point to the HTTP proxy server in order to monitor the traffic. With Wireshark, you tell it to capture traffic from your network card, and it can then capture any traffic going through that network.
So if your mobile device is on the same wifi network as your Wireshark machine's wifi card. Then you can capture the wifi traffic, filter it, and then monitor the HTTP traffic from your mobile device.
Why would I want to do that?Because sometimes the mobile app you are testing does not honour the proxy settings of the device and goes direct, so you don't see the traffic.
And because you can start learning more about the network traffic layers being used by your application and your device in general.
(It's also fun to hook into hotel wifi and airport lounge wifi - but don't tell anyone.)
But the serious point, is that we know we want to observe the http traffic. If we have the issue that we can't because we can't configure the app to point to the proxy then we need other options. We need to increase our flexibility to approaching the observation. So we have a new option - work at the network traffic level, rather than proxy.
Our aim is to keep looking for new ways of achieving our outcomes. Not finding tools, for the sake of tools. But finding new approaches.
On WindowsThe Windows install is simple. Just download and run.
On MacMac install was a little harder for me and it didn't work out the box so I had to do the extra steps to add the application to XQuartz
- Install X-Windows - X11 XQuartz - http://xquartz.macosforge.org/landing/
- Download the img for wireshark
- install Wireshark
- Start wireshark - it might take a while, but should work
- "open /Applications/Wireshark.app/Contents/MacOs/Wireshark"
- or "open wireshark"
- or "wireshark"
You might find these links helpful if you are on a mac:
On LinuxI haven't tried the install on linux - I imagine the instructions on the Wireshark website work fine.
First UsageWireshark, can seem intimidating initially to work with.
It is a complicated tool and there is a lot to learn about it.
Start a CaptureOn the main page, select your network card hooked to the wifi network. Then click "Capture Options".
In Capture options table. Check to see that "Mon. Mode" says enabled, for the interface you want to use. If it doesn't, you'll only see your own traffic.
To change "Mon. Mode", double click the item in the table, and choose "Capture packets in promiscuous mode" and "Capture packets in monitor mode", press
If you are on an encrypted network then you might need to decrypt the traffic.
I sometimes have to fiddle with the IEEE 802.11 preferences: changing them, hitting apply, changing them, etc. Until I see the actual http traffic.
I also have to disconnect the android device from the network, and then reconnect it, so that it sends the initial network connection and decryption packets. Feel free to test on open networks where you don't have these type of issues because they are insecure if you want to.
Filter the captureAt this point your going to start seeing a lot of traffic flowing through your network.
So you want to filter it.
in the filter text editor type "ip.addr eq 192.168.1.143" or whatever the ip of your device is, to start seeing that traffic.
then if you just want to see the http traffic you can do
"ip.addr eq 192.168.1.143 and http"
or, to see just the GET requests:
"ip.addr eq 192.168.1.143 and http.request.method eq "GET""
This is a useful tool to have in your toolbox, for those moments where you have less control over the application under test, but still want to observe the traffic in your testing.
You can see examples of Wireshark in action to help test an Android app in our Technical Web Testing 101 online course.
Monday, September 08, 2014 16:26 PM
I make quite a lot of notes and prep for my talks before I present them, and so in this post I will walk you through some of the notes, and the process I used to get ready for the talk.
And I'll use the medium of the blog to expand on the topic a little with additional lessons learned from pulp authors, relating to test planning and preparation.
I think a lightning talk is 'as hard' as a full talk, in some ways it is harder. For those people who present a lightning talk as their 'first talk' because they think it will be easier. The only 'easy' part is that you are on and off the stage faster. I find I have to work much harder to condense my message into such a small time frame.
I made notes on a few different topics, but eventually decided upon the theme:
- "Are you ready to start testing tomorrow?"
- "A sense of urgency"? or "A sense of readiness"?
My first step when preparing a talk is...
- to just talk
For me, this is a purely temporary measure, and I delete these artifacts afterwards.
Then I collate my notes into a small first person script like essay.
Which in this instance came out as a single thread below. By single thread I mean, one topic, one path through, no 'asides' or references to analogous material.
Over the years, I've been on site and people have been talking about a "sense of urgency", 'people' generally means management. And "sense of urgency" generally means "why aren't these people working harder to meet the deadlines that we have arbitrarily imposed upon them.
When I get involved, I don't usually try and solve that problem. What I like to focus on is a sense of readiness.
Because I often see testers - not ready to test the software. They are writing the strategy and the approach and everything else they are asked to, but they aren't getting ready.
This is really basic but I ask people "could you start testing the software tomorrow"? If you could then you're in a pretty good place. If you're not ready, then you're in a pretty good place because you have your todo list to get ready, by asking 'what do we need in order to be ready'. Everything else - strategy, policy, approach, etc. is a bonus. Because if you're ready - you can communicate your readiness, and your 'documentation' is a result of taking the time to write it down.
And you need to be ready to test at different levels. functionality, requirements, domain, technology. But all of that is for nothing if you don't yet have the attitude that you could test this thing at the drop of a hat. Mentally building that 'testing' sense of readiness so that you could test it now, if you had to.
So I encourage you all. Build a sense of readiness. Are you ready to test a week from now, tomorrow, an hour from now, can you test it now? If not - work out why not and and stick that on your preparation list. and get ready.
This had the basics of what I wanted to cover. And I left it to sit for a while. Because really, I wanted to try multi-threading the talk. Adding in some analogous threads, creating and closing open loops as I talked. I hadn't tried this approach for a short talk before, but since this wasn't a 'Lightning Talk', it was supposed to be a 'Lightning Keynote', I wanted to add more texture to the presentation.
And during the 'sitting' period I read a Novel called "Silvertip's Search" by Max Brand.
You can see my copy above. The London, Hodder and Stoughton edition, first published in 1948. (Max Brand died in 1944)
And in here, I found a passage that I thought fit my topic. A conversation between the head bad guy and one of his minions.
Throughout the book, both the head bad guy, and the hero, are 'ready for anything' and 'at any time'. And In this paragraph, the head bad guy explains his secret.
"Are you laughing at me, chief?" he asked. "You know that nobody in the world can stand up to you."
"Nobody? Ah, ah, the world is larger than we are," said the criminal. "I should never pretend that nobody can stand up against me. All I know is that I keep myself in practice, patiently, every day, working away my hours." He sighed. "A little natural talent, and constant preparation. That's all it needs. You fellows are my equals, every one of you. Taking a little pains is all the difference between us..."This is on page 70 of the edition I own.
Given this, I thought I could weave into the presentation: the text of the book, and additionally, Max Brand and his writing strategies.
That would then give me at least three threads. One personal, one fictional, and one cross domain.
So my next set of notes looked like this.
Some managers like to talk about a “Sense of Urgency”, which in management speak means - why are my staff not working hard enough to meet these arbitrary deadlines we’ve set.
I read a lot of pulp novels. Most written to very tight deadlines. Generally filled with life and death decisions, made quickly, based on minimal information and minimal planning. Urgency in a pulp novel gets you killed, or lets the villain get away. Readiness defines the best heroes.
Max Brand knew a lot about deadlines. His official biography lists over 500 novels. and 400 short stories He was so prolific, that new books based on his outlines continue to be written and published after his death.
I see a lot of testers on site being busy, writing stuff like policies, strategies, plans, approaches etc. They think they are getting ‘ready’, most often they are complying with a ‘sense of urgency’ that says we need a strategy, or we need a plan. They are getting ready. They’re getting ready for their next meeting. But they are not getting ready for their testing.
And if you ask them. Are you ready? They’ll typically tell you about all the things they are waiting for, and they are in a holding pattern.
And that’s not what I mean by readiness.
Readiness works at different levels. Could you test an application that you don’t know anything about about? But you understand the technology it is built on? Or you understand the domain it sits in. There are lots of models around readiness: skills, domain, the app requirements, techniques, technology, and these models all overlap.
And if you were ready, you could test the app from the point of view of any of these models and add value. And gain enough time to develop one of the other models and test from another perspective. Your strategies, and plans and policies become a communication and explanation of your readiness.
A Sense of Readiness leads to a confidence and flexibility that you could test something if it was delivered to you tomorrow, or now.
So back to Max Brand, and specifically his novel "Silvertip’s Search".
One of the bad guys has betrayed his gang, and he’s up before the head bad guy trying to convince them not to kill him.
And I’ll paraphrase, here. Max Brand is a better writer than I’m making him sound like here.
The bad guy persuading for is life says “I wouldn't betray you boss, nobody can stand up to you”
And the boss disagrees, and Max Brand, or Faust, then has the lead bad guy describe his approach to writing. and it is “Nope, I don’t promise no-one is better than me. I just keep myself in practice, a little every day, constant preparation. We’re the same. Taking pains is all the difference between us”
Max Brand describes his prolific approach to writing. And how we go about developing a sense of readiness, because we don’t know what is going to come at us. All we can do is work on ourselves so that we have the confidence to tackle it if it comes in next week, or tomorrow, or today, or ten minutes from now.
I emboldened the first part of the sentence because that becomes the outline that I commit to memory to inform my talk.
At this point, I discovered the Internet Archive contains a version of the novel. The quoted paragraph is on page 86 of the Internet Archive version. Yes, if you want to read this novel, you can.
So I decided to download the novel to my Kindle and wrap the hardback cover around the Kindle as a 'prop' for the talk. Since I didn't want slides, and I was talking about the novel, having a physical representation of it seemed like a useful stage device.
And I could possibly build some tension by 'teasing' a reveal early in the talk, then reading from the novel at the end of the talk.And I added the following lines into the outline.
I brought along a pulp novel for you. This is Silvertip’s Search. A western published in 1945, based on his pulp story published in 1933, written by Max Brand. Or Frederick Faust - his real name. He created the character of Destry, and probably most famously Dr Kildare.
And hidden In this novel, Max Brand describes the secret of his writing success.
You can watch the talk and see how closely it matches the outline above. I think I missed out some stuff and I think I added a little more.
And now, in this blog, I can expand a little further - with information I wouldn't include in a lightning talk.
If I was using this in a longer talk then I might well include the information I'm about to give you below.
Additional reasons I like pulp as an example of readiness...
The pulp authors worked from small outlines:
- A Title
- A Paragraph
- A Blurb
- A short plot outline
- They wrote for money.
- So they had to pitch the story, and didn't want to spend the time writing a full treatment, so they pitched outlines. Sometimes they pitched titles, to see what grabbed attention.
- They could expand them, quickly, when needed.
- Sometimes they would be asked to contribute a story to a magazine with only a few days notice because some other author had let the editor down. And out would come, either an earlier story that hadn't sold, or an expanded form from the outline.
And because pulp authors worked like this, they often left behind lots of outlines, scraps of ideas, blurbs etc. Which hadn't been sold, or used, or expanded. Which is how pulp authors continue to publish work long after they are dead. Someone manages to take their fast preparation and turn it into something else.
And so, to relate this back to testing...
A lot of the time in testing we see promoted the idea that you have to prep in advance, and that your advanced prep has to have copious detail.
I don't think you have to. My experience of testing tells me otherwise.
I work to be 'ready' as fast as I can. I know there will be gaps in my readiness. But I know I could start, and add value, fast. And with each passing moment that allows me more time to prepare I increase my readiness, until at some point, I test.
And one external source of validation I use for this, is the work, and approach, of pulp authors.
Pulp authors used "a sense of readiness" to help them. We can too.
Thursday, August 14, 2014 21:22 PM
I couldn't work around the recent bug in VirtualBox version 4.3.14. It conflicts with my Anti-virus software under windows.
When I upgraded to version 4.3.15, I no longer experienced the anti-virus crash, but my networking was trashed and I couldn't get it working. So I decided to try and migrate over to VMWare.
I use VMWare Fusion on the Mac to run my Windows and Linux VMs, and VMWare VMs are cross platform. It seems like a sensible move, even though it will cost me some cash to buy the VMWare Player on Windows.
- Step 1 - convert VirtualBox to an Appliance
- Step 2 - load the appliance into VMWare
- Step 3 - uninstall the VirtualBox addons
- Step 4 - install the VMWare addons
- Step 5 - edit the .vmxf file to change the network settings
- Step 6 - reauthenticate the Windows license
- Step 7 - enjoy your ported VM
- Step 8 - delete the VirtualBox VMs
Step 1 - convert VirtualBox to an ApplianceIn VirtualBox I exported the VM as an appliance using the "File \ Export Appliance..." menu.
Step 2 - load the appliance into VMWareUsing VMWare Player, I open the appliance using "Open a Virtual Machine".
VMWare complains about certificates, but we tell it to retry and continue.
Then it prompts to save as a VM location.
Voila, the VM is converted - but it doesn't work yet.
Step 3 - uninstall the VirtualBox addonsStart the VMWare machine, and uninstall the VirtualBox addons.
Step 4 - install the VMWare addonsInstall the VMWare tools.
At this point, we should be finished, but I wasn't. I had to fix the networking.
It took me a while to find the right online resource for the next step.
Step 5 - edit the .vmxf file to change the network settingsI could not get the drivers for the ethernet adapters to work with the VMWare Windows XP VM.
I had to edit the .vmxf file that configures the virtual machine, and remove the "e1000" line.
Then, when I restart the machine, the correct drivers from the VMWare tools are used for the ethernet device.
I also had to make sure that the "Automatic Bridging Settings" in the VM Settings were set to bridge the correct networks, including the "Microsoft Wi-Fi Direct Virtual Adapter" and the "Microsoft Hosted Network Virtual Adapter".
Step 6 - re-authenticate the Windows licenseAt this point Windows complains that the hardware configuration has changed, and needs to re-authenticate.
For some reason, it wouldn't re-authenticate when I was logged in, so I had to restart the machine, and re-authenticate prior to the login.
Step 7 - enjoy your ported VMI couldn't face a re-install of Windows, and downloading all the service packs etc. So I wanted to get this conversion working, which is why I persisted, and you now see my notes as a blog post.
So now. I'll use VMWare for my VMs on Windows, as well as Mac.
Hopefully, Oracle will fix VirtualBox fully, as I like to have options.
But I think VirtualBox has lost me as a user for my licensed Windows VMs. The conversion from VMWare to VirtualBox is not as easy as the conversion from VirtualBox to VMWare.
Step 8 - delete the VirtualBox VMsSince I use licensed versions of Windows, in addition to the modern.ie VMs, I had to remember to delete the VirtualBox VMs.
Monday, February 17, 2014 16:16 PM
Those of us that have worked with computers for most of our lives, take the command line for granted. We know it exists, we know basically how to use it, and we know how to find the commands we need even if we can't remember them.
But, not everyone knows how to use the command line. I've had quite a few questions on the various courses I conduct because people have no familiarity with the command line. And the worst part was, I could not find a good resource to send them to, in order to learn the command line.
As a result, I created a short 6 minute video that shows how to start the windows command line, change to a specific directory, run some commands, and how to find out more information.
Start Command line by:
- clicking on start \ "Command Prompt"
- Start \ Run, "cmd"
- Start \ search for "cmd"
- Win+R, "cmd"
- Windows Powertoy "Open Command Window Here"
- Shift + Right Click - "Open Command Prompt Here"
- type "cmd" in explorer (Win+e, navigate, "cmd")
- Windows 8 command from dashboard
- dir - show directory listing
- cd .. - move up a directory
- cd directoryname - change to a subdirectory
- cls - clear the screen
- title name - retitle a command window
- help - what commands are available
- help command - information on the command
If anyone wants more videos like this then please either leave comments here, or on YouTube and let me know. Or if you know of any great references to point beginners at then I welcome those comments as well.
Thursday, February 13, 2014 22:02 PM
My install of VirtualBox prompted me to update today. And I realised that I hadn't written much about VirtualBox, and I find any videos I had created about it.
Which surprised me since I use Virtual Machines. A lot.
No Matter, since I created the above video today.
In it, I show the basic install process for VirtualBox. A free Virtualisation platform from Oracle which runs on Windows, Mac and Linux.
Also, Modern.IE, which I know I have mentioned before. The Microsoft site where you can download virtual machines for each version of MS Windows - XP through to Windows 8, with a variety of IE versions.
Perfect for 'compatibility' testing - the main use case I think Microsoft envisioned for the site. Or for creating sandbox environments and for running automation against different browsers, which I often use it to do.
I even mention TurnkeyLinux, where you can find pre-built virtual machines for numerous open source tools.
In fact the version of RedMine that I used on the Black Ops Testing Workshops to demonstrate the quick automation I created. I installed via a TurnkeyLinux virtual machine.
Oracle, even host a set of pre-built virtual machines.
A New Feature in VirtualBox (that I only noticed today)I noticed some functionality had crept in to VirtualBox today.
The cool 'Seamless Mode' which I had previously noticed in Parallels on the Mac (as 'Coherence' mode) and on VM Fusion on the Mac called ('Unity' mode). This allows 'windows' on the virtual machine to run as though they were 'normal' windows on your machine - so not contrained within the virtual machine window.
I love this feature. It means I no longer have to keep switching in and out of a VM Window and can run the virtualised apps alongside native apps. And with shared clipboard and drag and drop, it seems too easy to forget that I ran the app from a VM.
If you haven't tried this yet. Download VirtualBox, install the Win XP with IE6 VM, and then run it in 'Seamless' Mode so you have IE6 running on the desktop of your shiny whiz bang monster desktop. Try it. Testing with IE6 becomes a fun thing to do - how often do you hear that?
Saturday, February 08, 2014 07:56 AM
Google Chrome continually changes, which usually means good news as new features appear. Unfortunately sometimes it means changes to our existing workflow.
This happened recently when Google released a new version of Chrome, but moved the Emulator settings.
I eventually found them, and show you how in the video below:
Or for those of you that prefer to read, read on. I've added references at the bottom.
We have to start by using the Overrides in the Chrome developer tools settings. All the emulation used to exist here, but it has moved.
Right click, and Inspect, to show the developer tools. Then click the cog on the right to show the Settings. And show the Override settings.
So the first thing we do is make sure that we have checked "Show 'Emulation' view in console drawer".
So now where is the console drawer?
Close the settings and in the dev tools on any of these tabs, we can display the "Console drawer" by pressing the escape key, and lo the drawer did appear and an emulation tab was present.
And we can use the emulation tab to help us test.
In the demo video I show this in action on the bbc site.
Choose a device to emulate. I pick the "Samsung Galaxy Note II" because I have a physical device for that on my desk, and if I encounter any issues I can try the same functionality on my device.
Choose Device, Click Emulate, and you can see the screen size refreshes to a scaled smaller size.
You can amend the display settings using the 'Screen' options. By default it is shown scaled, but you can make if full size if you want.
But we still don't have the mobile site yet. So I refresh the screen. Using Ctrl+F5. And because Chrome is now sending the correct mobile headers for the Note II, we are directed to the Mobile site.
And now the issues.
I try and use the site. Click on the links. And nothing happens.
So, I change sensors and switch off the emulate touch screen. And we have a working site again.
This works on the Note, so it might be a BBC issue, or it might be a Chrome issue. But really it shows us the problems of testing through emulation, when we find suspected issues, we have to replicate them on a better emulator or a physical device.
But the Chrome emulation is so convenient on the desktop that for a first run check on the site, and certainly checking how your server responds to mobile headers, these are a great first step.
And you can stop the emulation by clicking the [Reset] button.
In the video I show a bonus, which I thought was an emulator bug, but seems to be by design by the BBC, where the Weather page does not redirect.
Chrome emulation? Very easy way to run a first check on the site, if you know how to access the functionality.
- Paul Irish: Chrome DevTools for Mobile Screencast and Emulation
- Chrome Developers Blog: Mobile Emulation
Tuesday, December 17, 2013 14:03 PM
I attended the Software Development Summit December 2013 in Helsinki.
I was fortunate in being asked to perform a keynote, and asked to fill in for a keynote speaker who unfortunately couldn't attend, so I did two keynotes. Lucky me.
You can find slides for the talks listed over on my Compendium Developments site.
I managed to catch up with Kristian Karl and learn more about the CI and testing regime at spotify, you can watch his Eurostar conference Experiences of Test Automation Webinar online (Q&A).
I was able to quickly hang out and see a few of the twitter enabled attendees: Johan Atting, Johan Jonasson, Aleksis Tolonen, and my fellow Eurostar Committee member Maaret Pyhäjärvi
I did receive a pointer to FreeNest - an open source platform put together by students which is designed to help teams get up to speed with a set of collaboration tools on Ubuntu quickly.
But it was all over very quickly and I had very little time to chat with Ilari Aegerter or Gojko Adzic.
The only downside I had was that I had to miss Gojko's Keynote because I thought I was flying out early. Of course the London Fog had other ideas so instead of learning from Gojko, I was stuck at the airport instead of enjoying the end of the conference :).
Tuesday, December 17, 2013 13:24 PM
I attended the SIGIST in December because I was asked to be part of a Panel with the starting discussion title of "Should testers be able to code?".
I was on the panel with Dorothy Graham, Paul Gerrard and Dr Stuart Reid
#sigist the panel gets going! pic.twitter.com/bHzcoSsiRm
— SIGiST (@SIGiST) December 5, 2013
Initial NotesI wrote the following in an email to the other panel members during the run up to the SIGIST. It wasn't polished but represented my notes and pre-conf prep.
I pretty much have to ignore the title "Should testers be able to code?"
In my mind "Should", at that point in the sentence equals "An obligation to..."
We don't work in an industry where testers have an obligation to code. So any question about obligation has no place in my reality.
I've met developers who do not seem to feel they have an obligation to be able to code. Similarly I've met testers who do not seem to feel they have an obligation to be able to test, or managers who do not seem to feel they have an obligation to be able to 'manage'. The process of Software Development has a lot of fungibility built in and can work with many different skill sets and skill levels on the project.
Personally, I do know how code, and while I can code some things as well as professional developers I consider my coding skills intermediate. Therefore I hope to phrase my answers in the form "When testers can code the advantages are ..." "When testers can not code the disadvantages are..." and "My experience of having up to date and intermediate coding ability has been..."
I do hold some opinions that might pop up:
- "Testers who can not code, should not write automation code"
- "Testers who can not code well, will write worse automation code than testers who can code well."
- and I suspect that "Many failed test automation programmes are a result of testers not knowing how to code".
- "I've also seen developers write awful automation code, I think automation code may require some different coding and design styles than application code."
But this panel isn't about automation. Automation != Testing.
Much of my recent on-line work has been about lowering the barriers to entry for people who do want to code or develop more technical skills. I prefer to help someone do something, if they express an interest.
Personally I try to learn as much as possible about the process, and skill sets involved in, Software Development. One small part of that involves 'coding'. Other parts include "Architecture", "Design", "Databases", "Modelling", "Protocols", "Tools", "Estimation", "Planning", "Communication" etc. etc. etc.
I don't think that any role ("Tester", "Developer", "Analyst") has exclusive right to a set of skills, techniques and knowledge: "testing", "coding", "modelling", "analysis" etc.
I value diverse skills across the team.
On the PanelMy brain, working the way it does, has forgotten most of the questions and answers, so I revisit the memory with some degree of trepidation, false memories may well rear their head.
I think many of the above notes were covered during the Q&A. I was able to pull on some of the material from my Helsinki Talk on 'Experiences with Exploratory Testing...' because someone in the audience said "Developers should not test their own code", and so I was able to riff off the T-Shirt slogan slide from that presentation.
Basically - statements like "Developers should not test their own code" and "Developers do not make good testers" are the kind of statements people post on internet forums, and we should relegate them to T-Shirt slogans so we can mock them and laugh at them. Developers do test their own code, and they can learn to test better. The sooner the 'test community' wipes this nonsense from its collective meme pool the better.
Because we were sitting down on the panel, my opinions were phrased in a less confrontational amanner nd with more humour than might appear from the written form on this page.
I remember saying things like:
- Projects depend on teams. So we need the team to have a diverse set of skills. And when we build teams look at the gaps in the skillsets.
- Keep investing in your staff to make sure they keep expanding and improving their skillsets.
- Becoming a better programmer has helped me test better.
- Becoming better at testing, has helped me write code better
- I recommend the book "Growing Object Oriented Software Guided by Tests"
- Teams are systems. As soon as you add a team member, or mandate that they do something, you change the system. Keep looking at and evaluating the system.
- Programming means lots of different approaches because there are different styles: OO, Functional, Procedural, etc. They require different skills and models
- Modelling is a vital skill for testers
We on the panel certainly had fun, I hope the discussions and alternative view points added value to the audience.
End NotesI think one message that came through from everyone on the panel was that testers need to have the ability to demonstrably add value to projects.
Having the role 'tester' does not mean you automatically add value.
Having the ability to write code does not mean you automatically add value (you might write really bad code).
Each tester needs to identify how they can add value.
The current market vogue is for testers with coding skills.
If that isn't your thang. Then it might mean becoming an expert in UX and psychology so that you can add more value for user focused testing. It might mean a whole bunch of things.
Each tester needs to figure out what they can do to add value, and what they can do to demonstrate their capabilities to the teams or potential employers.
And keep improving.
Friday, November 22, 2013 12:36 PM
A long time ago I coded a now defunct modelling tool to help me with my testing. Half the battle with managing and reporting testing involves deciding how you will model it for the project you work on.
Lightweight Subjective Status Reporting
How to do this using Jira?
Use Dashboards to make things visible
But people don't like writing reports
You can report Daily, with mininal overhead
Friday, November 01, 2013 18:40 PM
I listed the results of my investigation into Android Screen Capture, Streaming and ScreenRecording tools for Mobile Testing, now time to turn to iOS.
Note that I'm not really covering static screen capture here, since short cuts are built into each operating system for static capture.
iOS is pretty locked down. And without jail breaking, your options are limited.
However, iOS has a built in screen sharing capability called AirPlay, designed to be used for streaming to your Apple TV. But that hasn't stopped some enterprising developers building AirPlay servers for both Windows and Mac OS computers.
Both are insanely affordable. And both have a version for Windows and Mac OS.
AirServer offers more configuration options, although it worked fine out of the box for me. AirServer offers a 7 day trial.
Reflector offers a trial where you can star as many sessions as you want. But each session only lasts for 10 minutes.
To capture your on-iOS-testing, 'airplay' the iOS screen to your desktop or laptop computer, and then use a screen recording tool like Camtasia, BB Flashback (or its big brother BB Test Assistant) and capture the screen movies there.
This doesn't let you interact with the actual iOS device from your computer, but goes some way to making your testing recordable and reportable.
Friday, November 01, 2013 17:07 PM
I'd like to embark on learning from your books and online course but should I do one before the other? Or does one set of materials supersede another?
A CorrespondentI receive this question often enough that I'm going to try and answer it fully on the blog.
On a timeline, I created the following products:
- Selenium Simplified Book
- Start Using Selenium WebDriver - a 'Getting Started' online course
- Selenium 2 WebDriver Basics online Course
- Java For Testers Book
- Technical Web Testing 101 online course
So, for Selenium WebDriver I created 3 products:
- To get started, setting up the tools Start Using Selenium WebDriver
- To provide detailed coverage of the WebDriver API without too much 'Java' hand holding Selenium 2 WebDriver Basics online Course
- To learn the Java needed to actually code automation Java For Testers
And since it seemed top heavy on Automation, when that only represents part of what I do in my daily work life, I created the Technical Web Testing 101 online course to introduce people to the tools and thought processes I use when testing Web Applications.
I think they complement each other rather than directly overlap or supersede each other. Java For Testers is designed as a stand alone introduction to Java Programming and the WebDriver course doesn't spend a lot of time explaining the Java used.
Friday, November 01, 2013 12:14 PM
Connecting an iOS device to an HTTP Proxy is much the same as we demonstrated on Android devices.
On your iOS settings:
- In Wifi
- Select the (i) information icon next to your wifi network
- On the bottom of the screen is the HTTP Proxy settings
- Set this to Manual
- Type in the Server IP address and port of your proxy
Friday, October 25, 2013 10:02 AM
We took delivery of the new iOS devices for mobile testing. And in order to activate them, you need to connect over Wi-Fi But they wouldn't connect to the Wi-Fi network during the activation wizard. What to do?
I decided to use my laptop as a local Wi-Fi hotspot. That way I could configure it with different passwords and encryption types to try the different options and hope that the iOS device would connect.
I had trouble sharing the Wi-Fi connection using the in built windows functionality (ably explained on lifehacker)
So I installed connectify.me to help me use my laptop as a hotspot. And LO! The iOS devices happily connected through my local hotspot and activation could ensue.
Of course - once I had activated them, the iOS devices had no problems connecting to the Wi-Fi network I originally wanted to use for activation.
I suspect that the local hotspot approach might have some useful secondary benefits that I may have to research:
- monitor traffic without having to setup a proxy
- throttle the network speed
Thursday, October 24, 2013 14:57 PM
I chain HTTP debug proxies.
That way I can use features from all of them, at the same time:
- Fiddler Autoresponders
- BurpSuite passive sitemap building
- ZAP's multiple breakpoints
FiddlerI usually work on Windows, so the first proxy I start is Fiddler. Fiddler hooks into the windows system seamlessly without any additional config. All other proxies I point at Fiddler as the down stream proxy.
When fiddler is running. Test your setup by pointing your browser through Fiddler.
BurpSuiteBurpSuite Options tab. Upstream Proxy Servers. Add an entry for Fiddler:
- Destination Host: *
- Proxy Host: localhost
- Proxy Port: 8888
At this point - test your setup again. Don't chain everything together and then try and figure out where the problem is. Point your browser at BurpSuite and check you can see traffic through them all.
If you get stuck, use the Alerts tab in BurpSuite to check for errors.
Hint: Firefox and Opera maintain their proxy settings independently from the Windows settings so test your setup with Firefox or Opera.
ZAPTools \ Options \ Connection. Then point it at BurpSuite
- Port: 8082 (or whatever you bound BurpSuite too)
And point the browser at this port now.
EndVoila, you should have it all chained.
If not, just revisit the last step. And don't panic.
Step by step. check each part of the journey. If its not working, it will probably just be some stupid error where you had previous config left over from a previous session.
Just remember to unwind them when you are done.
I have a video in the Technical Web Testing 101 course that shows this in more detail.
Wednesday, October 23, 2013 12:46 PM
- You need to find the right IP address on your desktop
- You need to change your proxy settings on your phone
- Make sure your proxy allows external connections
- Make sure both Mobile Device and Proxy Machine are on the same network
- Phone connected to wifi network
- Windows Desktop connected to same network via wired connection (or laptop connected via wireless)
- Start Fiddler
- Check that Fiddler has "Allow remote computers to connect" (via Tools \ Fiddler Options \ Connections)
- Start a command prompt and use "ipconfig" to show you the current list of ip addresses for you computer
- Open Settings
- Wi-Fi settings
- Long press on the wireless network you are using to access the connection settings
- Modify Network Config
- Proxy Settings - Manual
- change the Proxy HostName to the IP Address of your desktop
- Add the port for your debug proxy e.g. 8888 for Fiddler
It makes a big difference to the visibility of your testing when working through mobile.
- For burpsuite, in "Proxy \ Options" edit the proxy listener to Bind to address "All interfaces"
Friday, October 25, 2013 09:43 AM
I looked at my mobile testing options and I realised that I didn't have a full toolbox to help me.
I first wanted to identify screen capture and screen recording options for my Android devices.
Most of the tools I found wanted root devices. When testing, you may not have this option, people might come across paranoid about interfering with the device state.
So I limited myself to tools which did not require root access. They all pretty much work the same way using ADB and Debugging over USB enabled.
If you bought all tools that I recommend in here, then the total cost would hit the dizzying height of £8.98, so I don't see a lot of point trying to roll my own solution.
To use these tools you pretty much need to have a working SDK setup. So work on that first. And if you can connect to your device with adb or monitor.bat then you're probably good to go.
In order to record the screen for some of these I use them in combination with a desktop screen recording tool like Camtasia or the Blueberry Software tools BB FlashBack or BB Test Assistant
Free and Open Source Tools
Droid@Screen is a pretty good wrapper around the adb.
The main GUI display shows a continually refreshed view of the device.
- You can take a screenshot very easily.
- The main GUI has easy orientation buttons to adjust the GUI display for landscape or GUI.
- You can capture screenshots to a folder automatically.
- You can view device properties
- You can scale the output view
Android Screen Monitor
Much the same as droid@screen, the GUI is simpler with a right click menu instead of icons.
Sometimes this is a little faster than droid@screen, sometimes droid@screen is a little faster.
ASC - Advanced Screen Capture
ASC performs on device screen capture, so it writes a movie file to the phone's memory. It has a bunch of options to adjust framerate. What I particularly like is that it will highlight the taps you make on the screen so you can view the interaction on the device.
On non rooted devices requres you to use an 'activation' program on the PC or mac. The desktop activator program acts as a simple way of making a connection to your device and taking a screenshot, so an easy way of accessing some of the sdk funtionality.
Looking at the popups as the screen 'activates' it is using adb in some way - I assume to enable the android screenshot api.
Application description on the play store says it only works on non-Tegra devices. The trial worked fine on my Samsung Galaxy Note II.
Buy through an in-app purchase for £3.99
Activation Notes: I had some trouble activating it after purchase, but after a few emails with the developer. I had to uninstall it, then re-install it, then click the 'buy' again (I wasn't charged twice). The activation does work, but a bit more fiddly than it needed to be.
VMLite VNC Server
A desktop program to start the server on your phone if you work non-rooted.
Once the server runs I can head off to http://<deviceip>:5801 to use the HTTP interface.
Or connect a desktop vnc server to <deviceip>:5901
The HTML5 viewer was about the same as Droid@Screen or Android Screen Monitor.
The Java Applet VNC is a little faster, and the best out of the desktop tools I tried.
The video was not as smooth as ASC, but remember that this has the advantage that I can interact with the android device as well from the desktop and use my mouse and keyboard.
Used But Can't Recommend Fully
- Android Screen Cast
- Faster than Droid@Screen
- But a bit flakey on my machine and threw some errors. I tried it on Windows 8, it might work well on other machines.
- Looks Good, but hung on my Windows 8 machine after one screenshot. It might work for you.
- Desktop Connection for Screenshots and low frame rate streaming
- VNC for higher framerate and interaction
- On Device Capture for High Frame Rate
Friday, September 27, 2013 08:00 AM
I seem to default to WinMerge for my file and directory comparisons.
Whenever I need to:
- compare two directories
- compare two files for differences
- copy a set of files between directories
- compare contents of zip files or rar files
When you install WinMerge:
- You can choose to use it as the merge view in Tortoise SVN. I tend not to do this because the built in Tortoise SVN diff works fine for me.
- Add WinMerge to your system path, which allows you to call it from the command line easily and use the command line options. I choose this option.
- Enable explorer context menu integration - an essential option.
- You can drag and drop like a crazy man
- drag and drop folders on to the desktop icon for comparison
- drag and drop on to the main pane to start a compare
- drag and drop into the input fields
- If you install 7-zip and the7-zip plugin you can compare rar and zip files
- 7-zip acts as my default archive management tool
- The tree mode makes life excellent, just make sure you enable it
- Look through the options and switch on all the "Shell Integration" options - particularly the "Include subfolders by default"
- I create a backup folder before testing. I test. I can compare and see what changed.
- I want to revert back to previous files selectively, so I compare dirs and selectively move changed files
- I have an oracle file and want to compare results
Wednesday, September 25, 2013 09:28 AM
A few days ago I realised that one of my use cases for Fiddler, did not stem from a need for debug proxying.
- Game Over? Stop using the tool?
- Mess around with config settings for a few hours trying to get it to work?
- Use Fiddler?
Wednesday, September 11, 2013 08:40 AM
We spent a good 5 or 10 minutes thinking we were crazy. "I'm sure the option used to live here..."
As ever, Google came to the rescue..
Why did Mozilla do this? Because of "Checkboxes that kill your product".
Unfortunately, as a tester I still experience moments where I need to kill the product, so How can I do that now?
Add-onsOne of the official forum Q&A items mentions an add on called QuickJava, just make sure that enable the "Add on bar" from the toolbar menu "View \ Toolbars \ Add-on bar"
Other plugins mentioned in this other official forum Q&A item
Friday, September 06, 2013 10:35 AM
I have some 'rules' that I apply when I take notes as I perform exploratory testing.
When I look back over how I took notes in the past I can see that I tried different experiments with my approach when building those 'rules'.
I recommend some of my experiments to you now:
- In Memory
- Only use pen and paper
- Only use a text editor
- Use a text editor and screenshot tool
- Record the screen and talk as you test
- Use a tool designed for exploratory testing
- Use a Mind Map
- Draw a diagram
- Automate the capture of logs
- Use a Spreadsheet
1 - In MemoryOnly use your memory to track your exploratory testing.
This experiment mainly helps me remember that I need to do more. I now feel very uncomfortable testing with just my memory, but when I started this felt natural. I changed.
2 - Only use pen and paperYup, you use your computer to test, but you make notes on pen and paper.
- different pens,
- different colours,
- different sized paper,
- loose paper,
- mind maps,
I find this works well with a single screen, and intense moments in the testing, but I try to re-transcribe or take a photo with my phone and have the image in evernote.
3 - Only use a text editorExperiment with different text editors, find an editor you like - I've pretty much settled on NotePad++ and Sublime Text, and I import the text files into evernote for later searching.
Different styles of note taking:
- Time Stamped Entries,
- Annotations like #test #bug etc.
Can you parse your logs at a later date automatically? Would that benefit you?
Touch typing helps. learn to touch type if you can't already.
4 - Use a text editor and a screenshot toolSometimes you need to capture the moment as a screenshot.
Do you just use Ctrl+Print? Do you use an image editor? Do you use a dedicated screenshot tool?
I tend to use SnagIt or Jing now. I've used lots of others in the past.
- Where do you store the images?
- What filename standard do you use?
- How do you cross reference your text edit notes to the screenshot?
- Do you think you would benefit from using a Word Processor and embedding the screenshots along side your text?
5 - Record the screen and Talk as you testEnvironment can get in the way for this if you work in a shared office.
Do you have equipment that you can comfortably use for long periods of time?
Talking, and thinking and doing takes practice and time.
6 - Use a tool designed for exploratory testingPeople have created a whole bunch of tools; designed, or marketed, as helping exploratory testing.
Try them. See if they work for you.
If your style clashes with the tool. consider if the tool benefits warrant a change of style from you.
If you could write your own, what features would it have? Perhaps you could use a combination of tools to gain those features now?
7 - Use a mind mapEveryone loves creating mind maps. Few people use mind maps like Buzan suggests. Who cares, use mind maps and do it your way.
What do you represent in the model?
- To Dos?
Over time, learn the features of the tool. consider which features you don't use. Should you? Would they help?
Perhaps you don't use enough of the features? Try a less featured tool and see if it still works for you?
8 - Draw a diagramPen and Paper works well for diagrams.
What do you diagram?
GraphViz lets you write text files that it compiles into automatically positioned graphics.
You can use draw.io as an online diagrammer.
At last count there existed a Bazillion diagramming approaches and tools. Try some of them.
9 - Automate the capture of logsYou can't argue with logs right? Why bother making notes when the logs will do it for you?
- Fiddler - for HTTPsessions
- tail system logs (logtail, multitail, etc)
Do the logs capture everything you need?
How do you cross reference your notes to your logs and to your screenshots?
10 - Use a spreadsheetWhat about a grid?
Would that help?
Try it and see.
RepeatThe above covers a lot of note taking styles
- Outline, Tree
Take care about your judgement because some of it didn't work due to your lack of experience - try it again. Some of it didn't work because it doesn't fit you, your environment, your system, etc.
Having done them all - try them again. Some of them will seem offensive. Some will feel restrictive. You gain gain more insight when you try it again.
SummaryEven though I gave you this as a "10 experiments to improve your exploratory note taking". I have not given you a quick fix.
- If you tried one each day, this would take you 2 working weeks.
- If you try variants in each of the experiments ( different tool, different paper sizes, etc.) This could take a month or more.
- If you repeat them and challenge yourself to master them, and change them, this could take up to 6 months.
- I still experiment with my approach. I have done for years.
What experiments would you recommend" or "What experiments you have conducted?". Let me know in the comments below.
You might want to watch "What is Good Evidence" by Griffin Jones, which reminded me I needed to write about this. Griffin's talk overlaps very nicely with this post. I recommend you watch it.
Friday, August 30, 2013 12:43 PM
Normally I add my automation posts to SeleniumSimplified.com but this particular case study demonstrates how I think about testing and incorporate automation into my test approach.
The scenario you face as a tester:
- You have a main web site www.eviltester.com
- You have a new mobile site m.eviltester.com
- You have a set of redirection rules that take you from www to m. based on the device
- And the device is identified by the user-agent header string
The first thought for testing?
- We need to get a bunch of devices to test this on.
- We could spoof the user-agent.
- Well, Chrome has the override settings where we could choose a different user-agent.
- We could have our debug proxy change the user-agent for us.
Where will we find the user-agents?We need an oracle source for our data set of user-agents. Fortunately there are a few sites out there that track what user-agents are in use:
useragentstring.com - if you have a preference that differs then leave a comment and let me know.
So I wrote some code. And I know about all the "testers shouldn't code", "tester's don't need to code", "blah blah blah" discussions.
I can code. It increases my ability to respond to the variety of conditions on a project. Requisite Variety. I encourage you to learn how to code. (Hey I'm writing a book about that.)
So I code.
I wrote a simple set of Java code that:
- Uses GhostDriver - the new headless driver wrapper around PhantomJS
- Visits useragentstring.com and scrapes off the user-agent strings
- Filters the user-agent strings to those that I consider 'mobile' devices
- Iterates over all those user-agents
- Creates a new GhostDriver with that user-agent and visits the www site
- Checks that I redirect to the mobile site
Surely it would be faster to use direct HTTP calls?
- Faster to run, but not necessarily faster to write. Yes.
- See I can use the WebDriver findElements commands when scraping the page and not have to remember how to parse XML in Java or download another Java library.
- I can use the WebDriver to visit the site and handle all the redirection for me, rather than write some redirect handling code for the Apache HTTP libraries.
I tidied it up a little for release to github so it isn't completely embarassing, but hey ho, it added value. I'll use it again. It looks pretty nasty, but it works.
Sometimes that's the type of automation I write when I test.
But that wasn't the requirement scope!
- True. It wasn't.
- The requirement scope was small.
- Sometimes we have to explore.
- I look for external oracles and comparative sites and rules to help me evaluate if the requirements meet the actual user need.
- In this instance I found a lot of user-agents that the redirect rules didn't cover.
But if it wasn't in the requirements we can't justify the testing!
- I can use a comparison with other sites handling of the user-agents (e.g. bbc or tfl)
- I can see if the gaps in the system under test are better or worse than theirs.
- BBC didn't handle 1 user-agent I found,
- TFL didn't handle 3,
- The system under test didn't handle 100+
What would you do?Do let me know how you would have done it differently.
Tuesday, July 23, 2013 20:40 PM
I feel anger when I stumble across very, very, very simple security issues. Especially when they compromise my data.
Yes I do. And I hope, as a tester, that you do too.
But I face a problem... As a tester, I can't say "Did no-one test this!" because I know that they might have done, and someone else might have chosen to go live anyway.
But on the off chance that no-one did 'test this', I offer you this post.
Security by obscurity
If I visit your site and I can gain access to top level pages that I shouldn't have, then I get angry, because that should never happen.
Even if you haven't told me about the URL, I can try to guess it from your other naming conventions.
Please make sure you secure your URLs.
Security by obscurity doesn't work for very long.
Validate Parameters on the server
And if I see parameters in your URLs. I'll change them.
Yes I will.
- Because I don't want the list of items to stay limited at 25, I want to see 2500
- No I don't care about the performance impact on your database - fix that a different way
- Because I want to skip ahead more pages than you have listed on the page. "I want to go to page 15 now!"
- No I don't care about the user experience you want me to have, I care about the user experience that I want to have
- Because I can
- Yes, because I can
Security by ignorance
When I visit your site, I look at the traffic you issue when I hit a top level URL.
I look at what requests you make to build the page.
Yes I do.
Most browsers have the functionality to view traffic built in now. Any user can view the traffic and see the URLs that you access.
And then I use that information...
I take the URLs you've used. And I use them.
Sometimes I change them. Simple idea, but sadly all too effective.
- So if you access
- http://site/api/123456/report (note: not a real domain)
- I'll access
- http://site/api/123455/report (note: I changed the number)
And if you haven't locked down the API to specific users, with specific role level controls. Then I'll probably see another user's report. Then I get annoyed, because as a user it means that other people can see my reports. And I don't like that.
No I don't.
Assume that anything you do automatically someone else will do manually.
Just because they can.
Make sure you have low level permissions on your API, don't assume that no-one will notice it.
I frequently do this because I want to bypass the horrendous GUIs that web sites put in my face, when I want to achieve a result, rather than experience your broken GUI. So I script it. Or if I can get away by posting a URL with some data, then I'll do it.
I bet other people do this too.
Testers should do this too, because...
Security by ignorance doesn't work.
Security through sequential numbering
And if you're using sequential IDs for reports, or users, or accounts, etc. you actively encouraged people to hack you.
Yes you did.
No-one has ever recommended - Security through Sequential numbering.
No they haven't. Never Ever.
Security through sequential numbering doesn't work.
Tips for Testing
So now, the inevitable 10 tips for testing:
- Play with the URLs
- Change URL Parameters
- to check that permissions surround the public level
- to check that request validation takes place
- Try URLs when logged out to make sure permissions apply
- Guess some URLs
- Use an HTTP Debug proxy and look at the traffic
- Investigate the traffic and see what the requests do
- Issue the traffic requests out of context on a page to understand the 'real' state rules in place
- Change the URL parameters in the traffic URLs
- to check that permissions surround the AP
- to check that request validation acts at the API level, not just the GUI level
- Issue the requests when logged out to check the permissions still apply
- And if you do test like this, and your organisation keeps ignoring these types of defects, check if you reported them effectively, and if you did, then leave because that company doesn't deserve you.
You wouldn't like me when I'm Angry
I didn't even describe security testing above. I described functionality testing.
And really basic functionality testing at that, just simple input variations. I haven't messed with cookies, I haven't done anything hard (because cookie editing ain't easy, 'right kids).
If you don't include this "really really simple stuff" level of test activity, then please let me know so that I can avoid your site and find a competitor quickly before we develop a user/supplier relationship.
I really don't like getting angry when I act as a user.
Trust me, you wouldn't want me as an Angry user.
- Yes, this blog post does describe problems found at a specific web site.
- No I will not name that site.
- Yes, I have already told them... and more than once.
- Yes I have started looking at alternatives, sigh.