To reduce the pain associated with installation testing we banked upon some small checklist for installation engineer and a home grown tool to compare registry values and file systems with expected values.

Checklist was very small and comprises of basic questions like all the appropriate versions are included or not? Should we expect new registry values? If yes, what and where? Should we expect some changes in the file system ? If yes, what and where? Our home grown tool was also very simple and was comparing new values with the known oracles which can be defined by installation engineer or tester fairly easily.

We found this approach very useful as having a small checklist made it easier for installation engineer to remove his mistakes and gave sufficient information to the qe team on what should be expected in new build.

+++++++++++++++++++++++++++++++++++++++++++++++++++

Matt Heusser

If I understood you correctly, I have a couple of ideas.

First off, Do you have some level of documentation for these?

In other words, can you say: "We're spending an average of X hours a week finding Y installation bugs"

You numbers don't have to be exact.

If you have something, you could present it as an engineering challenge to the developers:

"Hey, guys, I know you perceive testing as a bottleneck.  Right now, we are spending a lot of time (tangible numbers go here) verifying the installation process.  If we could decrease the time we spend doing that, we could focus on verifying other aspects of the software, and decrease our overall testing time.  Do you have any ideas?"

Gentle directions to push the dev in ->

1) Offer to pair program on the install, so less bugs come through
2) Write some simple scripts to do the verification for you (a dir or ls command, piped to a file for the oracle, do the install, do another dir or ls command for the actual results, then diff)
3) Find some other way that the dev team can decrease the defect count before it gets to you
4) If this work is manual and repetitive, outsource it. (Seriously, could some stay at home mom do this work for $8/hr?  Maybe 10 or 12?  An Intern?)


A few years ago, I would have pushed hard on #3 - the code is in an unacceptable state before it gets to test; the developers have to take some responsibility and fix the !@#$% problem.

A few grey hairs later, I'm not so sure.  Now I would say that the key is to either get your management or the developers to take ownership and the credit for it.  Let it be their idea.  Give it to them as an engineering problem; let them solve it.  Don't make it be an embarrassment; instead, let it be a win for them.

It's amazing what you can get done when you don't care about the credit.
+++++++++++++++++++++++++++++++++++++++++++++++++++

Adam Goucher

At HP, we partially addressed the installer testing problem in the following ways
- Do most of the installation testing through Install Shield's command line or silent modes; the logic of the gui installer is still executed the same, but the presentation mode is different. This lets things get driven through automation (python in this case)
- To verify the files, we would spider the installation directory and get a checksum for each of the files which would be matched against one generated by the build process (pre packaging into the installer)
- Do a code review / inspection of the build files / system to ensure that basic sanity measures are in place. At one point we discovered that we were doing a version control update instead of a full clean and re-sync.
+++++++++++++++++++++++++++++++++++++++++++++++++++

Walter Gorlitz

We use a single build machine. This eliminates the need to have developers drop the files in place. After the build has been completed, the process that started the build tells our installer to grab the required files. This addresses the repeatability of creating the installer. Our installer guy just has to approve all changes to the installer scripts. We still have to do installer testing.

The problems are:
1) when developers change their compiler code and DLLs get built  elsewhere, we ship old DLLs until we delete the entire binary location.
2) occasionally, when we use checked-in DLLs (particularly common with 3rd party code) and a developer moves that DLL, the installer fails to complete.

In Another Post

We use Wise. We just call Wise from the command line with a parameter of the script to be used.

This is how installer scripts work: The installer application goes out and grabs all the components required from their locations and creates an installer. This installer knows where to put these various files.

The files could be laid out like this
install\bar.dll
install\foo.dll
projectx\coolgui.exe

The installer can take all of those, ask the user where she wants the  installed files to be dropped and you come out the other end with
c:\programme\bar.dll
c:\programme\coolgui.exe
c:\programme\foo.dll
(assuming you're using a German Windows system).

This is a very simple example. I would imagine that there are more complicated ones available on the net.

+++++++++++++++++++++++++++++++++++++++++++++++++++

Joao

I share your concern.

It is indeed a huge task to verify setups, because if you need to repeat the test you tipically must:

1-Uninstall

2-Check that everything was removed (like registry stuff, etc) – Probably you have to verify the uninstall feature as well…

3-Install

4-Perform the verifications

In my company we have patches every week, and you cannot imagine the immense diversity of problems such as:

-This file is missing because we forgot to include it

-This file is missing because we did not know we needed to include it

-This other team did not tell us this shared file was changed, so we did not include it in our module.

-There were “compilation problems” (corrupted files)

It seems to me that the way to resolve this problem is more on their side. I mean, they should find procedures and methods that are more reliable, more automated (in the sense that they don’t have to think every time about what files they should include, etc). The more tasks they have to do “by hand”, the more likely they are to miss some. They should get check lists or any other tools to help them.

I ought to say that if a file is missing from a release, the responsibility is from who released the product.

I don’t see our function (tester function) as a parachute to developers work, because I guess it is impossible to verify everything.

Quality is not responsibility of the quality department (QA, Test team, whatever you call it); it is responsibility of the whole R&D team. (I read this on paper “classic testing mistakes”, I think)

+++++++++++++++++++++++++++++++++++++++++++++++++++

Dave

I've had some limited contact with install testing before. I have used a tool called inctrl5 (http://www.pcmag.com/article2/0,1759,9882,00.asp). It's fairly cheap as test tools go - you can download it for $8. It will watch an install and compare before and after snapshots of the disk, registry, and specified files.

You might look into putting together an automated smoke test that can run through some installs and report any obvious problems quickly.

+++++++++++++++++++++++++++++++++++++++++++++++++++

Michael Bolton

I got a clear whiff of that from your initial report. I had in mind a somewhat different answer from most of the respondents: to my mind, your testing job may be done when you've identified the amount of time and effort that is being expended on the problem as it is. One important part of that story is the opportunity cost: every moment you spend on investigating and reporting a problem that could otherwise have been anticipated and addressed is a moment that you can't spend on getting test coverage on the rest of the product. The problems that you know about are therefore concealing problems that you don't know about yet, and that's usually a pretty serious risk.

As for driving change when there's an untouchable developer involved, there be dragons. Be careful out there.

+++++++++++++++++++++++++++++++++++++++++++++++++++

Neeraj Tikku

In my last company I was also on the other side of the fence, that is, I was the one who had the responsibility of creating the installer besides testing them. To add to the complexity, the product I created installer for, was supported on windows platform plus 3 flavours of UNIX [HP, AIX & Solaris].

Below is the approach I took:

1. Created a single Installation script file (we used Install Anywhere so that helped) for all platforms.
2. Created a single ANT script that did the following:
a) First, it would check the build date of all the files that are to be packaged and compare them to the last check-in date (since all files were sourced from a Version Control System)
b) Second check was to make sure that none of the file is of zero size.
C) Then created installer using Install Anywhere's command line utility.
d) Once the installer was created, we executed the installer in silent mode and did a complete installation for each platform.
e) Once the installation was complete, we again check that all the files are copied in appropriate directory structures and the values in properties files are changed appropriately (compared to baseline files).
f) Then brought up the product for Testing team to continue working.
f) Once all the checks passed, the installer was copied to a central location.

If any of the check failed, a mail was sent to appropriate people with all the details. Plus if the post-installation check failed, then the script used to remove the new installation and restore the backup of working setup.

All the installation were done on the setups that were used daily by testing team. Now since you are not the one who is creating the installers, you'll need to have development's buy-in for setting up such kind of process. But if you can translate this in to scenario specific to you, I'm sure the value derived would be too great for development to look other side.

+++++++++++++++++++++++++++++++++++++++++++++++++++

Eric

In a review, I would point out that the test case presented should be optimized for the problem space being solved for.

I’ve been doing install testing since Installshield was a pup, and there’s no way I would expect uninstall to set the correct base state for the next install testing round. I don’t mean this to seem like an Installshield problem, it’s common for the install developer to miss things. And have QA test quality into the installer.

The gross-level steps I’d follow for the install/uninstall testcase (there are so many more than this one) look like this:

1.Restore base image (from Ghost, VMWare, MS VPC, format and reinstall Windows, something)

2.Install anything else you want to have there as part of the test (i.e. back when apps conflicted due to DLL Hell)

3. Take filesystem and registry snapshot

4. Install

5. Verify File system, registry, any apps (DLL Hell again, but not always…)

6. Uninstall

7. Verify File system, registry, any apps (DLL Hell again, but not always…)

 Of course, this TC does not specify implementation method (manual, automated, etc.). In my case, automated, automated, automated. I’m the laziest, while being thorough, tester in the world and automate everything that gets me ROI while balancing for objectives. Before we get onto that tangent, let’s be clear that manual testing (ad hoc and structured) is a MUST for any GUI work to ensure a great customer experience. But from a functionality regression resource perspective, it’s not the right tool IMHO.