Continuous performance Monitoring

Performance testing is an important and integral part of most testing projects. This type of testing corresponds to Q4 of the Agile testing quadrant. You can find interesting insights on the agile testing quadrants in this post by Lisa Crispin.

Usually performance testing teams are different from functional testing teams and their reports / data etc are not easily available to to the entire team. I wanted to have more visibility, integration and feedback about the performance of application - essentially I was looking for Continuous Performance Monitoring.

In this post I will discuss what is continuous performance monitoring and how useful it is to report performance trends for every build.

In my current project, I am using TeamCity as the build server. TeamCity supports custom charts for any data. I thought, It should be possible to have performance data from all the teams in a particular format and have TeamCity display those results in its custom charts. This could be the solution I was looking for continuous performance monitoring.

What is Continuous Performance Monitoring?

So what is Continuous performance monitoring? Well, I have based it on continuous integration. Continuous integration means - integrating system with every check-in and executing some checks against the integration to check sanity of check-ins. So continuous performance monitoring is - monitoring performance of application with every check-in and making results visible as charts.

Continuous performance monitoring allow teams to identify performance bottle necks after every check-in - thus making it easier to find them and rectify them.

With continuous performance monitoring, teams can get historical view of performance data which allows them to drill down to specific commits/builds causing performance degradation.

Let's have a quick look at how easy it is to identify broken builds from the performance perspective with a sample graph produced by continuous performance monitoring -

Steps to report performance data as graphs in TeamCity

So before starting steps, I suggest you make yourself familiar with the workflow described in following diagram

Hope you got basic idea about the workflow and lets start with how to make use of TeamCity to report data in the form of graphs :

• Produce data by tests in XML format which TeamCity can consume.
• Use build script to pass service messages to TeamCity.
• Use build script to copy data from this file into another build script.
• Get TeamCity to report the data as charts.
• Setting up the build configuration.

Lets have a look at these steps in detail.

Produce data by tests in XML format which TeamCity can consume

Producing and collecting data from the performance test is the first step to produce these reports. Performance data can be reported in the form of key-value pairs where Key may be the service operations and value is the time in milliseconds- time taken for that operation.

Lets have a look at one sample XML file - MyPerformanceData.xml

<?xml version='1.0'?>

<project Name='TestProject' xmlns='http://nant.sf.net/release/0.85/nant.xsd'>

<property name='chart1Key' value='900'/>

<property name='chart2Key' value='50'/>

Use the build script to pass service messages to TeamCity

Teamcity allows interaction between build scripts. This is useful to send service messages to Teamcity. In my case, I had to send custom statistics to Teamcity so I made use of TeamCityReportStatsValue service message provided by Teamcity. Have a look at the snippet of build script that was used to send the service message.

Note that key name is the same as the one reported in MyPerformanceData.xml This is added in the Target node of the build script.

<TeamCityReportStatsValue Key='chart1Key' Value='$(chart1Key)' />

<TeamCityReportStatsValue Key='chart2Key' Value='$(chart2Key)' />

Use the build script to copy the data from this file into build script

The value performance value of the key will be fed dynamically from the performance data file into the build script. This requires XML file read task in the MSbuild. This is done by using the MsBuilCommunity task. I downloaded and copied it locally(or on the build agent). This task allows you to read performance data XML file and assign the values to the build script. I have copied the build script snippet to copy the XML values for keys.

   <MSBuildCommunityTasksPath>MsBuildTasks</MSBuildCommunityTasksPath>

   </PropertyGroup>

   <UsingTask                     AssemblyFile='C:\Personal\MSBuild.Community.Tasks.v1.2.0.306Build\MSBuild.Community.Tasks.dll'  TaskName='MSBuild.Community.Tasks.XmlRead'></UsingTask>

   <Target Name='BuildAll' DependsOnTargets='Compile' />

   <Target Name='Compile'>

   <Message Text='=== COMPILING $(Configuration) configuration ===' />

  <MSBuild Projects='$(SolutionFile)' Properties='Configuration=$(Configuration)' />

  <XmlRead Namespace='http://nant.sf.net/release/0.85/nant.xsd' Prefix='n' Path='/n:project/n:property[@name='chart1Key']/@value' XmlFileName='MyPerformanceData.xml'>

   <Output TaskParameter='Value' PropertyName='Chart1Key' />

   </XmlRead>

   <XmlRead Namespace="http://nant.sf.net/release/0.85/nant.xsd" Prefix="n"  Path="/n:project/n:property[@name='chart2Key']/@value" XmlFileName="MyPerformanceData.xml">

  <Output TaskParameter="Value" PropertyName="chart2Key" />

</XmlRead>

I used XML Read twice to copy the values for two keys. Note that namespace is also same as used in our performance data xml file.

Get Teamcity to report the data as charts

This is the last piece of puzzle - charts can be reported in TeamCity at two levels.

• At the configuration level
• At the project level.

If we want to report charts at the configuration level then we need to modify the main-config.xml file on the TeamCity server. This file is found in ...BuildServerconfig folder. This will start appearing in all the build configurations but will appear only if the build is sending service messages.

Have a look at the main-config.xml I used to report data in the 'settings' tab of every configuration.

main-config.xml

   <?xml version="1.0" encoding="UTF-8"?>

  <server rootURL="http://lon-komals20:85">

  <version number="454" />

 <db-compact>

  <scheduler hour="3" minute="0" />

 </db-compact>

  <auth-type>

 <login-module class="jetbrains.buildServer.serverSide.impl.auth.DefaultLoginModule" />

 <guest-login allowed="true" guest-username="guest" />

 <free-registration allowed="true" />

 </auth-type>

 <graph title="Performance data for-Read order service(in ms)" seriesTitle= "Read Order" >

  <valueType key="chart1Key" title="Read Order"/>

 </graph>

<graph title="Performance data for-Open order service(in ms)" seriesTitle="Open order"     withFilters="true">

 <valueType key="chart2Key" title="Open Order"/>

</graph>

But if you want to report performance data for a specific project, then just modify the plugin-settings.xml in the project folder. For my test project, this file was at

...BuildServerconfigTestProjectforDataCollectionplugin-settings.xml.

Please see below the code snippet for the plugin-settings.xml file. Note that you have to specify build configuration here i.e the build configuration which will send data to this chart. In my case the build type was bt2 and you can find it's reference at the end of the url for the project ‘buildTypeId=bt2’.

plugin-settings.xml

  <?xml version="1.0" encoding="UTF-8"?>

  <settings>

  <custom-graphs>

   <graph title="chart1Key" defaultFilters="" hideFilters="" seriesTitle="Komal test main">

   <valueType key="chart1Key" title="Some Graph Title" buildTypeId="bt2" />

   </graph>

  <graph title="chart2Key" defaultFilters="" hideFilters="" seriesTitle="Komal another main">

    <valueType key="chart2Key" title="Another Graph Title" buildTypeId="bt2" />

  </graph>

 </custom-graphs>

</settings>

Setting up the build configuration

Now it is time to setup the build configuration. This can be setup in two ways depending on how you are collecting your performance data.

** Collecting data from UI tests

It can be setup as a two build step process(if you are using UI automated tests to collect data). In the first step just produce the performance data in the xml and in the second step run the custom build file and notice the data reported in TeamCity. In the build step just call the custom build file which is sending messages and copying data.

Please find below the Result.build file used in the configuration above and also mentioned in previous sections

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

<PropertyGroup>

<WorkingFolder>$(MSBuildProjectDirectory)</WorkingFolder>

<Configuration>Debug</Configuration>

<SolutionFile>OMTests.sln</SolutionFile>

<MSBuildCommunityTasksPath>MsBuildTasks</MSBuildCommunityTasksPath>

 </PropertyGroup>

 <UsingTask                  AssemblyFile="C:\Personal\MSBuild.Community.Tasks.v1.2.0.306Buid\MSBuild.Community.Tasks.dll"  TaskName="MSBuild.Community.Tasks.XmlRead"></UsingTask>

  <Target Name="BuildAll" DependsOnTargets="Compile" />

   <Target Name="Compile">

   <Message Text="=== COMPILING $(Configuration) configuration ===" />

   <MSBuild Projects="$(SolutionFile)" Properties="Configuration=$(Configuration)" />

   <XmlRead Namespace="http://nant.sf.net/release/0.85/nant.xsd" Prefix="n" XPath="/n:project/n:property[@name='chart1Key']/@value" XmlFileName="MyPerformanceData.xml">

   <Output TaskParameter="Value" PropertyName="chart1Key" />

   </XmlRead>

   <XmlRead Namespace="http://nant.sf.net/release/0.85/nant.xsd" Prefix="n" XPath="/n:project/n:property[@name='chart2Key']/@value" XmlFileName="MyPerformanceData.xml">

    <Output TaskParameter="Value" PropertyName="chart2Key" />

     </XmlRead>

     <TeamCityReportStatsValue Key="chart1Key" Value="$(chart1Key)" />

     <TeamCityReportStatsValue Key="chart2Key" Value="$(chart2Key)" />

     <TeamCityReportStatsValue Key="chart2Key" Value="$(chart2Key)" />

      </Target>

      </Project>

Collecting performance data from component tests

If you are collecting your performance data through components tests/through maven(i.e the build agent is Lunux and not windows) then you can create a new build configuration to report data. Please ensure that your component tests are publishing the performance.xml as an artifact to teamcity. Create another configuration to consume that artifact. Please take a look this sample configuration for more details:

Please take a look at the fig. below for more details:

Now add the artifact dependency to this build.

Okay - so now performance data of the projects I am involved with are visible to everyone through TeamCity. Hope you found this approach and useful. Do leave your comments if you have any opinion about this approach. Please share this article on twitter, facebook or google+ if you feel it might be useful for your network. Thanks for your time.