ALM Open-Source Tools – Adding Hudson CI to the Mix

Posted on August 23, 2010

0


Hudson Continuous Integ. Server has been claimed to be better than the Cruise CI server and without a CI server, an Agile development team tool-set is not complete. But, what do we expect from a CI-server or a CI-server integrated Change-Mgmt. tool, apart from the basic build automation that it provides?

The diagram below shows the almost complete Open-Source ALM Tool-Set with Hudson as CI:

First of all, build automation can not be achieved without a decent build script using some build & dependency system such as Ant or Maven.

Second, build automation is just not enough, we need to have test automation with a set of regression test cases for checking the sanity of the build. Also, integration tests for new features need to be added/modified and their execution need to be automated from the build script. From the project manager’s point-of-view the test-related tasks also have to be planned & managed and the CI server or the PM/task tracking/change mgmt. tool (mostly the later) also need to provide visibility into the status of these tasks.  Ultimately, a specific release of the software product is made of features and  from the status available from different repository (mainly task/feature & config. mgmt), the PM should be able to get a quick insight into how many of the features planned for release can meet the date? Meaning, instead of just giving whether the feature is implemented & checked into SCM, one should be able to get granular information such as the following:

  1. How many Regression Test Cases (RTCs)  – critical or important TCs – of old features have passed in the latest build?
  2. If any RTC has failed, whose modifications have broken that old feature & whether he/she has been informed about it, automatically?
  3. Whether any integ. TCs have been written for the new features & how many have passed in the latest build, feature-wise?
  4. Whether any Unit TCs have been written for the new features & how many have passed in the latest build, feature-wise?
  5. What is the code coverage achieved after running these test cases, for each of the features?

Only from these input information, a PM can decide which features can be included for the target release date. I have not included the task estimate or the ‘time aspect’ into these requirements, those will be important for getting at a realistic delivery date for all the planned features.

These requirements can be achieved only if there is traceability from feature to test artifacts (in SCM) and this be achieved in the same way as the traceability from feature to development artifacts. Architecturally, there are three ways of implementing these requirements depending on the vendor:

  1. Make it as a plug-in for CI server which pulls additional traceability information from SVN which is anyway available as commit log. But it needs to pull more information if we have to find answer to question #2 above. Also, PMs are not used to looking at the UIs of CI server I guess.
  2. Make it as plug-in to Change Mgmt server (in this case Trac). This is better but it still need to pull in the build log information.
  3. Make as separate product and called it ‘Project Management Dashboard’ – if the vendor wants to make more money !!!

I evaluated the current integ. available from Trac to Hudson or the Trac plug-in for Hudson – to check if any or both of these are good enough for meeting these requirements. Even though there is some features for JUnit TCs execution status drill-down, Code coverage & Test coverage, I am not sure if we there is anything to correlate with the features. Hudson plug-in in Trac only provides consolidated timeline with build results added to it and the Trac plugin in Hudson only provides ‘skin-deep integration’.

Edgewall Bitten seems to have goals similar to what I want as said in their whitepaper below:

…many projects include build scripts that do a lot more than just compile, link and package the end product: unit tests are run, code coverage by the tests is recorded, conformance to a defined coding style is checked, XML files are validated, and so on. Some of these extra steps can be made to “break” the build (such as test failures or validation errors), while others exist solely to produce information about the code base (such as code coverage analysis). Either way, a lot of interesting data is generated during the build; information that can be important for understanding and managing a software project.

But it does not integrate with Hudson from what I have read. Also it does not work with Trac 0.12 version officially, but can be patched to make it work.

Edited on 14th Sep: While using the HudsonTracPlugin, you will encounter a problem while seeing the timeline of all events including Hudson builds. This is a bug in Hudson as described in HUDSON-7299. If you use any Hudson release including & after 1.375 this should not be there. But showing the Hudson events as part of the timeline of events is the minimal integration one can expect. I will layout what is required for a tighter integration between Trac Wiki – Trac Tickets/Tasks – Hudson builds in my next post.

Advertisements
Posted in: Process