Requirements Testing Supports Extensive Efficiency & Quality Gains At Major Telco Organisation

Sopra Group has been working with this telcommunications company since February 2003.  During this time, as part of a Test Optimisation Programme (TOP), we have helped them, amongst other things, to realise extensive cost savings and quality gains through a requirements testing approach.

In addition, Sopra Group continues to provide a flexible team of test managers, analysts and engineers that the organisation can use to assist with projects, as and when required.

Background

During 2004 – 2005 it became clear to the joint Sopra Group/Telco organisation Testing team that the cost of testing as a whole was impacting the overall cost benefit of each project undertaken.  Many faults were being located within the user acceptance phase, and even worse within the live environment.

This situation was exacerbated by the inclusion of a variety of third party suppliers who took the organisation's initial requirements, built the system in accordance with their understanding of the requirements and then presented the system (sometimes after many months) back to the organisation for acceptance testing.  It became clear that in some cases the supplier's understanding of the requirements was not always the same as that of the organisation.  However in many cases this was not discovered until acceptance testing had begun in earnest.  This led to an increase in change control activities, which in turn had an impact on project end dates.

A costing exercise was undertaken to ascertain the cost of testing at each project phase and the financial impact of locating faults 'downstream'.  It became clear that the earlier a fault could be located within the lifecycle the cheaper it was to fix and re-apply.  The cheapest point being at the time of the initial requirement review.

Approach

Sopra Group reviewed the above key drivers in order to ascertain what was required to drive up the level of business requirement quality to satisfy the organisation's key drivers.  Several sets of requirements were reviewed to ascertain their quality.  Once the initial review was completed the following key points were noted:

• The quality of requirements varied from project to project and in some cases function by function
• Some requirements were ambiguous, some were opinions, some were duplicates and some were incomplete
• Measurable definitions were vague
• Not all requirements were focused on the key benefits – some formed more of a 'wish-list'
• The requirement documents reviewed had not been initially considered from a development or testing viewpoint, that is, the cost of developing and testing the requirements had not been considered.

To combat these challenges Sopra Group defined a standard eight-point check that all requirements should follow.  Note: This has since been expanded to incorporate two further checks.

A pilot study was identified and the eight-point check applied to it.  The result was that every requirement failed at least 50% of the eight-point check.  The requirements were reviewed with the relevant Business Analysts and amended as appropriate.  On reapplying the eight-point check 80% of the requirements passed all eight points.  The remainder were re-reviewed and amended accordingly.

To understand what impact the eight-point check would have on the following test phases the whole process (defined as Requirements Testing) needed to be rolled out across several key projects.

Roll Out & Results

The eight-point check approach is based on common sense and thus is very straight forward to understand.  However when applying the approach across a company this large and diverse, it became clear that it had to be 'sold' to a great number of parties.  These included:

• The organisation's business community involved with creating the Business Requirements
• The Business Analysts
• The organisation's third party developers
• The Acceptance Testers
• Project Management
• Project Support
• Technical and Security support.

Sopra Group embarked on a roadshow to 'spread the gospel' and highlight the potential cost reduction and reduced time to market that could be gleaned from employing requirements testing. This was backed up by a fully documented process and requirement test tools that recorded the reviewing of each set of test requirements.

It was found that the most effective method of reviewing documents was to do so in a collaborative manner i.e. a variety of interested parties review the documents as a team and employ the eight-point check. This enabled the approach to be employed from a variety of parties who would normally be impacted by poor quality requirements further down the project lifecycle.

Once the approach had been employed against a variety of projects it was possible to start measuring the cost and time saved per project.  Sopra Group reviewed similar projects in size, complexity and functionality.

Benefits

• Impressive cost savings and quality gains by finding and fixing defects further upstream
• Accelerated time to market due to reduced rework
• Open communication between specialist technical staff, business heads and IT leaders, clarifying and aligning project objectives
• A more easily identified cost benefit from your IT investment, as programme effectiveness is more easily, and more usefully, measured against requirements which are highly relevant to the business.

Conclusion

At Sopra Group, our consultants will work with both business and technical teams from a programme's inception, to ensure that the business priorities drive the programme requirements rather than the project programme or technical specification.

Our 10 Point Check of the system requirements enables early identification of potential functionality issues, bugs or security vulnerabilities before the application is built.  This also ensures that the requirements are traceable throughout the programme with clear business ownership.