Thorough Testing Ensures Smooth Introduction Of New Services And Improved Systems

We have been working with Orange since February 2003 and, during this time, have helped amongst other things to develop a risk-based testing approach.

In addition, Sopra Group has provided a flexible team of test managers, analysts and engineers that Orange can use to assist with projects, as and when required.  Automated regression testing replaces a labour-intensive manual system to allow faster, more comprehensive and lower cost testing.

With more than 50 different customer management systems, Orange was spending a significant amount of time and money conducting manual system tests.  The testing cycle was taking, on average, five people five days per month to complete.  Orange realised it would be more time and cost-effective to use an automated testing system, enabling them to make better use of resources by freeing up the testing team to work on other business–critical projects.

Orange required an automated regression testing solution that would check business systems on a monthly basis, ensuring any changes made to one part of the system didn't generate bugs or errors elsewhere.  Working onsite with the Orange team, Sopra Group testing consultants' first task was to develop a tailored solution over an 18 month period.

Following a thorough review of the manual testing system, a pilot system was developed to prove the automated regression testing would work.

Once the pilot was successfully completed, Orange agreed to roll-out automated testing across all the systems.  In terms of the time and money that Orange is saving, the results speak for themselves:

• Previously, a system requiring six test scripts to cover its regression tests took up to eight hours to execute.  Using Orange's existing automated test tools, Mercury's Winrunner, reduced the test cycle from eight hours to under one hour.
• It took up to two weeks to create the data required to support the test execution for two systems.  Mercury's Winrunner was again used so the data could be created in overnight runs.

It has also delivered significant tangible benefits:

• Using an automated testing process means improved accuracy.
• The very nature of manual testing with the ever-present risk of human keying errors means it is rarely 100 percent reliable.
• Automation reduces the error rate to zero and ensures that the right tests are run in the right sequence – total consistency every time.
• Automated regression is much quicker, which is vital in today's fast moving business environment.
• Automated tests can be run in conjunction with other testing.

By implementing automated regression testing Orange realised that the less interesting, repetitive testing work is eliminated.  Skilled people are still needed to set the tests up initially and then to interpret the results if a problem arises.

Paul Bevan commented: "As one of the UK's leading mobile phone operators we need to ensure our business systems are 100 percent reliable at all times.  Testing is vital to this process, allowing us to identify any issues within the system as early and quickly as possible.  By testing smarter we will not only save money, but we will be able to optimise our resources.  Because this automates the more tedious aspects of the work, it keeps our people interested and helps us to make optimum use of their talents."

In addition to the automated regression testing, Sopra Group consultants within the test tools team also helped Orange to help set up Test Director to be used more efficiently.  Although Orange staff had previously used Test Director to manage and monitor the testing lifecycle, it was only being used on an ad hoc basis.

Sopra Group's staff played a key part in training staff, so test personnel could maximise use of all the modules within Test Director.  This has helped to increase the Test Director user community by approximately 300 percent in just over a year.  Today, all Orange projects that come through the test management process are managed via Test Director.  This has helped reduce time spent in the defect management cycle, and has introduced better traceability between requirements, test cases and defects.  By customising Test Director to support the Orange project framework, the tool also supports the risk-based testing approach, as it has helped test managers to make objective judgement calls to assess the risk involved in any given project, based on a wealth of project intelligence.