How to narrow down focus areas for test process improvement?

Once organization is convinced there is scope for them to increase effectiveness and efficiency of testing, then the next challenge is to find where to start the improvement. In my view one should look at THREE different perspective to see opportunity for improvement:

1. Process (the way testing is done)

2. People (who do testing at different levels)

3. Technology and tools (efficiency and accuracy agents used during testing activities) 

Process perspective:

It is important to look at what we have agreed as process of starting and ending testing in the organization. What are the standard deliverable to demonstrate the process defined is followed. The business scenario always demand light process so that time to accomplish the task is exactly what one can budget in the given scenario. How to know which part of the process needs a new way of doing ? There are SIX attributes I recommend to measure where the process needs fine-tuning.

Effectiveness - Here goal clarity for doing any assigned work is the key. If what is expected at the end of any assigned task is very clear there should be very less rework
Consistent - Common terminologies, common templates, repeatable across projects, reuse focus good
Efficient - Ability to do as per budgeted time/effort, do with less cost, start taking less time than previous cycle
Salable - Adaptable to any growing demand, increased people, more variety of projects
Visible - Clear mirror to see how we look at any stage from all perspective of project end-date, quality, cost
Agile - Achieved all goals for short cycles always

People perspective:

Here again it is important to set expectation for roles as clearly as possible. The competency is more looked as skills than how many times or years he/she had done that task. Here I strongly recommend methodology based testing. Look at HBT (Hypothesis Based Testing by STAG) personal test methodology so that whoever does testing activity are guided by well defined goals at any levels of  testing. 

http://slidesha.re/HBT-Solution-2of6

Technology and Tools perspective: 

Here the technology and tools must be looked as agent for efficiency. Any work which can be done with accuracy, with less time and less effort through tools and technology must be inducted. The common tools we see are defect tracking tool, functional test automation tools, load testing tools. But we have landscape of other tools which may accelerate unit testing, test data creation, reporting and management tools. How are we penetrating these tools keeping ROI (Return On Investment) in focus is the challenge.

Software Quality Management via Test Process Improvement

Ever since the software engineering discipline was born, there is lot of investment to address software quality related issues. The software quality is becoming a continuously challenging as complexity of software and the software solution users expectations are going up significantly. People looked at solution for software quality issue by addressing the software development process looking at standard for process improvements like ISO 9001 and CMMI. The TMMi framework has been developed by the TMMi Foundation as a guideline and reference framework for test process improvement and is positioned as a complimentary model to the CMMI version 1.2 addressing those issues important to test engineering community.

From my experience many projects will budget 30-40% of total cost of project to testing but we always land up in spending more in many cases due to late stage detection of defects or insufficient testing leading to defect escape to field and so on. It is important to look at controlling this software testing budget overflow by focusing on "Effectiveness and Efficiency" factor for all testing efforts in the entire software development life cycle.

I feel by looking at TMMi framework one can find the gaps in people competency to perform a specific role, process over heads in end-to-end software development life cycle, and gaps in usage of right set of tools  which helps to increase efficiency, accuracy and sufficiency factor of testing in this business driven challenging situations. Thought will share the practical application of process driven approach to increase effectiveness and efficiency of testing using TMMi in series of postings. 

Is there a crude way to know where we are currently in test process maturity in your organization? Take a look at these slides.  

http://slidesha. re/MCN-Quick-TP-Assess

TMMi is a registered trademark of TMMi Foundation.
TMMi Foundations makes no warranties of any kind who prefer to use this framework for test process improvement

Appreciate readers views and comments.

ROI (return on investment) for investing on Test Process Improvement

I am sure every organization will have project analysis after a major release and followed by a formation of a  task team to implement some set of actions resulted out of project analysis meeting. Some of them which will always appear in the list are
1. Improve unit testing and define some criteria for integration and system testing team to accept build for their testing
2. Automate many of our tests which are used at various stages repeatedly (nightly build test, regression test)
3. Regression test automation % should progressively grow release after release
4. There should be clear production release criteria and risk assessment before release
5. Improve estimation
and so on

Many will finally agree that test process improvement seems to cover majority of the list  in the action items coming after project review.

Let me also share from my experience who will get the advantage investing in test process improvement. You need to answer "YES" or "NO" to following questions. It may look common sense but we ignore these patterns.
1. Are you able to make your last release with exact number of planned test cycles?
2. Can you assure that majority of the tests done by development team is not repeated again in next level of testing?
3. Number of patch releases made after final production release is under control therefore support team and customers are not cribbing?
4. You have a well defined maintenance team which work on customer issues and enhancements and none of your product next version road map plan are impacted due to maintenance release pressure?
5. When you analyse the defect found after release majority of them were not traced to "missing test case" or "incomplete regression test" due to lack of time?

If your answer is "No" to any two of the questions above I feel you must look at formal test process improvement in your organization. Some TIPs on how to go about it.
http://slidesha.re/MCN-Quick-TP-Assess

Like to hear from our software engineering community on their views.

Managing stakeholders - Test Manager perspective

Test manager will be addressing on day-to-day basis questions like:

• Why actual QA effort was more than planned?
• Why are we exceeding from number QA cycles planned in the project plan?
• Why do we need to do so much testing for few changes done to the code?
• Why are we still doing unit level tests at system level tests?
• What can we do to test cases attributes to qualify which helps us to decide on priority, during test execution cycle?
• How many defects logged are not traced to test cases? What are we doing about this?
• What risks were anticipated during test plan? How they were mitigated?

I feel these can be addressed better with focus on following:

• Good test case architecture where all levels of test cases fit and with right set of attributes defined helps test case selection on need basis
• Quality gates or levels of testing better defined with defect types to be uncovered as focus than use same old Unit, Integration and System testing as levels of tests
• Introduce escape tracking at each quality gate
• Define criteria to move testing from one level to another level and improve the definition continuously
• Introduce better tracking report to look at progress aspects, cleanliness aspects, test coverage aspects
• User lessons learnt and fix the gaps for future cycles of testing
• Use defect centered activity breakdown during planning phase

Take a look at http://slidesha.re/HBT6of6

I like to hear any comments or different views from the community

Effective project review

You may be wondering, in spite of we spending so much time on project reviews why some of our projects are failing? The challenge here is, during review which are the aspects you focus and what measurements you have to judge where you stand on each of these aspects. It is recommended apart from progress aspects, watch out for product test coverage and cleanliness aspects.

The challenge here is not only defining right metrics to collect, the place and frequency to collect must also be defined. The phases of measurement journey in the organization are: measurements defined, collection happening in many projects, analysis and feedback done, accuracy improved and we have organization trend or control limits for key metrics. have a look at PPT below

http://slidesha.re/HBT4of6

Test automation challenges

Organization may be having the best test automation practice but test assets are inadequate. Such automation investment is of no use. Management will be wondering on so much money invested- on tools, training, hiring skilled engineers and scripting effort but defect escape to production is still bothering them.

Many parents who wanted their children to be successful in their life, sponsor them for whatever certification courses their son/daughter wanted to do after their graduation. Common mistake which I have observed is, parents sending them for some test automation tool training and get the certification. If you see the data points on how many were successful with that path, it is not that encouraging. Why? What went wrong? Automation of wrong tests may not yield anything as business value to organization. They need to understand what are the right test to automate is before starting on test automation. If that skill is not there difficult fro them to be successful in this path.

Another issue is organization management tracking on what percentage of tests automated than how it was done? They get some shocks like the automated scripts not working on new version of product but effort required to maintain scripts not possible to allocate with release pressure and scripts were dumped as useless assets. No frame work used and no standards in scriting maintenance is complicated.

The tool used for automation will not be supported from vendor from that year!. (Phased out tool), but automated scripts is really reducing test effort, helping in doing better regression tests, helping multi-platform tests etc. Oops! what to do? How can we accelerate porting these automated test solution to another tool?

HBT helps to address some of these challenges. Take a look at the following PPT

http://slidesha.re/HBT5of6

Effective and cost-efficient validation strategy

The standard practice is to divide product testing into different levels like Unit, Integration, System and UAT. There is always a claim in all the organization that they have planned some of these tests and executed. What is surprising is everything in software terminology is evolving with time but these definition for levels of testing remains same even now! But what you can observe is the confusion on definition of an "unit" within the team. Can we look at differently the definition of level of testing which is acceptable to all with no confusion? How about coming up with defect types and define the quality levels where these defects types must be uncovered. HBT recommends 9 levels. This goal-centric approach to testing will also help us to organize test assets better and helps to demonstrate the adequacy of test assets. Overtime it will also help us to prioritize tests for execution in each cycle of testing, select right type of test for automation and determine the right competency required for execution of certain types of tests. All these focus finally leads to effective and cost-efficient validation strategy acceptable to any stakeholders.

Take a look at how personal test methodology HBT helps you here http://slidesha.re/HBT3of6

Intensely goal focused testing

I am sure some of the challenging questions Test Manager will face when things don't go well is - why did these defects escape our team testing? Do we have sufficient test cases to test our product? Why are we spending so much money on re-work and re-testing?

This is where I strongly recommend to re-look at how did we set the goal for testing team in a project. What did we learn from history? Can we predict upfront potential defect types which impacts customer rating on quality of our product? Is there something we can generate as defect catalog for our domain and check the completeness of test assets we have using this catalog as reference? By finding answers to these question you will land up in defining a very clear goal for testing.

It is important to realize that sharper the goal definition for all levels of testing higher is the return on investment (ROI) made in test engineering.

Take a look at how personal test methodology help you here http://slidesha.re/HBT-Solution-2of6

Understanding customer expectation

One of the toughest job is to clearly understand customer expectation. I feel it is a staged approach but getting the entry into the customer's organization is a major break through. This will happen only when customer realize that they are talking to the organization which has systematic way of extracting all their needs step-by-step. Credibility increases when customer see value added by organization in other engagements. During the journey it is important that delivery team delivers whatever they promise and also one should focus on sharing with customer what problems were observed at each key stage of engagement and how it got eliminated or reduced. That is one of the key factor for engagement health to strengthen over time.

In the validation space, starting from understanding customer products & existing test assets to deliver the clean software to production, we realize there are various existing problems one can focus and solve in the organization. Look for some information shared here as 1 of 6 series which I am planning to share.

hbt-solutions10f6

I appreciate comments and look forward to learn from community in the process.

Quick assessment of test process maturity

If you believe that increasing effectiveness of test process in the organization impacts positively on top and bottom line of your business, you must find ways and means to know how is your test process behaving at present? and is there a systematic way to improve the test process?

Here I am sharing some symptoms to quickly assess the test process maturity and if you see consensus among decision makers at your organization to focus on process improvement, I recommend you to look at TMMi framework. The steps involved according to me are:

1. Understand TMMi framework
2. Assess where the current process maturity is
3. Identify gaps to reach a specific level of test process maturity
4. Find appropriate way to fix those gaps effectively

An improved process along with a good test methodology like HBT (coming from STAG) you will see accelerated path to achieve your goal.

http://www.slideshare.net/stagsoft/an-introduction-to-hypothesis-based-testing

http://www.slideshare.net/stagsoft/hbt-a-revolutionary-approach-to-testing-software

http://slidesha.re/MCN-Quick-TP-Assess

Solving business problem with Test Process Improvement

What kind of problem if you have should take a look at improving existing test process?

I suggest you look at any of these problem symptom given below, if 50% 0f the core management team feels "yes" to two or more potential problems below, then I strongly recommend to take Test Process improvement initiative as management sponsored project.

1. Releases are delayed significantly due to quality issues
2. Test cases we have seems to be not adequate resulting in bad releases
3. Major defects are encountered only during system testing resulting in more QA cycles than planned
4. Though organization encourage multi-leveled test focus but planning process and competency of team not helping to achieve this effectively
5. Product is evolving more regression cycles than planned and unable to cope up with commitment to major releases
6. Risks are not analyzed correctly after test results are out resulting in wrong decisions
7. Product seems to be stable, more customization and releases happening resulting in lot of effort going into test execution phase
8. All aspects of quality not assessed like Load, Stress, Performance, and Scalability (LSPS) thereby Marketing/Sales unable to position product correctly in the market place
9. UAT by customer not happening as per plan resulting in delayed revenue realization against plan
10. Unable to get right set of people for projects when needed

I also recommend to baseline the above problem quantitatively like in what percentage of projects these problem exists and see the improvement with same measurement after you have modified the processes and used by team for 3-6 months.

Test Process Improvement

How to decide whether we need to invest our time and effort on improving our test process? How much to invest? What ROI (Return on investment) we can anticipate? Is there a systematic way to improve test process maturity? Where do we stand as on today?

Let me take first question and see whether I can answer all other questions

How to decide whether test process improvement required?

First main factor is degree of customer satisfaction. Each organization will have their own way to measure this. Just for understanding purpose, let us assume we measure this in a scale of 1 to 5 (5 being very happy customer). If we have randomly distributed number of customers in all these five levels may be worth looking at what does it take to achieve 3 and more. Many times answer will be look at process, people, technology and tools used to do the required activities. This is where I feel process improvement focus will help because process can be improved only when you look at all the other three factors above.

First thing in process improvement is to set a goal (where do you want to reach) and assess where do we stand as on today. I recommend to use TMMi framework to assess what you have as practice and how it meets stated practice for all maturity levels. This framework will guide you to set goals and improve the process step by step.

In all processes we do know that people play major role. Technology and tools may help in consistency and speed. How to make sure all in the organization irrespective of their previous experience follow standard discipline for doing specific jobs in a test life cycle activities so that we ensure consistency in the output at each stage of conversion of input to output. This is the place where I encourage people to look test methodology HBT coming from STAG.

Yes, an effective test process powered by HBT ( http://www.slideshare.net/stagsoft/an-introduction-to-hypothesis-based-testing ) had shown significant cost reduction from our experience. HBT helps to accelerate process improvement initiative in the organization. http://www.stagsoftware.com/blog/?p=324

Testing team organization

Introduction

Many organizations are experimenting with different operational models to sustain a strong testing team with various business challenges they are facing. I am sharing my observations made in entire journey as software professional. Each model attempted had its own issues and challenges. It is important that organization attempting any of this model to aware of these potential issues and take necessary steps to counter them.

http://www.slideshare.net/mcnagaraj/testing-team-7816329

Effective test process and long sustenance of test engineers were observed more in the case of “Independent testing team under Test Manager” model. Test engineers started seeing career path for their growths. Test process improvement initiatives focused at organization level than project level.