Thursday, August 11, 2011

Managing stakeholders - Test Manager perspective

Test manager will be addressing on day-to-day basis questions like:
  • Why actual QA effort was more than planned?
  • Why are we exceeding from number QA cycles planned in the project plan?
  • Why do we need to do so much testing for few changes done to the code?
  • Why are we still doing unit level tests at system level tests?
  • What can we do to test cases attributes to qualify which helps us to decide on priority, during test execution cycle?
  • How many defects logged are not traced to test cases? What are we doing about this?
  • What risks were anticipated during test plan? How they were mitigated?

I feel these can be addressed better with focus on following:
  • Good test case architecture where all levels of test cases fit and with right set of attributes defined helps test case selection on need basis
  • Quality gates or levels of testing better defined with defect types to be uncovered as focus than use same old Unit, Integration and System testing as levels of tests
  • Introduce escape tracking at each quality gate
  • Define criteria to move testing from one level to another level and improve the definition continuously
  • Introduce better tracking report to look at progress aspects, cleanliness aspects, test coverage aspects
  • User lessons learnt and fix the gaps for future cycles of testing
  • Use defect centered activity breakdown during planning phase


I like to hear any comments or different views from the community



Friday, August 5, 2011

Effective project review

You may be wondering, in spite of we spending so much time on project reviews why some of our projects are failing? The challenge here is, during review which are the aspects you focus and what measurements you have to judge where you stand on each of these aspects. It is recommended apart from progress aspects, watch out for product test coverage and cleanliness aspects.

The challenge here is not only defining right metrics to collect, the place and frequency to collect must also be defined. The phases of measurement journey in the organization are: measurements defined, collection happening in many projects, analysis and feedback done, accuracy improved and we have organization trend or control limits for key metrics. have a look at PPT below

Test automation challenges

Organization may be having the best test automation practice but test assets are inadequate. Such automation investment is of no use. Management will be wondering on so much money invested- on tools, training, hiring skilled engineers and scripting effort but defect escape to production is still bothering them.

Many parents who wanted their children to be successful in their life, sponsor them for whatever certification courses their son/daughter wanted to do after their graduation. Common mistake which I have observed is, parents sending them for some test automation tool training and get the certification. If you see the data points on how many were successful with that path, it is not that encouraging. Why? What went wrong? Automation of wrong tests may not yield anything as business value to organization. They need to understand what are the right test to automate is before starting on test automation. If that skill is not there difficult fro them to be successful in this path.

Another issue is organization management tracking on what percentage of tests automated than how it was done? They get some shocks like the automated scripts not working on new version of product but effort required to maintain scripts not possible to allocate with release pressure and scripts were dumped as useless assets. No frame work used and no standards in scriting maintenance is complicated.

The tool used for automation will not be supported from vendor from that year!. (Phased out tool), but automated scripts is really reducing test effort, helping in doing better regression tests, helping multi-platform tests etc. Oops! what to do? How can we accelerate porting these automated test solution to another tool?

HBT helps to address some of these challenges. Take a look at the following PPT

Monday, July 25, 2011

Effective and cost-efficient validation strategy

The standard practice is to divide product testing into different levels like Unit, Integration, System and UAT. There is always a claim in all the organization that they have planned some of these tests and executed. What is surprising is everything in software terminology is evolving with time but these definition for levels of testing remains same even now! But what you can observe is the confusion on definition of an "unit" within the team. Can we look at differently the definition of level of testing which is acceptable to all with no confusion? How about coming up with defect types and define the quality levels where these defects types must be uncovered. HBT recommends 9 levels. This goal-centric approach to testing will also help us to organize test assets better and helps to demonstrate the adequacy of test assets. Overtime it will also help us to prioritize tests for execution in each cycle of testing, select right type of test for automation and determine the right competency required for execution of certain types of tests. All these focus finally leads to effective and cost-efficient validation strategy acceptable to any stakeholders.

Take a look at how personal test methodology HBT helps you here http://slidesha.re/HBT3of6

Monday, July 18, 2011

Intensely goal focused testing

I am sure some of the challenging questions Test Manager will face when things don't go well is - why did these defects escape our team testing? Do we have sufficient test cases to test our product? Why are we spending so much money on re-work and re-testing?

This is where I strongly recommend to re-look at how did we set the goal for testing team in a project. What did we learn from history? Can we predict upfront potential defect types which impacts customer rating on quality of our product? Is there something we can generate as defect catalog for our domain and check the completeness of test assets we have using this catalog as reference? By finding answers to these question you will land up in defining a very clear goal for testing.

It is important to realize that sharper the goal definition for all levels of testing higher is the return on investment (ROI) made in test engineering.

Take a look at how personal test methodology help you here http://slidesha.re/HBT-Solution-2of6

Friday, July 8, 2011

Understanding customer expectation

One of the toughest job is to clearly understand customer expectation. I feel it is a staged approach but getting the entry into the customer's organization is a major break through. This will happen only when customer realize that they are talking to the organization which has systematic way of extracting all their needs step-by-step. Credibility increases when customer see value added by organization in other engagements. During the journey it is important that delivery team delivers whatever they promise and also one should focus on sharing with customer what problems were observed at each key stage of engagement and how it got eliminated or reduced. That is one of the key factor for engagement health to strengthen over time.

In the validation space, starting from understanding customer products & existing test assets to deliver the clean software to production, we realize there are various existing problems one can focus and solve in the organization. Look for some information shared here as 1 of 6 series which I am planning to share.


I appreciate comments and look forward to learn from community in the process.

Monday, June 20, 2011

Quick assessment of test process maturity

If you believe that increasing effectiveness of test process in the organization impacts positively on top and bottom line of your business, you must find ways and means to know how is your test process behaving at present? and is there a systematic way to improve the test process?

Here I am sharing some symptoms to quickly assess the test process maturity and if you see consensus among decision makers at your organization to focus on process improvement, I recommend you to look at TMMi framework. The steps involved according to me are:
  1. Understand TMMi framework
  2. Assess where the current process maturity is
  3. Identify gaps to reach a specific level of test process maturity
  4. Find appropriate way to fix those gaps effectively
An improved process along with a good test methodology like HBT (coming from STAG) you will see accelerated path to achieve your goal.

Thursday, June 16, 2011

Solving business problem with Test Process Improvement

What kind of problem if you have should take a look at improving existing test process?

I suggest you look at any of these problem symptom given below, if 50% 0f the core management team feels "yes" to two or more potential problems below, then I strongly recommend to take Test Process improvement initiative as management sponsored project.
  1. Releases are delayed significantly due to quality issues
  2. Test cases we have seems to be not adequate resulting in bad releases
  3. Major defects are encountered only during system testing resulting in more QA cycles than planned
  4. Though organization encourage multi-leveled test focus but planning process and competency of team not helping to achieve this effectively
  5. Product is evolving more regression cycles than planned and unable to cope up with commitment to major releases
  6. Risks are not analyzed correctly after test results are out resulting in wrong decisions
  7. Product seems to be stable, more customization and releases happening resulting in lot of effort going into test execution phase
  8. All aspects of quality not assessed like Load, Stress, Performance, and Scalability (LSPS) thereby Marketing/Sales unable to position product correctly in the market place
  9. UAT by customer not happening as per plan resulting in delayed revenue realization against plan
  10. Unable to get right set of people for projects when needed

I also recommend to baseline the above problem quantitatively like in what percentage of projects these problem exists and see the improvement with same measurement after you have modified the processes and used by team for 3-6 months.

Tuesday, June 14, 2011

Test Process Improvement

How to decide whether we need to invest our time and effort on improving our test process? How much to invest? What ROI (Return on investment) we can anticipate? Is there a systematic way to improve test process maturity? Where do we stand as on today?

Let me take first question and see whether I can answer all other questions

How to decide whether test process improvement required?
First main factor is degree of customer satisfaction. Each organization will have their own way to measure this. Just for understanding purpose, let us assume we measure this in a scale of 1 to 5 (5 being very happy customer). If we have randomly distributed number of customers in all these five levels may be worth looking at what does it take to achieve 3 and more. Many times answer will be look at process, people, technology and tools used to do the required activities. This is where I feel process improvement focus will help because process can be improved only when you look at all the other three factors above.

First thing in process improvement is to set a goal (where do you want to reach) and assess where do we stand as on today. I recommend to use TMMi framework to assess what you have as practice and how it meets stated practice for all maturity levels. This framework will guide you to set goals and improve the process step by step.

In all processes we do know that people play major role. Technology and tools may help in consistency and speed. How to make sure all in the organization irrespective of their previous experience follow standard discipline for doing specific jobs in a test life cycle activities so that we ensure consistency in the output at each stage of conversion of input to output. This is the place where I encourage people to look test methodology HBT coming from STAG.

Yes, an effective test process powered by HBT ( http://www.slideshare.net/stagsoft/an-introduction-to-hypothesis-based-testing ) had shown significant cost reduction from our experience. HBT helps to accelerate process improvement initiative in the organization. http://www.stagsoftware.com/blog/?p=324

Friday, April 29, 2011

Testing team organization

Introduction

Many organizations are experimenting with different operational models to sustain a strong testing team with various business challenges they are facing. I am sharing my observations made in entire journey as software professional. Each model attempted had its own issues and challenges. It is important that organization attempting any of this model to aware of these potential issues and take necessary steps to counter them.

http://www.slideshare.net/mcnagaraj/testing-team-7816329


Effective test process and long sustenance of test engineers were observed more in the case of “Independent testing team under Test Manager” model. Test engineers started seeing career path for their growths. Test process improvement initiatives focused at organization level than project level.