Move from too many metrics to few right metrics

Many of us wonder when we get non-conformance report (NCR) during process audits regarding not following the documented process. NCR like "Collection and analysis of metrics is not as per documented process as in QMS" many times common in internal or external audit. Where are we going wrong? Is it wrong metrics proposed to be collected? People are not clear why these data required? or all metrics collected and dumped in some manager's hard disk and no analysis? or metrics collected are inaccurate and looked like collected for the sake of compliance to stated process?

If we do this as survey, you may get equal response to all the above questions when you count number of "yes". How to improve this situation so that people see metrics program in the organization is effective? From my experience metrics are collected to know about where we are with respect to a specific goal set for an attribute? Where we would like to be is the expected improvement goal? What we need to do is the outcome of analysis of metrics trend observed across teams?

Let me give an example why metrics program not implemented correctly which may confuse everyone

Organization has one goal which states "We wanted to grow our business every quarter for all business units". The brainstorming was done and they arrived at let us measure following metrics to see how we perform this year against this goal. The list of measurements selected finally were:

1. No. of new customers added  every quarter

2. Revenue % growth quarter by quarter

CEO was reviewing the reports from two divisional heads of the organization after three quarters, the report summary high-lighted looks like this.

Div 1: 

We added 2 new customers every quarter in the last three quarters

The revenue % growth from quarter to quarter is maintained at 10%

Div 2:

We were able to add 2 new customers in Q1 only but no new customers were added in next 2 quarters

The revenue % growth from quarter to quarter was 15% with existing customers giving more additional business

Which division is doing well?

I know you need some more accurate data to answer this. The key point here is the right metrics are required to measure progress towards team goal and metrics are not the right candidate to reward someone or some team. Normally such metrics program to be used for measuring improvement from baseline value. Division one added customers who may give more revenue in coming quarters but revenue growth % was 10% quarter to quarter. Division 2 though gave 15% growth in revenue quarter to quarter but not adding customers in Q2 and Q3 may not go well with CEO. Also 10% and 15% are not apple to apple comparison as baseline revenues of a division may be different.

May be a goal like min 5 customers/per qtr and revenue growth of 15-25% quarter to quarter as organization goal for this year will really bring meaning to the metrics collection program.

More than someone spending time to collect these metrics the tool they use must help in computing them. Also many times we feel the questions which come during management review had no answers at that point. A good dashboard and reports in tool also is a great advantage covering metrics planned. Sometime helps to derive on te spot answers to management questions. 

Appreciate your views and comments.

How to narrow down focus areas for test process improvement?

Once organization is convinced there is scope for them to increase effectiveness and efficiency of testing, then the next challenge is to find where to start the improvement. In my view one should look at THREE different perspective to see opportunity for improvement:

1. Process (the way testing is done)

2. People (who do testing at different levels)

3. Technology and tools (efficiency and accuracy agents used during testing activities) 

Process perspective:

It is important to look at what we have agreed as process of starting and ending testing in the organization. What are the standard deliverable to demonstrate the process defined is followed. The business scenario always demand light process so that time to accomplish the task is exactly what one can budget in the given scenario. How to know which part of the process needs a new way of doing ? There are SIX attributes I recommend to measure where the process needs fine-tuning.

Effectiveness - Here goal clarity for doing any assigned work is the key. If what is expected at the end of any assigned task is very clear there should be very less rework
Consistent - Common terminologies, common templates, repeatable across projects, reuse focus good
Efficient - Ability to do as per budgeted time/effort, do with less cost, start taking less time than previous cycle
Salable - Adaptable to any growing demand, increased people, more variety of projects
Visible - Clear mirror to see how we look at any stage from all perspective of project end-date, quality, cost
Agile - Achieved all goals for short cycles always

People perspective:

Here again it is important to set expectation for roles as clearly as possible. The competency is more looked as skills than how many times or years he/she had done that task. Here I strongly recommend methodology based testing. Look at HBT (Hypothesis Based Testing by STAG) personal test methodology so that whoever does testing activity are guided by well defined goals at any levels of  testing. 

http://slidesha.re/HBT-Solution-2of6

Technology and Tools perspective: 

Here the technology and tools must be looked as agent for efficiency. Any work which can be done with accuracy, with less time and less effort through tools and technology must be inducted. The common tools we see are defect tracking tool, functional test automation tools, load testing tools. But we have landscape of other tools which may accelerate unit testing, test data creation, reporting and management tools. How are we penetrating these tools keeping ROI (Return On Investment) in focus is the challenge.

Software Quality Management via Test Process Improvement

Ever since the software engineering discipline was born, there is lot of investment to address software quality related issues. The software quality is becoming a continuously challenging as complexity of software and the software solution users expectations are going up significantly. People looked at solution for software quality issue by addressing the software development process looking at standard for process improvements like ISO 9001 and CMMI. The TMMi framework has been developed by the TMMi Foundation as a guideline and reference framework for test process improvement and is positioned as a complimentary model to the CMMI version 1.2 addressing those issues important to test engineering community.

From my experience many projects will budget 30-40% of total cost of project to testing but we always land up in spending more in many cases due to late stage detection of defects or insufficient testing leading to defect escape to field and so on. It is important to look at controlling this software testing budget overflow by focusing on "Effectiveness and Efficiency" factor for all testing efforts in the entire software development life cycle.

I feel by looking at TMMi framework one can find the gaps in people competency to perform a specific role, process over heads in end-to-end software development life cycle, and gaps in usage of right set of tools  which helps to increase efficiency, accuracy and sufficiency factor of testing in this business driven challenging situations. Thought will share the practical application of process driven approach to increase effectiveness and efficiency of testing using TMMi in series of postings. 

Is there a crude way to know where we are currently in test process maturity in your organization? Take a look at these slides.  

http://slidesha. re/MCN-Quick-TP-Assess

TMMi is a registered trademark of TMMi Foundation.
TMMi Foundations makes no warranties of any kind who prefer to use this framework for test process improvement

Appreciate readers views and comments.

ROI (return on investment) for investing on Test Process Improvement

I am sure every organization will have project analysis after a major release and followed by a formation of a  task team to implement some set of actions resulted out of project analysis meeting. Some of them which will always appear in the list are
1. Improve unit testing and define some criteria for integration and system testing team to accept build for their testing
2. Automate many of our tests which are used at various stages repeatedly (nightly build test, regression test)
3. Regression test automation % should progressively grow release after release
4. There should be clear production release criteria and risk assessment before release
5. Improve estimation
and so on

Many will finally agree that test process improvement seems to cover majority of the list  in the action items coming after project review.

Let me also share from my experience who will get the advantage investing in test process improvement. You need to answer "YES" or "NO" to following questions. It may look common sense but we ignore these patterns.
1. Are you able to make your last release with exact number of planned test cycles?
2. Can you assure that majority of the tests done by development team is not repeated again in next level of testing?
3. Number of patch releases made after final production release is under control therefore support team and customers are not cribbing?
4. You have a well defined maintenance team which work on customer issues and enhancements and none of your product next version road map plan are impacted due to maintenance release pressure?
5. When you analyse the defect found after release majority of them were not traced to "missing test case" or "incomplete regression test" due to lack of time?

If your answer is "No" to any two of the questions above I feel you must look at formal test process improvement in your organization. Some TIPs on how to go about it.
http://slidesha.re/MCN-Quick-TP-Assess

Like to hear from our software engineering community on their views.

Challenges for colleges to enhance job opportunities for their students

The colleges will have placement cell to ensure their students get into a best job after they complete the degree. They look forward to invite prestigious organizations to come for campus interviews. Organizations apart from campus interviews have tried various other models also to hire fresh talents and I could see some challenges they are facing with speed of acquiring business not going as per plan. There are different perspectives to look at the challenges all players are facing in industry - student’s perspective, college's perspective and organization's perspective. In this blog I am talking about college's perspective challenges.

The college concerned people will be thinking, which organization we should give chance first for campus interview when they approach? The one which was prompt in making reasonable offer and honoring the same last year or one which gave very good offer but failed in adhering to the commitments as in the offer letter last year? Or should we tie-up with some organization who will partner with us to ensure our student’s employment index goes up year on year? Getting prestigious organizations as preferred by students for campus interview also is another BIG challenge. What should college showcase to attract the prestigious organization to visit them for campus interview?

The colleges may look for solutions where students are taken closer to “deployment ready” kind of talent for software projects so that organizations will look at them differently when they interview during campus recruitment. As there is a constant communication from some top industry leaders from India that the academy and industry is not working together and there are gaps in skills when the engineers come out of graduation. The investment required on training engineers from the college taking time and effort for organization, delaying the business acquisition and hence the growth. Therefore college is looking for platform like Sandbox environment where students will learn in parallel along with academics, various skills required by the software industry and apply them on well designed exercises to sharpen their skills and come out confidently to face interviews.

Love to hear different views or comments from the college management community. 

Challenges for students to start their career in software engineering

The students will look forward to the college placement office to organize campus interviews from prestigious organization. Apart from this they may look at options to build skills and competencies which may increase their probability of getting the right job while they are in final year or after completing their course. There are different perspectives to look at the challenges all players are facing in industry - student’s perspective, college's perspective and organization's perspective. In this blog I am talking about student’s perspective.

Not knowing exactly what one should take up as their career will attend the first campus interview and do their best to get selected, with an acceptable offer in hand but no date of reporting mentioned in that! College placement center as a policy will not allow this student to participate in another campus interview organized later. Student will get confused if someone who is less performed academically compared to him/her gets an offer with date of joining also mentioned. Added to this challenge, they also get a letter sometimes from the organization where they have offer letter in hand, that the recruitment plan and training moved to next quarter due to business reasons. Students are really in confused status at that stage. The parents of such students are more worried about what to do when their qualified son/daughter is idle with offer letter in hand but not sure when he/she will start the career.

Parents may look for opportunity to sponsor for some other skill building training so that their probability of getting job increases. Potential skills to enter into software engineering are: Basics knowledge of software development life cycle (SDLC), Process models (like ISO 9001-2008, CMMi, TMMi), Test engineering skills. This will help them to understand the role of software testing in building quality software. From this stage either they can take career in test engineering path or software development path (after learning their choice of programming languages and/or various other software development tools).

Love to hear different views or comments from the student or impacted parents community. 

Challenges for organization to scale their team

I observed a pattern where many organizations will prefer to hire fresh engineers from prestigious college and invest reasonable time and effort in training them on various skills before start realizing contribution to the business from those new hires. Organizations have tried various other models also to hire fresh talents and I could see some challenges they are facing with speed of acquiring business not going as per plan. There are different perspectives to look at the challenges all players are facing in industry - student’s perspective, college perspective and organization perspective. In this blog I am talking about organization perspective challenges.

HR/Talent acquiring division gets requirements of new talents from different business units keeping in mind business plan under discussion. The budget of new talent acquisition is carefully used, HR team visits all top colleges, evaluate students as per organization process and make offer to the best candidates keeping the joining date approximate (subject to change) as requested by business units hiring them. Business situation force HR to move the date further and they do their best to get connected to the potential new hire but they see a risk of losing some of the right talent when they get clarity to bring them on board. Even if everything goes well they see some best students drop out giving admissions they got for higher studies as reasons, or not happy with role which is not their passion (forced test engineer role or forced development engineer role) or region/place of work not as expected by them etc. Oops! All investment to acquire talent and train them is going waste.

The organization must be wondering, how nice if they have a solution where they can get a potential talent pool from different sources just in time (JIT) to start contributing from day 1 of joining to the project. 

Assuming few steps normally happens when you hire fresh talent from campus – talent acquisition team effort, travel and stay, training investment on new hires for minimum three months on various skills from internal academy or external vendor, may cost totally per student approximately rupees 2.5 to 3 lakhs. Assuming a minimum salary plus over heads for three months, minimum training charges investment per day, total hiring cost spent by talent acquisition team divided by number of students hired we may find Rs. 3 lakhs as investment per student before they start contributing in project assigned.. This may go up if you are providing accommodation and food also during training period. From my experience there will be few exits due to job allocation which is not in their line of interest. That is another loss on investment made so far. 

JIT hiring opportunity of engineers well trained and groomed by a professional organization who understands the needs of industry, easily will save Rs. 3 lakhs minimum per student by going through this path. Assuming organization hiring in bulk like 100 in a batch every year will save rupees 300 lakhs!. The software professionals hired in this model are productive from first week after you train them on organization process and domain on which they are going to work.

Love to hear different views or comments from the software community.