Quality Cost Analysis: Benefits and Risks

 Copyright © Cem Kaner

January, 1996

Joseph Juran, one of the world’s leading quality theorists, has been advocating the analysis of quality-related costs since 1951, when he published the first edition of his Quality Control Handbook. Feigenbaum made it one of the core ideas underlying the Total Quality Management movement.[2] It is a tremendously powerful tool for product quality, including software quality.

What is Quality Cost Analysis?

Quality costs are the costs associated with preventing, finding, and correcting defective work. These costs are huge, running at 20% - 40% of sales. [3] Many of these costs can be significantly reduced or completely avoided. One of the key functions of a Quality Engineer is the reduction of the total cost of quality associated with a product.

Here are six useful definitions, as applied to software products. Figure 1 gives examples of the types of cost. Most of Figure 1’s examples are (hopefully) self-explanatory, but I’ll provide some additional notes on a few of the costs:[4]

• Prevention Costs: Costs of activities that are specifically designed to prevent poor quality. Examples of "poor quality" include coding errors, design errors, mistakes in the user manuals, as well as badly documented or unmaintainably complex code.

• Appraisal Costs: Costs of activities designed to find quality problems, such as code inspections and any type of testing.

• Failure Costs: Costs that result from poor quality, such as the cost of fixing bugs and the cost of dealing with customer complaints.

• Internal Failure Costs: Failure costs that arise before your company supplies its product to the customer. Along with costs of finding and fixing bugs are many internal failure costs borne by groups outside of Product Development. If a bug blocks someone in your company from doing her job, the costs of the wasted time, the missed milestones, and the overtime to get back onto schedule are all internal failure costs.

• External Failure Costs: Failure costs that arise after your company supplies the product to the customer, such as customer service costs, or the cost of patching a released product and distributing the patch.

• Total Cost of Quality: The sum of costs: Prevention + Appraisal + Internal Failure + External Failure.

Figure 1. Examples of Quality Costs Associated with Software Products.

What Makes this Approach Powerful?

Over the long term, a project (or corporate) cost accounting system that tracks quality-related costs can be a fundamentally important management tool. This is the path that Juran and Feigenbaum will lead you down, and they and their followers have frequently and eloquently explained the path, the system, and the goal.

I generally work with young, consumer-oriented software companies who don’t see TQM programs as their top priority, and therefore my approach is more tactical. There is significant benefit in using the language and insights of quality cost analysis, on a project/product by project/product basis, even in a company that has no interest in Total Quality Management or other formal quality management models.[12]

Here’s an example. Suppose that some feature has been designed in a way that you believe will be awkward and annoying for the customer. You raise the issue and the project manager rejects your report as subjective. It’s "not a bug." Where do you go if you don’t want to drop this issue? One approach is to keep taking it to higher-level managers within product development (or within the company as a whole). But without additional arguments, you’ll often keep losing, without making any friends in the process.

Suppose that you change your emphasis instead. Rather than saying that, in your opinion, customers won’t be happy, collect some other data:[13]

 

• Ask the writers: Is this design odd enough that it is causing extra effort to document? Would a simpler design reduce writing time and the number of pages in the manual?

 

• Ask the training staff: Are they going to have to spend extra time in class, and to write more supplementary materials because of this design?

 

• Ask Technical Support and Customer Service: Will this design increase support costs? Will it take longer to train support staff? Will there be more calls for explanations or help? More complaints? Have customers asked for refunds in previous versions of the product because of features designed like this one?

 

• Check for related problems: Is this design having other effects on the reliability of the program? Has it caused other bugs? (Look in the database.) Made the code harder to change? (Ask the programmers.)

 

• Ask the sales staff: If you think that this feature is very visible, and visibly wrong, ask whether it will interfere with sales demonstrations, or add to customer resistance.

 

• What about magazine reviews? Is this problem likely to be visible enough to be complained about by reviewers? If you think so, check your impression with someone in Marketing or PR.

You won’t get cost estimates from everyone, but you might be able to get ballpark estimates from most, along with one or two carefully considered estimates. This is enough to give you a range to present at the next project meeting, or in a follow-up to your original bug report. Notice the difference in your posture:

• You’re no longer presenting your opinion that the feature is a problem. You’re presenting information collected from several parts of the company that demonstrates that this feature’s design is a problem.

• You’re no longer arguing that the feature should be changed just to improve the quality. No one else in the room can posture and say that you’re being "idealistic" whereas a more pragmatic, real-world businessperson wouldn’t worry about problems like this one. Instead, you’re the one making the hard-nosed business argument, "This design is going to cost us $X in failure costs. How much will it cost to fix it?"

• Your estimates are based on information from other stakeholders in this project. If you’ve fairly represented their views, you’ll get support from them, at least to the extent of them saying that you are honestly representing the data you’ve collected.

Along with arguing about individual bugs, or groups of bugs, this approach opens up opportunities for you (and other non-testers who come to realize the power of your approach) to make business cases on several other types of issues. For example:

• The question of who should do unit testing (the programmers, the testers, or no one) can be phrased and studied as a cost-of-quality issue. The programmers might be more efficient than testers who don’t know the code, but the testers might be less expensive per hour than the programmers, and easier to recruit and train, and safer (unlike newly added programmers, new testers can’t write new bugs into the code) to add late in the project.

• The depth of the user manual’s index is a cost-of-quality issue. An excellent index might cost 35 indexer-minutes per page of the manual (so a 200 page book would take over three person-weeks to index). Trade this cost against the reduction in support calls because people can find answers to their questions in the manual.

• The best investment to achieve better quality might be additional training and staffing of the programming group (prevent the bugs rather than find and fix them).

• You (in combination with the Documentation, Marketing, or Customer Service group) might demonstrate that the user interface must be fixed and frozen sooner because of the impact of late changes on the costs of developing documentation, packaging, marketing collaterals, training materials, and support materials.

Implementation Risks

Gryna (1988) [14] and Juran & Gryna (1980) [15] point out several problems that have caused cost-of-quality approaches to fail. I’ll mention two of the main ones here.

First, it’s unwise to try to achieve too much, too fast. For example, don’t try to apply a quality cost system to every project until you’ve applied it successfully to one project. And don’t try to measure all of the costs, because you probably can’t. [16]

Second, beware of insisting on controversial costs. Gryna (1988) [17] points out several types of costs that other managers might challenge as not being quality-related. If you include these costs in your totals (such as total cost of quality), some readers will believe that you are padding these totals, to achieve a more dramatic effect. Gryna’s advice is to not include them. This is usually wise advice, but it can lead you to underestimate your customer’s probable dissatisfaction with your product. As we see in the next section, down that road lies LawyerLand.

The Dark Side of Quality Cost Analysis

Quality Cost Analysis looks at the company’s costs, not the customer’s costs. The manufacturer and seller are definitely not the only people who suffer quality-related costs. The customer suffers quality-related costs too. If a manufacturer sells a bad product, the customer faces significant expenses in dealing with that bad product.

The Ford Pinto litigation provided the most famous example of a quality cost analysis that evaluated company costs without considering customers’ costs from the customers’ viewpoint. Among the documents produced in these cases was the Grush-Saunby report, which looked at costs associated with fuel tank integrity. The key calculations appeared in Table 3 of the report: [18]

In other words, it looked cheaper to pay an average of $200,000 per death in lawsuit costs than to pay $11 per car to prevent fuel tank explosions. Ultimately, the lawsuit losses were much higher. [19]

This kind of analysis didn’t go away with the Pinto. For example, in the more recent case of General Motors Corp. v. Johnston (1992), [20] a PROM controlled the fuel injector in a pickup truck. The truck stalled because of a defect in the PROM and in the ensuing accident, Johnston’s seven-year old grandchild was killed. The Alabama Supreme Court justified an award of $7.5 million in punitive damages against GM by noting that GM "saved approximately $42,000,000 by not having a recall or otherwise notifying its purchasers of the problem related to the PROM."

Most software failures don’t lead to deaths. Most software projects involve conscious tradeoffs among several factors, including cost, time to completion, richness of the feature set, and reliability. There is nothing wrong with doing this type of business tradeoff, consciously and explicitly, unless you fail to take into account the fact that some of the problems that you leave in the product might cost your customers much, much more than they cost your company. Figure 2 lists some of the external failure costs that are borne by customers, rather than by the company.

Figure 2. Comparison of External Failure Costs Borne by the Buyer and the Seller

The point of quality-related litigation is to transfer some of the costs borne by a cheated or injured customer back to the maker or seller of the defective product. The well-publicized cases are for disastrous personal injuries, but there are plenty of cases against computer companies and software companies for breach of contract, breach of warranty, fraud, etc.

The problem of cost-of-quality analysis is that it sets us up to underestimate our litigation and customer dissatisfaction risks. We think, when we have estimated the total cost of quality associated with a project, that we have done a fairly complete analysis. But if we don’t take customers’ external failure costs into account at some point, we can be surprised by huge increased costs (lawsuits) over decisions that we thought, in our incomplete analyses, were safe and reasonable.

1. Gryna, F. M. (1988) "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4^th Ed.), Juran’s Quality Control Handbook, McGraw-Hill, page 4.2.

2. Feigenbaum, A.V. (1991, 3^rd Ed. Revised), Total Quality Control, McGraw-Hill, Chapter 7.

3. Gryna, F. M. "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4^th Ed.), Juran’s Quality Control Handbook, McGraw-Hill, page 4.2. I’m not aware of reliable data on quality costs in software.

4. These are my translations of the ideas for a software development audience. More general, and more complete, definitions are available in Campanella, J. (Ed.) (1990), Principles of Quality Costs, ASQC Quality Press, as well as in Juran’s and Feigenbaum’s works.

5. Principles of Quality Costs, ASQC Quality Press, Appendix B, "Detailed Description of Quality Cost Elements."

6. "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4^th Ed.), Juran’s Quality Control Handbook, McGraw-Hill, pages 4.9 - 4.12.

7. The product is scheduled for release on July 1, so your company arranges (far in advance) for an advertising campaign starting July 10. The product has too many bugs to ship, and is delayed until December. All that advertising money was wasted.

8. If the product had to be shipped late because of bugs that had to be fixed, the direct cost of late shipment includes the lost sales, whereas the opportunity cost of the late shipment includes the costs of delaying other projects while everyone finished this one.

9. Note, by the way, that you can reduce external failure costs without improving product quality. To reduce post-sale support costs without increasing customer satisfaction, charge people for support. Switch from a toll-free support line to a toll line, cut your support staff size and you can leave callers on hold for a long time at their expense. This discourages them from calling back. Because these cost reductions don’t increase customer satisfaction, the seller’s cost of quality is going down, but the customer’s is not.

10. This is the cost of cooperating with a government investigation. Even if your company isn’t charged or penalized, you spend money on lawyers, etc.

11. Some penalties are written into the contract between the software developer and the purchaser, and the developer pays them if the product is late or has specified problems. Other penalties are imposed by law. For example, the developer/publisher of a computer program that prepares United States taxes is liable for penalties to the Internal Revenue Service for errors in tax returns that are caused by bugs or design errors in the program. The publishers are treated like other tax preparers (accountants, tax lawyers, etc.). See Revenue Ruling 85-189 in Cumulative Bulletin, 1985-2, page 341.

12. I am most definitely not saying that a tactical approach is more practical than an integrated, long-term approach. Gryna notes that there are two common approaches to cost-of-quality programs. One approach involves one-shot studies that help the company identify targets for significant improvement. The other approach incorporates quality cost control into the structure of the business. (Gryna, 1988, in Juran, J. M. & Gryna, F. M. (1988, 4^th Ed.), Juran’s Quality Control Handbook, McGraw-Hill, pages 4.2 onward.) The one-shot, tactical approach can prove the benefit of the more strategic, long-term system to a skeptical company.

13. Be sensitive to how you do this. If you adopt a tone that says that you think the project manager and the programming staff are idiots, you won’t enjoy the long-term results.

14. in Juran, J. M. & Gryna, F. M. (1988, 4^th Ed.), Juran’s Quality Control Handbook, McGraw-Hill, pages 4.27-4.28.

15. Quality Planning and Analysis (2^nd Ed.), McGraw-Hill, pages 30-32. Also, see Brown, M. G., Hitchcock, D. E. & Willard, M. L. (1994), Why TQM Fails and What To Do About It, Irwin Professional Publishing.

16. As he says in Juran, J.M. (1992), Juran on Quality by Design, The Free Press, p. 119, Costs of poor quality "are huge, but the amounts are not known with precision. In most companies the accounting system provides only a minority of the information needed to quantify this cost of poor quality. It takes a great deal of time and effort to extend the accounting system so as to provide full coverage. Most companies have concluded that such an effort is not cost effective. [¶] What can be done is to fill the gap by estimates, which provide managers with approximate information as to the total cost of poor quality and as to where the major areas of concentration are."

17. "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4^th Ed.), Juran’s Quality Control Handbook, McGraw-Hill, pages 4.9 - 4.12.

18. This table is published in Keeton, W. P., Owen, D.G., Montgomery, J. E., & Green, M.D. (1989, 2^nd Ed.) Products Liability and Safety, Cases and Materials, Foundation Press, page 841 and Posner, R.A. (1982) Tort Law: Cases and Economic Analysis, Little Brown & Co., page 225.

19. Grimshaw v. Ford Motor Co. (1981), (California Court of Appeal), California Reporter, volume 174, page 348.

20. Southern Reporter, 2^nd Series, volume 592, pages 1054 and 1061.