WHAT IS COST ANALYSIS?
Cost analysis (also called economic evaluation, cost allocation, efficiency assessment, cost-benefit analysis, or cost-effectiveness analysis by different authors) is currently a somewhat controversial set of methods in program evaluation. One reason for the controversy is that these terms cover a wide range of methods, but are often used interchangeably.
At the most basic level, cost allocation is simply part of good program budgeting and accounting practices, which allow managers to determine the true cost of providing a given unit of service (Kettner, Moroney, & Martin, 1990). At the most ambitious level, well-publicized cost-benefit studies of early intervention programs have claimed to show substantial long-term social gains for participants and cost savings for the public (Berreuta-Clement, Schweinhart, Barnett, et al., 1984). Because these studies have been widely cited and credited with convincing legislators to increase their support for early childhood programs, some practitioners advocate making more use of cost-benefit analysis in evaluating social programs (Barnett, 1988, 1993). Others have cautioned that good cost-benefit or cost-effectiveness studies are complex, require very sophisticated technical skills and training in methodology and in principles of economics, and should not be undertaken lightly (White, 1988). Whatever position you take in this controversy, it is a good idea for program evaluators to have some understanding of the concepts involved, because the cost and effort involved in producing change is a concern in most impact evaluations (Rossi & Freeman, 1993).
THREE TYPES OF COST ANALYSIS IN EVALUATION:
Cost allocation, cost-effectiveness analysis, and cost-benefit analysis represent a continuum of types of cost analysis which can have a place in program evaluation. They range from fairly simple program-level methods to highly technical and specialized methods. However, all have specialized and technical aspects. If you are not already familiar with these methods and the language used, you should plan to work with a consultant or read some more in-depth texts (see some suggested references at the end of this discussion) before deciding to attempt them.
COST ALLOCATION: Cost allocation is a simpler concept than either cost-benefit analysis or cost-effectiveness analysis. At the program or agency level, it basically means setting up budgeting and accounting systems in a way that allows program managers to determine a unit cost or cost per unit of service. This information is primarily a management tool. However, if the units measured are also outcomes of interest to evaluators, cost allocation provides some of the basic information needed to conduct more ambitious cost analyses such as cost-benefit analysis or cost-effectiveness analysis. For example, for evaluation purposes, you might want to know the average cost per child of providing an after-school tutoring program, including the costs of staff salaries, snacks, and other overhead costs.
Besides budget information, being able to determine unit costs means that you need to be collecting the right kind of information about clients and outcomes. In many agencies, the information recorded in service records is based on reporting requirements, which are not always in a form that is useful for evaluation. If staff in a prenatal clinic simply report the number of clients served by gender, for example, you might know only that 157 females were served in March. For an evaluation, however, you might want to be able to break down that number in different ways. For example, do young first-time mothers usually require more visits than older women? Do single mothers or women with several children miss more appointments? Is transportation to appointments more of a problem for women who live in rural areas? Are any client characteristics commonly related to important outcomes such as birth weight of the the baby? Deciding how to collect enough client and service data to give useful information, without overburdening staff with unnecessary paperwork requirements, requires a lot of planning. Larger agencies often hire experts to design data systems, which are called MIS or management-and-information-systems.
If you are working for an existing agency, your ability to separate out unit costs for services or outcomes may depend on the systems that are already in place for budgeting, accounting, and collecting service data. However, if you are in a position to influence these functions, or need to supplement an existing system, there are a number of texts that discuss the pros and cons of different ways of budgeting, accounting, and designing MIS or management-and-information-systems (see Kettner, Moroney & Martin, 1990).
COST-EFFECTIVENESS AND COST-BENEFIT STUDIES
Most often, cost-effectiveness and cost-benefit studies are conducted at a level that involves more than just a local program (such as an individual State Strengthening project). Sometimes they also involve following up over a long period of time, to look at the long-term impact of interventions. They are often used by policy analysts and legislators to make broad policy decisions, so they might look at a large federal program, or compare several smaller pilot programs that take different approaches to solving the same social problem. People often use the terms interchangeably, but there are important differences between them.
COST-EFFECTIVENESS ANALYSIS: Cost-effectiveness analysis assumes that a certain benefit or outcome is desired, and that there are several alternative ways to achieve it. The basic question asked is, "Which of these alternatives is the cheapest or most efficient way to get this benefit?" By definition, cost-effectiveness analysis is comparative, while cost-benefit analysis usually considers only one program at a time. Another important difference is that while cost-benefit analysis always compares the monetary costs and benefits of a program, cost-effectiveness studies often compare programs on the basis of some other common scale for measuring outcomes (eg., number of students who graduate from high school, infant mortality rate, test scores that meet a certain level, reports of child abuse). They address whether the unit cost is greater for one program or approach than another, which is often much easier to do, and more informative, than assigning a dollar value to the outcome (White, 1988).
COST-BENEFIT ANALYSIS: The basic questions asked in a cost-benefit analysis are, "Do the economic benefits of providing this service outweigh the economic costs" and "Is it worth doing at all"? One important tool of cost-benefit analysis is the benefit-to-costs ratio, which is the total monetary cost of the benefits or outcomes divided by the total monetary costs of obtaining them. Another tool for comparison in cost-benefit analysis is the net rate of return, which is basically total costs minus the total value of benefits.
The idea behind cost-benefit analysis is simple: if all inputs and outcomes of a proposed alternative can be reduced to a common unit of impact (namely dollars), they can be aggregated and compared. If people would be willing to pay dollars to have something, presumably it is a benefit; if they would pay to avoid it, it is a cost. In practice, however, assigning monetary values to inputs and outcomes in social programs is rarely so simple, and it is not always appropriate to do so (Weimer & Vining, 1992; Thompson, 1982; Zeckhauser, 1975).
An example will illustrate some of the differences between Cost-Effectiveness and Cost-Benefit studies, what they can tell you, and some of the issues that neither can effectively address:
"Suppose the drop-out rate in an inner-city high school is 50%. Prevention Program A enrolls 20 students, costs $20,000, and 15 of the 20 students (75%) graduate. Thus Program A resulted in 5 additional graduates at a cost of $20,000, or one additional graduate for every $4,000. Prevention Program B enrolls 20 students, costs $15,000, and 12 of the 20 students (60%) graduate. Thus Program B resulted in 2 additional graduates at a cost of $15,000, or one additional graduate for every $7,500 spent. Although Program B is cheaper ($15,000 compared to $20,000), Program A is more COST-EFFECTIVE ($4,000/each additional graduate, compared to $7,500/additional graduate). A COST-BENEFIT ANALYSIS in this situation, instead of comparing unit costs, would require estimating the dollar value of high school graduation (for example, by projecting the difference in lifetime earning capacity of graduates over drop-outs, and lifetime social service costs), and comparing the monetary value of producing more graduates to the monetary cost of providing the program in the first place. Neither method effectively addresses more intangible outcomes of graduation, such as increased self-esteem, or the value of a peer support system." (White, 1988, p. 430)
WHAT COST ANALYSES CAN TELL YOU:
WHAT COST ANALYSES CANNOT TELL YOU:
USING COST ANALYSES WITH THE STATE STRENGTHENING EVALUATION GUIDE:
If you are using the Five-Tiered Approach to Program Evaluation outlined in the State Strengthening Evaluation Guide, cost analyses can be used at several levels:
Tier 1 - Program Definition
At this stage, you will probably be using cost studies based on other people's experience in similar programs, since you are unlikely to have cost data of your own yet. This means that the estimates you use will only be approximations, and may not accurately reflect what your program's experience will be. However, "ex-ante" cost analyses, done in the planning stages before implementing a program, can potentially prevent some very costly mistakes. If you have access to cost-effectiveness studies of programs similar to the one you are considering, especially if they allow you to compare the relative costs and benefits of several different ways of delivering a service, before you have made substantial investments of time or money, some program design decisions may be easier. One common example in community-based programs is staffing (eg., deciding whether to use highly-trained professionals to deliver services, or to rely on less highly-paid paraprofessionals or volunteers). While many people assume that the paraprofessionals or volunteers are always less expensive, cost-effectiveness studies in some cases have found that the professionals may be less costly in the long run because they can see more clients, require less supervision time, or are more effective. Of course, costs also need to be weighed against other considerations, such as the fact that paraprofessionals recruited from the community served may more easily gain the trust of clients.
Tier 2 - Accountability
Clearly, fiscal accountability is one of the primary reasons for using any kind of cost analysis as part of your evaluation. Any responsible program should keep service statistics and financial records that are accurate and up-to-date enough to be able to determine some very basic information about unit costs, and funders usually require this. However, the minimal information routinely collected by programs for fiscal and reporting purposes is not always in a form that lends itself to evaluation uses. Often, unless advance planning has taken place, this data is too aggregated to reflect outcomes of interest to researchers and evaluators. At this stage (or earlier), careful consideration should be given to the kinds of client and cost data that will be needed later, so that it can be built into the accounting and record-keeping systems of the program (see Kettner, Moroney & Martin, 1990).
Tier 3 - Understanding and Refining
Like any other type of information gathered for evaluation purposes, the cost information collected in Tier 2 for accountability purposes provides programs with a basis for mid-course adjustments and program refinements, either at the end of a funding cycle, or in the course of implementation.
Tier 4 - Progress Toward Objectives
Using cost information in Tier 4 is closely tied to the program design issues of Tier 1, and the accountability issues of Tier 2. If appropriate program outcomes and indicators have been identified in Tier 1, and the appropriate unit cost information is included in the routine data that is collected as part of Tier 2, then the job of identifying progress toward objectives in Tier 4 becomes much easier.
Tier 5 - Program Impact
When it has been possible to conduct a full-scale cost-benefit analysis over a long period of time, and it shows significant long-term gains and cost savings in a particular population or problem area, the policy implications may be great. One of the best-known examples is the Perry Preschool Study (discussed earlier), which has been credited with persuading lawmakers to sustain or significantly increase their support for early intervention programs, including Head Start. It is widely believed that one reason that this study was so influential was the fact that it included a cost-benefit analysis. While monetary cost is not the only basis for policy decisions, it is usually a very salient one for voters and politicians.
ADVANTAGES OF USING COST ANALYSES:
DISADVANTAGES OF USING COST ANALYSES:
HOW TO BUDGET & ALLOCATE COSTS FOR COST EFFECTIVENESS STUDIES:
The type of budgeting and accounting system your program or agency uses may well determine how much useful cost data is available for evaluating your program, or comparing it to others. Three major types of budgeting formats commonly used in social service programs will provide different types and amounts of information (Kettner, Moroney, & Martin, 1990).
The most common format is the Line-Item Budget format, which simply looks at revenues (money coming in from various sources, including grants, user fees or United Way funds) and expenditures (costs broken down into broad categories like salaries, rent, utilities, and postage), and tries to ensure that they balance. The main purpose of a line-item budget is financial control, and the categories are usually too broad to give much information about the cost of providing a particular service or obtaining a particular result.
The Functional Budget format starts with a line-item budget, and takes it a step further. It focuses on process, or the cost of providing a service. For example, with a Functional Budget, we could determine that it cost an adoption agency $45,000 to conduct 100 home studies (an activity which is a necessary part of the process of placing children in permanent homes).
The Program Budget, which also starts with a line-item budget, looks at the same information from the point of view of outcomes, or the cost of achieving a result. For example, if the 100 home studies resulted in actually placing 50 children in adoptive homes, the Program Budget would allow us to say that it cost the agency $45,000 to place 50 children, which is an outcome.
Another way to look at this is that functional budgets measure productivity, and program budgets measure the cost of achieving goals and objectives.
COMMON STEPS IN DEVELOPING PROGRAM & FUNCTIONAL BUDGETS (Kettner, et al., 1990):
1. Develop a line-item budget that shows all expenditures. This is the minimal level of budgeting and accounting that is required by many funders, such as the United Way. Some funders require a specific format, so that the categories used are standard across the programs that they fund.
2. Determine the agency's program structure. A distinct program is a set of activities or services designed to accomplish a specific set of agency goals and objectives. Many agencies have several different programs.
3. Identify all direct costs and indirect costs. Direct costs are those that benefit only one program (for example, salaries of staff who work only for one program, or supplies and equipment used only for that program). Indirect costs or "overhead" costs are those that benefit or are shared by more than one program (for example, several programs in an agency might share the same building, and be served by the same bookkeeping and secretarial staff, utilities, or janitorial services).
4. Assign direct costs to the appropriate program or project. This is usually fairly straightforward. If one county agent has full-time responsibility for operating your State Strengthening project, for example, then 100% of his or her salary and benefits would be assigned as an expense to that project in the budget. If a staff member spends 50% of his or her time on the State Strengthening project and 50% on another assignment, then half of that person's salary and benefits would be assigned to the State Strengthening project as a direct cost.
5. Allocate indirect costs to programs. Deciding how to divide up the indirect (shared) cost pool among several programs in the agency can be much more complicated and technical. The actual practice of allocating or dividing up the indirect costs is usually best left to an accountant. There are several methods for doing this, each with particular advantages and disadvantages (see Kettner, et al., 1990). Although cost allocation of indirect costs can be a time-consuming step, it is considered well worth doing because of the increased information it provides about the real costs of providing services.
6. Determine total program costs. The total cost of a particular program (such as your State Strengthening project) is the sum of the direct costs, and the portion of indirect costs that is allocated to that program.
Once we have this information about total program costs, then we can calculate unit costs. For a Functional Budget, this involves defining the units of service for each program (eg., hours of day care provided, meals delivered, home studies conducted), and calculating the cost per unit of service. In the adoption agency example above, the unit cost of conducting a home study would be $450 (total program cost divided by number of units of service provided). For a Program Budget, the final steps are determining the total cost of achieving the outcome objectives for the year, and calculating the cost per outcome. Using the adoption example again, we can say that the adoption agency described above successfully placed children in adoptive homes at a unit cost of $900/child (total program costs divided by the number of successful outcomes).
COST-EFFICIENCY & COST-EFFECTIVENESS STUDIES: From the point of view of program evaluation, both the Program and Functional Budgeting systems are more useful than a Line-Item Budget. Unit cost information allows for useful comparisons of the costs of delivering services and getting results. With this information, we can look at the unit cost of one adoption agency compared to another, to see whether one operates more efficiently. We can also compare the unit cost (per child) of adoptive placement to the unit cost (per child) of placement in foster care or residential treatment. This is basically what happens in a cost-effectiveness study.
In general, a cost-effectiveness study is more appropriate than a cost-benefit analysis when your goals or outcomes can't easily be quantified or monetized, or when there are multiple competing goals. As with budgeting and cost allocation, there are a variety of approaches to cost-effectiveness studies. The approach that is best for your purposes will depend on a number of factors. A good source of more detailed information about deciding what approach is most appropriate, and conducting the various types of cost-effectiveness studies, is Weimer & Vining (1992).
HOW TO CONDUCT A COST-BENEFIT ANALYSIS:
Cost-benefit analysis is by far the most complex and controversial of the three methods of costs analysis we have discussed. It should not be attempted by those who lack technical expertise in this area. However, for some purposes, it is also one of the most powerful methods. For those who decide to undertake a cost-benefit analysis in spite of the difficulties, Barnett (1993) outlines a nine step process. Various standard texts are recommended for more in-depth information (see below).
1. Barnett, W. S. (1993). The economic evaluation of home visiting programs. The Future of Children: Home Visiting, 3. Center for the Future of Children, the David and Lucile Packard Foundation, 93-112.
The Future of Children journal is available on-line. Some issues can be viewed or downloaded at the following World Wide Web site, and older hard copy issues can be ordered free of charge at: http://www.futureofchildren.org
Barnett is a strong advocate of more use of cost-benefit or cost-effectiveness analyses in evaluations of social programs. He has been involved in conducting several influential cost-benefit studies of home visiting and early intervention programs (including the well-known Perry Preschool Program study), and reviews several others. He argues that it is more feasible than most evaluators believe, and provides a relatively detailed discussion of steps and procedures.
2. Barnett, W.S. (1985). Benefit-cost analysis of the Perry Preschool program and its policy implications. Educational evaluation and policy analysis, 7. 333-342.
3. Berreuta-Clement, J. R., Schweinhart, L. J., Barnett, W. S. , et al. (1984). Changed lives: The effects of the Perry Preschool program on youths through age 19. Monographs of the High/Scope Educational Research Foundation, no. 8. Ypsilanti, MI: High/Scope Press.
One of the best-known and most influential longitudinal studies of the effects of early intervention programs. The influence of this study on funders and policy makers is often attributed to its use of a cost-benefit analysis to show that funding intensive programs for low-income preschoolers can be a good long-term investment for society.
4. Callor, S., Betts, S. C., Carter, R., & Marczak, M. (1997). State Strengthening Evaluation Guide. Tucson, AZ: USDA/CSREES & University of Arizona.
Program development and evaluation manual designed by the Evaluation Collaboration for use in State Strengthening Projects. Based on the five-tiered model of evaluation of Jacobs (1988). The manual may be viewed or downloaded from the World Wide Web at: ttp://ag.arizona.edu/sfcs/cyfernet/cyfar/evalgde.htm
5. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin Co.
This is a standard reference on quasiexperimental research designs, which are commonly used for evaluation of community-based programs.
6. Jacobs, F.H. (1988). The five-tiered approach to evaluation: Context and implementation. In H.B. Weiss, & F.H. Jacobs (Eds.), Evaluating family programs. New York: Aldyne de Gruyter.
This chapter describes a model of evaluation of community-based programs which served as the basis for the model used in the State Strengthening Evaluation Guide.
7. Kettner, P. M., Moroney, R. M., & Martin, L. L. (1990). Designing and managing programs: An effectiveness-based approach. Newbury Park, CA: Sage.
Useful discussion of the pros and cons of several budgeting systems (line-item, functional, and program budgeting) in terms of facilitating the calculation of unit costs. Emphasizes that cost allocation is time-consuming and requires careful choices about methodology. Concludes that despite the difficulties of setting up such a budgeting system, it is worth the time and effort for programs to be able to accurately determine true unit costs and program costs. Although the advantages are primarily discussed in terms of practical benefits for program managers, the ability to accurately allocate costs and value of services is also a prerequisite for cost-benefit analysis as a long-term evaluation strategy.
8. Mishan, E. J. (1988). Cost-benefit analysis (4th ed.). London: Unwin Hyman.
An economics textbook, and one of the standard references for cost-benefit analysis techniques. Although fairly technical and mathematically oriented, this book is intended to be accessible to serious non-economists who are willing to devote some time and effort to learning the theory and techniques. Includes a sometimes whimsical discussion of the problems of valuing intangibles, such as time.
9. Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury Park: Sage Publications, pp. 363-401.
A standard evaluation text. Rossi and Freeman take the position that while "efficiency assessment" (their term for cost-benefit and cost-effectiveness analyses) is highly technical therefore not always feasible, all program evaluators should at least have some understanding of the basic concepts involved. This is because impact evaluations always implicitly involve issues of how much effort or cost is required to achieve a desired outcome.
10. Thompson, M. S. (1980). Benefit-cost analysis for program evaluation. Beverly Hills: Sage.
Another basic text, which elaborates on the quantitative aspects of benefit-cost (or cost-benefit) analysis. Includes in-depth discussions of issues of discounting and cost adjustments.
11. Weimer, D. L., & Vining, A. R. (1992). Policy analysis: Concepts and practice (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
Fairly technical discussion aimed at professional policy analysts; assumes a background in economics. Useful discussion and flow chart distinguishes five related approaches and when they are appropriate to use: Benefit-Cost Analysis, Qualitative Benefit-Cost Analysis, Modified Benefit-Cost Analysis, Cost-Effectiveness Analysis, and Multi-Goal Analysis. Appropriateness of each method is determined primarily by 1) whether the relevant impacts and inputs can be readily quantified and monetized, and 2) whether there are goals or benefits other than economic efficiency which need to be incorporated into the analysis.
12. White, K. R. (1988). Cost analyses in family support programs (pp. 429-443). In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs. New York: Aldyne de Gruyter.
Suggests use of general term "cost analyses". Basically takes a skeptical approach, noting that "present conceptualizations of cost analysis are overly simplistic, expectations are unrealistically high, and much of what is labeled as cost-effectiveness or cost-benefit research suffers from serious conceptual and methodological inadequacies" (p.429).
13. Zeckhauser, R. (1975). Procedures for valuing lives. Public Policy, 23(4), 419-464.
Serious discussion of philosophical, legal, and economic aspects of attempts to assign economic value to human lives that are saved, injured, or lost. More than most people may want to know, but a key ethical issue in cost-benefit analyses of many social and health related policies and programs.
Back to Alternative Methods website.