Chapter 14 Measuring the performance of health services and programmes

Take the chapter quiz before and after you read this chapter.

Open chapter quiz

Contents

• Unit 31: Types of performance measures in health services
• Objectives
• Case studies
• Unit 32: Adverse event monitoring
• Objectives
• Case studies
• Unit 33: Cost analysis
• Objectives
• Case studies
• Unit 34: Programme monitoring and evaluation
• Objectives
• Case studies

Unit 31: Types of performance measures in health services

Objectives

When you have completed this unit you should be able to:

• List the various approaches to measuring performance.
• Define standards for structures and processes.
• Choose a good indicator to measure performance.

31-1 Why is it necessary to measure performance in health systems?

Performance in health systems is measured to see whether standards are met. These may have been set by a local team to measure their own performance, or may have been set by managers or an outside organisation. This makes it possible to identify what is not working well and to put measures in place to address it.

31-2 What are standards?

There are different types of standards:

• Structure standards refer to facilities, staffing, equipment and administration. For example, the presence of filled soap dispensers.
• Process standards refer to activities or performance. For example, how often people wash their hands between patients. Process standards of performance are more difficult to measure.

Sets of standards for structures and processes are created for the purpose of accreditation.

31-3 What is accreditation of a health facility mean?

Accreditation of a health facility means that the facility has met the standards set. Sometimes medical aid or insurance schemes require a facility to be accredited before they pay for their members to use it. The process of accreditation is important to maintain high standards of care.

Facilities can be accredited based on structure and process standards.

31-4 What are the approaches to measuring performance in health systems?

There are 3 basic approaches to performance monitoring in healthcare:

• “Target” type assessments to see whether targets or standards are met. These include:
  • Audits of standards of structures and processes in health facilities.
  • Targets for activities or results
• Assessments that ask why there is a gap between what is happening and what should happen, and try to close that gap:
  • Quality improvement cycles.
  • Programme monitoring and evaluation.
• Assessments where things have gone wrong. Problems are identified and analysed:
  • Adverse event reporting.
  • Mortality and morbidity meetings.

31-5 Who sets standards and decides whether a facility can be accredited?

Often an independent body, such as the Council for Health Service Accreditation of South Africa, will develop standards and offer accreditation. There are also core standards for health facilities published by the National Department of Health’s Office of Standards Compliance. Many accreditation bodies offer support and capacity building to help a facility become accredited.

A number of organisations will accredit facilities if standards are met.

31-6 Does accreditation improve the quality of care?

No. By itself the accreditation process does not improve quality of care but it ensures an environment in which quality care can happen. Standards of accreditation provide a target for staff and managers to aim at when they are trying to improve healthcare facilities.

31-7 Who sets targets for activities and results?

Targets can be set by a variety of managers, leaders or team members. For example:

• Targets for activities (such as the number of patients seen) and results (such as the number of wound infections) are usually part of the performance agreement that district managers have with provincial departments of health. This is part of health service governance.
• The Millennium Development Goals were targets for poverty reduction and health improvement that were agreed between member governments of the United Nations. They were replaced in 2015 by the Sustainable Development Goals 2030. Goal 3 is Good Health and Well-being.
• Facilities, or departments within facilities may also pick their own targets.

The Sustainable Development Goals 2030 are international targets to improve the quality of life for all within a sustainable environment.

Note

The Sustainable Development Goals 2030 can be accessed on the United Nations Development Program website.

31-8 What is the basic tool for measuring performance?

The tool used for measuring performance is called an indicator. Indicators are markers that can be used to measure an activity or a result that is seen as important. A good indicator should:

• Be valid: It should really reflect the thing you are interested in.
• Be possible and convenient: It should be possible to collect the information you have asked for without too much trouble.

The tool used to measure performance is called an indicator.

31-9 What is meant by clinical audit?

An audit is an inspection or assessment to make sure that something is up to standard. The term clinical audit is used in 2 different ways. It is sometimes used to describe a “report card” type of performance assessment, but the term “audit cycle” is also used to mean a quality improvement process.

An audit is an assessment to make sure that something meets a set standard.

Case study 1

A student wants to know how good the diabetic control is in a community clinic. He thinks of 3 possible indicators.

1. Which of the following do you think would be the best?

Option 1: The number of hospitalisations in each diabetic patient.
Option 2: A 6-hourly measure of serum glucose using a haemoglucotest over the course of the next month in each diabetic patient.
Option 3: The 6-monthly check of glycosylated haemoglobin (HbA1C).

Option 1 is not a valid indicator because the patient may have bad glucose control without being hospitalised, or he or she might be hospitalised for a reason unconnected to the diabetes. Option 2 would give valid information, but is very unlikely to be possible, unless the diabetic is already regularly checking their blood glucose levels accurately at home every 6 hours. There are very few diabetics in South Africa who do this. Therefore option 3 is the best answer, as a check of HbA1C is usually part of standard care. The HbA1C will show the glucose control over the previous 3 months and it should be easy to find this information from the patient records, or online laboratory results if these are available.

Unit 32: Adverse event monitoring

Objectives

When you have completed this unit you should be able to:

• Explain what an adverse event is.
• Plan a mortality and morbidity meeting.
• Look for underlying causes using a “Why Tree”.
• Design systems for improving patient flow.

32-1 What is an adverse event?

An adverse event is a situation where some part of the healthcare process has gone wrong, and this has resulted in some sort of harm to a patient.

The harm might be major harm or death or it could be something that health professionals consider quite minor, such as a patient having to spend the night in casualty instead of going home because they did not know where the patient transport services station was to be taken home .

An adverse event is a situation where part of the healthcare process has failed resulting in some sort of harm to a patient.

32-2 Why should we monitor adverse events?

If we can work out what has gone wrong during the management of the patient, we can identify failed activities and processes that we can feed into a quality improvement programme and try to stop the same problem occurring for other patients.

Adverse events can feed into a quality improvement programme if the underlying causes are identified.

32-3 How can adverse events be monitored?

Adverse events can be informally monitored within the clinical unit or facility and then discussed at a mortality and morbidity (M&M) meeting.

Note

A mortality and morbidity meeting is also known as a deaths and complications (D&C) meeting.

There are also formal error reporting systems that are not widely used in South Africa. The disadvantage of these systems is that information goes “up” the chain of management and may not go back to the frontline clinicians who can make best use of it.

The World Health Organisation has a system for record review to monitor and list adverse events. It can show trends and compare countries. Because records contain incomplete information, this approach has a low to moderate reliability.

32-4 What is a “near miss”?

In a “near miss”, some part of the health system has gone wrong but the problem was picked up and managed without causing harm to patients. For example, if a patient was put on an operating list for an amputation of the wrong leg, but the theatre nurse realised this when the patient was handed over to her. It is also important to work out what went wrong and learn from near misses.

A “near miss” is something that nearly went wrong but the error was detected in time to prevent it happening.

32-5 How should mortality and morbidity meetings be run?

It is important that meetings are held in a “no blame” setting. If somebody feels they will be criticised or singled out in the meeting, they may hide their complications or avoid attending. This means that others will not have the benefit of learning from them and you are unlikely to get to the underlying causes of the problem. Always ask “how could we have done better” rather than how could an individual have done better. It may be necessary to put these “ground rules” in writing.

Mortality and morbidity meetings must be held in a “no blame” setting.

All deaths should be listed, and all staff members should be allowed and encouraged to put complications, adverse events and near misses on the agenda for discussion as well:

• The person running the meeting should review these cases beforehand. Many of the deaths will be unpreventable and not too much time should be spent discussing them. There may just be one or 2 cases that deserve an in-depth discussion.
• If a patient has had a post-mortem, the results should be tracked and shared.
• If a misunderstanding or misdiagnosis is identified beforehand, it is useful to have a short learning presentation prepared while the problem is fresh in people’s minds.
• All relevant team members should be invited. For example, if a missing record has led to a death or complication, it is appropriate to invite the head of records.
• Do not mention the name of the staff member who has made a mistake. A mistake may be due to an “honest error”, a lack of knowledge or clinical skills, overwork or lack of care.

A mortality and morbidity meetings is one of the best opportunities to learn especially if the whole group is asked to identify the mistake and say how it could be avoided in future.

A mortality and morbidity meeting is an excellent opportunity to learn from mistakes and find ways of avoiding them.

32-6 How can the underlying causes be identified when something has gone wrong?

It is important to try and identify the underlying causes of problems. They may not be obvious. For example, a confused patient may be given a drug to which they are allergic, but this could be because their records were missing and that in turn might be because of an unsatisfactory filing system. The best way of uncovering the underlying causes of a problem is by brainstorming with the relevant people in your team during the mortality and morbidity meeting. There may be an obvious underlying cause and solution. If not, there are tools such as the “5 whys” that you can use to get to the root of the problem. This is sometimes called a “root cause analysis”.

32-7 What are the 5 whys?

Many have had the experience of trying to explain something to a young child, and found the response to every explanation to be “but why?” until the adult comes to a point when they have to say: “because I say so”. This process of asking “why” until you cannot come up with another response is a useful approach to root cause analysis. The approach is sometimes called the “5 whys” because it is said that you have to ask “why?” 5 times to get to the root problem. Of course, you may ask “why?” fewer or more than 5 times. Here is an example of 5 whys:

1. Q: Why didn’t Mr Felies get his antibiotic this morning? A: Because Sr A didn’t give it to him.

2. Q: Why didn’t Sr A give Mr Felies his antibiotic? A: Because there wasn’t any Augmentin in the trolley.

3. Q: Why wasn’t there any Augmentin in the trolley? A: Because it was out of stock at the pharmacy.

4. Q: Why was Augmentin out of stock at the pharmacy? A: Because Mr P didn’t order it.

5. Q: Why didn’t Mr P order it? A: Because he broke his leg and is on sick leave and nobody has replaced him.

It is possible take this example further and ask why Mr P broke his leg, or why management did not find a replacement, but after asking “why” 5 times we are getting close to the underlying reason why Mr Felies did not receive his antibiotic. We can also identify possible solutions to prevent similar stock outs that may have an impact on patient care.

The “5 whys” method helps to examine in detail the many factors which lead to an adverse event.

Figure 32-1: Simplified example of a “why tree” for overcrowding of an emergency unit over weekends.

32-8 What is a why tree?

Sometimes the answer to a question raises more than one other question and this means that other chains of “whys” branch off from the original question. For example, in question 2 in the example above, one might also ask why Sr A didn’t ask the medical officer for an alternative prescription and then find out several other interesting things. When answers to the question “why?” branch in this way, it is useful to draw the problem as a tree. When using this approach for a real life problem, it is most effective to brainstorm the “whys” with others (See Figure 32-1).

In real life, a final “why tree” will probably be a lot more complicated. When the exercise has been completed and the problem has been picked apart as much as possible, all the underlying reasons need to be addressed in some way (see Table 32-1).

Table 32-1: Why is casualty overcrowded at the weekend?

Health Team Factors	Health Service Factors	Social and Economic
Some senior staff do not do weekend shifts	Primary care clinic not open at weekend	Law enforcement does not like to enter community because relations with community are poor
Clinical staff are not coding their caseload		Employers not paying sick leave
		Employment is informal/casual
		Low standards at local secondary school

Organising the root causes in this way helps to make sense of a complex problem. The causes can also be scored for ease and impact, so that easy to implement high impact changes can be prioritised.

32-9 What does “patient safety” mean?

Patient safety refers to the design and use of systems to reduce adverse events. Safety in health systems is often compared unfavourably with safety in the airline industry, which has a much more stringent approach to the prevention of accidents. This is not completely fair because an aeroplane never has to make an emergency take off or to carry more passengers than it has seats for. However, there are still many improvements that health systems can and should make.

Patient safety means designing and using systems that reduce the risk of adverse events.

32-10 How can systems be designed to reduce adverse events?

Many adverse events occur because somebody has done something wrong. Making an error is part of being a human, and because healthcare delivery depends so much on human choices and actions, there will always be errors. The important thing is to design systems so that errors can be caught before they cause harm. One way of thinking about systems to catch errors is the “Swiss Cheese Model”. The various steps in patient management are seen as slices of Swiss cheese – they are each designed to protect the patient, but each step has holes (errors) that could allow harm to happen. When the holes are lined up, all defences have failed and harm can result (see Figure 32-2). Patient safety involves redesigning a slice, or inserting a new protective slice.

For example: We wish to make sure that urgent cases coming to casualty are managed as quickly as possible. The process for admitting patients to casualty is as follows:

1. The patient registers at front desk.
2. The front desk clerk takes the folder to triage.
3. The triage nurse calls the patient and makes observations to determine how urgent the situation is.
4. Urgent patients are taken to one of the clinical areas and the doctor is informed.
5. A nurse is assigned to the urgent clinical areas to monitor all patients there.

What happens in this situation when the ambulance brings in an urgent patient and the front desk clerk has gone to search for a missing folder? The Emergency Medical Services (EMS) provider may take the patient straight to an urgent clinical area. But what if the nurse from that area is taking a tea break? If the Emergency Medical Services provider leaves the patient in an urgent clinical area because they must attend to another call, it is possible that the patient will just lie there and deteriorate. All the holes in the system have lined up and led to patient harm.

One possible extra “slice” to add, is a policy that the Emergency Medical Services provider must hand over patients to a qualified staff member. The sister in charge at the time can be clearly identified and be tasked with taking handovers if no trained staff are in the clinical area at the time. Trollies or beds containing patients who have been delivered by the Emergency Medical Services provider and bypassed triage could also be clearly identified with a board or a ribbon.

Figure 32-2: Successive layers of defences, barriers and safeguards. All the holes in the system have lined up and led to patient harm.

Case study 1

It is 3am in St Jude’s casualty department. Having worked all day and all night, Dr Prudence has just placed an intercostal drain in a patient with a stabbed chest. Shortly afterwards, a staff nurse asks her why the drain is in the left side when the stab is on the right. Horrified, Dr Prudence realises that the X-ray was wrongly labelled and she has put the drain in on the wrong side.

1. What should she do?

The right thing to do is go straight to the patient, explain what has happened and put it right (in this case, unfortunately it would mean putting in another drain on the correct side). She should make good notes, including her conversation with the patient, and report it to her senior as soon as possible.

She reports the event to Dr Retief, the family physician, the following morning. Dr Prudence is very embarrassed and worried.

2. Should Dr Retief discuss the event at the next Mortality and Morbidity meeting?

Yes. The purpose of a Mortality and Morbidity meeting is for everybody to learn from what has gone wrong. There are ways of discussing mistakes in ways that do not make it worse for the person who has had an event.

• Dr Prudence’s name should not be mentioned and she should not be singled out for discussion.
• The discussion should be around general lessons for all, such as: mistakes will happen, but:
  • Be particularly careful when you know you are tired.
  • Treat the patient, not the X-ray.
• Positives should also be mentioned for learning, such as the way in which “the doctor concerned” managed the error and communicated with the patient.

Unit 33: Cost analysis

Objectives

When you have completed this unit you should be able to:

• Describe the importance of perspective when assessing costs.
• Explain the economic analysis tools that health leaders use to compare the value for money offered by programmes and procedures.
• Consider the use of these economic analysis tools in health planning.

33-1 Why should clinicians know about cost analysis in health systems?

Healthcare costs rise as new innovations become available and public expectations of what can and should be done increase. Because there is a fixed amount of money available in the budgets of health facilities and health services, decisions may need to be made about what the available funds should be spent on. Clinicians need to be aware of costing issues so that they can:

• Be aware of the costs of items such as blood tests so they can use the available budget wisely and avoid unnecessary expense.
• Have an input into decisions about whether new technologies should be made available and whether they can be afforded.

33-2 How are healthcare costs analysed?

It is useful to understand:

• How costs are calculated for particular procedures, such as a Caesarean section. This is called a costing.
• The tools that managers use to look at the costs of achieving health outcomes. This is to compare options and decide which procedures represent value for money in healthcare. This is called economic evaluation.

Calculating the costs for a procedure is called costing.

33-3 What is included in a costing?

There are 4 categories of healthcare costs. These are:

• Resources used within the health sector.
• Resources used by the patient and their families.
• Resources used within other sectors.
• The costs of lost productivity.

What is included depends on the “perspective” of the costing.

33-4 What is the meaning of the “perspective” of the costing?

The “perspective” of the costing refers to whose money is being spent. Common costing perspectives are:

• The clinical manager’s perspective: This relates to direct costs that are billed to the manager’s budget. For example, the cost of a laboratory test or a piece of equipment.
• The patient’s perspective: This includes healthcare costs that are billed to the patient, together with personal transport costs and indirect costs such as time off work and payment for childcare.
• The health system perspective: As well as billed costs, these include overhead costs for the running of the health facility, building costs, staff costs and so on.
• The “global” (overall) perspective: This takes into account all costs, including the cost to the economy from lost labour productivity.

Clinicians do not need to be able to do a costing, but it is useful to understand that different cost items are important to different role players, including the patient.

The perspective of costing refers to whose money is being spent.

33-5 What is a patient day equivalent?

The patient day equivalent (PDE) is a measure commonly used by managers in South Africa to assess the efficiency of a hospital. The total cost of running the hospital is divided by the number of inpatient-days plus half the number of day cases plus a third of outpatient emergency consultations. This gives an average cost per patient seen. The patient day equivalent is used by managers to compare running costs for hospitals of similar size with similar caseloads. It is not a perfect measure because it does not show the effectiveness of care. For example, a patient who lies in a hospital bed without receiving much attention for a month uses the same number of patient day equivalent at a lower cost than 15 patients who occupy a bed for 2 days each but are quickly investigated and treated.

The patient day equivalent is a measure used to assess the efficiency of a hospital.

33-6 What tools are used to determine “value for money” in health spending?

By itself, a costing does not indicate whether money was well spent, or whether it could have been better spent elsewhere. For that it is necessary to measure not only the cost but also the outcomes. There are 3 different economic analysis tools (“economic evaluations”) that are used to compare the value for money of different interventions. They differ in the way that the outcomes are presented. They are:

• Cost effectiveness analysis
• Cost utility analysis
• Cost benefit analysis

33-7 What is cost effectiveness analysis?

The outcomes measured in cost effectiveness analysis are direct outcomes, or the things we want to achieve with the intervention that are specific to the disease. For example, TB treatment completion rates or hernia recurrence rates. A cost effectiveness assessment might be presented as cost per additional case prevented, or cost per additional child immunised. With cost effective analysis one estimates what will be achieved for the money spent.

Cost effectiveness analysis measures the financial cost of direct outcomes of an intervention.

33-8 What is cost utility analysis?

In cost utility analysis, the outcome is expressed as a health gain, usually in terms of QALYs (Quality Adjusted Life Years) gained or DALYs (Disability Adjusted Life Years) avoided. For conditions that have a significant fatal disease burden, years of life lost rather than QALYs or DALYs may be used as the outcome. A typical cost utility assessment might be presented as additional benefit in years of life gained. Cost utility analysis is not about the money.

Cost utility analysis estimates the health benefits gained as a result of an intervention.

33-9 What is cost benefit analysis?

In cost benefit analysis, the health outcomes are given a monetary value. This might include obvious financial benefits, such as the ability to work and earn a salary, but also less obvious ones such as the financial equivalent of being pain free.

Cost benefit analysis estimates the financial benefits as a result of an intervention.

33-10 Which is the most commonly used economic evaluation tool?

Cost effectiveness analysis is the most commonly used economic evaluation tool. Cost utility and cost benefit analyses seem appealing at first but in practice they are rarely used because so many assumptions are required to assign financial values and DALYs to health benefits that most people don’t really trust them. Most commonly, health economists will do cost effectiveness analyses, present disease specific health gains against financial cost, and let managers decide which outcomes should be prioritised.

Cost effective analysis is the most commonly use tool to decide how best to spend health funding.

33-11 Is the “value for money” argument controversial?

Yes. Using economic analysis to decide which treatments should be made available has been the dominant approach for health leaders, including the World Health Organisation, since the 1980s. Those who favour a human rights based approach have challenged it. There are also objections to the idea that somebody should be automatically denied treatment that is more costly.

Note

The main objection to economic analysis is that it assumes there is something “natural” and unchangeable in the prices charged by companies working in a free market.

Case study 1

A non-governmental organisation (NGO) was concerned about low immunisation rates in Northern India. One reason why mothers did not attend the clinic was that government nurses often did not turn up for work. The NGO decided to provide some well-publicised immunisation camps. In addition, in some villages, they provided a 1kg bag of lentils as an incentive for mothers bringing children for immunisation. Full immunisation rates in villages with no camps was 6%, it was nearly 20% in villages with immunisation camps, and nearly 40% in villages with immunisation camps and lentil incentives. Although each bag of lentils cost approximately US$1, the NGO was able to conclude that the incentive was highly cost effective.

1. Why might lentil incentives be highly cost effective when lentils only add to the cost of the programme?

The costing for providing the immunisation camps includes salaries, transport, refrigeration and so on. These are fixed costs that the NGO has to pay even if no children attend. The additional costs for each child – vaccine, consumables, and a bag of lentils for some, are quite small in comparison. Cost effectiveness is the cost per additional child immunised. If lentils mean that many more children attend the camps, then the cost per child goes down. In this case, the cost of immunising a child fell to about a half when the lentil incentive was used.

Case study 2

In 1993 the World Bank, using economic analysis, reported that DOTS (directly observed short course treatment for TB) was one of the most highly cost effective treatments in healthcare. As a result, the World Health Organisation declared TB a global emergency and DOTS became a well-funded, priority programme in global health. There were already a certain number of cases of multiple drug resistant (MDR) TB in all countries, but this required a longer treatment (2 years) with different drugs that at the time were quite expensive. With growing concern about MDR TB, the World Health Organisation issued guidelines stating that MDR TB was “too difficult to treat”, and that poor countries should focus on DOTS and not treat MDR TB.

1. Discuss the problems with the World Health Organisation’s response to MDR TB. Did the cost effectiveness approach have a negative effect on the global control of TB?

There are several problems with the World Health Organisation’s approach:

• Human rights activists would argue that it is inhumane to deny treatment to those with MDR strains of TB because it is more expensive to treat them than to treat patients with drug sensitive TB.
• TB is an infectious disease. Not treating MDR means that these strains were likely to become more dominant. This in fact happened, and MDR strains now make up about half of new cases of TB in parts of India.
• The cost effectiveness analysis assumes that the prices are fixed and unchangeable. Actually, many of the drugs used for MDR TB are older drugs such as streptomycin that are off patent and could have been manufactured more cheaply.

Unit 34: Programme monitoring and evaluation

Objectives

When you have completed this unit you should be able to:

• Explain the difference between monitoring and evaluation.
• Draw up a theory of change for a programme.
• Create a logistic framework for a programme.
• Analyse and explain to external stakeholders how and why a programme is working or not working.

34-1 What is monitoring and evaluation?

Monitoring and evaluation (M and E) is a performance assessment that compares what we want to happen with what actually happens. It shares many tools and ideas with the quality improvement processes, but it is a way of assessing the working of a whole programme rather than selected activities, issues or problems. While quality improvement processes tend to be used to track the progress of “in house” changes, the M and E approach tends to be used for reporting to external stakeholders, for example funders. It provides a common language for people inside and outside a programme to discuss what is working, what is not, and why.

34-2 What is monitoring?

Monitoring asks how well the programme is being implemented and whether the activities are being carried out as planned. The indicators chosen for monitoring tell us about the quantity or quality of programme activities.

Monitoring determines whether a programme is being implemented as planned.

34-3 What is evaluation?

Evaluation asks whether the activities have succeeded in achieving the aims of the programme. The indicators chosen for evaluation tell us about the quantity or quality of results.

Evaluation assesses whether the aims of a programme have been achieved.

34-4 How can you decide how a programme is expected to work?

In order to see if a programme is working as expected, it is necessary to first define and write down how it is expected to work. This method of deciding how change happens is usually called a Theory of Change. To make a Theory of Change, it is usually best to start with what you wish to achieve (the goal) and then work backwards to see what should happen first:

1. What is the overall goal of a programme? In M and E terms, this is usually called the impact you wish to have.
2. What is it that your particular programme will try to achieve in order to have that impact? This is called an outcome and there are often several outcomes in a programme.
3. What services or activities are needed to achieve this outcome? These are called outputs.
4. What resources you need to make these activities happen, these are the inputs.

Figure 34-1: Example of a very simple Theory of Change for a programme of workshops on contraception with the aim to prevent teenage pregnancies.

A Theory of Change explains how a programme is expected to lead to the desired impact.

34-5 What is a logical framework?

The Theory of Change defines how you expect the programme to work. It is now important to identify reasons why it may or may not work, and how you will know whether it is working. When this information is added to the theory of change, it is called a logical framework. The following steps are necessary:

• List assumptions and threats for the different parts of the programme (impact, outcomes, outputs, inputs): After deciding how the programme is expected to work, it is necessary to think about the conditions necessary in order to make it work, and the risks that may stop the programme being successful – these are called assumptions and threats. They should be listed.
• Choose indicators: It is necessary to measure whether each stage of the Theory of Change has been achieved. This means choosing appropriate indicators and identifying where the information will come from (sources of verification). This is the same as the measurement stage in quality improvement projects.

This information can now be placed into a table. This is the “logical framework”, sometimes shortened to “log framework” because it shows the logic of why the programme should work (see Table 34-1).

34-6 Why is it necessary to list assumptions and threats?

Listing assumptions and threats is important, because they need to be thought about, investigated and perhaps tackled before launching into a programme. For example, we might assume that teenagers will be willing to attend contraception workshops but find on talking to them that they would be embarrassed about attending the workshop because other students might draw conclusions about their sexual activity. The programme plan might then be changed to make the workshop part of the teenagers’ regular life skills class.

34-7 Do all managers and academics use an identical approach to the Theory of Change and log framework?

No. There are many different ways of producing a Theory of Change and a log framework, and it can be confusing when somebody else is using a slightly different version. People may even have different ideas about what is an input and what is an output. However the important principles are the same:

• Thinking about how and why a programme should work. Writing down how and why you believe the chosen inputs and activities will produce the change you want to see.
• Deciding to measure how well each stage is working and where this information will come from.
• Thinking about and writing down the reasons why each stage may fail and investigating or troubleshooting possible problems.

Case study 1

Sr Dlamini and her team have made good progress in their quality improvement project. Waiting times are down, the department is clean and the toilets have been fixed. However, she has noticed a large part of the caseload is uncontrolled, overweight type 2 diabetics who require admission. She decides to do something about it.

She thinks that adherence is a problem and would like to train community workers as adherence counsellors. However, the superintendent tells her there is no money to pay for this.

Fortunately, Dr Prudence has also become an improvement enthusiast and, while chatting in the tea room, she tells Sr Dlamini that there is a fund for small community projects offered by the social investment branch of a large company. Sr Dlamini calculates that she will need 5 full time community care workers to give adherence support, and that she would like to do this for 2 years to see if she can make a difference. She writes a grant proposal and the funders are enthusiastic, but request an M and E plan.

1. Help Sr Dlamni to state a desired impact.

An impact should be measureable and include a time frame. It should be ambitious but possible to achieve. An example of a good impact statement might be: “Reduce the numbers of admissions with uncontrolled diabetes by at least a half within 2 years”.

2. What would be suitable outcomes?

The outcomes are the things the programme needs to achieve in order to make the impact. Examples of suitable outcomes for this programme might include:

• Increase the number of patients attending all their appointments.
• Increase their understanding about how and when to take medication.
• Improve the glucose control in diabetics.

3. What would be suitable outputs?

The outputs are the programme activities. In this case, the number and type of consultations could be recorded.

4. What inputs need to be considered?

This would include training for the community workers, salary and transportation. If the adherence counselling will take place at St Jude’s Hospital, a suitable room must be identified. Information materials for the patients might also be necessary.

Activity: Using either your own theory of change, or the impact, outcomes, outputs and inputs listed above, draw up a logical framework. Include suitable indicators, assumptions and threats to the programme to improve diabetic care.

Table 34-1: An example of a log framework is shown for the teenage pregnancy workshop

	Objectives	Indicators	Sources of Verification	Assumptions/Threats
Impact	Lower rates of teenage pregnancy	Teenage pregnancy rates	Maternity records	Contraception is used effectively
Outcome	Increased use of contraceptive services by teenagers	Use of contraception by people under the age of 18	Clinic register	Teenagers must feel comfortable asking clinic staff for contraception
Outputs	20 workshops delivered to total of 200 teenagers Teenagers knowledgeable about how and why to access contraception	How many workshops happen How many students attend Students knowledge improved after workshop	Attendance register Before and after knowledge quiz	Teenagers will attend Teenagers will not be embarrassed Parents will not object Information is given in an understandable way
Inputs	People, place and resources available for 20 workshops	See action plan	See action plan	Schools will allow us to use their premises Budget is approved by district office