14 Measuring the performance of health services and programmes

Take the chapter quiz before and after you read this chapter.

Open chapter quiz

Close quiz

First time? Register for free. Just enter your email or cell number and create a password.

Close quiz

Contents

Unit 31: Types of performance measures in health services

Objectives

When you have completed this unit you should be able to:

31-1 Why is it necessary to measure performance in health systems?

Performance in health systems is measured to see whether standards are met. These may have been set by a local team to measure their own performance, or may have been set by managers or an outside organisation. This makes it possible to identify what is not working well and to put measures in place to address it.

31-2 What are standards?

There are different types of standards:

Sets of standards for structures and processes are created for the purpose of accreditation.

31-3 What is accreditation of a health facility mean?

Accreditation of a health facility means that the facility has met the standards set. Sometimes medical aid or insurance schemes require a facility to be accredited before they pay for their members to use it. The process of accreditation is important to maintain high standards of care.

Facilities can be accredited based on structure and process standards.

31-4 What are the approaches to measuring performance in health systems?

There are 3 basic approaches to performance monitoring in healthcare:

31-5 Who sets standards and decides whether a facility can be accredited?

Often an independent body, such as the Council for Health Service Accreditation of South Africa, will develop standards and offer accreditation. There are also core standards for health facilities published by the National Department of Health’s Office of Standards Compliance. Many accreditation bodies offer support and capacity building to help a facility become accredited.

A number of organisations will accredit facilities if standards are met.

31-6 Does accreditation improve the quality of care?

No. By itself the accreditation process does not improve quality of care but it ensures an environment in which quality care can happen. Standards of accreditation provide a target for staff and managers to aim at when they are trying to improve healthcare facilities.

31-7 Who sets targets for activities and results?

Targets can be set by a variety of managers, leaders or team members. For example:

The Sustainable Development Goals 2030 are international targets to improve the quality of life for all within a sustainable environment.

Note
The Sustainable Development Goals 2030 can be accessed on the UNDP’s website.

31-8 What is the basic tool for measuring performance?

The tool used for measuring performance is called an indicator. Indicators are markers that can be used to measure an activity or a result that is seen as important. A good indicator should:

The tool used to measure performance is called an indicator.

31-9 What is meant by clinical audit?

An audit is an inspection or assessment to make sure that something is up to standard. The term clinical audit is used in 2 different ways. It is sometimes used to describe a “report card” type of performance assessment, but the term “audit cycle” is also used to mean a quality improvement process.

An audit is an assessment to make sure that something meets a set standard.

Case study 1

A student wants to know how good the diabetic control is in a community clinic. He thinks of 3 possible indicators.

1. Which of the following do you think would be the best?

Option 1: The number of hospitalisations in each diabetic patient.
Option 2: A 6-hourly measure of serum glucose using a haemoglucotest over the course of the next month in each diabetic patient.
Option 3: The 6-monthly check of glycosylated haemoglobin (HbA1C).

Option 1 is not a valid indicator because the patient may have bad glucose control without being hospitalised, or he or she might be hospitalised for a reason unconnected to the diabetes. Option 2 would give valid information, but is very unlikely to be possible, unless the diabetic is already regularly checking their blood glucose levels accurately at home every 6 hours. There are very few diabetics in South Africa who do this. Therefore option 3 is the best answer, as a check of HbA1C is usually part of standard care. The HbA1C will show the glucose control over the previous 3 months and it should be easy to find this information from the patient records, or online laboratory results if these are available.

Unit 32: Adverse event monitoring

Objectives

When you have completed this unit you should be able to:

32-1 What is an adverse event?

An adverse event is a situation where:

The harm might be major harm or death or it could be something that health professionals consider quite minor, such as a patient having to spend the night in casualty instead of going home because they did not know where the patient transport services station was to be taken home .

An adverse event is a situation where part of the healthcare process has failed resulting in some sort of harm to the patient.

32-2 Why should we monitor adverse events?

If we can work out what has gone wrong during the management of the patient, we can identify failed activities and processes that we can feed into a quality improvement programme and try to stop the same problem occurring for other patients.

Adverse events can feed into a quality improvement cycle if the underlying causes are identified.

32-3 How can adverse events be monitored?

Adverse events can be informally monitored within the clinical unit or facility and then discussed at a mortality and morbidity (M&M) meeting.

Note
A mortality and morbidity meeting is also known as a deaths and complications (D&C) meeting.

There are also formal error reporting systems that are not widely used in South Africa. The disadvantage of these systems is that information goes “up” the chain of management and may not go back to the frontline clinicians who can make best use of it.

The World Health Organisation has a system for record review to monitor and list adverse events. It can show trends and compare countries. Because records contain incomplete information, this approach has a low to moderate reliability.

32-4 What is a “near miss”?

In a “near miss”, some part of the health system has gone wrong but the problem was picked up and managed without causing harm to patients. For example, if a patient was put on an operating list for an amputation of the wrong leg, but the theatre nurse realised this when the patient was handed over to her. It is also important to work out what went wrong and learn from near misses.

A “near miss” is something that nearly went wrong but the error was detected in time to prevent it happening.

32-5 How should mortality and morbidity meetings be run?

It is important that meetings are held in a “no blame” setting. If somebody feels they will be criticised or singled out in the meeting, they may hide their complications or avoid attending. This means that others will not have the benefit of learning from them and you are unlikely to get to the underlying causes of the problem. Always ask “how could we have done better” rather than how could an individual have done better. It may be necessary to put these “ground rules” in writing.

Mortality and morbidity meetings must be held in a “no blame” setting.

All deaths should be listed, and all staff members should be allowed and encouraged to put complications, adverse events and near misses on the agenda for discussion as well:

A mortality and morbidity meetings is one of the best opportunities to learn especially if the whole group is asked to identify the mistake and say how it could be avoided in future.

A mortality and morbidity meeting is an excellent opportunity to learn from mistakes and find ways of avoiding them.

32-6 How can the underlying causes be identified when something has gone wrong?

It is important to try and identify the underlying causes of problems. They may not be obvious. For example, a confused patient may be given a drug to which they are allergic, but this could be because their records were missing and that in turn might be because of an unsatisfactory filing system. The best way of uncovering the underlying causes of a problem is by brainstorming with the relevant people in your team during the mortality and morbidity meeting. There may be an obvious underlying cause and solution. If not, there are tools such as the “5 whys” that you can use to get to the root of the problem. This is sometimes called a “root cause analysis”.

32-7 What are the 5 whys?

Many have had the experience of trying to explain something to a young child, and found the response to every explanation to be “but why?” until the adult comes to a point when they have to say: “because I say so”. This process of asking “why” until you cannot come up with another response is a useful approach to root cause analysis. The approach is sometimes called the “5 whys” because it is said that you have to ask “why?” 5 times to get to the root problem. Of course, you may ask “why?” fewer or more than 5 times. Here is an example of 5 whys:

  1. Q: Why didn’t Mr Felies get his antibiotic this morning? A: Because Sr A didn’t give it to him.

  2. Q: Why didn’t Sr A give Mr Felies his antibiotic? A: Because there wasn’t any Augmentin in the trolley.

  3. Q: Why wasn’t there any Augmentin in the trolley? A: Because it was out of stock at the pharmacy.

  4. Q: Why was Augmentin out of stock at the pharmacy? A: Because Mr P didn’t order it.

  5. Q: Why didn’t Mr P order it? A: Because he broke his leg and is on sick leave and nobody has replaced him.

It is possible take this example further and ask why Mr P broke his leg, or why management did not find a replacement, but after asking “why” 5 times we are getting close to the underlying reason why Mr Felies did not receive his antibiotic. We can also identify possible solutions to prevent similar stock outs that may have an impact on patient care.

The “5 whys” method helps to examine in detail the many factors which lead to an adverse event.

Figure 32-1: Simplified example of a “why tree” for overcrowding of an emergency unit over weekends.

Figure 32-1: Simplified example of a “why tree” for overcrowding of an emergency unit over weekends.

32-8 What is a why tree?

Sometimes the answer to a question raises more than one other question and this means that other chains of “whys” branch off from the original question. For example, in question 2 in the example above, one might also ask why Sr A didn’t ask the medical officer for an alternative prescription and then find out several other interesting things. When answers to the question “why?” branch in this way, it is useful to draw the problem as a tree. When using this approach for a real life problem, it is most effective to brainstorm the “whys” with others.

In real life, a final “why tree” will probably be a lot more complicated. When the exercise has been completed and the problem has been picked apart as much as possible, all the underlying reasons need to be addressed in some way (see Table 32-1).

Table 32-1: Why is casualty overcrowded at the weekend?

Health Team Factors Health Service Factors Social and Economic
Some senior staff do not do weekend shifts Primary care clinic not open at weekend Law enforcement does not like to enter community because relations with community are poor
Clinical staff are not coding their caseload Employers not paying sick leave
Employment is informal/casual
Low standards at local secondary school

Organising the root causes in this way helps to make sense of a complex problem. The causes can also be scored for ease and impact, so that easy to implement high impact changes can be prioritised.

32-9 What does “patient safety” mean?

Patient safety refers to the design and use of systems to reduce adverse events. Safety in health systems is often compared unfavourably with safety in the airline industry, which has a much more stringent approach to the prevention of accidents. This is not completely fair because an aeroplane never has to make an emergency take off or to carry more passengers than it has seats for. However, there are still many improvements that health systems can and should make.

Patient safety means designing and using systems that reduce the risk of adverse events.

32-10 How can systems be designed to reduce adverse events?

Many adverse events occur because somebody has done something wrong. Making an error is part of being a human, and because healthcare delivery depends so much on human choices and actions, there will always be errors. The important thing is to design systems so that errors can be caught before they cause harm. One way of thinking about systems to catch errors is the “Swiss Cheese Model”. The various steps in patient management are seen as slices of Swiss cheese – they are each designed to protect the patient, but each step has holes (errors) that could allow harm to happen. When the holes are lined up, all defences have failed and harm can result (see Figure 32-2). Patient safety involves redesigning a slice, or inserting a new protective slice.

For example: We wish to make sure that urgent cases coming to casualty are managed as quickly as possible. The process for admitting patients to casualty is as follows:

  1. The patient registers at front desk.
  2. The front desk clerk takes the folder to triage.
  3. The triage nurse calls the patient and makes observations to determine how urgent the situation is.
  4. Urgent patients are taken to one of the clinical areas and the doctor is informed.
  5. A nurse is assigned to the urgent clinical areas to monitor all patients there.

What happens in this situation when the ambulance brings in an urgent patient and the front desk clerk has gone to search for a missing folder? The Emergency Medical Services (EMS) provider may take the patient straight to an urgent clinical area. But what if the nurse from that area is taking a tea break? If the Emergency Medical Services provider leaves the patient in an urgent clinical area because they must attend to another call, it is possible that the patient will just lie there and deteriorate. All the holes in the system have lined up and led to patient harm.

One possible extra “slice” to add, is a policy that the Emergency Medical Services provider must hand over patients to a qualified staff member. The sister in charge at the time can be clearly identified and be tasked with taking handovers if no trained staff are in the clinical area at the time. Trollies or beds containing patients who have been delivered by the Emergency Medical Services provider and bypassed triage could also be clearly identified with a board or a ribbon.

Figure 32-2: Successive layers of defences, barriers and safeguards. All the holes in the system have lined up and led to patient harm.

Figure 32-2: Successive layers of defences, barriers and safeguards. All the holes in the system have lined up and led to patient harm.

Case study 1

It is 3am in St Jude’s casualty department. Having worked all day and all night, Dr Prudence has just placed an intercostal drain in a patient with a stabbed chest. Shortly afterwards, a staff nurse asks her why the drain is in the left side when the stab is on the right. Horrified, Dr Prudence realises that the X-ray was wrongly labelled and she has put the drain in on the wrong side.

1. What should she do?

The right thing to do is go straight to the patient, explain what has happened and put it right (in this case, unfortunately it would mean putting in another drain on the correct side). She should make good notes, including her conversation with the patient, and report it to her senior as soon as possible.

She reports the event to Dr Retief, the family physician, the following morning. Dr Prudence is very embarrassed and worried.

2. Should Dr Retief discuss the event at the next Mortality and Morbidity meeting?

Yes. The purpose of a Mortality and Morbidity meeting is for everybody to learn from what has gone wrong. There are ways of discussing mistakes in ways that do not make it worse for the person who has had an event.

Unit 33: Cost analysis

Objectives

When you have completed this unit you should be able to:

33-1 Why should clinicians know about cost analysis in health systems?

Healthcare costs rise as new innovations become available and public expectations of what can and should be done increase. Because there is a fixed amount of money available in the budgets of health facilities and health services, decisions may need to be made about what the available funds should be spent on. Clinicians need to be aware of costing issues so that they can:

33-2 How are healthcare costs analysed?

It is useful to understand:

Calculating the costs for a procedure is called costing.

33-3 What is included in a costing?

There are 4 categories of healthcare costs. These are:

What is included depends on the “perspective” of the costing.

33-4 What is the meaning of the “perspective” of the costing?

The “perspective” of the costing refers to whose money is being spent. Common costing perspectives are:

Clinicians do not need to be able to do a costing, but it is useful to understand that different cost items are important to different role players, including the patient.

The perspective of costing refers to whose money is being spent.

33-5 What is a patient day equivalent?

The patient day equivalent (PDE) is a measure commonly used by managers in South Africa to assess the efficiency of a hospital. The total cost of running the hospital is divided by the number of inpatient-days plus half the number of day cases plus a third of outpatient of emergency consultations. This gives an average cost per patient seen. The patient day equivalent is used by managers to compare running costs for hospitals of similar size with similar caseloads. It is not a perfect measure because it does not show the effectiveness of care. For example, a patient who lies in a hospital bed without receiving much attention for a month uses the same number of patient day equivalent at a lower cost than 15 patients who occupy a bed for 2 days each but are quickly investigated and treated.

The patient day equivalent is a measure used to assess the efficiency of a hospital.

33-6 What tools are used to determine “value for money” in health spending?

By itself, a costing does not indicate whether money was well spent, or whether it could have been better spent elsewhere. For that it is necessary to measure not only the cost but also the outcomes. There are 3 different economic analysis tools (“economic evaluations”) that are used to compare the value for money of different interventions. They differ in the way that the outcomes are presented. They are:

33-7 What is cost effectiveness analysis?

The outcomes measured in cost effectiveness analysis are direct outcomes, or the things we want to achieve with the intervention that are specific to the disease. For example, TB treatment completion rates or hernia recurrence rates. A cost effectiveness assessment might be presented as cost per additional case prevented, or cost per additional child immunised. With cost effective analysis one estimates what will be achieved for the money spent.

Cost effectiveness analysis measures the financial cost of direct outcomes of an intervention.

33-8 What is cost utility analysis?

In cost utility analysis, the outcome is expressed as a health gain, usually in terms of QALYs (Quality Adjusted Life Years) gained or DALYs (Disability Adjusted Life Years) avoided. For conditions that have a significant fatal disease burden, years of life lost rather than QALYs or DALYs may be used as the outcome. A typical cost utility assessment might be presented as additional benefit in years of life gained. Cost utility analysis is not about the money.

Cost utility analysis estimates the health benefits gained as a result of an intervention.

33-9 What is cost benefit analysis?

In cost benefit analysis, the health outcomes are given a monetary value. This might include obvious financial benefits, such as the ability to work and earn a salary, but also less obvious ones such as the financial equivalent of being pain free.

Cost benefit analysis estimates the financial benefits as a result of an intervention.

33-10 Which is the most commonly used economic evaluation tool?

Cost effectiveness analysis is the most commonly used economic evaluation tool. Cost utility and cost benefit analyses seem appealing at first but in practice they are rarely used because so many assumptions are required to assign financial values and DALYs to health benefits that most people don’t really trust them. Most commonly, health economists will do cost effectiveness analyses, present disease specific health gains against financial cost, and let managers decide which outcomes should be prioritised.

Cost effective analysis is the most commonly use tool to decide how best to spend health funding.

33-11 Is the “value for money” argument controversial?

Yes. Using economic analysis to decide which treatments should be made available has been the dominant approach for health leaders, including the World Health Organisation, since the 1980s. Those who favour a human rights based approach have challenged it. There are also objections to the idea that somebody should be automatically denied treatment that is more costly.

Note
The main objection to economic analysis is that it assumes there is something “natural” and unchangeable in the prices charged by companies working in a free market.

Case study 1

A non-governmental organisation (NGO) was concerned about low immunisation rates in Northern India. One reason why mothers did not attend the clinic was that government nurses often did not turn up for work. The NGO decided to provide some well-publicised immunisation camps. In addition, in some villages, they provided a 1kg bag of lentils as an incentive for mothers bringing children for immunisation. Full immunisation rates in villages with no camps was 6%, it was nearly 20% in villages with immunisation camps, and nearly 40% in villages with immunisation camps and lentil incentives. Although each bag of lentils cost approximately US$1, the NGO was able to conclude that the incentive was highly cost effective.

1. Why might lentil incentives be highly cost effective when lentils only add to the cost of the programme?

The costing for providing the immunisation camps includes salaries, transport, refrigeration and so on. These are fixed costs that the NGO has to pay even if no children attend. The additional costs for each child – vaccine, consumables, and a bag of lentils for some, are quite small in comparison. Cost effectiveness is the cost per additional child immunised. If lentils mean that many more children attend the camps, then the cost per child goes down. In this case, the cost of immunising a child fell to about a half when the lentil incentive was used.

Case study 2

In 1993 the World Bank, using economic analysis, reported that DOTS (directly observed short course treatment for TB) was one of the most highly cost effective treatments in healthcare. As a result, the World Health Organisation declared TB a global emergency and DOTS became a well-funded, priority programme in global health. There were already a certain number of cases of multiple drug resistant (MDR) TB in all countries, but this required a longer treatment (2 years) with different drugs that at the time were quite expensive. With growing concern about MDR TB, the World Health Organisation issued guidelines stating that MDR TB was “too difficult to treat”, and that poor countries should focus on DOTS and not treat MDR TB.

1. Discuss the problems with the World Health Organisation’s response to MDR TB. Did the cost effectiveness approach have a negative effect on the global control of TB?

There are several problems with the World Health Organisation’s approach:

Unit 34: Programme monitoring and evaluation

Objectives

When you have completed this unit you should be able to:

34-1 What is monitoring and evaluation?

Monitoring and evaluation (M and E) is a performance assessment that compares what we want to happen with what actually happens. It shares many tools and ideas with the quality improvement processes, but it is a way of assessing the working of a whole programme rather than selected activities, issues or problems. While quality improvement processes tend to be used to track the progress of “in house” changes, the M and E approach tends to be used for reporting to external stakeholders, for example funders. It provides a common language for people inside and outside a programme to discuss what is working, what is not, and why.

34-2 What is monitoring?

Monitoring asks how well the programme is being implemented and whether the activities are being carried out as planned. The indicators chosen for monitoring tell us about the quantity or quality of programme activities.

Monitoring determines whether a programme is being implemented as planned.

34-3 What is evaluation?

Evaluation asks whether the activities have succeeded in achieving the aims of the programme. The indicators chosen for evaluation tell us about the quantity or quality of results.

Evaluation assesses whether the aims of a programme have been achieved.

34-4 How can you decide how a programme is expected to work?

In order to see if a programme is working as expected, it is necessary to first define and write down how it is expected to work. This method of deciding how change happens is usually called a Theory of Change. To make a Theory of Change, it is usually best to start with what you wish to achieve (the goal) and then work backwards to see what should happen first:

  1. What is the overall goal of a programme? In M and E terms, this is usually called the impact you wish to have.
  2. What is it that your particular programme will try to achieve in order to have that impact? This is called an outcome and there are often several outcomes in a programme.
  3. What services or activities are needed to achieve this outcome? These are called outputs.
  4. What resources you need to make these activities happen, these are the inputs.

Figure 34-1: Example of a very simple Theory of Change for a programme of workshops on contraception with the aim to prevent teenage pregnancies.

Figure 34-1: Example of a very simple Theory of Change for a programme of workshops on contraception with the aim to prevent teenage pregnancies.

A Theory of Change explains how a programme is expected to lead in the desired impact.

34-5 What is a logical framework?

The Theory of Change defines how you expect the programme to work. It is now important to identify reasons why it may or may not work, and how you will know whether it is working. When this information is added to the theory of change, it is called a logical framework. The following steps are necessary:

This information can now be placed into a table. This is the “logical framework”, sometimes shortened to “log framework” because it shows the logic of why the programme should work.

34-6 Why is it necessary to list assumptions and threats?

Listing assumptions and threats is important, because they need to be thought about, investigated and perhaps tackled before launching into a programme. For example, we might assume that teenagers will be willing to attend contraception workshops but find on talking to them that they would be embarrassed about attending the workshop because other students might draw conclusions about their sexual activity. The programme plan might then be changed to make the workshop part of the teenagers’ regular life skills class.

34-7 Do all managers and academics use an identical approach to the Theory of Change and log framework?

No. There are many different ways of producing a Theory of Change and a log framework, and it can be confusing when somebody else is using a slightly different version. People may even have different ideas about what is an input and what is an output. However the important principles are the same:

Case study 1

Sr Dlamini and her team have made good progress in their quality improvement project. Waiting times are down, the department is clean and the toilets have been fixed. However, she has noticed a large part of the caseload is uncontrolled, overweight type 2 diabetics who require admission. She decides to do something about it.

She thinks that adherence is a problem and would like to train community workers as adherence counsellors. However, the superintendent tells her there is no money to pay for this.

Fortunately, Dr Prudence has also become an improvement enthusiast and, while chatting in the tea room, she tells Sr Dlamini that there is a fund for small community projects offered by the social investment branch of a large company. Sr Dlamini calculates that she will need 5 full time community care workers to give adherence support, and that she would like to do this for 2 years to see if she can make a difference. She writes a grant proposal and the funders are enthusiastic, but request an M and E plan.

1. Help Sr Dlamni to state a desired impact.

An impact should be measureable and include a time frame. It should be ambitious but possible to achieve. An example of a good impact statement might be: “Reduce the numbers of admissions with uncontrolled diabetes by at least a half within 2 years”.

2. What would be suitable outcomes?

The outcomes are the things the programme needs to achieve in order to make the impact. Examples of suitable outcomes for this programme might include:

3. What would be suitable outputs?

The outputs are the programme activities. In this case, the number and type of consultations could be recorded.

4. What inputs need to be considered?

This would include training for the community workers, salary and transportation. If the adherence counselling will take place at St Jude’s Hospital, a suitable room must be identified. Information materials for the patients might also be necessary.

Activity: Using either your own theory of change, or the impact, outcomes, outputs and inputs listed above, draw up a logical framework. Include suitable indicators, assumptions and threats to the programme.

Table 34-1: An example of a log framework is shown for the teenage pregnancy workshop

Objectives Indicators Sources of Verification Assumptions/Threats
Impact Lower rates of teenage pregnancy Teenage pregnancy rates Maternity records Contraception is used effectively
Outcome Increased use of contraceptive services by teenagers Use of contraception by people under the age of 18 Clinic register Teenagers must feel comfortable asking clinic
staff for contraception
Outputs 20 workshops delivered to total of 200 teenagers
Teenagers knowledgeable about how and why to access contraception
How many workshops happen
How many students attend
Students knowledge improved after workshop
Attendance register
Before and after knowledge quiz
Teenagers will attend
Teenagers not be embarrassed
Parents will not object
Information is given in an understandable way
Inputs People, place and resources available for 20
workshops
See action plan See action plan Schools will allow us to use their premises
Budget is approved by district office