Propositional Evaluation & Outcomes Assurance by Andrew Hawkins is licensed under CC BY 4.0

 

FAQs

Your Questions Answered

Do I need to totally change my approach to do a Propositional Evaluation?

No, for many practicing evaluators Propositional Evaluation simply provides a theoretical justification for what they already do and provides a more logical structure for the reporting of findings. You might need to adjust your program logic diagrams to make use of 'Propositional Design Logic' and treat the diagram as a 'causal package' like a recipe for an outcome, rather than a 'chain of cause and effect' or 'model for a theory of change'. You may have to think more about what is 'necessary' and for what the program must be 'sufficient'. You may also need to pay more attention to assumptions in your diagrams because these are just as important as what the intervention does for generating intended outcomes - but most of your methods and analysis will not need to change.

Will suggesting Propositional Evaluation 'freak out' my funders or the commissioners of evaluation?

Very few people concerned with the design and delivery of effective programs and interventions will find Propositional Evaluation objectionable. The same quantitative and qualitative methods that are commonly used in the social sciences can be used - from experimental design through to lived experience. The only requirement is clarity on how certain facts obtained using these methods provide evidence to support specific claims people make about a program or intervention. For example, does an experimental design like a randomised controlled trial provide facts about causal processes and 'what works', or is it a good method to rule out alternative explanations for an observed change and provide facts about 'what happened' as a result of the intervention? 


Generally, people who are not technical social science experts will love the simplicity and rigour Propositional Evaluation provides for deliberation and decision making. It takes the power out of the hands of a professional class of evaluators for making overall judgements. It is easily understandable and focuses on the same sorts of claims and evidence that everyone thinks and uses in their everyday discussions. It invites policy makers, practitioners and citizens to contribute to the evaluation about what makes this a good idea. 


Be sure to see the answers to other FAQs below about when to use Propositional Evaluation, and when not to use it. People who approach the evaluation of public policy programs as a domain for developing and testing scientific theories may not appreciate the simplicity of Propositional Evaluation and its focus on applied social science and rational action.

When should I use Propositional Evaluation?

Propositional Evaluation is useful when you are designing a new policy or program. It can help you to systematically think through and discuss with others what will actually be necessary, what assumptions you need to rely on, and what your policy or program may actually be sufficient for achieving.

Propositional Evaluation is also very helpful when you are trying to deliver a policy or program and you want to make sure it is on track. This makes it an ideal approach for performance monitoring and managing the risk of program failure. It helps you to identify what needs to be monitored, that is, whether the necessary conditions did in fact materialise and how the intended outcomes (or sufficient conditions) are changing.

How is this different to other forms of evaluation?

Propositional Evaluation is different from evaluation undertaken within a scientific or research paradigm, such as experimental and realist evaluation. These forms of evaluation focus on the internal and external validity of knowledge claims about the value of a course of action. They aim to identify 'what works' in a scientific sense - that is, by establishing the validity of causal claims. These claims are made about whole programs in the experimental tradition, or about the hidden or latent structures and causal mechanisms that if activated will regularly generate an outcome in the realist tradition.

Propositional Evaluation is grounded in the idea that practical or applied knowledge is necessary for sustainable change in the real world in preference to abstract causal claims about hard-to-specify interventions whose past impacts may not generalise to other times and places. As Stephen Toulim wrote in Return to Reason we want to avoid 'theories that apply everywhere in general and nowhere in particular.' Scientific research is about establishing facts and developing theories about what things there are in the natural world and how they behave. Public policy and evaluation are concerned with action and manipulating the world. The scientific method has a place in public policy but logic and ethics are the means by which we can determine the value (or evaluate) a rational course of action; before, during, and after delivery.

Propositional evaluation is about rational action and is more closely aligned to complexity science and a systems perspective. On this understanding, causal relationships are usually only apparent in retrospect and do not necessarily repeat. This is because the world is full of interdependent actors who change their behaviour based on the behaviour of others. This means evaluation of 'what happened' is rarely an answer to 'what works'? 

Propositional Evaluation was born out of a critique of the more 'scientific' or 'positivist' approaches to the idea of causal impact and the ideas of validity that dominated the debate about high-quality evaluation in the 20th Century. These are not the only approaches to evaluation. Constructivist approaches that focus less on 'valid' knowledge and more on meaning can be useful. Related to this tradition there are approaches that focus less on the act of forming a judgment of value and more on the use to which evaluation is put, such as developmental and empowerment evaluation whose goal is to provide people with a process for surfacing values and putting knowledge to work.

Propositional evaluation can work with all these traditions, founded as it is in logic, ethics, and deliberation. It is a tool for rational, deliberative and collective decision-making in a world of great complexity where there is much uncertainty about the value of any action.

Should I focus more on monitoring or evaluation?

In Propositional Evaluation there is no real distinction between monitoring and evaluation. The purpose is the same, to test whether conditions are being brought about and to identify, manage or mitigate design and operational risks to program failure.

The difference is simply a question of granularity. Monitoring focuses on readily available longitudinal data and usually tracks whole conditions. Evaluation might need more in-depth cross-sectional data to understand the reasons why certain conditions are or are not necessary, and why certain combinations are or are not sufficient and how things might be improved. 

Monitoring or evaluation will be more useful depending on how complicated or complex is the intervention, the nature of the risks in any given plan, and the time and resources available for probing the system for answers.

Are there any methods best suited to Propositional Evaluation?

All the same methods commonly used in program evaluation can be used with Propositional Evaluation. The difference is the stronger focus on the logic of the design and on whether the program is necessary and sufficient rather than contributing to knowledge base of what works.

Qualitative Comparative Analysis (QCA) is a method for causal inference that is particularly relevant for Propositional Evaluation. QCA focuses on identifying 'necessary' and 'sufficient' conditions for another condition or outcome. 

When should I NOT use Propositional Evaluation?

You should never use a form of evaluation if it will not be helpful to answering your evaluation questions or informing decisions.


If your primary concern is to establish and test stable cause and effect relationships - whether that is about a whole program, or about a component of your program other methods such as experimental design, or realist impact evaluation will be more useful. Similarly, if your only goal is to measure the effect size of intervention then Propositional Evaluation will not be the most useful approach - experimental or quasi-experimental methods will be more appropriate in these instances.


Propositional Evaluation is great for working out where things are going wrong, and which parts of your program are not working - but if you want fully detailed and scientific answers to questions about why they are not working, a Realist approach will be more useful. This will help you get inside the 'black box' to understand the generative mechanisms and contexts that are, or are not, leading to the anticipated outputs and outcomes for different people. Propositional Evaluation focuses on the conditions that your program is attempting to bring about, it includes the reasons why it should work but is not designed to develop theories about psychology and sociology.

Finally, you should not expect a Propositional Evaluation to work out what you should be doing. Systems Evaluation and methods such as Root Cause Analysis are better suited to the task of understanding the nature of a particular system and finding the weaknesses in a system for which an intervention is needed. Propositional Evaluation is less about the creative act of an idea for an intervention. This is the place of insight or the 'Eureka' moment. Propositional evaluation is an approach for testing whether what you propose to do is a sound proposition. It provides a guide for management and monitoring whether, in reality, it is a well-grounded proposition, as determined by the empirical data that you are collecting.

How does Propositional Evaluation think about modelling and forecasting?

There is great potential for a long-term alliance between Propositional Evaluation and Bayesian causal modeling.

Propositional Evaluation can guide the search for necessary and sufficient conditions for an outcome. Realist, systems, and complexity science data may be used to ensure any enduring relationships between these conditions are appropriately modeled.

The utility of modeling in the social world is still fairly nascent - it may work ok with the weather not very well for the economy, and not at all for the granular domain of human interaction. We don't know, and we don't know if we should know if modeling here is possible or permissible. 


Propositional Evaluation is confident that if any modeling is going to work, it will need to focus on the necessary and sufficient conditions for a change in conditions, rather than on abstract causal theories.

But not everything is logical, why such a narrow focus on logic?

It is difficult to conceive of any other option for democratic and collective deliberation about the value of a course of action. Personal conviction? A higher moral authority? Vested interests? There are many ways to 'know' what to do in the world, but I am unaware of any other way to do anything in a truly democratic manner other than through rational deliberation.

This does not mean a naive assumption that politicians or public servants are driven only by the logic of the public good without regard for their career, that would be illogical. If we can create incentive systems for more rational action and hold our plans to a more rational evaluation then we will be heading in the right direction.


And of course, it is only logical to assume that humans will behave emotionally and seek to justify their actions after the fact regardless of the true reasons for action. Interesting research has shown how reasoning may have evolved as a means of 'self-talk and post hoc justification. Other research has shown that while individuals are prone to many biases and fallacies, reasoning when done in small groups is often very effective.


Nothing about Propositional Evaluation suggests a humorless or unemotional and technocratic world-view. Propositional Evaluation is about getting things done. Action is just one part of life. Being free to make mistakes, developing deep emotional bonds, exploring our place in the universe, knowing that we don't know are all valuable and sensible - we don't need propositional logic to justify them (although I would suggest you could!). But if you want to get something done in the world, and especially if it involves other people, it will always be more likely to succeed if it is logical.