Is it right to say ‘Goal-Free Evaluation Methods’?

[Hur Hassnain]

Is it right to say ‘Goal-Free Evaluation Methods’ or should it be rather ‘Objective-Free’?

This is a question put forward by our esteemed evaluation colleague Michelle Garred to the IEAC community.

Please share your thoughts while I invite Michelle to expand on this further if possible.

• This topic was modified 3 years, 4 months ago by Hur Hassnain.

[Michelle Garred]

Dear colleagues,

I’m an enthusiastic practitioner and proponent of ‘goal-free methods’ (including Outcome Harvesting, QuIP etc). As Hur Hassnain mentioned below, I increasingly wondering whether the term ‘goal-free’ has outlived its original usefulness, and may require an update. (Or, to be even more daring, perhaps the underlying concept could also use clarification and refreshing.)

I say this for several reasons:

1. None of the programs to which I apply ‘goal-free’ methods in the peacebuilding space (broadly defined) are without goals. They all have a goal, albeit sometimes a loosely defined one. We do in fact center that goal during the evaluation process; if not, how would we discern whether or not the program is moving and influencing in the intended direction? However those programs are often without specific objectives, or else they have objectives that are set aside for purposes of a particular evaluation exercise. Hence the potentially awkward alternative term ‘objective-free evaluation.’

2. The term ‘goal-free’ doesn’t allude in any helpful way to the reasons why such methods are currently used. As I understand it, the original reason for using goal-free methods was largely to avoid confirmation bias. More recently, the common reasons center around the impossibility or undesirability of accurately predicting results in advance in certain settings. This applies to social change efforts in highly complex fast-changing contexts and also, less commonly, to programs that are driven by iterative participant-led planning. The alternative term that I prefer here is ’emergent’- the program results are emergent, and so are the evaluation methods.

3. The term ‘goal-free’ seems to come across as alien, rigid and unrealistic with most audiences. Only the ‘evaluation nerds’ among us know what it means. Speaking for myself, this is a very real problem as I try to promote awareness of the benefits and limitations of goal-free methods among my clients and constituencies. I find myself avoiding the term ‘goal-free’ and talking instead about ’emergence.’ I don’t think I’m alone in this. I believe that the potentially outdated terminology may actually be holding back the development of the field.

What do others think about this challenging issue?

With thanks,
Michelle Garred
Ripple – Peace Research & Consulting LLC

[Michelle Garred]

Dear colleagues,

Following up the below, I recently had the privilege of taking an eval theory class with Melvin Mark at the Evaluators’ Institute. We discussed this issue, and came to the shared conclusion that Outcome Harvesting should NOT be called ‘goal-free.’

The logic is as follows: while OH is often called ‘goal-free,’ that is misleading because the underlying reasons for going ‘goal free’ are very different than the reasons underlying the original use of that term. Scriven coined the term ‘goal free’ to address bias reduction. Wilson-Grau developed OH to address contextual complexity. The theoretical underpinnings do not align, so it’s best to maintain clarity by using different terms.

As a result, I continue to call OH ’emergent.’ This is largely my own invented term. I still see a need for consensus within the field, so I’m eager to hear what others think.

Cheers,
Michelle Garred, PhD
Ripple Peace Research & Consulting LLC

[ICD]

Checking to see whether or not, or the extent to which, an intervention’s intended results were achieved is performance audit, review, research, etc. Evaluation is about valuing, independently of what the stated goals may be. Evaluation is, among other things, finding out what the intervention is actually doing and using a valuing frame that draws on those of the claims and rights holders, social and climate justice, considerations relative to the climate and bio-diversity breakdown, etc.
So one way to think about the “goal free” question is as an attribute of independence, of which there are a few. It is also useful to distinguish between independence, impartiality and autonomy of evaluative considerations, including evaluative thinking and reasoning.
Can this be done in practice when commissioners of evaluations hold most of the cards and are by definition partial? Yes, sometimes, if you have the stature & gravitas and/or are independently wealthy.
One of the raison-d’être of the Academy is to support and catalyze what it calls “evaluator directed” evaluation, meaning evaluation that is truly independent.
Ian

[Author]

Posts