Evaluation designs: participatory approaches and comparative designs

If we are to explore service experiences, relationships and improvement suggestions a process approach, such as participatory action research (Smith et al 2010; Adelman 1993), may be very useful. It was extensively practiced in CoSIE with positive results. Action research not only allows to include relevant stakeholders in joint reflection of their practices and analysis of possible solutions, but also take a lead in how power relations may be restructured.

To be able to determine whether co-creation has had any impact it requires comparisons and appropriate designs. Either design allowing a comparison of intervention outcomes for groups affected by co-creation against a similar service context with no co-creative attempts (so-called quasi-experimental counterfactual design) or comparing against a context before the intervention, that is comparing the ‘before’ and ‘after’ service status, relationships and experiences. Yet, cultural change that is often implied by co-creation takes effort and time, and we have to be aware at what point measuring outcomes might be relevant and when they are still pre-mature.

• In CoSIE, only one pilot (UK) was able to effect a quasi-experimental counterfactual evaluation design within the project timeframe. The conditions that made this possible were the short length of the pilot intervention, a tightly scoped pilot within an existing service, and availability of reliable data on an equivalent ‘untreated’ group.
• Other pilot results were extracted from case studies by employing before and after comparative approach to tracing co-creation based outcomes, and relying on participants’ experiences expressed in narratives (including digitally curated Community Reporter stories), for example the Dutch pilot.

Arguably, personal narratives of lived experience are a kind of evidence uniquely aligned with co-creation (see Trowbridge and Willoughby (forthcoming) and Geelhoed, 2020) and a valuable source of evidence for those working closely with targeted (“hard to reach”) groups, while public authorities tend to favour measured, easily comparable outcomes.

Below you find a conversation on methods for evaluating co-creation