From ‘systematic’ to ‘systemic’ research and policy analysis – progressing the application of systems approaches to policy evaluations at a European Union research agency (Eurofound)

Abstract

This study explores the relevance and applicability of using systems approaches to enhance policy evaluation projects conducted by Eurofound(European Foundation for the Improvement of Living and Working Conditions) aa an exploratory case study in 'action research' mode.

This European Union (EU) agency is mandated to support European policymakers in their endeavours to improve the working and living conditions in Europe, by providing “scientifically sound and unbiased, high quality information” .  

The key question of this exploration was: How can evaluation of relevant policies be accomplished as part of Eurofound’s role, and give justice to the complexities of the policies assessed and multiple perspectives and stakes involved?

The overall aim of this research was to investigate the effectiveness of current approaches and methods in policy evaluation activities of complex social policies and phenomena undertaken by Eurofound, and to test the ‘systemic desirability and cultural feasibility’ (Checkland, 1990) of moving beyond a ‘first-order science’ research tradition within Eurofound to advance opportunities for applying systems concepts and approaches in these activities. 

Methodology

The action research consisted in a specially designed systemic enquiry designed by the author, and conducted jointly with research staff of the agency as they engaged in the implementation of Eurofound policy analysis (research) project consisting of a series of analyses and workshops using different systems approaches (elements of SSM and CSH). This systemic enquiry was applied to a selected case study: a then current Eurofound research project ‘Delivering public services: a greater role for the private sector? Hospital services’ (Eurofound, 2015a, p. 39). This project aimed to analyse the ‘implications of privatisation for access, quality and effectiveness of services’ in the hospital sector across EU Member States. This situation features a number of ‘complex’ characteristics. To test the usefulness and applicability of selected systems approaches was the aim of the action research implemented in this case study.

Conclusions

   The experience from this action research applied to the ‘hospital services’ case has shown that the standard approach is not always sufficient for a deeper understanding of a situation to be researched at the start of a project, and can usefully be complemented and enhanced by systems approaches.

The experience and validation interviews with researchers in Eurofound suggested that there could be real value for research and policy evaluation projects in Eurofound to embed systemic inquiry streams into the overall project design, under some conditions:

• For projects with wide and fuzzy scope (boundary explorations to inform research questions);
• Where new territory is explored (exploratory social learning and scoping);
• In complex policy situations with diverse and multiple stakeholders (interrelationships and multiple perspectives).

Whilst it would be ‘systemically desirable’ to broaden a systemic approach to researching similar policies with shared characteristics, there are institutional barriers to the ‘cultural feasibility’, as the mainstream way of conducting research continues to dominate and imposes important constraints to such opportunities. These consist in short research project cycle times, and pressures to deliver results fast, leaving little time and scope for reflexivity and exploratory methods. Additionally, introducing systems approaches into policy evaluation projects exposes paradigmatic and epistemic tensions. The default way of doing research and evaluation projects ‘systematically’ sits uncomfortably with systemic approaches, which implicitly challenge the conventional way of framing and approaching research inquiries. This experience resonates with Argyris’ and Schön’s ‘theory of action’, in which there is a gap between a ‘theory in use’ (the ‘normal’ ways things are done in organisations) compared to ‘espoused theory’ (what is claimed to be done in rhetoric), (Argyris and Schön (1974, 1978, 1996), Argyris (1990). What Eurofound states in its work programme is one thing (‘to assess policies and practices’) (‘espoused theory’), however, in practice it currently falls short of this ambition, as the standard approaches for normal research projects continue to be used (for example standard case study methodology).

 In order to move forward from this, a ‘shift’ will be needed. As suggested by Argyris and Schön, this calls for ‘double-loop’ learning to bring about transformational change in the organisational and academic culture in research organisations like Eurofound, and to dissolve the contradictions and ‘organisational defensive routines’. This will require time, and finding and engaging further ‘allies’ across the organisation to pursue this aim. Opportunities exist at several levels:

1. Transfer the experiences from the ‘hospital services’ project to other projects and researchers, through internal conversations about these experiences.
2. Raise this discussion in the ‘official discourse’ in the organisation, to increase interest in ‘reflexivity’ and give it legitimacy within the ‘projectified world’ of the organisation.
3. Future research methodology developments might provide space to enhance the methodological frameworks of the agency.
4. Further explorations could be conducted within the context of agencies' networks, to explore further how systemic approaches could be used by other agencies to enhance their scientific processes for policy areas with systemic and complex characteristics, such as inter-relationships, multiple perspectives and boundary judgements.