European Capitals of Culture: Success Strategies and Long-Term Effects
____________________________________________________________________________________________ Methodological challenges Compiling a representative database of all published material relating to the ECoC Programme was a complex and time-consuming task. These difficulties were accentuated by the commitment of the study to collating several different types of document – each of variable quality and reliability – rather than just focusing on the refereed, English-language academic literature or documents officially sanctioned by EU organisations. Nevertheless, completing this exercise to the best ability of the research team was a key added value for the study. It enabled the team to argue with confidence about existing evidence and overarching trends, whilst at the same time creating greater certainty as to where the most significant knowledge gaps currently exist. Some of the key challenges encountered during the document mapping exercise, which have influenced the structure of this study and its ability to address key research questions, are as follows:
Balancing a macro (meta-evaluation) and micro (dedicated ECoC city analysis) approach: As this study is essentially a meta-evaluation of the ECoC Programme, its validity relies on retaining a comprehensive lens and ensuring as broad a perspective as possible on the overall Programme experience. Due to the greater availability of data for ECoC hosts from 2005 onwards, it is also possible to provide some closer analysis of the last phase of this Programme (referred to throughout as Phase 3 and characterised by a much clearer operational model and consistent monitoring framework). This is in line with the explicit request for the study to offer analysis of individual host cities from 2005 onwards. Given the resources and time available, this study is not focused on producing new evidence but, rather, on making the most of existing evidence and pointing at the most noticeable trends and problematic gaps in the material at hand.
Lack of actual evidence, particularly for long-term effects: Many documents reflect an ambiguity or confusion as to what constitutes an actual impact, what counts as a long-term effect, and what is simply a strategy aimed at achieving these outcomes. While there has been a significant growth in awareness and commitment to attaining long-term effects, the evidence is rarely available. Instead, many reports and journal articles refer to ‘impact’ when in fact they are simply covering aspirations for impact, or strategies to achieve impact and legacy. Such distinctions are noted in the analysis of material throughout Chapter 5, which is dedicated to the assessment of short- and long-term effects. Moreover, where appropriate, data on aspirations (rather than actual impact) is presented as an example of strategic intent or objectives, and discussed as an example of a delivery approach or success strategy.
Access/availability of material: Whilst there is an impressive volume of ECoCrelated literature available online – both through free-to-access sources and subscription-only journals – there are nevertheless a number of lacunae in the material, most notably in the case of earlier ECoCs, for which little evidence is readily available. This scarcity of particular types of material compounds problems concerning the quality of available documents.
Aside from the above, the two most important challenges that have had a notable impact on the capacity of this study to provide comprehensive data analysis across the ECoC Programme are the quality and reliability of data and the comparability of data.
Quality and reliability of data: An important goal of this study is to account for documents produced in as many European countries as possible, reflecting a wide range of academic, policy and practitioner backgrounds. Such an approach has allowed the identification of published materials on almost every single ECoC host city, and this, in itself, provides a rich source of information for identifying predominant and changing trends from a thematic point of view, across very different geographical and political 33