Bill Nichols Second Marker: Christine Parsons
Buckinghamshire New University Faculty of Design, Media and Management School of Applied Business Management
B.A.HONS Public Relations & Marketing Communications Which evaluation method is most accurate in the PR Industry? Dissertation 9th May 2013 Word Count: 11,152
I certify that this submission is my own work and understand Buckinghamshire New Universities regulations about, and the consequences of plagiarism and cheating. Signed: Date: 9th May 2013
Contents Acknowledgement .............................................................................................................................. 4 Abstract ............................................................................................................................................... 5 1.1- Introduction ................................................................................................................................. 7 1.2 - Research Objectives .................................................................................................................... 8 1.3 - Research Context ........................................................................................................................ 8 2.1 - What is Evaluation? .................................................................................................................. 10 2.2 - What is Evaluation in Public Relations? .................................................................................... 10 2.3 - Business Benefit & Objectives .................................................................................................. 13 2.4 - Historical Context...................................................................................................................... 15 2.5 - Input vs. Output vs. Outcome ................................................................................................... 16 2.6 - Placement vs. Effects ................................................................................................................ 18 2.7 - Output Methods ....................................................................................................................... 19 2.8 - Outcome Methods .................................................................................................................... 19 2.9 - Pros and Cons of Current Evaluation ........................................................................................ 21 2.10 - Best Practice Evaluation ......................................................................................................... 21 3.1 -Research Philosophy .................................................................................................................. 24 3.2 - Research Approach ................................................................................................................... 24 3.3 - Research Strategy ..................................................................................................................... 25 3.4 - Research Choices ...................................................................................................................... 26 3.5 - Research Time Horizon ............................................................................................................. 27 3.6 - Critique...................................................................................................................................... 27 4.1 - Introduction .............................................................................................................................. 29 4.2 - Agency Opinion on AVE ............................................................................................................ 30 4.3 - The Need of Setting Goals & Objectives ................................................................................... 31 4.4 - Multiple Medium Measurement .............................................................................................. 32 4.5 - International Evaluation Culture............................................................................................... 33 4.6 - Overspending on Evaluation ..................................................................................................... 34 4.7 - Key Findings Conclusion............................................................................................................ 35 5.0 - Conclusion:................................................................................................................................ 37 6.0 - Recommendations: ................................................................................................................... 40 6.1 - Goals & Objectives Recommendation ...................................................................................... 40 6.2 - Overspending Recommendation .............................................................................................. 41 6.3 - Multi Medium Evaluation Recommendation ........................................................................... 42
3|Page 6.4 - International Evaluation Culture Recommendation ................................................................. 43 6.5 - Recommendation Conclusion ................................................................................................... 44 Reference List & Bibliography ........................................................................................................... 46 Appendix 1.1 - Pyramid of PR Research ............................................................................................ 52 Appendix 1.2 - Interviewee Profile Table .......................................................................................... 52 Appendix 1.3 - Coding Table ............................................................................................................. 52 Appendix 1.4 - Sample ...................................................................................................................... 74 Appendix 1.5 - Managerial Recommendations................................................................................. 75 Appendix 2.1 - Transcript - Interviewee Coding A (two respondents) ............................................. 76 Appendix 2.2 - Transcript - Interviewee Coding B ............................................................................ 84 Appendix 2.3 - Transcript - Interviewee Coding C ............................................................................ 86 Appendix 2.4 - Transcript - Interviewee Coding D ............................................................................ 94 Appendix 2.5 - Transcript - Interviewee Coding E........................................................................... 100 Appendix 2.6 - Transcript - Interviewee Coding F ........................................................................... 102 Appendix 2.7 - Transcript - Interviewee Coding G .......................................................................... 104 Appendix 2.8 - Transcript - Interviewee Coding H .......................................................................... 106 Appendix 2.9 - Transcript - Interviewee Coding I ........................................................................... 109 Appendix 3.1 - Permission Letter - Coding A .................................................................................. 113 Appendix 3.2 - Permission Letter - Coding B .................................................................................. 113 Appendix 3.3 - Permission Letter - Coding C................................................................................... 114 Appendix 3.4 - Permission Letter - Coding D .................................................................................. 114 Appendix 3.5 - Permission Letter - Coding E ................................................................................... 115 Appendix 3.6 - Permission Letter - Coding F ................................................................................... 115 Appendix 3.7 - Permission Letter - Coding G .................................................................................. 116 Appendix 3.8 - Permission Letter - Coding H .................................................................................. 116 Appendix 3.9 - Permission Letter - Coding I.................................................................................... 117 Appendix 4.1 - Supervisor Meetings ............................................................................................... 117
Acknowledgement I would like to thank the following people for helping me through this dissertation.
I would like to begin by thanking my supervisor and course tutor Dr Bill Nichols, for his support and guidance through the research process, as well as through my PR training at university.
I would also like to thank the entire Marketing faculty at Buckinghamshire New University, in particularly John Hathaway, for his good humoured support whenever it was needed, Vic Davies, for helping me to find primary research respondents and Ruth Hickmott for helping me choose PR as a profession.
I would further like to thank my family and friends for their support through this process.
Abstract This dissertation was written to evaluate the measurement of contemporary Public Relations. This topic was chosen as it questions a long standing industry issue, which not only affects the UK PR industry but the industry across the globe. It was found from the literature review that many academic texts are arguing the point that PR should be measured using methods such as AVE (Advertising Value Equivalent) or OTS (Opportunity to See), which whilst providing figures for evaluation, only presents figures based on PRO (Public Relations Officers) outputs, as opposed to their client's business outcomes. The primary research data, collected through face to face semi structured interviews, which allow for free speech during the interview. This method was selected as it allows for the respondent and interviewer to discuss items that may have come up during the stages of the interview, which might not have been identified at the planning stage. The second selected method was e-mail interviews. This method was selected for respondents who were very busy, abroad or a great distance away from the researcher. The format for these was structured written interviews, as there was no opportunity for conversations between the researcher and interviewee to take place. The questions selected for the interviews were chosen at the end of the literature review, as these demonstrated the issues that are affecting the industry. Therefore, the author of this dissertation chose to select these, to form key findings. These questions were chosen to assist in attaining key findings for the dissertation. This dissertation produced five key findings, which are that the AVE methodology is not the most commonly used method of evaluation, as the literature review stated. Secondly, this dissertation found that in order to evaluate PR effectively, there is a need for PRO's to set communications goals and objectives at the start of a campaign. Third, the dissertation found a split in the opinion on how different forms of PR should be measured. The fourth finding evaluated how international PR is measured, to establish if local culture affects the process. The final key finding is one that was not anticipated, which is overspending on evaluation. 20923417
The dissertation proceeded by making four strategic recommendations, the first, to set communications goals and objectives at the start of every campaign or quarter, secondly, to monitor how long is spent on evaluation, the third recommendation was to measure different forms of PR differently, as they require different techniques, making their culture different. The final recommendation is regarding international PR, and this dissertation recommends that international PR, should, be measured in accordance with the local cultured taking priority.
1.1- Introduction â€˜Measure what is measurable, and make measurable what is not so.â€™ (Phillips, D, 2001) This dissertation has been written to identify the most effective method of evaluation in contemporary Public Relations. The study will be analysing various methods of evaluation used within the industry. By holding interviews with various people within the industry, from academics, practicing PRO's (Public Relations Officers), from both agencies and in house teams around the UK and abroad, members of trade bodies such as the PRCA, members of teams whom specialise in PR evaluation, this dissertation will be able to obtain a rounded view of what the industry is currently doing, and how evaluation methods are being changed or altered for the future, to incorporate social media and an increase in the use of digital PR, such as blogs and online news websites. Alison Theaker has stated that 'measurement and evaluation in the public relations field is not a brand new issue or topic that has suddenly emerged in the past few years. It is an issue and topic that has been widely discussed, actually carried out and grown and evolved over a 60-year period of time.' (Theaker A 2012: 196)
1.2 - Research Objectives The objectives set in this dissertation were to: Research Objectives
Establish if AVE is still the most widely used method of evaluation in the industry, both in house and in agencies Critically discuss the requirements of how to set up an effective PR evaluation process Critically discuss the culture of evaluation in the UK and Abroad, as well as discuss whether evaluation is something that PRO's do because, they have to, or because, they want to Make recommendations as to how online evaluation (including social media) can take place Make recommendations on how the Public Relations agencies can use evaluation to its fullest extent by giving clients accurate data.
These objectives have been set in line with the industry issues, of PRO's measuring outputs, as opposed to outcomes. The above research objectives will be used in order to coordinate the primary research questions.
1.3 - Research Context The aims of this study were to identify the strengths and weaknesses, of the various evaluation methods, which are used to evaluate the results of public relations. The Advertising Value Equivalent (AVE) system is the unwritten evaluation method, however, not always accurate. This system is used with a multiplier of normally, 3 or 5, depending on the agency. However, even though this method is used on a regular basis, by many PRO's in the industry, it is a method that only measures outputs, as opposed to the outcome that a client's business will benefit from .
This dissertation aimed to seek out methods used by PRO's, to analyse whether the industry has any form of evaluation, which accurately measure business outcomes, and if so, how does the measurement process work. This paper was written, as the topic of evaluation in Public Relations, is one that is very important in the industry, as without having accurate methodologies to measure work in, It would be very difficult for PRO's to justify their existence in the work place. The dissertation looked at the various PR evaluation methods, including, AVE, clip and message counting, as well as others and then, critically evaluating the methods, to establish which would give clients, the most accurate figure for their PR activities. Tench & Yeomans describes evaluation as â€˜the evergreen topic of entire practice and one of the areas where both practitioners and academics have a vast common interestâ€™ (Tench and Yeomans 2006: P213). This study focused on evaluating the accuracy of these methods, by comparing the various models, and analysing which will in turn, bring the agency and client, a form of win win. Win for the client, will be having a successful campaign, resulting in an increase of sales, and for the agency, win, would result in an increase of business from that client. There are many different texts available on the topic of the PR evaluation, there is also a heavy online presence on the latest trends and developments on this topic. However, a high amount of online sources, are proving to be arguing the same point in the same way as others. Therefore proving only one angle of evaluation in PR. Therefore not showing the true story of what is going on in the industry.
10 | P a g e
Literature Review 2.1 - What is Evaluation? Evaluation is the process of measurement of any form. Collins Dictionary, have defined the verb 'evaluation' as 'to ascertain or set the amount or value of' (Collins Dictionary 2013). Evaluation is a process which is used in everyday life, not only in the Public Relations industry, but also stems into all parts of business and academic life. The need to evaluate ones work in every industry is very important, as it allows organisations to ensure they are meeting their goals and objectives.
2.2 - What is Evaluation in Public Relations? Evaluation in the PR industry is used for PROâ€™s to give an ROI for their efforts. By giving information on the ROI for PR campaigns, the PRO can present back to the client, proving that PR has a valid place in a modern business. Evaluation in public relations has been described as â€˜as elusive as finding the Holy Grailâ€™ (Pavlik, 1987: 65). This comment has been made because a high number of agencies and inhouse teams which use their own methods of evaluation, it has been found that one accurate key method, which looks at outcomes, compared to outputs, is very difficult to find (Watson 2012: 19). A further reason for the industry not having any accurate models, compared to advertisers, is that many agencies use their own internal forms of measurement, therefore, the industry is not able to analyse them all, in order to find an accurate and effective methodology (Watson and Noble 2007). RD Smith has stated that there are many different forms of evaluation in Public Relations, from 'output evaluation, 'message production' and 'message distribution' (Smith 2009: P245).
11 | P a g e
Watson and Nobleâ€™s book has stated over the years, there have been many ideas thrown about the industry. During the transition to the digital world, it has been found that word of mouth, or interpersonal influence has become more common with the use of social media, allowing the public to talk to their peers about a certain product or company easily, from almost any location worldwide (Watson and Noble, 2007, P14). However, as the authors have not discussed specific models or theories regarding social media, it is not possible to establish whether any methods were approved or altered to help the evaluation of digital PR. In 2012, CIPR launched â€˜Share Thisâ€™, a book about the online world of PR, where the authors state that online PR measurement should be completed using the following framework.
Meaning Is there a best practice way to identify who influencers are?
Has the data been analysed by human or machine? What is the error rate? Understand exactly what engagement
means, a follower on twitter does not mean engagement. How fast are the researchers at finding
relevant content, how broad are searches, how much data returned is valid?
Are you sure that the correct target audience is being met?
Source: CIPR 2012: P169
Tench and Yeomans have expressed that evaluating Public Relations efforts, is extremely important, as it reflects effectiveness, professionalism, whilst giving accountability for the efforts (Tench and Yeomans 2006: P224). This view has been reflected Anne Gregory, who has described evaluation as the driving force which helps PR prove its effectiveness, and its position at the board. Without doing accurate evaluation, PR will be at risk of being a
12 | P a g e
communicator that is intangible or ‘far too important to be measured’. Something, which many board members, do not believe in (Gregory 2010: P164). An interview article in PR Week, from July 2012, with Philip Sheldrake & Mark Westaby, states that PR measurement has not changed in the last 20 years. The article quotes Westaby saying ‘A real problem is the PR industry thinks its job is to get media coverage or run a good event – that’s not the case. The PR industry’s job is to influence and change behaviour of consumers and stakeholders.’ (PR Week 2012). Sheldrake comments that the industry is wasting time trying to find a universal solution to evaluation, as it is something that he feels will never be found (PR Week 2012). The PR Week article agrees with Sheldrake’s book, as he states that PR should measure what it needs to, and current methods used are doing. Therefore, until PRO’s learn they only need to evaluate their work on its own, PR evaluation will never be accurate (Sheldrake 2012: P47). Tench & Yeomans have incorporated a survey carried out by PR Week in 1999 that PRO’s prefer to evaluate their work through press cuttings (Tench and Yeomans, 2006, P226). This system is AVE, which looks at the cost of the space taken by PR efforts in terms of advertising. Which only provide figures of output, compared, to the actual result of the campaign to the company (RomyR PR, n.d). The author of this dissertation agrees with this comment, as the industry needs to focus on this, by having a clear understanding of the work, measuring success would become easier, as there is a set bench mark for the outcome to live up to. There are a handful of external organisations who offer accurate media evaluation, by looking at a number of different parameters, including position on the page, the wording used and whether the coverage from the campaign is solely on the PRO's client, or its competitors also. The more parameters that are met the higher, the evaluation will be. However, this process is very costly, therefore, not many PR agencies would not be able to afford this, and also clients would expect this to be included in the price. Therefore, if this process is carried out for each client, the agency would not make any money from their client.
13 | P a g e
There are many external organisations that offer this service, examples of these include Kantar Media, Mantra, Metrica (part of the Gorkana group) and Apollo.
2.3 - Business Benefit & Objectives From the side of a PR agency, the organisation will have their own set of brand values and will also have their own business objectives - which will include ensuring all clients are profitable to the company. However, as a client of a PR agency, the values and business objectives will be very different, the clients business objectives may include increasing market share, through a mixture of marketing communications including PR. Therefore, the biggest job for the PRO, is to prove to the client, that the medium of PR has proven effective to the overall business. The pitch to prove the effectiveness, must be successful, as some larger corporate companies may have internal marketing managers, who need to protect their budgets, therefore passing the information onto their organisations boards of directors. Therefore, from a PR agency's perspective, the benefit of PROâ€™s carrying out evaluation is important, by allowing clients to have a tangible figure to an intangible operation, this allows for a greater understanding, allowing for PR to obtain a greater power in the modern business place. Tom Watson found from a 1994 survey, that many managers believe, the best form of evaluation, is when clients renew contracts (Watson, 2012:19). While this can be considered to be true, clients would only renew if they are happy with the service, many larger cooperationâ€™s still require a higher level of information, of where funds are going, and what their ROI is. From a Public Affairs consultancy approach, this is considered true, due to the need of concrete objectives at the outset, it is somewhat easier to measure a successful outcome to a campaign, however, there will also be some form of ongoing evaluation which will take place during the campaign to measure outcomes, this is something that is more difficult compared to the end result which is essentially easier to measure (Underwood, J and Stevenson J, 2013).
14 | P a g e
Anne Gregory has endorsed the view by Tom Watson, that many PRO’s are reserved, regarding evaluation, as it is believed to be done to make themselves look better, rather than giving an answer of how the media relations worked (Gregory, 2010, P158). This maybe as PRO’s could give the impression to clients, that their job is easy, potentially meaning, the client takes their PR in house. Gregory also states that advising clients on how well the media relations worked, it would entice them to invest more time and money into PR efforts, resulting in PRO’s becoming more profitable for their company, as they will be promoting brands and services through methods of communications, which has proved to give a high ROI (Gregory 2010: P159). This may have an adverse effect, as not all campaigns are guaranteed to be successful, meaning that sometimes PR can also be unhelpful for an organisation. The author has discussed the issues regarding evaluating PR campaigns, as many firms may run campaigns alongside sales promotions, therefore causing the PRO to find a balance between the two campaigns, as both are triggers which potentially cause the preferred outcome. Tench and Yeomans have also stated, that PRO’s have been known to overemphasise evaluation of print media, and struggling to keep up with the fast paced technological world, as well as finding ways to evaluate communications, such as TV and images (Tench and Yeomans 2006: P224). PR, also has a need to evaluate live PR stunts, in order to give an accurate measurement of the work (Tench and Yeomans 2006: P225). Should this not be done, it will be difficult to prove its worthiness, as there would be no results for the various tasks carried out. Potentially making it unbeneficial for companies to invest in.
15 | P a g e
2.4 - Historical Context The 1940’s saw evaluation become a big part of PR strategies, with various systems being used at government level, however, Tom Watson's article in the Journal of Public Relations, 2013, does not talk about agency use with the activities similar of Ivy Lee and Edward Bernays. Nor does the article discuss any methods used, making it difficult for the reader to understand what methods were available. This issue has been reflected in the majority of texts read during the research stage, of this section. 1949 saw the birth of the UK’s IPR (Institute of Public Relations), which was mainly run by the Governments communicators, who saw the need for a good method of evaluation for their work, something which was discussed in their first book, which helped train communicators on how to evaluate (IPR 1958). The above mentioned book, is one that is very difficult to obtain, therefore no analysis can be made to the ideas put forward by the authors in 1958, except from the content which has been delivered in this journal, mixed with the perception of Tom Watson. This article goes onto to compare and contrast comments made over the years, regarding whether PR is measureable or not. Watson compares quotes made by Ivy Lee, and the quote of John Marston in 1963, where the two men, have taken two very different approaches to the issue of evaluation in this industry. As the quotes are around 50 years apart, it is possible to understand, how over the years impressions and opinions of measurement in the PR industry have changed. Ivy Lee has claimed PR cannot be measured, whereas John Marston, in ‘Nature of Public Relations’, has claimed that they cannot be bothered to measure success (Watson, 2012:19). This is an interesting angle, as it opens a new route of investigation, something which the primary research will also be asking, as it will help establish if the evaluation reports made by PRO’s, are accurate, or altered to look good for the client. Watson goes on to discuss how the rise of the service agencies, such as Hill & Knowlton, and Burson-Marsetller, helped kick the start the consumer public relations sector, increasing the demand for more university programmes (Watson, 2012:11). As these two agencies became international very quickly, there was an urgent need to send evaluation reports, back to the
16 | P a g e
US headquarters, for CEO’s to see what was going on in their other offices (Watson, 2012:11). This would help the directors of the company, to monitor business and results given to clients, regardless of where they are located in the world. However, over the years, many agencies have created their own theory behind evaluation, and use these systems effectively. While all maybe accurate, questions still need to be asked as to how accurate they really are compared to competitors systems.
2.5 - Input vs. Output vs. Outcome The PRO's input to the campaign is dependent, as the brief, given to them by clients should be clear, where PRO’s can decipher the issues with clarity, allowing the campaign in which they plan to contain greater insights, allowing for better targeting, placing and content. Anne Gregory has described evaluation, as being an on-going process, from start to finish, and should give live figures, allowing for any changes to be made, should the campaign not perform in the intended manner. She also states the importance of evaluating the final results, in order to look back at the campaign on a time scale, to find the most effective aspects (Gregory 2010: P161). This is something that is being echoed in other author’s works. Experts such as Tom Watson, have discussed the need to start evaluation methods at the start of the campaign, by setting goals (Watson and Noble 2007: P176), however Anne Gregory has taken this further, by evaluating campaigns while they are running (Gregory 2010: P161). AMEC have also endorsed this as the Barcelona Principles, which start with the importance of setting goals (AMEC, 2012). However, this is not a new idea, as Smiths 9 steps to PR success stated in the 2009 book, Strategic Planning for Public Relations, where he states that any PR campaign should start with goal setting at the top of the agenda (Watson and Noble 2007: P55). Watson and Noble discuss the use of non-predictive methods. This would work well, as it would not give a fixed route that PRO’s need to follow, to establish if their campaigns have been successful (Watson and Noble 2007: P41) For example, if a campaign has been started to change the perception of a local government agency, then AVE would not give a relevant answer, as it is being based upon the advertising value, rather than the opinions of the people. Therefore, the more positive coverage and online comments, which are being
17 | P a g e
made, would prove whether the campaign has been a success, as the public’s perception has changed, therefore giving the PRO a choice of methods of evaluation which is most suited to the campaign. Philip Sheldrake has endorsed in his book, ‘The Business of Influence’, the comments made by The Holmes Report and CIPR, one of the industry bodies for PR, as they state that PRO’s should focus more, on measuring outcomes, compared to outputs (CIPR,2010). The reason this is being agreed with, is because the AVE method only allows for the PRO to measure the value of their success, thus only allowing for output, whereas the outcome, for the business can be extremely different. Appendix 1.1, shows the PR Research Pyramid, (Watson and Noble 2007: 85) which shows the questions that should be asked, at the start of a PR campaign, by the PRO, in order to gain a greater understanding, of what it is that the client wishes to achieve from this campaign. By understanding this, the campaign will have the ability to target the correct audience, as well as sending the correct messages to this audience. As the pyramid demonstrates, the starting process starts with the formative research, which in turns leads and the creation of the inputs and the pilot testing of messages, in order to ensure that the correct messages are taken out of it. Once this stage is completed, the PRO beings to talk to the media, and sell their inputs, to lead to outputs. The current industry un written standard, AVE, is the method which measures outputs, compared to the actual outcomes for the client. A real life example of this would be of an online retailer, who has used a PRO to create a campaign for a product. The inputs would be from the client and the PRO, following this outputs would include, the press releases and the number of pieces of coverage obtained. Finally, the most important part of the process, is analysing what this coverage means for the client. To establish this, it is possible to obtain website visitor reports, which can be compared, with the previous 12 months, to establish if there has been an increase in the number of visitors on the site, during the time of the campaign. If there has been an increase, this can be put down to the PR campaign, to analyse this fully, it is possible to ask customers when they visit the site, as to how they learnt of the company. This will provide insights into how effective the campaign has been.
18 | P a g e
Watson and Noble, have stated in the chapter on evaluation in their academic book, that many PRO's often opt to use the simplified method of evaluation, rather than any other form (Watson and Noble 2007). This shows how the industry is viewing evaluation, as something that is only done, because it has to be. Therefore, hindering the quality of the evaluation results.
2.6 - Placement vs. Effects The PR industry has seen two different approaches to evaluation. The first is placement, where the PRO would look at press cuttings, and quantity, compared to quality of where the article is being published. However, as placement looks at quantity, it is unable to address the key messages, and what context they were used in as they ignore the fact, whether they are positive or negative. Placement offers figures on the space used, and in which publication, therefore, again referring to outputs, not outcomes. This does not give answers as to how good the coverage received was, it only gives a reassuring number for the client to see (Nichols, 2012). The second method is measuring effects, this is where the change in perception, consumer behaviour, or opinions are measured. In order to get an accurate measurement, it is required for the PRO, to carry out qualitative research, such as opinion polls and surveyâ€™s, to measure the effectiveness of any changes, that the campaign has made (Nichols, 2012). Whilst these two methods of evaluation, will have strengths and weaknesses, which will be found by the industry, as stated above, the first methodology does not answer the question of evaluation, as it only looks at the outputs. Also, the second methodology, would be very costly for PRO's, as, it requires a series of opinion studies on the topic, to ensure that the objectives have been met.
19 | P a g e
2.7 - Output Methods There are many methods in the PR industry, which allow for a PRO to evaluate their work in their terms. The most renowned method, is the Advertising Value Equivalent method, or AVE, this is where the PRO measures the value of the coverage, by looking at how much it would have cost, should that space have been purchased at rate card price. All texts have clearly stated that PRO's add a multiplier onto this value, thus further increasing the figure. However, AVE's do not measure the value against the value of advertising, they only measure against the cost (Sheldrake 2011). Watson discusses measurement of outputs, through, clip counting, clip size and message direction. The method of clip size lead to AVE, as well as the system of â€˜pepping upâ€™, this was used, when images had been published, with the PRO enlarging the image on the clip, thus claiming more space, making it seem the coverage was much larger than it was (Harlow, 1942). This is the issue that AMEC are trying to resolve today, with a method ,which reflects PR activities effectively and accurately. For more information on the output process, please see Appendix 1.1. Other options of PR models, which evaluate outputs, include the use of the Opportunity to See method, where an ROI can be given on the circulation figures established by the PRO, in relation to how many people the campaign theoretically reached (Tench and Yeomans 2006).
2.8 - Outcome Methods In order to measure Public Relations effectively, it is essential for PRO's to measure the outcome to an organisation, or their client, as opposed to measuring outputs. In section 2.7 of this dissertation, it was stated that measuring outputs, only allowed the PRO, to report to their clients about the value of their work, not the valuable or tangibility of the campaign to their clients organisation.
20 | P a g e
In order to measure outcomes, it has been found, that it is essential to set communications goals and objectives in order for the PRO to have a fixed protocol to measure against. An example of this would be, a client wanting to increase enquires of a certain product by 15%, by the end of the month, or increase traffic to a webpage or website by 15%, by the end of the quarter. By having such clear targets set, it is possible for PRO's to use the Yardstick Model, developed in 1998, by American PRO and research, Walter Lindenmann, the method has three stages, similar to set up of Appendix 1.1, this model also endorses the need to set goals and objectives at the start of campaign, the second is known as PR Outgrowth, this is where it is analysed, whether the target audience attended to the campaign and if they understood and retained the key messages. Finally, the third stage of the process measures the outcomes to the campaign, by analysing opinion, attitude and behavioural changes within the target market(Theaker 2012: 200). In 2010, AMEC teamed up with the CIPR, as well as other international PR representative bodies, to try find an industry wide solution, to this issue. Seminars around the world led to the creation of the Barcelona Principles, seven steps that will allow for PROâ€™s to report accurate ROI figures. The stages of this evaluation process include, setting goals before any measurement should start, measure media with quantity and quality metrics, understand the value of change in relation to PR for the public and business, use the same measurement for social media as efforts are still the same and ensure that all measurement is transparent (AMEC, 2012). However, no information has been announced as to how the principles will work, and how they will be monitored, as the step regarding quality can vary from person to person. There has been much discussion in the industry, regarding this new system of evaluation. Peter Sheldrake does not endorse the Barcelona Principles, claiming that they are too basic. The author endorses the Holmes Report, as AVEâ€™s do not measure the value of PR, it only analyses the cost, in relation to advertising, at the un-negotiated rate card price (Holmes Report, 2010).
21 | P a g e
AMECâ€™s Executive Director, Barry Leggetter, has described the Barcelona Principles as â€˜baby steps towards a unified solutionâ€™ (Leggetter, 2010).
2.9 - Pros and Cons of Current Evaluation The current methods of evaluation in the PR industry are extremely varied, as this literature review has expressed from the start, the biggest weakness of current methods stated in academic textbooks, industry press and journals, are that many PRO's are only evaluating because they have to, and by doing so, only using the most simple methods available. By using such a simple method, PRO's, are not doing the industry any good, as they are potentially giving the impression that PR, can only be measured in terms of outputs, compared to the all important outcomes. The pro's of PR evaluation, are that, it can be proven, how cost effective the medium is, compared to others, such as advertising, or digital marketing. Many business owners know that PR is an effective medium, as they understand, that when a client has been included in an article, it is simply because the journalist has chosen that company, over a competing firm, thus delivering the message through a third party, whom the public trust and connect with.
2.10 - Best Practice Evaluation This dissertation, was written to establish the most accurate method of evaluation in the Public Relations industry. Following the above reading, as well as much other background information research, the author has established, that evaluation in PR is something which is considered a dark subject. This means that the measurement results can be increased, thus causing the client, to believe they are getting a higher ROI. It was also found that there has been many ideas thrown around the industry over the years, however, very few have been put into practice. The last few years have seen the Barcelona Principles, something which is aiming to combat the negative side of evaluation in PR, however, as stated above, some academics and
22 | P a g e
professionals, believe it is too little too late, as the industry has created the illusion of greater ROI's (Holmes Report 2010). It has been found from the above literature, that best practice would involve the need to set goals and objectives, prior to the start of the campaign, to give the PRO a metric to measure against. Watson and Noble endorsed this, as it allows the client to obtain an accurate evaluation figure, illustrating whether the goals set have been met effectively (Watson and Noble 2007: 48) - A view, which has been heavily recommended, by the other authors. Watson and Noble, endorsed the theory of McCoy & Hargie, that PR has to create awareness, leading to public knowledge, aiming to form a favourable attitude change, leading finally, to changes in behavioural patterns (McCoy and Hargie 2003:305). Smaller agencies, and in house PR teams, use the most common form of evaluation, AVE (Tench and Yeomans 2006: P229). However, as this measures output, the true value of PR is not being discovered. Following this literature review, primary research aimed to identify a method of evaluation, which will allow PRO’s to accurately uncover the value of their outcomes, compared to outputs. It was vital to understand from PRO’s, if they felt the mass media impacts the public behaviour, and how the impact of digital, made them alter their evaluation methods to incorporate this. The research programme, will also look to establish, whether PRO’s feel that PR is intangible, in terms of results, and how important evaluation is for them. It is also imperative to establish, how, they evaluate other methods of PR, including radio, TV and online, as whether they feel this is accurate. The study, will look at evaluation cultures in agencies, and in-house teams, the study will also aim uncover, if PRO’s are being trained in AVE or other methods.
23 | P a g e
By looking at various methods, used by PR agencies, and in-house teams, it will be vital to establish, whether the process requires goals to be set at the start of the campaign, and if they require measurement, to run simultaneously with the campaign. The research also needs to establish how international agencies evaluate PR abroad, to see if they are using one unified system, or if local culture comes into effect. Finally, it would also be important, to establish, whether PROâ€™s feel that methods used within their organisation, give accurate figures, while trying to understand what they would do differently for clients, or, if they are being told by their superiors to make evaluation reports look better, or, if they are aware of this malpractice. Below is the methodology for the primary research, where the research objectives, as stated above will be answered.
24 | P a g e
Methodology 3.1 -Research Philosophy The philosophy chosen for this study was critical realism. This was selected, as it reflects the topic of this dissertation as well as the conclusion to the literature review, as each source reviewed gave a different opinion, on which method of PR evaluation, gives the most accurate result. The realism philosophy, is based on the belief that reality is independent of human thoughts and beliefs, thus meaning, each person has their own views, even though their industry may suggest another way (Saunders, Lewis, & Thornhill 2003: P81).
3.2 - Research Approach The research approach to this dissertation, was deductive, as this study looked at the various methods of evaluation in PR, and which are the most accurate, rather than methods such as AVE, and its associated multipliers, which vary from agency to agency. This method, also, ensured that by proposing a relationship, between two variables that were not fixed, can be changed, after the primary research results have been collected and analysed (Milyankova n.d).
Dr Burney, has stated that a deductive approach, starts, with general content, and then follows down to specifics, where as an inductive approach, can be used to create a theory, from looking at specifics (Burney 2008). In order to gain the data, it was vital to speak to practicing PROâ€™s, both in house and consultancy, to discuss the various ways which their departments evaluate their work. This was conducted, by holding face to face interviews, as well as some email interviews with PROâ€™s, who have worked in the UK and possibly abroad, to discuss how the culture of evaluation varies, in the countries that they have been working in.
25 | P a g e
Wilson, has described face to face surveys, as a form of research which involves meeting respondents face to face and interviewing them, using paper or digital methods (Wilson 2012). This will provide a form of qualitative data, as it will give insight into what PROâ€™s and academics feel on the subject. Bernard further quotes, that 'if a respondent doesn't understand a question in a personal interview, you can fill in, and, if you sense that the respondent is not answering fully, you can probe for complete data' (Bernard, H 2006: 256). This required an outline plan, of what the interviewer wished to establish from the interview, in order to lead the conversation effectively (Lindlof and Taylor, 2002). This is also further supported, by Bernard's quote above, that the interviewer is able to lead the conversation, if the respondent is not answering the questions in the correct manner. It also allowed the interviewer, to take the discussion into an area that may have not been anticipated at the planning stage of the study, depending on the responses given.
3.3 - Research Strategy As there are various theories, surrounding the topic of evaluation in Public Relations, the strategy for the research, was based around a grounded theory, as this dissertation discussed various models, in order to ascertain which method will allow the PR industry to give the most accurate return on investment figures.
A ground research, is a method, which works in a systematic manner, while still having flexible guidelines, allowing for a free flowing conversation, with the respondent able to mention any facts, that will reinforce the point behind their answer. This allowed the interviewer to take the respondent, into a feeling of freedom, by asking open questions, allowing for a free flowing conversation (Charmaz K, 2006). The British Medical Journal, stated that grounded theory studies, are appropriate when the study aims to explain a process, and not to test, of verify, any existing theory (British Medical Journal 2008).
26 | P a g e
The research method, which was undertaken for this dissertation, were semi structured interviews, and structured email interviews, with persons from the PR industry, including PROâ€™s in house and in agencies from the UK and the US. The interviewer, also spoke to academics about their views, on the accuracy of evaluation in the industry, as well as the various methods available, finally, the dissertation approached various PRO's who were apart of trade bodies, such as the PRCA.
'First, they are well suited for the exploration of the perceptions and opinions of respondents regarding complex and sometimes sensitive issues and enable probing for more information and clarification of answers.' (LK, 1994)
The target number of interviews stood at 12, as this was felt that, it would allow for a overall rounded opinion to be formed on the topic.
3.4 - Research Choices The choice of research chosen for this dissertation, was a qualitative mono-approach. This was chosen, as it enabled the dissertation to obtain a rich insight, into the minds and emotions, of PROâ€™s and academics, to understand exactly what it is, that they feel about the evaluation processes, that are being used in the industry today.
The result of this study, gave the dissertation a form of qualitative data, providing a high level of insight, to the topic of this dissertation. By gaining insight, from the target audience outlined above, the study was able to gain opinions and ideas, of what the industry thinks and feels about the subject of evaluation.
This allowed the author, to add this to the review of literature above, allowing for further information to be uncovered, giving the industry a better insight. By understanding what the industry is doing, it will be possible, to establish a common ground, on what it is that PROâ€™s are doing in terms of evaluation. As this study set out to talk to people, who work with public relations, it is only these people who will need to be interviewed.
27 | P a g e
The reason that this target audience was chosen, was due to active PROâ€™s holding insights, into how their organisation undertakes evaluation, and its culture within their company. The second target group were academics, by interviewing this group, insight were found of how evaluation is being taught in universities, and how the lecturers feel about the Barcelona Principles. Finally the researcher, contacted those in trade bodies and those who work in evaluation companies. Please see appendix 1.2 for the profile of those interviewed.
3.5 - Research Time Horizon The time horizon chosen for this dissertation, was a cross-sectional study. This type of study takes place over a fixed period of time, rather than being longitudinal, where a study would be monitoring changes, and developments around the research topic. The cross sectional method, works differently, as the dissertation looked to establish the differences between agency cultures, in terms of evaluation of public relations. Cross Sectional Study â€“ A study based on observation represented at a single point in time (Babbie E, 2010). This strategy was chosen, as it will be possible to complete the study in the limited time scale available.
3.6 - Critique Following the study, it was found that research methodology chosen worked well for this study. The dissertation interviewed ten people from the PR industry, please see appendix 1.2 for the profiles of those interviewed. By holding a semi structured interview, it was possible for the interviewer and respondents to hold an open conversation, rather than a strict conversation where the interviewee can only answer the questions they've been asked (Sage Research Publications, n.d). Having completed the study, a reflective summary can be created.
28 | P a g e
It was found that access to people in the industry was the biggest barrier which was faced. As many PRO's work to deadlines, agreeing a suitable time and date to interview them was very difficult, as many of the times offered by the interviewees clashed with core lectures that the author of this dissertation had to attend. If this study was repeated, it would be recommended to allow a longer period of time to access PRO's in the industry. An appropriate time period would be an extra four weeks, as this will allow for drop outs to be replaced. Other than this, it has been found that the study carried out was very successful and helped to prove the research objectives. Please see appendix 1.2 for a table which has the profiles for those interviewed. These people were selected to be part of the interview, as the researcher wished to obtain a rounded view from people from all corners of the industry, as well as those from the UK and America, incorporating those working in agencies, in house and from measurement organisations. However, the limitations are that the research project is only talking to very few people in a mixture of position, regardless of agency or in house. A second research limitation is the study is only talking to two respondents based in the US, with the remainder in the UK. The data capture methods used, was extremely effective, as it allowed for the researcher to probe into the mind and emotions of PRO's, regarding, evaluation and measurement in Public Relations. By holding semi-structured interviews, it was possible to discuss in great depth around the topic, also allowing respondents to bring new parts of the issue to the table which they feel are important.
29 | P a g e
Key Findings 4.1 - Introduction This section of the dissertation, analyses, key findings below. The findings are based on the research objectives set with the key findings, which are: Research Objectives
Establish if AVE is still the most widely To explain the findings on the subject of used method of evaluation in the agencies opinions on AVE industry, both in house and in agencies Critically discuss the requirements of how To discuss an industry issue of over to set up an effective PR evaluation servicing clients in terms of evaluation process Critically discuss the culture of evaluation To explain the findings on the subject of in the UK and Abroad, as well as discuss international evaluation culture whether evaluation is something that PRO's do because, they have to, or because, they want to Make recommendations as to how online To explain the findings on the way that evaluation (including social media) can various mediums are evaluated take place Make recommendations on how the To explain the findings on the needs to Public Relations agencies can use set goals and objectives at the start of a evaluation to its fullest extent by giving campaign clients accurate data. These five points were chosen to be analysed, as it was these that allow for the author to analyse, what is best practice, and what steps that can be taken, to ensure that evaluation in Public Relations can be carried out accurately.
A coding table (appendix 1.3) has been used, as it enables the research to see they key data in one table, allowing for trends to be identified, thus engaging with the data (Miles and Hubberman, 1994; Yin, 1994).
30 | P a g e
4.2 - Agency Opinion on AVE It was found that many agencies, while offering the AVE method, do not recommend this to clients, as it refers only to outputs, not outcomes. This is something which links to the literature review, as it was found that many authors stated, that AVE was an outdated methodology, as well as being highly inaccurate, thus not giving the true value of PR. Respondent C stated in their interview that 'it doesn't answer the problems, it is very outdated.' This quote, was countered by respondent A, with the comment 'you're talking their language.' To put this comment in context, respondent A, was discussing the use of AVE, as a language that many small business owners understand, as they may have commissioned advertising in the past, therefore understand the currency. It was found from the primary research, that out of the people interviewed, only two PR organisation used AVE on a regular However, other agencies, offering it as a methodology to clients , other methods found included cost per 1000, where the agency is able to work out, how many people were reached, using the allocated budget. This again helps to measure outputs, with a slight amount of outcomes, as the PRO is able to put a figure on readership, this allows clients to understand exactly how many potential future customers they are reaching. It was also found from the primary research that, many agencies do not stick to one method, the methods they use, are dependent on the client's needs and wants from their campaign, and what they want to know from the evaluation. Other methods to measure good or bad PR, is by measuring reputation, as the fundamental task of PR, is to change opinion of the public. However, respondent D stated in the interview, that ' What we haven't done yet is make that link, to show how things that affect reputation can affect share price can be affected by good PR.' This quote discussed that while the industry has moved forward, in terms of accurate evaluation, there is still a long way to go, before the answer will be around.
31 | P a g e
Following the literature review, it was noted, that many authors have shown in their work, that finding one methodology, for evaluating public relations, is as elusive as the holy grail(Pavlik, 1987:65). As PR covers such a vast part of the business place, the requirement, to measure their outcomes are extremely important. As quoted by respondent F, who stated very clearly that evaluation in PR is 'very' important.
4.3 - The Need of Setting Goals & Objectives The second main theme, found from the primary research stage, was the need to set goals and objectives at the start of a campaign, this is vital, as it allows for the PRO to have some of parameter to measure against. As respondent B quoted 'just because we have 250 press cuttings, what does that mean to a client?' This quote states, that without communication goals and objectives, numbers are simply meaningless. It was found from the research carried out, that this was fundamental, with some agencies not doing this as much, and simply agreeing to 'just market' clients products and hope for the best. The research found, smaller agencies do not do this on a regular basis, with respondent A claiming that by setting goals and objectives, 'it can allow the PRO to be more realistic in what they need to achieve', and not to over promise the client. Also by setting goals and objectives, the client will understand, what they the aims are from the campaign, and can also be assured, that the PRO is working at an acceptable speed with their deliverables. One reason why this may not happen as often, was found from another question in the interview process, regarding the difference, between the in house and agency culture. It was found that in house teams, may be prone to have more information, than would normally be released to an agency. Again, by quoting respondent A, it was found that 'Because they can be much more open about their business plans and objectives than they can be to perhaps an outsider.' This is an
32 | P a g e
interesting quote, as many businesses, may not wish to publicise their plans, in fear or having them copied, or being second to the mark with their own plans, as competitors many have beaten them to the finish. It was found from respondent G, who is in support of the AVE system, that this methodology cannot use goals and objective at the start, with the comment 'No, not at all.' It was stated by respondent H, that on social media, their agency is able to set a numeric targets on how much engagement is required from the campaign. 'we will set a numerical goal in terms of retweets, likes, hashtag uses.' Writings such as those of Tom Watson, who have used the Pyramid Model of PR Research (see appendix 1.1), which shows the process that PRO's should undertake this, when creating a campaign, this is similar to the model of Smith's 9 step process, which shows, that when the client is open and willing to share information with the PRO to gain a better insight, the outputs given, will be very different, as the PRO is able to understand, exactly, what is wanted to be achieved from the campaign in question.
4.4 - Multiple Medium Measurement This dissertations third key finding related to the evaluation of various media's, as each media is different, with different media's, having different capabilities and different parameters to measure, it is has been felt by the author of this dissertation that different ways are required to evaluate the PR results on different mediums. Respondent E, who chooses to evaluate various mediums, with different methods, claimed that 'Clients want to measure click throughs but only some clients have this technology to do so.' Therefore, although the respondent prefers to measure different mediums, with different techniques, it is not always possible, thus meaning each client is evaluated differently, depending on needs, resources or budget. Whereas, respondent C, stated that they would agree with the need, to evaluate different mediums in different ways.
33 | P a g e
This was further confirmed by the quote of respondent G, that 'I think evaluation methods do differ for different media.' The literature review again has a mixed perception over this, with various authors arguing their personal views on the subject, with many saying that PR, should have one form of measurement, for the client to understand the value, with others arguing, that different media's require different things, therefore, requiring different methods of evaluation. The CIPR have stated, that online PR evaluation, should follow one protocol. Again, as mentioned above, the issue with evaluation, is that PRO's attempt, to measure what they need, as well as what they don't need to. This is something, that as outlined above, is a big issue in the industry, as it means that PRO's, may be preoccupied, measuring the unimportant aspects of the campaign, compared to ones which are going to make a difference, to the clients business.
4.5 - International Evaluation Culture The fourth key finding, discussed the need to understand, the local culture of international agencies, when a UK agency, is acting as a lead agent. This happens when working with clients, who require international PR. It was discovered from the primary research, that a 66% of respondents, felt, that as long as the media allowed the same methods of evaluation, including multipliers in AVE, methods used by the lead agency should stand. However, as each media landscape is different, respondent B correctly discussed the way media culture in different countries vary, thus meaning evaluation methodology would need to be altered to reflect this. Respondents quotes to this point, include respondent F, who stated 'For a Global Comms teams, one method is important in terms of framework and success metrics. The targets however have to adjust to take into account market differences.' However, respondent D, gave a different opinion. As they stated that, 'one that is set in the home country is the one that resonates around the world, if they can afford it, if the media in the country allows it.'
34 | P a g e
Nevertheless, a third quote, from the primary research came from respondent B, who states that ' Yes - for example, in China, where press release reprints are worth a lot , here they are not, and if you tell a Chinese PRO this, they will be very offended as it is their system.' Respondent B's quote came with local culture in mind, as the media landscape is different compared to the one which operates in the UK, PRO's in China, would have to evaluate with a different methodology. By altering methodologies for different countries, international clients will be able to monitor their progress, in various countries around the world.
However, as stated by respondent I 'Global media trends'. This quote shows how many media corporations will use the same stories in the same manner, however, it may not always work this way, as different organisations use different methods of persuasion depending on their agenda and audience.
This point was further confirmed by respondent H, with the comment 'I definitely think that local culture and ethics make a huge difference. There may be an agency out there who doesn’t properly measure their PR results, and in result, they may continue making a mistake over and over again with different clients.' The reason as to why agencies feel that the methodologies should remain the same, was discovered in the literature review, as Watson described, by explaining, that as agencies such as Burson Marstellar and Hill & Knowlton become international very quickly, there was a need for PRO's to send evaluation figures to their head office, in order for CEO's to monitor the progress.
4.6 - Overspending on Evaluation An issue that was found from the primary research, is that many agencies regularly, overspend on evaluation, with some agencies spending up to 40%, of the budget on this task. It is important for agencies to address this point, as it is currently reducing outputs, for example, a client budget of £1 Million, £400,000 is being spent on evaluating £600,000's worth of PR. If clients became aware that such high proportions of the budget, was being spent on evaluation, they may feel that their money is not causing a valuable difference to their business, potentially scrapping their PR programme.
35 | P a g e
This point was raised by respondent B, who claimed that 'It is important not to over spend on evaluation, many clients want figures, but they don't realise it can be costly and effective, or cheap and effective, in this case cost doesn't really impact results. I feel the average spend for evaluation should not exceed 15% of the overall budget, where as many cases are over 40%!' This was also resonated in a quote made by respondent C, who stated, over servicing is a industry wide issue. 'A PR agency will very rarely make money out of an account for the first quarter, it's only because of the amount of money you put into winning that account, time and effort, if you take that away from the fees, and in fact in that first quarter, you over service it, you're getting up to speed, there is a lot of content, you've got to hit the ground running'. This is something which was not covered in the literature review, therefore, a new find, which the industry may not have considered. However, due to the nature of PR, when starting with a new client, the first and second quarter, working with the client may not be profitable for the agency, due to the vast amount of time and money put into winning the business, prior to the client signing up with the agency. Therefore over servicing is an issue in the industry which cannot be resolved, as from the first day, the PRO is having to evaluate their work.
4.7 - Key Findings Conclusion In conclusion to this section, it was established, that while PRO's offer AVE as a form of evaluation, it was not a method which a majority of respondents choose to use. This therefore disregarded the literature review, as many authors have stated that PRO's use the AVE methodology to evaluate their work on a regular basis. The second key finding, was regarding the needs to set goals and objectives, prior to the start of a PR campaign. This was found to be an important stage in the evaluation process, as many people interviewed have claimed that it is not possible to evaluate PR, if there are no communications goals or objectives. The literature review also showed the importance of this, with Smiths 9 step model, which incorporates this into the planning process of PR.
36 | P a g e
Thirdly, multiple media evaluation is one topic, which the literature review and primary research has provided mixed views on. Many PRO's, and academics argue that PR should have one form of measurement, in order to use the same metrics, however, others argue that different mediums require different methods of evaluation to be accurate and efficient. The author of this dissertation believes that multiple mediums, should be measured in different ways, as each have their own pros and cons. The fourth recommendation listed, was based around international evaluation culture, and whether local culture comes into consideration, when evaluating international PR campaigns when partner agencies are employed. It is believed by the author of the dissertation, that as the local culture and media landscape may not always operate in the same manner as the UK media landscape, the way international PR is evaluated should differ to account for this. The literature review, showed little information on this topic, with one source claiming that the early PR agencies commissioned one method of evaluation, as it allowed for PR agency CEO's to monitor how regional offices are performing. The final key finding was on overspending of evaluation in the industry. While not many respondents discussed this, it was found that two respondents are concerned about over servicing in the industry.
37 | P a g e
5.0 - Conclusion: In conclusion to this dissertation, it was found that due to the high number of internal methods of evaluation, this dissertation has only been able to evaluate a small number of these methods. Following the literature review, the key issues identified, were that many PRO's attempted to measure too much, this means, they are measuring things that would not have made a difference to their client, in order to make themselves look better. It was also found that many PRO's, actively increased the value of their evaluation figures, again to impress clients and giving them the perception that PR was much more cost effective than it actually was. Questions also remained unanswered, regarding the evaluation in international agencies, as well as the cultural differences, between in house and agency staff, as the business objectives and goals are different. This may influence the way that agencies would like to evaluate their work, as it was found from the primary research, there was a need to evaluate outputs in agencies, but with in-house teams, the need to evaluate outcomes to the business were much greater. Part of the research methodology used, was a deductive semi structured interview approach, this allowed the interviewer, and interviewee, to hold a free discussion on the topic, compared to a structured interview, where the interviewer takes full control in the conversation, not allowing the respondent to give varied details. The research was carried out, using face to face and e-mail interviews, which were offered to PRO's, both agency side and in house, however, as explained in the critique section, people had to drop out due to business related issues. Therefore finding respondents who kept their appointments was an issue.
38 | P a g e
The objectives set were to: Research Objectives
Establish if AVE is still the most widely used method of evaluation in the industry, both in house and in agencies Critically discuss the requirements of how to set up an effective PR evaluation process Critically discuss the culture of evaluation in the UK and Abroad, as well as discuss whether evaluation is something that PRO's do because, they have to, or because, they want to Make recommendations as to how online evaluation (including social media) can take place Make recommendations on how the Public Relations agencies can use evaluation to its fullest extent by giving clients accurate data. By comparing the primary research findings with the research objectives, it was been found that some objectives have been disproven, while others had been confirmed, potentially leading to some industry wide changes in terms of online evaluation in PR. It was discovered from the primary research, that the AVE system, is not the most common method used in the PR industry, therefore this objective had been disproven. This was proven extensively, as all but one respondent claimed they use various methods, depending on the clients wants, and their budget. It was found that the main theme from the literature review, was to set goals and objectives at the start of any campaign. This was a trend, which was heavily promoted in the primary research, with many respondents stating, that without this happening, it would not be possible to measure accurately, thus returning an issue highlighted in the literature review. Where PRO's have been known for evaluating much more than they need to. It was also planned to discover, whether the culture of international agencies made a difference, in the way that international colleagues evaluated work. It was found that there
39 | P a g e
was a split approach in the way that UK PRO's see the culture and methodologies, with some believing that methodologies, should be the same in partner agencies, rather than being different in other cultures.
40 | P a g e
6.0 - Recommendations: It was established from the primary research data, in section four of this dissertation, that it was not possible to ascertain a unified method of evaluation, for the Public Relations industry, which was cost effective and accurate. However, this section of the dissertation, will make strategic recommendations, as to how evaluation in the industry, can be improved, in a cost effective manner, with little risk to the industry.
6.1 - Goals & Objectives Recommendation The first recommendation, was based on the finding of the importance, to agree a set of communication goals and objectives, at the start of the campaign, or at the start of a quarter. This is an essential process, as it will allow PRO's to understand, how to target the media outlets, and with which stories. By understanding what the client wants to achieve, it is possible to plan a campaign around that, thus, giving real parameters for PRO's to measure against. By having such solid parameters, such as, driving 20% more traffic to a website, or gaining 35% more inquires, it will possible for PRO's to work with clients, to measure this, by monitoring website traffic, or enquiries, it can be found out whether the campaign has generated this increase. If it is found that the campaign has provided this increase, then the campaign has been a success, however, it is important for the campaign, to run side by side, with the monitoring process, as it will enable the PRO, to make alterations, should things go wrong. This recommendation, is one that should be spread out across the industry, as there are many different forms of evaluation, used in the industry, it is felt by the author of this study, that this will ensure accuracy of evaluation, can be improved.
41 | P a g e
6.2 - Overspending Recommendation The second recommendation made, is for agencies to monitor expenditure, in terms of how many hours are being spent on evaluation. As outlined in section 4.6, it was found that PR agencies, regularly over spent on evaluation. In order to combat this, it is recommended, that PR agencies take steps, at the start of the campaign to ensure that they are only evaluating what is needed, and not anything more. As PR is a medium that is constantly trying to increase its presence, as a form of marketing, which is better value for money than advertising, many agencies, are spending too many hours, simply trying to accomplish this, therefore, effectively wasting money, on tasks where cost, does not affect the quality of the final figure. Therefore, this key recommendation, is for account executives and managers, to monitor how long PRO's are spending on evaluation work. In order to help PRO's measure what is needed, it is important for the agency, at the start of the campaign, to sit down with their client, to establish which publications, websites, and blogs, are the most influential in the clients business sector. By establishing this, not only can the correct media be targeted, but the PRO can then make a decision, whether other media outlets will follow suit. Should this be the case, it will be possible, for the PRO, to only measure the effectiveness of the coverage, from the media outlets that will be most influential, thus making a larger difference to the clients business, compared to the smaller publications which the clients customers, may not read. This will also require the PRO, to get a greater understanding of the clients business sector, allowing them to learn more about the client's choice of media. By understanding this, it will be possible for the PRO, to understand the client's business much clearer, by having such a clear comprehension, they will be able to provide better outputs for their clients, as they will be able to sell PR to journalists with greater ease.
42 | P a g e
6.3 - Multi Medium Evaluation Recommendation The third recommendation, from this dissertation, is to advise PRO's of the need to have different evaluation models, for different types of media. As stated in the literature review, the CIPR have provided a template of parameters to follow. By following this table, it will be possible to establish exactly what result, a digital campaign has achieved. It is important that the first step for PRO's, is to divide their evaluation, from online and traditional. Following this, the online method can be further adapted for the use of social media. Online PR can be tracked by the use of Google Analytics, or other tracking services, this will allow for the PRO, or client to monitor what traffic is going to a web page, and where it is coming from. By analysing this, a PRO would be able to measure the success of a campaign, depending on the objective set, as, if the objective was increase traffic to a specific webpage, this can be monitored with ease as it is a solid objective, with a clear outcome. However, if page visitors are not interacting with the website, and are simply spending time on the webpage, the ROI value may not be as high as a client would have liked. However, on sources such as Google Analytics, the user is able to set a conversation rate, which is in line with their business. The tracking software, will then provide reports as to what sources, are providing the best form of ROI for a client. This is where the client will be able to see, where their website traffic and engagement is coming from, and with what value. The value is something, which is advisable to be preset with the PRO, before the campaign. It is possible to find information on bounce rates, this provides a value of how many people have dropped off the website without engaging in any form. The industry average for this stands at between 30 and 40% of visitors (Marketing Journal Blog, 2012). By allowing for this drop off rate, a value can be put onto the value of online PR.
43 | P a g e
It is recommended, that social media PR, should be measured by the number of engagements which happen per post, it is recommendable, for PRO's sit with clients, to discuss the business's financial value for each engagement, following this, an accurate and personal ROI can be given to each client, as to what each social platform is providing. The further recommendation, is for agencies to do with this all clients, to calculate an average value overall. By doing this, the industry can determine an industry standard. This will allow for the social media platform to be evaluated effectively. This will allow for businesses, to see a financial tangible value on their social media efforts, helping them to analyse whether or not social media is working for them.
6.4 - International Evaluation Culture Recommendation The forth recommendation, is working with international agencies, it was found from the primary research, that 50% of respondents, felt there was a difference in culture, and that local culture and ethics, influence the way in which media is evaluated abroad. Also, as media is different in various countries, evaluation would have to be undertaken in a different manner, as values and landscape would be different. Therefore, models used in the United Kingdom, and the United States of America, may not be valid in China, or Russia. Due to this fact, international agencies, or agencies with partner agencies abroad, will need to discuss ways to evaluate work effectively and accurately, depending on the local culture. It was found from the primary research, that one respondent, clearer stated in China, press release reprints or 'lazy editing' are normal, however, in the UK, this is not normal practice, therefore, the value, could be considered to be less, as the journalist has not written the article in the style they would normally have, nor may it be in the personal tone that the readers are used to. As this is the norm in other counties, PR colleagues based in these countries, may become very offended, should a UK PRO discuss the value of their outputs, and claim they are very low. In order to combat this, PRO's need to discuss the local culture, and norms with agencies abroad, in order for a seamless evaluation process to take place, as this will allow the lead 20923417
44 | P a g e
agency, to inform their client of the various ROI's abroad. This can help the client, as they can decide exactly what it is that is going on around the world, in their PR efforts, while obtaining accurate ROI figures for each nation they are operating PR in. Should lead agencies expect partner agencies to evaluate using the same methods, inaccurate figures can be reached. Therefore, making the ROI figures produced inaccurate.
6.5 - Recommendation Conclusion In summary, the recommendations proposed in this section, are advising PRO's, to monitor the time they spend on evaluation, in order to give clients the best possible value of money, with around 10 to 15% of the overall budget, being spent on evaluation purposes. The second recommendation proposed, to set communication goals and objectives at the start of a campaign, in order to allow for the PRO to have a set of solid targets, they will be able to accurately measure their work. Thirdly, it is proposed, that PRO's should use two methods of online evaluation, first is the use of Google Analytics and bounce rates, by setting an ROI figure for each type of conversion, from key word searches, to click throughs, it will be possible to establish how much ROI a campaign has given a company, as mentioned above, the PRO and client would need to decide a financial figure for each conversion method. This can then be monitored throughout the campaign, to ensure the objectives are being met. Secondly, a similar approach is proposed for social media, with every form of engagement given a personal value for each client, in regards to how much value, the client feels they are getting from engaging with a member of public in a certain way, through their social network portal. By setting individual figures for clients, agencies will be able to set agency standards, which in time will lead into industry standards. This is something that can be done, for both social media, as well as other online methods used. The fourth and final recommendation proposed in this dissertation, is to ensure that all PRO's understand the culture and ethics of partner agencies, who operate abroad, in order
45 | P a g e
to understand how they undertake their evaluation, as media landscapes are different in every country, where some countries, such as China, still operate a Government controlled media, where as others, such as the UK operate free media. It is felt that these recommendations will help the PR industry, to improve the accuracy of evaluation while providing an initial answer to working with agencies abroad, as some agencies are still naive in believing local culture and media landscapes vary, the recommendations will also allow a frame work for evaluating online PR and social media, as this is something which is currently seen to be evaluated in the same way as traditional PR, even though the landscape of the outlet is very different. By carrying out the above recommendations, it will be possible for PRO's to offer a cheap and effective method of evaluation, in both traditional and online PR, including social media.
46 | P a g e
Reference List & Bibliography
AMEC, 2012, AVEs not an awards answer, [Online], AMEC, Available at: http://amecorg.com/2012/06/ave%E2%80%99s-not-an-awards-answer/, [Accessed 13th December 2012]
AMEC International Association for the Measurement and Evaluation of Communications, (2010). Barcelona Declaration of Measurement Principles, [Online], London, AMEC, Available from: http://www.amecorg.com/newsletter/BarcelonaPrinciplesforPRMeasurementslides. pdf. [Accessed 31 December 2012]
AMEC, n.d, International association for the measurement and evaluation of communication, [Online], AMEC, Available at: www.amecorg.com, [Accessed 24th September 2012]
AMEC, 2011, Valid metrics for pr measurement, [Online], AMEC, Available at: http://ameceuropeansummit.org/amecorgdocs/ValidMetricsFramework7June2011P rintVersion.pdf, [ Accessed 14th October 2012]
Anderson FW, Hadley L, Rockland D & Weiner M, 2009, Guidelines for setting measureable public relations objectives: an update, [Online], Anderson FW, Hadley L, Rockland D & Weiner M, Available at: http://www.institueforp.org/files/uploads/Setting_PR_Objectives.pdf, [Accessed 13th April 2013]
Babbie E, 2010, The practice of social research, 12th ed, Canada, Nelson Education LTD
Baran SJ, Davies DK, 2008, Mass communication theories: foundations, ferment and future, Boston, Cengage Learning.
Barry A, 2004, PR power, London, Virgin Books
Baskin O, Hahn J, Seamen S and Reines D, 2010, Perceived effectiveness and implementation of public relations measurement and evaluation tools among European providers and consumers of PR services, Public Relations Review 36 (2), PP 105-111
47 | P a g e
Bengston DN, Fang DP, 1999, An innovative method for evaluating strategic goals in a public agency, Evaluation Review, 23, (1), pp77-100
Bernard HM, 2006, Research methods in anthropology: qualitative And quantitative approaches, Oxford, AltaMira Press
Bernays E, 1923, Crystallizing public opinion, New York, Boni and Livenight
Botan CH, Hazleton V, 1989, Public relations theory, Hillsdale, Lawrence Erlbaum Associates.
British Medical Journal, 2008, Qualitative Research, [Online], British Medical Journal, Available at: http://www.bmj.com/content/337/bmj.39602.690162.47, [Accessed 28th April 2013]
Broom GM, 2009, Cutlip and centre's effective public relations, 10th Ed, Upper Saddle River, Pearson
Bryman A, 2008, Social research methods, 3rd ed, Oxford University Press, Oxford
Burney, SM, 2008, Inductive and deductive research approach [Online], Dr SM Aqil Burney, Available at: http://www.drburney.net/INDUCTIVE%20&%20DEDUCTIVE%20RESEARCH%20APPR OACH%2006032008.pdf, [Accessed 30th April 2013]
Carey, B, 2013, Active PRO's, S Mashru, 10th January 2013, 17.30
Charmaz K, 2006, Constructing ground theory, a practical guide through qualitative analysis, London, Sage Publications.
CIPR, 2010, Barcelona principals, the end of ave?, [Online], Jay O’Connor, Available at: http://www.cipr.co.uk/content/news-opinion/presidents-blog/4912/barcelonaprinciples-the-end-of-ave-, [Accessed 12th October 2012]
CIPR, 2012, Share this, West Sussex, John Whiley.
Collins Dictionary, 2013, Definition of evaluation, [Online], Collins Dictionary, Available at: http://www.collinsdictionary.com/dictionary/english/evaluation?showCookiePolicy= true, [Accessed 24th April 2013]
48 | P a g e
Cutlip, S.M. (1994) The unseen power: Public relations, a history, Hillsdale, N.J, Erlbaum Associates.
Dey ,I, 1993, Qualitative data analysis, Routledge, London
Gregory, A. (2010) Planning and managing Public Relations Campaigns. 3rd Ed, London UK: Kogan Page.
Guardian, 2012, Eastenders wins Christmas day ratings, [Online], John Plunkett, Available at: http://www.guardian.co.uk/media/2012/dec/26/eastenders-winschristmas-day-ratings, [Accessed 30th December 2012]
Harlow, R.F. (1942) Public relations in war and peace, New York, Harper and Brothers.
Heath, R. (2000), Handbook of public relations, Sage Publications, London
Holmes, P, 2010, A stake through the heart of advertising value equivalency, [Online], Paul Holmes, Available at: http://www.holmesreport.com/blog/index.cfm/2010/6/21/A-Stake-Through-theHeart-of-Advertising-Value-Equivalency, [Accessed 22nd December 2012]
IPR Institute of Public Relations. (1958) A guide to the practice of public relations. London, Newman Neame.
Jefkins F, 1992, Public relations technique, Butterworth-Heinemann LTD, Oxford
Kimmel A, 1996, Ethical issues in behavioural research, Blackwell Publishers, Oxford
L’Etang, J. (2004) Public relations in Britain, Mahwah, NJ, Lawrence Erlbaum Associates.
Lincoln Y, Denzin N, 2003, The landscape of qualitative research, 2nd ed, Sage Publishers, London
Lindlof T.R. and Taylor B.C., 2002, Qualitative communication research methods 2nd ed, Sage Publications, Thousand Oaks, CA
Marketing Journal Blog, 2012, What is the average bounce rate?, [Online], Marketing Journal Blog, Available at: http://www.marketingjournalblog.com/2012/01/what-isaverage-bounce-rate.html, [Accessed 21st April 2013]
Marston, J. (1963) The nature of public relations, New York, McGraw-Hill
49 | P a g e
Maxfield M, Babbie E, 2012, Basic research methods, Cengage Education, USA
McCoy, M, Hargie, O. (2003) Implications of mass communication theory for asymmetric public relations evaluation, Journal of Communication Management, (4), pp 304–16
McGuire, WJ. (1984) Attitudes and attitude change, in Handbook of Social Psychology, II, 3rd ed, ed.
Merriam S, 2009, Qualitative research, Jossey Bass, USA
Miles MB, Hubberman AM, 1994, Qualitative data analysis, 2nd ed, Sage Publications, London
Milyankova, R,(n.d), Choosing research & approach, [PowerPoint Slides] Russia: International University College, Available at: www.iucedu.eu%2Fgroup%2Fsem1_L2%2F2012%2520PDEVR%2F_7_PDR_Choosing%2520str ategy_method.ppt&ei=vq3uUP3VBseY1AW27oDACg&usg=AFQjCNEQBgN32qYP3KAk _ONUIzUPYL32pw&bvm=bv.1357700187,d.d2k, [Accessed 10th January 2013]
Morehouse R, Maykut P, 1994, Beginning qualitative research, Falmer Press London
Nichols, B. (2013) PR strategy part V: research & evaluation 2, [PowerPoint Slides], Presented at MG606 at Buckinghamshire New University.
Paine KD. (2011) Measure what matters: online tools for understanding customers, social media and key relationships, West Sussex, John Wileys & Sons LTD
Palvik, J, V. (1987) Public Relations: what research tells us, Newbury Park, NJ, Sage
Philips, D. (2001) The PR evaluation evaluationists, 6 (14), p2
PRCA, 2004, What is PR, London, PRCA
PRmoment.com, 2010, The AMEC principles of PR evaluation, [Online], Ben Smith, Available at: http://blog.prmoment.com/the-amec-principles-of-pr-evaluation/ [Accessed 13th December 2012]
50 | P a g e
PR Week, 2012, Measurement in pr has ‘not changed in 20 years’, [Online], Sara Luker, Available at: http://www.prweek.com/uk/news/1139832/, [Accessed 3 rd January 2012]
Riley J, 1996, Getting the most from your data, a handbook of practical idea on how to analyse qualitative data, 2nd ed, Bristol, Technical & Educational Services
Robson C, 2002, Real world research, 2nd ed, Blackwell Publishing, Oxford
Romy R PR,(n.d), How to measure PR effectiveness, [Online], RomyR PR, Available at: http://www.romyr.com/PRinsights/PR_effectiveness?cs_lang=en, [Accessed 28th January 2013]
Sage Research Publications, n.d, Semi-structured interviews, [Online], Sage Publications, Available at: http://srmo.sagepub.com/view/the-sage-encyclopedia-ofsocial-science-research-methods/n909.xml , [Accessed 29th April 2013]
Saunders M, Lewis P & Thornhill A, 2003, Research methods for business students, 3rd ed, Harlow, Pearsons Education LTD
Scott, DM, 2010, The new rules of marketing & pr, 2nd Ed, New Jersey, John Whiley & Sons
Sheldrake, P, 2011, The business of influence, New Jersey, John Whiley & Sons
Silverman D, 2010, Doing qualitative research, 3rd ed, Sage Publications, London
Strauss A, Corbin J, 1998, Basics of qualitative research, 2nd ed, Newbury Park CA, Sage
Tench R, Yeomans L. (2006) Exploring public relations, 2nd Ed, Harlow, Essex, Pearsons Education LTD
Theaker A.2012, The public relations handbook, 4th Ed, Oxon, Routledge
Turner S, 2011, Why PR evaluation needs objectives to succeed, [Online], Freshfield Communications, Available at: http://www.freshfield.com/pr-evaluation-debate-takesnew-direction/, [Accessed 3rd October 2012]
Underwood J and Stevenson J, 2013, Public Affairs, S Mashru, 19th March 2013, 10.45am
51 | P a g e
Watson, T, Noble, P. (2007) Evaluating Public Relations. 2nd Ed, London, Kogan Page.
Watson, T. (1994) Public relations evaluation: nationwide survey of practice in the United Kingdom, Paper presented to the International Public Relations Research Symposium, Bled, Slovenia, July 1994.
Watson T. (2012) The evolution of evaluation – the accelerating march towards the measurement of public relations effectiveness. Public Relations Review. 38 (3), p390398.
Wilson, Alan, 2012, Marketing research an integrated approach, 3rd ed, Harlow, Pearsons Education LTD
52 | P a g e
Appendix 1.1 - Pyramid of PR Research
Watson and Noble, 2007, P85 1
Appendix 1.2 - Interviewee Profile Table Interviewee
A 2 Respondents: 1, 2
1) Managing Director (PR) 2) Account Manager (PR) B) Vice President (PR) C) Board Director (PR) D) Corporate Affairs Partner (PR) E) Junior Account Director (PR) F) Head of Analysis and Insight G) Director of Communications H) PR & Comms Specialist. I) Head of Social Media
B C D E F G H I
Appendix 1.3 - Coding Table
Gender 1) F 2) F
Years in Industry 1) 10 Years 2) 5 Years
B) M C) M C) M
B) 13 Years C) 13 Years C)24 Years
E) 6 Years
F) 11 Years
G) 6 Years
H) 7 Years
I) 12 Years
53 | P a g e
Methods of Evaluation
Quote to use
A- Line: 11
A- AVE - Advertising Value Equivalent.
E- Line: 2-3
E - AVE, Circulation, total number of cuttings.
F- Line: 2-6
F- Start with what is cheap and easy, media content analysis is often used as a basis for the success of PR.
G - Line: 2
G - AVEs probably, although I have no Interviewee G states that they evidence of this. feel AVE is the most common method used.
Interviewee A and B have taken opposing views on the B- Line:3-4 B- Depends on client needs, measure requirements of evaluation. Watson, 2012: It has been found that one what they want, not what we want. Interviewee C has taken a accurate key method which looks at outcomes similar approach to that of compared to outputs is very difficult to find. C- Line: 7, 10- C- one size doesn't fit all, what agency B where they measure 12, 14-15, methodology suits your business, we on the client's needs. Gregory, 2010: Without doing accurate 17,20, 27, 44- use cost per 1000, Apollo measures evaluation, PR will be at risk of being a 45, 49, 55, share of voice, Metrica is very Specialist agencies who communicator that is intangible or â€˜far too 64,-67 expensive. evaluate work are very important to be measuredâ€™. expensive, PR's need to be D- Line: 7D- Media monitoring, measuring flexible on the methods they 10,12 sentiment, and reputation changing use for clients studies. Interviewee D has worked from a corporate angle, where Nothing Obvious the need to measure reputation is key. Interviewee E offers few forms of evaluation, with basic results monitoring outputs. Interviewee F works for a media evaluation company who feels that the delivery of the content is key.
54 | P a g e
H - Line: 2,3
Methods of Evaluation
I - Line: 2-5
I - The most common method of evaluation unfortunately is still based on OTS (opportunity to see). This is probably based on clients’ familiarity with the notion of “reach” from ad campaigns and a requirement to translate PR into comparable return on investment.
A - Yes
B - Line: 7-8
B- Yes- if you don't set targets, what are you measuring?
C - Line 74,79-80
C – Absolutely, if not you are going blind...
D- Line 17
D - When running specific campaigns yes, but when in corporate there is a continuous measurement looking at reputation changes.
Do your methods require goals and
H - Google analytics for the website, Interviewee H specialises in digital and social media PR, retweets on Twitter, likes on Facebook, Google alerts, hash-tags on they analyse the engagement of social media. Twitter and Instagram.
objectives to be set at the start of the campaign? E- Line: 7-8
Interviewee I feels that the OTS method as it can give the client an understand of who they have spoken to.
All interviewees are in agreement here, except interviewee G who feels that even AVE's do not need goals and objectives. Need solid parameters to measure against.
E - Set with clients in accordance with Not all campaigns need to business goals. have set goals and campaigns.
Tench and Yeomans, 2006 Expressed that evaluating Public Relations efforts is extremely important, as it reflects effectiveness, professionalism, whilst giving accountability for the efforts.
55 | P a g e
F - Line 9-13
F - If done properly, depends on the type of campaign.
G- Line: 5
G- No, not at all.
H - Line: 6,8
H - Yes. For example, if we begin a social media campaign and want to build a buzz about a new product, we will set a numerical goal in terms of retweets, likes, hashtag uses.
I- Line: 8-14
I- It often does, although OTS can be easily gamed by securing online coverage. If OTS is a key measurement then demographic should be part of this from the outset (i.e. the right people see the coverage). Also a list of key media should be defined up front to avoid coverage being secured in nonrelevant media and being counted in the final evaluation tally. Often, there is a volume target set at the start of a campaign (e.g. 15 x pieces of national coverage, 25 x pieces of regional, 2 x broadcast, etc.).
Social Media means that it is possible to set numeric goals in terms of engagement.
The OTS system is required to set goals and objectives, as the target audience will need to be addressed effectively.
56 | P a g e
A- Line:35-39 A- Product PR is hard to get negative PR, generally always positive, if something doesn't work it is replaced, not recalled.
B - Line: 13 Campaign Failure Recalls
B- Only add if the content is suitable.
Interviewee A & B have agreed that as long as the plan was still valid the content can be tweaked.
C- Not so much gone wrong, but more C- Line 93-97, of looking at it around a month in and Interviewees C & D have 99-103 PR Week, 2012 quote ‘A real problem is the saying, this will not achieve what we stated that if something does PR industry thinks its job is to get media want to, let's review it. It's about not work how they wanted it coverage or run a good event – that’s not the understand what success looks like. to or if something else is case. The PR industry’s job is to influence and change behaviour of consumers and D- I've advised the client to stop doing happening that will affect the D- Line 41-46 something, because the original idea stakeholders.’ outcome, they will advise their clients put their plans on was not a good one. hold. So we advised them to stop running the campaign and re think the offer to the market. E- Line: 13 F- Line: 18-21
E - No F- PR doesn't work with recalling, it reprioritises, if something doesn't work it should turn to damage limitation.
Interviewee E has not had to recall or alter any campaign. Interviewee F claims that should things go wrong, PR should concentrate on containing the damage, not
57 | P a g e
worry about fixing it.
H - No
Interviewee G and H has never across this in their career.
G- Line: 10 Failure H - Line: 15 Recalls I- Line: 22-25 I- No. Often, the strategy or KPIs are revised depending on initial results. So, if a brand has a new product range, KPIs could be increased. Likewise, if a key component of a campaign is no longer possible to activate, KPIs can also be rejigged to make the most of the campaign fee. A- Line:44-47, A- Some clients want to be 54-55 everywhere, and they want to be where you can't always get them. B - Line: 1618
B- Mass Media here is not as large as others abroad.
Interviewee I - has not had to recall a campaign, as the KPI's are active and can be increased should the campaign fail.
A believes there is a mass media Tench & Yeomans, 2006 state that PR's Interviewee B & C do not feel overemphasize the value of print media to that the UK has a mass media make it look competitive with digital sources. scheme anymore.
C - Line: 117- C- Target specific titles, choose them 121, 123-124, as clients customers are reading them, 128-129 Germany has much more media than Interviewee D claims that due Mass Media affect us, doesn't impact it. to the mass media effect in on way PR is seen in the UK and around the world, the business place? D - Line: 57, D- People are more aware of what can if things go wrong, they go 59-60 go wrong and what can go right. wrong quickly, however, good Globalisation of media has made PR news also travels very quickly. an important.
CIPR Share This, 2012, has stated that there are five steps that should be taken when it comes to carrying out digital PR evaluation, these steps include: identify the influences, check for the current evaluation and error rates, what and when engagement occurs,
58 | P a g e
E- Line: 16-17 E- I think the large choice of media allows businesses to reach people in different ways and allows PRs to be truly creative and proactive in their pitching of ideas.
Interviewee E feels that the looking at the number of data sources picked Mass Media acts as a positive up, and understanding how many could be missed, and finally, ensuring the correct for the industry. target is being met. Interviewee F feels that there is not a mass media, and that it is more important to F - Line: 24-31 F- Mass media is not connected, social media is more key, people can talk to concentrate on social media and digital forums, as this a business directly. affects the way PR is seen in the business place.
Mass Media affect on way PR is seen in G- Line: 13-15 G- Not sure what you mean - I do the business place? think that the media is influential on the way PR is seen in a number of ways. If you mean with regards to evaluation, I do not see any connection.
H - Line: 18,22
H - The industry of PR can have a negative reputation because many people who are not a part of the industry think we â€œspinâ€? stories. This is not true, because most PR professionals are well-versed in whatâ€™s ethically right in terms of communication with the public and a
Interviewee G feels the mass media does not affect the way PR is seen.
Interviewee H feels that PR will never have a negative image, as PR is not about spin, it is, it is about being ethical.
59 | P a g e
private organization. I do not call it spinning, it is more like finding whatâ€™s newsworthy and running with it. I- Line: 28-31 I- There is still a misconception of Mass Media affect what PR actually is both within the on way PR is seen in wider public and within businesses in the business place? the UK. Particularly in the small business space, PR is seen as a poor relation of advertising whereas the reality is that it can often deliver better long-term results for companies when compared against paid-for media. A- Line:76-77 A- No, there is room for manoeuvre, but no generally no.
Unified Method of evaluation for all clients?
B - Line:20-23 B - No silver cloud, everyone is different, where many agencies go wrong. C- Line: 172
C- It depends on the clients budgets and needs.
Interviewee I feels that PR gives better long term results compared against paid for advertising.
Interviewee A treats all clients equally, giving them one form of measurement which they say their clients know and use. Interviewee B and C base their plans on what the client wants to achieve and how much they want to spend.
Interviewee D used evaluation D-Line: 93-96 D- I got very involved with evaluation to measure the change in from the start and try to push for share prices following a PR metrics that allow reputation change campaign by looking at tracking as well.
60 | P a g e
reputation change. E- Line: 20
F- Line: 33-34
Unified Method of evaluation for all clients?
E- No, we tailor our methods to their reporting needs and business processes.
Interviewee E is able to measure what the client wants measured.
F- Smart PROs are using best practice frameworks fitting into clients needs.
Interviewee F feels that some PRO's do have one method, but the better ones can measure anything.
G- Line: 17-18 G - No, there are many methods, and many agencies have their own methods that they have used themselves. AVEs do seem to be the most prevalent however.
Interviewee G feels that the AVE system is still the widest used.
H - Line: 24,25
H - Yes. We have our clients fill-out a survey about their experience, and if we delivered the results they wanted and expected.
Interviewee H's agency uses surveys to establish what clients want from their evaluation results.
I - Not in my experience. I know that Interviewee I believes that PR we try to use qualitative and should have different quantitative methods of evaluation methodologies for evaluation depending on the mediums. for all clients but have to tweak our evaluation methodology to align them with our clientsâ€™ business objectives. So, if a client is interested purely in inbound clicks to their website(s), our
61 | P a g e
evaluation must prove that we have delivered on this regardless of how many pieces of coverage we secured. A- Line:89-95 A- Yes, few tools available, estimate, language business people understand, All interviewees are in they've used advertising so they know agreement here, all agree that how it works. PRO's are being too kind by giving figures that are B- Yes they do, and they have been increased, meaning that they B - Line:26 caught for doing this. are not giving clients the true value of the PR to their business . Harlow, 1942 states that the system of C: I'm sure there are some people like PRO's increase the C- Line: 145, â€˜pepping upâ€™, this was used when images had this, but we often report to a 149-151, been published, with the PRO enlarging the marketing manager, so we try to make When reporting to a value of evaluation them look good, not us. marketing manager, they need image, thus claiming more space, making it seem the coverage was much larger than it D-Agencies & In house are guilty of to make sure their budgets to make themselves D- Line: 67-68 this, over enthusiastic, media volumes don't get cut, so a degree of was. with multipliers. reality needs to be kept. look good. E- No E- Line: 23 F - The one way to do this is by using F - Line: 37-43 AVE's, which are nonsensical, does not answer the question, no research can be done to find out what multiplier used for AVE is accurate due to too many parameters. G - Line: 2124
G- There is one example of how this seems to happen - many PROs multiply the value of their AVEs by 3,
Interviewee F feels this is only done if PROs use the AVE system, and believes that PR holds more value than advertising, but is not sure how much. Interviewee G has stated the AVE method, allows for this, therefore inaccurate.
62 | P a g e
justifying this by claiming that PR is 3 times as effective as advertising. This seems to be completely unscientific and just there to increase the value.
PRO's increase the value of evaluation H - Line: 28
H - Not that I know of, that would be unethical.
Interviewee H feels that ethics comes into this in a big way, therefore, ethical issues have been identified.
to make themselves look good.
I - Line: 49,50 I - Part of our job is to work to whatever methods of presentation best suit our clients. That said, we would never fabricate results.
A- See it computerised, complicated due to high number of parameters.
B - Line:29-34 B- They can't be changed, only improved, putting methods together, altering methods to suit client needs and wants, make it cheap as possible, cost does not affect effectiveness. Also the need for human interaction is vital, a computer can't read the tone How and why would of an article. you change C- Line: 193evaluation models? 194
C- You can't change them, merging them will help make it clearer to clients.
Interviewee I believes that this is not an industry issue.
This question was asked to establish what would be done differently in possible. Interviewee A believes that if possible computerisation should take place to make life easier. Interviewee B believes that this shouldn't happen as computers can't be as accurate as humans. Whereas interviewee A would like to see computerisation take place.
Nothing obvious Recommendation: Keep AVE to measure outputs internally, but give clients information on how the outcomes will affect
63 | P a g e
D- Line: 101107 E- Line: 2628
F - Line: 4548
How and why would G - Line: 2629 you change evaluation models?
H - Line: 31
I - Line: 52-58
D- Need to find the link between good PR and reputation change, leading to the ultimate bottom line to the client. E- More emphasis on sales as a result of a click through from online coverage. F - By looking at how PR can help the business, thinking upfront means you don't need to be spurious with evaluation. G - A method based on the full length of the campaign, which focuses on the initial requirements, and whether the right people we actually reached. The AMEC meeting in June is expected to put forward a new form of evaluation which I would suggest will be useful to have a look at. H - I would not, I think they do a great job. I- I would fully embrace a uniform system that takes on board the Barcelona Declaration of Measurement Principles that set out to include volume as well as impact of campaigns. These principles go beyond pure OTS and offer a
Interviewee C and D agree that merging multiple methods together will allow for further clarification of the outcome for the client.
their business, product PR - check to see if any enquiries are being received for campaigned products.
Interviewee F feels that the PR industry should start with brain storming, to understand every business is different with different needs.
Interviewee G would like to see evaluation based on whether the correct people were reached.
Interviewee H believes that the methods used in their agency answer the question effectively. Interviewee I feels that PR should have a unified evaluation around the Barcelona Principles.
64 | P a g e
better 360 degree understanding of how effective a PR campaign is. It also takes into consideration social media and online coverage which goes beyond the pure vanity of the number of Likes and Followers (which can be unscrupulously bought) and focuses more on actions such as engagement. A- Line:123
Importance of Evaluation
Both interviewees A & B see evaluation as an important aspect, however, agency B B - Line:40-44 B- It is very important, prove our existence, also CEO's do not have time have gone into evaluation in a way t hat allows their clients to read a 140 page report, they need a simple dashboard with the pros and to see the bottom line, only knowing the main cons to make their decision. information. C- Line: 82,83 C- If you are sure what success looks Interviewee C and D are like, it is possible to evaluate discussing the significance effectively. between finding out what success looks like and how D- : Yes we have very few tools, and D-80-81 much clients want to spend because we haven't found, it's called on evaluation. scientific methodology yet, that is cheap and easy to apply. Interviewee E feels that E- Line: 32-33 evaluation makes people work E-Very Important - allows for people even harder. to work to their fullest. F - Line: 50
F - Very
Interviewee F is very upfront about their feelings on the
Tench and Yeomans, 2006: Shows professionalism and gives accountability.
No further links have been found.
65 | P a g e
subject. Importance of Evaluation
G - Line: 3133
H - Line: 3337
G- Quite important, but perhaps its importance is over-stated. The PR industry does very good work and that should not be forgotten just because the means of measuring it are not totally accurate.
Interviewee G feels that evaluation is important, but there are more important tasks in PR.
H - Of course it has to be one of the Interviewee H feels that it is essential to understand what most important aspects of a campaign the client needs. This will because you need to be able to see show in the understanding of what type of impact your campaign the impact on the public and initiatives had on the public. Without clients. evaluation, it would be impossible to measure return on investment as well. Not to mention, the client needs to see proof that you are doing something right with their campaign.
I- Incredibly important. Without I - Line: 60,62 measuring success, you can’t justify spend. That said, some things are more difficult to evaluate than others. For example, measuring people’s warmth towards a brand (beyond £) is difficult, although not immeasurable.
Interviewee I feels evaluation is very important, as it allows the PRO to measure success. However, things are more difficult to evaluate than others, such as reputation change and intangible factors.
66 | P a g e
A- Line: 128129
Understanding of Evaluation
A- Yes, if you have an understanding of you, you will do it better, as you know why it needs to be done.
All interviewees agree that if a person doesn't understand B- If you don't understand something, why evaluation is being done, B- Line:47 you won't do it well. they won't be able to do it properly. ‘A real problem is the PR industry thinks its CIf you don't know what you're C- Line: 205job is to get media coverage or run a good It is down to agency to teach 206, 211-214 doing you are going blind. event – that’s not the case. The PR industry’s their staff how to evaluate D- It depends on how good their job is to influence and change behaviour of properly, and look at the D - Line: 125degree is, experience counts as some trends that are given from the consumers and stakeholders.’ 136 journalists might be negative, or not journalists, in terms of their as negative as normal, making the language and tone. (PR Week, 2012). coverage good, but new PRO's may not understand that. Interviewee E has discussed E: Line - 3637
F - Line: 53-54
G - Line: 36,37
H - Line:40,42
E - If PROs understand their client's needs they can provide a better all round service. F- Results that show better what PR has done to move the needle for the business? Yes.
the industry as a whole as it is believed that the industry can work better with a greater understanding.
G - I suspect that this may not be the case, and that good PR skills are not linked to evaluation.
Interviewee F has stated that if everyone understands evaluation, everyone can work to a better goal. Interviewee G feel that PRO's should not need to be good at evaluation.
H - Well, if you are a professional, you must know what evaluation your work is a must. For example, if a pro
Interviewee H states that if a PRO is professional, it will be possible to understand
67 | P a g e
doesnâ€™t evaluate their process, they will not know what worked and what did not work. Understanding of Evaluation
Interviewee I feels that I - Line: 65,65 I - Yes. Evaluation isnâ€™t an end-process evaluation should be the most that you do post-campaign. It should important process, in PR, as it be an ongoing process that is central helps them show that to the execution of PR strategy. PROs everything they do has a that get this, will make sure that positive impact to the company. everything they do has a positive impact on evaluation. A- Line: 143147
Various Media Evaluations
evaluation and the process. If the PRO is not professional, they will not understand the pros and cons of it.
A- Same way, debate on how much AVE should be worth, depends on unique visitors which changes monthly.
B- Line: 50-51 B- No, as with some clients, they like to see where all their links have come from, so you need one form of evaluation to make sense of it all. C-Line: 227
C- Yes I'd agree that different medias have to be evaluated in different ways.
D- 139-141 D- Different media's give different things, outcomes may be the same but they have to monitored separately.
Interviewees A and B recommend it is best to evaluate PR with one methodology for consistency. However, interviewees C and D take opposing views, with them recommending that different media's require different methodologies as they are achieving different tasks.
During the transition to the digital world, it has been found that word of mouth, or interpersonal influence has become more common with the use of social media, allowing the public to talk to their peers about a certain product or company easily, from almost any location worldwide. (Watson and Noble, 2007, P14)
Interviewee E claims that they can measure click throughs
68 | P a g e
E: Line - 4142
F - Best practice should not allow for changes, even though metrics change.
Various Media Evaluations
E - Measure click throughs but only some clients have this technology to do so.
F - Line 57-58 G - I think evaluation methods do G - Line: 40,4 differ for different media.
H - Line:45-47
I - Line: 70-73
online but only with certain clients. Interviewee F feels that PRO's should measure everything the same way. Interviewee G feels that evaluation already does differ.
H - No, because we mainly Interviewee H feels that as concentrate on digital marketing and they concentrate on online PR PR and most of our impressions come they use one method, unless clients ask for their evaluation digitally. Unless a client asks for a to be done in a traditional traditional PR route for their manner. campaign. I- For clients that have an onlinecentric model, it is easier to show Interviewee I feels that clients impact of PR through clickable links in with online campaigns can obtain accurate and easy coverage that can be measured on a measurements, as there are source by source basis. These clients more methods to measure seem to be more switched on to this accurately. other forms of quantitative evaluation (SEO, PPC, email newsletters, etc.)
69 | P a g e
A- Line: 169170, 172
A- Still have to justify existence, but more defined targets, with more openness.
All interviewees agree with the question, with the general consensus being that in house teams still take up space and B- Line: 54-56 B- Yes, we have to make ourselves need to prove their look good to clients and our bosses, In worthiness in the house teams have to make organisation. themselves look viable to their In house and agency bosses. differences However, due to the cultural C- Lines: 244C- Massive Difference, marketers have difference, it is possible to 253 assume that in house teams 20 things to do a day and PR is just undertake a more intensive one of these things, we make their approach to PR, in terms that lives easier. D- Lines: 155the work that they achieve 156 will be of a higher standard D- Simply - agencies look for outputs, and specialism compared to In house looks for outcomes. agencies.
E- Line: 46 F - 61-65
E - Only worked in agency F - No clear distinction - good and bad in both.
G - Line: 44,45
G - I think that agencies â€˜ownâ€™ evaluation and that in house teams leave it to them to do most the evaluation work.
H - Line: 5056
Anne Gregory, 2010, states that if PRO's are too open about evaluation, clients may feel it is easy and then take the account in house and do it themselves.
Interviewee F feels that both in house teams and agencies would encourage businesses by measuring accurately, however, some still try to impress with big figures. Interviewee G feels that in house teams do not evaluate, but it is only agencies that do so.
H- I work in an agency type of environment, so I am used to dealing
70 | P a g e
with a few clients at a time. It’s important to discuss and understand Interviewee H feels that what the client wants and execute agencies need to explain what they need. These are clients evaluation to clients, and for who demand results and rely on us to PRO's to understand what make a PR buzz in any industry they clients want from evaluation. are a part of. We survey the client, we measure the “buzz” digitally, and show the clients what worked and what didn’t work based off of our PR measurements. I feel like in house PR also use evaluations to show profit to their seniors. That would be the difference.
In house and agency differences
I - Line: 76-78 I - In-house teams tend to have more pressure on delivering comparable pound for pound ROIs on different marketing disciplines (SEO, Direct Marketing, advertising, PR, etc.) whereas agencies will focus on delivering excellent PR results. International Evaluation
A- Line: 183186
A- May use the same idea's but in a different method, using different multipliers, but culture can come into it.
Culture B- Line:59-61 B- Yes, as different media's have
Interviewee I feels that agencies have more pressure to prove every pound they spent works, compared to agencies.
Interviewees A, C and D all agree that local culture and ethics affect the way that evaluation takes place.
As these two agencies became international very quickly, there was an urgent need to send evaluation reports back to the US headquarters for CEO’s to see what was going
71 | P a g e
different values that we may not have here, and if you insult a colleague abroad by saying this they won't be happy. However, interviewee C is part of a large network which C- Lines: 276CSome cultures are more laid back makes partner agencies use 278 so they don't do it on time, but the their methods of evaluation methodology is generally the same when it comes to work around the world. sharing. International Evaluation Culture
D-Lines: 168D- If the media allows it they will use 169, 172-174 one, if you want to compare two different countries, you need to be using the same yardstick. E - Line: 49E - Only one client US based, use 50 similar methods. F - Line: 68-70
G - Line 48,49
H - Line: 5962
F - It is important to use one method however, you have to allow for the local market adjustments.
on in their other offices (Watson, 2012:11)
Nothing obvious to promote the cultural differences.
Interviewee E uses the same methods for all clients, regardless.
Interviewee F feels that the framework should remain the same, however, with local culture coming into play.
G - I think it depends from one company to another, but there will be some instances of international companies using one method across the world.
Interviewee G feels that one method should be used, but there is room for other methods if needs be.
H - I definitely think that local culture and ethics make a huge difference. There may be agencies who doesnâ€™t properly measure their PR results, and in result, they may continue
Interviewee H feels that local culture does make a difference, and agencies which do not account for this, is making a mistake over and
72 | P a g e
making a mistake over and over again with different clients. International Evaluation Culture
Client Contract Renewals
I - Line:81-83 I - Global media trends and localised Interviewee I feels that one method should be used news will affect the effectiveness of internationally, as global PR but general universal PR media trends, however, the evaluation methods should be applied Barcelona Principles should be to all campaigns (see above: considered. Barcelona Principles)
A- Line: 199203
A- Best form of confirmation that things are going well and they value the work that has been carried out.
B- Line: 63
B- Yes! Nothing is better than a good pat on the back for your work.
C- Lines:281C - Definitely, A pat on the back is the 286 best form of evaluation, it means your work is not going unnoticed. D-Lines: 185D- Client doesn't need to understand 190 everything, that is why they hire an agency, as long as the agency is making money for the client they are happy.
All interviewees agree that this is the best form of evaluation, however, it still does not answer the questions, as some clients may change their agency to get a fresh perspective on their business.
Interviewee E feels that
Tom Watson found from a 1994 survey, that many managers believe the best form of evaluation is when clients renew contracts. (Watson, 1004)
73 | P a g e
E- Lines: 53- E- Best form as owners see the results, changing beliefs is more 56 important though.
F - Lines: 72- F - Yes, also when the client expands their network with the PRO, It doesn't 79 mean you are necessarily evaluating properly or affecting the target audience. Chemistry could come into play.
Client Contract Renewals
G- Line: 51
G - Probably not. No!
H - Line: 64
H - Well, they renewed the contract! So that should say it all.
I -Line: 85
I - No, but it always helps
changing beliefs is more important Interviewee F feels that the PRO can keep contracts by having a good relationship and understanding with the client, and not always about finances.
Interviewee G feels that the clients renewals would not be the best form of measurement. Interviewee H agrees with this comment. Interviewee I feels that client renewals are good for business, but not a good source of evaluation.
74 | P a g e
Appendix 1.4 - Sample The research sample for this study, was selected from PR agencies and in house teams, which have a population of around 9500 (Carey B, 2013) CIPR members, who will be based around the United Kingdom, as well as those from America, who were able to provide international insights, the study also contacted members of trade bodies and evaluators. This sample was created, by using the non-probabilistic, purposeful sampling method, as it allowed for the researcher, to select who to talk to. This study spoke to in house and agency PRO’s, who are CIPR members, and located in the South East of England. However, it excluded free-lance PRO’s, as they may not have the funding or skill set to use more accurate models. The sample also contained academics, who were teaching Public Relations at Buckinghamshire New University at the time of the study. Non Probability - This is where the respondents are chosen without the need of working out a sample size, for example, establishing to choose every 3rd male on the street. This is because the study will need respondents, who are from a specific sector, regardless of age, gender, or level of career (Maxfield MG, Babbie E, 2011). Purposeful Sampling Method - This is where the population of the sample, have been chosen from specific target, in order to gain a purposeful and meaningful result (Merriam, S, 2009). The limitations from this primary research were, that due to the allocated time scale, not all sizes of PR agencies and in house teams, could be interviewed, also, as some PRO’s may not have been available to answer questions, the interviewer, had to search for new respondents, the final limitation, was that the academics interviewed, will only be from Buckinghamshire New University.
75 | P a g e
Appendix 1.5 - Managerial Recommendations The managerial recommendations, which are made from this dissertation, are
To educate PRO's on the importance to set goals and objectives with clients at the start of every campaign, or client meeting;
To allow for PRO's working abroad to evaluate Public Relations in a manner which allows for local media culture and landscape to be accounted for;
To monitor how many hours are being spent on evaluation, to ensure that clients are not overspending on evaluation; and
To put the digital media evaluation process from this dissertation to be put into practice, as this is a low cost method, which can be trailed risk free.
76 | P a g e
Appendix 2.1 - Transcript - Interviewee Coding A (two respondents) R: This is the recording (erm) at 2.15 on the 20th March (erm) for the dissertation: 'Which evaluation method is the most accurate in the PR industry. (erm) Respondent 1 and respondent 2. Do you agree to be recorded? Respondent 1? 1. Yes R: Respondent 2 2. Yes R: Ok, firstly we will start with (erm), what method of evaluation does your company use (erm)respondent 1? 1.AVE - Advertising Value equivalent R: Ok and respondent 2 works for the same company so it's going to be the same? 2. Yes R: Respondent 2, do you feel this method requires goals and objectives to be set at the start of the campaign ? 2. (pause) Yes R:Ok, do you feel that you can measure against the during the campaign as well? 2. (erm)Yes (pause) because all publications, depending on their position of the magazine and things varies, so if we know what we are aiming for. 1. Yes definitely, although, I have to say, it is something we should do more of, the danger (pause) is that if you set targets of advertising standard, advertising value equivalent, at the beginning of a campaign, you might tend to be slightly more realistic and achieve a lot more than you target yourself to do, because you are worried of falling short. R: Ok 1. So I think it is a good thing to put in as an objective and KPIs, it could be, it could backfire on the client, so we don't always do it. Unless we have a very reluctant client and we want to convince them to have more confidence. R: Ok, and secondly, does (erm) your evaluation process run from the start of the campaign to the end? 1 & 2: Yes (Simultaneously) R: It does, ok, Say (pause) you send out a campaign to a journalist of group or journalists, it goes into the public domain, and it doesn't (erm) react in the same way that you've hoped, and its getting bad
77 | P a g e evaluation reports and advertising value equivalent figures, have you ever had to recall a campaign because of this. 1: No, I think because we are working with mainly product PR and not opinion PR, it's difficult to get bad PR. 2: Its generally always positive 1: Yeah, and if a particular product has been used a press release or campaign, doesn't work very well, we will replace it with something else , but we wouldn't recall it. R: Respondent 2, do you feel that the mass media (erm) has an impact on the way PR is seen in the clients eyes, do you feel, that its, seen that, the mass media makes PR a viable medium? 2: As in the reputation of PR? R: Yes 2. (pause) You do get a lot a lot of dubious people when it comes to PR, but I think it depends what sort of industry you are in, because, when it comes to people, celebrities and things, it is quite a negative thing, politics as well. But (erm), when it comes to products and brands, I think it's generally positive. R: And yourself (To 1) 1: Would you mind repeating the question again, it was quite a long one wasn't it. R: Yeah, the (pause) mass media, do you feel that it has an impact on the way that see PR and how it relates to the business objectives as such? 1: Right so because there is so much media? R: Yes 1: Because there is so much, some clients would want to be everywhere, which is unrealistic, and sods law is that they will want to be in the publications that you haven't managed to get them into. R: Yeah 1: So I think it can detrimental actually, in some countries there is nothing like the number of publications, and I think it's something easier there. Having said that, it does give us better scope. R: Yeah (erm), and also, going back to you saying the clients want to be where you can't get them at times, do you feel that clients are not disappointed with the results they get, but (pause), they're not as happy as they would be? 1: if they are not in certain publications? R: Yeah 1: Well I think most clients have a list of publications they want to be in, and that is part of our key performance indicators or brief at the beginning of the campaign. So we would try very hard to get
78 | P a g e them into them, so they're not disappointed. However (pause), nature of the PR consultants, is that you are good at spin, you can always spin the value of what, and perhaps change their mind and turn their disappointment into contentment. R: Ok, and also do you feel that if your client is in a certain regional of the country and you can get them into that Regional publications, do you feel that perhaps it's worth more than being in an national paper? 1: It depends how they sell, if they sell online, it is worth less, if they have a local business (2: Shops), yeah, then it is obviously worth, yes I think it is worth not more, but equal. R: Equal, ok, (erm) and also as the agency do you have one unified evaluation method for all of your clients, or does it vary from client to client? 2: We have a method as a rule, occasionally there is room for manoeuvre, but generally (erm) it is three times the AVE. R: Have any clients ever asked you to edit or alter the method. 1: No 2: No R: No, ok 1: Some are a little bit cynical about it and ask us to justify it. R: Yeah, 1: Especially in the current climate, when advertising rates are heavily discounted, so you have, before you start on this, you have to say, of course, we know no one pays rate card at the moment, so the advertising equivalent, is you know perhaps a little bit inflated, however, you know, we use it. R: (erm) Do you agree, that some PR officers, use evaluation reports and evaluation figures, to impress clients? 1: Yes! 2: Yeah 1: Because it is one of the few tools you have really. 2: It is a good way of proving value of PR, although it is an estimate (pause, erm), you know, everyone will see facts and figures, especially business owners, because they're all sales minded, but it's also a good back up. 1: And quite often they will advertise as well, so it's a currency they understand. R: (pause) If you could change the way that you measure PR, how would you, what would you do and how would you do it?
79 | P a g e 1: Well, I'd love to see it change really, because I do feel that this is quite an old fashioned way of doing it, but I can't see how you could change it. Ideally, you could, you know, something that is set up, computerised perhaps, to say this is, your client has this bit of coverage and it is so this on the page, or it's only one of three, but your client has a web address and the others don't or your client has something positive said about it and the others don't or whatever, so the value then is estimated on what that is worth, to the client (pause) and not just in what it would have cost to have bought that space. Because it is catch all, and I don't think it really works, but nobody has come up with anything better, especially for the type of PR that we do, which is product PR (pause) and I think it would be great if somebody could come up with some software or whatever. 2:Because it would get incredibly complicated, because you would have to have a value for an image, a value for a text and a value for the website, stockist details. 1: If it's positive or negative, if its glowing, if it's an endorsement or if it is just stating facts. 2:And its position in the newspaper or magazine. R: Ok, so you feel almost opinion polls, almost swaying towards that side, the opinion of the journalists how they write resonates, that sort of thing would impact? 1: Oh I see, I think that would help, again because of the type of PR that we do, nearly it's all positive, because by default the journalists are endorsing it, because they are using in the feature, its great if they say something really positive about the client. R: Yeah. 1: Specifically, which doesn't happen very often, if it does then that coverage is definitely worth a lot more than three times the advertising equivalent. But because the coverage is analysed and picked up by computer, (erm) it is very difficult to do anything more, and we could o it ourselves but we are not independent or unbiased it would be quite difficult to do. R: Yeah, ok so (erm), from what you've been saying, I understand that evaluation is very important to both of you. Both: Yes R: Do you feel that (pause), as you understand evaluation quite well, you provide better and more accurate results, compared to a colleague who doesn't have so much of an understanding, so like a someone who is new in the industry maybe or someone who is naive and does not want to understand it. 2: Yes, I think you need to have a good understanding of why it is worth what it is and its, you know (pause) how you work it out, the technical aspect, is important and also the reasoning behind it, because (pause) if they understand why they're doing it, then they are going to read it in a different way, because you have always got to (pause) take different things into account, like it might only be a 3cm mention, however, it's part of a big feature and it's the feature as a whole that has the power. 1: Definitely, I think also, it's great for trainees and graduates to look at coverage (pause), not only do you learn to analyse the evaluation, but you see the circulation of a title, so you see that
80 | P a g e something that has been in the Sunday Times Style Magazine is going to be worth more than something that has been in the Daily Express or whatever, so you suddenly get a much clearer view and a more perspective view of the current media landscape and if the circulations have gone down over the last few years and they fluctuate quite a lot, so I think it's a really good thing for trainees to do, as long as they are supervised, as they learn a lot about the industry as well. R: Ok, (erm) and also (erm), When you are working with clients who gain coverage online and offline, (erm, pause) do you measure it in a different way, are there any other ways you measure online PR? 1: There isn't at the moment is there? 2: No, we always (erm), have a little debate when we have online PR, because it is difficult, I mean you have your advantages because you have a direct link to the website, so people are directed to the client, (erm) and you have a mass audience (pause), but then are they actually just clicking on it and not reading it, you don't know. So it's sort (erm) of a difficult one. R: Yes, that's almost similar to traditional PR, where if someone has bought a newspaper, it doesn't necessarily mean they are going to read the article, so you would say it's the same issue really with online. It's harder to control? 1: It's harder to get the data isn't it, with the Daily Mail, you know that their circulation is 1.9million or whatever, but if it is on I don't know, House to Home or whatever, you know what their unique visitors monthly, what do you gauge it on, is it, unique users, visitors, is it, how do you measure it, you know it's very difficult. Is it unique number of click through's, you know you can't get that information. So it really is a difficult one I think,(erm)and also our cuttings agency don't give us the value of online coverage. 2: No they don't seem to value online or give you any details, so I think with online it can fluctuate so quickly. 1: Yeah 2: So one month could be 1000's higher than another month 1:Yeah, with national newspapers and magazines, you have the National Audit Circulation, which audits the national newspapers and magazines and whatever, so that you know exactly how many people should be buying that newspaper or magazine at any one time. So you have a much more accurate figure, and with online you don't have that. Somebody needs to work on this I think! It would be great! R: Ok, nearly done now, do you feel that there is a difference in the culture of evaluation between agency and in house PR teams? 1: I've never worked in house, they probably, could say it's quite similar Suraj, because they have to justify their existence, in the same way as external agencies do . They are probably given quite more defined targets though. R: Yeah
81 | P a g e 1: Because they can be much more open about their business plans and objectives than they can be to perhaps an outsider, maybe they put a lot more value if their target is to get into The Sunday Times Style, if they achieve that then maybe the value is inflated, if they get into the Hemel Hempsted Gazette, that doesn't have the same impact, so I don't know is the long winded answer to that, it would be interesting to find out. R: Yeah 2: I've never worked in house either R: You've never worked in house, ok, this one might be a little bit tricky to answer as well, International PR agencies, do you feel that they evaluate using the same method in all of their offices around the world, or do you feel local culture comes into it, I mean culture comes into everything in business really. 1: I'm sure it must do! I'm sure it must do, and they might have an advertising, they might use the advertising value equivalent, and might be 5:1 or 10:1 or 2:1 other than ours which is 3:1. We use which has been around for ages (pause), so they probably do, I'm sure they do, but it's probably not the same formula. R: Have you ever worked with any clients who are offer your PR services to, to market their products outside the UK? 1: Yes, we are at the moment actually. R: Ok, and how are you planning on evaluating that, is it the same way or?? 1: I think it will be the same way, because it is the way we work, so I think it's the way we understand and the way the client understands. R: Ok (erm), and finally, bit of a cheeky question this, but do you feel that the best form of evaluation for a PR agency is when the client renews the contract? 1:Sorry? R: When the client renews their contract, do you feel that this is the best form of evaluation, because they come back to you? 1: Oh I see. 2: It's the best form of confirmation (pause, erm), to say well done you've done a good job, continue! 1: (pause)Yeah, I think they've valued your work and your worth and they've said yes its working so we will renew, (pause, erm) I think we are reporting back all the time to our clients its almost an ongoing confirmation actually. 2: And ultimately our evaluation results and our coverage, and everything that works towards getting them to confirm it.
82 | P a g e R: Yeah 1: One thing you could perhaps think about Suraj in all of this is what targets for evaluation there is, you asked us that, there is sort of an industry standard of return on investment, for every pound you spend you get I think it's nine at the moment, eight or nine. R: Ok 1: Back, and its only setting targets at the beginning, so rather than saying we want to get ÂŁ500 or ÂŁ50,000 of coverage, what we want to say is we want an investment of 20:1 or something like that and you can really work towards that and keep a check on it every month. R: Yeah 1: And the other thing that we don't use very often, but I know other people do, is the opportunity to see target. R: Ok 1: Rather than just the financial (pause) value, so I would like this campaign to be seen by 3.5 million people and you know that might be possible within a couple of months or whatever (erm). So that is always very useful, but we don't always use it in our reporting, we are beginning to use it more actually. So it's not financial but it's another return. 2: Return on income is something 1: Return on investment 2: Return on investments is something that the clients really listen to as well, as it's R: It's hard facts and figures, you're talking their language. 2: And you're saying don't worry, we know what you've spent (pause) and this is what you're getting for it, so they know it's worth it. R: Yeah 2:So they know it's worth 10 times, or 20 times or even 30 times on what they've spent with us. R: Obviously you have clients who would potentially have internal marketing manager or internal PR manager, who has to report to their CEO's and CFO's, (pause) so would you have a hard time justifying (erm) your worthiness to them, so they can go back and really sound positive about it so they don't have their budgets cut? 1: We've got two or three clients like that haven't we. 2: Yeah, it's sort of just an (pause) age old thing with business isn't it, you always have to justify. 1: They have to justify their position, in the main they want to use a PR company, because, they don't really have the time or inclination to do it themselves, and the danger in the recession is that clients think ok we have so and so in our marketing department, they can do the PR, its really easy,
83 | P a g e but they don't have any tools or resources that we have, so yeah I think in the main they are happy to use us. So they try and positively (pause), what's the word, sort of sell us in I suppose, I think they are on our side. R: Because you're making their lives easier? 1: Yeah 2: And PR is a valuable part of a marketing plan 1: Its essential, (pause) it's the only important part. R: Ok, that's great! Thank you for your time (pause), I don't know if you want to add anything else? 1: It's a tricky but really interesting subject. R: Yea, it is very, because there are so many internal agency methods, so it's difficult to find out which one is going to be the most accurate. 1: Ours is a bit different because it is product PR, but I don't know if you were doing Hoover or Panasonic or something, which I know are products as well, there is good and bad press. 2: It's such a huge brand that people talk about them in general and their products . 1: it might be quite interesting to talk to, I don't know if it's possible, but someone who does PR for a political party. R: Yes, I am actually speaking to some people , another agency who deal with public affairs . 1: Excellent! R: I am talking to them 1: That would be a good way. R: Definitely, because they evaluate it very differently. 1: There is obviously loads of coverage they put no value on at all because it is negative R: Yes , yeah 1: Is that ok? R: Yeah, that is great, thank you for your time and thank you for your insights. 1: Thank you
84 | P a g e
Appendix 2.2 - Transcript - Interviewee Coding B 1. What method of evaluation does your company use? My organisation does not use a single type of evaluation, it is dependent on the client's needs, it is possible to measure what the clients, not what we want. 1a. Does this method require goals and objectives to be set at the start of the campaign? Yes! If you don't set targets, you could be measuring anything, just because we got 250 pieces of coverage, what does it mean in a business sense? 1b. Does the evaluation process run from the start of the campaign to the end? We begin our evaluation process in around the first month until the campaign ends. 1c. Have you ever had to re call a campaign as measuring it when it was live gave a negative result? No, we would add to it, but that is only if the content is not suitable. 2. Do you feel that the Mass Media or the large choice of media has an effect on the way that PR is seen in the business place? There is not really such a thing as mass media in the UK anymore, compared to most other countries including Eastern European Countries, China, Russia, India, USA as well as many others have much more choice of media outlets compared to ours. 3. Does your company use one unified method of evaluation for all clients? Again - no, our evaluation model depends on the campaign and what the client is looking for... There is no silver cloud when it comes to this, and this is where many agencies go wrong, as they are trying so hard to justify their existence figures they give may not mean anything to the client. 4. Do you agree that some PRO's increase the value of their evaluation reports to impress clients? Of course. However, many PRO's have been found out for doing this. 5. If you could change your company's method of evaluation what would you do and why? I don't think we can change them, they can be improved and that is by putting more than one method together to make results stronger. Also, it is important not to over spend on evaluation, many clients want figures, but they don't realise it can be costly and effective, or cheap and effective, in this case cost doesn't really impact results. I feel the average spend for evaluation should not exceed 15% of the overall budget, where as many cases are over 40%! The way we tend to work is by sitting down with a client, and establishing which 50 print titles are most important to them and why. By doing this, we can then establish which ones are the most influential, this means that other publications will follow what they say, therefore there is only the need to evaluate these top 50 titles. 6. How important do you feel evaluation is? It is very important, it is there as we do need to prove our existence to clients, but it has to be done realistically. It is important as many CEO's don't want a 140 page report on how
85 | P a g e campaigns have been, they want a simple dashboard where they can see what it is that is going on, with key facts and figures saying what went well and what didn't. 6a. Do you feel that PRO's who understand evaluation get more accurate results than their colleagues who don't have such an understanding? Yes - if you don't know what you're doing, you won't do it very well. 7. When working with clients who gain coverage on various media's away from print does your evaluation model differ? No, as with some campaigns, such as online, you can see where click through's come from. 8. Do you feel there is a difference between the culture of in house and agency evaluation? There is, we have to report to our bosses and clients, who have to report to their bosses, so we have to help them to look good rather than making us look good. IN House teams need to make themselves looks financially viable to their bosses. 9. Do international companies use one method across the world, or does local culture and ethics make a difference? Yes - for example, in China, where press release reprints are worth a lot , here they are not, and if you tell a Chinese PRO this, they will be very offended as it is their system. 10. Is the best form of evaluation when the client renews the contract? Yes! Nothing is better than a pat on the back for good work.
86 | P a g e
Appendix 2.3 - Transcript - Interviewee Coding C R: Ok, this is the recording for (erm) the evaluation in public relations dissertation, (erm) it's the 5th April at 11 o clock, (erm) firstly (pause), do you give your consent to be recorded. 1: Absolutely R: Ok that's great! (erm) firstly if we begin with (erm) what method of evaluation your company uses. 1: Ok, so the reality is, we don't just use on form of evaluation, we've got a variety of clients, from B2B, to tech, and consumer, so they way that say Go Ape Tree Top Adventure would evaluate their PR is very very different to Information security clients Kaspersky, if that makes sense. (Pause) What I've done, I've listed down a couple of the most popular, and what we do, is we sit down with the client, either when we're pitching or we've won the business and say, there isn't one size fits all, but we'll work to whatever, methodology suits your business. R: Yeah 1: We use (pause) quite frequently, opportunity to see, so OTS, looking at the circulations, the amount of online traffic and stuff like that. R: Ok 1: We use AVE, which I'm sure you've come across R: Yes 1: Which is now being see as quite a dated methodology, but it is still a way that some clients like to work, they feel familiar with it, they understand it, we (erm) frequently use cost per 1000 R: OK 1: So we will look at the actual spend that (erm) a client is spending, and then look at the amount of people that they've managed to get in front of, then you can start wheeling it down to every 1000 it costs 23p or something like that. That (pause) proves reasonably popular, because (erm) it means that the client can go back and actually give a bit of ROI to their board, if that makes sense. R: Yeah 1: A lot of them, just simply want to go by volume, volume of coverage in their (pause, erm) tiered titles. So when we start the campaign we create a press list or the (erm) online titles that we want to go in for, and they will agree a goals list. So these are like number one this is where we are aspiring to be, so some clients might say, we want to get five pieces of coverage in that every quarter, we expect 20 in the next layer, and 30 in the layer below that. R: Ok
87 | P a g e 1: So, (erm) again targets are agreed, it's important they get, likewise you know, and we're not just talking, say just like the, (pause, erm) biggies, we want to be on the BBC website, or broadcast or anything like that. For some clients it's just really really important that they are in their tier, sorry, their trade publication. R: Yeah 1: Every single issue. R: I suppose that that would be worth more to them. 1: It's worth more to them, it gets in front of all of their customers, if they're not there, then that's when they see an actual dip in the amount of phone calls they have. R: Ok 1: Increasingly we're seeing traffic, as a massive thing, so people say, I don't actually care what pieces of coverage you get me, or where it is, all I'm interested in is the amount of traffic you can drive to my site or you can drive to this specific page. Because that is giving me leads. R: Ok 1: We quite like that, because, it starts making it a bit more accountable, (pause) you can sit there and say, you spent X amount on PR, and we can actually prove that PR has driven X amount of leads, because using Google Analytics, they will be able to see where the links have come from, and they will be able to link them to the coverage that we've got, And hopefully we will also be able to see how many of those leads have been converted, so you can actually start to get a price for the leads you're getting. R: Sure 1: Then there are two paid for methodologies that we, some of our clients us, or that we put forward, one is Apollo (erm) and they measure share of voice, so, you can go onto Apollo and (pause) and they give you a free overview of the top 100 tech companies in the world, and it gives you a flavour of, ok if say we had, I'll use Kaspersky as a good example, and they actually use this, so the antivirus market is quite busy, there's Symantec, McAfee, Sophos, there's a wide variety of them, when we started working with them, they were a distant thif, in terms of the share of voice every month. There were a lot of other companies making a lot more noise, they were just another one, not really getting their voice out there, they are now number two or number one, two years later, and that is something that they are very proud of And that is something that is very important to them. That they are seen out there as very visible, it helps with their thought leadership strategy. The other one is Metrica, Metrica is (erm) extremely expensive, but it is (erm) run by a company that does clicking analysis, and they basically take all of the coverage and they analyse it all on a variety of different things, key messages that are in there, who has been spoken to, who's the spokesperson, imagery, all of this stuff, and the figure comes from there. It is quite pricey. R: Sure
88 | P a g e 1: The reality is, what method do we use, we use a lot, there isn't one size fits all, you do what suits the client. R: Yes, so, I mean, as you were just explaining, I gather that a lot of your methods do require goals to be set at the start, so to drive traffic, you want X amount of people to visit this page or this website in the period of time. 1: Absolutely R: Do you feel that that is the most important aspect (erm) when it comes to measuring? 1: I think it's not just measuring, it's actually one of the most fundamental key parts of any PR campaign, whether it's traditional or using social media, or if you're doing digital such as video, any project or campaign you're going to do if you just start working and you haven't sat down and thought why are we actually doing this, what are we wanting to achieve, and what will success look like, you haven't actually got that, then you're shooting yourself in the foot. R: Yes 1: You have to be a 100% clear what success is going to look like, then you can come up with the evaluation process to actually fit that. The reality is that some clients sit down and say yeah, we're really interested in, (pause) seeing the volume of our coverage go up over the next three quarters, but if you get to the end of those 3 quarters and the volume has gone up but sales have not gone up, and that's what they're actually really being judged on, that's what they really what success is going to be measured against, you've actually failed. So it's really important as PR agency, to sit down and actually really dig deep and find out what success is going to look like, and how we can actually help them with that. R: (erm) Next question was (pause) sort of about, a campaign that might have gone wrong, once you've sent it out, I don't know if this has ever happened to you in your career before. 1: Less than its gone wrong, a good PR agency is always evaluating, so you don't wait until the end. And again, this is a (pause) key part to that first conversation at the start, about understanding what success is going to look like, you have to be flexible enough to be say, a month into a campaign, to be able to say what we are doing isn't going to achieve what we need to do, what we need to do is to address this, we need to change, if you don't have that flexibility, you are going to fail and the client isn't going to stay with you, it's as simple as that really. So yeah, I've got (pause) I can't give you specific examples, because the client won't be happy about me sharing that. But yeah we have had examples of, you know we've had a product launch and we have been working towards a certain date, and actually we've turned round and said look, this won't work, because (pause) there's another conference going on, or there is something else, so we're going to have to change the deliverables on that. So yeah (pause) it often happens, but quite a lot of campaigns change, you have to think on your feet. R: Yeah, I suppose public holiday's, that comes into everything, and even holiday's abroad or something.
89 | P a g e 1: The big thing is finding out if there is a big thing happening, at the same time, as a major launch. So you have to look at when exhibitions are happening, you need to look at, not just players in your market, but major players, if Apple, is having the launch for a new iPhone, on the same day that you want to launch your product, you're probably not going to get a lot of coverage, because someone else is making a lot more noise than you, and is a lot more interesting. It might sound harsh, but that is the truth, sometimes you have to sit down with a client and say, all the press are going to be going to that, so no one is going to come to your event if you hold it on the same day. R: Ok, (pause) do you think that, because we have such a large choice of media, especially compared to other countries, do you think that it has an effect on the way PR is seen by clients, in a sense that, oh there is so much media, will we get into the right publications. Or do you feel that the PR agency can combat that? 1: I think that that is a responsibility of the PR agency, to make sure that you're speaking to the right people, and sending out news scatter gun to everyone and their dog, to try and get everything, you're not doing the right job, what you actually need to be doing is targeting specific titles, tailoring your pitches for them, building relationships with them, and you choose those ones based on clients customers are reading it, or that's something they watch or that is somewhere that they go, so you need to understand who your clients customers are, and then you can get to the right titles, so the fact that there is a lot of media in the UK, no that doesn't impact it, in actual fact, yeah, there is a lot of media, but if you compare, it say Germany, WOW, we are nowhere near them! Nowhere near! So I run, international campaigns, and that is our biggest challenge, for this specific client, that volume is and the OTS is a driving factor for them to evaluate their public relations. Germany has five or six times as many titles, so you're always struggling to, on that front. And if you beat them, wow you've obviously had a good month! So, I don't think it impacts it, I think there is a big element, that it is on the shoulders of the PR agency to be targeting the right people. The industry has changed, if you send out a press release, and send it out to 200 people, the reality is that there is every chance you will get named and shamed! Journalists are so fed up with that approach, of it being untailored, its spam, so you are sending something out and it's just no relevance to their publication. There is every chance that they are going to go to twitter, they are going to email people, and go, what the hell is this agency doing?!, they are just amateurs, this is the second time they have done this! I will never deal with them or their clients again! Now if your clients see that, they will not be happy! As you are losing publications, and they don't want to be represented by someone who is doing shoddy work. R: I mean, also, (pause) for example, the AVE system, I don't know if your agency uses a multiplier, on that. 1: We very rarely use the AVE system now, it (pause) you go on to any sort of PR Week, PR Moments, anything like that, you will see that there a school of thought, that it is out dated, it isn't answer the problems, it just doesn't the evaluation questions of PR anymore. We really, we sort of put it forward to clients, but we rarely use it. R: Do you feel that some PR people, actually increase the value of the coverage, just to impress the clients? 1: Yeah, I'm sure there are some unscrupulous people, who (pause), yeah I think there is an element of it, the other part is, actually, you shouldn't be about, impressing yourself to the client, you
90 | P a g e actually need to understand, (pause) so if it take a step back, so say for us for example, we win an account, more often than not we will be dealing with a marketing manager, or marketing director, they actually have to report their board, or a VP of marketing, or something like that. What is actually more important to us, is not making ourselves look good, it's actually making that marketing, our contact, at the client look good. So we need to understand what they need to say, to their bosses if that makes sense. So if you know that the VP is looking solely on volume, if the VP is looking solely on traffic, or something like that, you are going to make sure, you're not going to doctor them, you are not going to falsify them, you're going to make sure that, you give the information to our contact at the client, that is going to help make them look good, and stop them getting pressure from above as to what the hell is going on with t his PR agency, they are not delivering? So (pause) I'm sure there are some unscrupulous ones, who have gone out there and been slightly dodgy with numbers, but you know what, it's going to come back and bite you, for us, the amount of time, energy and money you put into winning a piece of business, you've, the only reason you would do is that for a long term relationship, you want to keep that client, two, three or even longer years. If you go in there and start lying about the amount coverage you got, or the amount of circulations or the AVE, and you get found out, your gone! To put it into context, the new business process, a PR agency will very rarely make money out of an account for the first quarter, and it can impact the second quarter as well, it's only because of the amount of money you put into winning that account, time and effort, if you take that away from the fees, and in fact in that first quarter, (pause) you over service it, you're getting up to speed, there is a lot of content, you've got to hit the ground running, because you've got to be evaluating it in three months time, so you need some coverage. R: Ok, I haven't actually thought of that point, so (pause) quite a good thought there, I mean, (pause), which evaluation method say for yourself, would you recommend out of the ones you mentioned to a client? R: (pause), again it depends on their budget, it depends, there really isn't one I would put my hat on, more often than not, cost per 1000 can be quite good, again, crude but it can give an ROI figure, which they can give to their boss, and say from this amount of money that you spent on PR, this is how many people we got in front of, online is very exciting to me, because there you can start putting a value against PR, you can actually say for your spend, we sent you these links, this many have been converted, so for this money, you have generated X amount, but there needs to be a level of sophistication at the client side as well to understand that. A lot of them are starting to get there, but a lot don't have that yet. R: With the cost per 1000, do you find that clients ask you, just because for example I'm in the daily mail today, and they have a circulation of 1.9million or whatever, how many people actually read it? Do you get that at all? 1: Of course, you are going to get that, I think, that's where you step back and it's important that you are targeting the right titles, you know, you got to make sure you are in the right publications, and the other part is they will start to see how effective it was as well from the amount of traffic that is going to their site, how many people have logged in, how many people have clicked through to the site, how many people have searched on them based upon something that was in there, more often than not if you hit the Daily Mail, the next day the client's website will see a massive spike, it will be
91 | P a g e the client name and the spokesperson, and you know where it has come from, because of that, so you can just about justify it. R: Also, out of any of the systems that your agency uses, if you could change one, how would you change it and why? 1: (pause) I don't think you can change any of them, I think they are what they are, I think it's less about changing them, and more about merging a couple of them, so on their own, they don't work particularly well, but alongside a couple of the others ,then that does work, say, you've got your opportunity to see, you're measuring the amount of volume you've got, so you know that the vast majority of your coverage, is in the titles that your customers are reading, and your opportunity to see in them is X, you can start doing it, but an element of that and an element of digital, I think you need to have, there isn't one size fits all, so many three or four of them inclined together. Does that make sense? R: Yeah, (pause) as a graduate myself, evaluation in PR in our degree is something that is touched upon, but we don't spend too much time on it, so do you feel that people who don't have a full understanding of evaluation, tend to not get as accurate results as those who do, because they don't see the point of it, they don't understand it in that sense as well? 1: Yeah, you're going blind, if you haven't got a way of evaluating what you are doing, you are never going to know, whether you have achieved success, and I suppose it is the same as doing your course, if you are not evaluated on that every term, as to how you're doing, or end of each year, you don't know how much harder the following term or year, to get your grades up, or you can sit there and say actually, I'm doing alright, my coursework is on track, I can concentrate on improving X over there, so it's exactly the same in the PR world, if you're not evaluating or evaluating properly, you're kind of going blind as you're not helping your clients, as you are just taking their money and running, you're not actually helping to improve it, you should always be striving to improve it, whether that's the results or the amount of sales they are getting, and it's only by doing that you will have a long term and successful relationship with them. R: OK (pause) 1: Evaluation, partially its hasn't been touched on so much, doesn't surprise me enormously, it's something I think, it's important to talk about it, but the reality is, the methodology for evaluating changes, it changes yearly, and it is rapidly changing with the impact that digital is having on PR, so the fact that it is not being in the curriculum, I kind of think it is the right thing, because, once that curriculum is set, it is out dated, as long as I think people on the courses understand the importance, it is actually the responsibility of the agencies you go into for work experience or your first job as a trainee to help you along those roads, and they will show you their different methods to understand it better if you are doing it. R: And also you feel that online and offline evaluation of PR should have a mix of all the methods really, so you can have an understanding of them all. So for examples, cost per 1000 and Google Analytics, would you agree? 1: Yes, I'd agree with that, I think, obviously, the world is moving to a digital phase, that is where a lot of reading is being done, it is where a lot of people click through and see stuff, but there still very
92 | P a g e much is a place for the hard copy, (pause), the vast majority of the news I read is online, because I like to know it as it's happening, but I still go out and buy Empire magazine, as I like to read big features about movies and things like that. I mean, for the reality is I am actually going to see a film or buy a DVD based upon the reviews and the big features I've read, and I still like that in a hard copy, but I check out the news of what's coming out online, So I'm not alone in that I'm sure that will change again in the next few years, but I really hope it doesn't ever die off hard copy, because, there is a place where you have to measure both, and they are slightly different. R: Ok, I don't know if you have worked in house at all or if you know anyone who has? 1: I haven't but I know a lot of people that have and do, and also at Berkeley, we like to employ people who have been in house as some point, as well as the people who have come straight from university, or just worked in house, because it gives us a different perspective. R: Do you feel that there is a cultural difference in evaluation, as an agency, you have your own business objective and goals and when reporting to a client they have their own, as well. So do you feel that although they are in house and may have their own department objectives, that the culture will be different? 1: There is a massive difference, in that we are a PR agency, we are totally blinkered, its PR and generating coverage and noise, and helping with crisis management and all of this stuff, that is all we think about. The reality is, in house, they very rarely, unless it is a massive company have a PR manger, you are actually reporting to a Marketing manager, and PR is a small mix of what they are responsible for, so where as it is all day every day for us, it's probably a day's worth of activity a week, because they also have all of their other stuff which they have to do. So for us it's all important, and for them it is just about results and it is an ease of life, they want to be seen that they are improving things to their boss, and also, it is making sure that it doesn't eat up too much of their time, so yes, there is a massive difference between the two. And the reason that we like to have people who have worked in house is so that we can understand that. R: Right sure. 1: But PR people go, oh god why is so and so not answering my calls, they're not sending this through, how are we going to achieve this? and it is actually because that is one of 20 other things they have to do that hour, and the other things are more important, at that point in time. Yes it is the most important thing in our world, but it is just critical at their end. R: Ok, obviously you deal with international clients, (pause) do you deal with partner agencies in other parts of the world? 1: Yes, So we have a network, called the Berkeley 360, which we are the lead agency of, that we have agencies, we just held our summit actually in February, and they came over to Reading, we have agencies in the Netherlands, Germany, France, Spain, Italy, Nordics, and there are a couple of other Europeans, I actually can't remember, we also have Singapore, which covers Asia Pacific, we've also got Australia and America. R: So you quite a large span network.
93 | P a g e 1: Those are agencies we share work through, but we also have a certain way of working and we do work on international clients together. R: do you feel that when it comes to measuring, when working with other agencies, the local culture at their end plays a role in how they evaluate their work, or are they open to use the methods you use here? 1: No they generally, more often than not, open, I think the way. Evaluation methods that are used in the UK are pretty much used around the world, so more often than not, stuff comes from the US first, then comes to the UK, so if you are looking at in terms of where new ideas and techniques come from, that's usually a rule of thumb, but some people might disagree with me, but I believe that. In terms of evaluation, it's pretty much the same around the world, Its more along the lines of getting people to do the evaluation and do it on time. It is sometimes that some cultures are a little more laid back about it. But yeah, the actual methodology is the same. R: Ok, and finally, a bit of a cheeky question this - do you feel the best form of evaluation is when the client comes back and says we want another 12 months with you? 1: Absolutely! Absolutely! That is not a cheeky question, it goes back to what I said, you need to understand what success is going to be like, there is no point of us kicking off a campaign and then sitting in a board room, six months later, look the volume of coverage, has gone up every month, and they go yeah but it is all in magazines that have no relevance to us, we've got no sales from this, your fired! That is what will happen, it is as simple as that, you have to make sure it starts, so yeah the biggest pat on the back for the work that your doing is that you are not going unnoticed. R: Ok, 1: Does that all make sense? R: Yeah, that is great thank you for your time. 1: No Worries and thanks for your questions. R: It's been great talking to you 1: Cool, perfect.
94 | P a g e
Appendix 2.4 - Transcript - Interviewee Coding D R: Firstly do you give your consent to being recorded? 1: I do R: Ok, that is great, thank you. (erm) Firstly, what evaluation method did you use when you were in the industry? 1: Erm, well, in industry, I mainly used more, 2 things, media evaluation, media evaluation, to monitor media sentiment and volumes and (pause) which way opinion was going, but I also did a lot of stakeholder opinion research, to see what stakeholders knew what our reputation was and how it was changing. R: Right, so sort of opinion polls. 1: It was opinion polls, but very specific commissioned research, because for example, investors, there was probably never enough, investors to do proper quantitative research, in mass market opinion polls, so we do them with target research, with almost full population samples . R: Right, these methods that you were saying, do they require goals and objectives to be set at the start of the campaign? 1: (erm) Yes, I mean, if we were doing specific campaigns then yes, but very often because these were very big organisations that I was working with, we would usually have a continuous research programme and a reputation monitoring programme running, so you would probably see what the results were within that, so if you did a twice annually, yearly, reputation survey, and if you did monthly based media evaluation, then you can see how (pause) the media opinion was going, how sentiment was going and you would fit campaigns into that. We did do a specific campaign monitoring as well. R: Ok 1: If I might just say, the role I am doing now, in consulting, much more of what I do is strategic consulting, so I am much more concerned with outcomes rather than outputs. In other words, has it affected the business? The continuity of a business, the probability of business growth and so on. R: Sure, so before you started the consultancy, your evaluation process ran from the start to the end of a campaign, and continued all the way through. 1: Yeah, I mean, the evaluation process ran all the time, and we would fit campaigns in it, for example, when I was with Rover Group, for seven years, whatever, nine years, we would launch usually a couple of products a year, so you would know within the years analysis, how coverage of that product had gone, and whether the media thought it was good or bad, how that affected the reputation of the company (pause) and you know, how other things were affected because of that. But what we didn't do very well in those days, was monitor how that coverage affected sales. R: Right sure. I mean, this might not be very relevant to your previous career in PR, but have you ever had to send out a campaign, say for a product launch, which backfired, or didn't work in the
95 | P a g e way that you wanted to? So you had to recall the campaign, or you had to edit it while in the public domain? 1: (erm) I can't recall doing it, necessarily because the campaign failed, sometimes, particularly since I've become a consultant, I've advised the client to stop doing something, because the original idea was not a good one. An example of that was an IPO, I was trying to gain awareness of an IPO, so an offering to the market, (pause), it became very clear when explaining it to the media, that the media did not think that it was a very good idea. This particular entrepreneurial business. So (pause) we knew they were not going to get good coverage, so we advised them to stop running the campaign and re think the offer to the market. R: Right 1: So, generally, no, in corporate life and in most consultancy work that I've done, I can't recall recalling a campaign. R: Sure, Also, we have quite a large choice of media outlets in the UK, and increasingly in other countries across the world, there's are growing as well, say China, India, Russia, Eastern Europe, they are very quickly catching up and overtaking us on the number of media outlets they have. Do you feel that this has an effect in the way that PR is seen in the way business place as there is so much media to choose from that PR professionals, they should know which publications to target? 1: Yeah (pause), I think, the proliferation of media, particularly social media, or online media let's call it as well as social media, has raised the profile of PR generally full stop. I think people are more and more aware now that things that can go wrong, as well as things that can go right, and need people in organisations or consultancies, that can advise on how best to handle that. So I think overall, the growth in and proliferation of media (pause) and the globalisation of media as you were saying has made PR a more important role in an organisation that it was 20 years ago. R: Sure, I mean, this might not have really affected you, as you were in house, but it is something that agencies, sometimes do tend to do, to justify their existence, to their clients, that they increase the value of their outcomes, to make themselves look good. I mean being in house, you would probably, your core business objectives are the same as the companies, you may have different departmental ones, but you are part of the same company, therefore potentially you wouldn't have to do that? 1: I think, in (pause) in house or externally, anybody in any business always tries to make themselves look better than a perhaps be seen by others. I think in PR we have the dual task, of convincing people that what we do is important, in managing reputations as opposed to managing media, so we had that task to do, then we had the task of persuading them that what we do in managing reputation, can be effective, so yes I think agencies and in house people can be guilty of over emphasising that and it is something that we do with our enthusiasm to get the message across in the first place, rather than just pretend to do better than we are, I mean in simple measurement terms, a lot of people over emphasise media volumes by multiplying by three of five when doing advertising value equivalents, or simply column inches and weighting column inches. To the extent that it becomes meaningless to the audience, (pause) I think the answer to your questions is yes, but sometimes with the right reasons and with the wrong reasons.
96 | P a g e R: Right sure, I mean do you feel that evaluation is important in PR, because it is one of the very few tools we have to justify our existence to clients? 1: Yes, exactly that, (pause) and because we haven't found, it's called scientific methodology yet, that is cheap and easy to apply, then we haven't got that easy route to persuading people, but I do think that the profession through the leadership of the CIPR and PRCA and other bodies, who are moving more and more to outcomes, rather than outputs, (erm) we are going to get closer to that end and getting the management of organisations to appreciate what we are doing is not just trying to get the biggest amount of coverage, or even the most positive amount of coverage, but trying to affect the bottom line, I think once we get into those conversations with people, they will start to set us objectives that will start to match the corporate objectives. (pause) I mean one of my current clients has done just that, we are trying to start with the corporate objectives, and then work out a PR programme that will help achieve those. One thing leads another - we hope. R: I mean (pause) over your career, you must have come across loads and loads of different methods of evaluation, if you could change any of the ones you have used, or merge them together, how and why would you do that? 1: (pause), well with my other hat on, in my other life as an economist - a statistician, I've got very involved from the beginning of the evaluation process with agencies that do the evaluation process, and pushed them a lot to try and become more aware of how we measure the factors that affect reputation, because we can see how a good media report, or a good media campaign, will affect some of those factors, but some of them are about relationships some are about other departments and how they work but they all affect reputation, very recently we've now got companies that are being very effective in measuring the impact of those factors on share price for example, which can be a good way of evaluating if something has worked, in changing the opinion, if it has changed the opinion of investors, it has usually been very effective in other areas as well. What we haven't done yet is make that link, to show how (pause) things that affect reputation can affect share price can be affected by good PR, so we are measuring the end result, we can find ways of measuring the end result but we can't necessarily find ways to measure the links in the chain that lead to that, so if I could (pause) take good media analysis, and project output analysis, and link that to the factors that affect reputation, then I think we would be in heaven in terms of persuading the boards that we know what we are doing, we know what works and this is how it works. R: I mean, also one thing that I've found through primary and secondary research is that, a lot of measuring agencies as such, or monitoring agencies, using technology more and more, but do you feel that computers can't read an article in a newspaper and measure the tone? 1: Yeah, I mean I'm using, doing some of that at the moment, I have a client who I am doing some media analysis for, and they've used a computerised system up until now, and it doesn't tell them much as you say, it reports words and when words are used, and they might have somebody who is reading the coding content analysis, to see whether it is positive or negative, but I think it does need to be a human being in my view, to read that sort of stuff, to really interpret the way things are going, because you can get a lot of coverage, you can get a lot of positive coverage, but the overall mood of the report when you read them together, and the impact of saying you've done a press release and it gets covered or events happen and it gets covered in the next day's report, you can read them together and take out a very negative view from them, rather than a computerised media
97 | P a g e analysis look at it and suggest a positive, so you're right, I do think you need it - there will always be a place for humans. It is what makes us human! R: Definitely! So do you feel that a person with a greater understanding of evaluation, would get or arrive a better result or better figure ROI figure, compared to a colleague who is either new in the industry and doesn't have a good understanding of why it is being done? 1: Yeah, it depends if the person who is new in the industry came from a good university degree, where they got access to the latest methodology and so on, (pause) then they would obviously be better, but I think sometimes there can be very strange nuances, sometimes you can rejoice that a journalist that has been negative is slightly less than negative, and you feel that you have achieved something, whereas in the straight forward physical response, or as you say, someone who is new to the industry or new to the company, looked at it, they would think it was a negative report. Somebody with experience might be happy that it is not as negative as it could be. And you need experience to understand those nuances, sometimes it can be how a media reports, Top Gear is famous for that in the car industry, they will look at some things in a way that its public although they know they are taking the mickey, they actually like the thing, but you might look at it from a straight forward use of words point of view and take out a negative view. So experience counts but new ideas are always welcome. R: Ok, And also, when you are working with clients who are using various media's, say digital and traditional, in one campaign or in one hit, would you alter your evaluation model to affect that? 1: Yeah, I think with different media's you are looking for different things, the end results, the outcomes and the reputation scores will stay constant, the way you measure those and monitor those. But the way you measure outputs and(pause) and activity levels in media will have to be different and the way you measure even Facebook from Twitter is different, you can measure likes on Facebook, but that doesn't really tell you very much at all, you know, you need to understand which part of the message people responded to, for example, measuring and understand number of likes is one thing, but what people did about it is something else. So on monitoring website hits, length of time people stay on pages, all of that sort of stuff, we need to develop a different monitoring mechanism, to measure social media, more importantly to understand what they means, for whether our campaign has been a success or not, people might the joke you made in an ad or in a Facebook video or whatever, but not really, but does that mean they like the product, they like the brand or the campaign, not necessarily, we need more research than just simple counting. R: And also, having worked in house, have you ever worked in an agency? 1: Yeah. R: You have, so what do you feel the main differences are between evaluation culture in in-house teams and agency teams? 1: agencies are much more focused on outputs, in house are much more focused on outcomes. I'll keep it that simple. R: Right
98 | P a g e 1:It is not that easy, a lot of in house teams are still very old fashioned and count and weigh things rather than understand the impact on a business, but what I said before about in house teams and PR generally wants to get up higher up the pecking order, the executive pecking order, by getting the executive to understand the role that PR plays in the business in building reputation, and I think that most of the big companies that I've worked with are geared that way, but most of the consultancies tend to be geared to either outputs or either campaign objectives achieved, you know where campaign objectives are measured by outtakes and outputs rather than outcomes. R: Ok, also when working with a PR agency or a Corporate who has companies or offices across the world, do you feel that their local culture and ethics come into play when it comes to evaluation or do you feel that the model set in the home country is the one that resonates around the world? 1: (erm pause) Usually, I think in big companies the one that is set in the home country is the one that resonates around the world, if they can afford it, if the media in the country allows it, trying to understand sentiment in Japanese or Chinese is very very difficult to get a computer to analyse that. Yes I think it is centrally driven, but in like all PR local cultures, but I do think you want to be able to compare one countries results to another. You want to compare one campaign with another, and how brands are doing in different market places, so you need to use the yard stick, the same statistic to measure that. R: Ok, last question - do you feel that the best form of evaluation is when a client comes to you and says ok I'm happy with the word you have done, I want you, to sign you for another year? 1: If you are an agency then yes, but, I'll say but, clients are most likely to do that if the agency can prove and demonstrate that their work has been effective, rather than just having a good relationship, and the client doesn't think that anybody else could do any better, the agencies need to demonstrate more and more that they are effective, and that does involve them getting monitoring outputs rather than simple outcomes. R: Sure 1: Sorry outcomes rather than outputs, got them the wrong way round! R: So if an agency promises you X Y and Z, they have to prove they have got X Y and Z. 1: Yeah, I mean the client, a good client would demand more of the agency all the time, and because, one of the things is, agencies have more of a broader range of staff, so have more capacity to do things that clients do, so although clients have the money, the clients don't necessarily understand the latest analysis techniques for evaluation so they could, agencies could demonstrate their worth, by bringing those to the attention of the client, especially social media evaluation, and offering that, to do that for the client, and grow the business as well. R: Ok, that is great! Thank you for your time and your insights! 1: Ok, My pleasure, I'm sorry I'm a bit half way between the two here. Former client and running a small agency.
99 | P a g e R: No it has been really helpful, as the people I've spoken to so far have all been agency staff, so having someone who has actually done both, and you've done the corporate side and the consumer side erm, it has put a whole new spin on things! Thank you very much.
100 | P a g e
Appendix 2.5 - Transcript - Interviewee Coding E 1. What method of evaluation does your company use? We currently use a spread of evaluation methods including EAV, ROI, circulation stats and total pieces of coverage. 1a. Does this method require goals and objectives to be set at the start of the campaign? We always set targets at the beginning of a campaign, in partnership with clients to align with their business objectives. 1b. Does the evaluation process run from the start of the campaign to the end? Yes 1c. Have you ever had to re call a campaign as measuring it when it was live gave a negative result? No 2. Do you feel that the Mass Media or the large choice of media has an effect on the way that PR is seen in the business place? I think the large choice of media allows businesses to reach people in different ways and allows PRs to be truly creative and proactive in their pitching of ideas.
3. Does your company use one unified method of evaluation for all clients? No, we tailor our methods to their reporting needs and business processes 4. Do you agree that some PRO's increase the value of their evaluation reports to impress clients or their board of directors? No 5. If you could change your company's method of evaluation what would you do and why? I would place more emphasis on 'impact on the ground' for example the number of sales as a result of click through a from online coverage or the number of tickets sold/entries to a race.
6. How important do you feel evaluation is? Evaluation is very important. It makes everyone work to their best ability and ensures efficient and tailored activity 6a. Do you feel that PRO's who understand evaluation get more accurate results than their colleagues who don't have such an understanding? I believe PROs who understand their client's needs and targets will achieve quality results e.g. Key press and USP inclusions
101 | P a g e
7. When working with clients who gain coverage on various media's away from print does your evaluation model differ? Yes, for online we measure click throughs but only a few clients have systems in place to monitor this on their websites.
8. What are the main differences do you feel that there are between evaluation culture of in house teams to agency teams? I'm not sure as I've only worked agency side 9. Do international companies use one method across the world, or does local culture and ethics make a difference? Can't really comment on this as the large majority of clients are UK based. We have one client in US and we use similar process of evaluation
10. Is the best form of evaluation when the client renews the contract? Renewal of a contract is very positive as it demonstrates the value of PR and relationships however I feel the best form of evaluation is business success and impact on the ground, with the consumers e.g changing beliefs, awareness of a product or service.
102 | P a g e
Appendix 2.6 - Transcript - Interviewee Coding F 1. What method of evaluation do you feel is most commonly used in the PR industry? As with most forms of measurement, people start with what is both easy and cost effective to measure. For PR this starts with a focus on media relations, which has an output that is easy to both capture and count. Media content analysis is often used as a basis for the success of PR, despite the fact that PR involves so much more than media relations. Basic output measures based on media content certainly dominate PR measurement. 1a. Does this method require goals and objectives to be set at the start of the campaign? It can do if it is done properly. Of course, perfect measurement involves climbing the ladder from outputs, to outtake and ultimately outcomes but any form of measurement can be rooted in best practice and begin with the setting of goals and objectives. Whether media content analysis is the right measure to demonstrate that those goals and objectives have been met is another matter. 1b. Does the evaluation process run from the start of the campaign to the end? Often is does, yes, as typically the work we do with clients is ongoing. 1c. Have you ever had to or heard of examples that a campaign has had to be recalled due to negative evaluation, whilst the campaign was in the public domain? PR doesn’t really work with the concept of recalling. Adapting or reprioritising might take place if a campaign does not gain traction. If something elicits an immediate negative response which is picked up by us, there may be the need for damage limitation.
2. Do you feel that the Mass Media or the large choice of media has an effect on the way that PR is seen in the business place? I don’t think these things are connected. I’m not sure mass media is the right phrase. Social media specifically, and the ability for the consumer to publically communicate back to an organisation, alongside the effect this can have on search e.g. my Dell Hell / Dell sucks coming top when you searched for Dell online back in 2005, made CEOs sit up and take notice. A lot of customer service problems were landed on PR in the early days, now there is the realisation that good PR cannot fix a fundamental problem in the business. The concept of social business is one that is really changing the way businesses operate. 3. Do you feel that PRO's use one unified method of evaluation for all clients? No. Smart PR agencies are using framework based in best practice and then selecting the right measures to fit client goals and strategies. 4. Do you agree that some PRO's increase the value of their evaluation reports to impress clients or their board of directors? The only way for a PR person to increase the ‘value’ of evaluation would be to start by using a value in the first place which I take to mean AVE. I have seen lots of PR people over the years apply a multiplier to AVE on the basis that editorial holds more value than advertising. Although third party endorsement does add value, it has never been calculated how much this should be (because it’s an impossible calculation dependent on so many variables). On top of that AVE is a nonsensical number that talks to nothing PR does for a business.
103 | P a g e
5. If you could change evaluation in PR, what would you do and why? I would ensure that every programme began with some brilliant thinking as to what the business was trying to achieve, how PR was helping this happen, what measures would accurately show PR was doing this successfully. Thinking upfront destroys the need for spurious measures such as AVE and basic counting of outputs. 6. How important do you feel evaluation is? Very. 6a. Do you feel that PRO's who understand evaluation get more accurate results than their colleagues who don't have such an understanding? I’m not sure what you mean by accurate results? Results that show better what PR has done to move the needle for the business? Yes. 7. When working with clients who gain coverage on various media's away from print does how do you feel evaluation models differ? Such as online and social? The model of best practice should not differ, the metrics used though can differ. 8. What are the main differences do you feel that there are between evaluation culture of in house teams to agency teams? Great agencies are taking the lead and pushing clients on setting measureable objectives and measuring them credibly. On the other end of the scale, some agencies use big figures to try and impress clients. Exactly the same is true of clients in-house. There’s no clear distinction between in-house and agency, there’s good and bad in both. 9. Do you feel international companies use one method across the world, or does local culture and ethics make a difference? For a Global Comms teams, one method is important in terms of framework and success metrics. The targets however have to adjust to take into account market differences. 10. Is the best form of evaluation when the client renews the contract? For an agency? Yes. And whether you grow that account into other territories or business areas etc. Retention, growth and recommendation are all indictors of whether a client is happy with you. It doesn’t necessarily mean you are evaluating properly or having an effect on the target audience. Chemistry and many other factors could be make this happen. It would be interesting to do some quantitative research to understand if there is a link between agencies evaluating properly and the retention / growth of their client base to see if there is a causal link. If you do, I’d love to see it!
104 | P a g e
Appendix 2.7 - Transcript - Interviewee Coding G 1. What method of evaluation do you feel is most commonly used in the PR industry? AVEs probably, although I have no evidence of this. 1a. Does this method require goals and objectives to be set at the start of the campaign? No, not at all. 1b. Does the evaluation process run from the start of the campaign to the end? No, it only covers the measurement of the press coverage. 1c. Have you ever had to or heard of examples that a campaign has had to be recalled due to negative evaluation, whilst the campaign was in the public domain? No 2. Do you feel that the Mass Media or the large choice of media has an effect on the way that PR is seen in the business place? Not sure what you mean - I do think that the media is influential on the way PR is seen in a number of ways. If you mean with regards to evaluation, I do not see any connection. 3. Do you feel that PRO's use one unified method of evaluation for all clients? No, there are many methods, and many agencies have their own methods that they have used themselves. AVEs do seem to be the most prevalent however. 4. Do you agree that some PRO's increase the value of their evaluation reports to impress clients or their board of directors? There is one example of how this seems to happen - many PROs multiply the value of their AVEs by 3, justifying this by claiming that PR is 3 times as effective as advertising. This seems to be completely unscientific and just there to increase the value. 5. If you could change evaluation in PR, what would you do and why? A method based on the full length of the campaign, which focuses on the initial requirements, and whether the right people we actually reached. The AMEC meeting in June is expected to put forward a new form of evaluation which I would suggest will be useful to have a look at.
105 | P a g e
6. How important do you feel evaluation is? Quite important, but perhaps its importance is over-stated. The PR industry does very good work and that should not be forgotten just because the means of measuring it are not totally accurate. 6a. Do you feel that PRO's who understand evaluation get more accurate results than their colleagues who don't have such an understanding? I suspect that this may not be the case, and that good PR skills are not linked to evaluation. 7. When working with clients who gain coverage on various media's away from print does how do you feel evaluation models differ? Don't understand the question - I think evaluation methods do differ for different media. 8. What are the main differences do you feel that there are between evaluation culture of in house teams to agency teams? I think that agencies â€˜ownâ€™ evaluation and that in house teams leave it to them to do most the evaluation work. 9. Do you feel international companies use one method across the world, or does local culture and ethics make a difference? I think it depends from one company to another, but there will be some instances of international companies using one method across the world. 10. Is the best form of evaluation when the client renews the contract? Probably not, no!
106 | P a g e
Appendix 2.8 - Transcript - Interviewee Coding H 1. What method of evaluation does your company use? Google analytics for the website, retweets on Twitter, likes on Facebook, Google alerts, hashtags on Twitter and Instagram 1a. Does this method require goals and objectives to be set at the start of the campaign? Yes. For example, if we begin a social media campaign and want to build a buzz about a new product, we will set a numerical goal in terms of retweets, likes, hashtag uses. 1b. Does the evaluation process run from the start of the campaign to the end? Yes. Absolutly. This is a must because one should be able to track and measure during what time in the campaign there were more impressions in the entire digital space. 1c. Have you ever had to re call a campaign as measuring it when it was live gave a negative result? No. 2. Do you feel that the Mass Media or the large choice of media has an effect on the way that PR is seen in the business place? The industry of PR can have a negative reputation because many people who are not a part of the industry think we “spin” stories. This is not true, because most PR professionals are well-versed in what’s ethically right in terms of communication with the public and a private organization. I do not call it spinning, it is more like finding what’s newsworthy and running with it. 3. Does your company use one unified method of evaluation for all clients? Yes. We have our clients fill-out a survey about their experience, and if we delivered the results they wanted and expected. 4. Do you agree that some PRO's increase the value of their evaluation reports to impress clients or their board of directors? Not that I know of. This would be unethical.
107 | P a g e
5. If you could change your company's method of evaluation what would you do and why? I would not, I think they do a great job. 6. How important do you feel evaluation is? Of course it has to be one of the most important aspects of a campaign because you need to be able to see what type of impact your campaign initiatives had on the public. Without evaluation, it would be impossible to measure return on investment as well. Not to mention, the client needs to see proof that you are doing something right with their campaign. 6a. Do you feel that PRO's who understand evaluation get more accurate results than their colleagues who don't have such an understanding? Well, if you are a professional, you must know what evaluation your work is a must. For example, if a pro doesn’t evaluate their process, they will not know what worked and what did not work. 7. When working with clients who gain coverage on various media's away from print does your evaluation model differ? No, because we mainly concentrate on digital marketing and PR and most of our impressions come digitally. Unless a client asks for a traditional PR route for their campaign. 8. What are the main differences do you feel that there are between evaluation culture of in house teams to agency teams? I work in an agency type of environment, so I am used to dealing with a few clients at a time. It’s important to discuss and understand what the client wants and execute what they need. These are clients who demand results and rely on us to make a PR buzz in any industry they are a part of. We survey the client, we measure the “buzz” digitally, and show the clients what worked and what didn’t work based off of our PR measurements. I feel like in house PR also use evaluations to show profit to their seniors. That would be the difference.
9. Do international companies use one method across the world, or does local culture and ethics make a difference?
108 | P a g e
I definitely think that local culture and ethics make a huge difference. There may be an agency out there who doesnâ€™t properly measure their PR results, and in result, they may continue making a mistake over and over again with different clients. 10. Is the best form of evaluation when the client renews the contract? Well, they renewed the contract! So that should say it all.
109 | P a g e
Appendix 2.9 - Transcript - Interviewee Coding I 1. What method of evaluation do you feel is most commonly used in the PR industry? The most common method of evaluation unfortunately is still based on OTS (opportunity to see). This is probably based on clients’ familiarity with the notion of “reach” from ad campaigns and a requirement to translate PR into comparable return on investment. 1a. Does this method require goals and objectives to be set at the start of the campaign? It often does, although OTS can be easily gamed by securing online coverage. If OTS is a key measurement then demographic should be part of this from the outset (i.e. the right people see the coverage). Also a list of key media should be defined up front to avoid coverage being secured in non-relevant media and being counted in the final evaluation tally. Often, there is a volume target set at the start of a campaign (e.g. 15 x pieces of national coverage, 25 x pieces of regional, 2 x broadcast, etc.). 1b. Does the evaluation process run from the start of the campaign to the end? In our agency, evaluation is done in real-time, so that we can spot trends, patterns and successes as we move along. This can be shared with clients on a weekly basis or reviewed at a 6 month evaluation meeting. Usually, the campaign has a main evaluation at the end of the year (for retained clients) or at the end of the project. 1c. Have you ever had to or heard of examples that a campaign has had to be recalled due to negative evaluation, whilst the campaign was in the public domain? No. Often, the strategy or KPIs are revised depending on initial results. So, if a brand has a new product range, KPIs could be increased. Likewise, if a key component of a campaign is no longer possible to activate, KPIs can also be rejigged to make the most of the campaign fee. 2. Do you feel that the Mass Media or the large choice of media has an effect on the way that PR is seen in the business place? There is still a misconception of what PR actually is both within the wider public and within businesses in the UK. Particularly in the small business space, PR is seen as a poor relation of
110 | P a g e
advertising whereas the reality is that it can often deliver better long-term results for companies when compared against paid-for media. The general public often confuse or mistake PR as being synonymous with political spindoctoring or Champagne-swilling at parties (ĂĄ la Absolutely Fabulous). PR is also seen as being similar to the services provided by celebrity publicists like Max Clifford which is a small part of what the industry actually has to offer. In fact, now that the media landscape has become fragmented and much larger, the opportunity to secure good PR has also grown. In addition to this, the way in which consumers and businesses actually engage with media has changed so much that it is now essential that PR takes on a bespoke strategy to ensure key audiences are reached via the media platforms that they consume. 3. Do you feel that PRO's use one unified method of evaluation for all clients? Not in my experience. I know that we try to use qualitative and quantitative methods of evaluation for all clients but have to tweak our evaluation methodology to align them with our clientsâ€™ business objectives. So, if a client is interested purely in inbound clicks to their website(s), our evaluation must prove that we have delivered on this regardless of how many pieces of coverage we secured. 4. Do you agree that some PRO's increase the value of their evaluation reports to impress clients or their board of directors? Part of our job is to work to whatever methods of presentation best suit our clients. That said, we would never fabricate results. 5. If you could change evaluation in PR, what would you do and why? I would fully embrace a uniform system that takes on board the Barcelona Declaration of Measurement Principles that set out to include volume as well as impact of campaigns. These principles go beyond pure OTS and offer a better 360 degree understanding of how effective a PR campaign is. It also takes into consideration social media and online coverage which goes beyond the pure vanity of the number of Likes
111 | P a g e
and Followers (which can be unscrupulously bought) and focuses more on actions such as engagement. 6. How important do you feel evaluation is? Incredibly important. Without measuring success, you can’t justify spend. That said, some things are more difficult to evaluate than others. For example, measuring people’s warmth towards a brand (beyond £) is difficult, although not immeasurable. 6a. Do you feel that PRO's who understand evaluation get more accurate results than their colleagues who don't have such an understanding? Yes. Evaluation isn’t an end-process that you do post-campaign. It should be an ongoing process that is central to the execution of PR strategy. PROs that get this, will make sure that everything they do has a positive impact on evaluation. 7. When working with clients who gain coverage on various media's away from print does how do you feel evaluation models differ? For clients that have an online-centric model, it is easier to show impact of PR through clickable links in coverage that can be measured on a source by source basis. These clients seem to be more switched on to other forms of quantitative evaluation (SEO, PPC, email newsletters, etc.). 8. What are the main differences do you feel that there are between evaluation culture of in house teams to agency teams? In-house teams tend to have more pressure on delivering comparable pound for pound ROIs on different marketing disciplines (SEO, Direct Marketing, advertising, PR, etc.) whereas agencies will focus on delivering excellent PR results. 9. Do you feel international companies use one method across the world, or does local culture and ethics make a difference? Global media trends and localised news will affect the effectiveness of PR but general universal PR evaluation methods should be applied to all campaigns (see above: Barcelona Principles)
112 | P a g e
10. Is the best form of evaluation when the client renews the contract? No. But it always helps!
113 | P a g e
Appendix 3.1 - Permission Letter - Coding A
Appendix 3.2 - Permission Letter - Coding B
114 | P a g e
Appendix 3.3 - Permission Letter - Coding C
Appendix 3.4 - Permission Letter - Coding D
115 | P a g e
Appendix 3.5 - Permission Letter - Coding E
Appendix 3.6 - Permission Letter - Coding F
116 | P a g e
Appendix 3.7 - Permission Letter - Coding G
Appendix 3.8 - Permission Letter - Coding H
117 | P a g e
Appendix 3.9 - Permission Letter - Coding I
Appendix 4.1 - Supervisor Meetings Date
Discuss topic (too broad), initial proposal content and structure.
Agreed new question and discussed initial proposal structure and content.
Initial feedback discussion, improve research objectives, research question.
Improve research objectives for next meeting, think of ways to improve question.
Discuss progress on literature review and methodology, structure, sources, methodology approach (onion)
Agreed other books to be used, finish methodology, brief outline of philosophy, aim to finish for original deadline.
Literature Review and Methodology 1st draft, discuss structure.
Alter structure with proposed one, insert more content for submission day.
118 | P a g e
Discuss feedback from literature review and methodology to establish where mistakes were and how to improve on them
Improve structure, find more academic sources, Make methodology and sample clearer. Improve primary research questions.
Discuss Coding with John Hathaway - Not sure if I was doing it correctly.
Coding done correctly. No action
Key Skills Review Through my time at Buckinghamshire New University, I have developed many new skills, from writing skills, presentation skills, public speaking skills as well as many others that will aid me in the transition from education to working in the real world. Firstly I will discuss my writing skills. During my three years at university, I have noticed my writing skills improved, as this is a fundamental part of working in Public Relations. My tutors helped me get to this stage by ensuring that tasks are set to help me to practice my writing skills, while providing feedback for the same. Following my internship, which was arranged by my tutor again, I have been actively writing press releases, blogs and social media postings on a regular basis, further improving my skills by obtaining further feedback from my employer. By having this opportunity, I was able to really stamp out bad habits I had when writing, such as, using commas incorrectly, while improving on grammar and increasing vocabulary with industry standard terms, which I feel will allow me to increase my chances in finding future employment.
119 | P a g e
The second skill which I feel has improved in my time at university, is the one of speaking in front of members of the public. Having completed public speaking examinations when younger, I had got out of the habit of speaking in front of large groups of people, thus making me feel very nervous, since being at university, and having to present in front of peers and external persons, who are industry experts helped me to improve my spoken presentations as well as teaching me to maintain eye contact with people while talking to them. I now feel extremely confident in approaching people and comfortably and confidently speaking with them, while having an insightful conversation. This has helped me from not only approaching lectures in university, but also with networking and generally in approaching people whom I have not spoken to previously. Now, I feel, if put on the spot to discuss a subject with people, I will be able to discuss the topic with ease, whether I know a vast amount on the subject or not. I feel that is skill will help me the most out of all of the skills in my chosen career, as in PR is important to be able to confidently speak and engage with people. The third skill that I have developed, is my ability to carry out research quickly, accurately and effectively, I feel that having access to a wide range of sources has helped me to carry out various analysis's on briefs for assignments. I feel that this was possible, due to the number of interesting briefs provided by the teaching staff, to help put the classroom and academic content into a real life perspective. The fourth main skill that I feel I have learnt during my time at university is the skill that has allowed me to further my career opportunities, was the presentation skills that we were taught. By having a knowledge in how to present effectively, it was possible to add this to my public speaking skill, meaning I can stand in front of a panel of professionals and discuss my idea's and plans with great ease. I feel that this will help me in my career as it will allow me to present to clients with pitch idea's and to keep them enticed in what I am saying by creating a mini play and maintaining eye contact with all members in the room.
120 | P a g e
Finally, my final main skill, is leadership, I feel that this skill is going to help me take control in a work environment if needs be. This skill will help me show future employers that I am a responsible individual, who is able to organise teams to carry out tasks, whilst ensuring that I still have work to carryout to help make a difference to the outcomes. I feel that the above skills outlined will help me to improve my work life as I feel I have developed well as a human being to become an excellent PR officer and team player.
Published on May 9, 2013
This dissertation was written to evaluate the measurement of contemporary Public Relations. This topic was chosen as it questions a long sta...