Experimental Design Case Study CIPP

MGT 5000 - Evaluating Research in Management Chapters 13, 14, 15 May 20, 2017 | Alexis Lake Question: Provide a major strength and weakness for each of the approaches covered in Chapters 13,14 & 15, and your opinion on the feasibility of using them for the follow up evaluation as requested in the Chapter 12 case study.

Chapter 13 - Experimental Design Strengths: Results are very clear and quantitative. The controlled and variable groups allows for clear path for what defines success - what works and what does not work. Also a high degree of credibility doe to the scientific nature of the approach.

Weaknesses: Challenging to design and execute especially in a social program where it may not be possible to totally control the variables. There are also ethical issues in terms of some receiving a treatment or benefit and others not. And again, it is difficult to control variables in the real world versus in a scientific lab.

Keeping in mind the strengths and weaknesses of the experimental design, I would not recommend the experimental design approach as a stand alone evaluation method for the Spirit of Consuelo case study.

In the traditional sense, the experimental method requires a randomized control and treatment group. While in medical testing this would be easy to establish, it which would be very challenging to apply in a social program like this housing development. It would be unethical to treat certain people one way and others another in terms of the benefits that they would receive in the community.

Additionally, the basis for this meta evaluation is focused on qualitative questions such as has the community avoided gentrification, have families kept up their properties, have residents become involved with each other in in a supportive community, etc. Experimental design would really give more quantitive data versus the more qualitative information which was requested as part of the meta evaluation.

Lastly, while I agree with Suchman’s principles that the experimental design could be used with other methods to indicate success and whether objectives are being fulfilled, using the experimental design leans more toward a formative approach. In this case, the community is already formed and the organization is looking for more summative information.

Chapter 14 - Case Study Evaluation Strengths: Examines the inputs, processes and the related outcomes. Includes the intended and unintended outcomes. It is also comprehensive, far reaching, and intensely descriptive in explaining a complex issue. Also, very popular and credible method of evaluation.

Weaknesses: Too narrowly focused on the original objective and focus. Also considered a quasi-evaluation approach by the authors. Focus more on qualitative versus quantitive analysis,

which may not be as rigorous as other evaluation methods. No set parameters for conducting case study research, which requires the evaluator to be very component and skilled in interpersonal dialogue.

In the Spirit of Conseulo meta evaluation, I would strongly recommend the case study approach because unlike the experimental design approach, it is inclusive to all stakeholders.

Again, in the request for the meta evaluation, the community leadership is looking for more summative information on what has happened since the community has been established. A case study would lend itself very well for gather this sort of qualitative information because the evaluators could conduct site visits and almost embed themselves in the community to ask these questions.

In terms of the tactics of the actual evaluation, it could be interviews with a cross functional sample of the community, as well as focus groups or surveys. While this would require evaluators with a high degree of social awareness and interpersonal skill, the final report or case study would be very easy to understand.

Additionally, the case study approach, as Robert Stake points out, includes the economic, political, legal, historical background, and physical setting case. All of these influence and impact the results of the community and are very important to the community’s leadership.

The one area to keep in mind would be the time requirement and budget for a case study approach. With the budget being such a concern and lesson learned from the first evaluation it would be important to take into account the travel costs and expenses for having the evaluators on site. These costs could easily stack up over time.

Chapter 15 - CIPP (Context, Input, Process, Product) Strengths: Assesses the needs, problems, opportunities, and assets. Helps decision makers to define goals, outcomes, and priorities. Reviews the current principles and alternatives in order to help a program make needed changes, and whether those changes are viable. Looks an intended and unintended consequences. The stakeholders and benefices

Weaknesses: This is a very idealistic notion of what an evaluation process should be and may be challenging to implement in a smaller setting. Very focused on the objectives and goals of the upper management and takes a top down approach. However, this approach may not take into account some of the politics in an organization’s decision making process. Also, this evolution is best utilized over a longer period of time.

The CIPP model was the original method used in the first Spirit of Consuelo evaluation, and it is the basis for the request and questions in the meta evaluation. This would lend itself very well for being considered a sound evaluation method for this case, however the concerns around the budget make it challenging to recommend.

As the authors outline, employing this method would mean employing much of the same team that did the original evaluation. This could save time versus the case study approach because

the team already has the historical background and context for the community. Additionally, the team might already know and remember many of the key and component stakeholders in the community, leaving time to find and interview newer members.

Additionally, this method clearly presents the intended and unintended consequences and gives clear results from which to derive actionable next steps. Also, the organization would be very familiar with how to read and interpret the results based on their past exposure to the CIPP model.

While these are certainly positive of the CIPP approach, as mentioned with the case study approach, the budget would need to be strongly considered. In the previous evaluation the costs has to be scaled back and with the reduced budget for this meta evaluation, I am unsure whether the CIPP model would be fully deployed and utilized. This would require a smaller scale version of the model, which may leave out some important results or outcomes that could impact the overall success of the community. There could be questions on the request that are limited or eliminated from the final meta evaluation that could be included with a case study approach.