Issuu on Google+

Full file at

Chapter 2: Steps in Program Evaluation Test Bank Multiple Choice Questions: 1. Kapp and Anderson believe the first step in program evaluation is: a. Identifying the evaluation question b. Conceptualizing the evaluation question c. Gathering program information d. Selecting an evaluation design 2. Which of the following steps are involved in conceptualizing an evaluation question? a. Developing evaluation instruments b. Clarifying key concepts c. Selecting data collection methods d. Interviewing participants 3. The “active context� of the evaluation is heightened by the: a. Agency funding b. Agency supplies c. Agency environment d. Agency technology 4. The purpose of program evaluation is the generation of quality information for the improvement of: a. Clients b. Staff c. Funding d. Services 5. Before any discussion of an evaluation can be complete, energy needs to be devoted to: a. Developing an evaluation design b. Developing data collection methods c. Developing a clear vision of the program d. Developing an evaluation question 6. Another name for a program description often used in evaluation is: a. A program design b. A logic model c. A technical model d. A strategic design 7. Discussion of how evaluation information will be used early in the process contributes to the _____________ of the evaluation. a. Accuracy b. Specificity


Full file at

c. Completeness d. Ownership 8. According to Kapp and Anderson, which of the following evaluation steps should be encompassed in identifying program information needs? a. Data collection, conceptualizing the evaluation question and report writing b. Conceptualizing the evaluation question, evaluation design, and data collection c. Identifying, conceptualizing and operationalizing the evaluation question d. Engaging stakeholders, conceptualizing the evaluation question and data collection 9. According to Kapp and Anderson, the last step in the evaluation process is: a. Utilization of evaluation information for service improvement b. Implementation of the evaluation c. Picking and defining evaluation methods d. Developing evaluation reports 10. The design and implementation of an evaluation needs to attend to which basic concept? a. The agency’s technology b. The agency’s supplies c. The agency’s location d. The agency setting True/False Questions: 1. The process for conducting an evaluation requires a basic structure. 2. The evaluation process is linear and consistent. 3. Evaluators always start the evaluation process by identifying the evaluation question. 4. An evaluation question may be influenced by subsequent steps in the evaluation process. 5. Political factors influence the design and implementation of evaluation. 6. It is best to use a single evaluation method for gathering necessary information for an evaluation. 7. An inclusive process should be used to actively involve stakeholders in an evaluation. 8. Developing ways for disseminating evaluation information requires attention at the beginning of the evaluation process. 9. Evaluation questions are the same as survey questions. 10. The program description is a description of the program evaluation.


Full file at

Short-Answer Questions: 1. What does it mean to conceptualize an evaluation question? Give an example. 2. What does it mean to operationalize an evaluation question? Give an example. 3. Explain the difference between a process evaluation and a summative evaluation. 4. What two fundamental notions frame the steps of the evaluation process? 5. Identify the 8 steps often used in a traditional program evaluation? Essay Questions: 1. What is involved in “understanding the agency context� when conducting a program evaluation? How does the agency context affect program evaluation? 2. Compare and contrast the more traditional steps in the evaluation process with those believed to be more effective by Kapp and Anderson. 3. What is involved in engaging key stakeholders in the evaluation process? What are some of the ways an evaluation can change based on stakeholder involvement?


Full file at

Answer Key Multiple Choice Questions: 1. c 2. b 3. c 4. d 5. c 6. b 7. d 8. c 9. a 10. d True/False Questions: 1. T 2. F 3. F 4. T 5. T 6. F 7. T 8. T 9. F 10. F Short Answer Questions: 1. Conceptualizing an evaluation question involves clarifying the key concepts by considering different approaches and selecting one that bets fits your interests and makes the conceptual framework clear. An example of an evaluation question is: Is the family preservation program effective? Conceptualization: Are families better off than if they did not receive the services? 2. Operationalize an evaluation question adds further detail by defining the specific concepts of the evaluation question. For example for the same evaluation question as above: What is the definition of family? What does better off mean? 3. A process evaluation is designed to improve the program while a summative evaluation focuses on assessing the effectiveness of the program. 4. (a) the use of information for service improvement; and (b) the context for evaluation is an agency where evaluation must compete with many enterprises for attention.


Full file at

5. The 8 steps are: (1) identify the evaluation question; (2) conceptualize the question; (3) operationalize the question; (4) pick and define the method; (5) select the sample; (6) data collection; (7) data management and analysis; and (8) utilization of information. Essay Questions: 1. Evaluators must understand the agency’s context in order to successfully carry out the evaluation process. Agency settings are dynamic and constantly focused on the delivery of services. One of the most important things involved in agency context is developing a solid grasp of the program before developing the evaluation question. Understanding the agency context also involves eliciting various points of view and encouraging assimilation of different ideas into a collective model depicting the program, where specific program information needs emerge. The nature of this process encourages the investment of key stakeholders in the definition of the program and information needs for it. Thus, the model for the program and the evaluation questions become more comprehensive and critical players have a stronger identification with the evaluation. 2. The traditional evaluation steps follow a traditional research process, starting with identifying the evaluation question and ending with utilization of information. They do not involve defining the program or engaging key stakeholders in the evaluation. Kapp and Anderson reframe these steps more broadly for agency-based evaluation, combining the first 4 steps from the traditional model as encompassed in the following 3 activities: defining the program with key stakeholders, supporting evaluation users, and implementing and reporting the design. They then combine the steps for implementing the evaluation into one step (e.g., sample, data collection and analysis), and end with utilization. They define utilization differently as ensuring that program evaluation information is used to improve services. They believe their process usually creates an evaluation that is more attentive to the agency context and the agency’s information needs, and one where information is actually used for program improvement. They also believe this process engages stakeholders more actively, thus creating more ownership for the evaluation and its eventual use. 3. Engaging key stakeholder involves them directly and inclusively in the evaluation process, including defining the program model, information needs, and methods, as well as how evaluation information will be used. This allows stakeholders to consider the most important information needs and how the information will be used. Consideration is given to the format that best suits the user and the organizational context. For example, if the users are individual program managers, then the form of the data and method may look different than it would for external funders who an agency may want to get additional resources from. Program managers would most likely want information that matches their specific programs, while funders may prefer information that would complement their standard evaluation and reporting methods.


Test bank agency based program evaluation 1st edition kapp