BuildaBridge Program Evaluation
BuildaBridge Program Evaluation Rosy Briones, Cleasha Brown, Brian Gilmore, Welton O’Neil Drexel University Policy 504 March 18, 2011 Dr. Isett
1
BuildaBridge Program Evaluation
BUILDABRIDGE PROGRAM EVALUATION Background The BuildaBridge Institute is a Philadelphia-based education and training program that imparts the knowledge and skills necessary for utilizing the arts in educational and community service projects. The training is intended for artists, educators, and community and religious leaders. Participants learn how to employ the arts in service to others. BuildaBridge provides a variety of training formats (See Table 1 below). Foundation and applied arts courses constitute the core of the training. These courses are presented in an intensive, five-day residency format, referred to as an Institute. They cover topics such as Foundational Theory, Arts Management and Education, and Applied Arts and Collaborative Art-Making. Institutes occur primarily in Philadelphia. BuildaBridge occasionally offers international Institutes that focus on a general educational theme (e.g. the study of iconic art and economic development in the Republic of Georgia).
Participants can enroll in the full, five-day Institute, or attend for a single day or course. Those who participate in two full Institutes and a twelve month internship are awarded a Certificate of Training in healing and community arts. Through a partnership with Eastern University, BuilaBridge also offers a Masters in Arts in Transformation. This twenty four month program includes two Institutes, two leadership practicum in community arts organizations, and additional university coursework. Since its inception, BuildaBridge has trained over five hundred individuals. This year marks the ten year anniversary of the first BuildaBridge Institute.
2
BuildaBridge Program Evaluation Table 1: Training Formats Offered by BuildaBridge Program Institute International Institute
Length 5 day residency Varies
Certificate Program
12 months
M.A. in Arts in Transformation
24 months
Components Foundation and Applied Arts Courses Country specific educational theme Two Institutes, online classes, and an internship with a community arts organization Two Institutes, two leadership practicum with a community arts organization, and additional Eastern University coursework
Purpose of the Evaluation The purpose of this evaluation is three-fold: To assess the effectiveness of the BuildaBridge training; to explore strategies for expanding and improving the Institute; and to develop methods for BuildaBridge to engage alumni. BuildaBridge has a large alumni base of over five hundred individuals. The past participants are geographically dispersed, which makes soliciting feedback and maintaining contact a challenge. This evaluation provides BuildaBridge an opportunity to reestablish contact with its alumni base. Through this alumni contact, BuildaBridge can evaluate and improve the Institute training program.
According to the stated goals of BuildaBridge, participants in the trainings should have the ability to: 1. Articulate a clear philosophical basis for the arts in community service 2. Write an initial strategy and goals for using the arts in community service 3. List critical indicators for a successful community arts program 4. Understand the role the arts play in the spiritual and psychological development of children and youth 3
BuildaBridge Program Evaluation 5. Understand the impact of the arts on learning and cognition 6. Demonstrate the foundations of one art form 7. Develop networks for support in developing a community arts program The evaluation will assess whether participants in the various BuildaBridge programs utilize the above knowledge and skills in their current fields of work.
In addition, BuildaBridge hopes to improve and expand the Institute by providing online courses and developing a scholarship fund. By assessing alumni interest in participating in online courses or contributing to a scholarship fund, BuildaBridge can evaluate the feasibility of these two strategies. Finally, BuildaBridge will re-engage alumni through this evaluation by inviting them to attend and participate in the ten-year anniversary celebration of the Institute.
BuildaBridge Goals for the Project The BuildaBridge Institute set forth goals to allow for the success of its future endeavors. Their stated goals for the evaluation survey were a result of the following long term, intermediate, and short term goals.
Long Term Goals: To achieve a complete database of alumni records and information, evaluate how alumni have implemented the education and training received through the BuildaBridge Institute, establish the level of interest of continued alumni contact with the BuildaBridge Institute through online courses, and obtain feedback in regards to possible donations for scholarship purposes.
4
BuildaBridge Program Evaluation Intermediate Goals: Organize information gathered from this survey analysis to inform strategic planning of the organization.
Short Term Goals: Organize a proper database and survey collection method that will best suit the needs of the program evaluation in order to achieve both intermediate and long term goals. With the 10 year anniversary approaching, the survey also asked if any past alumni would participate in the events scheduled to take place.
Survey Implementation Method In order to achieve the goals stated above, a survey was created and distributed to past alumni. BuildaBridge provided contact information for participants from 2005 to present day. The data included participants from all types of BuildaBridge trainings. Given the wide geographic distribution of alumni, and constraints of time and cost, a web-based survey design was chosen. BuildaBridge had email addresses for the majority of the alumni, making the web-based approach an appropriate method for gathering responses.
The survey was distributed to participants’ emails through BuildaBridge’s Survey Monkey account. Three hundred twenty four alumni received a request to complete the survey. The initial survey request included a brief description of the purpose of the survey and the expected length of time for completing the survey (approx. thirty minutes). Those who did not respond to the initial request received two reminders at four day intervals. At most, respondents received a total of three requests to complete the survey.
5
BuildaBridge Program Evaluation The survey was divided into ten sections that corresponded to the questions to be addressed through the evaluation. Questions appeared in a variety of formats, including yes/no questions, open-ended questions, and questions using a five-point likert scale. The variety of question formats was necessary given the variety of information being requested by BuildaBridge. The first section collected historical information about the respondents including how and when they participated in the BuildaBridge trainings. This section also asked respondents about their current level of contact with the organization and their general satisfaction with the program (using the five-point likert scale). Sections two and three collected both demographic information and information about respondents current work situation (i.e. type of organization, title/position, and size of the organization).
The remainder of the survey focused on the questions specific to the BuildaBridge trainings and the proposed strategies to expand and improve the Institute. As series of yes/no questions based on the seven goals for all Institute participants (see Background section of a list) rated respondents’ level of implementation of the trainings. This was followed by a series of openended questions related to BuildaBridge’s strategic planning.
The next two sections of the survey focused on proposed strategies for improving the BuildaBridge Institute. A series of questions utilizing the five-point likert scale assessed respondents’ feelings about and interest in online classes offered by BuildaBridge. This was followed by a variety of questions about donating to a scholarship, contributing in other ways to the organization, and reflecting on the impact of the training. The survey concluded with a series of questions inviting respondents to attend the ten year anniversary of the institute and
6
BuildaBridge Program Evaluation highlighting the variety of ways alumni can participate in the celebration. Finally, respondents were thanked for their participation and given the option to update their contact information with BuildaBridge.
Data & Results The following section highlights data and results from the survey. Quantitative and qualitative information is presented. BuildaBridge can access the full data through their Survey Monkey account. In addition, the survey will remain open, allowing BuildaBridge to collect more responses, which may alter these results. Section I: Quantitative Results Figure I Question asked: Respondents were asked if they had designed or participated in a program that has applied a philosophical basis for the arts in the community service?
7
BuildaBridge Program Evaluation Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 29 (70.7%) responded Yes; 9 (22%) responded No; and 3 (7.3%) responded as Not Applicable. Figure II Question asked: Respondents were asked if they written/ or assisted in an initial strategy and goals for using the arts in community service.
Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 28 (68.3%) responded Yes; 9 (22%) responded No; and 4 (9.8%) responded as Not Applicable.
8
BuildaBridge Program Evaluation Figure III Question asked: Respondents were asked if they created or participated in a program that has established critical indicators for a successful community arts program?
Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 17 (41.5%) responded Yes; 16 (39%) responded No; and 8 (19.5%) responded as Not Applicable.
9
BuildaBridge Program Evaluation Figure IV Question asked: Respondents were asked if they applied or increased the role the arts play in the spiritual and psychological development of children and youth?
Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 31 (75.6%) responded Yes; 5 (12.2%) responded No; and 5 (12.2%) responded as Not Applicable.
Figure V Question asked: Respondents were asked if they have engaged the arts as a way to improve academic learning and cognition?
10
BuildaBridge Program Evaluation Figure V
Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 30 (73.2%) responded Yes; 7 (17.1%) responded No; and 4 (9.8%) responded as Not Applicable.
Figure VI Question asked: Respondents were asked if they have created or participated in the demonstration of the foundations of one art form?
11
BuildaBridge Program Evaluation Figure VI
Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 29 (70.7%) responded Yes; 8 (19.5%) responded No; and 4 (9.8%) responded as Not Applicable.
Figure VII Question asked: Respondents were asked if they created or participated in the development of networks for support in developing a community arts program?
12
BuildaBridge Program Evaluation Figure VII
Results: Out of the 47 people who started the survey, 41 responded to this question. Out of the 41 people, 22 (55%) responded Yes; 13 (32.5 %) responded No; and 5 (12.5%) responded as Not Applicable.
Q: Rate your agreement with the following statement: The information I gained through BuildaBridge could be taught in an online format: Answer Choices
Response Percent
Response Count
Strongly Agree
10.50%
4
Agree
52.60%
20
Neither Agree nor Disagree
26.30%
10
Disagree
13.20%
5
Strongly Disagree
0%
0
answered question
38
skipped question
9
13
BuildaBridge Program Evaluation
Q: How likely would you be to participate in online seminars or courses offered by BuildaBridge? Answer Choices
Response Percent
Response Count
Very Likely
23.70%
9
Likely
39.50%
15
Neutral
26.30%
10
Unlikely
7.90%
3
Very Unlikely
2.60%
1
answered question
38
skipped question
9
Q: How likely would you be to recommend the online trainings or courses to others? Answer Choices
Response Percent
Response Count
Very Likely
29.40%
10
Likely
50.00%
17
Neutral
17.60%
6
Unlikely
2.90%
1
Very Unlikely
0%
0
answered question
34
skipped question
13
14
BuildaBridge Program Evaluation
Q: How likely would you be to contribute to a scholarship fund for future BuildaBridge participants? Answer Choices
Response Percent
Response Count
Very Likely
14.30%
5
Likely
31.40%
11
Neutral
40.00%
14
Unlikely
11.40%
4
Very Unlikely
2.90%
1
answered question
25
skipped question
12
Q: Overall, how satisfied are you/were you with the BuildaBridge training or education you received?
Answer Choices
Response Percent
Response Count
Very Satisfied
74.50%
35
Somewhat Satisfied
21.30%
10
Neither Satisfied nor Dissatisfied
4.30%
2
Somewhat Dissatisfied
0%
0
Very Dissatisfied
0%
0
answered question
47
skipped question
0
15
BuildaBridge Program Evaluation
Rate your agreement with the following statement: I found the skills I learned at BuildaBridge to be useful.
Answer Choices
Response Percent
Response Count
Very Likely
64.30%
27
Likely
28.60%
12
Neutral
7.10%
3
Unlikely
0%
0
Very Unlikely
0%
0
answered question
42
skipped question
5
Section II Qualitative Results Method applied: Inductive Analysis - The inductive approach/analysis can be defined as a systematic procedure that can be used when analyzing qualitative data. This type of analysis is guided by various yet specific evaluation objectives along with using various methodological approaches when applying inductive analysis. The analysis approach focuses mainly on the detailed readings of data in order to determine ideas, themes, or a type of model that involves interpretations (Thomas 2006). By applying this type of analysis, the evaluators were able to determine that most of the feedback provided for the open ended questions was positive feedback. Examples of questions and responses are below.
I. Q: What one statement would you use to describe the BuildaBridge training?
16
BuildaBridge Program Evaluation Out of 41 respondents, 36 answered this question. Although the responses were varied, they all gave positive feedback. Most of the respondents used key words such as informative, fun, powerful, intense, and challenging. Examples of responses are below. 1) “A dynamic, engaging, creative training that opens your mind to being innovative and excited about using the arts in transformation.” 2)“…passionate, practical and preparational insights into serving and equipping marginalized people to better understand themselves and their community through the arts.” 3) “…evidence based, theory informed, exploration of the power of arts and creative expression to facilitate a transformation towards health and wellbeing on an individual and collective scale.” 4) “A practical, philosophical and experiential study of Transformative Arts.”
II. Q: What did you find most beneficial about the BuildaBridge program? Out of 41 respondents, 30 answered this question. Feedback was positive. A majority of the respondents felt that the program they participated in provided hands on training, networking between people very passionate within their fields, and provided business partnerships. See below for example responses. 1) “…getting together with other educators and artists and learning about what they were doing and also getting the opportunity to see them in action working with kids in urban communities and being able to participate in that as well.” 2) “The chance to work with kids, to experience teaching alongside mentors, the integration of arts therapy, creative expression, and child and human development.” 3) “…on site participation - the buildabridge classroom experience at the shelter. “ 4) “…the training, the network of people passionate about the same thing as me.”
III. Q: What phrase would you use in your recommendation of the program?
17
BuildaBridge Program Evaluation Out of 41 respondents, 29 answered this question. Most of the feedback consisted of respondents recommending the program to a person by using words/phrases such as inspiring, a way to use the arts to help people, hands on experience, and one that is unique. 1) “A proven and effective method for bringing hope and healing to children, families and communities within a structured and wholesome environment.” 2) “An arts approach to social change that utilizes learning from social, academic and spiritual constructs.” 3) “Joins people together across lines of cultural, ethnic, and aesthetic experience. “ 4) “Wonderful, unique, positive, unique, experience that brought together many people from diverse backgrounds to work on a common goal “
IV. Q. What skills from the training have you utilized? Out of 47 respondents, 47 answered this question. Some utilized the skills of management within the classroom, arts, and lesson content. Others utilized the skill of networking. 1) “Classroom management, lesson planning. “ 2) “Classroom management, lesson writing, engaging students, creating a safe place to create.” 3) “ linking metaphora to artistic activities, involving children in lessons, creating curricula and lesson planning, understanding homelessness surroundings and how to deal with trauma behaviors, some knowledge of non-profit organization management, communication with children and participants.” 4) “The positive, reaffirming aspects of interacting with people (children) who are daring to make art and engage with a world that has often treated them harshly.”
18
BuildaBridge Program Evaluation Section III Discussion The success of the survey was determined by the number of respondents, truthful responses, and amount of time allotted for survey participation. Alumni that participated in the survey gave a great deal of positive feedback in regards to the seven stated goals for all participants in the BuildaBridge training. This suggests that the program directors and facilitators were able to properly train and instill the skills needed to further advance the use of art therapy in community service. The majority of respondents felt they were meeting the goals set for by BuildaBridge. There was a high non-response rate for the open-ended questions, which negatively impacted the potential of collecting information on the programs past participant’s experiences. However, those who did responded provided positive feedback. The training was described as engaging, fun, interactive, wonderful, and unique. Over ninety percent of respondents felt the skills they learned at BuildaBridge were useful. Respondents clearly felt they increased their knowledge and skills regarding using the arts in service to others, and enjoyed the experience.
The responses to the strategies for improving and expanding the training program were not as positive. Although a majority of respondents felt that the training could be taught in an online format and indicated that they would be interested in classes, the feedback was less positive when compared to the training feedback. When asked to describe the training, respondents called the training “challenging” and “interactive”, and stated that they enjoyed the ability to meet and network with others in their field. It is possible that the success of the training is contingent on the challenging and interactive nature of the experience. Respondents may have felt that an online format for the training would fail to capture that experience. There is a possibility the online training would not be as versatile, and that these skills are better taught in person. Or, if an 19
BuildaBridge Program Evaluation online course is produced, the training might have to be less complex and challenging, and may not achieve the same positive results.
In addition, the alumni were impressed by the program and would like to network more with artists. However, they appeared reluctant to donate to a scholarship fund. Their reluctance to donate does not seem to be a result of the alumni dissatisfaction, as ninety five percent of responded reported satisfaction with the training. It may be that alumni do not feel connected to the program after the training is complete. If BuildaBridge were to plan more social events or develop other ways to engage past participants, the alumni may be more amenable to donating to a scholarship fund. One purpose of the survey was to do just that, by inviting alumni to the ten year anniversary of the Institute.
Suggestions for Improving the Survey The following are a few suggestions for improving the survey. The survey is forty one questions long, with a variety of question formats, including open-ended questions. Due to its extensive length, it takes longer for respondents to complete the survey. Thus, respondents are more likely to not finish the survey, skip questions, and/or partially answer questions. Gelesic & Bosnjak (2009) evaluated the impact and actual length of the survey in their article “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey.� Their results indicated that response rates and data quality are inversely related to expected and actual questionnaire rate in web surveys. That is, as the length of the questionnaire increases, response rate and data quality decreases. A longer survey represents an increased burden, which reduces the likelihood that individuals will begin the survey. If they do complete the survey their
20
BuildaBridge Program Evaluation lack of motivation will prevent them from providing high quality answers. Reducing the survey length could increase response rates and data quality. Also, lower response rates may have been due to the length of time since respondents participated in the training. Conducting surveys closer to the end of the program might increase response rates and data quality. Finally, administering a pre-training survey could provide BuildaBridge a baseline measure of participants’ knowledge and skills prior to the training and increase the validity of their assessment of the training.
Conclusion The purpose of the ex-post survey evaluation was to understand the effectiveness of the BuildaBridge Institute, which provides knowledge and skills to participants utilizing the arts in community service, and exploring two strategies for improving and expanding the Institute, through online classes and a scholarship program. The results of the survey showcased the amount of skills the institute trained each past participant in and how they were able to apply those skills in their current programs. The results were overwhelmingly positive and it is clear that BuildaBridge Institutional programs teach the abilities needed to implement a successful art therapy community program. The results indicate a positive response to the two strategies for improving the institute, although the response was not as positive as compared to questions specific to the training. Through the evaluation, BuildaBridge successful engaged forty seven of its past participants, who provided valuable feedback. It is recommended that BuildaBridge continue to collect information from past and future participants in the Institute in order to have a complete and accurate understanding of the impact training, and to inform BuildaBridge’s strategic plan for the future.
21
BuildaBridge Program Evaluation References Thomas, D. R. (2006). A General Inductive Approach for Analyzing Qualitative Data. American Journal of Evaluation Galesic, M. & Bosnjak, M. (2009) Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly. Vol. 73 (2), pp 349-360
22