BlueCabin ImpactFramework
\\February2024
Introduction
TialthasbeencommissionedbyBlueCabintodevelopalearningandevaluationframeworkfortheorganisation.Thisbuilds ontheStoryofChangeandevaluationpracticesthatBlueCabinhasbeenusinginrecentyears.Wearenotre-inventingthe wheel.Wewanttoenhancetheextensiveworkthathasalreadygoneintohowtheorganisationexploresanddemonstrates itsimpact,ideallymakingtheapproachmorefocused,communicableandfitforpurpose.Ourapproachtothisprojectis:
Iterative–WeknowthatBlueCabinaretheexpertsintheirpractice.Wehaveexperienceindesigningevaluationframeworks andmethods,butwecanonlygettoapointoffindingaworkableapproachbyiteratingtheframeworkwiththeBlueCabin team.
Creative–Wewanttheframework,methodsandtools,tobeembeddedinthecreativepracticeoftheorganisation.Wethink evaluationpracticeshouldnotbesoseparatefromcreativepracticeandwillseektomaketheapproachfeelmorenaturalto theorganisation,practitionersand,mostimportantly,participants.
Inclusive–Atallstageswewillchallengeourselvesandeachothertoconsiderhowtheevaluationprocesswillbe‘received’ byparticipantsandhowwecanensurethatpeoplefeelincontrolofthedatatheyshareandwhatitmaybeusedfor.
Thisframeworkoutlinesthekeyevaluationquestions,logicmodel,assumptions,methods,andresponsibilities,which hopefullyfeelslikeaworkableapproachfortheorganisationmovingforward.
What’snew?
TialthaverefocusedBlueCabin’scurrent‘Storyof Change’to:
● Definingtwoclearoutcomesacrossallprojects
● Highlightingintrinsicandextrinsicimpactsof projects
ThesedevelopmentswillallowBlueCabintobetter articulatetheirstoryasawhole(throughtwoclear outcomes),whilstnotunderminingthevarietyof impactsoccurringinprojects.
Thesedevelopmentsalsoallowforaclearlensthrough whichtoexploreandevaluateprojects,guidedbythe4 evaluationquestions(detailedonthenextslide).
EvaluationQuestions
● WhatistheimpactoftheworkofBlueCabin acrossitsprogrammesandasan organisation?
● HowcanBluecabinlearnfromtheirworkin awaythatallowsthemtodevelopand improvetheirimpact?
● HowcanBlueCabinuselearningfromtheir ownprogrammestobuildonexistingtheory andpractice?
● HowcanBlueCabinsharethislearningand thestoryofBlueCabinmorebroadlywith theirpeers,fundersandsupporters?
BlueCabin’sCurrent
‘StoryofChange’
ExistingPrinciples
BlueCabinhasalreadyembeddedarangeofprinciplesinitsworkandhasestablishedaneffectivestoryofchange, basedontheprincipleofmeaningfulrelationshipsbetweencareexperiencedindividuals,andthepeopleand organisationswhoarepartoftheirlives.Creativityisattheheart.Thisincludes:
Aunique,Trauma-Informed Approach,acrossareas,such as:
● Creativeprogrammes andinterventionswith CareExperienced Individuals(CEI)
● Creativelifestorywork withCEIpartneringwith LocalAuthorities(LA)
● SupportingCEIinprisons
● Widersectorsupport
● Improvingrepresentation ofCEI
Indicatorsandevidenceof practiceinclude,butarenot limited,to:
● NumbersofCEI,trusted adults,LA,organisations andfreelancerstaking partinprojects
● IncreasednumberofCEI employed/volunteering withBlueCabin
● Numberof podcasts, learningeventsand informationsharing opportunities
Outcomes,includebutare limitedto:
● Increasedloveofthearts
● Developingrelationships
● Increasedemotional literacy
● Enhancingskills
● Improvingconfidence
● Increasedsenseof belonging
● Voicesandopinionsof CEIinformingBlueCabin practice
Furtherconsiderations
There’salot:duetotherangeandscaleofBlue Cabin’swork,therewillbelotsofdifferentoutcomes fordifferentpeople,farmorethanmostarts organisations.
Capturingtheoutcomes:withsuchavarietyof outcomesanticipatedfromtheprojects,capturing theoutcomeseffectivelyiscomplex,takesalotof timeandneedsdifferentmethodsforeachproject.
Tellingthewholestory:duetothenumberand varietyofanticipatedoutcomes,itiscomplexfor BlueCabintoshareacomprehensivebuteasyto understandstoryoftheorganisationanditswork.
Indicatorsandevidence:thecurrentindicatorsand evidenceBlueCabinusemaynotbecapturingthe depthofoutcomesacrossprojects
BlueCabin’sUpdated ‘TheoryofChange’
BlueCabinworksacrossmultiplestrandstoachievesystemschange forcareexperiencedchildrenandyoungpeople…
Learning projects
Creative LifeStory work
Workwith Local Authorities
Workwith cultural professionals
Creative Aid Systemschange
Bespokeinterventionsforcare experiencedyoungpeople
Sectorsupportand professionaldevelopment
Enhancingthesectorand promotingbestpractice
Specificprojectfundingfor careexperiencedyoung peopleandcarersfromvarious trustsandfoundations(incl. EsméeFairbairnFoundation, SegelmanTrust,SirJames KnottTrust,GarfieldWeston, CommunityFoundationTyne andWearandThe LeathersellersFoundation,Arts CouncilEngland,National LotteryCommunityFundand others).
ServicessoldtoLocal Authoritiesforstafftraining anddevelopment.
MatchfundingfromLocal Authoritiesfor
Timeandresourcesfromcore staffandfreelancers,boardof trustees
CreativeProgrammesand Interventions–tacklingCEIbarriers accessingarts(CoolThings,Thisisthe Place,FeelCreative,others…)
CreativeLifeStoryWork–supporting CEIexploringprecareandcare experiences
Prisons–Providingcreativeprovisions forCEIsinprisons(CreativeAid)
SectorSupport–Developand increaseknowledgeofbestpractice ofworkingwithCEI(SouthTyneside DFEproject,TheStudioofFostered Heroes,others…)
Representation–IncreaseCEI representationinBlueCabin (employment,volunteering,others…)
TraumaInformed–Becominga traumainformedorganisationacross activities
NumbersofCEIs,trustedadults, LAs,organisationsandfreelancers takingpartincreativeprojects, trainingetc.
NumberofCEI employed/volunteeringwithBlue Cabin
Numberof podcasts,learning eventsandinformationsharing
Numberofcreativesessions delivered
Numberoffreelanceartists employed
Numberandrangeofpartners
04
Outcomes
Impacts
05
BlueCabin’smainwayof achievingimpactis through...
DevelopingRelationships
BetweenCEIs,carers,trusted adults,families, organisations,BlueCabin, artists,LAs,professional, funders,thesectorand others…
Strengthening Relationships
BetweenCEIs,carers,trusted adults,families, organisations,BlueCabin, artists,LAs,professionals, funders,thesectorand others…
CEIs:
Intrinsicimpacts:
• Improvedconfidence
• Increasedemotionalliteracy
• Increasedresilience
• Increasesenseofbelonging
• Increasesenseofpride
Extrinsicimpacts:
• Beingseen,heardandvalued
• Developing/increasingloveofthe arts/creativity
• Skillsdevelopment
• Accreditation
• Improvedrelationships
• Recordoflifestory
• Lesshome/schoolmoves
• Beingrecognisedasartists Artists:
Intrinsicimpacts:
• ImprovedconfidenceworkingwithCEIs
• Increasedresilience(throughsupervision) Extrinsicimpacts:
• Skillsdevelopment(traumainformed practicesaswellasbroaderartistic practice)
• Regularemployment,sustainingartistic practice
ThoseconnectedtoCEIs(trustedadults,families, carers,others…):
Intrinsicimpacts:
• Improvedconfidence
• Increasedresilience
Extrinsicimpacts:
• Developing/increasingloveofthearts/creativity
–recognisingvalueofarts
• Skillsdevelopment
• Improvedrelationships(withCEIsaswellas others…)
Organisations(BlueCabin,LAs,arts organisations,thewidersector,others…):
Intrinsicimpacts:
• BCstaffandfreelancersresilientandenjoytheir work(throughsupervisionandothersupport provided)
• ImprovedconfidenceinworkingwithCEI
Extrinsicimpacts:
• LongtermLAcontracts
• Enhancednetworks
• Improvedskillsandknowledge
• BCrepresentativeofCEIviewsandopinions
• RetentionofpeoplewithinBC
• Improvedabilitytoadvocateforworkaswellas artsandcreativityworkingwithCEI
Unpackingtheprogrammes, outcomes&impacts(1/2)
StrengtheningRelationships–CEIsbeinggiventhe opportunitytosustaintheirrelationshipswithmany people,aswellastothemselves,throughcreativity (suchascarers,orwithfamiliesthroughcontacttime, virtualschoolstaffandothers)
ForexamplethroughCreativeLifeStorywork:carving outregularprotectedtimewithcarerstoshare
DevelopingRelationships–CEIsbeinggiventhe opportunitytodeveloptheirrelationshipswithmany people,aswellastothemselves,throughcreativity (suchascarers,orwithfamiliesthroughcontacttime, virtualschoolstaffandothers)
ForexamplethroughCoolThings:shortopportunitiesto connectthroughcreativefunprojects,forCEIsand carers,aswellasorganisationswiththoseattending CoolThings
Unpackingtheprogrammes, outcomes&impacts(2/2)
WhyDevelopingandStrengthening Relationships:
ThehugevarietyofimpactsinBlue Cabin’scurrent‘StoryofChange’ makeitdifficulttosharethestoryof BlueCabineasily.Separating outcomesfromimpactsmeansBlue Cabincaneasilyidentifyandshare outcomes(developingand sustainingrelationships),which encapsulatethehugevarietyand depthandbreadthoftheimpactsof BlueCabin’swork.
WhyIntrinsicandExtrinsicimpacts:
BlueCabindeliverprojectswhich createbothtypesofimpacts.Being abletohighlightwhichprojectis anticipatedtohavewhichtypeof impactallowsforbetterknowledge ofappropriateindicators,evidence andevaluationmethodstobeused. Italsohighlightstostakeholders, fundersetc.thebreadthanddepthof BlueCabin’swork.
IndicatorsandSources ofEvidence
Indicatorsandsourcesof evidence:CEIs
Outcomes Quantitativeindicators
Qualitativeindicators Sourcesof evidence
CEIs:
• Developing Relationships
• Strengthening Relationships
− NumberofCEItakingpartin projects
− Numberssustainingparticipation overtime(throughCoolThings, CLSWandothers…)
− NumberofCEI volunteering/employedatBlue Cabin
− Lesshome/schoolmovesofCEIs
− CEIsexpressmeaningful connectionsandrelationships beingformed
− CEIsdescribehowtheyhave triednewcreativethingsandare opentodoingmore
− CEIsdescribeachangeinhow theythinkaboutthemselvesand whattheyarecapableof
− CEIsexpressfeelingheardand valued
− CEIsdescribewillingnessand motivationtoengageinmore creativeactivities
− Monitoringdata collectedacrossall projects
− Qualitative informationfrom staffdelivering projects
− Additional qualitative methods(explored onpage21)
Indicatorsandsourcesof evidence:Artists
Outcomes Quantitativeindicators
Qualitativeindicators Sourcesof evidence
Artists:
• Developing Relationships
• Strengthening Relationships
− Numberofartistsemployedin projects
− Numberofartistsregularly employedbyBlueCabin
− Numberofartistsupervision sessions
− Numberofartistsworking togethercollaborativelyon projects
− Artistsexpress learning/developingnewskills
− Artistsexpressconfidencein workingwithCEIs
− Artistsfeelsecurein employmentwithBlueCabin
− Artistsexpressfeelingresilient andenjoyingtheirworkwith BlueCabin
− Monitoringdata collectedacross allprojects
− Qualitative information fromartists
− Additional qualitative methods (exploredon page21)
Indicatorsandsourcesofevidence: ThoseconnectedtoCEIs
Outcomes Quantitativeindicators Qualitativeindicators
Sourcesof evidence
Thoseconnectedto CEIs:
• Developing Relationships
• Strengthening Relationships
− Numberofthoseconnected toCEItakingpartinprojects
− Numberssustaining participationovertime (throughCoolThings,CLSW andothers…)
− ThoseconnectedtoCEIs expressmeaningful connectionsandrelationships beingformed
− ThoseconnectedtoCEIs describehowtheyhavetried newcreativethingsandare opentodoingmore
− ThoseconnectedtoCEIs describewillingnessand motivationtoengageinmore creativeactivities
− Monitoringdata collectedacross allprojects
− Qualitative information fromstaff delivering projects
− Additional qualitative methods (exploredon page21)
Indicatorsandsourcesof evidence:Organisations
Outcomes
Quantitativeindicators
Organisations:
• Developing Relationships
• Strengthening Relationships
− NumberofLAcontracts
− LengthofLAcontracts
− Numberoforganisationsworked withthroughprojects
− Numberofpodcast,reportand learningmaterials downloaded/visited
− Participantsincreative curriculumwithinprison
− Numberoforganisations regularlyworkingwith/seeking supportfromBlueCabin
− Numberofsupportand supervisionsessionsdeliveredto BCandLAstaff
− StaffretentionwithinBC
Qualitativeindicators Sourcesof evidence
− StaffaswellasLAstaffand partnerorganisationsexpress meaningfulconnectionsand relationshipsbeingformed
− Organisationsexpressskills developmentandknowledge developmentofworkingwith CEIs
− CEIexpressfeelingrecognised withinBCstrategyand organisationmorebroadly
− Organisationsexpressimproved understandingintrauma, attachmentandrecovery
− Monitoringdata collectedacross allprojects
− Qualitative informationfrom staffand organisations
− Additional qualitative methods (exploredon page21)
Methods
QuantitativeMethods
Projectandparticipantmonitoring–WewillsupportBlueCabintoexplorethe wayinwhichparticipationstatisticsarebeingstoredandanalysed(e.g., numbersanddemographicsofparticipantsacrossprojectsandprogrammes). WewillaimtosupportBlueCabininconsideringhowinformationisstored,what informationcanbegatheredandhowtobestgatherevidenceofBlueCabin’s work.
Internalrecordkeeping–Aswiththeotherquantitativemethodshere,theremay beadditionalevaluationmaterialthatisgeneratedbyprojectsandcanbe storedinconnectionwithprojectmonitoringdataandtheparticipantfeedback records(e.g.,producedbyLAstaff,referralpartners,emailscarersetc).Wewill encouragetheteamtotakeaconsistentapproachtohowthisdataisstoredand used,andwhenitmaybeusefulforongoingadhocinternalevaluation.Wewill alsosupportBlueCabininconsideringbestwaysofusingthisdataandhowit canbeevaluatedwithease.
QualitativeMethods(1/2)
ExperienceCards(seefollowingslides)–WithBlueCabin,tialtwilldevelopasetofexperiencecards,whichcanbeused acrossprojectstotellthestoryofBlueCabinacrosstherangeofitsprojectsandprogrammes.AsBlueCabinworkacrossa rangeofprojectsandprogrammes,itisdifficulttosharetheentirestoryofBlueCabin:theseexperiencecardswillhelpwith that.TheywillalsosupportCEIs,aswellasothers,toexploreandexpresstheirexperiencesofBlueCabinprojectswithout havingtorelyonwords,takingthepressureoffofCEIs(andothers)whenevaluatingprojectsandprogrammes.Thesecards willmirrorthecreativepracticesofBlueCabinitself.
Three-questionfeedbackformforartists-WeknowBlueCabinpractitionersarereflectiveintheirpracticeanyway.Thisis abouttryingtofindawaytoregularlycapturethesereflectionsinlinewiththeintendedoutcomes.Italsoreflectsthe incredibleworkBlueCabindoeswithartistsandsupportstheconceptofBlueCabinasanartisticdevelopmentorganisation. Usingtypeform(orasimilarplatform),wherepractitionerscaneitherwrite(usingphoneorlaptop)oruploadvoice notes/picturesetc.(orboth).Askingthemtorespondtothefollowingthreequestionsafterasession:
Whathaveyoulearned(today/thisweek/thisproject)thatyouthinkwillbemostusefultoyourwork? Howhasthisprojectmadeyoufeel?
Whatisyourresponsetothethemeof‘developingandstrengtheningrelationships’inrelationtoyourexperienceonthis project?
ThesequestionswillalsohelpBlueCabinsense-checkthenewoutcomes(sustaininganddevelopingrelationships)during projectsandexplorewhetherthe framingofoutcomesinthosetermsisusefulanduseable.
QualitativeMethods(2/2)
UpdatedCEIfeedbackpostcards(seefollowingslides)–BlueCabinalreadyusepostcardsastoolsforgatheringfeedback onprojectswithCEIs,howevertheyarenotalwaysregularlyused.Tialtwilldevelopmethodsofembeddingpostcard feedbackintoBlueCabin’scurrentpracticetoconsiderhowBlueCabincanreceivemorefeedback
Words/numbers-CurrentlywithinCreativeAidartistsandpartnerscheckinatthebeginningandendofsessionswith participantsselectingaword,andanumberfrom1-10.Thisisallowingtheprojecttoreflectovereachsessiononparticipants individualexperiencesandintrinsicimpacts.
Updatedevaluationquestions–TilatarepartneringwithBluecabinontheexplorationoftheirprojectCreativeAid,based withinDeerboltPrisonsupportingyoungCEItoexplorecreativityandartandtogainanArtsAwardaccreditation.Tialtwill workwiththeCreativeAidteamtoconsiderwhatquestionsarebeingasked,when,andhowtobestreflecttheexperiences oftheparticipantstakingpartintheCreativeAidproject.
ActionLearningsets–TialtarepartneringwithBlueCabintoexploretheprojectArtfulAlliance,workingwithorganisations andvirtualschoolstodelivercreativeprojectsforCEIs.Theseopen-endedreflectiveandcarefocusedsessionswillprovidein depthqualitativedataabouttheexperiencesofdeliveringtheproject.Thelearningfromthismethodcanbereplicatedas appropriateacrossotherworkstreams.
Potentialcreativemethods: experiencecards&postcards
ExperienceCards:
TialtwillworkwithBlueCabintocommissionan artisttodevelopan‘evaluationgame’,which participantssuchasorganisations,CEIs,carers, staff,artistsandmore,canplaytocapturetheir experiencesofBlueCabin’sworkvisually.
ThecardswillbedevelopedwithBlueCabintobest reflecttheirvisualidentityandtrauma-informed practiceswiththeaimthattheycanbeusedto:
• Exploredevelopmentandchangeof experiencesovertime(suchasthroughCLSW)
• Immediateexperiencesinshorttermprojects, suchasduringCoolThings
• Beusedtoreflectandexploreinastructured gamesessiontheworkofBlueCabin
EvaluationCEIsPostcards:
BlueCabinalreadyusepostcardstoexplorethe impactsoftheirwork.TialtwillsupportBlueCabin toreflectonhowtheyarecurrentlyused,andhow theycanbedevelopedwithinprojectstoensure theyarebeingembeddedwithinprojectsandthat artists,staffandpractitionersarerobustlytrained inhowtomakethemostofthesepostcards throughtheiranalysisandpresentationas‘data’.
What’sDifferentAbout theApproach&What toExpectNext
What’sdifferent&whyisituseful?
Byre-focussingthreekeyareasinBlueCabin’soriginalstoryofchange:
- Definingtwoclearoutcomesacrossallprojects
- Highlightingintrinsicandextrinsicimpacts
- Exploringhowdifferentlevelsofimpactcanleadtosystemschange
BlueCabinshouldbeabletobetterarticulateoutcomesofprojectswithease,aswellas identifywhatmethodsofevaluationareneededfordifferentprojectsandprogrammesof work.ThisshouldallalsosupportBlueCabinintellingthestoryofthewholeorganisation Thestructureoutlinedhereallowsforthefull‘impactnarrative’tobeexploreandshared internallyandexternally
BlueCabinwillbeusingbespokeevaluationmethods,co-createdwiththeteamtobest capturetheexperiencesofthosetakingpartinprojects,whichwillbeiterativeindesign:this meansmethodswillbeconsciouslytestedandre-assessedoveraperiodoftimesothat BlueCabinwilleventuallyhavedevelopedevaluationmethodsandtechniquesthatbest gathertheevidenceoftheirpractices.
ThebespokeevaluationmethodsalsoformauniqueEvaluationToolbox,whichcanbe pickedupandeasilyused,notrequiringnewevaluationmethodstobedevelopedforeach project,streamliningBlueCabin’sevaluationapproach.
Potentialprocesstimeline
Develop qualitativetools withstaff(cards, postcards,action learning)
Reviewmethods andframework withstaffteam
Exploremethods inprojects(Core, CreativeAid, ArtfulAlliance)
Workwithtialtto update2023/24 organisational impactreport
Continuetotestand trialframework throughout2024
