Dataset of expert novelty scores: a collection of expert-assigned novelty ratings for a sample of recent academicpapersacrossvariousfieldsanddisciplines.Itwilloperateas thetestenvironmentforsubmittedindicators.
Entrant: an individual, team, organisation, orconsortiawhohaveexpressedaninterestin competinginthechallenge
EOI Expression of Interest): the process where prospective participants make their submissionstobecomechallengers.
Metadata: the list of all papers in the dataset (i.e. DOIs & Title), that will be sharedwith challengers.
Metadata Indicator results: the datasets submitted by challengers that include novelty scoresforeachofthepapersincludedinthemetadata.
Novelty: a key virtue often aligning with perceptions of "good research." We are intentionally not providing a prescriptive definition of novelty, as we want to avoid influencingchallengersʼapproachesandallowforabroaderinterpretation
Novelty Indicator: a scalable metric or methodology designed to identify and measure noveltyinresearch
TheMetascienceNoveltyIndicatorsChallengeisaglobalinitiativetodevelopandvalidate scalable indicators that can identify novelty in research. We invite academics, data scientists, researchers, and innovators worldwide to create and submit their novelty indicators to be validated against a dataset of human expert assessments – the best-performing indicator will win a prize of £300,000 alongside an opportunity to have theirtoolimplementedinareal-worldcontext
WhyarewerunningtheChallenge?
Metascience indicators could have a profound impactontheglobalresearchecosystem. Better, timely, and responsibly implemented indicators could shift researcher incentives, enhance our understanding of the factors that make excellence more likely, and in turn, improve the pace of scientific progress – ultimately, helping funders, governments, and academicinstitutionstoyieldmorehigh-qualityresearchoutoflimitedresources
Atthemoment,citationandjournal-basedindicators,albeitcontroversial,arewidelyused across the research and innovation ecosystem. In some ways, these measures do track ourintuitionsonwhat‘goodresearchʼis,butitisanopenquestionwhethertheyareableto captureakeyvirtueofresearch:novelty.
Metascience academics overthepastdecadehavedevisedvariousmeasuresofnovelty, withnumerousexamplesavailableinliterature.Thesemeasuresaregaininginterestinthe world of metascience but have not yet translated into mainstreamuseoracceptabilityin thewiderresearchecosystem.Wethinkthemainreasonisthatthesemeasuresarenotyet validatedatscaleandacrossdifferentfieldsofresearch;inparticular,wedonʼtknowthat theycoherewithourexpertjudgementsofwhatnovelresearchis Thissortofvalidationis expensiveandtechnicallychallenging,buttheMetascienceNoveltyIndicatorsChallenge istakingitonbecausewebelieveinthepotentialimpactsuchanindicatorcoulddeliver.
The Metascience Novelty Indicators Challenge is now inviting individuals or teams who can generate novelty scores forscholarlypublicationsacrossawiderangeofdisciplines toexpresstheirinterestincompetingasachallenger
The winner of the £300,000 grand prize will be the challenger whose indicator best replicates human ratersʼ expert evaluations of research novelty. In particular, we are looking for scalable indicators that can be applied across a wide range of academic disciplines, and that can measure the novelty of a high volume ofpapersatthetimeof their publication. We are not looking for indicators that measure novelty by lookingata paper'slong-termimpact(ex-postnovelty),suchasitscitationcount.
WhoisInvolved?
The Metascience Novelty Indicators Challenge is delivered through a unique consortium,bringingtogether:
The MetascienceUnitlaunchedthemulti-partnerresearchchallenge,withthesupportof Elsevier. Open Philanthropy will be providing the £300,000 prize to the selected winner throughChallengeWorks.
UKMetascienceUnitUKRI
UK Metascience Unit UKRI is a small team spanning the UKʼs Department for Science, InnovationandTechnologyandUKResearchandInnovation,conductingboldexperiments on funding processes; cultivating a larger and more policy-facing metascience research community; advancing global metascience through large, multi-actor projects; and contributingtoananalyticallyenrichedpublicsciencesystem.
Elsevier
Elsevier is a global leader in advanced information and decisionsupportforscienceand healthcare. Elsevier believes that by working together with the communities it serves, humanprogresscanbeshapedtogofurther,happenfaster,andbenefitall.
OpenPhilanthropy
Open Philanthropy is a grantmaking organisation which aims to use itsresourcestohelp othersasmuchasitcan.OpenPhilanthropyisnotinvolvedintherunningoradministration of this prize, but will be providing the £300,000 prize to the selected winner through ChallengeWorks.
SPRU and RAND Europe have supported the conceptualisation, design, and implementation of the challenge in collaborationwiththeMetascienceUnit,Elsevier,and OpenPhilanthropy.
RANDEurope
RAND Europe is a not-for-profit research organisation that aims to improve policy and decision-making in the public interest through research and analysis. RANDʼs clients include European governments, institutions, non-governmental organisations and firms withaneedforrigorous,independent,multidisciplinaryanalysis TheMetascienceNovelty Indicators Challenge is delivered as part of RAND Europeʼs Research and Innovation Ecosystems portfolio,whichfocusesonsupportingtheresearchcommunity,industryand policymakers in the development of sustainable research and innovation ecosystems worldwide
U y y esearchUnitSPRU
Withalmost60yearsofexperience,SPRUisinternationallyrecognisedasaleadingcentre ofresearchonscience,technology,andinnovationpolicy.Foundedin1966byChristopher Freeman,apioneerofwhatisnowknownasinnovationstudies,SPRUwasoneofthefirst interdisciplinary research centres in the field of science and technology policy and management.Today,withover60facultymembers,SPRUremainsattheforefrontofnew ideas, problem-orientated research, inspiring teaching, and creative, high impact engagementwithdecisionmakersacrossgovernment,businessandcivilsociety.
ChallengeWorksisasocialenterprisefoundedbytheUKʼsinnovationagencyNesta Over the past decade, Challenge Works has facilitated over 100 challenge prizes, engaging more than16,000innovatorsandunlockingover£310millioninfundingtotacklesomeof the world's biggest challenges. Challenge Works believes no challenge is unsolvable, partnering with non-profits, governments, and other organisations around the globe to unearth entrepreneurs and theirinnovationsthatcansolvethegreatestchallengesofour time
te,youneedtosubmit yourEOIviaSubmitta bmit)tobeconsidered for thechallenge.We mittedviaSubmittable. EOIentriesmustbew notwritteninEnglish.
The EOI entry include rs, including contact details, a general out mentation plan, and youracceptanceoftheprize'sTermsandConditions.
EOI entries will be reviewed after the deadline, and entrants will be notified of whether they were successful, and invited to progress to the full challenge competition, in mid-November
Pleasenotethatbybeingacceptedasachallenger,youarenotbeingendorsedbyanyof our challenge partners. Full eligibility and due diligence checks will take place after the finalsubmissiondeadlineon2359GMTon2February2026.
Beforeexpressingyourinterest,pleaseensurethat:
● You have read andunderstoodtheMetascienceNoveltyIndicatorChallenge rules(thishandbook)
The Metascience Novelty Indicators Challenge is a global initiative to develop and validate scalable indicators that can identify novelty in research. We are challenging academics, data scientists, researchers, and innovators worldwide to create tools that could reshape how we evaluate novelty and thereby support scientific progress in the fieldofmetascience.
The Metascience Novelty Indicators Challenge is open toindividualsorteamswhocan generate novelty scores for scholarly outputs at the pointofpublication,acrossawide rangeofdisciplines,inascalable,automatedmanner.
The winning indicator will be selected based on how they best meet the following conditions:
The winner of the Metascience Novelty Indicators Challenge will be awarded to the indicator that can achieve the highest level of accuracy towards the dataset of expert noveltyscores
Thewinningindicatormustalsobe:
● Scalable: The Indicator should have the ability to efficiently and effectively process a high volume of publications across a wide range of academic disciplines.Weexpecttheindicatortobeautomatedandabletooperatewithout highmanualefforts
● Applicable at time of publication: The indicator should have the capability to assessnoveltyofscholarlypublicationsatthetimeofpublication.Weexpectthe indicator tomeasuretheimmediatenoveltyofnewresearchpapers.Wearenot looking for indicators that measure ex-post novelty,bylookingatthelong-term impact of a paper, i.e. indicators based on how many citations a paper will receiveoverseveralyears
TheDatasetofExpertNoveltyScores
The winning novelty indicator will be the one that most accurately reflects expert judgements of novelty Toachievethis,wearecurrentlydevelopingatestenvironmentin the form of a dataset of expert novelty scores for a sampleofrecentacademicpapers across all fields and disciplines.Thedevelopmentofthisdatasetistakingplacebetween SeptemberandDecember2025.Wewillsharefurtherdetailsaboutthemethodologyused to generate this dataset and its characteristics following the completion of aninitialpilot phase.
The challengers' goal will be to create a novelty indicator that generates noveltyratings thatmatchthoseinourdatasetascloselyaspossible.Thedatasetwillbeusedtoevaluate theperformanceoftheindicatorssubmittedbychallengersforthefinalsubmissionofthe challenge.Weoutlineourtimelinebelow.
Disclaimer
Pleasenotethattheexpertnoveltyscoredatasetisbeingdevelopedconcurrentlywiththe Metascience Novelty Indicator Challenge. The selection of the winneriscontingentupon theexpertnoveltyscoresdatasetbeingbuiltsuccessfully.Therefore,thechallengemaybe cancelledifunforeseenissuesarisewhenbuildingthedataset.
FinalSubmission&WinnerSelectionProcess
Challengers will be expected to have developed a fully working indicator by the final submissiondeadlineof2359GMTon2February2026.
The set of novelty ratings for the list of papers submitted by challengers for the final submission will be tested against the expert novelty score dataset. We are currently developing the best approach to delivering this test in a way that matches the characteristicsoftheexpertraterdataset
Challengers will be provided with a detailed description of this method alongside the metadata,whichwillbesharedinDecember2025.
Timeline
The Metascience Novelty Indicator Challenge runs from late September 2025 to April 2026
Individuals or teams who can generate novelty scores for journal publications across a widerangeofdisciplinesmustexpresstheirinterestincompetingasachallengerby2359 GMTon3November2025.
Successful applicants will be informed that they are an official challenger in mid-November,andwillhaveuntil2359GMTon2February2026todevelopandrefine theirindicators.
In December 2025, the metadata used in the expert data set will be shared with challengers.
Challengers will generate novelty scores for each of these papers using their indicator.
By 2359 GMT on 2 February 2026 challengers will submit their indicator results alongside a methodology description. Assessments will be based primarily on accuracy againsttheexpertratings.
ThewinningteamwillbeinformedinMarch2026,andreceivethe£300,000grandprize alongwithpartnershipopportunitiestoadvancetheirnewindicatorandtheopportunityto contribute to a peer-reviewed publication reporting the various indicators used in the challenge.ThechallengeresultswillbepubliclyannouncedinApril2026.
What we are offering
Asachallenger,youwillhavetheopportunityto:
● Contribute to advancing the field of metascience, including research assessmentandthemeasurementofnovelty,
● Prize award: The winner will receive a £300,000 grant to develop or commercialisetheirwinningsolutionfurther
● Optional partnership opportunities to advance your new indicator: The winningteamwillhavetheoptiontoexploredevelopingtheirindicatorfurther across the scientific corpus. This is not mandatory, but the winner will be encouraged to explore product development with Elsevier and access their hugedatasetfortesting
● The opportunity to be featured within an academic paper: Thewinnermay be invited to contribute to a peer-reviewed publication reporting the various indicatorsusedinthechallenge.
Informationabouttheacademicpaper
OneofthekeygoalsoftheUKMetascienceUnitistoadvancethefieldofmetascienceandexpandour knowledge of how to identify and measure novelty, with potential impacts in areas such as research assessment.Therefore,followingthechallenge,wewillpublishanacademicpaperdescribingitsresults, providing the field with crucial knowledge about how well various approaches to measuring novelty performagainstexpertassessments.
● A description of the various challenge entries, both in aggregate and in specific, independent of ranked performance We will describe high-level cohort characteristics such as countries of provenance,typesofindicatorapproachesetc.
● Only high-level methodologies and scores of selected indicators (at the discretion of the challenge partners) willbeshared,whilekeepingfullalgorithmsconfidential Beforepublication,wewillalways takeintoaccountyourconcernsaboutconfidentialityandintellectualpropertyrightsinyourentry
● A transparent and detailed description of the methods we usedtogeneratetheexpertraterdataset andtorankindicators.
By submitting an EOI and yourindicatorenteringthechallenge,youwillbeagreeingtohaveyourentry and its ranking featured on a ‘leaderboardʼ on our website and in a major journal article about the challenge.Yourparticipationandworkcouldalsobehighlightedinpublicandacademicevents.Youare freetopublishseparatelyaboutyourentry
Who can enter
Whocanentertobeachallenger?
TheMetascienceNoveltyIndicatorsChallengeisopentoindividualsandteamswhohave already developed novelty indicators, or who possess the capability to deliver a fully workingscalableindicatorby2359GMTon2February2026.Anyonewithasmartidea andthetechnicalskillsrequiredmayenter,especially:
In addition to these requirements, teams will need to confirm theiraffiliationandthe legal entity acting as lead organisation to submit their final submission by 2359 GMTon2February,andpassduediligencecheckspriortoreceivinganyaward
The lead organisationfromthewinningteamwillconfirmthesignatory(anindividual from the lead organisation with signatory authority) who will enter into the winner's agreement.Theprizeawardwillbesenttotheleadorganisationʼsbankaccount.
Whatisexpectedofchallengers?
Challengers are the individuals and teams who submitted an EOI before the deadline of 2359GMTon3November2025withasuccessfuloutcomeallowingthemtoparticipate inthechallenge
EntrantswillbenotifiedoftheoutcomeoftheirEOIinmid-November.Successfulentrants will be onboarded as challengers, and be added to a mailing list to receive key details throughoutthedurationofthechallenge.
Through submitting an EOI, team leads must confirm that their team will comply with all TermsandConditionsthroughoutthechallenge.
● Rights to develop the indicator: Challengers must own or have permission to usetheintellectualpropertyoftheirsolutionthroughoutthechallengeduration. For example, if you are an academic team entering as a newly incorporated company,youwillneedtoensurethattheuniversityhasgrantedrightstoyouof both their background IP and the foreground IP, for submission through the challenge Thewinningteamwillneedtohaverightstodevelopanddeploytheir indicatorbeyondtheendofthechallenge
● Multiple entries: We accept multiple entries from the same organisation (e.g. the same university), but an individual can only lead one submission An individual couldbeateammemberinmorethanoneteam.Individualsworking across multiple submissions must ensure they manage this arrangement and any potential conflict of interest responsibly through acting lawfully, ethically andingoodfaith.
● Your Intellectual Property: Your models remain yours, and how/what you choose to publish about your methodology and code will be up to you. It is expected that all competing challengers will be open to being featured in a journalarticleaboutthechallenge.
