Rankings, Bibliometrics and their Contribution to Higher Education Management

WORKSHOP Rankings, Bibliometrics and their Contribution to Higher Education Management Rectorate Universidade NOVA de Lisboa June 29th 2012 https://www.unl.pt/nova-2011-2012/rankings-and-bibliometrics

P R O G R A M Rankings, Bibliometrics and their Contribution to Higher Education Management 09.15h Opening Prof. João Crespo, Vice-Rector, UNL

09.30h Multidimensional rankings. The U-Multirank project Dr. Gero Federkeil, CHE Centre for Higher Education, Germanyl

10.00h Measuring UNL research: the use and interpretation of bibliometric indicators Dr. Martijn Visser, Center for Science and Technology Studies, Leiden University, The Netherlands

10.30h The scientific production of UNL: evolvement and prospects Prof. João Crespo, Vice-Rector, UNL

11.00h Bibliometric analysis in Social Sciences & Humanities: paradigm shift and mutual challenges Prof. João Costa, Vice-Dean, Faculdade de Ciências Sociais e Humanas, UNL

11.30h Bibliometric analysis and its contribution for university governance Prof. Carlos Costa, former Dean, Faculdade de Engenharia, Universidade do Porto

12.00h Open Discussion 13.00h Closing Prof. António Rendas, Rector, UNL

B i ographical notes of speakers Gero Federkeil is Project Manager at the Centre for Higher Education (CHE), Guetersloh, Germany, and Vice-President of IREG Observatory on Academic Rankings and Excellence since October 2009. At CHE he is responsible for international ranking activities, including the U-Multirank project funded by the European Union. He is an internationally recognized expert in the field of rankings and his publications focus on rankings, performance indicators, benchmarking, quality assurance and employability/university – labour market relations. He is a member of the German Association of Evaluation and a team member of the CHERPA Network. Before joining the CHE in 2000 he worked 7 years for the German Council of Science and Humanities in the field of higher education policy, investments in higher education, evaluation and university medicine. He holds a Master Degree in Sociology (1989) from Bielefeld University.

Martijn Visser holds a MA in History from Leiden University. Since 1996 he is affiliated with the Centre of Science and Technology Studies (CWTS) at Leiden University. His main activities at CWTS concerned managing, research assessment and benchmarking studies. In conjunction with that he has been responsible for developing and maintaining the data infrastructure and software routines to facilitate these studies. In the past ten years, he (co-) authored around 20 publications mainly on the development and application of indicators for the measurement research performance. Recently, he changed his role within CWTS and embarked on a research project to study the limitations and potentialities of new data sources for bibliometric studies.

João Costa is a Professor of Linguistics at Faculdade de Ciências Sociais e Humanas – UNL, where he currently is Vice-Dean for Research and Human Resources. He received his PhD degree in Linguistics from Leiden University in 1998, and his undergraduate degree from the University of Lisbon in 1994. He was a visiting scholar at MIT, and taught as a visiting professor in universities in Brazil, Macau, The Netherlands, and Spain. His research focuses on syntax and language acquisition in typical and atypical development. He has authored 7 books and around 100 papers (in journals, book chapters and proceedings), integrates the editorial board of 11 journals and book series and he is currently involved in 5 research projects and 3 international networks on syntax, language acquisition and parsing. He coordinated projects related to curriculum and teacher training for the Portuguese Ministry of Education.

Carlos Costa retired in 2010 as Professor of Chemical Engineering at Faculdade de Engenharia da Universidade do Porto (FEUP) and assumed the Presidency of Tâmega Entrepreneurship Institute and become non-executive administrator of RIMA, S.A. In 2011 he also assumed the direction of TER (Association for Innovation and Development of Renewable Energies). During 9 years (2001-2010) he was FEUP director and member of the General Councils of the UPorto Business School and of the Gomes Teixeira Foundation. Before (1995–2001) he was director of FEUP Chemical Engineering Department and founder and coordinator of LEPAE (Laboratory for Process, Environment and Energy Engineering) at FEUP. In all these management activities he always integrated the use of quantitative methods and feedback control to help better decision making and performance.

F O R E W O R D The goal of this workshop is to promote the discussion about the role of bibliometrics and university rankings in higher education management and university governance. Universities are increasingly confronted by ranking and classification initiatives – both at the national and international level. Despite all objections about the methodologies and criteria of rankings, there is a growing recognition that such initiatives are here to stay. A particular focus will be given to U-Multirank, the EU sponsored project for the development of a ranking that considers not only research and bibliometrics but also other dimensions of higher education institutions, from teaching/learning to international profile and knowledge transfer.

This workshop will also present and discuss the results of the third bibliometric study produced for NOVA by the Centre for Science and Technology Studies of Leiden University. This study analyses the scientific production of NOVA in different areas and subfields and compares the impact and output of NOVA with reference universities.

This study shows that the normalized impact of publications of NOVA, indexed to Web of Science, has improved consistently since 2000 and reached a value similar to the international average. The number of highly-cited papers has also increased significantly. This study identifies scientific areas with significant production and impact close to, or above, international average such as Biological Sciences - Animals & Plants, Chemistry, Geosciences, Applied Physics & Chemistry, Clinical Medicine, Biological Sciences – Humans, and Engineering.

As far as scientific subfields are concerned, the impact of NOVA is above the international average in Applied Physics, Materials Science Ceramics, Food Science & Technology, Plant Sciences, Physical Chemistry, Electrical & Electronic Engineering and Multidisciplinar Chemistry.

A group of 35 universities (from Portugal or other European countries) was selected for a benchmarking study. The impact of NOVA is significantly above the impact of Portuguese universities in the following main scientific areas: Biological Sciences Animals & Plants, Chemistry, Economics, Psychology, Psychiatry & Behavioral Sci and Humanities & Arts. Concerning subfields, the impact of NOVA is higher than that of the 35 benchmark universities in Applied Physics, Materials Science Ceramics, Chemical Engineering and Electrical & Electronic Engineering.

This event is designed to be an informal workshop, open not only to NOVA but also to the national scientific community at large, aiming at discussing the role of rankings (bibliometric and multidimensional) in the management of universities. 13.00h

Update of the bibliometric study of NOVA (publications indexed to Web of Science 2004-2010)

RESULTS

Benchmark

NOVA with other Portuguese and European Universities

Identify areas of excellence, strengths, and weaknesses

Assess the impact of the publications (articles, letters, reviews) in journals indexed to WoS and compare with the results of the previous studies (2000-06 & 2002-08)

Main objectives:

Solid conclusions can be withdrawn for the majority of the Natural and Exact Sciences, where the coverage of Web of Science is excellent and the publication in periodicals of international circulation is the most common way of dissemination of results

Study carried out by the Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands

Bibliometric analysis

3350 3995 (+19%)* 4671 (+39%)*

2000-2006

2002-2008

2004-2010

0.98 (+13%)*

0.95 (+9%)*

0.87

Impact CPP/FCSm

1.02 (+7%)*

1.0 (+5%)*

0.95

Impact MNCS

If CPP/FCSm or MNCS values are between 0.8-1.2 the impact is similar to the international average. A value >1.2 means that the impact is above average

MNCS, Mean Normalized Citation Score – New indicator that replaced CPP/FCSm in 2010.

It is also a normalized indicator, but the calculation method is different

citations excluding self-citations (CPP) obtained by a research unit to the international reference value (FCSm, the mean field(s) citation score). It is a normalized indicator that corrects for differences in “citation habits” among scientific areas

CPP/FCSm, Field Normalized Citation Impact - this indicator compares the average number of

* percentages are relative to 2000-2006

Production

Publication period

Production and normalized impact

0.66

0.72

0.75

0.82

0.82 0.80

0.88 0.87

0.89

1.00

1.03

1.05

The impact of NOVA publications has been increasing and reached 1.05 in 2007-2010

0

0.2

0.4

0.6

0.8

1

1.2

Evolution of Impact

Normalized impact (CPP/FCSm)

0

100

200

300

400

500

600

154

Top20%

559

57

Top10%

28 Top5%

133

3 Top1%

35

2004-2008**

2002-2005

2001-2004

2000-2003

1999-2002

1998-2001

1997-2000

** 2004-2008 papers counted in 5 years; all other data corresponds to 4 years

283

Highly-cited papers (2004-2008)

Collaboration

Arrows identify areas with significant output (>200) and impact close / above international average: Biological Sciences-Animals & Plants (1.51), Chemistry (1.17), Geosciences (1.07), Applied Physics & Chemistry (1.03), Clinical Medicine (0.99), Biological Sciences-Humans (0.97), Engineering (0.94)

Scientific areas of highest impact and production

Subfields of highest impact and production

“ Chemical Engineering”, “Microbiology” and several areas of “Chemistry” have both high impact and significant output

The impact is above international average (>1.2) for 8 subfields: Applied Physics; Material Sci Ceramics; Chemical Eng; Food Sci & Technol; Plant Sci; Physical Chemistry; Elect & Electr Eng; Multidisc Chem

Approximately 50% of publications result from international collaboration

The number of highly cited papers has been increasing consistently

The field normalized citation impact of NOVA, CPP/FCSm, increased 13% from 0.87 in 2000-2006 to 0.98 in 2004-2010. The Mean Normalized Citation Score, MNCS, increased from 0.95 to 1.02

The number of publications increased 39% from 3350 in 2000-2006 to 4671 in 2004-2010

Conclusions

Benchmarking

Austria Technical Univ Wien

Germany Stuttgart Technical Univ Dresden

Belgium Katholieque Univ Leuven Libre Bruxelles Switzerland ETH Zurich

Portugal Porto Lisboa Técnica de Lisboa Coimbra Aveiro Minho Algarve

Ireland Dublin City Univ United kingdom Bath Southampton Leeds

Italy Perugia Poland Krakov

France Toulouse 3 Paul Sabatier Paris 5 René Descartes

Spain Granada Sevilha Autónoma Barcelona

Greece Aristotle Univ Thessaloniki

Benchmark universities

Czech Republic Masaryk Univ Brno

Norway Bergen Norwegian Univ of Sci & Technol Trondheim

The Netherlands Utrecht Twente

Denmark Aarhus Univ of Southern Denmark

Finland Helsinki Turku Abo Akademi Univ

NOVA vs 35 benchmark universities: subfields with highest impact at NOVA

* Differences in impact higher than 0.15

Applied Physics Mat Sci, Ceramics Chemical Eng Elec & Electr Eng Physical Chemistry

The impact of UNL is significantly above* the benchmark universities in:

* Differences in impact higher than 0.15

Applied Physics Applied Physics Mat Sci, Ceramics Chemical Eng Elec & Electr Eng Plant Sciences Physical Chemistry Multidisc Chemistry Parasitology Microbiology Organic Chemistry Applied Mathematics

The impact of UNL is significantly above* the impact of Portugal in:

NOVA vs 35 benchmark universities: subfields with highest impact at NOVA

universities that produce more than 700 publications per year indexed to Web of Science

Position of NOVA among 365 top world universities

Ranking

↑ ↑ ↑

Engineering - 265

Biol Sci Humans - 269

↑ ↓

Multidisciplinary Sci - 259 ↑

Applied Physics & Chemistry - 249

Social Sci Medicine - 242

Other Social Sci - 235

Geosciences - 226

Chemistry - 221

↑

Economics - Ranked 189

World Top176-270 in 9 areas:

↑

World Top126-150 Humanities & Arts - Ranked 140

World Top51-75 Biol Sci Animals & Plants - Ranked 66

Rank in 2004-2010 and change in position compared to 2000-2006

Ranking based on impact (2004-2010) and comparison with 2000-2006