Online Learning Lab Report 2015

ONLINE LEARNING LAB REPORT 2015 Innovating the way we teach and learn

Innovating the way we teach and learn

TABLE OF CONTENTS Executive summary 6 Foreword 9 Introduction 11 The Year In Numbers 12 Exploring Online Education 17 Design & Delivery: Learning Experiences Iteratively improving course features Exploring storytelling, pedagogy and new media Storytelling & Pedagogy: Innovation & pedagogy Flexibility & production Taking a course whenever you want; the rise of Netflix University Personal and Glocal: Small Private Online Courses (SPOCS) Specializations: What’s next?

Activate and Empower: Learners as community builders Sustainable MOOC development Crowdfunding open content

17 17 18 18 18 20 20 22 23 23 25 25

Democratization through education

27

Impact on on-campus education

41

MOOCs as a Platform for research

49

Demographics Areas of employment Education Course quality and satisfaction rates Do our learners return for more?

Performance by age

Where the digital and physical classrooms meet to form the next experience in higher education Designing the classroom of the future Lessons learned from Flipping the Classroom How Leiden University students use our MOOCs Facilitating research through tooling and analytics Data management policy Github tools Data as a service Online learning research Information density and understanding Effective learning strategies: retrieval practice Peer reviews: online performance and collaboration

27 29 31 32 37 40 41 41 44 45 49 49 50 50 50 51 52 52

Collaborations 53 Next steps 54 Exploring new technologies Recognition of MOOCs and online learning towards credit Piloting online proctoring Designing for deep learning Adopting an online mindset for our physical spaces Experiment with alternative monetization structures Stimulate sustainable development of course content Exploring the online master space

54 54 55 55 55 55 56 56

6 - Innovating the way we teach and learn

7 - Innovating the way we teach and learn

EXECUTIVE SUMMARY This report presents the result of the Online Learning Lab providing a quantitative and qualitative analysis of the Leiden open and online initiatives from October 2014 to December 2015. So far: • Leiden Massive Open Online Courses (MOOCs) reached almost 200K enrollments from 196 countries (EU 37%, Asia 17%, Africa 5%) or approximately 10.000 monthly users.

• More than 100K participants enrolled in the course Terrorism & Counterterrorism: Comparing Theory and Practice.

• Over 52K participants enrolled in Miracles of Human Language; an Introduction to Linguistics, making it the single MOOC run with most enrollments in 2015.

• More than 11.000 exams were submitted and over 12.000 Statements of Accomplishment were rewarded.

• More than one million videos were watched, over 30.000 posts and more than 15.000 comments were made on the Leiden MOOC social media forums in 2015.

• Over 90% of the learners had a ‘good’ to ‘excellent’ experience in the MOOCs. • More than 358 Video Lectures were produced for MOOC content (over 50 hours of educational content) and 71 video’s for other projects, recorded at 15 locations on campus and beyond.

• Over 400 volunteers translated subtitles for the MOOC Miracles of Human Language into 26 languages.

• 8,5% of active MOOC learners is active in the Facebook groups established around the courses.

• Started with Small Private Online Course (SPOC) design that combines a virtual classroom with a real classroom.

• For the first time a course successfully crowdfunded an additional module.

General Findings: • MOOCs beyond the hype: MOOCs seem to have outlived the hype and there is still a lot to experiment and learn from this type of online education.

• MOOCs and SPOCs help to innovate on-campus education: Your own students are ready to use new learning formats. Online learning spaces do not make students less willing to learn if combined with intensive and rich on-campus learning experiences. For example, the use of a good mix of storytelling and pedagogy ensures that such materials are engaging and leverage their maximum potential in terms of educational experience and outcomes.

• The rise of 21st century skills: Students increasingly expect academic education and knowledge gathering to go hand in hand with (new) skills such as planning, finance, coding, data visualization and so on.

• Faculty engagement is key: To catalyse the conversation regarding online learning, the development of faculty wide Online Learning Strategy Plans is useful.

• Growth of requests for MOOCs and SPOCs: MOOCs and SPOCs have been included as key strategic points in relation to on-campus education innovation, and there is a significant rise in the number of requests to use online learning facilities of the Online Learning Lab. • The impact of technological developments: Research, education and working Technological advances (be it digital or material innovations) increasingly shape the physical environment in which students learn, study and work; the university building of the future will look different from the ones we have today.

• Recruitment and brand awareness through MOOCs: The global and massive reaches of MOOCs have recruitment potential. MOOCs and SPOCs are seen by several Faculties as a way to reach new student groups.

• Networking potential through MOOCs: MOOCs offer possibilities to build a network of professionals around the world.

• Importance of flexibility due to pace of technological change: As technologies become ever more readily accessible due to prices dropping, a limited budget can have a big impact in online learning.

• Online Learning Research calls for an interdisciplinary approach: There is a growing need to cross disciplines due to increasing complexities in scientific development.

There is a need to create new institutional incentives to bring students, teachers and researchers from different backgrounds closer together, and to connect to broader networks outside the university (inside out and outside in).

Recommendations: • Explore the opportunity space between 100% residential and full online MOOCs. Be clear about what the residential student experience is. Be explicit and intentional about what students and faculty get when they engage in class.

• Develop a coherent vision on what an academic working and learning environment should look like in the future and use broad experimentation to test different approaches.

• Dare to question the origin of the curriculum structure. Don’t keep doing what you do because you’re comfortable doing it. E.g. experiment with short cohort settings, mix of tools and delivery modes, and teacher-mentor coaches. Focus on the residential experience you provide students with.

• Stimulate Social Learning in course design.

Students learn more (and retain knowledge longer) from interaction and discussion with each other and the teacher than simply listening to a lecture, watching a

•

• •

•

•

• •

video or reading a text. Social contact in a community, online as well as face to face, is central to the academic experience.

Design for change in learning spaces. Develop a comprehensive approach for classroom and learning space design and furniture.

Find a way to leverage the potential of MOOCs to attract new students from previously hard-to-reach areas.

Stimulate the use of open online materials.

Understand how students and alumni use our own and other institutes’ online materials.

Start innovation and pedagogy experimentation.

Explore cutting-edge techniques while harbouring sound educational principles. E.g. the impact and use of virtual reality and 360 video as new media tools in education.

Invest in flexible equipment and small studio environments and high quality staff. Since new technologies are changing the online learning landscape quickly and new inventions make previous technologies obsolete, ensure that quickly changing the composition of your team and the tools on which they rely are possible throughout the entire year.

Explore new, alternative funding approaches for online education e.g. crowdfunding institutional partnerships.

Invest in online communities. This includes enlisting learners to update/upgrade your materials. Learners are proud of ‘their’ MOOCs. Give them ownership.

“ Even with my english, that isn’t good enough, this course showed me a new world, a really interesting world, an invisible world that only a few people can see how it works. From now on, always that i look a big corporation on TV, Internet or when i buy some product or service, I will remember what I learned with Leiden University.” Student in the MOOC Rethinking International Tax Law .

8 - Innovating the way we teach and learn

9 - Innovating the way we teach and learn

FOREWORD Leiden University has always strived to nurture a coherent vision of how new ideas can quickly enter the university, if they are relevant, and to either find ways to scale up, or to abandon such ideas and initiatives in a timely manner. As research confirms “ such space is necessary […] and [for] the definition and management of dynamic capabilities to renew competences in line with changing environments”.1

by Vice Rector Magnificus prof. dr. Simone Buitendijk We are in the midst of a technological revolution, which is significantly impacting the way we live, learn and work. Universities need to determine in what way they want to adapt to these new technologies and how they wish to use them in teaching and training the new generation for the job market and the world of tomorrow. Present day students will encounter a more globalized and interconnected society than previous generations did. These developments are shaping a new type of student, who more and more needs to be a ‘digital native’, and who will learn differently from students 10 years ago. Many present day students need to be prepared for jobs that do not yet exist.

The mission of the research-intensive university is to drive innovation, help respond to major national and global problems, and provide the narratives that make it possible to understand a rapidly changing and increasingly volatile world.

We started in 2012 on a journey to explore how we could offer our knowledge to a larger, global audience and to democratize education through technology, with our partner Coursera. We remain committed to this mission to open-up education and TO increase access globally. At the same time we are welcoming requests from staff and teachers within our university to change the way we work and to rethink our campus education: what does online & open education mean for (the future of) the university and how can it enhance the learning experience of students?

The report in front of you looks back at 2015 and reflects on the various innovation experiments we conducted. We have tried to formulate a number of recommendations for ourselves as a research and teaching organisation, but we hope they will inspire and be of benefit to a larger community of highereducation innovators.

1 Teece, D. J., Pisano, G., & Shuen, A. (1997). Dynamic capabilities and strategic management. Strategic management journal, 18(7), 509-533, Navarro, J. R., & Gallardo, F. O. (2003). A model of strategic change: Universities and dynamic capabilities. Higher education policy, 16(2), 199-212.

10 - Innovating the way we teach and learn

11 - Innovating the way we teach and learn

INTRODUCTION “We know what we are, but not what we might be” William Shakespeare Through the Online Learning Lab (OLL), Leiden University primarily aims to explore new forms of education, pedagogy and novel target groups. By providing a truly global audience access to our education, we hope to make a contribution to the democratisation of education. Through the experience and insight of these explorations, we aim to leverage innovation and to strengthen our on-campus education. This impact on campus education is fortified by research for which MOOCs provide an excellent platform.

Last but not least, our frontrunner position in online learning helps establish and reinforce (inter)national collaborations. In the following chapters we will evaluate the progress on these four aims.

After launching the first MOOCs and SPOCs in 2013-2014, we introduced several innovations in the delivery of MOOCs. We enhanced our working methods, established of a data-driven research team, and a growing interest in MOOCs and SPOCs from the part of the faculty which lead to over 25 newly approved MOOC, SPOC and Flipping the Classroom projects. These projects are currently in different stages of development. One of the driving forces which impacted online learning within the university were the new Online Learning Strategy plans written by the Leiden University Faculties, for which we provided advice. From online masters and pre-masters to active learning in a Flipped Classroom, new flexible learning spaces, academic skills education

and blended learning for study success are put into each faculty’s context and are prioritized.

At the same time, the Centre for Innovation’s Online Learning Lab maintained the role of frontrunner, always in co-creation with other pioneers, to explore trends that Faculties might be interested in tomorrow.

This way, we hope to fulfill our role to help Leiden University prepare for the future of education. To stay ahead of the curve we decided to become an innovation partner of Coursera which enables us to test new ideas quicker and push the boundaries of online learning a bit further.

The future of online and open learning is just beginning, and by exploring and harnessing the possibilities presented by rapidly developing information technologies, we aim to be at the forefront of teaching and learning innovation. It is not yet possible to say what MOOCs could/should look like in the coming 3 to 5 years and how online learning will be embedded in our institutions; therefore investment in new initiatives and scalability of working processes is needed in the coming years.

The report in front of you will provide facts and figures around the various innovation projects we ran last year with MOOCs. Marja Verstelle & Gideon Shimshon, Co-founders of Leiden University’s Online Learning Lab @ The Centre for Innovation

12 - Innovating the way we teach and learn

13 - Innovating the way we teach and learn

THE YEAR IN NUMBERS Table 1: Innovation production figures at a glance

MOOCs delivered

MOOCs Rerun

• Miracles of Human Language; an Introduction to Linguistics by prof. dr. M. van Oostendorp.

• Federalism & Decentralization; Evaluating Africa’s track record by dr. J.G. Erk • Rethinking International Tax Law by prof. mr. S. Douma.

• The Changing Global order (2 reruns in 2015) by prof. dr. M. Hosli; • Configuring the World (2 reruns in 2015) by prof. dr. R. Griffiths • Global affairs Capstone Project (2 reruns in 2015) by prof. dr. R. Griffiths and Prof. Dr. M. Hosli • Miracles of Human Language (1 rerun in 2015) by prof. dr. M. van Oostendorp.

MOOCs available On-Demand

MOOCs in development in 2015

• Terrorism & Counterterrorism: Comparing Theory and Practice by prof. dr. E. Bakker.

• Federalism & Decentralization; by dr. J.G. Erk • Rethinking International Tax Law; by prof. mr. S. Douma.

• International Law in Action: a guide to the International Courts & tribunals in the Hague (Jan 2016) by prof. dr. L. van den Herik et al.

• The Rooseveltian Century (Jan 2016) by prof. dr. G. Scott-Smith et al.

• Clinical Kidney Transplantation (Jan 2016)

by Dr. M. Reinders et al.

• Be Persuasive: write a Convincing Position Paper or Policy Advice.

Project centred course (Feb 2016) by prof. dr. R. Griffiths.

• Anatomy of the abdomen & Pelvis (April 2016) By prof. dr. M. De Ruiter et al.

• International Law in Action: Investigating and prosecuting International Crimes (Spring 2016) by prof. dr. C. Stahn.

• On Being a Scientist (Summer 2016) by prof. dr. B. Haring et al.

• Heritage under Threat (Summer 2016) by dr. S. Mire et al.

• Evolution Today (Fall 2016) by prof. dr. M. Schilthuizen et al.

SPOCs delivered

Flipping the Classes delivered

• Configuring the World; a Global Adventure, by prof. dr. R. Griffiths.

• Terrorism & Counterterrorism (2 projects) prof. dr. E. Bakker.

• Federalism & Decentralization dr. J.G. Erk • Rethinking International Tax Law, by prof. mr. S. Douma.

Special audio visual projects

Guides and workshops

• Documentary series of 4 Lectures about the History of Leiden University (for the 440th anniversary of Leiden University); • Production of 30 high quality online video lectures ‘On Minorities’ with the Faculty of Humanities; • 3-D rotation model ‘Embryonic Intestines’ for Anatomy MOOC; • A customized video design and trailers for 7 new MOOCs; • Special recording Vice-President of the European Commission Federica Mogherini in collaboration with Campus The Hague, used in the MOOC ‘The Changing Global Order'.

• Extending studio and faculty services: greenscreen, new media formats, cameras and equipment

Guides • How to write & create a knowledge clip?

• Innovation of physical learning spaces Workshops: • How to develop a SPOC workshop (Conference on Innovation in Teaching) • Screencast recording (Faculty of Science) • Communication management Training (TU Delft and E-merge) • Development of knowledge clips at LU (Conference on Innovation in Teaching)

14 - Innovating the way we teach and learn

15 - Innovating the way we teach and learn

Session-based 333.655 Enrollers

Total 164.716 Learners 24.557 15.689 3.263 6.660 1.616

Exam Takers Certificates Honours signature track Financial aid

423.586 197.389 17.660 8.320

On-demand 89.931 Enrollers

32.673 Learners 1.971 Certificates 1.660 Signature track

Figure 1: Overview of statistics for on-demand and session-based MOOCs. Data is calculated cumulatively over the years 2013-2015.

Enrollers Learners Certificates Signature track

16 - Innovating the way we teach and learn

17 - Innovating the way we teach and learn

EXPLORING ONLINE EDUCATION

Creating engaging and impactful online education requires an increasingly interdisciplinary and iterative approach. By bringing together domain experts, pedagogy expertise, video/media specialists, coaching capabilities, and analytics into the course design the development process becomes highly experimental.

Bringing together all this knowhow to deliver a course that is efficient in production while at the same time allows for exploration and innovation required a process which was flexible yet timeboxed.

Design / Innovate Leiden MOOC Learners per 100.000 Internet Users

1

192

- Course design • Pedagogy • Community • Video - Student experience

Figure 2: Sign-up rates, per country and completion rates, per region. Based on data of roughly 170.000 learners from 10 session-based MOOCs

Develop - Video scripts - Assignments - Video production - Course environment - Tests

Figure 3: Overview delivery process Online Learning Lab

Design and Delivery: Learning Experiences Based on the work of G. Salmon (University of Western Australia), who developed an education development method called Carpe Diem, we adapted the method as part of the Innovate step in the delivery process (figure 3). In contrast to previous development methods, Carpe Diem involves organizing an intensive 2-day workshop with domain experts and MOOC teams to kick-start the course design process.

This speeds up the completion time for course development in the Design phase and delivers a more coherent approach to content, assignments, assessment and video. It also fosters trust between the various team members, which results in a strong commitment to get the project done together.

Deliver - Course launch - Interaction and feedback

Evaluate - Course evaluation - Define improvements - Provide research data

18 - Innovating the way we teach and learn

Exploring Storytelling, Pedagogy and New Media How can we translate learning materials into an online setting? What tools do we use when teaching online and how can we keep students engaged in a setting so different than our regular classrooms?

These are some of the questions the Online Learning Lab seeks to answer for educators who want to develop online learning materials. The following elements are leading in our approach to video didactics: • Storytelling and pedagogy ensure that such materials are engaging and leverage their maximum potential in terms of educational experience.

• Innovation and pedagogy experimentation with cutting-edge techniques while harbouring sound educational principles.

• Flexibility and production ensures that we incorporate a variety of formats suitable to the course content.

Storytelling and Pedagogy: Video and interactive online media are a key aspect of online learning environments. Similar to on-campus education, online teaching materials take well-established pedagogical models as their point of departure. However, many of the materials used in online education rely on new media (as opposed to physical classroom interaction), and therefore require additional techniques to achieve the desired level of engagement. One technique we apply is storytelling.

The essence of great storytelling is well known in (news) media, (web) documentary production and filmmaking, and known as one of the oldest forms of communication1.

1 Bratitsis, T., Barroca, A., Fruhmann, P., & Broer, Y. (2014).

European Educators ’Training Needs For Applying Digital Storytelling In Teaching Practice, 194–204.

19 - Innovating the way we teach and learn

We apply similar techniques by working with educators to remodel learning materials and help them to engage with students by posing intriguing dilemmas and questions. Therefore, the focus becomes showing instead of simply telling. The first results of using storytelling are promising and we are looking to intensify the use of these techniques in 2016.

Innovation and Pedagogy The convergence of technology in devices means that these new forms are readily accessible and that the boundaries between types of media begins to fade2.

For each production, we require the implementation of a specific innovation which can be tested against pedagogical insights. Such innovations can focus on incorporating specific technologies, one example of which is the 3D animations created for the MOOC Anatomy of the Abdomen and Pelvis. They can also focus on creating more engaging video’s, such as our upcoming experiments using a 360 degree camera.

To us, the process of innovation in online learning materials is enabled by a collaboration between experienced documentary makers who look at video formats and the emotional connection between the viewer and the subject, as well as pedagogical expertise geared towards learning outcomes and formal learning models. Using new media in online education is relatively novel, but virtual reality (VR) and augmented reality (AR) are widely recognised to have a promising future in higher education.3

2 Jenkins, H. (2004). The Cultural Logic of Media Convergence. International Journal of Cultural Studies, 7(1), 33–43. doi:10.1177/1367877904040603 3 Exploring new media for education is done in collaboration with Remco van Schadewijk from the Leiden Science Faculty

Video and Pedagogy To celebrate Leiden University’s 440th birth year Professor Willem Otterspeer, a historian at Leiden University developed four knowledge clips.

Each clip in this web series highlights the communicative power of storytelling in educational content by taking viewers on a journey through four centuries of freedom in Leiden.

Using engaging video: which format is right for you?

In a MOOC, instructors translate their scientific knowledge into videos, animations and other new media using techniques from documentary production, storytelling. The images below show several of these techniques as they are used in our MOOCs. Lecture video - ‘Talking Head’

Use of informants

On-site locations

3D animation

20 - Innovating the way we teach and learn

Flexibility and Production The interaction between instructors and students is essential for learning outcomes. Therefore, instead of taking a ‘one size fits all’ approach, each MOOC is redesigned to fit the instructor’s needs. In the case of online learning materials, this translates into a flexible studio setting and a production team that is active both in The Hague and Leiden. In terms of design this means that every MOOC has a different look and feel. Additionally the MOOC-videos are shot on a wide variety of locations. In 2015 the video team shot lecture video’s on 15 locations in and around the University. From the operating room for the course Clinical Kidney Transplantation to the Linguistics lab for the course Miracles of Human Language.

With a relatively small team of one director and one editor, and one flexible staff if needed, the media team produced 358 Video Lectures and over 50 hours of educational content. It further produced another 71 videos for additional video projects. The formats range from discussion videos, interviews, animations and studio recordings. As the number of requests to use online learning facilities are rising, there is a continuous search for optimisation of the production output. The flexible studio format used at the Online Learning Lab, allows for this by making sure that innovative add-ons to the workflow can be added the year round.

21 - Innovating the way we teach and learn

Taking a Course Whenever You Want: the Rise of Netflix University Traditionally, MOOCs were offered in sessions of five to eight weeks. These sessions would run whenever the instructor could make the time for them, which led to a mismatch in supply and demand. In 2015, Coursera launched its “on-demand platform”, allowing students to begin a course whenever it suits them. While this feature is accommodating to the schedule of learners, it introduces the issue that without deadlines, one might never actually start the course.

Coursera further introduced “cohorts”, which provides learners with deadlines and therefore reintroduces a part of the session-based system.

As such, this approach attempts to combine the best of both platforms by allowing flexibility and providing incentives to work on the course materials.

The results of the new system are mixed. Below a short analysis comparing the ‘session based’ with the ‘cohort based’ approach.

Terrorism session-based (session 3 & 4, each 5 weeks)

Terrorism on-demand with cohorts (from 30-06-2015 to 31-12-2015)

% change

Sign-ups

30.991

33.290

+7.4%

Course completers (of total sign-ups)

6.5%

3.6%

-44.6%

Payments (of total sign-ups)

2.5%

2.7%

+8%

Hours spent OLL staff (over entire period)

36

27

-25%

Hours spent academic staff (over entire period)

30

27

-10%

Engagement Professor

Weekly

Stand-by basis

-

Table 2: Comparison indicators for the on-demand and session-based courses of Terrorism & Counterterrorism.

We compared six months of the continuous ondemand session with the results of two session-based courses for the MOOC Terrorism and Counterterrorism. These are our main observations: • Overall, the number of course completers dropped.

However, the number of payments1 went up. As a percentage of total course completers, payments represent roughly 39% in the session-based platform and 75% in the on-demand platform.

• As the on-demand platform improved, both retention and number of certificates increased. The number of paid certificates increased faster than did the non-paid certificates.

• The social interaction in the experience also saw a drop with less students posting on the forum.

1 With ‘payments’, we mean payments that count towards a verified certificate

These results call for a rethinking of our course designs on the one hand and the need to keep experimenting with platform features from the Coursera side and explore alternative monetization strategies without hindering the learning process.

In 2015, we joined the Innovation Partners group of Coursera. This enables us to test out and deliver feedback on additional functionality and to be at the forefront of MOOC platform developments. So far, Leiden University opted-in for most didactical tests (6 out of 7), and declined the monetization related tests (2), as they do not fully align with our vision on open education. To us, offering learners the possibility to expand their knowledge is the most important feature of open education. Our efforts for 2016 will continue on this path.

22 - Innovating the way we teach and learn

Personal and Glocal: Small Private Online Courses (SPOCS) Unlike MOOCs, which cater to massive audiences, SPOCs are geared towards smaller, private groups ranging from 20 to 250 and sometimes 500 students.

SPOCs feature intensive moderation by teachers, and are fully online courses. Earlier SPOC experiment gave us insight in the development and support processes required to scale up. In September, we opened a call for SPOCs, which resulted in a selection of nine new, very diverse SPOCs to be developed and evaluated in 2016.

The SPOCs that we are developing are targeted at several groups: • Regular students can use them as an online and flexible course to practice, for example, academic skills.

• Students from several universities can follow SPOCs together as a means to foster cross-university community building.

• Professionals can follow SPOCs to deepen and widen their skill set.

• Pre-master students can follow SPOCs to prepare for their on-campus education, for example when they live abroad.

23 - Innovating the way we teach and learn

Learning From Experimental SPOCs In the summer of 2015, Professor Dr. Richard T. Griffiths organized a Small Private Online Course (SPOC) which was based on his successful MOOC “Configuring the world: a critical political economy approach”. Students of Dr. Griffiths’ summer school in China participated in the SPOC together with pre-university students. While the Chinese students also received on-campus classes by Dr. Griffiths, the other students did not. One of the goals of the course was to develop a lively community between students of different backgrounds residing in different geographic areas and taking different versions of the same course.

The experiment did not have the desired results.

SPOCs need their own approach to community management that pays more attention to individual learners than one would in a MOOC, just like material might need an adjustment for this specific format. Induction and e-tivities embedded into the course design are important to lower the barrier to interaction, as is the constant attention of the moderator, which in a SPOC is the instructor. In addition thought needs to go into the motivation of students to go online. The Chinese students not only felt shy about their English, but had live access to the professor during the day which did not motivate them to go online after hours to interact with the virtual group of pre-university students.

There simply was no need for them to do so.

This experiment was helpful in preparing Online Learning Lab develop a strategy for community management and e-moderation for the 9 SPOC experiments lined up for 2016.

In 2015 Coursera closed down its private course functionality we used in some former SPOC experiments, and we started a selection process for a new SPOC platform, which we evaluated along the lines of the following criteria: • • • • •

User-friendliness Mobile accessibility Integrated web conferencing Easy integration of 3rd party tools like TurnitIn A paywall and easy access for non-Leiden students

Based on these criteria we selected an experimental platform as a SPOC platform for 2016. This platform offers us the opportunity to explore online education for private groups. In the last chapter of this yearly report we will look ahead at those developments that we aim to explore in 2016.

Specializations: What’s Next?

In collaboration with Geneva University, we composed the Specialization Global Affairs consisting of three MOOCs and a capstone project. While the average MOOC of four weeks requires a study load comparable to 1 ECTS, a Specialization compares more to a semester long 5-10 ECTS course.

After finishing the course, learners have considerable in-depth knowledge of a topic. We joined the first launch of Specializations by Coursera, which was also the first Specializations in the field of social sciences.

The experiences are positive. We continue to look for topics to develop as a Specialization from scratch, to offer our learners an in depth learning experience.

Activate and Empower: Learners as Community Builders Online education should be available and open to everybody. One way to achieve this is to lower the barrier to entry by offering a diverse range in video captions. We have found that when engaged properly online learners are happy to give back to the community of students by performing a variety of tasks. Our most successful project involving volunteer translators resulted in the translation of all captions in the MOOC Miracles of Human Language: an Introduction to Linguistics to 26 languages. To date, this effort made it the most widely translated MOOC on Coursera.

24 - Innovating the way we teach and learn

25 - Innovating the way we teach and learn

“Testing online interaction” One of Leiden’s popular online courses is given

by Marc van Oostendorp and called ‘Miracles of Human Language’. In an ideal situation Marc van Oostendorp teaches students in small groups, making the MOOC a test on how to bring this personal interaction online: “It seemed like a nice challenge to investigate how to imitate interaction in online education. In fact, that is why I started teaching the MOOC.” Creating an online community including 17.000 participants in an Facebook group was key to the success of the MOOC. Marc van Oostendorp: “Being together with students online, made the MOOC experience totally new to me. Knowing that there are thousands of students at the same time learning the same content is very stimulating.

That is why for me a MOOC is the best new thing to study self paced, it is stimulating to learn together. Faculty of Humanities. Instructor Marc van Oostendorp, course: Miracles of Human Language.

An Introduction to Linguistics.

Russian Brazilian Portuguese Chinese Greek Italian Spanish Ukranian Arabic Bahasa Indonesia Korean Hungarian Dutch Turkish Vietnamese Romanian Portuguese Bulgarian Croatian Japanese Swedish Danish Malay 0

5

10

15

20

25 translators

Figure 4: Number of translators, per language

Sustainable MOOC development In the course of 2015, Coursera has been experimenting to find additional revenue streams for sustainable financing. One of the most visible consequences of this approach resulted in an overhaul of Coursera’s motto from “the best courses in the world, for free” to “...accessible for everyone” and later to “Take the world’s best courses, online”. From Coursera’s perspective, these changes were meant to open opportunities to explore revenue models, while allowing Coursera to maintain its commitment to 100% financial aid for all learners for whom payment remains a prohibitive barrier. While individual courses still can be joined for free, courses that are part of a specializa-

tion feature a popup window with the choice between 1) payment or 2) financial aid. Since Specializations gathering in prominence on the Coursera website, many visitors we speak to get the impression that MOOCs no longer available for free. We regret this because our aim is that Leiden MOOCs are always free to take, and that one only pays (or applies for financial aid) for an -optional- verified certificate. We are working closely with Coursera to ensure learners are well-informed of the options available to them.

In part as a result of our feedback, Coursera has made recent changes to the user interface to make the financial aid option more prominent. We are not opposed to experiments with new monetization models as a financially healthy platform provider and the continuity of the platform are important for Leiden University. Also, the share the university receives helps financing MOOC updates and to keep them running on-demand. Our conviction is that monetization forms like sponsoring, crowdfunding, pay forward (for others) and pay-what-it-is-worth-to-you are monetization forms that fit our aims of sharing knowledge better. Therefore we will experiment with those optional forms of payment in the coming year.

A first promising experience is described in the next paragraph. MOOCs organized by Leiden University will remain free to take.

Crowdfunding Open Content Even though many of our learners are not interested in a course certificate, they are willing to donate money. Our crowdfunding experiment in 2015 resulted in raising enough money for the development of a new module for a course the learners enjoyed. This shows that users are willing to donate money to enable their own learning as well as that of others, and to keep content open and free.

26 - Innovating the way we teach and learn

27 - Innovating the way we teach and learn

DEMOCRATIZATION THROUGH EDUCATION

40% Percentage of contributors

Demographics Figure 6 shows the 10 most featured countries of origin for our learners. Most of them come from the United States. Only three non-Western countries are featured in this list (India, China and the Russian Federation). When compared to the Coursera average, Leiden University attracts a lower percentage of learners from upcoming economies, but attracts a higher number of European students.

Based on survey data in our session-based MOOCs, we gather information about the age, occupation, and course experience of our learners. These statistics are useful in evaluating our courses, and serve as a basis to our iterative approach to create better content.

Approximately 5-10% of all learners complete our surveys. Simultaneously, we use other sources of data to get a better understanding of user demographics and engagement.

60%

United States

20%

United Kingdom

Netherlands

India

0%

<€5

€5-€10

€10-€25

€25-€50

€50 - €100

€100 - €500

>€500

Spain

China

Figure 5: Results of our crowdfunding campaign to finance an additional module for the MOOC ‘Miracles of Human Language’ Canada

Germany

Russian Federation

France

0%

5%

10%

15%

Figure 6: Ten most featured countries of origin for our learners. Data is based on roughly 90.000 learners of 7 MOOCs

20%

25%

28 - Innovating the way we teach and learn

29 - Innovating the way we teach and learn

When comparing age groups between regions, we see that for every continent the largest group falls into the category of 25-34 years (figure 7). With the exception of Oceania and, to some extent, the Americas, learners of age 45 and higher are scarce. Asia, in particular, is represented by younger learners aged 18-24 and 25-34.

As for gender differences across continents, we observe that (with the exception of Africa) the overall gender distribution is roughly equal. With respect to the Coursera average (roughly 60% Male, 40% Female), this is a more equal distribution.

AGE GROUPS

EUROPE

40% 35% 30% 25% 20% 15% 10% 5%

30% 25%

AMERICAS

AFRICA

GENDER BALANCE

Roughly 38% of respondents for the course International Tax Law have a background in law, and 20% of respondents for the same course have a background in accountancy, banking or finance (figure 9). When compared to other courses and the Coursera average (2.4% for the former category and 14.3% for the latter category), this is a remarkably high rate. Although not surprising given the subject matter of the course, this fact, coupled with the observation that the course has a much lower adoption rate among other employment areas, indicates that this course is specifically popular among professionals.

65+ 55-64 45-54 35-44

F: 51%

M: 49% Student (college/university) (15.9%)

25-34 18-24 <18

Teaching and education (11%)

65+ 55-64

20%

45-54

15%

35-44

10%

25-34

5%

18-24

40% 35% 30% 25% 20% 15% 10% 5%

Areas of Employment Our courses are popular with both university students (figure 8, 16% of total) and those employed in the education sector (11% of total). The former group is relatively stable over all courses. The second group is remarkably higher (22% versus average of 8% for other courses and 5.3% Coursera average) in Miracles of Human Language MOOC. This indicates a high adoption rate among teachers for this particular course.

Law (7.3%)

F: 53%

M: 47% Accountancy, banking and finance (6%)

<18

65+

M: 68%

55-64 45-54 35-44

F: 32%

Retired (6%)

Business, consulting and management (5.8%)

25-34 18-24 <18

IT and information services (5.2%)

ASIA

40% 35% 30% 25% 20% 15% 10% 5%

25% 20%

OCEANIA

15% 10% 5%

65+ 55-64 45-54

F: 45%

Public sector (5%)

35-44 25-34

M: 55%

18-24 <18

65+ 55-64 45-54

Health and social care (2.8%)

F: 50%

35-44 25-34 18-24 <18

Seeking work (3.8%)

Other (12.4%)

M: 50%

Figure 7: Breakdown per continent for age groups and gender. Graph is based on several data sources of roughly 15.000 learners

0%

5%

Figure 8: percentage of participants, per area of employment. Aggregated over all courses

10%

15%

20%

30 - Innovating the way we teach and learn

A similar result can be observed (although less strong) for the course Wheels of Metals, Urban Mining for a Circular Economy. This course, which was primarily aimed at professionals, attracted a higher-than-average number of learners in areas such as engineering and manufacturing (14%) and business, consultancy and management (10%).

31 - Innovating the way we teach and learn

A relatively large group of MOOC participants (6%) indicate that they are retired. This is interesting especially because most survey respondents are between the ages of 25 and 35. When compared to the Coursera average (4.5%), Leiden courses are slightly more popular amongst retired learners.

Video lecture by dr. Jan Erk for his course Federalism and decentralization: Evaluating Africaâ&#x20AC;&#x2122;s track record.

Education When we compare the highest level of education attained for our courses in 2015 with the results for 2014, we see that the figures remain almost identical.

Most respondents (roughly 70% for both years) indicate that they have enjoyed university education

at the undergraduate or postgraduate level. This is higher than the Coursera average (roughly 65%).

Conversely, our courses are less popular among less-well educated learners when compared to the Coursera average.

39.4% Postgraduate university

38.6% 31%

Undergraduate university

31.5% 11.5%

College

11.7% 10.3%

Completed high school

Year

10.2%

Some additional training

3.1%

2014

3.3%

2015

2.5% Some secondary school

2.6% 2%

Prefer not to say

1.9% 0.3%

Primary school

0.2%

0%

10%

20%

Figure 10: Highest level of education among respondents for 2014 and 2015

30%

40%

50%

32 - Innovating the way we teach and learn

33 - Innovating the way we teach and learn

Course Quality and Satisfaction Rates Most students indicate that they want to learn new things from our MOOCs (figure 11). This number is higher among courses that are aimed at a wide audience (e.g. Federalism & Decentralization and Miracles of Human Language) rather than professionals (e.g. Wheels of Metals and International Tax law).

For the latter group, improving career prospects is an important reason to take the course.

“This course gives an opportunity to study for those people who don’t have a chance to study in the university. That’s a great thing that you are doing.” Student in the MOOC Rethinking International Tax Law.

As in 2014, most learners get what they want from the courses they follow (figure 12). In 2015, Roughly 50% indicate that they got what they wanted out of the course, and an additional 36% indicated that the course exceeded their expectations. These figures vary little across courses, which reflects learner satisfaction across the board.

“You have pushed me positively into the becoming the Diplomat I always love to be”. Student in the MOOC Terrorism and Counterterrorism; Comparing Theory and Practice.

40.8% Learn new things

35.4%

Yes, the course exceeded my expectations

45.9%

35.9%

21.3% Improve my career prospects

49.6% Yes, completely

19.5% 13.5%

Some additional training

To some extent

11.9%

Year

9.5%

See what MOOCs are

2015

13.4% 0.8%

6.4%

2014

6.2%

2015

5.7% Meet new people

2014

14.1%

12.8%

Get a feeling for education at Leiden University

Unsure

Year

50.2%

No

0.5%

0%

10%

20%

30%

40%

5.6% 0.5% 0.5%

0%

10%

20%

30%

40%

Figure 11: Responses to the question “What would you like to get out of this course?”, aggregated for 2014 and 2015

50% Figure 12: Responses to the question ‘did you get what you wanted from the course?’, aggregated for 2014 and 2015

50%

60%

34 - Innovating the way we teach and learn

35 - Innovating the way we teach and learn

Learner satisfaction is further reflected in the course ratings (figure 13 for session-based courses, figure 14 for on-demand courses).

In both cases, learners indicate they are happy with the quality of Leiden MOOCs. By far, most respondents (>80%) indicate that they find the course ‘excellent’ or ‘very good’.

100%

80%

42.6%

Excellent

42.6% 60%

44.2%

Very good

40.4%

Year

10.2%

Good

2014

13.5%

2015

2.5%

Satisfactory

40%

20%

3%

Poor

0.5% 0%

0.5%

5

0%

10%

20%

30%

40%

4

3

2

1

4

3

2

1

5

4

3

2

50%

Federalism 17 reviews

Figure 13: Respondents’ answers to the question ‘how would you rate the overall quality of this course’, aggregated for 2014 and 2015

5

Figure 14: Ratings for each of our on-demand courses

Tax law 66 reviews

Terrorism & counterterrorism 555 reviews

1

36 - Innovating the way we teach and learn

37 - Innovating the way we teach and learn

When we look at the engagement of learners with course components, we see that most students (roughly 83%) watch videos and make quizzes (75%). Conversely, few students indicate that they engage with the forum on a regular basis.

Do Our Learners Return for More?

Roughly 40% of all our learners has signed up for more than one MOOC organized by Leiden University (figure 16 ). This means that there exists a large student body (> 100.000) of learners who return to follow more MOOCs from Leiden University.

“I am a policy analyst from South Sudan and we are debating both decentralization and federalism in South Sudan and this knowledge will be quite handy for me and i am going to be a better policy analyst. I would be honored to take another course with you.” Student in the MOOC Federalism and Decentralization; evaluating Africa’s Track Record.

100%

never seldom some weeks most weeks every week

90% 80% 70%

184.418

60%

59.1% | 1 course

16356

4 courses | 5.2%

50%

59.916

40%

19.2% | 2 courses

12.983

5 courses | 4.2%

30%

29.247

20%

9.4% | 3 courses

8.925

>5 courses | 2.9%

10%

Forum discussions

In-video quizzes

Quizzes, exams and assignments

Readings

Watching video lectures

Figure 15: Respondents’ answers to questions about their engagement with the course components, aggregated for all courses in 2015.

Figure 16: Figure shows what percentage of learners have respectively signed up for one, two, three, four, five or more Leiden MOOCs over both 2014 and 2015

25.4%

17.9%

24.2%

45.2%

44%

46.3%

38.8%

32.9%

18.3%

29.9%

38.9%

63.6%

Septemb

82.1% 75.8%

54.8%

61.1%

New learners

53.7%

201

3

Se

Ter Co rorism u (ro ntert & und err 2) orism

Returning learners

40% 30%

Sep tem ber

& ism m or ris err rro ert Te unt 2) Co und (ro

50%

61.2% 56%

ism & Terror terrorism er Count 3) (round

60%

70.1%

4

67.1%

01

74.6%

e2

70%

81.7%

Jun

79.5%

er 2014

80%

Co wo nfig rld ur (ro ing un the d1 )

20.5%

The ch an Globa ging l (round Order 1)

100% 90%

Overall, courses are most clearly linked through time and content. That is, specialized courses attract fewer (but different) learners, and learners often sign up for the next Leiden course.

4

100%

We see that the courses the Changing Global Order (run 1) and Configuring the World (run 1) most clearly overlap in users. This is likely due to the fact that these courses are part of the same specialization.

01

Figure 18 clusters the courses together based on the overlap in course sign-ups. Here, we see that one cluster (purple text) is comprised of mainly early courses (end of 2013 and beginning of 2014), the light blue cluster concerns courses that were organised in 2015, and the dark blue cluster is comprised of several unrelated courses.

pte mb er 2

When we look at figure 17, we see that a sizeable minority of each course is comprised of learners who have already followed at least one other Leiden MOOC.

4

39 - Innovating the way we teach and learn

May 2 01

38 - Innovating the way we teach and learn

014

y2 uar Jan etals of M eels Wh

36.4% The Changing Global Order (round 2)

Introduction to EU Law (round 1)

June 2013 20%

Fe De dera ce lis nt m ra liz & ati on

201

4

20 Ju ne

April 201

15

20

5

y ar

2015 March

Figure 17: Courses (x-axis) are ordered chronologically. A ‘returning learner’ is a learner who has signed up for at least one earlier course.

No vem ber

u br Fe

15

Intern at Law (r ional Tax ound 1)

5

201

he g t 2) rin d gu un nfi (ro Co orld W

e Jun

Human les of 1) Mirac ge (round Langua

) ) r 1 n x er er 3) 4) ld ld ls de EU ) d1 d2 or ) nd ma ) l ta ) ord ) eta 1) ord 1) wor 1) und or ) nd on 1 un un e w nd 2 f hu nd 1 iona nd 1 bal nd 3 (rou al nd 2 (rou f Mund obal und the und (ro ro h cti und (ro b ( t o o t u u u u u o o u l o l a ro gl (ro ism s o g o d ro m m ars (ro ng g (ro ring (ro rism g g (r orism rin (r racle ge (r ternaw ( ing ris tro ( ris o ral Ge n l sin si s In Law erro i gu gu rro de err err Ri Ri M ngua I Ri nfi nfi T T T Te o o Fe a C C L

Ter Co rorism u (ro ntert & und err 4) orism

g ngin Cha er The al ord b Glo nd 3) u o (r

10%

October 2014

Figure 18: Result of applying a hierarchical clustering method on learner enrolments. There are three clusters (purple, grey and light blue). Both between and within the clusters, we see a strong time-based component in learner sign-ups. Additionally, we observe that the first rounds of ‘the changing global order’ and ‘configuring the world’ are more deeply clustered. This is likely to be the result of the courses belonging to the same specialization track.

40 - Innovating the way we teach and learn

41 - Innovating the way we teach and learn

IMPACT ON ON-CAMPUS EDUCATION Performance by Age Figure 19 shows the final course grade distribution by age group. All the distributions are bimodal (i.e. they have two peaks), indicating that students either do very poorly or do very well.

0.025

0.020

Nonetheless, the learning experience offered to these students does not reflect this mindset. Much like the boundary between television and the internet has been torn down in the past 20 years, hereby rendering television sets, mobile phones, tablets and computers ‘portals’ to access the same multimedia content, we aim to explore the ‘wall’ between online and on-campus education and figure out how to form the next learning experience in blended education.

0.015

Density

Today’s university students are digital/networked natives who are just as comfortable moving around in the digital world as they are in the physical world. They are used to quick fact-checking on their phones using google, sharing images, video and thoughts via social media, and possess great aptitude at scanning large amounts of information to determine its relevance.

0.010

0.005

> 18 18 - 24 > 25

0.000

0%

25%

50%

75%

100%

Figure 19: Density plot of course grades, per age group. Only grades > 0 are included in the graph. Figure is based on the survey responses and performance of roughly 10.000 learners in eight MOOCs.

Under 18 year-old learners (upcoming university students) are less likely to receive a high grade for their course. This age group often indicates that the course difficulty was more difficult than the average learner. For this group, the bimodal distribution is least pronounced when compared to the other groups.

18 to 24-year-old learners (current university students) are more strictly distributed between poor performers and good performers.

Above 25-year-old learners (life-long learners) are the best performers, and also show a less uniform distribution in passing grades.

At the heart of all this rests the belief that, ultimately, content and learning experience are what matters.

Contrarily, the medium through which education is offered should not have an effect on the student’s experience. To this effect, all learning spaces need to be geared towards optimizing the learning experience in order to cater to today’s students.

Where the Digital and Physical Classrooms Meet to Form the Next Experience in Higher Education In 2015, the Centre for Innovation Online Learning Lab has made several concrete steps towards further congruence of learning spaces at Leiden University.

We set up a Flipped Classroom experiment which received a very positive reaction and we intensified the use of our Living Lab, a space modeled after Stanford’s d-school to better reflect the needs of current-day students and located at the Faculty of Governance and Global Affairs. Additionally,

we visited other institutions to gain more experience and to help guide our goals for 2016.

Designing the Classroom of the Future In 2014, a special commission at MIT published the Institute-wide Task Force on the Future of MIT Education (URL: goo.gl/je85Qy). In this report, the authors forward several recommendations to integrate new technologies with traditional education.

In part, these recommendations focus on an overhaul of the physical learning spaces and on the need to experiment with pedagogy and various forms of blended learning in education to remain relevant and ahead of the curve.

Despite recent advances in online education, our on-campus educational format still relies on a model that predates the internet. That is, today’s students are used to having access to large amounts of information and a digital freedom of movement that is reflected in online courses, but not in our on-campus education.

Too many classes are still focused on ‘broadcasting’ knowledge in a linear fashion. For a large part, this discrepancy arises from learning spaces that are suited for such a learning experience rather than allowing students to collaborate, research and shape their physical space to fit their needs. Future learning spaces could to be re-imagined in order to meet new learning needs of students with potentially significant impact on university wide classroom design. The blended approach and a new (digital) type of student are posing new requirements for classroom setups. In recent years, Harvard moved from high tech classroom to flat floor, everything on wheels, whiteboard walls with minimal tech (apart from flex beamers/

42 - Innovating the way we teach and learn

projectors and power outlets). Setting up the classroom differently is important for future classroom setup for teacher innovation. Universities can change the culture through space changes, providing teachers with a new environment for learning.

Since its inception in 2012, the Living Lab has been a collaborative hub in which students and faculty can organize lectures, seminars, workshops and other activities. The main goal of the Living Lab is to promote collaboration between students and faculty by providing a space that is flexible and can serve many different purposes depending on the needs.

The Living Lab provides students with flexible furniture, a large, open space, beamers, meeting areas and furniture that can be used as a whiteboard. The effect we see on the instructor in this new structure asks

Taking students outside of the traditional classroom.

43 - Innovating the way we teach and learn

questions such as: where do I stand? How do I relate to my students in this new setup? While students ask whatâ&#x20AC;&#x2122;s my role as a student in this space?

As institutions it is important to realize that students are not looking for technology for the sake of technology. Rather, they are looking for a learning experience that fits their needs, and we need to design for this in our online and physical classrooms and other learning spaces. Key takeaways from the various visits and working within the Living Lab area are: Put collaboration central in the classroom (flexible), Informal spaces are key around the classroom for deep work and learning and Provide for makerspaces/ testing spaces (also for students of social science, law and humanities).

44 - Innovating the way we teach and learn

45 - Innovating the way we teach and learn

Lessons Learned from Flipping the Classroom In 2015, students who were enrolled in the oncampus Masters course Tax Treaties simultaneously participated in the MOOC Rethinking International Tax Law together with some 14.000 participants from all around the world. As the on-campus course mainly consisted of traditional lectures, the MOOC supplemented this course through participation in the online MOOC forums, peer reviews, and by providing additional content.

The data shows that the Leiden University students outperformed their peers in the other groups (figure 20). Furthermore, Leiden Students were better at selecting content relevant to them, and received higher grades for the peer-graded assignments.

Leiden students indicated that they enjoyed participating in the MOOC. They especially enjoyed the flexibility with which the format presented them, the supplementary skills they learned, and the additional

35

30

Grade

25

20

information they received from the online lectures and readings. Additionally, it became clear that the online format substantially lowered the barrier to engagement with the content. Simultaneously, the course instructors of the course were pleased with the results.

We learned several key lessons from this blended experiment: • Students experienced this form of blended learning as highly valuable. Chiefly, they enjoyed the freedom with which they could consume course content.

• Even though there was no strict check on the engagement with the course content, Leiden University students watched 80% of all videos. Additionally, students identified lectures important to them and re-watched these at a higher frequency than other types of students.

• A tight integration between on-campus and online courses are essential both in content as in set up. In future blended projects, this will be one of the focus points.

How Leiden University Students use our MOOCs For the years 2013 to 2015, we matched our database of MOOC learners with the on-campus student body database. We wanted to see whether we could identify students who followed a MOOC before joining an on-campus course, and wanted to understand if, and if so, how, our current on-campus students used our MOOCs.

15

10

5

Regular Coursera Students

Advanced track Students

Leiden University Students

Figure 20: Violin plot of grade distributions for regular Coursera students, advanced track students and Leiden University students that participated in the course ‘International tax law’. The boxplots show the summary statistics of the final exam for each group. The shape around the boxplot shows the grade distribution for students in each group using a double density plot.

“It is a fun way to learn at your own pace”. Leiden student in the MOOC “Rethinking International Tax Law” “I really like learning through MOOCs. It allows me greater control over my schedule“ Leiden student in the MOOC “Rethinking International Tax Law”’

Teaching assistants interviewing Professor of Linguistics N. Chomsky.

46 - Innovating the way we teach and learn

47 - Innovating the way we teach and learn

We identified a total of 1.961 students who participated in one of the session-based MOOCs of Leiden University.

Of these students, we identified 254 (13%) students as having followed a MOOC before attending an on-campus course. The remaining students either followed a MOOC while already studying at Leiden University or were alumni.

terrorism & coun. terrorism (round 4) terrorism & coun. terrorism (round 3) terrorism & coun. terrorism (round 2) terrorism & coun. terrorism (round 1) intro to European law (round 1) international tax law (round 2) international tax law (round 1) miracles of human language (round 1) miracles of human language (round 1) rising global order (round 3) rising global order (round 2)

254

rising global order (round 1)

13% | Potentially recruited students

federalism & decentralization (round 1) configuring the world (round 3) configuring the world (round 2)

1.707

87% | Leiden students

Cu ltu re le

an tr op ol o

gi

Fi s Ja pa e e caal ns n Pu re tu o ch bl n di tw t ic es (g ik Ad ra ke m du lin in a gs ist so te) r In ci Ps atio o te yc lo n rn g ho ( at lo g ra ie io gy du na M lR id a ( gr te) de ela ad ntio ua O ns os te ) te an ns d tu D d ip P ie lo s m sych ac ol y og ( i gr e Cr a isi Cr dua sa te Po im nd lit in ) Se ic ol al cu og Sc Fis rit ie ie c yM In a n a te ce lr an rn ec ag ( at io em gra ht na du l R ent at e) ela ( tio grad ns ua te (g ) Li ra b G dua Re era l A es te ch ) rt chi tsg ele s an ede ni d er Sc s dh ie ei nc d es (g ra du a te Po ) lit i Be colo Re stuu gie ch rs In tsge kun te de l e rn er dh at io e i na d lS tu di es

configuring the world (round 1)

Less students from a particular study enrolled in MOOC

More students from a particular study enrolled in MOOC

Figure 21: Results of matching the USIS database with MOOC participants. Our study indicates that MOOCs may help in recruiting new students.

Figure 22: overview of Leiden University student participation in Leiden MOOCs. The x-axis features 15 on-campus programmes which are ordered by the total amount of enrollments in Leiden MOOCs (shown on the y-axis). Darker color shading indicates higher enrolments in a given MOOC; lighter color shading indicates less enrolments

48 - Innovating the way we teach and learn

49 - Innovating the way we teach and learn

MOOCS AS A PLATFORM FOR RESEARCH In 2016, we will conduct further research to understand the connection between online education and taking on-campus education for these students.

“Lasting effects” Professor Stefaan van den Bogaert was the first in the Netherlands to produce a Massive Open Online Course: ‘The Law of the European Union’. Starting the MOOC was a big step into the unknown, but according to Van den Bogaert it left lasting results: “When starting the online course we had no idea what the effect would be, we were pioneering. During recording one cannot see the students, only the red-light of the camera, but in fact the whole world is watching”. The MOOC drew some 40.000 participants, following video lectures making quizes and discussing EU-Law on the forum. Started already two years ago, the results are still visible today. Van den Bogaert: “After having given the course online, we noticed a growth in the number of applications for the Master in EU Law from both within and outside of the EU. In fact students approached me to say that they have chosen Leiden University because of the course. I can say now, that there is a clear MOOC effect in Leiden.” Faculty of Law. Instructor Stefaan van den Bogaert, course: The Law of the European Union. An Introduction (2014).

In the nine months during which our research team was active in 2015 we focused on developing our data infrastructure, a research track, and on implementing actionable learning analytics. Together with researchers at Leiden University and partner institutions, we focused our efforts on the following areas: • We started building a data infrastructure to manage the deluge of learner data given to us by Coursera and finding secure ways to pass this data onto other researchers.

We designed a proper infrastructure to store Coursera data in a fast, flexible and secure way, and further created tools and documentation to share our expertise.

• We engaged in fundamental research and set up several research tracks that focus on different aspects of online learning, such as learning materials, educational interventions, student behavior, and learning outcomes. We further published four peer-reviewed research papers, including one open-access publication at a leading conference on learning analytics.

• Learning analytics allows us to impact online learning materials with practical insights. Among other things, we published one report in cooperation Dr. Maartje van den Bogaard (ICLON) in which we evaluated the use of the MOOC Rethinking International Tax Law in the on-campus course Tax Treaties, organized by Professor Sjoerd Douma at the Law faculty.

Facilitating Research Through Tooling and Analytics Several tools and services have been created in 2015 that researchers can use to do analysis with. These consists of a data management policy which defines how data can be shared and used, a set of github tools which provide analytics capabilities to researchers.

Video lecture by Dr. Ester van der Voet for her course Wheels of Metal: Urban Mining for a circular economy.

Data Management Policy When it comes to managing large datasets, few topics are as important as user privacy. In order to ensure the protection of our user’s data, we created a data management policy in which we outline how we store and retrieve data, and under which circumstances and conditions we share this data with third parties or researchers. In 2016, we further develop this policy in light of new European privacy laws and new data formats.

Our tools are open-source The Online Learning Lab maintains several repositories containing tools to facilitate analytics with Coursera MOOC data.

These tools are geared towards facilitating researchers within Leiden University, as well as researchers from other universities.

Check out our tools here: https://github.com/LU-C4i

50 - Innovating the way we teach and learn

51 - Innovating the way we teach and learn

Online Learning Research Next to the provision of tools and procedures for learning analytics, the team also started working on a research agenda with the University wide Higher Education Research Group (CROHO), the universityâ&#x20AC;&#x2122;s School of Education (ICLON) and the Centre for Education and Learning (CEL, a collaboration between the universities of Leiden, Delft and Rotterdam). MOOCs offer new ways to look at learning.

The approach blends three disciplines; educational sciences to investigate teaching and learning aspects, computer science for data analysis and modeling, and psychology to understand and study individual processes such as motivation and information processing.

Github Tools We realize that the expertise to handle large datasets is not abounding at our partner institutions, we therefore have released several tools on the online sharing platform GitHub. These tools allow researchers to store, retrieve and manipulate MOOC research data.

In 2016, we will further develop these tools to include a comprehensive package for the R statistical language.

By evaluating data from 500,000 interaction sessions with 100 videos from our MOOCs we found that, indeed, our students need more time finishing a video when the complexity is either low or high, supporting our hypotheses.

6

This research line is exemplar for answering fundamental questions within an applied setting. Using large scale, actual usage data we applied a psycholinguistic model of information processing difficulty and confirmed classical laboratory research hypotheses about human attention and interest. Simultaneously, we pave the way for selecting videos of an optimal level of difficulty for each studentâ&#x20AC;&#x2122;s individual knowledge.

4

- Application and verification of psycholinguistic findings - Retrieval practice 2

PSY - Instructional design of videos - MOOC design - Community

EDU

CS

Actionable learning analytics - Segment of users on level of understanding - Offer different / extra content

Information Density and Understanding Our most recent research publication addresses a fundamental issue in didactical videos. Our hypothesis is that students struggle to understand difficult videos, but similarly struggle to concentrate when the difficulty is low.

8

Dwelling rate

Data as a Service For researchers at the University of Leiden, we offer the opportunity to make appointments to discuss and receive access to our wealth of learner data. We also provide assistance in the safe storage of data in a format familiar to the researcher.

Together, these three areas are important to provide feedback to instructional designers, to improve actionable learner analytics, and to better understand students and their learning process.

0 0.5

1.0

1.5

2.0

2.5

rate rate and dwelling rate.

Figure 23: The relationship betweenInformation video information The former is a measure of both the complexity of information and the time in which the information is presented. The latter relates to the amount of a video that a student sees.

52 - Innovating the way we teach and learn

53 - Innovating the way we teach and learn

COLLABORATIONS Effective Learning Strategies: Retrieval Practice To promote meaningful learning, an instructor should employ educational strategies, which are consistent with human cognitive learning mechanisms.

In the case of online education, this implies that the range of instructional study objects (e.g. videos, animations, book chapters, articles and discussion platforms) should be designed and implemented in accordance with these principles. The main purpose of this line of research is to identify the core active ingredients of instructional study objects and get a better understanding of how, for whom and in what context they have an impact on learning. Specifically, we aim to understand how students process information in an online environment, how we can enhance the learning process to make it more efficient and more effective, and how we can Increase the pedagogical quality of MOOCs

These studies will strengthen the body of literature and provide instructors with evidence-based guidelines for how to design, enhance and optimize open online courses. This, in turn, will increase the educational benefits of MOOCs for the students.

Over the past year a number of collaborations have been strengthened or started.

Peer Reviews: Online Performance and Collaboration Peer assessment is more than merely a scalable assessment procedure for more complex assignments. In addition to alternatives such as automated essay scoring, peer assessment serves a formative purpose as well; participants are considered to benefit from both providing and receiving peer assessments, as well as from their encounter with others’ work and perspectives. An explorative study by Bart Huisman, a Ph.D student at the Graduate School of Teaching (ICLON), investigates this formative aspect of the peer assessment procedure. More specifically, it investigates the extent to which the ability of peer reviewers influences the subsequent performance of participants.

• The GitHub tools we developed have been shared with Coursera partners, and formed the basis for successful further exchange of experiences. It formed a stimulus for Coursera to launch the Coursera Research Community.

• We participated in a research summit at the University of Michigan about viable new avenues for research in MOOCs. • The Online Learning Lab staff works closely together with PHD-candidates from various organisations e.g. Centre for Education and Learning from Leiden-Delft-Erasmus.

• University of Utrecht and Edinburgh, we initiated a meeting for MOOC and online learning researchers of all LERU partners who aim to connect and collaborate. A second meeting will be organised in april 2016.

In 2016, our research projects will use MOOCs and SPOCs as a vehicle for research to answer fundamental questions about teaching and learning in an applied setting. In doing so, we simultaneously try to explain student behavior and to stimulate productive student behavior by answering design questions on how to promote positive learning outcomes for our students.

Research: Collaboration and connections with other researchers in the field of MOOCs and online learning:

Coursera: With Coursera there are a number of collaboration projects which will be built on this coming year:

• In 2015 we decided to become an innovation partner of Coursera which enables to test new ideas quicker and know earlier than most how the MOOC space is developing.

• In 2016 the Coursera Partner conference will be hosted

by Leiden University and University of Geneva.

Collaboration with University partners Looking for ways to collaborate with other partner universities can be helpful in developing new programmes or influencing policy.

• Discussions regarding the accreditation of MOOCs with LERU partners and the Dutch ministry of education.

• Further develop specialisations together with University of Geneva • Explore the setting up of a Global Master’s programme with two other universities.

54 - Innovating the way we teach and learn

55 - Innovating the way we teach and learn

NEXT STEPS In 2016 the Online Learning Lab will take further steps in the following innovation lines.

Exploring New Technologies Focus on storytelling, visual technologies and pedagogy. Online learning initiatives represent a new field of education in which there is still a need to experiment. First attempts have shown that implementing traditional methods such as lectures and talking heads in an online environment do not make use of the full potential of online learning. As such, the question becomes: how do we move forward to an evidence based, flexible approach, that allows us to further test and implement new educational technologies within a rapidly changing technological landscape? The aim is to bring together instructors and creative professionals to create work as Virtual Reality (VR) and interactive media that will allow students to engage with these questions through purposeful projects that unite storytelling with innovative visual technologies.

Recognition of MOOCs and Online Learning Towards Credit How can we ensure that universities recognize each other’s online courses towards credit? This is both a question for open courses that currently offer non-formal certificates and for closed online accredited courses that offer ECTS. Which regulations apply and, if necessary, how should they be adapted? How can we guarantee the validity of exams at a distance?

What quality should the instructional design of the courses have? How far can we go in offering personalized choices while maintaining a coherent curriculum? And how do we scale this processes within our departments?

In collaboration with the department of Academic Affairs, faculty, LERU and other (inter)national

partners, we will elaborate solutions for recognition and accreditation of open and closed online courses.

Our vision is that Leiden University students, within a few years, can benefit from a qualified offer extending beyond the Leiden programs and vice versa, and can personalize their studies to prepare themselves optimally for their future careers.

Piloting Online Proctoring In 2015 we published Effective online education requires valid online assessment: could online proctoring offer the answer? coauthored by Marja Verstelle (Online Learning Lab) and Marinke Sussenbach (TU Delft) in the SURF Open and Online Education Trend Report 2015. The coming year we will organize an online proctoring pilot to better understand and develop the monitoring of exams at a distance. Several of the SPOCs we are developing with faculties in 2016 will offer online proctored exams. We will report on the possibilities and requirements needed to scale this service.

Designing for Deep Learning One of the main goals of online learning is to promote “deep learning”, a term commonly used to signify learning beyond remembering and reproducing. Deep learning challenges the student to apply, analyze and evaluate and finally to create new knowledge. It leads to long-term retention of the learned as opposed to short-term retention used only to pass the course. The Online Learning Lab regards the integration of deep learning in online material as one of the main challenges for 2016. On the one hand, our research team looks into ways to stimulate this in a variety of ways. On the other hand, online learning can harness deep learning in contact hours,

often referred to as Flipping the Classroom. We will run 5 Flipped Classroom experiments and evaluate the impact on deep learning.

Another way to look at this is by asking instructors what the most important things they want to research about learning and teaching in their specific field/ subject. What are the concepts students most struggle with and can we design experiments to figure out what will fit students best? These can then be modeled into experiments in an online learning context.

Adopting an Online Mindset for Our Physical Spaces In the coming year we focus more on designing the classrooms and learning spaces. Technological advances (be it digital or material innovations) increasingly shape the physical environment in which students learn, study and work; the university building of the future could look very different from the ones we have today. All spaces are collaborative spaces which are adaptable which can be tailored to a specific learning approach. One vision is that teachers have a deeper understanding of the use of didactics in a physical environment. This is where social learning and individual coaching based on data driven learning analytics come into play. Though tracking students in the classroom teacher will see a student’s performance and will be able to define appropriate action in real time while operating in an interactive learning environment.

Experiment with Alternative Monetization Structures In order to secure sustainable funding to keep our MOOCs running and up to date, we aim to experiment with new forms of user contributions. Only

56 - Innovating the way we teach and learn

57 - Innovating the way we teach and learn

ACKNOWLEDGEMENTS a small part of learners aims to get a certificate, while we know many of them would love to support us in some way to keep making and updating free MOOCs. The crowdfunding pilot described earlier in this report proved to be a positive experience. Our conviction is that monetization forms like sponsoring and crowdfunding are monetization structures that fit within our vision to maintain open content.

The upcoming MOOC De-mystifying Mindfulness is a good example of a topic in which these type of contributions from participants are more opportune than fees for certificates. In this MOOC we will run and evaluate a pilot in 2016 with different forms of contributions and share the outcomes with Coursera and the MOOC community.

Stimulate sustainable development of course content Most faculties that are developing MOOCs, SPOCs and other online materials have the intention of using the online materials in their on-campus education.

The effort and costs of developing the materials is high, put pays itself back when the materials can be used in different settings and over many years. As such, the Online Learning Lab needs to focus more and more on modularity: making small chunks of content which are independent of the specific context in which they are produced first. For example, this means leaving out numbering, references to recent events, and splitting content that will not change soon from topics that might, although it is not always easy to foresee (who had expected two years ago that the Schengen Treaty is under threat today?).

A second approach towards sustainability is the reuse of materials of others, which we think is as much a cultural issue as a technical and practical one. Therefore, we start with challenging ourselves to use existing online (materials in our) courses for faculty: teach as you preach. We aim to develop SPOCs in such a way that they can be cloned for use in other domains of Leiden University. And for faculty we aim to develop a simple support addressing three concerns: • Where can I quickly find high-quality materials?

• How can I adapt them for my class?

• How am I allowed to use it and if needed, how can I clear copyrights?

Exploring the Online Master Space Leiden University has at least one blended master (Advanced Master Air and Space Law) where students do part of the work remotely. Over the past few years a number of full-fledged online master programmes have been developed which provide more flexibility for students and a greater reach of existing programmes. In 2016 we will be exploring the possibilities to develop a number of online (blended) programmes with various faculties. For example the Leiden Medical School’s (LUMC) future lab would like to create a masters programme which will consist for 75% of online components. In the law (Tax Law) and Archeology departments there is also interest in the development of online masters. Online education can also help develop new and deeper partnerships with other universities and organisations, for example through teaming up with a number of universities and develop a global master programme where students travel to different campuses for short intensive periods of time and to parts of the degree online.

This report is the collective achievement of a large number of colleagues who spent massive amounts of time developing methods, tools and courses.

Without their time, enthusiasm, support and ideal all this would not have been possible.

Online course teams Rethinking International Tax Law (MOOC) (one session-based MOOC and transition to on-demand) Prof. mr. S. Douma, Mr. drs. Judith Reijnen, Max van Boxel volunteers: Pablo Angel, Paula Beneitez, Stewart Cotteril, Vasileios Dafnomilis, Pedro Luz, Mahesh Nayak, Jessica (Chu) Shi. Federalism & Decentralization; Evaluating Africa’s Track Record (MOOC) (one session-based MOOC and transition to on-demand) Dr. J.G. Erk, Miranda Verboon, Gerrit Krol volunteers: Terresa Lewis, Saskia Kellenbach.

Miracles of Human Language; an Introduction to Linguistics (MOOC) (session 1 and 2) Prof. dr. Marc van Oostendorp, Inge Otto, Marten Vermeulen volunteers: Joshua Asmah, Valeria Hernandez Reyes, Saskia Kellenbach, Terresa Lewis, Kaori Muramatsu, Jasper Spierenburg, Kate Taylor, and 400 translators.

Terrorism & Counterterrorism: Comparing Theory and Practice (MOOC) (session 4 and the transition to on-demand) Prof. Dr. Edwin Bakker, Jeanine de Roy van Zuijdewijn

Volunteers: Sana Abdeljalil, Ali Ahmed Dyrs, Joshua Asmah, Stewart Cotteril, Mark Hatz, Valeria Hernandez Reyes, Saskia Kellenbach, Kaori Muramatsu, Iris Soliman.

The Changing Global Order (MOOC) (session 2 and 3) Prof. dr. Madeleine Hosli, Denise de Buck Volunteers: Joshua Asmah, Stewart Cotteril, Terresa Lewis.

Configuring the World; A Critical Political Economy Approach (MOOC) (session 2 and 3) Prof. dr. Richard Griffiths, Einat Shitrit Volunteers: Danny Damen, Saskia Kellenbach, Leo Landoll (Sandy), Kaori Muramatsu, Stewart Cotteril.

Configuring the World; A Global Adventure (SPOC) Prof. dr. Richard Griffiths, Einat Shitrit.

Capstone Global Affairs (session 2) Prof. dr. Richard Griffiths, Prof. dr. Madeleine Hosli, Einat Shitrit.

The course teams of courses that started in 2015 and launch in 2016 International Law in Action: A guide to the International Courts and Tribunals in the Hague (MOOC) - January 2016 Prof. mr. Larissa van den Herik, dr. Yannick Radi, dr. Cecily Rose, Lieneke Louman.

Volunteers: Stewart Cotteril, Tatiana Ramirez, Jaymie Wink, Louis Léonet.

58 - Innovating the way we teach and learn

59 - Innovating the way we teach and learn

THE ONLINE LEARNING LAB Centre for Innovation online learning team The Rooseveltian Century (MOOC) - January 2016 Prof. dr. Giles Scott-Smith, dr. Dario Fazzi Volunteers: Stewart Cotteril, Saskia Kellenbach.

Clinical Kidney Transplantation (MOOC) - January 2016 Dr. Marlies Reinders, Prof. Frans Claas, prof. Cees van Kooten, dr. Sebastiaan Heidt, dr. Volkert Huurman, prof. dr. Hans de Fijter, prof. dr. Eelco de Koning, dr. Andre Baranski, prof. dr. Ton Rabelink, dr. Aiko de Vries, dr. Jan Nico Bouwes Bavinck, prof. dr. Leo Visser, dr. Ingeborg Bajema, Bart Oerlemans, Erik Pasveer, Manon Dijkerman.

Volunteers: Maarten Engelse, Bram Voorzaat, J.R. Bank, Greetje Dreyer.

Be Persuasive: write a Convincing Position Paper or Policy Advice (MOOC) - February 2016 Prof. dr.Richard Griffiths, Miranda Verboon.

Volunteers: Dragana Djordjevic, David Talbot.

Anatomy of the Abdomen and Pelvis; a journey from basis to clinic. (MOOC) - April 2016 Prof. dr. Marco de Ruiter, Daniel Jansma MSc, MD Paul Gobee, Dr. Beerend Hierck, Bas Boekestijn MSc. Volunteers: Teressa Lewis, Denise Arnold, Yara Willems, Aglaia Hage, Thomas Kluck.

International Law in Action; Investigating and prosecuting International Crimes (MOOC) - Spring 2016 Prof. dr. Carsten Stahn, Lieneke Louman.

Heritage under Threat (MOOC) - Summer 2016 Dr. Sada Mire, drs. Mara de Groot, Krijn Boom, Katrin Hannen, Prof.dr. Maarten Janssen, Joseph Powderly, dr. Henrike Florusbosch, Prof. dr. Gerard Persoon, Prof. dr. Petra Sijpensteijn, drs. Marieke van Haren

Gideon Shimshon Co-founder Online Learning Lab & Director of Centre for Innovation

Marja Verstelle Co-founder Online Learning Lab & Learning Innovation at Centre for Innovation

Evolution Today (MOOC) - Fall 2016 Prof. dr. Menno Schilthuizen, dr. Rutger Vos, dr. Maurijn van der Zee.

Jasper Ginn Data analyst & research

Tanja de Bie Project Coordinator Learning Innovation & Community manager

ICTO Program steering committee Simone Buitendijk, Kurt De Belder, Jan Willem Brock, Frank Damen, Henk Dekker, Hans van Dommele, Evert Fortuin, Marlou Grobben, Joost Kok, Rick Lawson, Paul Nieuwenburg, Jos Schaeken, Bernard Steunenberg, Marja Verstelle, Han de Winde.

Annemieke van den Bijllaardt Online Learning expert

Thomas Hurkxkens New Media director

Leontine van Melle Project Coordinator Learning Innovation

Einat Shitrit Project Coordinator Learning Innovation

Nestor Romero Clemente Video editor Open & Online Learning

Frans van der Sluis Postdoctoral Researcher

On Being a Scientist (SPOC, MOOC) - Summer 2016 Prof. dr. Bas Haring, Prof. dr. Frans van Lunteren, Prof. ing. Ionica Smeets, Remco Schadewijk, Saskia Kellenbach.

Coursera Partnership Team Daphne Koller, Emma Webb, Melanie Lei, Meera Ramakrishnan, Evan Shore, Isaac Guttman, Alex Sarlin, and Coursera partner support team.

60 - Innovating the way we teach and learn

61 - Innovating the way we teach and learn

CENTRE FOR INNOVATION - ONLINE LEARNING LAB STRUCTURE

APPENDIX I: METRICS PER COURSE

The Online Learning Lab is founded and managed by the Centre for Innovation at Leiden University.

The Centre for Innovation helps to prepare Leiden University for the digital future by identifying and exploring technology-induced trends in education and research.

The mission of the Online Learning lab is to experiment with new developments in online and open education. It further provides an online teaching platform for teachers to use as starting point for their own ideas about - and new possibilities of the use of - online courses. The operating principles regarding open and online learning are: • Innovate: develop new concepts around MOOCs and Online Learning by accelerating new initiatives in online learning and keep positioning UL at the forefront of online learning developments.

• Service: Further develop services in online learning, didactics and production for campus education that can be used by all faculties in the university.

• Scale: Scale up our current MOOC engagement base from 150.000 to 1.000.000 joiners by 2017.

• Execute: Delivery of the online learning project proposals together with university partners.

• Standardize: Develop a professional production team for online education (MOOCs and SPOCs) that functions for the entire university.

• Develop: SPOC’s and new online projects with faculty for education to on campus students.

• Lessons Learned: Based on experience and lessons learned contribute to strategic discussions of Leiden University around the new LMS platform and long term physical infrastructure vision.

The core support team is responsible for learning and teaching innovation, and project management.

This includes instructional design, video production, coaching, technical support, development support, online community management, communication and participant recruitment, evaluation and data analysis.

The organisation is based on a network model and deploys agile methods to ensure that innovation and production quality ar aligned at all times.

A broader network is involved to help with issues around copyrights clearance (Library UBL) and for contract and IP issues (LURIS) Also the lab works in collaboration with various researchers at the Leiden University centers for research into higher education (ICLON-Leiden University Graduate School of Teaching and CROHO) and at the Centre for Education and Learning (Leiden University, TU Delft and Erasmus University). On press and recruitment approaches the Lab works with the Leiden Strategic Communication and Marketing Department (SCM).

During the production process MOOC teachers are supported by a junior teacher or student assistant who takes care of the course coordination together with the project management team of Online Learning Lab.

Session-based courses Course

Session

Enrolments

Signature Track

completed learners

Configuring the World

2

20.225

541

910 (4.5%)

Configuring the World

4**

8.957

428

642

Rethinking Tax Law

1

14.203

538**

767 (5.4%)

Changing Global Order

2

14.899

576

990 (6.6%)

Changing Global Order

3

12.848

569

823 (6.4%)

Federalism and Decentralization

1

3.788

84

223 (5.9%)

Miracles of Human language

2

16.058

309

745

(7.1%)

(4.6%) Miracles of Human language

1

49.924

435

2536 (5%)

Terrorism and Counterterrorism

4

11.769

231

703 (5.9%)

Course

Running since

# Total enrolled students

# Paid Certificates

# course completers

Federalism and Decentralization

September 2015

2.903

62

22

Rethinking international Tax Law

November 2015

10.468

240

105

Terrorism and Counterterrorism

March 2015

76.560

1.660

1.844

On-demand courses

** Of which approximately 53 Leiden University students for whom the certificate was gifted by the university. ** There is no third session for this course.

62 - Innovating the way we teach and learn

63 - Innovating the way we teach and learn

Ontwerp: Studio Piraat