TU Delft Extension School – QA Handbook

Page 4

Quality Assurance Handbook

Photo by Russ Ward on Unsplash
2 Authors: Clelia Paraluppi, QA Officer and Nida van Leersum, Policy Officer Approval and autorisation: Arno Smets, Director ES Education, and ES Management Team Date: 1 January 2023 Foreword ..3 Quality culture ..4 Quality standards and indicators .......................................................................... 8 Quality Assurance cycles.................................................................................... 12 Fraud, Code of Conduct, complaints .................................................................. 20 Evaluation instruments 24 Conclusions ........................................................................................................ 26 Annex 1: Table for the operationalisation of the Quality Standards & Indicators 28 Annex 2: Example of an evaluation instrument: Post-survey ............................. 42 Annex 3: List of accredited courses 47 Contents

Foreword

This Quality Assurance Handbook describes all the Quality Assurance (QA) activities relating to the work of the Extension School (ES). It comprises a summary of all short and long-term QA activities, information on when they will take place, and our implementation strategy

Quality has always been a guiding principle of our work. Moreover, when on 1 January 2021 ES became part of the University Corporate Office, we used the opportunity offered by this organisational restructure to look more actively at areas of improvement, and made QA one of the core tasks of the new organization.

To date, ES has taken huge steps in furthering its quality agenda. Some key achievements are:

• setting-up a Quality Assurance Board (QA Board) that also comprises independent education professionals. The QA Board is responsible for the quality of examinations, final assessments and certificates, and for monitoring that QA processes are being followed;

• establishing Learning and Assessment Regulations;

• hiring a full-time QA Officer, which is a new position within ES. The QA Officer is responsible for all quality-related activities as well as to for monitoring that quality processes are being followed;

• defining Quality Standards and Indicators at Institutional, Portfolio and Course level;

• running the first ‘ES quality & working culture survey’ completed by ES staff. This gives insights into the current status of quality within our organisation and indicates areas of improvement for the coming years;

• holding a quarterly meeting with all ES staff entirely dedicated to quality. Staff jointly brainstormed about quality values and how we define and co-create an improved ES quality culture based on those shared values.

In addition, we finalised and implemented a range of crucial policy documents and regulations, ensuring our commitment to quality in all aspects of our work. However, this is just the beginning, as we are on a journey to continuously improve by reflecting on what we can do better.

Thus far, we have identified three main areas of improvement: (A) having a quality culture that permeates everything we do, (B) operationalising our defined and shared quality standards and indicators and, (C) having robust QA processes to embed our quality standards and to help us to effectively deliver our quality activities whilst identifying areas for improvement.

This handbook describes the above three areas and, wherever possible, it outlines who is responsible or accountable for certain activities and how these are included in our working and reporting cycles. The handbook serves as a reference and guide for ES management and staff. We are proud of what we have achieved; everything we do is driven by our underlying commitment to enhance the quality of campus and online education.

3
The Extension School Management Team

Quality culture

One of ES’ overarching goals is to have a strong quality culture; it is one of the Strategic Targets 2025 (see figure on the right). Additionally, one of our core tasks is to manage the QA of the ES portfolio.

The Harvard Business Review defines a ‘true quality culture’ as an environment in which employees not only follow quality guidelines but also consistently see others taking quality-focused actions, hear others talking about quality, and feel quality all around them. ES’ ultimate goal is to create a culture in which employees feel an intrinsic drive and ownership to further strengthen quality within our organization through their daily actions.

In June 2022, we carried out a pulse survey to gauge the quality and working culture at ES. This was the first time we asked ES staff for their input on these topics. The findings from the survey serve as a baseline for the existing quality culture in our organisation and will be used to improve on those aspects that need it. Following the advice of the QA Board, the survey will be repeated in two years.

The survey results were also shared with ES staff in the quarterly

meeting held in August 2022 that focussed on quality. This meeting also served to collect input on the values that staff find important for a quality culture. Using the collective feedback, ES defined a vision on quality based on four shared values: collaboration, commitment to continuous improvement, professionalism, and trust.

4
Photo by Natasya Chen on Unsplash

Vision on quality

The quality culture we embrace at ES is based on a strong collaborative spirit with the shared goal of delivering an excellent experience to all learners.

We can be trusted to use our knowledge and expertise to deliver an impactful experience because we care about our diverse community of learners and their lifelong learning development.

Our teams work proactively with a focus on solutions: this approach enables us to improve the quality of our products and services. As an organisation, we foster a culture of learning, both for our learners and for ourselves; in this way we continue to grow and improve.

“Committed to excellence, because we care”
Photo by Jonny Caspari on Unsplash

Stakeholders and roles

Having a good quality culture means that quality is in everyone’s remit. However, the final responsibility regarding the quality of education is shared amongst different persons, teams

STAKEHOLDER

Executive Director

Director of ES Education

Quality Assurance Officer

Manager Education Support

Manager Business Development

Policy Advisor

Quality Assurance Board Management Team

Education Management Team

Academic Portfolio Directors

Faculty Coordinators

Administrative Officer Course Team

Portfolio & Product Managers

Learning Developers

Learning Technology Team

Moderation Staff

Educational Data Analyst

Communication Manager

Administration Team

6

and Boards. Some of these roles are legally prescribed in the ES Management Regulations (ES Beheersreglement).

ROLE IN QUALITY

Quality of ES strategy, innovation and operations.

Quality of ES education.

ES QA processes (process manager).

Quality of educational support and QA cycles (process owner).

Quality of marketing, enrolment processes and reporting cycles (process owner).

Regulations and all ES policy documents.

Quality of examinations and final assessments and thus certificates.

Advisory role on quality processes and quality of educational portfolio.

QA of the education offered.

Quality of education. Advisory role on QA and portfolio development of ES education.

Quality of portfolio for each ES educational theme.

Course evaluations and improvement plans.

Reporting and annual improvement plans (process manager).

Quality of course content and operations. Course team includes lecturers, teaching assistants and course coordinator (if appointed).

Quality of portfolio development and alignment. Portfolio and product management and review.

Course design and didactic quality.

Quality of technology.

Quality of course delivery.

Platform and learners’ data for course evaluations.

Sharing best practices on continuing education and lifelong learning.

Quality of administrative processes and systems.

7

Quality standards and indicators

In 2022 for the first time, ES developed and implemented a set of Quality Standards & Indicators to successfully benchmark all its quality activities. Currently, in the Netherlands there are no agreed standards for continuing education at the national level. However, to align with campus education, ES builds upon the existing quality standards of the Accreditation Organisation of the Netherlands and Flanders (NVAO) and makes these fit for purpose for online education courses and programmes1.

The ES Quality Standards & Indicators apply to three levels: Institutional, Portfolio, and Course. This three-level approach allows us to better organize and execute our QA System to guarantee a closer alignment with the existing quality culture, activities, and policies in place at TU Delft. For every standard and indicator, there are key stakeholders responsible or accountable for meeting the indicators and, ultimately, the quality standard. A ‘responsible’ stakeholder is the one who carries out the work individually or with a team to complete a task.

1 The quality standards are informed by other internationally renowned standards e.g., Online Course Quality Review Rubric (State University of New York;) E-Excellence Benchmark (European Association of Distance Teaching Universities); Quality Matters.

An ‘accountable’ stakeholder is the ‘owner’ of the work that they must sign off (approve) and is ultimately responsible for meeting the indicator; there are no financial nor legal implications associated with this role, with the exception of members of the Management Team as prescribed in the ES Management Regulations (ES Beheersreglement). There can only be one accountable person specified for each task or deliverable, and at least one responsible person, although others can be delegated to assist in the work required.

A comprehensive description, with measurements for each indicator and responsible stakeholders, can be found in Annex 1: Table for the operationalisation of the Quality Standards & Indicators. The three tables in the next section summarise those in Annex 1, giving for each standard the responsible and accountable stakeholders at institutional, portfolio and course level. Annex 1 provides the full description of the measurements and the stakeholders for each of the indicators; these will be evaluated and any necessary changes will be updated in future versions of this handbook.

8
Photo by Diana Polekhina on Unsplash

Institutional quality standards

The Institutional quality standards encompass core elements such as ES’ institutional philosophy, policies, and impact strategies. They ensure that as an organisation, we are successfully embedded within the university and a key player in the larger ecosystem of continuing education and lifelong learning. The table shows the five institutional standards (all associated indicators are included in Annex 1).

STANDARD

Responsible Accountable PHILOSPHY

ES’ mission is to educate the world and enhance the quality of TU Delft campus and continuing education. There is a corresponding strategic plan to support its educational philosophy.

IMPLEMENTATION

ES’ Educational Philosophy is realised in an effective, inclusive and learner-centred manner that is supported by a high-quality culture.

• Management Team

• Executive Director

• Management Team

• Policy Advisor

• QA Officer

• Management Team

EVALUATION AND MONITORING

ES systematically evaluates whether the intended policy objectives relating to educational quality are achieved.

DEVELOPMENT

ES focusses on the continuous improvement of its education, innovation and research.

• Policy Advisor

• QA Officer

• QA Board

• Process Managers

• Administrative Officer

• Management Team

• Administrative Officer

• QA Officer

• Education Support Team

• Communication Manager

IMPACT

Continuing education impacts TU Delft and beyond.

• Management Team

• Executive Director

• Director ES Education

• Executive Director

• Director ES Education

• Executive Director

• Chair of QA Board

• Manager Education Support

• Manager

Business Development

• Executive Director

• Director ES Education

9

Portfolio quality standards

The Portfolio Quality Standards reflect our strategy to provide courses and programs on specific themes, which are based on the expertise of the University, can attract the right target audience, and have a beneficial impact on TU Delft’s education and beyond. Therefore, our educational portfolio themes are of market and societal relevance, and we deliver online education in different formats to respond to the individual needs of lifelong learners and professionals. There are four Portfolio Quality Standards: the summary below identifies the relevant actors.

STANDARD Responsible Accountable

PORTFOLIO STRATEGY

The ES’ portfolio strategy should encompass themes based on the expertise of the University and on learners’ demand.

EVALUATION AND MONITORING

The ES portfolio is systematically evaluated for educational coherency and relevance

DEVELOPMENT, ACCESS AND REACH

The ES portfolio is broad and serving a diverse target audience across the globe.

• Academic Portfolio Directors

• Manager Business Development

• Portfolio & Product Managers

• Manager Business Development

• Portfolio & Product Managers

• Academic Portfolio Directors

• Portfolio & Product Managers

• Learning Developers

• Marketers

• Academic Portfolio Directors

IMPACT

The ES portfolio impacts TU Delft and beyond

• Manager Business Development

• Manager Education Support

• Portfolio & Product Managers

• Faculty Coordinators

• Director ES Education

• Director ES Education

• Director ES Education

• Academic Portfolio Directors

• Executive Director

• Director ES Education

• Academic Portfolio Directors

10

Course quality standards

The Course Quality Standards highlight the pedagogical model used in our unique course design: the cutting-edge course curriculum, learning activities, and technologies that create an optimal learning experience whilst fulfilling the relevant learning objectives.

STANDARD RESPONSIBLE ACCOUNTABLE

COURSE DESIGN AND DEVELOPMENT

Courses are designed and developed to provide a high-quality, stimulating, and engaging learning experience.

COURSE CURRICULUM

Courses enable learners to integrate knowledge and professional skills in the fields of Science, Engineering, and Design.

ASSESSMENT

Courses have an adequate assessment system in place according to the type of course.

COURSE DELIVERY

Course delivery methods encourage learners’ engagement and active learning through consistent moderation and feedback.

MONITORING AND EVALUATION

Courses are systematically evaluated for their educational quality and relevance.

TECHNOLOGY

Technology adequately supports the courses’ learning outcomes.

• Responsible Lecturers

• Course Teams

• Learning Developers Coordinator

• Learning Developers

• Learning Technology Team

• Responsible Lecturers

• Course Teams

• Faculty Coordinators

• Learning Developers

• Course Teams

• QA Officer

• Policy Advisor

• Faculty Coordinators

• Learning Experience Coordinator

• Course Teams

• Course Moderators

• Learning Developers

• Policy Advisor

• Data Analyst

• Course Teams

• Learning Developers

• Portfolio & Product Managers

• QA Officer

• Data Analyst

• Course Teams

• Learning Developers

• Learning Technology Team

• Learning Experience Coordinator

• Policy Advisor

• Portfolio & Product Managers

• Manager Education Support

• Process Manager Open Education

• Responsible Lecturers

• Manager Business Development

• Manager Education Support

• Responsible Lecturers

• Director ES Education

• Chair QA Board

• Responsible Lecturers

• Manager Business Development

• Manager Education Support

• Learning Experience Coordinator

• Manager Education Support

• Chair QA Board

• Executive Director

• Manager Education Support

11

Quality Assurance cycles

Following the approach described in the TU Delft Education Quality Assurance Plan, ES also bases its QA System12 on the Plan–Do–Check–Act (PDCA) cycle as shown in the figure on this page. The system provides a clear and continuous QA process for the ES organisation, its portfolio, and the individual courses – with the only exception of the online academic courses, as they are already part of the QA process for campus education. Their evaluation, therefore, sits within the faculty offering them.

In practice, the documentation ES uses in applying the model may differ from the standardised one used by faculty as it regards the programs, delivery methods, learners and legal status of the particular education offered, which diverges from that of regular campus education.

We apply the PDCA cycle to each of our three defined levels (Institutional, Portfolio, Course) to ensure a better alignment of quality standards and indicators.

It means that ES can organise the QA cycles for each level more effectively, taking into consideration the involved stakeholders, specific deliverables, and relevant time frames.

The iteration of the steps in the cycle guarantees the continuous improvement of our institution, products and services; its successful execution also rests on having in place appropriate QA governance, including comprehensive descriptions of who is responsible for carrying out which tasks and following which timeline.

2 A PDCA circle or Deming circle is a set of four successive, mutually connected activities with which the quality of an object can be systematically improved.

12
Photo by Cats Coming on Pexels
CHECK
ACT PLAN
Continuous improvement
DO

Institutional QA cycle

Our QA governance at Institutional level is realized in consultation with the ES Management Team, Strategy Board, Educational Advisory Board, and QA Board. The main reports are shared with the University’s Executive Board (EB) and the faculties. The PDCA cycle is implemented as follows:

Define the ES Strategic Plan, Annual Planning and Yearly Budget – taking into consideration the ES Governance Structure and its Educational Philosophy –in order to accurately define ES’ objectives.

Ensure transparency of findings and evaluations by disseminating these through the ES quarterly and annual reports for the faculties and the EB, as well as via the annual evaluation and the 5-year institutional evaluation, as included in the ES QA and reporting cycle. Additionally, analyse and evaluate the impact of online education on campus education, so that recommendations can guide an improved integration of online offerings into campus education and beyond.

Collect highly aggregated learner and platform data, and revenue streams, to measure results against the ES’ objectives defined in the Strategic Plan. Data is also collected by surveying instructors on the impact of online courses on their campus teaching.

Wherever possible implement the recommendations for continuous improvement of ES’ education and services, which result from the ES evaluation processes: these will be included in a summary section in the ES QA Annual Report.

13
PLAN DO CHECK ACT Multi-year strategic plan Annual planning Budget Governance structure ES Philosophy Recomendations for Continuous Improvement Institutional evaluation Learning Platform Data Instructors surveys Data Learners surveys Data Annual reports Campus use and impact report Annual evaluation QA board annual report Quarterly Update and Reports Institutional QA Cycle PLAN DO ACT CHECK

Portfolio QA cycle

The educational content of the ES courses is guaranteed by faculty members; whereas the processes that facilitate education, including the QA process, are under the responsibility of ES. Therefore, the Portfolio QA Cycle can only be realized in close consultation between these two parties.

PLAN

Ensure the development of relevant strategies for the educational themes of the ES portfolio, including their alignment and coherency, and the existence of an appropriate management and governance structure to allow the realization of the strategy for each educational theme. The Director of ES Education is primarily accountable for the development of a portfolio of courses and programmes offered by ES that meets academic standards societal demand, market relevance, and the needs of external learners. The Academic Portfolio Directors (APDs) advise on the direction and opportunities within their portfolio theme in close collaboration with TU Delft faculties and the ES Portfolio & Product Managers (PPMs). The Faculty Coordinators are key in safeguarding alignment with faculty strategy, while the APDs guarantee portfolio alignment with the university-wide strategic themes.

DO CHECK ACT

Make recommendations regarding the ES portfolio offering and overall strategy based on the assessments and consultation on the abovementioned information and data analysis. The Portfolio Review will indicate which categories of courses require a more in-depth review and which should be prioritized. Strategic recommendations may address portfolio development, business and education strategies, and the availability of data and information management.

Conduct a yearly Portfolio Review covering the entire ES offering; this is executed by the PPMs who assess information and data about these five perspectives:

• pedagogical quality

• business performance

• course relevance for the portfolio theme

• relevance for faculty

• learner feedback.

Faculty Coordinators are crucial in providing data on the impact of courses for campus education and their relevance for faculty. In this process, Learning Developers provide input concerning the didactic aspects of all courses. The Portfolio & Marketing Team provide input on the business performance against industry benchmarks. APDs check whether courses align with the portfolio themes (fit with portfolio). The Educational Data Analysis Team provides learner feedback through the elaboration of aggregated survey results. Put into practice the recommendations from the Portfolio Review, including executing improvement plans for the revision and update of individual courses, or for (dis) continuing them. Strategy sessions are planned regularly to review and update the portfolio strategies for relevance and alignment; the ES Management Team will consider recommendations regarding policy and procedures as well as information management. Included in such sessions are also the recommendations from the analysis on the impact of online education on campus education to facilitate further integration between the two.

14

Portfolio Themes Strategy and Alignment

LLL Strategy Management structure

LLL Strategy Review

Campus Impact Report

Portfolio Strategy Report

Portfolio Review Report

Campus Impact Evaluation

Portfolio Review Recommendations

Portfolio Strategy Review

LLL Projects Evaluations

Market Research

Portfolio Review

Learners’ and Platform Data

15
DO
PLAN
ACT CHECK Portfolio QA Cycle

Course QA cycle

The Course QA Cycle comes to life through the systematic iteration of the four PDCA steps.

Course Improvement Plan

Course Development Roadmap Pedagogical Model

Accessibility and Usability Guideliness

Trainings and Course Creation Resources

Platform Data

Learners’ Survey Data

Course

Moderation Feedback Log

Assesment Results

Complaints Log

Course Evaluation Report/Survey

Course Evaluation Meeting

Feedback from Instructors and Learning Developers

16
PLAN DO ACT CHECK Course QA Cycle

Ensure that our approach to course and curriculum design and development is carried out across our entire portfolio using pertinent course development processes, training and resources. Our approach follows our unique pedagogical model, based on the Online Learning Experience Principles, a learnercentred approach, and open practices. Learning Developers are responsible for providing Course Teams with the necessary guidance to adhere to the Quality Standards by following quality control checks throughout the course creation process. Staff need to have access to resources and be able to complete the necessary training before they can start any moderation activity as part of the course delivery, and in compliance with the relevant quality standards. Moderators monitor activity in the courses and resolve or escalate any issues that may arise and hinder learners from completing the course.

PLAN CHECK

Conduct a course evaluation at the conclusion of every 1st and 2nd course runs for continuous improvement. After the first two runs, courses will be evaluated at regular intervals of three runs (see figure on the next page) taking into considerations the five perspectives identified in the Portfolio Review.

Participants involved in the evaluations are:

• Course Team

• Faculty Coordinator

• Marketer, Learning Developer, Portfolio & Product Manager.

The evaluations take the following input into account:

• course evaluation report (for MOOCs 1st runs and Programmes) or learner survey results for 2nd runs, ProfEds, and further runs

• moderation feedback

• learning developer feedback

• instructor feedback.

Part E contains a complete list of these evaluation instruments.

Utilise a whole range of learner data gathered through the completion of pre and post surveys, aggregated platform data and activity, course assessment results, and learner feedback. The Data Analyst in the Learning Technology Team collects and analyses data to inform the full evaluation that takes place once the course is concluded.

Produce and share a course improvement plan that includes considerations based on the evaluation. The improvement plan may refer, as necessary, to the:

• didactic quality of the course

• value proposition/About Page, promotional activities

• course delivery methods (instructor-paced vs self-paced, moderation by Learning Experience Team, trainings necessary for TA)

• list of actions for improving the course before the next run.

17
DO
ACT

Evaluation cycle of individual courses

Evaluations will be planned at regular intervals of three runs. This will allow to systematically monitor the course quality and learners’ satisfaction.

Course evaluation process

The teams in the figure below are involved in the various phases of the evaluation process and are responsible for defining and implementing the course improvements.

18 Evaluation team Course team Share evaluation report or survey results Collect learner and platform data Provide moderation feedback Course Team, Learning Developer, Product & Portfolio Manager, Marketer, Faculty Coordinator Define necessary improvements Evaluate course Make adaptations Course team Course starts
course evaluation course evaluation course evaluation course evaluation evaluation upon request evaluation upon request evaluation upon request evaluation upon request Run 1 Run 2 Run 5 Run 6 Run 7 Run 8 Regular course evaluation frequency Evaluation in exceptional cases only Run 3 Run 4

Reporting cycles and structure

The table below provides an overview of all reporting activities organized by timeline, divided by type of activity and expected reporting material, and by the responsible teams or Boards. Wherever possible, ES staff will have access to transparent and timely reporting materials.

Activity Report Responsibilities

ES Strategic Plan divided in:

ES Multi-Year Plan

(MYP) - every 6 years

ES Management Team

ES Strategy Board

Multi year

Quality Standards & Indicators review

Annual roadshows

Annual planning

Annual evaluation

Mid-term updateevery 3 years

ES institutional evaluation – every 5 years

ES Management Team

QA Officer

ES Management Team

ES Management Team

ES Management Team

ES Annual Report

ES Annual Faculty Report

ES Management Team

ES Management Team

Annual

ES Strategy Board Meeting

Quarterly update

ES QA Board Annual Report

ES Education Advisory Board Portfolio Review report

ES Education Advisory Board Portfolio Strategy report

Quality Assurance Board

Education Advisory Board

Education Advisory Board

ES Strategy Board

ES Executive Director

ES Quarterly Report to faculties

Quarterly

Education Advisory Board meeting (bi-annual)

QA Board meeting

ES Management Team

Education Advisory Board

QA Board

19

Fraud, Code of Conduct, complaints

To ensure learners have a good learning experience, the ES Learning and Assessment Regulations (LEARN) have been put into place, and we expect all learners to comply with the TU Delft Code of Conduct. An important part of these regulations concerns fraud and how to deal with it when it occurs.

According to the ES LEARN, fraud is committed when it is not possible to accurately assess the knowledge of learners. Fraud includes the use of forbidden materials during assessments, copies of the assessment questions or assignments beforehand, or sharing assessment questions with others. It also includes committing plagiarism (such as copying the work of others and presenting it as own work, either on purpose or due to carelessness and/or incomplete referencing).

In cases of fraud and/or plagiarism, the necessary steps to be taken, and who are the stakeholders involved, varies. The process for each type of course is explained below. Note that there is no ES process in the case of Online Academic Courses as these are a faculty responsibility: if fraud is suspected or detected during the final assessment, the Board of Examiners of the faculty offering the course can decide on the applicable sanctions. In such cases, the responsible lecturer should reach out to the relevant Secretary of the Board of Examiners.

20
Photo by Tim Gouw on Unsplash

Fraud and code of conduct in MOOCs

The main lecturer is responsible for dealing with individual cases. In case of large-scale fraud (relative to the number of course participants) or a breach of the TU Delft Code of Conduct, Course Teams should contact the ES QA Board via email (QABoard.ES@tudelft.nl). The Board can advise the Course Team to allow a re-take of the assessment/assignment, or to stop/revoke the issuance of the course certificate. The QA Board will react in writing within five days from the time the Course Team has reached out for advice. Should the Course Team be unsure whether they face a case of large-scale fraud, they should reach out to the QA Board for advice. The figure below describes the full process.

21
Individual learner commits fraud in MOOC Course Team is consulted Re-take is allowed Learner is informed and certificate issuance is stopped or revoked Learner is informed and may do a retake Learner completes the course Learner does not complete the course Large-scale fraud is commited MOOC Course Team is informed QA Board advises Course Team Yes No

Fraud and Code of Conduct in ProfEd courses and programmes

The Course Team/lecturer should contact the ES QA Board via email (QABoard. ES@tudelft.nl) regarding cases of fraud or breach in the Code of Conduct in ProfEd courses as indicated in the Learning and Assessment Regulations

The Board can decide on the applicable sanctions (depending on the seriousness of the case) and will communicate its decision within five working days to the Course Team and the learner(s).

22 Fraud or breach of Code of Conduct is detected in ProfEd Course Team is consulted Decision is taken that results are invalid and a retake is allowed Learner is informed and certificate issuance is withdrawn or revoked Learner(s) informed and allowed a retake Course is completed Course is not completed QA Board advises Course Team Yes No

Handling complaints

The complaints procedure is an important QA aspect. It helps to identify the elements that can hamper the quality of education. ES follows an internal complaints process when dealing with complaints – this is described in the figure at the bottom of the page.

All learners are made aware of the TU Delft Terms & Conditions for Contract Education at the time of enrolment. Article 13 of these Terms & Conditions gives information on the procedure that applies if learners wish to express a complaint about one of our products, the ES organization, or a ES staff member. The complaint should be submitted as soon as possible, and no later than four weeks after the completion of the course (Article 13.1). Learners must do so by stating in writing the reasons for their complaint, providing additional information such as the name and address of the complainant, description of the conduct about which the complaint is being made, and the name of the related course (Article 13.2).

The Administration Team receives complaints from learners enrolled in ProfEd courses via the functional mailbox or via the dedicated feedback form on the TU Delft Online Learning website. The Administration Team determines which department should handle the complaint or escalates to the relevant Course Team. The Administration Team also advises the complainant about the expected response time.

The edX Support Team receives complaints from learners enrolled in a MOOC via the dedicated online form available on third party platforms. Depending on the nature of the complaint, the Learning Experience Team in charge of moderation will either directly address the complaint or escalate to the Course Team. They will also record the frequency and type of complaints in a feedback form; such information will be used to inform future course evaluation. Additionally, Learning Developers will record complaints that required an escalation to the Course Team in a dedicated repository, which is shared with the QA Officer.

In case the learner is not satisfied with the proposed solution, they may contact the Central Complaints Desk, which will escalate their complaint to the ES Executive Director.

Should the nature of the complaint be a dispute on assessment results, Course Teams unable to resolve this can escalate the case to the ES QA Board for a binding settlement.

On a quarterly basis, an overview of all complaints is shared with the ES Director of Education and the QA Board, who provide advice on the actions to take (if any) to improve ES products and services.

23 Learner(s) expresses complaint ES staff receives complaint Complaint can be resolved by staff Complaint is escalated to Course Team and the complaint is resolved Decision is communicated to learner Complaint is closed, case is recorded in complaints repository Complaint is closed, case is recorded in complaints repository Yes No Learner escalates complaint to Central Complaint Desk Central Complaint Desk escalates to ES Executive Director and complaint is resolved No No Yes

Evaluation instruments

Several instruments are devised and used to measure to what extent our quality criteria are met; for example these could be surveys or analysis of platform data on learners’ completion rates and their level of interaction within the course. The tables below describe the instruments currently in use.

Enrolments analysis

What Information on number, type of learners (verified vs audit) and demographics. Information is based on platform data

Target group Learners

Function Provides information on the target audience

When At the conclusion of every course run

Feedback

Shared and discussed with Course Teams during course evaluation for 1st runs.

For following runs information is made available to Course Teams

Pass rate analysis

What Analysis of assignment attempts and pass rate of the course. Information is based on platform data

Target group Learners

Function Provides information on the subject level and student learning

When At the conclusion of every course run

Feedback Shared with Course Teams during course evaluation

Course engagement analysis

What Analysis of learners’ activity and engagement with the course (material). Information is based on platform data.

Target group Learners

Function Provides information on the type and frequency of the learners’ engagement with the course material

When At the conclusion of every course

Feedback Shared with Course Teams during course evaluation

24
Photo by Ali Kazal on Unsplash

Pre-survey

What Survey received by all learners

Target group Learners

Function Provides learner demographics information, educational and professional background, motivation for studying

When At the start of each course

Feedback

Post-survey

Shared with Course Teams during course evaluation. For subsequent runs information is made available to Course Teams

What Survey received by all learners

Target group Learners

Function Provides information on the general quality of and satisfaction with the course

When At the end of each course

Feedback

Shared with Course Teams during course evaluation for 1st runs. For subsequent runs information is made available to Course Teams

Quality and workplace culture survey

What Survey about the perceived quality culture within the organization and workplace satisfaction

Target group

ES staff and affiliated members of the ES (Academic Portfolio Directors, Faculty Coordinators)

Function Provides information on the organisational culture with regards to quality and the degree of work satisfaction that ES employees experience

When Every 2 years

Shared with ES Management Team and presented at ES quarterly meetings

Feedback

Campus impact survey

What Survey on the reuse and impact of online education and material into campus education

Target group Lecturers

Function Provides information on the extent and impact that online courses have on campus education and beyond

When Every 2 years

Shared with whole ES

Feedback

25

Conclusions

This first edition of the ES QA Handbook represents a significant step towards documenting an improved QA system. In future, we will carry out concrete checks and plan evaluations to measure the efficacy of the described PDCA cycle at all levels. Given the very recent publication of the ES Quality Standards & Indicators, and implementation of the QA cycle at Institutional and Portfolio level, adjustments are to be expected.

In a similar manner, we expect updates and amendments to this handbook and its annexes to be made after one year, including a review of the measurements and stakeholders defined in the operationalisation of the standards (Annex 1). Changes will result from ongoing consultations with several stakeholders and from their feedback on the implementation of the standards and quality cycles, and including the handling of complaints and fraud cases.

26
Photo by Artur Aldyrkhanov on Unsplash

After the first year, the QA Handbook will be updated every three years and be, wherever possible, aligned to the TU Delft Education Quality Handbook (Onderwijskwaliteit Handboek).

It is the intention to keep the conversation around the quality of education ongoing and place this topic at the centre of our activities. Therefore, anticipating changes to the way we plan, organize and execute our QA is a positive sign that we aim to continuously improve our activities and the quality of education and services we offer our learners.

27
We are committed to excellence, because we care.

Annex 1: Table for the operationalisation of the Quality Standards & Indicators

Institutional Standards

PHILOSOPHY

ES mission is to educate the world and enhance the quality of TU Delft campus and continuing education. There is a corresponding strategic plan to support its educational philosophy.

Standard Measurements

Responsible Accountable

1.1

ES EDUCATIONAL PHILOSOPHY IS IN PLACE AND ALIGNED WITH THE PHILOSOPHY OF TU DELFT

ES Educational Philosophy defined and communicated.

Yearly review as part of QA cycle

Manager Education Support Director ES Education

1.2

ES STRATEGIC PLAN IS IN PLACE TO ACHIEVE ITS MISSION AND EDUCATIONAL PHILOSOPHY

ES Yearly Strategic Plan. Strategic targets and timelines are shared with ES staff including at a dedicated Quarterly meeting. The input is used to inform the following year Strategic Plan.

1.3

OPEN EDUCATION IS A CORE ASPECT OF THE ES PHILOSOPHY

Included in the ES Educational Philosophy.

ES follows TU Delft OER Policy. OCW policy. OCW publication process.

Management Team Executive Director

Management Team Executive Director

1.4

ES COMMUNICATES ITS VISION AND STRATEGIC PLAN TO ALL STAKEHOLDERS

Strategic targets and timelines are shared with ES staff during a dedicated quarterly meeting. The input is used to inform the following year Strategic Plan.

1.5

ES CLEAR GOVERNANCE STRUCTURE UNDERPINS ITS PHILOSOPHY

1.6 ES HAS THE REQUIRED RESOURCES TO SUPPORT ITS OPERATIONS.

Internal Audit Evaluation for adequate governance structure. Evaluation and feedback on whether ES governance really facilitates the achievement of its goals.

Yearly budget.

Annual Planning including personnel and systems requirements.

Invite HR advisor to MT meetings

Management Team Executive Director

Executive Director Director ES Education

Executive Director

Management Team Executive Director

28

IMPLEMENTATION

The ES educational philosophy is realised in an effective, inclusive and learnercentred manner that is supported by a high-quality culture.

Standard Measurements Responsible Accountable

Online Learning Experience Principles guide course design.

2.1

THE ES PHILOSOPHY IS TRANSLATED INTO POLICY ACTIONS

2.2 ES QUALITY ASSURANCE SYSTEM IS IN PLACE

Range of policies in place:

• LEARN regulations

• Privacy statement

• T&Cs

• Licensing policy.

QA handbook.

ES Policy Advisor Management Team

Director ES Education

Institutional QA Cycle in place. ES QA Officer Director ES Education

2.3

PROCESSES ARE IN PLACE TO SUPPORT THE SYSTEMATIC DELIVERY OF ES EDUCATION AND SERVICES

Directory with processes.

Manager Education Support Manager Business Development

Executive Director

2.4

PROCESSES ARE IN PLACE TO COLLECT AND COLLATE DATA AND LEARNERS’ FEEDBACK TO IMPROVE EDUCATIONAL SERVICES

Course evaluation process. Portfolio review process. Research process.

Manager Education Support Manager Business Development

Executive Director

2.5

STAKEHOLDERS KNOW ABOUT AND ADHERE TO THE ES QUALITY CULTURE

ES quality values and definition created and shared.

Quality survey data and feedback.

ES quarterly meeting dedicated to quality culture.

Management Team Executive Director

29

EVALUATION AND MONITORING ES systematically evaluates whether the intended policy objectives relating to educational quality are achieved.

Standard Measurements

Responsible Accountable

3.1

THE ES STRATEGIC PLAN IS REVIEWED AND PERIODICALLY UPDATED FOR ITS CONTINUING RELEVANCE

ES Strategic Plan review process

Management Team

Executive Director

3.2

QUALITY METRICS ARE DEFINED AND REVIEWED ON A FREQUENT BASIS

Quality Standards & Indicators review process.

QA Officer

QA Board

Policy Advisor

Director ES Education

3.3

APPROPRIATE MEASUREMENT AND EVALUATION ACTIVITIES ARE INTEGRAL PART OF THE ES PROCESSES

Measurements used: learner surveys and other stakeholders, as needed. Dashboards evaluation activities:

• ES Annual Report to faculties and EB

• Annual evaluation (EB and ES-MT)

• ES Institutional evaluation

Management Team

Executive Director

3.4

THE CONTINUOUS IMPROVEMENT OF EDUCATION AND SERVICES IS ONE OF THE CONCRETE OUTCOMES OF THE ES EVALUATION PROCESSES

Continuous improvement section included in QA Annual Report.

Process Managers

Director ES Education Executive Director

3.5

EVALUATION RESULTS ARE PUBLISHED AND ACCESSIBLE TO RELEVANT STAKEHOLDERS.

ES reporting structure and documentation (reports).

Administrative Officer

Executive Director

30

DEVELOPMENT

ES focusses on the continuous improvement of its education, innovation and research.

Standard Measurements

Responsible Accountable

4.1

RESULTS OF THE ES QUALITY ASSURANCE ACTIVITIES ARE DISCUSSED ON A CYCLICAL BASIS AND AN IMPROVEMENT PLAN IS PUT IN PLACE

QA Board Annual Report.

QA Board Secretary Administrative Officer

Chair of QA Board Executive Director

4.2

BUDGET IS AVAILABLE FOR THE CONTINUING DEVELOPMENT OF ES STAFF AND FOR THE CONTINUOUS IMPROVEMENT OF TECHNOLOGIES AND SERVICES

Budget allocated for professional development, technologies and services.

Management Team

Executive Director

4.3

BEST PRACTICE ON CONTINUING EDUCATION AND LIFELONG LEARNING IS CAPTURED AND SHARED

Resources are published on ES website/Hub. Recurring item on all teams’ meeting agendas.

Communication Manager Education Support Team

Manager Education Support Manager Business Development

31

IMPACT

Continuing education impacts TU Delft and beyond.

Standard Measurements Responsible Accountable

5.1

ES CONTRIBUTES TO IMPROVEMENTS IN CAMPUS EDUCATION AND VICE VERSA

Report on impact on campus survey. Webinars and workshops dedicated to campus teaching staff.

Management Team Executive Director

5.2

ES CONTRIBUTES TO THE CONTINUOUS DEVELOPMENT OF THE LEARNER

Learners survey and feedback.

Management Team

Executive Director Director ES Education

5.3

ES ANNUAL INNOVATION AND RESEARCH BUDGET IS USED TO IMPROVE CONTINUING AND CAMPUS EDUCATION

ES budget allocation for innovation and research is in place

5.4

ES CONTRIBUTES TO THE UN SDGS

Mapping between Portfolios and UN SDGs.

Manager Education Support Manager Business Development

Executive Director

Education Management Team ES Education Director

5.5

ES ESTABLISHES A STRONG NETWORK WITHIN INDUSTRY AS A PARTNER AND SOUNDING BOARD FOR CONTINUING EDUCATION

Education Advisory Board includes industry professionals.

Manager Business Development Executive Director

5.6

ES BECOMES A SUSTAINABLE ORGANISATION AND IS EMBEDDED WITHIN THE UNIVERSITY AND IN THE CONTINUING EDUCATION/ LIFELONG LEARNING ECO SYSTEM AND BEYOND.

ES has a permanent status within TU Delft . Yearly budget dedicated to ES.

ES has a LLL strategy.

Management Team Executive Director

32

Portfolio Standards

PORTFOLIO STRATEGY

The ES portfolio strategy should encompass themes based on the expertise of the University and on learners’ demand.

1.1

THE ES PORTFOLIO THEMES ALIGN WITH THE UNIVERSITY’S STRATEGIC THEMES CONTRIBUTING TO FINDING SOLUTIONS TO SOCIETAL AND MARKET CHALLENGES

Checkbox over strategic alignment in Portfolio review.

Academic Portfolio Directors

Portfolio & Product Manager

1.2

ES HAS A PORTFOLIO STRATEGY FOR EACH THEME WITH A RESPONSIBLE PERSON (ACADEMIC PORTFOLIO DIRECTOR) IN PLACE

1.3

EACH PORTFOLIO THEME STRATEGY ADDRESSES COHERENCY AND ALIGNMENT IN ITS PORTFOLIO OFFERING

Portfolio strategy. Portfolio review process explores both angels:

• academic

• market relevance. Internal audit evaluation.

Portfolio review process takes into consideration alignment and coherency.

Academic Portfolio Directors

Portfolio & Product Manager

Academic Portfolio Directors

Portfolio & Product Manager

Director ES Education

Director ES Education

Director ES Education

1.4

ES HAS A CLEAR MANAGEMENT STRUCTURE THAT ENABLES IT TO SUPPORT ITS PORTFOLIO STRATEGIES

Internal audit evaluation addresses adequate management structure.

Manager Business Development Director ES Education

1.5

ES ALLOCATES SUFFICIENT RESOURCES TO EFFECTIVELY SUPPORT THE PORTFOLIO STRATEGY

Budget (multi-year plan and yearly).

Internal audit evaluation. Manager Business Development Director ES Education

33
Standard Measurements Responsible Accountable

EVALUATION AND MONITORING

The ES Portfolio is systematically evaluated for educational coherency and relevance.

Portfolio & Product Manager

2.1

THE ES PORTFOLIO STRATEGY IS REVIEWED AND PERIODICALLY UPDATED FOR ITS CONTINUING RELEVANCE

Portfolio strategy review process.

Manager Business Development

Academic Portfolio Directors

Director ES Education

2.2

LEARNERS’ ANALYTICS AND MARKET INFORMATION ARE USED TO IMPROVE THE PORTFOLIO STRATEGY

Market research is regularly conducted. Portfolio review.

Portfolio recommendations from edX.

Manager Business Development Director ES Education

2.3

APPROPRIATE MEASUREMENT, EVALUATION PROCESSES AND ACTIVITIES ARE EMBEDDED AT PORTFOLIO LEVEL.

Portfolio review. Portfolio strategy review.

DEVELOPMENT, ACCESS AND REACH

Portfolio & Product Manager

Manager Business Development

Director ES Education

The ES Portfolio is broad and serving a diverse target audience across the globe.

3.1

THE ES PORTFOLIO PROVIDES UP-TO-DATE AND STATE-OF-THE-ART EDUCATION

Portfolio review process.

Portfolio & Product Manager

Academic Portfolio Directors

3.2

THE ES PORTFOLIO ENGAGES BOTH INTERNAL TU DELFT STUDENTS AND EXTERNAL LEARNERS ENCOURAGING CONTINUING EDUCATION/ LIFELONG LEARNING

Annual Report. Reuse of OER in accredited and continuing education. Input from lecturers gathered in evaluation

3.3

THE ES PORTFOLIO ATTRACTS AND CATERS FOR DIVERSE, YET THE RIGHT KIND OF LEARNERS.

Diversity is taken into account in the course design and promotion.

Net Promoter Score and course grade; learner satisfaction.

Portfolio & Product Manager Director ES Education

Learning Developers

Marketers

Manager Education Support

34
Standard Measurements Responsible Accountable
Standard Measurements Responsible Accountable

IMPACT

The ES portfolio impacts TU Delft and beyond.

4.1

EDUCATION MATERIALS ARE USED FOR CAMPUS EDUCATION (AND VICE VERSA) AND BEYOND TU DELFT

Annual Report. Re-use of OER in accredited and continuing education. Input from lecturers gathered in evaluation.

4.2

THE ES PORTFOLIO CONNECTS TO THE HUMAN CAPITAL AGENDA OF THE UN SDGs

Portfolio Strategy is informed by market and trend analysis. Themes relate to the UN SDGs

Portfolio & Product Manager

Faculty Coordinator

Director ES Education

Portfolio & Product Manager

Academic Portfolio Directors

4.3

ES IS VISIBLE AND ACTIVE IN THE ECOSYSTEM OF CONTINUING EDUCATION/ LIFELONG LEARNING AT NATIONAL AND GLOBAL LEVELS.

ES participates actively in (inter) national LLL activities, especially with regards to the portfolio themes.

ES is promoted and visible in (inter) national platforms.

Manager Education Support

Manager Business Development

Academic Portfolio Directors

Academic Portfolio Directors

35
Measurements Responsible
Standard
Accountable

Course Standards

COURSE DESIGN AND DEVELOPMENT

Courses are designed and developed to provide a high-quality, stimulating, and engaging learning experience.

1.1

EACH COURSE HAS A QUALIFIED COURSE TEAM WITH THE EXPERTISE REQUIRED TO ACHIEVE THE COURSE GOALS

1.2

THE COURSE DEVELOPMENT PROCESS ENSURES COURSE TEAMS ARE PROVIDED WITH RELEVANT RESOURCES

Addressed at kick-off meeting and division of tasks.

In re-runs: Course Team completeness/needs are checked as part of the evaluation.

Roadmap for creating an online course is in place and followed.

We have a process in place to review and update trainings and resources.

Responsible Lecturer

Product Portfolio Manager

Learning Developers Coordinator

Manager

Education Support

1.3

THE ES ONLINE LEARNING EXPERIENCE PRINCIPLES GUIDE THE COURSE DESIGN

Blueprint template reflects OLE principles. Alignment of intentions and results (with respect to OLE) by Course Team at evaluation stage. Reflection included in Evaluation Summary.

1.4

THE LEARNING OBJECTIVES ARE ALIGNED WITH SPECIFIC ASSESSMENTS AND LEARNING ACTIVITIES FOR THE GIVEN TARGET AUDIENCE

Story board follows constructive alignment.

Learning Developer Manager Education Support

Course Team

Responsible Lecturer

1.5

COURSES ARE DEVELOPED BASED ON OPEN PRACTICES, WHERE POSSIBLE

MOOCs are designed and created for reuse, meaning that course material created has an open license. Content re-used (and modified) is available in open access and correctly attributed.

1.6

THE COURSE DESIGN FOLLOWS ACCESSIBILITY AND USABILITY GUIDELINES

Accessibility checklist is used for course development.

Course Team

Learning Developers Learning Technology Team

Process Manager Open Education

Manager Education Support

36
Standard Measurements
Responsible Accountable

COURSE CURRICULUM

Courses enable learners to integrate knowledge and professional skills in the fields of Science, Engineering, and Design.

Standard Measurements

Responsible Accountable

2.1

COURSES FIT WITHIN THE FIELDS OF SCIENCE, ENGINEERING AND DESIGN

Courses that fit in one of the Portfolios themes have priority - tender process check.

Responsible Lecturers

Faculty Coordinators

Manager Business Development

2.2

THE COURSES’ ACTIVITIES, MATERIALS, AND LEARNING EXPERIENCE ENABLE LEARNERS TO ACHIEVE THE INTENDED LEARNING OUTCOMES

Learning outcomes are part of the Story Board together with activities and course material. Pass rate analysis.

Post-survey.

2.3

THE COURSES’ SYLLABUS OUTLINES THE COURSE OVERVIEW, LEARNING OBJECTIVES, STUDY LOAD, AND ASSESSMENT DETAILS SO THAT THE LEARNER KNOWS WHAT TO EXPECT

Information is stated in the Syllabus and course outline.

Course Team

Responsible Lecturers

Course Team Learning Developers

Responsible Lecturers

2.4

COURSES MATCH THE DIVERSE TARGET AUDIENCES IN ACCORDANCE WITH THE ES DIVERSITY AND INCLUSION STATEMENT

Course content is designed for broad accessibility.

Target audience in the About Page.

Post-survey.

Course Team Learning Developers

Manager Education Support

37

ASSESSMENT

Courses have an adequate assessment system in place according to the type of course.

Standard Measurements

Responsible Accountable

3.1

LEARNING OUTCOMES ARE ASSESSED USING A BALANCE OF FORMATIVE AND SUMMATIVE ASSESSMENTS APPROPRIATE TO THE CURRICULUM

Assessment information and type is clearly stated in the Syllabus.

Learning taxonomies are used to determine the right level and types of assessment for the course.

3.2

ASSESSMENTS ARE AUTHENTIC, VARIED AND INCLUSIVE

Review assessments created during Review and Reality Check. Post-survey.

3.3

ASSESSMENT CRITERIA AND RESULTS ARE CLEARLY AND TIMELY COMMUNICATED TO LEARNERS

Rubrics and grading tables in the course are available and comprehensive.

Course Team Responsible Lecturers

Course Team Responsible Lecturers

Course Team Responsible Lecturer

3.4

LEARNERS MAY BE ALLOWED TO DO RETAKES WITH THE APPROVAL OF COURSE TEAMS WHERE APPLICABLE

LEARN and T&Cs in place, which allow retakes.

QA Officer Policy Advisor Director ES Education

3.5

THERE IS A PROCESS IN PLACE TO HANDLE DISPUTES AND FRAUD CASES

LEARN and T&Cs in place. Fraud and complaints process described in QA Handbook. Fraud and complaints repository.

3.6

THE QUALITY ASSURANCE BOARD UPHOLDS THE QUALITY STANDARDS WITH REGARDS TO ASSESSMENT AND CERTIFICATION

Certificates are signed on behalf of the QA Board. Disputes about assessment results and fraud are addressed to the QA Board.

QA Officer Policy Advisor Director ES Education

QA Officer Chair QA Board

38

COURSE DELIVERY

Course delivery methods encourage learners’ engagement and active learning through consistent moderation and feedback.

Standard Measurements

Responsible

Accountable

4.1

COURSE TEAM MEMBERS IN CHARGE OF MODERATION ARE FULLY TRAINED BEFORE STARTING TO MODERATE COURSES

TAs on-boarding guide in place.

Building And Moderating (BAM) an online course training is mandatory for staff before the course starts.

4.2

COURSE MODERATION IS CONSISTENT AND OF GOOD QUALITY

E-moderation guidelines, training and resources are available.

Post-survey.

4.3

EACH COURSE INCLUDES (IN) FORMAL FEEDBACK TO HELP LEARNERS IMPROVE THEIR PERFORMANCE

Moderation guidelines include instructions on how to provide feedback to learners.

Post-survey question to learner about feedback.

LET Coordinator

Faculty Coordinator

Learning Developers

Course Team Course Moderator

Responsible Lecturer

4.4

LEARNERS ARE INFORMED OF AND HELD ACCOUNTABLE FOR ADHERING TO THE ES CODE OF CONDUCT, AND TO THE FORUM AND COLLABORATION GUIDELINES

All learners are required to review T&Cs and Code of Conduct upon enrolment.

Discussion Forum Guidelines, Collaboration Guidelines and Honour Pledge in place and embedded in course shell and syllabus.

Course Team Course Moderator

Responsible Lecturer

LET Coordinator

Responsible Lecturer LET Coordinator

Manager

Policy Advisor

Learning Developers

Business Development Manager Education Support

4.5

COURSE DELIVERY IS EVALUATED THROUGH THE ANALYSIS OF PLATFORM DATA, INPUT FROM LEARNERS ON THEIR EXPERIENCE, AND FEEDBACK FROM THE COURSE TEAM

Learners post-survey.

Platform data analysis of learners engagement

Moderation feedback log/ report or in course hidden section.

Data Analyst

Course Team

Learning Developers

Manager Education Support

39

MONITORING AND EVALUATION

Courses are systematically evaluated for their educational quality and relevance. Standard

Accountable

5.1

THERE IS A QA CYCLE IN PLACE FOR CONTINUOUS IMPROVEMENT

Course evaluation process. QA Officer Manager Education Support

5.2

LEARNERS’ DATA AND COURSE TEAM FEEDBACK CONTRIBUTE TO THE IMPROVEMENT PLAN

Evaluation report and survey results. Course Team feedback Learning developers’ observations. Moderation feedback report.

Data Analyst

Course Team Learning Developers

Manager Education Support

5.3

COURSES ARE REGULARLY REVIEWED AND IMPROVED (INCLUDING IN OPEN COURSEWARE) TO REMAIN RELEVANT FOR LEARNERS

Evaluation process and improvement plan after 1st run and every 3 runs after the 2nd

Portfolio & Product Manager

Manager Education Support

5.4

THE QUALITY ASSURANCE BOARD UPHOLDS THE QUALITY OF EDUCATION BY MONITORING THE COURSE EVALUATION PROCESS

QA Board regularly requests to be provided documentation concerning course evaluation process

QA Officer/ QA Board Secretary

QA Board Chair

40
Measurements Responsible

TECHNOLOGY Technology adequately supports the courses’ learning outcomes.

Standard Measurements

Responsible Accountable

6.1

A VARIETY OF TECHNOLOGIES SUPPORTS THE ACHIEVEMENTS OF THE LEARNING OUTCOMES

Assessed during Review and Reality Check and/or Story Board, includes type and variety of technology

Learning Developers Course Team

Manager Education Support

6.2

COURSE TEAMS ARE SUPPORTED IN THE DEVELOPMENT AND USE OF TECHNOLOGIES

ES has a process in place for the development of new technologies. Trainings and resources available on the use of technologies. edX Support Team is in place.

Learning Technology Team LET Coordinator

Manager Education Support

6.3

LEARNERS ARE ADVISED OF THE MINIMUM TECHNICAL REQUIREMENTS NEEDED TO BE ABLE TO FOLLOW A COURSE

System requirements information and general restrictions for all courses are indicated in ES website. Course-specific additional technical requirements are indicated in the About Page.

Learning Technology Team Course Team

Manager Education Support

6.4

LEARNERS ARE INFORMED ON HOW THEIR DATA AND PRIVACY ARE PROTECTED

ES and 3rd party platform Privacy Statement, and electronic consent in place. Policy Advisor Executive Director

41

Annex 2: Example of an evaluation instrument: Post-survey

Q196 The course has ended. We would like to thank you for your participation and contribution to its success! We are very interested to know about your experience during this course. With your feedback we can improve our courses, and enable more people to access, enjoy, and complete them.

This survey should only take 5-10 minutes to complete. Participation is completely voluntary and all information will be treated confidentially.

The data will be reported only in aggregate form and no individual will be identified. Your answers here, combined with your edX course data, will be used for evaluation purposes, and potentially in scientific research related to online education, which is why we need your permission.

Please feel free to contact us if you have specific questions about this survey, by sending an email to eduanalytics@tudelft.nl.

Thank you very much for helping us improve our courses!

Q198 This survey is in accordance with the edX Privacy Policy.

Q1edX ELECTRONIC CONSENT: Please select your choice below. Selecting the “agree” button below indicates that:

You have read and understand the above information. You voluntarily agree to participate. You give us permission to use your input for scientific purposes. You are at least 16 years of age.

Agree (1)

Disagree (0)

Q1 Based on your entire experience with TU Delft Online Learning, on a scale from 0-10, how likely are you to recommend us to a friend or colleague?

Q2 Please help us by explaining your rating.

Q2.1 Since the start of the course, how would you describe your participation level? Choose the answer that best applies to your situation.

In order for the survey to unfold correctly you need to answer this question.

• I did not participate in this course at all (1)

• I did not participate in this course, but I did browse around a little bit (2)

• I only looked at specific parts of the course that I was interested in (3)

• I participated in the course, but I stopped participating along the way (4)

• I participated in the course until the end (5)

42
• 0 (0) • 1 (1) • 2 (2) • 3 (3) • 4 (4) • 5 (5) • 6 (6) • 7 (7) • 8 (8) • 9 (9) • 10 (10)

Q2.2.1 Could you please describe the reason(s) why you did not start the course? ___________________________________________________________________________

Q2.2.2 Could you please describe which specific parts of the course you were interested in and why?

Q3 The next questions are about your overall experience and impression of the course. Q3.1 On a scale from 1 to 10, what overall grade would you give the course?

very poor, 10: excellent)

Q3.2 What did you find most valuable in the course? Please explain.

Q3.3 Which aspects of this course would you like us to improve? Please explain.

Q4 What is your current type of enrollment in the course?

• Audit - I am enrolled for free (1)

• Verified - I paid (2)

Q4.1 What additional value did you get from the verified track? ___________________________________________________________________________

Q4.2 How would you rate the following aspects of the course? The course was ...

Strongly disagree (1)

Somewhat disagree (2)

Neither agree nor disagree (3)

Strongly agree (5) Unique (Q4.5_1) Useful (Q4.5_2) Interesting (Q4.5_3)

Somewhat agree (4)

43
Very poor Excellent Overall grade: () 1 2 3 4 5 6 6 7 8 9 10
(1:

Q4.3 How would you rate the difficulty level of the course?

• Far too difficult (1)

• Too difficult (2)

• About right (3)

• Too easy (4)

• Far too easy (5)

Q4.4 How would you describe the amount of work required for the course?

• Far too little (1)

• Too little (2)

• About right (3)

• Too much (4)

• Far too much (5)

Q4.5 How would you describe the breadth of topics covered in the course?

• Far too narrowly (1)

• Too narrowly (2)

• About right (3)

• Too broad (4)

• Far too broad (5)

Q4.6 How would you describe the length of the course (i.e. number of weeks)?

• Far too short (1)

• Too short (2)

• About right (3)

• Too long (4)

• Far too long (5)

Q5 The next couple of questions are about how you participated in the course and how you perceived different elements of the course.

Q5.1 On average, how many hours per week did you work on this course? This includes assignments, reading material, and video lectures. Hours 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20

Q5.2 Which elements of the course did you use or participate in? Choose all that apply.

• Videos (1)

• Reading materials (2)

• Forums (3)

• Exercises, quizzes, assignments (4)

• Group work (5)

44

Q5.2.1 How satisfied were you with the following elements of this course?

Videos (Q5.2.1_x1)

Reading materials (Q5.2.1_x2)

Forums (Q5.2.1_x3)

Exercises, quizzes, assignments (Q5.2.1_ x4)

Group work (Q4.2.1_ x5)

Feedback by instructors (Q4.2.1_x6)

Possibility to work on your own project (Q4.2.1_x7)

Q5.2.2 How valuable were the following elements of this course? Not at all valuable (1) Somewhat valuable (2) Moderately valuable (3) Valuable (4) Very valuable (5)

Videos (Q5.2.1_x1)

Reading materials (Q5.2.1_x2)

Forums (Q5.2.1_x3)

Exercises, quizzes, assignments (Q5.2.1_ x4)

Group work (Q4.2.1_ x5)

Feedback by instructors (Q4.2.1_x6)

Possibility to work on your own project (Q4.2.1_x7)

45 Very dissatisfied (1) Somewhat dissatisfied (2) Neither satisfied nor dissatisfied (3) Somewhat satisfied (4) Very satisfied (5)

Q5.2.3 Why did you not use or participate in ${Q4.2/ChoiceGroup/UnselectedChoices}?

Q5.3 What was the biggest challenge in completing this course?

• Using the learning platform (1)

• Meeting the deadlines for assignments and exams (2)

• Grasping the course content (3)

• Allocating sufficient time and combining it with other life obligations (4)

• Other: (5)

• I didn’t experience any challenges (6)

Q5.3.1 At the beginning of the survey you said that you participated in the course, but stopped participating along the way. Why did you not participate in the course until the end? Choose the answer that applies the most to your situation.

• Due to personal obligations. (1)

• Due to an unexpected change in available time. (2)

• Due to a general lack of time. (3)

• The course was not what I expected. (4)

• The course was too difficult for me. (5)

• I found a better course somewhere else. (7)

• Other: (6)

46
_____________________________________________________________
_____________________________________________________________

Annex 3: List of accredited courses

Online Academic Courses – Course accreditation as part of the Master Aerospace Engineering at TU Delft

• Aeroelasticity

• Modeling, Simulation and Application of Power and Propulsion Systems

• Helicopter Performance, Stability and Control

• Non-linear Modeling

• Design of Lightweight Structures I: Composites & Metals

• Fatigue of Structures & Materials

• Linear Modeling (including FEM)

• Advanced Dynamics

• Rotor and Wake Aerodynamics

Accreditation period: 21 January 2020 to 01 November 2025.

Solar Energy Engineering MicroMasters® – Course accreditation as part of the TU Delft Master Sustainable Energy Technology and the Master Electrical Engineering.

The accreditation periods are respectively: 30 November 2017 to 01 November 2023, and 31 August 2017 to 01 May 2023.

Aerobic Granular Sludge Technology for Wastewater Treatment – Accredited by Chartered Institution of Water and Environmental Management (CIWEM).

Accreditation period: 21 November 2021 to 20 November 2023.

Nanofiltration and Reverse Osmosis in Water Treatment – Accredited by Chartered Institution of Water and Environmental Management (CIWEM).

Accreditation period: 21 November 2021 to 20 November 2023.

47

TU Delft Extension School for Continuing Education

Since 2013, our online courses and programs help thousands worldwide to make an impact on areas of great relevance to the environment and society. By delivering expert knowledge and applicable skills for engineers, we support working professionals to stay on top of their game – whether they are looking to ch ange career, broaden their skillset, or acquire further academic qualifications.

www.tudelft.nl/extension-school extension-school@tudelft.nl

@TUDelftOnline

Building 32, Landbergstraat 15, 2628CE Delft, The Netherlands

Please attribute TU Delft Extension School CC-BY-NC-SA.

Turn static files into dynamic content formats.

Create a flipbook
Issuu converts static files into: digital portfolios, online yearbooks, online catalogs, digital photo albums and more. Sign up and create your flipbook.