Patient-Led Research Playbook

The Promise of Patient-Led Research Integration into Clinical Registries and Research

NESTED PLAYBOOK

May 3, 2023

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

1

PLRC

CMSS

Gina Assaf

Helen Burstin, MD

Hannah Davis

Elizabeth O’Keefe

Lisa McCorkell

Julia Peterson

Anisha Sekar

Suzanne Pope

Hannah Wei

This playbook was funded through a Patient-Centered Outcomes Research Institute ® (PCORI ®) Eugene Washington PCORI Engagement Award (21376-CMSS).

The statements presented in this work are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute ® (PCORI ®), its Board of Governors or Methodology Committee.

To cite this document, please use: The Promise of Patient-Led Research Integration into Clinical Registries and Research: Nested Playbook.

Council of Medical Specialty Societies, Patient-Led Research Collaboration.

May 2023. www.cmss.org. Accessed [fill in date].

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

ACKNOWLEDGMENT

CMSS/PLRC Project Team

2

INTRODUCTION

The Promise of Patient-Led Research Integration into Clinical Registries and Research Project is a collaborative project of the Council of Medical Specialty Societies (CMSS) and the

Patient-Led Research Collaborative (PLRC). Funded through the Eugene Washington PCORI Engagement Awards program, an initiative of the Patient-Centered Outcomes Research Institute (PCORI), this project aims to develop a sustainable model for patient communities to use in collaborating with partners in patient-led research as well as tools and resources for traditional research entities to overcome challenges in the organization and sustainability of patient-led outcomes research teams.

CMSS and PLRC have worked collaboratively with stakeholders representing patients, patient advocates, funders, the research and medical community and others to explore perceptions of patient communities and traditional medical research entities on the integration of patient-led outcomes research into the comparative clinical effectiveness research (CER) enterprise, including promises and limitations of using COVID-19 as the basis for CER infrastructure development across a wide range of conditions, expand the understanding of best practices in the recruitment of diverse populations to patient-led outcomes research and CER.

While the medical community has moved toward incorporation of the patient voice into research, additional effort is required to incorporate patient-led outcomes research and CER into the medical research framework. The contribution of patient-led outcomes research in COVID-19 should lead to recalibration of research roles for patients and the medical community. The tools and strategies developed by CMSS and PLRC support capacity building and sustainability models for patient-led CER, build bridges to the medical research community, and foster new models of collaborative CER that extends beyond COVID-19, including the role of patient-led research in comparative effectiveness research.

_________

The Council of Medical Specialty Societies (CMSS) was founded in 1965 to provide an independent forum for discussion of issues facing medical specialists. Today, CMSS is a coalition of 47 specialty societies representing more than 800,000 physicians across the house of medicine. Its mission includes catalyzing improvement through convening and collaborating as well as impacting the future of healthcare and the patients CMSS members serve.

The Patient-Led Research Collaborative (PLRC), comprised of scientists, data analysts and other experts in public policy, human-centered design, health activism, and participatory design, was self-organized in 2020 as a result of its patient-led research around the Long COVID experience. PLRC has demonstrated the potential of patient-led outcomes research and has published several groundbreaking papers.

The Patient-Centered Outcomes Research Institute (PCORI) is an independent, nonprofit organization authorized by Congress in 2010 to fund comparative effectiveness research that will provide patients, their caregivers, and clinicians with the evidence needed to make better-informed health and healthcare decisions. PCORI is committed to seeking input from a broad range of stakeholders to guide its work.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

3

Overview and Purpose of the

PLAYBOOK The COVID-19 crisis has exposed significant limitations in our research and dissemination enterprise. Though many clinical registries and specialty societies engage patients in their work through advisory committees, these efforts stop short of full collaboration and leadership by patients in clinical effectiveness research. Newer patient-led participatory outcomes research for Long COVID demonstrates the power and contribution potential of patient-led outcomes research to assess effective treatment options for the clinical and functional outcomes that matter most to patients.

While evidence-based information and guidelines were rapidly transmitted to practicing clinicians by specialty societies and other trusted sources, many gaps remained in emerging diagnostic and treatment strategies. Clinical registries have traditionally supported research and discovery using data from electronic sources, including electronic health records (EHRs), claims data, and clinician-entered data. Clinical registries also incorporate patient-entered data on symptoms and outcomes, including health status and related disability. Many specialty societies developed or adapted existing registry platforms to support surveillance for patients at highest risk (e.g., cancer, autoimmune diseases) and used clinical registries to collect data on diagnosis, imaging, and prognosis across specialties (e.g., VIRUS registries).

In July-September 2020, CMSS supported a webinar series to address key questions related to the rapid development, deployment and implementation of COVID-19 focused clinical registries and clinical repositories by specialty societies and academia. One of the CMSS webinars, “Prioritizing Patient Engagement and Inclusion of Patient-Generated COVID-19 Data” featured the research of the Patient-Led Research Collaborative (PLRC).

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

4

Overview and Purpose of the

PLAYBOOK

The PLRC is a self-organized group of patients working collaboratively to lead and share research around the Long COVID experience. The team consists of researchers across a wide range of fields, including participatory design, neuroscience, public policy, data collection and analysis, human-centered design, and health activism. As patients, the team also has intimate knowledge of living with COVID-19. Body Politic started a support group when members of their team fell ill and needed to rely on each other for emotional support. In creating the COVID-19 support group, Body Politic wanted to provide others with that same opportunity to feel connected and supported through infection, symptoms, and recovery. As an extension of this patient support group, the PLRC was established to drive research to capture and share a bigger picture of the experiences of patients suffering from COVID-19 with prolonged symptoms using a data driven approach. The team developed survey content and research analysis to be “patient-centric” and conducted through community-based participatory research (CBPR). Survey questions and symptoms were aggregated and curated by patients themselves with expertise in research and survey design. Analysis was also conducted by patients themselves with expertise in both quantitative and qualitative data analysis. As noted in a PLRC paper, “the point of research is to provide access to trusted information, and to contribute to our shared knowledge and understanding of an illness while finding treatments that improve patients’ quality of life.”

In May 2020, the PLRC published their first report on Long Covid. Dr. Francis Collins recently referred to the group as “citizen scientists” and their work was widely cited within the broader research and clinical enterprise, as well as the lay press. The U.S. Centers for Disease Control has acknowledged that a significant proportion of patients have persistent symptoms and a recent study from Geneva demonstrated that up to 35 percent of patients have persistent symptoms beyond 30-45 days. The PLRC has advocated for the value of patient-centric and patient-led outcomes research, calling for the following: the acknowledgement of Long COVID as an illness, an accurate estimate of the prevalence of Long COVID, publicly available basic symptom management, care and research to not be limited to those with positive PCR and antibody tests, and aggressive research and investigation into the pathophysiology of symptoms.

The gold standard for clinical research has been large, anonymized, randomized trials, organized from the top down by experts. While traditional research has an important role, it tends to remain silent on a vast number of questions that are vitally important to patient health and well-being. Patient-centered clinical effectiveness research closes these gaps by addressing research questions most important to patients. Without a strong patient voice, research may still have important gaps, including the ability to personalize evidence on treatments for the individual patient. With the diversity of patient needs and questions, coupled with the skills and experiences required to address them, participatory, peer-to-peer production is required. The equitable engagement and contribution of patients across all phases of research is a hallmark of the CBPR strategy.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

5

Overview and Purpose of the

PLAYBOOK

Further engagement with CBP researchers, like the PLRC, can offer important insights for CER designed to meet the needs of the end-users of research and effectively translate findings into practice.

COVID-19 has highlighted the need to build more outcomes research capacity in the patient community, build more institutional support for CER efforts, and build bridges from patient initiatives to the medical research community. This emerging CBPRinformed paradigm for CER can be applied to additional research priorities. It should move patient-centered outcomes research beyond patient inclusion on advisory panels toward a systematic, long term funding commitment. Such a commitment will support patient-led outcomes research and build the institutional capacity to engage traditional biomedical research on a more equal basis.

CMSS and PLRC were guided in this effort by the following key questions for the future of patient-led outcomes research in COVID-19 and beyond: • How can the research enterprise be more supportive of patient-led outcomes research?

• What barriers, including underlying attitudes and beliefs about patients and clinical researchers limit collaboration across clinical outcomes research, clinical registries, and patient researchers? * How can patient-led r esearch and patient-generated outcomes data be further incorporated into the clinical and research enterprise for CER? For example, how can PLRC-generated outcomes data support comparative effectiveness of treatment options for Long COVID? * How can patient-led r esearch incorporate needed medical expertise into their research? * How can patient-led r esearch incorporate perspectives from diverse patient populations into their research?

• How can patient-led r esearch results be integrated into broader CER on potential treatments for Long COVID? How can patients lead these CER efforts, rather than advising on others’ research efforts?

• What are the pathways for sustainability for patient-led outcomes research and CER? How can specialty societies, clinical registries, research agencies, and academic medicine provide needed institutional support?

• What can we learn from COVID-19 patient-led outcomes research about needed skills and scientific capacity that can be incorporated into sustainable models of patient-led CER in other clinical areas? What do patient-led CER teams need to effectively disseminate actionable information on evidence-based treatment options to patients?

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

6

Overview and Purpose of the

PLAYBOOK

As work in patient-based participatory research advances, the findings and limitations of PLRC’s Long COVID research will serve as a model for collaborative, participatory, CER integration that advances the current traditional biomedical research framework.

The CMSS and PLRC team determined that a nested playbook and toolkit with linked resources and tools directed at patients, patient advocacy organizations, researchers, funders, and the broader medical community will be most meaningful. This playbook includes targeted resources for different stakeholders that are built on the same language and approach to ensure shared accountability for meaningful collaboration.

These resources will enable patient partners to assess whether research grantees are complying with patient-led and patient-empowered research principles as well as for researchers to assess prior to submission of a funding proposal if patients will be true partners in the research. Furthermore, patient researchers can use these tools to identify potential partners with unacceptable levels of patient engagement.

This nested playbook provides a shared lexicon and harmonized tools across patient, medical research, and funder communities. Shared language and tools can support meaningful recalibration of the research roles for patients and foster new models of collaborative research. This nested approach provides clear boundaries for unacceptable models of engagement with patients. The playbook includes targeted resources for different stakeholders that are built on the same language and approach to ensure shared accountability for meaningful collaboration. The engagement of those underrepresented in research, including Black, Indigenous and People of Color (BIPOC), is woven into the playbook.

The aligned tools and nested playbook builds off inputs from the advisory committee, key informant interviews, and desk research on current models and collaborative frameworks. Additionally, the tools and playbook build off PLRC’s experience leading research as a patient group.

The elements of the nested playbook include: 1. Shared glossary/lexicon of key definitions/concepts that have emerged from our work (e.g., power imbalance, burden of participation) 2. Aligned checklists with key questions that can be used to demonstrate meaningful engagement 3. Emerging collaborative frameworks and models built on prior work 4. Specific tools, including scorecards and case studies that can be used to drive improvement

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

7

Research Inputs into

PLAYBOOK DEVELOPMENT In parallel to the desk research and co-working sessions between CMSS and PLRC, PLRC developed a survey of its active members to assess their baseline perceptions of the medical, patient, and research community and to inform PLRC's initiatives going forward. PLRC’s survey of its members resulted in 16 responses from long COVID patients who joined the PLRC research group from March 2020 – July 2021. Members cited the value of PLRC’s work as one of the key lessons learned in the 15 months since PLRC began its work. Specifically, PLRC members noted the following: • •

Medical research led and informed by patients has ensured that the patient perspective and what is of value to patients is prioritized, and produces timely information Multi-disciplinar y skills with a diverse group of people coming together organically with a more democratic decision-making process has led to excellent work

PLRC members emphasized the importance of research led by patients so that the results are meaningful to those living with the condition. Furthermore, bringing together a team of multi-disciplinary stakeholders with broad skills and a reliance on democratic governance is critical to high quality, timely results that will truly advance the field and benefit patients. PLRC will build upon these learnings to develop a model that reflects these themes and advances comprehensive, thoughtful, accurate and timely research.

Additionally, PLRC and CMSS project staff surveyed the Project Advisory Panel to guide its efforts and identify areas where the group should focus its research. Participant culture, leadership/governance, and barriers/lessons learned emerged as critical topics to explore further with key informants. Advisory Panel members cautioned that the question is not “if” patients should be in these leadership roles but rather how they can be in these roles and how this process can be improved. There was also a concern about structural racism and how to ensure representation for all patient perspectives. Furthermore, the Advisory Panel expressed interest in exploring how patient-led research can help instill trust in the research process in populations that have been previously hesitant to engage or have been historically exploited. Advisory Panel members called for training for patients with critical lived experience who are looking to get more involved in research and cited the gap of a lack of basic scientific expertise in patients which can be a barrier to their involvement in research. PLRC and CMSS project staff utilized this guidance in conducting key informant interviews. Survey results can be found here: https://www.survey monkey.com/stories/SM-7N76V8NJ/. The appendix includes the Project Advisory Committee Member Roster and Advisory Committee Baseline Survey Questions.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

8

Research Inputs into

PLAYBOOK DEVELOPMENT Environmental Scan: Key Informant Interviews and Data Synthesis PLRC and CMSS project staff conducted key informant interviews with a variety of patients, patient advocates, researchers, funders, and others in the medical research community to assess collaborative patient-led outcomes research approaches.

Key Informant Interviews: Kristin Carman, PCORI, funder • Donna Cryer, Global Liver Institute, patient advocate • Prashila Dullabh, NORC at the University of Chicago, researcher • Rachael Fleurence, National Institutes of Health, researcher • Gregg Gonsalves, Yale School of Public Health, patient advocate • Karla Monterroso, Leadership Coach, patient advocate • Margaret O’Hara, University of Birmingham, patient researcher • Sally Okun, Clinical Trial Transformation Initiative, patient advocate • Eleanor Perfetto, National Health Council, patient advocate • Jaime Seltzer, ME Action, patient advocate • Emily Sirotich, Global Rheumatology Alliance, patient researcher • Sharon Terry, Genetic Alliance, patient advocate • The appendix includes the key informant discussion guide. Guided by Advisory Panel input, the project team conducted key informant interviews focused on the following overarching questions: Participant culture • What are the biggest obstacles in building collaborative models?

• What are successful strategies in recruiting/engaging diverse patients?

Leadership/Governance • What are some examples of productive governance structures in patient-led research?

• What are some leadership challenges in engaging the community?

• What successes ha ve you had in overcoming these challenges?

Barriers and lessons learned • What other sectors ha ve shifted from researcher to researched? Lessons learned?

• What is the biggest barrier for collaborative models?

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

9

Research Inputs into

PLAYBOOK DEVELOPMENT Beyond COVID, the team has explored a variety of established and emerging patient communities and interviewed patients, advocates, researchers, and other stakeholders in clinical research organizations living with and working in the fields of the following disease conditions: • Myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) • Genetic Disorders • Hyperemesis gravidarum • Liver Disease • Arthritis • Parkinson’s Disease • Amyotrophic Lateral Sclerosis (ALS) • Human Immunodeficiency Virus (HIV) From these diverse perspectives, the team identified cross-cutting themes that are critical in patient-led research and in aligning patient-led outcomes research approaches. Many key informants cited opportunities for improvement that PLRC should focus on, including the significant burden of participation for patients. Patients and their advocates often have limited time, energy and resources and this must be considered when engaging patients in research. PLRC has grappled with these challenges and found unique ways to overcome some of them through the use of technology, such as Slack and Google docs, that allow patients to contribute when they have time and energy.

A synthesis of key informant interviews was visually depicted and thematically sorted into areas of focus which informed the dimensions of the scorecard model. The team leveraged Miro software for collaborative thematic analysis.

Key Themes on Patient-Led Research in CER As a result of synthesis of the data collected through the baseline assessment and environmental scan, the team identified the following key themes: •

Motivations of involvement and biases – Researchers and patients alike enter collaborations with their own biases. To enable true patient-led research in CER, it is important to identify motivating factors that drive participants so that these potential biases can be overcome.

•

Power dynamic in collaboration – It is critical that patients have authority and power in the collaboration, rather than researchers exclusively holding the power and leading the effort.

•

Data and technology's role in empowering patients or reinforcing problematic dynamics – Data and technology can provide knowledge and power but can also exacerbate inequities and affect the collaboration; it is important to be aware of these issues and take steps to mitigate them.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

10

Research Inputs into

PLAYBOOK DEVELOPMENT •

Capacity building – With the right level of support, the collaboration is an opportunity for patients to build their capacity, receive professional benefits like visibility and networking, and improve their scientific skills.

•

Value of “nimbleness” with rigor – In its work, PLRC coupled scientific rigor with flexibility to adapt in a rapidly changing environment during the early days of the pandemic when there was so much to learn.

•

Patient groups models of operation – Many patient organizations follow dynamic and non-linear paths to success. It is important to recognize that these strategies can lead to effective organizations.

•

Funding dilemmas for patients – Patient organizations face real challenges regarding funding as they must balance the need for resources to continue their work with the threat of losing autonomy. Moral alignment is not enough to ensure they can remain committed to their mission while obtaining necessary funding.

•

Patient-centric values – In CER, it is vital to acknowledge the burden of patient participation and, therefore, patients must see direct value to continue their participation in patient-led research. CER must be designed around patient-centric values to truly engage patients as collaborators.

The team then delved into these key themes further and reviewed existing models of collaboration in light of these important themes.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

11

SCORECARD DIMENSIONS The following sections provide descriptions for each of the key dimensions of the scorecard and anchor descriptions for each dimension (non-collaboration, minimal collaboration, acceptable collaboration, great collaboration, and ideal collaboration). Cross-cutting consideration for BIPOC communities is woven into the scorecard.

In using the scorecard, each dimension and domain should be assessed individually. For example, under Accessible Engagement in the Patient Burden scorecard, the user should evaluate how the research organization designs engagement avenues and score them accordingly. The scores should not be combined across subdomains. So an organization may score -2 for Accessible Engagement, 1 for Trauma-Informed Practices, 0 for Responsiveness to Patients, and -1 for Compensation. These scores should not be combined or averaged as that would not accurately reflect patient burden.

Dimension

Why it is important to assess

Patient/Partner Governance

Decision-making power and governance should be shared between patient groups and partner groups

Integration into Research Process

Patients are the experts of their condition, and they bring value and skills to cross-domain collaboration. Given the value of the lived experience, patients must be involved in every phase of the research process and key committees, including study design, protocols, trial inclusion, analysis, and reporting.

Patient Burden

Measures the degree to which patient burden and associated trauma is addressed, including accommodating patients who are dealing with illness and symptoms, compensation for patients’ time and skills.

Research Organization Readiness

Measures the ability of the research organization to engage in meaningful patient partnership. This readiness assessment allows patients to discern the research organization’s level of collaboration and willingness to share control.

Patient Group Readiness

Measures the ability of the patient organization to engage in meaningful collaboration. This readiness assessment allows research organizations to discern the level of expertise, collaborative culture, and diversity of the patient group.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

12

DIMENSION ANCHORS

-2

Non-Collaboration

-1

Minimal Collaboration

0

Acceptable Collaboration

1

Great Collaboration

2

Ideal Collaboration

— New baseline of acceptable collaboration

— Patient-informed

— Patients as partners and/or patient-led

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

13

SCORECARD Patient-Led Collaboration RUBRICS PATIENT BURDEN

Based on extensive testing, a 5-point scale with an anchor at 0 for acceptable collaboration was selected.

The Patient Burden dimension includes the following domains: • Accessible Engagement • Trauma-Informed Practices • Responsiveness to Patients • Compensation

PATIENT BURDEN 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, ensures patients can easily request additional accommodations, and works with patients to co-design systemic updates in response to requests.

Patients co-create engagement avenues from the outset to ensure that full participation is accessible and minimally harmful across patient sub-populations.

Accessible Engagement Research organization dictates engagement avenues with no consideration of the patient population's access needs.

Full participation may be impossible; carry a high time, effort, or monetary cost; or cause patients harm.

Research organization considers the patient population when designing engagement avenues, but rarely provides additional accommodations when requested.

Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, and provides individuals with additional accommodations upon request.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

14

SCORING RUBRICS PATIENT BURDEN 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research organization implements traumainformed practices throughout the study, and collaborates with patients to co-design adjustments to those practices during the study. Requests for resources and support are honored at a systemic level for all patients.

A diverse array of patients, representative of the study's subpopulations, collaborates from the outset to co-create a safe, inclusive, mutually respectful environment; implement and adjust trauma-informed practices throughout the research process; and ensure all patients proactively receive sufficient, comprehensive resources and support.

Trauma-Informed Practices Research organization does not consider the trauma burden to patients. Patients may experience discrimination, hostility, new or recalled trauma, or other harms as a result of participation. No trauma-informed practices are in place, and patients receive no resources or support for the trauma caused by participation.

Research organization is aware of a possible trauma burden, but no systemic traumainformed practices are in place, and patients receive no resources or support for their trauma.

Research organization recognizes the trauma burden, and some trauma-informed practices are in place.

Resources and support are provided to patients upon request.

PATIENT BURDEN 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research organization creates clear, accessible, safe channels for patient input throughout the research process; acknowledges patient feedback; and establishes mechanisms for patients to co-design systemic changes to the current study.

Patients co-lead the study from end to end, including creating clear, accessible, safe channels for input, using that input to inform the research process, and acknowledging its impact. Members of the research organization are excited about and fully engaged in patient collaboration.

Responsiveness to Patients No formal channels for patient input are established. Research organization does not address patient feedback, and may exclude or retaliate against patients who voice concerns.

Patients find channels for input to be unclear, difficult to access, or unsafe from retaliation.

Patient feedback may be acknowledged, but rarely results in changes to the current study.

Research organization creates clear, accessible, safe channels for patient input only after the research process has begun. Patient feedback is acknowledged, resulting in changes to analysis, presentation, or communication; and ad-hoc changes to the current study.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

15

SCORING RUBRICS PATIENT BURDEN 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research organization sets patient compensation at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront.

Requests for additional compensation and/or payment options result in systemic changes that benefit all patients.

Patients have decision-making roles in setting and adjusting compensation. Patients are compensated at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk; in the method and timing of their choice. Requests benefit all patients.

Patients receive non-monetary compensation in the form of visibility, professional development, authorship, and awareness of their impact.

Compensation Patients are compensated below market rate for their domain expertise and experience level, with no or limited options for when and how they are paid. Expenses, harm, and risk assumed from participation are not compensated.

Patients are compensated at market rate for their expertise and experience, with no or limited payment options. Expenses, harm, and risk are not compensated.

Research organization sets patient compensation at market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront.

Requests for additional compensation and/or payment options are honored ad-hoc.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

16

SCORING RUBRICS RESEARCH ORGANIZATION READINESS

The Research Partner Readiness dimension includes the following domains: • Recognition of Biases • Collaboration Process • Knowledge in Disease Subject

RESEARCH ORGANIZATION READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research organization is aware of own biases and is open to feedback from patient group and actively iterates on feedback given.

Research organization is aware of own biases and is open to listening to feedback from patient group. Actively iterates on feedback given.

Other patient groups can attest to a positive working relationship.

Research organization has a systemic process for accepting input from patients and patient groups.

Recognition of Biases Research organization does not recognize bias and ignores feedback from patients.

Research organization has limited awareness of own biases and listens to some feedback from patients.

Research organization is aware of own biases, is open to feedback from patients, and implements some of the feedback.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

17

SCORING RUBRICS RESEARCH ORGANIZATION READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Collaboration Process Research organization has no dedicated infrastructure for collaborating with patients.

Research organization has minimal resources/infrastructure for collaborating with patients.

Research organization has dedicated some resources and infrastructure for collaborating with patients (ie. patient panels); has at least one coordinating personnel focused on meeting the patient group's needs; conducts limited training to build skills to engage with patients.

Research organization Research organization has an established has an established infrastructure and infrastructure and process for collaboraprocess for collaborating and codesigning ting with patients that with patients including has been vetted by other at least one dedicated patients/patient groups; person focused on has at least one meeting the patient dedicated person who is group's needs and focused on meeting the advocating to the rest of patient group's needs. the research organizaThe partner is recogtion; conducts routine nized as a patient ally training to build skills to vetted by other patients engage with patients. and patient groups with background in disability justice. Conducts extensive training on meaningful engagement with patients.

RESEARCH ORGANIZATION READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research organization has more than one year worth of knowledge/ experience of the disease being researched.

Research organization has extensive knowledge and direct experience with the disease being researched and those with knowledge are in decision-making roles.

Research organization has a systemic way to keep on top of information from the patient community as well as the latest research.

Knowledge in Disease Subject Research organization has no knowledge/ experience with the disease being researched

Research organization has minimal knowledge/experience (less than one year) with the disease being researched.

Research organization has at least one year worth of knowledge/ experience with the disease being researched.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

18

SCORING RUBRICS PATIENT/PARTNER GOVERNANCE

The Patient/Partner Governance dimension includes the following domains: • Meaningful Decision-making between groups • Accountability between groups

PATIENT/PARTNER GOVERNANCE 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Meaningful Decision-making between groups Decision-making for significant decisions (funding, study design, publication, etc.) is not communicated transparently and/or the research organization decides the decision making process without patient input.

Decision-making process Decision-making process for significant decisions for significant decisions (funding, study design, (funding, study design, publication, etc.) publication, etc.) is not communicated is well communicated and/or agreed upon. and agreed upon Patients have limited or between patients and not meaningful research organization. decision-making power.

Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group.

Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group with sufficient support to make the decisions.

PATIENT/PARTNER GOVERNANCE 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups.

Deference is given to patient groups to define the engagement.

Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups.

Deference is given to patient groups to define the engagement with sufficient support.

Accountability between groups There is a lack of understanding of the rules of engagement/ culture between groups with no written agreement and no defined consequences for not following through.

There is an understanding of the rules of engagement/ culture but no written agreement and/or defined consequences for not following through between groups.

There is a shared understanding and written agreement of the rules of engagement/ culture with defined consequences for not following through between groups.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

19

SCORING RUBRICS INTEGRATION INTO RESEARCH PROCESS

The Integration into Research Process dimension includes the following domains: • Hypothesis Generation • Study Design • Analysis • Publication • Attribution

INTEGRATION INTO RESEARCH PROCESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Research goals proactively address patients’ priorities with sufficient ongoing collaboration. Patient organization's inquiries and lived experiences are included when generating research hypotheses. Patient organizations work with patients to co-design research hypothesis.

Research goals are based on patients’ priorities and co-written by patient organization or patient-researchers.

Patient's inquiries and lived experiences share an equal weight with research organization’s interests when generating research hypotheses.

Hypothesis Generation Research goals are siloed from patients’ priorities. Patients' questions and experiences are not included and/or are dismissed when generating research hypotheses.

Research goals attempt to involve patients’ priorities, but limited by communication or collaboration. Patients' inquiries and lived experiences are rarely included when generating research hypotheses. Patients may have suggested the research question with no further involvement.

Research goals take into account patients’ priorities. Patients' inquiries and lived experiences are included when generating research hypotheses.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

20

SCORING RUBRICS INTEGRATION INTO RESEARCH PROCESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Study Design Research organization does not include patients in the study design process.

Patients do not have the opportunity to provide input on study design.

Patient groups are utilized for recruitment purposes only, if at all.

Research organization does not include patients in the study design process. Patients may be invited to review study design but feedback is rarely incorporated and no functioning accountability system is in place.

Select patient voices are Patient organization and approached to inform their community’s input the study design. are proactively invited to Patients are invited to help inform the study review study design and design. Patient organizahave an impact on the tions are invited to study design. co-design and review study design and patient feedback changes the study design.

Study design is cowritten and reviewed by a diverse array of patient-researchers representative of the study's sub-populations.

If applicable, protocol testing is done by the patient community.

INTEGRATION INTO RESEARCH PROCESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Analysis Patients do not have input in what data to prioritize for analysis and methods of analysis.

Patients are asked to review manuscript drafts but have little say in what data to prioritize for analysis and methods of analysis.

Patients are involved in interpreting data and carrying out analysis in some capacity.

Patient-researchers Patients or patient co-lead on the organizations are invited interpretation and and involved in interpreting data and analysis and/or work carrying out analysis concurrently with partner anywhere in the study. organization’s research team to carry out analysis.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

21

SCORING RUBRICS INTEGRATION INTO RESEARCH PROCESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Publication Study results are inaccessible to patients and/or behind an academic paywall.

Findings are not communicated in lay terms.

Research organization summarizes findings in lay terms, but study results are inaccessible to patients and/or are behind an academic paywall.

Study results are freely Study results are freely accessible to patients accessible to patients and the public. and the public.

Findings are summarized Findings are summarized in lay terms in ways that in lay terms and are are informative to the actively disseminated to patient population. patient population.

Patient-researchers co-write the interpretation and analysis.

Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population. Patient organizations invite patients to co-write findings and reports.

A channel of communication is available for patients to ask questions of the research organization.

INTEGRATION INTO RESEARCH PROCESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Patient group is credited in all public-facing communication and included as authors on papers, to the extent that the patient group wishes to be named.

Patient group was consulted on how they prefer to be attributed.

Patients are acknowledged specifically for what they did throughout the engagement process, are credited in all public-facing communication, and included as authors on papers, to the extent that the patient group wishes to be named.

Patient group was consulted on how they prefer to be attributed.

Attribution Patients' work is attributed to others and/or patients are not attributed at all.

Patients are listed as being involved without a description of how they were involved.

Patients were not consulted on how they prefer to be attributed.

Patients are acknowledged/credited in major public facing communication (press, announcements, papers), to the extent that patients wish to be named. Patients were consulted on how they prefer to be attributed.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

22

SCORING RUBRICS PATIENT GROUP READINESS

The Patient Group Readiness Scorecard dimension includes the following domains: • Expertise • Bias and Representativeness • Accountability • Accommodation • Culture of Collaboration

PATIENT GROUP READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Patient group is up-to-date on research and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.

Patient group has done research on condition, is up-to-date on research, and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.

Expertise Patient group is not versed in research on condition or speaks on only a narrow representation of the condition.

Patient group promotes research that is harmful to the community.

Patient group is not versed in research on condition or speaks on only a narrow representation of the condition.

Patient group is up-to-date on research associated with condition and speaks on the diversity of the condition.

PATIENT GROUP READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

The majority of the patient group are patients themselves or caregivers who can speak on behalf of patients. The majority of the patient group are representative of the group they are representing. Patient group prioritizes and surfaces views brought by on diverse patient population into research collaboration.

The entire patient group are patients or caregivers who can speak on behalf of patients.

The group is completely representative of the group they are representing. Patient group is well versed on own biases and centers the views of underrepresented patient population.

Bias and Representativeness The majority of the patient group is not patients or caregivers who can speak on behalf of patients. Patient diversity isn't prioritized or intentionally planned for, as such, participation in collaboration is severely limited and patient group leaders may be gatekeepers of research opportunities.

The leadership of the patient group is not patients or caregivers who can speak on behalf of patients.

The leadership is not demographically representative of the group they are representing. Participation in collaboration is limited to a select few members of the group.

Leadership of patient group are patients themselves or caregivers who can speak on behalf of patients. Leadership is demographically representative of the patient group they are representing. Patient diversity is prioritized and their participation is planned for.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

23

SCORING RUBRICS PATIENT GROUP READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.

Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group is transparent in sharing research outputs as well as decision-making that affect the patient population.

Accountability Patient group has explicit conflicts of interest with greater community good.

Patient group gaslights or bullies the patient population.

Patient group does not seek out or respond to feedback from the patient population.

Patient group is opaque in their involvement of the collaboration.

Patient group responds to the broader patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.

PATIENT GROUP READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Accommodation Patient group doesn't acknowledge or accommodate access needs of the illness.

Patient group acknowledges access needs of the illness but does not accommodate them.

Patient group acknowledges access needs of the illness. Patient group accommodates most of the access needs of the illness when able.

Patient group acknowPatient group acknowledges access needs of ledges access needs of the illness. Patient group the illness. Patient group accommodates most of accommodates all of access needs of the the access needs of the illness when able. Patient illness when able. group advocates for Patient group advocates access needs of its for access needs of its members. members and the wider patient population.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

24

SCORING RUBRICS PATIENT GROUP READINESS 5-point scale (-2, -1, 0, 1, 2) by subdimensions

-2

-1

0

1

2

Patient group has productive relationships with each other and with other groups. In the event of disagreements, patient group has a process to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.

Patient group has aligned values and practices and seamlessly collaborates with each other and with other groups. Patient group has policies developed to address collaboration dynamics with the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.

Culture of Collaboration There are unsolvable disagreements within the group and/or the group often is in disagreement with other patient groups.

Patient group has no agreed upon code of conduct and/or rules of engagement.

There are concerning disagreements on core values and/or inequitable practices within the group and/or with other patient groups that have caused tension. Patient group has code of conduct and/or rules of engagement that is not followed.

Patient group is able to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

25

USABILITY Considerations Given the goal of providing actionable information for improvement for users of the score- card, we will provide further guidance on how to use the scorecard. Furthermore, it will be important to consider usability by a variety of intended audiences, including what they want to know, and the dimensions that should be evaluated. Based on further testing, the playbook and subsequent training programs will link the score with potential next steps, including linkages to specific illustrative case studies. The example below illustrates how a patient group leader who is engaged in research may use the scorecard.

Going forward it will be important to test several assumptions regarding the scorecard and its use. Key issues to consider and assess are listed in the table below.

1. Who am I Example: Leader of a patient group

2. What would I like to know?

3. Dimensions to evaluate

4. How did I score?

Are partners treating me equitably by patient-led standards?

Patient/Partner Governance, Patient Burden, Integration into Research Process, Research Partner Readiness

Patient/Partner Governance: 4

Addressing patient burden

Patient burden: 2

What’s an organization similar to mine and how did they navigate to a better position?

How can I advocate for better treatment?

How can I grow my team to take on more equitable collaborations?

Assumptions to test: Are our user personas reflective of people’s realities and their intentions?

Will people approach the scorecard with specific questions they want answered about their own organization or collaborators?

Is there an opportunity to narrow down what’s relevant to them? (e.g., if the patient org doesn’t yet have capacity to do research, then certain dimensions won’t be applicable to them)

Integration into Research Process: 2

5. What are the next steps?

Research Partner Readiness: 2

Will people want to pick and choose which dimensions to assess?

Will they use and score all of the recommended dimensions?

Is scoring by dimension or or overall score meaningful?

Are they curious how the model is put together?

Are they curious Is there something about other we haven’t considimensions? dered that should go into scoring?

Do they want more info on how they Is the negative can do better? scoring helpful?

Are we overfitting or underfitting situations with the Do results need to sub-dimensions be accompanied that we have? by some context? (e.g., high patient organization composition could lead to team burnout)

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

26

CONCLUSIONS This project moves beyond patient engagement toward a solution where patient-generated data and patient-led outcomes research become an essential component of medical research, leading to more patient-centric comparative effectiveness research. Patients and patient organizations, funders, research institutions and other traditional biomedical research teams can collaboratively build the infrastructure for patient-led CER. CMSS and PLRC have developed a sustainable collaborative model of CER based on information from and the expertise of patient communities, researchers, funders, and clinical research organizations.

In year two of the project, CMSS and PLRC developed CER training webinars aimed at clinical research organizations and specialty societies that were incorporated into virtual programming for physician leaders and clinical registries to ensure these tools are available to broad audience for widespread use.

Long COVID has presented a unique opportunity to integrate patient-led research into traditional approaches, advancing the field of patient-led outcomes research.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

27

GLOSSARY Capacity

The amount an organization possesses to interact with patients in research.

Collaboration

Working as co-researchers with patients at every stage of the research process so that patients contribute to study design, goals, r esearch questions, methodology, analysis, and reporting; power is shared by patients and traditional researchers.

Control

The power to influence research and direct the research effort.

Decolonized Data

Moving away from making assumptions about a person’s social, economic and political contexts and towards active listening to the person to improve healthcare research; research often extracts and siloes data from the communities it studies.

Engagement

Involving patients in research via a partnership; rather than tokenizing patients, patients are equal partners contributing their expertise and lived experience which is valued equally with academic research expertise.

Leading vs. Co-Leading

Patients leading research puts patients into decision-maker roles at every stage of the research process; co-leading occurs when patients and research partners each contribute their expertise, come to agreement on important decisions, and respect each other’s unique and valuable expertise.

Most-impacted

Patients who have been most affected by an illness and/or societal structures.

Ownership

The state of possessing, accessing, cr eating, and modifying data and controlling the information.

Patient Burden/Trauma

The perceived negative outcomes of a medical encounter, resulting from an increased healthcare workload and diminished capacity to manage the negativ e outcomes that manifest emotionally, relationally, physically, communicatively, and systemically. (from Nottingham K, Chadwick A. Conceptualizing a new definition of patient burden. The Annals of Family Medicine. 2022 Apr 1;20 (Supplement 1).).

Patient-Led Research

Research in which patients are the decision-makers at every stage of the process; the patient is driving the pr ocess and ensuring that those living with the disease are contributing their skills and expertise and managing the effort.

Power

The capacity to direct the course of research and influence the process and be seen as an equal partner with valuable contributions to make.

Readiness

An organization has the ability to engage in meaningful patient-researcher partnership; an organization has a deep understanding of patient-led research along with the infrastructure to collaborate meaningfully .

Research Partner

The research team who is collaborating or wants to collaborate with a patient group; includes individual researchers and coordinating staff with direct contact with the patient group but does not extend to the entir e academic institution.

Tokenization

The practice of inv olving patients in research only for the purpose of appearing to engage patients.

Trauma-informed Research

Research that is conducted with the recognition that a person with lived experience responds in ways that reflect adapting and coping with symptoms of trauma. This is based on understanding the effects of trauma and trauma triggers and making a connection with the person with lived experience to minimiz e harm.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

28

APPENDIX BACKGROUND RESEARCH Existing Models and Collaborative Frameworks As part of the prototype model implementation, the project team has reviewed existing models and frameworks, rather than developing new model frameworks that do not add to existing models or advance the field in a unique way. A key aspect of the prototype model development focused on the characteristics of the PLRC that should be embedded in development of prototype models for meaningful engagement. For example, informants emphasized the nimbleness of the PLRC team and its governance structure which has allowed them to balance speed with scientific rigor. PLRC was able to advance its patient-led research quickly without sacrificing the quality and rigor of the evidence.

Additionally, the project team has identified the engagement of BIPOC communities as a critical developmental element of patient-led research model development. The Advisory Panel recommended that the project team embed best practices in the engagement of diverse populations in patient-led research and CER throughout all deliverables. Key informants identified engagement of diverse communities as an area for improvement and the project team is committed to developing a model and tools that not only engage the BIPOC community but also enable them to drive decision-making. Transparency, trustworthiness, and prioritization of their needs are key components in building these critical relationships.

The project team reviewed the following models and collaborative frameworks and incorporated elements from each of the following sources into the prototype models of patient-led research: • Learning Health Systems • Arnstein’s Ladder of Citizen Participation • Design Strategies in Online Citiz en Science Platforms (Shirk, et al) • Forms and Functions of Participation (Sarah White) • Felicity Callard’s Model • Research Partnership Maturity Model for Patient Organizations (FasterCures)

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

29

APPENDIX Baseline Assessment As part of its baseline assessment, CMSS and PLRC conducted surveys of their respective memberships to identify areas for capacity building and key lessons learned regarding patient engagement. It was critical to understand perceptions of patient communities and traditional medical research entities on the integration of patient-led outcomes and CER into the broader research enterprise as well as to explore how patient and disease/condition characteristics could influence acceptability of patient-led outcomes research, including promises and limitations of using COVID-19 as the basis for sustainable infrastructure development for patient-led CER. Furthermore, CMSS and PLRC identified established and emerging patient communities (outside of COVID-19) and clinical research organizations with an interest and goal of aligning toward patient-led outcome research approaches and endeavored to understand best practices in the recruitment of diverse populations to patient-led outcomes research initiatives.

2021 SURVEY RESULTS CMSS conducts an annual survey of specialty societies and clinical registry leaders and recently conducted its 2021 survey in June 2021. The survey was circulated to Quality Improvement and Registry Directors at the CMSS member societies and included focused questions on patient engagement.

Respondents included 17 specialty societies representing 23 clinical registries.

66.67%

Have not yet engaged patients or patient groups in their registry and performance measurement efforts

46.67%

Are in the process of engaging patients in the selection of outcomes and patient-reported outcome-based performance measures (PRO-PMs)

This represents an opportunity for this collaborative project to provide CMSS members with much-needed tools and resources to ensure meaningful engagement of patients in these important initiatives.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

30

APPENDIX Shared Opportunities for Improvement

In conducting key informant interviews, CMSS and PLRC identified burden of participation for all patients as a critical issue impacting patient-led research. Patients are often dealing with medical crises, limited energy, brain fog, and other limitations that hinder their time and energy to participate fully in research. BIPOC communities are particularly impacted by these issues and often carry with them the burden of historical abuse which has severely impacted their trust in research and research institutions.

Researchers must acknowledge these barriers facing patients and identify ways to accommodate those living with the disease. For example, requiring patients to participate in weekly conference calls during work hours can be problematic. Patients may have other jobs or may not have the energy to participate in lengthy conference calls at a pre-scheduled time.

Furthermore, researchers are often inviting patients to revisit their trauma but then do not acknowledge it or assist in addressing it in any way. Patient surveys may not be reviewed by a social worker or therapist who could minimize harm from the trauma.

Advances in technology can address some, but not all, of these issues. For example, patients can be invited to participate on their schedule, when their time and energy allow.

Patient surveys can be completed at any time. Asynchronous work can be done in collaborative digital environments, such as Google docs, Slack, etc.

Patients may want to be engaged in research on their own terms. That is, they may not be able to take on the burden of co-authorship but can contribute meaningfully in other ways.

Additionally, the messenger can be as important as the message itself. BIPOC communities must feel welcomed and represented if they are to be open to and accept the research findings. It is critical to have a diverse team that can engage diverse communities and prioritize the needs of these communities. Building trust with BIPOC communities can take time to establish reciprocal terms of partnership and create an equitable environment for meaningful collaboration.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

31

APPENDIX Scorecard Dimensions Development

SCORING RUBRICS

The dimensions represented in the nested scorecard build on the themes, models, and examples of patient engagement in research across a wide continuum. The intent of the scorecard is to identify models of engagement with patients in research that are unacceptable, minimally acceptable, and moving toward ideal levels of collaboration with patients.

The definitions of the scorecard dimension will remain the same across different users.

However, different users may use the scorecard for different purposes: • Research organizations may be interested in baseline assessment and strategies to move toward “patient-led” models; • Research funders may use the scorecard to evaluate partner organizations and/or patient groups seeking funding for patient-led research; • Patients and patient groups may use the tool to assess their readiness in leading research and/or evaluating potential partner research organizations.

Existing toolkits and scorecards were examined to identify potential models that would achieve the goals of the project, including: • IDEO Human Centered Design Thinking Toolkit • Rural Health Community Toolkit • NHC Patient Experience Journey Map • IBD Surgery Decision-Making Tool • Community Resilience Toolkit To meet the goals of driving patient-led research, the scorecard output is intended to: • Establish a new baseline of acceptable patient engagement • Provide a shared language between stakeholders • Center people and stories • Starts small with room to iterate and expand • Actionable and accessible The scorecard is intended to be both evaluatory (i.e., evaluate partnerships from a patient-led perspective) and instructive (i.e., how to ideally support and collaborate on patient-led research as a patient or collaborator). Therefore, the output of the scorecard may differ to ensure that it is useful and accessible for each audience.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

32

APPENDIX Development

The initial scorecard prototype was developed by the team following a synthesis of primary and secondary research. First, we gathered insights from an initial round of 12 key informant interviews, as well as feedback from a survey shared among members of Patient-Led Research Collaborative. We also conducted a literature review of existing models, and evaluated several types of frameworks, their usefulness, and gaps. We decided on a scorecard model for its simplicity and immediate deployability in addressing collaboration dynamics between patients / patient orgs with researchers / partner orgs.

The team conducted a round of testing on this version of the Patient-Led Collaboration Scorecard. The goal of the testing was to answer the following questions to help refine the scorecard by its focus and real-world use as well as to inform the next iteration.

•

Understanding of the targeted Users o Who should we prioritiz e as users? o When would this be used? o What use cases should we prioritiz e? o Will the users by self-evaluating or others or both?

•

Scoring Models o What scoring models would be most eff ective given the use cases? o How do we define each score? Clarity and messaging of each score o What drawbacks are involved with scoring? Behavioral considerations o Is negative scoring helpful? o Is scoring by dimension / overall score meaningful?

Is there something we haven’t considered that should go into scoring?

•

Dimensions o Which dimensions ar e a high priority to focus? By needs and expertise o Are we overfitting or underfitting situations with the sub-dimensions that we have? o Are participants curious about other dimensions?

•

Results o Do results need to be accompanied by some explanatory language? o Do they want more information on how they can do better? o What outputs/outline do we cr eate for next steps?

•

Overarching model output o What could we do with our entire model to improve?

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

33

APPENDIX Methodology for Testing & Validation of Model The team conducted a combination of asynchronous and synchronous directed user testing sessions to gather feedback on the prototype. To explore emerging use cases and account for gaps, the team recruited three different types of users to test the scorecard based on their background and context of usage: • • •

Patient community leaders in Long COVID and chronic illnesses Researchers with a variety of exposure to patient-led research based on the team’s assessment Funders of health research

The team collected feedback from 11 individuals in addition to a group feedback session from the advisory panel, and a group feedback form from PCORI officers which was composed of multiple responses. In total, the team collected feedback from 20 potential users. Additionally, the team consulted with a behavioral scientist on best practices for scoring. The team synthesized the feedback and responses into actionable updates to be made to the model for this version and future iterations of the model.

The following points are a summary of the main areas of feedback, how we addressed them, and next steps:

Feedback

Updates/Next steps

Users/Use Cases:We drew two distinct use-cases of the • scorecard from testers’ feedback 1. Patients evaluating partner organizations feel that they can use the scorecard as a way to facilitate conversations to advocate for better collaboration standards, and develop a needs assessment when entering collab- • orations 2. Testers who were champions of patient involvement within research organizations are keen to use the scorecards evaluate their orgs and use the feedback to make adjustments on future studies

Based on the type of users who had given feedback, we prioritized our efforts in this version on scorecards that address patient-burden Next version consider creating different user experiences for the patient and the patient champion within partner organizations

Scoring model: Testers had varying opinions on what scor- • ing model would be most effective (Likert scale 1-5; red/ yellow/green; A, B, C, D, F; keeping as is). Testers, as well as the behavioral scientist, indicated that the messaging/ titles of the scores seem to be the most important aspect in addition to what advice is given once someone is assigned • a score.

Given that we received mixed input on the scoring model, we will maintain the current scoring model with some changes to the text and language We will consider testing other scoring models and language to determine effectiveness

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

34

APPENDIX Methodology for Testing & Validation of Model

Feedback

Updates/Next steps

Dimensions: Generally, the majority of testers found the dimensions comprehensive and intuitive.

•

The testers prioritized the following scorecards over others as being more relevant to their experience: Patient Burden, Integration into Research Process, and Governance.

Data Collection and Ownership was also prioritized, but several reviewers indicated that it required more contextual nuance than what was included and may not be applicable to every type of research study.

•

A few testers mentioned that there were too many dimensions and it was overwhelming to navigate them. In addition, a few other dimensions and subdimensions were mentioned as ones they expected to be included but were not such as: • knowledge of disease (under Partner Organization Readiness), ethics, and intentions of research.

Based on the feedback and prioritization provided by participants, the team decided to move forward with 4 of the 7 initially proposed dimensions for this iteration: patient burden, integration into research process, governance, and research partner readiness. These were chosen based on feedback that they were comprehensive and useful to testers.

The team updated the language describing the dimensions and updated the glossary to include additional definitions.

The patient readiness scorecard was added as a result of feedback provided during the training webinars.

Some testers also commented on the lack of clarity of the • language used to describe some of the dimensions, such as “partner organization” as being too broad.

The team will consider further work on the other dimensions in future iterations.

Each Dimension Scorecard:The testers provided feedback • on each individual scorecard, suggesting language updates, changes to subdimensions, etc.

The team updated the prioritized scorecards with suggested edits.

•

The team will develop recommendations for ways to improve the user’s score so that the scorecard can be used for evaluation, as well as improvement.

Additional tools may be identified and developed during implementation.

Results & Next Steps Section:Testers suggested developing clear, actionable recommendations for improvement.

They suggested including checklists as a useful tool.

Several cited case studies, indicating how they would help users understand how to use the scorecards in different scenarios.

•

Overall Feedback on Model:Testers provided more generally • positive feedback on the model and prototype. Many testers agreed with the model logic and found the tools intuitive to use. However, a few points of constructive feedback for the team to consider were the following: • • Some commenters noted that the language was at times judgmental and unclear • Navigating and going through multiple scorecards felt complicated/overwhelming • How do we ensure that these scorecards are taken seriously by the research community?

•

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

The team updated some of the language so it is less “judgmental” and provided a comprehensive glossary – This will help clarify the unclear language.

The scorecards will be presented differently in the final output and we reduced the number of dimensions for evaluation, which will help with navigating the scorecards The team will continue iterating on the model 35

APPENDIX Key Informant Discussion Guide Introduction Thanks for taking the time to speak with us today. I am ____, and my co-researcher ____ is also on the call listening-in and taking notes.

We are with PLRC and CMSS working on a PCORI funded project to look at patient-led research models and explore the key components of a successful model of collaboration between traditional research and patient-led research organizations. PCORI is an independent, nonprofit organization authorized by Congress in 2010 to fund comparative effectiveness research that will provide patients, their caregivers, and clinicians with the evidence needed to make better-informed health and healthcare decisions. PCORI is committed to seeking input from a broad range of stakeholders to guide its work.

We will develop tools and resources for traditional research entities and patient-led participatory research communities to share the PLRC’s successes in collaboration as well as help communities overcome the challenges in initiating patient-led outcomes research teams. To that end, your input is critical in helping us fully understand models of collaboration.

[As you know, you were recommended to us by___.] We will be asking you questions about your experience as a (Researcher/Patient/Funder, X) and will have a general guided discussion with you on the topic of patient-led research.

Just so you know, anything you say won’t directly be attributed to you or your organization without explicit permission for something we may want to quote you on. **Request to Record Interview** Before we begin, can I ask for your permission to record this interview? It will only be used by us, to make sure we heard everything.

Do you have any questions before we start? **Notes to interviewer** What to look for to guide analysis - action and consequences (I did X and Y happened) - ask followup q’s about consequences, might lead to feelings, opinions, values - feelings (when X happened, I felt Y) - opinions (I think X is Y) - dig deeper, might lead to a guiding principle - guiding principles / values / motives (I value X, so I did Y)

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

36

APPENDIX Key Informant Discussion Guide

This part of the guide will give us context on their background and how it impacts and influences their answers. Manage this discussion to keep short and to the point and doesn’t take up too much time.

A. Role/Background • Tell us about your role or experience as a (Patient, researcher, funder).

• Can you briefly tell us what does your organization do? What do you do for them?

For how long?

• What led you to this work/experience?

This part of the guide is to understand their perceptions of PLRC. Make sure to tell them we want to hear their unbiased opinions so we can improve on models for Patient-led Research.

B. General Impressions on Patient-Led • Have you heard of PLRC before (Patient -led research collaborative)?

• What was your first interaction with them? Or when did you first hear of them?

• If yes, What were your first and now general impressions of them? (Probe for positive/ negative impressions) What do you think they did right/doing well? What do you wish they did better or more of?

• Have you heard or seen of other patient-led research initiatives? Which ones? What impressed you about them? What did these groups do well/not so well or struggle with?

• How would you differentiate between patient-led and patient-centered research?

This part of the guide is to understand patients’ interaction with understanding their illness and the research around it and their involvement. If they were ever involved in research, try to understand frustrations, feelings, motivations of being involved in research.

If Patient • Tell us about your illness? What are areas that need research or are under researched for your illness or other related ones?

• Where do you see gaps in the progress of research for your illness?

• Have you participated in research? In what capacity?

• What was the experience lik e?

– What was positiv e about it?

– Any challenges? What wer e the challenges?

– What did y ou wish was available to you?

• Have you collaborated with other traditional researchers? How was the experience?

– What was positiv e?

– What was challenging?

– What did y ou wish was different or could be better?

• (Important) Have you involved other patients from your community in the research?

Have you seen others who have?

• Do you consider yourself an advocate? How so? Has that impacted your research? If not, why not?

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

37

APPENDIX Key Informant Discussion Guide

This part of the guide is to understand medical research and how they involve patients. If a patient, who initiated medical research and didn’t ask above, check and see if they involved other patients. How? What obstacles/challenges did they face and what went well.

If Researcher • What is your main focus of your research work?

• What research work are you most proud of? Tell us why?

• Have you collaborated with (other) patients? Or has your organization? In what capacity?

• What went well? What didn ’t go well?

• What do you wish was different?

This part of the guide is to understand how minorities/BiPOCS were involved in medical research. If they have not done it themselves, try to probe into who they have heard about that did it.

Involving Different Communities (BIPOC/Etc.) • As a researcher have you involved BIPOC communities or other communities typically not included in research?

• Tell us about that experience?

• Where have you seen this done well?

• What challenges ha ve you seen in engaging these communities? (if needed, ask about skill building, ownership credit, trust) • Where are there other types of research who have done in this area?

This part of the guide is to dig into funding of medical research led or with patients. Try to find out where there are examples of it and what has enabled it. Also, if regular medical research or funders of medical researchers dig into understanding how the space may be navigated by new researchers seeking funding.

If Funder/Or knowledge about funding?

• Can you give examples when patient researchers have received funding?

• What are factors that may have triggered or hindered research efforts related to patients being allocated resources (i.e. digital spaces, number of patient/researcher intersections, crowdsourcing funding models, etc.)?

• (If medical researcher) What issues related to funding of research are “common knowledge” pitfalls to academic or medical researchers yet patient groups may not be aware of (i.e., organization qualifications to apply, conflicts of interest, disclosures, etc.)?

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

38

APPENDIX Key Informant Discussion Guide

The goal of this part of the guide is to understand their values and opinions when it comes to research and who/where it comes from.

C. General Research perception • Do you follow medical research? What kind? How do you find good sources of research?

• Are there sources you trust/don’t trust more than others? Why? • How do you evaluate the quality of research? What do you look for?

• Where do you see gaps in the medical research field?

• What is the ideal le vel of patient involvement in research? Wwhy?

• Is it important to distinguish research and advocacy?

The goal of this part of the guide is to get more leads on folks or resources. Also, into what areas we should maybe add to our research based on interest.

Research Leads/Other • What are some research questions that you might have for yourself on this topic of patient-led research and models for engaging patients in research?

• Who do you think would be good folks to speak to on this topic? What resources should we check out?

• Are there any communities, organizations, or people that you feel are usually left out of these conversations? (We’d like to include them if so!)

Wrap-up Thank you so much for your time. Do you have any questions for us? What else do we need to know?

If any other thoughts come up on this topic, please feel free to reach out via email.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

39

APPENDIX CMSS/PLRC Project Advisory Panel Nisreen Alwan, MBChB MRCP FFPH FHEA PGCAP MPH MSc PhD

Associate Professor of Public Health, University of Southampton; Leading researcher on Long COVID; Awarded MBE for services to Medicine and Public Health during the COVID19 pandemic in UK

Donna Cryer, JD

Founder, President and Chief Executive Officer of the Global Liver Institute; Public Member, CMSS Board of Directors

Rachael Fleurence, PhD

Special Assistant to the NIH Director, COVID-19 Diagnostics; Former Program Director; PCORNet, Patient-Centered Outcomes Research Institute (PCORI)

Susannah Fox

Principal, Internet Geologist LLC; Former Chief Technology Officer, US Department of Health and Human Services

Janna Friedly, MD, MPH

Professor, Vice Chair for Clinical Affairs, University of Washington

Harlan Krumholz, MD, SM

Harold H. Hines, Jr., Professor of Medicine and Epidemiology and Public Health at Yale University School of Medicine; Director, Center for Outcomes Research and Evaluation (CORE); Founder, Hugo Health

Fiona Lowenstein

Founder + Editor-in-Chief Body Politic

Monica L. Lypson MD, MHPE

Vice Dean for Education, Vagelos School of Physician and Surgeons, Columbia University

Greg Martin, MD, MSc

President, Society of Critical Care Medicine; Professor and Master Clinician, Pulmonary, Allergy, Critical Care and Sleep Medicine, Emory University School of Medicine; Director, Georgia CTSA Clinical Research Centers; Research Director, Emory Critical Care Center

Karla Monterroso

CEO, Code 2040, non-profit organization focused on Black and Latinx participation and leadership in the innovation economy; Long COVID patient

Dona Kim Murphey, MD, PhD

Director of Scientific Affairs, Corticare; Long COVID Patient

Jaime Seltzer, MS

ME/CFS patient advocate and Director of Scientific and Medical Outreach at MEAction

Craig Spencer, MD, MPH

Director of Global Health in Emergency Medicine at New YorkPresbyterian/Columbia University Medical Center

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

40

APPENDIX Advisory Panel Survey Results

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

41

APPENDIX Advisory Panel Survey Results

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

42

APPENDIX Advisory Panel Survey Results

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

43

APPENDIX Advisory Panel Survey Results

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

44

APPENDIX

-2

Noncollaboration

-1

SCORECARDS Minimal collaboration

0

Acceptable collaboration

1

Great collaboration

2

Ideal collaboration

Accessible Engagement Research organization dictates engagement avenues with no consideration of the patient population's access needs. Full participation may be impossible; carry a high time, effort, or monetary cost; or cause patients harm.

Research organization considers the patient population when designing engagement avenues, but rarely provides additional accommodations when requested.

Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, and provides individuals with additional accommodations upon request.

Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, ensures patients can easily request additional accommodations, and works with patients to co-design systemic updates in response to requests.

Patients co-create engagement avenues from the outset to ensure that full participation is accessible and minimally harmful across patient sub-populations.

Research organization implements trauma-informed practices throughout the study, and collaborates with patients to co-design adjustments to those practices during the study.

Requests for resources and support are honored at a systemic level for all patients.

A diverse array of patients, representative of the study's sub-populations, collaborates from the outset to co-create a safe, inclusive, mutually respectful environment; implement and adjust traumainformed practices throughout the research process; and ensure all patients proactively receive sufficient, comprehensive resources and support.

Research organization creates clear, accessible, safe channels for patient input throughout the research process; acknowledges patient feedback; and establishes mechanisms for patients to co-design systemic changes to the current study.

Patients co-lead the study from end to end, including creating clear, accessible, safe channels for input, using that input to inform the research process, and acknowledging its impact.

Members of the research organization are excited about and fully engaged in patient collaboration.

Research organization sets patient compensation at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront.

Requests for additional compensation and/or payment options result in systemic changes that benefit all patients.

Patients have decision-making roles in setting and adjusting compensation. Patients are compensated at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk; in the method and timing of their choice. Requests benefit all patients. Patients receive non-monetary compensation in the form of visibility, professional development, authorship, and awareness of their impact.

Patient Burden

Trauma-Informed Practices Research organization does not consider the trauma burden to patients. Patients may experience discrimination, hostility, new or recalled trauma, or other harms as a result of participation. No trauma-informed practices are in place, and patients receive no resources or support for the trauma caused by participation.

Research organization is aware of a possible trauma burden, but no systemic trauma-informed practices are in place, and patients receive no resources or support for their trauma.

Research organization recognizes the trauma burden, and some trauma-informed practices are in place. Resources and support are provided to patients upon request.

Responsiveness to Patients No formal channels for patient input are established. Research organization does not address patient feedback, and may exclude or retaliate against patients who voice concerns.

Patients find channels for input to be unclear, difficult to access, or unsafe from retaliation.

Patient feedback may be acknowledged, but rarely results in changes to the current study.

Research organization creates clear, accessible, safe channels for patient input only after the research process has begun. Patient feedback is acknowledged, resulting in changes to analysis, presentation, or communication; and ad-hoc changes to the current study.

Compensation Patients are compensated below market rate for their domain expertise and experience level, with no or limited options for when and how they are paid. Expenses, harm, and risk assumed from participation are not compensated.

Patients are compensated at market rate for their expertise and experience, with no or limited payment options.

Expenses, harm, and risk are not compensated.

Research organization sets patient compensation at market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront.

Requests for additional compensation and/or payment options are honored ad-hoc.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

45

Patient/Partner Governance

APPENDIX

-2

Noncollaboration

SCORECARDS

-1

Minimal collaboration

0

Acceptable collaboration

1

Great collaboration

2

Ideal collaboration

Meaningful Decision-making between groups Decision-making for significant decisions (funding, study design, publication, etc.) is not communicated transparently and/or the research organization decides the decision making process without patient input.

Decision-making process for significant decisions (funding, study design, publication, etc.) is not communicated and/or agreed upon. Patients have limited or not meaningful decision-making power.

Decision-making process for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patients and research organization.

Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group.

Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group with sufficient support to make the decisions.

Accountability between groups There is a lack of understanding of the rules of engagement/culture between groups with no written agreement and no defined consequences for not following through.

There is an understanding of the rules of engagement/culture but no written agreement and/or defined consequences for not following through between groups.

There is a shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups.

Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups. Deference is given to patient groups to define the engagement.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups. Deference is given to patient groups to define the engagement with sufficient support.

46

APPENDIX

-2

Noncollaboration

SCORECARDS

-1

Minimal collaboration

0

1

Acceptable collaboration

Great collaboration

2

Ideal collaboration

Research Organization Readiness

Recognition of Biases Research organization does not recognize bias and ignores feedback from patients

Research organization has limited awareness of own biases and listens to some feedback from patients.

Research organization is aware of own biases, is open to feedback from patients, and implements some of the feedback.

Research organization is aware of own biases and is open to feedback from patient group and actively iterates on feedback given

Research organization is aware of own biases and is open to listening to feedback from patient group. Actively iterates on feedback given.

Other patient groups can attest to a positive working relationship. Research organization has a systemic process for accepting input from patients and patient groups.

Research organization has an established infrastructure and process for collaborating and codesigning with patients including at least one dedicated person focused on meeting the patient group's needs and advocating to the rest of the research organization; conducts routine training to build skills to engage with patients.

Research organization has an established infrastructure and process for collaborating with patients that has been vetted by other patients/patient groups; has at least one dedicated person who is focused on meeting the patient group's needs. The partner is recognized as a patient ally vetted by other patients and patient groups with background in disability justice. Conducts extensive training on meaningful engagement with patients.

Research organization has more than one year worth of knowledge/experience of the disease being researched.

Research organization has extensive knowledge and direct experience with the disease being researched and those with knowledge are in decision-making roles.

Research organization has a systemic way to keep on top of information from the patient community as well as the latest research.

Collaboration Process Research organization has no dedicated infrastructure for collaborating with patients.

Research organization has minimal resources/infrastructure for collaborating with patients.

Research organization has dedicated some resources and infrastructure for collaborating with patients (ie. patient panels); has at least one coordinating personnel focused on meeting the patient group's needs; conducts limited training to build skills to engage with patients.

Knowledge in Disease Subject Research organization has no knowledge/experience with the disease being researched

Research organization has minimal knowledge/experience (less than one year) with the disease being researched.

Research organization has at least one year worth of knowledge/experience with the disease being researched.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

47

APPENDIX

-2

Noncollaboration

-1

SCORECARDS Minimal collaboration

0

Acceptable collaboration

1

Great collaboration

2

Ideal collaboration

Integration Into Research Process

Hypothesis Generation Research goals are siloed from patients’ priorities.

Patients' questions and experiences are not included and/or are dismissed when generating research hypotheses.

Research goals attempt to involve patients’ priorities, but limited by communication or collaboration. Patients' inquiries and lived experiences are rarely included when generating research hypotheses. Patients may have suggested the research question with no further involvement.

Research goals take into account patients’ priorities.

Patients' inquiries and lived experiences are included when generating research hypotheses.

Research goals proactively Research goals are based on address patients’ priorities with patients’ priorities and co-written sufficient ongoing collaboration. by patient organization or patient-researchers. Patient's Patient organization's inquiries and lived experiences are inquiries and lived experiences included when generating share an equal weight with research hypotheses. Patient research organization’s interests organizations work with patients when generating research hypotheses. to co-design research hypothesis.

Study Design Research organization does not include patients in the study design process.

Patients do not have the opportunity to provide input on study design.

Patient groups are utilized for recruitment purposes only, if at all.

Research organization does not include patients in the study design process.

Patients may be invited to review study design but feedback is rarely incorporated and no functioning accountability system is in place.

Select patient voices are approached to inform the study design. Patients are invited to review study design and have an impact on the study design.

Patient organization and their community’s input are proactively invited to help inform the study design.

Patient organizations are invited to co-design and review study design and patient feedback changes the study design.

Study design is co-written and reviewed by a diverse array of patient-researchers representative of the study's sub-populations. If applicable, protocol testing is done by the patient community.

Patients or patient organizations are invited and involved in interpreting data and carrying out analysis anywhere in the study.

Patient-researchers co-lead on the interpretation and analysis and/or work concurrently with partner organization’s research team to carry out analysis.

Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population.

Patient-researchers co-write the interpretation and analysis.

Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population. Patient organizations invite patients to co-write findings and reports. A channel of communication is available for patients to ask questions of the research organization.

Patient group is credited in all public-facing communication and included as authors on papers, to the extent that the patient group wishes to be named. Patient group was consulted on how they prefer to be attributed.

Patients are acknowledged specifically for what they did throughout the engagement process, are credited in all public-facing communication, and included as authors on papers, to the extent that the patient group wishes to be named. Patient group was consulted on how they prefer to be attributed.

Analysis Patients do not have input in what data to prioritize for analysis and methods of analysis.

Patients are asked to review manuscript drafts but have little say in what data to prioritize for analysis and methods of analysis.

Patients are involved in interpreting data and carrying out analysis in some capacity.

Publication Study results are inaccessible to patients and/or behind an academic paywall. Findings are not communicated in lay terms.

Research organization summarizes findings in lay terms, but study results are inaccessible to patients and/or are behind an academic paywall.

Study results are freely accessible to patients and the public. Findings are summarized in lay terms in ways that are informative to the patient population.

Attribution Patients' work is attributed to others and/or patients are not attributed at all.

Patients are listed as being involved without a description of how they were involved.

Patients were not consulted on how they prefer to be attributed.

Patients are acknowledged/ credited in major public facing communication (press, announcements, papers), to the extent that patients wish to be named. Patients were consulted on how they prefer to be attributed.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

48

APPENDIX

-2

Noncollaboration

-1

SCORECARDS Minimal collaboration

0

1

Acceptable collaboration

Great collaboration

2

Ideal collaboration

Expertise Patient group is not versed in research on condition or speaks on only a narrow representation of the condition. Patient group promotes research that is harmful to the community.

Patient group is not versed in research on condition or speaks on only a narrow representation of the condition.

Patient group is up-to-date on research associated with condition and speaks on the diversity of the condition.

Patient group is up-to-date on research and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.

Patient group has done research on condition, is up-to-date on research, and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.

The majority of the patient group are patients themselves or caregivers who can speak on behalf of patients. The majority of the patient group are representative of the group they are representing.

Patient group prioritizes and surfaces views brought by on diverse patient population into research collaboration.

The entire patient group are patients or caregivers who can speak on behalf of patients. The group is completely representative of the group they are representing. Patient group is well versed on own biases and centers the views of underrepresented patient population.

Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.

Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group is transparent in sharing research outputs as well as decision-making that affect the patient population.

Patient group acknowledges access needs of the illness.

Patient group accommodates most of access needs of the illness when able. Patient group advocates for access needs of its members.

Patient group acknowledges access needs of the illness.

Patient group accommodates all of the access needs of the illness when able. Patient group advocates for access needs of its members and the wider patient population.

Patient group has productive relationships with each other and with other groups. In the event of disagreements, patient group has a process to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.

Patient group has aligned values and practices and seamlessly collaborates with each other and with other groups. Patient group has policies developed to address collaboration dynamics with the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.

Patient Group Readiness

Bias and Representativeness The majority of the patient group is not patients or caregivers who can speak on behalf of patients.

Patient diversity isn't prioritized or intentionally planned for, as such, participation in collaboration is severely limited and patient group leaders may be gatekeepers of research opportunities.

The leadership of the patient group is not patients or caregivers who can speak on behalf of patients. The leadership is not demographically representative of the group they are representing.

Participation in collaboration is limited to a select few members of the group.

Leadership of patient group are patients themselves or caregivers who can speak on behalf of patients.

Leadership is demographically representative of the patient group they are representing. Patient diversity is prioritized and their participation is planned for.

Accountability Patient group has explicit conflicts of interest with greater community good.

Patient group gaslights or bullies the patient population.

Patient group does not seek out or respond to feedback from the patient population.

Patient group is opaque in their involvement of the collaboration.

Patient group responds to the broader patient population and other patient groups of related illnesses.

Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.

Accommodation Patient group doesn't acknowledge or accommodate access needs of the illness.

Patient group acknowledges access needs of the illness but does not accommodate them.

Patient group acknowledges access needs of the illness.

Patient group accommodates most of the access needs of the illness when able.

Culture of Collaboration There are unsolvable disagreements within the group and/or the group often is in disagreement with other patient groups. Patient group has no agreed upon code of conduct and/or rules of engagement.

There are concerning disagreements on core values and/or inequitable practices within the group and/or with other patient groups that have caused tension. Patient group has code of conduct and/or rules of engagement that is not followed.

Patient group is able to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

49

CASE STUDY #1

Patient Group Evaluating Potential Partner USE CASE

SUGGESTED USE OF SCORECARDS

A patient group is considering partnering, or is in the middle of a partnership, with a research team on a study.

• If the collaboration is in its planning phase, all

Research team asks representatives from patient group to be present on advisory calls and help with recruiting research participants, but is open to suggestions on working with the patient group.

4 scorecards can help guide discussion around shared expectations. The Research Organization Readiness scorecard helps patient groups determine whether or not to partner and communicate which changes can be made for a fair collaboration.

• The Integration into Research Process scorecard

can help the patient group be aware of their nature of involvement and enable parties to discuss and agree on acceptable and ideal levels of collaboration in the study.

• The patient group can use the Patient Burden

and Patient/Partner Governance scorecard during or after the completion of the collaboration to help assess own involvement and suggest improvements going forward.

IDEAL OUTCOMES

•

Patient group is better equipped to decide on who to partner with.

•

Patient group knows which areas to ask for decision-making power, resources and accommodations, and can quickly align with partner on expectations going into a collaboration.

•

Patient group has more resources and support to lead on initiatives where they have expertise, and develop additional capacities alongside partners.

EXAMPLE

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

CASE STUDY #2

Research Partner Evaluating Themselves USE CASE

SUGGESTED USE OF SCORECARDS

A research team who is working with patients or patient groups is looking for feedback and ways to improve on involving patients going forward.

• If the research project is in its planning phase,

Research team has some capacity for patient engagement and would like to set aside resources to do so early on.

all 4 scorecards can help guide grant-writing and establish mutual collaboration with patient groups.

•

•

IDEAL OUTCOMES

• •

If the study is already underway, the research group can use the Research Organization Readiness and Patient-Partner Governance scorecards to help assess the level of meaningful engagement and implement improvements for a more effective and sustainable collaboration with patients The research team can use the Integration into Research Process and Patient Burden scorecard to find ways to meaningfully engage with patients and plan for accommodations.

EXAMPLE

The Research Partner applies for funding that prioritizes scope and budget for meaningful patient involvement Research Partner proactively addresses the needs of patient group, and the study is made strong by patients’ https://bmjopen.bmj.com/content/12/3/e060826

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

CASE STUDY #3

Funder Evaluating Research Partner USE CASE

SUGGESTED USE OF SCORECARDS

A funder wants to support research projects that are directed by patients or at least have meaningful engagement of patients to ensure the best and most patient-focused outcomes.

• Funder uses all 4 scorecards to develop criteria for

The funder has the ability to set criteria for rating proposals, add patients to their review panels, and is willing to set aside adequate funds for meaningful patient engagement.

evaluating proposals.

Funder shares all 4 scorecards with applicants while they’re developing proposals.

Funder evaluates proposals based on how well the

• proposals score on each of the 4 scorecards.

•

IDEAL OUTCOMES

•

Funder incentivizes patient-led research, enabling researchers to have the capacity to properly engage patients.

•

Funder prioritizes research partners that value patient-led research and have a plan and capacity to meaningfully engage patients.

•

Funder can help give guidance on how to meaningfully engage patients.

•

Funder meaningfully engages patients in their evaluation of proposals.

As funded projects launch, funder checks in on each project to ensure execution of patient-led research plan by evaluating projects using all 4 scorecards.

Funder uses the Research Organization Readiness, Patient-Partner Governance, and Patient Burden scorecards to evaluate their own engagement of patients in the review process.

EXAMPLE

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

The Promise of Patient-Led Research Integration into Clinical Registries and Research

NESTED PLAYBOOK

May 3, 2023

THE PROMISE OF PATIENT-LED RESEARCH INTEGRATION INTO CLINICAL REGISTRIES AND RESEARCH

42