This playbook was funded through a Patient-Centered Outcomes Research Institute ® (PCORI ®) Eugene Washington PCORI Engagement Award (21376-CMSS).
The statements presented in this work are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute ® (PCORI ®), its Board of Governors or Methodology Committee.
To cite this document, please use: The Promise of Patient-Led Research Integration into Clinical Registries and Research: Nested Playbook. Council of Medical Specialty Societies, Patient-Led Research Collaboration. May 2023. www.cmss.org. Accessed [fill in date].
AC K NOWLE D
The Promise of Patient-Led Research
Integration into Clinical Registries and Research Project
is a collaborative project of the Council of Medical Specialty Societies (CMSS) and the Patient-Led Research Collaborative (PLRC). Funded through the Eugene Washington PCORI Engagement Awards program, an initiative of the Patient-Centered Outcomes Research Institute (PCORI), this project aims to develop a sustainable model for patient communities to use in collaborating with partners in patient-led research as well as tools and resources for traditional research entities to overcome challenges in the organization and sustainability of patient-led outcomes research teams.
CMSS and PLRC have worked collaboratively with stakeholders representing patients, patient advocates, funders, the research and medical community and others to explore perceptions of patient communities and traditional medical research entities on the integration of patient-led outcomes research into the comparative clinical effectiveness research (CER) enterprise, including promises and limitations of using COVID-19 as the basis for CER infrastructure development across a wide range of conditions, expand the understanding of best practices in the recruitment of diverse populations to patient-led outcomes research and CER.
While the medical community has moved toward incorporation of the patient voice into research, additional effort is required to incorporate patient-led outcomes research and CER into the medical research framework. The contribution of patient-led outcomes research in COVID-19 should lead to recalibration of research roles for patients and the medical community. The tools and strategies developed by CMSS and PLRC support capacity building and sustainability models for patient-led CER, build bridges to the medical research community, and foster new models of collaborative CER that extends beyond COVID-19, including the role of patient-led research in comparative effectiveness research.
IN T RODUC
The Council of Medical Specialty Societies (CMSS) was founded in 1965 to provide an independent forum for discussion of issues facing medical specialists. Today, CMSS is a coalition of 47 specialty societies representing more than 800,000 physicians across the house of medicine. Its mission includes catalyzing improvement through convening and collaborating as well as impacting the future of healthcare and the patients CMSS members serve.
The Patient-Led Research Collaborative (PLRC), comprised of scientists, data analysts and other experts in public policy, human-centered design, health activism, and participatory design, was self-organized in 2020 as a result of its patient-led research around the Long COVID experience. PLRC has demonstrated the potential of patient-led outcomes research and has published several groundbreaking papers.
authorized by Congress in 2010 to fund comparative effectiveness research that will provide patients,
The Patient-Centered Outcomes Research Institute (PCORI) is an independent, nonprofit organization their caregivers, and clinicians with the evidence needed to make better-informed health and healthcare decisions. PCORI is committed to seeking input from a broad range of stakeholders to guide its work.
N
Over view and Purpose of the P L A Y BOO K
The COVID-19 crisis has exposed significant limitations in our research and disseminatheir work through advisory committees, these efforts stop short of full collaboration and leadership by patients in clinical effectiveness research. Newer patient-led participatory outcomes research for Long COVID demonstrates the power and contribution potential of patient-led outcomes research to assess effective treatment options for the clinical and functional outcomes that matter most to patients.
tion enterprise. Though many clinical registries and specialty societies engage patients in
While evidence-based information and guidelines were rapidly transmitted to practicing clinicians by specialty societies and other trusted sources, many gaps remained in emerging diagnostic and treatment strategies. Clinical registries have traditionally supported research and discovery using data from electronic sources, including electronic health records (EHRs), claims data, and clinician-entered data. Clinical registries also incorporate patient-entered data on symptoms and outcomes, including health status and related disability. Many specialty societies developed or adapted existing registry platforms to support surveillance for patients at highest risk (e.g., cancer, autoimmune diseases) and used clinical registries to collect data on diagnosis, imaging, and prognosis across specialties (e.g., VIRUS registries).
In July-September 2020, CMSS supported a webinar series to address key questions related to the rapid development, deployment and implementation of COVID-19 focused clinical registries and clinical repositories by specialty societies and academia. One of the CMSS webinars, “Prioritizing Patient Engagement and Inclusion of Patient-Generated COVID-19 Data” featured the research of the Patient-Led Research Collaborative (PLRC).
Overview and Purpose of the
P L AY BOO K
shared knowledge and understanding of an illness while finding treatments that improve
The PLRC is a self-organized group of patients working collaboratively to lead and share research around the Long COVID experience. The team consists of researchers across a wide range of fields, including participatory design, neuroscience, public policy, data collection and analysis, human-centered design, and health activism. As patients, the team also has intimate knowledge of living with COVID-19. Body Politic started a support group when members of their team fell ill and needed to rely on each other for emotional support. In creating the COVID-19 support group, Body Politic wanted to provide others with that same opportunity to feel connected and supported through infection, symptoms, and recovery. As an extension of this patient support group, the PLRC was established to drive research to capture and share a bigger picture of the experiences of patients suffering from COVID-19 with prolonged symptoms using a data driven approach. The team developed survey content and research analysis to be “patient-centric” and conducted through community-based participatory research (CBPR). Survey questions and symptoms were aggregated and curated by patients themselves with expertise in research and survey design. Analysis was also conducted by patients themselves with expertise in both quantitative and qualitative data analysis. As noted in a PLRC paper, “the point of research is to provide access to trusted information, and to contribute to our patients’ quality of life ”
In May 2020, the PLRC published their first report on Long Covid. Dr. Francis Collins recently referred to the group as “citizen scientists” and their work was widely cited within the broader research and clinical enterprise, as well as the lay press. The U.S.
Centers for Disease Control has acknowledged that a significant proportion of patients
have persistent symptoms and a recent study from Geneva demonstrated that up to 35 percent of patients have persistent symptoms beyond 30-45 days. The PLRC has advocated for the value of patient-centric and patient-led outcomes research, calling for the following: the acknowledgement of Long COVID as an illness, an accurate estimate of the prevalence of Long COVID, publicly available basic symptom management, care and research to not be limited to those with positive PCR and antibody tests, and aggressive research and investigation into the pathophysiology of symptoms.
The gold standard for clinical research has been large, anonymized, randomized trials, organized from the top down by experts. While traditional research has an important role, it tends to remain silent on a vast number of questions that are vitally important to patient health and well-being. Patient-centered clinical effectiveness research closes these gaps by addressing research questions most important to patients. Without a strong patient voice, research may still have important gaps, including the ability to personalize evidence on treatments for the individual patient. With the diversity of patient needs and questions, coupled with the skills and experiences required to address them, participatory, peer-to-peer production is required. The equitable engagement and contribution of patients ac ross all phases of research is a hallmark of the CBPR strategy
Overview and Purpose of the
P L AY BOO K
Further engagement with CBP researchers, like the PLRC, can offer important insights for CER designed to meet the needs of the end-users of research and effectively translate findings into practice.
COVID-19 has highlighted the need to build more outcomes research capacity in the patient community, build more institutional support for CER efforts, and build bridges from patient initiatives to the medical research community. This emerging CBPRinformed paradigm for CER can be applied to additional research priorities. It should move patient-centered outcomes research beyond patient inclusion on advisory panels toward a systematic, long term funding commitment. Such a commitment will support patient-led outcomes research and build the institutional capacity to engage traditional biomedical research on a more equal basis.
CMSS and PLRC were guided in this effort by the following key questions for the future of patient-led outcomes research in COVID-19 and beyond:
• How can the r esearch enterprise be more supportive of patient-led outcomes research?
• What barriers, including underlying attitudes and beliefs about patients and clinical researchers limit collaboration across clinical outcomes research, clinical registries, and patient researchers?
* How can patient-led r esearch and patient-generated outcomes data be further incorporated into the clinical and research enterprise for CER? For example, how can PLRC-generated outcomes data support comparative effectiveness of treatment options for Long COVID?
* How can patient-led r esearch incorporate needed medical expertise into their research?
* How can patient-led r esearch incorporate perspectives from diverse patient populations into their research?
• How can patient-led r esearch results be integrated into broader CER on potential treatments for Long COVID? How can patients lead these CER efforts, rather than advising on others’ research efforts?
• What ar e the pathways for sustainability for patient-led outcomes research and CER? How can specialty societies, clinical registries, research agencies, and academic medicine provide needed institutional support?
• What can we learn from COVID-19 patient-led outcomes research about needed skills and scientific capacity that can be incorporated into sustainable models of patient-led CER in other clinical areas? What do patient-led CER teams need to effectively disseminate actionable information on evidence-based treatment options to patients?
Overview and Purpose of the P L AY BOO K
of PLRC’s Long COVID research will serve as a model for collaborative, participatory,
As work in patient-based participatory research advances, the findings and limitations CER integration that advances the current traditional biomedical research framework.
The CMSS and PLRC team determined that a nested playbook and toolkit with linked resources and tools directed at patients, patient advocacy organizations, researchers, funders, and the broader medical community will be most meaningful. This playbook includes targeted resources for different stakeholders that are built on the same language and approach to ensure shared accountability for meaningful collaboration. These resources will enable patient partners to assess whether research grantees are complying with patient-led and patient-empowered research principles as well as for researchers to assess prior to submission of a funding proposal if patients will be true partners in the research. Furthermore, patient researchers can use these tools to identify potential partners with unacceptable levels of patient engagement.
This nested playbook provides a shared lexicon and harmonized tools across patient, medical research, and funder communities. Shared language and tools can support meaningful recalibration of the research roles for patients and foster new models of collaborative research. This nested approach provides clear boundaries for unacceptable models of engagement with patients. The playbook includes targeted resources for different stakeholders that are built on the same language and approach to ensure shared accountability for meaningful collaboration. The engagement of those underrepresented in research, including Black, Indigenous and People of Color (BIPOC), is woven into the playbook.
The aligned tools and nested playbook builds off inputs from the advisory committee, key informant interviews, and desk research on current models and collaborative frameworks. Additionally, the tools and playbook build off PLRC’s experience leading research as a patient group.
The elements of the nested playbook include:
1. Shared glossary/lexicon of key definitions/concepts that have emerged from our work (e.g., power imbalance, burden of participation)
2. Aligned checklists with key questions that can be used to demonstrate meaningful engagement
3. Emerging collaborative frameworks and models built on prior work
4. Specific tools, including scorecards and case studies that can be used to drive improvement
Research Inputs into P L AY BOO K
DE V E L OPM E N T
In parallel to the desk research and co-working sessions between CMSS and PLRC, PLRC developed a survey of its active members to assess their baseline perceptions of the medical, patient, and research community and to inform PLRC's initiatives going forward. PLRC’s survey of its members resulted in 16 responses from long COVID patients who joined the PLRC research group from March 2020 – July 2021. Members cited the value of PLRC’s work as one of the key lessons learned in the 15 months since PLRC began its work. Specifically, PLRC members noted the following:
• Medical r esearch led and informed by patients has ensured that the patient perspective and what is of value to patients is prioritized, and produces timely information
• Multi-disciplinar y skills with a diverse group of people coming together organically with a more democratic decision-making process has led to excellent work
PLRC members emphasized the importance of research led by patients so that the results are meaningful to those living with the condition. Furthermore, bringing together a team of multi-disciplinary stakeholders with broad skills and a reliance on democratic governance is critical to high quality, timely results that will truly advance the field and benefit patients. PLRC will build upon these learnings to develop a model that reflects these themes and advances comprehensive, thoughtful, accurate and timely research.
Additionally, PLRC and CMSS project staff surveyed the Project Advisory Panel to guide its efforts and identify areas where the group should focus its research. Participant culture, leadership/governance, and barriers/lessons learned emerged as critical topics to explore further with key informants. Advisory Panel members cautioned that the question is not “if” patients should be in these leadership roles but rather how they can be in these roles and how this process can be improved. There was also a concern about structural racism and how to ensure representation for all patient perspectives. Furthermore, the Advisory Panel expressed interest in exploring how patient-led research can help instill trust in the research process in populations that have been previously hesitant to engage or have been historically exploited. Advisory Panel members called for training for patients with critical lived experience who are looking to get more involved in research and cited the gap of a lack of basic scientific expertise in patients which can be a barrier to their involvement in research. PLRC and CMSS project staff utilized this guidance in conducting key informant interviews. Survey results can be found here: https://www.survey monkey.com/stories/SM-7N76V8NJ/. The appendix includes the Project Advisory Committee Member Roster and Advisory Committee Baseline Survey Questions.
Research Inputs into P L AY BOO K DE V E L OPM E N T
Environmental Scan: Key Informant Interviews and Data Synthesis
PLRC and CMSS project staff conducted key informant interviews with a variety of patients, patient advocates, researchers, funders, and others in the medical research community to assess collaborative patient-led outcomes research approaches.
•
•
Key Informant Interviews:
Kristin Carman, PCORI, funder
Donna Cryer, Global Liver Institute, patient advocate
Prashila Dullabh, NORC at the University of Chicago, researcher
Rachael Fleurence, National Institutes of Health, researcher
Gregg Gonsalves, Yale School of Public Health, patient advocate
Karla Monterroso, Leadership Coach, patient advocate
Margaret O’Hara, University of Birmingham, patient researcher
Eleanor Perfetto, National Health Council, patient advocate Jaime Seltzer, ME Action, patient advocate
Emily Sirotich, Global Rheumatology Alliance, patient researcher
Sharon Terry, Genetic Alliance, patient advocate
The appendix includes the key informant discussion guide. Guided by Advisory Panel input, the project team conducted key informant interviews focused on the following overarching questions:
Participant culture
• What ar e the biggest obstacles in building collaborative models?
• What ar e successful strategies in recruiting/engaging diverse patients?
Leadership/Governance
• What ar e some examples of productive governance structures in patient-led research?
• What ar e some leadership challenges in engaging the community?
• What successes ha ve you had in overcoming these challenges?
Barriers and lessons learned
• What other sectors ha ve shifted from researcher to researched? Lessons learned?
• What is the biggest barrier for collaborative models?
Research Inputs into P L AY BOO K
DE V E L OPM E N
T
Beyond COVID, the team has explored a variety of established and emerging patient communities and interviewed patients, advocates, researchers, and other stakeholders in clinical research organizations living with and working in the fields of the following disease conditions:
From these diverse perspectives, the team identified cross-cutting themes that are critical in patient-led research and in aligning patient-led outcomes research approaches. Many key informants cited opportunities for improvement that PLRC should focus on, including the significant burden of participation for patients. Patients and their advocates often have limited time, energy and resources and this must be considered when engaging patients in research. PLRC has grappled with these challenges and found unique ways to overcome some of them through the use of technology, such as Slack and Google docs, that allow patients to contribute when they have time and energy.
A synthesis of key informant interviews was visually depicted and thematically sorted into areas of focus which informed the dimensions of the scorecard model. The team leveraged Miro software for collaborative thematic analysis.
Key Themes on Patient-Led Research in CER
As a result of synthesis of the data collected through the baseline assessment and environmental scan, the team identified the following key themes:
• Motivations of involvement and biases – Researchers and patients alike enter collaborations with their own biases. To enable true patient-led research in CER, it is important to identify motivating factors that drive participants so that these potential biases can be overcome.
• Power dynamic in collaboration – It is critical that patients have authority and power in the collaboration, rather than researchers exclusively holding the power and leading the effort.
• Data and technology's role in empowering patients or reinforcing problematic dynamics – Data and technology can provide knowledge and power but can also exacerbate inequities and affect the collaboration; it is important to be aware of these issues and take steps to mitigate them.
Research Inputs into P L AY BOO K
DE V E L OPM E N T
• Capacity building – With the right level of support, the collaboration is an opportunity for patients to build their capacity, receive professional benefits like visibility and networking, and improve their scientific skills.
• Value of “nimbleness” with rigor – In its work, PLRC coupled scientific rigor with flexibility to adapt in a rapidly changing environment during the early days of the pandemic when there was so much to learn.
• Patient groups models of operation – Many patient organizations follow dynamic and non-linear paths to success. It is important to recognize that these strategies can lead to effective organizations.
• Funding dilemmas for patients – Patient organizations face real challenges regarding funding as they must balance the need for resources to continue their work with the threat of losing autonomy. Moral alignment is not enough to ensure they can remain committed to their mission while obtaining necessary funding.
• Patient-centric values – In CER, it is vital to acknowledge the burden of patient participation and, therefore, patients must see direct value to continue their participation in patient-led research. CER must be designed around patient-centric values to truly engage patients as collaborators.
The team then delved into these key themes further and reviewed existing models of collaboration in light of these important themes.
SC O RECAR D DI M E N S ION S
The following sections provide descriptions for each of the key dimensions of the scorecard and anchor descriptions for each dimension (non-collaboration, minimal collaboration, acceptable collaboration, great collaboration, and ideal collaboration). Cross-cutting consideration for BIPOC communities is woven into the scorecard.
In using the scorecard, each dimension and domain should be assessed individually. For example, under Accessible Engagement in the Patient Burden scorecard, the user should evaluate how the research organization designs engagement avenues and score them accordingly. The scores should not be combined across subdomains. So an organization may score -2 for Accessible Engagement, 1 for Trauma-Informed Practices, 0 for Responsiveness to Patients, and -1 for Compensation. These scores should not be combined or averaged as that would not accurately reflect patient burden.
Dimension
Patient/Par tner Governance
Why it is important to assess
Decision-making power and governance should be shared between patient groups and par tner groups
Integration into Research Process
Patient Burden
Research Organization Readiness
Patients are the exper ts of their condition, and they bring value and skills to cross-domain collaboration. Given the value of the lived experience, patients must be involved in every phase of the research process and key committees, including study design, protocols, trial inclusion, analysis, and repor ting.
Measures the degree to which patient burden and associated trauma is addressed, including accommodating patients who are dealing with illness and symptoms, compensation for patients’ time and skills.
Measures the ability of the research organization to engage in meaningful patient par tnership. This readiness assessment allows patients to discern the research organization’s level of collaboration and willingness to share control.
Patient Group Readiness
Measures the ability of the patient organization to engage in meaningful collaboration. This readiness assessment allows research organizations to discern the level of expertise, collaborative culture, and diversity of the patient group.
DI M E N S IO N A N CH O R S
Non-Collaboration
Ideal Collaboration -2 -1 0 1 2 — New baseline of acceptable collaboration — Patient-informed — Patients as partners and/or patient-led
Minimal Collaboration
Acceptable Collaboration
Great Collaboration
Patient-Led Collaboration
SC O RECAR D R UB RIC S
PATIENT BURDEN
Research organization dictates engagement avenues with no consideration of the patient population's access needs.
Full participation may be impossible; carry a high time, effort, or monetary cost; or cause patients harm.
Based on extensive testing, a 5-point scale with an anchor at 0 for acceptable collaboration was selected.
The Patient Burden dimension includes the following domains:
• Accessible Engagement
• Trauma-Informed Practices
• Responsiveness to Patients
• Compensation
PATIENT BURDEN
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Research organization considers the patient population when designing engagement avenues, but rarely provides additional accommodations when requested.
Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, and provides individuals with additional accommodations upon request.
Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, ensures patients can easily request additional accommodations, and works with patients to co-design systemic updates in response to requests.
Patients co-create engagement avenues from the outset to ensure that full participation is accessible and minimally harmful across patient sub-populations.
Research organization does not consider the trauma burden to patients. Patients may experience discrimination, hostility, new or recalled trauma, or other harms as a result of participation. No trauma-informed practices are in place, and patients receive no resources or support for the trauma caused by participation.
SC O RI N G R UB RIC S
PATIENT BURDEN
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Research organization is aware of a possible trauma burden, but no systemic traumainformed practices are in place, and patients receive no resources or support for their trauma.
Research organization recognizes the trauma burden, and some trauma-informed practices are in place. Resources and support are provided to patients upon request.
No formal channels for patient input are established. Research organization does not address patient feedback, and may exclude or retaliate against patients who voice concerns.
PATIENT BURDEN
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Responsiveness to Patients Trauma-Informed Practices
Patients find channels for input to be unclear, difficult to access, or unsafe from retaliation. Patient feedback may be acknowledged, but rarely results in changes to the current study.
Research organization creates clear, accessible, safe channels for patient input only after the research process has begun. Patient feedback is acknowledged, resulting in changes to analysis, presentation, or communication; and ad-hoc changes to the current study.
Research organization implements traumainformed practices throughout the study, and collaborates with patients to co-design adjustments to those practices during the study. Requests for resources and support are honored at a systemic level for all patients.
A diverse array of patients, representative of the study's subpopulations, collaborates from the outset to co-create a safe, inclusive, mutually respectful environment; implement and adjust trauma-informed practices throughout the research process; and ensure all patients proactively receive sufficient, comprehensive resources and support.
Research organization creates clear, accessible, safe channels for patient input throughout the research process; acknowledges patient feedback; and establishes mechanisms for patients to co-design systemic changes to the current study.
Patients co-lead the study from end to end, including creating clear, accessible, safe channels for input, using that input to inform the research process, and acknowledging its impact. Members of the research organization are excited about and fully engaged in patient collaboration.
Patients are compensated below market rate for their domain expertise and experience level, with no or limited options for when and how they are paid. Expenses, harm, and risk assumed from participation are not compensated.
SC O RI N G R UB RIC S
PATIENT BURDEN
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Patients are compensated at market rate for their expertise and experience, with no or limited payment options. Expenses, harm, and risk are not compensated.
Compensation
Research organization sets patient compensation at market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront. Requests for additional compensation and/or payment options are honored ad-hoc.
Research organization sets patient compensation at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront. Requests for additional compensation and/or payment options result in systemic changes that benefit all patients.
Patients have decision-making roles in setting and adjusting compensation. Patients are compensated at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk; in the method and timing of their choice. Requests benefit all patients. Patients receive non-monetary compensation in the form of visibility, professional development, authorship, and awareness of their impact.
RESEARCH ORGANIZATION READINESS
SC O RI N G R UB RIC S
The Research Partner Readiness dimension includes the following domains:
• Recognition of Biases
• Collaboration Process
• Knowledge in Disease Subject
RESEARCH ORGANIZATION READINESS
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Recognition of Biases
Research organization does not recognize bias and ignores feedback from patients.
Research organization has limited awareness of own biases and listens to some feedback from patients.
Research organization is aware of own biases, is open to feedback from patients, and implements some of the feedback.
Research organization is aware of own biases and is open to feedback from patient group and actively iterates on feedback given.
Research organization is aware of own biases and is open to listening to feedback from patient group. Actively iterates on feedback given. Other patient groups can attest to a positive working relationship. Research organization has a systemic process for accepting input from patients and patient groups.
Research organization has no dedicated infrastructure for collaborating with patients.
SC O RI N G R UB RIC S
RESEARCH ORGANIZATION READINESS
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Collaboration Process
Research organization has minimal resources/infrastructure for collaborating with patients.
Research organization has dedicated some resources and infrastructure for collaborating with patients (ie. patient panels); has at least one coordinating personnel focused on meeting the patient group's needs; conducts limited training to build skills to engage with patients.
Research organization has an established infrastructure and process for collaborating and codesigning with patients including at least one dedicated person focused on meeting the patient group's needs and advocating to the rest of the research organization; conducts routine training to build skills to engage with patients.
Research organization has an established infrastructure and process for collaborating with patients that has been vetted by other patients/patient groups; has at least one dedicated person who is focused on meeting the patient group's needs. The partner is recognized as a patient ally vetted by other patients and patient groups with background in disability justice. Conducts extensive training on meaningful engagement with patients.
Research organization has no knowledge/ experience with the disease being researched
Research organization has minimal knowledge/experience (less than one year) with the disease being researched.
Research organization has at least one year worth of knowledge/ experience with the disease being researched.
Research organization has more than one year worth of knowledge/ experience of the disease being researched.
Research organization has extensive knowledge and direct experience with the disease being researched and those with knowledge are in decision-making roles. Research organization has a systemic way to keep on top of information from the patient community as well as the latest research.
SC O RI N G R UB RIC S
PATIENT/PARTNER GOVERNANCE
The Patient/Partner Governance dimension includes the following domains:
• Meaningful Decision-making between groups
• Accountability between groups
PATIENT/PARTNER GOVERNANCE
Decision-making for significant decisions (funding, study design, publication, etc.) is not communicated transparently and/or the research organization decides the decision making process without patient input.
Decision-making process for significant decisions (funding, study design, publication, etc.) is not communicated and/or agreed upon. Patients have limited or not meaningful decision-making power.
Decision-making process for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patients and research organization.
PATIENT/PARTNER
Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group.
Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group with sufficient support to make the decisions.
There is a lack of understanding of the rules of engagement/ culture between groups with no written agreement and no defined consequences for not following through.
There is an understanding of the rules of engagement/ culture but no written agreement and/or defined consequences for not following through between groups.
There is a shared understanding and written agreement of the rules of engagement/ culture with defined consequences for not following through between groups.
Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups. Deference is given to patient groups to define the engagement.
Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups. Deference is given to patient groups to define the engagement with sufficient support.
SC O RI N G R UB RIC S
INTEGRATION INTO RESEARCH PROCESS
The Integration into Research Process dimension includes the following domains:
• Hypothesis Generation
• Study Design
• Analysis
• Publication
• Attribution
INTEGRATION INTO RESEARCH PROCESS
Hypothesis Generation
Research goals are siloed from patients’ priorities. Patients' questions and experiences are not included and/or are dismissed when generating research hypotheses.
Research goals attempt to involve patients’ priorities, but limited by communication or collaboration. Patients' inquiries and lived experiences are rarely included when generating research hypotheses. Patients may have suggested the research question with no further involvement.
Research goals take into account patients’ priorities. Patients' inquiries and lived experiences are included when generating research hypotheses.
Research goals proactively address patients’ priorities with sufficient ongoing collaboration. Patient organization's inquiries and lived experiences are included when generating research hypotheses. Patient organizations work with patients to co-design research hypothesis.
Research goals are based on patients’ priorities and co-written by patient organization or patient-researchers. Patient's inquiries and lived experiences share an equal weight with research organization’s interests when generating research hypotheses.
Research organization does not include patients in the study design process. Patients do not have the opportunity to provide input on study design. Patient groups are utilized for recruitment purposes only, if at all.
SC O RI N G R UB RIC S
INTEGRATION INTO RESEARCH PROCESS
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Research organization does not include patients in the study design process. Patients may be invited to review study design but feedback is rarely incorporated and no functioning accountability system is in place.
Study Design
Select patient voices are approached to inform the study design. Patients are invited to review study design and have an impact on the study design.
Patient organization and their community’s input are proactively invited to help inform the study design. Patient organizations are invited to co-design and review study design and patient feedback changes the study design.
INTEGRATION INTO RESEARCH PROCESS
5-point scale (-2, -1, 0, 1, 2) by
Patients do not have input in what data to prioritize for analysis and methods of analysis.
Patients are asked to review manuscript drafts but have little say in what data to prioritize for analysis and methods of analysis.
Patients are involved in interpreting data and carrying out analysis in some capacity.
Patients or patient organizations are invited and involved in interpreting data and carrying out analysis anywhere in the study.
Study design is cowritten and reviewed by a diverse array of patient-researchers representative of the study's sub-populations. If applicable, protocol testing is done by the patient community.
Patient-researchers co-lead on the interpretation and analysis and/or work concurrently with partner organization’s research team to carry out analysis.
Study results are inaccessible to patients and/or behind an academic paywall. Findings are not communicated in lay terms.
SC O RI N G R UB RIC S
INTEGRATION INTO RESEARCH PROCESS
5-point scale (-2, -1, 0, 1, 2) by
Research organization summarizes findings in lay terms, but study results are inaccessible to patients and/or are behind an academic paywall.
Study results are freely accessible to patients and the public. Findings are summarized in lay terms in ways that are informative to the patient population.
Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population. Patient-researchers co-write the interpretation and analysis.
INTEGRATION INTO RESEARCH PROCESS
Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population. Patient organizations invite patients to co-write findings and reports. A channel of communication is available for patients to ask questions of the research organization.
Patients' work is attributed to others and/or patients are not attributed at all.
Patients are listed as being involved without a description of how they were involved. Patients were not consulted on how they prefer to be attributed.
Patients are acknowledged/credited in major public facing communication (press, announcements, papers), to the extent that patients wish to be named. Patients were consulted on how they prefer to be attributed.
Patient group is credited in all public-facing communication and included as authors on papers, to the extent that the patient group wishes to be named. Patient group was consulted on how they prefer to be attributed.
Patients are acknowledged specifically for what they did throughout the engagement process, are credited in all public-facing communication, and included as authors on papers, to the extent that the patient group wishes to be named.
Patient group was consulted on how they prefer to be attributed.
PATIENT GROUP READINESS
SC O RI N G R UB RIC S
The Patient Group Readiness Scorecard dimension includes the following domains:
• Expertise
• Bias and Representativeness
• Accountability
• Accommodation
• Culture of Collaboration
PATIENT GROUP READINESS
Patient group is not versed in research on condition or speaks on only a narrow representation of the condition. Patient group promotes research that is harmful to the community.
Patient group is not versed in research on condition or speaks on only a narrow representation of the condition.
Expertise
Patient group is up-to-date on research associated with condition and speaks on the diversity of the condition.
PATIENT GROUP READINESS
Patient group is up-to-date on research and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.
Patient group has done research on condition, is up-to-date on research, and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.
The majority of the patient group is not patients or caregivers who can speak on behalf of patients. Patient diversity isn't prioritized or intentionally planned for, as such, participation in collaboration is severely limited and patient group leaders may be gatekeepers of research opportunities.
Bias and Representativeness
The leadership of the patient group is not patients or caregivers who can speak on behalf of patients. The leadership is not demographically representative of the group they are representing. Participation in collaboration is limited to a select few members of the group.
Leadership of patient group are patients themselves or caregivers who can speak on behalf of patients. Leadership is demographically representative of the patient group they are representing. Patient diversity is prioritized and their participation is planned for.
The majority of the patient group are patients themselves or caregivers who can speak on behalf of patients. The majority of the patient group are representative of the group they are representing. Patient group prioritizes and surfaces views brought by on diverse patient population into research collaboration.
The entire patient group are patients or caregivers who can speak on behalf of patients. The group is completely representative of the group they are representing. Patient group is well versed on own biases and centers the views of underrepresented patient population.
Patient group has explicit conflicts of interest with greater community good. Patient group gaslights or bullies the patient population.
SC O RI N G R UB RIC S
PATIENT GROUP READINESS
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Patient group does not seek out or respond to feedback from the patient population. Patient group is opaque in their involvement of the collaboration.
Accountability
Patient group responds to the broader patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.
Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.
Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group is transparent in sharing research outputs as well as decision-making that affect the patient population.
PATIENT GROUP READINESS
Accommodation
Patient group doesn't acknowledge or accommodate access needs of the illness.
Patient group acknowledges access needs of the illness but does not accommodate them.
Patient group acknowledges access needs of the illness. Patient group accommodates most of the access needs of the illness when able.
Patient group acknowledges access needs of the illness. Patient group accommodates most of access needs of the illness when able. Patient group advocates for access needs of its members.
Patient group acknowledges access needs of the illness. Patient group accommodates all of the access needs of the illness when able.
Patient group advocates for access needs of its members and the wider patient population.
There are unsolvable disagreements within the group and/or the group often is in disagreement with other patient groups. Patient group has no agreed upon code of conduct and/or rules of engagement.
SC O RI N G R UB RIC S
PATIENT GROUP READINESS
5-point scale (-2, -1, 0, 1, 2) by subdimensions
Culture of Collaboration
There are concerning disagreements on core values and/or inequitable practices within the group and/or with other patient groups that have caused tension. Patient group has code of conduct and/or rules of engagement that is not followed.
Patient group is able to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.
Patient group has productive relationships with each other and with other groups. In the event of disagreements, patient group has a process to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.
Patient group has aligned values and practices and seamlessly collaborates with each other and with other groups. Patient group has policies developed to address collaboration dynamics with the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.
U SA B I L I T Y
Considerations
Given the goal of providing actionable information for improvement for users of the score- card, we will provide further guidance on how to use the scorecard. Furthermore, it will be important to consider usability by a variety of intended audiences, including what they want to know, and the dimensions that should be evaluated. Based on further testing, the playbook and subsequent training programs will link the score with potential next steps, including linkages to specific illustrative case studies. The example below illustrates how a patient group leader who is engaged in research may use the scorecard.
Going forward it will be important to test several assumptions regarding the scorecard and its use. Key issues to consider and assess are listed in the table below.
1. Who am I 2. What would I like to know?
Example:
Leader of a patient group
Assumptions to test: Are our user personas reflective of people’s realities and their intentions?
Are par tners treating me equitably by patient-led standards?
How can I advocate for better treatment?
How can I grow my team to take on more equitable collaborations?
Will people approach the scorecard with specific questions they want answered about their own organization or collaborators?
Is there an opportunity to narrow down what’s relevant to them?
(e.g., if the patient org doesn’t yet have capacity to do research, then cer tain dimensions won’t be applicable to them)
3. Dimensions to evaluate
Patient/Par tner Governance, Patient Burden, Integration into Research Process, Research Par tner Readiness
4. How did I score?
Patient/Par tner Governance: 4
Patient burden: 2
Integration into Research Process: 2
Research Par tner Readiness: 2
5. What are the next steps?
Addressing patient burden
What’s an organization similar to mine and how did they navigate to a better position?
Will people want to pick and choose which dimensions to assess? Will they use and score all of the recommended dimensions?
Are we overfitting or underfitting
situations with the sub-dimensions that we have?
Is scoring by dimension or or overall score meaningful?
Is there something we haven’t considered that should go into scoring?
Is the negative scoring helpful?
Do results need to be accompanied by some context?
(e.g., high patient organization composition could lead to team burnout)
Are they curious how the model is put together?
Are they curious about other dimensions?
Do they want more info on how they can do better?
C ON C L U SI ON S
This project moves beyond patient engagement toward a solution where patient-generated data and patient-led outcomes research become an essential component of medical research, leading to more patient-centric comparative effectiveness research. Patients and patient organizations, funders, research institutions and other traditional biomedical research teams can collaboratively build the infrastructure for patient-led CER. CMSS and PLRC have developed a sustainable collaborative model of CER based on information from and the expertise of patient communities, researchers, funders, and clinical research organizations.
In year two of the project, CMSS and PLRC developed CER training webinars aimed at clinical research organizations and specialty societies that were incorporated into virtual programming for physician leaders and clinical registries to ensure these tools are available to broad audience for widespread use.
Long COVID has presented a unique opportunity to integrate patient-led research into traditional approaches, advancing the field of patient-led outcomes research.
Capacity
G L O SSA R Y
Collaboration
Control
Decolonized Data
Engagement
Leading vs. Co-Leading
Most-impacted
Ownership
Patient Burden/Trauma
Patient-Led Research
Power
Readiness
Research Partner
Tokenization
Trauma-informed Research
The amount an or ganization possesses to interact with patients in research.
Working as co-researchers with patients at every stage of the research process so that patients contribute to study design, goals, r esearch questions, methodology, analysis, and reporting; power is shar ed by patients and traditional researchers.
The power to influence research and direct the research effort.
Moving away from making assumptions about a person’s social, economic and political contexts and towar ds active listening to the person to improve healthcare research; research often extracts and siloes data fr om the communities it studies.
Inv olving patients in research via a partnership; rather than tokenizing patients, patients are equal par tners contributing their expertise and lived experience which is valued equally with academic research expertise.
Patients leading research puts patients into decision-maker roles at every stage of the research process; co-leading occurs when patients and research partners each contribute their expertise, come to agreement on important decisions, and respect each other’s unique and valuable expertise.
Patients who have been most affected by an illness and/or societal structures.
The state of possessing, accessing, cr eating, and modifying data and controlling the information.
The perceived negative outcomes of a medical encounter, resulting from an increased healthcare workload and diminished capacity to manage the negativ e outcomes that manifest emotionally, relationally, physically, communicatively, and systemically. (from Nottingham K, Chadwick A. Conceptualizing a new definition of patient burden. The Annals of Family Medicine. 2022 Apr 1;20 (Supplement 1).).
Resear ch in which patients are the decision-makers at every stage of the process; the patient is driving the pr ocess and ensuring that those living with the disease are contributing their skills and exper tise and managing the effort.
The capacity to direct the course of research and influence the process and be seen as an equal par tner with valuable contributions to make.
An organization has the ability to engage in meaningful patient-researcher partnership; an or ganization has a deep understanding of patient-led research along with the infrastructure to collaborate meaningfully
The research team who is collaborating or wants to collaborate with a patient group; includes individual r esearchers and coordinating staff with direct contact with the patient group but does not extend to the entir e academic institution.
The practice of inv olving patients in research only for the purpose of appearing to engage patients.
Resear ch that is conducted with the recognition that a person with lived experience responds in ways that reflect adapting and coping with symptoms of trauma. This is based on understanding the eff ects of trauma and trauma triggers and making a connection with the person with lived experience to minimiz e harm.
BACKGROUND RESEARCH APPENDIX
Existing Models and Collaborative Frameworks
As part of the prototype model implementation, the project team has reviewed existing models and frameworks, rather than developing new model frameworks that do not add to existing models or advance the field in a unique way. A key aspect of the prototype model development focused on the characteristics of the PLRC that should be embedded in development of prototype models for meaningful engagement. For example, informants emphasized the nimbleness of the PLRC team and its governance structure which has allowed them to balance speed with scientific rigor. PLRC was able to advance its patient-led research quickly without sacrificing the quality and rigor of the evidence.
critical developmental element of patient-led research model development. The Advisory
Additionally, the project team has identified the engagement of BIPOC communities as a Panel recommended that the project team embed best practices in the engagement of diverse populations in patient-led research and CER throughout all deliverables. Key informants identified engagement of diverse communities as an area for improvement and the project team is committed to developing a model and tools that not only engage the BIPOC community but also enable them to drive decision-making. Transparency, trustworthiness, and prioritization of their needs are key components in building these critical relationships.
The project team reviewed the following models and collaborative frameworks and incorporated elements from each of the following sources into the prototype models of patient-led research:
• Learning Health Systems
• Arnstein ’s Ladder of Citizen Participation
• Design Strategies in Online Citiz en Science Platforms (Shirk, et al)
• F orms and Functions of Participation (Sarah White)
• F elicity Callard’s Model
• Resear ch Partnership Maturity Model for Patient Organizations (FasterCures)
APPENDIX
Baseline Assessment
As part of its baseline assessment, CMSS and PLRC conducted surveys of their respective memberships to identify areas for capacity building and key lessons learned regarding patient engagement. It was critical to understand perceptions of patient communities and traditional medical research entities on the integration of patient-led outcomes and CER into the broader research enterprise as well as to explore how patient and disease/condition characteristics could influence acceptability of patient-led outcomes research, including promises and limitations of using COVID-19 as the basis for sustainable infrastructure development for patient-led CER. Furthermore, CMSS and PLRC identified established and emerging patient communities (outside of COVID-19) and clinical research organizations with an interest and goal of aligning toward patient-led outcome research approaches and endeavored to understand best practices in the recruitment of diverse populations to patient-led outcomes research initiatives.
CMSS conducts an annual survey of specialty societies and clinical registry leaders
2021 SURVEY RESULTS and recently conducted its 2021 survey in June 2021. The survey was circulated to Quality Improvement and Registry Directors at the CMSS member societies and included focused questions on patient engagement. Respondents included 17 specialty societies representing 23 clinical registries.
66.67%
Have not yet engaged patients or patient groups in their registry and performance measurement efforts
46.67%
Are in the process of engaging patients in the selection of outcomes and patient-reported outcome-based performance measures (PRO-PMs)
This represents an opportunity for this collaborative project to provide CMSS members with much-needed tools and resources to ensure meaningful engagement of patients in these important initiatives.
APPENDIX
Shared Opportunities for Improvement
tion for all patients as a critical issue impacting patient-led research. Patients are often
In conducting key informant interviews, CMSS and PLRC identified burden of participadealing with medical crises, limited energy, brain fog, and other limitations that hinder their time and energy to participate fully in research. BIPOC communities are particularly impacted by these issues and often carry with them the burden of historical abuse which has severely impacted their trust in research and research institutions.
Researchers must acknowledge these barriers facing patients and identify ways to accommodate those living with the disease. For example, requiring patients to participate in weekly conference calls during work hours can be problematic. Patients may have other jobs or may not have the energy to participate in lengthy conference calls at a pre-scheduled time.
Furthermore, researchers are often inviting patients to revisit their trauma but then do not acknowledge it or assist in addressing it in any way Patient surveys may not be reviewed by a social worker or therapist who could minimize harm from the trauma.
Advances in technology can address some, but not all, of these issues. For example, patients can be invited to participate on their schedule, when their time and energy allow. Patient surveys can be completed at any time. Asynchronous work can be done in collaborative digital environments, such as Google docs, Slack, etc.
Patients may want to be engaged in research on their own terms. That is, they may not be able to take on the burden of co-authorship but can contribute meaningfully in other ways.
Additionally, the messenger can be as important as the message itself. BIPOC communities must feel welcomed and represented if they are to be open to and accept the research findings. It is critical to have a diverse team that can engage diverse communities and prioritize the needs of these communities. Building trust with BIPOC communities can take time to establish reciprocal terms of partnership and create an equitable environment for meaningful collaboration.
APPENDIX
Scorecard Dimensions Development
The dimensions represented in the nested scorecard build on the themes, models, and examples of patient engagement in research across a wide continuum. The intent of the scorecard is to identify models of engagement with patients in research that are unacceptable, minimally acceptable, and moving toward ideal levels of collaboration with patients.
The definitions of the scorecard dimension will remain the same across different users.
However, different users may use the scorecard for different purposes:
• Resear ch organizations may be interested in baseline assessment and strategies to move toward “patient-led” models;
• Resear ch funders may use the scorecard to evaluate partner organizations and/or patient groups seeking funding for patient-led research;
• Patients and patient groups may use the tool to assess their readiness in leading research and/or evaluating potential partner research organizations.
Existing toolkits and scorecards were examined to identify potential models that would achieve the goals of the project, including:
• IDEO Human Center ed Design Thinking Toolkit
• Rural Health Community Toolkit
• NHC Patient Experience Journey Map
• IBD Sur gery Decision-Making Tool
• Community Resilience Toolkit
To meet the goals of driving patient-led research, the scorecard output is intended to:
• Establish a new baseline of acceptable patient engagement
• Provide a shared language between stakeholders
• Center people and stories
• Star ts small with room to iterate and expand
• Actionable and accessible
The scor ecard is intended to be both evaluatory (i.e., evaluate partnerships from a patient-led perspective) and instructive (i.e., how to ideally support and collaborate on patient-led research as a patient or collaborator). Therefore, the output of the scorecard may differ to ensure that it is useful and accessible for each audience.
APPENDIX Development
The initial scorecard prototype was developed by the team following a synthesis of primary and secondary research. First, we gathered insights from an initial round of 12 key informant interviews, as well as feedback from a survey shared among members of Patient-Led Research Collaborative. We also conducted a literature review of existing models, and evaluated several types of frameworks, their usefulness, and gaps. We decided on a scorecard model for its simplicity and immediate deployability in addressing collaboration dynamics between patients / patient orgs with researchers / partner orgs.
The team conducted a round of testing on this version of the Patient-Led Collaboration Scorecard. The goal of the testing was to answer the following questions to help refine the scorecard by its focus and real-world use as well as to inform the next iteration.
• Understanding of the targeted Users
o Who should we prioritiz e as users?
o When would this be used?
o What use cases should we prioritiz e?
o Will the users b y self-evaluating or others or both?
• Scoring Models
o What scoring models would be most eff ective given the use cases?
o How do we define each score? Clarity and messaging of each score
o What drawbacks ar e involved with scoring? Behavioral considerations
o Is negativ e scoring helpful?
o Is scoring b y dimension / overall score meaningful? Is there something we haven’t considered that should go into scoring?
• Dimensions
o Which dimensions ar e a high priority to focus? By needs and expertise
o Are we overfitting or underfitting situations with the sub-dimensions that we have?
o Are participants curious about other dimensions?
• Results
o Do results need to be accompanied by some explanatory language?
o Do the y want more information on how they can do better?
o What outputs/outline do we cr eate for next steps?
• Overarching model output
o What could we do with our entire model to improve?
APPENDIX
Methodology for Testing & Validation of Model
The team conducted a combination of asynchronous and synchronous directed user testing sessions to gather feedback on the prototype. To explore emerging use cases and account for gaps, the team recruited three different types of users to test the scorecard based on their background and context of usage:
• Patient community leaders in Long COVID and chronic illnesses
• Resear chers with a variety of exposure to patient-led research based on the team’s assessment
• F unders of health research
The team collected feedback from 11 individuals in addition to a group feedback session from the advisory panel, and a group feedback form from PCORI officers which was composed of multiple responses. In total, the team collected feedback from 20 potential users. Additionally, the team consulted with a behavioral scientist on best practices for scoring. The team synthesized the feedback and responses into actionable updates to be made to the model for this version and future iterations of the model.
The following points are a summary of the main areas of feedback, how we addressed them, and next steps:
Feedback
Users/Use Cases:We drew two distinct use-cases of the scorecard from testers’ feedback
1. Patients evaluating par tner organizations feel that they can use the scorecard as a way to facilitate conversations to advocate for better collaboration standards, and develop a needs assessment when entering collaborations
2. Testers who were champions of patient involvement within research organizations are keen to use the scorecards evaluate their orgs and use the feedback to make adjustments on future studies
Scoring model: Testers had varying opinions on what scoring model would be most effective (Liker t scale 1-5; red/ yellow/green; A, B, C, D, F; keeping as is). Testers, as well as the behavioral scientist, indicated that the messaging/ titles of the scores seem to be the most impor tant aspect in addition to what advice is given once someone is assigned a score.
Updates/Next steps
• Based on the type of users who had given feedback, we prioritized our effor ts in this version on scorecards that address patient-burden
• Next version consider creating different user experiences for the patient and the patient champion within par tner organizations
• Given that we received mixed input on the scoring model, we will maintain the current scoring model with some changes to the text and language
• We will consider testing other scoring models and language to determine effectiveness
APPENDIX
Methodology for Testing & Validation of Model
Feedback Updates/Next steps
Dimensions: Generally, the majority of testers found the dimensions comprehensive and intuitive.
The testers prioritized the following scorecards over others as being more relevant to their experience: Patient Burden, Integration into Research Process, and Governance.
Data Collection and Ownership was also prioritized, but several reviewers indicated that it required more contextual nuance than what was included and may not be applicable to every type of research study.
A few testers mentioned that there were too many dimensions and it was overwhelming to navigate them. In addition, a few other dimensions and subdimensions were mentioned as ones they expected to be included but were not such as: knowledge of disease (under Par tner Organization Readiness), ethics, and intentions of research.
Some testers also commented on the lack of clarity of the language used to describe some of the dimensions, such as “ par tner organization” as being too broad.
Each Dimension Scorecard:The testers provided feedback on each individual scorecard, suggesting language updates, changes to subdimensions, etc.
Results & Next Steps Section: Testers suggested developing clear, actionable recommendations for improvement. They suggested including checklists as a useful tool. Several cited case studies, indicating how they would help users understand how to use the scorecards in different scenarios.
• Based on the feedback and prioritization provided by par ticipants, the team decided to move forward with 4 of the 7 initially proposed dimensions for this iteration: patient burden, integration into research process, governance, and research par tner readiness. These were chosen based on feedback that they were comprehensive and useful to testers.
• The team updated the language describing the dimensions and updated the glossar additional definitions. y to include
• The patient readiness scorecard was added as a result of feedback provided during the training webinars.
• The team will consider further work on the other dimensions in future iterations.
• The team updated the prioritized scorecards with suggested edits.
• The team will develop recommendations for ways to improve the user’s score so that the scorecard can be used for evaluation, as well as improvement.
• Additional tools may be identified and developed during implementation.
Overall Feedback on Model:Testers provided more generally positive feedback on the model and prototype. Many testers agreed with the model logic and found the tools intuitive to use. However, a few points of constructive feedback for the team to consider were the following:
• Some commenters noted that the language was at times judgmental and unclear
• Navigating and going through multiple scorecards felt complicated/overwhelming
• How do we ensure that these scorecards are taken seriously by the research community?
• The team updated some of the language so it is less “judgmental” and provided a comprehensive glossary – This will help clarify the unclear language.
• The scorecards will be presented differently in the final output and we reduced the number of dimensions for evaluation, which will help with navigating the scorecards
• The team will continue iterating on the model
APPENDIX
Key Informant Discussion Guide
Introduction
Thanks for taking the time to speak with us today. I am ____, and my co-researcher ____ is also on the call listening-in and taking notes.
We are with PLRC and CMSS working on a PCORI funded project to look at patient-led research models and explore the key components of a successful model of collaboration between traditional research and patient-led research organizations. PCORI is an independent, nonprofit organization authorized by Congress in 2010 to fund comparative effectiveness research that will provide patients, their caregivers, and clinicians with the evidence needed to make better-informed health and healthcare decisions. PCORI is committed to seeking input from a broad range of stakeholders to guide its work.
We will develop tools and resources for traditional research entities and patient-led participatory research communities to share the PLRC’s successes in collaboration as well as help communities overcome the challenges in initiating patient-led outcomes research teams. To that end, your input is critical in helping us fully understand models of collaboration.
[As you know, you were recommended to us by___.] We will be asking you questions about your experience as a (Researcher/Patient/Funder, X) and will have a general guided discussion with you on the topic of patient-led research.
Just so you know, anything you say won’t directly be attributed to you or your organization without explicit permission for something we may want to quote you on.
**Request to Record Interview**
Before we begin, can I ask for your permission to record this interview? It will only be used by us, to make sure we heard everything.
Do you have any questions before we start?
**Notes to interviewer**
What to look for to guide analysis
- action and consequences (I did X and Y happened)
- ask followup q’ s about consequences, might lead to feelings, opinions, values
- feelings (when X happened, I felt Y)
- opinions (I think X is Y)
- dig deeper, might lead to a guiding principle
- guiding principles / values / motives (I value X, so I did Y)
APPENDIX
Key Informant
Discussion Guide
This par t of the guide will give us context on their background and how it impacts and influences their answers. Manage this discussion to keep shor t and to the point and doesn’t take up too much time.
A. Role/Background
• Tell us about your role or experience as a (Patient, researcher, funder).
• Can you briefly tell us what does your organization do? What do you do for them? For how long?
• What led y ou to this work/experience?
This par t of the guide is to understand their perceptions of PLRC. Make sure to tell them we want to hear their unbiased opinions so we can improve on models for Patient-led Research.
B. General Impressions on Patient-Led
• Have you heard of PLRC before (Patient -led research collaborative)?
• What was your first interaction with them? Or when did you first hear of them?
• If yes, What were your first and now general impressions of them? (Probe for positive/ negative impressions) What do you think they did right/doing well? What do you wish they did better or more of?
• Have you heard or seen of other patient-led research initiatives? Which ones? What impressed you about them? What did these groups do well/not so well or struggle with?
• How would y ou differentiate between patient-led and patient-centered research?
This par t of the guide is to understand patients’ interaction with understanding their illness and the research around it and their involvement. If they were ever involved in research, try to understand frustrations, feelings, motivations of being involved in research.
If Patient
• Tell us about your illness? What are areas that need research or are under researched for your illness or other related ones?
• Where do you see gaps in the progress of research for your illness?
• Have you participated in research? In what capacity?
• What was the experience lik e?
– What was positiv e about it?
– Any challenges? What wer e the challenges?
– What did y ou wish was available to you?
• Have you collaborated with other traditional researchers? How was the experience?
– What was positiv e?
– What was challenging?
– What did y ou wish was different or could be better?
• (Important) Have you involved other patients from your community in the research? Have you seen others who have?
• Do y ou consider yourself an advocate? How so? Has that impacted your research? If not, why not?
APPENDIX
Key Informant Discussion Guide
This par t of the guide is to understand medical research and how they involve patients. If a patient, who initiated medical research and didn’t ask above, check and see if they involved other patients. How? What obstacles/challenges did they face and what went well.
If Researcher
• What is y our main focus of your research work?
• What r esearch work are you most proud of? Tell us why?
• Have you collaborated with (other) patients? Or has your organization? In what capacity?
• What went well? What didn ’t go well?
• What do y ou wish was different?
This par t of the guide is to understand how minorities/BiPOCS were involved in medical research. If they have not done it themselves, try to probe into who they have heard about that did it.
Involving Different Communities (BIPOC/Etc.)
• As a researcher have you involved BIPOC communities or other communities typically not included in research?
• Tell us about that experience?
• Where have you seen this done well?
• What challenges ha ve you seen in engaging these communities? (if needed, ask about skill building, ownership credit, trust)
• Where are there other types of research who have done in this area?
out where there are examples of it and what has enabled it. Also, if regular medical research
This part of the guide is to dig into funding of medical research led or with patients. Try to find or funders of medical researchers dig into understanding how the space may be navigated by new researchers seeking funding.
If Funder/Or knowledge about funding?
• Can y ou give examples when patient researchers have received funding?
• What ar e factors that may have triggered or hindered research efforts related to patients being allocated resources (i.e. digital spaces, number of patient/researcher intersections, crowdsourcing funding models, etc.)?
• (If medical researcher) What issues related to funding of research are “ common knowledge” pitfalls to academic or medical researchers yet patient groups may not be aware of (i.e., organization qualifications to apply, conflicts of interest, disclosures, etc.)?
APPENDIX
Key Informant Discussion Guide
The goal of this par t of the guide is to understand their values and opinions when it comes to research and who/where it comes from.
C. General Research perception
• Do you follow medical research? What kind? How do you find good sources of research?
• Are there sources you trust/don’t trust more than others? Why?
• How do y ou evaluate the quality of research? What do you look for?
• Where do you see gaps in the medical research field?
• What is the ideal le vel of patient involvement in research? Wwhy?
• Is it impor tant to distinguish research and advocacy?
The goal of this par t of the guide is to get more leads on folks or resources. Also, into what areas we should maybe add to our research based on interest.
Research Leads/Other
• What ar e some research questions that you might have for yourself on this topic of patient-led research and models for engaging patients in research?
• Who do y ou think would be good folks to speak to on this topic? What resources should we check out?
• Are there any communities, organizations, or people that you feel are usually left out of these conversations? ( We’d like to include them if so!)
Wrap-up
Thank you so much for your time. Do you have any questions for us? What else do we need to know?
If any other thoughts come up on this topic, please feel free to reach out via email.
C M S S / PL R C Project Advisory Panel APPENDIX
Nisreen Alwan, MBChB MRCP
FFPH FHEA PGCAP MPH
MSc PhD
Donna Cryer, JD
Rachael Fleurence, PhD
Susannah Fox
Janna Friedly, MD, MPH
Harlan Krumholz, MD, SM
Associate Professor of Public Health, University of Southampton; Leading researcher on Long COVID; Awarded MBE for services to Medicine and Public Health during the COVID19 pandemic in UK
Founder, President and Chief Executive Officer of the Global Liver
Institute; Public Member, CMSS Board of Directors
Special Assistant to the NIH Director, COVID-19 Diagnostics; Former Program Director; PCORNet, Patient-Centered Outcomes Research Institute (PCORI)
Principal, Internet Geologist LLC; Former Chief Technology Officer,
US Depar tment of Health and Human Services
Professor, Vice Chair for Clinical Affairs, University of Washington
Harold H. Hines, Jr., Professor of Medicine and Epidemiology and Public Health at Yale University School of Medicine; Director, Center for Outcomes Research and Evaluation (CORE); Founder, Hugo Health
Fiona Lowenstein
Monica L. Lypson MD, MHPE
Greg Mar tin, MD, MSc
Karla Monterroso
Dona Kim Murphey, MD, PhD
Jaime Seltzer, MS
Craig Spencer, MD, MPH
Founder + Editor-in-Chief Body Politic
Vice Dean for Education, Vagelos School of Physician and Surgeons, Columbia University
President, Society of Critical Care Medicine; Professor and Master Clinician, Pulmonary, Allergy, Critical Care and Sleep Medicine, Emory University School of Medicine; Director, Georgia CTSA Clinical Research Centers; Research Director, Emory Critical Care Center
CEO, Code 2040, non-profit organization focused on Black and COVID patient
Latinx par ticipation and leadership in the innovation economy; Long
Director of Scientific Affairs, Corticare; Long COVID Patient
ME/CFS patient advocate and Director of Scientific and Medical Outreach at MEAction
Director of Global Health in Emergency Medicine at New YorkPresbyterian/Columbia University Medical Center
Advisory Panel Survey Results
Advisory Panel Survey Results
Advisory Panel Survey Results
Advisory Panel Survey Results
APPENDIX
Research organization dictates engagement avenues with no consideration of the patient population's access needs. Full participation may be impossible; carry a high time, effort, or monetary cost; or cause patients harm.
Research organization does not consider the trauma burden to patients. Patients may experience discrimination, hostility, new or recalled trauma, or other harms as a result of participation. No trauma-informed practices are in place, and patients receive no resources or support for the trauma caused by participation.
Research organization considers the patient population when designing engagement avenues, but rarely provides additional accommodations when requested.
Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, and provides individuals with additional accommodations upon request.
Research organization is aware of a possible trauma burden, but no systemic trauma-informed practices are in place, and patients receive no resources or support for their trauma.
Research organization recognizes the trauma burden, and some trauma-informed practices are in place. Resources and support are provided to patients upon request.
Research organization designs engagement avenues to offer sufficient time and accessibility for the patient population's needs, ensures patients can easily request additional accommodations, and works with patients to co-design systemic updates in response to requests.
Research organization implements trauma-informed practices throughout the study, and collaborates with patients to co-design adjustments to those practices during the study. Requests for resources and support are honored at a systemic level for all patients.
Patients co-create engagement avenues from the outset to ensure that full participation is accessible and minimally harmful across patient sub-populations.
No formal channels for patient input are established. Research organization does not address patient feedback, and may exclude or retaliate against patients who voice concerns.
Patients find channels for input to be unclear, difficult to access, or unsafe from retaliation. Patient feedback may be acknowledged, but rarely results in changes to the current study.
Patients are compensated below market rate for their domain expertise and experience level, with no or limited options for when and how they are paid. Expenses, harm, and risk assumed from participation are not compensated.
Patients are compensated at market rate for their expertise and experience, with no or limited payment options. Expenses, harm, and risk are not compensated.
Research organization creates clear, accessible, safe channels for patient input only after the research process has begun. Patient feedback is acknowledged, resulting in changes to analysis, presentation, or communication; and ad-hoc changes to the current study.
Research organization sets patient compensation at market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront. Requests for additional compensation and/or payment options are honored ad-hoc.
Research organization creates clear, accessible, safe channels for patient input throughout the research process; acknowledges patient feedback; and establishes mechanisms for patients to co-design systemic changes to the current study.
Research organization sets patient compensation at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk. Multiple payment options are offered upfront. Requests for additional compensation and/or payment options result in systemic changes that benefit all patients.
A diverse array of patients, representative of the study's sub-populations, collaborates from the outset to co-create a safe, inclusive, mutually respectful environment; implement and adjust traumainformed practices throughout the research process; and ensure all patients proactively receive sufficient, comprehensive resources and support.
Patients co-lead the study from end to end, including creating clear, accessible, safe channels for input, using that input to inform the research process, and acknowledging its impact. Members of the research organization are excited about and fully engaged in patient collaboration.
Patients have decision-making roles in setting and adjusting compensation. Patients are compensated at or above market rate for their expertise and experience; and for anticipated expenses, harm, and risk; in the method and timing of their choice. Requests benefit all patients. Patients receive non-monetary compensation in the form of visibility, professional development, authorship, and awareness of their impact.
Decision-making for significant decisions (funding, study design, publication, etc.) is not communicated transparently and/or the research organization decides the decision making process without patient input.
There is a lack of understanding of the rules of engagement/culture between groups with no written agreement and no defined consequences for not following through.
Decision-making process for significant decisions (funding, study design, publication, etc.) is not communicated and/or agreed upon. Patients have limited or not meaningful decision-making power.
Decision-making process for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patients and research organization.
1
Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group.
Decision-making for significant decisions (funding, study design, publication, etc.) is well communicated and agreed upon between patient and partner group, with deference given to patient group with sufficient support to make the decisions.
There is an understanding of the rules of engagement/culture but no written agreement and/or defined consequences for not following through between groups.
There is a shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups.
Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups. Deference is given to patient groups to define the engagement.
Shared understanding and written agreement of the rules of engagement/culture with defined consequences for not following through between groups. Deference is given to patient groups to define the engagement with sufficient support.
Research Organization Readiness
Research organization does not recognize bias and ignores feedback from patients
Research organization has no dedicated infrastructure for collaborating with patients.
Research organization has limited awareness of own biases and listens to some feedback from patients.
Recognition of Biases
Research organization is aware of own biases, is open to feedback from patients, and implements some of the feedback.
Research organization is aware of own biases and is open to feedback from patient group and actively iterates on feedback given
Research organization has minimal resources/infrastructure for collaborating with patients.
Collaboration Process
Research organization has dedicated some resources and infrastructure for collaborating with patients (ie. patient panels); has at least one coordinating personnel focused on meeting the patient group's needs; conducts limited training to build skills to engage with patients.
Research organization has an established infrastructure and process for collaborating and codesigning with patients including at least one dedicated person focused on meeting the patient group's needs and advocating to the rest of the research organization; conducts routine training to build skills to engage with patients.
Research organization is aware of own biases and is open to listening to feedback from patient group. Actively iterates on feedback given. Other patient groups can attest to a positive working relationship. Research organization has a systemic process for accepting input from patients and patient groups.
Research organization has an established infrastructure and process for collaborating with patients that has been vetted by other patients/patient groups; has at least one dedicated person who is focused on meeting the patient group's needs. The partner is recognized as a patient ally vetted by other patients and patient groups with background in disability justice. Conducts extensive training on meaningful engagement with patients.
Research organization has no knowledge/experience with the disease being researched
Research organization has minimal knowledge/experience (less than one year) with the disease being researched.
Knowledge in Disease Subject
Research organization has at least one year worth of knowledge/experience with the disease being researched.
Research organization has more than one year worth of knowledge/experience of the disease being researched.
Research organization has extensive knowledge and direct experience with the disease being researched and those with knowledge are in decision-making roles. Research organization has a systemic way to keep on top of information from the patient community as well as the latest research.
Research goals are siloed from patients’ priorities. Patients' questions and experiences are not included and/or are dismissed when generating research hypotheses.
Research organization does not include patients in the study design process. Patients do not have the opportunity to provide input on study design. Patient groups are utilized for recruitment purposes only, if at all.
Patients do not have input in what data to prioritize for analysis and methods of analysis.
Study results are inaccessible to patients and/or behind an academic paywall. Findings are not communicated in lay terms.
Research goals attempt to involve patients’ priorities, but limited by communication or collaboration. Patients' inquiries and lived experiences are rarely included when generating research hypotheses. Patients may have suggested the research question with no further involvement.
Research organization does not include patients in the study design process. Patients may be invited to review study design but feedback is rarely incorporated and no functioning accountability system is in place.
Patients are asked to review manuscript drafts but have little say in what data to prioritize for analysis and methods of analysis.
Research goals take into account patients’ priorities. Patients' inquiries and lived experiences are included when generating research hypotheses.
Research organization summarizes findings in lay terms, but study results are inaccessible to patients and/or are behind an academic paywall.
Select patient voices are approached to inform the study design. Patients are invited to review study design and have an impact on the study design.
Research goals proactively address patients’ priorities with sufficient ongoing collaboration. Patient organization's inquiries and lived experiences are included when generating research hypotheses. Patient organizations work with patients to co-design research hypothesis.
Research goals are based on patients’ priorities and co-written by patient organization or patient-researchers. Patient's inquiries and lived experiences share an equal weight with research organization’s interests when generating research hypotheses.
Patients' work is attributed to others and/or patients are not attributed at all.
Patients are listed as being involved without a description of how they were involved. Patients were not consulted on how they prefer to be attributed.
Patients are involved in interpreting data and carrying out analysis in some capacity.
Study results are freely accessible to patients and the public. Findings are summarized in lay terms in ways that are informative to the patient population.
Patient organization and their community’s input are proactively invited to help inform the study design. Patient organizations are invited to co-design and review study design and patient feedback changes the study design.
Patients or patient organizations are invited and involved in interpreting data and carrying out analysis anywhere in the study.
Study design is co-written and reviewed by a diverse array of patient-researchers representative of the study's sub-populations. If applicable, protocol testing is done by the patient community.
Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population. Patient-researchers co-write the interpretation and analysis.
Patient-researchers co-lead on the interpretation and analysis and/or work concurrently with partner organization’s research team to carry out analysis.
Study results are freely accessible to patients and the public. Findings are summarized in lay terms and are actively disseminated to patient population. Patient organizations invite patients to co-write findings and reports. A channel of communication is available for patients to ask questions of the research organization.
Patients are acknowledged/ credited in major public facing communication (press, announcements, papers), to the extent that patients wish to be named. Patients were consulted on how they prefer to be attributed.
Patient group is credited in all public-facing communication and included as authors on papers, to the extent that the patient group wishes to be named. Patient group was consulted on how they prefer to be attributed.
Patients are acknowledged specifically for what they did throughout the engagement process, are credited in all public-facing communication, and included as authors on papers, to the extent that the patient group wishes to be named. Patient group was consulted on how they prefer to be attributed.
Patient group is not versed in research on condition or speaks on only a narrow representation of the condition. Patient group promotes research that is harmful to the community.
Patient Group Readiness
The majority of the patient group is not patients or caregivers who can speak on behalf of patients.
Patient diversity isn't prioritized or intentionally planned for, as such, participation in collaboration is severely limited and patient group leaders may be gatekeepers of research opportunities.
Patient group is not versed in research on condition or speaks on only a narrow representation of the condition.
Patient group is up-to-date on research associated with condition and speaks on the diversity of the condition.
The leadership of the patient group is not patients or caregivers who can speak on behalf of patients. The leadership is not demographically representative of the group they are representing. Participation in collaboration is limited to a select few members of the group.
Leadership of patient group are patients themselves or caregivers who can speak on behalf of patients. Leadership is demographically representative of the patient group they are representing. Patient diversity is prioritized and their participation is planned for.
Patient group is up-to-date on research and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.
Patient group has explicit conflicts of interest with greater community good. Patient group gaslights or bullies the patient population.
Patient group does not seek out or respond to feedback from the patient population. Patient group is opaque in their involvement of the collaboration.
Patient group doesn't acknowledge or accommodate access needs of the illness.
Patient group acknowledges access needs of the illness but does not accommodate them.
Patient group responds to the broader patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.
The majority of the patient group are patients themselves or caregivers who can speak on behalf of patients. The majority of the patient group are representative of the group they are representing. Patient group prioritizes and surfaces views brought by on diverse patient population into research collaboration.
Patient group has done research on condition, is up-to-date on research, and has existing expertise (doing research, disability justice background) and includes diverse patient experts of the illness.
The entire patient group are patients or caregivers who can speak on behalf of patients. The group is completely representative of the group they are representing. Patient group is well versed on own biases and centers the views of underrepresented patient population.
There are unsolvable disagreements within the group and/or the group often is in disagreement with other patient groups. Patient group has no agreed upon code of conduct and/or rules of engagement.
There are concerning disagreements on core values and/or inequitable practices within the group and/or with other patient groups that have caused tension. Patient group has code of conduct and/or rules of engagement that is not followed.
Patient group acknowledges access needs of the illness.
Patient group accommodates most of the access needs of the illness when able.
Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group advocates for sharing research outputs and is transparent in their involvement of the collaboration.
Patient group acknowledges access needs of the illness. Patient group accommodates most of access needs of the illness when able. Patient group advocates for access needs of its members.
Patient group proactively seeks feedback from the patient population and other patient groups of related illnesses. Patient group is transparent in sharing research outputs as well as decision-making that affect the patient population.
Patient group is able to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.
Patient group has productive relationships with each other and with other groups. In the event of disagreements, patient group has a process to work through any disagreement within the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.
Patient group acknowledges access needs of the illness. Patient group accommodates all of the access needs of the illness when able. Patient group advocates for access needs of its members and the wider patient population.
Patient group has aligned values and practices and seamlessly collaborates with each other and with other groups. Patient group has policies developed to address collaboration dynamics with the group and with other groups. Patient group has code of conduct and/or rules of engagement that is followed.
Patient Group Evaluating Potential Partner
USE CASE
A patient group is considering partnering, or is in the middle of a partnership, with a research team on a study.
Research team asks representatives from patient group to be present on advisory calls and help with recruiting research participants, but is open to suggestions on working with the patient group.
SUGGESTED USE OF SCORECARDS
If the collaboration is in its planning phase, all 4 scorecards can help guide discussion around shared expectations. The Research Organization Readiness scorecard helps patient groups determine whether or not to partner and communicate which changes can be made for a fair collaboration.
The Integration into Research Process scorecard can help the patient group be aware of their nature of involvement and enable parties to discuss and agree on acceptable and ideal levels of collaboration in the study.
The patient group can use the Patient Burden and Patient/Partner Governance scorecard during or after the completion of the collaboration to help assess own involvement and suggest improvements going forward.
IDEAL OUTCOMES
Patient group is better equipped to decide on who to partner with.
Patient group knows which areas to ask for decision-making power, resources and accommodations, and can quickly align with partner on expectations going into a collaboration.
Patient group has more resources and support to lead on initiatives where they have expertise, and develop additional capacities alongside partners.
EXAMPLE
Research Partner Evaluating Themselves
USE CASE
A research team who is working with patients or patient groups is looking for feedback and ways to improve on involving patients going forward.
Research team has some capacity for patient engagement and would like to set aside resources to do so early on.
IDEAL OUTCOMES
The Research Partner applies for funding that prioritizes scope and budget for meaningful patient involvement
Research Partner proactively addresses the needs of patient group, and the study is made strong by patients’
SUGGESTED USE OF SCORECARDS
If the research project is in its planning phase, all 4 scorecards can help guide grant-writing and establish mutual collaboration with patient groups.
If the study is already underway, the research group can use the Research Organization Readiness and Patient-Partner Governance scorecards to help assess the level of meaningful engagement and implement improvements for a more effective and sustainable collaboration with patients
The research team can use the Integration into Research Process and Patient Burden scorecard to find ways to meaningfully engage with patients and plan for accommodations.
https://bmjopen.bmj.com/content/12/3/e060826
Funder Evaluating Research Partner
USE CASE
A funder wants to support research projects that are directed by patients or at least have meaningful engagement of patients to ensure the best and most patient-focused outcomes.
The funder has the ability to set criteria for rating proposals, add patients to their review panels, and is willing to set aside adequate funds for meaningful patient engagement.
SUGGESTED USE OF SCORECARDS
Funder uses all 4 scorecards to develop criteria for evaluating proposals.
Funder shares all 4 scorecards with applicants while they’re developing proposals.
Funder evaluates proposals based on how well the proposals score on each of the 4 scorecards
As funded projects launch, funder checks in on each project to ensure execution of patient-led research plan by evaluating projects using all 4 scorecards
Funder uses the Research Organization Readiness, Patient-Partner Governance, and Patient Burden scorecards to evaluate their own engagement of patients in the review process.
IDEAL OUTCOMES
Funder incentivizes patient-led research, enabling researchers to have the capacity to properly engage patients.
Funder prioritizes research partners that value patient-led research and have a plan and capacity to meaningfully engage patients.
Funder can help give guidance on how to meaningfully engage patients.
Funder meaningfully engages patients in their evaluation of proposals.
