Program
Day 1: October 26
Day 2: October 27

Presentations - October 26

Keynote Presentation

Integrating analytics, epidemiology and operational insights for real-world impact

Dr. Laura Rosella and Shalu Bains

There is a clear need for epidemiological and analytics expertise to solve operational problems in the hospital. This was brought to the forefront during COVID-19, when there was the wide availability of public health data and several models created; however, these data and analytic outputs did not provide the insights needed to inform action in the hospital for critical operational decisions in real-time. Furthermore, in many hospital settings, there is a disconnect between complex analytics and models that were academically interesting yet disconnected from operational reality, which limited the ability for real-world impact. We present a use case where we overcame this disconnect in Canada’s largest volume hospital. This presentation brings unique perspectives from both the research and operational side to describe the necessary building blocks to integrate diverse data sources, epidemiology, data and operational domains to solve pressing hospital issues. We will reflect on our COVID-19 experience as a case study where learnings can be adapted for other pressing health systems and population health issues. The session will include an interactive discussion and a scalable approach that addresses the structures and processes needed to enable population health analytics to achieve maximal impact for health systems.

Real-world Implementation Insights

Designing an LHS-Informed and Theory-Guided Audit and Feedback Intervention to Improve Surgical Incident Reporting: A Qualitative Case Study

Dr. Rama Mwenesi Musalia

Background: Patient safety errors are often underreported, with national rates as high as 86-95%. To address this issue, audit and feedback (A&F) is recommended as a key strategy  to reduce underreporting and promote clinician engagement. However, existing A&F interventions lack theoretical guidance and evidence on how they improve reporting, particularly in surgical learning health systems (LHS). This study aimed to  design an LHS-informed and theory-guided A&F intervention with specific mechanisms of action to enhance incident reporting practices among surgical staff. The focus was on surgical count-related errors, (defined as unintentionally retained surgical items and their near-misses) as they represent the top patient safety concern for perioperative nurses at U.S. academic medical centers.
Methods: This qualitative case study took place at a single U.S. academic medical center and was guided by the Clinical Performance Feedback Intervention Theory (CP-FIT) which suggests that the effectiveness of A&F is influenced by recipient characteristics, contextual factors, and the feedback process itself. We used data from a previous assessment of implementation determinants and explored the LHS infrastructure for surgical incident reporting to identify recipient and context-related factors. Semi-structured interviews were conducted with 20 purposively sampled perioperative nurses (RNs) responsible for reporting surgical count-related errors. These interviews identified the remaining feedback process variables. Two focus group discussions were held to refine the A&F strategy and describe its mechanisms of action. All interviews and discussions were transcribed verbatim, and thematic analysis was conducted until saturation was achieved. Findings were validated through member checking with a representative stakeholder reference group.
Results: We designed an LHS-informed and theory-guided A&F strategy with specific mechanisms of action to improve incident reporting among perioperative nurses. The strategy’s context and feedback variables were perceived to have the greatest influence on successful feedback cycles among RNs, while recipient characteristics had a lesser impact. Key mechanisms of action for the proposed strategy included “compatibility” with organizational beliefs, systems, and processes; leveraging “social influence” among providers to induce behavioral change; “resource matching” to the organization and providers’ capacity to engage with feedback; and “actionability” through direct facilitation of problem-solving behaviors.
Conclusion: By applying LHS approaches and the CP-FIT, we developed an implementable feedback intervention that may enhance surgical count error reporting among perioperative nurses. Future research will focus on prototyping, implementing, and evaluating the effectiveness of this strategy in improving reporting behavior and advancing surgical patient safety.

Applying User Centered Design to Prescriber Feedback in Acute Outpatient Care Settings in the Veterans Health Administration

Dr. Michael Ward

Introduction: Harm from prescriptions written for patients with contraindications to their use is common and preventable; nearly one-third of all antibiotic and nonsteroidal anti-inflammatory drug (NSAID) prescriptions in outpatient care settings are potentially inappropriate and contribute to adverse drug events. Audit and feedback is a promising and effective intervention to change clinician behavior. However, adoption can be challenging to scale and failure to place the user (i.e., clinician) at the center of the design contributes to underuse, workarounds, and unintended consequences. We focused on user needs to determine workflow integration and visualization needs to develop a tool for clinicians to quickly and easily view and better understand their prescribing practice to improve adherence to guideline concordant care. 
 Methods: In this study, we employed a user-centered design framework, a four-phase, iterative approach to prototype development that starts with understanding user needs (Phase 1) and requirements (Phase 2), which then progresses to formative (Phase 3) and summative (Phase 4) evaluation through an iterative process of increasingly higher fidelity prototypes. We developed and refined a clinician prescribing feedback system, entitled “CRAFT” (Care Review, Assessment, and Feedback Tool), for acute outpatient clinicians in the Veteran’s Health Administration in the Tennessee Valley Healthcare System in Middle Tennessee. 
We developed a low-fidelity prototype (PowerPoint slide) using input from the research team and ED clinical champions. This initial prototype used hypothetical data that aggregated overall clinician performance such as appropriateness of prescribing and “notable events” including return visits and adverse drug events. In addition to aggregated data, we provided patient-level detail for each prescription. 
To better understand the user needs and requirements for CRAFT, we conducted 25 interviews with physicians, advanced practice providers, and pharmacists, and asked about their prescribing practices and response to the low-fidelity prototype. From these data, we further iterated the prototype, addressing usability, data visualization, and information prioritization concerns. We then conducted four 60-minute design sessions with clinicians, developers, and biostatisticians to create an interactive prototype that contained derived but realistic data verified by clinicians. Prototypes were updated between sessions. Once the prototype was deemed complete by the study team and ready to be evaluated by potential users, we progressed to individual usability evaluation sessions. 
Six ED physicians completed individual 60-minute virtual usability evaluation sessions where the clinicians interacted with the prototype using one of two counterbalanced scenarios based on performance by drug class: Scenario One showed the clinician good antibiotic and poor NSAID performance and Scenario Two showed poor antibiotic and good NSAID performance. During the sessions, clinicians were asked to “talk aloud” about what they were seeing and expected to see. At the conclusion of these sessions, key issues were brought back to the team along with recommendations for changes. The team then iterated design recommendations and a refined prototype was created. This prototype was then evaluated by five physicians using the same procedure as discussed above.  
Results: Overall, there were six themes represented from the interviews including: usability, clarity and unintended consequences of language, actionability, comparators, and features. Participants expressed that they were busy and inundated with information, and there was a need for communicating key information efficiently. Respondent input identified that overall organization seemed to be disconnected and that there was too much information to digest on the welcome page.  
This initial round of usability testing also saw participants have issues and recommendations around wording and labeling, peer comparison, and navigation. Participants questioned the use of the labels “notable events” and “eligible prescriptions” as they seemed vague even after reading the detail available in the information icon. They also expressed concern over the negative connotation of the word “inappropriate,” and the need to clearly define the guidelines and appropriateness assessments. Physicians questioned the use of peer comparison and how it might influence behavior, especially if it was below the clinician reported performance. There were usability issues associated with simplifying ways to navigate from the aggregated clinician data to detailed patient information in the most efficient way possible.  
With these results, we refined the prototype, also added a visualization of all eligible prescriptions that was used in a second round of usability testing. Continued usability testing identified that respondents did not understand why some prescriptions were inconsistent with recommendations and how to act accordingly. They also noted that rather than a peer or historical comparison with their own performance, they preferred an alternative form of comparison. Further, the visualization of “nonevent prescriptions” did not contribute much and their primary interest involved the prescriptions with either an unexpected return visit or an adverse event. This resulted in the final prototype that was used to build the user interface in a pilot implementation of CRAFT. 
Discussion and Conclusions: Overall, participants saw value in the tool’s concept and indicated that they would likely use it, but had important optimizations, clarifications, and changes they noted during the user-centered design process that underscored the importance of this type of visualization and workflow integration evaluation.  
Importantly, the usability evaluation results revealed that substantial changes were needed to address navigation issues, to clarify language and connection to guidelines, and finally to provide a simplified message that clinicians could easily understand and act upon. We hypothesize that this will improve usability and acceptance by clinicians, and ongoing usability testing in the summative evaluation (Phase 4) that will be conducted following the pilot implementation of CRAFT and prior to a randomized trial in the VA health system.

Evaluating Audit and Feedback Strategies to Reduce Antibiotic Prescribing in Primary Care: A Randomized Controlled Trial

Dr. Kevin Schwartz

Background: An estimated 25-50% of antibiotic prescriptions in primary care are unnecessary.
Aims: Primary: To evaluate if providing family physicians with audit and feedback (A&F) on antibiotic prescribing compared to their peers reduces antibiotic use.
Secondary: To evaluate if adjusting for case-mix in feedback reports is superior to providing unadjusted data and whether emphasizing antibiotic-associated harms improves impact.
Methods: We performed a pragmatic physician randomized controlled trial (4:1 allocation) of an A&F mailed letter to family physicians compared to no letter in Ontario, Canada. We embedded within the intervention arm a 2×2 factorial trial evaluating i) case-mix adjusted comparators versus unadjusted, and ii) emphasis, or not, on harms of antibiotics. Eligible physicians who did not opt out received a mailed letter in January 2022with peer comparison antibiotic A&F of patients aged ≥65 years. The primary outcome was antibiotic prescribing rate (APR) per 1,000 patient visits at 6 months using Poisson regression models.
Results:
5,097 physicians were included and 4,076 received a letter. At 6 months, APR was 59.95 in the control arm and 56.43 in the intervention arm (relative rate 0.95 (95%CI,0.94-0.96). The intervention was most impactful on younger physicians and those with baseline high prescribing. No significant incremental reduction was seen for adjusted case-mix data or harms messaging.
Discussion:
Peer comparison A&F letters significantly reduced overall antibiotic prescribing with no additional benefit through case-mix adjustment or harms messaging. A&F is an effective intervention for antimicrobial stewardship in primary care with further studies needed to optimize its impact.

Obstacles and facilitators for Audit & Feedback implementation in General Practice and Emergency care: an experience from Lazio Region, Italy within the EASY NET project

Carmen Angioletti

*To accommodate the schedule, this presentation is a combination of the two following abstracts

Abstract 1- Planning Audit and Feedback interventions in health care organizations. An account from an Italian national program for Audit and Feedback implementation
Background & Objective: Audit & Feedback (A&F) consist of multidimensional quality improvement activities. The optimal design is still unknown. In 2019 the Italian Ministry of Health launched a research program EASY-NET, aimed at exploring the worth of A&F interventions, with participating seven regions conducting projects applying A&F initiatives in different settings. Aim of this work is to outline how interventions were designed at an early stage, to explore the extent to which current recommendations on desirable characteristics of an “ideal” A&F procedure are adopted.
Methods: Information on the A&F interventions design were collected through a form and administered to project leaders. It consisted of six sections dealing with the following items: description of the working group; targeted clinical behaviors; selected indicators and sources; feedback procedures to be adopted; actions expected from the target health workers.
Information gathered through the template was then classified into four main topics (nature of the desired action, type of data available for feedback, feedback display and feedback delivery), in line with the categorisation used by Brehaut et al. 2016.
Results:
Nature of the desired action
A&F procedure were often aimed at changing a narrow focused, identifiable clinical behavior. Moreover, the type of actions that intervention designers expected seems to be generic. All the projects identified clinicians as the recipients of the information. Managers and other professionals with organizational responsibilities were explicitly considered in 8 projects.
Nature of the data available for feedback
An average of 27 indicators were planned to be developed from administrative databases, sometimes integrated by ad hoc data collection. Outcome measures were included in 5 projects. Comparators were identified as reference standards drawn from the scientific literature or from different territorial realities. The provision of feedback was scheduled semi-annually, annually, every 3 months, on demand.
Feedback display and delivery
All the feedback allow access to aggregated data that can be displayed through graphs and tables. Sending options were: web platforms, e-mails, workshops and individual meetings. The use of economic incentives to encourage clinicians’ participation was mentioned.
Conclusion: At least at an early stage, the projects were mostly intended as “wide focused” to generally improve the quality of care. However, changes in the design and delivery of A&F were introduced during the implementation of the projects, which are still ongoing, to consider suggestions from experts. Results will offer interesting insights on effectiveness of A&F strategies in Italy and their adherence to current best practice.

Abstract 2 – Obstacles and facilitators for Audit & Feedback strategies in General Practice: an experience from Rome, Italy

Introduction: We know that Audit & Feedback is an effective and widely used strategies for healthcare quality improvement, but its effectiveness is heterogeneous suggesting the need of performing studies aimed at understanding the ways to increase A&F effectiveness. Within an Italian research program called EASY-NET (project-code NET-2016-02364191), researchers from the Lazio Region (Work Package 1) is experimenting an A&F intervention involving mainly General Practitioners for improving healthcare quality for patients affected by COPD and diabetes mellitus type II.
The intervention was delivered during the year 2022 within an education & training course involving GP as “trainees”, a selected group of these GPs as “tutors”, and health management physicians of the Local Health District (LHD) as “representatives”. The intervention was articulated in frontal lectures, and practical work in small groups. Regarding the feedback, each GP collected data about his/her own practice through the professional practice management software, then the representative and tutor figures calculated selected indicators and fed-back results to GPs during in-person meeting.
Objective: The aim of the present qualitative study was to explore facilitating factors and obstacles encountered during the implementation of the described A&F strategy.
Methods: We organized a series of focus groups (FG) including all the professionals participating to the intervention. Separate FG were planned for each type of professionals with the same role, for a total of four FGs. Eight to twelve participants were expected for each FG. We prepared the protocol according to a phenomenological framework and drafted the questions to guide the discussion. FGs were audio recorded. The consent to be audio-recorded and to personnel data treatment were collected before the start of each FG along with anonymous information about demographic and professional characteristics of the participants. FGs were then transcribed, and encoded by two researchers, independently. We firstly identified single comments and then grouped them in categories and major teams.
Preliminary results: At this time, we conducted the first FG involving seven out of 10 invited facilitators. The FG lasted 90 minutes. Preliminary results suggested that giving to GPs knowledge pills about Audit &feedback characteristics (instead of lectures), reducing the number of meetings, reducing the amount of time to spend, having the opportunity to discuss results also with other professionals (i.e. specialists, feedback providers, health managers) emerged as example of facilitating factors.

Here we described some preliminary results that could be consolidated and expanded during the further FG that will involve participating GPs.

How can the national stroke audit in England drive quality improvement in the evolving post-acute setting?

Lal Russell

Background: The Sentinel Stroke National Audit Programme (SSNAP) began in 2013 and collects a clinical dataset for stroke patients in England, Wales and Northern Ireland (85,000 patients annually). SSNAP has historically focused on hospital-based care and evidence suggests the audit has been successful in driving improvements.
The audit has more recently expanded to include post-acute (community) services and the impact in this setting has yet to be established. Challenges exist in collecting national data beyond the hospital setting as community services are diverse and evolving with variations in models of commissioning and service delivery. Key questions have been raised as to how best capture multidisciplinary activity and how this relates to patient outcomes.
This study explores how the audit is perceived by post-acute stakeholders and what factors influence its success in driving quality improvement in this evolving context.
Methods: This study comprises two sequential phases. Phase one was an online mixed-methods survey. Findings from phase one shaped the exploration of the in-depth interviews in the second phase. Participants were employees who worked in, commissioned or managed community stroke rehabilitation in England.
Results: Phase one achieved a national sample of stakeholders, with representation from administration, clinical, leadership and commissioning (n=206). Participants described using SSNAP to support a range of improvement activities, including funding additional staff, resources and service reorganisation. However, several challenges were identified that were explored in-depth in phase two.
Phase two interview participants included administrators, clinicians, service leaders and commissioners(n=20).
Interviews highlighted several contextual features that influence the ability of the audit to drive quality improvement in this setting. These include the organisational culture, the format of the report, communication across the pathway and stakeholder perceptions of the data they submit.
Conclusion: Stakeholders are actively engaged with the post-acute audit and describe committing significant efforts to support participation. Despite the challenges highlighted, SSNAP feedback is used to inform quality improvements and service developments in this evolving healthcare landscape. Key messages from this study include the importance of organisational support for teams to engage with the audit cycle beyond data collection alone.
Efforts are required from rehabilitation teams, healthcare organisations and SSNAP in order to realise the potential of national clinical audit as a tool for quality improvement in the post-acute setting.

Audit & Feedback in Learning Health Systems

The evolution, impacts and challenges of a primary care implementation laboratory

Professor Robbie Foy

Background: Primary care is responsible for most prescribing in the UK National Health Service (NHS), with well-recognised inappropriate variations in prescribing. Our partnership of researchers and primary care commissioners has responded to this problem with a rolling programme of research and quality improvement for over a decade. We first demonstrated the effectiveness of audit and feedback in reducing high-risk and opioid prescribing respectively in rigorous trial and quasi-experimental evaluations involving over 300 West Yorkshire general practices. We then scaled up and diversified our feedback campaigns, mainly drawing upon NHS funding matched by the National Institute for Health and Care Research (NIHR) Yorkshire and Humber Applied Research Collaboration.
We describe the further development of our ‘implementation laboratory,’ impacts and challenges.
Methods: We designed our feedback campaigns using current best evidence and relevant theory. These have variously targeted gabapentinoid, antibiotic and (most recently) asthma inhaler prescribing. The reports use routinely collected data and content is produced by academic general practitioners, pharmacists and NHS managers. For each campaign, general practices typically receive two-monthly comparative feedback reports electronically.
Results: Since 2019, our feedback campaigns have been taken up across the region over varying durations, reaching over 900 general practices. However, although our feedback reports have generally been well-received, our ability to discern impact of the longest running campaign, on antibiotic prescribing trends, has been hindered by the impact of Covid-19 and variable regional adoption over time. Crude comparisons with national data suggest little overall impact on antibiotic prescribing but there may be specific impacts on appropriate prescribing.

Discussion: We have established much of the infrastructure and experience for a nascent learning health system. We now face some challenges and decisions, e.g.

  • How to decide which campaigns to start, stop or modify.
  • What level of rigour is needed to monitor and evaluate each iteration of feedback or any new campaigns.
  • Given a wide range of known and unknown confounders from national and local events and initiatives, whether randomisation will always be necessary in evaluating incremental effects.
  • How to secure sustained external research funding for embedded rigorous evaluations.
  • Whether and when we can apply our approach to other quality improvement interventions, such as computerised decision support systems, and to other clinical targets, such as test ordering.

ANDA-Evaluating Facilitated Feedback Enhancement - a Cluster randomised Trial (ANDAEFFECT)

Dr. Matthew Quigley

Background: Despite clinician participation in annual audit and feedback (A&F) and advances in diabetes technology and medications, glycaemic control remains suboptimal for many Australians living with diabetes. Previous qualitative work investigated the barriers and enablers to use of A&F in Australian diabetes centres, which informed redesign of the feedback and development of cointerventions to assist in use of feedback for quality improvement (QI) activities.
Objectives: This cluster randomised trial (CRT) tested the effect of redesigned audit feedback and educational and community of practice cointerventions on:
  1. HbA1c at the patient level (6 months after delivery of the interventions); and
  2. Acceptability and utility of the interventions at the practitioner level (~3 months after delivery of the interventions).
Methods: Australian diabetes centres (clusters) participating in this two-armed CRT nested within an existing diabetes A&F activity were randomised to the feedback plus cointerventions (intervention) or feedback (control) group, stratified by type of diabetes centre and location. Both groups received redesigned feedback; the intervention group also received educational and support resources to inform local QI activities. A mixed linear effects model was used to examine between-group differences at 6-month follow-up.
Results: Participating centres were allocated to the intervention group (n=23) and the control group (n=19). Follow-up clinical data were available for 20 intervention and 14 control centres.
At 6-month follow-up, no statistically significant between-group differences were found for mean HbA1c, or for secondary clinical outcomes including mean systolic blood pressure and lipid measures. Small changes in medication use in both groups resulted in statistically significant between-group differences for use of antihypertensive therapy (p = 0.030) and use of select medications including thiazides, ARBs and statins (p = 0.012, 0.002, 0.009, respectively).
Acceptability and utility survey data were available for 17 intervention and 12 control centres. All survey respondents reported using the redesigned feedback report. Of the intervention group, 59% (n = 10) reported use of any cointervention, but only 24% (n = 4) reported use of all cointerventions.
Conclusions: The interventions delivered in this trial were not effective in reducing HbA1c at 6-month follow-up. The potential effects of non-paired data and challenges of implementation during the COVID-19 pandemic, including low engagement with the cointerventions, warrant further understanding and may assist other diabetes and implementation researchers.
Trial registration: Prospectively registered on June 21, 2021 with the Australian and New Zealand Clinical Trials Registry ACTRN12621000765820.

How do teams tailor improvements in diabetes care: Preliminary findings from a Process Evaluation study

Elaine O'Halloran

Background: Feedback interventions may be more effective when they target health professionals with greater quality improvement capability (Brown et al., 2019). Assessing influences and aligning improvement actions to these influences – also known as tailoring (Powell et al., 2017) – is a common component of implementation strategies. To enhance effectiveness, audit and feedback should consider and address recipient and contextual variables (Desveaux et al., 2023).
Since 2008, NICE guidelines have recommended insulin pump therapy for patients with Type 1 Diabetes and an HbA1c over 69mmol/mol. However, in England and Wales there are about 90,000 patients who meet these criteria but who do not use a pump (National Diabetes Audit, unpublished), and significant variation by deprivation, ethnicity, sex, and location (National Diabetes Audit, 2021). Much of this variation is likely to be attributable to staff and local organisational factors (Llewellyn et al., 2014).
All diabetes services across England and Wales were invited to participate in a trial evaluating the effectiveness of a Quality Improvement Collaborative (QIC) aligned to the National Diabetes Audit to increase the use of insulin pumps. The QIC supports diabetes specialist teams to select, and generate commitment for, improvement actions aligned to their local influences and contexts. Within the QIC, the Theoretical Domains Framework (Atkins et al., 2017) is used by clinical teams to identify influences upon care. Teams then undertake a virtual logic model exercise to align improvement strategies to these influences. We aim to describe how teams enact tailoring as part of their work to respond to national audit to inform future support for quality improvement capabilities.
Methods: We use observations, documentary analysis and semi-structured interviews to explore how teams undertake tailoring work during the initial workshops and throughout the 15-month QIC. We categorise the selected and enacted improvement actions using the Expert Recommendations for Implementing Change (ERIC) (Powell et al., 2015).
Results: Preliminary findings from the QIC workshops describing the links between the diabetes care pathway, identified influences and proposed improvement strategies will be presented. Influences relate to patient (e.g., skills, emotion), staff (e.g., motivation, beliefs about capacity) and contextual factors (e.g., environmental context, social influences).
Conclusions: Exploring how teams identify the factors that influence their practice, and how and why these influences link to the strategies selected by teams to improve quality in their local contexts will inform our understanding of the effectiveness of the QIC and the support for quality improvement capabilities of A&F recipients.

Presentation not shared due to unpublished data.

Audits and feedback across sectors: transferring experience from Health to education in Middle Africa

Rigobert Pambe

There is a huge gap between evidence and practice in countries of the lake basin. Despite a growing body of knowledge emanating from sub–Saharan Africa, they are hardly embedded within policies and practices in countries of this region. This “evidence hesitancy” is due to a variety of reasons ranging from the complexity of the available evidence disseminated to simple resistance to change. eBASE Africa within the years has successfully bridge the evidence gap in health, using audits and feedback to improve clinicians’ compliance to malaria treatment and COVID19 recommendations. Giving precarity in delivery of basic services in middle Africa it will be interesting to explore transferability of experiences in health to other priority areas, notably education.
Educational attainment in countries of the Lake Chad Basin is amongst the lowest in the globe with teachers in middle Africa continuing to prioritize rote learning hardly implementing innovative evidence-based approaches for teaching and learning. Practical Teachers Professional Development (TPD) programs that supports specific well-evidenced changes in pedagogy can be highly effective in improving pupil learning and are essential in curbing these challenges. The Pedagogic Audits and Feedback (PAF) is in line with this logic, as it is a blend of TPD, implementation sciences and behavioral sciences.
This evidence implementation project was conducted in 48 schools in Cameroon, Niger, Nigeria and Chad. We engaged 96 teachers (58 females, 38 males, 11 indigenous teachers) from primary and secondary schools reaching over 5000 pupils in both rural and urban areas. Our approach is adapted from experiences and lessons from Audits and Feedback developed by the Joanna Briggs Institute. Audit criteria were developed from the eBASE Teaching and Learning Toolkit which uses a database of synthesized best available evidence for feedback in Middle Africa. We identified 10 evidence-based categories broken down into 49 evidence criteria which we used to assess teaching and learning practice.
We compared compliance with best practice recommendations at baseline against a follow-up compliance at three months. Compliance rates improved overall from 35% (R: 22–51) to 71 (R: 61-86) for all criteria and sites, with differences noticed between sites. Twelve barriers were identified, classified into infrastructure, material, and processes with respective coping mechanisms and strategies to ensure behavior change and systematization.
Despite barriers, PAF is effective in getting research into practice and does improve quality of teaching and learning.

Keynote Presentation

Building implementation labs into healthcare systems – what has to be true for this to work

Jane London

In this session, Jane will outline the real world considerations from a service provider and funder perspective when considering how to engage with implementation labs. Developing enduring learning systems that can be utilised for both research and service provider objectives requires a reframing of both research and service delivery. To get the best out of health services for consumers, we need to approach everything with a ‘we don’t know whether this will work’ mindset and create pragmatic ‘real world’ trial designs. She will explore the building blocks to this approach from a service provider angle, highlighting where the research can be utilised and the practical considerations for embedding this approach in the healthcare space.

Presentations - October 27

Keynote Presentations

Precision feedback in learning health systems

Dr. Zach Landis-Lewis

GEMINI: Harnessing hospital data to improve care

Dr. Fahad Razak and Dr. Surain Roberts

Insights for Audit & Feedback Design

Comparing paper Letters in addition to Emailed Audit and feedback in Refining Asthma treatment to Improve clinical and environmental Results in primary care: The CLEAR AIR study

Dr. Sarah Alderson

Background: The UK National Health Service (NHS) produces around 4% of all UK greenhouse gas emissions, on a par with the airline industry. Delivering a ‘Net Zero NHS’, including setting clear targets for decarbonisation, is a high priority. Respiratory inhalers, used to treat asthma, produce a disproportionate amount (3%) of the total NHS greenhouse gas production. Pressurised Metered Dose Inhalers (pMDI) contain hydrofluorocarbons, with an eighteen times higher carbon footprint than Dry Powdered Inhalers (DPI). England has a disproportionately high usage of pMDIs (71.6% of all prescribed inhalers) compared to its European neighbours (<50%), leading to calls to switch inhaler prescriptions in England.
Our UK regional audit and feedback campaigns have delivered improvements for high priority prescribing issues (previously addictive painkillers and antibiotics). This researcher and primary care commissioning partnership has now evolved over seven years into a Learning Health System (LHS), aiming to answer questions relating to improving effectiveness of our campaigns and thereby improving care. One such question is the impact on effectiveness of delivering paper copies of feedback alongside electronic (email) copies in primary care.
Method:A cluster randomised controlled trial (RCT) of paper reports in addition to emailed reports. Bimonthly theoretically and evidence-informed feedback will be delivered to practices (n=277) across the NHS West Yorkshire Integrated Care Board for 1 year from May 2023. Reports will highlight the number of patients with poor asthma control, the percentage of DPI prescribing in relation to pMDIs, and compare to other practices within the region. We will randomise groups of practices (primary care networks, n=52) to the addition of posted paper reports (intervention) to emailed reports alone (control). Primary outcome will be the number of DPI inhalers prescribed as a percentage of the total prescribed inhalers for asthma at the practice level.
Results: We will describe the trial protocol in detail, including the challenges of obtaining ethical and governance approvals for randomisation of practices without consent, the changing structure of primary care in England and the impact of the formation of primary care networks, sample size considerations and conducting an RCT as an LHS without additional external funding.
Discussion: Our established regional audit and feedback quality improvement programme has overcome multiple obstacles to become a LHS with the CLEAR-AIR trial. We propose this proof-of-concept trial will enable the LHS approach to be scaled up and diversified to answer further questions of effectiveness and priorities in improving patient care.

Repurposing the Ordering of Routine Laboratory Tests in Hospitalised Medical Patients (RePORT): results of a cluster randomised stepped-wedge quality improvement study

Dr. Douglas Woodhouse and Dr. Anshula Ambasta

This research was published in BMJ Quality & Safety on 10 May 2023: http://dx.doi.org/10.1136/bmjqs-2022-015611
A multifaceted intervention bundle using education and facilitated multilevel social comparison was associated with a safe and effective reduction in use of routine daily laboratory testing in hospitals.
We evaluated an intervention bundle to reduce repetitive use of routine laboratory testing in hospitalised patients. We used a stepped-wedge design to implement an intervention bundle across eight medical units. Our intervention included educational tools and social comparison reports followed by peer-facilitated report discussion sessions. Our study included a total of 125 854 patient-days.
There was a 14% overall reduction in ordering of routine tests, a 15% reduction in ordering of all common tests, a 20% increase in routine test-free patient-days and a 14% reduction in costs of routine testing. No worsening was noted in patient safety endpoints.

Exploring the components of feedback facilitation co-interventions: A systematic review

Dr. Michael Sykes

Background: Health systems are investing in quality improvement support for feedback recipients (e.g. HSE Ireland, 2019; HQIP, 2021) because increasing the knowledge and skills to respond to feedback may increase the effectiveness of audit and feedback (A&F) interventions (Brown et al, 2019). Quality improvement support is a form of ‘feedback facilitation’. The effectiveness of A&F with or without a feedback facilitation co-intervention is being investigated through the Cochrane review of randomised controlled trials. Our study aimed to describe the content (e.g. structured analysis of determinants) and delivery (e.g. educational meeting) of feedback facilitation co-interventions used in trials of A&F within healthcare.
Method: Papers were identified from the latest Cochrane review of audit and feedback. Within the Cochrane review, feedback facilitation “could be training about how to use feedback, or to do quality improvement in the practice, or set goals and plans, etc.”
We extracted data from papers describing the trial of audit plus feedback facilitation, and from companion papers. A total of 8 reviewers extracted data from the included studies using a specifically designed and piloted proforma. The proforma reflected an adapted TIDieR framework (Hoffmann et al 2014). Data was recorded in Excel. Each paper was reviewed independently by two reviewers and any disagreements were resolved through discussion.
At the time of submission, two members of the team are analysing the data graphically and narratively. We will present the analysis to the research team for challenge and synthesis. If possible, we will describe the effectiveness of subgroupings of feedback facilitation interventions.
Results: 105 trials were included. We are currently undertaking analysis, and present tentative early findings here: Feedback facilitation includes 20 different implementation strategies (Powell et al, 2015). There is little use of theory and few studies describe the programme theory of how the intervention has its effect. There is variation in whether and how interventions identify priorities, explore influences upon performance and select actions. There is variation in mode of delivery, deliverer and recipient, frequency, duration and timing.
Conclusion: Feedback facilitation is a heterogenous grouping of interventions. The ability to replicate a study underpins both implementation science and impact; Few studies provide sufficient detail to enable replication. We will present, and seek to draw lessons from, the different designs of feedback facilitation so as to inform researchers and health systems delivering quality improvement support to audit and feedback recipients.
Presentation not shared due to unpublished data.

Advancing the Science

Claims-based Audit & Feedback, development of indicators & acceptance by physicians

Dr. Vera de Weerdt

Introduction: Comparative effectiveness research (CER) is conducted to improve quality of care, but often fails to create impact since results are not implemented into clinical practice. Audit & Feedback (A&F) could be used to stimulate implementation of CER. Claims-data provides a cost effective data source for A&F on CER implementation but may not be accepted by professionals as a valid data source in this context. No existing methods are available to develop indicators specifically for CER studies. We examined whether we could develop claims-based A&F, which is accepted by professionals in the context of CER implementation.
Methods: We conducted two studies: First, we conducted a co-creation study to develop A&F for six CER studies. Second, we conducted five  focus groups to examine whether medical specialists accepted the claims-based A&F for CER studies. 
Results: During the co-creation method we developed claims-based A&F for six CER studies, which was accepted by the involved medical  experts in four out of six cases. The experts only accepted claims-based A&F indicators in which they deemed the level of over- or  underestimation of the target population marginal.
In focus groups, we presented the claims-based A&F for five CER studies, of which two were accepted by medical specialists. Arguments mentioned in favor of claims-based A&F were: (1) A&F stimulates reflective learning and improvement (2) claims-based A&F is perceived as more reliable than other A&F (3) claims-based A&F prevents administrative burden. Arguments in opposition were that (1) A&F is insufficient to create behavioral change (2) A&F lacks clinically meaningful interpretation, (3) claims-based A&F is unreliable, and (4) claims data is invalid for feedback on QI. Furthermore, participants describe several conditions for implementation of A&F which shape their acceptance.
Discussion: Using claims-based A&F for QI is, for some clinical topics and under certain conditions, accepted by professionals. Training  physicians in how to interpret and act upon A&F may further increase acceptance of claims-based A&F. Currently, claims data is the most  resource efficient data source for A&F interventions. Thus, when designing A&F it should weighed whether claims data can be used or whether it is necessary to collect more specific data for A&F aiming to improve quality.

Fielding feedback: Getting feedback to intended recipients

Dr. Anne Sales

Background: Feedback interventions are demonstrated to be moderately effective. Recommendations for optimizing feedback interventions gained through systematic review provide important principles for feedback design include considerations of design, cognitive burden, salience and validity of the information provided. There is discussion in the literature about presentation mode (oral, written, graphic), but very little about how feedback gets to the intended recipient(s). Method of distribution, or fielding, of a feedback intervention is as important as fielding a survey instrument but is rarely discussed. When it is discussed, reports about how successful the fielding was in actually ensuring that intended recipients receive the feedback are rare. In this presentation, we will provide a brief overview of what we know from the literature and case studies from our own work about methods of fielding feedback reports, reports of success rates in fielding feedback reports, and possible next steps for the future
Findings: We report on two different studies as case examples of different methods of fielding feedback interventions. In one study, conducted in Edmonton, Alberta, Canada, we distributed written feedback reports by hand directly to intended recipients (all personnel working on the study units in four continuing care facilities) every month, and observed what recipients did with the paper report when they received it. These observations were reported in the process evaluation published for this study; briefly, over the 13 months of the intervention, distribution shifted from reports being handed to an individual 68% of the time in the first month to 12% being directly handed to an individual at study end. Leaving the report somewhere instead of handing to an individual increased from 15% to 87% of the time. In the second study conducted in 19 U.S. Veterans Health Administration nursing homes, we delivered monthly feedback reports by email to champions who took responsibility for report distribution. In a follow-up survey, we found that almost none of the champions passed on the feedback reports to the intended recipients, who were frontline clinicians responsible for taking required actions (manuscript in process).
Implications: Theory underlying effective feedback interventions assumes that intended recipients have received the feedback as intended, but there is little in the published literature to support that assumption. We recommend development of reporting guidelines specific to feedback interventions which would encompass best practices in many domains of feedback intervention, including providing information about fielding methods and success rates in reaching intended recipients.

Examining how Audit & Feedback trials describe sustainability, spread, and scale: a theory informed, qualitative, secondary analysis of a systematic review

Zeenat Ladak on behalf of Dr. Celia Laur

Background: Little is known about if or how Audit and Feedback (A&F) interventions are sustained, or if trials plan for the intervention to be applied elsewhere (spread/scale). This study examines how A&F trials describe sustainability, spread, and scale.
Methods: A secondary analysis of an update of the Cochrane systematic review of A&F trials. All trials from the larger review published in 2011 and later were included. Keyword searches for terms related to sustainability, spread and scale were conducted; trials with at least one relevant keyword were extracted. Data were qualitatively analyzed using the Integrated Sustainability Framework (ISF), and the Framework for Going to Full Scale (FGFS), with additional codes and themes identified inductively. A forward citation search was conducted to identify relevant additional follow-up studies.
Results: From the larger review, 162 trials met eligibility criteria. 78% (n=127) of trials included at least one keyword on sustainability, 49% (n=62; 38% overall) were grouped as frequently mentioning sustainability (keyword found in 3+ sections, [abstract, introduction etc.]). For spread/scale, 62% (n=100) of trials included at least one keyword related to spread/scale, with 51% (n=51; 31% overall) of those trials grouped for frequently mentioning spread/scale (keywords in 2+ sections). Within the keyword search, “sustain*” was mentioned most frequently (n=142); “scal*” (n=85) was mentioned more than “spread” (n=12). Although trials mentioned the need to consider sustainability, detail was lacking regarding if or how this was planned or implemented. Results mapped to the broader Domains of ISF, but not all Factors. The most frequent sustainability period duration was 12 months (range 2-24 months). For spread/scale, strong alignment was found with the FGFS for Phases of Scale-up, and Support Systems (Infrastructure), but not for Adoption Mechanisms. Three new themes were identified: aligning affordability and scalability; balancing fidelity and scalability; and balancing effect size and scalability. Within the FGFS, some trials mentioned Learning Systems, mainly focusing on the benefits of implementation laboratories, clinical networks, or taking a Learning Health Systems approach.
Conclusion: A&F trials should plan for sustainability and scalability so if the trial is effective, benefits can continue. A deeper understanding of the factors impacting sustainability is needed. Scalability planning should go beyond cost and infrastructure to consider other adoption mechanisms. Future research should explore if the effect of an A&F trial is continued, for how long, and whether this is with or without continuation of the A&F intervention.
Trial Registration: Registered with Prospero in May 2022. CRD42022332606

Evaluating the effectiveness of a multifaceted intervention to reduce low-value care in adults hospitalised following trauma

Dr. Lynne Moore

Background: While simple Audit & Feedback (A&F) has shown modest effectiveness for reducing low-value care, there is a knowledge gap on the effectiveness of multifaceted interventions to support de-implementation efforts. Given the need to make rapid decisions in a context of multiple diagnostic and therapeutic options, trauma is a high-risk setting for low-value care. Furthermore, trauma systems are a favorable setting for de-implementation interventions as they have quality improvement teams with medical leadership, routinely collected clinical data, and performance linked to accreditation. We aim to evaluate the effectiveness of a multifaceted intervention for reducing low-value clinical practices in acute adult trauma care.
Methods: We will conduct a pragmatic cluster randomized controlled trial (cRCT) embedded in a Canadian provincial quality assurance program. Level I-III trauma centers (n=30) will be randomized (1:1) to receive simple A&F (control) or a multifaceted intervention (intervention). The intervention, developed using extensive background work and UK Medical Research Council guidelines, includes an A&F report, educational meetings, and facilitation visits. The primary outcome will be the use of low-value initial diagnostic imaging, assessed at the patient level using routinely collected trauma registry data. Secondary outcomes will be low-value specialist consultation, low-value repeat imaging after patient transfer, unintended consequences, determinants for successful implementation, and Incremental Cost-Effectiveness Ratios.
Discussion: The proposed intervention targets a problem identified by stakeholders, is based on extensive background work, was developed using a partnership approach, is low-cost, and is linked to accreditation. There will be no attrition, identification, or recruitment bias as the intervention is mandatory in line with trauma center designation requirements and all outcomes will be assessed with routinely collected data. However, investigators cannot be blinded to group allocation and there is a possibility of contamination bias that will be minimized by conducting intervention refinement only with participants in the intervention arm.
Trial registration: This protocol has been registered on ClinicalTrials.gov
(February 24th 2023, #NCT05744154, https://clinicaltrials.gov/ct2/show/NCT05744154)