AI tool built with Posit decreases unexpected deaths of hospitalized patients by 26%

Unity Health Toronto has launched over 50 artificial intelligence (AI) and analytics tools to make better decisions, increase hospital efficiency and improve patient care.

One such project is an Early Warning System aimed at reducing in-hospital mortality for General Internal Medicine inpatients. The AI tool was developed to predict and prevent unexpected deterioration in hospitalized patients, a major cause of unplanned ICU admissions. At the outset of this project, roughly 8% of these patients either passed away or were admitted to the ICU.

By providing a prediction score, the Early Warning System sought to improve care quality and inform clinical interventions, thereby decreasing mortality rates. 

Improvements to quality of care included:

• Earlier notification to ICU team before transfer to ICU
• Shorter ICU length of stay
• Better allocation of nurses to patients, based on illness severity
• Better communication with patients and families about prognosis and goals of care
• Earlier involvement of palliative care team for patients with expected death 

Back in 2017, the DSAA team at Unity Health Toronto received a suggestion from Amol Verma, a doctor within General Internal Medicine, that predicting deaths could be a key area where machine learning could make a positive difference.

Chloé and the DSAA team started developing a model using Python and R that would then become the CHARTWatch tool. The model was trained on ~20,000 patient visits consisting of: laboratory values, vital measurements, and demographics with a goal of predicting which patients are at risk of deterioration to improve their quality of care.

For every patient, the model outputs a risk group: Low, Medium, or High that is delivered to different end-users with different methods of delivery depending on the end-user.

• Emails with a list of patients and their risk group are sent to charge nurses and the Palliative Care team
• Updates to their front-end (Electronic Sign-Out Tool) and embedding the CHARTWatch Risk Group into the dashboard used by clinicians
• Alerts sent to clinician’s phones for patients that are at high-risk of deterioration

The successful development, deployment, and implementation of this model across the organization required collaboration to determine the most effective strategies. Chloé Pou-Prom, Senior Data Scientist at Unity Health Toronto, adds, “Model development is only a small part of the project. Engagement with different stakeholders is really essential to get things moving and get things implemented.”

• Create an implementation team: With so many people across the hospital involved in the success of this project, they created an implementation team across various departments (General Internal Medicine, Intensive Care Unit, Palliative Care, Clinical Informatics, IT, Data Science). End-user engagement was crucial and would not have been possible with just the Data Science team.
• Determine workflow together: The implementation team created a clinical workflow to outline the next steps (e.g., reviewing recent medications and imaging orders) after CHARTwatch flags a patient as High Risk. This allows the team to determine an appropriate intervention.
• Fit into existing processes: The new workflow was designed to align with existing clinical practices for deteriorating patients. The timing of alert notifications, such as the twice-daily emails to charge nurses, was also tailored to fit existing shift schedules.
• Pilot phase: The DSSA team first deployed to only two General Internal Medicine teams and during that time they had weekly meetings to debug any issues they ran into and go into the patients flagged by CHARTwatch to make sure everything was correct.

• Connections to databases: The hospital has various data systems (for labs, surgical procedures, etc.) To help the data science team work more effectively, they developed an internal package called chartDB to work with hospital databases, using connections that follow the same pattern.
• Reproducible environment, allowing multiple people to collaborate on a project: Their DSAA team utilizes renv with Posit Workbench to specify to different packages and package versions a project relies on.
• A secure way to download internally developed packages: To share code and knowledge with others and stop copying & pasting code, the team started writing their own internal packages that they deploy to Posit Package Manager. For their team it was important to be able to download packages while being disconnected from the internet. 
• A staging environment: The DSAA team uses an environment as close to the “real” deployment environment as possible when they need to make updates to CHARTwatch.

• Authentication: Deployed applications run on Posit Connect and it interacts with the hospital’s Active Directory so users can use their hospital username and password. Active Directory groups allow them to more easily manage permissions across the organization.
• Scheduling: DSAA uses automatic scheduling with a service account on Posit Connect (with tags, environment variables, scheduling settings, and access controls).
• Silent Deployment Period: Stage where the entire pipeline is running, but the final outputs are silent. This is useful for addressing unexpected changes and bugs, such as sodium tests being interpreted as “NA” (Not Available).
• Alerts for Downtime: The team developed a package called jarvis to monitor production applications and send automatic alerts via Slack and email. Due to the critical nature of the production model for patients and its integration into clinical care, specific downtime protocols are in place. If any issues arise, an email notification is sent to all hospital end-users.

The CHARTWatch early warning system resulted in a significant 26% reduction in unexpected deaths among hospitalized patients. This result was based on a comparison between 13,000 admissions to St. Michael’s General Internal Medicine ward to thousands of admissions in other subspecialty units.

“Open Access Clinical Evaluation of a Machine Learning–based Early Warning System for Patient Deterioration,” 2024)

Following the successful implementation of this tool, CHARTWatch expanded to the surgery department in 2022 to predict the level of support individual patients would require. The DSAA Team has also led the development of solutions for AI medical imaging, hospital bed capacity planning, and assignment/scheduling. Their ongoing efforts continue to enhance decision-making, optimize hospital efficiency, and elevate both patient care and outcomes.