Vol. 5 No.04 (2026) | Page: 318-326 |Review Article ![]()
External Validation and Clinical Utility of Explainable Artificial Intelligence Models for Early Prediction of Clinical Deterioration in Hospitalized Patients
Anil Kuma 1*
-
- Department of Medicine, King George’s Medical University, Lucknow, India.
kuma.a2019@gmail.com
ABSTRACT
Background: Clinical deterioration among hospitalized patients is associated with increased morbidity, mortality, prolonged hospital stays, and higher healthcare costs. Conventional early warning systems rely on predefined physiological thresholds and may not adequately identify complex patterns associated with adverse outcomes.
Objective: To evaluate the external validation, explainability, and clinical utility of AI-based models for predicting clinical deterioration among hospitalized patients.
Methods: A narrative literature review was conducted to assess studies investigating explainable AI models for predicting clinical deterioration. Studies were evaluated based on model discrimination, calibration, clinical utility, and implementation considerations.
Results: AI-based prediction models consistently demonstrated strong predictive performance and frequently outperformed conventional early warning scores in identifying patients at risk of clinical deterioration. Models integrating multiple clinical data sources achieved improved discrimination and risk stratification.
Conclusion: Explainable AI has considerable potential to improve the early detection of clinical deterioration and support timely clinical interventions. Successful implementation requires rigorous external validation, fairness assessment, continuous performance monitoring, and demonstrated clinical benefit.
Keywords: AI, clinical deterioration, ML, early warning systems
Key points:
- Can explainable AI models accurately predict clinical deterioration and improve risk identification among hospitalized patients?
- AI-based prediction models demonstrate superior predictive performance compared with conventional early warning scores.
- Rigorous external validation, fairness assessment, and continuous monitoring are essential for safe clinical implementation and timely interventions.
- INTRODUCTION
Clinical deterioration, defined as a worsening of physiological instability, changes to an intensive care unit (ICU) admission, cardiac arrest, respiratory failure, and death, are a persistent problem in today’s healthcare systems [1]. Patients at risk of deteriorating should be identified early in order to allow timely interventions that can enhance clinical outcomes and prevent adverse events. Hospitals have traditionally adopted a clinical judgement approach and/or the use of the National Early Warning Score (NEWS) or Modified Early Warning Score (MEWS), that are based on thresholds of physiological parameters to recognize high-risk patients [2]. These have proven to be clinically useful but are less successful at summarizing the interplay of multiple clinical factors [3]. With greater availability of electronic health records (EHRs), continuous monitoring systems and the advent of large clinical datasets, it has been possible to develop AI-powered prediction models for the early identification of clinical deterioration. Machine learning (ML) algorithms can learn patterns from large amounts of structured and unstructured clinical data, such as vital signs, laboratory values, medications, comorbidities, and demographic information, that are linked to adverse outcomes [4]. AI models could offer more personalized and dynamic risk predictions, which can aid earlier clinical decision-making, in comparison with traditional statistical methods. While the potential for prediction is promising, a large number of AI models have been created and evaluated in retrospectively collected data from a single center, raising concerns about generalizability and applicability of these models in the real world. It is therefore important for external validation and assessment of clinical utility, when planning the implementation into routine clinical practice [5]. Moreover, there are many sophisticated AI systems that operate as “black-box” systems, which lack transparency and clinician confidence. To tackle these challenges, there are approaches that have been developed to provide interpretable information on how models are making predictions and what clinical factors are involved in risk estimation: the so-called explainable artificial intelligence (XAI) approaches [6]. The aim of this literature review is to summarise current evidence on the external validation and the clinical applicability of explainable AI models for early prediction of clinical deterioration of hospitalised patients, with a focus on model performance, interpretability, implementation and future directions.
- ARTIFICIAL INTELLIGENCE MODELS FOR PREDICTING CLINICAL DETERIORATION
To identify deteriorating patients early, AI-based clinical prediction models have been increasingly studied. Support vector machines, logistic regression, random forest, gradient boosted machine and deep learning models have shown promise in predicting the outcomes of interest, including sepsis, need for intensive care unit admission, mortality, and respiratory failure [7]. Deep learning techniques, especially recurrent neural networks (RNNs) and long short-term memory (LSTM) networks, can process sequential clinical information and identify trends in it over time. In the hospitalized patient, these models are useful because physiological measurements may change over time, but often do not reach abnormal thresholds before deteriorating [8]. There are several studies that mention that AI models can detect high-risk patients sooner than traditional scoring systems. But a high predictive accuracy does not equal clinical effectiveness. Reliability, reproducibility, and usefulness within real-world healthcare workflows are the important prerequisites for the broad implementation of a model [9].
- IMPORTANCE OF EXTERNAL VALIDATION OF AI MODELS
External validation is an essential process to determine the performance of an AI model when tested on independent patient groups or healthcare environments. Differences in patient characteristics, disease prevalence, clinical practices, patient documentation and healthcare resources between hospitals may lead to differences in how a model performs when it is transferred to another hospital [10]. It is common for AI prediction models to perform well during internal testing and less effectively when applied to external data. This decrease can happen because of the difference between the development and implementation environments, which is referred to as dataset shift or model degradation [11]. External validation assesses a variety of key points of an AI model’s performance which helps to determine if a prediction model can be used in real-world clinical settings. Discrimination is the ability of a model to correctly distinguish between those patients who will deteriorate clinically and those patients who will remain stable. It is measured by a variety of methods including area under receiver operating characteristic curve (AUROC) and concordance index (C-index). A model that has good discrimination can correctly distinguish the risk level of patients, which can better identify individuals who need closer monitoring or early intervention [12]. But a high level of discrimination is not enough to make a model clinically useful as a model could classify high-risk patients as high risk and low-risk patients as low risk, but still be incorrect. Calibration assesses the agreement between the predicted probability of deterioration generated by the AI model and the actual observed outcomes in clinical practice. If a model can predict a 20% risk of deterioration in a group of patients, then about 20% of the patients should deteriorate. When a calibration is poor, there is a risk of the risk being over- or under-estimated, and thus either unnecessary intervention as well as delays in treatment. Hence, calibration assessment, both from calibration plots and calibration slope, and calibration-in-the-large, is necessary prior to clinical use [13]. Generalizability is defined as the ability to consistently perform well across multiple hospitals, health systems, geographic areas and patient populations outside of the training data. Model performance may vary due to differences in clinical practice, patient characteristics, disease prevalence, and documentation procedures, as well as resource availability. For instance, an AI system trained in a tertiary academic hospital may not be as effective at a community hospital that has different patient demographics. There is a need for external validation, across multiple centers, to ensure that the model is robust and transferable across a variety of clinical settings [14]. Fairness is a measure of the accuracy and equity of AI predictions across population and clinical subgroups, such as by age, sex, ethnicity, socioeconomic status, and comorbidities. Bias may arise if the training data do not contain equal representation of certain populations or if they may contain historical healthcare inequalities [15]. The overall model may be good, but less accurate in predicting a particular population of patients may lead to disparities in care. For this reason, equitable performance of AI systems is now a key consideration for validation, including an analysis of subgroups performance, possible algorithmic bias, and ways to enhance AI algorithm equity. These evaluation domains help to assess whether an AI-driven clinical deterioration prediction model is statistically accurate and reliable, clinically meaningful, and appropriate for clinical use. External validation helps build trust in the ability of AI systems to assist with clinical decision making, reducing the risk of incorrect predictions, transferability, and inequitable performance among patient groups [16].
- EXPLAINABLE ARTIFICIAL INTELLIGENCE IN HEALTHCARE
As AI models become more complex, there are concerns about transparency and interpretability. Clinicians might be reluctant to use predictions from algorithms if they are not aware of the logic behind their decisions. Explainable artificial intelligence (XAI) is a solution to this problem, which seeks to offer comprehensible explanations of model results. For clinical deterioration prediction, XAI techniques can identify key clinical indicators such as abnormal vital signs, inflammatory markers, laboratory abnormalities, medication changes, and comorbid conditions. These explanations can help clinicians understand the technology and enable the uptake of AI recommendations into clinical practice [17]. To make complicated machine learning models more transparent and interpretable, common Explainable Artificial Intelligence (XAI) techniques have been developed for clinical prediction. These approaches seek to provide insights into how AI systems make predictions and the clinical factors that influence patient risk assessments. One of the most popular XAI techniques, and is cooperative game-theory based, is SHapley Additive Explanations (SHAP). SHAP estimates the impact of each feature on the final prediction of a model, giving a quantitative estimate of the influence of each individual feature. For clinical deterioration prediction, the SHAP approach can reveal key predictors like trends in vital signs, laboratory data anomalies, age, comorbidities, or patterns of physiological deterioration that raise or lower a patient’s chances of deteriorating. This allows clinicians to determine the reason behind the alert they are receiving from an AI product and assess the likelihood of the prediction being clinically reasonable [18].
Another popular XAI technique is local interpretable model-agnostic explanations (LIME), which offers local explanations for specific predictions. Finally, LIME differs from approaches that explain the overall behavior of the model in that it explains why the model predicted a certain outcome for a specific patient. The approach generates simplified local models for each individual prediction and determines the top features that influenced the AI’s prediction. For instance, LIME can show that a high respiratory rate, abnormal laboratory values or falling blood pressure were most important in predicting deterioration in a patient that was hospitalized and had a high risk of deterioration. These personalized explanations can increase the clinician’s confidence and help the clinician make individualized decisions [19]. A simpler interpretability technique is feature importance analysis to determine which variables have the most significant impact on model performance and predictions. This method allows to rank features by their importance in the model, and helps to understand the most relevant features to be used for predicting the risk. For clinical deterioration models, feature importance analysis could identify key factors like oxygen desaturations, heart rate, respiratory rate, inflammatory markers, renal function markers or past medical history, which would help determine patient risk. While the feature importance methods are useful for understanding the behavior of the model, they may not be sufficient to fully explain the interactions between the variables, especially in more complex models like advanced deep learning architectures. These XAI methods, when combined, contribute to a more transparent view of AI predictions, making them comprehensible for a clinical context. SHAP, LIME, and feature importance analysis can enhance the trust of clinicians, enable more secure use of AI in decision support, and assist in the seamless integration of predictive models into clinical workflow within hospitals. XAI explanations, however, should be taken with a grain of salt: being interpretable does not prove that the variables identified in the explanation have causal relationships with clinical outcomes [20]. But, explainability is not synonymous with clinical validity. Certain explanation methods offer simplified versions of complex models, and need to be carefully analysed in order to demonstrate that they capture the decision process of the model [21].
- CLINICAL UTILITY AND IMPLEMENTATION OF EXPLAINABLE AI MODELS
The clinical utility of an AI model is its ability to enhance healthcare outcomes, decision-making or the use of resources. Any system that is clinically useful in predicting deterioration should allow earlier interventions, enhance patient safety and healthcare resource efficiency. The use of AI-powered early warning systems could give the time for clinicians to implement interventions to escalate monitoring or move patients to the next level of care. There is limited evidence of direct improvements to patient outcomes, however. Adoption requires alignment with workflows, uptake by clinicians, setting of suitable alarm limits, and proper response strategies [22]. Alert fatigue is one of the challenges – too many alerts reduce clinician attention and decreases system effectiveness. So, it is essential that the AI models alert clinically relevant events, not just maximize sensitivity.
- CHALLENGES AND FUTURE DIRECTIONS OF AI-BASED CLINICAL DETERIORATION PREDICTION MODELS
While there has been remarkable progress in Artificial Intelligence (AI) based prediction models for early detection of clinical deterioration, several key challenges lie ahead in the successful implementation of these systems in the clinical practice of routine care. While AI algorithms have shown promise in research environments, there are challenges to overcome when implementing them in clinical practice, such as data quality, algorithmic bias, model maintenance, workflow integration, ethical considerations, and assessing the impact of real-world clinical use [23]. The importance of these challenges is especially significant given the potential use of AI systems in healthcare settings where misjudgments can have a serious impact on patient safety and health outcomes.
6.1. Data Quality and Bias
The accuracy of AI clinical prediction models is dependent on the quality, completeness and representativeness of the data for model development and validation. Many deterioration prediction models rely on electronic health records (EHRs) as a key source of data, but these can be incomplete, inconsistent, or inaccurately documented. Laboratory measurements, vital signs, clinical events, and the coding of these can be missed, vital signs may not be recorded as often as they should be, clinical events may not be documented until an appreciable period after they occur, coding may differ, and different data collection methods may be used in different healthcare institutions, all of which can negatively influence model performance. While machine learning algorithms can detect patterns in big data that might otherwise not be found by traditional statistical methods, they can also fall prey to patterns that are already contained in the data set that they are being trained with. Algorithmic bias is one of the leading concerns in the field of AI in healthcare, as it leads to significantly different rates of accuracy when applied to various patient populations. Bias can occur when specific groups are underrepresented, such as older adults, racial and ethnic minorities, those with multiple chronic conditions, or those from low-resource populations, in training populations. Consequently, an AI model might work well for the general population but may not do as well for certain sub-populations. If the deterioration prediction model is largely created from data from a tertiary academic hospital, for instance, then it may not be valid to use the model for predicting risk in a rural hospital or a healthcare system with different patient characteristics and clinical practices. In addition, healthcare datasets may be a reflection of inequities in the past in access to healthcare, diagnosis and treatment. Such patterns, when fed into the algorithms of AI without proper intervention, can lead to unintentionally perpetuating inequalities. To ensure the fairness and reliability of AI systems, it is crucial to consider the diversity in datasets, assess the performance of models in various demographic groups, and implement measures to mitigate biases. Balanced sampling, including fairness-aware machine learning, subgroup analysis and transparent reporting of model limitations, are increasingly recommended for improving the reliability of AI-based clinical tools and ensuring their equity [11,19]. The distinction between statistical association and clinical relevance is another point that needs to be kept in mind. It is possible for AI models to detect variables that are highly correlated with deterioration, but are not clinically meaningful or modifiable risk factors. Thus, the cooperation between the data scientists and healthcare professionals is critical to ensure that these identified predictors are interpreted correctly and that they coincide with medical knowledge.
6.2 Model Updating and Continuous Monitoring
Clinics are changing constantly, as are healthcare systems with shifting treatment guidelines, disease demographics, patient populations, technology, and the approach to healthcare delivery. However, there can be a degradation in accuracy over time of AI models that perform well during initial development, which is called model drift. Model drift is the phenomenon in which the model’s relationship between the input variables and clinical outcomes change post-deployment, leading to reduced predictive power of the model. For instance, shifts in hospital admission rates, the inclusion of new drugs, updates to clinical management protocols, or variations in documentation systems can impact the data fed into AI models and consequently the results they generate.For instance, any changes in hospital admission patterns, introduction of new medications, changes to clinical management protocols, or changes in documentation systems can impact the characteristics of the data used by AI models and consequently the results they produce. A model developed prior to the broad use of some types of therapy or diagnostic technology may not be as accurate when applied years later. AI clinical prediction tools, therefore, need ongoing monitoring of their performance and regular updating for continued reliability. After deployment, regular evaluation should involve assessment of discrimination, calibration, fairness, and clinical impact. Monitoring systems should detect changes in the patient population, deterioration in the predictive performance, or unexpected differences in outcomes for patient groups. Validating AI models is crucial to ensure they remain accurate and that changes do not lead to unintended biases or lack of transparency. Therefore, a balance should be achieved between model accuracy and the stability, repeatability and compliance of the model. Furthermore, there should also be appropriate governance structures that will establish the responsibilities that are associated with the maintenance of the model, data management, cybersecurity and performance monitoring of the AI system deployed in the hospital. While this is the case for many traditional clinical tools, which tend to be fairly stable once validated, AI systems are technologies that continue to grow and change over time and should be evaluated on an ongoing basis.
6.3. Integration into Clinical Practice
The integration of such AI-driven deterioration prediction models into clinical workflows is also crucial for their successful implementation, alongside technical accuracy. If the clinicians do not trust, understand, or respond appropriately to the prediction of a highly accurate algorithm, patient outcomes will not be enhanced. So, AI systems should be used as a clinical decision support system that assists healthcare professionals, not to replace them. Factors that affect the acceptance by the clinicians are transparency of predication, usability, integration with existing hospital information systems, and perceived clinical usefulness. Explainable AI methods could help to enhance acceptance by offering understandable explanations for model predictions to the clinician. If healthcare providers can determine which patient factors led to an alert, they may be more inclined to consider and implement AI recommendations.
But if the implementation is not appropriate, it can cause new challenges. Too many alerts can lead to alert fatigue and may cause clinicians to refrain from taking the time to actually attend to important alerts. Thus, AI systems need to be precisely tuned to optimize sensitivity and specificity, guaranteeing that alerts are meaningful and actionable. It is necessary to have clear response protocols, staff training, and evaluation of workflow impact in integration strategies. It is equally important for multiple stakeholders to work together for successful adoption, such as clinicians, data scientists, information technology specialists, hospital administrators, and policymakers. Data scientists deliver expertise in algorithmic development and evaluation, while clinicians deliver expert knowledge in processes and clinical priorities. Administrators in the healthcare sector have a crucial part to play in ensuring availability of infrastructure, compliance with regulations, and sustainable implementation. More importantly, AI is not a replacement for clinical expertise but a technology that is supportive of the clinical process. Computational models of patient care simply cannot account for all aspects of patient care, such as patient preferences, ethics, social factors and clinical judgment. However, human oversight is still crucial to ensure that AI recommendations are understood in the context of the entire clinical journey.
- FUTURE RESEARCH DIRECTIONS
Further studies are needed to enhance the reliability, generalizability and clinical utility of AI-based deterioration prediction models. There is a need for large-scale and multi-centre prospective validation studies to assess the performance of AI models across a range of healthcare settings and patient cohorts. Prospective evaluation is especially important since retrospective studies may overestimate the performance of the model because of limited data sets and the lack of clinical complexity in the real world. Randomized clinical trials and real-world implementation studies will also need to be done to see if using AI for decision support will increase meaningful patient outcomes including reduced mortality, fewer admissions to the intensive care unit, shorter hospital stays, and better care. In addition to the predictive accuracy, future studies must look at usability, clinician acceptance, cost-effectiveness, and impact on healthcare workflows. Standardized reporting of AI prediction studies is also important. A standard reporting format like TRIPOD+AI will increase transparency, reproducibility and comparability of studies. Researchers should give comprehensive details about data, model development methods, validation strategies, data missing, fairness assessment, and limitations. Patient centered outcomes and ethical issues should also be considered for future AI systems. Models must be developed with attention to protection of privacy, security of data, fairness, and responsible patient information handling. For the development of trustworthy systems that can assist the clinician and enhance patient care, explainability, external validation, and continuous monitoring will be crucial. In conclusion, AI-driven deterioration prediction holds significant promise for reshaping the landscape of hospital care, but its adoption demands more than just algorithm creation; it calls for thorough assessment of its reliability, usability, and clinical impact. To ensure the ongoing success of AI in healthcare, it is crucial to develop systems that are accurate, transparent, adaptable, and seamlessly integrated with human clinical expertise.
- CONCLUSION
Explanatory AI models are a promising way of predicting clinical deterioration in hospitalized patients. These models have shown their capabilities to perform complex analysis of clinical data and to predict patients with high risk earlier than some traditional methods. But, external validation, calibration and fairness assessment, and measurable clinical benefit must, however, be done with rigor for successful clinical translation. While explainability techniques can help enhance transparency and trust from clinicians in AI-generated predictions, interpretability needs to be paired with strong validation and cautious implementation. Future studies should focus on prospective multicenter studies that evaluate predictive performance, as well as workflow integration, clinician adherence, and impact on patient outcomes. Accurate, externally validated and explainable AI models could play a part in creating safer and more effective clinical decision-support systems within hospital settings.
AUTHOR CONTRIBUTION STATEMENT
Anil Kuma: Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work, Drafting the work or reviewing it critically for important intellectual content; AND Final approval of the version to be published, and Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
ACKNOWLEDGEMENTS: No specific individuals, institutions, or organizations contributed additional support or assistance requiring formal acknowledgement for this study.
DECLARATIONS:
Funding: This research received no specific grant, financial support, or funding from any public, commercial, or non-profit organization.
Conflicts Of Interest: The authors declare that there are no conflicts of interest, financial or otherwise, that could have influenced the research, analysis, or interpretation of the findings presented in this article.
Data Availability: The datasets and information used in this study are publicly available from the sources cited within the article. No proprietary or restricted data were used.
Trial Registration: Trial registration is not applicable to this study, as it does not involve clinical trials, human participants, or experimental interventions.
Generative AI Disclosure: Generative artificial intelligence tools were not used for generating research data, conducting analyses, or producing scientific conclusions in this study. The authors take full responsibility for the content and accuracy of the manuscript
REFERENCES
- Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Medical Informatics and Decision Making, 20, Article 310. https://doi.org/10.1186/s12911-020-01332-6
- Boulitsakis Logothetis, S., Green, D., Holland, M., & Al Moubayed, N. (2023). Predicting acute clinical deterioration with interpretable machine learning to support emergency care decision making. Scientific Reports, 13, Article 13563. https://doi.org/10.1038/s41598-023-40661-0
- Brankovic, A., Hassanzadeh, H., Good, N., Mann, K., Khanna, S., Abdel-Hafez, A., & Cook, D. (2022). Explainable machine learning for real-time deterioration alert prediction to guide pre-emptive treatment. Scientific Reports, 12, Article 11734. https://doi.org/10.1038/s41598-022-15877-1
- Campbell, V., et al. (2020). Predicting clinical deterioration with Q-ADDS compared to NEWS, Between the Flags, and eCART track and trigger tools. Resuscitation, 153, 28–34. https://doi.org/10.1016/j.resuscitation.2020.05.027
- Choi, A., Choi, S. Y., Chung, K., Chung, H. S., Song, T., Choi, B., & Kim, J. H. (2023). Development of a machine learning-based clinical decision support system to predict clinical deterioration in patients visiting the emergency department. Scientific Reports, 13, Article 8561. https://doi.org/10.1038/s41598-023-35617-3
- Choi, A., Lee, K., Hyun, H., Kim, K. J., Ahn, B., Lee, K. H., Hahn, S., Choi, S. Y., & Kim, J. H. (2024). A novel deep learning algorithm for real-time prediction of clinical deterioration in the emergency department for a multimodal clinical decision support system. Scientific Reports, 14, Article 30116. https://doi.org/10.1038/s41598-024-80268-7
- Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., et al. (2024). Evaluation of clinical prediction models (part 1): From development to external validation. BMJ, 384, e074819. https://doi.org/10.1136/bmj-2023-074819
- Escobar, G. J., Liu, V. X., Schuler, A., Lawson, B., Greene, J. D., & Shah, N. H. (2020). Automated identification of adults at risk for in-hospital clinical deterioration. The New England Journal of Medicine, 383(20), 1951–1960. https://doi.org/10.1056/NEJMsa2001090
- Garcia-Gutiérrez, S., Esteban-Aizpiri, C., Lafuente, I., Barrio, I., Quiros, R., Quintana, J. M., Uranga, A., & COVID-REDISSEC Working Group. (2022). Machine learning-based model for prediction of clinical deterioration in hospitalized patients by COVID-19. Scientific Reports, 12, Article 7097. https://doi.org/10.1038/s41598-022-09771-z
- Gerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P. J., & Collins, G. S. (2020). Early warning scores for detecting deterioration in adult hospital patients: Systematic review and critical appraisal of methodology. BMJ, 369, m1501. https://doi.org/10.1136/bmj.m1501
- Guan, G., Lee, C. M. Y., Begg, S., Crombie, A., & Mnatzaganian, G. (2022). The use of early warning system scores in prehospital and emergency department settings to predict clinical deterioration: A systematic review and meta-analysis. PLOS ONE, 17(3), e0265559. https://doi.org/10.1371/journal.pone.0265559
- Gupta, R. K., Harrison, E. M., Ho, A., Docherty, A. B., Knight, S. R., van Smeden, M., et al. (2021). Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: A prospective cohort study. The Lancet Respiratory Medicine, 9(4), 349–359. https://doi.org/10.1016/S2213-2600(20)30559-2
- Kamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., Valley, T. S., West, L. R., Singh, K., Blumberg, S., Donnelly, J. P., Shenoy, E. S., Ayanian, J. Z., Nallamothu, B. K., Sjoding, M. W., & Wiens, J. (2022). Early identification of patients admitted to hospital for COVID-19 at risk of clinical deterioration: Model development and multisite external validation study. BMJ, 376, e068576. https://doi.org/10.1136/bmj-2021-068576
- Lauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020). Explainable artificial intelligence model to predict acute critical illness from electronic health records. Nature Communications, 11, Article 3852. https://doi.org/10.1038/s41467-020-17431-x
- Li, C., et al. (2021). Machine learning based early mortality prediction in the emergency department. International Journal of Medical Informatics, 155, 104570. https://doi.org/10.1016/j.ijmedinf.2021.104570
- Muralitharan, S., et al. (2021). Machine learning-based early warning systems for clinical deterioration: Systematic scoping review. Journal of Medical Internet Research, 23(2), e25187. https://doi.org/10.2196/25187
- Noy, O., Coster, D., Metzger, M., Atar, I., Shenhar-Tsarfaty, S., Berliner, S., Rahav, G., Rogowski, O., & Shamir, R. (2022). A machine learning model for predicting deterioration of COVID-19 inpatients. Scientific Reports, 12, Article 2630. https://doi.org/10.1038/s41598-022-05822-7
- O’Brien, C., Goldstein, B. A., Geller, N. L., & others. (2020). Development, implementation, and evaluation of an in-hospital optimized early warning score for patient deterioration. MDM Policy & Practice, 5(1). https://doi.org/10.1177/2381468319899663
- Rockenschaub, P., Akay, E. M., Carlisle, B. G., Hilbert, A., Wendland, J., Meyer-Eschenbach, F., Näher, A.-F., Frey, D., & Madai, V. I. (2025). External validation of AI-based scoring systems in the ICU: A systematic review and meta-analysis. BMC Medical Informatics and Decision Making, 25, Article 5. https://doi.org/10.1186/s12911-024-02830-7
- Romero-Brufau, S., et al. (2021). Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS). Journal of the American Medical Informatics Association, 28(6), 1207–1215. https://doi.org/10.1093/jamia/ocaa347
- Rockenschaub, P., Hilbert, A., Kossen, T., Elbers, P., von Dincklage, F., Madai, V. I., & others. (2024). The impact of multi-institution datasets on the generalizability of machine learning prediction models in the ICU. Critical Care Medicine. https://doi.org/10.1097/CCM.0000000000006359
- Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020). An overview of clinical decision support systems: Benefits, risks, and strategies for success. npj Digital Medicine, 3, Article 17. https://doi.org/10.1038/s41746-020-0221-y
- Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023). There is no such thing as a validated prediction model. BMC Medicine, 21, Article 70. https://doi.org/10.1186/s12916-023-02779-w
SUPPLEMENTAL MATERIAL
No supplementary files, datasets, figures, or additional materials are associated with this study. Any information necessary to understand and evaluate the work is included in the main article
Citation
Anil Kuma (2026). External Validation and Clinical Utility of Explainable Artificial Intelligence Models for Early Prediction of Clinical Deterioration in Hospitalized Patients. Dinkum Journal of Medical Innovations, 5(04): 318-326. https://doi.org/10.71017/djmi.5.04.d-0604
DOI
https://doi.org/10.71017/djmi.5.04.d-0604
Identification
D-0604
Publication History
Submitted: 19-03-2026
Accepted: 19-04-2026
Published: 30-04-2026
Copyright
© The Author(s) 2026
Publishing License
This is an Open Access article distributed under the terms of the Creative Commons Attribution (CC BY 4.0) License .
