Deep learning algorithm performance compared to experts in visual evaluation of interior vena cava collapse on ultrasound to determine intravenous ﬂuid need in dehydration management
1Department of Medicine, University of South Carolina School of Medicine, Department of Emergency Medicine, St. Francis Hospital, Columbus, GA 769209, USA
2Michigan State University, East Lancing, MI 48825, USA
3Department of Emergency Medicine, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA
Submitted: 12 May 2021 Accepted: 23 June 2021
Online publish date: 14 July 2021
Objectives: To create a deep learning (DL) algorithm capable of analyzing real time ultrasound video of the inferior vena cava (IVC) for complete collapse in pediatric patients being evaluated for intravenous fluid (IVF) resuscitation.
Methods: Researchers employed a VGG-16 based DL architecture, running inside a Long Short Term Memory algorithm design, to analyze prospectively obtained ultrasound video from pediatric patients presenting with dehydration to a busy urban ED, obtained for a prior clinical study. All videos were de-identified and no patient information was available. A total of 184 patient IVC ultrasound videos were used in the study. All videos were previously reviewed and graded by two blinded POCUS experts (PedEM fellow and PedEM attending with 20 years experience) and split into two categories, those showing complete (95 patients) and those with incomplete (89 patients) IVC collapse. Approximately 10% (9) patient videos were randomly removed from each original data groups to be used for algorithm testing after training was completed. A standard 80%/20% training and validation split was used on the remaining 166 patient videos for algorithm training. Training accuracy, losses and learning curves were tracked and various training parameters such as learning rates and batch sizes were optimized throughout training. As a final real world test, the DL algorithm was tasked with analyzing the 18 previously unseen, randomly selected IVC videos. Cohen’s kappa was calculated for each of the blinded POCUS reviewers and DL algorithm.
Results: This DL algorithm completed analysis of each previously unseen real world test video and is the first such algorithm to analyze IVC collapse through visual estimation in real-time. The algorithm was able to deliver a collapse result prediction for all 18 test IVC videos and there were no failures. Algorithm agreement with PedEM POCUS attending was substantial with a Cohen’s kappa of 0.78 (95% CI 0.49 to 1.0). Algorithm agreement with PedEM Fellow was substantial with Cohen’s kappa of 0.66 (95% CI 0.31 to 1.0). The PEM fellow and PEM POCUS attending also had substantial agreement, yielding a Cohen’s kappa of 0.66 (95% CI 0.32 to 1.0).
Conclusions: This DL algorithm developed on prospectively acquired IVC video data from patients being studied for an IVF resuscitation study proved accurate at identifying when the IVC collapsed completely in real time. There was substantial agreement with POCUS reviewers of the same videos. Such an algorithm could allow novice clinicians to rapidly identify complete IVC collapse in children and the need for IVF administration. This could expand patient access to point of care technology by enabling novices with little training to use the diagnostic tool at bedside and decide if patients require intravenous fluid administration.
Deep learning; Artificial intelligence; Long short term memory; Point-of-care ultrasound; Emergency medicine; Critical care; Inferior vena cava; Fluid responsiveness
Michael Blaivas,Laura N Blaivas,James W Tsung. Deep learning algorithm performance compared to experts in visual evaluation of interior vena cava collapse on ultrasound to determine intravenous ﬂuid need in dehydration management. Signa Vitae. 2021.doi:10.22514/sv.2021.128.
 Alpern ER, Stanley RM, Gorelick MH, Donaldson A, Knight S, Teach SJ, et al. Pediatric Emergency Care Applied Research Network. Epidemiology of a pediatric emergency medicine research network: the PECARN Core Data Project. Pediatric Emergency Care. 2006; 22: 689–699.
 Modi P, Glavis-Bloom J, Nasrin S, Guy A, Chowa EP, Dvor N, et al. Accuracy of Inferior Vena Cava Ultrasound for Predicting Dehydration in Children with Acute Diarrhea in Resource-Limited Settings. PLoS ONE. 2016; 11: e0146859.
 Gorelick MH, Shaw KN, Murphy KO. Validity and reliability of clinical signs in the diagnosis of dehydration in children. Pediatrics. 1997; 99: E6.
 Yen K, Riegert A, Gorelick MH. Derivation of the DIVA score: a clinical prediction rule for the identification of children with difficult intravenous access. Pediatric Emergency Care. 2008; 24: 143–147.
 Goldman RD, Friedman JN, Parkin PC. Validation of the clinical dehydration scale for children with acute gastroenteritis. Pediatrics. 2008; 122: 545–549.
 Kinlin LM, Freedman SB. Evaluation of a Clinical Dehydration Scale in Children Requiring Intravenous Rehydration. Pediatrics. 2012; 129: e1211–e1219.
 Pershad J, Myers S, Plouman C, Rosson C, Elam K, Wan J, et al. Bedside limited echocardiography by the emergency physician is accurate during evaluation of the critically ill patient. Pediatrics. 2005; 114: e667–e671.
 Orso D, Paoli I, Piani T, Cilenti FL, Cristiani L, Guglielmo N. Accuracy of Ultrasonographic Measurements of Inferior Vena Cava to Determine Fluid Responsiveness: a Systematic Review and Meta-Analysis. Journal of Intensive Care Medicine. 2020; 35: 354–363.
 Corl KA, Azab N, Nayeemuddin M, Schick A, Lopardo T, Zeba F, et al. Performance of a 25
 Bussmann BM, Sharma S, Mcgregor D, Hulme W, Harris T. Observa-tional study in healthy volunteers to define interobserver reliability of ultrasound haemodynamic monitoring techniques performed by trainee doctors. European Journal of Emergency Medicine. 2019; 26: 217–223.
 Bowra J, Uwagboe V, Goudie A, Reid C, Gillett M. Interrater agreement between expert and novice in measuring inferior vena cava diameter and collapsibility index. Emergency Medicine Australasia. 2015; 27: 295–299.
 Akkaya A, Yesilaras M, Aksay E, Sever M, Atilla OD. The interrater reliability of ultrasound imaging of the inferior vena cava performed by emergency residents. The American Journal of Emergency Medicine. 2013; 31: 1509–1511.
 Zhou AZ, Green RS, Haines EJ, Vazquez MN, Tay ET, Tsung JW. Interobserver Agreement of Inferior Vena Cava Ultrasound Collapse Duration and Correlated Outcomes in Children with Dehydration. Pediatric Emergency Care. 2020. (in press)
 Shokoohi H, LeSaux MA, Roohani YH, Liteplo A, Huang C, Blaivas M. Enhanced Point-of-Care Ultrasound Applications by Integrating Automated Feature‐Learning Systems Using Deep Learning. Journal of Ultrasound in Medicine. 2019; 38: 1887–1897.
 Blaivas M, Blaivas L. Are all Deep Learning Architectures Alike for Point-of-Care Ultrasound?: Evidence from a Cardiac Image Classifica-tion Model Suggests otherwise. Journal of Ultrasound in Medicine. 2020; 39: 1187–1194.
 Çınar A, Tuncer SA. Classification of normal sinus rhythm, abnormal arrhythmia and congestive heart failure ECG signals using LSTM and hybrid CNN-SVM deep neural networks. Computer Methods in Biomechanics and Biomedical Engineering. 2021; 24: 203–214.
 Lai CQ, Ibrahim H, Abd Hamid AI, Abdullah JM. Classification of Non-Severe Traumatic Brain Injury from Resting-State EEG Signal Using LSTM Network with ECOC-SVM. Sensors. 2020; 20: 5234.
 Ko H, Chung H, Lee H, Lee J. Feasible Study on Intracranial Hemorrhage Detection and Classification using a CNN-LSTM Network. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2020; 2020: 1290–1293.
 Blaivas M, Blaivas L, Philips G, Merchant R, Levy M, Abbasi A, et al. Development of a Deep Learning Network to Classify Inferior Vena Cava Collapse to Predict Fluid Responsiveness. Journal of Ultrasound in Medicine. 2020. (in press)
 Blaivas M, Adhikari S, Savitsky EA, Blaivas LN, Liu YT. Artificial intelligence versus expert: a comparison of rapid visual inferior vena cava collapsibility assessment between POCUS experts and a deep learning algorithm. Journal of the American College of Emergency Physicians Open. 2020; 1: 857–864.
 Mwikirize C, Kimbowa AB, Imanirakiza S, Katumba A, Nosher JL, Hacihaliloglu I. Time-aware deep neural networks for needle tip localization in 2D ultrasound. International Journal of Computer Assisted Radiology and Surgery. 2021; 16: 819–827.
 Dastider AG, Sadik F, Fattah SA. An integrated autoencoder-based hybrid CNN-LSTM model for COVID-19 severity prediction from lung ultrasound. Computers in Biology and Medicine. 2021; 132: 104296.
 Chen L, Kim Y, Santucci KA. Use of ultrasound measurement of the inferior vena cava diameter as an objective tool in the assessment of children with clinical dehydration. Academic Emergency Medicine. 2007; 14: 841–845.
 Kornblith AE, van Schaik S, Reynolds T. Useful but not used: pediatric critical care physician views on bedside ultrasound. Pediatric Emergency Care. 2015; 31: 186–189.
 Sawe HR, Haeffele C, Mfinanga JA, Mwafongo VG, Reynolds TA. Predicting Fluid Responsiveness Using Bedside Ultrasound Measure-ments of the Inferior Vena Cava and Physician Gestalt in the Emergency Department of an Urban Public Hospital in Sub-Saharan Africa. PLoS ONE. 2016; 11: e0162772.
 Mesin L, Pasquero P, Albani S, Porta M, Roatta S. Semi-automated tracking and continuous monitoring of inferior vena cava diameter in simulated and experimental ultrasound imaging. Ultrasound in Medicine & Biology. 2015; 41: 845–857.
 Chen J, Li J, Ding X, Chang C, Wang X, Ta D. Automated Identification and Localization of the Inferior Vena Cava Using Ultrasound: an Animal Study. Ultrasonic Imaging. 2018; 40: 232–244.
 Belmont B, Kessler R, Theyyunni N, Fung C, Huang R, Cover M, et al. Continuous Inferior Vena Cava Diameter Tracking through an Iterative Kanade–Lucas–Tomasi-Based Algorithm. Ultrasound in Medicine & Biology. 2018; 44: 2793–2801.
 Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet. 2018; 392: 2388–2396.
 Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers. Korean Journal of Radiology. 2019; 20: 405–410.
Science Citation Index Expanded (SciSearch) Created as SCI in 1964, Science Citation Index Expanded now indexes over 9,200 of the world’s most impactful journals across 178 scientific disciplines. More than 53 million records and 1.18 billion cited references date back from 1900 to present.
Journal Citation Reports/Science Edition Journal Citation Reports/Science Edition aims to evaluate a journal’s value from multiple perspectives including the journal impact factor, descriptive data about a journal’s open access content as well as contributing authors, and provide readers a transparent and publisher-neutral data & statistics information about the journal.
Chemical Abstracts Service Source Index The CAS Source Index (CASSI) Search Tool is an online resource that can quickly identify or confirm journal titles and abbreviations for publications indexed by CAS since 1907, including serial and non-serial scientific and technical publications.
IndexCopernicus The Index Copernicus International (ICI) Journals database’s is an international indexation database of scientific journals. It covered international scientific journals which divided into general information, contents of individual issues, detailed bibliography (references) sections for every publication, as well as full texts of publications in the form of attached files (optional). For now, there are more than 58,000 scientific journals registered at ICI.
Geneva Foundation for Medical Education and Research The Geneva Foundation for Medical Education and Research (GFMER) is a non-profit organization established in 2002 and it works in close collaboration with the World Health Organization (WHO). The overall objectives of the Foundation are to promote and develop health education and research programs.
Scopus: CiteScore 0.5(2019) Scopus is Elsevier's abstract and citation database launched in 2004. Scopus covers nearly 36,377 titles (22,794 active titles and 13,583 Inactive titles) from approximately 11,678 publishers, of which 34,346 are peer-reviewed journals in top-level subject fields: life sciences, social sciences, physical sciences and health sciences.
Embase Embase (often styled EMBASE for Excerpta Medica dataBASE), produced by Elsevier, is a biomedical and pharmacological database of published literature designed to support information managers and pharmacovigilance in complying with the regulatory requirements of a licensed drug.