Deep learning algorithm performance compared to experts in visual evaluation of interior vena cava collapse on ultrasound to determine intravenous fluid need in dehydration management

Objectives: To create a deep learning (DL) algorithm capable of analyzing real time ultrasound video of the inferior vena cava (IVC) for complete collapse in pediatric patients being evaluated for intravenous fluid (IVF) resuscitation.

Methods: Researchers employed a VGG-16 based DL architecture, running inside a Long Short Term Memory algorithm design, to analyze prospectively obtained ultrasound video from pediatric patients presenting with dehydration to a busy urban ED, obtained for a prior clinical study. All videos were de-identified and no patient information was available. A total of 184 patient IVC ultrasound videos were used in the study. All videos were previously reviewed and graded by two blinded POCUS experts (PedEM resident and PedEM attending with 20 years experience) and split into two categories, those showing complete (95 patients) and those with incomplete (89 patients) IVC collapse. Approximately 10% (9) patient videos were randomly removed from each original data groups to be used for algorithm testing after training was completed. A standard 80%/20% training and validation split was used on the remaining 166 patient videos for algorithm training. Training accuracy, losses and learning curves were tracked and various training parameters such as learning rates and batch sizes were optimized throughout training. As a final real world test, the DL algorithm was tasked with analyzing the 18 previously unseen, randomly selected IVC videos. Cohen’s kappa was calculated for each of the blinded POCUS reviewers and DL algorithm.

Results: This DL algorithm completed analysis of each previously unseen real world test video and is the first such algorithm to analyze IVC collapse through visual estimation in real-time. The algorithm was able to deliver a collapse result prediction for all 18 test IVC videos and there were no failures. Algorithm agreement with PedEM POCUS attending was substantial with a Cohen’s kappa of 0.78 (95% CI 0.49 to 1.0). Algorithm agreement with PedEM resident was substantial with Cohen’s kappa of 0.66 (95%CI 0.31 to 1.0). The PEM resident and PEM POCUS attending also had substantial agreement, yielding a Cohen’s kappa of 0.66 (95% CI 0.32 to 1.0).

Conclusions: This DL algorithm developed on prospectively acquired IVC video data from patients being studied for an IVF resuscitation study proved accurate at identifying when the IVC collapsed completely in real time. There was substantial agreement with POCUS reviewers of the same videos. Such an algorithm could allow novice clinicians to rapidly identify complete IVC collapse in children and the need for IVF administration. This could expand patient access to point of care technology by enabling novices with little training to use the diagnostic tool at bedside and decide if patients require intravenous fluid administration.

Deep learning; Artificial intelligence; Long short term memory; Point-of-care ultrasound; Emergency medicine; Critical care; Inferior vena cava; Fluid responsiveness

[1] Alpern ER, Stanley RM, Gorelick MH, Donaldson A, Knight S, Teach SJ, et al. Pediatric Emergency Care Applied Research Network. Epidemiology of a pediatric emergency medicine research network: the PECARN Core Data Project. Pediatric Emergency Care. 2006; 22: 689–699.

[2] Modi P, Glavis-Bloom J, Nasrin S, Guy A, Chowa EP, Dvor N, et al. Accuracy of Inferior Vena Cava Ultrasound for Predicting Dehydration in Children with Acute Diarrhea in Resource-Limited Settings. PLoS ONE. 2016; 11: e0146859.

[3] Gorelick MH, Shaw KN, Murphy KO. Validity and reliability of clinical signs in the diagnosis of dehydration in children. Pediatrics. 1997; 99: E6.

[4] Yen K, Riegert A, Gorelick MH. Derivation of the DIVA score: a clinical prediction rule for the identification of children with difficult intravenous access. Pediatric Emergency Care. 2008; 24: 143–147.

[5] Goldman RD, Friedman JN, Parkin PC. Validation of the clinical dehydration scale for children with acute gastroenteritis. Pediatrics. 2008; 122: 545–549.

[6] Kinlin LM, Freedman SB. Evaluation of a Clinical Dehydration Scale in Children Requiring Intravenous Rehydration. Pediatrics. 2012; 129: e1211–e1219.

[7] Pershad J, Myers S, Plouman C, Rosson C, Elam K, Wan J, et al. Bedside limited echocardiography by the emergency physician is accurate during evaluation of the critically ill patient. Pediatrics. 2005; 114: e667–e671.

[8] Orso D, Paoli I, Piani T, Cilenti FL, Cristiani L, Guglielmo N. Accuracy of Ultrasonographic Measurements of Inferior Vena Cava to Determine Fluid Responsiveness: a Systematic Review and Meta-Analysis. Journal of Intensive Care Medicine. 2020; 35: 354–363.

[9] Corl KA, Azab N, Nayeemuddin M, Schick A, Lopardo T, Zeba F, et al. Performance of a 25

[10] Bussmann BM, Sharma S, Mcgregor D, Hulme W, Harris T. Observa-tional study in healthy volunteers to define interobserver reliability of ultrasound haemodynamic monitoring techniques performed by trainee doctors. European Journal of Emergency Medicine. 2019; 26: 217–223.

[11] Bowra J, Uwagboe V, Goudie A, Reid C, Gillett M. Interrater agreement between expert and novice in measuring inferior vena cava diameter and collapsibility index. Emergency Medicine Australasia. 2015; 27: 295–299.

[12] Akkaya A, Yesilaras M, Aksay E, Sever M, Atilla OD. The interrater reliability of ultrasound imaging of the inferior vena cava performed by emergency residents. The American Journal of Emergency Medicine. 2013; 31: 1509–1511.

[13] Zhou AZ, Green RS, Haines EJ, Vazquez MN, Tay ET, Tsung JW. Interobserver Agreement of Inferior Vena Cava Ultrasound Collapse Duration and Correlated Outcomes in Children with Dehydration. Pediatric Emergency Care. 2020. (in press)

[14] Shokoohi H, LeSaux MA, Roohani YH, Liteplo A, Huang C, Blaivas M. Enhanced Point-of-Care Ultrasound Applications by Integrating Automated Feature‐Learning Systems Using Deep Learning. Journal of Ultrasound in Medicine. 2019; 38: 1887–1897.

[15] Blaivas M, Blaivas L. Are all Deep Learning Architectures Alike for Point-of-Care Ultrasound?: Evidence from a Cardiac Image Classifica-tion Model Suggests otherwise. Journal of Ultrasound in Medicine. 2020; 39: 1187–1194.

[16] Çınar A, Tuncer SA. Classification of normal sinus rhythm, abnormal arrhythmia and congestive heart failure ECG signals using LSTM and hybrid CNN-SVM deep neural networks. Computer Methods in Biomechanics and Biomedical Engineering. 2021; 24: 203–214.

[17] Lai CQ, Ibrahim H, Abd Hamid AI, Abdullah JM. Classification of Non-Severe Traumatic Brain Injury from Resting-State EEG Signal Using LSTM Network with ECOC-SVM. Sensors. 2020; 20: 5234.

[18] Ko H, Chung H, Lee H, Lee J. Feasible Study on Intracranial Hemorrhage Detection and Classification using a CNN-LSTM Network. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2020; 2020: 1290–1293.

[19] Blaivas M, Blaivas L, Philips G, Merchant R, Levy M, Abbasi A, et al. Development of a Deep Learning Network to Classify Inferior Vena Cava Collapse to Predict Fluid Responsiveness. Journal of Ultrasound in Medicine. 2020. (in press)

[20] Blaivas M, Adhikari S, Savitsky EA, Blaivas LN, Liu YT. Artificial intelligence versus expert: a comparison of rapid visual inferior vena cava collapsibility assessment between POCUS experts and a deep learning algorithm. Journal of the American College of Emergency Physicians Open. 2020; 1: 857–864.

[21] Mwikirize C, Kimbowa AB, Imanirakiza S, Katumba A, Nosher JL, Hacihaliloglu I. Time-aware deep neural networks for needle tip localization in 2D ultrasound. International Journal of Computer Assisted Radiology and Surgery. 2021; 16: 819–827.

[22] Dastider AG, Sadik F, Fattah SA. An integrated autoencoder-based hybrid CNN-LSTM model for COVID-19 severity prediction from lung ultrasound. Computers in Biology and Medicine. 2021; 132: 104296.

[23] Chen L, Kim Y, Santucci KA. Use of ultrasound measurement of the inferior vena cava diameter as an objective tool in the assessment of children with clinical dehydration. Academic Emergency Medicine. 2007; 14: 841–845.

[24] Kornblith AE, van Schaik S, Reynolds T. Useful but not used: pediatric critical care physician views on bedside ultrasound. Pediatric Emergency Care. 2015; 31: 186–189.

[25] Sawe HR, Haeffele C, Mfinanga JA, Mwafongo VG, Reynolds TA. Predicting Fluid Responsiveness Using Bedside Ultrasound Measure-ments of the Inferior Vena Cava and Physician Gestalt in the Emergency Department of an Urban Public Hospital in Sub-Saharan Africa. PLoS ONE. 2016; 11: e0162772.

[26] Mesin L, Pasquero P, Albani S, Porta M, Roatta S. Semi-automated tracking and continuous monitoring of inferior vena cava diameter in simulated and experimental ultrasound imaging. Ultrasound in Medicine & Biology. 2015; 41: 845–857.

[27] Chen J, Li J, Ding X, Chang C, Wang X, Ta D. Automated Identification and Localization of the Inferior Vena Cava Using Ultrasound: an Animal Study. Ultrasonic Imaging. 2018; 40: 232–244.

[28] Belmont B, Kessler R, Theyyunni N, Fung C, Huang R, Cover M, et al. Continuous Inferior Vena Cava Diameter Tracking through an Iterative Kanade–Lucas–Tomasi-Based Algorithm. Ultrasound in Medicine & Biology. 2018; 44: 2793–2801.

[29] Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet. 2018; 392: 2388–2396.

[30] Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers. Korean Journal of Radiology. 2019; 20: 405–410.