AI-based cancer pain assessment through speech emotion recognition and video facial expressions classification

The effective assessment of cancer pain requires a meticulous analysis of all the components that shape the painful experience collectively. Implementing Automatic Pain Assessment (APA) methods and computational analytical approaches, with a specific focus on emotional content, can facilitate a thorough characterization of pain. The proposed approach moves towards the use of automatic emotion recognition from speech recordings alongside a model we previously developed to examine facial expressions of pain. For training and validation, we adopted the EMOVO dataset, which simulates six emotional states (the Big Six). A Neural Network, consisting of a Multi-Layered Perceptron, was trained on 181 prosodic features to classify emotions. For testing, we used a dataset of interviews collected from cancer patients and selected two case studies. Speech annotation and continuous facial expression analysis (resulting in pain/no pain classifications) were carried out using Eudico Linguistic Annotator (ELAN) version 6.7. The model for emotion analysis achieved 84% accuracy, with encouraging precision, recall, and F1-score metrics across all classes. The preliminary results suggest the potential use of artificial intelligence (AI) strategies for continuous estimation of emotional states from video recordings, unveiling predominant emotional states, and providing the ability to corroborate the corresponding pain assessment. Despite limitations, the proposed AI framework exhibits potential for holistic and real-time pain assessment, paving the way for personalized pain management strategies in oncological settings. Clinical Trial registration: NCT04726228.

Automatic pain assessment; Pain; Cancer pain; Artificial intelligence; Speech analysis; Computational language analysis; Speech emotion recognition

[1] van den Beuken-van Everdingen MH, Hochstenbach LM, Joosten EA, Tjan-Heijnen VC, Janssen DJ. Update on prevalence of pain in patients with cancer: systematic review and meta-analysis. Journal of Pain and Symptom Management. 2016; 51: 1070–1090.e9.

[2] Cascella M, Vittori A, Petrucci E, Marinangeli F, Giarratano A, Cacciagrano C, et al. Strengths and weaknesses of cancer pain management in italy: findings from a nationwide SIAARTI survey. Healthcare. 2022; 10: 441.

[3] Caraceni A, Shkodra M. Cancer pain assessment and classification. Cancers. 2019; 11: 510.

[4] Giordano V, Deindl P, Olischar M. The limitations of pain scales—reply. JAMA Pediatrics. 2020; 174: 623.

[5] Baamer RM, Iqbal A, Lobo DN, Knaggs RD, Levy NA, Toh LS. Utility of unidimensional and functional pain assessment tools in adult postoperative patients: a systematic review. British Journal of Anaesthesia. 2022; 128: 874–888.

[6] Aung MSH, Kaltwang S, Romera-Paredes B, Martinez B, Singh A, Cella M, et al. The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal EmoPain dataset. IEEE Transactions on Affective Computing. 2016; 7: 435–451.

[7] Gkikas S, Tsiknakis M. Automatic assessment of pain based on deep learning methods: a systematic review. Computer Methods and Programs in Biomedicine. 2023; 231: 107365.

[8] Cascella M, Vitale VN, Mariani F, Iuorio M, Cutugno F. Development of a binary classifier model from extended facial codes toward video-based pain recognition in cancer patients. Scandinavian Journal of Pain. 2023; 23: 638–645.

[9] Gilam G, Gross JJ, Wager TD, Keefe FJ, Mackey SC. What is the relationship between pain and emotion? Bridging Constructs and Communities. Neuron. 2020; 107: 17–21.

[10] Asghar A, Sohaib S, Iftikhar S, Shafi M, Fatima K. An Urdu speech corpus for emotion recognition. PeerJ Computer Science. 2022; 8: e954.

[11] Atmaja BT, Sasou A. Sentiment analysis and emotion recognition from speech using universal speech representations. Sensors. 2022; 22: 6369.

[12] Deshpande G, Schuller BW, Deshpande P, Joshi AR. Automatic breathing pattern analysis from reading-speech signals, 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Sydney, Australia 24th–27th July 2023. 2023.

[13] Chowdhary KR. Natural language processing. In Chowdhary KR (ed.) Fundamentals of Artificial Intelligence (pp. 603–649). Springer: New Delhi. 2020.

[14] Srinivasan R, Subalalitha CN. Sentimental analysis from imbalanced code-mixed data using machine learning approaches. Distributed and Parallel Databases. 2023; 41: 37–52.

[15] Ku PKM, Vlantis AC, Yeung ZWC, Ho OYM, Cho RHW, Lee AKF, et al. Perceptual voice and speech analysis after supraglottic laryngeal closure for chronic aspiration in head and neck cancer. Laryngoscope. 2021; 131: E1616–E1623.

[16] Husain M, Simpkin A, Gibbons C, Talkar T, Low D, Bonato P, et al. Artificial Intelligence for Detecting COVID-19 with the aid of human cough, breathing and speech signals: scoping review. IEEE Open Journal of Engineering in Medicine and Biology. 2022; 3: 235–241.

[17] Kerdvibulvech C, Chen L. The power of augmented reality and artificial intelligence during the COVID-19 outbreak, HCI International 2020—Late Breaking Papers: Multimodality and Intelligence. Copenhagen, Denmark, July 19–24, 2020. Springer International Publishing. 2020.

[18] Xie X, Cai H, Li C, Wu Y, Ding F. A Voice Disease Detection Method Based on MFCCs and Shallow CNN. Journal of Voice. 2023. [Preprint].

[19] Koops S, Brederoo SG, de Boer JN, Nadema FG, Voppel AE, Sommer IE. Speech as a biomarker for depression. CNS & Neurological Disorders Drug Targets. 2023; 22: 152–160.

[20] Yokoi K, Iribe Y, Kitaoka N, Tsuboi T, Hiraga K, Satake Y, et al. Analysis of spontaneous speech in Parkinson’s disease by natural language processing. Parkinsonism & Related Disorders. 2023; 113: 105411.

[21] Costantini G, Iaderola I, Paoloni A, Todisco M. EMOVO corpus: an Italian emotional speech database, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). Reykjavik, Iceland. May 26-31, 2014. 2014.

[22] Cowie R, Cornelius RR. Describing the emotional states that are expressed in speech. Speech Communication. 2003; 40: 5–32.

[23] Khare Y. Hands-on-guide to Librosa for handling audio files. 2024. Available at: https://www.analyticsvidhya.com/blog/2024/01/hands-on-guide-to-librosa-for-handling-audio-files/ (Accessed: 04 January 2023).

[24] Jeevitha M. Exploring Librosa: a comprehensive guide to audio feature extraction from WAV files. 2023. Available at: https://www.linkedin.com/pulse/exploring-librosa-comprehensive-guide-audio-feature-extraction-m/ (Accessed: 04 January 2023).

[25] Kingma DP, Ba LJ. Adam: a method for stochastic optimization, International Conference on Learning Representations (ICLR). San Diego, May 7–9, 2015.

[26] Mende-Siedlecki P, Qu-Lee J, Lin J, Drain A, Goharzad A. The delaware pain database: a set of painful expressions and corresponding norming data. PAIN Reports. 2020; 5: e853.

[27] Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I. Painful data: the UNBC-McMaster shoulder pain expression archive database, 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA. 2011.

[28] Cascella M. Dataset for binary classifier_Pain. 2023. Available at: https://doi.org/10.5281/zenodo.7557362 (Accessed: 04 January 2023).

[29] Cuomo A, Cascella M, Forte CA, Bimonte S, Esposito G, De Santis S, et al. Careful breakthrough cancer pain treatment through rapid-onset transmucosal fentanyl improves the quality of life in cancer patients: results from the BEST multicenter study. Journal of Clinical Medicine. 2020; 9: 1003.

[30] Zhang M, Zhu L, Lin SY, Herr K, Chi CL, Demir I, et al. Using artificial intelligence to improve pain assessment and pain management: a scoping review. Journal of the American Medical Informatics Association. 2023; 30: 570–587.

[31] Cascella M, Schiavo D, Cuomo A, Ottaiano A, Perri F, Patrone R, et al. Artificial intelligence for automatic pain assessment: research methods and perspectives. Pain Research and Management. 2023; 2023: 6018736.

[32] Nagireddi JN, Vyas AK, Sanapati MR, Soin A, Manchikanti L. The analysis of pain research through the lens of artificial intelligence and machine learning. Pain Physician. 2022; 25: E211–E243.

[33] Sankaran R, Kumar A, Parasuram H. Role of artificial intelligence and machine learning in the prediction of the pain: a scoping systematic review. Proceedings of the Institution of Mechanical Engineers. Part H, Journal of Engineering in Medicine. 2022; 236: 1478–1491.

[34] Tsai FS, Weng YM, Ng CJ, Lee CC. Embedding stacked bottleneck vocal features in a LSTM architecture for automatic pain level classification during emergency triage, Proceedings of the 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). San Antonio, TX, USA, 23–26 October 2017. 2017.

[35] Li JL, Weng YM, Ng CJ, Lee CC. Learning conditional acoustic latent representation with gender and age attributes for automatic pain level recognition, Proceedings of the Interspeech 2018. Hyderabad, India, 2–6 September 2018. 2018.

[36] Schneiders E, Williams J, Farahi A, Seabrooke T, Vigneswaran G, Bautista JR, et al. TAME pain: trustworthy assessment of pain from speech and audio for the empowerment of patients, Proceedings of the First International Symposium on Trustworthy Autonomous Systems. New York, July 2023. 2023.

[37] Bellini V, Cascella M, Cutugno F, Russo M, Lanza R, Compagnone C, et al. Understanding basic principles of artificial intelligence: a practical guide for intensivists. Acta Biomedica. 2022; 93: e2022297.

[38] Dahan D. Prosody and language comprehension. Wiley Interdisciplinary Reviews: Cognitive Science. 2015; 6: 441–452.

[39] Speer S, Blodgett A. Prosody. In: Traxler M, Gernsbacher MA, eds. Handbook of psycholinguistics (pp. 505–537). 2nd edn. Academic Press: San Diego, CA. 2006.

[40] Carlson K. How prosody influences sentence comprehension. Language and Linguistics Compass. 2009; 3: 1188–1200.

[41] Schiavo D, Cumo A, Nocerino D, Monaco F, Cascella M. The body of pain. The experience of pain in the cancer patient. Recenti Progressi in Medicina. 2023; 114: 410–413. (In Italian)

[42] Siler S, Borneman T, Ferrell B. Pain and suffering. Seminars in Oncology Nursing. 2019; 35: 310–314.

[43] Erol O, Unsar S, Yacan L, Pelin M, Kurt S, Erdogan B. Pain experiences of patients with advanced cancer: a qualitative descriptive study. European Journal of Oncology Nursing. 2018; 33: 28–34.

[44] Venkitakrishnan S, Wu YH. Facial Expressions as an index of listening difficulty and emotional response. Seminars in Hearing. 2023; 44: 166–187.

[45] Zhang H, Chen X, Chen S, Li Y, Chen C, Long Q, Yuan J. Facial expression enhances emotion perception compared to vocal prosody: behavioral and fMRI studies. Neuroscience Bulletin. 2018; 34: 801–815.

[46] Cascella M, Vitale VN, D’Antò M, Cuomo A, Amato F, Romano M, et al. Exploring biosignals for quantitative pain assessment in cancer patients: a proof of concept. Electronics. 2023; 12: 3716.

[47] De Carolis B, Macchiarulo N, Palestra G, De Matteis AP, Lippolis A. FERMOUTH: facial emotion recognition from the MOUTH region. In: Foresti GL, Fusiello A, Hancock E (eds). Image Analysis and Processing—ICIAP 2023. Springer: Cham. 2023.

[48] Castellano G, De Carolis B, Macchiarulo N. Automatic facial emotion recognition at the COVID-19 pandemic time. Multimedia Tools and Applications. 2023; 82: 12751–12769.

[49] Samadiani N, Huang G, Cai B, Luo W, Chi CH, Xiang Y, et al. A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors. 2019; 19: 1863.

[50] Hajarolasvadi N, Demirel H. 3D CNN-based speech emotion recognition using K-means clustering and spectrograms. Entropy. 2019; 21: 479.

[51] Seyala N, Abdullah SN. Cluster analysis on longitudinal data of patients with kidney dialysis using a smoothing cubic B-spline model. International Journal of Mathematics, Statistics, and Computer Science. 2024; 2: 85–95.

[52] Kunz M, Lautenbacher S. The faces of pain: a cluster analysis of individual differences in facial activity patterns of pain. European Journal of Pain. 2014; 18: 813–823.

[53] Aung MS, Kaltwang S, Romera-Paredes B, Martinez B, Singh A, Cella M, et al. The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal EmoPain dataset. IEEE transactions on affective computing. 2015; 7: 435–451.

[54] Cascella M, Semeraro F, Montomoli J, Bellini V, Piazza O, Bignami E. The breakthrough of large language models release for medical applications: 1-year timeline and perspectives. Journal of Medical Systems. 2024; 48: 22.

[55] Jang EH, Rak B, Kim SH, Sohn JH. Emotion classification by machine learning algorithm using physiological signals, 2012 IACSIT Hong Kong Conferences. Singapore, 26–27th October 2012. IACSIT Press. 2012.