International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072
Integrating Multi-Modal Healthcare Data Using Hybrid Deep Learning Techniques
Dr. R.MURUGANANTHAM 1 , R. Jyoshita 2, K. Abhinav 3, N. Sai Priya 4 , M.Nihal 5
1Professor, Department of IT, TKR College of Engineering and Technology, Telangana, India 2,3,4,5B.Tech Students, Department of IT, TKR College of Engineering and Technology, Telangana, India
Abstract - The rapid advancement of smart healthcare technologies has resulted in the continuous generation of heterogeneous medical data from wearable sensors, IoT devices, electronic medical records, and medical imaging systems. Although deep learning has shown strong potential for healthcare prediction, its performanceishighlyaffectedby missing or incomplete data, which is common in real-world clinical environments. This paper proposes a Hybrid MultiModal Deep Learning Framework that integrates multiple single-modal neural networks through aCollaborativeConcat Layer (CCL) to achieve robust healthcare prediction under incompleteinputconditions.Theproposedframeworkemploys dedicated deep learning models for time-series, structured clinical data, and image modalities, and combines them intoa unified prediction architecture. To address missingfeatures,a correlation-driven Weight Matrix and collaborative node mechanism are used to estimate unavailable variables using learned relationships among health parameters. Unlike traditional fusion methods that fail when modalities are absent, the proposed CCLdynamicallyadaptstomissinginputs without requiring model retraining. Experimental evaluation shows that the framework achieves approximately 89–91% prediction accuracy even when key variables are missing, outperforming conventional deep learning fusion techniques. The proposed system is modular, scalable, and suitable for practical smart healthcare applications requiring reliable multi-modal integration and stable prediction performance.
Key Words: Multi-Modal Healthcare, Hybrid Deep Learning, Collaborative Concat Layer (CCL), Missing Data Handling, Correlation Analysis, Wearable Sensors, IoT Healthcare, Medical Prediction, Feature Fusion, Deep Neural Networks
1. INTRODUCTION
Modernhealthcareisrapidlyevolvingduetotheintegration of Artificial Intelligence (AI), Internet of Things (IoT), wearabledevices,andcloud-basedmedicalplatforms.These technologiesgeneratelarge-scaleandheterogeneoushealth datasuchasphysiologicalsignals,clinicalreports,laboratory values,lifestylerecords,andmedicalimages.Deeplearning models have demonstrated strong performance in healthcare applications including disease prediction, diagnosis, and clinical decision support [1], [2]. However, real-worldhealthcaredataisoftenincomplete,inconsistent, and collected from multiple independent sources, which
creates major challenges in developingreliable prediction systems[8],[15].
Multi-modalhealthcarelearninghasemergedasapowerful solution to combine different modalities into a unified prediction framework. By integrating structured clinical data, time-series signals, and imaging information, multimodaldeeplearningsystemsimprovepredictionaccuracy andclinicalreliabilitycomparedtosingle-modalmodels[1], [16].Atthesametime,hybriddeeplearningarchitectures thatcombineCNN,LSTM,andDNNmodelsenableeffective featureextractionacrossdiversedatatypes[4],[5].Despite theseadvances,missingdataremainsoneofthemostcritical issuesaffectingmulti-modalhealthcaresystems.
1.1 Background and Motivation
TheincreasingadoptionofwearablesensorsandIoT-based medical devices enables continuous monitoring of patient health, supporting early detection of diseases and personalized healthcare services [15]. At the same time, hospitals maintain Electronic Medical Records (EMR) containingpatienthistory,clinicalnotes,diagnosisreports, andlaboratoryresults.MedicalimagingmodalitiessuchasXrays, CT scans, and ultrasound images provide additional diagnostic evidence for clinicians [2], [11].Although these datasourcesoffercomprehensivehealthinsights,theyare oftencollectedasynchronouslyandstoredseparately.Many healthcare systems still process each modality independently, leading to reduced clinical effectiveness. Deeplearninghasproventobehighlyeffectiveinextracting meaningfulpatternsfromcomplexdata;forexample,CNNbased models have achieved near-human performance in medical image diagnosis tasks [2], [5]. Similarly, LSTM networks are widely used for time-series physiological signalsduetotheirabilitytolearntemporaldependencies [4], [8]. Therefore, integrating multiple modalities into a hybridlearningframeworkisessentialforbuildingrobust andaccuratehealthcarepredictionsystems[1],[16].
1.2 Challenges in Multi-Modal Healthcare Prediction
Eventhoughmulti-modaldeeplearningimprovesprediction capability, practical healthcare environments introduce several challenges. A major limitation is missing or incomplete data caused by sensor failure, irregular monitoring, device heterogeneity, and human errors in
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072
medical record entry [8]. Most traditional deep learning models assume complete input features and degrade significantlywhenoneormoremodalitiesaremissing. Another critical challenge is that conventional fusion methods rely on direct concatenation of features from multiple models. Such approaches fail to adapt when a modality becomes unavailable during inference. Missing variablescancauseunstablepredictionsandforceretraining oftheentiremulti-modalmodel,whichiscomputationally expensive and impractical for real-world healthcare deployments [7], [8]. Furthermore, most systems lack mechanismstoestimatemissingfeaturesintelligentlybased oncorrelationsamonghealthvariables,limitingprediction reliability.
2. PROPOSED SYSTEM
The Pproposed Hybrid Multi-Modal Deep Learning Framework is designed to integrate heterogeneous healthcare data and generate accurate predictions even whensomemodalitiesorfeaturesaremissing.Thesystem combinesmultipleindependentsingle-modaldeeplearning networks into a unified architecture using a Collaborative ContacLayer(CCL).Unlikeconventionalfusionmethods,the proposed framework dynamically adapts to incomplete inputs by estimating missing features using correlationbasedinference.Thisimprovespredictionstability,reduces dependency on complete datasets, and avoids frequent retraining,makingthesystemmoresuitableforreal-world smarthealthcareenvironments.
2.1 System Architecture
Theproposedsystemfollowsamodularhybridarchitecture thatsupportsmulti-modaldataintegrationandintelligent featurefusion.Thearchitectureincludesseparateprocessing pathsfortime-seriesdata,structuredclinicaldata,andimage data. Each modality is processed using a specialized deep learningmodelsuchasLSTMfortime-seriessignals,DNNfor numericalclinicalrecords,andCNNformedicalimages.The extracted features are then merged through the CollaborativeConcatLayer(CCL),whichperformsmissingdatadetection,collaborativefeatureestimation,andunified featurefusion.Thefusedrepresentationisfinallypassedinto thepredictionlayertogeneratethefinalhealthcareriskor diseaseclassificationoutput.
2.2 Multi-Modal Data Collection andPreprocessing
Thesystemcollectshealthcaredatafrommultiplereal-world sourcesincludingwearablesensors,IoTdevices,electronic medicalrecords,andmedicalimagingsystems.Sincethese sourcesgenerateheterogeneousdataformats,preprocessing is performed to ensure consistency and compatibility for deep learning. During preprocessing, raw data is cleaned, normalized, and transformed into structured
representations.Missingorincompletevaluesareidentified and marked for later estimation. Time-series signals are standardizedtomaintaintemporalconsistency,numerical featuresarescaledforstablelearning,andimageinputsare resizedandenhancedforCNN-basedfeatureextraction.This stageensuresthatthe input data isproperlypreparedfor independentsingle-modallearning.
2.3 Hybrid Multi-Modal Learning and CCL-Based Fusion
After preprocessing, each modality is passed through its corresponding neural network model to extract modalityspecificfeatures.Thesefeaturevectorsarenotmergedusing atraditionalconcatenationapproach.Instead,theproposed system introduces the Collaborative Concat Layer (CCL), which acts as an intelligent fusion module. The CCL uses correlation-based learning to build a Weight Matrix representingrelationshipsamonghealthvariables.Whenan inputfeatureormodalityismissing,thecollaborativenodes within the CCL estimate the missing feature values using correlated parameters and similar data patterns. This enables the hybrid model to generate stable predictions without retraining, even when key input variables are unavailable.Thefinalfusedfeaturerepresentationproduced bytheCCLisusedbythepredictionlayertooutputreliable healthcarepredictionresults.
3. IMPLEMENTATION DETAILS
The proposed Hybrid Multi-Modal Deep Learning Framework is implemented as a modular system that supports multi-source healthcare data processing, independentsingle-modallearning,andcollaborativefusion using the Collaborative Concat Layer (CCL). The implementation is designed to handle incomplete input conditions without retraining and to support scalable integrationofnewmodalities.Theoverallworkflowincludes data ingestion, preprocessing, model training, CCL-based fusion, and prediction through a web-based deployment interface.
3.1 System Architecture Implementation
The system is implemented using a layered hybrid architecturethatsupportsmulti-modaldataintegrationand missing-data handling. The architecture includes a data preprocessing pipeline, independent single-modal deep learning models, a correlation-based Weight Matrix generator,andaCollaborativeConcatLayer(CCL)forfusion andmissingfeatureestimation.Eachmodalityisprocessed separately,andtheextractedfeaturevectorsarecombined throughtheCCLtogenerateaunifiedfeaturerepresentation for final prediction. This modular structure enables scalability, improves reusability of trained models, and
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072
ensures stable prediction performance even when one or moreinputsaremissing.
Fig – 1 : System Architecture of Proposed Hybrid Multi-Modal Healthcare Prediction Framework
3.2 Model Training and Feature Extraction
Theimplementationincludesindependenttrainingforeach modality-specificmodel.Structuredclinicaldatasetssuchas heart disease and diabetes records are processed using preprocessingstepsincludingcleaning,normalization,label encoding, and feature selection. After preprocessing, multiplemachinelearninganddeeplearningclassifiersare trainedandevaluatedtoidentifythebest-performingmodel foreachdiseasecategory.Featureextractionisperformed through the trained models, and the most relevant parametersareselectedforpredictiontoreducecomplexity andimproveaccuracy.Time-seriesandimagemodalitiesare supported through LSTM/RNN and CNN-based feature extraction modules, enabling the framework to learn temporalandspatialpatternsfromphysiologicalsignalsand medicalimaginginputs.
3.3 Collaborative Contac Layer Integration and Deployment
Aftertrainingtheindividualmodels,thesystemintegrates them into a unified prediction framework using the Collaborative Concat Layer (CCL). The CCL acts as an intelligent fusion layer that detects missing features at runtimeandestimatesabsentvaluesusingcorrelation-based inference. A Weight Matrix is generated from dataset correlations and is used by collaborative nodes to infer missing inputs based on related health parameters. This allows the system to maintain stable prediction accuracy withoutretraining,evenwhendataisincomplete.Thefinal hybridmodelisdeployedusingaFlask-basedwebinterface, allowinguserstoinputhealthparametersthroughauserfriendlyformorAPIrequestandreceivereal-timedisease predictionoutputs.
4. RESULTS AND PERFORMANCE ANALYSIS
The proposed Hybrid Multi-Modal Deep Learning Framework was evaluated using real-world structured healthcare datasets containing multiple physiological and clinicalattributes.Thedatasetincludespatient-levelhealth parameters such as age, annual health risk factors, blood pressure values, body measurements, and laboratory test values including haemoglobin, platelet count, and other blood-related indicators. These features represent heterogeneousclinicalvariablesthatarecommonlyusedfor diseaseriskpredictionandhealthcareclassificationtasks.
During experimentation,the datasetwaspre-processed to remove noise and ensure consistency across variables. Missing and incomplete values were identified, and the proposedCollaborativeConcatLayer(CCL)mechanismwas applied to estimate unavailable inputs dynamically using correlation-based inference. This allowed the system to performpredictions evenwhencertainhealthparameters were missing, which reflects realistic smart healthcare environments.
The system performance was measured using standard evaluationmetricssuchasaccuracy,precision,recall,andF1score. The proposed model achieved stable performance across multiple prediction tasks, demonstrating strong robustnesscomparedtoconventionaldeeplearningfusion methods.Theexperimentalresultsindicatethatthehybrid architecturemaintainshighpredictionaccuracyevenunder missing-data conditions. In particular, the framework achievedapproximately89–91%accuracywhenkeyinput variableswereunavailable,provingtheeffectivenessofthe correlation-drivenWeightMatrixandcollaborativefeature estimationstrategy.
Additionally,themodulardesignoftheframeworkreduced theneedforfrequentretraining.Sinceeachmodality-specific model is trained independently and later fused using the CCL,thesystemsupportsscalabilityandefficientintegration of new data types. Overall, the results confirm that the proposed framework improves prediction reliability, stability,andreal-worldusabilityformulti-modalhealthcare analytics.
5. CONCLUSION
ThispaperpresentedaHybridMulti-ModalDeepLearning Framework for intelligent healthcare prediction using heterogeneous medical data collected from wearable sensors,IoTdevices,electronicmedicalrecords,andmedical imaging systems. The proposed approach effectively addresses one of the major limitations of traditional deep learningmodels,whichisperformancedegradationcaused bymissingorincompleteinputdata.
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072
The system integrates multiple single-modal learning modelsintoa unifiedarchitecturethrougha Collaborative Concat Layer (CCL). Unlike conventional feature fusion methods,theproposedCCLdynamicallyestimatesmissing variables using correlation-based learning and a Weight Matrix, enabling stable and reliable predictions without requiringretraining.Experimentalevaluationconfirmsthat the framework achieves robust prediction performance, maintainingapproximately89–91%accuracyevenwhenkey healthparametersaremissing.
Overall, the proposed framework provides a scalable, modular,andefficientsolutionformodernsmarthealthcare analytics.Itimprovespredictionstability,supportsflexible integrationofnewdatamodalities,andenhancesreliability for real-world clinical decisionsupport and remotehealth monitoringapplications.
6. FUTURE WORK
AlthoughtheproposedHybridMulti-ModalDeepLearning Frameworkachievedstrongpredictionperformanceunder missing-data conditions, several enhancements can be explored in future work. The system can be extended to supportreal-timestreaminghealthcaredatafromwearable andIoTdevices,enablingcontinuousmonitoringandearly disease detection. Future improvements may include integrating explainable AI (XAI) techniques to improve interpretabilityandincreaseclinicaltrustintheprediction outcomes. The framework can also be enhanced using federatedlearningtopreservepatientprivacybyenabling model training across multiple hospitals or institutions without sharing sensitive data. In addition, the proposed modelcanbeexpandedtosupportmorediseasecategories, additional medical imaging modalities, and larger multipopulationdatasetstoimprovegeneralizationandfairness. Theseenhancementswillfurtherstrengthenthescalability, reliability, and real-world applicability of the proposed intelligenthealthcarepredictionsystem.
REFERENCES
[
1] Y. Li, J. Huang, L. Zhou, and X. Li, “Multi-modal deep learning for healthcare: A review,” IEEE Journal of BiomedicalandHealthInformatics,vol.26,no.4,pp.1450–1464,2022.
[2]A.Esteva,B.Kuprel,R.A.Novoa,J.Ko,S.M.Swetter,H.M. Blau,andS.Thrun,“Dermatologist-levelclassificationofskin cancer with deep neural networks,” Nature, vol. 542, no. 7639,pp.115–118,2017.
[3] G. E. Hinton, A. Krizhevsky, and S. D. Wang, “Transformingauto-encoders,”inProc.Int.Conf.onArtificial NeuralNetworks,2011,pp.44–51.
[4] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”NeuralComputation,vol.9,no.8,pp.1735–1780, 1997.
[5]K.He,X.Zhang,S.Ren,andJ.Sun,“Deepresiduallearning for image recognition,” in Proc. IEEE Conf. on Computer VisionandPatternRecognition(CVPR),2016,pp.770–778.
[6]D.P.KingmaandJ.Ba,“Adam:Amethodforstochastic optimization,” in Proc. Int. Conf. on Learning Representations(ICLR),2015.
[7]J.Yoon,J.Jordon,andM.vanderSchaar,“GAIN:Missing dataimputationusinggenerativeadversarialnets,”inProc. Int.Conf.onMachineLearning(ICML),2018,pp.5689–5698.
[8] Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu, “Recurrentneuralnetworksformultivariatetimeserieswith missing values,” Scientific Reports, vol. 8, no. 1, pp. 1–12, 2018.
[9]E.Choi,M.T.Bahadori,J.Sun,J.Kulas,A.Schuetz,andW. F.Stewart,“RETAIN:Aninterpretablepredictivemodelfor healthcareusingreversetimeattentionmechanism,”inProc. Advances in Neural Information Processing Systems (NeurIPS),2016,pp.3504–3512.
[10] A. Vaswani et al., “Attention is all you need,” in Proc. Advances in Neural Information Processing Systems (NeurIPS),2017,pp.5998–6008.