International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 12 Issue: 10 | Oct 2025 www.irjet.net p-ISSN: 2395-0072
Multimodal deep learning system for Diagnosis of Autoimmune Hepatitis Using Clinical and Imaging Data
Akshay Bankar1 , Harshwardhani Kewate1 , Sahil Tandekar1, Shejal Meshram1
1,2,3,4Dept. of CSE, Gurunanak Institute of Engineering and Technology, Nagpur, Maharashtra, India ***
Abstract - Autoimmune Hepatitis (AIH) is a chronic inflammatory disorder of the liver, and its diagnosis requires an intricate integration of clinical observations, serological findings, and histopathological evidence. The multifactorial nature of this process often poses significant challenges, demanding both time and specialized medical expertise.Tomitigatethesechallenges,thisstudyintroduces aproof-of-conceptmultimodalwebapplicationdevelopedto support hepatologists in diagnosing AIH. The proposed system seamlessly combines unstructured clinical text data with liver biopsy imagery and employs an advanced multimodal large language model (MLLM) for comprehensive analytical processing. Following data synthesis, the application produces two distinct outputs: a structured diagnostic support report for clinicians and an accessible educational handout for patients. The diagnostic report provides a probabilistic estimation of AIH, a modelderived confidence score, key clinical insights, differential diagnostic considerations, and preliminary management suggestions. In parallel, the patient handout translates complex medical information into comprehensible language to enhance health literacy and engagement. This research highlights the transformative potential of generative artificialintelligenceinunifyingheterogeneous clinical data sources, thereby improving diagnostic precision, optimizing clinical decision-making, and promoting patient-centered communicationinthecontextofcomplexhepaticdisorders.
Key Words: Autoimmune Hepatitis, Multimodal AI, Large Language Models, Diagnostic Support Systems, Medical Informatics, Histopathology
1.INTRODUCTION
Autoimmune Hepatitis (AIH) is a chronic immunemediated liver disorder characterized by hepatocellular inflammation, interface hepatitis on histopathological examination, and the presence of circulating autoantibodies [1]. Diagnosing AIH poses substantial challengesduetoitsheterogeneousclinicalmanifestations andthefrequentoverlapofitsfeatureswithotherhepatic conditions, including viral hepatitis, drug-induced liver injury, and primary biliary cholangitis [2]. Accurate diagnosis requires a holistic evaluation that integrates clinical history, laboratory parameters such as elevated aminotransferase and immunoglobulin G (IgG) levels and, most critically, expert interpretation of liver biopsy specimens. Although established scoring systems like the
revised original and simplified AIH criteria provide structured diagnostic guidance, their practical use can be laborious, and histopathological interpretation remains inherentlysubjective,reliantonthepathologist’sexpertise [4].
The ongoing digitization of medical data and pathology slides has created new avenues for computational innovations that can complement clinical expertise. Recent progress in artificial intelligence, particularly the rise of large multimodal language models (MLLMs), has enabled systems capable of integrating and reasoning across diverse data modalities such as textual and visual inputs [5]. These models can correlate semantic content from clinical documentation with morphological patterns in medical images, effectively replicating certain aspects ofclinicalreasoning.
Inthis context, the present studyproposesa novel proofof-concept web-based system that utilizes an MLLM to assist in the diagnostic workflow of AIH. The platform accepts clinical information either manually entered or extracted from uploaded records alongside corresponding liver biopsy images. It performs a unified analysis of both data types to generate two outputs: a structured diagnostic support report for clinicians and a patient-oriented educational summary. Rather than substitutingclinicaljudgment,thesystemaimstoenhance it, offering a data-driven, multimodal synthesis of key diagnostic indicators to support more accurate, efficient, and informed decision-making in the evaluation of autoimmuneliverdisease.
2. Related Work
The integration of artificial intelligence into hepatology and pathology has advanced rapidly in recent years, with early research primarily centered on developing machine learning algorithms capable of predicting liver fibrosis stages from clinical variables or classifying histopathological patterns from whole-slide images (WSIs) [6]. Convolutional Neural Networks (CNNs), in particular, have achieved near-expert accuracy in identifying histological features such as steatosis, inflammation,andhepatocellularballooningwithinbiopsy specimens [7]. Despite these successes, most existing systems remain unimodal, focusing exclusively on either structured clinical data or imaging data, thereby limiting
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 12 Issue: 10 | Oct 2025 www.irjet.net p-ISSN: 2395-0072
their capacity to capture the complex interplay between diversediagnosticmodalities.
The emergence of multimodal data fusion represents a critical evolution in medical artificial intelligence. Integrating imaging data with electronic health record (EHR) information has been shown to enhance both diagnostic and prognostic accuracy across multiple disease domains [8]. For example, multimodal frameworks that combine histopathological imagery with genomic profiles have demonstrated improved cancer subtype classification. Nonetheless, many of these approachesrelyheavilyonstructureddatasetsandlaborintensive feature engineering, constraining their adaptability and scalability in real-world clinical environments.
The introduction of large language models (LLMs) and their multimodal variants (MLLMs) signifies a fundamental shift in this landscape. These models are capable of understanding and reasoning over unstructured text and visual information simultaneously, enabling correlation between narrative clinical data and corresponding medical images without requiring manually curated features or labels [9]. Although the deployment of such models in hepatology and gastroenterology remains in its early stages, initial research indicates promising applications, including automated radiology report generation and clinical question answering based on imaging content [10]. Buildingonthisfoundation,thepresentworkintroducesa domain-specific application focused on autoimmune hepatitis diagnostics, employing an end-to-end multimodal analytical framework that not only provides structured diagnostic support for clinicians but also produces patient-oriented educational summaries to facilitateinformedhealthcareengagement.
3. SYSTEM ARCHITECTURE AND METHODOLOGY
The proposed system follows a client–server architecture implemented as a web-based application, emphasizing modularity, usability, and secure data management throughout the diagnostic workflow. The design ensures thatuserinteraction,dataprocessing,andAIinferenceare handled through distinct layers, thereby maintaining scalabilityanddataintegrity.
Architectureofthemultimodaldiagnosticsystem integratingclinicalandimagingdata.
Data Input: Usersinteractwithawebinterfacetoprovide relevant clinical information. This can be achieved either by completing a structured form or by uploading existing clinicaldocumentationsuchasconsultationsummariesor laboratory reports in PDF format. In addition, a representativeliverbiopsyimageinstandardformats(JPG orPNG)isuploadedtoaccompanytheclinicaldata.
Backend Processing: Upon submission, the input data is securely transmitted to the backend via encrypted communication channels. The backend server includes a dedicated API endpoint capable of handling multipart form data, effectively managing both textual and imagebasedinputs.
Data Preprocessing: For uploaded PDF files, textual information is extracted using automated text-parsing algorithms. Both clinical text and image data are subsequentlypreprocessedandencodedinbase64format, producing a unified JSON payload optimized for multimodalmodelingestion.
AI Inference: The encoded payload is forwarded to a multimodal large language model (MLLM) that performs integratedreasoningoverbothdatamodalities.Themodel identifies and interprets key diagnostic indicators associated with autoimmune hepatitis, generating structuredanalyticalinsights.
4. Results
The developed multimodal diagnostic system was evaluated through a series of simulated clinical scenarios representing diverse presentations of autoimmune hepatitis (AIH). The system demonstrated robust
Fig-1:
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 12 Issue: 10 | Oct 2025 www.irjet.net p-ISSN: 2395-0072
performance in integrating textual clinical data and liver biopsy images, producing coherent and contextually relevant diagnostic summaries. The generated clinicianoriented reports consistently included accurate likelihood estimations of AIH, well-structured differential diagnoses, and concise interpretative remarks reflecting clinically meaningfulreasoning.Similarly,thepatient-facingreports effectivelytranslatedtechnicalterminologyintoaccessible language, improving interpretability and educational value. Preliminary usability testing with a small group of clinicians and students indicated that the web interface was intuitive and required minimal training. Processing time per case averaged under one minute, including file upload, AI inference, and PDF generation, confirming the system’s suitability for near-real-time clinical support. These results collectively validate the technical feasibility andpractical utilityofa multimodal largelanguagemodel (MLLM)-basedsysteminhepatologydiagnostics.
Quantitative Evaluation
To assess diagnostic reliability, model outputs were compared with expert annotations across 50 test cases. The multimodal system achieved an overall diagnostic accuracy of 0.88, with a precision of 0.91, recall of 0.85, andF1-scoreof0.88foridentifyingAutoimmuneHepatitis versus other hepatic disorders (Chart-1). Average inference and report-generation time per case was 57 seconds,confirmingnearreal-timefeasibility.
Chart-1: EvaluationMetrics
5. CONCLUSIONS
This study presented a proof-of-concept multimodal artificial intelligence system designed to support the diagnosis of Autoimmune Hepatitis (AIH) by integrating clinical text data with histopathological imagery. The system successfully demonstrates how generative AI can synthesize unstructured medical information into coherent,structuredoutputsthatservebothcliniciansand patients. By automating the generation of diagnostic supportreportsandeducationalhandouts,theframework not only streamlines the diagnostic workflow but also enhances interpretability, accessibility, and communication within clinical settings. The results validate the feasibility of leveraging multimodal large
language models (MLLMs) as effective decision-support tools capable of augmenting human expertise in hepatology.
6. REFERENCES
[1] European Association for the Study of the Liver. (2015). EASL Clinical Practice Guidelines: Autoimmune hepatitis. Journal of Hepatology, 63(4), 971-1004.
[2] Manns,M.P.,etal.(2010).Diagnosisandmanagement of autoimmune hepatitis. Hepatology, 51(6), 21932213.
[3] Hennes, E. M., et al. (2008). Simplified criteria for the diagnosisofautoimmunehepatitis. Hepatology,48(1), 169-176.
[4] Alvarez, F., et al. (1999). International Autoimmune Hepatitis Group Report: review of criteria for diagnosis of autoimmune hepatitis. Journal of Hepatology,31(5),929-938.
[5] Moor, M., et al. (2023). Foundation models for generalist medical artificial intelligence. Nature, 616(7956),259-265.
[6] Litjens, G., et al. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60-88.
[7] Taylor-Weiner,A.,etal.(2021).Adeeplearning-based system for identifying and grading liver steatosis on histology slides. The American Journal of Pathology, 191(9),1596-1607.
[8] Spathis, D., & Kleftakis, S. (2022). Multimodal Medical AI. AnnualReviewofBiomedicalDataScience,5,25-45.