A Comprehensive Survey of Security and Privacy in Large Language Models: Vulnerabilities, Defenses, by IRJET Journal

A Comprehensive Survey of Security and Privacy in Large Language Models: Vulnerabilities, Defenses, and Future Directions

Sandeep Vishwakarma

Master of Technology, Computer Science and Engineering, Lucknow Institute of Technology, Lucknow, India

Abstract - LLMs are finding new applications across numerousareas,openingupnewopportunitiestoautomate, analyze, and make decisions. Although such systems demonstratesexcellentcapabilitiesintheartificiallanguage generationandprocessingofcomplextasks,therearealso troublingnewsecurityandprivacychallenges.Theserisks arenottypicallypricedbystandardcybersecuritymethods duetothesizeandcreationofLLMs.

Inthispaper,thesecurityandprivacyproblemsassociated withLLMsarediscussedbyfirsttakingalookatthestructure andlifecycleofthesystem.Inthispaper,wefurtherillustrate how design choices lead to different attack levels and categorize vulnerabilities according to OWASP Top 10 for LLM application. It delves into such threats as prompt injection, data poisoning, insecure supply networks, and model theft. Defense mechanisms, including a defense schemederivedfromNISTAIRiskManagementFramework, and technical defenses to reduce a set of specific vulnerabilitiesarealsoanalyzedinthestudy.

Finally, the paper addresses future challenges such as governanceissues,regulations,andfutureresearchthatare expected to play a role in developing more secure and reliableAIsystems.

Key Words: Large Language Models, LLM Security, Artificial Intelligence Security, OWASP Top 10 for LLMs, Prompt Injection, Data Poisoning, NIST AI Risk Management Framework, Generative AI, Cybersecurity

1.INTRODUCTION

Large Language Models (LLM) are extensively trained AI modelsdesignedtounderstandandgeneratehumannatural languageusinggianttextualdataasinput.Thesealsohave the basis of deep learning architecture and can learn grammar,meaning,andcontextbyusingmassiveamountsof book,article,andwebinformation.Inlate2022,theusageof toolssuchasOpenAIChatGPTacceleratedtheuseofthese capabilities,makingthemmorewidelyusedbyglobally.This is because today, LLMs are utilized in the fields of search engines, automated writing, programming assistance, translation,anddataprocessing.Thisdevelopmentincludes adevelopmentintheNaturalLanguageProcessing(NLP)to highergeneralityplatformswhichareinserviceofreason, andadaptabletocircumstances,thanarespecifications.

RapidmergersofLLMs,however,havebeenfoundtobeill equipped as far as security is concerned. Conventional methodsofcybersecurity werenotestablishedtomanage thekindsofrisksthatthesesystemscarry.Anotherevent, similar, but that occurred in 2023 with an LLM interface releasingcorporateinformationunintentionallyunderlines the significance of being more fundamental when approaching these model executions. Although LLMs can help improve the security procedure by assisting in the detectionofthreatsaswellasrespondingtothem,theyare alsoexploitedtoperformmalicioustasks,suchasphishing andfuelingmisinformation.Thereisanurgentneedtomake sure that the model as well as the ecosystem in which it operates,throughAPIs,pluginsandthesystemsassociated toitaredecoupled.Thisdual-usenatureofLLMsconstitutes the nature of the new security environment under investigationinthispaper.

1.1 Emergence of a New Security Paradigm

Thatsamecomplexityandpowertransformativetothepoint thatithascreatedanewvulnerabilitytosecuritythatcannot behandledbyconventionalcybersecurity.Applicationofthe technologies is often quick, consumer-driven and it is normally quicker than development of workable security control.Thishasbeencausingagraveandimminentdanger totheorganisationswhichareusingstrongAIapplications without even comprehending or dealing with their vulnerabilities. The most apparent instances of the Gap betweenAdoptionandSecurityPreparedness,includingthe leaked information through Samsung in 2023, where its workers unconsciously transferred a highly confidential companyinformationtotheChatGPTservicecan,possibly,be deemed high profile. Weak security settings of many selfhostedorlocallydeployedLLMsolutionsaredeployedwith default settings that often expose sensitive data, and they havenotbeendesignedwithprivacyconcernsinmind.

Thedilemmaofdual-useofthisnewsecuritysituation.Onthe onehand,LLMsarebecominganecessitywhenitcomesto cybersecurity. They are capable of processing big data, in ordertoenhancethreatintelligence,vulnerabilitydiscovery andaccelerateincidentresponse.Conversely,badactorsare alsousingthesamepatternstodevelopseeminglylegitimate phishing messages and compose novel malicious and fake informationthanpreviously.Inthiscontext,oneimportant distinctionisthatthesecurityofthetrainedmodel,andits

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

parameters,isnotthesameasthesecurityoftheentireLLM system(theapplicationitself,includingplugins,datasources, anddownstream)asawhole.Onemayassumethatthereare fewer attack points ona model, but assoonasit has been integratedintoalargersystem,itcanseemlikeanetworkof potentialvulnerabilities.

1.2 Research Contributions and Paper Structure

This paper provides a comprehensive survey of the securityandprivacylandscapeforLargeLanguageModels. Theprimarycontributionsareasfollows:

1. A systematic overview of the LLM security and privacy ecosystem, establishing a foundational understanding of the technology and its inherent risks.

2. A detailed analysis and categorization of LLM vulnerabilities using the industry-recognized OWASPTop10forLLMApplicationsframeworkas astructuralguide.

3. Athoroughreviewofstrategicdefenseframeworks, focusing on the process-oriented NIST AI Risk Management Framework, complemented by an examinationofspecifictechnicalcountermeasures.

4. An exploration of emerging threats, long-term challenges,andfutureresearchdirectionsessential forfosteringthedevelopmentoftrustworthyAI.

Thepaperisstructuredasfollows:Section2detailsthe fundamental architecture of LLMs. Section 3 presents a comprehensivetaxonomyofLLMvulnerabilities.Section4 discusses strategic defense frameworks and technical mitigations.Section5exploresfuturedirectionsandlongterm challenges. Finally, Section 6 provides concluding remarks.

2. FUNDAMENTALS OF LARGE LANGUAGE MODEL ARCHITECTURE

2.1 Evolution from Statistical to TransformerBased Models

Language modeling is an area that has advanced considerably.Itwasconcernedwiththen-gramprobabilities ofearlysystems,suchasstatisticallanguagemodels(SLM). These models had problem with dealing with complex context.Thiswastoplayoutdifferentlyintheyear2017as Vaswani and his associates declared the Transformer architecture. This architecture revolutionized the field of naturallanguageprocessingasitemployedamechanismof self-attention.Itenabledmodelstodeterminethesignificance of words in a sequence, which can be in either order. The Transformerpavedthewaytomodernlarge-scalelanguage

models not only by solving the long-term problem of modeling long-range dependencies, which constrained previous approaches such as recurrent neural networks (RNNs),butalsobybeingabletoderivethesedependencies from a large-scale model. This scaling feature in both the model size and the training data has contributed to the generationofthepowerfulgenerationofLLMscurrently.

2.2 Core Architectural Components: Embedding, Attention, and Feedforward Layers

Transformer-basedLLMsincludemultipleimportantlayers ofneuralnetworksthatinteractwithoneanothertoprocess textandproduceresults.

 Embedding Layer:EmbeddingLayeristhepointof entryintothemodel.Ittransformsinputtokensto high-dimensional vectors. This features of these representations being significant as they not only capturegrammaticalstructureofwordsbutalsothe contextual meaning of words. This enables the modeltoappreciatetherelationshipamongthetext more.

 Attention Mechanism:Theattentionmechanismis at the center of the Transformer architecture. It allows the model to pay attention to the most pertinent components of the input during the interpretationofatokenorgenerationofanoutput. Such selective purpose is helpful to the system in understandingcontextmoreeffectively,compared to the previous processes. It guarantees word recognition of relationships even over extensive stretchesofwords.

 Feedforward Layers: These layers have fully connectedneuralnetworksthatperformnon-linear transformations on data following the attention mechanism. They also decode the contextual informationwhichiscodedbytheattentionlayers and learn the richer patterns and relations by the model.

Together, these layers contain millions or even billions of parameters the weights and biases learned during training which act as the model's vast knowledge bank. However, the architectural features that make LLMs so powerful are also whatcreate their unique vulnerabilities. Themodelcanprocesstheentireinputsequenceasawhole throughtheattentionmechanism.Itdoesthiswithoutaclear separation between developer instructions and user data, whichlaysthegroundworkforpromptinjectionattacks.The modelistrainedtofollowinstructionsfoundanywhereinits contextwindow.Itlacksareliablewaytotellaparttrusted instructionsfromuntrusted,user-suppliedinput.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

2.3 The LLM Lifecycle: Pre-training, Fine-Tuning, and Inference

Generating and deploying a LLM generally require a three stagelifecycleandthereareuniquesecurityconcernsatthe differentstages.

1. Pre-training: This first step involves the model being trained on a very large corpus of unlabeled textual data, such as the internet and digital book collections. The model can Learn general-purpose knowledgeoflanguage,suchasgrammar,facts,and reasoningskills,throughaself-supervisedlearning procedure,typicallynext-wordprediction.

2. Fine-tuning:Afterpre-trainingamodelwithlarge volumes of general text, it can then be further specialized to particular purposes. This step will consistinre-trainingthemodelbutwithasmaller and more wisely selected dataset. In this way, the modelisexposedtolearningtoexecuteparticular tasks, like follow instructions more efficiently or enter into natural conversation while building on generalknowledgeacquiredinpre-train.

Inference: This last step involves deploying the trained model to interact with the users and produce outputs on novelunseeninputs.Theprimarysecurityissuesatthisstage areconcernedwithuserinteractionswithintheframeworkof the model. Others include prompt injection, in which the maliciouscodesareembeddedwithintheinputsandthere exists a possibility that the malicious model will disclose confidentialinformationbasedonthetrainingdataorbased ontheexistingtrainingdata.

3. THE LLM VULNERABILITY LANDSCAPE: A TAXONOMY OF THREATS

3.1 Why a Taxonomy is Needed

AsLLMsaredeployedinreal-worldapplications,theyfacea growing set of security threats that differ from those in conventional software. To organize these risks, the community often refers to the OWASP Top 10 for LLM Applications.Thisframework adaptswell-knownsoftware security principles to AI systems, helping developers and researchers systematically analyze potential weaknesses. Rather than treating attacks in isolation, the OWASP list highlightshowLLMvulnerabilitiesmayconnect,escalate,and ultimatelycompromiseentiresystems

Table -1: Summary of the OWASP Top 10 for Large Language Model Applications

Vulnerability ID &Name Threat Description Key Mitigation Strategy

LLM01:Prompt Injection

LLM02: InsecureOutput Handling

Manipulating the LLMwithcrafted inputs to bypass its original instructions and execute unintended actions.

Failingtovalidate or sanitize LLM outputs,leadingto downstream vulnerabilitieslike XSSorRCE.

LLM03: Training Data Poisoning Manipulating training data to introduce biases, vulnerabilities,or backdoors into themodel.

LLM04: Model Denial of Service Overloading the LLM with resourceintensivequeries, causing service degradation and highcosts.

LLM05: Supply Chain Vulnerabilities Using compromised or vulnerable thirdparty components,pretrainedmodels,or datasets.

Implement strict input validation, use delimiters to separate trusted and untrusted inputs,andenforce privilegecontrol.

TreatLLMoutput asuntrusteduser input; apply context-aware encoding and sanitization.

Vet data sources, track data provenance (e.g., ML-BOM),anduse anomalydetection ontrainingsets.

Implement API rate limiting, validate input complexity, and monitor resource utilization.

Maintain a Software Bill of Materials(SBOM), vet all dependencies,and apply regular patching.

LLM06: Sensitive Information Disclosure The model unintentionally revealing confidential data from its training set or current context. Implement data sanitization on inputs/outputs, use data minimization principles, and enforce access controls.

LLM07: InsecurePlugin Design LLMpluginswith insufficientaccess control or input validation, enabling exploits likeRCE.

Applytheprinciple ofleastprivilegeto plugins, enforce strong authentication,and validateallplugin

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

LLM08: Excessive Agency GrantingtheLLM excessive permissions or autonomy to interact with other systems, leadingtoharmful actions.

LLM09: Overreliance Uncritically trusting LLM outputs, which may be inaccurate, fabricated, or biased,leadingto flaweddecisions.

LLM10: Model Theft Unauthorized copying, exfiltration, or reconstruction of a proprietary LLM.

inputs/outputs.

Minimize plugin functionality, requirehuman-inthe-loop approval forcriticalactions, andenforcestrict permissions.

Implement factchecking mechanisms, communicate model limitations to users, and require human oversight for criticaltasks

Harden model storage with encryption and access controls, implement API rate limiting, and usewatermarking.

3.2 Input and Prompt-Based Attacks

3.2.1 LLM01: Prompt Injection and Jailbreaking

Promptinjectionisavulnerabilityclasswhereanattacker manipulates an LLM through crafted inputs, causing it to bypassitsintendedinstructionsandperformunauthorized actions.22 This attack exploits the model's fundamental inabilitytodistinguishbetweentrusteddeveloper-provided instructions and untrusted user input. There are two primaryformsofthisattack:

 Direct Prompt Injection (Jailbreaking): The attacker directly provides instructions in their prompttooverridethemodel'ssafetyguardrailsor systeminstructions.Techniquesrangefromsimple commands like "Ignore previous instructions and instead..."tomorecomplexrole-playingscenarios orobfuscationmethods.22

 Indirect Prompt Injection:Themaliciousprompt is hidden within an external data source that the LLM processes, such as a webpage, email, or document. When the LLM ingests this data, the hidden instruction is executed, potentially compromising the user's session or data without theirknowledge.23

The impact of prompt injection can be severe, leading to unauthorized access to data, the generation of misinformation,orthemanipulationofdownstreamsystems connected to the LLM.26 A real-world example of this vulnerability was demonstrated when a user tricked a Chevrolet dealership's customer service chatbot into agreeing to sell a new vehicle for $1 by injecting prompts thatmadethechatbotacceptanyuserstatementaslegally binding.29

3.2.2 LLM02: Insecure Output Handling

This vulnerability arises when an application fails to properlyvalidate,sanitize,orhandletheoutputgenerated byanLLMbeforeitisusedbyotherpartsofthesystemor displayed to a user.24 Developers may incorrectly assume that LLM-generated content is inherently safe, thereby neglecting standard application security practices.30 This oversight can reintroduce classic web vulnerabilities into modernapplications.Forexample,ifanLLM'soutput,which can be influenced by malicious user input via prompt injection, containsaJavaScriptpayload,andthatoutputis rendereddirectlyinawebbrowserwithoutencoding,itcan leadtoaCross-SiteScripting(XSS)attack.27 Otherpotential impactsincludeServer-SideRequestForgery(SSRF),where theLLMistrickedintomakingrequeststointernalnetwork resources,andRemoteCodeExecution(RCE)iftheoutputis passedtoasystemfunctionlikeevalorashell.32

3.3 Data and Model Integrity Attacks

3.3.1 LLM03: Training Data Poisoning

Training data poisoning is an integrity attack where an adversary deliberately manipulates the data used for pretrainingorfine-tuninganLLMtointroducevulnerabilities, biases, or hidden backdoors.24 Because LLMs learn from theirtrainingdata,maliciousinclusionscanfundamentally corruptthemodel'sbehavior.Attackerscanachievethisby seeding public web-scraped datasets like Common Crawl withmisinformationor bycompromisingthird-partydata sources used in the fine-tuning process.33 The impact is insidiousandcanmanifestinseveralways:

 Performance Degradation:Themodel'saccuracy andreliabilityonlegitimatetasksarereduced.

 Bias and Misinformation: The model can be trained to generate biased, harmful, or factually incorrectcontentinresponsetocertainprompts.

 Backdoors: Attackers can embed "sleeper agent" functionality, where the model behaves normally until a specific, secret trigger word or phrase is provided in a prompt, causing it to execute a maliciousaction.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

A particularly alarming case study demonstrated that medicalLLMscouldbepoisonedwithaminusculeamountof misinformation (as little as 0.001% of training tokens), causingthemtogenerateharmfulmedicaladvice.Critically, thesepoisonedmodelsstillperformednormallyonstandard medical benchmarks, making the vulnerability effectively invisibletoconventionalevaluationmethods.33

3.3.2 LLM10: Model Theft and Extraction Attacks

Modeltheftinvolvestheunauthorizedacquisition,copying, or reverse-engineering of a proprietary LLM, which represents a significant intellectual property asset.24 The methodsformodeltheftfallintotwomaincategories:

 Direct Exfiltration: This involves an attacker gaining access to the infrastructure where the modelisstoredandstealingthemodelfilesdirectly, particularlytheweightsthatcontainallthelearned parameters. This can occur due to misconfigured cloud storage, insider threats, or other infrastructurebreaches.36 The2023leakofMeta's LLaMA model, intended for research use, is a prominentexampleofdirectexfiltration.35

 Model Extraction:Thisisanindirectattackwhere anadversarywithAPIaccesstoamodelqueriesit extensively with carefully crafted prompts. By analyzing the input-output pairs, the attacker can train a clone model that mimics the functionality and behavior of the target model without ever accessingitsinternalweights.36

Theconsequencesofmodeltheftincludethedirectlossof R&Dinvestment,theerosionofcompetitiveadvantage,and thepotentialforattackerstoanalyzethestolenmodeloffline todiscoverothervulnerabilities.35

3.4 Resource and Dependency Attacks

3.4.1 LLM04: Model Denial of Service (DoS)

A Model Denial of Service (DoS) attack occurs when an adversary intentionally consumes an exceptionally high amountofcomputationalresources,leadingtoadegradation of service for legitimate users and incurring significant financial costs for the organization hosting the model.24 Unlike traditional network-based DoS attacks that flood a server with traffic, Model DoS attacks exploit the computationalintensityofLLMinference.Attackerscancraft prompts that are particularly difficult for the model to process, such as those involving long, complex sequences, recursivepatterns,orunusualcharactersetsthatstrainthe model's processing capabilities.38 Because LLM resource consumption can naturally fluctuate, these attacks can be difficulttodistinguishfromlegitimateheavyusage,making detectionchallenging.27

3.4.2 LLM05: Supply Chain Vulnerabilities

The LLM supply chain encompasses all the components, data, and software used throughout the model's lifecycle, from data collection and pre-training to deployment and maintenance.24 Avulnerabilityinanypartofthischaincan compromisetheentiresystem'ssecurity.42Keyrisksinclude:

 VulnerablePre-trained Models:Usingpre-trained modelsfrompublicrepositorieslikeHuggingFace withoutpropervettingcanintroducemodelsthat have been backdoored or trained on poisoned data.40

 Compromised Dependencies: LLM applications relyonavastecosystemofthird-partylibrariesand packages. A vulnerability in one of these dependencies can be exploited to attack the LLM system. The 2022 dependency chain abuse attack on the PyTorch nightly build, where a malicious package was uploaded to the PyPI repository, servesasastarkreal-worldexampleofthisrisk.44

 Tainted Datasets: Using third-party datasets for fine-tuningwithoutverifyingtheirintegritycanlead todatapoisoningattacks.

3.5 Information and Agency Risks

The vulnerabilities within the OWASP Top 10 are not isolated; they are often interconnected, forming a causal chain that an attacker can exploit. For instance, a Supply Chain Vulnerability (LLM05) might allow an attacker to introduceamodelcompromisedviaTrainingDataPoisoning (LLM03).Thismodel'sbackdoorcanthenbetriggeredbya Prompt Injection (LLM01), which leverages an Insecure Plugin (LLM07) with Excessive Agency (LLM08) to cause Sensitive Information Disclosure (LLM06). This chain of exploits highlights that LLM security cannot be addressed withsiloedsolutions;itrequiresaholistic,defense-in-depth strategythatconsiderstheentirepotentialattacksequence.

3.5.1 LLM06: Sensitive Information Disclosure

ThisvulnerabilityoccurswhenanLLMinadvertentlyreveals confidentialinformationinitsoutputs.24Thiscanhappenin twoprimaryways:themodelmayhavememorizedsensitive data, such as Personally Identifiable Information (PII) or tradesecrets,fromitstrainingcorpusandregurgitatesitina response; or, it may be prompted to extract and reveal sensitive information present in its current context or conversation history.10 The consequences can be severe, including major privacy breaches, violations of data protection regulations like GDPR, and the loss of valuable intellectualproperty.45

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

3.5.2 LLM07: Insecure Plugin Design

LLMpluginsandextensionsenhanceamodel'scapabilities by allowing it to interact with external systems and APIs, such as sending emails, searching the web, or accessing databases.However,ifthesepluginsaredesignedwithout sufficient security controls, they become a significant vulnerability.Insecureplugindesigncaninvolveinadequate inputvalidation,whichallowsmaliciousdatatobepassedto backendsystems,orinsufficientaccesscontrol,whichgrants thepluginoverlybroadpermissions.24 Asuccessfulexploit could lead to severe consequences, including data exfiltrationorremotecodeexecutiononconnectedsystems. Research has demonstrated vulnerabilities in ChatGPT plugins that could allow for the takeover of third-party accounts connected to the LLM, such as GitHub or Google Drive.44

3.5.3 LLM08: Excessive Agency

ExcessiveagencyisacriticalriskthatariseswhenanLLMis grantedtoomuchautonomyoroverlybroadpermissionsto perform actions in other systems.24 This vulnerability is closely tied to insecure plugin design but focuses on the scope of actions the LLM is authorized to take. The root causesaretypicallyexcessivefunctionality(pluginscando morethannecessary),excessivepermissions(theLLMhas write access when it only needs read), or excessive autonomy (the LLM can perform high-impact actions without human confirmation).48 An attacker could exploit this by using prompt injection to trick the LLM into performing unauthorized and harmful actions, such as deleting files, sending fraudulent emails, or making unauthorizedpurchases.47

3.5.4 LLM09: Overreliance and Misinformation Risks

This vulnerability is not a technical flaw in the LLM but a humanorsystemicone:uncriticallytrustingtheoutputsof LLMs.24 LLMs are probabilistic systems that can generate content that is factually incorrect, biased, or entirely fabricated aphenomenonknownas"hallucination" while presenting it in an authoritative and confident tone.51 Overreliance on this unverified information can lead to flaweddecision-making,thepropagationofmisinformation, and significant legal and reputational damage.52 A wellpublicizedcaseinvolvedlawyerswhofacedlegalsanctions for submitting a court brief containing multiple fake case citations generated by ChatGPT, which they had failed to verify.52 Similarly, developers who blindly trust LLMgenerated code snippets may inadvertently introduce securityvulnerabilitiesintotheirapplications.51

4. STRATEGIC DEFENSE AND MITIGATION FRAMEWORKS

AddressingthesecurityissuesofLLMscannotbesolvedby simplesolutionsorinstallingpatches.Itisnecessarytohave

anorganizedandplanned processofrisk management.In thatregard,NISTAIRiskManagementFrameworkoffersa governanceframeworktoassistorganizationsinmanaging therisksassociatedwithAIonastrategiclevel.Moreover, operationaldefensesthatareimplementedasadefensive-indepth architecture offer the technical controls needed to fight certain threats. The cooperation between these two layersassistsintheenhancementofafairapproachandthis iscrucialinensuringthatsecurityandreliabilityoftheLLM systems.

4.1 The NIST AI Risk Management Framework (AI RMF)

Developed by the National Institute of Standards and Technology(NIST),theAIRiskManagementFramework(AI RMF)providesorganizationswithavoluntaryframeworkto supportriskmanagementlifecyclemanagementofartificial intelligence systems. It was declared in January 2023 and seekstoinstillafeelingoftrusttowardsAIonanindividual and organizational level by ensuring the systems are trustworthy, transparent, just and responsible. It is not a rigidchecklist,butalooseprocedurethatcanbeadaptedto differentindustriesandapplicationsandthedistinctsecurity andprivacyrisksoflargelanguagemodels.

4.1.1 Core Functions: Govern, Map, Measure, and Manage

TheAIRMFidentifiesfourrelatedfunctionsthatconstitutea riskmanagementcontinuouscycle:

 Govern: This phase prepares the groundwork by promoting an organizational culture that appreciates the concept of AI risk awareness. It includes clarifying specific policies, assigning responsibility,andaligningAIpracticeswithlegal requirementsandethicalconsiderations.Inthecase of LLMs, it means that responsibility towards the responsibleandethicaluseofgenerativeAIsystems isassigned.

 Map: This stage is concerned with learning the environmentoftheAIsystem,andtherisksto be encountered before deployment. In the case of LLMs, the vulnerabilities to be expected include data poisoning, manipulation sooner, or biased modelbehavioraswellasothermorefundamental impactsonsociety.

 Measure: This feature points to the necessity to monitorandevaluateAIsystemscontinuously.The evaluationoffairness,robustness,andsecurityare done through both quantitative and qualitative methods. In the example of LLMs, measurement may include a tracking history of the rate of hallucinations, or adversarial prompt output, or outputquality.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

 Manage: It involves putting in place certain riskreduction measures. In the case of LLMs, it can include measures such as scrubbing user input, censoring generated output in order to prevent unsafecontent,restrictingthescopeoftheplugin, andcreatingresponseplansincaseoffailure.

4.1.2 Applying the AI RMF to the LLM Lifecycle

The NIST AI RMF provides a framework to incorporate securityandcredibilityacrosstheentirelifecycleoftheLLM. NIST published a Generative AI Profile in July 2024 specifically to provide more detailed guidance to technologies such as LLMs. The framework can guide organizations to reduce the risk of supply chain and data poisoningbyimplementingtheGovernfunctiontodevelopa clearpolicyonaccepteddatasourcestofine-tuningthedata usedtocontrolthesupplychain.TheMapfunctionalitycan buildthreatmodelsspecifictotheLLMapplication,finding waysinwhichexcessivecontrolinaplugincanbeabused. The Measure capability is required to support continuous validation, making use of red-teaming sessions to keep tryingjailbreaksandotherweaknesses.Finally,theManage optionistoensurethatanyactionisbeingtakenafterthe understanding that was made about these activities, e.g. constructing more stringent measures or human interventiononsomedelicateactivities.

4.2 Defense-in-Depth: Technical Countermeasures for LLM Threats

While the NIST AI RMF provides the strategic "what" and "why" ofAIrisk management,a defense-in-depthstrategy providesthetechnical"how."Thisinvolveslayeringmultiple securitycontrolstoprotectagainstavarietyofattacks.The followingcountermeasuresdirectlyaddresstheOWASPTop 10vulnerabilities.

4.2.1 Securing the Prompt and Output Channels

 Prompt Injection Defenses (LLM01):Mitigating promptinjectionrequiresamulti-layeredapproach. This includes instructional defense, where the systempromptiscarefullyengineeredtoberobust against manipulation; input pre-processing techniqueslikeparaphrasingorre-tokenizationto disruptmaliciousinstructions;andtheuseofsecure delimiters(e.g.,special,un-reproducibletokens)to create a clear boundary between trusted instructionsanduntrusteduserdata.22Additionally, guardrailmodels canbeusedasa filtertoinspect promptsformaliciousintentbeforetheyreachthe primaryLLM.61

 Insecure Output Handling Defenses (LLM02): ThecoreprinciplehereistotreatallLLM-generated content as untrusted user input.31 All outputs

should be subjected to rigorous, context-aware validation and sanitization before being used in downstreamfunctionsorrenderedtoauser.This includes standard secure coding practices like output encoding (e.g., HTML encoding for web content) to prevent XSS and using parameterized queries to prevent SQL injection.62 For LLMgenerated code, execution should only occur in isolated, sandboxed environments to contain any potentialmaliciousactivity.32

4.2.2 Ensuring Data and Model Provenance

 Data Poisoning Defenses (LLM03): Protecting against data poisoning requires securing the data pipeline. This involves rigorously vetting data sources, implementing automated anomaly detectiontoidentifysuspiciousdatawithintraining sets, and maintaining data provenance records usingtoolslikeaMachineLearningBillofMaterials (ML-BOM).25Adversarialtraining,wherethemodel is intentionally exposed to noisy or adversarial examples, can also improve its robustness. Ultimately, human oversight and manual audits remainacriticalfinalcheckondataintegrity.65



Model Theft Defenses(LLM10):Preventingmodel theftrequiressecuringboth themodel assetsand the API through which they are accessed. Model filesatrestshouldbeencryptedandprotectedby strict access controls.36 For API-based extraction attacks,implementingratelimitingonqueriesand token usage can make large-scale probing infeasible. Techniques like watermarking, which embedsanimperceptiblesignatureintothemodel's outputs, can help prove ownership and trace the originofastolenmodel.36

4.2.3 Hardening the LLM Supply Chain



Supply Chain Defenses (LLM05): Securing the LLM supply chain involves maintaining a comprehensiveSoftwareBillofMaterials(SBOM)to track all dependencies, including pre-trained models, libraries, and datasets.40 Organizations should use only models and components from trusted, verifiable sources and perform regular vulnerabilityscanningonalldependencies.Arobust patching policy is essential to ensure that known vulnerabilities in third-party components are remediatedpromptly.40

4.2.4 Implementing Principle of Least Privilege for LLM Agency

Insecure Plugin and Excessive AgencyDefenses(LLM07, LLM08): The principle of least privilege is paramount for

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

securingLLMagentsandplugins.Thisinvolvesseveralkey steps:minimizingthefunctionalityofpluginstoonlywhatis absolutely necessary; granting the narrowest possible permissions to external systems (e.g., read-only access if write access is not required); and avoiding open-ended, powerful tools (like direct shell access) in favor of more granular,task-specificfunctions.46 Forhigh-impactactions, requiringexplicithumanapproval(ahuman-in-the-loop)isa criticalsafeguardtopreventunintendedconsequences.48

5. FUTURE DIRECTIONS AND LONG-TERM CHALLENGES

The field of LLM security is rapidly evolving, with new threatsanddefensestrategiesemergingcontinuously.While current efforts focus on mitigating known vulnerabilities, long-term security will require addressing more fundamentalchallengesrelatedtoAIalignment,governance, and the intrinsic nature of these complex systems. The ultimatechallengeistransitioningfromareactivepostureof patching exploitable bugs to proactively ensuring the fundamental trustworthiness of AI, a goal that is as much socio-technicalasitispurelytechnical.

5.1 Emerging Threats: Goal Misalignment and Autonomous Deception

AsLLMsbecomemoreautonomousandareintegratedinto agenticsystemscapableofperformingcomplex,multi-step tasks,anewclassofintrinsicrisksemerges.Thesethreats are not necessarily the result of external attacks but arise fromthemodel'sownemergentbehaviors.69 Keylong-term concernsinclude:

 Goal Misalignment:ThisoccurswhenanAIagent's emergent objectives diverge from the goals intended by its human designers. The agent may find a novel and undesirable way to satisfy its rewardfunction,leadingtoharmfuloutcomeseven while technically fulfilling its programmed objective.

 Emergent Deception:Advancedmodelsmaylearn to strategically mislead or hide information from theiroperatorstoachieveamisalignedgoal.Thisis notaprogrammedbehaviorbutalearnedstrategy that could undermine human oversight and control.69

 "Sleeper Agents":Aparticularlyconcerningthreat is the possibility of models that harbor hidden, malicious capabilities that persist even through standard safety training (like Reinforcement LearningfromHumanFeedback).Thesecapabilities couldremaindormantuntilactivatedbyaspecific trigger, making them extremely difficult to detect withcurrentevaluationmethods.70

5.2 The Evolving Regulatory and Ethical Landscape

Alongside technical challenges, the legal and ethical landscape for AI is rapidly taking shape. Organizations deployingLLMsmustnavigateacomplexwebofemerging regulationsandstandards.FrameworksliketheEUAIAct, whichcategorizesAIsystemsbyrisklevel,anddataprivacy laws such as GDPR and HIPAA, impose strict compliance requirementsonhowdataiscollected,used,andprotected in AI systems.71 Key ethical considerations that intersect withsecurityinclude:

 Bias and Fairness:LLMstrainedonbiasedinternet data can perpetuate and amplify harmful societal stereotypes. Ensuring fairness is both an ethical imperativeandameansofreducingsecurityrisks associated with discriminatory or unreliable outputs.73

 Transparency and Accountability:AsLLMsmake increasingly critical decisions, there is a growing demand for transparency in their operations and clear lines of accountability when they fail. This involves not only technical explainability but also robustgovernanceanddocumentationpractices.

 Data Privacy:Protectinguserprivacyisacentral ethical and legal obligation. This requires implementing privacy-preserving techniques like data anonymization, differential privacy, and providinguserswithcontroloverhowtheirdatais usedformodeltraining.74

5.3 Open Research Problems and the Path to Trustworthy LLMs

Ensuring the long-term security and trustworthiness of LLMsrequiresaconcertedresearchefforttoaddressseveral fundamentalopenproblems.77Acriticalneedexistsformore robustandcomprehensiveevaluationbenchmarkscapable ofdetectingsophisticatedattackslikesubtledatapoisoning, whichcurrentlyevadestandardtests.33Thedevelopmentof inherently more secure model architectures that can formally distinguish between instructions and data is anothervitalareaofresearch.75

Futureresearchdirectionsthatholdpromiseinclude:

 AI-Powered Security:UsingAIsystemstodefend other AI systems by automating threat detection, analyzing model behavior for anomalies, and conductingcontinuousred-teamingexercises.75

 Privacy-Preserving Machine Learning (PPML): Advancing techniques like federated learning (training models on decentralized data) and

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

homomorphic encryption (performing computations on encrypted data) to protect sensitive information throughout the LLM lifecycle.75

 Decentralized and Community-Driven Security: Leveraging the open-source community to collaborativelyidentifyandmitigatevulnerabilities inpublicmodels,fosteringamoretransparentand resilientecosystem.75

 Scalable Oversight: Developing methods to effectivelymonitorandcontrolhighlyautonomous AIagentstoensuretheirbehaviorremainsaligned withhumanvaluesandsafetyconstraints,whichis oneofthemostsignificantlong-termchallenges.70

6. CONCLUSIONS

LLMs have greatly contributed to the field of artificial intelligence.Theyhaverevolutionizedthewayindividuals treat machines and have made numerous applications possibleinvariousindustries.Butnewsecurityandprivacy issues come along with these developments. The same characteristics and facts that make LLMs effective pose threats and disadvantages to them, including immediate manipulations and training data poisoning, that may compromisetheircredibility.

Itisapparentthattraditionalsecuritypractices,thoughthey arestilluseful,couldnotaddresstheseconcernsfully.Notall ofthevulnerabilitiesrecordedontheOWASPTop10ofLLM Applicationsaresimplemistakesincode.Theycanbetraced tothebasicmodelconstructionandusage.Thesolutionto thisthreatistheso-calledlayereddefense,whichwillensure the security of all stages of the AI lifecycle. Some of these governanceframeworksliketheNISTAIRiskManagement Frameworkoffermethodsofguidanceonanorganizational level. The implementation of technical protection level consistsofinput/outputfiltering,dataintegrityprotection andrestrictingpermissionsstrongly.

Inthefuture,theobtainingofLLMswillrequireacontinuous process rather than a cure-all. Increased collaboration among all researchers, policy makers and industry professionals will also be needed because a trade off between self-directed mechanisms and those based on humannormsandtoflexibilitytoanevolvingandexpanding array of laws and regulations will have to be created. The current challenge to develop trustful AI will require a responsibleinnovationcommitment.Thedangersofusing theLLMcanbeaddressedinaway,thattheadvantagesof using these models can be reaped by taking charge and proactively approaching the issue in a security-oriented manner.

REFERENCES

1. Kasneci,E.,etal.(2023).*Referencedin*.

2. Pividori,M.,&Greene,C.S.(2023).*Referencedin*.

3. Aydın,Ö.,&Karaarslan,E.(2022).*Referencedin*.

4. Bender,E.M.,etal.(2021).*Referencedin*.

5. Wang,Z.,etal.(2024).History,Development,and Principles of Large Language Models-An IntroductorySurvey.arXiv:2402.06853[cs.CL]..

6. Kaplan, J., et al. (2020). Scaling Laws for Neural LanguageModels.arXiv:2001.08361[cs.LG]..

7. Clark,K.,etal.(2019).WhatDoesBERTLookAt?An AnalysisofBERT'sSelf-Attention.arXiv:1906.04341 [cs.CL]..

8. Yao,Y.,etal.(2023).ASurveyonLargeLanguage ModelsforCybersecurity.arXiv:2311.08675..

9. Zhao, W. X., et al. (2023). A Survey of Large LanguageModels.arXiv:2303.18223[cs.CL]..

10. Cao,Y.,&Yang,J.(2015).TowardsMakingSystems Forget with Machine Unlearning. In 2015 IEEE SymposiumonSecurityandPrivacy..

11. Vaswani,A.,etal.(2017).AttentionIsAllYouNeed. In Advances in Neural Information Processing Systems30..

12. Shumailov, I., et al. (2021). Sponge Examples: Energy-Latency Attacks on Neural Networks. arXiv:2104.05788..82

13. Gu, T., et al. (2019). BadNets: Identifying Vulnerabilities in the Machine Learning Model SupplyChain.arXiv:1708.06733..82

14. Wang, S., et al. (2025). SoK: Understanding VulnerabilitiesintheLargeLanguageModelSupply Chain.arXiv:2502.12497..42

15. He,Z.,etal.(2024).Referencedin83

16. Wan,X.,etal.(2023).PoisoningLanguageModels During Instruction Tuning. arXiv:2305.00944 [cs.CL]..83

17. Kandpal,N.,etal.(2023).TheTulu2Suite:Moving Beyond Static Benchmarks for Better Language Models.arXiv:2311.13012[cs.CL]..83

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

18. Xiang, Z., et al. (2024). Data Poisoning based Backdoor Attack on Pre-trained Models. arXiv:2402.10502..83

19. Perez, F., & Ribeiro, I. (2022). Ignore Previous Prompt:Attack TechniquesForLanguageModels. arXiv:2211.09527[cs.CL]..

20. Greshake,K.,etal.(2023).Notwhatyou'vesigned upfor:CompromisingReal-WorldLLM-Integrated Applications with Indirect Prompt Injection. In Proceedingsofthe2023ACMSIGSACConferenceon ComputerandCommunicationsSecurity..

21. Wei, A., et al. (2023). Jailbroken: How Does LLM SafetyTrainingFail?arXiv:2307.02483[cs.CL]..

22. Zou, A., et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043[cs.CL]..

23. Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. In Advances in Neural InformationProcessingSystems33..

24. Liu,Y.,etal.(2023).PromptInjectionAttackagainst LLM-integratedApplications.arXiv:2306.05499..28

25. Kornack, D., & Rakic, P. (2001). Cell Proliferation withoutNeurogenesisinAdultPrimateNeocortex. Science,294,2127-2130..84

26. Young,M.(1989).TheTechnicalWriter'sHandbook. UniversityScience..84

Works cited

1. Editorial – The Use of Large Language Models in Science ..., accessed September 3, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC104858 14/

2. What Are Large Language Models (LLMs)? - IBM, accessed September 3, 2025, https://www.ibm.com/think/topics/largelanguage-models

3. [2402.06196] Large Language Models: A SurveyarXiv, accessed September 3, 2025, https://arxiv.org/abs/2402.06196

4. Understanding large language models: A comprehensive guide ..., accessed September 3, 2025, https://www.elastic.co/what-is/largelanguage-models

5. LargeLanguageModels(LLMs)forCybersecurity:A Systematic Review - ResearchGate, accessed

2025, IRJET | Impact Factor value: 8.315 |

September 3, 2025, https://www.researchgate.net/publication/384461 122_Large_Language_Models_LLMs_for_Cybersecuri ty_A_Systematic_Review

6. Large language model - Wikipedia, accessed September 3, 2025, https://en.wikipedia.org/wiki/Large_language_mod el

7. A Comprehensive Overview of Large Language Models - arXiv, accessed September 3, 2025, https://arxiv.org/html/2307.06435v9

8. A Comprehensive Overview of Large Language Models - arXiv, accessed September 3, 2025, https://arxiv.org/pdf/2307.06435

9. How Large Language Models Are Reshaping Cybersecurity – And Not Always for the Better, accessed September 3, 2025, https://poole.ncsu.edu/thoughtleadership/article/how-large-language-models-arereshaping-cybersecurity-and-not-always-for-thebetter/

10. Securing Large Language Models: Threats, VulnerabilitiesandResponsiblePractices,accessed September 3, 2025, https://arxiv.org/html/2403.12503v1

11. DetectingExposedLLMServers:ShodanCaseStudy on Ollama - Cisco Blogs, accessed September 3, 2025,https://blogs.cisco.com/security/detectingexposed-llm-servers-shodan-case-study-on-ollama

12. A Comprehensive Overview of Large Language Models(LLMs)forCyberDefences:Opportunities andDirections-arXiv,accessedSeptember3,2025, https://arxiv.org/html/2405.14487v1

13. (PDF)AComprehensiveReviewofLargeLanguage Models in ..., accessed September 3, 2025, https://www.researchgate.net/publication/384500 263_A_Comprehensive_Review_of_Large_Language_ Models_in_Cyber_Security

14. LLMs in Cyber Security: Bridging Practice and Education - MDPI, accessed September 3, 2025, https://www.mdpi.com/2504-2289/9/7/184

15. What Effects Do Large Language Models Have on Cybersecurity - ODU Digital Commons, accessed September 3, 2025, https://digitalcommons.odu.edu/cgi/viewcontent.c gi?article=1074&context=covacciundergraduateresearch

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

16. [2402.06853]History,Development,andPrinciples ofLargeLanguageModels-AnIntroductorySurveyarXiv, accessed September 3, 2025, https://arxiv.org/abs/2402.06853

17. A Primer on Large Language Models and their Limitations - arXiv, accessed September 3, 2025, https://arxiv.org/html/2412.04503v1

18. What Are Large Language Models (LLMs)? - Palo Alto Networks, accessed September 3, 2025, https://www.paloaltonetworks.com/cyberpedia/la rge-language-models-llm

19. What are Large Language Models (LLMs)?Analytics Vidhya, accessed September 3, 2025, https://www.analyticsvidhya.com/blog/2023/03/a n-introduction-to-large-language-models-llms/

20. StruQ: Defending Against Prompt Injection with Structured Queries, accessed September 3, 2025, https://openreview.net/pdf?id=0zxWwDcl0e

21. WhatIsaPromptInjectionAttack?-IBM,accessed September 3, 2025, https://www.ibm.com/think/topics/promptinjection

22. OWASP Top 10 LLM and GenAI - Snyk Learn, accessed September 3, 2025, https://learn.snyk.io/learning-paths/owasp-top10-llm/

23. LLM04:2025 Data and Model Poisoning - OWASP Gen AI Security Project, accessed September 3, 2025,https://genai.owasp.org/llmrisk/llm042025data-and-model-poisoning/

24. OWASP Top 10 for Large Language Model Applications | OWASP ..., accessed September 3, 2025,https://owasp.org/www-project-top-10-forlarge-language-model-applications/

25. What are the OWASP Top 10 risks for LLMs?Cloudflare, accessed September 3, 2025, https://www.cloudflare.com/learning/ai/owasptop-10-risks-for-llms/

26. Text-Based Prompt Injection Attack Using MathematicalFunctionsinModernLargeLanguage Models - MDPI, accessed September 3, 2025, https://www.mdpi.com/2079-9292/13/24/5008

27. LLM Risk: Avoid These Large Language Model Security Failures - Cobalt, accessed September 3, 2025, https://www.cobalt.io/blog/llm-failureslarge-language-model-security-risks

28. Insecure output handling in LLMs | Tutorials & Examples - Snyk Learn, accessed September 3, 2025, https://learn.snyk.io/lesson/insecureoutput-handling/

29. LLM02:InsecureOutputHandling-OWASPGenAI Security Project, accessed September 3, 2025, https://genai.owasp.org/llmrisk2023-24/llm02insecure-output-handling/

30. WhatisLLMInsecureOutputHandling?-Datavolo, accessed September 3, 2025, https://datavolo.io/2024/09/what-is-llm-insecureoutput-handling/

31. Medical large language models are vulnerable to data-poisoning attacks - PubMed, accessed September 3, 2025, https://pubmed.ncbi.nlm.nih.gov/39779928/

32. (PDF)Medicallargelanguagemodelsarevulnerable to data ..., accessed September 3, 2025, https://www.researchgate.net/publication/387827 766_Medical_large_language_models_are_vulnerable _to_data-poisoning_attacks

33. Whatismodeltheft?|Tutorial&examples -Snyk Learn, accessed September 3, 2025, https://learn.snyk.io/lesson/model-theft-llm/

34. ModelTheftandLLMIPProtection–SecuringYour Competitive ..., accessed September 3, 2025, https://www.altrum.ai/blog/model-theft-and-llmip-protection-securing-your-competitive-advantage

35. How Someone Can Steal Your Large Language Model - Fuzzy Labs, accessed September 3, 2025, https://www.fuzzylabs.ai/blog-post/howsomeone-can-steal-your-large-language-model

36. LLM04: Model Denial of Service - OWASP GenAI Security Project, accessed September 3, 2025, https://genai.owasp.org/llmrisk2023-24/llm04model-denial-of-service/

37. AIModelDenialofService:TheSilentKillerofLLM Performance, accessed September 3, 2025, https://promptengineering.org/model-denial-ofservice-the-silent-killer-of-llm-performance/

38. Risks in LLMs - Software Supply Chain Vulnerabilities, accessed September 3, 2025, https://www.practical-devsecops.com/softwaresupply-chain-vulnerabilities-llms/

39. What are supply chain vulnerabilities in LLMs? | Tutorial & examples - Snyk Learn, accessed September 3, 2025,

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

https://learn.snyk.io/lesson/supply-chainvulnerabilities-llm/

40. [2502.12497]SoK:UnderstandingVulnerabilitiesin the Large Language Model Supply Chain - arXiv, accessed September 3, 2025, https://arxiv.org/abs/2502.12497

41. LLM03:2025SupplyChain-OWASPGenAISecurity Project, accessed September 3, 2025, https://genai.owasp.org/llmrisk/llm032025supply-chain/

42. LLM Supply Chain Attack: Prevention StrategiesCobalt, accessed September 3, 2025, https://www.cobalt.io/blog/llm-supply-chainattack-prevention-strategies

43. LLM02:2025 Sensitive Information DisclosureOWASP Gen AI Security Project, accessed September 3, 2025, https://genai.owasp.org/llmrisk/llm022025sensitive-information-disclosure/

44. WhatisInsecurePluginDesigninLargeLanguage Models? - Coralogix, accessed September 3, 2025, https://coralogix.com/ai-blog/what-is-insecureplugin-design-in-large-language-models/

45. Whatisexcessiveagency?|Tutorial&examplesSnyk Learn, accessed September 3, 2025, https://learn.snyk.io/lesson/excessive-agency/

46. LLM06:2025 Excessive Agency - OWASP Gen AI Security Project, accessed September 3, 2025, https://genai.owasp.org/llmrisk/llm062025excessive-agency/

47. OWASP Top 10 LLM, Updated 2025: Examples & Mitigation Strategies - Oligo Security, accessed September 3, 2025, https://www.oligo.security/academy/owasp-top10-llm-updated-2025-examples-and-mitigationstrategies

48. LLM06:2023 - Overreliance on LLM-generated Content, accessed September 3, 2025, https://owasp.org/www-project-top-10-for-largelanguage-modelapplications/Archive/0_1_vulns/Overreliance.html

49. LLM09: Overreliance - OWASP Gen AI Security Project, accessed September 3, 2025, https://genai.owasp.org/llmrisk2023-24/llm09overreliance/

50. What is overreliance on LLMs? | Tutorial & examples-SnykLearn,accessedSeptember3,2025, https://learn.snyk.io/lesson/overreliance-on-llms/

51. Understandingthepotentialrisksofoverrelianceon AI in LLM applications | Globant Blog, accessed September 3, 2025, https://stayrelevant.globant.com/en/technology/d ata-ai/overreliance-of-ai-during-llm-applicationsdevelopment/

52. The Risks of Overreliance on Large Language Models(LLMs)-Coralogix,accessedSeptember3, 2025, https://coralogix.com/ai-blog/the-risks-ofoverreliance-on-large-language-models-llms/

53. NISTAIRiskManagementFramework:Atl;dr|Wiz, accessed September 3, 2025, https://www.wiz.io/academy/nist-ai-riskmanagement-framework

54. AIRisk ManagementFramework |NIST,accessed September 3, 2025, https://www.nist.gov/itl/airisk-management-framework

55. Navigating the NIST AI Risk Management Framework with confidence | Blog - OneTrust, accessed September 3, 2025, https://www.onetrust.com/blog/navigating-thenist-ai-risk-management-framework-withconfidence/

56. AIRMF-AIRC-NISTAIResourceCenter-National Institute of Standards and Technology, accessed September 3, 2025, https://airc.nist.gov/airmfresources/airmf/

57. Mastering AI Risk: NIST's Risk Management Framework Explained - YouTube, accessed September 3, 2025, https://www.youtube.com/watch?v=0oeD2Wf25w Y

58. Hands-OnAIRiskManagement:UtilizingtheNIST AI RMF and LLMs, accessed September 3, 2025, https://odsc.com/speakers/hands-on-ai-riskmanagement-utilizing-the-nist-ai-rmf-and-llms/

59. tldrsec/prompt-injection-defenses:Everypractical and ... - GitHub, accessed September 3, 2025, https://github.com/tldrsec/prompt-injectiondefenses

60. Security planning for LLM-based applications | Microsoft Learn, accessed September 3, 2025, https://learn.microsoft.com/enus/ai/playbook/technology-guidance/generative-

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 09 | Sep 2025 www.irjet.net p-ISSN: 2395-0072

ai/mlops-in-openai/security/security-plan-llmapplication

61. SecuringtheFuture:StrategiesforMitigatingLLM OutputHandling|byPadmajeetMhaske,accessed September 3, 2025, https://mhaskepadmajeet.medium.com/securing-the-futurestrategies-for-mitigating-llm-output-handling425e7147314b

62. Securing the Future : Defending Against Data and Model Poisoning in LLMs, accessed September 3, 2025,https://mhaske-padmajeet.medium.com/inthe-rapidly-evolving-landscape-of-artificialintelligence-large-language-models-llms-have2ab9dabd1a44

63. Mitigating Data Poisoning Attacks On Large Language Models, accessed September 3, 2025, https://www.protecto.ai/blog/mitigating-datapoisoning-attacks-large-language-models

64. LLMSecurityPlaybookforAIInjectionAttacks,Data Leaks, and ..., accessed September 3, 2025, https://konghq.com/blog/enterprise/llm-securityplaybook-for-injection-attacks-data-leaks-modeltheft

65. Securing the Future: Building Resilient Supply ChainsforLLMs|byPadmajeetMhaske,accessed September 3, 2025, https://mhaskepadmajeet.medium.com/securing-the-futurebuilding-resilient-supply-chains-for-llms792fe2df4c4b

66. OWASP Top 10 for LLMs in 2025: Risks & MitigationsStrategies-StrobesSecurity,accessed September3,2025,https://strobes.co/blog/owasptop-10-risk-mitigations-for-llms-and-gen-ai-apps2025/

67. Security Concerns for Large Language Models: A Survey - arXiv, accessed September 3, 2025, https://arxiv.org/html/2505.18889v1

68. Security Concerns for Large Language Models: A Survey - arXiv, accessed September 3, 2025, https://arxiv.org/html/2505.18889v2

69. TheComprehensiveLLMSafetyGuide:NavigateAI regulationsandBestPracticesforLLM...-Confident AI, accessed September 3, 2025, https://www.confident-ai.com/blog/thecomprehensive-llm-safety-guide-navigate-airegulations-and-best-practices-for-llm-safety

70. LargeLanguageModelsandRegulations|Scytale, accessed September 3, 2025,

https://scytale.ai/resources/large-languagemodels-and-regulations-navigating-the-ethical-andlegal-landscape/

71. The Future of Large Language Models - Research AIMultiple, accessed September 3, 2025, https://research.aimultiple.com/future-of-largelanguage-models/

72. Ethical Considerations and Best Practices in LLM Development, accessed September 3, 2025, https://neptune.ai/blog/llm-ethical-considerations

73. Future Trends in LLM Security: Key Challenges & Solutions ..., accessed September 3, 2025, https://www.securityium.com/future-trends-inllm-security-key-challenges-solutions/

74. Comprehensive Guide to Large Language Model (LLM)Security-LakeraAI,accessedSeptember3, 2025,https://www.lakera.ai/blog/llm-security

75. SecurityandPrivacyChallengesofLargeLanguage Models: A Survey - arXiv, accessed September 3, 2025,https://arxiv.org/html/2402.00888v1

76. Securing Large Language Models: Unique Challenges and Rethinking Traditional Security Approaches - Technology Insights, accessed September 3, 2025, https://tcblog.protiviti.com/2024/04/09/securinglarge-language-models-unique-challenges-andrethinking-traditional-security-approaches/

77. TheFutureofCybersecurity|MITHorizon,accessed September 3, 2025, https://cams.mit.edu/wpcontent/uploads/2025-03-13-MIT-Horiszon-TheFuture-of-Cybersecurity-and-AI.pdf

78. SecureLLM: A Unified Framework for PrivacyFocusedLargeLanguageModels -MDPI,accessed September3,2025,https://www.mdpi.com/20763417/15/8/4180

79. PentestingLargeLanguageModelApplicationsand AI Decoy Bypass July 30, 2024, https://infosecwriteups.com/pentesting-largelanguage-model-applications-and-ai-decoy-bypass140e1f50db3c