Agentic_AI_Security_Privacy_Challenges.

Artificial Intelligence systems are increasingly being designed to do more than generate a single response. Agentic systems can interpret a goal, create a plan, call tools, observe results, revise their plan, and continue until a task is completed or a stopping condition is reached. A typical agent may interact with browsers, databases, code interpreters, email services, enterprise APIs, or local files. This ability can improve productivity, but it also expands the system’s attack surface. Traditional application security often assumes that software follows predefined workflows and that users explicitly initiate consequential operations. An agent may instead select tools dynamically and make decisions based on retrieved documents or messages that contain untrusted instructions. If safeguards are weak, a malicious web page or document may influence the agent, or a valid tool permission may be used in an unintended way. Privacy concerns also arise when an agent collects, stores, summarizes, or forwards personal and organizational information across multiple steps. A. Problem Statement The main problem is how to enable useful autonomous behaviour while preventing unauthorized actions, disclosure of sensitive information, and unsafe reliance on untrusted content. Security cannot be addressed only by asking the model to behave responsibly; controls must also exist in the application architecture, tool layer, data store, and monitoring process. B. Aim and Objectives The objectives are to: (1) identify major security and privacy risks in agentic AI; (2) map risks to the agent lifecycle; (3) propose a layered framework of preventive, detective, and corrective controls; (4) define a practical threat-based evaluation plan; and (5) discuss limitations and responsible deployment considerations. C. Scope and Contribution This paper focuses on software-based agents that use tools, external information, and optional memory to complete tasks. It provides a conceptual framework suitable for student research and prototype planning. It does not present a completed penetration test, measured risk reduction, or a claim that one control can eliminate all agent-related threats.

A. Agentic AI and the Expanded Attack Surface An agentic application typically combines a language model with planning logic, tool adapters, context retrieval, memory, and an execution environment. Each component introduces a different trust boundary. The model may misinterpret instructions; retrieved content may be adversarial; a tool may expose broad permissions; and a memory store may preserve information beyond its useful lifetime. Therefore, the security boundary includes the full application rather than the model alone. B. Security Guidance and Risk Taxonomies The NIST AI Risk Management Framework provides a general structure for governing, mapping, measuring, and managing AI risks [1]. OWASP’s guidance on large language model applications describes risks such as prompt injection, sensitive information disclosure, excessive agency, and insecure output handling [2]. MITRE ATLAS documents tactics and techniques associated with attacks against AI-enabled systems [3]. These resources are useful foundations, although agent-specific deployments still require threat modelling of their particular tools, permissions, and workflows. C. Privacy by Design Privacy protection requires purpose limitation, data minimization, access control, retention limits, and transparency about how personal information is used. Agent memory and retrieved context make these principles especially important: information supplied for one task should not automatically become available to every future task. Data protection must be considered during collection, processing, storage, sharing, and deletion. D. Identified Research Gap A useful practical approach must connect high-level AI risk categories to concrete runtime controls. For example, identifying prompt injection as a risk is not sufficient unless the system also limits what the agent can do after encountering suspicious content. This paper addresses that design-level need by linking threat categories to controls, monitoring evidence, and test scenarios.

A. Framework Architecture The proposed framework places security checks around the agent rather than relying on a single model instruction. The user request first passes through identity and policy checks. The planner proposes a sequence of steps, but each tool call is evaluated by a policy enforcement layer. Retrieved content is treated as untrusted data, and outputs are checked before being returned or used to trigger consequential actions. Logging and monitoring operate across the workflow. User request Identity / scope Agent planner Plan / reason Policy gate Permission check Tool layer API / files Validator Output checks Cross-cutting controls: privacy audit logs monitoring human approval

Fig. 1. Proposed layered security and privacy architecture for an agentic AI system. B. Major Risk Categories Risk area Example threat Primary control Prompt / instruction injection A retrieved page attempts to override task instructions. Separate instructions from data; screen and constrain tool actions. Excessive agency Agent sends messages, deletes files, or makes changes without adequate review. Least privilege, action limits, and human approval for high-impact actions. Sensitive data disclosure Private context appears in an answer, log, or external tool request. Data minimization, redaction, destination checks, and access control. Insecure tool integration Tool accepts unsafe arguments or has overly broad credentials. Schema validation, scoped credentials, sandboxing, and allowlists. Memory and retention Unnecessary personal information persists across tasks. Purpose-bound memory, expiry, user controls, and deletion procedures. Weak monitoring A multi-step failure is difficult to reconstruct. Trace IDs, immutable audit events, alerting, and incident response. C. Core Design Principles The framework follows six principles: least privilege; explicit trust boundaries; human oversight for irreversible or high-impact actions; data minimization; fail-safe defaults; and traceability. Tool permissions should be scoped to the task, and the system should stop or request clarification when a request exceeds its authorized scope. Model-generated text should not be treated as proof that an action is safe or that an external operation succeeded.

A. Threat Actors and Assets Potential threat actors include malicious users, compromised third-party services, publishers of adversarial web content, and insiders with inappropriate access. Important assets include user prompts, private documents, credentials, memory records, tool outputs, audit logs, and actions performed on external systems. The threat model should also consider accidental misuse, model error, stale information, and failures caused by poorly configured integrations. B. Prompt Injection and Untrusted Content An agent may retrieve an email, document, or web page containing instructions that conflict with the user’s request or system policy. The content should be processed as data rather than granted authority to redefine the agent’s goals. Defences include clear instruction hierarchy, content provenance labels, restricted tool permissions, and checks before sensitive actions. Because detection may be imperfect, the strongest protection is to limit the damage an injected instruction can cause. C. Tool Misuse and Excessive Agency Tools should expose narrowly scoped operations instead of unrestricted shell, database, or account access. Validate arguments against schemas, enforce destination allowlists where appropriate, and separate read operations from write operations. Require confirmation before sending external communications, changing permissions, deleting data, making purchases, or executing other consequential actions. Rate limits, transaction boundaries, and rollback mechanisms can reduce the impact of mistakes. D. Privacy and Data Leakage The application should collect only the information required for the task and avoid placing secrets in prompts, telemetry, or model-visible context unless necessary. Sensitive fields can be masked before external processing. Access to conversation history and memory should be purpose-bound and user-aware, with retention periods and deletion workflows. Logs should record security-relevant events without duplicating entire confidential documents or credentials. E. Identity, Credentials, and Isolation Every tool call should be associated with an authenticated user or service identity. Credentials should be stored in a secrets manager or protected environment configuration, not embedded in source code. Use short-lived tokens when supported, rotate exposed credentials, isolate code execution, and apply network restrictions. Separate development and production environments, and prevent one user’s context from being reused in another user’s session. F. Monitoring and Incident Response For each task, record a correlation identifier, requested operation, policy decision, tool name, outcome, timestamp, and relevant model or configuration version. Avoid logging secrets or unnecessary personal data. Monitoring should detect repeated policy denials, unusual tool-call volume, unexpected destinations, and failed authorization. An incident process should support disabling affected tools, revoking credentials, preserving relevant evidence, and informing responsible operators.

A. Limitations Agent behaviour can vary across model versions, prompts, tools, and context lengths, making reproducibility difficult. Attack patterns evolve, and a test suite cannot represent every real-world environment. Privacy requirements also depend on jurisdiction, application domain, and the nature of the data. Controls can reduce risk but cannot guarantee that an agent will never produce an unsafe response or make an error. B. Ethical and Responsible Use Users should be informed when an agent acts on their behalf, what data it accesses, and when human approval is required. High-impact decisions should remain subject to appropriate human accountability. Personal data should not be retained indefinitely or reused for unrelated purposes. System owners should document intended use, known limitations, escalation procedures, and responsibilities for handling incidents. C. Future Scope Future work can evaluate policy enforcement in multi-agent workflows, develop standardized agent threat benchmarks, investigate privacy-preserving memory, and measure how well controls transfer across models. Additional research may explore formal verification of tool permissions, provenance-aware retrieval, confidential computing, and continuous red-team testing. Real-world studies should assess both security outcomes and the effect of safeguards on task completion and usability.

Agentic AI introduces security and privacy challenges because models can plan actions, use external tools, and carry information across multiple steps. Major risks include prompt injection, excessive agency, insecure tool integration, sensitive-data leakage, weak identity controls, unsafe memory, and inadequate monitoring. This paper proposed a layered framework based on least privilege, trust-boundary enforcement, human approval for consequential operations, privacy-aware data handling, validation, and auditable monitoring. A threat-based evaluation plan was also defined without inventing experimental outcomes. Responsible deployment requires testing the complete system—including its model, tools, data, and operational procedures—and continually reviewing risks as capabilities and integrations change.