Cyber Knowledge · Domain 11 of 11 · Practitioner field guide
AI Security
AI security protects the complete socio-technical system around models: data pipelines, model artifacts, prompts, retrieval, application code, identities, plugins, agents, MCP servers, infrastructure, people, and decisions. This guide connects engineering, adversarial testing, monitoring, response, and governance so teams can move from a demo that “uses AI” to a system with explicit boundaries and reviewable evidence.
Scope and safety boundary
Use adversarial techniques only against systems you own or are explicitly authorized to assess. Model output is untrusted data, not authority. Never give an AI system broader tool, data, or network access than the task requires, and never treat an automated score as proof of security, privacy, compliance, or model correctness.
AI security is not a synonym for prompt filtering. A production system may combine a hosted or local model, system prompts, retrieval indexes, user content, memory, APIs, tools, agents, browsers, files, queues, databases, and human approvals. Each component has a different owner and trust level. The security objective is therefore to preserve confidentiality, integrity, availability, accountability, and safe human decision-making across the whole chain.
You will be able to
Draw a data-flow and trust-boundary model for an AI application.
Separate model behavior risks from ordinary application and infrastructure vulnerabilities.
Design least-privilege retrieval, agent, and MCP tool access.
Build abuse cases, tests, telemetry, response playbooks, and release evidence.
Evidence you should produce
System inventory, model/data provenance, and owner register.
Threat model with assumptions, abuse cases, and control owners.
Adversarial test corpus, results, residual-risk decisions, and regression gates.
Runtime audit trail, incident playbooks, and governed change history.
Map the AI system before testing it
Security is a property of the connected system—not the model in isolation.
Begin with three distinct questions: can an attacker compromise the application around the model, can untrusted content manipulate model-mediated behavior, and can the system cause unacceptable harm even without a conventional exploit? These overlap but require different tests and owners. An authorization flaw is still an authorization flaw; a retrieved document that changes an agent’s goal is an indirect prompt-injection problem; a plausible but false answer is a reliability and decision-control problem.
Architecture, assets, trust boundaries, and AI threat modeling
Create both a component diagram and a data-flow diagram. Mark every transition where content, identity, authority, or provenance changes. Model user input, uploaded documents, web retrieval, system instructions, conversation memory, vector search, model calls, tool calls, human approval, and output publication as separate flows. Shared infrastructure does not imply shared authorization.
Extend ordinary STRIDE-style analysis with AI-specific abuse cases: instruction override, retrieval poisoning, model extraction, membership inference, unsafe tool planning, resource exhaustion, insecure output consumption, and cross-tenant context leakage. For each case, document preconditions, attacker-controlled inputs, affected asset, observable signal, preventive control, recovery control, and residual uncertainty.
Threat-model record
System claim: what the feature is allowed to do—and explicitly cannot do.
Authority: whose identity is used for retrieval and action.
Untrusted channels: prompts, files, pages, connectors, model output, and tool output.
Decision gates: deterministic policy checks and human approvals outside the model.
Training data, ingestion, retrieval, RAG, and knowledge integrity
Data security spans collection, licensing, classification, preprocessing, training, embedding, indexing, retrieval, caching, retention, and deletion. A RAG system creates a new authorization surface: a document may be legitimate to store but illegitimate to retrieve for a particular user, tenant, task, or model. Embeddings are not an access-control boundary, and vector similarity is not evidence quality.
Preserve source URL or object identity, author/publisher, collection time, tenant, handling marking, checksum, parser version, chunk offsets, ACL, and supersession state. Apply authorization before retrieval results reach the prompt. Quarantine unknown formats, scan uploads, constrain parsers, and prevent fetched content from being interpreted as system authority. Detect duplicate, stale, contradictory, and poisoned material.
RAG integrity workflow
Accept only permitted sources and record provenance.
Normalize without destroying the original evidence.
Classify sensitivity and bind source ACLs to every chunk.
Validate parser output and reject active or malformed content.
Measure retrieval relevance, authorization correctness, citation accuracy, and poisoning resistance.
Reconcile additions, updates, deletions, and embedding-model changes.
1200km implementation context: compare these controls with the AdversaryGraph unified RAG and MCP guide, where source-linked CTI, IOC, CVE, and ATT&CK context must remain attributable and analyst-reviewable.
Models are executable dependencies with large transitive trust. Record provider, repository, model card, license, intended use, training disclosures, digest, serialization format, tokenizer, adapters, quantization, runtime, evaluation set, and deployment approval. Pin immutable revisions. Never load an untrusted pickle-like artifact into a privileged environment merely because it is called a model.
Apply ordinary software supply-chain controls to Python packages, containers, inference runtimes, GPU drivers, prompt templates, evaluation code, browser automation, MCP servers, and model gateways. Generate an SBOM where supported, scan artifacts, verify signatures or attestations, isolate conversion jobs, and retain a reproducible build path. Provider model aliases can change behavior without changing your code; production systems should detect and review that drift.
Release evidence
Approved model and dependency inventory with owners and end-of-life dates.
Artifact hashes, signature/attestation status, malware scan, and isolated load test.
Task-specific quality and abuse evaluations against the exact deployed configuration.
Rollback candidate and compatibility evidence for prompts, embeddings, and tools.
Prompt injection, instruction hierarchy, and secure output handling
Prompt injection is the attempt to make untrusted content influence instructions or decisions. Direct injection arrives from the user; indirect injection arrives through retrieved documents, email, web pages, images, tool responses, or memory. Delimiters, hidden prompts, and “ignore previous instructions” filters may improve robustness, but they do not create a security boundary.
Keep policy enforcement outside the model. Separate instructions from content in the application data model, label provenance, minimize context, and let deterministic code authorize tools and output destinations. Treat every generated value as untrusted: validate JSON against a strict schema, constrain enumerations and lengths, encode for the destination, parameterize database operations, and never execute generated shell, SQL, HTML, or URLs without independent controls.
Test corpus
Direct and multilingual override attempts.
Instructions embedded in HTML, PDFs, images, metadata, and tool output.
Requests to reveal system prompts, secrets, memory, or other tenants’ data.
Malformed structured output, dangerous URLs, command substitution, and stored payloads.
Multi-turn attacks that progressively alter state or exploit approval fatigue.
Accept only when: high-impact actions remain blocked by authorization and approval controls even when the model follows the attacker’s instruction.
Privacy, secrets, confidential data, and model leakage
Map personal data, credentials, source code, customer content, security evidence, and regulated records across prompts, files, logs, traces, caches, vector stores, fine-tuning datasets, provider retention, and support channels. “Do not train on my data” addresses only one part of the lifecycle. Define purpose, minimization, locality, retention, deletion, access, export, and incident obligations for every copy.
Use secret detection before model submission, but do not rely on redaction alone: transformed or encoded values can remain sensitive. Prefer short-lived scoped credentials delivered only at the tool boundary. Prevent the model from reading raw secret stores. Evaluate memorization and extraction risk for custom models; test cross-session and cross-tenant leakage; and ensure logs preserve security evidence without silently becoming a second sensitive-data lake.
Review questions
Which provider and region receive each class of content?
Can users delete conversations, uploaded sources, embeddings, and derived memory?
Who can access prompts and traces for support or monitoring?
What happens when content is quoted in an evaluation dataset or incident ticket?
Agents, tools, plugins, browsers, and MCP security
An agent turns model output into state change. Its risk is determined by reachable capability, credential scope, data visibility, autonomy, iteration count, and reversibility—not by the model’s marketing name. MCP standardizes discovery and invocation of tools and resources, but it does not automatically make a server, tool description, returned content, or authorization policy trustworthy.
Maintain an allowlist of server identities and pinned versions. Authenticate both sides where applicable. Validate tool schemas, constrain arguments, and authorize each invocation using the requesting user and task—not a universal agent credential. Separate read from write tools; isolate high-risk tools in containers or sandboxes; block arbitrary command execution and unrestricted egress; cap recursion, time, tokens, cost, and result size; and require explicit approval for destructive, external, or privilege-changing actions.
Agent transaction pattern
Planner proposes a bounded action with reason and expected effect.
Deterministic policy evaluates user, tenant, tool, arguments, data class, and risk.
Human approval is requested for material external effects.
Executor uses a scoped, short-lived credential in an isolated environment.
Result is treated as untrusted content and validated before reuse.
Audit record links request, policy, approval, execution, result, and rollback.
Identity, authorization, tenancy, and human approval
Authenticate the human or workload at the application boundary, propagate identity deliberately, and re-authorize at retrieval and action boundaries. Do not let the model infer permissions from natural-language claims. A user who may ask about a record does not necessarily have permission to retrieve its raw source, export it, or invoke an action against it.
Use tenant-aware object identifiers, row/document-level controls, explicit workspace membership, and service identities per environment. Protect system prompts and administrative configuration as privileged objects. Approval dialogs must state the exact action, target, data leaving the boundary, credential context, and rollback—not merely ask “Allow AI?”. Prevent replay and bind approval to immutable arguments and expiry.
Abuse cases
Cross-tenant retrieval through ambiguous entity names or cached context.
Confused-deputy action using a more privileged service account.
Privilege escalation by editing a tool description, prompt template, or routing rule.
Approval substitution where displayed arguments differ from executed arguments.
Infrastructure, network, resource, and availability security
Protect inference endpoints, gateways, notebooks, training jobs, vector stores, queues, object stores, GPU nodes, and management planes with ordinary hardening and segmentation. Private deployment does not remove supply-chain, insider, or tenant risks. Restrict network egress so model-mediated fetches cannot become SSRF, metadata-service access, internal discovery, or data exfiltration.
Availability threats include expensive prompts, unbounded context, decompression bombs, high-cardinality retrieval, recursive agents, adversarial inputs that trigger slow paths, provider quota exhaustion, and cascading retries. Enforce input and output limits, concurrency budgets, per-tenant quotas, circuit breakers, queue backpressure, timeouts, and cost alerts. Design a degraded mode that disables risky automation while preserving essential operations.
Baseline evidence
Network and identity policy for every AI service and data store.
Secret delivery, rotation, and emergency revocation test.
Capacity, quota, timeout, and cost-abuse test results.
Backup, restore, regional/provider failure, and kill-switch exercises.
AI red teaming, evaluation, and reproducible security testing
Security evaluation must be task-, architecture-, language-, and impact-specific. Separate model robustness tests from end-to-end security controls. Build a versioned corpus from threat models, production incidents, public techniques, internal policy, and expected user behavior. Include benign controls so the team measures over-blocking as well as bypass.
Record exact model/provider, temperature and relevant parameters, system prompt revision, retrieval snapshot, tool set, policy version, tester, time, attempts, and result. Because model behavior is probabilistic, repeat tests and report rates with sample size instead of presenting a single transcript as proof. Sanitize external test data and avoid sending protected content to unapproved services.
Assessment sequence
Confirm authorization, targets, data handling, stop conditions, and communications.
Map exposed functions and ordinary web/API vulnerabilities.
Test prompt, retrieval, output, tool, tenancy, privacy, and availability abuse cases.
Validate impact through safe canaries and reversible actions.
Retest controls, add regression cases, and document residual risk.
Logging, monitoring, detection, and security analytics
Log enough to reconstruct decisions without indiscriminately copying sensitive content. Useful events include authenticated actor and tenant, session/request ID, model and prompt-policy version, retrieval source IDs, policy decisions, tool proposals and calls, approval events, output validation, safety-control outcomes, latency, tokens, cost, errors, and data destination. Protect logs against tampering and apply strict access and retention.
Detect unusual tool sequences, new servers, denied-action spikes, cross-tenant lookup attempts, repeated extraction patterns, sudden output-volume or cost changes, sensitive-data indicators, retrieval of quarantined sources, prompt-template changes, and policy bypass. Correlate AI events with IAM, endpoint, cloud, proxy, and application telemetry. Behavioral alerts are investigation leads, not automatic proof of malicious intent.
Detection engineering loop
Define the abuse hypothesis and required telemetry.
Generate a safe positive case and ordinary negative cases.
Write, test, and version the analytic.
Measure visibility, precision, latency, and investigation cost.
Attach response actions and rerun after model or workflow changes.
Define AI-specific incident categories before an event: unauthorized disclosure, poisoned knowledge, compromised model or dependency, abusive automation, unauthorized tool action, cross-tenant leakage, integrity failure, availability/cost attack, and unsafe published output. Establish ownership across security, ML/AI engineering, application, privacy, legal, provider management, and business operations.
Preserve the exact prompt and policy versions, conversation and request IDs, retrieved source identities, model endpoint/version, tool arguments and results, approval record, relevant configuration, and surrounding cloud/application events. Avoid assuming a transcript alone explains the decision: orchestration, retries, hidden context, caches, and policy layers matter. Revoke credentials and disable tools independently from shutting down the entire service.
Response sequence
Declare severity from observed impact and reachable capability.
Contain identities, tools, connectors, sources, routes, or model versions.
Preserve evidence with handling and access controls.
Determine whether the failure was compromise, abuse, misconfiguration, drift, or expected model uncertainty.
Recover with fixed policy and regression evidence.
Notify affected owners and meet contractual or legal duties.
Governance, risk, assurance, transparency, and human factors
Maintain an inventory of AI use cases, owners, models, providers, data classes, affected users, decision authority, connected tools, deployment status, and review date. Classify risk from context and impact rather than model size alone. Shadow AI and embedded vendor features belong in the inventory when they process organizational data or influence decisions.
Document intended use, prohibited use, limitations, human oversight, user notice, appeal/escalation path, evaluation evidence, change triggers, incident history, and residual-risk acceptance. Claims such as “secure,” “private,” “compliant,” or “bias-free” require scoped evidence. Track provider and regulatory change, but consult qualified counsel for applicability; a field guide cannot determine legal obligations.
Assurance questions
Can an independent reviewer trace each material claim to evidence?
Are human reviewers trained, given time, and able to reject automation?
Do metrics include harmful misses, false blocks, near misses, and downstream impact?
Does a material model, data, prompt, tool, or purpose change trigger reassessment?
Secure AI development lifecycle and production release gate
Embed AI security into product discovery, architecture, data onboarding, implementation, evaluation, release, monitoring, and retirement. Maintain controls as code where appropriate, but keep accountable decisions visible. Every material change to model, provider, embedding, knowledge source, system prompt, tool, permission, or output destination can invalidate earlier evidence.
A production gate should combine ordinary application security with AI-specific checks. Require model and dependency provenance, data and privacy approval, threat model closure, retrieval authorization tests, prompt-injection and output-validation tests, tool-policy tests, tenancy tests, monitoring coverage, incident playbook, rollback, and explicit residual-risk acceptance. A successful benchmark or frontend build alone is not a release decision.
Minimum release gate
Named owner, approved use case, system map, and risk classification.
Pinned deployable configuration and rollback candidate.
No unresolved critical conventional or AI-specific abuse path.
High-impact actions protected by deterministic authorization and approval.
Evaluation report includes configuration, samples, rates, limitations, and negative controls.
Telemetry, alert routing, response playbook, and kill switch tested.
Change and periodic review dates recorded.
Release decision: approve only the tested configuration and scope. Record exceptions with owner, expiry, compensating control, and retest date.
Build a small document summarizer. Seed one document with an indirect instruction, enforce source labeling, validate structured output, and demonstrate that no model response can authorize a filesystem or network action.
Lab 2 — Tenant-safe RAG
Create two tenant corpora with canary records. Implement authorization before retrieval, run cross-tenant query variations, test deletion/reindexing, and prove canaries never enter unauthorized context.
Lab 3 — Least-privilege MCP
Expose one read-only and one state-changing tool. Pin server identity, validate arguments, require approval for the write, cap execution, record an audit chain, and test a malicious tool response.
Lab 4 — AI detection and response
Generate safe prompt-injection, enumeration, denied-tool, and cost-spike events. Correlate them with identity and application logs, investigate, disable a connector, preserve evidence, and rerun a regression test.
Every model, data source, connector, tool, and owner is inventoried.
Trust boundaries and high-impact abuse cases are documented.
Retrieval and actions use caller-aware authorization.
Model output is validated and encoded for its destination.
Secrets and sensitive content are minimized and governed across providers and logs.
Agent and MCP capabilities are allowlisted, scoped, bounded, and auditable.
Adversarial tests cover direct/indirect injection, tenancy, leakage, tools, and availability.
Monitoring can reconstruct a decision without exposing unnecessary content.
Containment, credential revocation, source quarantine, rollback, and kill switch are tested.
Residual risk is accepted by a named owner for the exact deployed configuration.
Authoritative frameworks and ecosystem reading
External primary references
NIST AI Risk Management Framework — governance vocabulary and lifecycle risk management; it is not a certification or a substitute for system-specific testing.
