AI Security Threat Landscape

Learning outcomes

Learning path

Block 1

1. Model the complete AI system

The six-layer working model

Follow one decision end to end

User request
  → authenticated application session
  → retrieval query with tenant and document authorization
  → retrieved chunks with provenance
  → prompt construction
  → model response
  → structured output validation
  → policy check
  → tool authorization
  → human approval for material action
  → execution with a scoped workload identity
  → audit, egress, and outcome telemetry

The important question is not simply “can the model be manipulated?” It is “what security-relevant state can change after manipulation, and which deterministic control prevents or records that change?”

Block 2

2. Evaluate threat evidence before making claims

The five-pass evidence method

1. Define the decision. Name the consumer, system, time horizon, and decision the analysis must support.
2. Establish provenance. Identify who observed the event, who reproduced it, who owns the affected product, and whether later articles merely repeat the same source.
3. Extract facts before judgments. Record directly supported actions, assets, dates, versions, preconditions, and outcomes. Keep vendor language and analyst interpretation separate.
4. Test alternatives and gaps. Ask what other explanation fits, what visibility the source lacks, and what evidence would change the conclusion.
5. State a bounded judgment. Use calibrated confidence, explain why it matters to the named system, and recommend an action proportionate to the evidence.

Evidence discipline

CTI evidence method for AI security

Cyber threat intelligence is useful here only when it changes a defensible security decision. A headline, framework label, or model-generated summary is not intelligence by itself. The analyst must connect a named consumer and decision to sourced facts, confidence, information gaps, observable behaviors, and a control or collection action.

1. Start with an intelligence requirement

An intelligence requirement is a question whose answer supports a decision. “Research prompt injection” is a topic. “Should Northbridge block unapproved MCP servers before enabling ticket creation?” is a decision requirement. A good requirement identifies the consumer, decision, system boundary, relevant conditions, and confidence needed for action.

2. Build a source and provenance record

Prefer the closest available source to the event or reproduction. A vendor advisory may establish affected versions and remediation but not exploitation. A researcher’s proof may establish feasibility but not victim impact. A provider misuse report may establish activity within the provider’s service but not the complete operation. Secondary reporting is useful for discovery and corroboration, not a substitute for the primary record.

3. Keep a claim ledger

Every important sentence in an assessment should be classifiable as a fact, judgment, assumption, information gap, or recommendation. Mixing these categories makes a report sound confident while hiding unsupported transitions.

4. Use calibrated confidence

Confidence is not a feeling and a CVSS score is not confidence. Explain why the evidence supports the judgment and what would change it. The course uses high, moderate, and low confidence as communication labels, each accompanied by an evidence rationale.

• High confidence: multiple independent, specific, first-hand or reproducible sources support the same bounded claim.
• Moderate confidence: the claim is plausible and supported, but visibility, corroboration, configuration, or outcome evidence is incomplete.
• Low confidence: the claim depends mainly on indirect reporting, untested assumptions, contested attribution, or a forecast.

5. Convert evidence into collection and controls

AI-specific evidence traps

Block 3

3. ShadowRay: AI infrastructure exploited

Internet-reachable Ray dashboard / Jobs API
  → unauthenticated job submission
  → arbitrary workload execution in the cluster
  → cryptominer or reverse shell
  → access to environment variables, credentials, models, and datasets
  → possible cloud, Kubernetes, registry, and data-store expansion

Why this is AI security

The entry technique is conventional infrastructure abuse, but the compromised environment concentrates AI-specific value: expensive accelerators, model weights, private datasets, experiment systems, model-hub tokens, and inference or training workflows. A complete AI-security program must defend this layer before prompt-level controls matter.

Block 4

4. Malicious model artifacts: the model is part of the supply chain

Public model repository
  → developer or pipeline downloads mutable artifact
  → loader trusts unsafe serialization or remote custom code
  → payload executes during deserialization
  → workstation or MLOps runner is compromised
  → tokens, datasets, registries, CI/CD, or cloud become reachable

Model-admission control

1. Allow only approved registries, owners, licenses, revisions, and cryptographic hashes.
2. Prefer formats that separate tensor data from executable object reconstruction.
3. Disable remote custom code and unsafe loaders by default.
4. Scan statically, but treat scanner results as hypotheses rather than proof.
5. Inspect and load untrusted artifacts only in an isolated, disposable environment with no secrets and denied egress.
6. Record provenance, analysis results, approver, intended use, and immutable release identity.
7. Run training and inference with scoped identities and protected access to datasets and registries.

Block 5

5. Chained LLM and agent failures

untrusted content
→ enterprise retrieval
→ instruction/data confusion
→ filter bypass
→ sensitive context in output
→ rendering / egress
→ demonstrated exfiltration

untrusted tool metadata
→ model instruction context
→ manipulated selection or arguments
→ action under delegated identity
→ data or state change

Why a model-only mitigation is incomplete

Block 6

6. Provider-observed adversarial use of generative AI

OpenAI, Google, Anthropic, and other providers publish disruption and threat reports based on activity visible within their services. This visibility can establish how particular accounts used the provider’s models, but it may not reveal the complete intrusion, final delivery, off-platform infrastructure, victim impact, or how representative the observed accounts are.

Analyze the underlying workflow

Additional exposure case: DeepSeek

Wiz reported an internet-accessible ClickHouse database associated with DeepSeek that allowed unauthenticated queries and contained chat history, secrets, and operational data. The case establishes a material exposure and illustrates the concentration of sensitive data around AI services. Discovery alone does not prove that a malicious actor accessed or exfiltrated the data before remediation.

Cross-case reasoning

What the cases collectively teach

These cases are not six versions of the same “AI attack.” They span infrastructure compromise, executable model artifacts, retrieval and rendering chains, agent metadata, provider visibility, and ordinary data exposure. The common security question is where untrusted input meets authority, state, or sensitive data—and whether a deterministic control records or limits that transition.

Block 7

7. Guided practical: evidence to architecture

Your role and decision

You are the AI product-security review team. Your consumer is the CISO and AI platform owner. Your decision question is:

Available evidence

• the six primary case reports linked in this module;
• the Northbridge architecture description above;
• known planned connections: private research documents, hosted model API, MCP ticket tool, public model hub, Ray cluster, cloud object storage;
• known gaps: deployment topology, identity scopes, egress policy, tool-definition lifecycle, logging coverage, and model-admission procedure.

Work sequence

Case and primary source:
Evidence status:
Directly established facts:
Attacker prerequisite or exposure condition:
Entry point:
Affected asset:
Trust-boundary failure:
Observed or demonstrated action:
Impact:
Known mitigation:
Known exploitation status:
Important limitation:
Organizational relevance:
Information that would change confidence:

Quality gate

• Every claim names its evidence status.
• Every framework entry maps a factual procedure, not a headline.
• Every recommended control has an owner and verification signal.
• At least one control reduces authority or blast radius.
• At least one evidence gap becomes a collection requirement.
• The analysis includes one tempting but unsupported claim that the team rejected.

Block 8

8. Use frameworks for the right job

Mapping rule

Write the factual procedure first. Then choose the framework entry that best describes it. Record the source and evidence status beside the mapping. If the procedure cannot be written without adding assumptions, do not map it as observed.

Weak:   ShadowRay → "AI attack" → ATLAS

Strong: An internet-reachable Ray Jobs API accepted unauthenticated job
        submissions that executed workloads; researchers reported miners
        and reverse shells in compromised clusters.
        → conventional public-service and execution behaviors
        → AI infrastructure context
        → reported in-the-wild exploitation
        → source and limitations recorded

9. Knowledge check

Answer before opening each explanation. Target score: 8/10.

Block 9

10. Final assessment

Submit a maximum 700-word technical threat assessment and deliver a concise briefing for engineering and security stakeholders. Identify the most relevant attack surfaces, separate observed activity from demonstrated capability, explain evidence gaps, and recommend controls and telemetry that later modules will validate.

Required submission

• AI-system attack-surface map
• Real-case classification matrix for all six cases
• Three complete case reconstructions with different evidence statuses
• Procedure-first framework mapping
• Prevent, constrain, detect, and preserve control matrix
• Maximum 700-word organizational threat assessment
• Concise technical briefing

Assessment rubric