AI Security Engineering · Module 01
Instructor Guide
Module: AI Security Threat Landscape Status: Draft for pilot
Before delivery
- Revalidate all primary sources and evidence statuses against the research cutoff.
- Select the individual or team format and provide the learner workbook.
- Confirm the Northbridge Health scenario and facilitator inject fit the audience.
- Prepare a six-layer architecture diagram and evidence-status badges.
- Do not reveal knowledge-check explanations or expected conclusions before the practical.
- Do not execute malicious model artifacts or test public AI services.
Evidence review protocol
Run every case through the same sequence. This keeps the lesson focused on AI security engineering while giving CTI a precise role in preventing overclaiming.
- Frame the decision: name the consumer, system, threat path, and decision the case must inform.
- Record provenance: identify the source owner, first-hand or reproduced access, affected version/configuration, and independent corroboration.
- Extract facts: write the attacker prerequisite, entry point, identity, action, affected asset, state change, and outcome.
- Mark the boundary: classify the case and write one sentence the evidence does not support.
- Derive action: assign preventive, authority-reducing, detective, and preservation controls with owners and verification signals.
- State uncertainty: identify visibility limits, assumptions, information gaps, confidence, and the collection that would change the judgment.
| Case type | Teach as | Do not teach as |
|---|---|---|
| In-the-wild incident | Observed exploitation or harm bounded by the report | Proof that every similar asset was compromised |
| Provider-observed misuse | Activity visible within a provider service | Complete victim-side campaign evidence |
| Production vulnerability | Reproducible weakness requiring remediation and validation | Automatic proof of criminal exploitation |
| Malicious artifact | Supply-chain behavior and admission risk | Proof that every downloader became a victim |
| Demonstrated research | Feasibility under tested conditions | Evidence of widespread operational use |
Run of show
| Block | Teach | Facilitate | Evidence of learning |
|---|---|---|---|
| 1 | Six-layer system and on/through/around lenses | Map one familiar AI workflow | Learners identify authority and trust boundaries |
| 2 | Evidence statuses and five-pass method | Three-claim classification check | Wording matches evidence |
| 3 | ShadowRay chain | Ask why conventional infrastructure abuse is AI security | Platform controls and telemetry identified |
| 4 | Malicious model artifacts | Build model-admission record | Learners distinguish artifact loading from inference |
| 5 | EchoLeak and MCP tool poisoning | Identify three independent chain breaks | Controls extend beyond prompt filtering |
| 6 | Provider CTI and DeepSeek exposure | Compare visibility and uplift claims | Provider observation is properly bounded |
| 7 | Northbridge guided practical | Issue the facilitator inject after the initial reconstruction | Teams update confidence and controls |
| 8 | Procedure-first framework mapping | Challenge unsupported mappings | Source and status accompany every mapping |
| 9 | Briefings and close | Concise delivery plus feedback | Decision-oriented recommendation and limitation |
Case teaching notes
ShadowRay
Start with the deployment assumption: Ray expected a trusted network, while the observed deployments exposed an execution-capable control plane. Expected conclusion: the first failure is platform exposure and missing boundary enforcement; model and dataset risk follows compromise.
Malicious model artifacts
Keep the distinction between tensor content, serialized objects, and remote custom code. Expected conclusion: model admission requires provenance, immutable identity, format policy, scanning, isolated loading, and least-privilege runners.
EchoLeak
Draw every independent boundary in the chain. Expected conclusion: no single prompt or classifier is sufficient; authorization, rendering, context minimization, egress, and forensic lineage all matter.
MCP tool poisoning
Emphasize that metadata can enter model context before tool invocation. Expected conclusion: pin and display complete definitions, bind approval to immutable parameters, reapprove changes, isolate tools, and log definitions and policy decisions.
Provider reporting
Ask what the provider can and cannot see. Expected conclusion: AI often changes speed, scale, language, personalization, or integration while underlying intrusion behavior remains recognizable.
DeepSeek exposure
Expected conclusion: the disclosure establishes exposure and sensitivity, but absent additional evidence it does not establish malicious access or exfiltration.
Practical facilitation
- Assign teams of two to four and name the CISO and platform owner as consumers.
- Require three cases with different evidence statuses.
- During framework mapping, reject any entry that lacks a written factual procedure.
- After the initial case reconstruction, issue the Ray/MCP/retrieval-logging inject from the learner page.
- During control design, require one control owner and verification signal per threat path.
- Before the briefing, require the unsupported claim the team deliberately rejected.
Prompt halfway through: Which sentence becomes false if “demonstrated” is silently replaced by “observed in the wild”?
Assessment rubric
| Criterion | Weight | Pass signal |
|---|---|---|
| System and trust boundaries | 20% | Six layers, identities, data flows, and authority are explicit |
| Technical reconstruction | 20% | Prerequisite, entry, action, asset, impact, and limitation are accurate |
| Evidence status | 15% | Observed, disclosed, demonstrated, and unknown remain distinct |
| Framework mapping | 10% | Procedure-first, source-backed, and appropriately scoped |
| Controls | 15% | Preventive, blast-radius, and detective controls extend outside the model |
| Telemetry | 10% | Signals support both detection and causal reconstruction |
| Communication | 10% | Concise, calibrated, and tied to the consumer’s decision |
Pass threshold: 70%, with no automatic revision trigger.
Automatic revision triggers
- Every case is described as a confirmed incident.
- A PoC becomes an active campaign or provider visibility becomes global visibility.
- A CVSS score or framework entry is treated as proof of organizational risk.
- The solution relies only on prompting, filters, or an “AI firewall.”
- Identity, infrastructure, supply chain, authority, or egress are omitted.
- Lack of public exploitation evidence is treated as proof exploitation never occurred.
Closing statement
AI security is operationally real, but its evidence is uneven. Infrastructure compromise and malicious artifacts have been observed. Providers see adversarial AI use. Production copilots and agent protocols have serious demonstrated weaknesses. Defenders must act on all of these without pretending they are the same kind of evidence.