AI Security Course, Module 00 — Part 1: Introduction, AI/ML Taxonomy, and Data Foundations

Before you secure an AI system, learn what the system is

AI security conversations often begin with prompt injection, jailbreaks, or a new red-team tool. That is understandable, but it creates a dangerous starting point.

An AI system is not just a model.

It is a chain of data pipelines, model artifacts, prompts, retrieval systems, applications, identities, tools, memory, infrastructure, operators, and downstream actions. The model may generate text or choose an action, but it does not automatically provide authorization, tenant isolation, provenance, auditability, or safe execution.

That is why the first module of the AI Security Engineering course is not an attack lab. It is a technical foundation module for security practitioners who need to reason accurately about the complete system.

This article is Part 1 of the chapter-by-chapter version of Module 00: AI, Machine Learning, and LLM Foundations. It covers the introduction, Chapter 1, and Chapter 2. Later articles will cover neural networks, Transformers, the LLM lifecycle, RAG, agents, and serving.

Course status: Under construction. The syllabus may change during creation. The scope, examples, references, labs, and assessment criteria may change before pilot delivery.

The purpose of Module 00

Module 00 creates a shared technical language for the rest of the course. It is not intended to turn security engineers into research scientists, and it does not assume advanced mathematics. It teaches the mechanisms that affect security decisions:

• how learning systems use data;
• how models are trained, adapted, evaluated, and served;
• how LLMs turn tokens into generated output;
• how RAG moves external data into model context;
• how agents connect model output to tools and authority;
• how deployment and observability determine the blast radius;
• how terminology affects threat modeling and incident reporting.

The objective is practical precision. A learner should be able to look at an AI architecture and answer:

1. What are the assets?
2. Which component has authority?
3. Which data crosses a trust boundary?
4. Which artifacts and configurations can change behavior?
5. What evidence would reconstruct a security-relevant action?

Audience, prerequisites, and skip paths

Module 00 is for security practitioners, AI platform engineers, MLOps engineers, threat-intelligence analysts, detection engineers, and technical risk owners who need a common operating vocabulary. Learners should be comfortable with basic security concepts such as identity, access control, logging, network boundaries, software dependencies, and incident evidence.

No advanced calculus, GPU programming, or model pretraining experience is required. Learners who already understand neural-network training may skim the optimization explanation in later parts, but should still complete the artifact, RAG, agent-authority, and observability traces. Learners who already operate LLM applications may skim the introductory definitions, but should not skip the security boundaries around retrieval, tool execution, caching, identity, and evidence.

The sequence is intentionally flexible. Instructors can deepen a topic, assign the glossary as reference, or use the skip paths without changing the learning contract. Every learner should still be able to explain the complete request path and produce the required artifacts.

1. AI, machine learning, deep learning, and generative AI are not synonyms

The first source of confusion is vocabulary. These terms describe overlapping scopes, not interchangeable products. The hierarchy below is useful as a map, but it is not a strict pipeline: some AI systems use no machine learning, some foundation models are not language models, and some generative systems are specialized rather than broadly reusable.

Artificial intelligence
├── symbolic AI: rules, search, planning, knowledge, optimization, robotics
└── Machine learning
    ├── supervised, unsupervised, self-supervised, and reinforcement learning
    ├── classical statistical and algorithmic ML
    └── Deep learning
        ├── discriminative models
        ├── encoder and representation models
        ├── embedding models
        ├── generative models
        └── foundation models
            ├── language models and LLMs
            ├── vision models
            └── multimodal models

The branches overlap. A foundation model may be generative, encoder-based, multimodal, or a combination. An embedding model may be a deep-learning model without being generative. Reinforcement learning is a learning paradigm that can use deep networks; it is not a sibling product category to “foundation model.” An LLM is a language model, and many current LLMs are language foundation models, but the terms are not interchangeable.

Artificial intelligence: the broadest category

Artificial intelligence (AI) is the broad field of machine-based systems that produce predictions, recommendations, decisions, or content for human-defined objectives. AI includes symbolic rules, search, planning, optimization, robotics, expert systems, statistical models, and neural networks.

Consider a rules-based transaction screening system:

if amount > approved_limit
and destination_country is restricted
and account_age < policy_threshold:
    require manual review

This is an AI-related decision system even though it does not learn parameters from data. Its security risks are still real: an attacker may manipulate input fields, bypass the policy path, abuse the review workflow, or compromise the service account. Calling it “not AI” would not make those risks disappear.

At the other end of the spectrum, DeepMind’s AlphaGo combined deep neural networks, search, and reinforcement learning to select moves. It is a useful reminder that an AI system may contain several techniques at once. The model is only one part of the decision loop; the search procedure, state, interfaces, and execution environment also matter.

Symbolic rules are explicit, human-authored logic: conditions, facts, policies, and actions written in a form a program can evaluate. A firewall rule, an allowlist, or “require review when a payment exceeds its limit” is symbolic behavior. It is usually easier to inspect and reproduce than learned behavior, but it can be brittle, incomplete, and vulnerable to input manipulation or rule-order mistakes. In a security investigation, preserve the rule version, evaluation order, input fields, and decision path; there are no learned weights to inspect, but the policy implementation is still a security-critical artifact.

Machine learning: behavior learned from data or experience

Machine learning (ML) uses data or interaction to learn a relationship, representation, policy, or decision instead of expressing all behavior as hand-written rules. Typical tasks include classification, regression, ranking, clustering, anomaly detection, and control.

Real-world examples include:

• an email system classifying messages as spam or not spam;
• a payment system estimating fraud risk;
• a search engine ranking results;
• a security platform grouping infrastructure by behavioral similarity;
• a forecasting model estimating demand or incident volume.

Google’s classification guide is a useful reference for the basic vocabulary of labels, predictions, thresholds, false positives, and false negatives.

The security boundary is the data and decision workflow around the model. A phishing classifier can be accurate and still be unsafe if an attacker poisons its training data, manipulates features, extracts sensitive examples, or causes an operator to treat a probability score as an authorization decision. The score is evidence for a policy; it is not the policy itself.

Deep learning: multilayer representation learning

Deep learning is ML based on neural networks with multiple layers that learn representations and functions. A deep model can identify patterns in images, audio, text, code, sensor data, or multimodal inputs.

The model may learn features that were not explicitly designed by an engineer. That is powerful, but it makes reasoning about provenance and failure more important. Security questions include:

• Which data created the representation?
• Which examples were overrepresented or mislabeled?
• What happens under distribution shift?
• Can an attacker craft an input that changes the prediction?
• Can a model artifact or dependency execute code before inference?
• What evidence shows which checkpoint and configuration produced the output?

The AlexNet paper is a landmark example of deep convolutional learning for image classification. The Transformer architecture later introduced attention-based processing that became central to modern language and multimodal models. These are technical milestones; the security properties still depend on data, deployment, identity, and controls.

Generative AI: producing new content

Generative AI produces new text, code, images, audio, video, or structured content. Generation may use autoregressive Transformers, diffusion models, generative adversarial networks, flow-based models, or other architectures.

Examples include:

• an LLM drafting a report or generating code;
• an image model creating a synthetic image from a text prompt;
• a speech model generating audio;
• a coding assistant proposing a patch;
• an agent generating a structured API call.

The output is not automatically an answer, a fact, or an authorized action. It is a model-produced artifact that must be validated in the context where it will be used. An output rendered as HTML has a different risk than an output shown as plain text. An output passed to a ticketing API has a different risk than an output read by an analyst.

OpenAI’s GPT-4 research report illustrates this distinction well: it describes a large multimodal model, its evaluations, limitations, and deployment considerations. Evaluation results describe measured behavior under defined conditions; they do not replace application authorization, identity controls, or production monitoring.

For image generation, the DDPM paper is a primary reference for diffusion-model foundations. The security lesson is not that every generative model has the same vulnerability. It is that each generated output becomes part of a downstream data and decision path.

Foundation models: broadly reusable starting points

A foundation model is trained on broad data and designed to support multiple downstream tasks or applications through prompting, adaptation, fine-tuning, retrieval, or additional system components. The concept is explained by Stanford’s What are Foundation Models? and the Bommasani et al. foundation-model report.

Foundation models create a concentration of dependency and supply-chain risk. One base model may be:

• fine-tuned by many teams;
• wrapped by many applications;
• downloaded from a public registry;
• combined with different adapters and tokenizers;
• connected to different retrieval systems and tools;
• deployed under identities with very different authority.

The same checkpoint can therefore be low-risk in an isolated research notebook and high-risk inside an agent with access to private documents, cloud APIs, or production systems. “The model is safe” is incomplete unless the model version, wrapper, data, identity, tools, and deployment are specified.

Large language models: token prediction at scale

An LLM is a large language model, usually based on a Transformer architecture, that predicts a probability distribution over tokens and generates sequences by repeatedly selecting the next token. It may support summarization, translation, classification, code generation, question answering, reasoning-like workflows, or tool selection.

The original Attention Is All You Need paper introduced the Transformer architecture that underlies many current language and multimodal systems. The GPT-4 research report linked above is a historical example of how a provider describes capabilities, evaluation, limitations, and system-level safety work; the course uses capability-based language so the lessons remain applicable to current model generations.

An LLM does not inherently provide:

• current or complete knowledge;
• truth verification;
• tenant authorization;
• secret management;
• stable identity;
• transactional integrity;
• safe tool execution;
• deterministic policy enforcement.

Those properties come from the application and operating environment.

Real-life examples: the label changes the security question

Security cases that make the distinction concrete

The course uses real reports to connect terminology to operational risk:

1. ShadowRay:

   

   Oligo reported active exploitation of exposed Ray AI infrastructure. The initial technique was conventional control-plane abuse, but the compromised environment contained AI workloads, models, datasets, credentials, and expensive compute. Read the ShadowRay report. The lesson is that AI security includes the platform around the model.
2. Malicious model artifacts:

   

   JFrog documented public ML artifacts whose loading could execute embedded code. Read JFrog’s model-supply-chain research. The lesson is that a model file may be an executable supply-chain input, not inert data.
3. EchoLeak:

   

   Microsoft’s advisory for CVE-2025-32711 documents a production vulnerability involving an enterprise AI assistant. The course treats it as a disclosed vulnerability and reproduced chain, not automatically as a criminal campaign. The lesson is that retrieved content, instructions, rendering, and outbound paths can combine into impact.
4. MCP tool poisoning:

   

   Invariant Labs demonstrated tool-poisoning attacks in which tool metadata could influence an agent’s planning context. The lesson is that tool definitions, approval state, delegated identity, and application authorization are security boundaries.

A practical test for terminology

When a report says “AI attack,” ask six questions before accepting the phrase:

1. Is the target a model, a data pipeline, an application, an agent, an identity, or infrastructure?
2. Is AI the target, the delivery mechanism, the enabling tool, or simply part of the environment?
3. Did the report establish feasibility, exposure, provider-observed activity, exploitation, or harm?
4. Which model, artifact, prompt, retrieval set, tool definition, identity, and runtime were involved?
5. What state or authority changed after the model produced its output?
6. Which deterministic control could have prevented, constrained, detected, or preserved the action?

If the report cannot answer these questions, it may still be useful as a lead, but it is not yet a complete threat model.

References for Chapter 1

• NIST AI 100-3: The Language of Trustworthy AI
• Google Machine Learning Crash Course: Classification
• DeepMind: AlphaGo
• Krizhevsky, Sutskever, and Hinton: ImageNet Classification with Deep Convolutional Neural Networks
• Bommasani et al.: On the Opportunities and Risks of Foundation Models
• Oligo: ShadowRay
• JFrog: Malicious Hugging Face ML Models
• Microsoft MSRC: CVE-2025-32711
• Invariant Labs: MCP Tool Poisoning Attacks

2. How learning systems use data

Security engineers need to understand where data enters the lifecycle and what happens to it.

collect and document data
  → clean, transform, label, and split
  → select architecture and objective
  → initialize parameters
  → train on batches
  → validate and tune hyperparameters
  → evaluate on held-out conditions
  → release a specific artifact
  → monitor inputs, behavior, outcomes, cost, and drift
  → retrain, roll back, replace, or retire

The key terms are different security objects. They have different owners, provenance, integrity controls, retention rules, and failure modes. Treating them all as “the model” makes an incident difficult to reproduce and can cause the wrong control to be applied.

Data

Data is any input used to learn, evaluate, retrieve, or generate. In a phishing classifier, the raw message, headers, URL, and analyst disposition may all be data. In a RAG application, the source documents, document chunks, user question, retrieved passages, and conversation history are different data sets. In an LLM service, the prompt sent by a user is production input data even though it is not part of model training.

The security question is not only “is the data accurate?” It is also “who could write it, read it, retain it, or cause it to cross a tenant boundary?” Training data can be poisoned; retrieval data can contain hidden instructions; prompts can contain secrets; logs can create a second copy of sensitive data. Preserve the source, owner, collection time, access policy, license or consent basis, content hash, split membership, and retention decision. NIST’s AI Risk Management Framework treats data quality, provenance, privacy, and monitoring as lifecycle concerns rather than one-time preprocessing tasks.

Features

Features are the representations consumed by a task or model. A classical phishing detector might use the sender-domain age, number of URLs, character n-grams, and authentication results. An image model receives normalized pixel tensors. An LLM receives token IDs and attention masks. A RAG system may convert a document into an embedding vector before placing it in a search index.

Features may be engineered by a person, extracted by a parser, or learned by earlier neural-network layers. That makes feature extraction a security boundary. An attacker can manipulate a URL spelling, image pixels, document chunking, tokenization, or embedding index to alter the representation while leaving the apparent business object unchanged. A train-serving mismatch can also make a production classifier behave differently from the evaluated one. Record the feature-extraction code and version, normalization rules, tokenizer or embedding model, dimensionality, missing-value behavior, and validation checks. Never assume that a feature is trustworthy merely because it was produced by an internal pipeline.

Labels

Labels identify the target a supervised system is expected to predict. Examples include phishing versus benign, malware family names, a named entity span, a severity category, or a preferred response in a human-feedback data set. A label is a recorded judgment under a policy; it is not automatically ground truth.

Labels can be supplied by analysts, vendors, heuristics, weak-labeling rules, user feedback, or another model. Security failures include poisoning the labeling queue, changing the label definition without versioning it, leaking the answer into a feature, and allowing a feedback loop to teach the system that its previous mistakes were correct. Preserve the label policy, annotator or source, timestamp, confidence, disagreement, adjudication notes, and policy version. For a phishing system, “malicious” might mean a confirmed payload, a suspicious campaign, or merely a message that violated a company rule; those are different targets and produce different controls.

Parameters

Parameters are values learned during optimization: neural-network weights and biases, embedding tables, normalization statistics, and, in an adapted LLM, adapter or low-rank update weights. A small email classifier may have thousands of parameters; a foundation model checkpoint may have billions. Parameters encode behavior, but they are still an artifact that must be identified and protected like a compiled binary.

The security boundary includes the model file, serialization format, loader, registry, and every derived checkpoint. An unexpected output can result from a tampered checkpoint, an unapproved adapter, a quantized copy, or a loader that interprets the file unsafely. JFrog’s research on malicious Hugging Face models is a useful reminder that model artifacts can be supply-chain inputs rather than inert data. Preserve a cryptographic digest, signature or attestation, source revision, parent checkpoint, adapter list, format, approved loader, and promotion history.

Hyperparameters

Hyperparameters are selected configuration values rather than values learned from the training examples. They include learning rate, batch size, number of epochs, optimizer, architecture depth, regularization, retrieval top-k, chunk size, and early-stopping policy. In a deployed generative system, decoding controls such as temperature, top-p, beam width, repetition penalty, and maximum output length are also configuration that changes behavior. A decision threshold is often a policy configuration rather than a model hyperparameter, but it belongs in the same change record because it changes the security outcome.

For example, the same phishing classifier can quarantine messages at a score of 0.5 or send them to review at 0.9; the weights did not change, but the operational risk did. Likewise, changing an LLM’s temperature may make a response less repeatable, while changing retrieval top-k may introduce an untrusted document into context. Record hyperparameters beside the model digest, not in an undocumented environment variable. Configuration drift, an overly permissive threshold, or a changed prompt template can defeat a control without any weight-level compromise.

Training

Training updates parameters using data, an objective, and an optimization procedure. Pre-training may learn next-token prediction from a large corpus. Supervised fine-tuning may teach a support assistant to follow examples. Preference optimization or reinforcement learning may change how the system responds to ranked outcomes. The training job is therefore a security-sensitive build process, not just a long computation.

For a security assistant, the runner may access incident reports, credentials, source code, package registries, and model checkpoints. Threats include poisoned data, compromised dependencies, unauthorized experiment code, secret leakage in logs, stolen checkpoints, and an attacker using training compute for another workload. Preserve the source-code revision, dependency lockfile, data manifest, objective, runner identity, permissions, seeds, logs, checkpoint lineage, and promotion decision. A training run that cannot be reproduced cannot be confidently investigated.

Validation

Validation is the development feedback loop used to choose models and configuration. Teams may use a validation set to select an alert threshold, choose an early-stopping checkpoint, compare prompt templates, or tune a retriever. Validation is not a final claim of generalization; repeated decisions against the same set gradually make that set part of the development process.

Consider a fraud model whose threshold is tuned until the validation false-positive rate looks acceptable. If analysts repeatedly adjust the threshold against that same set, the reported number becomes optimistic. For an LLM, repeatedly editing a system prompt until a known jailbreak suite passes can similarly overfit the suite. Protect the validation set from unauthorized edits, record each decision and rationale, and watch for duplicate or near-duplicate examples crossing between training and validation. Google’s discussion of generalization and overfitting explains why performance on development data can stop predicting behavior on new inputs.

Testing

Testing estimates behavior on held-out conditions that were not used to fit parameters or make routine configuration decisions. A useful phishing test set may be a later time period, a new campaign, or a separate organization. A RAG test may ask whether a user from tenant A can retrieve tenant B’s document, not merely whether the answer sounds fluent. Security testing can include adversarial examples, prompt-injection cases, authorization checks, abuse-rate tests, and failure-injection scenarios.

Testing can still mislead when the set is contaminated, too small, selected after seeing failures, or unlike production. “99% on the test set” does not establish safety under distribution shift, a new attacker, a different language, or a changed dependency. Preserve the exact test manifest, generation or sampling method, environment, model and configuration digests, expected outcomes, observed outputs, evaluator version, and exceptions. Keep final test evidence separate from the validation loop so a release decision remains auditable.

Inference

Inference is the execution of a released artifact to produce a prediction, score, embedding, classification, or generation. Examples include classifying an incoming email, generating an answer from retrieved policy documents, or proposing a tool call for an agent. Inference is where untrusted input meets model behavior and where an output may cross into a consequential system.

The model is only one part of the inference boundary. Record the caller identity and tenant, model and adapter digests, tokenizer, prompt or feature template, retrieved document IDs and authorization result, tool definitions, guardrail decisions, output, latency, and any side effect. A prompt injection can change the context; a retrieval bug can supply data from the wrong tenant; a route change can silently select a different model; an output parser can turn text into an unauthorized API call. The security decision must therefore be enforced by deterministic authorization and validation around inference, not inferred from the model’s confidence or wording.

The same bytes can have different security roles

The same text can be a different security object at different points in the lifecycle:

1. A policy document included in supervised fine-tuning is training data.
2. The same document indexed for search is retrieval data.
3. A copy placed in a user prompt is inference input.
4. The document’s embedding is a feature for retrieval.
5. A human’s “answer is grounded” judgment can be a label.
6. The influence learned from many documents is encoded in parameters.

The bytes may be identical, but the owner, access rule, retention period, audit trail, and threat model are not. During incident response, identify the role of each copy before deciding whether it should be deleted, quarantined, reindexed, retrained on, or disclosed.

These distinctions matter during investigations. An unexpected result may come from a changed dataset, a new adapter, a different tokenizer, a prompt-template update, a retrieval change, a runtime upgrade, or a decoding configuration—not only from a changed base model. The NIST AI 100-3 terminology report and the original Retrieval-Augmented Generation paper provide useful technical vocabulary for tracing those boundaries.

Learning paradigms

Module 00 covers the main learning paradigms at a practitioner level:

• Supervised learning: input-target pairs, such as malicious/benign classification.
• Unsupervised learning: structure discovery without explicit labels, such as clustering.
• Self-supervised learning: targets are derived from the data, such as next-token prediction.
• Semi-supervised learning: a smaller labeled set is combined with a larger unlabeled set.
• Reinforcement learning: a policy learns from states, actions, rewards, and returns.
• Transfer learning: representations or parameters learned in one setting are adapted to another.

Each paradigm creates different attack and evidence questions. Poisoning a training set is not the same as manipulating a retrieval index. A reward-model failure is not the same as a prompt injection. A model can perform well on a benchmark and still leak data or fail under distribution shift.

Worked example: a phishing classifier

Suppose a security team builds a binary classifier that labels incoming messages as benign or phishing. The security analysis must include more than the algorithm:

message and metadata
  → parser and feature extraction
  → classifier score
  → threshold and policy
  → quarantine, delivery, or analyst review
  → analyst feedback and future training data

The threat model includes poisoned feedback, attacker-controlled features, evasion through formatting or language changes, threshold manipulation, analyst over-trust, and leakage from logs or training examples. A false negative can deliver a malicious message; a false positive can block a critical business message and teach operators to ignore alerts.

The evaluation record should preserve the dataset population, class balance, split method, threshold, precision, recall, false-positive and false-negative counts, calibration, drift assumptions, and known blind spots. “The model is 98% accurate” is not enough to choose a quarantine policy.

Security questions across the learning lifecycle

This is why ML literacy is a security requirement: the evidence needed to explain a model decision is created throughout the lifecycle, not only at inference.

Key takeaways

• AI is a broad category that includes symbolic rules, search, optimization, classical ML, deep learning, and generative systems.
• Symbolic rules are explicit logic; learned models derive behavior from data or interaction. Both are security-critical and require versioned evidence.
• “LLM” describes a language model, not an authorization system, data-governance system, or safe execution environment.
• Data, features, labels, parameters, hyperparameters, training, validation, testing, and inference are distinct security objects.
• A model score or generated output is evidence for a policy decision, not the policy itself.
• During incident response, identify the exact artifact, configuration, input, context, identity, and downstream action before claiming that “the AI” caused an outcome.

What comes next

Chapter 2 — How Learning Systems Use Data continues with data, features, labels, model artifacts, configuration, evaluation, inference, and CTI evidence. Later parts will trace neural networks, tokens and Transformer context assembly, the LLM lifecycle, RAG authorization, agent and tool authority, and serving observability.