Neural Networks and Optimization

Why the mechanism matters to security

Security teams do not need to become research mathematicians, but they do need a causal model of what changes when a prediction changes. A neural network is not a mysterious “AI brain.” It is a parameterized computation, trained with an objective, evaluated under particular conditions, and released with configuration and dependencies.

That distinction is essential in incident response. A changed output may result from a modified input, parser, tokenizer, checkpoint, adapter, threshold, random seed, library, or serving route. The first question is not “was the model attacked?” It is “which observable component changed, and what evidence connects that change to the output?”

input tensor
  → layers and activations
  → output logits or values
  → loss against a target (training only)
  → gradients through backpropagation (training only)
  → optimizer update to parameters (training only)
  → released artifact + configuration
  → inference output and downstream policy

Course rule: describe the computation and the evidence chain before naming an attack. “Adversarial” is a claim that needs a threat model, a reproducible condition, and an evidence level.

1. Neural-network security objects

A neural-network system has more security objects than its weight file. The architecture describes the computation graph; parameters are learned weights and biases; activations are intermediate values for one input; and gradients describe how an objective changes with respect to parameters or inputs.

For a malware classifier, the same executable can be represented as bytes, an n-gram vector, an API-call sequence, or an embedding. For a language model, the same sentence can become different token IDs under a tokenizer revision. Security analysis must identify the representation actually used.

2. The forward pass

The forward pass applies the network to an input. In a simple layer, the model computes a weighted sum, adds a bias, and applies an activation function:

z = W · x + b
a = activation(z)
output = f(a)

Many layers compose this pattern. Convolutional networks learn local spatial filters; recurrent networks maintain a state over sequences; Transformer blocks use attention and feed-forward transformations. The security principle is the same: the output depends on the input, the ordered operations, and the exact parameter/configuration state.

During an investigation, capture a minimal request fixture and rerun it with the approved artifact. Compare preprocessing output, intermediate shapes, logits, post-processing, and final policy. This separates “the network produced a different score” from “the application sent a different tensor.”

3. Loss functions and objectives

A loss function turns a prediction and target into a number that the training process tries to minimize. Cross-entropy is common for classification; mean-squared error is common for regression; ranking and contrastive objectives are common for retrieval and representation learning.

The objective encodes what “better” means. Class weights, focal loss, label smoothing, reward models, safety penalties, and data filtering can all move the system toward different behavior. Security questions therefore include: who chose the objective, which examples had the greatest influence, and were safety constraints measured separately from task performance?

4. Gradients and backpropagation

Backpropagation applies the chain rule to compute how the loss changes with respect to each parameter. A gradient is not an attacker by itself; it is information about local sensitivity. In a white-box evasion test, an authorized evaluator may use input gradients to search for a small change that alters the prediction.

forward:  x → activations → loss
backward: loss → gradients for each operation
update:   parameter ← parameter − learning_rate × gradient

Gradients can also support diagnostics. Exploding gradients, vanishing gradients, saturated activations, or unstable validation loss may indicate an implementation or data problem. Preserve training logs and representative fixtures so a later analyst can tell a numerical failure from intentional manipulation.

5. Optimization in practice

An optimizer chooses how parameter updates use gradients. Stochastic gradient descent (SGD) uses batches of examples; momentum smooths updates; Adam adapts step sizes using running statistics. The learning rate, batch size, number of epochs, schedule, weight decay, initialization, and random seed can materially change the resulting artifact.

Optimization is a build process. Treat the training runner, dependencies, data manifest, configuration, logs, and checkpoint as a software supply chain. A model digest without its build context cannot fully explain behavior.

6. Generalization and regularization

Generalization is performance on conditions not used to fit or repeatedly tune the model. Overfitting occurs when the model learns training-specific patterns that do not transfer. Underfitting means the model has not captured enough useful structure. Both are security-relevant: overfitting can hide a backdoor or leak memorized data, while underfitting can create an evasion gap.

Regularization methods—weight decay, dropout, data augmentation, label smoothing, early stopping, and simpler architectures—can improve generalization, but they are not a security guarantee. Evaluate rare classes, campaign-held-out samples, temporal drift, and authorized adversarial transformations separately.

7. Evaluation, leakage, and reproducibility

Training fits parameters. Validation supports configuration choices. Testing estimates behavior on held-out conditions. Production monitoring checks whether those conditions still apply. Reusing a test set during tuning turns it into validation data and weakens the claim.

For a phishing detector, random splitting can place near-duplicate messages from the same campaign in both train and test. A time-based, sender-held-out, or campaign-held-out split provides stronger evidence. For a RAG system, test authorization boundaries, not only answer quality. For an LLM classifier, record tokenizer and prompt-template revisions.

8. Adversarial examples and evasion

An adversarial example is an input intentionally modified to cause an unwanted model result under a specified threat model. The change may be visually or semantically small while being large in the representation consumed by the model. In security operations, this is an evasion capability—not automatically evidence of a real intrusion.

Goodfellow, Shlens, and Szegedy demonstrated gradient-based adversarial examples; Madry and colleagues framed robust optimization against first-order adversaries. These are foundational research results. To claim that a production detector is vulnerable, reproduce the result against the released artifact, document access assumptions, record the transformation, and measure the operational consequence.

9. Poisoning and backdoors

Poisoning changes training data or the training process so the learned model behaves incorrectly. A backdoor is a hidden condition—such as a trigger pattern—that causes a targeted behavior while ordinary validation appears normal. The attacker may target a dataset, labeling workflow, augmentation step, dependency, checkpoint, or model loader.

Security controls should therefore cover the full build: authenticated data sources, append-only manifests, reviewer separation for labels, isolated runners, pinned dependencies, signed artifacts, safe model loading, release approvals, and rollback. A clean model hash does not prove a clean dataset or training run.

When investigating a suspected poisoning event, preserve the original dataset and manifest, compare lineage against the approved version, search for unusual clusters or trigger correlations, rerun with a clean reference, and record what remains unknown. Do not overwrite the only copy by “cleaning” it in place.

10. CTI case study: a malware classifier

Consider a classifier that scores Windows binaries for triage. It consumes byte n-grams and imported API names, then routes high-scoring files to quarantine. A threat-intelligence report describes a new loader family, and analysts add samples to the next dataset.

reported sample and provenance
  → extraction version and feature manifest
  → training run and checkpoint
  → held-out campaign test
  → score, threshold, and quarantine decision
  → analyst disposition and feedback

Several hypotheses can explain a missed sample:

1. The sample is outside the training distribution.
2. The parser or feature extractor changed.
3. The threshold or route was changed.
4. The sample was deliberately modified for evasion.
5. A label, split, or training artifact was corrupted.

Map the report to the exact observable claim: sample hash, collection source, timestamp, behavior, confidence, and corroboration. Then connect it to local evidence—feature output, model digest, configuration, inference event, and side effect. The CTI report guides the hypothesis; it does not replace local evidence.

11. Controls and analyst exercise

Choose a local classifier or course-owned demo. Draw its computation graph, identify one parameter and one hyperparameter, rerun a fixed fixture, and write an evidence-based hypothesis for one changed output. Do not upload confidential data or test systems without authorization.

1. Record the input and preprocessing output.
2. Record artifact, dependency, and configuration digests.
3. Compare logits or scores before and after the suspected change.
4. Separate observed facts, reproductions, inferences, and unknowns.
5. Propose one deterministic control and one monitoring signal.

Key takeaways

• A neural network is a parameterized computation, not an isolated model file.
• Forward passes produce outputs; loss, gradients, and optimizer updates belong to training.
• Learning rate, seed, data split, preprocessing, and dependencies can change behavior as decisively as weights.
• Generalization and robustness claims require held-out conditions and an explicit threat model.
• Research demonstrations establish capability; CTI and incident claims require evidence about the environment in scope.
• Preserve data, code, configuration, artifacts, and inference events as a causal evidence graph.

References

• Rumelhart, Hinton, and Williams — Learning representations by back-propagating errors
• Goodfellow, Shlens, and Szegedy — Explaining and Harnessing Adversarial Examples
• Madry et al. — Towards Deep Learning Models Resistant to Adversarial Attacks
• Kingma and Ba — Adam: A Method for Stochastic Optimization
• NIST AI 100-2 E2025 — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems
• Google Machine Learning Crash Course — Neural networks

What comes next

Chapter 4 will connect this mechanism to Transformers, tokenization, embeddings, attention, and LLM generation—then trace the new attack surfaces and CTI evidence requirements.