Autoregressive foundation models trained on limit order book messages offer state-of-the-art market simulation, yet remain black boxes. This paper presents the first mechanistic interpretability study of such a model: an 8.2M-parameter Mamba-3 state-space model trained only on the tokenised message stream of eight large-cap US equities. Much as sequence models build an internal board in games, it learns an internal representation of the market and its dynamics, held in its recurrent state. Order-flow imbalance decodes at R² ≈ 0.59 against 0.12 from a book snapshot, the representation extends to unseen stocks, and steering the state along a feature’s axis shifts the model’s next-message beliefs as that feature implies, monotonically in dose.

Autoregressive foundation models trained on limit order book data are the state of the art for market simulation, but their cross-entropy objective neither penalises the compounding of sampling errors over free-running rollouts nor targets desired market properties. This paper introduces a post-training method using a Wasserstein GAN objective: a critic, trained online on rollouts through the true, non-differentiable matching engine, guides evolution-strategies updates over rank-4 LoRA perturbations of the pre-trained generator under a KL trust region, using the memory-efficient EGGROLL variant. On a held-out panel, post-training improves LOB-Bench realism by 17.6% (WS-21) and 17.0% (Mean-L1) for under 50M tokens and 12 GPU-hours per seed, and slows error accumulation over 500-message rollouts from 3.1× to 2.3×, while selection-matched and compute-matched controls yield no improvement.

In large language models, attention concentrates heavily on the first token, an ‘attention sink’ believed to protect deep models against over-mixing. Graph transformers face a related pressure, yet GRIT, a state-of-the-art graph transformer, rarely forms sinks on molecules, and sinks less when trained on larger graphs. The graph is not the barrier: its symmetry and locality limit the attention any node can receive, but on molecules neither limit prevents sinks. The attention scoring function is the main cause. Replacing GRIT’s gated logit with a dot product raises its sink rate from 0.09 to 0.68 on identical graphs. Attention sinks are therefore a property of dot-product attention rather than of Transformers.

Autoregressive foundation models simulate the market by generating the message flow of limit order books. This thesis asks what these models learn about market microstructure, and introduces a post-training stage that scores full sequences rather than single messages. Probing the recurrent state of a Mamba-3 model uncovers a representation of the market as a flow that can be steered along its encoded features. Training an adversarial critic through the non-differentiable matching engine and updating the generator with a memory-efficient variant of evolution strategies improves LOB-Bench realism by 17.6% with a thousandth of the data and under 10% of the compute.

Unsustainable hunting drives biodiversity loss, and acoustic detection with low-cost audio loggers is a practical way to monitor hunting pressure over wide tropical forests. We first optimise a two-stage pipeline using EfficientNets and Mel Spectrograms for gunshot detection, improving F1 score by 5.72% and AUPRC by 6.67% over the state of the art. For real-time alerts, we then move detection onto AI-oriented microcontrollers. A compact convolutional network, distilled from the larger model and deployable on an Arduino Nano 33 BLE Sense Rev2, reaches an F1 score of 0.840 and an AUPRC of 0.849, near state-of-the-art accuracy at a fraction of the cost.