Real ML Training Pipeline

The Autoresearch Contract

XeeNet's training pipeline follows the autoresearch pattern pioneered by Andrej Karpathy: every experiment is a self-contained script that runs for a fixed compute budget and reports a single comparable metric.

Hyperparameter Search Space

The CharLMConfigGenerator defines the search space for character-level language model experiments. Each task samples a unique configuration from this space using a deterministic seed.

The Training Script

experiments/train_char_lm.py is a ~250-line self-contained training script. It implements a character-level GPT: a pre-norm decoder-only transformer with causal masking.

Architecture

• Tokenisation: Character-level (~65 vocabulary tokens from TinyShakespeare)
• Model: Pre-norm decoder-only transformer with causal attention masking
• Layers: Configurable depth (2-6 layers), width (64-256 d_model), and heads (2-4)
• Parameters: ~50K to ~2M depending on configuration
• Dataset: TinyShakespeare (~1MB, auto-downloaded and cached)

Interface

python train_char_lm.py --config-file config.json

The config file is a flat JSON object with all hyperparameters. The script reads it, trains the model, evaluates periodically, and outputs a single JSON line to stdout on completion:

{
  "val_bpb": 3.5906,
  "train_loss": 2.1847,
  "steps_completed": 1531,
  "wall_time_seconds": 9.02,
  "device_used": "cpu",
  "model_params": 198721
}

All other logging goes to stderr. This separation is critical: the worker parses stdout for the result JSON, while stderr is available for debugging.

Dual Deadline Pattern

Training tasks use a two-layer timeout system to guarantee termination:

1. Soft deadline (in-script): The training loop checks elapsed time each step. At 90% of the time budget, it stops training, runs a final evaluation, and exits cleanly with metrics.
2. Hard deadline (worker-side): The worker sets a kill timer at time_budget + 15 seconds. If the script hasn't exited by then, the process is terminated. This catches hangs, infinite loops, and GPU driver issues.

Config-via-Temp-File Pattern

Hyperparameter configs are passed to the training script via temporary JSON files, not command-line arguments. This is a deliberate design choice:

• No shell escaping issues: Windows cmd.exe and PowerShell handle quotes differently. JSON in a file avoids all quoting problems.
• Arbitrary complexity: Nested configs, arrays, and special characters work without serialisation concerns.
• Clean process interface: The script has a single argument (--config-file) regardless of how many hyperparameters exist.
• Auditability: The temp file can be preserved for debugging if a task fails.

# Worker writes config to temp file
config = {
    "lr": 0.003, "lr_schedule": "cosine",
    "n_layers": 4, "n_heads": 4, "d_model": 128,
    "d_ff": 512, "batch_size": 64, "context_length": 256,
    "seed": 42, "time_budget_seconds": 60
}
with open(tmp_path, "w") as f:
    json.dump(config, f)

# Spawn training subprocess
proc = subprocess.run(
    [python, "train_char_lm.py", "--config-file", tmp_path],
    capture_output=True, timeout=75
)

Worker Execution Flow

Both the Python worker agent and the Electron desktop worker follow this same execution pattern. The key steps:

1. Resolve script: Locate the training script from code_package_ref relative to the project root (Python) or process.resourcesPath (Electron).
2. Check PyTorch: Verify torch is importable. If not, fall back to simulated metrics with a clear warning.
3. Write config: Serialise the task's config dict (with injected seed and time budget) to a temporary JSON file.
4. Spawn subprocess: Execute the training script as a child process with stdout/stderr capture.
5. Enforce timeout: Hard kill at time_budget + 15s.
6. Parse result: Read the last line of stdout as JSON metrics.
7. Clean up: Delete the temp config file.

Verified Results

Task detail: completed training run showing real val_bpb metric, hyperparameter config, and resource requirements