This project is inspired by the Deep Red film trilogy from the creators of "Iron Sky". It aims to create a custom LLM model that has only knowledge up to July 1969 - the launch date of the fictional Mars mission in the movie - and responds in a style that aligns with a "Soviet utopia" setting controlled by the fictional "chess playing" AI.
These steps document how to perform a Model Training from Scratch procedure using a continued pre-training approach for temporal and thematic alignment.
- How to setup an AMD "Strix Halo" device (Fedora) — primary development and training system; includes automated setup script
- How to setup an optional second NVIDIA device (A4000) — dedicated GPU for training/inference; includes automated setup script
- How to set up the Wikipedia MCP server and data pipeline — extraction, indexing, search, and MCP server for Wikipedia content
- How to extract year-based historical topics from Wikipedia — enriched event data for temporal training
- How to augment Wikipedia with temporal metadata — YAGO/Wikidata parsing, normalization, and database augmentation for time-period filtering
- How to retrieve Project Gutenberg literature — thematically relevant books for training data
- How to prepare the chess training corpus — chess content retrieval, PGN conversion, and corpus preparation
- How to augment the chess game corpus - corpus augmentation to create custom chess-game content for the Deep Red persona
- How to tokenize and prepare the training corpus — tokenization, shuffling, and train/val splitting for continued pre-training
- How to train the Deep Red model — continued pre-training on the temporally-filtered pre-1969 corpus using dev (SmolLM2-360M) or prod (TinyLlama-1.1B) profiles
- How to train Deep Red on Gemma-3 (SFT) — parallel supervised fine-tuning track using Gemma-3-4B-IT / 12B-IT via TRL
SFTTrainer, mirroring the kyuz0/amd-strix-halo-llm-finetuning setup - How to use the generated GGUF models with LM Studio — loading, testing, and comparing trained model checkpoints in LM Studio
- Prototypes
- DeepRed-gemma-4b-2026-05-23-5-final.gguf - see corpus and model details
- gemma-4b-balanced-v1-small-1500-final.gguf - see balanced run details and examples
- gemma-4b-temporal-v1-10d-final.gguf - temporal-cutoff run (reintroduces
retain/unlearnat the 1969-07-20 cutoff); see run details and examples.
- Production
- gemma-4b-temporal-v1-10d-2-final.gguf - Full 10-day follow-up calibrated from previous run: 2026-06-17 runbook (2.32 GB)
- Intermediate checkpoints (10%, 25%, 50%, 75%):
Prebuilt chess augmentation archives can be downloaded directly from:
https://www.ferzkopp.net/Data/chess_games.jsonl.gzhttps://www.ferzkopp.net/Data/augmented_chess_games.jsonl.gz
Each URL uses the pattern https://www.ferzkopp.net/Data/[filename].
For augmentation workflow details, see the full guide:
documentation/ChessAugmentation-Setup.md.
/documentation- Setup guides and planning documents for the project/notebooks- Jupyter notebooks for testing embeddings and OpenSearch functionality/patches- System patches (network driver fix for AMD Strix Halo for older kernels)/scripts- Python scripts for Wikipedia extraction/indexing, temporal augmentation (YAGO/Wikidata), Gutenberg and chess content retrieval, MCP server, and system setup/services- Systemd service files for automated startup (inference servers, MCP server, OpenSearch, web GUI)/webapp- React-based web interface for Wikipedia search with Vite configuration