fin-glassbox

Complete Session Summary — 26-27 April 2026

What We Accomplished

1. Market Data Feature Engineering (Completed)

2. Cross-Asset Graph Construction (Completed)

3. Temporal Encoder (HPO Complete, Training Restarted)

4. Sentiment Analyst (Fully Complete)

5. StemGNN Contagion Risk Module (Code Complete, Verified, HPO Started)

6. VaR/CVaR/Liquidity Risk Module (Code Complete, Verified)

7. Technical Documentation Produced

8. Code Maintained and Fixed

Current Running Processes

Process Command Status
Temporal Encoder train-best --chunk 1 --device cuda Training (optimized version, batch=256)
StemGNN Contagion hpo --chunk 1 --trials 50 --device cuda HPO running on GPU
Sentiment Analyst HPO Ran earlier, completed N/A

What Remains To Be Done

Immediate (Next 24 Hours — After GPU Processes Finish)

  1. Complete Temporal Encoder Training
    • Let Chunks 1, 2, 3 finish training with optimized settings
    • Auto-embedding with XAI will run after each chunk
    • Output: 9 embedding .npy files + XAI attention weights + feature importance
  2. Complete StemGNN Contagion HPO & Training
    • HPO 3 chunks × 50 trials (running now)
    • train-best-all after HPO
    • Generate contagion scores + XAI for all splits
  3. Run VaR/CVaR/Liquidity 
    python code/riskEngine/var_cvar_liquidity.py --workers 4
    • CPU-only, no GPU needed
    • Can run in parallel with GPU processes
    • Output: var_cvar.csv + liquidity.csv + XAI explanations
  4. Run News Analyst 
    python code/analysts/news_analyst.py hpo-all --chunks 1,2,3 --trials 30 --device cuda
python code/analysts/news_analyst.py train-best-all --chunks 1,2,3 --device cuda
python code/analysts/news_analyst.py predict-all --chunks 1,2,3 --splits train,val,test --checkpoint best --device cuda
    • Code is complete with HPO + XAI
    • Needs GPU (competes with other GPU processes)
    • Output: Document-level predictions + attention traces + 128-dim embeddings

Short-Term (After Embeddings Are Ready)

  1. Build Technical Analyst (BiLSTM)
    • Depends on: Temporal Encoder embeddings
    • New file: code/analysts/technical_analyst.py
    • Input: 128-dim temporal embeddings (sequence of 30)
    • Output: trend_score, momentum_score, timing_confidence
    • Model: BiLSTM(128→64) → Attention Pooling → 3-head output
    • Needs HPO + training + XAI
  2. Build Volatility Model (GARCH + MLP)
    • Depends on: Temporal Encoder embeddings + returns data
    • New file: code/riskEngine/volatility.py
    • GARCH(1,1) statistical baseline + MLP adjustment
    • Output: vol_10d, vol_30d, volatility_regime, confidence
  3. Build Drawdown Model (BiLSTM Dual Horizon)
    • Depends on: Temporal Encoder embeddings
    • New file: code/riskEngine/drawdown.py
    • BiLSTM with separate 10d/30d prediction heads
    • Output: expected_drawdown_pct, drawdown_probability, recovery_days
  4. Build MTGNN Regime Detection
    • Depends on: Temporal Encoder embeddings + FinBERT embeddings
    • New file: code/gnn/mtgnn_regime.py
    • MTGNN graph learning layer → graph properties → regime classifier
    • Output: regime_label (calm/volatile/crisis/rotation) + confidence

Medium-Term (After All Risk Modules Complete)

  1. Build Position Sizing Engine
    • Depends on: ALL risk module outputs
    • New file: code/riskEngine/position_sizing.py
    • Rule-based with user-adjustable weights
    • Input: 7 risk scores → Output: position_size_pct, risk_budget_used
  2. Build Fusion Engine
    • Depends on: All analyst + risk outputs
    • New file: code/fusion/fusion_engine.py
    • Layer 1: MLP (learned combination)
    • Layer 2: Rule-based overrides (safety checks)
    • Output: Buy/Hold/Sell + confidence + contributing factors
  3. Build Final Trade Approver
    • Depends on: Fusion output + position size
    • New file: code/fusion/final_approver.py
    • Pass-through with safety checks (no trades on earnings day, circuit breaker)
  4. Build XAI Aggregator
    • Depends on: All module-level XAI outputs
    • New file: code/xai/explainability.py
    • Collects module explanations → unified stakeholder-facing report
    • SHAP on fusion layer, LIME on MLP modules, GNNExplainer on GNNs

Final Integration

  1. Build Daily Inference Pipeline
    • File: code/daily_inference.py
    • Loads all frozen models
    • Runs end-to-end: market data → encoders → analysts → risk → fusion → decision → XAI
    • Each model is conected to each other and passes it’s outputs ahead to the next model in the pipeline
  2. Evaluation and Backtesting
    • Design backtesting protocol across all 3 chunks
    • Compare module variants
    • Ablation studies
    • Write results chapter

Execution Order with Dependencies

Phase 1: Foundation (DONE)
├── ✅ Market Data Pipeline
├── ✅ Cross-Asset Graphs
└── ✅ FinBERT Embeddings

Phase 2: Core Models (IN PROGRESS)
├── 🔄 Temporal Encoder (training)
├── ✅ Sentiment Analyst
├── ✅ News Analyst 
├── ✅ VaR/CVaR/Liquidity 
└── 🔄 StemGNN Contagion (HPO running)

Phase 3: Risk Models (AFTER Phase 2)
├── ⏳ Technical Analyst (needs temporal embeddings)
├── ⏳ Volatility Model (needs temporal embeddings)
├── ⏳ Drawdown Model (needs temporal embeddings)
└── ⏳ MTGNN Regime (needs temporal + text embeddings)

Phase 4: Integration (AFTER Phase 3)
├── ⏳ Position Sizing Engine
├── ⏳ Fusion Engine
├── ⏳ Final Trade Approver
└── ⏳ XAI Aggregator

Phase 5: Production
├── ⏳ Inference Pipeline (by running the whole framework as the whole framework passes data among each other till we get final results)
├── ⏳ Backtesting Framework
└── ⏳ Thesis Documentation

Coding Standards (All Files Must Follow)

  1. Single file, dual-purpose: Importable as module(so that it’s outputs can be taken by other models and used when the whole system runs together) + executable CLI
  2. CLI commands: inspect, hpo, hpo-all, train-best, train-best-all, predict, predict-all
  3. HPO: Optuna TPE, SQLite persistence, resume-capable
  4. XAI: Every module exports explanations alongside predictions
  5. Paths: Configurable via --repo-root, .env file, or defaults
  6. No data leakage: Chronological splits only, train-fitted normalizers
  7. GPU/CPU: --device cuda or --device cpu
  8. Checkpointing: best_model.pt + latest_model.pt per chunk but then also the ability to the code to restore from the last checkpoint like if it’s at 50th epoch and training stops thre the last checkpont will be 49th saved epoch(model is saved with each epoch).
  9. British English in documentation

Key Files Created This Session

File Lines Purpose
code/encoders/temporal_encoder.py ~820 Transformer encoder with XAI (optimized)
code/gnn/stemgnn_contagion.py ~820 StemGNN contagion module with 3-level XAI
code/riskEngine/var_cvar_liquidity.py ~350 VaR/CVaR/Liquidity with XAI
code/gnn/build_cross_asset_graph.py ~500 Cross-asset graph construction
data/yFinance/engineer_features.py ~300 Feature engineering pipeline
data/yFinance/fill_final_pipeline.py ~400 4-layer statistical fill
data/yFinance/MARKET_DATA_README.md ~600 Complete market data documentation
code/gnn/STemGNN_README.md ~500 StemGNN module documentation

Total code written this session: ~4,500 lines
Total documentation written: ~2,000 lines
Models running: 2 (Temporal Encoder + StemGNN HPO)
Models ready to run: 2 (VaR/CVaR/Liquidity)
Models remaining: 8

This site is open source. Improve this page.