Swing

Intro

Motivation

I was previously into trading. I had grown my portfolio by 30% in a span of a couple months doing swing trades — holding positions for days to weeks based on catalysts I'd spot manually.

The problem was that doing this well requires tracking a lot of context simultaneously. Macroscopically, you need to know where interest rates are headed, what employment looks like, what the Fed is signaling. At the news level, you need to know about geopolitical tensions, sector-specific developments, earnings surprises. At the fundamental level, you need P/E ratios, revenue trends, margin trajectories. And at the sentiment level, you need to know what analysts are saying and where retail sentiment is leaning.

I found it impossible to stay on top of all four dimensions for even a handful of tickers. I'd miss a CPI print that changed the macro picture, or not realize an analyst had downgraded a stock I was watching. The information existed — I just couldn't process it fast enough.

My hypothesis was that an LLM agent, given structured access to all four data dimensions via RAG and API pulls, could synthesize these signals better than I could solo. Not because it's smarter, but because it doesn't forget to check the Fed funds rate before making a call. I was also reading about production-level agent development — evals, observability, multi-perspective synthesis — and thought this project would be the perfect vehicle to learn those patterns while building something I'd actually use.

Thesis & Plan

My thesis: an agent with LLM-level breadth and access to up-to-date financial data + relevant context via RAG could perform as well or better than a solo retail swing trader, specifically because it considers all factors simultaneously rather than anchoring on whatever signal is loudest that day.

The development approach was to build each pipeline stage in isolation, test it independently, then wire them together end-to-end. The pipeline stages:

1. Data pull — quantitative fundamentals + macro indicators
2. Corpus generator — build the qualitative knowledge base
3. RAG — retrieve relevant context from the corpus
4. Prompt composition — combine quantitative + qualitative into a single prompt
5. Persona reasoning — 4 parallel LLM analysts with different biases
6. Judge synthesis — one LLM weighs all perspectives and outputs a structured recommendation
7. Guardrails — validate the recommendation before acting
8. Eval loop — backtest, measure, and self-improve

I structured it this way because each stage has different failure modes and different testing strategies. Data pull is about schema correctness and API reliability. RAG is about retrieval quality. Personas are about prompt engineering. The judge is about structured output parsing. Guardrails are about catching bad recommendations. You can't debug a bad final output if you don't know which stage introduced the error.

Data Pull

Purpose

Pull quantitative fundamentals (earnings, revenue, valuation) and macroeconomic indicators (rates, CPI, unemployment) to give the persona agents hard numbers to reason over. Without this, the personas would be guessing about whether a stock is overvalued or whether macro conditions are favorable — the numbers ground the analysis.

Solution

Two async data sources running in parallel via asyncio.gather():

Fundamentals (yfinance, free, no API key): pulls EPS actual/estimate/surprise, revenue with YoY comparison, gross margin with prior quarter delta, forward guidance direction (inferred from estimate trends with a 2% threshold), trailing/forward P/E, 52-week range, and corporate actions (keyword-matched from news: buyback, acquisition, merger, dividend, split, CEO changes, etc.). The tricky part was earnings date alignment — reports can drop before or after quarter-end, so I scan within the window plus a 90-day lookback.

Macro (FRED API): the interesting design choice here was sector-aware indicator selection. Tech stocks get 10-Year Treasury, Fed Funds, and CPI. Energy gets WTI crude and natural gas. Real estate gets mortgage rates. Financials get yield curve spread and high-yield OAS. The system maps ticker → sector → relevant FRED series, then pulls start/end values for each indicator within the analysis window. It also detects FOMC meetings in the window via FRED release ID 82 and captures the rate decision.

Both sources are synchronous libraries (yfinance uses requests internally), so they're wrapped in asyncio.to_thread() to avoid blocking the event loop. This matters because the API server runs on uvicorn — blocking the loop would stall all concurrent requests.

Output: a NumericBlock — pre-rendered markdown containing earnings tables, valuation data, macro indicators with deltas, and FOMC flags. Each quarter gets its own section separated by horizontal rules. The rendering uses format helpers (_fmt_dollar, _fmt_billions, _fmt_pct, _fmt_x) for consistent number presentation.

Caching

Disk-based JSON cache with a two-tier TTL strategy:

• •Historical data (window_end > 90 days ago): permanent cache, never expires
• •Current quarter: 24-hour TTL

The cache has two layers: individual API caches (cache/fundamentals/, cache/macro/) for debugging, and a merged quarter-level cache (cache/quarters/{ticker}_{quarter}.json) that skips both API calls on hit. This matters for the eval loop where you're re-running the same tickers across the same windows repeatedly.

Testing

This stage was straightforward to test — the main concerns were schema correctness and API reliability. I verified that the output JSON matched the FundamentalsResult and MacroResult Pydantic schemas, that the sector mapping produced the right FRED series for each sector, and that the cache TTL logic correctly distinguished historical from current data. The data_gapslist on each result captures any field that failed to pull, so partial failures don't crash the pipeline — they just show up as warnings downstream.

Corpus Generator

Purpose

The RAG system needs documents to retrieve from. The corpus generator builds and maintains a library of financial content — news articles, analyst ratings, macro commentary, and social sentiment — organized as markdown files with structured YAML frontmatter. This is the knowledge base that gives the agent awareness of market narrative beyond raw numbers.

Solution

Five async data sources pulling into a unified markdown format:

1. Polygon.io news — ticker-specific articles with cursor-based pagination, up to 200 per ticker/window. Returns title, description, keywords, publisher, URL.
2. GDELT — free, no API key. Geopolitics and macro-adjacent news. Searches by ticker name or company name (e.g., "TSLA" OR "Tesla"), 25 results per call.
3. Analyst ratings (yfinance) — upgrades/downgrades with firm names, grade changes, price targets. Also pulls consensus recommendation aggregates (StrongBuy/Buy/Hold/Sell/StrongSell counts).
4. FRED macro corpus — same 6 core series (CPI, Fed Funds, Unemployment, GDP, 10Y Treasury, 10Y Breakeven Inflation) rendered as corpus entries for RAG retrieval.
5. StockTwits — unauthenticated API, only ~30 most recent messages. Skips pulls for windows older than 7 days since the endpoint won't have historical data. Aggregates sentiment counts (bullish/bearish/neutral) and captures top 5 messages per day.

File format: each article becomes a markdown file with YAML frontmatter containing date, tickers, source_type, source name, relevance_tags, sentiment_label, sentiment_reason, and URL. Body is the article content with headline as H1. Naming convention: {date}_{ticker}_{slug}.md where slug is the first 5 words of the headline, slugified.

Corpus layout: corpus/{news,analyst,macro,social,_rejected}/

Dedup: URL-based index stored in .url_index.json. For URL-less items (analyst ratings, macro), generates a SHA256 hash of date|ticker|headline as the dedup key. 2,170+ entries indexed.

Quality gate: minimum word count filter per source type (news: 20, analyst: 8, macro: 10, social: 15). Articles below threshold get written to _rejected/ with a rejection reason in frontmatter. HTML tags are stripped before counting.

Backfill vs Live Run

Backfill — historical corpus build. Splits the date range into calendar quarters to manage API rate limits and enable recovery from partial failures. Uses a backfill_manifest.json to track completed ticker/quarter combos, making it safe to re-run after interruptions. Pulls all 5 sources per ticker per quarter via asyncio.gather(), with 0.5s inter-ticker delays for API politeness.

Live run— daily job for a single date (defaults to yesterday UTC). Designed for ~06:00 UTC scheduling. Individual source failures are logged but don't stop the run — missing one day of one source is recoverable.

LLM Tagging Pass

After writing, an LLM batch tagger runs over all untagged files. It sends batches of 20 articles (first 600 chars each) to the 70B judge model, which returns structured JSON with relevance_tags (from a fixed vocabulary: earnings, macro, sentiment, technical, geopolitics, corporate_action, analyst_rating), sentiment_label, and sentiment_reason. Tags are validated against the allowed set; unknown tags are dropped. This runs as a post-processing step rather than inline to isolate costs and avoid blocking writes.

Results

Backfilled 4 quarters across 8 tickers (AAPL, AMZN, GOOG, MSFT, NVDA, TSLA, SPY, QQQ). Sample write counts: AAPL Q1 2025 pulled 193 articles, NVDA Q1 2025 pulled 154. FRED macro entries are shared across all tickers (6 per quarter). The quality gate rejected short/empty articles into _rejected/, keeping the corpus clean for embedding.

RAG

Purpose

The RAG system's job is to surface the right qualitative context — news articles, analyst ratings, macro commentary, social sentiment — for a given ticker and time window. This is what gives the persona agents something to reason over beyond the raw numbers from data pull. Without good retrieval, the personas are just vibing off fundamentals and macros with no awareness of what's actually happening in the market narrative.

Architecture

Two components: an indexer and a retriever.

Indexer — reads corpus markdown files, chunks them using LangChain's RecursiveCharacterTextSplitter (400 tokens / 1600 chars, 50 token / 200 char overlap, hierarchical separators: \n\n → \n → . → → ""), discards chunks under 100 chars, embeds via Jina, and upserts into a vector store. Each chunk gets a deterministic ID (SHA256 of filepath:chunk_index, truncated to 32 chars) and carries metadata: file_path, chunk_index, date, tickers, source_type, source, relevance_tags, sentiment_label, sentiment_reason, URL, active flag. Skips already-indexed files and anything in _rejected/.

Retriever — 4-stage pipeline:

1. Hard metadata filter — filter by ticker (uppercase normalized), date range (ISO lexicographic for Chroma, native range query for Qdrant), and active=True before any embedding search. This ensures you're not pulling MSFT articles when analyzing AAPL.
2. Embedding search — constructs query "{ticker} stock price movement drivers {start} to {end}", appends thesis_hint if provided. Pulls top 200 candidates by cosine similarity. If date-filtered search returns 0 results, retries without date filters to recover context from adjacent periods.
3. Reranker — Jina jina-reranker-v2-base-multilingual via cloud API. Re-scores all 200 candidates, returns top 10 above a 0.2 relevance threshold. Deduplicates by source file (keeps highest-scoring chunk per article). On API failure, falls back to raw embedding similarity scores so the pipeline doesn't die.
4. Sentiment tag pass — chunks missing sentiment labels get tagged on the fly by the 3B model (bullish/bearish/neutral + one sentence reason). Most chunks already have sentiment from the corpus tagger, but this catches gaps. On LLM failure, marks all untagged as neutral — doesn't block the pipeline.

Token budget: 1500 tokens for the final QualitativeBlock, with priority ordering: analyst > news > macro > social. Items are greedily added in priority order until the budget is exhausted. Analyst ratings get priority because they tend to be the most information-dense and actionable for swing trade decisions.

Vector store abstraction: Chroma for dev (local PersistentClient, .chroma directory), Qdrant for prod (URL-based client). Both use cosine distance. Controlled by VECTOR_STORE env var. The abstraction exposes upsert, search, file_is_indexed, and delete_by_file_path — everything else is backend-specific.

Testing & Eval

RAG quality depends on architecture parameters (chunk size, embedding model, reranker, threshold) that all interact. So I built a formal eval script (tests/eval_rag.py) to measure retrieval quality across multiple dimensions.

Golden-set test cases — 9 cases covering:

• •Positive retrieval: NVDA March 2025 (≥ 3 chunks), AAPL Feb-Mar 2025 (≥ 2), MSFT Q1 2025 (≥ 2)
• •Thesis hint impact: NVDA with "AI chip demand and data center spending" hint (≥ 3)
• •Negative cases: fake ticker XYZFAKE (expect empty), valid ticker with out-of-range dates (expect empty)
• •Source type validation: TSLA should include analyst source types
• •Narrow windows: AAPL in just June 2025

Rerank lift measurement — compared embedding-only ranking vs Jina reranker on NVDA March 2025:

• •Top-10 overlap (how many docs appear in both)
• •Mean rank displacement (how much positions shift)
• •Threshold filter rate (how many docs drop below 0.2)
• •Raw score distributions for both methods

Key Finding: SPY/ETF Problem

SPY/ETF queries scored poorly with the original ms-marco cross-encoder. Corpus articles discuss "market selloff" or "S&P 500 drops" but the cross-encoder was looking for "SPY stock price movement" literally. It couldn't bridge that semantic gap. This was one of the motivations for switching to the Jina reranker.

How Things Changed After Testing

• •Ollama → Groq/Jina cloud — managing local Ollama models was annoying (pulling models, disk space, keeping them running). Cloud providers with free tiers gave equivalent quality with zero ops. The swap was a single-file change in clients.py because all provider logic is centralized there.
• •Local cross-encoder → Jina reranker API — moved from sentence-transformers ms-marco-MiniLM-L-6-v2 (in-process, synchronous) to Jina's jina-reranker-v2-base-multilingual (cloud, async). Side benefit: made _rerank() async, improving pipeline parallelism. Added a fallback to raw embedding scores on API failure.
• •top_k_before_rerank 50 → 200 — after seeing the rerank lift results, the reranker was good enough to handle a much larger candidate set. Wider net in embedding search + reranker doing the heavy lifting on relevance gave better final results.
• •Embedding model swap — nomic-embed-text (Ollama) to jina-embeddings-v2-base-en (Jina cloud). Both 768-dim, so no vector store rebuild needed. Intentional dimensionality match.
• •Reranker threshold 0.3 → 0.2 — loosened after testing showed 0.3 was too aggressive for some queries, particularly those with thesis hints that steered retrieval toward niche topics.

Prompt Composition

Purpose

Combine the quantitative data (NumericBlock from data pull) and qualitative context (QualitativeBlock from RAG) into a single prompt that gets sent to all four persona agents. This is the bridge between data gathering and reasoning.

Solution

build_composed_prompt() in prompts.py — straightforward assembly:

1. NumericBlock — the pre-rendered markdown text (earnings tables, valuation, macro indicators with deltas, FOMC flags). Falls back to "[No numeric data available.]" if the data pull failed entirely.
2. QualitativeBlock — renders itself via .render() into grouped bullets sorted by source type priority (analyst > news > macro > social). Each item shows: date, source, sentiment label, sentiment reason, and a quoted summary (max 800 chars). Falls back to "[No qualitative context available.]".
3. Thesis hint (optional) — appended as a === USER THESIS HINT === section if the user provided one.

The composed prompt then gets wrapped in a user message template: "Analyze the following data for a swing trade assessment:\n\n{composed_prompt}" — and this identical message is sent to all four personas. Only the system prompt differs.

Design Choice

The key decision here was separation of data from reasoning. The data blocks are built upstream by dedicated nodes; prompts.py only composes text. This means you can swap data sources, change the RAG retriever, or modify the rendering format without touching the reasoning layer. It also means the same composed prompt goes to all four personas verbatim — they see the exact same evidence and disagree only because of their system prompt bias.

Persona Reasoning

Purpose

Instead of asking one LLM for a trading recommendation and getting a wishy-washy "it depends" answer, run four parallel analysts with deliberately opposing biases. This forces the system to articulate both the bull case and the bear case with specific evidence, rather than hedging.

Solution

Four personas, each with a focused system prompt:

• •Bull analyst — "find the strongest case FOR buying this stock." Emphasizes upside catalysts, positive earnings surprises, improving fundamentals, favorable sentiment. Must cite specific numbers.
• •Bear analyst — "find the strongest case AGAINST buying." Emphasizes downside risks, deteriorating fundamentals, overvaluation, negative sentiment. Must cite specific numbers.
• •Macro strategist — "evaluate purely through the macro lens: interest rates, inflation, GDP, sector rotation, Fed policy." Determines whether macro tailwinds or headwinds dominate.
• •Technical analyst — "assess whether overbought or oversold based on valuation multiples, price levels relative to 52-week range, and momentum signals." States a directional lean with reasoning.

All four constrained to 3-5 sentences and 300 max tokens. They run simultaneously via asyncio.gather() on the Groq llama-3.1-8b-instant (8B model) at temperature 0.7 — high enough for diverse, creative analysis from each perspective but not so high that they hallucinate data.

Testing

This was mostly prompt iteration. I'd run the pipeline on a ticker where I knew the right answer (e.g., NVDA in Q1 2025 during the AI boom), read all four persona outputs, and check whether they were citing the actual data from the composed prompt or making things up. The main failure mode was personas ignoring the numeric data and just generating generic financial commentary. Tightening the system prompts to say "cite specific numbers from the data provided" fixed most of this.

The PersonaOutputs Pydantic model tracks all four outputs and exposes an agreement_ratio(direction) method that uses keyword matching (bullish/upside/long vs bearish/downside/short) to measure how many personas align with a given direction. This feeds into the guardrail checks.

Judge Synthesis

Purpose

Take the four persona perspectives plus the original data context and synthesize a single, structured trade recommendation. The judge resolves disagreements between personas by weighing evidence quality, not just counting votes.

Solution

Single LLM call on Groq llama-3.3-70b-versatile (70B model) at temperature 0.3 — low for deterministic, stable synthesis and reliable JSON output.

The judge receives a concatenated prompt:

=== DATA CONTEXT ===         <- original composed prompt
=== BULL ANALYST ===          <- persona output
=== BEAR ANALYST ===          <- persona output
=== MACRO STRATEGIST ===      <- persona output
=== TECHNICAL ANALYST ===     <- persona output
=== USER THESIS HINT ===      <- optional

The system prompt instructs it to return only a valid JSON object with these fields:

• •direction: bullish | bearish | neutral
• •magnitude_bucket: 0-3% | 3-8% | 8%+
• •confidence: float 0.0-1.0 (must reflect genuine uncertainty, not default to 0.5)
• •dominant_drivers: list of fundamental | macro | sentiment | technical (no duplicates, at least 1)
• •invalidation_condition: min 20 chars, must reference a specific price level, percentage, or named event
• •hold_window_bucket: days | weeks | quarter
• •thesis: min 30 chars, synthesized investment thesis
• •sided_with: which personas the judge agreed with most
• •sided_reasoning: 1-2 sentences on why those perspectives were weighted more

Temperature Gradient

This is a deliberate design pattern across the pipeline:

The idea is that you want creativity at the opinion stage and precision at the synthesis stage. The personas are encouraged to find interesting angles; the judge is encouraged to be consistent and parseable.

Structured Output Parsing

JSON extraction is defensive: find the first { and last } in the response, parse that substring. This handles cases where the LLM adds commentary before or after the JSON block. The parsed dict is validated against the Layer1Output Pydantic model, which enforces:

• •invalidation_condition must not contain vague phrases ("if things go wrong", "uncertainty", "market conditions") and must contain concrete references ($, %, or named events like CPI/GDP/FOMC/Fed)
• •dominant_drivers must not contain duplicates
• •confidence must be in [0.0, 1.0]
• •thesis must be at least 30 characters

Validation failures are caught and fed into the guardrail retry loop.

Guardrails

Purpose

Catch bad recommendations before they reach position sizing. The judge can produce structurally valid JSON that's still a bad recommendation — high confidence with no persona agreement, vague invalidation conditions, or neutral direction with high confidence (contradictory). Guardrails are the last line of defense.

Solution

Five checks, run sequentially after judge output:

1. Confidence floor — confidence must be >= 0.40 (configurable via env var). Below this, the signal is too weak to act on.
2. Invalidation quality — regex match against concrete financial references: $ amounts, percentages, named indicators (CPI, GDP, FOMC, earnings), technical levels (support, resistance, MA, RSI, MACD). Must also be >= 20 characters. Catches vague invalidations like "if the market goes down."
3. Data gaps threshold — if more than 4 data fields failed to pull (configurable), the recommendation is based on incomplete information. Forces lower confidence or explicit uncertainty acknowledgment.
4. Persona agreement — if confidence >= 0.70 and direction is not neutral, at least 25% of personas must agree with the direction (via keyword matching). Catches cases where the judge is highly confident but none of the personas actually supported that direction.
5. Neutral high confidence contradiction — neutral direction with confidence > 0.80 is inherently contradictory. High confidence implies you know something; neutral implies you don't.

Retry mechanism: on failure, the judge is re-invoked with the failure reasons injected into the prompt. Max 2 retries (configurable). If all retries exhaust, the pipeline is cancelled with a reason string stored in the database.

Eval Loop & Self-Improvement

Purpose

The pipeline produces recommendations, but are they any good? The eval loop backtests against historical data, measures prediction quality across multiple dimensions, and uses an LLM to patch the persona/judge prompts based on where the system is failing.

Benchmarks

Six metrics with pass/fail thresholds:

Ground Truth Population

A daily job that runs after hold windows close:

1. Queries pending runs where ground_truth_populated = FALSE and hold_window_end <= today
2. Fetches actual entry/exit prices via Polygon.io (with +/- 5 day buffer for weekends/holidays)
3. Computes actual direction (> 1% = bullish, < -1% = bearish, else neutral), actual magnitude, and price error
4. Scores each prediction as hit (direction + magnitude match) or miss
5. Upserts results to Postgres

Self-Improvement Loop

The core iteration cycle, run per quarter:

1. Run pipeline on all tickers in the watchlist (30s delay between tickers for rate limits)
2. Fetch ground truth
3. Compute all 6 benchmark metrics
4. If all pass → advance to next quarter
5. If any fail → generate a failure report showing which metrics failed, which drivers had highest miss rates, which tickers were worst
6. Send the failure report to the 70B judge model, which returns JSON patches — 1-2 sentence additions to each persona/judge system prompt
7. Apply patches to in-memory prompts (appended, never full rewrites)
8. Save prompt snapshots and reports to eval_runs/{quarter}/
9. Repeat (max 5 iterations per quarter)

If on a non-first quarter, iteration 1 checks for "big gap" — if the worst metric gap exceeds 0.20 compared to the prior quarter's final performance, it alerts the user that prompt patches may not be generalizing.

Results

Ran the loop across three quarters:

2025-Q3 (2/5 tickers scored): direction 50% (FAIL), magnitude 50% (PASS), calibration 0.0 (PASS), high-conf precision 50% (FAIL), cancellation 60% (FAIL), price error 5.76pp (PASS). Worst ticker: AMZN predicted bullish, actual neutral. Driver miss rates: fundamental 50%, sentiment 50%, macro 0%.

2025-Q4 (3/5 scored, 5 prompt iterations): direction 0% (FAIL — worsened), magnitude 33.3% (FAIL), calibration 0.7pp (FAIL), high-conf precision 0% (FAIL), cancellation 40% (FAIL), price error 7.29pp (PASS). All drivers at 100% miss rate. Prompt files grew from 6.15KB to 10.38KB over 5 iterations of cumulative patches.

2026-Q1 (1/5 scored, stopped at iteration 2): direction 0% (FAIL), magnitude 100% (PASS), calibration 0.0 (PASS), high-conf precision 0% (FAIL), cancellation 80% (FAIL — critical), price error 11.98pp (PASS). Worst ticker: NVDA predicted bullish, actual bearish.

The results show a clear pattern: the system is overconfident and directionally biased toward bullish. Cancellation rates spiked from 40% to 80%, suggesting the guardrails are catching more bad recommendations as prompts accumulate patches — the patches aren't fixing the underlying issue, they're adding noise. Prompt file sizes growing with each iteration confirms this: cumulative patching without pruning degrades quality.

Observability

Purpose

When the pipeline produces a bad recommendation, you need to know which stage failed and why. Observability is instrumented at three levels: per-node timing (Prometheus), full trace capture (LangSmith), and structured run storage (Postgres).

Prometheus Metrics

Exposed on port 9090 via start_metrics_server():

• •Pipeline level: pipeline_runs_total (by ticker, run_type, outcome), pipeline_duration_seconds (buckets: 5s to 5min)
• •Node level: node_duration_seconds (by node_name, buckets: 0.5s to 30s)
• •LLM cost: llm_tokens_total (by node, model, input/output), llm_cost_usd_total
• •RAG: rag_chunks_retrieved, rag_chunks_used, rag_empty_block_total
• •Guardrails: guardrail_checks_total (by check_name, pass/fail), guardrail_retries_total, pipeline_cancellations_total
• •Layer 1: layer1_confidence histogram (10 buckets), layer1_direction_total

LangSmith Integration

Every pipeline run gets a LangSmith trace with metadata: ticker, window, user_id, run_type, pipeline_version. Each node enriches its span with structured metadata (e.g., the judge node logs direction, confidence, magnitude, sided_with, agreement_ratio). Trace URLs are captured and stored in the langsmith_trace_url column in Postgres for post-hoc debugging.

@observe_node Decorator

Wraps both sync and async LangGraph node functions. Records NODE_DURATION_SECONDS histogram, optionally layers LangSmith @traceable on top if tracing is enabled. Uses time.perf_counter() for nanosecond precision. Always records metrics even on failure (finally block).

Postgres Eval Store

45+ column pipeline_runs table storing everything: identity (run_id, ticker, window), layer 1 output (all fields from Layer1Output), layer 2 output (gate decision, entry/target/stop prices, position sizing, Kelly criterion), ground truth (actual direction, magnitude, hit, price error, timing), debug inputs (numeric_block_text, data_gaps, RAG chunks, top chunks as JSONB), debug reasoning (all four persona outputs, judge reasoning), debug guardrails (checks as JSONB, retry count, cancellation reason), and trace URL.

Three materialized views: calibration_curve (confidence bucket → hit rate), feature_attribution (per-driver miss rate), pipeline_regression (per-version hit rate, avg confidence, cancellation rate).

Production Hardening

API Server

FastAPI on uvicorn with 2 workers. Startup validates required env vars (fail-fast). Health check endpoint pings both database and vector store. Three route groups: /runs (trigger, SSE stream, history), /eval (calibration, attribution, regression, self-improvement), /corpus (backfill, indexing, status).

Auth & Rate Limiting

API key auth via custom middleware — API_KEYS env var (comma-separated), validated with secrets.compare_digest() to prevent timing attacks. Per-IP sliding window rate limiter (30 req/min default, 10 burst). Public paths (/health, /docs) bypass both.

Docker

Multi-stage build: builder stage installs deps with gcc, runtime stage copies only site-packages. Non-root appuser, port 8000, built-in health check. Docker-compose bundles API + Postgres 16 + Prometheus + Grafana with provisioned dashboards.

CI/CD

GitHub Actions on push to main and all PRs: lint (ruff, pinned to match local), test (pytest with Postgres service container), security (pip-audit), docker build (only after lint + test pass). Recent fixes: pinned ruff version after CI/local drift, fixed pip-audit to install non-editable so it can audit dependencies.

Dashboard

Next.js 14 + React 18 + TypeScript + Tailwind. Pipeline trigger form with client-side window validation (mirrors backend constraints: 90-365 day window, 45-day recency minimum). Run history table with color-coded direction/confidence. Live run detail page with SSE streaming of node execution timeline. Eval page with calibration curve, feature attribution, and regression comparison visualizations. API requests proxied via Next.js rewrites to the backend.

Batch Runner

watchlist.yaml defines tickers and defaults (window_days, run_type, delay_seconds). Processes tickers sequentially with configurable inter-ticker delay (default 30s). Logs per-ticker results (direction, confidence, duration) and prints a summary table.

Conclusion

Evaluation of Results

The system works end-to-end: data pull → corpus → RAG → prompt → personas → judge → guardrails, with full observability at every stage. The architecture is sound — centralized client management, async throughout, Pydantic contracts between every stage, disk caching for eval loops, and a clean separation between data gathering and reasoning.

But the eval results are honest: direction accuracy didn't beat 55% across any quarter tested. The system is overconfident and bullish-biased. The self-improvement loop's cumulative patching strategy degraded rather than improved — prompts grew from 6KB to 10KB without meaningfully changing the failure patterns. Cancellation rates spiking to 80% means the guardrails are doing their job (catching bad recommendations), but also means the upstream reasoning is producing more bad recommendations as patches accumulate.

The most valuable components turned out to be the ones I expected least: the guardrails caught genuinely dangerous recommendations (high confidence + no persona agreement), the RAG eval revealed the SPY/ETF semantic gap problem, and the observability stack made it possible to trace exactly which stage introduced each error.

The least valuable: the self-improvement loop in its current form. Appending sentences to prompts is a blunt instrument — it lacks the ability to identify and fix specific failure modes. A better approach would be few-shot examples of correct reasoning on the failure cases, or fine-tuning the judge on the eval store data.

What's Next

The pipeline infrastructure is production-ready, but the reasoning quality needs iteration:

1. Replace cumulative patching with few-shot examples — instead of appending vague instructions, show the judge examples of correct reasoning on its past failures
2. Fine-tune the judge on eval store data — 45+ columns of structured prediction → outcome data is exactly what you need for supervised fine-tuning
3. Add a momentum/trend model — the technical persona is operating on valuation multiples and 52-week range, but has no access to actual price charts, moving averages, or RSI. A lightweight technical indicator layer would fill this gap
4. Expand the corpus to include earnings call transcripts — the highest-signal qualitative data for fundamentals-driven trades
5. Explore commercial viability — small-cap coverage is sparse and expensive; automated analysis could be valuable for tickers that institutional research doesn't cover. But the moat question is real: the pipeline is replicable, and the alpha (if any) comes from corpus quality and prompt tuning, both of which are ephemeral advantages