Root cause analysis of the source volume disparity, with a concrete expansion plan to reach Sonar-level insight density
The original analysis (Feb 28) was based on a Steam-only prototype with 80 insights per game. The pipeline has been rebuilt with dual-stream (press + player), multi-source data, and GitHub Actions automation. Here's what changed:
| Change | Before (Feb 28) | After (Mar 2026) | Status |
|---|---|---|---|
| Data streams | Steam only | Press articles + Steam reviews | ✅ Done |
| Sources per game | 100 (Steam reviews only) | 100–122 (Steam + press articles) | ✅ Done |
| Category framework | 8 categories (keyword match) | 26 categories (AI extraction) | ✅ Done |
| Reference score | None | Steam All-Time % + Metacritic | ✅ Done |
| OpenCritic | Planned | ❌ Auth now required (was "free public") | Blocked |
| Planned | ❌ HTTP 403 from GitHub Actions IPs (Azure block) | Blocked | |
| YouTube transcripts | Phase 2 | ⏸ YT_BLOCKED=True (needs transcript proxy) | Backlog |
| GitHub Actions automation | None (local only) | workflow_dispatch — run from any browser | ✅ Done |
Using Balatro as the benchmark: Sonar shows it publicly in their press coverage, giving a clean apples-to-apples comparison point.
That's a 3.3× insight gap on a single game — down from 4.6× in the Feb baseline. The remaining gap is almost entirely explained by YouTube (disabled) and Reddit (blocked on GitHub Actions IPs).
| Game | Press % | Player % | Total Ins | Sources | Steam All-Time | Metacritic | Sprung Sonar |
|---|---|---|---|---|---|---|---|
| Black Myth: Wukong | 58% | 58% | 151 | 102 | 94% | 81 | — |
| Palworld | N/A | 84% | 77 | 100 | 95% | TBD | — |
| Helldivers 2 | 93% | 56% | 139 | 103 | 83% | 82 | — |
| Balatro | 100% | 77% | 112 | 103 | 98% | 90 | 88% · 370 ins |
| Hell Is Us | 54% | 49% | 243 | 101 | 87% | 77 | — |
| Outer Wilds | 100% | 76% | 144 | 122 | 95% | 85 | 81% · 610 ins |
| Disco Elysium | 89% | 72% | 131 | 102 | 92% | 91 | — |
| Pentiment | 95% | 67% | 173 | 102 | 95% | 86 | — |
| Citizen Sleeper | 86% | 67% | 184 | 102 | 94% | 82 | 75% · 389 ins |
| Signalis | 84% | 78% | 154 | 102 | 97% | 81 | — |
⚠ We now have MORE source documents than Sprung (103 vs 56) — but our 100 are short Steam reviews; theirs are long-form YouTube, articles, community. Quality > quantity here.
Category coverage up from 8 → 16 for Balatro, 19–25/26 for other games. Balatro's lower coverage (16/26) due to its focused card-game design rather than open-world variety.
The insight gap comes from three separable problems, not one.
What each source type provides, what's available for free, and estimated implementation effort.
| Source | Sonar Uses? | We Use? | Insights / Source Doc | API Access | Effort |
|---|---|---|---|---|---|
| Steam Reviews | ❓ Unconfirmed | ✅ Yes (100/game) | ~1.1 (AI extraction) | Free, no key | Done |
| Gaming Press Articles | ✅ Confirmed | ✅ Yes (3–5/game) | 5–15 per article | Google Search + scrape | Done |
| YouTube Videos (transcripts) | ✅ Primary source | ⏸ Disabled | 10–40 per video | YouTube Data API v3 (free quota) | YT_BLOCKED=True — needs proxy |
| Reddit (r/gaming, game-specific) | ✅ Likely (community boards) | ❌ Blocked | 2–8 per thread | Reddit OAuth — Azure IPs blocked without auth | Blocked on GH Actions |
| Metacritic User Reviews | ❓ Unconfirmed | ❌ No | ~1 per review (like Steam) | Scrape only (no API) | Medium |
| OpenCritic (critic reviews) | ❓ Unconfirmed | ❌ Blocked | 8–15 per full review | Now requires API key — was listed as free | Email developers@opencritic.com |
| Metacritic Critic Reviews | ❓ Unconfirmed | ❌ No | 8–15 per review | Scrape only | Medium |
| Steam Discussion Forums | ❓ Unconfirmed | ❌ No | 2–6 per thread | ISteamPublishedFile API (free) | Easy |
| Twitch/Streaming VODs | ❓ Unlikely (mentioned videos) | ❌ No | Variable (reaction-heavy) | Twitch API — free for metadata | Hard (transcription cost) |
With Steam only (1,000 reviews across 10 games), here's how mentions distribute across our 8 categories. Note: these are keyword matches, not discrete insights per Sonar's definition.
| Rank | Source → Category | Total Mentions | % of All Matches | Gap vs. Sonar Equivalent |
|---|---|---|---|---|
| #1 | Steam Reviews → Gameplay | 532 | 32.4% | Well-covered — Sonar's equivalent likely similar |
| #2 | Steam Reviews → Player Experience | 393 | 24.0% | Comparable, but emotionally shallow (short text) |
| #3 | Steam Reviews → Aesthetics | 323 | 19.7% | YouTube is massively better here (visual reactions) |
| #4 | Steam Reviews → Mechanics | 100 | 6.1% | Undercounted — Steam reviews rarely breakdown loops |
| #5 | Steam Reviews → Game & World Systems | 91 | 5.5% | Highly undercounted — better in long-form articles |
Four concrete phases, prioritized by ROI (insight gain per hour of dev work). Fully executable with the existing Python pipeline.
What to do:
1. Change max_reviews=100 to max_reviews=500 in analyze.py
2. Add fetch_opencritic_reviews(game_name) — search opencritic.com/api/game/search?criteria={name}, then fetch reviews at opencritic.com/api/review?game={id}. No API key needed.
3. Feed critic review text into the same classify pipeline. Each review is long — let AI extract multiple insights per review.
What to do:
1. pip install youtube-transcript-api google-api-python-client
2. Search YouTube: f"{game_name} review" site:youtube.com — target video reviews, not "let's play" content. Filter by: duration 8–25 min (full reviews), view count >10k, published within 2 years.
3. Fetch auto-captions via youtube-transcript-api. Clean the transcript, chunk into 800-token blocks.
4. Run each chunk through Claude with a multi-insight extraction prompt: "Extract all distinct player experience insights from this transcript. For each insight, identify the category, sentiment, and a 1-sentence quote."
5. Target 5–10 videos per game → 150–400 insights per game from YouTube alone.
What to do:
1. Reddit: use PRAW (Python Reddit API Wrapper). Free tier allows 60 req/min. Search r/{game_subreddit} for top posts tagged "criticism", "feedback", "ux", "accessibility". Also search r/gaming with game name + keywords.
2. Steam Discussions: endpoint is https://store.steampowered.com/appreviews/{appid} — Steam's IGetDiscussionList API also available for forum threads (no auth needed for public games).
3. Filter threads for signal: sort by upvotes, minimum comment count >5. Each quality thread = 4–8 extractable insights.
What to do:
1. Change the classification prompt from: "Which category does this review primarily discuss?" to: "Extract all distinct insights from this review. For each, return: category, sentiment (positive/negative/mixed), severity (minor/notable/major), and a verbatim quote of 1–2 sentences."
2. Update the schema from one object per review to a list of insight objects. Update the HTML renderer to aggregate insights per category rather than reviews per category.
3. Expected output: 2–4 insights per Steam review, 6–12 per OpenCritic piece, 15–40 per YouTube transcript chunk.
| After Phase | Sources Active | Estimated Insights/Game | Δ vs Current | Δ vs Sonar |
|---|---|---|---|---|
| Baseline — Mar 2026 ✅ | Steam ×100 + press articles | 112 | +1.4× vs Feb | −70% |
| Next: Reddit OAuth | + Reddit (GitHub Secret) | ~145 | +29% | −61% |
| Next: YouTube | + YouTube (5–10 vids/game) | ~350–450 | +3–4× | ≈ parity or above |
| Future: OpenCritic | + OpenCritic (API key req.) | ~400–500 | +3.6–4.5× | +8–35% |
| Full build | All sources + higher Steam cap | ~600–1000 | +5–9× | +60–170% vs Sonar |
{"message":"API key is required. Email developers@opencritic.com"}. This was documented as a free public API in the original plan — that's no longer the case. Contact them for a key, or pivot to scraping Metacritic critic reviews directly.