Starling

Financial forums are peculiar places. They’re simultaneously the most democratic form of investment research and the least accountable. Anyone can post a stock tip. Nobody checks if it was right.

This creates an information asymmetry in reverse. The experienced trader who’s been consistently accurate for years looks identical to the newcomer making their first wild guess. There’s no reputation system, no track record, no empirical basis for trust.

Starling asks: What if we could systematically track every prediction and measure who’s actually right?

The Credibility Gap

Traditional financial credibility comes from credentials - CFA certifications, institutional affiliations, Bloomberg terminals. But retail investor communities operate outside this framework. They’re credential-free zones where a anonymous username can move markets.

This gap isn’t just theoretical. It has real consequences:

Information asymmetry - Experienced traders with genuine insight are drowned out by volume. The signal-to-noise ratio approaches zero.

Manipulation vulnerability - Without accountability, pump-and-dump schemes thrive. Bad actors face no consequences for consistently wrong or misleading predictions.

Wasted attention - Retail investors spend hours parsing forum posts with no way to prioritize whose opinions actually matter.

The Extraction Pipeline

Building credibility from chaos requires a systematic approach. We process forum data through a four-stage pipeline that transforms unstructured discussion into measurable prediction accuracy.

Sentiment Extraction

The AI analysis layer processes entire forum threads in single API calls - a cost optimization that also provides better context. Individual posts are analyzed within the conversational flow of the discussion.

The extraction handles ambiguity. Not every post contains a tradeable prediction - many are questions, reactions, or general commentary. The AI distinguishes between:

• Explicit predictions - Clear directional calls with price targets
• Implicit sentiment - General bullish/bearish tone without specific targets
• Neutral content - Questions, news sharing, or factual discussion

Time Horizon Buckets

Predictions aren’t binary. A correct 7-day call is different from a correct 90-day thesis. We validate across three distinct windows:

Accuracy Metrics

We measure prediction quality across multiple dimensions. Direction accuracy alone isn’t sufficient - magnitude matters.

A user who correctly predicts direction 80% of the time but wildly overestimates magnitude isn’t as valuable as one who’s accurate on both dimensions. The composite score captures this nuance.

The Leaderboard Emerges

After processing 6 months of HotCopper data, patterns emerge. The distribution of accuracy is not normal - it’s heavily skewed.

Most users perform at or below chance level - consistent with noise. But the right tail tells a different story. Roughly 2% of users maintain accuracy above 65% across 50+ predictions. This is statistically significant signal.

User Personas

The data reveals distinct user archetypes based on prediction behavior and accuracy patterns:

System Architecture

The technical implementation prioritizes cost efficiency and scalability. Processing millions of posts through AI analysis requires careful batching and caching.

Key optimizations:

• Thread-level batching - One Perplexity API call per forum thread, not per post
• Activity-based scheduling - Active threads scraped every 6 hours, dormant threads weekly
• Response caching - Duplicate content detection avoids redundant AI processing
• Resumable checkpoints - Long-running scrapes can resume from interruption

The Console

The system exposes a Next.js dashboard for exploring credibility data in real-time. Key interfaces include user rankings, prediction signals, and operational monitoring.

Key Research Questions

Future Applications

This research focuses on the Australian Securities Exchange (ASX) via HotCopper. For collaboration inquiries: research@drksci.com