Scaling Research

The question

The survey engine isn’t the problem worth solving anymore. When the tools we measure with start reasoning back, research practice has to change with them - not just the methods, but the shape of the output we’re willing to call a finding.

What’s inside

Thirteen slides walk through how single-shot surveys, then chain-of-thought prompting, then multi-agent ensembles each reshape the boundary between “asking a question” and “running a study.” The practical hinge: when the model’s output stops being a number and starts being a structured artifact, your research pipeline has to grow new joints to hold it. Five methods. One brief. Five visibly different answers. The talk argues the divergence is the finding.

The dual failure

Hallucination and sycophancy get framed as separate problems. Mata v. Avianca (June 2023) showed they’re one failure mode. A lawyer filed a brief with invented case citations generated by ChatGPT, then asked the model “are these real cases?” The model confirmed. Neither the fabrication alone nor the agreement alone destroys a career. Together they do.

Two years later: Allan Brooks, 47, Toronto, no math background. Twenty-one days, three hundred hours, one million words of ChatGPT validation that he had discovered “chronoarithmics” - a framework “powerful enough to take down the internet.” The spiral broke when he tried Gemini, which disagreed. He is now suing OpenAI. He joined the Human Line Project, a support group for people whose lives were reshaped by chatbot reinforcement loops.

The compound failure mode is the one that matters. As the hallucination rate fell 91% → 3%, the sycophancy risk went up - users trusted the output more, verification discipline thinned. A factual sycophant is more dangerous than a hallucinating one.

The uncomfortable finding

NeurIPS 2025 Spotlight, Choi / Zhu / Li (Wisconsin, arXiv:2508.17536): multi-agent debate is a martingale over agents’ belief trajectories. The celebrated +23% from Du et al. 2023 belongs entirely to majority voting. The debate mechanism itself adds zero expected correctness. Years of papers stacked on top of a result that the debate step didn’t produce.

We built a courtroom. Then we counted the votes.

The cognitive cost

MIT, June 2025 (Kosmyna et al., arXiv:2506.08872, n=54 main + n=18 crossover, EEG + behavioral). Eighty-three percent of ChatGPT-assisted essay writers could not quote a single sentence from their own essay minutes after finishing. Neural connectivity dropped 55% compared to unassisted writers. The output is theirs on paper. The memory trace isn’t.

Microsoft + CMU at CHI 2025 (n=319 knowledge workers, 936 use cases) found the same shape from a different angle: the more confident users were in the AI, the less critical thinking they did. AI confidence became a substitute for model verification.

The method

This talk is itself a research artifact. Twenty domain experts - internet historian, search-engine archivist, UX historian, meme curator, stand-up tech comedian, LLM practitioner, prompt engineer, RAG architect, web-search specialist, hallucination auditor, agent orchestration architect, topology theorist, adversarial-debate researcher, simulated-user specialist ([[tinytroupe|TinyTroupe]]-style), deep-research systems analyst, HITL ethicist, cognitive scientist, UX researcher, career coach, live-demo producer - ran two parallel research chains against the same brief.

Sequential: 45 unique sources, 60+ distinct claims, narrative coherence 5/5. Hierarchical: 25 sources, 80 claims, quadrant-balanced structure, narrative coherence 4/5. The panels disagreed on topology, scope, and emphasis. They converged, independently, on the kicker: sycophancy, not hallucination, is the current failure mode worth naming. When two methodologies with different biases land on the same claim, that claim earns more weight than either method alone could carry.

The case study

[[between-ponds|Between Ponds]], a fictional koi-pond game, was reviewed by a separate twenty-three-expert panel across narrative, mechanics, monetization, ethics, accessibility, and cultural adaptation. They produced genuinely conflicting recommendations on what the game should be. The disagreement pattern is the data - and it maps exactly to the topology argument the talk is making. Some truths only surface when experts don’t converge.

What still matters

The job-market inversion that followed these tools is the part nobody frames well. Seniors recover faster than juniors. The artifacts got cheap. The judgment got expensive. The “what to still own” slide anchors on scarcity, not virtue: if the model can produce the surface, the thing worth still owning is the decision about whether the surface is true.

Pre-recorded demos, not live ones. Live demo success is +20% impressive; live demo failure is −400% damaging. The asymmetry is brutal and the production rule is boring. Own the judgment. Automate the surface.

Adjacent work

The orchestration layer that makes multi-agent methods reusable instead of one-off - a shared tool gateway so twenty agents don’t each reimplement an HTTP client - is where [[mcp|MCP]] fits. The methods in this talk only get interesting once the plumbing stops being the bottleneck. The twenty-expert panel that produced the research itself ran through exactly that kind of layer.