(New) 28 October 2026, Computational Social Choice Seminar, Jason Hartline

Abstract

Scoring rules are foundational in decision theory and, therefore, are foundational for a developing theory of artificial intelligence. Just as simple models from decision theory provide context for understanding the decisions of complex humans, so too can they for complex AI systems. Bayesian decision theory considers an agent receiving a signal that is correlated with the state, choosing an action, and obtaining a payoff that depends on both the state and action. With Bayesian updating and the revelation principle, the signal becomes a posterior belief and the decision problem becomes a scoring rule. Given a scoring rule, baseline performance is the optimal score under the prior; benchmark performance is the optimal score under the posterior; and the optimal scoring rule — framed as a mechanism design problem — maximizes the difference between them. The talk reviews this theory and applies it to (a) evaluate prediction as a service, (b) behavioral experiments on human-AI decision making, (c) develop proper scoring rules for text (e.g. for training language models to hallucinate less).