Uncertainty-Aware AI: Integrating Reliable Human Preferences and ML Predictions in Societal Dimensions

The rapid integration of machine learning (ML) or artificial intelligence (AI) into societal research has transformed how we study human behavior, preferences, and societal phenomena. However, this integration presents methodological challenges at the intersection of society and AI. Two critical issues become prominent: the reliability of (ML/AI) systems in capturing genuine societal (or human) preferences through ranking algorithms, and the reliability of AI model predictions when applied to complex (societal) data structures.
This talk addresses these challenges through a statistical lens. First, I will discuss how ranking algorithms??"crucial for everything from recommendation systems to policy decisions??"can be validated to ensure they capture meaningful societal preferences rather than algorithmic randomness. Our methodology provides a unified uncertainty quantification framework for individual-level rankings, top-choice sets, and preference heterogeneity across different demographic and social groups, providing further guidance to examine the fairness of the recommendations.
Second, I aim to address the ‘black-box’ nature of ML/AI model predictions through two innovative approaches: (1) developing robust prediction sets that can effectively account for complex societal data structures, such as networks, hierarchical data, and image content, and (2) designing novel, unified methods for calibration and uncertainty quantification tailored specifically to the evaluation of Large Language Model outputs. These advancements seek to enhance the reliability and interpretability of AI tools for societal research while upholding statistical rigor.