Abstract
High-stakes evaluation problems – to estimate the quality of items or people – arise in many real-world applications such as admissions, hiring, grading, and peer review. My research focuses on understanding and improving these evaluation problems along the dimensions of accuracy, fairness, efficiency, and reliability. In this talk, I describe one line of my research on correcting human bias under different behavioral models. The first type of bias arises in a sequential setting such as sports competitions and court decisions, where an evaluator rates candidates in an online, irrevocable fashion. I propose a natural model for the evaluator’s rating process that captures the lack of calibration inherent to this task, and conduct crowdsourcing experiments to support various facets of the model. I propose an efficient bias correction algorithm and show that it is information-theoretically optimal in terms of common ranking error metrics. I also briefly describe a second type of bias arising from people’s experiences that are irrelevant to the evaluation objective. For example, in teaching evaluation, students who receive higher grades are more positive towards their instructors. In such scenarios, I propose mild non-parametric assumptions to model the bias, design an adaptive correction algorithm, and prove its consistency guarantees.
Bio
Jingyan Wang is a President’s postdoctoral fellow in Industrial and Systems Engineering at the Georgia Institute of Technology. She received her Ph.D. from the School of Computer Science at Carnegie Mellon University, advised by Nihar Shah, and her B.S. in Electrical Engineering and Computer Sciences with a minor in Mathematics from the University of California, Berkeley. She uses tools from statistics and machine learning to understand and improve decision-making systems such as those involving hiring, admissions, and peer review. Her interdisciplinary research has been published in statistics, machine learning, artificial intelligence, human computation, and economics and computation. She is the recipient of the Best Student Paper Award at AAMAS 2019, and was selected as a Rising Star in EECS and in Data Science.
