The original e-mail inviting students to participate in the COMBO survey offered them the chance to win prizes in exchange for their time. While monetary incentives are never an ideal way to randomize a sample, the implications are particularly dire for a survey that attempts to investigate the effects of "social and economic stratification." Basic economic theory predicts that, other things being equal, a poor student will value the chance of winning a gift certificate more highly than a rich student will. One might hypothesize, then, that students of lower socio-economic status will be over-represented in this kind of non-random sample. Looking at the data, however, we do not see the pool of respondents skewed to the lower end of the income scale. Indeed, the Vice Provost for Institutional Research Jed Marsh points out in the COMBO report that the respondent pool was "fairly representative of the larger student body" with respect to family income.
So why did wealthy students participate in the survey? Presumably, they didn't care about the gift certificates. The primary choice a wealthy student faces is between participating in the survey and enjoying an extra 20 minutes of leisure time (poor students face the same trade-off, but they are also incentivized by the chance to win a prize). Most of us tend to value leisure most when we have little of it; if you don't believe me, just ask a senior in April. As an economist would put it, the marginal utility of leisure is high at low levels of leisure and decreases as the amount of leisure increases.
Because the value of leisure time decreases with more leisure, wealthy students deciding between leisure time and participation in the survey will be more likely to respond if they already have lots of leisure. And since leisure is, to an extent, correlated with happiness - for those who are interested, there is a large literature on the relationship between leisure and quality of life - rich, happy students will be more likely to respond to the survey than rich, unhappy students. Since poor students are more motivated by the prizes, their participation is less contingent on leisure levels and their sample may be reasonably diversified across a spectrum of happiness. But if rich students are more likely to participate if they enjoy more leisure time and are happier, their sample will be biased toward happier respondents. A sample that under-represents unhappy wealthy students may suggest that happiness at Princeton is correlated with family income when in fact it may not be. Note that it is entirely irrelevant to this conclusion that the pool of respondents turned out to be "fairly representative of the larger student body with respect to academic class year and family income," as Marsh noted in the report.
So what are we to do? Unfortunately, the only way to avoid sample bias is to repeat the study, this time selecting a truly random pool of respondents. There is no doubt that by encouraging the campus community to discuss the effects of socioeconomic differences on life at Princeton, COMBO has made a positive contribution. In its current form, however, the survey does not allow us to reach USG president Josh Weinstein '09's conclusion that "Princeton still has a long way to go to equalize the undergraduate experience." Princeton may or may not have a long way to go; all that's certain is that a non-random survey will not be able to tell.
This is a general problem for survey-taking at Princeton and elsewhere. The only way to avoid issues of sample bias is for survey designers to randomly select respondents at the outset of the study. At a time when seniors are beginning to design their own thesis surveys, discussing the validity of the COMBO study (and indeed the validity of any study based on a non-random sample) may be a highly instructive exercise.
Holger Staude is an economics major from Frankfurt, Germany. He can be reached at hstaude@princeton.edu.
Read More
Why can’t lecturers win the President’s Award for Distinguished Teaching?
Jerry ZhuLet’s acknowledge those who make “boutique teaching” possible, those who work tirelessly to make our rigorous curricula more accessible, and those who dedicate their time to advising and guiding us through all the challenges we face at this University. Lecturers should be eligible for the President’s Award for Distinguished Teaching.
Let’s acknowledge those who make “boutique teaching” possible, those who work tirelessly to make our rigorous curricula more accessible, and those who dedicate their time to advising and guiding us through all the challenges we face at this University. Lecturers should be eligible for the President’s Award for Distinguished Teaching.
Princeton’s administration of justice is Trumpian
Frances BroganThe disparity between the University’s approach to Giberson and its approach to pro-Palestinian protestors reflects a concerning trend of appeasing an increasingly emboldened right rather than resisting the threat it poses to academic freedom and the wellbeing of Princeton students.
Secular Opening Exercises? Don’t throw out the baby with the bath water
Frank StrasburgerThe secular makes no room for the spiritual. What Opening Exercises are meant to convey is the spirit of the institution so that it may be kept alive from generation to generation.
The secular makes no room for the spiritual. What Opening Exercises are meant to convey is the spirit of the institution so that it may be kept alive from generation to generation.
