I Don't Really Trust Surveys | The Renegade Coder

Recently, you might know that I've been writing a bit about the red button vs. blue button debate that's been making the rounds on social media. You might also know that I recently went on a trip to Japan where I saw several research groups show off their work. Hell, you might even know that I am (or rather was) a qualitative researcher. While those might seem like unrelated events, they've sort of converged on a thought I've been having: I don't really trust surveys. Let me explain. Following the Thread A few weeks back, I got into a somewhat heated debate with one of my colleagues about survey research. It all followed after a workshop I had just enjoyed with my students. We were tasked with taking a kind of personality test, plotting the results of the group, and discussing the clustering. To start, I am deeply skeptical of personality tests, despite having participated in them many times. I've done the Myers-Briggs assessment (i.e., 16personalities), which was INFP ages ago and has shifted to INFJ as of writing this article. I've also done the Clifton Strengths assessment twice, getting a completely different set of strengths both times. It's not necessarily the lack of repeatability that bugs me, though that seems like a problem. It's that the questions are open to a wide range of interpretation, yet the results are surprisingly narrow. Let me give you an example of what I mean. At the workshop, we completed an assessment that was meant to "measure" our neuroticism (N) alongside extroversion (E), though to be honest I don't remember the exact constructs. E could have been emotional stability or something. In any case, after rating like 80 statements, we got a score in each category. Personally, I scored high on N and high on E, which placed me close to the upper right portion of the the plot. The most frustrating part about this entire process was that there was a secret third score, L. L apparently was your "lie" score, and if you scored above a certain threshold, you were considered a liar. Me and like basically all of our students were deemed liars, meaning our data couldn't be used for the study. Though, we were never asked to consent in the first place, so you can already imagine how much stock I take in the work. Intuitively, I thought maybe the L score was calculated based on how many of your answers diverged. For example, you might agree with one statement and disagree with another with slightly different wording. Of course, I take issue with that on principle for the same reason I stated before: different wording can lead to different interpretations. That said, I could accept that answer. However, when we asked how L was measured, we were told it was based on a few key statements. For example, one of the statements was roughly phrased as follows: "You would not be upset if your worst enemy was successful" I don't think it's completely unreasonable to say "agree" here, but apparently doing so contributes to your L score. This is just my broad problem with trying to quantify a person, whether it's with an IQ test, a grade, or a personality assessment. It's always in an attempt, whether explicit or not, to strip a person of their humanity and their complexity. It's inherently reductionist while simultaneously used as a weapon to generalize (e.g., all X's are like A, Y's don't belong at B, etc.) Naturally, I still had some fun at this workshop. It's fun to sit around and discuss the results in the same way that horoscopes are fun. But like horoscopes, these personality tests really shouldn't be treated seriously. For the record, I am amazed how much that article linked in the last paragraph aligns with my thoughts. I was going to make jokes about BuzzFeed quizzes in this article, and I already made the horoscope reference. It's good to know that my thoughts here are less of a hot take than I thought. Of course, I can't speak to the author. Maybe I shouldn't be this excited. He might be the next Jordan Peterson. How the Button Problem Relates At this point, whether or not you agree with my position, you have to be wondering how the red/blue button debate fits into this. See, to prove that phrasing matters in a survey or assessment, consider the red/blue button debate. Apparently, this was the wording in the original Reddit thread: In front of you appears red and blue button. If more than 50% of the people presses the red button, everybody who pressed the blue button dies. If more than 50% of the people presses the blue button, nobody dies. What button will you press? Tim Urban later rephrased the question as follows: Everyone in the world has to take a private vote by pressing a red or blue button. If more than 50% of people press the blue button, everyone survives. If less than 50% of people press the blue button, only people who pressed the red button survive. Which button would you press? Do you see the subtle differences that might lead to different interpretations? For example, the second framing explicitly uses the word "everyone." In the first framing, it's a bit more ambiguous. Likewise, the second framing uses the word "survives" rather than "dies." If you don't feel like phrasing matters, consider how much rephrasing is employed in the debate. So many people try to rephrase the question to make their choice more obvious. For example, you will hear folks describe blue as the choice to jump in front of a train. When posed this way, choosing red feels obvious. Often, you will also hear folks refer to red as the "does nothing" button or the "murder" button, depending on the goals of the speaker. I find this fascinating, not because I particularly care about the button debate, but because it speaks to my broader problem with surveys (among other quantitative measures of social concepts). How do you know that the statement you wrote is being interpreted in the way you intend it to be? Like, there are survey questions that you can write that are unambiguous, of course. "Who do you intend to vote for in the upcoming election?" seems pretty cut-and-dry. But, I suspect that writing Likert scale questions is significantly harder than the vibes-based statements you'll see in a personality quiz. Additional Problems Broadly, there are things you can try to control for in a survey. The very first thing would be reliability. In short, you probably want the results of your survey to be reproducible. This is the type of thing that a lot of these personality quizzes claim to provide. For example, I was a bit surprised when my MBTI came back only one letter off. That quiz seems reliable. Even then, if you happen to have multiple statements/questions that are meant to assess the same construct, then you need to make sure that the responses correlate. This was my problem with the liar score I mentioned previously, especially if it was calculated by looking at diverging responses. That's more an issue of question wording than an issue of participant honesty. But, reliability is not enough. You need validity. You need to know that your question is actually getting at the construct you intend to assess. This is why I sometimes laugh at the concept of grades because none of us are checking the validity of our assessments. Do our assessments even "measure" what we're trying to assess? How can we be sure? Like, at least some of us are writing assessments based on learning objectives, but I feel like an institutional climate survey is held to a higher standard than a classroom exam. Still, it seems like quantitative researchers are aware of these concerns, and they hopefully develop their instruments with that level of rigor. That said, even controlling for reliability and validity, I am still skeptical. Consider student evaluations of instruction. You may know that these kind of surveys go out to students every semester, and their results are used to give faculty promotions. As it turns out, despite our best intentions to develop an instrument for determining effective teaching, there's a glaring flaw: how do you account for misogyny and racism? There are so many variables that you simply can't control for, even if your instrument is otherwise valid. It makes me question their use as a research tool. Like, we use them because they're cheap and easy to use, but do they really tell us what we think they do? Like, if none of the examples resonate with you so far, consider political polling. How many times have polls been catastrophically wrong about election results? You might want to blame sampling, or maybe you want to blame how certain demographics were weighted. To me, there seems to be something wrong with the entire polling process. The Importance of Surveys in Research Broadly, I wouldn't say that I am anti-survey. They're just not a research tool that I would personally employ. With that said, I rely on them heavily in my own work. Who doesn't like to have some correlational evidence to help construct an argument? I've done this plenty of times myself. Numbers are powerful tools for convincing an audience. However, I really think results from survey research should be taken with a massive grain of salt. Any correlational findings can only be used to setup future work, either qualitative work meant to explain the findings or quantitative work meant to establish causation. Again, that doesn't mean surveys are bad. They're an important part of scientific progress. I'm more speaking to the interpretability of their findings. You can't really give someone a survey and turn around and say "oh, you're X kind of person" or "I can infer Y about you." That seems absurd on its face. Personally, you might think I am coming at this from some positivist angle where I only care about objectivity, but it's actually the opposite. I care about interpretability, and surveys leave way to much to the imagination. I need to get in there and talk to the respondents to understand their choices. I just don't really trust survey data on its own.