Wednesday, May 17, 2017

Bad Surveys

Yet another survey concludes that people are pretty dull when it comes to finances. This time it’s the Teachers Insurance and Annuity Association (TIAA) Institute who asked just over a thousand Americans 28 financial questions. The respondents didn’t do very well. But sometimes, it’s the designers of the study who are dull.

A Wall Street Journal article quotes one of the survey’s 28 questions:

There’s a 50/50 chance that Malik’s car will need engine repairs within the next six months which would cost $1,000. At the same time, there is a 10% chance that he will need to replace the air conditioning unit in his house, which would cost $4,000. Which poses the greater financial risk for Malik?

Anyone mathematically inclined sees instantly that the expected cost is $500 for the engine and $400 for the air conditioner. But the question is which potential repair “poses the greater financial risk for Malik?”

In the field of assessing threats and vulnerabilities, “risk” is defined as the product of probability and the amount of loss. This is the same as the expected value of the loss. Based on this definition, we would choose the engine as the greater risk.

In finance, we usually use standard deviation as the measure of “risk.” For the engine the standard deviation is $500, and for the air conditioner the standard deviation is $1200. Few people would do this exact calculation, but they may understand it intuitively. A 10% chance of a $4000 cost seems riskier than a 50% chance of a $1000 cost, and the math backs up this feeling. Based on this definition we would choose the air conditioner as the greater risk.

In case the argument based on standard deviation isn’t compelling enough, imagine that we replace the potential air conditioner cost with a 0.1% chance of losing $400,000. This is still an expected loss of $400. However, faced with a 50% chance of losing $1000 and a 0.1% chance of losing $400,000, reasonable people would focus more on the potential huge loss.

But the word “risk” isn’t owned by any one technical field. The everyday use of “risk” is imprecise and doesn’t conform exactly to either of the technical definitions.

Some people might look at this question and decide that it’s reasonable to be able to absorb a $1000 loss into their short-term finances, but $4000 would put them into a cycle of high-interest debt and digging out would take time. In this scenario, the air conditioner is the greater risk.

Another way of looking at this question is that engines will cost $1000 per year in repairs and air conditioners will cost $4000 every 5 years or so. So, engines are more expensive, and even though the word “risk” isn’t a good fit, a person who thinks about the question this way would choose the engine as more risky.

As it turns out, the survey designers think the correct answer is that the engine is riskier because its expected cost is higher. I wonder how many knowledgeable respondents understood expected cost but chose the air conditioner anyway because of their view of what “risk” means. I could easily have chosen the air conditioner had I participated in the study.

I agree that most people know too little about personal finances, but in this case, the study designers seem unable to ask clear questions.


  1. Agreed, this isn't a clear cut question with a right and wrong answer. Another aspect is how urgent the repair is. If your air conditioner breaks you might be able to tough it out and use fans until you can save up to repair it. If your car breaks down you might miss work and lose income until you get it repaired.

    1. @Greg: You're right. In addition to people having different ideas about what "risk" means, they also have different ideas about the relative importance of fixing their cars and air conditioners.

  2. As a market research professional, I always get astonished at the poor quality of the surveys that are featured prominently to the general public. The worst part is that these often end up being built into bad policy, business or personal decisions. To give some credit to TIAA, they at least built a "situational" question, which makes people think what they would do under specific circumstances, as opposed to the easier to answer "attitudinal" questions that fail to take context into account. That said, all is lost by the bad use of the risk construct, for the reasons you clearly articulated

    1. @UI: I wish you the best of luck in improving the quality of surveys. It's amazing how often I find a big disconnect between the way research is portrayed in popular media and reality of what was actually found. Sometimes the problem is with those who take the message to the masses, and sometimes the problem is with the researchers.