Most Reviews Are Biased and “Noisy”

[USMA81]

In the book, Noise, Nobel Laureate Daniel Kahneman discusses how most judgements exceedingly are fraught with error. We as humans are exceptionally prone to any number of factors that hinder good judgement. For example, in one study criminal judges looking at identical case files varied their sentencing on average by 3.8 years for the same case. This is an example of both bias and “noise.” Even the same person from day-to-day shows high variability in judgment. Time of day, the weather, how we are feeling, all (and more) can affect our judgment. The “Endowment Effect“ means we love something much more once we own it than we did before we bought it. Compounding the problem, we all think our judgment is far better than it truly is.

Noise likely is what caused U.S. News and World Report to rank the Macan EV 18th out of 26 luxury SUV EV’s (the Mercedes G-Class was #1 to my great surprise), while Car and Driver ranked the Macan 1 out of 14 Compact Luxury SUV EV’s. Motorweek recently awarded it the Best EV of the year. Dave OOS seemingly loves the vehicle, but scored it poorly in a recent quantitative review on YouTube (as a group, we were pretty hard on him, but in hindsight, the issue, on both sides, was noise / bias). Confirmation bias means we really like reading in detail the Car and Driver ranking, but the U.S. News review we just skim over.

How should we proceed to make a decision in a noisy world? Avoid first impressions, use a structured approach to select important criteria, rate these criteria individually, avoid reliance on gut feeling, average outside judgments to get different perspectives. And more…, but my key takeaway is that these reviews are error-prone and should play a small role in our overall decision process. But also know that we will have only a small amount of available information upon which to decide. One of Kahneman’s quotes from another of his books is my favorite: “What you see is all there is.”

Sponsored

 

[tmrqs]

How should we proceed to make a decision in a noisy world? Avoid first impressions, use a structured approach to select important criteria, rate these criteria individually, avoid reliance on gut feeling, average outside judgments to get different perspectives.

Thoroughly agree!

Just one question: for the rating of individual criteria (bolded above), should we assign negative values to items that have a 0 to 5 score range? Or is that a non starter?

Asking for a former member.

 

[Fun TC Driving]

We are smart and all take in a lot of others’ views, then deciding over time that this YT reviewer (saveagegeese) is one I subscribe to because they are incite ful and knowledgeable, yet reviewer Y and Z are click bait and/or have such biases that I am not going to their site again. Same is true of written media evaluations.

Kind of like a paraphrasing of that old truism, that we vote with our pocket OOS/wallet/credit credits where and what we buy, and so so we use our brain and past experiences to appreciate evaluations by X yet stop following Y.

 

[dgkhn]

How should we proceed to make a decision in a noisy world? Avoid first impressions, use a structured approach to select important criteria, rate these criteria individually, avoid reliance on gut feeling, average outside judgments to get different perspectives.

Except: I would say if one is publishing a review of a car, that's fine. If one is evaluating a car that one might obtain, probably yes. If one is evaluating a car one already owns, probably not. Because before you buy the car, you are looking for as much objective information as you can obtain to decide whether you will be happy with the car after you obtain it. Once you have it, then your subjective experience is primarily what matters and any objective factors become irrelevant. IMHO.

 

[dgkhn]

In the book, Noise, Nobel Laureate Daniel Kahneman discusses how most judgements exceedingly are fraught with error. We as humans are exceptionally prone to any number of factors that hinder good judgement. For example, in one study criminal judges looking at identical case files varied their sentencing on average by 3.8 years for the same case. This is an example of both bias and “noise.” Even the same person from day-to-day shows high variability in judgment. Time of day, the weather, how we are feeling, all (and more) can affect our judgment. The “Endowment Effect“ means we love something much more once we own it than we did before we bought it. Compounding the problem, we all think our judgment is far better than it truly is.

Just entertaining myself here, so please nobody take this reply too seriously. However, a definition of "judgment" is "the ability to make considered decisions or come to sensible conclusions."

So, "considered," "sensible." Not necessarily "correct." I think all of us know people who generally provide "considered" and/or "sensible" opinions, but who are not always "correct." Judges (one hopes) are called upon to exercise good judgment, not to always be correct. The difference between the set of "sensible" decisions and presumably narrower set of "correct" ones is, presumably, noise.

 

[DTaxman]

Except: I would say if one is publishing a review of a car, that's fine. If one is evaluating a car that one might obtain, probably yes. If one is evaluating a car one already owns, probably not. Because before you buy the car, you are looking for as much objective information as you can obtain to decide whether you will be happy with the car after you obtain it. Once you have it, then your subjective experience is primarily what matters and any objective factors become irrelevant. IMHO.

Exactly….reading reviews after you’ve purchased a car is just an exercise in confirmation bias or to torture yourself that others don’t agree with you. If you love your car and others hate it, who cares….unless the sole purpose of the vehicle was as a status symbol.

 

[krissrock]

great post... and i think the main issue today is that everyone (the media) almost all get the cars around the same time, and it's a very short time.
So they're trying do everything they can with the car in 1 hr...and then make a video. It doesnt work well.
and one i realize, is that a these of these content creators aren't "in to" the car...and don't do any research on it...and thus, they don't appreciate the car, or understand the intended purpose of decision the engineers made...
but anyway, yes... a lot of noise out there

 

[cottony]

Exactly….reading reviews after you’ve purchased a car is just an exercise in confirmation bias or to torture yourself that others don’t agree with you. If you love your car and others hate it, who cares….unless the sole purpose of the vehicle was as a status symbol.

i actually like to read reviews after i buy something lol. it helps me calibrate the various reviewers against my own personal review and identify whose opinions i should look to down the line.

im still waiting for a review that mentions the absolute agony of having to carry around a physical key for this car -- whoever does that will win my trust completely cause i will know we share the exact same values

 

[Gorge1969]

In the book, Noise, Nobel Laureate Daniel Kahneman discusses how most judgements exceedingly are fraught with error. We as humans are exceptionally prone to any number of factors that hinder good judgement. For example, in one study criminal judges looking at identical case files varied their sentencing on average by 3.8 years for the same case. This is an example of both bias and “noise.” Even the same person from day-to-day shows high variability in judgment. Time of day, the weather, how we are feeling, all (and more) can affect our judgment. The “Endowment Effect“ means we love something much more once we own it than we did before we bought it. Compounding the problem, we all think our judgment is far better than it truly is.

Noise likely is what caused U.S. News and World Report to rank the Macan EV 18th out of 26 luxury SUV EV’s (the Mercedes G-Class was #1 to my great surprise), while Car and Driver ranked the Macan 1 out of 14 Compact Luxury SUV EV’s. Motorweek recently awarded it the Best EV of the year. Dave OOS seemingly loves the vehicle, but scored it poorly in a recent quantitative review on YouTube (as a group, we were pretty hard on him, but in hindsight, the issue, on both sides, was noise / bias). Confirmation bias means we really like reading in detail the Car and Driver ranking, but the U.S. News review we just skim over.

How should we proceed to make a decision in a noisy world? Avoid first impressions, use a structured approach to select important criteria, rate these criteria individually, avoid reliance on gut feeling, average outside judgments to get different perspectives. And more…, but my key takeaway is that these reviews are error-prone and should play a small role in our overall decision process. But also know that we will have only a small amount of available information upon which to decide. One of Kahneman’s quotes from another of his books is my favorite: “What you see is all there is.”

I always wanted a Porsche. Never test drove the Macan EV before ordering. It's a Porsche. I watched endless reviews but they were all positive. I suppose thats what you do if you want to curry favour and get access to new models and invites to launches. Reviews are just opinions. I like the look of the new Macan EV and am 100% certain I am going to love it. There you are

 

[rcomeau]

Thoroughly agree!

Just one question: for the rating of individual criteria (bolded above), should we assign negative values to items that have a 0 to 5 score range? Or is that a non starter?

Asking for a former member.

This goes against basic brain biology. We tend to make decisions with our "gut" and then selectively look for facts to rationally support the decision we already made (confirmation bias). To be honest, there is no objective list that would put the Macan EV on top. There is a saying: "People don't buy what you do, they buy why you do it". The Macan and more widely, the Porsche brand, reflects something we want to express. The facts and figures are really the noise. You won't feel pleasure driving it because it has a 100kW battery (vs. whatever the other EV contenders had), you will feel pleasure because the steering feels right in your hands and it looks good to you when you walk up to the door. Embrace instinct.

 

[tmrqs]

This goes against basic brain biology. We tend to make decisions with our "gut" and then selectively look for facts to rationally support the decision we already made (confirmation bias). To be honest, there is no objective list that would put the Macan EV on top. There is a saying: "People don't buy what you do, they buy why you do it". The Macan and more widely, the Porsche brand, reflects something we want to express. The facts and figures are really the noise. You won't feel pleasure driving it because it has a 100kW battery (vs. whatever the other EV contenders had), you will feel pleasure because the steering feels right in your hands and it looks good to you when you walk up to the door. Embrace instinct.

Yes and no.

Yes- I understand what you're trying to say, that the experience goes beyond a spec sheet and the size of the battery won't change the feelings you get when you drive. But the battery being 95kWh (or the range being X to stay on the battery topic) will make it a better/worse choice when it comes to buying the car.

No- there is hard data that can be used to compare vehicles and that should also be taken into account. As much or as little as the driving experience, based on your personal choice.

The point of my post however was more that if someone decides to use a scoring card (say a mark between 0 and 5) to assess the value of a car, they should stick to it... and not give scores outside of that range. Be they lower (-10) or higher (+10), that just doesn't make sense.

 

[dgkhn]

I think, deciding whether or not to buy a new Porsche is a particular "thing" because of the associated cost; unless you're just made of money, or sometimes even if you are, you have to justify to yourself that the value is worth the cost. "Objective" reviews are an entrance to making that evaluation. Forums like this are extremely helpful because they expose you to people's experiences which point the way to the experiences you might have.

I've owned over 40 cars in my lifetime. Most of them bought new. Many of them have been interesting vehicles; some of them just utilitarian. I have disliked some, hated a few, liked a bunch, and loved a relatively small number over their lifetime. I have loved all four of my Porsche's and I think that means something.

I admit it's fun when someone says to you "nice car" or gives you a thumbs up. But really I choose a car for its usefulness or fun or value to me, not because of any statement it might make. So far I am feeling that my Macan 4 was a great choice, although I do have in the back of my mind some of the significant issues people have reported and I can only hope my particular vehicle was a lucky draw.

 

[USMA81]

So, "considered," "sensible." Not necessarily "correct."

The measure of a decision is its “correctness.“ For example, if we buy a vehicle anticipating it will meet our preconceived views on how it will enhance our lives (in whatever way), and it does so, then that would be viewed as a correct decision. Our very human tendencies to error (bias and noise) and our lack of complete information cause the problems in deciding.

Without going down the rabbit hole, noise is the variability in decisions, within ourselves over time (“We are not the same person all the time”), or among people who should be making the same decisions with the same information, but don’t. For example, some of the Macan reviewers might be reviewing the Macan on a cold, overcast, miserable day, and that influences their views on the vehicle; if they had done the review on a spring-like day, they might have come to different conclusions. Porsche inviting journalists to beautiful, sunny, coastal southern France nine months ago to do a test drive and review is not by chance. They were “priming” the conditions for favorable reviews.

 

[dgkhn]

The measure of a decision is its “correctness.“

I think it depends on the nature of the decision, and also in what sense the decision is "correct". As they say, even a stopped clock is correct twice a day. Do you want to rely on a stopped clock to decide what time it is? How does one decide whether a jail sentence handed down by a judge is "correct"?

Yes, pretty much if you feel a car you bought has lived up to your expectations, then you could say you made a "correct" decision to obtain the car. But, even then, there's "I wonder if I should have gotten the rear wheel steering"!

Then, of course, there's that forum member who bought that highly specced Macan 4S, and whose wife rejected it. Correct decision? Well, it might've been, but I guess as it turned out not.

 

[USMA81]

How does one decide whether a jail sentence handed down by a judge is "correct"?

Then, of course, there's that forum member who bought that highly specced Macan 4S, and whose wife rejected it. Correct decision? Well, it might've been, but I guess as it turned out not.

Kahneman wasn’t focused on the length of the sentence, which is influenced by cultural norms, laws, etc. He was interested in the fact that multiple judges looking at exactly the same case each could sentence the perpetrator to terms that differed by huge amounts, years. That’s not what society wants. Kahneman explores why many decisions can be inconsistent: bias and noise.

The forum member made a decision that turned out not to be what he expected. Maybe he was highly biased to buy Porsche, maybe he was feeling overly optimistic that his wife would like the vehicle, maybe he didn’t have sufficient information to make the best decision. Maybe something entirely different was at play. Clearly that didn’t go as he planned.

Outside of how this body of knowledge affects the reviews of the Macan, the book is a good read for any decision-maker.

Sponsored