[Finally] Data proves why you shouldn't trust (donated) product reviews.

[george graves]

I love a good product review!  But to get a better sense of what you're looking to buy , my go-to places are Amazon review, YouTube, and EEVblog videos for test equipment.

But...I've long suspected that when someone receives a item for free, that the review is almost always a bit more positive then if they paid for it out of pocket.  Some reviewers say "Oh, no!  I'm so legit, that I put my thinking cap on, and it doesn't effect me at all!"  Blah! My spidy scenes told me otherwise.  We're all human, and respond to thing in the same way.  Just a quirk of human nature.

And that why I think every review, despite if you think you are above being influenced or not, should always say at the start if you paid for the item with your own money (even if it's youtube money)  or if it's a gift you don't (or do) have to return.

IMHO, This really should become the standard for all product reviews in the test/measurement world. There is nothing to loose by saying you got the item for free - just be honest and open about it.  Let the viewer know that info. Hiding things is what politicians do.

Anyways, this vid will explain it better then I can. It speaks to Amazon reviews, but more to human nature.

[Cerebus]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

I used to work as a full-time review journalist, as Labs Manager for PC Magazine and Tech Editor for PC Pro with later occasional freelance work for other titles. Both magazines that pioneered, in the UK, using objective test data as the core of reviews (hence "Labs Manager"). The norm there was to borrow products for review from the manufacturers and return them after review. Although this removes the "getting it to keep for free" bias and the "I bought it so it must be good" bias it doesn't eliminate bias altogether. For one, you've got PR companies associated with the manufacturers who can affect you in a number of ways from pestering you, threatening you, offering trips to "see the manufacturing plant" (booze-ups in sunny places), right down to simply knowing journalists innate biases and steering product towards or away from individual journos. Both my former homes were excellent at ensuring pressure from the advertising side of the magazine didn't affect editorial but that isn't necessarily the standard everywhere so that adds an additional possible bias.

The only sure fire way to eliminate all but some residual personal biases is to: 1) Buy the products to be reviewed anonymously, 2) Dispose of the products after review, 3) use objective measurable data as the core of reviews, 4) have objective, stated, criteria for what's 'best'. The problem is that this costs a *lot* of money and people just aren't prepared to pay for the end product nowadays.

[dannyf]

Data proves why you shouldn't trust (donated) product reviews.

if someone needs data to prove such a point, fake product reviews are the least of his/her problem.

[EEVblog]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

Yes, that has a huge effect in most people.

The only sure fire way to eliminate all but some residual personal biases is to: 1) Buy the products to be reviewed anonymously, 2) Dispose of the products after review, 3) use objective measurable data as the core of reviews, 4) have objective, stated, criteria for what's 'best'. The problem is that this costs a *lot* of money and people just aren't prepared to pay for the end product nowadays.

5) Have a policy to mention every nit picking negative thing you can find about the product regardless of the consequences.

[Cerebus]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

Yes, that has a huge effect in most people.

The only sure fire way to eliminate all but some residual personal biases is to: 1) Buy the products to be reviewed anonymously, 2) Dispose of the products after review, 3) use objective measurable data as the core of reviews, 4) have objective, stated, criteria for what's 'best'. The problem is that this costs a *lot* of money and people just aren't prepared to pay for the end product nowadays.

5) Have a policy to mention every nit picking negative thing you can find about the product regardless of the consequences.

Yeah. I'd argue "drawbacks" are part of any good review and I can tell you from experience that when you've got 30-50 otherwise identical products (i.e. A product category deliberately picked to be of roughly comparable products) you grasp at something wrong just to give you something, anything, to write about that doesn't sound the same as the other 10 reviews you've just written. In fact, in the slog of doing those kind of reviews both faults and good features can be magnified out of proportion and you have to guard carefully against doing so.

[R_Gtx]

There is also the case where the reviewed item is a "Golden sample", for example, containing all Japanese electrolytics, but delivered consumer units contain some lower spec. Chinese caps.
An example that comes to mind is here.

[zapta]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

Yes, that has a huge effect in most people.

That's a reasonable bias because it predicts my own opinion, as a customer, after purchasing that product.

A clear and full disclosure will go a long way, in the video itself ('my viewers already know' doesnt count) and explaining the scope of the incentive ('was sent to me' doesn't clarify if you keep it or if there were other incentives or business relations). Same goes to product placement of other than your own products.

[linux-works]

I used to spend time on SLR camera forums, and if you want to talk about golden samples - lenses from manufacturers and even camera bodies were always suspect of being hand-picked samples.  lens quality (canon comes to mind) varies a lot, even for very expensive lenses.  they even call it a 'lens lottery', or they used to (been years since I was in that scene, though).

as for user-based reviews, you do have to be very careful as many are fakes.  I was looking at solar-cell backed usb 'phone' battery packs and it seems 80% or more were 'samples given by vendor for review'.  they were all positive, too; but there was another tone of writing and from those folks, you could tell they paid for their own and they didn't have such a positive view of the product.

amazon has not fixed their review system and so you have to take what you read and consider all influences.  look at the reviewer and see their history and what they tend to write about.  is there something there that sticks out and yells 'shill' ?  its often obvious and does not take much digging.

[madires]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

Yes, that has a huge effect in most people.

Yep, the same effect why audiophoolery works: https://en.wikipedia.org/wiki/Cognitive_dissonance

[T3sl4co1l]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

I used to work as a full-time review journalist, as Labs Manager for PC Magazine and Tech Editor for PC Pro with later occasional freelance work for other titles. Both magazines that pioneered, in the UK, using objective test data as the core of reviews (hence "Labs Manager"). The norm there was to borrow products for review from the manufacturers and return them after review. Although this removes the "getting it to keep for free" bias and the "I bought it so it must be good" bias it doesn't eliminate bias altogether. For one, you've got PR companies associated with the manufacturers who can affect you in a number of ways from pestering you, threatening you, offering trips to "see the manufacturing plant" (booze-ups in sunny places), right down to simply knowing journalists innate biases and steering product towards or away from individual journos. Both my former homes were excellent at ensuring pressure from the advertising side of the magazine didn't affect editorial but that isn't necessarily the standard everywhere so that adds an additional possible bias.

The only sure fire way to eliminate all but some residual personal biases is to: 1) Buy the products to be reviewed anonymously, 2) Dispose of the products after review, 3) use objective measurable data as the core of reviews, 4) have objective, stated, criteria for what's 'best'. The problem is that this costs a *lot* of money and people just aren't prepared to pay for the end product nowadays.

Well said.

Remember further: human experience is nonphysical.  It isn't an evolutionary advantage for us to experience objective reality.  You are your biases, basically.

It therefore takes quite a lot of work to condition oneself to avoid or work around those limitations.  I pride myself on being as rational as possible, but I make no mistake about my own human susceptibility and fallibility!

Tim

[rstofer]

I like Dave's reviews for the simple reason that he presents the good, bad and ugly.  Bias may still creep in but if it exists, it is fairly easy to spot and it's right there on the video.  Look at the stuff on the shelf.  That's likely the stuff that gets used.  I really respect the opinions of people who actually use equipment and have experience with other similar devices.

In some ways it reminds me of a line from a TV show where a test pilot is gushing over the new fighter plane: "Best fighter plane I have ever flown".  Later it comes out that it is the ONLY fighter plane he has ever flown.  "The DS1054Z is the best scope I have ever owned!".  Fine, how many and what kind of scopes have they owned?

All that said, I totally ignore Amazon reviews and just about every other review on the Internet.  They lack credibility for two reasons:  First, I have no idea if they are qualified to do much of anything.  It seems as those some of those folks could break a ball bearing.  Second, by nature, there will be more negative reviews simply because people who are satisfied seldom spend any time blabbing about it.  This is particularly true for technology products.  Some people simply should not have computers or cell phones.

[elgonzo]

Look at the stuff on the shelf.  That's likely the stuff that gets used.

Incoming nitpick:
The reason stuff is on the shelf behind Dave is because it is not used. To see the stuff that likely gets used, look on the bench... and the office/lab floor.

All that said, I totally ignore Amazon reviews and just about every other review on the Internet.  They lack credibility for two reasons:  First, I have no idea if they are qualified to do much of anything.  It seems as those some of those folks could break a ball bearing.  Second, by nature, there will be more negative reviews simply because people who are satisfied seldom spend any time blabbing about it.  This is particularly true for technology products.  Some people simply should not have computers or cell phones.

Customer reviews such as on Amazon are of somewhat limited use. Unless you know the preferences and biases of the customer reviewer in question, which is pretty much unrealistic. Customers often tend to be too optimistic and too positive (confirmation bias, or lacking expertise) or too negative (venting their grievances).

However, there is value in customer reviews if a product has a 'sufficiently high' number of reviews made over a 'larger period' of time  (however you want to quantify "sufficiently high" and "larger period"). Despite encountering dilettantish reviews, with enough customer reviews you can often tell whether a product has quality problems to a much larger degree than you are comfortable with. Basically, customer reviews are almost useless to make a purchase decision, but in a set they can be helpful as some kind of filter to narrow down the list of candidates for your purchase.

[joeqsmith]

Imagine a manufacturer giving me their hand held meters to test their limits.  I don't think that's going to happen any time soon.  So you get meters that were purchased from normal channels (no hand picked camera lenses).  They are all ran on the same equipment with the same waveforms which are verified before each test.   No touching the leads together for the continuity test.   I would rather measure it with some standard method and document it.   Then finally, the results get placed into a spreadsheet for tracking.   

I fully admit that my personal bias towards the various brands has changed as I continue to run more meters.  Sure there are people who will be upset to see their beloved meter destroyed after they have been perfectly happy with it for the last several years.  I'll get things like, we only used brand (fill in the blank) at work and we never see them fail.   Or people who accuse me of being a fan boy of (name a brand).  The worst are the people who somehow gleam that the meters I test show no signs of being unsafe (and are therefore safe!)  or that because they saw a meter take a few KV that it somehow equates that they can measure the output of a MOT directly.   

I doubt you are going to find too many reviews where the data speak for itself.  If I test a Fluke 87V and it gets damaged at some level, then I go back and repeat the test, you get to see it.  UNI-T after UNI-T dies on the grill starter people get concerned that the grill starter is just too hard of a test to pass.  So I document the grill starters waveforms just for you.  I don't thing Fluke or UNI-T would send me meters.    Of course, there is the flip side.  The Asian made Flukes have been the most robust of the meters I have looked at.  And I have even gone so far as to show some of the problems with the UT61E's design and how it could be addressed if the UNI-Trend decided to do so.     

Yea, some of the meters that I can not salvage may get exposed to transient from the half cycle line simulator, dropped from the roof, shot with arrows, hooked to a neon sign transformer and on a rare occasion get shot with a cork, none of this is data I collect.   

One day, Dave will have his custom made meter for sale and it will be ran the same way, good or bad you will get to see the results.

"added safe comment"

[AntiProtonBoy]

If you buy something with your own money you've got a different bias to counter, confirmation bias. That is, "I've paid good money for this so I'd better prove to myself I didn't make a mistake".

Or you could see bias in the opposite extreme, "I've paid good money for this, buying this was a fucking mistake!!", then gives it rage rating of 1 out of 5. In reality the product might not even be that bad.

[SeanB]

Have the NST, might go find a cheapie meter to fry with it sometime. Have a choice of them as well, little ones and some larger more lethal ones.

[dannyf]

I think you should approach this "bias" question differently.

bias, or broadly speaking subjectivity, is not avoidable. consciously or subconsciously, we all have our own biases.

To me, it is futile to find an "unbiased" review.

With that said, I would take it on step further and say that bias / subjectivity is why you read other people's reviews. it is the reviewer's value-add: it allows you to see other people's experience with a product, how they think of it. Specs are for the most part not nearly as useful as real experience.

The key is full disclosure: let others know in advance if you may be subject to conflict of interests.

[joeqsmith]

Have the NST, might go find a cheapie meter to fry with it sometime. Have a choice of them as well, little ones and some larger more lethal ones.

NST?  Are you thinking to run the real IEC combo waveforms on some meters?   If so, post the model of your setup.  I'm interested!   Actually, if this what you are suggesting it would be great to have you run some of the meters I ran just so see how the results compare.   

[System Error Message]

thats why i never read 5 star or 1 star reviews on amazon. The best i give is 4 stars. The only time i give a 1 star is if it absolutely terrible or a fake product(such as usb cables that keep disconnecting because of poor construction).

[SeanB]

Have the NST, might go find a cheapie meter to fry with it sometime. Have a choice of them as well, little ones and some larger more lethal ones.

NST?  Are you thinking to run the real IEC combo waveforms on some meters?   If so, post the model of your setup.  I'm interested!   Actually, if this what you are suggesting it would be great to have you run some of the meters I ran just so see how the results compare.

NST Neon Sign Transformer. I know it kills things, especially after I made a dead linear halogen lamp, which had evaporated most of the filament as it blew open, darn near melt the quartz glass envelope after letting it run for a minute with the arc inside it. Also made the Jacobs ladder wires burning hot after a minute of operation.

Also can really fry chips and blow electrolytic capacitors with it. Arcs over any resistors as well, no matter the size or value.

[joeqsmith]

Have the NST, might go find a cheapie meter to fry with it sometime. Have a choice of them as well, little ones and some larger more lethal ones.

NST?  Are you thinking to run the real IEC combo waveforms on some meters?   If so, post the model of your setup.  I'm interested!   Actually, if this what you are suggesting it would be great to have you run some of the meters I ran just so see how the results compare.

NST Neon Sign Transformer. I know it kills things, especially after I made a dead linear halogen lamp, which had evaporated most of the filament as it blew open, darn near melt the quartz glass envelope after letting it run for a minute with the arc inside it. Also made the Jacobs ladder wires burning hot after a minute of operation.

Also can really fry chips and blow electrolytic capacitors with it. Arcs over any resistors as well, no matter the size or value.

Here I had my hopes up.   I don't think that last HV probe I made would stand up to the large transformer I showed.  Fun to play with. 

[VK3DRB]

According to a very trusted source who was an insider with no axe to grind, there is an electronic components vendor who published multiple "fantastic service and product" reviews with fake names all written online by the CEO of the company. Such behaviour is tantamount to fraud. I do not consider them when buying components.

I always wanted the x-ray specs and Polaris submarine. But the products were not what was advertised.
http://mentalfloss.com/article/30420/11-shameless-comic-book-ads-cost-us-our-allowance-money.

Whilst technology has changed, there will always be scammers around.

[helius]

X-ray Spex only cost a dollar, and they did pretty much what they advertised. If consumers (of whatever age) assumed that meant something that's impossible, they deserve all of the blame.

[rrinker]

 But those Sea Monkeys though.....   

[dannyf]

"multiple "fantastic service and product" reviews with fake names all written online by the CEO of the company. "

Unless it is a mom and pop company, a CEO fixated on writing online reviews is a terrible CEO and you should avoid his products, regardless of the reviews.