Thoughts on experts, predictions, and what to make of them.

[rockstar.esq]

I see a lot of articles with some version of the following in them;  "Expert who predicted XYZ says this will happen next".

Every time I see that, I think about a statistics class where we went over the prediction accuracy of some pretty basic stuff, weather, stocks, sports.  The most popular and  published experts in every one of these fields was less accurate than random chance.  In fact, in most cases, a person would have better odds of success by opposing whatever the expert predicted.

Yet in the past fifteen years or so, there's this incredible situation where we have experts who accurately predicted quite a few things in print, yet they seem to be inconsistent when placed under the spotlight.  What is going on?

I think the short answer is human nature, which is often ridiculous.  

I believe the first fundamental of this whole thing is publishing standards.  Journalists cite facts from respected agencies and individuals in their articles.  Since people often disagree, it's entirely possible for a journalist to find an expert or agency which supports their story.  There doesn't seem to be much investigating into whether or not the marketable expert or agency is consistently worthy of respect.  

The second fundamental I've noticed is the absolute absence of effective peer review.  An individual speaking for an agency is often quoted as the respected agency.  This same individual can do an interview where they render their own, opposing opinion or prediction.  Peers often treat this as a protected free-speech issue, without considering the individual's scientific, or moral integrity.  This practice works against public trust because "both sides" of any issue can claim the same experts testimony.

The third fundamental is what I call the time value of information integrity.  Since the individual in question is obviously predicting two different outcomes, they have to time their predictions such that they are close enough to claim credit for the success, but far enough apart to claim that "new information came to light" to excuse the failure.  With the narrative in place, these "experts" are able to compile a mighty list of accurate predictions, all of which were published in the media.  

The fourth fundamental seems to place everything is a zero-sum (existential) crisis.  In my industry (construction), it's fairly common to schedule new ground-up construction for late spring starts.  This minimizes weather delays, and generally allows the building envelope to be closed in before rain or snow become an issue.  Yet every trade publication treats the seasonality like it's an industry destroying boom or bust.   We're either dying from no work, or we're starving for new hires.  I can't help thinking that if the industry would quit panicking, they might apply themselves differently.  

The fifth fundamental I see is the "precautionary principle" which is an approach to innovations with considerable risk when available scientific information is lacking.  Very simply put, it's advocating for corrective action which might prove harmful, to forestall a perceived threat, in a situation where there is insufficient science to confirm the threat, or the corrective action.  "Shoot first, ask questions later", and "better safe than sorry" are a pithy summations of this principle.  I think this is a last-ditch approach that tends to assume that any action has even odds of success.  Life doesn't work that way.  Every action has a consequence, but the outcome is not always limited to good or bad.  I've found that unintended consequences cause most of humanities crises.  It's incredibly tragic how often our panic-driven action in an imaginary crises generates a real crisis as an unintended consequence.

With all that said, I think the reasonable individual has to take a few steps back.  In most cases, there is no crisis, and it's actually pretty rare that we get inflection points that clearly swing one way or the other.  In my industry, I can say with some confidence that the trade associations can rarely be expected to see anything that contradicts their preferred worldview / marketing narrative.  These human herds tend to be pretty dumb, even by herd animal standards.  

I've reduced my stress significantly by focusing less on the advertised crisis, and more on the actual challenges in my life.  I've seen quite a few crises go from constant coverage to total silence without ever encountering the most fervently predicted outcomes.  For what it's worth, I've also noticed how frequently there are order of magnitude math errors on all sides, all along.  On some level, I have to wonder if the goal is to separate the crisis from the facts so that people can abuse the precautionary principle to advance their tribal preferences.

[Irondragon Forge ClayWorks]

My definition of an "Expert". Someone who studies more and more about less and less, until they know everything about nothing.

[George N. M.]

Irondragon: Or, you can be a generalist and know less and less about more and more until you know nothing about everything.

Rockstar:  I've heard an expert defined as some SOB with a briefcase who is more than 50 miles away from home.

I think another contributing factor is the media's focus on attention grabbing news which sells papers, books, air time, band width, etc..  "Blood leads" is the driving factor on covering anything.  Good news doesn't draw much attention.  In your trade magazines there is probably an emphasis on some crisis, real or imagined, every month with some suggested remedy.  That's what sells.

And "experts" usually have an economic interest in some crisis or solution.  In the law it is usual to be able to find an expert to support your position no matter what it is.  Court cases are often a duel of the experts.  And what is probably more important than expertise in a particular area is the ability to credibly convey it to a judge or jury.  I've had experts who really knew their stuff but were bad at telling a story or who folded like wet paper under cross examination.  And I've had experts who would charm a jury and could have them believing the sun came up in the west if the expert said so.  The best expert witness is someone who knows their field and who is charming and can express themselves well and who is testifying pro bono because they believe in the case.  That is someone who is very hard to discredit.

"By hammer and hand all arts do stand."   

[ThomasPowers]

I find interesting the use of experts in one field being used to support contentions in a totally different field where they have little if any training and experience.

I think we suffer from a short news cycle where something new that is going to destroy our lives has to occur every couple of weeks; everyone jumps on the band wagon; often a bad law gets passed to show that they are *doing* *something* about it and then the *next* one shows up!  (Bad laws are ones that are often rushed and not thought out and examined for unintended consequences.  Often they do not solve the root cause of an issue but merely the visible surface and may end up causing more harm than good.)

[rockstar.esq]

There seems to be a consensus that the qualifications to be an expert are secondary to short-term aesthetic utility.  

George, you brought up conveying credibility, which is a real problem in industries like mine where trade associations do nothing to police their ranks.  The biggest estimators association in my area lists their canons of belief, including very explicit language against unethical bid practices.  It seems to have escaped the leaderships notice that unethical estimators have no compunction about lying, especially when the lie conveys credibility.  The local membership roster is peppered with unethical individuals I have first hand experience with.   

As I was thinking about that, it occurred to me that leadership in these associations might fear legal repercussions for expelling a dodgy member.  Even if they had a strong evidence in support of their action, they might be concerned about the cost of an effective legal defense.  

[Chris Williams]

"One good test is worth a thousand expert opinions." -- Werner von Braun 

You want an expert that recognizes the (limited) value of their opinion, and that can and will recommend the right test for a given problem. 

[ThomasPowers]

Empirical trumps Theoretical!

[rockstar.esq]

1 hour ago, ThomasPowers said:

Empirical trumps Theoretical!

If more people believed that, we'd have mandatory applicable skills testing for every job in society.  Not just for the initial hire, but for continued employment.

As a thought experiment, I think the only thing more disruptive than shedding incompetents from job ranks would be the incredible reckoning many industries would have about what they actually do.  A few local contractors would probably have to change the wording on their job openings.  Something like "...a criminal record is preferred, but not required."

 

 

 

[Chris Williams]

6 minutes ago, rockstar.esq said:

If more people believed that, we'd have mandatory applicable skills testing for every job in society.  

I hope you are not proposing that HR departments are qualified to establish skills tests that accurately determine worker capabilities. I don't know how many times I've heard "an engineer is an engineer" with respect to highly specialized positions. 

[ThomasPowers]

Not to mention "A manager can properly manage something they don't know beans about!"  Had a few that didn't understand physical constraints; basics like speed of light and direction of time.

[rockstar.esq]

Chris,

I have noticed that resource misalignment's are more common wherever competition and transparency are limited.  

I think it's critical to align incentives such that the wrong people find it profitable to do the right thing.  

This almost never happens because many people studiously avoid considering the complexity, and the potential, of their own nature.  Consider the Pareto principal, which states that 80% of the consequences, come from 20% of the causes. 

On the one hand, every student who's ever done all the work for a group project knows something fundamental about human nature.

On the other hand, it's like George Carlin once said "Everyone thinks of themselves as an excellent driver".

 

[Frosty]

On 10/13/2020 at 1:13 PM, rockstar.esq said:

On the other hand, it's like George Carlin once said "Everyone thinks of themselves as an excellent driver".

One example noted by Dunning & Kruger.

I believe R. A. Heinlein referred to the, Inverse IQ law, "The larger the group the dumber it is. Probably the first place I read it.

Working for the state I can attest to the positive feedback loop bad ideas and decisions create. One higher up makes a mistake and to save face finds or invents to justify it. Sycophants take it up to curry favor or because they don't know any better. Shortly it becomes institutionalized and everybody's stuck. 

I don't comment on your threads too often or I start getting bitter. 

Frosty The Lucky.