The Flaw of Averages

[rockstar.esq]

The other day I stumbled into an old article about an even older event.  Back in the 1940's the US Air Force was experimenting with jet planes which caused an incredible increase in crashes, errors, deaths, and injuries.  It's a very interesting story, but the important bit is that the Air Force commissioned a study of the physical attributes of 4,000 or so US Air Force pilots to determine if there had been any significant change in their average size since the previous study in 1926.

The new study took 140 separate measurements of all 4,000 pilots.  Keep in mind that all 4,000 were within the strict tolerances for height, weight, etc. Armed with all 560,000 data points, they calculated the average for the 140 dimensions.  The guy running the study chose the 10 most relevant average measurements, and went through the dimensions of the 4,036 pilots to see how many of them were "average" size.

There were none.

The guy reduced the list to three average dimensions, and learned that only 3.5% (141) of the 4,036 were close.

This lead to a revolutionary idea.  If there is no such thing as an "average" pilot, then the planes had to be custom fitted.  Initial response from industry was to protest that this would be incredibly expensive.  In time, solutions began to come forward.  Truly novel ideas like adjustable seats, or helmets with adjustable straps.  Stuff we think of as commonsense today.

So what does this have to do with business?

I've met a whole lot of entrepreneurs who target imaginary averages.  "If I could just land a contract for $X amount every month, I'd be set"  I've even encountered people who would negotiate against themselves when their estimate came in higher than their average.  There's this trust in a familiar average that leads to poor decisions.  Success in a capitalist business is a risk versus reward relationship.  Competition constrains the reward, so the business must turn it's attention to managing risk.  Just like the Air Force, a business needs to adjust to reality instead of building to an average that doesn't exist.

 

[Glenn]

 Average number of people per household in the United States was in 1989 at 2.62. It has decreased to 2.52 people per household in 2019.  Average number of vehicles per household in the United States in 2001 was 1.89 and has decreased to 1.88 vehicles in 2017.

I have never seen the .52 people or the .88 vehicles.  

Averages exist, but only as numbers on paper. 

[Irondragon Forge ClayWorks]

If you torture figures long enough, they will admit to anything you want.

[John McPherson]

"Figures do not lie, but liars sure can figure."

This is where the love of abstracts blinds people to reality. If you stopped 100 people on the street, I doubt that 10 could tell you the difference between the mean, median and mode. But they are all averages.

You can take a mountain of data and simplify it to a quad chart or graph to make it easier to wrap your head around, but all you have done is glossed over the individual truths. The plural of anecdote is data.

The map is not the terrain, it is a gross oversimplification of the terrain that your tiny pea-brain can fix on and relate to, but only one small piece at a time. You can't see the forest for the trees. And vice versa.

We humans like to label things. If we have a handle on something, we like to think that we know everything about it, and so don't have to think about it anymore.

Finally, George Carlin: "Think about how stupid the average person is, and then remember; half of them are dumber that that!"

Why would anyone want to settle for average?

[ThomasPowers]

Shoot; I see .88 vehicles and less at the scrapyard every week!   I do try to avoid situations where the .52 person might occur.

[rockstar.esq]

Glenn,  I hope I never do meet a .52 person, that sounds pretty sad.

Irondragon, that's a funny way to explain it.  I actually have a book entitled "How to lie with Statistics".  It explains how the figures are "tortured" to generate the desired message.  

John,  I like your analogy with the map and the terrain.  In my line of work, averages aren't too useful on their own.  Where they start to have some relevance is when the averages of two related things are compared.  For example, I track my bid results by dollar value, and by bids per year.  The average of winning bids has the potential to be heavily skewed by a wide range in winning bid values.  In most cases, my average winning bid amount doesn't look anything like my typical contract.

In contrast, the average percentage of winning bids by count, is just going to show how selective I'm being with my bids.  If I have a low percentage, I'm probably chasing stuff that I can't win.

As I've already indicated, the average winning bid amount isn't reflective of my contracts.  The hit rate by count is nice to know, but it's not telling the whole story.  A relatively high hit-rate by count implies that I'm only targeting work I can win.  If I only chased stuff that I was the only bidder for, my hit rate would be 100%.  If I only chased stuff with one other bidder, I'd have a theoretical 50% chance of winning.  

So both averages have relatively obvious skew.  However comparing the hit rate by value to the hit rate by count tends to reveal a few things that wouldn't be observable any other way.  

Lets say I've got a client who regularly puts me up against one other competitor.  Over the course of a year, they bid ten jobs of varying value, and I won five of them.  True to form, the average winning bid amount doesn't match any one of the jobs that were bid.  Obviously the five winning bids out of ten total gives me a 50% hit rate by count.  If my hit rate by dollar value is less than 50%, it's an indicator that I'm winning the cheap jobs.

We could rerun that example where maybe I won six out of ten bids, but only won 30% of the total dollars bid.  That suggests that they have four large bids a year that I'm not winning.  Focusing on the 60% hit rate by count would lead a lot of people to believe that this client is a source of steady work.  However, if you're only contracting 30% of their annual construction budget, you should be asking yourself why that is.

I also track "low" bids which means that I was the lowest bid to my client, but the bid didn't lead to contract.  Finally, I sum the won and the low for a composite percent which illustrates how often I'm the lowest bidder in the field.  Again, I calculate averages for all of this by count and by dollar value.  Over ten years, I can see that my composite percentages for value and count have grown closer.  For each of the last five years there's less than 3 tenths of a percent separating the averages for count and value.   I believe that's because I've become far more selective about what I pursue, and I've worked my way onto invite lists where I only have one real competitor out of the bunch.  Maybe 1% of my annual revenue (won) is negotiated agreements, so I have to win 99% of our revenue by hard bids.  I'm hovering in the 47%-48% hit rate range for composite percentages.  One of the most frustrating things about this work is that I'm the lowest bidder in the market roughly half the time, but I only get a contract out of half those wins.  

The main reason for this is that there are a lot of clients who will put their job out to bid repeatedly.  I've had jobs where I won six consecutive bids on the same project before they finally awarded it to me.  

I know for a fact that there are a whole lot of firms that win less than 5% of their bids.  It's pretty obvious from the outside because these people bid everything they can, on the assumption that they can expect one contract for every twenty bids or so.  Since they average nineteen losses to every win, their estimators don't worry too much about quality, accuracy or ethics.  They have no strategy beyond pursuit, which is only made more efficient by losing quickly.  

[Buzzkill]

72% of all statistics are made up on the spot.

[Stash]

I knew a guy who ate himself out of an F16. He became a statistic.

Steve

[John McPherson]

Statistically, the average human has one testicle and one ovary....

[PWS]

Statistics is an evaluation of a set of values-no more no less...The collation of  values is solely dependent upon subject matter...The Subject Matter  and values  contained within the investigation of the subject matter is where Statistics becomes manipulative...Cheers.

[rockstar.esq]

PWS,  I think there's an easily overlooked element in your statement.  

On 4/26/2020 at 9:57 AM, PWS said:

Statistics is an evaluation of a set of values-no more no less...

You're referring to an entire field of study that primarily exists to impose order and regularity on data sets.  The primary application of statistics is to reduce a data set to a smaller set of indicative figures.  In short, the entire practical purpose of statistical analysis is to trade accuracy for brevity.

That trade-off is exactly the sort of thing that specialists like to overlook, because it admits to limitations.   It's therefore understandable that people feel they've misplaced their trust in these specialists when reality doesn't match the model.

If those 1940's era air-force specialists had less faith in precision, and more interest in accuracy, they wouldn't have killed so many pilots before deciding to make an adjustable seat.

[Frosty]

There is a HUGE practical difference between, "accuracy," and "useful," or "practical." For example a company needs to buy shoes for 500,000 employees. Today an employee would enter his/er shoe size and special requirements on their employment app. and the computer would take care of it. However in a situation where this can't be done. Do you demand accuracy and have every employee's shoe data collected and processed? How many employees would it take to collect the data, how many to process it before supply can place the orders, and deliver the shoes? two, three thousand people to process acquiring specialty shoes for half a million workers? Then there are the later hires, do you keep the entire staff on hand to process and supply a few pairs of shoes a month?  Hmmmm? It's the accurate way.

Or, do you buy shoes on a bell curve (One way of averaging) and accept a % of wastage? Of course it's not REALLY wastage there is sure to be somebody show up with that show size so you'll either need cataloged storage for a couple thousand pairs of shoes and a distribution system. So you build another room on Supply's warehouse and let the head guys organize, and distribute shoes. It's their job after all you might not have to hire anybody but if you do it's part of the business.

Which would you choose? Getting every detail correct or getting the data close enough to do the job with minimum wastage of time and materials?

Back in the day I dealt with a lot of intelligent well educated folks with little practical field experience. Their main job was writing reports, not collecting samples, the guys directly handling our data and reports from the lab usually benefited from spending some time in the field with us. The designers on the other hand often had no idea of how things actually work. Alaska seems to afford designers right out of college where most states with more mature highway design experience don't let a designer plan speed bumps or the width of sidewalk lines till they have a few years experience.

Anyway, we'd get tasked to do a soils investigation with specific directions and test requests. Often we'd get comments from new designers that we just needed to confirm what they already knew, they'd flown over the location. A couple short time designers tried to alter the data to meet THEIR plans. By short time I mean they rarely got to plan anything if they actually submitted a report so distorted. There is a 3 tier review system to make sure as few mistakes get through the final design and go to construction. Unfortunately some of the bright boys have slipped through so we're having to live with round abouts that are too small and have landscaped centers you can't see past. A lot of the new intersections and flow patterns appear to have been designed by folks who don't drive.

Anyway, statistics are very useful, a must in many cases but they're just a tool. If you don't know: when, where or how to use a tool, things aren't likely to go well.

Frosty The Lucky.

[rockstar.esq]

Frosty,  Well said!

 

[Frosty]

[notownkid]

Please take this in the light it is offered.  You get the .88% of a vehicle at the same time you get the .52% of the human, It's called a Crash.  In my nearly 30 yrs. of Volunteer Fire Rescue time I  unfortunately saw the after effects of some of these Crashes, didn't set about a fine measurement of any but they are never forgotten. 

 

Also agree with Buzzkill  72% of Statistics are made up on the spot.  

Like Political math 2 + 2 = what ever it needs to to prove a point.    

[George N. M.]

"Lies, d-----d lies, and statistics"