Category Archives: Opinion

Opinion about data science

p-values (or not)?

Much has been debated about the validity of p-values in determining statistical relationships between data elements. Here is a long list of 402 Citations Questioning the Indiscriminate Use of Null Hypothesis Significance Tests in Observational Studies. And here is a current and measured contribution to the debate appearing over at the Simply Statistics blog. How do you weigh in? Do you feel the use of p-values produces the kind of false positives we always see in the mainstream press?

The p-value conundrum


Linear Regression Fallacies


As a data scientist, I’m continually amused with the on-going misuse of the principle of linear regression.  You see it all the time in the press. Lately, regression theory is underlying much of the talk surrounding the demise of American entitlement programs like Medicare and Social Security. Forecasts abound, saying that the country can no longer afford retirement benefits and elder healthcare coverage. But you must stop to ask what these so-called forecasts are based on?

Many forecasts unwisely project healthcare costs far into the future by assuming that the trends of the past would continue unaltered. But this mentality ignores realty. To look at this another way, just because your son is 4 feet tall at age 6 doesn’t mean he’ll be 12 feet tall at age 18. And just because the average American born today will live to the age of 78 doesn’t mean that the baby born in 2032 will live to 100. Forecasting something like healthcare is just as prickly.

Making bets on the distant future is not wise because it’s unknowable. Too many exceptional events can occur to affect life along the regression line – wars, financial bubbles, financial crashes, extreme weather, political demagoguery, etc. No one – no business, no government agency – makes plans today based on a vision of the world 20 years ahead. Apple doesn’t do it. Google doesn’t do it. The Department of Defense doesn’t do it. You and I don’t do it. Not even insurance companies do it and much of their business is tied to the future. But many in our U.S. government prodded along by the anti-entitlement lobby are pushing to gut these programs while relying on projecting the past experience into the future without adjusting for changes in behavior or policy.

All this does not mean that uncertainty in economic forecasts means there’s no point in economic planning at all. There are indeed good reasons for looking ahead, in small increments, along the regression line, just not good reason or making sweeping changes in programs as complex and far reaching as Social Security and Medicare. So let’s accept life on the regression line for what it is – a limited and isolated view of the future.

Starbridge Partners

Welcome to STARBRIDGE PARTNERS! We Help Build Careers in Data Science. We Specialize in Placing Data Scientists & Data Engineers.

Follow the Data

A data driven blog

Sponsored by AMULET Analytics

Simply Statistics

Sponsored by AMULET Analytics

Data Science 101

Learning To Be A Data Scientist

R and Data Mining

Sponsored by AMULET Analytics