Content
X

Create Account

Or log in with Facebook

X

Log in

Or log in with Facebook

What Stat Geeks Don't Get

By Category

What Stat Geeks Don't Get

This is good fodder for the weekend.

Recently, a bunch of sports-related talking heads got up on a stage at M.I.T. for something called the Sloan Sports Analytics Conference. The panel of note, at least for me, was "What Geeks Don't Get: The Limits of Moneyball" which featured Jonathan Kraft and Bill Polian along with Mark Cuban and Bill Simmons.

More interesting? The take of Aaron Schatz, founder of Football Outsiders, a site that most serious football fans are familiar with. Schatz, needless to say, was not very happy with the proceedings. Money quote:

The most annoying part, for me, is when Polian basically says "there's nobody doing analysis worth listening to." This is about an hour after his son had personally told me that everyone in the Colts front office reads FO and they generally feel we're on the right track. So I was a bit pissed off.

OK, first things first.

I've been reading Football Outsiders for just about the entire time they've been on the web. I'm a big fan. They were the first people to give you something more than "you need to establish the run" as commentary. I met Aaron at the Combine, we had dinner and got along famously. He's a passionate football fan, as dedicated to the game as he is his work, which is superb.

But that doesn't mean Polian doesn't have a point, even if he made it in his characteristically abrasive fashion.

Even someone as dedicated to statistical analysis as Schatz is has to recognize that there's a limit to what we can do as fans. No matter how many times we watch our DVR copies of games, no matter how many times we log in to NFL Game Rewind, no matter how many incredibly interesting though thoroughly perplexing stats we come up with on our own, we are essentially hobbyists dabbling in someone else's profession.

That's not to say we don't have things of consequence to say and it doesn't mean we shouldn't be listened to. I understand Aaron's frustration - he's worked long and hard over the course of the last decade and people that matter in NFL circles know his work. I talked to tons of people in Indy who are regular readers, both in the NFL and in the journalistic community. And to be fair, his site isn't strictly numbers - what gets lost is that there is some downright amazing writing on there. (Tanier's Walkthrough and Farrar's Cover-3 are simply weekly must-reads for any football fan)

But all the writing and all the numbers in the world by fans doesn't capture a complete picture of what happens on every play of every football game. The football people have that. Polian himself even eludes to this saying:

There's nobody out there working on quantifying all that goes on, 22 guys, 60 times a game

There's a reason for this - the NFL guards the coaches tape as though it were a state secret, which, for the NFL, it kind of is. The obvious inference here? If the stat geeks ever got a hold of the coaches film, they would pull the curtian back on Oz even more than they already have.

The football people, on the other hand and armed with said coaches tape, have every tendency, every formation, every medical report on every player, every piece of history of every coordinator in the league including, at the press of a button, a cut-up of every third down throw from the right hashmark that the slot receiver caught for a first down, etc.

All of this, and they've been doing it for longer than a lot of these stat guys have been alive.

What drives me to distraction about most stat-driven analysis is how its proponents hold it up as some kind of talisman that they and they alone are capable of understanding. They bandy about numbers as though they are their own facts rather than the part-to-the-puzzle they truly are. Even worse, stat geeks often fail to provide any context to the numbers they hold so dear.

To be fair, I know I sound like I'm completely anti-stats, completely anti-FO. This is far from the truth, and I love this from Schatz:

I take pride in the fact that Football Outsiders leads the league in couching our opinions in caveats

Knowing what you do and don't know is a very important thing - some stat geeks are better at this than others.

Enjoy the video and I'd love to get your thoughts on the subject.

Comments (58)

hollycrat's picture

Hmm, yes. It is a frequent flaw that we stats geeks justify our numbers by looking at other numbers. I am as guilty as anyone of looking for truth behind stats, but I am also painfully aware of just how little I know about this sport I love to watch.
I wish Polian wasn't quite so blunt about saying it, but he does have a point. And you're right - Aaron - the reason there's no one "doing analysis worth listening to" is that no one's been given the chance. Maybe the mystique is good for the sport. Maybe they just like holding the cards. Who knows.

Keith's picture

Eh, I dunno Holly. Sure, there is a lot we don't know but I also think the league and it's personnel like to make the game more complicated than it is. I think Aaron can be a bit too reverential towards the people who run the teams (I get it, to get respect you have to not only earn it but also give it.) I personally believe Aaron shortchanges people like himself and Schatz by referring to themselves as "hobbyists."

Keith's picture

Polian is so full of crap, but why should he care? He just gave a great sound byte that also looks good in a dialogue box.
-----
What I think Polian, and our Aaron, missed about the stat debate as it pertains to football is sample size. Unlike a sport like baseball, where there are 162 games and tons of atbats to verify the data, football seasons (and even player's careers) go bye in the blink of an eye. It's really hard to come to any strong conclusions from such a small sample. Also, matchups matter. Since the season is so short, all teams don't get to play one another. Thus, a lot of projections rely on common opponent or similar opponent and while this is somewhat helpful its basically just advanced tea leaf reading, imo.
-----
Another issue that is unique to football is context. For example, I've argued with a friend about who had the better peak between Favre or Warner. My friend points to stats, but I quickly remind him that Warner played in a wide open offense with 2 elite wide receivers (Bruce and Holt) and a HOF RB (Faulk.) Meanwhile, Favre had to deal with a motley crew of WRs, turning guys like Bill Schroeder into Pro Bowlers. Favre also played outside, in the elements, while Warner played in a controlled environment while with the Rams. So who's right? Who knows? My point is that stats don't really do much for me in football, because success in football depends on 11 guys doing their job correctly. If one lineman misses a block it can lead to failure.
-----
It's funny you posted this today because I have spent most of today (day off from classes) prepping for my 2 fantasy baseball drafts this weekend. When I prep for fantasy baseball, I head over to Baseball Prospectus, get their PECOTA projections and rankings based on my league settings and I am good to go. When I draft, I mostly stick to the rankings until the later rounds, when I will grab some young prospects I've either seen play or read up on and who have a chance to outproduce their projections. But by and by, established players generally meet their projections. By contrast, I pretty much throw stats out the window when I draft my fantasy football team (aside from RB carries) because success on the football field depends on so many variables and is a more team oriented game than baseball.
-----
Ultimately, my point is this: I am a total stat-geek when it comes to baseball and trust the data completely, but when it comes to football I remain more of a stat atheist. I know they're out there and probably contain answers, but I'm not convinced anyone has really figured out the best way to access these answers.

Keith's picture

So I watched about 20 minutes and Polian and Kraft made the points that I discussed above. Obviously, I think Polian's points about sample size is the crux of the issue when it comes to stats and football.

Keith's picture

One more note re: sample size. 2007 is a great example of this. If you go back and listen to Bill Simmons' podcasts with Aaron Schatz where they discuss the NFL playoffs, Schatz you'll note how every round they brushed off how the Giants were playing, relying more on "the data." Well, as a Packer fan living in NYC, I was very nervous going into the NFCC because that Giants pass rush was ferocious and they finally seemed to "get" Spagnuolo's defense. Long story short: this was not the same Giants team I had watched every Sunday for 15 weeks.
-----
Not to brush off what Football Outsiders does, because I think their product is excellent. I just feel that when it comes to football the stats serve as more of a complement to watching the games where in a sport like baseball, you could go into free agency and make decisions almost solely on statistics (assuming the player isn't a sociopath ala Elijiah Dukes.)

foundindaho's picture

If Holly doesn't know football, I am a total dork loser in comparison. Well, I guess I am anyway.

To be honest, I don't pay a lot of attention to stats. I paid quite a bit to you know who's and it misled me to think he was a stronger player than he was (and yes, I still think he was/is good). I watch the game, try to keep on top of what the play is and who should be where, and yell "GO PACK GO!" a lot.

I appreciate you educating me though. Oh, and Jersey Al.

deelux's picture

Gotta agree w/ Polian on this. FO (and other stat-geek sources) can be innovative, insightful and highly informative, but DVOA and other advanced stats are largely backward-looking and only predictive on a macro level.

For example, you can count how many times Daryn College has a brain fart and forgets how to block, but you cannot predict when it'll happen. If only...

If turnovers or anomalously poor red-zone efficiency cost a team victories despite otherwise stellar statistical performance, the coach's next gig is still going to be as a quality control asst. in St. Louis or Detroit, bitchin' DVOA or not.

Polian's reference to 22 guys, 60 times a game is really on the money. Football success is determined by the semi-subjective outcomes of gobs of 1-on-1 and 2-on-1 battles, but the quants' can only analyze the quantifiable.

Charting a 20-yard run on 2nd & 7 from your own 45 from the left hash with 7:42 remaining in the 2nd qtr while you're team's behind by 6 doesn't account for the pulling guard taking just the right angle to block the OLB, or that safety was thinking pass and put himself out of position.

In a nutshell: there are no winners in the Pythagorean playoffs.

packeraaron's picture

You absolutely nail it here. Well done. Had something akin to your 2nd and 7 example in mind but couldn't get it in the post while at work - you sum my thoughts up perfectly here. Thanks much :)

Franklin Hillside's picture

People can come up with statistics to prove anything, Keith. 14% of people know that.

FITZCORE1252's picture

But Frankiln,

73% of the 14% say stats only have a 25% influence on their stance... common knowledge.

GBP 4 LIFE

Keith's picture

Was I not clear that I do not think stats work as well for football analysis as they do for baseball analysis?

PackersRS's picture

Coming from soccer, stats in football are overwhelming. They say a lot, the right ones. But they'll never say enough.

Until someone comes up with stats for personality, mood, team chemistry, refs interference, and luck, a team strategy cannot be BASED by stats, rather be assisted by it.

Stats are so big in US sports that it sometimes overshadows play itself...

FITZCORE1252's picture

"Coming from soccer"...

Well, I'm glad we can offer you American football, because I for the life of me CAN'T see any redeeming qualities in "European Football".

I have tried.

Several guys I work with are Sounder FC fans. I've watched a few games with them over a few brews, needless to say the beer was all that I enjoyed.

I don't get it.

Millions of people worldwide do though, so there must be something there.

I think I was just spoiled with the best sport and the best franchise on the planet from an early age, subsequently, nothing else stands a chance.

GBP 4 LIFE

jerseypackfan's picture

Manchester United 4 LIFE!

PackersRS's picture

That's what's wrong. You've been watching too much European football. You should watch more South American football.

The only thing worth watching in Europe is Barça.

FITZCORE1252's picture

By "European Football", I just meant Soccer.

I can honestly say that I will never give Soccer another minute of my time. I'm probably just an ignorant American in that regard... it's bliss.

Draft here yet?

GBP 4 LIFE

Keith's picture

HALA MADRID!

CJ in Guatemala's picture

"Well, I’m glad we can offer you American football, because I for the life of me CAN’T see any redeeming qualities in European Football."

This happens to be inversely true for almost anybody that holds soccer as their true sport though. People who grow in their lives watching soccer, probably wouldn't understand the appeal of Football at all either, even if they tried like you do on "European Soccer".

I think all of this happens because of two major factors, which are lost in the "translation" if you will from one game to another:

1)Knowledge of the game. For example, it is important to know why each position of each player is important to the game, and not just what the general goal of the game is. You need to know why certain plays are penalized and why some others aren’t.
When you have more of a general understanding about the game, I think you can actually enjoy it.

2)Broadcasting color. This is very important, because if you don’t listen to a proper play by play commentator, then all the magic is gone. It’s just like listening to Joe Buck the guy has no spark and he does NOTHING for me in a live Football “primetime” setting. As it is the case I’ve heard some soccer broadcasts in English, in which you could just fall asleep, and I would feel sorry for the people who actually has to watch soccer that way.

“I think I was just spoiled with the best sport and the best franchise on the planet from an early age, subsequently, nothing else stands a chance.”

Now this line holds completely what I feel about the sport and the Packers, because while I’m not from the US or started watching Football until I was 15 or so, I DO think nothing else stands a chance, not even soccer.

Now about stats, I generally think that YOU HAVE TO PLAY THE GAME OUT, because stats alone aren’t going to decide the outcome of a game. I mean I love a good stat once in a while, and most of them are peculiar and entertaining, just like the “Defensive Hog Index” which Aaron pointed out little after the regular season ended. But by stats and matchups alone we had that Dallas game last year LOST. Then the Pack went to play and won superbly in a game which not many where hopeful to win.

-Carlos Jurado
Go Pack Go!

P.S. Over a year reader, first time poster. Please excuse any writing or grammatical errors, as I mostly read or listen to English, I haven’t wrote a paper for discussion in a while :)

packeraaron's picture

This is brilliant - and for the record, I can't wait for the World Cup to get here. ;)

jerseypackfan's picture

Agree Aaron, The World Cup action is out of this world. I was stationed in Germany when they won the World Cup back in 1990. The entire country all partying at once. It was a one of the greatest experiences of my life.
As for this upcoming World Cup. I can't wait for the USA/England match. England is a total mess right now.

packeraaron's picture

USA/England is going to be World War III in my house...

FITZCORE1252's picture

Simple, whomever is rooting for England... by them a plane ticket so they can go live there.
I can't think of any reason for an American to root against America in any capacity (although it IS their right... as an "American"). It's just, how shall I say... UN-American.

GBP 4 LIFE

PackersRS's picture

Nice twist. First this is a stats post... The na douche makes it about soccer.

And now, out of the blue... TCHAM! BAMN! A patriotism topic!!

;)

packeraaron's picture

Um, my wife is English and all three of my daughters are half-English. Sending them to live England would, in a word, suck.

FITZCORE1252's picture

Haha,

I was just tryin' to get a rise. I figured it was something along those lines... if everyone liked the same team, sports would be kinda pointless.

Really though, you should raise your girls "correctly", if you get my drift. USA, USA, US...

GBP 4 LIFE

packeraaron's picture

They're Packer fans. That's all that matters.

Matthew's picture

Aaron,

on a realted topic…. my daughters are half-kuwaiti…my oldest (9) is a die-hard Packer and Brewer fan. it confuses my poor mother-in-law so much. We went to visit her in Saudi arabia last year and my daughter was talking all about Aaron rodgers and trying to explain her grandma all about football. My mother-in-law, who can’t speak a word of English was soo confused..lol

packeraaron's picture

That is brilliant Matthew.

FITZCORE1252's picture

True dat.

Matthew's picture

great post and so true.....I never ever liked futbol (south american/european) mainly due to the fact that I never grew up watching it...no atmosphere. But this changed when I lived in Istanbul for 3 years (very close to Fenerbace Stadium). Needless to say I was a convert after going to my first match. Atmosphere is very similar to a Badger football game (in the student section).....once you watch quality futbol matches it is an amazing sport. Also was fortunate to go see some matches down in Medillin, Colombia (Postbon). Great sport and world cup will be great!!.

Go pack and Cheers to all my Packer friends from around the world

PackersRS's picture

Okay, my last post about soccer... Turkey stadiums are the closest to a South American atmosphere. It's the fans chanting for 90 minutes nonstop. IT's the colors, the flags and the lights. It's beautiful.

Until someone throws a chair at the opponent's team, that is...

FITZCORE1252's picture

CJ,

Nice post. Nobody gives 2 shitz about spelling around here if you're making sense, which you certainly did. I totally agree with your take. Kudos.

GBP 4 LIFE

FITZCORE1252's picture

On a side note,

Any MMA fans here?

Tomorrow's card is gonna be f'ing S-I-C-K!

GBP 4 LIFE

aussiepacker's picture

Dude, Love the ufc. Went to see 110 when they came to australia and the atmosphere was amazing. Am looking forward to 111 and although gsp is a ledgend i am rooting for dan hardy and i also hope carwin knocks frank mir's teeth out.

Matthew's picture

GSP is amazing and one of my favorite athletes in any sports. Just a nice human being as well. I'm a huge MMA fan and have been to my share of fights..mostly supporting Rofus Sports out of Milwaukee. Eric "Red" Schaffer, Pat Berry, Razak Al-Hassan and up and commer Anthony Pettis all train out of there...and all packer fans :)

FITZCORE1252's picture

Matthew,

Good to know those guys are GBP fans, I'll pull for them.

I'm kind of the same way with the guys from the Pat Miletich camp. It's pretty close to where I'm from originally. He's had a lot of studs go through his doors... Matt Hughes, Tim Sylvia, Jens Pulver, Robbie Lawler, Mark Coleman, Spencer Fisher, Jeremy Horn... to name several.

Matthew's picture

FitzCore,

yeah Miletich is on a whole nother level with his stable of fighters. Go pack!

FITZCORE1252's picture

Very cool. I was so impressed by George Sotiropoulos' total "ownage" of Joe Daddy, dude's gonna be hard to deal with.

For me, gotta go with GSP. I just love the way the guy handles his "bitniss". Should be a great fight though, Hardy's game.

With you on Carwin. I've been looking for a Carwin vs. Lesnar matchup for quite a while.

Getting pumped.

WOOT!

Graffin's picture

G! S! P!

FITZCORE1252's picture

That was nothing short of TOTAL DOMINATION. I just wish GSP would have stopped it, nonetheless... ASS-WHOOP'N.

packeraaron's picture

I have to say, having finally had the chance to watch the whole thing, I agree with Polian almost 100 percent.

Jayme's picture

Well.. some people just appreciate stats more than others. I can't find it now, but I know that I've read that Belichick has a stats guy who spends the entire game examining statistics and probabilities of play calls and success rates. When push comes to shove and the game is tight, the only guy that Belichick wants to hear from is the stats guy. That's why he makes calls like going for it on 4th and 2 at the end of the game to try to close out the Colts, instead of punting and playing the field position game. Granted, it doesn't always work, but statistics aren't about what will work every time, it's what is most likely to work in a given circumstance.

Don't get me wrong, it's not all about stats, but they definitely can augment what a human can see.

http://freakonomics.blogs.nytimes.com/2009/11/16/bill-belichick-is-great/

bomdad's picture

Bellicheat does more than monitor stats during a game!

Graham's picture

Everybody lets keep this in perspective. Now Bill Polian knows what he does. However, there was a SERIOUS debat to go with Manning or Leaf. This guy still has a job because he guessed right. If he guessed wrong we would not be talking about him today.

So why these guys know more than we do, they should they spend 12 hours a day EVERYDAY on this stuff with better info, more at stake, ect, these guys are not the best and brightest in the world. They are just regular guys who have an eye for talent, a ton of money, or some other good attributes. However they are just guys/girls nothing more nothing less.

ZeroTolerence's picture

Statistics are like a bikini.
What's revealed is interesting, concealed vital.

NickGBP's picture

Just watched the whole video. Was more interesting from a business perspective. Great link, thanks.

Matthew's picture

great topic and something very close to my heart. I agree whole-heartedly with Polian and Nagler's thesis :)

Exhibit A) Aaron Kampman against Minnesota last year. I heard many people talk about how Kampman was coming on right before he got hurt. But really he was a complete no-show against a rookie RT in the 2 most important games last year for the Packers.

baboons's picture

Polian really shouldn't criticise stats analysis when he uses phrases like "pretty strong likelihood" and "going to have a good chance". When talking about "4th&2 gate". Likelihood based on what?

Good chance based on what? Stats or Gut? If it's gut, just say it. Don't try to quantify it.

Oppy's picture

Bingo. There are many people whom abhor stats, and down play them, but yet rely on statistical analysis for many crucial decisions and functions- whether they fully realize it or not.

The world runs on stats and data. Yes, there can be massive failure if the stats cited aren't in proper context or the data is incomplete, or even if things just don't work out the way the odds suggest is most likely. But then, the same can be said about a NFL scout/coach/GM's appraisal of talent from film study or watching a prospect play live.

Oppy's picture

I've been stabbed at by Aaron about stats in the past. However, I've never tried to used statistics in football as a predictor of future outcomes.

I think that often what we believe to be true is tainted by our perception at the time- we often "feel" like someone is out-performing someone else. This is why I like to see some numbers from time to time.

Last example where I cited numbers and was jeered for it; when everyone was saying Ahman Green was performing better than Ryan Grant, especially in the screen game. I understand WHY people FELT this should be true- Grant seemed lack-luster until Green stepped in, and Green looks more physical in the inside running game. He's always been regarded as a great screen RB, while Grant has been regarded as poor. But the numbers- records of what HAD transpired from last year, showed that the *average* production of Grant was better than the Average production of Green, even in the screen game. Of course, I was lambasted for the comments (I think Aaron commented on how many more attempt Grant has had compared to Green, which was why I used yardage averages instead of yardage totals.) My only point- Just as stats alone don't tell the whole story without strict context, sometimes what you perceive is tinged by personal bias as opposed to being purely what occurred.

Matthew's picture

Oppy, like the breakdown and I think stats are great to in addition to what Polian states. i agree with biases and how sometimes it can mess up our perspective. But there is also the point that stats can mess up reality also.

PackersRS's picture

I just brought soccer in because in soccer, and any non-US sports in general, stats aren't as big a deal as it is in US. I have no idea what's the avg possession % my team has, or what's it's rank in steals... I do know how many goals it's scored, who's the leading scorer, etc...

I'm never mentioning soccer again in football posts. Sorry, people!

FITZCORE1252's picture

Haha,

You are forgiven.

Asshalo's picture

Thank god this wasn't an ass kissing of industry insiders or an angsty rant on the under-valuation of outsiders.

packeraaron's picture

Is that what you've come to expect? (Hangs head in shame...) ;)

Ron LC's picture

Linear regression doesn't work for football. Variables, too damn many variables.
_____
Milano v. Madrid - guarenteed riot within the first five minutes.
_____
Austrailian football - mans sport! No one has any teeth and their nostrils are closer to their ear than the middle of their face. No concussions though!

Cuphound's picture

(1) I know it's shocking, but there's no reason that qualitative and quantitative methods can't be synthesized in a single analysis. Methodological diversity is a healthy thing.

(2) I'm deeply against Aaron's women being deported to Merry Old England. Plus, I've seen her wear a Packers cap in photos. Two years ago, she probably didn't ask questions like, "Are you sure you want to keep rooting for this team? They keep losing by four points or less." My partner is an American. That's one of his favorite questions. Of course, I don't want to send my partner to England either, unless there's a ticket for me and lots of sightseeing involved.

D.D. Driver's picture

This is the best post in the thread.

You are exactly right. I'll take it a little further and say that stats and film review play different and complementary roles. Stats give you a macro view. It's important to have a 30,000 foot view.

You wouldn't (and shouldn't) base a critical play call primarily on statistics. On the other side of the coin, you line up all the genius GMs in the league and have them predict next years' standings, and you compare those predictions to FOs predictions. The FO guys will kick their ass every year.

Let's also keep in mind: Bob McGinn knows 3 scouts that this time last year thought that Jake Delhomme and Vince Young were better QBs than Aaron Rodgers. Did the tape lie? Did these scouts discount statistical prodution a little too much?

Log in to comment, upload your game day photos and more!

Not a member yet? Join free.

If you have already commented on Cheesehead TV in the past, we've created an account for you. Just verify your email, set a password and you're golden.

Or log in with Facebook

Must Read

Quote

"A school without football is in danger of deteriorating into a medieval study hall. "
"The Bears still suck!"
"I firmly believe that any man’s finest hour, the greatest fulfillment of all that he holds dear, is that moment when he has worked his heart out in a good cause and lies exhausted on the field of battle – victorious."