Quick view of bracket model performance in past nine tourneys

Throughout the upcoming week, as teams punch their dance tickets, I’ll be assembling the massive 80-stat Excel sheet that I make available to Insiders late on Selection Sunday evening. Then some time around midnight, I’ll start cranking out a set of statistical models for filling out your bracket. Last year, I committed to doing 10, then wound up offering 15. This year, I’m shooting for 12. We’ll see how it goes.

Some of these models—like the higher-seed baseline strategy, KenPom, BPI and Nate Silver—are relatively easy to crank out. Others, like Final Four/Champ and Seed Match-up, take time…and tax my sleep-deprived, numbers-groggy brain. But I usually get them all taken care of by Monday afternoon. Usually.

In advance of the big stats-crunching week, I put together a handy-dandy chart showing the performance of the bracket models in the ESPN Tourney Challenge going back to 2005. The numbers refer to the model’s percentile achievement against all competing brackets; they aren’t a measure of pick accuracy. Here’s how the 14 models I’ve offered over years have fared:

2014_Model_Results

Three of the top four performing models are based on the indexes of popular statisticians—Nate Silver, Dean Oliver (ESPN’s BPI) and Ken Pomeroy. Of course, the Silver and BPI systems only have one year of results. And KenPom only accounts for the past six dances. In fact, if you take away the 2005-07 dances from Final Four/Champ, that model has beaten KenPom 82.7 to 76.2.

My Outcome Matching experiment has performed well in its two tourneys. This model “force-fits” the average number of upsets per round and follows historical in toss-up games to make the results look like a real bracket should. It’s a high-risk strategy that could just as easily go down in flames. So far, though, I’ve avoided disaster.

There are two models in this list that aren’t constrained by any statistical. One is my “From the Gut” picks and the other is my “Keeper Bracket.” The gut model is nothing more than my first impression of how I think the bracket will play out. I take no more than five minutes to fill it out. The keeper model is the bracket I play in my small pool with friends. Sometimes I go contrarian on that, but for the most part, this is the bracket where I try to meld the numbers and my instincts. Not surprisingly—and somewhat cruelly—the gut model performs significantly better than the keeper model.

What is it that Malcolm Gladwell said in Blink? “On straight-forward choices, deliberate analysis is best. When questions of analysis and personal choice start to get complicated—when we have to juggle many different variables—then our unconscious thought processes may be superior.”

Word.

This entry was posted in Bracket Tools. Bookmark the permalink.

14 Responses to Quick view of bracket model performance in past nine tourneys

  1. Tommy says:

    Looking through the 2013 Models, I didn’t see the Pomeroy or Silver models PDF. Was that posted anywhere?

  2. Tommy says:

    Excuse me, Silver was the only one I couldn’t find. Thanks.

  3. Blazers! says:

    I have an idea about Malcolm Gladwell quote as it happened to me. Back in 2010 march madness, I dreamed about the biggest upset of march madness of the time, where a green-colored team took down a dark-blue colored team (turns out it was 14 ohio over 3 georgetown but I couldn’t translate that because their jerseys were blank in my dream). In 2012, I dream looking at the final scores and saw 10 Purdue over 7 St. Mary’s (CA) 72-69, by three points. AND THEY DID WIN!! BY THREE POINTS (74-71)!! Last year, one person at my park bragged about his pick of Harvard over New Mexico and says my consious tells me to pick it.

  4. Ryan says:

    One question about the BPI strategy. Aren’t those stats inflated due to the fact that they reevaluate the teams after each round instead of filling out the entire bracket in advance? I remember one of your columns from a while back that indicated that BPI and Ken Pom strategies were less than 2% more successful than merely picking the higher seed.

    • ptiernan says:

      You are correct that I wrote a blog about BPI’s accuracy claims. That said, it was in the 94th percentile of the ESPN Tourney Challenge. That’s a different thing than it’s accuracy percentage in picking games. If you search on “BPI” you’ll find the updated blogpost on the system’s accuracy.

  5. Gary Diny says:

    Looks like the even years are when KenPom efficiency numbers really shine…93 or better. Oh Yeah its 2014!!!!

    Gary

  6. Ryan Tressler says:

    hey Pete, good to be back (though ive been checking blog posts for months, lol), I can’t wait for Sunday and the spreadsheet . . . gotta go for that perfect bracket this year (thanks Warren Buffett, haha)

  7. Mark S says:

    Any way to tie each models success to the level of Chitwood Madness score each year?

    • ptiernan says:

      Hmm…I’ll have to look into that. These numbers are always hugely affected by whether you peg the champion, especially in the ESPN Tourney Challenge, where points double each round.

  8. the maven says:

    Keeper @ 52%. Key Pete…..I have a game…..wanna play??? :)
    There are teams with no Champ Credentials at the moment that I won’t be able to leave behind.
    Maybe this year will be a point off the curve…..or maybe not.

Leave a Reply

Your email address will not be published. Required fields are marked *

* Copy This Password *

* Type Or Paste Password Here *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>