Forecast 2003

The results

By Tangotiger

This article will present the results of the Forecast experiment. New readers are invited to read the concept behind the Forecast series before continuing.

A special thanks to Sylvain Cognet for offering to compile, and compiling, the 2003 OPS and ERA for the 32 players in this study. (I love Primer readers! I sometimes get these surprises, where a dedicated reader will just compile useful data, and send it along. This was certainly a time-saver for me.)

This table presents each player's:

  • Baseline Forecast: average of last three years, with an age adjustment
  • Systematic Forecasters: average of the 6 forecasting engines (Palmer, Shandler, Silver, Szymborski, Tippett, Warren)
  • Primer Readers: average of the 165 ballots from the Primer readers
  • 2003: actual results
  • Next 3 columns (Absolute): absolute difference between the picks and actual, after adjusting for league average (e.g, a pick of .830 with a league expectation of .730, against an actual of .840 with a league average of .740 is a perfect pick, giving a difference of .000)
  • Last 3 columns (Relative): taking the 3 preceding columns, but comparing to the minimum differential among the 3 picks (e.g., if the differentials were .060, .050, .020 in the preceding 3 columns, then the values in these last 3 columns would be .040, .030, .000).

     
    AbsoluteRelative
    Player Baseline Forecasters Primer Readers 2003 Baseline Forecasters Primer Readers Baseline Forecasters Primer Readers
    Barry Bonds 1.231 1.248 1.268 1.278 0.052 0.025 0.006 0.046 0.019 0
    Jim Thome 0.979 1.03 1.019 0.958 0.016 0.077 0.065 0 0.061 0.049
    Gary Sheffield 0.949 0.952 0.939 1.023 0.079 0.066 0.08 0.013 0 0.014
    Troy Glaus 0.949 0.883 0.887 0.807 0.137 0.081 0.084 0.056 0 0.003
    Luis Gonzalez 0.934 0.931 0.871 0.934 0.005 0.002 0.059 0.003 0 0.057
    J.D. Drew 0.895 0.865 0.873 0.886 0.004 0.016 0.009 0 0.012 0.005
    Pat Burrell 0.895 0.892 0.956 0.713 0.177 0.184 0.247 0 0.007 0.07
    Moises Alou 0.869 0.816 0.816 0.819 0.045 0.002 0.001 0.044 0.001 0
    Richard Hidalgo 0.858 0.821 0.798 0.957 0.104 0.131 0.155 0 0.027 0.051
    Jeremy Giambi 0.84 0.872 0.898 0.696 0.139 0.181 0.206 0 0.042 0.067
    Sean Casey 0.808 0.786 0.766 0.758 0.045 0.033 0.012 0.033 0.021 0
    Roberto Alomar 0.797 0.776 0.793 0.682 0.11 0.099 0.115 0.011 0 0.016
    Jacque Jones 0.795 0.822 0.814 0.797 0.007 0.03 0.021 0 0.023 0.014
    Torii Hunter 0.79 0.837 0.843 0.763 0.022 0.079 0.084 0 0.057 0.062
    Rich Aurilia 0.773 0.79 0.798 0.735 0.033 0.06 0.067 0 0.027 0.034
    Rondell White 0.77 0.726 0.776 0.829 0.064 0.098 0.049 0.015 0.049 0
    Adam Kennedy 0.766 0.74 0.75 0.743 0.018 0.002 0.011 0.016 0 0.009
    Jeromy Burnitz 0.741 0.737 0.724 0.786 0.05 0.044 0.058 0.006 0 0.014
    Jeff Cirillo 0.74 0.689 0.723 0.555 0.18 0.139 0.172 0.041 0 0.033
    Jose Hernandez 0.717 0.831 0.836 0.634 0.078 0.202 0.206 0 0.124 0.128
    Marquis Grissom 0.673 0.717 0.692 0.79 0.122 0.068 0.094 0.054 0 0.026
     
    Freddy Garcia 3.41 3.807 3.86 4.51 1.2 0.63 0.57 0.63 0.06 0
    Javier Vazquez 3.41 3.665 3.61 3.24 0.07 0.5 0.45 0 0.43 0.38
    Ryan Dempster 4.19 4.62 4.56 6.54 2.45 1.84 1.9 0.61 0 0.06
    Kip Wells 4.32 4.227 4.1 3.28 0.94 1.03 0.9 0.04 0.13 0
    Chan Ho Park 4.59 4.697 4.64 7.58 3.09 2.81 2.86 0.28 0 0.05
    Matt Clement 4.6 3.843 3.95 4.11 0.39 0.19 0.08 0.31 0.11 0
    Jamey Wright 4.76 4.887 5.1 4.26 0.4 0.7 0.92 0 0.3 0.52
    Aaron Sele 4.77 4.647 4.83 5.77 1.1 1.05 0.86 0.24 0.19 0
    Shawn Estes 4.91 4.337 4.79 5.73 0.92 1.32 0.86 0.06 0.46 0
    Kenny Rogers 5.35 4.41 4.52 4.57 0.68 0.08 0.03 0.65 0.05 0
    Todd Ritchie 5.62 4.768 5.16 5.08 0.44 0.23 0.16 0.28 0.07 0
    League OPS 0.76 0.75 0.751 0.755 0.088 0.093 0.108 0.016 0.022 0.031
    League ERA 4.49 4.312 4.31 4.39 1.33 1.15 1.16 0.28 0.16 0.09

    The last 2 line needs a little explanation. The results under the "absolute" columns are the standard deviations of the values in those columns. The results under the "relative" columns are the average of the values in those columns.

    Interpreting the Absolute differences

    Jumping straight to the standard deviations, we see that the Baseline forecast did the best job with the hitters and worst with the pitchers. The Primer readers did the worst with the hitters. The Systematic Forecasters just beat out the Primer readers for pitchers.

    You can go through the individual picks in these 3 columns, but I find it more interesting to look at...

    Interpreting the Relative differences (Pitchers)

    We see here that the Primer readers just nailed the pitcher forecasts. Of the 11 pitchers, the Primer readers had the closest pick in 7 of them. In 2 others, they were within .10 runs from the closest forecast pick. Primer readers blew it on Javy Vazquez and Jamey Wright.

    How did the Systematic Forecasters do with their vaunted research and engine? They picked 2 pitchers better than the other forecasters. Another 6 were within .20 runs. They blew it on 3 pitchers, including Javy Vazquez.

    And the Baseline, the thought-free system? They were closest on the 2 pitchers that the Primer readers misread the most (Vazquez and Wright). They had another 2 within .20 runs. And the rest were just bad. The 3 worst picks out of the whole group were all from the Baseline: Freddy Garcia, Ryan Dempster, and Kenny Rogers.

    There is a certain amount of information about a pitcher that is not captured in the stats, especially ERA. The Readers and Forecaster knew that there was something a bit more special about Clement that wasn't captured, for example, and they made their picks accordingly. Javy Vazquez, a pitcher near and dear to me, for some reason did not elicit the same faith from the Readers or Forecasters.

    All in all, Primer readers did a great job in forecasting the pitchers.

    On to the Hitters

    It seems that the amount of intuition required with pitchers is not at all required with hitters. The lesson here is: trust the numbers.

    For best picks, the Baseline nailed 9 of the 21, the Systematic Forecasters got 8 of them, and the Primer readers just 4. If we look at the blown picks (difference of greater than .050), the Primer readers blew it on 6 hitters (Gonzalez, Burrell, Hidalgo, Giambi, Hunter, and Hernandez), while the Baseline blew it on 2 (Glaus, Grissom), and the Forecasters on 3 (Hernandez, Thome, and Hunter).

    Overall, the Baseline did the best job with the hitters, both in terms of the absolute and relative differences, and the Primer readers the worst.

    Recap

    The 32 players were chosen because they had at least 3 years of performance to analyze, and that their year-to-year performance was very inconsistent (whether by luck, design, or injury). The test in this experiment was to see if the Systematic Forecasters would be able to interpret the numbers in a special way, or if the Primer reader's gut feelings would see something extra. Among the hitters, both of these groups failed to see anything beyond what the numbers said. In fact, they saw things that just weren't there. Among pitchers, the peripherals found by the Forecasters, or the "stuff" established by the Primer readers was enough to thoroughly beat the Baseline.

    Conclusion? Trust the numbers you see (which is enough for hitters), and fill-in the information missing (which is the case with pitchers, be it health or mechanics). Any extra nuance that you find just doesn't have the impact you'd hope. None of the 3 groups dominated the others. This was about as close to a draw as you'd expect.

    Next week, I'll look at the picks of 165 Primer readers, and crown a champion. I'll also break down the picks of the 6 Systematic Forecasters.