See copyright notice at the bottom of this page.
List of All Posters
Silver: The Science of Forecasting (March 12, 2004)
Discussion ThreadPosted 8:54 p.m.,
March 12, 2004
(#10) -
Ex-Ed
Tango, we don't know enough about how PECOTA works to answer the question. The number of PAs should affect the forecast standard errors for the usual reasons, but imagine that the models for the four different player types shown by Nate vary in terms of their predictive power and hence by the standard error of the esimtate. If so, then the forecast standard errors will vary by PA *and* by player type.
But we really don't know, since PECOTA is a black box.
Silver: The Science of Forecasting (March 12, 2004)
Posted 9:43 a.m.,
March 13, 2004
(#14) -
Ex-Ed
I've beefed on the similarity issue before, but one of the many things we haven't seen is that the model works better by using similar players. That is a strong assumption, and by definition, non-similar players give you information about each other.
Say you have one independent variable, e.g. By selecting similiar players, you are selecting on the independent variable, restricting the range of your x's, and lowering your r-squared or your s.e.e., or whatever fit statistic you care about. That's research design 101.
Silver: The Science of Forecasting (March 12, 2004)
Posted 8:49 p.m.,
March 14, 2004
(#24) -
Ex-Ed
Regarding similiarity scores, we know that PECOTA uses ANOVA to identify predictive components (BP 2003).
How it uses ANOVA or why it uses ANOVA rather than factor analysis, cluster analysis, MDS, or other more tradtional techniques, we know not.
Whether player i's comparables are weighted by similarity to i in making the forecasts, we don't know. But it has always seemed to me that the knife edge assumption of comparable/not comparable is a very strong one.
Silver: The Science of Forecasting (March 12, 2004)
Posted 1:55 p.m.,
March 15, 2004
(#28) -
Ex-Ed
Regarding these last two posts, I think both tango and wally are right.
Having said that, Wally does not need to remind us that nate does not need to reveal the ingredients in his special secret sauce in response to what I am about to say.
From the various PECOTA writeups, it's not clear that the playing time forecasts are estimated simultaneously with the forecasted rate stats. If they are, then one would expect to see greater forecast variance for pitchers with fewer forecasted BF. But if they are not, and the rate stats and the BF stats are forecasted independently, then multiplied together to get counting stats forecasts, then we wouldn't necessarily see the reliever/starter forecast standard error patterns that tango rightly expects to see.
Silver: The Science of Forecasting (March 12, 2004)
Posted 7:51 a.m.,
March 16, 2004
(#44) -
Ex-Ed
I raised this in an earlier pecota thread, but testing the utility of the measures would be very easy if one had all the data.
1. 2 x 2 chi-squared test of independence
improve/~improve x predicted improve/~predicted improve
2. simple regression/difference in means for collapse
dep var = % chance collapse
ind var = collapse/~collapse
3. repeat 2 for breakout.
In all three cases the null would be that having the measures in hand gives you no more information than a coinflip (where p can vary).
Sophomore Slumps? (March 23, 2004)
Posted 3:32 p.m.,
March 23, 2004
(#9) -
Ex-Ed
"For a large enough group of players (in order to smooth out the random fluctuations), any above average season is ALWAYS followed by a decline, and any below average season is always followed by an "improvement." Period! It doesn't matter whether you look at ROY players, MVP players, MVP runners up, best players on the team, silver sluggers, etc., etc., etc.!"
Regression to the mean is a group phenomenon, not an indivdual phenomenon, so one should not say that "any above average season is ALWAYS followed by a decline."
Rather, the right way to say it is that "for any group of players that experienced above average seasons, on average those players will tend to decline in the following season."
Sophomore Slumps? (March 23, 2004)
Posted 9:39 a.m.,
March 29, 2004
(#21) -
Ex-Ed
Anyone see this non-answer in Davenport's chat? Pretty defensive.
***
Chris (North Carolina): Clay, every year EQA shifts around, making it impossible to determine its accuracy, predictive power, and overall value. When are you going to come up with a measurement that doesn't shift every year? And is BP abandoning EQA in favor of other, newer, better measurement tools?
Clay Davenport: I think you are confusing EQA with DTs. The formula for EQA hasn’t changed in five years at least, and the formula is published and readily available. The translation procedure, however, does change, as I test different procedures, discover old biases, and generally learn things I didn’t already know to try to incorporate it. In addition, some of the inputs to the program change from one year to the next. The park factor I used for San Francisco in 2003 is currently based on an average from 2001-2003; in next year’s book, the 2003 park factor will be an average of 2001-2004, and it won’t be until after 2005 that it will become a stable value. Likewise, the difficulty rating for leagues is reassessed each year, as I can now see how players who left that league did, not just players who came into that league.