DMB/SAT Repository

Thread on DMB forum regarding splits and projections disks [click for full thread ] :

dmbluke

At this point I just don’t know if we’ll do [a projection disk] or not. It’s not a big seller but it does generate a lot of PR and buzz.

….

Baseball4Good

If we get V10 and 2008 season disk out as planned, one of us can always put out a projection disk or can get a zips projection from Dan Szymborski at the baseball thinkfactory website.

jfish26101

Don’t get me wrong, I’m happy they decided to continue DMB. It’s just that without the season disk, I’ll have no reason to use it. I don’t like the seasons disks and don’t have any desire to play past season disks. That is just sort of the way I feel. Also while Szymborski’s disk is an option, his projections don’t have splits and rely on the default splits that aren’t accurate…well accurate as in everyone has the same modification for their splits as it was explained to me. Nothing against him but taking .10 off your BA and a little bit here and there off the other stats for every single player isn’t my idea of splits.

TedRoethke

I’ll just repost a link to part of a thread regarding splits and projections for the benefit of folks who might be spooked off by the issue, and that’s the last I’ll say about the subject.

http://dmbforum.yuku.com/…-anyone-.html#reply-41586

diesel

iverson3ai wrote:

Last time I looked, Dan’s projections don’t account for splits, and all the other detailed areas that DMB focuses on.

That’s not entirely true. The ZiPS projections use DMB’s built-in generic splits which provide realistic platoon splits based on historical norms. In addition, Dan makes adjustments for switch hitters where appropriate.

DSzymborski wrote:

I’ve always taken it for granted that any split created with overall would have the event table apply generic historical splits as that’s always been my experience - I’ve played a ridiculous amount of DMB. If I thought that “Overall” generated an event table was the same for lefties as righties, I’d definitely make more of an effort to project splits. The only thing I’ve done differently this year (though not for the first release) is regenerate tables for starting switch-hitters with strong platoon splits (like Orlando Hudson) by changing them to the side of the plate they’re in tune with, regenerating the event table, and then putting them back to switch.

What other detailed areas are you referring to?

Daniel Evensen

I prefer the generic splits to real life ones, for both projection disks and season disks. It’s a sample size thing.

3rdandKing

The smaller the sample size, the less accurate the results. Is there a cutoff point where it makes more sense to use overall stats than splits? For example, if you have 75 at bats or less, it’s better use overall, but over that it’s ok to use splits?

diesel

3rdandKing wrote:

The smaller the sample size, the less accurate the results. Is there a cutoff point where it makes more sense to use overall stats than splits? For example, if you have 75 at bats or less, it’s better use overall, but over that it’s ok to use splits?

For player’s that you either create or modify yourself, DMB doesn’t make any adjustments to the event tables to account for sample size.

What you see is what you get.

David

diesel

Update:

I checked with Luke about this and it turns out that the game does do some smoothing for user created/modified players with limited playing time by adjusting their internal event tables. It was my belief that this was only done during the creation of DMB’s season disks.

This smoothing applies to batters and pitchers and is more drastic the fewer the at bats (or batters faced) a player has. Someone did a study awhile ago that zeroed in on the usage threshold that DMB uses as the cutoff. I think it was somewhere in the area of 30-40 PAs or BFs before you could expect to see a player perform at the level of his stats.

David

DSzymborski

The smaller the sample size, the less accurate the results. Is there a cutoff point where it makes more sense to use overall stats than splits? For example, if you have 75 at bats or less, it’s better use overall, but over that it’s ok to use splits?

There is a cutoff point where L/R split breakdown starts to predict future L/R split breakdown better than a generic L/R split breakdown does, but it’s much, much farther out. Andy Dolphin did the legwork and found that the point where this happens is well into the thousands of at-bats (4-to-5 thousand overall at-bats for hitters).

After the last discussion about this (at least that I was involved in), I went through the hitters and calculated, based on size of splits and sample size, which hitters had platoon splits and/or sample sizes large enough to make the actual platoon splits the better idea (the less at-bats, the larger a difference in splits you need for it to be relevant and vice-versa). I ended up with 13 hitters.

jfish26101

Whatever you say but I’d rather have the splits. There are LHed hitters that hit LHP better and vice versa not to mention some hit them equally well while others are horrible against one or the other. Generic splits are not realistic no matter how you wish to spin it. There is no telling whether the splits DMB puts on the disk are realistic either but I think on a player by player basis it’s more probable than using the same generic splits for every player in the database.

clueless

jfish26101 wrote:

Whatever you say …

Heh. With three words, ~1000 words of quality info are succinctly dismissed.

Ted, thanks for culling and formatting all of that. Interesting, informative, and thought-provoking.

jfish26101

I didn’t dismiss it, I just don’t agree with it and have read the conversation before. You can’t tell me using basically the same method for all the splits of every player on the disk is more accurate then looking at their splits and trying to project out what might happen. Dan makes those disks for fun and for free for the public and while I appreciate it I prefer the disk DMB puts out. Sorry.

DSzymborski

You’re missing the crux of the argument. As a factual matter, until a player has 4-5 thousand total at-bats, generic platoon splits more accurately predict the future platoon splits of an individual player better than actual past platoon splits by the player. So by definition, a database that uses generic platoon splits is going to more accurately predict platoon splits of the players in the database simply because the vast majority of players with platoon splits that differ from normal don’t have enough of a career to determine objectively if their platoon splits are predictive.

jfish26101

No, they wont be more accurate necessarily, you are simply saying you don’t know whether their RL splits are accurate or not so why put in the effort of trying. How can you say subtracting .10 off their BA and adding .10 to their BA depending on what arm the pitcher they are facing uses is more accurate? I don’t really care what sort of research you used, it simply doesn’t make sense. I’m fine with you saying you wont know how to truly judge what a player is going to do till 4-5k ABs so you have a large enough sample size to make a real educated guess but that doesn’t mean none of what they do until then matters. You can keep saying that all you want until you are blue in the face but your simply wrong. Just because you feel it isn’t a large enough sample to truly guess what their splits will be doesn’t mean ignoring the data makes it more accurate. I’m sure if a coach in RL were to base his lineup decisions on the same logic you keep repeating it wouldn’t work out to well. Heck BC does, he benches Kelly Johnson often when a LHP is on the mound and yet I believe his numbers have been better agaisnt LHP for 3 straight seasons. What sort of sense does that make? Bench a lefty is on the mound simply because the batter is left handed as well despite having better numbers against them? Perhaps when you look at the whole sim and every single player in it, maybe the projections would be closer using generic splits but I doubt it. It’s a freaking projection disk anyhow and projecting every player to have the same effects when facing RH or LH pitchers is ridiculous and I’m positive less accurate on a player by player basis.

DSzymborski

No, they would be more accurate, by definition. The reason we don’t know whether or not they are accurate is because the sample size is too small for it to be useful as a predictive manner. It’s a matter of mathematics, not personal preference.

If I flip a coin 10 times and it comes up heads 7 times, that’s not enough of a sample size to conclude that the coin is weighted. So if I’m predicting what the next 10 coin flips would be, I would still guess 5 heads and 5 tails as being the most likely. By the logic you’re employing, we’d predict 7 heads because we want to put in the effort of trying. Players aren’t coins, obviously, but the fact is that platoon split departures from generic act like coins until quite a long-term.

jfish26101

Haha

It’s amazing what can pass as facts these days. I’m no mathmatician, perhaps you are, but I do understand the concept of a small sample size. Just because someone said you can’t guess platoon splits accurately until ______ doesn’t mean you completely dismiss what data you have which is exactly what you are doing. The majority of the projection disk is based off of small sample sizes. There are players every year that are on the disk that don’t even have a single pro AB or IP. The fact that a sample size is small doesn’t disprove the data which is basically what you are saying. You are saying that it is DEFINITELY more accurate without a shadow of a doubt on a player by player basis to take every single player on the disk and use generic splits? That that is a fact?

Edit: By the way, this is the last time I will comment on your disk. I had this discussion on your site last year as well as this forum I believe. When I asked my league which disk they would prefer, every single person who commented said DMB’s disk. When asked if DMB doesn’t make a disk, is your disk OK? A few said yes but more said they would rather look into using a different product. /me shrugs

Edit 2: Also let me add I’m not trying to be a dick or trash your work. I’m just saying I’d rather have different splits for each player by looking at what they have done in both the minors and majors just like you would do with the other stats.

DSzymborski

You are saying that it is DEFINITELY more accurate without a shadow of a doubt on a player by player basis to take every single player on the disk and use generic splits? That that is a fact?

Of course not. That’s not how probability works.

Also let me add I’m not trying to be a dick or trash your work. I’m just saying I’d rather have different splits for each player by looking at what they have done in both the minors and majors just like you would do with the other stats.

I’m not even talking about the disk. I’m talking about the predictive value of actual platoon splits vs. generic platoon splits. If DMB did generic platoon splits and I did platoon splits based on actual history, then I’d be the one to be wrong.

You’re not being a dick, but you’re certainly being dense in a discussion of how probability and predictive value works. It’s not a democracy or a popularity contest, it’s a question of math and looking at what happened in actual baseball games. And as such, it’s a matter of fact, until a player has several thousand at-bats, actual platoon splits will predict future platoon splits with less accuracy overall than generic platoon splits and when you’re only talking a couple of seasons, using a player’s actual platoon splits for any kind of predictive exercise is essentially mathematical lying.

3rdandKing

I’m not getting into the argument between Jfish and Dszm. I just am curious about the real-life outliers. For example, Russell Branyan in 2008 was 33 for 118 against right handed pitchers, a .280 average. He was 0 for 14 against left handers. In my 2008 projection season, I’m using Branyan’s real life splits. He’s 4 for 14 (.286) against lefties and only hitting .226 against righthanders. What kind of performance would one expect from Branyan based on generic splits? If I had put in generic splits, those number might be reversed. And what about numerous batters who hit better against their own hand pitching than opposite?

JB50

Let’s take Dan’s example of a coin flip a bit further, perhaps that will clarify what he is saying.

If you flipped a coin a thousand times, and the results were 543 heads and 457 tails, what would the odds be for the next thousand flips? 500 and 500, of course. Now you don’t actually expect to get exactly 500 each. In fact, if you do, you’re rather amazed. But the “predictive value” of 500 and 500 has the most likelihood of being closer to the actual result of the next thousand tosses than the 543 and 457 from the previous thousand. Anyone who persisted in saying we should go with the 543 and 457 would be unreasonable. So when you are dealing with small sample sizes like a single season of MLB batting splits, even for full-time players, it is more reliable to us generic splits for players with less than 4-5,000 at bats. That’s what Dan is saying.

DrArbiter

Actually, it’s much worse than that, for three reasons.

In doing splits, what we are really doing is computing the difference between two samples. So, a better analogy would be that we have two coins, say of different types, and we want to find out if one comes up heads more often than the other. We flip each one 500 times, and take the difference in proportions of heads. The magnitude of the noise in this experiment is several times that of the one JB50 describes, for two reasons. First, because with each coin we are flipping only half the original 1000, our observations of each coin are noisier. But, when we take a difference of the (noisily observed) counts from the two coins, the noise in the difference is the *sum* of the noises in the observations of the individual coins! (Formally, the variances are added together.) Both those facts increase the uncertainty in our observation in determining which coin comes up heads more often.

Now, to bring that back to splits. We already start with a rather noisy estimate of a player’s true skill level. (See the thread on “season disk distortions” that was bumped a few weeks ago.) Then, we split up our observations of that player into two categories, which, by definition, will have smaller sample sizes, and therefore more noise. Then we are implicitly taking a difference between those observations — which adds even more noise.

And, then, there’s one more practical kicker. In the coin-tossing experiment above, we tossed each coin an equal number of times. That is optimal (in terms of minimizing the noise) if we don’t have any knowledge which coin is in fact the one that would come up heads more. If the 1000 tosses are divided unequally between the two coins, the noise in the difference of proportions of heads increases. And, that’s just what typically happens in baseball; most players don’t have an approximately equal number of plate appearances against pitchers (or batters) of each hand.

Add that all up, and you get the conclusion that Dan has been citing: We just don’t have nearly enough observations to reliably estimate splits. They’re not entirely noise, but there’s a lot of noise in them — way, way more noise than even in whole-season non-split statistics — and the most reliable predictor of future splits is something which, except for long-career players, is very close to the “neutral” split.

3rdandKing wrote:

And what about numerous batters who hit better against their own hand pitching than opposite?

They don’t exist. Or, at least, they’re not numerous. An implication of what Dan’s been saying (and I outline the essential intuition behind in a previous post) is that the noise in single-season, or even multiple-season, splits is huge. The noise is so huge that it is very easy for even a guy who has an above-average platoon differential to post a reverse differential in a single season. That doesn’t mean he actually hits his own hand pitching better; it’s just luck.

On the other hand, there are a few pitchers who do seem to tend to be more effective against opposite-side batters. For instance, take Bruce Sutter:

http://www.retrosheet.org/boxesetc/S/Lsuttb0010.htm

However, even Sutter’s career isn’t long enough that we can conclude that lefties “really” hit him for a lower batting average than righties (although it’s very close to being long enough).

DSzymborski

On the other hand, there are a few pitchers who do seem to tend to be more effective against opposite-side batters. For instance, take Bruce Sutter:

It’s important to note screwballers, too, who have a very strong tendency to have a reverse platoon split for simple mechanical reasons. The break-even point for pitchers seems to be significantly lower.

I should also note that it’s not just sample size, but magnitude as well. Very large unusual platoon splits require fewer plate appearances to validate while ones that differ only slightly need even more - I’m not sure if anyone’s ever played enough to demonstrate that a split 10 points of OPS smaller than normal is predictive (it’s too late at night for me to start dragging Statistica into this).

My point remains that for the vast majority of players, either their playing time is too short or the platoon splits are close enough to normal that it’s simply not worthwhile to try and predict their platoon splits from past platoon splits.