But hmm… what about just an arbitrary polynomial (or more complex equation) fit? I need to go learn non-parametric statistics, I think.

(Also, when I wrote “reduces the R^2 by only 0.79″ I clearly meant “by only 0.0079″ >.>)

The problem essentially comes down to the fact that I’m trying to extrapolate from a data set (and it’s quite an extrapolation, I might add!), which we of course know is something you have to be very careful when doing (and I wasn’t).

As you mentioned, the tail seems to be decreasing faster than an exponential distribution, which makes it difficult to fit a curve to. I looked as several other distributions (such as Poisson, which decreases much too quickly, and weird distributions like a Weibull, which doesn’t decrease quickly enough) before settling on the modified exponential. I was really excited after seeing the torus results because it just seems like an exponential should describe the lifespan frequencies — I’m a little curious and would be very excited to see a distribution that works better on the plane.

I don’t really see why fitting a continuous distribution to a discrete one would be a problem though – if you just restrict a continuous distribution to the integers, you get a discrete distribution that behaves essentially the same way. This sort of thing is done quite often in statistics (probably most frequently when approximating a binomial distribution by a normal distribution).

As for independence – yeah, that’s a toughy, and independence is probably a bit of a stretch on the plane. While I’m fairly comfortable with the assumption of “almost independence” for the majority of low-lifespan patterns, it gets harder to picture what’s happening for high-lifespan patterns, and it’s quite possible (probable, even), that the independence assumption bumped up my maximal lifespan estimate by a fair bit.

For one thing, it’s not very stable – switching the exponent applied from 0.75 up to 0.8, for example, reduces the R^2 by only 0.79 (to 0.978), which is still a perfectly legitimate result, but changes the equation to y=1550.4e^(-0.032x), which results in a maximal lifespan estimate of just 86371. Conversely, choosing 0.7 (and discarding some data so excel would be able to make an exponential trendline) gets you a lifespan of 152820.

For another, I have to wonder if your improvement in R^2 from going to 0.75 is not largely the result of the aforementioned discarding of data. Using the same dataset (as is chosen by default in the sheet), but with an exponent of 1 gets me R^2=0.9475, which is worse, but by no means terrible. Specifically, it would seem to me the data thus discarded suggests that the tail is thinner than the distribution would suggest, which would lead me to expect a lower maximum than these projections predict.

Finally, I’m somewhat leery of casually applying a continuous distribution to a discrete data set like this. It would be nice to try fitting some discrete distributions to it (notably Poisson, which is highly related to the exponential anyway) to see if any of them work well. Although I’m unaware of a program that would do this for me, so this is staying at the level of speculation for now.

(I’m also not entirely convinced by the assumption of independence, especially for the plane. Although one potential way to look at the matter is that if the exponential distribution applies to it, then the assumption probably holds, and if it does not, neither does the assumption.)