[Broken Stick Rule] - A New Kind of Science: The NKS ForumA New Kind of Science: The NKS Forum
Pages:1
Broken Stick Rule
(Click here to view the original thread with full colors/images)
Posted by: John Reith
In 1968 I wrote a paper for publication, titled "The Generalized Distribution Rule", while working at E.I. DuPont (retired in 1981, now 84 years old). This rule is long known in statistics as "the broken stick rule". My contribution was to show that its predicted outcome for competitive activity closely fit the observed distributions that I observed in every field for which I could get data. Most of my searches were concerned with market share distributions in various SIC industrial categories, as a function of N, the number of competitors, and incomes. These "quantized" outcomes are strikingly non-uniform. The work was never approved for publication by my supervision since "it questioned the whole role of DuPont management".
This rule gives the expected share if any magnitude (or good) is randomly divided into N parts and the largest down to the smallest shares assigned to the presently existing rank order of largest to smallest competitor: the industry leader would get the largest share, the 2nd rank would get the 2nd largest share etc. Repeated application of this random division leads to the final expected values. I have a 9 line program that gives these expected shares as a function of N, the number of competitors. After reading "A New Kind of Science" I realized that the broken stick rule (BSR), like the rules cited in the NKS book are all "survival of the fittest rules". But I believe my observation that the BSR rule matches so closely the world we live in suggests that it should be tested by NKS graphic procedures. I offer the following suggestions: the meaning of "fittest" is a synonym for present size, so a random division of a new magnitude would be assigned in order of existing size, i.e. the number of existing squares of a particular color. New competitors (shown as new colors) would be generated by some random process consistent with the historical record for the appearance and disappearance of new species in our existing world. Mating within a species would be similar to mergers in industry. There is another underlying statistical rule, often labeled as "the theory of long runs" (see William Feller's "Introduction to Probability"). Once a particular competitor secures an advantage (for example a lead in the flipping of an unbiased coin) he will maintain that lead for extremely long time periods. Until I realized this connection I was puzzled why market shares in given industries are suprisingly stable for long periods.
Wolfram's suggestion that the presently observed complexities in this world are the result of a few, simple rules is consistent with my own observations in some 37 years of research. My stated conclusion in a course I briefly taught is that most complex outcomes, once understood, never require more than 3-5 independent variables:color, DNA, space, mechanics, electricity etc. I would appreciate comment on the points I have raised.
Posted by: Jason Cawley
Others have noticed some related phenomena. For what they are worth, here are some links to a few of them I've run across before -
http://www.cut-the-knot.org/do_you_know/zipfLaw.shtml
http://mathworld.wolfram.com/BenfordsLaw.html
http://mathworld.wolfram.com/Heads-...stribution.html
http://mathworld.wolfram.com/topics/RandomWalks.html
Anyone else have more on the subject?
Posted by: Gunnar Tomasson
I just checked out John Reith's posting and, in response to Jason's question, Anyone else have more on the subject?, it occurred to me to re-post to NKS a message on the Frequentist View of Probability which I posted in September 2002 to Gang8 and PKT.
The message reads as follows:
Yesterday, I had occasion to revisit Keynes' 'Treatise on Probability', in which he challenged the Frequentist View of Probability and which Bertrand Russell described late in life as "an extremely able" work on the subject matter.
In short order, I formulated a simple thought experiment whereby elementary physics are shown to debunk the Coin-flipping version of the Frequentist View.
In view of the importance which Keynes attached to the subject matter, and considering the essential role which the Frequentist View of Probability plays in much of modern economic theorizing, I am sketching below a brief outline of my argument.
1. An Internet site offers the following definition of the 'Relative Frequency Interpretation of Proabability': "Recall that we defined the probability of event A as P(A) = # of ways A can occur/Total # of ways anything can occur. This is called the classical probability definition. Another way to interpret probability is as the long-run relative frequency (long-run fraction) of the event. That is, if I flip a fair coin hundreds and hundreds of times, the fraction of heads will be very close to 0.5. The more I repeat the experiment, the closer to 0.5 the relative frequency will be. This is the same result the classical definition gives us. The relative frequency interpretation of probability works especially well for repeatable events, e.g., flipping a coin, rolling dice, drawing cards etc."
2. "Why," I asked a supporter of the Frequentist View of Probability on an Internet Forum, "is the set of possible outcomes in the toss of a perfectly balanced coin specified as H + T, where H = Heads and T = Tails, rather than H + T + E, where E = Edge?"
3. He replied: "In abstract mathematics, the sample space is {H, T}, but in the real world it's {H, T, E}."
4. My response was as follows:
"Is that so?
a. Consider a perfectly balanced coin suspended in a contraption at the top of a perfect vacuum such that it is in perfect alignment with the vertical.
b. Now let the contraption's grip on the coin be relaxed so that the coin falls down.
c. Given the assumed vacuum and absent any other 'non-coin' related factors, why should the coin not land on its edge?"
5. Within a few minutes, a physics-literate third forumite provided the following input:
"If you assume that the coin "falls" then you must also assume that there is a gravitational force pulling it down. When the coin reaches the bottom or end of its fall the nearest point of the coin will stop first transmitting the energy of the coin into the "bottom" from the bottom of the coin to the top of the coin. I was trying to prove that the coin would fall over because the coin would try to compress from bottom up and since the coin can not compress or absorb all the energy released when it came to rest it would fall over, but I was wrong. It would remain on its edge. This could not be proven in a laboratory setting because you would need a perfectly uniform coin so that the energy released was perfectly released in a uniform way."
6. To which I replied:
"Thanks!
It may have taken you all of five minutes to reason your way through the physics of the thought experiment."
Which is readily shown to imply that the H/T Frequency Distribution is a function of the non-coin aspects of Coin-tossing.
7. As background, I had posted Harrod's summary of the key points made by Keynes in 'Treatise on Probability' (The Life Of John Maynard Keynes, Penguin Books, 1978, pp. 772-75). With reference thereto, another forumite asked: "What was Keynes's point?"
8. I referenced the above summary and added:
"Now, why did Keynes - and Whitehead, Ramsey, and Russell - either reject or question the logical adequacy of the Frequency Approach?
"The answer can be formulated in any number of ways - the key point at issue concerns the Humean view of the relationship between theory and practice which, in the case of economics, Keynes summarized as follows in 1922:
The Theory of Economics does not furnish a body of settled conclusions immediately applicable to a policy. It is a method rather than a doctrine, an apparatus of the mind, a technique of thinking, which helps its possessor to draw correct conclusions.
The distinction drawn here by Keynes between 'theory' and 'practice' mirrors that between pure and applied geometry - a distinction which has disappeared from much of modern physics and, apparently, has yet to appear on the NDS [Neo-Darwinian Synthesis] stage.
As for the Coin-tossing approach to the logical foundations of the Probability Calculus, it is nonsensical in the sense that an attempt to formulate the propositions of analytic geometry in terms of empirically observed lines, circles, and points is nonsensical."
The thrust of this argument is familiar to Gang8 and PKTers - until yesterday, I had not thought of relating it to Keynes' challenge to the Frequentist View of Probability.
Gunnar
Posted by: Karl Smith
I am not exactly sure what you are trying to say.
I don't think anyone is trying to say that if you took an actual coin and flipped it an increasing number of times the frequency will tend towards .5/.5.
For one thing actual coins are not fair. For another, actual coin flips are not fair.
The construct "fair coin" is meant to describe a phenemon that may not have an actual analog in the real world.
Posted by: Gunnar Tomasson
Re. the following:
I am not exactly sure what you are trying to say.
Comment:
The physics-literate fellow was exactly sure what I was trying to communicate.
Posted by: Karl Smith
I understand that if you in fact conducted the experiment that under ideal conditions (floor being perfectly flat, uniformly elastic, etc) you would in fact find that completely symmetrical coin would land on its edge.
However, what does that have to do with flipping a fair a coin many times?
Posted by: Gunnar Tomasson
Re. the following:
However, what does that have to do with flipping a fair a coin many times?
Comment:
The point at issue concerns the LOGICAL foundations of Probability Theory.
Flipping a coin n times sheds no light thereon.
Posted by: Karl Smith
Right but the fair coin analogy is simply an example.
It is to say that if one had a coin which was fair, that is having equally probability of falling on either heads or tails and that heads or tails exhausted the set of options then one would find that the law of large numbers holds.
If for example you take a coin a flip it many times and you do not observe any sort of movement towards .5/.5 then your coin is not fair.
Maybe I am totalling missing your point here.
I'd just like to note in an attempt to clarify that the law of large numbers is not axiomatic to probability theory. It is a consequence.
Posted by: Karl Smith
I've researched more into what you've been discussing
I see now that it refers not to formal probability theory but to the philosophical issues surrounding why we might observe probability theory holding in real life.
Thats is certainly a hard question because it doesn't really allow us to the mathematical conclusions of probability theory.
I could easily say that a fair coin, if one exists, would show certain properties. If you were then to give me a coin that did not satisfy those properties I would say, it must not be fair.
However, that leaves one unsatisfied if the search is for the applicaibility of probability theory. What good is probability theory if there are no fair coins?
I would offer the thought though that perhaps when I say, X is random, what I am saying is that X has no disernable pattern. If X has no disernable pattern then in practice the empirical law of averages must hold.
Why? Because if it didn't hold then X would have a pattern.
So, and perhaps this is dangerously close to Subjective Probability, when I say X is will occur with probability .3 I am saying that I believe that IF I were to conduct this experiment a thousand times I would see X occur approximately 300 times.
Why do I believe that? Because hitherto I have not been able to establish any pattern associated with X.
Posted by: Gunnar Tomasson
Indeed, the point at issue concerns "the philosophical issues surrounding why we might observe probability theory holding in real life".
As such, the point concerns the broader epistemological issue of the relationship between the conceptual apparatus of science - something which, as noted by Stephen W. Hawking in 'A Brief History Of Time' (Ch. 1), "exists only in our minds and does not have any other reality (whatever that might mean)."
In this respect, the foundations of Probability Theory have long been, as it were, in intellectual no man's land between Philosophy and Mathematics - a situation where either side appears to proceed on the implicit assumption that the foundations have been adequately established by the other side.
In my view, clarification of outstanding issues with respect to the foundations of Probability Theory would represent the single most important intellectual advance in scientific thought since the Relativity and Quantum Revolutions of the twentieth century.
Specifically, it would permit us to put in perspective - and adjudicate on solid grounds - the conflict between Einstein and his peers on which Hawking wrote as follows:
"Einstein spent most of his later years unsuccessfully searching for a unified theory, but the time was not ripe: there were partial theories for gravity and the electromagnetic force, but very little was known about the nuclear forces. Moreover, Einstein refused to believe in the reality of quantum mechanics, despite the important role he had played in its development. Yet it seems that the uncertainty principle is a fundamental feature of the universe we live in. A successful unified theory must therefore necessarily incorporate this principle." (Op. cit., Ch. 10)
In the context, this may be construed as Hawking's response to the view-point which Einstein summarized as follows:
"Roughly stated [Einstein's] conclusion is this: Within the framework of the statistical quantum theory there is no such thing as a complete description of the individual system. More cautiously it might be put as follows: The attempt to conceive the quantum-theoretical description as the complete description of the individual systems leads to unnatural theoretical interpretations, which become immediately unnecessary if one accepts the interpretation that the description refers to ensembles of systems and not to individual systems. In that case the whole "egg-walking" performed in order to avoid the "physically real" becomes superfluous. There exists, however, a simple psychological reason for the fact that this most nearly obvious interpretation is being shunned. For if the statisical quantum theory does not pretend to describe the individual system (and its development in time) completely, it appears unavoidable to look elsewhere for a complete description of the individual system; in doing so it would be clear from the very beginning that the elements of such a description are not contained within the conceptual scheme of the statistical quantum theory. With this one would admit that, in principle, this scheme could not serve as the basis of theoretical physics." ('Reply to Criticisms', in 'Albert Einstein, Philosopher-Scientist', Open Court, Third Edition, 1988, pp. 671-672)
In my view, Hawking's statement that "it seems that the uncertainty principle is a fundamental feature of the universe we live in" cannot be reconciled with his own view of "theory" as something which "exists only in our minds and does not have any other reality..."
Moreover, the rhetorical transition from a qualified "seems" - "it seems that the uncertainty principle is a fundamental feature of the universe we live in" - to a categorical "must" - "A successful unified theory must therefore necessarily incorporate this principle." - speaks volumes about Hawking's failure to address Einstein's view-point on its merits.
For what it is worth, I am persuaded that Einstein's position is irrefutable on strictly epistemological grounds - that use of the Probability Calculus serves to 'mask' mis-specification of the axiomatic premises of Quantum Mechanics.
Posted by: Karl Smith
I myself am deeply in league with Hawkins.
Though I have never had the pleasure of interacting with him on a direct basis, from reading his material I imagine him to be a positivist of the firmest sort.
QM succeeds in my mind as a theory because it allows one to derive conclusions which fit the data with astonishing accuracy. To me that is the key element of good theory. Beyond that there is elegance, simplicity and a sense of integrability with other theories that I usually refer to as fecundity.
Perhaps on a related question people often ask me if I think that the tenants of micro-economic theory are correct. To be people really maximize utility? To firms really maximize profit?
To which, if time permits, I ask, what do you mean 'really'? Do you really see me standing in front of you. I mean really. There is data picked up by your eyes, analysed by your optical lobe and fit to a model, a theory if you will in your head.
You believe you are seeing a person because you have recieved very similiar data before and have derived a model which explains it. However, if late someone were to ask the details of what you saw you might not be able to recall, what color my eyes were, whether I had a mole or not, whether my nose slanted to the left or right.
Why? Because you never really saw those things. You saw enough to confirm the expectations of your model and then concluded that I was probably a person. But, you never really stopped to look carefully at what was in front of you.
So, did you see me?
If by that I mean, you collected data sufficient to confirm the predictions of your model then yes. If you mean did you actually examine the object in front of you and take in all of the colors and shapes that make it up then, no.
Such is the same with profit maximization. If we went over IBMs books I am sure that we would find that even taking dynamics and uncertainity into play IBM is not maximizing profit.
However, if we take a more general look at its behavior does IBM appear to be conforming to profit maximization. For the most part, yes.
Posted by: Gunnar Tomasson
Re. the following:
QM succeeds in my mind as a theory because it allows one to derive conclusions which fit the data with astonishing accuracy. To me that is the key element of good theory.
Comment:
The differences between Einstein and his peers (here exemplified by Hawking) did not concern QM's "fit" with the data.
The question was (and remains) whether excellence of "fit" should be construed as presumptive PROOF that QM's axiomatic pre-suppositions accord in some one-to-one sense with "fundamental feature[s] of the universe we live in".
In this respect, the parenthetical part of Hawking's statement that a scientific theory "exists only in our minds and does not have any other reality (whatever that might mean)" exemplifies what Einstein termed the "egg-walking" performed by his QM peers around the "physically real".
Namely, REFUSAL to engage Einstein in debate on the LOGICAL admissibility of the proposition that "the uncertainty principle IS a fundamental feature of the universe we live in. [And that a] successful unified theory MUST therefore NECESSARILY incorporate the principle."
Posted by: Karl Smith
QUOTE
The question was (and remains) whether excellence of "fit" should be construed as presumptive PROOF that QM's axiomatic pre-suppositions accord in some one-to-one sense with "fundamental feature[s] of the universe we live in".
ENDQUOTE
But as a positivist I would reply that such questions are without basis. There is no "fundemental feature". There is no world beyond the data.
There is data and there are models. Thats it.
The nature of the "real world", whatever that might be, I would say is something that one should take up with his preist or deity but has no business being discussed in a science.
If you were to ask me as a person what I thought about it, I would say: Reality is a fiction. It is a useful model we employ to deal with the data we are regularly receiving via our senses.
With something of a nod to Betrand Russell I will admit that we might as well concede that there is a real world. That might as well lasting so long as no one presses questions such as the ones you or Einstien seem to propose.
Once you start asking me to differtiate between reality and the models I create based on data I must demand that there is no reality beyond the model.
Posted by: Gunnar Tomasson
Re. the following:
But as a positivist I would reply that such questions are without basis. There is no "fundemental feature". There is no world beyond the data.
And:
Once you start asking me to differtiate between reality and the models I create based on data I must demand that there is no reality beyond the model.
Comment:
How, then, do you construe Hawking's statement:
"....it seems that the uncertainty principle is a fundamental feature of the universe we live in. A successful unified theory must therefore necessarily incorporate this principle."?
Posted by: Karl Smith
The reality model is compelling.
No. But, seriously I can't answer for Hawkins but I can venture a guess.
I think he means that no model without uncertainity is likely to succeed. Uncertainity has proven such an crucial and powerful element of the model.
Therefore, one might say that the reailty model is seemingly incompatible with ultimate determinism.
But that is just a guess.
Forum Sponsored by Wolfram Research
© 2004-2009 Wolfram Research, Inc. | Powered by vBulletin 2.3.0 © 2000-2002 Jelsoft Enterprises, Ltd. |
Disclaimer
vB Easy Archive Final - Created by Xenon and modified/released by SkuZZy from the Job Openings