The Pareto distribution and the 20/80 rule

I mentioned in the previous post Pareto's 20/80 rule. Here, I will discuss Pareto's distribution, insisting on how (and in what conditions) it gives rise to this result. I had some trouble understanding the derivation as presented in various sources, so I will go through it in detail.

The functional form of the Pareto distribution is a power law, over an interval $(L,H)$ such that $0<L<H\leq \infty$. I will use the notations of the Wikipedia page unless stated otherwise. Its probability density function (PDF) $p(x)$ and cumulative distribution function (CDF) $F(x)$ are ($\alpha$ is real and strictly positive):

$$p(x) = \dfrac{\alpha}{1-(L/H)^{\alpha}} \dfrac{1}{x} \left ( \dfrac{L}{x} \right ) ^{\alpha}\quad ; \quad F(x) =  \dfrac{1-(L/x)^{\alpha}}{1-(L/H)^{\alpha}}$$

One often uses the complementary CDF (or survival function) defined as:

$$S(x) = 1 - F(x) = \dfrac{1}{1-(L/H)^{\alpha}}\left [ \left ( \dfrac{L}{x} \right )^{\alpha} - \left ( \dfrac{L}{H} \right )^{\alpha}\right ]$$

Note that the survival function is very similar to the PDF multiplied by $x$: $S(x) \simeq \dfrac{x}{\alpha} p(x)$, the difference being due only to the final truncation term. However, this is only true for power laws, as one can easily check by writing $p(x) = F'(x)$ and solving the resulting ODE. We should therefore carefully distinguish $x p(x)$ (which is, for instance, the integrand to use for computing the mean of the distribution) and $S(x)$ which "has already been integrated", so to speak.

Let us use this continuous model to describe the distribution of publications (neglecting for now its intrinsically discrete character). $x$ stands for the number of publications by one author, bounded by $L$ and $H$. The number of authors that published $x$ books is given by $N_0 \, p(x)$. $N_0$ is the total number of authors. 

• The first question is: who are the first $f$ more prolific authors (in Pareto's case, $f = 0.2 = 20$%)? More precisely, what is the threshold number of publications $x_f$ separating them from the less prolific ones?

This is quite easy: if we go through the list of authors (ordered by increasing $x$) when we reach $x_f$ we will have counted the lower fraction, so $\int_{L}^{x_f} p(x) \text{d}x = F(x_f) = 1-f$. Thus, the survival function is $S(x_f) = \int_{x_f}^{H} p(x) \text{d}x = f$ and we can simply invert this dependency to get $x_f =S^{-1}(f)$.

•  The second question is: how many publications did these top $f$ authors contribute?

We need to count the authors again, but with an additional factor of $x$, since there are $N_0 \, p(x)$ authors with exactly $x$ publications, for a total contribution of $x \, N_0 \, p(x)$. The fraction of publications contributed by the top $f$ authors $v$ is then:

$$v = \dfrac{\int_{x_f}^{H} x \, N_0 \, p(x) \text{d}x}{\int_{L}^{H} x \, N_0 \, p(x) \text{d}x} = \dfrac{\int_{x_f}^{H} x \, p(x) \text{d}x}{ \mu}$$

where $\mu$ is the mean of the distribution and $N_0 \mu$ is the total number of publications.

In the simple case $H = \infty$ (which requires $\alpha > 1$), one has:

$$p(x) = \dfrac{\alpha}{x} \left ( \dfrac{L}{x} \right )^{\alpha}, \quad \text{with} \quad \mu = \dfrac {\alpha}{\alpha-1} L$$

$$f  = S(x_f) =\left ( \dfrac{L}{x_f} \right )^{\alpha} \Rightarrow x_f = L f^{-1/\alpha}$$

 

Plugging the above into the equation for $v$ yields:

$$v = \dfrac{1}{\mu} \int_{x_f}^{\infty} x \, p(x) \text{d}x = \dfrac{\alpha}{\mu} \int_{x_f}^{\infty}  \left ( \dfrac{L}{x} \right ) ^{\alpha} \text{d}x = \left ( \dfrac{L}{x_f} \right ) ^{\alpha-1} =f^{\frac{\alpha-1}{\alpha}} \Rightarrow f = v^{\frac{\alpha}{\alpha-1}}$$  

Pareto's rule $f=0.2$ and $v=0.8$ requires $\alpha \simeq 1.161$: a power law with this exponent will obey the rule, irrespective of the values of $L$ and $N_0$. Despite the neat coincidence in the established statement of the principle, there is absolutely no need that $f+v=1$! For instance, the same $\alpha$ implies that, for $v=0.5$, $f \simeq 0.065$, a result I have already used in the previous post.