Benford's Law Frank Benford, a physicist working in the 1930...

Benford's Law Frank Benford, a physicist working in the 1930 s, discovered an interesting fact about some sets of numbers. While you might expect the first digits of numbers such as street addresses or checkbook entries to be randomly distributed, Benford showed that in many cases the distribution of leading digits is not random, but rather tends to have more ones, with decreasing frequencies as the digits get larger. ${ }^{13}$ Table 7.17 shows the proportions of leading digits for data that satisfy Benford's law. Professor Rick Cleary of Bentley University has given several public lectures about Benford's law. As part of his presentation, he rips out pages of a telephone book and asks audience members to select entries at random and record the first digit of the street address. Counts for the leading digits of 1188 such addresses are shown in Table 7.18 and stored in a variable called Address in the dataset Benford. Test if these counts are inconsistent with the probabilities given by Benford's law.

Key Concepts

Benford's Law

Benford's Law is a statistical principle that predicts the frequency distribution of leading digits in many naturally occurring collections of numbers. Instead of being uniformly distributed, lower digits occur as the first digit more frequently than higher ones. This counterintuitive phenomenon is used in various applications, from financial forensics to data validation, by comparing the expected digit distribution to observed data.

Hypothesis Testing

Hypothesis testing is a method in statistics used to decide whether there is enough evidence in a sample to infer that a certain condition is true for the entire population. In this context, it involves setting up a null hypothesis that the data follows the expected distribution (as specified by Benford's Law) and an alternative hypothesis that it does not. The process then determines if deviations from the expected distribution are statistically significant.

Chi-Square Goodness-of-Fit Test

The Chi-Square Goodness-of-Fit Test is a statistical test used to compare the observed frequency distribution of a categorical variable to an expected distribution. In the context of Benford's Law, this test is useful to assess whether the frequencies of the first digits in the data set conform to the predicted proportions. It calculates a test statistic that, under the null hypothesis, follows a chi-square distribution, thereby enabling a determination of the p-value and significance.

Observed vs. Expected Frequencies

Comparing observed versus expected frequencies is a fundamental concept in statistics, particularly in goodness-of-fit tests. Observed frequencies are the actual counts collected in the data, while expected frequencies are the counts predicted by a theoretical model such as Benford's Law. The differences between these frequencies are used to calculate a test statistic, which helps in assessing whether the deviations are due to random chance or suggest a systematic discrepancy.

Transcript

00:01 We have a question here about benford's law, which shows that in many cases the distribution of leading digits is not equal, but instead it shows decreasing frequency as the digits decrease.

00:13 And you can see in the table below, here in the second row, are the proportions under benford's law.

00:20 What we also have is the counts of the leading digits in 1 ,188 addresses that were randomly selected from a four.

00:30 Homebook.

00:31 And what we want to do is test whether or not the counts are consistent with the proportions shown in benford's law.

00:37 So one thing we need to do is calculate the expected counts.

00:42 And to find the expected counts, we multiply p, which is the proportion shown under benford's law, so 0 .301 in our first for digit 1 times the total count, 1 ,188.

00:59 So, if we keep doing this for each of the nine digits, we'll be able to calculate the expected counts.

01:06 Let's take a look at our hypotheses.

01:09 We're going to be testing whether these proportions, p1, p2, all the way through p9, are as stated here in benford's law.

01:19 So you can see the statement, p1 equals 0 .301, p2 equals 0 .176, p3 equals 0 .125, and so on.

01:30 The alternative hypothesis then is that at least one of those proportions is incorrect.

01:37 Now, to calculate our kai square statistic, remember what we're going to do is take the observed count minus the expected count, we'll square that difference and divide by the expected count, and then we will sum for all the cells.

01:53 And when we do that, we can see that we end with a kai square statistic.

02:00 Of 8 .15.

02:02 The degrees of freedom that we have are going to be the number of cells, which is nine minus one, so we have eight degrees of freedom.

02:11 Now, we can save a little bit of work on this problem because this data set is in stat key.

02:18 And if you were to take a look here at our stat key main screen, let me show you this real quick...

Please give Ace some feedback