In Chapter 9 we saw that the F test and the likelihood ratio test can give different p-values for the same comparison. That’s because the \(\chi^2\) distribution is what the \(F\) distribution turns into when the denominator degrees of freedom go to infinity, so it has a little less mass in its tail.
That was just one example. In this appendix we’ll look at this more systematically. We’ll also add a third statistic, the Wald test, which you’ll see in the output of most generalized linear models.
B.1 Three ways to test the same thing
Let’s take the simplest possible comparison: an intercept-only model vs. a model with one predictor. There are three statistics we could use.
The F statistic compares the reduction in SSE to the SSE that’s left over:
We compare it to an \(F\) distribution with 1 and \(n - 2\) degrees of freedom.
The Wald statistic takes that same number and compares it to a \(\chi^2\) distribution instead. This ignores the fact that the denominator was estimated.
The likelihood ratio statistic compares log-likelihoods. For the normal model, this turns out to be a function of the ratio of the two SSEs:
\[G^2 = n \log\left(\frac{\text{SSE}_C}{\text{SSE}_A}\right)\]
We also compare this one to a \(\chi^2\) distribution with one degree of freedom.
All three test the same null hypothesis, but none of them gives exactly the same answer.
B.2 Watching them converge
We can keep the association the same and just change the sample size. MASS::mvrnorm() with empirical = TRUE makes data where the sample correlation is exactly what we ask for. That gets rid of sampling noise, so the only thing changing is \(n\).
# create dataset with exact known correlation# then extract F, G^2, and the F, Wald, and LRT chi-square p-valuesget_stats <-function(n, r) { Sigma <-matrix(c(1, r, r, 1), 2, 2) mat <- MASS::mvrnorm(n, mu =c(0, 0), Sigma = Sigma, empirical =TRUE) x = mat[,1] y = mat[,2] m0 <-lm(y ~1) m1 <-lm(y ~ x) rss0 <-deviance(m0) rss1 <-deviance(m1) f_stat <- ((rss0 - rss1) /1) / (rss1 / (n-2)) g2 <- n *log(rss0 / rss1) fp <-1-pf(f_stat, 1, n-2) waldp <-1-pchisq(f_stat, 1) lrtp <-1-pchisq(g2, 1) log_fp <-log(fp) log_waldp <-log(waldp) log_lrtp <-log(lrtp) tmp <- tibble::tibble(f = f_stat,g2 = g2,fp = fp,waldp = waldp,lrtp = lrtp,log_fp = log_fp,log_waldp = log_waldp,log_lrtp = log_lrtp )return(tmp)}# for a given PRE (r2), get the data and compare stats for increasing Nmytests <-expand_grid(n =30:80,r =sqrt(.1)) |>rowwise() |>mutate(stats =list(get_stats(n, r))) |>unnest_wider(stats)mytests |>select(n, f, g2, fp, waldp, lrtp) |>head(4) |>mutate(across(everything(), \(z) round(z, 4)))
Note that f and g2 are not the same statistic. They’re close here (and they get closer as \(n\) gets bigger), but they’re calculated differently and compared to different distributions.
Every one of these datasets has exactly the same association (\(R^2 = .1\)), so any difference between the columns comes from the test, not the data.
The F test is the most conservative. Its p-value is always the biggest of the three, because it’s the only one that accounts for the fact that the denominator was estimated from the data.
The gap gets smaller as \(n\) goes up. By the right side of the plot, you can barely tell the three curves apart. They’re all the same as \(n\) goes to infinity, but in small samples they can lead to different conclusions.
At \(n = 30\), none of the three is below .05. At \(n = 40\), all of them are. In between, which test you use can decide whether you reject the null. This is another reason not to take .05 too seriously (see Chapter 6).
B.3 Which to use
For the normal linear model, use F. It’s exact rather than a large-sample approximation, and being a bit conservative in small samples is a good thing.
For generalized linear models, we can’t use the F test, so we have to choose between Wald and the likelihood ratio test. Wald is what you get by default in summary() output because it’s easy to calculate (it only needs the one fitted model). The likelihood ratio test needs both models, but it usually works better, especially when coefficients are large. If the two give noticeably different answers, I’d go with the likelihood ratio test.
