4.2 Estimation and Testing

Chapters and Sections in HTML Format×

2. Linear Models for Continuous Data
2.1. Introduction to Linear Models
2.2. Estimation of the Parameters
2.3. Tests of Hypotheses
2.4. Simple Linear Regression
2.5. Multiple Linear Regression
2.6. One-Way Analysis of Variance
2.7. Two-Way Analysis of Variance
2.8. Analysis of Covariance Models
2.9. Regression Diagnostics
2.10. Transforming the Data

3. Logit Models for Binary Data
3.1. Introduction to Logistic Regression
3.2. Estimation and Hypothesis Testing
3.3. The Comparison of Two Groups
3.4. The Comparison of Several Groups
3.5. Models With Two Predictors
3.6. Multi-factor Models: Model Selection
3.7. Other Choices of Link
3.8. Regression Diagnostics for Binary Data

4. Poisson Models for Count Data
4.1. Introduction to Poisson Regression
4.2. Estimation and Testing
4.3. A Model for Heteroscedastic Counts
4.A. Models for Over-Dispersed Count Data

6. Multinomial Response Models
6.1. The Nature of Multinomial Data
6.2. The Multinomial Logit Model
6.3. The Conditional Logit Model
6.4. The Hierarchical Logit Model
6.5. Models for Ordinal Response Data

7. Survival Models
7.1. The Hazard and Survival Functions
7.2. Censoring and The Likelihood Function
7.3. Approaches to Survival Modeling
7.4. The Piece-Wise Exponential Model
7.5. Infant and Child Mortality in Colombia
7.6. Discrete Time Models

A. Review of Likelihood Theory
A.1. Maximum Likelihood Estimation
A.2. Tests of Hypotheses

B. Generalized Linear Model Theory
B.1. The Model
B.2. Maximum Likelihood Estimation
B.3. Tests of Hypotheses
B.4. Binomial Errors and Link Logit
B.5. Poisson Errors and Link Log

The log-linear Poisson model described in the previous section is a generalized linear model with Poisson error and link log. Maximum likelihood estimation and testing follows immediately from the general results in Appendix B. In this section we review a few key results.

4.2.1 Maximum Likelihood Estimation

The likelihood function for $n$ independent Poisson observations is a product of probabilities given by Equation 4.1. Taking logs and ignoring a constant involving $\log (y_i!)$, we find that the log-likelihood function is

where ${\mu }_i$ depends on the covariates ${\mathit{x}}_i$ and a vector of $p$ parameters $\mathit{\beta }$ through the log link of Equation 4.2.

It is interesting to note that the log is the canonical link for the Poisson distribution. Taking derivatives of the log-likelihood function with respect to the elements of $\mathit{\beta }$, and setting the derivatives to zero, it can be shown that the maximum likelihood estimates in log-linear Poisson models satisfy the estimating equations

where $\mathit{X}$ is the model matrix, with one row for each observation and one column for each predictor, including the constant (if any), $\mathit{y}$ is the response vector, and $\hat{\mathit{\mu }}$ is a vector of fitted values, calculated from the m.l.e.’s $\hat{\mathit{\beta }}$ by exponentiating the linear predictor $\mathit{\eta }={\mathit{X}}^{\prime }\hat{\mathit{\beta }}$. This estimating equation arises not only in Poisson log-linear models, but more generally in any generalized linear model with canonical link, including linear models for normal data and logistic regression models for binomial counts. It is not satisfied, however, by estimates in probit models for binary data.

To understand equation 4.3 it helps to consider a couple of special cases. If the model includes a constant, then one of the columns of the model matrix $\mathit{X}$ is a column of ones. Multiplying this column by the response vector $\mathit{y}$ produces the sum of the observations. Similarly, multiplying this column by the fitted values $\hat{\mathit{\mu }}$ produces the sum of the fitted values. Thus, in models with a constant one of the estimating equations matches the sum of observed and fitted values. In terms of the example introduced at the beginning of this chapter, fitting a model with a constant would match the total number of children ever born to all women.

As a second example suppose the model includes a discrete factor represented by a series of dummy variables taking the value one for observations at a given level of the factor and zero otherwise. Multiplying this dummy variable by the response vector $\mathit{y}$ produces the sum of observations at that level of the factor. When this is done for all levels we obtain the so-called marginal total. Similarly, multiplying the dummy variable by the fitted values $\hat{\mathit{\mu }}$ produces the sum of the expected or fitted counts at that level. Thus, in models with a discrete factor the estimating equations match the observed and fitted marginals for the factor. In terms of the example introduced at the outset, if we fit a model that treats marital duration as a discrete factor we would match the observed and fitted total number of children ever born in each category of duration since first marriage.

This result generalizes to higher order terms. Suppose we entertain models with two discrete factors, say $A$ and $B$. The additive model $A+B$ would reproduce exactly the marginal totals by $A$ or by $B$. The model with an interaction effect $AB$ would, in addition, match the totals in each combination of categories of $A$ and $B$, or the $AB$ margin. This result, which will be important in the sequel, is the basis of an estimation algorithm known as iterative proportional fitting.

In general, however, we will use the iteratively-reweighted least squares (IRLS) algorithm discussed in Appendix B. For Poisson data with link log, the working dependent variable $\mathit{z}$ has elements

and the diagonal matrix $\mathit{W}$ of iterative weights has elements

where ${\hat{\mu }}_i$ denotes the fitted values based on the current parameter estimates.

Initial values can be obtained by applying the link to the data, that is taking the log of the response, and regressing it on the predictors using OLS. To avoid problems with counts of 0, one can add a small constant to all responses. The procedure usually converges in a few iterations.

4.2.2 Goodness of Fit

A measure of discrepancy between observed and fitted values is the deviance. In Appendix B we show that for Poisson responses the deviance takes the form

The first term is identical to the binomial deviance, representing ‘twice a sum of observed times log of observed over fitted’. The second term, a sum of differences between observed and fitted values, is usually zero, because m.l.e.’s in Poisson models have the property of reproducing marginal totals, as noted above.

For large samples the distribution of the deviance is approximately a chi-squared with $n-p$ degrees of freedom, where $n$ is the number of observations and $p$ the number of parameters. Thus, the deviance can be used directly to test the goodness of fit of the model.

An alternative measure of goodness of fit is Pearson’s chi-squared statistic, which is defined as

The numerator is the squared difference between observed and fitted values, and the denominator is the variance of the observed value. The Pearson statistic has the same form for Poisson and binomial data, namely a ‘sum of squared observed minus expected over expected’.

In large samples the distribution of Pearson’s statistic is also approximately chi-squared with $n-p$ d.f. One advantage of the deviance over Pearson’s chi-squared is that it can be used to compare nested models, as noted below.

4.2.3 Tests of Hypotheses

Likelihood ratio tests for log-linear models can easily be constructed in terms of deviances, just as we did in logistic regression models. In general, the difference in deviances between two nested models has approximately in large samples a chi-squared distribution with degrees of freedom equal to the difference in the number of parameters between the models, under the assumption that the smaller model is correct.

One can also construct Wald tests as we have done before, based on the fact that the maximum likelihood estimator $\hat{\mathit{\beta }}$ has approximately in large samples a multivariate normal distribution with mean equal to the true parameter value $\mathit{\beta }$ and variance-covariance matrix $\text{var}(\hat{\mathit{\beta }})={\mathit{X}}^{\mathbf{\prime }}\mathit{W}\mathit{X}$, where $\mathit{X}$ is the model matrix and $\mathit{W}$ is the diagonal matrix of estimation weights described earlier.

Math rendered by