|
| 1 | +--- |
| 2 | +jupytext: |
| 3 | + formats: md:myst |
| 4 | + text_representation: |
| 5 | + extension: .md |
| 6 | + format_name: myst |
| 7 | +kernelspec: |
| 8 | + display_name: Python 3 |
| 9 | + language: python |
| 10 | + name: python3 |
| 11 | +--- |
| 12 | + |
| 13 | +(Chap_LogIntro)= |
| 14 | +# Logistic Regression Model |
| 15 | + |
| 16 | +This chapter has an executable [Google Colab notebook](https://colab.research.google.com/drive/1kNMOMvoKzuzNq_rw1yz86B3N98hgTaZ8?usp=sharing) with all the same code, data references, and images. The Google Colab notebook allows you to execute the code in this chapter in the cloud so you don't have to download Python, any of its packages, or any data to your local computer. You could manipulate and execute this notebook on any device with a browser, whether than be your computer, phone, or tablet. |
| 17 | + |
| 18 | +The focus of this chapter is to give the reader a basic introduction to the logistic regression model, where it comes from, and how it can be interpreted. |
| 19 | + |
| 20 | + |
| 21 | +(Sec_LogQuantQual)= |
| 22 | +## Quantitative versus Qualitative Data |
| 23 | +The linear regression models of chapter {ref}`Chap_BasicEmpirMethods` have continuous quantitative variables as dependent variables. That is, the $y_i$ variable takes on a continuum of values. We use a different class of models to estimate the relationship of exogenous variables to *qualitative* or *categorical* or *discrete* endogenous or dependent variables. |
| 24 | + |
| 25 | +Examples of qualitative or categorical variables include: |
| 26 | + |
| 27 | +* Binary variables take on two values ($J=2$), most often 0 or 1. Examples: Male or female, dead or alive, accept or reject. |
| 28 | +* General categorical variables can take on more than two values ($J\geq 2$). Examples: red, blue, or green; teenager, young adult, middle aged, senior. |
| 29 | + |
| 30 | +Note with general categorical variables that order and numerical distance do not matter. As an example let $FlowerColor_i=\{red=1, blue=2,green=3\}$ be a function of $neighborhood_i$, $season_i$, and $income_i$. |
| 31 | + |
| 32 | +$$ FlowerColor_i = \beta_0 + \beta_1 neighborhood_i + \beta_2 season_i + \beta_3 income_i + u_i $$ |
| 33 | + |
| 34 | +We could mathematically estimate this regression model, but would that make sense? What would be wrong with a regression model? |
| 35 | + |
| 36 | + |
| 37 | +(Sec_LogQuantQualClassSet)= |
| 38 | +### The classification setting |
| 39 | +Let $y_i$ be a qualitative dependent variable on $N$ observations with $i$ being the index of the observation. Each observation $y_i$ can take on one of $J$ discrete values $j\in\{1,2,...J\}$. Let $x_{p,i}$ be the $i$th observation of the $p$th explanatory variable (independent variable) such that $X_i=\{x_{1,i}, x_{2,i}, ... x_{P,i}\}$. Then the general formulation of a classifier comes in the following two forms, |
| 40 | + |
| 41 | +```{math} |
| 42 | + :label: EqLog_GenClassModel |
| 43 | + Pr(y_i=j|X_i,\theta) = f(X_i|\theta) \quad\forall i, j \quad\text{or}\quad \sum_{j=1}^J I_j(y_i=j) = f(X_i|\theta) \quad\forall i, j |
| 44 | +``` |
| 45 | + |
| 46 | +where $I_j$ in the second formulation is an indicator function that equals 1 when $y_i=j$ and equals 0 otherwise. |
| 47 | + |
| 48 | + |
| 49 | +(Sec_LogRegClass)= |
| 50 | +## Logistic Regression Classifier |
| 51 | +In this section, we will look at two models for binary (0 or 1) categorical dependent variables. We describe the first model--the linear probability (LP) model--for purely illustrative purposes. This is because the LP model has some serious shortcomings that make it almost strictly dominated by our second model in this section. |
| 52 | + |
| 53 | +The second model--the logistic regression (logit, binary classifier) model--will be the focus of this section. There is another variant of this model, the probit model. But the logistic model is the more flexible, more easily interpretable, and more commonly used of the two. |
| 54 | + |
| 55 | + |
| 56 | +(Sec_LogLPM)= |
| 57 | +### The linear probability (LP) model |
| 58 | + |
| 59 | +One option in which a regression is barely acceptable for modeling a binary (categorical) dependent variable is the linear probability (LP) model. When the dependent variable has only two categories, it can be modeled as $y_i\in\{0,1\}$ without loss of generality. Let the variable $z_i$ be interpreted as the probability that $y_i=1$ given the data $X_i$ and parameter values $\theta=\{\beta_0,\beta_1,...\beta_P\}$. |
| 60 | + |
| 61 | +```{math} |
| 62 | + :label: EqLog_LPM |
| 63 | + z_i = Pr(y_i=1|X_i,\theta) = \beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i} + ... \beta_P x_{P,i} + u_i |
| 64 | +``` |
| 65 | + |
| 66 | +The LP model can be a nice, easy, computationally convenient way to estimate the probability of outcome $y_i=1$. This could also be reinterpreted, without loss of generality, as the probability that $y_i=0$. This is equivalent to a redefinition of which outcome is defined as $y_i=1$. |
| 67 | + |
| 68 | +The main drawback of the LP model is that the predicted values of the probability that $y_i=1$ or $Pr(y_i=1|X_i,\theta)$ can be greater than 1 and can be less than 0. It is for this reason that it is very difficult to publish any research based on an LP model. |
| 69 | + |
| 70 | + |
| 71 | +(Sec_LogLogit)= |
| 72 | +### The logistic (logit) regression classifier |
| 73 | + |
| 74 | +In contrast to the linear probability model, a good classifier tranforms numerical values from explanatory variables or feature variables into a probability that is strictly between 0 and 1. More specifically this function must take any numer on the real line between $-\infty$ and $\infty$ and map it to the $[0,1]$ interval. In addition, we want a monotonically increasing relationship between $x$ and the function $f(x)$. What are some functions with this property? Candidates include the following functions. |
| 75 | + |
| 76 | +* $f(x)=\text{max}\Bigl(0, \,\text{min}\bigl(1, x\bigr)\Bigr)$ |
| 77 | +* $f(x)=\frac{e^x}{1 + e^x}$ |
| 78 | +* $f(x) = \arctan(x)$ |
| 79 | +* $f(x) = \text{cdf}(x)$ |
| 80 | + |
| 81 | +Why don't functions like $\sin(x)$, $\cos(x)$, and $\frac{|x|}{1+|x|}$ fit these criteria? |
| 82 | + |
| 83 | +The second function in the bulletted list above is the logistic function. The logistic regression model is a binary dependent variable classifier that constrains its predicted values to be stricly between 0 and 1. The logistic function is the following, |
| 84 | + |
| 85 | +```{math} |
| 86 | + :label: EqLog_Logistic |
| 87 | + f(x) = \frac{e^x}{1 + e^x} \quad\forall x |
| 88 | +``` |
| 89 | + |
| 90 | +and has the following general shape. |
| 91 | + |
| 92 | +```{code-cell} ipython3 |
| 93 | +:tags: ["hide-input", "remove_output"] |
| 94 | +
|
| 95 | +import numpy as np |
| 96 | +import matplotlib.pyplot as plt |
| 97 | +
|
| 98 | +x_vals = np.linspace(-6, 6, 500) |
| 99 | +y_vals = np.exp(x_vals) / (1 + np.exp(x_vals)) |
| 100 | +plt.plot(x_vals, y_vals, color="blue") |
| 101 | +plt.scatter(0, 0.5, color="black", s=15) |
| 102 | +plt.title(r"Logistic function for $x\in[-6,6]$") |
| 103 | +plt.xlabel(r'$x$ values') |
| 104 | +plt.ylabel(r'$f(x)$ values') |
| 105 | +plt.grid(color='gray', linestyle=':', linewidth=1, alpha=0.5) |
| 106 | +plt.show() |
| 107 | +``` |
| 108 | + |
| 109 | +```{figure} ../../../images/basic_empirics/logit/logit_gen.png |
| 110 | +:height: 500px |
| 111 | +:name: FigLogit_logit_gen |
| 112 | +
|
| 113 | +Logistic function for $x\in[-6,6]$ |
| 114 | +``` |
| 115 | + |
| 116 | +The logistic regression function is the specific case of the logistic function where the value of $x$ in the general logistic function {eq}`EqLog_Logistic` is replaced by a linear combination of variables $\beta_0 + \beta_1 x_{1,i} + ...\beta_P x_{P,i}$ similar to a linear regression model. |
| 117 | + |
| 118 | +```{math} |
| 119 | + :label: EqLog_Logit_std |
| 120 | + Pr(y_i=1|X_i,\theta) = \frac{e^{X_i\beta}}{1 + e^{X_i\beta}} = \frac{e^{\beta_0 + \beta_1 x_{1,i} + ...\beta_P x_{P,i}}}{1 + e^{\beta_0 + \beta_1 x_{1,i} + ...\beta_P x_{P,i}}} |
| 121 | +``` |
| 122 | + |
| 123 | +or equivalently |
| 124 | + |
| 125 | +```{math} |
| 126 | + :label: EqLog_Logit_neg |
| 127 | + Pr(y_i=1|X_i,\theta) = \frac{1}{1 + e^{-X_i\beta}} = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_{1,i} + ...\beta_P x_{P,i})}} |
| 128 | +``` |
| 129 | + |
| 130 | +We could estimate the paramters $\theta=\{\beta_0,\beta_1,...\beta_P\}$ by generalized method of moments (GMM) using nonlinear least squares or a more general set of moments to match.[^GMM] But maximum likelihood estimation is the most common method for estimating the parameters $\theta$ because of its more robust statistical properties.[^MaxLikeli] Also, the distributional assumptions are built into the model, so they are not overly strong. |
| 131 | + |
| 132 | + |
| 133 | +(Sec_LogLogitNLLS)= |
| 134 | +#### Nonlinear least squares estimation |
| 135 | +If we define $z_i = Pr(y_i=1|X_i,\theta)$, then the error in the logistic regression is the following. |
| 136 | + |
| 137 | +```{math} |
| 138 | + :label: EqLog_LogitNLLS_err |
| 139 | + \varepsilon_i = y_i - z_i |
| 140 | +``` |
| 141 | + |
| 142 | +The GMM specification of the nonlinear least squares method of estimating the parameter vector $\theta$ would then be the following.[^GMM] |
| 143 | + |
| 144 | +```{math} |
| 145 | + :label: EqLog_LogitNLLS_gmm |
| 146 | + \begin{split} |
| 147 | + \hat{\theta}_{nlls} = \theta:\quad &\min_{\theta} \sum_{i=1}^N\varepsilon_i^2 \quad = \quad \min_{\theta}\sum_{i=1}^N\bigl(y_i - z_i \bigr)^2 \quad \\ |
| 148 | + &= \quad \min_{\theta} \sum_{i=1}^N\Bigl[y_i - Pr(y_i=1|X_i,\theta)\Bigr]^2 |
| 149 | + \end{split} |
| 150 | +``` |
| 151 | + |
| 152 | + |
| 153 | +(Sec_LogLogitMLE)= |
| 154 | +#### Maximum likelihood estimation |
| 155 | +We characterized the general likelihood function for a sample of data as the probability that the given sample $(y_i,X_i)$ came from the assumed distribution given parameter values $Pr(y_i=1|X_i,\theta)$. |
| 156 | + |
| 157 | +```{math} |
| 158 | + :label: EqLog_LogitMLE_like |
| 159 | + \mathcal{L}(y_i,X_i|\theta) = \prod_{i=1}^N Pr(y_i=1|X_i,\theta)^{y_i}\bigl[1 - Pr(y_i=1|X_i,\theta)\bigr]^{1 - y_i} |
| 160 | +``` |
| 161 | + |
| 162 | +The intuition of this likelihood function is that you want the probability of the observations for which $y_i=1$ to be close to one $Pr(X)$, and you want the probability of the observations for which $y_i=0$ to also be close to one $1 - Pr(X)$. |
| 163 | + |
| 164 | +The log-likelihood function, which the MLE problem maximizes is the following. |
| 165 | + |
| 166 | +```{math} |
| 167 | + :label: EqLog_LogitMLE_loglike |
| 168 | + \ln\bigl[\mathcal{L}(y_i,X_i|\theta)\bigr] = \sum_{i=1}^N\Bigl(y_i\ln\bigl[Pr(y_i=1|X_i,\theta)\bigr] + (1 - y_i)\ln\bigl[1 - Pr(y_i=1|X_i,\theta)\bigr]\Bigr) |
| 169 | +``` |
| 170 | + |
| 171 | +The MLE problem for estimating $\theta$ of the logistic regression model is, therefore, the following.[^MaxLikeli] |
| 172 | + |
| 173 | +```{math} |
| 174 | + :label: EqLog_LogitMLE_maxprob |
| 175 | + \hat{\theta}_{mle} = \theta:\quad \max_{\theta} \ln\bigl[\mathcal{L}(y_i,X_i|\theta)\bigr] |
| 176 | +``` |
| 177 | + |
| 178 | +(Sec_LogLogitTitanic)= |
| 179 | +#### Titanic example |
| 180 | +A good example of logistic regression comes from a number of sources. But I am adapting some code and commentary from [http://www.data-mania.com/blog/logistic-regression-example-in-python/](http://www.data-mania.com/blog/logistic-regression-example-in-python/). The research question is to use a famous Titanic passenger dataset to try to identify the characteristics that most predict whether you survived $y_i=1$ or died $y_i=0$. |
| 181 | + |
| 182 | +```{code-cell} ipython3 |
| 183 | +:tags: [] |
| 184 | +
|
| 185 | +import pandas as pd |
| 186 | +
|
| 187 | +url = ('https://raw.githubusercontent.com/OpenSourceEcon/CompMethods/' + |
| 188 | + 'main/data/basic_empirics/logit/titanic-train.csv') |
| 189 | +titanic = pd.read_csv(url) |
| 190 | +titanic.columns = ['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', |
| 191 | + 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'] |
| 192 | +titanic.describe() |
| 193 | +``` |
| 194 | + |
| 195 | +The variable descriptions are the following: |
| 196 | +* `Survived`: Survival (0 = No; 1 = Yes) |
| 197 | +* `Pclass`: Passenger class (1 = 1st; 2 = 2nd; 3 = 3rd) |
| 198 | +* `Name`: Name |
| 199 | +* `Sex`: Gender |
| 200 | +* `Age`: Age |
| 201 | +* `SibSp`: Number of siblings/spouses aboard |
| 202 | +* `Parch`: Number of parents/children aboard |
| 203 | +* `Ticket`: Ticket number |
| 204 | +* `Fare`: Passenger fare (British pound) |
| 205 | +* `Cabin`: Cabin |
| 206 | +* `Embarked`: Port of embarkation (C = Cherbourg; Q = Queenstown; S = Southampton) |
| 207 | + |
| 208 | +Let's first check that our target variable, `Survived`, is binary. Since we are building a model to predict survival of passangers from the Titanic, our target is going to be the `Survived` variable from the titanic dataframe. To make sure that it is a binary variable, let's use Seaborn's `countplot()` function. |
| 209 | + |
| 210 | +```{code-cell} ipython3 |
| 211 | +:tags: [] |
| 212 | +
|
| 213 | +titanic['Survived'].value_counts() |
| 214 | +``` |
| 215 | + |
| 216 | + |
| 217 | +(SecLogFootnotes)= |
| 218 | +## Footnotes |
| 219 | + |
| 220 | +The footnotes from this chapter. |
| 221 | + |
| 222 | +[^GMM]: See the {ref}`Chap_GMM` chapter of this book. |
| 223 | + |
| 224 | +[^MaxLikeli]: See the {ref}`Chap_MaxLikeli` chapter of this book. |
0 commit comments