Logistic Regression ¶
TLDR
class LogisticRegression(pf.CategoricalModel):
def __init__(self, d):
self.w = pf.Parameter([d, 1]) #weights
self.b = pf.Parameter([1, 1]) #bias
def __call__(self, x):
return pf.Bernoulli(x @ self.w() + self.b())
model = LogisticRegression(x.shape[1])
or simply
model = pf.LogisticRegression(x.shape[1])
and then
model.fit(x, y)
In the last example, both \(x\) and \(y\) were continuous variables (their values ranged from \(-\infty\) to \(\infty\)). What if our output variable is binary? That is, suppose the output variable can take only one of two values, 0 or 1, and so we need a classification model.
Let’s create a dataset which has 3 continuous features, and a target variable with 2 classes:
# Imports
import probflow as pf
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
randn = lambda *x: np.random.randn(*x).astype('float32')
# Settings
N = 1000 #number of datapoints
D = 3 #number of features
# Generate data
x = randn(N, D)
w = np.array([[2.], [0.1], [-2.]]).astype('float32')
noise = randn(N, 1)
y = np.round(1./(1.+np.exp(-(x@w+noise))))
# Plot it
for i in range(D):
plt.subplot(1, D, i+1)
sns.violinplot(x=y[:, 0], y=x[:, i])
Building a Logistic Regression Manually¶
A logistic regression is a model where our output variable is categorical. It’s basically the same thing as a linear regression, except we pipe the linearly predicted value through a nonlinear function to get the probability of the output class:
where \(f\) is usually the logistic function. Or, with \(>2\) classes, a softmax.
If our target variable has only 2 possible classes, we can model this using a Bernoulli distribution:
To create this model in ProbFlow, we’ll create a class which inherits
CategoricalModel
, because the target variable is categorical
(either 0 or 1). Again, in the __init__
method we define the parameters
of the model, and in the __call__
method we compute probabilistic
predictions given the parameters and the input data:
class LogisticRegression(pf.CategoricalModel):
def __init__(self, dims):
self.w = pf.Parameter([dims, 1], name='Weights')
self.b = pf.Parameter([1, 1], name='Bias')
def __call__(self, x):
return pf.Bernoulli(x @ self.w() + self.b())
import torch
pf.set_backend('pytorch')
class LogisticRegression(pf.CategoricalModel):
def __init__(self, dims):
self.w = pf.Parameter([dims, 1], name='Weights')
self.b = pf.Parameter([1, 1], name='Bias')
def __call__(self, x):
x = torch.tensor(x)
return pf.Bernoulli(x @ self.w() + self.b())
Note that by default, the Bernoulli
distribution treats its inputs
as logits (that is, it passes the inputs through a sigmoid function to get the
output class probabilities). To force it to treat the inputs as raw
probability values, use the probs
keyword argument to the Bernoulli
constructor.
Then we can instantiate our model class,
model = LogisticRegression(D)
And fit it to the data!
model.fit(x, y, lr=0.01)
Now we can plot the posterior distributions for the weights and the bias, and can see that the model recovered the values we used to generate the data:
model.posterior_plot(ci=0.9)
Using the LogisticRegression module¶
An even easier way to do a logistic regression with ProbFlow is to use the
pre-built LogisticRegression
model:
model = pf.LogisticRegression(D)
model.fit(x, y, lr=0.01)
model.posterior_plot(ci=0.9)
Multinomial Logistic Regression¶
The LogisticRegression
model even handles when \(y\) has multiple
classes (that is, a
Multinomial logistic regression).
Let’s generate some data with 4 features, where the target has 3 possible
classes:
# Settings
N = 1000 #number of datapoints
D = 4 #number of features
K = 3 #number of target classes
# Generate data
x = randn(N, D)
w = randn(D, K)
noise = randn(N, 1)
y = np.argmax(x@w+noise, axis=1).astype('float32')
# Plot it
for i in range(D):
plt.subplot(2, 2, i+1)
sns.violinplot(x=y, y=x[:, i])
The k
keyword argument to LogisticRegression
sets the number of
classes of the dependent variable.
model = pf.LogisticRegression(D, k=K)
model.fit(x, y, lr=0.01, epochs=200)
model.posterior_plot()
And we can predict the target class given the features:
>>> model.predict(x[:5, :])
array([0, 0, 1, 2, 1], dtype=int32)
Or even compute the posterior predictive probability of the target class for test datapoints:
x_test = randn(1, D)
model.pred_dist_plot(x_test)