Regression
This module contains all of the regression modules you will need.
- class sealion.regression.LinearRegression
While KNNs (K-Nearest-Neighbors) may be the simplest ML algorithms out there, Linear Regression is probably the one you heard first. You may have used it on a TI-84 before - all it does is it fit a line to the data. It does this through the gradient descent algorithm or a neat normal equation. We will use the normal equation as usually it works just the same and is faster, but for much larger datasets you should explore neural networks or dimensionality reduction (check the algos there.) The equation in school taught is
y = mx + b, but we’ll denote it asy_hat = m1x1 + m2x2 ... mNxN + b. The hat is meant to resemble the predictions, and the reason we do it fromm1...mNis because our data can be in N dimensions, not necessarily one.Some other things to know is that your data for
x_trainshould always be 2D. 2D means that it is[[]]. This doesn’t mean the data is necessarily 2D (this could look like[[1, 2], [2, 3]]) - but just means that its lists inside lists.y_trainis your labels, or the “correct answers” so it should be in a 1D list, which is just a list. This library assumes just a bit of knowledge about this - and it isn’t too difficult - so feel free to search this up.Another thing to note here is that for our library you can enter in numpy arrays of python lists, but you will always get numpy arrays back (standard practice with other libs, too.)
A lot of the methods here are consistent and same to a lot of the other classes of the library, so reading this will make it a lot easier down the line.
The goal of this module is it for it to be a useful algorithm, but I also hope this is inspiring to your journey of machine learning. It isn’t nearly as hard as it seems.
- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
r^2score
- fit(x_train, y_train)
- Parameters
x_train – 2D training data
y_train – 1D training labels
- Returns
None
- predict(x_test)
- Parameters
x_test – 2D prediction data
- Returns
predictions in a 1D numpy array
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.LogisticRegression(accuracy_desired=0.95, learning_rate=0.01, max_iters=1000, show_acc=True)
Logistic Regression is in the sweet spot of being easy to understand and useful. Despite having “regression” in the name, what it does is binary classification. Say you had a dataset of people’s heights and weights, along with other attributes (we call them features) and you wanted to predict whether they were healthy (0) or unhealthy (1). You may choose to use logistic regression as this task is binary classification (classifying 2 categories.) Make sure your labels are 0 and 1.
Logistic Regression doesn’t have a closed form solution, so we’ll have to use the gradient descent algorithm. It may take longer but we’ve provided a progress bar for you to see how it’s going.
You may want to look into is the sigmoid function, it’s what really is at the core of distinguishing logistic and linear regression. It’ll make more sense after you look at the differences in their output equations.
With that in mind, we can get started!
- __init__(accuracy_desired=0.95, learning_rate=0.01, max_iters=1000, show_acc=True)
- Parameters
accuracy_desired – the accuracy at which the fit() method can stop
learning_rate – how fast gradient descent should run
max_iters – how many iterations of gradient descent are allowed
show_acc – whether or not to show the accuracy in the fit() method
- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
what % of its predictions on x_test were correct
- fit(x_train, y_train)
- Parameters
x_train – training data (2D)
y_train – training labels (1D)
- Returns
- predict(x_test)
- Parameters
x_test – prediction data (2D)
- Returns
predictions in a 1D vector/list
- reset_params()
Run this after the fit() method has been run please. :return: Nothing, just redoes the weight init for all parameters.
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.SoftmaxRegression(num_classes, accuracy_desired=0.95, learning_rate=0.01, max_iters=1000, show_acc=True)
Once you know logistic regression, softmax regression is a breeze. Logistic regression is really a specific type of softmax regression, where the number of classes is equal to 2. Whereas logistic regression only can predict for 2 classes (0 and 1), softmax regression can predict for N number of classes. This can be 5, 3, or even a thousand! To define the number of classes in this model, insert the
num_classesparameter in the init. ALL parameters in this class are the same as in logistic regression except for that argument.Another note, if you use softmax regression with 2 classes - you just end up using logistic regression. In general you should use logistic regression if there are only 2 classes as it is faster and optimized as such.
If you’re interested in the theory, the primary change is from the sigmoid function to the softmax function. Also look into the log loss and crossentropy loss, both of them are at the heart of softmax and logistic regression. Maybe interesting to read up on that.
- __init__(num_classes, accuracy_desired=0.95, learning_rate=0.01, max_iters=1000, show_acc=True)
:param num_classes : number of classes your dataset has :param accuracy_desired: the accuracy at which the fit() method can stop :param learning_rate: how fast gradient descent should run :param max_iters: how many iterations of gradient descent are allowed :param show_acc: whether or not to show the accuracy in the fit() method
- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
what % of its predictions on x_test were correct
- fit(x_train, y_train)
- Parameters
x_train – training data (2D)
y_train – training labels (1D)
- Returns
- predict(x_test)
- Parameters
x_test – prediction data (2D)
- Returns
predictions in a 1D vector/list
- reset_params()
Run this after the fit() method has been run please. :return: Nothing, just redoes the weight init for all parameters.
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.RidgeRegression(alpha=0.5)
Imagine you have a dataset with 1000 features. Most of these features will usually be irrelevant to the task your solving; only a few of them will really matter. If you don’t want to use Dimensionality Reduction (see the algorithms there), you may want to consider using this. What ridge regression does is try to keep the weights as small as possible, a.k.a. regularization. This is because if a weight of a feature is not needed you want it to be 0 - you don’t want it to be 0.01 because of overfitting or the particular instances of the training data. Therefore it will work well with many features as it reduces the weights, hence making the model overfit less and generalize more (do we really need those 0.1s and 0.2s in the weights?) As StatQuest said, it “desensitizes” the training data (highly recommend you watch that video - regularization can be tough.)
You’ll be glad to know that this uses a closed form solution (generally much faster than iterative gradient descent.) There are other algorithms for regularization in regression like Lasso and Elastic Net, a combination of Lasso and Ridge, that are available in this module.
There’s only one parameter you need to worry about, which is alpha. It is simply how much to punish the model for its weights (especially unnecessary ones). It’s typically set between 0 and 1.
- __init__(alpha=0.5)
Set the alpha parameter for the model.
- Parameters
alpha – default 0.5, ranges from 0 - 1
- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
r^2 score
- fit(x_train, y_train)
- Parameters
x_train – 2D training data
y_train – 1D training labels
- Returns
- predict(x_test)
- Parameters
x_test – 2D prediction data
- Returns
predictions in a 1D numpy array
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.LassoRegression(alpha=0.5, accuracy_desired=0.95, learning_rate=0.01, max_iters=10000, show_acc=True)
Regularizer for regression models just like Ridge Regression. A few notable differences, but for the most part will do the same thing. Lasso regression tries to minimize the weights just like ridge regression, but one of its big differences is its tendency to make the weights of the regression model 0. This greatly decreases overfitting by making sure unnecessary features aren’t considered in the model.
Another difference is the use of gradient descent instead of a closed form solution like Ridge Regression. It shares the same alpha parameter to determine how much you want to “punish” (i.e. reduce) the weights, especially those not needed.
- __init__(alpha=0.5, accuracy_desired=0.95, learning_rate=0.01, max_iters=10000, show_acc=True)
- Parameters
alpha – penalty for the weights, applied to both lasso and ridge components
Check above documentation for all other parameters.
- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
r^2 score for the predictions of x_test and y_test
- fit(x_train, y_train)
- Parameters
x_train – training data (must be 2D)
y_train – training labels (must be 1D)
- Returns
None, just the model has the weights and biases stored
- predict(x_test)
- Parameters
x_test – testing data (must be 2D)
- Returns
predictions (1D vector/list)
- reset_params()
Resets all weights and biases of the model. :return: none
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.ElasticNet(l1_r=0.5, alpha=0.5, accuracy_desired=0.95, learning_rate=0.01, max_iters=1000, show_acc=True)
Elastic Net is a combination of Ridge and Lasso Regression. It implements both penalties, and you just decide how much weight each should have. The parameter
l1_rin the__init__of this class is the amount of “importance” lasso regression has (specifically the regularization term), on a scale from 0 - 1. If lasso regression gets an “importance” of 0.7, then we give the ridge regression part of this model an “importance” 0.3. Uses gradient descent, as there is no closed form solution.- __init__(l1_r=0.5, alpha=0.5, accuracy_desired=0.95, learning_rate=0.01, max_iters=1000, show_acc=True)
- Parameters
l1_r – The weight that lasso regression gets in this model. Default 0.5, but setting it higher tips the scale. Setting it to 0 or 1 makes it just ridge or lasso regression.
alpha – penalty for the weights, applied to both lasso and ridge components
Check above documentation for all other parameters.
- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
r^2 score for the predictions of x_test and y_test
- fit(x_train, y_train)
- Parameters
x_train – training data (must be 2D)
y_train – training labels (must be 1D)
- Returns
None, just the model has the weights and biases stored
- predict(x_test)
- Parameters
x_test – testing data (must be 2D)
- Returns
predictions (1D vector/list)
- reset_params()
Resets all weights and biases of the model. :return: none
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.ExponentialRegression
Say you’ve got some curved data. How can a line possibly fit that data? Glad you ask - that’s why this class exists. Exponential Regression does something very simple. All it does is just take the log transformation of exponential data, so it becomes a line and model that. Because now we can find
y_transformed_log = mx + b, we can turn thaty_transformed_logback intoyby raising it to powere.- evaluate(x_test, y_test)
- Parameters
x_test – testing data (2D)
y_test – testing labels (1D)
- Returns
r^2 score
- fit(x_train, y_train)
- Parameters
x_train – training data (2D)
y_train – training labels (1D)
- Returns
- predict(x_test)
- Parameters
x_test – prediction data (2D)
- Returns
predictions in a 1D vector/list
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.
- class sealion.regression.PolynomialRegression
Polynomial Regression is like an extended version of Linear Regression (sort of like Exponential Regression.) All it does is turn the equation from y = m1x1 + m2x2 … mNxN + bias to y = m1x1^1 + m2x2^2 … mNxN^N + bias. It just adds those power combinations to make the regression module model the data better. Neural networks can also model those functions similarily.
Please normalize your data with this module with the (X - mu) / sigma or just by dividing by the maximum value. This will help with faster convergence.
Not as famous as some other regression algorithms, so may you need a bit of experimentation.
- evaluate(x_test, y_test)
- Parameters
x_test – data to be evaluated on
y_test – labels
- Returns
r^2 score
- fit(x_train, y_train)
- Parameters
x_train – training data (2D)
y_train – training labels (1D)
- Returns
- predict(x_test)
- Parameters
x_test – points to be predicted on. Has to be stored in 2D array, even if just one data point. Ex.
[[1, 1]]or[[1, 1], [2, 2], [3, 3]]- Returns
flattened numpy array of your data going through the forward pass (that’s rounded as it’s either 0 or 1)
- visualize_evaluation(y_pred, y_test)
- Parameters
y_pred – predictions from the predict() method
y_test – labels for the data
- Returns
a matplotlib image of the predictions and the labels (“correct answers”) for you to see how well the model did.