Is it possible to customize the activation function in scikit-learn's MLPRegressor?

Question

Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway.

Is it possible to change the activation function of the output layer in an MLPRegressor neural network in scikit-learn?

I would like to use it for function approximation. I.e.

y = f(x)

where x is a vector of no more than 10 variables and y is a single continuous variable.

So I would like to change the output activation to linear or tanh. Right now it looks like sigmoid.

If not, I fail to see how you can use scikit-learn for anything other than classification which would be a shame.

Yes, I realise I could use tensorflow or PyTorch but my application is so basic I think scikit learn would be perfect fit (pardon the pun there).

Is it possible to build a more customized network with MultiLayerPerceptron or perhaps from individual layers (sknn.mlp)?

UPDATE:

In the documentation for MultiLayerPerceptron it does say:

For output layers, you can use the following layer types: Linear or Softmax.

But then further down it says:

When using the multi-layer perceptron, you should initialize a Regressor or a Classifier directly.

And there is no example of how to instantiate a MultiLayerPerceptron object.

Answer 1

I see in the code for the MLPRegressor, that the final activation comes from a general initialisation function in the parent class: BaseMultiLayerPerceptron, and the logic for what you want is shown around Line 271.

# Output for regression
if not is_classifier(self):
    self.out_activation_ = 'identity'
# Output for multi class
...

Then during a foward pass this self.out_activation_ is called (defined here):

# For the last layer
output_activation = ACTIVATIONS[self.out_activation_]
activations[i + 1] = output_activation(activations[i + 1])

That ominous looking variable ACTIVATIONS is simply a dictionary ,with the keywords being the descriptions you can choose as a parameter in your MLP, each mapping an actual function. Here is the dictionary:

ACTIVATIONS = {'identity': identity, 'tanh': tanh, 'logistic': logistic,
               'relu': relu, 'softmax': softmax}

With all of this information, you might be able to come up with a few ways of putting in your custom function. Off the top of my head, I can't see a quick way to simply provide a function. You could for example:

1. define your function where all the other activation functions are defined
2. add it to that ACTIVATIONS dictionary
3. make self.out_activation_ equal to your custom function (or even a new parameter in MLPRegressor
4. cross your fingers it doesn't break something somewhere else
5. run it and solve the inevitable small adaptations that will be necessary in a few places

I'm, afraid I have never looked at the source code of that library before, so cannot give more nuanced advice. Perhaps there is a beautifully elegant way to do it that we have both overlooked.

Answer 2

I tried to inject a modified initialization, which allows you to set the output activation:

from sklearn.neural_network import MLPRegressor
model = MLPRegressor()
from sklearn.neural_network._base import ACTIVATIONS, DERIVATIVES
def inplace_capped_output(X):
    """Compute a capped linear function inplace.
    Parameters
    ----------
    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The input data.
    """
    np.clip(X, -40,40)
ACTIVATIONS["custom"]=inplace_capped_output
model.old_initialize=model._initialize
def _initialize(self, y, layer_units, dtype):
    self._old_initialize(y, layer_units, dtype)
    self.out_activation="custom"
model._initialize = _initialize.get(model)

The binding of the modified initialize function follows this post.

You need to do something similar with regard to DERIVATIVES which is needed due to the backpropagation algorithm.

Be aware that you need to adapt https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/neural_network/_multilayer_perceptron.py#L274 because deltas[last] = activations[-1] - y is not true if you provide arbitrary, custom output activations.