Multiply weights after using dropout in training - PyTorch

Question

I have a Pytorch regression model as follows:

    model = nn.Sequential(nn.Linear(n_in, n_h_1),
                      nn.ReLU(),
                      nn.Dropout(p=0.1),
                      nn.Linear(n_h_1, n_h_2),
                      nn.ReLU(),
                      nn.Linear(n_h_2, n_out))

As you can see, I have used a Dropout regularization layer with dropout probability of 0.1. When the training is completed, we should disable the dropout. This can be achieved using model.eval(). Reference to this StackOverflow answer and other resources, we should multiply the output of hidden layer with (1-p) during inferencing of model.

My question is: do we have to do this manually using pytorch or the library handles this itself after applying model.eval()? If we have to do it ourselves, how can we achieve thisfor the given model?

score 1 · Accepted Answer · answered Jul 09 '19 at 07:42

1

PyTorch handles this with scaling the output of the dropout layer at training time with this probability:

So it handles this itself after applying model.eval(). You can lookup their code here:

answered Jul 09 '19 at 07:42

fhaase

171
3

Multiply weights after using dropout in training - PyTorch

1 Answers1