tensor.expand should be used as t.expand(a, b, c). However, when the size arguments are passed in a list, i.e., t.expand([a, b, c]). Forward is fine but backward fails. The following code shows this issue (from a discussion https://discuss.pytorch.org/t/solved-backward-error-when-using-expand/4273/6)
import torch
from torch.autograd import Variable
class TwoLayerNet(torch.nn.Module):
def __init__(self, D_in, H, D_out, h, w):
super(TwoLayerNet, self).__init__()
self.linear1 = torch.nn.Linear(D_in, H)
self.linear2 = torch.nn.Linear(H * h * w, D_out)
def forward(self, x):
h_relu = self.linear1(x).clamp(min=0)
h_relu = torch.unsqueeze(torch.unsqueeze(h_relu, 2), 3) # -> N x H x 1 x 1
h_expand = h_relu.expand([64, H, h, w]).contiguous().view(64, -1) # -> N x H x h x w
y_pred = self.linear2(h_expand) # -> N x D_out
return y_pred
x = Variable(torch.randn(N, D_in), requires_grad=True)
y = Variable(torch.randn(N, D_out), requires_grad=False)
model = TwoLayerNet(D_in, H, D_out, h, w)
criterion = torch.nn.MSELoss(size_average=False)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
for t in range(500):
y_pred = model(x)
loss = criterion(y_pred, y)
print(t, loss.data[0])
optimizer.zero_grad()
loss.backward()
optimizer.step()(N is batch size; D_in is input dimension;
H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out, h, w = 64, 1000, 100, 10, 6, 6)
The traceback is as follows:
Traceback (most recent call last):
File "try.py", line 31, in
loss.backward()
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py", line 152, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/__init__.py", line 98, in backward
variables, grad_variables, retain_graph)
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/function.py", line 91, in apply
return self.forwardcls.backward(self, *args)
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/_functions/pointwise.py", line 289, in backward
return grad_output * mask, None
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py", line 802, in mul
return self.mul(other)
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py", line 311, in mul
return Mul.apply(self, other)
File "/home/qizhe/tool/anaconda2/lib/python2.7/site-packages/torch/autograd/_functions/basic_ops.py", line 48, in forward
return a.mul(b)
RuntimeError: inconsistent tensor size, expected r_ [1 x 100 x 6 x 6], t [1 x 100 x 6 x 6] and src [64 x 100] to have the same number of elements, but got 3600, 3600 and 6400 elements respectively at /home/qizhe/tool/pytorch/torch/lib/TH/generic/THTensorMath.c:875
The issue exists for both the bleeding-edge version and the stable version torch-0.1.12.post2-cp27. Should forward throw an exception to help debuging?
tensor.expand should be used as t.expand(a, b, c). However, when the size arguments are passed in a list, i.e., t.expand([a, b, c]). Forward is fine but backward fails. The following code shows this issue (from a discussion https://discuss.pytorch.org/t/solved-backward-error-when-using-expand/4273/6)
(N is batch size; D_in is input dimension;
H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out, h, w = 64, 1000, 100, 10, 6, 6)
The traceback is as follows:
The issue exists for both the bleeding-edge version and the stable version torch-0.1.12.post2-cp27. Should forward throw an exception to help debuging?