-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Runtime Error raised by torch.nn.modules.activation.MultiheadAttention
when bias=False, batch_first=True
#88669
Comments
torch.nn.modules.activation.MultiheadAttention
when bias=True, batch_first=True
torch.nn.modules.activation.MultiheadAttention
when bias=False, batch_first=True
Thank you for opening the issue @shakedbr - does this issue persist with newer versions of PyTorch or nightlies? |
@cpuhrsch, this also happens in version 1.13.0 and in a nightly version 1.14.0.dev20221113, but in these versions, to reproduce the bug you need to include an even number of heads e.g.: import torch
x = torch.rand((1, 5, 10))
model = torch.nn.modules.activation.MultiheadAttention(10, num_heads=2, bias=False, batch_first=True)
model.eval()
model(x, x, x) |
@mikekgfb can you please link the fix to the issue? |
Also needs #88854 |
Thank you Michael. #89855 (comment) is the cherry-pick to release/1.13 |
closing since cherry-pick is included in the release |
馃悰 Describe the bug
Hi,
When creating an object of
torch.nn.modules.activation.MultiheadAttention
withbias=False
andbatch_first=True
, activating evaluation mode, and calling the forward pass you get an exception:It seems that the following lines don't handle the case where a parameter is
None
.pytorch/torch/nn/modules/activation.py
Lines 1117 to 1119 in 8cb5c55
Versions
[pip3] numpy==1.23.4
[pip3] torch==1.12.1
[pip3] torch-scatter==2.0.9
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.2.2
[conda] numpy 1.23.4 py39hefdcf20_0 conda-forge
[conda] pytorch 1.12.1 py3.9_0 pytorch
[conda] torch-scatter 2.0.9 pypi_0 pypi
[conda] torchaudio 0.12.1 py39_cpu pytorch
[conda] torchvision 0.2.2 py_3 pytorch
cc @jbschlosser @bhosmer @cpuhrsch @erichan1
The text was updated successfully, but these errors were encountered: