Transformer模型介绍

Transformer模型介绍

文献：$Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]. Advances in neural information processing systems. 2017: 5998-6008.$

详解：http://nlp.seas.harvard.edu/2018/04/03/attention.html The Annotated Transformer.

视频：https://www.youtube.com/watch?v=ugWDIIOHtPA 李宏毅：Transformer

1. encoder-decoder

2. self attention

import torch
import torch.nn as nn
import torch.nn.functional as F

class SelfAttention(nn.Module):
    def __init__(self, input_dim,hidden_dim):
        super().__init__()
        self.key = nn.Parameter(torch.randn(input_dim,hidden_dim))
        self.query = nn.Parameter(torch.randn(input_dim,hidden_dim))
        self.value = nn.Parameter(torch.randn(input_dim,hidden_dim))
        
    def forward(self, inputs):
        '''
        inputs dim : [batch, seq_len, dim]
        '''
        Q,K,V = torch.matmul(inputs, self.query), \
                torch.matmul(inputs, self.key), \
                torch.matmul(inputs, self.value)
        # Q * K.T => [batch, seq_len, seq_len]
        A = torch.bmm(Q, K.transpose(2,1))
        # 每个batch，每行进行softmax
        A = F.softmax(A, dim = 2)
        # V.T * A.T
        O = torch.bmm(V.transpose(2,1),A.transpose(2,1))
        
        return O

hidden_dim = 64
input_dim = 32
batch = 32
seq_len = 5

model = SelfAttention(input_dim,hidden_dim)
inputs = torch.randn((batch,seq_len,input_dim))

model(inputs).shape
# 输出：torch.Size([32, 64, 5])

3. multi head attention

4. attention和self-attention区别