Assignment 4: Transformer, Pretrained Transformer, and LLM¶

With your environment activated int the terminal, run:

mamba env create -n cs5293-4 python=3.10
pip install -r requirements.txt
##Your VSCode may complain sometime you need to install ipykernel using the following commands. If not, then  just ignore this. 
#mamba install -n cs5293-4 ipykernel  --force-reinstall 

In this assignment, you have to submit two things:

• (1) The whole folder with your code (not with .cache or your model checkpoints)
• (2) A report to summarize your experiments
• (3) AI Usage Statement

Part 1: Machine Tranlation with Seq2Seq, Attention, Transformer (60')¶

In this part, you will train a machine translation model from French to English to explore the invention of Seq2Seq model and Attention Mechanism, and finally transformer.

Section 1.1. Setup and Data Exploration (0', But run this tutorial)¶

In [2]:

# please fix byself to make the corresponding packages work properly.
from __future__ import unicode_literals, print_function, division
from io import open
import unicodedata
import re
import random

import torch
import torch.nn as nn
from torch import optim
import torch.nn.functional as F

import numpy as np
from torch.utils.data import TensorDataset, DataLoader, RandomSampler
from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence
from torch.nn import GRU, LSTM, TransformerEncoder, TransformerEncoderLayer, MultiheadAttention, TransformerDecoder, TransformerDecoderLayer

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# please fix byself to make the corresponding packages work properly. from __future__ import unicode_literals, print_function, division from io import open import unicodedata import re import random import torch import torch.nn as nn from torch import optim import torch.nn.functional as F import numpy as np from torch.utils.data import TensorDataset, DataLoader, RandomSampler from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence from torch.nn import GRU, LSTM, TransformerEncoder, TransformerEncoderLayer, MultiheadAttention, TransformerDecoder, TransformerDecoderLayer device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

The English to French pairs in data/eng-fra.txt are seperated by a tab separated list of translation pairs:

{.sh}
I am cold.    J'ai froid.

In [3]:

# Let us first explore the data with some shell commands we have learned in assignment 1.
# Explore the data simply with shell command (linux or macOS,windows users please use git bash or wsl)
# echo -e will enable interpretation of backslash escapes
# \e[1;34m will set the color to blue and bold for "English"
# \e[1;32m will set the color to green and bold for "French"
# \e[0m will reset the color to default
# head will show the first 10 lines of the data file
# cut will extract the first two columns (seperated by tab) from the data file
# column will format the output into a table with tab as the seperator
!(echo -e "\e[1;34mEnglish\t\e[1;32mFrench\e[0m"; head ../data/eng-fra.txt | cut -f1,2) | column -t -s $'\t'

# Let us first explore the data with some shell commands we have learned in assignment 1. # Explore the data simply with shell command (linux or macOS,windows users please use git bash or wsl) # echo -e will enable interpretation of backslash escapes # \e[1;34m will set the color to blue and bold for "English" # \e[1;32m will set the color to green and bold for "French" # \e[0m will reset the color to default # head will show the first 10 lines of the data file # cut will extract the first two columns (seperated by tab) from the data file # column will format the output into a table with tab as the seperator !(echo -e "\e[1;34mEnglish\t\e[1;32mFrench\e[0m"; head ../data/eng-fra.txt | cut -f1,2) | column -t -s $'\t'

English  French
Go.            Va !
Run!           Cours !
Run!           Courez !
Wow!           Ça alors !
Fire!          Au feu !
Help!          À l'aide !
Jump.          Saute.
Stop!          Ça suffit !
Stop!          Stop !
Stop!          Arrête-toi !

We'll need a unique index per word to use as the inputs and targets of the networks later. To keep track of all this we will use a helper class called Lang which has word → index (word2index) and index → word (index2word) dictionaries, as well as a count of each word word2count which will be used to replace rare words later.

In [4]:

SOS_token = 0
EOS_token = 1

class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.n_words
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1

SOS_token = 0 EOS_token = 1 class Lang: def __init__(self, name): self.name = name self.word2index = {} self.word2count = {} self.index2word = {0: "SOS", 1: "EOS"} self.n_words = 2 # Count SOS and EOS def addSentence(self, sentence): for word in sentence.split(' '): self.addWord(word) def addWord(self, word): if word not in self.word2index: self.word2index[word] = self.n_words self.word2count[word] = 1 self.index2word[self.n_words] = word self.n_words += 1 else: self.word2count[word] += 1

The files are all in Unicode, to simplify we will turn Unicode characters to ASCII, make everything lowercase, and trim most punctuation.

In [5]:

# Turn a Unicode string to plain ASCII, thanks to
# https://stackoverflow.com/a/518232/2809427
def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
    )

# Lowercase, trim, and remove non-letter characters
def normalizeString(s):
    s = unicodeToAscii(s.lower().strip())
    s = re.sub(r"([.!?])", r" \1", s)
    s = re.sub(r"[^a-zA-Z!?]+", r" ", s)
    return s.strip()

# Turn a Unicode string to plain ASCII, thanks to # https://stackoverflow.com/a/518232/2809427 def unicodeToAscii(s): return ''.join( c for c in unicodedata.normalize('NFD', s) if unicodedata.category(c) != 'Mn' ) # Lowercase, trim, and remove non-letter characters def normalizeString(s): s = unicodeToAscii(s.lower().strip()) s = re.sub(r"([.!?])", r" \1", s) s = re.sub(r"[^a-zA-Z!?]+", r" ", s) return s.strip()

To read the data file we will split the file into lines, and then split lines into pairs. The files are all English → Other Language, so if we want to translate from Other Language → English I added the reverse flag to reverse the pairs.

In [6]:

def readLangs(lang1, lang2, reverse=False):
    print("Reading lines...")

    # Read the file and split into lines
    lines = open('../data/%s-%s.txt' % (lang1, lang2), encoding='utf-8').\
        read().strip().split('\n')

    # Split every line into pairs and normalize
    pairs = [[normalizeString(s) for s in l.split('\t')] for l in lines]

    # Reverse pairs, make Lang instances
    if reverse:
        pairs = [list(reversed(p)) for p in pairs]
        input_lang = Lang(lang2)
        output_lang = Lang(lang1)
    else:
        input_lang = Lang(lang1)
        output_lang = Lang(lang2)

    return input_lang, output_lang, pairs

def readLangs(lang1, lang2, reverse=False): print("Reading lines...") # Read the file and split into lines lines = open('../data/%s-%s.txt' % (lang1, lang2), encoding='utf-8').\ read().strip().split('\n') # Split every line into pairs and normalize pairs = [[normalizeString(s) for s in l.split('\t')] for l in lines] # Reverse pairs, make Lang instances if reverse: pairs = [list(reversed(p)) for p in pairs] input_lang = Lang(lang2) output_lang = Lang(lang1) else: input_lang = Lang(lang1) output_lang = Lang(lang2) return input_lang, output_lang, pairs

Since there are too many example sentences and we want to train something quickly in this notebook, we'll trim the data set to only relatively short and simple sentences. Here the maximum length is 10 words (that includes ending punctuation) and we're filtering to sentences that translate to the form "I am" or "He is" etc. (accounting for apostrophes replaced earlier).

In [7]:

MAX_LENGTH = 10

eng_prefixes = (
    "i am ", "i m ",
    "he is", "he s ",
    "she is", "she s ",
    "you are", "you re ",
    "we are", "we re ",
    "they are", "they re "
)

def filterPair(p):
    return len(p[0].split(' ')) < MAX_LENGTH and \
        len(p[1].split(' ')) < MAX_LENGTH and \
        p[1].startswith(eng_prefixes)


def filterPairs(pairs):
    return [pair for pair in pairs if filterPair(pair)]

MAX_LENGTH = 10 eng_prefixes = ( "i am ", "i m ", "he is", "he s ", "she is", "she s ", "you are", "you re ", "we are", "we re ", "they are", "they re " ) def filterPair(p): return len(p[0].split(' ')) < MAX_LENGTH and \ len(p[1].split(' ')) < MAX_LENGTH and \ p[1].startswith(eng_prefixes) def filterPairs(pairs): return [pair for pair in pairs if filterPair(pair)]

The full process for preparing the data is:

• Read text file and split into lines, split lines into pairs
• Normalize text, filter by length and content
• Make word lists from sentences in pairs

In [8]:

def prepareData(lang1, lang2, reverse=False):
    input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse)
    print("Read %s sentence pairs" % len(pairs))
    pairs = filterPairs(pairs)
    print("Trimmed to %s sentence pairs" % len(pairs))
    print("Counting words...")
    for pair in pairs:
        input_lang.addSentence(pair[0])
        output_lang.addSentence(pair[1])
    print("Counted words:")
    print(input_lang.name, input_lang.n_words)
    print(output_lang.name, output_lang.n_words)
    return input_lang, output_lang, pairs

input_lang, output_lang, pairs = prepareData('eng', 'fra', True)
print(random.choice(pairs))

def prepareData(lang1, lang2, reverse=False): input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse) print("Read %s sentence pairs" % len(pairs)) pairs = filterPairs(pairs) print("Trimmed to %s sentence pairs" % len(pairs)) print("Counting words...") for pair in pairs: input_lang.addSentence(pair[0]) output_lang.addSentence(pair[1]) print("Counted words:") print(input_lang.name, input_lang.n_words) print(output_lang.name, output_lang.n_words) return input_lang, output_lang, pairs input_lang, output_lang, pairs = prepareData('eng', 'fra', True) print(random.choice(pairs))

Reading lines...
Read 135842 sentence pairs
Trimmed to 11445 sentence pairs
Counting words...
Counted words:
fra 4601
eng 2991
['vous etes un bon garcon', 'you are a good boy']

Section 1.2 Seq2Seq Model for Machine Translation (10')¶

A Sequence to Sequence network, or seq2seq network, or Encoder Decoder network, is a model consisting of two RNNs called the encoder and decoder. The encoder reads an input sequence and outputs a single vector, and the decoder reads that vector to produce an output sequence. The RNN encoder and decoder could also be replaced with other architectures such as Transformer, CNN, or GNN depends on your inputs and outputs.

Unlike sequence prediction with a single RNN, where every input corresponds to an output, the seq2seq model frees us from sequence length and order, which makes it ideal for translation between two languages.

Consider the sentence Je ne suis pas le chat noir → I am not the black cat. Most of the words in the input sentence have a direct translation in the output sentence, but are in slightly different orders, e.g. chat noir and black cat. Because of the ne/pas construction there is also one more word in the input sentence. It would be difficult to produce a correct translation directly from the sequence of input words.

With a seq2seq model the encoder creates a single vector which, in the ideal case, encodes the "meaning" of the input sequence into a single vector --- a single point in some N dimensional space of sentences.

Task 1.2.1 The Encoder (5')¶

The encoder of a seq2seq network is a RNN that outputs some value for every word from the input sentence. For every input word the encoder outputs a vector and a hidden state, and uses the hidden state for the next input word.

In [9]:

# Define the Encoder RNN with GRU(a verionsion of RNN with less parameters than LSTM)
class EncoderRNN(nn.Module):
    # Initialize the encoder
    # define embedding layer, GRU layer and dropout layer
    # input size: size of the input vocabulary
    # hidden size: size of the hidden state vector
    # dropout_p: dropout probability
    def __init__(self, input_size, hidden_size, dropout_p=0.1):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        # Here we initialize a new embedding layer from scratch with input vocabulary size and hidden size
        self.embedding = nn.Embedding(input_size, hidden_size)
        # We could use pretrained embedding such as GloVe or Word2Vec by loading the weights into the embedding layer
        # Please try this in self-exploration!!!
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.dropout = nn.Dropout(dropout_p)

    def forward(self, input):
        embedded = self.dropout(self.embedding(input))
        # here we simply use GRU to process the embedded input
        # Please return the whole output and the last hidden state.
        # What is their shapes? Are the different when batch_first=True or False?
        # Print the shapes for better understanding
        # Add your code here!!!
        raise NotImplementedError("Please implement the forward function of EncoderRNN")

# Define the Encoder RNN with GRU(a verionsion of RNN with less parameters than LSTM) class EncoderRNN(nn.Module): # Initialize the encoder # define embedding layer, GRU layer and dropout layer # input size: size of the input vocabulary # hidden size: size of the hidden state vector # dropout_p: dropout probability def __init__(self, input_size, hidden_size, dropout_p=0.1): super(EncoderRNN, self).__init__() self.hidden_size = hidden_size # Here we initialize a new embedding layer from scratch with input vocabulary size and hidden size self.embedding = nn.Embedding(input_size, hidden_size) # We could use pretrained embedding such as GloVe or Word2Vec by loading the weights into the embedding layer # Please try this in self-exploration!!! self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True) self.dropout = nn.Dropout(dropout_p) def forward(self, input): embedded = self.dropout(self.embedding(input)) # here we simply use GRU to process the embedded input # Please return the whole output and the last hidden state. # What is their shapes? Are the different when batch_first=True or False? # Print the shapes for better understanding # Add your code here!!! raise NotImplementedError("Please implement the forward function of EncoderRNN")

Task 1.2.2 The Decoder (5')¶

In the simplest seq2seq decoder we use only last output of the encoder. This last output is sometimes called the context vector as it encodes context from the entire sequence. This context vector is used as the initial hidden state of the decoder.

At every step of decoding, the decoder is given an input token and hidden state. The initial input token is the start-of-string <SOS> token, and the first hidden state is the context vector (the encoder's last hidden state).

In [10]:

### Your implementation of DecoderRNN forward function with teacher forcing ###
class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size):
        super(DecoderRNN, self).__init__()
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.out = nn.Linear(hidden_size, output_size)

    def forward(self, encoder_outputs, encoder_hidden, target_tensor=None):
        batch_size = encoder_outputs.size(0)
        decoder_input = torch.empty(batch_size, 1, dtype=torch.long, device=device).fill_(SOS_token)
        decoder_hidden = encoder_hidden
        decoder_outputs = []

        # Run the decoder for MAX_LENGTH time steps
        for i in range(MAX_LENGTH):
            decoder_output, decoder_hidden  = self.forward_step(decoder_input, decoder_hidden)
            decoder_outputs.append(decoder_output)

            if target_tensor is not None:
                # Teacher forcing: Feed the target as the next input
                # add your code here!!! Particularly, set decoder_input to the i-th token of target_tensor
                # Pay attention to the shape of decoder_input
                raise NotImplementedError("Please implement teacher forcing in DecoderRNN forward function")
            else:
                # Without teacher forcing: use its own predictions as the next input
                _, topi = decoder_output.topk(1)
                decoder_input = topi.squeeze(-1).detach()  # detach from history as input

        decoder_outputs = torch.cat(decoder_outputs, dim=1)
        decoder_outputs = F.log_softmax(decoder_outputs, dim=-1)
        return decoder_outputs, decoder_hidden, None # We return `None` for consistency in the training loop for future attention

    def forward_step(self, input, hidden):
        output = self.embedding(input)
        output = F.relu(output)
        output, hidden = self.gru(output, hidden)
        output = self.out(output)
        return output, hidden

### Your implementation of DecoderRNN forward function with teacher forcing ### class DecoderRNN(nn.Module): def __init__(self, hidden_size, output_size): super(DecoderRNN, self).__init__() self.embedding = nn.Embedding(output_size, hidden_size) self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True) self.out = nn.Linear(hidden_size, output_size) def forward(self, encoder_outputs, encoder_hidden, target_tensor=None): batch_size = encoder_outputs.size(0) decoder_input = torch.empty(batch_size, 1, dtype=torch.long, device=device).fill_(SOS_token) decoder_hidden = encoder_hidden decoder_outputs = [] # Run the decoder for MAX_LENGTH time steps for i in range(MAX_LENGTH): decoder_output, decoder_hidden = self.forward_step(decoder_input, decoder_hidden) decoder_outputs.append(decoder_output) if target_tensor is not None: # Teacher forcing: Feed the target as the next input # add your code here!!! Particularly, set decoder_input to the i-th token of target_tensor # Pay attention to the shape of decoder_input raise NotImplementedError("Please implement teacher forcing in DecoderRNN forward function") else: # Without teacher forcing: use its own predictions as the next input _, topi = decoder_output.topk(1) decoder_input = topi.squeeze(-1).detach() # detach from history as input decoder_outputs = torch.cat(decoder_outputs, dim=1) decoder_outputs = F.log_softmax(decoder_outputs, dim=-1) return decoder_outputs, decoder_hidden, None # We return `None` for consistency in the training loop for future attention def forward_step(self, input, hidden): output = self.embedding(input) output = F.relu(output) output, hidden = self.gru(output, hidden) output = self.out(output) return output, hidden

Task 1.2.3 Define the Seq2Seq and Training Setup (0')¶

In [11]:

def indexesFromSentence(lang, sentence):
    return [lang.word2index[word] for word in sentence.split(' ')]

def tensorFromSentence(lang, sentence):
    indexes = indexesFromSentence(lang, sentence)
    indexes.append(EOS_token)
    return torch.tensor(indexes, dtype=torch.long, device=device).view(1, -1)

def tensorsFromPair(pair):
    input_tensor = tensorFromSentence(input_lang, pair[0])
    target_tensor = tensorFromSentence(output_lang, pair[1])
    return (input_tensor, target_tensor)

def get_dataloader(batch_size):
    input_lang, output_lang, pairs = prepareData('eng', 'fra', True)

    n = len(pairs)
    input_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32)
    target_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32)

    for idx, (inp, tgt) in enumerate(pairs):
        inp_ids = indexesFromSentence(input_lang, inp)
        tgt_ids = indexesFromSentence(output_lang, tgt)
        inp_ids.append(EOS_token)
        tgt_ids.append(EOS_token)
        input_ids[idx, :len(inp_ids)] = inp_ids
        target_ids[idx, :len(tgt_ids)] = tgt_ids

    train_data = TensorDataset(torch.LongTensor(input_ids).to(device),
                               torch.LongTensor(target_ids).to(device))

    train_sampler = RandomSampler(train_data)
    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
    return input_lang, output_lang, train_dataloader

def indexesFromSentence(lang, sentence): return [lang.word2index[word] for word in sentence.split(' ')] def tensorFromSentence(lang, sentence): indexes = indexesFromSentence(lang, sentence) indexes.append(EOS_token) return torch.tensor(indexes, dtype=torch.long, device=device).view(1, -1) def tensorsFromPair(pair): input_tensor = tensorFromSentence(input_lang, pair[0]) target_tensor = tensorFromSentence(output_lang, pair[1]) return (input_tensor, target_tensor) def get_dataloader(batch_size): input_lang, output_lang, pairs = prepareData('eng', 'fra', True) n = len(pairs) input_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32) target_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32) for idx, (inp, tgt) in enumerate(pairs): inp_ids = indexesFromSentence(input_lang, inp) tgt_ids = indexesFromSentence(output_lang, tgt) inp_ids.append(EOS_token) tgt_ids.append(EOS_token) input_ids[idx, :len(inp_ids)] = inp_ids target_ids[idx, :len(tgt_ids)] = tgt_ids train_data = TensorDataset(torch.LongTensor(input_ids).to(device), torch.LongTensor(target_ids).to(device)) train_sampler = RandomSampler(train_data) train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size) return input_lang, output_lang, train_dataloader

To train we run the input sentence through the encoder, and keep track of every output and the latest hidden state. Then the decoder is given the <SOS> token as its first input, and the last hidden state of the encoder as its first hidden state.

"Teacher forcing" is the concept of using the real target outputs as each next input, instead of using the decoder's guess as the next input. Using teacher forcing causes it to converge faster but when the trained network is exploited, it may exhibit instability.

You can observe outputs of teacher-forced networks that read with coherent grammar but wander far from the correct translation -intuitively it has learned to represent the output grammar and can "pick up" the meaning once the teacher tells it the first few words, but it has not properly learned how to create the sentence from the translation in the first place.

Because of the freedom PyTorch's autograd gives us, we can randomly choose to use teacher forcing or not with a simple if statement. Turn teacher_forcing_ratio up to use more of it.

This is a helper function to print time elapsed and estimated time remaining given the current time and progress %.

In [12]:

import time
import math

def asMinutes(s):
    m = math.floor(s / 60)
    s -= m * 60
    return '%dm %ds' % (m, s)

def timeSince(since, percent):
    now = time.time()
    s = now - since
    es = s / (percent)
    rs = es - s
    return '%s (- %s)' % (asMinutes(s), asMinutes(rs))

import time import math def asMinutes(s): m = math.floor(s / 60) s -= m * 60 return '%dm %ds' % (m, s) def timeSince(since, percent): now = time.time() s = now - since es = s / (percent) rs = es - s return '%s (- %s)' % (asMinutes(s), asMinutes(rs))

In [13]:

# Train for one epoch
def train_epoch(dataloader, encoder, decoder, encoder_optimizer,
          decoder_optimizer, criterion):

    total_loss = 0
    # Iterate through the dataloader
    for data in dataloader:
        input_tensor, target_tensor = data
        # Zero gradients for two optimizers
        encoder_optimizer.zero_grad()
        decoder_optimizer.zero_grad()
        # Forward pass through encoder and decoder
        encoder_outputs, encoder_hidden = encoder(input_tensor)
        decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor)
        # Compute loss and backpropagate
        loss = criterion(
            decoder_outputs.view(-1, decoder_outputs.size(-1)),
            target_tensor.view(-1)
        )
        loss.backward()

        encoder_optimizer.step()
        decoder_optimizer.step()

        total_loss += loss.item()

    return total_loss / len(dataloader)

# Train for one epoch def train_epoch(dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion): total_loss = 0 # Iterate through the dataloader for data in dataloader: input_tensor, target_tensor = data # Zero gradients for two optimizers encoder_optimizer.zero_grad() decoder_optimizer.zero_grad() # Forward pass through encoder and decoder encoder_outputs, encoder_hidden = encoder(input_tensor) decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor) # Compute loss and backpropagate loss = criterion( decoder_outputs.view(-1, decoder_outputs.size(-1)), target_tensor.view(-1) ) loss.backward() encoder_optimizer.step() decoder_optimizer.step() total_loss += loss.item() return total_loss / len(dataloader)

The whole training process looks like this:

• Start a timer
• Initialize optimizers and criterion
• Create set of training pairs
• Start empty losses array for plotting

Then we call train many times and occasionally print the progress (% of examples, time so far, estimated time) and average loss.

Plotting is done with matplotlib, using the array of loss values plot_losses saved while training.

In [ ]:

import matplotlib.pyplot as plt
plt.switch_backend('agg')
import matplotlib.ticker as ticker
import numpy as np

def showPlot(points):
    plt.figure()
    fig, ax = plt.subplots()
    # this locator puts ticks at regular intervals
    loc = ticker.MultipleLocator(base=0.2)
    ax.yaxis.set_major_locator(loc)
    plt.plot(points)
    # finally show the plot
    plt.show()

import matplotlib.pyplot as plt plt.switch_backend('agg') import matplotlib.ticker as ticker import numpy as np def showPlot(points): plt.figure() fig, ax = plt.subplots() # this locator puts ticks at regular intervals loc = ticker.MultipleLocator(base=0.2) ax.yaxis.set_major_locator(loc) plt.plot(points) # finally show the plot plt.show()

In [15]:

# The whole training process
def train(train_dataloader, encoder, decoder, n_epochs, learning_rate=0.001,
               print_every=100, plot_every=100):
    start = time.time()
    plot_losses = []
    print_loss_total = 0  # Reset every print_every
    plot_loss_total = 0  # Reset every plot_every

    # Initialize optimizers and criterion
    # Why we need two optimizers here?
    # Because we have two different networks to optimize: encoder and decoder
    # Each network has its own parameters
    # In this way, we can optimize them separately
    # However, we can also use a single optimizer to optimize both networks
    # by passing the parameters of both networks to the optimizer
    # Please try this in self-exploration with parameter grouping!!!
    encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)
    decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate)
    criterion = nn.NLLLoss()

    for epoch in range(1, n_epochs + 1):
        loss = train_epoch(train_dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
        print_loss_total += loss
        plot_loss_total += loss

        if epoch % print_every == 0:
            print_loss_avg = print_loss_total / print_every
            print_loss_total = 0
            print('%s (%d %d%%) %.4f' % (timeSince(start, epoch / n_epochs),
                                        epoch, epoch / n_epochs * 100, print_loss_avg))

        if epoch % plot_every == 0:
            plot_loss_avg = plot_loss_total / plot_every
            plot_losses.append(plot_loss_avg)
            plot_loss_total = 0

    showPlot(plot_losses)

# The whole training process def train(train_dataloader, encoder, decoder, n_epochs, learning_rate=0.001, print_every=100, plot_every=100): start = time.time() plot_losses = [] print_loss_total = 0 # Reset every print_every plot_loss_total = 0 # Reset every plot_every # Initialize optimizers and criterion # Why we need two optimizers here? # Because we have two different networks to optimize: encoder and decoder # Each network has its own parameters # In this way, we can optimize them separately # However, we can also use a single optimizer to optimize both networks # by passing the parameters of both networks to the optimizer # Please try this in self-exploration with parameter grouping!!! encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate) decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate) criterion = nn.NLLLoss() for epoch in range(1, n_epochs + 1): loss = train_epoch(train_dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion) print_loss_total += loss plot_loss_total += loss if epoch % print_every == 0: print_loss_avg = print_loss_total / print_every print_loss_total = 0 print('%s (%d %d%%) %.4f' % (timeSince(start, epoch / n_epochs), epoch, epoch / n_epochs * 100, print_loss_avg)) if epoch % plot_every == 0: plot_loss_avg = plot_loss_total / plot_every plot_losses.append(plot_loss_avg) plot_loss_total = 0 showPlot(plot_losses)

Evaluation is mostly the same as training, but there are no targets so we simply feed the decoder's predictions back to itself for each step. Every time it predicts a word we add it to the output string, and if it predicts the EOS token we stop there. We also store the decoder's attention outputs for display later.

In [16]:

# Evaluate the model on a given sentence
def evaluate(encoder, decoder, sentence, input_lang, output_lang):
    with torch.no_grad():
        input_tensor = tensorFromSentence(input_lang, sentence)

        encoder_outputs, encoder_hidden = encoder(input_tensor)
        # decoder_attention is None for now when using basic decoder without attention
        decoder_outputs, decoder_hidden, decoder_attn = decoder(encoder_outputs, encoder_hidden)

        _, topi = decoder_outputs.topk(1)
        decoded_ids = topi.squeeze()

        decoded_words = []
        for idx in decoded_ids:
            if idx.item() == EOS_token:
                decoded_words.append('<EOS>')
                break
            decoded_words.append(output_lang.index2word[idx.item()])
    return decoded_words, decoder_attn

# Evaluate the model on a given sentence def evaluate(encoder, decoder, sentence, input_lang, output_lang): with torch.no_grad(): input_tensor = tensorFromSentence(input_lang, sentence) encoder_outputs, encoder_hidden = encoder(input_tensor) # decoder_attention is None for now when using basic decoder without attention decoder_outputs, decoder_hidden, decoder_attn = decoder(encoder_outputs, encoder_hidden) _, topi = decoder_outputs.topk(1) decoded_ids = topi.squeeze() decoded_words = [] for idx in decoded_ids: if idx.item() == EOS_token: decoded_words.append('<EOS>') break decoded_words.append(output_lang.index2word[idx.item()]) return decoded_words, decoder_attn

We can evaluate random sentences from the training set and print out the input, target, and output to make some subjective quality judgements:

In [17]:

def evaluateRandomly(encoder, decoder, n=10):
    for i in range(n):
        pair = random.choice(pairs)
        # input french sentence
        print('>', pair[0])
        # target english sentence
        print('=', pair[1])
        output_words, _ = evaluate(encoder, decoder, pair[0], input_lang, output_lang)
        output_sentence = ' '.join(output_words)
        # generated english sentence
        print('<', output_sentence)
        print('')

def evaluateRandomly(encoder, decoder, n=10): for i in range(n): pair = random.choice(pairs) # input french sentence print('>', pair[0]) # target english sentence print('=', pair[1]) output_words, _ = evaluate(encoder, decoder, pair[0], input_lang, output_lang) output_sentence = ' '.join(output_words) # generated english sentence print('<', output_sentence) print('')

In [18]:

# Use the above evaluate to evaluate the whole dataset and compute the BLEU score
def evaluateDataset(encoder, decoder, input_lang, output_lang, pairs):
    from nltk.translate.bleu_score import sentence_bleu
    total_bleu_score = 0
    for pair in pairs:
        reference = [pair[1].split(' ')]
        output_words, _ = evaluate(encoder, decoder, pair[0], input_lang, output_lang)
        # remove <EOS>
        if output_words[-1] == '<EOS>':
            output_words = output_words[:-1]
        candidate = output_words
        bleu_score = sentence_bleu(reference, candidate, weights=(0.5, 0.5))
        total_bleu_score += bleu_score
    average_bleu_score = total_bleu_score / len(pairs)
    return average_bleu_score

# Use the above evaluate to evaluate the whole dataset and compute the BLEU score def evaluateDataset(encoder, decoder, input_lang, output_lang, pairs): from nltk.translate.bleu_score import sentence_bleu total_bleu_score = 0 for pair in pairs: reference = [pair[1].split(' ')] output_words, _ = evaluate(encoder, decoder, pair[0], input_lang, output_lang) # remove <EOS> if output_words[-1] == '<EOS>': output_words = output_words[:-1] candidate = output_words bleu_score = sentence_bleu(reference, candidate, weights=(0.5, 0.5)) total_bleu_score += bleu_score average_bleu_score = total_bleu_score / len(pairs) return average_bleu_score

Task 1.2.4 Training the Machine Translation Model(0')¶

With all these helper functions in place (it looks like extra work, but it makes it easier to run multiple experiments) we can actually initialize a network and start training.

Remember that the input sentences were heavily filtered. For this small dataset we can use relatively small networks of 256 hidden nodes and a single GRU layer. After about 10-20 minutes on CPU we'll get some reasonable results.

!!! Important !!! Please use the "interrupt"(not "restart") in jupernotebook to help your debug. During the training, the weight will be updated, "interrupt" will stop the training but the weights has been updated. So you could evaluate the "already updated" model weight to debug. In this case, you don't need to wait until finishing.

In [19]:

hidden_size = 128
batch_size = 32

input_lang, output_lang, train_dataloader = get_dataloader(batch_size)

encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)
decoder = DecoderRNN(hidden_size, output_lang.n_words).to(device)

train(train_dataloader, encoder, decoder, 80, print_every=5, plot_every=5)

hidden_size = 128 batch_size = 32 input_lang, output_lang, train_dataloader = get_dataloader(batch_size) encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device) decoder = DecoderRNN(hidden_size, output_lang.n_words).to(device) train(train_dataloader, encoder, decoder, 80, print_every=5, plot_every=5)

Reading lines...
Read 135842 sentence pairs
Trimmed to 11445 sentence pairs
Counting words...
Counted words:
fra 4601
eng 2991

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[19], line 9
      6 encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)
      7 decoder = DecoderRNN(hidden_size, output_lang.n_words).to(device)
----> 9 train(train_dataloader, encoder, decoder, 80, print_every=5, plot_every=5)

Cell In[15], line 22, in train(train_dataloader, encoder, decoder, n_epochs, learning_rate, print_every, plot_every)
     19 criterion = nn.NLLLoss()
     21 for epoch in range(1, n_epochs + 1):
---> 22     loss = train_epoch(train_dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
     23     print_loss_total += loss
     24     plot_loss_total += loss

Cell In[13], line 13, in train_epoch(dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
     11 decoder_optimizer.zero_grad()
     12 # Forward pass through encoder and decoder
---> 13 encoder_outputs, encoder_hidden = encoder(input_tensor)
     14 decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor)
     15 # Compute loss and backpropagate

File ~/mamba/envs/cs5293-4/lib/python3.10/site-packages/torch/nn/modules/module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
   1773     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1774 else:
-> 1775     return self._call_impl(*args, **kwargs)

File ~/mamba/envs/cs5293-4/lib/python3.10/site-packages/torch/nn/modules/module.py:1786, in Module._call_impl(self, *args, **kwargs)
   1781 # If we don't have any hooks, we want to skip the rest of the logic in
   1782 # this function, and just call forward.
   1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1784         or _global_backward_pre_hooks or _global_backward_hooks
   1785         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786     return forward_call(*args, **kwargs)
   1788 result = None
   1789 called_always_called_hooks = set()

Cell In[9], line 25, in EncoderRNN.forward(self, input)
     19 embedded = self.dropout(self.embedding(input))
     20 # here we simply use GRU to process the embedded input
     21 # Please return the whole output and the last hidden state.
     22 # What is their shapes? Are the different when batch_first=True or False?
     23 # Print the shapes for better understanding
     24 # Add your code here!!!
---> 25 raise NotImplementedError("Please implement the forward function of EncoderRNN")

NotImplementedError: Please implement the forward function of EncoderRNN

Set dropout layers to eval mode

In [ ]:

# why we need to set eval mode here?
# Because some layers like dropout and batchnorm behave differently during training and evaluation.
# In evaluation mode, dropout is disabled and batchnorm uses running statistics instead of batch statistics.
# Be sure to call model.eval() before running inference and model.train() before training.
encoder.eval()
decoder.eval()
evaluateRandomly(encoder, decoder)

# why we need to set eval mode here? # Because some layers like dropout and batchnorm behave differently during training and evaluation. # In evaluation mode, dropout is disabled and batchnorm uses running statistics instead of batch statistics. # Be sure to call model.eval() before running inference and model.train() before training. encoder.eval() decoder.eval() evaluateRandomly(encoder, decoder)

> je n en suis pas sur
= i am not certain about that

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[20], line 7
      5 encoder.eval()
      6 decoder.eval()
----> 7 evaluateRandomly(encoder, decoder)

Cell In[17], line 8, in evaluateRandomly(encoder, decoder, n)
      6 # target english sentence
      7 print('=', pair[1])
----> 8 output_words, _ = evaluate(encoder, decoder, pair[0], input_lang, output_lang)
      9 output_sentence = ' '.join(output_words)
     10 # generated english sentence

Cell In[16], line 6, in evaluate(encoder, decoder, sentence, input_lang, output_lang)
      3 with torch.no_grad():
      4     input_tensor = tensorFromSentence(input_lang, sentence)
----> 6     encoder_outputs, encoder_hidden = encoder(input_tensor)
      7     # decoder_attention is None for now when using basic decoder without attention
      8     decoder_outputs, decoder_hidden, decoder_attn = decoder(encoder_outputs, encoder_hidden)

File ~/mamba/envs/cs5293-4/lib/python3.10/site-packages/torch/nn/modules/module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
   1773     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1774 else:
-> 1775     return self._call_impl(*args, **kwargs)

File ~/mamba/envs/cs5293-4/lib/python3.10/site-packages/torch/nn/modules/module.py:1786, in Module._call_impl(self, *args, **kwargs)
   1781 # If we don't have any hooks, we want to skip the rest of the logic in
   1782 # this function, and just call forward.
   1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1784         or _global_backward_pre_hooks or _global_backward_hooks
   1785         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786     return forward_call(*args, **kwargs)
   1788 result = None
   1789 called_always_called_hooks = set()

Cell In[9], line 25, in EncoderRNN.forward(self, input)
     19 embedded = self.dropout(self.embedding(input))
     20 # here we simply use GRU to process the embedded input
     21 # Please return the whole output and the last hidden state.
     22 # What is their shapes? Are the different when batch_first=True or False?
     23 # Print the shapes for better understanding
     24 # Add your code here!!!
---> 25 raise NotImplementedError("Please implement the forward function of EncoderRNN")

NotImplementedError: Please implement the forward function of EncoderRNN

In [ ]:

# get the bleu score on the whole dataset
bleu_score = evaluateDataset(encoder, decoder, input_lang, output_lang, pairs)
print("BLEU score on the whole dataset: %.4f" % bleu_score)

# get the bleu score on the whole dataset bleu_score = evaluateDataset(encoder, decoder, input_lang, output_lang, pairs) print("BLEU score on the whole dataset: %.4f" % bleu_score)

/Users/jcao/mamba/envs/cs5293-4/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 2-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)

BLEU score on the whole dataset: 0.5094

Section 1.3 Seq2Seq with Context Vector At Every Decoding (10')¶

Problem 1: Faded Context Vector¶

The decoder only knows the source text via the single context.

• The context vector will get faded after decoding for a few steps.
• It hopes to use this single context vector to summarize all previous inputs (bottleneck)

Please fix this by providing the context vector to every step of decoding instead of just to the first step of decoding. There are different ways to implement this, please simply inject the context vector into the GRU input before feeding into GRU.

Task 1.3.1 ContextalDecoderRNN (10')¶

In [ ]:

##### Problem 1: Faded Context Vector
# Please fix this by providing the context vector to every step of decoding instead of just to the first step of decoding.
class ContextualDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size):
        super(DecoderRNN, self).__init__()
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.out = nn.Linear(hidden_size, output_size)

    def forward(self, encoder_outputs, encoder_hidden, target_tensor=None):
        # Your code here!!! Please modify the decoder to take the context vector (encoder_outputs) at every step of decoding
        # Hint: 
        # 1. You need to concatenate the context vector with the embedded input at every step
        # 2. Pay attention to the shapes of the tensors when concatenating, You need to use torch.cat() function to concatenate tensors
        # 3. Learn from the previous DecoderRNN implementation with teacher forcing and without teacher forcing
        # Particularly, pay attention to the shape of decoder_input at every step
        # Add your code here!!!
        raise NotImplementedError("Please implement the forward function of ContextualDecoderRNN")

##### Problem 1: Faded Context Vector # Please fix this by providing the context vector to every step of decoding instead of just to the first step of decoding. class ContextualDecoderRNN(nn.Module): def __init__(self, hidden_size, output_size): super(DecoderRNN, self).__init__() self.embedding = nn.Embedding(output_size, hidden_size) self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True) self.out = nn.Linear(hidden_size, output_size) def forward(self, encoder_outputs, encoder_hidden, target_tensor=None): # Your code here!!! Please modify the decoder to take the context vector (encoder_outputs) at every step of decoding # Hint: # 1. You need to concatenate the context vector with the embedded input at every step # 2. Pay attention to the shapes of the tensors when concatenating, You need to use torch.cat() function to concatenate tensors # 3. Learn from the previous DecoderRNN implementation with teacher forcing and without teacher forcing # Particularly, pay attention to the shape of decoder_input at every step # Add your code here!!! raise NotImplementedError("Please implement the forward function of ContextualDecoderRNN")

In [ ]:

# train a new model with ContextualDecoderRNN
hidden_size = 128
batch_size = 32

input_lang, output_lang, train_dataloader = get_dataloader(batch_size)

# let redefine a new encoder and contextual decoder
encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)
contextual_decoder = ContextualDecoderRNN(hidden_size, output_lang.n_words).to(device)
train(train_dataloader, encoder, contextual_decoder, 80, print_every=5, plot_every=5)

# train a new model with ContextualDecoderRNN hidden_size = 128 batch_size = 32 input_lang, output_lang, train_dataloader = get_dataloader(batch_size) # let redefine a new encoder and contextual decoder encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device) contextual_decoder = ContextualDecoderRNN(hidden_size, output_lang.n_words).to(device) train(train_dataloader, encoder, contextual_decoder, 80, print_every=5, plot_every=5)

In [ ]:

# why we need to set eval mode here?
# Because some layers like dropout and batchnorm behave differently during training and evaluation.
# In evaluation mode, dropout is disabled and batchnorm uses running statistics instead of batch statistics.
# Be sure to call model.eval() before running inference and model.train() before training.
encoder.eval()
contextual_decoder.eval()
evaluateRandomly(encoder, contextual_decoder)

# why we need to set eval mode here? # Because some layers like dropout and batchnorm behave differently during training and evaluation. # In evaluation mode, dropout is disabled and batchnorm uses running statistics instead of batch statistics. # Be sure to call model.eval() before running inference and model.train() before training. encoder.eval() contextual_decoder.eval() evaluateRandomly(encoder, contextual_decoder)

In [ ]:

# get the bleu score on the whole dataset
bleu_score = evaluateDataset(encoder, contextual_decoder, input_lang, output_lang, pairs)
print("BLEU score on the whole dataset: %.4f" % bleu_score)

# get the bleu score on the whole dataset bleu_score = evaluateDataset(encoder, contextual_decoder, input_lang, output_lang, pairs) print("BLEU score on the whole dataset: %.4f" % bleu_score)

Section 1.4 Seq2Seq with Attention-based Context Vector (15')¶

Task 1.4.1 Attention Decoder (15')¶

If only the context vector is passed between the encoder and decoder, that single vector carries the burden of encoding the entire sentence.

Attention allows the decoder network to "focus" on a different part of the encoder's outputs for every step of the decoder's own outputs. First we calculate a set of attention weights. These will be multiplied by the encoder output vectors to create a weighted combination. The result (called attn_applied in the code) should contain information about that specific part of the input sequence, and thus help the decoder choose the right output words.

Calculating the attention weights is done with another feed-forward layer attn, using the decoder's input and hidden state as inputs. Because there are sentences of all sizes in the training data, to actually create and train this layer we have to choose a maximum sentence length (input length, for encoder outputs) that it can apply to. Sentences of the maximum length will use all the attention weights, while shorter sentences will only use the first few.

Bahdanau attention, also known as additive attention, is a commonly used attention mechanism in sequence-to-sequence models, particularly in neural machine translation tasks. It was introduced by Bahdanau et al. in their paper titled Neural Machine Translation by Jointly Learning to Align and Translate. This attention mechanism employs a learned alignment model to compute attention scores between the encoder and decoder hidden states. It utilizes a feed-forward neural network to calculate alignment scores.

However, there are alternative attention mechanisms available, such as Luong attention, which computes attention scores by taking the dot product between the decoder hidden state and the encoder hidden states. It does not involve the non-linear transformation used in Bahdanau attention.

In this tutorial, we will be using Bahdanau attention. However, it would be a valuable exercise to explore modifying the attention mechanism to use Luong attention.

In [ ]:

class BahdanauAttention(nn.Module):
    def __init__(self, hidden_size):
        super(BahdanauAttention, self).__init__()
        self.Wa = nn.Linear(hidden_size, hidden_size)
        self.Ua = nn.Linear(hidden_size, hidden_size)
        self.Va = nn.Linear(hidden_size, 1)

    def forward(self, query, keys):
        scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys)))
        scores = scores.squeeze(2).unsqueeze(1)

        weights = F.softmax(scores, dim=-1)
        context = torch.bmm(weights, keys)

        return context, weights

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1):
        super(AttnDecoderRNN, self).__init__()
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.attention = BahdanauAttention(hidden_size)
        self.gru = nn.GRU(2 * hidden_size, hidden_size, batch_first=True)
        self.out = nn.Linear(hidden_size, output_size)
        self.dropout = nn.Dropout(dropout_p)

    def forward(self, encoder_outputs, encoder_hidden, target_tensor=None):
        # Your code here!!! Please implement the attention-based decoder
        # Hint: At every step of decoding, compute the attention weights and context vector using the BahdanauAttention module
        # Then concatenate the context vector with the embedded input before feeding into the GRU
        # Learn from the previous DecoderRNN implementation with teacher forcing and without teacher forcing
        # Particularly, pay attention to the shape of decoder_input at every step
        # Add your code here!!!
        raise NotImplementedError("Please implement the forward function of AttnDecoderRNN")

class BahdanauAttention(nn.Module): def __init__(self, hidden_size): super(BahdanauAttention, self).__init__() self.Wa = nn.Linear(hidden_size, hidden_size) self.Ua = nn.Linear(hidden_size, hidden_size) self.Va = nn.Linear(hidden_size, 1) def forward(self, query, keys): scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys))) scores = scores.squeeze(2).unsqueeze(1) weights = F.softmax(scores, dim=-1) context = torch.bmm(weights, keys) return context, weights class AttnDecoderRNN(nn.Module): def __init__(self, hidden_size, output_size, dropout_p=0.1): super(AttnDecoderRNN, self).__init__() self.embedding = nn.Embedding(output_size, hidden_size) self.attention = BahdanauAttention(hidden_size) self.gru = nn.GRU(2 * hidden_size, hidden_size, batch_first=True) self.out = nn.Linear(hidden_size, output_size) self.dropout = nn.Dropout(dropout_p) def forward(self, encoder_outputs, encoder_hidden, target_tensor=None): # Your code here!!! Please implement the attention-based decoder # Hint: At every step of decoding, compute the attention weights and context vector using the BahdanauAttention module # Then concatenate the context vector with the embedded input before feeding into the GRU # Learn from the previous DecoderRNN implementation with teacher forcing and without teacher forcing # Particularly, pay attention to the shape of decoder_input at every step # Add your code here!!! raise NotImplementedError("Please implement the forward function of AttnDecoderRNN")

In [ ]:

# train a new model with AttnDecoderRNN
hidden_size = 128
batch_size = 32

input_lang, output_lang, train_dataloader = get_dataloader(batch_size)

# let redefine a new encoder and attention decoder
encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)
attn_decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)
train(train_dataloader, encoder, attn_decoder, 80, print_every=5, plot_every=5)

# train a new model with AttnDecoderRNN hidden_size = 128 batch_size = 32 input_lang, output_lang, train_dataloader = get_dataloader(batch_size) # let redefine a new encoder and attention decoder encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device) attn_decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device) train(train_dataloader, encoder, attn_decoder, 80, print_every=5, plot_every=5)

In [ ]:

# why we need to set eval mode here?
# Because some layers like dropout and batchnorm behave differently during training and evaluation.
# In evaluation mode, dropout is disabled and batchnorm uses running statistics instead of batch statistics.
# Be sure to call model.eval() before running inference and model.train() before training.
encoder.eval()
attn_decoder.eval()
evaluateRandomly(encoder, attn_decoder)

# why we need to set eval mode here? # Because some layers like dropout and batchnorm behave differently during training and evaluation. # In evaluation mode, dropout is disabled and batchnorm uses running statistics instead of batch statistics. # Be sure to call model.eval() before running inference and model.train() before training. encoder.eval() attn_decoder.eval() evaluateRandomly(encoder, attn_decoder)

In [ ]:

# get the bleu score on the whole dataset
bleu_score = evaluateDataset(encoder, attn_decoder , input_lang, output_lang, pairs)
print("BLEU score on the whole dataset: %.4f" % bleu_score)

# get the bleu score on the whole dataset bleu_score = evaluateDataset(encoder, attn_decoder , input_lang, output_lang, pairs) print("BLEU score on the whole dataset: %.4f" % bleu_score)

A useful property of the attention mechanism is its highly interpretable outputs. Because it is used to weight specific encoder outputs of the input sequence, we can imagine looking where the network is focused most at each time step.

You could simply run plt.matshow(attentions) to see attention output displayed as a matrix. For a better viewing experience we will do the extra work of adding axes and labels:

In [ ]:

def showAttention(input_sentence, output_words, attentions):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    cax = ax.matshow(attentions.cpu().numpy(), cmap='bone')
    fig.colorbar(cax)

    # Set up axes
    ax.set_xticklabels([''] + input_sentence.split(' ') +
                       ['<EOS>'], rotation=90)
    ax.set_yticklabels([''] + output_words)

    # Show label at every tick
    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))
    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))

    plt.show()


def evaluateAndShowAttention(input_sentence):
    output_words, attentions = evaluate(encoder, decoder, input_sentence, input_lang, output_lang)
    print('input =', input_sentence)
    print('output =', ' '.join(output_words))
    showAttention(input_sentence, output_words, attentions[0, :len(output_words), :])


evaluateAndShowAttention('il n est pas aussi grand que son pere')

evaluateAndShowAttention('je suis trop fatigue pour conduire')

evaluateAndShowAttention('je suis desole si c est une question idiote')

evaluateAndShowAttention('je suis reellement fiere de vous')

def showAttention(input_sentence, output_words, attentions): fig = plt.figure() ax = fig.add_subplot(111) cax = ax.matshow(attentions.cpu().numpy(), cmap='bone') fig.colorbar(cax) # Set up axes ax.set_xticklabels([''] + input_sentence.split(' ') + ['<EOS>'], rotation=90) ax.set_yticklabels([''] + output_words) # Show label at every tick ax.xaxis.set_major_locator(ticker.MultipleLocator(1)) ax.yaxis.set_major_locator(ticker.MultipleLocator(1)) plt.show() def evaluateAndShowAttention(input_sentence): output_words, attentions = evaluate(encoder, decoder, input_sentence, input_lang, output_lang) print('input =', input_sentence) print('output =', ' '.join(output_words)) showAttention(input_sentence, output_words, attentions[0, :len(output_words), :]) evaluateAndShowAttention('il n est pas aussi grand que son pere') evaluateAndShowAttention('je suis trop fatigue pour conduire') evaluateAndShowAttention('je suis desole si c est une question idiote') evaluateAndShowAttention('je suis reellement fiere de vous')

Section 1.5 Transformer-based Machine Translation (25')¶

There two excellent transformer jupyternote books at large, which covers a great amount details of transformer that we could cannot cover more in the class: "The Illustrated Transfomer" by Jay Alammar and "The Annotated Transformer" by harvardnlp lab.

Some one combined this two and create a single jupternotebook here. Please read through them to understand the details of transformer. Train a tranfromer-based model on the above machine translation dataset from scratch.

In [ ]:

# please implement a transformer-based model for the above translation task
# define a seq2seq transformer model for machine translation
class Seq2SeqTransformer(nn.Module):
    def __init__(self, input_vocab_size, output_vocab_size, hidden_size, 
                 nhead, num_encoder_layers, num_decoder_layers, dim_feedforward=256, dropout=0.1):
        # dim_feedforward: the dimension of the feedforward network model in nn.TransformerEncoder and nn.TransformerDecoder
        # in this turorial, given the small size of the dataset, we use a small dim_feedforward
        # You may try a larger dim_feedforward in self-exploration!!!
        super(Seq2SeqTransformer, self).__init__()
        # besides the embedding layers, we also add a positional encoding layer to identify the position of each token in the sequence
        # it is shared between source and target
        


    def forward(self, src, tgt):
        # besides the embedding layers, we also need to add the positional encoding to the embedded input
        # assume source and target are already on the same device as the model.
        output = None
        return F.log_softmax(output, dim=-1)

# please implement a transformer-based model for the above translation task # define a seq2seq transformer model for machine translation class Seq2SeqTransformer(nn.Module): def __init__(self, input_vocab_size, output_vocab_size, hidden_size, nhead, num_encoder_layers, num_decoder_layers, dim_feedforward=256, dropout=0.1): # dim_feedforward: the dimension of the feedforward network model in nn.TransformerEncoder and nn.TransformerDecoder # in this turorial, given the small size of the dataset, we use a small dim_feedforward # You may try a larger dim_feedforward in self-exploration!!! super(Seq2SeqTransformer, self).__init__() # besides the embedding layers, we also add a positional encoding layer to identify the position of each token in the sequence # it is shared between source and target def forward(self, src, tgt): # besides the embedding layers, we also need to add the positional encoding to the embedded input # assume source and target are already on the same device as the model. output = None return F.log_softmax(output, dim=-1)

Hint:

• Complete the above Seq2SeqTransformer Model, according to the above tutorial in RNN, and Language model tutorial here. https://github.com/pytorch/examples/blob/main/word_language_model/model.py#L107
• Rewrite the training functions for the transformer mode here
• Rewrite the evaluation functions for the transformer model
• You may find that your transformer fails at the beginning, please try different configurations to fix it, e.g., reduce your the parameters size or increase teaching forcing rate, or enlarge the dataset.

Part 2: Transformer and Pretrained Language Models(Total: 65)¶

In Assignment 2 and 3, you have worked on a 5-way sentiment classification dataset (SST-5), which has 5 labels: very positive, positive, neutral, negative, very negative. However, the performance is realtive low on regular logistic regression, and get improved when using word embedding via Deep Average Network and LSTM.

In this part, based on the same SST-5 dataset, you have two tasks:

• Section 2.1. Using builtin pytorch transformer encoder layers with the same given word embedding(used in Assignment 3) to improve the performance, and compare with previous models. (10')
• Section 2.2 Using the library of Huggingface Transformers to improve your model with BERT Finetuning.(20')

Important Hints:

1. You will find a ton of existing code for this two tasks, it is ok to refer or reuse some of those code.
2. To run a batch job on OSCER, you need to reassemble the code from the notebook into regular source code to submit a slurm job.
3. No matter use tranformer from scratch or pretrained BERT, the classification head and loss are the same as initial logistic regression model.

What to turn in: You need to turn in the code and a pdf report to show the performance of your two new models on SST-5 dataset. Performance Metrics:

• Classification Report in Sklearn (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html)
• Confusion Matrix (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html)

Section 2.1: Transformer-based Sementiment Classification Model(20')¶

Based on the assignment on DAN and LSTM-based sentiment classifier, please replace the encoder layers for encoding a sentence with transformer encoder. Please write your best model trials with transformer encoder for the sentimental classification. Hint:

• Please still use the given pretrained glove word embedding for fair comparasion
• You need to find the best hypermeterapter for your model.
• Don't missing the positional embedding, Here is a tutorial of using transformer to training a language model, please adapt it to classification task. https://github.com/pytorch/examples/blob/acc295dc7b90714f1bf47f06004fc19a7fe235c4/word_language_model/model.py#L107

Section 2.2 Huggingface Transformer, Various Pretrained LMs (45')¶

Your goal: use the huggingface transformers library to rebuilt your sentiment classifier There are a lot of old version codes of using BERT for SST-5, DON"T use them but read them will help you understand. They are obsolete but they give more details on how to assemble your own pytorch model with BERT. Such as this one. https://github.com/munikarmanish/bert-sentiment/blob/master/bert_sentiment/train.py

The most recent version of huggingface is easy to use but hide many details. I hope to make your life easier, please use the eaiser pytorch version(using Huggingface Trainer) of this following tutoral for SST-5.

Section 2.2.1 Tutorial on Hugginface Transformer Trainer for BERT(25')¶

Please run through the following tutorial. https://huggingface.co/docs/transformers/en/tasks/sequence_classification

When you run the above jupternobook, you will get prompted to two accounts:

1. Create a huggingface account and create a access token to login in this note book.
2. Create a wandb account to keep the log into the wandb, which has been automaitically integrated into hugginface trainer to track your experiments. See the details here. https://docs.wandb.ai/guides/integrations/huggingface/

The above tutorial will take you about 40 minutes to run on an old Nvidia Tesla T4 GPU with the free version of Colab, and sometimes it is slow. So try the Oscer first, then paid Colab GPU. Through this tutorial, you will learn how to use trainer, evaluator, pipeline, accelerator in huggingface library.

Your Task:

• See if you could replace the dataset with SST-5, and the model/tokenizers/configs with "bert-base-cased" and get familiar with this new framework for your sentiment classifier, report the final performance.

(HINT: you almost only need to change the parameters in AutoTokenizer, and AutoModelXX, the learning rate for finetuning is often small (around 10^-3 to 10^-5), the epoch is also around 3 to 10)

What to Report:

• Your training hyperparameters.
• Performance Metrics

Section 2.2.3 Sentimental Classification via Prompting LLMs (20')¶

In this section, you need to write a program to use AutoModelForCausalLM to load the pretrained model and prompt Google's "gemma-3-1b-it" (if 1b version not work for you, please use the 270m version) for the SST-5 classification task, and report the performance on test set only.

Hint:

• Create a Hugging Face account.
• Accept Google terms on the gemma-3-1b-it model page.
• Generate an Access Token with read permission.
• Set your Access Token as an environment variable, not in your code : export HF_TOKEN, see here: https://medium.com/@oadaramola/a-pitfall-i-almost-fell-into-d1d3461b2fb8
• Read through the pretrained models tutorial at https://community.ibm.com/community/user/blogs/ruslan-idelfonso-magaa-vsevolodovna/2023/10/05/how-to-work-with-pretrained-models-with-transforme
• Read the tutorial about structured output part, for each sentence in the SST-5 test set, prompt "gemma-3-1b-it" to produce one of the five labels. https://huggingface.co/docs/inference-providers/en/guides/structured-output
• Use OSCER to submit this job if you had try your best on your laptop.
• Report the performance and compare with the previous models.

Task 2.2.3.1 Sentiment Classifier without Constraint (5')¶

In [27]:

# use https://huggingface.co/google/gemma-3-1b-it to infer on the SST-5 dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import BitsAndBytesConfig
# Ignore this you don't have the GPU or 
# quantization_config = BitsAndBytesConfig(load_in_8bit=True)
# model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it", quantization_config=quantization_config).eval()

model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it").eval() 
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")
inputs = tokenizer("The move is nice. The sentiment is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=5)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])

# use https://huggingface.co/google/gemma-3-1b-it to infer on the SST-5 dataset from transformers import AutoModelForCausalLM, AutoTokenizer from transformers import BitsAndBytesConfig # Ignore this you don't have the GPU or # quantization_config = BitsAndBytesConfig(load_in_8bit=True) # model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it", quantization_config=quantization_config).eval() model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it").eval() tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it") inputs = tokenizer("The move is nice. The sentiment is", return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=5) print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])

The move is nice. The sentiment is appropriate and the focus is

We found that the above output is relative open. The output is not only within the 5 labels in sst-5. Evenif you control that into 2-3 tokens, it still goes wrong. Of course, you could give more guidance in the prompt as follows

In [31]:

inputs = tokenizer("The move is so great. Which is the sentiment among very positive, positive, neutral, negative, very negative.", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=5)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])

inputs = tokenizer("The move is so great. Which is the sentiment among very positive, positive, neutral, negative, very negative.", return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=5) print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])

The move is so great. Which is the sentiment among very positive, positive, neutral, negative, very negative.

There's a

However, the output still requires some parsing and random. Hence, in the next step, we will introduce pydantic and outlines to help you build your prompt and parse your results.

Task 2.2.3.2 Chat Format, Pydantic and Outlines (5')¶

ifferent LLM will use different details to support the chat format, such as adding special tokens or special separators between messages. For example LlaMA-3 has the following

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Write a haiku about autumn.<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

While the OpenAI/Mistral will use the <|im_start|> and <|im_end|>

You could use the Hugginface tokenizer.apply_chat_format to understand the details. The following is an example to understand Gemma3-1b, which is another different format.

Here are more tutorials on pydantic and outlines here. https://github.com/dottxt-ai/outlines

In [ ]:

from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer
from typing import Literal
import outlines

import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
trans_model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")

# use the outlines to wrap the model
# https://dottxt-ai.github.io/outlines/latest/features/models/transformers/
model = outlines.from_transformers(
    trans_model, tokenizer
)

# Define the output categories using Literal
SentimentCategory = Literal[
    "very positive",
    "positive",
    "neutral",
    "negative",
    "very negative"
]

# define the general chat messages template
messages = [
    [
        {
            "role": "system",
            "content": [{"type": "text", "text": "You are a helpful assistant."},]
        },
        {
            "role": "user",
            "content": [{"type": "text", "text": "Classify this sentence into 5 categories: very positive, positive, neutral, negative, very negative.\n Text: {{ text }}"},]
        },
    ],
]

# Different LLM will use different details to support the above chat format. 
# such as adding special tokens or special separators between messages.

prompt_template = tokenizer.apply_chat_template(messages, tokenize=False)[0]
print(prompt_template)
outline_template = outlines.Template.from_string(prompt_template)

texts =[
    "I loved the soundtrack but the story was weak."
    "The movie is not great, but it has some good moments.",
]

sentiment_rst = model(
    outline_template(text=texts[0]),
    SentimentCategory, max_new_tokens=10
    )

print(sentiment_rst)

from pydantic import BaseModel from transformers import AutoModelForCausalLM, AutoTokenizer from typing import Literal import outlines import os os.environ["TOKENIZERS_PARALLELISM"] = "false" trans_model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it") tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it") # use the outlines to wrap the model # https://dottxt-ai.github.io/outlines/latest/features/models/transformers/ model = outlines.from_transformers( trans_model, tokenizer ) # Define the output categories using Literal SentimentCategory = Literal[ "very positive", "positive", "neutral", "negative", "very negative" ] # define the general chat messages template messages = [ [ { "role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."},] }, { "role": "user", "content": [{"type": "text", "text": "Classify this sentence into 5 categories: very positive, positive, neutral, negative, very negative.\n Text: {{ text }}"},] }, ], ] # Different LLM will use different details to support the above chat format. # such as adding special tokens or special separators between messages. prompt_template = tokenizer.apply_chat_template(messages, tokenize=False)[0] print(prompt_template) outline_template = outlines.Template.from_string(prompt_template) texts =[ "I loved the soundtrack but the story was weak." "The movie is not great, but it has some good moments.", ] sentiment_rst = model( outline_template(text=texts[0]), SentimentCategory, max_new_tokens=10 ) print(sentiment_rst)

<bos><start_of_turn>user
You are a helpful assistant.

Classify this sentence into 5 categories: very positive, positive, neutral, negative, very negative.
 Text: {{ text }}<end_of_turn>

neutral

• Please change the above code and prompt for whole your sst-5 test datasets.
• Open exploration: please use batch processing to speedup. (optional)
• If you laptop doesnot work for inference this 1b model, please write it as sbatch job to run on OSCER.
• You are encouraged to run larger models or other models who support this structured output

Task 2.2.3.3 Sentiment Classifier with Explaination (10')¶

Still with the pydantic and outlines, could you design a program, (1) first think step by step on generating an explaination for a classification. (2) output that label.

For example, the input sentence "I loved the soundtrack but the story was weak.". Your prompt should be like Analyze the sentiment of this sentence with reasoning steps first, and then classifier it into 5 categories: very positive, positive, neutral, negative, very negative.\n

Your model should output a structured output as following two parts in a json file.

{"reason": "The sentence expresses both positive sentiment towards the soundtrack and a slight criticism of the story.", "label": "neutral"}

Hint:

• This is an open design question, please feel free to use any prompt or tools, such as LangChain, LlamaInex. But we suggest you use pydantic and outlines. They are all similar.
• Please read more exmaples about complex structures to support both the reasoning and label. https://github.com/dottxt-ai/outlines
• Use model_validate_json in pydantic. https://docs.pydantic.dev/latest/concepts/models/#basic-model-usage