Assignment 2: Sentence Classification and Logistic Regression¶

Programming Assignment (Total: 100 points)¶

For this assignment we will be implementing a naive bayes baseline classifier. Additionally, we will be using pytorch to implement a binary logistic regression classifier. Our task is sentiment classification for hotel reviews. The input to your model will be a text review, and the output label is a 1 or 0 marking it as positive or negative.

We have provided a util.py file for loading the data, and some of the basic modeling. Your task is to fill in the functions below in order to train as accurate a classifier as possible!

We suggest browsing the util.py script first. Additionally, make sure to install dependencies from the provided requirements.txt file in a similar fashion to the pytorch tutorial. With your environment activated int he terminal, run:

mamba env create -n cs5293-2 python=3.10
pip install -r requirements.txt
##Your VSCode may complain sometime you need to install ipykernel using the following commands. If not, then  just ignore this. 
#mamba install -n cs5293-2 ipykernel  --force-reinstall 

In [2]:

from typing import List
import spacy
import torch
import random
import os
import sys
sys.path.append("../src")
print(sys.path)

from typing import List import spacy import torch import random import os import sys sys.path.append("../src") print(sys.path)

['/Users/jcao/mamba/envs/cs5293-2/lib/python310.zip', '/Users/jcao/mamba/envs/cs5293-2/lib/python3.10', '/Users/jcao/mamba/envs/cs5293-2/lib/python3.10/lib-dynload', '', '/Users/jcao/mamba/envs/cs5293-2/lib/python3.10/site-packages', '../src', '../src']

Section 1: Dataset Exploration (Total: 20 Points)¶

The training data for this task consists of a collection of short hotel reviews. The data is formatted as one review per line. Each line starts with a unique identifier for the review (as in ID-2001) followed by tab and the text of the review. The reviews are not tokenized or sentence segmented in any way (the words are space separated). The positive reviews and negative reviews appear in separate files namely hotelPosT-train.txt and hotelNegT-train.txt.

In [5]:

from util import load_train_data
pos_datapath = "../data/hotelPosT-train.txt"
neg_datapath = "../data/hotelNegT-train.txt"
all_texts, all_labels = load_train_data(pos_datapath, neg_datapath)
assert len(all_texts) == len(all_labels), "Mismatch between texts and labels"
print(f"Loaded {len(all_texts)} texts in training data.")

from util import load_train_data pos_datapath = "../data/hotelPosT-train.txt" neg_datapath = "../data/hotelNegT-train.txt" all_texts, all_labels = load_train_data(pos_datapath, neg_datapath) assert len(all_texts) == len(all_labels), "Mismatch between texts and labels" print(f"Loaded {len(all_texts)} texts in training data.")

Loaded 189 texts in training data.

Lets look at what is in the data¶

In [11]:

def random_sample(texts, labels, label):
    data_by_label = {}
    for lab, text in zip(labels, texts):
        if lab not in data_by_label:
            data_by_label[lab] = []
        data_by_label[lab].append(text)
    return random.choice(data_by_label[label])

print("--- Positive Example ---")
print(random_sample(all_texts, all_labels, label=1))
print("\n--- Negative Example ---")
print(random_sample(all_texts, all_labels, label=0))

def random_sample(texts, labels, label): data_by_label = {} for lab, text in zip(labels, texts): if lab not in data_by_label: data_by_label[lab] = [] data_by_label[lab].append(text) return random.choice(data_by_label[label]) print("--- Positive Example ---") print(random_sample(all_texts, all_labels, label=1)) print("\n--- Negative Example ---") print(random_sample(all_texts, all_labels, label=0))

--- Positive Example ---
On my last vacation, I decided I wanted to go to a place that was quaint and quiet, but still had a suitable level of culture to partake in. This place turned out to be Charleston, SC. After doing research on hotels in the area, I decided on the Planters Inn on Market Street, a quite luxurious hotel for the area featuring a 4-star dining experience. Upon reaching the hotel the staff was friendly and greeted me warmly. Check-in was a breeze and soon I was off to my king room accommodations. Outside the room was a little round, iron-wrought table set for two overlooking the courtyard. The tall white columns were freshly washed and looked new and fresh plants were out. (The weather was a bit dreary the entire time, so I didn't get to enjoy that feature of the hotel very much.) Once inside with my luggage, the first thing I noticed was a card with my name on it and a teddy bear in the middle of my ginormous bed. The card welcomed me and the bear ultimately became a keepsake for my nephew. The bedroom was beautiful, and the bathroom was just as nice. The first thing I did was take a nice long, hot bath in the jacuzzi. It was the perfect way to unwind after my drive from Atlanta where I'd just barely fled some unseasonable snow. After the bath, I ordered the duck from the 4-star Peninsula Grill and a glass of champagne. It was my first vacation in almost a decade and I planned to live it up! My stay lasted a week, and all in all, each day was the a slightly different rendition of the first. Between sightseeing, visiting shops for gifts and keepsakes, and hitting up a few bead shops for my jewelry line on Etsy, I got to relax in some of the most luxurious accommodations I have ever had the pleasure of partaking in. The staff, from the front desk to the housekeepers, were all friendly and quick to answer any request. (Once I saw a bellman delivering chocolate covered strawberries that were not on the menu, he made sure I got some too!) It was a fabulous hotel and a place I plan to visit again in the future. Thanks Planters Inn!

--- Negative Example ---
This Red Roof Inn in North Charleston, South Carolina is pretty inexpensive, but you definitely get what you pay for. The location is terrible. It’s right off the interstate in a very busy, loud commercial area. It should also be noted that it’s not as close to the historic downtown and beach areas in Charleston as you might be lead to believe. We got there a little early and the room wasn’t ready, so we went to a nearby Starbucks for about an hour. We came back after check-in time, and the room still wasn’t ready. They lady at the desk was nice but incompetent. When we finally got into a room, it was what you’d expect from a cheap hotel, and it smelled of cigarettes and cheap cleaning products. I think the worst part of it all was the other clientele there. We had a room on the second floor, which required climbing two flights of stairs. Some very unsavory individuals were always hanging out on the stairs and the landing. It made me really uncomfortable to walk to and from my car. Next time, I’ll pay extra and stay somewhere else.

Test Data ( WAIT TILL DEADLINE)¶

This is the test dataset that you will need to use to report the results on. This set is the unseen dataset meaning, you are not in anyway supoose to look what is in this dataset.

In [12]:

### RUN THIS ONLY ON DEADLINE ###
# Load the test data

from util import load_inference_data
from typing import List, Tuple, Any

def load_test_data(filepath: str) -> Tuple[List[Any], List[Any]]:
    """Load the test data, producing a List of texts, labels
    Args:
        filepath (str): Path to the training file

    Returns:
        Tuple[List[Any], List[Any]]: The texts and labels
    """
    lab_map = {'POS': 1, 'NEG': 0}
    texts = []
    labels = []
    with open(filepath, "r") as file:
        for line in file:
            idx, text, label = line.rstrip().split("\t")
            texts.append(text)
            labels.append(lab_map[label])

    return texts, labels

test_datapath = "../data/HW2-testset.txt"
test_texts, test_labels = load_test_data(test_datapath)

### RUN THIS ONLY ON DEADLINE ### # Load the test data from util import load_inference_data from typing import List, Tuple, Any def load_test_data(filepath: str) -> Tuple[List[Any], List[Any]]: """Load the test data, producing a List of texts, labels Args: filepath (str): Path to the training file Returns: Tuple[List[Any], List[Any]]: The texts and labels """ lab_map = {'POS': 1, 'NEG': 0} texts = [] labels = [] with open(filepath, "r") as file: for line in file: idx, text, label = line.rstrip().split("\t") texts.append(text) labels.append(lab_map[label]) return texts, labels test_datapath = "../data/HW2-testset.txt" test_texts, test_labels = load_test_data(test_datapath)

Task 1.1: Print the number of "positive" and "negative" samples (5 Points)¶

It is important to know the distribution of the training examples. More often than not, you will have to work with datasets that are not "balanced" with respect to the labels of the samples. For this task, print out the number of examples that have label = 1 and label = 0, respectively, in std:out or plot a pie chart.

In [6]:

import matplotlib.pyplot as plt
# What is matplotlib? Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.

import matplotlib.pyplot as plt # What is matplotlib? Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.

In [14]:

### ENTER CODE HERE ###

# Note since we have them in two seperate files, 
# this can also be done with bash commands
def label_distribution(labels):
    """
    TODO: Replace the line `raise NotImplementedError` with your code
    to print the labels distribution.
    """
    raise NotImplementedError
    ### ENTER CODE HERE ###


label_distribution(all_labels)

### ENTER CODE HERE ### # Note since we have them in two seperate files, # this can also be done with bash commands def label_distribution(labels): """ TODO: Replace the line `raise NotImplementedError` with your code to print the labels distribution. """ raise NotImplementedError ### ENTER CODE HERE ### label_distribution(all_labels)

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[14], line 14
     10     raise NotImplementedError
     11     ### ENTER CODE HERE ###
---> 14 label_distribution(all_labels)

Cell In[14], line 10, in label_distribution(labels)
      5 def label_distribution(labels):
      6     """
      7     TODO: Replace the line `raise NotImplementedError` with your code
      8     to print the labels distribution.
      9     """
---> 10     raise NotImplementedError

NotImplementedError: 

Task 1.2: Split Training and Development Sets (5 Points)¶

For the purpose of coming with the best parameters for the model you will have to split the overall "training" dataset into training and development sets. Make sure the splits follow the similar distribution.

In [15]:

### ENTER CODE HERE ###

def split_dataset(texts, labels):
    """
    Split the dataset randomly into 80% training and 20% development set
    Make sure the splits have the same label distribution
    """
    train_texts = []
    train_labels = []
    dev_texts = []
    dev_labels = []
    raise NotImplementedError
    return train_texts, train_labels, dev_texts, dev_labels

train_texts, train_labels, dev_texts, dev_labels = split_dataset(all_texts, all_labels)

print('Train Label Distribution:')
label_distribution(train_labels)

print('Dev Label Distribution:')
label_distribution(dev_labels)

### ENTER CODE HERE ### def split_dataset(texts, labels): """ Split the dataset randomly into 80% training and 20% development set Make sure the splits have the same label distribution """ train_texts = [] train_labels = [] dev_texts = [] dev_labels = [] raise NotImplementedError return train_texts, train_labels, dev_texts, dev_labels train_texts, train_labels, dev_texts, dev_labels = split_dataset(all_texts, all_labels) print('Train Label Distribution:') label_distribution(train_labels) print('Dev Label Distribution:') label_distribution(dev_labels)

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[15], line 15
     12     raise NotImplementedError
     13     return train_texts, train_labels, dev_texts, dev_labels
---> 15 train_texts, train_labels, dev_texts, dev_labels = split_dataset(all_texts, all_labels)
     17 print('Train Label Distribution:')
     18 label_distribution(train_labels)

Cell In[15], line 12, in split_dataset(texts, labels)
     10 dev_texts = []
     11 dev_labels = []
---> 12 raise NotImplementedError
     13 return train_texts, train_labels, dev_texts, dev_labels

NotImplementedError: 

Task 1.3: Evaluation Metrics (10 Points)¶

Implement the evaulation metrics: Accuracy, Precision, Recall and F1 score

In [ ]:

### ENTER CODE HERE ###

def accuracy(predicted_labels, true_labels):
    """
    Accuracy is correct predictions / all predicitons
    """
    raise NotImplementedError

def precision(predicted_labels, true_labels):
    """
    Precision is True Positives / All Positives Predictions
    """
    raise NotImplementedError

def recall(predicted_labels, true_labels):
    """
    Recall is True Positives / All Positive Labels
    """
    raise NotImplementedError

def f1_score(predicted_labels, true_labels):
    """
    F1 score is the harmonic mean of precision and recall
    """
    raise NotImplementedError

### ENTER CODE HERE ### def accuracy(predicted_labels, true_labels): """ Accuracy is correct predictions / all predicitons """ raise NotImplementedError def precision(predicted_labels, true_labels): """ Precision is True Positives / All Positives Predictions """ raise NotImplementedError def recall(predicted_labels, true_labels): """ Recall is True Positives / All Positive Labels """ raise NotImplementedError def f1_score(predicted_labels, true_labels): """ F1 score is the harmonic mean of precision and recall """ raise NotImplementedError

In [ ]:

### DO NOT EDIT ###
import sklearn.metrics as metrics
em_test_labels = [0]*6 + [1]*4
em_test_predictions = [0]*8 + [1]*2

# using sklearn metrics as the ground truth to test your own implementation
# 0.8
em_test_accuracy = metrics.accuracy_score(em_test_labels, em_test_predictions)
# 1.0
em_test_precision = metrics.precision_score(em_test_labels, em_test_predictions) 
# 0.5
em_test_recall = metrics.recall_score(em_test_labels, em_test_predictions)
# 2/3
em_test_f1 = metrics.f1_score(em_test_labels, em_test_predictions)

assert accuracy(em_test_predictions, em_test_labels) == em_test_accuracy
assert precision(em_test_predictions, em_test_labels) == em_test_precision 
assert recall(em_test_predictions, em_test_labels) == em_test_recall
assert f1_score(em_test_predictions, em_test_labels) == em_test_f1

print('All Test Cases Passed!')

### DO NOT EDIT ### import sklearn.metrics as metrics em_test_labels = [0]*6 + [1]*4 em_test_predictions = [0]*8 + [1]*2 # using sklearn metrics as the ground truth to test your own implementation # 0.8 em_test_accuracy = metrics.accuracy_score(em_test_labels, em_test_predictions) # 1.0 em_test_precision = metrics.precision_score(em_test_labels, em_test_predictions) # 0.5 em_test_recall = metrics.recall_score(em_test_labels, em_test_predictions) # 2/3 em_test_f1 = metrics.f1_score(em_test_labels, em_test_predictions) assert accuracy(em_test_predictions, em_test_labels) == em_test_accuracy assert precision(em_test_predictions, em_test_labels) == em_test_precision assert recall(em_test_predictions, em_test_labels) == em_test_recall assert f1_score(em_test_predictions, em_test_labels) == em_test_f1 print('All Test Cases Passed!')

Section 2: Logsitic Regression (Total: 50 Points)¶

It is important to come up with baselines for the classifications to compare the more complicated models with. The baselines are also useful as a debugging method for your actual classfication model. You will create two baselines:

• Task 2.1. Baseline: Random Chance Classifier (10')
• Task 2.2. Logstic Classifier (40')

Task 2.1: Baseline: Random Chance Classifier (10 Points)¶

2.1.1: Implementing Random Chance Classifier (10 Points)¶

A random chance classifier predicts the label according to the label's distribution. As an example, if the label 1 appears 70% of the times in the training set, you predict 70 out of 100 times the label 1 and label 0 30% of the times

In [ ]:

### ENTER CODE HERE ###

def predict_random(train_labels, num_samples):
    """
    Using the label distribution, predict the label num_sample number of times
    """
    raise NotImplementedError

### ENTER CODE HERE ### def predict_random(train_labels, num_samples): """ Using the label distribution, predict the label num_sample number of times """ raise NotImplementedError

2.1.2: Random Baseline Results¶

Report the results you achieve with the random baselines by running the following cell:

In [ ]:

### DO NOT EDIT ###

### DEV SET RESULTS

## predict 
devset_prediction_random = predict_random(train_labels, num_samples=len(dev_labels))
print('Random Chance F1:', f1_score(devset_prediction_random, dev_labels))

### DO NOT EDIT ### ### DEV SET RESULTS ## predict devset_prediction_random = predict_random(train_labels, num_samples=len(dev_labels)) print('Random Chance F1:', f1_score(devset_prediction_random, dev_labels))

In [ ]:

### DO NOT EDIT ###
### RUN THIS ONLY ON DEADLINE ###
### TEST SET RESULTS

testset_prediction_random = predict_random(train_labels, num_samples=len(test_labels))
print('Random Chance F1:', f1_score(testset_prediction_random, test_labels))

### DO NOT EDIT ### ### RUN THIS ONLY ON DEADLINE ### ### TEST SET RESULTS testset_prediction_random = predict_random(train_labels, num_samples=len(test_labels)) print('Random Chance F1:', f1_score(testset_prediction_random, test_labels))

Task 2.2: Logistic Regression on Features (Total: 40 Points)¶

Now let's try building a logistic regression based classifier on hand-engineered features.

The following tasks are going to be the implementation of the components required in building a Logistic Regressor.

Task 2.2.1: Preprocessing and Feature Extraction (20 Points)¶

To tokenize the text and help extract features from text, we will use the popular spaCy model (https://spacy.io)

2.2.1.1 Play with Spacy Model (0')¶

In [ ]:

### DO NOT EDIT ###

# Initialize the spacy model
nlp = spacy.load('en_core_web_sm')

### DO NOT EDIT ### # Initialize the spacy model nlp = spacy.load('en_core_web_sm')

In [ ]:

### ENTER CODE HERE ###

test_string = "This is an amazing sentence"

# parse the string with spacy model
test_doc = nlp(test_string)

print('Token', 'Lemma', 'Is_Stopword?')
for token in test_doc:
    print(token, token.lemma_, token.is_stop)

### ENTER CODE HERE ### test_string = "This is an amazing sentence" # parse the string with spacy model test_doc = nlp(test_string) print('Token', 'Lemma', 'Is_Stopword?') for token in test_doc: print(token, token.lemma_, token.is_stop)

2.2.1.2 Processing the Text (5')¶

In [ ]:

### ENTER CODE HERE ###

def pre_process(text: str) -> List[str]:
    """
    remove stopwords and lemmatize and return an array of lemmas
    """
    raise NotImplementedError
    
test_string = "This sentence needs to be lemmatized"

assert len({'sentence', 'need', 'lemmatize', 'lemmatiz'}.intersection(pre_process(test_string))) >= 3

print('All Test Cases Passed!')

### ENTER CODE HERE ### def pre_process(text: str) -> List[str]: """ remove stopwords and lemmatize and return an array of lemmas """ raise NotImplementedError test_string = "This sentence needs to be lemmatized" assert len({'sentence', 'need', 'lemmatize', 'lemmatiz'}.intersection(pre_process(test_string))) >= 3 print('All Test Cases Passed!')

2.2.1.3: Feature Extraction (10 points)¶

This is perhaps the most challenging part of this assignment. In the class, we went over how to featurize text for a classification system for sentiment analysis. In this assignment, you should implement and build upon this to accuractely classify the hotel reviews.

This task requires a thorough understanding of the dataset to answer the important question, "What is in the data?". Please go through some of the datapoints and convert the signals that you think might help in identifying "sentiment" as features.

Please refer to the section in Jim's book that illustrates the process of feature engineering for this task. We have attached an image of the table below:

Please use the files with postive and negative words attached in the assignment: positive_words.txt and negative_words.txt

In [ ]:

### ENTER CODE HERE ###

def make_test_feature(text: spacy.tokens.doc.Doc):
    return "happy" in [t.lemma_ for t in text]


def extract_features(text: spacy.tokens.doc.Doc):
    features = []
    # TODO: Replace this with your own feature extraction functions.
    features.append(make_test_feature(text))
    # TODO: add more features to the feature vector
    
    return features

### ENTER CODE HERE ### def make_test_feature(text: spacy.tokens.doc.Doc): return "happy" in [t.lemma_ for t in text] def extract_features(text: spacy.tokens.doc.Doc): features = [] # TODO: Replace this with your own feature extraction functions. features.append(make_test_feature(text)) # TODO: add more features to the feature vector return features

In [ ]:

### DO NOT EDIT ###

def featurize_data(texts, labels):
    features = [
        extract_features(doc) for doc in nlp.pipe(texts)
    ]
    return torch.FloatTensor(features), torch.FloatTensor(labels)

### DO NOT EDIT ### def featurize_data(texts, labels): features = [ extract_features(doc) for doc in nlp.pipe(texts) ] return torch.FloatTensor(features), torch.FloatTensor(labels)

2.2.1.4: Feature Scaling (5 Points)¶

In this task we will use the data normalization technique to ensure the scales of the feature are consistent. After featurizing the dataset, we need to call the following function before passing it to the classifier

• Normalization Formula

In [ ]:

### ENTER CODE HERE ###

def normalize(features: torch.Tensor) -> torch.Tensor:
    """
    return the features transformed by the above formula of normalization
    """
    raise NotImplementedError

### ENTER CODE HERE ### def normalize(features: torch.Tensor) -> torch.Tensor: """ return the features transformed by the above formula of normalization """ raise NotImplementedError

Task 2.2.2 Training a Logistic Regression Classifier (Total: 20 Points)¶

In this section, you will implement the components needed to train the binary classifier using logistic regression

• Here we define our pytorch logistic regression classifier (DO NOT EDIT THIS)

In [ ]:

class SentimentClassifier(torch.nn.Module):
    def __init__(self, input_dim: int):
        super().__init__()
        # We force output to be one, since we are doing binary logistic regression
        self.output_size = 1
        self.coefficients = torch.nn.Linear(input_dim, self.output_size)
        # Initialize weights. Note that this is not strictly necessary,
        # but you should test different initializations per lecture
        initialize_weights(self.coefficients)
        
    def forward(self, features: torch.Tensor):
        # We predict a number by multipling by the coefficients
        # and then take the sigmoid to turn the logits into probabilities.
        raise NotImplementedError

class SentimentClassifier(torch.nn.Module): def __init__(self, input_dim: int): super().__init__() # We force output to be one, since we are doing binary logistic regression self.output_size = 1 self.coefficients = torch.nn.Linear(input_dim, self.output_size) # Initialize weights. Note that this is not strictly necessary, # but you should test different initializations per lecture initialize_weights(self.coefficients) def forward(self, features: torch.Tensor): # We predict a number by multipling by the coefficients # and then take the sigmoid to turn the logits into probabilities. raise NotImplementedError

2.2.2.1 : Initialize the weights. (5 Points)¶

Initialization of the parameters is an important step to ensure the SGD algorithm converges to a global optimum. Typically, we need to try different initialization methods and compare the accuracy we achieve for the development set. In this task, implement the function that initializes the parameters to ...

In [ ]:

### ENTER CODE HERE ###

def initialize_weights(coefficients):
    """
    TODO: Replace the line `raise NotImplementedError` with your code.
    Initialize the weights of the coefficients by assigning the parameter
    coefficients.weights.data = ...
    """
    raise NotImplementedError

### ENTER CODE HERE ### def initialize_weights(coefficients): """ TODO: Replace the line `raise NotImplementedError` with your code. Initialize the weights of the coefficients by assigning the parameter coefficients.weights.data = ... """ raise NotImplementedError

Let's build a training function similar to the linear regressor from the tutorial

2.2.2.2: Cross Entropy Loss Function (5 Points)¶

In [ ]:

### ENTER CODE HERE ###

def cross_entropy_loss(prediction: torch.Tensor, label: torch.Tensor) -> torch.Tensor:
    """
    TODO: Implement the logistic loss function between a prediction and label.
    """
    raise NotImplementedError

def negative_log_likihood_loss(prediction: torch.Tensor, label: torch.Tensor) -> torch.Tensor:
    """
    TODO: Implement the negative log likelihood loss function between a prediction and label.
    """
    raise NotImplementedError

### ENTER CODE HERE ### def cross_entropy_loss(prediction: torch.Tensor, label: torch.Tensor) -> torch.Tensor: """ TODO: Implement the logistic loss function between a prediction and label. """ raise NotImplementedError def negative_log_likihood_loss(prediction: torch.Tensor, label: torch.Tensor) -> torch.Tensor: """ TODO: Implement the negative log likelihood loss function between a prediction and label. """ raise NotImplementedError

2.2.2.3: Create an SGD optimizer (0 Points)¶

In the lecture, we only briefly mentioned the optimizer SGD. Here we offered another example for you to create your simple optimizer by implementing the gradient, learning rate, and weight updates. In real prictice, you could simply use existing optimier code in the assignments.

Consider the function you hope to learn is a quadrtic function

In [ ]:

### DO NOT EDIT ###
import numpy as np

def quadratic_loss(x1, x2):
    """
    Assuming we have a loss function, which is a quadratic function of two weight paramters.
    Obviously, this is not a logistic loss function, but we will use it for testing purposes.
    The mimumum of this function is 0 at (1, 1)
    :param x1: first coordinate in weight vector [x1, x2]
    :param x2: second coordinate in weight vector [x1, x2]
    :return:
    """
    return (x1 - 1) ** 2 + 8 * (x2 - 1) ** 2

### DO NOT EDIT ### import numpy as np def quadratic_loss(x1, x2): """ Assuming we have a loss function, which is a quadratic function of two weight paramters. Obviously, this is not a logistic loss function, but we will use it for testing purposes. The mimumum of this function is 0 at (1, 1) :param x1: first coordinate in weight vector [x1, x2] :param x2: second coordinate in weight vector [x1, x2] :return: """ return (x1 - 1) ** 2 + 8 * (x2 - 1) ** 2

In [ ]:

### ENTER CODE HERE ###
def quadratic_loss_grad(x1, x2):
    """
    Should return a numpy array containing the gradient of the quadratic function defined above evaluated at the point
    :param x1: first coordinate in weight vector [x1, x2]
    :param x2: second coordinate in weight vector [x1, x2]
    :return: a one-dimensional numpy array containing two elements representing the gradient
    """
    raise Exception("Implement me!")

### ENTER CODE HERE ### def quadratic_loss_grad(x1, x2): """ Should return a numpy array containing the gradient of the quadratic function defined above evaluated at the point :param x1: first coordinate in weight vector [x1, x2] :param x2: second coordinate in weight vector [x1, x2] :return: a one-dimensional numpy array containing two elements representing the gradient """ raise Exception("Implement me!")

In [ ]:

### DO NOT EDIT ###
import numpy as np
import matplotlib.pyplot as plt

def sgd_test_quadratic(lr: float, epochs: int):
    xlist = np.linspace(-3.0, 3.0, 100)
    ylist = np.linspace(-3.0, 3.0, 100)
    X, Y = np.meshgrid(xlist, ylist)
    Z = quadratic_loss(X, Y)
    plt.figure()

    # Track the points visited here
    points_history = []
    curr_point = np.array([0., 0.])
    for iter in range(0, epochs):
        grad = quadratic_loss_grad(curr_point[0], curr_point[1])
        if len(grad) != 2:
            raise Exception("Gradient must be a two-dimensional array (vector containing [df/dx1, df/dx2])")
        next_point = curr_point - lr * grad
        points_history.append(curr_point)
        print("Point after epoch %i: %s" % (iter, repr(next_point)))
        curr_point = next_point
    points_history.append(curr_point)
    cp = plt.contourf(X, Y, Z)
    plt.colorbar(cp)
    plt.plot([p[0] for p in points_history], [p[1] for p in points_history], color='k', linestyle='-', linewidth=1, marker=".")
    plt.title('SGD on quadratic')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show()

### DO NOT EDIT ### import numpy as np import matplotlib.pyplot as plt def sgd_test_quadratic(lr: float, epochs: int): xlist = np.linspace(-3.0, 3.0, 100) ylist = np.linspace(-3.0, 3.0, 100) X, Y = np.meshgrid(xlist, ylist) Z = quadratic_loss(X, Y) plt.figure() # Track the points visited here points_history = [] curr_point = np.array([0., 0.]) for iter in range(0, epochs): grad = quadratic_loss_grad(curr_point[0], curr_point[1]) if len(grad) != 2: raise Exception("Gradient must be a two-dimensional array (vector containing [df/dx1, df/dx2])") next_point = curr_point - lr * grad points_history.append(curr_point) print("Point after epoch %i: %s" % (iter, repr(next_point))) curr_point = next_point points_history.append(curr_point) cp = plt.contourf(X, Y, Z) plt.colorbar(cp) plt.plot([p[0] for p in points_history], [p[1] for p in points_history], color='k', linestyle='-', linewidth=1, marker=".") plt.title('SGD on quadratic') plt.xlabel('x') plt.ylabel('y') plt.show()

In [ ]:

sgd_test_quadratic(0.1, 20)

sgd_test_quadratic(0.1, 20)

In the above figure, you will see the point from the (0, 0, 9) moving to the minimum of the quardic loss is obtained at (1,1,0) We have already provided the implementation of how to create the SGD optimizer. However, in real practise, our loss are much more complicated than this, especially for the neural networks. Recently large language models are trained with more 650B parameters. In late semester, we also learned that sometimes we need not to train all the parameters during the so-called finetuning stage. We may try different optimizing algorithms more than SGD, such as Adam, AdamW or more. https://pytorch.org/docs/stable/optim.html In this example, you should not change this code, so we use SGD. For the Task 2.3, when you train you own classifier from scratch. You could try any of optimizer.

In [ ]:

def make_optimizer(model, learning_rate) -> torch.optim:
    """
    Returns an Stocastic Gradient Descent Optimizer
    See here for algorithms you can import: https://pytorch.org/docs/stable/optim.html
    """
    return torch.optim.SGD(model.parameters(), learning_rate)

def make_optimizer(model, learning_rate) -> torch.optim: """ Returns an Stocastic Gradient Descent Optimizer See here for algorithms you can import: https://pytorch.org/docs/stable/optim.html """ return torch.optim.SGD(model.parameters(), learning_rate)

2.2.2.4: Converting Logits into Predictions (5 Points)¶

In [ ]:

### ENTER CODE HERE ###

def predict(model, features):
    with torch.no_grad():
        """
        TODO: Replace the line `raise NotImplementedError`
        set a decision boundary, and convert the predicted
        sigmoid into prediction labels (0, 1)
        """
        p = model(features)
        raise NotImplementedError

### ENTER CODE HERE ### def predict(model, features): with torch.no_grad(): """ TODO: Replace the line `raise NotImplementedError` set a decision boundary, and convert the predicted sigmoid into prediction labels (0, 1) """ p = model(features) raise NotImplementedError

2.2.2.5: Training Function (DO NOT EDIT THIS)¶

In [ ]:

### DO NOT EDIT ###

from tqdm.autonotebook import tqdm
import random


def training_loop(
    num_epochs,
    batch_size,
    train_features,
    train_labels,
    dev_features,
    dev_labels,
    optimizer,
    model
):
    samples = list(zip(train_features, train_labels))
    random.shuffle(samples)
    batches = []
    for i in range(0, len(samples), batch_size):
        batches.append(samples[i:i+batch_size])
    print("Training...")
    for i in range(num_epochs):
        losses = []
        for batch in tqdm(batches):
            # Empty the dynamic computation graph
            features, labels = zip(*batch)
            features = torch.stack(features)
            labels = torch.stack(labels)
            optimizer.zero_grad()
            # Run the model
            pred_prob = model(features)
            # Compute loss
            loss = cross_entropy_loss(torch.squeeze(pred_prob), labels)
            # In this logistic regression example,
            # this entails computing a single gradient
            loss.backward()
            # Backpropogate the loss through our model
            
            # Update our coefficients in the direction of the gradient.
            optimizer.step()
             # For logging
            losses.append(loss.item())
        
        # Estimate the f1 score for the development set
        dev_f1 = f1_score(predict(model, dev_features), dev_labels)
        print(f"epoch {i}, loss: {sum(losses)/len(losses)}")
        print(f"Dev F1 {dev_f1}")
        
    # Return the trained model
    return model

### DO NOT EDIT ### from tqdm.autonotebook import tqdm import random def training_loop( num_epochs, batch_size, train_features, train_labels, dev_features, dev_labels, optimizer, model ): samples = list(zip(train_features, train_labels)) random.shuffle(samples) batches = [] for i in range(0, len(samples), batch_size): batches.append(samples[i:i+batch_size]) print("Training...") for i in range(num_epochs): losses = [] for batch in tqdm(batches): # Empty the dynamic computation graph features, labels = zip(*batch) features = torch.stack(features) labels = torch.stack(labels) optimizer.zero_grad() # Run the model pred_prob = model(features) # Compute loss loss = cross_entropy_loss(torch.squeeze(pred_prob), labels) # In this logistic regression example, # this entails computing a single gradient loss.backward() # Backpropogate the loss through our model # Update our coefficients in the direction of the gradient. optimizer.step() # For logging losses.append(loss.item()) # Estimate the f1 score for the development set dev_f1 = f1_score(predict(model, dev_features), dev_labels) print(f"epoch {i}, loss: {sum(losses)/len(losses)}") print(f"Dev F1 {dev_f1}") # Return the trained model return model

2.2.2.6: Train the classifier (5 Points)¶

Run the following cell to train a logistic regressor on your hand-engineered features.

In [ ]:

### DO NOT EDIT ###

num_epochs = 100

train_features_tensor, train_labels_tensor = featurize_data(train_texts, train_labels)
normalized_train_features_tensor = normalize(train_features_tensor)
dev_features_tensor, dev_labels_tensor = featurize_data(dev_texts, dev_labels)
normalized_dev_features_tensor = normalize(dev_features_tensor)
model = SentimentClassifier(normalized_train_features_tensor.shape[1]) 
optimizer = make_optimizer(model, learning_rate=0.01)

trained_model = training_loop(
    num_epochs,
    16,
    normalized_train_features_tensor,
    train_labels_tensor,
    normalized_dev_features_tensor,
    dev_labels_tensor,
    optimizer,
    model
)

### DO NOT EDIT ### num_epochs = 100 train_features_tensor, train_labels_tensor = featurize_data(train_texts, train_labels) normalized_train_features_tensor = normalize(train_features_tensor) dev_features_tensor, dev_labels_tensor = featurize_data(dev_texts, dev_labels) normalized_dev_features_tensor = normalize(dev_features_tensor) model = SentimentClassifier(normalized_train_features_tensor.shape[1]) optimizer = make_optimizer(model, learning_rate=0.01) trained_model = training_loop( num_epochs, 16, normalized_train_features_tensor, train_labels_tensor, normalized_dev_features_tensor, dev_labels_tensor, optimizer, model )

Get the predictions on the Test Set using the Trained model and print the F1 score.

In [ ]:

### DO NOT EDIT ###

### DEV SET RESULTS
dev_features_tensor, dev_labels_tensor = featurize_data(dev_texts, dev_labels)
normalized_dev_features_tensor = normalize(dev_features_tensor)
preds_dev = predict(trained_model, normalized_dev_features_tensor)
print('Logistic Regression Results:')
print('Accuracy:', accuracy(preds_dev, dev_labels))
print('F1-score', f1_score(preds_dev, dev_labels))

### DO NOT EDIT ### ### DEV SET RESULTS dev_features_tensor, dev_labels_tensor = featurize_data(dev_texts, dev_labels) normalized_dev_features_tensor = normalize(dev_features_tensor) preds_dev = predict(trained_model, normalized_dev_features_tensor) print('Logistic Regression Results:') print('Accuracy:', accuracy(preds_dev, dev_labels)) print('F1-score', f1_score(preds_dev, dev_labels))

In [ ]:

### DO NOT EDIT ###
### TEST SET RESULTS
test_features_tensor, dev_labels_tensor = featurize_data(test_texts, test_labels)
normalized_test_features_tensor = normalize(test_features_tensor)
preds_test = predict(trained_model, normalized_test_features_tensor)
# use this to get unnormalized results, will this predictions differ from noramalized version?
#preds_test_unnorm = predict(trained_model, test_features_tensor)

print('Logistic Regression Results:')
print('Accuracy:', accuracy(preds_test, test_labels))
print('F1-score', f1_score(preds_test, test_labels))

### DO NOT EDIT ### ### TEST SET RESULTS test_features_tensor, dev_labels_tensor = featurize_data(test_texts, test_labels) normalized_test_features_tensor = normalize(test_features_tensor) preds_test = predict(trained_model, normalized_test_features_tensor) # use this to get unnormalized results, will this predictions differ from noramalized version? #preds_test_unnorm = predict(trained_model, test_features_tensor) print('Logistic Regression Results:') print('Accuracy:', accuracy(preds_test, test_labels)) print('F1-score', f1_score(preds_test, test_labels))

Section 3: Multinomial Logistic Regression (Total: 30 points = 20 programming and 10 written reports)¶

According to the previous tutorial on using pytorch to train a logistic regression model for binary classification, now you will be given a new multiclass classification task with a new sentiment dataset (SST-5), which has 5 labels: very positive, positive, neutral, negative, very negative . Please implement a new pytorch model from scratch for Multinomial Logistic Regression. This code should be in the seperate python file sentiment_classifier.py. You are free to add any other utility files.

Ideally, this Task 3 should be decomposed in two subtasks:

1. Task 3.1 Exploratory Data Analysis to understand the new dataset (Please show some distribution analysis in the notebook by creating new cells below).
2. Task 3.2 Using the task 2.2 as an pytorch example, please build your own Multiclass classifier on this new dataset.

The expected submission for Task 3 has two parts:

1. (20', Programming) (1) (5') Completing the cells in this jupiter nootbook for data exploration. (2) (15') Your own sentiment_classifier.py from scratch.(Please zip the whole folder with a readme to setup your running. Please do remove the cache folders, such as pycache, .env .conda)
2. (10', Written Report) A single pdf for the findings in Task 3.1 and 3.2, with both your exploration for the dataset, and your experiments for the sentiment classification

Task 3.1 Exploratory Data Analysis on SST-5¶

In [7]:

# Load the dataset
from datasets import load_dataset

ds = load_dataset("SetFit/sst5")

# Load the dataset from datasets import load_dataset ds = load_dataset("SetFit/sst5")

/Users/jcao/mamba/envs/cs5293-2/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Repo card metadata block was not found. Setting CardData to empty.

In [8]:

train_dataset = ds['train']
dev_dataset = ds['validation']
test_dataset = ds['test']
train_dataset[0]

train_dataset = ds['train'] dev_dataset = ds['validation'] test_dataset = ds['test'] train_dataset[0]

Out[8]:

{'text': 'a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
 'label': 4,
 'label_text': 'very positive'}

In [ ]:

# The dataset is already split into train, dev, and test. So you don't need to split it again.
len(train_dataset), len(dev_dataset), len(test_dataset) 

# The dataset is already split into train, dev, and test. So you don't need to split it again. len(train_dataset), len(dev_dataset), len(test_dataset)

Out[ ]:

(8544, 1101, 2210)

In [ ]:

### ENTER CODE FOR EXPLORATORY HERE ###

### ENTER CODE FOR EXPLORATORY HERE ###

Task 3.2 Build Your MultiClass Sentiment Classifier From Scratch.¶

Your code should report the Accuracy, Precision, Recall, and F1 score for each label, and macro F1 for a combined score. (You don't need to reimplement all your metrics in Task 2.2. Please directly use classification_report to report the performance on dev set and test set. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html)

Training this may take around 30 minutes (depends on the features and machine you use) according your implementation. Hence we may not train your model from scratch to obtain a results. Please write a report to demonstrate your training, improvement, results and findings.

Hints:

1. You have to adapt the above binary classification code to support your "multiclass classifer", including dataset reading, using softmax function, cross-entropy loss, the features, and the model, and many details are not listed here. You may find existing libraries such as sklearn or keras for logistic regression, but please only use pytorch~(such as Task 2.2, but in a seperate python file) on this. Your code in this task 3 will also be improved in future assignments.
2. You need to explore your own features for this multinomial logistic regression (The performance might be bad. No worries. We will improve it later).