Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [1]:
%%capture
!pip install watermark
!pip install torchinfo datasets
In [2]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 torch: 2.10.0+cu128
Character-Level GRU (Char-GRU) Language Model on Tiny Shakespeare¶
The network in this notebook is an implementation of a Character-Level Gated Recurrent Unit (Char-GRU) network [2] trained on the Tiny Shakespeare dataset [7]. Moving from word-level classification to character-level generation is the "ultimate test" for a recurrent model. While a classifier identifies a label, a Language Model (LM) must learn the very fabric of language: spelling, punctuation, syntax, and even the "voice" of a specific author—one character at a time.
Network Architecture:
A Char-GRU is a Many-to-Many architecture. Unlike a classifier model that only cares about the final state, this model produces a prediction ($y_t$) at every single time step ($t$). For an input sequence of length $N$, the model makes $N$ predictions. The goal is to minimize the difference between the predicted next character and the actual next character in the sequence.
Many-to-Many RNN Structure. (Every input $x_t$ produces a corresponding output $y_t$ representing the probability of the next character.)
(Source)
The Internal Logic of the GRU Cell. (The same Gating mechanism used in classification is here tasked with "remembering" how to spell words.)
(Source)
Key Concepts of the Char-GRU Architecture:¶
- Sub-word Granularity: By operating at the character level, the model has a tiny vocabulary (usually ~65-100 characters). This makes the
Embeddinglayer extremely memory-efficient but forces the GRU to do the heavy lifting of learning word boundaries and semantics. - Stateful Inference: During text generation, the hidden state ($h_t$) is passed back into the model for the next character prediction. This "memory" allows the model to maintain context over long strings, such as ensuring a bracket
[eventually meets a closing]. - Temperature Scaling: A hyperparameter ($T$) used during generation to control "creativity." By scaling the output logits before the Softmax, we can force the model to be very conservative ($T < 0.5$) or highly experimental ($T > 1.0$).
Quick Comparison Table: Word-Level vs. Character-Level Modeling¶
| Feature | Word-Level (AG News) | Character-Level (Tiny Shakespeare) |
|---|---|---|
| Vocabulary Size | Large (10,000 - 50,000 words) | Tiny (~65 - 100 characters) |
| Sequence Length | Short (~58 tokens) | Long (100 - 500+ characters) |
| Task | Many-to-One (Classification) | Many-to-Many (Generation) |
| Output Layer | Linear(Hidden, Num_Classes) |
Linear(Hidden, Vocab_Size) |
| Difficulty | High-level Semantics | Low-level Syntax & Spelling |
| Inference Mode | Deterministic (Argmax) | Stochastic (Sampling with Temperature) |
| Memory Usage | High (Embedding Matrix) | Low (Embedding Matrix) |
REFERENCES:
- Karpathy, Andrej. "The Unreasonable Effectiveness of Recurrent Neural Networks." 2015. (The "Karpathy Special" Inspiration)
- Cho, Kyunghyun, et al. "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." arXiv preprint arXiv:1406.1078, 2014. (Original GRU Paper)
- Pascanu, Razvan, et al. "How to Construct Deep Recurrent Neural Networks." arXiv preprint arXiv:1312.6026, 2013. (Stacking & Depth)
- Chung, Junyoung, et al. "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling." arXiv preprint arXiv:1412.3555, 2014.
- Sutskever, Ilya, et al. "Sequence to Sequence Learning with Neural Networks." NeurIPS, 2014.
- Google. (2026). "Gemini 3 Flash" [Large language model]. Google AI
- Karpathy, Andrej. "Tiny Shakespeare Dataset." 2015.
Character-Level GRU (The Karpathy Special): Technical Reference¶
This document serves as a detailed reference for the Char-GRU implementation. Inspired by Andrej Karpathy's "The Unreasonable Effectiveness of Recurrent Neural Networks," this model shifts from word-level classification to character-level generation (Language Modeling).
Architecture Overview¶
Input (Character)
│
├── Embedding Layer (VOCAB_SIZE ≈ 65 → EMBED_DIM)
│ └── Maps unique characters (a, b, C, !, \n) to dense vectors.
│
├── Stacked GRU Layers (2-Layer Scratch Implementation)
│ ├── Layer 1: [Batch, Seq, Embed] → [Batch, Seq, Hidden]
│ └── Layer 2: [Batch, Seq, Hidden] → [Batch, Seq, Hidden]
│ └── Captures hierarchical patterns (Spelling → Grammar → Tone).
│
├── Linear Header (Hidden → VOCAB_SIZE)
│ └── Outputs "logits" for every single character in the sequence.
│
└── Softmax / Sampling
└── Converts logits to probabilities for the next character.
Data Pipeline: Shifting Windows¶
Unlike classification, where we have a static label, the target for a Language Model is the input sequence shifted by one position.
- Raw Text:
“To be, or not to be” - Input ($X$):
“To be, or not to b” - Target ($Y$):
“o be, or not to be”
The get_batch Logic¶
To ensure the model is robust, we sample random chunks from the training text (e.g., Tiny Shakespeare) rather than reading linearly from the start every time. This prevents the model from overfitting on the specific order of the training file.
The Generation Engine (Inference)¶
Generating text requires a "Stateful" loop. We don't just pass a string; we pass a hidden state that carries the memory of the previous characters.
The Sampling Process¶
- Seed: Start with a string (e.g.,
"ROMEO:"). - Forward Pass: Pass the seed through the GRU to get the initial hidden state $h$.
- Loop:
- Input the last predicted character into the model.
- Apply Temperature to the resulting logits.
- Sample the next character using
torch.multinomial. - Update the hidden state $h$ for the next iteration.
Temperature Control¶
Temperature ($T$) is a hyperparameter used to scale the logits before the Softmax. It controls the "creativity" of the model.
| Temperature | Effect | Output Quality |
|---|---|---|
| 0.2 (Low) | Confident / Sharp | Very safe, repetitive, perfect spelling. |
| 0.7 (Sweet Spot) | Balanced | Readable Shakespearean style with some flair. |
| 1.2 (High) | Chaotic / Creative | Inventive words, erratic punctuation, risky. |
| 5.0 (Extreme) | Random | Total gibberish; nearly uniform distribution. |
Training Specifics¶
Loss Function¶
We use nn.CrossEntropyLoss. Because our model outputs a prediction for every character in the sequence, we must flatten the tensors before calculating loss:
- Logits:
[Batch * Seq_Len, Vocab_Size] - Targets:
[Batch * Seq_Len]
Initial Loss "Robustness" Check¶
A mathematically sound Char-RNN should start with a loss approximately equal to the negative log of the vocabulary size:
$$-\ln\left(\frac{1}{V}\right) = \ln(V)$$
For Tiny Shakespeare ($V \approx 65$), the starting loss should be $\approx 4.17$. If it is significantly higher or lower, there is likely an initialization error in the GRU weights.
Performance Metrics¶
| Metric | Goal (Shakespeare) | Meaning |
|---|---|---|
| Training Loss | < 1.50 | Model has learned basic English spelling and names. |
| Validation Loss | < 1.70 | Model is generalizing to unseen dialogue patterns. |
| Perplexity | Lower is better | Measures how "surprised" the model is by the next char. |
Known Quirks & Guards¶
- Exploding Gradients: Because the sequences are long (100+ chars), gradients can grow exponentially. Guard:
nn.utils.clip_grad_norm_(max_norm=1.0). - The "The" Plateau: The model will spend the first few hundred iterations just learning that "t" is followed by "h" and then "e". This is normal.
- State Detachment: When training, we often "detach" the hidden state between batches to prevent BPTT (Backpropagation Through Time) from going too deep into the past, which would crash the VRAM.
Troubleshooting & Common Failure Modes¶
Even a robustly built Char-GRU can fall into specific traps. If your model starts generating gibberish or getting stuck, check these three common scenarios:
| Issue | Symptom | Likely Cause |
|---|---|---|
| The Loop of Death | Output is eeeeeeee or |
High Learning Rate or No Gradient Clipping. The model's weights "exploded," forcing all probabilities toward one character. |
| The Spelling Bee | Words are spelled correctly but make no sense | Hidden Dimension too small. The model learned local patterns (spelling) but lacks the "memory" to hold a sentence together. |
| Cold Start | Loss doesn't drop for 500+ iterations | Vanishing Gradients. Ensure you are using tanh for your candidate hidden state and sigmoid for your gates. |
Understanding Sequence Loss: The Flattening Trick¶
When training a Char-GRU, the model doesn't just produce one prediction at the end of the sentence; it produces a prediction for every single character. This creates a dimensional mismatch between your model's output and the standard PyTorch CrossEntropyLoss function.
1. The Dimension Problem¶
Model Output (Logits):
[Batch Size, Sequence Length, Vocab Size]Example:
[32, 100, 65]— 32 sentences, each 100 characters long, with 65 possible character scores for each.Targets (Actual Characters):
[Batch Size, Sequence Length]Example:
[32, 100]— The correct character index for every position in those 32 sentences.
2. The Solution: Flattening to 2D¶
CrossEntropyLoss expects the input to be a 2D matrix of shape [Total Samples, Classes]. To bridge this gap, we "squash" the Batch and Sequence dimensions together.
# During the training loop:
logits, h = model(inputs, h)
# Reshape from [32, 100, 65] -> [3200, 65]
# Reshape targets from [32, 100] -> [3200]
loss = criterion(logits.view(-1, VOCAB_SIZE), targets.view(-1))
3. Why this is "Robust"¶
By flattening the tensors, you are essentially telling the loss function to treat every character position as an independent classification problem. The GRU's internal hidden state ensures they aren't actually independent, but the math becomes much cleaner for the optimizer to handle.
Visualizing the Loss Landscape¶
In a Char-GRU, you will notice the loss follows a very specific "staircase" pattern:
- Initial Plateau (Loss $\approx 4.17$): The model is guessing randomly.
- The Space Drop: The loss drops sharply once the model realizes that spaces occur every 4-5 characters.
- The "The" Slope: A steady decline as the model learns common English trigrams (the, ing, and).
- The Structural Plateau: The final struggle to learn complex grammar and Shakespearean character names.
In [3]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from sklearn.model_selection import train_test_split
from torch.utils.data import DataLoader, Dataset
from torch.nn.utils.rnn import pad_sequence
from datasets import load_dataset, DatasetDict
from collections import Counter
# from torch.cuda.amp import GradScaler, autocast
from torch.amp import GradScaler, autocast
from torchinfo import summary
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
import re
import requests
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
In [4]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
In [5]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 123
BATCH_SIZE = 256
LEARNING_RATE = 0.002
NUM_EPOCHS = 10
SEQ_LEN = 100 # How many characters the model looks at to predict the next one
EMBEDDING_DIM = 64
HIDDEN_DIM = 512
# Set device
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(DEVICE)
cuda
In [6]:
# 1. Load Tiny Shakespeare
url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
text = requests.get(url).text
# 2. Build Character-Level Vocabulary
chars = sorted(list(set(text)))
vocab = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)} # integer-to-string for generation
VOCAB_SIZE = len(chars)
# 3. Custom Dataset for Language Modeling (Shifting Window)
class CharDataset(Dataset):
def __init__(self, raw_text, seq_len):
self.seq_len = seq_len
# Numericalize the entire corpus once
self.data = torch.tensor([vocab[c] for c in raw_text], dtype=torch.long)
def __len__(self):
return len(self.data) - self.seq_len
def __getitem__(self, idx):
# Input: "The quick brow"
# Target: "he quick brown"
x = self.data[idx : idx + self.seq_len]
y = self.data[idx + 1 : idx + self.seq_len + 1]
return x, y
# 4. Initialize DataLoaders
# Split text (90% train, 10% val)
n = int(0.9 * len(text))
train_ds = CharDataset(text[:n], SEQ_LEN)
val_ds = CharDataset(text[n:], SEQ_LEN)
# No special collate_fn needed since all seqs are SEQ_LEN
train_loader = DataLoader(train_ds, batch_size=BATCH_SIZE, shuffle=True)
val_loader = DataLoader(val_ds, batch_size=BATCH_SIZE, shuffle=False)
In [7]:
class GRUCellFromScratch(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
# We need weights for 3 operations:
# Update gate (z), Reset gate (r), and Candidate hidden state (h_tilde)
self.gate_weights = nn.Linear(input_size + hidden_size, 2 * hidden_size)
self.candidate_weights = nn.Linear(input_size + hidden_size, hidden_size)
def forward(self, x, h_prev):
# x: [batch, input_size]
# h_prev: [batch, hidden_size]
# 1. Calculate Update (z) and Reset (r) gates
combined = torch.cat((x, h_prev), dim=1)
gates = torch.sigmoid(self.gate_weights(combined))
z_gate, r_gate = gates.chunk(2, dim=1)
# 2. Calculate Candidate Hidden State (h_tilde)
# We use the Reset gate to "mask" the previous hidden state
combined_candidate = torch.cat((x, r_gate * h_prev), dim=1)
h_tilde = torch.tanh(self.candidate_weights(combined_candidate))
# 3. Final Hidden State update
# This is a linear interpolation between past and candidate
h_curr = (1 - z_gate) * h_prev + z_gate * h_tilde
return h_curr
In [8]:
class CharGRUFromScratch(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(vocab_size, embed_dim)
# Using your scratch cell
self.gru_cell = GRUCellFromScratch(embed_dim, hidden_size)
# Output is VOCAB_SIZE (prob of each char)
self.fc = nn.Linear(hidden_size, vocab_size)
def forward(self, x, h=None):
batch_size, seq_len = x.shape
x = self.embedding(x) # [Batch, Seq, Embed]
if h is None:
h = torch.zeros(batch_size, self.hidden_size).to(x.device)
outputs = []
for t in range(seq_len):
h = self.gru_cell(x[:, t, :], h)
# Predict character at this time step
out = self.fc(h)
outputs.append(out.unsqueeze(1))
# Return [Batch, Seq, Vocab] and the final hidden state (for generation)
return torch.cat(outputs, dim=1), h
In [9]:
set_all_seeds(RANDOM_SEED)
model = CharGRUFromScratch(VOCAB_SIZE, EMBEDDING_DIM, HIDDEN_DIM).to(DEVICE)
# In Char-GRU, we use SEQ_LEN instead of MAX_LEN
summary(model,
# Note: CharGRU forward only takes (x, h), not (x, lengths).
# But to keep your summary call robust, we just pass x:
input_data=torch.zeros((BATCH_SIZE, SEQ_LEN), dtype=torch.long).to(DEVICE),
col_names=["input_size", "output_size", "num_params", "mult_adds"],
device=DEVICE)
Out[9]:
============================================================================================================================================ Layer (type:depth-idx) Input Shape Output Shape Param # Mult-Adds ============================================================================================================================================ CharGRUFromScratch [256, 100] [256, 100, 65] -- -- ├─Embedding: 1-1 [256, 100] [256, 100, 64] 4,160 1,064,960 ├─GRUCellFromScratch: 1-2 [256, 64] [256, 512] -- -- │ └─Linear: 2-1 [256, 576] [256, 1024] 590,848 151,257,088 │ └─Linear: 2-2 [256, 576] [256, 512] 295,424 75,628,544 ├─Linear: 1-3 [256, 512] [256, 65] 33,345 8,536,320 ├─GRUCellFromScratch: 1-4 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-3 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-4 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-5 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-6 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-5 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-6 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-7 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-8 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-7 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-8 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-9 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-10 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-9 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-10 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-11 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-12 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-11 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-12 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-13 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-14 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-13 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-14 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-15 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-16 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-15 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-16 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-17 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-18 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-17 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-18 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-19 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-20 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-19 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-20 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-21 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-22 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-21 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-22 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-23 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-24 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-23 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-24 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-25 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-26 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-25 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-26 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-27 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-28 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-27 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-28 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-29 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-30 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-29 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-30 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-31 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-32 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-31 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-32 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-33 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-34 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-33 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-34 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-35 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-36 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-35 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-36 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-37 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-38 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-37 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-38 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-39 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-40 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-39 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-40 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-41 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-42 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-41 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-42 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-43 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-44 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-43 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-44 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-45 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-46 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-45 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-46 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-47 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-48 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-47 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-48 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-49 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-50 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-49 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-50 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-51 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-52 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-51 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-52 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-53 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-54 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-53 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-54 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-55 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-56 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-55 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-56 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-57 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-58 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-57 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-58 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-59 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-60 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-59 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-60 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-61 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-62 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-61 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-62 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-63 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-64 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-63 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-64 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-65 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-66 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-65 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-66 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-67 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-68 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-67 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-68 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-69 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-70 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-69 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-70 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-71 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-72 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-71 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-72 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-73 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-74 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-73 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-74 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-75 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-76 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-75 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-76 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-77 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-78 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-77 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-78 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-79 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-80 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-79 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-80 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-81 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-82 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-81 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-82 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-83 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-84 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-83 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-84 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-85 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-86 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-85 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-86 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-87 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-88 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-87 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-88 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-89 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-90 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-89 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-90 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-91 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-92 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-91 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-92 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-93 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-94 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-93 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-94 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-95 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-96 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-95 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-96 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-97 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-98 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-97 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-98 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-99 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-100 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-99 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-100 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-101 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-102 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-101 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-102 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-103 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-104 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-103 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-104 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-105 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-106 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-105 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-106 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-107 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-108 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-107 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-108 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-109 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-110 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-109 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-110 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-111 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-112 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-111 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-112 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-113 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-114 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-113 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-114 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-115 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-116 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-115 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-116 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-117 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-118 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-117 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-118 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-119 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-120 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-119 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-120 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-121 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-122 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-121 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-122 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-123 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-124 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-123 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-124 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-125 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-126 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-125 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-126 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-127 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-128 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-127 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-128 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-129 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-130 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-129 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-130 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-131 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-132 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-131 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-132 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-133 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-134 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-133 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-134 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-135 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-136 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-135 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-136 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-137 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-138 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-137 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-138 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-139 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-140 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-139 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-140 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-141 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-142 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-141 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-142 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-143 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-144 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-143 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-144 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-145 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-146 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-145 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-146 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-147 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-148 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-147 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-148 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-149 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-150 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-149 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-150 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-151 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-152 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-151 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-152 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-153 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-154 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-153 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-154 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-155 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-156 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-155 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-156 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-157 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-158 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-157 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-158 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-159 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-160 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-159 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-160 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-161 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-162 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-161 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-162 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-163 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-164 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-163 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-164 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-165 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-166 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-165 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-166 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-167 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-168 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-167 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-168 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-169 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-170 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-169 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-170 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-171 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-172 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-171 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-172 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-173 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-174 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-173 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-174 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-175 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-176 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-175 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-176 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-177 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-178 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-177 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-178 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-179 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-180 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-179 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-180 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-181 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-182 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-181 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-182 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-183 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-184 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-183 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-184 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-185 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-186 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-185 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-186 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-187 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-188 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-187 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-188 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-189 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-190 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-189 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-190 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-191 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-192 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-191 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-192 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-193 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-194 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-193 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-194 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-195 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-196 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-195 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-196 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-197 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-198 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-197 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-198 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-199 [256, 512] [256, 65] (recursive) 8,536,320 ├─GRUCellFromScratch: 1-200 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-199 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-200 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-201 [256, 512] [256, 65] (recursive) 8,536,320 ============================================================================================================================================ Total params: 923,777 Trainable params: 923,777 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 23.54 ============================================================================================================================================ Input size (MB): 0.20 Forward/backward pass size (MB): 340.99 Params size (MB): 3.70 Estimated Total Size (MB): 344.89 ============================================================================================================================================
In [10]:
def generate_text(model, seed="JULIET: ", length=50, temperature=0.8):
"""
Generates text by feeding the model a seed and sampling the next character.
"""
model.eval()
with torch.no_grad():
# 1. Convert seed string to integers
chars_idx = [vocab[c] for c in seed]
input_tensor = torch.tensor(chars_idx).unsqueeze(0).to(DEVICE) # [1, seed_len]
# 2. Feed the seed into the model to get the 'context' hidden state
# We don't care about the logits from the seed, just the final hidden state h
_, h = model(input_tensor)
# 3. Start generating from the very last character of the seed
current_char_tensor = input_tensor[:, -1:] # [1, 1]
generated_output = seed
for _ in range(length):
# Forward pass with the single current character and the previous hidden state
logits, h = model(current_char_tensor, h)
# 4. Apply Temperature Scaling
# Higher temp = more random/creative; Lower temp = more predictable
logits = logits.squeeze() / temperature
# 5. Sample from the distribution (instead of just taking the max)
probs = torch.softmax(logits, dim=-1)
next_char_idx = torch.multinomial(probs, num_samples=1).item()
# 6. Append to result and update current_char_tensor for next loop
generated_output += itos[next_char_idx]
current_char_tensor = torch.tensor([[next_char_idx]]).to(DEVICE)
return generated_output
In [11]:
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
In [12]:
def validate_perplexity(model, loader, device):
model.eval()
total_loss = 0
with torch.no_grad():
for inputs, targets in loader:
inputs, targets = inputs.to(device), targets.to(device)
logits, _ = model(inputs)
# Use CrossEntropy exactly like training
loss = nn.functional.cross_entropy(
logits.view(-1, VOCAB_SIZE),
targets.view(-1)
)
total_loss += loss.item()
avg_loss = total_loss / len(loader)
# PERPLEXITY CALCULATION
perplexity = torch.exp(torch.tensor(avg_loss)).item()
return avg_loss, perplexity
# --- Inside your main loop after the epoch finishes ---
val_loss, val_ppl = validate_perplexity(model, val_loader, DEVICE)
print(f"Validation Loss: {val_loss:.4f} | Perplexity: {val_ppl:.2f}")
Validation Loss: 4.1869 | Perplexity: 65.82
In [13]:
start_time = time.time()
print_interval = max(1, len(train_loader) // 5) # 5 prints per epoch
history = {
'train_loss': [],
'val_loss': [],
'val_ppl': [],
'lrs': []
}
for epoch in range(NUM_EPOCHS):
model.train()
total_loss = 0
running_batch_loss = 0
for i, (inputs, targets) in enumerate(train_loader):
inputs, targets = inputs.to(DEVICE), targets.to(DEVICE)
logits, _ = model(inputs)
loss = criterion(logits.view(-1, VOCAB_SIZE), targets.view(-1))
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
total_loss += loss.item()
running_batch_loss += loss.item()
# --- BATCH LOGGING (Keeps you sane while waiting) ---
if (i + 1) % print_interval == 0:
avg_batch_loss = running_batch_loss / print_interval
print(f"Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | "
f"Batch {i+1:04d}/{len(train_loader):04d} | "
f"Loss: {avg_batch_loss:.4f}")
running_batch_loss = 0
# --- EPOCH END: LOGGING & VALIDATION ---
# Move metrics out of the batch loop to save only once per epoch
avg_train_loss = total_loss / len(train_loader)
# Run validation to get fresh metrics for history
v_loss, v_ppl = validate_perplexity(model, val_loader, DEVICE)
history['train_loss'].append(avg_train_loss)
history['val_loss'].append(v_loss)
history['val_ppl'].append(v_ppl)
history['lrs'].append(optimizer.param_groups[0]['lr'])
# Optional: scheduler.step() if you have one
elapsed = (time.time() - start_time) / 60
print(f"\n{'-'*30}")
print(f"EPOCH {epoch+1} COMPLETE | Time: {elapsed:.2f} min")
print(f"Train Loss: {avg_train_loss:.4f} | Val Loss: {v_loss:.4f} | Val PPL: {v_ppl:.2f}")
print(f"{'-'*30}")
# GENERATION CHECK
# Juliet usually has something to say!
print(f"GENERATED TEXT:\n{generate_text(model, seed='JULIET: ', length=1000)}")
print(f"{'='*70}\n")
Epoch: 001/010 | Batch 0784/3921 | Loss: 1.4598 Epoch: 001/010 | Batch 1568/3921 | Loss: 1.0449 Epoch: 001/010 | Batch 2352/3921 | Loss: 0.9046 Epoch: 001/010 | Batch 3136/3921 | Loss: 0.8373 Epoch: 001/010 | Batch 3920/3921 | Loss: 0.7920 ------------------------------ EPOCH 1 COMPLETE | Time: 7.38 min Train Loss: 1.0077 | Val Loss: 2.0460 | Val PPL: 7.74 ------------------------------ GENERATED TEXT: JULIET: your alliant, In you more bare-falcon would be consul. QUEEN ELIZABETH: Then let him swallow goes, let us sup of the duke's or boxtor. First Citizen: Let it be quainty. If that be behind a man; for more than a promise to himself are incensed But to command, as I do see this bed When it serves me again down to flatter; No bigg'd, and to be loath to him that keeps it cannot hit my name but death. Boy: Good aunt, sir. Lord: Here comes the open air, as I said with me: And now, Camillo, till I be king, or death! QUEEN ELIZABETH: For what thou lack of you of such a person? speak. Yes, and I cannot tell: the slave of her Is that thou hast, deaf it is the morning, Is this our cockers here thou sakest, Is fair and advertore every bosom of the moon As doth such songing to thy head in her In madge the people; not before I came. KING HENRY VI: A clock mean to woe, I warrant it, if He could not meet our clothings shall attend you at the lord. HASTINGS: Who, my lord, I'll forbit out thy husba ====================================================================== Epoch: 002/010 | Batch 0784/3921 | Loss: 0.7552 Epoch: 002/010 | Batch 1568/3921 | Loss: 0.7313 Epoch: 002/010 | Batch 2352/3921 | Loss: 0.7111 Epoch: 002/010 | Batch 3136/3921 | Loss: 0.6952 Epoch: 002/010 | Batch 3920/3921 | Loss: 0.6818 ------------------------------ EPOCH 2 COMPLETE | Time: 14.83 min Train Loss: 0.7149 | Val Loss: 2.3097 | Val PPL: 10.07 ------------------------------ GENERATED TEXT: JULIET: your grace sovereign music False owe another, sir, a virgin too. Tranio, If love the face to save the time, Which for thing woman, you might have brought To think so letters, thou art foul a man. Ray't my memory midnight: Come, come; come, you shall go with us unto the queen! QUEEN ELIZABETH: There's no matter, as he is a thing me stay upon the ground? Not at Somerpet: Faith, as he were dishonour doth he speaks, Spare not: God is mild, Lord: He sits more fair best. SLY: Yes; I am resolved. First Murderer: How can you gone to-morrow? TYRREL: He that is thy crown, and give the world 'foreh: Too lately this paty from Lord Aumerle, I mean the senate-house: he hath been by the shearnesty I should not for my poor upon that word; Your praises are to be answer'd. MERCUTIO: No less prepared for a thing impecited her better dotingle, my lord, to Aloom'd son, Or I am such a case again: black-bod! Page: Ghost of BUCYRIOREL: Good night, as I sent for, to the sea suffice As being Coriolanus, ====================================================================== Epoch: 003/010 | Batch 0784/3921 | Loss: 0.6680 Epoch: 003/010 | Batch 1568/3921 | Loss: 0.6580 Epoch: 003/010 | Batch 2352/3921 | Loss: 0.6505 Epoch: 003/010 | Batch 3136/3921 | Loss: 0.6423 Epoch: 003/010 | Batch 3920/3921 | Loss: 0.6361 ------------------------------ EPOCH 3 COMPLETE | Time: 22.36 min Train Loss: 0.6510 | Val Loss: 2.4466 | Val PPL: 11.55 ------------------------------ GENERATED TEXT: JULIET: your good upon him, sir, he has, His bad is near, and out the other arm readving. See, I have sat hour in his bird! Oh, he is merey, soon but despair. A thousand-favour! Nurse: Madam, worthy sirrator, here's far That Clifford's prison, come soil Will well become and crowns to hear I do: And given in great mother, for your grace's honour, I hold it not, or then; I warrant him to her daughter. GREGORY: But some unlove to see, by temorance he tooks, Of this several men, are clouded so. DUKE VINCENTIO: Why withwalk your honours use to death? Then let the law that almost forget My brother, and nothing fear'd and publing-trick; And no, or comfort her: what after the fault Against the Duke of Buckingham. MAMILLO: I say, I pray thee, you can do't: Be in my tent: the which is mine enemy? Fear, ay, his army is now not much confederate it. DUKE OF YORK: Be much eyes do all other state, To thee the forfeit of the mind of our enmity! PRINCE: A murdering, bear witness good, and my love's cheap ====================================================================== Epoch: 004/010 | Batch 0784/3921 | Loss: 0.6270 Epoch: 004/010 | Batch 1568/3921 | Loss: 0.6235 Epoch: 004/010 | Batch 2352/3921 | Loss: 0.6182 Epoch: 004/010 | Batch 3136/3921 | Loss: 0.6134 Epoch: 004/010 | Batch 3920/3921 | Loss: 0.6092 ------------------------------ EPOCH 4 COMPLETE | Time: 29.80 min Train Loss: 0.6182 | Val Loss: 2.5542 | Val PPL: 12.86 ------------------------------ GENERATED TEXT: JULIET: your grace comes. And many a meats I show'd it. Take thou again. KING RICHARD II: I had forgot myself that do prove the king; Therefore think it still makes thee any of his eyes, Rather than marry but malifests Shall Romeo bear them from himself to see. JOHN OF GAUNT: O, no! thou diesers, with that true Resseng! AUFIDIUS: I know your fresh words were not grace. I have it yet see where now fares this land! Now, Marsapp'd in Tybalt and England's queen; And, Rivers, so with me it were a great deal of wrath, That gives not Mercutio, who hate here. KING RICHARD III: You speak saddle and more come; I must return The slaved of marriage him out a suit, As it was better than a villain and the gentle Warwick. You are thus eyes of your years old, of my brother; Our dimples of his grave puts forth his his ease:' here I could sings ere they love Hastings with his body that hath left undone and heart To draw out a teamous love. I beseech you, Give me confessed art thyself, Which both our hostes ====================================================================== Epoch: 005/010 | Batch 0784/3921 | Loss: 0.6029 Epoch: 005/010 | Batch 1568/3921 | Loss: 0.6012 Epoch: 005/010 | Batch 2352/3921 | Loss: 0.5979 Epoch: 005/010 | Batch 3136/3921 | Loss: 0.5943 Epoch: 005/010 | Batch 3920/3921 | Loss: 0.5917 ------------------------------ EPOCH 5 COMPLETE | Time: 37.16 min Train Loss: 0.5976 | Val Loss: 2.6122 | Val PPL: 13.63 ------------------------------ GENERATED TEXT: JULIET: your grant, your lawful king: Fear not, my lord: God poor brother! Rather, that Edward shall be leisure your dissolute With honours must have acceptrut to my nurs, And cry 'Tis brothers gives Lord Stanley--banishment Than imagine powers the more. Clown: Alas, poor manarrio, and as we have passing Than been preciously he that takest thou froudded my virtue! Juggld me, nor now the reason Froth here, To supply this ring. My better hated than in the end, excelling marriage with him; And in the prime of kingment here, Till the supplianteranchor look torn,-- And what shall we too, of all the case, The helm of grants as we once disdain against it give out a better horn: that's the way. ANTIGONUS: I take my leave. BUCKINGHAM: My lord, let's speak; methinks, you'll fight when bails on many That stabb'd his valour cased, but still should go. LADY ER: How should they stay without a kind of fault, Nor need you know. KING RICHARD III: So proud that dogs what they dance? FLORIZEL: I am strange ====================================================================== Epoch: 006/010 | Batch 0784/3921 | Loss: 0.5866 Epoch: 006/010 | Batch 1568/3921 | Loss: 0.5847 Epoch: 006/010 | Batch 2352/3921 | Loss: 0.5823 Epoch: 006/010 | Batch 3136/3921 | Loss: 0.5802 Epoch: 006/010 | Batch 3920/3921 | Loss: 0.5780 ------------------------------ EPOCH 6 COMPLETE | Time: 44.56 min Train Loss: 0.5824 | Val Loss: 2.6641 | Val PPL: 14.36 ------------------------------ GENERATED TEXT: JULIET: your grace, when Romeo? I shall entreat you to it. CLPULITES: Ha! what shall we do? CLARENCE: GLOUCESTER: KING EDWARD IV: Madam, yourself are never tribunes forgettles thee: But death hath strike not, there thy kinsmen field. The secrets of Hereford, all's i' the princess With sometimell'd streets, or destes our wag, Since the most you are, not so much at this face Between the street child. I do be brief,--if ever henceforth thou truth, I'll keep in foreign sleep, and find the worst. COMINIUS: I'll find out right and honourable. KING EDWARD IV: Clarence and consul, not her hence: What my divorce is sliped by Simonts fellow. To weeping possible, you have made his account, And to be glad of Buckingham, and extremity cannot Relable men, as an eye with thee the king! LEONTES: Ha! AUFIDIUS: You have a soldier as the woefullest discourse, Which sixteen mine eyes did hatch'd like sovereign. KING HENRY VI: Had a thousand stooping executed from all which lately king, And more than me, ====================================================================== Epoch: 007/010 | Batch 0784/3921 | Loss: 0.5738 Epoch: 007/010 | Batch 1568/3921 | Loss: 0.5732 Epoch: 007/010 | Batch 2352/3921 | Loss: 0.5705 Epoch: 007/010 | Batch 3136/3921 | Loss: 0.5698 Epoch: 007/010 | Batch 3920/3921 | Loss: 0.5677 ------------------------------ EPOCH 7 COMPLETE | Time: 51.82 min Train Loss: 0.5710 | Val Loss: 2.7035 | Val PPL: 14.93 ------------------------------ GENERATED TEXT: JULIET: your guilt name, and Montague, Who loved the children of your tender pains Become to hear this bastard's life. Amen, amen. On pain on my part from lose; Let's wrong and splinter'd even no other, hark you are some house, or say that she was beguiled, O'er-ty drops of sulmen, the very deadly grandfil Whereto your name? and what stars i' the royal business to welcome them, For mother than the mid blood doth with them As the deep designs of death in mine honour's grave And would usurp the present peace o' Thumb, So three our father of the fleeting to a fresher! On my sight! I'll to my red us now, my lord! ESCALULIO: Urge it betray'd the matter with our subject? CATESBY: I do the like. First Citizen: The beggary turn your counsel: to thy own! Sirron, so should to Pomfret, not my care; I'll leave you in thy knee. CORIOLANUS: Away! Second Citizen: But reason with the rock Tranio, well see you To give you judge. KING EDWARD IV: Where is thy command, the hour aid once, Young Romano, shall ====================================================================== Epoch: 008/010 | Batch 0784/3921 | Loss: 0.5639 Epoch: 008/010 | Batch 1568/3921 | Loss: 0.5632 Epoch: 008/010 | Batch 2352/3921 | Loss: 0.5624 Epoch: 008/010 | Batch 3136/3921 | Loss: 0.5600 Epoch: 008/010 | Batch 3920/3921 | Loss: 0.5591 ------------------------------ EPOCH 8 COMPLETE | Time: 59.11 min Train Loss: 0.5617 | Val Loss: 2.7377 | Val PPL: 15.45 ------------------------------ GENERATED TEXT: JULIET: your slanders of the king, of succession. You shall not see, I drink we dost make me swory; You said they were banished! But if thou love The broke of gentle come in sour holent. LADY ANNE: A horse! ah, were I thrown my woes fair-a fool. GLOUCESTER: But when you say that he hath wedded friend, And left thee but a man that did not move: Sirrah, no, not a man that you do sequest. SOMERSET: So smile the devil should lish years: Say 'Be; who, on my state that come but present, So true tongues by stomp to the rock again. ROMEO: I think so, not all; and, these words can your bisson must In hopes by nature's zeal of noble mother; For your suit of England; for willingnamen's arms, Because as in his title enough of my love. Dost thou we but sin as nigning warms not within world! For though what duke? or are his love? ROMEO: Thou wilt stay to hold. MENENIUS: The king enough to-morrow? VOLUMNIA: I thank you. Fare you well. Go, sign of warriages, To tires that have banish'd. Now to Bolingbr ====================================================================== Epoch: 009/010 | Batch 0784/3921 | Loss: 0.5561 Epoch: 009/010 | Batch 1568/3921 | Loss: 0.5559 Epoch: 009/010 | Batch 2352/3921 | Loss: 0.5539 Epoch: 009/010 | Batch 3136/3921 | Loss: 0.5532 Epoch: 009/010 | Batch 3920/3921 | Loss: 0.5521 ------------------------------ EPOCH 9 COMPLETE | Time: 66.60 min Train Loss: 0.5542 | Val Loss: 2.7575 | Val PPL: 15.76 ------------------------------ GENERATED TEXT: JULIET: you and honour, Incapate as lief as you did. DUKE VINCENTIO: The queen gay short of your opinion. First Murderer: 'Fair'dst thou, whose own part of pretty Montague? Second Murderer: Tybalt, her brothers too, glose Edward's champion Of these hands I give him gave us our secret honour, Or, ripe of his kindness. Yet, masters, take it to the princess, Be even in pure word, thou shalt not do't. ISABELLA: O, let him marry her, you greeting of On the virtue of your delight, But you must right me thus, Howe'er I do rother. DUKE VINCENTIO: By heaven, fond wrath, we hear, grow to us, you may stay. KING EDWARD IV: Ay, but thou comest in't; who, which been prevented To nothing terms to right our father saw. BENVOLIO: Romeo! my cousin Romeo! MENENIUS: The sentence moved to thee and holds up what nature? ISABELLA: I am a king's gloriage of a man mad. Nurse: I am a-weary, give me leave awhile, Yet as the gates fear paper and old! Citizens: Deservice hence, his sake, life, with one half and ====================================================================== Epoch: 010/010 | Batch 0784/3921 | Loss: 0.5492 Epoch: 010/010 | Batch 1568/3921 | Loss: 0.5488 Epoch: 010/010 | Batch 2352/3921 | Loss: 0.5483 Epoch: 010/010 | Batch 3136/3921 | Loss: 0.5467 Epoch: 010/010 | Batch 3920/3921 | Loss: 0.5457 ------------------------------ EPOCH 10 COMPLETE | Time: 74.12 min Train Loss: 0.5477 | Val Loss: 2.7940 | Val PPL: 16.35 ------------------------------ GENERATED TEXT: JULIET: your good atoney? ROMEO: O, help me, help to make his new speak. BUCKINGHAM: If is the city must Caius Marcius hath been The heavens is not so. DUCHESS OF YORK: Clinsing the sweet bies with silver all. Go in safety fitten to death. Your honesty Is right wellows there, his prosperity, And let them do breathe mine heirs: ha! That Edward is a cover: What are they? Huntsman! a pair of mine. Cousin of Exeter, what says Romeo? ROMEO: At any cold, and fested at all; For either presently your father there? BALTHASAR: Roie. Second Citizen: And so, my good lord mayor. My matter Lood Clarence, bictory is desperate, with it stands To take on that present country are requited But with such lawful king: O, homach your exchances Wherein this to them, whose usual suffer'd me And wantonted happiness to them! And, black, destin; come there be new scat disk'd Hath he disciss not till now, an instead off you as he willingly. CAMILLO: Evanors as I leave her; for I can speak Eget will I mean to my me ======================================================================
In [15]:
def plot_training_results(history):
epochs = range(1, len(history['train_loss']) + 1)
plt.figure(figsize=(15, 5))
# Plot 1: Cross-Entropy Loss
plt.subplot(1, 3, 1)
plt.plot(epochs, history['train_loss'], 'o-', label='Train Loss')
plt.plot(epochs, history['val_loss'], 's-', label='Val Loss')
plt.title('Training & Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.grid(True, alpha=0.3)
# Plot 2: Perplexity (The "Intelligence" Metric)
plt.subplot(1, 3, 2)
plt.plot(epochs, history['val_ppl'], 'g^-', label='Val Perplexity')
plt.title('Model Perplexity')
plt.xlabel('Epochs')
plt.ylabel('Perplexity')
plt.legend()
plt.grid(True, alpha=0.3)
# Plot 3: Learning Rate Decay
plt.subplot(1, 3, 3)
plt.plot(epochs, history['lrs'], 'orange', label='LR')
plt.title('Learning Rate Schedule')
plt.xlabel('Epochs')
plt.ylabel('Learning Rate')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# plt.savefig()
plot_training_results(history)
In [20]:
def evaluate_final_model(model, loader, device, vocab_size):
model.eval()
total_loss = 0
print(f"Calculating final metrics on {len(loader)} validation batches...")
with torch.no_grad():
for inputs, targets in loader:
inputs, targets = inputs.to(device), targets.to(device)
logits, _ = model(inputs)
# Use the passed-in vocab_size
loss = nn.functional.cross_entropy(
logits.view(-1, vocab_size),
targets.view(-1)
)
total_loss += loss.item()
avg_val_loss = total_loss / len(loader)
perplexity = torch.exp(torch.tensor(avg_val_loss)).item()
print("\n" + "="*40)
print(f"{'FINAL EVALUATION REPORT':^40}")
print("="*40)
print(f"Validation Loss: {avg_val_loss:.4f}")
print(f"Perplexity: {perplexity:.2f}")
print("="*40)
# 500 characters is the "Goldilocks" length for checking consistency
print("\nLONG GENERATION (Temp=0.7):")
print("-" * 40)
print(generate_text(model, seed="KING RICHARD: ", length=500, temperature=0.7))
print("-" * 40)
In [21]:
import requests
# 1. Fetch the Sonnets (Distinct from the 'Plays' in Tiny Shakespeare)
sonnets_url = "https://www.gutenberg.org/cache/epub/1041/pg1041.txt"
sonnets_text = requests.get(sonnets_url).text
# 2. Basic cleaning (Project Gutenberg adds headers/footers)
# We look for the actual start and end of the poems
start_idx = sonnets_text.find("I")
end_idx = sonnets_text.rfind("FINIS")
sonnets_clean = sonnets_text[start_idx:end_idx]
# 3. CRITICAL: Filter for characters your model actually knows
# Your model will crash if it sees a character it wasn't trained on (like a weird symbol)
sonnets_filtered = "".join([c for c in sonnets_clean if c in vocab])
# 4. Create the Out-of-Distribution (OOD) Test Set
test_ds_sonnets = CharDataset(sonnets_filtered, SEQ_LEN)
test_loader_sonnets = DataLoader(test_ds_sonnets, batch_size=BATCH_SIZE, shuffle=False)
print(f"New Test Set (Sonnets) loaded with {len(test_loader_sonnets)} batches.")
New Test Set (Sonnets) loaded with 447 batches.
In [22]:
print("--- INTERNAL VALIDATION (Same style) ---")
evaluate_final_model(model, val_loader, DEVICE, VOCAB_SIZE)
print("\n--- EXTERNAL TEST (Sonnets - Different style) ---")
evaluate_final_model(model, test_loader_sonnets, DEVICE, VOCAB_SIZE)
--- INTERNAL VALIDATION (Same style) ---
Calculating final metrics on 436 validation batches...
========================================
FINAL EVALUATION REPORT
========================================
Validation Loss: 2.7940
Perplexity: 16.35
========================================
LONG GENERATION (Temp=0.7):
----------------------------------------
KING RICHARD: use that thou art not her?
Your house, and come about her breast:
As never-let her hence to France to fell for love
As doth the town and find the scandal of bitter
Some tribunes, and be agreed;
Our doth relent, as the air with all dissending them.
O, he would have alone resolute of my traitor!
You were not be a lace-worse, or approved well;
Im absolmition are a bawd.
AUTOLYCUS:
Clown:
Prepare you, lords: O, he is grown your hands,
Thine being but a mind to her acceptain thee:
Thou, brace that
----------------------------------------
--- EXTERNAL TEST (Sonnets - Different style) ---
Calculating final metrics on 447 validation batches...
========================================
FINAL EVALUATION REPORT
========================================
Validation Loss: 3.9944
Perplexity: 54.29
========================================
LONG GENERATION (Temp=0.7):
----------------------------------------
KING RICHARD: use that thou, brace, at names,
And with the watching castle shall be dead.
KING RICHARD II:
We will descend and follow you; I should say
My tongue that may be spoke for a pattern, and my friends
Fertivation, for your father's death!
LADY CAPULET:
We think 'twill give him all unknown, and stinted
Lord England; but the last, and fast in a block,
Who has not our king, and who shuts upon the gain
To see it temposs me to the Tower.
Right coward, with him as I pupt a villain:
I have a stringle plan
----------------------------------------
README: Understanding Our Results¶
1. The Metric: Perplexity ($PPL$)¶
Perplexity is our primary gauge of the model's "certainty." Mathematically, it is $e^{\text{Loss}}$.
- Layman's Terms: It represents the branching factor. A $PPL$ of 16 means that for every character the model predicts, it is effectively choosing between 16 likely candidates.
- The Goal: As the model learns, the $PPL$ should drop. A $PPL$ equal to the Vocabulary Size (65) is equivalent to random guessing.
2. Comparison Table¶
| Metric | Internal (Plays) | External (Sonnets) |
|---|---|---|
| Loss | 2.79 | 3.99 |
| Perplexity | 16.35 | 54.29 |
| Performance | Strong | Weak |
3. The OOD (Out-of-Distribution) Concept¶
Our results demonstrate a classic machine learning challenge: Domain Shift.
- The Distribution: Our model was trained on the "Plays" distribution. This includes specific patterns like
CHARACTER NAME:,ACT I, andSCENE II. - The Shift: When we tested on the Sonnets, we moved Out-of-Distribution (OOD). Even though the author (Shakespeare) and the characters (A-Z) are the same, the structure is entirely different.
- Why the PPL Spiked: The model "expects" a character name and dialogue. When it encounters the continuous poetic flow of a Sonnet, its hidden state becomes "confused," leading to a $PPL$ of 54.29—nearly as high as a random guess.
Key Takeaway: High OOD Perplexity proves that our model hasn't just learned "English"—it has specifically learned "The Formatting of 16th Century Play Scripts."
In [19]:
%watermark -iv
datasets : 4.0.0 matplotlib : 3.10.0 numpy : 2.0.2 pandas : 2.2.2 re : 2.2.1 requests : 2.32.4 sklearn : 1.6.1 torch : 2.10.0+cu128 torchinfo : 1.8.0 torchvision: 0.25.0+cu128