Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [1]:
%%capture
!pip install watermark
!pip install torchinfo datasets
In [2]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 torch: 2.10.0+cu128
Character-Level GRU (Char-GRU) Language Model on WikiText-2¶
The network in this notebook is an implementation of a Character-Level Gated Recurrent Unit (Char-GRU) network trained on the WikiText-2 dataset. Moving from Shakespeare to Wikipedia is a major step up in difficulty. The model must learn capitalization, factual-sounding sentence structures, numbers, and technical vocabulary.
Network Architecture:
A Char-GRU is a Many-to-Many architecture. Unlike a classifier model that only cares about the final state, this model produces a prediction ($y_t$) at every single time step ($t$). For an input sequence of length $N$, the model makes $N$ predictions. The goal is to minimize the difference between the predicted next character and the actual next character in the sequence.
Many-to-Many RNN Structure. (Every input $x_t$ produces a corresponding output $y_t$ representing the probability of the next character.)
(Source)
The Internal Logic of the GRU Cell. (The same Gating mechanism used in classification is here tasked with "remembering" how to spell words.)
(Source)
Key Concepts of the Char-GRU Architecture:¶
- Sub-word Granularity: By operating at the character level, the model has a tiny vocabulary (usually ~85-100 characters after cleaning WikiText). This makes the
Embeddinglayer extremely memory-efficient but forces the GRU to do the heavy lifting of learning word boundaries and semantics. - Stateful Inference: During text generation, the hidden state ($h_t$) is passed back into the model for the next character prediction. This "memory" allows the model to maintain context over long strings.
- Temperature Scaling: A hyperparameter ($T$) used during generation to control "creativity." By scaling the output logits before the Softmax, we can force the model to be very conservative ($T < 0.5$) or highly experimental ($T > 1.0$).
Character-Level GRU: Technical Reference¶
Architecture Overview¶
Input (Character)
│
├── Embedding Layer (VOCAB_SIZE ≈ 85 → EMBED_DIM)
│ └── Maps unique characters (a, b, C, !, \n) to dense vectors.
│
├── Stacked GRU Layers (2-Layer Scratch Implementation)
│ ├── Layer 1: [Batch, Seq, Embed] → [Batch, Seq, Hidden]
│ └── Layer 2: [Batch, Seq, Hidden] → [Batch, Seq, Hidden]
│ └── Captures hierarchical patterns (Spelling → Grammar → Tone).
│
├── Linear Header (Hidden → VOCAB_SIZE)
│ └── Outputs "logits" for every single character in the sequence.
│
└── Softmax / Sampling
└── Converts logits to probabilities for the next character.
Training Specifics¶
Loss Function¶
We use nn.CrossEntropyLoss. Because our model outputs a prediction for every character in the sequence, we must flatten the tensors before calculating loss:
- Logits:
[Batch * Seq_Len, Vocab_Size] - Targets:
[Batch * Seq_Len]
Initial Loss "Robustness" Check¶
A mathematically sound Char-RNN should start with a loss approximately equal to the negative log of the vocabulary size:
$$-\ln\left(\frac{1}{V}\right) = \ln(V)$$
For WikiText ($V \approx 85$), the starting loss should be $\approx 4.44$. If it is significantly higher or lower, there is likely an initialization error in the GRU weights.
In [3]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset
from datasets import load_dataset
from collections import Counter
from torchinfo import summary
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
import re
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
In [4]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
In [5]:
##########################
### SETTINGS
##########################
# Hyperparameters upgraded for WikiText complexity
RANDOM_SEED = 123
BATCH_SIZE = 256
LEARNING_RATE = 0.002
NUM_EPOCHS = 10
SEQ_LEN = 150 # Increased context window for Wikipedia facts
EMBEDDING_DIM = 64
HIDDEN_DIM = 512
# Set device
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(DEVICE)
cuda
In [6]:
# 1. Load WikiText-2 Robustly
dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
raw_text = "\n".join(dataset["text"])
# 2. Clean the text (Remove Wikipedia headers and excessive newlines)
clean_text = re.sub(r'^\s*=+\s*[^=]+\s*=+\s*$', '', raw_text, flags=re.MULTILINE)
clean_text = re.sub(r'\n{2,}', '\n', clean_text)
# 3. Build Character-Level Vocabulary (Frequency Filtered)
char_counts = Counter(clean_text)
# Keep characters that appear at least 100 times to avoid rare unicode/math symbols
chars = sorted([ch for ch, count in char_counts.items() if count > 100])
vocab = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
VOCAB_SIZE = len(chars)
print(f"Original Unique Chars: {len(char_counts)}")
print(f"Filtered VOCAB_SIZE: {VOCAB_SIZE}")
# Filter the text to only include known characters
filtered_text = "".join([c for c in clean_text if c in vocab])
# 4. Custom Dataset for Language Modeling
# Instead of a step of 1, jump by SEQ_LEN
# This makes every batch "fresh" data and reduces batch count by 150x!
class CharDataset(Dataset):
def __init__(self, corpus, seq_len):
self.seq_len = seq_len
self.data = torch.tensor([vocab[c] for c in corpus], dtype=torch.long)
def __len__(self):
# STRIDE the data so we don't repeat sequences
return (len(self.data) - self.seq_len) // self.seq_len
def __getitem__(self, idx):
# Jump ahead by seq_len
start_idx = idx * self.seq_len
x = self.data[start_idx : start_idx + self.seq_len]
y = self.data[start_idx + 1 : start_idx + self.seq_len + 1]
return x, y
# 5. Initialize DataLoaders (90/5/5 Split)
n_train = int(0.9 * len(filtered_text))
n_val = int(0.05 * len(filtered_text))
train_ds = CharDataset(filtered_text[:n_train], SEQ_LEN)
val_ds = CharDataset(filtered_text[n_train:n_train+n_val], SEQ_LEN)
test_ds = CharDataset(filtered_text[n_train+n_val:], SEQ_LEN)
# The "Pro" version for larger datasets:
train_loader = DataLoader(
train_ds,
batch_size=BATCH_SIZE,
shuffle=True,
num_workers=2, # Uses 2 CPU cores to prep data in parallel
pin_memory=True # Speeds up the transfer from CPU to GPU
)
val_loader = DataLoader(val_ds, batch_size=BATCH_SIZE, shuffle=False, pin_memory=True)
test_loader = DataLoader(test_ds, batch_size=BATCH_SIZE, shuffle=False, pin_memory=True)
print(f"Train batches: {len(train_loader)} | Val batches: {len(val_loader)} | Test batches: {len(test_loader)}")
Original Unique Chars: 1013 Filtered VOCAB_SIZE: 100 Train batches: 255 | Val batches: 15 | Test batches: 15
In [7]:
class GRUCellFromScratch(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.gate_weights = nn.Linear(input_size + hidden_size, 2 * hidden_size)
self.candidate_weights = nn.Linear(input_size + hidden_size, hidden_size)
def forward(self, x, h_prev):
combined = torch.cat((x, h_prev), dim=1)
gates = torch.sigmoid(self.gate_weights(combined))
z_gate, r_gate = gates.chunk(2, dim=1)
combined_candidate = torch.cat((x, r_gate * h_prev), dim=1)
h_tilde = torch.tanh(self.candidate_weights(combined_candidate))
h_curr = (1 - z_gate) * h_prev + z_gate * h_tilde
return h_curr
In [8]:
class CharGRUFromScratch(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.gru_cell = GRUCellFromScratch(embed_dim, hidden_size)
self.fc = nn.Linear(hidden_size, vocab_size)
def forward(self, x, h=None):
batch_size, seq_len = x.shape
x = self.embedding(x)
if h is None:
h = torch.zeros(batch_size, self.hidden_size).to(x.device)
outputs = []
for t in range(seq_len):
h = self.gru_cell(x[:, t, :], h)
out = self.fc(h)
outputs.append(out.unsqueeze(1))
return torch.cat(outputs, dim=1), h
In [9]:
set_all_seeds(RANDOM_SEED)
model = CharGRUFromScratch(VOCAB_SIZE, EMBEDDING_DIM, HIDDEN_DIM).to(DEVICE)
summary(model,
input_data=torch.zeros((BATCH_SIZE, SEQ_LEN), dtype=torch.long).to(DEVICE),
col_names=["input_size", "output_size", "num_params", "mult_adds"],
device=DEVICE)
Out[9]:
============================================================================================================================================ Layer (type:depth-idx) Input Shape Output Shape Param # Mult-Adds ============================================================================================================================================ CharGRUFromScratch [256, 150] [256, 150, 100] -- -- ├─Embedding: 1-1 [256, 150] [256, 150, 64] 6,400 1,638,400 ├─GRUCellFromScratch: 1-2 [256, 64] [256, 512] -- -- │ └─Linear: 2-1 [256, 576] [256, 1024] 590,848 151,257,088 │ └─Linear: 2-2 [256, 576] [256, 512] 295,424 75,628,544 ├─Linear: 1-3 [256, 512] [256, 100] 51,300 13,132,800 ├─GRUCellFromScratch: 1-4 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-3 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-4 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-5 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-6 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-5 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-6 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-7 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-8 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-7 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-8 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-9 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-10 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-9 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-10 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-11 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-12 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-11 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-12 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-13 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-14 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-13 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-14 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-15 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-16 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-15 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-16 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-17 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-18 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-17 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-18 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-19 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-20 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-19 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-20 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-21 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-22 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-21 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-22 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-23 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-24 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-23 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-24 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-25 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-26 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-25 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-26 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-27 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-28 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-27 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-28 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-29 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-30 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-29 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-30 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-31 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-32 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-31 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-32 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-33 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-34 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-33 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-34 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-35 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-36 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-35 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-36 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-37 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-38 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-37 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-38 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-39 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-40 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-39 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-40 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-41 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-42 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-41 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-42 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-43 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-44 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-43 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-44 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-45 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-46 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-45 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-46 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-47 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-48 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-47 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-48 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-49 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-50 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-49 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-50 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-51 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-52 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-51 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-52 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-53 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-54 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-53 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-54 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-55 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-56 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-55 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-56 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-57 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-58 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-57 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-58 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-59 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-60 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-59 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-60 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-61 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-62 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-61 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-62 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-63 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-64 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-63 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-64 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-65 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-66 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-65 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-66 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-67 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-68 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-67 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-68 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-69 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-70 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-69 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-70 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-71 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-72 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-71 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-72 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-73 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-74 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-73 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-74 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-75 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-76 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-75 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-76 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-77 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-78 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-77 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-78 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-79 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-80 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-79 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-80 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-81 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-82 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-81 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-82 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-83 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-84 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-83 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-84 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-85 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-86 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-85 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-86 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-87 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-88 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-87 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-88 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-89 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-90 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-89 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-90 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-91 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-92 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-91 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-92 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-93 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-94 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-93 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-94 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-95 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-96 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-95 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-96 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-97 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-98 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-97 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-98 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-99 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-100 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-99 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-100 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-101 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-102 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-101 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-102 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-103 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-104 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-103 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-104 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-105 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-106 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-105 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-106 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-107 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-108 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-107 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-108 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-109 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-110 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-109 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-110 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-111 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-112 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-111 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-112 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-113 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-114 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-113 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-114 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-115 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-116 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-115 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-116 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-117 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-118 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-117 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-118 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-119 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-120 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-119 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-120 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-121 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-122 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-121 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-122 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-123 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-124 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-123 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-124 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-125 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-126 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-125 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-126 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-127 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-128 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-127 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-128 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-129 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-130 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-129 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-130 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-131 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-132 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-131 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-132 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-133 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-134 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-133 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-134 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-135 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-136 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-135 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-136 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-137 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-138 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-137 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-138 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-139 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-140 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-139 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-140 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-141 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-142 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-141 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-142 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-143 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-144 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-143 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-144 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-145 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-146 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-145 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-146 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-147 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-148 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-147 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-148 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-149 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-150 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-149 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-150 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-151 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-152 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-151 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-152 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-153 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-154 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-153 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-154 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-155 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-156 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-155 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-156 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-157 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-158 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-157 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-158 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-159 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-160 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-159 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-160 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-161 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-162 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-161 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-162 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-163 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-164 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-163 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-164 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-165 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-166 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-165 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-166 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-167 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-168 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-167 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-168 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-169 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-170 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-169 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-170 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-171 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-172 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-171 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-172 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-173 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-174 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-173 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-174 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-175 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-176 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-175 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-176 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-177 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-178 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-177 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-178 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-179 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-180 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-179 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-180 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-181 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-182 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-181 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-182 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-183 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-184 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-183 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-184 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-185 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-186 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-185 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-186 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-187 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-188 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-187 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-188 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-189 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-190 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-189 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-190 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-191 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-192 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-191 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-192 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-193 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-194 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-193 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-194 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-195 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-196 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-195 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-196 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-197 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-198 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-197 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-198 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-199 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-200 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-199 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-200 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-201 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-202 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-201 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-202 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-203 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-204 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-203 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-204 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-205 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-206 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-205 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-206 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-207 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-208 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-207 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-208 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-209 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-210 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-209 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-210 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-211 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-212 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-211 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-212 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-213 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-214 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-213 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-214 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-215 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-216 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-215 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-216 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-217 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-218 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-217 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-218 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-219 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-220 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-219 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-220 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-221 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-222 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-221 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-222 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-223 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-224 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-223 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-224 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-225 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-226 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-225 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-226 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-227 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-228 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-227 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-228 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-229 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-230 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-229 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-230 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-231 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-232 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-231 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-232 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-233 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-234 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-233 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-234 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-235 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-236 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-235 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-236 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-237 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-238 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-237 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-238 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-239 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-240 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-239 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-240 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-241 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-242 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-241 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-242 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-243 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-244 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-243 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-244 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-245 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-246 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-245 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-246 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-247 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-248 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-247 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-248 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-249 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-250 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-249 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-250 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-251 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-252 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-251 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-252 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-253 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-254 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-253 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-254 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-255 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-256 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-255 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-256 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-257 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-258 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-257 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-258 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-259 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-260 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-259 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-260 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-261 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-262 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-261 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-262 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-263 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-264 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-263 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-264 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-265 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-266 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-265 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-266 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-267 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-268 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-267 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-268 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-269 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-270 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-269 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-270 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-271 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-272 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-271 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-272 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-273 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-274 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-273 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-274 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-275 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-276 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-275 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-276 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-277 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-278 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-277 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-278 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-279 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-280 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-279 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-280 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-281 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-282 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-281 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-282 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-283 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-284 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-283 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-284 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-285 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-286 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-285 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-286 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-287 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-288 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-287 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-288 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-289 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-290 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-289 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-290 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-291 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-292 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-291 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-292 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-293 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-294 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-293 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-294 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-295 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-296 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-295 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-296 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-297 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-298 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-297 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-298 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-299 [256, 512] [256, 100] (recursive) 13,132,800 ├─GRUCellFromScratch: 1-300 [256, 64] [256, 512] (recursive) -- │ └─Linear: 2-299 [256, 576] [256, 1024] (recursive) 151,257,088 │ └─Linear: 2-300 [256, 576] [256, 512] (recursive) 75,628,544 ├─Linear: 1-301 [256, 512] [256, 100] (recursive) 13,132,800 ============================================================================================================================================ Total params: 943,972 Trainable params: 943,972 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 36.00 ============================================================================================================================================ Input size (MB): 0.31 Forward/backward pass size (MB): 522.24 Params size (MB): 3.78 Estimated Total Size (MB): 526.32 ============================================================================================================================================
In [10]:
def generate_text(model, seed="The history of ", length=150, temperature=0.8):
model.eval()
with torch.no_grad():
chars_idx = [vocab[c] for c in seed]
input_tensor = torch.tensor(chars_idx).unsqueeze(0).to(DEVICE)
_, h = model(input_tensor)
current_char_tensor = input_tensor[:, -1:]
generated_output = seed
for _ in range(length):
logits, h = model(current_char_tensor, h)
logits = logits.squeeze() / temperature
probs = torch.softmax(logits, dim=-1)
next_char_idx = torch.multinomial(probs, num_samples=1).item()
generated_output += itos[next_char_idx]
current_char_tensor = torch.tensor([[next_char_idx]]).to(DEVICE)
return generated_output
In [11]:
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
In [12]:
def validate_perplexity(model, loader, device):
model.eval()
total_loss = 0
with torch.no_grad():
for inputs, targets in loader:
inputs, targets = inputs.to(device), targets.to(device)
logits, _ = model(inputs)
loss = nn.functional.cross_entropy(
logits.view(-1, VOCAB_SIZE),
targets.view(-1)
)
total_loss += loss.item()
avg_loss = total_loss / len(loader)
perplexity = torch.exp(torch.tensor(avg_loss)).item()
return avg_loss, perplexity
In [13]:
start_time = time.time()
print_interval = max(1, len(train_loader) // 5) # 5 prints per epoch
history = {
'train_loss': [],
'val_loss': [],
'val_ppl': [],
'lrs': []
}
for epoch in range(NUM_EPOCHS):
model.train()
total_loss = 0
running_batch_loss = 0
for i, (inputs, targets) in enumerate(train_loader):
inputs, targets = inputs.to(DEVICE), targets.to(DEVICE)
logits, _ = model(inputs)
loss = criterion(logits.view(-1, VOCAB_SIZE), targets.view(-1))
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
total_loss += loss.item()
running_batch_loss += loss.item()
if (i + 1) % print_interval == 0:
avg_batch_loss = running_batch_loss / print_interval
print(f"Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | "
f"Batch {i+1:04d}/{len(train_loader):04d} | "
f"Loss: {avg_batch_loss:.4f}")
running_batch_loss = 0
# --- EPOCH END LOGGING ---
avg_train_loss = total_loss / len(train_loader)
v_loss, v_ppl = validate_perplexity(model, val_loader, DEVICE)
history['train_loss'].append(avg_train_loss)
history['val_loss'].append(v_loss)
history['val_ppl'].append(v_ppl)
history['lrs'].append(optimizer.param_groups[0]['lr'])
elapsed = (time.time() - start_time) / 60
print(f"\n{'-'*30}")
print(f"EPOCH {epoch+1} COMPLETE | Time: {elapsed:.2f} min")
print(f"Train Loss: {avg_train_loss:.4f} | Val Loss: {v_loss:.4f} | Val PPL: {v_ppl:.2f}")
print(f"{'-'*30}")
print(f"GENERATED TEXT:\n{generate_text(model, seed='The history of ', length=200)}")
print(f"{'='*70}\n")
Epoch: 001/010 | Batch 0051/0255 | Loss: 2.6533 Epoch: 001/010 | Batch 0102/0255 | Loss: 2.0313 Epoch: 001/010 | Batch 0153/0255 | Loss: 1.7995 Epoch: 001/010 | Batch 0204/0255 | Loss: 1.6554 Epoch: 001/010 | Batch 0255/0255 | Loss: 1.5629 ------------------------------ EPOCH 1 COMPLETE | Time: 0.82 min Train Loss: 1.9405 | Val Loss: 1.5874 | Val PPL: 4.89 ------------------------------ GENERATED TEXT: The history of is a woble and other book ( 500 bone . Marcharawa had strong the sce of the congract , and incruised the new archienced by the Somery Howe I and hosside they Vore and Bape . Since of its after the Mal ====================================================================== Epoch: 002/010 | Batch 0051/0255 | Loss: 1.4988 Epoch: 002/010 | Batch 0102/0255 | Loss: 1.4562 Epoch: 002/010 | Batch 0153/0255 | Loss: 1.4255 Epoch: 002/010 | Batch 0204/0255 | Loss: 1.4005 Epoch: 002/010 | Batch 0255/0255 | Loss: 1.3811 ------------------------------ EPOCH 2 COMPLETE | Time: 1.59 min Train Loss: 1.4324 | Val Loss: 1.4487 | Val PPL: 4.26 ------------------------------ GENERATED TEXT: The history of on Melicies Bewit he ended the popularizy of newly , among the song " , and the New Low and so that Parliament , and an exchange of this as " dell 's head " and it is recorded with the networks . Ga ====================================================================== Epoch: 003/010 | Batch 0051/0255 | Loss: 1.3534 Epoch: 003/010 | Batch 0102/0255 | Loss: 1.3411 Epoch: 003/010 | Batch 0153/0255 | Loss: 1.3329 Epoch: 003/010 | Batch 0204/0255 | Loss: 1.3239 Epoch: 003/010 | Batch 0255/0255 | Loss: 1.3180 ------------------------------ EPOCH 3 COMPLETE | Time: 2.35 min Train Loss: 1.3339 | Val Loss: 1.4058 | Val PPL: 4.08 ------------------------------ GENERATED TEXT: The history of , experience , the territory of the Virginian Times and Sociali and Wales ' held for the burght retirement with the north when a result of projected joining the types of Abu @-@ That arrived in displa ====================================================================== Epoch: 004/010 | Batch 0051/0255 | Loss: 1.2946 Epoch: 004/010 | Batch 0102/0255 | Loss: 1.2909 Epoch: 004/010 | Batch 0153/0255 | Loss: 1.2837 Epoch: 004/010 | Batch 0204/0255 | Loss: 1.2831 Epoch: 004/010 | Batch 0255/0255 | Loss: 1.2756 ------------------------------ EPOCH 4 COMPLETE | Time: 3.12 min Train Loss: 1.2856 | Val Loss: 1.3843 | Val PPL: 3.99 ------------------------------ GENERATED TEXT: The history of which became apparently care hard at the inner cartoby , the content of the English event , the moon fantased with the recognition are available to investigate the speculation of all making successful ====================================================================== Epoch: 005/010 | Batch 0051/0255 | Loss: 1.2566 Epoch: 005/010 | Batch 0102/0255 | Loss: 1.2573 Epoch: 005/010 | Batch 0153/0255 | Loss: 1.2576 Epoch: 005/010 | Batch 0204/0255 | Loss: 1.2535 Epoch: 005/010 | Batch 0255/0255 | Loss: 1.2499 ------------------------------ EPOCH 5 COMPLETE | Time: 3.89 min Train Loss: 1.2550 | Val Loss: 1.3712 | Val PPL: 3.94 ------------------------------ GENERATED TEXT: The history of Pier . Although the administer of any of the enen to added the story in the late 19th century , and the Terhignes and songs in the 2012 – 00 season . There are an understanding way race , and the to ====================================================================== Epoch: 006/010 | Batch 0051/0255 | Loss: 1.2335 Epoch: 006/010 | Batch 0102/0255 | Loss: 1.2338 Epoch: 006/010 | Batch 0153/0255 | Loss: 1.2333 Epoch: 006/010 | Batch 0204/0255 | Loss: 1.2331 Epoch: 006/010 | Batch 0255/0255 | Loss: 1.2325 ------------------------------ EPOCH 6 COMPLETE | Time: 4.64 min Train Loss: 1.2332 | Val Loss: 1.3646 | Val PPL: 3.91 ------------------------------ GENERATED TEXT: The history of 22 and 30 June , who was nominated by the 1985 – 30 American Act , but the equivalent of dramatically built before he was a pop symbolic association for Subui Tomas . After the commercial of the adver ====================================================================== Epoch: 007/010 | Batch 0051/0255 | Loss: 1.2153 Epoch: 007/010 | Batch 0102/0255 | Loss: 1.2174 Epoch: 007/010 | Batch 0153/0255 | Loss: 1.2152 Epoch: 007/010 | Batch 0204/0255 | Loss: 1.2186 Epoch: 007/010 | Batch 0255/0255 | Loss: 1.2158 ------------------------------ EPOCH 7 COMPLETE | Time: 5.41 min Train Loss: 1.2165 | Val Loss: 1.3627 | Val PPL: 3.91 ------------------------------ GENERATED TEXT: The history of 25 @-@ 99 , also known as Muslims and the territory . = = Death = = In January 1850 , the National Hurricane Championship review in Ionator and Pressures , Saunteral Secrotter , where he won the S ====================================================================== Epoch: 008/010 | Batch 0051/0255 | Loss: 1.1997 Epoch: 008/010 | Batch 0102/0255 | Loss: 1.2033 Epoch: 008/010 | Batch 0153/0255 | Loss: 1.2031 Epoch: 008/010 | Batch 0204/0255 | Loss: 1.2044 Epoch: 008/010 | Batch 0255/0255 | Loss: 1.2071 ------------------------------ EPOCH 8 COMPLETE | Time: 6.18 min Train Loss: 1.2035 | Val Loss: 1.3587 | Val PPL: 3.89 ------------------------------ GENERATED TEXT: The history of 273 feet 2 in early 1997 , and the Zimmenberg were abolished by the Second Doces on the 1980s . In December 2002 the Sunday River , the Star Department of the war to the New Zealand Day , and the Gura ====================================================================== Epoch: 009/010 | Batch 0051/0255 | Loss: 1.1879 Epoch: 009/010 | Batch 0102/0255 | Loss: 1.1939 Epoch: 009/010 | Batch 0153/0255 | Loss: 1.1899 Epoch: 009/010 | Batch 0204/0255 | Loss: 1.1968 Epoch: 009/010 | Batch 0255/0255 | Loss: 1.1954 ------------------------------ EPOCH 9 COMPLETE | Time: 6.95 min Train Loss: 1.1928 | Val Loss: 1.3552 | Val PPL: 3.88 ------------------------------ GENERATED TEXT: The history of earlier gaes were closely thought to be used in total of an interface break to the site but . The poor forms of chemical buildings into a run for the state without care intensity and broken by the mou ====================================================================== Epoch: 010/010 | Batch 0051/0255 | Loss: 1.1781 Epoch: 010/010 | Batch 0102/0255 | Loss: 1.1816 Epoch: 010/010 | Batch 0153/0255 | Loss: 1.1847 Epoch: 010/010 | Batch 0204/0255 | Loss: 1.1852 Epoch: 010/010 | Batch 0255/0255 | Loss: 1.1866 ------------------------------ EPOCH 10 COMPLETE | Time: 7.69 min Train Loss: 1.1832 | Val Loss: 1.3549 | Val PPL: 3.88 ------------------------------ GENERATED TEXT: The history of level and performer in the area . When all reinfronting in the same run episode of the Best Lucan in Permanett writers had approved of the remaining cities of the matter Cubeb ( literally operated by ======================================================================
In [19]:
def plot_training_results(history):
epochs = range(1, len(history['train_loss']) + 1)
# Adjusted to 12x5 for a better aspect ratio with 2 plots
plt.figure(figsize=(12, 5))
# 1. Loss Plot (Training vs Validation)
plt.subplot(1, 2, 1)
plt.plot(epochs, history['train_loss'], 'o-', label='Train Loss')#, color='#1f77b4')
plt.plot(epochs, history['val_loss'], 's-', label='Val Loss')#, color='#ff7f0e')
plt.title('Cross-Entropy Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
# 2. Perplexity Plot (Exponential of Loss)
plt.subplot(1, 2, 2)
plt.plot(epochs, history['val_ppl'], 'g^-', label='Val Perplexity')#, color='#2ca02c')
plt.title('Model Perplexity (Uncertainty)')
plt.xlabel('Epochs')
plt.ylabel('Perplexity')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
plt.tight_layout()
plt.show()
# Run this after training
plot_training_results(history)
In [15]:
def evaluate_final_model(model, loader, device, vocab_size):
model.eval()
total_loss = 0
print(f"Calculating final metrics on {len(loader)} test batches...")
with torch.no_grad():
for inputs, targets in loader:
inputs, targets = inputs.to(device), targets.to(device)
logits, _ = model(inputs)
loss = nn.functional.cross_entropy(
logits.view(-1, vocab_size),
targets.view(-1)
)
total_loss += loss.item()
avg_val_loss = total_loss / len(loader)
perplexity = torch.exp(torch.tensor(avg_val_loss)).item()
print("\n" + "="*40)
print(f"{'FINAL EVALUATION REPORT':^40}")
print("="*40)
print(f"Test Loss: {avg_val_loss:.4f}")
print(f"Perplexity: {perplexity:.2f}")
print("="*40)
print("\nLONG GENERATION (Temp=0.7):")
print("-" * 40)
print(generate_text(model, seed="During the war, ", length=500, temperature=0.7))
print("-" * 40)
# Run final test
evaluate_final_model(model, test_loader, DEVICE, VOCAB_SIZE)
Calculating final metrics on 15 test batches...
========================================
FINAL EVALUATION REPORT
========================================
Test Loss: 1.3509
Perplexity: 3.86
========================================
LONG GENERATION (Temp=0.7):
----------------------------------------
During the war, to the stabblish consent .
= = = Criticism = = = =
The new government led by NOK the propoceduated hard role of the tropical dunt for the fine at the start of Hurricane American support of the storm for the New Year 's Anti @-@ Horm Grand Road . In the last author , he had all remembered in the 1972 Shored State Lead , with the site of eight captivation in the city . As a move with the Missouri River aircraft manager with 16 people that may have photograph through heavy missions of Southern
----------------------------------------
In [16]:
# 1. Load the OOD Dataset (News)
ood_data = load_dataset("ag_news", split="test")
ood_raw_text = "\n".join(ood_data["text"][:1000]) # Sample for testing
# 2. Perform the Coverage Audit
ood_chars = set(ood_raw_text)
train_chars = set(vocab.keys())
missing_chars = ood_chars - train_chars
coverage = len(ood_chars & train_chars) / len(ood_chars) * 100
print(f"--- VOCABULARY AUDIT ---")
print(f"Character Coverage: {coverage:.2f}%")
if missing_chars:
print(f"Characters in News but NOT in Training: {sorted(list(missing_chars))}")
else:
print("Perfect Match: All characters in the OOD set were seen during training.")
# 3. Clean and Load for Evaluation
# We filter out missing chars to prevent the model from crashing (IndexError)
ood_filtered = "".join([c for c in ood_raw_text if c in vocab])
ood_ds = CharDataset(ood_filtered, SEQ_LEN)
ood_loader = DataLoader(ood_ds, batch_size=BATCH_SIZE, shuffle=False)
print(f"OOD Test Set (News) loaded with {len(ood_loader)} batches.")
--- VOCABULARY AUDIT --- Character Coverage: 97.53% Characters in News but NOT in Training: ['#', '\\'] OOD Test Set (News) loaded with 7 batches.
In [17]:
print("--- INTERNAL TEST (Wikipedia Style) ---")
evaluate_final_model(model, test_loader, DEVICE, VOCAB_SIZE)
print("\n--- EXTERNAL OOD TEST (News Style) ---")
evaluate_final_model(model, ood_loader, DEVICE, VOCAB_SIZE)
--- INTERNAL TEST (Wikipedia Style) ---
Calculating final metrics on 15 test batches...
========================================
FINAL EVALUATION REPORT
========================================
Test Loss: 1.3509
Perplexity: 3.86
========================================
LONG GENERATION (Temp=0.7):
----------------------------------------
During the war, whether these activity , and a common window entitle generation and a total of the storm . The trees attacked and withdraws are competition in the world . In the canoots possessed the other movement and described as the front game woman had to be used to be seen in the stadium . This structure was used to that of the extensive long . ( August 2014 ) built a survey of all to have been the largest management and returned to the fungus role . The previously population of the first two year people w
----------------------------------------
--- EXTERNAL OOD TEST (News Style) ---
Calculating final metrics on 7 test batches...
========================================
FINAL EVALUATION REPORT
========================================
Test Loss: 2.2103
Perplexity: 9.12
========================================
LONG GENERATION (Temp=0.7):
----------------------------------------
During the war, 4 metanchs ( 1 @,@ 000 m ) . Each to three from the attack , teachers are not contracted , less gravel preserves , and widened , and have been the tower 's place .
= = = Tropical Deputy Times , also interaction = = =
The Vice @-@ class writer and professional harbor with the Langue Parcher and the Health Regiment could not feature the Industrial Science Studio and Victorian Canadian unit . The archives compete are a semi @-@ scale reservoir , three other ' difficulties of people should not b
----------------------------------------
In [20]:
def print_comparison(wiki_results, news_results):
print("="*40)
print(f"{'Metric':<15} | {'Wiki':<10} | {'News':<10}")
print("-"*40)
print(f"{'Loss':<15} | {wiki_results['loss']:<10.4f} | {news_results['loss']:<10.4f}")
print(f"{'Perplexity':<15} | {wiki_results['ppl']:<10.2f} | {news_results['ppl']:<10.2f}")
gap = (news_results['ppl'] / wiki_results['ppl'] - 1) * 100
print("-"*40)
print(f"OOD Perplexity Increase: {gap:.1f}%")
print("="*40)
# Example usage with your results:
wiki_res = {'loss': 1.3509, 'ppl': 3.86}
news_res = {'loss': 2.2103, 'ppl': 9.12}
print_comparison(wiki_res, news_res)
======================================== Metric | Wiki | News ---------------------------------------- Loss | 1.3509 | 2.2103 Perplexity | 3.86 | 9.12 ---------------------------------------- OOD Perplexity Increase: 136.3% ========================================
Conclusion: Domain Adaptation & OOD Analysis¶
The results demonstrate that while the GRU successfully mastered the WikiText-2 internal distribution, it suffered significant performance degradation when shifted to an external News domain.
Key Findings: Domain Sensitivity & Robustness¶
The model was evaluated on both its native distribution (WikiText-2) and an out-of-distribution (AG News) dataset to test its generalization limits.
1. Performance Gap¶
The transition from Wikipedia to AG News OOD data caused a 136% increase in Perplexity ($3.86 \rightarrow 9.12$). This confirms that the model’s predictive power is heavily "locked" into encyclopedic linguistic patterns rather than universal English syntax.
2. Structural Overfitting¶
The "News" generation results revealed deep structural bias. The model consistently hallucinated:
- Markdown Headers: Using
= = =formatting. - Preprocessing Artifacts: Generating
@-@tokens unique to the WikiText source. - Contextual Anchors: References to "August 2014" or "The history of," proving that the global style of the training data overrides the local context of the OOD prompt.
3. Vocabulary Drift¶
A vocabulary audit showed 97.53% coverage. The remaining 2.47% (characters like # and \) represent "blind spots" where the model is forced to guess, contributing to the higher OOD loss.
4. Robustness Insight¶
This gap confirms that character-level models are highly sensitive to the global structure of their training corpus, not just the local grammar.
In [18]:
%watermark -iv
datasets : 4.0.0 matplotlib: 3.10.0 numpy : 2.0.2 pandas : 2.2.2 re : 2.2.1 torch : 2.10.0+cu128 torchinfo : 1.8.0