Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [1]:
%%capture
!pip install watermark
!pip install torchinfo datasets
In [2]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 torch: 2.9.0+cu126
Stacked Bidirectional Gated Recurrent Unit (Stacked Bi-GRU) Classifier¶
The network in this notebook is a sophisticated implementation of a Stacked Bidirectional Gated Recurrent Unit (Bi-GRU) network [1, 2, 3]. While a standard Stacked GRU processes information in a single temporal direction (past to future), a Stacked Bi-GRU processes the sequence in both directions simultaneously across multiple layers. This "dual-context" approach is critical for AG News, where the meaning of a word often depends as much on the words that follow it as the words that precede it.
Network Architecture:
In a Stacked Bi-GRU, each layer consists of two independent GRU components: one processing the sequence forward ($h_t^{\rightarrow}$) and one backward ($h_t^{\leftarrow}$). The concatenated output of these two directions at Layer 1 becomes the input for Layer 2. This creates a deep, "future-aware" hierarchical representation, allowing the model to resolve ambiguities in news headlines by looking at the entire context of the sentence before making a classification.
(Source, [6])
Key Concepts of the Stacked Bi-GRU Architecture:¶
- Contextual Omniscience: Unlike unidirectional models, a Bi-GRU has a "look-ahead" capability. By the time the model reaches the final classification layer, every hidden state has "seen" the entire sentence from both ends.
- Cross-Layer Feature Fusion: The hidden states of Layer 1 (capturing bidirectional syntax) are fused and passed to Layer 2, which refines these features into high-level semantic categories (e.g., distinguishing "World" news from "Business" based on global sentence structure).
- Increased Representational Power: Because each layer concatenates two hidden vectors, the model effectively doubles its capacity to store information about the sequence at every stage of the stack.
Quick Comparison Table: Stacked GRU vs. Stacked Bi-GRU¶
| Feature | Stacked GRU (Unidirectional) | Stacked Bi-GRU (Bidirectional) |
|---|---|---|
| Data Flow | Past $\rightarrow$ Future | Past $\leftrightarrow$ Future (Both) |
| Context | Causal (Only looks back) | Global (Looks back & forward) |
| Hidden State | $h_t$ | $[h_t^{\rightarrow} ; h_t^{\leftarrow}]$ (Concatenated) |
| Complexity | High | Very High (2$\times$ parameters per layer) |
| Computation | Sequential | Parallelizable (FWD/BWD can run at once) |
| Best For | Generative Task / Real-time | Text Classification / Sentiment / NER |
REFERENCES:
- Schuster, Mike, and Kuldip K. Paliwal. "Bidirectional recurrent neural networks." IEEE Transactions on Signal Processing, 1997. (The Bidirectional Foundation)
- Graves, Alex, and Jürgen Schmidhuber. "Framewise phoneme classification with bidirectional LSTM and other network architectures." Neural Networks, 2005.
- Pascanu, Razvan, et al. "How to construct deep recurrent neural networks." arXiv preprint arXiv:1312.6026, 2013. (Deep/Stacked RNN logic)
- Cho, Kyunghyun, et al. "Learning Phrase Representations using RNN Encoder-Decoder." arXiv, 2014. (Original GRU)
- Google. (2026). "Gemini 3 Flash" [Large language model]. Google AI
In [3]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from sklearn.model_selection import train_test_split
from torch.utils.data import Sampler, DataLoader
from torch.nn.utils.rnn import pad_sequence
from datasets import load_dataset, DatasetDict
from collections import Counter
from sklearn.metrics import accuracy_score, f1_score, confusion_matrix, classification_report
# from torch.cuda.amp import GradScaler, autocast
from torch.amp import GradScaler, autocast
from torchinfo import summary
import seaborn as sns
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
import re
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
In [4]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
In [5]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 123
BATCH_SIZE = 128
LEARNING_RATE = 0.0005 # 0.001
NUM_EPOCHS = 10 # News topics usually converge faster than char-level text
# NUM_CLASSES = 4 # [World, Sports, Business, Sci/Tech]
# Vocabulary & Architecture
VOCABULARY_SIZE = 20000
EMBEDDING_DIM = 128
HIDDEN_DIM = 128 #256
DROPOUT_P = 0.5 # Crucial for Stacked architectures to prevent overfitting
# Sequence Constraints
# MAX_LEN = 100 # AG News sentences are shorter than Wikipedia/Shakespeare chunks
# Set device
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(DEVICE)
cuda
In [6]:
# 1. Load Dataset
DATASET_CONFIG = "ag_news" # Use "imdb", "ag_news", "amazon_reviews_multi", etc.
raw_datasets = load_dataset(DATASET_CONFIG)
# 2. Split 10% of 'train' to create a 'validation' set (seed for reproducibility)
train_val_split = raw_datasets["train"].train_test_split(test_size=0.1, seed=RANDOM_SEED)
ds_splits = DatasetDict({
'train': train_val_split['train'],
'validation': train_val_split['test'], # The 10% we just carved out
'test': raw_datasets['test']
})
print(ds_splits)
# ------------Robust Extraction of Class Names----------
dataset_label_features = raw_datasets['train'].features['label']
if hasattr(dataset_label_features, 'names'):
class_names = dataset_label_features.names
NUM_CLASSES = len(class_names)
else:
# Fallback for datasets that don't have metadata names
NUM_CLASSES = len(set(raw_datasets['train']['label']))
class_names = [f"Class {i}" for i in range(NUM_CLASSES)]
# ----------Auto-detect Binary vs Multi-class-------------
# Note: For IMDB, NUM_CLASSES is 2. For AG News, it's 4.
# IS_BINARY = True if NUM_CLASSES <= 2 else False
is_imdb = DATASET_CONFIG.lower() == "imdb"
IS_BINARY = True if is_imdb else False
# --------DYNAMIC HYPERPARAMETERS ----------------------
# If binary, output layer needs 1 node (for Sigmoid: BCEWithLogitsLoss); else num_classes (for Softmax)
MODEL_OUTPUT_SIZE = 1 if IS_BINARY else NUM_CLASSES
print(f"\n[ROBUST CONFIG COMPLETE]")
print(f"Dataset: {DATASET_CONFIG} | Num of Classes : {NUM_CLASSES} | Classes: {class_names}")
print(f"Is Binary: {IS_BINARY} | Model Output Nodes: {MODEL_OUTPUT_SIZE}")
# 2. Build Vocabulary (From Scratch Logic)
tokenizer = lambda text: re.findall(r'\w+', text.lower())
word_counts = Counter()
# Important: We only iterate over ds_splits['train']
for text in ds_splits['train']['text']:
word_counts.update(tokenizer(text))
# Filter rare words and create mapping: : <PAD>=0, <UNK>=1, then top 20k words
vocab = {word: i + 2 for i, (word, count) in enumerate(word_counts.most_common(VOCABULARY_SIZE))}
vocab['<PAD>'] = 0
vocab['<UNK>'] = 1
print(f"Vocab size: {len(vocab)}")
VOCAB_SIZE = len(vocab)
# 3. Custom Dataset Class
class TextDataset(torch.utils.data.Dataset):
def __init__(self, dataset_split):
self.data = dataset_split
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
# Return raw text and label. Let collate_fn handle the rest.
item = self.data[idx]
return item['text'], item['label']
# 4. Robust Text Pipeline
def text_pipeline(text, max_len=128):
tokens = tokenizer(text)
# Use .get(t, 1) because 1 is your <UNK> index
return [vocab.get(t, 1) for t in tokens][:max_len]
# 5. Robust Collate Batch
def collate_batch(batch):
label_list, text_list, lengths = [], [], []
for _text, _label in batch:
# 1. Labels: Use LONG for AG News (Multi-class)
label_list.append(_label)
# 2. Numericalize and Truncate
processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)
text_list.append(processed_text)
lengths.append(processed_text.size(0))
# 3. Padding logic
# We use <PAD> index 0. We use vocab.get just in case names change.
pad_idx = vocab.get('<PAD>', 0)
# This dynamically pads to the longest sequence in THIS batch (more efficient)
text_list_padded = pad_sequence(text_list, batch_first=True, padding_value=pad_idx)
label_tensor = torch.tensor(label_list, dtype=torch.long)
length_tensor = torch.tensor(lengths, dtype=torch.int64)
return text_list_padded.to(DEVICE), label_tensor.to(DEVICE), length_tensor.to(DEVICE)
class BucketSampler(Sampler):
def __init__(self, data_source, batch_size, shuffle=True):
self.data_source = data_source
self.batch_size = batch_size
self.shuffle = shuffle
# Pre-calculate indices sorted by length
# This assumes your Dataset returns (text_tensor, label)
# or has access to lengths
self.indices = sorted(range(len(data_source)),
key=lambda i: len(data_source[i][0]))
def __iter__(self):
# Create batches of indices
batches = [self.indices[i:i + self.batch_size]
for i in range(0, len(self.indices), self.batch_size)]
if self.shuffle:
# Shuffle the order of the batches, not the items inside them
import random
random.shuffle(batches)
# Flatten and return
return iter([idx for batch in batches for idx in batch])
def __len__(self):
return len(self.data_source)
def collate_batch(batch):
label_list, text_list, lengths = [], [], []
for _text, _label in batch:
label_list.append(_label)
# Note: text_pipeline should ideally happen in the Dataset.__getitem__
# to save time, but keeping it here for now is okay.
processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)
text_list.append(processed_text)
lengths.append(processed_text.size(0))
pad_idx = vocab.get('<PAD>', 0)
# Pad on CPU
text_list_padded = pad_sequence(text_list, batch_first=True, padding_value=pad_idx)
label_tensor = torch.tensor(label_list, dtype=torch.long)
length_tensor = torch.tensor(lengths, dtype=torch.int64)
# DO NOT .to(DEVICE) HERE
return text_list_padded, label_tensor, length_tensor
# 6. Initialize DataLoaders
train_ds = TextDataset(ds_splits['train'])
val_ds = TextDataset(ds_splits['validation'])
test_ds = TextDataset(ds_splits['test'])
# Define common kwargs for robustness
loader_kwargs = {
'batch_size': BATCH_SIZE,
'collate_fn': collate_batch, # <--- Plug it in here
'pin_memory': True,
'num_workers': 2
}
# Training Loader with the BucketSampler
train_loader = DataLoader(
train_ds,
sampler=BucketSampler(train_ds, BATCH_SIZE, shuffle=True),
**loader_kwargs
)
# Val/Test Loaders
val_loader = DataLoader(val_ds, shuffle=False, **loader_kwargs)
test_loader = DataLoader(test_ds, shuffle=False, **loader_kwargs)
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
DatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 108000
})
validation: Dataset({
features: ['text', 'label'],
num_rows: 12000
})
test: Dataset({
features: ['text', 'label'],
num_rows: 7600
})
})
[ROBUST CONFIG COMPLETE]
Dataset: ag_news | Num of Classes : 4 | Classes: ['World', 'Sports', 'Business', 'Sci/Tech']
Is Binary: False | Model Output Nodes: 4
Vocab size: 20002
In [7]:
import torch.nn as nn
class GRUCellFromScratch(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.x2h = nn.Linear(input_size, 3 * hidden_size, bias=True)
self.h2h = nn.Linear(hidden_size, 3 * hidden_size, bias=True)
def forward(self, x, h):
# Fusion happens here
gate_x = self.x2h(x)
gate_h = self.h2h(h)
# Chunking into Reset, Update, and New gates
i_r, i_z, i_n = gate_x.chunk(3, 1)
h_r, h_z, h_n = gate_h.chunk(3, 1)
reset_gate = torch.sigmoid(i_r + h_r)
update_gate = torch.sigmoid(i_z + h_z)
new_gate = torch.tanh(i_n + (reset_gate * h_n))
h_new = (1 - update_gate) * new_gate + update_gate * h
return h_new
# To make it even faster, wrap the logic in JIT:
# scripted_cell = torch.jit.script(GRUCellFromScratch(dim, hid))
In [8]:
class GRUCellFromScratch(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
# We need weights for 3 operations:
# Update gate (z), Reset gate (r), and Candidate hidden state (h_tilde)
self.gate_weights = nn.Linear(input_size + hidden_size, 2 * hidden_size)
self.candidate_weights = nn.Linear(input_size + hidden_size, hidden_size)
def forward(self, x, h_prev):
# x: [batch, input_size]
# h_prev: [batch, hidden_size]
# 1. Calculate Update (z) and Reset (r) gates
combined = torch.cat((x, h_prev), dim=1)
gates = torch.sigmoid(self.gate_weights(combined))
z_gate, r_gate = gates.chunk(2, dim=1)
# 2. Calculate Candidate Hidden State (h_tilde)
# We use the Reset gate to "mask" the previous hidden state
combined_candidate = torch.cat((x, r_gate * h_prev), dim=1)
h_tilde = torch.tanh(self.candidate_weights(combined_candidate))
# 3. Final Hidden State update
# This is a linear interpolation between past and candidate
h_curr = (1 - z_gate) * h_prev + z_gate * h_tilde
return h_curr
In [9]:
class BiStackedGRUFromScratch(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden_size, output_size, dropout_p=0.5):
super().__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(vocab_size, embed_dim)
# Layer 1 Cells
self.gru1_fwd = GRUCellFromScratch(embed_dim, hidden_size)
self.gru1_bwd = GRUCellFromScratch(embed_dim, hidden_size)
# Layer 2 Cells (Input is 2 * hidden_size)
self.gru2_fwd = GRUCellFromScratch(hidden_size * 2, hidden_size)
self.gru2_bwd = GRUCellFromScratch(hidden_size * 2, hidden_size)
self.dropout = nn.Dropout(dropout_p)
self.fc = nn.Linear(hidden_size * 2, output_size)
def _process_layer(self, x, lengths, fwd_cell, bwd_cell):
batch_size, seq_len, _ = x.shape
# 1. Pre-flip input for the backward pass
x_rev = torch.flip(x, [1])
h_f = torch.zeros(batch_size, self.hidden_size, device=x.device)
h_b = torch.zeros(batch_size, self.hidden_size, device=x.device)
fwd_states, bwd_states = [], []
# 2. Parallel Step: Process fwd and bwd time steps in one loop
for t in range(seq_len):
# Forward Logic
h_f_next = fwd_cell(x[:, t, :], h_f)
mask_f = (t < lengths).float().unsqueeze(1)
h_f = (mask_f * h_f_next) + ((1 - mask_f) * h_f)
fwd_states.append(h_f.unsqueeze(1))
# Backward Logic (on flipped input)
h_b_next = bwd_cell(x_rev[:, t, :], h_b)
# Flipped mask logic: index t in flipped x corresponds to seq_len-1-t
mask_b = (t < lengths).float().unsqueeze(1)
h_b = (mask_b * h_b_next) + ((1 - mask_b) * h_b)
bwd_states.append(h_b.unsqueeze(1))
# 3. Reconstruct full sequence
fwd_seq = torch.cat(fwd_states, dim=1)
# Flip backward states back to temporal order
bwd_seq = torch.flip(torch.cat(bwd_states, dim=1), [1])
return torch.cat([fwd_seq, bwd_seq], dim=-1)
def forward(self, x, lengths):
batch_size, seq_len = x.shape
x = self.embedding(x)
# Pass through Layer 1
l1_out = self._process_layer(x, lengths, self.gru1_fwd, self.gru1_bwd)
l1_out = self.dropout(l1_out)
# Pass through Layer 2
l2_out = self._process_layer(l1_out, lengths, self.gru2_fwd, self.gru2_bwd)
# --- EFFICIENT HIDDEN STATE EXTRACTION ---
# Forward: take state at lengths - 1
fwd_idx = (lengths - 1).view(-1, 1, 1).expand(batch_size, 1, self.hidden_size)
fwd_last = l2_out[:, :, :self.hidden_size].gather(1, fwd_idx).squeeze(1)
# Backward: always take index 0 (the 'beginning' of the sequence for BWD)
bwd_last = l2_out[:, 0, self.hidden_size:]
combined = torch.cat([fwd_last, bwd_last], dim=-1)
return self.fc(combined)
In [10]:
def get_robust_maxlen(dataset, tokenizer, p=95, absolute_max=512):
# 1. Get the statistical length
lengths = [len(tokenizer(ex[0])) for ex in dataset]
p_length = np.percentile(lengths, p)
# 2. Find the next power of 2
# math.log2 gives the exponent, ceil rounds up, 2** raises it back
# We use max(16, ...) to ensure we don't get a tiny length like 4 or 8
robust_len = 2 ** math.ceil(math.log2(max(16, p_length)))
# 3. Cap at absolute_max
final_len = min(robust_len, absolute_max)
print(f"{p}th Percentile: {p_length:.2f}")
print(f"Nearest Power of 2: {robust_len}")
print(f"Final Robust MAX_LEN: {final_len}")
return final_len
# Usage
MAX_LEN = get_robust_maxlen(train_ds, tokenizer, p=95)
95th Percentile: 58.00 Nearest Power of 2: 64 Final Robust MAX_LEN: 64
In [11]:
DATASET_CONFIG, IS_BINARY, MODEL_OUTPUT_SIZE, VOCAB_SIZE, MAX_LEN,
Out[11]:
('ag_news', False, 4, 20002, 64)
In [13]:
set_all_seeds(RANDOM_SEED)
model = BiStackedGRUFromScratch(
vocab_size=VOCAB_SIZE,
embed_dim=EMBEDDING_DIM,
hidden_size=HIDDEN_DIM,
output_size=MODEL_OUTPUT_SIZE
).to(DEVICE)
model = torch.compile(model)
summary(model,
input_data=(torch.zeros((BATCH_SIZE, MAX_LEN), dtype=torch.long).to(DEVICE),
torch.ones((BATCH_SIZE,), dtype=torch.long).to(DEVICE) * MAX_LEN),
col_names=["input_size", "output_size", "num_params", "mult_adds"],
device=DEVICE)
W0318 04:11:16.068000 55 torch/_dynamo/convert_frame.py:1358] [5/8] torch._dynamo hit config.recompile_limit (8) W0318 04:11:16.068000 55 torch/_dynamo/convert_frame.py:1358] [5/8] function: 'hook' (/usr/local/lib/python3.12/dist-packages/torchinfo/torchinfo.py:592) W0318 04:11:16.068000 55 torch/_dynamo/convert_frame.py:1358] [5/8] last reason: 5/7: global_layer_info[132805340711104].macs == 0 # self.macs += self.output_size[0] * cur_params # torchinfo/layer_info.py:242 in calculate_macs W0318 04:11:16.068000 55 torch/_dynamo/convert_frame.py:1358] [5/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". W0318 04:11:16.068000 55 torch/_dynamo/convert_frame.py:1358] [5/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
Out[13]:
============================================================================================================================================ Layer (type:depth-idx) Input Shape Output Shape Param # Mult-Adds ============================================================================================================================================ OptimizedModule [128, 64] [128, 4] -- -- ├─BiStackedGRUFromScratch: 1-1 [128, 64] [128, 4] -- -- │ └─Embedding: 2-1 [128, 64] [128, 64, 128] 2,560,256 327,712,768 │ └─GRUCellFromScratch: 2-2 [128, 128] [128, 128] -- -- │ │ └─Linear: 3-1 [128, 256] [128, 256] 65,792 8,421,376 │ │ └─Linear: 3-2 [128, 256] [128, 128] 32,896 4,210,688 │ └─GRUCellFromScratch: 2-3 [128, 128] [128, 128] -- -- │ │ └─Linear: 3-3 [128, 256] [128, 256] 65,792 8,421,376 │ │ └─Linear: 3-4 [128, 256] [128, 128] 32,896 4,210,688 │ └─GRUCellFromScratch: 2-4 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-5 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-6 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-5 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-7 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-8 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-6 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-9 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-10 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-7 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-11 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-12 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-8 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-13 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-14 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-9 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-15 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-16 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-10 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-17 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-18 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-11 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-19 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-20 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-12 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-21 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-22 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-13 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-23 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-24 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-14 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-25 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-26 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-15 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-27 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-28 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-16 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-29 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-30 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-17 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-31 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-32 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-18 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-33 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-34 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-19 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-35 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-36 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-20 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-37 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-38 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-21 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-39 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-40 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-22 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-41 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-42 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-23 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-43 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-44 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-24 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-45 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-46 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-25 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-47 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-48 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-26 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-49 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-50 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-27 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-51 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-52 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-28 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-53 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-54 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-29 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-55 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-56 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-30 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-57 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-58 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-31 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-59 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-60 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-32 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-61 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-62 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-33 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-63 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-64 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-34 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-65 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-66 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-35 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-67 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-68 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-36 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-69 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-70 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-37 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-71 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-72 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-38 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-73 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-74 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-39 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-75 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-76 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-40 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-77 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-78 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-41 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-79 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-80 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-42 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-81 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-82 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-43 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-83 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-84 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-44 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-85 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-86 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-45 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-87 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-88 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-46 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-89 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-90 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-47 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-91 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-92 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-48 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-93 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-94 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-49 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-95 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-96 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-50 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-97 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-98 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-51 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-99 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-100 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-52 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-101 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-102 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-53 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-103 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-104 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-54 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-105 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-106 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-55 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-107 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-108 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-56 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-109 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-110 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-57 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-111 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-112 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-58 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-113 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-114 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-59 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-115 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-116 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-60 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-117 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-118 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-61 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-119 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-120 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-62 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-121 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-122 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-63 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-123 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-124 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-64 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-125 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-126 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-65 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-127 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-128 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-66 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-129 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-130 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-67 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-131 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-132 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-68 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-133 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-134 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-69 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-135 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-136 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-70 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-137 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-138 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-71 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-139 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-140 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-72 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-141 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-142 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-73 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-143 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-144 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-74 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-145 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-146 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-75 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-147 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-148 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-76 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-149 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-150 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-77 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-151 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-152 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-78 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-153 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-154 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-79 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-155 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-156 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-80 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-157 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-158 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-81 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-159 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-160 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-82 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-161 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-162 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-83 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-163 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-164 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-84 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-165 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-166 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-85 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-167 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-168 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-86 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-169 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-170 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-87 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-171 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-172 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-88 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-173 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-174 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-89 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-175 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-176 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-90 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-177 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-178 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-91 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-179 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-180 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-92 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-181 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-182 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-93 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-183 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-184 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-94 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-185 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-186 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-95 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-187 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-188 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-96 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-189 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-190 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-97 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-191 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-192 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-98 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-193 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-194 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-99 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-195 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-196 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-100 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-197 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-198 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-101 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-199 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-200 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-102 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-201 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-202 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-103 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-203 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-204 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-104 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-205 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-206 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-105 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-207 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-208 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-106 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-209 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-210 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-107 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-211 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-212 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-108 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-213 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-214 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-109 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-215 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-216 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-110 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-217 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-218 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-111 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-219 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-220 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-112 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-221 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-222 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-113 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-223 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-224 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-114 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-225 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-226 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-115 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-227 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-228 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-116 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-229 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-230 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-117 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-231 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-232 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-118 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-233 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-234 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-119 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-235 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-236 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-120 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-237 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-238 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-121 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-239 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-240 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-122 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-241 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-242 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-123 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-243 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-244 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-124 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-245 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-246 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-125 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-247 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-248 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-126 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-249 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-250 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-127 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-251 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-252 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-128 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-253 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-254 [128, 256] [128, 128] (recursive) 4,210,688 │ └─GRUCellFromScratch: 2-129 [128, 128] [128, 128] (recursive) -- │ │ └─Linear: 3-255 [128, 256] [128, 256] (recursive) 8,421,376 │ │ └─Linear: 3-256 [128, 256] [128, 128] (recursive) 4,210,688 │ └─Dropout: 2-130 [128, 64, 256] [128, 64, 256] -- -- │ └─GRUCellFromScratch: 2-131 [128, 256] [128, 128] -- -- │ │ └─Linear: 3-257 [128, 384] [128, 256] 98,560 12,615,680 │ │ └─Linear: 3-258 [128, 384] [128, 128] 49,280 6,307,840 │ └─GRUCellFromScratch: 2-132 [128, 256] [128, 128] -- -- │ │ └─Linear: 3-259 [128, 384] [128, 256] 98,560 12,615,680 │ │ └─Linear: 3-260 [128, 384] [128, 128] 49,280 6,307,840 │ └─GRUCellFromScratch: 2-133 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-261 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-262 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-134 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-263 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-264 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-135 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-265 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-266 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-136 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-267 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-268 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-137 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-269 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-270 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-138 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-271 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-272 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-139 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-273 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-274 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-140 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-275 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-276 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-141 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-277 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-278 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-142 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-279 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-280 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-143 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-281 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-282 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-144 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-283 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-284 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-145 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-285 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-286 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-146 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-287 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-288 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-147 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-289 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-290 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-148 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-291 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-292 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-149 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-293 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-294 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-150 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-295 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-296 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-151 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-297 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-298 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-152 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-299 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-300 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-153 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-301 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-302 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-154 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-303 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-304 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-155 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-305 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-306 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-156 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-307 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-308 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-157 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-309 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-310 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-158 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-311 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-312 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-159 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-313 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-314 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-160 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-315 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-316 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-161 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-317 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-318 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-162 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-319 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-320 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-163 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-321 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-322 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-164 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-323 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-324 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-165 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-325 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-326 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-166 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-327 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-328 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-167 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-329 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-330 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-168 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-331 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-332 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-169 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-333 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-334 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-170 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-335 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-336 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-171 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-337 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-338 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-172 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-339 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-340 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-173 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-341 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-342 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-174 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-343 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-344 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-175 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-345 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-346 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-176 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-347 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-348 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-177 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-349 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-350 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-178 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-351 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-352 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-179 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-353 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-354 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-180 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-355 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-356 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-181 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-357 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-358 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-182 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-359 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-360 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-183 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-361 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-362 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-184 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-363 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-364 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-185 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-365 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-366 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-186 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-367 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-368 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-187 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-369 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-370 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-188 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-371 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-372 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-189 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-373 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-374 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-190 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-375 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-376 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-191 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-377 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-378 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-192 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-379 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-380 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-193 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-381 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-382 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-194 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-383 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-384 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-195 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-385 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-386 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-196 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-387 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-388 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-197 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-389 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-390 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-198 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-391 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-392 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-199 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-393 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-394 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-200 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-395 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-396 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-201 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-397 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-398 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-202 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-399 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-400 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-203 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-401 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-402 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-204 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-403 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-404 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-205 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-405 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-406 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-206 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-407 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-408 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-207 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-409 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-410 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-208 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-411 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-412 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-209 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-413 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-414 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-210 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-415 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-416 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-211 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-417 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-418 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-212 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-419 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-420 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-213 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-421 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-422 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-214 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-423 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-424 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-215 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-425 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-426 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-216 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-427 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-428 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-217 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-429 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-430 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-218 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-431 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-432 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-219 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-433 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-434 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-220 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-435 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-436 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-221 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-437 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-438 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-222 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-439 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-440 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-223 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-441 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-442 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-224 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-443 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-444 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-225 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-445 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-446 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-226 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-447 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-448 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-227 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-449 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-450 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-228 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-451 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-452 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-229 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-453 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-454 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-230 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-455 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-456 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-231 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-457 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-458 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-232 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-459 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-460 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-233 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-461 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-462 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-234 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-463 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-464 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-235 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-465 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-466 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-236 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-467 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-468 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-237 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-469 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-470 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-238 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-471 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-472 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-239 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-473 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-474 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-240 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-475 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-476 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-241 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-477 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-478 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-242 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-479 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-480 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-243 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-481 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-482 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-244 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-483 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-484 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-245 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-485 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-486 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-246 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-487 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-488 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-247 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-489 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-490 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-248 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-491 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-492 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-249 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-493 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-494 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-250 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-495 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-496 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-251 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-497 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-498 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-252 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-499 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-500 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-253 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-501 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-502 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-254 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-503 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-504 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-255 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-505 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-506 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-256 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-507 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-508 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-257 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-509 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-510 [128, 384] [128, 128] (recursive) 6,307,840 │ └─GRUCellFromScratch: 2-258 [128, 256] [128, 128] (recursive) -- │ │ └─Linear: 3-511 [128, 384] [128, 256] (recursive) 12,615,680 │ │ └─Linear: 3-512 [128, 384] [128, 128] (recursive) 6,307,840 │ └─Linear: 2-259 [128, 256] [128, 4] 1,028 131,584 ============================================================================================================================================ Total params: 3,054,340 Trainable params: 3,054,340 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 4.37 ============================================================================================================================================ Input size (MB): 0.07 Forward/backward pass size (MB): 109.06 Params size (MB): 12.22 Estimated Total Size (MB): 121.34 ============================================================================================================================================
In [14]:
def count_parameters(model):
total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
embedding_params = sum(p.numel() for p in model.embedding.parameters())
gru_params = total_params - embedding_params - sum(p.numel() for p in model.fc.parameters())
print(f"{'Component':<15} | {'Parameters':<12}")
print("-" * 30)
print(f"{'Embeddings':<15} | {embedding_params:,}")
print(f"{'Stacked GRU':<15} | {gru_params:,}")
print(f"{'Classifier':<15} | {sum(p.numel() for p in model.fc.parameters()):,}")
print("-" * 30)
print(f"{'TOTAL':<15} | {total_params:,}")
count_parameters(model)
Component | Parameters ------------------------------ Embeddings | 2,560,256 Stacked GRU | 493,056 Classifier | 1,028 ------------------------------ TOTAL | 3,054,340
In [19]:
def evaluate_classification(model, data_loader, device, is_binary=False, class_names=None, plot=False):
model.eval()
all_preds = []
all_labels = []
total_loss = 0
# Selection of loss function
criterion = torch.nn.CrossEntropyLoss() if not is_binary else torch.nn.BCEWithLogitsLoss()
with torch.no_grad():
for inputs, labels, lengths in data_loader:
# FIX 1: Move lengths to device for masking logic
inputs, labels, lengths = inputs.to(device), labels.to(device), lengths.to(device)
logits = model(inputs, lengths)
# FIX 2: Handle shape for Binary Classification
if is_binary:
loss = criterion(logits.squeeze(), labels.float())
preds = (torch.sigmoid(logits.squeeze()) > 0.5).int()
else:
loss = criterion(logits, labels.long())
preds = torch.argmax(logits, dim=1)
# Use inputs.size(0) to weigh the loss correctly by batch size
total_loss += loss.item() * inputs.size(0)
all_preds.extend(preds.cpu().numpy())
all_labels.extend(labels.cpu().numpy())
# Calculate metrics
avg_loss = total_loss / len(data_loader.dataset)
acc = accuracy_score(all_labels, all_preds)
f1 = f1_score(all_labels, all_preds, average='binary' if is_binary else 'macro')
# Only print and plot if we are doing a final audit, not every epoch
if plot:
print(f"Loss: {avg_loss:.4f} | Accuracy: {acc:.2%} | F1: {f1:.2%}")
print("\nClassification Report:\n", classification_report(all_labels, all_preds, target_names=class_names))
cm = confusion_matrix(all_labels, all_preds)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=class_names, yticklabels=class_names)
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()
return avg_loss, acc, f1
In [24]:
def train_model(model, train_loader, val_loader, optimizer, criterion, scheduler, num_epochs, device,
is_binary=False, patience=5):
# --- 1. PERFORMANCE & EARLY STOPPING INIT ---
scaler = torch.amp.GradScaler('cuda')
start_time = time.time()
best_val_f1 = 0.0
epochs_no_improve = 0 # Counter for Early Stopping
history = {
'train_loss': [], 'val_loss': [],
'train_acc': [], 'val_acc': [],
'val_f1': [], 'lrs': []
}
print(f"Starting training on {device} (Patience: {patience})...")
for epoch in range(num_epochs):
# --- 2. TRAINING PHASE ---
model.train()
train_running_loss, train_correct, train_total = 0.0, 0, 0
for inputs, labels, lengths in train_loader:
inputs = inputs.to(device, non_blocking=True)
labels = labels.to(device, non_blocking=True)
lengths = lengths.to(device)
optimizer.zero_grad(set_to_none=True)
with torch.amp.autocast('cuda'):
logits = model(inputs, lengths)
loss = criterion(logits.squeeze(), labels.float() if is_binary else labels.long())
scaler.scale(loss).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(optimizer)
scaler.update()
train_running_loss += loss.item() * inputs.size(0)
preds = (torch.sigmoid(logits.squeeze()) > 0.5).long() if is_binary else torch.argmax(logits, dim=1)
train_correct += (preds == labels).sum().item()
train_total += labels.size(0)
# --- 3. VALIDATION PHASE ---
epoch_train_loss = train_running_loss / train_total
epoch_train_acc = train_correct / train_total
epoch_val_loss, epoch_val_acc, epoch_val_f1 = evaluate_classification(
model, val_loader, device, is_binary=is_binary, plot=False
)
# --- 4. EARLY STOPPING & CHECKPOINT LOGIC ---
if epoch_val_f1 > best_val_f1:
best_val_f1 = epoch_val_f1
torch.save(model.state_dict(), 'best_model.pth')
epochs_no_improve = 0
status_msg = " * (Saved Best)"
else:
epochs_no_improve += 1
status_msg = f" (No improve: {epochs_no_improve}/{patience})"
# --- 5. LOGGING ---
scheduler.step()
current_lr = optimizer.param_groups[0]['lr']
for k, v in zip(history.keys(), [epoch_train_loss, epoch_val_loss, epoch_train_acc, epoch_val_acc, epoch_val_f1, current_lr]):
history[k].append(v)
print(f'Epoch {epoch + 1:03d}/{num_epochs:03d} | '
f'Acc (T/V): {epoch_train_acc:.2%} / {epoch_val_acc:.2%} | '
f'F1 (V): {epoch_val_f1:.2%} | '
f'Loss (T/V): {epoch_train_loss:.4f} / {epoch_val_loss:.4f} | '
f'LR: {current_lr:.6f}{status_msg}')
# Trigger Early Stopping
if epochs_no_improve >= patience:
print(f"\nEarly stopping triggered after {epoch + 1} epochs.")
break
total_time = (time.time() - start_time) / 60
print(f'\nDone! Total Time: {total_time:.2f} min | Best Val F1: {best_val_f1:.2%}')
# Reload the best weights
model.load_state_dict(torch.load('best_model.pth'))
return history
In [25]:
def plot_history(history):
epochs = range(1, len(history['train_loss']) + 1)
# Check for all possible metrics
has_acc = 'val_acc' in history
has_f1 = 'val_f1' in history
has_gen = 'val_ppl' in history
# Universal plots: Loss + Learning Rate = 2
num_plots = 2 + int(has_acc) + int(has_f1) + int(has_gen)
plt.figure(figsize=(num_plots * 5, 5))
# --- Plot 1: Loss (Universal) ---
plt.subplot(1, num_plots, 1)
plt.plot(epochs, history['train_loss'], 'o-', label='Train Loss', color='#1f77b4')
plt.plot(epochs, history['val_loss'], 's--', label='Val Loss', color='#ff7f0e')
plt.title('Cross Entropy Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.grid(True, alpha=0.3)
plot_idx = 2
# --- Plot 2: Accuracy (%) ---
if has_acc:
plt.subplot(1, num_plots, plot_idx)
plt.plot(epochs, np.array(history['train_acc']) * 100, 'o-', label='Train Acc', color='#1f77b4')
plt.plot(epochs, np.array(history['val_acc']) * 100, 's--', label='Val Acc', color='#ff7f0e')
plt.title('Accuracy (%)')
plt.xlabel('Epochs')
plt.ylabel('Percentage')
plt.legend()
plt.grid(True, alpha=0.3)
plot_idx += 1
# --- Plot 3: F1 Score (%) ---
if has_f1:
plt.subplot(1, num_plots, plot_idx)
plt.plot(epochs, np.array(history['val_f1']) * 100, 'd-', label='Val F1', color='#2ca02c')
plt.title('F1 Score (%)')
plt.xlabel('Epochs')
plt.ylabel('Percentage')
plt.legend()
plt.grid(True, alpha=0.3)
plot_idx += 1
# --- Plot 4: Generative Metrics (Dual Axis) ---
if has_gen:
plt.subplot(1, num_plots, plot_idx)
ax1 = plt.gca()
# Perplexity (Purple)
ax1.plot(epochs, history['val_ppl'], 'o-', label='PPL', color='purple')
ax1.set_ylabel('Perplexity', color='purple', fontweight='bold')
ax1.tick_params(axis='y', labelcolor='purple')
if 'val_bpc' in history:
ax2 = ax1.twinx()
# BPC (Red)
ax2.plot(epochs, history['val_bpc'], 'x--', label='BPC', color='red')
ax2.set_ylabel('Bits Per Character', color='red', fontweight='bold')
ax2.tick_params(axis='y', labelcolor='red')
ax2.grid(False)
plt.title('Generative Metrics')
ax1.set_xlabel('Epochs')
ax1.grid(True, alpha=0.3)
plot_idx += 1
# --- Plot 5: Learning Rate (Universal) ---
plt.subplot(1, num_plots, num_plots)
plt.plot(epochs, history['lrs'], label='LR', color='black', linewidth=2)
plt.title('LR Schedule')
plt.xlabel('Epochs')
plt.ylabel('Rate')
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
In [26]:
DATASET_CONFIG, IS_BINARY, MODEL_OUTPUT_SIZE, VOCAB_SIZE, MAX_LEN,
Out[26]:
('ag_news', False, 4, 20002, 64)
In [27]:
# 1. Setup
criterion = nn.BCEWithLogitsLoss() if IS_BINARY else nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=1e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=NUM_EPOCHS)
# 2. Train
NUM_EPOCHS = 25
history = train_model(model, train_loader,
val_loader, optimizer,
criterion, scheduler,
NUM_EPOCHS, DEVICE,
is_binary=IS_BINARY, patience=5)
Starting training on cuda (Patience: 5)... Epoch 001/025 | Acc (T/V): 86.83% / 85.08% | F1 (V): 84.99% | Loss (T/V): 0.3791 / 0.4265 | LR: 0.000498 * (Saved Best) Epoch 002/025 | Acc (T/V): 87.74% / 87.55% | F1 (V): 87.62% | Loss (T/V): 0.3535 / 0.3662 | LR: 0.000492 * (Saved Best) Epoch 003/025 | Acc (T/V): 89.52% / 89.24% | F1 (V): 89.27% | Loss (T/V): 0.3067 / 0.3171 | LR: 0.000482 * (Saved Best) Epoch 004/025 | Acc (T/V): 91.33% / 90.46% | F1 (V): 90.43% | Loss (T/V): 0.2575 / 0.2979 | LR: 0.000469 * (Saved Best) Epoch 005/025 | Acc (T/V): 92.30% / 89.34% | F1 (V): 89.36% | Loss (T/V): 0.2283 / 0.3241 | LR: 0.000452 (No improve: 1/5) Epoch 006/025 | Acc (T/V): 93.06% / 91.30% | F1 (V): 91.29% | Loss (T/V): 0.2048 / 0.2690 | LR: 0.000432 * (Saved Best) Epoch 007/025 | Acc (T/V): 93.58% / 90.51% | F1 (V): 90.48% | Loss (T/V): 0.1870 / 0.3008 | LR: 0.000409 (No improve: 1/5) Epoch 008/025 | Acc (T/V): 93.93% / 90.30% | F1 (V): 90.32% | Loss (T/V): 0.1755 / 0.3004 | LR: 0.000384 (No improve: 2/5) Epoch 009/025 | Acc (T/V): 94.38% / 91.06% | F1 (V): 91.04% | Loss (T/V): 0.1646 / 0.2782 | LR: 0.000356 (No improve: 3/5) Epoch 010/025 | Acc (T/V): 94.67% / 90.84% | F1 (V): 90.85% | Loss (T/V): 0.1547 / 0.2944 | LR: 0.000327 (No improve: 4/5) Epoch 011/025 | Acc (T/V): 94.87% / 90.64% | F1 (V): 90.65% | Loss (T/V): 0.1483 / 0.3018 | LR: 0.000297 (No improve: 5/5) Early stopping triggered after 11 epochs. Done! Total Time: 38.29 min | Best Val F1: 91.29%
In [28]:
# Visualize & Audit
plot_history(history)
In [29]:
# --- Final Evaluation (The "Audit") ---
print("\n" + "="*50)
print(" FINAL TEST SET AUDIT ")
print("="*50)
# We set plot=True here to get the Heatmap and Classification Report
test_loss, test_acc, test_f1 = evaluate_classification(
model,
test_loader,
DEVICE,
is_binary=IS_BINARY,
class_names=class_names,
plot=True
)
print("="*100)
print(f'\nFinal Summary | Loss: {test_loss:.4f} | Accuracy: {test_acc:.2%} | F1-score: {test_f1:.2%}')
print("="*100)
==================================================
FINAL TEST SET AUDIT
==================================================
Loss: 0.2691 | Accuracy: 91.41% | F1: 91.41%
Classification Report:
precision recall f1-score support
World 0.92 0.90 0.91 1900
Sports 0.96 0.97 0.96 1900
Business 0.90 0.87 0.89 1900
Sci/Tech 0.87 0.91 0.89 1900
accuracy 0.91 7600
macro avg 0.91 0.91 0.91 7600
weighted avg 0.91 0.91 0.91 7600
==================================================================================================== Final Summary | Loss: 0.2691 | Accuracy: 91.41% | F1-score: 91.41% ====================================================================================================
In [30]:
tokenizer
Out[30]:
<function __main__.<lambda>(text)>
In [31]:
def predict_robust(model, sentence, vocab, tokenizer, class_names, is_binary=False):
model.eval()
# Step A: Tokenize exactly like training
tokens = tokenizer(sentence)
# Step B: Numericalize (Default to <UNK> index 1)
indexed = [vocab.get(t, 1) for t in tokens]
# Safety: Handle empty or purely non-alphanumeric input
if len(indexed) == 0:
return "N/A", 0.0
# Step C: Prepare Tensors [Batch Size 1, Seq_Len]
tensor = torch.LongTensor(indexed).unsqueeze(0).to(DEVICE)
length = torch.LongTensor([len(indexed)]).to(DEVICE)
with torch.no_grad():
logits = model(tensor, length)
if is_binary:
# --- BINARY MODE (IMDB) ---
# Flattens [1, 1] to [1] and squashes to 0-1
prob = torch.sigmoid(logits.view(-1)).item()
prediction_idx = 1 if prob > 0.5 else 0
# Confidence is distance from the 0.5 decision boundary
confidence = prob if prediction_idx == 1 else 1 - prob
else:
# --- MULTI-CLASS MODE (AG News, etc.) ---
# Converts logits to a probability distribution across N classes
probs = torch.softmax(logits, dim=1)
prediction_idx = torch.argmax(probs, dim=1).item()
confidence = probs[0][prediction_idx].item()
# Map the index back to the human-readable string from metadata
label = class_names[prediction_idx]
return label, confidence
# ================================================================================
def run_test_suite(model, tests, vocab, tokenizer, class_names, is_binary):
"""
Runs a batch of tests and prints a professional, formatted report.
"""
print(f"{'Input Sentence (First 50 chars)':<55} | {'Prediction':<12} | {'Conf'}")
print("-" * 80)
for text in tests:
# Our universal prediction function
label, conf = predict_robust(
model, text, vocab, tokenizer, class_names, is_binary
)
# Simple threshold for "High Confidence" (e.g., 80%)
# In a notebook, this makes identifying weaknesses easy
marker = "★" if conf > 0.8 else " "
print(f"{text[:50]:<55} | {label:<12} | {conf:>6.1%} {marker}")
In [32]:
# 1. Define the test cases
ag_news_tests = [
"The Federal Reserve is expected to keep interest rates steady as inflation cools.", # Business
"The star striker scored a hat-trick in the final minutes of the world cup qualifier.",# Sports
"Astronomers have discovered a new Earth-like planet orbiting a distant star system.",# Sci/Tech
"Protests erupted in the capital city following the controversial election results.", # World
"New software update fixes critical security vulnerabilities in mobile operating systems.", # Sci/Tech
"Wall Street closes higher as tech stocks rally despite global trade tensions.", # Business
"NASA's rover successfully collected soil samples from the Martian surface.", # Sci/Tech
"The prime minister announced a new coalition government after weeks of deadlock.", # World
"The underdog team secured a historic victory in the grand slam final today.", # Sports
"A massive cyberattack has compromised thousands of corporate database records." # Sci/Tech
]
# 2. Run the Robust Test Suite
print(f"{'Headline (First 58 chars)':<60} | {'Prediction':<12} | {'Confidence'}")
print("-" * 90)
for text in ag_news_tests:
# Use the universal predict_robust function we built
topic, conf = predict_robust(
model, text, vocab, tokenizer, class_names, is_binary=IS_BINARY
)
# Visual confidence indicator (Pro touch)
marker = "★" if conf > 0.85 else " "
# Truncate text for a clean table view
print(f"{text[:58]:<60} | {topic:<12} | {conf:>6.1%} {marker}")
Headline (First 58 chars) | Prediction | Confidence ------------------------------------------------------------------------------------------ The Federal Reserve is expected to keep interest rates ste | Business | 94.8% ★ The star striker scored a hat-trick in the final minutes o | Sports | 99.4% ★ Astronomers have discovered a new Earth-like planet orbiti | Sci/Tech | 97.0% ★ Protests erupted in the capital city following the controv | World | 92.0% ★ New software update fixes critical security vulnerabilitie | Sci/Tech | 96.4% ★ Wall Street closes higher as tech stocks rally despite glo | Business | 91.5% ★ NASA's rover successfully collected soil samples from the | Sci/Tech | 98.3% ★ The prime minister announced a new coalition government af | World | 85.9% ★ The underdog team secured a historic victory in the grand | Sports | 92.5% ★ A massive cyberattack has compromised thousands of corpora | Sci/Tech | 75.2%
In [33]:
%watermark -iv
datasets : 4.0.0 matplotlib : 3.10.0 numpy : 2.0.2 pandas : 2.3.3 re : 2.2.1 seaborn : 0.13.2 sklearn : 1.6.1 torch : 2.9.0+cu126 torchinfo : 1.8.0 torchvision: 0.24.0+cu126