Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [1]:
%%capture
!pip install watermark
!pip install torchinfo datasets
In [2]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 torch: 2.10.0+cu128
Gated Recurrent Unit (GRU) Classifier on IMDB Reviews¶
The network in this notebook is an implementation of a Gated Recurrent Unit (GRU) network [1] on IMDB reviews dataset [8]. The model is a Single-Layer GRU implemented from scratch. It utilizes a streamlined gating structure to maintain long-term dependencies without the complexity of a separate cell state, making it a robust and efficient alternative to the LSTM.
Network Architecture:
The GRU was introduced as a simpler, more computationally efficient evolution of the LSTM [1]. While the LSTM uses three gates and a dual-state system (Hidden + Cell), the GRU simplifies this into two gates—Update and Reset—and a single Hidden State ($h_t$). The GRU simplifies the LSTM by merging the cell and hidden states and reducing the gate count to two.
The repeating module in a standard RNN contains a single layer. (Source, [6])
The repeating module in an LSTM contains four interacting layers. (Source, [6])
GRU Unit Cell (Source, [6])
Key Concepts of the GRU Architecture:¶
- Unified Hidden State ($h_t$): Unlike the LSTM, the GRU does not have a separate "cell state." Instead, the hidden state acts as both the working memory and the long-term memory track. This reduces the memory footprint and simplifies the implementation.
- Gating Mechanisms: GRUs use two specialized gates to control the flow of information:
- Update Gate ($z_t$): Determines how much of the previous hidden state should be retained and how much of the new candidate state should be mixed in. It effectively combines the "Forget" and "Input" functions of an LSTM.
- Reset Gate ($r_t$): Controls how much of the past information to "forget" before calculating the new candidate hidden state.
- Linear Interpolation: The final hidden state is a weighted average (interpolation) between the previous state and the new candidate. This allows the gradient to persist through many time steps, effectively solving the vanishing gradient problem.
Improvements of GRU over LSTM and Simple RNN:¶
- Efficiency and Speed: With only two gates instead of three (and fewer matrix multiplications), the GRU has approximately 33% fewer parameters than an LSTM. This makes it faster to train and less prone to overfitting on smaller datasets.
- Stable Gradients: Similar to the LSTM, the GRU uses additive updates. This prevents the gradients from shrinking exponentially, allowing the model to learn context across long news articles.
- Performance Parity: On many NLP tasks, such as topic classification (AG News) or sentiment analysis (IMDB), the GRU performs as well as, or sometimes better than, the LSTM while being more computationally "robust" for resource-constrained environments.
Quick Comparison Table: RNN vs. LSTM vs. GRU¶
| Feature | Simple RNN | LSTM | Gated Recurrent Unit (GRU) |
|---|---|---|---|
| State Count | Single ($h_t$) | Dual ($h_t, c_t$) | Single ($h_t$) |
| Gate Count | 0 | 3 (Forget, Input, Output) | 2 (Update, Reset) |
| Complexity | Low | High | Medium (Efficient) |
| Memory Usage | Lowest | Highest | Moderate (Optimized) |
| Gradient Flow | Multiplicative | Additive | Additive (Interpolation) |
| Implementation | Simple | Complex | Streamlined |
| Parameter Count | $W$ | $\approx 4 \times W$ | $\approx 3 \times W$ |
REFERENCES:
- Cho, Kyunghyun, et al. "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." arXiv, 2014. (Original GRU paper)
- Hochreiter, Sepp, and Jürgen Schmidhuber. "Long Short-Term Memory." 1997.
- Elman, Jeffrey L. "Finding Structure in Time." 1990.
- PyTorch. "Recurrent Layers: nn.GRU." 2026.
- Zhang, Aston, et al. "Gated Recurrent Units (Chapter 10.2)." Dive into Deep Learning. 2023.
- Olah, Christopher. "Understanding LSTM Networks." colah's blog, 2015.
- Karpathy, Andrej. "The Unreasonable Effectiveness of Recurrent Neural Networks." Andrej Karpathy blog, 2015.
- Maas, Andrew L., et al. "Learning Word Vectors for Sentiment Analysis." Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
- Google. (2026). "Gemini 3 Flash" [Large language model]. Google AI
In [3]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from sklearn.model_selection import train_test_split
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence
from datasets import load_dataset, DatasetDict
from collections import Counter
# from torch.cuda.amp import GradScaler, autocast
from torch.amp import GradScaler, autocast
from torchinfo import summary
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
import re
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
In [4]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
In [5]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 123
BATCH_SIZE = 128
LEARNING_RATE = 0.001
NUM_EPOCHS = 20
NUM_CLASSES = 1
VOCABULARY_SIZE = 20000
EMBEDDING_DIM = 128
HIDDEN_DIM = 256
# OUTPUT_DIM = 1
# Set device
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(DEVICE)
cuda
In [6]:
# 1. Load Dataset (IMDB)
raw_datasets = load_dataset("imdb")
# 2. Split 10% of 'train' to create a 'validation' set (seed for reproducibility)
train_val_split = raw_datasets["train"].train_test_split(test_size=0.1, seed=RANDOM_SEED)
ds_splits = DatasetDict({
'train': train_val_split['train'],
'validation': train_val_split['test'], # The 10% we just carved out
'test': raw_datasets['test']
})
print(ds_splits)
# 2. Build Vocabulary (From Scratch Logic)
tokenizer = lambda text: re.findall(r'\w+', text.lower())
word_counts = Counter()
# Important: We only iterate over ds_splits['train']
for text in ds_splits['train']['text']:
word_counts.update(tokenizer(text))
# Filter rare words and create mapping: : <PAD>=0, <UNK>=1, then top 20k words
vocab = {word: i + 2 for i, (word, count) in enumerate(word_counts.most_common(VOCABULARY_SIZE))}
vocab['<PAD>'] = 0
vocab['<UNK>'] = 1
print(f"Vocab size: {len(vocab)}")
# 3. Custom Dataset Class
class TextDataset(torch.utils.data.Dataset):
def __init__(self, dataset_split):
self.data = dataset_split
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
# Return raw text and label. Let collate_fn handle the rest.
item = self.data[idx]
return item['text'], item['label']
# 4. Robust Text Pipeline
def text_pipeline(text, max_len=128):
tokens = tokenizer(text)
# Use .get(t, 1) because 1 is your <UNK> index
return [vocab.get(t, 1) for t in tokens][:max_len]
# 5. Robust Collate Batch
def collate_batch(batch):
label_list, text_list, lengths = [], [], []
for _text, _label in batch:
# 1. Labels: Use LONG for AG News (Multi-class)
label_list.append(_label)
# 2. Numericalize and Truncate
processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)
text_list.append(processed_text)
lengths.append(processed_text.size(0))
# 3. Padding logic
# We use <PAD> index 0. We use vocab.get just in case names change.
pad_idx = vocab.get('<PAD>', 0)
# This dynamically pads to the longest sequence in THIS batch (more efficient)
text_list_padded = pad_sequence(text_list, batch_first=True, padding_value=pad_idx)
label_tensor = torch.tensor(label_list, dtype=torch.long)
length_tensor = torch.tensor(lengths, dtype=torch.int64)
return text_list_padded.to(DEVICE), label_tensor.to(DEVICE), length_tensor.to(DEVICE)
# 6. Initialize DataLoaders
train_ds = TextDataset(ds_splits['train'])
val_ds = TextDataset(ds_splits['validation'])
test_ds = TextDataset(ds_splits['test'])
train_loader = DataLoader(train_ds, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch)
val_loader = DataLoader(val_ds, batch_size=BATCH_SIZE, shuffle=False, collate_fn=collate_batch)
test_loader = DataLoader(test_ds, batch_size=BATCH_SIZE, shuffle=False, collate_fn=collate_batch)
DatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 22500
})
validation: Dataset({
features: ['text', 'label'],
num_rows: 2500
})
test: Dataset({
features: ['text', 'label'],
num_rows: 25000
})
})
Vocab size: 20002
How do we figure out a good max_len? see code below
def get_robust_maxlen(dataset, tokenizer, p=95, absolute_max=512):
# 1. Get the statistical length
lengths = [len(tokenizer(ex[0])) for ex in dataset]
p_length = np.percentile(lengths, p)
# 2. Find the next power of 2
# math.log2 gives the exponent, ceil rounds up, 2** raises it back
# We use max(16, ...) to ensure we don't get a tiny length like 4 or 8
robust_len = 2 ** math.ceil(math.log2(max(16, p_length)))
# 3. Cap at absolute_max
final_len = min(robust_len, absolute_max)
print(f"{p}th Percentile: {p_length:.2f}")
print(f"Nearest Power of 2: {robust_len}")
print(f"Final Robust MAX_LEN: {final_len}")
return final_len
# Usage
__MAX_LEN = get_robust_maxlen(train_ds, tokenizer, p=95)
In [7]:
class GRUCellFromScratch(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
# We need weights for 3 operations:
# Update gate (z), Reset gate (r), and Candidate hidden state (h_tilde)
self.gate_weights = nn.Linear(input_size + hidden_size, 2 * hidden_size)
self.candidate_weights = nn.Linear(input_size + hidden_size, hidden_size)
def forward(self, x, h_prev):
# x: [batch, input_size]
# h_prev: [batch, hidden_size]
# 1. Calculate Update (z) and Reset (r) gates
combined = torch.cat((x, h_prev), dim=1)
gates = torch.sigmoid(self.gate_weights(combined))
z_gate, r_gate = gates.chunk(2, dim=1)
# 2. Calculate Candidate Hidden State (h_tilde)
# We use the Reset gate to "mask" the previous hidden state
combined_candidate = torch.cat((x, r_gate * h_prev), dim=1)
h_tilde = torch.tanh(self.candidate_weights(combined_candidate))
# 3. Final Hidden State update
# This is a linear interpolation between past and candidate
h_curr = (1 - z_gate) * h_prev + z_gate * h_tilde
return h_curr
In [8]:
class GRUFromScratch(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden_size, output_size):
super().__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.cell = GRUCellFromScratch(embed_dim, hidden_size)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x, lengths):
batch_size, seq_len = x.shape
x = self.embedding(x) # [batch, seq, embed]
# Initialize h_0 as zeros
h = torch.zeros(batch_size, self.hidden_size).to(x.device)
# To handle padding robustly, we store ALL hidden states
all_h = []
for t in range(seq_len):
h = self.cell(x[:, t, :], h)
all_h.append(h.unsqueeze(1)) # [Batch, 1, Hidden]
# Stack them: [Batch, Seq_Len, Hidden]
all_h = torch.cat(all_h, dim=1)
# ROBUST TRICK: Select the hidden state at the ACTUAL end of each sentence
# We subtract 1 because lengths are 1-indexed, but indices are 0-indexed
idx = (lengths - 1).view(-1, 1, 1).expand(batch_size, 1, self.hidden_size) # [batch, 1, hidden]
last_h = all_h.gather(1, idx).squeeze(1) # [batch, hidden]
return self.fc(last_h)
In [9]:
VOCAB_SIZE = len(vocab)
MAX_LEN = 256
In [10]:
# --- Configuration Switch ---
DATASET_NAME = "IMDB"
if DATASET_NAME == "IMDB":
IS_BINARY = True
NUM_CLASSES = 1
# criterion = nn.BCEWithLogitsLoss()
else:
IS_BINARY = False
NUM_CLASSES = 4 # AG News has 4 classes
# criterion = nn.CrossEntropyLoss()
In [11]:
set_all_seeds(RANDOM_SEED)
model = GRUFromScratch(
vocab_size=VOCAB_SIZE,
embed_dim=EMBEDDING_DIM,
hidden_size=HIDDEN_DIM,
output_size=NUM_CLASSES
).to(DEVICE)
summary(model,
input_data=(torch.zeros((BATCH_SIZE, MAX_LEN), dtype=torch.long).to(DEVICE),
torch.ones((BATCH_SIZE,), dtype=torch.long).to(DEVICE) * MAX_LEN),
col_names=["input_size", "output_size", "num_params", "mult_adds"],
device=DEVICE)
Out[11]:
============================================================================================================================================ Layer (type:depth-idx) Input Shape Output Shape Param # Mult-Adds ============================================================================================================================================ GRUFromScratch [128, 256] [128, 1] -- -- ├─Embedding: 1-1 [128, 256] [128, 256, 128] 2,560,256 327,712,768 ├─GRUCellFromScratch: 1-2 [128, 128] [128, 256] -- -- │ └─Linear: 2-1 [128, 384] [128, 512] 197,120 25,231,360 │ └─Linear: 2-2 [128, 384] [128, 256] 98,560 12,615,680 ├─GRUCellFromScratch: 1-3 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-3 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-4 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-4 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-5 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-6 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-5 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-7 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-8 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-6 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-9 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-10 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-7 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-11 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-12 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-8 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-13 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-14 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-9 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-15 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-16 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-10 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-17 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-18 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-11 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-19 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-20 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-12 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-21 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-22 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-13 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-23 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-24 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-14 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-25 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-26 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-15 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-27 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-28 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-16 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-29 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-30 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-17 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-31 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-32 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-18 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-33 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-34 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-19 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-35 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-36 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-20 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-37 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-38 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-21 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-39 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-40 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-22 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-41 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-42 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-23 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-43 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-44 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-24 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-45 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-46 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-25 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-47 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-48 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-26 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-49 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-50 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-27 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-51 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-52 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-28 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-53 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-54 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-29 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-55 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-56 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-30 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-57 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-58 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-31 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-59 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-60 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-32 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-61 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-62 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-33 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-63 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-64 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-34 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-65 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-66 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-35 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-67 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-68 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-36 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-69 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-70 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-37 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-71 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-72 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-38 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-73 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-74 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-39 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-75 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-76 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-40 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-77 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-78 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-41 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-79 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-80 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-42 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-81 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-82 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-43 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-83 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-84 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-44 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-85 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-86 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-45 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-87 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-88 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-46 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-89 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-90 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-47 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-91 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-92 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-48 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-93 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-94 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-49 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-95 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-96 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-50 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-97 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-98 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-51 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-99 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-100 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-52 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-101 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-102 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-53 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-103 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-104 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-54 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-105 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-106 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-55 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-107 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-108 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-56 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-109 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-110 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-57 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-111 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-112 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-58 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-113 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-114 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-59 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-115 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-116 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-60 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-117 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-118 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-61 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-119 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-120 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-62 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-121 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-122 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-63 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-123 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-124 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-64 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-125 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-126 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-65 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-127 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-128 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-66 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-129 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-130 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-67 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-131 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-132 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-68 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-133 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-134 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-69 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-135 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-136 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-70 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-137 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-138 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-71 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-139 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-140 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-72 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-141 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-142 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-73 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-143 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-144 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-74 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-145 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-146 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-75 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-147 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-148 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-76 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-149 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-150 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-77 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-151 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-152 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-78 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-153 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-154 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-79 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-155 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-156 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-80 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-157 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-158 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-81 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-159 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-160 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-82 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-161 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-162 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-83 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-163 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-164 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-84 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-165 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-166 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-85 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-167 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-168 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-86 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-169 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-170 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-87 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-171 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-172 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-88 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-173 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-174 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-89 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-175 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-176 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-90 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-177 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-178 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-91 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-179 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-180 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-92 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-181 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-182 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-93 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-183 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-184 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-94 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-185 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-186 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-95 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-187 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-188 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-96 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-189 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-190 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-97 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-191 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-192 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-98 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-193 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-194 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-99 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-195 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-196 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-100 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-197 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-198 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-101 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-199 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-200 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-102 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-201 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-202 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-103 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-203 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-204 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-104 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-205 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-206 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-105 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-207 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-208 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-106 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-209 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-210 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-107 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-211 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-212 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-108 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-213 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-214 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-109 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-215 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-216 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-110 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-217 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-218 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-111 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-219 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-220 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-112 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-221 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-222 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-113 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-223 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-224 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-114 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-225 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-226 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-115 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-227 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-228 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-116 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-229 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-230 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-117 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-231 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-232 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-118 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-233 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-234 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-119 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-235 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-236 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-120 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-237 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-238 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-121 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-239 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-240 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-122 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-241 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-242 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-123 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-243 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-244 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-124 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-245 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-246 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-125 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-247 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-248 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-126 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-249 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-250 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-127 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-251 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-252 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-128 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-253 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-254 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-129 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-255 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-256 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-130 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-257 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-258 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-131 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-259 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-260 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-132 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-261 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-262 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-133 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-263 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-264 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-134 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-265 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-266 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-135 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-267 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-268 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-136 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-269 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-270 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-137 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-271 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-272 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-138 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-273 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-274 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-139 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-275 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-276 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-140 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-277 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-278 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-141 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-279 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-280 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-142 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-281 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-282 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-143 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-283 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-284 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-144 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-285 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-286 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-145 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-287 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-288 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-146 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-289 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-290 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-147 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-291 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-292 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-148 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-293 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-294 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-149 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-295 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-296 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-150 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-297 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-298 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-151 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-299 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-300 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-152 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-301 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-302 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-153 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-303 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-304 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-154 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-305 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-306 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-155 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-307 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-308 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-156 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-309 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-310 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-157 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-311 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-312 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-158 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-313 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-314 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-159 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-315 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-316 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-160 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-317 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-318 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-161 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-319 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-320 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-162 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-321 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-322 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-163 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-323 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-324 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-164 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-325 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-326 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-165 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-327 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-328 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-166 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-329 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-330 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-167 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-331 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-332 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-168 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-333 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-334 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-169 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-335 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-336 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-170 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-337 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-338 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-171 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-339 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-340 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-172 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-341 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-342 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-173 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-343 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-344 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-174 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-345 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-346 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-175 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-347 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-348 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-176 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-349 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-350 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-177 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-351 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-352 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-178 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-353 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-354 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-179 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-355 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-356 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-180 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-357 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-358 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-181 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-359 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-360 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-182 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-361 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-362 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-183 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-363 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-364 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-184 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-365 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-366 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-185 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-367 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-368 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-186 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-369 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-370 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-187 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-371 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-372 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-188 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-373 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-374 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-189 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-375 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-376 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-190 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-377 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-378 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-191 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-379 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-380 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-192 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-381 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-382 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-193 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-383 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-384 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-194 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-385 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-386 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-195 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-387 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-388 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-196 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-389 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-390 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-197 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-391 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-392 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-198 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-393 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-394 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-199 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-395 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-396 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-200 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-397 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-398 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-201 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-399 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-400 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-202 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-401 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-402 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-203 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-403 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-404 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-204 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-405 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-406 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-205 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-407 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-408 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-206 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-409 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-410 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-207 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-411 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-412 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-208 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-413 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-414 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-209 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-415 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-416 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-210 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-417 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-418 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-211 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-419 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-420 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-212 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-421 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-422 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-213 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-423 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-424 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-214 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-425 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-426 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-215 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-427 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-428 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-216 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-429 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-430 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-217 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-431 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-432 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-218 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-433 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-434 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-219 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-435 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-436 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-220 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-437 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-438 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-221 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-439 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-440 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-222 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-441 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-442 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-223 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-443 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-444 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-224 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-445 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-446 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-225 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-447 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-448 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-226 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-449 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-450 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-227 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-451 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-452 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-228 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-453 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-454 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-229 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-455 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-456 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-230 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-457 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-458 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-231 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-459 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-460 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-232 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-461 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-462 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-233 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-463 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-464 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-234 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-465 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-466 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-235 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-467 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-468 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-236 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-469 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-470 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-237 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-471 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-472 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-238 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-473 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-474 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-239 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-475 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-476 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-240 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-477 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-478 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-241 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-479 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-480 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-242 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-481 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-482 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-243 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-483 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-484 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-244 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-485 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-486 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-245 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-487 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-488 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-246 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-489 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-490 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-247 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-491 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-492 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-248 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-493 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-494 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-249 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-495 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-496 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-250 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-497 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-498 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-251 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-499 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-500 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-252 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-501 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-502 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-253 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-503 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-504 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-254 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-505 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-506 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-255 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-507 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-508 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-256 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-509 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-510 [128, 384] [128, 256] (recursive) 12,615,680 ├─GRUCellFromScratch: 1-257 [128, 128] [128, 256] (recursive) -- │ └─Linear: 2-511 [128, 384] [128, 512] (recursive) 25,231,360 │ └─Linear: 2-512 [128, 384] [128, 256] (recursive) 12,615,680 ├─Linear: 1-258 [128, 256] [128, 1] 257 32,896 ============================================================================================================================================ Total params: 2,856,193 Trainable params: 2,856,193 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 10.02 ============================================================================================================================================ Input size (MB): 0.26 Forward/backward pass size (MB): 234.88 Params size (MB): 11.42 Estimated Total Size (MB): 246.57 ============================================================================================================================================
In [12]:
def calculate_accuracy(loader, model, device, is_binary=True):
model.eval()
correct, total = 0, 0
with torch.no_grad():
for inputs, labels, lengths in loader:
inputs, labels, lengths = inputs.to(device), labels.to(device), lengths.to(device)
logits = model(inputs, lengths)
if is_binary:
predicted = (torch.sigmoid(logits.view(-1)) > 0.5).long()
else:
_, predicted = torch.max(logits.data, 1)
total += labels.size(0)
correct += (predicted == labels.long()).sum().item()
return 100 * correct / total
In [13]:
IS_BINARY, NUM_CLASSES
Out[13]:
(True, 1)
In [14]:
# 1. Update Optimizer and Scheduler
# Binary uses BCEWithLogitsLoss; Multi-class uses CrossEntropy
criterion = nn.BCEWithLogitsLoss() if IS_BINARY else nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)
# Robust Scheduler: Cosine Annealing
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=NUM_EPOCHS)
In [15]:
start_time = time.time()
history = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': [], 'lrs': []}
print(f"Starting training on {DEVICE}...")
for epoch in range(NUM_EPOCHS):
model.train()
running_loss = 0.0
# 1. Training Phase
for i, (inputs, labels, lengths) in enumerate(train_loader):
inputs, labels, lengths = inputs.to(DEVICE), labels.to(DEVICE), lengths.to(DEVICE)
optimizer.zero_grad()
# Forward pass (passing lengths for masking logic)
logits = model(inputs, lengths)
if IS_BINARY:
loss = criterion(logits.squeeze(), labels.float())
else:
loss = criterion(logits, labels.long())
loss.backward()
# --- ROBUST RNN FIX: Gradient Clipping ---
# Prevents gradients from "exploding" during the backprop through time (BPTT)
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
running_loss += loss.item() * inputs.size(0)
# 2. Metrics Calculation
# Note: Using the updated calculate_accuracy that handles (inputs, lengths)
train_loss = running_loss / len(train_ds)
train_acc = calculate_accuracy(train_loader, model, DEVICE, is_binary=IS_BINARY)
# 3. Validation Phase
model.eval()
val_loss = 0.0
with torch.no_grad():
for inputs, labels, lengths in val_loader:
inputs, labels, lengths = inputs.to(DEVICE), labels.to(DEVICE), lengths.to(DEVICE)
logits = model(inputs, lengths)
if IS_BINARY:
loss = criterion(logits.squeeze(), labels.float())
else:
loss = criterion(logits, labels.long())
val_loss += loss.item() * inputs.size(0)
val_loss /= len(val_ds)
val_acc = calculate_accuracy(val_loader, model, DEVICE, is_binary=IS_BINARY)
scheduler.step()
current_lr = optimizer.param_groups[0]['lr']
# 4. Logging
history['train_loss'].append(train_loss)
history['val_loss'].append(val_loss)
history['train_acc'].append(train_acc)
history['val_acc'].append(val_acc)
history['lrs'].append(current_lr)
print(f'Epoch {epoch + 1}/{NUM_EPOCHS} | '
f'Loss (train/val): {train_loss:.4f} / {val_loss:.4f} | '
f'Acc (train/val): {train_acc:.2f}% / {val_acc:.2f}% | '
f'LR: {current_lr:.6f}')
print('\nTotal Training Time: %.2f min' % ((time.time() - start_time)/60))
Starting training on cuda... Epoch 1/20 | Loss (train/val): 0.6772 / 0.6710 | Acc (train/val): 57.83% / 58.60% | LR: 0.000994 Epoch 2/20 | Loss (train/val): 0.6113 / 0.5678 | Acc (train/val): 73.63% / 72.08% | LR: 0.000976 Epoch 3/20 | Loss (train/val): 0.4668 / 0.4457 | Acc (train/val): 85.28% / 79.60% | LR: 0.000946 Epoch 4/20 | Loss (train/val): 0.3399 / 0.4118 | Acc (train/val): 90.31% / 81.48% | LR: 0.000905 Epoch 5/20 | Loss (train/val): 0.2540 / 0.4114 | Acc (train/val): 93.69% / 82.12% | LR: 0.000854 Epoch 6/20 | Loss (train/val): 0.1818 / 0.4458 | Acc (train/val): 96.46% / 82.40% | LR: 0.000794 Epoch 7/20 | Loss (train/val): 0.1148 / 0.5336 | Acc (train/val): 98.08% / 82.28% | LR: 0.000727 Epoch 8/20 | Loss (train/val): 0.0669 / 0.6637 | Acc (train/val): 98.69% / 82.52% | LR: 0.000655 Epoch 9/20 | Loss (train/val): 0.0336 / 0.6918 | Acc (train/val): 99.79% / 81.36% | LR: 0.000578 Epoch 10/20 | Loss (train/val): 0.0136 / 0.8760 | Acc (train/val): 99.93% / 82.56% | LR: 0.000500 Epoch 11/20 | Loss (train/val): 0.0060 / 0.9594 | Acc (train/val): 100.00% / 82.04% | LR: 0.000422 Epoch 12/20 | Loss (train/val): 0.0024 / 1.1272 | Acc (train/val): 100.00% / 82.32% | LR: 0.000345 Epoch 13/20 | Loss (train/val): 0.0013 / 1.1747 | Acc (train/val): 100.00% / 82.24% | LR: 0.000273 Epoch 14/20 | Loss (train/val): 0.0007 / 1.2574 | Acc (train/val): 100.00% / 82.20% | LR: 0.000206 Epoch 15/20 | Loss (train/val): 0.0005 / 1.2982 | Acc (train/val): 100.00% / 82.20% | LR: 0.000146 Epoch 16/20 | Loss (train/val): 0.0004 / 1.3212 | Acc (train/val): 100.00% / 82.16% | LR: 0.000095 Epoch 17/20 | Loss (train/val): 0.0004 / 1.3425 | Acc (train/val): 100.00% / 82.28% | LR: 0.000054 Epoch 18/20 | Loss (train/val): 0.0004 / 1.3462 | Acc (train/val): 100.00% / 82.24% | LR: 0.000024 Epoch 19/20 | Loss (train/val): 0.0003 / 1.3519 | Acc (train/val): 100.00% / 82.20% | LR: 0.000006 Epoch 20/20 | Loss (train/val): 0.0003 / 1.3534 | Acc (train/val): 100.00% / 82.28% | LR: 0.000000 Total Training Time: 12.90 min
In [16]:
# --- Final Evaluation ---
test_accuracy = calculate_accuracy(test_loader, model, DEVICE, is_binary=IS_BINARY)
print(f'\nFinal Test Accuracy: {test_accuracy:.2f}%')
Final Test Accuracy: 80.86%
In [17]:
def plot_history(history):
"""
Plots the training and validation loss, accuracy, and learning rate
from a PyTorch history dictionary.
Args:
history (dict): Dictionary containing training metrics.
Expected keys: 'train_loss', 'val_loss', 'train_acc', 'val_acc', 'lrs'.
"""
epochs = range(len(history['train_loss']))
plt.figure(figsize=(18, 5))
# Plot 1: Loss
plt.subplot(1, 3, 1)
plt.plot(epochs, history['train_loss'], label='Training Loss')
plt.plot(epochs, history['val_loss'], label='Validation Loss')
plt.title('Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
# plt.grid(True)
# Plot 2: Accuracy
plt.subplot(1, 3, 2)
plt.plot(epochs, history['train_acc'], label='Training Accuracy')
plt.plot(epochs, history['val_acc'], label='Validation Accuracy')
plt.title('Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
# plt.grid(True)
# Plot 3: Learning Rate
plt.subplot(1, 3, 3)
plt.plot(epochs, history['lrs'], label='Learning Rate', color='orange')
plt.title('Learning Rate')
plt.xlabel('Epochs')
plt.ylabel('Learning Rate')
plt.legend()
# plt.grid(True)
plt.tight_layout()
plt.show()
# plt.savefig()
plot_history(history)
In [18]:
# 1. Reuse your exact training tokenizer
tokenizer = lambda text: re.findall(r'\w+', text.lower())
def predict_sentiment(model, sentence):
model.eval()
# Step A: Tokenize exactly like training
tokens = tokenizer(sentence)
# Step B: Numericalize using the <UNK> index (1)
# We use .get(t, 1) to handle words not in your top 20k vocab
indexed = [vocab.get(t, 1) for t in tokens]
# Step C: Prepare Tensors [Batch Size 1, Seq_Len]
tensor = torch.LongTensor(indexed).unsqueeze(0).to(DEVICE)
length = torch.LongTensor([len(indexed)]).to(DEVICE)
with torch.no_grad():
# Step D: Forward Pass
logits = model(tensor, length)
# Step E: Convert Logit to Probability (0.0 to 1.0)
# Using .view(-1) for safety with batch size 1
prob = torch.sigmoid(logits.view(-1)).item()
return prob
# --- Test ---
review = "This film was an absolute masterpiece of modern cinema."
score = predict_sentiment(model, review)
print(f'Review: "{review}"')
print(f'Positive Probability: {score:.4f}')
print(f'Final Sentiment: {"Positive" if score > 0.5 else "Negative"}')
Review: "This film was an absolute masterpiece of modern cinema." Positive Probability: 1.0000 Final Sentiment: Positive
In [19]:
imdb_tests = [
"I wanted to like this movie, but the acting was wooden and the plot was a mess.", # Negative
"An absolute triumph of cinema that I would recommend to everyone I know.", # Positive
"Not the worst thing I've ever seen, but definitely not the best either.", # Neutral/Leaning Neg
"The special effects were stunning, though the story felt a bit rushed.", # Mixed
"I fell asleep halfway through; avoid this waste of time at all costs." # Highly Negative
]
print(f"{'Review':<60} | {'Prob':<6} | {'Sentiment'}")
print("-" * 80)
for text in imdb_tests:
prob = predict_sentiment(model, text)
sentiment = "Positive" if prob > 0.5 else "Negative"
print(f"{text[:58]:<60} | {prob:.3f} | {sentiment}")
Review | Prob | Sentiment -------------------------------------------------------------------------------- I wanted to like this movie, but the acting was wooden and | 0.000 | Negative An absolute triumph of cinema that I would recommend to ev | 0.997 | Positive Not the worst thing I've ever seen, but definitely not the | 0.600 | Positive The special effects were stunning, though the story felt a | 1.000 | Positive I fell asleep halfway through; avoid this waste of time at | 0.000 | Negative
In [20]:
predict_sentiment(model, "This film is quite interesting")
Out[20]:
0.1331896185874939
In [21]:
predict_sentiment(model, "This film is terrible")
Out[21]:
0.0036119988653808832
In [22]:
%watermark -iv
datasets : 4.0.0 matplotlib : 3.10.0 numpy : 2.0.2 pandas : 2.2.2 re : 2.2.1 sklearn : 1.6.1 torch : 2.10.0+cu128 torchinfo : 1.8.0 torchvision: 0.25.0+cu128