Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [1]:
!pip install watermark
Collecting watermark Downloading watermark-2.6.0-py3-none-any.whl.metadata (15 kB) Requirement already satisfied: ipython>=6.0 in /usr/local/lib/python3.12/dist-packages (from watermark) (7.34.0) Requirement already satisfied: importlib-metadata>=1.4 in /usr/local/lib/python3.12/dist-packages (from watermark) (8.7.0) Requirement already satisfied: setuptools in /usr/local/lib/python3.12/dist-packages (from watermark) (75.2.0) Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.12/dist-packages (from importlib-metadata>=1.4->watermark) (3.23.0) Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.19.2) Requirement already satisfied: decorator in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (4.4.2) Requirement already satisfied: pickleshare in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.7.5) Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (5.7.1) Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (3.0.52) Requirement already satisfied: pygments in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (2.19.2) Requirement already satisfied: backcall in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.2.0) Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.1.7) Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (4.9.0) Requirement already satisfied: parso<0.9.0,>=0.8.4 in /usr/local/lib/python3.12/dist-packages (from jedi>=0.16->ipython>=6.0->watermark) (0.8.5) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.12/dist-packages (from pexpect>4.3->ipython>=6.0->watermark) (0.7.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.0->watermark) (0.2.14) Downloading watermark-2.6.0-py3-none-any.whl (13 kB) Installing collected packages: watermark Successfully installed watermark-2.6.0
In [2]:
!pip install torchinfo
Requirement already satisfied: torchinfo in /usr/local/lib/python3.12/dist-packages (1.8.0)
In [3]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 torch: 2.8.0+cu126
In [4]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from sklearn.model_selection import train_test_split
# from torch.cuda.amp import GradScaler, autocast
from torch.amp import GradScaler, autocast
from torchinfo import summary
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
In [5]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
In [6]:
# Check if GPU is available and set the device
if torch.cuda.is_available():
num_gpus = torch.cuda.device_count()
print("Num GPUs Available: ", num_gpus)
for i in range(num_gpus):
gpu_name = torch.cuda.get_device_name(i)
print(f"GPU {i}: {gpu_name}")
else:
print("No GPU available.")
Num GPUs Available: 2 GPU 0: Tesla T4 GPU 1: Tesla T4
ResNet: Residual Blocks¶
- Single Residual Block: The fundamental innovation of a Residual Block is the "shortcut connection" (or skip connection). Instead of forcing the layers to learn the underlying mapping H(x), we let them learn the residual mapping F(x) = H(x) - x. The output becomes F(x) + x. This allows the gradient to flow through the shortcut during backpropagation, effectively solving the vanishing gradient problem and allowing for much deeper architectures.
- Various Usages of Activation in residual Block: This diagram illustrates the placement of Batch Normalization (BN) and ReLU activation relative to the convolutional layers. While the original ResNet (2015) used Post-activation (applying ReLU after the addition), subsequent research (He et al., 2016) introduced Pre-activation ResNet. In the pre-activation version, BN and ReLU are applied before the weight layers, which eases optimization and improves the regularization of the model.
Types of Residual Block: ResNet architectures use two primary building blocks depending on the depth of the network:
- Basic Block: Used in ResNet-$18$ and ResNet-$34$. It consists of two consecutive $3 \times 3$ convolutional layers.
- Bottleneck Block: Used in deeper versions like ResNet-$50$, $101$, and $152$. It uses a triplet of $1 \times 1$, $3 \times 3$, and $1 \times 1$ convolutions. The $1 \times 1$ layers are used to first reduce (bottleneck) and then restore dimensions, significantly reducing the number of parameters and computational cost while maintaining performance.
ResNet Block Diagram¶
Network Architecture:
The diagram below shows a ResNet block with and without the 1x1 convolution. The 1x1 convolution (with stride) can be used to change the shape of the array, which is necessary for residual connection through an upsampling/downsampling layer.
ResNet-101 Classifier on CIFAR-10¶
Network Architecture: The network in this notebook is an implementation of the ResNet-101 [1] architecture on the Cifar-10 dataset to train an image classifier.
| layer name | output size | 18-layer | 34-layer | 50-layer | 101-layer | 152-layer |
|---|---|---|---|---|---|---|
| conv1 | 112×112 | 7×7, 64, stride 2 | 7×7, 64, stride 2 | 7×7, 64, stride 2 | 7×7, 64, stride 2 | 7×7, 64, stride 2 |
| conv2_x | 56×56 | 3×3 max pool, stride 2 | 3×3 max pool, stride 2 | 3×3 max pool, stride 2 | 3×3 max pool, stride 2 | 3×3 max pool, stride 2 |
| $\begin{bmatrix} 3\times3, 64 \\ 3\times3, 64 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 64 \\ 3\times3, 64 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 64 \\ 3\times3, 64 \\ 1\times1, 256 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 64 \\ 3\times3, 64 \\ 1\times1, 256 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 64 \\ 3\times3, 64 \\ 1\times1, 256 \end{bmatrix} \times 3$ | ||
| conv3_x | 28×28 | $\begin{bmatrix} 3\times3, 128 \\ 3\times3, 128 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 128 \\ 3\times3, 128 \end{bmatrix} \times 4$ | $\begin{bmatrix} 1\times1, 128 \\ 3\times3, 128 \\ 1\times1, 512 \end{bmatrix} \times 4$ | $\begin{bmatrix} 1\times1, 128 \\ 3\times3, 128 \\ 1\times1, 512 \end{bmatrix} \times 4$ | $\begin{bmatrix} 1\times1, 128 \\ 3\times3, 128 \\ 1\times1, 512 \end{bmatrix} \times 8$ |
| conv4_x | 14×14 | $\begin{bmatrix} 3\times3, 256 \\ 3\times3, 256 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 256 \\ 3\times3, 256 \end{bmatrix} \times 6$ | $\begin{bmatrix} 1\times1, 256 \\ 3\times3, 256 \\ 1\times1, 1024 \end{bmatrix} \times 6$ | $\begin{bmatrix} 1\times1, 256 \\ 3\times3, 256 \\ 1\times1, 1024 \end{bmatrix} \times 23$ | $\begin{bmatrix} 1\times1, 256 \\ 3\times3, 256 \\ 1\times1, 1024 \end{bmatrix} \times 36$ |
| conv5_x | 7×7 | $\begin{bmatrix} 3\times3, 512 \\ 3\times3, 512 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 512 \\ 3\times3, 512 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 512 \\ 3\times3, 512 \\ 1\times1, 2048 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 512 \\ 3\times3, 512 \\ 1\times1, 2048 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 512 \\ 3\times3, 512 \\ 1\times1, 2048 \end{bmatrix} \times 3$ |
| 1×1 | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | |
| FLOPs | 1.8 × 10⁹ | 3.6 × 10⁹ | 3.8 × 10⁹ | 7.6 × 10⁹ | 11.3 × 10⁹ |
Table adapted from ResNet paper
Key Notes on the Architecture:
- Building Blocks: The 18-layer and 34-layer networks use basic blocks (two 3×3 convs), while the deeper 50/101/152-layer networks use bottleneck blocks (1×1, 3×3, and 1×1 convs).
- Downsampling: Downsampling is performed by
conv3_1,conv4_1, andconv5_1with a stride of 2. - Global Average Pooling: This is used at the end of the convolutional layers to reduce the spatial dimensions to 1×1 before the fully connected layer.
The following figure illustrates residual blocks with skip connections such that the input passed via the shortcut matches the dimensions of the main path's output, which allows the network to learn identity functions.
The ResNet-18/34 architecture actually uses residual blocks with skip connections such that the input passed via the shortcut matches is resized to dimensions of the main path's output. Such a residual block is illustrated below:
The ResNet-101 uses a bottleneck as shown below:
Improvements over LeNet, AlexNet, VGG:
- Skip Connections: Unlike VGG and AlexNet, ResNet uses "identity mapping" which solves the vanishing gradient problem.
- Global Average Pooling: Instead of a huge Flatten + Dense layer (like in AlexNet), we use
avg_pool2d. This significantly reduces parameter count and prevents overfitting. - Batch Normalization: ResNet relies heavily on BN after every convolution to stabilize training at a higher learning rate.
REFERENCES:
He, K., Zhang, X., Ren, S., & Sun, J. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. (770-77).
Johnson, J. (2022, February 2). Lecture 8: CNN Architectures [PowerPoint Slides]. University of Michigan EECS 498/598: Deep Learning for Computer Vision. pp 77-109. https://web.eecs.umich.edu/~justincj/slides/eecs498/WI2022/598_WI2022_lecture08.pdf
Mahdid, Y. (2024.). deep-learning-model-explained: ResNet [Source code]. GitHub. https://github.com/yacineMahdid/deep-learning-model-explained/tree/main/resnet
In [7]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 123
BATCH_SIZE = 128 #256
LEARNING_RATE = 0.001
NUM_EPOCHS = 50
INPUT_SIZE = 32 # CIFAR-10 is 32x32
NUM_CLASSES = 10
# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
cuda
In [8]:
# Mean and Std Dev for CIFAR-10 (3 channels)
CIFAR_MEAN = (0.4914, 0.4822, 0.4465)
CIFAR_STD = (0.2471, 0.2435, 0.2616)
train_transforms = transforms.Compose([
transforms.RandomRotation(15),
transforms.RandomCrop(32, padding=4, padding_mode='reflect'),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(CIFAR_MEAN, CIFAR_STD)
])
test_transforms = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(CIFAR_MEAN, CIFAR_STD)
])
# Load Datasets
train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=train_transforms)
test_dataset = datasets.CIFAR10(root='./data', train=False, download=True, transform=test_transforms)
# Split training set into training and validation sets
train_size = int(0.9 * len(train_dataset))
val_size = len(train_dataset) - train_size
train_subset, val_subset = torch.utils.data.random_split(train_dataset, [train_size, val_size])
# Create DataLoaders
train_loader = DataLoader(train_subset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2, pin_memory=True)
val_loader = DataLoader(val_subset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)
100%|██████████| 170M/170M [00:01<00:00, 106MB/s]
In [11]:
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, in_planes, planes, stride=1):
super(BasicBlock, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.relu = nn.ReLU(inplace=True)
self.shortcut = nn.Sequential()
if stride != 1 or in_planes != self.expansion*planes:
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes)
)
def forward(self, x):
residual = self.shortcut(x)
out = self.relu(self.bn1(self.conv1(x))) # conv->batchnorm->relu
out = self.bn2(self.conv2(out)) # conv->batchnorm
out += residual # skip connection
out = torch.relu(out) # relu
return out
class Bottleneck(nn.Module):
expansion = 4 # The output of the block is 4x the width of the bottleneck
def __init__(self, in_planes, planes, stride=1):
super(Bottleneck, self).__init__()
# 1x1 Convolution to reduce dimensions
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
# 3x3 Convolution for spatial processing
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
# 1x1 Convolution to expand dimensions (planes * 4)
self.conv3 = nn.Conv2d(planes, self.expansion * planes, kernel_size=1, bias=False)
self.bn3 = nn.BatchNorm2d(self.expansion * planes)
self.shortcut = nn.Sequential()
# Shortcut needs to match dimensions (if stride != 1 or channels change)
if stride != 1 or in_planes != self.expansion * planes:
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion * planes, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion * planes)
)
def forward(self, x):
residual = self.shortcut(x)
out = torch.relu(self.bn1(self.conv1(x))) # conv->batchnorm->relu
out = torch.relu(self.bn2(self.conv2(out))) # conv->batchnorm->relu
out = self.bn3(self.conv3(out)) # conv->batchnorm
out += residual # skip connection
out = torch.relu(out) # relu
return out
class ResNet(nn.Module):
def __init__(self, block, num_blocks, num_classes=10):
super(ResNet, self).__init__()
self.in_planes = 64
self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(64)
self.relu = nn.ReLU(inplace=True)
self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.linear = nn.Linear(512*block.expansion, num_classes)
def _make_layer(self, block, planes, num_blocks, stride):
strides = [stride] + [1]*(num_blocks-1)
layers = []
for s in strides:
layers.append(block(self.in_planes, planes, s))
self.in_planes = planes * block.expansion
return nn.Sequential(*layers)
def forward(self, x):
out = torch.relu(self.bn1(self.conv1(x)))
out = self.layer1(out)
out = self.layer2(out)
out = self.layer3(out)
out = self.layer4(out)
out = self.avgpool(out)
out = torch.flatten(out, 1)
out = self.linear(out)
return out
def init_weights(m):
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
nn.init.normal_(m.weight, 0, 0.01)
nn.init.constant_(m.bias, 0)
def ResNet101():
return ResNet(Bottleneck, [3, 4, 23, 3])
In [12]:
def test_model_dims():
x = torch.randn(2, 3, 32, 32).to(device) # Create a dummy batch: 2 images, 3 channels, 32x32 size
net = ResNet101().to(device)
y = net(x)
print(f"Input Shape: {x.shape}")
print(f"Output Shape: {y.shape}") # Should be [2, 10]
if y.shape == (2, 10):
print("✅ Sanity Check Passed: Dimensions are correct for CIFAR-10!")
else:
print("❌ Sanity Check Failed: Check your stride or pooling layers.")
test_model_dims()
Input Shape: torch.Size([2, 3, 32, 32]) Output Shape: torch.Size([2, 10]) ✅ Sanity Check Passed: Dimensions are correct for CIFAR-10!
In [13]:
set_all_seeds(RANDOM_SEED)
model = ResNet101()
model.apply(init_weights) # Initialize weights while model still in CPU RAM
model = model.to(device)
# Model parameter details
summary(model,
input_size=(BATCH_SIZE, 3, INPUT_SIZE, INPUT_SIZE), # verbose=1,
col_names=["input_size", "output_size", "num_params", "kernel_size", "mult_adds"]
)
Out[13]:
===================================================================================================================================================================== Layer (type:depth-idx) Input Shape Output Shape Param # Kernel Shape Mult-Adds ===================================================================================================================================================================== ResNet [128, 3, 32, 32] [128, 10] -- -- -- ├─Conv2d: 1-1 [128, 3, 32, 32] [128, 64, 32, 32] 1,728 [3, 3] 226,492,416 ├─BatchNorm2d: 1-2 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ├─Sequential: 1-3 [128, 64, 32, 32] [128, 256, 32, 32] -- -- -- │ └─Bottleneck: 2-1 [128, 64, 32, 32] [128, 256, 32, 32] -- -- -- │ │ └─Sequential: 3-1 [128, 64, 32, 32] [128, 256, 32, 32] 16,896 -- 2,147,549,184 │ │ └─Conv2d: 3-2 [128, 64, 32, 32] [128, 64, 32, 32] 4,096 [1, 1] 536,870,912 │ │ └─BatchNorm2d: 3-3 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-4 [128, 64, 32, 32] [128, 64, 32, 32] 36,864 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-5 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-6 [128, 64, 32, 32] [128, 256, 32, 32] 16,384 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-7 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 │ └─Bottleneck: 2-2 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- │ │ └─Sequential: 3-8 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- │ │ └─Conv2d: 3-9 [128, 256, 32, 32] [128, 64, 32, 32] 16,384 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-10 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-11 [128, 64, 32, 32] [128, 64, 32, 32] 36,864 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-12 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-13 [128, 64, 32, 32] [128, 256, 32, 32] 16,384 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-14 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 │ └─Bottleneck: 2-3 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- │ │ └─Sequential: 3-15 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- │ │ └─Conv2d: 3-16 [128, 256, 32, 32] [128, 64, 32, 32] 16,384 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-17 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-18 [128, 64, 32, 32] [128, 64, 32, 32] 36,864 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-19 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-20 [128, 64, 32, 32] [128, 256, 32, 32] 16,384 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-21 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 ├─Sequential: 1-4 [128, 256, 32, 32] [128, 512, 16, 16] -- -- -- │ └─Bottleneck: 2-4 [128, 256, 32, 32] [128, 512, 16, 16] -- -- -- │ │ └─Sequential: 3-22 [128, 256, 32, 32] [128, 512, 16, 16] 132,096 -- 4,295,098,368 │ │ └─Conv2d: 3-23 [128, 256, 32, 32] [128, 128, 32, 32] 32,768 [1, 1] 4,294,967,296 │ │ └─BatchNorm2d: 3-24 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-25 [128, 128, 32, 32] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-26 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-27 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-28 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 │ └─Bottleneck: 2-5 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─Sequential: 3-29 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─Conv2d: 3-30 [128, 512, 16, 16] [128, 128, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-31 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-32 [128, 128, 16, 16] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-33 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-34 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-35 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 │ └─Bottleneck: 2-6 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─Sequential: 3-36 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─Conv2d: 3-37 [128, 512, 16, 16] [128, 128, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-38 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-39 [128, 128, 16, 16] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-40 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-41 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-42 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 │ └─Bottleneck: 2-7 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─Sequential: 3-43 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─Conv2d: 3-44 [128, 512, 16, 16] [128, 128, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-45 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-46 [128, 128, 16, 16] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-47 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-48 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-49 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 ├─Sequential: 1-5 [128, 512, 16, 16] [128, 1024, 8, 8] -- -- -- │ └─Bottleneck: 2-8 [128, 512, 16, 16] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-50 [128, 512, 16, 16] [128, 1024, 8, 8] 526,336 -- 4,295,229,440 │ │ └─Conv2d: 3-51 [128, 512, 16, 16] [128, 256, 16, 16] 131,072 [1, 1] 4,294,967,296 │ │ └─BatchNorm2d: 3-52 [128, 256, 16, 16] [128, 256, 16, 16] 512 -- 65,536 │ │ └─Conv2d: 3-53 [128, 256, 16, 16] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-54 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-55 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-56 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-9 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-57 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-58 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-59 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-60 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-61 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-62 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-63 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-10 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-64 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-65 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-66 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-67 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-68 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-69 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-70 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-11 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-71 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-72 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-73 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-74 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-75 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-76 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-77 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-12 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-78 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-79 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-80 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-81 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-82 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-83 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-84 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-13 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-85 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-86 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-87 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-88 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-89 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-90 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-91 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-14 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-92 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-93 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-94 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-95 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-96 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-97 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-98 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-15 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-99 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-100 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-101 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-102 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-103 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-104 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-105 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-16 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-106 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-107 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-108 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-109 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-110 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-111 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-112 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-17 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-113 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-114 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-115 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-116 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-117 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-118 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-119 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-18 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-120 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-121 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-122 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-123 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-124 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-125 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-126 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-19 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-127 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-128 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-129 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-130 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-131 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-132 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-133 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-20 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-134 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-135 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-136 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-137 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-138 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-139 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-140 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-21 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-141 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-142 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-143 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-144 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-145 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-146 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-147 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-22 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-148 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-149 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-150 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-151 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-152 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-153 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-154 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-23 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-155 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-156 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-157 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-158 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-159 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-160 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-161 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-24 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-162 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-163 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-164 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-165 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-166 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-167 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-168 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-25 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-169 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-170 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-171 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-172 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-173 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-174 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-175 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-26 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-176 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-177 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-178 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-179 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-180 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-181 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-182 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-27 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-183 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-184 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-185 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-186 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-187 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-188 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-189 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-28 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-190 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-191 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-192 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-193 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-194 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-195 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-196 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-29 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-197 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-198 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-199 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-200 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-201 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-202 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-203 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 │ └─Bottleneck: 2-30 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Sequential: 3-204 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─Conv2d: 3-205 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-206 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-207 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-208 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-209 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-210 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ├─Sequential: 1-6 [128, 1024, 8, 8] [128, 2048, 4, 4] -- -- -- │ └─Bottleneck: 2-31 [128, 1024, 8, 8] [128, 2048, 4, 4] -- -- -- │ │ └─Sequential: 3-211 [128, 1024, 8, 8] [128, 2048, 4, 4] 2,101,248 -- 4,295,491,584 │ │ └─Conv2d: 3-212 [128, 1024, 8, 8] [128, 512, 8, 8] 524,288 [1, 1] 4,294,967,296 │ │ └─BatchNorm2d: 3-213 [128, 512, 8, 8] [128, 512, 8, 8] 1,024 -- 131,072 │ │ └─Conv2d: 3-214 [128, 512, 8, 8] [128, 512, 4, 4] 2,359,296 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-215 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 │ │ └─Conv2d: 3-216 [128, 512, 4, 4] [128, 2048, 4, 4] 1,048,576 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-217 [128, 2048, 4, 4] [128, 2048, 4, 4] 4,096 -- 524,288 │ └─Bottleneck: 2-32 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- │ │ └─Sequential: 3-218 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- │ │ └─Conv2d: 3-219 [128, 2048, 4, 4] [128, 512, 4, 4] 1,048,576 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-220 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 │ │ └─Conv2d: 3-221 [128, 512, 4, 4] [128, 512, 4, 4] 2,359,296 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-222 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 │ │ └─Conv2d: 3-223 [128, 512, 4, 4] [128, 2048, 4, 4] 1,048,576 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-224 [128, 2048, 4, 4] [128, 2048, 4, 4] 4,096 -- 524,288 │ └─Bottleneck: 2-33 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- │ │ └─Sequential: 3-225 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- │ │ └─Conv2d: 3-226 [128, 2048, 4, 4] [128, 512, 4, 4] 1,048,576 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-227 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 │ │ └─Conv2d: 3-228 [128, 512, 4, 4] [128, 512, 4, 4] 2,359,296 [3, 3] 4,831,838,208 │ │ └─BatchNorm2d: 3-229 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 │ │ └─Conv2d: 3-230 [128, 512, 4, 4] [128, 2048, 4, 4] 1,048,576 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-231 [128, 2048, 4, 4] [128, 2048, 4, 4] 4,096 -- 524,288 ├─AdaptiveAvgPool2d: 1-7 [128, 2048, 4, 4] [128, 2048, 1, 1] -- -- -- ├─Linear: 1-8 [128, 2048] [128, 10] 20,490 -- 2,622,720 ===================================================================================================================================================================== Total params: 42,512,970 Trainable params: 42,512,970 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 321.29 ===================================================================================================================================================================== Input size (MB): 1.57 Forward/backward pass size (MB): 10452.22 Params size (MB): 170.05 Estimated Total Size (MB): 10623.84 =====================================================================================================================================================================
In [15]:
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)
# optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=200)
scaler = GradScaler('cuda')
In [16]:
# Function to calculate accuracy
def calculate_accuracy(loader):
model.eval()
correct = 0
total = 0
with torch.no_grad():
for inputs, labels in loader:
inputs, labels = inputs.to(device), labels.to(device)
with autocast('cuda'):
outputs = model(inputs)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
return 100 * correct / total
# Training Loop
start_time = time.time()
history = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': [], 'lrs': []}
print(f"Starting training on {device}...")
for epoch in range(NUM_EPOCHS):
model.train()
running_loss, correct_train, total_train = 0.0, 0, 0
for i, (inputs, labels) in enumerate(train_loader):
inputs, labels = inputs.to(device), labels.to(device)
optimizer.zero_grad()
with autocast('cuda'):
outputs = model(inputs)
loss = criterion(outputs, labels)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
running_loss += loss.item() * inputs.size(0)
_, predicted = torch.max(outputs, 1)
total_train += labels.size(0)
correct_train += (predicted == labels).sum().item()
train_loss = running_loss / len(train_subset)
train_acc = 100 * correct_train / total_train
# Calculate validation loss and accuracy
model.eval()
val_loss, correct_val, total_val = 0.0, 0, 0
with torch.no_grad():
for inputs, labels in val_loader:
inputs, labels = inputs.to(device), labels.to(device)
with autocast('cuda'):
outputs = model(inputs)
loss = criterion(outputs, labels)
val_loss += loss.item() * inputs.size(0)
_, predicted = torch.max(outputs, 1)
total_val += labels.size(0)
correct_val += (predicted == labels).sum().item()
val_loss /= len(val_subset)
val_acc = 100 * correct_val / total_val
scheduler.step()
current_lr = optimizer.param_groups[0]['lr']
history['train_loss'].append(train_loss)
history['val_loss'].append(val_loss)
history['train_acc'].append(train_acc)
history['val_acc'].append(val_acc)
history['lrs'].append(current_lr)
print(f'Epoch {epoch + 1}/{NUM_EPOCHS} | '
f'Train Loss: {train_loss:.5f}, '
f'Train Accuracy: {train_acc:.2f}% | '
f'Val Loss: {val_loss:.5f}, '
f'Val Accuracy: {val_acc:.2f}% | '
f'Learning Rate: {current_lr:.8f}')
print('\nTotal Training Time: %.2f min' % ((time.time() - start_time)/60))
# Final Evaluation
test_accuracy = calculate_accuracy(test_loader)
print(f'\nTest Accuracy: {test_accuracy:.2f}%')
Starting training on cuda... Epoch 1/50 | Train Loss: 1.73820, Train Accuracy: 35.74% | Val Loss: 1.47205, Val Accuracy: 45.78% | Learning Rate: 0.00099994 Epoch 2/50 | Train Loss: 1.27544, Train Accuracy: 53.54% | Val Loss: 1.30111, Val Accuracy: 53.26% | Learning Rate: 0.00099975 Epoch 3/50 | Train Loss: 1.06881, Train Accuracy: 61.93% | Val Loss: 1.06140, Val Accuracy: 63.38% | Learning Rate: 0.00099944 Epoch 4/50 | Train Loss: 0.91323, Train Accuracy: 67.73% | Val Loss: 1.08218, Val Accuracy: 62.64% | Learning Rate: 0.00099901 Epoch 5/50 | Train Loss: 0.78971, Train Accuracy: 72.51% | Val Loss: 0.82381, Val Accuracy: 71.94% | Learning Rate: 0.00099846 Epoch 6/50 | Train Loss: 0.70174, Train Accuracy: 75.32% | Val Loss: 0.97319, Val Accuracy: 67.62% | Learning Rate: 0.00099778 Epoch 7/50 | Train Loss: 0.63035, Train Accuracy: 77.93% | Val Loss: 0.61579, Val Accuracy: 78.38% | Learning Rate: 0.00099698 Epoch 8/50 | Train Loss: 0.58140, Train Accuracy: 79.76% | Val Loss: 0.71429, Val Accuracy: 75.98% | Learning Rate: 0.00099606 Epoch 9/50 | Train Loss: 0.54626, Train Accuracy: 80.88% | Val Loss: 0.68792, Val Accuracy: 77.30% | Learning Rate: 0.00099501 Epoch 10/50 | Train Loss: 0.51177, Train Accuracy: 82.39% | Val Loss: 0.59383, Val Accuracy: 78.92% | Learning Rate: 0.00099384 Epoch 11/50 | Train Loss: 0.47501, Train Accuracy: 83.63% | Val Loss: 0.50329, Val Accuracy: 82.56% | Learning Rate: 0.00099255 Epoch 12/50 | Train Loss: 0.58464, Train Accuracy: 80.21% | Val Loss: 0.54814, Val Accuracy: 81.06% | Learning Rate: 0.00099114 Epoch 13/50 | Train Loss: 0.51057, Train Accuracy: 82.31% | Val Loss: 0.56994, Val Accuracy: 78.82% | Learning Rate: 0.00098961 Epoch 14/50 | Train Loss: 0.45271, Train Accuracy: 84.37% | Val Loss: 0.54800, Val Accuracy: 80.38% | Learning Rate: 0.00098796 Epoch 15/50 | Train Loss: 0.40547, Train Accuracy: 85.96% | Val Loss: 0.48327, Val Accuracy: 83.88% | Learning Rate: 0.00098618 Epoch 16/50 | Train Loss: 0.40468, Train Accuracy: 86.12% | Val Loss: 0.50700, Val Accuracy: 82.72% | Learning Rate: 0.00098429 Epoch 17/50 | Train Loss: 0.36500, Train Accuracy: 87.30% | Val Loss: 0.40062, Val Accuracy: 86.98% | Learning Rate: 0.00098228 Epoch 18/50 | Train Loss: 0.34610, Train Accuracy: 87.90% | Val Loss: 0.44578, Val Accuracy: 85.02% | Learning Rate: 0.00098015 Epoch 19/50 | Train Loss: 0.33112, Train Accuracy: 88.36% | Val Loss: 0.42004, Val Accuracy: 86.08% | Learning Rate: 0.00097790 Epoch 20/50 | Train Loss: 0.32059, Train Accuracy: 89.02% | Val Loss: 0.38263, Val Accuracy: 86.96% | Learning Rate: 0.00097553 Epoch 21/50 | Train Loss: 0.30468, Train Accuracy: 89.46% | Val Loss: 0.45773, Val Accuracy: 85.00% | Learning Rate: 0.00097304 Epoch 22/50 | Train Loss: 0.29603, Train Accuracy: 89.75% | Val Loss: 0.36429, Val Accuracy: 87.42% | Learning Rate: 0.00097044 Epoch 23/50 | Train Loss: 0.28024, Train Accuracy: 90.12% | Val Loss: 0.37969, Val Accuracy: 86.56% | Learning Rate: 0.00096772 Epoch 24/50 | Train Loss: 0.26916, Train Accuracy: 90.58% | Val Loss: 0.38021, Val Accuracy: 87.00% | Learning Rate: 0.00096489 Epoch 25/50 | Train Loss: 0.25884, Train Accuracy: 90.91% | Val Loss: 0.37079, Val Accuracy: 87.48% | Learning Rate: 0.00096194 Epoch 26/50 | Train Loss: 0.24947, Train Accuracy: 91.13% | Val Loss: 0.32556, Val Accuracy: 88.90% | Learning Rate: 0.00095888 Epoch 27/50 | Train Loss: 0.23422, Train Accuracy: 91.79% | Val Loss: 0.38719, Val Accuracy: 87.56% | Learning Rate: 0.00095570 Epoch 28/50 | Train Loss: 0.23235, Train Accuracy: 91.89% | Val Loss: 0.32240, Val Accuracy: 88.84% | Learning Rate: 0.00095241 Epoch 29/50 | Train Loss: 0.21757, Train Accuracy: 92.41% | Val Loss: 0.35564, Val Accuracy: 88.30% | Learning Rate: 0.00094901 Epoch 30/50 | Train Loss: 0.21353, Train Accuracy: 92.50% | Val Loss: 0.34349, Val Accuracy: 88.98% | Learning Rate: 0.00094550 Epoch 31/50 | Train Loss: 0.20248, Train Accuracy: 92.82% | Val Loss: 0.33670, Val Accuracy: 88.84% | Learning Rate: 0.00094188 Epoch 32/50 | Train Loss: 0.19188, Train Accuracy: 93.20% | Val Loss: 0.43624, Val Accuracy: 88.60% | Learning Rate: 0.00093815 Epoch 33/50 | Train Loss: 0.18771, Train Accuracy: 93.54% | Val Loss: 0.34170, Val Accuracy: 88.52% | Learning Rate: 0.00093432 Epoch 34/50 | Train Loss: 0.18070, Train Accuracy: 93.63% | Val Loss: 0.31523, Val Accuracy: 89.88% | Learning Rate: 0.00093037 Epoch 35/50 | Train Loss: 0.17392, Train Accuracy: 93.98% | Val Loss: 0.29710, Val Accuracy: 90.18% | Learning Rate: 0.00092632 Epoch 36/50 | Train Loss: 0.16564, Train Accuracy: 94.16% | Val Loss: 0.29282, Val Accuracy: 90.40% | Learning Rate: 0.00092216 Epoch 37/50 | Train Loss: 0.16301, Train Accuracy: 94.11% | Val Loss: 0.31887, Val Accuracy: 89.62% | Learning Rate: 0.00091790 Epoch 38/50 | Train Loss: 0.15196, Train Accuracy: 94.58% | Val Loss: 0.32087, Val Accuracy: 89.38% | Learning Rate: 0.00091354 Epoch 39/50 | Train Loss: 0.15219, Train Accuracy: 94.57% | Val Loss: 0.32911, Val Accuracy: 89.02% | Learning Rate: 0.00090907 Epoch 40/50 | Train Loss: 0.14179, Train Accuracy: 95.00% | Val Loss: 0.31527, Val Accuracy: 90.40% | Learning Rate: 0.00090451 Epoch 41/50 | Train Loss: 0.13858, Train Accuracy: 95.13% | Val Loss: 0.31327, Val Accuracy: 90.28% | Learning Rate: 0.00089984 Epoch 42/50 | Train Loss: 0.12972, Train Accuracy: 95.37% | Val Loss: 0.33701, Val Accuracy: 89.90% | Learning Rate: 0.00089508 Epoch 43/50 | Train Loss: 0.12984, Train Accuracy: 95.46% | Val Loss: 0.30477, Val Accuracy: 90.42% | Learning Rate: 0.00089022 Epoch 44/50 | Train Loss: 0.12158, Train Accuracy: 95.66% | Val Loss: 0.32564, Val Accuracy: 90.16% | Learning Rate: 0.00088526 Epoch 45/50 | Train Loss: 0.11848, Train Accuracy: 95.92% | Val Loss: 0.33334, Val Accuracy: 89.84% | Learning Rate: 0.00088020 Epoch 46/50 | Train Loss: 0.11468, Train Accuracy: 96.08% | Val Loss: 0.73017, Val Accuracy: 84.16% | Learning Rate: 0.00087506 Epoch 47/50 | Train Loss: 0.11624, Train Accuracy: 95.90% | Val Loss: 0.31309, Val Accuracy: 90.64% | Learning Rate: 0.00086982 Epoch 48/50 | Train Loss: 0.10418, Train Accuracy: 96.36% | Val Loss: 0.37016, Val Accuracy: 89.52% | Learning Rate: 0.00086448 Epoch 49/50 | Train Loss: 0.10126, Train Accuracy: 96.39% | Val Loss: 0.32509, Val Accuracy: 90.10% | Learning Rate: 0.00085906 Epoch 50/50 | Train Loss: 0.10251, Train Accuracy: 96.39% | Val Loss: 0.31691, Val Accuracy: 90.50% | Learning Rate: 0.00085355 Total Training Time: 82.48 min Test Accuracy: 91.84%
In [19]:
def plot_history(history):
"""
Plots the training and validation loss, accuracy from a PyTorch history dictionary.
Args:
history (dict): Dictionary containing training metrics.
Expected keys: 'train_loss', 'val_loss', 'train_acc', 'val_acc'.
"""
epochs = range(len(history['train_loss']))
plt.figure(figsize=(18, 5))
# Plot 1: Loss
plt.subplot(1, 3, 1)
plt.plot(epochs, history['train_loss'], label='Training Loss')
plt.plot(epochs, history['val_loss'], label='Validation Loss')
plt.title('Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
# plt.grid(True)
# Plot 2: Accuracy
plt.subplot(1, 3, 2)
plt.plot(epochs, history['train_acc'], label='Training Accuracy')
plt.plot(epochs, history['val_acc'], label='Validation Accuracy')
plt.title('Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
# plt.grid(True)
# Plot 3: Learning Rate
plt.subplot(1, 3, 3)
plt.plot(epochs, history['lrs'], label='Learning Rate', color='orange')
plt.title('Learning Rate')
plt.xlabel('Epochs')
plt.ylabel('Learning Rate')
plt.legend()
# plt.grid(True)
plt.tight_layout()
plt.show()
# plt.savefig()
plot_history(history)
In [18]:
%watermark -iv
matplotlib : 3.10.0 numpy : 2.0.2 pandas : 2.2.2 sklearn : 1.6.1 torch : 2.8.0+cu126 torchinfo : 1.8.0 torchvision: 0.23.0+cu126
In [ ]: