Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
InĀ [1]:
!pip install watermark
Requirement already satisfied: watermark in /usr/local/lib/python3.12/dist-packages (2.6.0) Requirement already satisfied: ipython>=6.0 in /usr/local/lib/python3.12/dist-packages (from watermark) (7.34.0) Requirement already satisfied: importlib-metadata>=1.4 in /usr/local/lib/python3.12/dist-packages (from watermark) (8.7.0) Requirement already satisfied: setuptools in /usr/local/lib/python3.12/dist-packages (from watermark) (75.2.0) Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.12/dist-packages (from importlib-metadata>=1.4->watermark) (3.23.0) Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.19.2) Requirement already satisfied: decorator in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (4.4.2) Requirement already satisfied: pickleshare in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.7.5) Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (5.7.1) Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (3.0.52) Requirement already satisfied: pygments in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (2.19.2) Requirement already satisfied: backcall in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.2.0) Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.2.1) Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (4.9.0) Requirement already satisfied: parso<0.9.0,>=0.8.4 in /usr/local/lib/python3.12/dist-packages (from jedi>=0.16->ipython>=6.0->watermark) (0.8.5) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.12/dist-packages (from pexpect>4.3->ipython>=6.0->watermark) (0.7.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.0->watermark) (0.2.14)
InĀ [2]:
!pip install torchinfo
Requirement already satisfied: torchinfo in /usr/local/lib/python3.12/dist-packages (1.8.0)
InĀ [3]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 torch: 2.9.0+cu126
InĀ [4]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from sklearn.model_selection import train_test_split
from torch.amp import GradScaler, autocast
from torchinfo import summary
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
InĀ [5]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
InĀ [6]:
# Check if GPU is available and set the device
if torch.cuda.is_available():
num_gpus = torch.cuda.device_count()
print("Num GPUs Available: ", num_gpus)
for i in range(num_gpus):
gpu_name = torch.cuda.get_device_name(i)
print(f"GPU {i}: {gpu_name}")
else:
print("No GPU available.")
Num GPUs Available: 1 GPU 0: Tesla T4
ResNet: Residual BlocksĀ¶
- Single Residual Block: The fundamental innovation of a Residual Block is the "shortcut connection" (or skip connection). Instead of forcing the layers to learn the underlying mapping H(x), we let them learn the residual mapping F(x) = H(x) - x. The output becomes F(x) + x. This allows the gradient to flow through the shortcut during backpropagation, effectively solving the vanishing gradient problem and allowing for much deeper architectures.
- Various Usages of Activation in residual Block: This diagram illustrates the placement of Batch Normalization (BN) and ReLU activation relative to the convolutional layers. While the original ResNet (2015) used Post-activation (applying ReLU after the addition), subsequent research (He et al., 2016) introduced Pre-activation ResNet. In the pre-activation version, BN and ReLU are applied before the weight layers, which eases optimization and improves the regularization of the model.
Types of Residual Block: ResNet architectures use two primary building blocks depending on the depth of the network:
- Basic Block: Used in ResNet-$18$ and ResNet-$34$. It consists of two consecutive $3 \times 3$ convolutional layers.
- Bottleneck Block: Used in deeper versions like ResNet-$50$, $101$, and $152$. It uses a triplet of $1 \times 1$, $3 \times 3$, and $1 \times 1$ convolutions. The $1 \times 1$ layers are used to first reduce (bottleneck) and then restore dimensions, significantly reducing the number of parameters and computational cost while maintaining performance.
ResNet-50 Classifier on CIFAR-10Ā¶
Network Architecture: The network in this notebook is an implementation of the ResNet-50 [1] architecture on the Cifar-10 dataset to train an image classifier.
| layer name | output size | 18-layer | 34-layer | 50-layer | 101-layer | 152-layer |
|---|---|---|---|---|---|---|
| conv1 | 112Ć112 | 7Ć7, 64, stride 2 | 7Ć7, 64, stride 2 | 7Ć7, 64, stride 2 | 7Ć7, 64, stride 2 | 7Ć7, 64, stride 2 |
| conv2_x | 56Ć56 | 3Ć3 max pool, stride 2 | 3Ć3 max pool, stride 2 | 3Ć3 max pool, stride 2 | 3Ć3 max pool, stride 2 | 3Ć3 max pool, stride 2 |
| $\begin{bmatrix} 3\times3, 64 \\ 3\times3, 64 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 64 \\ 3\times3, 64 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 64 \\ 3\times3, 64 \\ 1\times1, 256 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 64 \\ 3\times3, 64 \\ 1\times1, 256 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 64 \\ 3\times3, 64 \\ 1\times1, 256 \end{bmatrix} \times 3$ | ||
| conv3_x | 28Ć28 | $\begin{bmatrix} 3\times3, 128 \\ 3\times3, 128 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 128 \\ 3\times3, 128 \end{bmatrix} \times 4$ | $\begin{bmatrix} 1\times1, 128 \\ 3\times3, 128 \\ 1\times1, 512 \end{bmatrix} \times 4$ | $\begin{bmatrix} 1\times1, 128 \\ 3\times3, 128 \\ 1\times1, 512 \end{bmatrix} \times 4$ | $\begin{bmatrix} 1\times1, 128 \\ 3\times3, 128 \\ 1\times1, 512 \end{bmatrix} \times 8$ |
| conv4_x | 14Ć14 | $\begin{bmatrix} 3\times3, 256 \\ 3\times3, 256 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 256 \\ 3\times3, 256 \end{bmatrix} \times 6$ | $\begin{bmatrix} 1\times1, 256 \\ 3\times3, 256 \\ 1\times1, 1024 \end{bmatrix} \times 6$ | $\begin{bmatrix} 1\times1, 256 \\ 3\times3, 256 \\ 1\times1, 1024 \end{bmatrix} \times 23$ | $\begin{bmatrix} 1\times1, 256 \\ 3\times3, 256 \\ 1\times1, 1024 \end{bmatrix} \times 36$ |
| conv5_x | 7Ć7 | $\begin{bmatrix} 3\times3, 512 \\ 3\times3, 512 \end{bmatrix} \times 2$ | $\begin{bmatrix} 3\times3, 512 \\ 3\times3, 512 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 512 \\ 3\times3, 512 \\ 1\times1, 2048 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 512 \\ 3\times3, 512 \\ 1\times1, 2048 \end{bmatrix} \times 3$ | $\begin{bmatrix} 1\times1, 512 \\ 3\times3, 512 \\ 1\times1, 2048 \end{bmatrix} \times 3$ |
| 1Ć1 | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | avg pool, 1000-d fc, softmax | |
| FLOPs | 1.8 Ć 10ā¹ | 3.6 Ć 10ā¹ | 3.8 Ć 10ā¹ | 7.6 Ć 10ā¹ | 11.3 Ć 10ā¹ |
Table adapted from ResNet paper
Key Notes on the Architecture:
- Building Blocks: The 18-layer and 34-layer networks use basic blocks (two 3Ć3 convs), while the deeper 50/101/152-layer networks use bottleneck blocks (1Ć1, 3Ć3, and 1Ć1 convs).
- Downsampling: Downsampling is performed by
conv3_1,conv4_1, andconv5_1with a stride of 2. - Global Average Pooling: This is used at the end of the convolutional layers to reduce the spatial dimensions to 1Ć1 before the fully connected layer.
The following figure illustrates residual blocks with skip connections such that the input passed via the shortcut matches the dimensions of the main path's output, which allows the network to learn identity functions.
The ResNet-18/34 architecture actually uses residual blocks with skip connections such that the input passed via the shortcut matches is resized to dimensions of the main path's output. Such a residual block is illustrated below:
The ResNet-50 uses a bottleneck as shown below:
Improvements over LeNet, AlexNet, VGG:
- Skip Connections: Unlike VGG and AlexNet, ResNet uses "identity mapping" which solves the vanishing gradient problem.
- Global Average Pooling: Instead of a huge Flatten + Dense layer (like in AlexNet), we use
avg_pool2d. This significantly reduces parameter count and prevents overfitting. - Batch Normalization: ResNet relies heavily on BN after every convolution to stabilize training at a higher learning rate.
REFERENCES:
He, K., Zhang, X., Ren, S., & Sun, J. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. (770-77).
Johnson, J. (2022, February 2). Lecture 8: CNN Architectures [PowerPoint Slides]. University of Michigan EECS 498/598: Deep Learning for Computer Vision. pp 77-109. https://web.eecs.umich.edu/~justincj/slides/eecs498/WI2022/598_WI2022_lecture08.pdf
Mahdid, Y. (2024.). deep-learning-model-explained: ResNet [Source code]. GitHub. https://github.com/yacineMahdid/deep-learning-model-explained/tree/main/resnet
InĀ [7]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 123
BATCH_SIZE = 128 #256
LEARNING_RATE = 0.001
NUM_EPOCHS = 100
INPUT_SIZE = 32 # CIFAR-10 is 32x32
NUM_CLASSES = 10
# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
cuda
InĀ [8]:
# Mean and Std Dev for CIFAR-10 (3 channels)
CIFAR_MEAN = (0.4914, 0.4822, 0.4465)
CIFAR_STD = (0.2471, 0.2435, 0.2616)
train_transforms = transforms.Compose([
transforms.RandomRotation(15),
transforms.RandomCrop(32, padding=4, padding_mode='reflect'),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(CIFAR_MEAN, CIFAR_STD)
])
test_transforms = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(CIFAR_MEAN, CIFAR_STD)
])
# Load Datasets
train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=train_transforms)
test_dataset = datasets.CIFAR10(root='./data', train=False, download=True, transform=test_transforms)
# Split training set into training and validation sets
train_size = int(0.9 * len(train_dataset))
val_size = len(train_dataset) - train_size
train_subset, val_subset = torch.utils.data.random_split(train_dataset, [train_size, val_size])
# Create DataLoaders
train_loader = DataLoader(train_subset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2, pin_memory=True)
val_loader = DataLoader(val_subset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)
2. Model CreationĀ¶
Notes:
- This implementation uses a $3 \times 3$ initial convolution (instead of $7 \times 7$) and removes the initial MaxPool to preserve the spatial resolution of the small $32 \times 32$ images.
InĀ [9]:
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, in_planes, planes, stride=1):
super(BasicBlock, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.relu = nn.ReLU(inplace=True)
self.shortcut = nn.Sequential()
if stride != 1 or in_planes != self.expansion*planes:
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes)
)
def forward(self, x):
residual = self.shortcut(x)
out = self.relu(self.bn1(self.conv1(x))) # conv->batchnorm->relu
out = self.bn2(self.conv2(out)) # conv->batchnorm
out += residual # skip connection
out = torch.relu(out) # relu
return out
class Bottleneck(nn.Module):
expansion = 4 # The output of the block is 4x the width of the bottleneck
def __init__(self, in_planes, planes, stride=1):
super(Bottleneck, self).__init__()
# 1x1 Convolution to reduce dimensions
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
# 3x3 Convolution for spatial processing
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
# 1x1 Convolution to expand dimensions (planes * 4)
self.conv3 = nn.Conv2d(planes, self.expansion * planes, kernel_size=1, bias=False)
self.bn3 = nn.BatchNorm2d(self.expansion * planes)
self.shortcut = nn.Sequential()
# Shortcut needs to match dimensions (if stride != 1 or channels change)
if stride != 1 or in_planes != self.expansion * planes:
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion * planes, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion * planes)
)
def forward(self, x):
residual = self.shortcut(x)
out = torch.relu(self.bn1(self.conv1(x))) # conv->batchnorm->relu
out = torch.relu(self.bn2(self.conv2(out))) # conv->batchnorm->relu
out = self.bn3(self.conv3(out)) # conv->batchnorm
out += residual # skip connection
out = torch.relu(out) # relu
return out
class ResNet(nn.Module):
def __init__(self, block, num_blocks, num_classes=10):
super(ResNet, self).__init__()
self.in_planes = 64
self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(64)
self.relu = nn.ReLU(inplace=True)
self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.linear = nn.Linear(512*block.expansion, num_classes)
def _make_layer(self, block, planes, num_blocks, stride):
strides = [stride] + [1]*(num_blocks-1)
layers = []
for s in strides:
layers.append(block(self.in_planes, planes, s))
self.in_planes = planes * block.expansion
return nn.Sequential(*layers)
def forward(self, x):
out = torch.relu(self.bn1(self.conv1(x)))
out = self.layer1(out)
out = self.layer2(out)
out = self.layer3(out)
out = self.layer4(out)
out = self.avgpool(out)
out = torch.flatten(out, 1)
out = self.linear(out)
return out
def ResNet50():
return ResNet(Bottleneck, [3, 4, 6, 3])
InĀ [10]:
def test_model_dims():
# Create a dummy batch: 2 images, 3 channels, 32x32 size
x = torch.randn(2, 3, 32, 32).to(device)
# Initialize model
net = ResNet50().to(device)
# Forward pass
y = net(x)
print(f"Input Shape: {x.shape}")
print(f"Output Shape: {y.shape}") # Should be [2, 10]
if y.shape == (2, 10):
print("ā
Sanity Check Passed: Dimensions are correct for CIFAR-10!")
else:
print("ā Sanity Check Failed: Check your stride or pooling layers.")
test_model_dims()
Input Shape: torch.Size([2, 3, 32, 32]) Output Shape: torch.Size([2, 10]) ā Sanity Check Passed: Dimensions are correct for CIFAR-10!
InĀ [11]:
set_all_seeds(RANDOM_SEED)
model = ResNet50().to(device)
InĀ [12]:
# Model parameter details
summary(model,
input_size=(BATCH_SIZE, 3, INPUT_SIZE, INPUT_SIZE),
# verbose=1,
col_names=["input_size", "output_size", "num_params", "kernel_size", "mult_adds"]
)
Out[12]:
===================================================================================================================================================================== Layer (type:depth-idx) Input Shape Output Shape Param # Kernel Shape Mult-Adds ===================================================================================================================================================================== ResNet [128, 3, 32, 32] [128, 10] -- -- -- āāConv2d: 1-1 [128, 3, 32, 32] [128, 64, 32, 32] 1,728 [3, 3] 226,492,416 āāBatchNorm2d: 1-2 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 āāSequential: 1-3 [128, 64, 32, 32] [128, 256, 32, 32] -- -- -- ā āāBottleneck: 2-1 [128, 64, 32, 32] [128, 256, 32, 32] -- -- -- ā ā āāSequential: 3-1 [128, 64, 32, 32] [128, 256, 32, 32] 16,896 -- 2,147,549,184 ā ā āāConv2d: 3-2 [128, 64, 32, 32] [128, 64, 32, 32] 4,096 [1, 1] 536,870,912 ā ā āāBatchNorm2d: 3-3 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ā ā āāConv2d: 3-4 [128, 64, 32, 32] [128, 64, 32, 32] 36,864 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-5 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ā ā āāConv2d: 3-6 [128, 64, 32, 32] [128, 256, 32, 32] 16,384 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-7 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 ā āāBottleneck: 2-2 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- ā ā āāSequential: 3-8 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- ā ā āāConv2d: 3-9 [128, 256, 32, 32] [128, 64, 32, 32] 16,384 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-10 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ā ā āāConv2d: 3-11 [128, 64, 32, 32] [128, 64, 32, 32] 36,864 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-12 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ā ā āāConv2d: 3-13 [128, 64, 32, 32] [128, 256, 32, 32] 16,384 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-14 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 ā āāBottleneck: 2-3 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- ā ā āāSequential: 3-15 [128, 256, 32, 32] [128, 256, 32, 32] -- -- -- ā ā āāConv2d: 3-16 [128, 256, 32, 32] [128, 64, 32, 32] 16,384 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-17 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ā ā āāConv2d: 3-18 [128, 64, 32, 32] [128, 64, 32, 32] 36,864 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-19 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 ā ā āāConv2d: 3-20 [128, 64, 32, 32] [128, 256, 32, 32] 16,384 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-21 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 āāSequential: 1-4 [128, 256, 32, 32] [128, 512, 16, 16] -- -- -- ā āāBottleneck: 2-4 [128, 256, 32, 32] [128, 512, 16, 16] -- -- -- ā ā āāSequential: 3-22 [128, 256, 32, 32] [128, 512, 16, 16] 132,096 -- 4,295,098,368 ā ā āāConv2d: 3-23 [128, 256, 32, 32] [128, 128, 32, 32] 32,768 [1, 1] 4,294,967,296 ā ā āāBatchNorm2d: 3-24 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 ā ā āāConv2d: 3-25 [128, 128, 32, 32] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-26 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-27 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-28 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 ā āāBottleneck: 2-5 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- ā ā āāSequential: 3-29 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- ā ā āāConv2d: 3-30 [128, 512, 16, 16] [128, 128, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-31 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-32 [128, 128, 16, 16] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-33 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-34 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-35 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 ā āāBottleneck: 2-6 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- ā ā āāSequential: 3-36 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- ā ā āāConv2d: 3-37 [128, 512, 16, 16] [128, 128, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-38 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-39 [128, 128, 16, 16] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-40 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-41 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-42 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 ā āāBottleneck: 2-7 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- ā ā āāSequential: 3-43 [128, 512, 16, 16] [128, 512, 16, 16] -- -- -- ā ā āāConv2d: 3-44 [128, 512, 16, 16] [128, 128, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-45 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-46 [128, 128, 16, 16] [128, 128, 16, 16] 147,456 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-47 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 ā ā āāConv2d: 3-48 [128, 128, 16, 16] [128, 512, 16, 16] 65,536 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-49 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 āāSequential: 1-5 [128, 512, 16, 16] [128, 1024, 8, 8] -- -- -- ā āāBottleneck: 2-8 [128, 512, 16, 16] [128, 1024, 8, 8] -- -- -- ā ā āāSequential: 3-50 [128, 512, 16, 16] [128, 1024, 8, 8] 526,336 -- 4,295,229,440 ā ā āāConv2d: 3-51 [128, 512, 16, 16] [128, 256, 16, 16] 131,072 [1, 1] 4,294,967,296 ā ā āāBatchNorm2d: 3-52 [128, 256, 16, 16] [128, 256, 16, 16] 512 -- 65,536 ā ā āāConv2d: 3-53 [128, 256, 16, 16] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-54 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-55 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-56 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ā āāBottleneck: 2-9 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāSequential: 3-57 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāConv2d: 3-58 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-59 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-60 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-61 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-62 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-63 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ā āāBottleneck: 2-10 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāSequential: 3-64 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāConv2d: 3-65 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-66 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-67 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-68 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-69 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-70 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ā āāBottleneck: 2-11 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāSequential: 3-71 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāConv2d: 3-72 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-73 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-74 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-75 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-76 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-77 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ā āāBottleneck: 2-12 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāSequential: 3-78 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāConv2d: 3-79 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-80 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-81 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-82 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-83 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-84 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ā āāBottleneck: 2-13 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāSequential: 3-85 [128, 1024, 8, 8] [128, 1024, 8, 8] -- -- -- ā ā āāConv2d: 3-86 [128, 1024, 8, 8] [128, 256, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-87 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-88 [128, 256, 8, 8] [128, 256, 8, 8] 589,824 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-89 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 ā ā āāConv2d: 3-90 [128, 256, 8, 8] [128, 1024, 8, 8] 262,144 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-91 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 āāSequential: 1-6 [128, 1024, 8, 8] [128, 2048, 4, 4] -- -- -- ā āāBottleneck: 2-14 [128, 1024, 8, 8] [128, 2048, 4, 4] -- -- -- ā ā āāSequential: 3-92 [128, 1024, 8, 8] [128, 2048, 4, 4] 2,101,248 -- 4,295,491,584 ā ā āāConv2d: 3-93 [128, 1024, 8, 8] [128, 512, 8, 8] 524,288 [1, 1] 4,294,967,296 ā ā āāBatchNorm2d: 3-94 [128, 512, 8, 8] [128, 512, 8, 8] 1,024 -- 131,072 ā ā āāConv2d: 3-95 [128, 512, 8, 8] [128, 512, 4, 4] 2,359,296 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-96 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 ā ā āāConv2d: 3-97 [128, 512, 4, 4] [128, 2048, 4, 4] 1,048,576 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-98 [128, 2048, 4, 4] [128, 2048, 4, 4] 4,096 -- 524,288 ā āāBottleneck: 2-15 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- ā ā āāSequential: 3-99 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- ā ā āāConv2d: 3-100 [128, 2048, 4, 4] [128, 512, 4, 4] 1,048,576 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-101 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 ā ā āāConv2d: 3-102 [128, 512, 4, 4] [128, 512, 4, 4] 2,359,296 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-103 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 ā ā āāConv2d: 3-104 [128, 512, 4, 4] [128, 2048, 4, 4] 1,048,576 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-105 [128, 2048, 4, 4] [128, 2048, 4, 4] 4,096 -- 524,288 ā āāBottleneck: 2-16 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- ā ā āāSequential: 3-106 [128, 2048, 4, 4] [128, 2048, 4, 4] -- -- -- ā ā āāConv2d: 3-107 [128, 2048, 4, 4] [128, 512, 4, 4] 1,048,576 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-108 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 ā ā āāConv2d: 3-109 [128, 512, 4, 4] [128, 512, 4, 4] 2,359,296 [3, 3] 4,831,838,208 ā ā āāBatchNorm2d: 3-110 [128, 512, 4, 4] [128, 512, 4, 4] 1,024 -- 131,072 ā ā āāConv2d: 3-111 [128, 512, 4, 4] [128, 2048, 4, 4] 1,048,576 [1, 1] 2,147,483,648 ā ā āāBatchNorm2d: 3-112 [128, 2048, 4, 4] [128, 2048, 4, 4] 4,096 -- 524,288 āāAdaptiveAvgPool2d: 1-7 [128, 2048, 4, 4] [128, 2048, 1, 1] -- -- -- āāLinear: 1-8 [128, 2048] [128, 10] 20,490 -- 2,622,720 ===================================================================================================================================================================== Total params: 23,520,842 Trainable params: 23,520,842 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 166.13 ===================================================================================================================================================================== Input size (MB): 1.57 Forward/backward pass size (MB): 7029.66 Params size (MB): 94.08 Estimated Total Size (MB): 7125.32 =====================================================================================================================================================================
InĀ [13]:
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)
# optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=200)
scaler = GradScaler('cuda')
InĀ [14]:
# Function to calculate accuracy
def calculate_accuracy(loader):
model.eval()
correct = 0
total = 0
with torch.no_grad():
for inputs, labels in loader:
inputs, labels = inputs.to(device), labels.to(device)
with autocast('cuda'):
outputs = model(inputs)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
return 100 * correct / total
# Training Loop
start_time = time.time()
history = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': [], 'lrs': []}
print(f"Starting training on {device}...")
for epoch in range(NUM_EPOCHS):
model.train()
running_loss, correct_train, total_train = 0.0, 0, 0
for i, (inputs, labels) in enumerate(train_loader):
inputs, labels = inputs.to(device), labels.to(device)
optimizer.zero_grad()
with autocast('cuda'):
outputs = model(inputs)
loss = criterion(outputs, labels)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
running_loss += loss.item() * inputs.size(0)
_, predicted = torch.max(outputs, 1)
total_train += labels.size(0)
correct_train += (predicted == labels).sum().item()
train_loss = running_loss / len(train_subset)
train_acc = 100 * correct_train / total_train
# Calculate validation loss and accuracy
model.eval()
val_loss, correct_val, total_val = 0.0, 0, 0
with torch.no_grad():
for inputs, labels in val_loader:
inputs, labels = inputs.to(device), labels.to(device)
with autocast('cuda'):
outputs = model(inputs)
loss = criterion(outputs, labels)
val_loss += loss.item() * inputs.size(0)
_, predicted = torch.max(outputs, 1)
total_val += labels.size(0)
correct_val += (predicted == labels).sum().item()
val_loss /= len(val_subset)
val_acc = 100 * correct_val / total_val
scheduler.step()
current_lr = optimizer.param_groups[0]['lr']
history['train_loss'].append(train_loss)
history['val_loss'].append(val_loss)
history['train_acc'].append(train_acc)
history['val_acc'].append(val_acc)
history['lrs'].append(current_lr)
print(f'Epoch {epoch + 1}/{NUM_EPOCHS} | '
f'Train Loss: {train_loss:.5f}, '
f'Train Accuracy: {train_acc:.2f}% | '
f'Val Loss: {val_loss:.5f}, '
f'Val Accuracy: {val_acc:.2f}% | '
f'Learning Rate: {current_lr:.8f}')
print('\nTotal Training Time: %.2f min' % ((time.time() - start_time)/60))
# Final Evaluation
test_accuracy = calculate_accuracy(test_loader)
print(f'\nTest Accuracy: {test_accuracy:.2f}%')
Starting training on cuda... Epoch 1/100 | Train Loss: 1.68479, Train Accuracy: 39.20% | Val Loss: 8.23171, Val Accuracy: 25.90% | Learning Rate: 0.00099994 Epoch 2/100 | Train Loss: 1.27674, Train Accuracy: 53.57% | Val Loss: 1.13974, Val Accuracy: 58.40% | Learning Rate: 0.00099975 Epoch 3/100 | Train Loss: 1.06348, Train Accuracy: 62.07% | Val Loss: 1.05485, Val Accuracy: 63.28% | Learning Rate: 0.00099944 Epoch 4/100 | Train Loss: 0.93236, Train Accuracy: 66.96% | Val Loss: 0.99227, Val Accuracy: 64.44% | Learning Rate: 0.00099901 Epoch 5/100 | Train Loss: 0.83547, Train Accuracy: 70.56% | Val Loss: 0.98299, Val Accuracy: 67.70% | Learning Rate: 0.00099846 Epoch 6/100 | Train Loss: 0.74949, Train Accuracy: 73.80% | Val Loss: 0.78712, Val Accuracy: 73.04% | Learning Rate: 0.00099778 Epoch 7/100 | Train Loss: 0.68434, Train Accuracy: 76.23% | Val Loss: 0.72795, Val Accuracy: 75.00% | Learning Rate: 0.00099698 Epoch 8/100 | Train Loss: 0.62323, Train Accuracy: 78.27% | Val Loss: 0.68470, Val Accuracy: 76.54% | Learning Rate: 0.00099606 Epoch 9/100 | Train Loss: 0.57522, Train Accuracy: 80.06% | Val Loss: 0.61790, Val Accuracy: 78.54% | Learning Rate: 0.00099501 Epoch 10/100 | Train Loss: 0.54272, Train Accuracy: 81.22% | Val Loss: 0.57593, Val Accuracy: 79.84% | Learning Rate: 0.00099384 Epoch 11/100 | Train Loss: 0.51755, Train Accuracy: 81.93% | Val Loss: 0.70098, Val Accuracy: 77.82% | Learning Rate: 0.00099255 Epoch 12/100 | Train Loss: 0.48993, Train Accuracy: 82.91% | Val Loss: 0.58946, Val Accuracy: 79.48% | Learning Rate: 0.00099114 Epoch 13/100 | Train Loss: 0.46773, Train Accuracy: 83.70% | Val Loss: 0.55465, Val Accuracy: 81.66% | Learning Rate: 0.00098961 Epoch 14/100 | Train Loss: 0.43728, Train Accuracy: 84.79% | Val Loss: 0.52569, Val Accuracy: 81.92% | Learning Rate: 0.00098796 Epoch 15/100 | Train Loss: 0.41841, Train Accuracy: 85.39% | Val Loss: 0.52094, Val Accuracy: 82.04% | Learning Rate: 0.00098618 Epoch 16/100 | Train Loss: 0.40263, Train Accuracy: 85.98% | Val Loss: 0.46924, Val Accuracy: 84.02% | Learning Rate: 0.00098429 Epoch 17/100 | Train Loss: 0.38727, Train Accuracy: 86.49% | Val Loss: 0.51035, Val Accuracy: 82.94% | Learning Rate: 0.00098228 Epoch 18/100 | Train Loss: 0.37088, Train Accuracy: 87.04% | Val Loss: 0.49431, Val Accuracy: 82.88% | Learning Rate: 0.00098015 Epoch 19/100 | Train Loss: 0.34892, Train Accuracy: 87.96% | Val Loss: 0.53233, Val Accuracy: 82.62% | Learning Rate: 0.00097790 Epoch 20/100 | Train Loss: 0.33990, Train Accuracy: 88.14% | Val Loss: 0.42996, Val Accuracy: 85.86% | Learning Rate: 0.00097553 Epoch 21/100 | Train Loss: 0.32223, Train Accuracy: 88.75% | Val Loss: 0.45012, Val Accuracy: 84.96% | Learning Rate: 0.00097304 Epoch 22/100 | Train Loss: 0.31307, Train Accuracy: 89.17% | Val Loss: 0.48949, Val Accuracy: 83.72% | Learning Rate: 0.00097044 Epoch 23/100 | Train Loss: 0.29532, Train Accuracy: 89.59% | Val Loss: 0.39340, Val Accuracy: 86.26% | Learning Rate: 0.00096772 Epoch 24/100 | Train Loss: 0.29214, Train Accuracy: 89.79% | Val Loss: 0.39544, Val Accuracy: 86.40% | Learning Rate: 0.00096489 Epoch 25/100 | Train Loss: 0.27305, Train Accuracy: 90.39% | Val Loss: 0.42701, Val Accuracy: 85.20% | Learning Rate: 0.00096194 Epoch 26/100 | Train Loss: 0.26470, Train Accuracy: 90.81% | Val Loss: 0.41314, Val Accuracy: 87.80% | Learning Rate: 0.00095888 Epoch 27/100 | Train Loss: 0.25956, Train Accuracy: 90.93% | Val Loss: 0.41408, Val Accuracy: 86.50% | Learning Rate: 0.00095570 Epoch 28/100 | Train Loss: 0.24204, Train Accuracy: 91.57% | Val Loss: 0.35460, Val Accuracy: 88.40% | Learning Rate: 0.00095241 Epoch 29/100 | Train Loss: 0.23766, Train Accuracy: 91.77% | Val Loss: 0.39048, Val Accuracy: 87.00% | Learning Rate: 0.00094901 Epoch 30/100 | Train Loss: 0.22540, Train Accuracy: 91.97% | Val Loss: 0.35636, Val Accuracy: 87.70% | Learning Rate: 0.00094550 Epoch 31/100 | Train Loss: 0.21376, Train Accuracy: 92.56% | Val Loss: 0.43152, Val Accuracy: 86.50% | Learning Rate: 0.00094188 Epoch 32/100 | Train Loss: 0.21005, Train Accuracy: 92.41% | Val Loss: 0.36545, Val Accuracy: 88.14% | Learning Rate: 0.00093815 Epoch 33/100 | Train Loss: 0.20067, Train Accuracy: 92.90% | Val Loss: 0.34037, Val Accuracy: 88.78% | Learning Rate: 0.00093432 Epoch 34/100 | Train Loss: 0.19222, Train Accuracy: 93.20% | Val Loss: 0.37795, Val Accuracy: 87.96% | Learning Rate: 0.00093037 Epoch 35/100 | Train Loss: 0.18639, Train Accuracy: 93.52% | Val Loss: 0.36051, Val Accuracy: 88.20% | Learning Rate: 0.00092632 Epoch 36/100 | Train Loss: 0.17928, Train Accuracy: 93.69% | Val Loss: 0.36817, Val Accuracy: 88.24% | Learning Rate: 0.00092216 Epoch 37/100 | Train Loss: 0.16976, Train Accuracy: 93.99% | Val Loss: 0.33700, Val Accuracy: 89.32% | Learning Rate: 0.00091790 Epoch 38/100 | Train Loss: 0.16750, Train Accuracy: 94.25% | Val Loss: 0.37073, Val Accuracy: 88.20% | Learning Rate: 0.00091354 Epoch 39/100 | Train Loss: 0.15700, Train Accuracy: 94.45% | Val Loss: 0.47850, Val Accuracy: 85.46% | Learning Rate: 0.00090907 Epoch 40/100 | Train Loss: 0.16049, Train Accuracy: 94.23% | Val Loss: 0.33806, Val Accuracy: 89.52% | Learning Rate: 0.00090451 Epoch 41/100 | Train Loss: 0.15086, Train Accuracy: 94.66% | Val Loss: 0.32877, Val Accuracy: 89.60% | Learning Rate: 0.00089984 Epoch 42/100 | Train Loss: 0.14441, Train Accuracy: 94.95% | Val Loss: 0.35669, Val Accuracy: 89.36% | Learning Rate: 0.00089508 Epoch 43/100 | Train Loss: 0.14018, Train Accuracy: 95.04% | Val Loss: 0.33891, Val Accuracy: 88.88% | Learning Rate: 0.00089022 Epoch 44/100 | Train Loss: 0.13380, Train Accuracy: 95.33% | Val Loss: 0.34134, Val Accuracy: 89.36% | Learning Rate: 0.00088526 Epoch 45/100 | Train Loss: 0.12450, Train Accuracy: 95.54% | Val Loss: 0.33519, Val Accuracy: 90.18% | Learning Rate: 0.00088020 Epoch 46/100 | Train Loss: 0.12511, Train Accuracy: 95.54% | Val Loss: 0.33836, Val Accuracy: 89.36% | Learning Rate: 0.00087506 Epoch 47/100 | Train Loss: 0.12066, Train Accuracy: 95.71% | Val Loss: 0.37871, Val Accuracy: 89.16% | Learning Rate: 0.00086982 Epoch 48/100 | Train Loss: 0.12085, Train Accuracy: 95.88% | Val Loss: 0.36020, Val Accuracy: 89.30% | Learning Rate: 0.00086448 Epoch 49/100 | Train Loss: 0.11147, Train Accuracy: 96.11% | Val Loss: 0.34076, Val Accuracy: 89.68% | Learning Rate: 0.00085906 Epoch 50/100 | Train Loss: 0.10818, Train Accuracy: 96.18% | Val Loss: 0.41493, Val Accuracy: 88.24% | Learning Rate: 0.00085355 Epoch 51/100 | Train Loss: 0.10732, Train Accuracy: 96.18% | Val Loss: 0.34973, Val Accuracy: 89.56% | Learning Rate: 0.00084796 Epoch 52/100 | Train Loss: 0.10261, Train Accuracy: 96.39% | Val Loss: 0.33579, Val Accuracy: 90.42% | Learning Rate: 0.00084227 Epoch 53/100 | Train Loss: 0.09287, Train Accuracy: 96.73% | Val Loss: 0.34471, Val Accuracy: 90.04% | Learning Rate: 0.00083651 Epoch 54/100 | Train Loss: 0.09541, Train Accuracy: 96.60% | Val Loss: 0.30876, Val Accuracy: 90.80% | Learning Rate: 0.00083066 Epoch 55/100 | Train Loss: 0.08941, Train Accuracy: 96.90% | Val Loss: 0.37591, Val Accuracy: 89.72% | Learning Rate: 0.00082472 Epoch 56/100 | Train Loss: 0.08920, Train Accuracy: 96.78% | Val Loss: 0.35401, Val Accuracy: 90.06% | Learning Rate: 0.00081871 Epoch 57/100 | Train Loss: 0.08642, Train Accuracy: 96.97% | Val Loss: 0.34533, Val Accuracy: 90.38% | Learning Rate: 0.00081262 Epoch 58/100 | Train Loss: 0.07891, Train Accuracy: 97.17% | Val Loss: 0.36477, Val Accuracy: 90.38% | Learning Rate: 0.00080645 Epoch 59/100 | Train Loss: 0.07792, Train Accuracy: 97.22% | Val Loss: 0.35051, Val Accuracy: 90.28% | Learning Rate: 0.00080021 Epoch 60/100 | Train Loss: 0.07880, Train Accuracy: 97.19% | Val Loss: 0.36324, Val Accuracy: 89.40% | Learning Rate: 0.00079389 Epoch 61/100 | Train Loss: 0.07476, Train Accuracy: 97.36% | Val Loss: 0.34877, Val Accuracy: 90.34% | Learning Rate: 0.00078750 Epoch 62/100 | Train Loss: 0.07058, Train Accuracy: 97.51% | Val Loss: 0.33860, Val Accuracy: 90.60% | Learning Rate: 0.00078104 Epoch 63/100 | Train Loss: 0.07174, Train Accuracy: 97.59% | Val Loss: 0.34371, Val Accuracy: 90.52% | Learning Rate: 0.00077451 Epoch 64/100 | Train Loss: 0.06815, Train Accuracy: 97.60% | Val Loss: 0.36183, Val Accuracy: 90.48% | Learning Rate: 0.00076791 Epoch 65/100 | Train Loss: 0.06911, Train Accuracy: 97.59% | Val Loss: 0.37606, Val Accuracy: 90.02% | Learning Rate: 0.00076125 Epoch 66/100 | Train Loss: 0.06555, Train Accuracy: 97.67% | Val Loss: 0.33831, Val Accuracy: 90.58% | Learning Rate: 0.00075452 Epoch 67/100 | Train Loss: 0.06625, Train Accuracy: 97.67% | Val Loss: 0.34669, Val Accuracy: 90.62% | Learning Rate: 0.00074773 Epoch 68/100 | Train Loss: 0.06047, Train Accuracy: 97.83% | Val Loss: 0.37616, Val Accuracy: 90.16% | Learning Rate: 0.00074088 Epoch 69/100 | Train Loss: 0.05909, Train Accuracy: 97.95% | Val Loss: 0.35047, Val Accuracy: 90.76% | Learning Rate: 0.00073396 Epoch 70/100 | Train Loss: 0.05549, Train Accuracy: 98.02% | Val Loss: 0.35381, Val Accuracy: 90.62% | Learning Rate: 0.00072700 Epoch 71/100 | Train Loss: 0.05548, Train Accuracy: 98.06% | Val Loss: 0.37252, Val Accuracy: 90.84% | Learning Rate: 0.00071997 Epoch 72/100 | Train Loss: 0.05487, Train Accuracy: 98.14% | Val Loss: 0.33494, Val Accuracy: 91.28% | Learning Rate: 0.00071289 Epoch 73/100 | Train Loss: 0.05343, Train Accuracy: 98.14% | Val Loss: 0.35435, Val Accuracy: 90.82% | Learning Rate: 0.00070576 Epoch 74/100 | Train Loss: 0.05399, Train Accuracy: 98.08% | Val Loss: 0.36311, Val Accuracy: 91.06% | Learning Rate: 0.00069857 Epoch 75/100 | Train Loss: 0.05115, Train Accuracy: 98.18% | Val Loss: 0.34578, Val Accuracy: 91.44% | Learning Rate: 0.00069134 Epoch 76/100 | Train Loss: 0.04872, Train Accuracy: 98.30% | Val Loss: 0.39113, Val Accuracy: 90.62% | Learning Rate: 0.00068406 Epoch 77/100 | Train Loss: 0.04698, Train Accuracy: 98.36% | Val Loss: 0.36687, Val Accuracy: 91.44% | Learning Rate: 0.00067674 Epoch 78/100 | Train Loss: 0.04604, Train Accuracy: 98.41% | Val Loss: 0.37298, Val Accuracy: 90.84% | Learning Rate: 0.00066937 Epoch 79/100 | Train Loss: 0.04270, Train Accuracy: 98.53% | Val Loss: 0.38693, Val Accuracy: 90.80% | Learning Rate: 0.00066196 Epoch 80/100 | Train Loss: 0.04940, Train Accuracy: 98.27% | Val Loss: 0.40585, Val Accuracy: 90.62% | Learning Rate: 0.00065451 Epoch 81/100 | Train Loss: 0.04322, Train Accuracy: 98.56% | Val Loss: 0.36105, Val Accuracy: 91.04% | Learning Rate: 0.00064702 Epoch 82/100 | Train Loss: 0.04111, Train Accuracy: 98.52% | Val Loss: 0.35413, Val Accuracy: 91.04% | Learning Rate: 0.00063950 Epoch 83/100 | Train Loss: 0.04098, Train Accuracy: 98.62% | Val Loss: 0.37258, Val Accuracy: 91.02% | Learning Rate: 0.00063194 Epoch 84/100 | Train Loss: 0.03631, Train Accuracy: 98.77% | Val Loss: 0.35972, Val Accuracy: 91.18% | Learning Rate: 0.00062434 Epoch 85/100 | Train Loss: 0.03580, Train Accuracy: 98.69% | Val Loss: 0.44476, Val Accuracy: 90.08% | Learning Rate: 0.00061672 Epoch 86/100 | Train Loss: 0.03877, Train Accuracy: 98.59% | Val Loss: 0.37136, Val Accuracy: 91.04% | Learning Rate: 0.00060907 Epoch 87/100 | Train Loss: 0.03915, Train Accuracy: 98.64% | Val Loss: 0.36113, Val Accuracy: 91.32% | Learning Rate: 0.00060139 Epoch 88/100 | Train Loss: 0.03352, Train Accuracy: 98.81% | Val Loss: 0.34581, Val Accuracy: 91.70% | Learning Rate: 0.00059369 Epoch 89/100 | Train Loss: 0.03423, Train Accuracy: 98.82% | Val Loss: 0.36797, Val Accuracy: 91.48% | Learning Rate: 0.00058596 Epoch 90/100 | Train Loss: 0.03280, Train Accuracy: 98.95% | Val Loss: 0.39512, Val Accuracy: 91.16% | Learning Rate: 0.00057822 Epoch 91/100 | Train Loss: 0.03448, Train Accuracy: 98.83% | Val Loss: 0.38865, Val Accuracy: 91.38% | Learning Rate: 0.00057045 Epoch 92/100 | Train Loss: 0.02971, Train Accuracy: 99.02% | Val Loss: 0.36548, Val Accuracy: 91.72% | Learning Rate: 0.00056267 Epoch 93/100 | Train Loss: 0.02815, Train Accuracy: 98.98% | Val Loss: 0.38599, Val Accuracy: 91.48% | Learning Rate: 0.00055487 Epoch 94/100 | Train Loss: 0.03330, Train Accuracy: 98.84% | Val Loss: 0.34104, Val Accuracy: 91.54% | Learning Rate: 0.00054705 Epoch 95/100 | Train Loss: 0.02700, Train Accuracy: 99.08% | Val Loss: 0.38822, Val Accuracy: 91.18% | Learning Rate: 0.00053923 Epoch 96/100 | Train Loss: 0.02940, Train Accuracy: 98.90% | Val Loss: 0.36798, Val Accuracy: 91.66% | Learning Rate: 0.00053140 Epoch 97/100 | Train Loss: 0.02646, Train Accuracy: 99.06% | Val Loss: 0.37093, Val Accuracy: 91.96% | Learning Rate: 0.00052355 Epoch 98/100 | Train Loss: 0.02615, Train Accuracy: 99.07% | Val Loss: 0.39168, Val Accuracy: 91.14% | Learning Rate: 0.00051571 Epoch 99/100 | Train Loss: 0.02754, Train Accuracy: 99.03% | Val Loss: 0.37287, Val Accuracy: 91.28% | Learning Rate: 0.00050785 Epoch 100/100 | Train Loss: 0.02128, Train Accuracy: 99.28% | Val Loss: 0.37799, Val Accuracy: 92.04% | Learning Rate: 0.00050000 Total Training Time: 98.61 min Test Accuracy: 92.81%
InĀ [15]:
def plot_history(history):
"""
Plots the training and validation loss, accuracy, and learning rate
from a PyTorch history dictionary.
Args:
history (dict): Dictionary containing training metrics.
Expected keys: 'train_loss', 'val_loss', 'train_acc', 'val_acc', 'lrs'.
"""
epochs = range(len(history['train_loss']))
plt.figure(figsize=(18, 5))
# Plot 1: Loss
plt.subplot(1, 3, 1)
plt.plot(epochs, history['train_loss'], label='Training Loss')
plt.plot(epochs, history['val_loss'], label='Validation Loss')
plt.title('Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
# plt.grid(True)
# Plot 2: Accuracy
plt.subplot(1, 3, 2)
plt.plot(epochs, history['train_acc'], label='Training Accuracy')
plt.plot(epochs, history['val_acc'], label='Validation Accuracy')
plt.title('Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
# plt.grid(True)
# Plot 3: Learning Rate
plt.subplot(1, 3, 3)
plt.plot(epochs, history['lrs'], label='Learning Rate', color='orange')
plt.title('Learning Rate')
plt.xlabel('Epochs')
plt.ylabel('Learning Rate')
plt.legend()
# plt.grid(True)
plt.tight_layout()
plt.show()
# plt.savefig()
InĀ [16]:
plot_history(history)
InĀ [17]:
%watermark -iv
matplotlib : 3.10.0 numpy : 2.0.2 pandas : 2.2.2 sklearn : 1.6.1 torch : 2.9.0+cu126 torchinfo : 1.8.0 torchvision: 0.24.0+cu126