Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [1]:
!pip install watermark
Collecting watermark Downloading watermark-2.6.0-py3-none-any.whl.metadata (15 kB) Requirement already satisfied: ipython>=6.0 in /usr/local/lib/python3.11/dist-packages (from watermark) (7.34.0) Requirement already satisfied: importlib-metadata>=1.4 in /usr/local/lib/python3.11/dist-packages (from watermark) (8.7.0) Requirement already satisfied: setuptools in /usr/local/lib/python3.11/dist-packages (from watermark) (75.2.0) Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.11/dist-packages (from importlib-metadata>=1.4->watermark) (3.23.0) Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (0.19.2) Requirement already satisfied: decorator in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (4.4.2) Requirement already satisfied: pickleshare in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (0.7.5) Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (5.7.1) Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (3.0.51) Requirement already satisfied: pygments in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (2.19.2) Requirement already satisfied: backcall in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (0.2.0) Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (0.1.7) Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.11/dist-packages (from ipython>=6.0->watermark) (4.9.0) Requirement already satisfied: parso<0.9.0,>=0.8.4 in /usr/local/lib/python3.11/dist-packages (from jedi>=0.16->ipython>=6.0->watermark) (0.8.4) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.11/dist-packages (from pexpect>4.3->ipython>=6.0->watermark) (0.7.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.11/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.0->watermark) (0.2.13) Downloading watermark-2.6.0-py3-none-any.whl (13 kB) Installing collected packages: watermark Successfully installed watermark-2.6.0
In [2]:
!pip install torchinfo
Requirement already satisfied: torchinfo in /usr/local/lib/python3.11/dist-packages (1.8.0)
In [3]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p torch
Author: Chizoba Obasi Python implementation: CPython Python version : 3.11.13 IPython version : 7.34.0 torch: 2.6.0+cu124
DenseNet-121 Classifier on CIFAR-10¶
Network Architecture:
The network in this notebook is an implementation of the DenseNet-121 [1] architecture on the CIFAR-10 dataset to train an image classifier.
DenseNet: Dense Block¶
The DenseNet paper introduced the idea that instead of connecting layers only to their immediate successors (like standard CNNs) or adding residual connections (like ResNet), each layer should receive inputs from all preceding layers and pass its own feature-maps to all subsequent layers. This helps alleviate the vanishing-gradient problem and encourages feature reuse.
DenseNet is characterized by Dense Blocks, where each layer receives the concatenated feature maps of all preceding layers, and Transition Layers, which reduce the spatial dimensions and the number of channels between blocks. The following figure below illustrates the main concept of DenseNet: within each "dense" block, each layer is connected with each previous layer -- the feature maps are concatenated.
Note that this is somewhat related yet very different to ResNets. ResNets have skip connections approx. between every other layer (but don't connect all layers with each other). Also, ResNets skip connections work via addition
$$\mathbf{x}_{\ell}=H_{\ell}\left(\mathbf{X}_{\ell-1}\right)+\mathbf{X}_{\ell-1}$$
whereas $H_{\ell}(\cdot)$ can be a composite function of operations such as Batch Normalization (BN), rectified linear units (ReLU), Pooling, or Convolution (Conv).
In DenseNets, all the previous feature maps $\mathbf{X}_{0}, \dots, \mathbf{X}_{\ell}-1$ of a feature map $\mathbf{X}_{\ell}$ are concatenated:
$$\mathbf{x}_{\ell}=H_{\ell}\left(\left[\mathbf{x}_{0}, \mathbf{x}_{1}, \ldots, \mathbf{x}_{\ell-1}\right]\right).$$
Furthermore, in this particular notebook, we are considering the DenseNet-121, which is depicted below:
REFERENCES:
- Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708).
In [4]:
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from sklearn.model_selection import train_test_split
from torchinfo import summary
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
import os
import random
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
In [5]:
def set_all_seeds(seed):
os.environ["PL_GLOBAL_SEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
def set_deterministic():
if torch.cuda.is_available():
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.set_deterministic(True)
In [6]:
# Check if GPU is available and set the device
if torch.cuda.is_available():
num_gpus = torch.cuda.device_count()
print("Num GPUs Available: ", num_gpus)
for i in range(num_gpus):
gpu_name = torch.cuda.get_device_name(i)
print(f"GPU {i}: {gpu_name}")
else:
print("No GPU available.")
Num GPUs Available: 2 GPU 0: Tesla T4 GPU 1: Tesla T4
In [7]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 123
BATCH_SIZE = 128 #256
LEARNING_RATE = 0.001
NUM_EPOCHS = 50
INPUT_SIZE = 32 # CIFAR-10 is 32x32
NUM_CLASSES = 10
# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
cuda
In [8]:
set_all_seeds(RANDOM_SEED)
In [9]:
# Mean and Std Dev for CIFAR-10 (3 channels)
CIFAR_MEAN = (0.4914, 0.4822, 0.4465)
CIFAR_STD = (0.2471, 0.2435, 0.2616)
# Data Augmentation (Crucial for VGG-16 on CIFAR-10)
train_transforms = transforms.Compose([
transforms.RandomCrop(32, padding=4), # Crop after padding
transforms.RandomHorizontalFlip(), # Flip
transforms.ToTensor(),
transforms.Normalize(CIFAR_MEAN, CIFAR_STD) # Mean Subtraction
])
# Test/Validation Transforms (No Augmentation)
test_transforms = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(CIFAR_MEAN, CIFAR_STD)
])
# Load Datasets
train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=train_transforms)
test_dataset = datasets.CIFAR10(root='./data', train=False, download=True, transform=test_transforms)
# Split training set into training and validation sets
train_size = int(0.9 * len(train_dataset))
val_size = len(train_dataset) - train_size
train_subset, val_subset = torch.utils.data.random_split(train_dataset, [train_size, val_size])
# Create DataLoaders
train_loader = DataLoader(train_subset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2, pin_memory=True)
val_loader = DataLoader(val_subset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)
100%|██████████| 170M/170M [00:04<00:00, 40.6MB/s]
In [10]:
# Check that shuffling works properly
# i.e., label indices should be in random order.
# Also, the label order should be different in the second
# epoch.
for images, labels in train_loader:
pass
print(labels[:10])
for images, labels in train_loader:
pass
print(labels[:10])
tensor([1, 5, 5, 4, 2, 8, 5, 2, 4, 2]) tensor([9, 2, 4, 3, 7, 5, 8, 6, 2, 8])
In [11]:
# Check that validation set and test sets are diverse
# i.e., that they contain all classes
for images, labels in val_loader:
pass
print(labels[:10])
for images, labels in test_loader:
pass
print(labels[:10])
tensor([1, 6, 5, 9, 9, 0, 6, 6]) tensor([7, 5, 8, 0, 8, 2, 7, 0, 3, 5])
In [12]:
class Bottleneck(nn.Module):
def __init__(self, in_planes, growth_rate):
super(Bottleneck, self).__init__()
self.bn1 = nn.BatchNorm2d(in_planes)
self.conv1 = nn.Conv2d(in_planes, 4*growth_rate, kernel_size=1, bias=False)
self.bn2 = nn.BatchNorm2d(4*growth_rate)
self.conv2 = nn.Conv2d(4*growth_rate, growth_rate, kernel_size=3, padding=1, bias=False)
def forward(self, x):
out = self.conv1(F.relu(self.bn1(x)))
out = self.conv2(F.relu(self.bn2(out)))
# Concatenate input with the output of this layer
out = torch.cat([out, x], 1)
return out
class Transition(nn.Module):
def __init__(self, in_planes, out_planes):
super(Transition, self).__init__()
self.bn = nn.BatchNorm2d(in_planes)
self.conv = nn.Conv2d(in_planes, out_planes, kernel_size=1, bias=False)
def forward(self, x):
out = self.conv(F.relu(self.bn(x)))
out = F.avg_pool2d(out, 2)
return out
class DenseNet(nn.Module):
def __init__(self, block, nblocks, growth_rate=12, reduction=0.5, num_classes=10):
super(DenseNet, self).__init__()
self.growth_rate = growth_rate
num_planes = 2*growth_rate
self.conv1 = nn.Conv2d(3, num_planes, kernel_size=3, padding=1, bias=False)
self.dense1 = self._make_dense_layers(block, num_planes, nblocks[0])
num_planes += nblocks[0]*growth_rate
out_planes = int(num_planes*reduction)
self.trans1 = Transition(num_planes, out_planes)
num_planes = out_planes
self.dense2 = self._make_dense_layers(block, num_planes, nblocks[1])
num_planes += nblocks[1]*growth_rate
out_planes = int(num_planes*reduction)
self.trans2 = Transition(num_planes, out_planes)
num_planes = out_planes
self.dense3 = self._make_dense_layers(block, num_planes, nblocks[2])
num_planes += nblocks[2]*growth_rate
self.bn = nn.BatchNorm2d(num_planes)
self.linear = nn.Linear(num_planes, num_classes)
self._initialize_weights()
def _initialize_weights(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
nn.init.kaiming_normal_(m.weight)
nn.init.constant_(m.bias, 0)
def _make_dense_layers(self, block, in_planes, nblock):
layers = []
for i in range(nblock):
layers.append(block(in_planes, self.growth_rate))
in_planes += self.growth_rate
return nn.Sequential(*layers)
def forward(self, x):
out = self.conv1(x)
out = self.trans1(self.dense1(out))
out = self.trans2(self.dense2(out))
out = self.dense3(out)
out = F.avg_pool2d(F.relu(self.bn(out)), 8)
out = out.view(out.size(0), -1)
out = self.linear(out)
return out
def DenseNet121():
return DenseNet(Bottleneck, [6, 12, 24, 16], growth_rate=32)
In [13]:
model = DenseNet121().to(device)
In [14]:
# Model parameter details
summary(model,
input_size=(BATCH_SIZE, 3, INPUT_SIZE, INPUT_SIZE),
# verbose=1,
col_names=["input_size", "output_size", "num_params", "kernel_size", "mult_adds"]
)
Out[14]:
===================================================================================================================================================================== Layer (type:depth-idx) Input Shape Output Shape Param # Kernel Shape Mult-Adds ===================================================================================================================================================================== DenseNet [128, 3, 32, 32] [128, 10] -- -- -- ├─Conv2d: 1-1 [128, 3, 32, 32] [128, 64, 32, 32] 1,728 [3, 3] 226,492,416 ├─Sequential: 1-2 [128, 64, 32, 32] [128, 256, 32, 32] -- -- -- │ └─Bottleneck: 2-1 [128, 64, 32, 32] [128, 96, 32, 32] -- -- -- │ │ └─BatchNorm2d: 3-1 [128, 64, 32, 32] [128, 64, 32, 32] 128 -- 16,384 │ │ └─Conv2d: 3-2 [128, 64, 32, 32] [128, 128, 32, 32] 8,192 [1, 1] 1,073,741,824 │ │ └─BatchNorm2d: 3-3 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-4 [128, 128, 32, 32] [128, 32, 32, 32] 36,864 [3, 3] 4,831,838,208 │ └─Bottleneck: 2-2 [128, 96, 32, 32] [128, 128, 32, 32] -- -- -- │ │ └─BatchNorm2d: 3-5 [128, 96, 32, 32] [128, 96, 32, 32] 192 -- 24,576 │ │ └─Conv2d: 3-6 [128, 96, 32, 32] [128, 128, 32, 32] 12,288 [1, 1] 1,610,612,736 │ │ └─BatchNorm2d: 3-7 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-8 [128, 128, 32, 32] [128, 32, 32, 32] 36,864 [3, 3] 4,831,838,208 │ └─Bottleneck: 2-3 [128, 128, 32, 32] [128, 160, 32, 32] -- -- -- │ │ └─BatchNorm2d: 3-9 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-10 [128, 128, 32, 32] [128, 128, 32, 32] 16,384 [1, 1] 2,147,483,648 │ │ └─BatchNorm2d: 3-11 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-12 [128, 128, 32, 32] [128, 32, 32, 32] 36,864 [3, 3] 4,831,838,208 │ └─Bottleneck: 2-4 [128, 160, 32, 32] [128, 192, 32, 32] -- -- -- │ │ └─BatchNorm2d: 3-13 [128, 160, 32, 32] [128, 160, 32, 32] 320 -- 40,960 │ │ └─Conv2d: 3-14 [128, 160, 32, 32] [128, 128, 32, 32] 20,480 [1, 1] 2,684,354,560 │ │ └─BatchNorm2d: 3-15 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-16 [128, 128, 32, 32] [128, 32, 32, 32] 36,864 [3, 3] 4,831,838,208 │ └─Bottleneck: 2-5 [128, 192, 32, 32] [128, 224, 32, 32] -- -- -- │ │ └─BatchNorm2d: 3-17 [128, 192, 32, 32] [128, 192, 32, 32] 384 -- 49,152 │ │ └─Conv2d: 3-18 [128, 192, 32, 32] [128, 128, 32, 32] 24,576 [1, 1] 3,221,225,472 │ │ └─BatchNorm2d: 3-19 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-20 [128, 128, 32, 32] [128, 32, 32, 32] 36,864 [3, 3] 4,831,838,208 │ └─Bottleneck: 2-6 [128, 224, 32, 32] [128, 256, 32, 32] -- -- -- │ │ └─BatchNorm2d: 3-21 [128, 224, 32, 32] [128, 224, 32, 32] 448 -- 57,344 │ │ └─Conv2d: 3-22 [128, 224, 32, 32] [128, 128, 32, 32] 28,672 [1, 1] 3,758,096,384 │ │ └─BatchNorm2d: 3-23 [128, 128, 32, 32] [128, 128, 32, 32] 256 -- 32,768 │ │ └─Conv2d: 3-24 [128, 128, 32, 32] [128, 32, 32, 32] 36,864 [3, 3] 4,831,838,208 ├─Transition: 1-3 [128, 256, 32, 32] [128, 128, 16, 16] -- -- -- │ └─BatchNorm2d: 2-7 [128, 256, 32, 32] [128, 256, 32, 32] 512 -- 65,536 │ └─Conv2d: 2-8 [128, 256, 32, 32] [128, 128, 32, 32] 32,768 [1, 1] 4,294,967,296 ├─Sequential: 1-4 [128, 128, 16, 16] [128, 512, 16, 16] -- -- -- │ └─Bottleneck: 2-9 [128, 128, 16, 16] [128, 160, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-25 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-26 [128, 128, 16, 16] [128, 128, 16, 16] 16,384 [1, 1] 536,870,912 │ │ └─BatchNorm2d: 3-27 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-28 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-10 [128, 160, 16, 16] [128, 192, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-29 [128, 160, 16, 16] [128, 160, 16, 16] 320 -- 40,960 │ │ └─Conv2d: 3-30 [128, 160, 16, 16] [128, 128, 16, 16] 20,480 [1, 1] 671,088,640 │ │ └─BatchNorm2d: 3-31 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-32 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-11 [128, 192, 16, 16] [128, 224, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-33 [128, 192, 16, 16] [128, 192, 16, 16] 384 -- 49,152 │ │ └─Conv2d: 3-34 [128, 192, 16, 16] [128, 128, 16, 16] 24,576 [1, 1] 805,306,368 │ │ └─BatchNorm2d: 3-35 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-36 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-12 [128, 224, 16, 16] [128, 256, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-37 [128, 224, 16, 16] [128, 224, 16, 16] 448 -- 57,344 │ │ └─Conv2d: 3-38 [128, 224, 16, 16] [128, 128, 16, 16] 28,672 [1, 1] 939,524,096 │ │ └─BatchNorm2d: 3-39 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-40 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-13 [128, 256, 16, 16] [128, 288, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-41 [128, 256, 16, 16] [128, 256, 16, 16] 512 -- 65,536 │ │ └─Conv2d: 3-42 [128, 256, 16, 16] [128, 128, 16, 16] 32,768 [1, 1] 1,073,741,824 │ │ └─BatchNorm2d: 3-43 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-44 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-14 [128, 288, 16, 16] [128, 320, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-45 [128, 288, 16, 16] [128, 288, 16, 16] 576 -- 73,728 │ │ └─Conv2d: 3-46 [128, 288, 16, 16] [128, 128, 16, 16] 36,864 [1, 1] 1,207,959,552 │ │ └─BatchNorm2d: 3-47 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-48 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-15 [128, 320, 16, 16] [128, 352, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-49 [128, 320, 16, 16] [128, 320, 16, 16] 640 -- 81,920 │ │ └─Conv2d: 3-50 [128, 320, 16, 16] [128, 128, 16, 16] 40,960 [1, 1] 1,342,177,280 │ │ └─BatchNorm2d: 3-51 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-52 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-16 [128, 352, 16, 16] [128, 384, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-53 [128, 352, 16, 16] [128, 352, 16, 16] 704 -- 90,112 │ │ └─Conv2d: 3-54 [128, 352, 16, 16] [128, 128, 16, 16] 45,056 [1, 1] 1,476,395,008 │ │ └─BatchNorm2d: 3-55 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-56 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-17 [128, 384, 16, 16] [128, 416, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-57 [128, 384, 16, 16] [128, 384, 16, 16] 768 -- 98,304 │ │ └─Conv2d: 3-58 [128, 384, 16, 16] [128, 128, 16, 16] 49,152 [1, 1] 1,610,612,736 │ │ └─BatchNorm2d: 3-59 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-60 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-18 [128, 416, 16, 16] [128, 448, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-61 [128, 416, 16, 16] [128, 416, 16, 16] 832 -- 106,496 │ │ └─Conv2d: 3-62 [128, 416, 16, 16] [128, 128, 16, 16] 53,248 [1, 1] 1,744,830,464 │ │ └─BatchNorm2d: 3-63 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-64 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-19 [128, 448, 16, 16] [128, 480, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-65 [128, 448, 16, 16] [128, 448, 16, 16] 896 -- 114,688 │ │ └─Conv2d: 3-66 [128, 448, 16, 16] [128, 128, 16, 16] 57,344 [1, 1] 1,879,048,192 │ │ └─BatchNorm2d: 3-67 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-68 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 │ └─Bottleneck: 2-20 [128, 480, 16, 16] [128, 512, 16, 16] -- -- -- │ │ └─BatchNorm2d: 3-69 [128, 480, 16, 16] [128, 480, 16, 16] 960 -- 122,880 │ │ └─Conv2d: 3-70 [128, 480, 16, 16] [128, 128, 16, 16] 61,440 [1, 1] 2,013,265,920 │ │ └─BatchNorm2d: 3-71 [128, 128, 16, 16] [128, 128, 16, 16] 256 -- 32,768 │ │ └─Conv2d: 3-72 [128, 128, 16, 16] [128, 32, 16, 16] 36,864 [3, 3] 1,207,959,552 ├─Transition: 1-5 [128, 512, 16, 16] [128, 256, 8, 8] -- -- -- │ └─BatchNorm2d: 2-21 [128, 512, 16, 16] [128, 512, 16, 16] 1,024 -- 131,072 │ └─Conv2d: 2-22 [128, 512, 16, 16] [128, 256, 16, 16] 131,072 [1, 1] 4,294,967,296 ├─Sequential: 1-6 [128, 256, 8, 8] [128, 1024, 8, 8] -- -- -- │ └─Bottleneck: 2-23 [128, 256, 8, 8] [128, 288, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-73 [128, 256, 8, 8] [128, 256, 8, 8] 512 -- 65,536 │ │ └─Conv2d: 3-74 [128, 256, 8, 8] [128, 128, 8, 8] 32,768 [1, 1] 268,435,456 │ │ └─BatchNorm2d: 3-75 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-76 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-24 [128, 288, 8, 8] [128, 320, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-77 [128, 288, 8, 8] [128, 288, 8, 8] 576 -- 73,728 │ │ └─Conv2d: 3-78 [128, 288, 8, 8] [128, 128, 8, 8] 36,864 [1, 1] 301,989,888 │ │ └─BatchNorm2d: 3-79 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-80 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-25 [128, 320, 8, 8] [128, 352, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-81 [128, 320, 8, 8] [128, 320, 8, 8] 640 -- 81,920 │ │ └─Conv2d: 3-82 [128, 320, 8, 8] [128, 128, 8, 8] 40,960 [1, 1] 335,544,320 │ │ └─BatchNorm2d: 3-83 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-84 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-26 [128, 352, 8, 8] [128, 384, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-85 [128, 352, 8, 8] [128, 352, 8, 8] 704 -- 90,112 │ │ └─Conv2d: 3-86 [128, 352, 8, 8] [128, 128, 8, 8] 45,056 [1, 1] 369,098,752 │ │ └─BatchNorm2d: 3-87 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-88 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-27 [128, 384, 8, 8] [128, 416, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-89 [128, 384, 8, 8] [128, 384, 8, 8] 768 -- 98,304 │ │ └─Conv2d: 3-90 [128, 384, 8, 8] [128, 128, 8, 8] 49,152 [1, 1] 402,653,184 │ │ └─BatchNorm2d: 3-91 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-92 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-28 [128, 416, 8, 8] [128, 448, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-93 [128, 416, 8, 8] [128, 416, 8, 8] 832 -- 106,496 │ │ └─Conv2d: 3-94 [128, 416, 8, 8] [128, 128, 8, 8] 53,248 [1, 1] 436,207,616 │ │ └─BatchNorm2d: 3-95 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-96 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-29 [128, 448, 8, 8] [128, 480, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-97 [128, 448, 8, 8] [128, 448, 8, 8] 896 -- 114,688 │ │ └─Conv2d: 3-98 [128, 448, 8, 8] [128, 128, 8, 8] 57,344 [1, 1] 469,762,048 │ │ └─BatchNorm2d: 3-99 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-100 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-30 [128, 480, 8, 8] [128, 512, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-101 [128, 480, 8, 8] [128, 480, 8, 8] 960 -- 122,880 │ │ └─Conv2d: 3-102 [128, 480, 8, 8] [128, 128, 8, 8] 61,440 [1, 1] 503,316,480 │ │ └─BatchNorm2d: 3-103 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-104 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-31 [128, 512, 8, 8] [128, 544, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-105 [128, 512, 8, 8] [128, 512, 8, 8] 1,024 -- 131,072 │ │ └─Conv2d: 3-106 [128, 512, 8, 8] [128, 128, 8, 8] 65,536 [1, 1] 536,870,912 │ │ └─BatchNorm2d: 3-107 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-108 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-32 [128, 544, 8, 8] [128, 576, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-109 [128, 544, 8, 8] [128, 544, 8, 8] 1,088 -- 139,264 │ │ └─Conv2d: 3-110 [128, 544, 8, 8] [128, 128, 8, 8] 69,632 [1, 1] 570,425,344 │ │ └─BatchNorm2d: 3-111 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-112 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-33 [128, 576, 8, 8] [128, 608, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-113 [128, 576, 8, 8] [128, 576, 8, 8] 1,152 -- 147,456 │ │ └─Conv2d: 3-114 [128, 576, 8, 8] [128, 128, 8, 8] 73,728 [1, 1] 603,979,776 │ │ └─BatchNorm2d: 3-115 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-116 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-34 [128, 608, 8, 8] [128, 640, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-117 [128, 608, 8, 8] [128, 608, 8, 8] 1,216 -- 155,648 │ │ └─Conv2d: 3-118 [128, 608, 8, 8] [128, 128, 8, 8] 77,824 [1, 1] 637,534,208 │ │ └─BatchNorm2d: 3-119 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-120 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-35 [128, 640, 8, 8] [128, 672, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-121 [128, 640, 8, 8] [128, 640, 8, 8] 1,280 -- 163,840 │ │ └─Conv2d: 3-122 [128, 640, 8, 8] [128, 128, 8, 8] 81,920 [1, 1] 671,088,640 │ │ └─BatchNorm2d: 3-123 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-124 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-36 [128, 672, 8, 8] [128, 704, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-125 [128, 672, 8, 8] [128, 672, 8, 8] 1,344 -- 172,032 │ │ └─Conv2d: 3-126 [128, 672, 8, 8] [128, 128, 8, 8] 86,016 [1, 1] 704,643,072 │ │ └─BatchNorm2d: 3-127 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-128 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-37 [128, 704, 8, 8] [128, 736, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-129 [128, 704, 8, 8] [128, 704, 8, 8] 1,408 -- 180,224 │ │ └─Conv2d: 3-130 [128, 704, 8, 8] [128, 128, 8, 8] 90,112 [1, 1] 738,197,504 │ │ └─BatchNorm2d: 3-131 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-132 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-38 [128, 736, 8, 8] [128, 768, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-133 [128, 736, 8, 8] [128, 736, 8, 8] 1,472 -- 188,416 │ │ └─Conv2d: 3-134 [128, 736, 8, 8] [128, 128, 8, 8] 94,208 [1, 1] 771,751,936 │ │ └─BatchNorm2d: 3-135 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-136 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-39 [128, 768, 8, 8] [128, 800, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-137 [128, 768, 8, 8] [128, 768, 8, 8] 1,536 -- 196,608 │ │ └─Conv2d: 3-138 [128, 768, 8, 8] [128, 128, 8, 8] 98,304 [1, 1] 805,306,368 │ │ └─BatchNorm2d: 3-139 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-140 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-40 [128, 800, 8, 8] [128, 832, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-141 [128, 800, 8, 8] [128, 800, 8, 8] 1,600 -- 204,800 │ │ └─Conv2d: 3-142 [128, 800, 8, 8] [128, 128, 8, 8] 102,400 [1, 1] 838,860,800 │ │ └─BatchNorm2d: 3-143 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-144 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-41 [128, 832, 8, 8] [128, 864, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-145 [128, 832, 8, 8] [128, 832, 8, 8] 1,664 -- 212,992 │ │ └─Conv2d: 3-146 [128, 832, 8, 8] [128, 128, 8, 8] 106,496 [1, 1] 872,415,232 │ │ └─BatchNorm2d: 3-147 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-148 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-42 [128, 864, 8, 8] [128, 896, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-149 [128, 864, 8, 8] [128, 864, 8, 8] 1,728 -- 221,184 │ │ └─Conv2d: 3-150 [128, 864, 8, 8] [128, 128, 8, 8] 110,592 [1, 1] 905,969,664 │ │ └─BatchNorm2d: 3-151 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-152 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-43 [128, 896, 8, 8] [128, 928, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-153 [128, 896, 8, 8] [128, 896, 8, 8] 1,792 -- 229,376 │ │ └─Conv2d: 3-154 [128, 896, 8, 8] [128, 128, 8, 8] 114,688 [1, 1] 939,524,096 │ │ └─BatchNorm2d: 3-155 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-156 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-44 [128, 928, 8, 8] [128, 960, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-157 [128, 928, 8, 8] [128, 928, 8, 8] 1,856 -- 237,568 │ │ └─Conv2d: 3-158 [128, 928, 8, 8] [128, 128, 8, 8] 118,784 [1, 1] 973,078,528 │ │ └─BatchNorm2d: 3-159 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-160 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-45 [128, 960, 8, 8] [128, 992, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-161 [128, 960, 8, 8] [128, 960, 8, 8] 1,920 -- 245,760 │ │ └─Conv2d: 3-162 [128, 960, 8, 8] [128, 128, 8, 8] 122,880 [1, 1] 1,006,632,960 │ │ └─BatchNorm2d: 3-163 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-164 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 │ └─Bottleneck: 2-46 [128, 992, 8, 8] [128, 1024, 8, 8] -- -- -- │ │ └─BatchNorm2d: 3-165 [128, 992, 8, 8] [128, 992, 8, 8] 1,984 -- 253,952 │ │ └─Conv2d: 3-166 [128, 992, 8, 8] [128, 128, 8, 8] 126,976 [1, 1] 1,040,187,392 │ │ └─BatchNorm2d: 3-167 [128, 128, 8, 8] [128, 128, 8, 8] 256 -- 32,768 │ │ └─Conv2d: 3-168 [128, 128, 8, 8] [128, 32, 8, 8] 36,864 [3, 3] 301,989,888 ├─BatchNorm2d: 1-7 [128, 1024, 8, 8] [128, 1024, 8, 8] 2,048 -- 262,144 ├─Linear: 1-8 [128, 1024] [128, 10] 10,250 -- 1,312,000 ===================================================================================================================================================================== Total params: 4,271,882 Trainable params: 4,271,882 Non-trainable params: 0 Total mult-adds (Units.GIGABYTES): 105.06 ===================================================================================================================================================================== Input size (MB): 1.57 Forward/backward pass size (MB): 6752.84 Params size (MB): 17.09 Estimated Total Size (MB): 6771.50 =====================================================================================================================================================================
In [15]:
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=5e-4)
# optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9, weight_decay=5e-4) # Use weight decay (L2) to combat overfitting
# scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=5)
def calculate_accuracy(loader):
model.eval()
correct = 0
total = 0
with torch.no_grad():
for inputs, labels in loader:
inputs, labels = inputs.to(device), labels.to(device)
outputs = model(inputs)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
return 100 * correct / total
# Training Loop
start_time = time.time()
history = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': [], 'lrs': []}
print(f"Starting training on {device}...")
for epoch in range(NUM_EPOCHS):
model.train()
running_loss = 0.0
for i, (inputs, labels) in enumerate(train_loader):
inputs, labels = inputs.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item() * inputs.size(0)
train_loss = running_loss / len(train_subset)
train_acc = calculate_accuracy(train_loader)
# Calculate validation loss and accuracy
model.eval()
val_loss = 0.0
with torch.no_grad():
for inputs, labels in val_loader:
inputs, labels = inputs.to(device), labels.to(device)
outputs = model(inputs)
loss = criterion(outputs, labels)
val_loss += loss.item() * inputs.size(0)
val_loss /= len(val_subset)
val_acc = calculate_accuracy(val_loader)
# scheduler.step(val_loss)
# current_lr = optimizer.param_groups[0]['lr']
# Log metrics
history['train_loss'].append(train_loss)
history['val_loss'].append(val_loss)
history['train_acc'].append(train_acc)
history['val_acc'].append(val_acc)
# history['lrs'].append(current_lr)
print(f'Epoch {epoch + 1}/{NUM_EPOCHS} | '
f'Train Loss: {train_loss:.5f}, '
f'Train Acc.: {train_acc:.2f}% | '
f'Val Loss: {val_loss:.5f}, '
f'Val Acc.: {val_acc:.2f}% | '
# f'LR: {current_lr:.8f}')
)
print('\nTotal Training Time: %.2f min' % ((time.time() - start_time)/60))
# --- Final Evaluation ---
test_accuracy = calculate_accuracy(test_loader)
print(f'\nTest Accuracy: {test_accuracy:.2f}%')
Starting training on cuda... Epoch 1/50 | Train Loss: 1.28938, Train Acc.: 60.14% | Val Loss: 1.16412, Val Acc.: 58.40% | Epoch 2/50 | Train Loss: 0.85381, Train Acc.: 64.32% | Val Loss: 1.18293, Val Acc.: 62.02% | Epoch 3/50 | Train Loss: 0.69381, Train Acc.: 73.73% | Val Loss: 0.81697, Val Acc.: 71.98% | Epoch 4/50 | Train Loss: 0.61682, Train Acc.: 76.76% | Val Loss: 0.73867, Val Acc.: 75.22% | Epoch 5/50 | Train Loss: 0.55805, Train Acc.: 78.57% | Val Loss: 0.69625, Val Acc.: 76.86% | Epoch 6/50 | Train Loss: 0.51576, Train Acc.: 78.87% | Val Loss: 0.66618, Val Acc.: 77.32% | Epoch 7/50 | Train Loss: 0.47948, Train Acc.: 79.74% | Val Loss: 0.65859, Val Acc.: 77.82% | Epoch 8/50 | Train Loss: 0.44716, Train Acc.: 84.11% | Val Loss: 0.53837, Val Acc.: 81.94% | Epoch 9/50 | Train Loss: 0.42224, Train Acc.: 84.77% | Val Loss: 0.53360, Val Acc.: 82.02% | Epoch 10/50 | Train Loss: 0.40031, Train Acc.: 84.69% | Val Loss: 0.54779, Val Acc.: 81.76% | Epoch 11/50 | Train Loss: 0.38732, Train Acc.: 86.89% | Val Loss: 0.45152, Val Acc.: 84.08% | Epoch 12/50 | Train Loss: 0.36669, Train Acc.: 86.41% | Val Loss: 0.49049, Val Acc.: 83.72% | Epoch 13/50 | Train Loss: 0.35379, Train Acc.: 88.97% | Val Loss: 0.42019, Val Acc.: 85.24% | Epoch 14/50 | Train Loss: 0.33676, Train Acc.: 88.28% | Val Loss: 0.43587, Val Acc.: 84.86% | Epoch 15/50 | Train Loss: 0.33249, Train Acc.: 88.13% | Val Loss: 0.44740, Val Acc.: 84.80% | Epoch 16/50 | Train Loss: 0.32420, Train Acc.: 86.14% | Val Loss: 0.51135, Val Acc.: 83.10% | Epoch 17/50 | Train Loss: 0.31054, Train Acc.: 89.94% | Val Loss: 0.39764, Val Acc.: 86.64% | Epoch 18/50 | Train Loss: 0.29667, Train Acc.: 90.16% | Val Loss: 0.37759, Val Acc.: 86.80% | Epoch 19/50 | Train Loss: 0.29461, Train Acc.: 88.26% | Val Loss: 0.45316, Val Acc.: 84.14% | Epoch 20/50 | Train Loss: 0.28682, Train Acc.: 90.48% | Val Loss: 0.39970, Val Acc.: 86.38% | Epoch 21/50 | Train Loss: 0.27553, Train Acc.: 90.94% | Val Loss: 0.39202, Val Acc.: 87.20% | Epoch 22/50 | Train Loss: 0.26756, Train Acc.: 91.03% | Val Loss: 0.38528, Val Acc.: 86.50% | Epoch 23/50 | Train Loss: 0.26530, Train Acc.: 90.72% | Val Loss: 0.38940, Val Acc.: 87.10% | Epoch 24/50 | Train Loss: 0.25869, Train Acc.: 89.58% | Val Loss: 0.42679, Val Acc.: 85.34% | Epoch 25/50 | Train Loss: 0.25475, Train Acc.: 92.16% | Val Loss: 0.34798, Val Acc.: 87.50% | Epoch 26/50 | Train Loss: 0.24929, Train Acc.: 90.30% | Val Loss: 0.41339, Val Acc.: 86.08% | Epoch 27/50 | Train Loss: 0.24653, Train Acc.: 91.32% | Val Loss: 0.37720, Val Acc.: 87.06% | Epoch 28/50 | Train Loss: 0.23998, Train Acc.: 91.57% | Val Loss: 0.37020, Val Acc.: 87.62% | Epoch 29/50 | Train Loss: 0.23568, Train Acc.: 91.68% | Val Loss: 0.36186, Val Acc.: 87.60% | Epoch 30/50 | Train Loss: 0.23346, Train Acc.: 91.33% | Val Loss: 0.38213, Val Acc.: 87.08% | Epoch 31/50 | Train Loss: 0.23313, Train Acc.: 91.59% | Val Loss: 0.39230, Val Acc.: 87.06% | Epoch 32/50 | Train Loss: 0.22949, Train Acc.: 91.02% | Val Loss: 0.40955, Val Acc.: 86.66% | Epoch 33/50 | Train Loss: 0.22137, Train Acc.: 92.97% | Val Loss: 0.34216, Val Acc.: 88.20% | Epoch 34/50 | Train Loss: 0.21845, Train Acc.: 91.97% | Val Loss: 0.35844, Val Acc.: 87.38% | Epoch 35/50 | Train Loss: 0.21806, Train Acc.: 93.13% | Val Loss: 0.34781, Val Acc.: 88.78% | Epoch 36/50 | Train Loss: 0.21142, Train Acc.: 92.91% | Val Loss: 0.35315, Val Acc.: 87.90% | Epoch 37/50 | Train Loss: 0.20953, Train Acc.: 92.86% | Val Loss: 0.36286, Val Acc.: 88.42% | Epoch 38/50 | Train Loss: 0.21127, Train Acc.: 92.63% | Val Loss: 0.36896, Val Acc.: 87.66% | Epoch 39/50 | Train Loss: 0.20944, Train Acc.: 92.99% | Val Loss: 0.36131, Val Acc.: 88.58% | Epoch 40/50 | Train Loss: 0.20694, Train Acc.: 93.29% | Val Loss: 0.33641, Val Acc.: 88.56% | Epoch 41/50 | Train Loss: 0.20343, Train Acc.: 93.79% | Val Loss: 0.31670, Val Acc.: 88.92% | Epoch 42/50 | Train Loss: 0.20029, Train Acc.: 93.69% | Val Loss: 0.33835, Val Acc.: 88.72% | Epoch 43/50 | Train Loss: 0.20297, Train Acc.: 93.05% | Val Loss: 0.34384, Val Acc.: 88.16% | Epoch 44/50 | Train Loss: 0.19723, Train Acc.: 92.44% | Val Loss: 0.36636, Val Acc.: 87.08% | Epoch 45/50 | Train Loss: 0.19548, Train Acc.: 93.15% | Val Loss: 0.33696, Val Acc.: 88.48% | Epoch 46/50 | Train Loss: 0.19373, Train Acc.: 92.93% | Val Loss: 0.35015, Val Acc.: 88.52% | Epoch 47/50 | Train Loss: 0.19753, Train Acc.: 92.82% | Val Loss: 0.35248, Val Acc.: 88.26% | Epoch 48/50 | Train Loss: 0.19052, Train Acc.: 94.26% | Val Loss: 0.32415, Val Acc.: 89.18% | Epoch 49/50 | Train Loss: 0.18848, Train Acc.: 92.78% | Val Loss: 0.35649, Val Acc.: 87.52% | Epoch 50/50 | Train Loss: 0.19074, Train Acc.: 94.20% | Val Loss: 0.32554, Val Acc.: 88.90% | Total Training Time: 153.70 min Test Accuracy: 89.73%
In [16]:
# --- Final Evaluation ---
test_accuracy = calculate_accuracy(test_loader)
print(f'\nTest Accuracy: {test_accuracy:.2f}%')
Test Accuracy: 89.73%
In [17]:
def plot_history(history):
"""
Plots the training and validation loss, accuracy, and learning rate
from a PyTorch history dictionary.
Args:
history (dict): Dictionary containing training metrics.
Expected keys: 'train_loss', 'val_loss', 'train_acc', 'val_acc', 'lrs'.
"""
epochs = range(len(history['train_loss']))
plt.figure(figsize=(18, 5))
# Plot 1: Loss
plt.subplot(1, 3, 1)
plt.plot(epochs, history['train_loss'], label='Training Loss')
plt.plot(epochs, history['val_loss'], label='Validation Loss')
plt.title('Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
# plt.grid(True)
# Plot 2: Accuracy
plt.subplot(1, 3, 2)
plt.plot(epochs, history['train_acc'], label='Training Accuracy')
plt.plot(epochs, history['val_acc'], label='Validation Accuracy')
plt.title('Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
# plt.grid(True)
plt.tight_layout()
plt.show()
# plt.savefig()
plot_history(history)
In [18]:
from sklearn.metrics import confusion_matrix
import seaborn as sns
def _compute_confusion_matrix(model, data_loader, device):
"""
Computes a confusion matrix for the given model and data.
Uses scikit-learn for robustness and speed.
"""
model.eval()
all_targets = []
all_predictions = []
with torch.no_grad():
for features, targets in data_loader:
features = features.to(device)
logits = model(features)
_, predicted_labels = torch.max(logits, 1)
# Move to CPU and convert to list/numpy immediately
all_targets.extend(targets.view(-1).cpu().numpy())
all_predictions.extend(predicted_labels.view(-1).cpu().numpy())
return confusion_matrix(all_targets, all_predictions)
def _plot_confusion_matrix(mat, class_names):
"""
Visualizes the confusion matrix using Seaborn for professional reporting.
"""
plt.figure(figsize=(10, 8))
sns.heatmap(mat, annot=True, fmt='d', cmap='Blues',
xticklabels=class_names, yticklabels=class_names)
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.title('Confusion Matrix: MobileNetV2 on CIFAR-10')
plt.show()
In [19]:
class_dict = {0: 'plane',
1: 'car',
2: 'bird',
3: 'cat',
4: 'deer',
5: 'dog',
6: 'frog',
7: 'horse',
8: 'ship',
9: 'truck'}
_mat = _compute_confusion_matrix(model=model, data_loader=test_loader, device=device)
_plot_confusion_matrix(_mat, class_names=class_dict.values())
In [20]:
%watermark -iv
matplotlib : 3.7.2 numpy : 1.26.4 pandas : 2.2.3 seaborn : 0.12.2 sklearn : 1.2.2 torch : 2.6.0+cu124 torchinfo : 1.8.0 torchvision: 0.21.0+cu124
In [ ]: