Deep Learning Models: A collection of various deep learning architectures, models, & their implementations (PyTorch or TensorFlow) in Jupyter Notebooks.
- Author: Chizoba Obasi
- Github Repo: https://github.com/chizkidd/deep-learning-models
In [34]:
!pip install watermark
Requirement already satisfied: watermark in /usr/local/lib/python3.12/dist-packages (2.5.0) Requirement already satisfied: ipython>=6.0 in /usr/local/lib/python3.12/dist-packages (from watermark) (7.34.0) Requirement already satisfied: importlib-metadata>=1.4 in /usr/local/lib/python3.12/dist-packages (from watermark) (8.7.0) Requirement already satisfied: setuptools in /usr/local/lib/python3.12/dist-packages (from watermark) (75.2.0) Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.12/dist-packages (from importlib-metadata>=1.4->watermark) (3.23.0) Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.19.2) Requirement already satisfied: decorator in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (4.4.2) Requirement already satisfied: pickleshare in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.7.5) Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (5.7.1) Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (3.0.52) Requirement already satisfied: pygments in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (2.19.2) Requirement already satisfied: backcall in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.2.0) Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (0.2.1) Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.12/dist-packages (from ipython>=6.0->watermark) (4.9.0) Requirement already satisfied: parso<0.9.0,>=0.8.4 in /usr/local/lib/python3.12/dist-packages (from jedi>=0.16->ipython>=6.0->watermark) (0.8.5) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.12/dist-packages (from pexpect>4.3->ipython>=6.0->watermark) (0.7.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.0->watermark) (0.2.14)
In [35]:
%load_ext watermark
# %reload_ext watermark
%watermark -a 'Chizoba Obasi' -v -p tensorflow
The watermark extension is already loaded. To reload it, use: %reload_ext watermark Author: Chizoba Obasi Python implementation: CPython Python version : 3.12.12 IPython version : 7.34.0 tensorflow: 2.19.0
Implementation of LeNet-5¶
LeNet-5 is a classic convolutional neural network employing the use of convolutions, pooling and fully connected layers that was developed in 1998 by Yann Lecun, Leon Bottou, Yoshua Bengio & Patrick Haffner. It was designed for handwritten digit recognition using the MNIST (Modified National Institute of Standards and Technology) dataset. Animated demonstrations of LeNet-5 can be found here. The original LeNet-5 paper, "Gradient Based Learning Applied to Document Recognition," can be found here or here. According to Yann Lecun, on his LeNet5 demo page:
Convolutional Neural Networks are are a special kind of multi-layer neural networks. Like almost every other neural networks they are trained with a version of the back-propagation algorithm. Where they differ is in the architecture.
Convolutional Neural Networks are designed to recognize visual patterns directly from pixel images with minimal preprocessing. They can recognize patterns with extreme variability (such as handwritten characters), and with robustness to distortions and simple geometric transformations.
LeNet-5 is our latest convolutional network designed for handwritten and machine-printed character recognition.
The MNIST dataset is basically the "Hello World" of computer vision in deep learning. Currently, most deep learning implementations attain $\textbf{~99%}$ accuracy in correct classification of the handwritten digits. The MNIST database can be found here.
The goal is to reproduce the LeNet-5 model in tensorflow as seen in the 1998 paper. To improve accuracy, further modifications that leverage current deep learning advancements will be done to the model.
This notebook implements the classic LeNet-5 convolutional network [1] and applies it to MNIST digit classification. The basic architecture is shown in the figure below:
LeNet-5 is commonly regarded as the pioneer of convolutional neural networks, consisting of a very simple architecture (by modern standards). The LeNet-5 architecture primarily consists of three major layer constructs:
- Convolutional layers,
C1, C3, C5 - Sub-sampling layers (pooling),
S2, S4 - Fully-connected layers (FC),
F6
In the figure above, convolutional layers are labeled $Cx$, sub-sampling layers are labeled $Sx$, and fully-connected layers are labeled $Fx$, where $x$ is the layer index. In total, LeNet-5 consists of only $7$ layers. $3$ out of these $7$ layers are convolutional layers ($C1, C3, C5$), which are connected by two average pooling layers ($S2, S4$). The penultimate layer is a fully connected layer ($F6$), which is followed by the final output layer. The additional details are summarized below:
Model: Input --> [ C1 ] -> [ S2 ] -> [ C3 ] -> [ S4 ] -----------> [ C5/F5 ] -> [ F6 ] -> Output
Shape: 32x32 28x28 14x14 10x10 5x5 1x1 84 10
Feature Maps: - 6 6 16 16 120 - -
Trainable Params: - 156 2,416 48,120 10,164 850
Implementation: ConvNet -> Avg. Pool -> ConvNet -> Avg. Pool -> (Flatten) ---> FC -------> FC -----> Softmax
- The input size is $32\times32$; here, we rescale the MNIST images from $28\times28$ to $32\times32$ to match this input dimension.
- All convolutional layers use $5\times5$ kernels with stride $1$.
- The two average pooling (subsampling) layers are $2\times2$ pixels wide with stride $2$.
- The output layer uses $10$ custom
Euclidean Radial Basis Function (RBF)neurons for the output layer. (In this notebook, we replace these withsoftmaxactivations).
References¶
- [1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, november 1998.
In [36]:
import tensorflow as tf
from tensorflow import keras
from keras.datasets import mnist
from keras.layers import Input, Conv2D, AveragePooling2D, MaxPooling2D, Dense, Flatten, Dropout, Activation, BatchNormalization
from keras.models import Model, Sequential
from keras.optimizers import Adam
from keras.utils import to_categorical
# from keras.preprocessing.image import ImageDataGenerator
from keras.regularizers import l2
from keras.callbacks import Callback, ReduceLROnPlateau
from sklearn.model_selection import train_test_split
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
import numpy as np
import math
import time
In [37]:
print(tf.__version__)
2.19.0
In [38]:
# Check if GPU is available
gpus = tf.config.list_physical_devices('GPU')
if gpus:
print("Num GPUs Available: ", len(gpus))
for gpu in gpus:
print(f"GPU: {gpu}")
else:
print("No GPU available.")
#------------------------------------------------------------------------------------------------------------------------
# Verify TensorFlow is using GPU
if tf.test.gpu_device_name():
print('GPU found: {}'.format(tf.test.gpu_device_name()))
else:
print("GPU not found.")
Num GPUs Available: 1 GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU') GPU found: /device:GPU:0
In [39]:
##########################
### SETTINGS
##########################
# Hyperparameters
RANDOM_SEED = 42
LEARNING_RATE = 0.001
BATCH_SIZE = 64
NUM_EPOCHS = 30
# Architecture
INPUT_SHAPE = (32, 32, 1)
NUM_CLASSES = 10
In [40]:
(train_x, train_y), (test_x, test_y) = keras.datasets.mnist.load_data()
train_x.shape, type(train_x)
Out[40]:
((60000, 28, 28), numpy.ndarray)
In [41]:
for i in range(5):
plt.subplot(1, 5, i+1)
plt.imshow(train_x[i, :, :])
In [42]:
def preprocess_data(mnist_data, split_ratio=0.08, batch_size=32, preprocess_type='normalize', random_seed=42):
"""
Preprocess MNIST data with specified method and return TensorFlow datasets.
Args:
mnist_data: Tuple containing the MNIST data ((train_images, train_labels), (test_images, test_labels)).
split_ratio: Float, the ratio of the training data to use for validation.
batch_size: Integer, the batch size for the datasets.
preprocess_type: String, 'normalize' or 'standardize' for data preprocessing method.
random_seed: Integer, the seed for shuffling the data.
Returns:
Tuple of TensorFlow datasets: (train_dataset, val_dataset, test_dataset)
"""
(train_images, train_labels), (test_images, test_labels) = mnist_data
# Resize images to 32x32 and add channel dimension
train_images = tf.image.resize(train_images[..., tf.newaxis], [32, 32])
test_images = tf.image.resize(test_images[..., tf.newaxis], [32, 32])
if preprocess_type == 'normalize':
# Normalize pixel values to [0, 1]
train_images = train_images / 255.0
test_images = test_images / 255.0
elif preprocess_type == 'standardize':
# Standardize pixel values (mean 0, std 1)
mean_px = tf.reduce_mean(train_images).numpy().astype(float)
std_px = tf.math.reduce_std(train_images).numpy().astype(float)
train_images = (train_images - mean_px) / std_px
test_images = (test_images - mean_px) / std_px
else:
raise ValueError("Invalid preprocess_type. Choose 'normalize' or 'standardize'.")
# Convert labels to one-hot encoding
train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)
# Split the training data into training and validation sets
num_val_samples = int(split_ratio * len(train_images))
val_images = train_images[-num_val_samples:]
val_labels = train_labels[-num_val_samples:]
train_images = train_images[:-num_val_samples]
train_labels = train_labels[:-num_val_samples]
# Create TensorFlow datasets
train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
val_dataset = tf.data.Dataset.from_tensor_slices((val_images, val_labels))
test_dataset = tf.data.Dataset.from_tensor_slices((test_images, test_labels))
# Shuffle and batch datasets
train_dataset = train_dataset.shuffle(buffer_size=len(train_images), seed=random_seed).batch(batch_size)
val_dataset = val_dataset.batch(batch_size)
test_dataset = test_dataset.batch(batch_size)
return train_dataset, val_dataset, test_dataset
In [43]:
# Load MNIST data
mnist_data = mnist.load_data()
# Preprocess data with normalization and a fixed random seed
train_dataset, val_dataset, test_dataset = preprocess_data(
mnist_data, split_ratio=0.08, batch_size=BATCH_SIZE, preprocess_type='normalize', random_seed=RANDOM_SEED)
In [44]:
# Assuming train_dataset is your tf.data.Dataset
for images, labels in train_dataset.take(1): # Take one batch
print("Image shape:", images.shape)
print("Label shape:", labels.shape)
Image shape: (64, 32, 32, 1) Label shape: (64, 10)
In [45]:
def LeNet5(input_shape=(32, 32, 1), num_classes=10):
"""
Builds a LeNet-5 model using the Sequential API.
Architecture summary: ConvNet -> Avg. Pool -> ConvNet -> Avg. Pool -> (Flatten) --> FC --> FC --> Softmax
Architecture:
- Conv2D: Convolutional layer with 6 filters, kernel size 5, activation 'tanh', padding 'valid'
- AveragePooling2D: Average pooling layer with pool size 2
- Conv2D: Convolutional layer with 16 filters, kernel size 5, activation 'tanh', padding 'valid'
- MaxPooling2D: Max pooling layer with pool size 2
- Flatten: Flatten layer to reshape the output for dense layers
- Dense: Fully connected layer with 120 units and 'tanh' activation
- Dense: Fully connected layer with 84 units and 'tanh' activation
- Dense: Output layer with 'softmax' activation for classification
Arguments:
input_shape -- tuple, shape of the input images (height, width, channels). Default is (32, 32, 1).
num_classes -- integer, number of classes for the output layer. Default is 10.
Returns:
model -- a Keras Sequential model instance with the defined architecture.
"""
model = Sequential([
# Input layer
Input(shape=input_shape, name='input_layer'),
# Layer 1: C1 + S2
Conv2D(filters=6, kernel_size=5, strides=1, activation='tanh', padding='valid', name='conv_1'),
AveragePooling2D(pool_size=2, strides=2, name='avg_pool_1'),
# Layer 2: C3 + S4
Conv2D(filters=16, kernel_size=5, strides=1, activation='tanh', padding='valid', name='conv_2'),
AveragePooling2D(pool_size=2, strides=2, name='avg_pool2'),
# Layer 3: Flatten + F5
Flatten(name='flatten'),
Dense(units=120, activation='tanh', name='fully_connected_1'),
# Layer 4: F6
Dense(units=84, activation='tanh', name='fully_connected_2'),
# Layer 5: Output
Dense(units=num_classes, activation='softmax', name='output')
])
model._name = 'LeNet5'
return model
3. Model Compiling, Training and Testing & Evaluation Plots (Loss, Accuracy)¶
Functions:
train_modelplot_history
In [46]:
def train_model(model, train_dataset, val_dataset, NUM_EPOCHS=20, lr_factor=0.2, lr_patience=2):
# compile the model
model.compile(optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
# # Define the custom callback to log train and validation accuracy & learning rate
class LogMetrics(Callback):
def __init__(self):
super(LogMetrics, self).__init__()
self.lrs = []
def on_train_begin(self, logs=None):
self.lrs = []
def on_epoch_end(self, epoch, logs=None):
val_logs = self.model.evaluate(val_dataset, verbose=0)
lr = self.model.optimizer.learning_rate.numpy()
self.lrs.append(lr)
print(f'Epoch {epoch + 1}/{NUM_EPOCHS} | '
f'Train Loss: {logs["loss"]:.5f}, '
f'Train Accuracy: {logs["accuracy"]:.5f} | '
f'Val Loss: {val_logs[0]:.5f}, '
f'Val Accuracy: {val_logs[1]:.5f} | '
f'Learning Rate: {lr:.8f}')
# Define the ReduceLROnPlateau callback
reduce_lr = ReduceLROnPlateau(
monitor='val_loss', # Metric to monitor
factor=lr_factor, # Factor by which the learning rate will be reduced
patience=lr_patience, # Number of epochs with no improvement after which learning rate will be reduced
)
# Create an instance of the LogMetrics callback
log_metrics = LogMetrics()
# Train the model with both the custom logging and ReduceLROnPlateau callbacks
start_time = time.time()
history = model.fit(train_dataset,
epochs=NUM_EPOCHS,
validation_data=val_dataset,
callbacks=[log_metrics, reduce_lr],
verbose=0)
print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))
return model, history, log_metrics, reduce_lr
In [47]:
# Plot the training history
def plot_history(history, lrs):
epochs = range(len(history.history['loss']))
plt.figure(figsize=(14, 5))
plt.subplot(1, 3, 1)
plt.plot(epochs, history.history['loss'], label='Training Loss')
plt.plot(epochs, history.history['val_loss'], label='Validation Loss')
plt.title('Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 3, 2)
plt.plot(epochs, history.history['accuracy'], label='Training Accuracy')
plt.plot(epochs, history.history['val_accuracy'], label='Validation Accuracy')
plt.title('Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.subplot(1, 3, 3)
plt.plot(epochs, lrs, label='Learning Rate')
plt.title('Learning Rate')
plt.xlabel('Epochs')
plt.ylabel('Learning Rate')
plt.legend()
plt.tight_layout()
plt.show()
In [48]:
# Create the model & Print out model summary
model = LeNet5(input_shape=INPUT_SHAPE, num_classes=NUM_CLASSES)
model.summary()
Model: "sequential_5"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv_1 (Conv2D) │ (None, 28, 28, 6) │ 156 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ avg_pool_1 (AveragePooling2D) │ (None, 14, 14, 6) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_2 (Conv2D) │ (None, 10, 10, 16) │ 2,416 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ avg_pool2 (AveragePooling2D) │ (None, 5, 5, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 400) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_1 (Dense) │ (None, 120) │ 48,120 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_2 (Dense) │ (None, 84) │ 10,164 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ output (Dense) │ (None, 10) │ 850 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 61,706 (241.04 KB)
Trainable params: 61,706 (241.04 KB)
Non-trainable params: 0 (0.00 B)
In [49]:
model, history, log_metrics, reduce_lr = train_model(model, train_dataset, val_dataset, NUM_EPOCHS=NUM_EPOCHS)
# Evaluate on test data using model
test_loss, test_accuracy = model.evaluate(test_dataset)
print(f'Test Loss: {test_loss:.4f}, Test Accuracy: {test_accuracy:.4f}')
# Plot loss and accuracy of training and validation dataset
plot_history(history, log_metrics.lrs)
Epoch 1/30 | Train Loss: 0.28284, Train Accuracy: 0.91614 | Val Loss: 0.11197, Val Accuracy: 0.96854 | Learning Rate: 0.00100000 Epoch 2/30 | Train Loss: 0.10670, Train Accuracy: 0.96790 | Val Loss: 0.07764, Val Accuracy: 0.97979 | Learning Rate: 0.00100000 Epoch 3/30 | Train Loss: 0.07099, Train Accuracy: 0.97821 | Val Loss: 0.05983, Val Accuracy: 0.98042 | Learning Rate: 0.00100000 Epoch 4/30 | Train Loss: 0.05436, Train Accuracy: 0.98339 | Val Loss: 0.05954, Val Accuracy: 0.98146 | Learning Rate: 0.00100000 Epoch 5/30 | Train Loss: 0.04089, Train Accuracy: 0.98746 | Val Loss: 0.05939, Val Accuracy: 0.98229 | Learning Rate: 0.00100000 Epoch 6/30 | Train Loss: 0.03491, Train Accuracy: 0.98877 | Val Loss: 0.05747, Val Accuracy: 0.98208 | Learning Rate: 0.00100000 Epoch 7/30 | Train Loss: 0.02946, Train Accuracy: 0.99038 | Val Loss: 0.04693, Val Accuracy: 0.98729 | Learning Rate: 0.00100000 Epoch 8/30 | Train Loss: 0.02327, Train Accuracy: 0.99239 | Val Loss: 0.04589, Val Accuracy: 0.98646 | Learning Rate: 0.00100000 Epoch 9/30 | Train Loss: 0.01957, Train Accuracy: 0.99380 | Val Loss: 0.04854, Val Accuracy: 0.98625 | Learning Rate: 0.00100000 Epoch 10/30 | Train Loss: 0.01788, Train Accuracy: 0.99417 | Val Loss: 0.04681, Val Accuracy: 0.98687 | Learning Rate: 0.00100000 Epoch 11/30 | Train Loss: 0.00742, Train Accuracy: 0.99795 | Val Loss: 0.04055, Val Accuracy: 0.99000 | Learning Rate: 0.00020000 Epoch 12/30 | Train Loss: 0.00399, Train Accuracy: 0.99947 | Val Loss: 0.04122, Val Accuracy: 0.99000 | Learning Rate: 0.00020000 Epoch 13/30 | Train Loss: 0.00312, Train Accuracy: 0.99958 | Val Loss: 0.04264, Val Accuracy: 0.98937 | Learning Rate: 0.00020000 Epoch 14/30 | Train Loss: 0.00226, Train Accuracy: 0.99989 | Val Loss: 0.04205, Val Accuracy: 0.99021 | Learning Rate: 0.00004000 Epoch 15/30 | Train Loss: 0.00204, Train Accuracy: 0.99987 | Val Loss: 0.04172, Val Accuracy: 0.99000 | Learning Rate: 0.00004000 Epoch 16/30 | Train Loss: 0.00186, Train Accuracy: 0.99991 | Val Loss: 0.04174, Val Accuracy: 0.99021 | Learning Rate: 0.00000800 Epoch 17/30 | Train Loss: 0.00183, Train Accuracy: 0.99989 | Val Loss: 0.04185, Val Accuracy: 0.99000 | Learning Rate: 0.00000800 Epoch 18/30 | Train Loss: 0.00178, Train Accuracy: 0.99991 | Val Loss: 0.04186, Val Accuracy: 0.99000 | Learning Rate: 0.00000160 Epoch 19/30 | Train Loss: 0.00178, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000160 Epoch 20/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000032 Epoch 21/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000032 Epoch 22/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000006 Epoch 23/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000006 Epoch 24/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000001 Epoch 25/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000001 Epoch 26/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000000 Epoch 27/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000000 Epoch 28/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000000 Epoch 29/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000000 Epoch 30/30 | Train Loss: 0.00176, Train Accuracy: 0.99991 | Val Loss: 0.04188, Val Accuracy: 0.99000 | Learning Rate: 0.00000000 Total Training Time: 1.65 min 157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9883 - loss: 0.0417 Test Loss: 0.0356, Test Accuracy: 0.9900
4. Modified LeNet-5 Architectures¶
Below are the modified versions of LeNet-5 architecture:
LeNet5v0: ConvNet → Max. Pool → ConvNet → Max. Pool → (Flatten) → FC → FC → Softmax
- Replace all the instances of
tanhactivation functions withReLUactivation functions in baseline original model from 1998 paper. - Replace all the instances ($S2, S4$) of
average poolinglayers withmax poolinglayers.
LeNet5v1: ConvNet → ConvNet → Max. Pool → Dropout → ConvNet → Max. Pool → Dropout → (Flatten) → FC → FC → Softmax
- Add a Convolutional layer directly after $C1$ and before $S2$.
- Add dropout layers (
dropout= $0.25$) directly after both pooling layers.
LeNet5v2: ConvNet → ConvNet → BatchNorm → Max. Pool → Dropout → ConvNet → ConvNet → BatchNorm → Max. Pool → Dropout → (Flatten) → FC → BatchNorm → FC → BatchNorm → FC → BatchNorm → Dropout → Softmax
- Add a
BatchNormlayer directly after $C2$ inLeNet5v1. - Add a
ConvNet& then aBatchNormlayer directly after new $C3$ inLeNet5v1. - Add a
FC& then aBatchNormlayer directly after flatten layer. - Add 2
BatchNormlayers, one directly before and one after $F8$ inLeNet5v1. - Add a Dropout layer right before the output layer.
- Increase number of filter channels (feature maps) for
Convnetlayers consecutively to $32, 32, 64, 64$. - Set number of hidden units in initial $FC$ layer (directly after flatten layer) and 2nd $FC$ layer (sandwiched with
BatchNorms) to $256$ and $128$ respectively. - The use of
BatchNormhandlesbiasso we can set bias toFalseor $0$ in layers right beforeBatchNormlayer. - Add L2 regularization (
reg_L2= $0.0005$) in $C1, C3$ layers to reduce overfitting
In [50]:
def LeNet5v0(input_shape=(32, 32, 1), num_classes=10, activation='relu'):
"""
Builds a modified LeNet-5 model using the Sequential API.
Architecture summary: ConvNet -> Max. Pool -> ConvNet -> Max. Pool -> (Flatten) --> FC --> FC --> Softmax
Arguments:
input_shape -- tuple, shape of the input images (height, width, channels). Default is (32, 32, 1).
num_classes -- integer, number of classes for the output layer. Default is 10.
Returns:
model -- a Keras Sequential model instance with the defined architecture.
"""
model = Sequential([
# Input layer
Input(shape=input_shape, name='input_layer'),
# Layer 1: C1 + S2
Conv2D(filters=6, kernel_size=5, strides=1, activation=activation, padding='valid', name='conv_1'),
MaxPooling2D(pool_size=2, strides=2, name='max_pool_1'),
# Layer 2: C3 + S4
Conv2D(filters=16, kernel_size=5, strides=1, activation=activation, padding='valid', name='conv_2'),
MaxPooling2D(pool_size=2, strides=2, name='max_pool_2'),
# Layer 3: Flatten + F5
Flatten(name='flatten'),
Dense(units=120, activation=activation, name='fully_connected_1'),
# Layer 4: F6
Dense(units=84, activation=activation, name='fully_connected_2'),
# Layer 5: Output
Dense(units=num_classes, activation='softmax', name='output')
])
model._name = 'LeNet5v0'
return model
#------------------------------------------------------------------------------------------------------------------------
def LeNet5v1(input_shape=(32, 32, 1), num_classes=10, activation='relu'):
"""
Builds a modified LeNet-5 model using the Sequential API.
Architecture summary: ConvNet -> [ConvNet] -> Max. Pool -> [Dropout] -> ConvNet -> Max. Pool -> [Dropout] -> (Flatten) -> FC -> FC -> Softmax
Arguments:
input_shape -- tuple, shape of the input images (height, width, channels). Default is (32, 32, 1).
num_classes -- integer, number of classes for the output layer. Default is 10.
Returns:
model -- a Keras Sequential model instance with the defined architecture.
"""
model = Sequential([
# Input layer
Input(shape=input_shape, name='input_layer'),
# Layer 1: C1
Conv2D(filters=6, kernel_size=5, strides=1, activation=activation, padding='valid', input_shape=input_shape, name='conv_1'),
# Layer 2: C2 + S2 + D3
Conv2D(filters=16, kernel_size=5, strides=1, activation=activation, padding='valid', name='conv_2'),
MaxPooling2D(pool_size=2, strides=2, name='max_pool_1'),
Dropout(rate=0.25, name='dropout_1'),
# Layer 3: C4 + S5 + D6
Conv2D(filters=16, kernel_size=5, strides=1, activation=activation, padding='valid', name='conv_3'),
MaxPooling2D(pool_size=2, strides=2, name='max_pool_2'),
Dropout(rate=0.25, name='dropout_2'),
# Layer 4: Flatten + F7
Flatten(name='flatten'),
Dense(units=120, activation=activation, name='fully_connected_1'),
# Layer 5: F8
Dense(units=84, activation=activation, name='fully_connected_2'),
# Layer 6: Output
Dense(units=num_classes, activation='softmax', name='output')
])
model._name = 'LeNet5v1'
return model
#------------------------------------------------------------------------------------------------------------------------
def LeNet5v2(input_shape=(32, 32, 1), num_classes=10, l2_reg=0.0005, activation='relu'):
"""
Builds a modified LeNet-5 model using the Sequential API.
Architecture summary: ConvNet → ConvNet → [BatchNorm] → Max. Pool → Dropout → ConvNet → [ConvNet] → [BatchNorm] → Max. Pool →
Dropout → (Flatten) → [FC] → [BatchNorm] → FC → [BatchNorm] → FC → [BatchNorm] → [Dropout] → Softmax
Arguments:
input_shape -- tuple, shape of the input images (height, width, channels). Default is (32, 32, 1).
num_classes -- integer, number of classes for the output layer. Default is 10.
Returns:
model -- a Keras Sequential model instance with the defined architecture.
"""
model = Sequential([
# Input layer
Input(shape=input_shape, name='input_layer'),
# Layer 1: C1_L2regularized
Conv2D(filters=32, kernel_size=5, strides=1, activation=activation, padding='valid', input_shape=input_shape, kernel_regularizer=l2(l2_reg), name='conv_1'),
# Layer 2: C2_no_bias + BN3 + S4 + D5
Conv2D(filters=32, kernel_size=5, strides=1, activation=activation, padding='valid', use_bias=False, name='conv_2'),
BatchNormalization(name='batch_norm_1'),
Activation(activation),
MaxPooling2D(pool_size=2, strides=2, name='max_pool_1'),
Dropout(rate=0.25, name='dropout_1'),
# Layer 3: C6_L2regularized
Conv2D(filters=64, kernel_size=3, strides=1, activation=activation, padding='valid', kernel_regularizer=l2(l2_reg), name='conv_3'),
# Layer 4: C7_no_bias + BN8 + S9 + D10
Conv2D(filters=64, kernel_size=3, strides=1, activation=activation, padding='valid', use_bias=False, name='conv_4'),
BatchNormalization(name='batch_norm_2'),
MaxPooling2D(pool_size=2, strides=2, name='max_pool_2'),
Dropout(rate=0.25, name='dropout_2'),
# Layer 5: Flatten + F11_no_bias + BN12
Flatten(name='flatten'),
Dense(units=256, activation=activation, use_bias=False, name='fully_connected_1'),
BatchNormalization(name='batch_norm_3'),
Activation(activation),
# Layer 6: F13_no_bias + BN14
Dense(units=128, activation=activation, use_bias=False, name='fully_connected_2'),
BatchNormalization(name='batch_norm_4'),
Activation(activation),
# Layer 7: F15_no_bias + BN16 + D17
Dense(units=84, activation=activation, use_bias=False, name='fully_connected_3'),
BatchNormalization(name='batch_norm_5'),
Activation(activation),
Dropout(rate=0.25, name='dropout_3'),
# Layer 8: Output
Dense(units=num_classes, activation='softmax', name='output')
])
model._name = 'LeNet5v2'
return model
In [51]:
factor = 0.5
patience = 4
In [52]:
# Create the model & Print out model summary
LeNet5v0_model = LeNet5v0(input_shape=INPUT_SHAPE, num_classes=NUM_CLASSES)
LeNet5v0_model.summary()
Model: "sequential_6"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv_1 (Conv2D) │ (None, 28, 28, 6) │ 156 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pool_1 (MaxPooling2D) │ (None, 14, 14, 6) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_2 (Conv2D) │ (None, 10, 10, 16) │ 2,416 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pool_2 (MaxPooling2D) │ (None, 5, 5, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 400) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_1 (Dense) │ (None, 120) │ 48,120 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_2 (Dense) │ (None, 84) │ 10,164 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ output (Dense) │ (None, 10) │ 850 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 61,706 (241.04 KB)
Trainable params: 61,706 (241.04 KB)
Non-trainable params: 0 (0.00 B)
In [53]:
LeNet5v0_model, history_v0, log_metrics_v0, reduce_lr_v0 = train_model(LeNet5v0_model, train_dataset, val_dataset, lr_factor=factor, lr_patience=patience)
# Evaluate on test data using model
test_loss_v0, test_accuracy_v0 = LeNet5v0_model.evaluate(test_dataset)
print(f'Test Loss: {test_loss_v0:.4f}, Test Accuracy: {test_accuracy_v0:.4f}')
# Plot loss and accuracy of training and validation dataset
plot_history(history_v0, log_metrics_v0.lrs)
Epoch 1/20 | Train Loss: 0.22722, Train Accuracy: 0.93158 | Val Loss: 0.07933, Val Accuracy: 0.97646 | Learning Rate: 0.00100000 Epoch 2/20 | Train Loss: 0.06941, Train Accuracy: 0.97882 | Val Loss: 0.05764, Val Accuracy: 0.98271 | Learning Rate: 0.00100000 Epoch 3/20 | Train Loss: 0.04921, Train Accuracy: 0.98478 | Val Loss: 0.05057, Val Accuracy: 0.98687 | Learning Rate: 0.00100000 Epoch 4/20 | Train Loss: 0.03977, Train Accuracy: 0.98759 | Val Loss: 0.04267, Val Accuracy: 0.98896 | Learning Rate: 0.00100000 Epoch 5/20 | Train Loss: 0.03164, Train Accuracy: 0.99020 | Val Loss: 0.04577, Val Accuracy: 0.98708 | Learning Rate: 0.00100000 Epoch 6/20 | Train Loss: 0.02700, Train Accuracy: 0.99125 | Val Loss: 0.04100, Val Accuracy: 0.99083 | Learning Rate: 0.00100000 Epoch 7/20 | Train Loss: 0.02095, Train Accuracy: 0.99322 | Val Loss: 0.04333, Val Accuracy: 0.98854 | Learning Rate: 0.00100000 Epoch 8/20 | Train Loss: 0.01973, Train Accuracy: 0.99319 | Val Loss: 0.04999, Val Accuracy: 0.98792 | Learning Rate: 0.00100000 Epoch 9/20 | Train Loss: 0.01583, Train Accuracy: 0.99471 | Val Loss: 0.05685, Val Accuracy: 0.98750 | Learning Rate: 0.00100000 Epoch 10/20 | Train Loss: 0.01526, Train Accuracy: 0.99504 | Val Loss: 0.04521, Val Accuracy: 0.98875 | Learning Rate: 0.00100000 Epoch 11/20 | Train Loss: 0.00608, Train Accuracy: 0.99815 | Val Loss: 0.04245, Val Accuracy: 0.99271 | Learning Rate: 0.00050000 Epoch 12/20 | Train Loss: 0.00420, Train Accuracy: 0.99879 | Val Loss: 0.04415, Val Accuracy: 0.99042 | Learning Rate: 0.00050000 Epoch 13/20 | Train Loss: 0.00395, Train Accuracy: 0.99875 | Val Loss: 0.05888, Val Accuracy: 0.98979 | Learning Rate: 0.00050000 Epoch 14/20 | Train Loss: 0.00405, Train Accuracy: 0.99864 | Val Loss: 0.04386, Val Accuracy: 0.99187 | Learning Rate: 0.00050000 Epoch 15/20 | Train Loss: 0.00144, Train Accuracy: 0.99969 | Val Loss: 0.04726, Val Accuracy: 0.99208 | Learning Rate: 0.00025000 Epoch 16/20 | Train Loss: 0.00060, Train Accuracy: 0.99986 | Val Loss: 0.04838, Val Accuracy: 0.99146 | Learning Rate: 0.00025000 Epoch 17/20 | Train Loss: 0.00066, Train Accuracy: 0.99989 | Val Loss: 0.05724, Val Accuracy: 0.99208 | Learning Rate: 0.00025000 Epoch 18/20 | Train Loss: 0.00062, Train Accuracy: 0.99978 | Val Loss: 0.05191, Val Accuracy: 0.99167 | Learning Rate: 0.00025000 Epoch 19/20 | Train Loss: 0.00030, Train Accuracy: 0.99996 | Val Loss: 0.05387, Val Accuracy: 0.99167 | Learning Rate: 0.00012500 Epoch 20/20 | Train Loss: 0.00015, Train Accuracy: 1.00000 | Val Loss: 0.05681, Val Accuracy: 0.99187 | Learning Rate: 0.00012500 Total Training Time: 1.06 min 157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9900 - loss: 0.0495 Test Loss: 0.0373, Test Accuracy: 0.9920
6. LeNet5v1¶
LeNet5v1: ConvNet → ConvNet → Max. Pool → Dropout → ConvNet → Max. Pool → Dropout → (Flatten) → FC → FC → Softmax
In [54]:
# Create the model & Print out model summary
LeNet5v1_model = LeNet5v1(input_shape=INPUT_SHAPE, num_classes=NUM_CLASSES)
LeNet5v1_model.summary()
Model: "sequential_7"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv_1 (Conv2D) │ (None, 28, 28, 6) │ 156 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_2 (Conv2D) │ (None, 24, 24, 16) │ 2,416 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pool_1 (MaxPooling2D) │ (None, 12, 12, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 12, 12, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_3 (Conv2D) │ (None, 8, 8, 16) │ 6,416 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pool_2 (MaxPooling2D) │ (None, 4, 4, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_2 (Dropout) │ (None, 4, 4, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_1 (Dense) │ (None, 120) │ 30,840 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_2 (Dense) │ (None, 84) │ 10,164 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ output (Dense) │ (None, 10) │ 850 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 50,842 (198.60 KB)
Trainable params: 50,842 (198.60 KB)
Non-trainable params: 0 (0.00 B)
In [55]:
LeNet5v1_model, history_v1, log_metrics_v1, reduce_lr_v1 = train_model(LeNet5v1_model, train_dataset, val_dataset, lr_factor=factor, lr_patience=patience)
# Evaluate on test data using model
test_loss_v1, test_accuracy_v1 = LeNet5v1_model.evaluate(test_dataset)
print(f'Test Loss: {test_loss_v1:.4f}, Test Accuracy: {test_accuracy_v1:.4f}')
# Plot loss and accuracy of training and validation dataset
plot_history(history_v1, log_metrics_v1.lrs)
Epoch 1/20 | Train Loss: 0.27430, Train Accuracy: 0.91351 | Val Loss: 0.06136, Val Accuracy: 0.98188 | Learning Rate: 0.00100000 Epoch 2/20 | Train Loss: 0.08686, Train Accuracy: 0.97270 | Val Loss: 0.05686, Val Accuracy: 0.98250 | Learning Rate: 0.00100000 Epoch 3/20 | Train Loss: 0.06794, Train Accuracy: 0.97828 | Val Loss: 0.03884, Val Accuracy: 0.99000 | Learning Rate: 0.00100000 Epoch 4/20 | Train Loss: 0.05759, Train Accuracy: 0.98165 | Val Loss: 0.03972, Val Accuracy: 0.98958 | Learning Rate: 0.00100000 Epoch 5/20 | Train Loss: 0.05031, Train Accuracy: 0.98409 | Val Loss: 0.03464, Val Accuracy: 0.99021 | Learning Rate: 0.00100000 Epoch 6/20 | Train Loss: 0.04642, Train Accuracy: 0.98514 | Val Loss: 0.03343, Val Accuracy: 0.99063 | Learning Rate: 0.00100000 Epoch 7/20 | Train Loss: 0.04114, Train Accuracy: 0.98694 | Val Loss: 0.03639, Val Accuracy: 0.98917 | Learning Rate: 0.00100000 Epoch 8/20 | Train Loss: 0.04100, Train Accuracy: 0.98692 | Val Loss: 0.03511, Val Accuracy: 0.98937 | Learning Rate: 0.00100000 Epoch 9/20 | Train Loss: 0.03859, Train Accuracy: 0.98795 | Val Loss: 0.02803, Val Accuracy: 0.99208 | Learning Rate: 0.00100000 Epoch 10/20 | Train Loss: 0.03360, Train Accuracy: 0.98928 | Val Loss: 0.02973, Val Accuracy: 0.99208 | Learning Rate: 0.00100000 Epoch 11/20 | Train Loss: 0.03398, Train Accuracy: 0.98913 | Val Loss: 0.03033, Val Accuracy: 0.99187 | Learning Rate: 0.00100000 Epoch 12/20 | Train Loss: 0.03080, Train Accuracy: 0.99000 | Val Loss: 0.03921, Val Accuracy: 0.99000 | Learning Rate: 0.00100000 Epoch 13/20 | Train Loss: 0.03002, Train Accuracy: 0.99031 | Val Loss: 0.02984, Val Accuracy: 0.99250 | Learning Rate: 0.00100000 Epoch 14/20 | Train Loss: 0.02074, Train Accuracy: 0.99303 | Val Loss: 0.02530, Val Accuracy: 0.99437 | Learning Rate: 0.00050000 Epoch 15/20 | Train Loss: 0.02027, Train Accuracy: 0.99361 | Val Loss: 0.02635, Val Accuracy: 0.99271 | Learning Rate: 0.00050000 Epoch 16/20 | Train Loss: 0.02090, Train Accuracy: 0.99315 | Val Loss: 0.02557, Val Accuracy: 0.99354 | Learning Rate: 0.00050000 Epoch 17/20 | Train Loss: 0.01825, Train Accuracy: 0.99393 | Val Loss: 0.02477, Val Accuracy: 0.99375 | Learning Rate: 0.00050000 Epoch 18/20 | Train Loss: 0.01794, Train Accuracy: 0.99409 | Val Loss: 0.02669, Val Accuracy: 0.99354 | Learning Rate: 0.00050000 Epoch 19/20 | Train Loss: 0.01803, Train Accuracy: 0.99388 | Val Loss: 0.02669, Val Accuracy: 0.99354 | Learning Rate: 0.00050000 Epoch 20/20 | Train Loss: 0.01822, Train Accuracy: 0.99361 | Val Loss: 0.02614, Val Accuracy: 0.99292 | Learning Rate: 0.00050000 Total Training Time: 1.43 min 157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9936 - loss: 0.0206 Test Loss: 0.0161, Test Accuracy: 0.9952
7. LeNet5v2¶
LeNet5v2: ConvNet → ConvNet → BatchNorm → Max. Pool → Dropout → ConvNet → ConvNet → BatchNorm → Max. Pool → Dropout → (Flatten) → FC → BatchNorm → FC → BatchNorm → FC → BatchNorm → Dropout → Softmax
In [56]:
# Create the model & Print out model summary
LeNet5v2_model = LeNet5v2(input_shape=INPUT_SHAPE, num_classes=NUM_CLASSES)
LeNet5v2_model.summary()
Model: "sequential_8"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv_1 (Conv2D) │ (None, 28, 28, 32) │ 832 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_2 (Conv2D) │ (None, 24, 24, 32) │ 25,600 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_norm_1 │ (None, 24, 24, 32) │ 128 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation_4 (Activation) │ (None, 24, 24, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pool_1 (MaxPooling2D) │ (None, 12, 12, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 12, 12, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_3 (Conv2D) │ (None, 10, 10, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv_4 (Conv2D) │ (None, 8, 8, 64) │ 36,864 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_norm_2 │ (None, 8, 8, 64) │ 256 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pool_2 (MaxPooling2D) │ (None, 4, 4, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_2 (Dropout) │ (None, 4, 4, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 1024) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_1 (Dense) │ (None, 256) │ 262,144 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_norm_3 │ (None, 256) │ 1,024 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation_5 (Activation) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_2 (Dense) │ (None, 128) │ 32,768 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_norm_4 │ (None, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation_6 (Activation) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ fully_connected_3 (Dense) │ (None, 84) │ 10,752 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_norm_5 │ (None, 84) │ 336 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation_7 (Activation) │ (None, 84) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_3 (Dropout) │ (None, 84) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ output (Dense) │ (None, 10) │ 850 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 390,562 (1.49 MB)
Trainable params: 389,434 (1.49 MB)
Non-trainable params: 1,128 (4.41 KB)
In [57]:
LeNet5v2_model, history_v2, log_metrics_v2, reduce_lr_v2 = train_model(LeNet5v2_model, train_dataset, val_dataset, lr_factor=factor, lr_patience=2)
# Evaluate on test data using model
test_loss_v2, test_accuracy_v2 = LeNet5v0_model.evaluate(test_dataset)
print(f'Test Loss: {test_loss_v2:.4f}, Test Accuracy: {test_accuracy_v2:.4f}')
# Plot loss and accuracy of training and validation dataset
plot_history(history_v2, log_metrics_v2.lrs)
Epoch 1/20 | Train Loss: 0.33472, Train Accuracy: 0.91446 | Val Loss: 0.07454, Val Accuracy: 0.98604 | Learning Rate: 0.00100000 Epoch 2/20 | Train Loss: 0.10614, Train Accuracy: 0.97679 | Val Loss: 0.05117, Val Accuracy: 0.98979 | Learning Rate: 0.00100000 Epoch 3/20 | Train Loss: 0.08341, Train Accuracy: 0.98234 | Val Loss: 0.06130, Val Accuracy: 0.98979 | Learning Rate: 0.00100000 Epoch 4/20 | Train Loss: 0.06774, Train Accuracy: 0.98525 | Val Loss: 0.04115, Val Accuracy: 0.99187 | Learning Rate: 0.00100000 Epoch 5/20 | Train Loss: 0.06057, Train Accuracy: 0.98694 | Val Loss: 0.04516, Val Accuracy: 0.99083 | Learning Rate: 0.00100000 Epoch 6/20 | Train Loss: 0.05472, Train Accuracy: 0.98707 | Val Loss: 0.04635, Val Accuracy: 0.99063 | Learning Rate: 0.00100000 Epoch 7/20 | Train Loss: 0.04118, Train Accuracy: 0.99129 | Val Loss: 0.03190, Val Accuracy: 0.99396 | Learning Rate: 0.00050000 Epoch 8/20 | Train Loss: 0.03215, Train Accuracy: 0.99308 | Val Loss: 0.03843, Val Accuracy: 0.99229 | Learning Rate: 0.00050000 Epoch 9/20 | Train Loss: 0.03195, Train Accuracy: 0.99283 | Val Loss: 0.02844, Val Accuracy: 0.99521 | Learning Rate: 0.00050000 Epoch 10/20 | Train Loss: 0.03271, Train Accuracy: 0.99304 | Val Loss: 0.03421, Val Accuracy: 0.99354 | Learning Rate: 0.00050000 Epoch 11/20 | Train Loss: 0.03002, Train Accuracy: 0.99317 | Val Loss: 0.03827, Val Accuracy: 0.99333 | Learning Rate: 0.00050000 Epoch 12/20 | Train Loss: 0.02228, Train Accuracy: 0.99520 | Val Loss: 0.03139, Val Accuracy: 0.99354 | Learning Rate: 0.00025000 Epoch 13/20 | Train Loss: 0.01848, Train Accuracy: 0.99659 | Val Loss: 0.03407, Val Accuracy: 0.99437 | Learning Rate: 0.00025000 Epoch 14/20 | Train Loss: 0.01630, Train Accuracy: 0.99667 | Val Loss: 0.02736, Val Accuracy: 0.99500 | Learning Rate: 0.00012500 Epoch 15/20 | Train Loss: 0.01425, Train Accuracy: 0.99732 | Val Loss: 0.02771, Val Accuracy: 0.99500 | Learning Rate: 0.00012500 Epoch 16/20 | Train Loss: 0.01246, Train Accuracy: 0.99784 | Val Loss: 0.02776, Val Accuracy: 0.99542 | Learning Rate: 0.00012500 Epoch 17/20 | Train Loss: 0.01150, Train Accuracy: 0.99804 | Val Loss: 0.02732, Val Accuracy: 0.99500 | Learning Rate: 0.00006250 Epoch 18/20 | Train Loss: 0.01086, Train Accuracy: 0.99810 | Val Loss: 0.02781, Val Accuracy: 0.99417 | Learning Rate: 0.00006250 Epoch 19/20 | Train Loss: 0.01061, Train Accuracy: 0.99819 | Val Loss: 0.02737, Val Accuracy: 0.99458 | Learning Rate: 0.00003125 Epoch 20/20 | Train Loss: 0.01000, Train Accuracy: 0.99828 | Val Loss: 0.02690, Val Accuracy: 0.99437 | Learning Rate: 0.00003125 Total Training Time: 1.84 min 157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9900 - loss: 0.0495 Test Loss: 0.0373, Test Accuracy: 0.9920
A. Model Performance for Various Modified Versions of LeNet-5¶
| S/o. | Model Changes | Epochs | Train loss | Train Acc. | Val loss | Val. Acc.(%) | Test Loss | Test Acc.(%) | Error rate (%) |
|---|---|---|---|---|---|---|---|---|---|
| 1. | Baseline | 10 | 0.0083 | 99.83 | 0.0437 | 98.98 | 0.0365 | 98.89 | 1.11 |
| 2. | Baseline | 30 | 0.00176 | 99.99 | 0.0417 | 98.83 | 0.0356 | 99.00 | 1.00 |
| 3. | LeNet5_v0 | 20 | 0.00015 | 99.99 | 0.0495 | 99.00 | 0.0373 | 99.20 | 0.80 |
| 4. | LeNet5_v1 | 20 | 0.01822 | 99.36 | 0.0206 | 99.36 | 0.0161 | 99.52 | 0.48 |
| 5. | LeNet5_v2 | 20 | 0.01000 | 99.83 | 0.0269 | 99.44 | 0.0373 | 99.20 | 0.80 |
In [59]:
%watermark -iv
tensorflow: 2.19.0 sklearn : 1.6.1 keras : 3.10.0 matplotlib: 3.10.0 numpy : 2.0.2 pandas : 2.2.2