The data, concept, and initial implementation of this notebook was done in Colab by Ross Wightman, the creator of timm. I (Jeremy Howard) did some refactoring, curating, and expanding of the analysis, and added prose.

timm

PyTorch Image Models (timm) is a wonderful library by Ross Wightman which provides state-of-the-art pre-trained computer vision models. It's like Huggingface Transformers, but for computer vision instead of NLP (and it's not restricted to transformers-based models)!

Ross has been kind enough to help me understand how to best take advantage of this library by identifying the top models. I'm going to share here so of what I've learned from him, plus some additional ideas.

The data

Ross regularly benchmarks new models as they are added to timm, and puts the results in a CSV in the project's GitHub repo. To analyse the data, we'll first clone the repo:

import altair as alt
alt.renderers.enable('html')


# # Ensure Kaggle renderer is active
# alt.renderers.enable('kaggle')

RendererRegistry.enable('html')

! git clone --depth 1 https://github.com/rwightman/pytorch-image-models.git
%cd pytorch-image-models/results

Cloning into 'pytorch-image-models'...
remote: Enumerating objects: 486, done.
remote: Counting objects: 100% (486/486), done.
remote: Compressing objects: 100% (385/385), done.
remote: Total 486 (delta 109), reused 258 (delta 97), pack-reused 0 (from 0)
Receiving objects: 100% (486/486), 3.27 MiB | 5.90 MiB/s, done.
Resolving deltas: 100% (109/109), done.
/kaggle/working/pytorch-image-models/results

Using Pandas, we can read the two CSV files we need, and merge them together.

%ls

README.md
benchmark-infer-amp-nchw-pt113-cu117-rtx3090.csv
benchmark-infer-amp-nchw-pt210-cu121-rtx3090.csv
benchmark-infer-amp-nchw-pt240-cu124-rtx3090.csv
benchmark-infer-amp-nchw-pt240-cu124-rtx4090-dynamo.csv
benchmark-infer-amp-nchw-pt240-cu124-rtx4090.csv
benchmark-infer-amp-nchw-pt291-cu128-4090-dynamo.csv
benchmark-infer-amp-nchw-pt291-cu128-4090.csv
benchmark-infer-amp-nchw-pt291-cu130-5090-dynamo.csv
benchmark-infer-amp-nchw-pt291-cu130-5090.csv
benchmark-infer-amp-nchw-pt291-cu130-pro6000maxq-dynamo.csv
benchmark-infer-amp-nchw-pt291-cu130-pro6000maxq.csv
benchmark-infer-amp-nhwc-pt113-cu117-rtx3090.csv
benchmark-infer-amp-nhwc-pt210-cu121-rtx3090.csv
benchmark-infer-amp-nhwc-pt240-cu124-rtx3090.csv
benchmark-infer-amp-nhwc-pt240-cu124-rtx4090.csv
benchmark-infer-amp_bf16-nchw-pt291-cu130-pro6000maxq-dynamo.csv
benchmark-infer-bf16-nchw-pt291-cu130-pro6000maxq-dynamo.csv
benchmark-infer-fp32-nchw-pt221-cpu-i9_10940x-dynamo.csv
benchmark-infer-fp32-nchw-pt240-cpu-i7_12700h-dynamo.csv
benchmark-infer-fp32-nchw-pt240-cpu-i9_10940x-dynamo.csv
benchmark-train-amp-nchw-pt112-cu113-rtx3090.csv
benchmark-train-amp-nhwc-pt112-cu113-rtx3090.csv
generate_csv_results.py
model_metadata-in1k.csv
results-imagenet-a-clean.csv
results-imagenet-a.csv
results-imagenet-r-clean.csv
results-imagenet-r.csv
results-imagenet-real.csv
results-imagenet.csv
results-imagenetv2-matched-frequency.csv
results-sketch.csv

#benchmark-infer-amp-nhwc-pt111-cu113-rtx3090.csv

import pandas as pd
df_results = pd.read_csv('results-imagenet.csv')

df_results['model_org'] = df_results['model'] 
df_results['model'] = df_results['model'].str.split('.').str[0]

We'll also add a "family" column that will allow us to group architectures into categories with similar characteristics:

Ross has told me which models he's found the most usable in practice, so I'll limit the charts to just look at these. (I also include VGG, not because it's good, but as a comparison to show how far things have come in the last few years.)

def get_data(part, col):
    # df = pd.read_csv(f'benchmark-{part}-amp-nhwc-pt240-cu124-rtx3090.csv').merge(df_results, on='model')
    if part == 'train':
        suffix = 'pt112-cu113-rtx3090'
    else:
        suffix = 'pt240-cu124-rtx3090'
        
    filename = f'benchmark-{part}-amp-nhwc-{suffix}.csv'
    df = pd.read_csv(filename).merge(df_results, on='model')
    
    df['secs'] = 1. / df[col]
    df['family'] = df.model.str.extract('^([a-z]+?(?:v2)?)(?:\d|_|$)')
    df = df[~df.model.str.endswith('gn')]
    df.loc[df.model.str.contains('in22'),'family'] = df.loc[df.model.str.contains('in22'),'family'] + '_in22'
    df.loc[df.model.str.contains('resnet.*d'),'family'] = df.loc[df.model.str.contains('resnet.*d'),'family'] + 'd'
    return df[df.family.str.contains('^re[sg]netd?|beit|convnext|levit|efficient|vit|vgg|swin')]

df = get_data('infer', 'infer_samples_per_sec')

Inference results

Here's the results for inference performance (see the last section for training performance). In this chart:

• the x axis shows how many seconds it takes to process one image (note: it's a log scale)
• the y axis is the accuracy on Imagenet
• the size of each bubble is proportional to the size of images used in testing
• the color shows what "family" the architecture is from.

Hover your mouse over a marker to see details about the model. Double-click in the legend to display just one family. Single-click in the legend to show or hide a family.

Note: on my screen, Kaggle cuts off the family selector and some plotly functionality -- to see the whole thing, collapse the table of contents on the right by clicking the little arrow to the right of "Contents".

# import plotly.express as px
# w,h = 1000,800

# def show_all(df, title, size):
#     return px.scatter(df, width=w, height=h, size=df[size]**2, title=title,
#         x='secs',  y='top1', log_x=True, color='family', hover_name='model_org', hover_data=[size])

# show_all(df, 'Inference', 'infer_img_size')

import altair as alt
alt.data_transformers.disable_max_rows()
alt.renderers.enable('html')

def show_all(df, title, size_col):
    # Ensure the data is clean for plotting
    plot_df = df.dropna(subset=['secs', 'top1', size_col]).copy()
    
    # 2. Create the base chart with logarithmic X-axis
    chart = alt.Chart(plot_df).mark_circle().encode(
        x=alt.X('secs:Q', 
                scale=alt.Scale(type='log'), 
                title='Seconds per Image (Log Scale)'),
        y=alt.Y('top1:Q', 
                scale=alt.Scale(zero=False), 
                title='Top-1 Accuracy (%)'),
        color=alt.Color('family:N', title='Model Family'),
        # Use range to control min/max marker size visually
        size=alt.Size(f'{size_col}:Q', 
                     scale=alt.Scale(range=[20, 1000]), 
                     title='Image Size'),
        tooltip=[
            alt.Tooltip('model_org:N', title='Model'),
            alt.Tooltip('family:N', title='Family'),
            alt.Tooltip('top1:Q', title='Top-1 Acc', format='.2f'),
            alt.Tooltip(f'{size_col}:Q', title='Img Size')
        ]
    ).properties(
        width=1000,
        height=800,
        title=title
    ).interactive() # Allows zooming and panning
    
    return chart

# 3. Call and display (Note: No .show() call)
# chart_infer = show_all(df, 'Inference Performance', 'infer_img_size')
# chart_infer
show_all(df, 'Inference Performance', 'infer_img_size')

print(df['infer_img_size'].describe())
print(f"Rows after dropna: {len(df.dropna(subset=['secs', 'top1', 'infer_img_size']))}")

count    1301.000000
mean      262.696387
std        67.215183
min       160.000000
25%       224.000000
50%       256.000000
75%       288.000000
max       512.000000
Name: infer_img_size, dtype: float64
Rows after dropna: 1301

That number of families can be a bit overwhelming, so I'll just pick a subset which represents a single key model from each of the families that are looking best in our plot. I've also separated convnext models into those which have been pretrained on the larger 22,000 category imagenet sample (convnext_in22) vs those that haven't (convnext). (Note that many of the best performing models were trained on the larger sample -- see the papers for details before coming to conclusions about the effectiveness of these architectures more generally.)

subs = 'levit|resnetd?|regnetx|vgg|convnext.*|efficientnetv2|beit|swin'

In this chart, I'll add lines through the points of each family, to help see how they compare -- but note that we can see that a linear fit isn't actually ideal here! It's just there to help visually see the groups.

# def show_subs(df, title, size):
#     df_subs = df[df.family.str.fullmatch(subs)]
#     return px.scatter(df_subs, width=w, height=h, size=df_subs[size]**2, title=title,
#         trendline="ols", trendline_options={'log_x':True},
#         x='secs',  y='top1', log_x=True, color='family', hover_name='model_org', hover_data=[size])

# show_subs(df, 'Inference', 'infer_img_size')

def show_subs(df, title, size_col):
    # Filter the dataframe based on the 'subs' variable (ensure 'subs' is defined globally)
    df_subs = df[df.family.str.fullmatch(subs)].copy()
    
    # Base chart for points
    base = alt.Chart(df_subs).encode(
        x=alt.X('secs', scale=alt.Scale(type='log'), title='Seconds per Image'),
        y=alt.Y('top1', scale=alt.Scale(zero=False), title='Top-1 Accuracy'),
        color='family'
    )
    
    # The Scatter layer
    points = base.mark_circle().encode(
        size=alt.Size(f'{size_col}:Q', scale=alt.Scale(range=[20, 1000])),
        tooltip=['model_org', 'family', 'top1', size_col]
    )
    
    # The Trendline layer (Linear Regression)
    # Note: Altair performs regression on the transformed values
    trendline = base.transform_regression(
        'secs', 'top1', groupby=['family', 'method'] if 'method' in df_subs else ['family'], method='poly', order=1
    ).mark_line()
    
    # Combine layers and set properties
    return (points + trendline).properties(
        width=1000,
        height=800,
        title=title
    ).interactive()

# Call the function and ensure 'subs' is defined beforehand (e.g., subs = 'resnet|vit')
chart_subs = show_subs(df, 'Inference', 'infer_img_size')
chart_subs

From this, we can see that the levit family models are extremely fast for image recognition, and clearly the most accurate amongst the faster models. That's not surprising, since these models are a hybrid of the best ideas from CNNs and transformers, so get the benefit of each. In fact, we see a similar thing even in the middle category of speeds -- the best is the ConvNeXt, which is a pure CNN, but which takes advantage of ideas from the transformers literature.

For the slowest models, beit is the most accurate -- although we need to be a bit careful of interpreting this, since it's trained on a larger dataset (ImageNet-21k, which is also used for vit models).

I'll add one other plot here, which is of speed vs parameter count. Often, parameter count is used in papers as a proxy for speed. However, as we see, there is a wide variation in speeds at each level of parameter count, so it's really not a useful proxy.

(Parameter count may be be useful for identifying how much memory a model needs, but even for that it's not always a great proxy.)

# px.scatter(df, width=w, height=h,
#     x='param_count_x',  y='secs', log_x=True, log_y=True, color='infer_img_size',
#     hover_name='model_org', hover_data=['infer_samples_per_sec', 'family']
# )

def show_param_vs_speed(df, title):
    # Clean data to avoid log issues with zero or NaN
    plot_df = df.dropna(subset=['param_count_x', 'secs', 'infer_img_size']).copy()
    plot_df = plot_df[(plot_df['param_count_x'] > 0) & (plot_df['secs'] > 0)]

    chart = alt.Chart(plot_df).mark_circle(size=60).encode(
        x=alt.X('param_count_x:Q', 
                scale=alt.Scale(type='log'), 
                title='Parameter Count (Log)'),
        y=alt.Y('secs:Q', 
                scale=alt.Scale(type='log'), 
                title='Seconds per Image (Log)'),
        color=alt.Color('infer_img_size:Q', 
                       scale=alt.Scale(scheme='viridis'), 
                       title='Image Size'),
        tooltip=[
            alt.Tooltip('model_org:N', title='Model'),
            alt.Tooltip('family:N', title='Family'),
            alt.Tooltip('param_count_x:Q', title='Params', format='.2s'),
            alt.Tooltip('secs:Q', title='Secs/Img', format='.4f'),
            alt.Tooltip('infer_samples_per_sec:Q', title='Throughput', format='.1f')
        ]
    ).properties(
        width=1000,
        height=800,
        title=title
    ).interactive()
    
    return chart

# Display the chart
chart_params = show_param_vs_speed(df, 'Parameter Count vs. Inference Speed')
chart_params

Training results

We'll now replicate the above analysis for training performance. First we grab the data:

tdf = get_data('train', 'train_samples_per_sec')

Now we can repeat the same family plot we did above:

show_all(tdf, 'Training', 'train_img_size')

...and we'll also look at our chosen subset of models:

show_subs(tdf, 'Training', 'train_img_size')

Finally, we should remember that speed depends on hardware. If you're using something other than a modern NVIDIA GPU, your results may be different. In particular, I suspect that transformers-based models might have worse performance in general on CPUs (although I need to study this more to be sure).