makemore: bigram exercises¶

  1. watch the "makemore: bigram" video on YouTube
  2. come back and complete these exercises to level up :)



Exercises¶

  1. Train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model?
  2. Split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see?
  3. Use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve?
  4. We saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of F.one_hot in favor of simply indexing into rows of W?
  5. Look up and use F.cross_entropy instead. You should achieve the same result. Can you think of why we'd prefer to use F.cross_entropy instead?
  6. Meta-exercise! Think of a fun/interesting exercise and complete it.

1. Trigram Model¶

a. Counting Approach¶

In [1]:
import math, random
import numpy as np
import pandas as pd
import seaborn as sns
import torch
import torch.nn.functional as F
import nltk
from collections import defaultdict
from tqdm import tqdm
from pprint import pprint


import matplotlib.pyplot as plt
from matplotlib.colors import Normalize
%matplotlib inline
In [2]:
words = open('../data/names.txt', 'r').read().splitlines()
words[:10]
Out[2]:
['emma',
 'olivia',
 'ava',
 'isabella',
 'sophia',
 'charlotte',
 'mia',
 'amelia',
 'harper',
 'evelyn']
In [3]:
print(f"the shortest word length in the corpus is: {min(len(w) for w in words)}")
print(f"the longest word length in the corpus is: {max(len(w) for w in words)}")
print(f"the length of the corpus document (num of words) is: {len(words)}")

the shortest word length in the corpus is: 2
the longest word length in the corpus is: 15
the length of the corpus document (num of words) is: 32033
In [4]:
t = {}
for w in words:
    chs = ['.'] + list(w) + ['.']
    for ch1, ch2, ch3 in zip(chs, chs[1:], chs[2:]):
        trigram = (ch1, ch2, ch3)
        t[trigram] = t.get(trigram, 0) + 1
pprint(f'the 10 most common trigrams: {sorted(t.items(), key = lambda kv: -kv[1])[:10]}')

("the 10 most common trigrams: [(('a', 'h', '.'), 1714), (('n', 'a', '.'), "
 "1673), (('a', 'n', '.'), 1509), (('o', 'n', '.'), 1503), (('.', 'm', 'a'), "
 "1453), (('.', 'j', 'a'), 1255), (('.', 'k', 'a'), 1254), (('e', 'n', '.'), "
 "1217), (('l', 'y', 'n'), 976), (('y', 'n', '.'), 953)]")
In [5]:
N = torch.zeros((27, 27, 27), dtype=torch.int32)

chars = sorted(list(set(''.join(words))))
stoi = {s:i+1 for i,s in enumerate(chars)}
stoi['.'] = 0
itos = {i:s for s,i in stoi.items()}

for w in words:
    chs = ['.'] + list(w) + ['.']
    for ch1, ch2, ch3 in zip(chs, chs[1:], chs[2:]):
        ix1 = stoi[ch1]
        ix2 = stoi[ch2]
        ix3 = stoi[ch3]
        N[ix1, ix2, ix3] +=1
In [6]:
N[0][1]
Out[6]:
tensor([  0, 207, 190,  31, 366,  55,  21,  17,  91, 154,  27,  75, 632, 384,
        623,  10,  17,   9, 482, 194,  72, 152, 243,   6,  27, 173, 152],
       dtype=torch.int32)
In [7]:
p = N[0][1].float()    
p = p / p.sum()

g = torch.Generator().manual_seed(2147483647)
ix = torch.multinomial(p, num_samples=1, replacement=True, generator=g).item()
p.shape, ix, itos[ix]
Out[7]:
(torch.Size([27]), 18, 'r')
In [8]:
P = (N+1).float() 
P /= P.sum(2, keepdim=True) 

g = torch.Generator().manual_seed(2147483647)

for i in range(5):
    out = []
    ix, jx = 0, 0
    while True:
        p = P[ix][jx]
        next_letter = torch.multinomial(p, num_samples=1, replacement=True, generator=g).item()
        ix = jx
        jx = next_letter
        out.append(itos[next_letter])
        if next_letter == 0:
            break
    print(''.join(out))

quia.
yu.
quinslyntien.
nolliahi.
ha.
In [9]:
%%time
log_likelihood = 0.0
n = 0
for w in words:
    chs = ['.'] + list(w) + ['.']
    for ch1, ch2, ch3 in zip(chs, chs[1:], chs[2:]):
        ix1 = stoi[ch1]
        ix2 = stoi[ch2]
        ix3 = stoi[ch3]
        prob = P[ix1, ix2, ix3]
        logprob = torch.log(prob)
        log_likelihood += logprob
        n += 1
        
print(f'\nlog_likelihood = {log_likelihood:.4f}')
nll = -log_likelihood
print(f'negative log_likelihood = {nll:.4f}\n')
avg_nll = nll/n
print(f'average negative log_likelihood = {avg_nll:.4f}')

log_likelihood = -410414.9688
negative log_likelihood = 410414.9688

average negative log_likelihood = 2.0927
CPU times: user 1.51 s, sys: 19.2 ms, total: 1.53 s
Wall time: 1.56 s

b. Neural Network Approach¶

In [10]:
def create_trigrams(words_list:list):
    xs, ys = [], []

    for w in words_list:
        chs = ['.'] + list(w) + ['.']
        for ch1, ch2, ch3 in zip(chs, chs[1:], chs[2:]):
            ix1 = stoi[ch1]
            ix2 = stoi[ch2]
            ix3 = stoi[ch3]
            #print(ch1, ch2, ch3)
            xs.append([ix1, ix2])
            ys.append(ix3)

    xs = torch.tensor(xs) 
    ys = torch.tensor(ys)
    
    return xs, ys
In [11]:
# Create dataset and initialize the neural net
xnn, ynn = create_trigrams(words)
num = ynn.nelement()
g = torch.Generator().manual_seed(2147483647)
W_nn = torch.randn((27*2, 27), generator=g, requires_grad=True)
In [12]:
num == ynn.shape[0]
Out[12]:
True
In [13]:
%%time
n_iter_nn, lr_nn, C_nn = 250, 50, 0.0
losses_nn = []
# gradient descent
for k in range(n_iter_nn):

    # forward pass
    xenc_nn = F.one_hot(xnn, num_classes=27).float()            # input to the network: one-hot encoding
    logits_nn = xenc_nn.view(-1, 27*2) @ W_nn                   # predict log-counts
    counts_nn = logits_nn.exp()                                 # counts, equivalent to N
    probs_nn = counts_nn / counts_nn.sum(1, keepdim=True)       # probabilities for next character
    loss_nn = -probs_nn[torch.arange(num), ynn].log().mean() + C_nn*(W_nn**2).mean() 
    #print(loss_nn.item())
    losses_nn.append(loss_nn)
    if k > (n_iter_nn - 5):
        print(loss_nn.item())

    # backward pass
    W_nn.grad = None # set to zero the gradient
    loss_nn.backward()

    # update
    W_nn.data += -lr_nn * W_nn.grad
print(f'\naverage negative log_likelihood = {loss_nn.item():.4f}\n')

2.245729923248291
2.24568247795105
2.245635747909546
2.2455894947052

average negative log_likelihood = 2.2456

CPU times: user 48.2 s, sys: 7.36 s, total: 55.6 s
Wall time: 25.3 s



2. Model Validation¶

In [14]:
def data_split(xs, ys, train_percent:int, val_percent:int, test_percent:int):
    '''
    split dataset into train, validation and test sets
    '''
    assert train_percent + val_percent + test_percent == 100
    
    total = len(xs)
    train_end, val_end = int(total * train_percent / 100), int(total * (train_percent + val_percent) / 100)
    Xtr, Ytr = xs[:train_end],  ys[:train_end]
    Xval, Yval = xs[train_end:val_end],  ys[train_end:val_end]
    Xtest, Ytest = xs[val_end:],  ys[val_end:]
    
    assert len(Xtr) == len(Ytr)
    assert len(Xval) == len(Yval)
    assert len(Xtest) == len(Ytest)
    
    return Xtr, Ytr, Xval, Yval, Xtest, Ytest
In [15]:
xnn2, ynn2 = create_trigrams(words)
x_tr, y_tr, x_val, y_val, x_test, y_test = data_split(xnn2, ynn2, 80, 10, 10)
g = torch.Generator().manual_seed(2147483647)
W_nn2 = torch.randn((27*2, 27), generator=g, requires_grad=True)
In [16]:
print(f'{len(x_tr)/len(xnn2)*100:.0f}% training set')
print(f'{len(x_val)/len(xnn2)*100:.0f}% validation set')
print(f'{len(x_test)/len(xnn2)*100:.0f}% test set')

80% training set
10% validation set
10% test set
In [17]:
%%time
losses_tr = []
n_iter_tr, lr_tr, C_tr = 250, 50, 0.0

# gradient descent
for k in range(n_iter_tr):

    # forward pass
    xenc_tr = F.one_hot(x_tr, num_classes=27).float()            
    logits_tr = xenc_tr.view(-1, 27*2) @ W_nn2                 
    counts_tr = logits_tr.exp()                                 
    probs_tr = counts_tr / counts_tr.sum(1, keepdim=True)  + C_tr*(W_nn2**2).mean()    
    loss_tr = -probs_tr[torch.arange(y_tr.shape[0]), y_tr].log().mean() 
    #print(loss_tr.item())
    losses_tr.append(loss_tr)
    if k > (n_iter_tr - 5):
        print(loss_tr.item())

    # backward pass
    W_nn2.grad = None # set to zero the gradient
    loss_tr.backward()

    # update
    W_nn2.data += -lr_tr * W_nn2.grad
    
#print(f'\naverage negative log_likelihood = {loss_tr.item():.4f}\n')

print(f"\ntrain_loss (final): {losses_tr[-1]:.4f}\n")
plt.plot(losses_tr)
plt.xlabel("epoch")
plt.ylabel("loss")
plt.title("Loss on training data")
plt.grid() 
plt.plot()

2.20723819732666
2.2071897983551025
2.207141637802124
2.207094192504883

train_loss (final): 2.2071

CPU times: user 37.7 s, sys: 6.08 s, total: 43.8 s
Wall time: 19.6 s
Out[17]:
[]
No description has been provided for this image
In [18]:
xenc_val = F.one_hot(x_val, num_classes=27).float()
logits_val = xenc_val.view(-1, 27*2) @ W_nn2
counts_val = logits_val.exp()
probs_val = counts_val/ counts_val.sum(1, keepdim=True)

loss_val = -probs_val[torch.arange(y_val.shape[0]), y_val].log().mean()
print(f"validation loss: {loss_val.item():.4f}")

validation loss: 2.4321
In [19]:
xenc_test = F.one_hot(x_test, num_classes=27).float()
logits_test = xenc_test.view(-1, 27*2) @ W_nn2
counts_test = logits_test.exp()
probs_test = counts_test/ counts_test.sum(1, keepdim=True)

loss_test = -probs_test[torch.arange(y_test.shape[0]), y_test].log().mean()
print(f"test loss: {loss_test.item():.4f}")

test loss: 2.4324

For the trigram model, the loss after training the model on the training set was 2.2071. Evaluating the model on the validation and test set yielded losses of 2.4321 and 2.4324 respectively. The test and validation losses are very similar and marginally higher than the traiing loss so that means the model is not overfitting (generalizes well to new unseen data: model robustness).



3. Regularization¶

In [20]:
num_tr, num_val, num_test = y_tr.nelement(), y_val.nelement(), y_test.nelement()
In [21]:
reg_param_list = [0, 0.001, 0.01, 0.02, 0.05, 0.075, 0.1, 0.25, 0.3, 0.4, 0.5]
#reg_param_list = np.geomspace(0.001, 1, 12)

loss_dict = {} # <reg_strength: [final train loss, final val loss]>
n_iter = 200
lr = 50

for C in tqdm(reg_param_list):
    # Initialize model & losses
    W_reg = torch.randn((27*2, 27), generator=g, requires_grad=True)
    #tr_losses, val_losses= [], []
    
    # gradient descent
    for k in range(n_iter):

        # forward pass
        xenc = F.one_hot(x_tr, num_classes=27).float()       
        logits = xenc.view(-1, 27*2) @ W_reg                 
        counts = logits.exp()                             
        probs = counts / counts.sum(1, keepdim=True)     
        tr_loss = -probs[torch.arange(num_tr), y_tr].log().mean() + C*(W_reg**2).mean() 
        #print(loss.item())
        #tr_losses.append(tr_loss)

        # Zero grad & backward pass
        W_reg.grad = None 
        tr_loss.backward()

        # update
        W_reg.data += -lr * W_reg.grad     


    xenc_val = F.one_hot(x_val, num_classes=27).float()      
    logits = xenc_val.view(-1, 27*2) @ W_reg                   
    counts = logits.exp()                             
    probs = counts / counts.sum(1, keepdim=True)     
    val_loss = -probs[torch.arange(num_val), y_val].log().mean()

       
    print(f'For regularization parameter = {C:.4f} ---> train loss: {tr_loss.item():.4f},\
    validation loss: {val_loss.item():.4f}')
    
    _tr_loss = np.round(tr_loss.item(), decimals=4)
    _val_loss = np.round(val_loss.item(), decimals=4)
    _C = np.round(C, decimals=4)
    loss_dict[_C] = [_tr_loss, _val_loss]

  9%|▉         | 1/11 [00:16<02:41, 16.11s/it]
For regularization parameter = 0.0000 ---> train loss: 2.2110,    validation loss: 2.4363

 18%|█▊        | 2/11 [00:34<02:30, 16.78s/it]
For regularization parameter = 0.0010 ---> train loss: 2.2108,    validation loss: 2.4331

 27%|██▋       | 3/11 [00:53<02:18, 17.34s/it]
For regularization parameter = 0.0100 ---> train loss: 2.2212,    validation loss: 2.4338

 36%|███▋      | 4/11 [01:14<02:09, 18.53s/it]
For regularization parameter = 0.0200 ---> train loss: 2.2312,    validation loss: 2.4321

 45%|████▌     | 5/11 [01:30<01:46, 17.70s/it]
For regularization parameter = 0.0500 ---> train loss: 2.2539,    validation loss: 2.4331

 55%|█████▍    | 6/11 [01:45<01:24, 16.88s/it]
For regularization parameter = 0.0750 ---> train loss: 2.2698,    validation loss: 2.4351

 64%|██████▎   | 7/11 [02:04<01:10, 17.53s/it]
For regularization parameter = 0.1000 ---> train loss: 2.2833,    validation loss: 2.4390

 73%|███████▎  | 8/11 [02:19<00:50, 16.99s/it]
For regularization parameter = 0.2500 ---> train loss: 2.3468,    validation loss: 2.4605

 82%|████████▏ | 9/11 [02:34<00:32, 16.26s/it]
For regularization parameter = 0.3000 ---> train loss: 2.3635,    validation loss: 2.4672

 91%|█████████ | 10/11 [02:51<00:16, 16.50s/it]
For regularization parameter = 0.4000 ---> train loss: 2.3929,    validation loss: 2.4798

100%|██████████| 11/11 [03:06<00:00, 16.91s/it]
For regularization parameter = 0.5000 ---> train loss: 2.4185,    validation loss: 2.4913





















In [22]:






loss_dict





















Out[22]:




{0: [2.211, 2.4363],
 0.001: [2.2108, 2.4331],
 0.01: [2.2212, 2.4338],
 0.02: [2.2312, 2.4321],
 0.05: [2.2539, 2.4331],
 0.075: [2.2698, 2.4351],
 0.1: [2.2833, 2.439],
 0.25: [2.3468, 2.4605],
 0.3: [2.3635, 2.4672],
 0.4: [2.3929, 2.4798],
 0.5: [2.4185, 2.4913]}




















In [23]:






plt.title(f"Effect of Regularization/Smoothing on Trigram Model")
plt.plot(list(loss_dict.keys()), [train_loss for train_loss, _ in loss_dict.values()], "o--", label="train")
plt.plot(list(loss_dict.keys()), [val_loss for _, val_loss in loss_dict.values()], "o--", label="val")
plt.xlabel("Regularization Strength, C")
plt.ylabel("Loss")
plt.grid()
plt.legend()
plt.plot()
#plt.show()





















Out[23]:




[]











No description has been provided for this image























Increasing the regularization parameter C increases both the validation and training losses, but the training losses grow faster than the validation losses.




























4. Indexing into Rows of W (Discard One-Hot Encoding)¶






















Utilising the inputs xnn and weight matrix W_nn from problem 1b (neural network approach for trigram LM). Let's compare the results for one-hot encoding xnn, (xenc_nn) and indexing into rows of W_nn.



















In [24]:






logits_idx = W_nn[xnn[:, 0]] + W_nn[xnn[:, 1] + 27]
counts_idx = logits_idx.exp()
probs_idx = counts_idx / counts_idx.sum(1, keepdim=True)
loss_idx = -1 * probs_idx[torch.arange(ynn.shape[0]), ynn].log().mean()
#loss_idx

print(f"validation loss via indexing into rows of W: {loss_idx.item():.4f}")

























validation loss via indexing into rows of W: 2.2455





















In [25]:






logits_ohe = xenc_nn.view(-1, 27*2) @ W_nn
counts_ohe = logits_ohe.exp()
probs_ohe = counts_ohe / counts_ohe.sum(1, keepdim=True)
loss_ohe = -1 * probs_ohe[torch.arange(ynn.shape[0]), ynn].log().mean()
#loss_ohe

print(f"validation loss via one-hot encoding: {loss_ohe.item():.4f}")

























validation loss via one-hot encoding: 2.2455


























As we can see from the cells above, we get the same results using either one-hot encoding or indexing into rows of W.




























5. F.cross_entropy¶


















In [26]:






logits = xenc_nn.view(-1, 2*27) @ W_nn
counts = logits.exp()
probs = counts / counts.sum(1, keepdim=True)
loss = -probs[torch.arange(ynn.shape[0]), ynn].log().mean()
print(f"Average Negative Log Likelihood Loss: {loss.item():.4f}")

























Average Negative Log Likelihood Loss: 2.2455





















In [27]:






loss_ce = F.cross_entropy(logits, ynn)
print(f"Cross Entropy Loss: {loss_ce.item():.4f}")

























Cross Entropy Loss: 2.2455

























We get the same results using both methods. However, F.cross_entropy is preferable because it combines the softmax step and loss function step together at once. Basically, F.cross_entropy incorporates transforming the neural net outputs to probabilities with calculating the loss in one step (logits -> losses)


    

  • NN outputs to probabilities: logits -> [exp(x)] -> counts -> [normalize(x)] -> probs
    • 

  • Probabilities to losses: probs -> [Average negative log likelihood(x)] -> losses
    • 





























6. meta-exercise: PyTorch Base Class for all Neural Network Modules¶






















Implementation steps of class:


    

  • Initialize
      

    • Apply a linear transformation to incoming data: y = xA.T + b, using torch.nn.Linear
      • 

    • Initialize weights with normal distribution (mean of 0 and STD of 1)
      • 

    

    • 

  • Forward function
      

    • No activation function


      • 

    

    • 



















In [44]:






xs, ys = [], []

for w in words:
    chs = ['.'] + list(w) + ['.']
    for ch1, ch2, ch3 in zip(chs, chs[1:], chs[2:]):
        ix1 = stoi[ch1]
        ix2 = stoi[ch2]
        ix3 = stoi[ch3]
        #print(ch1, ch2, ch3)
        xs.append([ix1, ix2])
        ys.append(ix3)

xs = torch.tensor(xs) 
ys = torch.tensor(ys)





















In [45]:






class MLP(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = torch.nn.Linear(2*27, 27) # first fully connected layer of MLP
        torch.nn.init.normal_(self.fc1.weight, mean=0, std=1)
        #torch.nn.init.xavier_uniform(self.fc1.weight)

    def forward(self, x):
        logits = W_nn[x[:, 0]] + W_nn[x[:, 1] + 27]
        return logits





















In [46]:






C = 0
model = MLP()
# gradient descent
for k in range(250):

    # forward pass
    logits = model(xs)
    loss = F.cross_entropy(logits, ys) 
    loss += C * (model.fc1.weight ** 2).mean() # added regularization term
    
#     if k % 10 == 0:
#         print(f'After {k} iterations, the loss is {loss.item():.4f}')

    # backward pass
    model.fc1.zero_grad() # set to zero the gradient (reset gradients of the layer)
    loss.backward()

    # update
    model.fc1.weight.data += -50 * model.fc1.weight.grad
print(f"After {k} iterations, the loss is {loss.item():.4f}")

























After 249 iterations, the loss is 2.2455





















In [47]:






model.fc1.weight, model.fc1.bias





















Out[47]:




(Parameter containing:
 tensor([[ 1.7855,  0.3460, -1.2854,  ...,  0.5336, -0.8839,  0.3031],
         [ 0.1225, -1.0614,  1.0801,  ..., -0.3535, -0.5087, -1.2049],
         [-0.0594, -0.4437,  0.3638,  ...,  0.1334,  0.1140,  0.4028],
         ...,
         [-1.1852,  1.0893, -1.8360,  ...,  0.0390,  1.0074,  0.5208],
         [ 1.0159, -0.2812,  1.5159,  ..., -0.1665,  1.3235, -0.6612],
         [ 2.1103,  0.6758, -1.7507,  ...,  0.7648,  0.6888, -0.0393]],
        requires_grad=True),
 Parameter containing:
 tensor([-0.0118,  0.0219,  0.0048,  0.0952, -0.0755, -0.1241,  0.0293, -0.0502,
         -0.1163,  0.0890,  0.1085, -0.1187, -0.0349,  0.0860, -0.1105, -0.0938,
         -0.0551, -0.1208, -0.0525, -0.0991, -0.0487,  0.1044, -0.0580, -0.1296,
          0.1280, -0.0386, -0.1206], requires_grad=True))




















In [48]:






C = 0
model2 = MLP()
optimizer = torch.optim.SGD(model2.parameters(), lr = 50)


# gradient descent
for k in range(250):

    # forward pass
    logits = model(xs)
    loss = F.cross_entropy(logits, ys) 
    loss += C * (model.fc1.weight.data ** 2).mean() # added regularization term
    
#     if k % 10 == 0:
#         print(f'After {k} iterations, the loss is {loss.item():.4f}')

    # Backward pass
    optimizer.zero_grad()
    loss.backward()

    # Update the parameters
    optimizer.step()
print(f"After {k} iterations, the loss is {loss.item():.4f}")

























After 249 iterations, the loss is 2.2455





















In [49]:






model2.fc1.weight.shape, model2.fc1.bias.shape





















Out[49]:




(torch.Size([27, 54]), torch.Size([27]))




















In [50]:






model2.fc1.weight, model2.fc1.bias





















Out[50]:




(Parameter containing:
 tensor([[-1.1215, -0.0396,  0.1958,  ..., -0.2544,  0.4298, -0.4357],
         [ 1.3369, -0.1873, -1.1479,  ..., -0.8545,  0.0814,  2.1752],
         [ 0.1292,  1.4875,  0.2979,  ...,  0.1874,  0.9836, -0.0892],
         ...,
         [-0.6958,  0.1216, -1.8677,  ...,  1.2131,  0.4728, -0.7404],
         [ 0.8460,  0.1320,  0.7196,  ..., -0.0559, -0.4908, -0.4020],
         [-0.6621,  1.7270, -1.1435,  ...,  1.6317,  0.1695,  0.8134]],
        requires_grad=True),
 Parameter containing:
 tensor([ 0.0715, -0.0081, -0.0726, -0.0865,  0.0413, -0.1155, -0.0556, -0.1025,
         -0.0778,  0.0568,  0.1281, -0.0204, -0.0799,  0.0951, -0.0248,  0.0988,
          0.1246, -0.0578, -0.0071,  0.0510,  0.0348, -0.1114,  0.0883, -0.0448,
         -0.0798, -0.1314, -0.0539], requires_grad=True))




















In [51]:






# forward pass
logits = model2(xnn)
print("Shape of logits:", logits.shape)
print("Shape of ynn:", ynn.shape)
loss = F.cross_entropy(logits, ynn)
loss

























Shape of logits: torch.Size([196113, 27])
Shape of ynn: torch.Size([196113])









Out[51]:




tensor(2.2455, grad_fn=<NllLossBackward0>)