PyTorch function song & examples: Autograd, Random Number Generation, Loss Functions, optimization

Autograd:

1. tensor.requires_grad_(True) – Enables gradient computation for the tensor.
2. tensor.backward() – Computes the gradient of the tensor (used in backpropagation).
3. tensor.grad – Retrieves the gradient for a tensor.
4. torch.autograd.grad(outputs, inputs) – Computes and returns the gradients of the output with respect to the inputs.

Random Number Generation:

1. torch.manual_seed(seed) – Sets the seed for generating random numbers for reproducibility.
2. torch.cuda.manual_seed(seed) – Sets the seed for random number generation on CUDA.

Loss Functions:

1. torch.nn.functional.mse_loss(output, target) – Mean squared error loss.
2. torch.nn.functional.cross_entropy(output, target) – Cross-entropy loss for classification.

Optimization:

1. torch.optim.SGD(params, lr) – Stochastic Gradient Descent optimizer.
2. torch.optim.Adam(params, lr) – Adam optimizer.
3. optimizer.step() – Updates model parameters based on gradients.
4. optimizer.zero_grad() – Resets gradients before backpropagation.

Examples for Autograd, Random Number Generation, Loss Functions, and Optimization in PyTorch:

Autograd

1. tensor.requires_grad_(True) (Enable gradient computation for the tensor)

import torch

# Create a tensor with requires_grad=True to track computation
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x ** 2  # Square the tensor (y = x^2)

# Perform backpropagation
y.sum().backward()

# Get the gradients (dy/dx)
print(x.grad)

Output:

tensor([4., 6.])

2. tensor.backward() (Computes the gradients)

x = torch.tensor([2.0, 3.0], requires_grad=True)
z = 3 * x ** 2 + 2 * x

# Perform backpropagation
z.sum().backward()

# Gradient of the function with respect to x
print(x.grad)

Output:

tensor([14., 20.])

3. tensor.grad (Access the gradient of the tensor after backward())

x = torch.tensor([2.0], requires_grad=True)
y = 5 * x ** 3  # y = 5 * x^3
y.backward()  # Compute gradients
print(x.grad)  # dy/dx = 15 * x^2

Output:

tensor([60.])

4. torch.autograd.grad(outputs, inputs) (Compute and return gradients without retaining computational graph)

x = torch.tensor([2.0], requires_grad=True)
y = 5 * x ** 3
grad = torch.autograd.grad(outputs=y, inputs=x)
print(grad)  # Same as x.grad, but won't keep the graph for future computations

Output:

(tensor([60.]),)

Random Number Generation

1. torch.manual_seed(seed) (Set the seed for random number generation)

torch.manual_seed(42)
tensor = torch.rand(2, 2)
print(tensor)

Output:

tensor([[0.8823, 0.9150],
        [0.3829, 0.9593]])

2. torch.cuda.manual_seed(seed) (Set the seed for random number generation on CUDA)

torch.cuda.manual_seed(42)
tensor = torch.rand(2, 2).cuda()  # Random tensor on GPU
print(tensor)

Loss Functions

1. torch.nn.functional.mse_loss(output, target) (Mean Squared Error Loss)

import torch.nn.functional as F

output = torch.tensor([0.5, 1.0, 2.0])
target = torch.tensor([0.0, 1.0, 2.0])

# Calculate Mean Squared Error Loss
loss = F.mse_loss(output, target)
print(loss)

Output:

tensor(0.0833)

2. torch.nn.functional.cross_entropy(output, target) (Cross-Entropy Loss for classification)

output = torch.tensor([[1.5, 2.0, 0.5]])  # Class scores (logits)
target = torch.tensor([1])  # True class label

# Calculate Cross Entropy Loss
loss = F.cross_entropy(output, target)
print(loss)

Output:

tensor(0.8971)

Optimization

1. torch.optim.SGD(params, lr) (Stochastic Gradient Descent optimizer)

# Define a simple model
model = torch.nn.Linear(1, 1)

# Define optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Dummy input and target
input = torch.tensor([[1.0]])
target = torch.tensor([[2.0]])

# Forward pass
output = model(input)
loss = F.mse_loss(output, target)

# Backward pass and optimization step
optimizer.zero_grad()  # Zero the gradients
loss.backward()        # Backpropagate the loss
optimizer.step()       # Update model parameters

print(list(model.parameters()))

2. torch.optim.Adam(params, lr) (Adam optimizer)

# Define model
model = torch.nn.Linear(1, 1)

# Define Adam optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

# Dummy input and target
input = torch.tensor([[1.0]])
target = torch.tensor([[2.0]])

# Forward pass
output = model(input)
loss = F.mse_loss(output, target)

# Backward pass and optimization step
optimizer.zero_grad()  # Zero the gradients
loss.backward()        # Backpropagate the loss
optimizer.step()       # Update model parameters

print(list(model.parameters()))

3. optimizer.step() (Update the model parameters based on the gradients)

This function is used after calling loss.backward() to update the model’s parameters based on the gradients calculated during backpropagation.

optimizer.step()  # Updates the parameters based on gradients

4. optimizer.zero_grad() (Reset gradients before backpropagation)

Before performing backpropagation, it’s important to zero out the gradients to prevent accumulation from previous iterations.

optimizer.zero_grad()  # Zero the gradients before backpropagation