PyTorch function song & examples: Autograd, Random Number Generation, Loss Functions, optimization

Autograd:

1. tensor.requires_grad_(True) – Enables gradient computation for the tensor.
2. tensor.backward() – Computes the gradient of the tensor (used in backpropagation).
3. tensor.grad – Retrieves the gradient for a tensor.
4. torch.autograd.grad(outputs, inputs) – Computes and returns the gradients of the output with respect to the inputs.

Random Number Generation:

1. torch.manual_seed(seed) – Sets the seed for generating random numbers for reproducibility.
2. torch.cuda.manual_seed(seed) – Sets the seed for random number generation on CUDA.

Loss Functions:

1. torch.nn.functional.mse_loss(output, target) – Mean squared error loss.
2. torch.nn.functional.cross_entropy(output, target) – Cross-entropy loss for classification.

Optimization:

1. torch.optim.SGD(params, lr) – Stochastic Gradient Descent optimizer.
2. torch.optim.Adam(params, lr) – Adam optimizer.
3. optimizer.step() – Updates model parameters based on gradients.
4. optimizer.zero_grad() – Resets gradients before backpropagation.

Examples for Autograd, Random Number Generation, Loss Functions, and Optimization in PyTorch:

---

Autograd

1. tensor.requires_grad_(True) (Enable gradient computation for the tensor)

import torch

# Create a tensor with requires_grad=True to track computation
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x ** 2  # Square the tensor (y = x^2)

# Perform backpropagation
y.sum().backward()

# Get the gradients (dy/dx)
print(x.grad)

Output:

tensor([4., 6.])

---

2. tensor.backward() (Computes the gradients)

x = torch.tensor([2.0, 3.0], requires_grad=True)
z = 3 * x ** 2 + 2 * x

# Perform backpropagation
z.sum().backward()

# Gradient of the function with respect to x
print(x.grad)

Output:

tensor([14., 20.])

---

3. tensor.grad (Access the gradient of the tensor after backward())

x = torch.tensor([2.0], requires_grad=True)
y = 5 * x ** 3  # y = 5 * x^3
y.backward()  # Compute gradients
print(x.grad)  # dy/dx = 15 * x^2

Output:

tensor([60.])

---

4. torch.autograd.grad(outputs, inputs) (Compute and return gradients without retaining computational graph)

x = torch.tensor([2.0], requires_grad=True)
y = 5 * x ** 3
grad = torch.autograd.grad(outputs=y, inputs=x)
print(grad)  # Same as x.grad, but won't keep the graph for future computations

Output:

(tensor([60.]),)

---

Random Number Generation

1. torch.manual_seed(seed) (Set the seed for random number generation)

torch.manual_seed(42)
tensor = torch.rand(2, 2)
print(tensor)

Output:

tensor([[0.8823, 0.9150],
        [0.3829, 0.9593]])

---

2. torch.cuda.manual_seed(seed) (Set the seed for random number generation on CUDA)

torch.cuda.manual_seed(42)
tensor = torch.rand(2, 2).cuda()  # Random tensor on GPU
print(tensor)

---

Loss Functions

1. torch.nn.functional.mse_loss(output, target) (Mean Squared Error Loss)

import torch.nn.functional as F

output = torch.tensor([0.5, 1.0, 2.0])
target = torch.tensor([0.0, 1.0, 2.0])

# Calculate Mean Squared Error Loss
loss = F.mse_loss(output, target)
print(loss)

Output:

tensor(0.0833)

---

2. torch.nn.functional.cross_entropy(output, target) (Cross-Entropy Loss for classification)

output = torch.tensor([[1.5, 2.0, 0.5]])  # Class scores (logits)
target = torch.tensor([1])  # True class label

# Calculate Cross Entropy Loss
loss = F.cross_entropy(output, target)
print(loss)

Output:

tensor(0.8971)

---

Optimization

1. torch.optim.SGD(params, lr) (Stochastic Gradient Descent optimizer)

# Define a simple model
model = torch.nn.Linear(1, 1)

# Define optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Dummy input and target
input = torch.tensor([[1.0]])
target = torch.tensor([[2.0]])

# Forward pass
output = model(input)
loss = F.mse_loss(output, target)

# Backward pass and optimization step
optimizer.zero_grad()  # Zero the gradients
loss.backward()        # Backpropagate the loss
optimizer.step()       # Update model parameters

print(list(model.parameters()))

---

2. torch.optim.Adam(params, lr) (Adam optimizer)

# Define model
model = torch.nn.Linear(1, 1)

# Define Adam optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

# Dummy input and target
input = torch.tensor([[1.0]])
target = torch.tensor([[2.0]])

# Forward pass
output = model(input)
loss = F.mse_loss(output, target)

# Backward pass and optimization step
optimizer.zero_grad()  # Zero the gradients
loss.backward()        # Backpropagate the loss
optimizer.step()       # Update model parameters

print(list(model.parameters()))

---

3. optimizer.step() (Update the model parameters based on the gradients)

This function is used after calling loss.backward() to update the model’s parameters based on the gradients calculated during backpropagation.

optimizer.step()  # Updates the parameters based on gradients

---

4. optimizer.zero_grad() (Reset gradients before backpropagation)

Before performing backpropagation, it’s important to zero out the gradients to prevent accumulation from previous iterations.

optimizer.zero_grad()  # Zero the gradients before backpropagation

---