```
import torch
= torch.ones(2, 4, requires_grad=True)
x x
```

```
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.]], requires_grad=True)
```

Author

Marie-Hélène Burle

PyTorch has automatic differentiation capabilities—meaning that it can track all the operations performed on tensors during the forward pass and compute all the gradients automatically for the backpropagation—thanks to its package torch.autograd.

Let’s have a look at this.

**Derivative of a function:**

Rate of change of a function with a single variable w.r.t. its variable.

**Partial derivative:**

Rate of change of a function with multiple variables w.r.t. one variable while other variables are considered as constants.

**Gradient:**

Vector of partial derivatives of function with several variables.

**Differentiation:**

Calculation of the derivatives of a function.

**Chain rule:**

Formula to calculate the derivatives of composite functions.

**Automatic differentiation:**

Automatic computation of partial derivatives by algorithms.

PyTorch does not track all the computations on all the tensors (this would be extremely memory intensive!). To start tracking computations on a vector, set the `requires_grad`

attribute to `True`

:

```
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.]], requires_grad=True)
```

`grad_fun`

attributeWhenever a tensor is created by an operation involving a tracked tensor, it has a `grad_fun`

attribute:

You don’t want to track more than is necessary. There are multiple ways to avoid tracking what you don’t want.

You can stop tracking computations on a tensor with the method `detach`

:

You can change its `requires_grad`

flag:

```
tensor([[0., 0., 0.],
[0., 0., 0.]], requires_grad=True)
```

Alternatively, you can wrap any code you don’t want to track under `with torch.no_grad()`

:

```
tensor([[2., 2., 2., 2.],
[2., 2., 2., 2.]])
```

Compare this with what we just did above.

Let’s imagine that \(x\), \(y\), and \(z\) are tensors containing the parameters of a model and that the error \(e\) could be calculated with the equation:

\[e=2x^4-y^3+3z^2\]

Let’s see how we would do this manually.

First, we need the model parameters tensors:

We calculate \(e\) following the above equation:

The gradients of the error \(e\) w.r.t. the parameters \(x\), \(y\), and \(z\) are:

\[\frac{de}{dx}=8x^3\] \[\frac{de}{dy}=-3y^2\] \[\frac{de}{dz}=6z\]

We can calculate them with:

For this method, we need to define our model parameters with `requires_grad`

set to `True`

:

\(e\) is calculated in the same fashion (except that here, all the computations on \(x\), \(y\), and \(z\) are tracked):

The backward propagation is done automatically with:

And we have our 3 partial derivatives:

The result is the same, as can be tested with:

Of course, calculating the gradients manually here was extremely easy, but imagine how tedious and lengthy it would be to write the chain rules to calculate the gradients of all the composite functions in a neural network manually.