PyTorch tensors

Before information can be processed by algorithms, it needs to be converted to floating point numbers. Indeed, you don’t pass a sentence or an image through a model; instead you input numbers representing a sequence of words or pixel values.

All these floating point numbers need to be stored in a data structure. The most suited structure is multidimensional (to hold several layers of information) and homogeneous—all data of the same type—for efficiency.

Python already has several multidimensional array structures (e.g. NumPy’s ndarray) but the particularities of deep learning call for special characteristics such as the ability to run operations on GPUs and/or in a distributed fashion, the ability to keep track of computation graphs for automatic differentiation, and different defaults (lower precision for improved training performance).

The PyTorch tensor is a Python data structure with these characteristics that can easily be converted to/from NumPy’s ndarray and integrates well with other Python libraries such as Pandas.

In this section, we will explore the basics of PyTorch tensors.

Importing PyTorch

First of all, we need to import the torch library:

import torch

We can check its version with:

torch.__version__

'2.0.1'

Creating tensors

There are many ways to create tensors:

• torch.tensor:   Input individual values
• torch.arange:   1D tensor with a sequence of integers
• torch.linspace:  1D linear scale tensor
• torch.logspace:  1D log scale tensor
• torch.rand:     Random numbers from a uniform distribution on [0, 1)
• torch.randn:     Numbers from the standard normal distribution
• torch.randperm:   Random permutation of integers
• torch.empty:     Uninitialized tensor
• torch.zeros:     Tensor filled with 0
• torch.ones:     Tensor filled with 1
• torch.eye:       Identity matrix

From input values

t = torch.tensor(3)

Your turn:

Without using the shape descriptor, try to get the shape of the following tensors:

torch.tensor([0.9704, 0.1339, 0.4841])

torch.tensor([[0.9524, 0.0354],
        [0.9833, 0.2562],
        [0.0607, 0.6420]])

torch.tensor([[[0.4604, 0.2699],
         [0.8360, 0.0317],
         [0.3289, 0.1171]]])

torch.tensor([[[[0.0730, 0.8737],
          [0.2305, 0.4719],
          [0.0796, 0.2745]]],

        [[[0.1534, 0.9442],
          [0.3287, 0.9040],
          [0.0948, 0.1480]]]])

Let’s create a random tensor with a single element:

t = torch.rand(1)
t

tensor([0.0582])

We can extract the value from a tensor with one element:

t.item()

0.05820673704147339

All these tensors have a single element, but an increasing number of dimensions:

torch.rand(1)

tensor([0.0809])

torch.rand(1, 1)

tensor([[0.1210]])

torch.rand(1, 1, 1)

tensor([[[0.7144]]])

torch.rand(1, 1, 1, 1)

tensor([[[[0.2860]]]])

You can tell the number of dimensions of a tensor easily by counting the number of opening square brackets.

torch.rand(1, 1, 1, 1).dim()

4

Tensors can have multiple elements in one dimension:

torch.rand(6)

tensor([0.5643, 0.4410, 0.4270, 0.8506, 0.8635, 0.8864])

torch.rand(6).dim()

1

And multiple elements in multiple dimensions:

torch.rand(2, 3, 4, 5)

tensor([[[[0.0870, 0.3299, 0.4416, 0.9001, 0.8895],
          [0.3249, 0.5699, 0.3589, 0.1634, 0.8901],
          [0.7038, 0.0706, 0.7664, 0.0512, 0.6184],
          [0.8186, 0.1754, 0.6099, 0.6695, 0.9044]],

         [[0.0778, 0.6769, 0.6762, 0.8832, 0.0775],
          [0.5978, 0.6909, 0.5128, 0.8141, 0.6769],
          [0.2519, 0.8887, 0.2718, 0.1048, 0.2765],
          [0.0758, 0.6950, 0.9207, 0.6944, 0.6033]],

         [[0.3973, 0.8712, 0.2578, 0.3369, 0.5643],
          [0.4111, 0.6432, 0.6648, 0.9249, 0.3345],
          [0.0290, 0.0351, 0.6928, 0.9561, 0.6171],
          [0.2387, 0.6789, 0.0634, 0.3986, 0.8932]]],


        [[[0.2385, 0.7671, 0.5207, 0.3065, 0.2753],
          [0.3112, 0.5486, 0.0513, 0.9793, 0.8458],
          [0.0583, 0.0789, 0.0734, 0.4873, 0.8102],
          [0.3111, 0.9091, 0.8887, 0.3369, 0.0989]],

         [[0.8715, 0.7742, 0.3734, 0.4324, 0.6747],
          [0.0925, 0.7034, 0.6770, 0.6588, 0.3616],
          [0.7486, 0.3784, 0.9325, 0.4455, 0.5678],
          [0.0194, 0.1014, 0.8829, 0.3390, 0.5708]],

         [[0.9122, 0.6654, 0.9463, 0.1395, 0.2253],
          [0.8338, 0.7130, 0.0494, 0.2842, 0.3587],
          [0.4219, 0.8805, 0.2924, 0.5384, 0.9167],
          [0.2243, 0.2419, 0.9765, 0.4751, 0.7793]]]])

torch.rand(2, 3, 4, 5).dim()

4

torch.rand(2, 3, 4, 5).numel()

120

torch.ones(2, 4)

tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.]])

t = torch.rand(2, 3)
torch.zeros_like(t)             # Matches the size of t

tensor([[0., 0., 0.],
        [0., 0., 0.]])

torch.ones_like(t)

tensor([[1., 1., 1.],
        [1., 1., 1.]])

torch.randn_like(t)

tensor([[ 0.1467, -1.7656,  1.6756],
        [ 0.2015, -0.5710,  1.1410]])

torch.arange(2, 10, 3)    # From 2 to 10 in increments of 3

tensor([2, 5, 8])

torch.linspace(2, 10, 3)  # 3 elements from 2 to 10 on the linear scale

tensor([ 2.,  6., 10.])

torch.logspace(2, 10, 3)  # Same on the log scale

tensor([1.0000e+02, 1.0000e+06, 1.0000e+10])

torch.randperm(3)

tensor([1, 2, 0])

torch.eye(3)

tensor([[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.]])

Conversion to/from NumPy

PyTorch tensors can be converted to NumPy ndarrays and vice-versa in a very efficient manner as both objects share the same memory.

From PyTorch tensor to NumPy ndarray

t = torch.rand(2, 3)
t

tensor([[0.4104, 0.4666, 0.5470],
        [0.2693, 0.6950, 0.6927]])

t_np = t.numpy()
t_np

array([[0.41038793, 0.46655822, 0.547011  ],
       [0.26930255, 0.69503874, 0.69269085]], dtype=float32)

From NumPy ndarray to PyTorch tensor

import numpy as np
a = np.random.rand(2, 3)
a

array([[0.49319847, 0.38358174, 0.72995909],
       [0.4151989 , 0.79181779, 0.4989544 ]])

a_pt = torch.from_numpy(a)
a_pt

tensor([[0.4932, 0.3836, 0.7300],
        [0.4152, 0.7918, 0.4990]], dtype=torch.float64)

Note the different default data types.

Indexing tensors

t = torch.rand(3, 4)
t

tensor([[0.2268, 0.3211, 0.3199, 0.0580],
        [0.5257, 0.4824, 0.9741, 0.0022],
        [0.3495, 0.0125, 0.8108, 0.0436]])

t[:, 2]

tensor([0.3199, 0.9741, 0.8108])

t[1, :]

tensor([0.5257, 0.4824, 0.9741, 0.0022])

t[2, 3]

tensor(0.0436)

A word of caution about indexing

While indexing elements of a tensor to extract some of the data as a final step of some computation is fine, you should not use indexing to run operations on tensor elements in a loop as this would be extremely inefficient.

Instead, you want to use vectorized operations.

Vectorized operations

Since PyTorch tensors are homogeneous (i.e. made of a single data type), as with NumPy’s ndarrays, operations are vectorized and thus fast.

NumPy is mostly written in C, PyTorch in C++. With either library, when you run vectorized operations on arrays/tensors, you don’t use raw Python (slow) but compiled C/C++ code (much faster).

Here is an excellent post explaining Python vectorization & why it makes such a big difference.

Data types

Default data type

Since PyTorch tensors were built with efficiency in mind for neural networks, the default data type is 32-bit floating points.

This is sufficient for accuracy and much faster than 64-bit floating points.

By contrast, NumPy ndarrays use 64-bit as their default.

t = torch.rand(2, 4)
t.dtype

torch.float32

Setting data type at creation

The type can be set with the dtype argument:

t = torch.rand(2, 4, dtype=torch.float64)
t

tensor([[0.5250, 0.1203, 0.2109, 0.2446],
        [0.7366, 0.1476, 0.5989, 0.0488]], dtype=torch.float64)

Printed tensors display attributes with values ≠ default values.

t.dtype

torch.float64

Changing data type

t = torch.rand(2, 4)
t.dtype

torch.float32

t2 = t.type(torch.float64)
t2.dtype

torch.float64

List of data types

dtype	Description
torch.float16 / torch.half	16-bit / half-precision floating-point
torch.float32 / torch.float	32-bit / single-precision floating-point
torch.float64 / torch.double	64-bit / double-precision floating-point
torch.uint8	unsigned 8-bit integers
torch.int8	signed 8-bit integers
torch.int16 / torch.short	signed 16-bit integers
torch.int32 / torch.int	signed 32-bit integers
torch.int64 / torch.long	signed 64-bit integers
torch.bool	boolean

Simple operations

t1 = torch.tensor([[1, 2], [3, 4]])
t1

tensor([[1, 2],
        [3, 4]])

t2 = torch.tensor([[1, 1], [0, 0]])
t2

tensor([[1, 1],
        [0, 0]])

Operation performed between elements at corresponding locations:

t1 + t2

tensor([[2, 3],
        [3, 4]])

Operation applied to each element of the tensor:

t1 + 1

tensor([[2, 3],
        [4, 5]])

Reduction

t = torch.ones(2, 3, 4);
t

tensor([[[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]],

        [[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]]])

t.sum()   # Reduction over all entries

tensor(24.)

Other reduction functions (e.g. mean) behave the same way.

Reduction over a specific dimension:

t.sum(0)

tensor([[2., 2., 2., 2.],
        [2., 2., 2., 2.],
        [2., 2., 2., 2.]])

t.sum(1)

tensor([[3., 3., 3., 3.],
        [3., 3., 3., 3.]])

t.sum(2)

tensor([[4., 4., 4.],
        [4., 4., 4.]])

Reduction over multiple dimensions:

t.sum((0, 1))

tensor([6., 6., 6., 6.])

t.sum((0, 2))

tensor([8., 8., 8.])

t.sum((1, 2))

tensor([12., 12.])

In-place operations

With operators post-fixed with _:

t1 = torch.tensor([1, 2])
t1

tensor([1, 2])

t2 = torch.tensor([1, 1])
t2

tensor([1, 1])

t1.add_(t2)
t1

tensor([2, 3])

t1.zero_()
t1

tensor([0, 0])

While reassignments will use new addresses in memory, in-place operations will use the same addresses.

Tensor views

t = torch.tensor([[1, 2, 3], [4, 5, 6]]); print(t)
t.size()
t.view(6)
t.view(3, 2)
t.view(3, -1) # Same: with -1, the size is inferred from other dimensions

Note the difference

t1 = torch.tensor([[1, 2, 3], [4, 5, 6]])
t1

tensor([[1, 2, 3],
        [4, 5, 6]])

t2 = t1.t()
t2

tensor([[1, 4],
        [2, 5],
        [3, 6]])

t3 = t1.view(3, 2)
t3

tensor([[1, 2],
        [3, 4],
        [5, 6]])

Logical operations

t1 = torch.randperm(5)
t1

tensor([4, 0, 1, 3, 2])

t2 = torch.randperm(5)
t2

tensor([4, 2, 0, 3, 1])

Test each element:

t1 > 3

tensor([ True, False, False, False, False])

Test corresponding pairs of elements:

t1 < t2

tensor([False,  True, False, False, False])

Device attribute

Tensor data can be placed in the memory of various processor types:

• the RAM of CPU,
• the RAM of a GPU with CUDA support,
• the RAM of a GPU with AMD’s ROCm support,
• the RAM of an XLA device (e.g. Cloud TPU) with the torch_xla package.

The values for the device attributes are:

• CPU:  'cpu',
• GPU (CUDA & AMD’s ROCm):  'cuda',
• XLA:  xm.xla_device().

This last option requires to load the torch_xla package first:

import torch_xla
import torch_xla.core.xla_model as xm

Creating a tensor on a specific device

By default, tensors are created on the CPU.

You can create a tensor on an accelerator by specifying the device attribute (our current training cluster does not have GPUs, so don’t run this on it):

t_gpu = torch.rand(2, device='cuda')

Copying a tensor to a specific device

You can also make copies of a tensor on other devices:

# Make a copy of t on the GPU
t_gpu = t.to(device='cuda')
t_gpu = t.cuda()             # Alternative syntax

# Make a copy of t_gpu on the CPU
t = t_gpu.to(device='cpu')
t = t_gpu.cpu()              # Alternative syntax

Multiple GPUs

If you have multiple GPUs, you can optionally specify which one a tensor should be created on or copied to:

t1 = torch.rand(2, device='cuda:0')  # Create a tensor on 1st GPU
t2 = t1.to(device='cuda:0')          # Make a copy of t1 on 1st GPU
t3 = t1.to(device='cuda:1')          # Make a copy of t1 on 2nd GPU

Or the equivalent short forms:

t2 = t1.cuda(0)
t3 = t1.cuda(1)