Before you turn this problem in, make sure everything runs as expected. First, restart the kernel (in the menubar, select Kernel$\rightarrow$Restart) and then run all cells (in the menubar, select Cell$\rightarrow$Run All).

Make sure you fill in any place that says YOUR CODE HERE or “YOUR ANSWER HERE”, as well as your name and collaborators below:

Numpy Tutorial¶

Ref: https://numpy.org/doc/stable/user/absolute_beginners.html

This tutorial has been adapted from Numpy tutorial to Pytorch tensors.

If you are running this in Google Colab. Try running it with Runtime > “Change Runtime Type” > “”

Python is good at managing code that runs single instruction on multiple data (SIMD)

You might occasionally hear an array referred to as a “ndarray,” which is shorthand for “N-dimensional array.” An N-dimensional array is simply an array with any number of dimensions. You might also hear 1-D, or one-dimensional array, 2-D, or two-dimensional array, and so on. The NumPy ndarray class is used to represent both matrices and vectors. A vector is an array with a single dimension (there’s no difference between row and column vectors), while a matrix refers to an array with two dimensions. For 3-D or higher dimensional arrays, the term tensor is also commonly used.

We can access the elements in the array using square brackets. When you’re accessing elements, remember that indexing in NumPy starts at 0. That means that if you want to access the first element in your array, you’ll be accessing element “0”.

If you start with these arrays:

You can concatenate them with t.cat().

Or, if you start with these arrays:

You can concatenate them with:

Can you reshape an array?¶

This section covers arr.reshape()

Yes!

Using arr.reshape() will give a new shape to an array without changing the data. Just remember that when you use the reshape method, the array you want to produce needs to have the same number of elements as the original array. If you start with an array with 12 elements, you’ll need to make sure that your new array also has a total of 12 elements.

If you start with this array:

You can use reshape() to reshape your array. For example, you can reshape this array to an array with three rows and two columns:

How to convert a 1D array into a 2D array (how to add a new axis to an array)¶

This section covers None

You can use None to increase the dimensions of your existing array.

Using None will increase the dimensions of your array by one dimension when used once. This means that a 1D array will become a 2D array, a 2D array will become a 3D array, and so on.

For example, if you start with this array:

You can use None to add a new axis:

You can explicitly convert a 1D array with either a row vector or a column vector using None. For example, you can convert a 1D array to a row vector by inserting an axis along the first dimension:

Or, for a column vector, you can insert an axis along the second dimension:

You can do the inverse operation using .squeeze() method which removes the dimensions of size 1. np.squeeze takes the axis that need to be squeezed as an arugment.

You can also provide a tuple of axis as argument to squeeze.

If you want to squeeze all the axis, you can call squeeze without an argument, although explicit is better than implicit.

Find more information about newaxis here

Indexing and slicing¶

You can index and slice NumPy arrays in the same ways you can slice Python lists.

You can visualize it this way:

You may want to take a section of your array or specific array elements to use in further analysis or additional operations. To do that, you’ll need to subset, slice, and/or index your arrays.

If you want to select values from your array that fulfill certain conditions, it’s straightforward with NumPy.

For example, if you start with this array:

You can easily print all of the values in the array that are less than 5.

You can also select, for example, numbers that are equal to or greater than 5, and use that condition to index an array.

You can select elements that are divisible by 2:

Or you can select elements that satisfy two conditions using the & and | operators:

You can also make use of the logical operators & and | in order to return boolean values that specify whether or not the values in an array fulfill a certain condition. This can be useful with arrays that contain names or other categorical values.

You can also use np.nonzero() to select elements or indices from an array.

Starting with this array:

You can use np.nonzero() to print the indices of elements that are, for example, less than 5:

In this example, an array was returned: one for each dimension. The first array represents the row indices where these values are found, and the second array represents the column indices where the values are found.

If you want to generate a list of coordinates where the elements exist, you can zip the arrays, iterate over the list of coordinates, and print them. For example:

You can also use np.nonzero() to print the elements in an array that are less than 5 with:

If the element you’re looking for doesn’t exist in the array, then the returned array of indices will be empty. For example:

Using Ellipsis or ...¶

Ellipsis or ... expands to the number of : objects needed for the selection tuple to index all dimensions. In most cases, this means that the length of the expanded selection tuple is x.ndim. There may only be a single ellipsis present.

This is equivalent to:

Each newaxis object in the selection tuple serves to expand the dimensions of the resulting selection by one unit-length dimension. The added dimension is the position of the newaxis object in the selection tuple. newaxis is an alias for None, and None can be used in place of this with the same result. From the above example:

This can be handy to combine two arrays in a way that otherwise would require explicit reshaping operations. For example:

Learn more about indexing and slicing here and here.

Read more about using the nonzero function at: nonzero.

How to create an array from existing data¶

This section covers slicing and indexing, np.vstack(), np.hstack(), np.hsplit(), .view(), copy()

You can easily create a new array from a section of an existing array.

Let’s say you have this array:

You can create a new array from a section of your array any time by specifying where you want to slice your array.

Here, you grabbed a section of your array from index position 3 through index position 8.

You can also stack two existing arrays, both vertically and horizontally. Let’s say you have two arrays, a1 and a2:

You can stack them vertically with vstack:

Or stack them horizontally with hstack:

You can split an array into several smaller arrays using hsplit. You can specify either the number of equally shaped arrays to return or the columns after which the division should occur.

Let’s say you have this array:

If you wanted to split this array into three equally shaped arrays, you would run:

If you wanted to split your array after the third and fourth column, you’d run:

Learn more about stacking and splitting arrays here.

You can use the view method to create a new array object that looks at the same data as the original array (a shallow copy).

Views are an important NumPy concept! NumPy functions, as well as operations like indexing and slicing, will return views whenever possible. This saves memory and is faster (no copy of the data has to be made). However it’s important to be aware of this - modifying data in a view also modifies the original array!

Let’s say you create this array:

Now we create an array b1 by slicing a and modify the first element of b1. This will modify the corresponding element in a as well!

Using the copy method will make a complete copy of the array and its data (a deep copy). To use this on your array, you could run:

Learn more about copies and views here.

Basic array operations¶

This section covers addition, subtraction, multiplication, division, and more

Once you’ve created your arrays, you can start to work with them. Let’s say, for example, that you’ve created two arrays, one called “data” and one called “ones”

You can add the arrays together with the plus sign.

You can, of course, do more than just addition!

Basic operations are simple with NumPy. If you want to find the sum of the elements in an array, you’d use sum(). This works for 1D arrays, 2D arrays, and arrays in higher dimensions.

To add the rows or the columns in a 2D array, you would specify the axis.

If you start with this array:

You can sum over the axis of rows with:

You can sum over the axis of columns with:

Learn more about basic operations here.

Broadcasting¶

There are times when you might want to carry out an operation between an array and a single number (also called an operation between a vector and a scalar) or between arrays of two different sizes. For example, your array (we’ll call it “data”) might contain information about distance in miles but you want to convert the information to kilometers. You can perform this operation with:

NumPy understands that the multiplication should happen with each cell. That concept is called broadcasting. Broadcasting is a mechanism that allows NumPy to perform operations on arrays of different shapes. The dimensions of your array must be compatible, for example, when the dimensions of both arrays are equal or when one of them is 1. If the dimensions are not compatible, you will get a ValueError.

Learn more about broadcasting here.

More useful array operations¶

This section covers maximum, minimum, sum, mean, product, standard deviation, and more

NumPy also performs aggregation functions. In addition to min, max, and sum, you can easily run mean to get the average, prod to get the result of multiplying the elements together, std to get the standard deviation, and more.

Let’s start with this array, called “a”

It’s very common to want to aggregate along a row or column. By default, every NumPy aggregation function will return the aggregate of the entire array. To find the sum or the minimum of the elements in your array, run:

Or:

You can specify on which axis you want the aggregation function to be computed. For example, you can find the minimum value within each column by specifying axis=0.

You can pass Python lists of lists to create a 2-D array (or “matrix”) to represent them in NumPy.

Indexing and slicing operations are useful when you’re manipulating matrices:

You can aggregate matrices the same way you aggregated vectors:

You can aggregate all the values in a matrix and you can aggregate them across columns or rows using the axis parameter. To illustrate this point, let’s look at a slightly modified dataset:

Once you’ve created your matrices, you can add and multiply them using arithmetic operators if you have two matrices that are the same size.

You can do these arithmetic operations on matrices of different sizes, but only if one matrix has only one column or one row. In this case, NumPy will use its broadcast rules for the operation.

Be aware that when NumPy prints N-dimensional arrays, the last axis is looped over the fastest while the first axis is the slowest. For instance:

There are often instances where we want NumPy to initialize the values of an array. NumPy offers functions like ones() and zeros(), and the random.Generator class for random number generation for that. All you need to do is pass in the number of elements you want it to generate:

You can also use ones(), zeros(), and random() to create a 2D array if you give them a tuple describing the dimensions of the matrix:

Read more about creating arrays, filled with 0’s, 1’s, other values or uninitialized, at array creation routines.

Generating random numbers¶

The use of random number generation is an important part of the configuration and evaluation of many numerical and machine learning algorithms. Whether you need to randomly initialize weights in an artificial neural network, split data into random sets, or randomly shuffle your dataset, being able to generate random numbers (actually, repeatable pseudo-random numbers) is essential.

With Generator.integers, you can generate random integers from low (remember that this is inclusive with NumPy) to high (exclusive). You can set endpoint=True to make the high number inclusive.

You can generate a 2 x 4 array of random integers between 0 and 4 with:

Read more about random number generation here.

Transposing and reshaping a matrix¶

This section covers arr.reshape(), arr.transpose(), arr.T

It’s common to need to transpose your matrices. NumPy arrays have the property T that allows you to transpose a matrix.

You may also need to switch the dimensions of a matrix. This can happen when, for example, you have a model that expects a certain input shape that is different from your dataset. This is where the reshape method can be useful. You simply need to pass in the new dimensions that you want for the matrix.

You can also use .transpose() to reverse or change the axes of an array according to the values you specify.

If you start with this array:

You can transpose your array with arr.transpose().

You can also use arr.T:

Sometimes you want to represent multiple matrices in a single array. Let’s say we have two matrices m1 and m2 concatenated in an array as ms

To take the transpose of each matrix, m1 and m2, you can use swapaxes or transpose

To learn more about transposing and reshaping arrays, see transpose, swapaxes and reshape.

Reshaping and flattening multidimensional arrays¶

You can use flatten to flatten your array into a 1D array.

Working with mathematical formulas¶

The ease of implementing mathematical formulas that work on arrays is one of the things that make NumPy so widely used in the scientific Python community.

For example, this is the mean square error formula (a central formula used in supervised machine learning models that deal with regression):

Implementing this formula is simple and straightforward in NumPy:

What makes this work so well is that predictions and labels can contain one or a thousand values. They only need to be the same size.

You can visualize it this way:

In this example, both the predictions and labels vectors contain three values, meaning n has a value of three. After we carry out subtractions the values in the vector are squared. Then NumPy sums the values, and your result is the error value for that prediction and a score for the quality of the model.

Common mathematical operations¶

1. Scalar-scalar multiplication

   $$f(a, b) = a b$$

   (1)

Numpy allows us to multiply many scalars at the same time.

2. Element-wise multiplication of matrices

Sclar-scalar multiplication in numpy is same as element-wise multiplication of matrices, also known as Hadamard product

2. Scalar-vector multiplication \newcommand{\bfv}{\mathbf{v}}

   $$f(\alpha, \bfv) = \alpha \bfv$$

   (4)

Numpy allows us to multiply a batch of scalars with a batch of vectors at the same time through broadcasting. Let’s say you want to multiply three scalars $\alpha_1 = 1, \alpha_2 = 2, \alpha_3 = 3$ with three vectors $\bfv_1 = [2; 3], \bfv_2 = [5; 7], \bfv_3 = [9; 11]$, respectively. You can combine them in single arrays $\alpha_s = [\alpha_1; \alpha_2; \alpha_3]$ and a matrix of vectors $\bfv_s = \begin{bmatrix} \bfv_1 \\ \bfv_2 \\ \bfv_3 \end{bmatrix}$

If we simply multiply alphas with vs, numpy will throw an error:

We need to carefully choose the axes along which we want to multiply alphas with vs. Making alpha.shape = (3, 1) will do the trick.

3. Vector-vector dot product \newcommand{\bfa}{\mathbf{a}} \newcommand{\bfb}{\mathbf{b}}

   $$f(\bfa, \bfb) = \bfa^\top \bfb$$

   (5)

Note that you do not need a.T because a is 1-D array and taking its transpose will not do anything.

Look at the rules for np.matmul which implements the @ operator.

1. If both arguments are 2-D they are multiplied like conventional matrices.

2. If either argument is N-D, N > 2, it is treated as a stack of matrices residing in the last two indexes and broadcast accordingly.

3. If the first argument is 1-D, it is promoted to a matrix by prepending a 1 to its dimensions. After matrix multiplication the prepended 1 is removed.

4. If the second argument is 1-D, it is promoted to a matrix by appending a 1 to its dimensions. After matrix multiplication the appended 1 is removed.

In this case rules 3 and 4 are in effect.

How can you multiply many vectors together. Let’s try to multiply 3 pairs of vectors ($\bfa_1, \bfa_2, \bfa_3$) and (($\bfb_1, \bfb_2, \bfb_3$) using numpy.

If we simply use @ between _as and bs, then we will get an error.

We can tranpose _as, but that will give us a (2,2) array while we only expect 3 scalars (not 4) each one from corresponding vector multiplication between $\bfa_i^\top \bfb_i$. There are multiple ways to accomplish this.

Approach 1: Make the last two dimensions compatible for matrix multiplication, then use @ with rule 2 of np.matmul., Use np.squeeze to remove the unnecessary dimensions with size 1.

Approach 2: Take element-wise product and sum over the last axis (which by definition is the vector-vector dot product).

4. Matrix-vector product

   $$f(A, \bfb) = A\bfb$$

   (6)

   Following the vector-vector product example, multiple matrix-vector pairs can be multiplied using the same tricks.

Approach 1: Make them compatiable with rule 2 of np.matmul

Approach 2: Use element-wise multiplication followed by sum along the right axis.

5. Matrix-matrix product

This is straightforward due to the rule 2 of np.matmul

6. Norm/magnitude of a vector

   $$f(\bfv) = \|\bfv\|$$

   (8)

Norm of many vectors

7. Inverse of a matrix

Inverse of many matrices

How to save and load Tensor objects¶

This section covers t.save, t.load,

You will, at some point, want to save your arrays to disk and load them back without having to re-run the code. Fortunately, there are several ways to save and load objects with torch. The ndarray objects can be saved to and loaded from the disk files with load and save functions.

The .pt files store data, shape, dtype, and other information required to reconstruct the ndarray in a way that allows the array to be correctly retrieved, even when the file is on another machine with different architecture.

It’s easy to save and load and array with t.save(). Just make sure to specify the array you want to save and a file name. For example, if you create this array:

You can save it as “filename.pt” with:

You can use np.load() to reconstruct your array.

If you want to check your array, you can run: