Linear Algebra Cheat Sheet for Machine Learning

Familiarity with Linear Algebra helps quite a bit while working on Machine Learning algorithms. As Software Programmers, we don’t get to do lot of serious Mathematics on regular basis tend to forget the principles. I would like to list some of the concepts here.

Matrices & Vectors

Starting with basics, A Matrix is a rectangular array of elements. Typical example is:

A Matrix is denoted by Number of Rows * Number of Columns. The above Matrix is normally called a 3 * 2 Matrix.
Similarly Vector is another often used Entity in Machine Learning algorithms. It’s special type of Matrix in the sense it always has only one column. It also called a n * 1 Matrix. A typical example of Vector would be:


Entries of Matrix or Vectors are denoted by A_{ij} where i is number of row and j is number of column. So in examples above, A_{32} (Matrix) will be 4 and a_{31} (Vector) will be 5.

Matrix Addition

We can perform addition operations over 2 matrices. In this, we add element at position i * j in Matrix M1 with element at the same position in Matrix M2. Let’s take an example :

We can perform subtraction in similar manner. One point to mention here is that we can perform these operations only on matrices of the same dimensions.

Scalar Multiplication & Division

We can also perform multiplication and divisions of Matrices and Vectors with real numbers as well. As you might have guessed, we take every number in the matrix and perform the corresponding operation with the provided number. Again an example will help:

Addition and subtraction can also be performed similarly.

Matrix-Vector Multiplication

Things get a bit interesting here. Firstly we can only perform multiplications between a Matrix and Vector when number of columns of Matrix equals number of rows of Vector. In mathematical terms m * n-dimensional Matrix can be multiplied with only n-dimensional Vector. The result of any Matrix – Vector multiplication is always a Vector. Let’s take help of an example to see how this is done.

As you can see, we are multiplying a 3 * 2 matrix with a 2 dimensional Vector and we get a 3 dimensional Vector in result.

Matrix-Matrix Multiplication

A Matrix-Matrix multiplication can be considered as an extended case of Matrix-Vector multiplication:

Above two multiplications can be done as explained in Matrix-Vector multiplications:

The final result would be :

As you can see we can only multiply a m * n matrix with n * p i.e. number of columns on 1st matrix should be equal to number of rows of 2nd matrix. The resulting matrix is of m * p dimensions.

Couple of points to keep in mind related to Matrix – Matrix multiplications. Contrary to real numbers where A * B = B * A, Matrix multiplication is not commutative i.e. M_{1} * M_{2} \neq M_{2} * M_{1}

Identity Matrix

An Identity Matrix is a square Matrix i.e. m * m matrix where there are 1s on the diagonal from top left to bottom right and 0s else where. An Identity Matrix is generally denoted by I or I_{n*n}. Couple of examples of Identity Matrices are:

All these operations are quite widely used in developing algorithms for Linear Regression and Gradient Descent. There are a lot of libraries in various popular programming languages which do these things for us but it helps to know the details for a better understanding.

There are a couple of other interesting operations like Transpose of a Matrix and Inverse of Matrix but I would leave them for you to dig further 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s