# We Learned the Wrong Way to Matrix Multiply

October 07, 2020
Mark Liu

I took an informal poll of my friends and it turns out we all intially learned a method of matrix multiplication in school which I call row oriented multiplication. However, the friends who had been working with linear algebra for a long time frequently used a more intuitive method which I call column oriented multiplication. Here I’ll explain the difference and try to show why the column oriented point of view should be a useful tool in your toolbox, especially if you are working in a field that is heavy in linear algebra (e.g. robotics).

## Row Oriented Multiplication

When I learned how to multiply vectors and matrices, it went something like this.

$\begin{bmatrix} & & \\ \rightarrow & \rightarrow & \rightarrow \\ & & \end{bmatrix} \begin{bmatrix} \downarrow \\ \downarrow \\ \downarrow \end{bmatrix} =\begin{bmatrix} \\ \cdot \\ \\ \end{bmatrix}$

Or if you prefer formulas, it went like this.
$A x = \begin{bmatrix} ... \\ A_{i1} x_1 + A_{i2} x_2 + ... + A_{i n} x_n \\ ... \end{bmatrix}$

I call this row oriented multiplication. It definitely works but it often hides the geometrical content of a matrix behind nasty algebra. Consider this $3D$ rotation matrix.

$R = \begin{bmatrix} \frac{\sqrt 3}2 & -\frac 1 2 & 0 \\ \frac 1 2 & \frac{\sqrt 3}2 & 0 \\ 0 & 0 & 1\end{bmatrix}$

We can verify that the axis of rotation is the z-axis with row oriented multiplication by seeing that a point on the z axis $\begin{bmatrix}0 & 0 & z\end{bmatrix}^T$ is unaffected by multiplication with $R$.

$R \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = \begin{bmatrix} \frac{\sqrt 3}2 & -\frac 1 2 & 0 \\ \frac 1 2 & \frac{\sqrt 3}2 & 0 \\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = \begin{bmatrix} \frac{\sqrt 3}2\cdot 0 + -\frac 1 2\cdot 0 + 0 \cdot z \\ \frac 1 2 \cdot 0 + \frac{\sqrt 3}2\cdot 0 + 0 \cdot z \\ 0 \cdot 0 + 0\cdot 0 + 1\cdot z\end{bmatrix} = \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix}$

Now I’ll introduce column oriented multiplication and how it not only simplifies the previous calculation but also reveals the geometrical content of the matrix.

## Column Oriented Multiplication

First let’s establish the notation that for any matrix $A$, $A_i$ means the $i^{th}$ column of $A$. We obviously have $A = \begin{bmatrix} A_1 | A_2 | ... | A_n\end{bmatrix}$.

Column oriented multiplication of $A x$ works like this. We use the columns of $A$ as vectors.

$A x = \begin{bmatrix} A_1 | A_2 | ... | A_n\end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = x_1 A_1 + x_2 A_2 + ... + x_n A_n$

There are multiple ways to verify that this leads to the same result as row oriented multiplication. I’ll show a really easy way in the appendix. But for now let’s see how column oriented multiplication can be applied.

## Using Column Oriented Multiplication

First let’s redo our previous calculation involving our rotation matrix $R$ using column oriented multiplication.

$R \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = 0 \cdot R_1 + 0 \cdot R_2 + z R_3 = z R_3$

We can completely ignore the first two columns of $R$, and compute the answer simply by reading off the third column of $R$ and multiplying by $z$. We arrive at the same answer we got with row oriented multiplication.

$z R_3 = \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix}$

Now recall the notation for the standard basis vectors.

$e_1 = \begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix}, \,\,\,\, e_2 = \begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}, \,\,\,\, e_3 = \begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix}$

Visualize $e_1, e_2, e_3$ as points on the x, y, and z axis respectively. Using column oriented multiplication we can quickly see these relationships.

$R e_1 = R_1, \,\,\,\, \ R e_2 = R_2, \,\,\,\, R e_3= R_3$

This means that the geometrical content of $R$ — the way it acts on the coordinate axes — can be entirely read off from its columns!

$e_1 \rightarrow \begin{bmatrix} \frac {\sqrt 3} 2 \\ \frac 1 2 \\ 0\end{bmatrix}, \,\,\,\, \ e_2 \rightarrow \begin{bmatrix} -\frac 1 2 \\ \frac{\sqrt 3} 2 \\ 0\end{bmatrix}, e_3 \rightarrow e_3$ (read the arrows as “rotates to”)

Try carrying out this rotation on your thumb as $e_1$, your index finger as $e_2$, and your middle finger as $e_3$. Also note that $e_2$ is redundant since it must be perpendicular to $e_1$ and $e_3$.

The story doesn’t end here. Once I got into the habit of seeing matrix multiplication this way, I started to have insights into the nature of other types of matrices, including Jacobian matrices which are ubiquitous in robotics algorithms. But I’ll save that for another post.

## Appendix

In all matrix multiply expressions that follow, assume that we are using row oriented multiplication.

$A x = A \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = A (\sum_{i=1}^n x_i e_i)$

$= A (\sum_{i=1}^n x_i e_i)$
$= \sum_{i=1}^n x_i A e_i$
$= \sum_{i=1}^n x_i A_i$ (verify, with row oriented multiplication that $A e_i = A_i$)

So we end up with the formula for column oriented multiplication $Ax = \sum_{i=1}^n x_i A_i$.