We Learned the Wrong Way to Matrix Multiply

October 07, 2020
Mark Liu

I took an informal poll of my friends and it turns out we all intially learned a method of matrix multiplication in school which I call row oriented multiplication. However, the friends who had been working with linear algebra for a long time frequently used a more intuitive method which I call column oriented multiplication. Here I’ll explain the difference and try to show why the column oriented point of view should be a useful tool in your toolbox, especially if you are working in a field that is heavy in linear algebra (e.g. robotics).

Row Oriented Multiplication

When I learned how to multiply vectors and matrices, it went something like this.

[][]=[]\begin{bmatrix} & & \\ \rightarrow & \rightarrow & \rightarrow \\ & & \end{bmatrix} \begin{bmatrix} \downarrow \\ \downarrow \\ \downarrow \end{bmatrix} =\begin{bmatrix} \\ \cdot \\ \\ \end{bmatrix}

Or if you prefer formulas, it went like this.
Ax=[...Ai1x1+Ai2x2+...+Ainxn...]A x = \begin{bmatrix} ... \\ A_{i1} x_1 + A_{i2} x_2 + ... + A_{i n} x_n \\ ... \end{bmatrix}

I call this row oriented multiplication. It definitely works but it often hides the geometrical content of a matrix behind nasty algebra. Consider this 3D3D rotation matrix.

R=[3212012320001]R = \begin{bmatrix} \frac{\sqrt 3}2 & -\frac 1 2 & 0 \\ \frac 1 2 & \frac{\sqrt 3}2 & 0 \\ 0 & 0 & 1\end{bmatrix}

We can verify that the axis of rotation is the z-axis with row oriented multiplication by seeing that a point on the z axis [00z]T\begin{bmatrix}0 & 0 & z\end{bmatrix}^T is unaffected by multiplication with RR.

R[00z]=[3212012320001][00z]=[320+120+0z120+320+0z00+00+1z]=[00z]R \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = \begin{bmatrix} \frac{\sqrt 3}2 & -\frac 1 2 & 0 \\ \frac 1 2 & \frac{\sqrt 3}2 & 0 \\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = \begin{bmatrix} \frac{\sqrt 3}2\cdot 0 + -\frac 1 2\cdot 0 + 0 \cdot z \\ \frac 1 2 \cdot 0 + \frac{\sqrt 3}2\cdot 0 + 0 \cdot z \\ 0 \cdot 0 + 0\cdot 0 + 1\cdot z\end{bmatrix} = \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix}

Now I’ll introduce column oriented multiplication and how it not only simplifies the previous calculation but also reveals the geometrical content of the matrix.

Column Oriented Multiplication

First let’s establish the notation that for any matrix AA, AiA_i means the ithi^{th} column of AA. We obviously have A=[A1A2...An]A = \begin{bmatrix} A_1 | A_2 | ... | A_n\end{bmatrix}.

Column oriented multiplication of AxA x works like this. We use the columns of AA as vectors.

Ax=[A1A2...An][x1x2...xn]=x1A1+x2A2+...+xnAnA x = \begin{bmatrix} A_1 | A_2 | ... | A_n\end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = x_1 A_1 + x_2 A_2 + ... + x_n A_n

There are multiple ways to verify that this leads to the same result as row oriented multiplication. I’ll show a really easy way in the appendix. But for now let’s see how column oriented multiplication can be applied.

Using Column Oriented Multiplication

First let’s redo our previous calculation involving our rotation matrix RR using column oriented multiplication.

R[00z]=0R1+0R2+zR3=zR3R \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = 0 \cdot R_1 + 0 \cdot R_2 + z R_3 = z R_3

We can completely ignore the first two columns of RR, and compute the answer simply by reading off the third column of RR and multiplying by zz. We arrive at the same answer we got with row oriented multiplication.

zR3=[00z]z R_3 = \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix}

Now recall the notation for the standard basis vectors.

e1=[100],e2=[010],e3=[001]e_1 = \begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix}, \,\,\,\, e_2 = \begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}, \,\,\,\, e_3 = \begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix}

Visualize e1,e2,e3e_1, e_2, e_3 as points on the x, y, and z axis respectively. Using column oriented multiplication we can quickly see these relationships.

Re1=R1, Re2=R2,Re3=R3R e_1 = R_1, \,\,\,\, \ R e_2 = R_2, \,\,\,\, R e_3= R_3

This means that the geometrical content of RR — the way it acts on the coordinate axes — can be entirely read off from its columns!

e1[32120], e2[12320],e3e3e_1 \rightarrow \begin{bmatrix} \frac {\sqrt 3} 2 \\ \frac 1 2 \\ 0\end{bmatrix}, \,\,\,\, \ e_2 \rightarrow \begin{bmatrix} -\frac 1 2 \\ \frac{\sqrt 3} 2 \\ 0\end{bmatrix}, e_3 \rightarrow e_3 (read the arrows as “rotates to”)

Try carrying out this rotation on your thumb as e1e_1, your index finger as e2e_2, and your middle finger as e3e_3. Also note that e2e_2 is redundant since it must be perpendicular to e1e_1 and e3e_3. diagram of hand rotation

The story doesn’t end here. Once I got into the habit of seeing matrix multiplication this way, I started to have insights into the nature of other types of matrices, including Jacobian matrices which are ubiquitous in robotics algorithms. But I’ll save that for another post.

Appendix

In all matrix multiply expressions that follow, assume that we are using row oriented multiplication.

Ax=A[x1x2...xn]=A(i=1nxiei)A x = A \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = A (\sum_{i=1}^n x_i e_i)

=A(i=1nxiei)= A (\sum_{i=1}^n x_i e_i)
=i=1nxiAei= \sum_{i=1}^n x_i A e_i
=i=1nxiAi= \sum_{i=1}^n x_i A_i (verify, with row oriented multiplication that Aei=AiA e_i = A_i)

So we end up with the formula for column oriented multiplication Ax=i=1nxiAiAx = \sum_{i=1}^n x_i A_i.


© 2020 Biro Inc.