7

Using LU Decomposition how can I solve for vector $x$ in the system $Ax = b$, given $A$ and $b$. For simplicities sake where $A$ is a 3x3 matrix and $b$ is a vector of size 3.

For example how to find x when: $$A= \begin{pmatrix} 3 & 1 & 2\\ 5 & 7 & 5\\ 1 & 2 & 3\\ \end{pmatrix}, b = \begin{pmatrix} 4 \\ 4 \\ 4 \\\end{pmatrix}$$

John
  • 231

2 Answers2

15

Suppose that you have an LU - Decomposition of matrix A: $$ A=LU, $$ Where U is upper-triangle and L is lower-triangle. Then the original system is: $$ LUx=b. $$ Let us break the task into two parts: first, we find $y$ such that $$ Ly=b. $$ Then,find x such that $$ Ux=y. $$ It is obvious by multiplying the last equation by L from the left that such $x$ will be the solution to the original problem.

Now, what makes LU - decomposition useful is that both sub-tasks can be exactly solved in one pass! (That is, the complexity is $O(n^2)$, where n is the dimensionality of $x$). This is because of the special triangle form of both L and U.

When solving the first system of equations for $y$, you can start from finding $y_1$ - this would be elementary because the first equation will be of the form $L_{11}y_1=b_1$. Then plug this value into the second equation, find $y_2$, and so on. Note that this is equivalent to simple Gaussian elimination when you don't have to do the first (actually, the hardest part) of converting the matrix to the triangular form - it is already triangular. Now, performing the same process for $x$ (but this time you start with $x_n$ and then proceed upwards), you would have the solution in $O(n^2)$).

In practice, though, this trick would be useful only for large matrices (definitely not 3x3) when implementing a numerical solver, because the hardest part of the process is actually finding the LU - decomposition. This procedure is not linear in complexity (refer to the Wikipedia page for some estimates), but this is still easier than $O(n^3)$ for simple Gaussian elimination.

Of course, when solving 3x3 matrices by hand, you better stick with simple Gaussian elimination ;)

Vossler
  • 1,080
  • How would you find L and U? does one get the diagonal and the other doesn't? – John Feb 24 '15 at 10:22
  • @littleO Of course, I am so sorry; I was actually going to write "in a single pass" but wrote "linear" instead. I would edit it. – Vossler Feb 24 '15 at 11:27
  • 1
    @littleO As for the second part, actually, you are wrong: fast matrix multiplication methods exist (see, for example, http://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm). It is still larger than $O(n^2)$, but definitely not O(n^3), so it is, after all, better than Gaussian elimination. – Vossler Feb 24 '15 at 11:30
  • @Boris Ah, you're right, I didn't realize that. Very good point. I'll delete my original comment, thanks for pointing out there are LU factorization algorithms with complexity better than $O(n^3)$. – littleO Feb 24 '15 at 11:39
  • 1
    @John Perhaps the easiest method for doing that is the so-called "Doolittle algorithm" which is equivalent to the numerically stable version of Gaussian elimination; I think it would be easier to Google it. You would not get the time advantage over Gaussian elimination by using it but for your purposes it would be more than enough. – Vossler Feb 24 '15 at 11:44
4

If $A=LU$, then $Ax=LUx=b$. Thus, first solve $Ly=b$, for $y$, then solve $Ux=y$ for x. Both solutions are found by back-substitution.

Tommy L
  • 329
  • 2
  • 7