The way I justify it is that, what row reduction normally leads into is solving systems of linear equations. Since our rows are representing the coefficients of variables in a linear equation, if we swap two rows, we are not changing the system of equations. e.g.

would correspond to the same system of equations as the matrix

, that is by swapping rows, we haven't changed the solution set.
As to finding the inverse matrix, we try to make the left into the identity matrix, leaving the right as the inverse matrix.

Firstly we can swap rows 1 and 3, so that we get a nice 1 in our 1,1 entry.
 & \left[\begin{array}{ccc|}<br />1 & 0 & -2<br />\\ 3 & -1 & 1<br />\\ 2 & 1 & -1<br />\end{array}\begin{array}{ccc}<br />0 & 0 & -1<br />\\ 0 & 1 & 0<br />\\ 1 & 0 & 0<br />\end{array}\right]<br />\end{alignedat}<br /> )
Now we want to form a pivot in the 2,2 entry, so to do this we need to clear the 2,1 entry out, so we can take a multiple of row 1 from row 2 to do this. We can also clear out the 3,1 entry by taking away a mutiple of row 1 from row 3.

Next we need to clear out the 3,2 entry to make the lower triangle of the matrix be all zeros. We can clear out this entry by taking away a mutiple of row 2 from row 3.
<br />\\ r_{3}\rightarrow\frac{1}{10}\left(r_{3}\right)\vphantom{\int_{a}^{b}}<br />\end{array} & \left[\begin{array}{ccc|}<br />1 & 0 & -2<br />\\ 0 & 1 & -7<br />\\ 0 & 0 & 1<br />\end{array}\begin{array}{ccc}<br />0 & 0 & -1<br />\\ 0 & -1 & -3<br />\\ \frac{1}{10}\vphantom{\int_{a}^{b}} & \frac{1}{10} & \frac{1}{2}<br />\end{array}\right]<br />\end{alignedat}<br /> )
Next we need to start working back upwards, to clear out the top right triangle, forming the identity matrix. So we first start by working on the entry 2,3. If we take a mutiple of row one away, we will affect our zeros we formed, but if we are to take a mutiple of row 3 away from row 2, then we will preserve these zeros, and be able to form another zero.

Now we work on the 1,3 entry. We can add a mutiple of row three away from row one without affecting our other zeros.

Now normally we wouldn't be finished yet, but since our 1,2 entry is what we want it to be, we have formed the identity matrix, so the matrix on the right is our inverse matrix. If we did need to work on the 1,2 entry, then we could take a mutiple of row 2 away from row 1 without affecting the other zeros.

Wow, that turned into a long post, anyways, hope that makes sense and hope it helps
