A is a 3x3 matrix, b is a 3x1 vector. I try to convert [A|b], a 3x4 matrix, to [I|0]. So the formula is right multiple [A|b] by a matrix:
$$ \begin{bmatrix} A^{-1} & -A^{-1}b\ 0 & 1\ \end{bmatrix} $$
Here's the code in Mathematica 10
A=Table[RandomReal[],{i,3},{j,3}]
b=Table[RandomReal[],{i,3}]
M=Transpose[Join[A, {b}]]
ret=M.Join[Transpose[Join[Inverse[A], {-Inverse[A].b}]], {{0, 0, 0, 1}}]
The last column of ret is always non-zero. The ret will often look like for example
$$ \begin{bmatrix} 1. & 0. & 0. & -0.979237\ 0. & 1. & 4.44089*10^{-16} & 2.17005\ -8.8817810^{-16} & -5.5511210^{-17} & 1. & -1.79119 \end{bmatrix} $$
How can I make Mathematica do it right?
