If m1 and m2 are your matrices try
Expand[m1.m2] //. a_. x^(m_ /; m >= 3) -> 0
Thanks for the advice, Hans, it worked for me.
