what do you mean by "4*4 byte binary matrix"?
a 4 x 4 matrix of integers in [-128, 127] where "+" is "+ mod 256" and "*" is "* mod 256"? This might not have an inverse.
a 4 x 4 matrix of integers in [0, 255] where "+" is "+ mod 256" and "*" is "* mod 256"? This might not have an inverse.
This is because some elements do not have inverses themselves. for example 1 = (2 * x) mod 256 has no solution if x must be an integer.
a 32 x 32 matrix of {0,1} where "+" is "xor" and "*" is "and"? This could have an inverse.
a 4 x 4 matrix of elements of a gf256? This could have an inverse.
These don't have numeric interpretations.