A matrix is a short hand way of summarizing many equations.
Each port has a power coming out of it composed of contributions from reflected power put into the port and transmitted power from power put into the other ports.
Once the matrix is used to summarize all of these simultaneous equations, mathematical operations can be done on the set of equations by doing matrix related operations on the matrix.