GMSK Tx is very simple. The first stage is conversion of binary digits to bipolar levels and then apply the Gaussian filter of suitable BT (=0.3 for GSM) and then integrate and find cos and sin yielding I and Q components of GMSK respectively. This is the complete Tx! You may use gaussfir (), filter(), cos() and sin() functions in MATLAB for simulating GMSK.