it doesn't makes sense that you measured "Smith Chart", Smith chart is only another representation of your S11 parameter data so u can plot S11 vs freq in a normal graph or you can plot the same on smith chart only representation is different, data i.e impedance at particular frequency will remain the same.
could you be more elaborate how did you actually plotted S11 in Matlab???
Smith chart on a NA shows the input impedance w.r.t frequency. If you want to plot it on matlab, first find Zin from S11 (it will be an array of complex numbers). Then use plotsmithchart function to plot it. You will get exactly the same figure as you were getting on NA.
Do you know what value you have (Impedance or S11 - each of them are complex values)?
To use the plotsmithchart.m you need the impedance values, if you have the S11 you need to calculate the impedance from the (complex and without dB) S11.
With the formula you mentioned - 10log[sqrt((real^2)+(Im^2))] - you can calculate the logarithm of the magnitude of S11. This represents the reflection coefficient which is the length in the smith chart.
For the angle you need another calculation.
The plotsmithchart.m will do the calculations for you.