After generating soldermask files from Altium Designer, I found that while for most of the pads the sodlermask is of the same shape and with slight expansion, and for round holes the soldermask is round, for BGA pins which are round discs I got non-round discs for them.
Does anyone know why this happen and the solution?
I suspect that the soldermask openings are correct (round) if you view them in the Gerber files - you should generate Gerbers and check there. It's actually difficult to generate a soldermask opening of that shape by accident. But I have seen a number of cases where soldermask opening is displayed incorrectly in Altium even thought it's correct in the Gerber file.
I think that soldermask opening must have been created manually (-1000 expansion would generate no opening) You should be able to go into the footprint's top solder layer, delete the bad shape, and manually place a full circle on the soldermask layer (or change the soldermask expansion to generate the circle you want).