I think that both has advantage and inconvenient.
In the case of rectangle fp, the pros are:
- easy to integrate in the total IC (if your digital block is an IP)
- the utilization is averaged on the total surface, which can help during routing (depend on the tools)
The cons:
- As the shape is fixed, if the rest of the macro are already fixed you will had a lost on area
For rectilinear, the pros are:
- you can have the desired shape. So if the rest of the chip macro (analog block) is already fixed, it will help the integration
- you can use the maximum area of your chip
The cons:
- depending on the tool you are using, if the shape is too complex cells are not efficiently placed and routing is difficult
Personally due to the shape constraints and to optimize area I almost only use rectilinear floorplan.