Solder mask cutouts can be rectangular. To change to rectangular cutout click on the Rectangle tab in the cutouts properties in the properties panel.
Rectangular cutout can have rounded corners.
without and with rounded corners
To edit the parameters of a cutout, first select it. The properties panel will display the properties for the selected cutout.
The x coordinate of the center of the cutout
Y
The y coordinate of the center of the cutout
Width
The width of the cutout.
Height
The height of the cutout.
Roundness
Drag to from left to right to increase the rounding of the corners.
Layer
Select the layer for the cutout