What you need to do is called a Momentum Co-Simulation.
In your existing layout, under Momentum, Parameters, add parameters (variables), and assign them as Subnetwork type, with a value. Say you have a stub transmission line that needs to be 30 mils long, so you make a variable called stub_length and give it a value of 30 (no units). In your polygon, you change Length = 30 mils to Length = stub_length mils.
Now, add ports to the in/outs of your circuit, and set up the substrate. Go to Momentum, Create/Update Components (don't recall exactly where it is... middle of the Momentum menu). In that window, define the Momentum simulation parameters.... upper/lower freq, mesh density per wavelength, Edge Mesh on/off, etc, and specify a Look-alike component. When you click OK, ADS will create a Layout Component for this design.
Open a new schematic window, go to the Component library, select your project from the tree, then drag/drop the layout design onto the schematic (the description will be "Layout Component"). You will get a circuit component that looks like your design with red port pins where you placed your in/out ports, and a variable listed by the name called stub_length = 30.
Set up your schematic as you would any other optimization... S-parameter controller, Optim block and goals (make the stub_length value an optimization variable). When you run the sim, the schematic will pick a value for stub_length, open up a temporary ADS layout, mesh the design, solve it, and return the values to the schematic component, which will the be evaluated by the S-parameter block. The optimizer will pick a new value for stub_length, open a new temp layout and repeat the process until the goals are met.