In general case support for new layout requires adding special support for this layout into GWT Designer.
There are no way to GWT Designer to know magically how to generate code, show editing feedbacks, etc.
In simplest case, when layout is flow-based, you can use wbp-component.xml description for your new layout.
See file FlowLayout.wbp-component.xml in "wbp-meta" folder of com.google.gdt.eclipse.designer.GXT plugin.
In more complex cases you need to write JavaInfo model, register your plugin as GWT toolkit contributor and write GEF EditPart, LayoutEditPolicy, etc.