Accessing Zynq FTM_GPO/I Pins for Low Overhead PS/PL Communication

[Michael Graf_Research]

Hello RapidWright (RW) Community,

I am attempting to use the ftm_gpo/i pins from the Zynq as a low overhead mechanism to stimulate my RW module instances. If successful, this example would be largely useful to the community as there's no existing RW examples that I could find that demonstrate a low overhead option to stimulate our DUTs. Unfortunately, what I thought would be relatively simple has become a rather complex issue.

Below is a screenshot of the Zynq US+ block diagram and the block diagram of example module instance design, slice_0, that I am attempting to connect via RW. In addition, I have included the RW java code to load the zynq design, place the slice module, make the connections, and save the result. In this example, I am attempting to connect a subset of the zynq_o ports to I on slice_0 and slice_0 o output to zynq_i. What appears to be occurring is that by default, the ftm_gpo/I connections are routed to pins by default by Vivado as shown in the zynq_example.dcp. This results in additional connections to these pins unroutable as there is no viable routing path from the pad output when attempting to route the design in Vivado. I've attempted to circumvent this issue by creating a zynq design that has an intermediary dummy input between the ftm_gpo/I connection but this approach results in complex hierarchical connection modifications that I have yet to get working (despite trying to follow the connectAcrossHierarchy method found in a previous post). With that being said, does anyone have any thoughts as to how best I could resolve this issue? I've included all the .dcp files, zynq_example.dcp (the base design), example (slice_0), and zynq_test.dcp (the output design) at the following link: https://drive.google.com/drive/folders/1PsbB70Bq1yFRDAWu-gEQhsgSplUgc16x?usp=sharing

public void Run_RW() {
        System.out.println("Working Directory = " + System.getProperty("user.dir"));

        String designName = "zynq_example.dcp";
        String designEdif = "zynq_example.edif";
        Design design = Design.readCheckpoint(designName, designEdif);

        Device device = design.getDevice();
        EDIFNetlist netlist = design.getNetlist();
        EDIFCell top = design.getTopEDIFCell();
        
        ModuleImpls modules = new ModuleImpls();
        String[] slice_name = {"example"};

        com.xilinx.rapidwright.design.Module m = new Module(Design.readCheckpoint("example.dcp", CodePerfTracker.SILENT), "example_0_metadata.txt");
        m.setName("example");
        m.getNetlist().changeTopName("example");
        m.getNetlist().renameNetlistAndTopCell("example");
        modules.add(m);
        
        m.calculateAllValidPlacements(m.getDevice());
        netlist.migrateCellAndSubCells(m.getNetlist().getTopCell());
        
        ModuleInst mi = null;
        Site[] sliceSites = device.getAllSitesOfType(SiteTypeEnum.SLICEL);
        for(Site site: sliceSites) {
            if(m.isValidPlacement(site, design.getDevice(), design)) {
                mi = design.createModuleInst("slice_0_1", m);
                mi.getCellInst().setCellType(m.getNetlist().getTopCell());
                mi.place(site);
                break;
            }
        }
        
        mi.connect("o", "zynq_i", 0);
        mi.connect("i", "zynq_o", 0);

        System.out.println("Write output file");
        design.writeCheckpoint("zynq_test.dcp", CodePerfTracker.SILENT);
    }

Please note that Vivado 2020.2 & RW 2020.2.2 were used for this process.

Thank you in advance for your assistance!

Best,

MC

[RapidWright]

Thanks for the post, sounds like a cool use case.  There are at least two ways to approach this.  

One approach might be to simply write some RapidWright code to unconnect the IBUFs/OBUFs and splice in the module you want.  

(I wrote some example code, see attached).

Another approach as you suggested, might be to create a dummy module with the interface that would match your desired FTM pins.  Then, you could use (in Vivado):

update_design -black_box -cells [get_cells <your_dummy_cell>]

write_checkpoint <your_zynq_example_with_bb.dcp>

Then, in RapidWright you could read in both your desired module and top-level zynq with black box and populate the black box with your desired module using the Design.createModuleInst() method where the instance name is the black box name (also ensuring that the interface--same ports/widths, etc---to both black box and your module are identical).  Place the module as desired and then route the connections.  

Hope that helps.

[RapidWright]

One more thing I forgot to mention.  If you want to use the example source code I shared, you'll have to update to the latest RapidWright (git pull) and get a patch:

cd $RAPIDWRIGHT_PATH

wget https://github.com/Xilinx/RapidWright/releases/download/v2020.2.7-beta/rapidwright_api_lib-2020.2.7-patch2.zip

unzip -o  rapidwright_api_lib-2020.2.7-patch2.zip

[Michael Graf_Research]

Thank you for the detailed response!

I updated RW as suggested and attempted to use the example code you provided. Establishing the connections from the zynq_o to i worked like a charm, but it seems to cause odd behavior when performing the connection to the zynq_i to o. Looking at the hierarchy, the connection appears to incorrectly be established at the top level and shows a connection to o[5] rather than index 0 as specified in the code. I've included an image of the hierarchy as shown in Vivado. Although Vivado says this connection can be routed, that single connection fails. I've also included the route_report_status output of the net.

Any thoughts as to what could be causing this issue or what next step I should take to diagnose the issue? 

Best,

MC

[RapidWright]

Sorry for the delayed response.  I think the easiest approach here is to just reverse the index direction into the port array (I think this is simply a matter of the RTL directionality specified "downto" vs. "to").  You can just modify the code slightly (see abbreviated edits to lines ~56 of the example code):

            //...

            EDIFPort port = mi.getCellInst().getPort(modPortName);

            newNet.createPortInst(modPortName, port.getWidth()-busIdx-1, mi.getCellInst());

            // Remove logical and physical net connected to port

            // ...

[Michael Graf_Research]

I implemented the fix you suggested which does correct the output connection issues as shown in the image below; however, it doesn't resolve the second issue where the signal remain unroutable despite Vivado indicating the Route status is `Routable but not routed`. I've included the RW output dcp after executing the `route_design` command in Vivado. I've also attempting to select the individual To confirm this wasn't a visual bug in Vivado, I executed the report_route_status command and a find query to for routes labeled as unrouted which showed that indeed these nets are not routed. This was done using Vivado 2020.2 on Linux X86, and the problem .dcp been uploaded here for review: https://drive.google.com/file/d/1MIy29735EZmyQKVci_Cy5MBZc3k_ChrO/view?usp=sharing

Please let me know if you have any suggestions for next steps or what additional information I should provide to assist in debugging.

Thanks,

MC

[RapidWright]

Apologies for getting back to you late.  The issue is that internal to sites, there are site wires that also need to be routed.  Site routing typically happens during placement and in the case of combining the two designs, some of the site routing was removed (as it was no longer correct).  Since Vivado's router doesn't redo site routing, it expects it to be there before beginning a routing run.  

RapidWright has a basic site router that can resolve the missing site routing.  You can invoke it by calling:

        netlist.resetParentNetMap();

        design.routeSites();

Just before you write out a design checkpoint.  This will allow the Vivado router to finish.  However, when I run route_design in Vivado on the resulting DCP, I still get one net that is unrouted, GND (GLOBAL_LOGIC0).  I haven't quite figured out why, but I thought I should at least give you an update.

Chris

[RapidWright]

One more update, I checked zynq_example.dcp and the same net is unrouted.  If I attempt to route that net (GND) in Vivado, it fails.  It appears there are several pins on the input of the ZYNQ site that are to be driven to GND but there is no way to configure routing to that point.  It doesn't appear that this is a RapidWright issue.

[Michael Graf_Research]

After playing around with the design a bit more, I was finally able to get it to work! The main issue stemmed with using the pblock to constrain the PS8 cell to the single cell in an attempt to use as few resources in the fabric as possible. Apparently, the PS8 utilizes numerous LUTs from the the adjacent SLICE column to drive some of the unused pins to ground. This was represented as GLOBAL_LOGIC within the design. This behavior seems extremely wasteful of PL resources, but I digress. One other correction I made was to reduce the port size of zynq_o/i to avoid dealing with handling design constraints for the unused pins. Lastly, I updated the example dcp to use a better LUT Mask that would be more visible in testing--currently just outputs 0.

With that being said, RapidWright stitched everything together, and it worked when tested using PetaLinux & Uboot.

Thanks again for all the help RapidWright team and hopefully this helps someone else in the future.

Best,

MC