`Design.writeCheckpoint` emits non-deterministic binary outputs

[korbin]

I run RapidWright and Vivado in Docker (but have run it in similar caching systems previously) which provides binary layer caching for the inputs and outputs to build stages. If the checksums for all of the inputs (files, arguments, etc.) don't change, intermediate stages can be cached and reused.

I have synth, opt, place, route build stages for a plain Vivado flow. In some builds, I use a RapidWright intermediate stage to manipulate Vivado DCPs.

If I run:

 val dcp = Design.readCheckpoint("vivado.dcp")
 dcp.writeCheckpoint("rapidwright1.dcp")
 dcp.writeCheckpoint("rapidwright2.dcp")

The emitted DCPs: rapidwright1.dcp and rapidwright2.dcp contain different binary data (and interestingly: on a 522MB Vivado DCP, the DCPs that RapidWright emits are only 356MB).

I assume a timestamp or something is getting inserted into the DCP output which is causing the file checksums to be different, breaking any kind of caching one might be using:

# sha256sum rapidwright*.dcp
3d88a165e38282320b92e8ac020b7ef4472fda2d1bc971f261b868dcdcbd8a31  rapidwright1.dcp
aa514d798478831d6ad232ddc75fdb6617670cefba4dd1cb1eb9abba77298ab1  rapidwright2.dcp

I would love the ability for rapidwright1.dcp and rapidwright2.dcp to have the same binary contents and checksum - even if this means fixing or explicitly passing the timestamp or whatever is changing internally. It's okay if vivado.dcp and rapidwright1.dcp don't match, as I'm sure these are radically different codebases, DCP generation routines, etc., but successive RapidWright runs should be able to result in consistent outputs.

[eddieh-xlnx]

Hi @korbin. Thanks for explaining your need.

I assume a timestamp or something is getting inserted into the DCP output

That is correct. There are a number of timestamps that I know of (and likely some I don't know of, since it's not been a problem for me to look). While I can see your reasoning, it's not clear whether we can omit all such timestamps -- I expect this same non-determinism problem also exists with Vivado too?

RapidWright also employs parallel threads to write out a DCP, such that the output binary may not be deterministic even if were to eliminate timestamps. Forcing that step to be deterministic will require it to be serialized with a corresponding impact to runtime.

[korbin]

I expect this same non-determinism problem also exists with Vivado too?

Yes, Vivado write_checkpoint does result in different DCPs as well (though Vivado seems to be completely deterministic for the underlying synth/implementation results.)

The way RapidWright is commonly used (RapidWright itself is interleaved into the end-user build process/program) makes this problem a bit more painful: alterations to the RapidWright-using scripts invalidate that part of the cache (causing downstream invalidation), whereas users of Vivado do not modify Vivado ever.

While I can see your reasoning, it's not clear whether we can omit all such timestamps

Omission is one possibility, being able to set them to a fixed value is another (via env var or direct argument.) I tried using libfaketime unsuccessfully to accomplish this.

RapidWright also employs parallel threads to write out a DCP

I'm surprised to learn this is possible, but yes that makes sense and would also be required (-ultrathreads in Vivado also causes thread-related implementation determinism issues).

[clavin-xlnx]

I would love the ability for rapidwright1.dcp and rapidwright2.dcp to have the same binary contents and checksum - even if this means fixing or explicitly passing the timestamp or whatever is changing internally. It's okay if vivado.dcp and rapidwright1.dcp don't match, as I'm sure these are radically different codebases, DCP generation routines, etc., but successive RapidWright runs should be able to result in consistent outputs.

Hi @korbin, I think it might be difficult in the short term to build out functionality that would support this. One potential work around idea, however, would be to instead of writing out the design as a DCP, you could write it out using the FPGA Interchange Format. This format doesn't include timestamps and should be deterministic (it is not currently multi-threaded). You can dump and reload from this format for those scenarios when you want file determinism, and then re-convert to DCP when needed. This process will take some extra time, but I think it might be able to satisfy the use case.

I ran a quick test to see if this idea holds (running rapidwright jython):

d = Design.readCheckpoint("test/RapidWrightDCP/microblazeAndILA_3pblocks_2024.1.dcp")
from com.xilinx.rapidwright.interchange import Interchange
Interchange.writeDesignToInterchange(d, "microblazeAndILA_3_pblocks_2024.1")
# I exited and re-started
d = Design.readCheckpoint("test/RapidWrightDCP/microblazeAndILA_3pblocks_2024.1.dcp")
from com.xilinx.rapidwright.interchange import Interchange
Interchange.writeDesignToInterchange(d, "microblazeAndILA_3_pblocks_2024.1_try2")

Then from the command line:

md5sum microblaze*
d26b8f0dd5b84210c235e3a6dc8a7149  microblazeAndILA_3_pblocks_2024.1.netlist
67228afcad30486b9c568d83c1d69686  microblazeAndILA_3_pblocks_2024.1.phys
d26b8f0dd5b84210c235e3a6dc8a7149  microblazeAndILA_3_pblocks_2024.1_try2.netlist
67228afcad30486b9c568d83c1d69686  microblazeAndILA_3_pblocks_2024.1_try2.phys

Note that there are two files that are needed to reconstitute the design (.netlist and .phys), however, both seem deterministic. Give it a shot and let us know how it goes.