Making non functional source changes triggers a .bin difference despite CONFIG_APP_REPRODUCIBLE_BUILD being set (IDFGH-14646)

[soliis-jones]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.5-dev-1860-g0d6099ec53, v5.3.2

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

Making non functional changes to a source file (specifically tested removing comments) does not result in a modified .bin file if CONFIG_APP_REPRODUCIBLE_BUILD is set.

What is the actual behavior?

Built .bin files differ because the contained sha256 of the underlying .elf file is modified.

Steps to reproduce.

1. idf.py build
2. xxd build/project.bin > /tmp/originalbin.hexdump
3. Remove a comment (or likely any non functional change) from a project source file
4. idf.py build
5. xxd build/project.bin > /tmp/removedcommentbin.hexdump
6. diff /tmp/originalbin.hexdump /tmp/removedcommentbin.hexdump

sdkconfig.txt

Build or installation Logs.

The below diffs occur at the beginning and end of the `.bin` file hex dumps, which apparently contain the sha256 of the associated elf file and the sha of the file itself.

$ diff /tmp/originalbin.hexdump /tmp/removedcommentbin.hexdump
12,13c12,13
< 000000b0: 163f 533d f2bf a826 23c8 8d9f 5c21 f0ad  .?S=...&#...\!..
< 000000c0: 5be2 4148 d495 e95d 8184 9398 d9cf 586e  [.AH...]......Xn
---
> 000000b0: d7bf 7294 3720 6627 bd25 02cc 4fe9 c427  ..r.7 f'.%..O..'
> 000000c0: 3cd3 2d4c 57be ab6c 7536 68ae 04fc 61a5  <.-LW..lu6h...a.
36044,36046c36044,36046
< 0008ccb0: 0000 0000 0000 0000 0000 0000 0000 003f  ...............?
< 0008ccc0: cf63 096c b559 17b8 0ea4 14dd eaca 8d5d  .c.l.Y.........]
< 0008ccd0: 28e7 355a 5024 ad92 bb3d da02 7749 ce54  (.5ZP$...=..wI.T
---
> 0008ccb0: 0000 0000 0000 0000 0000 0000 0000 00db  ................
> 0008ccc0: bc3f e26c 8240 7425 647c 1f81 2284 b455  .?.l.@t%d|.."..U
> 0008ccd0: 7b8f 7d7e 7267 172e 8810 1978 30cb 5a03  {.}~rg.....x0.Z.

More Information.

Performing clean builds and/or resetting ccache does not affect this issue.

I am inheriting an old project, but as far as I can tell there isn't anything in the sdkconfig or the source itself (FILE, DATE, etc) that would explain this.

I have made some slight modifications to the top level CMakeLists.txt in an attempt to eliminate this problem:

cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(name_of_project)

idf_build_set_property(COMPILE_OPTIONS -fdiagnostics-color=always -Wdate-time APPEND)
idf_build_set_property(CCACHE_ENABLE true)
idf_build_set_property(SOURCE_DATE_EPOCH 0)
idf_build_set_property(BUILD_RPATH_USE_ORIGIN true)
idf_build_set_property(CMAKE_SKIP_RPATH true)

If this can't be directly addressed, I would also be interested in steps to produce a .bin file that omits the .elf sha256.

[fhrbata]

Hello @soliis-jones ,

the reason is that even if you do a non-functional change, the dwarf sections will change in the elf file, because of changed line references. I tested this with esp32c3 and the hello_world example by deleting one comment from the main/hello_world_main.c.

$ for sec in $(readelf -SW build1/hello_world.elf | sed -n 's/.*] \(\.[^ ]\+\).*/\1/p'); do diff -u <(readelf -x $sec build1/hello_world.elf) <(readelf -x $sec build2/hello_world.elf ) || echo "Section $sec differs" ;done
--- /dev/fd/63	2025-02-14 13:58:58.969747566 +0100
+++ /dev/fd/62	2025-02-14 13:58:58.969747566 +0100
@@ -41708,11 +41708,11 @@
   0x000a2e90 2898c701 000d3c06 67110000 17671100 (.....<.g....g..
   0x000a2ea0 00000d04 ed090000 29237301 00021006 ........)#s.....
   0x000a2eb0 38570042 60010000 019c0213 00002a9c 8W.B`.........*.
-  0x000a2ec0 c7010002 1615ed09 00000291 642abec7 ............d*..
-  0x000a2ed0 01000217 0ef70800 00029160 2be3c901 ...........`+...
-  0x000a2ee0 0002210e 94000000 66fb0200 2b49ca01 ..!.....f...+I..
-  0x000a2ef0 0002220e 94000000 93fb0200 2c204e00 .."........., N.
-  0x000a2f00 00061200 002d6900 022e0e8d 000000c6 .....-i.........
+  0x000a2ec0 c7010002 1515ed09 00000291 642abec7 ............d*..
+  0x000a2ed0 01000216 0ef70800 00029160 2be3c901 ...........`+...
+  0x000a2ee0 0002200e 94000000 66fb0200 2b49ca01 .. .....f...+I..
+  0x000a2ef0 0002210e 94000000 93fb0200 2c204e00 ..!........., N.
+  0x000a2f00 00061200 002d6900 022d0e8d 000000c6 .....-i..-......
   0x000a2f10 fb02002e 68580042 3e110000 f5110000 ....hX.B>.......
   0x000a2f20 2f015a05 03e43202 3c2f015b 02780000 /.Z...2.</.[.x..
   0x000a2f30 30745800 42041100 002f015a 02086400 0tX.B..../.Z..d.
Section .debug_info differs
--- /dev/fd/63	2025-02-14 13:58:59.100749363 +0100
+++ /dev/fd/62	2025-02-14 13:58:59.101749377 +0100
@@ -18165,7 +18165,7 @@
   0x00046f20 74656d2e 68000c00 00746173 6b2e6800 tem.h....task.h.
   0x00046f30 0d00003c 6275696c 742d696e 3e000000 ...<built-in>...
   0x00046f40 00000402 05010005 02385700 42031001 .........8W.B...
-  0x00046f50 05050301 09060001 0304090c 00010301 ................
+  0x00046f50 05050301 09060001 0303090c 00010301 ................
   0x00046f60 09000001 03010900 00010301 09060001 ................
   0x00046f70 05150603 02090000 01051603 01090400 ................
   0x00046f80 01052003 00090200 01050503 7d090400 .. .........}...
Section .debug_line differs

As you can see the .debug_info and .debug_line sections differ, which is expected. The "problem" is that the sha256 of the source elf file, wrt esptool, is inserted into the app description in the generated bin file. Meaning the bin files are different , because the sha256 of the source source elf is different. This can be avoided by not providing --elf-sha256-offset argument to the elf2image command. In that case the bin files will be the same.

I guess this is expected behavior.

Thank you

[soliis-jones]

@fhrbata Thank you for your response!

As you can see the .debug_info and .debug_line sections differ, which is expected. The "problem" is that the sha256 of the source elf file, wrt esptool, is inserted into the app description in the generated bin file. Meaning the bin files are different , because the sha256 of the source source elf is different.

I was aware of this behavior, and confirmed by looking at the .elf file that this was the case. For those reading that would like some more info on this, see the relevant esp-idf documentation and esptool documentation.

This can be avoided by not providing --elf-sha256-offset argument to the elf2image command. In that case the bin files will be the same.

Maybe I'm missing something, but I confirmed that the arguments were removed after adding the following lines in my CMakeLists.txt:

message("esptool_elf2image_args BEFORE: ${esptool_elf2image_args}")
list(REMOVE_ITEM esptool_elf2image_args --elf-sha256-offset 0xb0)
message("esptool_elf2image_args AFTER: ${esptool_elf2image_args}")

And seeing the following output after executing idf.py fullclean build:

esptool_elf2image_args BEFORE: --flash_mode;dio;--flash_freq;80m;--flash_size;4MB;--elf-sha256-offset;0xb0;--min-rev;3;--min-rev-full;3;--max-rev-full;199
esptool_elf2image_args AFTER: --flash_mode;dio;--flash_freq;80m;--flash_size;4MB;--min-rev;3;--min-rev-full;3;--max-rev-full;199

But the sha is still present in the output .bin file.

I also attempted to list(APPEND esptool_elf2image_args --dont-append-digest) as that seemed like a relevant option but it also had no effect. Is there a different way I'm intended to modify these arguments?

[soliis-jones]

After some poking around in esp-idf/esptool and learning more about cmake I discovered the following:

There is a add_custom_command call here that latches the value of the esptool_elf2image_args before my project's CMakeLists.txt commands to modify it are executed.

The only (hacky) way I could successfully get idf.py build to generate the same bin without the elf's sha was to append to the existing custom command to repeat the generation with the correct arguments. Since this effectively repeats the esptool.py elf2image command it's not ideal, but build time impact is minimal.

list(REMOVE_ITEM esptool_elf2image_args --elf-sha256-offset 0xb0)
add_custom_command(OUTPUT "${build_dir}/.bin_timestamp" APPEND
  COMMAND echo "Repeating .bin file generation with the following elf2image arguments: ${esptool_elf2image_args}"
  COMMAND ${ESPTOOLPY} elf2image ${esptool_elf2image_args}
      -o "${build_dir}/${unsigned_project_binary}" "$<TARGET_FILE:$<GENEX_EVAL:${elf}>>"
  COMMAND ${CMAKE_COMMAND} -E md5sum "${build_dir}/${unsigned_project_binary}" > "${build_dir}/.bin_timestamp"
)

My tests confirm this produces the desired result: non functional diffs and build environment changes (different users/computers) produce equivalent .bin files.

I'm sure I'm not the only esp-idf user that desires this result while still being able to use the idf.py wrapper. Hopefully someone can inform me of a more elegant solution.

I have 1 last question: does omitting this elf file sha have any negative impact on the OTA process?

[igrr]

Just chiming in regarding the OTA process: it depends on the method you use to compare the versions available locally and offered over the air. If you don't rely on the ELF file hash in this process, then there should be no impact.

However please keep in mind that for production builds it is highly recommended to keep the ELF files for each application binary you release, as that later helps debugging application crashes, should they occur. In this case, having the ELF file hash recorded in the binary image header helps avoid any mixup between the ELF files. Accurate debug info is also important if you are collecting core dumps from devices in the field.

By the way, I am curious to learn why you consider that nonfunctional changes to source code should not affect the built binary. It doesn't seem to be a part of the common definition of reproducible builds. It's probably something reasonable, but we definitely haven't considered this requirement when designing the reproducible builds feature in IDF.