Forward port changes from v0.6 release branch

Merge a number of fixes from release-0.6 into main.
Most importantly: fix refcount calculation when copying data to heaps #1498.

Author: bettio

.github/workflows/build-and-test-macos.yaml

@@ -109,6 +109,11 @@ jobs:
       run: |
         ./tests/test-enif
 
+    - name: "Test: test-heap"
+      working-directory: build
+      run: |
+        ./tests/test-heap
+
     - name: "Test: test-mailbox"
       working-directory: build
       run: |

.github/workflows/build-and-test-on-freebsd.yaml

@@ -104,6 +104,11 @@ jobs:
           echo "%%"
           ./tests/test-enif
 
+          echo "%%"
+          echo "%% Running test-heap ..."
+          echo "%%"
+          ./tests/test-heap
+
           echo "%%"
           echo "%% Running test-mailbox ..."
           echo "%%"

.github/workflows/build-and-test-other.yaml

@@ -170,12 +170,15 @@ jobs:
         make AtomVM &&
         make test-erlang &&
         make test-enif &&
+        make test-heap &&
         make test-mailbox &&
         make test-structs &&
         file ./tests/test-erlang &&
         ./tests/test-erlang -s prime_smp &&
         file ./tests/test-enif &&
         ./tests/test-enif &&
+        file ./tests/test-heap &&
+        ./tests/test-heap &&
         file ./tests/test-mailbox &&
         ./tests/test-mailbox &&
         file ./tests/test-structs &&

.github/workflows/build-and-test.yaml

@@ -375,6 +375,13 @@ jobs:
         ./tests/test-enif
         valgrind ./tests/test-enif
 
+    - name: "Test: test-heap"
+      working-directory: build
+      run: |
+        ulimit -c unlimited
+        ./tests/test-heap
+        valgrind ./tests/test-heap
+
     - name: "Test: test-mailbox"
       working-directory: build
       run: |

.github/workflows/build-linux-artifacts.yaml

@@ -227,12 +227,15 @@ jobs:
         VERBOSE=1 make AtomVM &&
         make test-erlang &&
         make test-enif &&
+        make test-heap &&
         make test-mailbox &&
         make test-structs &&
         file ./tests/test-erlang &&
         ./tests/test-erlang -s prime_smp &&
         file ./tests/test-enif &&
         ./tests/test-enif &&
+        file ./tests/test-heap &&
+        ./tests/test-heap &&
         file ./tests/test-mailbox &&
         ./tests/test-mailbox &&
         file ./tests/test-structs &&

CHANGELOG.md

@@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `net:gethostname/0` on platforms with gethostname(3).
 - Added `socket:getopt/2`
 - Added `supervisor:terminate_child/2`, `supervisor:restart_child/2` and `supervisor:delete_child/2`
+- Added support for 'erlang:--/2'.
 - Added `esp:partition_read/3`, and documentation for `esp:partition_erase_range/2/3` and `esp:partition_write/3`
 - Added support for list insertion in 'ets:insert/2'.
 - Support to OTP-28
@@ -27,9 +28,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Added the ability to run beams from the CLI for Generic Unix platform (it was already possible with nodejs and emscripten).
-- Added support for 'erlang:--/2'.
+- Added the ability to run beams from the CLI for Generic Unix platform (it was already possible
+with nodejs and emscripten)
 - Added preliminary support for ESP32P4 (no networking support yet).
+- Added memory info in `out_of_memory` crash logs to help developers fix memory issues.
 
 ### Fixed
 
@@ -38,11 +40,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Adding missing check, passing a non numeric argument to a function expecting a floating point
 might lead to a crash in certain situations.
 - Fixed several bugs in `http_server` (#1366)
-- Fixed generic\_unix `socket_driver` to return `{gen_tcp, closed}` when socket is closed on Linux instead of `{gen_tcp, {recv, 104}}`
+- Fixed generic\_unix `socket_driver` to return `{gen_tcp, closed}` when socket is closed on Linux
+instead of `{gen_tcp, {recv, 104}}`
 - Fixed a memory leak where modules were not properly destroyed when the global context is destroyd
 - alisp: fix support to variables that are not binaries or integers.
 - Fixed destruction of ssl-related resources
-- Fix corruption when dealing with specific situations that involve more than 16 x registers when
+- Fixed corruption when dealing with specific situations that involve more than 16 x registers when
 certain VM instructions are used.
 - Fixed ESP32 GPIO interrupt trigger `none`
 - Fixed an issue where a timeout would occur immediately in a race condition
@@ -51,19 +54,22 @@ certain VM instructions are used.
 - Fixed a race condition affecting multi-core MCUs where a timeout would not be properly cleared
 - Fixed a double free when esp32 uart driver was closed, yielding an assert abort
 - Fixed compilation with latest debian gcc-arm-none-eabi
-- Fix `network:stop/0` on ESP32 so the network can be started again
-- Fix matching of binaries on unaligned boundaries for code compiled with older versions of OTP
-- Fix a memory corruption caused by `binary:split/2,3`
-- Fix deadlock in socket code
-- Fix bug in opcode implementation (`select_val`): when selecting a value among many others a
+- Fixed `network:stop/0` on ESP32 so the network can be started again
+- Fixed a memory corruption caused by `binary:split/2,3`
+- Fixed deadlock in socket code
+- Fixed bug in opcode implementation (`select_val`): when selecting a value among many others a
 shallow comparison was performed, so it was working just for plain values such as atoms and small
 integers
 - Fixed support for setting esp32 boot_path in NVS.
 - Fixed race conditions in network:start/stop.
 - Fixed crash calling network:sta_rssi(), when network not up.
-- Fix error handling when calling `min` and `max` with code compiled before OTP-26: there was a
-bug when handling errors from BIFs used as NIFs (when called with `CALL_EXT` and similar opcodes)`
-- Fix matching of binaries on unaligned boundaries for code compiled with older versions of OTP
+- Fixed error handling when calling `min` and `max` with code compiled before OTP-26: there was a
+bug when handling errors from BIFs used as NIFs (when called with `CALL_EXT` and similar opcodes)
+- Fixed matching of binaries on unaligned boundaries for code compiled with older versions of OTP
+- Added missing out of memory handling in binary_to_atom
+- Fixed call to funs such as fun erlang:'not'/1, that make use of BIFs
+- Fixed potential crashes or memory leaks caused by a mistake in calculation of reference counts
+and a race condition in otp_socket code
 
 ## [0.6.5] - 2024-10-15
 

doc/release-notes.md.in

@@ -33,6 +33,7 @@ AtomVM will run BEAM files that have been compiled using the following Erlang an
 | ✅ OTP 24 | ✅ 1.14 |
 | ✅ OTP 25 | ✅ 1.14 |
 | ✅ OTP 26 | ✅ 1.15 |
+| ✅ OTP 27 | ✅ 1.17 |
 | ✅ OTP 28 | ✅ 1.17 |
 
 ```{note}
@@ -63,6 +64,7 @@ AtomVM currently supports the following [Espressif ESP SoCs](https://www.espress
 | [ESP32h2](https://www.espressif.com/sites/default/files/documentation/esp32-h2_datasheet_en.pdf) | ✅ |
 | [ESP32s2](https://www.espressif.com/sites/default/files/documentation/esp32-s2_datasheet_en.pdf) | ✅ |
 | [ESP32s3](https://www.espressif.com/sites/default/files/documentation/esp32-s3_datasheet_en.pdf) | ✅ |
+| [ESP32p4](https://www.espressif.com/sites/default/files/documentation/esp32-p4_datasheet_en.pdf) Datasheet Pending | ✅ |
 
 AtomVM currently supports the following versions of ESP-IDF:
 
@@ -97,13 +99,13 @@ AtomVM tests this build on the latest Ubuntu github runner.
 
 ### Raspberry Pi Pico Support
 
-AtomVM supports deployment on the [Raspberry Pico RP2040](https://www.raspberrypi.com/documentation/microcontrollers/rp2040.html) architecture.
+AtomVM supports deployment on the [Raspberry Pico RP2040](https://www.raspberrypi.com/documentation/microcontrollers/pico-series.html#pico-1-family) architecture.
 
 AtomVM currently supports the following Raspberry Pico development boards:
 
 | Development Board | AtomVM support |
 |------------------------------|----------------|
-| [Raspberry Pico and Pico H](https://www.raspberrypi.com/documentation/microcontrollers/raspberry-pi-pico.html#raspberry-pi-pico-and-pico-h) | ✅ |
-| [Raspberry Pico W and Pico WH](https://www.raspberrypi.com/documentation/microcontrollers/raspberry-pi-pico.html#raspberry-pi-pico-w-and-pico-wh) | ✅ |
+| [Raspberry Pico and Pico H](https://www.raspberrypi.com/documentation/microcontrollers/pico-series.html#pico-1-technical-specification) | ✅ |
+| [Raspberry Pico W and Pico WH](https://www.raspberrypi.com/documentation/microcontrollers/pico-series.html#picow-technical-specification) | ✅ |
 
 Building the AtomVM virtual machine for Raspberry Pico is optional.  In most cases, you can simply download a release image from the AtomVM [release](https://github.com/atomvm/AtomVM/releases) repository.  If you wish to work on development of the VM or use one on the additional drivers that are available in the [AtomVM repositories](https://github.com/atomvm) you will to build AtomVM from source.  See the [Build Instructions](build-instructions.md) for information about how to build AtomVM from source code.

doc/src/atomvm-tooling.md

@@ -347,11 +347,6 @@ For instructions about how to install AtomVM on the `generic_unix` platform, see
 
 The [`ExAtomVM`](https://github.com/atomvm/ExAtomVM) plugin supports flash targets for various device types.  These targets are described in more detail below.
 
-```{attention}
-Currently, the [`ExAtomVM`](https://github.com/atomvm/ExAtomVM) tool only supports flash targets for the ESP32 and
-STM32 platforms.
-```
-
 #### ESP32 flash task
 
 To flash AtomVM packbeam file to an ESP32 device, use the `mix.esp32.flash` target.  Users will typically specify the device port and baud rate as command-line options to this target.
@@ -401,7 +396,7 @@ You can now use a serial console program such as [minicom](https://en.wikipedia.
     I (922) AtomVM: Starting esp32init.beam...
     ---
     AtomVM init.
-    I (932) sys: Loaded BEAM partition main.avm at address 0x210000 (size=1048576 bytes)
+    I (932) sys: Loaded BEAM partition main.avm at address 0x250000 (size=1048576 bytes)
     Starting application...
     Hello World
     AtomVM finished with return value: ok

doc/src/build-instructions.md

@@ -231,6 +231,11 @@ $ cd <atomvm-source-tree-root>
 $ cd src/platforms/esp32
 ```
 
+If you want to build an image with Elixir modules included you must first have a version of Elixir installed that is compatible with your OTP version, then add the following line to sdkconfig.defaults:
+```shell
+CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-elixir.csv"
+```
+
 Start by updating the default build configuration of local `sdkconfig` file via the `idf.py reconfigure` command:
 
 ```shell
@@ -378,11 +383,7 @@ The AtomVM Flash memory is partitioned to include areas for the above binary art
 
 The flash layout is roughly as follows (not to scale):
 
-    +-----------------+  ------------- 0x0000
-    |    secure       |
-    |     boot        | 4KB
-    |                 |
-    +-----------------+  ------------- 0x1000
+    +-----------------+  ------------- 0x0 | 0x1000 | 0x2000
     |                 |             ^
     |   boot loader   | 28KB        |
     |                 |             |
@@ -404,9 +405,9 @@ The flash layout is roughly as follows (not to scale):
     |                 |             |
     |                 |             |
     +-----------------+             |
-    |     boot.avm    | 256KB       v
-    +-----------------+  ------------- 0x210000
-    |                 |             ^
+    |     boot.avm    | 256-512KB   v
+    +-----------------+  ------------- 0x210000 for Erlang only images or
+    |                 |             ^  0x250000 for images with Elixir modules
     |                 |             |
     |     main.avm    | 1MB+        | Erlang/Elixir
     |                 |             | Application
@@ -418,22 +419,25 @@ The following table summarizes the partitions created on the ESP32 when deployin
 
 | Partition | Offset | Length | Description |
 |:----------|:-------|:-------|:------------|
-| Secure Boot | 0x00 | 4kB | Initialization vectors and other data needed for ESP32 secure boot. |
-| Bootloader | 0x1000 | 28kB | The ESP32 bootloader, as built from the IDF-SDK.  AtomVM does not define its own bootloader. |
+| Bootloader | 0x0 \| 0x1000 \| 0x2000 | 28kB | The ESP32 bootloader, as built from the IDF-SDK.  AtomVM does not define its own bootloader. The offset of the bootloader varies by chip.|
 | Partition Table | 0x8000 | 3kB | The AtomVM-defined partition table. |
 | NVS | 0x9000 | 24kB | Space for non-volatile storage. |
 | PHY_INIT | 0xF000 | 4kB | Initialization data for physical layer radio signal data. |
 | AtomVM virtual machine | 0x10000 | 1.75mB | The AtomVM virtual machine (compiled from C code). |
 | boot.avm | 0x1D0000 | 256k | The AtomVM BEAM library, compiled from Erlang and Elixir files in the AtomVM source tree. |
-| main.avm | 0x210000 | 1mB | The user application.  This is where users flash their compiled Erlang/Elixir code |
+| main.avm | `0x210000` \| `0x250000` | 1mB | The user application.  This is where users flash their compiled Erlang/Elixir code |
+
+```{warning}
+There is an important difference in the partition layout between the minimal images and those build with Elixir support. To accommodate the extra Elixir modules the boot.avm partition on these images is larger, and the application offset is moved accordingly. When working with Elixir supported images it is important to always use the offset `0x250000` whether using `mix` or the `atomvm_rebar3_plugin` (possibly to test an Erlang app), otherwise part of the boot.avm partition (specifically the area where many Elixir modules are located) will be overwritten with the application, but the VM will still be trying to load from the later `0x250000` offset. This should be kept in mind reading the rest of build instructions, and [AtomVM Tooling](./atomvm-tooling.md) sections of the docs that cover the use of rebar3, for these sections an Erlang only image is assumed.
+```
 
 ### The `boot.avm` and `main.avm` partitions
 
 The `boot.avm` and `main.avm` partitions are intended to store Erlang/Elixir libraries (compiled down to BEAM files, and assembled as AVM files).
 
 The `boot.avm` partition is intended for core Erlang/Elixir libraries that are built as part of the AtomVM build.  The release image of AtomVM (see below) includes both the AtomVM virtual machine and the `boot.avm` partition, which includes the BEAM files from the `estdlib` and `eavmlib` libraries.
 
-In contrast, the `main.avm` partition is intended for user applications.  Currently, the `main.avm` partition starts at address `0x210000`, and it is to that location to which application developers should flash their application AVM files.
+In contrast, the `main.avm` partition is intended for user applications.  Currently, the `main.avm` partition starts at address `0x210000` for thin images or `0x250000` for images with Elixir modules, and it is to that location to which application developers should flash their application AVM files.
 
 The AtomVM search path for BEAM modules starts in the `main.avm` partition and falls back to `boot.avm`.  Users should not have a need to override any functionality in the `boot.avm` partition, but if necessary, a BEAM module of the same name in the `main.avm` partition will be loaded instead of the version in the `boot.avm` partition.
 
@@ -456,20 +460,31 @@ you target a different build directory when running CMake.
 
 Running this script will generate a single `atomvm-<sha>.img` file in the `build` directory of the esp32 source tree, where `<sha>` is the git hash of the current checkout.  This image contains the ESP32 bootloader, AtomVM executable, and the `eavmlib` and `estdlib` Erlang libraries in one file, which can then be flashed to address `0x1000` for the esp32. The bootloader address varies for other chip variants. See the [flashing a binary image to ESP32](./getting-started-guide.md#flashing-a-binary-image-to-esp32) section of the [Getting Started Guide](./getting-started-guide.md) for a chart with the bootloader offset address of each model.
 
-The `mkimage.sh` script is run from the `src/platform/esp32` directory as follows:
+To build a thin image with only Erlang libraries `mkimage.sh` script is run from the `src/platform/esp32` directory as follows:
 
 ```shell
 $ ./build/mkimage.sh
-Writing output to /home/joe/AtomVM/src/platforms/esp32/build/atomvm-esp32-0.6.0
--dev+git.602e6bc.img
+Writing output to /home/joe/AtomVM/src/platforms/esp32/build/atomvm-esp32.img
+=============================================
+Wrote bootloader at offset 0x1000 (4096)
+Wrote partition-table at offset 0x8000 (32768)
+Wrote AtomVM Virtual Machine at offset 0x10000 (65536)
+Wrote AtomVM Core BEAM Library at offset 0x1D0000 (1114112)
+```
+
+To build a full image with Erlang and Elixir libraries the path to the previously (during the generic_unix build) built `elixir_esp32boot.avm` must be passed to the `mkimage.sh` script as follows (Note: this is still run from the AtomVM/src/platforms/esp32 directory for the relative path to work - feel free to use the absolute path to this file):
+
+```shell
+$ ./build/mkimage.sh --boot ../../../build/libs/esp32boot/elixir_esp32boot.avm
+Writing output to /home/joe/AtomVM/src/platforms/esp32/build/atomvm-esp32.img
 =============================================
 Wrote bootloader at offset 0x1000 (4096)
 Wrote partition-table at offset 0x8000 (32768)
 Wrote AtomVM Virtual Machine at offset 0x10000 (65536)
-Wrote AtomVM Core BEAM Library at offset 0x110000 (1114112)
+Wrote AtomVM Core BEAM Library at offset 0x1D0000 (1114112)
 ```
 
-Users can then use the `esptool.py` directly to flash the entire image to the ESP32 device, and then flash their applications to the `main.app` partition at address `0x210000`,
+Users can then use the `esptool.py` directly to flash the entire image to the ESP32 device, and then flash their applications to the `main.app` partition at address `0x210000`, (or `0x250000` for Elixir images)
 
 But first, it is a good idea to erase the flash, e.g.,
 
@@ -488,7 +503,7 @@ Hard resetting...
 
 #### Flashing Release Images
 
-After preparing a release image you can use the `flashimage.sh`, which is generated with each build that will flash the full image using the correct flash offset for the chip the build was configured for.
+After preparing a release image you can use the `flashimage.sh`, which is generated with each build that will flash the full image using the correct flash offset for the chip the build was configured for using the either the default Erlang only `partitions.cvs` table, or the `partitions-elixir.cvs` table if that was used during the configuration.
 
 ```shell
 $ ./build/flashimage.sh
@@ -497,7 +512,7 @@ $ ./build/flashimage.sh
 To perform this action manually you can use the `./build/flash.sh` tool (or `esptool.py` directly, if you prefer):
 
 ```shell
-$ FLASH_OFFSET=0x1000 ./build/flash.sh ./build/atomvm-esp32-0.6.0-beta-0.img
+$ FLASH_OFFSET=0x1000 ./build/flash.sh ./build/atomvm-esp32-0.6.6.img
 esptool.py v2.8-dev
 Serial port /dev/tty.SLAB_USBtoUART
 Connecting........_
@@ -525,7 +540,8 @@ have a way to recover and re-write any such data, if you need to retain it.
 
 ### Flashing Applications
 
-Applications can be flashed using the `flash.sh` script in the esp32 build directory:
+Applications can be flashed using the `flash.sh` script in the esp32 build directory (the application offset is set
+correctly depending on the build configuration):
 
 ```shell
 $ ./build/flash.sh ../../../build/examples/erlang/esp32/blink.avm
@@ -562,12 +578,12 @@ applications for the AtomVM platform.
 
 #### Flashing the core libraries
 
-If you are doing development work on the core Erlang/Elixir libraries and wish to test changes that do not involve the `C` code in the core VM you may flash `atomvmlib.avm` to the avm.lib partition (offset 0x1D0000) by using the `flash.sh` script in the esp32 build directory as follows:
+If you are doing development work on the core Erlang/Elixir libraries and wish to test changes that do not involve the `C` code in the core VM you may flash `esp32boot.avm` (or `elixir_esp32boot.avm` when using an Elixir partition table) to the boot.avm partition (offset 0x1D0000) by using the `flash.sh` script in the esp32 build directory as follows:
 
 ```shell
-$ build/flash.sh -l ../../../build/libs/atomvmlib.avm
+$ build/flash.sh -l ../../../build/libs/esp32boot.avm
 %%
-%% Flashing ../../../build/libs/atomvmlib.avm (size=116k)
+%% Flashing ../../../build/libs/esp32boot.avm (size=116k)
 %%
 esptool.py v4.5.1
 Serial port /dev/ttyUSB0