Questions about Optiboot Flasher

[technoblogy]

I'm planning to use your Optiboot Flasher on the ATmega1284P, but there are a few things I don't understand:

• I'm not sure what you're referring to by 'buffer' in the documentation in the header of SerialReadWrite.ino. When you say:

  Buffer must be page aligned (see declaration of flash_buffer)
  

  There is no flash_buffer in the code. I assume this means the block of flash you're writing to, flashSpace[]?

• In:

  Writing to EEPROM destroys temporary buffer
  You can write only once into one location of temporary buffer
  

  By 'temporary buffer' do you mean:

  uint8_t ramBuffer[SPM_PAGESIZE];
  

  and by EEPROM do you mean flash? Why does it get destroyed by writing it to flash?

• Why might you want to do fill-erase-write rather than erase-fill-write?

• Instead of allocating the page(s) of flash in the middle of PROGMEM as you've done here, could I specify an explicit address for flashSpace?

  For example, if I understand correctly the bootloader occupies two pages from word addresses 0xFF00 to 0xFFFF. I would like to write to one page of flash just below that, from word addresses 0xFE80 to 0xFEFF. So could I do this?

  const *uint8_t flashSpace PROGMEM = 0xFE80;
  

• Is it essential to use a RAM buffer, ramBuffer[256], or if I'm short of RAM could I write the data to flash byte by byte, treating it more like an SD card? I realise I would have to provide my own alternative to optiboot_writePage().

• Finally, I'm puzzled by the code in optiboot_readPage() in optiboot.h:

  if(read_character != 0 && read_character != 255)                      
    storage_array[j] = read_character;
  

  Why is it skipping bytes if they are 0 or 255?

Sorry about the flood of questions!

[MCUdude]

Hi!

I wrote this crude sketch five years ago, so a lot has happened to my coding skills since since then. I haven't really used this functionality myself either, but I'll try to answer your questions as good as I can. Note that this sketch relies on optiboot.h, which is mostly written by @majekw. This is the sketch](https://github.com/majekw/optiboot/blob/spm/optiboot/examples/flash_program/flash_program.ino) SerialReadWrite.ino is based on. As you can see in this sketch, here's where I've copied all the "important things to know" text.

I'm not sure what you're referring to by 'buffer' in the documentation in the header of SerialReadWrite.ino. When you say:
Buffer must be page aligned (see declaration of flash_buffer)
There is no flash_buffer in the code. I assume this means the block of flash you're writing to, flashSpace[]?

Yes, I'm referring to flashSpace[] here.

In:
Writing to EEPROM destroys temporary buffer
You can write only once into one location of temporary buffer
By 'temporary buffer' do you mean:
uint8_t ramBuffer[SPM_PAGESIZE];
and by EEPROM do you mean flash? Why does it get destroyed by writing it to flash?

When looking at the code now, there is no reason why the ramBuffer should be destroyed when writing to EEPROM. ramBuffer is just an array living in RAM, and should not be affected. But it would be great if you could try this just to check if the statement from majekw still is true (for some reason).

Why might you want to do fill-erase-write rather than erase-fill-write?

I am doing erase-fill-write. See code here.

Instead of allocating the page(s) of flash in the middle of PROGMEM as you've done here, could I specify an explicit address for flashSpace? For example, if I understand correctly the bootloader occupies two pages from word addresses 0xFF00 to 0xFFFF. I would like to write to one page of flash just below that, from word addresses 0xFE80 to 0xFEFF. So could I do this?

I've not tried this myself, and I don't have a good answer to your question. However, you can always try this and use Avrdude to dump the flash and look at its content afterward to see if it did work? Perhaps maybe @majekw can give us a straight answer? 🙂

Is it essential to use a RAM buffer, ramBuffer[256], or if I'm short of RAM could I write the data to flash byte by byte, treating it more like an SD card? I realize I would have to provide my own alternative to optiboot_writePage().

As far as I know, you can only write a full page to the flash memory at once. You could of course create a wrapper that treats each page as a byte array, but you'll always have to "write" all the bytes, even though only one has changed.

Finally, I'm puzzled by the code in optiboot_readPage() in optiboot.h:
...
Why is it skipping bytes if they are 0 or 255?

The code is based on the read routine from the original sketch. It may be that 0x00 or 0xff represents an empty space? But I do agree that 0 and 255 should be read out as well. If not, it's pretty much useless for anything else than characters.

I can change the optiboot.h and the SerialReadWrite sketch based on what you figure out what works and what don't. Looking forward to seeing what you'll come up with! Is it something that you'll write about on your blog?

[technoblogy]

Thanks for the answers! I'll let you know what I find out after I've done some experiments.

The application I've got in mind is to improve my version of uLisp for the ATmega1284P. Currently it allows you to save the workspace from RAM to the EEPROM on the chip, which limits you to saving 4Kbytes of the 16Kbytes. By saving it to flash instead I could save the entire RAM.

[JAndrassy]

I use optiboot.h in the ArduinoOTA library to store the uploaded sketch binary in upper half of the flash memory. I don't use a memory buffer. The MCU has a buffer for the page.
https://github.com/jandrassy/ArduinoOTA/blob/master/src/InternalStorageAVR.cpp

btw: fill-erase-flash is only possible in last 4 flash pages to support advanced bootloader burning

[technoblogy]

@JAndrassy Thanks, I'll take a look at your code too.

[majekw]

Lot of questions :-)

1. flash_buffer is unfortunate name which conflicts with AVR's flash buffer used to store temporary flash page before writing it. 'flash_buffer' variable is just memory in flash where we want to write. That's also a reason for page aligning, because in fact we must write whole pages.
2. Flash vs. EEPROM note: during writing we must write to internal AVR flash buffer then 'commit' write to real flash location. This internal flash buffer could be erased during operations on EEPROM. It's somehow connected in chip and this limitation is described in AVR documentation.
3. fill-erase-write vs. erase-fill-write: when you want to preserve some contents of erased page, you don't need RAM to store old contents of erased page in first case. Preparing contents of page in RAM and write all in one go is just more convenient and easier to implement, so second way is more common. There also could be some magic with NRWW/RWW sections, but I don't remember all these quirks right now. For better understanding I recommend to read chip documentation about bootloader, NRWW, RWW sections and self programming.
4. regarding 'if' with 0 and 255: it was just to output some reasonable characters to serial console, so I was replacing them to dots :-) No other magic here :-)

[technoblogy]

OK, I've got it to work! Here is some feedback. This generally relates to the ATmega1284P and may need changing for the smaller chips.

• I couldn't get anything to work at first until I remembered that my ATmega1284P board had an old bootloader on it from Jack Christensen's Mighty 1284P core. After updating the bootloader from your MightyCore everything started to work!

  So I think it might be useful to provide a function like the one in Spence Konde's DxCore called Flash.checkWritable() which checks for obvious things. I'm not sure how to write this.

• I got very confused about whether I was working with byte addresses or word addresses. It doesn't help that the memory map in the Atmel datasheet shows the top of memory on an ATmega1284P as 0xFFFF. It would be worth saying that all the addresses used by the Optiboot Flash routines are byte addresses.

• The demo program SerialReadWrite.ino makes it appear that you have to allocate your own 256-byte RAM buffer, but this isn't necessary as there's a temporary buffer in the ATmega1284P that you can fill on the fly with optiboot_page_fill().

• I don't think the function optiboot_readPage() should be included in optiboot.h as it's more of a demo function, and it wouldn't be useful for reading real data as it ignores 0x00 and 0xFF bytes. Also it wouldn't work in the upper half of flash; you need to use pgm_read_byte_far() for that. It's not really needed anyway as you can just use pgm_read_byte() or pgm_read_byte_far().

• In the documentation there's a bit of confusion over the terminology. I suggest:

  Flash buffer: the temporary page in flash filled by optiboot_page_fill().
  Flash area: the page(s) of flash you are writing to (which must be page aligned). Don't call it a buffer (it's not temporary).
  RAM area: the area in RAM you're transferring to flash, although I suggest dropping this concept.

Finally, here are my demo programs which I've tried to keep as simple as possible.

Writing to an explicit page in far flash

/*
  Test Optiboot Flash on ATmega1284P - explicit page
*/

#include "optiboot.h"

// Bootloader takes from 0x1fc00 to 0x1ffff (byte addresses).
// Write in page immediately below that
uint32_t BaseAddress = 0x1fb00;

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println("Erase");
  optiboot_page_erase(BaseAddress);
  
  Serial.println("Write data");
  uint32_t addr = BaseAddress;

  // Fill flash buffer. Page is 128 words
  for (unsigned int i=0; i<128; i++) {
    optiboot_page_fill(addr, i); addr++; addr++;
  }

  // Write flash buffer to flash area
  optiboot_page_write(BaseAddress);
  
  Serial.println("Read data");
  addr = BaseAddress;  
  for (unsigned int i=0; i<128; i++) {
    Serial.print(pgm_read_word_far(addr));
    addr++; addr++; Serial.print(' ');
  }
  Serial.println();
  
  delay(20000);
}

Writing to a page allocated in PROGMEM

/*
  Test Optiboot Flash on ATmega1284P - allocate page in PROGMEM
*/

#include "optiboot.h"

// Allocate one page of 256 bytes
const uint8_t flashSpace[256] __attribute__ (( aligned(256) )) PROGMEM = { };
uint32_t BaseAddress = (uint32_t)flashSpace;

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println("Erase");
  optiboot_page_erase(BaseAddress);
  
  Serial.println("Write data");
  uint32_t addr = BaseAddress;

  // Fill flash buffer. Page is 128 words
  for (unsigned int i=0; i<128; i++) {
    optiboot_page_fill(addr, i); addr++; addr++;
  }

  // Write flash buffer to flash area
  optiboot_page_write(BaseAddress);
  
  Serial.println("Read data");
  addr = BaseAddress;  
  for (unsigned int i=0; i<128; i++) {
    Serial.print(pgm_read_word(addr));
    addr++; addr++; Serial.print(' ');
  }
  Serial.println();
  
  delay(20000);
}

[technoblogy]

I forgot to say: thanks for providing this really useful feature!

[MCUdude]

@technoblogy thanks for the feedback! I'll go through it later and apply some of the improvements you've suggested.

BTW you can write to the upper part of flash by using the PROGMEM1 attribute:

const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on a page far away"
};

EDIT: Not, that this doesn't work for some reason. Haven't had the time to investigate why. I did change pgm_read_byte_far to pgm_read_byte_far in optiboot_readPage(), but still no success. For reference, PROGMEM1 is defined here.

[technoblogy]

Here's another example, which is what I'm using in uLisp. You simply keep calling FlashWriteInt() to write values to the flash (up to 16Kbytes in this example), and the only other thing you have to do is call FlashEndWrite() at the end. It takes care of the correct sequence of optiboot_page_erase, optiboot_page_fill, and optiboot_page_write calls for you:

const uint32_t BaseAddress = 0x1bc00;

void FlashWriteInt (uint32_t *addr, int data) {
  if (((*addr) & 0xFF) == 0) optiboot_page_erase(BaseAddress + ((*addr) & 0xFF00));
  optiboot_page_fill(BaseAddress + *addr, data);
  if (((*addr) & 0xFF) == 0xFE) optiboot_page_write(BaseAddress + ((*addr) & 0xFF00));
  (*addr)++; (*addr)++;
}

void FlashEndWrite (uint32_t *addr) {
  if (((*addr) & 0xFF) != 0x00) optiboot_page_write((BaseAddress + ((*addr) & 0xFF00)));
}
 
void loop() {
  uint32_t addr = 0;

  for (int word = 0; word<500; word++) FlashWriteInt(&addr, word);
  
  FlashEndWrite(&addr);
  
  for(;;);
}

[MCUdude]

Is there a "proper way" to read content that's stored in "far mem" (64-128kB)? I've tried to modify the optiboot_readPage function, but it seems like the pgm_get_far_address macro requires its parameter to be known at compile time. I'm getting this error:

/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_544907/sketch/SerialReadWrite.ino.cpp.o: In function `optiboot_readPage(unsigned char const*, unsigned char*, unsigned int, char)':
/Users/hans/Documents/Arduino/hardware/MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h:137: undefined reference to `r30'
/Users/hans/Documents/Arduino/hardware/MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h:137: undefined reference to `r30'
/Users/hans/Documents/Arduino/hardware/MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h:137: undefined reference to `r30'
collect2: error: ld returned 1 exit status
Using library Optiboot_flasher at version 1.0

// Function to read a flash page and store it in an array (storage_array[])
void optiboot_readPage(const uint8_t allocated_flash_space[], uint8_t storage_array[], uint16_t page, char blank_character)
{
  uint8_t read_character;
  for(uint16_t j = 0; j < SPM_PAGESIZE; j++) 
  {
    
    read_character = pgm_read_byte_far(pgm_get_far_address(allocated_flash_space) + j + SPM_PAGESIZE*(page-1));
    if(read_character != 0 && read_character != 255)
      storage_array[j] = read_character; 
    else
      storage_array[j] = blank_character;   
  }
}

[JAndrassy]

RAMZ register

[MCUdude]

The idea is to see if I can create a usable example where I utilize PROGMEM1.

// This array allocates the space you'll be able to write to
static const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on page one"
};

I want to see if there's possible to read and write to the array when it's located in the far (64kB+) memory space. So far I've been able to read from it by "manually" passing it's memory address, but it's not very elegant. When I try to write to it I either corrupt the bootloader area or the program gets permanently corrupted.

Preferably, this is what I want to acheive:

// This array allocates the space you'll be able to write to
static const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on page one but still far away!"
};

// ...

// Write content to page 1:
optiboot_writePage(flashSpace, ramBuffer, 1);

// Read page 1 of the allocated space:
optiboot_readPage(flashSpace, ramBuffer, 1); 
// Print page content
Serial.print(F("\nContent of page 1: "));
Serial.println((char*)ramBuffer);

[JAndrassy]

maybe my copy_flash_pages function's source code can help

MightyCore/avr/bootloaders/optiboot_flash/optiboot_flash.c

Lines 1190 to 1209 in 0990937

	void copy_flash_pages(uint32_t dest_page_addr, uint32_t src_page_addr, uint16_t page_count, uint8_t reset_mcu) {
	int i, j;
	for (i = 0; i < page_count; i++) { // do standard spm steps for every page
	do_spm_rampz(dest_page_addr, __BOOT_PAGE_ERASE, 0); // erase page
	for (j = 0; j < SPM_PAGESIZE; j += 2) { // fill the bytes for the page
	#ifdef RAMPZ // only devices with RAMPZ have pgm_read_word_far()
	do_spm_rampz(dest_page_addr + j, __BOOT_PAGE_FILL, pgm_read_word_far(src_page_addr + j));
	#else
	do_spm(dest_page_addr + j, __BOOT_PAGE_FILL, pgm_read_word(src_page_addr + j));
	#endif
	}
	do_spm_rampz(dest_page_addr, __BOOT_PAGE_WRITE, 0); // write the page
	dest_page_addr += SPM_PAGESIZE;
	src_page_addr += SPM_PAGESIZE;
	}
	if (reset_mcu) {
	watchdogConfig(WATCHDOG_16MS); // for a reset of the MCU
	while (1); // to prevent return to application in the 15MS to reset
	}
	}

[MCUdude]

Preferably, this is what I want to achieve:

OK, I got it working, but I still can't get around the fact that the address to the array has to be known at compile-time and thus passed to the optiboot_readPage and optiboot_writePage functions. But this isn't really a deal-breaker. I'll continue working on some of the suggestions @technoblogy came up with! There may be breaking changes to optiboot.h, but these will be minor, and absolutely for the best (like getting rid of the 0 and 255 guards in optiboot_read_page for instance).