Adds support for flash encryption

[4rzael]

Introduction

This PR adds support for flash encryption, by adding a --encrypt flag to flash and write-bin commands.
Requested by #70

As written below, I still need to perform a few tests (waiting for new chips, I hope to have it done in a few days).

Implementation

When performing encrypted writes, espflash will use non-compressed flashing instructions to the bootloader, as compression + encryption is not supported. Similarly, skip & verify features are disabled on encrypted partitions.

When passed to the write-bin command, espflash perform an encrypted flash directly.

The encrypt flag is stored in WriteBinArgs, and then directly passed to the created RomSegment.

When passed to the flash command, espflash will perform encrypted flashes on bootloader, partition, and application partitions.

The parameter flow is a bit more complex there: we start in FlashArgs, which is used to construct FlashData, which is used to construct IdfBootloaderFormat, which finally populates the appropriate RomSegments.

Tests:

• write-bin with --encrypt
• write-bin without --encrypt
• flash with --encrypt
• flash without --encrypt
• cargo-espflash flash with --encrypt
• cargo-espflash flash without --encrypt

[jessebraham]

Thanks for the PR. Unfortunately, this include breaking changes and will require a new major release, however we are currently discussing a v4 release anyway, so probably this is not too much of an issue. If I can just ask you to be patient, once we come up with a game plan I can get back to you with regards to merging this. Hopefully within a week or two we will have a better idea of what our plans are moving forward.

[4rzael]

Hi ! Thanks for the quick response.
Yes, I understand. I'll wait for your input as to how to get it merged to v4.

[jessebraham]

We did some planning this week for the v4 release. I do not think we will start working on this quite yet, probably not for a few weeks still, but in the meantime I will publish one final v3 release on Monday, and then we can at least unblock this PR (and others).

[ivmarkov]

@4rzael Please excuse my ignorance... I do not understand how espflash can write encrypted payload given that there is no encryption key around?

[ivmarkov]

@4rzael Also I'm a bit lost as to what exactly is requested in #70 ?

My understanding as to what @jessebraham demonstrates in #70, is just one way to do encryption, which is - "first flash plaintext images, and then the bootloader will encrypt them on-the-fly" (that is, given that the bootloader is compiled with encryption support in the first place, which I doubt to be the case with the bootloaders provided by espflash itself and embedded inside it with include_bytes! or whatever). By the way, this is done by the bootloader by it doing a lot of (dangerous!) e-fuse operations, like generating a random encryption key, burning it, then read+write protecting it and a few others.

This is by the way (and to my understanding - I'm still debugging the bootloader so I might very well be wrong!) a one time operation? If you can confirm my understanding is correct?

Like, once it is done, you can't just burn plaintext images and hope the bootloader would encrypt them? Or am I missing something here? My understanding is, once the initial bootloader-driven encryption is done, that's it end of story. The bootloader does not repeat the process anymore, and will assume all new images you upload to be pre-encrypted. and since you don't know the encryption key, you can't pre-encrypt them in the first place? That is, given that the UART downloader was left enabled (which depends on whether you are in DEVELOPMENT or RELEASE mode w.r.t. encryption).

I'm myself following the "other" process, where the encryption key is "externally" provided, and the payload (bootloader, part-table, app image, nvs keys etc.) are pre-encrypted. Seems to be what Espressif recommends as well?:
https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32s3/security/security-features-enablement-workflows.html

If you have an opinion on this stuff - please share. I'm about to enter production with my approach...

[4rzael]

@ivmarkov
Hey. Sorry about the slow answer there.

My understanding as to what @jessebraham demonstrates in #70, is just one way to do encryption, which is - "first flash plaintext images, and then the bootloader will encrypt them on-the-fly"

Yes. IIRC, this used to be the recommended/documented workflow if you didn't want to keep the keys.
The process you linked is equivalent (as long as you destroy the key afterwards), and a is bit faster in the production line (the other approach requires the esp encrypting the flash at first boot, which can easily take like 30s).

The goal of this PR, however, is to replicate esptool's encrypt feature, which is indeed only useful in development mode, where the bootloader's ability to perform encrypted-flashes is not disabled.
You're completely right about it being impossible to re-flash once all security options are enabled in release mode.

Basically, after encryption has been enabled (by whichever of the 2 methods you prefer), you can make the bootloader reflash a plaintext image, and encrypt it on the fly, allowing developers to flash firmwares on devices with encryption enabled (and thus as similar as possible to the end product).

[ivmarkov]

@ivmarkov Hey. Sorry about the slow answer there.

My understanding as to what @jessebraham demonstrates in #70, is just one way to do encryption, which is - "first flash plaintext images, and then the bootloader will encrypt them on-the-fly"

Yes. IIRC, this used to be the recommended/documented workflow if you didn't want to keep the keys. The process you linked is equivalent (as long as you destroy the key afterwards), and a is bit faster in the production line (the other approach requires the esp encrypting the flash at first boot, which can easily take like 30s).

The goal of this PR, however, is to replicate esptool's encrypt feature, which is indeed only useful in development mode, where the bootloader's ability to perform encrypted-flashes is not disabled. You're completely right about it being impossible to re-flash once all security options are enabled in release mode.

Basically, after encryption has been enabled (by whichever of the 2 methods you prefer), you can make the bootloader reflash a plaintext image, and encrypt it on the fly, allowing developers to flash firmwares on devices with encryption enabled (and thus as similar as possible to the end product).

@4rzael Thank you for your response. I completely agree with your statements. Ideally, espflash should not get in the way of supporting either of the two workflows (flashing pre-encrypted images or flashing plain-text images and then leaving it up to the bootloader to encrypt).

The goal of this PR, however, is to replicate esptool's encrypt feature, which is indeed only useful in development mode, where the bootloader's ability to perform encrypted-flashes is not disabled. You're completely right about it being impossible to re-flash once all security options are enabled in release mode.

This ^^^ is the one thing I'm struggling with. I can't see an "encrypt" command on esptool.py?

ivan@ivan-laptop:~/rp$ esptools tool -h
Tool esptool mounted as `/home/ivan/.cache/esptools/78c195b08bd57bf27167b323600d417a546d3393/esptool`
Executing `esptool`
usage: esptool [-h]
               [--chip {auto,esp8266,esp32,esp32s2,esp32s3beta2,esp32s3,esp32c3,esp32c6beta,esp32h2beta1,esp32h2beta2,esp32c2,esp32c6,esp32c61,esp32c5,esp32c5beta3,esp32h2,esp32p4}]
               [--port PORT] [--baud BAUD] [--port-filter PORT_FILTER]
               [--before {default_reset,usb_reset,no_reset,no_reset_no_sync}]
               [--after {hard_reset,soft_reset,no_reset,no_reset_stub}] [--no-stub] [--trace]
               [--override-vddsdio [{1.8V,1.9V,OFF}]] [--connect-attempts CONNECT_ATTEMPTS]
               {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,read_flash_sfdp,merge_bin,get_security_info,version}
               ...

esptool.py v4.8.1 - Espressif chips ROM Bootloader Utility

positional arguments:
  {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,read_flash_sfdp,merge_bin,get_security_info,version}
                        Run esptool.py {command} -h for additional help
    load_ram            Download an image to RAM and execute
    dump_mem            Dump arbitrary memory to disk
    read_mem            Read arbitrary memory location
    write_mem           Read-modify-write to arbitrary memory location
    write_flash         Write a binary blob to flash
    run                 Run application code in flash
    image_info          Dump headers from a binary file (bootloader or application)
    make_image          Create an application image from binary files
    elf2image           Create an application image from ELF file
    read_mac            Read MAC address from OTP ROM
    chip_id             Read Chip ID from OTP ROM
    flash_id            Read SPI flash manufacturer and device ID
    read_flash_status   Read SPI flash status register
    write_flash_status  Write SPI flash status register
    read_flash          Read SPI flash content
    verify_flash        Verify a binary blob against flash
    erase_flash         Perform Chip Erase on SPI flash
    erase_region        Erase a region of the flash
    read_flash_sfdp     Read SPI flash SFDP (Serial Flash Discoverable Parameters)
    merge_bin           Merge multiple raw binary files into a single file for later flashing
    get_security_info   Get some security-related data
    version             Print esptool version

optional arguments:
  -h, --help            show this help message and exit
  --chip {auto,esp8266,esp32,esp32s2,esp32s3beta2,esp32s3,esp32c3,esp32c6beta,esp32h2beta1,esp32h2beta2,esp32c2,esp32c6,esp32c61,esp32c5,esp32c5beta3,esp32h2,esp32p4}, -c {auto,esp8266,esp32,esp32s2,esp32s3beta2,esp32s3,esp32c3,esp32c6beta,esp32h2beta1,esp32h2beta2,esp32c2,esp32c6,esp32c61,esp32c5,esp32c5beta3,esp32h2,esp32p4}
                        Target chip type
  --port PORT, -p PORT  Serial port device
  --baud BAUD, -b BAUD  Serial port baud rate used when flashing/reading
  --port-filter PORT_FILTER
                        Serial port device filter, can be vid=NUMBER, pid=NUMBER, name=SUBSTRING
  --before {default_reset,usb_reset,no_reset,no_reset_no_sync}
                        What to do before connecting to the chip
  --after {hard_reset,soft_reset,no_reset,no_reset_stub}, -a {hard_reset,soft_reset,no_reset,no_reset_stub}
                        What to do after esptool.py is finished
  --no-stub             Disable launching the flasher stub, only talk to ROM bootloader. Some features will not be available.
  --trace, -t           Enable trace-level output of esptool.py interactions.
  --override-vddsdio [{1.8V,1.9V,OFF}]
                        Override ESP32 VDDSDIO internal voltage regulator (use with care)
  --connect-attempts CONNECT_ATTEMPTS
                        Number of attempts to connect, negative or 0 for infinite. Default: 7.
Status Ok(ExitStatus(unix_wait_status(0)))

See? And that's because esptool.py does not care whether its flashing an encrypted image or not.

Now, there is a command "encrypt_flash_data" on espsecure.py, if that's what you actually meant, but this command is for pre-encrypting the image on the host machine, which is not your use case anyway.

What am I missing?

[4rzael]

@ivmarkov Yeah, sorry, wasn't super clear. What I'm talking about is the --encrypt argument in esptool's write_flash command, which tells the bootloader to encrypt the supplied plaintext blob during the transfer.
While I haven't experimented with the workflow where you directly encrypt on the host, I agree with you that, AFAIK, it should be completely transparent to esptool.