Boosted performance, board.json metadata, more mimxrt, rp2, samd features

This release of MicroPython sees a boost to the overall performance of the VM and runtime. This is achieved by the addition of an optional cache to speed up general hash table lookups, as well as a fast path in the VM for the LOAD_ATTR opcode on instance types. The new configuration options are MICROPY_OPT_MAP_LOOKUP_CACHE and MICROPY_OPT_LOAD_ATTR_FAST_PATH. As part of this improvement the MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE option has been removed, which provided a similar map caching mechanism but with the cache stored in the bytecode, which made it not useful on bare metal ports. The new mechanism is measured to be at least as good as the old one, applies to more map lookups, has a constant RAM overhead, and applies to native code as well as bytecode.

These performance options are enabled on the esp32, mimxrt, rp2, stm32 and unix ports. For esp32 and mimxrt some code is also moved to RAM to further boost performance. On stm32, performance increases by about 20% for benchmarks that are heavy on name lookups, like misc_pystone.py and misc_raytrace.py. On esp32 performance can increase by 2-3x, and on mimxrt it is up to 6x.

All boards in all ports now have a board.json metadata file, which is used to automatically build firmware and generate a webpage for that board (among other possibilities). Auto-build scripts have been added for this purpose and they build all esp32, mimxrt, rp2, samd and stm32 boards. The generated output is available at https://micropython.org/download.

Support for FROZEN_DIR and FROZEN_MPY_DIR has been deprecated for some time and was finally removed in this release. Instead of these, FROZEN_MANIFEST can be used. The io.resource_stream() function is also removed, replaced by the pure Python version in micropython-lib.

The search order for importing frozen Python modules is now controlled by the ".frozen" entry in sys.path. This string is added by default in the second position in sys.path. User code should adjust sys.path depending on the desired behaviour. Putting ".frozen" first in sys.path will speed up importing frozen modules.

A bug in multiple precision integers with bitwise of -0 was fixed in commit 2c139bb.

The platform module has been added to allow querying the compiler and underlying SDK/HAL/libc version. This is enabled on esp32, mimxrt and stm32 ports.

The mpremote tool now supports seek, flush, mkdir and rmdir on PC-mounted filesystems. And a help command has been added.

The documentation has seen many additions and improvements thanks (for a second time) to the Google Season of Docs project. The rp2 documentation now includes a reference for PIO assembly instructions, a PIO quick reference and a PIO tutorial. The random and stm modules have been documented, along with sys.settrace, manifest.py files and mpremote. There is also now more detail about the differences between MicroPython and standard Python 3.5 and above.

The esp32 port sees support for ESP32-S3 SoCs, and new boards GENERIC_S3, ESP32_S2_WROVER, LOLIN_S2_MINI, LOLIN_S2_PICO and UM_FEATHERS2NEO. The PWM driver has been improved and now supports all PWM timers and channels, and the duty_u16() and duty_ns() methods, and it keeps the duty constant when changing frequency. The machine.bitstream() function has been improved to use RMT, with an option to select the original bit-banging implementation.

The mimxrt port gained new hardware features: SDRAM and SD card support, as well as network integration with a LAN driver. The machine.WDT class was added along with the machine.reset_cause(), machine.soft_reset(), machine.unique_id() add machine.bitstream() functions. DHT sensor support was added, and f-strings were enabled.

The rp2 port now has support for networking, and bluetooth using NimBLE. The Nina-W10 WiFi/BT driver is fully integrated and supported by the new Arduino Nano RP2040 connect board. I2S protocol support is added along with a machine.bitstream() driver and DHT sensor support. The PWM driver had a bug fix with the accuracy of setting/getting the frequency, and the duty value is now retained when changing the frequency.

On the samd port there is now support for the internal flash being a block device, and for filesystems and the os module. Pin and LED classes have been implemented. There are more time functions, more Python features enabled, and the help() function is added. SEEED_WIO_TERMINAL and SEEED_XIAO board definitions are now available.

The stm32 port now has support for F427, F479 and H7A3(Q)/H7B3(Q) MCUs, and new board definitions for VCC_GND_H743VI, OLIMEX_H407, MIKROE_QUAIL, GARATRONIC_PYBSTICK26_F411, STM32H73B3I_DK. A bug was fixed in the SPI driver where a SPI transfer could fail if the CYW43 WiFi driver was also active at the same time.

On the windows port the help() function has been enabled, and support for build variants added, to match the unix port.

The zephyr port upgraded Zephyr to v2.7.0.

The change in code size since the previous release for various ports is (absolute and percentage change in the text section):

   bare-arm:  -1520  -2.605%
minimal x86:  -2256  -1.531%
   unix x64:   -457  -0.089%
unix nanbox:   -925  -0.204%
      stm32:   +312  +0.079% PYBV10
     cc3200:   -176  -0.096%
    esp8266:   +532  +0.076% GENERIC
      esp32: +27096  +1.820% GENERIC
        nrf:   -212  -0.121% pca10040
        rp2:  +9904  +2.051% PICO
       samd: +35332 +33.969% ADAFRUIT_ITSYBITSY_M4_EXPRESS

The changes that dominate these numbers are:

• bare-arm, minimal: use of new MICROPY_CONFIG_ROM_LEVEL_MINIMUM option and subsequent disabling of remaining optional features
• unix, cc3200, nrf: general code size reductions of the core
• stm32: performance improvements, addition of platform module
• esp8266: enabling f-strings
• esp32: use of -O2 instead of -Os
• rp2: machine.I2S and other new hardware features
• samd: filesystem support and other new hardware features

Thanks to everyone who contributed to this release: Alan Dragomirecký, Alexey Shvetsov, Andrew Leech, Andrew Scheller, Antoine Aubert, Boris Vinogradov, Chris Boudacoff, Chris Fiege, Christian Decker, Damien George, Daniel Gorny, Dave Hylands, David Michieli, Emilie Feral, Frédéric Pierson, gibbonsc, Henk Vergonet, iabdalkader, Ihor Nehrutsa, Jan Hrudka, Jan Staal, jc_.kim, Jim Mussared, Jonathan Hogg, Laurens Valk, leo chung, Lorenzo Cappelletti, Magnus von Wachenfeldt, Matt Trentini, Matt van de Werken, Maureen Helm, Michael Bentley, Michael Buesch, Mike Causer, Mike Teachman, Mike Wadsten, Ned Konz, NitiKaur, oli, patrick, Patrick Van Oosterwijck, Peter Boin, Peter Hinch, Peter van der Burg, Philipp Ebensberger, Pooya Moradi, retsyo, robert-hh, roland van straten, Scott Armitage, Sebastian Wicki, Seon Rozenblum, Sergei Silnov, Simon Baatz, Stewart Bonnick, stijn, Tobias Thyrrestrup, Tomas Vanek, YoungJoon Chun.

What follows is a detailed list of changes, generated from the git commit history, and organised into sections.

Main components

all:

• remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE
• update Python formatting to latest Black version 21.12b0
• remove support for FROZEN_DIR and FROZEN_MPY_DIR

py core:

• parse: simplify parse nodes representing a list
• emitnative: ensure load_subscr does not clobber existing REG_RET
• mpconfig.h: define initial templates for "feature levels"
• vm: add a fast path for LOAD_ATTR on instance types
• map: add an optional cache of (map+index) to speed up map lookups
• builtinimport: forward all debug printing to MICROPY_DEBUG_PRINTER
• add wrapper macros so hot VM functions can go in fast code location
• runtime: fix crash when exc new doesn't return an exc instance
• mpconfig.h: define the "extra" feature level
• mpconfig.h: revert MICROPY_REPL_INFO to disabled at all levels
• gc: add hook to run code during time consuming GC operations
• showbc: print unary-op string when dumping bytecode
• modsys: replace non-ASCII quote char with ASCII char
• runtime: allow types to use both .attr and .locals_dict
• lexer: support nested [] and {} characters within f-string params
• objfun.h: remove obsolete comments about entries in extra_args
• builtinimport: refactor module importing
• showbc: fix printing of raw bytecode header on nanbox builds
• modio: remove io.resource_stream function
• only search frozen modules when '.frozen' is found in sys.path
• mkrules.cmake: set frozen preprocessor defs early
• runtime: allow initialising sys.path/argv with defaults
• mpstate.h: only include sys.path/argv objects in state when enabled
• mpz: fix bugs with bitwise of -0 by ensuring all 0's are positive
• qstr: reset mpstate.qstr_last_chunk before raising an error
• modbuiltins: add additional macro for extending builtins
• mpconfig.h: define MICROPY_PY_USSL_FINALISER only if not defined

extmod:

• machine_i2c: make SoftI2C configurable via macro option
• machine_spi: make SoftSPI configurable via macro option
• modonewire: make _onewire module configurable via macro option
• machine_pwm: factor out machine.PWM bindings to common code
• move modnetwork and modusocket from stm32 to extmod
• modnetwork: add STA_IF and AP_IF constants
• modnetwork: add extended socket state
• modusocket: add read/write stream methods to socket object
• modnetwork: define network interfaces in port config files
• network_cyw43: make consistent use of STA and AP constants
• modnetwork: remove STM32 references
• modnetwork: remove modnetwork socket u_state member
• mpbthci.h: add mp_bluetooth_hci_uart_any prototype
• nimble: add nimble CMake fragment file
• add platform module
• moduplatform: improve implementation for PC ports
• vfs_posix_file: support MP_STREAM_POLL in vfs_posix_file_ioctl
• modbluetooth: add connection interval to gap_connect
• nimble: update...

F-strings, new machine.I2S class, ESP32-C3 support and LEGO_HUB_NO6 board

This release of MicroPython adds support for f-strings (PEP-498), with a few limitations compared to normal Python. F-strings are essentially syntactic sugar for "".format() and make formatting strings a lot more convenient. Other improvements to the core runtime include pretty printing OSError when it has two arguments (an errno code and a string), scheduling of KeyboardInterrupt on the main thread, and support for a single argument to the optimised form of StopIteration.

In the machine module a new I2S class has been added, with support for esp32 and stm32 ports. This provides a consistent API for transmit and receive of audio data in blocking, non-blocking and asyncio-based operation. Also, the json module has support for the "separators" argument in the dump and dumps functions, and framebuf now includes a way to blit between frame buffers of different formats using a palette. A new, portable machine.bitstream function is also added which can output a stream of bits with configurable timing, and is used as the basis for driving WS2812 LEDs in a common way across ports.

There has been some restructuring of the repository directory layout, with all third-party code now in the lib/ directory. And a new top-level directory shared/ has been added with first-party code that was previously in lib/ moved there.

The docs have seen further improvement with enhancements and additions to the rp2 parts, as well as a new quick reference for the zephyr port. The terms master/slave have been replaced with controller/peripheral, mainly relating to I2C and SPI usage. And u-module references have been replaced with just the module name without the u-prefix to help clear up the intended usage of modules in MicroPython.

For the esp8266 and esp32 ports, hidden networks are now included in WLAN scan results. On the esp32 the RMT class is enhanced with idle_level and write_pulses modes. There is initial support for ESP32-C3 chips with GENERIC_C3 and GENERIC_C3_USB boards.

The javascript port has had its Makefile and garbage collector implementation reworked so it compiles and runs with latest the Emscripten using asyncify.

The mimxrt port sees the addition of hardware I2C and SPI support, as well as some additional methods to the machine module. There is also support for Hyperflash chips.

The nrf port now has full VFS storage support, enables source-line on traceback, and has .mpy features consistent with other ports.

For the rp2 port there is now more configurability for boards, and more boards added.

The stm32 port has a new LEGO_HUB_NO6 board definition with detailed information how to get this LEGO Hub running stock MicroPython. There is also now support to change the CPU frequency on STM32WB MCUs. And USBD_xxx descriptor options have been renamed to MICROPY_HW_USB_xxx.

Thanks to everyone who contributed to this release: Amir Gonnen, Andrew Scheller, Bryan Tong Minh, Chris Wilson, Damien George, Daniel Mizyrycki, David Lechner, David P, Fernando, finefoot, Frank Pilhofer, Glenn Ruben Bakke, iabdalkader, Jeff Epler, Jim Mussared, Jonathan Hogg, Josh Klar, Josh Lloyd, Julia Hathaway, Krzysztof Adamski, Matúš Olekšák, Michael Weiss, Michel Bouwmans, Mike Causer, Mike Teachman, Ned Konz, NitiKaur, oclyke, Patrick Van Oosterwijck, Peter Hinch, Peter Züger, Philipp Ebensberger, robert-hh, Roberto Colistete Jr, Sashkoiv, Seon Rozenblum, Tobias Thyrrestrup, Tom McDermott, Will Sowerbutts, Yonatan Goldschmidt.

What follows is a detailed list of changes, generated from the git commit history, and organised into sections.

Main components

all:

• fix signed shifts and NULL access errors from -fsanitize=undefined
• update to point to files in new shared/ directory

py core:

• mpstate: make exceptions thread-local
• mpstate: schedule KeyboardInterrupt on main thread
• mperrno: add MP_ECANCELED error code
• makeqstrdefs.py: don't include .h files explicitly in preprocessing
• mark unused arguments from bytecode decoding macros
• objexcept: pretty print OSError also when it has 2 arguments
• makeversionhdr: add --tags arg to git describe
• vm: simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode
• objexcept: make mp_obj_exception_get_value support subclassed excs
• support single argument to optimised MP_OBJ_STOP_ITERATION
• introduce and use mp_raise_type_arg helper
• modsys: optimise sys.exit for code size by using exception helpers
• objexcept: make mp_obj_new_exception_arg1 inline
• obj: fix formatting of comment for mp_obj_is_integer
• emitnative: reuse need_reg_all func in need_stack_settled
• emitnative: ensure stack settling is safe mid-branch
• runtime: fix bool unary op for subclasses of native types
• builtinimport: fix condition for including do_execute_raw_code()
• mkrules: automatically build mpy-cross if it doesn't exist
• implement partial PEP-498 (f-string) support
• lexer: clear fstring_args vstr on lexer free
• mkrules.mk: do submodule sync in "make submodules"

extmod:

• btstack: add missing call to mp_bluetooth_hci_uart_deinit
• btstack: check that BLE is active before performing operations
• uasyncio: get addr and bind server socket before creating task
• axtls-include: add axtls_os_port.h to customise axTLS
• update for move of crypto-algorithms, re1.5, uzlib to lib
• moduselect: conditionally compile select()
• nimble: fix leak in l2cap_send if send-while-stalled
• btstack/btstack.mk: use -Wno-implicit-fallthrough, not =0
• utime: always invoke mp_hal_delay_ms when >= to 0ms
• modbluetooth: clamp MTU values to 32->UINT16_MAX
• nimble: allow modbluetooth binding to hook "sent HCI packet"
• nimble: add "memory stalling" mechanism for l2cap_send
• uasyncio: in open_connection use address info in socket creation
• modujson: add support for dump/dumps separators keyword-argument
• modlwip: fix close and clean up of UDP and raw sockets
• modbluetooth: add send_update arg to gatts_write
• add machine.bitstream
• modframebuf: enable blit between different formats via a palette

lib:

• tinyusb: update to version 0.10.1
• pico-sdk: update to version 1.2.0
• utils/stdout_helpers: make mp_hal_stdout_tx_strn_cooked efficient
• axtls: switch to repo at micropython/axtls
• axtls: update to latest axtls 2.1.5 wih additional commits
• re1.5: move re1.5 code from extmod to lib
• uzlib: move uzlib code from extmod to lib
• crypto-algorithms: move crypto-algorithms code from extmod to lib
• update README's based on contents of these dirs

drivers:

• neopixel: add common machine.bitstream-based neopixel module
• neopixel: optimize fill() for speed
• neopixel: reduce code size of driver
• cyw43: fix cyw43_deinit so it can be called many times in a row
• cyw43: make wifi join fail if interface is not active

mpy-cross:

• disable stack check when building with Emscripten

Support components

docs:

• library: document new esp32.RMT features and fix wait_done
• library: warn that ustruct doesn't handle spaces in format strings
• esp8266/tutorial: change flash mode from dio to dout
• replace master/slave with controller/peripheral in I2C and SPI
• rp2: enhance quickref and change image to Pico pinout
• rp2: update general section to give a brief technical overview
• library/utime.rst: clarify behaviour and precision of sleep ms/us
• library/uasyncio.rst: document stream readexactly() method
• library/machine.I2S.rst: fix use of sd pin in examples
• zephyr: add quick reference for the Zephyr port
• library/zephyr: add libraries specific to the Zephyr port
• templates: add unix and zephyr quickref links to top-index
• rename ufoo.rst to foo.rst
• replace ufoo with foo in all docs
• library/index.rst: clarify module naming and purpose
• library/builtins.rst: add module title
• library/network.rst: simplify socket import
• add docs for machine.bitstream and neopixel module
• library: fix usage of :term: for frozen module reference
• esp8266: use monospace for software tools
• reference: mention that slicing a memoryview causes allocation

examples: no changes specific to this component/port

tests:

• extmod: make uasyncio_heaplock test more deterministic
• cpydiff/modules_struct_whitespace_in_format: run black
• extmod/ujson: add tests for dump/dumps separators argument
• run-multitests.py: add broadcast and wait facility
• multi_bluetooth/ble_subscribe.py: add test for subscription
• extmod/vfs_fat_finaliser.py: ensure alloc at never-used GC blocks
• basics: split f-string debug printing to separate file with .exp
• pybnative: make while.py test run on boards without pyb.delay

tools:

• autobuild: add scripts to build release firmware
• remove obsolete build-stm-latest.sh script
• ci.sh: run apt-get update in ci_powerpc_setup
• makemanifest.py: allow passing flags to mpy-tool.py
• autobuild: add mimxrt port to build scripts for nightly builds
• pyboard.py: add cmd-line option to make soft reset configurable
• mpremote: swap order of PID and VID in connect-list output
• ci.sh: build unix dev variant as part of macOS CI
• ci.sh: build GENERIC_C3 board as part of esp32 CI
• autobuild: use separate IDF version to build newer esp32 SoCs
• autobuild: add FeatherS2 and TinyS2 to esp32 auto builds
• mpremote: add seek whence for mounted files
• mpremote: raise OSError on unsupported RemoteFile.seek
• autobuild: add the MIMXRT1050_EVKB board to the daily builds
• ci.sh: add mpy-cross build to nrf port
• codeformat.py: include ports/nrf/modules/nrf in code formatting
• gen-cpydiff.py: don't rename foo to ufoo in diff output
• autobuild: add auto build for Silicognition wESP32
• mpremote: fix connect-list in case VID/PID are None
• mpremote: add "devs" shortcut for "connect list"
• mpremote: remove support for pyb.USB_VCP in/out specialisation
• autobuild: don't use "-B" for make, it's already a fresh build
• pyboard.py: move --no...

New mpremote tool, and the mimxrt port gets Pin, ADC, UART, RTC and VFS

This release of MicroPython includes a new command-line tool called "mpremote", which is intended to be the main way to remotely control a MicroPython-based device from the command line. It features a serial terminal, filesystem access, support to mount a local directory on the remote device, and a macro language to define custom commands. This tool can be installed from PyPI via "pip3 install mpremote", and it works on Linux, Windows and Mac. As part of this, improvements were made to pyboard.py including opening serial ports in exclusive mode to more easily manage multiple devices.

In the Python core, OSError exceptions now support the ".errno" attribute, and an option was added to compile MicroPython without error messages to further reduce code size where needed. The REPL was improved so that it does not tab-complete private methods (those starting with underscore, if no underscore has been typed yet), and it also now tab completes built-in module names after "import" is typed.

There has been a minor breaking change to a relative import exception: what was previously a ValueError was changed to ImportError, following the same change in CPython. See commit 53519e3.

In the extmod components, uctypes has a fix for the size and offset calculation for ARRAY of FLOAT32, uhashlib now raises an exception if a hash is reused after digest is called, and urandom supports passing 0 to getrandbits (following the CPython change). In uasyncio, the readinto method is added to the Stream class, and two race conditions were fixed: one with start_server and wait_closed, and the other with cancelling a task waiting on finished task; see de2e081 and 514bf1a respectively.

The esp32 port now supports specifying FROZEN_MANIFEST with new CMake build system, has NeoPixel support on GPIO32 and GPIO33, network.LAN support in IDF v4.1 and above, and a new "reconnects" option in the WLAN STA interface to configure how many (if any) reconnection attempts are made if the WiFi goes down.

Many features have been added to the mimxrt port, including: VFS filesystem support with internal flash storage, Pin, Pin.irq and ADC support, UART, SoftI2C and SoftSPI bus support, Timer and RTC classes, and floating point numbers.

The rp2 port now has the machine.RTC class to configure the RTC, as well as new board definition files for SparkFun's Thing Plus RP2040 and Pro Micro boards.

The stm32 port now supports static soft timers with a C-based callback, and mboot has been made more configurable, in particular the LEDs and reset mode selection can now be fully customised by a board. Two new boards have been added: VCC_GND_F407VE and VCC_GND_F407ZG. A bug fix was made to the SDIO driver to make sure DMA doesn't turn off mid-transfer; this affected WLAN operation when certain SPI buses were being used. See commit a96afae for details. Pin configuration of UART has been modified so pull-up is now configured only on RX and CTS, not TX and RTS; see 748339b. The USB_VCP class has a new irq method to set a callback on USB data RX events. The Ethernet driver now supports low-power mode, and has a fix so the link status is reported correctly.

On the zephyr port, scheduled callbacks are now run at idle REPL and during sleeps, and there is an initial ubluetooth module which supports BLE scanning and advertising. Configuration is provided for the nucleo_wb55rg board.

What follows is a detailed list of changes, generated from the git commit history, and organised into sections.

Thanks to everyone who contributed to this release!

Main components

all:

• rename mp_keyboard_interrupt to mp_sched_keyboard_interrupt
• replace busses with buses

py core:

• objexcept: support errno attribute on OSError exceptions
• add option to compile without any error messages at all
• dynruntime.h: add mp_obj_get_array() function
• profile: use mp_handle_pending() to raise pending exception
• scheduler: add mp_sched_exception() to schedule a pending exception
• scheduler: add optional port hook for when something is scheduled
• runtime: remove commented-out code from mp_deinit()
• scheduler: add missing MICROPY_WRAP_MP_SCHED_EXCEPTION usage
• repl: filter private methods from tab completion
• repl: enter four spaces when there are no matches
• repl: refactor autocomplete to reduce nesting
• repl: refactor autocomplete, extracting reusable parts
• repl: autocomplete builtin modules
• gc: make gc_lock_depth have a count per thread
• mkenv.mk: don't emit info about BUILD_VERBOSE if it's set
• objarray: prohibit comparison of mismatching types
• objarray: implement more/less comparisons for array
• objarray: use mp_obj_memoryview_init helper in mp_obj_new_memoryview
• objarray: fix constructing a memoryview from a memoryview
• nlraarch64: add underscore prefix to function symbols for Darwin ABI
• nlrx64: correct the detection of Darwin ABI
• asmx64: support use of top 8 regs in src_r64 argument
• emitnative: fix x86-64 emitter to generate correct 8/16-bit stores
• mkrules.cmake: add MPY_LIB_DIR and BOARD_DIR to makemanifest call
• asmarm: use builtin func to flush I- and D-cache on ARM 7 archs
• compile: raise an error on async with/for outside an async function
• gc: access the list of root pointers in an asan-compatible way
• repl: don't read past the end of import_str
• builtinimport: change relative import's ValueError to ImportError
• emitglue: always flush caches when assigning native ARM code
• stackctrl: prevent unused-var warning when stack checking disabled
• gc: only use no_sanitize_address attribute for GCC 4.8 and above

extmod:

• uasyncio: use .errno instead of .args[0] for OSError exceptions
• remove old comments used for auto-doc generation
• moductypes: remove double blank lines and debugging printf's
• moductypes: replace numbers with macro constants
• moductypes: fix size and offset calculation for ARRAY of FLOAT32
• moduhashlib: put hash obj in final state after digest is called
• modurandom: add error message when getrandbits has bad value
• modurandom: support an argument of bits=0 to getrandbits
• uasyncio: fix start_server and wait_closed race condition
• uasyncio: add readinto() method to Stream class
• uasyncio: fix race with cancelled task waiting on finished task
• nimble: remove TODO comment about notify_custom freeing om

lib:

• utils: remove unused PYEXEC_SWITCH_MODE from pyexec.h
• utils: add ARM semihosting utility functions
• lwip: switch to use GitHub mirror repo
• mbedtls: switch to currently latest commit of LTS branch v2.16

drivers:

• sdcard: add sleep_ms(1) delay in SDCard.readinto sync loop
• cyw43/cyw43_ctrl: use new sdio enable API functions
• cyw43/cywbt: add compile option for RF switch
• cyw43/cywbt: remove hard-coded UART6 alternate function setting
• display/ssd1306.py: add rotate method
• display/ssd1306.py: add support for 72x40 displays

mpy-cross: no changes specific to this component/port

Support components

docs:

• library/machine: specify initial machine.PWM class
• library/machine: add machine.bootloader docs
• esp8266: add note about simultaneous use of STA_IF and AP_IF
• esp8266: add instructions on entering programming mode manually
• esp8266: clarify limitations of SSL in esp8266 and fix typos
• fix some spelling mistakes
• pyboard: fix typo in pyb.Switch tutorial
• esp32: add UART to quickref
• esp32: add WDT to quickref
• esp32: add SDCard to quickref
• esp8266: add WDT to quickref
• library: add initial API reference for rp2 module and its classes
• library/rp2.rst: fix typo overriden->overridden
• esp32: add APA106 to quickref
• esp32: mention Signal in GPIO section of quickref
• esp8266: mention Signal in GPIO section of quickref
• esp8266: add SSD1306 to quickref and tutorial
• library: clarify what type of algorithm is implemented in heapq
• library: add a blank line to fix formatting for ussl docs
• library/pyb.Pin.rst: update the arguments for Pin.init()
• rp2: add skeleton docs for the rp2 port
• library/machine.RTC.rst: document datetime method and fix ex code
• esp32: document WLAN "reconnects" config option

examples: no changes specific to this component/port

tests:

• use .errno instead of .args[0] for OSError exceptions
• run-multitests.py: provide some convenient serial device shorcuts
• multi_bluetooth: add performance test for gatt char writes
• thread: make stress_create.py test run on esp32
• thread: make stress_aes.py test run on bare-metal ports
• thread: make exc1,exit1,exit2,stacksize1,start1 tests run on rp2
• run-perfbench.py: fix native feature check
• run-multitests.py: flush stdout for each line of trace output
• run-tests.py: parallelize running tests by default
• cpydiff: add test and workaround for function.module attr
• make float and framebuf tests skip or run on big-endian archs
• extmod/btree_gc.py: close the database to avoid a memory leak
• basics: split out literal tests that raise SyntaxWarning on CPy
• run-multitests.py: allow to work without sys.stdout on target
• multi_bluetooth/ble_gap_advertise.py: allow to work without set
• unix: add ffi test for integer types
• cpydiff: add test for array constructor with overflowing value
• float: make bytes/bytearray construct tests work with obj repr C

tools:

• metrics.py: add rp2 port to table of ports that can be built
• upip.py: use .errno instead of .args[0] for OSError exceptions
• pyboard.py: support opening serial port in exclusive mode
• gen-cpydiff.py: fix formatting of doc strings for new Black
• makemanifest.py: show directory name if there is a FreezeError
• mpy_ld.py: support R_X86_64_GOTPCREL reloc for x86-64 arch
• pydfu.py: remove default VID/PID values...

The esp32 port moves to CMake and has S2 support, new features for rp2

This release of MicroPython adds general support in the core for using CMake as a build system. The rp2 port is consolidated to use the new CMake files, and the esp32 and zephyr ports have switched to build as pure CMake projects. These three ports have SDKs which are built around CMake and this change should make them easier to maintain and use.

As part of this work, CMake based ports now have support for user C modules. Authors of user C modules should now provide both .mk and .cmake configuration files (following the documentation and examples).

A bug was fixed in the multiple precision integer library, an arithmetic overflow in the long division routine. Prior to this fix certain integer divisions would take excessive time and produce incorrect results. See commit 0a59938 for details. There was also a fix for regular expressions, to check and report byte overflow errors when compiling expressions. See commit 172fb52.

In the uasyncio library, a new MicroPython extension has been added, ThreadSafeFlag, which can be set from outside the asyncio event loop, such as other threads, IRQs or scheduler context. It allows preemptive code like IRQs to signal asyncio tasks, which are by nature cooperative (non-preemptive). asyncio.current_task() has also been added, with the same semantics as CPython.

As mentioned above, the esp32 port has switched to a full CMake-based project, and traditional make capability has been removed (although a simple helper Makefile remains to keep top-level build/deploy commands consistent with other ports). Because of the move to CMake, network.LAN support has been removed, to be added back in the future (use a prior release if LAN is needed). Basic support for Non-Volatile-Storage is added to the esp32 module. And there is also preliminary support for ESP32S2 SoCs and USB, with a GENERIC_S2 board defined.

On the mimxrt and samd ports, USB CDC TX handling has been fixed so that it now works reliably.

The rp2 port has had many more core Python features enabled, along with the enabling of ubinascii.crc32(), the uos.VfsFat and machine.Signal classes, and the uerrno module. uos.urandom() has been added and machine.freq() can now change the CPU clock frequency. The machine.UART class now has support for timeout/timeout_char, inverted TX/RX lines, and buffered TX/RX with configurable sized buffers. For PIO, StateMachine has added restart(), rx_fifo() and tx_fifo() helper functions, and support for FIFO joining. There is now support for user C modules (via CMake) and for building different board configurations (the default remains the PICO board). USB reliability has been improved.

For the stm32 port, there is now more configuration options for boards, such as USBD VID/PID and fine-grained selection of modules. The UART class now supports LPUART on L0, L4, H7 and WB MCUs. WB MCUs have a fix for a race condition accessing the BLE ACL free buffer list, and a workaround for a low-level BLE bug.

The zephyr port has been updated to use zephyr v2.5.0, and now builds MicroPython as a CMake target.

What follows is a detailed list of changes, generated from the git commit history, and organised into sections.

Main components

all:

• rename BYTES_PER_WORD to MP_BYTES_PER_OBJ_WORD
• add .git-blame-ignore-revs for fixing up git blame output

py core:

• gc: don't include mpconfig.h and misc.h in gc.h
• remove BITS_PER_WORD definition
• rename BITS_PER_BYTE to MP_BITS_PER_BYTE
• rename WORD_MSBIT_HIGH to MP_OBJ_WORD_MSBIT_HIGH
• gc: change include of stdint.h to stddef.h
• mpz: fix overflow of borrow in mpn_div
• add core cmake rule files
• expand lists in core cmake custom commands
• mkrules.cmake: rename QSTR_DEFS variables to QSTRDEFS
• mkrules.cmake: add MICROPY_QSTRDEFS_PORT to qstr build process
• nlr: implement NLR for AArch64
• nlrx64: fix typo in comment
• vm: for tracing use mp_printf, and print state when thread enabled
• rename remaining object types to be of the form mp_type_xxx
• py.cmake: move qstr helper code to micropy_gather_target_properties
• py.cmake: introduce MICROPY_INC_CORE as a list with core includes
• profile: resolve name collision with STATIC unset
• runtime: make sys.modules preallocate to a configurable size

extmod:

• vfs_posix_file: allow closing an already closed file
• btstack: add HCI trace debugging option in btstack_hci_uart
• btstack: add stub functions for passkey, l2cap bindings
• btstack: enable SYNC_EVENTS, PAIRING_BONDING by default
• uasyncio: add asyncio.current_task()
• add core cmake rule files
• nimble: ensure handle is set on read error
• moduselect: fix unsigned/signed comparison for timeout!=-1
• uasyncio: add ThreadSafeFlag
• nimble/hal/hal_uart: fix HCI_TRACE format specifiers
• modbluetooth: allow NimBLE to use Zephyr static address
• modussl: fix ussl read/recv/send/write errors when non-blocking
• btstack: use MICROPY_HW_BLE_UART_BAUDRATE for first UART init
• modbluetooth: separate enabling of "client" from "central"
• extmod.cmake: add modonewire.c to MICROPY_SOURCE_EXTMOD list
• modbluetooth: free temp arrays in gatts register services
• re1.5: check and report byte overflow errors in _compilecode
• extmod.cmake: add support to build btree module with CMake

lib:

• tinyusb: update to version 0.8.0
• mbedtls: switch to currently latest commit of LTS branch v2.16
• utils/gchelper_generic: implement AArch64 support
• pico-sdk: update to latest version 1.1.0

drivers: no changes specific to this component/port

mpy-cross: no changes specific to this component/port

Support components

docs:

• library/uasyncio.rst: add docs for ThreadSafeFlag
• develop/cmodules.rst: document C-modules and micropython.cmake
• develop: improve user C modules to properly describe how to build

examples:

• usercmodule: add micropython.cmake to the C and CPP examples
• usercmodules: simplify user C module enabling
• embedding: fix example so it compiles again

tests:

• extmod/vfs_posix.py: add more tests for VfsPosix class
• extmod: add test for ThreadSafeFlag
• multi_bluetooth: add basic performance tests
• rename run-tests to run-tests.py for consistency
• run-tests.py: reformat with Black
• multi_bluetooth: skip tests when BLE features are unsupported
• extmod/vfs_fat_fileio2.py: close test file at end of test
• run-tests.py: provide more info if script run via pyboard crashes
• feature_check: check for lack of pass result rather than failure
• net_inet: add 'Strict-Transport-Security' to exp file

tools:

• add filesystem action examples to pyboard.py help
• ci.sh: change esp32 CI to work with idf.py and IDF v4.0.2
• makemanifest.py: allow passing option args to include()
• ci.sh: update zephyr docker image to v0.11.13
• verifygitlog.py: show required format regexp in error message
• pydfu.py: support DFU files with elements of zero size
• ci.sh: add CI for CMake USER_C_MODULE support
• ci.sh: build user C modules for esp32
• metrics.py: fix esp32 output filename due to move to CMake
• ci.sh: build esp32 using IDF v4.0.2 and v4.3

The ports

all ports:

• remove def of MP_PLAT_PRINT_STRN if it's the same as the default
• update to build with new tinyusb

bare-arm port:

• clean up the code, make it run on an F405, and add a README

cc3200 port: no changes specific to this component/port

esp8266 port:

• modules: fix fs_corrupted() to use start_sec not START_SEC

esp32 port:

• add support to build using IDF with cmake
• esp32_rmt: don't do unnecessary check for unsigned less than zero
• add explicit initialisers to silence compiler warnings
• remove traditional "make" capability
• boards: remove old IDF v3 sdkconfig values
• boards: enable BLE on all boards
• README: update based on new IDF v4 cmake build process
• add support to build with ESP-IDF v4.1.1
• add support to build with ESP-IDF v4.2
• remove obsolete IDF v3 code wrapped in MICROPY_ESP_IDF_4
• modsocket: remove unix socket error code translation
• set MICROPY_USE_INTERNAL_ERRNO=0 to use toolchain's errno.h
• boards: enable size optimisation for builds
• add support to build with ESP-IDF v4.3 pre-release
• add basic support for Non-Volatile-Storage in esp32 module
• make machine.soft_reset() work in main.py and reset_cause()
• define MICROPY_QSTRDEFS_PORT to include special qstrs
• Makefile: specify port and baud on erase_flash command
• machine_hw_spi: use default pins when making SPI if none given
• restore USER_C_MODULE support with new CMake build system
• fix multiple definition errors with mp_hal_stdout_tx functions
• enable btree module
• modsocket: correctly handle poll/read of unconnected TCP socket
• add initial support for ESP32S2 SoCs
• add support for USB with CDC ACM
• boards: add GENERIC_S2 board definition
• machine_pin: use rtc_gpio_deinit instead of gpio_reset_pin

javascript port: no changes specific to this component/port

mimxrt port:

• fix USB CDC handling so it works reliably
• boards: add MIMXRT1050_EVK board, based on MIMXRT1060_EVK
• enable CPYTHON_COMPAT, PY_ASYNC_AWAIT, PY_ATTRTUPLE options

minimal port: no changes specific to this component/port

nrf port:

• drivers/usb: add USBD_IRQHandler which calls tud_int_handler

pic16bit port: no changes specific to this component/port

powerpc port: no changes specific to this component/port

qemu-arm port: no changes specific to this component/port

rp2 port:

• machine_adc: only initialise the ADC periph if not already enabled
• micropy_rules.cmake: fix makemoduledefs vpath to work with abs path
• use local tinyusb instead of the one in pico-sdk
• enable MICROPY_PY_UBINASCII_CRC32 to get ubinascii.crc32()
• enable VfsFat class for FAT filesystem support
• machine_uart: add timeout/timeout_char to read and write
• machine_u...

New rp2 port, BLE pairing, bonding and l2cap APIs, and docs on internals

This release of MicroPython sees the addition of a new "rp2" port for the new Raspberry Pi RP2040 microcontroller, a dual-core Cortex-M0+ MCU with programmable IO. This port is built on the pico-sdk and MicroPython sits very close to the bare metal. The programmable IO is supported via the @rp2.asm_pio decorator which allows writing inline assembler to be run on the PIO state machines.

The ubluetooth module has seen significant improvements, fixes and additions. BLE events are now synchronous on unix and stm32 so that user Python callbacks are called directly from the BLE stack (from NimBLE). This helps with the new pairing and bonding API. There is also a new l2cap API for much faster raw data transfer between BLE devices.

Many new sections have been added to the existing "Developing and building MicroPython" chapter to make it all about the internals of MicroPython, including details about the MicroPython compiler, and how to port to a new system. This documentation work was done as part of Google's Season of Docs 2020.

The native emitter and inline assembler now fully support ARMv6M, for Cortex-M0+ CPUs. There is also support added for C++ user modules on unix, stm32 and esp32. And reproducible builds can now be achieved by setting SOURCE_DATE_EPOCH in the shell build environment. Qstr preprocessing is now done in parallel to speed up building, and there is a new tools/ci.sh script with all functionality to run the continuous integration tests.

The machine.I2C and machine.SPI classes have been modified so they no longer construct both software- and hardware-based peripheral instances. Such construction is now split to explicit soft and non-soft types by the introduction of machine.SoftI2C and machine.SoftSPI classes. See below for more information.

The wait_for function in uasyncio now handles cancellation correctly, VfsLfs supports mounting in read-only mode, and the urandom module will randomise its seed on import on stm32, esp8266, esp32 and rp2 ports.

A new raw-paste mode is added to the REPL which includes flow control between the device and host, as well as a mechanism where the device pulls in data from the host as it parses the input. This makes pasting code faster, more reliable and use less memory. This feature is supported on all ports that have a raw REPL and still supports previous REPL behaviour. See the documentation for more detail on this raw-paste mode.

The stm32 port sees improved rfcore support for WB MCUs, including flash locking for writes when BLE is active, and a script for automatic update of FUS/WS firmware. New hooks are added for boards to configure the top level behaviour of the system in stm32_main, and there is now support for Ethernet on H7 processors. Mboot now allows signed and encrypted firmware updates.

Breaking changes in this release are:

• The "sys" module is renamed to "usys". Ports that do not enable module weak links must replace "import sys" with "import usys as sys". See commit 40ad8f1

• machine.I2C and machine.SPI constructors are changed when using software-based I2C and SPI. Code that constructed I2C/SPI peripherals in the following way will need to be changed (or else their use will emit a warning which will eventually be an error):

    machine.I2C(-1, ...)            ->  machine.SoftI2C(...)
    machine.I2C(scl=scl, sda=sda)   ->  machine.SoftI2C(scl=scl, sda=sda)

    machine.SPI(-1, ...)            ->  machine.SoftSPI(...)
    machine.SPI(sck=sck, mosi=mosi, miso=miso)
                        ->  machine.SoftSPI(sck=sck, mosi=mosi, miso=miso)

Code which uses machine.I2C and machine.SPI classes to access hardware peripherals does not need to change. See commits 39d50d1 and 98182a9

• In ubluetooth, the BLE.irq() method now has positional only arguments. See commit 6a6a5f9

• In ubluetooth, arguments passed to the BLE.irq() callback that were previously bytes objects are now memoryview objects.
  See commit 81f2162

What follows is a detailed list of changes, generated from the git commit history, and organised into sections.

Main components

all:

• rename "sys" module to "usys"
• rename absolute time-based functions to include "epoch"

py core:

• objfloat: fix handling of negative float to power of nan
• fix handling of NaN in certain pow implementations
• showbc: pass in an mp_print_t struct to all bytecode-print functions
• parse: pass in an mp_print_t to mp_parse_node_print
• dynruntime.h: add mp_import_* and mp_load/store_*
• objstr: make bytes(bytes_obj) return bytes_obj
• objarray.h: add mp_obj_memoryview_init() helper function
• makeversionhdr.py: match only git tags which look like versions
• parse: expose rule-name printing as MICROPY_DEBUG_PARSE_RULE_NAME
• objdict: add mp_const_empty_dict_obj, use it for mp_const_empty_map
• objtype: handle dict attribute when type has no locals
• scope: name and use id_kind_type_t
• objexcept: compare mp_emergency_exception_buf_size signed
• use unsigned comparison of chars
• add explicit initializers for default values
• vmentrytable: ignore GCC -Woverride-init
• introduce and use MP_FALLTHROUGH macro
• makeqstrdefs.py: fix beaviour when scanning non-C preprocessed files
• makeqstrdefs.py: process C++ files as well
• mkrules.mk: add target for compiling C++ files
• makeqstrdefs.py: support preprocessing C++ files for QSTR generation
• workaround clang error when building misc.h with C++ compiler
• py.mk: support C++ code for user C modules
• mpz: do sign extension in mpz_as_bytes for negative values
• binary: fix sign extension setting wide integer on 32-bit archs
• mpprint: prevent case fall-through when assert is disabled
• mpprint: fix length calculation for strings with precision-modifier
• modmath: simplify handling of positional args to reduce code size
• mkrules.mk: remove stray vpath and unused -Itmp, add $(Q) for $(AR)
• objfun: support fun.globals attribute
• asmthumb: add support for ARMv6M in native emitter
• emitnative: ensure encoding to load prelude_offset doesn't change sz
• emitinlinethumb: exclude code using #if when ARMV7M disabled
• emitnative: support binary ops on ARMv6M without use of ite instr
• qstr.h: remove QSTR_FROM_STR_STATIC macro
• gc: fix debug printing of pointer
• persistentcode: introduce MICROPY_PERSISTENT_CODE_SAVE_FILE option
• makeversionhdr: honor SOURCE_DATE_EPOCH if present

extmod:

• modbluetooth: rename logging macro to be just DEBUG_printf
• modbluetooth: refactor stack/hci/driver/port bindings
• nimble: set struct alignment correctly on 64-bit arch
• nimble: make nimble_malloc work with allocated size
• nimble: implement NimBLE mutex
• modbluetooth: implement configuration of address modes
• modbluetooth: allow using mp_hal_get_mac as a static address
• btstack: add btstack support for _IRQ_GATTS_READ_REQUEST
• btstack: detect HCI UART init failure
• nimble: add timeout for HCI sync on startup
• modussl_axtls: reduce size of code that makes exception
• modbluetooth: fix handling of optional data/uuid args
• modbluetooth: make BLE.irq() method positional only
• modbluetooth: print UUIDs correctly
• modbluetooth: implement MTU
• vfs: fix lookup of entry in root dir so it fails correctly
• modbluetooth: change module-owned bytes objects to memoryview
• modure: allow \ in re.sub replacements
• machine_i2c: rename type to SoftI2C and add custom print method
• machine_i2c: remove "id" arg in SoftI2C constructor
• machine_spi: remove "id" arg in SoftSPI constructor
• utime_mphal: add generic utime.time_ns() function
• add explicit initializers for default values
• disable -Wmissing-field-initializers for lfs2
• introduce and use MP_FALLTHROUGH macro
• vfs_lfs: support mounting LFS filesystems in read-only mode
• modurandom: support urandom.seed() without an argument
• btstack/btstack.mk: add -Wimplicit-fallthrough=0
• nimble/nimble.mk: add -Wno-old-style-declaration
• moductypes: fix storing to (U)INT64 arrays on 32-bit archs
• machine_mem: only allow integers in machine.memX subscript
• modbluetooth: make UUID type accessible outside modbluetooth.c
• modbluetooth: combine gattc-data-available callbacks into one
• nimble: poll startup directly rather than using NimBLE sem
• modbluetooth: re-instate optional no-ringbuf modbluetooth
• nimble: make stm32 and unix NimBLE ports use synchronous events
• machine_i2c: add init protocol method for generic I2C bindings
• modbluetooth: add API for L2CAP channels
• nimble/modbluetooth_nimble: fix build when l2cap unavailable
• uasyncio: delay calling Loop.call_exception_handler by 1 loop
• uasyncio: add Task.done() method
• uasyncio: fix cancellation handling of wait_for
• nimble: fail read if the characteristic is too big
• modbluetooth: add _IRQ_CONNECTION_UPDATE event
• modbluetooth: allow setting char/desc enc/auth options
• modbluetooth: allow user-specified reason in read request IRQ
• modbluetooth: add compile-config flag to enable pairing/bonding
• modbluetooth: add _IRQ_ENCRYPTION_UPDATE event
• modbluetooth: allow configuration of pairing/bonding parameters
• modbluetooth: add gap_pair(conn_handle) func to intiate pairing
• modbluetooth: add support for bonding (key persistence)
• modbluetooth: simplify synchronous invoke_irq_handler signature
• modbluetooth: add support for passkey authentication
• nimble: generate and persist a unique IRK
• modubinascii: update code, docs for hexlify now CPython has sep
• vfs: raise OSError(ENODEV) if mounting bdev without a filesystem
• nimble: reset NimBLE BSS in mp_bluetooth_init
• nimble: don't assert on save-IRK failure
• modbluetooth: add ble.hci_cmd(ogf, ocf, req, resp) function
• vfs: check block 0 and ...

New uasyncio module, code formatting, BTstack bindings with unix support

This release of MicroPython includes a new implementation of the uasyncio module which aims to be more compatible with CPython's asyncio module. The main change is to use a Task object for each coroutine, allowing more flexibility to queue tasks in various places, eg the main run loop, tasks waiting on events, locks or other tasks. It no longer requires pre-allocating a fixed queue size for the main run loop. A pairing heap is used to queue Tasks, and the core is optionally implemented in C.

Most code in this repository is now auto-formatted using uncrustify for C code and Black for Python code. The tools/codeformat.py script is used to apply this formatting. Options for uncrustify were chosen to match as closely as possible the existing C style, but there were nevertheless quite a few changes to make the code style consistent.

BlueKitchen BTstack bindings have been added for the ubluetooth module, as an optional alternative to the NimBLE stack. The unix port can now be built with BLE support using these bindings, which works with USB Bluetooth adaptors (this feature is enabled by default on the dev and coverage variants).

Other Bluetooth additions include: new events for service/characteristic/ descriptor discovery complete; new events for read done and indicate acknowledgement; and support for active scanning in BLE.gap_scan(). See also below for breaking changes to the ubluetooth BLE API.

PEP 526 has been implemented which adds new syntax for variable annotations, such as "x:int = 1". PEP 572 has also been implemented, which adds the ":=" operator for assignment expressions, such as "if x := 1:...".

The whole code base now supports compressing error messages (those strings used as arguments to exception instances) using a "common word" scheme, which can be used to reduce code size.

Other new features include: support for non-boolean results for equality and inequality tests; the addition of the micropython.heap_locked() function to test the state of the heap; binary operations for viper uint operands in the native emitter; and mtime support on littlefs v2 filesystems (which can be disabled if needed).

There has been an important bug fix when importing ARM machine code from an .mpy file: the system now correctly tracks the executable memory allocated to the machine code so this memory is not reclaimed by the garbage collector.

For testing, a multi-instance test runner has been added (see tests/run-multitests.py) which allows running a synchronised test across two or more MicroPython targets. This is currently used for network and BLE tests that require communication between multiple devices.

In the unix port different builds are now organised into build variants, analogous to boards on bare-metal. And PEP 475 has been implemented which retries syscalls failing with EINTR.

The stm32 port sees improved support for STM32WB MCUs, with ADC, SPI and DMA support, along with support in mboot for these MCUs. Mboot also adds optional littlefs support.

The esp8266 port now includes three generic board definitions: GENERIC_512K, GENERIC_1M and GENERIC, with the latter for modules with 2M or more of flash. This port also changed the default filesystem to littlefs, and no longer supports hard pin IRQs. See breaking changes below for more details.

The esp32 port now enables Bluetooth support (via NimBLE) on both IDF v3 and v4 builds. It also changes the default filesystem from FAT to littlefs v2 (but existing devices with FAT filesystems will continue to work without change).

There is also a new mimxrt port which targets NXP i.MX RT series CPUs.

Breaking changes in this release are:

• async-for no longer await's on the result of aiter. The aiter method should return an async-iterable object but is not itself awaitable. See commit 37e1b5c

• In ubluetooth, the IRQ event constants have all changed value (from a bitfield to a sequential integer) and Python code should be updated accordingly by replacing the const definitions with the new ones, which can be found at https://docs.micropython.org/en/latest/library/ubluetooth.html#ubluetooth.BLE.irq See also commit e6881f0

• In ubluetooth, the IRQ data for the _IRQ_SCAN_RESULT event has had the "connectable" entry changed to "adv_type". The existing connectable value was a boolean and True now becomes 0x00, False becomes 0x02. See commit dd0bc26

• For esp8266, the default filesystem has changed from FAT to littlefs v2. And the flash layout of the firmware and filesystem has changed to give more space to the firmware and frozen bytecode. When upgrading an existing esp8266 device the filesystem will be reformatted so files should be backed up first. See commit e0905e8

• For esp8266, machine.Pin no longer supports configuring a pin IRQ with "hard=True" (because the interrupt handling code is moved to iRAM). Existing code using pin IRQs should remove any "hard=True" argument. See commit 0bd58a5

• For the unix port, binaries are renamed to micropython-variant (no longer micropython_variant). See commit 977b532

What follows is a detailed list of changes, generated from the git commit history, and organised into sections.

Main components

all:

• reformat C and Python source code with tools/codeformat.py
• convert exceptions to use mp_raise_XXX helpers in remaining places
• remove spaces between nested paren and inside function arg paren
• remove spaces inside and around parenthesis
• fix implicit casts of float/double, and signed comparison
• use MP_ERROR_TEXT for all error messages
• clean up error strings to use lowercase and change cannot to can't
• fix implicit floating point promotion
• fix implicit conversion from double to float
• fix implicit floating point to integer conversions
• enable extra conversion warnings where applicable
• remove commented-out include statements
• format code to add space after C++-style comment start
• factor gchelper code to one place and use it for unix & ARM ports
• fix auto-enable of MICROPY_GCREGS_SETJMP to select GC behaviour
• update Python code to conform to latest black formatting

py core:

• remove commented-out debug printf's from emitbc and objlist
• obj.h: remove comments about additional mp_buffer_info_t entries
• objobject: add object.setattr function
• objobject: add object.delattr function
• nlrx86: silence possible warnings about unused nlr argument
• asmx86: fix stack to be 16-byte aligned for entry and sub-call
• asmx86: remove unused 5th argument facility
• runtime: don't allocate iter buf for user-defined types
• objsingleton: use mp_generic_unary_op for singleton objects
• introduce MP_ROM_NONE macro for ROM to refer to None object
• introduce MP_ROM_FALSE/MP_ROM_TRUE for ROM to refer to bool objects
• objstr: don't use inline GET_STR_DATA_LEN for object-repr D
• obj.h: use 32-bit shift in MP_OBJ_NEW_QSTR calc for obj-repr D
• clean up commented-out code and comments about exception hierarchy
• objslice: add support for indices() method on slice objects
• objslice: inline fetching of slice paramters in str_subscr()
• make mp_obj_get_type() return a const ptr to mp_obj_type_t
• mkenv.mk: move usage of 32-bit flags to py.mk
• unicode: add unichar_isalnum()
• runtime: move MICROPY_PORT_INIT_FUNC near the end of mp_init()
• nativeglue: use mp_const_X instead of &mp_const_X_obj
• obj.h: redefine qstr object encoding to add immediate obj encoding
• obj: add MICROPY_OBJ_IMMEDIATE_OBJS option to reduce code size
• obj: optimise mp_obj_get_type for immediate objs with repr A and C
• mpconfig.h: define BITS_PER_BYTE only if not already defined
• objint: add mp_obj_int_get_uint_checked() helper
• pairheap: add generic implementation of pairing heap data structure
• gc: don't include or init gc_mutex when GIL is enabled
• qstr: don't include or init qstr_mutex when GIL is enabled
• objgenerator: use mp_obj_new_exception_arg1 to make StopIteration
• objexcept: optimise mp_obj_new_exception[_arg1/_args] functions
• obj.h: add and use mp_obj_is_bool() helper
• release GIL during syscalls in reader and writer code
• emitnative: use NULL for pending exception (not None)
• emitnative: stop after finding an unwind target
• mpthread.h: use strong type for mp_thread_set_state() argument
• objtype: make mp_obj_type_t.flags constants public, moved to obj.h
• support non-boolean results for equality and inequality tests
• compile: allow 'return' outside function in minimal builds
• scheduler: add "raise_exc" argument to mp_handle_pending
• scheduler: move mp_keyboard_interrupt from lib/utils to py core
• scheduler: allow a port to specify attrs for mp_keyboard_interrupt
• scheduler: move clearing of kbd traceback to mp_keyboard_interrupt
• expand type equality flags to 3 separate ones, fix bool/namedtuple
• add mp_raise_type helper macro and use it where appropriate
• add mp_raise_msg_varg helper and use it where appropriate
• factor out definition of mp_float_union_t to one location
• objexcept: rename mp_obj_new_exception_msg_varg2 to ..._vlist
• objtuple: remove code that handles tuple-subclass equality test
• objtype: allow mp_instance_cast_to_native_base to take native obj
• dynruntime.h: add implementation of mp_obj_cast_to_native_base
• objarray: turn on MP_TYPE_FLAG_EQ_CHECKS_OTHER_TYPE for memoryview
• removing dangling "else" to improve code format consistency
• bc0.h: shift comment to start of line to improve format consistency
• builtinimport: adjust if-block order in find_file to clean up #if's
• malloc: put { on separate line for funcs that have selective sigs
• un-nest configuration #if/#endif's for selection of complex code
• parse: add paren...

PYBD boards, BLE and littlefs, dynamic native modules, samd and powerpc

This release sees a reduction in overall bytecode size due to compression of the bytecode prelude, saving 7 bytes of bytecode per function for roughly 80% of functions. The bytecode opcode values are also redefined to group them based on their argument size and format. Support is also added for the matrix multiplication operator "@" (PEP 465).

The .mpy file format moved to version 5 to support the new bytecode encoding, and added support for relocation of native machine code, along with separate rodata and BSS sections. This allows a MicroPython system to import dynamic native modules, .mpy files that are generated from C code. Examples of this feature are provided in "examples/natmod/" and documentation in "docs/develop/natmod.rst".

Some code-size saving optimisations were implemented leading to a reduction in size of minimal firmware: bare-arm reduced by 584 bytes, minimal x86 by 5476 bytes (partly due to changed compiler optimisation flags) and minimal ARM Thumb 2 by 1440 bytes (all measured with gcc 9.2.0).

A new mechanism to freeze scripts into firmware is provided - a frozen manifest - whereby scripts to freeze are listed in a Python file (eg manifest.py). All ports are updated to use this new feature.

When weak module links are enabled (via MICROPY_MODULE_WEAK_LINKS) an import will now automatically search for the built-in u-variant of a module if the non-u-variant fails (eg searches for "ufoo" if "foo" fails), so ports no longer provide an explicit list of these. The unix and windows ports now enable this feature.

A performance benchmarking test suite is added which can be used to compare changes in absolute performance when optimising features, as well as compare across different ports. See "tests/run-perfbench.py".

Bluetooth (BLE only) support is added via the "ubluetooth" module and provides the ability to implement the four BLE roles. The BLE stack is used is Mynewt Nimble and it currently runs on PYBD boards, STM32WB55 MCUs and ESP32 boards.

Support for littlefs filesystems is added through the MicroPython VFS interface, and it works on the unix, stm32, esp8266 and esp32 ports.

A new "machine.ADC.read_u16()" method is defined and implemented on stm32, esp8266, esp32 and nrf ports, providing a consistent way to read an ADC that returns a value in the range 0-65535. This new method should be preferred to the existing "ADC.read()" method.

The stm32 port sees support for the new PYBD range of pyboards which include a fully integrated CYW43xx WiFi/BT chip. USB is enhanced to support VCP+MSC+HID mode and up to 3x simultaneous VCP interfaces. Support is also added for STM32WBxx MCUs including BLE. There is a new machine.ADC class which is compatible with other ports, along with a new machine.Timer class that implements a software timer with millisecond resolution and number of active timers only limited by RAM. Support for littlefs is available by default via uos.VfsLfs2. Documentation for using littlefs is found in "docs/reference/filesystem.rst".

The esp8266 port has switched to use per-board configurations and builds, as well as the new frozen manifest feature. It also has optional littlefs support (requires a separate build).

The esp32 port has the following new features: native code generation, machine.SDCard, hardware I2C, mDNS queries and responder, esp32.Partition, esp32.RMT, BLE (requires IDF v4), and built-in support for VfsLfs2. It has switched to use per-board configurations and builds, and frozen manifests. The default SSL output buffer is resized from 16kiB down to 4kiB to save RAM.

There are new, minimal ports to Microchip SAMDxx microcontrollers, and the bare metal PowerPC architecture.

A code-of-conduct based on the PSF's code is added in CODEOFCONDUCT.md.

The following new Git submodules are added: mbedtls, asf4, tinyusb, mynewt-nimble. And a new third-party library: littlefs.

A detailed list of changes follows.

py core:

• nativeglue: make private glue funs all static, remove commented code
• nativeglue: remove dependency on mp_fun_table in dyn-compiler mode
• obj: optimise small-int comparison to 0 in mp_obj_is_true
• nlrthumb: save and restore VFP registers s16-s21 when CPU has them
• mkrules.mk: use $(CPP) not $(CC) -E for preprocessor rule
• define EMIT_MACHINE_CODE as EMIT_NATIVE || EMIT_INLINE_ASM
• persistentcode: fix compilation with load and save both enabled
• persistentcode: ensure prelude_offset is always initialised
• asmarm: use __clear_cache on Linux/GCC when creating new asm code
• nlrthumb: check thumb2 instead of ARM_ARCH_6M
• objgenerator: add missing #if guard for PY_GENERATOR_PEND_THROW
• asmarm: use __builtin___clear_cache instead of __clear_cache
• makeqstrdata.py: allow using \r\n as a qstr if a port requires it
• scheduler: rename sched_stack to sched_queue
• objstringio: guard bytesio_stream_p struct w/ MICROPY_PY_IO_BYTESIO
• objdict: quote non-string types when used as keys in JSON output
• builtinimport: populate file when importing frozen or mpy files
• runtime: allow to override builtins.import with Python func
• modio: call mp_import_name to do resource stream import
• allow to pass in read-only buffers to viper and inline-asm funcs
• showbc: fix off-by-one when showing address of unknown opcode
• implement new sys.atexit feature
• objarray: fix amount of free space in array when doing slice assign
• modmath: implement math.isclose() for non-complex numbers
• introduce MP_UNREACHABLE macro to annotate unreachable code
• nlr: use MP_UNREACHABLE at the end of arch-specific nlr_jump funcs
• compile: improve the line numbering precision for comprehensions
• emitbc: rewrite switch in load_const_tok to reduce code size
• emitbc: make all emit_write_bytecode_* funcs take a stack_adj arg
• objgenerator: move defn of mp_const_GeneratorExit_obj here
• vm: shorten error message for not-implemented opcode
• vm: don't add traceback info for exc's propagated through a finally
• vm: don't add traceback info for exceptions that are re-raised
• add global default_emit_opt variable to make emit kind persistent
• compile: improve the line numbering precision for lambdas
• bc: factor out code to get bytecode line number info into new func
• profile: add initial implementation of sys.settrace feature
• integrate sys.settrace feature into the VM and runtime
• profile: add debugging for sys.settrace feature
• objtuple: allow compatible subclasses of tuple in mp_obj_tuple_get
• modstruct: fix struct.unpack with unaligned offset of native type
• modstruct: fix struct.pack_into with unaligned offset of native type
• binary: change mp_uint_t to size_t for index, size, align args
• bc: fix size calculation of UNWIND_JUMP opcode in mp_opcode_format
• mkenv.mk: add GDB variable
• vm: factor cached map lookup code to inline function
• mkrules.mk: add QSTR_GLOBAL_REQUIREMENTS variable for qstr auto-gen
• persistentcode: enable persistent code saving for Windows ports
• lexer: reorder operator tokens to match corresponding binary ops
• parse: use calculation instead of table to convert token to operator
• compile: use calculation instead of switch to convert token to op
• add support for matmul operator @ as per PEP 465
• bc0: order opcodes into groups based on their size and format
• bc: replace big opcode format table with simple macro
• introduce and use constants for multi-opcode sizes
• split RAISE_VARARGS opcode into 3 separate ones
• nlr.h: factor out constants to specific macros
• emitnative: factor sizeof/offsetof calculations to macros
• rename MP_QSTR_NULL to MP_QSTRnull to avoid intern collisions
• nativeglue: make mp_fun_table fixed size regardless of config
• persistentcode: bump .mpy version to 5
• ringbuf: add helpers for put16/get16
• bc: change mp_code_state_t.exc_sp to exc_sp_idx
• add n_state to mp_code_state_t struct
• compress first part of bytecode prelude
• rework and compress second part of bytecode prelude
• bc: don't include mp_decode_uint funcs when not needed
• compile: disallow 'import *' outside module level
• makeqstrdefs.py: remove unused blacklist
• vm: fix handling of unwind jump out of active finally
• runtime: fix PEP479 behaviour throwing StopIteration into yield from
• emitnative: add support for archs with windowed registers
• emitnative: add support for using setjmp with native emitter
• emitnative: add support for archs that cannot read executable data
• asmxtensa: add support for Xtensa with windowed registers
• add new Xtensa-Windowed arch for native emitter
• persistentcode: make .mpy more compact with qstr directly in prelude
• objtype: add type.bases attribute
• mkrules.mk: add warning/error for invalid frozen config
• objstr: size-optimise failure path for mp_obj_str_get_buffer
• automatically provide weak links from "foo" to "ufoo" module name
• modarray: rename "array" module to "uarray"
• runtime: reorder some binary ops so they don't require conditionals
• stream.h: add MP_STREAM_POLL_NVAL constant
• nativeglue: remove unused mp_obj_new_cell from mp_fun_table
• objgenerator: remove globals from mp_obj_gen_instance_t
• objgenerator: allow pend_throw to an unstarted generator
• persistentcode: move declarations for .mpy header from .c to .h file
• modsys: report .mpy version in sys.implementation
• emitnative: fix typo, REG_PARENT_ARG_RET should be REG_PARENT_RET
• emitnx86: make mp_f_n_args table match order of mp_fun_kind_t
• objdict: support ujson.dump() of OrderedDict objects
• ringbuf: add peek16 method
• compile: coalesce error message for break/continue outside loop
• builtinimport: raise exception on empty module name
• qstr: raise exception in qstr_from_strn if str to intern is too long
• objstringio: slightly optimize stringio_copy_on_write for code size
• objenumerate: check for valid args in enumerate constructor
• persistentcode: add abilit...

Improved mpy format with support for native code, and new JavaScript port

In this release the mpy file format has been moved to version 4 and has some significant improvements: mpy file size is reduced on average by about 35%, loading time of mpy files is reduced by about 40%, and they now have support to save native, viper and inline assembler code (or machine code generated from any other source). Size reduction of mpy files was achieved by adding a qstr window to reuse past qstrs when encoding them, by packing qstrs directly in the bytecode, and by defining a static qstr set.

Some VM opcodes were also changed to fix a bug when doing a break/continue out of a finally block, and to make some simplifications. In particular POP_BLOCK and POP_EXCEPT opcodes were replaced with POP_EXCEPT_JUMP.

Most uppercase macros have been converted to lowercase to make a more consistent C API, including all MP_OBJ_IS_xxx and MP_xxx_SLOT_IS_FILLED macros.

The default PYTHON makefile variable is now changed from "python" to "python3", but Python 2 is still supported via "make PYTHON=python2".

The mpy-cross compiler supports the new mpy version 4 and has new command line options: "-march=" to select the native emitter, and "--version" to print the MicroPython version and the mpy version. Also mpy-tool.py has support for freezing native code.

A module system for external, user C modules has been implemented and documentation for this is available in the new "docs/develop" section.

A new "javascript" port has been added which targets JavaScript as the machine via Emscripten. This allows to run MicroPython as an application within node.js, and to run it within a browser (among other things).

All bare-metal ports have the following improvements: machine.sleep() is now machine.lightsleep(), and both lightsleep() and deepsleep() now take an optional argument which is the maximum time to sleep in milliseconds. These ports also now allow freezing of boot.py and main.py using the usual methods. And a new I2C method i2c.writevto(addr, vect) is added which can be used to write a tuple/list of buffers all at once to an I2C device.

The stm32 port now has a fully integrated Ethernet MAC driver (see the network.LAN class) using lwIP for the TCP/IP stack, and sockets were made significantly more robust. Support for F413 MCUs was added. There are also some minor user-facing changes to this port:

• machine.UART (and pyb.UART) now defaults to timeout=0 (was 1000ms) which gives UARTs a non-blocking behaviour.
• The USB REPL is now configurable using uos.dupterm, and by default pyb.USB_VCP(0) is put on dupterm slot 1 before boot.py is executed. To disable the default REPL use: uos.dupterm(None, 1).
• pyb.DAC(id) will now only reset the DAC the first time it is called; to get the old behaviour pass the bits parameter like: pyb.DAC(id, bits).
• pyb.DAC.noise() and pyb.DAC.triangle() now output at full scale.
• The system will not recreate boot.py if it's missing (it will only create it if the filesystem is corrupt, or when doing a factory reset).

For the esp32 port, the build process has been updated to align better with the ESP IDF and now uses sdkconfig to configure features. Dual core mode is now enabled by default, SPIRAM is in memory-mapped mode so all of it can be used for the MicroPython heap, there is support to change the CPU frequency, and the WDT now panics and resets the device if it times out.

A detailed list of changes follows.

py core:

• remove calls to file reader functions when these are disabled
• add optional support for 2-argument version of built-in next()
• compile: swap order of pop_block/pop_except in "except as" handler
• warning: support categories for warnings
• builtinhelp: only print help re FS modules if external import enabled
• downcase all MP_OBJ_IS_xxx macros to make a more consistent C API
• downcase MP_xxx_SLOT_IS_FILLED inline functions
• mkenv.mk: change default PYTHON variable from "python" to "python3"
• qstr: evaluate find_qstr only once then pass to Q_GET_HASH macro
• obj.h: remove obsolete mp_obj_new_fun_viper() declaration
• objfun: make fun_data arg of mp_obj_new_fun_asm() a const pointer
• eliminate warnings about unused arguments when debugging disabled
• compile: fix handling of unwinding BaseException in async with
• compile: add optimisation to compile OrderedDict inplace
• objexcept: fix hash of exc str created in mp_obj_new_exception_msg
• py.mk: update lwip build config to work with latest lwip version
• fix VM crash with unwinding jump out of a finally block
• vm: remove currently_in_except_block variable
• replace POP_BLOCK and POP_EXCEPT opcodes with POP_EXCEPT_JUMP
• persistentcode: add a qstr window to save mpy files more efficiently
• persistentcode: pack qstrs directly in bytecode to reduce mpy size
• persistentcode: define static qstr set to reduce size of mpy files
• add independent config for debugging sentinel object values
• emitnative: consolidate where HASCONSTS is set to load-const-obj fun
• emitnative: provide concentrated points of qstr emit
• emitnative: adjust accounting of size of const_table
• emitglue: remove union in mp_raw_code_t to combine bytecode & native
• add support to save native, viper and asm code to .mpy files
• persistentcode: bump .mpy version to 4
• allow registration of modules at their definition
• implement a module system for external, user C modules
• update and rework build system for including external C modules
• move mp_native_type_from_qstr() from emitnative.c to nativeglue.c
• nativeglue: rename native convert funs to match other native helpers
• compile: add support to select the native emitter at runtime
• compile: support multiple inline asm emitters
• compile: check that arch is set when compiling native, viper or asm
• runtime: remove long-obsolete MICROPY_FSUSERMOUNT init code
• scheduler: convert micropythyon.schedule() to a circular buffer
• nlrthumb: add support for iOS where the C func is _nlr_push_tail
• makedefs: use io.open with utf-8 encoding when processing source
• runtime: optimise to not create temp float for int to power negative
• runtime: fix mp_unpack_ex so seq can't be reclaimed by GC during use
• mpprint: support printing %ld and %lu formats on 64-bit archs
• asmthumb: support asm_thumb code running on normal ARM processors
• native: improve support for bool type in viper functions
• remove "if (0)" and "if (false)" branches
• objgenerator: fix handling of None passed as 2nd arg to throw()
• objgenerator: remove unneeded forward decl and clean up white space
• misc.h: rename _MP_STRINGIFY to not use leading underscore in ident
• persistentcode: change "len" type to size_t for mp_obj_str_get_data
• objarray: add support for memoryview.itemsize attribute
• objarray: add decode method to bytearray
• update makefiles to use $(CAT) variable instead of hard coded "cat"
• update makefiles to use $(TOUCH) instead of hard coded "touch"

extmod:

• moduzlib: update to uzlib 2.9.2
• modlwip: add support for polling UDP sockets for writability
• moduhashlib: include implementation of sha256 only when required
• convert legacy uppercase macro names to lowercase
• moduwebsocket: refactor websocket to uwebsocket
• modlwip: change #ifdef to #if for check of MICROPY_PY_LWIP
• modlwip: fix bug when polling listening socket with backlog=1
• modlwip: add concurrency protection macros
• modwebrepl: fix logic to handle a put of file of size 0
• vfs_fat: update for new oofatfs version
• modlwip: fix case where concurrency lock isn't released on error
• moduselect: adjust select_select and poll_register to use size_t
• vfs_fat: fallback to FAT32 if standard FAT16/SFD format fails
• modlwip: handle case of connection closing while on accept queue
• modlwip: handle case of accept callback called with null PCB
• modlwip: protect socket.accept with lwIP concurrency lock
• modlwip: free any stored incoming bufs/connections on TCP error
• modlwip: use correct listening socket object in accept callback
• modlwip: abort TCP conns that didn't close cleanly in a while
• modurandom: add init method to seed the Yasmarang generator
• machine_signal: fix fault when no args are passed to Signal()
• modussl_mbedtls: support non-blocking handshake
• modussl_axtls: add non-blocking mode support
• moducryptolib: add optional AES-CTR support
• modujson: handle parsing of floats with + in the exponent
• machine_i2c: change C-level API to allow split I2C transactions
• machine_i2c: remove need for temporary memory in writemem() call
• machine_i2c: add i2c.writevto() that can write a vector of bufs
• modlwip: free any incoming bufs/connections before closing PCB
• modlwip: register TCP close-timeout callback before closing PCB

lib:

• utils/pyexec: implement paste mode with event driven REPL
• utils/printf: exclude __GI_vsnprintf alias for gcc 9 and above
• utils/gchelper_m3: add license header and clean up code
• utils/gchelper_m3: add gc_helper_get_sp() function
• utils/gchelper: add gchelper.h header file for assembler functions
• netutils: add function to print tracing info for Ethernet frames
• oofatfs: update oofatfs library to R0.13c working branch
• oofatfs: update ffconf.h config for new oofatfs version
• oofatfs: update oofatfs library to fix issue with logic not
• stm32lib: update library to fix F7 MMC capacity calculation
• utils/pyexec: add pyexec_file_if_exists() helper function
• utils: make pyexec_file_if_exists run frozen scripts if they exist
• utils/interrupt_char: invalidate interrupt char at start up
• stm32lib: update library to fix UART9/10 baudrate on F4 MCUs
• nrfx: upgrade nrfx to v1.7.1 plus a UART bug fix

drivers:

• memory/spiflash: rework wait_sr to fix uninit'd variable 'sr'
• display/ssd1306.py: change to use new i2c.writevto() method

tools:

• mpy-tool.py: add support for freezing native code
• upip.py: use "raise arg" instead of no-arg raise for...

Vast improvements to native emitter, new nrf port, unified documentation

In this release there are a wide range of improvements and additions to both the core and the ports. In the core the main improvement was to the native emitter to have much more comprehensive support for general Python features, such as generators and complex exception handling, and the generated machine code is smaller and retains its efficiency. Elsewhere, fuzzy testing was used to find and eliminate some corner-case bugs, user classes were optimised when they don't use special accessors, underscores in numeric literals are now supported (PEP515), and the uio.IOBase class was added to allow user defined streams.

For the extended modules there is now a VfsPosix filesystem component, a new ucryptolib module with AES support, addition of ure.sub() and of uhashlib.md5, and the lwIP socket implementation now has working TCP listen/accept backlog.

Compared to the last release the minimal baseline core code size is reduced by about 2.2%, down by roughly 1500 bytes for bare-arm port and 3500 bytes for minimal x86 port. Most other ports have increased in size due to the addition of new features (eg ucryptolib, ure.sub).

The stm32 port sees the introduction of a new bootloader -- mboot -- which supports DFU upload via USB FS, USB HS, as well as a custom I2C protocol, and also support to program external SPI flash. There is significant refactoring of the USB device driver, improved VCP throughput, and support for 2x VCP interfaces on the one USB device. lwIP has been integrated, and support added for SDRAM. Cortex-M0 CPUs are now supported along with STM32F0 MCUs.

For the esp8266 port the heap is increased by 2kbytes, the radio is automatically put to sleep if no WLAN interfaces are active, and the UART can now be disconnected from the REPL and its RX buffer length configured. Upon soft-reset sockets are now cleaned up.

The esp32 port has added support for external SPI RAM, PPPoS functionality, improved performance and stability when using threads, and other general bug fixes.

There is a new nrf port for Nordic MCUs, currently supporting nRF51x and nRF52x chips.

The docs have now been unified so there is just one set of documentation covering all ports. And initial documentation for the esp32 port is added.

There are two changes at the Python API level that are not backwards with previous versions:

• uos.dupterm now requires that a stream passed to it is derived from uio.IOBase (or is a native stream object).
• The esp32 neopixel driver has had its default timing changed from 400kHz to 800kHz; existing code that didn't explicitly specify the "timing" parameter in the NeoPixel constructor may need to be updated to specify this as "timing=0" to get 400kHz timing.

A detailed list of changes follows.

py core:

• mpconfig.h: be stricter when autodetecting machine endianness
• mpstate.h: adjust start of root pointer section to exclude non-ptrs
• nlrx86: use naked attribute on nlr_push for gcc 8.0 and higher
• vm: adjust #if logic for gil_divisor so braces are balanced
• objfun: fix variable name in DECODE_CODESTATE_SIZE() macro
• vm: use enum names instead of magic numbers in multi-opcode dispatch
• vm: improve performance of opcode dispatch when using switch stmt
• repl: fix handling of unmatched brackets and unfinished quotes
• misc.h: add MP_STATIC_ASSERT macro to do static assertions
• compile: change comment about ITER_BUF_NSLOTS to a static assertion
• emitbc: avoid undefined behavior calling memset() with NULL 1st arg
• parsenum: use int instead of mp_int_t for parsing float exponent
• parsenum: avoid undefined behavior parsing floats with large exponents
• objfloat: fix undefined shifting behavior in high-quality float hash
• mpz: avoid undefined behavior at integer overflow in mpz_hash
• objfloat: fix undefined integer behavior hashing negative zero
• gc: when GC threshold is hit don't unnecessarily collect twice
• parsenum: adjust braces so they are balanced
• modbuiltins: add support for rounding integers
• objgenerator: save state in old_globals instead of local variable
• objgenerator: protect against reentering a generator
• emit: combine fast and deref into one function for load/store/delete
• emit: combine yield value and yield-from emit funcs into one
• emit: combine build tuple/list/map emit funcs into one
• emit: combine name and global into one func for load/store/delete
• emit: combine load/store/delete subscr into one emit function
• emit: combine load/store/delete attr into one emit function
• emit: combine break_loop and continue_loop into one emit function
• emit: combine import from/name/star into one emit function
• emit: merge build set/slice into existing build emit function
• emit: combine setup with/except/finally into one emit function
• objtype: fix assertion failures in mp_obj_new_type by checking types
• objtype: fix assertion failures in super_attr by checking type
• stream: move definition of mp_stream_p_t from obj.h to stream.h
• reader: allow MICROPY_VFS_POSIX to work with MICROPY_READER_POSIX
• mpconfig.h: add default MICROPY_VFS_FAT config value
• obj.h: introduce a "flags" entry in mp_obj_type_t
• objtype: don't expose mp_obj_instance_attr()
• objtype: optimise instance get/set/del by skipping special accessors
• gc: add gc_sweep_all() function to run all remaining finalisers
• lexer: add support for underscores in numeric literals
• modio: add uio.IOBase class to allow to define user streams
• mkrules.mk: regenerate all qstrs when config files change
• stream: introduce and use efficient mp_get_stream to access stream_p
• objarray: replace 0x80 with new MP_OBJ_ARRAY_TYPECODE_FLAG_RW macro
• add checks for stream objects in print() and sys.print_exception()
• compile: combine break and continue compile functions
• compile: combine subscript_2 and subscript_3 into one function
• compile: combine global and nonlocal statement compile functions
• compile: combine or_test and and_test compile functions
• compile: combine expr, xor_expr and and_expr into one function
• compile: handle return/break/continue correctly in async with
• objgenerator: eliminate need for mp_obj_gen_wrap wrapper instances
• obj.h: fix broken build for object repr C when float disabled
• simplify some cases of accessing the map of module and type dict
• objdict: make mp_obj_dict_get_map an inline function
• objmodule: make mp_obj_module_get_globals an inline function
• obj.h: give compile error if using obj repr D with single-prec float
• malloc: give a compile warning if using finaliser without GC
• objgenerator: implement name with normal fun attr accessor code
• emitnative: optimise for iteration asm code for non-debug build
• runtime: use mp_obj_new_int_from_ll when return int is not small
• stream: introduce MP_STREAM_GET_FILENO ioctl request
• objstr: in format error message, use common string with %s for type
• asmthumb: optimise native code calling runtime glue functions
• mpconfig.h: introduce MICROPY_DEBUG_PRINTER for debugging output
• fix compiling with debug enabled and make more use of DEBUG_printf
• emitnative: factor common code for native jump helper
• emitnative: fix x86 native zero checks by comparing full word
• asmx86: use generic emit function to simplify cmp emit function
• emitnative: fix native locals stack to start at correct location
• emitnative: simplify handling of exception objects from nlr_buf_t
• emitnative: allocate space for local stack info as it's needed
• compile: for dynamic compiler, widen literal 1 to get correct shift
• py.mk: don't hardcode path to libaxtls.a
• gc: in gc_alloc, reset n_free var right before search for free mem
• objarray: allow to build again when bytearray is disabled
• stream: adjust mp_stream_posix_XXX to take void*, not mp_obj_t
• emitnative: use small tables to simplify handling of local regs
• asmxtensa: handle function entry/exit when stack use larger than 127
• asm*: support assembling code to jump to a register, and get PC+off
• emitnative: optimise and improve exception handling in native code
• asmxtensa: fix bug with order of regs in addi encoding
• asmxtensa: optimise loading local addr and support larger offsets
• emitnative: fix bug with store of 16 and 32 values in viper ARM mode
• asmxtensa: use narrow version of add instr to reduce native code size
• emitnx86: fix number of args passed to mp_setup_code_state, 4 not 5
• vm: fix handling of finally-return with complex nested finallys
• emitnative: cancel caught exception once handled to prevent reraise
• emitnative: add support for return/break/continue in try and with
• compile: factor code that compiles start/end of exception handler
• py.mk: build axtls library directly from its source files
• runtime: fix incorrect test for MICROPY_PORT_DEINIT_FUNC
• objarray: bytearray: allow 2nd/3rd arg to constructor
• emitnative: fix try-finally in outer scope, so finally is cancelled
• fix native functions so they run with their correct globals context
• optimise call to mp_arg_check_num by compressing fun signature
• asmx64: fix bug in assembler when creating disp with r13 and 0 offset
• {asmx86,asmx64}: extend test_r8_with_r8 to accept all 8 lower regs
• emit: move MP_EMIT_OPT_xxx enums from compile.h to emitglue.h
• emit: remove need to call set_native_type to set native/viper mode
• emit: remove need to call set_native_type to set viper return type
• emit: completely remove set_native_type, arg type is set in compiler
• compile: merge viper annotation and normal param compilation stages
• compile: factor code that compiles viper type annotations
• emitnative: reuse mp_native_type_from_qstr when searching for a cast
• make viper functions have the same entry signature as native
• emitnative: support arbitrary number of arguments to viper functions
• emitnative: use macros instead of raw offsetof for slot locations
• emitnative: make viper f...

Parser size reduced, new Python stack, stm32 improvements, new esp32 port

This release brings some significant size reductions to the parser, as well as removal of unused code and additional tests to improve coverage of the core. A new optional, internal Python stack is introduced for scoped allocation and can be used instead of alloca to provide more efficient and flexible temporary memory; see MICROPY_ENABLE_PYSTACK. There have been many improvements and internal code refactors of the stm32 port, and the port to the Espressif ESP32 SoC has been merged from its development repository.

Compared to the last release the change in code size is (in bytes, using gcc 7.3):

bare-arm: -1832
minimal x86: -2608
unix x64: -5129
unix nanbox: +1024
stm32: +2120
cc3200: -928
esp8266: +336

The decrease is mainly due to the reduced size of the parser, where the table of rule pointers was compressed to a table of offsets. The increase in the stm32 and esp8266 ports is due to additional features, such as more colour formats in the framebuf module and the addition of ujson.dump().

Code coverage (measured by gcov) has improved since the last version (v1.9.3 on the left, v1.9.4 on the right):

py: 15202/15447 = 98.4% -> 15391/15513 = 99.2%
extmod: 2227/ 2363 = 94.2% -> 2291/ 2430 = 94.3%

Changes and additions seen by the Python user include: improvement of dir() and tab-completion to list all available attributes of an object; addition of efficient ucollections.deque type with fixed size; better handling and polling support of sockets that get into an error state; implementation of key and cert keyword arguments in ussl.wrap_socket (for axtls); uos.ilistdir now return 4-tuples with the file size in the fourth position.

Changes at the C level include: removal of "make_qstr_if_not_already" argument from mp_obj_new_str; rename of mp_exc_recursion_depth to mp_raise_recursion_depth; removal of mp_const_MemoryError_obj; switching of stream close operation from method to ioctl; refactoring of how native emitter code is compiled with a file per architecture.

The stm32 port has seen a lot of improvements and additions, as well as some significant internal refactoring to better support configuring custom boards. STM32F7 support is improved, USB HS is supported for F723 and F733, the CAN class has improvements to handle bus errors, ctrl-C can now interrupt running code when the REPL is over UART, and the ADC class added the read_timed_multi static method. Board configuration has seen some internal changes, in particular with setting LD_FILES, TEXT0_ADDR, and TEXT1_ADDR in mpconfigboard.mk (previously LD_FILE, FLASH_ADDR, TEXT_ADDR). The pin_X and pyb_pin_X identifiers have been changed to be pointers to objects rather than objects, and main is renamed to stm32_main.

A detailed list of changes follows.
py core:

• compile: use alloca instead of qstr_build when compiling import name
• mpconfig: introduce reusable MP_HTOBE32(), etc. macros
• objtype: mp_obj_new_type: name base types related vars more clearly
• objnamedtuple: add _asdict function if OrderedDict is supported
• emitnative: clean up asm macro names so they have dest as first arg
• mkenv.mk: use $(PYTHON) consistently when calling Python tools
• objstr: remove "make_qstr_if_not_already" arg from mp_obj_new_str
• objstr: make mp_obj_new_str_of_type check for existing interned qstr
• objstr: when constructing str from bytes, check for existing qstr
• add config option to disable multiple inheritance
• objnamedtuple: allow to reuse namedtuple basic functionality
• objfloat: allow float() to parse anything with the buffer protocol
• modbuiltins: slightly simplify code in builtin round()
• runtime: simplify handling of containment binary operator
• opmethods: include the correct header for binary op enums
• runtime: add MP_BINARY_OP_CONTAINS as reverse of MP_BINARY_OP_IN
• parsenum: improve parsing of floating point numbers
• objdict: reuse dict-view key iterator for standard dict iterator
• annotate func defs with NORETURN when their corresp decls have it
• qstr: rewrite find_qstr to make manifest that it returns a valid ptr
• gc: in gc_realloc, convert pointer sanity checks to assertions
• objgenerator: remove unreachable code for STOP_ITERATION case
• misc.h: add m_new_obj_var_with_finaliser()
• mp_call_function_*_protected(): pass-thru return value if possible
• modbuiltins: use standard arg-parsing helper func for builtin print
• mpprint: make "%p" format work properly on 64-bit systems
• mpprint: support "%lx" format on 64-bit systems
• malloc: allow to use debug logging if !MICROPY_MALLOC_USES_ALLOCATED_SIZE
• gc: add CLEAR_ON_SWEEP option to debug mis-traced objects
• emitbc, asmbase: only clear emit labels to -1 when in debug mode
• asmbase: revert removal of clearing of label offsets for native emit
• objint_longlong: check for zero division/modulo
• runtime: when tracing unary/binary ops, output op (method) name
• gc: factor out a macro to trace GC mark operations
• gc: in sweep debug output, print pointer as a pointer
• objfun: factor out macro for decoding codestate size
• objfun, vm: add comments on codestate allocation in stackless mode
• objfun: factor out macro for initializing codestate
• mkrules.mk: add "clean-frozen" target to clean frozen script/modules dir
• runtime: move mp_exc_recursion_depth to runtime and rename to raise
• introduce a Python stack for scoped allocation
• convert all uses of alloca() to use new scoped allocation API
• runtime: use the Python stack when building *arg and **kwarg state
• objexcept: use INT_FMT when printing errno value
• extend nan-boxing config to have 47-bit small integers
• objtype: implement better support for overriding native's init
• objtype: refactor object's handling of new to not create 2 objs
• builtinimport: call init for modules imported via a weak link
• objgenerator: allow to pend an exception for next execution
• map: don't include ordered-dict mutating code when not needed
• mpz: fix pow3 function so it handles the case when 3rd arg is 1
• modio: use correct config macro to enable resource_stream function
• malloc: remove unneeded code checking m_malloc return value
• nlrthumb: fix use of naked funcs, must only contain basic asm code
• nlr: clean up selection and config of NLR implementation
• nlr: factor out common NLR code to macro and generic funcs in nlr.c
• parse: split out rule name from rule struct into separate array
• parse: break rule data into separate act and arg arrays
• parse: pass rule_id to push_result_rule, instead of passing rule_t*
• parse: pass rule_id to push_result_token, instead of passing rule_t*
• parse: remove rule_t struct because it's no longer needed
• parse: compress rule pointer table to table of offsets
• parse: update debugging code to compile on 64-bit arch
• parse: fix macro evaluation by avoiding empty VA_ARGS
• mpz: simplify handling of borrow and quo adjustment in mpn_div
• mpz: in mpz_as_str_inpl, convert always-false checks to assertions
• nlr: fix nlr functions for 64bit ports built with gcc on Windows
• compile: combine compiler-opt of 2 and 3 tuple-to-tuple assignment
• objtype: check and prevent delete/store on a fixed locals map
• modbuiltins: for builtin_chr, use uint8_t instead of char for array
• vm: simplify stack sentinel values for unwind return and jump
• parsenum: fix parsing of floats that are close to subnormal
• objfloat: fix case of raising 0 to -infinity
• unicode: clean up utf8 funcs and provide non-utf8 inline versions
• modbuiltins: simplify casts from char to byte ptr in builtin ord
• objrange: implement (in)equality comparison between range objects
• objexcept: remove long-obsolete mp_const_MemoryError_obj
• nlrthumb: do not mark nlr_push as not returning anything
• pystack: use "pystack exhausted" as error msg for out of pystack mem
• objdict: disallow possible modifications to fixed dicts
• gc: reduce code size by specialising VERIFY_MARK_AND_PUSH macro
• gc: rename gc_drain_stack to gc_mark_subtree and pass it first block
• gc: make GC stack pointer a local variable
• qstr: add QSTR_TOTAL() macro to get number of qstrs
• modbuiltins: simplify and generalise dir() by probing qstrs
• repl: generalise REPL autocomplete to use qstr probing
• objstr: protect against creating bytes(n) with n negative
• objmodule: factor common code for calling init on builtin module
• builtinimport: add compile-time option to disable external imports
• modmicropython: allow to have stack_use() func without mem_info()
• objstr: remove unnecessary check for positive splits variable
• objint: use MP_OBJ_IS_STR_OR_BYTES macro instead of 2 separate ones
• objdeque: implement ucollections.deque type with fixed size
• objdeque: protect against negative maxlen in deque constructor
• objdeque: use m_new0 when allocating items to avoid need to clear
• py.mk: split list of uPy sources into core and extmod files
• py.mk: remove .. path component from list of extmod files
• use "GEN" consistently for describing files generated in the build
• compile: adjust c_assign_atom_expr() to use return instead of goto
• asm*.c: remove unnecessary check for num_locals<0 in asm entry func
• mpz: in mpz_clone, remove unused check for NULL dig
• mpstate.h: add repl_line state for MICROPY_REPL_EVENT_DRIVEN
• vm: fix case of handling raised StopIteration within yield from
• vm: simplify handling of special-case STOP_ITERATION in yield from
• formatfloat: fix rounding of %f format with edge-case FP values
• formatfloat: fix case where floats could render with a ":" character
• formatfloat: fix case where floats could render with negative digits
• objint: remove unreachable code checking for int type in format func
• misc.h: remove unused count_lead_ones() inline function
• obj.h: clean up by removing commented-out inline versions of macros
• obj.h: move declaration of mp_o...