Introduction of ports subdirectory where all ports are moved to

The main change in this release is the introduction of a "ports/" subdirectory at the top-level of the repository, and all of the ports are moved here. In the process the "stmhal" port is renamed to "stm32" to better reflect the MCU that it targets. In addition, the STM32 CMSIS and HAL sources are moved to a new submodule called "stm32lib".

The bytecode has changed in this release, compared to the previous release, and as a consequence the .mpy version number has increased to version 3. This means that scripts compiled with the previous mpy-cross must be recompiled to work with this new version.

There have also been various enhancements and optimisations, such as: check for valid UTF-8 when creating str objects, support for reverse special binary operations like __radd__, full domain checking in the math module, support for floor-division and modulo in the viper emitter, and addition of stack overflow checking when executing a regex.

The stm32 port sees improved support for F7 MCUs, addition of a new board B_L475E_IOT01A based on the STM32L475, and support for the Wiznet W5500 chipset along with improved socket behaviour.

A detailed list of changes follows.

py core:

• objstr: startswith, endswith: check arg to be a string
• nlrx86,x64: replace #define of defined() with portable macro usage
• objtype: handle NotImplemented return from binary special methods
• objtype: mp_obj_class_lookup: improve debug logging
• map: remove unused new/free functions
• make m_malloc_fail() have void return type, since it doesn't return
• modstruct: in struct.pack, stop converting if there are no args left
• modstruct: check and prevent buffer-read overflow in struct unpacking
• modstruct: check and prevent buffer-write overflow in struct packing
• nlrthumb: get working again on standard Thumb arch (ie not Thumb2)
• objfloat: fix binary ops with incompatible objects
• obj: fix comparison of float/complex NaN with itself
• objtype: implement fallback for instance inplace special methods
• objtuple: properly implement comparison with incompatible types
• objstr: add check for valid UTF-8 when making a str from bytes
• objlist: properly implement comparison with incompatible types
• runtime0.h: move relational ops to the beginning of mp_binary_op_t
• runtime0.h: move MP_BINARY_OP_DIVMOD to the end of mp_binary_op_t
• objtype: make sure mp_binary_op_method_name has full size again
• runtime0.h: put inplace arith ops in front of normal operations
• builtinhelp: simplify code slightly by extracting object type
• runtime: implement dispatch for "reverse op" special methods
• nlrx86: fix building for Android/x86
• builtinhelp: change signature of help text var from pointer to array
• runtime.h: change empty mp_warning macro so var-args are non empty
• modbuiltins: implement abs() by dispatching to MP_UNARY_OP_ABS
• {objfloat,objcomplex}: optimise MP_UNARY_OP_ABS by reusing variables
• mpconfig.h: add note that using computed gotos in VM is not C99
• objstr: strip: don't strip "\0" by default
• objexcept: prevent infinite recursion when allocating exceptions
• stream: remove unnecessary checks for NULL return from vstr_add_len
• vstr: raise a RuntimeError if fixed vstr buffer overflows
• vm: use lowercase letter at start of exception message
• persistentcode: define mp_raw_code_save_file() for any unix target
• add config option to print warnings/errors to stderr
• objfloat: support raising a negative number to a fractional power
• objset: simplify set and frozenset by separating their locals dicts
• objset: check that RHS of a binary op is a set/frozenset
• objset: include the failed key in a KeyError raised from set.remove
• objtype: change type of enum-to-qstr table to uint16_t to save space
• objstr: make empty bytes object have a null-terminating byte
• mpprint: only check for null string printing when NDEBUG not defined
• objtype: clean up unary- and binary-op enum-to-qstr mapping tables
• persistentcode: bump .mpy version number to version 3
• bc: update opcode_format_table to match the bytecode
• modmath: add full checks for math domain errors
• modmath: convert log2 macro into a function
• formatfloat: don't print the negative sign of a NaN value
• formatfloat: use standard isinf, isnan funcs instead of custom ones
• modbuiltins: use existing utf8_get_char helper in builtin ord func
• emitnative: implement floor-division and modulo for viper emitter
• objtype: use CPython compatible method name for sizeof
• objtype: fit qstrs for special methods in byte type
• objtype: define all special methods if requested
• objtype: introduce MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS

extmod:

• modubinascii: only include uzlib/tinf.h when it's really needed
• modussl_mbedtls: allow to compile with MBEDTLS_DEBUG_C disabled
• machine_pinbase: put PinBase singleton in ROM
• re1.5: upgrade to v0.8.2, adds hook for stack overflow checking
• modure: add stack overflow checking when executing a regex
• uos_dupterm: update uos.dupterm() and helper funcs to have index
• uos_dupterm: swallow any errors from dupterm closing the stream
• vfs: replace VLA in proxy func with small, static sized array
• modussl: add finaliser support for ussl objects
• modussl_mbedtls: allow to compile with unix coverage build

lib:

• add new submodule, stm32lib containing STM32 CMSIS and HAL source
• embed/abort_: use mp_raise_msg helper function
• libm: fix tanhf so that it correctly handles +/- infinity args
• libm: remove implementation of log2f, use MP_NEED_LOG2 instead
• axtls: update, support for SSL_EAGAIN return code
• berkeley-db-1.xx: update, allow to override MINCACHE, DEFPSIZE

drivers:

• memory/spiflash: change from hard-coded soft SPI to generic SPI
• display/ssd1306.py: improve performance of graphics methods
• nrf24l01: make nRF24L01 test script more portable
• display/ssd1306: implement SSD1306_I2C poweron method
• display/ssd1306: make poweron() work the same with SSD1306_SPI
• wiznet5k: improve the performance of socket ops with threading
• wiznet5k: get low-level W5500 driver working

tools:

• upip: upgrade to 1.2.2
• pyboard: use repr() when quoting data in error messages
• pyboard: update docstring for additional device support

tests:

• object_new: better messages, check user new() method
• class_new: add checks for init being called and other improvements
• class_new: add another testcase for new/init interaction
• class_inplace_op: test for inplace op fallback to normal one
• run-bench-tests: update locations of executables, now in ports/
• class_reverse_op: test for reverse arith ops special methods
• run-tests: skip class_inplace_op for minimal profile
• run-tests: fix copy-paste mistake in var name
• cpydiff: add cases for locals() discrepancies
• extmod: add test for ure regexes leading to infinite recursion
• extmod: add test for '-' in character class in regex
• run-tests: close device under test using "finally"
• net_inet: update tls test to work with CPython and incl new site

unix port:

• rename modsocket.c to modusocket.c
• modusocket: remove #if MICROPY_SOCKET_EXTRA code blocks
• enable MICROPY_PY_REVERSE_SPECIAL_METHODS

stm32 port:

• modmachine: make machine.bootloader() work when MPU is enabled
• modmachine: improve support for sleep/deepsleep on F7 MCUs
• compute PLL freq table during build instead of at run time
• modmachine: for F7 MCU, save power by reducing internal volt reg
• boards/pllvalues.py: make script work with both Python 2 and 3
• Makefile: use lib/stm32lib instead of local cmsis and hal files
• remove cmsis and hal files, they are now a submodule
• Makefile: automatically fetch stm32lib submodule if needed
• update to new STM Cube HAL library
• fix clock initialisation of L4 MCUs
• rename stmhal port directory to stm32
• remove unused usbd_msc.c file
• boards: change remaining stm32f4xx_hal_conf.h to unix line ending
• boards: change linker scripts to use "K" instead of hex byte size
• boards: fix I2C1 pin mapping on NUCLEO_F401RE/F411RE boards
• i2c: when scanning for I2C devices only do 1 probe per address
• modnwwiznet5k: release the GIL on blocking network operations
• boards: add new board B_L475E_IOT01A based on STM32L475
• make-stmconst.py: make sure mpz const data lives in ROM
• timer: make pyb.Timer() instances persistent
• mpconfigport.h: add configuration for max periphs on L4 series
• usbdev: make the USBD callback struct const so it can go in ROM
• usbdev: change static function variable to non-static
• usbdev: put all CDC state in a struct
• usbdev: put all HID state in a struct
• usbdev: simplify CDC tx/rx buffer passing
• usbdev: simplify HID tx/rx buffer passing
• usbdev/core: add state parameter to all callback functions
• usbdev: put all state for the USB device driver in a struct
• usbdev: simplify pointers to MSC state and block dev operations
• usbdev: merge all global USB device state into a single struct
• usbdev: make device descriptor callbacks take a state pointer
• usbdev: move all the USB device descriptor state into its struct
• timer: enable ARPE so that timer freq can be changed smoothly
• modnwwiznet5k: get the IP address of an established socket
• boards: fix typos in stm32f767_af.csv table
• usbd_cdc_interface: don't reset CDC output buf on initialisation
• modnwwiznet5k: implement WIZNET5K.isconnected() method
• modusocket: make getaddrinfo() work when passed an IP address
• modusocket: return OSError(-2) if getaddrinfo fails
• mpconfigport.h: add MICROPY_THREAD_YIELD() macro
• modnwwiznet5k: add support for W5500 Ethernet chip
• modnwwiznet5k: increase SPI bus speed to 42MHz
• modnwwiznet5k: implement stream ioctl for the Wiznet driver
• mphalport: improve efficiency of mp_hal_stdout_tx_strn_cooked
• make uos.dupterm() conform to specs by using extmod version

cc3200 port:

• enable micropy...

Double precision math library and support on pyboard, and improved ussl

This release brings general improvements and bug fixes to the core and
various ports, as well as documentation additions, clean-ups and better
consistency. And effort has been made to clean up the source code to
make it more consistent across the core and all ports.

There is a new tool "mpy_bin2res.py" to convert arbitrary (binary) files
to Python resources for inclusion in source code (frozen or otherwise).

The ussl module has seen improvements, including implementation of
server_hostname (for axtls) and server_side mode (for mbedtls).

There is now a double-precision float math library and stmhal has support
to build firmware with software or hardware double-precision.

A detailed list of changes follows.

py core:

• formatfloat: fix number of digits and exponent sign when rounding
• modthread: raise RuntimeError in release() if lock is not acquired
• compile: raise SyntaxError if positional args are given after /*
• objint: support "big" byte-order in int.to_bytes()
• objint: in to_bytes(), allow length arg to be any int and check sign
• compile: fix bug with break/continue in else of optimised for-range
• compile: optimise emitter label indices to save a word of heap
• builtinimport: remove unreachable code for relative imports
• objnamedtuple: simplify and remove use of alloca building namedtuple
• mpprint: remove unreachable check for neg return of mp_format_float
• binary: add missing "break" statements
• runtime: mark m_malloc_fail() as NORETURN
• objstr: remove unnecessary "sign" variable in formatting code
• vm: make "if" control flow more obvious in YIELD_FROM opcode
• modmath: check for zero division in log with 2 args
• makeversionhdr.py: update to parse new release line in docs/conf.py
• objdict: factorise dict accessor helper to reduce code size
• change mp_uint_t to size_t in builtins code
• repl: change mp_uint_t to size_t in repl helpers
• compile: combine arith and bit-shift ops into 1 compile routine
• compile: use switch-case to match token and operator
• objgenerator: allow to hash generators and generator instances
• gc: refactor assertions in gc_free function
• vm: make n_state variable local to just set-up part of VM
• asmx64: support moving a 64-bit immediate to one of top 8 registers
• modmicropython: cast stack_limit value so it prints correctly
• builtinevex: add typechecking of globals/locals args to eval/exec
• py.mk: make berkeley-db C-defs apply only to relevant source files
• mperrno: allow mperrno.h to be correctly included before other hdrs
• mpz: make mpz_is_zero() an inline function
• implement raising a big-int to a negative power
• mkrules.mk: show frozen modules sizes together with executable size
• objtuple: allow to use inplace-multiplication operator on tuples
• objstr: raise an exception for wrong type on RHS of str binary op
• modsys: initial implementation of sys.getsizeof()
• binary.c: fix bug when packing big-endian 'Q' values
• add verbose debug compile-time flag MICROPY_DEBUG_VERBOSE
• binary: change internal bytearray typecode from 0 to 1
• objstringio: prevent offset wraparound for io.BytesIO objects
• objstringio: fix regression with handling SEEK_SET
• stream: seek: Consistently handle negative offset for SEEK_SET
• mkrules.mk: use "find -path" when searching for frozen obj files
• compile: remove unused pn_colon code when compiling func params
• objcomplex: remove unnecessary assignment of variable
• formatfloat: don't post-increment variable that won't be used again
• use "static inline" for funcs that should be inline
• asmthumb: use existing macro to properly clear the D-cache

extmod:

• modussl_axtls: update for axTLS 2.1.3
• modussl_axtls: implement server_hostname arg to wrap_socket()
• move modonewire.c from esp8266 to extmod directory
• modure: if input string is bytes, return bytes results too
• modubinascii: add check for empty buffer passed to hexlify
• modussl_axtls: allow to close ssl stream multiple times
• modussl_mbedtls: support server_side mode
• modussl_mbedtls: when reading and peer wants to close, return 0
• modframebuf: fix invalid stride for odd widths in GS4_HMSB fmt
• modussl_mbedtls: make socket.close() free all TLS resources
• modframebuf: consistently use "col" as name for colour variables
• modussl_mbedtls: implement non-blocking SSL sockets
• machine_signal: fix parsing of invert arg when Pin is first arg
• modframebuf: use correct initialization for .locals_dict
• modlwip: implement setsockopt(IP_ADD_MEMBERSHIP)
• modussl_mbedtls.c: add ussl.getpeercert() method
• modubinascii: rewrite mod_binascii_a2b_base64
• modubinascii: don't post-increment variable that won't be used
• modonewire: rename public module to mp_module_onewire
• for uos.stat interpret st_size member as an unsigned int
• use "static inline" for funcs that should be inline

lib:

• axtls: upgrade to axTLS 2.1.3 + MicroPython patchset
• libm/math: remove implementations of float conversion functions
• add libm_dbl, a double-precision math library, from musl-1.1.16

drivers:

• onewire: move onewire.py, ds18x20.py from esp8266 to drivers
• onewire: enable pull-up when init'ing the 1-wire pin

tools:

• gen-cpydiff: use case description as 3rd-level heading
• pyboard: add license header
• mpy_bin2res: tools to convert binary resources to Python module
• mpy-tool.py: don't generate const_table if it's empty
• mpy-tool.py: fix missing argument in dump() function

tests:

• net_inet/test_tls_sites.py: integration test for SSL connections
• net_inet: add tests for accept and connect in nonblocking mode
• basics: add tests for for-else statement
• net_inet: move tests which don't require full Internet to net_hosted
• connect_nonblock: refactor towards real net_hosted test
• auto detect floating point capabilites of the target
• import: add a test for the builtin import function
• import: update comment now that uPy raises correct exception
• basics/namedtuple1: add test for creating with pos and kw args
• unix/extra_coverage: add test for mp_vprintf with bad fmt spec
• basics: add tests for arithmetic operators precedence
• cpydiff/modules_deque: elaborate workaround
• cpydiff/core_class_mro: move under Classes, add workaround
• cpydiff/core_arguments: move under Functions subsection
• cpydiff/core_class_supermultiple: same cause as core_class_mro
• cpydiff: improve wording, add more workarounds
• cpydiff: add case for str.ljust/rjust
• rename exec1.py to builtin_exec.py
• basics/builtin_exec: test various globals/locals args to exec()

minimal port:

• Makefile: enable gc-sections to remove unused code
• remove unused stmhal include from Makefile
• use size_t for mp_builtin_open argument

unix port:

• modtime: replace strftime() with localtime()
• mpconfigport.mk: update descriptions of readline and TLS options
• Makefile: disable assertions in the standard unix executable
• modjni: convert to mp_rom_map_elem_t
• for uos.stat interpret st_size member as an unsigned int

stmhal port:

• mpconfigport.h: remove config of PY_THREAD_GIL to use default
• make error messages more consistent across peripherals
• add initial implementation of Pin.irq() method
• add .value() method to Switch object, to mirror Pin and Signal
• move pybstdio.c to lib/utils/sys_stdio_mphal.c for common use
• add "quiet timing" enter/exit functions
• make available the _onewire module, for low-level bus control
• modules: provide sym-link to onewire.py driver
• boards/stm32f405.ld: increase FLASH_TEXT to end of 1MiB flash
• sdcard: allow a board to customise the SDIO pins
• add possibility to build with double-precision floating point
• boards: enable double-prec FP on F76x boards
• Makefile: use hardware double-prec FP for MCUs that support it
• Makefile: rename FLOAT_IMPL to MICROPY_FLOAT_IMPL to match C name
• Makefile: add CFLAGS_EXTRA to CFLAGS so cmdline can add options
• mpconfigport.h: allow MICROPY_PY_THREAD to be overridden
• boards: add configuration files for NUCLEO_F429ZI
• boards/NUCLEO_F429ZI: change USB config from HS to FS peripheral
• reduce size of ESPRUINO_PICO build so it fits in flash
• servo: make pyb.Servo(n) map to Pin('Xn') on all MCUs
• servo: don't compile servo code when it's not enabled
• use "static inline" for funcs that should be inline

cc3200 port:

• modusocket: simplify socket.makefile() function
• make non-zero socket timeout work with connect/accept/send
• modusocket: fix connect() when in non-blocking or timeout mode
• use the name MicroPython consistently in code

esp8266 port:

• Makefile: bump axTLS TLS record buffer size to 5K
• Makefile: allow FROZEN_DIR,FROZEN_MPY_DIR to be overridden
• Makefile: add LIB_SRC_C variable to qstr auto-extraction list
• make onewire module and support code usable by other ports
• modonewire: move low-level 1-wire bus code to modonewire.c
• modonewire: make timings static and remove onewire.timings func
• reinstate 1-wire scripts by sym-linking to drivers/onewire/
• move mp_hal_pin_open_drain from esp_mphal.c to machine_pin.c
• enable MICROPY_ENABLE_FINALISER
• README: make "Documentation" a top-level section
• machine_rtc: use correct arithmetic for aligning RTC mem len
• mpconfigport_512k: use terse error messages to get 512k to fit
• mpconfigport.h: make socket a weak link
• modesp: remove unused constants: STA_MODE, etc
• general: add known issue of WiFi RX buffers overflow
• use size_t for mp_builtin_open argument
• fix UART stop bit constants

zephyr port:

• Makefile: rework dependencies and "clean" target
• Makefile: revert prj.conf construction rule to the previous state
• remove long-obsolete machine_ptr_t typedef's
• Makefile: explicitly define default target as "all"
• modusocket: allow to use socketized net_context in upstream
• modusocket: socket, close: switch to native Zephyr socket calls
• modusocket: bind...