Skip to content

Discussion of Python 3.6 support #2415

Description

@dpgeorge

Python 3.6 beta 1 was released on 12 Sep 2016, and a summary of the new features can be found here: https://docs.python.org/3.6/whatsnew/3.6.html . Being in beta stage means we can provide feedback on any of the new features/changes if we like.

(See #1329 for Python 3.5 support, which is still pending.)

New syntax features:

New built-in features:

  • PEP 468 - Preserving the order of **kwargs in a function
  • PEP 487 - Simpler customization of class creation
  • PEP 520 - Preserving Class Attribute Definition Order

Standard library changes:

  • PEP 495 - Local Time Disambiguation
  • PEP 506 - Adding A Secrets Module To The Standard Library [not relevant to uPy core]
  • PEP 519 - Adding a file system path protocol

CPython internals (not exposed at the Python level so not relevant to uPy):

  • PEP 509 - Add a private version to dict
  • PEP 523 - Adding a frame evaluation API to CPython

Linux/Windows changes:

  • PEP 524 - Make os.urandom() blocking on Linux (during system startup)
  • PEP 528 - Change Windows console encoding to UTF-8 (provisional)
  • PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional)

Other Language Changes

  • A global or nonlocal statement must now textually appear before the first use of the affected name in the same scope. Previously this was a SyntaxWarning.
  • It is now possible to set a special method to None to indicate that the corresponding operation is not available. For example, if a class sets __iter__() to None, the class is not iterable.
  • Long sequences of repeated traceback lines are now abbreviated as "[Previous line repeated {count} more times]"
  • Import now raises the new exception ModuleNotFoundError when it cannot find a module. Code that currently checks for ImportError (in try-except) will still work.
  • Class methods relying on zero-argument super() will now work correctly when called from metaclass methods during class creation.

Changes to MicroPython built-in modules

  • array - Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended.
  • asyncio (many, may need another ticket)
  • binascii - The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline character is appended to the return value.
  • cmath
    • The new cmath.tau (τ) constant has been added.
    • New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infj and cmath.nanj to match the format used by complex repr.
  • collections
    • The new Collection abstract base class has been added to represent sized iterable container classes.
    • The new Reversible abstract base class represents iterable classes that also provide the __reversed__() method.
    • The new AsyncGenerator abstract base class represents asynchronous generators
    • The namedtuple() function now accepts an optional keyword argument module, which, when specified, is used for the __module__ attribute of the returned named tuple class.
    • The verbose and rename arguments for namedtuple() are now keyword-only.
    • Recursive collections.deque instances can now be pickled.
  • hashlib
    • hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2.
    • BLAKE2 hash functions were added to the module. blake2b() and blake2s() are always available and support the full feature set of BLAKE2.
    • The SHA-3 hash functions sha3_224(), sha3_256(), sha3_384(), sha3_512(), and SHAKE hash functions shake_128() and shake_256() were added.
    • The password-based key derivation function scrypt() is now available with OpenSSL 1.1.0 and newer.
  • json - json.load() and json.loads() now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32.
  • math - The tau (τ) constant has been added to the math and cmath modules.
  • os
    • See the summary of PEP 519 for details on how the os and os.path modules now support path-like objects.
    • A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol.
    • On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security.
    • The Linux getrandom() syscall (get random bytes) is now exposed as the new os.getrandom() function.
  • re
    • Added support of modifier spans in regular expressions. Examples: '(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON'; '(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'.
    • Match object groups can be accessed by __getitem__, which is equivalent to group(). So mo['name'] is now equivalent to mo.group('name').
    • Match objects now support index-like objects as group indices.
  • socket
    • The ioctl() function now supports the SIO_LOOPBACK_FAST_PATH control code.
    • The getsockopt() constants SO_DOMAIN, SO_PROTOCOL, SO_PEERSEC, and SO_PASSSEC are now supported.
    • The setsockopt() now supports the setsockopt(level, optname, None, optlen: int) form.
    • The socket module now supports the address family AF_ALG to interface with Linux Kernel crypto API. ALG_*, SOL_ALG and sendmsg_afalg() were added.
    • New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added.
  • ssl
    • ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2.
    • 3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added.
    • SSLContext has better default configuration for options and ciphers.
    • SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance.
    • The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority.
    • All constants and flags have been converted to IntEnum and IntFlags.
    • Server and client-side specific TLS protocols for SSLContext were added.
    • Added SSLContext.post_handshake_auth to enable and ssl.SSLSocket.verify_client_post_handshake() to initiate TLS 1.3 post-handshake authentication.
  • struct - now supports IEEE 754 half-precision floats via the 'e' format specifier.
  • sys - The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames.
  • zlib - The compress() and decompress() functions now accept keyword arguments.

(Changes to non-built-in modules will need to be documented elsewhere.)

Activity

  1. dpgeorge commented on Sep 13, 2016

    @dpgeorge
    MemberAuthor

    PEP 468, preserving the order of kwargs, is of particular interest to us because it might be hard to implement efficiently. CPython 3.6 will have dict == OrderedDict (ie all dicts are ordered), using the PyPy implementation to make it efficient. Essentially all dicts have 2 tables, one which is the ordered array of (key, value) pairs (ordered by order of insertion), and the other is a (compact) hash table that indexes the ordered array.

  2. pfalcon commented on Sep 13, 2016

    @pfalcon
    Contributor

    I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented. Easy and helpful. The rest, I'm more and more getting disappointed where CPython goes.

  3. dpgeorge commented on Sep 13, 2016

    @dpgeorge
    MemberAuthor

    I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented

    +10. It would be pretty easy to add, and I agree really nice to then write something like machine.freq(42_000_000).

  4. dhylands commented on Sep 13, 2016

    @dhylands
    Contributor

    I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented

    +10. It would be pretty easy to add, and I agree really nice to then write something like machine.freq(42_000_000).

    +1 here too. I've been using that while coding in rust and while a bit odd at first, it makes reading the larger integers easier.

  5. dbc commented on Sep 16, 2016

    @dbc
    Contributor

    I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented. Easy and helpful.

    Oh yes, oh yes. This is one my favourite language features in any language. I love it for hex and binary literals. Interestingly, at PyCon 2016 in Portland a few months ago, when the BDFL was reading over the list of PEP's, I cheered and applauded loudly for this one.... and about 2 thousand other people in the same room with me were dead silent. :/ So I'm thinking us embedded programmer types have different priorities.

    As to the rest, I'm thinking that preserving order in kwargs is hardly worth the expense of memory in Micropython, for as often as it is used. OTOH, it may be a forced march if it becomes important for code portability. OTOOH, since we have 'micro' versions of the important libraries anyway, perhaps porting is a given in any case.

    PEP530 just tickles me. Not sure it's of great value to Micropython, but I love it.

    (Yes, I'm back! Been away from Micropython since about last December. Sometimes, life happens, and good stuff falls below the cut-off line on the to-do list. Hopefully, Micropython can crawl back above the line for a while.)

  6. peterhinch commented on Sep 16, 2016

    @peterhinch
    Contributor

    @dbc Welcome back! I don't follow your point about PEP530 though - improved async support is surely welcome. Many embedded applications rely on asynchronous code.

  7. dbc commented on Sep 16, 2016

    @dbc
    Contributor

    @peterhinch re: PEP530 -- I should have been more clear about my fear of RAM bloat. Totally agree that asynch is great for embedded, I'm a big fan. I'd use async comprehensions if they were available, certainly -- like a cat on catnip. And of course we want Micropython to be as close to 100% source compatible with CPython as is practical. My fear is the "practical" part in Micropython -- if asynch comprehensions cause the SRAM footprint to grow significantly for programs using non-async list comprehensions or non-async generator expressions, then I'd be wary of it.

    Thinking about it... the semantics of an async comprehension are satisfied by just doing a normal comprehension. The completion order is arbitrary, after all. So maybe the syntax can be accepted but the functionality could be a normal comprehension.

  8. peterhinch commented on Sep 16, 2016

    @peterhinch
    Contributor

    @dbc Will the availability of the async keyword affect the bytecode emitted when the keyword is absent? This surely depends on the cleverness of the compiler. But I'll leave this to people who actually understand compilers :)

  9. pohmelie commented on Sep 20, 2016

    @pohmelie
    Contributor

    PEP 498 - Literal String Formatting. It's step forward and "replace both existing % and str.format" looks like best solution for me. Definitely I want it in µpy.
    PEP 525 - Asynchronous Generators (provisional). This is just awesome, since this can replace pretty ugly async for context class. It is just same as generator and __enter__/__iter__ methods of object. But, this requires too much changes, since this can't be done just as sugar for async/await in µpy right now. And probably no one need this in µpy.

  10. dbc commented on Sep 20, 2016

    @dbc
    Contributor

    PEP 498: I read that in detail now. So.. PEP 498 looks like the future. I never use %-formatting in new code. When PEP 498 strings happen in CPython, I will probably go exclusively to PEP498 format strings for new code and never look back, they are much more readable.

    @dpgeorge says " likely too much bloat "

    Ignoring PEP 498 means sacrificing CPython compatibility in a very visible way. Removing .format() in favor of literal string formatting is a very visible breaking change. Both of those options hit every user every day.

    My naive assumption would be that much of the low-level formatting code could be factored out of .format() and literal string formatting. Pre-processing the PEP 498 strings would add to the compiler, certainly. Or is it RAM bloat and not code bloat that is the concern? An f-string becomes a byte-code expression in RAM instead of a simple string. Is this the concern?

    I guess if you twisted my arm, I'd choose dropping %-formatting and .format() in favor of f-strings if I had to choose between f-strings and .format().

  11. pohmelie commented on Sep 20, 2016

    @pohmelie
    Contributor

    @dbc, one thing is pretty annoying. If you have only f-strings and got template from somewhere as string you should do something like

    >>> s = "{x}"
    >>> x = 1
    >>> eval(f"f'{s}'")
    '1'
  12. deshipu commented on Sep 20, 2016

    @deshipu
    Contributor

    You can still do:

    >>> s = "{x}"
    >>> x = 1
    >>> s.format(x=x)
  13. pohmelie commented on Sep 20, 2016

    @pohmelie
    Contributor

    @deshipu, @dbc and I talking about «replace both existing % and str.format», so there is not str.format/format/%

  14. deshipu commented on Sep 20, 2016

    @deshipu
    Contributor

    I see, I didn't catch that. In any case, they are not getting removed, so more power to you.

    Argh, and now I re-read the whole conversation, and see how stupid my comment is, sorry.

  15. pohmelie commented on Sep 20, 2016

    @pohmelie
    Contributor

    @deshipu, as @dpgeorge said in first post this is possible in theory.

  16. dbc commented on Sep 20, 2016

    @dbc
    Contributor

    If you have only f-strings and got template from somewhere as string you should do something like ...

    Yes, that is somewhat klunky.

  17. peterhinch commented on Sep 21, 2016

    @peterhinch
    Contributor

    @pohmelie @dbc said

    Removing .format() in favor of literal string formatting is a very visible breaking change. Both of those options hit every user every day.

    I agree that removing .format() would be very bad news. Likewise the idea of removing the string modulus operator. In my view such changes would break far too much code and would be a worse departure from cPython compatibility than opting not to support f-strings.

  18. ARF1 commented on Mar 2, 2017

    @ARF1

    +1 For PEP 487 - Simpler customization of class creation

    This is deceptively humbly named. It is a MUCH saner and light-weight replacement for metaclasses in many cases. Also, it should be fairly simple to implement given that the machinery will be fairly similar to __new__.

  19. s-celles commented on Sep 20, 2018

    @s-celles

    f-string is something I'd love to see in MicroPython
    PEP 498 -- Literal String Interpolation

  20. kamikaze commented on Dec 8, 2018

    @kamikaze
    Contributor

    Kill the .format() , voting for f-strings ! broken code is an abandoned code in 99% cases. I don't see any reason to not update the code for this change.

  21. mattytrentini commented on Apr 1, 2019

    @mattytrentini
    SponsorContributor

    I believe PEP-15 is implemented in at least 1.10. I think we can strike it through!

  22. dpgeorge commented on Apr 2, 2019

    @dpgeorge
    MemberAuthor

    PEP 515 (underscores in numeric literals) was implemented in 6a445b6

  23. asottile commented on May 2, 2019

    @asottile

    for those looking for another way to get f-strings, I've added better support to future-fstrings for use as a preprocessor: https://github.com/asottile/future-fstrings#transform-source-for-micropython

    It has a PEP-498 compliant source transformer that produces equivalent '...'.format(...) calls

  24. pmp-p commented on Sep 20, 2019

    @pmp-p
    Contributor

    thank you @asottile but sadly it seems, we won't benefit of bpo-36817 since asottile-archive/future-fstrings#40 was closed.

    Where is the "Discussion of Python 3 support" status ? i may have missed it. I hardly see any 3.x future support with notimpl on major language features ( including some that exist since 2.x )

    i only found 3.5 #1329

  25. asottile commented on Sep 20, 2019

    @asottile

    that issue is still doable, I just don't really feel motivated to work on it -- if someone wants to pick it up then by all means do so :)

  26. jimmo commented on Sep 20, 2019

    @jimmo
    Member

    For f-strings, did you see #4998 referenced above?

  27. pmp-p commented on Sep 22, 2019

    @pmp-p
    Contributor

    @jimmo no i'm not searching for 3.6/3.7/3.8 totallly optionnal features from #4998

    Apart from the warm thanks addressed to the person deserving them for adaptation ( after all he did not have to take the time to add that ) of his usefull tool for use with micropython in replacement of clumsy https://github.com/pmp-p/fstrings_helper my question was :

    Where is the "Discussion of Python 3 support" status ? i may have missed it. I hardly see any 3.x future support with notimpl on major language features ( including some that exist since 2.x )

    i only found 3.5 #1329, i'm more interested for discussion upon completion of 3.4 and reaching stability.

  28. kevinkk525 commented on Oct 30, 2019

    @kevinkk525
    Contributor

    Any chance to implement PEP 526 (Syntax for Variable Annotations (provisional))?

    I understand that it doesn't have any impact beyond the parser but would be hugely beneficial in development (at least to me it would be). I guess if it is too much work then using strip-hints is an option. Running it on each script before uploading/freezing would be an acceptable workaround if it works reliable.

  29. dpgeorge commented on Oct 31, 2019

    @dpgeorge
    MemberAuthor

    Any chance to implement PEP 526 (Syntax for Variable Annotations (provisional))?

    Yes I would like to see this implemented soon. It would then allow more sophisticated type hinting, and can also get use in viper annotations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    rfcRequest for Comment

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions