New MicroPython hardware API

@dhylands

Edit: for easy reference, the proposed new API is at https://github.com/micropython/micropython/wiki/Hardware-API

Since we have the WiPy TODO list (#1425) Damien and I spent some time today discussing how to do things properly, once and for all, and do it before the WiPy ships. It involves a redefinition of the API in many aspects and it has implications all over the place. I'll do my best to layout all the points we touched:

~~Use attributes instead of functions when we simply need to get/set things. I am going to quote @dhylands on issue Functions vs attributes in object API #378.~~

I think that to use attributes then you need to use nouns. If you can't represent it as a noun, then I think it should stay as a function

~~So for instance, pin.value() will change to pin.value to get it and pin.value = x to set it. This will be the way to go in other peripherals, e.g. timer.frequency, timer.period, etc.~~

Ruled out becaue attributes have a few drabacks which are a deal breaker.

The whole Pin API will look like this:

# create & init a pin
# id, mode are mandatory and positional (also can be named)
# pull is optional and positional (also can be named)
# rest of args are kwonly
# only "value" is required for a port to implement (initial value if given)
# rest are optional, and a port can define them and also define others
pin = Pin(id, mode, pull=Pin.PULL_NONE, *, value, drive, slew, ...)

# constants
# OPEN_DRAIN is much more descriptive than OD
for mode: Pin.IN, Pin.OUT, Pin.OPEN_DRAIN, Pin.ALT, Pin.ALT_OPEN_DRAIN
for pull: Pin.PULL_UP, Pin.PULL_DOWN, Pin.PULL_NONE
optional pull: Pin.PULL_UP_STRONG, Pin.PULL_DOWN_WEAK, ...
for drive: Pin.LOW_POWER, Pin.MED_POWER, Pin.HIGH_POWER
for slew: Pin.FAST_RISE, Pin.SLOW_RISE

pin.init(...) # re init

pin.high() # set high
pin.low() # set low

pin.value() # get value
pin.value(x) # set value

# toggle
pin.toggle()

# functions to get/set most common settings
# this duplicates a bit what init can already do, but it is useful in the case of pins
# for instance when bit banging like it's done by the one-wire driver,
pin.id() # this one is only a getter
pin.mode()
pin.pull()
pin.drive()
...

In general then any extensions to a function/method can happen in 3 ways:
1. the port provides extra constants for an existing arg (eg Pin.PULL_UP_WEAK).
2. the port provides extra keyword args.
3. the port provides extra functions/methods.
A new module called hardware will be created to replace the pyb module. pyb will still exist in the pyboard in order to provide backwards compatibility and to avoid confusions between the old and the new API.
The new hardware module will contain Pin, SPI, UART, I2C, Timer, CAN, and so on.
~~The new hardware classes will use attributes where applicable.~~
A new module named machine will hold all functions related to reset, interrupt and sleep management.
The UART already supports the stream protocol and we would like to bring it to SPI and I2C as well. Right now we have the recv functions that accepts either an integer or a buffer object, which is kind of weird and can get confusing. If we make SPI and I2C stream capable, then we have read/readinto/readchar, write/writechar. This will make the API across communication peripherals consistent and homogeneous. One problem of the stream approach is how to select the slave (for SPI) or the destination address (for I2C). We had two ideas:
1. Simply add kwarg to read/write like addr, memaddr, slave_addr, etc. This will deviate from the standard stream API.
2. Create and endpoint subclass which is the one actually providing the read/write methods and holds the information relevant to the destination like chip select, i2c address, addres size, etc. Doesn't solve the issue of having to add kwarg to the read/write methods in the case of the memaddr for I2C (not practical to have an endpoint for each memaddr).

I prefer option 1...

It's a lot to discuss in one ticket, but we think it's better because all things are related and it shows the complete picture.

The idea is to have this new API implemented (for the WiPy at least) in 2 weeks...

There are still some leftover functions/classes in pyb that don't yet have a home (eg LED, Servo, Accel, USB stuff). But these are not present on the WiPy so deciding upon them is not urgent. They will stay in pyb for now. Most likely the USB stuff will move to a separate module, so that boards that have USB will have this module, and boards that don't, won't.

I think that the whole hardware word already lets you know that what you see there relates to the hardware you are running MicroPython on. Your board doesn't have USB? then it doesn't show in hardware. You can do dir(hardware) and see what's supported. So, USB and SD (SDcard) which are very common but some boards have it and some not, should fit well into the hardware module. Maybe LED too...

Specific stuff like Servo and Accel can go into something like pyboard. The WiPy for instance has a HeartBeat class. This could go into wipy.

Things like UART and SPI are low level peripherals.

Things like Servo and LED are built on top of this. I'd like to see Servo and LED be much more flexible than they currently are. Currently it's impossible for the user to add additional Servos or LEDs to the ones which are "builtin". In theory I should be able to instantiate a Servo and associate it with a timer and/or a pin. For Servo and LED there are each a couple of different kinds.

1 - For Servo, you can have a Servo which is tied to a PWM output (much like the current pyb.Servo class). But you can also use a single timer to drive multiple servos which are connected to arbitrary GPIOs that don't necessarily have a PWM output (this is the model that teensy uses).

2 - Similarly, for LED, you can have a simple on/off LED, or you could have a PWM controlled LED.

I think that designing some of these "higher level" classes to be a bit more flexible makes them much more usable across various boards and board setups.

Regarding attributes vs functions (as previously discussed in #378, but this point I'm making here was not raised in that issue): the big downside of using attributes is that assignment is a statement and not an expression, so you would not be able to set the pin value (or do other attr assignments) in a lambda. Eg you can't do:

timer.callback(lambda tim: pin.value = 1)

With functions acting as getters/setters (based on whether 0 or 1 argument is supplied), which is how things work at the moment, you can do:

timer.callback(lambda tim: pin.value(1))

I think this is a deal-breaker for using attributes. Ignoring the implementation details and/or efficiency, and looking instead from a user/programmability point of view, the API should always use functions because they are expressions (more powerful than statements) and you can take a "pointer" to a function (eg pin_value_fun = pin.value) but not an attribute.

So, even though it's more Pythonic to use attributes for things that are clearly accessed using a getter and a setter (eg pin.value=1), I propose that we instead use functions that take 0 or 1 args (eg pin.value(1)).

So, even though it's more Pythonic to use attributes for things that are clearly accessed using a getter and a setter (eg pin.value=1), I propose that we instead use functions that take 0 or 1 args (eg pin.value(1)).

OK, let's only use functions then.

the API should always use functions because

I'm glad that we came to the same conclusion as we did before ;-).

I'm glad that we came to the same conclusion as we did before ;-).

Yes indeed. But since the topic here is to discuss a complete revamp of the API, I think it's useful that we had the discussion (again, but in a new light).

Just updated the original post scratching all the attribute ideas.

So do we then want to provide a "call" method as a faster way of getting/setting the pin:

pin() # get pin value
pin(1) # set pin value

(this will be in addition to having the pin.value method)?

Rationale: pin getting/setting is a very common operation so benefits from a shortcut (shorter to type, smaller script, smaller bytecode). Also, it's common to bit-bang using pins, and doing a call is much faster than looking up the method (pin() is roughly 3 times faster than pin.value()).

Rationale: pin getting/setting is a very common operation so benefits from a shortcut (shorter to type, smaller script, smaller bytecode). Also, it's common to bit-bang using pins, and doing a call is much faster than looking up the method (pin() is roughly 3 times faster than pin.value()).

Even though it might seem a bit adhoc, and that it also duplicates what pin.value() already does, given the advantages, I think that in this case is a good idea and we should have the call method to set/get the value of a pin.

Right now there's a pyb.mount() function which works like this:

pyb.mount(device, mount_point, *, readonly, mkfs)

I propose to move this function to the os module and remove the mkfs option. So it ends up being:

os.mount(device, mount_point, *, readonly)

and then add a separate function, also in the os module, to make a file system, i.e.:

os.mkfs(device or path, *, options...)

The required parameter can be a device just like the one accepted by os.mount or a path in case that the device is already mounted (like /flash). Maybe we need some keyword only arguments for extra option(s), maybe not.

Thoughts?

@dpgeorge

@dpgeorge asked me to create a wiki page to show how the new API will look like. I created it, and added the Pin class: https://github.com/micropython/micropython/wiki/Hardware-API

I've also implemented the new API for the WiPy. For now is in my personal repo:
danicampora@0061f9f

So, I saw https://github.com/micropython/micropython/wiki/Hardware-API (is that right place to do documentation? docs should be in RST form in repo/on a site, RFC is in tracker, not sure how overflow to wiki happens), and wonder: how did we end up with yet another "hardware" module? Last time it was discussed we seemed to agree that "machine" would be a place to transition elaborated APIs to (from "pyb").

So, I saw https://github.com/micropython/micropython/wiki/Hardware-API (is that right place to do documentation? docs should be in RST form in repo/on a site, RFC is in tracker, not sure how overflow to wiki happens),

OK, sorry, I corrected my post. The documentation itself will be done via sphinx just as before, the idea of the wiki is to show how the new API will look like before actually implementing it. It's easier to see it there than via the discussion in this PR. We can remove that link when the API is finished.

and wonder: how did we end up with yet another "hardware" module? Last time it was discussed we seemed to agree that "machine" would be a place to transition elaborated APIs to (from "pyb").

Well, the new idea is to use machine for reset, interrupt, power management, and memory access.
hardware will contain the peripherals.

machine.idle()
machine.sleep()
machine.enable_irq()
machine.reset()
machine.mem8()
...

hardware.SPI
hardware.UART
hardware.Timer
hardware.WDT
...

Don't you think it looks cleaner this way? ;-)

Don't you think it looks cleaner this way? ;-)

Not sure, I don't see clear pattern what should go into one module vs another. And having 2 modules instead of 1 takes more memory for boilerplate code.

@dhylands

@dhylands mbed support sounds really cool! I can't wait to see it.

@tannewt

@tannewt you gave me a crazy idea: write the whole machine module in Python, using a _machine C module with very minimal and unpythonic API bindings for the hardware.

@deshipu

@deshipu thats one way to do it! :-)

@dhylands

@dhylands ++1 for mbed and Teensy 3.5/3.6

@deshipu

@deshipu thanks for the feedback. Here are some comments.

I have been recently shown an alternative approach to this, which instead puts all the Python-related code in the modmachine.c file that is shared among all the ports, and which just calls into HAL library to do all the port-specific work. I think that this approach is much superior to what we have at the moment

This is "just" an implementation issue and we should separate this from the discussion of the user facing API. But see now that stmhal is using the common Python bindings from extmod for some of the machine.SPI methods.

makes any extensions or non-standard APIs go into separate modules.

The question here is: should extensions be done as extra methods / arguments to existing machine classes (as I proposed many times), or put in separate modules/classes. I think the only way the latter can work is if all extensions are functions that operate on objects from the machine module, eg stm.background_dma_transfer(spi, write_buf, read_buf). To me that is quite ugly.

I think that objects corresponding to hardware peripherals should all be
singletons, so that attempting to create a new copy merely returns an already
existing copy, possibly re-setting its internal settings.

I agree. This is how stmhal works and it should be how the other ports work as well. But for software implementations, you can create as many instances as you like (and they will all be independent).

Moreover, the information about the current state of the peripheral should be taken from the actual hardware registers

stmhal works like this. But it's mostly an implementation detail. I agree though that the "truth" should always be displayed.

Then it defines wrapper methods for all the i2c methods that it uses, basically doing little more than adding address to the call and wrapping integer arguments in 1-element bytearrays.

There is no doubt that sending/receiving a single byte is common. In the original stmhal pyb API it allows either buffers or integers (acting as single bytes) in transfer functions. When designing the machine API we thought it's best not to overload the types of the arguments like this, and also didn't want to add heaps of extra byte-methods like write_byte, read_byte. But maybe these decisions should be revisited. Probably allowing to overload the argument (either a buf or int) is a good way to do it, but that doesn't help with reading single bytes which would still need a separate method.

Therefore, I think it would make sense to have some kind of a "device" object, both for SPI and I²C, and possibly also for other protocols, that would wrap and abstract away those details.

The idea of endpoints has been discussed a few times before. Having SPI and I2C endpoint classes that have equivalent method signatures and behaviour is a good idea. Putting these classes in a separate module sounds good, but then if all drivers use such a module, that module needs to be included in all ports anyway. It's important to find a "minimal common denominator" upon which drivers are written / rely upon, so that all ports can use the drivers (otherwise we are back at square one regarding complexity and may as well put everything in the machine.SPI / I2C class to begin with).

The init method should be renamed to config and not re-set the settings that are not included as its arguemnts -- instead, it should leave them at their current values.

Yes I agree that it should not re-set all the settings. That's how some of the classes work (eg new stmhal machine.SPI, also esp8266 SPI).

Regarding the name: init() was chosen to be symmetric with deinit(), but I guess it could easily be config() and close(), and reset() could be added where appropriate.

The low-level I²C API ... The readfrom and writeto methods should be renamed to read and write
(and the readfrominto to readinto). They are the most commonly used methods and those names are pretty popular.

The reason they shouldn't be called read/write is because of duck-typing: methods should be called the same only if they behave the same. In this case, read() should behave like file.read() and write, like file.write(). Since they take address arguments they can't have the names read/write. read/write can be used for the endpoint methods where the address is already specified in the constructor.

There should be a standard way to send a device address. It could be either added to the start method

Yes I agree with this. I don't think there's any scenario where you send a start bit that's not followed by the address.

The mem operations should lose their "to" and "from", and possibly use register in place of memaddr (maybe even 'reg' instead of mem).

As mentioned by @tannewt these should probably go in the higher-level class (ie the endpoint). They could be replaced in the low-level I2C class by a single method that does write-followed-by-read: i2c.write_read(address, wr_buf, rd_buf).

The I2C.MASTER and I2C.SLAVE constants should be dropped, and instead there should be two separate classes for master and for slave.

Yes I agree, and this is how it now works (or should work). esp8266 and stmhal's machine module don't have a mode anymore (if they do it's a bug).

Software peripherals should be created with separate classes, possibly not even residing in the machine module

I think the way it's defined at the moment is pretty clean: machine.SPI(sck=p0, miso=p1, mosi=p2) creates an SPI instance on the given pins. It will use hardware if possible, else software. If you want a specific HW instance then name it, eg machine.SPI('X').

The worst offender I encountered so far seems to be the RTC module, which uses data structures which are similar, but have completely different meanings between platforms.

Yes RTC needs to be fixed. The problem is that old pyb.RTC propagated to machine.RTC and hasn't yet been cleaned up.

The ADC and Timer classes on the ESP8266 doesn't follow the established pattern of channels

How to deal with channels on these objects, in a way that works across many ports, is still an unsolved problem.

SPI constructors take varying number of obligatory arguments

Please raise issues specifically for this because I thought it was pretty much fixed now.

The question here is: should extensions be done as extra methods / arguments to existing machine classes (as I proposed many times), or put in separate modules/classes. I think the only way the latter can work is if all extensions are functions that operate on objects from the machine module, eg stm.background_dma_transfer(spi, write_buf, read_buf). To me that is quite ugly.

Well, I have seen several pull requests being shot down recently on the basis of not being compatible with the hardware API, precisely because they added extra methods, optional arguments or additional constants. If this is really the preferred way, perhaps it could be documented, so that authors of the pull requests can point to that in their defense?

As for putting the extensions in separate classes, I think that wrapping a class from machine in an "advanced" class, adding the extra methods and arguments, could work quite well -- either through inheritance, or through composition (passing an instance of the wrapped class to the wrapper's constructor). Often the "advanced" versions wouldn't even share any code with the machine versions (like the DMA stuff, which would use different calls), so they would be completely separate classes.

Theres a lot to talk about in the last few replies but I want to focus on the organization question only.

Putting all of the python related code into a single place that is port agnostic isn't just an implementation issue because it forces certain properties of an API. In particular, it forces the API to be identical across ports. I think that is a very good thing. Having identical APIs for modules of the same name is hugely beneficial in Python because it reduces the risk of RuntimeErrors. That is why I think all method and class definitions should be shared. Consistency is hugely important.

The low-level I²C API ... The readfrom and writeto methods should be renamed to read and write (and the readfrominto to readinto). They are the most commonly used methods and those names are pretty opular.

The reason they shouldn't be called read/write is because of duck-typing: methods should be called the same only if they behave the same. In this case, read() should behave like file.read() and write, like file.write(). Since they take address arguments they can't have the names read/write. read/write can be used for the endpoint methods where the address is already specified in the constructor.

Duck typing applies to interfaces that do the same, or an equivalent thing. An i2c.read() doesn't and can't possibly do the same thing as file.read(). You can't just pass an I2C bus object to something expecting a file and expect that to work -- the whole protocol is based on transactions, and that doesn't map to UNIX file semantics. It could perhaps work if you had that "I2CDevice" endpoint class. Perhaps we should go back to i2c.recv() or something similar then? That's at least consistent with what Arduino had.

It would be great if Micropython would use this opportunity to also introduce CAN (Controller Area Network) support. Over at python-can, we figured that our library is probably not the right place to provide that support, but this project is. (see python-can #93)

Hello,

Dose anyone see a chance to use any USB-TO-CAN adapter (maybe over python-can) on a micropython windows build?

many thanks

These questions will find a bigger audience in the forum. There are people using CAN bus with the Pyboard which has two CAN channels.

okay many thanks.

Merge pull request micropython#1430 from adafruit/header-logo

I'm going to close this issue, I don't think there's anything to gain from keeping it open. The machine module has been around for a while and some classes are settled, while others are still being developed, and their development/discussion should happen in separate, dedicated issues.

Navigation Menu

Repository navigation

New MicroPython hardware API #1430

302 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

New MicroPython hardware API #1430

Description

Activity

dpgeorge commented on Aug 16, 2015

danicampora commented on Aug 16, 2015

dhylands commented on Aug 16, 2015

dpgeorge commented on Aug 17, 2015

danicampora commented on Aug 17, 2015

pfalcon commented on Aug 17, 2015

dpgeorge commented on Aug 17, 2015

danicampora commented on Aug 17, 2015

dpgeorge commented on Aug 17, 2015

danicampora commented on Aug 17, 2015

danicampora commented on Aug 18, 2015

danicampora commented on Aug 20, 2015

pfalcon commented on Aug 20, 2015

danicampora commented on Aug 20, 2015

pfalcon commented on Aug 20, 2015

302 remaining items

tannewt commented on Oct 12, 2016

deshipu commented on Oct 12, 2016

tannewt commented on Oct 12, 2016

profra commented on Oct 13, 2016

dpgeorge commented on Oct 18, 2016

deshipu commented on Oct 18, 2016

tannewt commented on Oct 18, 2016

deshipu commented on Oct 20, 2016

felixdivo commented on May 30, 2018

mgiaco commented on Jul 23, 2018

peterhinch commented on Jul 24, 2018

mgiaco commented on Jul 24, 2018

dpgeorge commented on May 21, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions