Repository navigation
Framework for coherent eth/wlan drivers in MicroPython #876
Description
Activity
Commmon stuff you'll need to be able to set for each device:
ip address, mask, gateway, and metric (all needed for static IP)gethostbyname isn't normally a per-interface thing. It's a per-machine thing. You would normally check your local cache, check /etc/hosts, and then check with the DNS server using whatever the primary interface is (normally each interface also has a metric and the one with the lowest metric is usually considered to be the "primary" one. So while you may have a DNS server per interface, you only use the one from the primary interface.
Things that you'll need to be able to do with wlan versus eth are things like setting the SSID to connect to, encryption & password, scanning for APs.
I'd consider sicking with the standard sockets api and keep the network stuff seperately for a number of reasons: it's a defacto standard, has been in use for years, is well-known even to beginners and allows to simply copy-paste socket samples, or applied to uPy: write your socket stuff on unix and then move it to the board without having to change code, just add network configuration. It maintains proper seperation of underlying platform-specific configuration and the platform-agnostic socket api. Im not saying that in general because something has been done for ages it is the best thing to do, but in this case this does apply imo.
The implementation disadvantage is of course that for client connections (using connect()) you cannot specify which of the network adapters to use using sockets (for server connections() the implementation of bind() should take care of that). On most userland OS this is implemented by checking the destination address of the connections vs the routing table and choose the most suitable interface. Possible solutions for pyBoard: IIRC calling bind() defore connect() doesn't hurt, just has no effect on typical OS, so pyBoard could implement it for client sockets in the same way it does for server sockets. Might be somewhat confusing. Another option is specifying a sort of routing table in the network module: before connecting, the code can tell the network module 'when connecting to this address (or range), use this interface'. Easily allows specifying defaults, comes closest to other OS. Another option, a bit messy though, is setting the 'active' NIC for the socket calls following it. These 3 options in pseudocode:
cc3k = network.CC3k(pyb.SPI(1), pyb.Pin.board.Y5, pyb.Pin.borad.Y6) wiz = network.WIZnet5k(...) #bind specifies address s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind((localAddress, ignoredLocalPort)) s.connect.. #uses NIC which has address equal to localAddress #routing table style network.routingTable[ default ] = cc3k; network.routingTable[ destinationAddress1 ] = cc3k; network.routingTable[ destinationAddress2 ] = wiz; s.connect((destinationAddress1, port)) #uses cc3k #active NIC syle network.setActive( cc3k ) s = socket.socket..... #uses cc3k network.setActive( wiz ) s = socket.socket..... #uses wizThanks for the feedback.
The point I made about making each driver (cc3k, wiznet, ...) behave like the socket module was so that you could use existing Python code (and knowledge of socket calls). But you'd need to make 1 change, namely use the driver object instead of importing socket. I guess even this 1 change is asking too much, and we want 100% compatibility with Python in this respect.
In than case I'd vote +1 for the routing table style.
+1 from me for routing-table style. Then network drivers will have a C-level API, and all multiplexing (routing) among them will be handled by socket (usocket) module. But what type of this C API should be is a bit hard to tell out of top of mind. As I already mentioned, it would nice to consider drivers for both "builtin stack" (e.g. wiznet) and "raw packet" (slip would be simplest) devices.
- added a commit that references this issue
on Sep 30, 2014 The basics of this are now implemented, in the network and usocket modules in stmhal. Some renaming and cleanup still to do.
This has been implemented in stmhal/ port as above, but it has some design flaws.
You need to know the IP address for connect/bind/sendto before opening a socket, in order that you can select the right NIC from the routing table. But what about setsockopt? Eg:
import socket s = socket.socket() # can't select NIC yet because don't know anything about the socket s.setsockopt(...) # still can't select NIC, so what to do with options?? s.connect(('192.168.0.1', 80)) # now we can select NIC, but then options are not applied
Yes, we could store all options and apply them after the socket is opened, but that seems very wasteful in RAM and leads to very messy code.
Can you show an example when setsockopt would need to be set up on fresh socket?
POSIXy way to resolve that would be bind() before calling .setsockopt(). Address to bind to would be injected from outside (e.g. specified by user). As that's not always possible, some alternative solution may be needed, like creating socket objects from adapter object, as you proposed in starter message.
Let's discuss how to name function to scan for WiFi APs. In wifi terminology that's usually called "scan", but maybe "list_ap", or "list_aps", or "scan_aps" are better names?
And then maybe what interface it should have? (Though for esp8266 it will take callback for starters).
Let's discuss how to name function to scan for WiFi APs. In wifi terminology that's usually called "scan", but maybe "list_ap", or "list_aps", or "scan_aps" are better names?
On linux I do "iwlist wlan0 scan". There is also an I2C.scan() method on pyboard to scan the I2C bus for device addresses. So "scan" keeps it short and simple, and if you have a wifi object it looks like "wifi.scan()" which is pretty obvious what it does :)
And then maybe what interface it should have? (Though for esp8266 it will take callback for starters).
wifi.scan() I guess. For ESP,
wifi.scan(callback=lambda ap:print(ap)).On linux I do "iwlist wlan0 scan"
Yep, that was the inspiration.
So "scan" keeps it short and simple,
Yes, it's also more reusable to technologies which don't use "access point" technology, like you point out with i2c. So, let's go with it.
We'll also need to agree on basic data structure for results. Implementation for esp8266 I have surely uses tuple, but that doesn't scale (different devices will supply different amounts and types of info), so apparently that should be dict with some basic keys defined.
33 remaining items
that devs actually stick to the agreed upon convention.
Yes, and that's how I hoped that it worked all this time (not one month by now) - that we'll discuss it and come to conclusions, then someone on the forefront of this work will implement it, to serve as actual code template for other ports. In my list, WiPy development is on this forefront, but I don't see much much of what discussed being implemented there. And I don't say that for finger-pointing, it's perfevtly clear that @danicampora has a lot of other important things to do for his port. In the meantime, we get submissions for esp8266 which either need to be ignored, rejected, or reworked. I finally take a step to lay good base for esp8266 by implementing all the latest ideas myself, take a final peek to compare WiPy and pyboard drivers, and see that not only WiPy doesn't implement latest ideas (which is know), but that it's drivers based on older ideas is inconsistent with your original implementation of pyboard drivers (which are kinda news).
So, "important [...] that devs actually stick to the agreed upon convention" doesn't work. And I don't blame devs, I blame conventions, which apparently too tangled and vague, and propose soemthing which can easily fits onto one head, so it will be easy to say what's right and what's not going forward.
We should decouple these things: design the API so it's Pythonic and obvious for the user
I don't see anything unpythonic in what I propose (again: query params one by one as a baseline), and that can be only more obvious to user than potentially long structures.
And that effort only needs to be spent once.
No, it will need to be spent again and again going forward, for each newly written or contributed drivers. So, the more complicated scheme we have, the more complicated it will be.
What do we actually need to configure?
The most generic answer is "anything", and mys scheme allows to configure anything. For example, adhoc .patch() method in some driver can be replaced with .config(patch=bytes).
How about we design a mock API for wipy and esp8266 independently, with all the required config values that they need/want, and then see where we stand.
We already have mock of API there. We already spent a lot of time discussing how it should be, with ~zero outcome so far. I already spent a whole coding session on that, usual outcome of that should be newly implemented or cleaned up functionality, but this time, outcome if zero (well, yet another proposal on how to get it right; well, final on my side). I don't see all that as sustainable or rewarding, given the context of this API (use once and forget), the fact that I don't consider working on esp8266 too beneficial as it stands, that there're other issues in WIP many of which are more important and interesting, etc.
So, the only way I see is to do this KISS and get over it.
In my list, WiPy development is on this forefront, but I don't see much much of what discussed being implemented there.
Apart from the things that we already know, what else do you see in the WiPy that doesn't follow what we have discussed?
On Jun 27, 2015, at 5:08 PM, Paul Sokolovsky [email protected] wrote:
that devs actually stick to the agreed upon convention.
Yes, and that's how I hoped that it worked all this time (not one month by now) - that we'll discuss it and come to conclusions, then someone on the forefront of this work will implement it, to serve as actual code template for other ports. In my list, WiPy development is on this forefront, but I don't see much much of what discussed being implemented there. And I don't say that for finger-pointing, it's perfevtly clear that @danicampora has a lot of other important things to do for his port. In the meantime, we get submissions for esp8266 which either need to be ignored, rejected, or reworked. I finally take a step to lay good base for esp8266 by implementing all the latest ideas myself, take a final peek to compare WiPy and pyboard drivers, and see that not only WiPy doesn't implement latest ideas (which is know), but that it's drivers based on older ideas is inconsistent with your original implementation of pyboa rd drivers (which are kinda news).
So, "important [...] that devs actually stick to the agreed upon convention" doesn't work. And I don't blame devs, I blame conventions, which apparently too tangled and vague, and propose soemthing which can easily fits onto one head, so it will be easy to say what's right and what's not going forward.
We should decouple these things: design the API so it's Pythonic and obvious for the user
I don't see anything unpythonic in what I propose (again: query params one by one as a baseline), and that can be only more obvious to user than potentially long structures.
And that effort only needs to be spent once.
No, it will need to be spent again and again going forward, for each newly written or contributed drivers. So, the more complicated scheme we have, the more complicated it will be.
What do we actually need to configure?
The most generic answer is "anything", and mys scheme allows to configure anything. For example, adhoc .patch() method in some driver can be replaced with .config(patch=bytes).
How about we design a mock API for wipy and esp8266 independently, with all the required config values that they need/want, and then see where we stand.
We already have mock of API there. We already spent a lot of time discussing how it should be, with ~zero outcome so far. I already spent a whole coding session on that, usual outcome of that should be newly implemented or cleaned up functionality, but this time, outcome if zero (well, yet another proposal on how to get it right; well, final on my side). I don't see all that as sustainable or rewarding, given the context of this API (use once and forget), the fact that I don't consider working on esp8266 too beneficial as it stands, that there're other issues in WIP many of which are more important and interesting, etc.
So, the only way I see is to do this KISS and get over it.
—
Reply to this email directly or view it on GitHub.Apart from the things that we already know, what else do you see in the WiPy that doesn't follow what we have discussed?
Unfortunately, I don't follow WiPy development too close. When I spot anything worth communicating, I do, like that case with time.sleep() taking milliseconds. And anyway, the point of this discussion is not WiPy specifically, but unbreaking networking interfaces.
I finally got time to implement the new proposed constructor for
WLAN. See this commit:
danicampora@0a99cf8If no objections, I will push to MicroPython master later. Examples:
from network import WLAN wifi = WLAN() wifi.iwconfig(mode=WLAN.STA) wifi = WLAN(mode=WLAN.AP, security=WLAN.WPA2, key='123456abc', channel=2, antenna=WLAN.INTERNAL) wifi.iwconfig(channel=4) # only change the channel, keep the rest as it is. # get the current config (namedtuple) wifi.iwconfig()
I also added an extra method to just get the MAC address
nic.mac()nic.ifconfig()remains as it was, it takes a 4-tuple with(ip, netmask, gateway, dns), or 'dhcp`. With no params the 4-tuple with the current config is returned.Thoughts?
@danicampora : Well, I guess it's better than was before. As for ultimate solution, it seems to be hanging in the air (at least that how I see it), so doing something is again better than waiting.
@pfalcon thanks. I guess you think it's OK, but it is still missing something. Feel free to propose changes.
EDIT:
From what you proposed during the long discussion of this ticket, he only thing that I think my implementation doesn't cover is the option to GET config params independently, they can be SET independently though.@danicampora : Well, for WiPy, it should be ok. For MicroPython as a whole, I gave my proposals above.
From what you proposed during the long discussion of this ticket, he only thing that I think my implementation doesn't cover is the option to GET config params independently, they can be SET independently though.
Yes, and having cleaner/easier naming of functions to get/set, and fewer such get/set functions.
- added a commit that references this issue
on Aug 9, 2015 There is now a relatively mature "network" module with a definition of its methods given by the docs.
We need a standard way of using internet modules on the pyboard that is powerful, extensible and easy to use. Powerful means being able to attach an ethernet and wifi module at the same time (and possibly multiple ones of these); extensible means we should be able to add new drivers without changing existing ones; easy to use means it should behave as expected for someone familiar with the Python socket module. Note that higher level abstractions (like http servers) can be built upon a proper functioning socket interface.
Let's first briefly review how things are done in Python and on a POSIX system. Python has the socket module which allows you to make socket objects, and do things like gethostbyname:
This is just a wrapper for the underlying POSIX calls.
The above calls don't know anything about the underlying internet device (be it eth0 or wlan0 for example). That, and the configuration of an IP address, is left up to the operating system.
On the pyboard, uPy is the operating system, and so we need a way (in Python) to initialise and configure the eth/wlan drivers, and also a way to select which one we want to use for, eg, a gethostbyname call.
A scenario might be: I have a pyboard with a CC3000 (wifi) and a WIZnet (eth) module connected. I need to initialise them on the correct pins, then perhaps run DHCP on the CC3000, but set a static IP for the WIZnet. I want the CC3000 to be a server and the WIZ to be a client, and I need to open sockets on both.
My proposed solution is to have a "meta socket" module which contains a class for each driver that is available. To initialise a driver, you make a new instance of the associated class. The instance object then behaves like a CPython socket module, and calling socket on this class makes a socket associated with that driver. Each driver has its own gethostbytname, etc.
Initialised drivers also register themselves with the "meta socket" module, so, for example, you can get a list of all initialised internet drivers (something like ifconfig on linux).
I thought about splitting into eth and wlan modules, but I think it makes sense to put all drivers into one module, since the application shouldn't care if a connection is wired or wireless.
Adding a new driver would be as simple as writing a new class and putting it into the "meta socket" module.
I propose calling the "meta socket" module
network. An example use would be:Comments, thoughts, suggestions?