New port PSOC Edge - #18554

Code size report:

Reference:  stm32/boards: Add linker script for STMF412xE with 512k flash. [4e98486]
Comparison: tools/psoc-edge/mpy-pse.py: Corrected spelling mistake. [merge of d11675c]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (78ff170) to head (d11675c).
⚠️ Report is 75 commits behind head on master.

@@            Coverage Diff             @@
##           master   #18554      +/-   ##
==========================================
- Coverage   98.38%   98.38%   -0.01%     
==========================================
  Files         171      171              
  Lines       22300    22298       -2     
==========================================
- Hits        21939    21937       -2     
  Misses        361      361              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hi @jaenrig-ifx , thanks for making this PR.

I have taken a first pass look over it and have the following high-level comments.

1. I'm unable to download any of the URLs in the README.md for the Modbus Toolbox. They all end up in the following error page. Are the URLs out-of-date?

2. What is the mpy-test-ext submodule for? Does it need to be included here? Please note that we have our own test runners for multiple targets, see tests/run-multitests.py. It would be best to use the existing test runners if possible.
3. Is it possible to enable the CI here in this PR so we can see it building the new psoc-edge port? Maybe not all the CI can be enabled, but at least compiling one board should work.

Hi @dpgeorge,

Thanks a lot for checking out this PR :)

1. Interesting... You mean links like this one: https://softwaretools-hosting.infineon.com/packages/com.ifx.tb.tool.modustoolbox/versions/3.6.0.17979/artifacts/modustoolbox_3.6.0.17979_Linux_x64.deb/download, isn't it?
   We've tried accessing it in multiple ways (browsers, networks, regions, machines, ...) and it works in all cases. In case it was a momentary network issue or the server was down, could you please try again?
   If it still doesn't work, we'll need to investigate further with the ModusToolbox team. I apologize :|

2. This is a script we use to manage the CI matrix (board vs tests) of our hardware-in-the-loop test setup. It already reuses the tests/run-tests.py and tests/run-multitests.py scripts internally. Since this script is only used in our fork, we can definitely remove the submodule from this PR.
   More details: In .github/workflows/ports_psoc-edge.yml, the HIL checks are only executed if the workflow is run within our fork/organization. In that stage, the mpy-test-ext utility is used -> https://github.com/Infineon/micropython-psoc-edge/blob/psoc-edge-pr1/.github/workflows/ports_psoc-edge.yml#L85

3. The CI build is already running in the PR -> https://github.com/micropython/micropython/actions/runs/20162464676/job/57877853930?pr=18554. As mentioned before, the non-relevant jobs and stages are disabled (they will only work under the Infineon organization). Or do you mean something different?

Interesting... You mean links like this one:

Yes. If I click that link in Firefox or Chrome I get: Application is not available. The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

Interesting... You mean links like this one:

Yes. If I click that link in Firefox or Chrome I get: Application is not available. The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

Same issue from Amsterdam

Interesting... You mean links like this one:

Yes. If I click that link in Firefox or Chrome I get: Application is not available. The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

Same issue from Amsterdam

Okay, it also does not work here (also Netherlands ;)) anymore.

It seems there is some maintenance work or issue with the Infineon Developer Center:

Very timely... 😓

Currently, we don´t have information about when this will be fixed. I expect this is solve ASAP (< hours, < days ?).
Let´s wait some time before the next attempt.

My apologies again 🙏

2. This is a script we use to manage the CI matrix (board vs tests) of our hardware-in-the-loop test setup. It already reuses the tests/run-tests.py and tests/run-multitests.py scripts internally. Since this script is only used in our fork, we can definitely remove the submodule from this PR.

OK, thanks for the explanation. I would then suggest removing it, in order to simplify this PR.

3. The CI build is already running in the PR

Aaah, you are right! My mistake.

I see it's using docker to set up the build environment. That is then an alternative to downloading the Modbus Toolbox. Maybe using docker should be mentioned in the README, and perhaps be the preferred way of building? Although it's still good to be able to set up a build environment from scratch using Modbus packages.

2. This is a script we use to manage the CI matrix (board vs tests) of our hardware-in-the-loop test setup. It already reuses the tests/run-tests.py and tests/run-multitests.py scripts internally. Since this script is only used in our fork, we can definitely remove the submodule from this PR.

OK, thanks for the explanation. I would then suggest removing it, in order to simplify this PR.

Fine, I´ll do so👍

3. The CI build is already running in the PR

Aaah, you are right! My mistake.

I see it's using docker to set up the build environment. That is then an alternative to downloading the Modbus Toolbox. Maybe using docker should be mentioned in the README, and perhaps be the preferred way of building? Although it's still good to be able to set up a build environment from scratch using Modbus packages.

Right, that is another alternative.

I'd like to give it a try to see if there are any inconveniences during the local development flow, and polish it a bit.
I don't foresee any big issues.

This is also part of our technical debt, as with the other PSOC6 port, we could reuse the same ModusToolbox toolchain container, and maybe some of the tools/ci.sh functions, etc. We will need to spend some time on this.

Hi @jaenrig-ifx . I now have a PSOC Edge E84 AI Kit dev board. It's a really neat board with lots of functionality!

I tried to get this PR building and deployed on that board, but I had some troubles. I'm using Arch Linux so can't directly follow the README to get the Modus components installed, but I did manage to make some good progress (I could start up an Ubuntu VM, but I wanted to see if it could work with Arch Linux).

Steps taken:

1. Install Modus Toolbox from the AUR: https://aur.archlinux.org/packages/modustoolbox , that seemed to work.
2. Install the Edge Protect Security Suite 1.6.0 by downloading the URL in the README, converting the .deb file to an Arch package using debtap and install it. That also seemed to work.
3. Install the Modus Programming tools 1.5.0 using debtap as in step 2 above.
4. I used the arm-none-eabi-gcc compiler that comes with Arch Linux.
5. export CY_TOOLS_PATHS=/opt/ModusToolbox/tools_3.6
6. export CY_COMPILER_GCC_ARM_DIR=/usr
7. export CY_TOOL_edgeprotecttools_EXE_ABS=/opt/Tools/ModusToolbox-Edge-Protect-Security-Suite-1.6/tools/edgeprotecttools/bin/edgeprotecttools
8. add /opt/Tools/ModusToolbox/tools_3.6/library-manager to $PATH
9. Run make BOARD=KIT_PSE84_AI.

This is successful! I could build the board. It's a bit awkward setting everything up, but it does work.

Then I tried to deploy it using make BOARD=KIT_PSE84_AI deploy, but that did not work, it gives:

...
Toolchain validation: PASS
Initializing build: proj_cm33_s Debug APP_KIT_PSE84_AI GCC_ARM

/bin/bash: line 2: : command not found
make[2]: *** [../../mtb_shared/mtb-dsl-pse8xxgp/release-v1.2.0/make/make/recipe/program_common.mk:151: qprogram_KitProg3] Error 127
...

I think that it cannot find the Modus programming tool. How does the toolchain find the programming tool, are there some environment variables that I need to set?

I also noticed a lot of these warnings:

egrep: warning: egrep is obsolescent; using grep -E

Hi @dpgeorge,

Thanks for checking out the port again ☺️.

Unfortunately, we have not tried out building MicroPython on Arch Linux. Neither the just ModusToolbox toolchain itself.
So I wonder if it is fully supported on this Linux distribution...

You mention that you installed The Programming tools in step 3. But it seems that some tools are not yet in the system path?

If possible I would try in Ubuntu.

Thanks!

I eventually managed to work out how to deploy the signed firmware to the KIT_PSE84_AI, using Infineon's custom version of OpenOCD. Doing that I got a REPL over UART on the board! So that's great.

I then also tried out MicroPython's zephyr port, and was able to build that for the KIT_PSE84_AI as well (using Zephyr 4.3, which supports this board). And after deploying that build to the board I got a REPL as well.

Note that zephyr does not need the Modus Toolbox to be installed, although it does still need the edgeprotecttools and custom OpenOCD. Instead of Modus it uses a set of git repositories with the relevant Infineon SDK code. These repositories are hosted by Infineon on GitHub, although Zephyr makes their own custom repo which combines many of the Infineon ones into a single, big repo.

@jaenrig-ifx did you consider using the zephyr MicroPython port running on the PSOC Edge, instead of creating a new bare-metal port? A bare metal port does have its benefits, but leveraging zephyr would mean that a lot of the work is already done.

Regarding this PR itself. The way the Modus Toolbox is set up and used, and the way the build is done, is rather complicated. The things that make it difficult are:

1. You need to download 4 separate packages (that really only work properly on Ubuntu), and the Modus Toolbox components are nearly 3GiB.
2. There is one submodule added in this PR, but when you build the firmware it downloads a whole lot of other git repositories and puts them in lib/mtb_shared. This is unconventional, you shouldn't need internet connectivity when you type make, and there shouldn't be any temporary files put in lib/.
3. After building, a lot of the build artefacts (the .o files) are placed inside the submodule at lib/mtb-psoc-edge-libs. This is unconventional (for this project), there should never be any stray files added to a submodule. All build artefacts should go in the output build directory for the board.

I was curious to see if all these issues could be overcome, so I made my own, minimal PSOC Edge port. See #18843. That has all the necessary SDK code as submodules (similar to Zephyr), uses standard arm-none-eabi-gcc to build, and puts all output files in the build directory.

At this point there are three different approaches to the PSOC Edge port:

1. This PR that's based on Modus Toolbox (with the advantage that most of the complexity of the chip and its configuration is handled by Modus Toolbox).
2. The existing MicroPython zephyr port, built for KIT_PSE84_AI.
3. The port in ports: add new PSOC Edge port (proof of concept that doesn't use ModusToolbox) #18843 that uses submodules and standard make instead of Modus Toolbox.

@jaenrig-ifx what is your opinion on these three different approaches?

From my point of view as a maintainer here, this PR would not be ready for merging without addressing point (3) above (making sure all build artefacts are in the build directory) and ideally also point (2) (not downloading during the build process).

Dear @dpgeorge ,

Thank you for your reply. I am happy to see all the energy going into the PSOC Edge board.

About enabling PSOC Edge through the Zephyr port

We are not considering the Zephyr approach. The enablement for the PSOC Edge might not cover the features that we would like to support in MicroPython.

If I am not mistaken, it seems only a small subset of the features have been enabled so far: UART, GPIO, clock and memory.

https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/infineon/kit_pse84_ai/kit_pse84_ai_m55.yaml
https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/infineon/kit_pse84_ai/kit_pse84_ai_m33.yaml

Besides, our (working group's) influence on the Zephyr enablement is constrained, and the bare-metal port gives us the autonomy to grow the MicroPython support without dependencies on the Zephyr roadmap and priorities.

Still, I don't see this as a mutually exclusive alternative to the bare-metal port. The Zephyr approach is always there as an alternative, and of interest depending on the use case and the level of enablement for each MCU/board.

About why we are using ModusToolbox and removal concerns

Very valid points and concerns. We are also aware of this and not fully satisfied with the ModusToolbox mtb-psoc-edge-libs submodule integration flow.

The reason we opted for this approach is mainly to delegate the Board Support Package (BSP) dependencies management and the integration of middleware libraries to ModusToolbox.
This process involves a custom proprietary system for managing dependencies beyond git version control, additional tools for custom configuration and code generations, and other intricacies that we are trying to avoid dealing with.

My concern with getting fully rid of ModusToolbox is that it may become a bit more troublesome to maintain and keep up to date, when trying to support multiple boards and a large number of middleware libraries.

About next steps and your requirements for merge

Anyhow, the current ModusToolbox approach isn't a hard requirement for us. Actually, easing the development and maintenance efforts is part of our goal. You already provided this new approach with that in mind, so give me some days to go through it, and find out more about its implications.

Upfront, the requirement of bringing the build artifacts to the build directory does not seem hard to fulfill, even with the ModusToolbox in place, that would be just copying the output files to the MicroPython build directory.
Or am I seeing it too simplistic?

About the second requirement, not downloading the packages during the build. Technically, they are downloaded during the project settings.
Subsequent builds of the same board won't require any download.
I guess that technicality does not make a difference, but just wanted to clarify it.

Definitively, adding each lib as a submodule is doable. Then:

One of the issues I foresee is when/if multiple boards use different versions of the same library. Do you have such a situation in any port? And if so, how are you handling it?

Another topic is all the code generation and configuration. This needs to be hardcoded and imported manually according to the BSP and subsequent potential modifications.
I assume that will then require using ModusToolbox regularly, at least until the maturity of the port development might minimize the need for code generation and configuration changes.
Thus, getting rid of ModusToolbox might not be fully achievable.

As said, let me check your PR, and find out about the best ways to move forward 😊

I am happy to see all the energy going into the PSOC Edge board.

Yes, I did spend quite some time with this PR, building using Zephyr, and creating my minimal port. Overall that has taught me a lot about this chip and the way it and the build system works.

The PSOC Edge chip itself is really nice and I would definitely like to see it supported in MicroPython!

Regarding Zephyr

We are not considering the Zephyr approach.
...
our (working group's) influence on the Zephyr enablement is constrained, and the bare-metal port gives us the autonomy to grow the MicroPython support without dependencies on the Zephyr roadmap and priorities.

I fully understand this point. A bare-metal port definitely gives more freedom. That's why we have a lot of bare-metal ports like stm32, mimxrt, samd and rp2.

Still, I don't see this as a mutually exclusive alternative to the bare-metal port. The Zephyr approach is always there as an alternative,

Yes that's very true.

OK, so the case for a bare-metal PSOC Edge port is clear, and we will concentrate on that.

Regarding ModusToolbox

We are also aware of this and not fully satisfied with the ModusToolbox mtb-psoc-edge-libs submodule integration flow.

OK, so at least we agree that there are things to improve in this PR with respect to the build system. As I mentioned, to help with ongoing maintenance and support within this repo, the closer this new PSOC Edge port aligns with existing ports, the better.

The reason we opted for this approach is mainly to delegate the Board Support Package (BSP) dependencies management and the integration of middleware libraries to ModusToolbox.
...
My concern with getting fully rid of ModusToolbox is that it may become a bit more troublesome to maintain and keep up to date, when trying to support multiple boards and a large number of middleware libraries.

Yes, that's definitely a valid reason to use ModusToolbox, to easily support multiple boards and libraries.

As two counterpoints to that:

1. We also need to consider keeping the port up to date with the rest of MicroPython. When things in MicroPython change or are added (eg a new wireless driver, improved IPv6 support, etc) the closer all ports are aligned in their implementation, the easier it is to change everything at once and keep the ports consistent.
2. I would hope that most of the middleware libraries can be provided by MicroPython, rather than by ModusToolbox. Eg filesystem drivers like FAT and littlefs, networking integration, Bluetooth support, USB via TinyUSB, among many other things.

I think we should discuss in more detail point (2). For example TCP/IP: we use lwIP as the TCP/IP library and have a relatively mature (non-OS) integration of that with the bare-metal ports. It works very well, is pretty efficient, and we have spent a lot of time to make sure the socket module in Python acts like it does on a desktop PC. Then on top of that sit all the other networking libraries, like requests, iperf3, microdot etc. It would be ideal if the PSOC Edge port could also use the lwIP integration provided by MicroPython, because then all the networking things would "just work".

Similarly with BLE, all the port needs to provide is a HCI UART/stream interface, and then MicroPython does the rest: using NimBLE or BTstack, and then our bluetooth module bindings, and then the aioble library on top of that. We really need to leverage all that existing infrastructure.

Hopefully that doesn't sound restrictive to you. After all, you didn't want to use the Zephyr port because you wanted more freedom with adding features. But I think there's still a lot of freedom to be had in a bare-metal PSOC Edge port, even if things like networking and BLE are provided by MicroPython.

Next steps

Upfront, the requirement of bringing the build artifacts to the build directory does not seem hard to fulfill, even with the ModusToolbox in place, that would be just copying the output files to the MicroPython build directory. Or am I seeing it too simplistic?

Simply moving files could work. But isn't it possible to pass some setting through to ModusToolbox for the output location?

About the second requirement, not downloading the packages during the build. Technically, they are downloaded during the project settings.
Subsequent builds of the same board won't require any download.

Maybe there can be a separate step (eg make BOARD=PSE84E_KIT_AI prepare) which does this download and only needs to be done once? But still that's not ideal.

Definitively, adding each lib as a submodule is doable. Then:

One of the issues I foresee is when/if multiple boards use different versions of the same library. Do you have such a situation in any port? And if so, how are you handling it?

Adding all required source as a submodule is definitely preferable.

That seems awkward if different boards use different versions of a library. Do you have an example of that, of such a library? Can't we just make sure everything steps forward with new library versions in sync?

As a related example, our lwIP bindings (in extmod/modlwip.c) do support multiple versions of lwIP. But we only have a single lwIP version in the submodule at lib/lwip.

Another topic is all the code generation and configuration. This needs to be hardcoded and imported manually according to the BSP and subsequent potential modifications.

Yes, it seems the BSP code is the hardest point to address. As an example, the mimxrt and renesas-ra ports do have generated BSP files that are committed verbatim to this repo. If you look at a renesas-ra board, eg ports/renesas-ra/boards/ARDUINO_PORTENTA_C33, you can see that each one contains BSP code.

Maybe we can have a separate make target (eg make BOARD=PSE84_KIT_AI generate-bsp) that generates the required files. And the files are committed to the repo so generating them is an optional step and only needed when they change.

Closing this PR. Continued in #18910.