kyrofa's blog

How to use RubyGems Trusted Publishing with Semantic Release

Mon, 07 Sep 2026 00:00:00 +0000

“Can I safely use this new release?”

When you’re a developer maintaining a piece of software, this is a question you ask every time you update one of its dependencies. How do you answer that question? Read the CHANGELOG? Release notes? Pray? No, you look at the version (first, at least). How to interpret that version generally depends on the project, but I’m personally partial to semantic versioning. I think it serves as an elegant contract between dependency author and consumer that can be used to easily answer this question.

That said, I don’t actually care about the version itself, I just care about the contract. Bugfixes bump the patch version, features bump the minor version, breaking changes bump the major version. Actually deciding when to do a release, determining the version to use, and then DOING that version bump and release is super boring. The same critique goes for the release notes. Thankfully, with a little commit discipline, both problems can be solved the same way: conventional commits. If each commit can be used to determine if a given change is a bugfix/feature/breaking change, then you can automate the rest with something like Semantic Release. That’s what I’ve been doing for the better part of a decade, and I can’t recommend it enough.

Historically, publishing a gem to RubyGems automatically from CI was easy: you create an API token scoped to specifically publish that gem, and use that token in your CI environment to push the gem. However, that token is one of the targets for supply chain attacks. Also, using tokens like that requires that your account doesn’t utilize multi-factor authentication (MFA), which you should definitely be using. So what is the solution? Trusted Publishing.

RubyGems Trusted Publishing

Trusted Publishing is pretty well documented elsewhere, so I don’t want to dwell on the details here for too long. For our purposes, let’s gain a high-level understanding of how this works by contrasting it with the API token approach mentioned above.

Without trusted publishing:

Create API token, scoped as tightly as possible to just pushing this gem
Put it in CI variables in as secure a manner as possible (e.g. only defined on your release branch(es), etc.)
When releasing the gem, use that variable to authenticate to RubyGems, and push the gem

Repeat (3) every time you need to release a new version. You can see how, if some part of your supply chain is compromised (e.g. your dependencies, or some part of your build system), that key can be stolen every time (3) runs. And since it’s a long-lived key, the attackers can use it until you happen to detect that it has been compromised. Yes, thanks to your carefulness in (1) the damage they can do is limited to your gem, but they can still turn your gem into an attack vector for your users by pushing malicious versions of it.

With trusted publishing:

Go to your gem’s settings in RubyGems, and add a new Trusted Publisher. This is identified by your GitHub repository, GitHub Actions workflow file, and environment name. You’re basically saying “allow this gem to be published from this GitHub Action running in this environment.”
When releasing the gem (from the workflow and environment you specified in (1)), obtain a new token from RubyGems, use it to authenticate, and push the gem

Repeat (2) every time you need to release a new version. On the surface, perhaps this doesn’t sound so different from the “without trusted publishing” flow, but it is: the token you fetch in (2) is incredibly short-lived. Even if CI is compromised, the window of opportunity for utilizing the stolen token is vanishingly small. This method of authenticating and pushing new releases is far superior.

How do we automatically push new gem versions with Trusted Publishing?

You may have noticed that step (2) above, with the “with trusted publishing” flow, was uselessly high level. I could have written “Magic happens here.” That’s because the process of actually contacting RubyGems to obtain that short-lived token is complicated enough to not really be intended to be done by you, dear developer. RubyGems kindly created a GitHub Action specifically for this, called release-gem. That is nifty, but to quote the end of the documentation:

With the trusted publisher configured and this workflow in your repository, you can release a new version of your gem by simply pushing a git tag. No API tokens, no manual gem push — just tag and push.

If you read the intro to this article, you’ll know that’s a problem for me. In order for me to push a tag, I have to select a version. I have to write the CHANGELOG. I have to do all the boring stuff that I don’t want to do. I want Semantic Release doing this for me! Can I make them work together?

Integrate Semantic Release with Trusted Publishing

First, we need to gain a better understanding of what exactly release-gem does. To glean those details we need to dig into its action.yml, which at the time of this writing looks like this:

name: "Release Gem"
# snip...
runs:
  using: "composite"
  steps:
    - name: Attribute commits to last committer
      run: |
        git config --global user.email "$(git log -1 --pretty=format:'%ae')"
        git config --global user.name "$(git log -1 --pretty=format:'%an')"        
      shell: bash
    - name: Configure Git using cached credentials
      run: |
        git credential-cache --timeout=300 store <<EOF
        protocol=https
        host=github.com
        username=x-access-token
        password=${{ inputs.token }}

        EOF
        git config --local credential.helper 'cache --timeout=300'        
      shell: bash
    - name: Fetch tags
      run: git fetch --tags --force
      shell: bash
    - name: Configure trusted publishing credentials
      if: ${{ inputs.setup-trusted-publisher == 'true' }}
      uses: rubygems/configure-rubygems-credentials@dc5a8d8553e6ee01fc26761a49e99e733d17954a # v2.1.0
    - name: Run release rake task
      run: bundle exec rake release
      shell: bash
      env:
        RUBYOPT: "${{ inputs.attestations == 'true' && format('-r{0}/rubygems-attestation-patch.rb {1}', github.action_path, env.RUBYOPT) || env.RUBYOPT }}"
      working-directory: ${{ inputs.working-directory }}
    - name: Wait for release to propagate
      if: ${{ inputs.await-release == 'true' }}
      run: gem exec rubygems-await pkg/*.gem
      shell: bash
      working-directory: ${{ inputs.working-directory }}
    - name: Clean up credentials
      if: always()
      run: git credential-cache exit || true
      shell: bash

From this, you can see that this isn’t the GitHub Action that actually does the “get my credentials” dance: that’s rubygems/configure-rubygems-credentials. If you just want something to fetch credentials for you to use in your own CI workflow, that is the GitHub Action you need to care about. That’s what we need in order to integrate Trusted Publishing with Semantic Release.

Semantic Release configuration

Let’s take a momentary detour to discuss the Semantic Release configuration file, .releaserc.yaml:

# Standard plugins, minus npm, plus rubygems
plugins:
  - "@semantic-release/commit-analyzer"
  - "@semantic-release/release-notes-generator"
  - "@semantic-release/github"
  - "@webhippie/semantic-release-rubygem"

You’ll notice I’m using the semantic-release-rubygem Semantic Release plugin to actually build/push gems, and the rest are standard.

CI Workflow

Alright, let’s put all the pieces together. Writing the CI workflow is almost exactly as if you had a long-lived token. Here’s an example deploy job:

deploy:
  runs-on: ubuntu-latest
  environment: rubygems

  # Only deploy if running on master, our release branch in this case
  if: github.ref == 'refs/heads/master'

  permissions:
    id-token: write # Required for trusted publishing
    contents: write # Required to push tags

  steps:
    - name: Checkout code
      uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
      with:
        persist-credentials: false

    - name: Setup node
      uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0
      with:
        node-version: '26'

    - name: Install semantic release
      run: npm install semantic-release@25 @webhippie/semantic-release-rubygem@7

    - name: Obtain RubyGems credentials
      uses: rubygems/configure-rubygems-credentials@dc5a8d8553e6ee01fc26761a49e99e733d17954a # v2.1.0

    - name: Run semantic release
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: npx semantic-release

That’s it. That CI workflow obtains a short-lived token from RubyGems, and then runs Semantic Release. Semantic Release will process all the commit messages that have taken place since the last release, determine first if a release should even be made, and if so, the version that should be used. Once that is determined, it will tag that commit and push it up. It will then update the gem version, build the gem, and then push it using the credentials obtained by configure-rubygems-credentials.

Conclusion

Semantic versioning is a great way to help your users understand if they can update to your new releases without issue. Combining conventional commits and Semantic Release lets you honor that contract without the tedious work of manually picking versions and writing changelogs. Trusted Publishing eliminates the long-lived API token from that workflow entirely, but the documentation and GitHub Actions make some assumptions that aren’t entirely compatible with Semantic Release. I hope this article was helpful in smoothing over that sharp edge and showing that, with a little care, you can take advantage of the increased security offered by Trusted Publishing without sacrificing any of that release automation. The result is magical.

New and improved way to review GitHub pull requests for snaps

Sat, 16 Sep 2023 00:00:00 +0000

Reviewing pull requests for snaps has been pretty terrible ever since snaps were introduced. There are a few very simple reasons:

Building snaps locally in order to review is too much work. This is a fine place to start, but it doesn’t scale, especially if it takes any amount of time to build the snap in question (mine takes over an hour).
Most CI engines use Docker. You can build snaps in docker with hacks and tweaks that sort of work sometimes, but it’s not a supported approach, and it’ll just randomly break next week.
Using private tokens to access the snap store from a fork will leak the token. You don’t want random people releasing snaps on your behalf. This concern is not limited to snaps, of course.

I dedicated an article to this back in 2019, introducing a solution that worked reasonably well for a number of years, but it had some downsides as well:

It used Docker. See (2), above.
It required your own infrastructure and a GitHub app, which was difficult to configure. No one wants to do that.

Ultimately, the GitHub Snap Builder fell apart for me personally because it used Docker, and Snapcraft broke my ability to build the snap I cared about (Nextcloud) in Docker. I needed a new solution, but I wasn’t in the mood to roll my own again. I bided my time, investigated, and experimented. Two years ago, I finally started down the path that ultimately led to success, and I’d like to share that with you.

GitHub Actions

For a long time, the Nextcloud snap used Circle CI as our CI system (Travis CI before that). It was the only CI system for years that could actually build snaps, since it supported a special runner type that was a virtual machine instead of being a Docker container. However, in 2021, they suddenly capped CI runtimes to one hour, with no way to override. We were forced to revisit the CI landscape, and were pleased to discover that GitHub Actions, introduced in the meantime, had all the capabilities we needed. We moved to that, and have been happy users ever since.

We were still in the lurch in terms of testing pull requests, though. We could build snaps in GitHub Actions, and run our tests against them, but without something like the now-defunct GitHub Snap Builder, we couldn’t upload them to the store. The Nextcloud snap is an open source project, and we receive pull requests from unknown contributors all the time. Using GitHub Actions to upload snaps to the store would leak our private tokens to those unknown contributors (see (3), above). Until 2020.

On December 15th, 2020, Jaroslav Lobacevski wrote a series of posts about GitHub Actions security. To be honest, I don’t actually know when the critical feature was added, but this series is what introduced me to it: the workflow_run trigger.

Documentation has always been the Achilles heel of GitHub Actions. All they have are reference docs that are so light on detail I find myself wince-laughing half the time. I still can’t read the docs on workflow_run and come away with any meaningful understanding, but Lobacevski’s blog posts convinced me it was what I needed. Let me explain by first explaining how the GitHub Snap Builder solved this problem.

GitHub Snap Builder

The GitHub Snap Builder obviously needs a token for it to be able to upload snaps to the snap store. It did this on pull requests from unknown contributors. How did it avoid leaking that token? By making a very clear delineation between trusted and untrusted code. It used two ephemeral containers:

A container to build the snap. This was an unprivileged container with no tokens to leak. It was responsible for building the snap from the untrusted code in the pull request.
A container to upload the snap. Once the snap was built by container 1, another container was fired up with the snap store token in its environment. It then used snapcraft to upload the snap to the store. Everything running in this container is trusted, coming from a known-good source. It’s uploading an untrusted snap, yes, but it’s not running anything out of that snap to do it. This process was hard-coded, and didn’t rely on (or run) anything in the pull request, so there’s nothing for a malicous pull request to compromise.

How does that relate to GitHub Actions?

The workflow_run trigger allows for the same untrusted/trusted code separation that was possible in the GitHub Snap Builder. It supports a pull request workflow like this:

Pull request is opened from fork foo/nextcloud-snap, branch bar
Snap is built of the untrusted code in the pull request, and automatically tested
That snap is saved as an artifact

Once that workflow finishes on that fork’s branch, it uses the workflow_run trigger to fire up a new workflow on the master branch: trusted code. That branch has access to the snap store token, downloads the snap from the pull request workflow’s artifacts, and uploads it. Again, all without running untrusted code.

Show me how

Ultimately you will need two workflow files, one for each side of the trust boundary. In other words, you need one to run on pull requests and build snaps, and one to run on master and upload snaps. I’ll reduce this to the minimum you need to make this happen, but if you want to see a real world example, see the ‘test.yml’ and ‘release-pr-snap.yml’ workflows in the Nextcloud snap

Generate new snapcraft token

The first thing you need to do is generate a token so you can authenticate with snapcraft without actually needing to provide a password. This also provides an opportunity to scope access to the specific snap and actions we’ll be doing, to minimize damage in case the token DOES leak. Do that by running a command like this:

$ snapcraft export-login token_file\
           --snaps=<snap-name>\
           --channels=<channel>\
           --acls=package_access,package_push,package_release

As you can see, that requires us to decide our release strategy: what channel do we use when releasing a snap built from a pull request? This is exactly what channel branches are for. In Nextcloud, we use latest/beta/pr-<pull request id>. So the command we use to generate a token that has the ability to upload ONLY the Nextcloud snap ONLY to latest/beta/<branch> (but NOT latest/beta or anything else) looks like this:

$ snapcraft export-login token_file\
           --snaps=nextcloud\
           --channels=latest/beta/*\
           --acls=package_access,package_push,package_release

Create a new environment: snapcraft

In order to protect access to the token you just generated, create a new environment in GitHub. I called mine “snapcraft.” Add a branch protection rule to only allow access from the master branch, and add a single secret, SNAPCRAFT_STORE_CREDENTIALS, set to the contents of the token_file you generated in the last step.

Untrusted code: build the snap

Alright, time for the workflow files. First create the workflow that is responsible for building (and potentially testing) the snap. I called mine .github/workflows/test.yml. Make it look something like this:

name: Run pull request tests

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-20.04

    steps:
      - name: Check out code
        uses: actions/checkout@v3

      - name: Install dependencies
        run: sudo apt-get update -qq && sudo apt-get remove -qq lxd lxd-client && sudo snap install lxd && sudo lxd init --auto && sudo snap install --classic snapcraft

      # Using sudo here because our CI user isn't a member of the lxd group
      - name: Build snap
        run: sudo snapcraft --provider lxd

      # Just an example, of course. Remove this if you have no real tests
      - name: Test the snap
        run: ./run-tests

      # Upload the snap so that we can download it in our other workflow
      - name: Upload snap artifact
        uses: actions/upload-artifact@v3
        with:
          name: snap
          path: '*.snap'

That simply checks the code out, installs snapcraft (and lxd, which we’re using to build here for a nice clean environment), builds and tests the snap, and then saves it as an artifact.

Trusted code: upload the snap

Alright, now we’re ready to create the workflow that is responsible for uploading the snap. I called mine .github/workflows/release-pr-snap.yml. Make it look something like this:

name: Release snap for pull request

on:
  workflow_run:
    # This maps to the name of our workflow above
    # "when one of these workflows finishes, trigger me"
    workflows: ["Run pull request tests"]
    types:
      - completed

jobs:
  upload:
    runs-on: ubuntu-latest
    environment: snapcraft # request access to the environment we created

    # Only run if what triggered us was a workflow from a pull request,
    # and that workflow finished successfully.
    if: >
      ${{ github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success' }}      
    steps:
      - name: Determine pull request commit hash
        uses: actions/github-script@v6
        id: determine-sha
        with:
          result-encoding: string
          script: |
            const workflowRun = await github.rest.actions.getWorkflowRun({
              owner: context.repo.owner,
              repo: context.repo.repo,
              run_id: ${{ github.event.workflow_run.id }},
            });

            return workflowRun.data.head_sha;            

      - name: Set initial commit status
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'pending',
              description: 'Preparing environment...',
            });            

      - name: Determine release channel
        uses: actions/github-script@v6
        id: determine-channel
        with:
          result-encoding: string
          script: |
            const pullRequests = await github.paginate(
              `GET /repos/${process.env.GITHUB_REPOSITORY}/pulls`,
              {
                owner: context.repo.owner,
                repo: context.repo.repo,
              }
            );

            const matchingPullRequests = pullRequests.filter(pr => pr.head.sha === '${{ steps.determine-sha.outputs.result }}');
            const pullRequestCount = matchingPullRequests.length;
            if (pullRequestCount != 1) {
              throw new Error(`Expected one pull request to contain commit, but got ${pullRequestCount}`);
            }

            return `latest/beta/pr-${matchingPullRequests[0].number}`;            

      - name: Download snap artifact
        uses: actions/github-script@v6
        with:
          script: |
            const artifacts = await github.rest.actions.listWorkflowRunArtifacts({
               owner: context.repo.owner,
               repo: context.repo.repo,
               run_id: ${{ github.event.workflow_run.id }},
            });

            const snapArtifacts = artifacts.data.artifacts.filter((artifact) => {
              return artifact.name == "snap"
            });
            if (snapArtifacts.length != 1) {
              throw new Error(`Expected one snap artifact, but got ${snapArtifacts.length}`);
            }

            var download = await github.rest.actions.downloadArtifact({
               owner: context.repo.owner,
               repo: context.repo.repo,
               artifact_id: snapArtifacts[0].id,
               archive_format: 'zip',
            });
            var fs = require('fs');
            fs.writeFileSync('${{github.workspace}}/snap.zip', Buffer.from(download.data));            

      - name: Extract snap artifact
        run: unzip snap.zip

      - name: Install snapcraft
        run: sudo apt-get update -qq && sudo snap install --classic snapcraft

      - name: Set uploading commit status
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'pending',
              description: 'Currently uploading/releasing snap...',
            });            

      - name: Upload snap
        env:
          SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }}
        run: snapcraft upload *.snap --release="${{ steps.determine-channel.outputs.result }}"

      - name: Set final commit status
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'success',
              description: "Snap built and released to '${{ steps.determine-channel.outputs.result }}'",
            });            

      - name: Set error commit status
        uses: actions/github-script@v6
        if: failure()
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'error',
              description: 'Snap failed to upload/release',
            });

Because this code is running on another branch than the pull request’s workflow, there don’t seem to be any primitives available to accomplish most of what we need, so we lean pretty heavily on the GitHub API. I’d be willing to bet there are premade actions out there that could simplify some of this, but I was happy enough with this that I didn’t investigate too hard.

The first thing we do is take the triggering workflow run ID, and determine the commit on which that workflow ran. We need that so we can create new commit statuses, which ultimately results in feedback in the pull request’s “checks” section. Indeed, that is the second thing we do: create a new status saying that the Snap Publisher is “Preparing environment.”

Then we use that commit hash again and determine the pull request that contains it. Using that information, we’re able to calculate the release channel for this snap using the pattern we discussed previously: latest/beta/pr-<pull request id>.

After that, we download and extract the artifact saved by the “test.yml” workflow. At that point, we install snapcraft, so we can actually upload the snap. We update the commit status again, saying we’re uploading/releasing the snap, and then do exactly that: upload the snap and release it to the calculated channel. You’ll notice that’s where we actually use the token, defining the SNAPCRAFT_STORE_CREDENTIALS environment variable, which snapcraft will use to authenticate.

When that’s done, we update the commit status once more saying that it’s done, and providing the channel into which it was released, like so:

The very last step only runs if an error occurs elsewhere in the pipeline, and it simply updates the commit status to point out the failure.

Conclusion

After a few years of painful reviews, the Nextcloud snap project is once again able to more easily evaluate its pull requests. This new workflow not only makes it easier for maintainers to review, but also lowers the barrier to entry for other members of the community to help. Now we can ask for participants in a particular issue to test out new features or fixes without teaching them to do anything beyond that with which they are already familiar. This technique has certainly helped our community, and it’s my hope that the sharing of it can help yours.

Monitoring a Nextcloud snap upgrade

Sun, 06 Mar 2022 00:00:00 +0000

The Nextcloud snap updates automatically. By default it might happen any time, although you can configure that. The snap provides a wonderful, user-friendly indicator that an upgrade is happening: it will become unresponsive and unvisitable until the upgrade has finished. Okay, I’m kidding, this is terrible and something we know we need to work on. My point is, there are times when your snap isn’t responding properly, and you want to know WHY. It’s probably because an upgrade is happening, so how do you monitor it?

Logs.

The Nextcloud snap has several services contained within it. There are four that are most important for upgrade purposes:

MySQL (nextcloud.mysql)
PHP (nextcloud.php-fpm)
Apache (nextcloud.apache)
Nextcloud Fixer (nextcloud.nextcloud-fixer)

These run in order and are dependent on each other, so it’s informative to check them in order. Let’s talk through them.

MySQL

MySQL will run its own upgrade process when the snap update includes a new version of it. This alone can take surprisingly long (several minutes on my own production install of the snap).

In most snap updates, though, a new MySQL version is not shipped. In that case, it should fire up pretty quickly. Either way, you can monitor it with:

$ snap logs -f nextcloud.mysql

That will open the log and follow it, so you can see what’s happening in real time. To stop following it, ctrl+c out of it.

If for whatever reason that log isn’t helpful to you, you can take a look at MySQL’s error log, located at var/snap/nextcloud/current/logs/mysql_errors.log.

If MySQL looks healthy, move on to PHP.

PHP

PHP doesn’t really have an upgrade process of its own, even when the snap includes a PHP update. Which means it fires up fast and rarely has issues. That said, it’s required by almost everything else, so it’s important to know about it and know how to check it. You can monitor it with:

$ snap logs -f nextcloud.php-fpm

Its error log is located at /var/snap/nextcloud/current/logs/php-fpm_errors.log. Assuming everything looks good, time to check Apache.

Apache

Apache is another component that doesn’t have its own upgrade process and fires up fast. However, its the service that is responsible for actually running Nextcloud’s upgrade process, so you’ll find that this is the most important service to monitor. You can do that with:

$ snap logs -f nextcloud.apache

Following the pattern we’ve already established, its error log is located at /var/snap/nextcloud/current/logs/apache_errors.log. If that looks good, then you should be able to visit Nextcloud at this time, but it might be in maintenance mode. That’s because it’s not the last step in the upgrade process– check the Nextcloud Fixer.

Nextcloud Fixer

The Nextcloud Fixer is the slick and creative name that I gave the service that works around all of Nextcloud’s upgrade quirks and weirdnesses that we’ve run into over the years. For example, Nextcloud likes to introduce requirements in updates (new database columns, indices, primary keys) but then not take care of creating them for you. This is where we do such things. You can monitor that process with:

$ snap logs -f nextcloud.nextcloud-fixer

It doesn’t have a logfile of its own, so that command should give you everything.

Conclusion

If all of the above services look healthy and you still can’t visit the snap, something else is broken, probably not related to the snap. Check your surrounding infrastructure, your firewall, your port-forwarding etc., and make sure your IP address didn’t change out from under you so your domain name isn’t pointing to the wrong place.

I hope that helps, but if you’re stumped and need some help, you’re always welcome to log an issue and discuss it.

Nextcloud snap finds a new home

Fri, 03 Sep 2021 00:00:00 +0000

If you’re an avid reader of the blog (and come on, who isn’t?), you’ll know that I’ve been maintaining a snap for Nextcloud since Nextcloud came into existence. Originally, though, that snap wasn’t called “nextcloud”, it was called “nextcloud-kyrofa”. My original goal was that I would bootstrap the snap for them, and then they’d take it over when they were ready. I hosted the code as a personal project on GitHub. Once the Nextcloud Box looked like a certainty, though, Jos and I talked about transferring things over to Nextcloud– both the code as well as the snap in the store, so we could call it “nextcloud” and rip my nick off the end.

Ultimately they were on board, but they asked that I continue maintaining it, which was fine with me because I had since fallen in love with it and began using it myself. So I ended up maintaining the Nextcloud snap with the code living in the Nextcloud GitHub org.

That has worked fine for years, but I’m posting today to explain why it no longer works, and why the snap’s code is amicably moving away from the Nextcloud org to one that I fully control.

What happened?

Nextcloud has gotten quite popular, and various pieces of it have attracted a number of contributors, which is wonderful. They are a very welcoming community, and I have found myself at odds with their open-by-default practices. I’ll admit I’m a bit of a grumpy old man, but allow me to explain.

The Nextcloud org has a very low barrier to entry. Essentially, if you contribute anything at all to any of the projects in the org, you’re added to the org. It’s their org, and it’s cool to be able to show that badge on your GitHub profile etc., so I can see how such a process encourages the growth of the community. As of this writing there are 619 people in that org.

Here’s the problem, though: the base role for the Nextcloud org is write access. That means that over 600 people have write access to any of the over-200 repositories in the Nextcloud org. Various Nextcloud apps, utilities, and yes, the snap. Over 600 people that I don’t know have write access to my project. Even as an admin of the project, GitHub seemingly provides no way for me to override the org’s base role. In my humble opinion, while I understand the desire to have an open community, this is poor opsec, and it makes me uncomfortable.

So protect your branches!

Those familiar with GitHub will suggest that I protect my branches. I do that on my personal projects, you can bet I do that were I have collaborators (like here). Sadly, that’s really the only level of protection that GitHub offers. My CI system leans heavily on tags, and anyone in the org can delete them or make them point to another commit. Write access provides quite a bit of access. Even protected branches provide only a single layer of protection. I’m human, after all. If I forget to protect a release branch, it’s game over. Not everyone signs their commits (by the way, you should sign your commits).

And this goes beyond the code, too. There have been numerous times where some random community member will run rampant over my issues without any communication, creating labels and applying them willy nilly. I hate to be ungrateful, but I have a system for things like that. Instead of being helpful, it creates more work for me.

Web of trust

My trust is earned, not granted by default. I like to invite individuals that have earned that trust to join the team that maintains a particular project. It leads to a much smaller community than Nextcloud’s approach, but that community is built on trust as opposed to random contributions to a completely different project. I don’t mean to imply that one approach is objectively better than the other– they serve different purposes.

Snaps automatically update. As a result, the people installing the Nextcloud snap are willingly ceding control of and responsibility for maintaining and updating their Nextcloud stack to the snap maintainers. They are putting their trust in us– in me– and I take that very seriously. If I then allow people I don’t trust to have write access to the snap, do I really deserve that trust? I think not. So I’m taking action.

What’s the plan?

It’s simple: with Nextcloud’s blessing, the Nextcloud snap has moved to the nextcloud-snap org on GitHub, which I own. Now I can promise that the only people with access are the people who have earned my trust. Nothing should change in the day-to-day maintenance of the snap, and GitHub is forwarding traffic from the old URL for those of you who have it cloned (just update your remotes when you get a chance).

As always, I appreciate your support and your trust, and I will do everything in my power to retain it. I look forward to continuing to work with you from our new location!

Demystifying Rails autoloading

Sat, 14 Aug 2021 00:00:00 +0000

When I first started learning Rails back in the day, it was my first introduction to Ruby: I was learning them both at the same time. As a result, the line between them was rather blurry; I didn’t know what was coming from Ruby, and what was coming from Rails. The Rails approach of monkey-patching Ruby didn’t help. If I’m being honest, I didn’t realize that Object#blank? wasn’t a Ruby method until only a few years ago.

I think it’s really important to understand the software you’re using, particularly big frameworks like Rails that follow convention over configuration and thus end up magically doing things for you. What are you supposed to do when the magic is gone?

Today I want to talk about one such piece of magic: why you need to require things in Ruby, and why you (generally) don’t in Rails.

First of all, you’ll note that the title refers to Rails autoloading. I did that for the SEO, but that’s not the only piece in play, here. In fact, there are three:

Ruby
Bundler
Rails

Let’s take each in turn. Note that for all my examples, I’m using version 3.0.1 of Ruby via RVM, with a gemset made particularly for this purpose.

$ rvm install 3.0.1
$ rvm use 3.0.1@demystify-autoloading --create

Ruby

Ruby does no hand-holding, and has no magic. In general, you must require everything. Let’s investigate this with a simple example.

Create a new directory for this example with two files in it, one of them executable:

$ mkdir ~/ruby-example
$ touch ~/ruby-example/foo.rb
$ touch ~/ruby-example/main.rb
$ chmod a+x ~/ruby-example/main.rb

Make ~/ruby-example/foo.rb look like this:

class Foo
  def hello
    puts 'hello, world!'
  end
end

This file defines a simple class with a single method that prints a message. Now let’s fill out ~/ruby-example/main.rb to use it:

#!/usr/bin/env ruby

foo = Foo.new
foo.hello

This simply instantiates our class and runs the method that should print our message. Let’s run it:

$ ~/ruby-example/main.rb
/home/kyrofa/ruby-example/main.rb:3:in `<main>': uninitialized constant Foo (NameError)

This doesn’t work because Ruby has no magic here: it has no idea what Foo is, despite the fact that the file defining it is sitting right next to the one you’re executing. You need to require it. Make ~/ruby-example/main.rb look like this:

#!/usr/bin/env ruby

require_relative 'foo'

foo = Foo.new
foo.hello

Run it again, and behold success:

$ ~/ruby-example/main.rb
hello, world!

In general you need to be pretty explicit in Ruby about your dependencies. In trade, it’s usually pretty easy to determine where your dependencies are coming from, since there’s an explicit require chain that pulls it in.

Bundler

One could make the case that needing to explicitly include everything you’re using is tedius. This is actually one of the roles that Bundler can play. Most folks know that Bundler can be used to manage your dependencies, but did you know that it can also be used to require those dependencies?

Create a new directory for this example with two files in it, one of them executable:

$ mkdir ~/bundler-example
$ touch ~/bundler-example/Gemfile
$ touch ~/bundler-example/main.rb
$ chmod a+x ~/bundler-example/main.rb

The ~/bundler-example/Gemfile is used by bundler to control your dependencies. Make it look like this:

source 'https://rubygems.org'
gem 'hello-world'

The hello-world gem is useful because it just prints a message when you require it. Install it:

$ cd ~/bundler-example/
$ bundle install

Now make ~/bundler-example/main.rb look like this:

#!/usr/bin/env ruby

require 'bundler/setup'
Bundler.require

That’s it. Run it:

$ ~/bundler-example/main.rb
hello world!
this is hello world library

That’s the message printed by the hello-world gem when you require it, as I mentioned above. That means Bundler included it for you, as a result of it being part of your Gemfile. You can hand Bundler.require different groups to limit what it requires for you, as well (production versus development or test, for example).

Now that you’re familiar with this technique, go to any Rails application and check out config/application.rb, and you’ll see something like this:

# ...

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

# ...

So Rails applications do this for you by default, which is why you generally don’t need to require any of your dependencies. There are caveats, however.

Lots of Rails developers (including me) have settled on a modular approach to developing Rails applications, which means that a given application is split across a number of gems, which are often Rails engines. If you’re also in this camp, one thing to keep in mind is that gems (including engines) do not have this functionality. Gems generally execute within the context of the main application’s Gemfile, so it can’t lean on Bundler like this. Gems are expected to require their own dependencies. That doesn’t mean you need to do it in every file, though, if that annoys you. Ruby’s require has global effect, which means once something is required in one place, it can be used everywhere. If you have a file which is generally loaded as part of using the gem (e.g. lib/<gem name>.rb), you can put your requires there and just use them everywhere.

Rails

So Bundler magic covers why you don’t need to require your gem dependencies, but what about other .rb files within the same app (or engine)? If you’re familiar with Rails, you may have noticed you generally don’t need to require those either. That flies in the face of our understanding of Ruby, and it’s thanks to Rails magic called autoloading.

There are other resources that describe Rails autoloading in detail. Those are recommended reading, so I won’t repeat them.

The general idea is this: if you access a constant that is missing in the current execution context (e.g. a class defined in another file), Rails will try to find and require the file that defines it for you by searching a pre-defined set of autoload paths following a convention where each namespace for the constant equals a directory. A simplistic example: if you try to access Bar and it’s not defined, Rails will quickly check to see if there’s a bar.rb directory in any of the autoload paths. If there’s not, it’ll either complain or you’ll get the standard Ruby error for trying to access an undefined thing. If you try to access MyModule::Bar, it’ll search those same autoload paths for my_module/bar.rb.

As a caveat, Rails engines don’t add their lib/ directory to the set of autoload paths by default, but you’ll notice engine developers often do it themselves for consistent behavior.

Conclusion

Rails has done a magnificent job of making it trivial to get up and running developing new web applications, thanks to its convention-over-configuration approach. Its use of Bundler and autoloading to handle requires is no small part of that overall philosophy, and I suspect it’s a large reason for its success. That said, if you never really try to understand what Rails is doing for you, you will eventually get bitten when it doesn’t do what you expect. Gaining at least a small insight into how this particular piece of the puzzle works will pay dividends down the road, and I hope this was helpful in accomplishing exactly that.

My phone isn't as smart as yours

Sat, 17 Apr 2021 00:00:00 +0000

I remember the first mobile phone I ever saw. A “car phone.” It looked a bit like an old corded telephone hooked up to a car battery in a bag that weighed 30 pounds. I thought it was awesome; my father was on the bleeding edge of a technological revolution, and he was pretty much a super hero. Was it pretty? No. Actually mobile? It couldn’t leave the car, but the car could move, so sure. Did it give us all cancer? Maybe.

In the beginning

I received my first personal cell phone when I was 16. A Nokia something-or-other. It wasn’t small enough to fit in my pocket, but it came with a case and a belt clip, so I looked like a boss. Texting was just starting to become a thing. I couldn’t understand it– why on earth would I punish myself trying to produce something legible by randomly mashing buttons (a.k.a. the super mario method, a.k.a. T9)? I’d rather call people instead. And you know what? Back then, people answered instead of ignoring you and texting you back, probably because they agreed texting was annoying, too. If they didn’t answer, you left a voicemail, which was a perfectly normal thing to do.

By the time I left for university, my cell phone was a Sony Ericsson Walkman flip phone. It was small enough to fit in my pocket, but I still wore it on a belt clip, because I was cool. That phone could actually play music, so I was also regularly wearing headphones, which really just elevated my cool factor.

During this journey, I wasn’t paying for my phone or the plan, my parents (bless their hearts) were. I never wanted to rock that boat: when it was time to get a new phone, they gave me my options, and I happily selected one. I knew that smartphones were a thing. Heck, I was a Computer Engineering major– I knew all about them. But they were expensive, and I never dared ask my parents. Then came graduation day, and First Real Job day, when I no longer mooched from my parents.

Smartphones forever!

I moved across the country for work, and purchased my own phone and my own phone plan. A smartphone, obviously: the HTC One. It was perfect. The perfect size, the perfect speed. Battery life was a little annoying but that was okay, and I could actually text on the thing!

It changed my life. I didn’t need maps anymore to get around. I started texting far more than I called. I received my email on my phone, allowing me to be more responsive and use my computer less. More than anything: it put the internet at my fingertips at all times. I could learn anything at the drop of a hat. I was hooked. I purchased a new smartphone every two years, upgrading to the best thing on the market at the time, unlimited data plan, the works.

To people who know me, I’m sure this is unsurprising. I’m a pretty nerdy dude. Come on, I wore cell phones on my hip. Why wouldn’t I stay on the forefront of innovation?

So it might surprise you to know that I’m approaching the third anniversary of owning my latest phone, an Alcatel GO FLIP. Yes, a flip phone. It doesn’t even play music. That surprises people, and I’ve gotten a lot of questions about it. What happened to me? That’s what I want to tell you about today.

All was not right with the world

It really comes down to something I’m loathe to admit: I have no self-discipline. For example, the only reason I don’t weigh 300 pounds is because my lovely wife knows that if she buys chips/cookies/what-have-you, I will eat them all without prejudice. So such things stay out of the house. I don’t seem to have a problem with binge eating broccoli.

This lack of self-discipline shows itself in various ways. One was addiction. No, thankfully I’ve never struggled with alcohol or drug addiction. For me, it was information. I love learning new things so much that it can easily become “the end” instead of “the means”. This happens to everyone in some form or another: you want to make sure you can explain gimbal lock properly, so you go to the wikipedia page, where you see Apollo 11 discussed, so you check that out, where you see talk of the ATOLL programming language, so you start reading about that before you realize that you should really finish the gimbal lock thing. It got to the point where having a smartphone with me all the time was virtually that, constantly. I wasn’t continually playing games or using snapchat like the kids of today, but I was no better. In fact, I was worse. I was always sucked into learning the ins and outs of whatever tickled my fancy at that moment, which required far more focus and withdrawal from the real world than snapchat ever could.

That’s not to say I couldn’t focus. I could, and I did. In fact, I believe that this propensity is properly harnessed when writing software, and thus is a significant part of why I like to think I’m good at my job. I am obsessed with writing the best code I can, which involves constant research and learning new ways to do things. As a result, I don’t really have the same struggles when on a computer working than I do with a smartphone. No, it was my free time that slowly disintegrated.

When I got off work, I’d go into the living room where the kids were playing. I adore my sweet kids; I’d give them all hugs and enjoy the excited “daddy’s home!” reactions, then I’d plop down into the recliner and pull out my phone. The kids eventually gave up on trying to get my attention, and I only had reason to look away from my screen when they were misbehaving and needed correcting. Which meant two things: 1) they misbehaved more to get my attention, and 2) our interactions were primarily limited to my disciplining them. Neither of those were great.

I hadn’t read a book for years. My free time was spent on my phone learning something or other, but learning it in a very specific, very shallow way.

My work at Canonical over the last few years had gotten more stressful. I had to keep a lot more in my head. At the same time, it seemed like my attention span was getting shorter and shorter. My brain felt like it was fracturing, and I started getting irritable. And I still plopped in that recliner at the end of every day and pumped more useless information into my overworked head.

Suddenly, my wife made an interesting admission: “I think I’m addicted to Facebook. I feel like I’m on my phone too much when I’m around the kids, and I don’t see the quiet positive things that happen as much as the loud misbehaving. And I think that is negatively impacting our relationship.” Suddenly everything clicked into place.

I know! Let’s rely on our self-discipline!

We started a policy of leaving our phones on the shelf after work. That was nice– the kids behaved better, our relationship improved, and my brain started to heal. As you might expect, though, the lack of self-discipline bit us over and over again. We would find after a few days that we fell back into old habits, and would need to re-institute the policy, only to fall back into old habits again.

The return of T9

Finally, we took the broccoli approach: we ordered flip phones. Used from ebay, because, you know, they don’t really exist anymore. We were thinking that maybe we just needed a detox. Maybe if we could actually be without smartphones for a while, we’d be able to go back and have more discipline with them.

The transition wasn’t easy, though. There have been a number of issues that we had to deal with that really ended up making this a complete life change.

No more camera in your pocket

My wife’s main gripe was the lack of a decent camera. Flip phones generally have cameras, but she liked the ability to snap the kids doing something memorable and save it forever, and no one wants to save the kind of picture that is taken by a flip phone camera. Also, she’s a photographer, so her standards are even higher than most. I actually ended up purchasing her a high-quality compact camera for her birthday. That fits in her purse, and shoots in RAW just like her real-deal cameras. She likes that, and has mostly stopped whining about this aspect of flip phones.

No more instant calendar

Something that has effected us both is the lack of a calendar system. I’ve had a Nextcloud instance for years that we used to sync our contacts and calendars on our phones. It was really nice to be able to organize things that way. My wife now maintains an actual calendar, a spiral bound book on our counter. I know that sounds annoying, but it has actually worked great. The only real downside is that you can only “check your calendar” when you’re home.

Group messaging is hit or miss

This one was a big deal. I didn’t realize 1) how often I texted all of my siblings as a group, and 2) that group messaging support in flip phones is hit or miss. It’s impossible to determine if a given flip phone supports them without trying it. My first two flip phones were misses: a Nokia and a Kyocera. I thus went years without being able to be included in a group text thread at all. I was seriously considering creating my own, to be honest. Thankfully, I think I figured it out: if the phone is based on AOSP or KaiOS, group messaging seems to be supported. The Alcatel GO FLIP uses the latter, and group messaging works great.

Seriously, T9?

Thankfully, not completely T9. Even dumbphone users aren’t that masochistic. The OS is smart enough to have a dictionary and predictive text, so it can mostly figure out what you’re trying to do. It’s still tremendously annoying though, I won’t lie. And we live in a smartphone world: people expect you to text back. You should see my wife, she could probably break some records with how fast she can crank out a text on that teeny keyboard. Me… not so much. My texts are very plain and to-the-point. And you know what… maybe that’s just the kind of person I should be.

Maps and GPS

I grew up in a small town, and to this day I still don’t know any street names; I drive by landmark. I have an extraordinary inability to navigate anywhere. It’s just not one of my skills, and that’s an endless source of mirth for my wife. Thankfully, it’s one of her skills.

When I moved to the big city, though, I wasn’t married yet. Virtually the only way I could get anywhere was to either buy a map, or use the GPS on the shiny new smartphone I had. You can probably guess the direction I went with that. I became entirely dependent on that thing.

Flip phones, barring a few outliers, don’t have maps. This has been a difficult adjustment for me. When I’m going somewhere I’ve never been, I generally will just print out Google Maps directions. If it’s somewhere a long ways away, though, where I might need to take numerous detours, I actually just bring my old smartphone with an offline map. That’s what I do whenever I travel internationally as well (where I don’t have signal), and it works perfectly.

Alternative sources of music

My family is one of those that listens to music virtually constantly. At home, in the car, on an airplane, everywhere. I always purchased smartphones with a micro SD card slot so I could fit my 40GB or so of music on there. My father got us a Spotify subscription for Christmas one year, and we started using that as well. Flip phones generally do neither of these things.

When we’re in the car, we’re now listening to CDs or the radio (how retro!). For longer trips, where I generally have the old smartphone for maps anyway, we use it for music just like the old (new?) days.

Severe lack of knowing

One of the beauties of having a computer in your pocket at all times is having instant access to all of mankind’s knowledge whenever you need it. Need a recipe for garlic knots? Done. Can’t remember that joke you read? Look it up real quick so you can tell it properly. Need to explain gimbal lock? As I said earlier, I took full advantage of this with a smartphone, but ultimately it’s what drove me to a dumbphone. I’m just not made to have constant access to information like that. I’ve developed a new skill: the ability to say “I don’t know” and be okay with it.

That’s not to say I’ve stopped learning and researching things. It’s just far more… natural. Instead of isolating myself in the moment, switching contexts, and instantly satisfying my curiosity, I make a mental note. If I remember later, I’ll research it on the computer when I have some free time. If I fail to remember, well, then it wasn’t really worth knowing, was it? I think that’s really how I’m made to live. The ability to forget is a gift, and I have it for a reason. Shoving useless information into my face was how I got here in the first place. Ultimately, I’ve found the lack of knowing to be freedom.

Not just a detox

Like I said earlier, we went into this thinking that we just needed a detox. It’s been three years though, I’m not really sure we can call it that anymore. Our lives have completely changed. It’s been a long road to my brain healing, but I’m stoked with the progress I’ve made. My attention span is fantastic now. I’ve generally found that I have a lot more free time, and I spend it on a few specific things instead of dabbling lightly in several without actually doing anything. I have some open-source software that I maintain, and I’ve gotten into woodworking and reading. I can’t even tell you how many books I’ve read.

This caters to other desires I’ve always had, too. I don’t like being sold, and I don’t like being tracked, and smartphone companies make money by doing both. “If you’re not doing anything wrong, you have nothing to fear” says the naive person who has never really considered all the ramifications.

Our relationships with our phones have changed completely. Half of the time I forget to bring mine when I leave the house. I’ll go an entire day without it before I realize it. I no longer fill silence with endless research: I have some empty space in my life. We have more meaningful discussions without any Facebook or Twitter notification sounds going off.

It definitely impacted our relationship with our kids, too. When they look up while playing, we’re actually there, we notice and give them a smile. We notice how well they’re playing together and are able to give them some positive affirmation instead of over-emphasizing the negative. We see and appreciate the sweet moments instead of only really getting jerked out of our bubble by misbehavior.

We’ve even noticed a social impact. For example, it can be awkward to sit in a doctor’s office where everyone has their nose in their phones except for you. However, we’ve noticed that people tend to randomly strike up conversations with us, and not with other people in the room. We attribute this to the fact that we’re the only people that don’t appear preoccupied. We’re suddenly more approachable, and this has made it easier to connect with people.

Ultimately, we’ve found that the upsides of this lifestyle outweigh the downsides. I suspect we’ll be “detoxing” for the forseeable future.

The end of my journey at Canonical, and new adventures ahead

Wed, 10 Mar 2021 00:00:00 +0000

I am a staff engineer at Canonical, where I have worked for six years. My time here has been filled with brilliant people and amazing technology, and I’ve had the pleasure and privilege of playing pivotal roles getting a number of new projects off the ground. As I’ve said in the past, I joined Canonical to help my technical growth, and they delivered in spades. I have really loved my time here, which made the decision to leave all the more difficult. However, here I am, with the clock ticking down on my last week as a Canonical engineer.

My journey

I was originally hired onto the Unity API team, helping create Unity 8 for mobile and desktop (you’re probably used to Unity 7). That team was filled with wonderful people that I really looked up to, from the kindest and most supportive manager I’d ever had, Alejandro, to C++ geniuses like Michi and Pete.

Over time I found myself drawn more and more to this whole “snaps” thing, and eventually I was invited to join the team creating the snapcraft CLI. This is where I met all-stars like Sergio and Leo. After a while Mark asked me to join the snapd team to help out with some specific features, which is where I got to know the amazing Michael. Once those features were complete, I found myself jumping between both snapcraft and snapd to implement larger features, and the snapcraft team ended up merging with the advocacy team, where I had the privilege of working with giants like Alan and Martin.

Finally my robotics background came home to roost, and I was asked to bootstrap a robotics team at Canonical. I said something along the lines of “sure, as long as I don’t have to be a manager” and launched off. I had the absolute pleasure of being mentored by Joe and Jamie, and I’m tremendously proud of what we’ve done and built in the robotics team at Canonical over the last few years!

Now that the robotics team is up and running though, I’m ready for a new challenge and a change of pace.

Does that mean you’ll stop maintaining <project>?

It’s true, you will probably see my activity on Canonical-owned projects decrease (e.g. snapcraft, snapd, and the like). However, everything that I did on my own time, I generally did because I was using it and relied on it (e.g. the nextcloud snap). Nothing should change for those types of projects.

I’m still a part of the Ubuntu and broader open source communities, so I hope our paths will continue to cross. You should feel free to reach out like you always have.

So what is next?

I’m joining Miriam Technologies, a software contracting and consulting company. I interned for Miriam in the past. It was my first software job as a kid, actually, and I’m looking forward to returning to where it all began and hopefully add a bit more value this time around. Every engineer there is friendly and brilliant, so if you ever need some custom software work done, get in touch.

How to build a snap using ROS 2 Foxy

Mon, 17 Aug 2020 00:00:00 +0000

The snapcraft CLI (the tool used to create snaps) has long had support for building snaps that use both ROS 1 and ROS 2. ROS 2 Foxy Fitzroy is the latest ROS 2 LTS, which runs on Ubuntu 20.04 (Focal Fossa). The snapcraft CLI recently gained experimental support for building Foxy snaps, so I wanted to walk you through doing exactly that with the goal of helping both of us: getting you familiar with this new feature, and at the same time getting some mileage on this so as to make it not experimental. Let’s get to it!

Prerequisites

In order to follow along, you’ll need at least v4.2 of the snapcraft CLI:

$ sudo snap install --classic snapcraft

Of course, some previous experience building a snap will also be helpful.

Let’s get started

The first step toward creating any snap is to create the snapcraft.yaml, which serves as the recipe instructing the snapcraft CLI on how to piece together your snap.

Create the snapcraft.yaml

Let’s create a new directory, and then initialize it with a snapcraft.yaml:

$ mkdir ~/ros2-foxy-snap
$ cd ~/ros2-foxy-snap
$ snapcraft init
Created snap/snapcraft.yaml.
Go to https://docs.snapcraft.io/the-snapcraft-format/8337 for more information about the snapcraft.yaml format.

Open that snap/snapcraft.yaml file, and make it look like this:

name: ros2-talker-listener
version: '0.1'
summary: ROS2 Talker/Listener Example
description: |
    This example launches a ROS2 talker and listener.

grade: devel
confinement: strict
base: core20

parts:
  ros-demos:
    plugin: colcon
    source: https://github.com/ros2/demos.git
    source-branch: foxy
    source-subdir: demo_nodes_cpp
    build-packages: [make, gcc, g++]
    stage-packages: [ros-foxy-ros2launch]

apps:
  ros2-talker-listener:
    command: opt/ros/foxy/bin/ros2 launch demo_nodes_cpp talker_listener.launch.py
    plugs: [network, network-bind]
    extensions: [ros2-foxy]

Let’s break that down by section.

name: ros2-talker-listener
version: '0.1'
summary: ROS2 Talker/Listener Example
description: |
    This example launches a ROS2 talker and listener.

This is the basic metadata required by all snaps. These fields are fairly self-explanatory, but note that the name must be globally unique among all snaps. You might consider appending your developer name to the end of the snap name, for example.

grade: devel
confinement: strict
base: core20

grade can be either stable or devel. If it’s devel, the store will prevent you from releasing into one of the two stable channels (stable and candidate, specifically). If it’s stable, you can release it anywhere.

confinement can be strict, devmode, or classic. strict enforces confinement, whereas devmode allows all accesses, even those that would be disallowed under strict confinement, and logs access that would be disallowed. classic is even less confined than devmode in that it doesn’t even get private namespaces anymore (among other things). There is more extensive documentation on confinement available.

As I’ve said multiple times in the past, I typically use strict confinement unless I know for sure that the thing I’m snapping won’t run successfully under confinement, in which case I’ll use devmode. I typically avoid classic unless I never intend for the snap to run confined (e.g. you’ll notice the snapcraft CLI is a classic snap, since it needs more access to the host than confinement would allow).

Finally, the base keyword specifies a special kind of snap that provides a minimal set of libraries common to most applications (e.g. libc). It will be the root filesystem for this snap. In this case, we’re using core20 which is a minimal rootfs based upon Ubuntu 20.04 (Focal).

parts:
  ros-demos:
    plugin: colcon
    source: https://github.com/ros2/demos.git
    source-branch: foxy
    source-subdir: demo_nodes_cpp
    build-packages: [make, gcc, g++]
    stage-packages: [ros-foxy-ros2launch]

The snapcraft CLI is responsible for taking many disparate parts and orchestrating them all into one cohesive snap. You tell it the parts that make up your snap, and it takes care of the rest. Here, we’re saying that we have a single part called ros-demos. We specify that it builds using the colcon plugin, and we point it to the ROS 2 demos GitHub repository (this could just as easily be a directory on disk). We point the colcon plugin at the subdirectory containing the C++ demo nodes as opposed to letting it build the whole thing. We also provide a list of packages that need to be installed in order to build (build-packages), and also ask that ros-foxy-ros2launch gets staged into the snap alongside the rest of the part to be used at runtime (specifically, we’ll use it in the app, below). To view all the options supported by the colcon plugin, run the command snapcraft help colcon.

apps:
  ros2-talker-listener:
    command: opt/ros/foxy/bin/ros2 launch demo_nodes_cpp talker_listener.launch.py
    plugs: [network, network-bind]
    extensions: [ros2-foxy]

This part is interesting, and even if you’ve built snaps in the past there might be something new here. When we build this snap, it will include a complete ROS 2 system: rclcpp, the demo_nodes_cpp workspace, etc. It could contain the entire system necessary for a robot in one installable blob. It’s a standalone unit: we’re in total control of how we want our users to interact with it. We exercise that control via the apps keyword, where we expose specific commands to the user. Here we specify that this snap has a single app called ros2-talker-listener (the same name as the snap itself), although it could be anything. The command that this app actually runs within the snap uses the ros-foxy-ros2launch that we staged to fire up the demo nodes' talker/listener launch file. We use plugs to specify that this app requires network access (read more about interfaces). Finally, the new bit: we specify that this uses the ros2-foxy extension. This is where a significant chunk of the magic happens, wrapping the app in the environment necessary to run ROS 2 Foxy stuff, etc. Every app that needs ROS 2 will need to use that extension, and those that don’t need ROS 2 don’t need the extension (read more about snapcraft extensions).

Build the snap

Now that we’ve defined the snapcraft.yaml, it’s time to build the snap. Make sure you’re in the directory we created earlier (the one that contained the snap/ directory), and run snapcraft with the --enable-experimental-extensions flag (I told you this was experimental):

$ cd ~/ros2-snap
$ snapcraft --enable-experimental-extensions
*EXPERIMENTAL* extensions enabled.
*EXPERIMENTAL* extension 'ros2-foxy' enabled.
<snip>
Snapped ros2-talker-listener_0.1_amd64.snap

Note that depending on your host and whether or not you’ve built snaps in the past, the snapcraft CLI may prompt you to install Multipass, a tool used by the snapcraft CLI to manage VMs for building snaps (rather than building straight on your host).

The build process will take a few minutes. You’ll see the snapcraft CLI install the build dependencies of the demo nodes, build the packages in the workspace and install them into the snap, and fetch their runtime dependencies and unpack them into the snap as well. This is a bit different than previous ROS snap build processes that used rosdep: this uses catkin_pkg directly to avoid rosdep’s issue of unpacking even build dependencies into the snap, which means ROS snaps using extensions will be significantly smaller than in the past (and take less time to rebuild). Anyway, at the end, you’ll have your snap.

Test the snap

Let’s install the snap we just built:

$ cd ~/ros2-snap
$ sudo snap install ros2-talker-listener_0.1_amd64.snap --dangerous

Note the use of the --dangerous flag. That’s required because we’re installing a snap from disk instead of using the store, and snapd (the daemon with which we’re communicating using the snap command) only trusts snaps that it can cryptographically verify as being from the store unless we tell it otherwise with this flag.

Finally, let’s run the app we defined in the snapcraft.yaml:

$ ros2-talker-listener
<snip>
[INFO] [talker-1]: process started with pid [21071]
[INFO] [listener-2]: process started with pid [21073]
[talker-1] [INFO] [1597444182.799512859] [talker]: Publishing: 'Hello World: 1'
[listener-2] [INFO] [1597444182.800671590] [listener]: I heard: [Hello World: 1]
[talker-1] [INFO] [1597444183.799313385] [talker]: Publishing: 'Hello World: 2'
[listener-2] [INFO] [1597444183.799922986] [listener]: I heard: [Hello World: 2]

CTRL+C to stop that. As you can see, it works great! You could hand this snap to anyone with a snap-capable system, even if they don’t have ROS installed, and it would work exactly the same way for them.

I hope this is helpful in your snap journey, and gives you a decent guide to building ROS 2 Foxy snaps using the new ros2-foxy extension. Please feel free to ask any questions here, on the Snapcraft forums, or on the ROS forums. I’d love to hear any feedback you have.

Four years of Nextcloud

Sat, 27 Jun 2020 00:00:00 +0000

I’m a few weeks late on celebrating Nextcloud’s four year anniversary, but I’m doing it anyway! My how time flies. I became a part of the ownCloud community back in 2012. Heck, you can still find blog posts about how I ran it on this very blog if you scroll down far enough. Then, in 2015, I saw the post about the Western Digital/ownCloud collaboration on what would eventually become the Nextcloud Box. But I’m getting ahead of myself.

The timing was excellent. We at Canonical were busy developing Ubuntu Core and snaps, and I had just recently joined the team responsible for developing snapcraft, the tool that creates snaps. I thought “What better way to learn the ins and outs of what I’m making than to participate in this competition!” So a few ownCloud community members and I worked on exactly that. It was ultimately selected as the platform to ship on the Box, just in time to lose every contact I had at ownCloud.

I was disappointed. I assumed the project was dead. Then I heard from my friends, and was told about Nextcloud. And they retained the Western Digital contacts, who now wanted to continue the project with them!

I was excited. Within days of Nextcloud making their first release I had created a Nextcloud snap. You can still use Nextcloud v9 today, in fact:

$ sudo snap install nextcloud --channel=9

I’ll always have fond memories of those pixelated clouds.

I was very happy with it. A one-line installation method for a production-ready Nextcloud. Let’s Encrypt integration. Everything. Despite being created to enable the Nextcloud Box, we made sure that it worked on all architectures, so folks could run it wherever they wanted to.

Before too long, the Nextcloud Box became a reality, becoming a joint project between Nextcloud, Western Digital Labs, and Canonical (where Canonical was maintaining the image). At that time, we were actually having to patch Nextcloud to make it work the way we needed it to. We had tried upstreaming that patch to ownCloud, but they turned it down. Which means if you look at the config file you’ll see a dirty little secret:

/**
 * Nextcloud doesn't support hosting the config file out of the application, so
 * this version of Nextcloud has been patched slightly to allow for it. As a
 * result, we fail the integrity check, so we'll just disable it.
 */
'integrity.check.disabled' => true,

Which I why I remember the Nextcloud v10 release so well: Nextcloud didn’t have the same problem with that patch, and as of v10 we no longer needed to patch it, and could re-enable the integrity check!

$ sudo snap install nextcloud --channel=10

That was a solid release, but it wasn’t long before v11 was released with all sorts of security features and a preview of what was to become Nextcloud Talk in a few years:

$ sudo snap install nextcloud --channel=11

There was a lot of similarity between versions 9, 10, and 11, but a ton of new development went into version 12. That was great, but it also meant that a number things broke in version 12. Snaps automatically update, and this is when we really took ownership of that responsibility. We decided we weren’t cool automatically breaking people. We’ve continued that attitude today, so if you’ve wondered why the snap lags upstream occasionally, that’s why. Anyway, eventually the issues were resolved, and we got everyone onto v12:

$ sudo snap install nextcloud --channel=12

This release finally got rid of that dual menu setup (app menu on the left, settings/users on the right) and unified them really nicely:

Version 12 is still one of my favorite releases simply because no new major releases came out for almost a year. Sadly, it was also during this timeframe that Western Digital Labs shut down, thus ending the Nextcloud Box, and thus officially turning the Nextcloud snap into a personal project for me. Despite that, we settled into a groove of maintenance, and our userbase (and group of testers, who are amazing) grew significantly.

Version 13 saw quite a leap in functionality and design:

$ sudo snap install nextcloud --channel=13

For example, this is where Nextcloud Talk officially made its debut. It’s also where they got rid of those pixelated clouds and replaced them with a nice abstract banner:

It was only a few months before the update drum started for v14, which came with all sorts of stuff like SMS 2FA and GDPR compliance:

$ sudo snap install nextcloud --channel=14

Nextcloud 15 took that even further:

$ sudo snap install nextcloud --channel=15

This is the release that added support for my beloved Yubikey, U2F, and TOTP. This is also where they introduced Nextcloud Social for a federated social network.

Nextcloud 16 came with a much-improved Nextcloud Talk, as well as more fine-grained permissions by way of an access control list:

$ sudo snap install nextcloud --channel=16

Version 17 is where the remote wipe functionality made its debut, and the shiny new Text app was introduced as the default text editor:

$ sudo snap install nextcloud --channel=17

Nextcloud 18 saw a slight rebranding. They started calling it Nextcloud Hub, and made it super easy to preinstall “recommended” apps, which included Only Office for collaborative document editing:

$ sudo snap install nextcloud --channel=18

And now we come to current day, version 19, where they switched to Collabora for collaborative document editing instead of Only Office:

$ sudo snap install nextcloud --channel=19

Along with Nextcloud 19, we at Canonical have started the Ubuntu Appliance portfolio, which includes Nextcloud. This is essentially a bring-your-own-box Nextcloud Box: an image with the Nextcloud snap already on there, waiting to be flashed onto a Raspberry Pi or a NUC or something. As soon as it’s flashed and booted, you have a Nextcloud install.

Anyway, I hope this brief walk through history was interesting. Note that all the snap tracks used here will be open forever, so you’re welcome to use even the first Nextcloud, but as a general rule of thumb, only the two most recent major releases are actually supported and receive updates. Of course, if you just want the latest stable version, you don’t need to specify a track at all:

$ sudo snap install nextcloud

It’s been a good four years. Here’s to many more!

PSA for ROS users: Some things to know as Python 2 approaches EOL

Wed, 23 Oct 2019 00:00:00 +0000

We got an interesting question from a customer today, and I think the answer might be helpful to a wider audience. Python 2 will reach end of life in two months. This shouldn’t be news to anyone who hasn’t been living under a rock, and plans are in place to use Python 3 in Noetic (whereas ROS 2 has always used Python 3). However, the question from our customer was this: What does that mean for existing ROS 1 distributions (Kinetic and Melodic)? They are still using (and will continue to use) Python 2.

The answer really depends on where you’re getting Python 2. Tl;dr: If you’re using Ubuntu Xenial (16.04) or Bionic (18.04), please know that Python 2 from the Ubuntu repositories will continue to be supported for the lifetime of the Ubuntu release, regardless of Python 2’s upstream support status.

Ubuntu’s package repositories are split into a number of components: main, restricted, universe, and multiverse. You can read that link to learn all about them, but let me briefly quote it so you can understand the difference between main and universe:

main:

The main component contains applications that are free software, can be freely redistributed and are fully supported by the Ubuntu team. […] When you install software from the main component, you are assured that the software will come with security updates and that commercial technical support is available from Canonical.

universe:

The universe component is a snapshot of the free, open-source, and Linux world. […] Canonical does not provide a guarantee of regular security updates for software in the universe component, but will provide these where they are made available by the community.

All Ubuntu releases prior to Eoan (19.10) included Python 2 in main. In Eoan it was demoted to universe. This means that for Xenial (16.04) and Bionic (18.04), Python 2 falls under the main guarantees quoted above. It will continue to be fully-supported and receive free security updates.

In other words, as long as you’re running Kinetic or Melodic against its recommended Ubuntu release targets, you should be golden.

We’re having a discussion about this in the ROS forum if you’d like to participate.

Robotics security: What is SROS 2?

Fri, 04 Oct 2019 00:00:00 +0000

We at Canonical have been hard at work on the security features of version 2 of the Robot Operating System (ROS 2). However, if we lift our collective heads up out of the weeds it’s easy to see folks completely misunderstanding how security works today in ROS 2. We’ve written some design articles to help distill all the moving pieces into something comprehensible, but I wanted to do the same here in a slightly less formal way.

ROS 2 uses DDS, an existing protocol with its own specs. It’s out of scope to explain DDS, but there are two important points to make for the purpose of this post:

DDS is just a spec; there are various DDS vendors that create both open source and commercial DDS implementations.
DDS supports various security features by way of its DDS-Security spec.

Both of those specs are just as much fun to read as any other spec, so let’s paint a simplified picture of how DDS-Security works.

What is DDS-Security?

DDS-Security is, again, just a spec, and an optional one at that (not all DDS implementations support it). It expands upon DDS by defining a set of plugins with specific purposes and APIs, as well as defining a set of built-in plugin implementations that must be supported by a given DDS implementation in order to be considered in compliance with the spec. It defines five plugins:

Authentication: Verify the identity of a given domain participant
Access control: Enforce restrictions on the DDS-related operations that can be performed by a participant
Cryptography: Handle all required encryption/signing/hashing
Logging: Provide ability to audit DDS-Security-related events
Data tagging: Provide the ability to add tags to data samples

ROS 2’s security features only utilize the first three of these plugins. Why? Because those last two aren’t required in order to be compliant with the spec, and thus not all DDS implementations support them. We need to cater to the lowest common denominator, here. Let’s talk a little more about those first three plugins.

Authentication

The Authentication plugin is central to the entire plugin architecture defined by DDS-Security. It provides the concept of a confirmed identity without which further enforcement would be impossible (e.g. it would be awfully hard to make sure a given ROS node could only access specific topics if it was impossible to securely determine which node it was).

The builtin plugin (which is what ROS 2 uses) uses the proven Public Key Infrastructure (PKI). It requires a public and private key per domain participant, as well as an x.509 certificate that binds the participant’s public key to a specific name. Each x.509 certificate must be signed by (or have a signature chain to) a specific Certificate Authority (CA) that the plugin is configured to trust.

Access control

The Access control plugin deals with defining and enforcing restrictions on the DDS-related capabilities of a given domain participant. It’s not apparmor or cgroups, it’s really about limiting a particular participant to a specific DDS domain, or only allowing the participant to read from or write to specific DDS topics, etc.

The builtin plugin (which is again what ROS 2 uses) also uses PKI. It requires two files per domain participant:

Governance file: A signed XML document specifying how the domain should be secured
Permissions file: A signed XML document containing the permissions of the domain participant, bound to the name of the participant as defined by the authentication plugin

Both of these files must be signed by a CA which the plugin is configured to trust. This may be the same CA as the Authentication plugin trusts, but that isn’t required.

Cryptographic

The Cryptographic plugin is where all the cryptography-related operations are handled: encryption, decryption, signing, hashing, etc. Both the Authentication and Access control plugins utilize this plugin to verify signatures, etc. This is also where the functionality to encrypt DDS topic communication resides.

So what is SROS 2? It’s ROS 2’s integration with DDS-Security

SROS 2 stands for “Secure Robot Operating System 2.” Understandably, this tends to make folks assume it’s some sort of ROS 2 fork that is somehow secure, and that’s not actually the case. The name is historical: SROS was an effort back in ROS 1 to lock it down, and it was essentially a fork. In ROS 2, the term “SROS 2” is given to the collective set of features and tools in various parts of ROS 2 that are used to enable integration with DDS-Security. So what is that “set of features and tools”?

ROS client library (RCL)

Most of the user-facing runtime support for SROS 2 is contained within the ROS Client Library, a core component of ROS 2. It’s responsible for coordinating the enablement of DDS-Security for each DDS implementation that supports it. It supports three main features:

Security files for each domain participant
Permissive and strict enforcement options
A master on/off switch for all security features

For more details on each of these please read the ROS 2 DDS-Security article.

SROS 2 utilities

Configuring a ROS 2 system to be secure in RCL involves a lot of technology that may be unfamiliar to the average roboticist (PKI, DDS governance and permissions files and their syntax, etc.). The SROS 2 utilities includes the ros2 security tool to help set up security in a way that RCL can use. It includes the following features:

Create CAs for the Authentication and Access control plugins
Create a directory tree containing all security files (keypairs, governance and permissions files, etc.)
Create new identity for a given node instance, generating a keypair and signing its x.509 certificate
Create a governance file that will encrypt all DDS traffic by default
Support specifying permissions in familiar ROS terms which are then automatically converted into low-level DDS permissions
Support automatically discovering required permissions from a running ROS system

Conclusion

I hope that helps clear up any misunderstandings you’ve had about what SROS 2 is and how it works today. We’re always here to help if you have any questions!

Canonical joins the ROS 2 Technical Steering Committee

Wed, 28 Aug 2019 00:00:00 +0000

We at Canonical care deeply about robotics. We firmly believe that robots based on Linux are cheaper to develop, more flexible, more secure, and faster to market. One of the contributing factors to this being the case is the Robot Operating System (ROS). ROS is by far the most popular middleware for creating Linux-powered robots. It provides all sorts of open source tools and libraries and pre-made components that solve common problems encountered during robot development. This allows roboticists to avoid needing to reinvent the wheel and instead focus on what really makes their robot unique. Of course, another reason we care about ROS is that most of the ROS community use Ubuntu. We love our users, and we want to make sure the experience they have on Ubuntu is consistently stellar!

We also care deeply about security, and that permeates everything we do. We’ve all seen how the IoT wave has been going in this regard: badly. IoT devices are low-margin, and no one has any incentive to keep them up to date or ensure that they’re secure in the first place. Manufacturers want to drive costs down, and users don’t consider the devices computers and don’t give a second thought to connecting them to the internet. It’s an unfortunate set of circumstances.

We think that the best way out of this situation is to make security and maintenance so easy that it becomes the obvious choice. If it was suddenly easier and cheaper for device manufacturers to create secure devices that can be automatically updated, why wouldn’t they do it? That’s the premise behind snaps and Ubuntu Core: by making complex topics like security and updates transparent and straightforward, we can make the entire ecosystem better for everyone.

Now, we aren’t quite that naive. A secure and robust operating system and packaging and update process are only pieces of a puzzle. If the software running on the devices has security holes, the picture is incomplete. Which brings us back to robotics and ROS.

Robotics is a realm of experimentation: evaluating different hardware and software, pushing ahead with what works, and culling what doesn’t. ROS 1, the version used by most production robots, was designed to be flexible to enable this experimentation. The flexibility of allowing existing components to be easily combined with new ones or swapped with others was valued above all else, at the cost of security.

We haven’t quite seen a wave of robots like we’ve seen with IoT, but we’re pretty sure it’s coming, and we want to make sure it doesn’t suffer the same fate. Just like IoT, the best way to make sure a production robot is secure is to make security as easy and transparent as possible. This is why we’re so interested in ROS 2.

ROS 2 is undergoing active development and is being built with the flexibility of ROS 1 while also supporting the technology necessary to secure it at its very core. A while ago, we joined the effort of fleshing out the ROS 2 security story; we’ve been helping with the design as well as software implementation. In order to effectively coordinate this work as well as show our commitment to it, we’re pleased to announce that we have become a member of the ROS 2 Technical Steering Committee.

We’re genuinely excited to be a part of such an open source powerhouse, and look forward to bringing our security expertise to bear as ROS 2 matures. With the right push during its development, we’re convinced ROS 2 will develop into an ecosystem where security comes naturally.

A better way to review GitHub pull requests for snaps

Fri, 03 May 2019 00:00:00 +0000

I’ve had a very particular itch for a long time, now. Something I love about open source is that it empowers developers to scratch their own itches, and a while back I decided to do exactly that. We’ve used it in production on the Nextcloud snap for a few months with relatively few issues, so I think it’s time to open it up for wider use… in case you share my itch. What is open source but a bunch of users scratching togeth…. okay this analogy is gross.

The problem

What is the problem that has annoyed me for years? It’s that reviewing pull requests for snaps sucks. It involves too much manual work. Sure, it’s relatively easy to automatically verify that the snap builds, as long as it’s a snap that actually builds within the Travis or CircleCI timeout. Then what? How do you verify that it actually works?

Testing that a snap actually works obviously depends on the snap in question. Nextcloud is a web application, so we actually have an entire suite of Capybara-powered tests to automatically verify the snap is sane. In some cases, that’s enough. For example, if a pull request is made that updates PHP, I can be relatively confident that none of those tests would pass if that PHP release was broken.

However, there are a whole host of things that can’t be automated. For Nextcloud, want to test that the Let’s Encrypt integration still works correctly? That’s all manual. What if it’s a GUI application? Those are a lot harder to fully test automatically. Besides, what if the PR changed the GUI itself? You probably want to take a look at it. Basically, you must install the snap.

Sadly, even if you build a snap as part of your CI, it’s usually lost after that ephemeral process. I mean, you can upload it to some sort of external service after it was built, but I’ve had mixed success with that, and it’s work. Besides, we already have one of those– the snap store! Why can’t we just upload there? Because it involves a very private token, which would need to be exposed on pull requests to the code random people submit, which means anyone can just steal your private token and start releasing snaps your behalf. I hope you agree that such a thing would be bad.

That turns the typical workflow for reviewing pull requests into this:

Code review
Pull the code
Build a snap locally
Test the snap locally

For every pull request. That’s actually a lot of work and a lot of time, and it meant that the snap moved quite a bit slower than I would have liked. I was always annoyed having to review pull requests (building the Nextcloud snap takes an hour), and I don’t want to be annoyed just because someone is trying to contribute! What a terrible attitude.

So I was on vacation a few months ago, and hacked together something that solved this problem, something that built and released snaps for every pull request without handing my token out to anyone who wanted it. I then reached deep into my imagination and came up with the name GitHub Snap Builder. This project is in beta and is far from finished (heck, it still has the default README). However, it’s in a usable state, and I wanted to write this blog post to get you started using it, if you’re interested. Let’s dive in, shall we?

The solution

GitHub Snap Builder (it just rolls off the tongue, right?) is a hosted solution. It runs on your own server, and you connect it to a GitHub App that you create in your organization (or under your user account). This allows it to run like any other CI system (e.g. Travis), report status on pull requests, etc. Let’s start by getting that GitHub App setup.

GitHub App

I’ll walk you through creating a GitHub App under your personal account, but doing the same for an organization is very similar. Let’s work off of the Creating a GitHub App page.

First, go to your profile settings, then to Developer settings in the left sidebar. From there click GitHub Apps in the left sidebar, and then hit the New GitHub App button.

Give it a name, and set the Homepage URL to whatever you want. The Webhook URL needs to be pointing at the GitHub Snap Builder, so use the URL of the server, e.g. http://domain-name.com. I know we haven’t talked about this yet, we’ll revisit it, just get something in there so you can create the app.

I know the Webhook secret says it’s optional, but if you’re using the GitHub Snap Builder (so fun to say) it’s not. Generate your own password, make it good, and put it in there. Keep track of it, you’ll need it in a minute.

Now for the permissions. GitHub Snap Builder requires the ability to read pull requests, and read/write commit statuses ( the place where your CI systems report success/failure). Under Permissions, scroll down until you see Pull requests and select Read-only in the drop-down. In that same section, find Commit statuses and select Read & write in the drop-down. Then continue scrolling down until you see the Subscribe to events section, and check the Pull request box (so GitHub Snap Builder is notified whenever a pull request is opened or updated).

Finally, hit the big green Create GitHub App button.

Now that your application is created, you need to do one more thing. Scroll down to the bottom of the page you’re on now (your new application) and you’ll see a big green Generate a private key button. Hit that. It’ll immediately download a private key that you’ll need in a minute. Keep the app page open, and move on to setting up your server.

Server setup

GitHub Snap Builder is written as a Ruby gem. As such, you can consume it in two ways. First of all, you can install the gem:

$ gem install github_snap_builder

Or you can install the snap:

$ sudo snap install github-snap-builder --beta --classic

We’re going to use the snap since it sets up the daemons and whatnot for us, but this post should still be useful for you even if you opt for the gem.

Once you install the snap, a daemon will be running. So how does this work? That daemon sits and waits for GitHub to tell it that a pull request event has happened (e.g. a pull request was created or updated). When that request comes in, GitHub Snap Builder will fire up an ephemeral container, build the snap, toast the container, create a new ephemeral container, push/release it to the proper channel, and toast the container again. Note the use of two different containers: it’s so we can’t possibly be running code from the pull request when we whip out the credentials necessary to upload the snap. GitHub Snap Builder was written so that the container tech is easily extended, but the only supported option today is Docker, so let’s install that:

$ sudo apt install docker.io

Why Docker, by the way? Because I have this thing deployed in a LXD container, and nested LXD containers have problems with snaps, but can run Docker just fine. Anyway, let’s configure GitHub Snap Builder. The following command will open up a YAML file in an editor:

$ sudo github-snap-builder.configure

You’ll see all sorts of options here with hopefully-helpful comments. Here’s where you need information from the GitHub App you just finished creating. Go to the App page, and grab its App ID. Set gitub_app_id in the config to that value. Set github_webhook_key equal to the webhook secret that you generated for the app. Grab the private key you downloaded, and open it up in a text editor. Copy the entire file (“BEGIN RSA PRIVATE KEY” and everything) and paste it into the github_app_private_key, formatting like the example given in the config.

Set build_type to “docker”. As I mentioned, nothing else is supported today.

You can set the bind address/port to whatever you want, but the defaults are given there (0.0.0.0:3000), and I left them alone.

Finally, let’s talk about that repos section. This is where you tell GitHub Snap Builder what repositories it’s authorized to build, what channel should be used for release, and provide the token that gives it permission to do so.

Each item under the repos section is a YAML object that is the slug of a given repo (<owner>/<repo name>). For the Nextcloud snap, that’s nextcloud/nextcloud-snap. Nested under that object are two options: channel and token.

When GitHub Snap Builder has built a snap, it will release it into a pull-request-specific branch in the store that only lives for 30 days, for example edge/pr-123. That branch is determined with the channel option, and is <channel>/pr-123.

The token option is something you’ll need to generate on your own machine. Run this command (replacing <snap-name> with the obvious thing and replacing <channel> with whatever you used for the channel option followed by a /*, e.g. edge/*:

$ snapcraft export-login\
           --snaps=<snap-name>\
           --channels=<channel>\
           --acls=package_push,package_release -

Yes, note the - at the end, which will print a token to the screen. Copy that token, and paste it into the token option, thereby giving permission for the GitHub Snap Builder to upload snaps to the set of branches required. You’ll need to go through this process for each repo/snap you want GitHub Snap Builder to support building.

We could stop here. We could make sure the GitHub App was hooked straight up to the GitHub Snap Builder. It would be a simple matter of editing the GitHub App settings and setting the Webhook URL to http://my-server:3000. However, doing that would send all that data in the clear (including your app password), which is a bad idea.

I suggest installing nginx and using it as an SSL terminator and proxy for the snap (getting SSL certs is an exercise left to the reader, but I suggest Let’s Encrypt, it’s free). The configuration I used was pretty simple:

server {
   listen *:443;
   ssl on;
   server_name <domain name>;
   ssl_certificate /etc/letsencrypt/live/<domain name>/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/<domain name>/privkey.pem;

   # teminate SSL and proxy to the actual internal web service.
   location / {
       proxy_pass       http://localhost:3000;
       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-Ssl on; # Optional
       proxy_set_header X-Forwarded-Port $server_port;
       proxy_set_header X-Forwarded-Host $host;
   }
}

At this point, let’s revisit the GitHub App settings, and make sure the Webhook URL hits that nginx endpoint. With that config, it ends up being https://my-server and that’s it.

Activate GitHub App on GitHub repo

You already configured GitHub Snap Builder to build a specific repo when pull requests happen, but now you need to go install your GitHub app on that specific repo to complete the circle and actually start sending events. On the GitHub App page, hit the Install App item in the left sidebar, and then hit the green Install button next to your username (again, you could do the same for organizations). You’ll be given the option to active it for all repositories, or you can select which repositories you have in mind. As long as the repository you configured in the GitHub Snap Builder is among those you select here, it’ll ignore the rest and do what you want. Finally hit the Install button, and you’re done! Submit a pull request for that repo and watch a snap be built and released. The branch it ends up using will be provided in the status string.

Install from that branch and you’ll get whatever the most recent change in that pull request produced. As the pull request is updated, GitHub Snap Builder will release updates into the same branch.

As always, I hope this is useful to you. Feel free to log issues, and pull requests are always welcome!

Speed up your ROS snap builds

Tue, 09 Apr 2019 00:00:00 +0000

A while back I wrote a post about distributing a ROS system among multiple snaps. If you want to enable some sort of add-on story, you need to have multiple snaps, and that remains the way to do it today with ROS. That approach works, but I’ll be the first to admit that it’s not terribly elegant, and we’re working on making it better. However, in my experience talking to various ROS users creating snaps, they oftentimes end up using multiple snaps not for an add-on or app store story, but for build speed. If you’re one of those people, keep reading, because there’s a better way to do this.

How can distributing one’s ROS system among multiple snaps increase speed? Because if you have a large number of dependencies that don’t change very often (e.g. upstream ROS components) you can package them all in their own snap. Then you can use its staging area when building the snap containing your own sweet, sweet code. This bypasses time spent fetching, unpacking, or building dependencies.

That’s great, and I’m happy that pattern has solved issues for folks. However, selecting it is also selecting the extra responsibility and maintenance burden that comes along with it. For example, handling the atomic update and rollback of a single snap is simple, but it quickly gets complex when you’re dealing with multiple snaps that need to be in lockstep. Basically, if you don’t need to distribute multiple snaps tied together with content sharing, I recommend avoiding it. Embrace bundling dependencies into one isolated snap that represents your product. Thankfully, there’s a way to do both: get the speed benefits of having multiple snaps, while only needing to distribute one.

`stage-snaps`

If you’ve built a snap in the past, you’re probably familiar with the [stage-packages](https://docs.snapcraft.io/snapcraft-parts-metadata/8336) option available to parts. It’s essentially a list of Debian packages for the snapcraft CLI to download, unpack, and distribute along with the rest of the part (i.e. bundle into the snap). As of version 3.2 of the snapcraft CLI, the stage-snaps option is also available, which offers the same functionality for snaps. Version 3.4 of the snapcraft CLI adds the functionality required to save as much time as possible when building ROS snaps using stage-snaps.

What this means is that, instead of needing to maintain a “producer” snap (which is a runtime dependency), as well as keep hold of its staging area for building a “consumer”, you can simply build a snap of all your dependencies and then bundle it straight into the “consumer” and only ship it as your end-product.

This is best explained with an example.

An example

Similar to the post about distributing a ROS system among multiple snaps, we’re going to create two snaps: one to contain common dependencies, and another to be our actual product. The word “base” has been overloaded by the snap project, so we’ll call our dependency snap our “foundation” and the product snap our “app” snap.

Create `ros-foundation`

Create a snap/snapcraft.yaml with the following contents:

name: ros-foundation-kyrofa
base: core18
version: '0.1'
summary: ROS foundational snap
description: |
 Contains roscore and basic ROS utilities. Meant for use as a stage-snap in
 other snaps. 

grade: stable
confinement: strict

parts:
 ros-foundation:
   plugin: catkin
   source: .
   include-roscore: true
   catkin-packages: []

You’ll notice I called mine ros-foundation-kyrofa to avoid any name clashes. It’s pretty useless on its own: it contains no apps, so it’s not like end-users can interact with it. It’s really just a blob of stuff. What stuff? roslib, roscore, std_msgs, etc. In this case it doesn’t contain any of my own Catkin packages, but it certainly could– got some libraries that don’t change very often? Put them here.

Run snapcraft on that, and after a little time you’ll have your “foundation” snap ready for use. Let’s get it into the stable channel of the store:

$ snapcraft register ros-foundation-kyrofa # You only need to do this once
$ snapcraft push ros-foundation-kyrofa_0.1_amd64.snap --release=stable

Create `ros-app`

The starting point for our “app” snap is the standard ROS talker/listener snap you’ve probably seen a thousand times. We’ll take it from the current source tree of the snapcraft CLI and tweak it a bit.

At this point, if you were distributing both foundation and app snaps, you’d go through the dance of tarring up the foundation staging area, adding a new part to the app snap, etc. We don’t need to do that in this case– our tweak is far simpler. Make the snap/snapcraft.yaml look something like this:

name: ros-app-kyrofa
base: core18
version: '0.1'
summary: ROS app snap
description: |
  Contains ROS talker/listener, including foundational ROS stuff from stage-snaps.

grade: stable
confinement: strict

apps:
 ros-talker-listener:
   command: roslaunch listener talk_and_listen.launch
   plugs: [network-bind]

parts:
 ros-app:
   plugin: catkin
   source: .
   include-roscore: false
   stage-snaps: [ros-foundation-kyrofa]

Other than the addition of the base option, the most important parts are the two lines at the bottom: the addition of the stage-snaps option, and the specification not to include roscore (because it’s already provided by our “foundation” snap). In order to see other effects of this new pattern, run snapcraft pull, which is responsible for downloading all dependencies of the snap:

$ snapcraft pull
<snip... installing build packages>
<snip... fetching roslib>
Downloading snap 'ros-foundation-kyrofa'
Pulling ros-app
<snip... fetching catkin, compilers, and rosdep>
Initializing rosdep database...
Updating rosdep database...
Determining system dependencies for Catkin packages...

That’s it. Normally, after “determining system dependencies,” you’d see it move on to actually fetching those dependencies. However, again as of v3.4, the snapcraft CLI is smart enough to realize that those dependencies are already contained within the “foundation” snap that we included in stage-snaps, so it doesn’t bother to fetch them again.

Go ahead and run snapcraft now to build the “app” snap the rest of the way, and then you can test it out without needing the “foundation” snap at runtime (since it’s bundled within the “app” snap now):

$ sudo snap install ros-app-kyrofa_0.1_amd64.snap --dangerous
ros-app-kyrofa 0.1 installed
$ ros-app-kyrofa.ros-talker-listener
<snip>
NODES
/
listener (listener/listener_node)
talker (talker/talker_node)
auto-starting new master
<snip>
[ INFO] [1554841533.046517090]: Hello world 0
[ INFO] [1554841533.146675333]: Hello world 1
[ INFO] [1554841533.246651599]: Hello world 2
[INFO] [1554841533.246916]: I heard Hello world 2
[ INFO] [1554841533.346644894]: Hello world 3
[INFO] [1554841533.347463]: I heard Hello world 3

As you can see, it works just as well as before, but now there’s no content sharing involved: just one snap. One snap that could be installed by anyone and work the exact same way.

Limitations

There are, of course, some limitations. For example. when building the “app” snap, you’ll notice that the snapcraft CLI fetches roslib, even though it’s already contained within the “foundation.” This happens because the snapcraft CLI doesn’t define the order in which it handles stage-snaps and stage-packages. This does waste some time (at least on the first pull; subsequent pulls should have it cached), and will need to be a future optimization.

Another limitation is that, oftentimes, dependencies are Debian packages, and there’s no simple way to break a Debian package’s dependency chain. For example, say your Catkin package depends upon A. If the stage-snap already contains A, great, the snapcraft CLI won’t fetch it again. However, let’s say your Catkin package depends upon B, which then depends upon A. In that case, unless the stage-snap also contains B, the snapcraft CLI will fetch both A and B, since the B Debian package depends upon the A Debian package, and that’s just the way the APT API works. There are some things we can try here, but again, it’s a future optimization.

Conclusion

If you’re one of the people trying to ship multiple snaps just because you want your app snaps to build faster, take a step back. Perhaps you could benefit from just using stage-snaps and only shipping a single snap to your users or devices. It will simplify your life, and you’ll still get the speed savings you were craving.

As always, feel free to ask any questions in the snapcraft forum.

Cipher update for the Nextcloud snap

Sun, 17 Mar 2019 00:00:00 +0000

Security is a always a balancing act between safety and practicality. If we’re talking about the internet, you can go all the way from something impenetrable and completely useless to something easy to use and also part of a botnet. In the case of the Nextcloud snap’s SSL cipher suite, as with any website, it’s a balance of supporting only the best ciphers while also ensuring that older clients aren’t left out in the cold.

The current cipher suite was selected several years ago, back when most folks were using Windows 7. We of course scan it periodically with SSL Labs and testssl.sh, and nightly with Nessus. We tweak it periodically to ensure it reflects the current security landscape and gets good results from those tools, but it hasn’t really changed that much.

As I mentioned earlier, one can always make the cipher suite more secure, but the side effect is that older clients who don’t support newer/better ciphers will no longer be able to communicate with the server. This is what our config currently looks like:

SSLProtocol all -SSLv3
SSLHonorCipherOrder On
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS;

This supports a lot of clients. Here’s the client simulation from testssl.sh:

Android 2.3.7               No connection
Android 4.1.1               TLSv1.0 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
Android 4.3                 TLSv1.0 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
Android 4.4.2               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Android 5.0.0               TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 6.0                 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 7.0                 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Chrome 51 Win 7             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Chrome 57 Win 7             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Firefox 49 Win 7            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Firefox 53 Win 7            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
IE 6 XP                     No connection
IE 7 Vista                  TLSv1.0 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
IE 8 XP                     No connection
IE 8 Win 7                  TLSv1.0 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
IE 11 Win 7                 TLSv1.2 DHE-RSA-AES256-GCM-SHA384, 4096 bit DH
IE 11 Win 8.1               TLSv1.2 DHE-RSA-AES256-GCM-SHA384, 4096 bit DH
IE 11 Win Phone 8.1 Update  TLSv1.2 DHE-RSA-AES256-GCM-SHA384, 4096 bit DH
IE 11 Win 10                TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Edge 13 Win 10              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Edge 13 Win Phone 10        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Opera 17 Win 7              TLSv1.2 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
Safari 5.1.9 OS X 10.6.8    TLSv1.0 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
Safari 7 iOS 7.1            TLSv1.2 ECDHE-RSA-AES256-SHA384, 256 bit ECDH (P-256)
Safari 9 OS X 10.11         TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Safari 10 OS X 10.12        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Apple ATS 9 iOS 9           TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Tor 17.0.9 Win 7            TLSv1.0 ECDHE-RSA-AES256-SHA, 256 bit ECDH (P-256)
Java 6u45                   No connection
Java 7u25                   TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
Java 8u31                   TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
OpenSSL 1.0.1l              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
OpenSSL 1.0.2e              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)

Here’s the same from SSL Labs:

That’s pretty darn good. Windows XP isn’t all that well supported, but heck, Vista is, and 7 and on definitely are. All these clients, and we still were able to get an A+ from SSL Labs.

Nowadays, though, some of these older clients just aren’t as important anymore. XP barely registers, Vista even less, and everyone gets a new phone every year or two (which is silly, by the way). The change in client landscape means it’s time to rebalance our cipher list against the clients we care to support. How do we go about determining that?

I won’t pretend I’m a security expert. I don’t at any one time know which ciphers are best and which are less best, which is why I rely on other folks and tools who do. So I decided to start with the strongest cipher list I could find and see what clients it eliminated. In order to find that list, I went to the Mozilla SSL config generator and turned everything up to 11 (i.e. checked the “modern” radio). It gave me this config:

SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

It actually supported a pretty good number of clients, but testssl.sh whined about it in the same way it whined about the current config:

LUCKY13 (CVE-2013-0169), experimental    potentially VULNERABLE, uses cipher block chaining (CBC) ciphers with TLS

Okay, so this ISN’T the ultimate locked down config. I scrolled down to testssl.sh’s cipher suite list to determine which ones were problematic:

Hexcode Cipher Suite Name (OpenSSL)      KeyExch.  Encryption Bits    Cipher Suite Name (RFC)
-----------------------------------------------------------------------------------------------------------------------------
 # ...
 xc028  ECDHE-RSA-AES256-SHA384          ECDH 256  AES        256     TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 # ...
 xc027  ECDHE-RSA-AES128-SHA256          ECDH 256  AES        128     TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

I removed the those ciphers from the list, and the config turned into:

SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256

And indeed, this got a clean bill of health from testssl.sh, but the number of clients it eliminated was… a lot:

Android 2.3.7               No connection
Android 4.1.1               No connection
Android 4.3                 No connection
Android 4.4.2               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Android 5.0.0               TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 6.0                 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 7.0                 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Chrome 51 Win 7             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Chrome 57 Win 7             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Firefox 49 Win 7            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Firefox 53 Win 7            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
IE 6 XP                     No connection
IE 7 Vista                  No connection
IE 8 XP                     No connection
IE 8 Win 7                  No connection
IE 11 Win 7                 No connection
IE 11 Win 8.1               No connection
IE 11 Win Phone 8.1 Update  No connection
IE 11 Win 10                TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Edge 13 Win 10              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Edge 13 Win Phone 10        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Opera 17 Win 7              No connection
Safari 5.1.9 OS X 10.6.8    No connection
Safari 7 iOS 7.1            No connection
Safari 9 OS X 10.11         TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Safari 10 OS X 10.12        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Apple ATS 9 iOS 9           TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Tor 17.0.9 Win 7            No connection
Java 6u45                   No connection
Java 7u25                   No connection
Java 8u31                   TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
OpenSSL 1.0.1l              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
OpenSSL 1.0.2e              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)

So, it seems I found the most locked-down config, and it was time to walk it back until I was comfortable. I definitely want to still support Internet Explorer on Windows 7 and 8, so I glanced back at the results from the current config and added DHE-RSA-AES256-GCM-SHA384 back to the list of supported ciphers. Still strong, still providing forward secrecy:

SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:DHE-RSA-AES256-GCM-SHA384

And it gave me the following results:

Android 2.3.7               No connection
Android 4.1.1               No connection
Android 4.3                 No connection
Android 4.4.2               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Android 5.0.0               TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 6.0                 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 7.0                 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Chrome 51 Win 7             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Chrome 57 Win 7             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Firefox 49 Win 7            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Firefox 53 Win 7            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
IE 6 XP                     No connection
IE 7 Vista                  No connection
IE 8 XP                     No connection
IE 8 Win 7                  No connection
IE 11 Win 7                 TLSv1.2 DHE-RSA-AES256-GCM-SHA384, 4096 bit DH
IE 11 Win 8.1               TLSv1.2 DHE-RSA-AES256-GCM-SHA384, 4096 bit DH
IE 11 Win Phone 8.1 Update  TLSv1.2 DHE-RSA-AES256-GCM-SHA384, 4096 bit DH
IE 11 Win 10                TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Edge 13 Win 10              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Edge 13 Win Phone 10        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Opera 17 Win 7              No connection
Safari 5.1.9 OS X 10.6.8    No connection
Safari 7 iOS 7.1            No connection
Safari 9 OS X 10.11         TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Safari 10 OS X 10.12        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Apple ATS 9 iOS 9           TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Tor 17.0.9 Win 7            No connection
Java 6u45                   No connection
Java 7u25                   No connectionJava 8u31
Java 8u31                   TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
OpenSSL 1.0.1l              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
OpenSSL 1.0.2e              TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)

And from SSL Labs:

That gives me client coverage I’m willing to accept given various usage statistics.

ROS 2 launch: required nodes

Mon, 11 Mar 2019 00:00:00 +0000

When using the Robot Operating System (ROS), it’s fairly common to want to shut down a launched system if a specific node (or set of nodes) exits. This is pretty easy in ROS1, because launch files support the required attribute on each node. As a result, crafting a two-node system where one of the nodes is required is straightforward:

<launch>
   <node name = "talker" pkg = "talker" type = "talker_node" />
   <node name = "listener" pkg = "listener" type = "talker_node" required = "true" />
</launch>

This launch file creates a talker/listener system where, if the talker exits the system continues trucking along, but if the listener exits the entire launched system is shut down.

In ROS 2 Crystal’s launch system, getting similar functionality involves a lot more boilerplate:

import launch
import launch_ros.actions

def generate_launch_description():
   talker = launch_ros.actions.Node(
       package="demo_nodes_py",
       node_executable="talker",
       node_name="talker",
       output="screen")

   listener = launch_ros.actions.Node(
       package="demo_nodes_py",
       node_executable="listener_qos",
       node_name="listener",
       output="screen",
       arguments=["--number_of_cycles", "1"])

   return launch.LaunchDescription([
       talker,
       listener,
       launch.actions.RegisterEventHandler(
           event_handler=launch.event_handlers.OnProcessExit(
               target_action=listener,
               on_exit=[
                   launch.actions.LogInfo(
                       msg="Listener exited; tearing down entire system."),
                   launch.actions.EmitEvent(event=launch.events.Shutdown())]))
   ])

This also creates a talker/listener system where, if the talker exits the system continues, but if the listener exits the launched system is shut down. However, you can see that, unlike the ROS 1 example, there are a few steps required to get there. Specifically, it requires the LaunchDescription to include an event handler to listen for an exit event for every required node which then emits a Shutdown event, which then FINALLY causes the launched system to shut down. That doesn’t scale particularly well to a real system where a large number of nodes may be required to run successfully.

We went through a few design iterations for how to best solve this, and decided that both the scaling and the boilerplate issues could be solved if the Node definitions could specify somehow that they were required. Rather than carry over that language from ROS 1, though, we decided to keep some commonality with OnProcessExit and simply add an on_exit action list directly to the Node definition. We also added a new action called Shutdown. Using these two new features together allows one to very simply specify that, if a given node exits, it should shut the entire launched system down. This greatly reduces boilerplate and scales far better than adding an event handler for each node. Take a look:

import launch
import launch_ros.actions

def generate_launch_description():
   return launch.LaunchDescription([
       launch_ros.actions.Node(
           package="demo_nodes_py",
           node_executable="talker",
           node_name="talker",
           output="screen"),

       launch_ros.actions.Node(
           package="demo_nodes_py",
           node_executable="listener_qos",
           node_name="listener",
           output="screen",
           arguments=["--number_of_cycles", "1"],
           on_exit=launch.actions.Shutdown())
   ])

This functionality is already available in master, and will of course be included in Dashing (scheduled for June).

Building ROS 2 snaps with Colcon

Thu, 28 Feb 2019 00:00:00 +0000

The snapcraft CLI has supported building ROS1 snaps for a while via the catkin plugin. We supported the ROS2 betas via the ament plugin, but that was before Open Robotics had a ROS 2 package repository setup, which meant that the ament plugin built the ROS 2 underlay from source, and it was predictably dreadfully slow. However, the stable releases of ROS2 introduced a new build system called colcon, and also had their own package repositories setup. Version 3.2 of the snapcraft CLI (just released today) sees the addition of a colcon plugin to support those new releases, and I’d like to give you a quick run-through of its capabilities.

Prerequisites

In order to follow along, make sure you have at least v3.2 of the snapcraft CLI. The best way to do that is to install the snap:

$ sudo snap install --classic snapcraft

Some previous experience building a snap will also be helpful.

Let’s get started

Create the `snapcraft.yaml`

First of all, create a new directory and initialize it with a snapcraft.yaml:

~$ mkdir ros2-snap
~$ cd ros2-snap/
~/ros2-snap$ snapcraft init
Created snap/snapcraft.yaml.
Go to https://docs.snapcraft.io/the-snapcraft-format/8337 for more information about the snapcraft.yaml format.

Open that snap/snapcraft.yaml file, and make it look like this:

name: ros2-talker-listener
version: "0.1"
summary: ROS2 Talker/Listener Example
description: |
  This example launches a ROS2 talker and listener.

grade: devel
confinement: strict
base: core18

parts:
 ros-demos:
   plugin: colcon
   source: https://github.com/ros2/demos.git
   source-branch: crystal
   colcon-rosdistro: crystal
   colcon-source-space: demo_nodes_cpp
   build-packages: [make, gcc, g++]
   stage-packages: [ros-crystal-ros2launch]

apps:
 run:
   command: opt/ros/crystal/bin/ros2 launch demo_nodes_cpp talker_listener.launch.py
   plugs: [network, network-bind]

Let’s break that down and go through it by section.

name: ros2-talker-listener
version: "0.1"
summary: ROS2 Talker/Listener Example
description: |
  This example launches a ROS2 talker and listener.

grade: devel
confinement: strict
base: core18

As I’ve said in the past, I typically use strict confinement unless I know for sure that the thing I’m snapping won’t run successfully under confinement, in which case I’ll use devmode. I typically avoid classic unless I never intend for the snap to run confined (e.g. you’ll notice the snapcraft CLI is a classic snap, since it needs more access to the host than confinement would allow).

Finally, the base keyword specifies a special kind of snap that provides a minimal set of libraries common to most applications (e.g. libc). It will be the root filesystem for this snap. In this case, we’re using core18 which is a minimal rootfs based upon Ubuntu Bionic (18.04).

parts:
 ros-demos:
   plugin: colcon
   source: https://github.com/ros2/demos.git
   source-branch: crystal
   colcon-rosdistro: crystal
   colcon-source-space: demo_nodes_cpp
   build-packages: [make, gcc, g++]
   stage-packages: [ros-crystal-ros2launch]

The snapcraft CLI is responsible for taking many disparate parts and orchestrating them all into one cohesive snap. You tell it the parts that make up your snap, and it takes care of the rest. Here, we’re saying that we have a single part called ros-demos. We specify that it builds using the colcon plugin, and we point it to the ROS 2 demos GitHub repository (this could just as easily be a directory on disk). We also specify that we’re using the newest ROS 2 release as of this writing: Crystal. We point the colcon plugin at the C++ demo nodes specifically. We also provide a list of packages that need to be installed in order to build (build-packages), and also ask that ros-crystal-ros2launch gets staged into the snap alongside the rest of the part to be used at runtime (specifically, we’ll use it in the app, below). To view all the options supported by the colcon plugin, run the command snapcraft help colcon.

apps:
 run:
   command: opt/ros/crystal/bin/ros2 launch demo_nodes_cpp talker_listener.launch.py
   plugs: [network, network-bind]

This part is interesting. When we build this snap, it will include a complete ROS 2 system: rclcpp, the demo_nodes_cpp workspace, etc. It could contain the entire system necessary for a robot in one installable blob. It’s a standalone unit: we’re in total control of how we want our users to interact with it. We exercise that control via the apps keyword, where we expose specific commands to the user. Here we specify that this snap has a single app called run. The command that this app actually runs within the snap uses the ros-crystal-ros2launch that we staged to fire up the demo nodes' talker/listener launch file. Finally, we use plugs to specify that this app requires network access (read more about interfaces).

Build the snap

~$ cd ros2-snap/
~/ros2-snap$ snapcraft
<snip>
Snapped ros2-talker-listener_0.1_amd64.snap

The build process will take a few minutes. You’ll see the snapcraft CLI fetch rosdep, which is then used to determine the dependencies of the packages in the colcon workspace. It then pulls those down and unpacks them into the snap, and finally builds the packages in the workspace and installs them into the snap as well. At the end, you’ll have your snap.

Test the snap

Let’s install the snap we just built:

~/ros2-snap$ sudo snap install --dangerous ros2-talker-listener_0.1_amd64.snap
ros2-talker-listener 0.1 installed

Finally, let’s run the app we defined in the snapcraft.yaml:

$ ros2-talker-listener.run
[INFO] [launch]: process[talker-1]: started with pid [26273]
[INFO] [launch]: process[listener-2]: started with pid [26274]
[INFO] [talker]: Publishing: 'Hello World: 1'
[INFO] [listener]: I heard: [Hello World: 1]
[INFO] [talker]: Publishing: 'Hello World: 2'
[INFO] [listener]: I heard: [Hello World: 2]

As you can see, it works great. You could hand this snap to anyone with a snap-capable system, even if they don’t have ROS installed, and it would work exactly the same way for them.

I hope this gives you a decent overview of the capabilities of the colcon plugin, and I look forward to seeing what you do with it! Please feel free to ask any questions on the Snapcraft forums, or on the ROS forums. I’d love to hear any feedback you have.

Are you passionate about open-source robotics? Join us.

Fri, 22 Feb 2019 00:00:00 +0000

Ubuntu is a very popular operating system for robotics, for a number of reasons. Perhaps the software stack one happens to use prefers Ubuntu, or the hardware one is using only distributes drivers for Ubuntu. Maybe one’s team is already familiar with it, or decision-makers like that it has long-term support releases and extended security maintenance. Perhaps one views snaps as the perfect way to support and update robots in the field, and so on. However, there’s a problem with this list: none of these reasons really have anything to do with robotics. That isn’t good enough.

Why is Ubuntu so popular in the cloud? There are a number of reasons (including some of the ones I already provided), but I want to talk about one in particular: because we at Canonical are experts in cloud technology. We have a team working on OpenStack. We have a team working on LXD. People know that if they have a problem, they can turn to us for help and we’ll be there. Heck, they can even have us manage their entire cloud for them if they want.

It’s time for us to do the same with robotics. We have a team with robotics expertise, and we’re looking for more talented engineers to join us. The work is challenging and diverse:

Working in upstream ROS1 and ROS2
Developing infrastructure to properly support ROS1 and ROS2 in Ubuntu
Designing and developing robotics-related features in snapd and snapcraft
Writing blog posts, creating videos, and presenting at conferences

There is really one big perk to this gig: you get to work with me. Mic drop.

Oh, that’s not enough? Sheesh, fine, you also get to work from home and travel the world, if you’re into that kind of thing.

If this sounds interesting to you, I’d love to talk. There are actually two different roles available that tackle slightly different aspects:

The roles are open as long as those links are live!

Develop your snaps faster

Thu, 25 Oct 2018 00:00:00 +0000

Snaps bundle their dependencies, which take time to fetch and unpack, and they’re typically made up of multiple parts, each of which need to be built and installed into the snap. Once you finish this process, you need to install the snap and test it out. Once you notice that your code is terrible and doesn’t work the way you thought, you have to make a change and go back through that process again.

The typical snap-building experience

This is the snap development process I see too often:

Run snapcraft, which fetches/builds dependencies and all parts, and creates a snap. This can take quite a chunk of time, depending on the snap.
Install the snap to test it out.
Realize it’s broken.
Tweak the code to fix.
Run snapcraft again.
One of two things happen, depending on the version of snapcraft being used: versions prior to 2.43 change nothing and happily create the exact same broken snap you just tested; later versions error out saying “hey, the source code changed.” You might find that latter option equally annoying, but it’s better, trust me. I’ll explain why in a minute.
As a result of (6), regardless of the outcome, the developer typically runs snapcraft clean and goes back to step (1).

Stop. There’s a better way. Then there’s actually an even better way.

The better way: careful cleaning

Let’s say you just finished step 4. If you run snapcraft clean right now, you’ll remove everything snapcraft has done: all the dependencies that have been unpacked for every part, all the source code fetched, everything. For the typical project, this shotgun approach is not necessary. Snapcraft supports cleaning individual parts, and even individual steps of individual parts if you want. In this example, you probably want to run:

$ snapcraft clean <part you changed>

As another example, say you just wanted to clean the stage step of “part1”:

$ snapcraft clean part1 --step=stage

Using this approach, you only clean exactly what you need cleaned, and nothing else. All the other parts will remain as they were, and when you run snapcraft again it’ll leave them alone and only update what you cleaned.

The best way: a smarter snapcraft CLI

As of version 3.0 of the snapcraft CLI, it’s gotten quite a bit smarter (2.43 also contains some of this, but it’s better in 3.0). The problem is, implanting the extra brains it needed involved some very significant changes, so we decided to leave it mostly disabled by default for now. The only indication you get that something has changed is that it bails if you change the source of a part. Let me show you how to flip the switch to enable the new behavior (with a caveat: it’s new, beware, and please log a bug if you see something weird).

Create a new file with the path ~/.config/snapcraft/cli.cfg, and put the following into it:

[Lifecycle]
outdated_step_action = clean

That’s it. Now your workflow should be more intuitive, like this:

Run snapcraft, which fetches/builds dependencies and all parts, and creates a snap. This can still take quite a chunk of time, depending on the snap.
Install the snap to test it out.
Realize it’s broken.
Tweak the code to fix.
Run snapcraft again.
Snapcraft should notice that your code changed, and will update that part to incorporate the changes. Return to step (2).

Much better, right? There’s one more improvement that can be made to this workflow.

Stop installing your snap to test

Snaps are squashfs images, which by definition are read-only. It takes time to create the image, and also requires you to install the snap again every time it changes. Both of these are problems in the development loop: you can’t tweak it easily, and needing to create a new one/install it again is time better spent coding. Allow me to introduce you to a feature that has existed for a while but about which few people seem to be aware: snap try.

When you run snapcraft with no arguments, you’re asking it to go through its entire lifecycle of steps and then finally create a snap. Specifically, that’s pull, build, stage, and finally prime. In the prime step, the snapcraft CLI creates a directory named prime/ and puts everything that makes up the final snap in there: the proper metadata that defines it as a valid snap, etc. The process of turning that into a snap is literally creating a squashfs image of the prime/ directory. If you then unsquashed that image, you’d end up with exactly the contents of the prime/ directory again. snap try is snapd’s way of supporting taking a snap for a spin from a directory instead of from a squashfs image. It essentially gives you a read/write snap, which changes your workflow again. Now it looks like this:

Run snapcraft prime, which fetches/builds dependencies and all parts, and creates all the metadata that turns it into a valid snap in the prime/ directory, but doesn’t create a snap out of it.
Run snap try prime/ to test out the snap defined in that directory.
Realize it’s broken.
Tweak the code to fix.
Run snapcraft prime again to incorporate your changes into the prime/ directory.
Test the snap out again (your changes are already available for use since they’re in the prime directory, no need to install or try it again). Return to step (3).

That’s a pretty nice workflow. You could even skip step 4 and 5 if you really just want to hack in the prime/ directory; your changes will immediately be reflected in the snap being tried.

I hope this proves useful to you. As always, if you have any questions, ask away in the Snapcraft Forum. Now go forth and develop snaps faster.

ROS Melodic on Fedora 27

Fri, 25 May 2018 00:00:00 +0000

Part of the snapcraft CLI’s development roadmap this cycle includes the ability to build snaps on distros other than Ubuntu. Part of that work includes investigating each plugin to determine if it’s possible to get it working on other distros. As a result, I’ve been looking into the Catkin plugin recently, using Fedora 27 as my proving ground (since that’s the distro with which I’m most familiar after Ubuntu).

The first thing you’ll notice trying to get ROS working on Fedora is that, unlike Ubuntu, where Open Robotics maintains infrastructure hosting a repository of Debian packages, there is no repository for you to use: you must build ROS from source. There’s some decent documentation for how to do this, but it’s severely lacking for Fedora, so I thought I’d share how I got it working.

The biggest issue I hit is that rosdep is completely useless for the dependencies of ROS itself. Honestly, I’m not sure why, a quick peek at the rules shows that it definitely supports Fedora. Maybe just none of the dependencies for ROS have Fedora 27 rules; I didn’t investigate.

So instead of using rosdep, I figured out the dependencies using good old trial and error. If you don’t feel like doing the same, you can borrow the ones I found (although I was only playing with ros_comm, no GUI tools):

# dnf install python-rosdep python-rosinstall_generator \
              python-rosinstall python-pyparsing \
              cmake gcc-c++ python-empy boost-devel console-bridge-devel \
              poco-devel python-devel tinyxml2-devel lz4-devel gtest-devel \
              bzip2-devel gpgme-devel python-defusedxml python-netifaces

Now create a workspace for ROS:

$ mkdir ~/ros_catkin_ws
$ cd ~/ros_catkin_ws

Fetch the source for Melodic:

$ rosinstall_generator-2.7 ros_comm --rosdistro melodic --deps --tar > melodic-ros_comm.rosinstall
$ wstool init -j8 src melodic-ros_comm.rosinstall

Now build ROS:

$ ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release

Finally, you can source the setup.bash for the workspace as usual:

$ source ~/ros_catkin_ws/install_isolated/setup.bash

Now you can run e.g. roscore:

$ roscore
... logging to /root/.ros/log/35<snip>85d360/roslaunch-fedora-ros-19796.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://fedora-ros:34509/
ros_comm version 1.14.1

SUMMARY
========

PARAMETERS
* /rosdistro: melodic
* /rosversion: 1.14.1

NODES

auto-starting new master
rocess[master]: started with pid [19806]

ROS_MASTER_URI=http://fedora-ros:11311/
setting /run_id to 35d993ce-6062-11e8-bcab-00163e85d360
process[rosout-1]: started with pid [19817]
started core service [/rosout]

Anyway, just wanted to get that out there for posterity, and in case it helps anyone else.

Your first robot: Sharing with others [5/5]

Fri, 16 Mar 2018 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the fifth (and final) blog post in this series about creating your first robot with ROS and Ubuntu Core. In the previous post we discussed methods of control, did a little math, and wrote the ROS driver for our robot. But it still required several nodes to be running at once, and sharing it with the world involved uploading your source code somewhere and convincing people to install ROS, build your package, and use it. Today we’re going to simplify both of those issues by discussing ROS launch files, and packaging everything we’ve written as a snap that can be installed by your friends with a few keystrokes, even without knowing anything about ROS.

Alright, let’s get started! Remember that this is also a video series, feel free to watch the video version of this post:

Prerequisites

This post will assume the following:

You know what snaps are, and have completed the “create your first snap” tutorial.
You have a store account at dashboard.snapcraft.io

Step 1: Create launch files

In parts 3 and 4, we needed to open several terminals because we needed to run several different ROS nodes. It’s difficult to remember every component of our system, and it gets exponentially harder as we start adding parameters, etc. Running individual nodes is handy for testing things out, but we’re done with that step. It’s time to make our system a little more of a product, something you can just run and use to control your robot. As you learned in the eighth ROS tutorial, ROS has a tool for dealing with this: launch files. These give us a simple language we can use to specify what nodes need to launch at the same time, and with what parameters.

Let’s get started writing launch files for the ROS system that controls our robot. We’ll put these launch files in the edukit_bot package we’ve been working on throughout this series, in a new launch/ folder. Make that folder now:

$ mkdir ~/edukit_bot_ws/src/edukit_bot/launch

In order for our robot (and launch files) to be reusable, we’re not actually going to create a single launch file. We want a single entry point (i.e. we want to be able to launch a single launch file and have it bring up the whole system), but we also want to keep things as modular as possible. So we’ll write one launch file per subsystem, and then one super launch file in charge of bringing up the entire system. We have two subsystems for our robot: low-level control, and teleoperation. Let’s write our “super” launch file first.

Step 1.1: Write robot launch file

Create a new file ~/edukit_bot_ws/src/edukit_bot/launch/edukit_bot.launch and open it up in your editor. Make it look like this:

<launch>
 <include file="$(find edukit_bot)/launch/includes/teleop.launch.xml" />
 <include file="$(find edukit_bot)/launch/includes/control.launch.xml" />
</launch>

That’s it. All this launch file does is “include” the launch files for our two subsystems (which we have yet to write). When this launch file is run, it runs those. Alright, let’s get started on our subsystems' launch files.

Step 1.2: Write teleop launch file

Why are we splitting these up, exactly? Let’s say in the future you want to work toward making this robot autonomous. In that case, you may no longer want the teleoperation nodes to be launched. Do you think you’ll remember which nodes relate to teleoperation and which don’t? I certainly won’t! It’s easier just to keep related pieces together, and remove the “include” line that launches the teleoperation nodes if you want to remove that functionality.

First, you need to create the includes directory into which we’ll place our subsystem launch files:

$ mkdir ~/edukit_bot_ws/src/edukit_bot/launch/includes

Create a new file ~/edukit_bot_ws/src/edukit_bot/launch/includes/teleop.launch.xml. Why do we end this one with .xml and not just .launch like the other one? Because roslaunch (the command that runs launch files) will tab-complete files that end in .launch. Since these subsystem launch files don’t do much by themselves, we don’t want them to be exposed to end users. Ending them with .xml essentially hides them from roslaunch, but still lets us use them with includes. Anyway, open that file in your editor. Make it look like this:

<launch>
 <node pkg="joy" type="joy_node" name="joystick">
   <param name="autorepeat_rate" value="1" />
 </node>

 <node pkg="teleop_twist_joy" type="teleop_node" name="joystick_to_twist">
   <param name="scale_angular" value="4" />
 </node>
</launch>

This file launches the two nodes that make up our teleoperation (i.e. what we discussed in part 3). Note that we’re using the autorepeat_rate parameter of the joy node. This parameter tells joy to rebroadcast joystick values, even if they haven’t changed, at a rate of at least 1 Hz (if the joystick moves, the rate will be much greater than 1 Hz, we just want to make sure we don’t run into our driver’s timeout functionality if the joystick simply hasn’t moved).

We’re also using the scale_angular parameter of the teleop_node so the robot actually turns (as we discussed in part 4).

Step 1.3: Update dependencies

We’ve just finished writing a launch file in our package that assumes the presence of other ROS packages (and their nodes). However, we’re not depending on those ROS nodes at all. Let’s add both joy and teleop_twist_joy to our list of dependencies in our edukit_bot’s package.xml, making it look something like this in the end:

<?xml version="1.0"?>
 <package format="2">
   <name>edukit_bot</name>
   <version>0.1.0</version>
   <description>The edukit_bot package</description>

   <maintainer email="you@you.com">You</maintainer>

   <license>GPLv3</license>

   <buildtool_depend>catkin</buildtool_depend>
   <build_depend>rospy</build_depend>

   <exec_depend>rospy</exec_depend>
   <exec_depend>geometry_msgs</exec_depend>
   <exec_depend>python-rpi.gpio</exec_depend>
   <exec_depend>joy</exec_depend>
   <exec_depend>teleop_twist_joy</exec_depend>
</package>

Step 1.4: Write the low-level control launch file

Create a new file ~/edukit_bot_ws/src/edukit_bot/launch/includes/control.launch.xml. Make it look like this:

<launch>
 <node name="driver" pkg="edukit_bot" type="driver_node" />
</launch>

This one is tremendously simple, as it’s just launching the driver we wrote in part 4. No parameters to tweak, since the default values are suitable (at least for my robot, perhaps yours is different).

We’re done making changes to our package, so let’s rebuild it:

$ cd ~/edukit_bot_ws
$ catkin_make

Step 1.5: Test launch system

Alright, let’s take our new launch files for a spin. This time, you only need a single terminal! Make sure it’s in your classic shell. First, as in the rest of the series, we need to make sure we have permission to access GPIO as a user (remember this resets upon reboot):

$ sudo chmod a+rw /dev/gpiomem

Now activate your workspace:

$ cd ~/edukit_bot_ws
$ source devel/setup.sh

Finally launch the super launch file, which will launch everything else:

$ roslaunch edukit_bot edukit_bot.launch

You’ll see all the nodes that make up our system come up, and you should be able to control your robot. So much easier than running them all individually!

Step 2: Package this workspace as a snap

At this point, we have our entire ROS system launchable by a single command. Wouldn’t it be neat to have this system running as soon as we boot our Raspberry Pi? Also, it would be cool to ask your friends to take your version of this series for a spin, and be able to test theirs out as well! Both of these are easily accomplished by packaging our ROS system as a snap. Note that the workspace (and snapcraft.yaml) used here is available for reference.

Step 2.1: Add install rules

The first step toward packaging anything is typically to ensure that we’re installing the components of our package correctly. With its use of workspaces, ROS makes this dangerously easy to ignore, but it’s still important. We have two components of our edukit_bot package: the driver, and the launch files. Open up edukit_bot’s CMakeLists.txt, and add the following two install rules to the bottom:

# ... <snip>

# Install driver node
install(PROGRAMS
 src/driver_node
 DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)

# Install launch files
install(DIRECTORY launch
 DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)

Now when we install our ROS package, it will install both our driver node as well as the launch files we’ve written.

Step 2.2: Install Snapcraft

In order to build a snap, we need to install a tool called snapcraft. From your classic shell:

$ sudo apt install snapcraft

Step 2.3: Write the `snapcraft.yaml`

We tell snapcraft how to create our snap by writing a snapcraft.yaml. We can get an initial version with snapcraft init:

$ cd ~/edukit_bot_ws
$ snapcraft init
Created snap/snapcraft.yaml.
Edit the file to your liking or run `snapcraft` to get started

Do as it asks, and open up snap/snapcraft.yaml in your editor. Make it look something like this:

name: edukit-bot-kyrofa
version: '1.0'
summary: ROS system for teleoperating edukit bot
description: |
 A ROS system for teleoperating the EduKit #3, the robotics kit.
 This accompanies the "Your First Robot" series written by Kyle
 Fazzari.

 Note that the `joystick` interface is needed for access to the
 gamepad, and the `physical-memory-control` interface is needed
 for access to GPIO. Both of these interfaces must be connected
 before this snap will function:

   $ sudo snap connect edukit-bot-kyrofa:joystick
   $ sudo snap connect edukit-bot-kyrofa:physical-memory-control 

grade: stable
confinement: strict

parts:
 workspace:
   plugin: catkin
   rosdistro: kinetic
   source: .
   catkin-packages: [edukit_bot]

   # Required to build RPi.GPIO
   build-packages: [python-dev, python-setuptools]

apps:
 launch:
   command: roslaunch edukit_bot edukit_bot.launch
   daemon: simple
   plugs:
     # Basic networking
     - network
     - network-bind

     # Needed for access to gamepad
     - joystick

     # Needed for GPIO memory-mapping
     - physical-memory-control

Let’s break this down piece by piece.

name: edukit-bot-kyrofa
version: '1.0'
summary: ROS system for teleoperating edukit bot
description: |
 A ROS system for teleoperating the EduKit #3, the robotics kit.
 This accompanies the "Your First Robot" series written by Kyle
 Fazzari.

 Note that the `joystick` interface is needed for access to the
 gamepad, and the `physical-memory-control` interface is needed
 for access to GPIO. Both of these interfaces must be connected
 before this snap will function:

   $ sudo snap connect edukit-bot-kyrofa:joystick
   $ sudo snap connect edukit-bot-kyrofa:physical-memory-control

This is just useful metadata for our snap. The name must be unique (that’s why I put my nickname at the end), but you’re pretty much free to put whatever you want for the rest. The description here hints at confinement, which we’ll cover in a moment.

grade: stable
confinement: strict

grade can be either stable or devel. If it’s devel, the store will prevent you from releasing into stable channels– think of it as a safety net to prevent accidental releases. If it’s stable, you can release it anywhere. I personally think this feature is a tad silly, and typically just set it to stable.

confinement can be strict, devmode, or classic. strict enforces confinement, whereas devmode allows all accesses, even those that would be disallowed under strict confinement (and logs accesses that would otherwise be disallowed for your reference). classic is even less confined than devmode, essentially running exactly like something you’d install from a deb. There is more extensive documentation on confinement available.

I personally always use strict confinement unless I know for sure that the thing I’m snapping won’t run successfully under confinement, in which case I’ll use devmode. I typically avoid classic unless I never intend for the app to run confined (e.g. if I was snapping vim, I know it would always need access to everything). In this case, this snap can run quite well under strict confinement. We’ll discuss this some more momentarily.

parts:
 workspace:
   plugin: catkin
   rosdistro: kinetic
   source: .
   catkin-packages: [edukit_bot]

   # Required to build RPi.GPIO
   build-packages: [python-dev, python-setuptools]

This is where we tell snapcraft how to build our Catkin workspace. Every snap is made up of a number of parts. In our case, there’s only one, called workspace. We say that it’s built using the catkin plugin, using kinetic (as opposed to Lunar or Indigo, as we discussed in part 2). We give it a list of Catkin packages to build, in this case just edukit_bot. Finally, we give it a list of packages that need to be installed in order to build our ROS package, or more specifically, our RPi.GPIO dependency.

apps:
 launch:
   command: roslaunch edukit_bot edukit_bot.launch
   daemon: simple
   plugs:
     # Basic networking
     - network
     - network-bind

     # Needed for access to gamepad
     - joystick

     # Needed for GPIO memory-mapping
     - physical-memory-control

Here we expose our single launch file that brings up the whole system. We create a new app called “launch” that runs the roslaunch command we used in step 4. We’re saying that this should be running at boot with daemon: simple. Finally, we provide a list of plugs, which is where confinement comes into play. (Note: if you’re using Ubuntu Core 18, you should use gpio-memory-control instead of physical-memory-control).

Since this snap is strictly confined, by default it doesn’t really have access to anything. No network, no GPIO, no controller. So we add plugs for each of those things: ROS needs the network for its nodes to function, we need access to the controller, and we need access to GPIO. Read more about interfaces.

That’s it! Let’s build it.

Step 2.4: Build the snap

This part is pretty easy:

$ cd ~/edukit_bot_ws
$ snapcraft

Note that the Raspberry Pi is not as blazingly fast as we’d all like it to be. This will take some time. You’ll see snapcraft fetch rosdep, which is then used to determine the dependencies of the ROS packages in the workspace. This is only edukit_bot in our case, which we know depends upon rospy, geometry_msgs, python-rpi.gpio, joy, and teleop_twist_joy. It then pulls those down and puts them into the snap along with roscore. Finally, it builds the requested packages in the workspace, and installs them into the snap as well. At the end, you’ll have your snap.

Step 2.5: Test your snap

Now that we have our snap built, let’s test it out! First, install it:

$ cd ~/edukit_bot_ws
$ sudo snap install edukit-bot-kyrofa_1.0_armhf.snap --dangerous
edukit-bot-kyrofa 0.1 installed

Now that the snap is installed, since we made our ROS system a daemon it’s already up and running. Unfortunately, it has already probably died because it can’t access the joystick or GPIO. We need to give it permission to do that by connecting the joystick and physical-memory-control interfaces:

$ sudo snap connect edukit-bot-kyrofa:joystick
$ sudo snap connect edukit-bot-kyrofa:physical-memory-control

We only need to do that once. Now that access has been granted, we just need to restart the service so it tries again:

$ sudo snap restart edukit-bot-kyrofa

Now you should be able to drive your robot around as before, but everything is running out of that single snap.

All you need to do to share your project with others is release the snap in the store!

Step 3.1: Tell Snapcraft who you are

We’re about to use snapcraft to register and upload a snap using the store account you created when satisfying the prerequisites. For that to work, you need to sign in with snapcraft. From your classic shell:

$ snapcraft login

Step 3.2: Register the snap name

As I mentioned before, snap names are globally unique, so only one developer can register and publish a snap with a given name. Before you can publish this snap you need to make sure that snap name is registered to you (note that this corresponds to the name field in the snapcraft.yaml we created moments ago). Again from your classic shell:

$ snapcraft register <my snap name>

Assuming that name is available, you can proceed to upload it. Otherwise you may need to rename/rebuild your snap to whatever name you managed to register (i.e. change the name field in the YAML and run snapcraft again, it won’t take long).

Step 3.3: Release the snap

There are four release channels available by default. In order of increasing stability, these channels are edge, beta, candidate, and stable. We’ve tested this snap and it’s strictly confined, so we can really release it wherever we want. Let’s go ahead and put it into the stable channel:

$ snapcraft push path/to/my.snap --release=stable

Once the upload and automated reviews finish successfully, anyone in the world can install your snap on the Raspberry Pi controlling their EduKit bot by running:

$ sudo snap install <my snap name>

Remember they also need to connect those interfaces so the snap can use the gamepad and GPIO. Mine (see the code) is also in the stable channel, so you can take it for a spin with:

$ sudo snap install edukit-bot-kyrofa

Conclusion

That’s all, folks! I hope you enjoyed this series, and I hope that this brief introduction to professional robotics will serve you well in the years to come. If this is a field that you’d like to pursue, I can’t recommend internships enough, especially if you’re at the university level. At this point, you can put experience with ROS and Ubuntu Core on your resume, which will help you stand out from the pack!

If you’re looking for ways to continue hacking on this robot using ROS, I suggest doing some research into ways to obtain the wheel speeds. Once you have that, you can start investigating morphing your driver node into something you can use with the diff_drive_controller, which goes back to ros_control, which we discussed in part 4. Who knows, maybe I’ll write another series about that someday.

Your first robot: The driver [4/5]

Fri, 09 Mar 2018 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the fourth blog post in this series about creating your first robot with ROS and Ubuntu Core. In the previous post we worked on getting data out of the wireless controller and into ROS in a format meant for controlling differential drive robots like ours: the Twist message. Today we’re going to create a ROS node that takes that Twist message and turns it into the appropriate wheel speeds to drive our robot.

Remember that this is also a video series, feel free to watch the video version of this post:

Alright, let’s get to it. There are a number of ways we could tackle this problem. If we were working on making our robot autonomous, we might use ros_control. However, it would add several more parts to this series, and we just don’t have the sensors available to make it worthwhile. Let me explain.

Open-loop versus closed-loop

Controlling a robot is always a loop of tasks done over and over again. For example, once we’re done with this blog post, our loop will look something like this:

Read data from controller
Turn controller data into velocity data
Turn velocity data into wheel speeds
Turn wheel speeds into duty cycles
Change the wheel duty cycle
Return to step 1

There are two types of control loops: open or closed. The difference is that a closed loop has feedback in it. As a simple example, let’s say you wanted to make some hot cocoa (it’s that time of year!) in the microwave. If you have a microwave like mine, you put the cocoa in, set the timer for a few minutes and come back when it’s done. The control loop of that microwave looks something like this:

Set power level to 10
Is time up yet? If not, return to step 1

There’s no feedback mechanism here: this is an example of open-loop control. The microwave is heating the cocoa only for a set amount of time, and that’s all it cares about. In fact, I often find it not hot enough and end up putting it back in.

However, maybe you have a fancy microwave. Maybe it includes a thermal sensor of some kind, perhaps a probe that you insert into the cocoa when you put it inside the microwave (gross, let’s say it’s infrared instead). Now the microwave can determine the temperature of the cocoa, which means you no longer have to set a timer, you just need to tell the microwave how hot you like your cocoa. Insert your desired temperature, and now the microwave’s control loop might look something like this:

Set power level to 10
How hot is the cocoa? Has it yet reached the desired temperature? If not, return to step 1

That thermal sensor gives the microwave some data for feedback in its loop, which means this is closed-loop control. This is probably the simplest one you can imagine, but there’s an entire field dedicated to closed-loop control, called control theory.

On our robot, we have no sensors that reasonably give us a way to determine a change in position. For example, if we had sensors on the wheels that told us how quickly they were spinning, we could use our knowledge of the wheel size to estimate how fast we’re going (this is called odometry). Since we have no such data, we have nothing to feed back: we’re limited to open-loop control right now.

That’s why using ros_control isn’t worth the effort: we don’t have the data necessary to close the loop, so we might as well just write our own super-simple open-loop controller instead. So let’s get on with it, shall we?

Prerequisites

There’s really only one new prerequisite here: a little more Python knowledge. We’ve kept it as simple as we could until now, but it’s time for our code to grow up a little. You’re already familiar with functions after following the worksheets. Today we’re going to use classes. Read this tutorial on classes, if needed.

Step 1: Calculate wheel speeds from Twist

Thanks to part 3, we have the controller generating Twist messages, which represent the desired linear and angular velocity of the robot (“go this fast forward/backward”, “turn this fast left/right”). At the end of the day, the only way the robot moves at all is by changing wheel direction and speed. We need to develop a way to convert the commanded velocities into wheel speeds that actually accomplish what was commanded. To do that, we need just a smidge of math.

Let’s talk about the linear velocity first, since it’s pretty simple. If we command the robot to move forward at 1 meter per second (m/s), in what direction (forward or backward) and at what speed (in m/s) does each wheel need to move? The answer should be intuitive: both wheels need to spin forward at 1m/s. So the formula for the linear velocity part of the wheel velocity is simply:

Angular velocity is a little more tricky.

If we command the robot to turn left at 90 degrees per second, in what direction (forward or backward) would each wheel need to turn? The answer is fairly intuitive, but it depends on how we want to make our robot turn. There are two options: single-wheel turning and double-wheel turning.

As you can see in the picture, above, single-wheel turning involves leaving one wheel stationary, and turning the other. Double-wheel turning involves rotating both wheels in opposite directions, thus sharing the work of the turn. If we elected to use single-wheel turning, then the answer to the question would be “The left wheel doesn’t turn at all, and the right wheel turns forward.” However, I prefer the double-wheel method, so my answer is “The left wheel turns backward, and the right wheel turns forward.”

Okay, we have the wheel directions. Now for the speed: if we command the robot to turn left at 90 degrees per second, at what speed (in meters per second) would each wheel need to turn? Note that this turn is a portion of a circle, the outside of which traces the desired path for our wheels. Recall the formula for the circumference of a circle:

How big of a portion of the circle are we wanting to turn? Well, we know that 360 degrees is an entire circle, so we can use a ratio and combine it with the circumference to create our formula for wheel speed (which is really just the formula for the length of an arc):

Alright, there’s only one more unknown in this formula: the radius. What is it? You can see that it’s actually half the distance between the two wheels (called the “track”). So our formula turns into this:

Awesome! Using this, we can now answer the question. I measured my robot’s track to be 0.091 meters. Using that, I calculate that the left wheel should be turning backward at 0.071 m/s, and the right wheel should be turning forward at the same speed.

We can actually simplify this formula, because the Twist message specifies angular velocities in radians per second as opposed to degrees. 360 degrees = 2π radians. If we change our ratio to use radians, things start canceling beautifully:

In addition, ROS conventions follow the right-hand rule, which means that a positive angular velocity means a counter-clockwise turn, and a negative angular velocity means a clockwise turn. Let’s decide right now that a positive value means that our wheels turn forward, and a negative value turns them backward. Using these two facts along with our formulas, we can come up with formulas for both wheel speeds:

Step 2: Convert wheel speeds into duty cycles

We got most of the math out of the way, but now we have a small issue that requires our attention. Our wheel speeds are in meters per second, but as you learned in CamJam worksheet #7, the way we actually control the motors is by applying a duty cycle between 0 and 100 (stopped and full speed, respectively). How do we get from meters per second to duty cycles? This is where closed-loop control would come in handy: if we had wheel speed sensors, we could compare how fast the robot was going to how fast it should be going, and say “it needs a higher duty cycle” or “it needs a lower duty cycle”. However, we already discussed that this needs to be an open-loop controller. We need to get our duty cycle by making some assumptions instead of using feedback data. We’ll do this by determining our robot’s maximum possible speed, and obtain the duty cycle by dividing the requested speed by the maximum speed.

How do we determine our robot’s maximum speed? The most accurate way would be to actually measure it. Essentially, measure out a meter, set the vehicle at the beginning, get out your stopwatch, and do a drag race.

Just for simplicity’s sake, though, we can cheat a little. If you noticed from part 3, by default the data from the controller has a maximum value of 0.5. So if we see a 0.5, we know the controller is maxed out. So if we just assume that our robot’s maximum speed is 0.5 meters per second, maxing out the controller will also max out our robot’s speed. It’s a little dirty, but it works for our case since we’re only using a controller anyway.

Step 3: Add geometry_msgs as a dependency

We’re about to rewrite the “driver” node we started in part 2 to handle Twist messages instead of String messages. This requires changing the std_msgs dependency of our edukit_bot package to geometry_msgs. Open up the edukit_bot package’s package.xml file and make it look something like this:

<?xml version="1.0"?>
<package>
 <name>edukit_bot</name>
 <version>0.1.0</version>
 <description>The edukit_bot package</description>

 <maintainer email="you@you.com">You</maintainer>

 <license>GPLv3</license>

 <buildtool_depend>catkin</buildtool_depend>
 <build_depend>rospy</build_depend>

 <exec_depend>rospy</exec_depend>
 <exec_depend>geometry_msgs</exec_depend>
 <exec_depend>python-rpi.gpio</exec_depend>
</package>

This is saying “I require catkin and rospy in order to build, and I require rospy, geometry_msgs, and python-rpi.gpio in order to run.”

Step 4: Write the driver

Now that we have the math out of the way, let’s write the ROS driver that utilizes it. In fact, we’ll rewrite the “driver” node in the edukit_bot package that we started in part 2. First, activate that workspace:

$ cd ~/edukit_bot_ws
$ source devel/setup.sh

Now make the ~/edukit_bot_ws/src/edukit_bot/src/driver_node file look like this (note that the entire package created in this part is available for reference):

#!/usr/bin/env python

import rospy
from geometry_msgs.msg import Twist

import RPi.GPIO as GPIO

# Set the GPIO modes
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

_FREQUENCY = 20

def _clip(value, minimum, maximum):
   """Ensure value is between minimum and maximum."""

   if value < minimum:
       return minimum
   elif value > maximum:
       return maximum
   return value

class Motor:
   def __init__(self, forward_pin, backward_pin):
       GPIO.setup(forward_pin, GPIO.OUT)
       GPIO.setup(backward_pin, GPIO.OUT)

       self._forward_pwm = GPIO.PWM(forward_pin, _FREQUENCY)
       self._backward_pwm = GPIO.PWM(backward_pin, _FREQUENCY)

   def move(self, speed_percent):
       speed = _clip(abs(speed_percent), 0, 100)

       # Positive speeds move wheels forward, negative speeds
       # move wheels backward
       if speed_percent < 0:
           self._backward_pwm.start(speed)
           self._forward_pwm.start(0)
       else:
           self._forward_pwm.start(speed)
           self._backward_pwm.start(0)

class Driver:
   def __init__(self):
       rospy.init_node('driver')

       self._last_received = rospy.get_time()
       self._timeout = rospy.get_param('~timeout', 2)
       self._rate = rospy.get_param('~rate', 10)
       self._max_speed = rospy.get_param('~max_speed', 0.5)
       self._wheel_base = rospy.get_param('~wheel_base', 0.091)

       # Assign pins to motors. These may be distributed
       # differently depending on how you've built your robot
       self._left_motor = Motor(10, 9)
       self._right_motor = Motor(8, 7)
       self._left_speed_percent = 0
       self._right_speed_percent = 0

       # Setup subscriber for velocity twist message
       rospy.Subscriber(
           'cmd_vel', Twist, self._velocity_received_callback)

   def _velocity_received_callback(self, message):
       """Handle new velocity command message."""

       self._last_received = rospy.get_time()

       # Extract linear and angular velocities from the message
       linear = message.linear.x
       angular = message.angular.z

       # Calculate wheel speeds in m/s
       left_speed = linear - angular*self._wheel_base/2
       right_speed = linear + angular*self._wheel_base/2

       # Ideally we'd now use the desired wheel speeds along
       # with data from wheel speed sensors to come up with the
       # power we need to apply to the wheels, but we don't have
       # wheel speed sensors. Instead, we'll simply convert m/s
       # into percent of maximum wheel speed, which gives us a
       # duty cycle that we can apply to each motor.
       self._left_speed_percent = (100 * left_speed/self._max_speed)
       self._right_speed_percent = (100 * right_speed/self._max_speed)

   def run(self):
       """The control loop of the driver."""

       rate = rospy.Rate(self._rate)

       while not rospy.is_shutdown():
           # If we haven't received new commands for a while, we
           # may have lost contact with the commander-- stop
           # moving
           delay = rospy.get_time() - self._last_received
           if delay < self._timeout:
               self._left_motor.move(self._left_speed_percent)
               self._right_motor.move(self._right_speed_percent)
           else:
               self._left_motor.move(0)
               self._right_motor.move(0)

           rate.sleep()

def main():
   driver = Driver()

   # Run driver. This will block
   driver.run()

   GPIO.cleanup()

if __name__ == '__main__':
   main()

Okay, that probably looks pretty new. Why don’t we break it down piece by piece.

#!/usr/bin/env python

import rospy
from geometry_msgs.msg import Twist

import RPi.GPIO as GPIO

# Set the GPIO modes
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

_FREQUENCY = 20

This is more or less straight out of part 2, although we’re importing the Twist message now instead of String. We’re also saving the frequency into a more conventionally-named variable, _FREQUENCY (the underscore indicates that it’s for internal use only, all-caps indicates that it’s a constant).

def _clip(value, minimum, maximum):
   """Ensure value is between minimum and maximum."""

   if value < minimum:
       return minimum
   elif value > maximum:
       return maximum
   return value

The _clip function is pretty simple: it makes sure a given value is between the given minimum and maximum. We’ll use it later to make sure we don’t try to make the motors move with a duty cycle less than 0 or greater than 100.

class Motor:

Here we’re creating a new class to represent a motor that we can move.

   def __init__(self, forward_pin, backward_pin):
       GPIO.setup(forward_pin, GPIO.OUT)
       GPIO.setup(backward_pin, GPIO.OUT)

       self._forward_pwm = GPIO.PWM(forward_pin, _FREQUENCY)
       self._backward_pwm = GPIO.PWM(backward_pin, _FREQUENCY)

This is the initializer for the Motor class. It’s called automatically whenever a new instance is created. It accepts two parameters, one for each pin involved in moving the motor (one forward, one backward). It then sets the pins up as outputs, and saves off the PWMs for use in the move() function:

   def move(self, speed_percent):
       speed = _clip(abs(speed_percent), 0, 100)

       # Positive speeds move wheels forward, negative speeds
       # move wheels backward
       if speed_percent < 0:
           self._backward_pwm.start(speed)
           self._forward_pwm.start(0)
       else:
           self._forward_pwm.start(speed)
           self._backward_pwm.start(0)

This is the main worker function of the Motor class: it’s how the motor moves. It accepts a positive or negative percent (between 0-100), where a positive value moves forward, and a negative value moves backward. Here’s where we utilize the _clip() function we discussed above to ensure the percentage requested does not fall outside the valid range of 0-100, which turns it into a valid duty cycle.

class Driver:

Here’s one more class, representing the ROS driver itself.

   def __init__(self):
       rospy.init_node('driver')

       self._last_received = rospy.get_time()
       self._timeout = rospy.get_param('~timeout', 2)
       self._rate = rospy.get_param('~rate', 10)
       self._max_speed = rospy.get_param('~max_speed', 0.5)
       self._wheel_base = rospy.get_param('~wheel_base', 0.091)

       # Assign pins to motors. These may be distributed
       # differently depending on how you've built your robot
       self._left_motor = Motor(10, 9)
       self._right_motor = Motor(8, 7)
       self._left_speed_percent = 0
       self._right_speed_percent = 0

       # Setup subscriber for velocity twist message
       rospy.Subscriber(
           'cmd_vel', Twist, self._velocity_received_callback)

Here’s the initializer for the Driver class, called automatically when a new instance is created. It doesn’t accept any parameters, since it supports changing its behavior by way of parameters from the ROS Parameter Server (which you learned a bit about in ROS tutorial number 7). We start out by initializing the ROS node, which begins communication with the ROS master. We then record the current time (which we’ll use later) and retrieve the values for all the parameters we support. After that, we create two instances of the Motor class defined above to represent the left and right motors, and initialize our speeds to zero.

A quick aside: these pins correspond to which motor is motor A, which is motor B, and the polarity that we used to hook them up. If you happened to connect yours differently than I connected mine (which is perfectly fine), you may need to switch these pins around a little or your robot will move hilariously badly.

Finally, we subscribe to our Twist topic, which is called cmd_vel (following convention, it means “commanded velocity”), and request that the _velocity_received_callback() function is called whenever a new command is received.

   def _velocity_received_callback(self, message):
       """Handle new velocity command message."""

       self._last_received = rospy.get_time()

       # Extract linear and angular velocities from the message
       linear = message.linear.x
       angular = message.angular.z

       # Calculate wheel speeds in m/s
       left_speed = linear - angular*self._wheel_base/2
       right_speed = linear + angular*self._wheel_base/2

       # Ideally we'd now use the desired wheel speeds along
       # with data from wheel speed sensors to come up with the
       # power we need to apply to the wheels, but we don't have
       # wheel speed sensors. Instead, we'll simply convert m/s
       # into percent of maximum wheel speed, which gives us a
       # duty cycle that we can apply to each motor.
       self._left_speed_percent = (100 * left_speed/self._max_speed)
       self._right_speed_percent = (100 * right_speed/self._max_speed)

This is the _velocity_received_callback() function, part of the Driver class. Thanks to our subscription in __init__(), this function is called whenever a new command message comes in. Here’s where the math we did in steps 1 and 2 come into play. First of all, we record the time we received the message (which we’ll discuss in a moment). Then we extract the linear and angular velocity components out of the message. Using this information, we use the wheel speed formula we derived in step 1 to calculate the left and right wheel speeds (in meters per second). Then we use the duty cycle formula derived in step 2 to turn those wheel speeds into left and right percentages. Since these values can be negative, they represent both the desired duty cycle as well as wheel direction.

Note that this function doesn’t actually change the wheel speeds at all, it just calculates what they should be. Why? We’ll discuss that in a moment.

   def run(self):
       """The control loop of the driver."""

       rate = rospy.Rate(self._rate)

       while not rospy.is_shutdown():
           # If we haven't received new commands for a while, we
           # may have lost contact with the commander-- stop
           # moving
           delay = rospy.get_time() - self._last_received
           if delay < self._timeout:
               self._left_motor.move(self._left_speed_percent)
               self._right_motor.move(self._right_speed_percent)
           else:
               self._left_motor.move(0)
               self._right_motor.move(0)

           rate.sleep()

This is the control loop of our Driver class. It’s where we actually apply the newly-calculated wheel speeds, in a loop that runs at a specific rate (10 Hz by default). What if we don’t receive a new commanded velocity by the time we loop again? We just use the previously-commanded one. That seems a little odd– why do we have a loop at all if we only ever use the commanded velocity?

Let’s say we pulled this logic out of the loop and put it in the message handler (the _velocity_received_callback() function) instead. What happens if we received a few commanded velocities, but then the network went down, or we otherwise lost communication with the controller? The robot would just continue moving at the previously-commanded velocities, and you would have to go chase it down. I’ve had that happen on a robot that weighed several hundred pounds– it went out of range of the control unit and just kept on trucking. My legs were sore for a week.

By putting this logic in a loop, and recording when commanded messages come in, we can implement a timeout to make sure our robot never runs away from us if issues occur. You can see this in the loop: we calculate how long it’s been since we last received a commanded velocity. If it’s less than the timeout (2 seconds by default) the calculated wheel speeds are used. If it’s greater than the timeout (i.e. we haven’t received a message for a while), then we halt the robot.

def main():
   driver = Driver()

   # Run driver. This will block
   driver.run()

   GPIO.cleanup()

This is the main part of the program, where we simply create a new instance of the Driver class, start its control loop, and clean up when it’s done.

if __name__ == '__main__':
   main()

This is the entry point of this whole node– we simply run the main() function.

Alright, we’re done with this node. Let’s build it before moving on:

$ cd ~/edukit_bot_ws
$ catkin_make

Step 5: Test the driver node

As in part 2, we need to make sure we have permission to access GPIO as a user (remember this resets upon reboot):

$ sudo chmod a+rw /dev/gpiomem

We’re also going to use what we learned in part 3 to get the controller giving us Twist messages. Open up four terminals running the classic shell. In the first one, run roscore:

$ roscore

In the second, run the joy node:

$ rosrun joy joy_node
[ INFO] [1515003691.568172834]: Opened joystick: /dev/input/js0. deadzone_: 0.050000.

In the third, run the teleop node:

$ rosrun teleop_twist_joy teleop_node
[ INFO] [1515003734.469189240]: Teleop enable button 0.
[ INFO] [1515003734.469664500]: Linear axis x on 1 at scale 0.500000.
[ INFO] [1515003734.469924707]: Angular axis yaw on 0 at scale 0.500000.

And finally, in the fourth terminal activate our workspace and run our new driver node:

$ cd ~/edukit_bot_ws
$ source devel/setup.sh
$ rosrun edukit_bot driver_node

Now turn on your controller and start moving your robot around! Oh wait, you tried to turn and nothing happened? Right, let me explain.

Remember in step 2 how we used the fact that the twist messages we were getting with the controller maxed out at 0.5? That works great for linear velocities, but the controller also maxes out at 0.5 radians per second for its angular velocity, which is so slow that the motors can’t actually pull it off (they just give off a quiet buzzing sound instead, as if they want you to know that they’re doing their best).

The solution is to request the teleop node to scale those values to more reasonable turning speeds. Go to the terminal where you ran the teleop node, ctrl+c it, and re-run it with an angular scaling factor. You can experiment with this to see how responsive/twitchy you like your robot, but I settled on a value of 4 (larger values will make it more sensitive):

$ rosrun teleop_twist_joy teleop_node _scale_angular:=4
[ INFO] [1515005430.818803229]: Teleop enable button 0.
[ INFO] [1515005430.819282707]: Linear axis x on 1 at scale 0.500000.
[ INFO] [1515005430.819531978]: Angular axis yaw on 0 at scale 4.000000.

NOW you should be able to successfully drive it around.

In the next (and final) post in this series, we’ll talk about how launch files save us from having to open a billion terminals just to run a few ROS nodes, and turn our ROS package into a snap that starts on boot and is easily installed by your friends!

Your first robot: The controller [3/5]

Sat, 10 Feb 2018 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the third blog post in this series about creating your first robot with ROS and Ubuntu Core. In the previous post you were introduced to the Robot Operating System (ROS), and got your robot moving by ROSifying one of the CamJam worksheets. Today we’re going to move beyond the CamJam worksheets, and work toward having our robot remotely controlled by focusing on our wireless controller: getting data out of it and into ROS messages.

As I mentioned in part 1, I’m using the wireless controller from the Pi Hut, but really the only requirement to follow this post exactly is that it shows up as /dev/input/jsX on the Pi (where X is some number, typically 0). I have a wired Xbox 360 controller that works the same way, for example.

Alright, let’s get started! Remember that this is also a video series, feel free to watch the video version of this post:

Step 1: Get data out of the controller

One of the benefits of using ROS is that a lot of the problems are already solved for us. In this case, there’s already a ROS node that exists solely to read data from controllers like ours, called joy. Get into your classic shell, and install it:

$ sudo apt install ros-kinetic-joy

Insert the USB receiver for the controller into the Pi, and you should see it appear as /dev/input/js0. Open up two others terminals, and get into your classic shell in each (so now you have three). On one terminal, run the master:

$ roscore

On another terminal, run the joy node:

$ rosrun joy joy_node
[ INFO] [1513274242.219112701]: Opened joystick: /dev/input/js0. deadzone_: 0.050000.

On the third terminal, let’s take a look at what’s coming out of the joy node. First, let’s see what topics are available:

$ rostopic list
/diagnostics
/joy
/rosout
/rosout_agg

Take note of the /joy topic. That’s being published by the joy node. Let’s take a look. Run:

$ rostopic echo /joy

Nothing should happen at first. Now turn on your controller, and start moving joysticks about and pressing buttons. You should see all sorts of stuff start spewing, looking something like this:

header:
seq: 1
stamp:
secs: 1513274448
nsecs: 114444863
frame_id: ''
axes: [0.0, 0.08846031129360199, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
buttons: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
---
header:
seq: 2
stamp:
secs: 1513274448
nsecs: 130450677
frame_id: ''
axes: [0.0, 0.26209455728530884, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
buttons: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
---

That’s a bunch of Joy messages representing the current state of the controller: joysticks, buttons, everything. Whenever you move a stick or press a button, more messages are sent showing the state change. That saved us a bunch of work! Go ahead and CTRL+C out of rostopic, the joy node, and roscore.

Step 2: How do we control our robot?

So we’re able to get data out of the controller, but that data is still somewhat raw. Remember that, at the end of the day, the robot moves and turns via changes in its wheel speeds. How do we get from these controller messages to wheel speeds? We could totally write a new node that subscribes to the controller messages, pulls out the joystick movements, and turns those into wheel speeds. However, that would tightly couple the robot driver to the controller, which would be short-sighted. Is there not a more generic way to request that the robot move and turn? Indeed there is.

The answer can be found by looking forward to how we would make the robot autonomous. This would probably utilize the ROS navigation stack, which was written for arbitrary robots, and outlines exactly how we should control our robot with its first hardware requirement:

“[This software] is meant for both differential drive and holonomic wheeled robots only. It assumes that the mobile base is controlled by sending desired velocity commands to achieve in the form of: x velocity, y velocity, theta velocity.”

~ The ROS Navigation Stack

Okay, so what does this mean? The robot we have is a differential drive robot. That means it can move forward, backward, and turn left and right, all with wheels that don’t steer like your car (which would be e.g. Ackermann steering). It does this by rotating the wheels at different speeds and/or in different directions. It cannot, however, move side-to-side. Wheeled holonomic systems are more magical in that they can move forward, backward, turn left and right, and move side-to-side. These typically use either omni wheels or mecanum wheels.

We can make this easier to understand by drawing the movements possible by each drive type:

On the left we have the differential drive system, like our robot. It can move in a linear fashion along the red line we labeled X (i.e. it can move forward and backward), and it can move in an angular fashion (rotate) around the green circle we labeled θ (theta). On the right we have the holonomic drive system, which can do exactly the same as the differential system but can also move along the blue line we labeled Y (i.e. it can move side-to-side).

Concretely, the ROS navigation stack supports both systems by supplying commands that are velocities in each of these directions (X, Y, and θ). So let’s make our robot support those types of commands! Since our system doesn’t support Y velocities, we’ll just be using X and θ. ROS has a standard message that is used for exactly this type of data: the Twist message, which consists of a set of linear and angular velocities. So let’s turn our Joy messages into Twist messages.

Step 3: Turn Joy messages into Twist messages

I might sound like a bit of a broken record, but guess what: this already exists, too! It’s called teleop_twist_joy. Let’s install it:

$ sudo apt install ros-kinetic-teleop-twist-joy

Now open four terminals running the classic shell (I know, this is getting ridiculous). In one, run roscore again:

$ roscore

In another, run the joy node again:

$ rosrun joy joy_node
[ INFO] [1513274242.219112701]: Opened joystick: /dev/input/js0. deadzone_: 0.050000.

In the third, run the newly-installed teleop node:

$ rosrun teleop_twist_joy teleop_node
[ INFO] [1513286128.076671780]: Teleop enable button 0.
[ INFO] [1513286128.077269945]: Linear axis x on 1 at scale 0.500000.
[ INFO] [1513286128.077666187]: Angular axis yaw on 0 at scale 0.500000.

And finally, in the fourth let’s poke at it. First, take a look at the available topics:

$ rostopic list
/cmd_vel
/diagnostics
/joy
/rosout
/rosout_agg

We know that the /joy topic contains the controller data, but /cmd_vel is new. That’s the conventional name for Twist-based velocity commands. Let’s take a look at those:

$ rostopic echo /cmd_vel

As before, nothing should happen at first. Then turn on your controller and start moving the left joystick. Psych! That doesn’t do anything either. Because the teleop node was written for remote control in mind, it implements a safety button as a sort of “dead-man’s switch” if you will. By default, this is button 0, which on my controller is the X button. Hold down that button, and then move the left joystick, and you should start seeing velocity commands start scrolling past, like this:

linear:
 x: 0.0
 y: 0.0
 z: 0.0
angular:
 x: 0.0
 y: 0.0
 z: 0.0
---
linear:
 x: 0.0225298888981
 y: 0.0
 z: 0.0
angular:
 x: 0.0
 y: 0.0
 z: 0.0
---

You’ll notice that while there are a bunch of components to this message, only two of the components actually change depending on how you move that joystick: the linear X component, and the angular Z component. This actually relates back to the picture in step 2: X corresponds to the red X line we drew, and Z corresponds to the green θ circle. So X values mean “move forward or backward this fast” and Z values mean “rotate left or right this fast”. Exactly what we need!

In the next post in this series, we’ll rewrite the simple “driver” node we created last time to accept Twist messages and turn them into wheel speeds.

Your first robot: Introduction to the Robot Operating System [2/5]

Sat, 20 Jan 2018 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the second blog post in this series about creating your first robot with ROS and Ubuntu Core. In the previous post we walked through all the hardware necessary to follow this series, and introduced Ubuntu Core, the operating system for IoT devices. We installed it on our Raspberry Pi, and used it to go through the CamJam worksheets. In this post, I’m going to introduce you to the Robot Operating System (ROS), and we’ll use it to move our robot. We’ll be using ROS throughout the rest of the series. Remember that this is also a video series, feel free to watch the video version of this post:

What is the Robot Operating System?

At its simplest, ROS is a set of open-source libraries and tools meant to ease development of robots. It also provides an infrastructure for connecting various robotic components together. For example, if you happened to go through all of the CamJam worksheets (particularly #9), you’ve written a single Python script that’s responsible for a bunch of things: controlling the motors, reading from the line detector, reading from the ultrasonic sensor, etc. What if we threw a wireless controller into the mix? That script quickly gets complicated, and if you happen to want to swap one component out for another, you need to rewrite the whole thing; i.e. these logically different components are tightly coupled together since they’re in the same script.

ROS provides a communication infrastructure that allows you to extract different logic into their own modules, and have them communicate with each other in a standard way, as depicted in the picture above. For example, if you wanted to switch ultrasonic sensors, and rewrite the “distance” module, you could do that without having to touch any of the other modules as long as the new distance module talked the same way as the old one.

This will all make more sense once we dive in, so let’s get started, shall we?

Step 1: Install ROS on the Raspberry Pi

As of this writing, ROS has three supported releases: Indigo Igloo, Kinetic Kame, and Lunar Loggerhead. Ubuntu Core series 16 (the one we’re using) is Ubuntu Xenial, which limits our options to Kinetic and Lunar. Lunar is technically newer and shinier, but like Ubuntu, ROS has Long-Term Support (LTS) releases that are supported for an extended period of time, and Kinetic is their most recent LTS. As a result, we’ll use Kinetic here.

SSH into your Pi, and get into your classic shell:

$ sudo classic

Let’s follow ROS Kinetic’s install guide. ROS maintains its own repository of Debian packages, which we need to add to our system:

$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'

We then need to add that repository’s keys to the list of keys we’ll accept (this verifies that packages in that repository really do come from ROS):

$ sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

Now we’ll re-index all the repositories we have configured, since we just added one:

$ sudo apt update

Now let’s install ROS. As you’ll see on the install guide, there are a bunch of meta packages available (packages that exist solely to pull in other packages). Let’s install the smallest, bare-bones one, ros-kinetic-ros-base, which will take up around 700MB. We’ll also install g++, the C++ compiler (it’s still required, even though we’re writing Python):

$ sudo apt install g++ ros-kinetic-ros-base

At this point, ROS is successfully installed, but none of its tools are available to run. That’s because ROS installs itself into what it calls a “workspace”, and provides a shell script that activates that workspace. We can make sure we activate that workspace upon login by adding it to the .bashrc file in our home directory:

$ echo "source /opt/ros/kinetic/setup.bash" >> ~/.bashrc
$ source ~/.bashrc

Now you should be able to run roscore without troubles:

$ roscore
<snip>
SUMMARY
========
PARAMETERS
* /rosdistro: kinetic
* /rosversion: 1.12.12
NODES
auto-starting new master
process[master]: started with pid [4987]
ROS_MASTER_URI=http://localhost.localdomain:11311/
setting /run_id to 1db9f4c6-e044-11e7-9931-b827eba43643
process[rosout-1]: started with pid [5000]
started core service [/rosout]

Go ahead and quit that by pressing CTRL+C.

Step 2: Get to know ROS

One of the reasons I like ROS so much (and, I think, one of the reasons it’s so popular) is that their introductory documentation is fantastic. They have a phenomenal set of tutorials that take you from knowing absolutely nothing to feeling more or less comfortable with the entire system. Each one is easily digestible in a few minutes. Because they are so good, rather than try and duplicate their hard work here, you should just start at the beginning and go through them at least until you complete #13, “Examining the Simple Publisher and Subscriber”. Note that there are two parallel tutorial tracks, one that uses C++ and one that uses Python. We’ll be using Python through this series, so you don’t need to worry about the C++ ones unless they interest you.

Step 3: Setup Python 2

Now that we’ve gained some familiarity with ROS, it’s almost time to make our robot move using it. However, there’s something we need to do first. Back in CamJam worksheet #1, they mention the following:

“When the Raspberry Pi was first released, some of the important Python libraries were only available for Python 2.7. However, almost every library, and all the ones used in these worksheets, are available for Python 3.2. It has been decided that all code for this EduKit will be developed for Python 3.2.”

~ CamJam worksheet #1

That’s all fine and dandy, and I agree with it, but unfortunately the Python bindings for ROS are only officially supported on Python 2, so we need to use Python 2 from now on instead of Python 3. Don’t worry, all the code from the worksheets should still work, but it means we need to install the Python 2 version of RPi.GPIO (we only have the Python 3 version right now):

$ sudo apt install python-dev python-pip python-setuptools
$ pip install RPi.GPIO

Step 4: Create ROS package for our robot

Alright, let’s have some fun! We’re going to rewrite the code we wrote for the CamJam Worksheet #7 using ROS. We’ll add some message handling to it, so, using ROS, we can command the robot to move forward, turn left, turn right, etc.

The first step is to create a new workspace. You learned how to do this in the first ROS tutorial. I’m calling mine “edukit_bot_ws”, if you call yours something else remember to change the directions accordingly:

$ mkdir -p ~/edukit_bot_ws/src
$ cd ~/edukit_bot_ws/src
$ catkin_init_workspace

Now let’s create a new package in that workspace. I’ll call mine “edukit_bot”, and it has three dependencies: rospy (the Python bindings for ROS), std_msgs (the standard ROS messages, e.g. numbers, strings, etc.), and python-rpi.gpio (RPi.GPIO, which we use for GPIO access):

$ cd ~/edukit_bot_ws/src
$ catkin_create_pkg edukit_bot rospy std_msgs -s python-rpi.gpio

Step 5: Write the ROS node

Time to write some code. First, create a new Python script in the ROS package’s src/ directory:

$ touch ~/edukit_bot_ws/src/edukit_bot/src/driver_node

The CamJam worksheets don’t discuss this, but if we set the script to be executable, we can run it directly instead of calling it like python path/to/script.py. Let’s do that:

$ chmod a+x ~/edukit_bot_ws/src/edukit_bot/src/driver_node

Open that script in a text editor, and make it look like this (note that the entire package used in this post is available for reference):

#!/usr/bin/env python

import rospy
from std_msgs.msg import String

import RPi.GPIO as GPIO

# Set the GPIO modes
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

# Set variables for the GPIO motor pins
pinMotorAForwards = 10
pinMotorABackwards = 9
pinMotorBForwards = 8
pinMotorBBackwards = 7

# How many times to turn the pin on and off each second
Frequency = 20
# How long the pin stays on each cycle, as a percent (here, it's 30%)
DutyCycle = 30
# Setting the duty cycle to 0 means the motors will not turn
Stop = 0

# Set the GPIO Pin mode to be Output
GPIO.setup(pinMotorAForwards, GPIO.OUT)
GPIO.setup(pinMotorABackwards, GPIO.OUT)
GPIO.setup(pinMotorBForwards, GPIO.OUT)
GPIO.setup(pinMotorBBackwards, GPIO.OUT)

# Set the GPIO to software PWM at 'Frequency' Hertz
pwmMotorAForwards = GPIO.PWM(pinMotorAForwards, Frequency)
pwmMotorABackwards = GPIO.PWM(pinMotorABackwards, Frequency)
pwmMotorBForwards = GPIO.PWM(pinMotorBForwards, Frequency)
pwmMotorBBackwards = GPIO.PWM(pinMotorBBackwards, Frequency)

# Start the software PWM with a duty cycle of 0 (i.e. not moving)
pwmMotorAForwards.start(Stop)
pwmMotorABackwards.start(Stop)
pwmMotorBForwards.start(Stop)
pwmMotorBBackwards.start(Stop)

# Turn all motors off
def StopMotors():
   pwmMotorAForwards.ChangeDutyCycle(Stop)
   pwmMotorABackwards.ChangeDutyCycle(Stop)
   pwmMotorBForwards.ChangeDutyCycle(Stop)
   pwmMotorBBackwards.ChangeDutyCycle(Stop)

# Turn both motors forwards
def Forwards():
   pwmMotorAForwards.ChangeDutyCycle(DutyCycle)
   pwmMotorABackwards.ChangeDutyCycle(Stop)
   pwmMotorBForwards.ChangeDutyCycle(DutyCycle)
   pwmMotorBBackwards.ChangeDutyCycle(Stop)

# Turn both motors backwards
def Backwards():
   pwmMotorAForwards.ChangeDutyCycle(Stop)
   pwmMotorABackwards.ChangeDutyCycle(DutyCycle)
   pwmMotorBForwards.ChangeDutyCycle(Stop)
   pwmMotorBBackwards.ChangeDutyCycle(DutyCycle)

# Turn left
def Left():
   pwmMotorAForwards.ChangeDutyCycle(Stop)
   pwmMotorABackwards.ChangeDutyCycle(DutyCycle)
   pwmMotorBForwards.ChangeDutyCycle(DutyCycle)
   pwmMotorBBackwards.ChangeDutyCycle(Stop)

# Turn Right
def Right():
   pwmMotorAForwards.ChangeDutyCycle(DutyCycle)
   pwmMotorABackwards.ChangeDutyCycle(Stop)
   pwmMotorBForwards.ChangeDutyCycle(Stop)
   pwmMotorBBackwards.ChangeDutyCycle(DutyCycle)

# Message handler
def CommandCallback(commandMessage):
   command = commandMessage.data
   if command == 'forwards':
       print('Moving forwards')
       Forwards()
   elif command == 'backwards':
       print('Moving backwards')
       Backwards()
   elif command == 'left':
       print('Turning left')
       Left()
   elif command == 'right':
       print('Turning right')
       Right()
   elif command == 'stop':
       print('Stopping')
       StopMotors()
   else:
       print('Unknown command, stopping instead')
       StopMotors()

rospy.init_node('driver')

rospy.Subscriber('command', String, CommandCallback)

rospy.spin()
print('Shutting down: stopping motors')
StopMotors()
GPIO.cleanup()

Lots of that should look familiar, but let’s break it down into chunks.

#!/usr/bin/env python

import rospy
from std_msgs.msg import String

The very first line of this file is called a shebang. Since we’ve marked this file as executable on its own, this defines the interpreter that will execute this program. In this case, we’re telling it that it needs the python command.

We then import rospy, which includes the ROS Python bindings, and we import the String message from the ROS std_msgs. We’ll use both of these a little later in the program.

import RPi.GPIO as GPIO

# Set the GPIO modes
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

# <snip you don't need to see all this again...>

# Turn Right
def Right():
   pwmMotorAForwards.ChangeDutyCycle(DutyCycle)
   pwmMotorABackwards.ChangeDutyCycle(Stop)
   pwmMotorBForwards.ChangeDutyCycle(Stop)
   pwmMotorBBackwards.ChangeDutyCycle(DutyCycle)

This entire section was lifted virtually verbatim from the CamJam Worksheet #7. It’s explained there, so I won’t repeat the explanation here.

# Message handler
def CommandCallback(commandMessage):
   command = commandMessage.data
   if command == 'forwards':
       print('Moving forwards')
       Forwards()
   elif command == 'backwards':
       print('Moving backwards')
       Backwards()
   elif command == 'left':
       print('Turning left')
       Left()
   elif command == 'right':
       print('Turning right')
       Right()
   elif command == 'stop':
       print('Stopping')
       StopMotors()
   else:
       print('Unknown command, stopping instead')
       StopMotors()

Here’s a new, ROS-specific part. The CommandCallback function is created to handle a String message. It simply takes a look at the data (i.e. the string itself) contained within the message, and takes action appropriately. For example, if the string is the word “forwards” it moves the robot forward by calling the Forwards function created in the worksheet. Similarly, if the word is “left” it turns the robot left by calling the Left function. If the command isn’t one of the recognized words, the function does the safest thing it can: stop.

rospy.init_node('driver')

rospy.Subscriber('command', String, CommandCallback)

rospy.spin()
print('Shutting down: stopping motors')
StopMotors()
GPIO.cleanup()

Here’s the main part of the program. First of all, we initialize the node and give it a name (“driver”). This begins communication with the ROS master. We then subscribe to a topic named “command”, and we specify that we expect that topic to be a String message. We then provide our CommandCallback function to request that it be called when a new messages comes in on that topic.

Then we call rospy.spin() which blocks and waits for messages to come in. Once the node is asked to quit (say, with a CTRL+C), that function will exit, at which time we ensure that the motors have been stopped. We don’t want the robot running away from us!

We’re done with our workspace for now, so let’s build it:

$ cd ~/edukit_bot_ws
$ catkin_make

Step 6: Move the robot using ROS

At this point, we have a ROS node created that will drive our robot as requested in a “command” message. It uses GPIO though, which still requires sudo. Instead of trying to get our workspace working using sudo, let’s temporarily change the permissions of GPIO so that we don’t need sudo (this will be reset upon reboot):

$ sudo chmod a+rw /dev/gpiomem

/dev/gpiomem is a device representing the memory dedicated to GPIO (i.e. not other, more important/dangerous memory). As a result, this operation is relatively safe, particularly compared to doing the same to e.g. /dev/mem.

Alright, let’s test this out! You’ll need to open three terminals for this, each one using the classic shell (remember, run sudo classic to enter the classic shell). We first need the ROS master, as without it publishers and subscribers can’t find one another. So in one terminal, run the ROS master:

$ roscore

In another terminal, make sure you activate our newly-built workspace, and run our “driver” node:

$ cd ~/edukit_bot_ws
$ source devel/setup.sh
$ rosrun edukit_bot driver_node

Finally, in the third terminal, we’ll start giving our commands to make the robot move. First of all, take note of the topics that either have publishers or subscribers:

$ rostopic list
/command
/rosout
/rosout_agg

Notice the /command topic. This is the topic on which our “driver” node is listening, because of the Subscriber we configured. It’s expecting a String message there, so let’s send it a command, say, to move forward:

$ rostopic pub -1 /command std_msgs/String "forwards"
publishing and latching message for 3.0 seconds

You should notice the “driver” node say that it’s moving forward, and then your robot’s wheels should start rotating forward! Try sending any of the strings we handled in CommandCallback (“left”, “backwards”, etc.), and commands that you know are invalid to ensure that it stops safely.

Congratulations, you’re quickly learning ROS! In the next post in this series, we’ll break free of the CamJam worksheets and strike out on our own. We’ll introduce the wireless controller, and begin working on making our robot remotely-controlled using ROS.

Your first robot: A beginner's guide to ROS and Ubuntu Core [1/5]

Sun, 17 Dec 2017 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

Some time ago I created a blog/video series that walked the reader through creating a prototype using the Robot Operating System (ROS) and taking it to production using Ubuntu Core. However, that series was intended more for robotics professionals; it assumed quite a bit of ROS knowledge, and required some costly equipment (the robot was about $1k). Well, Ubuntu is also for hobbyists (and kids!) who don’t want to shell out $1k to play with robots. Thus, this series was born: one that doesn’t assume any ROS knowledge, and uses hardware that’s so inexpensive you could give it as a Christmas gift. I present you with a robot that costs less than $100: The CamJam EduKit #3, which is a wheeled robot kit you assemble and control with a Raspberry Pi. Let me break down the entire list of hardware you’ll need to follow this series:

The EduKit, £18, so, say, $25
A Raspberry Pi (I’m using a Raspberry Pi 2, you need either the 2 or 3), $35
Battery for Raspberry Pi (I used this one), $15
- This is optional, but if you don’t have one your Pi will be tethered by its power supply.
Wireless controller, £14, so, say, $20
Total, including optional components: $95

In my previous robotics series, one of the reasons the Turtlebot was so expensive is because it has a plethora of software already written for it, like ROS drivers to control its movement, read from its sensors, etc. This little robot is different: we’ll need to write everything ourselves. Thankfully, the CamJam folks provide a number of worksheets meant to introduce you to the platform and get you started writing software to control the wheels, read from the ultrasonic sensor, and so on. We can leverage those to get up and running quickly, so throughout this series we’ll cover the following topics:

Your first robot: Introduction to the Robot Operating System [2/5]
- What is the Robot Operating System (ROS)? Why is it worth learning? How do I use it?
Your first robot: The controller [3/5]
- Getting data out of the wireless controller and into ROS.
Your first robot: The driver [4/5]
- Learn a little control theory, write the driver for the robot.
Your first robot: Sharing with others [5/5]
- How do you start the system without needing to SSH in? How do you share your final product with your friends?

As I mentioned, the CamJam worksheets will get us a long way toward what we need to learn for this series. However, rather than use Raspbian, we’re going to use Ubuntu Core. The worksheets are still perfectly applicable, and I’ll prove it by walking you through getting Ubuntu Core setup on your Raspberry Pi, and then we’ll go through one of the CamJam worksheets. Note that this is also a video series, feel free to watch the video version of this post:

Prerequisites

While this is meant to be an introduction, I want to be clear on some of the assumptions I’m making. This series assumes you’re at least somewhat familiar with Ubuntu (or other Debian-based distro, like Raspbian), particularly using the command-line. I don’t expect you to be a shell-scripting pro, but you should know how to navigate around the file system and use terminal-based editors (vi, nano, etc.)

What is Ubuntu Core?

Ubuntu Core is a specialized distribution of Ubuntu that is dedicated to devices that fit into the “Internet of Things” (IoT) bucket. This includes routers, thermostats, and of course, robots. It’s my hope that, in introducing you to the operating system (and tools like ROS) that the pros use for their robots, the road is paved for you to go on and accomplish great things in this field!

Ubuntu Core is a bit different from Raspbian. It’s even a bit different than classic Ubuntu (or Ubuntu MATE)! Let’s get started installing it, and I’ll explain what I mean.

Step 1: Install Ubuntu Core

Update: I no longer recommend using Ubuntu Core to learn this stuff. It was used initially here because it was the only official Ubuntu image that ran on the Pi, but that’s no longer the case. You can install Ubuntu Server or even the full-blown Ubuntu Desktop, and still follow all the snap-related steps outlined in these tutorials without needing to deal with the somewhat-foreign Ubuntu Core.

This is already well-documented, but I want to add a few notes here before you follow the steps. First of all, a keyboard and monitor are required, but only for the first boot. Unlike Raspbian or Ubuntu MATE, there is no default username/password or install process where you create a user. It doesn’t even use a mouse– this is more like Ubuntu Server. In fact, by default there’s no local login at all: it’s all done over SSH (a remote login over the network), with cryptographic keys instead of passwords. This is part of Ubuntu Core’s “secure by default” philosophy. There are numerous examples of default credentials being abused (see the Mirai botnet), and as an IoT-focused operating system, Ubuntu Core can’t have that. That’s why you’ll see an Ubuntu SSO account and SSH keys as prerequisites in the install guide (don’t worry, both are easy).

Alright, go ahead and follow the Raspberry Pi 2 or 3 install guide (make a new topic on the forum if you have any questions). At the end of this step, you should have Ubuntu Core installed, and you should be able to SSH into the Pi.

Step 2: Prepare developer setup

As an IoT-focused operating system, Ubuntu Core uses a different packaging format from Raspbian and classic Ubuntu. Rather than using Debian packaging (which involves familiar tools like apt), Ubuntu Core uses a packaging format called snaps. Snaps are far more robust when it comes to reliable updates and security, both of which are important to IoT devices, including robots. There is no apt on this system. To give you a taste of what using the snap command is like, ensure your system is completely up-to-date by running snap refresh.

So how does one install all the developer tools to which one is accustomed? There’s a special snap for that called classic, which gives you access to all your familiar tools (including apt).

Install the classic snap by following the “developing on target” section of the Developer setup guide (again, create a new forum topic if you have any questions). At the end of this step, you should be able to run sudo classic and gain access to a shell where you can install debs, which we’ll use for hacking on our robot! Later we’ll walk through creating your own snap to control the robot, which makes sharing your project with your friends super easy.

Step 3: CamJam time

Alright, we’re now to the point where we can begin following CamJam’s worksheets! Why don’t we start with the very first one. This one really assumes you’re running Raspbian, which we’re not, so while you can read the whole thing, we’re going to skip straight to the “Identifying your Version of Raspbian” step. No, we’re still not using Raspbian, but this section makes an interesting point that I want to make sure you notice.

Ubuntu Core, like Debian Wheezy, considers GPIO access via memory mapping to be a privileged operation. As a result, any of the code throughout these worksheets that uses GPIO will need to be run with sudo.

Alright, let’s get started. Make sure you’re in your classic shell by running sudo classic. vi is available here by default, but you can sudo apt install nano if you want. Following the worksheet, open up 1-helloworld.py and fill it with the following:

# Print Hello World!
print('Hello World!')

Then save and exit. You can now run the code in that file with this command:

$ python3 1-helloworld.py
Hello World!

Go ahead and read through the rest of the worksheet, it will be helpful to you later on.

I hope this gets you off to a good start using Ubuntu Core to hack on your Raspberry Pi projects. As homework for the rest of the series, also complete worksheets 2-4, and 7. You can totally do the other worksheets as well, but we won’t be using the ultrasonic sensor or line detector in this series, so you don’t have to. Before you do any other worksheets, though, you need to get the RPi.GPIO package installed. While in your classic shell, run:

$ sudo apt install gcc python3-dev python3-pip python3-setuptools
$ pip3 install RPi.GPIO

Also, remember that you’ll need to use sudo for any python scripts that utilize GPIO. The next post in this series will be an introduction to ROS, what it is, why it’s useful, why you need to know it, and how to get it on your Ubuntu-Core-powered Raspberry Pi.

Google Code-in 2017

Mon, 27 Nov 2017 00:00:00 +0000

Tomorrow, it begins. The culmination of weeks of research, documentation, and preparation by multiple open-source projects, just waiting to start hacking away with students. Tomorrow, students aged 13-17 can begin participating in Google Code-in 2017.

Ubuntu is one of the mentoring organizations, with several of my colleagues and me as mentors. Just in case you haven’t heard of it, Ubuntu is an open-source operating system for servers, desktops, and embedded devices. Even those who have never heard of it use it daily: Paypal, Netflix, Wikipedia, Pinterest, Reddit, Snapchat, Spotify– all these run on Ubuntu. It’s an awesome project, and very rewarding to work on!

One of the best ways to get involved in computer science is with open-source software (like Ubuntu). One of the best ways to get involved in open-source software is with mentorship, which is exactly what Google Code-in provides– with prizes, including a trip to Google’s headquarters! Tasks range from creating documentation (tutorials, etc.) to translations, all the way to hard-core coding tasks. There’s something there for everyone, and the product of your work will be used by millions of people!

If you fall in this age range, I know I can speak for all my colleagues when I say that we’re looking forward to working with you. If you’re local, feel free to reach out by commenting below and hitting the “notify me of other comments” checkbox and entering your email address (which will be kept private). If there’s interest, perhaps we can put a workshop together.

Nextcloud snap to be updated to v12

Mon, 20 Nov 2017 00:00:00 +0000

If you’re a user of the Nextcloud snap, you may have noticed that upstream Nextcloud released v12 a while back, but you’re still on v11. That’s because v12 introduced several regressions that made us maintainers decide to hold off on the upgrade. Thankfully, things are finally looking up.

As of today, it seems that all the blocking regressions have been fixed. Which means that, barring any additional regressions being introduced between now and then, v12.0.4 will be the release that sees the Nextcloud snap join the party.

What does this mean for you, oh treasured Nextcloud snap user? If you, like the majority of Nextcloud users, want to be on v12, it means nothing. Just wait for it to automatically update. However, what if you don’t want v12? Maybe an app you’re using hasn’t been updated yet, or you just really like v11? Then this post is for you.

The snap is broken down into tracks, of which there are currently three: 11, 12, and latest. Each track is composed of four channels, not all of which are utilized by the Nextcloud snap, but I’ll outline their usages here:

11
- edge: Daily builds of Nextcloud’s stable11 branch (i.e. what will be coming down the pipe in the next v11 point release).
- beta/candidate: Not used
- stable: The latest stable release of Nextcloud v11. Will never update to v12.
12
- edge: Daily builds of Nextcloud’s stable12 branch (i.e. what will be coming down the pipe in the next v12 point release).
- beta/candidate/stable: As of now, nothing. As soon as the snap is based on v12, the stable channel will contain the latest stable release of Nextcloud v12. Will never update to v13.
latest
- edge: Daily builds of Nextcloud’s master branch (look where development is heading)
- beta: The development branch of the snap itself
- candidate: Used for release candidates and calls for testing
- stable: The latest stable release of Nextcloud.

So by default, when you snap install nextcloud, you’re on the stable channel of the latest track (denoted as latest/stable). This means that, by default, you’ll upgrade automatically from v11 to v12 when it comes out. If you don’t want that, and would prefer to stay on v11 (including its point releases), you need to be using the stable channel of the 11 track. To switch to that channel, run this command:

$ sudo snap refresh nextcloud --channel=11/stable

As soon as you run that command, you’ll begin tracking 11/stable, and will only update when new v11 point releases come out. Then, if/when you’re ready, you can switch channels again to upgrade to v12:

$ sudo snap refresh nextcloud --channel=12/stable

Now you’ll only update when new v12 point releases come out. If you want to jump back to getting the newest stable release (including v13 when it comes out) refresh back to latest/stable:

$ sudo snap refresh nextcloud --stable

It’s important to note that snaps allow changing channels at any time, from any channel to any other channel. That includes, say, changing from 12/stable to 11/stable. While this is technically possible, Nextcloud does not support this. Downgrading is not supported and will be met with a significant lack of happiness.

Anyway, I just wanted to give you a heads up that this will be happening soon, so you can take action if necessary.

Ubuntu 17.10 Release Party

Thu, 19 Oct 2017 00:00:00 +0000

Were you busy today? Did you forget that Ubuntu 17.10 was released today (no way)? Perhaps you agree that Thursdays are lame for a party anyway? Walla Walla University’s ACM club is having their kick-off event, a joint retro video game and Ubuntu 17.10 release party, on Sunday, October 22nd from 3pm to 5pm!

This event will be taking place in room 348 of Kretschmar Hall on the Walla Walla University campus in Walla Walla, WA. The schedule looks something like this:

Introduction
Ubuntu presentation (me. You can use this time to go through the swag I’m bringing)
Social break
Retro video games/installation support/networking/general hanging out

If you happen to be in the area, I’d love to see you there!

Snap configuration: the configure hook

Tue, 12 Sep 2017 00:00:00 +0000

You’ve heard it a million times: snaps bundle their dependencies. People seem to understand and accept the technical aspects of this, but today I want to talk about a more philosophical aspect. If you’re used to more traditional packaging, then you’re used to each project being standalone, e.g. Apache is its own package, with its own configuration; PHP is its own package, with its own configuration; and so on. As you begin creating a snap, perhaps one that bundles Apache, it’s easy to automatically gravitate toward wanting to make those config files available to your users as well. That’s fine, and doing something like that is covered in an earlier post about the install hook, but I want to challenge your thinking.

Think about what you’re creating. Continuing with the Apache example, is it just an Apache snap? Meant to host any web application? If so, then yeah, it probably needs to be super flexible and exposing the full Apache config to the user may be a reasonable thing to do. However, oftentimes you’re bundling Apache together with other things to create a specific product, perhaps a web application. If so, I suggest you stop thinking of your snap as a collection of parts (Apache, PHP, MySQL), and think of it as quite simply your product. What experience do you want your user to have when interacting with it? Does your user even need to care that you use Apache? What if you want to use NGINX later? Do you want to force your user to edit the full Apache config when the only thing they want to do is change the port? Perhaps it would be better to have a slick way for the user to ask your product to simply listen on a different port. Snapd actually provides a standardized configuration interface that interacts with your snap by way of the configure hook.

As a reminder, hooks are simply executables that are shipped in the snap which snapd calls at predefined times. When is the configure hook called? In a few different situations:

When the snap is first installed, after services are started
When the snap is refreshed (updated)
Whenever snap set is called to alter the snap’s configuration

(Honestly, I think the first two situations was a mistake, but I digress.)

Bear with me, now. I want to show you what I mean with a quick tutorial.

Tutorial

In an earlier post about the install hook, we created a very simple snap containing a service that said “Hello, World!” at a rate determined by a configuration file. I’d like to build on that example here, by no longer using a configuration file at all, and using snapd to manage the configuration for us.

Let’s start with that snap. If you don’t happen to have it around, you don’t need to read the other post– just grab it off of GitHub.

Our first step toward letting snapd manage our configuration is to stop using a configuration file. That file is created in the install hook, so let’s start there. Right now it looks like this:

#!/bin/sh -e

# Create a default config file
echo "sleep_time 5" > "$SNAP_DATA/hello.conf"

Instead of creating that config file, we can ask snapd to record that configuration for us by using the snapctl command (see snapctl -h to learn more about it). Change the install hook to look like this:

#!/bin/sh -e

# Initialize the sleep time to a default
snapctl set sleep-time=5

Using snapctl in this way saves the value 5 into the sleep-time variable of this particular snap’s configuration (e.g. another snap could use snapctl set sleep-time=2 and this one would still be 5).

Now let’s change our service to use snapctl instead of the config file as well:

#!/bin/sh -e

while true; do
   # First, determine our rate by determining how long we should sleep
   sleep_time="$(snapctl get sleep-time)"

   # Now be nice and greet
   echo "Hello, World!"

   # Now sleep for the time requested
   sleep "$sleep_time"
done

At this point we could build and install this snap, and it would behave exactly the same way as in the previous post (printing its greeting every 5 seconds), but we wouldn’t be using a configuration file anymore. Of course, that also means that the user has no way to change this behavior, and here I was bragging about this “standardized configuration interface” blah blah, what about that, you ask?

Well, in the same way that snapctl set/get can be used from within the snap, snap get/set can be used from outside the snap (by the user). However, snap set can only be used once we implement a configure hook, so let’s do that.

First, create the hook and make it executable:

$ touch snap/hooks/configure
$ chmod a+x snap/hooks/configure

Now make that file look like this:

#!/bin/sh -e

# Obtain sleep-time value
sleep_time="$(snapctl get sleep-time)"

# Validate it
if ! expr "$sleep_time" : '^[0-9]*$' > /dev/null; then
   echo "\"$sleep_time\" is not a valid sleep time" >&2
   exit 1
fi

So here’s how this works. When the user calls snap set with key-value pairs, snapd calls the configure hook. If the configure hook exits successfully, snapd applies (saves) the configuration. If the configure hook exits non-zero, snapd considers the configuration bad and doesn’t save it. So all we need to do in the configure hook is ensure that the sleep time is valid, and if not, exit non-zero.

An important point before we move on: if the configure hook exits non-zero during an install or a refresh, that operation is rolled back (e.g. if a refresh, the snap is rolled back to the previously-working snap). (How does the hook know that it’s running during one of those times? It doesn’t, which is one of the reasons I think this design was a mistake and you should use the install and post-refresh hooks for that stuff, but I digress again.)

Alright, now that we have that out of the way, let’s build and install this snap (remember, --dangerous because this isn’t from the store):

$ snapcraft
Preparing to pull my-service 
Pulling my-service 
Preparing to build my-service 
Building my-service 
Staging my-service 
Priming my-service 
Snapping 'my-snap-name' |
Snapped my-snap-name_0.1_amd64.snap
$ sudo snap install my-snap-name_0.1_amd64.snap --dangerous
my-snap-name 0.1 installed

Now check the journal:

$ journalctl -fu snap.my-snap-name.hellod.service 
-- Logs begin at Mon 2017-09-11 07:51:07 PDT. --
Sep 11 13:35:40 Pandora my-snap-name.hellod[18026]: Hello, World!
Sep 11 13:35:45 Pandora my-snap-name.hellod[18026]: Hello, World!

As expected, we see a greeting every 5 seconds. We can verify that configuration setting using snap get:

$ snap get my-snap-name sleep-time
5

There we see the value that we set in the install hook. We can change it using snap set:

$ snap set my-snap-name sleep-time=1
$ snap get my-snap-name sleep-time
1

You can see that the new value applied. Let’s check the journal:

$ journalctl -fu snap.my-snap-name.hellod.service 
-- Logs begin at Mon 2017-09-11 07:51:07 PDT. --
Sep 11 13:40:17 Pandora my-snap-name.hellod[18026]: Hello, World!
Sep 11 13:40:18 Pandora my-snap-name.hellod[18026]: Hello, World!

There we go, a greeting once a second. Now let’s see what happens if we try to set an invalid value:

$ snap set my-snap-name sleep-time=foo
error: cannot perform the following tasks:
- Run configure hook of "my-snap-name" snap (run hook "configure": "foo" is not a valid
 sleep time)
$ snap get my-snap-name sleep-time
1

As you can see, the invalid value was caught by our configure script, and it was not applied.

Conclusion

I hope you can see how this might apply to your snap, and how giving your product a consistent interface for configuration might lead to a better overall user experience than requiring them to edit configuration files. It’s also easier for you to validate!

Snap install-time setup: the install hook

Mon, 11 Sep 2017 00:00:00 +0000

When it comes to developing snaps, there’s a particular confusion out there that I see over and over again: build-time versus run-time. For example: “I’m building a snap, but I can’t seem to convince Snapcraft to place my config file in $SNAP_DATA.” In this post, I want to show you how to get the results you want.

First of all, we need to clear something up. Snapcraft can’t actually do this for you. Why? Because Snapcraft is responsible only for building the my-cool-thing_v1.0.snap. Once that snap is built, you can install it. Before the snap is installed/running (e.g. while you’re building the snap), $SNAP, $SNAP_DATA, etc. do not exist. They are defined by snapd when the snap is installed.

To put it another way, the snap created by Snapcraft is completely immutable and self-contained. The snap does not contain $SNAP_DATA, and has no concept of $SNAP_DATA until it is actually installed. Which means Snapcraft doesn’t have that concept either, and can’t help you.

However, as of v2.27, snapd does have the ability to help you, by way of the install hook. Generically, hooks are the way snapd tells a snap that something interesting has happened, and gives it a chance to do something about it. At its most basic, a hook is an executable contained in the snap that snapd calls when it feels like it should. In the case of the install hook, it’s letting the snap know that it’s currently being installed, and provides the opportunity for the snap to do something as part of that process, before services are fired up, and potentially bail out of the entire install process if required.

To be clear: the install hook runs only upon initial install, not a refresh or remove (there are other hooks for those situations).

Let me explain with a tutorial.

Tutorial

The problem

First of all, let’s get started by creating a brand new snap that suffers from a pretty typical problem: a service that requires a config file, but that config file needs to be in a writable place (e.g. $SNAP_DATA). Once we have that, we’ll discuss how the install hook solves that problem.

So, get into the directory that will contain our snap, and run:

$ snapcraft init
Created snap/snapcraft.yaml.
Edit the file to your liking or run `snapcraft` to get started

Now, let’s create our service. We’ll keep it simple: this service simply says “Hello, World!” over and over again, at a rate determined by a config file. First, create the service executable itself, and make sure it’s executable:

$ mkdir -p src/bin
$ touch src/bin/hellod
$ chmod a+x src/bin/hellod

Now make that file look like this:

#!/bin/sh -e

config_file="$SNAP_DATA/hello.conf"

while true; do
   # First, determine our rate by determining how long we should sleep
   sleep_time="$(awk '/^sleep_time/{print $2}' "$config_file")"

   # Now be nice and greet
   echo "Hello, World!"

   # Now sleep for the time specified in the config file
   sleep "$sleep_time"
done

Update your snap/snapcraft.yaml to include this service by adding both a part to install it as well as an app to expose it as a service. While we’re at it, let’s change the confinement type to strict instead of devmode. Make it look something like this (most of it is still the template):

name: my-snap-name # you probably want to 'snapcraft register <name>'
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
 This is my-snap's description. You have a paragraph or two to tell the
 most important story about your snap. Keep it under 100 words though,
 we live in tweetspace and your description wants to look good in the snap
 store. 

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: strict

parts:
 my-service:
   plugin: dump
   source: src/

apps:
 hellod:
   command: hellod
   daemon: simple

Let’s go ahead and build this snap, and then install it (--dangerous because it’s not from the store):

$ snapcraft
Preparing to pull my-service 
Pulling my-service 
Preparing to build my-service 
Building my-service 
Staging my-service 
Priming my-service 
Snapping 'my-snap-name' |
Snapped my-snap-name_0.1_amd64.snap
$ sudo snap install my-snap-name_0.1_amd64.snap --dangerous
my-snap-name 0.1 installed

Now as you probably know, as soon as you install the snap, its services are fired up. Let’s take a look at the output from our service:

$ journalctl -u snap.my-snap-name.hellod.service 
-- Logs begin at Mon 2017-09-11 07:51:07 PDT, end at Mon 2017-09-11 10:26:03 PDT. --
Sep 11 10:23:18 Pandora systemd[1]: Started Service for snap application my-snap-name.hellod.
Sep 11 10:23:19 Pandora my-snap-name.hellod[8527]: awk: fatal: cannot open file `/var/snap/my-snap-name/x1/hello.conf' for reading (No such file or directory)
Sep 11 10:23:19 Pandora systemd[1]: snap.my-snap-name.hellod.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 11 10:23:19 Pandora systemd[1]: snap.my-snap-name.hellod.service: Unit entered failed state.

Ah, well darn, our service is failing because it can’t find the config file we didn’t put there– how unexpected!

Sarcasm aside, while this example is contrived, this is a very common problem. Oftentimes config files need to be writable by the user, which means they simply cannot be held within the snap itself, which is by definition immutable. However, one is left with the problem of needing to get a config file in a writable area before the service that needs it is started, exactly like this example. Before snapd v2.27, the only solution was to wrap one’s service in a shell script that checked for the config file and created it if it wasn’t there. This was tedius and annoying, but no more! Let’s solve this the Right Way™ by creating an install hook. First though, let’s remove the snap we already have installed:

$ sudo snap remove my-snap-name

The solution

Let’s start by creating a new directory to hold our hooks:

$ mkdir snap/hooks

Now let’s create the install hook, and make sure it’s executable:

$ touch snap/hooks/install
$ chmod a+x snap/hooks/install

Now we’ll make that file look like this:

#!/bin/sh -e

# Create a default config file
echo "sleep_time 5" > "$SNAP_DATA/hello.conf"

Believe it or not, that’s it. Snapcraft knows this is a hook because you put it in the snap/hooks/ dir and it’s executable, and it’ll place it in the right spot in the snap. Snapd knows it’s the install hook because the file name is “install”. Note that the code for this snap is available on GitHub for reference.

NOTE: Hooks run confined, just like apps. This particular hook requires no special permission, so nothing extra was required for it to work. However, let’s say our hook required access to the network. In that case, it would need to use the network plug the same way as an app would need to. The way to specify this requirement is to add a section to your YAML that looks like this:

hooks:
 install:
   plugs: [network]

Again, that’s not required here, but keep it in mind.

Alright back to it: let’s build the snap from scratch again, and install it once more:

$ snapcraft clean
Cleaning up priming area
Cleaning up staging area
Cleaning up parts directory
$ snapcraft
Preparing to pull my-service 
Pulling my-service 
Preparing to build my-service 
Building my-service 
Staging my-service 
Priming my-service 
Snapping 'my-snap-name' |
Snapped my-snap-name_0.1_amd64.snap
$ sudo snap install my-snap-name_0.1_amd64.snap --dangerous
my-snap-name 0.1 installed

Now take a look at the service journal, and you’ll see it working:

$ journalctl -fu snap.my-snap-name.hellod.service 
-- Logs begin at Mon 2017-09-11 07:51:07 PDT. --
Sep 11 10:59:54 Pandora my-snap-name.hellod[11625]: Hello, World!
Sep 11 10:59:59 Pandora my-snap-name.hellod[11625]: Hello, World!
Sep 11 11:00:04 Pandora my-snap-name.hellod[11625]: Hello, World!

How is this working? Well, now that you provided an install hook, snapd ran it as part of the installation. You can see specifically when by taking a look at the change representing the snap installation. First, we need to determine the change ID:

$ snap changes
<snip>
1138 Done <snip> Install "my-snap-name" snap from file "my-snap-name_0.1_amd64.snap"

Now take a look at that particular change:

$ snap change 1138
Status <snip> Summary
Done           Prepare snap "/tmp/snapd-sideload-pkg-840816098" (unset)
Done           Mount snap "my-snap-name" (unset)
Done           Copy snap "my-snap-name" data
Done           Setup snap "my-snap-name" (unset) security profiles
Done           Make snap "my-snap-name" (unset) available to the system
Done           Setup snap "my-snap-name" (unset) security profiles (phase 2)
Done           Set automatic aliases for snap "my-snap-name"
Done           Setup snap "my-snap-name" aliases
Done           Run install hook of "my-snap-name" snap if present
Done           Start snap "my-snap-name" (unset) services
Done           Run configure hook of "my-snap-name" snap if present

Note in particular where the ‘Run install hook of “my-snap-name” snap if present’ is happening: right before services are started. Which gives it a chance to set things up so that services have what they require before they try to run. If the install hook runs into a problem, it can exit non-zero and cause the entire installation to abort.

After installation, one can update that config file to shorten or lengthen the sleep time as required. While that may not be particularly interesting for this snap, imagine if this was an Apache configuration file, or MySQL. Depending on the snap, it may be quite useful to expose the configuration this way, and of course can be used for other things as well (e.g. plugins, assets, generate an initial database, etc.) In a follow-up post, we’ll discuss another way to manage configurations, by way of the configure hook.

Integrating your Rails project with GitLab review apps

Mon, 24 Jul 2017 00:00:00 +0000

My first foray into software development when I was a kid was as a web developer. I don’t admit this to many people, but the first language I ever learned halfway decently was PHP. I led a confused childhood. After that, though, I was introduced to Ruby on Rails, and I’ve been using it ever since.

I don’t do web development professionally anymore, but I have a number of web-based side projects, making websites for friends or family. Most of the time, the people I work with are the brains of the operation, and I’m the brawn: they decide what they want, and I make it for them. Also, most of the time the people I’m working with are non-technical. So as I make progress on their project and want to obtain feedback, I’m rarely looking for feedback on my code (that’s always perfect anyway, of course). I want them to play around with the site itself.

There are some obvious ways to do that. We could meet in person and they could play with it on my laptop, or they could VPN in and access my local server remotely, or I could deploy to a staging server, and so on. These are fine solutions, but they’re synchronous and don’t scale well. Perhaps more importantly, they get in my way: they can only review essentially one thing at a time.

These projects are all using Rails 5, and are hosted on a private GitLab instance. While doing some research into how I might be able to move a bit faster, I read about GitLab’s Review Apps, and realized I’d hit a gold mine. This post documents how I got it working for my Rails projects in the hope that it will be useful to you. Review Apps

I was already using Gitlab CI to test my rails project, as well as deploy it to staging and production. Deploying requires the use of environments, and once you have those figured out review apps build naturally on top of them. Deploying is pretty cool: once the deploy finishes successfully, GitLab shows the URL you specified in the environment as a way to view the deployment, which makes things pretty seamless. Well, review apps are just like that, but the environment is actually dynamic. The environment name is dynamic, and the environment url is dynamic. These are typically based on the branch name. It means you get a separate deployment per branch, which means if you structure your project correctly, you can get nicely isolated deployments for every merge request you make, even at the same time.

Prerequisites

Review apps aren’t magical. You need the infrastructure to support them.

Ability to handle unique, branch-based URLs

Review apps will just assume that a URL it generates by combining the branch name with the base domain you specify actually points to the app in the branch. We’ll get into the technical details of this in a bit, but in my case, my URL looks like this:

url: http://$CI_COMMIT_REF_SLUG.example.com

So if I push up a branch named support_review_apps that will turn into http://support-review-apps.example.com. Review apps does nothing to ensure that exists, it just assumes it does. In my case, that means I needed to have a wildcard DNS entry for *.example.com. Since my registrar is 1&1 (who doesn’t support wildcard DNS entries), I started using Namecheap for my DNS. It’s pretty awesome. If you’re unfamiliar, you essentially create a *.example.com subdomain pointing to a given IP. Then foo.example.com, which doesn’t have an entry of its own, will resolve to that same IP. So will bar.example.com, you get the idea. Why is this useful? Because then we can use it in our web server to make sure that traffic ends up in the right place. We’ll get to that in a minute.

Somewhere you can dynamically deploy

This might look like docker to you. Or a kubernetes cluster. For me, it’s a single LXD container configured for my project (the right version of Ruby, etc.) and using bundler in --deployment mode to put all the gems in a single tree instead of installing them on the system. Then I can install different branches of the same project in different directories using capistrano, which we’ll cover in a minute.

Get on with it

Configuring the project

So we need to deploy the same project multiple times to the same infrastructure. This is a perfect use-case for docker, but that doesn’t run well within LXD (update: it does nowadays if you enable nesting), so I was limited to using a single container. Since I wanted to keep each instance isolated, and also because they didn’t need to be particularly performant, I didn’t want to have to deal with a shared, server-based database, with different credentials depending on instance, etc. In other words, I decided that the Rails environment I wanted to use was not going to be production, where I use PostgreSQL. So I created a new environment called “review” in config/environments/. It’s really just a copy of the “production” environment, but it allows us to specify a database for that environment in config/database.yml that looks like this:

review:
 <<: *default
 database: db/review.sqlite3

Now our database is specific to that instance. We’ll also need a secret key base for that environment in config/secrets.yml:

review:
 secret_key_base: 9c9cdd6<snip>dd5667b

Of course for production this should be in the environment rather than being committed, but I’m okay with this being hard-coded for review.

Configuring the web server

So we have a dynamic DNS pointing to an LXD container nearly setup for dynamic deployment. We still need to answer the question: “How do we ensure that traffic to a branch-based URL ends up on that branch’s deployed site?”

Since all deployments for review apps on this project were going to the same container, I configured nginx to be smart about it, and grab the name from the URL in order to figure out where the traffic should go. That configuration looked like this:

server {
   listen 80;
   server_name ~^(www\.)?(?<sname>.+?).example.com$;
   root /var/www/$sname/current/public;

   location / {
       proxy_pass http://unix:/var/www/$sname/shared/tmp/sockets/puma.sock;
       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   }

   location ~* ^/assets/ {
       expires 1y;
       add_header Cache-Control public;
       add_header Last-Modified "";
       add_header ETag "";
       break;
   }

   access_log /var/log/nginx/$sname-access.log;
   error_log /var/log/nginx/reviews-error.log debug;
}

Since *.example.com was already hitting nginx, this made it so visiting foo.example.com tried to proxy to a puma socket in /var/www/foo/shared/tmp/sockets/puma.sock. Of course, if that didn’t exist nginx told you so. Now we just needed a deployment method that actually deployed sites in sub-directories of /var/www/ based on branch name.

Configuring capistrano to deploy the way we need

All my apps are using puma. So to deploy with capistrano, I add the following to the development group of my Gemfile:

gem 'capistrano'
gem 'capistrano-rails'
gem 'capistrano3-puma'

I don’t intend to make this post a capistrano tutorial, there are already several out there. However, assuming you already have it set up, there are three things we need to change from the typical capistrano behavior.

First of all, since we want to use the new “review” environment, make sure you create an environment-specific deploy script in config/deploy/review.rb (just copy the “production” one and modify as necessary). Since I’m using an sqlite database, I want to make sure it’s shared across revisions of the deployment, so in that file I placed code to do exactly that:

append :linked_files, "db/review.sqlite3"

namespace :deploy do
   namespace :check do
       # The database needs to exist before it's symlinked in or the symlink
       # will fail.
       before :linked_files, :create_db do
           on primary :db do
               execute "touch #{shared_path}/db/review.sqlite3"
           end
       end
   end
end

Second, by default, capistrano deploys to /var/www/<application>. capistrano3-puma also uses the application name to determine where to place puma’s socket. So if we can support changing the application name on the fly, we can support the branch-specific nesting that our nginx configuration will be expecting. We could use rake parameters for this, but I liked environment variables better, so in config/deploy.rb I set the application like this:

set :application, ENV.fetch('application', 'example.com')

So it defaults to the production application name, but now I can call cap review deploy application=foo and it’ll deploy to /var/www/foo/, which is exactly what I want.

Third, by default capistrano deploys the master branch. With review apps we want to deploy a specific branch, so that’s another line in config/deploy.rb like this:

set :branch, ENV.fetch('branch', 'master')

Still defaulting to master, but now we can call cap review deploy branch=my-branch-name and it’ll deploy the my-branch-name branch instead of master.

Finally, what we’ve discussed so far has a “leak” issue. Every branch that’s pushed will be deployed onto this same container. That deployment stays there, even after the branch has been merged and deleted! Eventually we’ll run out of hard drive space. Thankfully, review apps support stopping once the branch is deleted, we just need to support such a thing. I added a simple undeploy task to config/deploy.rb that looks like this:

namespace :deploy do
   task :undeploy do
       on roles(:app) do
           invoke 'puma:stop'
           execute "rm -rf #{fetch(:deploy_to)}"
       end
   end
end

So then stopping and removing a deployment is as simple as calling cap review deploy:undeploy.

Putting all of this together and using review apps

We now have a Rails project with a “review” environment, a capistrano configuration for deploying that environment, and the basic infrastructure necessary to host the deployments. Now we just need to flip the switch to enable GitLab Review Apps.

That of course goes into the .gitlab-ci.yml file. You probably already had a deploy stage, you just need to add jobs for deploying and stopping the review apps, which looks like this:

deploy_review:
 stage: deploy
 tags: [ruby-2.4.1]

 script:
   - eval "$(ssh-agent -s)"
   - ssh-add <(echo -e "$SSH_PRIVATE_KEY")
   - bundle exec cap review deploy branch=$CI_COMMIT_REF_NAME application=$CI_COMMIT_REF_SLUG

 environment:
   name: review/$CI_COMMIT_REF_NAME
   url: http://$CI_COMMIT_REF_SLUG.example.com
   on_stop: stop_review

 only:
   - branches
 except:
   - master

stop_review:
 stage: deploy
 tags: [ruby-2.4.1]
 variables:
   GIT_STRATEGY: none

 script:
   - eval "$(ssh-agent -s)"
   - ssh-add <(echo -e "$SSH_PRIVATE_KEY")
   - bundle exec cap review deploy:undeploy application=$CI_COMMIT_REF_SLUG

 environment:
   name: review/$CI_COMMIT_REF_NAME
   action: stop
 when: manual

Let’s walk through the deployment job first, piece by piece.

deploy_review:
 stage: deploy
 tags: [ruby-2.4.1]

This defines the job, putting it into the deploy stage (which in my pipeline comes after test). It needs a runner with Ruby v2.4.1 on it, so I tag it as such.

 script:
   - eval "$(ssh-agent -s)"
   - ssh-add <(echo -e "$SSH_PRIVATE_KEY")
   - bundle exec cap review deploy branch=$CI_COMMIT_REF_NAME application=$CI_COMMIT_REF_SLUG

This is where the rubber meets the road. Capistrano works over SSH, so I have a private key saved as a variable in GitLab that I load into ssh-agent and then deploy using the $branch and $application environment variables we discussed earlier. I use $CI_COMMIT_REF_SLUG for the application name to ensure that it’s safe for use in a URL.

 environment:
   name: review/$CI_COMMIT_REF_NAME
   url: http://$CI_COMMIT_REF_SLUG.example.com
   on_stop: stop_review

Here’s the dynamic environment. Note how the name and url both use the branch name (although the url uses the slug to ensure it’s safe for use in a URL). This is the URL that will be shown on a merge request to view that deployment. Finally, we specify the job that needs to run in order to stop this deployment, stop_review. GitLab will run this job automatically once the branch is deleted. Let’s take a look at that job now, piece by piece.

stop_review:
 stage: deploy
 tags: [ruby-2.4.1]
 variables:
   GIT_STRATEGY: none

Here again we name the job, and specify that it requires a runner with Ruby v2.4.1. We also specify GIT_STRATEGY: none so the runner doesn’t try to check out code that was deleted. This is a little odd to me since I’m still able to use capistrano here, so I wonder if I’m taking advantage of the fact that this ends up running on the same runner and the code is already there. I need to investigate that further, but this works for now.

 script:
   - eval "$(ssh-agent -s)"
   - ssh-add <(echo -e "$SSH_PRIVATE_KEY")
   - bundle exec cap review deploy:undeploy application=$CI_COMMIT_REF_SLUG

As before, we load the SSH key into ssh-agent, and then execute our undeploy job with the same application variable to make sure we blow away the right one.

 environment:
   name: review/$CI_COMMIT_REF_NAME
   action: stop
 when: manual

Note that this environment name corresponds to the deploy_review job as well, and the action is stop, which tells GitLab this is how it stops that environment. We specify that this is a manual action so it doesn’t happen when we push, but it’s still automatic since we specified this job name in the on_stop of the initial deployment.

Conclusion

It took me a while to distill the documentation down into those steps, but once you’ve gone through the process you realize that it makes sense and it’s easy to do again. This has drastically sped up my development time as reviews can now happen asynchronously and simultaneously. I really can’t recommend it enough, and I hope this post helps you accomplish the same thing!

Ubuntu Core: Making a factory image with private snaps

Thu, 06 Jul 2017 00:00:00 +0000

This is a follow-up to the ROS prototype to production on Ubuntu Core series to answer a question I received: “What if I want to make an image for the factory, but don’t want to make my snaps public?” This question is of course not robotics-specific, and neither is its answer. In this post we’ll cover two ways to do this.

Before we start, you’ll need a little bit of an Ubuntu Core imaging background. If you followed the ROS prototype to production series (part 5 specifically) you already have the required background, but if you didn’t, check out the docs for creating your own Ubuntu Core image.

Assuming you’re up-to-speed and know what I’m talking about when I say “model definition” or “model assertion,” let’s get started on a few different methods for creating an Ubuntu Core image with private snaps.

Method 1: Don’t put your snap in the store at all

It really doesn’t get simpler. Take a look at this example model definition, amd64-model.json:

{
 "type": "model",
 "series": "16",
 "model": "custom-amd64",
 "architecture": "amd64",
 "gadget": "pc",
 "kernel": "pc-kernel",
 "authority-id": "4tSgWHfAL1vm9l8mSiutBDKnnSQBv0c8",
 "brand-id": "4tSgWHfAL1vm9l8mSiutBDKnnSQBv0c8",
 "timestamp": "2017-06-23T21:03:24+00:00",
 "required-snaps": ["kyrofa-test-snap"]
}

Let’s go ahead and turn that into a model assertion:

$ cat amd64-model.json | snap sign -k my-key-name > amd64.model
You need a passphrase to unlock the secret key for
user: "my-key-name"
4096-bit RSA key, ID 0B79B865, created 2016-01-01
...

Now you have your model assertion: amd64.model. If you hand that to ubuntu-image right now you’ll run into a problem:

$ sudo ubuntu-image -c stable amd64.model 
Fetching core
Fetching pc-kernel
Fetching pc
Fetching kyrofa-test-snap
error: cannot find snap "kyrofa-test-snap": snap not found
COMMAND FAILED: snap prepare-image --channel=stable amd64.model /tmp/tmp6p453gk9/unpack

The snap with the name kyrofa-test-snap isn’t actually in the store. But that’s important to note: the model definition (and thus assertion) only contains a list of snap names. If you have a snap locally with that name, even if it’s not in the store, you can tell ubuntu-image to use it to satisfy that name in the assertion with the --extra-snaps option:

$ sudo ubuntu-image -c stable \
                   --extra-snaps /path/to/kyrofa-test-snap_0.1_amd64.snap \
                   amd64.model
Fetching core
Fetching pc-kernel
Fetching pc
Copying "/path/to/kyrofa-test-snap_0.1_amd64.snap" (kyrofa-test-snap)
kyrofa-test-snap already prepared, skipping
WARNING: "kyrofa-test-snap" were installed from local snaps disconnected from
a store and cannot be refreshed subsequently!
Partition size/offset need to be a multiple of sector size (512).
The size/offset will be rounded up to the nearest sector.

There. You now have an Ubuntu Core image (named pc.img) with your snap preinstalled, without the snap ever needing to be in the store. This works, but it has a big disadvantage which ubuntu-image points out with a warning: preinstalling a snap that isn’t connected to the store means you have no way to update it once devices are flashed with this image. Your only update mechanism would be to ship new images to be flashed.

Method 2: Use a brand store

When you create a store account and visit dashboard.snapcraft.io, you’re viewing your snaps in the standard Ubuntu store. If you install snapd fresh on your system, this is the store it uses by default. While you can release snaps privately on the Ubuntu store, you can’t preinstall those in an image because only you (and the collaborators you’ve added) can obtain access to it. The only way you can make an image in this case would be to make the snaps publicly available, which defeats the whole purpose of this post.

For this use-case, we have what are called brand stores. Brand stores are still hosted in the Ubuntu store, but they’re a custom, curated version of it, meant to be specific to a given company or device. They can (optionally) inherit from the standard Ubuntu store, and be open to all developers or locked down to a specific group (which is what we want in our case, to keep things private).

Note that this is a paid feature. You need to request a brand store by contacting Canonical. Once your request has been granted, you’ll see your new store by visiting “stores you can access” under your name.

There you’ll see the various stores to which you have access. You’ll have at least two: the normal Ubuntu store, and your new brand store. Select the brand store (red rectangle). While you’re here, record your store ID (blue rectangle): you’ll need it in a moment.

From there, registering names/uploading snaps works the same way, but now they go into your brand store instead of the standard one, and assuming you have it unlisted, those snaps are not available to external users. The only caveat today is that at least the first upload for the snap needs to be via the web interface. After that, you can continue to use Snapcraft like normal.

So how does this change things? My “kyrofa-store” inherits snaps from the Ubuntu store, and also contains a “kyrofa-branded-test-snap” published into the stable channel. This snap isn’t available in the Ubuntu store, as you can see if you search for it:

$ snap find kyrofa-branded
The search "kyrofa-branded" returned 0 snaps

But using the store ID we recorded earlier, we can make a model assertion that pulls from the brand store instead of the Ubuntu store. We just need to add the “store” key to the JSON document, making it look like this:

{
 "type": "model",
 "series": "16",
 "model": "custom-amd64",
 "architecture": "amd64",
 "gadget": "pc",
 "kernel": "pc-kernel",
 "authority-id": "4tSgWHfAL1vm9l8mSiutBDKnnSQBv0c8",
 "brand-id": "4tSgWHfAL1vm9l8mSiutBDKnnSQBv0c8",
 "timestamp": "2017-06-23T21:03:24+00:00",
 "required-snaps": ["kyrofa-branded-test-snap"],
 "store": "ky<secret>ek"
}

Sign it just as we did in Method 1, and we can create an Ubuntu Core image with our private, brand-store snap preinstalled as simply as:

$ sudo ubuntu-image -c stable amd64.model
Fetching core
Fetching pc-kernel
Fetching pc
Fetching kyrofa-branded-test-snap
Partition size/offset need to be a multiple of sector size (512).
The size/offset will be rounded up to the nearest sector.

Now, like at the end of Method 1, you have a pc.img ready for the factory. However, with this method, the snaps in the image are all coming from the store, which means they will automatically update as usual.

Conclusion

These are the only two options for doing this today. When I started writing this post I thought there was a third (keeping one’s snap private and creating an image with it), but that turns out to not be the case.

Note that we’ve also received numerous requests for some sort of on-premises/enterprise store, and while such a product is not yet available, the store team is working on it. Once this is available, I’ll write a new post about it.

I hope this proves useful!

How to use Snapcraft v2 on Trusty

Thu, 08 Jun 2017 00:00:00 +0000

As of this writing there are two major versions of Snapcraft: v1 and v2. Snapcraft v1 creates snaps that target Ubuntu Core 15, and Snapcraft v2 creates snaps that target Ubuntu Core 16. Series 16 snaps are much more widely used, being supported not only in Ubuntu Core 16 but also classic Ubuntu Xenial (desktop and server) and more recently even Trusty! It’s also supported across a range of other Linux distributions. However, while series 16 snaps can run in a number of places, currently Snapcraft v2 (which makes series 16 snaps) only runs in Xenial. The Trusty archives contains Snapcraft v1. For the vast majority of cases this works fine: build your snaps on Xenial, run them anywhere. However, there are cases when you really need to run Snapcraft on Trusty. Case in point: ROS (Robot Operating System) Indigo.

ROS maintains a number of releases at a time. The ones most heavily used tend to be the LTS releases: that is, the releases that correspond to the LTS releases of Ubuntu. There are two LTS versions of Ubuntu supported right now, Trusty and Xenial. The corresponding versions of ROS are Indigo and Kinetic, respectively.

These are the most popular releases because switching one’s entire project to another release of ROS is not an easy task. In my experience, most teams tend to stick with a single release until it’s either no longer supported or new versions introduce features they need. Indeed, this is backed up by the metrics presented at ROSCon last year: over 70% of ROS users are still using Indigo, and about 10% are using Kinetic.

This (among other things) has made making Snapcraft v2 available on Trusty a priority for us. However, due to the SRU process (i.e. the process of updating software in already-released versions of Ubuntu), it’s not as simple as just releasing an update upgrading the Snapcraft v1 already in the Trusty archives to v2. There are a lot of dependencies that are Xenial-only. The perfect solution is to start shipping Snapcraft as a snap, which we’re doing, but it’s still early days there. When running the Snapcraft snap on Trusty, it’s currently only intended to serve as an entry point into cleanbuild, which uses a Xenial container to build a snap. That won’t work for Indigo: we need to build on Trusty, and use Trusty’s archives for dependencies. We’re still working out the kinks to support exactly that, but until then, I know some folks who are blocked by this. If you’re part of that group, then this post is for you: I’ll walk you through running Snapcraft v2 from source on Trusty.

Update: While this post may still be useful, it’s mostly historical. As of Snapcraft v2.35, the Snapcraft snap runs on Trusty. That’s not yet released as of the time of this writing, but you can install it right now from the edge channel:

$ sudo snap install --classic --edge snapcraft

Alright, back to your regularly-scheduled programming.

Prerequisites

Python 3.5

Snapcraft uses circular imports, which means it requires at least Python v3.5. Trusty’s archives have v3.4. In the Ruby community we run into this problem all the time, and have tools like rvm to solve it. Python has a similar tool: pyenv. We’ll use it to get Python v3.5. First, you need git:

$ sudo apt install git

Now you can use the pyenv installer to get up and running (it places pyenv into ~/.pyenv):

$ curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash

When that has finished, you’ll see it recommending you place a few lines into ~/.bash_profile. I just placed those lines at the end of my ~/.bashrc:

export PATH="/home/ubuntu/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Now either start a new terminal session or simply re-source it, and verify that it works by running an update (which should do nothing other than say stuff is up-to-date):

$ source ~/.bashrc
$ pyenv update

pyenv will build Python, for which you’ll need a few dependencies:

$ sudo apt install gcc make libssl-dev libbz2-dev libreadline-dev liblzma-dev libsqlite3-dev

Now we’re ready to build Python v3.5, but we have a small issue. pyenv will build upstream Python, which doesn’t actually detect Ubuntu correctly. Snapcraft does different things depending on the OS, so Ubuntu needs to be detected correctly. Fortunately there’s a distro patch for that, so let’s just use it. You’ll need a few tools:

$ sudo apt install bzr patchutils

Now fetch the Ubuntu source for Python v3.5:

$ bzr branch lp:~doko/python/pkg3.5-debian

Alright, now let’s get Python v3.5 (specifically v3.5.2, which is what’s currently in Xenial), and tell pyenv to apply that distro patch before building:

$ cat pkg3.5-debian/patches/platform-lsbrelease.diff | filterdiff --strip=1 | pyenv install -p 3.5.2
Downloading Python-3.5.2.tar.xz...
-> https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tar.xz
Installing Python-3.5.2...
patching file Lib/platform.py
Hunk #1 succeeded at 268 (offset 2 lines).
Hunk #2 succeeded at 297 (offset 2 lines).
Hunk #3 succeeded at 333 (offset 1 line).
patching file Lib/test/test_platform.py
Hunk #1 succeeded at 320 (offset 16 lines).
Installed Python-3.5.2 to /home/ubuntu/.pyenv/versions/3.5.2

Now you can see that you have multiple Python versions available (with the current one represented by the asterisk):

$ pyenv versions
* system (set by /home/ubuntu/.pyenv/version)
 3.5.2

Let’s go ahead and change the active one to 3.5.2:

$ pyenv global 3.5.2
$ pyenv versions
 system
* 3.5.2 (set by /home/ubuntu/.pyenv/version)
$ python3 --version
Python 3.5.2

Now we’re using v3.5.2. You can set it back later by using pyenv global system if you like.

libsodium

Snapcraft depends on libsodium for communication with the store, but it’s only in the Xenial archives, not Trusty’s. Thankfully there’s a PPA that provides it:

$ sudo add-apt-repository ppa:chris-lea/libsodium
$ sudo apt update
$ sudo apt install libsodium-dev

Install Snapcraft

We’re going to install Snapcraft via pip, as outlined in its hacking guide. Doing that on Trusty requires a few tweaks, so use my special trusty-support branch:

$ git clone -b trusty-support https://github.com/kyrofa/snapcraft
Cloning into 'snapcraft'...
remote: Counting objects: 18235, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 18235 (delta 9), reused 13 (delta 5), pack-reused 18204
Receiving objects: 100% (18235/18235), 6.84 MiB | 6.46 MiB/s, done.
Resolving deltas: 100% (12342/12342), done.
Checking connectivity... done.

Now install its dependencies (taken from the hacking guide):

$ sudo apt install g++ libffi-dev libapt-pkg-dev libarchive13

As far as how/where to actually install Snapcraft, that’s up to you. Personally, I like to use virtualenv so I don’t clutter my system. Pretty much exactly as outlined in the hacking guide (guess who wrote that bit):

$ mkdir ~/venv
$ python3 -m venv ~/venv/snapcraft
$ source ~/venv/snapcraft/bin/activate
(snapcraft) $ pip install --upgrade pip
(snapcraft) $ pip install -r requirements.txt .

As long as you’re within that virtual environment (by sourcing its activate script) you’ll have Snapcraft v2 available to you! Now you can go build your snaps like normal, but you’re running on Trusty now, using the Trusty archives for stage-packages, etc. I’ve tested this specifically for ROS Indigo, but of course other things should work as well unless there’s a hard dependency on Xenial. I hope this is helpful until the snap fully supports Trusty!

Continuous acceptance tests for complex applications

Mon, 05 Jun 2017 00:00:00 +0000

So you’re developing a complex application. Maybe it has some really specific dependencies, or requires a lot of setup. In many cases (such as my own), it’s a web application. I have a suite of tests, varying all the way from unit tests through integration tests. The latter typically uses Selenium, and I often integrate it with Sauce Labs. I’ve written an article about this before. However, even those integration tests aren’t testing the real application: it’s not running against a production database, it’s not running with a production web server, and so on. Doing this is not easy; oftentimes the solutions I see look like this:

Create custom docker images with the rest of the system (dependencies, database, web server, etc.) setup correctly
Inject the code into it
Run tests against it

Not only this, but one needs to ensure one’s image stays in line with one’s actual dependencies and continues to mirror a production deployment.

There’s a better way: package your application as a snap.

I’ve written about the Nextcloud snap a number of times. It’s one of the more complex snaps out there, bundling Apache, MySQL, PHP-FPM, etc. It’s a production-ready snap, following Nextcloud’s recommended settings. This makes it a perfect testing target! Except… there are no hosted CI options that can actually run snaps. All the ones I tried didn’t have a kernel configured properly for snaps (e.g. Travis). I was starting to investigate integrating my private GitLab instance (with its own amazing CI) with GitHub just to run tests against the snap.

Then I heard about Circle CI. By default they run docker on Ubuntu Trusty, but they have actual VMs available as a beta (the machine executor), which have a Xenial kernel and can actually run snaps! It’s a bit limited, as I’ll discuss, but it solves a real need at least for the Nextcloud snap, and I’d like to walk you through doing the same thing.

Backstory

The Nextcloud snap publishes on a number of different channels:

stable: The current stable release of Nextcloud (currently v11.0.3. Yes, I know v12 is out, we’ll talk about that in a sec)
candidate: Release candidates, used for testing calls before releasing to stable
beta: Snap development branch. Automatically built as new features are added to the snap (e.g. when pull requests are merged for the snap)
edge: Daily builds of Nextcloud’s master branch
11/edge: Daily builds of Nextcloud’s v11 maintenance branch
12/edge: Daily builds of Nextcloud’s v12 maintenance branch

Any one of these could immediately be installed with:

$ sudo snap install nextcloud --channel=<channel>

It’s actually pretty neat: want to see what work went into Nextcloud yesterday? snap install --edge nextcloud. It’ll automatically update daily, giving you the opportunity to easily track development.

We have all these things happening automatically, but with zero CI because as I mentioned, we couldn’t find any hosted CI solutions that were shiny enough to actually run snaps. Of course we didn’t release to beta much less candidate or stable without some extensive manual testing, but it wasn’t good enough: no one was testing edge.

So Nextcloud version 12 was just released upstream, and we quickly discovered that it doesn’t work in the snap. If we were actually able to run tests against the snap we could have caught this before v12 was actually released!

The solution

A few weeks ago, I saw a forum post from Alan detailing his use of Circle CI for building/pushing snaps. Our builds are already automated so I wasn’t so interested in that, but what caught my eye was the fact that he was installing Snapcraft from the snap, i.e. Circle CI could run snaps! I took a closer look as soon as I could.

It turns out that, like Travis, Circle CI runs on Ubuntu 14.04 (Trusty). Snaps actually can run on Trusty, but the kernel was too old. However, instead of using Docker as an executor type, one can use the (still-in-beta) machine executor type, which spins up a VM that is still based on Trusty, but uses a newer v4.4 kernel.

Once I enabled the Circle CI integration for the project, it was a simple matter of actually creating acceptance tests to run, and then creating the correct Circle CI configuration to build the snap and run the tests against it.

Creating acceptance tests

Nextcloud actually already has a suite of acceptance tests. However, they make a lot of assumptions about where they are and how they’re run, such that they wouldn’t work for our purposes.

I have a lot of experience writing Rails apps, and one of my favorite tools from that world is Capybara. I’ve never used it in a Ruby-only project, but it was a perfect fit for what I needed to do, so I went for it. I needed five gems– here’s my Gemfile:

source 'https://rubygems.org'

# Test driver
gem 'capybara'

# Webdriver (for javascript)
gem 'capybara-webkit'

# Create xvfb from ruby (needed by webkit)
gem 'headless'

# Use nice acceptance tests DSL
gem 'rspec'

# Run tests with a decent CLI
gem 'rake'

Install those gems and their dependencies:

$ sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev
$ gem install bundler
$ bundle install

Then setup rspec:

$ rspec --init
 create  .rspec
 create  spec/spec_helper.rb

That spec/spec_helper.rb is where we put some setup code for the tests:

require 'capybara'
require 'capybara/dsl'
require 'capybara/rspec'
require 'capybara-webkit'
require 'headless'

include Capybara::DSL

Capybara.configure do | config |
   # Set javascript driver to webkit (selenium is the default)
   config.default_driver = :webkit

   # Assume the snap is available at localhost. Change here if otherwise.
   config.app_host = 'http://localhost'

   # The snap will already be running, no need to run a server.
   config.run_server = false
end

Capybara::Webkit.configure do |config|
   # Don't raise errors when SSL certificates can't be validated
   config.ignore_ssl_errors

   # Raise JavaScript errors as exceptions
   config.raise_javascript_errors = true

   # Allow pages to make requests to any URL without issuing a warning.
   config.allow_unknown_urls
end

RSpec.configure do |config|
   config.expect_with :rspec do |expectations|
       expectations.include_chain_clauses_in_custom_matcher_descriptions = true
   end

   config.mock_with :rspec do |mocks|
       mocks.verify_partial_doubles = true
   end

   config.shared_context_metadata_behavior = :apply_to_host_groups

   # Webkit needs a framebuffer in order to work, so create one before the tests
   # run, and destroy it afterward.
   config.before(:all) do
       @headless = Headless.new
       @headless.start
   end

   config.after(:all) do
       @headless.destroy
   end
end

With the generic setup out of the way, I decided to write a simple test that would have caught the issue introduced in v12: just login with and without valid credentials (this test assumes the existence of an “admin” user whose password is “admin”).

feature "Logging in" do
   scenario "Logging in with correct credentials" do
       visit "/"
       fill_in "User", with: "admin"
       fill_in "Password", with: "admin"
       click_button "Log in"
       expect(page).to have_content "Documents"
   end

   scenario "Logging in with incorrect credentials" do
       visit "/"
       fill_in "User", with: "wronguser"
       fill_in "Password", with: "wrongpassword"
       click_button "Log in"
       expect(page).to have_content "Wrong password"
   end
end

Short and sweet. In order to run this test, I created a Rakefile in the directory containing the spec/ folder, with the following contents:

require 'rake'
require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:test) do |t|
   t.pattern = Dir.glob('spec/**/*_spec.rb')
end

task :default => :test

Now the entire suite of tests (even when more specs are added) can be run with rake test. I verified that these tests pass normally, and throw terrible javascript errors in v12, so this simple suite would indeed catch the issue that bit us. Time to get it into CI!

Circle CI Config

Since I needed to use that new executor type, I needed to use Circle CI 2.0. Here’s the YAML I ended up using:

version: 2
jobs:
 build:
   working_directory: ~/nextcloud-snap
   machine: true
   steps:
     - checkout
     - run:
         command: |
           sudo apt update
           sudo apt install -y snapd
           docker run -v $(pwd):$(pwd) -t ubuntu:xenial sh -c "apt update -qq && apt install snapcraft -y && cd $(pwd) && snapcraft"           
     - run:
         command: |
           sudo snap install *.snap --dangerous
           sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev -y
           sudo gem install bundle
           cd tests
           bundle install --deployment
           sudo nextcloud.manual-install admin admin
           bundle exec rake test

It’s beyond the scope of this post to teach the entire Circle CI syntax, but I want to go over the important parts in more detail: the two run steps (they’re run in order).

- run:
   command: |
     sudo apt update
     sudo apt install -y snapd
     docker run -v $(pwd):$(pwd) -t ubuntu:xenial sh -c "apt update -qq && apt install snapcraft -y && cd $(pwd) && snapcraft"

As I mentioned, the machine executor type is Trusty. Well, the Nextcloud snap depends on packages in the Xenial archives, so it needs to build on Xenial. So we fire up a Xenial Docker image just to build the snap. We mount the current working directory into place so that the built snap ends up on the host once complete.

- run:
   command: |
     sudo snap install *.snap --dangerous
     sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev -y
     sudo gem install bundle
     cd tests
     bundle install --deployment
     sudo nextcloud.manual-install admin admin
     bundle exec rake test

This step runs after the one that builds the snap, so the first thing we do is install the built snap. This works because as I said, snaps run on Trusty (given the right kernel). We then install the prerequisites for our test suite, create an “admin” user with its password set to “admin”, and then run the tests.

Conclusion

Building, installing/running, and testing the complete, complex snap is awesome. We don’t have to worry about any differing dependencies between production and testbed (because they’re literally the same binary), and Capybara works wonderfully well standalone in this manner. I’m really thankful for Circle CI actually supporting running snaps, particularly after hitting a wall with e.g. Travis.

Right now the snap is built and tested for every pull request in the snap. We need to start running these tests daily against the daily snaps as well, but unlike Travis (which recently introduced cron jobs), Circle CI does not natively support running jobs periodically. It does support triggering jobs via a web API though, so I may just trigger it from cron running on my own infrastructure. Beyond that, I look forward to fleshing the test suite out further. If you have good ideas, make a pull request! We’d love to see what you have in mind.

Call for testing: Nextcloud snap v11.0.3snap3

Wed, 24 May 2017 00:00:00 +0000

We’re getting ready to release a new version of the Nextcloud snap. v11.0.3snap3-rc1 is now in the candidate channel:

$ sudo snap install --candidate nextcloud

This update includes:

An update to MySQL v5.7.18
An update to certbot v0.14.1

We’d love this new version (particularly features that involve those updates) to get some exercise before we release it. Feel like helping? We’ve put together some smoke tests to get you started.

Please test all you can, but if you only have time to focus on the smoke tests covering the new features, that would still be very useful. We’re particularly looking to cover that:

There are no regressions regarding obtaining certificates from Let’s Encrypt
There are no issues upgrading from the current Nextcloud snap to the new one with the new version of MySQL.

Thank you for your help, and let me know if you have any questions.

ROS production: behind the scenes

Fri, 19 May 2017 00:00:00 +0000

I’m an engineer, and am pretty clueless when it comes to film-making. However, I’ve been asked by a few different people how I made the ROS prototype to production on Ubuntu Core video series. While I’m not deluding myself into thinking I’m a pro, I’m happy with how they turned out, and they were made on a shoestring budget with open-source software. This post is for those who are curious about what I did, what I used, and how I did it.

The hardware

The camera

I don’t care how awesome you are at post-processing, there’s not a lot you can do to make up for a terrible camera. For this series, I used a Canon Rebel T5i, which is pretty good (I wouldn’t call it cheap, but it’s far from top-of-the-line).

I already owned it, but if you have the opportunity to buy a camera, here are some things to consider:

Resolution

The T5i can only shoot video in 1080p. Perhaps you’re wanting to make 4k video: you can stop right here. But perhaps you’re happy with making 1080p video: this may work fine, but you won’t be able to punch into the image at all without degrading quality (e.g. create footage like this).

Frame rate

At 1080p, the T5i maxes out at 30fps (you can get a bit more if you shoot at a lower resolution, but who wants that?). The videos I produce are 24fps, so that’s acceptable. But it means I can’t move very quickly without unacceptable blur, and I can’t do slow motion at all without it looking jumpy. You’ll notice that I do, though, and it doesn’t look jumpy. I faked it: the robot was barely moving. The “slow motion” was really full-speed, and the faster stuff was sped up. Yeah, I recommend a higher-frame-rate camera if you’re able to swing it (assuming you love slow-mo vibes as much as I do).

Formats

When I shoot photos I shoot in RAW, and edit them in Darktable. When I purchased the camera that’s really all I wanted, but now that I’m making video with it I’m saddened by the fact that, when making video, it only supports h264. I think I could get more out of it if I used Magic Lantern, but just because I run Linux doesn’t mean I want to tinker with everything (there’s a reason I use Ubuntu). Perhaps I’ll get there eventually, but until then, the alterations I can make while editing are somewhat limited.

The microphone

Camera microphones are even worse than the microphones built into your laptop. If you actually need audio while you’re taking the video (e.g. recording an interview), I recommend something like the Rode NTG4. I’d love one of those. Fortunately, I had no need to use audio from the video. I could record the footage and then voice over it later. I actually do voice work for radio, so I happen to already have a good mic (an MXL 009, to be specific).

Lighting

I shot all the footage for the series in my living room and kitchen (had to kick the family out every time, they loved that). There are some big windows in there and a good amount of daylight, but I was shooting toward the middle of the house, and the light just wasn’t good enough to shoot at the ISO I wanted (800 or less), so I needed some lights. However, I liked how the light coming from the windows looked, so I wanted to make sure I kept that appearance. I did some research, and decided I wanted a soft box. Two, actually: a key light, and a fill light (to soften the shadows made by the key light). I ran into a few issues here:

The soft boxes themselves are not that expensive, but the lights that go inside them can be. I was probably looking at a few hundred bucks (at least).
Even if Canonical was okay with that cost (I doubted they would be), I needed to order them and I was on a deadline (anyone else notice that Amazon’s two-day shipping == five-day shipping these days?)

So I went with an alternative.

What you see here is a cheap styrofoam cooler covered in parchment paper to diffuse the light. I poked a work lamp through the back of it and put a daylight (5700K) LED bulb in there. I made two of them. Not durable in the least but they worked well for this project and still exist, so apparently durable enough.

Note: I selected LED bulbs because I didn’t want to put anything hot inside styrofoam, but if you’re using a fast shutter speed, beware of flicker. I was using commercial LED bulbs, and they flickered quite a bit if I went too high.

Often a three-point light system is used that includes a back light, but I found the light coming from the windows worked well enough for that. How I positioned the lights varied depending on the shot I was trying to get, but most of the time it looked something like this (the vertical lines are the windows):

The most difficult parts were the moving shots, where I really didn’t want to cast a shadow by crossing the light and the subject. Those shots were obtained by rotating the light setup a bit, or putting lights on chairs, etc. I never moved the lights while shooting to keep it looking like the light was coming from windows.

Slider

Lots of the footage I captured was in motion, with most of it at floor-level. I needed to move slowly and smoothly, which meant some sort of slider/dolly track. Those can start getting up there in price, and again, I had no time. I really wanted to make one of these, but decided I didn’t have time for that either. So again, I went with an alternative.

Even this I had to get on loan from my two year old.

My wooden floors have a texture to them that make them lovely to look at and walk on, but dreadful for moving my professional-grade slider smoothly, so my yoga mat is there to even it out.

This setup worked for the situations when I was shooting alongside the action (e.g. the intro for part 1), but there were other times when I needed to move straight toward the action, when the yoga mat would be in the way. Also, the Hess 2016 Edition does not have a nice way to mount the camera facing forward. I should complain. Regardless, when moving along the rough floor I found painted (i.e. slick) cardboard worked acceptably well.

You’ll see footage captured from that in the intro for part 1 as well as part 2. The moving footage well above the floor was done via a tripod set on this same piece of cardboard. A fluid head would have been handy.

In the intro for part 5 I needed to get some moving footage where I was front-facing and tilted downward, very close to the surface of my kitchen counter. The mount situation on the Hess bit me here, so I had to break out the Adventure Wheels.

The software

Capturing my screen

Man, I’ll tell you. The screen recording software situation on Linux is so sad. I would say it’s like a barren wasteland, except that there are several options. All of which suck. I think I tried them all. Every one of them either couldn’t handle my 4k screen (recording only the upper-left quarter of it) or the color was way off.

The least bad option was RecordMyDesktop. It was the one exception to the previous broad brush: it could handle 4k and the colors looked okay. It was insanely slow and was only happy recording at 15fps (I tried switching to 24 and it ended up a garbled mess), and even then a bunch of frames got encoded out of order. Watch some of my videos closely: that wasn’t a typo that got deleted and re-typed, it was RecordMyDesktop shuffling frames randomly. But it did the job well enough.

You might be asking: “Why didn’t you just bump your resolution down to 1920x1080 to record?” First of all, I shouldn’t have to. Really though, it’s because I wanted to be able to zoom into portions of the video without losing quality (e.g. step 1 of the video for part 4).

Most of my recordings were of full-screen applications: I’d make them full-screen (which hid the clock and task bar), make the text larger so things would be readable on a smaller video, select that window within RecordMyDesktop, and start recording.

I did things in the smallest chunks possible, because RecordMyDesktop would take ages to encode. For example, if I needed to show three separate commands in the terminal one after the other, I’d record each of them in separate takes, simply making sure not to touch the terminal between takes so they pieced together nicely. I didn’t fake anything here, although I certainly sped pieces up in post-processing.

After I was done, I’d hand them off to ffmpeg to convert from 15fps to 24 so I didn’t need to deal with different framerates in Blender. I wish RecordMyDesktop would just do that for me.

Recording audio

I have a lot of experience with Audacity, so that was my obvious choice. However, this time around (running v2.1.2 on Xenial) I ran into some stability issues. It crashed on me about once a session, and it didn’t always cut the track where I actually asked it to cut (sometimes I didn’t notice and ended up having to re-record a track). I still used it, and it did a good job of recovering when it crashed, but that was frustrating. Your mileage may vary (at least on that version). (Update: On Bionic, Audacity no longer seems to be crashing, but it’ll still cut weird. I’ve started copying the original recordings into their own project before starting any editing.)

I recorded each paragraph of the script as a separate track, edited them individually (e.g. removed the whirring sound of my server in the background), and then used “export multiple” to export them all at once so I could put them where they needed to go in the video editor.

Video editor

Believe it or not, I use one of the prides of open-source: Blender. You’re thinking “Wait, isn’t Blender a 3D modeling program?” Yes, it is. But it also has the best video sequence editor (VSE) I’ve ever used. In fact, the only time I used its 3D capabilities was for the title floating in on part 5. You probably didn’t notice. It took me four hours. Shut up.

Blender is rock solid, but it has a lot of features and can be difficult to learn. Mike Meyers (mikeycal) created an excellent series of tutorials for learning pretty much everything you need to know about the VSE, and I can’t recommend it enough.

The only feature that’s missing at least in v2.76 that I’m using on Xenial is a simple way to do good-looking text. It has a text effect, but there’s no font control so it ends up just making me laugh. It sounds like they’re well aware of the limitation though– I expect this is already fixed in later versions.

As a work-around, I used Inkscape (and the wonderful Ubuntu font) to create the text you see in the videos. I made sure to place all the text in the same place (aligned left, vertically centered) and exported the entire page as a transparent PNG (so every text image was the same size). This allowed me to easily align the text in the editor and be confident they were all the same size.

You’ll see that it took me a few videos to converge on a style: part 1 brought text into frame in a way that I felt was slightly gimmicky, so improving the text was a large focus in part 2. I liked that better, and you’ll see that I continued that style throughout the rest of the series.

Editing took the majority of my time for each video, but it went faster as I gained experience. For example, editing was always the last step, but the order in which I actually did the editing morphed over time. For the first video, I pretty much perfected the video before adding the audio in. Then I realized that I had too much audio for the cuts I made, so I had to do all the cutting and transition work again. Like the efficient machine I am, I went through pretty much the same process for part 2 before I realized how much time I was wasting. Things went much smoother once I started importing all the raw footage and audio at the same time and aligning them before working on any cutting or transitions. Seems obvious in retrospect.

Conclusion

I had a good time with this project, and I see more videos in my future. I’ll probably slowly purchase better gear as time goes on, starting with a better slider and fluid head. But for the most part, this process worked well, and cost very little. I hope this post helps somebody out there!

Call for testing: Darktable v2.2.4snap1

Thu, 18 May 2017 00:00:00 +0000

The Darktable snap (darktable-kyrofa) is terribly out-of-date. I’d like to bring it up to the latest release, but I could use your help verifying that things still work. If you’re a photographer, particularly if you have some experience using Darktable, mind putting it through its paces? v2.2.4snap1-rc1 is in the candidate channel:

$ sudo snap install --candidate darktable-kyrofa

Please let me know if you run into any issues.

Update: v2.2.4snap1 has now been released into the stable channel.

Call for testing: Nextcloud snap v11.0.3snap2

Thu, 11 May 2017 00:00:00 +0000

We’re getting ready to release a new version of the Nextcloud snap with a few new features, including:

Supporting changing ports from the default 80/443.
Supporting Samba shares.

We’d love this new version (particularly these new features) to get some exercise before we release it. Feel like helping? We’ve put together some smoke tests to get you started.

Please test all you can, but if you only have time to focus on the smoke tests covering the new features, that would still be very useful. Thank you!

Let me know if you have any questions.

Update: Thanks to testers, we found issues with both major features. We’ve removed them from this release, which makes 11.0.3snap2 purely a security release. It has been released.

ROS production: create an Ubuntu Core image with our snap preinstalled [5/5]

Thu, 04 May 2017 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the fifth (and final) blog post in this series about ROS production. In the previous post we created a gadget snap to allow confined access to the Turtlebot. In this post, we’re going to put all the pieces from this series together and create an Ubuntu Core image with our ROS snap preinstalled, ready to send to the factory for production.

If you download one of the generic Ubuntu Core images for a reference device and run snap list on it immediately, you’ll see three snaps installed: a core snap, a kernel snap, and a gadget snap.

Every Ubuntu Core image consists of at least these three snaps. In my case, since I’m using an Intel NUC, I see core, pc-kernel and pc-amd64-gadget. You may recognize that last one from part 3, because it’s the gadget we started with when creating our custom one.

We want to create an Ubuntu Core image that uses our custom gadget, and includes our ROS snap so that as soon as the device boots for the first time, the robot starts moving.

Alright, let’s get started! Remember that this is also a video series: feel free to watch the video version of this post:

Step 1: Request interface auto-connection

As soon as the device boots for the first time, we want the Turtlebot to begin moving. In order for this to happen, the ROS snap needs access to the Turtlebot at boot. We added the correct slot to our gadget and plug to our app, but this interface isn’t automatically connected, which means at boot, the ROS snap doesn’t actually have the appropriate permission to access the Turtlebot. This works okay for testing purposes, but to actually have a production image, we need this interface automatically connected. Eventually this will be automated, but currently it’s a manual process. You have to:

Put the ROS snap in the store (which we did in part 3).
Put the gadget snap in the store (which we did in part 4).
Create a new topic in the Snapcraft forum requesting that the kobuki interface (or whatever you called it) be automatically connected between these two snaps. This is done via an assertion on the gadget in the store.

UPDATE: This is no longer the case. You can now specify auto-connection straight in the gadget instead of only relying on the store. As it should be.

Step 2: Install required tools

Ubuntu Core will verify that the image it’s booting actually came from you. In order for that to happen, you need to create a signing key and register it with the store. You need Snapcraft and Snapd to generate, register, and use the signing key, and you need Ubuntu Image to actually create the image. You already have most of this from following the rest of the series, but for completeness:

$ sudo apt install snapcraft snapd ubuntu-image

Step 3: Create and register your signing key

Now let’s create a signing key:

$ snapcraft create-key my-key-name

Check that everything worked and your key is ready:

$ snapcraft list-keys
  Name          SHA3-384 fingerprint
* my-key-name   Qjdfpj0EW<snip>iZ41H4CROy (not registered)

Now register your new key with the store to associate it with your account:

$ snapcraft register-key
Registering key ...
Done. The key "my-key-name" (Qjdfp<snip>H4CROy) may be used to sign your assertions.

Step 4: Define the model

In order to create an Ubuntu Core image, we need to create what’s called the “model definition.” The “model” in question is your device. The definition is simply a JSON document that answers the questions “what is this device, who created it, and what software is running on it?” The model definition and assertion I created are available for reference.

Create a file named turtlebot-model.json and make it look something like this:

{
 "type": "model",
 "series": "16",
 "model": "turtlebot-demo",
 "architecture": "amd64",
 "kernel": "pc-kernel",
 "gadget": "<my gadget snap>",
 "authority-id": "<my account ID>",
 "brand-id": "<my account ID>",
 "timestamp": "<timestamp>",
 "required-snaps": ["<my turtlebot snap>"]
}

We specify that this is targeting Ubuntu Core series 16, name the model “turtlebot-demo” (you can use whatever you wish), specify the architecture for this model as well as the kernel snap to use (these would change if your arch wasn’t amd64). Note that we’re using the kernel maintained and supported by Canonical here. The gadget field needs to contain the name of your gadget snap. The authority-id and brand-id need to be the account ID from your store account. The timestamp needs to be the output of

$ date -Iseconds --utc

(I would explain why if I knew.) Finally, the required-snaps is a list of snaps to be preinstalled on this image. In our case, that’s the name of the ROS snap we created in part 3.

Step 5: Create the model assertion

Now that we have a model definition, we need to use our key to sign it, thereby turning it into what’s called a “model assertion”:

$ cat turtlebot-model.json | snap sign -k my-key-name > turtlebot.model
You need a passphrase to unlock the secret key for
user: "my-key-name"
4096-bit RSA key, ID 0B79B865, created 2016-01-01
...

After giving your passphrase, a turtlebot.model file is created containing the assertion.

Step 6: Create the image

The assertion we just created is all we need for ubuntu-image to actually put the image together:

$ sudo ubuntu-image -c stable -o turtlebot.img turtlebot.model

This will pull down the various components of the image (gadget, kernel, core, and any additional snaps from required-snaps) from the stable channel, and create an Ubuntu Core image out of them. Well, what if the gadget (or other) snap is your own, and isn’t available in the store yet? The assertion doesn’t change, but you can provide local snaps to satisfy the snap names specified in the assertion with the --extra-snaps parameter. For example, say your gadget snap isn’t available in the store; you could use the local one with:

$ sudo ubuntu-image -c stable -o turtlebot.img \
                    --extra-snaps /path/to/gadget.snap \
                    turtlebot.model

We already mentioned the downsides to this, but it’s worth mentioning again: installing local snaps like this will prevent those snaps from updating, and will also prevent the serial-port interface from auto-connecting.

Once ubuntu-image completes successfully, you’ll have an Ubuntu Core image ready to flash to your device!

Step 7: Test the image

You can take the image you just created for a test drive by following the flashing directions for the device you used, or if you use amd64 you can even test it out in VirtualBox or KVM.

Once the image boots (assuming the Turtlebot is connected and powered on), the Turtlebot will begin moving. Congratulations, you’re done!

Step 7a: Connect interfaces if necessary

If you were unable to follow Step 1, then the Turtlebot probably didn’t begin moving for you once you powered the device on. This is due to the lack of interface auto-connection. To work around that, go through the first boot wizard as discussed on the flashing directions, SSH into the device, and run:

$ snap interfaces
Slot                   Plug
[...]
my-gadget-snap:kobuki  -
-                      my-turtlebot-snap:kobuki

As you can see, this interface is not connected. So connect them:

$ sudo snap connect my-turtlebot-snap:kobuki my-gadget-snap:kobuki
$ snap interfaces
Slot                   Plug
[...]
my-gadget-snap:kobuki  my-turtlebot-snap:kobuki

The ROS snap now has permission to access the Turtlebot, but the service has probably errored out by now. So let’s start it again by disabling and re-enabling the snap:

$ sudo snap disable my-turtlebot-snap
$ sudo snap enable my-turtlebot-snap

Now you should see the Turtlebot start moving as expected. This is persistent across reboots.

Conclusion

At this point, we’ve taken a ROS prototype running on classic Ubuntu all the way to a production-ready Ubuntu Core image with the ROS prototype preinstalled and running on boot. Every component of this image will automatically update, including your ROS snap. Admittedly there are a few rough edges, but I hope this shows you how powerful snaps and Ubuntu Core can be.

Hopefully you managed to follow along successfully, but if you ran into issues, here’s the reference material for this series:

Turtlebot demo ROS snap
- Note that this is published as turtlebot-demo-kyrofa in the store for all supported architectures (stable channel)
Turtlebot amd64 gadget
- Note that this is published as pc-turtlebot-kyrofa in the store (stable channel)
Turtlebot Dragonboard gadget
- Included for reference. Note that this is published as dragonboard-turtlebot-kyrofa in the store (stable channel)
Model definition and assertion used
- This includes both amd64 and dragonboard model definitions/assertions

If you need any help with this, or anything to do with ROS on snaps and/or Ubuntu Core, please feel free to reach out directly, ask in the Snapcraft forum, or ask in the ROS forum.

ROS production: obtaining confined access to the Turtlebot [4/5]

Wed, 26 Apr 2017 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the fourth blog post in this series about ROS production. In the previous post we created a snap of our prototype, and released it into the store. In this post, we’re going to work toward an Ubuntu Core image by creating what’s called a gadget snap. A gadget snap contains information such as the bootloader bits, filesystem layout, etc., and is specific to the piece of hardware running Ubuntu Core. We don’t need anything special in that regard (read that link if you do), but the gadget snap is also currently the only place on Ubuntu Core and snaps in general where we can specify our udev symlink rule in order to obtain confined access to our Turtlebot (i.e. allow our snap to be installable and workable without devmode). Eventually there will be a way to do this without requiring a custom gadget, but this is how it’s done today.

Alright, let’s get started. Remember that this is also a video series– feel free to watch the video version of this post:

Step 1: Start with an existing gadget

Canonical publishes gadget snaps for all reference devices, including generic amd64, i386, as well as the Raspberry Pi 2 and 3, and the DragonBoard 410c. If the computer on which you’re going to install Ubuntu Core is among these, you can start with a fork of the corresponding official gadget snap maintained in the github.com/snapcore organization. Since I’m using a NUC, I’ll start with a fork of pc-amd64-gadget (my finished gadget is available for reference, and here’s one for the DragonBoard). If you’re using a different reference device, fork that gadget and you can still follow the rest of these steps.

Step 2: Select a name for your gadget snap

Open the snapcraft.yaml included in your gadget snap fork. You’ll see the same metadata we discussed in the previous post. Since snap names are globally unique, you need to decide on a different name for your gadget. For example, I selected pc-turtlebot-kyrofa. Once you settle on one, register it:

$ snapcraft register my-gadget-snap-name

If the registration succeeded, change the snapcraft.yaml to reflect the new name. You can also update the summary, description, and version as necessary.

Step 3: Add the required udev rule

If you’ll recall, back in the second post in this series, we installed a udev rule to ensure that the Turtlebot showed up at /dev/kobuki. Take a look at that rule:

$ cat /etc/udev/rules.d/57-kobuki.rules
# On precise, for some reason, USER and GROUP are getting ignored.
# So setting mode = 0666 for now.
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="kobuki*", MODE:="0666", GROUP:="dialout", SYMLINK+="kobuki"
# Bluetooth module (currently not supported and may have problems)
# SUBSYSTEM=="tty", ATTRS{address}=="00:00:00:41:48:22", MODE:="0666", GROUP:="dialout", SYMLINK+="kobuki"

It’s outside the scope of this series to explain udev in depth, but the two values I’ve made bold are how the Kobuki base is uniquely identified. We’re going to write the snapd version of this rule, using the same values. At the end of the gadget’s snapcraft.yaml, add the following slot definition:

# Custom serial-port interface for the Turtlebot 2
slots:
 kobuki:
   interface: serial-port
   path: /dev/serial-port-kobuki
   usb-vendor: 0x0403
   usb-product: 0x6001

The name of the slot being defined is kobuki. The symlink path we’re requesting is /dev/serial-port-kobuki. Why not /dev/kobuki? Because snapd only supports namespaced serial port symlinks to avoid conflicts, without documentation, and with silent errors (it’s lovely, I’m not grumpy about it). You can use whatever you like for this path (and the slot name), but make sure to follow the /dev/serial-port-<whatever> pattern, and adjust the rest of the directions in this series accordingly.

Step 4: Build the gadget snap

This step is easy. Just get into the gadget snap directory and run:

$ snapcraft

In the end, you’ll have a gadget snap.

Step 5: Put the gadget snap in the store

You don’t technically need the gadget snap in the store just to create an image, but there are two reasons you will want it there:

Without putting it in the store you have no way of updating it in the future
Without putting it in the store it’s impossible for the serial-port interface we just added to be automatically connected to the snap that needs it, which means the image won’t make the robot move out of the box (Update: This is no longer true, you can specify auto-connections within the gadget now)

Since you’ve already registered the snap name, just push it up (we want our image based on the stable channel, so let’s release into that):

$ snapcraft push /path/to/my/gadget.snap --release=stable

You will in all likelihood receive an error saying it’s been queued for manual review since it’s a gadget snap. It’s true, right now gadget snaps require manual review (although that will change soon). You’ll need to wait for this review to complete successfully before you can take advantage of it actually being in the store (make a post in the store category of the forum if it takes longer than you expect), but you can continue following this series while you wait. I’ll highlight things you’ll need to do differently if your gadget snap isn’t yet available in the store.

Step 6: Adjust our ROS snap to run under strict confinement

As you’ll remember, our ROS snap currently only runs with devmode, and still assumes the Turtlebot is available at /dev/kobuki. We know now that our gadget snap will make the Turtlebot available via a slot at /dev/serial-port-kobuki, so we need to alter our snap slightly to account for this. Fortunately, when we initially created the prototype, we made the serial port configurable. Let’s pat ourselves on the back for a moment, then edit our snapcraft.yaml a bit:

name: my-turtlebot-snap
version: '0.2'
summary: Turtlebot ROS Demo
description: |
  Demo of Turtlebot randomly wandering around, avoiding obstacles and cliffs.

grade: stable

# Using strict confinement here, but that only works if installed
# on an image that exposes /dev/serial-port-kobuki via the gadget.
confinement: strict

parts:
 prototype-workspace:
   plugin: catkin
   rosdistro: kinetic
   catkin-packages: [prototype]

plugs:
 kobuki:
   interface: serial-port

apps:
 system:
   command: roslaunch prototype prototype.launch device_port:=/dev/serial-port-kobuki --screen
   plugs: [network, network-bind, kobuki]
   daemon: simple

Most of this is unchanged from version 0.1 of our snap that we discussed in the previous post. Let’s cover only the parts that changed.

version: 0.2

We’re modifying the snap here, so I suggest changing the version. This isn’t required (the version field is only for human consumption, it’s not used to determine which snap is newest), but it certainly makes support easier (“what version of the snap are you on?").

# Using strict confinement here, but that only works if installed
# on an image that exposes /dev/serial-port-kobuki via the gadget.
confinement: strict

You wrote such a nice comment, it hardly needs more explanation. The point is, now that we have a gadget that makes this device accessible, we can switch to using strict confinement.

plugs:
 kobuki:
   interface: serial-port

This one defines a new plug in this snap, but then says “the kobuki plug is just a serial-port plug.” This is optional, but it’s handy to have our plug named kobuki instead of the more generic serial-port. If you opt to leave this off, keep it in mind for the following section.

apps:
 system:
   command: roslaunch prototype prototype.launch device_port:=/dev/serial-port-kobuki --screen
   plugs: [network, network-bind, kobuki]
   daemon: simple

Here we take advantage of the flexibility we introduced in the second post of the series, and specify that our launch file should be launched with device_port set to the udev symlink defined by the gadget instead of the default /dev/kobuki. We also specify that this app should utilize the kobuki plug defined directly above it, which grants it confined access to the serial port.

Rebuild your snap, and release the updated version into the store as we covered in part 3, but now you can release to the stable channel. Note that this isn’t required, but the reasons for putting this snap in the store are the same as the reasons for putting the gadget in the store (namely, updatability and interface auto-connection).

In the next (and final) post in this series, we’ll put all these pieces together and create an Ubuntu Core image that is ready for the factory.

ROS production: our prototype as a snap [3/5]

Wed, 19 Apr 2017 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the third blog post in this series about ROS production. In the previous post we came up with a simple ROS prototype. In this post we’ll package that prototype as a snap. For justifications behind why we’re doing this, please see the first post in the series.

We know from the previous post that our prototype consists of a single launch file that we wrote, contained within our prototype ROS package. Turning this into a snap is very straight-forward, so let’s get started! Remember that this is also a video series: feel free to watch the video version of this post:

Prerequisites

This post will assume the following:

You’ve followed the previous posts in this series
You know what snaps are, and have taken the tour
You have a store account at dashboard.snapcraft.io
You have a recent Snapcraft installed (2.28 is the latest as of this writing)

Create the snap

The first step toward a new snap is to create the snapcraft.yaml. Put that in the root of the workspace we created in the previous post:

$ cd ~/workspace
$ snapcraft init
Created snap/snapcraft.yaml.
Edit the file to your liking or run `snapcraft` to get started

Do as it says, and make that file look something like this:

name: my-turtlebot-snap # This needs to be a unique name
version: '0.1'
summary: Turtlebot ROS Demo
description: |
  Demo of Turtlebot randomly wandering around, avoiding obstacles and cliffs.

grade: stable
confinement: devmode

parts:
 prototype-workspace:
   plugin: catkin
   rosdistro: kinetic
   catkin-packages: [prototype]

apps:
 system:
   command: roslaunch prototype prototype.launch --screen
   plugs: [network, network-bind]
   daemon: simple

Let’s digest that section by section.

name: my-turtlebot-snap version: 0.1 summary: Turtlebot ROS Demo description: | Demo of Turtlebot randomly wandering around, avoiding obstacles and cliffs.

This is the basic metadata that all snaps require. These fields are fairly self-explanatory. The only thing I want to point out specifically here is that the name must be globally unique among all snaps. If you’re following this tutorial, you might consider appending your developer name to the end of this example.

grade: stable
confinement: devmode

confinement can be strict, devmode, or classic. strict enforces confinement, whereas devmode allows all accesses, even those that would be disallowed under strict confinement (and logs accesses that would otherwise be disallowed for your reference). classic is even less confined than devmode, in that it doesn’t even get private namespaces anymore (among other things). There is more extensive documentation on confinement available.

I personally always use strict confinement unless I know for sure that the thing I’m snapping won’t run successfully under confinement, in which case I’ll use devmode. I typically avoid classic unless I never intend for the app to run confined. Also, classic confinement requires manual review in the store, a process that generally isn’t that much fun.

In this case, I know from experience this snap won’t run confined as-is, and will require devmode for now (more on that later).

parts:
 prototype-workspace:
   plugin: catkin
   rosdistro: kinetic
   catkin-packages: [prototype]

You learned about this in the Snapcraft tour, but I’ll cover it again real quick. Snapcraft is responsible for taking many disparate parts and orchestrating them all into one cohesive snap. You tell it the parts that make up your snap, and it takes care of the rest. Here, we tell Snapcraft that we have a single part called prototype-workspace. We specify that it builds with Catkin, and also specify that we’re using Kinetic here (as opposed to Jade, or the default, Indigo). Finally, we specify the packages in this workspace that we want included in the snap. In our case, we only have one: that prototype package we created in part 2.

apps:
 system:
   command: roslaunch prototype prototype.launch --screen
   plugs: [network, network-bind]
   daemon: simple

This is where things get a little interesting. When we build this snap, it will include a complete ROS system: roscpp, roslib, roscore, roslaunch, your ROS workspace, etc. It’s a standalone unit: you’re in total control of how the user interacts with it. You exercise that control via the apps keyword, where you expose specific commands to the user. Here, we specify that this snap has a single app, called system. The command that this app actually runs within the snap is the roslaunch invocation we got from part 2. We use plugs to specify that it requires network access (read more about interfaces), and finally specify that it’s a simple daemon. That means this app will begin running as soon as the snap is installed, and also run upon boot. All this, and the user doesn’t even need to know that this snap uses ROS!

That’s actually all we need to make our prototype into a snap. Let’s create the snap itself:

$ cd ~/workspace
$ snapcraft

That will take a few minutes. You’ll see Snapcraft fetch rosdep, which is then used to determine the dependencies of the ROS packages in the workspace. This is only prototype in our case, which you’ll recall from the previous post depends upon kobuki_node and kobuki_random_walker. It then pulls those down and puts them into the snap along with roscore. Finally, it builds the requested packages in the workspace, and installs them into the snap as well. At the end, you’ll have your snap.

Test the snap

Even though we’re planning on using this snap on Ubuntu Core, snaps run on classic Ubuntu as well. This is an excellent way to ensure that our snap runs as expected before moving on to Ubuntu Core. Since we already have our machine setup to communicate with the Turtlebot, we can try it out right here. The only hitch is that /dev/kobuki isn’t covered by any interface on classic Ubuntu (we can make this work for Ubuntu Core, though, more on that later). That’s why we used devmode as the confinement type in our snap. We’ll install it with devmode here:

$ sudo snap install --devmode path/to/my.snap

Right after this completes (give it a second for our app to fire up), you should hear the robot sing and begin moving. Once you remove the snap it’ll stop moving:

$ sudo snap remove my-turtlebot-snap

How easy is that? If you put that in the store, anyone with a Turtlebot (no ROS required) could snap install it and it would immediately begin moving just like it did for you. In fact, why don’t we put it in the store right now?

Put the snap in the store

Step 1: Tell Snapcraft who you are

We’re about to use Snapcraft to register and upload a snap using the store account you created when satisfying the prerequisites. For that to happen, you need to sign in with Snapcraft:

$ snapcraft login

Step 2: Register the snap name

Snap names are globally unique, so only one developer can register and publish a snap with a given name. Before you can publish the snap, you need to make sure that snap name is registered to you (note that this corresponds to the name field in the snapcraft.yaml we created a few minutes ago):

$ snapcraft register <my snap name>

Assuming that name is available, you can proceed to upload it.

Step 3: Release the snap

In the tour you learned that there are four channels available by default. In order of increasing stability, these channels are edge, beta, candidate, and stable. This snap isn’t quite perfect yet since it still requires devmode, so let’s release it on the beta channel:

$ snapcraft push path/to/my.snap --release=beta

Once the upload and automated reviews finish successfully, anyone in the world can install your snap on the computer controlling their Turtlebot as simply as:

$ sudo snap install --beta --devmode my-turtlebot-snap

In the next post in this series, we’ll discuss how to obtain real confined access to the Turtlebot’s udev symlink on Ubuntu Core by creating a gadget snap, moving toward our goal of having a final image with this snap pre-installed and ready to ship.

ROS production: our prototype [2/5]

Wed, 12 Apr 2017 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

This is the second blog post in this series about ROS production. In the previous post we discussed why Ubuntu Core was a good fit for production robotics. In this post we’ll be on classic Ubuntu, creating the example ROS prototype that we’ll use throughout the rest of the series as we work toward using Ubuntu Core.

I mentioned in the previous post that this series was going to use the Turtlebot 2. However, there are a number of possible configurations for the Turtlebot. Some use a Kinect, others use an Astra Pro, still others use a RealSense, etc. I want this prototype to be runnable by everyone who has a Turtlebot 2, so I’m going to cater to the lowest common denominator here: the Kobuki base. We’ll keep it very simple and make it randomly wander around, bumping into things and correcting itself, utilizing both the bumper sensors as well as the cliff sensors.

I also mentioned how the ROS community has standardized around the Turtlebot as the introductory platform. That makes this prototype incredibly easy to create, as every package we need has already been written: we just need to put them together. Let’s get started! Remember that this is also a video series: feel free to watch the video version of this post:

Prerequisites

This post will assume the following:

You’re running on Ubuntu 16.04 (Xenial)
You have ROS Kinetic installed (I installed ros-kinetic-ros-base)
You have gone through at least the first 8 ROS tutorials (e.g. you should know what a launch file is)

The computer being used doesn’t matter so much as this prototype should work on any standard architecture, but for your reference I’m using an Intel NUC (the DE3815TYKE, specifically).

The prototype

Step 1: Install the Kobuki driver

The first Turtlebot-specific package we’ll need is the ROS wrapper for the Kobuki driver. It contains the node that’s responsible for communicating with the Kobuki itself, receiving sensor data and publishing it on ROS, etc.

$ sudo apt install ros-kinetic-kobuki-node

Step 2: Configure udev rule

By default, the driver will look for the Turtlebot at /dev/kobuki. Getting the Kobuki there takes a udev rule. Thankfully, what we just installed includes a utility for installing this udev rule:

$ rosrun kobuki_ftdi create_udev_rules

It may prompt for your sudo password. After this completes successfully, connect your Turtlebot, turn it on, and you should see the /dev/kobuki symlink show up.

Step 3: Test the Kobuki driver

We can prove that this is working by running the driver’s minimal launch file:

$ roslaunch kobuki_node minimal.launch

After the system comes up, you should hear the Turtlebot sing a little tune. If that doesn’t happen, check to make sure that the udev rule is working. If it does happen, leave it running and continue.

Step 4: Install the Kobuki Random Walker

The Kobuki driver is most of what we need. Its minimal launch file brings up enough nodes to obtain data from the wheel drop sensors, cliff sensors, etc., but most importantly exposes the ability to command the robot to move. We just need to make use of it. We want it to randomly walk around, so we’ll use the Kobuki Random Walker (note that this may already be installed):

$ sudo apt install ros-kinetic-kobuki-random-walker

Step 5: Test the Kobuki Random Walker

To run it (the minimal.launch file from kobuki_node still needs to be running):

$ roslaunch kobuki_random_walker safe_random_walker_app.launch

When that system finishes launching, the robot will begin randomly wandering around. If it collides with something using its bumper sensors, it’ll back up and rotate, trying to get around it. If it notices it’s about to drive off a cliff (e.g. your steps) it’ll stop and rotate again.

Go ahead and stop (ctrl+c) both launch files.

Step 6: Put the pieces together

At this point, if you have the appropriate 3d sensors, you could start doing the really fun stuff with the navigation stack, but as I mentioned before, we’re going to keep it simple here. These two launch files make up the entirety of the prototype we’ll use for this series. However, if we leave it as-is it has a small limitation: the only udev symlink location supported is /dev/kobuki. That works on our current system, but what if another user has it elsewhere? Or what if we couldn’t use that exact symlink in a snap (a hint for the next post in this series)? So let’s distill those two launch files into a single launch file that represents our prototype, and make it a little more configurable.

First of all, create a new ROS workspace for this prototype (you can of course put this anywhere):

$ mkdir -p ~/workspace/src
$ cd ~/workspace/src
$ catkin_init_workspace

Now create a new package within that workspace to contain our prototype’s launch file. It should depend upon the two packages we know we need (kobuki_node and kobuki_random_walker):

$ cd ~/workspace/src
$ catkin_create_pkg prototype kobuki_node kobuki_random_walker

Now we’re going to create our launch file:

$ cd ~/workspace/src/prototype
$ mkdir launch
$ touch launch/prototype.launch

Open that launch/prototype.launch file in your favorite editor. In your terminal, open up the minimal.launch file from the Kobuki driver:

$ rosed kobuki_node minimal.launch

We’ll copy the contents of that launch file into ours, clean it up into only the bits we need, and add an argument that makes the serial port configurable. Finally, we’re also going to include the random walker launch file so that launching this single launch file brings up our entire prototype:

<launch>
   <!-- Make the device port configurable. -->
   <arg name="device_port" default="/dev/kobuki"/>

   <!--
       Nodelet manager for the Kobuki driver as well as the safe
       walker.
   -->
   <node pkg="nodelet" type="nodelet"
         name="mobile_base_nodelet_manager" args="manager"/>

   <!-- The ROS wrapper for the Kobuki driver. -->
   <node pkg="nodelet" type="nodelet" name="mobile_base"
         args="load kobuki_node/KobukiNodelet mobile_base_nodelet_manager">
       <rosparam file="$(find kobuki_node)/param/base.yaml" command="load"/>
       <param name="device_port" value="$(arg device_port)"/>
       <remap from="mobile_base/odom" to="odom"/>
       <remap from="mobile_base/joint_states" to="joint_states"/>
   </node>

   <!--
       The safe random walker launch file that causes the robot
       to wander while trying not to destroy itself by
       utilizing the bumper sensors as well as the cliff
       sensors.
   -->
   <include file="$(find kobuki_random_walker)/launch/safe_random_walker_app.launch" />
</launch>

Save and exit. We also need to modify ~/workspace/src/prototype/CMakeLists.txt to actually install that launch file:

cmake_minimum_required(VERSION 2.8.3)
project(prototype)

find_package(catkin REQUIRED)

catkin_package()

install(DIRECTORY launch
   DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)

Save and exit. Note that the entire ROS project used here is available for reference. Finally, build the workspace:

$ cd ~/workspace
$ catkin_make

Now test out our final prototype:

$ source ~/workspace/devel/setup.sh
$ roslaunch prototype prototype.launch

And the robot should sing its little tune and immediately begin moving randomly again. So now we have our prototype contained in a single launch file, and we can use a different serial port like so:

$ roslaunch prototype prototype.launch device_port:=/dev/my-serial-port

In the next post in this series, we’ll discuss the process of packaging this prototype as a snap.

From ROS prototype to production on Ubuntu Core [1/5]

Wed, 05 Apr 2017 00:00:00 +0000

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics.

My background is pretty heavily littered with robotics. A natural side effect of this is that I’ve published numerous posts discussing snaps, Ubuntu Core, and different robotics frameworks (ROS and MOOS specifically). But my robotics experience was professional, which meant I didn’t really have a reason (or money, these things can be expensive) to buy any robotic systems personally. As a result, most of my posts have been pretty high level. Well today, that changes. Today, I have what you see above, which is a Turtlebot 2.

Well, not really. What you see here is the iClebo Kobuki, which is the base of the Turtlebot 2. The barebones version of the Turtlebot 2 is the Kobuki plus some mounting hardware that goes on top. You can save some significant money if you can do without the mounting hardware: skip the kit, and buy the Kobuki itself.

Anyway, the Turtlebot 2 is the go-to robotic platform for people learning ROS. Not just because there’s no time needed for researching components and writing the necessary software to build your own, but because this is what the ROS community has standardized around. It already has numerous ROS packages written for it, as well as its own set of excellent tutorials on the ROS wiki.

This situation makes it the perfect guinea pig for the ROS blog series I’ve been itching to write: taking your ROS prototype to production with snaps and Ubuntu Core. This series will cover the following topics:

ROS production: our prototype [2/5]
- Creating an example prototype using classic Ubuntu and ROS Kinetic.
ROS production: our prototype as a snap [3/5]
- Packaging that prototype as a snap.
ROS production: obtaining confined access to the Turtlebot [4/5]
- Creation of a gadget snap to allow explicit access to the Turtlebot.
ROS production: create an Ubuntu Core image with our snap preinstalled [5/5]
- Creation of an image that is ready for the factory.

In this first post, I want to discuss why Ubuntu Core is such a good fit for production robotics. Note that this is also a video series, feel free to watch the video version of this post:

ROS is great for research and development. Getting started is as simple as installing a few packages from the ROS archive, creating a workspace, and diving in. This is all I ever did in the past. When it was time to try my changes out on the real robotic system, I’d SSH into it, fetch my changes from git, and run the source on the robot. This is great for prototyping, but what happens when you’re ready to ship something?

Going to production opens up a host of new concerns. You have to ask yourself a number of questions, such as:

What is my factory process?
How is the base OS updated so as to remain secure?
How do I update the robotic software I have on top?
How is the device recovered if an update goes wrong?

Let’s take a look at each of these questions in turn.

1. What is your factory process?

In other words: how do you initially deploy Ubuntu, ROS, and your special sauce on a device you’re about to ship? By hand? That doesn’t scale and is error prone. Seed a custom Ubuntu distro with everything? That would actually work, but it’s not easy to maintain. I know some ROS users use Chef or another automation tool for this, but there’s a serious learning curve there. With Ubuntu Core, you’ll end up with a single flashable image that includes the operating system as well as your ROS system, all ready to go. I’ll show you how in this series.

2. How is the base OS updated so as to remain secure?

Perhaps with your complex Chef setup you’ve also enabled automatic security updates. Good thinking, but there are other issues with this (see question 4).

With Ubuntu Core, assuming you use one of the reference devices (this includes just generic amd64) and don’t need to mess with the kernel, the kernel and core snaps are maintained by Canonical for the lifetime of the LTS. They’ll automatically update out of the box, too.

3. How do I update the robotic software I have on top?

A question to be answered first is: How is your software packaged? Is it packaged at all? Are you still running it from source? Do you have it on the ROS build farm? What if it isn’t public? Perhaps you use bloom and then build debs locally, but then you need to maintain your own archive infrastructure. None of these are fun solutions, nor do they lead to an easy upgrade process.

If you package your ROS system as a snap, it includes all its dependencies (including its greater ROS dependencies). If your factory process included placing a snap on the robotic system, then it will automatically update. All you need to do is release the update in the store, and all the deployed devices with automatically update themselves.

4. How is the device recovered if an update goes wrong?

My wife had this happen recently. She brought her laptop to me with a screen showing the grub prompt, unable to boot. After some poking, it turns out that “somehow” (she denies any knowledge of how this may have happened) the computer was forcibly powered off while the kernel was being updated (an automatic security update, no less). I manually told grub to boot the previous kernel, reinstalled the new one, and all was right with the world. But that would have been a lot harder if her laptop was actually a headless, production robot in my warehouse that I didn’t even know ran Linux.

Ubuntu Core is based entirely upon snaps, which are really just squashfs images containing software and their dependencies. This allows them to have some neat properties, one of which is that the upgrade process is as simple as downloading the new image, unmounting the old one, and mounting the new one into place (this is all done automatically, behind the scenes). Another one is that the old snap, once unmounted, is actually kept around, in case something goes wrong with the new snap. If a problem is detected, either by automated health checks, your own health checks within the snap, or in the case of a kernel, a panic, the snap will be reverted to the old one automatically. The atomic nature of snaps also means that either an installation succeeded or failed, there’s no intermediate state like what happened above to my wife’s laptop. If power is pulled in the middle of an update, snapd will remember that it was updating next time it boots, and finish the update.

I hope I’ve given some good justification on why Ubuntu Core is a good fit for production robotics. The next post in this series will be all classic Ubuntu and ROS, discussing a basic prototype with the Turtlebot that we’ll use as an example throughout the rest of the series as we work toward production.

Distributing a ROS system among multiple snaps

Mon, 20 Mar 2017 00:00:00 +0000

One of the key tenets of snaps is that they bundle their dependencies. The fact that they’re self-contained helps their transactional-ness: upgrading or rolling back is essentially just a matter of unmounting one snap and mounting the other. However, historically this was also one of their key downsides: every snap must be standalone. Fortunately, snapd v2.0.10 saw the addition of a content interface that could be used by a producer snap to make its content available for use by a consumer snap. However, that interface was very difficult to utilize when it came to ROS due to ROS’s use of workspaces for both building and running. At long last, support is landing in Snapcraft for building a ROS system that is distributed among multiple snaps, and I wanted to give you a preview of what that will look like.

Why would you want to do that?

Like I said, snaps bundling their dependencies is typically a good thing, and this applies to ROS-based snaps as well. Having an entire ROS system in a single snap that updates transactionally is awesome, useful for most deployment cases, and is generally what I recommend. However, there are some use-cases where this breaks down.

For example, say I’m manufacturing an unmanned aerial vehicle. I want to sell it in such a state that it’s only capable of being piloted via remote control. This is done with a ROS system, which in a simple world would be made up of:

One node to act as a driver for the RC radio
One node to drive the motors
Launch file to connect the two

You get the idea. In addition to that basic platform, I want my users to be able to buy add-on packs. For example, perhaps the vehicle includes a GPS sensor (as well as basic pose sensors). I’d like to sell an add-on pack that adds a very basic “fly here” autopilot, or perhaps a “follow me” mode. That’s another ROS system, perhaps something like:

One node to act as a driver for the GPS
One node (or perhaps a few) to act as a driver for the pose sensors.
One node to plan a path
One node to take the path and turn it into motor controls
A launch file to bring up this system

If we build both of these snaps to be standalone, we quickly run into issues:

Lots of duplication between them, as the autopilot snap will need to include most of the base behavior snap.
They both include (and will try to launch) their own roscore.
The duplicated snaps in each will try to access their respective hardware. This is a race condition: the first one up will win, the second will die. Or, depending on the hardware interface, they’ll both control it. That’s fun.

Using content sharing, we can actually make the autopilot snap depend upon and utilize the base behavior snap.

Alright, what does this look like?

Let’s simplify our previous example into two snaps: a “ros-base” snap that includes the typical stuff: roscore, roslaunch, etc., and a “ros-app” snap that includes packages that actually do something, specifically the classic talker/listener example. A quick reminder: this will only be possible in Snapcraft v2.28 or later. Also note that the example I’m about to walk through is a demo in Snapcraft, in case you want to look at the whole thing.

Create `ros-base`

(Note added after the fact: The “base” terminology here predates snap bases, a concept introduced for core18. If it was writing this post today, I’d use the word “foundation” instead of “base” to avoid the clash. That’s really the only core/base/foundational-type word not yet claimed by the snap ecosystem, heh. Anyway, don’t get confused here.)

To create the base snap, create a snap/snapcraft.yaml file with the following contents:

name: ros-base
version: '1.0'
grade: stable
confinement: strict
summary: ROS Base Snap
description: Contains roscore and basic ROS utilities.

slots:
 # This is how we make a part of this snap readable by other snaps.
 # Consumers will need to access the PYTHONPATH as well as various libs
 # contained in this snap, so share the entire $SNAP, not just the ROS
 # workspace.
 ros-base:
   content: ros-base-v1
   interface: content
   read: [/]

parts:
 ros-base:
   plugin: catkin
   rosdistro: kinetic
   include-roscore: true
   catkin-packages: []

That’s it. Run snapcraft on it, and after a little time you’ll have your base snap (the “provider” snap regarding content sharing). This particular example doesn’t do a whole lot by itself, so let’s move on to our ros-app snap (the “consumer” snap regarding content sharing).

Create `ros-app`

The starting point for ros-app is the current standalone ROS demo. We’ll use the exact same ROS workspace, but we’ll add a few more things and tweak the YAML a bit.

The recommended way to build a “consumer” snap (assuming it has a build-time dependency on the content shared from the “producer” snap, which ros-app does indeed have on ros-base) is to create a tarball of the producer’s staging area, and use it as a part to build the consumer.

Concretely, we can tar up the staging area of ros-base and use it to build ros-app, but then filter it out of the final ros-app snap (so as to not duplicate the contents of ros-base).

So let’s do that now. Get into the directory containing the now-built ros-base snap, tar up its staging area, then move it off into the ros-app area:

$ tar czf ros-base.tar.bz2 stage/
$ mv ros-base.tar.bz2 /path/to/ros-app

Now, in /path/to/ros-app alter the snap/snapcraft.yaml to look something like this:

name: ros-app
version: '1.0'
grade: stable
confinement: strict
summary: ROS App Snap
description: Contains talker/listener ROS packages and a .launch file.

plugs:
 # Mount the content shared from ros-base into $SNAP/ros-base
 ros-base:
   content: ros-base-v1
   interface: content
   target: /ros-base

apps:
 launch-project:
   command: run-system
   plugs: [network, network-bind, ros-base]

parts:
 # The `source` here is the tarred staging area of the ros-base snap.
 ros-base:
   plugin: dump
   source: ros-base.tar.bz2
   # This is only used for building-- filter it out of the final snap.
   prime: [-*]

 # This is mostly unchanged from the standalone ROS example. Notable
 # additions are:
 # - Using Kinetic now (other demo is Indigo)
 # - Specifically not including roscore
 # - Making sure we're building AFTER our underlay
 # - Specifying the build- and run-time paths of the underlay
 ros-app:
   plugin: catkin
   rosdistro: kinetic
   include-roscore: false
   underlay:
     # Build-time location of the underlay
     build-path: $SNAPCRAFT_STAGE/opt/ros/kinetic

     # Run-time location of the underlay
     run-path: $SNAP/ros-base/opt/ros/kinetic
   catkin-packages:
     - talker
     - listener
   after: [ros-base]

 # We can't just use roslaunch now, since that's contained in the
 # underlay. This part will tweak the environment a little to
 # utilize the underlay.
 run-system:
   plugin: dump
   stage: [bin/run-system]
   prime: [bin/run-system]

 # We need to create the $SNAP/ros-base mountpoint for the content
 # being shared.
 mountpoint:
   plugin: nil
   install: mkdir $SNAPCRAFT_PART_INSTALL/ros-base

Other than the ROS workspace in src/ (which remains unchanged from the other demo so we won’t discuss it here), we need to create a bin/run-system executable that looks something like this:

#!/bin/bash

# Would sure be nice if snapd gave us the triplet as well.
# It doesn't, so we'll just create it here. I'm only adding
# support for amd64 here, but one could add this logic for
# any arch they wanted (assuming ROS builds there, of
# course).
case $SNAP_ARCH in
amd64)
   export TRIPLET=x86_64-linux-gnu
   ;;
*)
   echo "Unsupported arch: $SNAP_ARCH"
   exit 1
   ;;
esac

export ROS_BASE=$SNAP/ros-base

# Add ros-base to the PYTHONPATH
export PYTHONPATH=$PYTHONPATH:$ROS_BASE/usr/lib/python2.7/dist-packages

# Add ros-base to LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROS_BASE/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROS_BASE/lib/$TRIPLET
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROS_BASE/usr/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROS_BASE/usr/lib/$TRIPLET

roslaunch listener talk_and_listen.launch

Why is this needed? Because the Catkin plugin can only do so much for you. The ros-base snap includes various python modules and libs outside of its ROS workspace that ros-app needs, so we extend the $PYTHONPATH and $LD_LIBRARY_PATH to utilize them.

From there, it’s as easy as running roslaunch (which by the way is contained in ros-base).

Run snapcraft on this, and after a few minutes (fairly quick since it’s re-using the base’s staging area to build) you’ll have a ros-app snap.

So now I have two ROS snaps. Now what?

You now have your ROS system split between multiple snaps. The first step is to install both snaps:

$ sudo snap install --dangerous ros-base_1.0_amd64.snap
ros-base 1.0 installed
$ sudo snap install --dangerous ros-app_1.0_amd64.snap
ros-app 1.0 installed

Now take a look at snap interfaces:

$ snap interfaces
Slot                     Plug
ros-base:ros-base        -
:alsa                    -
:avahi-observe           -
...
<snip>
...
-                        ros-app:ros-base

You’ll see that ros-base:ros-base is an available slot, and ros-app:ros-base is an available plug. This interface is currently not connected, so content sharing is not yet taking place. Let’s connect them:

$ sudo snap connect ros-app:ros-base ros-base:ros-base

Taking another look at snap interfaces you can see they’re now connected:

$ snap interfaces
Slot                     Plug
ros-base:ros-base        ros-app
:alsa                    -
:avahi-observe           -
...
<snip>

And now you can launch this ROS system you now have distributed between two snaps:

$ ros-app.launch-project
<snip>
NODES
 /
   listener (listener/listener_node)
   talker (talker/talker_node)
<snip>
process[talker-2]: started with pid [10649]
process[listener-3]: started with pid [10650]
[ INFO] [1487121136.757225517]: Hello world 0
[ INFO] [1487121136.860879281]: Hello world 1
[ INFO] [1487121136.960885723]: Hello world 2
[ INFO] [1487121137.057481265]: Hello world 3
[INFO] [1487121137.058298]: I heard Hello world 3
<snip>

Conclusion

Multiple ROS users have mentioned that the fact that a ROS snap must be completely self-contained is a problem. Typically it either interferes with their workflow or their business plan. We’ve heard you! We can’t pretend that the snap world of isolated blobs and the ROS world of workspaces merge perfectly, but the content interface takes a big step toward blending these two worlds, and the new features in Snapcraft’s Catkin plugin hopefully makes it as easy as possible to utilize.

I personally look forward to seeing what you do with this!

ROS on arm64 with Ubuntu Core

Thu, 26 Jan 2017 00:00:00 +0000

Previous Robot Operating System (ROS) releases only supported i386, amd64, and armhf. I even tried building ROS Indigo from source for arm64 about a year ago, but ran into dependency issues with a missing sbcl. Well, with surprisingly little fanfare, ROS Kinetic was released with support for arm64 in their prebuilt archive! I thought it might be time to take it for a spin with Ubuntu Core and its arm64 reference board, the DragonBoard 410c, and thought I’d take you along for the ride.

Step 1: Install Ubuntu Core

The first order of business is to install Ubuntu Core on the DragonBoard. Just follow the documentation. I want to mention two things for this step, though:

I’ve never had good luck with wifi on the DragonBoard, it seems incredibly unstable. I recommend not even bothering with it and using a USB to ethernet adapter, since with Ubuntu Core at least your first login must be over SSH.
There’s a bug that causes the first boot wizard to take quite some time between entering your SSO password and finishing. Don’t worry, just leave it alone, it’ll finish (mine took about 7 minutes).

Step 2: Make sure it’s up-to-date

SSH into the machine (or if you set a password, login locally, it doesn’t matter), and run the following command to ensure everything is completely up-to-date:

$ sudo snap refresh

If it updated, you may need to reboot. Go ahead and do that now, we’ll wait.

Step 3: Install Snapcraft

You may or may not be familiar with Ubuntu Core, but the first thing people typically notice is that it doesn’t use Debian packages (.debs), it uses snaps (read up on them, they’re pretty awesome). However, in this case, they don’t give us what we need, which is a development environment. We need to build a ROS workspace into a snap, which means we’ll need to utilize ROS’s archive as well as the snap packaging tool, snapcraft. All of these are available as .debs. Fortunately, the environment we want is available as a snap:

$ snap install classic --edge --devmode
$ sudo classic
<unpacking stuff... snip>
(classic)kyrofa@localhost:~$

The (classic) prompt modifier tells you that you’re now in a classic shell. Now you can do familiar things such as update the package index and install Snapcraft, both of which you should do now:

(classic)kyrofa@localhost:~$ sudo apt update
(classic)kyrofa@localhost:~$ sudo apt install snapcraft

Step 4: Build your snap

Probably the most simple ROS workspace you can imagine is a single talker and a single listener. Snapcraft has exactly that as one of its demos, so for this example we’ll just use that (you might consider reading that demo’s walkthrough). First though, that means fetching the Snapcraft source code:

(classic)kyrofa@localhost:~$ sudo apt install git
(classic)kyrofa@localhost:~$ git clone --branch 2.28 https://github.com/snapcore/snapcraft
(classic)kyrofa@localhost:~$ cd snapcraft/demos/ros

The ROS demo in question will by default actually build for Indigo. But you’re interested in arm64! You can’t use Indigo. You need something newer… something shinier, and perhaps less purple. You need to use Kinetic. You tell Snapcraft this by utilizing the rosdistro property. Open up the snapcraft.yaml and edit the appropriate section to look like this:

[...]
parts:
 ros-project:
   plugin: catkin
   source: .
   rosdistro: kinetic
   catkin-packages:
     - talker
     - listener
   include-roscore: true

Save that file and exit. Finally, it’s time to run snapcraft and watch that workspace get built into a snap (this will take a bit, the DragonBoard is not a blazing workhorse; if you want something faster use the snap builders):

(classic)kyrofa@localhost:~/snapcraft/demos/ros$ snapcraft
<snip pulling, building, staging>
Priming ros-project
Snapping 'ros-example'
Snapped ros-example_1.0_arm64.snap

In the end you have a snap. You should exit out of the classic shell at this point (e.g. ctrl+d).

Step 5: Install and run it!

Now it’s time to install the snap you just built. Even though you’ve exited the classic shell, your $HOME remained the same, so you can install the snap like so:

$ cd snapcraft/demos/ros
$ sudo snap install --dangerous ros-example_1.0_arm64.snap

(remember the --dangerous flag is because you just built the snap locally, so snapd can’t verify its publisher. You’re telling it that’s okay.)

Finally, run the application contained within the snap (it’ll launch the talker/listener system):

$ ros-example.launch-project
<snip>
SUMMARY
========
PARAMETERS
 * /rosdistro: kinetic
 * /rosversion: 1.12.6
NODES
 /
   listener (listener/listener_node)
   talker (talker/talker_node)

auto-starting new master
<snip>
process[talker-2]: started with pid [25827]
process[listener-3]: started with pid [25828]
[ INFO] [1485394763.340461416]: Hello world 0
[ INFO] [1485394763.440354547]: Hello world 1
[ INFO] [1485394763.540334917]: Hello world 2
[ INFO] [1485394763.640330599]: Hello world 3
[ INFO] [1485394763.740335917]: Hello world 4
[ INFO] [1485394763.840366912]: Hello world 5
[ INFO] [1485394763.940342594]: Hello world 6
[ INFO] [1485394764.040321141]: Hello world 7
[ INFO] [1485394764.140323334]: Hello world 8
[ INFO] [1485394764.240328548]: Hello world 9
[ INFO] [1485394764.340319074]: Hello world 10
[INFO] [1485394764.341486]: I heard Hello world 10
[ INFO] [1485394764.440333194]: Hello world 11
[INFO] [1485394764.441476]: I heard Hello world 11
[ INFO] [1485394764.540333772]: Hello world 12
[INFO] [1485394764.541450]: I heard Hello world 12

Conclusion

I haven’t yet pushed ROS’s arm64 support very hard, but I’m thoroughly pleased that support is present. Particularly paired with snaps and Ubuntu Core, I think this opens the door to a lot of amazing robotic possibilities.

Snap updates: automatic rollbacks

Mon, 12 Dec 2016 00:00:00 +0000

When researching snaps, one of the main advantages everyone talks about is the fact that they’re transactionally updated. That is, an upgrade either succeeds or fails, it doesn’t leave the snap in a broken state. If you have snap “A” installed and an update for it is released, it’ll automatically update. If that update is somehow broken, the snap will roll back to the previously-working revision. However, no one has really talked about how that works. How does a snap know that an update is broken? That’s what I want to talk about today.

Snapd won’t check the health of your snap for you, as it doesn’t know enough about your snap to perform a decent health check. However, it’s pretty easy to check the health yourself.

Snapd recently gained support for “hooks”, which is a way for snapd to notify individual snaps about various events. There’s one hook in particular that supports configuration. It’s beyond the scope of this post to walk through configuration itself, but there’s something important about this hook that’s worth knowing: it runs upon initial install, and it runs when the snap is updated, after the snap’s services are started. This makes it a decent place to perform a health check. (Update: there are probably better hooks for this now, such as the pre-refresh hook, but that doesn’t change how to use them)

I’d like to explain with an example, but first we need to make sure we’re on the same page. This support is pretty new as of this writing, and you need to be running the right version of snapd.

Prerequisites

First of all, verify that you’re at least on Xenial running version 2.17.1ubuntu1:

$ snap --version
snap   2.17.1ubuntu1
snapd  2.17.1ubuntu1
series 16
ubuntu 16.04

Also make sure your core snap (it could also be called ubuntu-core, that’s okay) is up to date (hook support was only recently added):

$ sudo snap refresh

Now get on with it

Now that we’re on the same page, let’s continue with our example. Hooks are simply executables contained within the meta/hooks/ directory inside the snap. Snapcraft doesn’t yet have explicit support for hooks (Update: it does now), but we can bend it to our will for this example.

Let’s first create a directory for our new snap, and create the snapcraft.yaml for it:

$ mkdir rollback-test
$ cd rollback-test && snapcraft init
Created snapcraft.yaml.
Edit the file to your liking or run `snapcraft` to get started

Edit that snapcraft.yaml and make it look like this:

name: rollback-test
version: '1.0'
summary: Snap to test rollback functionality.
description: You heard me.

grade: stable
confinement: strict

parts:
 hook:
   plugin: dump
   source: .

Now create a new file: meta/hooks/configure. The tree should look like this when you’re done:

$ tree
.
├── meta
│  └── hooks
│      └── configure
└── snapcraft.yaml

Edit the meta/hooks/configure file and make it look like this:

#!/bin/sh
echo "This will succeed."
exit 0

Make that file executable (hooks must be executable):

$ chmod a+x meta/hooks/configure

And finally, create the snap:

$ snapcraft
Preparing to pull hook
Pulling hook
Preparing to build hook
Building hook
Staging hook
Priming hook
Snapping 'rollback-test' |
Snapped rollback-test_1.0_amd64.snap

Now you have a snap containing a configure hook that does exactly nothing other than exiting successfully. Install the snap:

$ sudo snap install --dangerous rollback-test_1.0_amd64.snap
rollback-test 1.0 installed

$ snap list
Name          Version Rev Developer Notes
core          16.04.1 641 canonical -
rollback-test 1.0     x1            -

Take a careful look at the revision for the snap we just installed: x1. As you may know, revision numbers are assigned by the store. However, this snap didn’t come from the store: we just created and installed it locally. Snapd knows working with revisions can be handy here, so it makes one up for you: The “x” indicates that it was a local snap, and the “1” indicates that it’s been installed once. You could actually install it again over the top of it and get the revision to increment:

$ sudo snap install --dangerous rollback-test_1.0_amd64.snap
rollback-test 1.0 installed

$ snap list
Name          Version Rev Developer Notes
core          16.04.1 641 canonical -
rollback-test 1.0     x2            -

You can see this works like an update from the store. You’ll notice that you now have two revisions on your disk (with only one active, of course):

$ ls -l /snap/rollback-test/
total 0
lrwxrwxrwx 1 root root 2 Dec 12 12:47 current -> x2
drwxrwxr-x 3 root root 27 Dec 12 12:42 x1
drwxrwxr-x 3 root root 27 Dec 12 12:42 x2

Alright, let’s get to the cool part. You probably didn’t notice it, but your configure hook did actually run as part of the build step. Check this out:

$ snap changes
ID  Status Spawn  Ready  Summary
7   Done   <snip> <snip> Install "rollback-test" snap [...]
8   Done   <snip> <snip> Install "rollback-test" snap [...]

I’ve cleaned the output up a little, but you get the idea. You see two changes there corresponding to the two back-to-back installs we did. Let’s take a closer look at that most recent change:

$ snap change 8
Status Spawn   Ready Summary
Done   <snip> <snip> Prepare snap "/tmp/snapd-sideload-pkg-798830678" (unset)
Done   <snip> <snip> Mount snap "rollback-test" (unset)
Done   <snip> <snip> Stop snap "rollback-test" services
Done   <snip> <snip> Make current revision for snap "rollback-test" unavailable
Done   <snip> <snip> Copy snap "rollback-test" data
Done   <snip> <snip> Setup snap "rollback-test" (unset) security profiles
Done   <snip> <snip> Make snap "rollback-test" (unset) available to the system
Done   <snip> <snip> Start snap "rollback-test" (unset) services
Done   <snip> <snip> Clean up "rollback-test" (unset) install
Done   <snip> <snip> Run configure hook of "rollback-test" snap if present

Notice that last step: “Run configure hook […] if present”. So you can see that the snap’s services start up, and then the configure hook runs. This means that your configure hook can make sure your services are running correctly, etc. But how would it tell snapd if something is wrong, i.e. the snap needs to be rolled back? That’s easy: fail.

Alter your meta/hooks/configure hook and make it look like this:

#!/bin/sh
echo "This will fail."
exit 1

Notice we’re exiting non-zero here– the hook will fail. Rebuild your snap so that it includes the new hook:

$ snapcraft clean
Cleaning priming area for hook 
Cleaning staging area for hook 
Cleaning build for hook 
Cleaning pulled source for hook 
Cleaning up snapping area

$ snapcraft
Preparing to pull hook 
Pulling hook 
Preparing to build hook 
Building hook 
Staging hook 
Priming hook 
Snapping 'rollback-test' |
Snapped rollback-test_1.0_amd64.snap

Now try to install it again:

$ sudo snap install --dangerous rollback-test_1.0_amd64.snap
error: cannot perform the following tasks:
- Run configure hook of "rollback-test" snap if present (This will fail.)

Neat, huh? The failing hook actually made the installation fail. What does that entail, then? What is the state of the snap? See for yourself:

$ snap list
Name          Version Rev Developer Notes
core          16.04.1 641 canonical -
rollback-test 1.0     x2            -

If the installation succeeded, we’d be on revision x3. However, the installation failed, so snapd took action: it went back to the revision it knew was previously working: x2.

Conclusion

I obviously walked through this example locally without touching the store, but the exact same principle applies if the snap is coming from the store and the update is caused automatically. If you published the first snap we made, and then published the second version of it as an update, the update would have reverted just like it did in our example. Give it a try yourself!

Admittedly this example was simplistic. However, hopefully you can see how this would expand to a snap-wide health check. Since the configure hook runs after the snap’s services are fired up, you can communicate with them, verify they’re running correctly, and exit non-zero if you detect a problem. This would allow snapd to take remedial action and undo the upgrade.

Ubuntu Core on VirtualBox

Mon, 21 Nov 2016 00:00:00 +0000

If you read the official Ubuntu Core docs, you’ll see that the recommended way to run Ubuntu Core virtually is with KVM. However, it’s probably even easier to use it in VirtualBox, and I’ve been getting some questions about that, so I wanted to devote a quick post to it.

Prerequisites:

A Linux machine (using Ubuntu 16.04 here). This guide will work on Windows as well, but the command invocations are a bit different and I don’t have a machine on which I can verify them before suggesting them.
VirtualBox installed and running successfully.

Guide

First of all, download the appropriate Ubuntu Core 16 image from here. I’m running 64-bit, so that’s the one I selected. Download it, and uncompress it:

$ wget http://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz
$ unxz ubuntu-core-16-amd64.img.xz

You now have a raw disk image. To make it usable within VirtualBox, convert it to a VDI:

$ VBoxManage convertdd ubuntu-core-16-amd64.img ubuntu-core-16-amd64.vdi --format VDI

You now have a VDI that is usable, but is small enough that it’s not very useful. Ubuntu Core will resize itself to fit in whichever disk you give it, so let’s make this disk a bit bigger (remember this is a VDI, so it won’t take any more space), I selected 20GB:

$ VBoxManage modifyhd ubuntu-core-16-amd64.vdi --resize 20480

Finally, create a new machine within VirtualBox. When it comes to the “Hard disk” section, select “Use an existing virtual hard disk file,” and browse to the ubuntu-core-16-amd64.vdi file you just created. Create the machine. I suggest you change the network interface to be bridged so you can actually SSH to the thing, and then fire it up.

The rest of the directions are the same as KVM. Make sure you have an Ubuntu SSO account with associated SSH keys. As the machine comes up for the first time, eventually it’ll ask you to “Press enter to configure.”

This will immediately jump to a screen asking you to configure your network. If you’re using DHCP, just hit “Done” and move on, where it’ll ask for your email address (it wants the one associated with your Ubuntu SSO account).

From here it’ll look up your account in the store, and allow you to login to the VM via the SSH keys you have in your SSO account.

Enjoy your Ubuntu Core experience!

Snapping Nextcloud: Nextcloud itself

Sat, 25 Jun 2016 00:00:00 +0000

In the previous post of this series, we discussed the process of and lessons learned from building Redis for the Nextcloud snap. That means we’ve covered all the major pieces except for Nextcloud itself, so for this post that’s what we’ll focus on getting that into the snap.

As I mentioned previously, Nextcloud is a PHP application. As such, it’s not difficult to get working with Apache or MySQL. However, it was written in such a way to make its deployment as easy as possible (e.g. untar it ), which led to a complication in the way I chose to package it.

Lesson 7: Read-only is both a blessing and a curse

Snaps are squashfs images, which by definition means they’re read-only. I thought this was a pretty big plus for Nextcloud: even if it was compromised, the attacker couldn’t modify any part of the application. However, there’s a downside here, too: Nextcloud has an admin interfaces that allows a user to alter its configuration. However, that configuration isn’t stored in the database: it’s stored within a config.php file which lives inside the application itself (in the config/ directory). Well, in a snap, that directory (and config file) is completely read-only. That means you can’t even configure email notifications, since those settings are stored in that config file.

This meant I had to choose one of two paths:

Store the entire application in a readable location (i.e. $SNAP_DATA somewhere)
Store the config in a writable location, separate from the read-only application

(1) would mean duplicating the application, because during the installation process I’d have to copy the application from the snap to the writable area, which that means there are two copies of the application. Nextcloud is not small (it’s about 90MB all by itself), so that seems wasteful. It also defeats what I consider to be an advantage of the snap: the fact that it is read-only in the first place! So (2) seemed to be the obvious win from my perspective.

However, while Nextcloud supports apps and data being hosted outside the main application, it does not support hosting the config outside of the main application. This meant I had to fork Nextcloud and apply a patch adding that support, and use that fork in the snap. I tried to move that patch upstream back when it was ownCloud, but I was unfortunately unable to gain any traction. Perhaps I should try again with Nextcloud. (Update: This was upstreamed in Nextcloud v10; the snap no longer uses a fork).

Speaking of hosting data outside the main application, that led to:

Lesson 8: Versioned data directories don’t cover all use-cases

If you read the available documentation and take a look at the environment in which snap applications and services execute, you’ll note that there are two writable areas available to all snaps:

$SNAP_DATA: system-wide writable area.
$SNAP_USER_DATA: user-specific writable area.

Both of these areas are contained within versioned directories, i.e. the data contained within them are version-specific. What do I mean by this? Let me explain with an example:

Say you install version 1 of snap “foo”. Say it writes files into both $SNAP_DATA and $SNAP_USER_DATA. Now an update to “foo” is released: version 2. As part of the upgrade process, snapd copies version 1’s data directories into version 2’s data directories, and then starts version 2 of “foo”. Version 2 fires up and is running on the data that was last touched by version 1. Maybe it runs some migrations on it.

What if those migrations fail? Well, then snapd can revert (or rollback) version 2, and start running version 1 again. Since snapd made a copy of version 1’s data before the migration occurred, version 1 is fired up running on the data exactly as it left it rather than on the data where the failed migration occurred.

That’s lovely, but it’s problematic when it comes to applications like Nextcloud. My personal Nextcloud collection of data is several terabytes. Automatically copying that data every time an upgrade occurrs just in case Nextcloud needs to rollback is not a scalable workflow unless you have some serious hard drive space. This led to the development of a new feature in snapd, a data directory that was common across all versions of a given snap. This feature is not yet documented because those directories are not yet represented by an environment variable, but the Nextcloud snap makes use of them anyway for storing its raw data. (Update: They are now: $SNAP_COMMON and $SNAP_USER_COMMON.)

Beyond those issues, Nextcloud was pretty straight-forward to get up and running in the snap. It provides the ability to pre-configure some settings which allowed me to make the initial setup nice and painless (e.g. pre-configure the database, etc.). I wish they exposed more things that way though– I’d also like to set the background jobs to using cron instead of ajax, but you can only do that after the application is installed (i.e. an admin user is created), so the snap is still using ajax. Maybe I’ll try proposing a change (Update: the cron type is now automatically changed once the cron job runs).

But all in all, I’m quite happy with the snap.

Snapping Nextcloud: The memcache

Fri, 24 Jun 2016 00:00:00 +0000

In the previous post of this series, we discussed the process of and lessons learned from building MySQL for the Nextcloud snap. The last piece of infrastructure we need to introduce is the memcache, for which we’ll use Redis. Nextcloud supports a number of memcache backends; I chose Redis because it can provide both a local cache as well as file locking, so for this post that’s what we’ll focus on getting into the snap.

Since I was already building all the other components from source, I decided to do the same for Redis. I hit Lesson 4 again (DESTIR is not standard) so I wrote a dead simple local plugin for it to call make install PREFIX=<installdir>, but hardly anything worth mentioning. However, I encountered what was very nearly a deal-breaking problem:

Lesson 6: Not all projects support environment variables in their configuration files

Redis is pretty neat in that it’ll happily run using a default configuration, but I didn’t like its default configuration. I didn’t want to listen on the network, I wanted to use unix sockets. I wanted to log and dump the database into $SNAP_DATA/redis/. You get the idea. But Redis doesn’t evaluate environment variables in its config file, so it was throwing errors complaining about not being able to open the file “${SNAP_DATA}/redis/redis.log”, etc.

So I took a careful look at its help page, and noticed this little nugget:

$ ./redis-server -h
Usage: <snip>
      ./redis-server - (read config from stdin)
      <snip>

So I decided to rewrite the config file on the fly, evaluating the environment variables myself, and hand it to its stdin:

#!/bin/sh

envsubst < "$SNAP/config/redis/redis.conf" | redis-server -

All that was left was to update the Nextcloud configuration to use Redis as the backend for the memcache and file locking, and voila!

The next (and final) post in the series will discuss the Nextcloud web application itself.

Snapping Nextcloud: MySQL

Thu, 23 Jun 2016 00:00:00 +0000

In the previous post of this series, we discussed the process of and lessons learned from building PHP 7 for the Nextcloud snap. Nextcloud supports a number of databases, but they recommend MySQL (the v9 docs that I used when making this decision are lost to the dust of time, but MySQL is still the recommended option in v21), so for this post that’s what we’ll focus on getting into the snap.

MySQL is included in the Ubuntu archives, and honestly something like this should work fine:

# ...
   parts:
       mysql:
           plugin: nil
           stage-packages: [mysql-server]
# ...

But it doesn’t.

Lesson 5: seccomp denials are not recoverable

I find that a lot of the people I talk to think of snaps as containers (they often compare them to docker). Yes, snaps are self-contained. Yes, snaps are isolated. But in my opinion, no, snaps are not containers. Admittedly they utilize some of the same underlying technology, but they aren’t in a chroot of any kind, they don’t get their own network namespace, etc. They’re simply very confined applications, and calling them containers is confusing to those who are familiar with other kinds of containers. It’s not my goal to discuss the entire security story behind snaps (check the whitepaper out if you’re interested), but you need to know a part of it for this lesson.

One of the confinement technologies used in snaps are seccomp filters, which allows one to provide a whitelist of system calls for a given application environment. This is one of the things handled by the concept of “interfaces”. When you specify interfaces for a given application, you expand the whitelist of syscalls it’s allowed to make (among other things). When configuring the whitelist, one specifies what should happen if a disallowed syscall is made. There are several options:

Kill the offending application immediately.
Notify the offending application via a signal.
Return an errno from the syscall.
Notify a tracer of the denial.
Allow the syscall.

Only two of these options were valid for the snap use-case: either kill the offending application, or return an errno from the syscall. The other options were I believe either racy or impossible to implement in a generic manner. The obvious winner there is to return an errno– let the application handle its own denial, maybe it can recover. However, therein lies a seccomp filters shortcoming: if one uses the errno option, none of the denials get logged! So the security team chose the last remaining option: kill the offending application immediately.

It’s terribly unfriendly to the application, but at least you get a warning about it in the syslog so you know what the issue is (and snappy-debug will make recommendations for you). It’s worth noting here that the security team is working with upstreams to enable logging for errno, so we hope to use that instead once it’s available. (Update: that work has completed, and snapd now uses errno instead of killing offending applications. Yes, it makes this lesson useless, but I’m leaving it intact for historical reasons.)

Anyway, MySQL attempts to control the priority of its page cleaner threads with a setpriority() syscall, which is not contained within any of snapd’s interfaces. MySQL makes this call, and rightly handles its failure, but snapd doesn’t give it a chance to handle the failure: it kills it immediately. The only way around this was to fork MySQL, apply a patch to make that syscall optional, and build the MySQL part from source. I’m trying to move that patch upstream, but the MySQL folks seem reluctant to accept it since they don’t understand why snapd doesn’t use errno instead of kill. I can’t really blame them, so I’ll probably be maintaining this fork until snapd uses errno.

Beyond that issue, MySQL was pretty easy to get working. It runs based on config files which can contain environment variables, so I just ship custom versions in the snap.

The next post in the series will discuss the memcache.

Snapping Nextcloud: PHP 7

Wed, 22 Jun 2016 00:00:00 +0000

In the previous post of this series, we discussed the process of and lessons learned from building Apache for the Nextcloud snap. Nextcloud is a web application written in PHP, so for this post we’ll focus on getting PHP into the snap.

PHP 7, like Apache, is included in the Ubuntu archives. However, I was planning on using the PHP Apache module, and since I was building Apache from source, I figured there’d be a lot of hoops for me to jump through if I tried to shoehorn the .deb into working for me. Since building Apache modules is really pretty straight-forward, I just went straight for building the PHP Apache module from source, as a new part in the snapcraft.yaml. Of course, it uses the various Apache headers and tools for building modules, which is another part in the YAML, and that relationship is specified by using the after keyword in snapcraft, something like this:

# ...
parts:
   apache:
       plugin: apache
       # ...
   php:
       plugin: autotools
       after: [apache]
       # ...
# ...

Using the after keyword like this tells snapcraft that the php part depends upon the apache part, and apache should be staged before php is built. However, going down this path quickly led me to:

Lesson 3: Project build systems aren’t really prepared for snapcraft

This is pretty similar to Lesson 2 in the previous post in that the problem is the same, but the effect is different. As I mentioned in the previous post, Apache writes its prefix everywhere, including its tools for building modules (e.g. apxs). This has the effect that modules built using apxs get installed into that same prefix. There doesn’t seem to be any way to redirect it elsewhere.

This is a problem because snapcraft was designed for parts to be completely independent. They may depend upon another part if it’s already in the staging area, but they cannot alter each others files or directories. We could work around this fact by rewriting the prefix in apxs to point to the staging area, but then we hit another problem: snapcraft was also designed for each part to go through an explicit lifecycle: pull, build, stage, and prime (followed by the final snap). Having parts install directly into the staging area means they essentially skip the build step, which throws snapcraft’s state tracking off and makes it unhappy. Future snapcraft developments may make this easier, but we’re not there yet.

The solution to this problem without fighting either snapcraft or apxs too much is to simply build Apache modules as part of the Apache plugin, thus keeping it within the same part (you’ll notice the third-party-modules config option within the plugin). When developing that code, I quickly ran into another issue:

Lesson 4: DESTDIR is not standard

I’ve mentioned the concept of relocatable builds a few times: snapcraft configures projects to install into location X, installs into location X, and snaps run from location Y. That’s actually not quite true. It’s actually more like: configure to install into location X, install into location Y, and run from location Z. What do I mean by this? Well, snapcraft builds each part within a parts/<partname>/build/ subdirectory, and installs them into parts/<partname>/install/. It does this even though the project might be configured to install into /usr/bin/, and it uses DESTDIR to do that.

For example, if you look at snapcraft’s make plugin, you’ll see that it runs make followed by make install DESTDIR=<installdir>. Many build systems respect this as a de-facto standard, and essentially install their artifacts into DESTDIR/<prefix>.

However, PHP does not. The PHP build system completely ignores DESTDIR and tries to install to the configured prefix, in the root filesystem somewhere, and dies because it doesn’t have permission to do so. It wouldn’t get into the snap that way anyway! This actually led to the creation of a new snapcraft feature for the autotools plugin: the install-via option. Using that option it’s possible to make the autotools plugin not use DESTIR, and instead use a series of incantations like this:

$ ./configure --prefix=parts/<partname>/install
$ make
$ make install

Note the lack of DESTDIR there; it’s just installing with the prefix instead. This allowed PHP to install correctly into the snap.

Note that I’m actually currently in the process of moving to PHP FPM instead of the Apache module, for various reasons, but these lessons still apply.

The next post in the series will discuss MySQL.

Snapping Nextcloud: The web server

Tue, 21 Jun 2016 00:00:00 +0000

The backbone of any web application is of course the web server, so that’s where I started when snapping Nextcloud. I went with Apache as opposed to Nginx for two reasons: 1) Apache is recommended by Nextcloud (the v9 docs that I used when making this decision are lost to the dust of time, but Apache is still the recommended option in v21), and 2) I’m much more familiar with Apache than Nginx.

Apache is in the Ubuntu archives, so it seemed that a reasonable starting point was to simply include it via stage-packages, something like this:

# ...
parts:
   apache:
       plugin: nil
       stage-packages: [apache2]
# ...

I quickly hit pain going this route, which leads me to:

Lesson 1: Debian packages only get you so far

Debian uses a different layout for Apache than the default. In particular, the method of determining which configuration, module, and/or site is enabled is not done directly in the main configuration file but via symbolic links. That makes it pretty easy to use, which I assume is the reason it was done, but the problem is this: those symlinks are setup in the Debian package’s postinst, not distributed in the package itself. In simplistic terms, that essentially means the steps taken to install the package are the following:

Unpack the package, placing things where they need to go.
Run the postinst script, which calls familiar commands like a2enmod etc. to setup the default modules and so on.

In snapcraft, the stage-packages are only unpacked– the postinst script never gets run. I’m sure there are various good reasons for this, but the obvious one is this: the stage-packages are unpacked into a build directory, not onto the system. Many postinst scripts have hard-coded paths which wouldn’t work in this case. Regardless, the side effect is that, if I wanted to make use of Ubuntu’s Apache package, I’d need to recreate all those symlinks myself. That, along with a few other reasons, led me toward compiling Apache from source instead. Where I immediately learned:

Lesson 2: Project build systems aren’t really prepared for snappy

The design of snaps pushes the envelope in a few places, so it’s understandable to see some incompatibilities. The one I’m talking about here is the concept of a relocatable build. Most projects typically use their build systems (autotools, cmake, etc.) to configure a project to install into location X, install the project from location X, and the project runs from location X. This works fine for Debian packaging, where it can unpack files into, say, /usr/bin/. However, snaps are confined to their own area in /snap/<snapname>/<version>/, so a build system that might be configured for installing into /usr/bin/ will actually end up running out of /snap/<snapname>/<version>/usr/bin/ (for example). It means the project is configured to install into location X, is installed into location X, and runs from location Y.

This isn’t the fault of autotools or cmake, necessarily. It’s perfectly possible to have a relocatable project that uses those tools, but projects historically haven’t needed to bother with such things. A great example of this is Apache. It’s an autotools project, and when you give it a prefix while configuring it, it writes that prefix everywhere. Config files, hard-coded in scripts, you name it. These paths have to be valid at build- and run-time. That’s great if you’re installing and running out of the same place… but with snaps, you aren’t.

Thankfully, Apache is one of the most configurable projects I’ve ever seen. Everything can be set via a config file and/or command-line switches. I essentially had to build Apache in two phases: install Apache and its modules, then clean it up. I did this with a local snapcraft plugin, which gave me full control over the build process. I installed Apache and any extra modules that were requested, and then did one massive search/replace to enable it to be run from a snap. I could probably simplify this further by shipping a custom Apache configuration in another snapcraft part, but this works for now.

The next post in this series will discuss PHP.

Installing Nextcloud can be a snap

Mon, 20 Jun 2016 00:00:00 +0000

I’ve been an ownCloud user for years, and I recently moved to using Nextcloud, which is based upon ownCloud. Both my wife and I use it to sync our calendars and contacts across all our devices, and also use it to sync our documents, music, pictures, and videos. It caters to my paranoid nature– the idea that all that information is being saved on my own in-house server instead of someone else’s greatly pleases me.

I’ve been on the snappy team for a little while now, and I still find one of the best ways to gain understanding about the system as well as prove out new functionality and find bugs is to simply create snaps. Snaps provide a particularly easy way to install software, and Nextcloud is a particularly complex system that requires a bit of work and knowledge to setup. Wouldn’t it be cool if you could install Nextcloud and it came all pre-configured and ready to go? Check this out:

$ sudo snap install nextcloud
120.87 MB / 120.87 MB [==================] 100.00 % 2.78 MB/s 
Name      Version     Rev Developer Notes
nextcloud 9.0.50snap2 5   canonical -

Visit localhost (or the device’s external IP) in your browser and you’ll see:

Create an admin account by entering a desired username and password there, and you have a working Nextcloud installation. It’s using Apache, PHP 7, MySQL for the database, and redis for the cache, all installed and setup with a few keystrokes. It even includes mDNS for network discovery in case you installed it on a headless box. It’s also completely confined, so if (heaven forbid) Nextcloud was compromised, the attacker wouldn’t be able to walk all over your system.

There are, of course, limitations.

The target audience: this snap is really for those users who just want to get Nextcloud up and running with recommended settings. The side effect of getting all this preconfiguration and easy upgrades is that it’s not very customizable. If you’re wanting to run Nextcloud behind a load balancer connected to a MySQL cluster, you’re probably better off setting it up yourself.
HTTPS is not yet supported. I don’t recommend the use of this snap on production systems until it is. I’m working on on integrating Let’s Encrypt into the snap, but I want to make it as user-friendly as possible, which will take a little time. (Update: HTTPS support was added in 9.0.53snap2, released on 08/22/16.)
Overall still a work in progress, and probably has unknown rough edges.

I’m pleased with how far it’s come, but getting there wasn’t easy. In the following series of blog posts, I’d like to share some of the stumbling blocks I hit along the way (as well as how I solved them). I hope that doing so will help you snap up your complex system.

The next post in this series will discuss the web server.

Building your snap on-device? There's a better way

Mon, 09 May 2016 00:00:00 +0000

I’m currently the maintainer of the ownCloud snap. One of the primary target devices for this snap is a Raspberry Pi 2, which is of course armhf. Snapcraft solves a lot of problems when building snaps, but one issue it doesn’t solve is cross-building (i.e. building on one architecture while targeting another). Back in Snappy Ubuntu Core 15.04, every ownCloud release I made for armhf was built on the Raspberry Pi 2. What a painful experience that was– it needed more RAM than the rpi2 had (so I needed to enable swap), and it took literally all day to build.

Nowadays there’s a much, much better way. Snapcraft still doesn’t fully support cross-building, but the Launchpad team recently introduced snap builders for several different architectures that run on very high-end hardware– available for free, for everyone! The only downside is that they aren’t very well documented. Maybe I can help with that.

What are you talking about?

The Launchpad snap builders are server instances spun up on demand with the sole purpose of building snaps using Snapcraft in a clean environment. They have servers for several different architectures– at the time of this writing, amd64, x86, arm64, armhf, and ppc64. I strongly encourage you to use them.

How do I use them?

First of all, you need a Launchpad account. Once you have that, you just need your code in a branch on Launchpad. You can do that by creating a project and associating you code with it, or by pushing to the +junk pseudo project. I’ll walk you through using a project, but the +junk project is really no different, it just cuts down a step. Launchpad supports both bzr and git– I’ll use git in this example.

Step 1: Register a project

Go ahead and create a project with which the code will be associated. I called mine “owncloud-snap”.

Step 2: Push the code

Since the ownCloud snap is hosted in GitHub, I just needed to add Launchpad as a remote and push up my branch:

$ git remote add lp git+ssh://kyrofa@git.launchpad.net/owncloud-snap
$ git push lp master

Now you should see your new branch in the Code tab of your project (e.g. here’s ownCloud’s).

Step 3: Create a snap associated with the branch you just pushed

Select the new branch in the Code tab. One of the options you’ll see is “Create snap package.”

Select that, and you’ll be able to name your snap, and specify what distribution to use.

There are many options here, but you either want to select Vivid (if targeting Snappy Ubuntu Core 15.04) or Xenial (if targeting Snappy Ubuntu Core 16)– ignore the rest. You can even opt to automatically upload completed snaps to the store here if you like. When you’re happy the settings, select “Create snap package.”

Step 4: Make sure it’s using the architectures you want.

After completing step 3 you should be on the “Snap package information” page for the snap you just created. By default (at least at the time of this writing), snaps are only built for amd64 and x86.

On the right, select “Edit snap package” and you’ll be able to select other architectures for which the snap will build. The store will accept any of these architectures, but you’ll see I’m only building for the four main ones.

Step 5: Build it!

Now that it’s configured for the architectures you want, get back on the “Snap package information” page and select “Request builds.”

Select the architectures for which you’d like to build (note the options here are limited to what you selected in step 4), as well as the pocket to use for building the snap. Pockets are essentially “sections” of the archive. The Release pocket reflects things as they were when that version of Ubuntu was released. Updates (the default) reflects all the updates that the version of Ubuntu has received since release. Security reflects security updates that the version has received since release… you get the idea. I always use Updates unless there’s a good reason not to, particularly because that’s where the Snapcraft updates land. Anyway, once you select your options, go ahead and hit “Request builds.”

Step 6: Hang out for a bit. But definitely not a day.

Sit back and relax while Launchpad spins up some instances and runs Snapcraft for you! For the ownCloud snap, which as I said took an entire day to build on the rpi2, Launchpad cranked out a snap for amd64, x86, arm64 and armhf in about 40 minutes. Total. Oh yes.

Step 7: Grab the built snap and use it.

Once a build has completed, it’ll give you a link to the built snap.

You can download it from there and sideload it right away, or upload it to the store and publish it (assuming you didn’t request Launchpad to do that for you)!

Limitations

Now, I love the Launchpad builders, but there are a few limitations of which you should be aware.

Internet access is only available in the pull step. If your build step for instance tries to hit the internet, it’ll fail. Refactor your snapcraft.yaml if necessary. (Update: This is no longer true, you should be able to hit the internet in any step now).
The builders are still beta. Sometimes proxy tokens time out, or other infrastructure-related issues occur. You’ll get an email if the builds fail with a link to the build log– take a look, and if it looks infrastructure-related, just request a new build.

If you run into any other issues, or have questions, hop in the #launchpad channel in Freenode and ask away!

That's one Snappy MOOS

Wed, 10 Feb 2016 00:00:00 +0000

So you’re a roboticist, looking at Ubuntu Core and Snappy, trying to decide if they’re a good fit for your project. You come across some ROS documentation and realize that the ROS support is first-class, but you’re not using ROS. No, you’re using the Mission Oriented Operating Suite (MOOS). Why is there no documentation on using MOOS? Because it’s too easy to need a document, that’s why (update: this remains true, but we wrote a quick one anyway).

Note that there are two commonly-used projects here, MOOS, and MOOS IvP. The latter uses the former, and adds some more modules on top. It also has a more complicated build system when it comes to packaging, so we’re only going to talk about MOOS here (I’ll make another post regarding MOOS IvP).

Typical MOOS build process

MOOS is just good-old CMake, so the typical build process looks pretty, well, typical. First we grab and extract the latest release:

$ wget https://github.com/themoos/core-moos/archive/10.0.2.a-release.tar.gz
$ tar xzf 10.0.2.a-release.tar.gz

Now we configure and build it:

$ mkdir build
$ cd build/
$ cmake ../core-moos-10.0.2.a-release/
$ make

There are even sensible install rules in there if you want to run make install. Creating a new module to make use of it is also easy– just refer to this example.

Packaging MOOS as a .snap

Snaps are built using a tool called Snapcraft. For this example we’re going to target Ubuntu Core 15.04, which requires Snapcraft v1 (the latest release in that line as of this writing is v1.0.2). For more information on installing/getting started with Snapcraft, head this direction.

Let’s get started. First of all, we’re going to borrow this example from MOOS, with one modification to the CMakeLists.txt: an install rule. Just put it at the bottom, like this:

# ...
INSTALL(TARGETS ${EXECNAME}
 RUNTIME DESTINATION bin
)

If this isn’t clear, the finished code for this post has been pushed here. We need the install rule, or Snapcraft won’t know which files to place where when creating the final .snap.

Now we’re going to create the following directory structure:

moos-example/
  snapcraft.yaml
  icon.svg
  src/
     CMakeLists.txt
     ex1010.cpp

The snapcraft.yaml file is the recipe used by Snapcraft to “craft” the snap. Open it up in your favorite editor.

Let’s begin by providing some general metadata about the .snap, so add something like the following:

name: moos-example
version: 1
vendor: me <me@me.com>
summary: MOOS Example
description: Simple ping/pong example for MOOS
icon: icon.svg

In Snapcraft v1 that icon is required, so if you don’t have one run this command to make one:

$ echo "<svg />" > icon.svg

Back in the snapcraft.yaml, we need to tell Snapcraft about the parts making up our .snap. In our case, there are two parts: the MOOS core, and our ping/pong example:

parts:
 core-moos:
   plugin: cmake
   source: https://github.com/themoos/core-moos/archive/10.0.2.a-release.tar.gz
   configflags:
     - -DCMAKE_INSTALL_PREFIX=/

 moos-example:
   plugin: cmake
   source: src/
   after: [core-moos] # Build this part after MOOS

The snapcraft.yaml we have so far will result in a valid .snap, but it doesn’t export any behavior. Let’s make MOOSDB run as a service, at boot:

services:
 MOOSDB:
   start: MOOSDB
   description: MOOS router

Let’s also add a few binaries to use our ping/pong example (binaries don’t run at boot, they’re run by the logged-in user):

binaries:
 ping:
   exec: ex1010 --ping
 pong:
   exec: ex1010

So that adds two binaries: ping and pong. Their functionalities should be obvious.

Believe it or not, we’re done here. Let’s save and close snapcraft.yaml, and create the snap:

$ cd moos-example/
$ snapcraft

Wait for a minute while MOOS is fetched and built, then the local code is built, and finally while the .snap is being created. Once it’s finished, take it for a spin!

Testing the resulting .snap

Note that you need an Ubuntu Core device with the same architecture as the snap you just created (it can be a real device, or virtualized). Once you have it up and running, you can scp your snap to it and install it by running this command on the device:

$ sudo snappy install /path/to/my.snap --allow-unauthenticated

Now, on the device, you can see MOOSDB running:

$ sudo snappy service status moos-example
Snap           Service  State
moos-example   MOOSDB   enabled; loaded; active (running)

So now we can run our ping/pong example. In one terminal (logged into the device), run:

$ moos-example.pong

In another terminal (also logged into the device), run:

$ moos-example.ping

And you’ll see them communicating, just as you’d expect.

Conclusion

“So what’s the big deal? What did I just accomplish?” I hear you asking. You actually accomplished a number of things:

You now have an entire MOOS system prepared for deployment/sharing. All dependencies are included in the .snap– it will run on any Ubuntu Core 15.04 installation.
You’ve configured exactly what should be run on boot for the robotic system, no need to fiddle with config files.
By using Ubuntu Core, you now have a transactional deployment/update strategy for free. You can even get automatic updates on your devices if they connect to the internet and you upload your .snap to the store.
Perhaps most importantly, your MOOS system is running completely confined. Even if it has an exploitable vulnerability, the confinement model of Ubuntu Core means an attacker’s potential damage is very limited.

If these things are important to you, I definitely suggest you spend some time experimenting with Ubuntu Core, Snappy, and Snapcraft. For your reference, the code used in this post is in GitHub.

Seccomp filters on the Ubuntu Phone

Thu, 07 Jan 2016 00:00:00 +0000

The officially-supported Ubuntu Phones run pretty old kernels. The Nexus 4 and BQ phones all run v3.4, and the Meizu MX4 runs v3.10. Device kernels typically lag a bit behind since so much effort goes into making them compatible with the hardware, but this posed a bit of a complication since the Click confinement model utilizes features of AppArmor v3, which isn’t even released yet. However, we like to push the envelope here at Canonical, so we backported it into our phone kernels anyway. Unfortunately, Snappy is another story entirely.

The Snappy confinement model utilizes not only AppArmor, but seccomp filters and cgroups (there’s probably more in there somewhere, too). Unfortunately, while seccomp has been around for a while, seccomp filters weren’t introduced in until v3.5, which means Snappy won’t run on our current phones.

Until now (pretend I had a narrator voice).

Seccomp filters have now been backported to and enabled on all supported Ubuntu phone kernels, allowing them to run Snappy and .snaps fully confined. These changes are hanging around in -proposed, and will be included in OTA9, which is currently set to release on January 27th.

That’s not to say that our current phones will ever move entirely to Snappy– that’s not my call, and Snappy isn’t ready yet anyway. But at least the underlying support is there.

Prototype: A GUI-friendly Snappy

Tue, 03 Nov 2015 00:00:00 +0000

So this is the week of the Ubuntu Online Summit, and many of the sessions are discussing Snappy. As you may know, Snappy is currently pretty geared toward embedded, headless devices. However, it is the successor to Click, and eventually the phones will be based upon it. To drive that effort forward, a few colleagues and I had a session (update: I used to link to the video here, but the Summit server is no more) where we discussed the path forward for supporting snaps on other devices, specifically the phone and the desktop.

As Alejandro mentioned in the video, the current packaging situation sucks, and we know it. Application developers have to package their application as .deb for the desktop, .click for the phone, and .snap for embedded devices. Debian packaging isn’t going away– we’ll always be based on Debian, but the current situation can be improved.

Check out the first part of the video for the session. I gave a demo of a prototype tool that allows snaps to be installed and run alongside .debs and .clicks on both the phone and the desktop. Eventually we’d like Snappy to include such functionality, but until then, you can experiment with this yourself. We provided a link in IRC for a tutorial that walks you through exactly what I did in the video, but if you’re watching the video after the fact you probably didn’t get it. This post is for you.

The result of the prototyping is called “Crackly” (snap… crackle… nevermind). It’s a wrapper around Snappy introducing some other limited functionality. Note that Crackly is not a product we’re proposing, simply a tool that enables our prototype, allowing Snaps to run transparently alongside .clicks (or .debs). It’ll go away as it’s superannuated by features in Snappy.

This tutorial will walk you through installing and using Crackly. It has been tested on the current phone image and a Unity7-based Vivid desktop install.

Tutorial

First of all, setup the Crackly PPA:

$ sudo apt-add-repository ppa:kyrofa/crackly
$ sudo apt-get update

Now install Crackly:

$ sudo apt-get install crackly

That should pull in the relevant dependencies and install two binary utilities: the snappy CLI and crackly CLI. For this tutorial you won’t be using the snappy CLI directly (it’s used to drop privileges when unpacking the .snap), you’ll be using crackly.

Now, you won’t be able to reach the Snappy store by default, so you’ll have to download (or create) a snap which you can sideload. Crackly specifically disallows installing services (since the phones are using Upstart, among other reasons), so make sure it only has binaries. If you don’t feel like making one, I made one for you, you just have to build it into a snap:

$ git clone https://github.com/kyrofa/calculator-snap
$ cd calculator-snap
$ snappy build .

You’ll end up with snappy-calculator_2.0.182.1_multi.snap, which can be installed on both armhf and amd64 (it’s QML– update the package.yaml if you want more architectures). Transfer it to the device on which Crackly is installed.

Alright, now on the device with Crackly, install the prerequisite for the calculator app (not necessary on the phone):

$ sudo apt-get install qmlscene

Now install the calculator snap:

$ sudo crackly install snappy-calculator_2.0.182.1_multi.snap

Finally, you should be able to open the dash (on Unity7) or the app scope (on Unity8), and you’ll notice a new calculator app (with the icon from the QML calculator) which you should be able to launch. Doing so will run the app under the control of Ubuntu Core Launcher, exactly like any other Snappy-based system.

Limitations

There are a number of limitations to be aware of when experimenting with this prototype:

Crackly functionality is limited to installing/removing. No update or hw-assign is available, for example.
If you’re running this on a current phone image, you’ll need it mounted read/write rather than read-only.
If you’re running this on the current phone image, unless you’re using an arale (or newer) your kernel won’t support seccomp filters until OTA9. This is an important part of the Snappy confinement model and Ubuntu Core Launcher will refuse to launch without it. As a workaround, included in the Crackly PPA is an Ubuntu Core Launcher with a patch applied to launch even if seccomp filters aren’t available (note that this is an unofficial patch and will not be released).
The calculator snap discussed above currently runs unconfined for various reasons. Still working on that bit.

V3 Technology Awards: Vote for Canonical!

Fri, 11 Sep 2015 00:00:00 +0000

I’m pleased to announce that Canonical has made the shortlist in several categories for the V3 Technology Awards! Now it’s all in your hands: Go and Vote for Canonical! Note that voting closes at 5pm on Friday, October 9th.

The categories in which we made the shortlist are:

Best cloud computing provider
- We made this as an overall company: Canonical
Best business software provider
- We also made this as an overall company: Canonical
Best Internet of Things product
- Made this as Canonical Snappy Ubuntu Core

Thank you for your support!

Snappy Scope: Progress Update

Thu, 20 Aug 2015 00:00:00 +0000

I’ve previously written about Snappy and Unity 8’s Snappy Scope, back when I was first getting started on it. Since then, both the scope and Ubuntu Core itself have come a long way, so I thought it was time for an update and a quick demo.

When I was working on Snappy back when I wrote the other post I was running Ubuntu Core virtually, since I was using my embedded devices for other tasks. Since then, Canonical has sent me an Orange Matchbox to use, onto which I promptly installed Ubuntu Core. I was really hoping to be able to demo some hardware for you, but neither GPIO or I2C was supported, so I just used it to continue developing the Snappy Scope.

The scope is now much more functional, having the ability to list installed and store snaps, as well as install and uninstall them. It’s only missing a few things at this point:

The ability to launch installed snaps (this is due to the .snap specification only having the definition of a service, not a launchable application).
More details on the previews.
Internationalization support.

Thankfully, not long after I got to that point, support for I2C and GPIO was finally added to the Raspberry Pi 2 image, so I was able to put a decent demo together of both items:

If you want to get up and running with the scope, the instructions haven’t really changed since the previous post (i.e. read the README). It’s also installed out-of-the-box on Ubuntu Personal, although please remember that the image overall is very rough. If you want to give it a spin anyway, you can create one for KVM (it won’t work with VirtualBox because, well, Mir) like so:

$ sudo ubuntu-device-flash personal rolling --channel edge --output personal.img

Post a comment if you have any questions, or hop on #snappy on Freenode.

Update

I’ve been asked a few times about the source code to PiGlow Top. It’s a project I wrote for this demonstration, and it’s on GitHub.

Update 2

It’s no longer possible to create an Ubuntu Personal image from ubuntu-device-flash. We’re switching some channels around and focusing on pocket desktop at the moment. Don’t worry, it’s not dead, just not being actively developed for the time being.

Travis CI and Sauce Labs

Sun, 05 Jul 2015 00:00:00 +0000

I think we can all agree that testing one’s Rails project is a good thing. I think we can also agree that those tests should be run on a regular basis (i.e. after every change). I think it’s a logical conclusion then that having a continuous integration environment setup for one’s project would be considered ideal. This is really pretty easy for open-source projects, since a number of companies provide free support for them. The go-to CI for Rails projects seems to be Travis CI, but it has a limitation: browser testing with javascript is hard.

The typical tool for Rails integration testing is Capybara. When I first got started with it, I began with the Selenium web driver for javascript tests since it was easy (Firefox just popped up and started running my tests). Unfortunately, when I wanted to get my project running with Travis CI, I quickly ran into a problem: Travis CI is headless. I couldn’t just launch Firefox and start running my tests.

Headless option: PhantomJS

I did a little research and came across PhantomJS, which would allow me to run my javascript tests completely headless. It was easy to get integrated with Capybara and running in my testing setup, but it didn’t work the same way as Selenium– several of my tests started failing, and I wasn’t sure why. Had I started out using PhantomJS, it probably would have been fine. Unfortunately, this was after my project was nearly feature-complete– I had many, many tests, and I wasn’t interested in rewriting them.

Better option: Sauce Labs

Fortunately, my work ethic is such that I’ll spend ages trying to do the least amount of work possible. That paid off: I came across Sauce Labs. Not only could I continue using Selenium, but I could run my Selenium tests across an enormous range of browsers and devices! I was thrilled, but my enthusiasm was quickly quelled by the lack of documentation.

The Problem

I read the documentation from Travis, but came away asking “How do I run my tests on multiple browsers?” I read the documentation from Sauce Labs only to realize it was basically the same document. I found some more documentation from Sauce Labs that introduced me to a gem for running multiple tests, only to learn after a while that I couldn’t use it with MiniTest. Also, that gem doesn’t work with Ruby 1.9.3, whereas the rest of my project worked with both 1.9.3 and 2.1. Using the gem would mean I couldn’t run 1.9.3 tests on Travis, which was a deal-breaker.

The Solution

I finally figured out the secret: the Travis build matrix. It was actually pretty easy to do this combining the original Travis documentation for Sauce integration (linked above) with the build matrix environment variables.

Let’s quickly discuss Travis environment variables. You can define them in your .travis.yml file like this:

env:
 - FOO=bar
 - BAZ=qux

My initial assumption was that the build in Travis would now have access to two environment variables: FOO and BAZ. That actually isn’t correct– each item in the env list causes a new Travis build to spin up! In other words, the above file would actually cause two Travis builds to begin, one with the FOO variable defined, and one with the BAZ variable defined. If I wanted the behavior I initially expected, I’d have to do this:

env:
 - FOO=bar BAZ=qux

That’s the key. I wanted a new build for each Sauce Labs combination I wanted to test, which meant that I needed a new env item for each. That ended up looking something like this:

env:
 # Linux tests
 - PLATFORM="Linux" BROWSER="firefox"
 - PLATFORM="Linux" BROWSER="chrome"

 # Mac OS X tests
 - PLATFORM="OS X 10.10" BROWSER="firefox"
 - PLATFORM="OS X 10.10" BROWSER="chrome"

 # Windows tests
 - PLATFORM="Windows 7" BROWSER="firefox"
 - PLATFORM="Windows 7" BROWSER="chrome"
 - PLATFORM="Windows 8.1" BROWSER="firefox"
 - PLATFORM="Windows 8.1" BROWSER="chrome"

That gave me 8 different Travis builds with those corresponding environment variables defined. Now all I had to do was use them.

I was able to tweak the information provided in the original Travis CI document about integrating with Sauce Labs, and came up with this to put in my test_helper.rb:

# <snip>
# Only run with Sauce Labs if we're running on Travis CI.
if ENV['TRAVIS']
   capabilities = Selenium::WebDriver::Remote::Capabilities.send ENV["BROWSER"]

   # Set the browser version. If this is nil, Sauce Labs will use the latest version.
   capabilities.version = ENV["VERSION"]

   # Set the operation system.
   capabilities.platform = ENV["PLATFORM"]

   # Set the tunnel ID to connect to the identified Sauce Connect tunnel from Travis.
   capabilities['tunnel-identifier'] = ENV['TRAVIS_JOB_NUMBER']

   # Give the build a name in Sauce Labs so they aren't just all "Unnamed Job."
   capabilities['name'] = "Travis ##{ENV['TRAVIS_JOB_NUMBER']}"

   Capybara.register_driver :selenium do |app|
       Capybara::Selenium::Driver.new(app,
                                      browser: :remote,
                                      url: "http://#{ENV['SAUCE_USERNAME']}:#{ENV['SAUCE_ACCESS_KEY']}@ondemand.saucelabs.com/wd/hub",
                                      desired_capabilities: capabilities)
   end
end

# <snip>

Finally, I had multiple Travis CI builds all talking to different Sauce Lab environments, running my tests. This setup has been working great for me so far, and I encourage you to give it a try.

Rails: Writing Engine Tests That Depend on Main Application Models

Tue, 02 Jun 2015 00:00:00 +0000

When developing a new Ruby on Rails project, one doesn’t always start from scratch. Often one first searches for an open-source project that will at least get one part of the way there. For example, one might want to do something as simple a create a website with a blog as well as image galleries. The first step in this journey might be to find some open-source blog software for Rails, e.g. Publify (previously called Typo). Great, one has a blog!

Now what about that image gallery stuff? One would surely like to utilize the same authentication/account structure that is in use in the blogging software, but there’s a problem: Projects such as Publify are distributed as an entire Rails application. Yes, it’s of course open-source, but it was written to be one’s entire website. Adding image galleries means one needs to figure out and alter the Publify source code, and that’s just not cool.

The Solution: Mountable Engines

Actually, that exact situation is what led to my writing Proclaim. I wanted something that didn’t try to be my entire website, and just focused on doing what it was supposed to do: provide a blog. It was written as a mountable engine, and it doesn’t even include users or authentication– it will make use of whatever the main application is using.

The Resulting Difficulties

That desire quickly led to a testing issue, though. The blog Post model doesn’t need to know anything about its author, and the engine doesn’t need to deal with accounts in any way, but it makes sense for posts to have an author, so the solution is to make the class for the author configurable and we say that the Post model belongs_to it. Then the main application can configure that author class and the engine will then use whatever the main application configured. However, how does one write tests for the Post model if the engine doesn’t contain a model for the author?

Let me answer that with a quick tutorial (see the official Rails Engines guide for more information).

Tutorial

Create Engine with Posts

First, create your mountable engine:

$ cd ~
$ rails plugin new blorgh --mountable

In this tutorial, all we care about is the cross-over between engine and main application, and for our engine that happens on the Post model (we don’t need any comments or anything). So create a scaffold for your Post model:

$ cd ~/blorgh/
$ rails g scaffold post author_id:integer title:string text:text
$ rake db:migrate

Notice the author_id bit. That’s where the Post will belong_to an author, even though the author class itself is not defined within the engine. Let’s add that belongs_to logic to the Post model now:

module Blorgh
  class Post < ActiveRecord::Base
     belongs_to :author, required: true, class_name: "User"
  end
end

This is saying that the Post model must belong to an author, which happens to be a model called User (no, such a model doesn’t exist).

Write Tests

Alright, we have a Post model, so before we start fleshing it out, let’s write some tests (which is really what I’m writing about anyway). We know that a Post should have an author, so let’s write a test to ensure that a Post cannot be saved if it doesn’t have an author. The scaffold generated a blorgh/test/models/blorgh/post_test.rb file for us, so we might as well use that:

require 'test_helper'

module Blorgh
  class PostTest < ActiveSupport::TestCase
     test "ensure author is required" do
        post = Post.new(title: "Foo", text: "Bar", author_id: nil)
        refute post.save, "Post should require an author_id"

        # Author with 12345 shouldn't exist
        post = Post.new(title: "Foo", text: "Bar", author_id: 12345)
        refute post.save, "Post should require a valid author"
     end
  end
end

Pretty simple, right?

First Test Run

Alright, time to run the tests:

$ cd ~/blorgh/
$ rake test
<snip>
1) Error:
Blorgh::PostTest#test_ensure_author_is_required:
NameError: uninitialized constant Blorgh::Post::User

Right, so: the Post model belongs_to a User model, but the User model doesn’t exist. Where should it go? We’ve already decided that it doesn’t belong in our engine. Fortunately, Rails is way ahead of you.

The Dummy Application

Your engine has to be mounted before it can do anything, including run tests. For this purpose, there’s an entire Rails application pre-generated for you in blorgh/test/dummy/. This is a perfect place for the User model to go:

$ cd ~/blorgh/test/dummy
$ rails g model User name:string

Now run the dummy application’s migrations:

$ cd ~/blorgh/
$ rake db:migrate RAILS_ENV=test

You should notice the “CreateUsers” migration running.

Second Test Run

Alright, let’s try our test again:

$ cd ~/blorgh/
$ rake test
<snip> 0 failures, 0 errors, 0 skips

Success!

Conclusion

The use of the dummy application took some getting used to, and I don’t feel like it was well-explained in the documentation. I was initially under the impression that, if I wrote models in the dummy application, I would also have to write tests in the dummy application, and that just seemed nasty. Comments in the Rails mailing list about how the dummy application shouldn’t be committed to version control didn’t help– I obviously disagree with that.

In the end, you’ll find that the dummy application is a perfect fit for testing the crossover between your engine and the main application. The test written in this tutorial didn’t use any fixtures or factories for simplicity, but note: if you use FactoryBot, don’t put factories in the dummy application (FactoryBot won’t look there). Put factories for models defined in the dummy (e.g. the User model) in blorgh/test/factories/ (not blorgh/test/factories/blorgh/, since it isn’t contained within the engine).

Unity 8's Snappy Scope

Sun, 24 May 2015 00:00:00 +0000

I recently began a new position at Canonical. I joined this company in particular to give myself more opportunities for growth. Now, when I say that, people tend to assume I’m talking about a managerial track. I’m not sure why they tend to jump there, but it’s not what I’m referring to here. I’m talking about technical growth. I feel at the top of my game when I’m learning something brand new, and I needed a company that facilitated that. Canonical has delivered.

My background is quite heavily focused on C++. My resume revolves around it, and the interviews each probed the depths of my C++ understanding. So, as logic dictates, my first technical tasking is in– you guessed it– Go.

This is fantastic. Not that I particularly care about Go (I’d never used it before now), but because it broadens my horizons. I’ve been entrenched in the Google-test-driven C++ world for a while, so it’s nice to be able to dig into a new language, with new testing idioms. This can’t not make me a better all-around engineer.

I joined the company amidst the fervor of snaps and Ubuntu Core– a transactionally-updated version of Ubuntu targeted mainly at embedded devices. In order to be transactionally updated, Ubuntu Core uses a different packaging mechanism than regular-old Ubuntu. Ubuntu, as a Debian derivative, obviously uses .debs, but Ubuntu Core uses a packaging format called Snap, which is the successor of Click (the packaging format currently used on the Ubuntu Phone). In addition, a large number of Ubuntu Core components are written in Go, thus the choice of language for my piece.

Quick Ubuntu Core Background

Ubuntu Core is brand new, and pretty beta. Things are changing quickly, and I’m pretty new to the whole thing, so take what I say here with a grain of salt.

The master plan, as I understand it, is for Ubuntu to dominate the Internet of Things (IoT). The problem with embedded devices today is that one typically buys them from the manufacturer with some firmware installed, and then that device runs forever without anyone touching it again. At first glance that’s a good thing– the device should run forever. But the side effect of this is that the software on the device never gets updated. A good example of how this is a problem is with routers: a public-facing device, with security holes, that never gets updated to fix the holes. Pretending that the manufacturer even releases new software to fix the problems, the consumer can’t be bothered to: A) check to see that there is an update, or B) apply said update.

Ubuntu Core is attempting to solve this problem by changing these manual firmware flashes to software updates that can be safely automated, because they’re completely transactional, i.e. they can be reversed if they cause problems.

In addition to this, because the embedded device is running Ubuntu, you should be able to install whatever applications you want. Of course, since these are embedded devices, you have no way to do that unless you want to SSH into them or something. We thought of that with Ubuntu Core: You can use WebDM to configure each device remotely!

Ubuntu Core + Ubuntu Mobile

Okay, so Ubuntu Core has a pretty solid story behind it. So does Ubuntu Mobile. However, Canonical wants to fuse the two together. This makes sense– they have very similar components already (security model, transactional updates, app store, etc.), but they still have a lot of differences, particularly regarding the packaging and software install/update/run model. That’s (partially) where I come in.

“Snappy Scope”

Ubuntu Core makes the assumption that the user will do nothing locally (since an embedded device is also typically a headless device). Adding Ubuntu Mobile changes the game a bit, since a user is now doing everything locally. One of the first components we need is essentially a local-only WebDM– a local app store. Since Ubuntu Mobile uses Unity 8, it makes sense for this to be a Unity Scope.

“But wait,” you’re thinking. “Doesn’t Ubuntu Mobile already have an app store?” Indeed it does, but that scope is for installing Click packages; we need one for snaps.

“But wait,” you may also be thinking. “The Unity Scope API documentation is only available for C++. Didn’t you say this was in Go?” Ah, you are astute! This isn’t officially supported just yet, but my colleagues James Henstridge and Xavi Garcia have been slaving away on Go bindings for the Unity Scope API, and they are quite functional.

Development

We’re hoping to throw together a proof-of-concept very soon, which gave us a hudle really early-on: We don’t want to duplicate the effort that has gone into WebDM. WebDM is also written in Go, so we were hoping we could at least share libraries or something. Unfortunately, WebDM wasn’t written that way– it has a REST API for the web interface, and that’s it.

That’s okay. Ubuntu Core is indeed pretty beta, so we’ll work on getting WebDM to where it needs to go. Until then, we still don’t want to duplicate effort talking to the overarching Ubuntu Store API, so in our proof-of-concept we’ll just be using the existing WebDM REST API. We’re doing this in a very modular way so we can rip it out easily later.

We’re still very early-on in the development process, but you can check it out (and even contribute!) if you like.

How to test

First of all, you should check out the project’s README. It will be more up-to-date than this post.

Prerequisites:

Ubuntu 15.04 (Vivid)
Ubuntu Core running somewhere (I just put it in a VM)
WebDM must be installed
WebDM must be accessible from where the scope is running
Unity Scope Tool
Golang
Bazaar

Install the required packages:

$ sudo apt-get install unity-scope-tool golang-go bzr

Now setup your Go workspace. This is what I use:

$ mkdir -p ~/src/go # Create the workspace directory

$ echo "export GOPATH=~/src/go" >> ~/.bashrc # Make this environment whenever you login
$ echo "export PATH=\$GOPATH/bin:\$PATH"

$ source ~/.bashrc # Make it your environment now

Now grab the scope itself:

$ cd $GOPATH
$ go get launchpad.net/unity-scope-snappy/snappy-scope

This should result in a snappy-scope binary being placed into $GOPATH/bin/. However, it can’t be run directly– this is where the Unity Scope Tool comes in.

Each Unity Scope requires a .ini file describing itself and how to launch it, and Go scopes are no different. The Snappy Scope’s .ini file is alongside the source code, so launch it like this:

$ cd $GOPATH
$ unity-scope-tool src/launchpad.net/unity-scope-snappy/snappy-scope/snappy-scope.ini

The scope by default uses the webdm.local hostname to communicate with WebDM. If that’s not working, edit the snappy-scope.ini file to tell it where to look for WebDM. Something like this:

[ScopeConfig]
<snip>
ScopeRunner = $GOPATH/bin/snappy-scope --runtime %R --scope %S -webdm=<ip:port of webdm>

As you can see, the functionality of the scope is currently quite limited. At the time of this writing, all it can do is display applications from the store. I’m currently working on making those applications installable.

I haven’t yet noticed anything missing from the Go Scope bindings. Definitely worth checking out!

VirtualBox: Internet Access With Host-Only Network

Wed, 11 Feb 2015 00:00:00 +0000

I ran into the need to access the internet from a VirtualBox VM via host-only networking. Of course, you’re saying “There’s a better way to do that!” And of course, I agree: with no other considerations, the best way to do this is to simply add another NIC to the VM configured as NAT instead of host-only. However, I’m trying to test out some new networking ideas for my OpenStack Network node, and it doesn’t have a NIC configured with NAT. Since I’m trying to reproduce a production system in VirtualBox for testing purposes (which is how I typically use it), it stands to reason that I want the VM setup as close to the real thing as possible.

This is actually pretty easy, especially after reading Ubuntu’s Connection Sharing wiki page. First, consider my development setup:

Development laptop NICs:
- wlan0: Wifi internet connection
- vboxnet0: VirtualBox’s host-only interface (creating these is well-documented). I disabled DHCP to reproduce production setup.
  - IP is 192.168.56.1
VirtualBox VM NIC:
- eth0: Host-only interface connecting to laptop’s vboxnet0.
  - static IP: 192.168.56.100/24
  - gateway: 192.168.56.1 (i.e. development laptop)

Normally, in this setup, the VM would attempt to use the development laptop as a NAT gateway for external access, but since our laptop isn’t setup that way, the VM can’t see the internet nor is it accessible from the internet. I didn’t care about the latter– I just needed internet access from the VM. That’s as simple as setting up IP forwarding on the development laptop and enabling a few firewall (iptables) rules to setup the NAT.

Let’s start with some iptables rules. Note that these will be flushed upon reboot. That’s not important to me, but if you want these settings to persist it’s not difficult, and is well-documented elsewhere (start with Ubuntu’s wiki, linked above).

Let’s first add a rule to forward new connections from the host-only network:

$ sudo iptables -A FORWARD -o wlan0 -i vboxnet0 -s 192.168.56.0/24 -m conntrack --ctstate NEW -j ACCEPT

Now add a rule to forward already-established and related connections:

$ sudo iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Now flush the NAT table and add a new rule to it to masquerade any packets flowing out of wlan0 to look as if they’re coming from the laptop (i.e. “do NATing”):

$ sudo iptables -t nat -F POSTROUTING
$ sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

At this point, the firewall is all setup to take care of NATing traffic from vboxnet0 to wlan0, but we still need to allow packet forwarding to occur at the kernel level. Again, the way we’re doing it here is temporary (will be gone at reboot):

$ sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

Finally, you should be able to access resources on the internet from the VM:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=14.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=14.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=15.3 ms
^C

Note, however, that domain names won’t work, since we didn’t set that up. You know you can access 8.8.8.8 (Google’s nameserver), so you may as well use that. Then you’re good to go!

The Face Detector

Fri, 31 Jan 2014 00:00:00 +0000

Several years ago, I was new to my specific group at work. I was working on a combination of image processing and a GUI for keeping track of specific objects in the camera’s view. I started on the GUI first, writing a server that received object locations and handed them off to the GUI which displayed them in a nifty way. I was using Qt and QML, which I have used in numerous personal projects, so I was pretty familiar and confident with what I was doing. In order to test/develop the GUI I threw a quick C++ program together to create red and blue OpenCV images and fire them over the network.

Right as I got close to getting the GUI far enough along that I was ready for real images, my supervisor came in and said that we had our project sponsors coming by to see how we were getting along. He wanted to show them my GUI work, and wanted it working with detecting faces in a live camera feed. Oh, and they were coming day after tomorrow.

What? I had never really done much image processing before (other than one class in college), and had NEVER used OpenCV before (other than generating red and blue images). Now I needed to throw something together in a day?

So I started doing some research. It turned out that Haar-cascade-based face detection was pretty easy in OpenCV, but it was too slow to do every frame in my case. So I went with initially detecting faces via Haar cascades, and then handed it off to a color tracker (based on Camshift). It wasn’t great, but it was acceptable for something I threw together in half a day. I spent the other half a day tying it together with the GUI.

With the Camshift-based color tracker, one needed to calibrate it for the color you want to track (and toss out unrelated colors). So the day of the demo, I set my stuff up where I wanted and played with the calibration until it detected my Italian face quite nicely. I saved those parameters so they would be used every time the detector/tracker software was run. Multiple people came by throughout the demo day– the setup was a hit! The initial face detection worked, and then people had fun trying to ditch the color tracker by trying to get it to track their hand, etc.

Then my supervisor came in with the sponsor in tow. He was black.

He stood in front of the camera with the other people playing with it, and it wouldn’t track him. He tried getting close: no cigar. Tilting his head… nada. I exchanged awkward glances with my coworkers and tapped the camera like some idiot. “Huh… looks like it’s not working anymore.” I quickly quit the program so he wouldn’t notice it tracking other people on my screen. I had no idea what to say. I had made a racist face detector.

He finally stood up and said “Huh… well cool,” and left. We’re still funded. Now that I know more, I would have used something like OpenTLD.

GitLab 6.3 with Apache and Passenger. On an ARM

Mon, 25 Nov 2013 00:00:00 +0000

As I’ve mentioned in previous posts, I run all my websites on a MiraBox– my nifty, low-power ARM server. As awesome as it is having a low-power server, it does make things more difficult. I’m a Ruby on Rails developer, and I run three websites on this thing, along with a gitolite installation so I can host my own code. However, I recently began a few projects collaborating with other people, and as the projects grow, gitolite is not the collaboration platform of success. I need groups and better permissions. I need issue tracking. I need wiki pages. I need GitLab.

There’s only one problem. If you follow its installation procedure, GitLab does not support the ARM architecture. Rather, gems upon which GitLab depends do not support the ARM architecture. Namely: therubyracer.

Server-side Javascript has long been a nightmare on ARM. therubyracer depends on the libv8 gem, which straight up cannot build on ARM. Which is annoying, since without too much convincing, V8 itself can build on ARM (even though it uses GYP, a build system that I simply cannot figure out).

I tried participating in therubyracer and libv8 gem development, but there’s not a lot of momentum behind ARM development there and later versions of V8 simply don’t compile with the old GCC in Squeeze anymore, so I gave up and punted: Hello Node.js. With some code tweaks, I was able to get THAT installed on my MiraBox. I then removed therubyracer from my Gemfile and Gemfile.lock, and was able to follow the GitLab installation directions. Well… mostly.

As I said before, I already had three different websites on my server using Apache and Passenger– I refuse to also use Nginx and Puma or Unicorn or whatever the kids are using these days. That made things a little harder though… was there something special about Puma/Unicorn? Did I need it? These were questions for Google.

…

No, I take that back– with an utter lack of results other than a few mentions of Apache with no one talking about Puma or Unicorn, they were questions for the mailing list.

…

No… I take that back– no one ever responded. So, I had to figure it out on my own.

It was a while ago that I learned the lesson not to experiment with this type of stuff on a live server. However, I took it even a step further. I had previous experience setting up a GitLab install, but that experience only involved following the GitLab installation instructions to a T. I wanted to do a few things differently in this situation: namely, I wanted to use Apache instead of NginX, I wanted to use Passenger instead of Puma/Unicorn, and I needed to use Node instead of therubyracer. Since I didn’t even know if this setup would work on the best of systems, I decided that testing this setup for the first time on a crawling ARM emulation wasn’t the best way forward. I wanted to see if this worked on a more powerful, normal machine. So I threw a server installation together in VirtualBox, using Ubuntu 12.04 64-bit.

I followed the GitLab installation instructions, but I skipped a few parts this time. GitLab 6.3 uses Unicorn (whereas some previous releases used Puma), and I specifically did not setup Unicorn at all– I left its configuration file at “unicorn.rb.example,” which I assume means it doesn’t run. This required an alteration to the gitlab init script, since it runs both Sidekiq and Unicorn. I simply removed all mention of Unicorn, and rewrote the script to use Sidekiq only.

Also, obviously, instead of setting up an NginX configuration for GitLab, I setup an Apache2 virtual host. Finally, before I ran the bundler to deploy the gems, I removed therubyracer from the Gemfile (and Gemfile.lock) and installed the newest Node.js from source. This setup actually worked perfectly, first try. There was hope! Also, that answered the question of Unicorn/Puma: They are not necessary for GitLab to function.

I then moved to duplicating each of these steps on the QEMU-emulated ARM server. That took ages, I’ll tell you. And after I finished I got nothing. It spun for ages and then said that the server didn’t respond. Okay… so not as good as the VirtualBox version. My investigation started in the GitLab logs. It actually looked like things were happening, but then Firefox always came back with “server didn’t respond.” After a little more digging I realized that everything did indeed look fine– the server just didn’t respond fast enough.

After some searching around, I realized that with Passenger 4.0.20 they introduced the “PassengerStartTimeout” option, which allows one to extend the loading window giving a good-sized application like GitLab time to fire everything up. I set it to 60000 (time out now, I dare you). Success!

It was time to put GitLab on my real server.

I went through the same steps– didn’t configure Unicorn and configured an Apache2 virtual host for Gitlab:

<VirtualHost *:80>
   ServerName www.url.com
   ServerAlias url.com
   DocumentRoot /home/git/gitlab/public
   <Directory /home/git/gitlab/public/>
       AllowOverride all
       Options -MultiViews
   </Directory>

   PassengerStartTimeout 60000 # Required since the MiraBox is so slow
   PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.3-p484@gitlab/ruby

   ErrorLog ${APACHE_LOG_DIR}/gitlab_error.log

   # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
   LogLevel warn
   CustomLog ${APACHE_LOG_DIR}/gitlab_access.log combined
</VirtualHost>

I removed therubyracer from both the Gemfile and Gemfile.lock, installed Node.js, and deployed all the required gems with Bundler according to the GitLab installation guide. I had everything setup identically to my QEMU-emulated server, and I ran it… and got a 500 error.

What the heck? So I started digging into my logs. I actually tail’d the GitLab production log, and watched it load. I watched it compile all the assets on the first request (as expected). I saw the request get the layout, and the layout partials… and then a 500 error after _head.html.haml. There wasn’t actually an error in the GitLab log, or the Apache log, or being shown in Firefox– just a 500. Normally one actually sees something like “function .empty() doesn’t exist for Nil.NilClass” or something along those lines… I saw nothing. It was totally bizarre. So I edited _head.html.haml and started old-school debugging: adding print (logger.debug) statements. Using this technique I discovered that the 500 happened right after application.js was being included. Interesting… so it involved the asset pipeline. Just for kicks I tried precompiling the assets with:

$ bundle exec rake assets:precompile RAILS_ENV=production

And I got an error! I’m afraid I don’t have the error message saved, but it involved an invalid byte and something about UTF-8. Regardless, it was failing to compile its assets when they were requested, so it barfed a 500 without actually giving any sort of descriptive error (which was annoying).

What do you do when assets need to be compiled but the server has errors when compiling them? Fortunately I already had the solution in place: Compile it in another, identical environment. I chose the VirtualBox server since it was way faster than the QEMU server. I ran the same precompile line on there, then tar’d up the generated assets and placed them in the correct spot on my production server.

The moment of truth: I restarted everything and it worked beautifully :) .

GitLab lessons learned:

Unicorn/Puma is not necessary, but not using them requires an altered boot script.
NginX is not necessary– Apache2 works great.
Node.js is a replacement for therubyracer, but it doesn’t seem 100% compatible with the ARM architecture. Asset compilation is wonky… compile elsewhere and deploy.

The only question I have is this: Why did Node.js fail to compile assets on my production server, but succeed to compile assets on my QEMU-emulated ARM server? I have yet to find an answer.

Upgrade makes Mirabox unbootable

Mon, 29 Jul 2013 00:00:00 +0000

For some reason (I can’t really put a finger on it exactly), I like to keep my servers up-to-date. I know! All those security patches, all those new features– they’re a killer!

In all seriousness, I host several websites on a Mirabox from right behind my TV. I hopped on the plug computing train way back when the SheevaPlug was just a dev kit, and I’ve been on it ever since. The reasons are pretty simple: reasonable power and capabilities while using less power than my desktop computer when it’s off (i.e. phantom power). I mean come on– the Mirabox in particular has two gigabit ethernet ports and two USB 3 ports. I have a fantastic RAID 1 setup running on that. Anyway, I digress.

The Mirabox comes with a custom flash of Debian Squeeze on it that worked just fine. A few weeks ago I decided to get it up-to-date.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get dist-upgrade # What was I thinking...

And reboot and… the Mirabox never came back. It just hung when it tried to bring the network interfaces back up. This is the end of the boot log:

<snip>
[ ok ] Setting kernel variables ...done.
[ ok ] Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
[FAIL] startpar: service(s) returned failure: module-init-tools ... failed!
pool #0: pkt_size=1536, buf_size=1632 - 2048 of 2048 buffers added
eth0: started
ADDRCONF(NETDEV_UP): eth0: link is not ready
mlan: module license 'Marvell Proprietary' taints kernel.
Disabling lock debugging due to kernel taint WLAN FW is active
ADDRCONF(NETDEV_UP): uap0: link is not ready
Bluetooth: Core ver 2.15
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.14
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM ver 1.11
WLAN FW already running! Skip FW download WLAN FW is active
ADDRCONF(NETDEV_UP): mlan0: link is not ready

This was, believe it or not, somewhat of a blessing in disguise, since I should NOT have been running the OS off of internal memory. Why, you ask? Because this just happened. There was no way to recover– I tried booting to different runlevels, just the shell, you name it.

I ended up rebuilding the server from scratch, but on an SD card this time so I could at the very least back it up (via dd) as I made progress.

And you know what the issue was? Check out the sources.list:

deb http://ftp.us.debian.org/debian/ squeeze main contrib non-free
deb http://http.us.debian.org/debian stable main contrib non-free
deb http://security.debian.org squeeze/updates main contrib non-free
deb http://www.backports.org/debian squeeze-backports main contrib non-free

Does anything jump out at you? Perhaps the fact that the second line is actually the same repository URL, first of all. But that’s not the bad part: notice the mixture of squeeze and stable repositories. Stable is now Wheezy! Whichever idiot at Globalscale who set this thing up just potentially screwed all of you by mixing releases. That person has wasted nearly two weeks of my time.

The solution is simple: delete the second line. The URL was redundant anyway.

Now you’re just using Squeeze!

You might ask: “Well, why not just update to Wheezy?” Because the wheezy repos contain the package updates that cause the Mirabox to no longer boot. Which packages, you ask? One of these five (they’re all interdependent… I’m not sure which one breaks it):

initscripts
netbase
rsyslog
wpasupplicant
ifupdown

You’ll notice in the install logs (if you go ahead with that madness) that “netbase breaks ifupdown.” I suspect the root cause to be somewhere around there.

If you figure out how to successfully upgrade to Wheezy, let me know! For now, I’m happy with Squeeze.

QEMU: MiraBox emulation

Sat, 20 Jul 2013 00:00:00 +0000

As I mentioned in a previous post, I host various websites (including this one in the past) on a MiraBox sitting behind my tv. In that same post I outlined how Globalscale screwed me by mixing releases in the sources.list. I didn’t catch it until I had an unbootable system, and I had to completely rebuild my server which took weeks. I very nearly tossed the whole thing in favor of hosting my websites on a “real computer, made by people who knew what they were doing,” but I reminded myself why I settled on the MiraBox in the first place: I like embedded stuff. I’m a computer engineer, this is what I find interesting. Is installing new things on this box harder than on an x86? Sure. But do you learn new things every time you play with it? Definitely.

So, I rebuilt my server on the MiraBox again. And now it’s working better than ever (now that I’ve fixed my sources.list). However, I also have an ownCloud install on here, which I recently tried to upgrade. Of course, I broke it and had to spend a few days rolling back (something weird happens to a PostgreSQL database when you update Owncloud 4.8 to 5… I haven’t figured it out yet). Some of you might be laughing, because I STILL hadn’t learned my lesson: You don’t do research and development on production systems. You get it working in a parallel environment, and then deploy it. I’ve got it now, I promise. However, there’s no way I’m shelling out for another MiraBox just to play the role of a staging server, so I’ve come up with another idea: using QEMU to emulate the MiraBox environment as closely as possible.

QEMU

QEMU is a simple, open-source machine emulator. When I decided to go the virtualization route, my first thought was obviously VirtualBox, but a quick google revealed that it doesn’t support other architectures. So after some research, I settled on QEMU for ARM emulation.

Install Debian

My MiraBox server is running Debian Squeeze, so the first step is to download Debian Squeeze to install on the emulated staging server. However, installing on an ARM is a bit different than installing on, say, an x86. Instead of using the .iso, you need to seperately download the ARMEL kernel image and installer:

$ mkdir -p ~/qemu/debian/squeeze
$ mkdir -p ~/qemu/images
$ cd ~/qemu/debian/squeeze
$ wget http://ftp.debian.org/debian/dists/squeeze/main/installer-armel/current/images/versatile/netboot/initrd.gz
$ wget http://ftp.debian.org/debian/dists/squeeze/main/installer-armel/current/images/versatile/netboot/vmlinuz-2.6.32-5-versatile

Now an image needs to be created for the VM. I like the QEMU’s copy-on-write strategy, so I use the qcow2 format. I also have Debian on a 32 GB SD card inside the MiraBox, so I’m making this image 32 GB.

$ qemu-img create -f qcow2 server.img 32G
Formatting 'server.rainveiltech.com.img', fmt=qcow2 size=34359738368 encryption=off cluster_size=65536 lazy_refcounts=off

Now it’s time to actually install Debian. I was really hoping for a way to emulate the MiraBox hardware as closely as possible, but it seems that the closest I can get with QEMU is the arm5 instruction set (the MiraBox can use arm7). I figured that would be good enough, so I walked through installing Debian:

$ qemu-system-arm -M versatilepb -kernel ../debian/squeeze/vmlinuz-2.6.32-5-versatile -hda server.rainveiltech.com.img -initrd ../debian/squeeze/initrd.gz -append "root=/dev/ram" -m 1G

After it had finished installing, I fired it up to make sure everything looked good:

$ qemu-system-arm -M versatilepb -kernel ../debian/squeeze/vmlinuz-2.6.32-5-versatile -hda server.rainveiltech.com.img -initrd ../kernels/initrd.img-2.6.32-5-versatile -append "root=/dev/sda1" -m 1G

It looked fine, but by default the user-mode networking stack won’t allow any incoming traffic (reference) so I decided to just forward port 80 on my host machine to the guest machine, with -redir tcp:<host port>::<guest port> or in my case:

$ qemu-system-arm -M versatilepb -kernel ../debian/squeeze/vmlinuz-2.6.32-5-versatile -hda server.rainveiltech.com.img -initrd ../kernels/initrd.img-2.6.32-5-versatile -append "root=/dev/sda1" -m 1G -redir tcp:80::80

Now I can install Apache etc. and test out new settings! I was having some trouble getting Gitlab setup on the MiraBox, so now I can play with it here instead of my production system.

I still wish I could get Debian running on an emulated arm7 system. If anyone knows how to do that, please share!

Dell Sputnik beta participant gift

Sun, 06 Jan 2013 00:00:00 +0000

Well, how nice! Look what the UPS man hid behind my garbage can (seriously… the thing was all damp; it must have been sitting out there for two weeks).

That’s pretty nice of them! It even included a note:

The one strange thing is that while it’s marked for Project Sputnik (which uses a Dell XPS 13), I’m not sure I would feel comfortable carrying such a small computer with it. Indeed, it’s better suited for a 17-incher (that’s a 17-inch macbook pro in there):

But it was a nice gesture. Thank you, Dell.

TSA: "Red is the Bad Color"

Sat, 08 Dec 2012 00:00:00 +0000

At work, a large part of my focus is stereo vision. Well, a coworker and I needed to attend a three day training seminar, which we expected would be quite boring. So, we grabbed a pair of cameras, secured their baseline by attaching them to a small metal beam, zip tied the beam to a 12V power supply, and shoved the whole thing into a pelican case so we could continue working on interesting things while we were there.

To the unfamiliar, stereo vision works with two cameras taking frames in sync with each other. In order to facilitate syncing, with these particular cameras, you call one camera the “master” and the other the “slave,” having the slave trigger frames when the master says to do so. This requires the cameras being connected in a sync circuit involving a 1.6K resistor. Each camera has a wiring harness included with it, which was used to connect the cameras to the power supply (black and red wires), and the master camera to one end of the 1.6K resister (yellow wire) and the slave camera to the other end of the resister (green wire). The sync circuitry needed to be powered as well, which was another red wire.

All of these wires were connected by way of an admittedly messy rail, with crescent connectors crimped onto the wires. I mean, come on, we just threw the thing together. The power supply we used was I guess fairly interesting-looking: black, covered in heat sink blades.

So: We checked the pelican case, and flew to our training with no issues. We messed with stuff for a few days, made no real scientific breakthroughs, and then flew back, checking the case again. When we got back to the lab, we cracked open the box to find a “TSA has inspected this” tag, and all of the red wires clipped off. I’m not kidding. The green wires weren’t touched; the yellow wires were fine; the black were also left alone; every red wire was cut.

Seriously TSA? You were so scared this thing was going to explode that you flashed back to your days watching 24 and decided to cut the red wires? If this was a bomb, do you really thing it would have defused after you started slicing through it?

Word to the wise: If you check electronics for a flight, make sure your wires aren’t red.

Dell Sputnik XPS 13 critiques

Wed, 05 Dec 2012 00:00:00 +0000

As I mentioned in my last post, I recently picked up an XPS 13 at a discount from Dell for the Sputnik beta project. When I say “recently,” I actually mean about two months ago, which means this is not a “first look” post. This has been my main computer ever since it arrived, which means I have some thoughts to share. Mostly critiques, at least for this post.

Hardware Design

Google pictures of this thing. It’s really quite pretty. It’s also light, and it has a backlit keyboard, which is still difficult to find even though Apple started doing it ages ago. It’s also pretty gutsy inside with an i7, while still retaining pretty good battery life (I have not explicitly tested battery performance). The annoying thing is that it only has 4GB of memory. Of course, now that Sputnik is officially released, it has 8. Too bad us beta testers don’t get a taste.

Another detractor from the design is the glossy display. That was the only option. I hate gloss. Who likes it? Who started that idea? Some narcissist who likes staring at his reflection? Give me a matte display any day. The only upside of this is that it’s Gorilla glass, which is pretty tough stuff.

But those few detractors don’t hold a candle to this one:

Notice the vents behind the hinge, kinda like every other computer you’ve seen. But this laptop is different– its intake is on the bottom. Seriously. The fan-powered intake for the vents is right where one would normally place the laptop on one’s lap. In fact, if you wear shorts and you have hairy legs (I do), you have to hold this laptop in a very special way not just because the intake fan starts screaming due to starvation, but because it starts grinding on your legs hairs! Who designed this?

This doesn’t even work well sitting on a table! There’s simply not enough space between a flat tabletop and the intake to get enough air if you’re running full-tilt. Compile a kernel or something and you’ll quickly realize the only way to get rid of the screaming fan is to set it teetering off the edge of the table like in the image at the top of this post.

This is a truly awful design, and would have caused me to pass this laptop up if it weren’t for the Sputnik program. However, these complaints relate only to the XPS 13 hardware, so let’s see where we stand on the Sputnik software.

Software

Out of the box (using the Ubuntu 12.04 Sputnik-specific ISO), this thing ran Ubuntu well. The only issue I had was with the touchpad, which worked embarrassingly badly. I hate tap-to-click, and the first thing I always do on a laptop is disable it. I just have big hands, and my palms always cause taps to occur. So: I disabled tap-to-click. But it was only honored for the top 3/4 of the touchpad– the bottom 1/4 still turned taps into clicks! This driver was incredibly untested. I knew I was using a beta and did what any beta tester should do– I posted a bug on the Sputnik Launchpad page and the Sputnik Forums.

Fast forward two weeks. No activity.

I’m not sure what deal Dell has with Canonical, but Dell does not seem to support this at all from a software standpoint. It seems that they just handed it off to Canonical and/or the community, and expected the “beta participants” to be their developers. I ended up fixing that bug myself. I don’t actually mind so much that Dell isn’t messing with it, but I think it’s indicative of their motives for this project.

Summary

After working with this platform for a while, it’s my belief that if this project succeeds, it will be due to the efforts of Kamal Mostafa and the folks at Canonical more than Dell. That said, I do hope it succeeds. I just wish the XPS 13 was a better hardware platform than it is. Fix the ventilation issues, and this platform is a contender. Fix that and the glossy-display-only stuff, and this thing is a winner.

My new laptop: Dell Sputnik XPS 13

Sun, 02 Dec 2012 00:00:00 +0000

My old laptop finally seemed to be on its last legs, so I began shopping. I’m the kind of guy who will do research for a month before spending $100 on a hand-held vacuum cleaner, so you can imagine how I deal with something having three or more zeros in its price tag.

I started using Debian when I was just a kid, and moved to Ubuntu a few years later. I was hoping that I could purchase a laptop in support of Linux– maybe System76, etc. But my artsy side shied away from every Linux-specific laptop I could find. I wouldn’t call them ugly, but they’re not at all appealing! I would like one that doesn’t look bullet-proof, please.

That’s when I happened upon Project Sputnik currently in beta status at Dell. I thought “Oh, this is perfect! My current laptop will probably last until the Fall of 2012, when the Sputnik project will be official.” So, I decided to wait while I researched the XPS 13’s hardware. But I also decided to post a comment on Barton George’s blog (the guy running Sputnik), and ask if they had a more specific release date than “Fall 2012.” Rather than reply to that question, he sent me a response saying there was still room in the beta if I wanted in. I’d get 20% off and by the way that code expires in one week.

20% off in trade for having zero time to research the XPS 13. But it’s supporting Linux! Okay… I did it.

Was it worth it? I’ll post a review shortly.

Mon, 01 Jan 0001 00:00:00 +0000

Who is Kyle Fazzari?

Mon, 01 Jan 0001 00:00:00 +0000

Kyle is a husband, father of seven, roboticist, and a prolific open source software developer (where he goes by the handle “kyrofa”). He maintains and contributes to a number of projects, including Ubuntu, Nextcloud, versions 1 and 2 of the Robot Operating System (ROS), as well as the snapcraft CLI and snapd, two key technologies behind snaps and Ubuntu Core.

Kyle’s day job is at Miriam Technologies, a software consulting and contracting company, where he works on everything from cloud infrastructure and security to industrial real-time product sorting machines. In the past he’s worked as a staff engineer on the security and robotics teams for Canonical (the company that publishes Ubuntu), and as a lead roboticist for the US Department of Defense.

He believes that a maintained and stable API is a work of art, and is easily bothered when they aren’t. He thinks C++ is the only real language, and will never admit a begrudging fondness for Ruby and Python. He tends to publish blog posts about anything technical that interests him.

kyrofa's blog

How to use RubyGems Trusted Publishing with Semantic Release

RubyGems Trusted Publishing

How do we automatically push new gem versions with Trusted Publishing?

Integrate Semantic Release with Trusted Publishing

Semantic Release configuration

CI Workflow

Conclusion

New and improved way to review GitHub pull requests for snaps

GitHub Actions

GitHub Snap Builder

How does that relate to GitHub Actions?

Show me how

Generate new snapcraft token

Create a new environment: snapcraft

Untrusted code: build the snap

Trusted code: upload the snap

Conclusion

Monitoring a Nextcloud snap upgrade

MySQL

PHP

Apache

Nextcloud Fixer

Conclusion

Nextcloud snap finds a new home

What happened?

So protect your branches!

Web of trust

What’s the plan?

Demystifying Rails autoloading

Ruby

Bundler

Rails

Conclusion

My phone isn't as smart as yours

In the beginning

Smartphones forever!

All was not right with the world

I know! Let’s rely on our self-discipline!

The return of T9

No more camera in your pocket

No more instant calendar

Group messaging is hit or miss

Seriously, T9?

Maps and GPS

Alternative sources of music

Severe lack of knowing

Not just a detox

The end of my journey at Canonical, and new adventures ahead

My journey

Does that mean you’ll stop maintaining <project>?

So what is next?

How to build a snap using ROS 2 Foxy

Prerequisites

Let’s get started

Create the snapcraft.yaml

Build the snap

Test the snap

Four years of Nextcloud

PSA for ROS users: Some things to know as Python 2 approaches EOL

Robotics security: What is SROS 2?

What is DDS-Security?

Authentication

Access control

Cryptographic

So what is SROS 2? It’s ROS 2’s integration with DDS-Security

ROS client library (RCL)

SROS 2 utilities

Conclusion

Canonical joins the ROS 2 Technical Steering Committee

A better way to review GitHub pull requests for snaps

The problem

The solution

GitHub App

Server setup

Activate GitHub App on GitHub repo

Speed up your ROS snap builds

stage-snaps

An example

Create ros-foundation

`stage-snaps`

Create `ros-foundation`

Create `ros-app`

Create the `snapcraft.yaml`

Step 2.3: Write the `snapcraft.yaml`