Stuart Hinson

Rocket v0.5 Error Handling Best Practices

2021-07-28T12:36:00+00:00

A common question in #rocket is how to go beyond the basics in error handling. Specifically, there’s a bit of a jump that’s needed to connect the dots between defining a custom error type (e.g. with thiserror) and the Responder trait.

We can put these together and write handlers with idiomatic early returns, e.g.

#[derive(Error, Debug)]
pub enum Error {
    #[error("HTTP Error {source:?}")]
    Reqwest {
        #[from] source: reqwest::Error,
    },
    #[error("SerdeJson Error {source:?}")]
    SerdeJson {
        #[from] source: serde_json::Error,
    },
}


#[get("/hello")]
async fn hello() -> Result<Json<Todo>, Error> {
    // a  https://docs.rs/reqwest/0.11.4/reqwest/struct.Error.html
    let response = reqwest::get("https://jsonplaceholder.typicode.com/todos/1")
        .await?
        .text()
        .await?;

    // https://docs.serde.rs/serde_json/struct.Error.html
    let todo: Todo = serde_json::from_str(&response)?;
    Ok(Json(todo))
}

Setting aside the contrived logic, this is error handling as we’d expect in Rust – we’re leaning on ? and its implicit into to map errors into our Error type.

But, in a real-world application, we’re not typically happy to simply render a 500 back to the user. How do we respond differently to different error conditions? How do we log an error into a monitoring service like Sentry?

Here (and as of 0.5), the best practice I’ve found is to implement Responder for your Error type

impl<'r, 'o: 'r> Responder<'r, 'o> for Error {
    fn respond_to(self, req: &'r Request<'_>) -> response::Result<'o> {
        // log `self` to your favored error tracker, e.g.
        // sentry::capture_error(&self);

        match self {
            // in our simplistic example, we're happy to respond with the default 500 responder in all cases 
            _ => Status::InternalServerError.respond_to(req)
        }
    }
}

With this approach, we have a natural place to centralize error tracking as well as flexibility in the responses we send from the server. Not bad!

Rocket v0.5 Error Handling Best Practices was originally published by Stuart Hinson at Stuart Hinson on July 28, 2021.

Effective (and Simple!) Predictive Recommendations

2020-01-22T12:36:00+00:00

I want to give a quick illustration of the idea that effective applications of machine learning – especially early versions – can be almost embarrassingly simple. This is the broad strokes of a system I implemented that produced a substantial lift in revenue for a large online marketplace.

This marketplace is a bit unusual in that its products are all one of a kind. Every day, we’d list a new batch of products for sale, and the problem I faced was how do I pick which products to recommend to each user.

That each product was unique meant that the most common solutions for generating recommendations like a correlation matrix couldn’t be applied since we don’t have historical data about what users who’ve bought the same products had also bought. I did notice, however, when I looked into our data that users had a high affinity for particular types of products. At a coarse level, that meant we could use our product categorization as the basis for recommendations.

The implementation is simple and took less than a plane ride to finish. I wrote a background job that scored each user for their affinity for a category based on usage signals we had available (say, viewing a product, saving a product a wishlist (better signal than just viewing!), purchasing a product (great signal!)), producing a set of scores normalized across our categories that represented each user’s preferences. Those scores were then used when a known user viewed new products to predict which they’d be most interested in.

Split tests showed a substantial lift in revenue from its introduction.

Notes:

There are, of course, a few wrinkles when deploying a system like this. One is the idea that our confidence in our predictions should be proportional to how well we know that user. The less that’s known about a user, the easier it is to make predictions that overfit for the little we do know.
Predictive recommendations for engagement are important, but definitely don’t speak to the whole picture for either a user or the business. The ranking system I wrote also incorporated factors like anticipated customer reviews, support workload, customer churn, profitability, etc.

Effective (and Simple!) Predictive Recommendations was originally published by Stuart Hinson at Stuart Hinson on January 22, 2020.

Connecting Remote Controlled Blinds to Alexa Smart Home

2018-09-08T12:36:00+00:00

My intuition is to begin with a bit about the pragmatic motivations for the project, but if I’m being honest this came from a place of ‘I bet I could do this’ rather than ‘what my life needs isn’t remote controlled blinds, it’s voice controlled blinds’.

Anyway. My coworking space recently bought some remote controlled blinds. This is a story about how I connected them to Alexa.

The idea is to have an architecture like so

And we’d flow data through it

Digging into the details, we’ll start with the last step in our diagram: How do we replicate the remote’s commands?

We’re off to an easy start as the remote lists its transmitting frequency on the back. Similar devices will have an FCC id that can be referenced to find the same information.

Next we’ll rely on a software defined radio and the amazing Universal Radio Hacker to record and analyze the remote’s messages.

After recording a few taps of the remote, we can use Universal Radio Hacker to decode the message

The process couldn’t have been easier. Looking at the raw signal, I took a (lucky?) guess that it used ASK modulation, and the software will then detect the bit length and produce an interpretation of the signal.

Next step is to check that we’re on the right path by transmitting the same code ourselves and making sure we can control the blinds. I used two components for this, an ESP8266 and a 433Mhz transmitter

Connected on a breadboard along with an LED give a cue when it’s transmitting

Now we can write some quick and dirty code to see whether we can move the blinds

And it works! I’m not sure I expected it to.

I feel self conscious posting that example using a String to store the command. Later we’ll switch to a more efficient encoding, but as a proof of concept, well, it proves the concept.

Moving on.

Next we’ll look at exposing the microcontroller over an HTTP interface. The plan here is to make a simple HTTP server to receive high-level actions like ‘Raise the blinds’ and pass them on the microcontroller in the form of the bits necessary to take that action.

This means we’ll have a process that’s listening on two ports. One port for incoming HTTP requests, and one port that our microcontrollers will connect to as they come online. We’ll then use the device information from the HTTP request to find the microcontroller to communicate with, look up its action mapping (e.g. for Stu’s blinds controller, the action ‘up’ means send 10101011’ at 300 bits / second), and encode and transmit the action to the microcontroller. To serialize the action we’ll use

<MESSAGE TYPE: 1 byte><BIT DURATION: 4 bytes><MSG LENGTH: 4 bytes><MSG: MSG LENGTH bytes>

The microcontroller side of this code was pleasingly simple to write. The ESP8266 has a built in wifi chip, and nice library support for creating a TCP connection. All we need to do is to write code to connect to our server and then handle the two message types our server sends: keep alives and actions.

We’re close now. We can control the blinds with curl, so we’re already living large. The last step is connecting our HTTP interface to the Alexa Smart Device system.

A couple of things need to happen here, and I’ll only touch on them as they’re well covered in Amazon’s docs. We need to write a lambda function that exposes our device’s interface and responds to voice commands from Alexa, and then provide an OAuth endpoint. All are well documented on Amazon.

Connecting Remote Controlled Blinds to Alexa Smart Home was originally published by Stuart Hinson at Stuart Hinson on September 08, 2018.

Building Clojure projects with the Jenkins Kubernetes Plugin

2015-11-05T10:30:00+00:00

The Jenkins Kubernetes plugin is a very cool bit of tech that allows Jenkins to dynamically provision workers from a pool of Docker hosts managed by Kubernetes.

Setting up a worker with lein is straightforward. With the Kubernetes plugin installed, visit your Jenkins configuration page and add a pod template along the lines of

The Docker image url refers to this Quay.io repository, based on a Dockerfile on Github that adds lein to the base jenkinsci/jnlp-slave image.

Configure your project to use the pod and you’re set.

Note that only lein is supported currently – please submit a PR or open an issue if you’d like to see additional build tools supported!

Building Clojure projects with the Jenkins Kubernetes Plugin was originally published by Stuart Hinson at Stuart Hinson on November 05, 2015.

Understanding Pedestal Interceptors and Selectively Injecting a ClojureScript REPL

2015-05-22T10:36:00+00:00

Pedestal represents interceptors, its analog to Ring middleware, as data. Let’s look at how we can use that to conditionally inject a ClojureScript REPL into responses.

Pedestal Terminology

Two working definitions as we start

Interceptor - a record containing keys corresponding functions for execution stages. Serves an analogous role to middleware in Ring.
context - a map with all data relating to processing a request. Contains Ring request and response maps along with additional data.

Marking Routes with Metadata

We’ll start by adding metadata indicating that a browser REPL should be included in the response. We can either add it directly to an endpoint

(require '[io.pedestal.interceptor.helpers :as h]
(require '[ring.util.response :refer [response content-type]])

(def hello-world
  (with-meta
    (h/handler
     (fn [request]
       (-> (response "<html><body><h1>hello world!!</h1></body></html>")
           (content-type "text/html"))))
    {:browser-repl true}))

or to an interceptor anywhere else in the chain

(def include-browser-repl
  (with-meta
    (h/before
     (fn [context] context))
    {:browser-repl true}))

;; in the route definition
["/foo" ^:interceptors [include-browser-repl]
  {:get foo-handler}]

Interceptor Application

Interceptors serve a similar role as middlewares in Ring but the implementation is substantially different. See Pedestal’s docs for a full discussion, but for now the important difference is in how middlewares and interceptors are combined. In Ring, middlewares compose functionally

(defn middleware-ex
  [handler transform-request transform-response]
  (fn [request]
    (let [response (handler (transform-request request))]
      (transform-response response))))

(def new-handler (-> some-handler
                     (middleware-ex fn-1 fn-2)
                     (middleware-ex fn-3 fn-4)))

;; new-handler is now a fn [request] that could be
;; composed with additional middleware

Pedestal represents interceptors sequentially, adding them as a queue to the request / response context (under :io.pedestal.impl.interceptor/queue). In broad strokes, requests are processed by taking an interceptor from the queue and applying its :enter function to the context, producing a new context that the next interceptor in the queue will act on. As each interceptor is visited, it’s added to a stack in the context that will be traversed on the :leave stage. The code is well worth a read.

Knowing that the other interceptors involved in processing a request are visible in the context’s :io.pedestal.impl.interceptor/queue, we can write an interceptor that scans the queue looking for the :browser-repl metadata and injects a repl into the response if it’s found

(require '[io.pedestal.interceptor.helpers :refer [around]])

(defn browser-repl
  "produces an interceptor that scans the context's queue on enter for
  brower-repl metadata, calls inject-repl on leave if found"
  [inject-repl]
  (around
   ::browser-repl
   (fn [context] ;; fn to call on :enter
     (cond-> context
       (some->> context ;; scan the queue for :browser-repl metadata
                :io.pedestal.impl.interceptor/queue
                (map meta)
                (some :browser-repl))
       (assoc :include-browser-repl true))) ;; add a flag to inject the repl
   (fn [{response :response :as context}] ;; on :leave
     (cond-> context
       (:include-browser-repl context) ;; did we find browser-repl metadata?
       (update :response inject-repl))))) ;; add a repl!

We use around to implement browser-repl so that we can examine the interceptor queue in the before stage and set a flag in the context called :include-browser-repl that we’ll read when a response is present in the leave phase and decide whether to call inject-repl.

inject-repl is a function that handles the mechanics of modifying the response, along the lines of

(defn inject-repl
  [{body :body :as response}]
  (let [html (if (string? body)
               body
               (slurp body))
        repl-template (enlive/template
                       (enlive/html-snippet html) []
                       [:body]
                       (enlive/append
                        (enlive/html [:script "goog.require(\"pedestal_browser_repl.dev\");"])))]

    (->> (repl-template)
         (apply str)
         (response))))

Note Brian Rowe and Alex Redington suggested a couple of very nice extensions to this idea that I’d recommend studying before implementing it.

Kinda cool, right? Interceptors are data, and neat approaches fall out. We use a similar pattern for authorization.

Also, big thanks to Brian Rowe and Rick Hall for a number of improvements to this post!

Understanding Pedestal Interceptors and Selectively Injecting a ClojureScript REPL was originally published by Stuart Hinson at Stuart Hinson on May 22, 2015.

Pixie Pi

2015-05-05T12:36:00+00:00

Pixie is a Clojure-inspired lisp that has a quick startup time and nice FFI support that allows it to make use of existing C libraries, making it a nice option for working with the Raspberry Pi.

Installing Pixie

Meeting Dependencies

From a base Raspbian install, Pixie’s dependencies (as of eb5886ff95) except for libuv-dev can be satisfied using aptitude

sudo apt-get install libffi-dev libedit-dev libboost-all-dev

libuv-dev (and libuv) are available from the Jesse packages. Use the armhf builds.

Use wget to grab the packages and install with

sudo dpkg -i libuv*armhf.deb

Building the Pixie VM

make build_no_jit

It’ll run for about an hour. Enjoy the excellent ascii art the RPython build process produces.

Cross compilation from a faster machine may be an option as well.

Interfacing with the Raspberry Pi’s GPIO

Using Pixie’s FFI, I made a library called wiringPixie wrapping wiringPi. Follow the instructions on the README to install

https://github.com/stuarth/wiringPixie

and start hacking (or controlling LEDs :))

(require wiring-pixie.gpio :refer :all)

(def pin 7)

(wiringPiSetup)         ;; required first!
(pinMode pin OUTPUT)    ;; pin to OUTPUT
(digitalWrite pin HIGH) ;; set pin 7 to HIGH

Pixie Pi was originally published by Stuart Hinson at Stuart Hinson on May 05, 2015.

Error handling in Pedestal 0.4.0

2015-02-13T09:30:00+00:00

Pedestal 0.4.0 includes io.pedestal.interceptor.error/error-dispatch, which uses core.match to make error handling very tidy. For example

(def my-error-handler
  (error-int/error-dispatch [context ex]
    ;; match errors from body-params interceptor
    {:interceptor :io.pedestal.http.body-params/body-params}
    (let [response (-> (ring.util.response/response "Invalid data given") :status 400)]
      (assoc context :response response))

     ;; match FileNotFoundException in the :enter stage
    {:exception-type :java.io.FileNotFoundException :stage :enter}
    ;; respond to this error differently

    ;; pass along the error if we don't match
    :else (assoc context :io.pedestal.impl.interceptor/error ex)
    ))

creates an error handler that tries to matches any Throwable from ,

The error-dispatch macro splices its forms after the binding vector ([context ex] above) into a core.match expression matching against the ex-data of our exception, which is defined in io.pedestal.impl.interceptor, and contains :execution-id, :stage, :interceptor, :exception-type, and :exception.

So my-error-handler will respond with a 400 for any Throwable from the interceptor named :io.pedestal.http.body-params/body-param, etc.

Giving a nice separation between the semantically-correct error for an interceptor to throw (e.g. a parser error for invalid JSON) and the response that’s generated (a status 400 for invalid input).

Error handling in Pedestal 0.4.0 was originally published by Stuart Hinson at Stuart Hinson on February 13, 2015.

Configuring CSRF protection in a simple Clojure / Pedestal App

2015-01-25T10:30:00+00:00

OWASP has an excellent description of CSRF attacks and why they’re a concern. The tl;dr is, without safegaurds, an attacker can cause users to take unintended actions on your app. It’s well worth spending a few minutes on the link above if you’re not familiar.

Using `io.pedestal.http.csrf` to prevent attacks

Pedestal’s io.pedestal.http.csrf namespace implements the synchronizer token pattern that gives us the tools we need.

We’ll start with a blank pedestal-service template

lein new pedestal-service csrf

And add Pedestal’s CSRF namespace to our service

(ns csrf.service
  (:require [io.pedestal.http :as bootstrap]
            [io.pedestal.http.route :as route]
            [io.pedestal.http.body-params :as body-params]
            [io.pedestal.http.route.definition :refer [defroutes]]
            [io.pedestal.http.csrf :as csrf] ;; add this
            [ring.util.response :as ring-resp]))

The trick to using csrf/anti-forgery with a form parameter (as of 0.3.1) is to realize that the io.pedestal.http.body-params/body-params interceptor, which is responsible for parsing parameters encoded in the request body, is not included by io.pedestal.http/default-interceptors. That means naively enabling CSRF protection in your service map, e.g.

(def service {...
              ::bootstrap/enable-csrf {} ;; doesn't work with form params!
              ...})

won’t work because the anti-forgery interceptor will attempt to read from form-params that haven’t been parsed by body-params yet. Resolving it is simple, we’ll update our service definition to include

(def service {...
              ::bootstrap/enable-session {}
              ...})

and include our csrf middleware on the appropriate routes after io.pedestal.http.body-params/body-params has been applied

(defroutes routes
  [[["/"
     ^:interceptors [(body-params/body-params)
                     bootstrap/html-body
                     (csrf/anti-forgery)]
     ["/csrf-protected "{:post csrf-protected-handler}]]]])

All that’s left is to include the request’s anti-forgery token in our forms, something like

(format "<input type=\"hidden\" name=\"__anti-forgery-token\" value=\"%s\"/>"
  (::csrf/anti-forgery-token request))

A full sample’s available on Github.

Configuring CSRF protection in a simple Clojure / Pedestal App was originally published by Stuart Hinson at Stuart Hinson on January 25, 2015.

Surprising timeout chan interactions in Clojure's core.async

2014-09-14T08:56:00+00:00

note that this behavior applies to core.async 0.1.338.0-5c5012-alpha

The issue distilled

Let’s measure the time it takes to create two timeout chans, close one, and read from the other.

(let [a (timeout 100)
      b (timeout 100)
      t0 (System/currentTimeMillis)]
  (close! a)
  (<!! b)
  (- (System/currentTimeMillis) t0))
;; => 0

What happened? Why don’t we see a value ~100ms corresponding to b’s timeout?

What’s going on?

Let’s dive into the code! The relevant bit’s in timers.clj

(defn timeout
  "returns a channel that will close after msecs"
  [^long msecs]
  (let [timeout (+ (System/currentTimeMillis) msecs)
        me (.ceilingEntry timeouts-map timeout)]
    (or (when (and me (< (.getKey me) (+ timeout TIMEOUT_RESOLUTION_MS)))
          (.channel ^TimeoutQueueEntry (.getValue me)))
        (let [timeout-channel (channels/chan nil)
              timeout-entry (TimeoutQueueEntry. timeout-channel timeout)]
          (.put timeouts-map timeout timeout-entry)
          (.put timeouts-queue timeout-entry)
          timeout-channel))))

Bells are probably going off when you see TIMEOUT_RESOLUTION_MS (defined above: (def ^:const TIMEOUT_RESOLUTION_MS 10)). Let’s unpack the surrounding code. First in our let block, we have timeout, which converts the relative msecs into a clock time to expire. We then set me to (.ceilingEntry timeouts-map timeout).

timeouts-map is a java.util.concurrent.ConcurrentSkipListMap and its method ceilingEntry “Returns a key-value mapping associated with the least key greater than or equal to the given key, or null if there is no such entry”. So me will be the entry (if present) in timeouts-map with the first value that’s greater than timeout.

The unexpected behavior comes from the next line

(when (and me (< (.getKey me) (+ timeout TIMEOUT_RESOLUTION_MS)))
          (.channel ^TimeoutQueueEntry (.getValue me)))

So, if we found an me, and its key is within the tolerance of TIMEOUT_RESOLUTION_MS, we return an existing channel rather than creating a new one! This explains the behavior at the top: a and b share a channel, so closing a immediately closed b, allowing the entire block to evaluate almost instantly.

Thanks!

A huge thanks to Brian Rowe

Surprising timeout chan interactions in Clojure's core.async was originally published by Stuart Hinson at Stuart Hinson on September 14, 2014.