Richard Kallos

ETS Match Specification Pitfall: Maps

2025-02-09T01:45:00Z

During my work, I had a problem where I needed to efficiently write tree-shaped data. I also needed the ability to query ranges of the data. I reached for ETS tables, using the ordered_set table setting, and relied on match specifications to read ranges from the table.

ETS tables can have different storage backends. See the documentation for ets:new/2 for details. These backends reside within in a union type.

In many cases, an ETS match specification will scan a whole table. I specifically wanted to avoid this. The documentation for ets:match/2 says this:

If the key is specified in the pattern, the match is very efficient. If the key is not specified, that is, if it is a variable or an underscore, the entire table must be searched. The search time can be substantial if the table is very large.

I learned of an interesting caveat to the above statement; When using ordered_set tables, matching can be made more efficient by specifying a partial key!

(Note: Code snippets are based on this commit of erlang/otp)

enum ms_key_boundness {
    /* Order significant, larger means more "boundness" => less iteration */
    MS_KEY_UNBOUND           = 0,
    MS_KEY_PARTIALLY_BOUND   = 1,
    MS_KEY_BOUND             = 2,
    MS_KEY_IMPOSSIBLE        = 3
};

/*
 * This structure is filled in by analyze_pattern() for the select
 * functions.
 */
struct mp_info {
    enum ms_key_boundness key_boundness;
    Eterm least;                /* The lowest matching key (possibly
                                 * partially bound expression) */
    Eterm most;                 /* The highest matching key (possibly
                                 * partially bound expression) */
    Binary *mp;                 /* The compiled match program */
};

(source)

The analyze_pattern() function determines the ‘boundness’ of the key, and will recursively calculate the least and greatest values that can match the provided partial key.

One interesting bit of the code that calculates key boundness is as follows:

else if (key != am_Underscore && db_is_variable(key) < 0 && !db_has_map(key)) {
    *keyp = key;
    return MS_KEY_PARTIALLY_BOUND;
}

(source)

If the key contains a map, it cannot be considered fully bound. Consider this comment in the db_is_fully_bound() function:

/* Like in Erlang code, "literal" maps in a pattern match any
* map that has the given elements, so they must be considered
* variable. */

(source)

The explanation is intuitive, but the ramifications are important. If you have an ordered_set ETS table, consider changing out maps (or Elixir structs) for tuples and records instead.

I’ve encountered this issue at least twice now. By being clever about the types and ordering of ETS key terms, one can massively improve the performance of ets:select/2 calls, even when the ETS tables contain thousands of rows.

Case Study: Using CRDTs to improve a distributed system

2021-09-10T22:00:00Z

At a previous job, I was lucky enough to be faced with the challenge of improving a distributed service’s robustness to network faults. In this post, I share how I attacked the problem using Conflict-free/Convergent Replicated Data Types (CRDTs).

A Chronicle of Counters

The service was responsible for managing the state of a set of counters. These counters had lifetimes and rules attached to them, dictating when they could be incremented. Many counters needed to be incremented steadily over time. Other counters needed to be incremented slowly at the beginning, then more quickly toward the end of the counter’s life. The service didn’t care what the counters stood for. The service’s job was to inform clients when counters could be incremented, and to acknowledge and remember when clients did increment counters.

The service supported four types of requests:

Clients send (ASK, abc, 10) requests, asking the service if counter abc can be incremented by 10. The server responds with YES or NO.
Clients send (TELL, abc, 5) requests, informing the service that counter abc should be incremented by 5.
Replicas broadcast (ANNOUNCE, abc, 5) messages, informing other replicas that the counter abc has been incremented by 5.
Sometime between one hour and the next, e.g. 10:00 and 11:00, a periodic job sends (STATE, 09:59, abc, 3) to every replica, stating that the state of counter abc at time 09:59 was 3. The service does its best to track the values of counters, but for reasons I won’t get in to, the service can be wrong about the states of counters. STATE requests are a source of truth. This will be very important later.

Some other facts about the system:

High performance. Peak hours saw over 1 million QPS with 99th percentile latency under 2ms.
Servers run NTP; their physical clocks are synchronized. I won’t really get in to why this matters in this post, but having bounds on clock drift is essential when one wants servers to coordinate using Logical Physical Clocks.
ASK requests do not always turn into TELL requests.
STATE requests are not comprised of the sum of all TELL requests made to the system. It’s close, but it’s different enough to be a source of chaos.
The distribution of requests is uneven. One server may get 30% of all ASK/TELL requests for counter abc, but 0% of requests for counter xyz.
The service tries to keep counters close to their limits at any given time. Over-incrementing is a bad outcome when a counter corresponds to a monetary budget. Under-incrementing is a bad outcome when a counter corresponds to achievement of some important goal.

Before I worked on this service:

Replicas maintained connections to all other replicas. The number of open connections in the system was N(N-1)/2. O(N²) isn’t that bad; Erlang defaults to a fully connected cluster and scales to a decently high number of machines. However, we felt that we needed to do better in order to scale out the service, because…
If any connection between any two connections was broken or stalled, all replicas would resort to the safe option of saying NO to every ASK request. With O(N²) connections, the likelihood of any connection breaking and entering the bad state of saying NO to every request would increase sharply as nodes get added to the cluster.

My task was to change the service so that it could be scaled out without increasing the likelihood of entering a bad state. To achieve this, I sought to:

Remove the requirement for full network connectivity for saying YES to ASK requests.
Relax the requirement from needing fresh state from all replicas to only needing fresh state from a majority of replicas.

Data Meets Math

One of the first things I noticed about the ANNOUNCE protocol was that it was very sensitive to messages being dropped. I changed the data in ANNOUNCE messages from being “everything a replica learned since the previous broadcast” to “everything a replica knows about the state of the system”. The ANNOUNCE messages got much larger in size, but I strongly believe the properties of these larger messages were worth the cost.

Messages changed from being independent events to being members of a partially ordered set. The message {(abc, 10), (xyz, 5)} could be considered greater than {(abc, 8), (xyz, 3)} or {(abc, 1)}, but not greater than {(xyz, 8)}. Orders, even partial ones, are transitive.
It became possible to ‘merge’ messages according to some least upper bound function. Given the messages {(abc, 10)} and {(xyz, 5)}, the LUB is the union; {(abc, 10), (xyz, 5)}. For sets of pairs where the first element is the same, a common LUB is to take the pairwise maximum, for example: {(abc, 10), (xyz, 5)} ⊔ {(xyz, 8)} would be {(abc, 10), (xyz, 8)}. In my case, I also had to account for replicas, so my messages resembled {(A, abc, 10), (A, xyz, 5), (B, abc, 3)}, where the state of counter abc would be 10 + 3 = 13. This merge function is associative, commutative and idempotent. Associativity can be useful when you have to merge a lot of messages (merging messages in a tree shape can be faster than merging in a list shape), but commutativity and idempotency were essential to making the counter service protocol more robust.
A partially ordered set with a least upper bound function is a join-semilattice. Semilattices are monotonic, which is another great property for data flowing in a distributed system.
A join-semilattice with an ‘update’ function, like a TELL message, can be considered a State-based CRDT, a form of data that provides strong eventual consistency; which means that once all replicas exchange and merge their state-based CRDTs; the replicas have equivalent state. Eventually consistent systems might need expensive reconciliation protocols, but this is not necessary in a system with strong eventual consistency.

Once applying ANNOUNCE messages to local state became idempotent, it became safe to retransmit messages without fear of introducing inconsistencies.

Once applying ANNOUNCE messages to local state became commutative and monotonic, it became almost harmless for the network to drop or reorder messages. ANNOUNCE messages contained the full knowledge of some replica, so it became easy to determine which ANNOUNCE message was the most recent; the greatest/maximum message according to our partial ordering. As long as ANNOUNCE messages were delivered in a timely manner, it did not matter if individual messages were dropped or reordered.

Once ANNOUNCE messages to local state became transitive, the service no longer relied on direct connections between replicas. If replica A ANNOUNCE’d at replica B, then replica B ANNOUNCE’d at replica C, the state would be the same as if replica A had also ANNOUNCE’d at replica C. Transitivity in ANNOUNCE messages relaxed the connectivity graph requirement from needing to be fully connected to needing to be strongly connected.

Problems in Paradise

Idempotency, transitivity, commutativity and monotonicity are all excellent properties to have in a distributed system protocol. However, there are obstacles that prevent CRDTs being used in production everywhere. I encountered two specific issues:

Counter state was not always monotonic. As I mentioned earlier, STATE messages were a source of truth that often declared that a counter’s state was lower than the service expected.
How could I shrink a message that was supposed to be monotonic?

Both problems were solved by the fact that STATE messages corresponded to some ground truth rooted at a specific point in the past.

STATE messages always reflected the true state of the system at the end of some hour. By changing my (Replica, Counter, State) triples into (Replica, Counter, Hour, State) tuples, and relying on the fact that STATE messages are always delivered to all replicas, a replica could stop sending tuples where Hour was before the latest STATE message. STATE messages helped put a practical bound on the size of ANNOUNCE messages.

Splitting counter state into hourly buckets also helped to account for nonmonotonic updates from STATE messages; While counter values are monotonically non-decreasing in ANNOUNCE messages, the Hour element of STATE messages were also monotonically non-decreasing. and I was able to change my least upper bound operation into a lexicographic least upper bound, which I feel is best explained by example: (0, 5) < (0, 500) < (1, 0). Applied to STATE messages, (08:59, 5000) < (09:59, 3).

Mo’ computers? No problem.

With that final change, I was finished. After writing, testing, and deploying the code to production, I was delighted to see that when a single instance of the service went down, whether due to a network partition, or perhaps a deployment, the rest of the cluster was healthy and answering YES to ASK requests.

I had a lot of fun working on that problem. I also had a really great time researching CRDTs and eventual consistency, implementing two different distributed system simulators, and eventually implementing the changes to the protocol. I’m hoping to write another post in the near future showcasing my favorite papers around this topic.

For the sake of brevity, many fascinating details about the service and protocol have been left out. I hope nevertheless that you remember and possibly consider CRDTs when you find yourself tasked with making some distributed system more robust.

Generating code with jq

2021-07-12T21:15:00Z

When I first learned about jq, I thought it was a handy tool for extracting from JSON. Little did I know that I was only scratching the surface of what jq can do.

jq is a “lightweight and flexible command-line JSON processor”. When I first heard of the tool, I mainly saw it being used for extracting relevant data from large JSON objects. If I had a sequence of large JSON objects, I could pipe them through a small jq program to extract the key(s) that I wanted.

$ jq '.first.inside_first.thing' < my_big_object.json

What a relief! No more trying to massage the json into a form that could be mined with sed or awk! No more writing and debugging Python scripts to transform JSON! No more copying and pasting into my browser’s dev console!

If your interest is piqued, I highly recommend going and watching Charles Chamberlain’s delightful !!Con 2017 talk about jq. It’s short (9 minutes 32 seconds), engaging, and shows glimpses at the true power of jq. If your interest is not piqued, I recommend watching the video anyway. Maybe it’ll pique your interest.

I happily added jq to my toolbox, and used it for all my JSON parsing and extracting needs.

Fast forward to earlier this year. I started writing a Pony port of the Mustache templating language. What I thought was kinda cool is that the test suite for mustache implementations is written in YAML and JSON.

In retrospect, this approach makes sense to me. If you have a specification of some pure function, and you want to make it easier for implementers to write implementations of your function, you could do worse than to use an extremely common data representation format to represent your tests. With specification tests in JSON, implementers might have an easier time testing their implementations as long as their language can parse JSON. I know I had an easier time because of this.

I found myself with a bunch of JSON objects representing tests. What I wanted were a bunch of Pony classes representing tests. Thinking about my problem, I remembered a tweet by Kate, describing how she used jq to generate C code.

In Pony, lists of tests are collected by actors (or possibly classes) that implement the TestList interface. A TestList has a function tests() that accepts a PonyTest object, and calls the PonyTest.apply() behavior with the name of each test class in the list. With these incoming messages, the PonyTest actor knows what tests to run. Here’s an example.

TestLists can also be comprised of other TestLists. Mustache’s tests are categorized, and so I adopted the same categorization for my tests.

Some of these lists of tests are quite long, and I did not want to write and maintain all these tests without any assistance. jq to the rescue!

jq programs are essentially composed of filters that operate on some input JSON. I have some preamble to add to my Pony test files, so I start with a string literal, which can be thought of as a filter that ignores its input and produces the desired string. I then use the comma operator, which concatenates the results of two filters together, and end with a filter that extracts and transforms the names of each test into an appropriate name for a private Pony class.

#!/usr/bin/jq -Mfr

# meant to be used with comments.json from
# https://github.com/mustache/spec

def testClass(n): "_Test" + (n | gsub("[() -]"; ""));

"use \"ponytest\"
use \"json\"
use \"../..\"

actor Main is TestList
  new create(env: Env) => PonyTest(env, this)

  new make() => None

  fun tag tests(test: PonyTest) =>",
(.tests[].name | ("    test(" + testClass(.) + ")")),

The only remaining thing to do is to generate the code for each class. We have a function for coming up with the name of a Pony test class. We have the templates, variable bindings, and expected results from the JSON test cases. I injected JSON string literals into the test files, and used Pony’s JSON package to parse those literals into Pony data.

(.tests[] | ("
class iso " + testClass(.name) + " is UnitTest
  fun name(): String => \"interpolation/" + .name + "\"

  fun apply(h: TestHelper) ? =>
    let template = " + (.template | @json) + "
    let expected = " + (.expected | @json) + "
    let data = (recover val
      JsonDoc.>parse(\"\"\"" + (.data | tojson) + "\"\"\")?
    end).data

    let m = Mustache(template)?
    h.log(m.print_tokens())

    h.assert_eq[String](expected, m.render(data))"))

Lastly, to generate a Pony file for the tests in interpretation.json, I pass interpretation.json through my jq script.

$ ./gen.jq < /path/to/interpretation.json > interpretation/_test.pony

Repeating these steps for the remaining Mustache test suites left me with Pony files comprised of Pony classes for each of the tests. After using these tests to debug my implementation of Mustache in Pony, my next task is to improve the API to make it easier for everyday use.

Hopefully you’ve learned that jq is useful for more than shell pipelines and one-liners! If you’d like to learn more, I strongly recommend reading through jq’s manpage. It’s dense, but comprehensive and precise. Thanks for reading!

What my choices told me about my priorities

2020-04-04T22:57:41Z

This was a useful exercise for me. I think I’ll try and check in again sometime soon. Here were my thoughts from today.

I was prioritizing long-term storage

When I started using plain text files to store my notes as Org mode files (which I am trying to replace)
When I started using TiddlyWiki, which can be opened/reused wherever I have access to a browser that executes Javascript. It doesn’t necessarily have the longevity of plain text, but it’s close.

I was prioritizing (re-)reading and reflection

When I stopped transcribing my notes into Org mode file, where I would never happen across them unless I specifically went looking for them.
When I started transcribing my handwritten notes into TiddlyWiki, breaking my notes into smaller, more reusable chunks (tiddlers), organizing them all, and making small changes to TiddlyWiki’s interface to encourage exploration.

I am prioritizing focus

When I step away from the computer to think with pen and paper.
When I write notes by hand while reading papers or watching talks.
When I close Slack after realizing I’ve wasted too much time clicking on unread channels and not enough time working.

A Bit of Math: Triple Elimination Tournament

2019-12-22T14:58:40Z

A while ago, my father, who is a competitive backgammon player, had a puzzle for me. He wanted to know the fraction of players remaining after a certain number of rounds of play in a triple-elimination tournament. This blog post goes over my work to figure out the answer.

A triple-elimination tournament is more-or-less how it sounds. If you lose 3 times, you’re eliminated. Since the problem doesn’t need to deal with any sort of bracket structure, I’m leaving it out completely.

In any triple-elimination tournament, there are 3 groups of players; those with zero, one, and two losses. The number of players with zero losses is easiest to figure out. After each match, there is one winner and one loser. After the first round, half of the players won, and half of the players lost. After the second round, half of the winning players won again. The proportion of players with zero losses after x rounds of play is

The number of players with one loss is a little trickier. After the first round, half of the players lost the first game. After the second round, the fraction of players with one loss is the sum of those who lost their first game and won the second, and those who won their first game and lost the second. Each of those groups comprise one quarter of the total number of players, so the fraction of players with one loss at the end of the second round is still half. After the third round, half of the half of players with one loss won their third game, and half of the quarter of players with zero losses lost their third game, so the fraction of players with one loss after round three is (1/4) + (1/8) = (3/8) After the fourth round, half of the three eighths of players with one loss won their fourth game, and one eighth of the players with zero losses lost their fourth game.

If we expand a little further, a clear pattern emerges. 1/2, 2/4, 3/8, 4/16, 5/32. The proportion of players with one loss after x rounds of play is

Finally, there are the players with two losses. Seeing as it’s not possible to lose twice in zero or one rounds of play, I’m going to start counting at x = 2 rounds.

After two rounds, one in four players lost twice. After the third round, half of those players lost for the third time, resulting in their elimination, and half won, giving one in eight players. These players are joined by the quarter of players with one loss who lost their second match, giving 3/8. After the fourth round, half of the 3/8 won, giving 3/16. These lucky few are joined by 3/16 of people with one loss who lost their fourth match, giving 6/16. After the fifth round, half of the 6/16 won, giving 6/32, who are joined by the 4/32 of people with one loss who lost their fifth match, giving 10/16. If we look at the number of rounds, and the sequence of values in the numerator, we see [(2, 1), (3, 3), (4, 6), (5, 10)], which matches the following equation:

That means the proportion of people with two losses is equal to

Now, the number of people in the tournament after x rounds (where x is greater than 2) is equal to

Presentation: Hybrid Logical Clocks @ PWLMTL

2019-10-05T11:22:37Z

This past thursday (2019-10-03), I presented this paper at Papers We Love Montreal. I had a really fun time!

Here are my slides.

Imperium in Imperio - A Bridge is Made of Planks; Every Plank is a Bridge

2019-04-22T10:30:00Z

In the previous three installments, I discussed visualizing ideal final states of things, examining the current state of your life, and how to bridge the two. In the final post in this series, I show how these techniques can be applied at any scale; from a daily to-do list to planning the course of your entire life.

In the previous post, I showed how how I write Structural Tension charts. The paragraph on top is where I write the ideal state of some thing that I want to accomplish, whether it’s a finished project, an instilled habit or an attained achievement. The paragraph on the bottom is where I describe as objectively and nonjudgmentally as possible the current reality of the thing I’m setting out to accomplish. The space in between the two paragraphs gets filled with a list of actionable steps I can take to move from where I am to where I want to be.

To me, these steps resemble planks on a bridge. The easier and smaller steps tend to go down near the bottom of the page, and the later tasks tend to be a bit more abstract, large in scope, and usually depend on previous tasks. Sometimes the steps I write out are really big items, like “Get a job as a software developer”. Writing this down doesn’t fill me with the will to go out and get things done. How do I even go about starting such a huge task? The answer: Make a ST chart for that task. If there are any tasks in that new chart that are too large, you can make yet more charts. In the end, you wind up with a forest (in the graph theory sense) of charts that all serve to plot a course for your life.

For example, the ST chart I shared with you in the previous post could have been a single step in a larger chart where I set a course to gain better insight to my emotions, and the step titled “Experiment with active forms of meditation” could have its own chart where I describe the styles I’ve tried, what’s gone well, and what has yet to be tested. Furthermore, I could have a completely separate chart about setting up a regular journaling habit where I list “Try keeping a meditation journal” as one of the steps.

So far the method that I’ve had the most success with is to keep track of these charts on paper, but as much as I deeply enjoy the feel of pen and paper, I can’t help but think that organizing these charts is a task that computers are well-suited to. I’ve noticed there is a lot of software that could be great at managing these ST charts, but I think the two most promising ones are Emacs and TiddlyWiki. I’ll hopefully have more to say in the future about if/how I’ve adapted my system to allow for the help of software.

In conclusion, the process of writing ST charts may lead you to write out steps that are a bit too large to tackle on their own. Fortunately, you can take advantage of the recursive nature of ST charts and break each step down into their own chart, and it becomes easier to plan out projects or ambitions of any size.

Inveniam viam - Building Bridges

2019-03-18T23:00:00Z

If you know where you are and where you want to be, you can start to plot a course between the two points. Following on the ideas presented in the previous two posts, I describe a practice I learned from reading Robert Fritz’s book Your Life as Art.

I spent a few years of my life voraciously consuming self-help books. I believe the journey started with Eckhart Tolle’s The Power of Now, and it more-or-less ended with Meditations, Stoicism and the Art of Happiness, and Your Life as Art. I’ll probably wind up writing about my path to Stoicism some other time. This post is about what I learned from Robert Fritz.

Your Life as Art is filled with insight about navigating the complex murky space of life while juggling the often competing aspects of spontaneity and rigid structure. My collection of notes about Your Life as Art is nearly a thousand lines long, and there’s definitely far too much good stuff to fit into a single blog post. At this point, I’ll focus on what Robert Fritz calls structural tension, and his technique for plotting a course with the help of a chart.

Hopefully I convinced you with the previous two posts about the importance of objectively “seeing” where you are and where you want to be. These two activites form the foundation of what Fritz calls structural tension, a force that stems from the contrast between reality and an ideal state and seeks to relieve the tension by moving you from your present state to your ideal state.

Writing is a handy exercise generating this force. A structural tension chart (or ST chart) has your desired ideal state at the top of a page, your current state at the bottom, and a series of steps bridging the gap between the two. First you write the ideal section, then the real section, and finally add the steps in the middle. It’s very important to be as objective and detailed as possible about your ideal and current states. Here’s an example:

--- Ideal ---
I meditate every day for at least 15 minutes. My mind is calm
and focused as I go about my daily activities. I feel comfortable
sitting for the duration of my practice, no matter how long it is.
-------------

- Try keeping a meditation journal
- Experiment with active forms of meditation
- Experiment with seating positions
- Give myself time in the morning to meditate
- Wake up at the same time every day

--- Real ----
I meditate approximately once per week. I have difficulty
finding a regular time during the day to devote to meditation,
making it difficult to create a habit that sticks. I find that
I become uncomfortable sitting with my legs crossed for more
than 5 minutes. I do not often remember how good I feel after
meditating, which results in difficulty deciding to sit.

If you’re interested in reading more, I highly recommend Your Life as Art. Robert Fritz’s books are filled with great ideas. While this is basically a slightly more detailed to-do list, I find the process to be very grounding.

In conclusion, once you know where you are and where you want to be, try writing a structural tension chart in order to set a course.

Esse quam videri - Seeing what's in front of you

2019-03-17T20:00:00Z

As humans, we are easily fooled. Our five senses are the primary way we get an idea of what’s happening around us, and it’s been shown time and time again that our senses are unreliable. In this post, I try to explain the importance of ‘seeing’ what’s in front of you, and how to practice it.

See this video a classic example of our incredible ability at missing important details.

Rembrandt used to train his students by making them copy his self-portraits. This exercise forced them to see their subject as objectively as possible, which was essential to make an accurate reproduction. Only after mastering their portraiturne skills did Rembrandt’s students go on to develop their own artistic styles.

It is important to periodically evaluate your position and course in life. It’s something you do whether you’re aware of it or not. When you plan something, you’re setting a course. When you’re reflecting on past events, you’re estimating your position. For the sake of overloading words like sight, vision, and planning, let’s refer to this act as life portraiture.

Life portraiture can be compared to navigating on land, air, or sea, except the many facets of our lives results in a space of many more dimensions. We can consider our position and course on axes like physical health, emotional health, career, finance, and social life. If we want finer detail, we can split any of those axes into more dimensions.

Objective life portraiture is not easy. We are all vulnerable to cognitive biases. Following the above analogy with navigation, our inaccuracy at objectively evaluating our lives is akin to inaccurately navigating a ship or airplane. If you’re not well-practiced at seeing, your only tool for navigation might be dead reckoning. If you practice drawing self-portraits of your life, you might suddenly find yourself in possession of a sextant and an almanac, so you can navigate using the stars. The ideal in this case would be to have something like GPS, which might look like Quantified Self with an incredible amount of detail.

It’s worth mentioning that our ability to navigate varies across different dimensions. This is an idea that doesn’t really carry over to navigating Earth, but it’s important to recognize. For example, if you’re thorough with your personal finances, you could have tools akin to GPS for navigating that part of your life. At the same time, if you don’t check in with your emotions, or do anything to improve your emotional health, you might be lost in those spaces.

There are ways to improve our navigating abilities depending on the spaces we’re looking at. To improve navigating your personal finances, you can regularly consult your banking statements, make budgets, and explore different methods of investing. To improve navigating your physical health, you can perform one of many different fitness tests, or consult a personal trainer. To improve navigating your emotional health, you could try journaling, or maybe begin seeing a therapist. Any and all of these could help you locate yourself in the vast space where your life could be.

In order to get where you want to go, you need to know where you are, and what direction you’re moving in.

Memento Mori - Seeing the End

2019-03-16T17:30:00Z

Memento Mori (translated as “remember death”) is a powerful idea and practice. In this post, I make the case that it’s important to think not just about your death, but to clearly define what it means to be finished in whatever you set out to do.

We are mites on a marble floating in the endless void. Our lifespans are blinks in cosmic history. Furthermore, for many of us, our contributions are likely to be forgotten soon after we rejoin the earth, if not sooner.

This is great news.

Whenever I’m feeling nervous or embarrassed, I start to feel better when I realize that nobody in front of me is going to be alive 100 years from now, and I doubt that they’ll be telling their grandchildren about that time when Richard made a fool of himself, because I try to make a fool of myself often enough that it’s usually not worth telling people about.

Knowing that we are finite is also pretty motivating. I feel less resistance to starting new things. It doesn’t have to be perfect, in fact, it’s probably going to be average. However, it’s my journey, so it’s special to me, and I probably (hopefully?) learned and improved on the way.

It’s important to think about the ends of things, even when we don’t necessarily want things to end. Endings are as much a part of life as beginnings are, to think otherwise is delusion. Endings tend to have a reputation for being sad, but they don’t always have to be.

For example, some developers of open source software get stuck working on their projects for far longer than they expected. It’s unfortunate that creating something that people enjoy can turn into a source of grief and resentment.

Specifying an end of any endeavor is an important task. If no ‘end state’ is declared, it’s possible that a project will continue to take up time and effort, perpetually staying on the back-burner of things you have going on, draining you of resources until you are no longer able to start anything new.

Spending time thinking about what your finished project will look like sets a target for you to achieve, which is a point I’ll elaborate on very soon. This exercise, along with evaluating where you are currently on your path toward achieving your goal/finishing your project, are immensely useful for getting your brain to focus on the intermediate tasks that need to be finished in order to get to that idealized ‘end state’.

All in all, while it’s sometimes nice to simply wander, it’s important to acknowledge that you are always going somewhere, even when you think you’re standing still. You should be the one who decides where you go, not someone else.

In Defense of gen_event

2018-05-22T18:00:00Z

gen_event has drawn ire from Erlang and Elixir developers for a while. Many posts have been written, and more than one conference talk has been given on various alternatives to gen_event. In this post, I make the case that despite its warts and age, gen_event is still a very useful module to know. I also make the case that its core design is ultimately a lot more flexible than many people give it credit for.

Certain design decisions give rise to flexible systems. For example, Erlang’s asynchronous sending of messages and synchronous receiving of messages with timeouts offer developers the full range of message passing behaviour. For synchronous sending, send your message, ideally with a unique reference, then do a receive, pattern matching on the reference you sent. This is basically what goes on when you use gen_*:call. For asynchronous receives, you can use a receive block with after 0 to instantly time out if no messages can be matched against. Here is some code:

  sync_send(Pid, Msg) ->
      Ref = make_ref(),
      Pid ! {Ref, Msg},
      receive
          {Ref, Reply} -> Reply
      end.

  async_recv() ->
      receive
          M -> {ok, M}
      after
          0 -> none
      end.

Question: If Erlang gave you synchronous sends and asynchronous receives, how would you go about implementing asynchronous sends and synchronous receives?

The core design of Erlang’s message passing is flexible enough to express the message passing semantics that it didn’t implement by default. Please keep this in mind as we turn our attention to gen_event.

For those of you who are unfamiliar with gen_event, it works more-or-less like this: gen_event:start_link starts what’s called an event manager process. When you write a module that implements the gen_event behaviour, you are writing what’s called an event handler. An event manager has a list of installed event handlers, each with their own state. when a process calls gen_event:notify, the event manager will invoke its installed handlers one at a time (after all, the event manager is just one process). I’ll repeat for emphasis; an event manager process executes all of its installed event handlers one at a time. This decision draws some ire, particularly from José Valim, who has a blog post on how to replace gen_event with a supervisor (as the event manager) and a bunch of gen_servers (as the event handlers).

I know I’m strange, but I think that the default behaviour of gen_event is perfectly fine, and ultimately much more flexible than the solution with concurrent handlers as suggested by Jose. If you want events to be handled concurrently, then you can trivally implement fanning out by installing event handlers that forward messages to existing processes or spawn processes to execute the event handling code. However, if conccurent handling of events is the default behaviour, how would you set about implementing sequential handling of events? I suspect that you’ll basically wind up implementing a less efficient version of gen_event; less efficient because you’ll be sending unnecessary messages.

Question: If gen_event ran handlers concurrently by default, how would you go about running handlers sequentially?

In conclusion, gen_event, much like Erlang’s message passing, provides a flexible base upon which you can easily implement different behaviour. I’m happy that gen_event exists and that it was decided that sequential handling of events would be the default behaviour.

Stream

2018-05-09T18:30:00Z

A very common piece of programming wisdom that gets passed around is to think before you code. My favorite expression of this idea is “Weeks of programming can save you hours of planning”. That being said, not all thinking/planning is equal. In this post, I share a creative writing technique I’ve put into practice to great effect at work; freewriting.

The idea is simple; you write down your thoughts as they arise. Forget about editing. Forget about grammar. Just write down a stream of consciousness. I prefer to use pen and paper for this, but I find the practice equally effective at the computer. I usually start each exercise with a sentence like “I don’t know what to do for this thing”, then go into detail about the problem, as well as any solutions that crop up and their respective tradeoffs. After a while, I find that as the words fill the page, my mind becomes more focused on the problem. More often than not, I come up with an effective solution to the problem I describe toward the beginning of the stream.

I’ve started freewriting whenever I feel stifled at home or at work. I find it to be a great way to break away from the computer while still thinking deeply about programming. In addition, I relish every opportunity to write with my fountain pens.

One reason why I think the technique is so effective is that you quickly begin using System 2 (borrowing Daniel Kahneman’s terminology) to digest the task at hand. Furthermore, by writing down your thoughts, you make yourself more resilient to distractions. Suppose someone comes by and ask you a question. If you’ve been writing down your thoughts, you can simply re-read what you’ve written in order to recover your train of thought. This can avoid situations like this (credit to Jason Heeris).

For your viewing pleasure, here (830KB) is a photo of one of my sessions. TL;DR: I’ve been wrestling with how to write a a good API for a set of NIFs I’m writing for a fast metrics library. In the end, I decided it’s a better idea to focus on implementing static configuration and putting machinery in place to allow for property testing.

Introducing wrek, a miniature Erlang graph engine

2018-04-22T15:15:00Z

This is a copy of this article, kept for posterity on my blog.

I gave a talk about Wrek at Code BEAM SF last month. You can view it here.

Wrek is an Erlang library I wrote for concurrently executing task dependency graphs. It’s intended purpose is to run a set of pre-defined tasks that have a partial ordering between them. In this post, I explain why I wrote wrek, what it can be used for, and how to use it.

Motivation

Wrek emerged as the result of two distinct forces. First, my amateurish enjoyment of graph theory made me try to see graphs wherever I could, laying the conceptual foundation for this library. Second, I realized that a project I was working on at Adgear would benefit from such a library, causing me to finally start writing Wrek.

Conceptual

The graph is a pervasive data structure in computing. When I learned about graph algorithms in school, I was amazed by their broad applications. Graphs play essential roles in compilers and build systems. Various graph algorithms form the backbone of how we communicate with each other over the internet.

Our everyday lives tend to be filled with lists. We have to-do lists, shopping lists, recipes, checklists, instructions, and much, much more. One day, I realized that some of these lists are deceiving. Some of these lists are actually graphs hiding in list’s clothing; these lists are topological orderings of directed acyclic graphs. Two of the more obvious cases of this are to-do lists and recipes. You can’t send a letter that you haven’t written yet, nor can you cook spaghetti if you haven’t boiled a pot of water.

Once the sneaky graphs were discovered, I spent some time thinking about how to benefit from treating various lists like the DAGs that they secretly are. The clearest analogies between these lists and DAGs were to-do lists and recipes. These very closely resembled dependency graphs. Unlike in their list representation, these dependency graphs show which vertices (individual tasks) can be executed concurrently. Sometimes this is obvious (e.g: I can chop vegetables while I wait for the pot of water to boil! I can begin preparing a second baking sheet of cookies while the first batch is in the oven!), but I had my hopes that the act of reformulating lists as dependency graphs could expose more opportunities for concurrency.

                Boil water -- Add pasta -- Cook pasta --.
                                                         \
  Purée tomatoes --.                                      \
                    \                                      \
  Chop vegetables -- Combine in saucepan -- Simmer sauce -- Combine pasta and sauce -- Serve
                    /
      Add spices --'

Edges between vertices in the above graph express a partial ordering. Vertices with no path between them can be executed concurrently. The relationship between vertices in the graph reflects the truth in the kitchen. We can cook the pasta while our sauce is simmering. It doesn’t matter whether we chop vegetables or purée tomatoes first; they both need to be done if we’re going to make any sauce.

Despite my best efforts, I am still a disaster in the kitchen. As a means of changing this inconvenient fact, I thought it would be fun to try representing recipes as directed graphs, with extra data like how long each step can be expected to take. This sat in my list of ‘someday projects’ for a long time, and I was only reminded of it when I begin to think about a project I was working on at $JOB.

Practical

One of the projects I accomplished last year at Adgear was removing an expensive computation that was taking place on each of our already maxed-out edge servers, and replaced it with a system that performed the expensive computation on a single machine, then distributed the result. The system worked when nobody touched it, but it was the programming equivalent of twigs and duct tape; cron jobs and shell scripts. This system continued to be painful to use, and more examples of expensive computations being done on CPU-starved machines were cropping up. At this point, it made sense start writing a more robust system.

These expensive computations decomposed nicely into lists of steps to be performed. Fetch this data, transform it, transform it some more, send some of the data to this group of servers, send this other data to some other group of servers. Shell scripts are pretty good at encoding these pipelines, so it was an acceptable choice at the time of implementation. After a while, it dawned on me that these lists of steps were not really lists; they were dependency graphs. I tried to expose the latent concurrency in the shell scripts by spicing them up with some background jobs and waitpid, but it was decided that it would make more sense to switch to Erlang and benefit from all that OTP has to offer.

Let’s get `wrek`ed

(I’m really sorry. I couldn’t resist.)

Wrek accepts dependency graphs like the above spaghetti recipe as input, and executes each vertex. The nature of concurrently executing actions as soon as they are able to execute is generic to any problem that can be represented by a dependency graph. The structure of a dependency graph, as well as the tasks involved in each of the graph’s vertices are specific to the user’s wishes. Following the same generic/specific split as OTP; the generic portion of this class of problems is solved by wrek and wrek_vert modules. The specific portion is provided by the user in the form of an Erlang map describing each vertex in the graph, and a set of callback modules that implement the wrek_vert behaviour.

The wrek_vert behaviour consists of a single callback run/2, where the first argument is a list of arguments to be sent to the callback function, and the second argument is the ID of a process which can provide information generated by other vertices. The expected result of this callback function is either {ok, Any :: any()} or {error, Reason :: any()}. If the callback function succeeds, Any will be taken by wrek and made available to other wrek_vert processes. If the callback function crashes or returns an error, the entire graph will be shut down.

Making Erlang Pasta

Following the contrived example above, let’s set about using wrek to make pasta. Of course, our program won’t really make pasta, but it’s output ought to fool somebody.

Looking at the above graph, each step seems to do one of three things:

Add an ingredient
Combine ingredients in a vessel
Do something to ingredients in a vessel

Let’s go ahead write some wrek_verts for each of those actions. If you don’t feel like following along in a text editor, the full code can be found here.

-module(cook_add).

-behaviour(wrek_vert).
-export([run/2]).

run([Ingredient, Quantity], _Pid) ->
    io:format("adding ~s. amount: ~s.~n", [Ingredient, Quantity]),
    {ok, #{added => [{Ingredient, Quantity}]}}.

That’s all for cook_add. It prints a message, then produces a map with a key added whose value is a proplist with a single pair.

-module(cook_heat).

-behaviour(wrek_vert).
-export([run/2]).

run([Verb, Noun], _Pid) ->
    io:format("~ping ~p.~n", [Verb, Noun]),
    {ok, #{}}.

cook_heat is also pretty short. It’s also very abstract. It could be used to print a message about Verbing any Noun, not just cooking ingredients!

Our final callback module is a little longer, because it does a little bit more than printing a message.

-module(cook_combine).

-behaviour(wrek_vert).
-export([run/2]).

run([Ingredients, Vessel], Pid) ->
    Fun = fun(Step, Acc) ->
              Stuff = wrek_vert:get(Pid, Step, added),
              io:format("combining ~p with ~p in ~p.~n", [Stuff, Acc, Vessel]),
              Stuff ++ Acc
          end,
    Stuff = lists:foldl(Fun, [], Ingredients),
    io:format("~p now contains: ~p.~n", [Vessel, Stuff]),
    {ok, #{added => Stuff}}.

Ingredients is expected to be a list of vertex names. We finally use our parent process’s Pid as an argument to wrek_vert:get/3. This lets us consume the data produced by the cook_add callback module. After combining everything, we returns a new collection of ingredients.

Alright! We’re nearly done describing the specific portion of our problem! The last step is to represent our dependency graph in terms of these callback modules and the arguments we want to pass to them.

-module(wrek_example).

-export([make_pasta/0]).

make_pasta() ->
    application:ensure_all_started(wrek),
    Graph = #{
      tomatoes => #{
        module => cook_add,
        args => ["pureed tomatoes", "1 can"],
        deps => []
       },
      vegetables => #{
        module => cook_add,
        args => ["chopped vegetables", "lots"],
        deps => []
       },
      spices => #{
        module => cook_add,
        args => ["spices", "to taste"],
        deps => []
       },
      saucepan => #{
        module => cook_combine,
        args => [[tomatoes, vegetables, spices], saucepan],
        deps => [tomatoes, vegetables, spices]
       },
      simmer_sauce => #{
        module => cook_heat,
        args => [simmer, sauce],
        deps => [saucepan]
       },
      boil_water => #{
        module => cook_heat,
        args => [boil, water],
        deps => []
       },
      add_pasta => #{
        module => cook_add,
        args => ["pasta", "1 handful"],
        deps => [boil_water]
       },
      cook_pasta => #{
        module => cook_heat,
        args => [cook, pasta],
        deps => [add_pasta]
       },
      mix_pasta_with_sauce => #{
        module => cook_combine,
        args => [[saucepan, add_pasta], saucepan],
        deps => [simmer_sauce, cook_pasta]
       }
     },
    wrek:start(Graph).

That’s an eyeful! What we’ve done here is created an Erlang map whose keys represent the names of vertices in our original dependency graph, and whose values are maps that specify a callback module, arguments to pass to the callback module, as well as any dependencies the vertex may have. I congratulate those of you who noticed that we are never straining the pasta; I did confess to being a disaster in the kitchen. I promise I learned my lesson.

Alright, we’re done coding! Let’s start up a shell and make some pasta!

$ rebar3 shell
[...]
1> wrek_example:make_pasta().
adding spices. amount: to taste.
adding puréed tomatoes. amount: 1 can.
adding chopped vegetables. amount: lots.
adding pasta. amount: 1 handful.
{ok,<0.133.0>}
2> boiling water.
cooking pasta.
combining [{"puréed tomatoes","1 can"}] with [] in saucepan.
combining [{"chopped vegetables","lots"}] with [{"puréed tomatoes","1 can"}] in saucepan.
combining [{"spices","to taste"}] with [{"chopped vegetables","lots"},
                                        {"puréed tomatoes","1 can"}] in saucepan.
saucepan now contains: [{"spices","to taste"},
                        {"chopped vegetables","lots"},
                        {"puréed tomatoes","1 can"}].
simmering sauce.
combining [{"spices","to taste"},
           {"chopped vegetables","lots"},
           {"puréed tomatoes","1 can"}] with [] in saucepan.
combining [{"pasta","1 handful"}] with [{"spices","to taste"},
                                        {"chopped vegetables","lots"},
                                        {"puréed tomatoes","1 can"}] in saucepan.
saucepan now contains: [{"pasta","1 handful"},
                        {"spices","to taste"},
                        {"chopped vegetables","lots"},
                        {"puréed tomatoes","1 can"}].

It isn’t pretty, but neither is most of my cooking. However, we’ve just used wrek to teach Erlang how to concurrently prepare pasta! If you run wrek_example:make_pasta() over and over, you may notice that the order of the output changes, but never enough that the steps followed in order don’t produce valid pasta.

Uses for wrek, additional features, and plans for the future

I believe that Wrek is useful for more than pretending to make pasta. I used Wrek at $JOB to concurrently execute various small shell scripts, collect their output, and pass up to Erlang through and down to other small shell scripts.

Wrek also contains features that I haven’t covered in this article. For instance, you can pass a gen_event process to wrek:start/2 to get Wrek to call gen_event:notify/2 when various events take place. This gives users a way to monitor the execution of their graphs. Combined with Sergey Aleynikov’s erlexec library, you can collect stdout and stderr from any shell command you run in a vertex using wrek_exec:exec/4. I invite interested readers to take a peek at the source code for Wrek.

Wrek is still a very young project, and I am still inexperienced at writing libraries. I am virtually certain that the API is going to go through significant changes in the near future. I believe this is for the best, because the whole process of specifying callback modules and passing data between vertices can be improved dramatically. Here are some of the ideas I have for improving Wrek:

Add support for limiting the number of active wrek_vert processes
Avoid forced namespacing by vertex names, perhaps through the use of ETS tables as a tuple space
(experimental) Describing dependency graphs by annotating vertices by the data they produce and consume, rather than by explicitly naming other vertices as dependencies.

Conclusion

I had a really nice time writing Wrek, and I’m excited to be using it in production at Adgear. I am also eagerly looking forward to presenting Wrek at Code BEAM SF in March 2018.

Thank you very much for reading!

P.S: As I mentioned above, this is a copy of a post that was published earlier in the year. As a result, the above paragraph doesn’t make much sense.

Erlang Debug VMs: What, Why and How

2018-04-22T15:00:00Z

Erlang can load and run C libraries. This can be useful if you want to speed up parts of your code, or if you want to interact with some OS-specific features. These special C libraries are written specially to work with the Erlang VM, and can’t easily be run in isolation without extra code. This makes these libraries difficult to debug. In this post, I introduce Erlang debug VMs, explain how to build them, and show why they can be tremendously useful for debugging linked-in drivers and NIFs.

Once you load and run arbitrary C code, the Erlang VM becomes very easy to kill.

$ erl
Erlang/OTP 19 [erts-8.2.2] [source] [64-bit] [smp:4:4]
[async-threads:10] [hipe] [kernel-poll:false]

Eshell V8.2.2  (abort with ^G)
1> test_nif:do_something_cool().
Segmentation fault
$ :(

There are so many great tools for debugging Erlang code (dbg, redbug, erlyberly, recon) as well as excellent tools for debugging C code (gdb, valgind, afl-fuzz). However, there is a bit of a shortage of good tooling for debugging C code that gets executed by the BEAM. There’s niffy, which is a neat tool that acts like a reasonable subset of the Erlang VM, letting you load and call NIFs. Niffy and your NIF can then be run through something like afl-fuzz or your own test suite to try and uncover bugs. As far as I am aware, no such tool exists for Port Drivers.

Enter Debug VMs

I’ve gotten into the habit of building Erlang/OTP from source. This affords me more flexibility than using a package manager. One particularly nice thing about building Erlang from source is that you can compile debug VMs, modified BEAM binaries intended to be run inside of tools like GDB and Valgrind. This lets us debug any C code we’ve written that gets executed by BEAM. In addition, there is also a bonus set of GDB scripts called ETP which can pretty-print Erlang terms from inside of gdb! Debug VM binaries and ETP make the ordeal of digging through core dumps of crashed BEAM processes much easier, as you can poke around various parts of memory, and print the Erlang terms they correspond to.

How I Build My Debug VMs

If you want to build some debug VMs for yourself, the first thing you need to do is clone the Erlang/OTP git repository. (For managing software compiled from source, I highly recommend GNU Stow).

Here’s the general workflow of building and installing Erlang from source, using the git repository. Please note that these commands are somewhat symbolic; if you copy and paste them into your terminal, they may not work.

Clone https://github.com/erlang/otp.git
export ERL_TOP="$(pwd)/otp"; cd $ERL_TOP
git checkout OTP-X.Y.Z
$PAGER HOWTO/INSTALL.md
./otp_build autoconf
./configure --help | $PAGER
Figure out which options I want from step 6.
./configure --prefix=/usr/local/stow/erlang-X.Y.Z --and-other-opts
make && make docs

The following steps are optional:

sudo make install install-docs
cd /usr/local/stow
readlink $(which erl) to figure out the current version in use
sudo stow -D erlang-X.(Y-1).W to unlink the old version
sudo stow erlang-X.Y.Z to link the new version

We haven’t built any debug VMs yet. Let’s do that now. There are quite a couple of different VMs you can build. The ones I use the most are the debug and valgrind variants.

$ cd $ERL_TOP/erts/emulator
$ make debug valgrind gcov gprof lcnt icount

After this, you should have a shell script called cerl in $ERL_TOP/bin. This is the script to invoke when you want to run a special BEAM VM. Adding that script to your path will let you invoke a debug VM whenever you like!

Invoking `cerl`

Now that you have built debug VMs, and have cerl somewhere in your path, we can start the BEAM inside of gdb or valgrind using cerl -rgdb or cerl -valgrind. After that, you can pass arguments to the script as if you were passing them to erl. NOTE: If you type cerl -gdb, the script will start Emacs to control your gdb session, which can be surprising!

Conclusion

I hope this inspires some of you to play around with Erlang debug VMs! There’s a lot of great stuff to be discovered! In the near future, I hope to publish another post showing off some use cases of debug VMs!

Starting a Blog, Take Two

2017-01-28T00:00:00Z

My attempt at blogging solely with Emacs and Org mode has come to an end. I blame my perfectionist tendencies and my lack of understanding of Org mode internals. I’ve since set up Frog, a static site generator written in Racket.

There are some things that I will miss from Org mode, like source blocks and embedded LaTeX, but Frog has some pretty nice features of its own, like integration with Pygments for syntax highlighting and Disqus for comments. I’m also looking forward to writing some posts using Scribble, in the likely event that I will want to do some complex bloggy things like executing Racket code in a blog post.

Static site generation with Org Mode

2017-01-02T00:00:00Z

UPDATE (2017-01-28): I no longer use Emacs and Org mode for blogging. I’ve since switched to Frog. I’ve kept this post up for shame and to those who may be interested in blogging with Org.

Let’s get this out of the way; I love Emacs and Org mode. I’ve been using Emacs as my daily editor for the past several years. I was using Vim before that, but tried Emacs and switched over when I wanted a system for organizing lecture notes. Don’t get me wrong, I think Vim is fantastic, and I still use it occasionally, but Emacs has won my heart.

`org-publish` as a blogging system

This site is generated using Org mode’s publishing features in org-publish. Each post is a .org file that gets transpiled to an HTML file when I run the org-publish command. My setup takes care of RSS, images, custom Javascript (if any) and CSS. I’m also thinking of incorporating either TRAMP or rsync to push changes to my website.

The rest of this blog post is going to go over the various settings I use. To start off, let’s pull in some org-mode libraries that we’ll use to generate our site map.

(require 'org)
(require 'ox-html)
(require 'ox-publish)
(require 'ox-rss)

Next, let’s set a couple of variables. I maintain 2 separate custom files in my \~/.emacs.d folder, one for Windows and another for Linux, so I create a variable that is editable through Emacs’s customize command. The rest of the variables are boilerplate HTML that will go into every page on the site.

(defcustom blog-base-dir "~/blog/" "Base directory for blog files")

(setq website-html-head "<link rel=\"stylesheet\" href=\"css/site.css\"
  type=\"text/css\"/>")

(setq website-html-preamble
  "<h1>Richard Kallos</h1>
<hr></hr>
<div class=\"nav\">
<ul>
<li><a href=\"/\">Home</a></li>
<li><a href=\"/static/about.html\">About</a></li>
<li><a href=\"https://github.com/rkallos\">GitHub</a></li>
<li><a href=\"/static/resume.html\">Résumé</a></li>
<li><a href=\"/index.xml\">RSS</a></li>
</ul>
</div>")

(setq website-html-postamble "<div class=\"footer\"> Copyright 2016 %a (%v
  HTML).<br> Last updated %C.<br> Built with %c.  </div>")

Finally, let’s dig into the meat and potatoes of the system,

(setq org-publish-project-alist
      `(("blog"
         :base-directory ,(concat blog-base-dir "/raw")
         :base-extension "org"
         :publishing-directory ,(concat blog-base-dir "/out")
         :publishing-function org-html-publish-to-html

         :export-with-tags nil
         :exclude-tags ("todo" "noexport")
         :exclude "level-.*\\|.*\.draft\.org"
         :with-title t
         :section-numbers nil
         :headline-levels 4
         :with-toc nil
         :with-date t

         :auto-sitemap t
         :sitemap-filename "index.org"
         :sitemap-title "Home"
         :sitemap-sort-files anti-chronologically
         :sitemap-file-entry-format "%d - %t"
         :sitemap-function org-publish-org-sitemap

         :html-doctype "html5"
         :html-html5-fancy t
         :html-head ,website-html-head
         :html-head-extra
         "<link rel=\"alternate\" type=\"application/rss+xml\"
                href=\"https://rkallos.com/posts.xml\"
                title=\"RSS feed\">"
         :html-preamble ,website-html-preamble
         :html-postamble ,website-html-postamble)

        ("images"
         :base-directory ,(concat blog-base-dir "/img")
         :base-extension "jpg\\|gif\\|png\\|svg"
         :publishing-directory ,(concat blog-base-dir "/out/img")
         :recursive t
         :publishing-function org-publish-attachment)

        ("js"
         :base-directory ,(concat blog-base-dir "/js")
         :base-extension "js"
         :publishing-directory ,(concat blog-base-dir "/out/js")
         :recursive t
         :publishing-function org-publish-attachment)

        ("css"
         :base-directory ,(concat blog-base-dir "/css")
         :base-extension "css"
         :publishing-directory ,(concat blog-base-dir "/out/css")
         :recursive t
         :publishing-function org-publish-attachment)

        ("rss"
         :base-directory ,(concat blog-base-dir "/raw")
         :base-extension "org"
         :publishing-directory ,(concat blog-base-dir "/out")
         :publishing-function (org-rss-publish-to-rss)
         :html-link-use-abs-url t

         :export-with-tags nil
         :exclude-tags ("todo" "noexport")
         :exclude "level-.*\\|.*\.draft\.org"
         :with-title t
         :section-numbers nil
         :headline-levels 4
         :with-toc nil
         :with-date t)

        ("static"
         :base-directory ,(concat blog-base-dir "/raw/static")
         :base-extension "org"
         :publishing-directory ,(concat blog-base-dir "/out/static")
         :recursive t
         :publishing-function org-html-publish-to-html

         :with-title t
         :section-numbers nil
         :headline-levels 4
         :with-toc nil

         :html-doctype "html5"
         :html-html5-fancy t
         :html-head ,website-html-head
         :html-preamble ,website-html-preamble
         :html-postamble ,website-html-postamble)

        ("website" :components ("blog" "static" "images" "js" "css" "rss"))))

This snippet defines 7 different ‘projects’ for org-publish; blog, images, js, css, rss, static, and the aggregate project website. When drafting a blog post, I can hit C-c C-e to open Org mode’s handy export buffer, then hit P p to publish the blog.

Reasons to love blogging with Org Mode

My editor, my way

I’ve spent years honing my Emacs configuration. I find reading and exiting text in Emacs to be a pleasure, and I find myself increasingly wanting to do everything I can inside of Emacs. However, that doesn’t justify why I don’t use a different format like Markdown.

Portability

Since Org mode and org-publish are written in Emacs Lisp, I can generate my blog anywhere where Emacs runs with virtually no hassle.

Single input, multiple documents

Org mode has many different output formats. With a single .org file, you can generate HTML, LaTeX, OpenDocument, and many other formats. If you want to generate multiple output files from a single input file, using Org mode makes a lot of sense. Should I so choose, I could turn my blog into a LaTeX Ebook without much effort. From what I understand, I think only Pandoc rivals Org mode in terms of compatible file formats. Well, guess what! Org mode supports exporting to Pandoc with the ox-pandoc library, so I can use all of Pandoc’s formats with Org mode files.

Living Inside Emacs

Much like a craftsman’s workshop, a text editor can be very personal and highly customized. I’ve spent a great deal of time molding both myself and my text editor to be as unobtrusive as possible. As a result, I prefer working in Emacs to anything else. At this point, blogging with Org Mode presents very little friction to me, as I am already accustomed (and addicted) to using Emacs and Org Mode.

Static HTML

As someone who debugs software for a living, I’ve grown to appreciate software with as few ‘moving parts’ as possible. Static HTML with a splash of CSS can give me a simple website that’s (hopefully) beautiful.

Conclusion

Emacs’s fabulous Org mode is extraordinarily flexible. Given my love of Emacs and Org mode, writing my blog using Org mode and org-publish seemed like a great choice. I can’t say I’ve used this system extensively yet, since this is my first post on this new domain, but I’m excited and hopeful for future posts!

Richard Kallos

ETS Match Specification Pitfall: Maps

Case Study: Using CRDTs to improve a distributed system

A Chronicle of Counters

Data Meets Math

Problems in Paradise

Mo’ computers? No problem.

Generating code with jq

What my choices told me about my priorities

I was prioritizing long-term storage

I was prioritizing (re-)reading and reflection

I am prioritizing focus

A Bit of Math: Triple Elimination Tournament

Presentation: Hybrid Logical Clocks @ PWLMTL

Imperium in Imperio - A Bridge is Made of Planks; Every Plank is a Bridge

Inveniam viam - Building Bridges

Esse quam videri - Seeing what's in front of you

Memento Mori - Seeing the End

In Defense of gen_event

Stream

Introducing wrek, a miniature Erlang graph engine

Motivation

Conceptual

Practical

Let’s get wreked

Making Erlang Pasta

Uses for wrek, additional features, and plans for the future

Conclusion

Erlang Debug VMs: What, Why and How

Enter Debug VMs

How I Build My Debug VMs

Invoking cerl

Conclusion

Starting a Blog, Take Two

Static site generation with Org Mode

org-publish as a blogging system

Reasons to love blogging with Org Mode

My editor, my way

Portability

Single input, multiple documents

Living Inside Emacs

Static HTML

Conclusion

Let’s get `wrek`ed

Invoking `cerl`

`org-publish` as a blogging system