A very useful (if somewhat misunderstood) feature of TCP is its built-in support for backpressure. Every TCP packet includes a 16-bit window field in its header which indicates how many more bytes the sender can accept. Typically this value is shifted by some number of bits known as the window scale set with a TCP option during the initial handshake (and potentially modified later) as defined in RFC 7323. A node, receiving a stream of data faster than it can handle it, applies backpressure by reducing the window size in its TCP acknowledgements. Crucially, the node may reduce this window size all the way down to zero, indicating that it can handle no more data. This backpressure mechanism is built into the operating system’s TCP implementation and the only thing that an application needs to do to access it is to stop reading data from the socket in question while it is overloaded. Once the overload has passed, the application reads from the socket again which causes the kernel to transmit a packet advertising that the window is open again, inviting the sender to continue sending data.

Infinite patience

One of the misunderstandings about this mechanism is the belief that there must be some kind of timeout after which a connection in the zero-window backpressure state indicates an error condition and should be closed. RFC 1122 says that this is not the case:

A TCP MAY keep its offered receive window closed
indefinitely.  As long as the receiving TCP continues to
send acknowledgments in response to the probe segments, the
sending TCP MUST allow the connection to stay open.

There is no need for any timeout here because the zero-window state is actively maintained by the two endpoints: the prospective sender repeatedly sends so-called zero-window probes to which the receiver responds, indicating that both ends remain alive but that the backpressure situation persists. This is essential because when the backpressure is released the receiver sends a single window-open advertisement but this packet may be lost. As long as the sender eventually sends another probe it will eventually discover that the backpressure has been released.

These repeated zero-window probes also ensure that the sender eventually detects a network partition by watching for a sufficiently long sequence of consecutive probes to which it has not received a response, as if it were sending TCP keepalives.

Probe timings

RFC 1122 has some recommendations about the exact timings of the zero-window probes:

The transmitting host SHOULD send the first zero-window
probe when a zero window has existed for the retransmission
timeout period (see Section 4.2.2.15), and SHOULD increase
exponentially the interval between successive probes.

[...]                   Exponential backoff is
recommended, possibly with some maximum interval not
specified here.

In practice a maximum is essential, or else the sender may wait for unreasonably long before discovering that the window has reopened. For example, if it backed off by a factor of 2 on every probe with no maximum and the window-opening packet went undelivered then the sender would effectively wait for twice the length of the backpressure period before discovering that the window has reopened.

But how exactly are these timings calculated?

In Linux by default the zero-window probes are scheduled similarly to regular retransmissions, starting at RTO_MIN (200ms) and backing off repeatedly by a factor of 2 up to a maximum of RTO_MAX (2 minutes), which it reaches after the backpressure has lasted for a little under 3½ minutes:

Thus the default behaviour is that the resolution of a backpressure situation which lasted for just a few minutes might not be noticed for a further two minutes (RTO_MAX) in the unfortunate, but not uncommon, event that the single window-opening packet goes undelivered. Occasionally the first probes after the window opens may also go unacknowledged, each time adding another two minutes to any backpressure-related delays. That seems awfully long to me.

This also raises the question of how the system deals with unacknowledged probes, such as would happen if the network were partitioned or the receiving process were no longer running.

The answer is that the tcp_retries2 sysctl works similarly on zero-window probes to how it works with regular retransmissions: if more than tcp_retries2 consecutive zero-window probes go unacknowledged then the connection fails. With the default value of 15, this means that a connection across a network partition might not be considered faulty for a whopping 30 minutes (15 × RTO_MAX) after the start of the partition. Yikes!

Less is more

The best way to reduce the time it takes to detect a network partition in a backpressure situation is to reduce tcp_retries2 to a more reasonable value, just as in the non-backpressure case. By setting tcp_retries2 to 5 the system will close the connection and report the failure to the application after just 5 unacknowledged probes in a row.

If the interval between the zero-window probes were allowed to grow up to RTO_MAX then waiting for 5 of them to go unacknowledged would still take a pretty dreadful 10 minutes. However, the tcp_retries2 sysctl also limits the time between the probes. I couldn’t find this mentioned in any documentation but this is where it’s implemented in tcp_send_probe0():

if (icsk->icsk_backoff < READ_ONCE(net->ipv4.sysctl_tcp_retries2))
    icsk->icsk_backoff++;

Here icsk->icsk_backoff is the backoff counter, visible using tools such as ss -tonie, and from which the re-probe interval is computed. The effect of this code is to stop increasing the backoff counter, and thus the re-probe interval, once it reaches tcp_retries2. By default this allows the re-probe interval to increase all the way to RTO_MAX, but if tcp_retries2 is 5 then the interval between zero-window probes will not increase beyond 32 × RTO_MIN which is a little over 6 seconds:

This means that a prospective sender will be able to pick up the open window within a few seconds even if the window-opening packet goes undelivered, losing only a few more seconds on each undelivered probe, and a network partition will be detected in 5 × 32 × RTO_MIN which is a little over 30 seconds, surely vastly preferable to the 30-minute default.

User timeouts

Cloudflare has a blog post about detecting dead TCP connections which concludes that typical applications sending data to the internet should set Linux’s TCP_USER_TIMEOUT socket option to be equal to the overall TCP keepalive timeout (i.e. TCP_KEEPIDLE + TCP_KEEPINTVL * TCP_KEEPCNT) so that sockets with nonempty send buffers can still detect network partitions in as timely a fashion as ones with empty send buffers.

Linux’s TCP_USER_TIMEOUT socket option has the following meaning according to man tcp:

TCP_USER_TIMEOUT (since Linux 2.6.37)
    This option takes an unsigned int as an argument.  When the
    value is greater than 0, it specifies the maximum amount of
    time in milliseconds that transmitted data may remain
    unacknowledged, or buffered data may remain untransmitted
    (due to zero window size) before TCP will forcibly close
    the corresponding connection and return ETIMEDOUT to the
    application.  If the option value is specified as 0, TCP
    will use the system default.

    [...]

    Further details on the user timeout feature can be found in
    RFC 793 and RFC 5482 ("TCP User Timeout Option").

However, RFC 5482 describes a subtly different timeout:

The Transmission Control Protocol (TCP) specification [RFC0793]
defines a local, per-connection "user timeout" parameter that
specifies the maximum amount of time that transmitted data may remain
unacknowledged before TCP will forcefully close the corresponding
connection.

This RFC specifies a TCP option allowing endpoints to communicate such a timeout to each other, but as far as I can tell Linux doesn’t make use of this facility even if TCP_USER_TIMEOUT is set.

Confusingly RFC 793 describes such a “user timeout” with different semantics again:

The timeout, if present, permits the caller to set up a timeout
for all data submitted to TCP.  If data is not successfully
delivered to the destination within the timeout period, the TCP
will abort the connection.  The present global default is five
minutes.

This is the timeout specified in Linux by the SO_SNDTIMEO socket option, not TCP_USER_TIMEOUT.

The difference between the timeout described in RFC 5482 and the implementation of the TCP_USER_TIMEOUT option in Linux is subtle but vitally important when considering TCP backpressure. The RFC 5482 timeout only considers unacknowledged data, but a TCP connection in a zero-window state has no unacknowledged data and thus this timeout should have no effect. In contrast, the TCP_USER_TIMEOUT socket option also considers untransmitted data and thus imposes a time limit on any backpressure situation after which the connection is closed, violating RFC 1122:

A TCP MAY keep its offered receive window closed
indefinitely.  As long as the receiving TCP continues to
send acknowledgments in response to the probe segments, the
sending TCP MUST allow the connection to stay open.

Unfortunately this makes this feature useless, indeed harmful, in a system that relies on TCP backpressure. I can imagine ways that it might be appropriate to use in Cloudflare’s particular situation, but it does not apply more generally.

Although the tcp_retries2 option does get a brief mention in Cloudflare’s post, the author fails to explore the consequences of reducing this from its unreasonably large default of 15 down to something more sensible. Had they done so, they might have concluded that this is a more effective solution to the problems they were describing than the backpressure-incompatible TCP_USER_TIMEOUT option.

The question is therefore how tenaciously to retry a delivery before eventually giving up and performing some corrective action. Of course this depends on the exact use-case but typically a few seconds of retries is going to be more than enough. Assuming for instance that network-congestion-signalling packet loss affects even a whopping 1% of packets, and that those packets are chosen at random, the probability of losing a few retransmissions in a row from the same connection quickly becomes vanishingly small.

The latest standard on the subject of retransmissions is RFC 1122, which uses R2 to denote the overall timeout on message deliveries and specifies the following:

The value of R2 SHOULD correspond to at least 100 seconds.

Note however that the scope of this RFC is rather narrow:

The requirements spelled out in this document are designed
for a full-function Internet host, capable of full
interoperation over an arbitrary Internet path.

Most nodes in a modern distributed system are not a full-function Internet host in this sense. Aside from nodes at the edge, very little traffic in such a system is sent over an arbitrary Internet path, and none goes over paths that behave like the internet did in October 1989 when this RFC was written. Instead, the nodes in the interior of the system will be communicating over a much more reliable and performant network and different configuration choices are appropriate for this kind of network environment.

See also this less-opinionated blog post by Marco Pracucci on the same subject.

Linux

In Linux, the retransmission behaviour is controlled by the net.ipv4.tcp_retries2 sysctl:

tcp_retries2 - INTEGER
    This value influences the timeout of an alive TCP connection,
    when RTO retransmissions remain unacknowledged.
    Given a value of N, a hypothetical TCP connection following
    exponential backoff with an initial RTO of TCP_RTO_MIN would
    retransmit N times before killing the connection at the (N+1)th RTO.

    The default value of 15 yields a hypothetical timeout of 924.6
    seconds and is a lower bound for the effective timeout.
    TCP will effectively time out at the first RTO which exceeds the
    hypothetical timeout.

The mentioned “exponential backoff” uses a factor of two, and is also bounded above by TCP_RTO_MAX, where the constants are defined as the following numbers of jiffies:

#define TCP_RTO_MAX_SEC 120
#define TCP_RTO_MAX ((unsigned)(TCP_RTO_MAX_SEC * HZ))
#define TCP_RTO_MIN ((unsigned)(HZ / 5))

HZ is a compile-time-configurable constant representing the number of jiffies per second, so these values work out to 200ms and 120s respectively. This is where the documented “hypothetical timeout of 924.6 seconds” comes from:

I don’t think I’ve ever encountered a situation where waiting for ≥15 minutes in case a message is finally delivered and acknowledged is the right thing to do, and even waiting the RFC-1122-specified ≥100 seconds for 8 retransmissions is almost always excessive.

If you instead set tcp_retries2=5 then the system will report failure to deliver a message after a little under 13s, allowing for much more prompt corrective action. This will only happen if the connection in question failed to deliver 6 packets in a row which is incredibly unlikely to happen naturally. Assuming random and independent packet loss due to congestion etc. of 1%, the loss of 6 packets in a row would have probability 0.0000000001%. Put differently, if you find you are getting connection timeouts with tcp_retries2=5 then the packet loss is almost certainly non-random, and therefore something deserving of investigation and a remedy.

Linux ≥6.15

Starting in Linux 6.15 the TCP_RTO_MAX value can be adjusted at runtime via the net.ipv4.tcp_rto_max_ms sysctl, affecting all connections on the system, and applications can also override the system-wide maximum on each socket via the TCP_RTO_MAX_MS socket option. For instance, setting the shortest permitted TCP_RTO_MAX_MS=1000 yields the following retransmission schedule:

A timeout in under 15 seconds is vastly preferable to waiting over 15 minutes. It remains to be seen whether such frequent retransmissions can have negative consequences, perhaps even causing additional network congestion and further packet loss, but I expect this will behave better in very many situations.

At least, you can do this as long as you have a way to say for each commit whether it is definitely good or definitely bad. But sometimes you might have a test case that started failing extremely rarely, and this is where traditional binary search struggles. If you can reproduce the failure on a commit then that commit is definitely bad, but if the failure doesn’t reproduce on some other commit then it could be that that commit is good or it could be that you just didn’t get lucky this time. So you run it again. And again. But how many good runs in a row do you need before you can consider the commit to be good?

To give some figures from a recent example, I was recently chasing down a test that started to fail approximately once in every ~2700 runs. Each run of the test took around 6 seconds so you could expect a bad commit to fail in around 4-and-a-half hours. The offending commit was somewhere in a range of about a thousand commits, so in principle should be findable with a binary search of 10 steps.

One possible strategy would be to consider a commit that gets 2700 successful runs in a row to be good, and carry on with the normal binary search process. This would take a couple of days to complete the 10 steps needed to narrow the search down to a single commit. But consider what happens with this strategy if we were just unlucky and a bad commit took a little longer than expected to fail. In this case, we discard the half of the commit range that actually contains the first bad commit, and the binary search process will terminate pointing to an earlier, and actually good, commit.

We can improve on this by looking for longer sequences of successful runs before considering a commit to be good. For instance, we could wait until we’ve seen twice as many successes in a row. But this means that on a good commit we’ll have to wait nine hours before moving to the next step. With ten steps to run that’d be around four days of work, and then we’d still have the concern that at least one of those steps was unlucky and didn’t capture the failure, leading to low confidence that the binary search really found the first bad commit. We could increase that confidence by running the tests for much longer on the (claimed) last good commit, but if any of the preceding runs was unlucky then we’d find the (claimed) last good commit actually to be bad, and from there we have limited options apart from restarting the binary search on the smaller range, because all those commits we thought to be good are now under suspicion again.

It’d be awfully nice if we could use the output of the preceding hours of test runs to influence the restarted search somehow.

Bayesian inference

Bayesian inference allows us to build a probabilistic model of our beliefs about the quality of each commit, and to update this model as we learn more about the system by running its tests on the different commits. The resulting process generalizes a traditional binary search into one which takes account of the probabilistic nature of the test results, iteratively updating its beliefs about which commit may have introduced the flakiness based on the history of observed failures and successes.

Rather than treating the outcome of a test run as a binary pass-or-fail thing, we model the system in terms of probabilities as follows. Consider a sequence of commits C_i for i ∈ {0..N-1} together with some k ∈ {1..N-1} such that the initial subsequence C_i<k are all good, reliably passing some test, and the remainder C_i≥k are all bad, occasionally failing the test with flakiness probability p. Assume also that the failures of distinct test runs are independent. Our goal is to identify the first bad commit C_k, which we will do by computing a discrete probability distribution P_i = P(C_i is the first bad commit) based on a history of successes and failures on the various commits in the sequence, and then selecting further tests to run so as to concentrate the probability mass on a single commit in the sequence as efficiently as possible.

The computation of the probability distribution works iteratively: given any prior distribution P_i and another test result at commit C_j, we can compute a posterior distribution P’_i which incorporates the evidence from the new test result into the distribution using Bayes’ theorem:

P’_i = P(C_i is the first bad commit | test result at C_j)
    = P(test result at C_j | C_i is the first bad commit)
          × P(C_i is the first bad commit)
          ÷ P(test result at commit C_j)
    ∝ P(test result at C_j | C_i is the first bad commit) × P_i

… noting that the denominator P(test result at commit C_j) is independent of i and is really more like a normalization factor that scales all the numbers so that they sum to 1 again. Thus we can compute the posterior distribution by multiplying each probability in the prior distribution by a factor which depends on the probability p of a test failing on a bad commit:

• P(test passes at C_j | C_i is the first bad commit) = (1-p) if i ≤ j else 1
• P(test fails at C_j | C_i is the first bad commit) = p if i ≤ j else 0

In words: given a prior distribution of probabilities that each commit in a sequence is the first bad commit, and another test result on a commit C_j, compute the posterior distribution of those probabilities as follows. If the test passed at C_j then multiply by (1-p) the probabilities of all C_i for i ≤ j to reflect that it has now become slightly less likely that any of these commits is the first bad one. In contrast, if the test failed on a commit C_j then it has now become impossible for any earlier commit to be the first bad one, so set to zero the probabilities of all commits C_i for j < i. In the latter case there is in fact no need to multiply the remaining probabilities by p, as the above formula suggests, because we are only working up to proportionality and we must renormalize everything to sum to 1 to compute the true probabilities anyway.

Doing this repeatedly for a sequence of test results yields a posterior probability distribution which reflects all the knowledge gathered from those test runs. The initial prior distribution is not particularly important, but a uniform distribution is easiest to implement so I suggest to use that.

Testing the median

Given such a probability distribution, it makes the most sense to do the next test run on the median commit, because the outcome of this run (pass or fail) most evenly divides the remaining probability space, meaning it provides the maximum possible expected reduction in uncertainty about the location of the first bad commit.

Repeatedly testing the median commit according to the posterior probability distribution has an intuitively useful behaviour: as tests pass the median will slowly creep upwards towards the first known-bad commit, and then whenever a test fails it will jump downwards again. In effect, it re-tests some commits that we were previously believed to be good, gathering more information about their quality and possibly discovering a failure on an earlier commit than the previous best. If we’ve seen a failure on the actual first-bad commit then the median will creep upwards until it is between the first-bad and last-good commits, where it will stay.

Note that the speed at which the median creeps towards the first known-bad commit is determined by p. The rarer the test failures on bad commits, the slower the median commit moves, so that the algorithm spends more time testing commits further from the known-bad ones. In contrast, if test failures are relatively likely then the median commits moves more quickly. In the limit, if the test is not flaky at all (so p is 1) and we start from a uniform prior distribution then testing the median will do a standard binary search exactly like regular bisection does.

Since this is a discrete probability distribution there typically is no commit which is the exact median. Instead, one must choose between the submedian commit, i.e. the last commit C_j such that ∑_i≤j P_i ≤ ½, and the supermedian commit, i.e. the first commit C_j such that ∑_i≤j P_i ≥ ½. When the algorithm is just starting out the difference is fairly unimportant, but as things converge it turns out that neither of these choices alone behaves quite as desired: we must repeatedly test the believed-last-good commit in case it turns out to be bad, but always choosing the submedian can end up focussing on the commit just before the believed-last-good commit, whereas always choosing the supermedian will end up focussing on the believed-first-bad commit instead.

My preferred solution is to choose whichever of the submedian or supermedian commits has had fewer test runs so far. In the endgame, this scheme means the algorithm alternates between the last-good and first-bad commits, which has some useful consequences. Running further tests on the first-bad commit helps to refine our estimate of the flakiness probability p (see below), while repeatedly checking the believed-last-good commit will eventually see a failure if this commit is in fact bad. Additionally, this scheme produces a sequence of test runs on a bad commit, some of which failed, and a similar length of sequence of test runs on the previous commit, all of which passed, which forms an intuitively compelling argument that we’ve really found the first bad commit even without having to resort to any probability calculations.

As a slight further refinement, I instead choose the commit with the smaller value of ⌊runs÷10⌋, preferring the submedian in case of a tie. Rather than strictly alternating between the two commits, using ⌊runs÷10⌋ runs ten tests in a row on each commit before switching to the other one. Doing several runs on each commit amortises the overheads associated with switching commits, such as having to recompile the system under test. You may find a different number works better in other cases according to your commit-switching overheads.

Estimating the flakiness probability p

The process described above assumes we know roughly how flaky the test is, i.e. we have a good estimate for p. In fact whatever value we use for p will still yield the right quantitative behaviour and find the right commit in the end, it just might move the median around at a suboptimal speed.

I find it works well to estimate p simply as the number of observed failures divided by the total number of test runs on all known-bad commits. When a failure is observed on a commit that was previously believed to be good, this estimate for p may drop significantly. The more successful runs there have been on the commits now known to be bad, the greater will be the reduction in the estimate for p, and therefore the earlier in the commit history will be the new median commit.

At the very start of the process, when no test failures have been observed, this estimate for p is not well-defined. Instead, seed the process by running the test repeatedly on the last commit in the sequence C_N-1 until it fails. For instance, if the test succeeds for the first nine runs on C_N-1 and fails on the tenth then the initial estimate for p will be ⅒.

Algorithm outline

A conceptual outline of the full Bayesian bisection process:

1. Initialize:
   • Define commit range: Establish the range of commits to investigate, identifying a known good commit at one end and a known bad commit at the other.
   • Initialize probability distribution: Assign a distribution P_i to represent the probability of each commit C_i being the first bad one, setting all these probabilities to be initially equal.
   • Find the first failure: Repeatedly run the test on the last commit C_N-1 until it fails for the first time.
2. Iterate:
   • Estimate flakiness probability p: Recompute the estimate of the flakiness probability p as the number of failures of commits known to be bad divided by the total number of runs on those commits.
   • Recompute posterior distribution: Given the new estimate for p and the history of other test results, compute the current probability distribution P_i.
   • Select next commit: Identify the submedian and supermedian commits according to the current probability distribution P_i. Choose whichever has smaller value of ⌊runs÷10⌋, preferring the submedian in case of a tie, and call it C_j.
   • Run test: Execute the flaky test on C_j.
3. Conclude: Once the probability mass is concentrated as desired on a single commit, and we have a sufficiently long run of successes on the previous commit, declare that commit as the most likely first bad commit.

A couple of recent Youtube videos have raised its profile, first one by James Grimes on Numberphile and then a second by Mike Boyd. In both videos they show that it’s possible to do the mental calculations quite quickly. I tried learning it some time ago and struggled to achieve that kind of speed on the algorithm as described in the literature. The first part of the computation involves counting leap years (i.e. dividing by 4) and the second part involves subtracting numbers mod 7 which means you have to get the sign right and sometimes have to deal with negative numbers. All definitely possible, but tricky to do quickly in your head.

I found that the effects of practice was basically to inline various bits of the strategy and commit the inlined version to memory, but the inlining wasn’t consistently applicable which meant that some dates ended up harder to compute than others. Since the point of practice seemed to be to commit various things to memory, I figured it’d be simpler to rework the algorithm to make use of memorisation directly.

There’s not actually that many things to remember, it only took me a couple of hours, and once I’d done the memorisation work I found I can run the algorithm pretty quickly even if distracted by other things such as continuing a conversation. I wonder if when other folks say they’re doing Conway’s method they’re really doing something more like this, or whether it’s just me that finds recalling memorised facts much easier than doing actual computations.

One handy feature of the method presented here is that you combine all the sub-computations together wih addition (mod 7) which is commutative and associative so you can process the components of the date in any order and reduce them to a single value at each step which saves on working memory slots. Here in the UK most folks say the date in day-month-year order; in the US month-day-year seems preferred, but really any order is reasonable.

Overview

Process the day-of-month, the month, the century and the year-of-century in any order you choose. Each component yields a value which you add to a running total, computed mod 7. The final running total after all four values are added (occasionally with a simple leap-year correction) indicates the day of the week of the target date, with 0 meaning Sunday, 1 meaning Monday and so on.

Day-of-month

When you hear the day of the month, add it to the running total, working mod 7.

Month

When you hear the month, convert it to a value according to the following table and add the value to your running total, working mod 7.

Just learn this mapping. There’s kind of a pattern but learning the values doesn’t take long and means you can process the month very quickly while listening for the rest of the date.

Century

When you hear the century—the first two digits of the year—convert it to a value according to the following table and add that value to your running total, working mod 7.

Again, just learn this mapping. There is a repeating pattern because the Gregorian calendar repeats every 400 years, but in practice you’ll mostly get one of these centuries so it’s simpler to remember them. You can reasonably reject years before 1700 because the UK and US didn’t start using the Gregorian calendar until 1752, and if you get a year after 2499 then it’s usually ok to take a bit longer to work out to which of 20xx, 21xx, 22xx or 23xx it’s equivalent.

Year-of-century

The year of the century—the last two digits of the year—is the trickiest bit to deal with. If it’s a multiple of 4 then it maps to a remainder mod 7 as follows which should be added to the running total:

The values for the nine multiples of 12 are just the result of dividing by 12. The other 16 are all ±4 away from a multiple of twelve so you can in principle compute the nearest multiple of 12 and then add or subtract 2 according to the pattern, but it’s only 16 values so still not that hard to just memorise the mapping.

If the year isn’t a multiple of 4 then add its remainder mod 4 to your running total, then round the year down to the nearest multiple of 4, grab the value from the table above, and add that to the running total as well. There’s various other ways you could achieve this, for instance you could memorise all 100 values, but representing the year as 4k+r and then processing 4k and r separately works best for me.

Leap year correction

If the target is in January or February of a leap year then decrement the running total by one. This is kind of irritating to remember. However note that leap years are slightly simpler to compute since they’re always a multiple of 4 so there’s no remainder mod 4 to deal with at the year-of-century step.

Finishing up

The final total represents the day of the week of the chosen date, with 0 meaning Sunday, 1 meaning Monday, and so on.

Worked examples

19 July 1989

The day-of-month 19 becomes 5 which is the initial running total. July has value 3, and adding this to the running total gives 1. The century 19xx is also 3, which adds to the running total to give 4. The year-of-century xx89 is 1 greater than a multiple of 4, namely xx88, so we add the 1 to the running total to give 5. Finally xx88 has value 5 which is added to the running total to give 3, which means that 19 July 1989 was a Wednesday.

February 27, 2204

February has value 0 so the running total starts out as zero. The day-of-month 27 becomes 6 which replaces the running total of zero. The century 22xx has value 5 which adds to the running total to give 4 (recalling that adding 6 is the same as subtracting 1). The year-of-century xx04 is a multiple of 4 with value 5 which adds to the running total to give 2. But 2204 is a leap year and the date is in February so we must subtract 1, giving a final total of 1 which means that 2 February 2204 will be a Monday.

1900-01-15

The century 19xx has value 3 so that’s the initial running total. The year-of-century xx00 is a multiple of 4 with value 0 so the running total is unchanged. January has value 4 which is added to the running total to give 0. The day-of-month 15 becomes 1 mod 7 so this is the new running total. The month is January and the year is a multiple of 4 but note that 1900 was not a leap year so no further correction is needed. 1900-01-15 was therefore a Monday.

How it works

The day and month values added together (minus one for January and February in leap years) tells you the offset in days from the year’s anchor day to the target date. You can check that the memorable doomsdays all add up to a multiple of seven: 4 April = 4 + 3 = 7; 9 May = 9 + 5 = 14; …

The century and year-of-century values added together tells you the anchor day for the year. The year-of-century table adds the number of years to the number of earlier leap years in the century, i.e. it computes 5n/4 mod 7 for multiples of 4.

Speed tips

Recalling a value from memory is enormously faster than working it out. I’d rather remember a table of 25 things instead of adding another computation step.

I spent some time actively memorising the remainders mod 7 of the numbers from 1 to 31, even though I can compute them fairly well, because you need to do this step quickly whilst distracted by listening for the rest of the date.

I also found it worth practicing doing sums of remainders mod 7. One of my very early maths teachers made us do speed exercises for adding small numbers, much like multiplication tables, which everyone hated at the time but I’ve since realised how powerful it is to be able to do these sums instantly rather than merely quickly. There’s not actually that many to learn in this case: the sums that don’t exceed 7 are just the regular ones; the sums that equal 7 are easy to map to zero. Sums involving zero are also easy, and adding 6 is easier to treat as subtracting 1. That just leaves 3+5, 4+4, 4+5 and 5+5 to learn.

I learned the year-of-century values in three batches: first the multiples of 12, then 04, 08, 16, 28, 32, 40, 76, 80, and finally 20, 44, 52, 56, 64, 68, 88, 92. Learning all 16 values at once didn’t work well for me, but working 8 at a time was very effective. The batches were pretty much randomly chosen except I made sure the split of values was as even as possible. There’s not much science behind that but it seemed to work.

I recommend learning the values for any years that you will use a lot, such as the current year and a couple of nearby ones. 2021’s is 0 and 2022’s will be 1.

If you’re coming at this cold then the only things you don’t already know are the values for the 12 months, 8 centuries, and the 16 years-of-century that aren’t multiples of 12, which is 36 individual items. To get faster you probably also need to practice the 31 remainders for days-of-the-month, 9 remainders for the multiple-of-12 years, and the 21 sums mod 7. That is 97 things in total, plus maybe the value for the current year ±1 to make it up to an even hundred. There’s loads of spaced-repetition flashcard apps out there that should help with this kind of thing. With a couple hours of up-front investment you can save literally seconds per month of looking up dates in a calendar.

Elasticsearch makes its native calls using JNA which is a deeply magical library that makes it easy for Java code to call into libraries written in C. There was definitely a suspicion that this was something to do with JNA, but the investigation fizzled out before it got very far.

Then in May 2016 Github user @fxh reported the same thing on Github. The investigation got further this time: the issue seemed to manifest only when SELinux was enabled, and seemed to be related to whether temporary files were permitted to contain executable code. Forbidding temporary executables is a security measure: anyone can write to /tmp so it’s possible to attack a system by writing a nefarious executable to /tmp and then tricking someone more privileged into running it.

In this context “executable” means more than just programs that you can run from the command line. Modern processors distinguish code from data at a very low level, with a flag on each page of memory that determines whether the data it contains can ever be interpreted as instructions that the CPU will execute. If /tmp is mounted with the noexec option then every page that’s associated with a file under /tmp will have the no-execute bit set. This forbids fully-fledged programs and dynamically-linked libraries and also, crucially, any other kind of memory-mapped executable page that is backed by a file in /tmp.

Some of JNA’s deep magic works by dynamically generating a temporary library (wrapping around the C library) into which the Java code can call. It’s sometimes possible to generate code dynamically in pages that aren’t backed by a file, but for security reasons the system might also proscribe pages from being both writeable and executable, and obviously we need write access to the memory in which we’re generating the code. The usual solution seems to be to write the code into a file and then use something like mmap() to load it again into read-only-but-executable pages. In order to do this we need some temporary space that isn’t mounted noexec.

However the crash wasn’t just caused by having /tmp mounted with the noexec option: if you do that then you get a different error and not a segmentation fault. And anyway you can tell JNA to create its temporary library in a different location by setting the java.io.tmpdir or jna.tmpdir system properties, which is a sensible workaround for when executables are forbidden in the default temporary directory. This doesn’t always fix the problem.

At this point it became hard to make further progress: there was no sufficiently well-locked-down SELinux system on which to analyse things further and it’s not a configuration that gets a lot of testing. We verified that the problem really wasn’t in Elasticsearch code itself and concluded that it must be a problematic and untested interaction between SELinux and JNA in the hope that a future version of SELinux and/or JNA would fix it.

A few months later user @vineet01 reported that they were having the same problem and that they fixed it by creating a home directory for the user as which Elasticsearch was running. They hypothesised that this was because the JVM wanted to create a usage-tracking file in the home directory. More recently user @cyamal1b4 reported the same fix and blamed the same usage-tracking file, although neither user gave an explanation of how a failure to write this file might lead to a segmentation fault in JNA-related code.

Over the years many users have reported this same crash. It still definitely exists on very locked-down systems. It always appears to be related to temporary executable files and is generally fixed by fiddling with environment variables/system properties/permissions until the segfault goes away. The trouble is that the users that have these very locked-down systems are also the users that can provide the least amount of debugging context, and that struggle to make changes to permissions or other environmental settings. It’s often quite a long process to find the right combination of settings that fix it, and all these cases consume much time from many engineers. At least the crash reliably happens at every startup. It’d be much worse if it were nondeterministic or took a long time to manifest, but Elasticsearch really should not be failing with a segmentation fault like this, and we really shouldn’t be spending this much time helping users solve the same issue over and over again.

Recently I decided it was worth taking another look to see if I could work out what was at the bottom of it.

JVM error log analysis

@fxh included the JVM error log file in the original Github issue which contains a fantastic amount of detail about the state of the JVM at the time of the crash. Here’s an excerpt of the important bits:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f424226f40a, pid=28216, tid=139922878629632
#
# JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [jna4948368637624641726.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
#

Registers:
RAX=0x00007f424226f8c2, RBX=0x00007f425579ed48, RCX=0x00007f424c44a7b0, RDX=0x00007f4242264590
RSP=0x00007f425579eae0, RBP=0x00007f425579eae0, RSI=0x00007f424c44a7d0, RDI=0x0000000000000000
R8 =0x00007f425007ae43, R9 =0x0000000000000002, R10=0x00007f425579e870, R11=0x00007f424226f3f0
R12=0x0000000000000000, R13=0x0000000000000008, R14=0x00007f424c44a7b0, R15=0x0000000000000004
RIP=0x00007f424226f40a, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
TRAPNO=0x000000000000000e

Instructions: (pc=0x00007f424226f40a)
0x00007f424226f3ea:   66 90 66 66 66 90 8b 06 55 41 b9 02 00 00 00 48
0x00007f424226f3fa:   89 e5 ff c8 83 f8 01 77 44 48 8b 05 c6 49 10 00
0x00007f424226f40a:   66 c7 07 49 bb 4c 89 47 0c 66 c7 47 0a 49 ba 48
0x00007f424226f41a:   89 47 02 8b 46 1c 48 89 77 18 48 89 57 20 48 89

Stack: [0x00007f42556a0000,0x00007f42557a1000],  sp=0x00007f425579eae0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [jna4948368637624641726.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
C  [jna4948368637624641726.tmp+0xd4dd]  Java_com_sun_jna_Native_registerMethod+0x45d
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
...
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+45
...
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8

The header tells us that Elasticsearch received a fatal SIGSEGV signal while executing the instruction at ffi_prep_closure_loc+0x1a, i.e. the one which starts 0x1a bytes into the function ffi_prep_closure_loc. This signal usually means the program attempted to dereference a pointer that doesn’t point to a valid memory location. The stack trace shows that Elasticsearch was in the process of executing JNANatives#definitelyRunningAsRoot() which looks like this:

static boolean definitelyRunningAsRoot() {
    if (Constants.WINDOWS) {
        return false; // don't know
    }
    try {
        return JNACLibrary.geteuid() == 0;
    } catch (UnsatisfiedLinkError e) {
        // this will have already been logged by Kernel32Library, no need to repeat it
        return false;
    }
}

This method is ultimately trying to call the C library’s geteuid() function, and it’s the first time we’ve touched the JNACLibrary class so we’re running the static constructor (<clinit>) which is setting up all the JNA magic.

The dump of instruction memory is useful too: the instruction pointer is at 0x00007f424226f40a and as mentioned above this is only 0x1a bytes into executing ffi_prep_closure_loc, which means the function starts at address 0x00007f424226f3f0 and hence we can disassemble all the instructions in this function leading up to the one that caused the crash:

0:  8b 06                   mov    eax,DWORD PTR [rsi]
2:  55                      push   rbp
3:  41 b9 02 00 00 00       mov    r9d,0x2
9:  48 89 e5                mov    rbp,rsp
c:  ff c8                   dec    eax
e:  83 f8 01                cmp    eax,0x1
11: 77 44                   ja     0x57
13: 48 8b 05 c6 49 10 00    mov    rax,QWORD PTR [rip+0x1049c6]        # 0x1049e0
1a: 66 c7 07 49 bb          mov    WORD PTR [rdi],0xbb49

The SIGSEGV happened on the last line which is trying to write something to the address to which register RDI points, and the crash dump also tells us that RDI is currently 0x0000000000000000 which is the null pointer and definitely not a valid address. This function hasn’t tried to write to RDI before it gets to the faulting instruction, which means it must have been expecting the caller to set RDI to a valid address.

If a function takes arguments then the caller is responsible for putting them in appropriate places so that the callee can find them. A calling convention is an agreement between caller and callee which defines (amongst other things) where the arguments to a function are when the function is called. Even on a particular processor architecture there are many possible calling conventions but in practice on a 64-bit system running Linux RDI will contain the first argument to the function. We can see from the stack dump that the caller is JNA’s Java_com_sun_jna_Native_registerMethod function, and here’s how it calls ffi_prep_closure_loc:

closure = ffi_closure_alloc(sizeof(ffi_closure), &code);
status = ffi_prep_closure_loc(closure, closure_cif, dispatch_direct, data, code);

The first argument is the closure which is returned from ffi_closure_alloc. The docs for this function say it allocates and returns a chunk of memory and doesn’t suggest that it might return NULL, but if we look at its source it’s clear that it does return NULL to indicate various kinds of failure.

Hurrah, we worked it out: the segmentation fault is because JNA isn’t checking for a failure to allocate this closure, which turns out to be a known issue which should be a small thing to fix.

But wait, there’s more

Although it’s definitely an improvement to throw a Java exception on this failure instead of a segmentation fault, this doesn’t actually solve anything. Elasticsearch will still fail to start up even with this fix: it’ll report a more descriptive message and shut down more gracefully but still users will need to fiddle around with permissions and ask for help to get Elasticsearch up and running. Ideally we need to make ffi_closure_alloc succeed or at least to understand better why it’s failing.

Note that from here on this investigation takes a couple of leaps of faith: I’ve only read the code, I don’t have a locked-down system on which to run experiments to verify any of this.

There are a number of different implementations of ffi_closure_alloc depending on operating system and selected by #ifdef pragmas but they all ultimately need to allocate some memory into which some machine code can be written: like JNA, libffi does some of its magic with dynamically-generated executable code.

The allocation mechanism is kinda complicated: there’s actually a whole separate implementation of malloc() and friends which relies on mmap() to actually acquire memory from the operating system, but then mmap() is redefined to call a custom implementation which does its best to allocate executable pages using different techniques until it finds one which succeeds. Fortunately there are some helpful comments about how this works on Linux:

#if !FFI_MMAP_EXEC_WRIT && !FFI_EXEC_TRAMPOLINE_TABLE
# if __linux__ && !defined(__ANDROID__)
/* This macro indicates it may be forbidden to map anonymous memory
   with both write and execute permission.  Code compiled when this
   option is defined will attempt to map such pages once, but if it
   fails, it falls back to creating a temporary file in a writable and
   executable filesystem and mapping pages from it into separate
   locations in the virtual memory space, one location writable and
   another executable.  */
#  define FFI_MMAP_EXEC_WRIT 1
#  define HAVE_MNTENT 1
# endif
...
#if FFI_MMAP_EXEC_WRIT && !defined FFI_MMAP_EXEC_SELINUX
# if defined(__linux__) && !defined(__ANDROID__)
/* When defined to 1 check for SELinux and if SELinux is active,
   don't attempt PROT_EXEC|PROT_WRITE mapping at all, as that
   might cause audit messages.  */
#  define FFI_MMAP_EXEC_SELINUX 1
# endif
#endif

That tells us that libffi will sometimes create temporary executable files, and it will always create them when running under SELinux. It’s important to note that this is completely independent of the fact that JNA creates temporary executable files: libffi is a language-independent library for calling foreign functions so it doesn’t know anything about Java and therefore doesn’t have access to the Java system properties java.io.tmpdir and jna.tmpdir which control where JNA does its work. Instead, on Linux libffi tries to create its temporary executable files in various places in the following order of preference:

• $LIBFFI_TMPDIR
• $TMPDIR
• /tmp
• /var/tmp
• /dev/shm
• $HOME

Elasticsearch doesn’t set any of these environment variables specially, so even if JNA is creating its temporary files somewhere that permits executables it’s entirely possible that libffi does not. This also helpfully resolves the mystery of why giving the elasticsearch user a writeable home directory seems to make the problem go away: when nothing else works, libffi will try writing to $HOME which typically does permit executable code as long as it exists.

Finally, this leads us to a proper fix: we don’t need to give the elasticsearch user a whole home directory, instead we should be able to set $LIBFFI_TMPDIR to point to the same directory that JNA uses.

Addendum 2021-08-31: a colleague pointed out that JNA contains a vendored version of libffi, and the version of libffi used by the latest release of JNA dates back to before support for the LIBFFI_TMPDIR environment variable. This’ll be the right fix eventually, but until then the best we can do is to set TMPDIR or HOME.

A successful run of either of these tools does not prove the absence of bugs, of course, nor does it say anything about corruptions that only occur when the data has been sitting on disk for an extended period of time. However a failure definitely indicates that your storage is not working as required. It’s always worth running tests like these when commissioning a new system.

Corruption on power loss

System calls like Linux’s fsync() are supposed to guarantee that some data has genuinely been written to durable storage and will still be there even if power is lost and subsequently restored. Durable writes can be expensive, so it’s pretty common to encounter systems which are configured to ignore fsync() calls which gives the impression of better performance in benchmarks. Ignoring fsync() calls is obviously very dangerous and will lead to data loss or corruption in a power outage. It’s also a very subtle misconfiguration since you probably can’t tell whether fsync() is really working or not without a genuine power outage. In particular if you’re testing VMs it’s unlikely to be enough to “power down” the VM and keep the host running: you really need to abruptly remove power from the host to find bugs of this nature.

The venerable diskchecker.pl script is pretty good at shaking out cases where fsync() is not working as it’s supposed to.

This script does quite a bit of I/O and involves pulling power cords out of things while they’re running so it’s not a great idea to run it on systems while they’re in production.

Write-time corruption

The excellent stress-ng tool recently merged a patch which improves its ability to detect corruption introduced at write time. I suggest invoking it something like this:

$ sudo stress-ng --hdd 32 \
                 --hdd-opts wr-seq,rd-rnd \
                 --hdd-write-size 8k \
                 --hdd-bytes 30g \
                 --temp-path $MOUNT_POINT \
                 --verify

Some notes on the options:

--hdd 32: This sets the number of concurrent processes writing to disk. I suggest setting it higher than the number of cores to force some context-switching which may help surface more bugs.

--hdd-opts wr-seq,rd-rnd: This says to write the file sequentially from start to finish and then perform random reads during verification. This most closely matches Lucene’s access pattern, although Lucene may use mmap() for reads instead of read() and friends. It’s also reasonable to use wr-seq,rd-seq.

--hdd-write-size 8k: Lucene writes in 8kB chunks by default.

--hdd-bytes 30g: This says how large a file each process should write. The total size of all the files written must of course not exceed the space on the filesystem, but should be large enough that the reads cannot all be served from pagecache. In this example there are 32 processes each writing 30GB of data, which adds up to about 1TB. You can also try dropping pagecache every now and then during the test with echo 3 | sudo tee /proc/sys/vm/drop_caches .

--temp-path $MOUNT_POINT: This says where to create the files used for the test, which must of course be on the filesystem you suspect to be buggy.

--verify: This says to check that the data read is what was written, which is required to detect cases of silent corruption. Without this option the test will still go through the motions of reading and writing data but only reports a failure if any of the I/O actually fails.

This is a pretty aggressive test so it’s not a good idea to run it on systems while they’re in production. By default it will run for 24 hours which I think is reasonable.

Make sure that you’re using at least version 0.12.01 of stress-ng too: older versions did not test things so thoroughly.

Apache Lucene has a simple but effective mechanism for detecting corruption which the lower layers missed: every relevant file in a Lucene index includes a CRC32 checksum in its footer. CRC32 is fast to compute and good at detecting the kinds of random corruption that happen to files on disk. A CRC32 mismatch definitely indicates that something has gone wrong, although of course a matching checksum doesn’t prove the absence of corruption. A mismatch is reported with an exception like this:

org.apache.lucene.index.CorruptIndexException:
    checksum failed (hardware problem?) : expected=335fe5fd actual=399b3f10 ...

Elasticsearch, which uses Lucene for most of its interactions with disk, shares this mechanism for detecting corruption. There are a few places where Elasticsearch throws its own CorruptIndexException with a subtly different message:

org.apache.lucene.index.CorruptIndexException:
    checksum failed (hardware problem?) : expected=e95zwd actual=fzexao ...

This is indicating the same issue except the expected and actual values are reported in base 36 rather than base 16, because that’s how Elasticsearch tracks these things internally.

Verifying a checksum is expensive since it involves reading every byte of the file which takes significant effort and might evict more useful data from the filesystem cache, so systems typically doesn’t verify the checksum on a file very often. This is why you tend only to encounter a CorruptIndexException when something unusual is happening. Users sometimes claim that corruptions are being caused by merges, or shard movements or snapshots in Elasticsearch, but this isn’t the case. These activities are examples of the rare times where reading a whole file is necessary, so they’re a good opportunity to verify the checksum at the same time, and this is when the corruption is detected and reported. It doesn’t tell us what actually caused the corruption or when it happened. It could have been introduced many months earlier.

All the relevant files in a Lucene index are written sequentially from start to end and then never modified or overwritten. This access pattern is important because it means the checksum computation is really simple and can happen on-the-fly as the file is written, and also makes it very unlikely that an incorrect checksum is due to a userspace bug at the time the file was written. The code that computes the checksum is straightforward and very well-tested, so we can have high confidence that a checksum mismatch really does indicate that the data that Lucene read is not the data that it previously wrote.

We do get reports of Elasticsearch detecting these silent corruptions in practice, which is to be expected since our userbase represents a huge quantity of data running in environments of sometimes-questionable quality. It’s not always the hardware: often the reports involve a nonstandard setup which hasn’t seen enough real-world use to shake out the bugs. Lucene’s behaviour during a power outage on properly configured storage is well-tested but many storage systems are not properly configured and may be vulnerable to corrupting data on power loss. Filesystem bugs, kernel bugs, drive firmware bugs and incompatible RAID controllers are amongst the many other possibilities. The more unusual or cutting-edge your storage subsystem is the higher the chances of encountering this sort of bug. But of course it could genuinely be faulty hardware too, maybe the RAID controller, maybe the drive itself, or maybe even your RAM. These things do happen.

The trouble with silent corruption is that it’s silent, it typically doesn’t result in log entries or other evidence of corruption apart from the checksum mismatch. In cases where the storage is managed by a separate team from the applications there’s usually at least one exchange where the storage team says it must be an application bug because there’s no evidence that anything went wrong at the storage layer, ignoring of course that a checksum mismatch itself is pretty convincing evidence that there was a problem with the storage system. Some simple empirical tests are probably a good idea.

When you hear hoofbeats, look for horses not zebras: Lucene’s mechanism for writing and checksumming files is simple, sequential and used by everyone, but there’s a lot of complexity and concurrency and variability on the other side of the syscalls it makes. It’s very very likely that the cause of corruption is infrastructural and out of the application’s control.

Technical details

Here are some technical details of how the checksum works and how to verify it independently. You can generally rely on Lucene getting this right, but hopefully the transparency is reassuring.

At time of writing, the last few bytes of a Lucene file will look something like this:

...
0000014a: 7265 6446 6965 6c64 7346 6f72 6d61 742e  redFieldsFormat.
0000015a: 6d6f 6465 0a42 4553 545f 5350 4545 4400  mode.BEST_SPEED.
0000016a: c028 93e8 0000 0000 0000 0000 335f e5fd  .(..........3_..
#                                       ^^^^^^^^^ checksum
#                             ^^^^^^^^^ 4 bytes of zeroes (not checksummed)
#                   ^^^^^^^^^ 4 bytes of zeroes (checksummed)
#         ^^^^^^^^^ 4-byte magic number (checksummed)

The footer is the last 16 bytes of which the first 12 bytes must be exactly c028 93e8 0000 0000 0000 0000 and the last 4 bytes are the CRC32 checksum of the whole file except its last 8 bytes.

There doesn’t seem to be a standard tool to compute and display the CRC32 checksum of a file on Linux, although it’s a very widely-used checksum algorithm. The most portable method I know is to run the relevant data through gzip and then extract the checksum from the footer of the resulting compressed stream, which can be done without needing to install anything special if you don’t mind burning a bunch of unnecessary CPU for the compression:

$ cat _13.si | head -c-8 | gzip --fast -c | tail -c8 | od -tx4 -N4 -An
 399b3f10

There are tools to do this computation too, although they’re not usually installed by default, and there’s almost certainly a short program in your language of choice to do the same thing. Reading the expected checksum from the last 4 bytes is simpler:

$ tail -c4 _13.si | od -tx4 -N4 -An --endian=big
 335fe5fd

These outputs don’t match so the file is corrupt, which was the problem reported by the example exception shown above. On a file that isn’t corrupt we get matching outputs:

$ tail -c16 _13.cfs | xxd
00000000: c028 93e8 0000 0000 0000 0000 0064 fc5d  .(...........d.]
$ cat _13.cfs | head -c-8 | gzip --fast -c | tail -c8 | od -tx4 -N4 -An
 0064fc5d
$ tail -c4 _13.cfs | od -tx4 -N4 -An --endian=big
 0064fc5d

This week I discovered that tcpdump has options -C, -G and -z that let me do exactly that, rendering obsolete my janky bash script that tried to do the same:

    -C size
        The file size (in multiples of 1e6 bytes) at which to roll over.
    -G age
        The file age (in seconds) at which to roll over.
    -z command
        Runs `command <filename>` on completion of a file.

The man page gives more details including a (somewhat vague) description of how tcpdump names files when using these options. I will be using something like this:

sudo tcpdump -G600 -C1000 -zgzip -Z${USER} -wcapture-%s.pcap -s128 -i eth0 tcp port 12345 ...
#                                adjust these bits according to taste ^^^^^^^^^^^^^^^^^^^^^^^

This says to roll over every 10 minutes, and every 1GB, and to run gzip to compress the file on rollover. The first file in every 10-minute period is named something like capture-1607780545.pcap, and if that file hits 1GB then subsequent files are named capture-1607780545.pcap1, capture-1607780545.pcap2, etc. The lack of leading zeroes is a bit of a pain: the eleventh file in each period is called capture-1607780545.pcap10 which tends to sort between capture-1607780545.pcap1 and capture-1607780545.pcap2. At the end of the ten-minute period it starts over again at capture-1607781145.pcap. The -Z${USER} bit means to do the file writing as the calling user rather than as root.

Note that tcpdump will spawn command each time it rolls over to a new file. It does not itself limit how many of them are running concurrently, so it’s your responsibility to make sure that the system can keep up with the traffic. Make sure to give tcpdump a suitably restrictive filter. If you don’t, you’ll hit some limit or other eventually, which probably won’t go well. If the system is under particularly heavy load then you can alternatively use tcpdump -w- to send the capture to stdout and then pipe it somewhere else that does have the capacity for further processing. If you do that, make sure to exclude the piped-elsewhere data from the packets you’re capturing.

The command is executed using execlp(3) so it searches your $PATH for the executable like a shell, but you cannot pass any other arguments. Note also that the command above runs tcpdump as root which means that command runs as root too. The man page suggests writing a script to do some more advanced processing but you may not want (or be permitted) to run a more complicated script as root. If you’d rather do most of the work as a different user then you can have tcpdump run gzip and then use inotifywait to receive a notification each time a compressed capture file is ready for further action:

inotifywait -e delete . -m --format %f \
    | sed -ue '/gz$/d;s/$/.gz/' \
    | xargs -n1 ./done.sh

This works because gzip deletes the original file once the compressed file is complete which triggers the notification. The inotifywait command writes out the names of files that are deleted from the current directory, the sed command adds a .gz to the end of each filename, and then the xargs -n1 runs the named script on each compressed capture file.

The done.sh script can do whatever you need, often offloading the data elsewhere and then removing the original file:

#!/bin/bash

echo Processing $1
aws s3 cp $1 s3://mybucket/captures/$HOSTNAME/$1
rm -vf $1

Invoking xargs like this will process the files in turn, with no parallelism, so if the done.sh script can’t keep up with the traffic then a backlog of compressed capture files might build up. Consider running these captures in their own filesystem to bound the disk space they can consume, or else adjust done.sh to simply drop the given file with no additional processing if disk usage is building up.

There’s a couple of feedback gotchas here. Firstly, deleting the compressed file triggers inotifywait again, which is why compressed files are skipped by the /gz$/d in the sed script. Secondly, if you’re sending the captured traffic back out over the network then there’s a risk that tcpdump will capture it all over again. Make sure you set up an appropriate filter in the original capture to avoid that.

Addendum 2020-12-15: An astute colleague pointed out that when we’re doing these kinds of network trace then we typically only care about the packet headers, and therefore -s128 is extremely effective at capturing what we need and dropping all the other junk in the payload. The headers compress pretty well too since there’s lots of stuff that appears in every one, whereas the payload is typically encrypted and therefore practically uncompressible. Tools like Wireshark correctly use the packet length reported in the header so they handle these truncated packets with no problems. I added that to the suggested tcpdump invocation above.

Spoilers follow. Stop reading if you’d like to try this puzzle yourself first.

Zach gives a geometric argument for the two-handed case which you can extend to the three-handed case by adding a dimension, so that you’re looking for intersections between lines in three-dimensional space which may reasonably simply not exist. However actually proving whether they do exist or not was really rather fiddly given all the different ways that the hands could line up with each other.

Two-handed case

For the two-handed case, the time t₁ is ambiguous if there is a different time t₂ such that {h(t₁), m(t₁)} = {h(t₂), m(t₂)}, i.e. h(t₁) = m(t₂) and h(t₂) = m(t₁), where h and m give the respective positions of the hour and minute hand. For simplicity’s sake we measure times as a fraction of 12 hours (so 0 is midnight and 1 is noon) and only consider times in [0,1), and measure the hand position as a fraction of the circle, so that h(t) = t and m(t) = 12t - ⌊12t⌋. Thus we want to solve

t₁ = 12t₂ - ⌊12t₂⌋
t₂ = 12t₁ - ⌊12t₁⌋

Substituting gives …

t₁ = 144t₁ - ⌊144t₁⌋ - ⌊144t₁ - ⌊144t₁⌋⌋

… which tells us that the only solutions are where 143t₁ must be an integer, say, n₁. If n₁ < 143 then h(t₁) = m(t₂) = n₁/143 and h(t₂) = m(t₁) = 12n₁ mod 143 / 143 is a solution to the equations, but this includes the cases where the hands coincide and t₁ = t₂ which are not ambiguous. Excluding those cases gives the answer.

Three-handed case

For the three-handed case, the time t₁ is ambiguous if there is a different time t₂ such that {h(t₁), m(t₁), s(t₁)} = {h(t₂), m(t₂), s(t₂)} where h and m are as before and s gives the position of the second hand, i.e. s(t) = 720t - ⌊720t⌋. There are a lot of different ways to satisfy the equality between those sets of hand positions, although at least we know for certain that h(t₁) ≠ h(t₂). Focussing just on the hour hands gives the following four cases:

h(t₁) = m(t₂) ∧ h(t₂) = m(t₁)
h(t₁) = s(t₂) ∧ h(t₂) = s(t₁)
h(t₁) = m(t₂) ∧ h(t₂) = s(t₁)
h(t₁) = s(t₂) ∧ h(t₂) = m(t₁)

Case 1

If h(t₁) = m(t₂) and h(t₂) = m(t₁) then this is the same as the two-handed case, so the solution is of the form t₁ = n/143 and t₂ = 12n mod 143 / 143. However,

s(t₁) = 720n mod 143 / 143 ∈ {h(t₂), m(t₂), s(t₂)}
                           = {h(t₁), m(t₁), s(t₂)}
                           = {n/143, 12n mod 143 / 143, 8640n mod 143 / 143}

Therefore 719n mod 143 = 0, 708n mod 143 = 0 or 7920n mod 143 = 0. But 719, 708 and 7920 are all coprime to 143 so in any case n = 0 is the only possible solution. This implies that t₁ = t₂ (= midnight) which isn’t ambiguous so this case leads to no ambiguities.

Case 2

This case is similar to the previous. If h(t₁) = s(t₂) and h(t₂) = s(t₁) then a similar argument to the two-handed case tells us that the solution is of the form t₁ = n/(720×720-1) = n/518399 and t₂ = 720n mod 518399 / 518399. However,

m(t₁) = 12n mod 518399 / 518399 ∈ {h(t₂), m(t₂), s(t₂)}
                                = {h(t₁), m(t₂), s(t₁)}
                                = {n/518399, 8640n mod 518399 / 518399, 720n mod 518399 / 518399}

Therefore 11n mod 518399 = 0, 8628n mod 518399 = 0 or 708n mod 518399 = 0. But 11, 8628 and 708 are all coprime to 518399 so in any case n = 0 is the only possible solution. Therefore this case also leads to no ambiguities.

Case 3

If h(t₁) = m(t₂) and h(t₂) = s(t₁) then

t₁ =  12t₂ -  ⌊12t₂⌋
t₂ = 720t₁ - ⌊720t₁⌋

Substituting gives

t₁ = 8640t₁ - ⌊8640t₁⌋ - ⌊8640t₁ - ⌊8640t₁⌋⌋

Therefore the solution is of the form t₁ = n/8639 and t₂ = 720n mod 8639 / 8639. However,

m(t₁) = 12n mod 8639 / 8639 ∈ {h(t₂), m(t₂), s(t₂)}
                            = {h(t₁), s(t₁), s(t₂)}
                            = {n/8639, 720n mod 8639 / 8639, 518400n mod 8639 / 8639}

Therefore 11n mod 8639 = 0, 708n mod 8639 = 0 or 518388n mod 8639 = 0. But 11, 708 and 518388 are all coprime to 8639 so in any case n = 0 is the only possible solution. Therefore this case also leads to no ambiguities.

Case 4

This case is the same the previous case, swapping the roles of t₁ and t₂, so also leads to no ambiguities.

Conclusion

Since an ambiguity is impossible in all cases, this shows that all positions of the three hands on the clock uniquely determine the time even if the hands are indistinguishable.

This was pretty fiddly. There are lots of other ways to split the equality {h(t₁), m(t₁), s(t₁)} = {h(t₂), m(t₂), s(t₂)} into cases, many of which look promising but lead to a lot of duplicated work. Getting just the right amount of case splitting took a few goes, and I reckon there’s a more elegant argument out there somewhere as even this solution really copy-pastes the same argument four times.