Avinash's Blog

projects

Mon, 01 Jan 0001 00:00:00 +0000

Most of my work is on private repositories, but I do find some time to learn new topics, contribute back to some of the open source packages I frequently use, or to create interesting tools.

Featured projects

ducktabe: Short for DuckDB Tabular Explorer. A small tool to run SQL on parquet files I generated, for “quick and dirty” analysis. I am working on improving it over time and packaging it for distribution - currently it is in a pre-alpha stage.
ducktabe-py: An offshoot of the Rust version above, but written entirely by AI. It is my first foray into spec-driven development, in a language and general area that I’ve got significant experience in. It is surprisingly capable, and is available on PyPI.
BorrowChecker: A play on the same concept in Rust, this is a simple web-app that allows you to split complex receipts with multiple people in a simple manner. Runs entirely in-browser. Made with Dioxus and Rust. Repository link.
PowerPointSnap: A mostly feature complete tool for PowerPoint on VBA that is filled with a lot of tricks to make it easy to consistently format presentations to impress clients - from my consulting days. Written in VBA. See accompanying blog post.

Other work or contributions

IntelligentReceiptSplitter: A relatively simple predecessor to BorrowChecker that focussed on using an OCR framework followed by an LLM based parser to read receipts that could be further split manually. This combination significantly reduced hallucinations from LLMs but was still very computationally intensive to run.
r.data.table.funs: A very small set of R functions that use data.table, that I found very useful earlier in my career to quicky churn out analyses. It is not ground-breaking, but rather something that anybody with sufficient basic skills in R and understand, and save an immense amount of time.
I wrote several chapters of the Polars Book, which have since been moved to the main Polars repository. Polars was a breadth of fresh air in terms of speed and ergonomics, which I had been sorely missing after switching to Python from R (where projects like data.table and dplyr dominated), so I was eager to make it better for everybody making the switch.

BasicsByHand: Logarithms

Sun, 15 Feb 2026 00:00:00 +0000

Premise

While reading a book on the history of $e$ and why it appears everywhere, the author focuses a little on how Napier first developed logarithms. While the book clarifies that Napierian logarithms are different from the definitions we use today, and demonstrates how they were calculated, it occured to me that I had no idea how to derive their values by hand.

Searching online had limited results, and it took me quite some time to find a version that described a simple method that didn’t rely on calculus (something that wasn’t developed fully when logarithms were first calculated).

I write this down almost entirely from that source, primarily as a means to make one more source available on the internet, and for my own curiosity.

Prerequisites

You need to know very little to read this article.

However, if you truly wish to do it quite literally by hand, you will also need to know how to calculate square roots by hand. I went down this (small) rabbit hole myself while doing this research, and have written down how (again, no calculus involved).

Logarithms and key properties

My “first principles” knowledge on Logarithms was a little rusty. I hope this helps you the same way it did me. However, if you remember your logarithms well, you should skip this section.

What is a logarithm?

The logarithm of a number $y$ is defined for a base $b$ as the value $x$ that satisfies the equation:

y=b^x

It is more popularly written as:

\log_b y=x

By definition, therefore,

\log_b b^x = x

and conversely

b^{\log_bx}=x

The converse is tricky, but becomes easier if you say $\log_b x=t$ , which implies that $b^t=x$ , which is exactly what $b^{\log_b x}$ becomes.

Restrictions on $b$ and $y$

It is important to note here that $\log_b y$ is defined only for $y>0$ and $b>0$ and $b \ne 1$ . These restrictions are necessary for the reasons laid out below:

If $b < 0$ , then $b^x$ is not always defined (for example, $(-2)^{0.5}$ is not real)
If $b = 0$ , $0^x$ is 0 for positive $x$ and undefined for $x \leq 0$ . It’s not invertible.
If $b = 1$ , $1^x = 1$ for all $x$ , which doesn’t give a useful $\log$ function.
If $y<0$ for valid $b$ , then no real $x$ can satisfy $b^x>0$ .
If $y=0$ for valid $b$ , then no real $x$ can satisfy $b^x=0$ .

Properties of logarithms

Sum of logs to the same base

A very basic property of exponents is that:

\begin{aligned} b^u\cdots b^v &= \underbrace{b\cdots b\ldots\cdots b}_\text{(u times)}\cdots \underbrace{b\cdots b\ldots\cdots b}_\text{(v times)}\\ &=b^{u+v} \end{aligned}

To extend this to logarithms, define $u=\log_b x$ and $v=\log_b y$ :

\begin{aligned} b^{u+v} &= b^u\cdots b^v\\ b^{\log_b x+\log_b y}&=b^{\log_b x}\cdots b^{\log_b y}\\ b^{\log_b x+\log_b y}&=xy\\ b^{\log_b x+\log_b y}&=b^{\log_b{(xy)}}\\ \log_bx+\log_by&=\log_b(xy) \end{aligned}

It is important that all logs are to the same base $b$ .

Logs raised to an arbitrary exponent

This follows directly from sum of logs:

\begin{aligned} \log_bx^a&=\log_b(\underbrace{x\cdots x\cdots\ldots\cdots x}_{\text{a cdots}})\\ &=\underbrace{\log_bx+\log_b x+\ldots\log_bx}_{\text{a times}}\\ &=a\log_bx \end{aligned}

Or, when $a$ isn’t positive or a whole number, we use the fact that:

(b^x)^a=b^{xa}=b^{ax}

which can be written as:

\begin{aligned} b^{\log_b(y^a)}&=y^a=(y)^a\\ &=(b^{\log_b y})^a\\ &=b^{a\log_b y}\\ \Rightarrow\log_b(y^a)&=a\log_b y \end{aligned}

Difference of logs

Following the first two properties:

\begin{aligned} \log_bx-\log_by&=\log_bx+(-1)\log_by\\ &=\log_bx+\log_by^{-1}\\ &=\log_bx+\log_b\frac{1}{y}\\ &=\log_b\frac{x}{y} \end{aligned}

Changing of base

This is an important one. We need to write:

\log_xy

in the form of just logs to a common base $b$ . We can write:

\begin{aligned} x^\frac{\log_by}{\log_bx}&=(b^{\log_bx})^\frac{\log_by}{\log_bx}\\ &=b^{\frac{\log_bx\cdots\log_by}{\log_bx}}\\ &=b^{\log_by}\\ &=y\\ &=x^{\log_xy}\\ \Rightarrow \dfrac{\log_by}{\log_bx}&=\log_xy \end{aligned}

The Euler Method

Let’s say that we want to calculate $\log_7 155$ . Euler’s method involved first changing the base:

\log_7 155=\dfrac{\log_{10} 155}{\log_{10} 7}

and then using the identity:

\log(\sqrt{x\cdot y})=\dfrac{\log(xy)}{2}=\dfrac{\log x+\log y}{2}

Let’s see how. From herewith, if only $\log$ is used, assume that it means $\log_{10}$ . This will simplify a lot of the visual noise.

Simplifying log calculations

We can first start by writing:

\log155=\log(100\cdot1.55)=2+\log1.55

Calculating $\log1.55$

Iteration 1

We know that from the identity above:

\log(\sqrt{1\cdot10}) = \dfrac{\log1 +\log10}{2}=\dfrac{1}{2}=0.5

In other words,

\log{\sqrt{10}} = 0.5

We know (or can calculate by hand) that $\sqrt{10}\approx3.16228$ . You can hopefully now see where this is going.

Iteration 2

Where does $1.55$ lie? Between $1$ and $\sqrt{10}$ , or between $1$ and $3.16228$ . Let’s now calculate:

\log{\sqrt{1\cdot3.16228}}=\dfrac{\log1+\log3.16228}{2}=\dfrac{0+0.5}{2}=0.25

Thus, $\sqrt{1\cdot3.16228}$ is $\sqrt{3.16228}=1.77828$ .

Iteration 3

Once again, we make more progress by finding the square root of $1.77828$ , because $1.55$ lies between $1$ and $1.77828$ . Therefore,

\log{\sqrt{1\cdot1.77828}}=\dfrac{\log1+\log{1.77828}}{2}=0.125

$\sqrt{1.77828}\approx1.33521$ . We’re getting rather close.

Iteration 4

We have a small change here. $1.55$ now lies between $1.33521$ and $1.77828$ . This means that the new value we need to find is:

\begin{aligned} \log{\sqrt{1.33521\cdot1.77828}}&=\dfrac{\log{1.33521}+\log{1.77828}}{2}\\ &=\dfrac{0.125+0.25}{2}\\ &=0.1875 \end{aligned}

Also, $\sqrt{1.33521\cdot1.77828}=1.54090$

Further iterations

The next step would be to determine if $1.54090$ is sufficiently close to $1.55$ for us to approximate that $\log 1.54090\approx\log 1.55$ , or we need to proceed further.

Of course, we can continue to refine the values until we’re sufficiently satisfied of our accuracy. What does a calculator tell us about $\log 1.55$ ?

1>>> from math import log
2>>> log(1.55, 10)
30.1903316981702915

So we were off by around… $0.0028$ , which isn’t bad at all, considering we did only around 4 iterations (and many more iterations for calculating the square root).

Back to the Euler method

We find that $\log_{10}155=2+0.1875$ . We calculate $\log_{10}7$ the same way, which should come to $0.845$ (according to my calculator), so

\log_7 155\approx\dfrac{2.1875}{0.845}=2.5

and calculators give us:

1>>> from math import log
2>>> log(155, 7)
32.591

Clearly, while this is close to the $2.5$ we got, it’s not quite correct - we’re off by $0.091$ , an order of magnitude more. If we’d used the true value, $0.1903$ , then we would have got:

\log_7 155\approx\dfrac{2.1903}{0.845}\approx2.592

Which is much closer to the true value. We find that approximating the true value of the log to a much larger number of decimal points is crucial to get the correct value.

Takeaway

A key takeaway for a programmer would be that Euler managed to implement binary search to efficiently (especially in terms of human effort) find the logarithm of a number

References

The primary reference for the material in this article is Bureau42. While the link works, it looks like the material that this used to point to doesn’t anymore. Some searching led me to a backup hosted on GitHub and another Hackernews linked source.

BasicsByHand: Square Roots

Sun, 15 Feb 2026 00:00:00 +0000

Premise

Long ago in middle school (or even earlier), along with the concept of what a square root is, I was also taught a method to calculate them by hand. This method did not involve Taylor series expansions, nor was it the popular numeric Newton-Raphson method - we didn’t have a lick of knowledge of Calculus to use them anyway.

It worked, and nobody ever quite taught why it worked. Given that I’m an adult now and free to do adult things, I wanted to cross this off my (admittedly non-existent) list as a part of something else I was trying to understand.

What method?

This is for my readers who may not be familiar with this method, or just need a refresher. Suppose you need to calculate the square root of 78129.329 while you’re stuck on an island. Maybe your captor is a math fanatic. This is how you would go about it.

Start with grouping the digits in pairs - starting from the decimal point, moving on either side of it:

       ┌─────────────────
       │  7,81,29.32,9

Now, we begin a method that suspiciously looks a lot like long-form division. Find the largest number whose square is equal to, or less than the left-most group.

           2
        ┌─────────────────
      2 │  7,81,29.32,9

Then, you subtract its square from the digit:

           2
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
        │  3

and multiply the number at the top with 2 (how arbitrary! or is it…?) and call it the new divisor you’re going to work with. The next task is to find a value for a digit $x$ such that $x \times (40+x)$ is less than or equal to $381$ :

           2  x
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     4x │  3,81

This will require some manual multiplication (since we’re doing this by hand!). Looks like $7$ meets this requirement (as $48 \times 8 = 384$ ). Apply the same subtraction:

           2  7
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     47 │  3,81
        │ -3,29
        │─────────────────
        │   52

and repeat the process once more. We need to first multiply $27$ by $2$ , which gives us $54$ . Similarly, $x \times (540+x)$ should then be less than or equal to $52,29$ :

           2  7  x
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     47 │  3,81
        │ -3,29
        │─────────────────
    54x │    52,29

I’ll forgive you if you use the calculator here! I’ll skip a step here (we end up at $9$ as $549\times 9=49,41<52,29$ ) to show that we need to now find $x$ such that $x \times (5580 + x)$ is less than or equal to $2,88,32$ :

           2  7  9. x
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     47 │  3,81
        │ -3,29
        │─────────────────
    549 │    52,29
        │   -49,41
        │─────────────────
   558x │     2,88,32

We settle on 5. Note here that the next number is $2795\times2$ , not $279.5\times2$ , a small subtlety here. The next $x$ is going to be $1$ :

           2  7  9. 5 x
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     47 │  3,81
        │ -3,29
        │─────────────────
    549 │    52,29
        │   -49,41
        │─────────────────
   5585 │     2,88,32     
        │    -2,79,25
        │─────────────────
  5590x │        9,07,90

and we get:

           2  7  9. 5 1
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     47 │  3,81
        │ -3,29
        │─────────────────
    549 │    52,29
        │   -49,41
        │─────────────────
   5585 │     2,88,32     
        │    -2,79,25
        │─────────────────
  55901 │        9,07,90
        │      - 5,59,01
        │─────────────────
        │        3,48,89

We could stop here, or go on further. I elect to stop here, and verify my result. Let’s go to trusty Python:

1>>> from math import sqrt
2>>> print(sqrt(7_81_29.32_9))
3279.51624103082094

Excellent, so we were right!

But… why? How did we get it right?

Building the Intuition

Getting our foot in the door

Let’s start with our previous example.

7,81,29.32,9

Note that commas intentionally retained every two digits instead of the usual three.

Why are commands retained every two digits?

This is because squares of numbers have, up to twice the number of digits of the original number. Anecdotally, consider $999^2$ . It will be less than $1000^2$ , which is $1,000,000$ and has 7 digits (and 7 is less than 4, the number of digits that $1000$ ) has.

On the other hand, smaller numbers like $4$ go to $16$ ( $1$ digit to $2$ digits). $1$ to $3$ remain at $1$ digit (1, 4, 9).

Remember that we usually also write our numbers in base $10$ . Since any power of $10$ is easy to calculate, this makes the estimate of our first digit rather easy. Let’s start by finding the closest square of $10$ that is less than or equal to $7,81,29.329$ .

\begin{array}{lrll} &1,00,00 &\leq 7,81,29.329 &\leq 10,00,00\\ \Rightarrow &10^4 &\leq 7,81,29.32,9 &\leq 10^5\\ \Rightarrow &(10^2)^2 &\leq 7,81,29.32,9 &\leq 10^5\\ \end{array}

Why did we write this as $(10^2)^2$ ? This tells us that our desired square root is greater than $100$ . What’s the closest multiple of hundred that is less than or equal to $7,81,29.32,9$ ? It’s 200, because

\begin{array}{rlr} 200^2 &= 4,00,00 &< 7,81,29.32,9\\ 300^2 &= 9,00,00 &> 7,81,29.32,9 \end{array}

In other words,

\begin{array}{lrll} &200^2 &\leq 7,81,29.32,9 &\leq 300^2\\ \end{array}

When we calculated that $200$ was the closest multiple of $100$ that was less than or equal to $7,81,29.32,9$ , we were calculating the first digit of the square root.

We continue by applying a common formula…

…which is derived below to minimize cognitive load, in case it’s been a long time since you’ve touched algebra.

\begin{aligned} (a+x)^2 &=(a+x)(a+x)\\ &=a(a+x) + x(a+x)\\ &=a^2 + ax + xa + x^2\\ &=a^2+2ax+x^2 \end{aligned}

Connecting the dots

Let’s consider the next digit. Because the true root lies between $200$ and $300$ , we can say that the next digit of the root will be the closest multiple of $10$ that is greater than $200$ and whose square is less than $7,81,29.32,9$ .

That means we’re trying to find a single digit $x$ that meets the criteria:

\begin{array}{rll} &(200+10x)^2 &\leq 7,81,29.32,9\\ \Rightarrow &200^2 + 2\cdot200\cdot10x + 100x^2 &\leq 7,81,29.32,9\\ \Rightarrow &200^2 + 100\cdot\boldsymbol{(40+x)x} &\leq 7,81,29.32,9\\ \Rightarrow &100\cdot\boldsymbol{(40+x)x} &\leq 3,81,29.32,9\\ \Rightarrow &\boldsymbol{(40+x)x} &\leq \boldsymbol{3,81}.29,32,9 \end{array}

Look closely at the bolded parts of the last equation:

\boldsymbol{(40+x)x} \leq \boldsymbol{3,81}

This is exactly what we did here:

           2  x
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     4x │  3,81

The single digit $x$ that satisfies this equation is $7$ , as we had obtained earlier.

One more step for clearer understanding

Let’s do one more round to inscribe the logic in our heads:

We know that $270$ is the next best approximation of the root of $7,81,29.329$ . We also know that to get the next digit, we need the closest multiple of $1$ that is between $270$ and $280$ whose square is less than $7,81,29.32,9$ .

Once again, in trying to find the single digit $x$ , the expressions become (note that in this iteration, $x$ is just $x$ , not $10x$ , because it’s a multiple of $1$ , not $10$ ):

\begin{array}{rll} &(270+x)^2 &\leq 7,81,29.32,9\\ \Rightarrow &270^2 + 2\cdot270\cdot x + x^2 &\leq 7,81,29.32,9\\ \Rightarrow &270^2 + \boldsymbol{(540+x)x} &\leq 7,81,29.32,9\\ \Rightarrow &\boldsymbol{(540+x)x} &\leq 52,29.32,9\\ \Rightarrow &\boldsymbol{(540+x)x} &\leq \boldsymbol{52,29}.32,9 \end{array}

and the bolded parts of the expression are:

\boldsymbol{(540+x)x} \leq \boldsymbol{52,29}

once again, matching exactly to:

           2  7  x
        ┌─────────────────
      2 │  7,81,29.32,9
        │ -4
        │─────────────────
     47 │  3,81
        │ -3,29
        │─────────────────
    54x │    52,29

Estimating to more decimal points

You also now understand why this process can continue on till as many decimal points are needed.

The rest of the estimation, however, is, um… left as an exercise to the reader 😬.

Conclusion

I hope you now understand that the so called “long-division” method is just encoding the technique that I laid out above step-by-step, and is not really doing division alone, it’s extending the binomial identity $(a+b)^2$ and refining it successively every iteration.

You will also notice that as we proceed further in calculating the root, the smaller the error becomes, allowing us to ignore it when we see fit.

References

The method laid out in the article is described in NIST article. However, the explanation of the method is my own, as I was not able to grasp how the explanation provided in that link actually worked “all the way” to the end.

Resolving I/O Bottlenecks for 100K Small Files with LMDB

Tue, 10 Feb 2026 00:00:00 +0000

Premise

I was building a model for processing images (OCR, classification, object detection all bundled in), and I found myself with another problem - too many images to store efficiently as individual files on disk.

I had several thousand images already. I was expecting several thousand more. My repository was tracking these images via DVC. My computer was also slowing down massively because of the sheer number of files. DVC itself was slowing down (after all, randomly accessing many files isn’t going to be fast). I also needed to access files at random for training/evaluating the model (lots of shuffling). Lastly, these images had their own associated metadata (labels, bounding boxes, “correct” text etc.), and they need to be stored along with the images - or least easily linkable to them.

I was primarily aiming for a “simple” solution, and didn’t need a productionizable codebase.

Potential Solutions

Partitioning

A typical solution for “too many files” is to partition them by their name. It’s ideal if their name is a hash, so you can store the first character of the hash, then the second character, and then the actual file. So, for example, the directory changes from:

1files/
2├── a1b2c3d4e5.txt
3├── b7f8a9c0d1.txt
4├── b7e4d2f1a0.txt
5├── cf1a2b3c4d.txt
6├── c0d1e2f3a4.txt
7└── a1f5e6d7c8.txt

 1files/
 2├── a/
 3│   └── 1/
 4│       ├── a1b2c3d4e5.txt
 5│       └── a1f5e6d7c8.txt
 6├── b/
 7│   └── 7/
 8│       ├── b7f8a9c0d1.txt
 9│       └── b7e4d2f1a0.txt
10└── c/
11    ├── 0/
12    │   └── c0d1e2f3a4.txt
13    └── f/
14        └── cf1a2b3c4d.txt

This isn’t novel - git and dvc both store their objects this way. This limits directory size, so file system look ups take less time.

This isn’t a perfect solution - it required that I store the images as their hash, and handle the directory structure correctly. I will also need to maintain my own mechanism to maintain the link between the hash and the metadata, which means creating some sort of index. Lastly, DVC will still track files individually, which means that its push, diff and pull commands will still be slow.

Separate (object) storage, maintain only the index

Another solution would be to offload the storage to a medium capable of handling a large number of files in any order, while maintaining random access - for example, S3. Now my task will reduce to just correctly maintaining the index so that I know how many images are stored on S3, and link their metadata via their hash.

If you’ve experienced retrieving a list of a large number of files stored on S3, you’d have first encountered the limit of 1000 objects that boto3 enforces per request. You’ll need to work around it with pagination, which while standard, is still more work. You would have also realized that even after all this, S3 will take quite some time to give you the list even after you’ve optimized as much as you can.

However, that means I need an active internet connection to access any data. It also introduces latency during training, and quite wasteful bandwidth in running multiple training sessions for the models. I can minimize these if I store the images in the same AWS region as I would be running the training in (say, an EC2), but that means I needed access to an EC2.

This still left me the task of maintaining my own index, which I really wanted to avoid, as it would mean additional maintenance burden for a relatively nascent project that hasn’t reached production status, while demanding production code for an even more nascent pipeline.

What about a… database?

This feels much like reaching for your nose by looping your hand behind your head instead of touching it directly with your fingers. A database is great for many things, but setting one up and maintaining it (even a local SQLite one) isn’t really a quick and painless process, and has many gotchas. For instance, most databases aren’t optimized for storing a large number of binary blobs.

I considered, and tested using a more modern embeddable analytical database like DuckDB for this purpose (I’m quite biased to using DuckDB and/or Polars to solve a large number of my data processing problems). I quickly found out that storing large binary blobs in it causes it to choke (which is fair, it isn’t really designed for that). Storing the files elsewhere while maintaining just the index in it still had the original problem - I needed to write the mechanism of maintaining the index.

Note: HuggingFace now provides many images datasets (such as MNIST) in the Parquet format, with the images stored using Arrow’s extension types (but still as binary blobs). My experience with storing binary data in Parquet hasn’t been great, but you could check this out to see if it meets your requirements.

The solution I landed on

What about a… different kind of database?

Let’s get down to first principles. What did I want to do? I wanted to store images. With those images, I also wanted to store its metadata. I wanted to access said data quickly. It became clearer to me that I was looking for a fast key-value store, and I stumbled upon LMDB.

LMDB

Wikipedia’s entry on LMDB indicates that it’s an incredibly small (64kB) piece of software that does one thing really well - be a ridiculously fast key-value store. I won’t pretend to understand how it works, the writeup on the Wiki provides plenty of good detail. I’ll focus, rather, on how I used it to solve my problem.

Storing and retrieving image data along with its metadata

LMDB is rather barebones. It exposes few features - the ability to write, and read a particular key (stored as a bytestring), that itself points to arbitrary bytes. I used its Python bindings.

I wrote a tiny class (~200 LoC) that did the following:

Read the file names and the data of the images that I had, along with their metadata as it currently was, into Python. Batched to avoid running out of memory.
Serialize the metadata, and read the image files as bytes, and link them to the keys f"{file_name}_image" and f"{file_name}_metadata" respectively.
Store these as key-value pairs into the LMDB database, which is a single file.
Provided a method to read the keys to identify all the images present in the database.
Provided a method to retrieve an arbitrary set of images and their metadata quickly from the saved database.

This has many advantages:

LMDB is fast - really, really fast. Random access? Check. Fast retrieval of available images? Check.
DVC tracking becomes simple - maintain a single file, and just version control that. No slow downs due to sheer number of files - either for DVC, or my computer.
No index to maintain - the images and their metadata are stored in the same location, and linkable via a mere change in the suffix to their file name.
Local access, practically zero latency.

Which solves… all of the problems I had! When new files come in, all I needed to do was add them to the DB. LMDB has a few options available - you can avoid overwriting the same key, ensure that the database is de-duplicated.

The code

I’ve provided a sample code below that demonstrates storing just the images (not metadata) for the Oxford 102 Category Flower dataset, which has around 8000 images.

  1from pathlib import Path
  2import lmdb
  3
  4
  5class ImageDB:
  6    def __init__(self, env_path: Path, max_size_as_mb: int):
  7        self.env_path = str(env_path)
  8        self.env = lmdb.open(self.env_path, map_size=max_size_as_mb * (2**20))
  9        self.db = self.env.open_db()
 10
 11    def save_image(
 12        self,
 13        name: str,
 14        image_path: Path,
 15    ) -> None:
 16        with self.env.begin(write=True) as txn:
 17            txn.put(name.encode(), image_path.read_bytes())
 18
 19    def read_image(self, name: str) -> bytes:
 20        with self.env.begin(write=False) as txn:
 21            if image_as_bytes := txn.get(name.encode()):
 22                return image_as_bytes
 23            else:
 24                raise KeyError()
 25
 26    def save_images(
 27        self,
 28        name_image: dict[str, Path],
 29    ) -> None:
 30        # Note: you might need to enforce a batch size here
 31        # to aovid running out of memory because this loads
 32        # all images sent to this function as bytes.
 33        with self.env.begin(write=True) as txn:
 34            item_tuples = [
 35                (k.encode(), image_path.read_bytes())
 36                for k, image_path in name_image.items()
 37            ]
 38            cursor = txn.cursor()
 39            consumed, added = cursor.putmulti(
 40                item_tuples, dupdata=False, overwrite=False
 41            )
 42            print(
 43                f"Saved {added:,} out of {len(name_image):,} images to the DB ({consumed - added:,} seem to already exist)."
 44            )
 45
 46    def load_images(
 47        self,
 48        names: list[str],
 49    ) -> dict[str, bytes]:
 50        names_as_bytestrings = [x.encode() for x in names]
 51        with self.env.begin(write=False) as txn:
 52            cursor = txn.cursor()
 53            return {
 54                k.decode(): image_as_bytes
 55                for k, image_as_bytes in cursor.getmulti(names_as_bytestrings)
 56            }
 57
 58    def delete_image(self, name: str):
 59        with self.env.begin(write=True) as txn:
 60            if txn.delete(name.encode()):
 61                print(f"Image {name} deleted successfully")
 62            else:
 63                raise KeyError()
 64
 65    def retrieve_names(self) -> list[str]:
 66        with self.env.begin(write=False) as txn:
 67            return [x.decode() for x in txn.cursor().iternext(keys=True, values=False)]
 68
 69
 70if __name__ == "__main__":
 71    db = ImageDB(Path("./db/"), 512)
 72    name_image: dict[str, Path] = dict()
 73
 74    # Save the results
 75    for image_path in Path("./data/jpg/").glob("*.jpg"):
 76        name_image[image_path.name] = image_path
 77        if len(name_image) >= 1000:
 78            db.save_images(name_image)
 79            name_image.clear()
 80    # Add last batch also
 81    db.save_images(name_image)
 82
 83    del name_image
 84    # How many images have been stored?
 85    print(f"The DB has {len(db.retrieve_names()):,} images stored")
 86
 87    name_image: dict[str, bytes] = dict()
 88    # Load the results from the DB and check if they match the files on disk
 89    for image_path in Path("./data/jpg").glob("*.jpg"):
 90        name_image[image_path.name] = image_path.read_bytes()
 91        if len(name_image) >= 1000:
 92            saved_name_image = db.load_images(list(name_image.keys()))
 93            assert name_image == saved_name_image
 94            name_image.clear()
 95    # Verify last batch also
 96    saved_name_image = db.load_images(list(name_image.keys()))
 97    assert name_image == saved_name_image
 98
 99    print("All images stored are byte identical to the original ones!")
100
101    db.env.close()

This should provide you a good starting point to implement additional features, such as storing metadata, filtering required input by metadata (such as extracting a specific label for evaluation) and so on.

Caveats

Avoid PIL, or pay the (small) price

One gotcha that I initially faced is that the images I saved wasn’t the same as the images that I retrieved. This wasn’t LMDB’s fault, this was because I was reading the images from disk via PIL, and storing them as bytes in LMDB. PIL decodes and encodes the image, so a roundtrip will not necessarily be identical, even for lossless file formats (other than bitmap images).

Don’t encode/re-encode the image before you store it, or be prepared for the stored data to not be byte-identical.

The `max_size_as_mb` argument

LMDB has an unusual design. You need to specify the upper bound of the DB size upon creation, and if it exceeds this size, it will fail. You can edit this later, with some caveats (on Windows, this will actually allocate the full size).

Concurrency and LMDB

LMDB, while extremely fast, has some considerations with concurrency. See the documentation for details. It may not be suited for distributed workloads.

Alternatives

This article covers a “quick and dirty” solution, and was before more purpose-built solutions were available. Some alternatives are:

If you’re comfortable operating directly on archives, a simple tar file will do - it can provide an offset index to provide random access to data.
Nvidia’s WebDataset. Modern, open source and purpose built for large scale deep learning.
LanceDB, which describes itself as “designed for multimodal” and “built for scale”. It’s built on top of Arrow, closely related to Parquet.
As mentioned, HuggingFace has multiple solutions to this, starting with Arrow backed storage, and their own datasets.

Use these if you want to scale to production level training.

Resolving Multiprocessing Bottlenecks in Legacy Batch Pipelines

Wed, 14 Jan 2026 00:00:00 +0000

Premise

I needed to use a codebase that had a mix of light numpy operations, coupled with a few heavy mathematical optimization problems that did not use numpy. The API that I had access to provided to be too slow (even asynchronously), so I thought I’ll run this locally on a faster system in parallel to speed things up. Turns out, that was easier said than done, and I picked up the importance of environment variables along the way.

In detail

Demonstration code

Let’s say that the “black box” code looked something like this.

 1import numpy as np
 2import sys
 3
 4n = 400
 5acc = 0.0
 6for _ in range(30):
 7    a = np.random.rand(n, n)
 8    b = np.random.rand(n, n)
 9    c = a @ b
10    d = np.linalg.inv(c + np.eye(n))
11    acc += np.linalg.svd(d, compute_uv=False).sum()
12
13result = acc + float(sys.argv[1])

The actual codebase was a full blown package that I wasn’t familiar with, except for the following points:

The author(s) had not attempted to parallelize the codebase.
It used several packages, but most notably numpy and a few niche optimization tools.
The most important bit: numpy was used sparingly, and the compute intensive part was in the optimization tools.

The problem

I needed to call this script multiple times, supplying different arguments each time. A single call doesn’t take much time on my system:

1> time python test.py 0
2
3real    0m0.861s
4user    0m26.207s
5sys     0m0.120s

I could, of course, call it sequentially, but that wouldn’t be very helpful:

1> time for i in {1..10}; do python test.py $i; done
2
3real    0m8.100s
4user    4m6.980s
5sys     0m1.193s

This isn’t that bad - it looks like it took slightly less than the expected 8.6s, but it’s no speedup.

What about switching to good ol’ GNU parallel? I have 32 cores, so it should be fast, right?

1> time seq 10 | parallel -j32 'python test.py {}'
2
3real    16m50.053s
4user    521m15.773s
5sys     0m42.995s

Wow, that’s actually around 125x SLOWER.

An experienced programmer at this point would likely scream one of two words: CONTENTION! OVERSUBSCRIPTION!

Let’s explore what that means.

But why?

numpy is a very optimized package. It tries to use all cores available under the hood so that it can do what you asked in the fastest possible time. This is a great idea when you have large matrices to operate on.

However, we called numpy via Python here 10 separate times. Each of those 10 processes were trying to access 32 threads on the machine. 320 threads on 32 cores, which means that there were significantly more threads than cores, all actively vying for those 32 cores!

The clearest signal of this is that a naive attempt at parallelization via multiprocessing being SLOWER than a sequential set of calls to the same operation. That’s why using a system with more cores and trying to parallelize blindly doesn’t always speed things up, and in the worst case, such as this, it considerably slows things down.

The solution

numpy uses different libraries for multithreading based on the system. Environment variables can be used to control the number of threads each process creates, a summary of which is provided below.

Variable	Backend
`OMP_NUM_THREADS`	OpenMP (used by many BLAS)
`OPENBLAS_NUM_THREADS`	OpenBLAS
`MKL_NUM_THREADS`	Intel MKL
`BLIS_NUM_THREADS`	BLIS
`VECLIB_MAXIMUM_THREADS`	Apple Accelerate

To avoid oversubscription, we need to tell numpy to use fewer threads than the default (which is all threads), because (1) we are aware that the process running is not compute intensive, and (2) we will handle parallelism ourselves.

On my system, I was using OpenMP, so all that I needed to do was:

1> time seq 10 | parallel -j32 'OMP_NUM_THREADS=1 python test.py {}'
2
3real    0m0.612s
4user    0m4.905s
5sys     0m0.236s

to make the slowest job among a set of 10 run faster than single call to the script!

The real world impact in the actual codebase was 15x response time, and 30x throughput. What used to take 120 machines earlier took only 4 now!

Ending notes

Why did this work here? Let’s revisit the “most important” bit of information from the demonstration code:

The most important bit: numpy was used sparingly, and the compute intensive part was in the optimization tools.

This technique of using environment variables to restrict the number of threads that a process can spawn, and then running it in parallel (via parallel based multiprocessing) will work in any case where multiple processes are spawned for each physical core, and you want to maximize throughput. In case the computations used by numpy are more compute intensive, you could experiment with allowing more threads while still paralllelizing:

1seq 10 | parallel -j8 'OMP_NUM_THREADS=4 python test.py {}'

This allows use of up to 4 threads, while spawning 8 processes in parallel, and in theory still using a total of 32 cores. You may need to adjust this to find out what works best for your system.

In addition, to make this work less dependent on the libraries that numpy can use, just set more (or all) of the environment variables:

1seq 10 | \
2parallel -j32 "OMP_NUM_THREADS=1 \
3OPENBLAS_NUM_THREADS=1 \
4MKL_NUM_THREADS=1 \
5python test.py {}"

An interesting side note is that the parallel, but single core run took 0.61 seconds, less than the time it took to run a single call to the script on multiple cores (via threading in numpy internally). For large matrices, this also indicates that the overhead associated with spawning 32 threads for the single task might actually be significant - and running on a single core might just be inherently better!

Footnotes

I’ve simplified a few things in this blog post:

This demo uses heavy numpy ops to clearly show the oversubscription effect. The actual codebase had much lighter usage, but the principle is the same.
I’ve used contention and oversubscription interchangeably here. The former is a case of threads competing for the same shared resource, and the latter is a higher thread count than CPU cores. Oversubscription here led to contention.
Modern systems have hundreds, or thousands of active threads for very few system cores. The difference with most of these threads is that they often are “sleeping”, and don’t vie for attention like the numpy ones.

Quick hacks to make client-ready presentations

Fri, 20 Oct 2023 00:00:00 +0000

Premise

When I worked in healthcare consulting, I often spent a LOT of my time creating PowerPoint presentations (decks in consulting lingo - not even slide decks). However, it was rather repetitive. Thus, was born PowerPointSnap.

What is it?

I’ll write this down as pointers.

It’s a VBA based PowerPoint add-on. Just a set of commands that work well with each other.
It’s Windows only - it’s unlikely to work on MacOS.
It’s installation-free and is not an executable, which makes it perfect for locked-down corporate environments, as long as you have the permission to download files.

How do I get it?

The project is available on this Github repo. The instructions to install it are available there, but here’s the down-low:

Download the Snap.ppam file to your system.
Enable the developer options.
Go to the Developer tab, and click on PowerPoint Add-ins.
Click on Add New. Choose the location of the file you just dowloaded. Click Close.
To uninstall, repeat the process, and simply click on Remove this time.

What can I do with it?

Frankly, a LOT. The base concept of this tool is:

“Set” a shape as the one you want to copy a property from.
Select any property from the list to automatically apply it.

Here’s a non-exhaustive list of all the options available.

Apply properties of shapes directly

This is the part of the interface that can be used for shapes (which include charts and tables).

To use, first select a shape object, click on “Set”. Then, choose the object you want to Snap its properties to (see how I got the inspiration for the name?). You should be able to copy all compatible properties - if something is not copy-able, the tool will show an error, and then let you exit.

Note that it’s probably not to apply a property of a shape to a table - if you want to make the entire table orange, there are probably better built-in ways to do it than to use Snap.

Beautify charts with Snappable properties

Charts are also supported, with dedicated features for it.

What do these features do? You should be able to hover over the option and get a tooltip that shows what it’s capable of, but here’s another summary just in case:

Sync Value/Date Axis: this will try to align the range, the ticks, the numeric values etc. of the “set” chart to the one you’ve selected. I couldn’t put in just $x$ and $y$ here because Microsoft internally doesn’t label them that way. Try either of these two options (you can undo!) and see what works best for your chart. This doesn’t work well yet for 3D charts.
Sync Plot/Title/Legend: often, you want to centre a title, or make sure that multiple charts that show nearly identical things for different variables all look exactly the same from a client perspective. But that’s usually difficult if you’ve already configured the charts a little - which can be remedied with this option!
Format Painter: this is simply a helper for the normal format painter to align the formats of the text that you’ve selected with the way it originally is in the “set” chart. The reason for this feature is simply to avoid going back to Home to click on the Format Painter option again.
Reset Axes Scales: in case you messed up somewhere, you can use this to rever to PowerPoint defaults.

The next two options deserve their own section.

Customize the labels programmatically

Your immediate senior in a consulting environment would frown at your chart, and then exclaim, “I think that’s too many labels for the data points. Can you show them every two/three/four labels? I know this is manual work, but it’s a one time thing!”

It’s never a one time affair. But don’t worry, we have this nice feature to help us. If you click on the Customize Label option, you will get this (without the “Set” option):

Never mind the rather unfriendly legend entries. They’re just here to demonstrate that you can do the following kinds of whacky abilities with your own chart!

Screenshots of the chart snapability

Of course, visuals will do it more justice. For example, look at this image:

Here’s what you can do:

Click on the left chart. Press “Set” in the toolbar for Snap.
Click on the right chart, and then go through the following:
1. In Shapes, click on Dim. This will align the shapes of the chart.
2. Use the guides that you get while moving the chart to align the positions of the two charts now that their shapes are equal.
3. You’ll notice that the chart area doesn’t still match, nor does the title.
4. In Charts, click on Sync Plot Area and Sync Title Area, and watch the magic unfold.
5. Now, click on the second chart, and click on “Set”. Let’s align the axes of the first chart to the second one.
6. Click on the first chart, and then in Charts, click Sync Value Axis.
Let’s bring that senior’s exclamation back into play - (s)he wants you to highlight only Profit labels, and that too every 2 iterations. To do this:
1. Click on Customize Labels after clicking on either chart.
2. You’ll get the screen shown in the previous section. Make sure to adjust the values such that it’s exactly like the screenshot there.
3. Click on “Save and Run”. This will save the configuration you’ve selected, and run it on the chart you’ve selected.
4. Click the other chart. Then, in Charts, click on Rerun Customization.

This is what your results should look like:

Of course, getting those calculations right is a whole different thing that will need some work.

Align table dimensions

Oftentimes, you have two tables that show similar values… you know the drill. Here’s what you can do in a scenario such as this:

This is what the Tables section of the tool looks like:

To align these tables together,

Click on the left table. Press “Set” in the toolbar for Snap.
Click on the right table.
Click on Shapes, inside it, Dim. Now the shapes of the table are the same.
In Tables, click on Sync Column Widths. Now the columns are also the same.
If you try to align by rows, it fails because the number of rows are not the same in the two tables.

Here’s what you’ll end up with:

Pretty neat, eh?

Finding representative samples efficiently for large datasets

Thu, 19 Oct 2023 00:00:00 +0000

Premise

In this day and age, we’re not short on data. Good data, on the other hand, is very valuable. When you’ve got a large amount of improperly labelled data, it may become hard to find to find a representative dataset to train a model on such that it generalizes well.

Let’s formalize the problem a little so that a proper approach can be developed. Here’s the problem statement:

You have a large-ish set of (imperfectly) labelled data points. These data points can be represented as a 2D matrix.
You need to train a model to classify these data points on either these labels, or on labels dervied from imperfect labels.
You need a good (but not perfect) representative sample for the model to be generalizable, but there are too many data points for each label to manually pick representative examples.

In a hurry?

Here’s what you need to do:

Read the premise and see if it fits your problem.
Go to the For the folks in a hurry! section at the end to find the generic solution and how it works.

Why do we need representative samples?

Generally, three things come to mind:

Allows the model to be generalizable for all kinds of data points within a category.
Allows for faster training of the model - you need fewer data points to get the same accuracy!
Allows maintaining the training set - if your training set needs validation by experts or annotations, this keeps your costs low!

Define the data

This data can be practically anything that can be represented as a 2D matrix.

There are exceptions. Raw image data (as numbers) might get difficult because even if you flatten them, they’ll be significant correlation between features. For example, a face can appear practically anywhere in the image, and all pixels centered around the face will be highly correlated, even if they are on different lines. A workaround in this case would be to pipe the image through a CNN model that has been trained on some generic task and produces a 1D representation of a single image in the final hidden layer before the output. Other data will need further processing along similar lines.

Get a specific dataset

For this specific article, I will use the ShopMania dataset on Kaggle. I apologize in advance for not using a more easily accessible dataset (you need to sign into Kaggle to download it) - and I’m not 100% sure if the GPL allows me to create a copy of the data and place it in my own repository. Nevertheless, the data (if you download it and choose to use it instead of some other dataset) will look like this:

NOTE: whenever I want to show an output along with the code I used for it, you’ll see the characters >> indicating the command used, and the output to be without those prefixes.

 1>> import polars as pl
 2>> data = pl.read_csv("archive/shopmania.csv")
 3>> data
 4shape: (313_705, 4)
 5┌────────────┬──────────────────────────────────────────────────────┬─────────────┬────────────────┐
 6│ product_ID ┆ product_title                                        ┆ category_ID ┆ category_label │
 7│ ---        ┆ ---                                                  ┆ ---         ┆ ---            │
 8│ i64        ┆ str                                                  ┆ i64         ┆ str            │
 9╞════════════╪══════════════════════════════════════════════════════╪═════════════╪════════════════╡
10│ 2          ┆ twilight central park print                          ┆ 2           ┆ Collectibles   │
11│ 3          ┆ fox print                                            ┆ 2           ┆ Collectibles   │
12│ 4          ┆ circulo de papel wall art                            ┆ 2           ┆ Collectibles   │
13│ 5          ┆ hidden path print                                    ┆ 2           ┆ Collectibles   │
14│ …          ┆ …                                                    ┆ …           ┆ …              │
15│ 313703     ┆ deago anti fog swimming diving full face mask        ┆ 229         ┆ Water Sports   │
16│            ┆ surface snorkel scuba fr gopro black s/m             ┆             ┆                │
17│ 313704     ┆ etc buys full face gopro compatible snorkel scuba    ┆ 229         ┆ Water Sports   │
18│            ┆ diving mask blue large/xtralarge blue                ┆             ┆                │
19│ 313705     ┆ men 039 s full face breathe free diving snorkel mask ┆ 229         ┆ Water Sports   │
20│            ┆ scuba optional hd camera blue mask only adult men    ┆             ┆                │
21│ 313706     ┆ women 039 s full face breathe free diving snorkel    ┆ 229         ┆ Water Sports   │
22│            ┆ mask scuba optional hd camera black mask only        ┆             ┆                │
23│            ┆ children and women                                   ┆             ┆                │
24└────────────┴──────────────────────────────────────────────────────┴─────────────┴────────────────┘

The data documentation on Kaggle states:

The first dataset originates from ShopMania, a popular online product comparison platform. It enlists tens of millions of products organized in a three-level hierarchy that includes 230 categories. The two higher levels of the hierarchy include 39 categories, whereas the third lower level accommodates the rest 191 leaf categories. Each product is categorized into this tree structure by being mapped to only one leaf category. Some of these 191 leaf categories contain millions of products. However, shopmania.com allows only the first 10,000 products to be retrieved from each category. Under this restriction, our crawler managed to collect 313,706 products.

For demonstration, I’ll just limit the categories to those that have exactly 10,000 occurences.

1data = (
2    data
3    .filter(pl.count().over("category_ID") == 10000)
4)

You’ll notice that there are only 17 categories in this dataset. Run this to verify that fact.

 1>>> data.get_column("category_label").unique()
 2shape: (17,)
 3Series: 'category_label' [str]
 4[
 5    "Kitchen & Dining"
 6    "Scarves and wraps"
 7    "Handbags & Wallets"
 8    "Rugs  Tapestry & Linens"
 9    "Cell Phones Accessories"
10    "Men's Clothing"
11    "Jewelry"
12    "Belts"
13    "Men Lingerie"
14    "Crafts"
15    "Football"
16    "Medical Supplies"
17    "Adult"
18    "Hunting"
19    "Women's Clothing"
20    "Pet Supply"
21    "Office Supplies"
22]

Note that this is very easy in Polars, which is the package I typically use for data manipulation. I recommend using it over Pandas.

Specify the task

Okay - so now we have exactly 10,000 products per category. We only have the title of the product that can be leveraged for categorization. So let me define the task this way:

Craft a small representative sample for each category.

Why small? It helps that it’ll make the model faster to train - and keep the training data manageable in size.

Finding representative samples

I mentioned earlier that we need to represent data as a 2D matrix for the technique I have in mind to work. How can I translate a list of text to a matrix? The answer’s rather simple: use SentenceTransformers to get a string’s embedding. You could also use more classic techniques like computing TF-IDF values, or use more advanced transformers, but I’ve noticed that SentenceTransformers are able to capture semantic meaning of sentences rather well (assuming you use a good model suited for the language the data is in) - they are trained on sentence similarity after all.

Getting `SentenceTransformer` embeddings

This part is rather simple. If you’re unable to install SentenceTransformers, please check their website.

1import sentence_transformers
2# See list of models at www.sbert.net/docs/pretrained_models.html
3ST = sentence_transformers.SentenceTransformer("all-mpnet-base-v2")
4title_embeddings = (
5    ST.encode(
6        data.get_column("product_title").to_list(),
7        show_progress_bar=True, convert_to_tensor=True)
8    .numpy())

This process will be slow (~30 minutes) if you don’t have a GPU. There are faster approaches, but they are slightly more involved than would be beneficial for a blog post. The wait will be worth it, I promise! In addition, the call to .numpy() at the end is to directly get a single numpy array - otherwise you get a list of numpy arrays, which is rather inefficient. Further, SentenceTransformers will try to run on the GPU if available, and if so, you will need to write .cpu().numpy() so that the tensor is copied from the GPU to the CPU.

NOTE: for a proof-of-concept implementation, or if you’re on the CPU, try the all-MiniLM-L6-v2 model. It’s a much smaller and much faster model, although you sacrifice a little in terms of accuracy.

The concept of approximate nearest neighbors

Performing any kind of nearest neighbor algorithm on medium scale datasets (even bordering 10,000 rows and tens of columns) tends to be slow. A primary driver of this was the need to calculate all, or nearly all distances between all data points. Approximate nearest neighbor (ANN) algorithms work around this through various approaches, which warrant their own blog post. For now, it would suffice to understand that there are shortcuts that ANN algorithms take to give you if not the exact nearest neighbor, at least one of the nearest neighbors (hence the term approximate).

There are several algorithms that you can use - I shall proceed with faiss, because it has a nice Python interface and is rather easy to work with. You can use any algorithm - a full list of the major ones are available here.

I’ll explain why we’re in the nearest neighbor territory in due course.

Building the database

To build the database, all we need is the title_embeddings matrix.

1import faiss
2def create_index(title_embeddings):
3    d = title_embeddings.shape[1]    # Number of dimensions
4    ann_index = faiss.IndexFlatL2(d) # Index using Eucledian Matrix
5    ann_index.add(title_embeddings)  # Build the index
6    
7    return ann_index # Faiss considers databases an "index"

This does create a database. But remember, we’re trying to find representative samples - which means we need to do this by the category (or label). So let’s design a function that sends only the necessary data as that for a particular category, and then create the database. We’ll need three pieces of information from this function:

The actual faiss database.
The actual subset of data that was used to build this index.
The label indices with respect to the original data that went into the faiss database.

(2) and (3) will help us later in rebuilding a “network graph” that will allow us to reference the original data points.

 1import faiss
 2import numpy as np
 3import polars as pl
 4
 5def create_index(label):
 6    faiss_indices = (
 7        data # this needs to be an argument if you want to create a generic function
 8        .with_row_count("row_idx")
 9        .filter(pl.col("category_label") == label)
10        .get_column("row_idx")
11        .to_list()
12    )
13    
14    faiss_data = title_embeddings[faiss_indices]
15    d = data.shape[1]               # Number of dimensions
16    faiss_DB = faiss.IndexFlatIP(d) # Index using Inner Product
17    faiss.normalize_L2(data)        # Normalized L2 with Inner Product search = cosine similarity
18    # Why cosine similarity? It's easier to specify thresholds - they'll always be between 0 and 1.4.
19    # If using Eucledian or other distance, we'll have to spend some time finding a good range
20    # where distances are reasonable. See https://stats.stackexchange.com/a/146279 for details.
21    faiss_DB.add(data)              # Build the index
22    
23    return faiss_DB, faiss_data, faiss_indices

Identifying the nearest neighbors

To proceed with getting a representative sample, the next step is to find the nearest neighbors for all data points in the database. This isn’t too hard - faiss index objects have a built-in search method to find the k nearest neighbors for a given index, along with the (approximate) distance to it. Let’s then write a function to get the following information: the label index for whom nearest neighbors are being searched, the indices of said nearest neighbors and the distance between them. In network graph parlance, this kind of data is called an edge list i.e. a list of pair of nodes that are connected, along with any additional information that specifies a property (in this case distance) of the edge that connects these nodes.

 1def get_edge_list(label, k=5):
 2    faiss_DB, faiss_data, faiss_indices = create_index(label)
 3    # To map the data back to the original `train[b'data']` array
 4    faiss_indices_map = {i: x for i,x in enumerate(faiss_indices)}
 5    # To map the indices back to the original strings
 6    title_name_map = {i: x for i,x in data.select("row_idx", "product_title").rows()}
 7    distances, neighbors = faiss_DB.search(faiss_data, k)
 8    
 9    return (
10        pl.DataFrame({
11            "from": faiss_indices})
12        .with_columns(
13            pl.Series("to", neighbors),
14            pl.Series("distance", distances))
15        .explode("to", "distance")
16        .with_columns(
17            pl.col("from")
18            .map_dict(title_name_map),
19            pl.col("to")
20            .map_dict(faiss_indices_map)
21            .map_dict(title_name_map))
22        .filter(pl.col("from") != pl.col("to"))
23    )

NetworkX and Connected Components

The next step in the process is to create a network graph using the edge-list. But why?

Remember that we have identified the (k=5) nearest neighbors of each data point. Let’s say that we have a point A that has a nearest neighbor B. C is not a nearest neighbor of A, but it is a nearest neighbor of B. In a network graph, if A and C are sufficiently similar enough to B within a particular minimum thershold, then A will be connected to C through B! Hopefully a small visual below would help.

What happens when such a concept is extended for many data points? Not all of them would be connected - because we’re applying a minimum threshold that they have to meet. This is the only hueristic part of the rather fast process. Here’s one more helpful visual:

Very starry night-eque vibes here. Let’s get to the code.

1import networkx as nx
2def get_cluster_map(label, k=5, min_cosine_distance=0.95):
3    edge_list = (
4        get_edge_list(label, k=k)
5        .filter(pl.col("distance") >= min_cosine_distance)
6    )
7    graph = nx.from_pandas_edgelist(edge_list.to_pandas(), source="from", target="to")
8    return {i: list(x) for i,x in enumerate(nx.connected_components(graph))}

Getting clusters

Now that all the parts of the puzzle are together, let’s run it to see what kind of clusters you get for Cell Phone Accessories.

1clusters = get_cluster_map("Cell Phones Accessories", 5, 0.95)

Make sure to configure the following if your results aren’t good enough:

Relax the min_cosine_distance value if you want bigger clusters.
Increase the number of nearest neighbors if you want more matches.

Viewing the components

There will likely be many clusters (you can see how many exactly with len(clusters)). Let’s look at a random cluster:

1>> clusters[3]
2['smartphone lanyard with card slot for any phone up to 6 yellow 72570099',
3 'smartphone lanyard with card slot for any phone up to 6 black 72570093',
4 'smartphone lanyard with card slot for any phone up to 6 lightblue 72570097',
5 'smartphone lanyard with card slot for any phone up to 6 blue 72570095',
6 'smartphone lanyard with card slot for any phone up to 6 green 72570101',
7 'smartphone lanyard with card slot for any phone up to 6 pink 72570091']

Let’s see another cluster that had 172(!) members in my run (the clusters themselves will be stable, but their indices may change in each run owing to some inherent randomness in the process).

 1>>> clusters[6]
 2['otm essentials iphone 8/7 modern clear printed phone case snowflakes iphone 8/7 op qq z051a',
 3 'otm essentials iphone 8/7 modern clear printed phone case iphone 8/7 arrows blue op qq a02 58',
 4 'otm essentials iphone 8/7/6s clear printed phone case single iphone 8/7/6s golden pineapple op qq z089a',
 5 'otm essentials iphone 8/7/6s clear printed phone case single iphone 8/7/6s butteryfly delight yellow op qq z029d',
 6 'otm essentials iphone 8/7 modern clear printed phone case iphone 8/7 luck of the irish op qq a01 45',
 7 'otm essentials iphone 8/7 modern clear printed phone case iphone 8/7 brides maid white op qq a02 16',
 8 ...
 9 'otm essentials iphone 8/7 modern clear printed phone case iphone 8/7 flying arrows white op qq hip 20',
10 'otm essentials iphone 8/7 modern clear printed phone case iphone 8/7 brides maid pink white op qq a02 17',
11 'otm essentials iphone 8/7 modern clear printed phone case iphone 8/7 anemone flowers white op qq z036a',
12 'otm essentials iphone 8/7 modern clear printed phone case mustache iphone 8/7 op qq hip 08',
13 'otm essentials iphone 8/7 modern clear printed phone case oh snap iphone 8/7 op qq z053a',
14 'otm essentials iphone 8/7/6s clear printed phone case single iphone 8/7/6s desert cacti orange pink op qq a02 22']

Running for all categories

This isn’t that hard (although it may take more than a moment). Just iterate it for each category!

1clusters = [get_cluster_map(x, 5, 0.95) for x in data.get_column("category_label").unique()]

For the folks in a hurry!

I get it - you often want a solution that “just works”. I can come close to it. See below for code and a succinct explanation. For those of my readers who aren’t in a hurry, this also serves as a nice summary (and copy-pastable code)!

The code

 1import sentence_transformers
 2import faiss
 3import polars as pl
 4import numpy as np
 5
 6# Data is read here. You download the files from Kaggle here: 
 7# https://www.kaggle.com/datasets/lakritidis/product-classification-and-categorization
 8data = pl.read_csv("archive/shopmania.csv", new_columns=[
 9    "product_ID", "product_title", "category_ID", "category_label"])
10data = (
11    data
12    .filter(pl.count().over("category_ID") == 10000)
13    .with_row_count("row_idx")
14)
15
16
17# See list of models at www.sbert.net/docs/pretrained_models.html
18ST = sentence_transformers.SentenceTransformer("all-mpnet-base-v2")
19title_embeddings = (
20    ST.encode(
21        data.get_column("product_title").to_list(),
22        # I'm on a MacBook, you should use `cuda` or `cpu`
23        # if you've got different hardware.
24        device="mps",
25        show_progress_bar=True, convert_to_tensor=True)
26    .cpu().numpy())
27
28# Code to create a FAISS index
29def create_index(label):
30    faiss_indices = (
31        data # this needs to be an argument if you want to create a generic function
32        .filter(pl.col("category_label") == label)
33        .get_column("row_idx")
34        .to_list()
35    )
36    
37    faiss_data = title_embeddings[faiss_indices]
38    d = faiss_data.shape[1]         # Number of dimensions
39    faiss_DB = faiss.IndexFlatIP(d) # Index using Inner Product
40    faiss.normalize_L2(faiss_data)  # Normalized L2 with Inner Product search = cosine similarity
41    faiss_DB.add(faiss_data)        # Build the index
42    
43    return faiss_DB, faiss_data, faiss_indices
44
45# Code to create an edge-list
46def get_edge_list(label, k=5):
47    faiss_DB, faiss_data, faiss_indices = create_index(label)
48    # To map the data back to the original `train[b'data']` array
49    faiss_indices_map = {i: x for i,x in enumerate(faiss_indices)}
50    # To map the indices back to the original strings
51    title_name_map = {i: x for i,x in data.select("row_idx", "product_title").rows()}
52    distances, neighbors = faiss_DB.search(faiss_data, k)
53    
54    return (
55        pl.DataFrame({
56            "from": faiss_indices})
57        .with_columns(
58            pl.Series("to", neighbors),
59            pl.Series("distance", distances))
60        .explode("to", "distance")
61        .with_columns(
62            pl.col("from")
63            .map_dict(title_name_map),
64            pl.col("to")
65            .map_dict(faiss_indices_map)
66            .map_dict(title_name_map))
67        .filter(pl.col("from") != pl.col("to"))
68    )
69
70# Code to extract components from a Network Graph
71import networkx as nx
72def get_cluster_map(label, k=5, min_cosine_distance=0.95):
73    edge_list = (
74        get_edge_list(label, k=k)
75        .filter(pl.col("distance") >= min_cosine_distance)
76    )
77    graph = nx.from_pandas_edgelist(edge_list.to_pandas(), source="from", target="to")
78    return {i: list(x) for i,x in enumerate(nx.connected_components(graph))}
79
80# Example call to a single category to obtain its clusters
81clusters = get_cluster_map("Cell Phones Accessories", 5, 0.95)
82# Example call to **all** categories to obtain all clusters
83clusters = [get_cluster_map(x, 5, 0.95) for x in data.get_column("category_label").unique()]

How the code works

If you want to write down an algorithmic way of looking at this approach,

Obtain a 2D representation of the labelled/categorized data. This can be embeddings for strings, the final hidden state output from a generic CNN model for images, or a good ol’ tabular dataset where all numbers are normalized and can be expressed as such.
Create an ANN database (based on a package such as faiss) that allows you fast nearest neighbor searches. Use cosine similarity for an easy threshold determination step.
Obtain an edge-list of k (from 5 to 100) nearest neighbors for all (or a sample of data points in case your dataset is incredibly HUGE) data points in the ANN database.
Apply a minimum threshold on similarity (completely based on heuristics), and obtain the connected components of the network graph from the filtered edge-list you just created.
Map all indices back to their source data-points that make sense, and pick any number of items from each cluster (usually, I end up picking one element from each cluster), and you now have your representative sample!

Overlap Joins: Number of docker trucks in an interval

Thu, 22 Jun 2023 00:00:00 +0000

Premise

I stumbled upon an interesting Stackoverflow question that was linked via an issue on Polars github repo. The OP asked for a pure Polars solution. At the time of answering the question Polars did not have support for non-equi joins, and any solution using it would be pretty cumbersome.

I’m more of a right-tool-for-the-job person, so I tried to find a better solution.

Problem Statement

Suppose we have a dataset that captures the arrival and departure times of trucks at a station, along with the truck’s ID.

 1import polars as pl # if you don't have polars, run 
 2                    # pip install 'polars[all]'
 3data = pl.from_repr("""
 4┌─────────────────────┬─────────────────────┬─────┐
 5│ arrival_time        ┆ departure_time      ┆ ID  │
 6│ ---                 ┆ ---                 ┆ --- │
 7│ datetime[μs]        ┆ datetime[μs]        ┆ str │
 8╞═════════════════════╪═════════════════════╪═════╡
 9│ 2023-01-01 06:23:47 ┆ 2023-01-01 06:25:08 ┆ A1  │
10│ 2023-01-01 06:26:42 ┆ 2023-01-01 06:28:02 ┆ A1  │
11│ 2023-01-01 06:30:20 ┆ 2023-01-01 06:35:01 ┆ A5  │
12│ 2023-01-01 06:32:06 ┆ 2023-01-01 06:33:48 ┆ A6  │
13│ 2023-01-01 06:33:09 ┆ 2023-01-01 06:36:01 ┆ B3  │
14│ 2023-01-01 06:34:08 ┆ 2023-01-01 06:39:49 ┆ C3  │
15│ 2023-01-01 06:36:40 ┆ 2023-01-01 06:38:34 ┆ A6  │
16│ 2023-01-01 06:37:43 ┆ 2023-01-01 06:40:48 ┆ A5  │
17│ 2023-01-01 06:39:48 ┆ 2023-01-01 06:46:10 ┆ A6  │
18└─────────────────────┴─────────────────────┴─────┘
19""")

We want to identify the number of trucks docked at any given time within a threshold of 1 minute prior to the arrival time of a truck, and 1 minute after the departure of a truck. Equivalently, this means that we need to calculate the number of trucks within a specific window for each row of the data.

Finding a solution to the problem

Evaluate for a specific row

Before we find a general solution to this problem, let’s consider a specific row to understand the problem better:

1"""
2┌─────────────────────┬─────────────────────┬─────┐
3│ arrival_time        ┆ departure_time      ┆ ID  │
4│ ---                 ┆ ---                 ┆ --- │
5│ datetime[μs]        ┆ datetime[μs]        ┆ str │
6╞═════════════════════╪═════════════════════╪═════╡
7│ 2023-01-01 06:32:06 ┆ 2023-01-01 06:33:48 ┆ A6  │
8└─────────────────────┴─────────────────────┴─────┘
9"""

For this row, we need to find the number of trucks that are there between 2023-01-01 06:31:06 (1 minute prior to the arrival_time and 2023-01-01 06:34:48 (1 minute post the departure_time). Manually going through the original dataset, we see that B3, C3, A6 and A5 are the truck IDs that qualify - they all are at the station in a duration that is between 2023-01-01 06:31:06 and 2023-01-01 06:34:48.

Visually deriving an algorithm

There are many cases that will qualify a truck to be present in the overlap window defined by a particular row. Specifically for the example above, we have (this visualization is generalizable, because for each row we can calculate without much difficulty the overlap window relative to the arrival and departure times):

Take some time to absorb these cases - it’s important for the part where we write the code for the solution. Note that we need to actually tell our algorithm to filter only for Cases 2, 3 and 4, since Cases 1 and 5 will not satisfy our requirements.

Writing an SQL query based on the algorithm

In theory, we can use any language that has the capability to define rules that meet our algorithmic requirements outlined in the above section to find the solution. Why choose SQL? It’s often able to convey elegantly the logic that was used to execute the algorithm; and while it does come with excessive verbosity at times, it doesn’t quite in this case.

Note here that we run SQL in Python with almost no setup or boilerplate code - so this is a Python based solution as well (although not quite Pythonic!).

Introducing the DuckDB package

Once again, in theory, any SQL package or language can be used. Far too few however meet the ease-of-use that DuckDB provides:

no expensive set-up time (meaning no need for setting up databases, even temporary ones),
no dependencies (other than DuckDB itself, just pip install duckdb),
some very friendly SQL extensions, and
ability to work directly on Polars and Pandas DataFrames without conversions

all with mind-blowing speed that stands shoulder-to-shoulder with Polars. We’ll also use a few advanced SQL concepts noted below.

Self-joins

This should be a familiar, albeit not often used concept - a join of a table with itself is a self join. There are few cases where such an operation would make sense, and this happens to be one of them.

A bullet train recap of non-equi joins

A key concept that we’ll use is the idea of joining on a range of values rather than a specific value. That is, instead of the usual LEFT JOIN ON A.column = B.column, we can do LEFT JOIN ON A.column <= B.column for one row in table A to match to multiple rows in B. DuckDB has a blog post that outlines this join in detail, including fast implementation.

The concept of `LIST` columns

DuckDB has first class support for LIST columns - that is, each row in a LIST column can have a varying length (much like a Python list), but must have the exact same datatype (like R’s vector). Using list columns allow us to eschew the use of an additional GROUP BY operation on top of a WHERE filter or SELECT DISTINCT operation, since we can directly perform those on the LIST column itself.

Date algebra

Dates can be rather difficult to handle well in most tools and languages, with several packages purpose built to make handling them easier - lubridate from the tidyverse is a stellar example. Thankfully, DuckDB provides a similar swiss-knife set of tools to deal with it, including specifying INTERVALs (a special data type that represent a period of time independent of specific time values) to modify TIMESTAMP values using addition or subtraction.

Tell me the query, PLEASE!

Okay - had a lot of background. Let’s have at it! The query by itself in SQL is (see immediately below for runnable code in Python):

 1SELECT
 2     A.arrival_time
 3    ,A.departure_time
 4    ,A.window_open
 5    ,A.window_close
 6    ,LIST_DISTINCT(LIST(B.ID)) AS docked_trucks
 7    ,LIST_UNIQUE(LIST(B.ID))   AS docked_truck_count
 8
 9FROM (
10    SELECT *
11        ,arrival_time   - (INTERVAL 1 MINUTE) AS window_open
12        ,departure_time + (INTERVAL 1 MINUTE) AS window_close
13    FROM data) A
14
15LEFT JOIN (
16    SELECT *
17        ,DATEDIFF('seconds', arrival_time, departure_time) AS duration
18    FROM data) B
19
20ON ((B.arrival_time <= A.window_open AND 
21    	(B.arrival_time   + TO_SECONDS(B.duration)) >=  A.window_open) OR
22    (B.arrival_time >= A.window_open AND 
23                                  B.departure_time  <= A.window_close) OR
24    (B.arrival_time >= A.window_open AND
25    	(B.departure_time - TO_SECONDS(B.duration)) <= A.window_close))
26GROUP BY 1, 2, 3, 4

A small, succinct query such as this will need a bit of explanation to take it all in. Here’s one below, reproducible in Python (make sure to install duckdb first!). Expand it to view.

SQL with explanation.

Avinash's Blog

projects

Featured projects

Other work or contributions

BasicsByHand: Logarithms

Premise

Prerequisites

Logarithms and key properties

What is a logarithm?

Restrictions on bbb and yyy

Properties of logarithms

Sum of logs to the same base

Logs raised to an arbitrary exponent

Difference of logs

Changing of base

The Euler Method

Simplifying log calculations

Calculating log⁡1.55\log1.55log1.55

Iteration 1

Iteration 2

Iteration 3

Iteration 4

Further iterations

Back to the Euler method

Takeaway

References

BasicsByHand: Square Roots

Premise

What method?

Building the Intuition

Getting our foot in the door

We continue by applying a common formula…

Connecting the dots

One more step for clearer understanding

Estimating to more decimal points

Conclusion

References

Resolving I/O Bottlenecks for 100K Small Files with LMDB

Premise

Potential Solutions

Partitioning

Separate (object) storage, maintain only the index

What about a… database?

The solution I landed on

What about a… different kind of database?

LMDB

Storing and retrieving image data along with its metadata

The code

Caveats

Avoid PIL, or pay the (small) price

The max_size_as_mb argument

Concurrency and LMDB

Alternatives

Resolving Multiprocessing Bottlenecks in Legacy Batch Pipelines

Premise

In detail

Demonstration code

The problem

But why?

The solution

Ending notes

Footnotes

Quick hacks to make client-ready presentations

Premise

What is it?

How do I get it?

What can I do with it?

Apply properties of shapes directly

Beautify charts with Snappable properties

Customize the labels programmatically

Screenshots of the chart snapability

Align table dimensions

Finding representative samples efficiently for large datasets

Premise

In a hurry?

Why do we need representative samples?

Define the data

Get a specific dataset

Specify the task

Finding representative samples

Restrictions on $b$ and $y$

Calculating $\log1.55$

The `max_size_as_mb` argument

Getting `SentenceTransformer` embeddings

The concept of `LIST` columns