Sql on staticnotes.org

TIL how to investigate missing data easily with DuckDB

Fri, 16 May 2025 14:00:00 +0000

In data engineering pipelines we commonly store output data in S3 buckets at various stages. In this particular case we stored the input data received from customers as csv files in the S3 bucket s3://customer-ingested/customerA.csv and after some transformations we stored the transformed data as csv files in another S3 bucket s3://customer-transformed/customerA.csv.

Problem

For some of the customers the csv file was missing in the first bucket customer-ingested while we were sure that all customers had a file in customer-transformed. However, there was a legitimate reason why the file could be missing which was that in the application the customer had a state of is_deactivated = True.

The goal of this investigation was to find all the active customers for which we were missing data in the bucket customer-ingested.

Solution

The nice thing about DuckDB in this scenario is that I can combine filesystem data with information from an application table and work in (familiar) SQL land.

Here is the query:

with transformed as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_transformed from glob('s3://customer-transformed/*.csv')
	),
ingested as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_ingested from glob('s3://customer-ingested/*.csv')
	),
customer_status as (
	select customer_name, is_deactivated from read_parquet('s3://application_table_snapshot/customers.parquet')
)

select * from transformed
left join ingested on transformed.filenames_transformed = ingested.filenames_ingested 
left join customer_status as cs on transformed.filenames_transformed = cs.customer_name
where is_deactivated = False and filenames_ingested is null;

┌───────────────────────┬────────────────────┬────────────────────┬────────────────┐
│ filenames_transformed │ filenames_ingested │   customer_name    │ is_deactivated │
│        varchar        │      varchar       │      varchar       │    varchar     │
├───────────────────────┼────────────────────┼────────────────────┼────────────────┤
│ customerT             │ NULL               │ customerT          │ False          │
│ customerW             │ NULL               │ customerW          │ False          │
│ customerZ             │ NULL               │ customerZ          │ False          │
└───────────────────────┴────────────────────┴────────────────────┴────────────────┘

In the CTE, I am creating a table transformed with all the filenames in the S3 bucket customer-transformed. I am using DuckDB’s utility function glob(search_path) to retrieve the filenames. I do the same for the bucket customer-ingested.

To get the customer status I read from a recent replication snapshot of the application database to get customer_status.

Finally, I join the three expressions to identify missing files (missing values) in the column filenames_ingested for active customers.

TIL that DuckDB now supports Avro files

Wed, 12 Mar 2025 16:00:00 +0000

In the data engineering pipelines of my current company we use two main file formats:

Parquet files for analytical workloads (columnar-storage)
Avro files for transactional / event-based messages (row-storage)

In Querying remote S3 files I wrote about how I use DuckDB to query parquet files stored in S3. Recently, I noticed that DuckDB started supporting reading of Avro files in the same way.

Install extension

To install the extension open DuckDB and run:

INSTALL avro FROM community;
LOAD avro;

Read local files

Local files can be queried with the read_avro function:

select * FROM read_avro('my_dataset.avro');

Read files stored in S3

Remote files can be queried by providing the S3 URI (after logging into the AWS account ):

select * FROM read_avro('s3://my_bucket/my_remote_dataset.avro');

A few limitations of the extension are listed here .

TIL about the QUALIFY statement in SQL

Mon, 16 Sep 2024 00:00:00 +0100

Today I came across the QUALIFY clause which is supported in some SQL dialects. It’s not part of the SQL standard but supported by the main analytical databases like BigQuery, Snowflake, Oracle, Databricks, DuckDB, etc. It’s part of Snowflake’s dialect which is the data warehouse that I use at work. The QUALIFY statement lets me filter the result of a query based on the result of a window function.

The order of execution is the following:

-- SELECT <columns>
--   FROM <table>
--   [GROUP BY ...]
--   [HAVING ...]
--   QUALIFY <window function predicate>

QUALIFY is for window functions what HAVING is for group-by statements. It allows me to filter based on the outputs of these operations without having to use a subquery or CTE. Therefore I can write more concise SQL.

I often deduplicate data using the row_number() window function. Sometimes I just want the latest event for a particular user in a dataset.

An example would be:

-- logs with some user events and timestamps
with logs_annotated as (
    select
        logs.*,
        row_number() over (partition by user_id order by event_timestamp desc) as row_number
    from logs
)
select * from logs_annotated where row_number = 1

With QUALIFY I can write one statement instead:

select * from logs
qualify row_number() over (partition by user_id order by event_timestamp desc) = 1

This works because the QUALIFY statement is evaluated after the window function result has been computed and can therefore be used in filtering.

TIL how to set a lower threshold on a column in SQL

Wed, 07 Aug 2024 00:00:00 +0100

For the last couple of months I have come across multiple instances where I wanted to write an SQL query to apply a lower threshold to all the values in a particular column in one of our Snowflake tables.

I used quite clunky workarounds to do that. It turns out you can use the SQL functions GREATEST or LEAST for it.

Example

I am creating a temporary table in the first CTE with a column DATA_COLUMN that has positive and negative numbers.

with example_table as (
SELECT * 
FROM (
    VALUES 
        (1, -100.423), 
        (2, 5.0), 
        (3, -0.001), 
        (4, 100.0), 
        (5, 12.0)
) AS t (row_nr, data_column)
)

select GREATEST(DATA_COLUMN, 0.) as thresholded 
from example_table

Output:

The function’s primary purpose is to take the greatest value among a number of referenced columns, e.g. GREATEST(column1, column2, column3).

DuckDB use cases for data scientists: Querying remote S3 files

Sun, 25 Feb 2024 00:00:00 +0000

DuckDB is a pretty cool in-process OLAP analytical database that I started to spin up on the fly for quick data analysis. What SQLite is to Postgres, DuckDB is to Snowflake. It is a single executable without dependencies and stores databases in local files.

I can think of four use cases for data science work:

DuckDB supports larger-than-memory workloads by loading data sequentially. You can use it to analyse datasets that are too large for Pandas (and too small to justify PySpark).
I can query CSV, parquet, and JSON files directly from remote endpoints, e.g. S3, using SQL.
I can replace Snowflake queries with DuckDB queries in unit / integration tests.
I can set up a (DuckDB + dbt) data warehouse for local development.

I want to share here my workflow for the second use case. Inspecting parquet files in AWS S3 is a pain because I can’t easily inspect them in the AWS console. For a few months now I have used DuckDB to load, inspect, and analyse parquet files from the command line. I found this reduced my cognitive load in situations where I quickly want to check a remote file, because I don’t have to download the parquet file and write a Python script to inspect it.

Installing DuckDB

I use macOS and can install DuckDB via homebrew: brew install duckdb. To work with remote files I also install the httpfs extension.

I start the DuckDB shell with duckdb and run the SQL commands:

INSTALL httpfs;
LOAD httpfs;

Authentication with AWS

To load files from S3, I need to configure AWS credentials. I am assuming here that your workplace has configured AWS SSO temporary credentials, but this works also with static credentials (ACCESS_KEY_ID, SECRET_ACCESS_KEY). There are two ways of doing this:

1. Set AWS credentials in the session:

I can set AWS credentials inside a DuckDB session like this:

LOAD httpfs;
SET s3_region = 'eu-west-2';
SET s3_access_key_id ='???';
SET s3_secret_access_key ='???';

2. Let aws vault handle credentials

I use the AWS credentials management tool aws vault (see here ) which manages my temporary AWS credentials in the background and exposes them to my shell sub-process. I create the DuckDB shell with aws-vault exec [profile-name] -- duckdb which ensures that AWS credentials are automatically set and updated.

Querying parquet files

Let’s assume that I have multiple parquet files stored in an S3 bucket work-project and they all have the same schema:

s3://work-project/clickdata_001.parquet
s3://work-project/clickdata_002.parquet
[...]
s3://work-project/clickdata_100.parquet

These files might be placed in this S3 bucket as part of an ETL pipeline before they are loaded into a table in a data warehouse. Let’s imagine I want to investigate a bug in the pipeline and need to inspect the files.

Inspect schema of parquet file

To familiarise myself with the schema I run parquet_schema

LOAD httpfs; -- once at the beginning of the session
SELECT * FROM parquet_schema('s3://work-project/clickdata_001.parquet');

which will return:

clicked_at::timestamp
user_id::int
event_type::string

Query a parquet file

I can use postgres SQL dialect to query the parquet files with the read_parquet function, e.g.

--- count number of events in the file
SELECT count(*) FROM read_parquet('s3://work-project/clickdata_001.parquet');

--- find the last event time of a particular user
SELECT max(clicked_at) FROM read_parquet('s3://work-project/clickdata_001.parquet') where user_id = 1234;

Query multiple parquet files

Say I want to find the users with the highest number of events across all files. I can use glob syntax with * to run a query against all files as if they were one table:

SELECT user_id, count(*) as num_events FROM read_parquet('s3://work-project/*.parquet') group by user_id order by num_events desc;

Create table from files

Let’s assume that all clickdata files together have a size of 500MB. It would be annoying if I had to download these files for every query that I want to run. Let’s instead create a table from the files.

create table clickdata as select * from read_parquet('s3://work-project/*.parquet');

This table is stored in memory until we close the shell, which allows me to run different queries against it until I am done with my analysis, e.g.

select * from clickdata;

I can also store the table in a local database on my computer in case I need to work with the data for a longer time. The following SQL statement

EXPORT DATABASE 'mydatabase' (FORMAT PARQUET);

will create a local folder mydatabase which stores the tables currently in memory. In a later DuckDB session I can reload the table from the database using the following command

IMPORT DATABASE 'mydatabase';

Query column statistics

A nice feature of the parquet file format is that it stores statistics about each column , e.g. min, max, null_count, distinct_count. If you are interested, you can inspect parquet file metadata using select * from parquet_metadata('file.parquet'); Assume the click event dataset is several GB large and I want to identify the earliest clicked_at time. I would run the query below.

SELECT min(clicked_at) FROM read_parquet('s3://work-project/*.parquet');

Fortunately, DuckDB uses the column statistics stored in the parquet files to compute the answer without having to download the whole dataset from S3. Simon Willison shows a more extreme example of this feature.

DuckDB is a great tool for quick investigations of remotely hosted files. Especially after having configured automatic handling of AWS credentials I can spin up a DuckDB shell with one command. I hope it becomes a time saver for you too.

TIL how to generate a dbt staging model description from Snowflake table

Wed, 04 Oct 2023 00:00:00 +0100

In my team we use dbt to create, document, and test data models in our data warehouse, Snowflake . We use different layers of models that start from the raw data and in each layer increase complexity and specialisation for the target use case:

source data in a raw table
thin staging models (standardisation of columns, no joins, no aggregations)
intermediate models (recurring building blocks that are defined once and used across models in the next layer)
mart models (most detailed models organised by department / business context / application)

Problem

The creation of staging models can sometimes be a bit mind-numbing, especially when the raw table has many columns. Let’s take a small example table USERS:

id	first_name	last_name	country	is_active
1	Peter	Clark	US	True
2	Sarah	Svenson	Sweden	False
3	Roberto	Gonzales	Spain	True

and dbt staging model stg_users.sql:

select 
id::number as id, 
first_name::varchar as first_name, 
last_name::varchar as last_name, 
country::varchar as country, 
is_active::boolean as is_active
from {{ source('RAW_DATA', 'USERS') }}

The staging model definition typically follows the pattern: raw-table-column-name::type as staging-model-column-name. I have used vim macros in the past to quickly write the staging model definition, but often end up copy-pasting and looking up the inferred types in Snowflake for ten minutes.

Solution

We can run two SQL queries in Snowflake to generate most of the dbt model definition:

desc table "RAW_DATA"."DATA_SCHEMA"."USERS";
select CONCAT("name", '::', "type", ' as ', "name", ',') as dbt_model_def from table(result_scan(LAST_QUERY_ID()))

The first line (DESCRIBE TABLE) outputs the table definition. The second command takes the column name and type information about the table and concatenates them into the right format:

LAST_QUERY_ID(): returns the id of the desc table query
RESULT_SCAN(): returns the result of the last command
TABLE(): converts the output to a queryable table

The output of the query is a table with each row formatted with the above pattern. We can copy-paste the output in one step into our staging model file.

TIL how to use Snowflake's VALUES sub-clause

Tue, 03 Oct 2023 00:00:00 +0100

Today I found out about the useful VALUES() sub-clause in Snowflake that allows you to dynamically generate a fixed, known set of rows inside your query.

Problem

I had a table exchange_rates of EUR-currency pairs with their corresponding exchange rates

date	base_currency	target_currency	exchange_rate
2023-10-03	‘EUR’	‘GBP’	0.87
2023-10-03	‘EUR’	‘USD’	1.05
2023-10-03	‘EUR’	‘JPY’	155.95

that I wanted to join with a table transactions which contained transaction prices in different currencies:

id	price	price_currency
1	100.53	‘GBP’
2	1021.68	‘EUR’
3	7.99	‘JPY’

My goal was to standardise all prices to EUR.

The SQL query to join the two tables would be:

select 
id,
price,
price_currency,
exchange_rate
from transactions as t
inner join exchange_rates as er on er.target_currency = t.price_currency

The issue was that exchange_rates did not have an EUR-EUR currency pair and I didn’t have access to the data source. So, the inner join would filter out all EUR transactions (or leave NULLs in case of a left join). This is a bit of a contrived problem to show an application of VALUES(). One could alternatively fill the NULLs with 1.0s after a left join.

Solution

I used Snowflake’s VALUES() to dynamically generate another row in the query. I use this with a UNION statement to complete the exchange_rates table before joining:

with exchange_rates_complete as (
  select * from exchange_rates
  union
  select * from (values( '2023-10-03', 'EUR', 'EUR', 1.0, 1))
)

select 
id,
price,
price_currency,
exchange_rate
from transactions as t
inner join exchange_rates_complete as er on er.target_currency = t.price_currency

The joined table looks like this:

id	price	price_currency	exchange_rate
1	100.53	‘GBP’	0.87
2	1021.68	‘EUR’	1.0
3	7.99	‘JPY’	155.95