• one week after the final submission deadline,
• after being able to see the actual winning solution,
• spending half of my Claude Pro weekly limits, and
• spending some extra 25$ from my pocket on NVIDIA Brev.

But let us focus on the positive side: what did I learn? I would say a lot. As someone relatively new to deep learning but relatively experienced in scientific Python (or at least I would like to think so), I was very curious to see how the “good practices” that I have learned over the years for computational pipelines that I run locally apply to deep learning pipelines that run on GPU servers. Here are some thoughts.

The CI2026 starter kit

All the participants started from a base repository set up by Tobias Finn (a big shout-out for the organization) with the following structure:

CI2026-StarterKit/
+-- configs/                  # Hydra settings
|   +-- data/                 # dataset paths and splits
|   +-- experiments/          # baseline experiment presets
|   +-- model/                # architecture hyperparameters
|   +-- suite/                # validation/test forecast suites
|   +-- test_data/            # test-set paths
|   +-- train.yaml            # default training config
|   +-- forecast.yaml         # default forecast config
|   +-- submit.yaml           # default submission config
+-- data/                     # local runtime data, not committed
|   +-- train_data/           # downloaded inputs and targets
|   +-- models/               # checkpoints and logs
|   +-- forecasts/            # generated netCDF predictions
+-- notebooks/                # exploration and analysis
+-- scripts/                  # Hydra entry points
|   +-- train.py              # train a model
|   +-- forecast.py           # generate forecasts
|   +-- evaluate.py           # score validation forecasts
|   +-- submit.py             # forecast and submit predictions
+-- starter_kit/              # installable Python package
    +-- data.py               # PyTorch datasets
    +-- layers.py             # input normalisation
    +-- model.py              # shared training interface
    +-- baselines/            # reference model implementations
        +-- mlp.py
        +-- parametric.py
        +-- sundquist.py

This is indeed a structure that I have seen quite often in GitHub but I had never actually used myself, essentially because it is designed to run on a GPU. Instead, I have sticked to my personal flavour of the cookiecutter-data-science template and my GPU requirements have been fullfilled serverlessly by writing modal apps, e.g., to run AIFS forecasts, or to run training/inference steps in computer vision. But we’ll get to that later — for this workshop, we had 100$ of Brev credits (graciously offered by Nvidia) to run on NVIDIA L4 GPUs throughout the hackathon, so we did not have to worry about GPU cost.

Snakemake, my dearest friend

After launching the provided Brev environment, accessing it through JupyterLab and setting up conda environment, the first step of the computational pipeline was to download (and unzip) the input dataset from HuggingFace. Then, we copied and pasted a few of the commands to run training, inference and evaluation with the provided baseline multi-layered perceptron (MLP) model:

# for the baseline MLP, per Hydra defaults
python scripts/train.py n_epochs=50 learning_rate=5e-4 device=cuda
python scripts/forecast.py device=cuda
python scripts/evaluate.py \
    --prediction_dir data/forecasts/baseline_mlp \
    --to_json \
    --output_path scores/baseline_mlp.json

as well as with the physical reference baseline following Sundqvist et al. (1989)¹

python scripts/train.py +experiments=baseline_sundquist device=cuda
python scripts/forecast.py +experiments=baseline_sundquist device=cuda
python scripts/evaluate.py --prediction_dir data/forecasts/baseline_sundquist
python scripts/evaluate.py \
    --prediction_dir data/forecasts/baseline_sundquist \
    --to_json \
    --output_path scores/baseline_sundquist.json

However, it only took me copy-pasting these commands to mess up and overwrite the MLP scores with its Sundqvist by forgetting to update --output_path. At that point, it immediately occured to me that I usually lean on Snakemake to avoid these mistakes, so why shouldn’t I do the same here? Enter Snakemake.

Since Hydra already takes care of managing paths for the train and forecast parts, the main gap was the scripts/evaluate.py step, where --output_path had to be specified manually. Adding a thin Snakemake wrapper around all three steps meant that paths flow automatically from the {run} wildcard through the whole chain via a directed acyclic graph (DAG): the right commands run in the right order and already-completed steps are skipped on re-runs. We also added the data download and unzip steps so that the full pipeline — from raw dataset to evaluation scores — could be expressed as a single snakemake call. Furthermore, region expansion, which would otherwise require looping over the four ERA5 and AIMIP domains by hand, became a one-liner with expand(). Convenience targets like all and test_forecasts replaced sequences like:

for region in era5_region1 era5_region2 aimip_region1 aimip_region2; do
    python scripts/forecast.py +experiments=baseline_mlp \
        +test_data=val_${region} device=cuda
done
python scripts/evaluate.py \
    --prediction_dir data/forecasts/baseline_mlp \
	--to_json \
	--output_path scores/baseline_mlp.json

with a single call:

snakemake --config runs=baseline_mlp -j1 --resources gpu=1

The checkpoint and forecast paths (e.g., data/models/{run}/best_model.ckpt) are declared in both Snakemake (for DAG construction) and in the Hydra configs (for runtime resolution). This is unavoidable: Snakemake must know file paths statically to determine what to run and what to skip. Nonetheless, in practice the overlap is small and the convention is simple enough that keeping both synchronized is a one-line change.

The value of adding a Snakefile did not stop there though: we realized that reference values for normalization were pre-computed from the training data and hardcoded to mlp.py, so in order to add further meteorological calculations as derived input features (computed on-the-fly in the forward pass), we needed to add the computation of reference values to the pipeline with a new compute_normalization rule. The overall pipeline at this point looked as follows:

flowchart TB
    download_data(download_data)
    data_ready["data/train_data/.download_complete"]
    compute_norm(compute_normalization)
    norm_stats["data/stats/normalization_{...}.json"]
    train(train)
    ckpt["data/models/{run}/best_model.ckpt"]

    subgraph forecasts["forecast for each split and region"]
        direction TB
        forecast_val(forecast)
        val_forecasts["val_{region}.nc<br/>for the four regions"]
        forecast_test(forecast)
        test_forecasts["test_{region}.nc<br/>for the four regions"]
    end

    evaluate(evaluate)
    score["scores/{run}.json"]
    all_target["all / val_scores"]
    test_target["test_forecasts"]

    download_data --> data_ready --> compute_norm --> norm_stats --> train --> ckpt
    data_ready --> train
    ckpt --> forecast_val --> val_forecasts
    ckpt --> forecast_test --> test_forecasts
    val_forecasts --> evaluate --> score --> all_target
    test_forecasts --> test_target

    classDef data fill:$DATA_FILL,stroke:$DATA_STROKE,color:$NODE_TEXT;
    classDef task fill:$TASK_FILL,stroke:$TASK_STROKE,color:$NODE_TEXT;
    classDef gpu fill:$GPU_FILL,stroke:$GPU_STROKE,stroke-width:2px,color:$NODE_TEXT;
    classDef target fill:$TARGET_FILL,stroke:$TARGET_STROKE,color:$NODE_TEXT;
    class data_ready,ckpt,val_forecasts,test_forecasts,score,norm_stats data;
    class train,forecast_val,forecast_test gpu;
    class download_data,evaluate,compute_norm task;
    class all_target,test_target target;
    style forecasts fill:$SUB_GPU_FILL,stroke:$SUB_GPU_STROKE,stroke-width:2px

Interlude: our submission

I am not going to develop much on our submitted network because we did not win. My colleague was right to suggest using a U-Net², to which we added sphere-aware padding (circular along the longitude axis, replicate along latitude) and added relative humidity as a derived input feature computed on-the-fly in the forward pass alongside the raw pressure-level fields and two static auxiliary fields (land-sea mask and orography). The best results network were obtained with a wide configuration (96 base channels, 3 downsampling stages).

From my experience in computer vision, I actually thought of including augmentation-like transforms to our training loop, but discarded it since I considered that these would disregard the semantics of the land-sea mask and orography. However, in retrospective, the most likely reason we fell short was actually including those static geographic fields. By seeing the land-sea mask and orography during training, the network learned to associate cloud patterns with specific landscape features of the training region rather than the underlying atmospheric dynamics, and those associations do not transfer to a held-out region with different geography.

The winning solution description actually pointed at this: a geography-agnostic design that drops all static inputs and relies solely on pressure-level fields generalizes across regions by construction.

Comparing experiments

The starter kit provided logged the training loss at each epoch to a CSV file of the form data/models/<exp_name>/train_log.csv. In order to better compare experiments, we set up integration with Weights & Biases. We could have also used MLflow but I could not find a free hosted solution which allowed multiple collaborators — something included within the free academic plan of Weights & Biases.

In any case, besides our submitted unet_brute_xlarge configuration, I added two additional experiments a posteriori. The first is geounet_wide_rh_hilr, a geography-agnostic UNet variant based ont the winning solution, to which I added on-the-fly relative humidity to the forward pass and set a high learning rate (0.001). The second is a (vibe-coded) SegFormer-B2 variant segformer_b2_rh, i.e., a transformer-based architecture that replaces the CNN backbone with a Mix Transformer encoder and a lightweight MLP decoder. The following validation loss curves from Weights & Biases summarize the comparison:

The segformer_b2_rh setup performed very poorly: its pre-training on natural images (ImageNet) and classification-oriented decoder were likely not the best choice. I may further add a U-Net with a Swin Transformer encoder, since its shifted-window attention is probably better to capture local spatial structure.

I also vibe-coded a notebook to visualize the outputs of each model:

and another notebook to inspect the distribution of each physical variable accross pressure levels and potentially identify improvements to the z-score normalization — although some preliminary experiments suggest that these may make very little effect.

On the overall development workflow: another case for serverless computer?

The setup allowed editing code directly in Brev using JupyterLab. However, as a long time emacs user, that was not my preferred approach³. Additionally, if I am not mistaken, only one user could access JupyterLab at a time, so we ended up adopting this workflow:

flowchart TB
    subgraph Local["Local laptop"]
        direction TB
        local_edit["Edit code, configs and workflows"]
        local_push["Commit and push changes"]
        local_refine["Refine code or configuration"]
    end

    subgraph GitHub["GitHub"]
        direction TB
        github_repo[("Repository")]
    end

    subgraph Brev["Brev GPU instance"]
        direction TB
        brev_pull["Pull latest changes"]
        subgraph BrevLoop["Experiment loop"]
            direction TB
            brev_run["Run training, forecasts and evaluation"]
            brev_inspect["Inspect results"]
            brev_update["Pull update"]
            brev_rerun["Re-run GPU scripts"]
        end
    end

    local_edit --> local_push --> github_repo --> brev_pull
    brev_pull --> brev_run --> brev_inspect --> local_refine
    local_refine --> local_push
    github_repo --> brev_update --> brev_rerun --> brev_inspect

    classDef local fill:$TASK_FILL,stroke:$TASK_STROKE,color:$NODE_TEXT;
    classDef github fill:$TARGET_FILL,stroke:$TARGET_STROKE,color:$NODE_TEXT;
    classDef gpu fill:$GPU_FILL,stroke:$GPU_STROKE,stroke-width:2px,color:$NODE_TEXT;
    class local_edit,local_push,local_refine local;
    class github_repo github;
    class brev_pull,brev_run,brev_inspect,brev_update,brev_rerun gpu;
    style Local fill:$SUB_TASK_FILL,stroke:$SUB_TASK_STROKE
    style GitHub fill:$SUB_NEUTRAL_FILL,stroke:$SUB_NEUTRAL_STROKE
    style Brev fill:$SUB_GPU_FILL,stroke:$SUB_GPU_STROKE,stroke-width:3px
    style BrevLoop fill:$SUB_GPU_FILL,stroke:$SUB_GPU_STROKE,stroke-width:2px

The positive part of this set up is that all changes need to go through git, which can enforce good practices by means of pre-commit hooks and potentially continuous integration (CI). The bad part is that the GPU was idle most of the time. In fact, after setting up the Brev instance on the first day of the hackathon, my colleague and I went for lunch and realized that about $2 of credits were gone becuse we simply did not “Stop” the Brev deployment.

Although the training of the heaviest models could take about 2 hours to complete on the L40 GPU, iterative development of Python and yaml files largely dominated our overall time to solution. Using a GPU server for that not only is overkill but also inconveniennt as it hampers the good practices provided by LSP features in integrated development environment (IDEs) — especially for Emacs users such as myself. I thus believe that this is another great use case for a serverless setup:

flowchart TB
    subgraph Local["Local laptop"]
        direction TB
        local_edit["Edit code, configs and workflows"]
        local_push["Commit and push changes"]
        subgraph LocalLoop["Experiment loop"]
            direction TB
            local_submit["Submit GPU job from local CLI"]
            local_inspect["Inspect results"]
            local_refine["Refine code or configuration"]
        end
    end

    subgraph GitHub["GitHub"]
        direction TB
        github_repo[("Repository")]
    end

    subgraph GPU["Serverless GPU"]
        direction TB
        gpu_run["Run training, forecasts and evaluation"]
    end

    local_edit --> local_push --> github_repo
    local_push --> local_submit --> gpu_run
    gpu_run -->|"logs and artifacts"| local_inspect
    local_inspect --> local_refine --> local_push

    classDef local fill:$TASK_FILL,stroke:$TASK_STROKE,color:$NODE_TEXT;
    classDef github fill:$TARGET_FILL,stroke:$TARGET_STROKE,color:$NODE_TEXT;
    classDef gpu fill:$GPU_FILL,stroke:$GPU_STROKE,stroke-width:2px,color:$NODE_TEXT;
    class local_edit,local_push,local_submit,local_inspect,local_refine local;
    class github_repo github;
    class gpu_run gpu;
    style Local fill:$SUB_TASK_FILL,stroke:$SUB_TASK_STROKE
    style GitHub fill:$SUB_NEUTRAL_FILL,stroke:$SUB_NEUTRAL_STROKE
    style GPU fill:$SUB_GPU_FILL,stroke:$SUB_GPU_STROKE,stroke-width:3px
    style LocalLoop fill:$SUB_LOOP_FILL,stroke:$SUB_TASK_STROKE,stroke-width:2px

I may explore a Modal version of this set up soon to verify whether it is another good case for serverless computing.

Notes

2012: The IPython team released the IPython Notebook, and the world has never been the same

– Jake Vanderplas¹

Whether you use Jupyter notebooks or not, it seems hard to disagree with the quote above. I actually did not know that the name “Jupyter” is a reference to the three programming languages to which the IPython notebook was extended in 2014, i.e., Julia, Python and R - again, thank you Wikipedia. By 2018, more than 100 languages were already supported, and quoting Lorena Barba, “For data scientists, Jupyter has emerged as a de facto standard”². In 2021, Jupyter Notebooks voted by Nature readers as the third software codes that had biggest impact in their work (after the Fortran compiler and the Fast Fourier transform)³.

Unlike Nature and Wikipedia, I would break down Jupyter notebooks into three components (instead of two): the user interface (UI) to edit code and text, the kernel that executes the code and the underlying JSON file format with the “.ipynb” extension into which notebooks are saved and shared. Altogether, the first two components are not much of a novelty since they essentially constitue a read–eval–print loop (REPL) environment, which were developed in the 1960s⁴. In my view, the novelty is more about how the first two occur within a document-like notebook file editing experience which allows not only to write code cells, but also to move, execute and delete them as desired until the overall look of the notebook is considered satisfactory. At this point, one can save and export the notebook, obtaining a visually-appealing document mixing code, documentation and results, which collectively tells a story, aka, a literate computational narrative⁵.

However, this editing freedom comes at a non-negligible cost, i.e., the risk of exporting an inconsistent computational pipeline after moving and deleting cells that have been executed (potentially several times). As a matter of fact, in 2019, a study collected a corpus of ~1.16M notebooks from GitHub and found that only 3.17% could be executed providing the same results. Many experienced programmers have raised warnings about how notebooks obscure the state, which can be especially dangerous for beginners⁶. Additionally, the JSON-based ipynb format of vanilla Jupyter notebooks poses many challenges when it comes to version control, practically requiring you to use external tools such as Jupytext and nbdime to prosper in the face of adversity. But I think that in any case, it is fair to say that, if used properly, Jupyter notebooks are a very powerful tool.

Let us now go back to the first of the three components of notebooks outlined above, namely the interface to edit notebooks. The Project Jupyter provides two options, both in the form of a web application: the classic Jupyter Notebook and JupyterLab, with the latter providing a more fully-featured integrated development environment (IDE) to edit notebooks and other text files, run consoles and manage files. Additionally, many other options have emerged in view of the popularity of Jupyter noteebooks, both web-based (e.g., Google Colab, Azure Machine Learning, GitHub Codespaces, DeepNote…) and client-based (e.g., VSCode, PyCharm, DataSpell…). Nevertheless, for Emacs users such as myself, these are hardly viable options, even when using extensions to emulate Emacs key bindings. Therefore, the goal of this post is to explore which Emacs configurations provide the best Jupyter-like experience.

Desired features

Before we go any further, it is worth noting that this post is inevitably highly opinionanted, or in other words, influenced by the way in which I use Jupyter notebooks, which is twofold. First, like most users, I use notebook to interactively explore, e.g., a dataset, a library or the like. As the notebook code becomes more mature, I may manually move it to Python modules or scripts. Such a development process can lead to both Python libraries or data science repositories. This brings me to the second way in which I use notebooks, namely to tell a story, which can either be the documentation or user guide for a Python library as well as the computational pipeline to reproduce the results of an academic article.

Based on what has been said so far, the main desired features, in no particular order, are the following:

• Beyond (Python) code: mixing Python (or another programming another language), markdown and inline plots can constitute a great recipe to create literate computational narratives, so it is important to ensure that the Emacs setup supports them - otherwise there is practically no reason to use a hardly human-readable JSON-based format instead of plain text “.py” files.
• Proper version control with the possibility to include cell outputs: following the previous point, I am convinced that Jupyter notebooks would not have been this successful if web platforms such as GitHub or GitLab did not offer a rich rendering of notebook files, as reading them directly in the browser can be a very convenient way to navigate computational narratives such as tutorials, computational pipelines to reproduce an academic paper or the like. From this perspective, it is essential that the version-controlled notebooks can include the cell outputs - I emphasize the can, because in some use cases, it may be better to strip out the cell outputs to reduce file sizes and to ease version control, yet it is important that the option to include the outputs exists.
• IDE features (environment-aware and notebook-wide): as highlighted in the introduction, the out-of-order editing and execution of notebooks can dangerously obscure the state, which can be prone to errors and elusive results. Features provided by IDEs such as completion, documentation, on-the-fly syntax checking, navigation, refactoring and the like can help prevent many of these issues. In Emacs, there are several options to provide these IDE features⁷ - the two main challenges are to ensure that IDE features are aware of the Python environment used in each Emacs buffer, and notebook-wide, i.e., using information (such as imports and variables) defined not only in the cell being edited but in all the notebook cells.

Overview of Emacs configurations for Jupyter notebooks

The following sections assume an overall setup in which multiple virtual Python environments coexist in the same computer. To this end, my configuration⁸ uses conda (or more precisely, its much faster reimplementation named mamba), with automatic detection and activation of the right conda environment for a buffer using conda.el. Then, Jupyter interacts with conda environments using the ipykernel package⁹.

Although I believe that conda features major advantages over other virtual environment and package managers¹⁰, it should be possible to achieve an analogous setup using alternative tools (e.g., virtualenv and pip¹¹). Similarly, my configuration uses pyright (which actually can be installed using conda)¹², a language server protocol (LSP) to obtain IDE features, but feel free to use another tool (e.g., eglot) if you prefer.

Let us now finally move on to experiencing Jupyter notebooks in Emacs.

EIN

The first option is Emacs IPython Notebook (EIN). Using it requires very little Emacs configuration and is quite straightforward: we need to launch a jupyter notebook server (e.g., by running jupyter notebook in a shell or running M-x ein:run in Emacs), then run M-x ein:login. Then, we can either open an existing notebook or select a kernel and create a new notebook. The key bindigns are very convenient and provide a very jupyter-like experience: C-c C-a and C-c C-b respectively create cells above and below the current cell, cells can be moved up and down using M-<up> and M-<down>, C-c C-c executes the current cell, C-c C-w copies the current cell and C-c C-y yanks it, and other bindings allow executing all cells, restarting the kernel and many more. Additionally, you can easily mix Python with markdown, IPython magics, inline plots and shell commands.

Creating a new notebook for a specific Jupyter kernel is also quite straight-forward, i.e., the ein:notebooklist buffer lets you select an option among the existing kernels, and then you just need to click the [New Notebook] button.

While EIN provides an Emacs interface to emulate the web-based Jupyter notebook with its major advantages (i.e., cell organization, markdown rendering, inline results and plots…), such a resemblance comes at the cost of raising several important issues that require careful consideration. First, the “undo” command only works within a given cell, leading to unintuitive undo behaviors when moving accross cells¹³. Furthermore, a critical aspect to consider is that there is no “undo” for cell deletion, which can easily prompt one to very strongly regret certain typing mistakes. Similarly, Emacs auto-saving and file recovery features do not apply to “ipynb” files edited with EIN, so you can easily lose valuable work if your computer crashes.

Finally, IDE features in EIN are provided using elpy, which provides notebook-wide completion, navigation and documentation, and can be environment aware by using pyvenv-activate or pyvenv-workon. However, based on my experience (and other users have reported similarly), the main issue with elpy is that the background elpy process quickly ends up consiming an entire CPU and provides very slow responses. See a minimal example of how completions and documentation easily become excessively slow:

Moreover, as of May 2023, elpy is unmantained, which I hope that prompts the EIN developers to switch to a LSP client or another package to provide IDE features.

code-cells and emacs-jupyter

Another notable option to make Emacs interact with Jupyter kernels is the emacs-jupyter package. The main difference with EIN is that emacs-jupyter does not provide a graphical UI to edit notebooks. Instead, it just provides the underlying interaction with Jupyter kernels within Emacs, thus giving the user more freedom to chose the notebook editing interface.

The most straight-forward way to use emacs-jupyter is to use the M-x jupyter-run-repl command to run a REPL using a specific Jupyter kernel, open a “.py” file (or whichever language extension corresponds to the Jupyter kernel), write Python code and send the desired lines, regions or the whole buffer to the REPL. While many developers may be perfectly happy with such an approach, it does not represent any major conceptual change with respect to the initial design of the REPL environments in the 1960s. Therefore, it is missing two of the main strengths of Jupyter notebooks, i.e., the organizational compartmentalization provided by notebook cells and the storytelling ability of the exported notebook file, mixing programming code, markdown and inline plots.

In order to emulate the cell-like organization of Jupyter notebooks, code-cells provides a lightweight mode to read, edit and write “.ipynb” files. To that end, it first converts the JSON-based “.ipynb” file to a plain-text script representation, in which certain Python comment syntax is interpreted as cell boundaries. By default, such a conversion is performed by Jupytext (which needs to be installed separately), but it can easily be configured to use any other command (e.g., pandoc) by customizing the code-cells-convert-ipynb-style. In fact, it is also possible to convert “.ipynb” files to another format such as markdown or org, and then edit them in the associated Emacs mode.

Notebooks as Python scripts

When opening an existing notebook using the default settings, it is automatically converted to a Python script, with line comments of the form # %% defining cell boundaries. Additionally, the converted file begins with a YAML-like header block enclosed by two comment lines of the form # ---, which includes notebook metadata such as the kernel information as well as Jupytext format and version information. In order to execute cells, we can run M-x jupyter-repl-associate-buffer to associate the buffer to an existing emacs-jupyter REPL or create a new one choosing the appropriate Jupyter kernel. Cells with shell commands and IPython magics can be commented, which avoids syntax checking errors and allows the other IDE features to work properly¹⁴. An associated caveat is that such cells have to be uncommented before they can be evaluated by emacs-jupyter - otherwise we are just evaluating Python comments and thus seeing no effect whatsoever¹⁵. The following screencast illustrates how a notebook can be edited as Python script and its cells can be executed interactively using emacs-jupyter:

The fact that we are editing a Python file-like buffer has several key advantages when compared to the web-like interface of EIN, i.e., IDE features work fast and other basic features such as undo, auto saving and recovery work as in any regular Emacs file. However, in my view, notebooks are displayed more nicely in EIN: there is no need for a YAML-like header block¹⁶, cells feel more demarcated, markdown cells can be properly rendered instead of appearing as Python comments and most importantly, cell outputs (including plots) appear within the same notebook buffer. The latter brings me to the main shortcoming of this approach: as of May 2023, it is not possible to include the cell outputs in script representations of Jupyter notebooks¹⁷. As discussed in the introduction, this can be a crucial deficiency as it disables one of the major strengths of Jupyter notebooks, namely their storytelling ability. Without displaying cell outputs, notebooks are no longer such a suitable medium for tutorials, documentation or supporting materials of an academic article. Nonetheless, the combination of code-cells, Jupytext and emacs-jupyter can be very well suited to use cases where cell outputs can be ommited.

Finally, the configuration provided for this section of the post includes a couple of changes. First, I have found the default key bindings (all prefixed with C-c %) rather impractical, so the cell navigation, moving and execution key bindings are redefined to mimic EIN. Secondly, unlike EIN, code-cells and Jupytext do not provide any command to create a new notebook (with its metadata), so the configuration includes a function named my/new-notebook, which takes the new notebook file path and the desired Jupyter kernel as arguments.

Notebooks as org files

As its name suggests, org mode was initially created as an Emacs to organize notes and tasks. Nevertheless, what started lightweight markup language quickly evolved into a much more powerful system, notably thanks to Babel, a feature of org mode¹⁸ that allows users to include and execute source code blocks within org files. This is actually quite reminiscent of Jupyter notebook - in fact, one may argue that org and Babel provide a considerable superset of features, as they allow using multiple programming languages in the same file and offer richer and more flexible export capabilities.

It is actually very easy to configure code-cells to automatically convert notebooks to org files and edit them accordingly - provided that pandoc is installed, code-cells-convert-ipynb-style:

(setq code-cells-convert-ipynb-style '(
	("pandoc" "--to" "ipynb" "--from" "org")
	("pandoc" "--to" "org" "--from" "ipynb")
	org-mode))

When opening the example notebook using the above configuration, the resulting org buffer converts Jupyter cells to org code blocks with the jupyter-python language identifier, with the markdown cells which are translated to org syntax (which start with a line of the form <<a0c84e00>> that corresponds to the Jupyter cell id). Like when editing notebooks as Python scripts, we need a running emacs-jupyter REPL to evaluate code blocks. However, to execute jupyter-python blocks in org mode, we also need a session, which is started by the Jupyter kernel and maintains the state, e.g., including any variables or objects that have been defined so that they can be shared between code blocks. Sessions can be defined using the session header argument, e.g.:

#+begin_src jupyter-python :session foo
import matplotlib.pyplot as plt
import rasterio as rio
from rasterio import plot
#+end_src

By default, emacs-jupyter uses the kernel associated with the currently active environment (see a dedicated section about this in the introduction above), yet this can be overridden by setting the :kernel header argument. Additionally, it is most certainly desirable to set the :async header argument to yes to allow asynchronous execution of code-blocks - otherwise the buffer will be completely stalled until the code execution is completed. Altogether, an example code block looks as follows:

#+begin_src jupyter-python :session foo :kernel jupyter-emacs :async yes
import matplotlib.pyplot as plt
import rasterio as rio
from rasterio import plot
#+end_src

While this highlights another advantage of org over Jupyter, namely the ability to use multiple languages and sessions in a single file, having to set these header arguments in each code block is quite verbose when only a single session and language is required - which surely corresponds to most use cases. Luckily, such a repetition can easily be avoided by setting the header-args property at the beginning of the file¹⁹:

#+PROPERTY: header-args:jupyter-python :session foo :kernel jupyter-emacs
#+PROPERTY: header-args:jupyter-python+ :async yes

so that all jupyter-python src blocks include the defined header arguments implicitly. Note that although it is beyond the scope of this post, it is further possible to set the session property to use remote kernels and much more²⁰. In any case, once the session and the kernel have been properly configured, code blocks can be executed using the C-c C-c key bindings. The screencast below shows how the example notebook can be set up and executed as an org file buffer:

Likewise notebooks as Python scripts, org files are plain text, so the basic Emacs undo, auto saving and recovery work out of the box. Additionally, code blocks feel more demarcated than comment-delimited cells in Python scripts, and results can be displayed inline, including images - to that end, run org-toggle-inline-images (by default bound to C-c C-x C-v) after a code block producing an image has been executed²¹. Regarding key bindings, new code blocks (and other kinds of org mode blocks) can be inserted using the org-insert-structure-template command, by default bound to C-c C-,, and Babel provides a series of useful functions bound to the C-c C-v prefix that facilitate navigating to the next or previous code block (using C-c C-v n or C-c C-v p respectively) as well as to execute all code blocks in the buffer (bound to C-c C-v b). Even though I find these key bindings to be less handy than its counterparts in EIN and code-cells, this is understandable as org mode provides much more features and flexibility. Following the screencast above, the one below displays how code blocks can be edited and executed:

In my view, the main issue of using org to emulate Jupyter is related to how code blocks are edited. First, in order to get proper indentation and IDE features, code blocks must be edited in a separate buffer by using the org-edit-special command, by default bound to C-c '. But besides the inconvenience of having to use multiple buffers²² (especially in small screens), separate code editing buffers have a major drawback, i.e., they make getting proper IDE features excessively intricate (besides the default inspection and completion provided by emacs-jupyter). In fact, even after asking around in Reddit and GitHub issues I have not managed to achieve it²³.

Finally, even though pandoc supports round-trip conversion from org to ipynb since version 2.19.2 (the org propertiy definitions at the start of the file are lost, but this is a minor detail), results from code blocks are ommited, hence the converted “.ipynb” files do not include cell outputs²⁴.

Overall, org is nothing but adaptable so if you are comfortable tweaking your Emacs configuration, it is likely possible to suit org to your needs. In my experience, my inability to obtain proper notebook-wide IDE features when editing code is too much of a drawback.

A final note on polymode

Before wrapping up, it is worth noting that many of the problems in getting proper edition of Python code in org files is because the Python major mode is not active by default in org buffers, which makes sense because code blocks can be in many languages. Nonetheless, there exist libraries to support multiple major modes in a single buffer in Emacs - a notable example is polymode. It is possible to use code-cells with Jupytext or pandoc, transform notebooks to a plain-text representation such as markdown and then open them in a single polymode buffer so that fenced code blocks with a language identifier, e.g.:

```python
print("Here is some Python code")
```

can be edited in the language-specific major mode and sent to a running REPL. Nevertheless, doing so requires non-trival configuration based on a thorough understanding of how polymode works. Like with org mode, I have searched through blogs, GitHub and Reddit and I not been able to find any working configuration to make polymode work properly with emacs-jupyter or LSP²⁵.

Conclusion

As one would expect, the choice of an appropriate setup largely depends on the use case. The key points to consider from the configurations reviewed in this post are summarized below.

TL;DR

Use EIN if you:

• want an interface as close as possible to Jupyter notebook and Jupyterlab, with a nice notebook display with code, rendered markdown and results
• want a simple Emacs setup that works out of the box but is inevitable less customizable
• do not mind slow IDE features (e.g., completion, documentation…)
• are going to be (very) careful not to mistakenly delete cells and save your notebooks constantly

Use notebooks as Python scripts (i.e., code-cells interacting with an emacs-jupyter kernel, with notebooks converted to Python scripts with Jupytext) if you:

• want all the advantages of working with plain text files (i.e., fast IDE features, undo, autosave…)
• do not mind a lightweight emulation of the Jupyter notebook experience, with outputs in separate buffers, lesser-demarcated cells and without rendered markdown

Use notebooks as org files (i.e., org-mode interacting with one or serveral emacs-jupyter kernels, with notebooks converted to org files with pandoc) if you:

• want a highly versatile and custumizable setup that largely supersedes the features offered by Juptyer (e.g., multiple kernels in the same file, checklists, agenda, calendar…), while keeping the advantages of working with plain text files
• are comfortable with hacky Emacs configurations and do not mind a steep learning curve (e.g., lots of metadata involved such as properties or header arguments, complex key bindings…)

Final (personal) thoughts

After much experimentation, my preference leans towards a setup where notebooks are by default edited as Python scripts (with code-cells, emacs-jupyter and jupytext as reviewed above), mainly because I find that the advantages of working with a plain-text file (i.e., fast IDE features, undo, autosave…) outweight the better notebook representation offered by EIN.

In my experience, the cell-centric feel provided by EIN and org-mode (as well as in Jupyter and Jupyterlab) ease the conceptual organization of code in a way that is not matched in code-cells, but this is merely a UI aspect that could be easily improved. On the other hand, I find the inability of the “notebooks as Python scripts” approach to include inline inputs¹⁷ to be a bit more troublesome, especially since it prevents including outputs in version control - again, hampering the storytelling ability of Jupyter notebooks. The latter is why my setup keeps the possibility of editing notebooks with EIN by running M-x ein:run or M-x ein:login and opening the notebooks from the EIN menu. I find this useful once the content of the notebook is solid and I just want to review it (e.g., follow its story, visualize outputs inline, proofread markdown…) before commiting it to version control. Actually, I often perform this stage in the browser so that I can see exactly what the notebook will look like in GitHub.

Finally, I have never needed to use more than one programming language or Jupyter kernel within the same file, therefore I have felt overwhelmed by org mode. But again, your needs may be different - in any case, I hope that this post provides a thorough and helpful overview of the strengths and weaknesses of each approach to write Jupyter notebooks within Emacs.

Notes

After some preprocessing of the GPS traces in Yaoundé, we have a CSV file with the locations and times of the origins and destinations of 23712 trips during three weeks. For the sake of simplicity, let us consider only the 8943 trips whose destination falls within the longitudes of 3.8° and 3.85° and the latitudes of 11.4° and 11.5°. We then have a geo-data frame of the following form:

trip_gdf.head()

Let us then use OSMnx to get the street network ot the area:

# get convex hull in lat/long to query OSM
convex_hull = trip_gdf['dest'].unary_union.convex_hull
# get the street network and project it
G = ox.graph_from_polygon(convex_hull, network_type='drive')
# the undirected graph will do for our purposes
G = ox.project_graph(G.to_undirected(reciprocal=False))

We obtain a network of 4032 nodes and 5344 edges which looks as follows:

fig, ax = ox.plot_graph(G)

Using the trips geo-data frame and the street network shown above, we want to compute, for each of the trip destinations, the total street length that falls within a 100 m radius of the destination location. Let us use the code snippet below to illustrate the situation:

# let us first reproject `trip_gdf` to the network's CRS
dest_gdf = gpd.GeoDataFrame(geometry=trip_gdf['dest'])
dest_gdf.crs = 'epsg:4326'
trip_gdf['dest'] = ox.project_gdf(loc_gdf)

# select the destination point of the first trip
point = trip_gdf.iloc[0]['dest']
point_buffer = point.buffer(buffer_dist)

fig, ax = ox.plot_graph(G, show=False, close=False)
ax.plot(point.x, point.y, 'x', color='r')
ax.plot(*point_buffer.exterior.xy, color='g')
plt.show()

Therefore, for each trip, we need to truncate the street network to the extent of the 100 m buffer around its destination location. To that end, we might use the truncate_graph_polygon funcion of OSMnx as in the code snippet below:

def get_buffer_street_length(point, buffer_dist, G):
    geom = point.buffer(buffer_dist)
    try:
        buffer_G = ox.truncate_graph_polygon(
            G, geom, truncate_by_edge=True, retain_all=True)
        return sum([d['length'] for u, v, d in buffer_G.edges(data=True)])
    except:
        return 0

which we can then call by providing the point variable defined above, i.e., the location of the destination of the first trip, a buffer distance of 100 m, and the projected street network G as in:

street_length = get_buffer_street_length(point, 100, G)

Nevertheless, the such call takes 19.4 s in my Lenovo X1C6 laptop, which is clearly too slow considering that we need to perform such computation for each of the 8943 trips.

First step: spatial join

There are two main issues that we need to address regarding the performance of our code.

• Asymptotic complexity: for each of our K trips, we need to test which of the N nodes fall within the 100 m radius of the destination location. A brute force algorithm would require $\mathcal{O}(K \cdot N)$. In order to improve that, we can use a tree-based spatial indexing such as an R-tree. This post by Geoff Boeing uses Python and GeoPandas to provide a nice overview of its advantages. Making use of such techniques can improve the asymptotic complexity to $\mathcal{O}((K + N) \cdot log(K) + f)$, where f is the number of interesections between minimum bounding rectangles found¹. Long story short, such gain becomes key when we deal with large datasets.
• Dynamic typing and interpreted execution in Python: Python is an interpreted language which is prone to significant execution overheads because of its dynamic type-checking. In short, this means that you should avoid iterating over large arrays and instead operate with NumPy typed arrays with ufuncs, aggregates (e.g., np.mean, np.max, …), slicing, masking and the like.

In Python, both issues can be addressed by making use of the spatial join operation of GeoPandas². To that end, we need to prepare two geo-data frames, i.e., a first one where the destination points of each trip are transformed into polygon geometries representing the circular buffers with a 100 m radius, and a second one with the point geometries of each node of the street network:

# left geo-data frame
# add a buffer around the point geometries
buffer_gdf = gpd.GeoDataFrame(
    geometry=trip_gdf['dest'].apply(lambda p: p.buffer(buffer_dist)),
    index=trip_gdf.index)
buffer_gdf.crs = G.graph['crs']

# right geo-data frame
# get list of nodes from the projected graph
node_geom = [
    geometry.Point(data['x'], data['y']) for _, data in G.nodes(data=True)
]
# create a GeoDataFrame with the node ids and their point geometries
node_gdf = gpd.GeoDataFrame(
    {
        'node': pd.Series(G.nodes()),
        'geometry': node_geom
    }, crs=G.graph['crs'])

Then we can perform the spatial join as in:

sjoin_gdf = gpd.sjoin(buffer_gdf, node_gdf, op='contains')

which will test, whether each buffer geometry of buffer_gdf contains the nodes of the street network represented in node_gdf. Since the resulting sjoin_gdf will have the same index as its first argument (i.e., the id of each buffer geometry, thus the id of each trip), we can use a group-by operation as in:

for trip_index, group_gdf in sjoin_gdf.groupby(sjoin_gdf.index):
    break  # here we just want to see the first group

which will, for each buffer geometry, yield a geo-data frame of the form:

group_gdf

Note that the index and geometry column are the same since they correspond to the same buffer geometry (i.e., and in turn to the same trip), and therefore the node column shows us which nodes of node_gdf are inside such buffer geometry.

Compute the street length with group by and apply

Given the result of the spatial join, we can use the G.subgraph method of NetworkX to get the subgraph that corresponds to the set of nodes each group_gdf, i.e.:

G.subgraph(group_gdf['node'])

In our case, in order to compute the street length of each buffer, we first need to define a function that will be called for each group data frame (such as the one shown above) of the group-by operation. Consider the following snippet:

# get street length
# first define function to compute street length given a list of nodes
# from G that fall within the buffer extent
def get_street_length(nodes, G):
    buffer_G = G.subgraph(nodes)
    return sum([
        d['length'] for u, v, d in buffer_G.edges(data=True)
    ])

Then we can compute the street length of each trip (remember that the elements of sjoin_gdf are indexed by the trip ids) as in:

trip_gdf['street_length'] = sjoin_gdf['node'].groupby(
    sjoin_gdf.index).apply(get_street_length, G)

There might be trips where their respective buffer geometry does not contain any graph node, which will be represented by a nan value (after the column assignment of the code snippet above). Since this means that no nodes of the street networks are found around the trip destination, we can fill such rows of trip_gdf with a 0.

trip_gdf['street_length'] = trip_gdf['street_length'].fillna(0)

For the 8943 trips, the combination of (i) the spatial join, (ii) group-by computation of the street length and (iii) filling nan values takes a total of 2.66 s in my laptop.

We can now plot our distribution of street lengths as in:

_ = trips_df['street_length'].hist()

Going one step further: including incomplete street segments

Although the computational workflow described above has provided us with a distribution of street lenghts in satisfactory time, it has a key conceptual shortcoming. Let us illustrate it by plotting the subgraph that correspond group_gdf used above:

fig, ax = ox.plot_graph(G.subgraph(group_gdf['node']),
                        margin=2,
                        show=False,
                        close=False)
ax.plot(point.x, point.y, 'x', color='r')
ax.plot(*point_buffer.exterior.xy, color='g')
plt.show()

As we can observe, the subgraph method returns a graph with only those nodes that are inside the buffer geometry and the edges between them. This might make sense for networks whose edges represent ethereal connections (e.g., social networks), but in networks whose edges represent physical connections (e.g., streets), we might be interested in retaining the edge segments that fall within the buffer geometry. To make such a point more clear, compare the subgraph plot above with the one below:

The edges are colored in green when they link the green nodes, i.e., those that are inside the buffer geometry, like the graph returned by the subgraph method. The red nodes are nodes from the street network that are outside the buffer geometry but link to a node inside it (i.e., the red edges). The orange nodes, which are not originally in the street network, represent the intersection between the red edges and the boundaries of the buffer geometry. Finally, the orange edges (which overlay part of the red edges) represent the street segments that lie inside the buffer zones but lead to a node that is outside of it.

It seems clear that such orange segments must be considered in order to obtain a fair impression of the characteristics of the street network inside each buffer zone. It seems rather hasty to assume beforehand that ommiting them has no significant effect over the resulting distribution of street lenghts or that such residuals will be normally distributed.

To obtain the orange segments, we can use the G.edges method, which takes as argument a list of nodes in G, and returns those edges that involve at least one of such nodes. We will then distinguish between two main cases: (a) edges such that both nodes are inside the buffer geometry (i.e., the green edges above) and (b) edges a node inside the buffer geometry and the other outisde it (i.e., the red edges above). In the latter case, we will use Shapely to get the interception between such edges and the boundaries of the buffer geometry (i.e., the orange nodes), and then add a synthetic edge between the inner node and the interception (i.e., the orange edges). Let us wrap the foregoing concepts into a function that can be called at each iteration of a group by loop as in:

def get_graph_in_geom(geom, nodes, G):
    # prepare the new graph
    # the `MultiGraph` type and the 'crs' and 'name' attributes are needed
    # to plot with osmnx
    G_in_geom = nx.MultiGraph()
    for attr_key in ['crs', 'name']:
        G_in_geom.graph[attr_key] = G.graph[attr_key]
    G_in_geom.add_nodes_from(G.subgraph(nodes).nodes(data=True))

    # iterate over all the edges that involve at least one node of `nodes`
    for u, v, d in G.edges(nodes, data=True):
        node_u = G.nodes[u]
        node_v = G.nodes[v]
        p_u = geometry.Point(node_u['x'], node_u['y'])
        p_v = geometry.Point(node_v['x'], node_v['y'])
        if p_u.within(geom):
            if p_v.within(geom):
                # both nodes are inside `geom`
                G_in_geom.add_edge(u, v, length=d['length'])
                continue
            else:
                # `u` is inside `geom` but `v` is outside
                node_in_id, node_out_id = node_u['osmid'], node_v['osmid']
                p_in, p_out = p_u, p_v
        else:
            # since the argument `nodes` (that we pass to `subgraph`
            # above) comes from the spatial join we know that at least one
            # node is inside `geom`
            # `v` is inside `geom` but `u` is outside
            node_in_id, node_out_id = node_v['osmid'], node_u['osmid']
            p_in, p_out = p_v, p_u

        # if we get here, it is because we have `p_in` inside `geom` and
        # `p_out` outside, otherwise (i.e., both nodes inside `geom`) the
        # `continue` statement would have brought us directly to the next
        # iteration
        intercept = geom.boundary.intersection(
            geometry.LineString([p_in, p_out]))

        # compute the distance between the node inside `geom` and the
        # interception between the edge and `geom`
        dist = p_in.distance(intercept)
        # add the node for the interception with the same id as the node
        # that is outside of `geom`
        G_in_geom.add_node(node_out_id, x=intercept.x, y=intercept.y)
        # add an edge from the node inside `geom` to the interception
        G_in_geom.add_edge(node_in_id, node_out_id, length=dist)

    return G_in_geom

Then we can finish our spatial join as in (note that using tqdm to display a progress bar is optional):

G_list = []
for trip_index, group_gdf in notebook.tqdm(
        sjoin_gdf.groupby(sjoin_gdf.index)):
    buffer_geom = group_gdf.iloc[0]['geometry']
    nodes = group_gdf['node']

    G_in_geom = get_graph_in_geom(buffer_geom, nodes, G)
    G_list.append(G_in_geom)

which will return a list of the graphs G_list where each position matches the index of trip_gdf features a NetworkX graph instance representing the street network inside the 100 m buffer around the destination of each trip. The execution of the above snippet takes 26.2 s in my laptop.

We can now compute the street length of each graph very easily with a function such as:

def compute_street_length(G):
    return sum([d['length'] for u, v, d in G.edges(data=True)])

which can be used in a loop as in:

street_lengths = []
for G_in_geom in notebook.tqdm(G_list):
    street_lengths.append(compute_street_length(G_in_geom))

and takes 395 ms in my laptop. We can now plot the difference between the two approaches:

figwidth, figheight = plt.rcParams['figure.figsize']
fig, (ax_l, ax_r) = plt.subplots(1, 2, figsize=(2 * figwidth, figheight))

# left plot
alpha = 0.7  # to overlay histograms
trip_gdf['street_length'].hist(alpha=alpha, label='inner only', ax=ax_l)
street_length_ser.hist(alpha=alpha, label='intercept', ax=ax_l)
ax_l.legend()
ax_l.set_title('Street lengths')

# right plot
(trip_gdf['street_length'] - street_length_ser).hist(ax=ax_r)
ax_r.set_title('Differences')

As noted in the left plot, the distribution of street lengths when considering the inner edges only and when including the street segments are quite different. From the plot in the right, we can see that such differences follow a normal distribution centered at -355.92 m.

On further performance improvements

The 26 s required to include the incomplete street segments in our spatial join is significantly slower than the 2.66 seconds required when ommiting them. Nonetheless, there is still room for many performance improvements. Firstly, the group by for loop where we call get_graph_in_geom is embarrassingly parallel, and therefore can easily be performed at scale with Dask. Secondly, NetworkX is a rather slow library to model networks when compared to other Python alternatives such as graph-tool or igraph. Thirdly, the geometric operations that we perform with shapely (i.e., interception of the red edges and the boundaries of the buffer geometries) are relatively simple and could be implemented with NumPy to avoid overheads. Finally, there exists a set of libraries that allow us to compile our Python code into faster machine languages and significantly improve the execution speeds e.g., Cython, Numba, Pythran…

I might at some point amend this blog post with the implementation of some of the ideas listed above. In the meantime, the proposed approach allows us to comfortably operate on networks of some thousand nodes in a laptop.

Notes

The aim of the urban footprinter package is to provide a repeatable convolution-based method to extract the extent of the urban regions.

The study area for this example is a rasterized extract of the official cadastral survey of the Canton of Zurich, Switzerland.

import matplotlib.pyplot as plt
import numpy as np
import rasterio as rio
from matplotlib import cm, colors
from scipy import ndimage as ndi

import urban_footprinter as ufp


raster_filepath = 'data/zurich.tif'

with rio.open(raster_filepath) as src:
    lulc_arr = src.read(1)
    res = src.res[0]  # only square pixels are supported

    # custom colormap to plot the raster
    # https://matplotlib.org/3.1.1/tutorials/colors/colormap-manipulation.html
    num_classes = len(np.unique(lulc_arr))
    terrain_r_cmap = cm.get_cmap('terrain_r', num_classes)
    color_values = terrain_r_cmap(np.linspace(0, 1, num_classes))
    blue = np.array([0, .8, 1, 1])
    color_values[13, :] = blue  # water
    color_values[14, :] = blue  # water
    cmap = colors.ListedColormap(color_values)

    # np.where is used because nodata values of 255 would distort the colormap
    plt.imshow(np.where(lulc_arr != 255, lulc_arr, -1), cmap=cmap)
    plt.colorbar()

According to the land cover classification of the dataset, the codes 0 to 7 correspond to urban land cover classes. Therefore, the array of urban/non-urban pixels looks as follows:

urban_lulc_classes = range(8)
urban_lulc_arr = np.isin(lulc_arr, urban_lulc_classes).astype(np.uint32)

plt.imshow(urban_lulc_arr)

How the urban footprinter works

The approach of the urban footprinter library is built upon the methods used in the “Atlas of Urban Expansion” by Angel et al. [3]. The main idea is that a pixel is considered part of the urban extent depending on the proportion of built-up pixels that surround it. Accordingly, the extraction of the urban extent can be customized by means of the following three parameters:

• kernel_radius: radius (in meters) of the circular kernel used in the convolution
• urban_threshold: proportion of neighboring (within the kernel) urban pixels after which a given pixel is considered urban
• buffer_dist (optional): buffer distance (in meters) to add around the computed mask

Convolution with a circular kernel

The main task consists in computing, for each pixel of the raster, the proportion of surrounding build-up pixels. This is done by means of a convolution with a circular kernel. In this example, the kernel_radius parameter will be set to 500 meters.

kernel_radius = 500

kernel_pixel_radius = int(kernel_radius // res)
kernel_pixel_len = 2 * kernel_pixel_radius + 1

y, x = np.ogrid[-kernel_pixel_radius:kernel_pixel_len -
                kernel_pixel_radius, -kernel_pixel_radius:
                kernel_pixel_len - kernel_pixel_radius]
mask = x * x + y * y <= kernel_pixel_radius * kernel_pixel_radius

kernel = np.zeros((kernel_pixel_len, kernel_pixel_len), dtype=np.uint32)
kernel[mask] = 1

The kernel will be an array of ones (the circle) and zeros which looks as follows:

plt.imshow(kernel)
plt.colorbar()

The convolution is computed by means of the convolve function of SciPy.

conv_result = ndi.convolve(urban_lulc_arr, kernel)

Since both urban_lulc_arr and kernel are arrays of ones and zeros, the result corresponds to the number of built-up pixels that lie within a 500m distance.

plt.imshow(conv_result)
plt.colorbar()

Filtering the convolution results

The result of the convolution is then used to classify pixels that are part of the urban extent. To this end, the urban_threshold parameter sets the proportion of surrounding built-up pixels after which a pixel is to be considered part of the urban extent. In this example, it will be set to 0.25 (i.e., 25%).

urban_threshold = 0.25

urban_mask = conv_result >= urban_threshold * np.sum(kernel)

The result looks as follows:

plt.imshow(urban_mask)

Optional: extract only the largest urban cluster

The fact that raster images are rectangular and real-world cities are not often entails that the raster includes more than just one urban settlement. In such case, the user might be interested in the extent that corresponds to the main urban settlement, namely, the largest urban patch of the raster. The largest urban cluster of the resulting urban_mask can be obtained by means of the label function of SciPy, which performs a connected-component labeling of the input array (i.e., urban_mask).

kernel_moore = ndi.generate_binary_structure(2, 2)
label_arr = ndi.label(urban_mask, kernel_moore)[0]
cluster_label = np.argmax(np.unique(label_arr, return_counts=True)[1][1:]) + 1
urban_mask = (label_arr == cluster_label).astype(np.uint8)

which produces the following output:

plt.imshow(urban_mask)

Optional: adding a buffer around the mask

In some contexts, e.g., the study of the urban-rural gradient in landscape ecology, it might be of interest to add a buffer around the urban extent. In this example, a buffer of 1000 meters will be added to the extracted urban extent. To this end, the binary_dilation function of Scipy can be used as follows:

buffer_dist = 1000

iterations = int(buffer_dist // res)
kernel_moore = ndi.generate_binary_structure(2, 2)
urban_mask = ndi.binary_dilation(
    urban_mask, kernel_moore, iterations=iterations)

which produces the following output:

plt.imshow(urban_mask)

The urban footprinter API

In the urban footprinter package, the procedure described above is encapsulated into a single function named urban_footprint_mask, which accepts the following arguments:

help(ufp.urban_footprint_mask)

Help on function urban_footprint_mask in module urban_footprinter:

urban_footprint_mask(raster, kernel_radius, urban_threshold, urban_classes=None, largest_patch_only=True, buffer_dist=None, res=None)
    Computes a boolean mask of the urban footprint of a given raster.

    Parameters
    ----------
    raster : ndarray or str, file object or pathlib.Path object
        Land use/land cover (LULC) raster. If passing a ndarray (instead of the
        path to a geotiff), the resolution (in meters) must be passed to the
        `res` keyword argument.
    kernel_radius : numeric
        The radius (in meters) of the circular kernel used in the convolution.
    urban_threshold : float from 0 to 1
        Proportion of neighboring (within the kernel) urban pixels after which
        a given pixel is considered urban.
    urban_classes : int or list-like, optional
        Code or codes of the LULC classes that must be considered urban. Not
        needed if `raster` is already a boolean array of urban/non-urban LULC
        classes.
    largest_patch_only : boolean, default True
        Whether the returned urban/non-urban mask should feature only the
        largest urban patch.
    buffer_dist : numeric, optional
        Distance to be buffered around the urban/non-urban mask. If no value is
        provided, no buffer is applied.
    res : numeric, optional
        Resolution of the `raster` (assumes square pixels). Ignored if `raster`
        is a path to a geotiff.

    Returns
    -------
    urban_mask : ndarray

kernel_radius = 500
urban_threshold = 0.25
urban_classes = list(range(8))

Therefore, the urban extent extracted above can be obtained as in:

urban_mask = ufp.urban_footprint_mask(raster_filepath,
                                      kernel_radius,
                                      urban_threshold,
                                      urban_classes=urban_classes)

plt.imshow(urban_mask)

Alternatively, the urban extent can also be obtained as a vector geometry (instead of a raster array) by means of the urban_footprint_mask_shp function:

urban_mask = ufp.urban_footprint_mask_shp(raster_filepath,
                                          kernel_radius,
                                          urban_threshold,
                                          urban_classes=urban_classes)

urban_mask

References

1. Oliveira, E. A., Andrade Jr, J. S., & Makse, H. A. (2014). Large cities are less green. Scientific reports, 4, 4235.
2. Rozenfeld, H. D., Rybski, D., Andrade, J. S., Batty, M., Stanley, H. E., & Makse, H. A. (2008). Laws of population growth. Proceedings of the National Academy of Sciences, 105(48), 18702-18707.
3. Angel, S., Blei, A. M., Civco, D. L., & Parent, J. (2016). Atlas of urban expansion - The 2016 edition, Volume 1: Areas and Densities. New York: New York University, Nairobi: UN-Habitat, and Cambridge, MA: Lincoln Institute of Land Policy.

The most troublesome of such practices was perhaps how I managed the data transformations required by my analysis workflow. For instance, after applying many complex transformation to a large pandas DataFrame, I would often conclude a notebook with a cell such as:

# dump the result
df.to_csv('df_after_complex_transformation.csv')

so that the following notebook could immediately pick up from through an initial cell as in:

# resume from where last notebook left off
df = pd.read_csv('df_after_complex_transformation.csv')

While many of such issues can be settled by choosing helpful file names, good organization, documentation and source control of the code, I felt that there was something missing in my coding toolbox. More precisely, I felt that for data-centric tasks, source control of the code was not enough. Imagine that I found out that I made a mistake in my complex transformation of the DataFrame above and fixed the code accordingly. Since data files like df_after_complex_transformation.csv are kept outside source control, when reopening the notebook after some days, perhaps even in another computer, it might be hard to know whether such interim data files are the ones that result from the older code with mistakes or, on the contrary, they are up to date with the fix.

It was not until some months ago, tired of sticking to toxic habits, that I started digging deep into blog posts and GitHub repositories in order to find a solution to this. The best solution came in the form of “A logical, reasonably standardized, but flexible project structure for doing and sharing data science work”, namely Cookiecutter Data Science.

As the reader may already know, there exist several tools devoted to manage such kind of computational workflows, and in fact, my initial idea was to write a post in which the appropriateness of such existing tools to geospatial data science was discussed. However, with the description of the example analysis case and the thorough explanation on how to use GNU Make to manage the corresponding computational workflow, the post has already become long enough. So a comparison with similar tools to manage geospatial computational workflows will be left for future posts. Now let us get to an example analysis case.

An example from geospatial data science: analysis of the spatiotemporal patterns of urbanization

I am currently working on a paper in which I intend to evaluate the spatiotemporal patterns of land cover change associated to urbanization in the main Swiss urban agglomerations. In order to do so, for each of the urban agglomerations I need to generate a time series of raster representations of the land cover. Two datasets are used for this purpose:

• The Corine Land Cover (CLC) datasets for the years 2000, 2006 and 2012. Each dataset consists of a single .tif raster file for the whole European extent that features its land cover, categorized into 44 land cover classes.
• The Swiss Generalized Municipal Boundaries (GMB), which consist of a .shp vector shape file featuring the administrative boundaries of the municipalities that configure the main Swiss urban agglomerations. The boundaries for a given agglomeration can be extracted by filtering the boundary shapes by their attribute values in the ANAME column.

The preprocessing of the data is as follows. For a given agglomeration, at a given year, we first clip the CLC raster of the corresponding year to the extent of the vector boundaries of the agglomeration in question. Since we only interested in the land cover change associated to urbanization, we will then reclassify the pixels of the clipped raster from the 44 land cover classes of CLC to a simplified urban/non-urban representation. The task is illustrated in the animation below.

We will then compute two landscape metrics: the proportion of landscape occupied by urban land cover, and the area-weighted mean fractal dimension of urban patches, which reflects the complexity of their shape. Finally, we will assemble the computed values for each agglomeration and year in order to generate the following figure, which will help to assess whether the land use change associated to urbanization is leading towards a landscape with more complex shapes.

Enter the computational workflow approach

Let us apply the Cookiecutter Data Science project structure and its principles to our example case. A repository following such structure, with all the materials needed to reproduce our analysis is available at GitHub. Following the analysis as a Directed Acyclic Graph (DAG) approach, a simplified version of the workflow for the task above can be represented as follows:

graph LR
DOWNGMB(Download GMB) --> GMB[GMB shp]
DOWNCLC(Download CLC) --> CLC[CLC tif]
GMB -.-> MAKEEXTRACT(Make extract)
CLC -.-> MAKEEXTRACT
subgraph for each agglomeration and year
MAKEEXTRACT --> URBANEXTRACT[Urban extract tif]
end
URBANEXTRACT -.-> MAKEFIGURE(Make figure)
MAKEFIGURE --> FIGURE[Figure png]
classDef task fill:$TASK_FILL,stroke:$TASK_STROKE,color:$NODE_TEXT;
classDef data fill:$DATA_FILL,stroke:$DATA_STROKE,color:$NODE_TEXT;
class DOWNGMB,DOWNCLC,MAKEEXTRACT,MAKEFIGURE task;
class GMB,CLC,URBANEXTRACT,FIGURE data;

The nodes with rounded corners represent tasks, whereas the ones with sharp corners represent data states (for instance, interim files). The dashed links towards the highlighted part denote that there should not be a single make extract and urban extract pair of nodes but a pair for each urban agglomeration and for each year. Since we consider the 3 largest Swiss urban agglomerations (Zurich, Geneva and Basel) for the four years 2000, 2006 and 2012, we will generate a total of 9 urban extracts, which will then be all used in order to produce the final figure.

Note that the computation of the two landscape metrics is done within the make figure task. Had it been a computationally-expensive task or had the computed metrics been required for any further step in our analysis, it might have been reasonable to add a dedicated step in the DAG, e.g., a compute metrics task between urban extract and make figure, perhaps dumping the computed metrics to an interim csv file.

Step 1: Download the raw datasets

The data immutability principle basically states that data should never be edited manually, and that anyone should be able to fully reproduce our analysis using only the raw datasets and our code. Accordingly, the download steps download the raw datasets to a data/raw directory of our repository (whose files will be ignored by git). While some datasets can be downloaded by anyone from open-data services like Zenodo, other datasets like CLC require a registration. In the latter case, I store the raw datasets in a personal S3 bucket, and then create a Python script that downloads the files from the bucket using the boto3 library. Overall, such approach to the download steps ensure that the raw datasets will be the same regardless on whether I am running the analysis in my laptop or my office’s desktop computer.

Step 2: Make the urban extracts

The make extract step perform the preprocessing task described above, taking datasets from data/raw as inputs and outputting the urban extracts raster files to data/processed.

Step 3: Make the figure

Finally, the make figure will first use the PyLandStats library in order to compute the landscape metrics for the urban extract located at data/processed, plot their evolution for each agglomeration, and then save the figure to reports/figures.

Automating the workflow with GNU Make

As noted above, we intend to automate the way we move the steps of our workflow, so that we do not need to ask whether we have applied or not certain data transformation nor whether such transformation has been performed according to the latest version of our code. Ideally, we want manage the execution of such steps explicitly (without hidden state) and in an automated manner, avoiding tricks such as caching an interim df_after_complex_transformation.csv file To that end, we will use GNU Make.

Many people (like myself) always thought of Make as a tool to compile executables from source code, especially in C. Their syntax is quite simple. We create a file named Makefile at the root of our repository, and write a set of rules there.

target: prerequisites
	commands

The way in which we can chain rules can be used to mimic our analysis DAG. A very straightforward way to do so would be with a Makefile such as:

download_gmb:
	python src/data/download_gmb.py

download_clc:
	python src/data/download_clc.py

extracts: download_gmb download_clc
	python src/data/make_urban_extracts.py

figure: extracts
	python src/visualization/make_figure.py

Then, from the repository’s root, we can run make figure, and Make will detect that the figure target depends on the extracts target, and that the latter depends on both download_gmb and download_clc, and therefore Make will automatically determine the right order in which the tasks must be executed in order to produce our figure. This is the idea beyond the concept of an “automated management of the computational workflow”. Nevertheless, as elegant as the above Makefile might look, and despite the fact that it would succeed in producing our figure, we would be making a poor use of the features that Make has to offer.

Detecting completed targets

Imagine that our src/visualization/make_figure.py has a mistake which is (for example) labeling our figure’s axes incorrectly. If we fix such mistake, and run make figure again, Make will blindly execute again all of the necessary steps, including the download of 551.9 MB of unchanged raw datasets, and the generation of the exact same 9 urban extracts. Since our datasets could be of gigabytes, and our steps could be computationally expensive algorithms, we want our management of the computational workflow to automatically detect whether a task must be re-executed or not.

This idea is in fact central to Make. In order to exploit it, we just need that targets match file paths. For instance, we could refactor our download scripts to take the destination file path as command-line argument and download the files to such location. Under these circumstances, we could use such destination file path as target of the two download rules and prerequisite of the extracts rule, as in:

data/raw/gmb/gd-b-00.03-875-gg18.zip:
	python src/data/download_gmb.py data/raw/gmb/gd-b-00.03-875-gg18.zip

and we could further DRY our rule by storing such path in a variable of our Makefile as in:

GMB_ZIP_FILEPATH = data/raw/gmb/gd-b-00.03-875-gg18.zip

$(GMB_ZIP_FILEPATH):
	python src/data/download_gmb.py $(GMB_ZIP_FILEPATH)

Note, however, that if we now want download the GMB dataset, the elegant command make download_gmb becomes the less appealing make data/raw/gmb/gd-b-00.03-875-gg18.zip. We might overcome this drawback by writing an additional dummy rule such as:

. . .

download_gmb: $(GMB_ZIP_FILEPATH)

so that now, whenever we call make download_gmb, Make will automatically detect whether the file path data/raw/gmb/gd-b-00.03-875-gg18.zip exists and accordingly decide whether the file should be downloaded (that is, running src/data/download_gmb.py) or not.

Two comments on Phony targets

In Make, targets which do not correspond to any file path, such as download_gmb, are called Phony targets. A typical example would be a rule which does not produce any output, for instance, a rule to delete compiled Python files. Two remarks are especially relevant in this situation:

• When a rule depends on a Phony target, Make will always execute the rule associated with the Phony target. This is in sharp contraposition to file path targets, which Make will ignore if the corresponding file is up to date. That means that dummy rules such as the proposed download_gmb cannot be used as rule prerequisites. That is, using

  # again, let us ignore `download_clc` for now
  extracts: download_gmb download_clc
  	python src/data/make_extracts.py
  

  instead of extracts: $(GMB_ZIP_FILEPATH) download_clc, implies that Make will no longer be able to detect whether downloading the GMB dataset is necessary or not. The bottom line is: you might add dummy rules with Phony targets to have a more elegant command-line interface (CLI) with your Makefile, but do not use such dummy rules as prerequisites of other rules unless you really want them to be executed regardless on the state of your workflow.

• Was there a file matching the path download_gmb, Make would assume that the target refers to such file. In order to avoid such conflicts, Phony targets can be explicitly declared as prerequisites of a rule with the special target .PHONY as in:

  .PHONY: download_gmb
  
  . . .
  
  download_gmb: $(GMB_ZIP_FILEPATH)
  

File modification times and order-only prerequisites

Some rules might require creating directories in our tree. For instance, the data/raw/gmb directory in which gd-b-00.03-875-gg18.zip is to be downloaded might not exist. There are many ways of dealing with such situation, for instance, within the Python script. Nonetheless, handling this explicitly within the Makefile might be preferable, in order to avoid potential sources of hidden state. We might thus add an additional rule that creates the data/raw/gmb directory only when such path does not exist (otherwise it would throw a File exists error). We can do so as follows:

GMB_DIR = data/raw/gmb
# Since `$(GMB_DIR)` is constant, we use `:=` so that Make expands the path
# only once instead of doing so every time `GMB_ZIP_FILEPATH` is used.
# See https://www.gnu.org/software/make/manual/make.html#Flavors
GMB_ZIP_FILEPATH := $(GMB_DIR)/gd-b-00.03-875-gg18.zip

$(GMB_DIR):
	mkdir $(GMB_DIR)

$(GMB_ZIP_FILEPATH): $(GMB_DIR)
	python src/data/download_gmb.py $(GMB_ZIP_FILEPATH)

Let us go further. Since our tasks will require the use of the GMB shapefiles rather than of the GMB zip, we might use the unzip command in order to extract them. We can even remove the GMB zip once the shapefiles are extracted. This could all be done as in:

GMB_DIR = data/raw/gmb
GMB_ZIP_FILEPATH := $(GMB_DIR)/gd-b-00.03-875-gg18.zip
GMB_SHP_FILEPATH := $(GMB_DIR)/g1a18.shp

$(GMB_DIR):
	mkdir $(GMB_DIR)

$(GMB_SHP_FILEPATH): $(GMB_DIR)
	python src/data/download_gmb.py $(GMB_ZIP_FILEPATH)
	unzip $(GMB_ZIP_FILEPATH) -d $(GMB_DIR)
	rm $(GMB_ZIP_FILEPATH)

A peculiar issue can arise with the above Makefile. When using unzip command, the modification times of the extracted files (such as $(GMB_SHP_FILEPATH)) will be those of their creation, whereas for $(GMB_DIR), the modification time will correspond to the moment in which the files were extracted under such directory. Accordingly, in the eyes of Make, the target $(GMB_SHP_FILEPATH) will appear older than its prerequisite $(GMB_DIR), and therefore the rule will always be executed.

There are two ways to work around such issue. On the one hand, we could use the touch command, which will set the modification date of $(GMB_SHP_FILEPATH)$ to the current date², as in:

. . .

$(GMB_SHP_FILEPATH): $(GMB_DIR)
	python src/data/download_gmb.py $(GMB_ZIP_FILEPATH)
	unzip $(GMB_ZIP_FILEPATH) -d $(GMB_DIR)
	touch $(GMB_SHP_FILEPATH)
	rm $(GMB_ZIP_FILEPATH)

Alternatively, we could also tell Make that $(GMB_DIR) is an order-only prerequisites, which ignore the modification dates and are executed only if the files or directories in question do not exist. Order-only prerequisites must be placed to the right of a pipe symbol |, which in our example would be:

. . .

# a target might have both normal and order-only prerequisites, e.g.,
# $(GMB_SHP_FILEPATH): some-normal-prerequisite-target | $(GMB_DIR)
$(GMB_SHP_FILEPATH): | $(GMB_DIR)
	python src/data/download_gmb.py $(GMB_ZIP_FILEPATH)
	unzip -j $(GMB_ZIP_FILEPATH) -d $(GMB_DIR)
	rm $(GMB_ZIP_FILEPATH)

Again, note that we could use Python’s zipfile library and deal with the extraction of the shapefiles within the script src/data/download_gmb.py, nevertheless this will further complicate the non-trivial task of ensuring that Make manages our computational workflow correctly.

Pattern rules and abstracting our workflow from the data

Let us now consider the task of downloading the CLC datasets. Like GMB, CLC datasets come as a zip file, from which we intend to extract a geotiff file (instead of shapefiles in GMB). The major difference with respect to GMB is that we now need to download three datasets, namely one for each year. Accordingly, src/data/download_clc.py needs to know which CLC dataset to download in every case. Since CLC names its datasets as in g100_clc00_V18_5, g100_clc06_V18_5 and g100_clc12_V18_5 (note the year codes 00, 06 and 12 right after the clc characters), let src/data/download_clc.py take such basename as first argument (to know which dataset to download), and as usual, the destination file path as second argument. Consider the following snippet:

# Note that we should first handle the creation of the `data/raw/clc`
# directory like explained above for `$(GMB_DIR)`
CLC_DIR = data/raw/clc

$(CLC_DIR):
	mkdir $(CLC_DIR)

# and now we can start with the CLC dataset for the year code 00 (2000)
CLC_00_BASENAME = g100_clc00_V18_5
CLC_00_ZIP_FILEPATH := data/raw/clc/$(CLC_00_BASENAME).zip

$(CLC_00_ZIP_FILEPATH): | $(CLC_DIR)
	python src/data/download_clc.py $(CLC_00_BASENAME) \
	    $(CLC_00_ZIP_FILEPATH)

Since writing the foregoing rule for each dataset year would clearly be over-repetitive and become hard to maintain, Make features pattern rules, which are like ordinary rules but use a pattern-matching mechanism on file paths. We can tell make that the commands to any zip target within $(CLC_DIR) are as follows:

. . .

$(CLC_DIR)/%.zip: | $(CLC_DIR)
	python src/data/download_clc.py $(notdir $@) $@

where the % is used to compose the target pattern. We could also tell Make, for example, to only match zip files starting by g100_clc as in $(CLC_DIR)/g100_clc%.zip. On the other hand, the full expanded path of the target (in the above example, that is data/raw/clc/g100_clc00_V18_5.zip) will be stored under the $@ automatic variable, which we can use within the rule commands. Given how we have assumed src/data/download_clc.py to work, we can use Make’s functions for file names notdir to extract the file name from the path (that is, g100_clc00_V18_5.zip from data/raw/clc/g100_clc00_V18_5.zip), and then basename to trim out the .zip extension.

In order to define an abstract computational workflow that is independent of the individual files³, we can define the datasets to be processed in a separate variable, as in the snippet below:

. . .

CLC_BASENAMES = g100_clc00_V18_5 g100_clc06_V18_5 g100_clc12_V18_5

CLC_ZIP_FILEPATHS := $(addsuffix .zip, \
    $(addprefix $(CLC_DIR)/, $(CLC_BASENAMES)))

$(CLC_DIR)/%.zip: | $(CLC_DIR)
	python src/data/download_clc.py $(notdir $@) $@

download_clc: $(CLC_ZIP_FILEPATHS)

Were we to consider an additional dataset in our analysis, only by adding it to CLC_BASENAMES, Make would be able to manage and execute our computational workflow accordingly. By applying the file name functions addprefix and addsuffix as shown above, CLC_ZIP_FILEPATHS will be a list of file paths (such as data/raw/clc/g100_clc00_V18_5.zip) for each base name in CLC_BASENAMES. Whenever we execute the Phony target download_clc, for each file path of CLC_ZIP_FILEPATHS Make will detect that it matches the $(CLC_DIR)/%.zip pattern and proceed accordingly. Furthermore, since the next steps within the computational workflow do not require the zip files but the shapefiles and geotiff files that contain, we could chain the pattern rules as in:

. . .

CLC_BASENAMES = g100_clc00_V18_5 g100_clc06_V18_5 g100_clc12_V18_5

CLC_TIF_FILEPATHS := $(addsuffix .tif, \
    $(addprefix $(CLC_DIR)/, $(CLC_TIF_FILENAMES)))

$(CLC_DIR)/%.zip: | $(CLC_DIR)
	python src/data/download_clc.py $(notdir $@) $@

$(CLC_DIR)/%.tif: $(CLC_DIR)/%.zip
	unzip $< '$(basename $(notdir $@)).*' -d $(CLC_DIR)
	touch $@

download_clc: $(CLC_TIF_FILEPATHS)

so that when executing make clc, Make first matches the $(CLC_DIR)/%.tif, and determines that such geotiff target requires its corresponding zip, which will in turn be matched by the above rule with the $(CLC_DIR)/%.zip target. Note that the $< right after unzip is another automatic variable of Make, which corresponds to the first prerequisite of the rule in question (in this case, the path to zip file). Under these circumstances, if there is no further reference to $(CLC_DIR)/%.zip, Make will consider the zip files as intermediate and will therefore remove them once the $(CLC_DIR)/%.tif targets have been produced. The same will not happen after extracting $(GMB_SHP_FILEPATH) from $(GMB_ZIP_FILEPATH) since the path to the zip file is stored in the latter variable and in such case Make does not remove it. We could achieve the same behavior by defining a pattern rule of the same form for GMB, even when we know that there is only a single file involved.

Metaprogramming: generating rules dynamically

Now comes the most convoluted part of our Makefile. In order to generate the urban extracts for each year and agglomeration, the script src/data/make_urban_extract.py needs the path to the GMB shapefile, a reference to the corresponding agglomeration (for instance, a slug as in basel) so that its boundaries can be extracted from the GMB shapefile (as detailed above, by filtering the values of the ANAME attribute column of the shapes), the CLC geotiff of the corresponding year, and a file path to dump the generated urban extract. We could make use of what we have seen so far and write a rule such as:

. . .

URBAN_EXTRACTS_DIR = data/processed/urban_extracts

$(URBAN_EXTRACTS_DIR):
	mkdir $(URBAN_EXTRACTS_DIR)
$(URBAN_EXTRACTS_DIR)/basel-%.tif: $(CLC_DIR)/%.tif $(GMB_SHP_FILEPATH) \
    | $(URBAN_EXTRACTS_DIR)
	$(PYTHON_INTERPRETER) src/data/make_urban_extract.py \
	    $(GMB_SHP_FILEPATH) basel $< $@

and so on for each agglomeration, but again, this would be hard to maintain as we include additional urban agglomerations. We will instead adopt a metaprogramming approach which consists in defining a variable with the list of the agglomeration slugs, and then dynamically generate a rule for each agglomeration as follows:

. . .

AGGLOMERATION_SLUGS = basel geneve zurich

define MAKE_URBAN_EXTRACT
$(URBAN_EXTRACTS_DIR)/$(AGGLOMERATION_SLUG)-%.tif: \
    $(CLC_DIR)/%.tif $(GMB_SHP_FILEPATH)| $(URBAN_EXTRACTS_DIR)
	python src/data/make_urban_extract.py $(GMB_SHP_FILEPATH) \
	    $(AGGLOMERATION_SLUG) $$< $$@
endef

$(foreach AGGLOMERATION_SLUG, $(AGGLOMERATION_SLUGS), \
    $(eval $(MAKE_URBAN_EXTRACT)))

The MAKE_URBAN_EXTRACT works as a template which GNU Make’s eval function will use to dynamically generate rules. Note automatic variables within the template have been escaped through a preceding additional $. This is because the templates that we pass as argument to eval (in this case, MAKE_URBAN_EXTRACT) are expanded before we call them, so the variables that must be expanded during the execution of the rule (for instance, any automatic variable) should be escaped⁴. The use of Make’s foreach function in order to dynamically generate a rule for each agglomeration is quite self-explanatory.

Subsequently, we will then use a nested loop to generate the full list of file paths of the form data/processed/urban_extracts/{agglomeration}-{clc_basename}.tif for each agglomeration and each CLC dataset, and create a Phony target to generate all the urban extracts:

. . .

URBAN_EXTRACTS_TIF_FILEPATHS := $(addprefix $(URBAN_EXTRACTS_DIR)/, \
    $(foreach CLC_BASENAME, $(CLC_BASENAMES), \
        $(foreach AGGLOMERATION_SLUG, $(AGGLOMERATION_SLUGS), \
            $(AGGLOMERATION_SLUG)-$(CLC_BASENAME).tif)))

urban_extracts: $(URBAN_EXTRACTS_TIF_FILEPATHS)

Given how pattern rules work, Make will be able to match each of the file paths for urban extract to the rule that we have dynamically created for the corresponding agglomeration.

Detecting edited source files

We will now proceed to the final part of the workflow, namely generating the figure and introduce another nice feature of Make while we are at it. There are many ways in which we can generate the figure from the urban extracts. We will use a script src/visualization/make_figure.py which will take as first argument the directory where the urban extracts are located, the output file path to store the figure as second argument, and three options allowing a list of values for the desired metrics, CLC basenames (each corresponding to a particular year) and the agglomeration slugs. The Python script will then deal with the association of each file path of the urban extracts to its respective agglomeration slug and CLC year.

. . .

FIGURE_FILEPATH = reports/figures/swiss-urbanization.png
METRICS = proportion_of_landscape fractal_dimension_am

$(FIGURE_FILEPATH): $(URBAN_EXTRACTS_FILEPATHS)
	python src/visualization/make_figure.py $(URBAN_EXTRACTS_DIR) \
	    $(FIGURE_FILEPATH) --metrics $(METRICS) \
	    --clc-basenames $(CLC_BASENAMES) \
	    --agglomeration-slugs $(AGGLOMERATION_SLUGS)

figure: $(FIGURE_FILEPATH)

There is nothing tricky in the above snippet. Just note that we are using $(URBAN_EXTRACTS_FILEPATHS) as prerequisite and not urban_extracts, since as explained above, Phony targets are always executed even if the required files are up to date. Unless we changed the code, at this point running make figure will, from any starting conditions, proceed to the end of the workflow in the most efficient manner.

Let us now introduce the last feature of Make covered by this example case. Imagine that we have successfully generated our figure but we have changed the code of src/visualization/make_figure.py in order to (for instance) change the axis labels. We could remove the former figure, and execute make figure again. Nevertheless, this would still require manual interaction. Furthermore, let us assume that in the same way that we might need to update src/visualization/make_figure.py, other reasons require us to update an earlier of our workflow such as src/data/make_urban_extract.py. Under such circumstances, we would not need to manually remove a single figure but perhaps all the urban extracts. In order to let Make deal with such circumstances in an automated manner, we can set the file path to the script as a prerequisite of its corresponding rule. For instance, the latter snippet could be refactored to:

...
MAKE_FIGURE_PY = src/visualization/make_figure.py

$(FIGURE_FILEPATH): $(URBAN_EXTRACTS_FILEPATHS) $(MAKE_FIGURE_PY)
	python $(MAKE_FIGURE_PY) $(URBAN_EXTRACTS_DIR) $(FIGURE_FILEPATH) \
	    --metrics $(METRICS) \
	    --clc-basenames $(CLC_BASENAMES) \
	    --agglomeration-slugs $(AGGLOMERATION_SLUGS)

Now whenever we edit src/visualization/make_figure.py, its modification time will be more recent than that of its respective target (that is, the figure file), and therefore if we execute make figure, Make will detect that the rule needs to be executed again. The above approach frees us from the duty of manually managing such situations, and might become crucial in other analysis cases, the amount of interim and processed data files might be in the range of hundreds, thousands or basically any number.

Concluding remarks

We have seen a thorough introduction to the use of GNU Make to manage computational workflows in geospatial data science. Here are some remarks from the perspective of someone coming from a Python background.

The main advantage of Make is its flexibility, which allows it to precisely manage any kind of computational workflow, from the compilation of source files to the example above. Nonetheless, this flexibility comes at the price of a syntax which is obviously less pleasant than Python’s, especially as we try to get the most out of Make and as a consequence Makefiles become inevitably abstract.

Avoid hidden state: let Make (not Python) carry the burden of managing your workflow

While many (like myself) would feel more comfortable tackling the organization of a computational workflow in Python, it is important to delegate such burden to the Makefile. We have accordingly implemented our scripts so that they take the destination file path as a command-line argument. While the management of the input and output paths could have been handled within the Python scripts, under such circumstances Makefiles would have to make assumptions about how the script works. Avoiding this is in accordance to the idea of encapsulation of object-oriented programming as well as to a core Pythonic principle, namely “explicit is better than implicit”. The only distinction is that this time we must apply the foregoing principle to Make rather than to Python. An explicit control of the inputs an output files within Makefiles is crucial towards avoiding potential sources of hidden state.

In order to make the most out of Make, it is preferable that the Python scripts focus on the data processing tasks and are as agnostic of the project structure as possible. Overall, a sharp knowledge of the involved Python libraries combined with the flexibility of GNU Make should allow an efficient management of the computational workflow of most analysis cases.

Python modules and detecting edited source files

Another remark can be made with respect to how we have organized our scripts. We often organize Python code in a way that a .py file might perform multiple tasks, especially when we are using the Click library to create CLI, which allows a command to have a group of subcommands attached.

In our example case, we could have implemented both src/data/download_gmb.py and src/data/download_gmb.py within a single Python module src/data/download_data.py. Nevertheless, if we set such single script as prerequisites of GMB and CLC respective rules (as we have shown above for src/visualization/make_figure.py), Make will always re-execute both rules when we edit its source, even if the amendments only concern one of the rules.

It is thus recommended that each .py file serves one and only one Make rule, even if that might sometimes feel strange for Python developers.

Alternatives to Make

There are indeed alternatives to GNU Make which might be better suited for certain analysis⁵. While many of them make reference to the fields of bioinformatics and machine learning, the same does not hold for geospatial data science. Nevertheless, most of the core principles apply similarly throughout all data-centric analysis cases.

My intention is to implement the management workflow of this post within Snakemake, which seems very aligned with Make while providing a more appealing syntax. On the other hand, before discovering the Cookiecutter Data Science project structure, I used dvc in a geospatial analysis case. While it has the nice asset of employing checksums on the data files to ensure their consistency within the workflow (and different computers), and features a nice integration with git and S3 storage, its CLI does not allow for a forthwith integration with the Cookiecutter Data Science project structure. Nonetheless, there seems to be some work in progress to overcome such drawback. On the other hand, I would also like to try the innovative approach of Renku (currently under early development stages), which appears similar to dvc but more general, that is, less centered on machine learning workflows. Either way, I will try to come back with another post on this topic soon.

Regarding the research question of the example case, can we say that the landscapes of the main three Swiss agglomerations are becoming increasingly complex? Well, it is hard to say with this simplified example, but a more thorough manuscript is under preparation.

Notes

The fact that we tend to associate entropy with disorder can be largely attributed to the legacy of Ludwig Boltzmann, the father of statistical mechanics. His interpretation of entropy can be clarified through the following example. Let’s consider we roll two dices. The image below illustrates all the 36 possible resulting configurations (microstates), grouped in columns according to the sum of the dices’ values (macrostates).

According to Boltzmann, the macrostate of maximum entropy is the one with most possible microstates. In our two-dice system, such macrostate corresponds to when the dices’ values add up to seven, given that there are six possible results that can bring us to such configuration. Instead, the two lateral ends of the image, that is, the macrostates where the dices’ values add up to two or twelve, correspond to the macrostates of minimal entropy, given that there is only one possible microstate (where we roll two ones or two sixes respectively).

Boltzmann was convinced that the macroscopic behavior of gases could be explained through the configurations formed by their molecules, quantifiable by such measure of entropy. Unfortunately, in a historical context where most of the scientific community did not believe in the existence of atoms, his contributions conflicted with many of his peers. In fact, apropos of the sadness of the second principle of thermodynamics mentioned above, it is worth pointing out that the constant criticisms on his work together with other personal circumstances lead Boltzmann to commit suicide during a family vacation in Trieste in 1906, just before the atoms became the zeitgeist of science during the twentieth century.

The idea that when we repeatedly throw two dices, the most recurring result will be a a combination where the values add up to seven is consistent with the second principle of thermodynamics in that closed systems tend to its configurations of maximum entropy. Nevertheless, this is not the behavior displayed by many of the systems that we encounter in nature or the contemporary society.

An interesting example is the behavior observed in ant colonies. The image below (and in more detail in this video) highlights that there exists strong collective order behind the movement of ant colonies.

From Boltzmann’s perspective, among all possible configurations, ant colonies tend to strongly ordered patterns, with very low entropy, and thus their occurrence should be very unlikely. Yet this is what we encounter.

Unlike with dices, the interactions between ants are not random, since they pursue collective objectives of locating and storing food. However, it remains surprising that, as suggested by a long line of investigations¹, the collective order observed in ant colonies emerges from local interactions within neighboring ants, with practically no global communication.

As one might imagine, ant colonies are not the only system which, in apparent defiance of the second principle of thermodynamics, tends to generate ordered patterns. In fact, concurrent to the advances on the understanding of the behavior of ant colonies², the Nobel laureate in Chemistry Ilya Prigogine suggested that the reason why certain systems do not evolve towards states of maximum entropy is that the second principle of thermodynamics as well as Boltzmann’s probabilistic interpretation of entropy only apply to closed systems, that is, systems that do not interact with their environment.

There exists thus, a class of systems which Prigogine called ‘dissipative’, which as result of certain interactions with their environment, evolve towards ordered configurations³. It seems reasonable to consider that cities - the subject of study of my PhD thesis - are part of this class of systems. A nice illustration of this kind of behavior that is applicable to contemporary cities is Schelling’s segregation model⁴. Let’s consider a city with two kinds of inhabitants, the reds and the blues. As far as this example is concerned, the city will be represented by a grid of 40x40 cells, where 50% of them will be occupied by reds, 40% by blues and 10% empty. The central idea is very simple. Each inhabitant will consider his or her cell-location satisfactory if and only if at least four out of their eight neighbors are of the same color. In the opposite case, they will try to relocate to another cell. Starting from a random configuration, this mechanism will be applied until all agents are satisfied. The evolution of the city is illustrated below⁵:

It is easy to see that in our city, analogously to ant colonies, a global order emerges from interactions that consider only local criteria. In the terminology of complex systems, these phenomena are said to exhibit emergent properties, which have not been directly incorporated within the design of the system but they have been an indirect result of their functioning - through a process of self-organization.

While Schelling’s model might be excessively simple and restricted to pedagogical purposes, contemporary cities also exhibit clear characteristics of self-organization and emerging order. A noteworthy example is the urban sprawl that we can observe in many Western cities, with Los Angeles, Atlanta or Houston as archetypal examples. The term urban sprawl has often been associated to pejorative connotations of disorganization and lack of planning.

Nonetheless, the commuting behavior of such sprawling cities reveal that, behind their haphazard visual impression, there is strong order underpinning their spatial development: despite notable suburban expansion of the largest metropolitan areas of the United States during the 80s and 90s, the commuting times have remained stable⁶. At the same time, the number of commuting trips with both origin and destination in the suburbs increased substantially, which suggests that residences and businesses mutually co-locate⁷. Therefore, while it is fair to consider that urban sprawl misuses space and induce automobile dependency, it is by no means a disordered phenomena. It is rather the result of a complex series of interactions mediated by many factors such as residential preferences, land value, existing infrastructure and the like.

To what extent systems that display self-organization and emerging properties are considered purely mechanical or some sort of animate organism is one of the fundamental questions of the last centuries of Western science and philosophy⁸. In any case, the way in which the interactions between inhabitants can generate an emerging order without the intervention of any planning agent is a clear defiance of “planning” as a rational activity. This does not mean that any attempt to planning is futile but rather that most of the urban planning movements developed throughout the latest two centuries are not adequate for the kind of problems that contemporary cities pose. This is in fact the fundamental criticism of the journalist Jane Jacobs in her seminal book “The Death and Life of Great American Cities”, where she wrote:

Merely to think about cities and get somewhere, one of the main things to know is what kind of problem cities pose . . . .

Cities happen to be problems in organized complexity, like the life sciences. They present “situations in which a half-dozen or even several dozen quantities are all varying simultaneously and in subtly interconnected ways . . . . The variables are many but they are not helter-skelter; they are “interrelated in to an organic whole.” . . . .

The theorists of conventional modern city planning have consistently mistaken cities as problems of simplicity. . . and have tried to analyze and treat them thus⁹

Similarly, and referring to the critique that the Marxist geographer and anthropologist David Harvey addresses to the models of classic urban economics, the also geographer Juval Portugali wrote the following:

[your models are] “incapable of saying anything of depth and profoundity . . .”; not because you are bad guys or second-rate professionals, but because like Don Quixote, you tilt at windmills, and by doing so not only do you not harm the real beast - the Capitalist mode of production - but you actually participate in its reproduction; not only that your sophisticated plans do not reduce injustice in the city, but they reproduce it; create it anew again and again. Capitalism with its city of injustice is advancing and flourishing behind the ideological false consciousness of your liberal science and planning[^portugali-harvey]

Regardless of what one might think of capitalism, the emergent properties of complex systems such as cities and society in general, for instance the distribution of wealth or social mobility, are the result of the numerous interactions between the agents that constitute it, inevitably guided by certain economic, social and political system (whether it is capitalism, socialism or another). In fact, the notion of emergent properties resulting from a process of self-organization is salient on the “invisible hand” principle of Adam Smith, and liberal economists such as Friederich Hayek have actually built upon them to defend capitalism, considering the free market as the only mechanism capable of taking advantage of the dispersed knowledge present in our society.

In any case, it seems increasingly clear that conventional planning tools such as master plans are not enough to overthrow the emerging order that emanates from the actions and decisions of citizens in the context of capitalism, increasingly influenced by the globalization, volatility of the job market, technological advances and the incursion of platforms such as Airbnb and Uber[^planning-failures].

Last summer I visited the Pueblos Mancomunados de la Sierra Norte de Oaxaca, in Mexico. It is a group of eight villages that conform a cooperative and community organization since pre-colonial times, which were formally constituted as an agrarian entity in 1961[^pueblos-mancomunados]. Their economic activities consist basically of agriculture, sustainable logging, and since 1995, a communal ecotourism program.

One of the days, a local guide took us to the ‘Amanecer’ lookout to see the sunrise, and I took the opportunity to ask some questions about the organization of the communities. He explained us that any person, regardless of his or her nationality, can move there - provided that he or she agrees to follow the community rules. Each family is entitled to cut three pines in order to build their house, expansible for large families. Their local economy offers a series of jobs, however nobody is bound to take any specific job. The main duty as a citizen are the “cargos”, which consist of some basic tasks for the functioning of the communities, such as logging, organization of citizen and school boards, or jobs derived from the ecotourism program, among others. Every for years, a member of each house, aged between 18 and 65, must perform the “cargos” during a whole year as an unremunerated service to the community. On the other hand, it is worth pointing out that all the profits from the ecotourism program, which does not come cheap considering Oaxacan and Mexican prices, are entirely devoted to community projects. Altogether, not only their community system yields a fairly egalitarian distribution of wealth (which is relatively unsurprising), but also their local economy seems to enjoy good health.

At this point, it is easy to argue that under the functioning mode of the Pueblos Mancomunados, humanity would have never discovered subatomic particles, penicillin nor the field-effect transistor. I will not proceed into the evaluation of such remarks, since I do not intend to showcase the Pueblos Mancomunados as the universal model of an ideal society. The final reflection that I would like to raise is another. There is a perception, quite spread among our society, that the government positions are bound for people with high-level university education, since it is only with such background that one might be prepared for the arduous task of governing a system as complex as our society.

Nonetheless, it seems that such perception does not prevail so much for local governments, where other less-technical skills are more valued. Regarding this, I will conclude with an excerpt from Hayek, which leaving aside what one might think of his ideology, I find very pertinent to this discussion:

There is . . . a body of very important but unorganized knowledge which cannot possibly be called scientific . . . : the knowledge of the particular circumstances of time and place. It is with respect to this that practically every individual has some advantage over all others because he possesses unique information of which beneficial use might be made, but of which use can be made only if the decisions depending on it are left to him or are made with his active cooperation . . . Central planning based on statistical information by its nature cannot take direct account of these circumstances of time and place[^hayek-moroni]

I hope that I have illustrated how in complex systems such as cities and the contemporary society, interventions do not always achieve the action-reaction effect that one might expect, not even when planned by experts with high academic qualifications. Therefore, I would like to raise a question concerning whether such interventions should be preferably managed by local governments, and by people that - no matter their academic training - know the particular circumstances of time and place of each neighborhood, village or city[^oaxaca-timezone], and to whom we can direct our concerns without the need to talk to three secretaries nor booking a flight to Madrid or Brussels. Although a network of local governments taking decisions that likely overlap between one city and another might seem inefficient when compared to a sole centralized government, according to Prigogine’s theory, it is in fact such dispersed knowledge and distribution of actions that allows complex systems to adapt to the changing circumstances of the world that surrounds us.

Notes

El fet que associem el concepte d’entropia amb el desordre es deu majoritàriament al llegat de Ludwig Boltzmann, el pare de la mecànica estadística. La seva interpretació de l’entropia es pot clarificar amb l’exemple següent. Imaginem que llencem dos daus. La imatge a continuació ens il·lustra totes les 36 configuracions resultats possibles (microestats), agrupats segons la suma dels valors dels daus (macroestats).

Segons Boltzmann, el macroestat amb màxima entropia és aquell que té més microestats possibles. En el nostre sistema de dos daus, aquest macroestat correspon al cas on els valors dels daus sumen set, ja que hi ha sis resultats possibles que ens porten a aquesta configuració. En canvi, en els dos extrems laterals de la imatge, és a dir els macroestats on els valors dels daus sumen dos o dotze, corresponen als macroestats de mínima entropia, ja que només hi ha un microestat possible (on els daus marquen dos uns o dos sisos respectivament).

Boltzmann estava convençut de que les configuracions que formaven les molècules dels gasos, quantificables amb aquesta mesura d’entropia, determinaven el seu comportament macroscòpic. Malauradament, en un context històric en el qual la comunitat científico-acadèmica no creia en la existència dels àtoms, les seves contribucions discrepaven amb molts companys de professió. De fet, ja que començàvem parlant de la tristesa i el segon principi de la termodinàmica, val la pena dir que els constants qüestionaments al seu treball, conjuntament amb altres circumstàncies personals, van portar a Boltzmann a suïcidar-se durant unes vacances familiars a Trieste el 1906, tot just abans que els àtoms esdevinguessin el zeitgeist de la ciència durant el segle XX.

La idea que si llencem aleatòriament dos daus repetides vegades, la configuració que trobarem en més ocasions serà aquella on els valors dels daus sumen set és consistent amb la idea del segon principi de la termodinàmica de que els sistemes tancats tendeixen a la màxima entropia possible. No obstant, aquest no és el comportament de molts dels sistemes que podem trobar en la natura o la societat contemporània.

Un exemple interessant és el comportament de les colònies de formigues. En la imatge a continuació (i amb més detall en aquest vídeo) es pot observar com existeix un gran patró d’ordre col·lectiu en el moviment de les colònies de formigues.

Des de la perspectiva de Boltzmann, entre totes les configuracions possibles, les colònies de formigues prenen formes molt ordenades, amb una entropia molt baixa, i per tant la seva ocurrència hauria de ser molt improbable. I tot i així aquestes són les formes que s’observen en la realitat.

A diferència dels daus, les interaccions entre formigues no són aleatòries, ja que responen a objectius col·lectius de localització i emmagatzematge de recursos alimentaris. Tanmateix, el que és sorprenent és que segons sembla indicar una llarga línia d’investigacions¹, les colònies de formigues aconsegueixen generar un ordre col·lectiu a partir d’interaccions locals entre formigues veïnes, pràcticament sense cap tipus de comunicació global.

Com hom pot imaginar, les colònies de formigues no són l’únic sistema que, en aparent desafiament al segon principi de la termodinàmica, tendeixen a generar patrons ordenats. De fet, simultàniament als avenços en la comprensió de les colònies de formigues², el premi Nobel de química Ilya Prigogine va demostrar que el motiu pel qual certs sistemes no tendeixen cap a estats de màxima entropia, és que el segon principi de la termodinàmica i l’entropia de Boltzmann només apliquen a sistemes tancats, és a dir, a sistemes que no interactuen amb el medi que els envolta.

Existeix doncs, una classe de sistemes, anomenats ‘dissipatius’ segons Prigogine, que a través de certes interaccions amb el medi que els envolta, tendeixen cap a configuracions ordenades³. Crec que és fàcil argumentar que les ciutats - el subjecte d’estudi de la meva tesi doctoral - formen part d’aquest grup. Una bona il·lustració de com podem trobar aquest tipus de comportaments en les ciutats contemporànies és el model de segregació de Schelling⁴. Imaginem una ciutat amb dos tipus d’habitants, que denotarem respectivament amb els colors vermell i blau. Per aquest exemple, la ciutat serà una quadrícula de 40x40 caselles, on un 50% seran ocupades per habitants vermells, 40% per habitants blaus, i 10% buides. La idea central és molt senzilla. Els habitants estaran satisfets a la seva casella si com a mínim quatre dels seus vuit veïns són del mateix color. En el cas contrari, intentaran mudar-se a una altra casella. A partir d’una configuració inicial aleatòria, aquest mecanisme s’aplicarà fins a que tots els agents estiguin satisfets. A continuació podem veure l’evolució de la ciutat⁵.

Es pot observar clarament com la nostra ciutat, de manera anàloga a les colònies de formigues, genera un ordre global a partir d’interaccions que es regeixen per criteris exclusivament locals. En la terminologia de sistemes complexos, es diu que aquests fenòmens exhibeixen propietats emergents, que no han estat directament incorporades en el disseny del sistema però han estat un resultat indirecte del seu funcionament, a través d’un procés d’autoorganització. La simulació amb el model de preferential attachment que vaig descriure en un article anterior sobre el debat urbanístic de Sant Feliu és un altre exemple d’aquest tipus de processos.

Si bé el model de Schelling és excessivament simplista i poc aplicable fora de contextos pedagògics, en les ciutats contemporànies també podem trobar processos d’autoorganització i ordres emergents. Un exemple interessant és la dispersió urbana que podem observar en moltes ciutats occidentals, sobretot a Estats Units, com ara Los Angeles, Atlanta o Houston. En anglès, aquest tipus de dispersió s’anomena urban sprawl, i és sovint associat a connotacions negatives de desorganització i fracàs en la seva planificació urbana.

No obstant, darrere una impressió visual de desgavell, els hàbits de mobilitat en aquestes ciutats disperses evidencien que hi ha un clar ordre vigent en el seu desenvolupament espacial: tot i la considerable expansió suburbana de les majors àrees metropolitanes dels Estats Units durant els 80 i els 90, els respectius temps de viatge per anar a treballar s’han mantingut pràcticament constants⁶. Simultàniament, la quantitat de trajectes que tenen tant l’origen com la destinació en els suburbis ha incrementat de manera significativa, el qual suggereix que no només els habitants s’han establert en els suburbis, sinó que també ho han fet moltes empreses i les seves respectives oportunitats laborals⁷. Així doncs, si bé podem considerar que aquest tipus de ciutats malbaraten l’espai i són ineficients en termes de la dependència en el cotxe que generen, en cap cas es tracta d’estructures desorganitzades, ans al contrari. Són el resultat d’un procés complex on hi intervenen preferències residencials, preus immobiliaris, infraestructures i molts altres factors.

Fins a quin punt considerem que els sistemes que exhibeixen processos d’autoorganització i propietats emergents són purament mecànics - o pel contrari, constitueixen algun tipus d’entitat més complexa - és una de les qüestions fonamentals dels darrers segles de la ciència i filosofia occidentals⁸. En qualsevol cas, la manera en la que les interaccions entre els habitants poden generar un ordre emergent sense la intervenció de cap agent extern és un clar desafiament a la “planificació” com a activitat racionalista. Això no vol dir que qualsevol intent de planificació sigui completament estèril, si no que la major part dels corrents principals de l’urbanisme desenvolupats durant els dos últims segles no són adequats pel tipus de problemes que trobem en les ciutats contemporànies. Aquesta és de fet la crítica principal de la periodista Jane Jacobs en el seu influent llibre “The Death and Life of Great American Cities”, on va escriure:

Per pensar en ciutats i arribar a algun lloc, una de les principals qüestions que cal conèixer és quin tipus de problemes plantegen les ciutats . . . .

Les ciutats resulten ser problemes de complexitat organitzada, com les ciències de la vida. Presenten “situacions en les que mitja dotzena o fins i tot diverses dotzenes de quantitats varien simultàniament i de maneres subtilment interconnectades . . . . Les variables són moltes, però no són un batibull; estàn “interrelacionades en un tot orgànic” . . . .

Els teòrics de la planificació convencional de la ciutat moderna han malentès consistentment les ciutats com a problemes simples . . . i han intentat analitzar-los i tractar-los com a tals⁹

De manera similar, i en referència a la crítica que el geògraf i antropòleg marxista David Harvey adreça als models clàssics d’economia urbana, el també geògraf Juval Portugali va escriure el següent:

[els vostres models són] “incapaços de dir res amb profunditat . . .”; no perquè sigueu males persones o professionals de segona, però perquè, com Don Quijote, esteu lluitant contra els molins de vent, i fent-ho no només no danyeu la bèstia real - el mode de producció Capitalista - sinó que de fet participeu en la seva reproducció; no només els vostres plans sofisticats no redueixen la injustícia a la ciutat, sinó que la reprodueixen; la tornen a crear novament una i altra vegada. El capitalisme amb la seva ciutat d’injustícia està avançant i florint darrere la falsa consciència ideològica de la vostra ciència i planificació liberal¹⁰

Independentment del que cadascú pensi del capitalisme, les propietats emergents de sistemes complexos com les ciutats i la societat en general, com ara la distribució de la riquesa o la mobilitat social, són fruit de les interaccions entre els habitants que en formem part, inevitablement guiades per cert sistema econòmic, social i polític (tant si es tracta de capitalisme, socialisme o altres). De fet, la noció de propietats emergents resultants de processos d’autoorganització són clarament darrere el principi de la “mà invisible” d’Adam Smith, i economistes liberals com ara Friederich Hayek les han emprat justament per defensar el capitalisme, considerant el lliure mercat com l’únic mecanisme capaç d’aprofitar els coneixements dispersos presents en la societat.

En qualsevol cas, resulta cada cop més evident que els instruments convencionals de planificació, com ara els plans directors urbanístics, no són suficients per a suplantar l’ordre emergent que emana de les accions i decisions dels ciutadans en el context del capitalisme, cada cop més influenciades per la globalització, la creixent volatilitat del mercat laboral, constants desenvolupaments tecnològics i les irrupcions de plataformes com Airbnb o Uber¹¹.

Aquest estiu vaig estar als Pueblos Mancomunados de la Sierra Norte de Oaxaca, a Mèxic. Es tracta d’un conjunt de vuit pobles organitzats de manera comunitària i cooperativa des de temps pre-hispànics, i que varen ser formalment constituïts en un únic nucli agrari el 1961¹². Les seves activitats econòmiques consisteixen bàsicament en la agricultura, l’explotació forestal sostenible, i des del 1995, un programa d’ecoturisme comunitari.

Un dels dies un guia local ens va portar al Mirador ‘amanecer’ a veure sortir el sol, i vam tenir temps per fer-li unes quantes preguntes sobre el funcionament del poble. Ens va explicar que qualsevol persona, independentment de la nacionalitat, hi pot anar a viure - sempre i quant es segueixin les mateixes normes segons les quals es regeix la comunitat. Cada família té dret a tallar tres pins per a construir la seva casa, ampliable si el nombre de membres de la família així ho requereix. La seva economia local ofereix una sèrie de llocs de treball, però ningú està obligat a treballar en res en concret. La principal obligació com a ciutadà són els anomenats “cargos”, que consisteixen en tasques bàsiques pel funcionament de les comunitats, com ara la gestió forestal, organització juntes ciutadanes i escolars, o feines derivades del programa d’ecoturisme, entre d’altres. Cada quatre anys, un membre d’entre 18 i 65 anys de cada casa ha de realitzar “cargos” durant un any sencer de manera no remunerada. D’altra banda, cal esmentar que els ingressos provinents del seu programa d’ecoturisme, gens barat pel que és Oaxaca i Mèxic en general, es dediquen exclusivament a projectes comunitaris. Tot plegat, no només aquest sistema genera una distribució de la riquesa força igualitària entre els ciutadans (fet que no hauria de ser excessivament sorprenent), sinó que la seva economia local també gaudeix d’una bona salut.

Arribats aquí, és fàcil argumentar que amb l’organització dels Pueblos Mancomunados la humanitat mai hagués arribat a descobrir les partícules subatòmiques, la penicil·lina ni l’efecte transistor. No entraré a valorar aquestes qüestions, ja que la meva intenció no és posar als Pueblos Mancomunados com a model universal de com hem d’estructurar la nostra societat. La reflexió final que voldria plantejar és una altra. Hi ha una percepció, força estesa entre la nostra societat, que els càrrecs de les administracions públiques han de ser ocupats per persones d’altes formacions universitàries, ja que són aquestes les úniques preparades per la difícil tasca de governar un sistema tant complex com la societat contemporània. Sembla però, que aquesta percepció es dilueix en la política local, on es valoren més altres capacitats menys tècniques. Sobre aquesta qüestió, acabaré amb una citació de Hayek, que independentment del que es pugui pensar de la seva ideologia, trobo que és molt pertinent en aquest context:

Hi ha . . . tot un conjunt molt important de coneixement desorganitzat que en cap cas es pot anomenar científic . . . : el coneixement de les circumstàncies particulars de moment i lloc. És en aquest sentit que pràcticament cada persona té algun avantatge respecte als altres ja que posseeix informació única de la qual se’n pot fer un ús beneficiós, però tal ús només és possible quan les decisions de les quals depèn les pren la persona en qüestió o es prenen amb la seva cooperació activa . . . La planificació centralitzada basada en informació estadística, per la seva pròpia naturalesa, no pot prendre en compte directament aquestes circumstàncies de moment i lloc.¹³

Espero haver il·lustrat que en sistemes tant complexos com les ciutats i societats contemporànies, les intervencions no sempre obtenen l’efecte acció-reacció que s’espera, ni tan sols quan les planegen persones amb importants formacions acadèmiques. Així doncs, m’agradaria plantejar si no seria millor que aquestes intervencions es gestionessin des del món local, per part persones que coneixen que coneixen les circumstàncies particulars de cada barri, poble o ciutat¹⁴, i a les que podem adreçar-nos quan quelcom no funciona com previst, sense necessitat de parlar amb tres secretàries per prendre una cita ni de desplaçar-nos a Madrid o Brussel·les. Si bé una xarxa de governs locals prenent decisions que possiblement es solapen entre poble i poble pot semblar ineficient comparat amb una administració centralitzada, segons la teoria de Prigogine, és justament aquesta distribució de coneixement i accions la que permet als sistemes complexos adaptar-se a les circumstàncies canviants del món que ens envolta.

Notes

Els quatre anys que vaig dedicar a estudiar Enginyeria Industrial em van acostumar a un tipus de problemes en els quals existeix una solució òptima que es pot trobar a través de mètodes estrictament matemàtics. Malauradament, aquestes situacions no es donen a l’hora d’intentar resoldre els problemes d’una societat cada cop més complexa. Actualment, com a candidat a doctor en enginyeria civil i ambiental, em centro en problemes de gestió del territori i urbanisme on s’utilitzen mapes, previsions i simulacions realitzades a partir de models que per més que ens esforcem, no poden ser més que una representació simplificada d’una realitat massa complicada com per traduir-la en un sistema d’equacions.

És en aquesta bretxa entre els límits del coneixement humà i la complexitat de la realitat on entra en joc l’hermenèutica - l’art de la interpretació de fenòmens a través d’una representació de coneixement, sigui aquesta un text sagrat, jurídic o un mapa resultant d’una simulació. Inevitablement, en estudis de fenòmens complexos com ara les ciutats del segle XXI, l’hermenèutica juga un paper clau des del principi: quines hipòtesis es formulen, quins elements es consideren rellevants i quins s’obvien, o com se n’interpreten els resultats. Això fa que un dia llegeixi articles d’acadèmics prestigiosos que defensen densificar les ciutats per a reduir l’ús del cotxe privat, i l’endemà en llegeixi d’altres on acadèmics d’igual prestigi defensen l’expansió urbana per a reduir les congestions a les carreteres. Aquestes interpretacions estan fortament influenciades per la personalitat, el context sociocultural i la ideologia política de cadascú. I per més que no m’agradi, el que pugui presentar a la meva tesi doctoral també estarà influenciat per aquests factors.

Com hom pot imaginar, l’hermenèutica juga un paper central en la disciplina del dret: les sentències judicials depenen de com els jutges interpreten les accions dels acusats i el codi penal. Per l’auto de Junqueras esmentat anteriorment, he pogut llegir una magistrada al·legant que impulsar als independentistes a la mobilització implicava també impulsar-los a enfrontaments físics amb les forces de l’ordre, a la vegada que tal raonament és qualificat com “esperpéntico” per part d’un catedràtic de dret constitucional o com a possible causant d’una sentència del Tribunal Europeu de Drets Humans segons un altre magistrat.

Davant el fet que el codi penal pugui donar peu a tal diversitat d’interpretacions, i considerant la influència del context personal de cadascú sobre aquestes, sembla que una manera d’evitar biaixos sistemàtics en les decisions judicials seria garantir una pluralitat sociocultural i ideològica en el sí del poder judicial. Si ens fixem doncs en els tres magistrats firmants de l’auto en qüestió, veiem que un ve d’una família que porta quatre generacions exercint en l’administració de la Justícia, un altre és nét d’un magistrat del Tribunal Suprem i l’últim és el fill del rector de la Universitat de Santiago que va investir a Franco com doctor honoris causa el 1965. Tot sigui dit, la família no es tria, i l’arbre genealògic no hauria d’impedir a ningú la pràctica de cap professió. No obstant cal destacar que durant la transició espanyola molts magistrats de l’antic Tribunal de Orden Público van ser automàticament recol·locats en les noves estructures de poder judicial, i com en nombroses ocasions es van permetre arxivar casos i deixar en llibertat a autors d’assassinats ocorreguts ja durant la transició. Així doncs, sembla que l’esmentada composició de la sala del Tribunal Suprem que manté a Junqueras a la presó no és una casualitat. De fet, el periodista d’investigació Alfredo Grimaldos, autor del llibre ‘Claves de la transición 1973-1986 para adultos’, afirma que “El sistema judicial actual es de extrema derecha”.

Reitero doncs, tornant al principi, que molts periodistes, politòlegs, influents usuaris de Twitter o jo mateix, no tenim els els coneixements necessaris per a valorar l’auto de Junqueras. El que sí que em permeto afirmar, és la importància que té l’hermenèutica en les resolucions judicials. Així doncs, més enllà dels anàlisis amateurs d’interlocutòries, potser el que cal és qüestionar si davant el ventall tant ampli d’interpretacions emanant d’experts en dret penal que podem trobar, l’evidència de quines resolucions acaba emetent el poder judicial espanyol permet afirmar que existeix un biaix sistemàtic contra certes ideologies o minories regionals. I aquí potser em precipito, però davant el continuisme del poder franquista durant la transició, la tortura a militants independentistes, els ja 420 dies de presó preventiva als joves d’Altsasu, la complicitat de les forces de l’ordre amb atemptats i agressions per part l’extrema dreta, o com el Ministerio de Justícia ignora exhorts de la Justícia Argentina per a investigar alts càrrecs del franquisme i la transició… crec que hi ha quelcom que fa pudor.

Aquests records no haurien d’estranyar massa a ningú, ja que a data del 2016 Sant Feliu comptava amb 44086 habitants, el qual representa un creixement del 20% respecte el 1995. Hom pot entendre doncs, que l’ampliació de la superfície urbana de Sant Feliu és una mera resposta a un creixement demogràfic, que de fet ha estat inferior al 26% corresponent a la mitjana catalana. El debat més calent que hi ha sobre la taula, segons les opinions que he llegit, és la manera en la que aquests nous edificis contribueixen a fer Sant Feliu una ciutat dormitori.

El que intento explicar a continuació, és com la transformació que porta a Sant Feliu cap a una ciutat dormitori no és més que el resultat d’un procés demogràfic que es manifesta arreu del món en diverses magnituds i variants, però compartint gran part dels mateixos fonaments.

Com es distribueix la població catalana?

L’arrel del problema s’amaga sota les distribucions estadístiques de la població a Catalunya. Per a començar amb un exemple, mireu el gràfic següent:

A l’esquerra i a la dreta es mostren respectivament les distribucions d’alçada i pes d’un grup d’individus, segons una enquesta de joves de 18 anys a Hong Kong. Veiem doncs que la gran majoria d’individus de l’estudi mesuren entre 165 i 180 centímetres i pesen entre 45 i 70 kilograms, amb mitjanes entre els 170-175cm i 55-60kg respectivament. El que s’observa correspon una distribució normal, que és raonablement com estem acostumats a pensar quan parlem de números. Per exemple, quan comparàvem notes a la classe de l’institut, probablement la mitjana estava entre el 5 i el 6, mentre que la majoria tenia notes entre el 3 i el 7. Desprès quedaven els pocs estudiants que havien tret entre un 1 i un 2 i a l’altre extrem també pocs estudiants que tenien un 9 o un 10.

Si ara observem com es distribueixen els habitants en els diferents municipis de Catalunya segons l’Idescat, ens trobem amb el següent:

La diferència entre aquesta distribució i les d’alçada i pes mostrades abans són evidents. La gran majoria de municipis es situen a l’esquerra del gràfic, amb poblacions molt petites, mentre que hi ha grup pràcticament inapreciable de municipis de més de 200000 habitants. El que s’observa correspon a una distribució lognormal amb una cua aproximable per una power-law segons la Llei de Zipf. El lingüista estatunidenc George Kingsley Zipf va popularitzar aquesta llei a l’observar que en la gran majoria de paraules d’un llenguatge s’utilitzen rarament, ja que hi ha un grup reduït de paraules que utilitzem molt sovint. Hi ha doncs una clara analogia amb el nostre gràfics: la gran majoria de municipis són poc poblats, mentre que hi ha un grup petit de municipis amb molts habitants. De fet, aquest comportament es replica en molts altres fenòmens de la ciència, com ara sistemes biològics, termodinàmics o conjunts fractals en matemàtiques. Una pràctica habitual per analitzar millor aquests fenòmens és representar-los en escala logarítmica. Si ho apliquem al nostre gràfic, trobem el següent:

S’observa així que la majoria de les poblacions tenen entre 100 i 10000 habitants, amb un únic municipi de més d’un milió d’habitants situat a l’extrem dret, que com bé podem imaginar correspon a la ciutat de Barcelona. Malgrat ser tant recurrents, el que sobta d’aquestes distribucions és la dificultat que ens comporta intentar predir o extrapolar-ne conclusions de manera intuïtiva. Per exemple, sabent que mesuro 177cm, tots seríem més o menys capaços d’estimar intuïtivament i amb relativa precisió que un 25% de les persones de la meva edat deuen ser més alts que jo. En canvi, malgrat que ens diguin que a Barcelona hi ha el 21% dels habitants de Catalunya, segurament no sabrem estimar tant fàcilment quin percentatge dels Catalans viuen en les 5 ciutats més grans (són un 33%, entre Barcelona, l’Hospitalet de Llobregat, Badalona, Terrassa i Sabadell).

Què ens ha portat a aquesta desproporció?

Qualsevol persona que conegui més o menys Catalunya es pot imaginar que el creixement demogràfic no s’ha repartit de manera simètrica. Probablement, els habitants de tercera edat la vida dels quals ha transcorregut en els petits pobles de la Catalunya interior ens podrien explicar el progressiu èxode dels seus fills i néts cap a poblacions més grans de la comarca o directament cap a Barcelona. Aquesta migració succeïx de manera retroactiva: com que les poblacions més grans ofereixen més oportunitats, més habitants nous hi arriben, que a la vegada fan créixer el municipi, que passa a oferir encara més oportunitats.

Aquest comportament es coneix com a avantatge acumulatiu o preferential attachment, i es pot resumir en l’aforisme de “els més rics es tornen més rics, i els més pobres més pobres”. La formalització d’aquest fenomen s’atribueix a les observacions del matemàtic escocès Udny Yule sobre el fet que els gèneres biològics amb més espècies són també els que tenen més capacitat de generar noves espècies.

Veiem un exemple de com funciona el preferential attachment, il·lustrat amb una xarxa d’amics, on cada node representa una persona, i un enllaç entre dos nodes simbolitza que les persones corresponents són amics. La situació inicial, il·lustrada en la imatge a continuació, és una persona central que és l’única amistat dels altres 9 membres.

Per a simular el preferential attachment, farem que arribin noves persones a la xarxa, que intentaran fer-se amics d’algú que ja tingui molts amics. Els detalls exactes de la implementació estan descrits en la documentació del programa utilitzat. Veiem doncs com evoluciona el sistema quan hi ha 25, 50, 100, 250 i 500 persones respectivament:

Podem observar doncs, que una idea molt senzilla com el preferential attachment pot generar patrons clarament estructurats. Les imatges de la xarxa probablement ens recordin a diferents fenòmens: enllaços entre pàgines web, amistats de Facebook, connexions entre Aeroports, xarxes de proteïnes en biologia o matèria condensada en física. Aquest tipus de xarxes agrupen sota el nom de scale-free networks, i comparteixen propietats molt similars en camps completament diferents de la ciència. Les xarxes que formen les ciutats i pobles, no en són una excepció.

Com es pot observar, a mesura que evoluciona el sistema, la persona central acapara gran part de les amistats. De fet, la distribució del numero d’amics que té cada persona de la xarxa, (representada en el gràfic de “Degree Distribution”) es va assemblant cada cop més a la distribució en municipis dels habitants de Catalunya. Al cap i a la fi, les decisions que guien l’evolució de les ciutats segueixen el mateix raonament que els nouvinguts a la xarxa: quan decidim mudar-nos a una ciutat, la majoria busquem aquelles que ens ofereixin més oportunitats, de la mateixa manera que els nouvinguts busquen fer-se amics d’aquells que tenen més amics.

L’Àrea Metropolitana de Barcelona i l’urban sprawl

Segons el raonament del preferential attachment, la ciutat de Barcelona hauria d’haver acaparat la majoria del creixement de Catalunya durant els últims anys. Però si mirem com ha evolucionat la població de les comarques de Catalunya en termes absoluts des del 1981 al 2011 segons l’Idescat, podrem observar com no ha estat així.

Veiem doncs que el Barcelonès és justament la comarca que més població ha perdut, concretament 208211 habitants. En canvi, les comarques de l’Àrea Metropolitana de Barcelona (AMB) i el Tarragonès són les que més han crescut. La hipòtesi més versemblant seria que ja no queda espai per allotjar nous residents al Barcelonès, i per tant aquests s’estableixen a les comarques de l’AMB. Però no només no s’ha pogut allotjar nous residents, sinó que hi ha hagut una pèrdua de població significativa. Això implica que força residents del Barcelonès que han decidit marxar-hi, molt possiblement per a establir-se en poblacions veïnes de l’AMB.

Aquesta migració cap a les rodalies normalment s’atribueix a les preferències per un context residencial més tranquil, espaiós i verd, evitant la congestió de les ciutats sense allunyar-se’n excessivament. La seva prevalença en el context nord americà ha popularitzat globalment el terme urban sprawl, que es refereix al model urbà de zones residencials disperses, separades dels centres d’activitats d’oci i comercials, sovint forçant als seus habitants a una forta dependència del transport motoritzat. Probablement això ens faci pensar a localitats del Vallès, el Maresme o del Baix Llobregat, com ara Sant Cugat, Cerdanyola, el Masnou, Mongat, Vallirana, Corbera, i també a barris Santfeliuencs recents com ara Les Grasses, Mas Lluhí o els nous habitatges al costat de la Ciutat Esportiva Joan Gamper.

Implicacions a l’escala Sant Feliu i de Catalunya

El que descriu aquest article doncs, es pot recapitular en les següents observacions:

• la distribució demogràfica de Catalunya està molt descompensada, amb una minoria de municipis concentrant la major part de la seva població
• aquesta distribució és resultat d’on procés retroalimentat que tendeix a desproporcionar-la encara més (el preferential attachment), amb Barcelona com a gran centre de gravetat
• la saturació de la ciutat de Barcelona, i les preferències per un context residencial més tranquil, han desencadenat una migració dels seus habitants cap a altres municipis de l’AMB

Com hom pot imaginar, aquestes observacions són extrapolables a la majoria de regions desenvolupades del món: segons les previsions United Nations, la proporció mundial de persones vivint en ciutats ha passat d’un 3% al 1800 a un 50% actual, amb estimació d’arribar al 60% al 2030. Seguint l’actual tendència global cap a l’urban sprawl, s’espera que la superfície total ocupada per aglomeracions es tripliqui els propers 40 anys.

Considerant la magnitud d’aquest procés demogràfic, cal entendre que probablement hi ha poques solucions implementables des de l’ajuntament de Sant Feliu. Els municipis metropolitans estem condemnats a jugar un paper clau en aquesta transformació, i per a no acabar sent satèl·lits invisibles de Barcelona, cal canviar la percepció territorial de Catalunya. Com remarca el periodista Marc Andreu en un extret del seu llibre Les ciutats invisibles, “Que en el concepte periodístic i polític de territori, tant interioritzat a Catalunya, no s’hi incloguin les realitats metropolitanes només s’explica com una vacuna mal receptada contra el centralisme de Barcelona”. D’altra banda, potser també són necessàries altres respostes a tal centralisme, com ara solucions a l’èxode juvenil dels pobles de la Catalunya oblidada “d’on als 18 anys marxes a estudiar i rarament hi tornes”, ja que en molts casos tals joves no s’estableixen a l’AMB per voluntat pròpia, sinó per manca d’oportunitats en la seva comarca d’origen.

Però inclús acceptant que la migració cap a l’AMB és inevitable, cal plantejar quin tipus de teixit urbà acomodarà aquest creixement. Un estudi de l’Universitat Autònoma de Barcelona va monitoritzar les transformacions de l’AMB entre el 1993 i el 2000, destacant com la proliferació de zones industrials i comercials segregades junt amb la privatització d’espais públics ens estan acostant cada cop més al model urbà nord americà. Ara bé, l’estudi distingeix una característica peculiar de l’AMB: l’existència de centres vitals a les afores de Barcelona ha frenat notablement l’urban sprawl, ja que sembla que els seus habitants tenim més preferència per habitar a prop d’aquests centres, a diferència dels països anglosaxons. El que potser podem fer des de Sant Feliu doncs, és aprofitar aquesta diferència per a evitar acabar sent una ciutat dormitori: com deia l’Arnau Picón al Fet a Sant Feliu, “Més enllà de les polítiques municipals necessàries, comença a ser hora que la ciutat s’espavili. Els qui més hi hem de fer som les entitats, els comerços i els ciutadans”.

No entraré a valorar quines són les decisions de l’ajuntament en clau urbanística que ens poden portar al model de ciutat dormitori o d’urban sprawl. Ara bé, el que sí que hem de reclamar tots els Santfeliuencs és el nostre dret a participar activament en aquest debat i la presa de decisions, ja que votar un cop cada quatre anys no és suficient per a hipotecar el model de ciutat que volem per als propers anys.

Avui es retroben a Seattle, al nord-oest dels Estats Units: en Miguel hi viu des de fa gairebé un any, ja que treballa a l’Hotel Sheraton; en Martí en canvi, tot just hi és de passada com a part d’un viatge que realitza a la Costa Oest abans de començar un doctorat a Suïssa. Des d’un café del modern barri de Capitol Hill, ens expliquen la desconeguda història de com vàren fer arribar el Reggaeton (i altres músiques llatines) a Catalunya.

Malgrat haver crescut els dos a les rodalies de Barcelona, proveniu d’entorns molt diferents. Com va ser la vostra infància abans de conèixer-vos?

M.C. Jo vaig créixer en un entorn on predominava la llengua castellana, molts dels meus amics eren del Madrid i el català era només una assignatura a l’escola. Quasi tots érem descendents de famílies procedents d’altres regions d’Espanya, predominantment de el Sur, en el meu cas de Múrcia i Rus, a Jaén. Probablement per això, la comunió era inqüestionable, tots portàvem els penjolls amb la creu, i sentíem els daltabaixos de la Roja com a nostres.

M.B. En l’arbre genealògic de la meva família, només hi podem recordar cognoms catalans, però sempre ha estat l’únic cas en la meva classe. Havent anat a l’escola pública al barri Falguera, vaig viure (en menor mesura) un ambient semblant al que explica en Miguel. D’altra banda, els caps de setmana feia vida principalment al barri centre, on el teixit associatiu ha contribuït sempre al fet que les tradicions d’arrel més catalana fossin sempre presents en els seus ciutadans. Formant part d’aquests dos móns em semblava increïble com conviviem tots en la mateixa ciutat d’una manera tant impermeable.

A Catalunya ja fa uns anys que a les festes majors i locals d’oci hi predomina la música llatina. Com us sentiu després d’haver aconseguit aquesta gesta?

M.C. A casa i al carrer, em vaig criar escoltant estils de música llatina i espanyola. En aquell llavors a l’Hospitalet, ningú de la nostra edat havia escoltat mai cançons de Lluís Llach o Marina Rossell. Per mi no és més que veure com allò amb el que he viscut des de petit s’ha extès a la resta del País. Tot i estar orgullós d’haver assumit la responsabilitat d’extrendre i fer entendre el Reggaeton a les poblacions amb més tradició catalana, no puc evitar sentir certa indignació al veure com els joves ara en gaudeixen sense preguntar-se quant va costar, ni qui se’n va fer càrrec.

M.B. Les úniques cançons castellanes que havien tingut presència al cotxe de la meva família eren “Y cómo es él” i “Me llamas” de José Luís Perales. Nosaltres escoltàvem Els Pets o Lax’n’Busto (mai Sopa de Cabra perquè no soportàvem en Gerard Quintana). Per això estic sorprès d’haver estat involucrat d’aquesta manera en el procés d’acceptació la música llatina a Catalunya, encara que he d’admetre que si no fos pel Carra mai l’hagués ni tan sols considerat.

Heu utilitzat expressions com “fer entendre” i “procés d’acceptació” en el context d’una música molt popular que s’escolta arreu del món. No utilitzeu termes molt exagerats?

M.C. Ni molt menys, deixa’m que et posi uns quants exemples. Quan vaig arribar a Sant Feliu al 2006, cada vegada que intentava posar una cançó de Reggaeton se’m titllava de cani, xungo i se’n reien dient “com es nota que ets de Hospi”. Això va seguir així durant anys, de fet quan vaig conèixer al Martí el 2008 res havia canviat. És més, ell va ser el que més se’n va riure de mi per el sol fet de que m’agradés aquesta música.

M.B. És cert, de fet jo sentia vergonya aliena per el latineo, el seu ritme descarat i les seves lletres dantesques. No va ser fins a l’estiu del 2010, quan amb la família estàvem de vacances a Paraguai que vem arribar a escoltar-ne una cançó sencera: a la ràdio va sonar “Noche de Sexo”, i al trobar-nos a l’Amèrica Llatina vem trobar pertinent deixar-la sonar. Tot i que no em va agradar, és cert que la vaig percebre de manera diferent.

cada vegada que intentava posar una cançó de Reggaeton se’m titllava de cani, xungo i se’n reien dient “com es nota que ets de Hospi”

Llavors, si en aquell moment tampoc et va agradar, què et va fer canviar d’opinió?

M.B. De fet va ser tot plegat casualitat. Poc després de tornar d’aquelles vacances, estàvem bevent a casa meva abans de sortir al Milenio, i el Carra va intentar posar una de les seves cançons. Tothom va reaccionar contràriament. A mi en canvi, em va recordar a la cançó que havia escoltat feia unes setmanes, i li vaig proposar que si la coneixia, la posés.

M.C. Evidentment la vaig reconèixer immediatament, però no la tenia al MP3, i en aquella època ningú tenia internet al mòbil. Per això li vaig proposar de demanar-la un cop arribéssim al Milenio. Però vem ser massa innocents de pensar que tot seria tant senzill. Jo sabia que no m’anaven a prendre seriosament ja que no hagués estat ni molt menys la primera vegada que demanés una cançó. Però pensàvem que seria diferent si la demanava en Martí…

M.B. Justament l’estiu del 2010 vàrem passar amb els amics moltes tardes mandroses de cafè a l’Ateneu, en les quals sovint el gerent del Milenio, en Miquel Albert s’autoconvidava a la nostra taula. Ell ens explicava què tenia preparat per aquell cap de setmana (per exemple, nits de Mojitos o Xupitos especials), i amb els amics hi anàvem i hi consumíem. De forma que vaig creure que no era agosarat anar a demanar una cançó, encara que fós d’un gènere tan controvertit. Al cap i a la fi, quan sortim de nit anem a divertir-nos, no?

I com va acabar la cosa?

M.B. Em vaig dirigir al DJ que en aquell llavors hi havia al Mile, l’Albert de Síbaris, i li vaig demanar si podia posar la cançó en questió. Ell, per cercar-la em va demanar l’autor. I jo, palplantat sense saber la resposta, vaig demanar-li ajuda al Carra.

M.C. Inevitablement obligat a intervenir, li vaig dir a l’Albert: “Noche de Sexo”, de Aventura ft. Wisin y Yandel. I la seva resposta va ser: “Aquí hace mucho que no se pone Reggaeton”.

I no us va donar cap més explicació?

M.C. No ens va donar cap explicació perquè no calia: la seva expressió de mofa no deixava dubtes. De manera que, esperant comprensió, vem explicar als nostres amics el que havia passat. Però malauradament la seva reacció va ser més de indiferència que de suport. Aquí és quan vaig adonar-me’n que calia fer un pas més enllà. Els havia de fer entendre que el Reggaeton no només s’escolta, es viu. Per això li vaig ensenyar al Martí el vídeo de “Ella y yo” de Aventura ft. Don Omar en directe al Madison Square Garden, on els artistes no només canten, sinó que interpreten el seu paper en el triangle amoròs que descriu la cançó.

M.B. La lletra de “Ella y yo” va ser la primera lletra de Reggaeton a la qual li vaig dedicar atenció. I em vaig emportar una grata sorpresa positiva, ja que va molt més enllà amoroses buides i superficials que tant sovint trobem en la música pop, i hi afegeix un nou element: la passió, encara més observable en l’actuació en directe. I llavors vàrem pensar que si aconseguiem transmetre aquella passió, els nostres amics voldrien tant com nosaltres escoltar Reggaeton en els moments de rauxa.

I què veu pensar per a aconseguir-ho?

M.C. Doncs llavors va arribar ja la nit de Halloween (com l’anomenem a Hospitalet) o la Castanyada (com dicta el chauvinisme català). En aquella nit, a alguns dels llavors caps del CAU se’ls va acudir utilitzar les instalacions per a activitats que poc tenen a veure amb l’escoltisme: beure begudes alcohòliques abans d’anar al Milenio. Empassats ja els dos primers cubates vem trobar que era el moment perfecte per posar “Ella y yo”, però un altre cop ens vem trobar que ningú la tenia al mòvil, i ningú tenia internet per a posar YouTube. Aleshores ens vem adonar que ho hauríem de posar tot de la nostra part: l’havíem d’interpretar nosaltres.

M.B. Jo ja l’havia escoltat prous vegades i em sabia la lletra de memòria. En Carra per suposat feia anys que es va memoritzar fins la última coma. Així que per el to de veu vàrem decidir que jo interpretaria la part del Romeo i en Carra faria de Don Omar. Va ser un gran èxit, gràcies al qual teníem un excèrcit d’amics units a la nostra causa. I aquella mateixa nit tots junts vàrem anar al Milenio a demanar que ens posessin “Ella y yo”.

Entenc doncs que aquest cop va haver-hi sort?

M.C. Tandebò hagués estat tan fàcil. Malgrat que els nostres amics aquella nit van insistir junt amb nosaltres, tant a l’Albert de Síbaris com al Miquel Albert els va ser igual. Van seguir pensant que la vitalitat del beat del Reggaeton els faria mal al seu negoci, com si el Mile tingués cap estàndard a mantenir.

M.B. Però de totes maneres, aquella nit va acabar sent un punt d’inflexió: els nostres amics havien gravat la nostra interpretació en vídeo, i la van publicar al Facebook. Allà no només vàrem aconseguir el suport del nostre grup més proper: molts altres Santfeliuencs ara volien veure també la performance al Milenio.

Llavors, quant temps més va poder resistir la Milenio a la pressió popular?

M.C. Relativament poc, però perquè s’hi va ajuntar un altre element determinant. Justament aquell novembre va sortir “Danza Kuduro”, que malgrat ser cantada majoritàriament per Don Omar tenia un beat més lleuger i inevitablement addictiu que va fer que fos acceptada per inclús els més reticents. Estic segur que fins i tot en Miquel Albert l’escoltava en privat. Així que cap al desembre ja es va anar deixant sentir per al Milenio, i amb ella van venir més cançons.

Per com ho dius sembla que va ser més aviat “Danza Kuduro” el punt d’inflexió en l’extensió del reggaeton…

M.C. Cal reconèixer que va ser una ajuda molt important, però si no haguéssim aclimatat prèviament als Santfeliuencs, posaria la mà al foc que no els hagués agradat tampoc “Danza Kuduro”.

M.B. Però cal contextualitzar encara més: estem parlant de Sant Feliu, que no deixa de formar part del Baix Llobregat. Justament aquell any havíem començat la universitat, i allà els meus companys eren de la Catalunya més vella i pura que no sabia ni situar Sant Feliu de Llobregat al mapa. Jo sempre els deia “Som la capital del Baix Llobregat”, però llavors no entenien com podia tenir vuit cognoms catalans. Això ajuntat amb la meva obsessió al reggaeton els trencava tots els esquemes del que havien cregut que podia ser Catalunya. Però no m’estranya: el vigatà i el gironí mai s’ha dignat a anar més al sud que la Ciutat Comptal. Per ells, qualsevol cosa en la lengua del imperio era considerada una heretgia, de manera que el reggaeton no tenia cap cabuda en la seva imatge platònica de Catalunya.

els meus companys d’universitat, originaris de la més pura Catalunya Vella on ningú ha votat mai al PSC no sabien ni situar Sant Feliu al mapa. Jo sempre els deia “Som la capital del Baix Llobregat”, però llavors no entenien com podia tenir vuit cognoms catalans. Això ajuntat amb la meva obsessió al reggaeton els trencava tots els esquemes del que havien cregut que podia ser Catalunya. Però no m’estranya: el vigatà i el gironí mai s’ha dignat a anar més al sud que la Ciutat Comptal.

I què va caler per acostar a un Olotí o Empordanenc a la sub-cultura de la música llatina?

M.B. Doncs vaig cada nit que fèiem pre-festa al pis d’algú, era molt insistent i els forçava a escoltar cançons senceres. Hi accedien perquè els semblava inevitablement divertit veure com em sabia totes les lletres de memòria. Però la clau va ser que llavors ja es deixaven sentir temes com “Ayer la ví” o “La despedida” a les discoteques Barcelonines, i a part sonaven en els moments crítics en què el cos et demana remembers com Chayanne o Sónia y Selena. I com que ja els havia donat les molles a la pre-festa, llavors i només llavors era quan els meus amics agironats no podien resistir la temptació i volien el pa sencer: el perreo. I vaja, després et demanàven les lletres, i poc a poc se’ls anava desfent el chauvinisme català/gironí que imperava al seu cervell.

Un cop aconseguit que un estudiant gironí escolti reggaeton, la resta és bufar i fer ampolles, no?

M.B. En efecte, aquesta va ser la victòria definitiva. El reggaeton va ser un terratrèmol que va trencar definitivament les barreres entre les dues Catalunyes. És l’element cohesiu més important què hi ha hagut mai a Catalunya, l’únic punt comú entre un frequentador de Foro-Coches i els fans de “Els Catarres”. Daddy Yankee i Don Omar pixen damunt del 80è aniversari del PSUC: amb una desena de temes de reggaeton ens han fet un sol poble.

Amb la rellevància que té tot el què dieu, com és que mai se us ha reconegut la vostra labor?

M.C. La cultura catalana està inevitablement polititzada, i als mitjans subvencionats ja els ha costat prou l’acceptació del reggaeton com per a sobre fer-ne un reportatge de com va arribar a Catalunya. Ni els cooperatives independents finançades a través de micromecenatge com el Fet a Sant Feliu s’han dignat a contactar-nos.

Personalment, un dels fets que em van afectar més va ser sentir en primícia “La Calor que em crema” de Trifàssic, la primera cançó de reggaeton en català. Els vaig demanar si podia participar en el seu proper tema i ells em van dir que hi estarien encantats. Encara espero la seva trucada. Des de llavors intento fer la meva carrera en solitari i no conto amb més suport que el dels meus amics

La cultura catalana està inevitablement polititzada, i als mitjans subvencionats ja els ha costat prou l’acceptació del reggaeton com per a sobre fer-ne un reportatge de com va arribar a Catalunya. Ni els cooperatives independents finançades a través de micromecenatge com el Fet a Sant Feliu s’han dignat a trucar-nos.

M.B. De totes maneres estem orgullosos d’haver aplanat el camí als joves que volen escoltar reggaeton en pubs, discoteques i festes majors. I encara més, també hem ajudat a que joves artistes com Bad Gyal hagin pogut fer-se un nom en el panorama musical català a través d’estils fins fa poc vetats per L’Aranguardia com el trap o reggaeton. No ens calen reconeixements personals. Les nostres motivacions van més enllà. Arribarà un dia en què en les paelles de la Pilar Rahola en Joan Laporta (que tornarà a ser president del Barça) convidarà als més quinquis del filial i junts combinaràn les “Paraules d’amor” d’en Serrat amb el “Llamado de Emergencia” de Daddy Yankee. Per què com deia Miquel Martí Pol, “tot està per fer, i tot és possible”.