Python contextvars don't persist across cells in notebook

Python 3.7 added a new contextvars package to "manage, store, and access context-local state".

However, if I set a ContextVar in one cell and then try to get it in another, it isn't persisted:

This may be an antipattern (not sure if there is a good use case to set a variable across many cells) and I worked around it by setting default, as recommended. However it is surprising and is different than how ipython works.

@Carreau

@Carreau is this related to the async stuff?

@njsmith

Likely. I might need @njsmith on that.

That's weird. ContextVars are like thread-locals until someone explicitly calls Context.run... is the kernel single-threaded? Does it use Context.run? Is some other lib calling it?

@minrk

In the notebook it's probably running the tornado eventloop... maybe @minrk knows ?

I believe tornado on py3 uses the asyncio loop by default, and if you're on py 3.7 then asyncio will automatically use different contexts for different tasks. And probably each cell execution counts as a different task as far as tornado/aio is concerned.

Sounds like the kernel should be explicitly holding a Context object and reusing it for each cell execution, instead of letting tornado/aio set the Context implicitly.

@njsmith

Thanks @njsmith! That sounds like a pointer in the right direction. Each execution is indeed an independent coroutine submission. I'm not quite sure if the fix for this will belong in core ipython, ipykernel, or both. Hopefully just ipython.

Also, related to this. It seems like context vars are not correctly propagated when initializing or calling code that get/sets vars in differently in async functions.

Here is some code to display the problem:

from IPython.terminal.embed import InteractiveShellEmbed
from traitlets.config.loader import Config

import contextvars

cvar = contextvars.ContextVar('foo')


def run():
    cvar.set('bar')

    def foo():
        cvar.set('foofunc')

    async def afoo():
        cvar.set('afoofunc')

    ipshell = InteractiveShellEmbed(config=Config())
    ipshell()


if __name__ == '__main__':
    run()

Here is in ipython prompt:

In [1]: cvar.get()
Out[1]: 'bar'

In [2]: foo()

In [3]: cvar.get()
Out[3]: 'foofunc'

In [4]: await afoo()

In [6]: cvar.get() == 'afoofunc'
Out[6]: False

Possible workround for ipython/ipython#11565

@njsmith

I recently had a package that was build with contextvars that suddenly broke in jupyter notebooks. It would be great if this issue could be somehow addressed, as it might become a real problem if more packages start to rely on contextvars and simply break unexpectedly when used within ipython/jupyter.

Two observations that I made are:

The problem does not occur, when Run all cells is used.
The problem also does not seem to occur when Python 3.6 and the backport of contextvars is used, but that might well be related to what @njsmith described above.

Add warning about GPU selection in Jupyter notebooks

This produces surprising results when changing the decimal precision (or other aspects of the decimal context) in notebooks, since decimal now uses a PEP 567 context variable for the decimal context.

Related: ipython/ipykernel#494, jupyter/notebook#5260

Note:
This does not impact IPython, or the xeus-python kernel.

Adjust tutorial colab to work correctly with older versions of ipytho…

Navigation Menu

Repository navigation

Python `contextvars` don't persist across cells in notebook #11565

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Python contextvars don't persist across cells in notebook #11565

Description

Activity

takluyver commented on Jan 18, 2019

Carreau commented on Jan 18, 2019

njsmith commented on Jan 18, 2019

Carreau commented on Jan 18, 2019

njsmith commented on Jan 18, 2019

minrk commented on Jan 21, 2019

vangheem commented on Oct 2, 2019

JarnoRFB commented on Oct 1, 2020

mdickinson commented on Mar 23, 2021

SylvainCorlay commented on Apr 9, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Python `contextvars` don't persist across cells in notebook #11565