Text Notebooks

Example Notebooks

With Jupytext, text files become Jupyter Notebooks

notebook.py

# %% [markdown]
# # Analysis
# A quick look at the data.

# %%
import pandas as pd
df = pd.read_csv("data.csv")
df.describe()

30+ languages supported. Works in VS Code, PyCharm, Spyder...

notebook.myst.md

# Analysis

A quick look at the data.

```{code-cell} ipython3
import pandas as pd
df = pd.read_csv("data.csv")
df.describe()
```

Best for rich technical docs & Jupyter Book — directives, cross-refs and more.

notebook.md

# Analysis

A quick look at the data.

```python
import pandas as pd
df = pd.read_csv("data.csv")
df.describe()
```

Best for documentation-first notebooks — GitHub Markdown syntax.

notebook.qmd

---
title: Analysis
---

A quick look at the data.

```{python}
import pandas as pd
df = pd.read_csv("data.csv")
df.describe()
```

For multi-language scientific publishing — Python, R, Julia and more, rendered by the Quarto system.

Jupytext’s Scope

I see Jupytext as a bridge between communities and tools

The .ipynb format is a Rosetta Stone for notebooks
Use whatever editor or renderer works best for you: Jupyter Lab, Jupyter Book, VS Code, Quarto, Sphinx, Pandoc, Marimo…

Jupytext has extensive coverage

The MyST and Markdown formats support any language for which there is a kernel (100+)
30+ languages supported in the auto:percent format
Also available: auto:light, auto:nomarker, py:marimo, md:pandoc, py:sphinx, R:spin

My own favorite formats

py:percent - editable in any IDE or with AI → research and experiments
md:myst - great for documentation and education → I use it for ITables’ documentation
md:markdown for simple notebooks that look nice on GitHub
qmd:quarto for presentations like this one!

Demo time

Text notebooks

Install Jupyter and Jupytext

pixi init && pixi add jupyterlab jupytext

Launch JupyterLab

pixi run jupyter lab

Install ITables

pixi add pandas itables

Ask Claude to create a notebook

Please write a short py:percent notebook to help me try ITables

Right-click the .py file → Open With → Notebook

Run the notebook, make edits, save → .py is updated accordingly
Make a change in the .py file outside of Jupyter
File → Reload default from disk (🤔 what is default?)
Kernel is still live but outputs are lost!

Paired notebooks

Create a jupytext.toml file with this content formats="ipynb,py:percent"
Run and save the notebook → saves to .ipynb and .py
Close and re-open, or reload: outputs are still there!
We can still modify the .py file outside of Jupyter
The inputs in the .py file have precedence over the .ipynb file.

Story time

From R Markdown to Jupytext

2012–2017: Big fan of RStudio, R Markdown, ggplot2, datatables, shiny, Rcpp… I also authored wikify: internal Confluence publisher, a precursor to quarto publish
But… soon I was the last employee working with R 😨
2018: Transitioned to Python 🎉. Yet, frustrated that I could only edit notebooks in Jupyter
Knew it could be different → started exchanging on GitHub with my employer’s blessing! Identified a few projects going in a direction that I liked: ipymd and notedown. Great projects, but they did not work well enough for me (Python 2 etc)
I first implemented Jupyter Notebooks as R Markdown (nbrmd), then realized that more Pythonic formats would be appropriate (nbsrc, .py scripts)
We found a better name: Jupytext, made sure the tool was safe (many tests on round trips) and easy to use (Jupyter plugin, Python 2 & 3)

What do I get from Jupytext?

Jupytext is useful to me every day
I am glad to contribute back, and I feel honored that many people use Jupytext!
Maintaining Jupytext is a responsibility that I take seriously (and I think it helps on the CV)
I have learnt so much by developing Jupytext, as users or would-be users told me about their own frameworks, findings or wishes
Sharing software in the open reminds me of the best part of my days as a university researcher: the collaboration and learning. Also what I like about software is that it’s (hopefully) easier to use than math papers!

Who contributes to Jupytext

Maintainers

There are two of us! Mahendra Paipuri did tremendous work on our JupyterLab extension (the Jupytext Menu)

Advanced Users

I have had very constructive and consistent feedback from some users, some of whom I am glad to meet here for the first time!

Jupyter Maintainers and QuantStack

The community of Jupyter developers has been incredibly helpful, and they helped me ensure compatibility with the major Jupyter updates.

105 contributors so far!

What’s next

Stable/Production

Jupytext is actively maintained but is fairly stable now

Improve awareness

I do meet many people that have not heard of Jupytext but would find it useful.

Brand new jupytext.org website! Unfortunately I can’t do monthly newsletters like Marimo 😅

Brand

Jupytext is still under my GitHub user name, should we move it to an organization?

Jupyter-plainb

TypeScript implementation by QuantStack!! Are Text Notebooks becoming a Jupyter standard? Also, plainb works in web-only contexts like JupyterLite!

Implementation details

Pain points

I wish we had autoreload in Jupyter
Paired notebooks mean two files per notebook — functional, but not ideal
If the .ipynb file is more recent than the .py file, Jupytext will refuse to open the document because the user is at risk of losing an edit. Should not happen but… you edited the .ipynb outside of Jupyter, or git clone was slow?
Merging text files is easy; however, if you also commit the .ipynb files you still have the challenge of merging them

The `py:percent` format

While the py:percent format looks (and is) mostly trivial, we do have the following requirements:

we need round trip stability for both the .ipynb and the .py file
the .py file should be correct Python

Detailed features

Code cells can have a title and metadata
Blank lines between cells are preserved on round trips (imports, functions…)
Jupyter magic commands are commented out in the .py file
Bash commands (ls, cat or !custom) are also commented out
Magic and bash commands are uncommented back in the .ipynb!

Synchronization

When does the synchronization occur?

Jupyter

Sync happens on load in memory, and is only materialized on disk when saving.

On load: inputs taken from .py, outputs from .ipynb
On save: write to .ipynb and then to .py

CLI

jupytext --sync takes inputs from the newest paired file.

Use --check-source-is-newer to force the file you pass as the source.

VS Code

With the Jupytext Sync extension by Victor Negîrneac, sync happens when either file is saved, using jupytext --sync --check-source-is-newer under the hood.

pre-commit

We also have a pre-commit hook for jupytext --sync!

You need to a) edit only one file or b) change both files in a consistent way.

Pairing in subfolders

Paired notebooks at the same location with formats = "ipynb,py:percent"
Paired notebooks with custom extensions: formats = "ipynb,.myst.md:myst"
Paired notebooks with three files: formats = "ipynb,py:percent,md:myst"
Notebook and script folders: formats = "notebooks///ipynb,scripts///py:percent"

Merging inputs and outputs for paired notebooks

Paired notebooks take the inputs from the most recent text file in the pair, and outputs from the .ipynb file.

Metadata that has been filtered out of the text notebook is also restored from the .ipynb file.

The merging algorithm allocates the outputs and cell metadata to the input cells in a way that resists simple edits, cell moves, and reformatting.

Native vs non-native formats

The percent, light, sphinx, markdown, and Rmd formats are fully implemented within Jupytext.

The quarto, pandoc, marimo formats simply call the corresponding tool to achieve the conversion. It’s still useful to have them in Jupytext as it provides the Jupyter integration.

The MyST format is an exception as it is partially implemented in Jupytext - we should probably delegate this entirely to MyST: #1543

QA on notebooks

Research notebooks

My research notebooks are scratch pads. They might not work anymore in two weeks.

Still, I commit them to a research repo. I use ruff check --fix and ruff format as pre-commit hooks on my .py notebooks to ensure minimal quality checks like no missing variable etc.

Documentation

ITables’ documentation notebooks are subject to

a test in the CI to make sure that they run
pre-commit ruff checks as above
plus, pyright type checks (hence the md:myst,py:percent pairing)

Text Notebooks with outputs

Jupytext’s issue #220 was about implementing a format with outputs.

My target:

Outputs are valid .txt, .png, .html files (etc) and they are saved in a folder with the same name as the notebook
The text notebook includes the outputs (but mainstream Markdown doesn’t have includes)
Our notebook with outputs looks good in at least one editor/renderer other than Jupyter (ideally, GitHub)

Did I put too many constraints and ended up with an empty solution set?

Conclusion

So nice to meet you all in person! 🙏
Happy to follow up and delegate the conversion to MyST
(count on me to minimize the noise on your notebooks)
Excited to see plainb landing, and text notebooks supported in JupyterLite and notebook.link!
Looking forward to more official support for text notebooks (but again, Jupyter has been amazingly supportive!)

Text Notebooks

Example Notebooks

Jupytext’s Scope

My own favorite formats

Demo time

Text notebooks

Paired notebooks

Story time

From R Markdown to Jupytext

What do I get from Jupytext?

Who contributes to Jupytext

What’s next

Implementation details

Pain points

The py:percent format

Synchronization

Pairing in subfolders

Merging inputs and outputs for paired notebooks

Native vs non-native formats

QA on notebooks

Text Notebooks with outputs

Conclusion

The `py:percent` format