Alice initializes her project.

cd /tmp
rm -rf /tmp/alice
mkdir -p alice && cd alice
git init
git commit --allow-empty -m "alice: init"
pdm init --runner mamba # using python 3.12 and defaults
git add .
git commit -m "alice: pdm init using python 3.12 and mamba"

Since this project will have both conda and PyPI dependencies, Alice also creates a conda environment that she’ll be using and makes pdm aware of it.

pdmcondaprofile="$(basename $(pdm venv create -cn alice | cut -d ' ' -f 2))"
pdm use --venv  "${pdmcondaprofile}" # populate the .pdm-python file with the name of the environment
echo "${pdmcondaprofile}"

STATUS: Creating virtualenv using mamba...
Using Python interpreter: /home/user/miniconda3/envs/alice-NLGZxbVC-3.12/bin/python (3.12)
alice-NLGZxbVC-3.12

Alice then installs dependencies for her project. However, instead of using pip or conda (or mamba) directly, she installs them via pdm and the pdm-conda plugin. Doing so ensures that the dependencies and their specific versions are noted in the project “lock” file. ¹⁴¹⁴For the benefit of her future self and others.

git status # confirm it's clean
pdm add --conda "postgresql>12.0" # may need multiple invocations for success

Having installed the dependencies, she confirms that they have been captured in the project “lock” file.

git status

On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   pyproject.toml

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	pdm.lock

no changes added to commit (use "git add" and/or "git commit -a")

She adds the “lock” file to the git repository index and commits the changes.

git add pdm.lock pyproject.toml
git commit -m "alice: install: postgresql>12"
git status

Alice will be using PyTorch in this project. She installs this from PyPI and commits the changes to the repository.

pdm add "torch>=2.3.0" # ✓
git add pdm.lock pyproject.toml
git status
git commit -m "alice: install: torch>=2.3.0"

During development, Alice also prefers some additional utilities to be available. Since these are not central to the project, but may be helpful to other collaborators, she installs these as development only dependencies.

pdm add --dev --conda "ipython"
pdm add --dev --conda "jupyterlab"

And then commits the changes.

git add pdm.lock pyproject.toml
git commit -m "alice: install: dev: ipython, jupyterlab"
git status

Thus far, Alice has all her dependencies installed and she has the ability to invoke the Python interpreter¹⁵¹⁵And other utilities. in the correct environment via pdm run:

pdm run which python
pdm run which postgres
pdm run python -c "import torch; print(torch.__file__)"
pdm run which ipython

/home/user/miniconda3/envs/alice-NLGZxbVC-3.12/bin/python
/home/user/miniconda3/envs/alice-NLGZxbVC-3.12/bin/postgres
/home/user/miniconda3/envs/alice-NLGZxbVC-3.12/lib/python3.12/site-packages/torch/__init__.py
/home/user/miniconda3/envs/alice-NLGZxbVC-3.12/bin/ipython

Note, however, that directly invoking python doesn’t do the “right” thing¹⁶¹⁶I.e., “right” here means invoking the version installed in the conda environment with access to the installed dependencies. — she has to prefix the command invocation with pdm run. ¹⁷¹⁷Even if Alice were using Rye and had the associated shim installed, directly invoking python would not have worked as intended. This is because she’s not using the project-local .venv, but is instead using an external conda environment. As such, the Rye shim wouldn’t have automatically been able to correctly resolve things.

which python
python -c "import torch; print(torch.__file__)"

/usr/bin/python
Traceback (most recent call last):
ModuleNotFoundError: No module named 'torch'

If Alice doesn’t want every command invocation to be prefixed with pdm run, she can accomplish that using direnv. In her case she has direnv installed via Guix.

which direnv

/home/user/.guix-profile/bin/direnv

She creates a .envrc file in her project with the following contents:

if [ -f ".pdm-python"  ]; then
    cmd="$(pdm venv activate 2>/dev/null)"
    if [ conda = "$(echo $cmd | cut -d ' ' -f 1)" ]; then
        # <https://github.com/direnv/direnv/issues/326#issuecomment-574779485>
        eval "$(conda shell.bash hook)"
    fi
    eval "$cmd"
else
    echo "Missing .pdm-python; ensure pdm has been initialized"
fi

She then “allows” the .envrc file so that it may be automatically evaluated.

direnv allow .

Now, running python directly, anywhere within the project directory,¹⁸¹⁸Including a sub-directory of it, and only when so. automatically invokes the correct Python interpreter. Prefixing the command invocation with pdm run is no longer necessary.

which python
python -c "import torch; print(torch.__file__)"

/home/user/miniconda3/envs/alice-NLGZxbVC-3.12/bin/python
/home/user/miniconda3/envs/alice-NLGZxbVC-3.12/lib/python3.12/site-packages/torch/__init__.py

Since Alice’s choice to use direnv is a personal one, she doesn’t subject others to it. She adds the .envrc file to .gitignore.

echo -e "\n.envrc" >> .gitignore
git add .gitignore
git commit -m "alice: gitignore: ignore .envrc"

Having done the initial setup to her liking, Alice continues developing, testing, extending her project. She takes care to ensure that every time she needs to add or alter a dependency

• she does so via pdm, and
• subsequently commits the changes to the pyproject.toml and pdm.lock files to git.

The above allows her to track her project dependencies with the project itself. It also allows her to track the dependencies in sufficient detail to allow her future self to recreate a previously known-to-be-working environment. As we’ll shortly see, doing the above also allows her to serve the needs of collaborators like Bob and users like Charlie at no extra cost.

Bob is Alice’s colleague, and he wants to be able to collaborate with Alice. He clones Alice’s repository so he can start working on it locally.

git clone /tmp/alice bob
cd bob

With a fresh checkout, this is what Bob’s view looks like. ¹⁹¹⁹Note, in particular, the absence of the .pdm-python file.

.  ..  .git  .gitignore  pdm.lock  pdm.toml  pyproject.toml  README.md  src  tests

Bob then proceeds to create a conda environment. ²⁰²⁰Alice, graciously, has noted the instructions in the README.

pdmcondaprofile="$(basename $(pdm venv create -cn bob | cut -d ' ' -f 2))"
pdm use --venv  "${pdmcondaprofile}"
echo "${pdmcondaprofile}"

STATUS: Creating virtualenv using mamba...
Using Python interpreter: /home/user/miniconda3/envs/bob-i2DthxFJ-3.12/bin/python (3.12)
bob-i2DthxFJ-3.12

Doing so creates a .pdm-python file with the correct entry.

/home/user/miniconda3/envs/bob-i2DthxFJ-3.12/bin/python

Finally, he installs the dependencies from the lock file. Since Bob will be contributing to the project, he installs both the development and the production dependencies. ²¹²¹This is the default behaviour of pdm install.

pdm install # may need multiple invocations for success

And now he has access to both Conda and PyPI dependencies.

pdm run which postgres
pdm run which python
pdm run python -c "import torch; print(torch.__file__)"
pdm run which ipython

/home/user/miniconda3/envs/bob-i2DthxFJ-3.12/bin/postgres
/home/user/miniconda3/envs/bob-i2DthxFJ-3.12/bin/python
/home/user/miniconda3/envs/bob-i2DthxFJ-3.12/lib/python3.12/site-packages/torch/__init__.py
/home/user/miniconda3/envs/bob-i2DthxFJ-3.12/bin/ipython

Not only can Bob now view Alice’s benchmarking and profiling results, but he can also run the Jupyter notebooks locally. He has been able to quickly and easily get to a point where he can lend his performance tuning expertise to Alice and collaborate with her.

Charlie is a user for whom Alice’s project is, simply, a means to an end. He would like to use Alice’s model on his own data. However, before he does so he would like to ensure that Alice’s project is correctly configured and he has all the requisite dependencies. ²²²²I.e., he would like to quickly verify that the test suite in Alice’s project successfully runs. He will, at present, not be contributing any changes to it. ²³²³In other words, Charlie is not interested in evaluating the benchmarks contained in the Jupyter notebooks. As such, he’s only interested in the production dependencies.²⁴²⁴As opposed to Bob.

git clone /tmp/alice charlie
cd charlie

Since the project has some “production” dependencies from conda, Charlie proceeds to create a conda environment following Alice’s instructions.

pdmcondaprofile="$(basename $(pdm venv create -cn charlie | cut -d ' ' -f 2))"
pdm use --venv  "${pdmcondaprofile}"
echo "${pdmcondaprofile}"

STATUS: Creating virtualenv using mamba...
Using Python interpreter: /home/user/miniconda3/envs/charlie-Z9GinPUb-3.12/bin/python (3.12)
charlie-Z9GinPUb-3.12

Finally, he installs the production dependencies from the lock file.

pdm install --prod # may need multiple invocations for success

As expected, he now has access to the production, but not the development dependencies.

pdm run which postgres
pdm run which python
pdm run python -c "import torch; print(torch.__file__)"
if [ $(pdm list | grep ipython >/dev/null) ]; then
    echo "STATUS ERROR: development dependencies were installed"
else
    echo "STATUS OK: No development dependencies were installed"
fi

/home/user/miniconda3/envs/charlie-Z9GinPUb-3.12/bin/postgres
/home/user/miniconda3/envs/charlie-Z9GinPUb-3.12/bin/python
/home/user/miniconda3/envs/charlie-Z9GinPUb-3.12/lib/python3.12/site-packages/torch/__init__.py
STATUS OK: No development dependencies were installed

Charlie can now proceed to run the test suite. Note that the test suite may still fail for other reasons. ²⁵²⁵For instance, if Alice has forgotten to pin some random seed. But Charlie is one step closer to understanding whether or not Alice’s project in its current state could meet his needs.

A non-exhaustive list:

• While the “lock” file generated by pdm is cross-platform by default, there is no guarantee that Alice hasn’t opted for the lock strategy to be platform specific.
• Additionally, even if the “lock” file is cross-platform, it doesn’t mean that the test suite has been confirmed to be successfully working on multiple platforms.
• The “lock” file doesn’t capture some environment specifics such as the OS kernel version, version of libc etc. ³⁶³⁶These gaps may or may not prove to be pertinent for the code to fulfill the developer’s intent.
• As noted previously, a “lock” file also doesn’t offer any protection against the specific versions of dependencies simply becoming unavailable at a future point in time.
• The approach followed by Alice in this post, while fulfilling some necessary criteria for reproducibility, is insufficient for reproducibility. Correctly capturing project dependencies doesn’t offer any protection from, say, the contents of a remote dataset changing over time. ³⁷³⁷Or a host of other possible issues that can affect reproducibility.

And yet, despite the above, if the approach outlined in this post is followed, the situation is already considerably better than what one routinely encounters in Python projects today.