Locks

This page is for people working on pikepdf itself. It describes the locks inside pikepdf, what each one protects, and how they behave on GIL-enabled and free-threaded Python. For what users of pikepdf need to know, see Callbacks and the thread safety section of Architecture.

Why pikepdf needs its own locks

qpdf is not thread-safe. A QPDF object, and every object handle that belongs to it, may only be used by one thread at a time. Even reading is unsafe, because qpdf loads objects from the file lazily: the first read of an indirect object moves the file position, parses, and fills the object cache.

On a free-threaded build nothing stops two threads from calling into the same QPDF at once, so a lock is obviously needed. On a GIL-enabled build it is needed too, which is less obvious. The GIL serializes Python bytecode, not qpdf operations, and it changes hands part way through many of them:

• when qpdf reads from a Python stream (readinto, seek and tell run Python code);

• when it writes to a Python stream, or calls a progress function;

• when it runs a pikepdf.TokenFilter or pikepdf.StreamParser callback;

• when it logs a warning through the logging module;

• when an allocation triggers a garbage collection, which can run any finalizer, including the one that destroys another Pdf;

• when pikepdf releases the GIL outright, as it does while it opens a file and while it writes one.

Each of these gives another thread a chance to enter the same QPDF while the first is still in the middle of an operation.

Overview

Lock	Protects	Kind
Per-Pdf lock (QpdfEntry::mutex)	one QPDF and all of its objects	ReentrantFtMutex
Unowned lock (unowned_mutex())	every Dictionary and Array that belongs to no Pdf	ReentrantFtMutex, global
Teardown lock (QpdfRegistry::teardown_mutex_)	objects of a QPDF that is being destroyed	ReentrantFtMutex, global
Registry lock (QpdfRegistry::map_mutex_)	the map from QPDF * to its entry	std::mutex, global
Wait table lock (ReentrantFtMutex::Waiting)	the table used to detect deadlocks	std::mutex, global
Adopted objects lock (QpdfEntry::adopted_mutex_)	the list of direct objects a Pdf has adopted	std::mutex, per Pdf
qpdf globals lock (qpdf_global_mutex)	qpdf’s global limits, through pikepdf.settings	std::mutex, global
Traversal locks (ObjectSet, ContentScopes)	the state of one graph traversal	std::mutex, per object
Pillow limit lock (_pil_limit_lock)	PIL.Image.MAX_IMAGE_PIXELS while pikepdf changes it	threading.Lock

Module-level settings that are a single value, such as the decimal precision and the default conversion mode, are constinit std::atomic variables, not locks. Per-thread state, such as the stack of conversion mode overrides, is thread_local.

On GIL-enabled and free-threaded Python

Lock	GIL-enabled build	Free-threaded build
Per-Pdf lock	Real. An uncontended lock is a few plain loads and stores, which the GIL makes safe. A thread that finds it held releases the GIL and sleeps on a condition variable.	Real. An uncontended lock is a compare-and-swap. A thread that finds it held detaches from the interpreter and sleeps on a condition variable.
Unowned lock	As the per-Pdf lock	As the per-Pdf lock
Teardown lock	As the per-Pdf lock	As the per-Pdf lock
Deadlock check	Runs only when a lock is contended	Runs only when a lock is contended
Registry, wait table, adopted objects, qpdf globals, traversal	std::mutex. Needed, because a thread can be inside qpdf with the GIL released. Each is held only briefly and never while calling Python.	std::mutex, held only briefly and never while calling Python
constinit std::atomic settings	Atomic, though the GIL alone would suffice	Atomic; required
Pillow limit lock	threading.Lock	threading.Lock

None of these locks compiles to nothing on a GIL-enabled build. Earlier versions of pikepdf relied on the GIL and made the per-Pdf lock a no-op there, which was wrong for the reasons listed above.

The per-Pdf lock

Every QPDF that pikepdf creates is registered in QpdfRegistry (src/core/qpdf_lock.h), which maps the QPDF * to a QpdfEntry. The entry holds the lock, the document’s conversion mode, and the list of adopted objects. Entries are reference counted: a thread that is locking a QPDF, or waiting to, holds a reference to the entry, so the lock outlives the QPDF if another thread destroys the document meanwhile.

The lock is re-entrant. One call often locks the same Pdf more than once: the binding locks it, then the type caster for the return value locks it again to resolve indirect objects; a Python method in _methods.py calls several bindings in turn; a callback calls back into pikepdf on the same Pdf.

How bindings take it

Bindings do not usually lock anything themselves. A class bound with locked_class_ (src/core/locked_bindings.h) wraps each method so that it locks the Pdf that owns each of its arguments before the body runs. Which Pdf owns a value of a given C++ type is declared once, by specializing LockOwner; a class whose type has no such declaration cannot be bound with locked_class_, so a forgotten lock is a compile error. A prek hook (cpp-locked-class in prek.toml) rejects plain py::class_ in the files that have been converted. A binding that must not be wrapped is registered as unlocked(f), so the exception is visible where it is made. A binding registered as shallow(f) is locked, but does not walk an argument that belongs to no Pdf; see below.

Code that locks by hand uses one of three RAII guards:

• QpdfLockGuard locks one QPDF;

• DualQpdfLockGuard locks two, for example when copying objects between documents;

• MultiQpdfLockGuard locks any number;

• ReachGuard locks what a list of objects needs, including the Pdfs reachable from those that belong to no Pdf, as the bindings do.

The guards that take several locks always take them in the same order (by the address of the registry entry), so two calls that each lock the same pair of Pdfs cannot deadlock. If a later lock throws, they release the ones already taken.

Pdf.lock() is the same lock, exposed to Python through _acquire_lock and _release_lock, so that users can make a sequence of calls atomic.

Python code in pikepdf

A method implemented in Python is a sequence of binding calls, each of which holds the lock only while it runs. A method that changes a Pdf in several dependent steps holds the lock across them: with pdf.lock():, or a decorator that does the same, as in sanitize.py and form.py. One that involves more than one Pdf, such as Pdf.add_pages_from or Page.add_overlay, uses with _OwnerLock((a, b)): instead. _OwnerLock (src/core/qpdf.cpp) locks the owner of each item it is given (a Pdf, an object, or an object helper such as a Page) through MultiQpdfLockGuard, so the locks are taken in the same order as the bindings take them. Nested pdf.lock() blocks would take them in the order the code names them, and two threads copying between the same pair of Pdfs in opposite directions would then deadlock; pikepdf would detect that and raise in one of them.

Methods that only read do not need this, and neither do the with blocks of Pdf.open_metadata() and Pdf.open_outline(): they hold the lock while they read the document and while they write it back, but not in between, since the block runs user code. tests/test_threading_python_layer.py checks each method that holds the lock, by interrupting it part way through and probing the lock from another thread.

How it works

The lock is owner_ (the identifier of the thread holding it, or 0) and depth_. The same algorithm is used on both kinds of build; only the way a free lock is taken differs:

• On a GIL-enabled build, only a thread that holds the GIL reads or writes owner_, so taking a free lock or re-entering a held one is a few plain loads and stores, with no atomic read-modify-write and no system call.

• On a free-threaded build, a free lock is taken with a compare-and-swap of owner_ from 0 to the thread’s identifier.

A thread that finds the lock held by another:

1. increments waiters_;

2. detaches from the interpreter, which releases the GIL if there is one;

3. waits on the condition variable released_ under sleep_mutex_;

4. releases sleep_mutex_ before attaching again, so a thread that is attached (holding the GIL, or holding up a stop-the-world garbage collection) never waits for sleep_mutex_;

5. tries to take the lock again, since another thread may have taken it in between, and goes back to step 2 if that fails.

unlock() notifies the condition variable only when waiters_ is non-zero, so an uncontended unlock costs nothing extra. It briefly takes sleep_mutex_ before notifying, which orders the release after a waiter’s check of owner_ and prevents a lost wake-up. On a free-threaded build, unlock() stores 0 to owner_ and then reads waiters_, while a waiter increments waiters_ and then tries to take the lock, all sequentially consistent: either the waiter finds the lock free or unlock() finds the waiter.

A free-threaded interpreter often runs with the GIL enabled, because importing a module that does not declare free-threading support (lxml, for example) turns it back on. The free-threaded implementation is used either way: which one is compiled depends on the build, not on whether the GIL happens to be enabled.

Why not PyMutex

The lock is deliberately not fair. A released lock goes to whichever thread asks for it next, usually the thread that has just released it and is still running. Earlier versions used nanobind’s ft_mutex, a PyMutex, on free-threaded builds. Once a waiter has waited for a millisecond, PyMutex hands the lock directly to it on release, and the lock then stays held until that thread is scheduled. When several threads take the same lock often, each handover waits for a sleeping thread, and on a machine whose CPUs are all busy that wait can be far longer than the work done under the lock. Threads reading one Pdf through a Python stream ran 4 to 50 times slower than with the current lock when the CPUs were oversubscribed.

Deadlock detection

A thread can hold one Pdf’s lock and then ask for another’s: from a callback that runs while a Pdf is locked, or from nested with pdf.lock() blocks. Two threads doing that in opposite orders would wait for each other for ever.

ReentrantFtMutex::Waiting keeps a global table of which thread is waiting for which lock, under its own std::mutex. A thread about to wait follows the chain from the lock it wants to that lock’s holder, to the lock that holder is waiting for, and so on. If the chain comes back to the thread itself, it throws DeadlockError, which reaches Python as RuntimeError. Unwinding releases the locks the thread holds, and the other threads in the cycle carry on. Only contended locks pay for this check.

Code that cannot throw waits without the check: lock_unchecked(), and QpdfLockGuard(q, std::nothrow). The QPDF deleter, the teardown lock and the return value type caster use these.

Rules

• Every lock() must be made while holding the GIL, or while attached to the interpreter on a free-threaded build. Nothing checks this.

• unlock() must be called by the thread that called lock(). The owner is a thread identifier, so a lock that is not released before its thread exits stays held, and a later thread that is given the same identifier would be let in as if it were the owner.

• Never call Python code while holding one of the std::mutex locks listed above. Python code can release the GIL, run a garbage collection or wait for a per-Pdf lock, any of which can deadlock against a thread that needs the std::mutex.

• A lazy iterator over a qpdf container, returned from a binding, keeps reading the Pdf after the binding has released the lock. Return a snapshot instead.

Objects that belong to no Pdf

A direct Dictionary or Array that has not been added to a Pdf has no owner. All such containers share one lock, unowned_mutex() in src/core/qpdf_lock.h. lock_target(h) says what to lock for an object: its Pdf, the stand-in unowned_owner() for a container that belongs to no Pdf, or nothing for a scalar that belongs to no Pdf, since scalars cannot be changed. LockOwner<QPDFObjectHandle> and the object helpers answer with lock_target(), and the guards take unowned_mutex() when given unowned_owner(). _ObjectList and _ObjectMapping also belong to no Pdf, but each locks itself (see src/core/object_list.cpp).

unowned_mutex() is always taken after any Pdf lock, never before, so that the order of every lock is fixed. Code that holds it must not wait for a Pdf lock. Two places need a Pdf lock while holding it: reading a value out of an unowned container that belongs to a Pdf (ChildLockGuard), and converting such a value to Python. Both lock as QpdfLockGuard::AfterUnowned: if the Pdf lock is not free at once, unowned_mutex() is set aside while the thread waits for it and taken back afterwards. That is only safe because the binding has finished with the container by then. _OwnerLock never takes unowned_mutex(), because it holds its locks while Python code runs.

An object’s owner changes when it is adopted into a Pdf, made indirect, or detached. Each of these holds both the Pdf’s lock and unowned_mutex() while the owner changes (object_adoption.cpp), so a thread that holds the lock of an object’s owner knows the owner cannot change until it lets go. A thread can still read an object’s owner, then wait for that lock while the owner changes. So the bindings (ArgsGuard in locked_bindings.h) and the guards that take an object read the owner again once they hold the lock, and start over if it changed. There is nothing to re-read when no lock was taken: a scalar that belongs to no Pdf never comes to belong to one, because adoption copies it.

The exception is the teardown of a QPDF, which clears the owner of each object it adopted without taking unowned_mutex(). A thread that finds an object with no owner waits for any teardown in progress before it uses it: see the teardown lock below.

An unowned container may hold objects of Pdfs: indirect objects, and containers adopted by a Pdf. A binding that reads through it to them, such as repr(), ==, index(), a name path lookup, is_rectangle or any Page method on an unowned page, needs their locks too, since reading an indirect object can make qpdf parse it from the file. So by default, when an argument of a locked binding is an unowned container, ArgsGuard walks its direct objects (add_reachable_owners() in qpdf_lock.h), stopping at anything that belongs to a Pdf, and collects those Pdfs. It walks with unowned_mutex() held, then releases everything and takes all the locks again through one MultiQpdfLockGuard, in the usual order, and walks again to check that nothing changed. The walk follows only what was built in Python, never a Pdf’s object graph, and an argument that belongs to a Pdf costs nothing extra. ReachGuard does the same for code outside the bindings: repr() itself, and the comparisons of _ObjectList.

A binding that only ever reads at most one child, under a ChildLockGuard, is registered as shallow(f) so that it skips the walk: indexing, item and attribute assignment and deletion, get(), keys(), len(), append(), the typed getters, and properties such as is_indirect. Without that, building a large unowned array with append() would take quadratic time. A binding that is not marked is safe, at worst slow on a large unowned container, so mark one only when it reads no more than one child.

Adoption keeps every object in a Pdf’s graph owned by that Pdf, which is what lets one lock cover them. Before adopting, refuse_mixed_owners walks the direct objects of the value being inserted, the same ones adoption visits, and raises ForeignObjectError if it finds an indirect object or a direct container that belongs to another Pdf. It runs before anything is changed, so a refused insertion leaves the target as it was. Every insertion path goes through adopt_into or make_direct_indirect; the few that adopt after qpdf has inserted the value (page insertion, the stream_dict setter, the Stream constructor) and those that insert several values (extend, slice assignment) call refuse_mixed_owners themselves first. Adoption on read does not check, so that a document already in a bad state can still be read.

The teardown lock

~QPDF clears the owner of each of its objects and then destroys their values. A thread that read one of those objects in between would find no owner, take no lock, and read a value that is being destroyed.

So the QPDF deleter (register_qpdf in src/core/qpdf.cpp) takes the document’s own lock, then calls QpdfRegistry::begin_teardown(), which takes the global teardown lock and increments a counter. It disconnects the adopted objects, deletes the QPDF, unregisters it, and calls end_teardown(). A thread about to use an object with no owner calls wait_for_teardown(), which does nothing unless the counter is non-zero, and otherwise takes and releases the teardown lock.

The teardown lock is re-entrant, so code run by the destroying thread itself, including the destruction of a second Pdf, passes straight through.

The std::mutex locks

These protect small data structures and are never held while Python code runs.

• Registry lock. Held while a QPDF is registered, unregistered or looked up. Lookups return a shared_ptr to the entry, so the caller can release the registry lock before taking the per-Pdf lock.

• Wait table lock. Held while a thread records that it is about to wait for a per-Pdf lock, checks for a cycle, and removes its record.

• Adopted objects lock. qpdf records an object’s owner as a raw pointer, and ~QPDF only clears that pointer on objects still in its cache. A direct object that pikepdf adopted and that was later detached would be left with a dangling pointer. Each QpdfEntry keeps weak references to the objects it adopted, and the deleter disconnects those still alive. This lock protects that list.

• qpdf globals lock. qpdf keeps its global limits in plain variables. This serializes the reads and writes made through pikepdf.settings. It cannot cover qpdf’s own reads while it parses, so limits should be set before threads are started.

• Traversal locks. An ObjectSet or ContentScopes (src/core/traversal.h) can hold objects of several documents, or of none, so it cannot use a per-Pdf lock. Each has a mutex of its own.

Testing

tests/test_threading.py and tests/test_threading_object.py run the same operations from several threads. tests/test_threading_unowned.py covers objects that belong to no Pdf: it probes the shared lock from another thread, checks that operations which read through such an object take the locks of the Pdfs it holds and that the shallow ones do not, and changes such objects while threads add them to a Pdf and take them out again. The frequent_thread_switches fixture lowers the switch interval so that a GIL-enabled interpreter changes threads as often as it can, which makes interleavings in the middle of a qpdf operation likely. run_threads reuses its threads from one call to the next: on a free-threaded build, starting a thread can take a tenth of a second when every CPU is busy, as it is on a CI runner shared by several test workers.

The ThreadSanitizer job (.github/workflows/tsan.yml, and Debugging) runs these tests on a free-threaded build with the GIL disabled. Free-threaded wheel jobs mostly run with the GIL enabled, because lxml turns it back on; set PYTHON_GIL=0 to keep it off.