Race and deadlock in polymorphic multiobject through-model registration

[bctiemann]

Plugin Version

main (unreleased, found in development past v0.6.0)

NetBox Version

v4.6.6

Python Version

3.12

Steps to Reproduce

While investigating #640, I found two independent, concretely reproducible concurrency bugs in MultiObjectFieldType.create_polymorphic_m2m_table() (called once, when a polymorphic multiobject field is first created). Neither reproduces #640's specific reported symptom (see the discussion on PR #648) -- these are separate bugs in the same function, verified with deterministic regression tests added in PR #648:

1. PolymorphicMultiObjectConcurrencyTestCase.test_forced_registration_interleaving_stays_consistent -- forces a concurrent get_model(no_cache=True) call to land inside the writer's build-then-repoint window (bug 1 below), via a mocked apps.register_model() hook.
2. PolymorphicMultiObjectConcurrencyTestCase.test_concurrent_double_submit_does_not_deadlock -- two threads calling CustomObjectTypeField.objects.create() for the identical (custom_object_type, name) at once (bug 2 below).

Bug 1: registration-before-repoint race against a concurrent reader

create_polymorphic_m2m_table() built and registered a through-model class with Django's app registry, and only afterward repointed its source FK at the caller's model class -- all without holding CustomObjectType._global_lock. CustomObjectType.get_model()'s own reuse-or-create check for polymorphic through models (_after_model_generation()) is lock-protected on its own side. A concurrent get_model(no_cache=True) call could land in the writer's build-then-repoint window, find the through model already registered, and repoint source at its own (different, but table-equivalent) model instance instead -- leaving the through's FK and whatever get_model() subsequently caches pointing at two different Python classes for the same table.

Bug 2: lock-ordering deadlock on a double-submit

Fixing bug 1 by holding _global_lock across the whole create_polymorphic_m2m_table() call (including the table-existence probe and schema_editor.create_model() DDL) introduces a different real deadlock: two threads racing to create the same field (e.g. a doubly-clicked "save" button, or a retried request) each build+register a through model for the same physical table before either knows which one will win the (name, custom_object_type) UniqueConstraint. Whichever thread's schema_editor.create_model() runs second blocks at the Postgres level waiting on the first thread's uncommitted CREATE TABLE (same table name) to resolve -- but the first thread's own CustomObjectTypeField.save() needs _global_lock again in clear_model_cache() before it can commit and release that wait.

Expected Behavior

A polymorphic multiobject field's through-model registration should stay consistent under concurrent field creation and reads, and concurrent field-creation attempts should not be able to deadlock.

Observed Behavior

Bug 1: the through model's source FK and the model class get_model() subsequently returns/caches can point at two different Python classes for the same table. This class-identity mismatch produces ValueError: Cannot query "X": Must be "TableYModel" instance. or a RecursionError when later deleting an object through the affected relation.

Bug 2: confirmed live via pg_stat_activity -- one thread idle-in-transaction waiting on _global_lock (needed again in clear_model_cache() to commit), the other actively blocked on Lock/transactionid executing CREATE TABLE for the same physical table, waiting on the first thread's uncommitted transaction. Neither thread can make progress; the request hangs.

Relationship to #640

#640 reports a different, more severe symptom (a deterministic FieldDoesNotExist on a fresh, single, non-concurrent process) that I was unable to reproduce on current main -- see the investigation notes on PR #648. This issue is scoped narrowly to the two concurrency bugs above, which PR #648 fixes with regression coverage. #640 remains open pending further reproduction details.