Non-obvious plugin load order of the root conftest w.r.t. pytest_plugins

[Anton3]

What's the problem this feature will solve?

We in userver like to create test libraries with mocks and utilities packed in pytest plugins.

In a service's root conftest.py we list the used plugins with pytest_plugins variable, then throw in some service-wide fixtures. It is also natural to put overrides of global fixtures there (where needed), this works as expected, in the spirit that a root conftest works as the latest plugin.

We've added a feature using pytest_plugin_registered that allows any plugin to provide a piece of JSON config by putting USERVER_CONFIG_PART global variable in the plugin. Our config plugin essentially walks through all the plugins using pytest_plugin_registered and merges their JSON patches, allowing the later plugins to override the config parts of the earlier ones.

One issue we've encountered is, suddenly conftest.py does not have priority over other plugins! Example:

# foo_lib_plugin.py
USERVER_CONFIG_PART = {
    'should-frobnicate': False,
}
# ...

# root conftest.py
pytest_plugins = [
    'config',
    'foo_lib_plugin',
    'bar_lib_plugin',
]
USERVER_CONFIG_PART = {
    'should-frobnicate': True,
}

In the end, the merged config looks like:

{
    "should-frobnicate": false
}

This is highly unexpected and has cost me a whole day of head-scratching.
Why is conftest.py not applied as "the latest plugin" here?

---

One possible fix in this case is to extract all the overrides into a separate plugin:

# foo_lib_plugin.py
USERVER_CONFIG_PART = {
    'should-frobnicate': False,
}
# ...

# my_service_plugin.py
USERVER_CONFIG_PART = {
    'should-frobnicate': True,
}

# root conftest.py
pytest_plugins = [
    'config',
    'foo_lib_plugin',
    'bar_lib_plugin',
    'my_service_plugin',
]

---

Now, thinking more generally, I can understand why this is the way it is. A test session can have 1 or more "root" conftest.py files (in the root test directory or higher), all of them are treated as fully-fledged plugins. As a plugin, a conftest.py can load other plugins using pytest_plugins or pytest_configure or in some other ways. In general, we'll only know the whole list of plugin dependencies after the plugin has been applied. So for consistency, let's register the conftest plugin completely, get, in particular, the list of requested plugins, then register them.

This is different to how our team thinks of conftest.py. We essentially think of the root conftest as playing two roles at the same time:

1. The "mother of all plugins", contains the list of all plugins needed for the service, we do not use other plugin sources.
2. The latest, most specific plugin that can be used for service-specific fixtures and overrides.

Unfortunately, these roles do not live along under the current pytest rules, as the plugin ordering becomes inconsistent according to this mental model.

Describe the solution you'd like

This quirk can at least be documented. While loading stages are somewhat well described, plugin ordering guarantees w.r.t. pytest_plugin_registered can be documented more thoroughly, especially with respect to pytest_plugins and conftest.py.

Still, despite not being a bug, I believe this is an inconsistency issue. A conftest.py can be thought of as "the last plugin" in the context of fixtures, but a deep dive breaks this simple mental model.

I don't have a good solution at hand, but would be happy to hear one.

[Anton3]

I've still not figured out completely why that happens. The weird order might be because config itself, which contains the pytest_plugin_registered hook, is loaded by the conftest, which also contains a USERVER_CONFIG_PART definition, which needs to be consumed by the config plugin.

[Anton3]

Reading the docs, it seems that for each plugin, pytest_plugin_registered is called before any of its hooks or pytest_plugins are processed. Because the conftest is seen first, it becomes the first plugin. Except that it behaves differently w.r.t. fixtures, which is by design, but somewhat inconsistent.

[RonnyPfannschmidt]

hoks are managed by pluggy, fixtures are managed by pytest
we explicitly order fixtures by location/conftest relation/test file relation to allow more controlled overrides/oriorities

we currently dont have such a ordering system for pluggy
we wont have such a ordering system in pluggy in near future - its something to consider after we add global async support to pluggy and pytest (which is very close)

[Anton3]

A solution could be as follows: pytest_plugins should be processed before the plugin itself. This is natural for both conftests and normal plugins.

The implementation is as simple as moving these two lines up a bit and computing plugin_name manually before super().register.

[RonnyPfannschmidt]

The link goes to a random disfuncional russian URL

As for changing the order that's potentially a breaking change than messes up other users code

[Anton3]

Fixed the link, sorry 🤦
This change might need to target the next major version bump. Still might be something to consider, it's important to be able to order plugins meaningfully.

[RonnyPfannschmidt]

i have at least one codebase where this change trigger errors and rightfully so

[Anton3]

Here are some extra details I discovered looking at the implementation of PytestPluginManager.register and how pytest_configure is called.

Of course, there is this inconsistency:

• If plugin A lists plugin B in pytest_plugins (A depends on B), pytest always orders them as A, B. pytest_configure, pytest_plugin_registered and other hooks are called in that order. (As I said above, this is not what I want)
• However, if plugin C is injected before A and depends on B, then suddenly B is before A (inconsistency)

There is one extra curious inconsistency in addition to that. If plugin A registers plugin B from its pytest_configure, and plugin B registers plugin C from its pytest_configure (all inside _do_configure), then the following will happen:

• Plugin A is registered during plugins collection stage
• A.pytest_configure
  • PytestPluginManager.register(B)
    • pluggy.PluginManager.register(B)
      • B.pytest_configure
        • PytestPluginManager.register(C)
          • pluggy.PluginManager.register(C)
            • C.pytest_configure
          • pytest_plugin_registered with C
    • pytest_plugin_registered with B

So pytest_plugin_registered of registered plugins will run for these plugins in order A, C, B. But pytest_plugin_registered of future plugins will run for these plugins in their order of registration, that is: A, B, C.

---

So as you say it seems some plugins already rely on the current order. One way out is to:

1. Add pytest_pre_plugins (TODO: bikeshed the name) that processes dependency plugins before the current one.
2. Add pytest_pre_register(???config???) (TODO: bikeshed the name), which is a hook that is called right before the plugin is registered.
   • It can be used to register dependency plugins before the current plugin is considered registered
   • That hook will likely have to be called in PytestPluginManager manually without the help of pluggy
   • That means manual calls of hook tracing
   • Because pluggy does not yet know that the current plugin is registered, dependency cycles can happen, which should be detected and banned (e.g. by keeping a set of plugins for which pytest_pre_register is in the process of being called)
     • That can happen with pytest_pre_plugins as well
   • The hook is similar to pytest_load_initial_conftests except that
     • It's available for all plugins, not only for early ones
     • Because it may be called before or during configure stage, parser is not available
     • This hook will be called pre-pluggy.PluginManager.register, not during it

Although it's not perfectly teachable that for a sane ordering of plugins one has to use those new entities instead of well-known ones.

---

Looking into pytest_pre_register further, it cannot accept config, as this hook is called for early-loaded plugins as well. It can accept (early_config, args), which is not very convenient.

pytest_pre_plugins, on the other hand, will be limited to module plugins. Although, if pytest_pre_register fails, then we can support pytest_plugins and pytest_pre_plugins attrs for class plugins as well.

So at least for guaranteed dependencies, and most are, sane plugin order is achievable.

[RonnyPfannschmidt]

This sounds like a case for something more nuanced than tryfirst/trylast

[RonnyPfannschmidt]

One detail we need to document better is that nobody should register plugins in the registrd hook

Its what pytest configure is for

[Anton3]

This sounds like a case for something more nuanced than tryfirst/trylast

In that case, pytest_plugin_registered will still run in weird order, and we rely on it to collect variables instead of proper pluggy hooks (although we maybe shouldn't)

[RonnyPfannschmidt]

consider plugin registred as a notification - unless its a weird hack its not supposed to be used for registring plugins at the same time

[Anton3]

I mean, tryfirst/trylast apply to pluggy hook functions, but for our hooks we instead use the following scheme:

def pytest_plugin_registered(plugin):
    if part := getattr(plugin, USERVER_CONFIG_PART, None):
        _config_parts.append(part)

Although you could say that we should switch to def userver_config_part() hook instead, to which the hypothetical advanced tryfirst/trylast would be applicable.

[RonnyPfannschmidt]

please outline your mechanism to avoid staying in the X/Y problem