Failure output order of dictionary keys is alphabetical instead of insertion order

[mcarans]

1. Create a file like this:

class TestDictOrder:
    def test_dict_order(self):
        a = {
            "Existing Hash": "",
            "Existing Modified": "",
            "Existing Size": "",
            "Existing Broken": "",
            "Set Broken": "N",
            "Head Status": "",
            "Head Error": "",
            "Get Status": "",
            "Get Error": "",
            "New ETag": "",
            "ETag Changed": "",
            "New Modified": "",
            "Modified Changed": "",
            "Modified Newer": "",
            "Modified Value": "",
            "New Size": "",
            "Size Changed": "",
            "New Hash": "",
            "Hash Changed": "",
            "Update": "N",
        }
        print(a)
        assert a == {}

2. run pytest with -vv
3. The keys in the output failure message are not in insertion order (the guaranteed order since Python 3.7) but are displayed alphabetically instead (presumably because that's easier to implement?). I think there should be an option to have the output order be insertion order (or it should be the standard output order) as that can be helpful with debugging large and nested dictionaries and is the order when you print the dictionary.

In the output from pytest below, "Left contains 20 more items:" and "Full diff:" are alphabetical instead of insertion order:

1 > pytest -vv test_dict_order.py 
=================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0 -- /home/mcarans/Code/VirtualEnvs/scratch/bin/python
cachedir: .pytest_cache
rootdir: /home/mcarans/Code/scratch
plugins: typeguard-4.4.2
collected 1 item                                                                                                                                                                                                                                                                          

test_dict_order.py::TestDictOrder::test_dict_order FAILED                                                                                                                                                                                                                           [100%]

======================================================================================================================================== FAILURES =========================================================================================================================================
______________________________________________________________________________________________________________________________ TestDictOrder.test_dict_order ______________________________________________________________________________________________________________________________

self = <test_dict_order.TestDictOrder object at 0x7adebb01f890>

    def test_dict_order(self):
        a = {
            "Existing Hash": "",
            "Existing Modified": "",
            "Existing Size": "",
            "Existing Broken": "",
            "Set Broken": "N",
            "Head Status": "",
            "Head Error": "",
            "Get Status": "",
            "Get Error": "",
            "New ETag": "",
            "ETag Changed": "",
            "New Modified": "",
            "Modified Changed": "",
            "Modified Newer": "",
            "Modified Value": "",
            "New Size": "",
            "Size Changed": "",
            "New Hash": "",
            "Hash Changed": "",
            "Update": "N",
        }
        print(a)
>       assert a == {}
E       AssertionError: assert {'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'} == {}
E         
E         Left contains 20 more items:
E         {'ETag Changed': '',
E          'Existing Broken': '',
E          'Existing Hash': '',
E          'Existing Modified': '',
E          'Existing Size': '',
E          'Get Error': '',
E          'Get Status': '',
E          'Hash Changed': '',
E          'Head Error': '',
E          'Head Status': '',
E          'Modified Changed': '',
E          'Modified Newer': '',
E          'Modified Value': '',
E          'New ETag': '',
E          'New Hash': '',
E          'New Modified': '',
E          'New Size': '',
E          'Set Broken': 'N',
E          'Size Changed': '',
E          'Update': 'N'}
E         
E         Full diff:
E         - {}
E         + {
E         +     'ETag Changed': '',
E         +     'Existing Broken': '',
E         +     'Existing Hash': '',
E         +     'Existing Modified': '',
E         +     'Existing Size': '',
E         +     'Get Error': '',
E         +     'Get Status': '',
E         +     'Hash Changed': '',
E         +     'Head Error': '',
E         +     'Head Status': '',
E         +     'Modified Changed': '',
E         +     'Modified Newer': '',
E         +     'Modified Value': '',
E         +     'New ETag': '',
E         +     'New Hash': '',
E         +     'New Modified': '',
E         +     'New Size': '',
E         +     'Set Broken': 'N',
E         +     'Size Changed': '',
E         +     'Update': 'N',
E         + }

test_dict_order.py:26: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------------------------------
{'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'}
================================================================================================================================= short test summary info =================================================================================================================================
FAILED test_dict_order.py::TestDictOrder::test_dict_order - AssertionError: assert {'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'} == {}
  
  Left contains 20 more items:
  {'ETag Changed': '',
   'Existing Broken': '',
   'Existing Hash': '',
   'Existing Modified': '',
   'Existing Size': '',
   'Get Error': '',
   'Get Status': '',
   'Hash Changed': '',
   'Head Error': '',
   'Head Status': '',
   'Modified Changed': '',
   'Modified Newer': '',
   'Modified Value': '',
   'New ETag': '',
   'New Hash': '',
   'New Modified': '',
   'New Size': '',
   'Set Broken': 'N',
   'Size Changed': '',
   'Update': 'N'}
  
  Full diff:
  - {}
  + {
  +     'ETag Changed': '',
  +     'Existing Broken': '',
  +     'Existing Hash': '',
  +     'Existing Modified': '',
  +     'Existing Size': '',
  +     'Get Error': '',
  +     'Get Status': '',
  +     'Hash Changed': '',
  +     'Head Error': '',
  +     'Head Status': '',
  +     'Modified Changed': '',
  +     'Modified Newer': '',
  +     'Modified Value': '',
  +     'New ETag': '',
  +     'New Hash': '',
  +     'New Modified': '',
  +     'New Size': '',
  +     'Set Broken': 'N',
  +     'Size Changed': '',
  +     'Update': 'N',
  + }
==================================================================================================================================== 1 failed in 0.03s ====================================================================================================================================
[

[RonnyPfannschmidt]

Pytest shows a order invariant difference between 2 dicts as dict equality doesn't require the order of items in a dictionary to match

Its not clear what you sre asking for

[mcarans]

@RonnyPfannschmidt Sorry for the lack of clarity. I'm thinking only of the way the differences are displayed rather than changing dict equality.

assert {"c": 3, "d": 4, "b": 2, "a": 1} == {"d": 4, "c": 3}

Currently the output is:

  Common items:
  {'c': 3, 'd': 4}
  Left contains 2 more items:
  {'a': 1, 'b': 2}
  
  Full diff:
    {
  +     'a': 1,
  +     'b': 2,
        'c': 3,
        'd': 4,
    }

I'm suggesting the output would follow the insertion order as far as possible and default to using the order of one of the sides for common items (I have chosen right hand side below) ie.:

  Common items:
  {'d': 4, 'c': 3}
  Left contains 2 more items:
  {'b': 2, 'a': 1}
  
  Full diff:
    {
        'd': 4,
        'c': 3,
  +     'b': 2,
  +     'a': 1,
    }

[RonnyPfannschmidt]

So the request is for the display order to orient on the insert order

Based on the code it might be a mistake that the full diff is printed

In any case we need a deep diff to correctly implement this .

Currently pformat is used to get diff compatible left/right

Unless a correct deep diff we can apply here is avaliable im against implementing this in pytest

[mcarans]

@RonnyPfannschmidt pformat has an option sort_dicts which defaults to True. It is described as "If True, dictionaries will be formatted with their keys sorted, otherwise they will be displayed in insertion order"

Is that what is sorting the keys alphabetically in the pytest output?

If not, I found a deep diff library called deepdiff. For the example above, it gives:

pprint(DeepDiff(a,b))
{'dictionary_item_removed': ["root['b']", "root['a']"]}

[nicoddemus]

Is that what is sorting the keys alphabetically in the pytest output?

pformat is used here:

pytest/src/_pytest/assertion/util.py

Lines 498 to 541 in 9e9633d

	def _compare_eq_dict(
	left: Mapping[Any, Any],
	right: Mapping[Any, Any],
	highlighter: _HighlightFunc,
	verbose: int = 0,
	) -> list[str]:
	explanation: list[str] = []
	set_left = set(left)
	set_right = set(right)
	common = set_left.intersection(set_right)
	same = {k: left[k] for k in common if left[k] == right[k]}
	if same and verbose < 2:
	explanation += [f"Omitting {len(same)} identical items, use -vv to show"]
	elif same:
	explanation += ["Common items:"]
	explanation += highlighter(pprint.pformat(same)).splitlines()
	diff = {k for k in common if left[k] != right[k]}
	if diff:
	explanation += ["Differing items:"]
	for k in diff:
	explanation += [
	highlighter(saferepr({k: left[k]}))
	+ " != "
	+ highlighter(saferepr({k: right[k]}))
	]
	extra_left = set_left - set_right
	len_extra_left = len(extra_left)
	if len_extra_left:
	explanation.append(
	f"Left contains {len_extra_left} more item{'' if len_extra_left == 1 else 's'}:"
	)
	explanation.extend(
	highlighter(pprint.pformat({k: left[k] for k in extra_left})).splitlines()
	)
	extra_right = set_right - set_left
	len_extra_right = len(extra_right)
	if len_extra_right:
	explanation.append(
	f"Right contains {len_extra_right} more item{'' if len_extra_right == 1 else 's'}:"
	)
	explanation.extend(
	highlighter(pprint.pformat({k: right[k] for k in extra_right})).splitlines()
	)
	return explanation

Looks like it would be just a matter of passing sort_dicts=False to those pformat calls.

[RonnyPfannschmidt]

We will have to ensure compatible dict ordering in nested structures to ensure minimal diffs

Order matching dicts in modern python make this easier but its still a caveat

[github-actions]

This issue is stale because it has the status: needs information label and requested follow-up information was not provided for 14 days.

[mcarans]

What information do I need to provide to resolve the stale label?