Add referential integrity check to `apply_metadata_in` Dagster job

[eecavanna]

Add real-time referential integrity checking to the apply_metadata_in Dagster job.

Relevant code

nmdc-runtime/nmdc_runtime/site/graphs.py

Lines 129 to 133 in d380b25

	@graph
	def apply_metadata_in():
	outputs = perform_mongo_updates(get_json_in())
	add_output_run_event(outputs)
	materialize_alldocs()

The perform_mongo_updates function uses validate_json under the hood:

nmdc-runtime/nmdc_runtime/site/ops.py

Lines 537 to 539 in d380b25

	rv = validate_json(
	docs, mongo.db
	) # use *exact* same check as /metadata/json:validate

Nowadays, the validate_json function has a Boolean parameter that enables real-time referential integrity checking. See the function's docstring for details.

[eecavanna]

We will add this to the Runtime Q4 squad board initially. We may remove it and document the limitation (and some mitigation tips), depending upon how complex implementation turns out to be.

[eecavanna]

To do:

• Add referential integrity check (this is a one-line change that involves passing an additional argument to the validate_json function)
• Add tests that target the containing perform_mongo_updates "op". Include a test case that confirms that the new referential integrity check detects a would-be-broken reference. Also include a test case that confirms innocuous payloads can still be inserted, like they can today.

How are "ops" normally tasted as part of this repo's test suite? (This is a question I have.)

There's an example of this specific op being tested, in: tests/test_ops/test_ops.py

[eecavanna]

I am removing this ticket from the Runtime Q4 squad board. Since the time we formed the squad, the referential integrity tickets were always ones that we had planned to first (a) gauge the complexity and then (b) decide whether we would be able to accomplish them during Q4.

In the case of this specific ticket, I began implementing it and then learned that the apply_metadata_in function is used for both inserts and updates; so simply adding a kwarg to the validate_json function that it uses under the hood (the kwarg would tell that function to perform referential integrity checks) would not suffice, since that kwarg only works for inserts, not updates.