Add contest export script #3039
Conversation
eldering
force-pushed
the
add-contest-export
branch
from
f55ed07 to
d13e643
Compare
misc-tools/export-contest.in
Outdated
| recurse_find_files(data[0], store_path, default_name) | ||
| else: | ||
| for i, item in enumerate(data): | ||
| recurse_find_files(item, store_path, f"{default_name}.{i}") |
There was a problem hiding this comment.
Initially we pass '' in default_name, so this becomes .{i}? (same with .{key} below)
There was a problem hiding this comment.
Hmm, I think you're right. The idea is that this builds up a default file name while recursively descending into the object's structure. So at the base level you always find the object itself as a dict, and that means that at line 94 default_name is set to the property key, but as you say with a prefix .. So for example for multiple JPEG files in the array under contest.logo, their default names would become .logo.0.jpg, .logo.1.jpg. So that first . must indeed be stripped.
There was a problem hiding this comment.
Fixed by using default argument default_name = None and adding . based on that.
misc-tools/export-contest.in
Outdated
|
|
||
| if ext == '.json': | ||
| data = json.loads(data) | ||
| store_path = name |
There was a problem hiding this comment.
Why use store_path here? Can't we just use name below?
There was a problem hiding this comment.
Yeah, I think so. I guess this is a left-over of previous refactoring. It's a bit annoying that there's too many similar concepts floating around: name of the endpoint, endpoint path in the URL, path to store the downloaded endpoint data to on disk, and filename on disk.
There was a problem hiding this comment.
Replaced store_path by name and documented the parameters better.
…via CLI Fetching the event feed from the API via `internalApiRequest` wasn't working since it returns a StreamedResponse. Fix this by only using an output buffer in `internalApiRequest` and only calling `ob_flush` from `getEventFeedAction` if the top-level output buffer allows it. Also return contents from `internalApiRequest` without JSON decoding it, since the event feed endpoint is NDJSON, *not* JSON. Explicitly decode the JSON at the other places calling internalApiRequest`.
This will be used by the contest export script for downloading the event feed (NDJSON) and other files. Also improve error reporting when an internal API call fails.
Also initialize `domjudge_api_url = None` and simply call the API via CLI if `domjudge_api_url` is unset.
Capture all function parameters and pass these as is to the function itself again, just now with the global `ca_check = False`.
This follows the specifications correctly, but beware that specifying a base URL that doesn't end with a `/` leads to strange results, see #2378 Replace `exit(1)` by an exception so this can be caught also if this code runs asynchronously in a thread or separate process.
This is needed for binary content that otherwise gets mangled when fetched through the CLI. Also force all but the `endpoint` and `method` parameters to `do_api_request` to be named parameters to prevent mistakes.
eldering
force-pushed
the
add-contest-export
branch
from
adb08ee to
71e854b
Compare
This gives more flexibility for extension in the future and leaves the default usage without arguments unchanged.
eldering
force-pushed
the
add-contest-export
branch
from
71e854b to
e93273e
Compare
Closes #2726