Anticipate different year values between Athena and Socrata assets #894
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wrridgeway
commented
Sep 5, 2025
Comment on lines
+246
to
+253
| query_dict = { | ||
| year: f"{query} WHERE year = {year}" | ||
| for year in years | ||
| if not pd.isna(year) | ||
| } | ||
|
|
||
| if any(pd.isna(years)): | ||
| query_dict.update({"nan": f"{query} WHERE year IS NULL"}) |
There was a problem hiding this comment.
Making sure we don't exclude data on socrata that has null year values.
wrridgeway
commented
Sep 5, 2025
| input_data[":deleted"] = None | ||
| input_data.loc[input_data["_merge"] == "right_only", ":deleted"] = True | ||
| input_data = input_data.drop(columns=["_merge"]) | ||
| if len(socrata_rows) > 0: |
There was a problem hiding this comment.
When we get new years in athena there won't be any comparable socrata data to gather and this was causing an issue for the permits asset.
wrridgeway
changed the title
wrridgeway
commented
Sep 5, 2025
| .split(",") | ||
| ) | ||
| # row id won't show up here since it's hidden on the open data portal assets | ||
| asset_columns += ["row_id"] |
There was a problem hiding this comment.
This is no longer needed since we exposed row_id on the open data assets.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addressing an issue where we get new years for data in Athena which don't already exist in Socrata, or years are deleted from Athena and the corresponding rows need to be removed from Socrata. Before, this would error out trying to find a
row_idcolumn in an empty dataframe if the year value wasn't present on Socrata, or would leave rows undeleted on Socrata if they had a year value that had been removed from Athena.This hasn't caused us an issue yet since we've almost exclusively been using the script to update the present year (2025) of data for each asset, and when I tested the script updating all years in the past enough time hadn't passed between runs for there to be differences in year values.
This run of the workflow successfully handled
year = 205which was causing the error here (205 had been added to athena but wasn't present in socrata), but ended up failing when athena got nuked.Here's a run I initialized after solving the data is on athena but not on socrata issue. There was still a one row difference between the athena and socrata assets due to a year value that no longer existed in athena. The run found the row on socrata and removed it.