Skip to content

Conversation

berri-teddy
Copy link
Collaborator

Title

Implement 2-Approval and Source Citation Requirement for Model Context Changes

Relevant issues

N/A

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • I have added a screenshot of my new test passing locally
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature

Changes

This PR introduces a new governance rule for changes to model context JSON files (e.g., model_prices_and_context_window.json). The primary goal is to ensure data accuracy, maintain traceability, and prevent accidental modifications to critical configuration.

Key changes include:

  • New GitHub Workflow (.github/workflows/model_context_approval.yml): Enforces a requirement of at least 2 approvals for any Pull Request modifying files matching **/model_context*.json or **/context_window*.json. It also validates JSON structure and warns about missing source citations in the PR description.
  • New Pre-commit Hook (.github/scripts/validate_model_context_changes.py): Prevents commits to model context files if the commit message or PR description does not include clear source citations, and validates JSON structure.
  • Updated Auto-update Workflow (.github/workflows/auto_update_price_and_context_window.yml): Ensures automated updates to model_prices_and_context_window.json include proper source citations.
  • Documentation (docs/MODEL_CONTEXT_APPROVAL.md and CONTRIBUTING.md): Comprehensive documentation explaining the new approval process, citation guidelines, automated checks, and troubleshooting.

Open in Cursor Open in Web

Co-authored-by: teddy <teddy@berri.ai>
Copy link

cursor bot commented Sep 26, 2025

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
Learn more about Cursor Agents

Copy link

vercel bot commented Sep 26, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
litellm Ready Ready Preview Comment Sep 26, 2025 7:45pm

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Comment on lines +12 to +103
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Validate JSON Files
run: |
echo "🔍 Validating JSON structure for model context files..."

# Validate main model context file
if [ -f "model_prices_and_context_window.json" ]; then
echo "Validating model_prices_and_context_window.json..."
python -c "
import json
import sys

try:
with open('model_prices_and_context_window.json', 'r') as f:
data = json.load(f)
print('✅ model_prices_and_context_window.json is valid JSON')

# Basic structure validation
if isinstance(data, dict):
print('✅ File contains a valid JSON object')

# Check for required fields in model entries
required_fields = ['litellm_provider', 'mode']
optional_fields = ['max_tokens', 'input_cost_per_token', 'output_cost_per_token']

for model_name, model_data in data.items():
if isinstance(model_data, dict):
# Check for required fields
missing_required = [field for field in required_fields if field not in model_data]
if missing_required:
print(f'⚠️ Model {model_name} missing required fields: {missing_required}')

# Check for at least one cost field
cost_fields = [field for field in optional_fields if field in model_data]
if not cost_fields and 'mode' in model_data and model_data['mode'] != 'image_generation':
print(f'⚠️ Model {model_name} has no cost information')
else:
print(f'❌ Model {model_name} data is not a valid object')
else:
print('❌ File is not a valid JSON object')
sys.exit(1)

except json.JSONDecodeError as e:
print(f'❌ JSON validation failed: {e}')
sys.exit(1)
except Exception as e:
print(f'❌ Validation error: {e}')
sys.exit(1)
"
fi

# Validate other model context files
for file in $(find . -name "*model_context*.json" -o -name "*context_window*.json" | grep -v node_modules | grep -v ".git"); do
if [ -f "$file" ]; then
echo "Validating $file..."
python -c "
import json
import sys

try:
with open('$file', 'r') as f:
data = json.load(f)
print('✅ $file is valid JSON')
except json.JSONDecodeError as e:
print(f'❌ $file JSON validation failed: {e}')
sys.exit(1)
except Exception as e:
print(f'❌ $file validation error: {e}')
sys.exit(1)
"
fi
done

echo "✅ All JSON files are valid"

- name: Check File Size
run: |
echo "🔍 Checking file sizes..."

if [ -f "model_prices_and_context_window.json" ]; then
FILE_SIZE=$(wc -c < model_prices_and_context_window.json)
echo "model_prices_and_context_window.json size: $FILE_SIZE bytes"

# Warn if file is unusually large (over 10MB)
if [ "$FILE_SIZE" -gt 10485760 ]; then
echo "⚠️ Warning: model_prices_and_context_window.json is larger than 10MB"
fi
fi No newline at end of file

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 24 days ago

How to, in general terms, fix the problem:
Explicitly declare the workflow/job's least required permissions using a permissions: block. For this workflow, only read-access to repository contents is required since actions include checking out and reading files only.

Detailed fix with location:
Add a permissions block to the root of the workflow file (.github/workflows/json-structure-validation.yml), immediately after the name: declaration and before on:. This block should specify contents: read. This will apply to all jobs unless overridden.

What is needed:
No imports or additional definitions are necessary; only a single YAML edit is required.


Suggested changeset 1
.github/workflows/json-structure-validation.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/json-structure-validation.yml b/.github/workflows/json-structure-validation.yml
--- a/.github/workflows/json-structure-validation.yml
+++ b/.github/workflows/json-structure-validation.yml
@@ -1,4 +1,6 @@
 name: JSON Structure Validation
+permissions:
+  contents: read
 
 on:
   pull_request:
EOF
@@ -1,4 +1,6 @@
name: JSON Structure Validation
permissions:
contents: read

on:
pull_request:
Copilot is powered by AI and may make mistakes. Always verify output.
Co-authored-by: teddy <teddy@berri.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants