-
-
Notifications
You must be signed in to change notification settings - Fork 4.4k
Require two approvals for model_context changes #14954
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Require two approvals for model_context changes #14954
Conversation
Co-authored-by: teddy <teddy@berri.ai>
Cursor Agent can help with this pull request. Just |
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Co-authored-by: teddy <teddy@berri.ai>
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout code | ||
uses: actions/checkout@v4 | ||
|
||
- name: Validate JSON Files | ||
run: | | ||
echo "🔍 Validating JSON structure for model context files..." | ||
|
||
# Validate main model context file | ||
if [ -f "model_prices_and_context_window.json" ]; then | ||
echo "Validating model_prices_and_context_window.json..." | ||
python -c " | ||
import json | ||
import sys | ||
|
||
try: | ||
with open('model_prices_and_context_window.json', 'r') as f: | ||
data = json.load(f) | ||
print('✅ model_prices_and_context_window.json is valid JSON') | ||
|
||
# Basic structure validation | ||
if isinstance(data, dict): | ||
print('✅ File contains a valid JSON object') | ||
|
||
# Check for required fields in model entries | ||
required_fields = ['litellm_provider', 'mode'] | ||
optional_fields = ['max_tokens', 'input_cost_per_token', 'output_cost_per_token'] | ||
|
||
for model_name, model_data in data.items(): | ||
if isinstance(model_data, dict): | ||
# Check for required fields | ||
missing_required = [field for field in required_fields if field not in model_data] | ||
if missing_required: | ||
print(f'⚠️ Model {model_name} missing required fields: {missing_required}') | ||
|
||
# Check for at least one cost field | ||
cost_fields = [field for field in optional_fields if field in model_data] | ||
if not cost_fields and 'mode' in model_data and model_data['mode'] != 'image_generation': | ||
print(f'⚠️ Model {model_name} has no cost information') | ||
else: | ||
print(f'❌ Model {model_name} data is not a valid object') | ||
else: | ||
print('❌ File is not a valid JSON object') | ||
sys.exit(1) | ||
|
||
except json.JSONDecodeError as e: | ||
print(f'❌ JSON validation failed: {e}') | ||
sys.exit(1) | ||
except Exception as e: | ||
print(f'❌ Validation error: {e}') | ||
sys.exit(1) | ||
" | ||
fi | ||
|
||
# Validate other model context files | ||
for file in $(find . -name "*model_context*.json" -o -name "*context_window*.json" | grep -v node_modules | grep -v ".git"); do | ||
if [ -f "$file" ]; then | ||
echo "Validating $file..." | ||
python -c " | ||
import json | ||
import sys | ||
|
||
try: | ||
with open('$file', 'r') as f: | ||
data = json.load(f) | ||
print('✅ $file is valid JSON') | ||
except json.JSONDecodeError as e: | ||
print(f'❌ $file JSON validation failed: {e}') | ||
sys.exit(1) | ||
except Exception as e: | ||
print(f'❌ $file validation error: {e}') | ||
sys.exit(1) | ||
" | ||
fi | ||
done | ||
|
||
echo "✅ All JSON files are valid" | ||
|
||
- name: Check File Size | ||
run: | | ||
echo "🔍 Checking file sizes..." | ||
|
||
if [ -f "model_prices_and_context_window.json" ]; then | ||
FILE_SIZE=$(wc -c < model_prices_and_context_window.json) | ||
echo "model_prices_and_context_window.json size: $FILE_SIZE bytes" | ||
|
||
# Warn if file is unusually large (over 10MB) | ||
if [ "$FILE_SIZE" -gt 10485760 ]; then | ||
echo "⚠️ Warning: model_prices_and_context_window.json is larger than 10MB" | ||
fi | ||
fi No newline at end of file |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 24 days ago
How to, in general terms, fix the problem:
Explicitly declare the workflow/job's least required permissions using a permissions:
block. For this workflow, only read-access to repository contents is required since actions include checking out and reading files only.
Detailed fix with location:
Add a permissions
block to the root of the workflow file (.github/workflows/json-structure-validation.yml
), immediately after the name:
declaration and before on:
. This block should specify contents: read
. This will apply to all jobs unless overridden.
What is needed:
No imports or additional definitions are necessary; only a single YAML edit is required.
-
Copy modified lines R2-R3
@@ -1,4 +1,6 @@ | ||
name: JSON Structure Validation | ||
permissions: | ||
contents: read | ||
|
||
on: | ||
pull_request: |
Co-authored-by: teddy <teddy@berri.ai>
Title
Implement 2-Approval and Source Citation Requirement for Model Context Changes
Relevant issues
N/A
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/
directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit
Type
🆕 New Feature
Changes
This PR introduces a new governance rule for changes to model context JSON files (e.g.,
model_prices_and_context_window.json
). The primary goal is to ensure data accuracy, maintain traceability, and prevent accidental modifications to critical configuration.Key changes include:
.github/workflows/model_context_approval.yml
): Enforces a requirement of at least 2 approvals for any Pull Request modifying files matching**/model_context*.json
or**/context_window*.json
. It also validates JSON structure and warns about missing source citations in the PR description..github/scripts/validate_model_context_changes.py
): Prevents commits to model context files if the commit message or PR description does not include clear source citations, and validates JSON structure..github/workflows/auto_update_price_and_context_window.yml
): Ensures automated updates tomodel_prices_and_context_window.json
include proper source citations.docs/MODEL_CONTEXT_APPROVAL.md
andCONTRIBUTING.md
): Comprehensive documentation explaining the new approval process, citation guidelines, automated checks, and troubleshooting.