-
Notifications
You must be signed in to change notification settings - Fork 943
feat(ui): add Token Estimator link to footer #337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
HmbleCreator
wants to merge
21
commits into
coderamp-labs:main
Choose a base branch
from
HmbleCreator:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
a3eef1a
feat(ui): add Token Estimator link to footer
HmbleCreator 6247f92
feat: Add token estimator documentation page and simplify footer link…
HmbleCreator cef47a2
feat: Add token estimator documentation page and simplify footer link…
HmbleCreator 48328ab
feat: use Jinja template for token estimator API docs
HmbleCreator 2ed86f6
Merge branch 'main' into main
filipchristiansen 3f0c1a5
Merge branch 'main' into main
HmbleCreator 98e5c45
Refactor token count API and routing, enforce API-only for /api/token…
HmbleCreator 5c271b5
Merge branch 'main' of https://github.com/HmbleCreator/gitingest
HmbleCreator 41ecea9
Merge branch 'main' into main
filipchristiansen 92b8660
Merge branch 'main' into main
filipchristiansen e1648c4
Merge branch 'main' into main
HmbleCreator 75daa98
Merge branch 'main' into main
filipchristiansen 5b282b5
Merge branch 'main' into main
HmbleCreator 806d2d5
Merge branch 'main' into main
HmbleCreator 79eb882
Merge branch 'main' into main
HmbleCreator 7852fde
Merge branch 'cyclotruc:main' into main
HmbleCreator b802ccf
Merge branch 'main' into main
filipchristiansen e1c5859
Merge branch 'main' into main
HmbleCreator 9e69ce8
Merge branch 'main' into main
HmbleCreator 260cb9b
Merge branch 'main' into main
HmbleCreator 99bd52a
Merge branch 'main' into main
HmbleCreator File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,37 +1,161 @@ | ||
"""Module defining the FastAPI router for the home page of the application.""" | ||
|
||
from fastapi import APIRouter, Request | ||
from fastapi import APIRouter, Depends, Request, Form, HTTPException | ||
from fastapi.responses import HTMLResponse | ||
from fastapi.templating import Jinja2Templates | ||
from autotiktokenizer import AutoTikTokenizer | ||
import tiktoken | ||
from typing import Optional | ||
|
||
from gitingest.utils.compat_typing import Annotated | ||
from server.models import QueryForm | ||
from server.query_processor import process_query | ||
from server.server_config import EXAMPLE_REPOS, get_version_info, templates | ||
from server.server_utils import limiter | ||
from pydantic import BaseModel, Field | ||
|
||
router = APIRouter() | ||
templates = Jinja2Templates(directory="server/templates") | ||
|
||
SUPPORTED_MODELS = { | ||
'GPT-2 (OpenAI)': 'openai-community/gpt2', | ||
'GPT-3 (OpenAI)': 'openai-community/gpt2', | ||
'GPT-3.5 (OpenAI)': 'openai-community/gpt2', | ||
'GPT-3.5-turbo (OpenAI)': 'openai-community/gpt2', | ||
'GPT-4 (OpenAI)': 'openai-community/gpt2', | ||
'Claude (approximate, uses GPT-2)': 'openai-community/gpt2', | ||
'Gemini (approximate, uses T5)': 't5-base', | ||
'Llama-2 (Meta)': 'meta-llama/Llama-2-7b-hf', | ||
'Llama-3 (Meta)': 'meta-llama/Meta-Llama-3-8B', | ||
'Mistral-7B (MistralAI)': 'mistralai/Mistral-7B-v0.1', | ||
'Mixtral-8x7B (MistralAI)': 'mistralai/Mixtral-8x7B-v0.1', | ||
'Phi-3-mini (Microsoft)': 'microsoft/phi-3-mini-4k-instruct', | ||
'Gemma-2B (Google)': 'google/gemma-2b', | ||
'Qwen2-7B (Alibaba)': 'Qwen/Qwen2-7B', | ||
'Yi-34B (01.AI)': '01-ai/Yi-34B-Chat', | ||
'Falcon-7B (TII)': 'tiiuae/falcon-7b', | ||
'MPT-7B (MosaicML)': 'mosaicml/mpt-7b', | ||
'Baichuan-7B (Baichuan)': 'baichuan-inc/Baichuan-7B', | ||
'XLM-RoBERTa-base (Facebook)': 'xlm-roberta-base', | ||
'RoBERTa-base (Facebook)': 'roberta-base', | ||
'DistilBERT-base-uncased': 'distilbert-base-uncased', | ||
'GPT-Neo-1.3B (EleutherAI)': 'EleutherAI/gpt-neo-1.3B', | ||
'GPT-J-6B (EleutherAI)': 'EleutherAI/gpt-j-6B', | ||
'GPT-Bloom-560m (BigScience)': 'bigscience/bloom-560m', | ||
'BERT-base-uncased': 'bert-base-uncased', | ||
'T5-base': 't5-base', | ||
} | ||
|
||
@router.get("/", response_class=HTMLResponse, include_in_schema=False) | ||
async def home(request: Request) -> HTMLResponse: | ||
"""Render the home page with example repositories and default parameters. | ||
|
||
This endpoint serves the home page of the application, rendering the ``index.jinja`` template | ||
and providing it with a list of example repositories and default file size values. | ||
|
||
Parameters | ||
---------- | ||
request : Request | ||
The incoming request object, which provides context for rendering the response. | ||
def get_tokenizer(model_id): | ||
return AutoTikTokenizer.from_pretrained(model_id) | ||
|
||
Returns | ||
------- | ||
HTMLResponse | ||
An HTML response containing the rendered home page template, with example repositories | ||
and other default parameters such as file size. | ||
def count_tokens(input_text, model_id): | ||
if model_id == 'openai-community/gpt2': | ||
enc = tiktoken.encoding_for_model("gpt-3.5-turbo") | ||
return len(enc.encode(input_text)) | ||
else: | ||
tokenizer = AutoTikTokenizer.from_pretrained(model_id) | ||
return len(tokenizer.encode(input_text)) | ||
|
||
""" | ||
@router.get("/", response_class=HTMLResponse, include_in_schema=False) | ||
async def home(request: Request) -> HTMLResponse: | ||
"""Render the home page with example repositories and default parameters.""" | ||
context = { | ||
"request": request, | ||
"examples": EXAMPLE_REPOS, | ||
"default_max_file_size": 243, | ||
} | ||
context.update(get_version_info()) | ||
|
||
return templates.TemplateResponse("index.jinja", context) | ||
|
||
@router.post("/", response_class=HTMLResponse) | ||
@limiter.limit("10/minute") | ||
async def index_post(request: Request, form: Annotated[QueryForm, Depends(QueryForm.as_form)]) -> HTMLResponse: | ||
resolved_token = form.token if form.token else None | ||
return await process_query( | ||
request, | ||
input_text=form.input_text, | ||
slider_position=form.max_file_size, | ||
pattern_type=form.pattern_type, | ||
pattern=form.pattern, | ||
is_index=True, | ||
token=resolved_token, | ||
) | ||
|
||
class TokenCountRequest(BaseModel): | ||
input_text: str = Field(..., description="The text to count tokens for") | ||
model_id: str = Field(default="openai-community/gpt2", description="The model ID to use for tokenization") | ||
|
||
class TokenCountResponse(BaseModel): | ||
token_count: int = Field(..., description="Number of tokens in the input text") | ||
model_id: str = Field(..., description="Model ID used for tokenization") | ||
character_count: int = Field(..., description="Number of characters in the input text") | ||
|
||
@router.post("/api/tokencount", response_model=TokenCountResponse) | ||
async def api_token_count( | ||
request: Optional[TokenCountRequest] = None, | ||
input_text: str = Form(None), | ||
model_id: str = Form(default="openai-community/gpt2"), | ||
): | ||
if request: | ||
text = request.input_text | ||
model = request.model_id | ||
else: | ||
text = input_text | ||
model = model_id | ||
|
||
if not text or not text.strip(): | ||
raise HTTPException(status_code=400, detail="Input text cannot be empty") | ||
|
||
if model not in SUPPORTED_MODELS.values(): | ||
raise HTTPException( | ||
status_code=400, | ||
detail=f"Unsupported model ID. Must be one of: {', '.join(SUPPORTED_MODELS.values())}" | ||
) | ||
|
||
try: | ||
token_count = count_tokens(text, model) | ||
return TokenCountResponse( | ||
token_count=token_count, | ||
model_id=model, | ||
character_count=len(text) | ||
) | ||
except Exception as e: | ||
raise HTTPException(status_code=500, detail=str(e)) | ||
|
||
@router.get("/tokencount", response_class=HTMLResponse) | ||
async def tokencount_ui(request: Request): | ||
return templates.TemplateResponse( | ||
"tokencount.jinja", | ||
HmbleCreator marked this conversation as resolved.
Show resolved
Hide resolved
|
||
{"request": request, "supported_models": SUPPORTED_MODELS, "input_text": "", "model_id": "openai-community/gpt2", "result": None, "error": None} | ||
) | ||
|
||
@router.post("/tokencount", response_class=HTMLResponse) | ||
async def tokencount_post(request: Request, input_text: str = Form(...), model_id: str = Form("openai-community/gpt2")): | ||
error = None | ||
result = None | ||
if not input_text or not input_text.strip(): | ||
error = "Input text cannot be empty." | ||
elif model_id not in SUPPORTED_MODELS.values(): | ||
error = f"Unsupported model ID. Must be one of: {', '.join(SUPPORTED_MODELS.values())}" | ||
else: | ||
try: | ||
token_count = count_tokens(input_text, model_id) | ||
result = { | ||
"token_count": token_count, | ||
"model_id": model_id, | ||
"character_count": len(input_text) | ||
} | ||
except Exception as e: | ||
error = str(e) | ||
return templates.TemplateResponse( | ||
"tokencount.jinja", | ||
{ | ||
"request": request, | ||
"supported_models": SUPPORTED_MODELS, | ||
"input_text": input_text, | ||
"model_id": model_id, | ||
"result": result, | ||
"error": error | ||
} | ||
) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
{% extends "base.jinja" %} | ||
{% block title %}Token Estimator{% endblock %} | ||
{% block content %} | ||
<div class="relative"> | ||
<div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div> | ||
<div class="rounded-xl relative z-20 p-8 sm:p-10 border-[3px] border-gray-900 bg-[#fff4da]"> | ||
<h1 class="text-3xl font-bold text-gray-900 mb-4">Token Estimator</h1> | ||
<form method="post" action="/tokencount" class="space-y-6"> | ||
<div> | ||
<label for="input_text" class="block mb-2 font-medium">Text to analyze:</label> | ||
<textarea name="input_text" id="input_text" rows="4" required class="w-full border-[3px] border-gray-900 rounded p-2 mb-2 bg-[#E8F0FE] focus:outline-none">{{ input_text if input_text else '' }}</textarea> | ||
</div> | ||
<div class="mb-10"> | ||
<label for="model_id" class="block mb-2 font-medium">Model:</label> | ||
<select name="model_id" id="model_id" class="w-full border-[3px] border-gray-900 rounded p-2 bg-[#E8F0FE] focus:outline-none"> | ||
{% for name, model in supported_models.items() %} | ||
<option value="{{ model }}" {% if model_id == model %}selected{% endif %}>{{ name }}</option> | ||
{% endfor %} | ||
</select> | ||
</div> | ||
<div> | ||
<button type="submit" class="bg-yellow-500 hover:bg-yellow-600 text-white font-bold py-2 px-4 rounded border-[3px] border-gray-900">Count Tokens</button> | ||
</div> | ||
</form> | ||
{% if result %} | ||
<div class="mt-6 p-4 border-[3px] border-gray-900 rounded bg-white"> | ||
<h2 class="text-xl font-semibold mb-2">Result</h2> | ||
<p><b>Token count:</b> {{ result.token_count }}</p> | ||
<p><b>Character count:</b> {{ result.character_count }}</p> | ||
<p><b>Model:</b> {{ result.model_id }}</p> | ||
</div> | ||
{% endif %} | ||
{% if error %} | ||
<div class="mt-6 p-4 border-[3px] border-red-600 rounded bg-red-100 text-red-800"> | ||
<b>Error:</b> {{ error }} | ||
</div> | ||
{% endif %} | ||
</div> | ||
</div> | ||
{% endblock %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
from fastapi.testclient import TestClient | ||
from src.server.main import app | ||
|
||
client = TestClient(app, base_url="http://localhost") | ||
|
||
|
||
def test_tokencount_valid(): | ||
HmbleCreator marked this conversation as resolved.
Show resolved
Hide resolved
|
||
response = client.post("/tokencount", json={"input_text": "Hello world!", "model_id": "openai-community/gpt2"}, headers={"host": "localhost"}) | ||
if response.status_code != 200: | ||
print("Response content:", response.content) | ||
assert response.status_code == 200 | ||
data = response.json() | ||
assert "token_count" in data | ||
assert isinstance(data["token_count"], int) | ||
assert data["token_count"] > 0 | ||
|
||
def test_tokencount_missing_input(): | ||
response = client.post("/tokencount", json={"model_id": "openai-community/gpt2"}, headers={"host": "localhost"}) | ||
if response.status_code != 400: | ||
print("Response content:", response.content) | ||
assert response.status_code == 400 | ||
data = response.json() | ||
assert "error" in data |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.