pull request 20241114 #951
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have read the CLA Document and I hereby sign the CLA 1 out of 4 committers have signed the CLA. |
nil-andreu
reviewed
Nov 16, 2024
Dockerfile
Outdated
| wget https://gitee.com/myhloli/MinerU/raw/master/scripts/download_models.py && \ | ||
| python3 download_models.py && \ | ||
| sed -i 's|cpu|cuda|g' /root/magic-pdf.json" | ||
|
|
||
| # install extents | ||
| COPY requirements-fastapi.txt /minerugw/requirements-fastapi.txt |
There was a problem hiding this comment.
To do not increase by default the size of the image, this could be set as a optional requirements.
返回当前队列长度
- Set PyMuPDF version to <= 1.24.14 in all requirements files - Prevent potential compatibility issues with future versions
- Merge title blocks that are close to each other horizontally - Adjust line insertion logic for title blocks- Increase image size and decrease confidence threshold for layout detection - Update DocLayoutYOLO model weights - Refactor drawing of bounding boxes for different block types
- Add remove_tilted_line function to filter out lines with angles between 2 and 88 degrees - Integrate the new function into the text extraction process - Improve the accuracy of text block processing by removing non-horizontal/vertical lines
- Add key length validation for ONNX model initialization - Move import statements to the top of the file - Wrap model initialization in a try-except block for better error handling - Refactor code to improve readability and maintainability
- Update model path from 'unimernet_small' to 'unimernet_small_2501' in multiple scripts and configuration files - This change affects download_models.py, download_models_hf.py, and model_configs.yaml
- Reduce YOLO_LAYOUT_BASE_BATCH_SIZE from 4 to 1 - Simplify batch ratio calculation for formula detection - Remove unused conditional logic in batch ratio determination
- Update GPU memory check and batch ratio calculation logic - Add support for virtual VRAM size environment variable - Improve logging for GPU memory and batch ratio
…e VRAM allocation logic to use 'VIRTUAL_VRAM_SIZE' environment variable - Reduce MFR (Math Formula Recognition) batch size from 64 to 32
- Reduce batch_ratio by 1 for better performance and stability - This change ensures more consistent memory usage when processing documents
- Improve batch ratio calculation based on GPU memory - Enhance performance for devices with 8GB or more VRAM
- Update conditions for batch ratio assignment: -8 <= gpu_memory < 10: batch_ratio = 2 - 10 <= gpu_memory <= 12: batch_ratio =4 - This fix ensures proper batch ratio selection for GPU memory sizes
- Restore commented code for filtering out characters with invalid bounding boxes - This change may affect the filtering of unnecessary characters in PDF parsing
- Add a check to return 0 when either bbox1_area or bbox2_area is zero - This prevents division by zero errors when calculating IoU
- Add timing measurement for formula, text, and title optimization using LLM - Log the execution time for each LLM aided process
- Change Miners homepage link from 'https://mineru.org.cn/home?source=online' to 'https://mineru.net/home?source=online' - Change Miners client download link from 'https://mineru.org.cn/client?source=online' to 'https://mineru.net/client?source=online'
- Add sub_model configuration option for rapid_table model - Provide two sub_model options: slanet_plus and unitable
…ilities: upgrade to latest doclayout_yolo(2501) and unimernet(2501) models - Improve performance: optimize resource usage and processing pipeline for faster parsing on high-end devices- Enhance parsing effects: add new heading classification feature to online demo - Refactor changelog structure for better readability and organization
…ability - Update online demo links in both English and Chinese README files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Please describe the motivation of this PR and the goal you want to achieve through this PR.
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
Checklist
Before PR:
After PR: