MCP Server Analysis Tool

Comprehensive scraper and analyzer for Model Context Protocol (MCP) servers.

Features

• Comprehensive Data Collection: Scrapes repository metadata, documentation, tools, and technical details
• LLM-Enhanced Analysis: Uses Ollama for intelligent content analysis and classification
• Multiple Output Formats: JSON, CSV, and summary statistics
• Rate Limiting: Respectful scraping with built-in delays
• GitHub API Integration: Enhanced metadata collection

Installation

pip install -r requirements.txt

Usage

Basic Usage

python mcp_scraper.py

Without LLM Analysis

python mcp_scraper.py --no-ollama

Custom Output Directory

python mcp_scraper.py --output my_data/

Data Schema

The scraper extracts the following information for each MCP server:

Basic Information

• Name and repository URL
• Description and category (reference/third-party/official)

Repository Metadata

• Stars, forks, last updated
• Programming language and license
• Author/organization

Technical Details

• Available tools and descriptions
• Installation type (local/API/both)
• API key requirements
• Free vs paid status
• Target platform/service

Documentation Quality

• README content analysis
• Examples and documentation availability
• Configuration requirements

Output Files

• mcp_servers.json: Complete dataset in JSON format
• mcp_servers.csv: Tabular data for analysis
• summary_stats.json: Aggregate statistics and insights

LLM Integration

The tool uses Ollama (llama3 model) to:

• Classify installation types
• Detect API requirements
• Identify pricing models
• Extract technical dependencies
• Analyze tool functionality

Ensure Ollama is installed and the llama3 model is available:

ollama pull llama3