feat: add plugins cli, external plugin support, plugin template

[araujof]

What issue(s) are addressed by this PR?

Closes #720 #773

Add MCP Plugins CLI and External Plugin Support

Implementation Summary

This pull request extends the Context Forge's plugin system, enabling developers to extend the gateway's functionality through external plugins deployed as standalone services. The implementation introduces a new mcpplugins CLI tool that streamlines plugin development, deployment, and integration workflows.

The external plugin integration leverages the Model Context Protocol's standardized interface to enable seamless communication between the Context Forge gateway and external plugin services, promoting modularity and extensibility.

Changes Made

New CLI Tool

• Added mcpplugins command-line interface for managing plugin lifecycle operations
• Implemented plugin scaffolding, build, deployment, and management commands
• Integrated with existing Context Forge configuration and service discovery mechanisms

Plugin Template System

• Introduced copier-based plugin template for rapid plugin bootstrapping
• Created standardized project structure with best practices for MCP plugin development
• Included configuration templates, documentation stubs, and example implementations
• Established consistent naming conventions and directory layouts for plugin projects

MCP Client/Server Implementation

• Developed MCP client components for gateway-to-plugin communication
• Implemented MCP server interfaces for plugin service endpoints
• Added protocol translation layers to handle different transport mechanisms (stdio, SSE, HTTP) based on chuck-mcp-runtime

Breaking Changes: None - this is a purely additive feature that maintains backward compatibility with existing Context Forge deployments.

Migration Notes: Existing installations will automatically gain access to the new plugin system without requiring configuration changes. The mcpplugins CLI will be available after package installation.

[crivetimihai]

Review in progress, don't touch ;-)

[crivetimihai]

PR #722 Review: Plugin CLI and External Plugin Support

TODO:

• Move copier to install-dev not install

Overview

This PR implements comprehensive plugin system enhancements for the MCP Context Forge Gateway, adding a CLI tool (mcpplugins) and external plugin support with MCP server integration.

Status: ✅ APPROVED & UPDATED

• Rebase: ✅ Successfully rebased against main (64 commits processed)
• Tests: ✅ All tests passing (2109 passed, 15 skipped, 79% coverage)
• Quality: ✅ Fixed all linting tools - now passing (Bandit, Pylint 10/10, Flake8)
• Dependencies: ✅ New dependencies installed and working
• Force Push: ✅ Updated araujof:feat/720-plugins-cli with fixes (commit: cf7df00)

Changes Summary

🆕 New Features Added

1. mcpplugins CLI Tool (mcpgateway/plugins/tools/cli.py)

• Bootstrap command: Creates new plugin projects from templates using Copier
• Template support: Both native and external plugin templates
• Git integration: Auto-detects author name/email from git config
• Rich markup support: Configurable via environment variables
• Entry point: Available as mcpplugins console script

2. External Plugin Framework (mcpgateway/plugins/framework/external/)

• MCP Client Integration: Connects to external MCP servers as plugins
• Multi-transport Support: stdio and streamable HTTP transports
• Plugin Server Runtime: Based on chuk-mcp-runtime for robust MCP handling
• Hook System: Pre/post processing hooks for tools, prompts, and resources

3. Plugin Templates (plugin_templates/)

• External Plugin Template: Full project scaffold with:
  • Containerfile for deployment
  • Makefile for development workflow
  • Test framework setup
  • Configuration templates
  • Documentation stubs
• Native Plugin Template: Lightweight template for internal plugins

4. Documentation & Configuration

• Plugin lifecycle documentation (docs/docs/using/plugins/lifecycle.md)
• Environment variable configuration for CLI behavior
• Copier template configuration at repository root

🔧 Key Technical Components

External Plugin Architecture

• ExternalPlugin class: Manages connections to remote MCP servers
• Transport abstraction: Supports stdio and HTTP streaming protocols
• Session management: Uses AsyncExitStack for proper resource cleanup
• Error handling: Comprehensive error conversion and logging

CLI Architecture

• Typer-based: Modern CLI framework with rich help and completion
• Template-driven: Uses Copier for project scaffolding
• Environment-aware: Configurable markup and completion behavior
• Git-integrated: Auto-detects user configuration

📈 Code Quality Metrics

• Files Changed: 126 files modified/added
• New Files: 69 new files added
• Test Coverage: 79% overall coverage maintained
• Tests Status: 2109 tests passing, 15 skipped
• Dependencies: All new dependencies properly integrated

🧪 Testing Coverage

New test suites added for:

• CLI functionality (tests/unit/mcpgateway/plugins/tools/test_cli.py)
• External plugin runtime (tests/unit/mcpgateway/plugins/framework/external/)
• Plugin framework components
• Hook system integration

🔒 Security Considerations

• Subprocess Safety: Uses secure subprocess invocation patterns
• Template Security: Templates use Jinja2 with proper escaping
• MCP Protocol: Leverages established MCP security model
• No Shell Injection: Avoids shell=True in subprocess calls

🚀 Deployment Impact

• Breaking Changes: None - purely additive features
• Dependencies: New optional dependencies added to pyproject.toml
• Entry Points: New mcpplugins console script available
• Configuration: New optional environment variables

📋 Quality Assurance

✅ Completed Checks

1. Rebase: Successfully rebased 64 commits against main
2. Dependencies: Installed copier, chuk-mcp-runtime, typer
3. Tests: All 2124 tests collected and executed successfully
4. Coverage: Maintained 79% code coverage
5. Linting: Pre-commit hooks applied formatting fixes
6. Security: Fixed all Bandit security issues (5 → 0)
7. Code Quality: Fixed all Pylint issues (achieved perfect 10/10 score)
8. Style: Fixed all Flake8 style/documentation issues

🛠️ Review Fixes Applied

During the review process, the following issues were identified and fixed:

Security Fixes (Bandit):

• Added explicit check=False to subprocess.run calls in CLI
• Added proper #nosec comments with security justifications
• Fixed subprocess security warnings for git command execution

Code Quality Fixes (Pylint):

• Fixed method signatures in plugin framework (replaced elif with if after return)
• Corrected Pydantic model validator parameter naming
• Fixed cyclic import by using direct imports instead of package-level imports
• Renamed global variables to follow UPPER_CASE naming conventions
• Updated test files to match renamed variables

Style Fixes (Flake8):

• Corrected Pydantic model validator docstrings (removed excess/missing parameters)
• Fixed method parameter naming for model validators
• Added appropriate pylint disable comments for Pydantic-specific patterns

🔧 Technical Details of Fixes

Files Modified During Review:

• mcpgateway/plugins/tools/cli.py: Security fixes for subprocess calls
• mcpgateway/plugins/framework/models.py: Pydantic validator signatures and docs
• mcpgateway/plugins/framework/manager.py: Control flow improvements
• mcpgateway/plugins/framework/utils.py: Control flow improvements
• mcpgateway/plugins/framework/loader/plugin.py: Python 3 inheritance cleanup
• mcpgateway/plugins/framework/external/mcp/client.py: Parentheses cleanup
• mcpgateway/plugins/framework/external/mcp/server/runtime.py: Global variable naming
• mcpgateway/plugins/framework/external/mcp/server/server.py: Import path fixes
• tests/unit/mcpgateway/plugins/framework/external/mcp/server/test_runtime.py: Variable name updates

Key Patterns Fixed:

• Subprocess Security: subprocess.run(cmd, check=False) with #nosec justifications
• Control Flow: elif → if after return/raise statements
• Pydantic Validators: Proper self parameter usage with #pylint: disable=bad-classmethod-argument
• Import Cycles: Direct imports instead of package-level imports
• Global Constants: snake_case → UPPER_CASE naming

✅ Final Quality Metrics

• Bandit: 0 security issues (was 5)
• Pylint: 10.00/10 perfect score (was failing with 14 issues)
• Flake8: 0 style issues (was 4 issues)
• Tests: All framework and CLI tests passing
• Functionality: Full preservation of all features

⚠️ Notes

• Some tests are skipped due to external dependencies (expected)
• New dependencies require installation for full functionality
• All fixes maintain backward compatibility

Recommendation

✅ APPROVED FOR MERGE - This PR successfully implements a comprehensive plugin system with:

🎯 Original Features

• Well-architected external plugin support
• Professional CLI tooling (mcpplugins)
• Comprehensive testing (2109 tests passing)
• Proper documentation and lifecycle guides
• No breaking changes
• Clean integration with existing codebase

[crivetimihai]

Post-PR #722 Fixes

Critical Dependency Issues

• Move copier>=9.9.0 from main dependencies to [project.optional-dependencies.plugins] (or just dev?)
• Move typer>=0.16.0 from main dependencies to [project.optional-dependencies.plugins] (or just dev?)
• Update installation docs to show pip install mcp-contextforge-gateway[plugins] for CLI

CLI Implementation

• Uncomment and implement mcpplugins install command from existing code
• Add mcpplugins list command to list available templates
• Add mcpplugins validate command to validate plugin configurations
• Improve error messages when git is not found
• Add better error handling for copier template failures

Error Handling

• Add timeout handling for external plugin MCP calls
• Add circuit breaker pattern for repeatedly failing external plugins
• Improve plugin validation error messages with specific field information
• Add graceful degradation when external plugins are unavailable
• Handle network failures in external plugin communication

Documentation

• Add mcpplugins CLI section to main README.md
• Create plugin development quick start guide in docs
• Document [plugins] and [chuck] optional dependency groups
• Add examples of using mcpplugins bootstrap command
• Document external plugin MCP server requirements

Performance

• Make plugin loading async where blocking is unnecessary
• Add resource monitoring for external plugin processes
• Add plugin execution time metrics and logging
• Consider plugin process isolation/sandboxing for security

Testing Gaps

• Add integration tests for mcpplugins CLI commands
• Add tests for plugin failure scenarios (timeout, crash, invalid config)
• Add tests for external plugin MCP communication edge cases
• Add performance tests for plugin execution overhead

Developer Experience

• Add plugin hot-reloading capability for development mode
• Improve Copier template validation and error messages
• Add validation for template variables before processing
• Create additional plugin template types (authentication, logging, monitoring)
• Add plugin debugging/tracing capabilities

Security

• Add plugin execution sandboxing for external plugins
• Add resource limits (CPU/memory) for plugin processes
• Add audit logging for plugin actions and modifications
• Consider plugin signing/verification for distributed plugins

Architecture Improvements

• Add plugin discovery mechanism for auto-loading from directories
• Add plugin versioning and compatibility checking
• Add inter-plugin dependency management
• Create plugin marketplace/registry foundation