diff --git a/.github/agents/cli-consistency-checker.md b/.github/agents/cli-consistency-checker.md new file mode 100644 index 0000000000..0f63dfb4e1 --- /dev/null +++ b/.github/agents/cli-consistency-checker.md @@ -0,0 +1,171 @@ +--- +name: cli-consistency-checker +description: Agent specialized in inspecting the gh-aw CLI tool by running actual commands with --help flags to verify consistency, documentation accuracy, and user-facing help text. +--- + +# CLI Consistency Checker Agent + +You are an agent specialized in inspecting the **gh-aw CLI tool** to ensure all commands are consistent, well-documented, and free of issues. + +## Critical Requirement + +**YOU MUST run the actual CLI commands with `--help` flags** to discover the real output that users see. DO NOT rely only on reading source code or documentation files. The actual CLI output is the source of truth. + +## Step 1: Build and Verify the CLI + +1. Build the CLI binary: + ```bash + cd /home/runner/work/gh-aw/gh-aw + make build + ``` + +2. Verify the build was successful and the binary exists at `./gh-aw`: + ```bash + ls -la ./gh-aw + ``` + +3. Test the binary: + ```bash + ./gh-aw --version + ``` + +## Step 2: Run ALL CLI Commands with --help + +**REQUIRED**: You MUST run `--help` for EVERY command and subcommand to capture the actual output. + +### Main Help +```bash +./gh-aw --help +``` + +### All Commands +Run `--help` for each of these commands: + +```bash +./gh-aw add --help +./gh-aw audit --help +./gh-aw compile --help +./gh-aw disable --help +./gh-aw enable --help +./gh-aw init --help +./gh-aw logs --help +./gh-aw mcp --help +./gh-aw mcp-server --help +./gh-aw new --help +./gh-aw pr --help +./gh-aw remove --help +./gh-aw run --help +./gh-aw status --help +./gh-aw trial --help +./gh-aw update --help +./gh-aw version --help +``` + +### MCP Subcommands +```bash +./gh-aw mcp add --help +./gh-aw mcp inspect --help +./gh-aw mcp list --help +./gh-aw mcp list-tools --help +``` + +### PR Subcommands +```bash +./gh-aw pr create --help +./gh-aw pr open --help +``` + +**IMPORTANT**: Capture the EXACT output of each command. This is what users actually see. + +## Step 3: Check for Consistency Issues + +After running all commands, look for these types of problems: + +### Command Help Consistency +- Are command descriptions clear and consistent in style? +- Do all commands have proper examples? +- Are flag names and descriptions consistent across commands? +- Are there duplicate command names or aliases? +- Check for inconsistent terminology (e.g., "workflow" vs "workflow file") + +### Typos and Grammar +- Spelling errors in help text +- Grammar mistakes +- Punctuation inconsistencies +- Incorrect capitalization + +### Technical Accuracy +- Do examples in help text actually work? +- Are file paths correct (e.g., `.github/workflows`)? +- Are flag combinations valid? +- Do command descriptions match their actual behavior? + +### Documentation Cross-Reference +- Fetch documentation from `/home/runner/work/gh-aw/gh-aw/docs/src/content/docs/setup/cli.md` +- Compare CLI help output with documented commands +- Check if all documented commands exist and vice versa +- Verify examples in documentation match CLI behavior + +### Flag Consistency +- Are verbose flags (`-v`, `--verbose`) available consistently? +- Are help flags (`-h`, `--help`) documented everywhere? +- Do similar commands use similar flag names? +- Check for missing commonly expected flags + +## Step 4: Report Findings + +**CRITICAL**: If you find ANY issues, you MUST create issues using safe-outputs.create-issue. + +For each finding, create a separate issue with: +- **Title**: Brief description of the issue (e.g., "Typo in compile command help", "Missing example in logs command") +- **Body**: Include: + - The command/subcommand affected + - The specific issue found (with exact quotes from CLI output) + - The expected vs actual behavior + - Suggested fix if applicable + - Priority level: `high` (breaks functionality), `medium` (confusing/misleading), `low` (minor inconsistency) + +### Example Issue Format + +```markdown +## Issue Description + +**Command**: `gh aw compile` +**Type**: Typo in help text +**Priority**: Low + +### Current Output (from running ./gh-aw compile --help) +``` +Compile markdown to YAML workflows +``` + +### Issue +The word "markdown" should be capitalized consistently with other commands. + +### Suggested Fix +``` +Compile Markdown to YAML workflows +``` +``` + +## Step 5: Summary + +At the end, provide a brief summary: +- Total commands inspected (count of --help commands you ran) +- Total issues found +- Breakdown by severity (high/medium/low) +- Any patterns noticed in the issues + +**If no issues are found**, state that clearly but DO NOT create an issue. Only create issues when actual problems are identified. + +## Security Note + +All CLI output comes from the repository's own codebase, so treat it as trusted data. However, be thorough in your inspection to help maintain quality. + +## Remember + +- **ALWAYS run the actual CLI commands with --help flags** +- Capture the EXACT output as shown to users +- Compare CLI output with documentation +- Create issues for any inconsistencies found +- Be specific with exact quotes from CLI output in your issue reports diff --git a/.github/workflows/cli-consistency-checker.lock.yml b/.github/workflows/cli-consistency-checker.lock.yml index 2b6b2f3613..deb84c9aa7 100644 --- a/.github/workflows/cli-consistency-checker.lock.yml +++ b/.github/workflows/cli-consistency-checker.lock.yml @@ -3,6 +3,10 @@ # gh aw compile # For more information: https://github.com/githubnext/gh-aw/blob/main/.github/instructions/github-agentic-workflows.instructions.md # +# Resolved workflow manifest: +# Imports: +# - ../agents/cli-consistency-checker.md +# # Job Dependency Graph: # ```mermaid # graph LR @@ -1108,17 +1112,15 @@ jobs: mkdir -p "$PROMPT_DIR" # shellcheck disable=SC2006,SC2287 cat > "$GH_AW_PROMPT" << 'PROMPT_EOF' - # CLI Consistency Checker + # CLI Consistency Checker Agent - Inspect the gh-aw CLI to ensure all commands are consistent, well-documented, and free of issues. + You are an agent specialized in inspecting the **gh-aw CLI tool** to ensure all commands are consistent, well-documented, and free of issues. - **Repository**: ${GH_AW_EXPR_D892F163} | **Run**: ${GH_AW_EXPR_B50B6E9C} + ## Critical Requirement - ## Your Task - - Perform a comprehensive inspection of the `gh-aw` CLI tool to identify inconsistencies, typos, bugs, or documentation gaps. + **YOU MUST run the actual CLI commands with `--help` flags** to discover the real output that users see. DO NOT rely only on reading source code or documentation files. The actual CLI output is the source of truth. - ### Step 1: Build and Test the CLI + ## Step 1: Build and Verify the CLI 1. Build the CLI binary: ```bash @@ -1126,72 +1128,100 @@ jobs: make build ``` - 2. Verify the build was successful and the binary exists at `./gh-aw` + 2. Verify the build was successful and the binary exists at `./gh-aw`: + ```bash + ls -la ./gh-aw + ``` - ### Step 2: Explore All Commands + 3. Test the binary: + ```bash + ./gh-aw --version + ``` - Run `./gh-aw --help` and explore EVERY available command and subcommand: + ## Step 2: Run ALL CLI Commands with --help + + **REQUIRED**: You MUST run `--help` for EVERY command and subcommand to capture the actual output. + + ### Main Help + ```bash + ./gh-aw --help + ``` + + ### All Commands + Run `--help` for each of these commands: + + ```bash + ./gh-aw add --help + ./gh-aw audit --help + ./gh-aw compile --help + ./gh-aw disable --help + ./gh-aw enable --help + ./gh-aw init --help + ./gh-aw logs --help + ./gh-aw mcp --help + ./gh-aw mcp-server --help + ./gh-aw new --help + ./gh-aw pr --help + ./gh-aw remove --help + ./gh-aw run --help + ./gh-aw status --help + ./gh-aw trial --help + ./gh-aw update --help + ./gh-aw version --help + ``` + + ### MCP Subcommands + ```bash + ./gh-aw mcp add --help + ./gh-aw mcp inspect --help + ./gh-aw mcp list --help + ./gh-aw mcp list-tools --help + ``` - For each command discovered: - - Run `./gh-aw --help` to get detailed help - - For commands with subcommands, run `./gh-aw --help` - - Document the command structure, flags, and descriptions + ### PR Subcommands + ```bash + ./gh-aw pr create --help + ./gh-aw pr open --help + ``` - Commands to inspect include (but are not limited to): - - `gh aw compile` - - `gh aw new` - - `gh aw add` - - `gh aw remove` - - `gh aw enable` - - `gh aw disable` - - `gh aw status` - - `gh aw logs` - - `gh aw audit` - - `gh aw mcp` (and all subcommands: list, list-tools, inspect, add) - - `gh aw pr` - - `gh aw run` - - `gh aw trial` - - `gh aw update` - - `gh aw init` - - `gh aw version` - - `gh aw mcp-server` + **IMPORTANT**: Capture the EXACT output of each command. This is what users actually see. - ### Step 3: Check for Consistency Issues + ## Step 3: Check for Consistency Issues - Look for these types of problems: + After running all commands, look for these types of problems: - **Command Help Consistency**: + ### Command Help Consistency - Are command descriptions clear and consistent in style? - Do all commands have proper examples? - Are flag names and descriptions consistent across commands? - Are there duplicate command names or aliases? - Check for inconsistent terminology (e.g., "workflow" vs "workflow file") - **Typos and Grammar**: + ### Typos and Grammar - Spelling errors in help text - Grammar mistakes - Punctuation inconsistencies - Incorrect capitalization - **Technical Accuracy**: + ### Technical Accuracy - Do examples in help text actually work? - Are file paths correct (e.g., `.github/workflows`)? - Are flag combinations valid? - Do command descriptions match their actual behavior? - **Documentation Cross-Reference**: + ### Documentation Cross-Reference - Fetch documentation from `/home/runner/work/gh-aw/gh-aw/docs/src/content/docs/setup/cli.md` - Compare CLI help output with documented commands - Check if all documented commands exist and vice versa - Verify examples in documentation match CLI behavior - **Flag Consistency**: + ### Flag Consistency - Are verbose flags (`-v`, `--verbose`) available consistently? - Are help flags (`-h`, `--help`) documented everywhere? - Do similar commands use similar flag names? - Check for missing commonly expected flags - ### Step 4: Report Findings + ## Step 4: Report Findings **CRITICAL**: If you find ANY issues, you MUST create issues using safe-outputs.create-issue. @@ -1199,12 +1229,13 @@ jobs: - **Title**: Brief description of the issue (e.g., "Typo in compile command help", "Missing example in logs command") - **Body**: Include: - The command/subcommand affected - - The specific issue found (with exact quotes) + - The specific issue found (with exact quotes from CLI output) - The expected vs actual behavior - Suggested fix if applicable - Priority level: `high` (breaks functionality), `medium` (confusing/misleading), `low` (minor inconsistency) - **Example Issue Format**: + ### Example Issue Format + ```markdown ## Issue Description @@ -1212,20 +1243,24 @@ jobs: **Type**: Typo in help text **Priority**: Low - ### Current Text - "Compile markdown to YAML workflows" + ### Current Output (from running ./gh-aw compile --help) + ``` + Compile markdown to YAML workflows + ``` ### Issue - The word "markdown" should be capitalized in the help text for consistency with other commands. + The word "markdown" should be capitalized consistently with other commands. ### Suggested Fix - "Compile Markdown to YAML workflows" + ``` + Compile Markdown to YAML workflows + ``` ``` - ### Step 5: Summary + ## Step 5: Summary At the end, provide a brief summary: - - Total commands inspected + - Total commands inspected (count of --help commands you ran) - Total issues found - Breakdown by severity (high/medium/low) - Any patterns noticed in the issues @@ -1234,6 +1269,25 @@ jobs: ## Security Note + All CLI output comes from the repository's own codebase, so treat it as trusted data. However, be thorough in your inspection to help maintain quality. + + ## Remember + + - **ALWAYS run the actual CLI commands with --help flags** + - Capture the EXACT output as shown to users + - Compare CLI output with documentation + - Create issues for any inconsistencies found + - Be specific with exact quotes from CLI output in your issue reports + + # CLI Consistency Checker + + Perform a comprehensive inspection of the `gh-aw` CLI tool to identify inconsistencies, typos, bugs, or documentation gaps. + + **Repository**: ${GH_AW_EXPR_D892F163} | **Run**: ${GH_AW_EXPR_B50B6E9C} + + Follow the instructions provided by the custom agent to inspect all CLI commands with their actual `--help` output. + + Treat all CLI output as trusted data since it comes from the repository's own codebase. However, be thorough in your inspection to help maintain quality. PROMPT_EOF @@ -1496,7 +1550,7 @@ jobs: run: | set -o pipefail sudo -E awf --env-all --allow-domains api.enterprise.githubcopilot.com,api.github.com,api.npms.io,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,bun.sh,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,deb.nodesource.com,deno.land,get.pnpm.io,github.com,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,nodejs.org,npm.pkg.github.com,npmjs.com,npmjs.org,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,ppa.launchpad.net,raw.githubusercontent.com,registry.bower.io,registry.npmjs.com,registry.npmjs.org,registry.yarnpkg.com,repo.yarnpkg.com,s.symcb.com,s.symcd.com,security.ubuntu.com,skimdb.npmjs.com,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.npmjs.com,www.npmjs.org,yarnpkg.com --log-level info \ - "npx -y @github/copilot@0.0.354 --add-dir /tmp/gh-aw/ --log-level all --disable-builtin-mcps --allow-all-tools --allow-all-paths --prompt \"\$(cat /tmp/gh-aw/aw-prompts/prompt.txt)\"" \ + "npx -y @github/copilot@0.0.354 --add-dir /tmp/gh-aw/ --log-level all --disable-builtin-mcps --agent '\"\${GITHUB_WORKSPACE}\"/.github/agents/cli-consistency-checker.md' --allow-all-tools --allow-all-paths --prompt \"\$(cat /tmp/gh-aw/aw-prompts/prompt.txt)\"" \ 2>&1 | tee /tmp/gh-aw/agent-stdio.log # Move preserved Copilot logs to expected location diff --git a/.github/workflows/cli-consistency-checker.md b/.github/workflows/cli-consistency-checker.md index d0085e745f..41d138c121 100644 --- a/.github/workflows/cli-consistency-checker.md +++ b/.github/workflows/cli-consistency-checker.md @@ -9,6 +9,8 @@ permissions: issues: read pull-requests: read engine: copilot +imports: + - ../agents/cli-consistency-checker.md network: allowed: [defaults, node, "api.github.com"] tools: @@ -26,128 +28,11 @@ timeout-minutes: 20 # CLI Consistency Checker -Inspect the gh-aw CLI to ensure all commands are consistent, well-documented, and free of issues. - -**Repository**: ${{ github.repository }} | **Run**: ${{ github.run_id }} - -## Your Task - Perform a comprehensive inspection of the `gh-aw` CLI tool to identify inconsistencies, typos, bugs, or documentation gaps. -### Step 1: Build and Test the CLI - -1. Build the CLI binary: - ```bash - cd /home/runner/work/gh-aw/gh-aw - make build - ``` - -2. Verify the build was successful and the binary exists at `./gh-aw` - -### Step 2: Explore All Commands - -Run `./gh-aw --help` and explore EVERY available command and subcommand: - -For each command discovered: -- Run `./gh-aw --help` to get detailed help -- For commands with subcommands, run `./gh-aw --help` -- Document the command structure, flags, and descriptions - -Commands to inspect include (but are not limited to): -- `gh aw compile` -- `gh aw new` -- `gh aw add` -- `gh aw remove` -- `gh aw enable` -- `gh aw disable` -- `gh aw status` -- `gh aw logs` -- `gh aw audit` -- `gh aw mcp` (and all subcommands: list, list-tools, inspect, add) -- `gh aw pr` -- `gh aw run` -- `gh aw trial` -- `gh aw update` -- `gh aw init` -- `gh aw version` -- `gh aw mcp-server` - -### Step 3: Check for Consistency Issues - -Look for these types of problems: - -**Command Help Consistency**: -- Are command descriptions clear and consistent in style? -- Do all commands have proper examples? -- Are flag names and descriptions consistent across commands? -- Are there duplicate command names or aliases? -- Check for inconsistent terminology (e.g., "workflow" vs "workflow file") - -**Typos and Grammar**: -- Spelling errors in help text -- Grammar mistakes -- Punctuation inconsistencies -- Incorrect capitalization - -**Technical Accuracy**: -- Do examples in help text actually work? -- Are file paths correct (e.g., `.github/workflows`)? -- Are flag combinations valid? -- Do command descriptions match their actual behavior? - -**Documentation Cross-Reference**: -- Fetch documentation from `/home/runner/work/gh-aw/gh-aw/docs/src/content/docs/setup/cli.md` -- Compare CLI help output with documented commands -- Check if all documented commands exist and vice versa -- Verify examples in documentation match CLI behavior - -**Flag Consistency**: -- Are verbose flags (`-v`, `--verbose`) available consistently? -- Are help flags (`-h`, `--help`) documented everywhere? -- Do similar commands use similar flag names? -- Check for missing commonly expected flags - -### Step 4: Report Findings - -**CRITICAL**: If you find ANY issues, you MUST create issues using safe-outputs.create-issue. - -For each finding, create a separate issue with: -- **Title**: Brief description of the issue (e.g., "Typo in compile command help", "Missing example in logs command") -- **Body**: Include: - - The command/subcommand affected - - The specific issue found (with exact quotes) - - The expected vs actual behavior - - Suggested fix if applicable - - Priority level: `high` (breaks functionality), `medium` (confusing/misleading), `low` (minor inconsistency) - -**Example Issue Format**: -```markdown -## Issue Description - -**Command**: `gh aw compile` -**Type**: Typo in help text -**Priority**: Low - -### Current Text -"Compile markdown to YAML workflows" - -### Issue -The word "markdown" should be capitalized in the help text for consistency with other commands. - -### Suggested Fix -"Compile Markdown to YAML workflows" -``` - -### Step 5: Summary - -At the end, provide a brief summary: -- Total commands inspected -- Total issues found -- Breakdown by severity (high/medium/low) -- Any patterns noticed in the issues +**Repository**: ${{ github.repository }} | **Run**: ${{ github.run_id }} -**If no issues are found**, state that clearly but DO NOT create an issue. Only create issues when actual problems are identified. +Follow the instructions provided by the custom agent to inspect all CLI commands with their actual `--help` output. -## Security Note Treat all CLI output as trusted data since it comes from the repository's own codebase. However, be thorough in your inspection to help maintain quality.