Skip to content

Conversation

jjain1259
Copy link
Member

@jjain1259 jjain1259 commented Oct 1, 2025

Summary

This PR introduces a configurable hard limit for the number of topic-to-external-resource mappings.

Changes Made

New Configuration Parameter

  • Config Name: max.external.resource.mappings
  • Default Value: 15
  • Type: Integer (minimum value: 1)
  • Description: The maximum number of topic-to-external-resource mappings allowed

Validation Logic

  • Added validation in getTopicToExternalResourceMap() method to enforce the mapping limit
  • Throws ConfigException with descriptive error message when limit is exceeded
  • Error message includes:
    • Current number of mappings configured
    • Maximum allowed limit
    • Configuration parameter name to modify the limit

Test Coverage

Added comprehensive test cases in ValidatorTest.java:

  • testValidMappingWithinLimit() - Verifies mappings within limit work correctly
  • testInvalidTooManyMappings() - Tests custom limit enforcement
  • testDefaultMappingLimit() - Validates default limit of 15 mappings

Configuration Usage

Users can now configure the connector with a custom limit:

# Set custom limit (optional, defaults to 15)
max.external.resource.mappings=25

# Configure topic mappings (validated against the limit)
external.resource.usage=INDEX
topic.to.external.resource.mapping=topic1:index1,topic2:index2,topic3:index3

Error Handling

When the limit is exceeded, users receive a clear error message:

Too many topic-to-external-resource mappings configured (16). Maximum allowed is 15. Reduce the number of mappings or increase the 'max.external.resource.mappings' configuration.

Backward Compatibility

  • ✅ Existing configurations continue to work with the default limit of 15
  • ✅ No breaking changes to existing APIs
  • ✅ Configuration is optional with default

Testing

  • All existing tests pass
  • New test cases cover both valid and invalid scenarios
  • Tests validate default and custom limit enforcement

@jjain1259 jjain1259 requested a review from a team as a code owner October 1, 2025 06:26
@Copilot Copilot AI review requested due to automatic review settings October 1, 2025 06:26
@jjain1259 jjain1259 changed the base branch from master to 14.1.x October 1, 2025 06:26
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a configurable limit for topic-to-external-resource mappings in the Elasticsearch connector, with a default maximum of 15 mappings to prevent performance issues or configuration errors.

  • Added a new configuration parameter max.external.resource.mappings with a default value of 15
  • Implemented validation logic to enforce the mapping limit during configuration parsing
  • Added comprehensive test coverage for the new validation functionality

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
ElasticsearchSinkConnectorConfig.java Added new configuration parameter and validation logic for mapping limits
ValidatorTest.java Added test cases to verify mapping limit validation behavior

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

mappingBuilder.append("topic").append(i).append(":index").append(i);
topicsBuilder.append("topic").append(i);
}

Copy link

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Missing newline at the end of the line before the closing brace. There should be consistent spacing around the closing brace.

Suggested change

Copilot uses AI. Check for mistakes.

@jjain1259 jjain1259 changed the title Added topic to external resource mapping limit to 15 and made it conf… Add configurable limit for topic-to-external-resource mappings Oct 1, 2025
@sonarqube-confluent

This comment has been minimized.

1 similar comment
@sonarqube-confluent
Copy link

Failed

  • 0.00% Coverage on New Code (is less than 80.00%)

Analysis Details

0 Issues

  • Bug 0 Bugs
  • Vulnerability 0 Vulnerabilities
  • Code Smell 0 Code Smells

Coverage and Duplications

  • Coverage 0.00% Coverage (0.00% Estimated after merge)
  • Duplications No duplication information (0.00% Estimated after merge)

Project ID: kafka-connect-elasticsearch

View in SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant