Notify users when the server might not be available when polling for messages

[aashikam]

Error Management in Ballerina Kafka Library

Identified Issues

• Consumer operates without a Kafka server: The Kafka consumer service can start and run even in the absence of a running Kafka server.
• Consumer continues running despite server downtime: Once the Kafka consumer has started and begun consuming messages, it will continue to operate indefinitely, even if the Kafka server goes down.

Root Cause Analysis

The issue arises from the design of the poll() method in the Kafka consumer. When the broker is unreachable, the poll() method:

• Returns an empty list (ConsumerRecords[]).
• Does not throw an exception or trigger a retry log, causing connection errors to be silently handled in the background.

Solutions

• Check Kafka Metrics: Add health checks using Kafka's built-in metrics API. (implemented in this PR)
• Capture Network client logs: Parse logs from org.apache.kafka.clients.NetworkClient programmatically. (Hook SLF4J for NetworkClient logger )

fixes: ballerina-platform/ballerina-library#8400

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

“aashikam” seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 80.95238% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.28%. Comparing base (91f785f) to head (0f2ee3e).

Files with missing lines	Patch %	Lines
...va/io/ballerina/stdlib/kafka/utils/KafkaUtils.java	62.50%	1 Missing and 2 partials ⚠️
...llerina/stdlib/kafka/nativeimpl/consumer/Poll.java	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1274      +/-   ##
============================================
- Coverage     82.74%   81.28%   -1.47%     
- Complexity      355      505     +150     
============================================
  Files            38       47       +9     
  Lines          2127     2591     +464     
  Branches        268      394     +126     
============================================
+ Hits           1760     2106     +346     
- Misses          270      319      +49     
- Partials         97      166      +69     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.