Add support for matched_fields with the unified highlighter #18166

nomoa · 2025-04-30T09:58:57Z

Description

This option was silently ignored when running the unified highlighter. Since lucene 10 it is now possible to pass a list of additional fields to fetch matches from and blend into the snippets.

Related Issues

Resolves #18164

Check List

Functionality includes testing.
~~[ ] API changes companion pull request created, if applicable.~~
Public documentation issue/PR: Document that the unified highlighter supports matched_field documentation-website#9793

github-actions · 2025-04-30T10:11:25Z

❌ Gradle check result for e9a0a85: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-30T11:37:02Z

❌ Gradle check result for ad13011: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-03T13:08:25Z

❌ Gradle check result for 32061e5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

nomoa · 2025-05-05T12:20:44Z

Failure I can see from the last build is:

> Task :test:fixtures:gcs-fixture:composeDown
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-third-party_1           ... 
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture_1                       ... 
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-other_1                 ... 
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-repositories-metering_1 ... 
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-repositories-metering_1 ... error
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture_1                       ... error
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-third-party_1           ... error
Stopping c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-other_1                 ... done

ERROR: for c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-repositories-metering_1  cannot stop container: 6be4186fdfda17109af09d376297dc67310ec862c6c4d760714806ab0172e90a: container 6be4186fdfda PID 37320 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes

ERROR: for c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture_1  cannot stop container: 1bfd96106dd37b667377653973bd6207eff410e6689127693cde6e6e1bf7645a: container 1bfd96106dd3 PID 37378 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes

ERROR: for c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-third-party_1  cannot stop container: 2c9315a4c36fa8f299580844e98e5576d7b5f57dcc255d3628f6acc31724938a: container 2c9315a4c36f PID 37470 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-third-party_1           ... 
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture_1                       ... 
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-other_1                 ... 
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-repositories-metering_1 ... 
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-other_1                 ... done
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture_1                       ... done
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-third-party_1           ... done
Removing c632c20032974ac355b722657aedda3e_gcs-fixture_gcs-fixture-repositories-metering_1 ... done
Removing network c632c20032974ac355b722657aedda3e_gcs-fixture_default

> Task :test:fixtures:krb5kdc-fixture:composeDown
Stopping 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_peppa_1 ... 
Stopping 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_hdfs_1  ... 
Stopping 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_peppa_1 ... done
Stopping 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_hdfs_1  ... done
Removing 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_peppa_1 ... 
Removing 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_hdfs_1  ... 
Removing 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_hdfs_1  ... done
Removing 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_peppa_1 ... done
Removing network 2ff70b1d4a8f216a2e2075eb5fa05c77_krb5kdc-fixture_default
build complete, generating: /var/jenkins/workspace/gradle-check/search/build/57600.tar.bz2

[Incubating] Problems report is available at: file:///var/jenkins/workspace/gradle-check/search/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

github-actions · 2025-05-05T12:48:55Z

❌ Gradle check result for a6fbda7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-08T19:18:49Z

✅ Gradle check result for fa503d1: SUCCESS

codecov · 2025-05-08T19:19:12Z

Codecov Report

Attention: Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.

Project coverage is 72.47%. Comparing base (93d5356) to head (dcd711a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...h/fetch/subphase/highlight/UnifiedHighlighter.java	63.63%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18166      +/-   ##
============================================
- Coverage     72.57%   72.47%   -0.11%     
+ Complexity    67446    67325     -121     
============================================
  Files          5488     5488              
  Lines        311069   311067       -2     
  Branches      45217    45218       +1     
============================================
- Hits         225757   225431     -326     
- Misses        66986    67229     +243     
- Partials      18326    18407      +81

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

msfroh

This looks great! Thanks a lot @nomoa!

...rnalClusterTest/java/org/opensearch/search/fetch/subphase/highlight/HighlighterSearchIT.java

nomoa · 2025-05-09T17:33:34Z

is the code coverage check being green mandatory? The code is being tested in integration tests but if you prefer I can introduce a dedicated test for these lines (might need to make some methods visible for testing, unless there are some facilities to easily create/mock FieldHighlightContext)

github-actions · 2025-05-09T20:39:52Z

❌ Gradle check result for 4d6a2ae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-12T14:19:46Z

❕ Gradle check result for 09717f8: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2025-05-12T19:32:38Z

❌ Gradle check result for 95d48bb: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This option was silently ignored when running the unified highlighter. Since lucene 10 it is now possible to pass a list of additional fields to fetch matches from and blend into the snippets. Signed-off-by: David Causse <dcausse@wikimedia.org>

github-actions · 2025-05-19T15:41:57Z

✅ Gradle check result for dcd711a: SUCCESS

nomoa · 2025-05-19T16:46:03Z

@andrross @msfroh thanks for the reviews! Is there anything I can do to help get this PR merged?

msfroh · 2025-05-19T17:51:52Z

@andrross @msfroh thanks for the reviews! Is there anything I can do to help get this PR merged?

One of the automated checks "Validate Gradle Wrapper" had failed with a timeout. I just retried it and it looks good now. 👍

Thanks @nomoa!

nomoa requested review from anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, VachaShah and a team as code owners April 30, 2025 09:58

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search Search query, autocomplete ...etc labels Apr 30, 2025

nomoa force-pushed the unified-highlighter-support-matched-fields branch from e9a0a85 to ad13011 Compare April 30, 2025 10:39

This was referenced Apr 29, 2025

[AUTOCUT] Gradle Check Flaky Test Report for CreateIndexIT #14312

Open

[AUTOCUT] Gradle Check Flaky Test Report for WarmIndexSegmentReplicationIT #18157

Open

nomoa force-pushed the unified-highlighter-support-matched-fields branch from ad13011 to 6f75191 Compare April 30, 2025 13:05

nomoa force-pushed the unified-highlighter-support-matched-fields branch from f7907c5 to 32061e5 Compare May 3, 2025 12:41

nomoa force-pushed the unified-highlighter-support-matched-fields branch from 32061e5 to a6fbda7 Compare May 5, 2025 12:23

This was referenced May 6, 2025

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobStoreRepositoryTests #14299

Open

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobContainerRetriesTests #17551

Open

nomoa force-pushed the unified-highlighter-support-matched-fields branch from a6fbda7 to fa503d1 Compare May 8, 2025 18:09

msfroh approved these changes May 8, 2025

View reviewed changes

andrross reviewed May 8, 2025

View reviewed changes

...rnalClusterTest/java/org/opensearch/search/fetch/subphase/highlight/HighlighterSearchIT.java Show resolved Hide resolved

andrross approved these changes May 8, 2025

View reviewed changes

nomoa force-pushed the unified-highlighter-support-matched-fields branch from fa503d1 to 4d6a2ae Compare May 9, 2025 20:06

This was referenced May 10, 2025

[AUTOCUT] Gradle Check Flaky Test Report for QueueResizableOpenSearchThreadPoolExecutorTests #14297

Closed

[AUTOCUT] Gradle Check Flaky Test Report for SecureReactorNetty4HttpServerTransportTests #17486

Open

nomoa force-pushed the unified-highlighter-support-matched-fields branch from 4d6a2ae to 09717f8 Compare May 12, 2025 13:09

nomoa force-pushed the unified-highlighter-support-matched-fields branch from 09717f8 to 95d48bb Compare May 12, 2025 17:30

opensearch-ci-bot mentioned this pull request May 7, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreIT #16145

Open

nomoa force-pushed the unified-highlighter-support-matched-fields branch from 95d48bb to dcd711a Compare May 19, 2025 14:33

msfroh approved these changes May 19, 2025

View reviewed changes

msfroh merged commit 553d0bf into opensearch-project:main May 19, 2025
29 of 30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for matched_fields with the unified highlighter #18166

Add support for matched_fields with the unified highlighter #18166

nomoa commented Apr 30, 2025 •

edited

Loading

github-actions bot commented Apr 30, 2025

github-actions bot commented Apr 30, 2025

github-actions bot commented May 3, 2025

nomoa commented May 5, 2025

github-actions bot commented May 5, 2025

github-actions bot commented May 8, 2025

codecov bot commented May 8, 2025 •

edited

Loading

msfroh left a comment

nomoa commented May 9, 2025

github-actions bot commented May 9, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 19, 2025

nomoa commented May 19, 2025

msfroh commented May 19, 2025

Add support for matched_fields with the unified highlighter #18166

Add support for matched_fields with the unified highlighter #18166

Conversation

nomoa commented Apr 30, 2025 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Apr 30, 2025

github-actions bot commented Apr 30, 2025

github-actions bot commented May 3, 2025

nomoa commented May 5, 2025

github-actions bot commented May 5, 2025

github-actions bot commented May 8, 2025

codecov bot commented May 8, 2025 • edited Loading

Codecov Report

msfroh left a comment

Choose a reason for hiding this comment

nomoa commented May 9, 2025

github-actions bot commented May 9, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 19, 2025

nomoa commented May 19, 2025

msfroh commented May 19, 2025

nomoa commented Apr 30, 2025 •

edited

Loading

codecov bot commented May 8, 2025 •

edited

Loading