Skip to content

Secondary indexes (streams, category, event type) #5115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 167 commits into from
Closed

Secondary indexes (streams, category, event type) #5115

wants to merge 167 commits into from

Conversation

alexeyzimarev
Copy link
Member

  • Add DuckDB init code
  • Add subscription for building the default index tables
  • Add index builders (stream, event type, category)
  • Add index readers

oskardudycz and others added 11 commits May 23, 2025 12:09
@oskardudycz
Added the first real DuckDB secondary index implementation
ab80600 
That includes DuckDB setup, default index, processor, reader, etc.

Extended also the database readers code to support reading real events based on the indexed position.
@oskardudycz
Configured the DuckDb path for integration tests and moved initialisa…
4667dac 
…tion code do DefaultIndex constructor
@oskardudycz
Added Query overloads for DuckDbAdvancedConnection
94cb8ba
@oskardudycz
Skiped for now tests around plugin enablement with the intention to f…
e266d12 
…ix them later
@oskardudycz
Added Category Index imlementation
3affe99
@oskardudycz
Added Event Type index implementation
459df55
@oskardudycz
Added stream index implementation
1163749
@alexeyzimarev
Merge branch 'master' into indexing/duck_db_default_index
5bfe24f
@alexeyzimarev
* Fixed the copyright notice
11cb0b2 
* Removed unused stuff (sync enumerator, etc)
* Renamed indices to indexes
* Removed LogV3 tests
@alexeyzimarev
Merge branch 'indexing/duck_db_default_index' into indexing/duck_db_c…
ddffb60 
…e_et_stream

# Conflicts:
#	src/KurrentDB.SecondaryIndexing.Tests/IntegrationTests/SecondaryIndexingPluginEnabledIntegrationTests.cs
#	src/KurrentDB.SecondaryIndexing/Indexes/Default/DefaultSecondaryIndexProcessor.cs
@alexeyzimarev
Merge the first PR
3640bed
@alexeyzimarev alexeyzimarev requested a review from a team as a code owner May 26, 2025 17:36
@alexeyzimarev alexeyzimarev marked this pull request as draft May 26, 2025 17:38
@alexeyzimarev alexeyzimarev mentioned this pull request May 26, 2025
Closed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 26, 2025 •
edited
Loading

Qodana for .NET

17 new problems were found

Inspection name Severity Problems
Parameter is only used for precondition check (private accessibility) 🔶 Warning 6
Redundant cast 🔶 Warning 2
Collection content is never queried (private accessibility) 🔶 Warning 1
Member initialized value ignored 🔶 Warning 1
Non-accessed field (private accessibility) 🔶 Warning 1
Redundant class or interface specification in base types list 🔶 Warning 1
Redundant name qualifier 🔶 Warning 1
Redundant using directive 🔶 Warning 1
Get-only auto-property is never assigned 🔶 Warning 1
Type member is never used (private accessibility) 🔶 Warning 1
Unused parameter (private accessibility) 🔶 Warning 1

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at qodana-support@jetbrains.com

alexeyzimarev and others added 15 commits May 26, 2025 19:59
@alexeyzimarev
Address Qodana warnings
362d854 
Remove unused internal API (ReadLogEvents)
@alexeyzimarev
Fix exception message test
e8d25b8
@alexeyzimarev
Fix disposed outside of closure warning
c880dcf
@alexeyzimarev
Refactored to sync
9ce4063
@alexeyzimarev
Simplify obvious names
4c336fc
@oskardudycz
Added crude implementation of using Stream Info Cache for Stream Seco…
68f585e 
…ndary Index
@oskardudycz
Added implementation of the subscription to default secondary index
dee33c6
@oskardudycz
Added tracking of committed sequence in the category stream
a0faa4c
@alexeyzimarev
* Use filtered subscription, we don't index system events
58e92d2 
* Fix the bug in `IsVirtualStream`
@alexeyzimarev
Revert back to non-filtered subscription and filter system events int…
847bd00 
…ernally
@alexeyzimarev
Refactoring
55413c1
@sakno
Reuse the writable connection for query
afad2b6
@sakno
Fixed disposable pattern implementation
d2f9105
@alexeyzimarev
Support in-flight messages, align data types, removed unnecessary abs…
1e91155 
…tractions
@alexeyzimarev
Make batch size and live buffer size configurable internally
040e934
oskardudycz and others added 27 commits July 3, 2025 09:06
@oskardudycz
Set stream metadata in secondary stream index
1b2ac7c
@oskardudycz
Added storing truncate_before and read ACL roles in secondary indexes
9f7d5c6
@oskardudycz
Stored additional information for stream secondary indexes
f060d7e
@oskardudycz
Removed doubled flush in DefaultIndexProcessor
ac99bae
@oskardudycz
Bumped the Quack version to apply the fix of handling nullable values
1c95fef
@oskardudycz
Added test for indexing stream without category
07d2e03
@oskardudycz
Fixed mapping of secondary index message data to correctly point to o…
750aec9 
…riginal event in the UI
@oskardudycz
Merge branch 'master' into indexing/secondary-indexes
6a5d204
@oskardudycz
Fixed skipped first event when read backwards in the UI
f5b0a64
@oskardudycz
Merge branch 'master' into indexing/secondary-indexes
0a3387d
@oskardudycz
Post merge fixes to the solution file
4bebe52
@oskardudycz
Added back load testing project
e9a6212
@oskardudycz
Fixed the last number of index all stream to subscribe correctly from…
1410a0c 
… the first event
@oskardudycz
Added tests checking DefaultIndexProcessor positions when no events w…
95b8442 
…ere processed yet
@oskardudycz
Fixed the flaky Integration tests setup by using class fixtures inste…
b651699 
…ad of base setup and collection fixtures

Each test will need to have a different setup either way not to mess with each other
@oskardudycz
Merge branch 'master' into indexing/secondary-indexes
a32a55f
@oskardudycz
Fixed reference to secondary indexing load testing project
6bb2a88
@oskardudycz
Added first set of secondary index default reader tests
daf25ac
@oskardudycz
Added more tests for Default Index Reader
13fdfeb
@oskardudycz
Replaced DummyReadIndex with ReadIndexStub encapsulating the setup
206017b
@oskardudycz
Fixed build
3560a2f
@oskardudycz
Extended test suite for reading backward and other methods for Defaul…
9e747a2 
…t Index Reader
@oskardudycz
Added matrix tests for inflight records
75a9d82
@oskardudycz
Adjusted returned next event and last event number values for Reading…
fd3e951 
… forwards and backward from the secondary stream
@shaan1337
Change gap unit to bytes
fcdf6cc
@shaan1337
fix typo
8e870b6
@shaan1337
Fix improper ticks conversion to milliseconds
8ed5c99 
Make lag positive
@CLAassistant
Copy link

CLAassistant commented Aug 1, 2025 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@alexeyzimarev alexeyzimarev deleted the indexing/secondary-indexes branch September 18, 2025 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shaan1337 shaan1337 shaan1337 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@alexeyzimarev @CLAassistant @shaan1337 @oskardudycz @sakno