Implemented optimized Neo4j queries and setup #593
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @northdpole, could you please take a look at this PR and provide feedback? |
|
@northdpole Could you please review this? I have fixed the linting errors. |
|
have you tried running gap analysis with the improved queries? the code looks good but have you managed to benchmark if there's an improvement? you can do so by importing only 2 standards and then calculating gap analysis for these two and measuring performance before/after your changes |
|
@northdpole Could you please review this? |
i am, can you please answer my question? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title: Improve Map Analysis Performance with Optimized Neo4j Queries
Description:
Summary: This PR implements several improvements to make the Map Analysis feature faster and more resource-efficient. It includes optimized Cypher queries, indexing for faster lookups, caching mechanisms to avoid redundant computations, and usage of APOC procedures for parallel processing.
Key Changes:
Added Indexes on Standard(name) and Framework(name) to reduce NodeByLabelScan operations.
Refined Cypher Queries using WITH and LIMIT clauses for smaller, more efficient batches.
Implemented Caching for frequently accessed paths, minimizing redundant path calculations.
Utilized APOC for timeboxed queries and parallel batch processing.
Why: Addresses Issue #587, ensuring faster response times and lower resource usage.
Testing:
Verified performance gains via PROFILE in Neo4j Browser.
Measured query execution time before and after applying indexes and caching.
Next Steps:
Further refine queries if new performance bottlenecks arise.
Update documentation as needed.
How to Test:
Pull these changes and install any required dependencies (e.g., python-dotenv, neo4j).
Start Neo4j and run setup_database.py to create nodes/indexes.
Run query_database.py and observe improved execution times using PROFILE or logs.