Extract SharedContext interface

[idanya]

Extract an interface for SharedContext and use the interface in the pipeline.
Make current SharedContext an in-memory interface implementation.

This will open the ability to have an external context that can be persistent and shared with other services. Also, will lower memory consumption.

[scottsdevelopment]

Hello,

After thinking about current use case scenarios that I am looking to accomplish, my suggestion would be to move the SharedContext to the PipelineRunner constructor, or possibly encapsulate the arguments into a Pipeline class and have the PipelineRunner accept multiple pipelines, piping the context to each following run.

What I am planning on:

• CandleCache warmer, that preloads the SharedContext with data from the MongoDbSource. The CacheWarmPipeline or some other crafty name.
• Multiple Pipeline, one for pre-processing and validation of existing data, and a hand off to two, a sequential Pipeline that handles the incoming data and executes the Strategy and Sink.

Let me know your thoughts on this.

Kind regards,

[idanya]

Encapsulating a Pipeline into a Pipeline and having the PipelineRunner accept multiple Pipeline (as an array) sounds good to me. That will enable us to create multiple pipelines running sequentially and share data between them.
We'll also need to enrich PipelineSpecification to accept multiple Pipeline instances.

So the action items as I see them are:

1. Create a Pipeline class that will hold the Source, processor, and terminator.
2. Extend PipelineRunner to accept a list of Pipeline
3. Move PipelineRunner.run() to Pipeline and implement a higher level run() on the PipelineRunner level (that invokes each Pipeline internal run(), sequentially.
4. PipelineSpecification would hold a list of the new Pipeline objects
5. inject SharedContext to the PipelineRunner and pass it to each Pipeline.run() call.