Refactor ETL threading to support consolidateOnShutdown on multiple dataStores for a PV #361
Conversation
…ches in DefaultConfigService. Instead try using Hz predicates ( which seemed to work well with GetDataAtTime ) This is a precursor to having reassignAppliance functionality. Some of the BPLs had to change as we need to set the typeinfo first before registering to the appliance
Show details based on StoragePlugin name rather than position in the array. This is needed for supporting varying datastore arrays where some PVS have STS -> MTS -> LTS and other may have STS -> SSD -> MTS -> LTS. Change some class names and variable names to reflect that ETL happens in stages. For example, ETLStage and ETLStageDetails Also some minor UI changes
We added support for this using Named Variables
Move away from one thread per lifetime to using virtual threads and sequencing one ETL stage after another. This should help support consolidateOnShutdown on multiple datastores. For example, we should now be able to support consolidateOnShutdown on both the STS and MTS.
We don't need to poll anymore
Let's see if the distributed query functions just as fast
| } | ||
| logger.debug("Running ETL for {} with stages {}", pvName, etlStages.getStages().size()); | ||
| try(ScheduledThreadPoolExecutor scheduleWorker = new ScheduledThreadPoolExecutor(1)) { | ||
| try(ExecutorService theWorker = Executors.newVirtualThreadPerTaskExecutor()) { |
There was a problem hiding this comment.
Is there a way to cancel these? I had a request for more finegrained control on the ETL, i.e. to trigger early or halt for some time.
Would also be useful for unit testing maybe.
There was a problem hiding this comment.
Re: Is there a way to cancel these?
We have a Future; so we should be able to cancel if we hang on to the future in the ETLStage/ETLStages. Should be safe to cancel mid-way I think.
Re: to trigger early
The consolidate BPL has this in some sense so we may just need to use this and maybe fix the nextETLStart. Or maybe just manipulate the nextETLStart
Re: halt for some time
Yeah; this might be useful. On a global scale, we can do this using named variables ( similar to what we do for retrieval ). On a per PV scale, we can maybe accomplish this by manipulating the nextETLStart
Let me know if this is not what you meant.
There was a problem hiding this comment.
Need to make sure that pause/resume/connect/disconnect are still working after a component restart etc.
There was a problem hiding this comment.
Checked pause/resume/connect/disconnect are still working after a component restart
There was a problem hiding this comment.
Anything more needed here? Don't mind pushing these largish changes onto a 2.1 branch.
Pause/resume using bulk operations.
|
|
Will resend this pull request once I merge properly. The conflicts are not complex but I have my repos in a messy state and I may need to redo this PR. |
This is a semi-largish change. While the unit tests seem to be passing, this needs to be tested in a production environment for impact. I plan to test this Jul-Aug during our downtime and hope for an official release in Sep.