Replace DataPusher #35
Description
DataPusher should be replaced
CKAN DataPusher is not a good choice for pushing data into CKAN datastore.
One core reason to replace DataPusher is that it is complicated to setup and extremly slow. Some more arguments are listed here.
I identified two candidates to replace DataPusher.
ckanext-xloader
Pros
- Comes as a CKAN extension and is easy to setup
- Up to 10x faster than Datapusher
Cons
- Needs to be included in the CKAN-SDDI image
- Can only be autoscaled by scaling CKAN instances
- All columns defined as text, and the Data Publisher will need to manually change the data types in the Data Dictionary and reload the data again.
DataPusher+
Pros
- Built on qsv, an ultra fast processing tool wirtten in Rust.
- Lives in a separate container and can be scaled individually
Cons
- Complicated setup