Switch raw dataset indexing from `fsspec` to `obstore`

[bhimrazy]

🚀 Feature

Switch raw dataset indexing from fsspec to obstore

obstore is 2 to 3x faster

Initial experiments resulted in some sort of deadlock in streaming. It worked for indexing but not for streaming — only one thing worked at a time.

Motivation

obstore is 2 to 3x faster for indexing. really useful for large datasets

Follow up to #652, commit 39c6e6e

Pitch

Alternatives

Additional context

[bhimrazy]

Might be useful to inspect multiprocessing start methods when debugging stuck processes.
It seems to work with spawn:

import multiprocessing as mp

mp.set_start_method("spawn", force=True)
print("Start method:", mp.get_start_method())