add dataloader with docs

[bghira]

No description provided.

[Copilot]

Pull Request Overview

This PR adds a new TarDataLoader with comprehensive documentation and tests for streaming TAR file entries from webshart datasets.

Key changes:

• New TarDataLoader and BatchDataLoader classes for streaming dataset entries
• Streaming infrastructure for processing TAR files efficiently
• Complete test suite for dataset streaming functionality

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/dataloader.rs	Implements TarDataLoader and BatchDataLoader for streaming TAR entries with buffering
src/streaming.rs	Core streaming infrastructure for local and remote TAR files
tests/test_dataset_streaming.py	Comprehensive test suite covering streaming functionality and error handling
src/lib.rs	Adds new dataloader exports and dependencies
src/metadata.rs	Adds files() method to ShardMetadata for accessing file information
python/webshart/init.py	Exports TarDataLoader for Python users
README.md	Documents TarDataLoader usage example
Cargo.toml	Adds rayon and pythonize dependencies
src/extract.rs	Removes unused variable
src/discovery.rs	Adds get_hf_token() method

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Infinite recursion: the trait implementation calls itself instead of the actual implementation method. This should call RemoteTarStreamer::stream_entries(&self) from line 254.

Suggested change

	impl TarStreamer for RemoteTarStreamer {
	fn stream_entries(&self) -> Result<Box<dyn Iterator<Item = Result<TarFileEntry>>>> {
	self.stream_entries()
	RemoteTarStreamer::stream_entries(self)

[Copilot]

Magic numbers 10 and 5 should be defined as constants. Consider const EARLY_ERROR_THRESHOLD: usize = 10 and const MAX_EARLY_ERRORS: usize = 5.