CrawlerRust

Features

• Send HTTP GET requests using the reqwest crate.
• Parse HTML responses using the scraper crate.
• Extract specific data from the parsed HTML.

Common links to use

• https://www.rust-lang.org/

Prerequisites

• Rust 1.54.0 or later
• Cargo 1.54.0 or later

Dependencies

• reqwest for sending HTTP requests.
• scraper for parsing HTML and extracting data.
• tokio for asynchronous programming.

Usage

1. Clone the repository.
2. Navigate to the project directory.
3. Run the project using the command cargo run.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

TODO

• GET https://crates.io/crates/rust-bert up and running for analysis