🕷️ Spidey 🕸️

Fully working search engine which can search by text or voice for webpages or images, it also has its own crawler, indexer, ranker, query engine and UI

👷 Dependencies

• java
• javac
• mysql-server
• Nodejs
• npm
• angular-cli

📦 Libraries

• jsoup
• jfreechart
• lucene-snowball
• mysql-connector-java
• progressbar

📝 Modules

• Crawler
• Indexer
• Ranker
  • Page Ranker
  • Query Results Fetcher
• Query Engine
• Performace Analysis
• DataBase Manager
• UI

🚀 Running Main Modules

1. Clone using vesion control using any IDE ,e.g: IntelliJ IDEA
2. Install mysql server sudo apt-get install mysql-server
3. Login to root user sudo mysql -u root -p
4. Create new user named admin with admin password CREATE USER 'admin'@'localhost' IDENTIFIED BY 'admin';
5. Grant all privileges to user admin on test_search_egnine database GRANT ALL PRIVILEGES ON test_search_engine.* TO 'admin'@'localhost';
6. Run Create Database
7. Run Crawler (preferred to wait 5 minutes before moving to the next step)
8. Run Indexer
9. Run Query Processor (wait until it lauches to move to the next step)
10. Run Performance Analysis

🎨 Running Front-End

1. Run Query Processor (wait until it lauches to move to the next step)
2. Open front end folder cd front_end/
3. Install packages npm install
4. Run ng serve for a dev server. Navigate to http://localhost:4200/. The app will automatically reload if you change any of the source files

📝 Note: use google chrome because we use its API in speech recognition.

📈 Analysis

Analysis with 500 pages

This analysis has been made with this parameters using Performance Analysis module

public static final int MAX_WEBSITES = 500;
public final static int SECONDS_TO_SLEEP = 5;

1. How many simultaneous search requests can your solution handle?

2. How is the latency of your solution affected by the number of simultaneous search requests?

3. How is the search request latency of your solution affected by the number of web pages crawled?

4. How is the search request latency of your solution affected by the size of the index table?

5. How is the search request latency of your solution affected by the ranking process?

Analysis with 5000 pages

This analysis has been made with this parameters using Performance Analysis module

public static final int MAX_WEBSITES = 5000;
public final static int SECONDS_TO_SLEEP = 120;

1. How many simultaneous search requests can your solution handle?

2. How is the latency of your solution affected by the number of simultaneous search requests?

3. How is the search request latency of your solution affected by the number of web pages crawled?

4. How is the search request latency of your solution affected by the size of the index table?

5. How is the search request latency of your solution affected by the ranking process?

🔥 Spidey On The Action

Search By Text

Navigate Result Pages

Search By Voice

Search For Images

Night Mode

©️ Developers

Name	Email
Abdulrahman Khalid Hassan	[email protected]
Shady Fahmy Abd Elhafez	[email protected]
AbdElRahman Muhammad Ahmad ElGamil	[email protected]
Yosry Mohammad Yosry	[email protected]