Repositories list

• KeywordAnalysis

  Public
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

  wordcountkeyword-extractioncluster-analysis+ 1commoncrawl

  11•57•0•0•Updated Jan 28, 2024Jan 28, 2024

• CI-HiBench

  Public
  Big Data benchmark from Intel called HiBench

  Shell

  •0•0•0•0•Updated Mar 5, 2018Mar 5, 2018

• CommonCrawlDocumentDownload

  Public
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types for mass-testing of frameworks like Apache POI and Apache Tika

  Java

  •

  BSD 2-Clause "Simplified" License

  •17•0•0•0•Updated Feb 9, 2018Feb 9, 2018

• cdx-index-client

  Public
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/

  Python

  •

  MIT License

  •48•1•0•0•Updated Feb 9, 2018Feb 9, 2018

• HiBench

  Public
  HiBench is a big data benchmark suite.

  Java

  •

  Other

  •772•0•0•0•Updated Jan 31, 2018Jan 31, 2018

• spark-Jupyter-AWS

  Public
  A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support

  Jupyter Notebook

  •18•1•0•0•Updated Apr 24, 2017Apr 24, 2017

• dkpro-c4corpus

  Public
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate removal, language detection, and near-duplicate removal.

  Java

  •

  Apache License 2.0

  •7•0•0•0•Updated Dec 20, 2016Dec 20, 2016

• common_crawl_index

  Public
  Index URLs in Common Crawl

  Python

  •47•0•0•0•Updated Sep 6, 2016Sep 6, 2016

• commoncrawl-examples

  Public
  A library of examples showing how to use the Common Crawl corpus.

  Java

  •43•0•0•0•Updated Aug 5, 2016Aug 5, 2016