Skip to content

Ungoliant v1.0.0
Compare
Choose a tag to compare
@Uinelj Uinelj released this 07 Sep 17:31
v1.0.0
a9a6421
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Ungoliant v1.0.0

This is the first release of Ungoliant, a project that provides tools to generate corpora from CommonCrawl.
Ungoliant also includes already established pipeline(s), in particular to generate OSCAR-like corpora.

Ungoliant also replaces goclassy.

Get the release from the Releases tab or via cargo: cargo install ungoliant.

Features

  • Feature: Downloading of CommonCrawl. Ungoliant features an asynchronous multithreaded downloader that is faster than the previous solution used for OSCAR.
  • Feature: Generation of both OSCAR v1 and OSCAR v1.1 corpora. The new OSCAR v1.1 is a backward compatible corpus including metadata.
  • Feature: Deduplication using runiq. Ungoliant currently uses a fork that enables library access.
  • Feature: Splitting, compression and packaging. These three operations facilitates generated corpora preparation for ulterior distribution. Note that these operations are not yet performed on the fly, and may need huge free space.

Changes

These changes are feature evolutions from goclassy

  • Pipelines and tools are available through the ungoliant command-line interface.
  • Downloading and compilation of fasttext is not needed anymore. Be sure to have cmake installed if you plan on compiling Ungoliant yourself.
  • General performance improvements when using implemented pipelines. This has been possible by using a multithreading of a finer granularity, using rayon.rs.
Assets 2
Loading