Skip to content

0.1.0
Compare
Choose a tag to compare
@jhpoelen jhpoelen released this 28 Jun 19:18
0.1.0
afd1f7d

Features

  1. support for remote gzipped tarballs . To help publish many files to existing publication platforms (e.g., Internet Archive, Zenodo), data dirs can be put into tar balls like preston-aa.tar.gz , where "aa" are the first two characters of a content hash of files contained in the tar ball. For example publication, see Poelen, Jorrit H. (2019). A biodiversity dataset graph: Biodiversity Heritage Library (BHL) (Version 0.0.1) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.3251134 and https://archive.org/details/preston-bhl .
  2. command line progress bar. To help indicate that stuff is being downloaded show a progress bar like:
$ preston clone https://archive.org/download/preston-bhl
[https://archive.org/download/preston-bhl/2a5de79372318317a382ea9a2cef069780b852b01210ef59e06b640a3539cb5a] 100.0% of 78 bytes at 0.07 MB/s completed in < 1 minute
[https://archive.org/download/preston-bhl/89926f33157c0ef057b6de73f6c8be0060353887b47db251bfd28222f2fd801a] 0.8% of 236 MB at 0.43 MB/s ETA: +9 minutes
  1. introduce clone command to explicitly clone a remote Preston repository. Can be chained to include many repositories. E.g.,
$ preston clone https://example.org https://archive.org/download/preston-bhl

would first try to get index / provenance and associated content via https://example.org, then try https://archive.org/download/preston-bhl. Allows for distributing content across arbitrary number of remotes.

  1. make local data/tmp dir configurable. Local data/tmp dir can now be configured with option --data-dir and --tmp-dir respectively.

Improvements

  1. stop processing when downstream pipe has closed (see #13)

Bugs

Assets 4
Loading