okh-scraper

A stand-alone service that scrapes(/crawls) Open Source Hardware (OSH) projects from different platforms and other hosting technologies.
The collected data conforms to the Open Know-How (OKH) and the Open Dataset (ODS) standards.

Using

How to get a Thingiverse REST API access token

To be able to scrape thingiverse (similar to many other platforms), you first need to generate an API access token, which you then insert into this scrapers configuration, so it can authorize itsself with the API.

NOTE Every few months, you might need to generate a new such token!

How to generate one:

1. create a thingiverse.com account and log in with it
2. go to https://www.thingiverse.com/developers/my-apps
3. click on the "Create an App" button
4. choose "Web App"
5. Fill out the form: choose random values for name and description, click the "I agree to the MakerBot API terms and Privacy Policy" box, and click on the button "CREATE & GET APP KEY" on the top of the page, next to "CANCEL".
6. If all went well, your "APP" will have been created, and you will be presented with its details. We are interested in the App Token value. It is a 32 characters long string like this:
   1234567890abcdef1234567890abcdef
   copy it!

In the next sections, we will see how to use it. Enter your access token into the configuration, replacing the string XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.

We will now learn how to run the scraper:

Setup

Using podman/docker

if which podman > /dev/null
then
  cont_app=podman
elif which docker > /dev/null
then
  cont_app=docker
else
  >&2 echo "ERROR: Neither podman nor docker are in PATH"
fi
if ! which okh-scraper > /dev/null
then
  alias okh-scraper="$cont_app"' run --interactive --volume "$PWD:/home/okh-scraper" oseg/okh-scraper:master'
fi

Compiling from source

You need to install Rust and Cargo, if you don't already have it.

Then you have to get the sources, either through git:

git clone https://github.com/OSEGermany/okh-scraper.git
https://github.com/OSEGermany/okh-scraper/archive/refs/heads/master.zip

or by downloading the sources as a ZIP file and extracting them.

After that, you change to the sources dir and compile and install:

cd okh-scraper
cargo install --locked --path .

NOTE To get a 64bit binary that is portable to all Linux systems, use

# setup (only execute this once)
mkdir -p run
git clone https://github.com/hoijui/rust-project-scripts.git run/rp
run/rp/setup

# building
run/rp/build

Running

mkdir okh-scraper-root
cd okh-scraper-root

cat > config.yml << EOF
database:
  type: file       # (opt) nothing else implemented
  path: ./workdir  # (opt)

scrapers:
  thingiverse.com:
    scraper_type: thingiverse
    config:
      retries: 3   # (opt) scraper specific number of retries
      timeout: 15000  # (opt) scraper specific request timeout in milliseconds [ms]
      access_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  # (req) app access token to use the Thingiverse API
      things_store_root: ./workdir/thingiverse_store/  # (req) Where to store the raw thingiverse API scraping results to
      things_range_min: 2000000  # (opt)
      things_range_max: 2999999  # (opt)
EOF

1. Fill out the config file (config.yml)
2. Make sure that the dir specified in the configuration under database:path exists, in our case: mkdir ./workdir
3. Start the scraper: okh-scraper
4. Results will (hopefully) start trickling in under ./workdir.

It will continuously collect and update OSH project data, found on the supported and configured platforms and other locations. The fetched and the converted data, plus the related scraping meta-data, get stored in structured, text based file formats -- JSON, TOML, YAML and Turtle -- and committed and pushed to a git repo. That repo is then synced with its forks, which are directly pushed to by other instances of this scraper. This basically constitutes a distributed scraping mechanism, if configured correctly: The different scraper instances should ideally al fetch all platforms, but different "sections" of the total of the hosted projects. For example in the case of Thingiverse, which as of Early 2025 hosts about 3 million projects, one crawler would grase the ID range of 0 to 499'999, the second one from 500'000 to 999'999, and so on.

okh-scraper --verbose

Testing

To run unit-, doc- and integration-tests:

run/rp/test