Skip to content
This repository was archived by the owner on Nov 7, 2019. It is now read-only.
This repository was archived by the owner on Nov 7, 2019. It is now read-only.

Add Morph Run Custom Crawl Model #19

@b5

Description

@b5

From our meeting notes:

  1. we're allowed to make POST calls
    • we'll call some sort commit / finalize function
    • last but not least, POST overview data about the run to datatogether, including details on how >to download data.sqlite payload.
  2. Data Together accepts the post, stores the result, kicks off a task, which uses an "archiving library" (which currently doesn't exist) that conforms the the file, urls, & tabular data formats form the archiving library, downloads that data, posts it to IPFS, and then does a small dance. It also grabs the scraping script from.

Post-Run Payload

ident.archivers.co/jwt?access_token=foo&action=morph_run
api.archivers.co/morph

{
    "jwt": "eiwhofanerjnkgf.aweiofhaeioheaw.asnivocaneo",
    "morphRunId": "329awgknlvorapfevjiaowefha",
    "dateCompleted": "2017-03-9",
    "githubRepo" : "https://github.com/foo/bar",
    "originalUrl: "http://everything.gov.com.au.it.co.uk",
    "sqliteChecksum": "qafhq32ufh4a32uq4eraf3whaudrsfaiaewf"
}

cc @ebenp

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions