Skip to content

Releases: tijme/not-your-average-web-crawler

1.5.0

04 Jun 22:39
Compare
Choose a tag to compare

New feature(s):

  • Scope option tld_must_match

Improvement(s):

  • Renamed domain_must_match scope option to hostname_must_match

Fixed bug(s):

  • Invalid subdomain check in complies_with_scope method

PyPi: https://pypi.python.org/pypi/nyawc/1.5.0

1.4.7

11 May 19:53
Compare
Choose a tag to compare

Improvement(s):

  • PyPi reStructuredText parsing
  • Removed reStructuredText lint dependency

PyPi: https://pypi.python.org/pypi/nyawc/1.4.7

1.4.6

11 May 19:06
Compare
Choose a tag to compare

New feature(s):

  • SVG scraping
  • CSS scraping

PyPi: https://pypi.python.org/pypi/nyawc/1.4.6

1.4.5

02 May 19:44
Compare
Choose a tag to compare

Improvement(s):

  • HTML scraping performance

Fixed bug(s):

  • HTML scraper did not use <base> element

PyPi: https://pypi.python.org/pypi/nyawc/1.4.5

1.4.4

01 May 21:34
Compare
Choose a tag to compare

Fixed bug(s):

  • Missing argument in null route callback

PyPi: https://pypi.python.org/pypi/nyawc/1.4.4

1.4.3

30 Apr 18:36
Compare
Choose a tag to compare

New feature(s):

  • JSON scraper

Fixed bug(s):

  • Invalid reference to queue on HTTP error

PyPi: https://pypi.python.org/pypi/nyawc/1.4.3

1.4.2

28 Apr 19:12
Compare
Choose a tag to compare

Fixed bug(s):

  • Folder traversal not working correctly
  • Multiple forms on the same page couldn't be scraped

PyPi: https://pypi.python.org/pypi/nyawc/1.4.2

1.4.1

26 Apr 20:07
Compare
Choose a tag to compare

Fixed bug(s):

  • Undefined queue items on crawler sigint

PyPi: https://pypi.python.org/pypi/nyawc/1.4.1

1.4.0

26 Apr 19:02
Compare
Choose a tag to compare

Improvement(s):

  • Refactored queue to reduce time complexity

Fixed bug(s):

  • Suddenly disappearing cookies
  • Slow backtrack in regular expressions
  • Infinite crawling recursion due to random input data

PyPi: https://pypi.python.org/pypi/nyawc/1.4.0

1.3.0

05 Apr 20:15
Compare
Choose a tag to compare

New feature(s):

  • Autofill forms with random data
  • Added form autofill callbacks
  • Added custom headers option

PyPi: https://pypi.python.org/pypi/nyawc/1.3.0