json interpreter?

[MarkJaroski]

I'm working on a site which has a json snippet for each event:

<script type="application/ld+json">{	"@context": "http://schema.org",
							  	"@type": "Event",
							  	"name": "Emily Jane & NikoLaLa (duo piano-voix) à Morges, au La Rive Lounge bar",
							  	"startDate": "2023-3-18T18-18-00-00",
							  	"endDate": "2023-3-18T22-22-00-00",
							  	"image":"https://www.dukes-bar.ch/wp-content/uploads/2023/03/2023-03-18-Emily-NicoLaLa-A4-reduit-300x208.jpg",
							  	"description":"Emily Jane and NikoLaLa are a duo of voice and piano, formed to take you on a musical journey through time starting in the 1940s and ending in the present",
							  	
							 }</script>

Which is a pretty interesting way to handle things.

[jakopako]

Yeah I've seen this too. Until know I was too lazy to implement this in goskyr because it was always possible to extract the same information from the html. But it's definitely worth considering. I'll make an issue in goskyr linking to this discussion.

[MarkJaroski]

I've found another one, and even more interesting. There's a festival being put on in part by EPFL, and as you might guess they've gone all the way with semantic data. The programme is assembled with JavaScript in the DOM from their events API!

So here's the question part: do you think it would be better to add this functionality to Goskyr, or to create a completely new tool?

[jakopako]

That's a very good question. I'm kind of indecisive at the moment. On the one hand it would probably make sense to have this functionality in goskyr. Especially for the first case you mentioned where json snippets are embedded in a website. We could probably also reuse some functionality like date parsing.

On the other hand I am not sure if it is worth the extra complexity or if it is better to separate those cases at the risk of duplicating some functionality. I am also not yet entirely sure what we would need to make this work with the event-api. Probably there'd need to be some kind of translation of the field names, too. Then again maybe it wouldn't be as complex as I think. For instance the configuration of a 'json-scraper' might have the same fields as the configuration now!? With the only difference that a selector will be a json path.

What do you think? What are your thoughts on this?

[MarkJaroski]

From what I can see in the goskyr code it does the selection in just one or two places, so I'm leaning more toward your second option. If you don't mind I think I'd like to take a shot at implementing it.

[jakopako]

Of course, go for it! I'm very happy if other people contribute to the code base!

[MarkJaroski]

Paginatination might be interesting ...

[jakopako]

Yeah true, pagination might be a bit more difficult. Depending on whether the 'next' url is contained somewhere in the json or just the page nr and max pages/items.

[MarkJaroski]

OK, I'm on leave for a couple of days (for Cully Jazz), so during the daytime I'll make a play branch, and try some things out. Then I'll come back here to discuss the format additions to the config.yml file.

[jakopako]

Nice, enjoy!

[MarkJaroski]

I'm pleased to announce that I have working code:

https://github.com/MarkJaroski/goskyr/tree/json-experimental

It seems to work with this config:

scrapers:
  - name: DukesBar
    url: https://www.dukes-bar.ch/evenements/
    item: div.event
    fields:
      - name: type
        type: text
        value: concert
      - name: "sourceUrl"
        value: "https://www.dukes-bar.ch/evenements/"
      - name: type
        type: text
        value: concert
      - name: "sourceUrl"
        value: "https://www.dukes-bar.ch/evenements/"
      - name: title
        type: text
        location:
          selector: div.evo_event_schema > script
          json_selector: "//name"
      - name: location # this is currently an itinerant venue
        type: text
        location:
          selector: div.evo_event_schema > script
          json_selector: "//name"
          regex_extract:
            exp: "(?i)(lounge bar|tacos bar|restaurant la rive)"
      - name: city # this is currently an itinerant venue
        type: text
        location:
          selector: div.evo_event_schema > script
          json_selector: "//name"
          regex_extract:
            exp: "(?i)(lausanne|morges|mex)"
      - name: comment
        type: text
        location:
          selector: div.evo_event_schema > script
          json_selector: "//description"
      - name: date
        type: date
        components:
          - covers:
              day: true
              month: true
              year: true
              time: true
            location:
              selector: div.evo_event_schema > script
              json_selector: "//startDate"
            transform: # for some reason the hour part is doubled
              - type: regex-replace
                regex: "T[0-9]{2}-"
                replace: "T"
            layout: ["2006-1-2T15-04-00"]
        date_location: "Europe/Berlin"

There are some issues with the target site's json, so there are a couple of hacks in the goskyr code to clean it up.

[MarkJaroski]

There are some trade-offs to consider, and I don't yet know what to do with a RESTfull API. It might just work, or it might require work. Hard to say.

[jakopako]

Thanks, awesome! I'll test this as soon as I can!

[jakopako]

Sorry that it takes me so long,I was on holidays. I'll look into it next week at the latest!

[MarkJaroski]

It's OK. No problem at all. I was at a team retreat in Budapest last week anyway, and am only now getting back to some kind of normalcy.

…

On Wed, 3 May 2023 at 10:32, jakopako ***@***.***> wrote: Sorry that it takes me so long,I was on holidays. I'll look into it next week at the latest! — Reply to this email directly, view it on GitHub <#223 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATLBWFXYMKMQKUAUOAQNLXEIJYTANCNFSM6AAAAAAV7PXNKQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[jakopako]

So I just looked at it and it looks good. I just had to make some minor changes related to some edge cases and error handling.

func extractJsonField(p string, s string) (string, error) {
	extractedString := s
	if p != "" {
		// HACK: json has some instances of meaningfull whitespace
		// (\n in values). Let's get rid of them
		spacecleaner := regexp.MustCompile(`\s+`)
		s = spacecleaner.ReplaceAllString(s, " ")
		// HACK: a dangling comma is another common json mistake
		regex := regexp.MustCompile(`,\s*}`)
		s = regex.ReplaceAllString(s, " }")
		doc, err := jsonquery.Parse(strings.NewReader(s))
		if err != nil {
			return "", fmt.Errorf("JSON: %+v : %s", err, s)
		}
		node := jsonquery.FindOne(doc, p)
		extractedString = fmt.Sprintf("%v", node.Value())
	}
	return extractedString, nil
}

This would be the modified function. (I didn't know how to quickly update your branch as I don't have permission)

There are still some other cases where json parsing comes into play but I think it's fine to merge this (partial) feature now and proceed in small steps rather then handling all the cases now. Just for the record, right now I can see two other cases involving json parsing:

1. A website contains one big json chunk with all the events inside, eg moods.ch
2. A RESTfull api that returns 'pure' json.

We can look at those two cases separately. So if you agree you can make a pull request with your current changes (+ my mods). Thanks for your work! :)

[MarkJaroski]

I'll try to get it in tomorrow. It's a really busy time of year at my work, so I haven't been around much.

[jakopako]

Sure, no worries!