Never get a full result set from a VueJS site using Axios

[rgbjay]

Hi, I am using the package to create a sitemap of a VueJS site (created with Vue CLI not Nuxt) and I never seem to get a full set of results.

The Vue site uses Axios to get its content from a CraftCMS Element API endpoint so I understand that the majority of links won't be present exactly at the point that the page is loaded.

Initially I thought that if I added a delay to the Browsershot instance then I would give the site a chance to load before it crawled it. However this still does not give me a full set of results, the strange thing is that it does appear to be getting some of the links that are generated by the axios call, just not all of them. I've extended the delay to (an extreme) 20 seconds just to see if that made a difference, it didn't.

I have put my code below - due to the issue mentioned here (#315) I have had to include a direct link to my Chromium installation.

Any pointers/help appreciated.

public function generateSitemap($site_id)
    {
        $site = Site::find($site_id);
        $site_url = 'https://' . $site->site_url;
        $storage = Storage::createS3Driver([
            'driver' => 's3',
            'key' => env('AWS_ACCESS_KEY_ID'),
            'secret' => env('AWS_SECRET_ACCESS_KEY'),
            'region' => env('AWS_DEFAULT_REGION'),
            'bucket' => $site->site_url,
        ]);

        $browsershot = (new \Spatie\Browsershot\Browsershot)
            ->setNodeBinary('/usr/bin/node')
            ->setNpmBinary('/usr/bin/npm')
            ->setChromePath('/usr/bin/chromium-browser')
            ->setDelay(5000);

        SitemapGenerator::create($site_url)
            ->configureCrawler(function (Crawler $crawler) use ($browsershot) {
                $crawler->executeJavaScript()
                        ->setBrowsershot($browsershot);
            })
            ->setMaximumCrawlCount(1000)
            ->getSitemap()
            ->writeToDisk('local', 'sitemap.xml');

        $sitemap_file = Storage::disk('local')->path('sitemap.xml');
        $storage->put('sitemap.xml', file_get_contents($sitemap_file), 'public');
        Storage::disk('local')->delete('sitemap.xml');
        return true;
    }