Adding support for local file scraping

[jonocodes]

It would be nice if this could be used 'offline'. So I could scrape something like 'file://home/me/site/mypage.html'
Or perhaps a way to feed the scraper raw html instead of a url.

[jonocodes]

For now I have a silly workaround, but it still visits a website even if it ignores the content.

class BodyScraperPlugin {

  constructor(body) {
    this.body = body;
  }

	apply(registerAction) {
		registerAction('afterResponse', async ({response}) => {
      
      if (response.headers['content-type'].includes('text/html')) {
        console.log("html AFTER REPONSE " + response.url)

        return this.body
      }

      return response.body
    });
	}
}


    const options = {
      urls: ['https://nodejs.org'],	// dummy since it does not use this content
      directory: saveDir,
      plugins: [ new BodyScraperPlugin(article.content) ]
    };

    var result = await scrape(options);

[aivus]

Hello @jonocodes

file protocol is not supported by got module we use for http requests which means we doesn't support it as well.

I would recommend to run a http server in the directory with needed files.
Simplest way to do it run of python -m SimpleHTTPServer 9000

[jonocodes]

Yeah, behind a server this works. But I'm trying to do this in a self contained script against local files. Having to start and stop a server just to read a text file is too much overhead for my simple task.

[aivus]

Module, as per name website, is not supposed to work with local files.