[Feature Request] Add a hook to customize "wait_for_load_state" behavior

[moonk-banksalad]

Is your feature request related to a problem? Please describe.
Hello, I am trying to scrap a SPA page where "domcontentloaded" or "networkidle" event does not guarantee content readiness.

Describe the solution you'd like
One solution would be adding a way to pass a hook function which gets called right after opening a web page and before getting content (https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/scrapegraphai/docloaders/chromium.py#L88). If the playwright.Page is passed to this hook, a caller can customize how to determine readiness of the page content.

Describe alternatives you've considered
I am currently trying to workaround this limitation by monkey patching playwright functions.

Additional context
N/A

[VinciGit00]

hi @moonk-banksalad,
we would be glad if you implement this feature

[dosubot]

Hi, @moonk-banksalad. I'm Dosu, and I'm helping the Scrapegraph-ai team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You suggested adding a hook function to the "wait_for_load_state" process in Playwright for better SPA content readiness management.
• This feature would allow executing a custom function post-page load, offering more flexibility than existing events.
• Currently, you're using a workaround by monkey patching Playwright functions.
• @VinciGit00 supported the idea and encouraged you to implement it, which you acknowledged.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the Scrapegraph-ai repository by commenting here.
• If there is no further activity, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!