Ability to add headers to source ( better integration with Jina AI)

[angelotc]

Is your feature request related to a problem? Please describe.
We can assign a url to the source. It would be nice if we could also pass in an headers such as API key to use our jina ai credits.

Describe the solution you'd like
Perhaps a source_config param?

    smart_scraper_graph = SmartScraperGraph(
        prompt="Find the yelp link, name, website, number of average yelp reviews , summary of yelp_reviews, specialties, phone,  and their website",
        source=f"https://s.jina.ai/{url_encoded_query}",
        source_config= {
            headers: {
                "Authorization": f"Bearer {JINA_API_KEY}"
            }
        },
        config=graph_config,
        schema = Contractor
    )

Describe alternatives you've considered
n/a

[VinciGit00]

Why you want embeddings? I never noticed about that and as I can see from the website Juba is used for embeddings

[angelotc]

Why you want embeddings? I never noticed about that and as I can see from the website Juba is used for embeddings

Less so about embeddings - moreso of the fact that they offer free / low-cost web search and LLM friendly site data.

https://jina.ai/reader/
https://www.youtube.com/watch?v=QxHE4af5BQE&t=1035s

[dosubot]

Hi, @angelotc. I'm Dosu, and I'm helping the Scrapegraph-ai team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The issue proposes adding a source_config parameter to include headers like API keys for better Jina AI integration.
• This feature aims to optimize the use of Jina AI credits by enhancing web search and LLM-friendly site data access.
• @VinciGit00 questioned the need for embeddings, but you clarified the focus is not on embeddings.
• No alternative solutions have been suggested, and the issue remains unresolved.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the Scrapegraph-ai repository by commenting here.
• If there is no further activity, the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!