feat(source-google-sheets): migrate low code #50843
Open
+7,541
−2,356
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…xtractor that matches schema properties with values
…in components resolver in order to get an indexed object of properties where index is the order. - Add partition router to retriver in components resolver so we can slice.
- Remove old SourceClass to temp file to be deleted and - Remove unused transformations - Make extractor ha transform data from component mapping
…anges to discard empty cells in first row
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
octavia-squidington-iii
added
the
area/documentation
|
/format-fix
|
|
/format-fix
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
We want to migrate google-sheets connector relying on google-api-python-client/google-auth-httplib2 to communicate with API to manifest using our HttpRequester and latest CDK features.
How
We removed helper and client file dependencies to use new CDK features like ComponentResolver, DynamicSchema and DynamicStreams.
Review guide
I will use this section to give a quick walkthrough of the manifest components, relations and custom components used in this connector.
Dynamic Streams
The dynamic_streams section is a list of DynamicDeclarativeStream that depends on stream_template and components_resolver we will start with components_resolver.
Components Resolver
The component resolver has its own SimpleRetriever retriever with a partition_router that depends on a static stream
get_spreadsheet_info_and_sheets, this stream is in charge of requesting spreadsheet info to get all the sheets.req:
{{ config["spreadsheet_id"] }}?includeGridData=false&alt=jsonNow, with the sheet IDs we use the component resolver retriever/requester to obtain information, mainly the first row of data and rows_count
req:
{{ config["spreadsheet_id"] }}?includeGridData=true&ranges={{ stream_partition.sheet_id }}!1:1&alt=jsonWith the response, we can start to pass information using the component_mapping like
['properties']['title']['data'][0]['rowData'][0]array (order is important) to match with records (will explains this a few lines below).['properties']['gridProperties']['rowCount']to continue...
User Impact
Can this PR be safely reverted and rolled back?