Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document BigQuery RPC settings #24638

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+53 −0
Open

Document BigQuery RPC settings #24638

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions docs/src/main/sphinx/connector/bigquery.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,59 @@ a few caveats:
- Enable using Apache Arrow serialization when reading data from BigQuery.
Please read this [section](bigquery-arrow-serialization-support) before using this feature.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove the "Please" here .. in a separate commit if you want

- `true`
* - `bigquery.channel-pool.initial-size`
- The initial size of the connection pool, also known as a channel pool,
used for gRPC communication.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we explain gRPC and RPC somewhere earlier in the doc? If not .. we might want to .. or at least link to some external doc

- `1`
* - `bigquery.channel-pool.min-size`
- The absolute minimum size of the connection pool, also known as a channel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The absolute minimum size of the connection pool, also known as a channel
- The minimum size of the connection pool, also known as a channel

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also ... size in what .. maybe better

The minimum number of connections in the connection pool, ...

pool, used for gRPC communication.
- `1`
* - `bigquery.channel-pool.max-size`
- The absolute maximum size of the connection pool, also known as a channel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The absolute maximum size of the connection pool, also known as a channel
- The maximum size of the connection pool, also known as a channel

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above also

pool, used for gRPC communication.
- `1`
* - `bigquery.channel-pool.min-rpc-per-channel`
- Threshold to start scaling down the channel pool.
When the average of the maximum number of outstanding RPCs in a single
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simplify to

Suggested change
When the average of the maximum number of outstanding RPCs in a single
When the average of outstanding RPCs in a single

minute drop below this threshold, channels will be removed from the pool.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
minute drop below this threshold, channels will be removed from the pool.
minute drop below this threshold, channels are removed from the pool.

- `0`
* - `bigquery.channel-pool.max-rpc-per-channel`
- Threshold to start scaling up the channel pool.
When the average of the maximum number of outstanding RPCs in a single
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as preceding

minute surpass this threshold, channels will be added to the pool.
- `2147483647`
* - `bigquery.rpc-retries`
- The maximum number of retry attempts to perform for the RPC calls.
If this value is set to `0`, the logic will instead use the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If this value is set to `0`, the logic will instead use the
If this value is set to `0`, the value from `bigquery.rpc-timeout` is used.

`bigquery.rpc-timeout` configuration value to determine retries.
In the event that both the `bigquery.rpc-retries` and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too many "both"

`bigquery.rpc-timeout` values are both `0`, the logic will not retry.
If this value is positive, and the number of attempts exceeds
`bigquery.rpc-retries` limit, the logic will give up retrying even if
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as preceding

the total retry time is still lower than `bigquery.rpc-timeout`.
- `0`
* - `bigquery.rpc-timeout`
- Timeout [duration](prop-type-duration) on when the retries for the
RPC call should be given up completely. The higher the timeout, the
more retries can be attempted. If this value is `0s`, then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then the value from bigquery.rpc-retries is used.

the logic will instead use `bigquery.rpc-retries` to determine retries.
In the event that `bigquery.rpc-retries` and `bigquery.rpc-timeout`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retry is deactivated when both .. and .. are 0.

values are both `0`, the logic will not retry.
If this value is positive, and the retry duration has reached the timeout
value, the logic will give up retrying even the number of attempts is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
value, the logic will give up retrying even the number of attempts is
value, retries stop.

lower than the `bigquery.rpc-retries` value.
- `0s`
* - `bigquery.rpc-retry-delay`
- The delay [duration](prop-type-duration) before the first retry attempt
for RPC calls.
- `0s`
* - `bigquery.rpc-retry-delay-multiplier`
- Controls the change in delay before the next retry.
The retry delay of the previous call is multiplied by the
`bigquery.rpc-retry-delay-multiplier` to calculate the retry delay
for the next RPC call.
- `1.0`
* - `bigquery.rpc-proxy.enabled`
- Use a proxy for communication with BigQuery.
- `false`
Expand Down
Loading