Kinetic Refactor
Pre-release
Pre-release
·
23 commits
to version3_package_refactor
since this release
RELEASE NOTES
- Producer
- refactored to abstract out the StreamWriter interface to support writing to Kinesis and Firehose
- refactored to remove the retry queue entirely
- we determined that the original implementation could lead to an overflow of goroutines (and ultimately unbounded memory issues)
- new implementation utilizes a dispatcher / worker model throttled by a rate limiter
- implemented shard monitoring to periodically check the number of active shards and adjust the producer's rate limiter automatically
- Consumer (previously Listener)
- refactored to abstract out the StreamReader interface to support reading from Kinesis using the aws-sdk-go library and the Kinesis Client Library (KCL) using the Multilang daemon protocol
- Added rate limiting on the GetRecords calls
- Added the KclReader which implements the StreamReader interface and uses KCL to retrieve messages from Kinesis
- Added the ability to checkpoint periodically (on demand and automatically, through configurable parameters)
- Updated stats collection interfaces and added a default stats collector which utilizes rcrowley metrics
- Major refactor to consolidate multiple packages down to a single (main) package
- API clean up by utilizing function option methods pattern to adjust configurable parameters
- Added a testing harness (testexec) which builds an executable binary with a CLI that can be used to test the Producer / Consumer in different modes of operation (read, write, readwrite)