Slice Callback Function with Batch Interval Functionality

[epociask]

Addressing issue #128

Added additional functionality for batch interval messages with a separate callback function that returns a slice of kinesis.Record than a single reference to a consumer.Record....

• Added WithBatchSecondInterval function in Options to assign second interval for reading
• Added ScanShardWithIntervalBatching function to send a slice of records upon a conditional evaluation of elapsed time
• Added batchInterval data member to Consumer struct in Consumer.go, DEFAULT=0

To use batch interval consumption, just specify your callback function as func([]*kinesis.Record) error and pass it as a parameter to ScanBatch

[harlow]

I'd rather not break the public API if possible. lemme put some thought into this one

[epociask]

Sounds good : )

[harlow]

steram -> stream

[harlow]

Hi thanks for submitting the PR; I'm struggling a little with the amt of duplication between the two function; my fear is "drift" between the two makes this code path harder to maintain over time.

Taking a step back I'm curious if something like this is possible:

// shape of the code
func Scan(ctx, fn ScanFn)
	items = ScanBatch(ctx, func(items) {
		for each items 
			fn(item)
	})

func ScanBatch(ctx, fn ScanBatchFn)
       // 

so the Scan func passes off to the ScanBatch func (which would eliminate the duplication of the two loops). I'll play around with this a little and see if anything emerges.

Or potentially the inverse where we add a convenience function ScanBatch which uses Scan

func Scan(ctx, fn ScanFn)
	// original function

func ScanBatch(ctx, fn ScanBatchFn)
       c.Scan()
       // do the batching here

[harlow]

I think this should be ScanBatchFunc

[harlow]

what is this doing? is it different from the scan interval?

[epociask]

Effectively just a poorer duplication of scan interval, when I wrote this initially I overlooked that component of the code... thank you for pointing this out

[harlow]

if you run a linter on this code you'll see a bunch of warnings around CamelCase vs snake_case