Automatic Chunking

[Material-Scientist]

Hi,

I recently had a problem where I got indexCacheBuf corrupt record state: dlen 46134798 slen 21 index 0 rl 46134820 lbuf 46134820 when trying to upload a 40MB file to the KV store, despite setting max_payload to 64MB in the NATS server config.

But maybe instead of increasing the maximum msg size, it would be a better idea to add a way for NATS to automatically chunk the object into 8MB fragments, and only store the references in the KV store.

AFAIK, that's how NATS object store works. So, why not just use the same automatic chunking everywhere (even pub/sub)?

It would push the burden of checking file-sizes and chunking from the user to the messaging system.

Thoughts?

[ripienaar]

That’s what the object store is for :)

Nats isn’t optimised for big payloads 2 or 3MB max is a good rule

[Material-Scientist]

I know. I mentioned it in the post.

I.e. KV or pub/sub could add the NATS object store interface to automatically chunk and store only the references & automatically fetch the data, rather than let the user deal with that.

[Jarema]

pub/sub does not persist any data, is at-most-once,
Using it with JetStream layer is not a way to go.

On the other hand, chunking messages for KV would bring a lot of challenges to KV API.
Maybe we can revisit some of those ideas in the future when batched publishes are implemented.

Meanwhile, you can use request many pattern from our extension libraries

It does not automatically chunk messages, but provides API for requesting chunked responses and strategies to handle chunks termination.