Uploading client-side-encrypted data for loading into Redshift

[edgao]

I'm trying to use the AmazonS3EncryptionClientV2 to upload an encrypted file to S3, and then run a COPY .... ENCRYPTED command to load that file into Redshift. However, Redshift is not compatible with the encryption format in this SDK. I'm following the instructions here, which simply links to the AWS SDK docs. There are two errors that I'm seeing:

The first is that the S3 object created by the SDK does not have the x-amz-meta-x-amz-key metadata set. It looks like the encryption library is now setting x-amz-meta-x-amz-key-v2; is it possible to configure the SDK to set the non-v2 metadata key as well?

ERROR: S3CurlException: Failed writing body (0 != 216) Cause: S3 object 'test_random_key.txt.enc' does not have 'x-amz-meta-x-amz-key' metadata, CurlError 23, multiCurlError 0, CanRetry 0, UserError 1 Detail: ----------------------------------------------- error: S3CurlException: Failed writing body (0 != 216) Cause: S3 object 'test_random_key.txt.enc' does not have 'x-amz-meta-x-amz-key' metadata, CurlError 23, multiCurlError 0, CanRetry 0, UserError 1 code: 8001 context: S3 key being read : s3://edgao-test/test_random_key.txt.enc query: 6154190 location: copy_s3_scanner.cpp:301 process: query0_119_6154190 [pid=678] ----------------------------------------------- [ErrorId: 1-6266d357-62563731450019df4644b352]

If I manually set the x-amz-meta-x-amz-key metadata entry, Redshift then says it was expecting a 16-byte IV (whereas the S3 object has a 12-byte IV). My understanding is that a 16-byte IV may weaken the AES-GCM cipher but this is mitigated by the envelope encryption generating a unique key per file; is there a way to force this behavior for the sake of interop between AWS services?

ERROR: S3CurlException: Failed writing body (0 != 216) Cause: S3 object encryption key iv is of invalid size. Size=12. Expected size 16, CurlError 23, multiCurlError 0, CanRetry 0, UserError 1 Detail: ----------------------------------------------- error: S3CurlException: Failed writing body (0 != 216) Cause: S3 object encryption key iv is of invalid size. Size=12. Expected size 16, CurlError 23, multiCurlError 0, CanRetry 0, UserError 1 code: 8001 context: S3 key being read : s3://edgao-test/test_random_key.txt.enc query: 6154221 location: copy_s3_scanner.cpp:301 process: query0_126_6154221 [pid=966] ----------------------------------------------- [ErrorId: 1-6266d3fb-420b44b61ce699e956e947a5]

For reference, these are the metadata values in question (this is all test data; note that I added the x-amz-meta-x-amz-key entry manually):

Type	Key	Value
System defined	Content-Type	text/plain
User defined	x-amz-meta-x-amz-matdesc	{}
User defined	x-amz-meta-x-amz-tag-len	128
User defined	x-amz-meta-x-amz-key	6Qo4S34/f0xgy4q+OMh+N7jm2EsZIp++dFDUEyvcfaG551kRfMKALQ==
User defined	x-amz-meta-x-amz-unencrypted-content-length	200
User defined	x-amz-meta-x-amz-wrap-alg	AESWrap
User defined	x-amz-meta-x-amz-key-v2	6Qo4S34/f0xgy4q+OMh+N7jm2EsZIp++dFDUEyvcfaG551kRfMKALQ==
User defined	x-amz-meta-x-amz-cek-alg	AES/GCM/NoPadding
User defined	x-amz-meta-x-amz-iv	Oq7a7O0tH+Vxy1oa

I'm pretty sure I've configured the client correctly, as when I UNLOAD ... ENCRYPTED from Redshift, I'm able to decrypt the resulting files using the AWS SDK (these files have a x-amz-meta-x-amz-key metadata entry, and a 16-byte x-amz-meta-x-amz-iv entry). I can attach a code sample / sample files if that's helpful.

[debora-ito]

Do you have references to the Redshift encryption requirements?

is it possible to configure the SDK to set the non-v2 metadata key as well?

I'm not sure, but probably x-amz-meta-x-amz-key was set by the non-V2 AmazonS3EncryptionClient, which is now deprecated.

[edgao]

I haven't been able to find a description of what Redshift requires, other than the docs that just link to this SDK. Everything I've figured out is just via trial and error, unfortunately.

I've ended up just rolling my own implementation using the javax.crypto libraries - not entirely ideal, but it does generate identical output to the UNLOAD command.

[debora-ito]

I'm sorry your experience wasn't great. Have you looked into the Encryption SDK? https://github.com/aws/aws-encryption-sdk-java/

[edgao]

to my understanding, that would have similar issues (it enforces good encryption practice by requiring aws-gcm with a 12-byte IV). But I didn't look too closely, and I also haven't experimented with trying to load an aes-gcm file into redshift