Very slow API response times for S3 when using dual stack on IPv6 only EC2 instance

[anca-jalbum]

When making calls to S3 using the Java SDK v2 on an EC2 instance that doesn't have a public IPv4 IP, but have IPv6 working, using dual-stack mode, the first API call takes 16 seconds. If the client is reused within 5 seconds those calls are quick (~20ms). But if that client idles for more than 5 seconds, the next API call takes 16 seconds again.

It looks like the SDK first tries using IPv4 3 times with a 5 second timeout, and then switches to IPv6.

If I use curl to access the dual stack S3 endpoint I get an immediate response. Same of course if I force curl to use IPv6 with curl -6. But it will of course fail using curl -4.

What might be wrong here?
Why isn't IPv6 used first when enabling dual-stack?
I don't really want to use "dual-stack". I want to use IPv6 only. Isn't that possible?

I'm basically using the AWS SDK for Java dual-stack endpoint example at https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html but rewritten for SDK v2, with v2.24.2 of the SDK and Java 21. I've also tried Java 8 and Java 17 with the same results. It's an Arm based Ubuntu machine in us-east-1.

[debora-ito]

It looks like the SDK first tries using IPv4 3 times with a 5 second timeout, and then switches to IPv6.

That is weird, indeed. Can you share any logs of this?
Are you using a proxy?
Can you share the exact same code you are running so I can reproduce on my end?

[anca-jalbum]

No proxy.

IPv4 is enabled for the private net at AWS, so the EC2 instance has a private IPv4 IP, and there are IPv4 routes that other servers in the same VPC us that has public IPv4 IPs. Don't think I can disable those for just this instance. But shouldn't matter if the SDK would try IPv6 first.

After enabling debug logging I see that it actually tries 8 different IPv4 IPs with a 2 second timeout. 9th attempt is using IPv6 which succeeds. (I've replaced my bucket with BUCKET.):

2024-02-15 15:36:12 DEBUG ExecutionInterceptorChain:85 - Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@2c532cd8' modified the message with its modifyHttpRequest method.
2024-02-15 15:36:12 DEBUG request:85 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=BUCKET.s3.dualstack.us-east-1.amazonaws.com, encodedPath=, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[list-type, max-keys])
2024-02-15 15:36:12 DEBUG SigningStage:85 - Using SelectedAuthScheme: aws.auth#sigv4
2024-02-15 15:36:12 DEBUG DefaultV4RequestSigner:85 - AWS4 Canonical Request Hash: REPLACED
2024-02-15 15:36:12 DEBUG DefaultV4RequestSigner:85 - AWS4 String to sign: AWS4-HMAC-SHA256
20240215T153612Z
20240215/us-east-1/s3/aws4_request
2d0bd45ac17d8be6b24a16805e924b9d35f7eaa6420384a111f846cb9083c427
2024-02-15 15:36:12 DEBUG RequestAddCookies:123 - CookieSpec selected: default
2024-02-15 15:36:12 DEBUG RequestAuthCache:77 - Auth cache not set in the context
2024-02-15 15:36:12 DEBUG PoolingHttpClientConnectionManager:267 - Connection request: [route: {s}->https://BUCKET.s3.dualstack.us-east-1.amazonaws.com:443][total available: 0; route allocated: 0 of 50; total allocated: 0 of 50]
2024-02-15 15:36:12 DEBUG PoolingHttpClientConnectionManager:312 - Connection leased: [id: 0][route: {s}->https://BUCKET.s3.dualstack.us-east-1.amazonaws.com:443][total available: 0; route allocated: 1 of 50; total allocated: 1 of 50]
2024-02-15 15:36:12 DEBUG MainClientExec:234 - Opening connection {s}->https://BUCKET.s3.dualstack.us-east-1.amazonaws.com:443
2024-02-15 15:36:12 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/54.231.226.34:443
2024-02-15 15:36:12 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/54.231.226.34:443 with timeout 2000
2024-02-15 15:36:14 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/54.231.226.34:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:14 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.147.86:443
2024-02-15 15:36:14 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.147.86:443 with timeout 2000
2024-02-15 15:36:16 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.147.86:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:16 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/16.182.107.210:443
2024-02-15 15:36:16 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/16.182.107.210:443 with timeout 2000
2024-02-15 15:36:18 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/16.182.107.210:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:18 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/16.182.32.146:443
2024-02-15 15:36:18 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/16.182.32.146:443 with timeout 2000
2024-02-15 15:36:20 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/16.182.32.146:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:20 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.32.82:443
2024-02-15 15:36:20 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.32.82:443 with timeout 2000
2024-02-15 15:36:22 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.32.82:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:22 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.168.126:443
2024-02-15 15:36:22 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.168.126:443 with timeout 2000
2024-02-15 15:36:24 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.216.168.126:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:24 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.217.204.226:443
2024-02-15 15:36:24 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.217.204.226:443 with timeout 2000
2024-02-15 15:36:26 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/52.217.204.226:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:26 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/54.231.203.90:443
2024-02-15 15:36:26 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/54.231.203.90:443 with timeout 2000
2024-02-15 15:36:28 DEBUG DefaultHttpClientConnectionOperator:166 - Connect to BUCKET.s3.dualstack.us-east-1.amazonaws.com/54.231.203.90:443 timed out. Connection will be retried using another IP address
2024-02-15 15:36:28 DEBUG DefaultHttpClientConnectionOperator:139 - Connecting to BUCKET.s3.dualstack.us-east-1.amazonaws.com/[2600:1fa0:812f:8da1:36e7:a03a:0:0]:443
2024-02-15 15:36:28 DEBUG SdkTlsSocketFactory:366 - Connecting socket to BUCKET.s3.dualstack.us-east-1.amazonaws.com/[2600:1fa0:812f:8da1:36e7:a03a:0:0]:443 with timeout 2000
2024-02-15 15:36:28 DEBUG SdkTlsSocketFactory:430 - Enabled protocols: [TLSv1.3, TLSv1.2]
2024-02-15 15:36:28 DEBUG SdkTlsSocketFactory:431 - Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2024-02-15 15:36:28 DEBUG SdkTlsSocketFactory:85 - socket.getSupportedProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello], socket.getEnabledProtocols(): [TLSv1.3, TLSv1.2]
2024-02-15 15:36:28 DEBUG SdkTlsSocketFactory:435 - Starting handshake
2024-02-15 15:36:28 DEBUG SdkTlsSocketFactory:465 - Secure session established

Test class is:

import software.amazon.awssdk.awscore.exception.AwsServiceException;
import software.amazon.awssdk.core.exception.SdkClientException;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.S3Object;

public class DualStackEndpoints {

public static void main(String[] args) throws InterruptedException {
	if (args.length < 2) {
		System.out.println("Usage: DualStackEndpoints <region> <bucket>");
		return;
	}
	Region region = Region.of(args[0]);
	String bucket = args[1];

	try {
		// Create an Amazon S3 client with dual-stack endpoints enabled.
		S3Client client = S3Client.builder()
				.region(region)
				.dualstackEnabled(true)
				.build();

		for (int i = 0; i < 20; ++i) {
			long start = System.currentTimeMillis();
			listObjects(client, bucket);
			System.out.println("...in " + (System.currentTimeMillis() - start) + "ms");

			System.out.println("Waiting " + i + "s...");
			Thread.sleep((long)i * 1000);
		}
	} catch (AwsServiceException e) {
		// The call was transmitted successfully, but Amazon S3 couldn't process
		// it, so it returned an error response.
		e.printStackTrace();
	} catch (SdkClientException e) {
		// Amazon S3 couldn't be contacted for a response, or the client
		// couldn't parse the response from Amazon S3.
		e.printStackTrace();
	}
}

private static void listObjects(S3Client client, String bucket) {
	ListObjectsV2Request request = ListObjectsV2Request
			.builder()
			.bucket(bucket)
			.maxKeys(1)
			.build();

	System.out.println("Listing 1st file in bucket " + bucket + ":");

	ListObjectsV2Response response = client.listObjectsV2(request);

	for (S3Object o : response.contents()) {
		System.out.println(o.key());
	}

}

}

Args are region (using us-east-1) and bucket. It waits longer and longer between calls.

Maven dependencies are:

	<dependencies>
		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>bom</artifactId>
			<version>2.24.2</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	
	</dependencies>

	<dependency>
		<groupId>software.amazon.awssdk</groupId>
		<artifactId>s3</artifactId>
	</dependency>

[debora-ito]

Did some research, and you need to tell your JVM to prefer IPv6:

By default, IPv4 addresses are preferred over IPv6 addresses, for example, when querying the name service 
(for instance, DNS service), IPv4 addresses would be returned ahead of IPv6 addresses. 

(...) 

This setting can be changed through the system property java.net.preferIPv6Addresses=<true|false>

Can you try using the system property -Djava.net.preferIPv6Addresses=true and see if it works?

[anca-jalbum]

Yes, setting java.net.preferIPv6Addresses=true will make my test use IPv6 first, and the first request is quick too. Good! 👍

The guide at https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html says that's a property that might have to be set "If you are using the AWS SDK for Java on Windows". But I'm using Java on a Linux machine, so I ignored that statement.

It also says "When you make a request to a dual-stack endpoint, the bucket URL resolves to an IPv6 or an IPv4 address.", implying that IPv6 is tried first. That would be much better, since that is the whole reason dual stack is enabled for the client...

I don't really want to set this for the whole web server. Everything else should use IPv4, since that is communication using private IPs, where IPv4 has to be used (or at least can't be disabled)... Can't the use of IPv6 be controlled per API or for each client? I simply want to disable IPv4 only for these S3 calls...

[debora-ito]

The guide at https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html says that's a property that might have to be set "If you are using the AWS SDK for Java on Windows". But I'm using Java on a Linux machine, so I ignored that statement.

Good call-out, I'll ask the documentation team to reword.

Can't the use of IPv6 be controlled per API or for each client? I simply want to disable IPv4 only for these S3 calls...

Unfortunately no, because the IPv6 preference is controlled at the underlying JDK level.