Connection reset by peer

[SapiensAnatis]

Hi, I'm running Testcontainers in an unusual environment, which is inside an OpenSUSE distrobox and using Podman. I appreciate this is not exactly supported, but it was working beautifully until it abruptly stopped today without any configuration changes on my part.

I use this ~/.testcontainers/properties:

ryuk.container.privileged=true
docker.host=unix:///run/user/1000/podman/podman.sock 

and now whenever I run a test, it fails at container.StartAsync() with

System.IO.IOException
Unable to read data from the transport connection: Connection reset by peer.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
   at Docker.DotNet.MultiplexedStream.ReadOutputAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /_/src/Docker.DotNet/MultiplexedStream.cs:line 81
   at Docker.DotNet.MultiplexedStream.CopyOutputToAsync(Stream stdin, Stream stdout, Stream stderr, CancellationToken cancellationToken) in /_/src/Docker.DotNet/MultiplexedStream.cs:line 188
   at Docker.DotNet.MultiplexedStream.ReadOutputToEndAsync(CancellationToken cancellationToken) in /_/src/Docker.DotNet/MultiplexedStream.cs:line 133
   at DotNet.Testcontainers.Clients.DockerContainerOperations.ExecAsync(String id, IList`1 command, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerContainerOperations.cs:line 156
   at Testcontainers.PostgreSql.PostgreSqlBuilder.WaitUntil.UntilAsync(IContainer container) in /_/src/Testcontainers.PostgreSql/PostgreSqlBuilder.cs:line 152
   at DotNet.Testcontainers.Containers.DockerContainer.CheckReadinessAsync(WaitStrategy waitStrategy, CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 542
   at DotNet.Testcontainers.Configurations.WaitStrategy.<>c__DisplayClass24_0.<<WaitUntilAsync>g__UntilAsync|0>d.MoveNext() in /_/src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 184
--- End of stack trace from previous location ---
   at DotNet.Testcontainers.Configurations.WaitStrategy.WaitUntilAsync(Func`1 wait, TimeSpan interval, TimeSpan timeout, Int32 retries, CancellationToken ct) in /_/src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 213
   at DotNet.Testcontainers.Containers.DockerContainer.CheckReadinessAsync(IEnumerable`1 waitStrategies, CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 560
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 480
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 283
   at DragaliaAPI.Integration.Test.TestContainersHelper.StartAsync() in /home/jay/Projects/Dawnshard/DragaliaAPI/DragaliaAPI.Integration.Test/TestContainersHelper.cs:line 106
   at DragaliaAPI.Integration.Test.CustomWebApplicationFactory.InitializeAsync() in /home/jay/Projects/Dawnshard/DragaliaAPI/DragaliaAPI.Integration.Test/CustomWebApplicationFactory.cs:line 38
   at Xunit.v3.FixtureMappingManager.GetFixture(Type fixtureType) in D:\a\xunit\xunit\src\xunit.v3.core\Utility\FixtureMappingManager.cs:line 189

What's weird is, if I use a debugger to break on the exception, the containers start correctly:

$ podman ps
CONTAINER ID  IMAGE                                            COMMAND               CREATED         STATUS         PORTS                              NAMES
51e225fd3f45  registry.opensuse.org/opensuse/distrobox:latest  --verbose --name ...  2 months ago    Up 16 minutes                                     tumbleweed
33a944c2bd70  registry.opensuse.org/opensuse/distrobox:latest  --verbose --name ...  2 months ago    Up 13 minutes                                     dotnet
ec2594b662e7  docker.io/testcontainers/ryuk:0.9.0              /bin/ryuk             28 seconds ago  Up 28 seconds  0.0.0.0:36333->8080/tcp            testcontainers-ryuk-6cb2078c-6760-4f10-a6b4-7f662717ac50
7bfb327b725b  docker.io/library/postgres:16                    -c fsync=off -c f...  28 seconds ago  Up 28 seconds  0.0.0.0:33329->5432/tcp            gallant_curran
756a0451f9e0  docker.io/redis/redis-stack:7.2.0-v6             /entrypoint.sh        3 seconds ago   Up 3 seconds   0.0.0.0:34625->6379/tcp, 8001/tcp  recursing_shtern

The last log line from Testcontainers before the exception looks something like

[testcontainers.org 00:00:08.08] Execute "pg_isready --host localhost --dbname testing --username testing" at Docker container 7bfb327b725b

except, sometimes this appears multiple times, indicating the command was issued, and sometimes the test even runs to completion, maybe about 10% of the time it passes.

I get this is probably too low-level to have anything to do with Testcontainers, but I was wondering if anyone had seen anything similar?

[HofmeisterAn]

It looks like there is an issue reading from the stream. I remember we had a similar exception in Docker.DotNet before: dotnet/Docker.DotNet#629. Maybe you can try to recreate the issue using Docker.DotNet. If it is related to reading from the stream, this should not be too difficult. Perhaps this test might help. Be aware that this test writes to the stream, but reading from it works in the same way. Also, note that there is a fork of Docker.DotNet that contains recent fixes and updates (you should use the forked version). Hope that helps.

[SapiensAnatis]

Thanks for the pointers. As I mentioned, I doubt it's an issue with Testcontainers itself, but it's great to get some input from someone more familiar with Docker.DotNet.

I've had a go at replicating what I think Testcontainers is doing using Docker.DotNet, but was unfortunately unable to reproduce the issue. Here's the code I was using:

https://gist.github.com/SapiensAnatis/1562fb87dfeef4bac0de3d9b3e7a71cc

I put together a console app which uses Testcontainers and that Docker.DotNet code:

// Runs to completion and performs the exec 10 times
await DockerDotNet.RunExecManually();

// Crashes after a non-deterministic number of ready checks
var postgresContainer = new PostgreSqlBuilder()
    .WithImage("postgres:16")
    .WithUsername("testing")
    .WithPassword("verystrongpassword")
    .WithDatabase("testing")
    .WithPortBinding(5432, true)
    .Build();

var redisContainer = new ContainerBuilder()
    .WithImage("redis/redis-stack:7.2.0-v6")
    .WithWaitStrategy(Wait.ForUnixContainer().UntilCommandIsCompleted("redis-cli PING"))
    .WithPortBinding(6379, true)
    .Build();

await Task.WhenAll(postgresContainer.StartAsync(), redisContainer.StartAsync());

Maybe I misunderstood something from the samples about how to use Docker.DotNet, but I am getting the output I would expect from my exec code, so it seems to do what I imagine the PostgreSqlContainer is doing. Maybe I'll have a closer look at how that's implemented in the Testcontainers side and see if there are any major differences.

[HofmeisterAn]

Right now I’m on mobile, can’t help much. The response returns a multiplex stream. Maybe we get or read the wrong target. It probably make sense to debug into this line. This is the spec of the steam.

[SapiensAnatis]

Thanks for all your help so far. I'll take a look at debugging that when I have time.

I've also noticed I can't reproduce the issue on Testcontainers 4.0.0 / Docker.DotNet (non-fork) 3.125.15. So it could be a regression in 4.1.0 (which it seems introduced the fork of Docker.DotNet?)

The fork doesn't look to have changed the file in question aside from updating some usings, however.

[HofmeisterAn]

I've also noticed I can't reproduce the issue on Testcontainers 4.0.0 / Docker.DotNet (non-fork) 3.125.15. So it could be a regression in 4.1.0 (which it seems introduced the fork of Docker.DotNet?)

I see, these are the changes. Not many, I guess the issue is related to the migration from JSON.NET to System.Text.Json then. I do not have an environment running Podman, sorry. Cannot test it.

[SapiensAnatis]

Crash seems to be here, reading the header.

If it's similar to the issue you posted earlier, wouldn't the if (n == 0) block cover it? I am not a socket expert so I don't know exactly what 'connection reset' means.

The code looks to line up with the spec, with some additional loops that I assume are handling the cases where an incomplete read of the header occurs. But we are coming into the exception from a direct call; i.e. it doesn't look like this is happening after an iteration where _remaining was 0

[HofmeisterAn]

Sorry, right now I have no idea what's going wrong. It's hard to tell without a reproducer (not running Podman or a similar environment as yours). I think we should first check if the issue relates to a recent change and see if we can identify a commit or if it already occurs in previous versions, and we were just lucky not to run into it before. Then, I think it makes sense to check why the behavior differs when toggling AttachStdin and what changes in the headers and responses.