2.8.2104 Occasional Redis Hangs Out Of Memory allocating

[BoasE]

We have a single standalone Redis for Windows node running on a server.

Two other server are communicating with this instance.

We use it with SignalR and for caching.
The dump has a size of about 700MB

From time to time we have hangs for 1-3 minutes. After this it recoverrs by it self.

The error seems to only occur when there is some traffic on our page.

In this time we get the exception you can see below

StackExchange.Redis.RedisConnectionException: SocketFailure on EVAL
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at Microsoft.Owin.Cors.CorsMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at

When I search through the Redis log i can ocasssionally find these errors:

[33144] 18 Feb 18:18:44.843 #

=== REDIS BUG REPORT START: Cut & paste starting from here === [33144] 18 Feb 18:18:44.844 # Out Of Memory allocating 308457 bytes. [33144]
18 Feb 18:18:44.844 # --- ABORT [33144] 18 Feb 18:18:44.844 # ---
STACK TRACE
redis-server.exe!LogStackTrace(c:\release\redis\src\win32_interop\win32_stacktrace.cpp:95)(0x00000016,
0x042E0028, 0x00000000, 0x00000001)
redis-server.exe!AbortHandler(c:\release\redis\src\win32_interop\win32_stacktrace.cpp:206)(0x00000001,
0x89EE7767, 0x40150880, 0xBB7A5ED7)
redis-server.exe!raise(f:\dd\vctools\crt\crtw32\misc\winsig.c:587)(0x00000001,
0x00000000, 0x0004B4E9, 0x042E0028)
redis-server.exe!abort(f:\dd\vctools\crt\crtw32\misc\abort.c:82)(0x00000001,
0x4013F888, 0x0004B4E9, 0x00008000)
redis-server.exe!redisOutOfMemoryHandler(c:\release\redis\src\redis.c:3397)(0x0004B4E9,
0x4007DA07, 0x042E0028, 0x4007A27B)
redis-server.exe!zmalloc(c:\release\redis\src\zmalloc.c:147)(0xBDF01150,
0x4007EB2C, 0xBDF01150, 0x446D6B10)
redis-server.exe!sdsnewlen(c:\release\redis\src\sds.c:59)(0xBDF01150,
0xBDF01150, 0x3E74FD95, 0x00000003)
redis-server.exe!_addReplyStringToList(c:\release\redis\src\networking.c:271)(0xBDF01150,
0xBDF01150, 0x042E0028, 0x400E34FE)
redis-server.exe!addReplyBulkCBuffer(c:\release\redis\src\networking.c:517)(0xFFFFFFFF,
0x042E0028, 0x01B77260, 0x01B77260)
redis-server.exe!luaReplyToRedisReply(c:\release\redis\src\scripting.c:792)(0x00000004,
0xBDF01150, 0x00000002, 0x00000002)
redis-server.exe!luaReplyToRedisReply(c:\release\redis\src\scripting.c:839)(0xFFFFFFFF,
0x00A7F690, 0x67897B20, 0xBDF01150)
redis-server.exe!evalGenericCommand(c:\release\redis\src\scripting.c:1048)(0x71E66870,
0x00000000, 0x00000001, 0x000000B2)
redis-server.exe!call(c:\release\redis\src\redis.c:2016)(0x56C60B04,
0x4008B000, 0x00000000, 0x000000B2)
redis-server.exe!processCommand(c:\release\redis\src\redis.c:2235)(0xBDF01150,
0x000000B2, 0x000023B5, 0x00000001)
redis-server.exe!processInputBuffer(c:\release\redis\src\networking.c:1274)(0xBDF01150,
0x00000000, 0x00000000, 0x00000001)
redis-server.exe!readQueryFromClient(c:\release\redis\src\networking.c:1329)(0xFFE51650,
0x00000001, 0x44726F20, 0x0000012C)
redis-server.exe!aeMain(c:\release\redis\src\ae.c:487)(0x56C5C7F8,
0x00000002, 0x00000000, 0x00000002)
redis-server.exe!redis_main(c:\release\redis\src\redis.c:3524)(0x0024BA50,
0x00000002, 0x56C5C7EB, 0x00000002)
redis-server.exe!main(c:\release\redis\src\win32_interop\win32_qfork.cpp:1363)(0x00000016,
0xFFFFFFFF, 0x00000016, 0x0023F3A0)
redis-server.exe!ServiceWorkerThread(c:\release\redis\src\win32_interop\win32_service.cpp:485)(0x4000B3D0,
0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(c:\release\redis\src\win32_interop\win32_service.cpp:485)(0xBB0113B0,
0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(c:\release\redis\src\win32_interop\win32_service.cpp:485)(0x00000000,
0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(c:\release\redis\src\win32_interop\win32_service.cpp:485)(0x00000000,
0x00000000, 0x00000000, 0x00000000) [33144] 18 Feb 18:18:44.857 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

maxheap is set to 3000mb
the server has a total of 64 GB RAM and about 10GB were free

Looking at the error message _ Out Of Memory _ does this mean redis reached the maxheap in this moment?

There is also one more thing , but i'm not sure wether it is really realted to the problem.

Most of the time the problem increases its frequency. Then when I reset the iis of one of the server the problem is gone for hours or days completly. I thought about there may be hanging / stacking signalR queues. But I don't have any further signs that this may be the case.

I Also attached the output of the redis info ( but was about an hour after the error)
redisinfo.txt

Is there any thinh i can log when the server is in this state?
Any hints about that?

PS: I also posted on SO (if someone iscollecting points) http://stackoverflow.com/questions/35489732/redis-occassional-hangs

[enricogior]

Hi @Gentlehag
Redis on Windows 2.8 uses a memory management architecture that is prone to this kind of problems.
The amount of free RAM is influential, the only thing that counts is the fragmentation of the preallocated maxheap (see the wiki for details), one the fragmentation reaches a critical point the memory allocator (dlmalloc) can't do anything about it. That problem has been fixed in 3.0, there isn't anymore a maxheap limitation, the heap is allocated on demand until all system resources are exhausted (if maxmemory is not set). 3.0 also uses a different allocator (jemalloc) that does a better job at preventing the heap fragmentation to grow to fast.
So my suggestion is to consider switching to 3.0.
Thank you.

[BoasE]

We currently have this hang each 2-5 hours. sometime redis even don'T recover it (we have cuirrently more data in ram)

When upgraden to redis 3.0 would the dump survive? and If I would have to downgrand can I use the dump which was modified with 3.0 again ?

And some time ago in an other issue (#379) you told us that you can provide a version with extended logging, could you create it for me so if I have this problem then you could see what it is? Or are there some tradeoffs with this logging?

Our cache is very valuable and I wan't to avoid it if not necessary.

[enricogior]

@Gentlehag

When upgraden to redis 3.0 would the dump survive? and If I would have to downgrand can I use the dump which was modified with 3.0 again ?

Redis on Windows has the same specification of Redis on Linux in regard of the dump file format and compatibility. Please refer to the official Redis documentation: http://redis.io/documentation

And some time ago in an other issue (#379) you told us that you can provide a version with extended logging, could you create it for me so if I have this problem then you could see what it is? Or are there some tradeoffs with this logging?

We provide private builds for issues occurring with the latest release (currently 2.8.2400 and 3.0.501) if there is a bug report specific to one of those releases.
The extra logging trade-offs are: the log will increase in size and may have a slight impact on performance.

[BoasE]

could you provide me this private biuld ? i'Ve the feeling that there is a high chance to get again this bug and in this case it would be very good to not have it twice run in this problem

[enricogior]

@Gentlehag
you want a private build for the issue 379?

[BoasE]

for the most current 3.0 that i should use to update :-) i'd assume its the 3.0.501 isn't it ? I just ask for this as some time ago we had the 3.0 and we had a bug. In these day there wasn't enough information in the log and we would have needed an updated version with more logs. see #379 (comment)

[enricogior]

@Gentlehag
there is a private build with extra logging prepared for a similar issue available here: http://1drv.ms/1oNafsd

I suggest to close this issue and keep track of any problem with the private build in a dedicated issue.
Thank you.

[BoasE]

Thank you very much !