redis throws timeout errors

Question

My server started throwing a couple of hours ago the following exception for an unknown reason. It's a web application. Can anyone help please? What can I do?

, Timeout performing GET SomeKey, inst: 1, mgr: ProcessReadQueue, err: never, queue: 12, qu: 0, qs: 12, qc: 0, wr: 0, wq: 0, in: 1702, ar: 1, clientName: SSD41ACCU10147, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=11,Free=8180,Min=4,Max=8191), Local-CPU: unavailable (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

, Timeout performing GET SomeKey2, inst: 2, mgr: ProcessReadQueue, err: never, queue: 33, qu: 0, qs: 33, qc: 0, wr: 0, wq: 0, in: 1141, ar: 1, clientName: SSD41ACCU10147, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=18,Free=8173,Min=4,Max=8191), Local-CPU: unavailable (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

Stacktrace

at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 2044 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 81 at StackExchange.Redis.RedisDatabase.HashGetAll(RedisKey key, CommandFlags flags) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 130 at ...

Edit: Is it possible that I have to much connections that are not closing?

I have a static Property:

public static ConnectionMultiplexer Redis = ConnectionMultiplexer.Connect("localhost:1234,abortConnect=false");

And then from each place I need to query the redis I do:

IDatabase db = GlobalStaticClass.redis.GetDatabase();

Is there a problem with this approach?

Edit 2 [Answers to marc]:

I have Some keys that are very large (1MB) and some that are merely numbers or small string. But this is not new.

It's been like that for couple of months. Suddenly today, 2 hours ago, ALL redis requests started throwing exceptions. (The attached exception is just an example)

Not to much. The redis was never close to be the bottle neck.
Slowlog shows nothing intersting.
How can I check that?
Trying that right now.

Edit 3: Reset IIS solves the problem for 15-30 minutes and then it's starts again and never stops.

Edit 4: It was the hosting company fault. Not really related to redis.

It caused all of our servers to be very slow and redis was the first one to return timeout errors.

Your answers did help me to find who is responsible.

Thanks all.

Marc Gravell · Accepted Answer · 2017-04-04 12:40:46Z

2

The overall approach should be fine; the outbound is clear, we have a small amount of data on the inbound, and we have an active reader - which seems healthy. So we get into context:

what is the size of the data in SomeKey?
how busy is the server?
is slowlog indicating any particularly long-running operations that would explain blocking at the server?
is the server log showing long database persistence operations or slaving operations that might explain things?
as a last resort: have you tried increasing the sync timeout? (see search for syncTimeout here)

answered Apr 4, 2017 at 12:40

Marc Gravell

1.1m273 gold badges2.6k silver badges3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Carl Dacosta · Accepted Answer · 2017-04-05 05:33:00Z

0

From your Timeout error message (note that the value for "Busy" is greater than the value of "Min"), you are experiencing Threadpool growth throttling.

Timeout performing GET SomeKey, inst: 1, mgr: ProcessReadQueue, err: never, queue: 12, qu: 0, qs: 12, qc: 0, wr: 0, wq: 0, in: 1702, ar: 1, clientName: SSD41ACCU10147, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=11,Free=8180,Min=4,Max=8191),

Here is an explanation and recommendations on a few things you can try out. https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md

answered Apr 5, 2017 at 5:33

Carl Dacosta

9114 silver badges13 bronze badges

Collectives™ on Stack Overflow

redis throws timeout errors

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related