3

I am getting the following error when trying to delete from my cache:

Timeout performing DEL test.com, inst: 0, mgr: ExecuteSelect, err: never, queue: 0, qu: 0, qs: 0, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: ORLWS052, serverEndpoint: Unspecified/pub-redis-16778.us-west-2-1.1.ec2.garantiadata.com:16778, keyHashSlot: 6928, IOCP: (Busy=3,Free=997,Min=4,Max=1000), WORKER: (Busy=4,Free=4091,Min=4,Max=4095), Local-CPU: 100% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

I also get an similar timeout when updating.

I have no idea how to read this, so if anyone knows that would be helpful.

The size of the key I am trying to update or delete is 5515 bytes.

I also ran the slowlog get command and there is none (the length is 0)

I also tried to increase the timeout, but that didn't work either

Improve this question

2 Answers 2

Reset to default
3

IOCP: (Busy=3,Free=997,Min=4,Max=1000), WORKER: (Busy=4,Free=4091,Min=4,Max=4095), Local-CPU: 100%

Please check your ICOP, WORKER busy threads and minimum threads. Also the CPU is 100%. The timeoutHelpLink was updated in the latest release.

https://stackexchange.github.io/StackExchange.Redis/Timeouts

If we look at an example error message from StackExchange.Redis (build 1.0.450 or later), you will see that it now prints ThreadPool statistics (see IOCP and WORKER details below).

System.TimeoutException: Timeout performing GET MyKey, inst: 2, mgr: Inactive, queue: 6, qu: 0, qs: 6, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, IOCP: (Busy=6,Free=994,Min=4,Max=1000), WORKER: (Busy=3,Free=997,Min=4,Max=1000)

In the above example, you can see that for IOCP thread there are 6 busy threads and the system is configured to allow 4 minimum threads. In this case, the client would have likely seen two 500 ms delays because 6 > 4.

Note that StackExchange.Redis can hit timeouts if growth of either IOCP or WORKER threads gets throttled.

Recommendation: Given the above information, it's recommend to set the minimum configuration value for IOCP and WORKER threads to something larger than the default value. We can't give one-size-fits-all guidance on what this value should be because the right value for one application will be too high/low for another application. This setting can also impact the performance of other parts of complicated applications, so you need to fine-tune this setting to your specific needs. A good starting place is 200 or 300, then test and tweak as needed.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Hi, thanks for the input. Where can I change these settings? What part of the code? Also how did you come up with 2 500 ms delays? Is it 500ms for every thread over the min?
@xaisoft Do you check the link?
Yeah I saw it after the fact. Thanks.
1

I faced the same issue and managed to solve it by changing of syncTimeout parameter. By default it's 1000 ms. https://stackexchange.github.io/StackExchange.Redis/Configuration.html

According to the sources it also impacts on AsyncTimeout. https://github.com/StackExchange/StackExchange.Redis/blob/3f7e5466c6bbff96a3ed1130b637a097d21f3fed/src/StackExchange.Redis/ConfigurationOptions.cs#L171

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.