Let's take a look at BETWEEN's documentation.
BETWEEN works with an interval, but if you take a look at this official example:
mysql> SELECT 2 BETWEEN 1 AND 3, 2 BETWEEN 3 and 1;
-> 1, 0
you will see that 2 is between 1 and 3, but 2 is not between 3 and 1. So, if the right side is bigger than the left side, like in the second case where 3 is the left and 1 is the right, then your element will not be between them.
So, 2 is between 1 and 3, but 2 is not between 3 and 1.
Let us take a look at your query:
SELECT * FROM user_actions WHERE timestamp BETWEEN date_sub(now(), interval 5 minute) AND date_sub(now(), interval 10 minute);
The left side contains the moment earlier than now with 5 minutes and the right side contains the moment earlier than now with 10 minutes. So, the right side is actually earlier (smaller) than the left side. We need to see whether we have values between 10 minutes ago and 5 minutes ago, not the other way around :)
So, this should be the query:
SELECT * FROM user_actions WHERE timestamp BETWEEN date_sub(now(), interval 10 minute) AND date_sub(now(), interval 5 minute);