My service is writing to clickhouse. I want to implement a cache which will help me with batching data for CH + it will store the data when CH is unavailable so i dont lose the data. I chose redis sorted set for that purpose, which uses row timestamp as a score and serialized row(including timestamp) as a member. I write to cache first with ZAdd and periodically read first n members from cache with ZRevRange(top n recent rows) to send the data to CH, and, after successful write, i clear those members from cache with ZRem.
The problem i'm facing is that i can't manage TTL properly in that case. TTL is set for the whole set(key), but i need it per-member, so when updating TTL with Expire method i dont overwrite TTL for the old data. I know this method supports additional options, like nx xx gt lt, but i don't see how they can help me with my issue.
I can switch to other data structure, which supports per-key TTL, but then i lose a possibility to easily read top n most recent rows (which is important because in case CH becomes available after a long period i need to write the most recent rows instead of the oldest, in other words i need a LIFO behavior).
Can you help me adjust my current solution, or maybe propose another?
