5

Following some other SO questions, I'm developing a website monitoring application as a pet project with the goal of learning more about Node.js + Redis.

What I have planned is to let users add urls and add them to a Redis SET. Every minute, I get the SET results, do a HTTP Get request and print the response.

This seem to work fine, however, I have a couple of questions:

  1. Given that Redis SET does not allow repeated keys (which will save me from doing a request to the same URL), how do I control when a user removes the URL from his account but another user has the same URL? Can I have an INCR value in the URL key so I know how many users have the URL in their account?

  2. Given that I do an HTTP request every minute and I want to use Redis to save the results (response time, up/down, etc), what's the best way to save all that data in Redis (results from the requests to each url every minute)? Shall I save each response in a unique Redis key?

  3. In order to show results to the user in real-time, what's the best way to query the results and parse it in real-time?

Thanks for the help.

Improve this question

1 Answer 1

Reset to default
6

I think you should start writing prototype in redis-cli. Also I would like to point out this very good article from Simon Willison explaining redis.

Given that Redis SET does not allow repeated keys (which will save me from doing a request to the same URL), how do I control when a user removes the URL from his account but another user has the same URL? Can I have an INCR value in the URL key so I know how many users have the URL in their account?

I would use SADD + INCR for that. 

SADD urls http://www.google.com
INCR http://www.google.com

To remove http://www.google.com I would simply do:

DECR http://www.google.com
#Only if DECR http://www.google.com => 0, then you should remove from SET
SREM urls http://www.google.com

Given that I do an HTTP request every minute and I want to use Redis to save the results (response time, up/down, etc), what's the best way to save all that data in Redis (results from the requests to each url every minute)?

I would use a unique key for every URL and write the data back to redis as json(JSON.stringify(obj)) using MSET.

MSET data:http://www.google.com "{json for google}" data:http://www.yahoo.com "{json for yahoo}"

In order to show results to the user in real-time, what's the best way to query the results and parse it in real-time?

I would get the results via MGET and parse the json(obj = JSON.parse(json-string)).

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Ok! It makes a lot of sense! Thanks! Just one question tough, you'll be needing to append the new results to JSON. Is this the best solution for it? Is there no scaling problems once the JSON file becomes huge? Can you detail that a little better? I have very few experience using JS/JSON. Thanks Alfred! ;)
If you need to keep historic data for each url rather than just the current data, use a list rather than a simple key. You lose MSET, but you don't need to worry about the previous data when writing and can easily trim the list. I'd also look at keeping a list of urls as well as or instead of a set - with a list set up as a queue you can easily spread out the requests rather than having to run them all at once.
@Donald I was on a holiday the last couple of days, but I am not appending anything. You should have another look at the documentation of MSET. But also you are a little bit right I guess it could get slow if the MSET string is too long. Maybe I would just use a lot of SETS instead that is also very fast( about 110000 SETs per second).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.