I'm trying to RELIABLY implement that pattern. For practical purposes, assume we have something similar to a twitter clone (in cassandra and nodejs).
So, user A has 500k followers. When user A posts a tweet, we need to write to 500k feeds/timelines.
Conceptually this is easy, fetch followers for user A, for each one: write tweet to his/her timeline. But this is not "atomic" (by atomic I mean that, at some point, all of the writes will succeed or none will).
async function updateFeeds(userId, tweet) {
let followers = await fetchFollowersFor(userId)
for(let f of followers) {
await insertIntoFeed(f, tweet)
}
}
This seems like a DoS attack:
async function updateFeeds(userId, tweet) {
let followers = await fetchFollowersFor(userId)
await Promise.all(followers.map(f => insertIntoFeed(f, tweet)))
}
How do I keep track of the process? How do I resume in case of failure? I'm not asking for a tutorial or anything like that, just point me in the right direction (keywords to search for) if you can please.
Promise.all()will never be what you want if you want to keep track of which feeds succeeded and which failed since it short circuits on first failure and doesn't report anything about all the other requests. And, yes it will look like at least a rate limiting violation, if not a DOS attack.