2

I'm trying to parse some RSS feeds using Golang. A package I found was https://github.com/jteeuwen/go-pkg-rss

My process is

  1. Get feeds from my Postgres DB
  2. For each feed, fetch the articles
  3. Store each article into the Postgres DB

My code is as below

func main() {
  db := dbConnect() // returns the DB connection variable

  feeds := getRssFeeds(db) // returns feeds from my DB
  for i := 0; i < len(feeds); i++ {
    getFeedArticles(feeds[i].url, 5, db)
  }
}

func getFeedArticles(uri string, timeout int, db *sql.DB) {
  // using the package to get RSS feed contents
  feed := rss.New(timeout, true, chanHandler, itemHandler)
  if err := feed.Fetch(uri, nil); err != nil {
    fmt.Fprintf(os.Stderr, "[e] %s: %s", uri, err)
    return
  }
}

func chanHandler(feed *rss.Feed, newchannels []*rss.Channel) {
  // no need to do anything...
}

func itemHandler(feed *rss.Feed, ch *rss.Channel, newitems []*rss.Item) {
  for i := 0; i < len(newitems); i++ {
    fmt.Println(newitems[i].Title)

    // insert the article into DB here
  }
}

How do I pass the db variable to itemHandler so I can insert the article into my DB? I suppose I could call dbConnect() to create a new db variable to insert, but that would seem wasteful.

Is there a better way to do this process?

1

1 Answer 1

3

You can use closures:

func makeHandler(db Database) rss.ItemHandler {
    return func(feed *rss.Feed, ch *rss.Channel, newitems []*rss.Item) {
         for i := 0; i < len(newitems); i++ {
             fmt.Println(newitems[i].Title)
             // Accessed via closure:
             db.InsertArticle(newitems[i])
         }
    }
}

Or some variation thereof. You can then call rss.New with makeHandler(db), or define itemHandler in getFeedArticle... In any case, itemHandler will have db being defined and accessible.

This tutorial and this part of the specs explain it in more details

Sign up to request clarification or add additional context in comments.

5 Comments

thanks for the response. I tried your suggestion of closure and called "makehandler(db)" in "rss.New" but I get "cannnot use makeHandler(db) (type handler) as type feeder.ItemHandler in function argument". Any ideas?
ok, instead of calling "makeHandler(db)" in rss.New, I did rss.New(timeout, true, chanHandler, func(feed *rss.Feed, ch *rss.Channel, newitems[]*rss.Item) { ... }) and this worked. Not sure why the first approach isn't working?
The first did not work because you provided a handler where a feeder.ItemHandler was expected, although they both refer to the same function signature. You can correct it by having makeHandler return a feeder.ItemHandler instead (it has the same signature). I only defined handler for clarity, but the correct way is to return a ItemHandler directly.
And indeed: github.com/jteeuwen/go-pkg-rss/blob/master/feed.go defines type ItemHandler func(f *Feed, ch *Channel, newitems []*Item) (i.e. same as handler)
I have edited the response so that it now returns the correct func type. Thank you for your feedback !

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.