How to parse rss-feeds / xml in a shell script

Question

I'd like to parse rss feeds and download podcasts on my ReadyNas which is running 24/7 anyway.

So I'm thinking about having a shell script checking periodically the feeds and spawning wget to download the files.

What is the best way to do the parsing?

Thanks!

Perhaps I have to add: I'm on a very slow line, that's why I'm not running my workstation. — Oli
– Oli, Commented Jan 14, 2009 at 17:48

Community · Accepted Answer · 2012-08-24 20:47:06Z

24

Sometimes a simple one liner with shell standard commands can be enough for this:

 wget -q -O- "http://www.rss-specifications.com/rss-podcast.xml" | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c

Sure this does not work in every case, but it's often good enough.

edited Aug 24, 2012 at 20:47

CommunityBot

11 silver badge

answered Jan 15, 2009 at 10:06

leo

3,7477 gold badges36 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

sourcejedi Over a year ago

Brilliant. A cautionary note (wget_1.13.4-3 with polipo_1.0.4.1-1.2): the "-c" option (to continue interrupted downloads) may not work if you also use a proxy server. It seems to keep retrying files which are already complete.

nwgat Over a year ago

this works, just a little thing, is there a way to only download the latest in the rss feed?, so you can run it via cron, i dont want to download 400 episodes of a show :/

nwgat Over a year ago

hmm, that didnt work on this one > wget -q -O- "feeds.twit.tv/sn_video_hd.xml" | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c

nwgat Over a year ago

my question again is howto download only the single latest entry in a rss feed

leo Over a year ago

@wiak It means that it takes the top entry of the feed. Which is usually the newest.

|

cddr · Accepted Answer · 2009-01-14 18:01:24Z

2

Do you have access to awk? Maybe you could use XMLGawk

answered Jan 14, 2009 at 18:01

cddr

3003 silver badges8 bronze badges

Comments

Oli · Accepted Answer · 2009-01-14 17:49:56Z

1

I read about XMLStartlet here and there

But is there a port to ReadyNas NV+ available?

answered Jan 14, 2009 at 17:49

Oli

1,7823 gold badges19 silver badges22 bronze badges

Comments

Community · Accepted Answer · 2017-05-23 12:08:38Z

1

I've wrote the following simple script for downloading XML from Amazon S3, so it would be useful for parsing different kind of XML files:

#!/bin/bash
#
# Download all files from the Amazon feed
#
# Usage:
#  ./dl_amazon_feed_files.sh http://example.s3.amazonaws.com/
# Note: Don't forget about slash at the end
#

wget -qO- "$1" | grep -o '<Key>[^<]*' | grep -o "[^>]*$" | xargs -I% -L1 wget -c "$1%"

This is similar approach to @leo answer.

edited May 23, 2017 at 12:08

CommunityBot

11 silver badge

answered Feb 14, 2013 at 13:38

kenorb

169k95 gold badges712 silver badges796 bronze badges

1 Comment

kenorb Over a year ago

I don't remember if any of part was used, as this is very basic syntax which I'm using very often out of hand. However, I've linked the post just in case.

Giacomo · Accepted Answer · 2009-01-14 18:15:29Z

0

You can use xsltproc from libxml2 and write a simple xsl stylesheet that parses the rss and outputs a list of links.

answered Jan 14, 2009 at 18:15

Giacomo

11.3k5 gold badges27 silver badges25 bronze badges

Collectives™ on Stack Overflow

How to parse rss-feeds / xml in a shell script

5 Answers 5

7 Comments

Comments

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

7 Comments

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related