Processing CSV file in Clojure in parallel

Question

I have a large CSV file that contains independent items that take a fair bit of effort to process. I'd like to be able to process each line item in parallel. I found a sample piece of code for processing a CSV file on SO here:

Newbie transforming CSV files in Clojure

The code is:

(use '(clojure.contrib duck-streams str-utils))                 ;;'
(with-out-writer "coords.txt"
  (doseq [line (read-lines "coords.csv")]
    (let [[x y z p] (re-split #"," line)]
      (println (str-join \space [p x y z])))))

This was able to print out data from my CSV file which was great - but it only used one CPU. I've tried various different things, ending up with:

(pmap println (read-lines "foo"))

This works okay in interactive mode but does nothing when running from the command line. From a conversation on IRC, this is because stdout isn't available by default to threads.

Really what I'm looking for is a way to idiomatically apply a function to each line of the CSV file and do so in parallel. I'd also like to print some results to stdout during testing if at all possible.

Any ideas?

`read-line' on a csv file may generate messy data, so one had better use a matured csv-processing lib. — WIZARDELF
– WIZARDELF, Commented Feb 5, 2016 at 20:48

subsub · Accepted Answer · 2011-04-04 07:37:45Z

12

If you want the results in the output be in the same order as in the input, then printing from pmap might not be a good idea. I would recommend creating a (lazy) sequence of the input lines pmap over that and then print the result of pmap. Something like this should work:

(dorun (map println (pmap expensive-computation (read-lines "coords.csv"))))

answered Apr 4, 2011 at 7:37

subsub

1,85710 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mac · Accepted Answer · 2018-04-05 08:50:33Z

7

If you want to do this at speed you might want to look at this article on how Alex Osborne solved the Widefinder 2 challenge posed by Tim Bray. Alex goes into all aspects of parsing, processing and collecting the results (in the Widefinder 2 case the file is a very large Apache log). The actual code used is here.

edited Apr 5, 2018 at 8:50

answered Apr 4, 2011 at 5:12

mac

10.1k4 gold badges38 silver badges51 bronze badges

3 Comments

PeterM Over a year ago

that article is actually really helpful. I will definitely be reading it again in more depth!

Olim Saidov Over a year ago

First link is broken

mac Over a year ago

Thanks, fixed the links.

Joost Diepenmaat · Accepted Answer · 2011-04-04 08:30:40Z

2

I would be extremely surprised if hat code can be sped up by using more cores. I'm 99% certain that the actual speed limit here is the file I/O, which should be a couple orders of magnitude slower than any single core you can throw at the problem.

And that's aside from the overhead you'll introduce when splitting these very minimal tasks over multiple CPUs. pmap isn't exactly free.

If you're sure that disk IO isn't going to be a problem and you've got a lot of CSV parsing to do, simply parsing multiple files in their own threads is going to gain you a lot more for a lot less effort.

answered Apr 4, 2011 at 8:30

Joost Diepenmaat

17.8k3 gold badges48 silver badges55 bronze badges

1 Comment

PeterM Over a year ago

It takes about 5 seconds on average to parse and crunch each CSV line - so it would definitely help to make it parallel. I agree though that in general this sort of work would be I/O not compute bound.

Collectives™ on Stack Overflow

Processing CSV file in Clojure in parallel

3 Answers 3

Comments

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related