parallel R execution problem in R

Question

I am using doSMP as a parallel backend in Windows 7, with R 2.12.2. I incur in an error, and would like to understand the likely cause. Here is some sample code to reproduce the error.

require(foreach)
require(doSMP)
require(data.table)
wrk <- startWorkers(workerCount = 2)
registerDoSMP(wrk)
DF = data.table(x=c("b","b","b","a","a"),v=rnorm(5))
setkey(DF,x)
foreach( i=1:2)  %dopar% {
    DF[J("a"),]
}

The error message is

Error in { : task 1 failed - "could not find function "J""

I'll ask the obvious Q - what is J() I have run your code and get the same error on a Linux box with R 2.13-0-alpha and I can't find J() anywhere on that system. — Gavin Simpson
– Gavin Simpson, Commented Apr 1, 2011 at 13:03
this is the same question as Gavin, but when you call DF[J("a"),] what is J? — JD Long
– JD Long, Commented Apr 1, 2011 at 13:11
Ah, ignore that, I see that DF[J("a"),] works when not in the foreach() wrapper so it must be something particular to data.table. Will investigate more. — Gavin Simpson
– Gavin Simpson, Commented Apr 1, 2011 at 13:16
ok, J is a data.table function. I see ;) It looks like the spawned worker R instances need to have the data.table package loaded. — JD Long
– JD Long, Commented Apr 1, 2011 at 13:30

Gavin Simpson · Accepted Answer · 2011-04-01 13:42:04Z

8

I've not used doSMP, but I did some digging around and it looks like this post gets at a similar issue.

so it looks like you should be able to do:

foreach( i=1:2, .packages="data.table")  %dopar% {
    DF[J("a"),]
}

I can't test as I don't have a Windows machine handy.

edited Apr 1, 2011 at 13:42

Gavin Simpson

176k28 gold badges405 silver badges461 bronze badges

answered Apr 1, 2011 at 13:31

JD Long

61k58 gold badges209 silver badges300 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

gappy · Accepted Answer · 2011-04-01 14:39:03Z

6

OK, I asked Revolution computing, and Steve Weller (of RC) replied:

The problem is a R scoping issue. By default, foreach() will look for variables defined in it's own 'environment'. Any objects defined outside of it's scope need to be explicitly passed to it via the '.export' argument.

In your case, you will need to modify your 'foreach()' call to pass in the objects 'DF' and 'J':

...

foreach(i=1:2, .export=c("DF","J")) %dopar% {
...

I haven't tried either solution yet, but I trust both JD and RC...

answered Apr 1, 2011 at 14:39

gappy

10.3k14 gold badges56 silver badges73 bronze badges

2 Comments

Gavin Simpson Over a year ago

I don't think you need "DF" in there. I get a warning if I use your line: already exporting variable(s): DF If you leave out "DF" it works without warning.

Steve Weston Over a year ago

Since your foreach loop needs to use the data.table package, you should use .packages, as in @JD Long's answer, and then exporting isn't necessary. Exporting may also hurt your performance since I believe it will serialize the whole package along with the function. Loading a package is the best way to "export" functions to the workers.

Collectives™ on Stack Overflow

parallel R execution problem in R

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related