Variables in function arguments do not pass to cluster when parallel computing

Question

I am having difficulties understanding how variables are scoped/passed to the functions when interacting with the parallel package

library(parallel)

test <- function(a = 1){
  no_cores <- detectCores()-1
  clust <- makeCluster(no_cores)
  result <- parSapply(clust, 1:10, function(x){a + x})
  stopCluster(clust)
  return(result)
}

test()
[1]  4  5  6  7  8  9 10 11 12 13

x = 1
test(x)

Error in checkForRemoteErrors(val) : 
3 nodes produced errors; first error: object 'x' not found

test() works but test(x) doesn't. When I modify the function as follows, it works.

test <- function(a = 1){
  no_cores <- detectCores()-1
  clust <- makeCluster(no_cores)
  y = a
  result <- parSapply(clust, 1:10, function(x){y + x})
  stopCluster(clust)
  return(result)
}

x = 1
test(x)

Can someone explain what is going on in memory?

mt1022 · Accepted Answer · 2019-01-02 06:27:32Z

2

This is due to lazy evaluation. The argument a is not evaluated in the function call untill its first use. In first case, the cluster does not known a since it has not been evaluated in the parent environment. You can fix it by forcing the evaluation:

test <- function(a = 1){
    no_cores <- detectCores()-1
    clust <- makeCluster(no_cores)
    force(a)    # <------------------------
    result <- parSapply(clust, 1:10, function(x){a + x})
    stopCluster(clust)
    return(result)
}

x = 1
test(x)
#  [1]  2  3  4  5  6  7  8  9 10 11

answered Jan 2, 2019 at 6:27

mt1022

17.4k6 gold badges53 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

F. Privé · Accepted Answer · 2019-01-02 08:38:38Z

0

I would preferably use foreach() instead of parSapply():

library(doParallel)

test <- function(a = 1) {
  no_cores <- detectCores() - 1
  registerDoParallel(clust <- makeCluster(no_cores))
  on.exit(stopCluster(clust), add = TRUE)
  foreach(x = 1:10, .combine = 'c') %dopar% { a + x }
}

You don't need to force a to be evaluated when using foreach(). Moreover, you can register the parallel backend outside the function if you want.

See a tutorial on using foreach() there (disclaimer: I'm the author of the tuto).

answered Jan 2, 2019 at 8:38

F. Privé

11.8k3 gold badges31 silver badges81 bronze badges

1 Comment

hjw Over a year ago

tyvm i'll definitely look at this package

Collectives™ on Stack Overflow

Variables in function arguments do not pass to cluster when parallel computing

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related