parallel::clusterExport how to pass nested functions from global environment?

Question

I'm making a function (myFUN) that calls parallel::parApply at one point, with a function yourFUN that is supplied as an argument.

In many situations, yourFUN will contain custom functions from the global environment.

So, while I can pass "yourFUN" to parallel::clusterExport, I cannot know the names of functions inside it beforehand, and clusterExport returns me an error because it cannot find them.

I don't want to export the whole enclosing environment of yourFUN, since it might be very big.

Is there a way for me to export only the variables necessary for running yourFUN?

The actual function is very long, here is a minimized example of the error:

mydata <- matrix(data = 1:9, 3, 3)

perfFUN <- function(x) 2*x

opt_perfFUN <- function(y) max(perfFUN(y))

avg_perfFUN <- function(w) perfFUN(mean(w))

myFUN <- function(data, yourFUN, n_cores = 1){

  cl <- parallel::makeCluster(n_cores)
  parallel::clusterExport(cl, varlist = c("yourFUN"), envir = environment())

  parallel::parApply(cl, data, 1, yourFUN)

}

myFUN(data = mydata, yourFUN = opt_perfFUN)
myFUN(data = mydata, yourFUN = avg_perfFUN)

 Error in checkForRemoteErrors(val) : one node produced an error: could not find function "perfFUN"

Thank you very much!

F. Privé · Accepted Answer · 2018-01-17 06:58:49Z

4

A possible solution, use:

myFUN <- function(data, yourFUN, n_cores = 1) {

  cl <- parallel::makeCluster(n_cores)
  on.exit(parallel::stopCluster(cl), add = TRUE)

  envir <- environment(yourFUN)
  parallel::clusterExport(cl, varlist = ls(envir), envir = envir)

  parallel::parApply(cl, data, 1, yourFUN)  
}

answered Jan 17, 2018 at 6:58

F. Privé

11.8k3 gold badges31 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

parallel::clusterExport how to pass nested functions from global environment?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related