R: How to source variable files on each parallel cluster node?

Question

I want to run the following code:

library(parallel)

cl <- makeCluster(detectCores())
requires <- c("fUnitRoots","fGarch")

for(req in requires) {
  clusterEvalQ(cl,require(req))
}

list1 <- clusterApply(cl,1:10,function(i) {
  x <- rnorm(100)
  y <- rnorm(100)
  m <- lm(y~x)
  res <- resid(m)
  t <- adfTest(res) ## this function is in {fUnitRoots}
  return(t@test$statistic)
})

stopCluster(cl)

However, fUnitRoots package is not loaded in any node. It is probably because clusterEvalQ(cl,expr) where expr is an expression. require(req) is treated as an expression where req is not regarded as the iterator variable as a character.

How should I refine the code to make it work?

Steve Weston · Accepted Answer · 2014-02-25 15:34:22Z

4

The "character.only" option is useful when calling "require" in this situation. Also, I would use "clusterCall" instead of "clusterEvalQ" to allow the package names to be passed as an argument to a worker function:

clusterCall(cl, function(pkgs) {
  for (req in pkgs) {
    require(req, character.only=TRUE)
  }
}, c("fUnitRoots","fGarch"))

This is also a bit more efficient since it loads all of the packages in a single cluster operation.

You can verify that the packages were correctly loaded using:

clusterEvalQ(cl, search())

answered Feb 25, 2014 at 15:34

Steve Weston

19.7k4 gold badges62 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user177196 Over a year ago

I tried your clusterCall code but it all returns FALSE for me. May you please offer some help as I need to load a large amount of library into each cluster node? Also, how do we remove some packages from every cluster nodes? I also wonder if clusterEvalQ(cl, library(dplyr)) helps put the library(dplyr) into each cluster node, as that was the only option working for me (and I tried clusterExport(cl, library(dplyr))` and clusterCall(cl, library(dplyr)...All I got were messages object: forcats not found)

Collectives™ on Stack Overflow

R: How to source variable files on each parallel cluster node?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related