Errors when parallelizing for loop

Question

I'm new to parallel-processing and attempting to parallelize a for loop in which I create new columns in a data frame by matching a column in said data frame with two other data frames. j, the data frame I'm attempting to create columns in is 400000 x 54. a and c, the two data frames I'm matching j with are 5000 x 12 and 45000 x 8 (respectively).

Below is my initial loop prior to the attempt at parallelizing:

for(i in 1:nrow(j)) {
   if(j$Inspection_Completed[i] == TRUE) {
      next
  }

  j$Assigned_ID <- a$Driver[match(j$car_name, a$CarName)]

  j$Title <- c$Title[match(j$Site_ID, c$LocationID)]

  j$Status <- c$Status[match(j$Site_ID, c$LocationID)]    
 

}

So far I have attempted the following:

cl <- snow::makeCluster(4)
doSNOW::registerDoSNOW(cl)

foreach::foreach(i = 1:nrow(j)) foreach::`%dopar%` {
   if(j$Inspection_Completed[i] == TRUE) {
      next
   }

  j$Assigned_ID <- a$Driver[match(j$car_name, a$CarName)]

  j$Title <- c$Title[match(j$Site_ID, c$LocationID)]

  j$Status <- c$Status[match(j$Site_ID, c$LocationID)] 
} 
stopCluster(cl)

However, when I run the code above I receive several errors.

Error: unexpected symbol in "foreach::foreach(i = 1:nrow(j)) foreach"

And:

Error: object 'i' not found

Lastly:

Error: unexpected '}' in "}"

I'm not sure why I'm getting these errors. None of the columns in any of the data frames are factors and I haven't been able to spot any mismatched parentheses or brackets. I've also done this without the snow and doSNOW packages and the result is the same. I've ran it without the tick marks around dopar as well with the same result.

The function name is %dopar%, not dopar. Try foreach::foreach(i = 1:nrow(j)) foreach::%dopar% .... — r2evans
– r2evans, Commented May 3, 2021 at 0:14
Not sure if you were thinking this, but just in case ... you cannot wrap a function in % and expect it to become an infix function. Quite the opposite, a function needs to be defined as being wrapped in % for R to use it as an infix operator. — r2evans
– r2evans, Commented May 3, 2021 at 0:17
@r2evans the absence of % with dopar was a typo on my part. Even with %dopar% the errors are the exact same. — ddw2303
– ddw2303, Commented May 3, 2021 at 0:57

r2evans · Accepted Answer · 2021-05-03 01:02:31Z

1

(I didn't know this before.)

R doesn't like infix operators with the ::-notation. Even if you're doing that for namespace management, R isn't having it:

1L %in% 1:2
# [1] TRUE
1L base::%in% 1:2
# Error: unexpected symbol in "1L base"
1L base::`%in%` 1:2
# Error: unexpected symbol in "1L base"

Workarounds:

Redefine your own infix that just mimics the other, as in
```
`%myin%` <- base::`%in%`
1L %myin% 1:2
# [1] TRUE
```
Use explicit namespace inclusion with library(foreach) before that point in your code, and just use %dopar%. (Not that it helps much, but using library(foreach) does not mean you cannot use foreach::foreach, though it is unnecessary.)

answered May 3, 2021 at 1:02

r2evans

167k8 gold badges92 silver badges176 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Errors when parallelizing for loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related