2

I am starting with just a small chunk of bigger code in R; this code does not require HPC but the steps following it do. The code behaves as expected when I run it in interactive mode. When running the code as a batch job it 'completed' according to slurm and the .out file was empty, but the expected output - a .csv - is not created.

I want to learn how to get R to save its error messages, warnings etc into a text file that I can access after the code has run as a batch file, so I can diagnose what is going wrong, in this instance and as I run more complex jobs.

Following this solution to a similar query, but about running R from the command line, I have tried to use sink(). My job now ends in an error:

"rror: unexpected input in "msg <- file("/niguanak/nigSp_1.Rout", open = "wt") Execution halted

The current code that ends in the error is below. The previous, which 'completed' but did not output the expected .csv file, was the same but lacking the top two and bottom two lines that are related to sink. niguanak is the folder I am running the job from and it is there I have created a blank file nigSp_1.Rout, which remains blank after running the below code.

msg  <- file("/niguanak/nigSp_1.Rout", open = "wt")
sink(msg, type = "message")

library(tidyverse)

#combine sp download and Dryas integrifolia missed from sp download
Ni.sp.dl <- read.csv("/hpcfs/users/a1233466/niguanak/data/0039174-240321170329656.csv")
DI.dl <- read.csv("/hpcfs/users/a1233466/niguanak/data/0045407-240321170329656.csv")
Ni.sp.dl <- rbind(Ni.sp.dl, DI.dl, make.row.names = FALSE)
rm(DI.dl)

#get rid of occurrenceStatus == Absent
Ni.sp.dl <- Ni.sp.dl %>%
  filter(occurrenceStatus == "PRESENT")

#get rid of any less than 15º for invasive
Ni.sp.dl <- Ni.sp.dl %>%
  filter(decimalLatitude > 15)

#Checked this with coordinates cleaner on laptop. None seemed problematic for this analysis.

###reshape and save
Ni.sp.dl_df <- as.data.frame(cbind(Ni.sp.dl$gbifID, Ni.sp.dl$species,
                              Ni.sp.dl$decimalLatitude, Ni.sp.dl$decimalLongitude))

colnames(Ni.sp.dl_df) <- c("ind_id", "tax", "lat", "lon")

#Make sure the numeric columns are numeric!
Ni.sp.dl_df$lat <- as.numeric(Ni.sp.dl_df$lat)
Ni.sp.dl_df$lon <- as.numeric(Ni.sp.dl_df$lon)
Ni.sp.dl_df$ind_id <- as.numeric(Ni.sp.dl_df$ind_id)

#write out
write.csv(Ni.sp.dl_df, file = "/hpcfs/users/a1233466/niguanak/Ni.sp.dl_df_b.csv")

sink(type="message")
close(msg)

I have read the examples and help file for sink() and file() but didn't figure out what I am doing wrong from there either.

Additionally, the complete code with sink etc as above, also seems to work as expected when run interactively.

EDIT:

In trying out various (desperate) solutions I ended up with the following code.

   #capture output
sink("nigSp_1.out", type =  "messages")

#load required packages
library(tidyverse)

#combine downlaod and Dryas integrifolia missed from sp download
Ni.sp.dl <- read.csv("/hpcfs/users/a1233466/niguanak/data/0039174-240321170329656.csv")
DI.dl <- read.csv("/hpcfs/users/a1233466/niguanak/data/0045407-240321170329656.csv")
Ni.sp.dl <- rbind(Ni.sp.dl, DI.dl, make.row.names = FALSE)
rm(DI.dl)

#get rid of occurrenceStatus == Absent
Ni.sp.dl <- Ni.sp.dl %>%
  filter(occurrenceStatus == "PRESENT")

#get rid of any less than 15 for invasive
Ni.sp.dl <- Ni.sp.dl %>%
  filter(decimalLatitude > 15)

#Checked this with coordinates cleaner on laptop. None seemed problematic for this analysis.

###reshape and save
Ni.sp.dl_df <- as.data.frame(cbind(Ni.sp.dl$gbifID, Ni.sp.dl$species,
                              Ni.sp.dl$decimalLatitude, Ni.sp.dl$decimalLongitude))

colnames(Ni.sp.dl_df) <- c("ind_id", "tax", "lat", "lon")

#Make sure the numeric columns are numeric!
Ni.sp.dl_df$lat <- as.numeric(Ni.sp.dl_df$lat)
Ni.sp.dl_df$lon <- as.numeric(Ni.sp.dl_df$lon)
Ni.sp.dl_df$ind_id <- as.numeric(Ni.sp.dl_df$ind_id)

#write out in case of issues
write.csv(Ni.sp.dl_df, file = "/hpcfs/users/a1233466/niguanak/Ni.sp.dl_df.csv")

#return output to console
sink()

This 'completes' in 1 second, but doesn't have any errors in the .err file, no output in the .out file, no "nigSp_1.out" is made nor "Ni.sp.dl_df.csv"

Improve this question
5
  • 1
    This is a syntax error. Unfortunately, I don't see any obvious syntax errors in your code. Could it be encoding issues? I wouldn't use the º character in code comments. Commented Apr 12, 2024 at 5:31
  • Unfortunately, remoting the degree sign did not solve the issue. I have tried quite a lot of variations now with no dice. Would it make sense that you cannot have a connection to a file outside of the node it is running on if you are sending the Rscript to be run on a HPC? Commented Apr 12, 2024 at 5:52
  • But why would that give a syntax error? I don't know anything about slurm, though. Commented Apr 12, 2024 at 5:55
  • I am not sure why it would through a syntax error. I did just come across this cran.r-project.org/doc/contrib/Lemon-kickstart/kr_scrpt.html which suggests there is something to do with needing to print in sink() with an Rscript, but I can't see what I actually need to do. Commented Apr 12, 2024 at 7:47
  • I got some advice on an alternative to sink to get my error data, adding #SBATCH -o <run_directory>/%j.out #SBATCH -e <run_directory>/%j.err to my .sh file. I have then removed the sync commands. It seems to now complete with no errors, but is still not making the .csv so there is something going wrong in there and I can't figure out how to get it to show me. Commented Apr 12, 2024 at 7:48

1 Answer 1

Reset to default
0

In answer to my original question, adding the following to my submission script does output the errors and console outputs (so long as your Rscript is not just a big comment!)

#SBATCH -o /hpcfs/users/username/folder/%j.out
#SBATCH -e /hpcfs/users/username/folder/%j.err

The underlying issue as to why this didn't appear to work at first was that nano was hiding a formatting issue. My code was authored on my computer and pasted into nano; I was using a short example as I got used to a new HPC system. It looked fine in nano but when my colleague looked at it in vim, it was using windows/dos line feeds not posix. Because of that it 'saw' my whole script as a comment, hence no errors or outputs. Having corrected that, it now behaves and outputs as expected.

I also found that you can control the line feeds in Rstudio under tools > global options > code > saving > line end conversion. Switched it to posix. I am now able to cut and paste from code I have working on my PC.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.