0

I'm trying compile a large model using ifort and with the -qopenmp flag: 

FC = mpif90 and FCFLAGS = -g -m64 -qopenmp -O3 -xHost -fp-model \
precise -convert big_endian -traceback -r8
FCDEFS = BLAS LITTLE LINUX INTEGER_IS_INT
LFLAGS = -qopenmp

CC = mpicc
CFLAGS = -g -O3
CCDEFS = BLAS LITTLE LINUX INTEGER_IS_INT _ABI64

OMP_NUM_THREADS=2
OMP_STACKSIZE=1000M
OMP_SCHEDULE=STATIC

ulimit -s unlimited

mpprun -n 192 master.exe -e "exp1" -f d1 -t 2700

However, when I try and run the model I get:

mpprun info: Starting impi run on 12 node ( 192 rank X 1 th ) for 22

==================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 19619 RUNNING AT n457
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault 
(signal 11)
mpprun info:   Job terminated with error

Now the thing is, if I compile this model without the OpenMP flag and run it with TotalView, there are no errors and the model executes without error. I'm trying to find a way to track down what is going wrong. Does anyone have any ideas? Where do I start? how can I do simple tests to see why OpenMP exited with a segmentation fault?

Appreciate the help

Improve this question
6
  • 2
    There are many ways OpenMP can trigger a segfault. If the program does not even start, one can think of stackoverflow.com/questions/3070587/… stackoverflow.com/questions/13870564/… stackoverflow.com/questions/13264274/… They should be probably linked as duplicates Commented Nov 19, 2016 at 14:44
  • Can linked libraries cause the program to crash at the start? How can I get error messages apart from "segmentation fault"? Commented Nov 19, 2016 at 16:20
  • 3
    Such an excessive setting for OMP_STACKSIZE is likely to exhaust even "unlimited" stack. Working applications using as much as 40M (thread stack) are rare. Note that mpif90 would not be the correct wrapper for Intel MPI (where it gives you gfortran) nor for OpenMPI (unless you aliased it to the correct underlying name). So you need to be sure that you haven't crossed incompatible MPI implementations. Commented Nov 19, 2016 at 19:05
  • 1
    Have you considered generating a core file and exploring it, also? Commented Nov 21, 2016 at 12:18
  • @Harald That sounds like a good idea. How can I generate one? I've also reduced the stack size to 128M but the problem persists. Commented Nov 21, 2016 at 16:51
Related questions
0
ifort mpi-openmp segmentation fault
3
The best way to solve Segmentation fault in OpenMP
3
MPI_Comm_size Segmentation fault
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.