Calling procedure from included .mod file, without explicit `use` statement

[rgba]

Can a procedure contained in a included (via flags) .mod file be called, without actually using the module in the caller program/module? I mean having something like

module caller_module
    implicit none
    ...
contains
    ...
    subroutine caller_code(args)
    ...
    call incriminated_procedure(args)
    ...
    end subroutine
    ...
end module

correctly compile and run, while nowhere in the source code there is a declaration for incriminated_procedure, nor a use statement for a module that defines it.

Well, apparently yes, provided that a .mod file for such a module is included via flags: what I have is a $(pkg-config --cflags library_name) $(pkg-config --libs library_name) flag passed to fpm / defined in a Makefile. This library contains a module defining a public incriminated_procedure, so that it would be present in one of the many .mod files that pkg-config is including with the -I path statement produced by --cflags. My naive assumption would have been that if a do not use this module, the compiler would not search for procedures inside its .mod file, but apparently I’m wrong.

Does this sound correct to you? Am I misunderstanding something? Isn’t it somewhat unsafe, I would expect the compiler (gfortran) to be way more intransigent, especially given that I’m passing also these flags:

-O0 -p -g -fimplicit-none -Wsurprising -Waliasing -fwhole-file -fcheck=all -pedantic -fbacktrace -ffree-line-length-none

[ivanpribec]

Take a look at this answer:

What does implicit none (type,external) do?

For reference, I’ll answer the subject question. implicit none (type,external) is a Fortran 2018 feature. implicit none(type) is the same as implicit none, disabling all implicit typing. implicit none(external) requires that any procedures referenced either have an explicit interface or be declared EXTERNAL. You can combine the options. Compilers supporting this may have a command line option to imply implicit none (external) in the same manner as options to disable implicit typing.

I’m guessing your module could have just picked up the procedure from the library and used it as an (implicit) external procedure. It does seem a bit weird though, as module procedures have their names mangled.

[rgba]

Oh, good catch! Didn’t know about that (I guess I assumed it was the default behavior).

Now that I added external to my implicit none statement the compiler correctly errors at the incriminated procedure. Thanks!

[ivanpribec]

What you describe could have easily occured if the incriminated procedure was truly an external procedure, but perhaps had a public interface block in a module.

[rgba]

The “safety” problem I see is that if I wish to set a “clean boundary” with the external library, such as with

use library_module, only: <some-stuff-but-not-that-procedure>

I’m not actually assured I would not pick up unwanted things from the (precompiled) module. I imagine that if I actually define my own incriminated_procedure it would shadow the external one and everything goes fine, but in this case I just forgot to do that and failed to realize, actually running some of my tests in an uncontrolled environment (the idea was to test just an isolated part of the module…).

[rgba]

@ivanpribec

truly an external procedure

Yeah maybe that’s the point I need to clarify. It is actually defined in a non-module file (legacy stuff), which is then included in the library module. I.e.

module library_module
    implicit none
    ...
    public :: incriminated_procedure
    ...
contains
    ...
    include "that_file.f90"
    ...
end module

Does this qualify as a “true” external procedure? I thought I should consider everything in a included file as if pasted there verbatim, but maybe I’m wrong.

[ivanpribec]

That’s a module procedure.

[rgba]

Ok, good. Considering that I’m not using that module (at all, the use, only: statement above was just an example to illustrate my concerns, maybe not really appropriate), in my code I have no use nor include statements that bring in that procedure name. Hence if I call it the compiler assumes it has to be external.

• that’s probably why the implicit none(type,external) statement makes it fail to compile.

The question is then, if I do not restrict implicit external procedures, the compiler starts to look for them… does/should it look inside “reachable” .mod files too? Does it care if they are module procedures in that context, if I do not use the module?

[ivanpribec]

Btw, if you are using fpm, or using globbing to collect source files in a Makefile, it could have been that_file.f90 got compiled too. Change the extension to .inc or something else perhaps.

[rgba]

@ivanpribec

it could have been that_file.f90 got compiled too

No, that I’m sure it could not happen. fpm knows nothing about where the source files are for the included library, it just includes the precompiled .mod files since I feed them as compiler flags, via pkg-config. Also, the very same behavior (compiling and running correctly before, now failing after adding external to implicit none) happens using the Makefile, too (and the Makefile adds every obj one by one).

EDIT: I mean, I could even delete the library source files and everything would happen regardless, since I’m checking out a completely independent repository, and linking the library relying on pkg-config only.

[ivanpribec]

My reasoning:

• absence of the incriminating procedure will only become apparent at link time (unless implicit none (external) is present)
• if linking is successful the incriminated procedure must have come from somewhere
• you can look at into the static libraries included via pkg-config using the nm command

[RonShepard]

I think this (@ivanpribec ) must be what is happening. That subroutine must have been compiled somewhere, and maybe added to some library somewhere, so that it could be found at link time. The fact that a copy of that subroutine was in an unused module is likely just a red herring (a misleading clue).

[rgba]

Thank you all, didn’t know about nm and probably I have some confusion about .mod files. I surely give the compiler library information too, through the pkg-config --libs library_name output. If I understand correctly then I should worry about what’s contained in those library files (so inspect it with nm) instead of the .mod file.

I’ll take a look and report back soon.

[rgba]

Ok then, let’s disclose some details to make easier to follow:

• the incriminated procedure is called bracket, and called in my test code as call bracket(ax,xx,bx,fa,fx,fb,func)

• the library I’m including via pkg-config is called scifor and the module that defines bracket is called sf_optimize

Very similarly to what @kargl has shown if I inspect the include folder containing all the .mod files of the scifor library, I get

$ cat *.mod | gunzip | grep bracket
3735 'bracket' 'sf_optimize' '' 1 ((PROCEDURE UNKNOWN-INTENT MODULE-PROC
3733 'boltzmann_constant_in_inverse_meters_per_kelvin' 0 3734 'bracket'
50 'bracket' 'sf_optimize' '' 1 ((PROCEDURE UNKNOWN-INTENT MODULE-PROC
('bisect' 0 42 'bracket' 0 50 'brent' 0 59 'brentq' 0 66 'broyden1' 0 73

While if I use nm to inspect the library file, I get

$ nm libscifor.a | grep bracket
000000000002bd60 T __sf_optimize_MOD_bracket
000000000001bbc0 T r8vec_bracket_

Finally if I inspect all the .o files as correctly compiled with either the makefile or fpm build --flag "$(pkg-config --cflags scifor) $(pkg-config --libs scifor)" I get

$ nm *.o | grep bracket
                 U bracket_

More specifically

$ nm src_optimize_cgfit_routines.f90.o | grep bracket
                 U bracket_

where optimize_cgfit_routines.90 is the source file for the modules that has calls to bracket.

If I understand correctly what nm is telling me here, my test code lacks a definition for the symbol bracket_ (why the underscore?), so at linking stage something has to be found elsewhere. The library file libscifor.a has not a matching symbol, due to mangling (right?) so it should be coming from there. The included .mod file for sf_optimize does instead have a bracket symbol indeed, so I’d expect it to be linked.

Crosscheck:

[gbellomi@login2 Test_CGfit]$ fpm build --flag "$(pkg-config --cflags scifor) $(pkg-config --libs scifor)"
minimize_krauth.f                      done.
optimize_cgfit_routines.f90            done.
minimize_sascha.f                      done.
VARS_GLOBAL.f90                        done.
OPTIMIZATION.f90                       done.
BATH_AUX.f90                           done.
BATH_FIT.f90                           done.
libCGfit.a                             done.
[100%] Project compiled successfully.

[gbellomi@login2 Test_CGfit]$ yes | fpm clean
Delete build, excluding dependencies (y/n)?  + rm -rf build/gfortran_07A77609A9445D9C

[gbellomi@login2 Test_CGfit]$ fpm build --flag "$(pkg-config --libs scifor)"
minimize_krauth.f                      done.
optimize_cgfit_routines.f90            done.
minimize_sascha.f                      done.
VARS_GLOBAL.f90                        failed.
[ 50%] Compiling...
././src/VARS_GLOBAL.f90:2:6:

   USE SF_CONSTANTS, only: zero
      1
Fatal Error: Cannot open module file ‘sf_constants.mod’ for reading at (1): No such file or directory
compilation terminated.
<ERROR> Compilation failed for object " src_VARS_GLOBAL.f90.o "
<ERROR>stopping due to failed compilation
STOP 1

[gbellomi@login2 Test_CGfit]$ yes | fpm clean
Delete build, excluding dependencies (y/n)?  + rm -rf build/gfortran_FEBAE927C8C8AD11

[gbellomi@login2 Test_CGfit]$ fpm build --flag "$(pkg-config --cflags scifor)"
minimize_krauth.f                      done.
optimize_cgfit_routines.f90            done.
minimize_sascha.f                      done.
VARS_GLOBAL.f90                        done.
OPTIMIZATION.f90                       done.
BATH_AUX.f90                           done.
BATH_FIT.f90                           done.
libCGfit.a                             done.
[100%] Project compiled successfully.

• if I do not pass the --cflags pkg-config output to fpm I cannot compile anything, since my code uses some of the modules from scifor (unfortunately it errors before we can see if it would find bracket or not)

• if I skip instead the --libs entry I get everything working (at least at build level), as much as it did by passing both… I believe this should be enough to prove that bracket is found in the .mod files, not in the library file(s).

To be clear, there is more than just libscifor.a in the --libs output, and I have not looked for the presence of bracket inside the MKLs etc, but the above test should rule out them having any effect on this:

$ pkg-config --libs scifor
-lscifor -L/opt2/sissa/compiler/intel/2020.4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -L/usr/lib64 -lpthread -lm -ldl 

[rgba]

Yes! That’s exactly the point. I removed the use sf_optimize statement on purpose since I’m trying to debug some nasty behavior therein and I prefer to do so in a isolated test code where I explicitly include only what I need for the tests. So the original idea was to copy paste a copy of bracket from sf_optimize to my test code (as I did with many other things).

Since I forgot to do that the expected behavior would have been to fail to compile (and then informing me about forgetting bracket).

It instead compiled correctly, without the use statement, and that’s the reason for the post.

As Ivan suggested above, changing implicit none to implicit none(type, external) in my test code has correctly made the compilation fail and now we were trying to understand how a module procedure could have been brought in as an external one.

Sorry if I miscommunicated what the problem is.

[urbanjost]

Note that if your loader is ld that adding flags for the linker (see “man ld”) can give you information in addition to what nm and ldd and objdump provide.

fpm build --link-flag '-Xlinker --trace -Xlinker --trace -Xlinker --trace-symbol=__m_cli2_MOD_set_args' --verbose |tee x.out

The “-Xlinker --trace -Xlinker --trace” option gives a nice compact list of the files you loaded from;
if you know the mangled name “–trace-symbol=$NAME” tells you who references it and where it was found.

objdump -x $FILENAME

and look for lines with your procedure name in them, with .text in front of them, and then look above and it might show you the filename the routine came from; but if it does not put the mangled name(s) onto the --trace-symbol option and it will tell you were it was loaded, and from what files, for example. I keep meaning to publish some fpm plugins that do some stuff like that and never get around to it.

There are some places where things can go wrong. If you keep your procedures in individual include files
that end in *.f90 but also load the include files into a module and are using wildcards or fpm you might compile a routine twice, once in a module and once externally for example.

So there are some fancy things you can do with objdump and ld options that supplement what nm can do but in this case can you just rename your module procedure so you know it is not being used and see if it still loads? I got a bit lost in the details above. It this in a public github/gitlab repo were someone could take a look at it?

So if the problem is you know the mangled name is bracket_ and you want to know where it is being
loaded from and are using ld(1) underneath it all to load and fpm(1) add

 --link-flag '--Xlinker --trace-symbol=bracket_' --verbose 

and it should tell you where you are loading bracket_ fromadd t

For example, from nm or objdump I know the mangled name I am looking for. It is __m_cli2_MOD_set_args. If I add the appropriate options, I can see who references it and where it comes from:

./usr/bin/ld: build/gfortran_2A42023B310FA28D/M_intrinsics/app_fpm-man.f90.o: reference to __m_cli2_MOD_set_args
/usr/bin/ld: build/gfortran_2A42023B310FA28D/M_intrinsics/libM_intrinsics.a(build_dependencies_M_CLI2_src_M_CLI2.F90.o): definition of __m_cli2_MOD_set_args

[urbanjost]

That is a good point. Also, if a huge number of variables or procedures are not in the modules, adding the “, only : name1, name2” on USE would prevent accidentally bringing baggage into scope.

[rgba]

Mmm, yes, that I could check.

But then why if I put implicit none(type, external) on top of the module that calls bracket then it stops compiling? Or equivalently, as you have seen nm is suggesting that bracket is unlinked, if another module is actually bringing it in that way, this should not happen right?

[urbanjost]

Perhaps this is simpler than we are making it. Did you actually create a program with the procedure loaded? If I make an object file

subroutine callsomething()
    call notthere()
end

and compile it with “gfortran -c callsomething.90” and do “nm callsomething.o” I will see “U notthere_”.
That is totally normal. At load time I can load in anything with the name “notthere”. The U just means it is not defined in that object file; ie. it is external. I was under the impression you loaded successfully.

The unmangled name pretty much ensures it it not trying to find this in a module, where you would usually get something like “__Module_name_MOD._name” (depends on the compiler). You can compile up a file with unsatisfied externals no problem.

If you did build, run "objdump -x $EXECUTABLE|grep -i bracket

Do you get some long mangled name or just “bracket_” or something simple like that?

If you get the simple name, reload with “-Xlinker --trace-symbol=bracket_” and it should tell you where it is coming from.

A big selling point of moduls is that with modules and the implicit none statement you can avoid name collisions like this, with some holes like @kargl mentioned, along with how to avoid it.

[rgba]

@urbanjost

So if the problem is you know the mangled name is bracket_ and you want to know where it is being
loaded from and are using ld(1) underneath it all to load and fpm(1) add

 --link-flag '--Xlinker --trace-symbol=bracket_' --verbose 

and it should tell you where you are loading bracket_

Oh this is so nice! I’ll try it. In general fpm is growing a lot of very ergonomic facilities, I’m still at an early stage of exploring what it can do, but totally intend to keep going. My ultimate goal is to move all the codebase of my research group to fpm (well not really substitute CMake for now, but at least coexisting).

@urbanjost

I got a bit lost in the details above. It this in a public github/gitlab repo were someone could take a look at it?

Yeah, sorry. It is quite difficult to give a clear clue of what’s happening without an explicit reference to the actual code.

Scifor is public, so that part of the problem can easily be inspected directly by anyone wanting to spend time on it:

• Source code in SF_OPTIMIZE that includes the file where bracket is defined, some line above bracket is declared public.

• The included file

Unfortunately the caller code I’m working on is instead private and I cannot decide on my own to change it public. But let me paste here the “header” of the only module that calls bracket, to clarify that there’s no way I’m bringing sf_optimize in by accident:.

MODULE CGFIT_ROUTINES 
   implicit none 
  
   abstract interface 
      function cgfit_func(a) 
        real(8),dimension(:)  ::  a 
        real(8)               ::  cgfit_func 
      end function cgfit_func 
      function cgfit_fjac(a) 
        real(8),dimension(:)       :: a 
        real(8),dimension(size(a)) :: cgfit_fjac 
      end function cgfit_fjac 
   end interface 
  
  
  
   integer                        :: ncom 
   real(8), dimension(:), pointer :: pcom,xicom 
   procedure(cgfit_func),pointer  :: func 
   procedure(cgfit_fjac),pointer  :: fjac 
  
 contains

As you can see no trace of any used module, and if I add the external key word to the implicit none statement I effectively break the linking. So yes, that should prove that bracket is treated as an external procedure in the “client” code, so the linker must find it somewhere else, i.e. your guess is right.

@urbanjost

The unmangled name pretty much ensures it it not trying to find this in a module, where you would usually get something like “__Module_name_MOD._name”

Exactly this is where I fail to follow. The whole point here:

@rgba

$ cat *.mod | gunzip | grep bracket
3735 'bracket' 'sf_optimize' '' 1 ((PROCEDURE UNKNOWN-INTENT MODULE-PROC
3733 'boltzmann_constant_in_inverse_meters_per_kelvin' 0 3734 'bracket'
50 'bracket' 'sf_optimize' '' 1 ((PROCEDURE UNKNOWN-INTENT MODULE-PROC
('bisect' 0 42 'bracket' 0 50 'brent' 0 59 'brentq' 0 66 'broyden1' 0 73

While if I use nm to inspect the library file, I get

$ nm libscifor.a | grep bracket
000000000002bd60 T __sf_optimize_MOD_bracket
000000000001bbc0 T r8vec_bracket_

is that, as far as I understand, the library file does contain a mangled name for bracket, while the .mod file for sf_optimize contains just bracket, so I’m arguing that the linker, when looking for an external procedure named bracket ends up picking it from the included sf_optimize.mod file. This I find surprising, since I have not a use sf_optimize statement in the header above.

@urbanjost

If you did build, run "objdump -x $EXECUTABLE|grep -i bracket

Do you get some long mangled name or just “bracket_” or something simple like that?

If you get the simple name, reload with “-Xlinker --trace-symbol=bracket_” and it should tell you where it is coming from.

Yes, I’d totally go for it now and tell you what I determine.