Repository navigation
BUG: f2py cannot compile files it used to be able to compile #24008
Description
Describe the issue:
I'm attempting to use f2py to compile some fortran code. I'm able to do this using the following numpy versions:
- 1.21.0
- 1.22.0
- 1.23.0
- 1.24.0
- 1.24.2
Starting on v1.24.3, the same compilation code no longer works using f2py. Strangely enough, if I precompile one of files (LAPACK.f), the compilation can work with v1.24.3.
Reproduce the code example:
# I don't see how I can produce runnable code using f2py, as it requires the fortran source code
# This code produces a .so file in my project's home directory on numpy <= 1.24.2 but doesn't work on numpy >= 1.24.3
from pathlib import Path
from numpy import f2py
project_path = Path(__file__).resolve().parent
disort_directory = project_path.joinpath('disort4.0.99')
module_name = 'disort'
fortran_source_filenames = ['BDREF.f', 'DISOBRDF.f', 'ERRPACK.f', 'LAPACK.f', 'LINPAK.f', 'RDI1MACH.f']
fortran_paths = [disort_directory.joinpath(f) for f in fortran_source_filenames]
with open(disort_directory.joinpath('DISORT.f')) as disort_module:
f2py.compile(disort_module.read(), modulename=module_name, extra_args=fortran_paths)
# If I precompile LAPACK.f using:
# /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -g -fno-second-underscore -fPIC -O3 -funroll-loops -c LAPACK.f
# then it works using numpy = 1.24.3. Note the only difference in the code is the LAPACK.f is now LAPACK.o
project_path = Path(__file__).resolve().parent
disort_directory = project_path.joinpath('disort4.0.99')
module_name = 'disort'
fortran_source_filenames = ['BDREF.f', 'DISOBRDF.f', 'ERRPACK.f', 'LAPACK.o', 'LINPAK.f', 'RDI1MACH.f']
fortran_paths = [disort_directory.joinpath(f) for f in fortran_source_filenames]
with open(disort_directory.joinpath('DISORT.f')) as disort_module:
f2py.compile(disort_module.read(), modulename=module_name, extra_args=fortran_paths)Error message:
There is no error message. f2py simply stops right before where it prints this line:
INFO: compiling Fortran sourcesRuntime information:
Line 1 output:
1.24.0
3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
(note that I have no idea why it says 1.24.0 when Pycharm assures me I'm using 1.24.2)
Line 2 output:
Exception ignored on calling ctypes callback function: <function ThreadpoolController._find_libraries_with_dl_iterate_phdr..match_library_callback at 0x7f592f9c00d0>
Traceback (most recent call last):
File "/repos/pyRT_DISORT/venv/lib/python3.10/site-packages/threadpoolctl.py", line 584, in match_library_callback/repos/pyRT_DISORT/venv/lib/python3.10/site-packages/threadpoolctl.py", line 725, in _make_controller_from_path
self._make_controller_from_path(filepath)
File "
lib_controller = lib_controller_class(
File "/pyRT_DISORT/venv/lib/python3.10/site-packages/threadpoolctl.py", line 842, in init/repos/pyRT_DISORT/venv/lib/python3.10/site-packages/threadpoolctl.py", line 810, in init
super().init(**kwargs)
File "
self._dynlib = ctypes.CDLL(filepath, mode=_RTLD_NOLOAD)
File "/usr/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: dlopen() error
[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}}]
None
Context for the issue:
I believe that LAPACK.f is one of the more widely used fortran codes. If it cannot compile in conjunction with other code, that could potentially disrupt a good number of users.
Activity
seberg commentedon Jun 21, 2023
Ping @HaoZeke, also to confirm, can you check if the issue persists on 1.25.0?
kconnour commentedon Jun 21, 2023
@seberg I can confirm that the issue persists on 1.25.0. It's actually why I noticed it in the first place. My code that used to run failed and then I tracked it down to the version number described above, in hopes that someone might have a better idea what caused it.
HaoZeke commentedon Jun 21, 2023
On
1.24.4the error is:Where it seems that the relevant error is from changes to character handling:
getstrlength: expected a signature of a string but got: {'typespec': 'character', 'charselector': {'*': ''}, 'attrspec': [], 'dimension': ['*']} getstrlength: expected a signature of a string but got: {'typespec': 'character', 'charselector': {'*': ''}, 'attrspec': [], 'dimension': ['*']} /tmp/tmp5l8spqw9/src.linux-x86_64-3.9/disortmodule.c: In function 'f2py_rout_disort_iparmq': /tmp/tmp5l8spqw9/src.linux-x86_64-3.9/disortmodule.c:4772:58: error: expected expression before ',' token 4772 | capi_name_as_array = ndarray_from_pyobj( NPY_STRING,,name_Dims,name_Rank, capi_name_intent,name_capi,capi_errmess); | ^ /tmp/tmp5l8spqw9/src.linux-x86_64-3.9/disortmodule.c:4787:58: error: expected expression before ',' token 4787 | capi_opts_as_array = ndarray_from_pyobj( NPY_STRING,,opts_Dims,opts_Rank, capi_opts_intent,opts_capi,capi_errmess); |OTOH I would suspect something in the character handling of #23356 or #23194
Diffing the outputs b/w
1.24.3and1.24.4only show:Indeed,
f2py -c --f90flags='-O3' -m disort BDREF.f DISOBRDF.f ERRPACK.f LAPACK.f LINPAK.f RDI1MACH.f skip: iparmqdoes seem to compile as well.Will investigate ASAP. P.S. @kconnour the code being tested is Pythonic-Distort right?
kconnour commentedon Jun 21, 2023
Hi @HaoZeke, thanks for investigating! I'm really glad to see someone else is getting a similar error.
Actually, the code I'm testing is on the api branch of my repo but it should be extremely similar to the code in the repo you linked. We're both apparently trying to make a front-end to a popular open-source fortran algorithm. Note that I coded a workaround to this problem in pyproject.toml, where I force it to install numpy==1.24.0 in order to circumvent this issue... so if you ran the installation script from my repo, it shouldn't encounter this error.
I'm happy to provide any additional info to help diagnose this issue!
Akahan12 commentedon Jul 7, 2023
We ran into similar issues with our code, and after testing, it's because of having a decimal inside of a parameter declaration. Once that happens, everything later stops processing properly. The same seems be happening in the examples above
If you look at the error log posted, #24008 (comment), you'll see the first error message is the attached.
You could confirm whether this also works for your code by modifying that parameter to just be ONE=1, ZERO=0, and recompiling. If it gets past that point, that's the problem.
We worked around this by reverting crackfortran.py to prior to the changes introduced here: https://github.com/numpy/numpy/pull/23637/files
As to speculation: I think that the changes to the if blocks in https://github.com/numpy/numpy/pull/23637/commits caused an issue that assumes the input is an integer. Even more speculative - I see the kind selector is in that portion, and the kind selector requires integers.
(https://numpy.org/doc/stable/f2py/advanced.html#dealing-with-kind-specifiers) . But I'm not sure - I didn't spend enough time on understanding crackfortran.py once we discovered that reverting crackfortran.py fixed our problem.
BUG: Fix parameter and length issues
BUG: Fix f2py F77 assumed length issues
BUG: Fix f2py F77 assumed length issues
BUG: Rework character array assignments (numpygh-24008)
BUG: Fix f2py F77 assumed length issues
BUG: Rework character array assignments (numpygh-24008)
BUG: Fix f2py F77 assumed length issues
BUG: Rework character array assignments (numpygh-24008)
BUG: Fix f2py F77 assumed length issues
BUG: Rework character array assignments (numpygh-24008)