[BUG]: libstdc++ issues / `GLIBCXX` not found

[agbuckley]

What happened?

Hi Miles, this is the issue flagged to you on Twitter -- sorry it's taken time to report, as I was drowning in exam meetings today!

The issue we saw was on a Centos7/RHEL system, which by default has Python 2.7 (or python3 = 3.6) and GCC 4.8.5 -- too old for much development these days. So we source installed RH software collections from /opt/rh to overload these system defaults with Python 3.8 and GCC8. Usually works fine, at least for straightforward Python, C++, and compilation between them using the tools found in the environment.

But we hit a problem when installing PySR. juliaup install worked fine, but the python3 -m pysr install command failed with a GCLIBCXX_* version-symbol mismatch: usually the indication of a mismatch between the active compiler and the C++ stdlib being linked against. (I'm afraid I don't have the direct terminal printouts handy: this happened yesterday when installing for a student, and we closed the terminal and moved on with installation on his Windows laptop.) Above the error message we could see the compile/link command explicitly using the system libstdc++ rather than the override from /opt/rh (e.g. /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libstdc++.so) -- again, I can provide the full output later if useful.

Seems to me that something in the pysr install code is not picking up the overloaded compiler/stdlib locations, maybe assuming that the system ones are valid. It'd be best to query the appropriate ones from the environment, but maybe there is a way around like exporting library-path variables while running the installer? Thanks!

Version

Latest from pip

Operating System

Linux

Package Manager

pip

Interface

Script (i.e., python my_script.py)

Relevant log output

TBC

Extra Info

No response

[MilesCranmer]

Thanks for the bug report!

Some notes:

1. Python 3.6 is past its end-of-life and is no longer maintained by the Python organization. See https://endoflife.date/python. For PySR I only test against Python 3.7 to 3.11, so I cannot guarantee that it will work on earlier versions. I would recommend you upgrade Python to a version that is still maintained, maybe using pyenv: https://github.com/pyenv/pyenv#installation. I've had good luck using pyenv on my institutional cluster where I do not have sudo access.

2. The python -m pysr install triggers some Julia install commands, but it does not install any C libraries itself, so C compilation/linking issues are very unexpected. My guess is that the Julia environment has a problem with it (or maybe the Python install itself). Could you try installing the Julia libraries manually, using the following command?

julia -e 'using Pkg; pkg"activate --temp"; pkg"add SymbolicRegression PyCall"; pkg"build PyCall"; pkg"precompile"'

The result of this will help me debug further. There is a chance its some issue of PyCall.jl trying to talk to Python.

[IamShubhamGupto]

Hey

I am not sure if it's related but I am having problems installing PySR in my GitHub actions. It seems to be working in the last couple of weeks so the change is something very recent.

Here's the logs where it seems to be failing

Precompiling PyCall...
Precompiling PyCall... DONE
PyCall is installed and built successfully.
ERROR: Unable to dlopen(cxxpath) in parent!
Message: /usr/share/miniconda/envs/L96M2lines/lib/python3.9/site-packages/scipy/sparse/../../../../libstdc++.so.60�P�L�: cannot open shared object file: No such file or directory
Error: Process completed with exit code 1.

complete logs can be found here

let me know if the issue is related or if I have gone wrong somewhere.

Thank you!

[MilesCranmer]

Okay I think this is related to a very nasty bug we have been working on for the past week at the intersection of conda-forge, PyCall.jl, and the new version of libstdcxx-ng in conda-forge. See, e.g., conda-forge/scipy-feedstock#238, conda-forge/pysr-feedstock#89, JuliaPy/PyCall.jl#1040, conda-forge/pysr-feedstock#90.

Apologies for the delay in fixing this. Until the conda-forge job in PySR is listed as "passing" again, I would recommend one of two options:

1. Switch to the pip install of PySR, with a juliaup based install of Julia
2. Create a new conda environment with the constraint libstdcxx-ng<13:

conda create -n pysr pysr 'libstdcxx-ng<13' -c conda-forge

[MilesCranmer]

@agbuckley @IamShubhamGupto it seems like the conda-forge build is working again (a temporary fix, but a fix nonetheless, until some upstream bugs are patched). Could you please re-install things and let me know how it goes?

[IamShubhamGupto]

@MilesCranmer
Hey, thanks for the update. We decided to try installing PySR using condaand it worked just fine. I hope it does not break in the future.
environment.yml

in the GitHub action, we just commented out the pip install commands
GitHub workflow

[agbuckley]

Hi @MilesCranmer , back to the original issue, I'm actually using Python 3.8 via the Software Collection: Py 3.6 is the system version of Py3 that we're specifically overriding, along with the compiler. Running again, here's the error message I get from the python3 -m pysr install action:

Precompiling project...
  9 dependencies successfully precompiled in 66 seconds. 3 already precompiled.

Precompiling PyCall...
Precompiling PyCall... DONE
PyCall is installed and built successfully.
ERROR: Unable to load dependent library /home/ppe/a/abuckley/.julia/juliaup/julia-1.9.1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1
Message:/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/ppe/a/abuckley/.julia/juliaup/julia-1.9.1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1)

This is the ABI version error I mentioned: it should be picking up the overloaded /opt/rh/devtoolset-8/root/lib/gcc/x86_64-redhat-linux/8/libstdc++.so library, not the system's /lib64/libstdc++.so.6 that is apparently being referenced by libjulia-internal.so.1 (or being found by the dynamic loader in the pysr install environment).

I'll try the manual Julia package installs now, to see if it is in the way pysr is executing that. Will post the result in a separate comment for clarity.

[agbuckley]

It was looking positive: the manual Julia package installs worked fine using your line above. And all the PySR import and model setup (on the first example, copied in verbatim) works fine at the Python 3.8 REPL. But then failure again with the wrong library pickup when PySR's model.fit() dispatches to Julia, via PyCall I presume:

...
>>> model.fit(X, y)
Compiling Julia backend...
ERROR: Unable to load dependent library /home/ppe/a/abuckley/.julia/juliaup/julia-1.9.1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1
Message:/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/ppe/a/abuckley/.julia/juliaup/julia-1.9.1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1)

So it does look like some mishandling of the environment via PyCall, picking up the system libs rather than the live environment's ones. Thanks.

[MilesCranmer]

The part I don’t understand is that PyJulia/PyCall never try to load libstdc++.so directly. They only try to load libpython.so. Are you able to run Julia code directly? (eg SymbolicRegression.jl?)

[MilesCranmer]

Ah, wait, I found this in the PyJulia docs:

Error due to ``libstdc++`` version

When you use PyJulia with another Python extension, you may see an error
like :literal:`version `GLIBCXX_3.4.22' not found` (Linux) or
``The procedure entry point ... could not be located in the dynamic link library libstdc++6.dll``
(Windows). In this case, you might have observed that initializing
PyJulia first fixes the problem. This is because Julia (or likely its
dependencies like LLVM) requires a recent version of ``libstdc++``.

Possible fixes:

-  Initialize PyJulia (e.g., by ``from julia import Main``) as early as
   possible. Note that just importing PyJulia (``import julia``) does
   not work.
-  Load ``libstdc++.so.6`` first by setting environment variable
   ``LD_PRELOAD`` (Linux) to
   ``/PATH/TO/JULIA/DIR/lib/julia/libstdc++.so.6`` where
   ``/PATH/TO/JULIA/DIR/lib`` is the directory which has
   ``libjulia.so``. macOS and Windows likely to have similar mechanisms
   (untested).
-  Similarly, set environment variable ``LD_LIBRARY_PATH`` (Linux) to
   ``/PATH/TO/JULIA/DIR/lib/julia`` directory. Using
   ``DYLD_LIBRARY_PATH`` on macOS and ``PATH`` on Windows may work
   (untested).

See: https://github.com/JuliaPy/pyjulia/issues/180,
https://github.com/JuliaPy/pyjulia/issues/223

Could you try those solutions?

[agbuckley]

Unfortunately...

$ python
Python 3.8.13 (default, Aug 16 2022, 12:16:29) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysr
>>> pysr.julia_helpers.init_julia()
ERROR: Unable to load dependent library /home/ppe/a/abuckley/.julia/juliaup/julia-1.9.1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1
Message:/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/ppe/a/abuckley/.julia/juliaup/julia-1.9.1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1)

I'm not quite sure what the issue is: the impression I got was that Julia commands standalone were correctly picking up the overloaded GCC8 libs (including libstdc++), but that this was broken when calling the Julia from Python. However, I have a lot of experience of using this Python+GCC overload for Python-C++ wrappers, and the binary compatibility there is fine (even though, as you can see above, the Python executable was built with GCC9).

[agbuckley]

Just noticed that that that import line appeared in the email version of your message, but not the one above. I guess you edited it. Importing from julia import Main gave a similar error, and the LD_PRELOAD wasn't happy to force-preload the GCC8 libstdc++.so. The Julia lib directory wasn't obvious within the .juliaup directory: that only seems to contain the bin dir and its executables. Sorry to not be more helpful.