Use Python 3 stable ABI to build MacVim, and better Python discovery

Vim added support for using Python 3 stable ABI in 9.0.1776, which
allows us to safely load Python libraries of a different version from
what Vim was built against. Modify our CI to use that. This allows user
to use whatever Python version they want as long as it's above the
minimum target. Given that macOS/Xcode still ships with 3.9 by default,
we build using 3.9 as the minimum version.

Also, change our Python detection script to work better. Change all
explicit versions in our paths to refer to the "Current" version instead
which for the most part should "just work" instead of requiring an exact
match every time we or Python update to a new version (e.g. Homebrew
will update the Current version to point to the latest Python3). Also
add support for finding Python 3 from Xcode Command Line Tools which was
previously not ok to use technically because it's 3.9 and before stable
ABI support we couldn't load it safely as MacVim was built using newer
versions.

Fix #1351.

Author: ychin

.github/workflows/ci-macvim.yaml

@@ -21,16 +21,15 @@ env:
   CC: clang
 
   VERSIONER_PERL_VERSION: '5.30'  # macOS default Perl installation uses this to determine which one to use
-  PYTHON3_VERSION: '3.11'         # Make sure to keep src/MacVim/vimrc synced with the Python version here for the Python DLL detection logic.
 
   vi_cv_path_python: /usr/local/bin/python
   vi_cv_path_python3: /usr/local/bin/python3
   vi_cv_path_plain_lua: /usr/local/bin/lua
   vi_cv_path_ruby: /usr/local/opt/ruby/bin/ruby
   vi_cv_dll_name_perl: /System/Library/Perl/%s/darwin-thread-multi-2level/CORE/libperl.dylib
   vi_cv_dll_name_python: /usr/local/Frameworks/Python.framework/Versions/2.7/Python
-  vi_cv_dll_name_python3: /usr/local/Frameworks/Python.framework/Versions/%s/Python
-  vi_cv_dll_name_python3_arm64: /opt/homebrew/Frameworks/Python.framework/Versions/%s/Python
+  vi_cv_dll_name_python3: /usr/local/Frameworks/Python.framework/Versions/Current/Python
+  vi_cv_dll_name_python3_arm64: /opt/homebrew/Frameworks/Python.framework/Versions/Current/Python
   vi_cv_dll_name_ruby: /usr/local/opt/ruby/lib/libruby.dylib
   vi_cv_dll_name_ruby_arm64: /opt/homebrew/opt/ruby/lib/libruby.dylib
   vi_cv_dll_name_lua_arm64: /opt/homebrew/lib/liblua.dylib
@@ -140,22 +139,8 @@ jobs:
             brew unlink perl
           fi
 
-          # With Perl and Python, we need to manually specify the version number for various reasons
-          # (e.g. library paths include the version number). Because of that, check that the version
-          # matches the expectation and fail if they don't, so we can go and fix it. Only do for
-          # Python because a wrong Perl version would just fail the build later.
-          if [[ "$(python3 --version)" != *"Python $PYTHON3_VERSION"* ]]; then
-            printf "\n"
-            echo "Wrong Python 3 version: $(python3 --version). Expected $PYTHON3_VERSION."
-            printf "\n"
-            echo "This likely happened because Homebrew was updated to point to a newer version of"
-            echo "Python 3. Update PYTHON3_VERSION to fix this."
-            exit -1
-          fi
-
+          # With Perl, we need to manually specify the version number because the dylib path depends on it.
           echo "vi_cv_dll_name_perl=$(printf $vi_cv_dll_name_perl $VERSIONER_PERL_VERSION)" >> $GITHUB_ENV
-          echo "vi_cv_dll_name_python3=$(printf $vi_cv_dll_name_python3 $PYTHON3_VERSION)" >> $GITHUB_ENV
-          echo "vi_cv_dll_name_python3_arm64=$(printf $vi_cv_dll_name_python3_arm64 $PYTHON3_VERSION)" >> $GITHUB_ENV
 
       # All set up steps are done. Build and test MacVim below.
 
@@ -176,6 +161,7 @@ jobs:
               --enable-perlinterp=dynamic
               --enable-pythoninterp=dynamic
               --enable-python3interp=dynamic
+              --with-python3-stable-abi=3.9 # macOS and Xcode currently ships 3.9, so we don't want go higher than that.
               --enable-rubyinterp=dynamic
               --enable-luainterp=dynamic
               --with-lua-prefix=/usr/local

src/MacVim/vimrc

@@ -10,34 +10,43 @@ set nocompatible
 set backspace+=indent,eol,start
 
 " Python2
-" MacVim is configured by default to use the pre-installed System python2
-" version. However, following code tries to find a Homebrew, MacPorts or
-" an installation from python.org:
+" MacVim is configured by default in the binary release to use the
+" pre-installed System python2 version. However, following code tries to
+" find a Homebrew, MacPorts or an installation from python.org:
 if exists("&pythondll") && exists("&pythonhome")
+  " Homebrew python 2.7
   if filereadable("/usr/local/Frameworks/Python.framework/Versions/2.7/Python")
-    " Homebrew python 2.7
     set pythondll=/usr/local/Frameworks/Python.framework/Versions/2.7/Python
+
+  " MacPorts python 2.7
   elseif filereadable("/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python")
-    " MacPorts python 2.7
     set pythondll=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python
+
+  " https://www.python.org/downloads/mac-osx/
   elseif filereadable("/Library/Frameworks/Python.framework/Versions/2.7/Python")
-    " https://www.python.org/downloads/mac-osx/
     set pythondll=/Library/Frameworks/Python.framework/Versions/2.7/Python
   endif
 endif
 
 " Python3
-" MacVim is configured by default to use Homebrew python3 version
-" If this cannot be found, following code tries to find a MacPorts
-" or an installation from python.org:
+" MacVim is configured by default in the binary release to set
+" pythonthreedll to Homebrew python3. If it cannot be found, the following
+" code tries to find Python3 from other popular locations.  Note that we are
+" using "Current" for the version, because Vim supports the stable ABI and
+" therefore any new version of Python3 will work.
 if exists("&pythonthreedll") && exists("&pythonthreehome") &&
       \ !filereadable(&pythonthreedll)
-  if filereadable("/opt/local/Library/Frameworks/Python.framework/Versions/3.11/Python")
-    " MacPorts python
-    set pythonthreedll=/opt/local/Library/Frameworks/Python.framework/Versions/3.11/Python
-  elseif filereadable("/Library/Frameworks/Python.framework/Versions/3.11/Python")
-    " https://www.python.org/downloads/mac-osx/
-    set pythonthreedll=/Library/Frameworks/Python.framework/Versions/3.11/Python
+  " MacPorts python
+  if filereadable("/opt/local/Library/Frameworks/Python.framework/Versions/Current/Python")
+    set pythonthreedll=/opt/local/Library/Frameworks/Python.framework/Versions/Current/Python
+
+  " macOS default Python, installed by 'xcode-select --install'
+  elseif filereadable("/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/Current/Python3")
+    set pythonthreedll=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/Current/Python3
+
+  " https://www.python.org/downloads/mac-osx/
+  elseif filereadable("/Library/Frameworks/Python.framework/Versions/Current/Python")
+    set pythonthreedll=/Library/Frameworks/Python.framework/Versions/Current/Python
   endif
 endif