Migrate codebase to Python 3 and fix related issues - #158
Open
CuriousAvenger wants to merge 56 commits into
Open
Migrate codebase to Python 3 and fix related issues#158CuriousAvenger wants to merge 56 commits into
CuriousAvenger wants to merge 56 commits into
Conversation
- Replace Python 2 print statements with print() calls - Replace except (ExcType, e): with except ExcType as e: syntax (mysqldef.py, sqlcl.py) - Replace map(None, X, Y) with zip(X, Y) in lscastrodef.py linreg() - Replace urllib/urllib2 with urllib.request/urllib.parse (externaldata.py, sqlcl.py) - Replace file() built-in with open() (util.py, externaldata.py) - Replace xrange() with range() and remove string module usage (cosmics.py, sqlcl.py) - Replace np.cast[] with np.asarray(..., dtype=) (cosmics.py) - Replace np.Inf with np.inf (cosmics.py) - Replace np.alen() with len() (cosmics.py) - Replace ndimage.morphology.*/ndimage.measurements.*/ndimage.filters.* with top-level ndimage.* - Replace pkg_resources with importlib.resources (util.py) - Remove Python 2/3 conditional in userinput() (util.py) - Replace raw string in re.sub() call (lscpsfdef.py, myloopdef.py) - Use integer floor division in rebin() (cosmics.py) - Add odrpack as fallback import for scipy.odr in lscabsphotdef.py - Update setup.py: distutils -> setuptools, add odrpack to requires - Update .gitignore: add build artifacts, __pycache__, .pytest_cache, editor/OS files, pyraf cache, and data file patterns
- cosmics.py (growmask): fix variable name typo 'dismask' -> 'dilmask' in the else-branch of growmask(); the function returned the correct 'dilmask' name but the assignment used the wrong name, causing NameError at runtime when size is neither 3 nor 5. - util.py (readkey3): fix incorrect string escape sequence '\#' -> '\#'; the single-backslash form is not a recognised Python escape sequence and was silently treating '#' without stripping the backslash, producing wrong header keyword values. - myloopdef.py (checkfilevsdatabase): fix missing index on lista['filename'] inside a for-loop; 'lista["filename"].replace(...)' operated on the whole column Series instead of the i-th element, causing incorrect .sn2.fits path construction and potential AttributeError.
Python 2 runs often avoided this failure because this fallback path was less frequently triggered and numeric values more often matched built-in float identity.\n\nIn Python 3 migration runs, FITS header values can be valid numeric scalar types that are not identical to float under exact type checks.\n\nThe previous logic used strict type identity, which is not robust across numeric representations.\n\nThis patch uses explicit float coercion with the same exception path for truly invalid values.
In Python 2-era environments, mixed string/numeric arrays passed through this code path with more permissive coercion behavior in the surrounding scientific stack.\n\nIn Python 3 migration runs with modern NumPy, np.where over string values (for example 'INDEF') and numeric fallback values (for example 9999) now triggers explicit dtype promotion checks. Under NumPy 2 this yields DTypePromotionError when no common non-object dtype exists.\n\nThis patch removes mixed-type np.where array construction and normalizes each photometry vector through explicit float coercion with a controlled sentinel fallback.\n\nResult: the SN2 FITS table columns are consistently float-typed, preserving prior semantics while remaining compatible with current Python 3/NumPy behavior.
MySQL/MariaDB clients on ARM Linux default to requiring TLS (--ssl-mode=REQUIRED), causing 'ERROR 2026: SSL is required' when connecting to a local MariaDB container that has no TLS certificates configured. This regression does not occur on x86 because the default client SSL policy differs between architectures and client versions. Fix: detect whether the installed mysql client supports --ssl-mode (MySQL 5.7.11+) or only --ssl=0 (older MariaDB clients), then pass the appropriate flag to suppress TLS for loopback container-to-container connections where encryption is unnecessary.
…backend In Python 2, raw_input() was implemented in C and cooperated with the OS scheduler in a way that allowed Tk's event loop to continue servicing Expose/paint events on the main thread between keystrokes. In Python 3, input() delegates to sys.stdin.readline(), which is a hard blocking call on the main thread. Because Matplotlib's TkAgg backend drives its canvas entirely on that same thread, the call to input() starves the Tk event queue: the window opens, receives one paint from the initial plt.draw(), then immediately blocks. On ARM Docker + XQuartz this race is lost and the window appears black. Fix (myloopdef.py): run userinput() in a daemon thread and spin plt.pause(0.1) on the main thread while waiting, keeping the Qt/Tk event loop alive throughout. Fix (util.py): guard userinput() against non-TTY stdin (docker exec -i) and EOFError, returning '' so the loop exits cleanly rather than crashing. Fix (Dockerfile / docker-compose.yml / bashrc): install pyqt via conda and set MPLBACKEND=QtAgg. TkAgg is structurally fragile under Python 3 + XQuartz + ARM because any blocking call on the main thread stalls rendering; QtAgg uses its own native event loop and is robust to the threading model used here.
The previous image hard-coded an x86 DS9 package, which fails on ARM64 at runtime with a missing dynamic loader (/lib64/ld-linux-x86-64.so.2). The build now resolves the DS9 package from uname -m and installs an architecture-matched binary (x86_64 or aarch64). This removes host-dependent runtime breakage and makes the container reproducible across heterogeneous Linux platforms.
iraf.display() requires a running XImtool/DS9 image display server bound to the IRAF image device (imtool). When no server is present the IRAF task terminates with an IrafError, aborting the PSF stage entirely. This is a runtime environment issue, not a code defect, and is equally possible under Python 2 and 3. Fix: wrap the display/tvmark block in a broad Exception catch so the PSF fitting pipeline continues uninterrupted and the user receives an informative warning rather than an unhandled traceback.
In Python 2, Docosmic used the bundled lsc.cosmics (L.A.Cosmic) implementation unconditionally. The Python 3 migration replaced this with astroscrappy, a compiled C extension offering the same algorithm with better performance, but made the dependency mandatory with an unconditional import. On environments where astroscrappy is not installed the import raises ModuleNotFoundError, which propagates through multiprocessing.Pool and aborts the cosmics stage entirely — a regression relative to Python 2 behaviour. Fix: wrap the astroscrappy import in a try/except ImportError inside the existing Python 3 branch. When astroscrappy is absent the code falls back to lsc.cosmics, preserving Python 2 parity. When astroscrappy is installed the fast path is taken unchanged.
In Python 2, zip() returned a materialised list of tuples with a known length
and shape, allowing NumPy and astropy.wcs to infer dtype('float64') directly.
In Python 3, zip() returns a lazy iterator with no __len__ or element-type
information; NumPy therefore allocates a 0-d object array (dtype('O')), which
astropy's wcs.p2s refuses to cast to float64, raising TypeError at the
wcs_pix2world call site.
Fix: wrap the three affected zip() calls in list() to restore the eager
evaluation semantics that Python 2 provided implicitly.
This reverts commit 7454aec.
Python 2 allowed float result from / 2 in array shape division, but Python 3 returns float and slice indices must be integers. This patch uses integer division for yctr/xctr to fix difftype 1 in lscdiff.py.
… logic" This reverts commit 06fa68a.
…d ingestion" This reverts commit ac0d26d.
In Python 3, numpy cannot compare None with str using '<', which causes argsort inside astropy's group_by to fail with TypeError when multiple standards in the same filter have different zcol1/zcol2 values (one NULL from MySQL, one a string like 'UB'). The fix replaces None values in zcol1/zcol2 columns with empty strings before calling group_by, allowing the sort to proceed without errors. This is a Python 2→3 regression: Python 2 allowed cross-type comparisons (None < 'str' returned True), but Python 3 raises TypeError. Co-Authored-By: Claude Opus 4.8 <[email protected]>
readnoi was set to gain and epadu to ron; corrected to readnoi=ron, epadu=gain to match lscastrodef/lscpsfdef. Fixes DAOPHOT photometric error terms.
Seed fsolve near the analytic SNR root (Nsigma^2/gain) instead of np.median(data). On difference images the median is ~0, which sits left of the SNR curve's minimum where the sqrt goes imaginary; fsolve stepped into the nan wall, returned ~0, and limmag blew up to ~104 instead of ~21. The old seed also carried cross-build float noise (~1e-13 median differences), making the stall flip per-file between environments. The new seed is a fixed constant, so limmag is now stable and correct on diff images.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.