refactor: decompose extract.py and __main__.py into focused modules by TPAteeq · Pull Request #1737 · Graphify-Labs/graphify

TPAteeq · 2026-07-08T20:50:23Z

Splits the two largest modules into cohesive, single-responsibility modules —
verbatim moves only, every original import path preserved via re-exports, and
the full suite unchanged (3036 passed, 29 skipped) after every commit.

Why

extract.py (17,054 LOC) and __main__.py (5,368 LOC) were by far the largest
files in the package and mixed several independent subsystems each. This is pure
decomposition: no behavior change, no renames, no reformatting.

Size impact

File	before	after
`graphify/extract.py`	17,054	4,740 (−72%)
`graphify/__main__.py`	5,368	673 (−87%)
`graphify/export.py`	1,671	962 (−42%)

What moved (one subsystem per commit)

extract.py → graphify/extractors/ (continues the MIGRATION.md / #1212 effort):

engine.py — the _extract_generic tree-sitter engine + its 67-function closure and 10 private type-tables (~4.1k LOC).
resolution.py — the cross-file symbol/import resolution passes (60 fns).
models.py — shared types (LanguageConfig, the _Symbol*Fact dataclasses) + two shared caches.
23 bespoke language extractors: dart, rust, go, powershell, fortran, sql, dm, bash, apex, terraform, sln, pascal_forms, json_config, verilog, markdown, pascal, objc, julia.

main.py:

install.py — the whole install/uninstall subsystem and the install/platform CLI dispatch (dispatch_install_cli).
cli.py — every non-install subcommand (dispatch_command).

export.py → graphify/exporters/: html.py, graphdb.py, base.py.

Why it's safe

Verbatim. Each move was validated by AST closure-privacy analysis and byte-identity checks; nothing was rewritten.
No import cycles. Strict layering extract → engine → resolution → models/base; __main__ → cli; the one back-edge (the graphify <path> redirect's recursive main()) uses a lazy import.
Nothing to update downstream. extract.py / __main__.py / export.py re-export every moved name, so from graphify.extract import extract_dart, from graphify.__main__ import claude_install, etc. all still resolve (object identity preserved). No test or importer changed.
Packaging. Added graphify.exporters to the setuptools package list; verified a fresh-venv wheel install imports every new module and runs graphify end-to-end (build a multi-language graph, query/path/explain, install/uninstall, the path-redirect).
ruff check clean; skillgen --check OK.

Left as-is (intentionally)

The rest of extract.py is the dispatcher + config-driven extractor family (js/ts/vue/svelte/astro/xaml) that share _JS_CONFIG/dispatch caches — kept together per MIGRATION.md. llm.py (backend layer entangled with orchestration) and callflow_html.py (one cohesive renderer) were assessed and left untouched.

…tall.py __main__.py was 5,368 LOC, more than half of it per-platform install/uninstall machinery interleaved with the CLI dispatcher. Move that subsystem — 68 functions (all *_install/*_uninstall, _copy_skill_file, _platform_skill_destination, _always_on, etc.) plus 21 module constants (_PLATFORM_CONFIG and the platform skill/hook payload constants) — into a new graphify/install.py, extracted verbatim. Behavior-preserving: - install.py lives in the same package dir as __main__ so packaged-asset lookups via Path(__file__).parent ("always_on"/"skills") resolve unchanged. - __main__ re-exports all 89 moved names, so `from graphify.__main__ import claude_install` (and every other import, incl. private helpers used by tests) keeps working, with object identity preserved. - The install cluster was verified (AST) to have zero back-references into __main__, so no circular import; __main__ imports _PLATFORM_CONFIG et al. one-way. __main__.py drops 5,368 -> 3,642 LOC. Full suite unchanged: 3036 passed, 29 skipped (excluding the env-only openai tests). ruff check clean; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…rs/ package Continues the graphify/extract.py -> graphify/extractors/ split (MIGRATION.md, upstream Graphify-Labs#1212), which already moved blade/zig/elixir/razor. Moves the independent bespoke extractors whose closures are fully private (no shared _extract_generic core, no shared mutable caches), verbatim, following the documented invariants: dart, rust, go, powershell (+psd1 manifest), fortran, sql, dm (dm/dmm/dmi/dmf), bash, apex, terraform, sln, pascal_forms (delphi .dfm + lazarus .lfm), json_config Each language's private helper funcs and constants move with it; only the `extract_<lang>` entry points (plus fortran's _cpp_preprocess, which has a direct unit test) are re-exported from extract.py's facade block, so every existing importer (__main__.py, watch.py, tests) is unchanged and object identity is preserved. Registry (extractors/__init__.py) grows 4 -> 22 langs. Verified: AST closure-privacy analysis (no symbol referenced from outside its moved set except via the facade); byte-identity of every moved span; extract._DISPATCH still resolves every extension; ruff clean; skillgen --check OK. extract.py drops 17,054 -> 13,121 LOC. Full suite unchanged: 3036 passed, 29 skipped (excluding env-only openai tests). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…orters/ export.py mixed nine independent export targets in one 1,671-LOC module. Move the two largest self-contained ones into a new graphify/exporters/ package, verbatim, re-exported from export.py so every importer (from graphify.export import to_html / push_to_neo4j) is unchanged: - exporters/html.py: to_html + its private helpers (_html_script, _html_styles, _hyperedge_script, _viz_node_limit) and MAX_NODES_FOR_VIZ. - exporters/graphdb.py: push_to_neo4j, push_to_falkordb. - exporters/base.py: COMMUNITY_COLORS (shared by the HTML/SVG/Obsidian exporters), homed here so per-format modules and export.py both import it without a cycle. Verified: AST closure-privacy analysis, facade object identity, ruff clean. export.py drops 1,671 -> 962 LOC. Full suite unchanged: 3036 passed, 29 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…ors/ Continues the extract.py -> extractors/ split. Both are self-contained bespoke extractors (verified: closure-private, byte-identical verbatim move, facade identity + _DISPATCH resolution intact): - extractors/verilog.py: extract_verilog + SystemVerilog helpers (_sv_*, _augment_systemverilog_semantics) and _SV_* constants. - extractors/markdown.py: extract_markdown + _resolve_markdown_link and _MD_* regexes. extract.py 13,121 -> 12,632 LOC. Full suite unchanged: 3036 passed, 29 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Moves the cross-file symbol-resolution / import-resolution / decl-def-merge passes — the largest remaining self-contained subsystem in extract.py — into two new modules, verbatim: - extractors/models.py: the shared data types (LanguageConfig, the _Symbol*Fact / _SymbolResolutionFacts dataclasses) and two shared caches (_WORKSPACE_PACKAGE_CACHE, _JS_CACHE_BYPASS_SUFFIXES). Homed here so both extract.py and resolution.py import them without a cycle; the mutable workspace cache keeps a single shared object identity (only ever .clear()'d in place), verified in the smoke test. - extractors/resolution.py: 60 resolution functions (_resolve_*, _collect_*, _apply_symbol_resolution_facts, _disambiguate_colliding_node_ids, _merge_decl_def_classes, the JS/TS/Python import walkers, tsconfig/workspace resolution) and their 12 private constants. extract.py re-exports every moved name, so importers (__main__, watch, tests that reach into _JS_RESOLVE_EXTS / _resolve_cross_file_imports / the fact dataclasses) are unchanged. Import direction is strictly extract.py -> resolution -> {models, base}; AST analysis confirmed no moved non-entry symbol is referenced from outside its new module. extract.py 12,632 -> 10,270 LOC (17,054 at the start of this branch, -40%). Full suite unchanged: 3036 passed, 29 skipped; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…tch_install_cli main() carried a 256-line if/elif chain routing every install-family command (install, uninstall, and the 22 per-platform targets: claude, codebuddy, gemini, cursor, vscode, copilot, kilo, kiro, devin, pi, amp, agents/skills, aider/codex/opencode/claw/droid/trae/trae-cn/hermes, antigravity). That block only ever reads sys.argv and calls functions that already live in install.py, so move it there verbatim as a guarded dispatch_install_cli(cmd) -> bool; main() now does `if dispatch_install_cli(cmd): return`. The command set is derived from the block's own conditions (so none is missed), and the branch's early `return` becomes `return True` so the help path still short-circuits. Verified end-to-end: `graphify claude install/uninstall`, unknown command, and the install test suite all behave identically. __main__.py 3,641 -> 3,388 LOC (5,368 at branch start, -37%). Full suite unchanged: 3036 passed, 29 skipped; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

setuptools uses an explicit package list, so the new graphify/exporters/ package (base, html, graphdb — split out of export.py) was absent from the built wheel, which would break `import graphify.exporters.*` for installed users. Add it alongside graphify and graphify.extractors. No version change. Verified: fresh-venv install of the rebuilt wheel imports every new module, resolves all backward-compat re-exports, builds a multi-language graph, and runs query/path/explain and install/uninstall. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…to extractors/engine.py Moves the config-driven extraction engine — _extract_generic (~2.2k LOC) and its 67-function closure of per-language tree-sitter walkers/type-collectors (_js_*, _ts_*, _python_*, _java_*, _csharp_*, _cpp_*, _swift_*, _kotlin_*, _php_*, _ruby_*, _scala_*) plus 10 engine-private constants (language builtin/type tables) — into graphify/extractors/engine.py, verbatim. AST analysis proved the engine closure references neither `extract` nor `_DISPATCH`, so the dispatcher facade (extract, _DISPATCH, _get_extractor, collect_files, the thin config-driven extractor wrappers) stays in extract.py and imports the engine. Import direction: extract.py -> engine -> {resolution, models, base}; no shared constant crosses the boundary (verified), and engine pulls a single name (_resolve_js_import_target) from resolution, so there is no cycle. extract.py re-exports every moved symbol, so importers and tests reaching into engine internals are unchanged. extract.py 10,270 -> 5,947 LOC (17,054 at branch start, -65%). Full suite unchanged: 3036 passed, 29 skipped; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…actors/ Now that the resolution passes and the tree-sitter engine live in their own modules, these three bespoke extractors are self-contained and move cleanly (verbatim), pulling only the specific engine/resolution/base helpers they need: - extractors/pascal.py: extract_pascal + regex helpers + _PAS_* constants - extractors/objc.py: extract_objc + Objective-C member-call resolution - extractors/julia.py: extract_julia Re-exported from extract.py and registered in extractors/__init__.py (27 langs). The remaining in-file extractors (js/ts config family + vue/svelte/astro/xaml) stay because they share _JS_CONFIG/_TS_CONFIG and the xaml dispatch caches with extract_js/extract_csharp and the dispatcher. extract.py 5,947 -> 4,740 LOC (17,054 at branch start, -72%). Full suite unchanged: 3036 passed, 29 skipped; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

main() was a 2,760-line function that was almost entirely a 2,538-line if/elif chain dispatching every non-install subcommand (query, path, explain, diagnose, affected, reflect, save-result, extract, update, cluster-only, label, build, global, merge-graphs, merge-driver, watch, tree, export, benchmark, clone, prs, provider, hook*, check-update, and the `graphify <path>` redirect). Move that chain, plus its 5 chain-only helpers (_StageTimer, _clone_repo, _default_graph_path, _enforce_graph_size_cap_or_exit, _run_hook_guard) and 4 chain-only constants (_SEARCH_NUDGE, _READ_NUDGE, _HOOK_SOURCE_EXTS, _GEMINI_NUDGE_TEXT), into graphify/cli.py as dispatch_command(cmd). main() now does its setup (encoding, stale-skill check, version/help) then `if dispatch_install_cli(cmd): return` else `dispatch_command(cmd)`. The chain ends in its own unknown-command exit, so it's called as a statement — no return-value plumbing. The one cli->__main__ coupling, the path-redirect's recursive main() call, is handled by a lazy import (_reenter_main), so import direction stays __main__ -> cli with no cycle. __main__ re-exports every moved symbol, so importers/tests are unchanged. __main__.py 3,388 -> 662 LOC (5,368 at branch start, -88%). Verified end-to-end: version/help/unknown-command, the `graphify <path>` redirect (rebuilds a graph), and query/path/explain. Full suite unchanged: 3036 passed, 29 skipped; skillgen OK. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

safishamsi · 2026-07-08T21:15:20Z

Merged into v8 (fast-forward, all 10 commits + authorship preserved) — thanks @TPAteeq, excellent decomposition.

Reviewed as a pure verbatim move: confirmed it's based on current HEAD (includes the recent #1726/#1734/#1729/#1731 fixes — verified each survives in the moved code), every original import path still resolves via the re-export shims, and the full suite passes 3091/3 identical to the pre-merge baseline on a grammar-rich environment (your 3036/29 was just fewer optional grammars installed → more skips). extract.py 17,054→4,740, main.py 5,368→673, export.py 1,671→962.

Heads-up to other contributors: this reshapes extract.py/main.py/export.py, so open PRs touching them (#1697, #1691, #1720, #1717) will need a rebase onto the new module layout.

Decompose extract.py / __main__.py / export.py into focused modules (#1737, verbatim, no behavior change). Plus fixes since 0.9.10: merge-graphs distinct repo tags (#1729), uninstall cleans Claude .local files (#1731), and extract --code-only for keyless code-only indexing of a mixed repo (#1734). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

@ivanzhl

…1700 Kotlin half, #1738) Kotlin enum entries weren't extracted: the walker never descended into the enum body (`enum_class_body` wasn't in _KOTLIN_CONFIG.body_fallback_child_types, so _find_body returned None). Add `enum_class_body` to the fallback body types and a `_kotlin_extra_walk` (dispatched for tree_sitter_kotlin, mirroring the Java/Swift handling) that emits each enum_entry as a node with a `case_of` edge to the enum. Re-applied from PR #1738 (@ivanzhl) onto the post-#1737 module layout: the walk engine now lives in graphify/extractors/engine.py while _KOTLIN_CONFIG stays in extract.py. Closes the Kotlin half of #1700 (Java was #1719). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

… calls across files (#1739) Both Pascal extractors resolved every call via a single file-wide {method_name: node_id} dict, so two unrelated classes declaring a same-named method (property accessors, generated COM/TLB wrapper classes) collapsed onto whichever declaration was inserted last, producing wrong cross-class `calls` edges. Resolution is now scoped: own class -> ancestor chain (inherits) -> file-level free function -> unambiguous file-wide match; ambiguous at every level emits no edge rather than guessing (same god-node guard as the Ruby resolver). Adds graphify/pascal_resolution.py, a corpus-wide post-extraction resolver (registered via resolver_registry) that walks the inherits chain across file boundaries, so a call from a manual descendant to a method it inherits from a base class in a separate unit (the generated-base/manual-descendant split) resolves. Also stops both extractors from emitting a duplicate base-class stub carrying the referencing file's source_file, which collided with the real node under cross-file id disambiguation. cache.py gives the new raw_calls bucket the same portable-path treatment as nodes/edges so it round-trips. Re-applied to the post-#1737 module layout (extractor hunks land in graphify/extractors/pascal.py; registration stays in extract.py). Added one adaptation the original PR predated: the cross-file resolver's god-node guard now counts DISTINCT method nids, because the tree-sitter extractor emits a method edge for both the interface declaration and the implementation, so the same method_nid arrives twice -- without deduping, every inherited call looked ambiguous and resolved to nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

@Cekaru

…ge split (#1721) The extract_terraform move #1721 proposed already landed on v8 via the #1737 decomposition (extractors/terraform.py exists, extract.py re-exports it, and extractors/LANGUAGE_EXTRACTORS registers it), so the code move is a no-op now. But the regression test @Cekaru added with it had no equivalent on v8. Salvage and generalize it: sweep every LANGUAGE_EXTRACTORS entry and assert graphify. extract re-exports the SAME object (facade identity) and the registry maps to it (registry identity), plus the concrete terraform anchor from the PR. This institutionalizes the re-export-identity guarantee the split relies on, so a future move that forgets a facade re-export fails loudly. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

TPAteeq and others added 10 commits July 9, 2026 00:20

safishamsi merged commit 247b2ae into Graphify-Labs:v8 Jul 8, 2026
4 checks passed

TPAteeq deleted the refactor/split-large-modules branch July 8, 2026 21:16

safishamsi mentioned this pull request Jul 8, 2026

fix(extract): extract Kotlin enum entries as nodes with case_of edges #1738

Closed

safishamsi mentioned this pull request Jul 8, 2026

fix(extract): scope Pascal/Delphi call resolution + resolve inherited calls across files #1739

Closed

3 tasks

safishamsi mentioned this pull request Jul 9, 2026

refactor(extract): move extract_terraform to extractors/terraform.py (verbatim) #1721

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

refactor: decompose extract.py and main.py into focused modules#1737

refactor: decompose extract.py and main.py into focused modules#1737
safishamsi merged 10 commits into
Graphify-Labs:v8from
TPAteeq:refactor/split-large-modules

TPAteeq commented Jul 8, 2026

Uh oh!

safishamsi commented Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

TPAteeq commented Jul 8, 2026

Why

Size impact

What moved (one subsystem per commit)

Why it's safe

Left as-is (intentionally)

Uh oh!

safishamsi commented Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants