PyLegend 2.0#307
Draft
deepyaman wants to merge 72 commits into
Draft
Conversation
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Decomposes Phase 1 (Fix Pure Foundation) into four parallel-optimized plans honoring all CONTEXT.md decisions (D-01 through D-08) and covering requirements PURE-01..05, TEST-01, TEST-02. Wave 1 (01 + 02) runs in parallel: Java test-server Execute registration + Maven rebuild alongside Python to_pure() bodies for service/function input frames. Wave 2 (03) adds LegendClient Pure execution + schema endpoints. Wave 3 (04) switches LegendQL frames to Pure end-to-end while preserving the SQL path for Legacy/Pandas frames per D-07. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Add import for org.finos.legend.engine.query.pure.api.Execute - Register new Execute(modelManager, planExecutor, routerExtensions, generatorExtensions.flatCollect(...)) after GrammarToJson registration - Exposes POST /api/pure/v1/execution/execute and /api/pure/v1/execution/generatePlan endpoints in test server
…input frames - Add tests/extensions/tds/abstract/__init__.py - Add TestLegendServiceInputFramePure with unit tests for pattern -> Pure path mapping - Add TestLegendFunctionInputFramePure with unit test for function path -> Pure string - Both unit tests currently fail (RED) since to_pure() raises RuntimeError - Integration tests with legend_test_server fixture skipped when JAVA_HOME absent
…nd LegendFunctionInputFrameAbstract - Service frame: pattern '/simplePersonService' -> '|pylegend::test::SimplePersonService.all()' Maps HTTP pattern to Pure path by stripping leading '/', capitalizing first letter, prepending 'pylegend::test::' package, appending '.all()' call form - Function frame: path 'pylegend::test::function::Func' -> '|pylegend::test::function::Func()' Wraps path in lambda prefix '|' and appends '()' to call the function - Replaces RuntimeError raises with concrete Pure root string generation (PURE-01, PURE-02) - Existing to_pure() on CsvInputFrameAbstract and TableSpecInputFrameAbstract untouched (PCT guard)
…ild JAR - Task 1: added Execute import and jersey registration to PyLegendSqlServer.java (commit 1600c9b) - Task 2: installed Maven via pixi, built pylegend-sql-server-1.0-shaded.jar (795 MB, 90s build) - Boot smoke check passed: /api/pure/v1/execution/execute returns 500 (non-404), /api/server/v1/info returns 200
…execute_input to LegendClient
…g_schema (xfail pending Pure expression resolution)
…RE-05) - Add depot_server_host/depot_server_port optional params to LegendClient - Add _build_depot_execute_input to fetch service/function metadata from depot and construct valid ExecuteInput (func+mapping+runtime for services, body lambda for functions) - Add _pure_to_sql_fallback for backward-compat when depot not configured - Modify get_pure_string_schema: tries Pure/generatePlan, falls back to depot approach, falls back to SQL schema (for backward compat) - Modify execute_pure_string: same cascade pattern - Switch LegendQLApiLegendServiceInputFrame init to get_pure_string_schema - Switch LegendQLApiLegendFunctionInputFrame init to get_pure_string_schema - Add execute_frame override on LegendQLApiBaseTdsFrame (routes Pure) - Add _get_legendql_input_project_coordinates helper - Add get_project_coordinates() getters on abstract service/function frames - Expose metadata_port in conftest.py legend_test_server fixture Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… (TEST-02) - Add test_legendql_api_legend_person_service_frame_pure_gen (asserts to_pure) - Add test_legendql_api_legend_person/trade/product_service_frame_pure_execution - Add monkeypatch test confirming execute_pure_string called (not execute_sql_string) - Parallel additions for function frame: pure_gen, pure_execution, monkeypatch test - Fix _tds_columns_from_plan_result_type: type-annotate tds_cols for mypy - Fix _build_depot_execute_input: annotate execution dict for mypy strict - Add TYPE_CHECKING guard for ProjectCoordinates in legendql_api_base_tds_frame Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ion path Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Create pylegend/utils/grammar_method.py with Apache 2.0 header and decorator - Update all 10 shared primitive files to import grammar_method from pylegend.utils.grammar_method - Preserves exact identity-decorator behavior (sets _is_grammar_method=True attribute)
…ory trees - Remove production trees: core/tds/legacy_api, core/tds/pandas_api, core/language/legacy_api, core/language/pandas_api, core/sql, core/database, extensions/tds/legacy_api, extensions/tds/pandas_api, extensions/database, samples/pandas_api - Remove orphaned files: legacy_api_tds_client.py, sql_query_helpers.py, extensions/tds/result_handler/to_pandas_df_result_handler.py - Remove test mirrors: tests/core/tds/legacy_api, tests/core/tds/pandas_api, tests/core/language/legacy_api, tests/core/database, tests/extensions/database, tests/extensions/tds/frames/legacy_api, tests/extensions/tds/frames/pandas_api, tests/samples/pandas_api, tests/test_legacy_api_tds_client.py - Retain: samples/legendql_api, samples/local_legend_env.py, core/tds/result_handler
…/pandas/sql trees deleted - Task 1: pylegend/utils/grammar_method.py created; 10 primitive files updated - Task 2: 10 production trees, 8 test trees, 3 files deleted via git rm - REMV-01, REMV-02, REMV-03 directory-level requirements satisfied
… and project_cooridnates - Remove core.sql.metamodel imports from all 10 shared primitive files - Remove FrameToSqlConfig imports from all primitive files - Remove abstract to_sql_expression from PyLegendPrimitive base class - Remove concrete to_sql_expression from all 9 primitive subclasses and precise_primitives - Remove sql_params() abstract+concrete methods from ProjectCoordinates and all subclasses - Remove core.sql.metamodel import from project_cooridnates.py - Remove import abc (abstractmethod gone); keep ABCMeta - Clean up now-unused PyLegendDict imports in all affected files
…ons, and legendql language files - Remove core.sql.metamodel and metamodel_extension imports from all 13 operation files - Remove to_sql_func parameter from binary/unary/nary/nullary base expression classes - Remove __to_sql_func static methods and SQL func args from all concrete operation classes - Remove abstract to_sql_expression from PyLegendExpression base class - Remove to_sql_expression from all shared expression files (column, literal, variable, tds_row) - Remove SQL methods from pylegend_custom_expressions (PyLegendSortInfo, PyLegendWindow, etc.) - Remove SQL methods from legendql_api_custom_expressions and legendql_api_tds_row - Remove column_sql_expression from AbstractTdsRow (SQL path no longer needed) - All to_pure_expression and __to_pure_func Pure methods preserved unchanged - flake8 clean on all 21 edited files
- Remove FrameToSqlConfig class from tds_frame.py; keep FrameToPureConfig - Remove execute_frame, execute_frame_to_string, to_sql_query abstracts from PyLegendTdsFrame - Remove to_sql_query_object, execute_frame* from BaseTdsFrame; keep to_pure_query - Remove to_sql_query_object and to_sql from AppliedFunctionTdsFrame/AppliedFunction - Remove get_sql_string_schema and execute_sql_string from LegendClient - Clear ToPandasDfResultHandler/PandasDfReadConfig exports from extensions result_handler
… function files - Remove to_sql_query_object and core.sql.metamodel imports from 4 abstract extension bases - Replace QualifiedName with plain list in TableSpecInputFrameAbstract (no sql.metamodel needed) - Remove to_sql method and all SQL/sql_query_helpers imports from all 17 LegendQL function files - Pure paths (to_pure) preserved on all function files; flake8 clean
…egend - Remove LegacyApiTdsClient, legacy_api_tds_client from pylegend/__init__.py - Remove agg, olap_rank, olap_agg from pylegend/__init__.py language imports - Remove pandas_api from pylegend/samples/__init__.py - Remove all legacy_api/olap/LegacyApi imports from pylegend/core/language/__init__.py - python -c "import pylegend" succeeds with no ImportError
… pylegend green - FrameToSqlConfig, execute_frame*, to_sql_query*, to_pandas* removed from frame hierarchy - LegendClient SQL methods removed; Pure-only HTTP methods retained - All 17 LegendQL function files SQL-free with to_pure preserved - Public __init__.py files cleaned of legacy/OLAP/pandas exports - python -c "import pylegend" succeeds with no ImportError
…lient tests, and tds_row test - Remove FrameToSqlConfig imports and to_sql_query assertions from 17 function test files - Remove SQL test methods from test_legend_client.py; add execute_pure_string coverage - Remove SQL test methods from test_legend_client_e2e.py; retain Pure/parse/compile tests - Remove SQL fixtures and methods from test_legendql_api_tds_row.py and AbstractTestTdsRow - Fix tests/core/language/shared/__init__.py to remove deleted imports (FrameToSqlConfig, R, PandasDfReadConfig) - Fix tests/test_helpers/test_legend_service_frames.py to remove deleted legacy/pandas API imports - pytest collection passes with 258 tests collected and 0 errors
…fix full suite collection - Remove SQL class vars, imports, and assertions from 10 shared primitive test files - Clean TestPreciseIntegerTypes, TestPreciseFloatTypes, and remove TestPrecisePrimitiveDirectInstantiation (SQL-only class) - Clean test_tds_row.py AbstractTestTdsRow base class to Pure-only assertions - Fix tests/__init__.py shared module to remove deleted imports (FrameToSqlConfig, R, PandasDfReadConfig) - Fix 12 additional test files across test_literal_expressions, result_handler tests, extension tests, and tds tests that had pre-existing import failures from prior-wave deletions - Full pytest collection: 678 tests collected with 0 errors (up from 598 before these fixes) - flake8 passes on tests/core/language/shared/ and all other modified test files
…ed test files Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…-dependent tests - Remove lazy imports of deleted pandas_api.pandas_api_series and pandas_api.pandas_api_groupby_series from unary_expression.py and binary_expression.py; the Series/GroupbySeries isinstance branch was pandas-only and is no longer reachable - Add missing frame_to_pure_config to TestPyLegendDecimalUnit - Rename TestPandasResultHandlerPreciseTypes and TestCastToPreciseTypesPandasResultHandler with leading underscore so pytest skips them (they test the deleted ToPandasDfResultHandler) - Remove test_table_spec_frame_execution_error which called the deleted execute_frame_to_string method
- Remove pandas, numpy, testcontainers from runtime dependencies - Remove pandas-stubs, mockito, sqlalchemy, pg8000, pymysql, cryptography, wrapt from dev deps - Add testcontainers>=3.0.0 to dev dependency group
…ained files - Re-lock uv.lock against trimmed pyproject.toml (requests + ijson runtime only) - Replace pandas CSV parsing in csv_tds_frame.py with stdlib csv/datetime - Update test_csv_tds_frame.py to expect ValueError instead of pd.errors.EmptyDataError - Replace mockito with unittest.mock in test_auth.py - Remove pandas import from test_legendql_api_tds_client.py; rewrite as pure query test
All 5 plans executed and verified. Legacy API, Pandas API, SQL metamodel layer removed. Runtime deps trimmed to requests+ijson. 63 unit tests pass.
- 01-VERIFICATION.md updated to human_needed / 4/4 (from prior session) - 02-PATTERNS.md generated during phase 02 planning
One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via: Supported
Alternatively, if the co-author should not be included, remove the Please update your commit message(s) by doing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.