Skip to content

feat(python-notebook-migration): add notebook-migration-service microservice in backend#5258

Open
zyratlo wants to merge 2 commits into
apache:mainfrom
zyratlo:migration-tool-backend-notebook-migration-service
Open

feat(python-notebook-migration): add notebook-migration-service microservice in backend#5258
zyratlo wants to merge 2 commits into
apache:mainfrom
zyratlo:migration-tool-backend-notebook-migration-service

Conversation

@zyratlo
Copy link
Copy Markdown
Contributor

@zyratlo zyratlo commented May 28, 2026

What changes were proposed in this PR?

Introduces the microservice that mediates between Texera and the JupyterLab docker stack landed in migration-tool-jupyter-docker. Adds a new SBT subproject notebook-migration-service plus shared config and a frontend dev-proxy route.

New SBT subproject notebook-migration-service/:

  • build.sbt and project/build.properties — module SBT setup; module depends on the existing Auth, Config, and DAO projects
  • src/main/scala/.../NotebookMigrationService.scala — Dropwizard Application entry point; sets Jersey URL pattern to /api/*, registers the resource class, initializes the shared SQL connection via
    SqlServer.initConnection(StorageConfig.jdbcUrl, …), and wires in RequestLoggingFilter.
  • src/main/scala/.../NotebookMigrationServiceConfiguration.scala — Dropwizard Configuration subclass.
  • src/main/scala/.../resource/NotebookMigrationResource.scala — five REST endpoints under /notebook-migration:
    • GET /get-jupyter-url — health-checks the Jupyter container and returns its base URL.
    • GET /get-jupyter-iframe-url — returns the iframe-ready URL for notebook.ipynb.
    • POST /set-notebook — receives a notebook JSON, PUTs it into JupyterLab via its /api/contents/work/{name} API.
    • POST /store-notebook-and-mapping — persists a notebook + workflow-notebook mapping into Postgres in a single transaction (writes to the notebook and workflow_notebook_mapping tables added by migration-tool-database-tables).
    • POST /fetch-notebook-and-mapping — returns the most recent notebook + mapping for a given (wid, vid).
  • src/main/resources/logback.xml — logging config.
  • src/main/resources/notebook-migration-service-web-config.yaml — Dropwizard server config (HTTP port 9098, DB connection refs).

Root build wiring:

  • build.sbt — declares the new NotebookMigrationService SBT subproject and adds it to the TexeraProject aggregation.

Shared config:

  • common/config/src/main/resources/storage.conf — new jupyter { url = "http://localhost:9100" } block, overridable via STORAGE_JUPYTER_URL.
  • common/config/src/main/scala/.../StorageConfig.scala — adds the jupyterURL accessor.

Frontend dev proxy:

  • frontend/proxy.config.json — routes /api/notebook-migration/* to http://localhost:9098.

Any related issues, documentation, discussions?

Closes #5257
Parent issue #4301

  • Hard dependency: must be merged after migration-tool-database-tables feat(python-notebook-migration): add database tables for Notebook Migration tool #5055 — the resource imports jOOQ-generated Notebook / WorkflowNotebookMapping classes that only exist once the schema PR is merged.
  • Soft dependency: the JupyterLab container from migration-tool-jupyter-docker is what StorageConfig.jupyterURL points to. Without it running, the Jupyter-related endpoints return a 500 with "Cannot connect to Jupyter server". Service still starts and the DB-persistence endpoints work in isolation.

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.7)

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file frontend Changes related to the frontend GUI common labels May 28, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 28, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.95%. Comparing base (d8c254c) to head (baea89a).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5258      +/-   ##
============================================
- Coverage     48.95%   48.95%   -0.01%     
  Complexity     2377     2377              
============================================
  Files          1048     1048              
  Lines         40270    40270              
  Branches       4272     4272              
============================================
- Hits          19714    19713       -1     
  Misses        19402    19402              
- Partials       1154     1155       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 39.53% <ø> (ø)
agent-service 33.76% <ø> (ø) Carriedforward from 0c3af5e
amber 51.57% <ø> (ø) Carriedforward from 0c3af5e
computing-unit-managing-service 0.00% <ø> (ø)
config-service 0.00% <ø> (ø)
file-service 37.99% <ø> (ø)
frontend 40.64% <ø> (ø)
python 90.79% <ø> (ø) Carriedforward from 0c3af5e
workflow-compiling-service 56.81% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@zyratlo zyratlo marked this pull request as ready for review May 28, 2026 04:34
@Yicong-Huang Yicong-Huang changed the title feat(python-notebook-migration, backend): add notebook-migration-service microservice in backend feat(python-notebook-migration): add notebook-migration-service microservice in backend May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Notebook Migration] Add notebook-migration-service for Texera and Jupyter communication and Database communication

2 participants