Refactor to multi-module Java 11 cloud storage SDK with official CSP SDKs - #65
Merged
Merged
Conversation
…tations Replace the single Scala/jclouds-based module with a multi-module pure Java 11 Maven project. Each CSP (AWS, Azure, GCP, OCI) gets its own module using the official cloud SDK, with OIDC-based auth as the primary approach and Access Key as a configurable fallback. Modules: - cloud-storage-sdk-api: Interface, models, factory (ServiceLoader), abstract base - cloud-storage-sdk-aws: AWS S3 + Ceph S3 support (AWS SDK v2) - cloud-storage-sdk-azure: Azure Blob Storage (azure-storage-blob + azure-identity) - cloud-storage-sdk-gcp: Google Cloud Storage (google-cloud-storage) - cloud-storage-sdk-oci: OCI Object Storage (oci-java-sdk-objectstorage) Key changes: - Drop jclouds, Scala, and Joda-Time dependencies entirely - Pure Java 11 with java.time for date operations - ServiceLoader-based CSP discovery via StorageServiceProvider SPI - StorageConfig uses Builder pattern with AuthType and StorageType enums - AbstractStorageService provides shared orchestration (retry, search, copy, etc.) - IStorageService extends AutoCloseable for try-with-resources support https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
- API module: unit tests for DateRangeUtil, FileUtil, and AbstractStorageService (in-memory implementation, no credentials) - CSP modules: integration tests extending BaseStorageServiceIntegrationTest that auto-skip via JUnit 5 assumeTrue() when env vars are absent - API POM: publish test-jar so CSP modules can extend the base test class - CSP POMs: depend on api test-jar for shared test infrastructure - TESTING.md: comprehensive guide covering env vars, auth types, and examples for running tests from cloud-permissioned VMs https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
Captures module layout, key source files, StorageConfig/AuthType enums, the 12 AbstractStorageService primitives, per-CSP auth patterns, service discovery via ServiceLoader, test strategy, build commands, package structure, and common modification patterns. https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
CircleCI is no longer used. Removes .circleci/config.yml and the corresponding reference in PLAN.md Phase 6. https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
- Add managed dependency for cloud-storage-sdk-api test-jar classifier in parent POM so CSP modules resolve the version correctly - Pin maven-jar-plugin to 3.3.0 in pluginManagement - Add GitHub Actions CI workflow (ci.yml) for build + test on push/PR - Add GitHub Actions SonarCloud workflow (sonar.yml) for static analysis https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
Compute a relativePath for files under a directory and join it to objectKey with an explicit '/' separator before calling upload. This prevents malformed object keys when objectKey doesn't end with a separator and preserves the directory structure during recursive (sync and async) uploads. The change extracts the relative path calculation and applies it in both upload loops.
Problem: Used .toList() method which is only available in Java 16+, but project targets Java 11. Solution: Replaced .toList() with .collect(Collectors.toList()) and added Collectors import.
…ent-encodes the blob name
…ploads and avoid 5 null arguments code
fix: Decoding blob url since azure sdk percent-encodes the blob name
fix: Update test cases to cover the optional slash in objectKey
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete refactor of the Sunbird Cloud Storage SDK from a single Scala/jclouds-based module to a multi-module pure Java 11 Maven project using official cloud provider SDKs. This is a breaking change released as version 2.0.0.
Key Changes
Architecture: Converted from monolithic Scala codebase to modular Maven structure with separate implementations for each cloud provider
cloud-storage-sdk-api: Core interfaces, models, factory, and shared abstractionscloud-storage-sdk-aws: AWS S3 and Ceph S3 (via endpoint override)cloud-storage-sdk-azure: Azure Blob Storagecloud-storage-sdk-gcp: Google Cloud Storagecloud-storage-sdk-oci: Oracle Cloud Infrastructure Object StorageLanguage & Runtime: Upgraded from Scala to pure Java 11, replacing all Scala source files with Java equivalents
Dependencies: Replaced jclouds with official cloud SDKs:
Core Implementation:
AbstractStorageService: Shared orchestration logic for upload, download, signed URLs, delete, copy, search, and metadata operationsStorageServiceFactory: ServiceLoader-based factory for dynamic provider discoveryStorageConfig: Builder pattern configuration supporting multiple auth types (ACCESS_KEY, OIDC, IAM, IAM_ROLE, INSTANCE_PROFILE)BlobandDeleteTargetmodels replacing Scala case classesUtilities: Replaced Joda-Time with Java 11
java.timeAPI inDateRangeUtil; addedFileUtilfor file operations and zip extractionTesting:
Documentation: Added comprehensive PLAN.md and TESTING.md guides
CI/CD: Updated GitHub Actions and CircleCI configurations for Java 11 builds
Notable Implementation Details
https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6