Fix sentencepiece strict hdrs_check failure when building //c:litert-lm on macOS - #3262
Open
yudaprama wants to merge 1 commit into
Open
Fix sentencepiece strict hdrs_check failure when building //c:litert-lm on macOS#3262yudaprama wants to merge 1 commit into
yudaprama wants to merge 1 commit into
Conversation
Upstream's sentencepiece v0.2.2 layout fails strict hdrs_check on macOS (generated sentencepiece_model.pb.h counted as undeclared inclusion). Adopt the cognee-dev fork's strip-to-src layout on v0.2.2 content: - WORKSPACE: strip_prefix sentencepiece-0.2.2/src + absl/protobuf-lite seds - PATCH.sentencepiece_darts: re-add vendored darts_clone/darts.h (newer DoubleArrayImpl API: copy_array/validate) lost to the strip - BUILD.sentencepiece: root-file layout, darts.h in hdrs, full absl deps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Building the C API shared library on macOS (Apple Silicon) fails during compilation of the sentencepiece dependency:
With the current
http_archivestanza (v0.2.2, no strip beyond the default), the generatedsentencepiece_model.pb.hlands in the package root of the external repo while the processor sources live undersrc/, so the quoted include resolves through theincludes = ["src"]path instead of through the declaringcc_proto_librarylabel — and strict hdrs_check rejects it. The same applies to//support/tokenizer:sentencepiece_tokenizer.Fix
Adopt a strip-to-src layout for sentencepiece v0.2.2:
WORKSPACE:strip_prefix = "sentencepiece-0.2.2/src"so sources and the generated.pb.hshare the package root (quoted includes, no cross-dir resolution), plus sed ofthird_party/absl/andthird_party/protobuf-lite/include prefixes to the real absl/protobuf.PATCH.sentencepiece_darts: re-adds the vendoredthird_party/darts_clone/darts.hthat the strip drops — v0.2.2 sources require its newerDoubleArrayImplAPI (copy_array,validate), which the older separately-pinned darts_clone does not have.BUILD.sentencepiece: root-file layout,third_party/darts_clone/darts.hinhdrs, and the full absl dep set v0.2.2 sources actually include.Verification
macOS 15, Apple Silicon (darwin_arm64), Bazel 7.6.1 via bazelisk. Linking the resulting
liblitert-lm.dylibfrom a Rust FFI consumer and running a Gemma 4 E2B (.litertlm) streaming session works end to end.