Skip to content

Fix sentencepiece strict hdrs_check failure when building //c:litert-lm on macOS - #3262

Open
yudaprama wants to merge 1 commit into
google-ai-edge:mainfrom
yudaprama:fix/macos-sentencepiece-hdrs-check
Open

Fix sentencepiece strict hdrs_check failure when building //c:litert-lm on macOS#3262
yudaprama wants to merge 1 commit into
google-ai-edge:mainfrom
yudaprama:fix/macos-sentencepiece-hdrs-check

Conversation

@yudaprama

Copy link
Copy Markdown

Problem

Building the C API shared library on macOS (Apple Silicon) fails during compilation of the sentencepiece dependency:

bazel build //c:litert-lm --config=macos_arm64
ERROR: external/sentencepiece/BUILD.bazel:23:11: Compiling src/word_model.cc failed: undeclared inclusion(s) in rule '@@sentencepiece//:sentencepiece_processor':
this rule is missing dependency declarations for the following files included by 'src/word_model.cc':
  'bazel-out/darwin_arm64-opt/bin/external/sentencepiece/sentencepiece_model.pb.h'

With the current http_archive stanza (v0.2.2, no strip beyond the default), the generated sentencepiece_model.pb.h lands in the package root of the external repo while the processor sources live under src/, so the quoted include resolves through the includes = ["src"] path instead of through the declaring cc_proto_library label — and strict hdrs_check rejects it. The same applies to //support/tokenizer:sentencepiece_tokenizer.

Fix

Adopt a strip-to-src layout for sentencepiece v0.2.2:

  • WORKSPACE: strip_prefix = "sentencepiece-0.2.2/src" so sources and the generated .pb.h share the package root (quoted includes, no cross-dir resolution), plus sed of third_party/absl/ and third_party/protobuf-lite/ include prefixes to the real absl/protobuf.
  • PATCH.sentencepiece_darts: re-adds the vendored third_party/darts_clone/darts.h that the strip drops — v0.2.2 sources require its newer DoubleArrayImpl API (copy_array, validate), which the older separately-pinned darts_clone does not have.
  • BUILD.sentencepiece: root-file layout, third_party/darts_clone/darts.h in hdrs, and the full absl dep set v0.2.2 sources actually include.

Verification

bazel build //c:litert-lm --config=macos_arm64 --jobs=6
INFO: Build completed successfully

macOS 15, Apple Silicon (darwin_arm64), Bazel 7.6.1 via bazelisk. Linking the resulting liblitert-lm.dylib from a Rust FFI consumer and running a Gemma 4 E2B (.litertlm) streaming session works end to end.

Upstream's sentencepiece v0.2.2 layout fails strict hdrs_check on macOS
(generated sentencepiece_model.pb.h counted as undeclared inclusion).
Adopt the cognee-dev fork's strip-to-src layout on v0.2.2 content:
- WORKSPACE: strip_prefix sentencepiece-0.2.2/src + absl/protobuf-lite seds
- PATCH.sentencepiece_darts: re-add vendored darts_clone/darts.h (newer
  DoubleArrayImpl API: copy_array/validate) lost to the strip
- BUILD.sentencepiece: root-file layout, darts.h in hdrs, full absl deps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant