Running on WSL2 Ubuntu 22.04 fails

[yhdanid]

First, thanks for this.

As litert still does not support native Windows, I tried to run it in WSL2 under Ubuntu 22.04.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

Nvidia and CUDA seem to be working OK. Also the GPU is set to be used in rendering.

glxinfo | grep "OpenGL renderer"
OpenGL renderer string: D3D12 (NVIDIA GeForce RTX 4080)

vulkaninfo | grep "deviceName"
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
        deviceName        = Microsoft Direct3D12 (NVIDIA GeForce RTX 4080 GPU)
        deviceName        = llvmpipe (LLVM 15.0.7, 256 bits)
        deviceName        = Microsoft Direct3D12 (Intel(R) Iris(R) Xe Graphics)

nvidia-smi works as expected and I can see the GPU.

I can run this command fine:

litert-lm run  \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm \
   --prompt="What is the capital of France?"

And I get a quick response:
The capital of France is **Paris**.

To get parlor up and running, I ran:

1. uv sync
   in parlor's src directory, and I get the .venv setup OK.
2. uv run python server.py
   causes an error. I'm including the output below.

It looks to me that the CPU engine backend (software rendering) is being loaded, instead of the GPU's.

nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/src$ uv run python server.py
Downloading litert-community/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm (first run only)...
INFO:     Started server process [737]
INFO:     Waiting for application startup.
/mnt/e/sandbox/parlor/src/server.py:66: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
  await asyncio.get_event_loop().run_in_executor(None, load_models)
Loading Gemma 4 E2B from /home/nemisis/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm/snapshots/616f4124e6ff216292f16e7f73ff33b5ba9a4dd4/gemma-4-E2B-it.litertlm...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1775483829.659911     758 litert_lm_loader.cc:126] LitertLmLoader::Initialize
I0000 00:00:1775483829.661337     758 litert_lm_loader.cc:149] mmap_status is ok
I0000 00:00:1775483829.661497     758 litert_lm_loader.cc:155] status: OK
I0000 00:00:1775483829.661511     758 litert_lm_loader.cc:156] major_version: 1
I0000 00:00:1775483829.661529     758 litert_lm_loader.cc:157] minor_version: 5
I0000 00:00:1775483829.661541     758 litert_lm_loader.cc:158] patch_version: 0
I0000 00:00:1775483829.661556     758 litert_lm_loader.cc:209] section_index: 0
I0000 00:00:1775483829.661568     758 litert_lm_loader.cc:210] section_data_type: LlmMetadataProto
I0000 00:00:1775483829.661579     758 litert_lm_loader.cc:212] section_begin_offset: 16384
I0000 00:00:1775483829.661582     758 litert_lm_loader.cc:213] section_end_offset: 28576
I0000 00:00:1775483829.661584     758 litert_lm_loader.cc:209] section_index: 1
I0000 00:00:1775483829.661602     758 litert_lm_loader.cc:210] section_data_type: SP_Tokenizer
I0000 00:00:1775483829.661614     758 litert_lm_loader.cc:212] section_begin_offset: 32768
I0000 00:00:1775483829.661618     758 litert_lm_loader.cc:213] section_end_offset: 4721781
I0000 00:00:1775483829.661744     758 litert_lm_loader.cc:190] model_type: tf_lite_embedder
I0000 00:00:1775483829.661765     758 litert_lm_loader.cc:209] section_index: 2
I0000 00:00:1775483829.661768     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661770     758 litert_lm_loader.cc:212] section_begin_offset: 4734976
I0000 00:00:1775483829.661771     758 litert_lm_loader.cc:213] section_end_offset: 108546696
I0000 00:00:1775483829.661782     758 litert_lm_loader.cc:190] model_type: tf_lite_per_layer_embedder
I0000 00:00:1775483829.661785     758 litert_lm_loader.cc:209] section_index: 3
I0000 00:00:1775483829.661796     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661807     758 litert_lm_loader.cc:212] section_begin_offset: 108560384
I0000 00:00:1775483829.661810     758 litert_lm_loader.cc:213] section_end_offset: 1393078776
I0000 00:00:1775483829.661812     758 litert_lm_loader.cc:190] model_type: tf_lite_audio_encoder_hw
I0000 00:00:1775483829.661815     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483829.661818     758 litert_lm_loader.cc:209] section_index: 4
I0000 00:00:1775483829.661819     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661830     758 litert_lm_loader.cc:212] section_begin_offset: 1393082368
I0000 00:00:1775483829.661832     758 litert_lm_loader.cc:213] section_end_offset: 1487134648
I0000 00:00:1775483829.661834     758 litert_lm_loader.cc:190] model_type: tf_lite_audio_adapter
I0000 00:00:1775483829.661836     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483829.661837     758 litert_lm_loader.cc:209] section_index: 5
I0000 00:00:1775483829.661839     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661840     758 litert_lm_loader.cc:212] section_begin_offset: 1487142912
I0000 00:00:1775483829.661841     758 litert_lm_loader.cc:213] section_end_offset: 1496584156
I0000 00:00:1775483829.661852     758 litert_lm_loader.cc:190] model_type: tf_lite_end_of_audio
I0000 00:00:1775483829.661854     758 litert_lm_loader.cc:209] section_index: 6
I0000 00:00:1775483829.661856     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661857     758 litert_lm_loader.cc:212] section_begin_offset: 1496596480
I0000 00:00:1775483829.661868     758 litert_lm_loader.cc:213] section_end_offset: 1496603252
I0000 00:00:1775483829.661870     758 litert_lm_loader.cc:190] model_type: tf_lite_vision_encoder
I0000 00:00:1775483829.661872     758 litert_lm_loader.cc:209] section_index: 7
I0000 00:00:1775483829.661883     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661886     758 litert_lm_loader.cc:212] section_begin_offset: 1496612864
I0000 00:00:1775483829.661888     758 litert_lm_loader.cc:213] section_end_offset: 1715702984
I0000 00:00:1775483829.661890     758 litert_lm_loader.cc:190] model_type: tf_lite_vision_adapter
I0000 00:00:1775483829.661892     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483829.661903     758 litert_lm_loader.cc:209] section_index: 8
I0000 00:00:1775483829.661905     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661907     758 litert_lm_loader.cc:212] section_begin_offset: 1715716096
I0000 00:00:1775483829.661909     758 litert_lm_loader.cc:213] section_end_offset: 1720443208
I0000 00:00:1775483829.661921     758 litert_lm_loader.cc:190] model_type: tf_lite_end_of_vision
I0000 00:00:1775483829.661932     758 litert_lm_loader.cc:209] section_index: 9
I0000 00:00:1775483829.661969     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661971     758 litert_lm_loader.cc:212] section_begin_offset: 1720451072
I0000 00:00:1775483829.661973     758 litert_lm_loader.cc:213] section_end_offset: 1720457844
I0000 00:00:1775483829.661975     758 litert_lm_loader.cc:190] model_type: tf_lite_prefill_decode
I0000 00:00:1775483829.661986     758 litert_lm_loader.cc:209] section_index: 10
I0000 00:00:1775483829.661989     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.661990     758 litert_lm_loader.cc:212] section_begin_offset: 1720467456
I0000 00:00:1775483829.661992     758 litert_lm_loader.cc:213] section_end_offset: 2538742640
I0000 00:00:1775483829.661995     758 litert_lm_loader.cc:190] model_type: tf_lite_mtp_drafter
I0000 00:00:1775483829.661998     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483829.662009     758 litert_lm_loader.cc:209] section_index: 11
I0000 00:00:1775483829.662019     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483829.662023     758 litert_lm_loader.cc:212] section_begin_offset: 2538749952
I0000 00:00:1775483829.662027     758 litert_lm_loader.cc:213] section_end_offset: 2583082648
I0000 00:00:1775483829.662482     758 engine_impl.cc:306] New model files have LlmModelType, loading tokenizer asynchronously
W0000 00:00:1775483829.662555     758 litert_lm_loader.h:144] TFLite model type: TF_LITE_PREFILL_DECODE not found for backend constraints. Skipping.
W0000 00:00:1775483829.662843     758 litert_lm_loader.h:144] TFLite model type: TF_LITE_VISION_ENCODER not found for backend constraints. Skipping.
I0000 00:00:1775483829.662853     758 engine_settings.cc:97] The Main backend constraint is not set.
I0000 00:00:1775483829.662866     758 engine_settings.cc:97] The Vision backend constraint is not set.
I0000 00:00:1775483829.662880     758 engine_settings.cc:94] The Audio backend constraint is matched: CPU
I0000 00:00:1775483829.662892     758 engine_settings.cc:337] The validated engine settings: EngineSettings:
  MainExecutorSettings: backend: GPU
backend_config:
max_top_k: 1

max_tokens: 4096
activation_data_type: Not set
max_num_images: 0
lora_rank: 0
cache_dir:
cache_file: Not set
litert_dispatch_lib_dir: Not set
model_assets: model_path: /home/nemisis/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm/snapshots/616f4124e6ff216292f16e7f73ff33b5ba9a4dd4/gemma-4-E2B-it.litertlm
fake_weights_mode: FAKE_WEIGHTS_NONE

advanced_settings: prefill_batch_sizes: []
num_output_candidates: 1
configure_magic_numbers: true
verify_magic_numbers: false
clear_kv_cache_before_prefill: true
num_logits_to_print_after_decode: 0
gpu_madvise_original_shared_tensors: true
is_benchmark: false
preferred_device_substr:
num_threads_to_upload: -1
num_threads_to_compile: -1
convert_weights_on_gpu: true
wait_for_weights_conversion_complete_in_benchmark: true
optimize_shader_compilation: true
cache_compiled_shaders_only: false
share_constant_tensors: true
sampler_handles_input: true
allow_src_quantized_fc_conv_ops: true
hint_waiting_for_completion: false
gpu_context_low_priority: false
enable_speculative_decoding: false
disable_delegate_clustering: true

  LlmMetadata: goo.gle/debugproto
start_token {
  token_ids {
    ids: 2
  }
}
stop_tokens {
  token_ids {
    ids: 1
  }
}
stop_tokens {
  token_ids {
    ids: 50
  }
}
stop_tokens {
  token_ids {
    ids: 106
  }
}
sampler_params {
  type: TOP_P
  k: 1
  p: 0.95
  temperature: 1
  seed: 0
}
llm_model_type {
  gemma4 {
    start_of_image_token {
      token_str: "<|image>"
    }
    end_of_image_token {
      token_str: "<image|>"
    }
    start_of_audio_token {
      token_str: "<|audio>"
    }
    end_of_audio_token {
      token_str: "<audio|>"
    }
    code_fence_start: "<|tool_call>"
    code_fence_end: "<tool_call|>"
    open_quote: "<|\"|>"
    close_quote: "<|\"|>"
    function_response_start: "<|tool_response>"
    use_template_for_fc_format: true
    patch_width: 16
    patch_height: 16
    max_num_patches: 2520
  }
}
jinja_prompt_template: "{%- macro format_parameters(properties, required) -%}\n    {%- set standard_keys = [\'description\', \'type\', \'properties\', \'required\', \'nullable\'] -%}\n    {%- set ns = namespace(found_first=false) -%}\n    {%- for key, value in properties | dictsort -%}\n        {%- set add_comma = false -%}\n        {%- if key not in standard_keys -%}\n            {%- if ns.found_first %},{% endif -%}\n            {%- set ns.found_first = true -%}\n            {{ key }}:{\n            {%- if value[\'description\'] -%}\n                description:<|\"|>{{ value[\'description\'] }}<|\"|>\n                {%- set add_comma = true -%}\n            {%- endif -%}\n            {%- if value[\'nullable\'] %}\n                {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}\n                nullable:true\n            {%- endif -%}\n            {%- if value[\'type\'] | upper == \'STRING\' -%}\n                {%- if value[\'enum\'] -%}\n                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}\n                    enum:{{ format_argument(value[\'enum\']) }}\n                {%- endif -%}\n            {%- elif value[\'type\'] | upper == \'OBJECT\' -%}\n                ,properties:{\n                {%- if value[\'properties\'] is defined and value[\'properties\'] is mapping -%}\n                    {{- format_parameters(value[\'properties\'], value[\'required\'] | default([])) -}}\n                {%- elif value is mapping -%}\n                    {{- format_parameters(value, value[\'required\'] | default([])) -}}\n                {%- endif -%}\n                }\n                {%- if value[\'required\'] -%}\n                    ,required:[\n                    {%- for item in value[\'required\'] | default([]) -%}\n                        <|\"|>{{- item -}}<|\"|>\n                        {%- if not loop.last %},{% endif -%}\n                    {%- endfor -%}\n                    ]\n                {%- endif -%}\n            {%- elif value[\'type\'] | upper == \'ARRAY\' -%}\n                {%- if value[\'items\'] is mapping and value[\'items\'] -%}\n                    ,items:{\n                    {%- set ns_items = namespace(found_first=false) -%}\n                    {%- for item_key, item_value in value[\'items\'] | dictsort -%}\n                        {%- if item_value is not none -%}\n                            {%- if ns_items.found_first %},{% endif -%}\n                            {%- set ns_items.found_first = true -%}\n                            {%- if item_key == \'properties\' -%}\n                                properties:{\n                                {%- if item_value is mapping -%}\n                                    {{- format_parameters(item_value, value[\'items\'][\'required\'] | default([])) -}}\n                                {%- endif -%}\n                                }\n                            {%- elif item_key == \'required\' -%}\n                                required:[\n                                {%- for req_item in item_value -%}\n                                    <|\"|>{{- req_item -}}<|\"|>\n                                    {%- if not loop.last %},{% endif -%}\n                                {%- endfor -%}\n                                ]\n                            {%- elif item_key == \'type\' -%}\n                                {%- if item_value is string -%}\n                                    type:{{ format_argument(item_value | upper) }}\n                                {%- else -%}\n                                    type:{{ format_argument(item_value | map(\'upper\') | list) }}\n                                {%- endif -%}\n                            {%- else -%}\n                                {{ item_key }}:{{ format_argument(item_value) }}\n                            {%- endif -%}\n                        {%- endif -%}\n                    {%- endfor -%}\n                    }\n                {%- endif -%}\n            {%- endif -%}\n            {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}\n            type:<|\"|>{{ value[\'type\'] | upper }}<|\"|>}\n        {%- endif -%}\n    {%- endfor -%}\n{%- endmacro -%}\n{%- macro format_function_declaration(tool_data) -%}\n    declaration:{{- tool_data[\'function\'][\'name\'] -}}{description:<|\"|>{{- tool_data[\'function\'][\'description\'] -}}<|\"|>\n    {%- set params = tool_data[\'function\'][\'parameters\'] -%}\n    {%- if params -%}\n        ,parameters:{\n        {%- if params[\'properties\'] -%}\n            properties:{ {{- format_parameters(params[\'properties\'], params[\'required\']) -}} },\n        {%- endif -%}\n        {%- if params[\'required\'] -%}\n            required:[\n            {%- for item in params[\'required\'] -%}\n                <|\"|>{{- item -}}<|\"|>\n                {{- \',\' if not loop.last -}}\n            {%- endfor -%}\n            ],\n        {%- endif -%}\n        {%- if params[\'type\'] -%}\n            type:<|\"|>{{- params[\'type\'] | upper -}}<|\"|>}\n        {%- endif -%}\n    {%- endif -%}\n    {%- if \'response\' in tool_data[\'function\'] -%}\n        {%- set response_declaration = tool_data[\'function\'][\'response\'] -%}\n        ,response:{\n        {%- if response_declaration[\'description\'] -%}\n            description:<|\"|>{{- response_declaration[\'description\'] -}}<|\"|>,\n        {%- endif -%}\n        {%- if response_declaration[\'type\'] | upper == \'OBJECT\' -%}\n            type:<|\"|>{{- response_declaration[\'type\'] | upper -}}<|\"|>}\n        {%- endif -%}\n    {%- endif -%}\n    }\n{%- endmacro -%}\n{%- macro format_argument(argument, escape_keys=True) -%}\n    {%- if argument is string -%}\n        {{- \'<|\"|>\' + argument + \'<|\"|>\' -}}\n    {%- elif argument is boolean -%}\n        {{- \'true\' if argument else \'false\' -}}\n    {%- elif argument is mapping -%}\n        {{- \'{\' -}}\n        {%- set ns = namespace(found_first=false) -%}\n        {%- for key, value in argument | dictsort -%}\n            {%- if ns.found_first %},{% endif -%}\n            {%- set ns.found_first = true -%}\n            {%- if escape_keys -%}\n                {{- \'<|\"|>\' + key + \'<|\"|>\' -}}\n            {%- else -%}\n                {{- key -}}\n            {%- endif -%}\n            :{{- format_argument(value, escape_keys=escape_keys) -}}\n        {%- endfor -%}\n        {{- \'}\' -}}\n    {%- elif argument is sequence -%}\n        {{- \'[\' -}}\n        {%- for item in argument -%}\n            {{- format_argument(item, escape_keys=escape_keys) -}}\n            {%- if not loop.last %},{% endif -%}\n        {%- endfor -%}\n        {{- \']\' -}}\n    {%- else -%}\n        {{- argument -}}\n    {%- endif -%}\n{%- endmacro -%}\n{%- macro strip_thinking(text) -%}\n    {%- set ns = namespace(result=\'\') -%}\n    {%- for part in text.split(\'<channel|>\') -%}\n        {%- if \'<|channel>\' in part -%}\n            {%- set ns.result = ns.result + part.split(\'<|channel>\')[0] -%}\n        {%- else -%}\n            {%- set ns.result = ns.result + part -%}\n        {%- endif -%}\n    {%- endfor -%}\n    {{- ns.result | trim -}}\n{%- endmacro -%}\n\n{%- set ns = namespace(prev_message_type=None) -%}\n{%- set loop_messages = messages -%}\n{{ bos_token }}\n{#- Handle System/Tool Definitions Block -#}\n{%- if (enable_thinking is defined and enable_thinking) or tools or messages[0][\'role\'] in [\'system\', \'developer\'] -%}\n    {{- \'<|turn>system\\n\' -}}\n\n    {#- Inject Thinking token at the very top of the FIRST system turn -#}\n    {%- if enable_thinking is defined and enable_thinking -%}\n        {{- \'<|think|>\' -}}\n    {%- endif -%}\n\n    {%- if messages[0][\'role\'] in [\'system\', \'developer\'] -%}\n        {%- if messages[0][\'content\'] is string -%}\n            {{- messages[0][\'content\'] | trim -}}\n        {%- elif messages[0][\'content\'] is sequence -%}\n            {%- for item in messages[0][\'content\'] -%}\n                {%- if item[\'type\'] == \'text\' -%}\n                    {{- item[\'text\'] | trim -}}\n                {%- endif -%}\n            {%- endfor -%}\n        {%- endif -%}\n        {%- set loop_messages = messages[1:] -%}\n    {%- endif -%}\n\n    {%- if tools -%}\n        {{- \'\\n\\n\' -}}\n        {%- for tool in tools %}\n            {{- \'<|tool>\' -}}\n            {{- format_function_declaration(tool) | trim -}}\n            {{- \'<tool|>\' -}}\n        {%- endfor %}\n    {%- endif -%}\n\n    {{- \'<turn|>\\n\' -}}\n{%- endif %}\n\n{#- Loop through messages -#}\n{%- for message in loop_messages -%}\n    {%- set role = \'model\' if message[\'role\'] == \'assistant\' else message[\'role\'] -%}\n        {%- if role != \'tool\' -%}\n            {{- \'<|turn>\' + role + \'\\n\' }}\n        {%- endif -%}\n\n            {%- if message[\'tool_calls\'] -%}\n                {%- for tool_call in message[\'tool_calls\'] -%}\n                    {%- set function = tool_call[\'function\'] -%}\n                    {{- \'<|tool_call>call:\' + function[\'name\'] + \'{\' -}}\n                    {%- if function[\'arguments\'] is mapping -%}\n                        {%- set ns_args = namespace(found_first=false) -%}\n                        {%- for key, value in function[\'arguments\'] | dictsort -%}\n                            {%- if ns_args.found_first %},{% endif -%}\n                            {%- set ns_args.found_first = true -%}\n                            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}\n                        {%- endfor -%}\n                    {%- elif function[\'arguments\'] is string -%}\n                        {{- function[\'arguments\'] -}}\n                    {%- endif -%}\n                    {{- \'}<tool_call|>\' -}}\n                {%- endfor -%}\n                {%- set ns.prev_message_type = \'tool_call\' -%}\n            {%- endif -%}\n\n            {%- if role == \'tool\' and message[\'content\'] is sequence -%}\n                {#- Tool Response handling -#}\n                {%- for item in message[\'content\'] -%}\n                    {{- \'<|tool_response>\' -}}\n                    {%- if item[\'response\'] is mapping -%}\n                        {{- \'response:\' + item[\'name\'] | default(\'unknown\') + \'{\' -}}\n                        {%- for key, value in item[\'response\'] | dictsort -%}\n                            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}\n                            {%- if not loop.last %},{% endif -%}\n                        {%- endfor -%}\n                        {{- \'}\' -}}\n                    {%- else -%}\n                        {{- \'response:\' + item[\'name\'] | default(\'unknown\') + \'{value:\' + format_argument(item[\'response\'], escape_keys=False) + \'}\' -}}\n                    {%- endif -%}\n                    {{- \'<tool_response|>\' -}}\n                {%- endfor -%}\n                {%- set ns.prev_message_type = \'tool_response\' -%}\n            {%- else -%}\n                {%- if message[\'content\'] is string -%}\n                    {%- if role == \'model\' -%}\n                        {{- strip_thinking(message[\'content\']) -}}\n                    {%- else -%}\n                        {{- message[\'content\'] | trim -}}\n                    {%- endif -%}\n                {%- elif message[\'content\'] is sequence -%}\n                    {%- for item in message[\'content\'] -%}\n                        {%- if item[\'type\'] == \'text\' -%}\n                            {%- if role == \'model\' -%}\n                                {{- strip_thinking(item[\'text\']) -}}\n                            {%- else -%}\n                                {{- item[\'text\'] | trim -}}\n                            {%- endif -%}\n                        {%- elif item[\'type\'] == \'image\' -%}\n                            {{- \'\\n\\n<|image|>\\n\\n\' -}}\n                        {%- elif item[\'type\'] == \'audio\' -%}\n                            {{- \'<|audio|>\' -}}\n                        {%- endif -%}\n                    {%- endfor -%}\n                {%- endif -%}\n                {%- set ns.prev_message_type = None -%} \n            {%- endif -%}\n        {%- if ns.prev_message_type == None -%}\n            {{- \'<turn|>\\n\' -}}\n        {%- endif -%}\n{%- endfor -%}\n\n{%- if add_generation_prompt -%}\n    {%- if ns.prev_message_type != \'tool_response\' -%}\n        {{- \'<|turn>model\\n\' -}}\n    {%- endif -%}\n{%- endif -%}"
channels {
  channel_name: "thought"
  start: "<|channel>thought\n"
  end: "<channel|>"
}
  BenchmarkParams: Not set
  VisionExecutorSettings: VisionExecutorSettings:
  ModelAssets: model_path: /home/nemisis/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it
I0000 00:00:1775483829.667374     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_PREFILL_DECODE
I0000 00:00:1775483829.667398     758 model_resources_litert_lm.cc:68] litert model size: 818275184
INFO: [environment.cc:30] Creating LiteRT environment with options
WARNING: [auto_registration.cc:192] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
INFO: [auto_registration.cc:279] Loading GPU accelerator(libLiteRtGpuAccelerator.so).
INFO: [auto_registration.cc:279] Loading GPU accelerator(libLiteRtWebGpuAccelerator.so).
INFO: [accelerator_registry.cc:54] RegisterAccelerator: ptr=0x7786ad011fc0, name=GPU WebGPU
INFO: [auto_registration.cc:287] Dynamically loaded GPU accelerator(libLiteRtWebGpuAccelerator.so) registered.
INFO: [accelerator_registry.cc:54] RegisterAccelerator: ptr=0x7786accabec0, name=CpuAccelerator
INFO: [auto_registration.cc:325] CPU accelerator registered.
I0000 00:00:1775483829.967055     758 llm_executor_settings_utils.cc:137] Setting serialization dir: /home/nemisis/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm/snapshots/616f4124e6ff216292f16e7f73ff33b5ba9a4dd4
INFO: [magic_number_utils.cc:97] Loaded: num_subgraphs=976, num_signatures=4
INFO: [magic_number_utils.cc:105]   signature=decode, subgraph_index=0, num_tensors=2703, num_inputs=35, num_outputs=32, num_ops=2068
INFO: [magic_number_utils.cc:105]   signature=prefill_1024, subgraph_index=1, num_tensors=1454, num_inputs=35, num_outputs=30, num_ops=1107
INFO: [magic_number_utils.cc:105]   signature=prefill_128, subgraph_index=2, num_tensors=1454, num_inputs=35, num_outputs=30, num_ops=1107
INFO: [magic_number_utils.cc:105]   signature=verify, subgraph_index=3, num_tensors=2887, num_inputs=35, num_outputs=32, num_ops=2243
INFO: [magic_number_utils.cc:109] Magic number configs: num_configs=1
INFO: [magic_number_utils.cc:116]   config[0]: magic_number=32003, target_number=4096, signature_prefix=null
INFO: [magic_number_utils.cc:425] 439 tensors of signature decode have been updated for magic number  32003 to target number 4096
INFO: [magic_number_utils.cc:425] 271 tensors of signature prefill_1024 have been updated for magic number  32003 to target number 4096
INFO: [magic_number_utils.cc:425] 270 tensors of signature prefill_128 have been updated for magic number  32003 to target number 4096
INFO: [magic_number_utils.cc:425] 444 tensors of signature verify have been updated for magic number  32003 to target number 4096
INFO: [compiled_model.cc:719] Flatbuffer model initialized directly from incoming litert model.
WARNING: Logging before InitGoogle() is written to STDERR
I0000 00:00:1775483830.047934     758 delegate_webgpu.cc:211] Create WebGPU environment with device substr:
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
Warning: Vulkan fullDrawIndexUint32 feature required.
 - While initializing adapter (backend=BackendType::Vulkan)
    at InitializeImpl (third_party/dawn/src/dawn/native/vulkan/PhysicalDeviceVk.cpp:253)

Warning: maxDynamicUniformBuffersPerPipelineLayout artificially reduced from 1000000 to 16 to fit dynamic offset allocation limit.
Warning: maxDynamicStorageBuffersPerPipelineLayout artificially reduced from 1000000 to 16 to fit dynamic offset allocation limit.
I0000 00:00:1775483832.253381     758 environment.cc:521] Found 1 adapters
I0000 00:00:1775483832.253420     758 environment.cc:534] adapter: llvmpipe (LLVM 15.0.7, 256 bits), arch=software, vendor=mesa, backend=Vulkan, adapterType=CPU / Software
I0000 00:00:1775483832.253473     758 environment.cc:556] Selected adapter: llvmpipe (LLVM 15.0.7, 256 bits), arch=software, vendor=mesa, backend=Vulkan, adapterType=CPU / Software
I0000 00:00:1775483832.307452     758 delegate_webgpu.cc:238] Created a WebGPU environment.
INFO: [environment.cc:41] Adding options to the existing LiteRT environment
INFO: [gpu_environment.cc:364] Failed to create OpenCL context.
INFO: [gpu_environment.h:152] Created LiteRT GpuEnvironment.
INFO: [environment.cc:41] Adding options to the existing LiteRT environment
I0000 00:00:1775483832.308061     758 delegate_webgpu.cc:632] # of threads to upload weights = 2
I0000 00:00:1775483832.308200     758 delegate_webgpu.cc:640] # of threads to compile kernels = 1
I0000 00:00:1775483832.388808     758 delegate_kernel.cc:279] Total 32 external tensors are used for delegate inputs and outputs
I0000 00:00:1775483832.388848     758 delegate_kernel.cc:716] Initializing WebGPU-based API from serialized data.
I0000 00:00:1775483832.633295     758 delegate_kernel.cc:616] Initialized InferenceContext from serialized data.
I0000 00:00:1775483832.645137     758 delegate_kernel.cc:279] Total 31 external tensors are used for delegate inputs and outputs
I0000 00:00:1775483832.645178     758 delegate_kernel.cc:716] Initializing WebGPU-based API from serialized data.
I0000 00:00:1775483832.756648     758 delegate_kernel.cc:616] Initialized InferenceContext from serialized data.
I0000 00:00:1775483832.765489     758 delegate_kernel.cc:279] Total 31 external tensors are used for delegate inputs and outputs
I0000 00:00:1775483832.765531     758 delegate_kernel.cc:716] Initializing WebGPU-based API from serialized data.
I0000 00:00:1775483832.839186     758 delegate_kernel.cc:616] Initialized InferenceContext from serialized data.
I0000 00:00:1775483832.857003     758 delegate_kernel.cc:279] Total 32 external tensors are used for delegate inputs and outputs
I0000 00:00:1775483832.857051     758 delegate_kernel.cc:716] Initializing WebGPU-based API from serialized data.
I0000 00:00:1775483833.001031     758 delegate_kernel.cc:616] Initialized InferenceContext from serialized data.
INFO: [environment.cc:30] Creating LiteRT environment with options
WARNING: [auto_registration.cc:192] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
INFO: [auto_registration.cc:279] Loading GPU accelerator(libLiteRtGpuAccelerator.so).
INFO: [auto_registration.cc:279] Loading GPU accelerator(libLiteRtWebGpuAccelerator.so).
INFO: [accelerator_registry.cc:54] RegisterAccelerator: ptr=0x5784a83d7a90, name=GPU WebGPU
INFO: [auto_registration.cc:287] Dynamically loaded GPU accelerator(libLiteRtWebGpuAccelerator.so) registered.
INFO: [accelerator_registry.cc:54] RegisterAccelerator: ptr=0x5784f0001330, name=CpuAccelerator
INFO: [auto_registration.cc:325] CPU accelerator registered.
I0000 00:00:1775483833.044445     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_END_OF_AUDIO
I0000 00:00:1775483833.044544     758 model_resources_litert_lm.cc:68] litert model size: 6772
I0000 00:00:1775483833.044822     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_END_OF_VISION
I0000 00:00:1775483833.044845     758 model_resources_litert_lm.cc:68] litert model size: 6772
I0000 00:00:1775483833.045567     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_EMBEDDER
I0000 00:00:1775483833.045591     758 model_resources_litert_lm.cc:68] litert model size: 103811720
INFO: [magic_number_utils.cc:97] Loaded: num_subgraphs=1, num_signatures=1
INFO: [magic_number_utils.cc:105]   signature=embedder, subgraph_index=0, num_tensors=16, num_inputs=1, num_outputs=1, num_ops=8
INFO: [magic_number_utils.cc:109] Magic number configs: num_configs=1
INFO: [magic_number_utils.cc:116]   config[0]: magic_number=32003, target_number=4096, signature_prefix=null
INFO: [compiled_model.cc:719] Flatbuffer model initialized directly from incoming litert model.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
I0000 00:00:1775483833.063724     758 embedding_lookup_text.cc:345] EmbeddingLookupText initialized: signature=embedder, rank=3, floats_per_token=1536
INFO: [magic_number_utils.cc:97] Loaded: num_subgraphs=1, num_signatures=1
INFO: [magic_number_utils.cc:105]   signature=eoi, subgraph_index=0, num_tensors=1, num_inputs=0, num_outputs=1, num_ops=0
INFO: [magic_number_utils.cc:109] Magic number configs: num_configs=1
INFO: [magic_number_utils.cc:116]   config[0]: magic_number=32003, target_number=4096, signature_prefix=null
INFO: [compiled_model.cc:719] Flatbuffer model initialized directly from incoming litert model.
INFO: [compiled_model.cc:1486] Tracked constant output tensor eoi_embedding with locked address
INFO: [magic_number_utils.cc:97] Loaded: num_subgraphs=1, num_signatures=1
INFO: [magic_number_utils.cc:105]   signature=eoa, subgraph_index=0, num_tensors=1, num_inputs=0, num_outputs=1, num_ops=0
INFO: [magic_number_utils.cc:109] Magic number configs: num_configs=1
INFO: [magic_number_utils.cc:116]   config[0]: magic_number=32003, target_number=4096, signature_prefix=null
INFO: [compiled_model.cc:719] Flatbuffer model initialized directly from incoming litert model.
INFO: [compiled_model.cc:1486] Tracked constant output tensor eoa_embedding with locked address
I0000 00:00:1775483833.070814     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_PER_LAYER_EMBEDDER
I0000 00:00:1775483833.070847     758 model_resources_litert_lm.cc:68] litert model size: 1284518392
INFO: [magic_number_utils.cc:97] Loaded: num_subgraphs=1, num_signatures=1
INFO: [magic_number_utils.cc:105]   signature=per_layer_embedder, subgraph_index=0, num_tensors=83, num_inputs=1, num_outputs=1, num_ops=42
INFO: [magic_number_utils.cc:109] Magic number configs: num_configs=1
INFO: [magic_number_utils.cc:116]   config[0]: magic_number=32003, target_number=4096, signature_prefix=null
INFO: [compiled_model.cc:719] Flatbuffer model initialized directly from incoming litert model.
I0000 00:00:1775483833.314252     758 embedding_lookup_text.cc:345] EmbeddingLookupText initialized: signature=per_layer_embedder, rank=4, floats_per_token=8960
I0000 00:00:1775483833.314397     758 litert_lm_loader.cc:126] LitertLmLoader::Initialize
I0000 00:00:1775483833.314454     758 litert_lm_loader.cc:149] mmap_status is ok
I0000 00:00:1775483833.314472     758 litert_lm_loader.cc:155] status: OK
I0000 00:00:1775483833.314474     758 litert_lm_loader.cc:156] major_version: 1
I0000 00:00:1775483833.314477     758 litert_lm_loader.cc:157] minor_version: 5
I0000 00:00:1775483833.314478     758 litert_lm_loader.cc:158] patch_version: 0
I0000 00:00:1775483833.314480     758 litert_lm_loader.cc:209] section_index: 0
I0000 00:00:1775483833.314481     758 litert_lm_loader.cc:210] section_data_type: LlmMetadataProto
I0000 00:00:1775483833.314483     758 litert_lm_loader.cc:212] section_begin_offset: 16384
I0000 00:00:1775483833.314485     758 litert_lm_loader.cc:213] section_end_offset: 28576
I0000 00:00:1775483833.314486     758 litert_lm_loader.cc:209] section_index: 1
I0000 00:00:1775483833.314487     758 litert_lm_loader.cc:210] section_data_type: SP_Tokenizer
I0000 00:00:1775483833.314488     758 litert_lm_loader.cc:212] section_begin_offset: 32768
I0000 00:00:1775483833.314489     758 litert_lm_loader.cc:213] section_end_offset: 4721781
I0000 00:00:1775483833.314491     758 litert_lm_loader.cc:190] model_type: tf_lite_embedder
I0000 00:00:1775483833.314494     758 litert_lm_loader.cc:209] section_index: 2
I0000 00:00:1775483833.314495     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314496     758 litert_lm_loader.cc:212] section_begin_offset: 4734976
I0000 00:00:1775483833.314497     758 litert_lm_loader.cc:213] section_end_offset: 108546696
I0000 00:00:1775483833.314499     758 litert_lm_loader.cc:190] model_type: tf_lite_per_layer_embedder
I0000 00:00:1775483833.314501     758 litert_lm_loader.cc:209] section_index: 3
I0000 00:00:1775483833.314502     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314503     758 litert_lm_loader.cc:212] section_begin_offset: 108560384
I0000 00:00:1775483833.314504     758 litert_lm_loader.cc:213] section_end_offset: 1393078776
I0000 00:00:1775483833.314505     758 litert_lm_loader.cc:190] model_type: tf_lite_audio_encoder_hw
I0000 00:00:1775483833.314507     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483833.314508     758 litert_lm_loader.cc:209] section_index: 4
I0000 00:00:1775483833.314509     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314510     758 litert_lm_loader.cc:212] section_begin_offset: 1393082368
I0000 00:00:1775483833.314512     758 litert_lm_loader.cc:213] section_end_offset: 1487134648
I0000 00:00:1775483833.314513     758 litert_lm_loader.cc:190] model_type: tf_lite_audio_adapter
I0000 00:00:1775483833.314514     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483833.314515     758 litert_lm_loader.cc:209] section_index: 5
I0000 00:00:1775483833.314516     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314518     758 litert_lm_loader.cc:212] section_begin_offset: 1487142912
I0000 00:00:1775483833.314519     758 litert_lm_loader.cc:213] section_end_offset: 1496584156
I0000 00:00:1775483833.314520     758 litert_lm_loader.cc:190] model_type: tf_lite_end_of_audio
I0000 00:00:1775483833.314521     758 litert_lm_loader.cc:209] section_index: 6
I0000 00:00:1775483833.314522     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314523     758 litert_lm_loader.cc:212] section_begin_offset: 1496596480
I0000 00:00:1775483833.314524     758 litert_lm_loader.cc:213] section_end_offset: 1496603252
I0000 00:00:1775483833.314536     758 litert_lm_loader.cc:190] model_type: tf_lite_vision_encoder
I0000 00:00:1775483833.314539     758 litert_lm_loader.cc:209] section_index: 7
I0000 00:00:1775483833.314541     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314542     758 litert_lm_loader.cc:212] section_begin_offset: 1496612864
I0000 00:00:1775483833.314543     758 litert_lm_loader.cc:213] section_end_offset: 1715702984
I0000 00:00:1775483833.314545     758 litert_lm_loader.cc:190] model_type: tf_lite_vision_adapter
I0000 00:00:1775483833.314546     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483833.314547     758 litert_lm_loader.cc:209] section_index: 8
I0000 00:00:1775483833.314549     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314550     758 litert_lm_loader.cc:212] section_begin_offset: 1715716096
I0000 00:00:1775483833.314551     758 litert_lm_loader.cc:213] section_end_offset: 1720443208
I0000 00:00:1775483833.314552     758 litert_lm_loader.cc:190] model_type: tf_lite_end_of_vision
I0000 00:00:1775483833.314553     758 litert_lm_loader.cc:209] section_index: 9
I0000 00:00:1775483833.314555     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314556     758 litert_lm_loader.cc:212] section_begin_offset: 1720451072
I0000 00:00:1775483833.314557     758 litert_lm_loader.cc:213] section_end_offset: 1720457844
I0000 00:00:1775483833.314558     758 litert_lm_loader.cc:190] model_type: tf_lite_prefill_decode
I0000 00:00:1775483833.314559     758 litert_lm_loader.cc:209] section_index: 10
I0000 00:00:1775483833.314560     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314561     758 litert_lm_loader.cc:212] section_begin_offset: 1720467456
I0000 00:00:1775483833.314562     758 litert_lm_loader.cc:213] section_end_offset: 2538742640
I0000 00:00:1775483833.314563     758 litert_lm_loader.cc:190] model_type: tf_lite_mtp_drafter
I0000 00:00:1775483833.314565     758 litert_lm_loader.cc:203] section_backend_constraint: cpu
I0000 00:00:1775483833.314567     758 litert_lm_loader.cc:209] section_index: 11
I0000 00:00:1775483833.314568     758 litert_lm_loader.cc:210] section_data_type: TFLiteModel
I0000 00:00:1775483833.314569     758 litert_lm_loader.cc:212] section_begin_offset: 2538749952
I0000 00:00:1775483833.314570     758 litert_lm_loader.cc:213] section_end_offset: 2583082648
I0000 00:00:1775483833.315671     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_VISION_ENCODER
I0000 00:00:1775483833.315706     758 model_resources_litert_lm.cc:68] litert model size: 219090120
I0000 00:00:1775483833.389173     758 model_resources_litert_lm.cc:67] model_type: TF_LITE_VISION_ADAPTER
I0000 00:00:1775483833.389222     758 model_resources_litert_lm.cc:68] litert model size: 4727112
INFO: [magic_number_utils.cc:97] Loaded: num_subgraphs=1, num_signatures=1
INFO: [magic_number_utils.cc:105]   signature=vision_280, subgraph_index=0, num_tensors=2502, num_inputs=2, num_outputs=2, num_ops=2245
INFO: [magic_number_utils.cc:109] Magic number configs: num_configs=1
INFO: [magic_number_utils.cc:116]   config[0]: magic_number=32003, target_number=4096, signature_prefix=null
INFO: [compiled_model.cc:719] Flatbuffer model initialized directly from incoming litert model.
INFO: [environment.cc:41] Adding options to the existing LiteRT environment
I0000 00:00:1775483833.395941     758 delegate_webgpu.cc:632] # of threads to upload weights = 0
I0000 00:00:1775483833.396008     758 delegate_webgpu.cc:640] # of threads to compile kernels = 0
I0000 00:00:1775483835.880817     758 delegate_kernel.cc:716] Initializing WebGPU-based API from serialized data.
E0000 00:00:1775483835.881750     758 delegate_webgpu.cc:370] Failed to create litert::ml_drift::DelegateKernelLiteRt: RESOURCE_EXHAUSTED: Buffer with size - 304819200 bytes can not be created. Max buffer size for this GPU - 134217728 bytes. Shape - {bhwdc, {12, 1, 2520, 1, 2520}}, data type - float32.
=== Source Location Trace: ===
third_party/ml_drift/common/task/tensor_desc.cc:1854
third_party/ml_drift/common/gpu_model_util.cc:232
third_party/ml_drift/common/gpu_model_util.cc:269
third_party/ml_drift/common/gpu_model_util.cc:432
third_party/odml/litert/ml_drift/delegate/delegate_kernel.cc:697
third_party/odml/litert/ml_drift/delegate/delegate_kernel.cc:627
third_party/odml/litert/ml_drift/delegate/delegate_kernel.cc:719
third_party/odml/litert/ml_drift/delegate/delegate_kernel.cc:284
third_party/odml/litert/ml_drift/delegate/delegate_kernel_litert.cc:158
ERROR: Failed to initialize kernel.
ERROR: Restored original execution plan after delegate application failure.
ERROR:    Traceback (most recent call last):
  File "/mnt/e/sandbox/parlor/src/.venv/lib/python3.14/site-packages/starlette/routing.py", line 638, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/nemisis/.local/share/uv/python/cpython-3.14.3-linux-x86_64-gnu/lib/python3.14/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/sandbox/parlor/src/server.py", line 66, in lifespan
    await asyncio.get_event_loop().run_in_executor(None, load_models)
  File "/home/nemisis/.local/share/uv/python/cpython-3.14.3-linux-x86_64-gnu/lib/python3.14/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/home/nemisis/.local/share/uv/python/cpython-3.14.3-linux-x86_64-gnu/lib/python3.14/concurrent/futures/thread.py", line 73, in run
    return fn(*args, **kwargs)
  File "/mnt/e/sandbox/parlor/src/server.py", line 52, in load_models
    engine = litert_lm.Engine(
        MODEL_PATH,
    ...<2 lines>...
        audio_backend=litert_lm.Backend.CPU,
    )
RuntimeError: python/litert_lm/litert_lm.cc:496: operator(): INTERNAL: ERROR: [runtime/executor/vision_litert_compiled_model_executor.cc:188]
└ ERROR: [external/litert/litert/cc/litert_compiled_model.h:836]

ERROR:    Application startup failed. Exiting.
INFO: [accelerator_registry.cc:43] DestroyAccelerator: ptr=0x5784f0001330, name=CpuAccelerator
INFO: [accelerator_registry.cc:43] DestroyAccelerator: ptr=0x5784a83d7a90, name=GPU WebGPU
nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/src$

Any idea how to resolve this, short of having to run it in a native Linux machine?

[fikrikarim]

Thanks for the detailed information. Unfortunately, I don't have a Windows machine myself so it's hard for me to debug it. I'll give it a try since there seems to be some people that show interest on running this on Windows. Although, I can't promise anything right now.

[fikrikarim]

@yhdanid could you try running the litert-lm CLI with the gpu backend? The default backend is CPU.

litert-lm run  \
   --backend=gpu \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm \
   --prompt="What is the capital of France?"

If that works, try running the litert-lm benchmark to double check whether it's actually running on the GPU:

litert-lm benchmark  \
   --backend=gpu \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm

and compare with

litert-lm benchmark  \
   --backend=cpu \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm

[yhdanid]

SUMMARY

I've tried with --backend=cpu, it works fine, and also seems to work when --backend=gpu is specified. Benchmarking fails on GPU backend but works fine in CPU. So I'm confused.

Note that I did this with model locally downloaded in /model directory (to save on bandwidth as I was testing), and while downloading from huggingface. All this was done in WSL2, Ubuntu 22.04.5 LTS.

GPU IS ACCESSIBLE FROM TORCH
As an experiment, I installed pytorch in another virtual environment and tested if it can access the GPU, and I got these results:

(.venv) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/test-cuda$ python torch_cuda_test.py
Is CUDA available? True
CUDA device count: 1
CUDA current device: 0
Torch CUDA device: <torch.cuda.device object at 0x703ae7443f10>
CUDA device name: NVIDIA GeForce RTX 4080 GPU

DOCKER
I also tried seeing if parlor works in docker as it is more widely available than WSL (image built FROM nvidia/cuda:12.8.0-devel-ubuntu24.04) with more or less the same results. "uv run python server.py" fails with the same error message as the one above from WSL. Running litert-lm run however on CPU succeeds but on GPU prints out: "Segmentation fault (core dumped)".

WSL: MODEL IN LOCAL MODEL DIRECTORY

CPU BACKEND

(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor$ ls
default.profraw  model  src
(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor$ litert-lm run \
    ./model/gemma-4-E2B-it.litertlm \
    --prompt="What is the capital of France?"
The capital of France is **Paris**.

GPU BACKEND

(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor$ litert-lm run \
    ./model/gemma-4-E2B-it.litertlm \
    --prompt="What is the capital of France?" \
    --backend=gpu
The capital of France is **Paris**.

WSL: DOWNLOADING MODEL FROM HUGGINGFACE

CPU BACKEND

(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/src$ litert-lm run \
     --backend=cpu \ 
     --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm  \
     gemma-4-E2B-it.litertlm  \
     --prompt="What is the capital of France?"
Downloading gemma-4-E2B-it.litertlm from litert-community/gemma-4-E2B-it-litert-lm...
The capital of France is **Paris**.

GPU BACKEND

(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/src$ litert-lm run  \
   --backend=gpu \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm \
   --prompt="What is the capital of France?"
Downloading gemma-4-E2B-it.litertlm from litert-community/gemma-4-E2B-it-litert-lm...
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
The capital of France is **Paris**.

WSL: BENCHMARKING

GPU

(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/src$ litert-lm benchmark  \
   --backend=gpu \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm
Downloading gemma-4-E2B-it.litertlm from litert-community/gemma-4-E2B-it-litert-lm...
Benchmarking model: gemma-4-E2B-it.litertlm (/home/nemisis/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm/snapshots/2a101e00c47f942975ce8b493c2498311ed9900d/gemma-4-E2B-it.litertlm)
Number of tokens in prefill: 256
Number of tokens in decode : 256
Backend                    : gpu
Speculative decoding       : auto
An error occurred during benchmarking
Traceback (most recent call last):
  File "/mnt/e/sandbox/parlor/src/.venv/lib/python3.14/site-packages/litert_lm_cli/model.py", line 388, in benchmark
    result = benchmark_obj.run()
RuntimeError: python/litert_lm/litert_lm.cc:380: operator(): INTERNAL: ERROR: [runtime/executor/llm_litert_compiled_model_executor.cc:765]
└ Failed to invoke the compiled model

CPU

(parlor) nemisis@nemisis-BLACK:/mnt/e/sandbox/parlor/src$ litert-lm benchmark  \
   --backend=cpu \
   --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
   gemma-4-E2B-it.litertlm
Downloading gemma-4-E2B-it.litertlm from litert-community/gemma-4-E2B-it-litert-lm...
Benchmarking model: gemma-4-E2B-it.litertlm (/home/nemsis/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm/snapshots/2a101e00c47f942975ce8b493c2498311ed9900d/gemma-4-E2B-it.litertlm)
Number of tokens in prefill: 256
Number of tokens in decode : 256
Backend                    : cpu
Speculative decoding       : auto
----- Results -----
Prefill speed:        76.20 tokens/s
Decode speed:         30.04 tokens/s
Init time:            19.4130 s
Time to first token:  3.3930 s

I hope this helps

[fikrikarim]

Thanks for the additional information.

Could you changing the litert_lm.Backend to be CPU on all backend on the server.py?

    engine = litert_lm.Engine(
        MODEL_PATH,
        backend=litert_lm.Backend.CPU,
        vision_backend=litert_lm.Backend.CPU,
        audio_backend=litert_lm.Backend.CPU,
    )

If that works, then there's a problem with the GPU inference on your setup.

If you could isolate and reproduce the issue just on python litert_lm without additional stuff from this repo, perhaps it'd be better to report the issue or ask for support from the main LiteRT-LM repo?

[yhdanid]

I have looked around briefly concerning this, and it seems the matter is not only related to litert (which seems to rely on Vulkan for GPU hardware acceleration), but I've also found out that even llama.cpp's Vulkan implementation won't use the dedicated NVIDIA GPU in WSL. Even when I specify --device MY_VULKAN_DEVICE (for NVIDIA) in the command. It still falls back to CPU acceleration. So this is not a litert issue, or llama.cpp issue, or a Parlor issue, but rather a Vulkan issue when running in WSL.

So I suggest closing this as there is nothing that can be done until a solution upstream is found for Vulkan running in WSL in GPU acceleration mode.

[yhdanid]

Just a heads up that as of yesterday [1], LiteRT-LM CLI is supported natively on Windows with both CPU and GPU backends. On litert-lm CLI page [2], more detail is provided including installation instructions. I've tried it and it works.

I have not yet tested Parlor with this new update, but will do so once I have more time. Unfortunately native Vulcan support is not a priority for the developers of WSL and Vulkan, since GPU emulation through CPU seems to be deemed sufficient [3].

[1] https://github.com/google-ai-edge/LiteRT-LM/releases/tag/v0.11.0
[2] https://ai.google.dev/edge/litert-lm/cli
[3] https://forums.developer.nvidia.com/t/critical-bug-rtx-5070-ti-mobile-missing-nvidia-icd-json-in-wsl2-causes-vulkan-initialization-failure-on-driver-591-86/359933

[fikrikarim]

Thank you for the update. Unfortunately I don't have a Windows machine to test Parlor directly. Hope it doesn't need too many changes to make it work on Windows.