Skip to content

scannet_4frames评测中部分样本生成重复循环,触发max_new_tokens截断并导致JSON解析失败 #71

Description

@NONO997

我在使用VG-LLM对scannet_4frames进行评测时,发现部分样本的生成结果会出现明显的重复循环,最终触发max_new_tokens上限,导致输出JSON被截断,后处理阶段解析失败。

Postprocessing: 87%|█████████████████████████████████████████████████████████████████████████▏ | 264/303 [00:09<00:01, 3
8.88it/s]2026-06-03 15:28:14.695 | ERROR | utils:threedod_process_results:372 - Error parsing prediction bbox: {"label": "book"
, "bbox_3d": [0.95, 0.99, 1.41, 0.18, 0.53, 0.26, -1.2, -1, Error: '[' was never closed (, line 1)
Postprocessing: 89%|██████████████████████████████████████████████████████████████████████████▌ | 269/303 [00:09<00:01, 3
1.94it/s]2026-06-03 15:28:14.768 | ERROR | utils:threedod_process_results:372 - Error parsing prediction bbox: {"label": "box",
"bbox_3d": [0.32, -0.78, 2.95, 0.18, 0.23, 0.23, -1.45, -0.2, Error: '[' was never closed (, line 1)
2026-06-03 15:28:14.803 | ERROR | utils:threedod_process_results:372 - Error parsing prediction bbox: {"label": "box", "bbox_3d
": [0.85, 1.17, 2.6, 0.31, Error: '[' was never closed (, line 1)
Postprocessing: 91%|████████████████████████████████████████████████████████████████████████████▌ | 276/303 [00:09<00:00, 3
0.07it/s]2026-06-03 15:28:15.021 | ERROR | utils:threedod_process_results:372 - Error parsing prediction bbox: {"label": "paper
", "bbox_3d": [0.21, -0.1, 1.5, 0.22, 0.1, Error: '[' was never closed (, line 1)

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch
--multi_gpu
--num_processes=8
--main_process_port 29501
-m lmms_eval
--model vgllm
--model_args "pretrained=vgllm-3d-vggt-4b,use_flash_attention_2=true,max_num_frames=32,max_length=12800"
--tasks scannet_4frames
--batch_size 1
--log_samples_suffix original
--log_samples
--output_path logs/$(TZ="Asia/Shanghai" date "+%Y%m%d")

任务配置中generation kwargs类似如下:
generation_kwargs:
max_new_tokens: 2000
temperature: 0
top_p: 1.0
num_beams: 1
do_sample: false

后续我也尝试将max_new_tokens提高到4096,但仍然有许多样本触发上限。该问题看起来不是max_length设置过小导致的,因为debug中输入长度只有约1116tokens。更可能的原因是模型在部分样本上没有正确生成结束符或没有正确结束JSON list,进入重复生成循环,最终耗尽max_new_tokens,导致输出被硬截断。
想请教一下:

  1. 这是已知问题吗?
  2. 官方评测scannet_4frames时推荐的max_new_tokens是多少?
  3. 对于这类重复循环导致的残缺JSON,官方评测中应该如何处理?是按原始后处理逻辑跳过解析失败行,还是有推荐的鲁棒解析方式?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions