NARU Bench

NARU Bench is a Japanese video understanding benchmark for multimodal models. It focuses on two hard capabilities: following narrative structure over time and interpreting culturally grounded communication in context.

flowchart TD
  V[Japanese Video] --> N[Narrative]
  V --> C[Cultural]
  N --> N1["1_1 Character evolution"]
  N --> N2["1_2 Sequential flow"]
  N --> N3["1_3 Plot progression"]
  N --> N4["1_4 Thematic development"]
  C --> C1["2_1 Aizuchi"]
  C --> C2["2_2 Reading the air"]
  C --> C3["2_3 Subtext"]
  C --> C4["2_4 Cultural context"]
  C --> C5["2_5 Sentiment"]

Loading

Current Release

• 1,481 multiple-choice questions across 9 task types
• 4 narrative tasks (1_1–1_4) and 5 cultural tasks (2_1–2_5)
• 155 unique Japanese YouTube videos
• Japanese prompt, question body, and answer choices
• Task definitions: 9 task types
• Dataset: infinimind/naru_benchmark on Hugging Face
• Evaluation-ready for lmms-eval

Tasks

ID	Task	Items
1_1	Character or entity evolution	185
1_2	Sequential topical flow	187
1_3	Plot or conflict progression	186
1_4	Thematic development	187
2_1	Aizuchi (backchanneling)	143
2_2	Reading the air (kuuki wo yomu)	147
2_3	Subtext interpretation	148
2_4	Cultural context recognition	149
2_5	Sentiment analysis	149

See 9 task types for detailed definitions and examples.

Why NARU Bench

• Many video benchmarks reward object recognition or short-horizon recall; NARU Bench targets longer-range narrative understanding and culturally grounded interpretation
• It stresses capabilities that matter in real Japanese media and communication, including implied meaning, social context, and temporal structure
• It is useful for benchmarking foundation models, evaluating product quality, and diagnosing failure modes that generic video QA often misses

Benchmark Results

Results on the full benchmark (1,481 questions):

Model	Parameters	Modalities	Accuracy (%)
DeepFrame Platform	—	—	55.3
Qwen3VL	8B	V+T	33.9
DeepFrame	8B	A+V+T	32.1
Qwen3-Omni	30B	A+V+T	29.9
InternVL3.5	8B	V+T	27.0
Qwen3VL	2B	V+T	25.2
Qwen2.5-Omni	7B	A+V+T	24.9

Modalities: A+V+T = Audio + Visual + Text, V+T = Visual + Text. Random chance is 25%.

All results use the strict scoring protocol described in Important notes below.

Evaluation with lmms-eval

NARU Bench integrates with lmms-eval via the --include_path mechanism. This means you do not need to fork or modify the lmms-eval repository — just point it at the tasks/ directory in this repo.

Prerequisites

pip install lmms-eval

Step 1: Set up video files

The dataset references videos via youtube://<id> URIs. You need a local directory with the corresponding .mp4 files named <youtube_id>.mp4.

export NARUBENCH_VIDEO_DIR=/path/to/naru/videos

Important: If NARUBENCH_VIDEO_DIR is not set, evaluation will fail with a clear error message explaining what to do.

Step 2: Run evaluation

# Run all 9 subtasks
lmms-eval \
  --model <model_name> \
  --tasks narubench \
  --include_path ./tasks \
  --batch_size 1 \
  --output_path ./results/

# Run a specific subtask
lmms-eval \
  --model <model_name> \
  --tasks narubench-narrative-1.1.char_evolution \
  --include_path ./tasks \
  --batch_size 1

Available task names

Task name	Description
narubench	All 9 subtasks (group)
narubench-narrative-1.1.char_evolution	Character evolution
narubench-narrative-1.2.sequential_topical_flow	Sequential & topical flow
narubench-narrative-1.3.plot_conflict_progression	Plot & conflict progression
narubench-narrative-1.4.idea_thematic_argumentative_dev	Thematic & argumentative development
narubench-cultural-2.1.aizuchi	Aizuchi interpretation
narubench-cultural-2.2.kuukiwoyomu	Reading the air
narubench-cultural-2.3.subtext_interpretation	Subtext interpretation
narubench-cultural-2.4.cultural_context_recognition	Cultural context recognition
narubench-cultural-2.5.sentiment_analysis	Sentiment analysis

How it works

The tasks/ directory contains:

File	Role
narubench.yaml	Group definition — registers all subtasks under the narubench group
base_multichoice.yml	Shared config — points to the HF dataset (infinimind/naru_benchmark), defines scoring (exact_match), delegates processing to utils.py
narrative-1.*.yaml, cultural-2.*.yaml	Per-subtask configs — inherit base, override data_files path
utils.py	Processing logic — resolves video URIs, formats Japanese MCQ prompts, provides doc_to_messages for chat-style models

The dataset is loaded from HuggingFace Hub (infinimind/naru_benchmark) where JSONL files are stored under eval/data/. The utils.py resolves youtube:// URIs to local .mp4 paths and filters out any examples whose video file cannot be found.

Important notes

• Video filtering: The process_docs step in utils.py silently filters out examples whose video cannot be resolved locally. If you see fewer examples than expected, check that all 155 video files are present in NARUBENCH_VIDEO_DIR.
• Greedy decoding: Generation uses temperature=0, max_new_tokens=100, do_sample=false.
• Metric (strict scoring protocol): exact_match (mean) — the model's raw output must exactly equal the correct option letter (A, B, C, or D). No answer extraction or normalization is applied: an output like A) ... or a full-sentence answer scores 0 even if the choice is correct. The Japanese system prompt explicitly instructs the model to reply with a single letter, so this metric also measures instruction following. All published baseline numbers were produced under this protocol — keep it unchanged when comparing against them.

Data Format

Each JSONL row:

{
  "uri": "youtube://<video_id>",
  "body": "<question>",
  "choice": ["...", "...", "...", "..."],
  "answer": "B",
  "task_type": "1_1",
  "uuid": "f4dd022ed0d25362be3d448047b5455d"
}

Field	Type	Description
uri	str	Video reference (youtube://<id>)
body	str	Question text (Japanese)
choice	list[str]	Answer options (4 choices, Japanese)
answer	str	Correct option letter (A–D)
task_type	str	Task identifier (1_1, 2_3, etc.)
uuid	str	Unique question identifier

Processed document (after validate_docs)

The task utilities transform each row before it reaches the model:

Field	Type	Description
video_path	str	Resolved local path to video file
text	str	System prompt + question + labeled options
options	list[str]	Option labels (["A","B","C","D"])
target	int	0-based index of correct answer

System prompt (Japanese):

提供された動画を詳細に分析し、以下の質問に対して最も適切な選択肢を選んでください。回答は選択肢の記号（A、B、C、D など）から 1文字のみで答えてください。

Repository Structure

naru_benchmark/
├── datafiles/
│   ├── README.md              # HuggingFace dataset card (uploaded with data)
│   └── refined/               # JSONL data files (9 subtask files)
├── tasks/
│   ├── narubench.yaml         # lmms-eval group definition
│   ├── base_multichoice.yml   # Shared task config
│   ├── narrative-1.*.yaml     # Narrative subtask configs
│   ├── cultural-2.*.yaml      # Cultural subtask configs
│   └── utils.py               # Processing utilities
├── scripts/
│   └── upload_to_hf.py        # Publishes the QA JSONL files to the HF dataset
├── docs/
│   └── task_types.md          # Detailed task type definitions
└── README.md                  # This file

Status

• Initial public release is multiple-choice only
• Current benchmark is based on Japanese online video data and is designed to expand to other Japanese video sources
• Dataset construction and evaluation settings are still being improved
• Japanese TV structure analysis is planned for a future release

Video Availability

Last checked: June 2026

Of the 155 source YouTube videos, 7 are no longer available on YouTube (removed by uploaders or platform). All 7 videos remain in our archived storage and are included in the video distribution package (42.7 GB total for all 155 videos).

Unavailable videos

Video ID	YouTube URL	Questions affected
-9mYs50JQcE	https://youtube.com/watch?v=-9mYs50JQcE	9
-EFgz52sFpE	https://youtube.com/watch?v=-EFgz52sFpE	9
3VCjf5ljPB4	https://youtube.com/watch?v=3VCjf5ljPB4	9
4WFQQJesyXc	https://youtube.com/watch?v=4WFQQJesyXc	9
5A2RoWMZ4HE	https://youtube.com/watch?v=5A2RoWMZ4HE	8
6-OSk5ysgIY	https://youtube.com/watch?v=6-OSk5ysgIY	9
6Vr7Nk1SQWs	https://youtube.com/watch?v=6Vr7Nk1SQWs	9

Total impact: 62 / 1,481 questions (4.2%), uniformly distributed across all 9 subtasks. This does not meaningfully change benchmark scores or relative model rankings.

Requesting video access

Users who cannot obtain the full set of videos from YouTube can request access to the archived copies (including the 7 unavailable videos) via:

• Google Forms: Request video access

The complete video package is approximately 42.7 GB (155 .mp4 files).

Citation

@misc{huang2026naru,
  title={NARU: A Benchmark for Japanese Video Narrative and Cultural Understanding},
  author={Yuheng Huang and Jianlang Chen and Jiayang Song and Hua Qi and Aza Kai and Vincent Markert and Edison Marrese-Taylor and Lei Ma},
  year={2026},
  note={Dataset available at https://huggingface.co/datasets/infinimind/naru_benchmark}
}

License

This project is split-licensed:

• The source code, evaluation scripts, and configuration files are licensed under the Apache-2.0 License.
• The dataset annotations, QA pairs, and metadata are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License.
• The referenced YouTube videos remain subject to their original creators' terms and YouTube's Terms of Service.