Syntactic Blind Spots

Overview

Syntactic Blind Spots is a research codebase designed to evaluate and analyze the failure modes of large language models (LLMs) on mathematical and reasoning tasks, focusing on syntactic constructs that commonly trip up model inference. The repository contains scripts for preprocessing datasets, generating model responses, paraphrasing mathematical word problems (MWPs), and quantifying syntactic complexity using Dependency Locality Theory (DLT) metrics.

This repository supports the experiments and analyses described in the paper submitted to EMNLP 2025.

Features

• Dataset Preprocessing: Converts GSM8K and other datasets into instruction-based JSON formats, suitable for prompting LLMs.
• Model Evaluation Pipelines: Automates prompt generation, model response collection, and answer extraction.
• Paraphrasing & Contrastive Analysis: Generates paraphrased MWPs to probe model sensitivity to syntactic variations.
• DLT Complexity Metrics: Quantifies the syntactic complexity of questions using DLT and correlates it with model outcomes.
• Error Analysis: Extracts correct/incorrect answers and compares reasoning traces.

Directory Structure

• data/preprocessing/: Scripts for preprocessing datasets (e.g., GSM8K).
• dlt/: DLT complexity computation and analysis.
• utils/: Post-processing and analysis utilities (e.g., answer extraction, JSON cleaning).
• paraphrasing/: Dependency parsing and paraphrase generation.
• qa/: Model QA pipelines.

Quickstart

Prerequisites

• Python 3.8+
• HuggingFace Transformers
• Datasets
• Additional dependencies: spacy, matplotlib, seaborn, fire, pandas, scipy, tqdm

Environment Setup

# Clone repository
git clone https://github.com/dwil2444/syntactic-blind-spots.git
cd syntactic-blind-spots

# Install dependencies
pip install -r requirements.txt

# Download spacy model
python -m spacy download en_core_web_sm

Running Experiments

All major pipelines can be run via the Makefile:

# Run all steps: preprocessing, generation, paraphrasing, DLT analysis
make all

# Clean up outputs and logs
make clean

Key steps include:

• make math_qa: Run QA on GSM8K or other datasets.
• make gen_mwp: Generate MWPs using the selected model.
• make clean_json: Clean generated MWPs.
• make pair_match: Find paraphrase pairs for contrastive analysis.
• make contrastive_paraphrase: Generate paraphrases for matched pairs.
• make dlt_complexity: Compute DLT complexity metrics.

Custom Usage

Individual modules can also be invoked directly using Python scripts with Fire:

python data/preprocessing/gsm8k.py --outfname data/gsm8k/eval.json --split test
python dlt/complexity.py analyze_complexity --correct_path outputs/correct.json --incorrect_path outputs/incorrect.json

Output Artifacts

• outputs/: Contains model answers, correct/incorrect instance lists, paraphrased datasets, and DLT complexity results.
• logs/: Training and inference logs for reproducibility.

Citation

If you use this code for research, please cite:

@inproceedings{williamson2025syntacticblindspots,
  title={Syntactic Blind Spots},
  author={Williamson, Dane},
  year={2025}
}

License

MIT License. See LICENSE for details.

Contact

For questions, please contact Dane Williamson.