LilyBench — Can LLMs understand LilyPond?

Abstract

Symbolic music evaluation for large language models remains fragmented across representations, datasets, and metrics. We introduce LilyBench, a LilyPond-based benchmark that jointly evaluates symbolic music generation and music understanding on the same family of open-weight LLMs. The benchmark includes a 200-prompt generation suite and ten understanding tasks adapted from ABC-Eval, covering syntax, metadata prediction, structural sequencing, and music recognition. Generation quality is measured with compile rate, MusPy descriptor distributions via Jensen–Shannon similarity, and a LilyBERT-based Fréchet Music Distance (FMD). Experiments on four open-weight models show that executable LilyPond generation is achievable zero-shot, while structural understanding tasks remain hard despite strong composer and genre recognition. The descriptor- and embedding-based metrics systematically disagree, suggesting that symbolic music evaluation benefits from metric triangulation rather than single-score ranking.

Two benchmarks, one family of models

🎹 Generation

A fixed bank of 200 metadata-conditioned prompts (composer, period, form, ensemble, part). Outputs are scored by compile rate, JS similarity over three MusPy descriptors, and a LilyBERT-FMD against in-domain (BMdataset) and out-of-domain (Mutopia) references.

🔎 Understanding

Ten ABC-Eval-adapted tasks over raw LilyPond text, grouped by reasoning depth: counting, metadata QA, bar sequencing, next-bar prediction, captioning, composer / genre / emotion recognition, and error detection. Greedy decoding, no chain-of-thought.

Category	Task	n	Source	Format	Metric
Basic	bar_count	100	Mutopia	integer	exact-match acc.
Basic	metadata_qa	60	Mutopia	4-way MC	accuracy
Segment	bar_sequencing	119	Mutopia	4-digit perm.	Kendall-τ (pen.)
	next_bar_prediction	119	Mutopia	4-way MC	accuracy
	metadata_prediction	60	Mutopia	4-way MC	accuracy
Sequence	music_captioning	60	Mutopia	4-way MC	accuracy
	composer_recognition	96	Mutopia	4-way MC	accuracy
	genre_recognition	132	Mutopia	4-way MC	accuracy
	emotion_recognition	120	EMOPIA	4-way MC	accuracy
	error_detection	220	Mutopia†	bar list	macro-F1

† Mutopia scores with five categories of synthetic corruption. The four backbones evaluated are Phi-4, Qwen2.5-Coder-14B, DeepSeek-Coder-V2-Lite and Codestral-22B.

Generation, heard

Each backbone receives the same metadata-conditioned prompt and must emit LilyPond. Below are the raw zero-shot outputs — engraved and sonified straight from the models, nothing cherry-picked bar-by-bar. Quality varies sharply: some models write idiomatic Baroque lines, others drift into ascending scale-runs (a real failure mode the compile/FMD metrics capture).

Generation results

Understanding, probed

Real benchmark questions with each model's greedy answer. Recognition is strong (composer, genre, title); one-shot structural reasoning — counting bars, locating a corrupted bar, placing emotion on the valence axis — collapses.

Understanding results

Citation

@inproceedings{spanio2026lilybench,
  title     = {Can LLMs understand LilyPond? A benchmark for symbolic
               music generation and understanding},
  author    = {Spanio, Matteo and Torabi, Mohammad and
               Poltronieri, Andrea and Rod{\`a}, Antonio},
  booktitle = {Proceedings of the 6th National Conference on Artificial
               Intelligence (Ital-IA 2026)},
  year      = {2026},
  address   = {Rome, Italy},
  publisher = {CEUR-WS},
  eprint    = {2606.08722},
  archivePrefix = {arXiv},
}