REWARD-GUIDED SYNTHETIC PREFERENCE DATA

Mitigating Hallucination in Multimodal LLMs
with Reward-Guided Synthetic Data

Robert WijayaNgoc-Bao NguyenNgai-Man Cheung

Singapore University of Technology and Design

Paper (under review) Code Dataset Model weights

Synthetic images. Reward-ranked responses. Preference-based post-training.

Abstract

Multimodal language models can generate fluent answers that contain details unsupported by an image. SynthAlign investigates reward-guided synthetic preference data for mitigating these hallucinations through direct preference optimization (DPO). The pipeline selects synthetic images with ImageReward, generates candidate responses with open-source multimodal models, and uses ArmoRM to construct chosen–rejected response pairs. Evaluations examine hallucination and general visual question answering, alongside analyses of the resulting data and the limitations of reward-based supervision.

THE FRAMEWORK

From synthetic images to preference pairs

Two reward-guided selection stages, followed by DPO post-training.

SynthAlign pipeline: ImageReward selects a synthetic image using its generation prompt; multimodal models generate candidate responses; ArmoRM ranks question–answer pairs; chosen and rejected responses are used for DPO.
Overview of SynthAlign. The selected image, question, and response pair form a multimodal preference example. Click any figure to view full resolution.
  1. 1

    Image rating & ranking

    Generate candidate images and select the highest ImageReward score, conditioned on the text-to-image prompt.

    Rimg(image, prompt)
  2. 2

    Response rating & ranking

    Generate image-conditioned answers with four multimodal models. ArmoRM ranks the question–answer pairs.

    Rresp(question, answer)
  3. 3

    DPO post-training

    Use higher- and lower-scoring responses as chosen and rejected answers to post-train the multimodal backbone.

    (image, question, y+, y−)

BENCHMARKS

Hallucination and visual question answering

Results reported in the revised manuscript, separated by backbone.

These tables reproduce the manuscript snapshot of 26 September 2026, with Qwen2-VL training sizes corrected to 2K. They do not include the ongoing full-9K A–F ablation runs. Published reference systems are omitted here to avoid treating cross-paper results as controlled comparisons.

MMHal-Bench score ↑

2.363.49

LLaVA-1.5-7B · vanilla → DPO-9K
MMHal hallucination rate ↓

51.0%25.0%

A 26.0 percentage-point reduction
POPE accuracy ↑

86.1%87.6%

A 1.5 percentage-point increase

Hallucination benchmarks

Bold indicates the best reported value within each backbone block, including ties. ↑ higher is better; ↓ lower is better.

ModelMMHal-BenchAMBERPOPE
Score ↑Hall. ↓CHAIR ↓Cog ↓Acc. ↑Precision ↑
LLaVA-1.5-7B
Vanilla2.3651.07.74.286.1%89.1%
+ SynthAlign-DPO-9K3.4925.06.43.187.6%97.8%
Qwen2-VL-7B
Vanilla4.118.76.85.988.3%93.0%
+ SynthAlign-SFT-2K3.924.05.32.988.5%93.0%
+ SynthAlign-DPO-2K4.714.95.83.188.4%93.0%

Source: revised manuscript, Table 1. LLaVA uses 9K preference pairs; Qwen2-VL uses 2K training examples. Results across these blocks do not constitute a data-budget-matched comparison.

General VQA benchmarks · LLaVA-1.5-7B

ModelScienceQA ↑MME ↑VizWiz ↑MMMU ↑SEED-img ↑MM-Vet ↑
Vanilla66.8355.750.035.966.131.1
+ SynthAlign-DPO-9K69.0367.555.336.166.334.1
+ SynthAlign-DPO-12K69.4378.255.236.266.333.4

Source: revised manuscript, Table 2. All rows use the Vicuna-7B language backbone. Metrics retain the manuscript’s reporting scale; no cross-benchmark average is computed.

INSIDE THE DATA

Inspect the preferences, not just the scores

Synthetic image candidates, response pairs, and dataset-level diagnostics.

Two synthetic image–question examples, a ceramic vase and a metro train, with reward-selected preferred and dispreferred responses shown side by side.
Examples of reward-selected response pairs from the manuscript. “Preferred” and “dispreferred” denote reward-model labels, not human-verified correctness.

How image candidates are selected

Four prompt groups with four Stable Diffusion candidates at guidance scales 5, 7, 9, and 11. ImageReward scores and the highest-scoring selected image are marked.
ImageReward compares candidates generated from the same text-to-image prompt. The highest-scoring image is selected within each group.

Prompt diversity

Heatmap of 15 categories across direct visual, interpretive visual, commonsense/advice, and mixed prompt classes. Mixed and commonsense prompts dominate many categories.
Prompt types vary across categories. Many require interpretation or external knowledge as well as visual information, so improvements need not reflect visual grounding alone.

Response length

Chosen and rejected response word-length histograms. Chosen median is 127 words, rejected median is 95 words; rejected responses also have a longer right tail.
Chosen and rejected responses differ in length. This motivates length-controlled analysis; the observed association alone does not prove that length causes reward preference.

QUALITATIVE EXAMPLE

What changes in the response?

A side-by-side example from the manuscript.

Vanilla LLaVA-1.5-7B and SynthAlign-trained LLaVA compare two surfboards. The SynthAlign response more accurately describes the predominantly white and red boards, but still adds interpretation about their owners.
LLaVA-1.5-7B before and after SynthAlign post-training.
QUESTION

“Compare the two surfboards in the photo.”

The aligned model describes the boards’ dominant colors more accurately in this example.

However, its interpretation of the owners’ preferences is not established by the image. The example illustrates a change in response quality, not the elimination of hallucination.

Read the paper for the full study