Comparing LLM-Based Query Rewriting Strategies Within RAG Pipelines for Domain-Routed Legal Question Answering

We compare three LLM-based query rewriting strategies-multi-query, decomposition, and step-back-within a retrieval-augmented generation (RAG) pipeline for Italian legal question answering, focusing on inheritance and divorce laws collected during CREA2, a European project that employs AI and blockchain to simplify EU civil dispute resolution and improve access to justice. Gemma3 27B handles answer generation, while Llama3.3 70B provides synthetic test data and end-to-end answer evaluation. We assess how closely each strategy’s final output matches ground truth, alongside overall inference latency. Experimental results show that step-back rewriting yields the highest accuracy across both legal domains, likely due to query abstraction and more efficient retrieval. These findings highlight the promise of step-back rewriting in legal question answering, showing potential for expanding its use to more jurisdictions and diverse legal fields.