In this work, we present an automated framework for discrete interpretable prompt optimization by combining Evolutionary Algorithms (EAs) with Large Language Models (LLMs). Building on the recently proposed EvoPrompt methodology, we adopt a Genetic Algorithm (GA) and Differential Evolution (DE) to optimize prompts for vision-language models, specifically focusing on CLIP prompt templates. Our approach leverages the Alpaca LLM to perform evolutionary operations, maintaining prompt coherence while exploring diverse prompt formulations. We evaluate the effectiveness of our framework across three datasets: ImageNet-1k, 102Flowers, and FGVC-Aircraft-2013b, using a similarity-based fitness function. Our experimental results demonstrate that larger population sizes and specific selection strategies enhance prompt optimization. This work highlights the potential of combining evolutionary search with LLMs to improve automated prompt engineering while preserving interpretability.

CLIP Prompt Optimization with Evolutionary-Driven LLMs / Ceron, A., Donà, I., Iacca, G.. - 16525:(2026), pp. 167-182. (29th European Conference on Applications of Evolutionary Computation, EvoApplications 2026, held as part of EvoStar 2026 Toulouse 8th April-10th April 2026) [10.1007/978-3-032-23607-4_11].

CLIP Prompt Optimization with Evolutionary-Driven LLMs

Giovanni Iacca
2026-01-01

Abstract

In this work, we present an automated framework for discrete interpretable prompt optimization by combining Evolutionary Algorithms (EAs) with Large Language Models (LLMs). Building on the recently proposed EvoPrompt methodology, we adopt a Genetic Algorithm (GA) and Differential Evolution (DE) to optimize prompts for vision-language models, specifically focusing on CLIP prompt templates. Our approach leverages the Alpaca LLM to perform evolutionary operations, maintaining prompt coherence while exploring diverse prompt formulations. We evaluate the effectiveness of our framework across three datasets: ImageNet-1k, 102Flowers, and FGVC-Aircraft-2013b, using a similarity-based fitness function. Our experimental results demonstrate that larger population sizes and specific selection strategies enhance prompt optimization. This work highlights the potential of combining evolutionary search with LLMs to improve automated prompt engineering while preserving interpretability.
2026
International Conference on the Applications of Evolutionary Computation (Part of EvoStar)
Cham, Switzerland
Springer
9783032236067
9783032236074
Ceron, Andrea; Donà, Ivan; Iacca, Giovanni
CLIP Prompt Optimization with Evolutionary-Driven LLMs / Ceron, A., Donà, I., Iacca, G.. - 16525:(2026), pp. 167-182. (29th European Conference on Applications of Evolutionary Computation, EvoApplications 2026, held as part of EvoStar 2026 Toulouse 8th April-10th April 2026) [10.1007/978-3-032-23607-4_11].
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/487213
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 1
  • OpenAlex 1
social impact