In this work, we present an automated framework for discrete interpretable prompt optimization by combining Evolutionary Algorithms (EAs) with Large Language Models (LLMs). Building on the recently proposed EvoPrompt methodology, we adopt a Genetic Algorithm (GA) and Differential Evolution (DE) to optimize prompts for vision-language models, specifically focusing on CLIP prompt templates. Our approach leverages the Alpaca LLM to perform evolutionary operations, maintaining prompt coherence while exploring diverse prompt formulations. We evaluate the effectiveness of our framework across three datasets: ImageNet-1k, 102Flowers, and FGVC-Aircraft-2013b, using a similarity-based fitness function. Our experimental results demonstrate that larger population sizes and specific selection strategies enhance prompt optimization. This work highlights the potential of combining evolutionary search with LLMs to improve automated prompt engineering while preserving interpretability.
CLIP Prompt Optimization with Evolutionary-Driven LLMs / Ceron, A., Donà, I., Iacca, G.. - 16525:(2026), pp. 167-182. (29th European Conference on Applications of Evolutionary Computation, EvoApplications 2026, held as part of EvoStar 2026 Toulouse 8th April-10th April 2026) [10.1007/978-3-032-23607-4_11].
CLIP Prompt Optimization with Evolutionary-Driven LLMs
Giovanni Iacca
2026-01-01
Abstract
In this work, we present an automated framework for discrete interpretable prompt optimization by combining Evolutionary Algorithms (EAs) with Large Language Models (LLMs). Building on the recently proposed EvoPrompt methodology, we adopt a Genetic Algorithm (GA) and Differential Evolution (DE) to optimize prompts for vision-language models, specifically focusing on CLIP prompt templates. Our approach leverages the Alpaca LLM to perform evolutionary operations, maintaining prompt coherence while exploring diverse prompt formulations. We evaluate the effectiveness of our framework across three datasets: ImageNet-1k, 102Flowers, and FGVC-Aircraft-2013b, using a similarity-based fitness function. Our experimental results demonstrate that larger population sizes and specific selection strategies enhance prompt optimization. This work highlights the potential of combining evolutionary search with LLMs to improve automated prompt engineering while preserving interpretability.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione



