Learning optimal policies from event logs through reinforcement learning: A comparison of deep and MDP-based approaches

Prescriptive Process Monitoring is an emerging area within Process Mining that focuses on recommending actions to optimize business outcomes. Most existing works focus on prescribing pre-defined interventions, that is, pre-defined (sets of) actions, on specific ongoing process executions, aimed at achieving a specific objective or Key Performance Indicator (KPI). In contrast, only a few approaches have explored the learning and evaluation of optimal behavioral policies, that is, general strategies that determine the best sequence of actions that constitute a process execution in order to maximize the desired KPI. In this paper, we address the problem of learning optimal behavioral policies by proposing an AI-based approach that learns an optimal policy directly from historical process executions using Reinforcement Learning (RL), with the goal of recommending the best actions to optimize a KPI of interest. To this end, we employ two distinct RL techniques. The first is a classical, model-based approach that extends previous work by the authors, overcoming its limitations, by constructing a Markov Decision Process (MDP) that captures the process behavior. The second is a model-free technique based on offline Deep RL, a rapidly advancing family of methods that have demonstrated strong performance across a variety of domains. Differently from state of the art work, we aim at building methods that minimize the usage of domain knowledge on the scenario at hand, and learn optimal policies directly from historical event data. In this way, we investigate whether, given a relevant KPI for the process under analysis, it is possible not only to learn when to apply an intervention, but also to discover which interventions are effective directly from data. Moreover, we aim at targeting complex scenarios, such as the ones modeling an interplay with customers or external actors, in which the process owner (and therefore the behavioral policy) may control only part of the process activities. Concerning the evaluation , we adopt an approach already used in the evaluation of pre-defined interventions of exploiting Business Process Simulation (BPS), but we adapt and customize it to the task of evaluating optimal behavioral policies. In particular, we build a data-driven BPS environment, to evaluate the discovered policies. Our results show that both methods consistently improve the targeted KPI with similar effectiveness, with the model-based approach outperforming offline Deep RL in terms of computational efficiency.