Neural-symbolic (NeSy) AI has gained a lot of popularity by enhancing learning models with explicit reasoning capabilities. Both new systems and new benchmarks are constantly introduced and used to evaluate learning and reasoning skills. The large variety of systems and benchmarks, however, makes it difficult to establish a fair comparison among the various frameworks, let alone a unifying set of benchmarking criteria. This paper analyzes the state-of-the-art in benchmarking NeSy systems, studies its limitations, and proposes ways to overcome them. We categorize popular neural-symbolic frameworks into three groups: model-theoretic, proof-theoretic fuzzy, and proof-theoretic probabilistic systems. We show how these three categories have distinct strengths and weaknesses, and how this is reflected in the type of tasks and benchmarks to which they are applied.
Benchmarking in Neuro-Symbolic AI / Manhaeve, Robin; Giannini, Francesco; Ali, Mehdi; Azzolini, Damiano; Bizzarri, Alice; Borghesi, Andrea; Bortolotti, Samuele; De Raedt, Luc; Dhami, Devendra; Diligenti, Michelangelo; Dumančić, Sebastijan; Faltings, Boi; Gentili, Elisabetta; Gerevini, Alfonso; Gori, Marco; Guns, Tias; Homola, Martin; Kersting, Kristian; Lehmann, Jens; Lombardi, Michele; Lorello, Luca; Marconato, Emanuele; Melacci, Stefano; Passerini, Andrea; Paul, Debjit; Riguzzi, Fabrizio; Teso, Stefano; Yorke-Smith, Neil; Lippi, Marco. - 16059 LNAI:(2025), pp. 238-249. ( 4th International Joint Conference on Learning and Reasoning, IJCLR 2024, and 33rd International Conference on Inductive Logic Programming, ILP 2024 chn 2024) [10.1007/978-3-032-09087-4_17].
Benchmarking in Neuro-Symbolic AI
Bortolotti, Samuele;Marconato, Emanuele;Passerini, Andrea;Teso, Stefano;
2025-01-01
Abstract
Neural-symbolic (NeSy) AI has gained a lot of popularity by enhancing learning models with explicit reasoning capabilities. Both new systems and new benchmarks are constantly introduced and used to evaluate learning and reasoning skills. The large variety of systems and benchmarks, however, makes it difficult to establish a fair comparison among the various frameworks, let alone a unifying set of benchmarking criteria. This paper analyzes the state-of-the-art in benchmarking NeSy systems, studies its limitations, and proposes ways to overcome them. We categorize popular neural-symbolic frameworks into three groups: model-theoretic, proof-theoretic fuzzy, and proof-theoretic probabilistic systems. We show how these three categories have distinct strengths and weaknesses, and how this is reflected in the type of tasks and benchmarks to which they are applied.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione
