"This Sounds Unclear": Evaluating ChatGPT capability in translating end-user prompts into ready-to-deploy Python Code

In this paper, we present a study aimed at evaluating how ChatGPT-4 understands end-users’ natural language instructions to express automation rules for smart home applications and how it translates them into Python code ready to be deployed. Our study used 34 natural language instructions written by end users who were asked to automate scenarios presented as visual animations. The results show that ChatGPT-4 can produce coherent and effective code even if the instructions present ambiguities or unclear elements, understanding natural language instructions and autonomously resolving 94% of them. However, the generated code still contains numerous ambiguities that could potentially affect safety and security aspects. Nevertheless, when appropriately prompted, ChatGPT-4 can subsequently identify those ambiguities. This prompts a discussion about prospective interaction paradigms that may significantly improve the immediate usability of the generated code.