SELP: A framework for implementing the perception–action loop in an AI agent

Speaker: Bonny Banerjee, Associate Professor, Institute for Intelligent Systems and Department of Electrical and Computer Engineering, University of Memphis

Date: 16 April 2026
YouTube link: https://youtu.be/uGPbWjWBx3g

In this talk, Bonny Banerjee spoke about SELP, a framework for implementing the perception–action loop in an AI agent. An agent is anything that can be viewed as perceiving its environment through sensors acting on that environment through actuators. The function that maps the percept sequence to the action is called the agent function. The goal of AI, at least in the field of AI agents, is to find out a smallish program that maps this percept sequence to the action.

The SELP cycle consists of four functions: (i) Surprise, (ii) Explain, (iii) Learn, and (iv) Predict.

At any point of time, the agent has an expectation from the environment; it is expecting some observations. When the actual observations come in, the agent computes the difference between the expected observations and the actual observations; here, the difference is not just subtraction; it can be complicated forms of difference and that difference gives it a surprise or prediction error.

The agent then tries to explain these errors; in order to explain, it can perform actions; for example, it can sample other agents or it can ask a large language model. This explain function is crucial and the agent can take time to do different actions to explain why the errors occurred. If it has inferred the causes and if it can explain the surprise, then it learns.

Then, using the updated internal model, it predicts what the next observation is going to be. If it cannot explain, it does not have the inferred causes, then no learning occurs, and it continues to predict using the existing model.

Banerjee presented a general-purpose (or foundational) embodied predictive AI agent model and applied it to six different applications, namely (i) detecting unusual objects, actions, and events in surveillance videos; (ii) learning to produce speech similar to a human infant; (iii) learning to generate handwritten numerals and alphabets; (iv) learning to predict intent during physical interaction with other agents; (v) learning to recognise emotion in speech; and (vi) learning when and with whom to communicate.

The model is inherently multimodal, consisting of perceptual and proprioceptive pathways. Thus, it monitors its own body just as it monitors its external environment. It can be easily extended to any number of modalities. It was found that the same model yields accuracy comparable to the state-of-the-art or better on benchmark datasets in all applications and data types. The model is more efficient than the state-of-the-art in each application in terms of model size (number of trainable parameters), data size (% data observed in each glimpse on average to make inference), and training time.

MENU