One of the most important things that constitute reinforcement learning (RL) is the reward function. These functions are context-specific and differ for every problem under RL. But what if RL is deprived of an exclusive external reward function? It may have a new perspective where the learning component in the RL algorithm has no goals at all!

The Element Of Curiosity

What distinguishes between humans and RL algorithms is the aspect of curiosity. Algorithms are not inquisitive about the environment and keep repeating tasks without exploring much. On the other hand, humans are quite fascinated by new things and keep digging more. Integrating this kind of curiosity in RL can present many possibilities in a machine-like atmosphere.

Now AI researchers are examining whether this could be the new reality of RL. First formulated into an actual learning framework by scholars at the University of California, Berkeley, curiosity-driven learning was introduced as a concept of error with curiosity being the error itself so that the learning model slowly realises its actions/tasks. D Pathak and others, who came up with curiosity-driven learning, say that the factor of motivation or curiosity should rather be internally within the learning environment than in the form of extrinsic rewards. In their paper, Curiosity-driven Exploration by Self-supervised Prediction, Pathak and team explain the reason behind this phenomenon.

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse or missing altogether, and it is not possible to construct a shaped reward function. This is a problem as the agent receives reinforcement for updating its policy only if it succeeds in reaching a pre-specified goal state. Hoping to stumble into a goal state by chance (i.e. random exploration) is likely to be futile for all but the simplest of environments.

This problem with traditional RL lies with external rewards, which are either very less or too vague to fit into the scenario. For example, video games that involve a lot of character movements don’t necessarily lead to completing the goal (say, completing a stage or level). Blending all of the learning elements in the form of extrinsic rewards will not only take time but also consume large amounts of memory to run the algorithm.

Do Intrinsic Rewards Let Go Reinforcement?  

In another paper by OpenAI and UC Berkeley, curiosity-driven learning is explored for various instances of intrinsic rewards such as in Atari games, Super Mario Bros., Roboschool Juggling, Roboschool Ant Robot and Multi-agent curiosity in Two-player Pong. Here, these are tested considering a dynamic setup for not just the above video games but also for physics engine simulations and virtual navigation tasks.

For this to happen, a number of considerations are taken for curiosity, that is, intrinsic rewards, to practically drive RL. They are listed below:

• A proximal policy optimisation (PPO) algorithm for a stable RL algorithm
• Normalisation across three dimensions — reward, advantages and observations to further enhance stability in the algorithm’s performance.
• More RL actors for faster training.
• Feature Normalisations  

Now all of these lead to a pure curiosity-based algorithm. Consequently, it keeps on foraying more into newer actions than traditional RL. In fact, the researchers show that in all of the video game instances mentioned earlier, the learning agent does not need rewards (extrinsic) at all. The authors also highlight how this intrinsic ‘curiosity’ present can shape RL along with the computing itself.

Learning without extrinsic rewards or fitness functions has also been studied extensively in the evolutionary computing where it is referred to as ‘novelty search’. There the novelty of an event is often defined as the distance of the event to the nearest neighbour amongst previous events, using some statistics of the event to compute distances. One interesting finding from this literature is that often much more interesting solutions can be found by not solely optimizing for fitness.

Concluding Note

That is why an RL algorithm devised on the basis of curiosity in an environment can go a long way in overcoming situations wherein RL agents have no iota of their interaction and why they are doing it so. Although this is at an early stage of development, if propelled into the RL, it can have a fruitful impact.