Kim So Hyeon, Lee Jee Hang

KIPS Transactions on Software and Data Engineering, Vol. 11, No. 8, pp. 331-338, Aug. 2022
https://doi.org/10.3745/KTSDE.2022.11.8.331,   PDF Download:
Keywords: SR Based Reinforcement Learning Algorithm, 2-Stage Markov Decision Task, State Transition Probability, Reward Function
Abstract

Successor representation (SR) is a model of human reinforcement learning (RL) mimicking the underlying mechanism of hippocampal cells constructing cognitive maps. SR utilizes these learned features to adaptively respond to the frequent reward changes. In this paper, we evaluated the performance of SR under the context where changes in latent variables of environments trigger the reward structure changes. For a benchmark test, we adopted SR-Dyna, an integration of SR into goal-driven Dyna RL algorithm in the 2-stage Markov Decision Task (MDT) in which we can intentionally manipulate the latent variables – state transition uncertainty and goal-condition. To precisely investigate the characteristics of SR, we conducted the experiments while controlling each latent variable that affects the changes in reward structure. Evaluation results showed that SR-Dyna could learn to respond to the reward changes in relation to the changes in latent variables, but could not learn rapidly in that situation. This brings about the necessity to build more robust RL models that can rapidly learn to respond to the frequent changes in the environment in which latent variables and reward structure change at the same time.

Statistics
Cite this article