TY - JOUR
T1 - Adiabatic Markov Decision Process: Convergence of Value Iteration Algorithm
PB - ASME
AU - Duong, Thai
AU - Nguyen-Huu, Duong
AU - Nguyen, Thinh
Y1 - 2016/04/06
N1 - 10.1115/1.4032875
JO - Journal of Dynamic Systems, Measurement, and Control
SP - 061009
EP - 061009-12
VL - 138
IS - 6
N2 - Markov decision process (MDP) is a well-known framework for devising the optimal decision-making strategies under uncertainty. Typically, the decision maker assumes a stationary environment which is characterized by a time-invariant transition probability matrix. However, in many real-world scenarios, this assumption is not justified, thus the optimal strategy might not provide the expected performance. In this paper, we study the performance of the classic value iteration algorithm for solving an MDP problem under nonstationary environments. Specifically, the nonstationary environment is modeled as a sequence of time-variant transition probability matrices governed by an adiabatic evolution inspired from quantum mechanics. We characterize the performance of the value iteration algorithm subject to the rate of change of the underlying environment. The performance is measured in terms of the convergence rate to the optimal average reward. We show two examples of queuing systems that make use of our analysis framework.
SN - 0022-0434
M3 - doi: 10.1115/1.4032875
UR - http://dx.doi.org/10.1115/1.4032875
ER -