0
Journal of Dynamic Systems, Measurement, and Control | Volume 131 | Issue 4 | Research Paper
Research Papers

Convergence Properties of a Computational Learning Model for Unknown Markov Chains

[+] Author and Article Information
Andreas A. Malikopoulos

Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109amaliko@umich.edu

J. Dyn. Sys., Meas., Control 131(4), 041011 (May 20, 2009) (7 pages) doi:10.1115/1.3117202 History: Received March 18, 2008; Revised February 04, 2009; Published May 20, 2009
Article
References
Figures
Citing Articles

The increasing complexity of engineering systems has motivated continuing research on computational learning methods toward making autonomous intelligent systems that can learn how to improve their performance over time while interacting with their environment. These systems need not only to sense their environment, but also to integrate information from the environment into all decision-makings. The evolution of such systems is modeled as an unknown controlled Markov chain. In a previous research, the predictive optimal decision-making (POD) model was developed, aiming to learn in real time the unknown transition probabilities and associated costs over a varying finite time horizon. In this paper, the convergence of the POD to the stationary distribution of a Markov chain is proven, thus establishing the POD as a robust model for making autonomous intelligent systems. This paper provides the conditions that the POD can be valid, and be an interpretation of its underlying structure.
FIGURES IN THIS ARTICLE
<>
Copyright © 2009 by American Society of Mechanical Engineers
Topics: Chain , Probability
Your Session has timed out. Please sign back in to continue.

References

1
Bertsekas, D. P., and Tsitsiklis, J. N., 1996, "Neuro-Dynamic Programming" (Optimization and Neural Computation Series Vol. 3 ), 1st ed., Athena Scientific, Nashua, NH.
2
Sutton, R. S., and Barto, A. G., 1998, "Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)", MIT, Cambridge, MA.
3
Borkar, V. S., 2000, “A Learning Algorithm for Discrete-Time Stochastic Control,” Probability in the Engineering and Informational Sciences, 14 , pp. 243–258.  [CrossRef]
4
Kaelbling, L. P., Littman, M. L., and Moore, A. W., 1996, “Reinforcement Learning: A Survey,” J. Artif. Intell. Res., 4 , pp. 237–285.
5
Mandl, P., 1974, “Estimation and Control in Markov Chains,” Adv. Appl. Probab., 6 , pp. 40–60.  [CrossRef]
6
Borkar, V., and Varaiya, P., 1979, “Adaptive Control of Markov Chains. I. Finite Parameter Set,” IEEE Trans. Autom. Control, AC-24 , pp. 953–957.  [CrossRef]
7
Borkar, V., and Varaiya, P., 1982, “Identification and Adaptive Control of Markov Chains,” SIAM J. Control Optim., 20 , pp. 470–489.  [CrossRef]
8
Kumar, P. R., 1982, “Adaptive Control With a Compact Parameter Set,” SIAM J. Control Optim., 20 , pp. 9–13.  [CrossRef]
9
Doshi, B., and Shreve, S. E., 1980, “Strong Consistency of a Modified Maximum Likelihood Estimator for Controlled Markov Chains,” J. Appl. Probab., 17 , pp. 726–734.  [CrossRef]
10
Kumar, P. R., and Becker, A., 1982, “A New Family of Optimal Adaptive Controllers for Markov Chains,” IEEE Trans. Autom. Control, AC-27 , pp. 137–146.  [CrossRef]
11
Kumar, P. R., and Lin, W., 1982, “Optimal Adaptive Controllers for Unknown Markov Chains,” IEEE Trans. Autom. Control, AC-27 , pp. 765–774.  [CrossRef]
12
Sato, M., Abe, K., and Takeda, H., 1982, “Learning Control of Finite Markov Chains With Unknown Transition Probabilities,” IEEE Trans. Autom. Control, AC-27 , pp. 502–505.  [CrossRef]
13
Sato, M., Abe, K., and Takeda, H., 1985, “An Asymptotically Optimal Learning Controller for Finite Markov Chains With Unknown Transition Probabilities,” IEEE Trans. Autom. Control, AC-30 , pp. 1147–1149.  [CrossRef]
14
Sato, M., Abe, K., and Takeda, H., 1988, “Learning Control of Finite Markov Chains With an Explicit Trade-Off Between Estimation and Control,” IEEE Trans. Syst. Man Cybern., 18 , pp. 677–684.  [CrossRef]
15
Kumar, P. R., 1985, “A Survey of Some Results in Stochastic Adaptive Control,” SIAM J. Control Optim., 23 , pp. 329–380.  [CrossRef]
16
Varaiya, P., 1982, “Adaptive Control of Markov Chains: A Survey,” Proceedings of the IFAC Symposium , New Delhi, India, pp. 89–93.
17
Agrawal, R., and Teneketzis, D., 1989, “Certainty Equivalence Control With Forcing: Revisited,” Proceedings of the IEEE Conference on Decision and Control Including the Symposium on Adaptive Processes , Tampa, FL, p. 2107.
18
Malikopoulos, A. A., 2008, “Real-Time, Self-Learning Identification and Stochastic Optimal Control of Advanced Powertrain Systems,” Ph.D. thesis, Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI.
19
Malikopoulos, A. A., Papalambros, P. Y., and Assanis, D. N., 2007, “A State-Space Representation Model and Learning Algorithm for Real-Time Decision-Making Under Uncertainty,” Proceedings of the 2007 ASME International Mechanical Engineering Congress and Exposition , Seattle, WA, Nov. 11–15.
20
Malikopoulos, A. A., Papalambros, P. Y., and Assanis, D. N., 2007, “A Learning Algorithm for Optimal Internal Combustion Engine Calibration in Real Time,” Proceedings of the ASME 2007 International Design Engineering Technical Conferences Computers and Information in Engineering Conference , Las Vegas, NV, Sept. 4–7.
21
Malikopoulos, A. A., Assanis, D. N., and Papalambros, P. Y., 2007, “Real-Time, Self-Learning Optimization of Diesel Engine Calibration,” Proceedings of the 2007 Fall Technical Conference of the ASME Internal Combustion Engine Division , Charleston, SC, Oct. 14–17.
22
Malikopoulos, A. A., Assanis, D. N., and Papalambros, P. Y., 2008, “Optimal Engine Calibration for Individual Driving Styles,” Proceedings of the SAE 2008 World Congress and Exhibition , Detroit, MI, Apr. 14–17, SAE Paper No. 2008-01-1367.
23
Kumar, P. R., and Varaiya, P., 1986, "Stochastic Systems", Prentice-Hall, Englewood Cliffs, NJ.
24
Bertsekas, D. P., 2001, "Dynamic Programming and Optimal Control (Volumes 1 and 2)" (Optimization and Neural Computation Series ), 1st ed., Athena Scientific, Nashua, NH.
25
Kemeny, J. G., and Snell, J. L., 1983, "Finite Markov Chains", 1st ed., Springer, New York.
26
Krishnan, V., 2006, "Probability and Random Processes", 1st ed., Wiley, New York.
27
Gubner, J. A., 2006, Probability and Random Processes for Electrical and Computer Engineers, 1st ed., Cambridge University Press, Cambridge.
28
Grimmett, G. R., and Stirzaker, D. R., 2001, "Probability and Random Processes", 3rd ed., Oxford University Press, New York.
29
Gosavi, A., 2003, "Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning", 1st ed., Springer, New York.

Figures

Grahic Jump Location
+Stochastic system model schematic
View Large  |  Save Figure  |  Download Slide (.ppt)
Stochastic system model schematic
Figure 1

Stochastic system model schematic

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
An Algorithm for the Quantification of Hybrid Causal Logic Models (PSAM-0339)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
Evaluation of Terrorist Risk Using Belief and Plausibility (PSAM-0384)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
QRAS Approach to Phased Mission Analysis (PSAM-0444)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
Pre-Accidental Situations Highlighted by RECUPERARE Method and Data (PSAM-0029)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
Space Propulsion System Phased-Mission Probability Analysis Using Conventional PRA Methods (PSAM-0252)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In