0
Journal of Dynamic Systems, Measurement, and Control | Volume 129 | Issue 5 | TECHNICAL PAPERS
TECHNICAL PAPERS

Decentralized Coordinated Motion Control of Two Hydraulic Actuators Handling a Common Object

[+] Author and Article Information
Mark Karpenko

Department of Mechanical and Manufacturing Engineering, University of Manitoba, Winnipeg, Manitoba, R3T 5V6, Canada

Nariman Sepehri1

Department of Mechanical and Manufacturing Engineering, University of Manitoba, Winnipeg, Manitoba, R3T 5V6, Canadanariman@cc.umanitoba.ca

John Anderson

Department of Computer Science, University of Manitoba, Winnipeg, Manitoba, R3T 5V6, Canada

1

Corresponding author.

J. Dyn. Sys., Meas., Control 129(5), 729-741 (Jan 25, 2007) (13 pages) doi:10.1115/1.2764516 History: Received June 23, 2006; Revised January 25, 2007
Article
References
Figures
Tables
Citing Articles

In this paper, reinforcement learning is applied to coordinate, in a decentralized fashion, the motions of a pair of hydraulic actuators whose task is to firmly hold and move an object along a specified trajectory under conventional position control. The learning goal is to reduce the interaction forces acting on the object that arise due to inevitable positioning errors resulting from the imperfect closed-loop actuator dynamics. Each actuator is therefore outfitted with a reinforcement learning neural network that modifies a centrally planned formation constrained position trajectory in response to the locally measured interaction force. It is shown that the actuators, which form a multiagent learning system, can learn decentralized control strategies that reduce the object interaction forces and thus greatly improve their coordination on the manipulation task. However, the problem of credit assignment, a common difficulty in multiagent learning systems, prevents the actuators from learning control strategies where each actuator contributes equally to reducing the interaction force. This problem is resolved in this paper via the periodic communication of limited local state information between the reinforcement learning actuators. Using both simulations and experiments, this paper examines some of the issues pertaining to learning in dynamic multiagent environments and establishes reinforcement learning as a potential technique for coordinating several nonlinear hydraulic manipulators performing a common task.
Copyright © 2007 by American Society of Mechanical Engineers
Your Session has timed out. Please sign back in to continue.

References

1
Vuckbrotovic, M., and Tuneski, A. I., 1998, “Mathematical Model of Multiple Manipulators: Cooperative Compliant Manipulation on Dynamical Environments,” Mech. Mach. Theory  [CrossRef], 33 , pp. 1211–1239.
2
Braun, B. M., Starr, G. P., Wood, J. E., and Lumia, R., 2004, “A Framework for Implementing Cooperative Motion on Industrial Controllers,” IEEE Trans. Rob. Autom.  [CrossRef], 20 , pp. 583–589.
3
Arimoto, S., Miyazaki, F., and Kawamura, S., 1987, “Cooperative Motion Control of Multiple Robot Arms or Fingers,” "Proceedings of the 1987 IEEE International Conference on Robotics and Automation", Raleigh, NC, pp. 1407–1412.
4
Kopf, C. D., and Yabuta, T., 1988, “Experimental Comparison of Master∕Slave and Hybrid Two Arm Position∕Force Control,” "Proceedings of the 1988 IEEE International Conference on Robotics and Automation", Philadelphia, PA, pp. 1633–1637.
5
Raibert, M. H., and Craig, J. J., 1981, “Hybrid Position∕Force Control of Manipulators,” ASME J. Dyn. Syst., Meas., Control, 102 , pp. 126–133.
6
Yoshikawa, T., and Zheng, X.-Z., 1993, “Coordinated Dynamic Hybrid Position∕Force Control for Multiple Robot Manipulators Handling One Constrained Object,” Int. J. Robot. Res.  [CrossRef], 12 (3), pp. 219–230.
7
Perdereau, V., and Drouin, M., 1996, “Hybrid External Contol for Two Robot Coordinated Motion,” Robotica, 14 , pp. 141–153.
8
Liu, Y.-H., and Arimoto, S., 1996, “Distributively Controlling Two Robots Handling an Object in the Task Space Without any Communicaiton,” IEEE Trans. Autom. Control  [CrossRef], 41 (8), pp. 1193–1198.
9
Uzmay, I., Burkan, R., and Sarikaya, H., 2004, “Application of Robust and Adaptive Control Techniques to Cooperative Manipulation,” Control Eng. Pract.  [CrossRef], 12 , pp. 139–148.
10
Woon, L. C., Ge, S. S., Chen, X. Q., and Zhang, C., 1999, “Adaptive Neural Network Control of Coordinated Manipulators,” J. Rob. Syst.  [CrossRef], 16 (4), pp. 195–211.
11
Nakayama, T., Arimoto, S., and Naniwa, T., 1995, “Coordinated Learning Control for Multiple Manipulators Holding an Object Rigidly,” "Proceedings of the 1995 IEEE International Conference on Robotics and Automation", Nagoya, Japan, 2 , pp. 1529–1534.
12
Schneider, S. A., and Cannon, R. H., 1992, “Object Impedance Control for Cooperative Manipulation: Theory and Experimental Results,” IEEE Trans. Rob. Autom.  [CrossRef], 8 (3), pp. 383–394.
13
Zeng, H., and Sepehri, N., 2005, “Nonlinear Position Control of Cooperative Hydraulic Manipulators Handling Unknown Payloads,” Int. J. Control  [CrossRef], 78 (3), pp. 196–207.
14
Mulder, M. C., and Malladi, S. R., 1991, “A Minimum Effort Control Algorithm for a Cooperating Sensor Driven Intelligent Multi-Jointed Robotic Arm,” "Proceedings of the 30th IEEE Conference on Decision and Control", Brighton, UK, 2 , pp. 1573–1578.
15
Zeng, H., and Sepehri, N., 2007, “On Tracking Control of Cooperative Hydraulic Manipulators,” Int. J. Control  [CrossRef], 80 (3), pp. 454–469.
16
Sutton, R. S., and Barto, A., 1998, "Reinforcement Learning: An Introduction", The MIT Press, Cambridge, MA.
17
Kaebling, L. P., Littman, M. L., and Moore, A. W., 1996, “Reinforcement Learning: A Survey,” J. Artif. Intell. Res., 4 , pp. 237–285.
18
Sutton, R. S., 1984, “Temporal Credit Assignment in Reinforcement Learning,” Ph.D. thesis, University of Massachusetts, Amherst.
19
Russell, S., and Norvig, P., 1995, "Artificial Intelligence: A Modern Approach", Prentice-Hall, Englewood Cliffs, NJ.
20
Watkins, C. J. C. H., 1989, “Learning With Delayed Rewards,” Ph.D. thesis, Cambridge University, Cambridge.
21
Williams, R. J., 1992, “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning,” Mach. Learn., 8 (3), pp. 229–256.
22
Barto, A., Sutton, R. S., and Anderson, C. W., 1983, “Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems,” IEEE Trans. Syst. Man Cybern., 13 , pp. 834–846.
23
Anderson, C. W., 1989, “Learning to Control an Inverted Pendulum Using Neural Networks,” IEEE Control Syst. Mag.  [CrossRef], 9 (3), pp. 31–37.
24
Gullapalli, V., Franklin, J. A., and Benbrahim, H., 1994, “Acquiring Robot Skills via Reinforcement Learning,” IEEE Control Syst. Mag.  [CrossRef], 14 , pp. 13–24.
25
Tzasfestas, S. G., and Rigatos, G. G., 2002, “Fuzzy Reinforcement Learning Control for Compliance Tasks of Robotic Manipulators,” IEEE Trans. Syst., Man, Cybern., Part B: Cybern.  [CrossRef], 32 , pp. 107–113.
26
Stone, P., and Veloso, M., 2000, “Multiagent Systems: A Survey From a Machine Learning Perspective,” Auton. Rob., 8 , pp. 345–383.
27
Gullapalli, V., 1990, “Stochastic Reinforcement Learning Algorithm for Learning Real-Valued Functions,” Neural Networks  [CrossRef], 3 (6), pp. 671–692.
28
Mataric, M. J., 1998, “Using Communication to Reduce Locality in Distributed Multi-Agent Learning,” J. Exp. Theor. Artif. Intell., 10 (3), pp. 357–369.
29
Merritt, H., 1967, "Hydraulic Control Systems", Wiley, New York.
30
Karpenko, M., and Sepehri, N., 2003, “Robust Position Control of an Electrohydraulic Actuator With a Faulty Actuator Piston Seal,” ASME J. Dyn. Syst., Meas., Control  [CrossRef], 125 (3), pp. 413–423.
31
Bishop, C. M., 1995, "Neural Networks for Pattern Recognition", Oxford University Press, New York.

Figures

Grahic Jump Location
+Benchmark experimental performance on test trajectory with starting and stopping motions: (a) desired position; (b) measured force
View Large  |  Save Figure  |  Download Slide (.ppt)
Benchmark experimental performance on test trajectory with starting and stopping motions: (a) desired position; (b) measured force
Figure 14

Benchmark experimental performance on test trajectory with starting and stopping motions: (a) desired position; (b) measured force

Grahic Jump Location
+Typical experimental performance with communication and increased object stiffness: (a) measured force; (b) trajectory corrections Δx1 and Δx2. Legend: (1) learning; (2) after learning.
View Large  |  Save Figure  |  Download Slide (.ppt)
Typical experimental performance with communication and increased object stiffness: (a) measured force; (b) trajectory corrections Δx1 and Δx2. Legend: (1) learning; (2) after learning.
Figure 13

Typical experimental performance with communication and increased object stiffness: (a) measured force; (b) trajectory corrections Δx1 and Δx2. Legend: (1) learning; (2) after learning.

Grahic Jump Location
+Typical experimental performance on coordination task with communication: (a) measured force; (b) trajectory corrections Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) learning; (2) after learning.
View Large  |  Save Figure  |  Download Slide (.ppt)
Typical experimental performance on coordination task with communication: (a) measured force; (b) trajectory corrections Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) learning; (2) after learning.
Figure 12

Typical experimental performance on coordination task with communication: (a) measured force; (b) trajectory corrections Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) learning; (2) after learning.

Grahic Jump Location
+Typical experimental performance of noncommunicating agents on coordination task: (a) measured force; (b) trajectory corrections Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) learning; (2) after learning.
View Large  |  Save Figure  |  Download Slide (.ppt)
Typical experimental performance of noncommunicating agents on coordination task: (a) measured force; (b) trajectory corrections Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) learning; (2) after learning.
Figure 11

Typical experimental performance of noncommunicating agents on coordination task: (a) measured force; (b) trajectory corrections Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) learning; (2) after learning.

Grahic Jump Location
+Benchmark experimental performance on sinusoidal trajectory 22 before learning: (a) measured force; (b) position trajectory errors, x1d−x1 and x2d−x2
View Large  |  Save Figure  |  Download Slide (.ppt)
Benchmark experimental performance on sinusoidal trajectory 22 before learning: (a) measured force; (b) position trajectory errors, x1d−x1 and x2d−x2
Figure 10

Benchmark experimental performance on sinusoidal trajectory 22 before learning: (a) measured force; (b) position trajectory errors, x1d−x1 and x2d−x2

Grahic Jump Location
+Photograph of test rig upon which experiments were carried out. Inset: Close-up view of deformable object and load cell.
View Large  |  Save Figure  |  Download Slide (.ppt)
Photograph of test rig upon which experiments were carried out. Inset: Close-up view of deformable object and load cell.
Figure 9

Photograph of test rig upon which experiments were carried out. Inset: Close-up view of deformable object and load cell.

Grahic Jump Location
+Actuator 1 trajectory correction ratios after learning using communication scheme
View Large  |  Save Figure  |  Download Slide (.ppt)
Actuator 1 trajectory correction ratios after learning using communication scheme
Figure 8

Actuator 1 trajectory correction ratios after learning using communication scheme

Grahic Jump Location
+Typical performance on coordination task with communication: (a) interaction force; (b) trajectory corrections, Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) before learning; (2) learning; (3) after learning.
View Large  |  Save Figure  |  Download Slide (.ppt)
Typical performance on coordination task with communication: (a) interaction force; (b) trajectory corrections, Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) before learning; (2) learning; (3) after learning.
Figure 7

Typical performance on coordination task with communication: (a) interaction force; (b) trajectory corrections, Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) before learning; (2) learning; (3) after learning.

Grahic Jump Location
+Performance of noncommunicating hydraulic actuators on coordination task after learning: (a) rms interaction force; (b) actuator 1 trajectory correction ratio
View Large  |  Save Figure  |  Download Slide (.ppt)
Performance of noncommunicating hydraulic actuators on coordination task after learning: (a) rms interaction force; (b) actuator 1 trajectory correction ratio
Figure 6

Performance of noncommunicating hydraulic actuators on coordination task after learning: (a) rms interaction force; (b) actuator 1 trajectory correction ratio

Grahic Jump Location
+Typical performance of noncommunicating hydraulic actuators on coordination task: (a) interaction force; (b) trajectory corrections, Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) before learning; (2) learning; (3) after learning.
View Large  |  Save Figure  |  Download Slide (.ppt)
Typical performance of noncommunicating hydraulic actuators on coordination task: (a) interaction force; (b) trajectory corrections, Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) before learning; (2) learning; (3) after learning.
Figure 5

Typical performance of noncommunicating hydraulic actuators on coordination task: (a) interaction force; (b) trajectory corrections, Δx1 and Δx2; (c) position trajectory errors, x1d−x1 and x2d−x2. Legend: (1) before learning; (2) learning; (3) after learning.

Grahic Jump Location
+Reinforcement function for evaluating actuator performance
View Large  |  Save Figure  |  Download Slide (.ppt)
Reinforcement function for evaluating actuator performance
Figure 4

Reinforcement function for evaluating actuator performance

Grahic Jump Location
+Schematic of coordinated positioning system: (a) decentralized control scheme using localized position and RLNN controllers; (b) architecture of local reinforcement learning control system showing signals used for learning as dashed lines
View Large  |  Save Figure  |  Download Slide (.ppt)
Schematic of coordinated positioning system: (a) decentralized control scheme using localized position and RLNN controllers; (b) architecture of local reinforcement learning control system showing signals used for learning as dashed lines
Figure 3

Schematic of coordinated positioning system: (a) decentralized control scheme using localized position and RLNN controllers; (b) architecture of local reinforcement learning control system showing signals used for learning as dashed lines

Grahic Jump Location
+Schematic of coordinated positioning task: (a) definition of reference trajectories x1d and x2d with respect to desired object position xd; (b) deformation of object due to relative actuator positioning error
View Large  |  Save Figure  |  Download Slide (.ppt)
Schematic of coordinated positioning task: (a) definition of reference trajectories x1d and x2d with respect to desired object position xd; (b) deformation of object due to relative actuator positioning error
Figure 2

Schematic of coordinated positioning task: (a) definition of reference trajectories x1d and x2d with respect to desired object position xd; (b) deformation of object due to relative actuator positioning error

Grahic Jump Location
+Schematic of a typical hydraulic actuator for mathematical modeling
View Large  |  Save Figure  |  Download Slide (.ppt)
Schematic of a typical hydraulic actuator for mathematical modeling
Figure 1

Schematic of a typical hydraulic actuator for mathematical modeling

Grahic Jump Location
+Typical experimental performance for trajectory following test task with communication: (a) measured force; (b) trajectory corrections Δx1 and Δx2. Legend: (1) after pretraining on sinusoidal trajectory for 30s; (2) additional learning.
View Large  |  Save Figure  |  Download Slide (.ppt)
Typical experimental performance for trajectory following test task with communication: (a) measured force; (b) trajectory corrections Δx1 and Δx2. Legend: (1) after pretraining on sinusoidal trajectory for 30s; (2) additional learning.
Figure 15

Typical experimental performance for trajectory following test task with communication: (a) measured force; (b) trajectory corrections Δx1 and Δx2. Legend: (1) after pretraining on sinusoidal trajectory for 30s; (2) additional learning.

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
On-Line a Predictive Model of Cutting Force in Turning with 3 Axis Acceleration Transducer Using Neural Network
International Conference on Advanced Computer Theory and Engineering (ICACTE 2009)
Comparisons of Gaussian ARTMAP and Distributed Gaussian ARTMAP Classifiers: The Category Proliferation Problem
Intelligent Engineering Systems through Artificial Neural Networks, Volume 16
A Novel Approach for LFC and AVR of an Autonomous Power Generating System
International Conference on Mechanical Engineering and Technology (ICMET-London 2011)
A New Exploratory Neural Network Training Method
Intelligent Engineering Systems through Artificial Neural Networks, Volume 16
Fitting a Function and Its Derivative
Intelligent Engineering Systems Through Artificial Neural Networks, Volume 17
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In