In this paper, we develop a unified framework to address the problem of optimal nonlinear analysis and feedback control for partial stability and partial-state stabilization of stochastic dynamical systems. Partial asymptotic stability in probability of the closed-loop nonlinear system is guaranteed by means of a Lyapunov function that is positive definite and decrescent with respect to part of the system state which can clearly be seen to be the solution to the steady-state form of the stochastic Hamilton–Jacobi–Bellman equation, and hence, guaranteeing both partial stability in probability and optimality. The overall framework provides the foundation for extending optimal linear-quadratic stochastic controller synthesis to nonlinear-nonquadratic optimal partial-state stochastic stabilization. Connections to optimal linear and nonlinear regulation for linear and nonlinear time-varying stochastic systems with quadratic and nonlinear-nonquadratic cost functionals are also provided. Finally, we also develop optimal feedback controllers for affine stochastic nonlinear systems using an inverse optimality framework tailored to the partial-state stochastic stabilization problem and use this result to address polynomial and multilinear forms in the performance criterion.

# Partial-State Stabilization and Optimal Feedback Control for Stochastic Dynamical Systems PUBLIC ACCESS

**Tanmay Rajpurohit**

**Wassim M. Haddad**

Contributed by the Dynamic Systems Division of ASME for publication in the JOURNAL OF DYNAMIC SYSTEMS, MEASUREMENT, AND CONTROL. Manuscript received November 30, 2015; final manuscript received March 6, 2017; published online June 5, 2017. Assoc. Editor: Suman Chakravorty.

*J. Dyn. Sys., Meas., Control*139(9), 091001 (Jun 05, 2017) (18 pages) Paper No: DS-15-1602; doi: 10.1115/1.4036033 History: Received November 30, 2015; Revised March 06, 2017

In Ref. [1], we extended the framework developed in Refs. [2,3] to address the problem of *optimal partial-state stabilization*, wherein stabilization with respect to a subset of the system state variables is desired. Partial-state stabilization arises in many engineering applications [4,5]. Specifically, in spacecraft stabilization via gimballed gyroscopes, asymptotic stability of an equilibrium position of the spacecraft is sought while requiring Lyapunov stability of the axis of the gyroscope relative to the spacecraft [5]. Alternatively, in the control of rotating machinery with mass imbalance, spin stabilization about a nonprincipal axis of inertia requires motion stabilization with respect to a subspace instead of the origin [4]. The most common application where partial stabilization is necessary is adaptive control, wherein asymptotic stability of the closed-loop plant states is guaranteed without necessarily achieving parameter error convergence.

In this paper, we extend the framework developed in Ref. [1] to address the problem of optimal partial-state *stochastic* stabilization. Specifically, we consider a notion of optimality that is directly related to a given Lyapunov function that is positive definite and decrescent with respect to part of the system state. In particular, an optimal partial-state stochastic stabilization control problem is stated, and sufficient Hamilton–Jacobi–Bellman conditions are used to characterize an optimal feedback controller. Another important application of partial stability and partial stabilization theory is the unification it provides between time-invariant stability theory and stability theory for time-varying systems [3,6]. We exploit this unification and specialize our results to address optimal linear and nonlinear regulation for linear and nonlinear time-varying stochastic systems with quadratic and nonlinear-nonquadratic cost functionals.

Our approach focuses on the role of the Lyapunov function guaranteeing stochastic stability of the closed-loop system and its connection to the steady-state solution of the stochastic Hamilton–Jacobi–Bellman equation characterizing the optimal nonlinear feedback controller. In order to avoid the complexity in solving the stochastic steady-state, Hamilton–Jacobi–Bellman equation, we do not attempt to minimize a *given* cost functional, but rather, we parameterize a family of stochastically stabilizing controllers that minimizes a *derived* cost functional that provides the flexibility in specifying the control law. This corresponds to addressing an *inverse optimal stochastic control problem* [7–13].

The inverse optimal control design approach provides a framework for constructing the Lyapunov function for the closed-loop system that serves as an optimal value function and, as shown in Refs. [11,12], achieves desired stability margins. Specifically, nonlinear inverse optimal controllers that minimize a *meaningful* (in the terminology of Refs. [11,12]) nonlinear-nonquadratic performance criterion involving a nonlinear-nonquadratic, non-negative-definite function of the state and a quadratic positive-definite function of the feedback control are shown to possess sector margin guarantees to component decoupled input nonlinearities in the conic sector $(1/2,\u221e)$.

The paper is organized follows. In Sec. 2, we establish notation, definitions, and present some key results on partial stability of nonlinear stochastic dynamical systems. In Sec. 3, we consider a stochastic nonlinear system with a performance functional evaluated over the infinite horizon. The performance functional is then evaluated in terms of a Lyapunov function that guarantees partial asymptotic stability in probability. We then state a stochastic optimal control problem and provide sufficient conditions for characterizing an optimal nonlinear feedback controller guaranteeing partial asymptotic stability in probability of the closed-loop system. These results are then used to address a stochastic optimal control problem for uniform asymptotic stabilization in probability of nonlinear time-varying stochastic dynamical systems.

In Sec. 4, we develop optimal feedback controllers for affine stochastic nonlinear systems using an inverse optimality framework tailored to the partial-state stochastic stabilization problem. This result is then used to derive time-varying extensions of the results in Refs. [14,15] involving nonlinear feedback controllers minimizing polynomial and multilinear performance criteria. In Sec. 5, we provide two illustrative numerical examples that highlight the optimal partial-state stochastic stabilization framework. In Sec. 6, we present conclusions and highlight some future research directions. Finally, we note that a preliminary version of this paper appeared in Ref. [16]. The present paper considerably expands on Ref. [16] by providing detailed proofs of all the results along with examples and additional motivation.

In this section, we establish notation, definitions, and review some basic results on partial stability of nonlinear stochastic dynamical systems [17–22]. Specifically, $\mathbb{R}$ denotes the set of real numbers, $\mathbb{R}+$ denotes the set of positive real numbers, $\mathbb{R}\xaf+$ denotes the set of non-negative numbers, $\mathbb{Z}+$ denotes the set of positive integers, $\mathbb{R}n$ denotes the set of *n* × 1 real column vectors, $\mathbb{R}n\xd7m$ denotes the set of *n* × *m* real matrices, $\mathbb{N}n$ denotes the set of *n* × *n* non-negative-definite matrices, and $\mathbb{P}n$ denotes the set of *n* × *n* positive-definite matrices. We write $B\epsilon (x)$ for the *open ball centered* at *x* with *radius ε*, $||\xb7||$ for the Euclidean vector norm or an induced matrix norm (depending on context), $\Vert \xb7\Vert F$ for the Frobenius matrix norm, *A*^{T} for the transpose of the matrix *A*, ⊗ for the Kronecker product, ⊕ for the Kronecker sum, and *I _{n}* or

*I*for the

*n*×

*n*identity matrix. Furthermore, $Bn$ denotes the

*σ*-algebra of Borel sets in $D\u2286\mathbb{R}n$, and $S$ denotes a

*σ*-algebra generated on a set $S\u2286\mathbb{R}n$.

We define a complete probability space as $(\Omega ,F,\mathbb{P})$, where Ω denotes the sample space, $F$ denotes a *σ*-algebra, and $\mathbb{P}$ defines a probability measure on the *σ*-algebra $F$; that is, $\mathbb{P}$ is a non-negative countably additive set function on $F$ such that $\mathbb{P}(\Omega )=1$ [20]. Furthermore, we assume that *w*(⋅) is a standard *d*-dimensional Wiener process defined by $(w(\xb7),\Omega ,F,\mathbb{P}w0)$, where $\mathbb{P}w0$ is the classical Wiener measure [22, p. 10], with a continuous-time filtration ${Ft}t\u22650$ generated by the Wiener process *w*(*t*) up to time *t*. We denote a stochastic dynamical system by $G$ generating a filtration ${Ft}t\u22650$ adapted stochastic process $x:\mathbb{R}\xaf+\xd7\Omega \u2192D$ on $(\Omega ,F,\mathbb{P}x0)$ satisfying $F\tau \u2282Ft,\u20090\u2264\tau <t$, such that ${\omega \u2208\Omega :x(t,\omega )\u2208B}\u2208Ft,\u2009t\u22650$, for all Borel sets $B\u2282\mathbb{R}n$ contained in the Borel *σ*-algebra $Bn$. Here, we use the notation *x*(*t*) to represent the stochastic process *x*(*t*, *ω*) omitting its dependence on *ω*.

We denote the set of equivalence classes of measurable, integrable, and square-integrable $\mathbb{R}n$ or $\mathbb{R}n\xd7m$ (depending on context) valued random processes on $(\Omega ,F,\mathbb{P})$ over the semi-infinite parameter space [0, *∞*) by $L0(\Omega ,F,\mathbb{P}),\u2009L1(\Omega ,F,\mathbb{P})$, and $L2(\Omega ,F,\mathbb{P})$, respectively, where the equivalence relation is the one induced by $\mathbb{P}$-almost-sure equality. In particular, elements of $L0(\Omega ,F,\mathbb{P})$ take finite values $\mathbb{P}$-almost surely (a.s.). Hence, depending on the context, $\mathbb{R}n$ will denote either the set of *n* × 1 real variables or the subspace of $L0(\Omega ,F,\mathbb{P})$ comprising $\mathbb{R}n$ random processes that are constant almost surely. All inequalities and equalities involving random processes on $(\Omega ,F,\mathbb{P})$ are to be understood to hold $\mathbb{P}$-almost surely. Furthermore, $E[\u2009\xb7\u2009]$ and $Ex0[\u2009\xb7\u2009]$ denote, respectively, the expectation with respect to the probability measure $\mathbb{P}$ and with respect to the classical Wiener measure $\mathbb{P}x0$.

Finally, we write tr(⋅) for the trace operator, $(\xb7)\u22121$ for the inverse operator, $V\u2032(x)\u225c((\u2202V(x))/\u2202x)$ for the Fréchet derivative of *V* at *x*, $V\u2033(x)\u225c((\u22022V(x))/\u2202x2)$ for the Hessian of *V* at *x*, and $Hn$ for the Hilbert space of random vectors $x\u2208\mathbb{R}n$ with finite average power, that is, $Hn\u225c{x:\Omega \u2192\mathbb{R}n:E[xTx]<\u221e}$. For an open set $D\u2286\mathbb{R}n,\u2009HnD\u225c{x\u2208Hn:x:\Omega \u2192D}$ denotes the set of all the random vectors in $Hn$ induced by $D$. Similarly, for every $x0\u2208\mathbb{R}n,\u2009Hnx0\u225c{x\u2208Hn:x=x0\u2009a.s.}$. Furthermore, C^{2} denotes the space of real-valued functions $V:D\u2192\mathbb{R}$ that are two-times continuously differentiable with respect to $x\u2208D\u2286\mathbb{R}n$.

In this paper, we consider nonlinear stochastic autonomous dynamical systems $G$ of the form

where, for every $t\u2265t0,\u2009x1(t)\u2208Hn1D$ and $x2(t)\u2208Hn2$ are such that $x(t)\u225c[x1T(t),\u2009x2T(t)]T$ is a $Ft$-measurable random state vector, $x(t0)\u2208Hn1D\xd7Hn2,\u2009D\u2286\mathbb{R}n1$ is an open set with $0\u2208D$, *w*(*t*) is a *d*-dimensional independent standard Wiener process (i.e., Brownian motion) defined on a complete filtered probability space $(\Omega ,F,{Ft}t\u2265t0,\mathbb{P}),\u2009x(t0)$ is independent of $(w(t)\u2212w(t0)),t\u2265t0$, and $f1:D\xd7\mathbb{R}n2\u2192\mathbb{R}n1$ is such that, for every $x2\u2208\mathbb{R}n2,\u2009f1(0,x2)=0$ and *f*_{1}(⋅, *x*_{2}) is locally Lipschitz continuous in *x*_{1}, and $f2:D\xd7\mathbb{R}n2\u2192\mathbb{R}n2$ is such that, for every $x1\u2208D,\u2009f2(x1,\xb7)$ is locally Lipschitz continuous in *x*_{2}. In addition, the function $D1:D\xd7\mathbb{R}n2\u2192\mathbb{R}n1\xd7d$ is continuous such that, for every $x2\u2208\mathbb{R}n2,\u2009D1(0,x2)=0$, and $D2:D\xd7\mathbb{R}n2\u2192\mathbb{R}n2\xd7d$ is continuous.

A $\mathbb{R}n1+n2$-valued stochastic process $x:[t0,\tau ]\xd7\Omega \u2192D\xd7\mathbb{R}n2$ is said to be a *solution* of Eqs. (1) and (2) on the interval [*t*_{0}, *τ*] with initial condition *x*(*t*_{0}) = *x*_{0} a.s., if *x*(⋅) is *progressively measurable* (i.e., *x*(⋅) is nonanticipating and measurable in *t* and *ω*) with respect to ${Ft}t\u2265t0,\u2009f(x1,x2)\u225c[f1T(x1,x2),f2T(x1,x2)]T\u2208L1(\Omega ,F,\mathbb{P}),\u2009D(x1,x2)\u225c[D1T(x1,x2),D2T(x1,x2)]T\u2208L2(\Omega ,F,\mathbb{P})$, and

where the integrals in Eq. (3) are Itô integrals. Note that for each fixed *t* ≥ *t*_{0}, the random variable $\omega \u21a6x(t,\omega )$ assigns a vector *x*(*ω*) to every outcome *ω* ∈ Ω of an experiment, and for each fixed *ω* ∈Ω, the mapping $t\u21a6x(t,\omega )$ is the *sample path* of the stochastic process *x*(*t*), *t* ≥ *t*_{0}. A pathwise solution $t\u21a6x(t)$ of Eqs. (1) and (2) in $(\Omega ,{Ft}t\u2265t0,\mathbb{P}x0)$ is said to be *right maximally* defined if *x* cannot be extended (either uniquely or nonuniquely) forward in time. We assume that all right maximal pathwise solutions to Eqs. (1) and (2) in $(\Omega ,{Ft}t\u2265t0,\mathbb{P}x0)$ exist on [*t*_{0}, *∞*), and hence, we assume that Eqs. (1) and (2) are *forward complete*. Sufficient conditions for forward completeness or *global solutions* to Eqs. (1) and (2) are given by Corollary 6.3.5 of Ref. [20].

Furthermore, we assume that $f:D\xd7\mathbb{R}n2\u2192\mathbb{R}n1+n2$ and $D:D\xd7\mathbb{R}n2\u2192\mathbb{R}(n1+n2)\xd7d$ satisfy the uniform Lipschitz continuity condition

and the growth restriction condition

for some Lipschitz constant *L* > 0, and hence, since $x(t0)\u2208Hn1D\xd7Hn2$ and *x*(*t*_{0}) is independent of $(w(t)\u2212w(t0)),t\u2265t0$, it follows that there exists a unique solution $x\u2208L2(\Omega ,F,\mathbb{P})$ of Eqs. (1) and (2) in the following sense. For every $x\u2208Hn1D\xd7Hn2$, there exists *τ _{x}* > 0 such that, if $xI:[t0,\tau 1]\xd7\Omega \u2192D\xd7\mathbb{R}n2$ and $xII:[t0,\tau 2]\xd7\Omega \u2192D\xd7\mathbb{R}n2$ are two solutions of Eqs. (1) and (2); that is, if $xI,xII\u2208L2(\Omega ,F,\mathbb{P})$, with continuous sample paths almost surely, solve Eqs. (1) and (2), then $\tau x\u2264min{\tau 1,\tau 2}$ and $\mathbb{P}(xI(t)=xII(t),\u2009t0\u2264t\u2264\tau x)=1$. Sufficient conditions for forward existence and uniqueness in the absence of the uniform Lipschitz continuity condition and growth restriction condition can be found in Refs. [23,24].

A solution $t\u21a6[x1T(t),x2T(t)]T$ is said to be *regular* if and only if $\mathbb{P}x0(\tau e=\u221e)=1$ for all $x(0)\u2208Hn1D\xd7Hn2$, where *τ*^{e} is the first *stopping time* of the solution to Eqs. (1) and (2) from every bounded domain in $D\xd7\mathbb{R}n2$. Recall that regularity of solutions implies that solutions exist for *t* ≥ *t*_{0} almost surely. Here, we assume regularity of solutions to Eqs. (1) and (2), and hence, *τ _{x}* =

*∞*[18, p. 75]. Moreover, the unique solution determines a $\mathbb{R}n1+n2$-valued, time-homogeneous Feller continuous Markov process

*x*(⋅), and hence, its stationary Feller transition probability function is given by (Refs. [18, Theorem 3.4] and [20, Theorem 9.2.8]) $\mathbb{P}(x(t)\u2208B|x(t0)=a.s.x0)=\mathbb{P}(t\u2212t0,x0,0,B)$ for all $x0\u2208D\xd7\mathbb{R}n2$ and

*t*≥

*t*

_{0}, and all Borel subsets $B$ of $D\xd7\mathbb{R}n2$, where $\mathbb{P}(s,x,t,B),t\u2265s$, denotes the probability of transition of the point $x\u2208D\xd7\mathbb{R}n2$ at time instant

*s*into the set $B\u2282D\xd7\mathbb{R}n2$ at time instant

*t*. Finally, recall that every continuous process with Feller transition probability function is also a strong Markov process [18, p. 101].

Definition 2.1 [22, Definition 7.7]. *Let x(**⋅**) be a time-homogeneous Markov process in*$Hn1D\xd7Hn2$* and let*$V:D\xd7\mathbb{R}n2\u2192\mathbb{R}$*. Then, the infinitesimal generator*$L$* of x(t), t ≥ 0, with x(0) = x _{0} a.s., is defined by*

If $V\u2208C2$ and has a compact support, and *x*(*t*), *t* ≥ *t*_{0}, satisfies Eqs. (1) and (2) , then the limit in Eq. (6) exists for all $x\u2208D\xd7\mathbb{R}n2$ and the infinitesimal generator $L$ of *x*(*t*), *t* ≥ *t*_{0}, can be characterized by the system drift and diffusion functions *f*(*x*) and *D*(*x*) defining the stochastic dynamical system (1) and (2) with system state *x*(*t*), *t* ≥ *t*_{0}, and is given by [22, Theorem 7.9]

Definition 2.2. *(i) The nonlinear stochastic dynamical system*$G$* given by Eqs.*(1)* and*(2)* is* Lyapunov stable in probability with respect to *x*_{1} uniformly in *x*_{20}*if, for every ε > 0 and ρ > 0, there exist*$\delta =\delta (\rho ,\epsilon )>0$* such that, for all*$x10\u2208B\delta (0)$

*(ii)*$G$* is* asymptotically stable in probability with respect to *x*_{1} uniformly in *x*_{20}*if*$G$* is Lyapunov stable in probability with respect to x _{1} uniformly in x_{20} and*

*(iii)*$G$* is* globally asymptotically stable in probability with respect to *x*_{1} uniformly in *x*_{20} if $G$* is Lyapunov stable in probability with respect to x _{1} uniformly in x_{20} and*$\mathbb{P}x0(limt\u2192\u221e\Vert x1(t)\Vert =0)=1$

*holds uniformly in x*$(x10,x20)\u2208\mathbb{R}n1\xd7\mathbb{R}n2$.

_{20}for all*Remark 2.1*. It is important to note that there is a key difference between the stochastic partial stability definitions given in Definitions 2.2 and the definitions of stochastic partial stability given in Ref. [21]. In particular, the stochastic partial stability definitions given in Ref. [21] require that both the initial conditions *x*_{10} and *x*_{20} lie in a neighborhood of origin, whereas in Definition 2.2 *x*_{20} can be arbitrary. As will be seen below, this difference allows us to unify autonomous stochastic partial stability theory with time-varying stochastic stability theory. An additional difference between our formulation of the stochastic partial stability problem and the stochastic partial stability problem considered in Ref. [21] is in the treatment of the equilibrium of Eqs. (1) and (2). Specifically, in our formulation, we require the weaker partial equilibrium condition *f*_{1}(0, *x*_{2}) = 0 and *D*_{1}(0, *x*_{2}) = 0 for every $x2\u2208\mathbb{R}n2$, whereas in Ref. [21] the author requires the stronger equilibrium condition $f1(0,0)=0,f2(0,0)=0,D1(0,0)=0$, and *D*_{2}(0, 0) = 0.

*Remark 2.2*. A more general stochastic stability notion can also be introduced here involving stochastic stability and convergence to an invariant (stationary) distribution. In this case, state convergence is not to an equilibrium point but rather to a stationary distribution. This framework can relax the vanishing perturbation assumption $D1(0,x2)=0,\u2009x2\u2208\mathbb{R}n2$, and requires a more involved analysis and synthesis framework showing stability of the underlying Markov semigroup [25].

As shown in Refs. [3] and [6], an important application of deterministic partial stability theory is the unification it provides between time-invariant stability theory and stability theory for time-varying systems. A similar unification can be provided for stochastic dynamical systems. Specifically, consider the nonlinear time-varying stochastic dynamical system given by