Zhong-Ping Jiang

This research, whose principal author is Ph.D. student Bo Pang, was directed by Zhong-Ping Jiang, professor in the Department of Electrical and Computer Engineering. 

As an important and popular method in reinforcement learning (RL), policy iteration has been widely studied by researchers and utilized in different kinds of real-life applications by practitioners. 

Policy iteration involves two steps: policy evaluation and policy improvement. In policy evaluation, a given policy is evaluated based on a scalar performance index. Then this performance index is utilized to generate a new control policy in policy improvement. These two steps are iterated in turn, to find the solution of the RL problem at hand. When all the information involved in this process is exactly known, the convergence to the optimal solution can be provably guaranteed, by exploiting the monotonicity property of the policy improvement step. That is, the performance of the newly generated policy is no worse than that of the given policy in each iteration. 

However, in practice policy evaluation or policy improvement can hardly be implemented precisely, because of the existence of various errors, which may be induced by function approximation, state estimation, sensor noise, external disturbance and so on. Therefore, a natural question to ask is: when is a policy iteration algorithm robust to the errors in the learning process? In other words, under what conditions on the errors does the policy iteration still converge to (a neighborhood of) the optimal solution? And how to quantify the size of this neighbourhood?

This paper studies the robustness of reinforcement learning algorithms to errors in the learning process. Specifically, they revisit the benchmark problem of discrete-time linear quadratic regulation (LQR) and study the long-standing open question: Under what conditions is the policy iteration method robustly stable from a dynamical systems perspective? 

Using advanced stability results in control theory, they show that policy iteration for LQR is inherently robust to small errors in the learning process and enjoys small-disturbance input-to-state stability: whenever the error in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded, and, moreover, enter and stay in a small neighbourhood of the optimal LQR solution. As an application, a novel off-policy optimistic least-squares policy iteration for the LQR problem is proposed, when the system dynamics are subjected to additive stochastic disturbances. The proposed new results in robust reinforcement learning are validated by a numerical example.

This work was supported in part by the U.S. National Science Foundation.

Zhong-Ping Jiang, professor of electrical and computer engineering (ECE) and member of the C2SMART transportation research center at NYU Tandon, directed this research. Leilei Cui, a Ph.D. student in the ECE Department is lead author. Zhengyou Zhang and Shuai Wang from Tencent are co-authors.

This paper studies the trajectory tracking and balance control problem for an autonomous bicycle — one that is ridden like a normal bicycle before automatically traveling by itself to the next user — that is a non-minimum phase, strongly nonlinear system.

As compared with most existing methods dealing only with approximate trajectory tracking, this paper solves a longstanding open problem in bicycle control: how to develop a constructive design to achieve asymptotic trajectory tracking with balance. The crucial strategy is to view the controlled bicycle dynamics from an interconnected system perspective.

More specifically, the nonlinear dynamics of the autonomous bicycle is decomposed into two interconnected subsystems: a tracking subsystem and a balancing subsystem. For the tracking subsystem, the popular backstepping approach is applied to determine the propulsive force of the bicycle. For the balancing subsystem, optimal control is applied to determine the steering angular velocity of the handlebar in order to balance the bicycle and align the bicycle with the desired yaw angle. In order  to tackle the strong coupling between the tracking and the balancing systems, the small-gain technique is applied for the first time to prove the asymptotic stability of the closed-loop bicycle system. Finally, the efficacy of the proposed exact trajectory tracking control methodology is validated by numerical simulations (see the video).  

"Our contribution to this field is principally at the level of new theoretical development," said Jiang, adding that the key challenge is in the bicycle's inherent instability and more degrees of freedom than the number of controllers. "Although the bicycle looks simple, it is much more difficult to control than driving a car because riding a bike needs to simultaneously track a trajectory and balance the body of the bike. So a new theory is needed for the design of an AI-based, universal controller." He said the work holds great potential for developing control architectures for complex systems beyond bicycles.

The work was done under the aegis of the Control and Network (CAN) Lab led by Jiang, which consists of about 10 people and focuses on the development of fundamental principles and tools for the stability analysis and control of nonlinear dynamical networks, with applications to information, mechanical and biological systems.

The research was funded by the National Science Foundation (Grant number 10.13039/100000001). 

COVID 19 has wreaked havoc across the planet. As of January 1, 2021, the WHO has reported nearly 82 million cases globally, with over 1.8 million deaths. In the face of this upheaval, public health authorities and the general population are striving to achieve a balance between safety and normalcy. The uncertainty and novelty of the current conditions call for the development of theory and simulation tools that could offer a fine resolution of multiple strata of society while supporting the evaluation of “what-if” scenarios. 

The research team led by Maurizio Porfiri proposes an agent-based modeling platform to simulate the spreading of COVID-19 in small towns and cities. The platform is developed at the resolution of a single individual, and demonstrated for the city of New Rochelle, NY — one of the first outbreaks registered in the United States. The researchers used New Rochelle not only because of its place in the COVID timeline, but because agent-based modelling for mid-size towns are relatively unexplored despite the U.S. being largely composed of small towns.

Supported by expert knowledge and informed by officially reported COVID-19 data, the model incorporates detailed elements of the spreading within a statistically realistic population. Along with pertinent functionality such as testing, treatment, and vaccination options, the model also accounts for the burden of other illnesses with symptoms similar to COVID-19. Unique to the model is the possibility to explore different testing approaches — in hospitals or drive-through facilities— and vaccination strategies that could prioritize vulnerable groups. Decision making by public authorities could benefit from the model, for its fine-grain resolution, open-source nature, and wide range of features.

The study had some stark conclusions. One example: the results suggest that prioritizing vaccination of high-risk individuals has a marginal effect on the count of COVID-19 deaths. To obtain significant improvements, a very large fraction of the town population should, in fact, be vaccinated. Importantly, the benefits of the restrictive measures in place during the first wave greatly surpass those from any of these selective vaccination scenarios. Even with a vaccine available, social distancing, protective measures, and mobility restrictions will still key tools to fight COVID-19.

The research team included Zhong-Ping Jiang, professor of electrical and computer engineering; post-docs Agnieszka Truszkowska, who led the implementation of the computational framework for the project, and Brandon Behring; graduate student Jalil Hasanyan; as well as Lorenzo Zino from the University of Groningen, Sachit Butail from Southern Illinois University, Emanuele Caroppo from the Università Cattolica del Sacro Cuore, and Alessandro Rizzo from Turin Polytechnic. The work was partially supported by National Science Foundation (CMMI1561134 and CMMI-2027990), Compagnia di San Paolo, MAECI (“Mac2Mic”), the European Research Council, and the Netherlands Organisation for Scientific Research.