PSO-PPO-based reinforcement learning control strategy for active suspension systems under multiple operating conditions

2.1. Pavement modeling

To implement agent training, the active suspension model requires real-time data generation. The data-driven approach obtains its input from road profiles, utilizing the statistical characteristic parameters of spatial frequency power spectral density (PSD) and its time-domain representation to characterize random road unevenness. A stochastic road input model is constructed, where the impact of road roughness on the vehicle suspension system's dynamic response is mathematically formulated as follows:

where $\dot{q}\left(t\right)$ is the vertical velocity excitation of road unevenness, $q\left(t\right)$ is the vertical displacement excitation of road unevenness, $v$ is the vehicle speed, $f_0=n_{0}v$ is the lower cutoff temporal frequency, $n_0$ is the lower cutoff spatial frequency, and $\omega \left(t\right)$ is zero-mean Gaussian white noise.

2.2. Dynamic model of suspension system under different operating conditions

During vehicle operation, actual driving environments exhibit complexity and variability. To realistically simulate vehicle operating conditions, three characteristic scenarios are selected: constant-speed cruising, launch, and emergency braking. A 4-DOF suspension system dynamic model under different operating conditions is established.

2.2.1. Constant-speed condition

Following Newton’s second law, the governing equations of the 4-DOF model are formulated through force- and moment-equilibrium analysis about the vehicle’s center of mass and translational equilibrium in the vertical direction:

2

$\left\{\begin{array}{l}{m}_2{\ddot{z}}_2+k_f(z_{2f}-z_{1f})+c_f({\dot{z}}_{2f}-{\dot{z}}_{1f})-F_{af}+k_r(z_{2r}-z_{1r})+c_r({\dot{z}}_{2r}-{\dot{z}}_{1r})-F_{ar}=0,\\ I_y\ddot{\theta }-a\left[k_f\right(z_{2f}-z_{1f})+c_f({\dot{z}}_{2f}-{\dot{z}}_{1f})-F_{af}]+b\left[k_r\right(z_{2r}-z_{1r})\\ +cr({\dot{z}}_{2r}-{\dot{z}}_{1r})-F_{ar}]=0.\end{array}\right.$

Fig. 1Half-car four-degree-of-freedom dynamic model

a) Constant-speed condition

b) Launch condition

c) Emergency braking condition

Following Newton’s second law, with the front- and rear-unsprung masses as dynamic subsystems, the governing differential equations are derived:

3

$\left\{\begin{array}{c}{m}_{1f}{\ddot{z}}_{1f}+{k_t}_f(z_{1f}-q_f)-c_f({\dot{z}}_{2f}-{\dot{z}}_{1f})-k_f(z_{2f}-z_{1f})=-F_{af},\\ m_{1r}{\ddot{z}}_{1r}+k_{tr}(z_{1r}-q_r)-c_r({\dot{z}}_{2r}-{\dot{z}}_{1r})-k_r(z_{2r}-z_{1r})=-F_{ar},\end{array}\right.$

where $m_2$ is the sprung mass, $I_y$ is the moment of inertia of the sprung mass about the $y$-axis, $m_{1f}$ and $m_{1r}$ are the front and rear wheel masses, $k_f$ and $k_r$ are the front and rear suspension stiffnesses, $c_f$ and $c_r$ are the front and rear suspension damping coefficients, $k_{tf}$ and $k_{tr}$ are the front and rear tire stiffnesses, $a$ and $b$ are the longitudinal distances from the front and rear axles to the vehicle's center of mass.

The state variables $x$, output variables $y$ and input variables $u$ are defined as follows:

4

$x={\left[\begin{array}{cccccccc}{x}_1& x_2& x_3& x_4& x_5& x_6& x_7& x_8\end{array}\right]}^T={\left[\begin{array}{cccccccc}{z}_2& {\dot{z}}_2& \theta & \dot{\theta }& z_{1f}& {\dot{z}}_{1f}& z_{1r}& {\dot{z}}_{1r}\end{array}\right]}^T,$

5

$y={\left[\begin{array}{cccccccc}{y}_1& y_2& y_3& y_4& y_5& y_6& y_7& y_8\end{array}\right]}^T$
${=\left[\begin{array}{cccccc}{\ddot{z}}_2& \ddot{\theta }& z_{2f}-z_{1f}& z_{2r}-z_{1r}& z_{1f}-q_f& z_{1r}-q_r\end{array}\right]}^T,$

6

$u={\left[\begin{array}{cccc}{F}_{af}& F_{ar}& q_f& q_r\end{array}\right]}^T,$

where $z_2$ is the vertical displacement of the vehicle body, $\theta$ is the pitch angle of the vehicle body, $z_{1f}$ and $z_{1r}$ are the vertical displacements of the front and rear wheels, $z_{2f}$-$z_{1f}$ and $z_{2r}$-$z_{1r}$ are the dynamic deflections of the front and rear suspensions, $q_f$ and $q_r$ are the road excitations at the front and rear wheels, $F_{af}$ and $F_{ar}$ are the active forces of the front and rear suspensions.

Transform the above equations into state-space equations, where A, B, C, D are the coefficient matrices of the state-space equations (coefficient matrices are shown in Appendix A1):

7

$\left\{\begin{array}{c}\dot{x}=Ax+Bu,\\ y=Cx+Du.\end{array}\right.$

3.1. Integrated workflow for active suspension control

This paper combines the PSO-PPO algorithm with active suspension systems. The overall process involves sensors collecting road information input to the suspension system model, while the Particle Swarm Optimization (PSO) algorithm pre-trains with the environment to generate the optimal initial policy required for Proximal Policy Optimization (PPO) training. The agent is trained through reinforcement learning, continuously optimizing the control policy based on obtained reward values until the control performance converges. The deployed agent then selects corresponding action as outputs according to the probability density function in the Actor network. The interaction process between the PSO-PPO algorithm and the suspension environment is shown in Fig. 2.

Fig. 2PSO-PPO algorithm suspension control architecture

3.2. PSO-PPO algorithm workflow

To address the issues of instability during the initial training phase and prolonged convergence time in reinforcement learning algorithms, this paper proposes a control method that integrates the PSO algorithm and the PPO algorithm. Specifically, the PSO algorithm is employed to generate parameters for the Actor network, which are pre-evaluated through environmental interaction using the cumulative rewards of trajectories as fitness criteria. The PPO algorithm then adopts the Actor network parameters with the highest fitness as the initial policy for agent training. A multi-objective optimization framework is designed to configure key parameters of the reinforcement learning control algorithm, including the reward function, learning rate, and discount factor. The interaction logic between the PSO and PPO algorithms is illustrated in Fig. 3.

In the field of suspension control, the initial policy generated by the PSO algorithm is applied to the suspension system. The cumulative reward calculated from vehicle-body responses induced by road-vehicle interaction serves as the fitness metric, with the fitness function formulated as:

The velocity update formula governing particle position and search speed in the PSO algorithm is expressed as:

where $r_1$ and $r_2$ are random numbers uniformly distributed in the interval [0, 1], introduced to enhance the stochastic exploration capability of the algorithm. The acceleration constants $c_1$ and $c_2$, which control the learning rates of individual and social cognition respectively, are set to $c_1=c_2=$1.5. The inertia weight $w$, governing the momentum preservation in particle motion, is assigned a value of $w=$0.5.

Fig. 3Interaction logic between the PSO and PPO algorithms

The PSO algorithm generates optimal initial policies required for subsequent reinforcement learning through preliminary interaction with the suspension environment, thereby reducing blind exploration during the initial training phase, improving sample efficiency, and accelerating convergence.

In reinforcement learning, policy gradient methods maximize expected returns by directly optimizing policy parameters. However, traditional policy gradient approaches are prone to high-variance gradient estimates, leading to unstable training. Moreover, policy updates may cause drastic fluctuations in policy performance, or even policy collapse. To address these issues, the PPO algorithm introduces a surrogate objective with clipped probability ratios, constraining policy update magnitudes. This mechanism ensures policy improvement while preventing gradient overshooting, with the clipping function formulated as:

where $r_t\left(\theta \right)$ denotes the probability ratio between new and old policies, and $\epsilon$ controls the clipping range.

In the PPO-based suspension system control algorithm, the neural network comprises an Actor network and a Critic network, responsible for generating control actions and evaluating the current state value, respectively. Both networks share an identical architecture: an input layer followed by two hidden layers. The network architecture parameters are detailed in Table 1.

By properly configuring these parameters, a neural network with moderate complexity can be constructed to enhance control effectiveness while maintaining efficiency and stability during training. The Actor network optimizes the parameters $\theta$ in the control policy ${\pi }_{\theta }\left(a\left|s\right.\right)$ through the gradient descent algorithm, directly learning action-output policies from the state space to maximize cumulative rewards. The PPO algorithm employs the Critic network to evaluate the quality of the Actor network’s policy. The core task of the Critic network is to compute the state value function $V\left(s\right)$, which predicts the expected cumulative rewards obtainable under the current policy when in state $s$. The target state value function $V_t\left(s\right)$ is calculated using the Generalized Advantage Estimation (GAE) method:

where $V\left(s\right)$ is the predicted value, $V_t\left(s\right)$ is the target value, $A^{GAE}$ is the GAE factor.

When the PPO algorithm completes a policy update, the new policy is used for subsequent iterations until policy convergence is achieved, ultimately applied to output the active force required for suspension control.

In summary, the training workflow of the PSO-PPO-based active suspension control method is as shown in Table 2.

3.3. PSO-PPO training parameter configuration

According to the characteristics of different road conditions and driving scenarios, the performance weighting factors are adjusted, enabling the suspension system to adapt dynamically under various road conditions to optimize both comfort and stability. This design approach reduces complexity in the reward function while ensuring that reward signals during the reinforcement learning process can effectively guide the agent's learning and optimization. To prevent excessive influence from single variables on the reward function, reward scaling is typically employed. The reward function is formulated as follows:

where $k_1$, $k_2$, $k_3$, $k_4$, $k_5$and $k_6$ represent the weighting coefficients for body acceleration, front suspension dynamic deflection, rear suspension dynamic deflection, front tire dynamic travel, rear tire dynamic travel, and body pitch angular acceleration, respectively.

In reinforcement learning, performance requirements vary with operating conditions, while adjusting weighting coefficients enhances control effectiveness. The learning rate is a critical hyperparameter that controls the step size of parameter updates during training. When vehicle body responses exhibit significant fluctuations, increasing the learning rate can improve adaptation capability. Proper learning rate tuning substantially improves training efficiency and convergence speed. The reward weighting coefficients and learning rates for each evaluation metric under various operating conditions are provided in Table 3.

The configuration of hyperparameters in reinforcement learning directly impacts training efficiency and convergence rate. Key parameters in the PSO-PPO algorithm include the clipping factor, neural network parameters, entropy coefficient, and discount factor. The hyperparameter configurations are listed in Table 4.

After configuring the reinforcement learning parameters, the agent training process is conducted. The training convergence curves of the baseline PPO algorithm and the PSO-PPO hybrid control method are compared in Fig. 4.

As shown, utilizing PSO for pre-training exploration to generate high-quality initial samples significantly reduces random exploration in the early training phase. This mechanism improves sample efficiency and enables accelerated policy convergence compared to standard PPO.

Fig. 4Reinforcement learning training process

3.4. Multi-condition-oriented control methodology

The PSO-PPO algorithm generates distinct control policies for three operational modes, each requiring dynamic weight allocation of suspension performance objectives. While scenario-specific reward weights are tuned for individual modes, real-world driving typically involves hybrid modes requiring real-time weight adaptation. This multi-condition suspension control problem constitutes a multi-objective optimization challenge with $m$ evaluation criteria and $n$ performance metrics, for which the entropy weighting method provides an ideal solution. The information entropy is defined as:

where $H$ is the information entropy, $q$ is the source alphabet size, $P\left(x_i\right)$ specifies the occurrence probability of information unit $x_i$.

The entropy weight method determines relative importance weights for multi-objective decision criteria through the following standardized steps:

4.1. Constant-speed condition

To verify the control effectiveness of the reinforcement-learning-based active suspension control method under constant-speed conditions, a comparative simulation analysis was conducted involving a passive suspension system, a conventional LQR-based active suspension system, and a PSO-PPO-algorithm-based active suspension system.

The comparative experiments were configured with three test scenarios:

1) Simulation experiments and result analysis of suspension systems under constant-speed driving at 20 m/s on class C roads.

2) Simulation experiments and result analysis of suspension systems under constant-speed driving at 20 m/s on class D roads.

3) Simulation experiments and result analysis of suspension systems under constant-speed driving on sequentially varying road surfaces (class A, B, C).

Fig. 5 shows the simulation results of body acceleration, pitch angular acceleration, front/rear suspension dynamic deflection, and front/rear tire dynamic travel when the vehicle travels at 20 m/s on class C roads.

Fig. 6 shows the simulation results of body acceleration, pitch angular acceleration, front/rear suspension dynamic deflection, and front/rear tire dynamic travel when the vehicle travels at 20 m/s on class D roads.

Tables 6 and 7 present the RMS values of various metrics for passive suspension, LQR-based active suspension, and PSO-PPO-algorithm-based active suspension under identical test conditions, along with the percentage improvement (“-“ indicates improvement; “+” indicates deterioration) of both active suspension control methods compared to passive suspension. Table 8 present the standard deviation of various metrics for 5 program runs.

Fig. 5Suspension response under class C road excitation

a) Body acceleration

b) Pitch angular acceleration

c) Front suspension dynamic deflection

d) Rear suspension dynamic deflection

e) Front tire dynamic travel

f) Rear tire dynamic travel

The PSO-PPO algorithm demonstrates superior control performance compared to both the LQR-based controller and passive suspension systems across all evaluated metrics. Under class C road conditions, the algorithm achieves 49.1 % reduction in body acceleration and 25.7 % suppression in pitch angular acceleration, while on class D roads, these improvements reach 39.1 % and 38.2 %, respectively, significantly outperforming the LQR method. This capability enables dual optimization of vehicle stability and ride comfort through effective vibration mitigation and dynamic response control. Furthermore, the algorithm exhibits outstanding suspension control performance, with front and rear suspension dynamic deflections improved by 25.4 % and 29.8 % respectively, surpassing LQR's optimization results. Although its improvement on tire dynamic travel is relatively modest, the PSO-PPO algorithm still demonstrates positive impacts on overall suspension system performance. Collectively, the PSO-PPO algorithm shows significant advantages over LQR in control precision and response stability, particularly in suppressing body acceleration and pitch angular acceleration. These capabilities enable superior control effectiveness, further enhancing the comprehensive performance of suspension systems.

Fig. 6Suspension response under class D road excitation

a) Body acceleration

b) Pitch angular acceleration

c) Front suspension dynamic deflection

d) Rear suspension dynamic deflection

e) Front tire dynamic travel

f) Rear tire dynamic travel

To verify the universal adaptability of the control algorithm, a continuously varying random road excitation was configured, where the road condition transitioned sequentially from class A to class B, then to class C, and finally back to class B. Each road segment lasted 10 seconds, with a total simulation time set to 40 seconds. The variation in random road excitation levels is shown in Fig. 7.

Fig. 8. shows the body acceleration curves of the vehicle traveling at 20 m/s on time-varying road surfaces.

As evidenced in Table 9, the PSO-PPO-algorithm-based active suspension demonstrates superior performance in body acceleration reduction compared to both passive suspension and LQR-based active suspension, with explicit percentage improvements explicitly quantified in the table. The table also present the standard deviation of Body acceleration for 5 program runs. These comparative simulations confirm the algorithm's generalization capabilities across progressively changing road profiles, maintaining effective vibration suppression when handling diverse pavement classifications.

Fig. 7Stochastic road excitation classification variation

Fig. 8Body acceleration under continuously varying road excitation

4.2. Launch condition

The launch condition simulation experiment was configured with the vehicle accelerating from rest on class C road under a constant driving force $F_t=\mathrm{ }$2200 N, with the simulation duration set to 6 seconds. Fig. 9. displays comparative results of body acceleration, pitch angular acceleration, pitch angle variation, front/rear suspension dynamic deflections, and front/rear tire dynamic travels for passive suspension, LQR-controlled active suspension, and PSO-PPO-algorithm-controlled active suspension systems under launch conditions.

Table 10 lists the RMS values of these metrics for all three suspension configurations, with percentage improvements of both active control methods calculated relative to the passive suspension baseline. Table 11 present the standard deviation of various metrics for 5 program runs.

Fig. 9Suspension response under launch condition

a) Body acceleration

b) Pitch angular acceleration

c) Front suspension dynamic deflection

d) Rear suspension dynamic deflection

e) Front tire dynamic travel

f) Rear tire dynamic travel

g) Pitch angle variation

Under launch conditions, the PSO-PPO-algorithm-controlled active suspension demonstrates 68.52 % optimization in front suspension dynamic deflection and 33.89 % optimization in rear suspension dynamic deflection compared to passive suspension, achieving 47.52 % reduction in vehicle body pitch angle. This enhancement in body posture stability is attained without adversely affecting other vehicle performance metrics. Additionally, the PSO-PPO-algorithm-controlled system surpasses the LQR-controlled active suspension in all performance indicators except body acceleration.

4.3. Emergency braking condition

Emergency braking condition simulation experiments were configured as follows:

1) Passive suspension vehicle (without ABS system) braking from 20 m/s to full stop with maximum braking force on class C roads.

2) Passive suspension vehicle (with ABS system) braking from 20 m/s to full stop with maximum braking force on class C roads.

3) Active suspension vehicle (with ABS system) braking from 20 m/s to full stop with maximum braking force on class C roads.

Fig. 10 compares suspension pitch angle, body acceleration, braking distance, vehicle speed, front/rear suspension dynamic deflection, and front/rear tire dynamic travel across the three control methods under emergency braking.

Table 12 quantifies RMS values of vehicle performance metrics for each control strategy. Table 13 presents the comparative analysis of braking distance and braking time across these control configurations. Table 14 details percentage improvement comparisons among control strategies. Table 15 present the standard deviation of various metrics for 5 program runs.

Fig. 10Suspension response under emergency braking condition

a) Body acceleration

b) Pitch angle variation

c) Break distance

d) Speed

e) Front suspension dynamic deflection

f) Rear suspension dynamic deflection

g) Front tire dynamic travel

h) Rear tire dynamic travel

From the perspective of braking performance, the ABS demonstrates 9.7 % reduction in braking distance and 8.06 % decrease in braking duration compared to non-ABS configurations, enhancing stopping efficiency. The PSO-PPO-algorithm-controlled active suspension further refines these parameters, achieving 14.83 % shorter braking distance and 15.29 % reduced braking time, establishing itself as the superior configuration.

From the perspective of body stability, although ABS improves braking performance, it increases body pitch angle by 7.86 %, and front/rear suspension dynamic deflections by 5.82 % and 9.52 % respectively, indicating compromised posture control. In contrast, the PSO-PPO-algorithm-controlled active suspension reduces pitch angle by 45.06 %, and decreases the front/rear suspension dynamic deflections by 17.63 % and 15.8 %, significantly enhancing stability during braking.

From the perspective of suspension dynamic performance, ABS reduces front-wheel dynamic travel by 23.97 % but increases rear-wheel dynamic travel by 16.08 %, revealing front/rear response imbalance. The PSO-PPO-algorithm-controlled active suspension increases front-wheel dynamic travel by 17.78 % and reduces rear-wheel dynamic travel by 17.25 %, improving wheel motion balance. It increases vertical body acceleration by 41.98 %, potentially affecting ride comfort and necessitating control strategy optimization.

In summary, the PSO-PPO-algorithm-controlled active suspension improves braking performance and body attitude control, reducing vehicle pitch motion and tire dynamic deflection. This 9.70 % braking distance improvement is achieved at the cost of ride comfort degradation, with vertical body acceleration RMS value worsening by 41.98 %.

4.4. Multi-operating condition

The active suspension multi-condition co-simulation experimental setup was configured in CarSim with three sequential phases: From 0-3 seconds, the vehicle accelerated from 0 to 20 m/s; subsequently maintained a constant speed of 20 m/s until the 8th second; and finally executed emergency braking from 8-10 seconds, completing the total 10-second simulation duration.

Fig. 11 compares body vertical acceleration, pitch angular acceleration, front-left/rear-left suspension dynamic deflections, front-right/rear-right suspension dynamic deflections, front-left/rear-left tire dynamic travel, and front-right/rear-right tire dynamic travel across passive suspension, LQR-controlled active suspension, and PSO-PPO-algorithm-controlled active suspension under multi-condition operations.

Table 16 quantifies RMS values of performance metrics for all three configurations (passive, LQR-controlled, PSO-PPO-algorithm-controlled), including percentage improvements of active control methods relative to the passive suspension baseline. Table 17 present the standard deviation of various metrics for 5 program runs.

Fig. 11Suspension response under multiple operating conditions

a) Body acceleration

b) Pitch angular acceleration

c) Front-left suspension dynamic deflection

d) Rear-left suspension dynamic deflection

e) Front-right suspension dynamic deflection

f) Rear-right suspension dynamic deflection

g) Front-left tire dynamic travel

h) Rear-left tire dynamic travel

i) Front-right tire dynamic travel

j) Rear-right tire dynamic travel

The multi-condition co-simulation experimental results demonstrate that the PSO-PPO-algorithm-controlled active suspension system achieves comprehensive performance superiority over the LQR-controlled counterpart. The PSO-PPO algorithm achieves a 29.92 % reduction in vertical body acceleration, outperforming the LQR controller’s 17.65 % improvement. In evaluations of suspension dynamic deflection reflecting system stability, the PSO-PPO algorithm delivers 26.19 %-34.09 % optimization, significantly exceeding the LQR controller’s narrow 4.55 %-7.14 % enhancement range. Regarding tire dynamic travel critical for roadholding safety, the PSO-PPO achieves 17.39 %-28.57 % improvements across all wheel stations, while LQR controller exhibits 8.7 % performance deterioration at rear wheel stations, revealing inherent limitations in multi-objective coordinated control scenarios.