A multi-scale CNN-transformer hybrid network with parallel attention mechanism and local linear unit for cross-condition fault diagnosis

2.1. Parallel attention mechanism (PAM)

The channel attention mechanism is an effective approach for improving convolutional operations and has demonstrated significant potential in enhancing the performance of deep CNNs. Traditional attention mechanisms typically focus only on feature responses along the channel dimension, while their capability to model temporal dependencies in one-dimensional time-series signals is limited. To overcome this limitation, S. Woo et al. [18] proposed the Convolutional Block Attention Module (CBAM), a lightweight structure that sequentially integrates channel attention and spatial attention to enhance feature selection capability. Recently, Y. Sun et al. [19] further developed the PAM, which models the importance of channel and temporal dimensions through two parallel branches. The outputs of these branches are integrated to enhance the discriminative capability of one-dimensional temporal features. Its typical mathematical formulation is given by:

where $X_{out}\in {\mathbb{R}}^{L\times C}$ denotes the output feature map, $AMMP(\cdot )$ represents adaptive mixing mean pooling, $\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}(\cdot )$ denotes the one-dimensional convolution, $GELU(\cdot )$ is the Gaussian Error Linear Unit activation function, and $\odot$ represents element-wise multiplication.

2.2. Multi-scale CNNs

As one of the most representative deep learning models, CNNs are naturally suitable for extracting local temporal features from one-dimensional vibration signals, owing to their properties of local connectivity, weight sharing, and spatial pooling [20]. However, under cross-condition scenarios, the time–frequency structures of fault signals are highly complex. Consequently, vibration signals often exhibit pronounced multi-scale characteristics. A single convolutional kernel has a limited receptive field and is therefore insufficient to comprehensively capture multi-band transient impact features induced by different operating conditions and fault classes (e.g., bearing pitting or gear tooth breakage). To address this issue, multi-scale CNN has been widely adopted. By deploying multiple convolution kernels of different sizes in parallel, the network can simultaneously capture fine-grained local details and coarse-grained contextual information within a single forward propagation, enabling the automatic learning of complementary and discriminative multi-scale diagnostic features from complex vibration signals [21]. Given an input signal $x\in R^L$, the output of the multi-scale CNN branch is defined as:

where ${Conv}_{k_i\times 1}(\cdot )$ denotes a one-dimensional convolution with kernel size $k_i$, $n$ is the number of parallel convolution branches, and $\oplus$ represents channel-wise concatenation.

2.3. Transformer

The Transformer [22] was originally designed for natural language processing tasks. It consists of a stack of Transformer blocks, forming a deep network architecture that includes multi-head self-attention (MHSA), feed-forward network (FFN), and layer normalization (LN). The core component, the multi-head self-attention mechanism, is capable of modeling long-range dependencies between any two-time steps in a sequence. For an input sequence $X\in {\mathbb{R}}^{L\times d}$, the MHSA operation is calculated as:

where $W_i^Q$, $W_i^K$, and $W_i^V$ are the learnable parameters, and $d_k$ is the dimension of query and key vectors.

2.4. Local linear unit (LLU)

Although Transformers possess strong global modeling capability, directly using a conventional Transformer architecture to extract fault features may result in the loss of detailed fault information due to the absence of the inherent inductive bias provided by CNNs [23]. To address this issue, C. Weng et al. [23] proposed introducing the LLU into the Transformer architecture to enhance its ability to capture local features. The output feature embedding is defined as:

where $f^{k\times 1;D}(\cdot )$ denotes the one-dimensional convolution function, $k\times 1$ and $D$ represent the kernel size and number of filters, respectively, $U_{t-1}\left(\bullet \right)$ denotes the input transformation function at layer $t-1$, and $GELU(\cdot )$ denotes the activation function.

2.5. DG Learning

The primary objective of DG is to learn transferable representations from multiple labeled source domains, enabling the model to maintain stable performance on unseen target domains, where target-domain data are completely unavailable during training [24]. Accordingly, the learning objective of DG is formulated as minimizing the expected risk on the unknown target-domain distribution $D_t$:

where $f$ denotes the learned function and $\mathcal{l}$ represents the loss function.

3.1. Overall architecture

The overall framework proposed in this paper is illustrated in Fig. 1. The model takes raw vibration signals as input and first processes them through parallel multi-scale CNN branches. Specifically, one-dimensional convolutions with kernel sizes of 31, 63, and 127 are used to capture local fault features at different temporal scales. The outputs of all branches are concatenated along the channel dimension and subsequently compressed through a 1×1 convolution layer to map them into a unified embedding dimension. The compressed high-dimensional features are then fed into the PAM module, which simultaneously models the channel importance and key temporal weights to enhance the discriminative capability of feature representations under cross-condition scenarios. The PAM-enhanced feature sequence is further compressed into a fixed-length token sequence using the adaptive max pooling (AMP). The resulting sequence is concatenated with the learnable classification token ([CLS] token) and added to the learnable positional embeddings before being fed into a stack of deep Transformer encoder layers with embedded LLU modules. Within each Transformer block, the depthwise separable convolution explicitly models local temporal dependencies and is fused with the self-attention pathway through residual connections, enabling collaborative learning of local details and global contextual information. Finally, the [CLS] is normalized using Layer Normalization (LN) and mapped to the predefined fault categories through a linear classifier to produce the final diagnosis result.

This architecture effectively combines the local inductive bias of CNNs with the long-range dependency modeling capability of Transformers. Meanwhile, the PAM module strengthens feature focusing under cross-condition scenarios, and the LLU module enhances the modeling of transient impact responses, thereby effectively alleviating performance degradation caused by domain shift and enabling robust and accurate fault diagnosis under unseen operating conditions.

Fig. 1Structure of the proposed domain-generalizable fault diagnosis method

3.2. Multi-Scale feature extraction and channel compression

To comprehensively capture multi-band transient fault features excited under different operating conditions, three parallel one-dimensional convolutional branches are adopted for feature extraction. The kernel sizes are 31, 63, and 127, which are intended to capture high-frequency impulsive features, mid-frequency modulation patterns, and low-frequency trend information. To avoid domain-specific bias caused by complex fusion mechanisms, the outputs of all branches are directly concatenated along the channel dimension. A 1×1 convolution layer is then applied to compress the concatenated feature map into a unified embedding dimension $d$, producing a high-dimensional feature map $Z\in {\mathbb{R}}^{B\times d\times L}$, where $B$ denotes the batch size and $L$ represents the sequence length.

The resulting feature map is subsequently input into the PAM module described in Section 3.3 to perform joint feature reweighting along both the channel and temporal feature dimensions. After attention enhancement, the Adaptive Max Pooling (AMP) operation is applied to compress the temporal length into a predefined number of tokens $T$, yielding the token sequence $Z_{tok}\in {\mathbb{R}}^{B\times d\times T}$. This strategy effectively reduces computational complexity while preserving the peak amplitude points of transient impact signals, which are crucial for accurate fault diagnosis.

3.3. PAM in the proposed method

Under cross-condition scenarios, different channel features and temporal signal segments often exhibit different response intensities to fault information. Consequently, some discriminative fault features may be weakened or even overwhelmed, thereby degrading the model’s capability to characterize critical fault patterns. To mitigate this issue, the modified PAM module is designed to adaptively reweight feature responses related to faults along both the channel and temporal dimensions.

Existing PAM approaches typically rely on the parameterized Adaptive Mixing Mean Pooling (AMMP) combined with a single convolution layer. However, such designs may introduce domain-specific bias and are often inadequate for modeling local temporal signal structures associated with transient impact. To address these limitations, this study proposes a simplified and more physically interpretable PAM variant. Specifically, the Global Average Pooling (GAP) is used to replace AMMP, eliminating additional parameterization; the temporal branch employs two consecutive 1×1 convolution layers to better capture local temporal patterns; the final attention outputs are normalized using a sigmoid activation function and fused through weighted summation. The proposed PAM operates on the high-dimensional features extracted by the multi-scale CNN. Given the input feature map $Z\in {\mathbb{R}}^{B\times d\times L}$, where $B$ is the batch size, $L$ is the sequence length, and $d$ denotes the embedding dimension, the channel and temporal attention branches are defined as:

where $GAP(\cdot )$ denotes Global Average Pooling, $Con{v}_{k\times 1}$ represents a one-dimensional convolution with kernel size $k$, $\sigma (\cdot )$ denotes the Sigmoid activation function, $Con{v}_{1\times 1}^{\left(1\right)}$ and $Con{v}_{1\times 1}^{\left(2\right)}$ represent the first and second 1×1 convolution layers in the channel attention branch.

The final output is obtained by element-wise summation of the channel-weighted and temporal-weighted features. This lightweight design adopts a fixed structure without introducing additional adaptive parameters, enabling effective joint focusing on fault-sensitive channels and key temporal segments, thereby providing a robust feature basis for subsequent domain-invariant representation learning.

3.4. Transformer encoder with embedded LLU

Although the multi-scale CNN effectively captures local transient fault features, its receptive field is still constrained by fixed convolution kernel sizes, making it difficult to model long-term fault evolution patterns across cycles. To overcome this limitation, the Transformer encoder is introduced to model long-range dependencies in the sequence using the Multi-Head Self-Attention (MHSA) mechanism. However, standard Transformers rely entirely on data-driven attention weights and lack local inductive bias, which may cause them to overlook critical local waveform details when processing short-duration high-energy impact signals such as bearing pitting.

Inspired by the work of C. Weng et al. [23], this study introduces the LLU based on depthwise separable convolution, which is embedded within each Transformer block in a residual manner, as shown in Fig. 2. This design enhances the Transformer’s capability to model local temporal structures, such as the waveform details of transient impact, while maintaining low computational cost. The forward computation of the LLU is given by:

where $Con{v}_{depthwise}^{\left(k\right)}$ denotes the depthwise separable convolution with kernel size $k$, $BN$ represents Batch Normalization, and $GELU(\cdot )$ denotes the Gaussian Error Linear Unit activation function. The extracted local features are then added to the input of the current Transformer block through a residual connection. After the Layer Normalization (LN), the features are further processed by the MHSA and the feed-forward network (FFN). The detailed computation procedure is as follows:

In addition, the Pre-LayerNorm structure is adopted, where normalization is performed before both the MHSA and FFN modules. This design enhances training stability, particularly in small-sample DG scenarios.

Fig. 2Structure of the proposed Transformer block with embedded LLU

3.5. Classification output

Finally, the model maps the learned feature representation to predefined fault categories through a linear classifier. Specifically, the [CLS] token in the Transformer output is adopted as the global feature representation. After the Layer Normalization (LN), it is input into a linear classification layer to generate classification logits, which are used for the final fault diagnosis.

4.1. Datasets and task settings

Three benchmark datasets were used in this study, namely Case Western Reserve University (CWRU), Paderborn University (PU), and PHM09 datasets. The operating condition variations considered in this study include variable rotational speed (PU and PHM09 datasets) and variable load (CWRU dataset). The key characteristics of each dataset are summarized in Table 1.

To ensure fair comparison with existing methods, the task partition strategy and naming convention strictly follow the experimental settings in Reference [2]. For the PU and PHM09 datasets, four diagnostic tasks are constructed for each dataset, resulting in eight tasks in total. For each task, three operating conditions from the same machine are selected as the source domains, and the remaining condition is used as the target domain. The operating conditions and task settings of the PU dataset are shown in Tables 2 and 3, respectively.

The PHM09 dataset task configuration is shown in Table 4.

In addition, the proposed method is compared with a representative CNN–Transformer hybrid architecture, namely the Efficient Convolution Transformer Network (ECTN) [10], to further evaluate its performance in small-sample DG scenarios. ECTN also adopts a CNN-Transformer hybrid architecture and has demonstrated effectiveness in cross-condition tasks on the CWRU dataset. Under the same DG task settings, the multi-source to single-target (3→1) tasks are further extended by constructing six single-source to single-target tasks, as shown in Table 5.

4.2. Experimental settings and comparison methods

The input data for all datasets consist of single-channel raw vibration signals. Each signal segment is first preprocessed using fixed-length truncation and sample-wise $z$-score normalization before being fed into the model. The signal lengths adopted in the experiments are 3200, 6660, and 3200 for the PU, PHM09, and CWRU datasets respectively. To ensure fairness across different datasets, the core hyperparameters of the model are kept consistent across all experiments. The embedding dimension is set to $d=$ 72, the number of stacked Transformer blocks is depth = 3, the batch size is set to 64, the token length is 64, and the number of attention heads in the multi-head self-attention mechanism is 4. In addition, to control model complexity and prevent overfitting, dropout with a rate of 0.1 is applied in the feed-forward network (FFN) within the Transformer blocks.

The model is trained using the Adam optimizer with an initial learning rate of $lr=$ 0.003 and a minimum learning rate of ${lr}_{min}=$ 0.0001. A cosine annealing learning rate scheduler with warm restarts is adopted, where the initial cycle length is set to $T_0=$ 30 and the cycle multiplication factor is $T_{mult}=$ 2, and the maximum number of training epochs is 90. To ensure reproducibility, the random seeds of both NumPy and PyTorch libraries are fixed in all experiments. Each task is independently conducted five times, and the average diagnostic accuracy along with the corresponding standard deviation is reported. All experiments are conducted on an NVIDIA GeForce RTX 4060 Laptop GPU with the PyTorch 2.4.1 framework.

To comprehensively and fairly evaluate the performance of the proposed method, comparisons are conducted with several representative approaches, whose implementations are provided in the survey paper [2]. These methods include the baseline approach AGG [2]; domain alignment methods such as Domain-Adversarial Neural Network (DANN) [14], Maximum Mean Discrepancy (MMD) [28], CORAL [29], and Triplet loss [30]; DG methods including Multi-domain Mixup [31] and Meta-Learning DG (MLDG) [32]; and the ensemble-based method Domain Adaptive Ensemble Learning (DAEL) [33].

To further validate the effectiveness of the key components of the proposed method, four ablation variants are designed as internal baseline models. The first variant, denoted as Ours-A, is a CNN-only baseline that retains only the multi-scale CNN and global average pooling, with the PAM, LLU, and Transformer modules removed. The second variant, denoted as Ours-B, is constructed by adding a standard Transformer encoder to Ours-A. The third variant, denoted as Ours-C, further integrates the PAM module into Ours-B. The fourth and final variant, denoted as Ours-D, corresponds to the complete proposed model, incorporating the Transformer encoder along with both the PAM and LLU modules.

4.3. Main results analysis

4.3.2. Visualization analysis

To visually verify the discriminative capability of the features learned by the proposed method under unseen operating conditions, t-SNE visualization was performed on the PU dataset for Task T3, accompanied by confusion matrix analysis to further examine the classification behavior.

As shown in Fig. 3, the features learned by Ours-D exhibit clearly separable cluster structures in the feature space across the 14 fault categories, which include inner-race and outer-race faults with single-point, multi-point, and repeated distributions, as well as distributed fatigue pitting and plastic indentation. The intra-class samples are tightly clustered with clear boundaries maintained between different fault categories, thus achieving an overall diagnostic accuracy of 98.66 %. The main confusion occurs when Fault 7 is occasionally misclassified as Fault 1.

In contrast, the feature distributions of CORAL (88.4 %) and AGG (88.8 %) exhibit noticeable overlap among multiple fault categories. CORAL exhibits severe interference between Fault 7 and Fault 1 and also misclassifies a large number of Fault 9 samples (inner-race fatigue pitting with repeated distribution) as Fault 7, indicating its difficulty in distinguishing between inner-race and outer-race damage. Furthermore, its modeling capability for different fault distribution patterns, such as “repeated distribution” and “distributed defects,” remains limited.

Although AGG performs slightly better, it still misclassifies many Fault 9 samples as Fault 6 (multi-point fatigue pitting on both inner and outer races) and Fault 7. Additionally, Fault 13 (inner-race fatigue pitting with double points) is sometimes misclassified as Fault 9 (inner-race fatigue pitting with repeated distribution), suggesting insufficient sensitivity to subtle morphological differences between double-point and repeated-distribution pitting patterns.

These observations demonstrate that the PAM, by jointly modeling channel and temporal attention, effectively focuses on fault-sensitive regions that are invariant to operating condition variations. Meanwhile, the LLU enhances the perception of transient fault details such as indentation morphology, the number of pitting defects, and distribution patterns through local linear modeling. The synergy between these two modules enables the model to accurately distinguish between different fault categories even under the unseen operating conditions, significantly outperforming existing DG approaches.

Fig. 3t-SNE feature visualization and the corresponding confusion matrices for Task T3 on the PU dataset

4.3.4. Sensitivity analysis of key hyperparameters

To investigate the influence of local modeling scales in PAM and LLU on diagnostic performance, a total of 20 different parameter combinations were tested on Task T0 of the PU dataset, covering temporal kernel sizes in the PAM $k_t\in \left\{\mathrm{3,5},\mathrm{7,9}\right\}$ and depthwise kernel sizes in LLU $k_l\in \left\{\mathrm{3,5},\mathrm{7,9},11\right\}$. The average diagnostic accuracies are shown in Fig. 4.

The results indicate that $k_t=$ 7 and $k_l=$ 11 achieve the optimal diagnostic performance for the current task, reaching an accuracy of 57.16 %. Compared with the kernel size in PAM, the kernel size in LLU has a more pronounced impact on model performance. When $k_t$ is fixed, the diagnostic accuracy generally increases with larger $k_l$, suggesting that larger LLU kernels are beneficial for capturing long-period modulation envelope components in bearing fault signals. On the other hand, the PAM exhibits an optimal range of kernel sizes. Small kernels can precisely localize transient impacts but are more susceptible to noise interference, whereas excessively large kernels tend to oversmooth temporal attention weights, weakening the model’s ability to respond to critical impulses.

Fig. 4Impact of local modeling scales in the PAM and LLU on diagnosis performance (T0 of PU dataset)

The optimal combination $k_t=$ 7 and $k_l=$ 11 reflects the collaborative effect between PAM and LLU. This configuration maintains high sensitivity to transient fault features while enhancing the perception of fault evolution trends. In practical applications, the convolution kernel sizes of the PAM and LLU should be selected according to the physical characteristics of rotating machinery fault signals, such as impact duration and modulation frequency. Although the current study adopts $k_t=$ 7 and $k_l=$ 7 (accuracy 50.93 %) to balance computational efficiency and robustness, further performance improvements may still be achieved through appropriate kernel size tuning.