# On a three-dimensional neural network model

1Department of Natural Sciences and Mathematics, Daugavpils University, Daugavpils, Latvia

2Institute of Mathematics and Computer Science, University of Latvia, Riga, Latvia

1Corresponding author

Vibroengineering PROCEDIA, Vol. 47, 2022, p. 69-73. https://doi.org/10.21595/vp.2022.23059
Received 14 November 2022; received in revised form 1 December 2022; accepted 6 December 2022; published 12 December 2022

61st International Conference on Vibroengineering in Udaipur, India, December 12-13, 2022

Copyright © 2022 Diana Ogorelova, et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Editor's Pick
Views 18
CrossRef Citations 0
Abstract.

The dynamics of a model of neural networks is studied. It is shown that the dynamical model of a three-dimensional neural network can have several attractors. These attractors can be in the form of stable equilibria and stable limit cycles. In particular, the model in question can have two three-dimensional limit cycles.

Keywords: neural networks, mathematical modeling, attractors.

#### 1. Introduction

The theory of neural networks appeared as an attempt to understand the structure and principles of the functioning of the human brain. Now it is rich in results and practically significant field of research in natural sciences. Artificial neural networks (ANN) can be understood as computing systems inspired by biological neural networks. Their mathematical models can be formulated in terms of systems of quasi-linear differential equations of the form Eq. (1). Each dependent variable ${x}_{i}$ is associated with a neuron. It accepts signals from other neurons (this is called input) and elaborates its own signal (it is called output) which is sent to a network. The nonlinearity is called the response function, or activation function. Usually, a sigmoidal function like $f\left(z\right)=1/\left(1+\mathrm{e}\mathrm{x}\mathrm{p}\left(-z\right)\right)$ or $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(z\right)$ is used. Recent attempts to introduce other response functions may be found in -. The system:

(1)

Appears in neurodynamics , . It is of general nature, and for appropriate choice of parameters ${a}_{i}$ and ${b}_{j}$ it may have rich dynamics. Moreover, for sufficiently large $n$ it can approximate (on a finite interval) any dynamical system . The dynamics of solutions is a valuable object of investigation. Especially future states of a modeled neural networks are important to know. For this, the analysis of the phase space is needed. Future states are heavily dependent on attractors of the system, , . In this note we will show that the three-dimensional system of the form Eq. (1) can have stable equilibria in the form of stable focuses. For the appropriate choice of parameters, it can have limit cycles attracting other solutions.

#### 2. Two-dimensional system

Consider the system:

(2)
$\left\{\begin{array}{c}\frac{d{x}_{1}}{dt}=\mathrm{tanh}\left({a}_{11}{x}_{1}+{a}_{12}{x}_{2}\right)-{b}_{1}{x}_{1},\\ \frac{d{x}_{2}}{dt}=\mathrm{tanh}\left({a}_{21}{x}_{1}+{a}_{22}{x}_{2}\right)-{b}_{2}{x}_{2}.\end{array}\right\$

Proposition 1. System Eq. (2) can have stable critical points of the type stable focus.

Proof by construction the example. Set ${a}_{11}=k$, ${a}_{12}=$ 1.5, ${a}_{21}=$ –1.5, ${a}_{22}=k$, $k=$ 0.2, 1, see Fig. 1.

Proposition 2. System Eq. (2) can have a limit cycle.

Proof by constructing the example. Set ${a}_{11}=k$, ${a}_{12}=$ 1.5, ${a}_{21}=$ –1.5, ${a}_{22}=k$, $k=$ 1.2, 1, see Fig. 2.

Fig. 1. Stable focus as in Proposition 1 Fig. 2. Limit cycle as in Proposition 2 #### 3. Three-dimensional system

Consider the three-dimensional system of the form Eq. (1):

(3)
$\left\{\begin{array}{c}\frac{d{x}_{1}}{dt}=\mathrm{tanh}\left({a}_{11}{x}_{1}+{a}_{12}{x}_{2}+{a}_{13}{x}_{3}\right)-{b}_{1}{x}_{1},\\ \frac{d{x}_{2}}{dt}=\mathrm{tanh}\left({a}_{21}{x}_{1}+{a}_{22}{x}_{2}+{a}_{23}{x}_{3}\right)-{b}_{2}{x}_{2},\\ \frac{d{x}_{3}}{dt}=\mathrm{tanh}\left({a}_{31}{x}_{1}+{a}_{32}{x}_{2}+{a}_{33}{x}_{3}\right)-{b}_{3}{x}_{3}.\end{array}\right\$

Proposition 3. System Eq. (3) can have three limit cycles.

Proof by construction the example. Let the coefficient matrix in Eq. (3) be:

and see Fig. 4.

The nullclines for the system Eq. (3) are given by the relations:

(4)
$\left\{\begin{array}{c}0=\mathrm{tanh}\left({a}_{11}{x}_{1}+{a}_{12}{x}_{2}+{a}_{13}{x}_{3}\right)-{b}_{1}{x}_{1},\\ 0=\mathrm{tanh}\left({a}_{21}{x}_{1}+{a}_{22}{x}_{2}+{a}_{23}{x}_{3}\right)-{b}_{2}{x}_{2},\\ 0=\mathrm{tanh}\left({a}_{31}{x}_{1}+{a}_{32}{x}_{2}+{a}_{33}{x}_{3}\right)-{b}_{3}{x}_{3}.\end{array}\right\$

There are three periodic solutions. The respective trajectories are located in three planes (blue ones in Fig. 3). The critical points inside the limit cycles have the following characteristic numbers: ${\lambda }_{1}=$ –0.32, ${\lambda }_{2,3}=$ 0.2±1.5i for the critical points at (0; 0; ±0.65857). The central critical point (0;0;0) has ${\lambda }_{1}=$ 0.2, 0.2 ±1.5i. Trajectories go away from the central critical point.

Fig. 3. Nullclines of the system Eq. (3), matrix $A$ Fig. 4. Three periodic trajectories of the system Eq. (3) #### 4. Perturbation of three-dimensional system

Let the coefficients of the system Eq. (3) be perturbed as:

(5)
${A}_{1}=\left(\begin{array}{ccc}1.2& 1.5& 0.1\\ -1.5& 1.2& -0.1\\ -0.2& -0.2& 1.2\end{array}\right).$

Fig. 5. Nullclines of the system Eq. (3) with the coefficient matrix ${A}_{1}$ Fig. 6. Two periodic attractors of the system Eq. (3), coefficient matrix ${A}_{1}$, and converging some other trajectories. The middle limit cycles are destroyed There are still three critical points at (–0.051; –0.039;0.68723), (0; 0; 0), (0.051; 0.039; –0.68723). Their characteristic numbers are: ${\lambda }_{1}=$ –0.3543, ${\lambda }_{2,3}=$ 0$.$19±1.4946i for the first and the third critical points, and 0.2266, 0.1866±1.5i for the central point.

Let the coefficients of the system Eq. (3) be perturbed as:

(6)
${A}_{2}=\left(\begin{array}{ccc}1.2& 1.5& 1.3\\ -1.5& 1.2& -0.1\\ -0.4& -0.1& 1.2\end{array}\right).$

There are still three critical points at (0.0527; 0.6533; –0.76188), (0;0;0), (–0.0527; –0.6533; 0.76188). Their characteristic numbers are: ${\lambda }_{1}=$ –0.4434, ${\lambda }_{2,3}=$ 0.089±1.1882i for the first and the third critical points, and ${\lambda }_{1}=$ 0.2921, ${\lambda }_{2,3}=$ 0.1539±1.6632™ for the central point.

Fig. 7. Nullclines of the system Eq. (3) with the coefficient matrix ${A}_{2}$ Fig. 8. Two periodic attractors of the system Eq. (3), coefficient matrix ${A}_{2}$, and converging trajectories #### 5. Conclusions

Dynamical systems, arising in neurodynamic, can have periodic attractors in the form of the limit cycles. Periodic attractors in two-dimensional systems appear as the result of Andronov-Hopf bifurcation, where the bifurcating parameter is the value at the main diagonal of the coefficients matrix $A$. Three-dimensional systems can have two attractors in the form of limit cycles. Small perturbation of the coefficient matrix $A$ can destroy limit cycles lying in a non-stable manifold (that is, in the middle plane in the example above). It seems that two side attractors (which were in stable manifolds) are preserved under the perturbations saving the types of the corresponding critical points. For practical purposes, special attention should be paid to perturbations that preserve the structure of the nullclines and, as a consequence, characteristics of equilibria (critical points).

#### Acknowledgements

ESF Project No. 8.2.2.0/20/I/003 “Strengthening of Professional Competence of Daugavpils University Academic Personnel of Strategic Specialization Branches 3rd Call”.

1. S. Haykin, Neural Networks. A Comprehensive Foundation. Prentice Hall, 1998. [Search CrossRef]
2. J. C. Sprott, Elegant Chaos. Singapore: World Scientic, 2010. [Publisher]
3. K.-I. Funahashi and Y. Nakamura, “Approximation of dynamical systems by continuous time recurrent neural networks,” Neural Networks, Vol. 6, No. 6, pp. 801–806, Jan. 1993, https://doi.org/10.1016/s0893-6080(05)80125-x [Publisher]
4. D. Ogorelova, F. Sadyrbaev, and V. Sengileyev, “Control in inhibitory genetic regulatory network models,” Contemporary Mathematics, Vol. 1, Oct. 2020, https://doi.org/10.37256/cm.152020538 [Publisher]
5. I. Samuilik and F. Sadyrbaev, “Modelling three dimensional gene regulatory networks,” WSEAS Transactions on Systems and Control, Vol. 16, pp. 755–763, 2021. [Publisher]
6. A. Afzal, J. K. Bhutto, A. Alrobaian, A. Razak Kaladgi, and S. A. Khan, “Modelling and computational experiment to obtain optimized neural network for battery thermal management data,” Energies, Vol. 14, No. 21, p. 7370, Nov. 2021, https://doi.org/10.3390/en14217370 [Publisher]
7. S.-L. Shen, N. Zhang, A. Zhou, and Z.-Y. Yin, “Enhancement of neural networks with an alternative activation function tanhLU,” Expert Systems with Applications, Vol. 199, p. 117181, Aug. 2022, https://doi.org/10.1016/j.eswa.2022.117181 [Publisher]
8. E. Trentin, “Networks with trainable amplitude of activation functions,” Neural Networks, Vol. 14, No. 4-5, pp. 471–493, May 2001, https://doi.org/10.1016/s0893-6080(01)00028-4 [Publisher]
9. H. Zhu, H. Zeng, J. Liu, and X. Zhang, “Logish: A new nonlinear nonmonotonic activation function for convolutional neural network,” Neurocomputing, Vol. 458, pp. 490–499, Oct. 2021, https://doi.org/10.1016/j.neucom.2021.06.067 [Publisher]