Approach to predictability via anticipated synchronization

Predictability of chaotic systems is limited, in addition to the precision of the knowledge of the initial conditions, by the error of the models used to extract the nonlinear dynamics from the time series. In this paper, we analyze the predictions obtained from the anticipated synchronization scheme using a chain of slave neural network approximate replicas of the master system. We compare the maximum prediction horizons obtained with those attainable using standard prediction techniques.


I. INTRODUCTION
Modeling and predicting the dynamics of nonlinear chaotic systems is a challenging problem with important realworld applications ͑stock market returns ͓1͔, weather forecast ͓2͔, etc.͒.It is well known that chaotic behavior implies long-term unpredictability, but the deterministic nature of chaotic systems allows the prediction of their dynamics to some extent.From a theoretical point of view, Lyapunov theory provides a sound framework for this problem, and the inverse of the largest Lyapunov exponent gives a theoretical limit for the prediction horizon attainable for a particular system.However, in practice, the original system is unknown and approximate models fitted to the available data are used to model and forecast its nonlinear dynamics ͑e.g., neural networks ͓3͔͒.In this situation, the attainable forecast horizon depends not only on the dynamics of the original system, but also on the error of the approximate model.
Anticipated synchronization is a recently discovered intriguing phenomenon which allows one to predict the dynamics of nonlinear systems using a particular achronal master-slave synchronization scheme ͓4͔.Theoretically, an arbitrary large forecast horizon can be obtained using a chain of anticipated synchronized exact replicas ͑slaves͒ of the original system ͓5͔ ͓see Fig. 1͑a͔͒.However, it has been recently shown that this scheme is unstable to propagating perturbations ͑the spatiotemporal character of the coupled chain introduces a convectivelike instability into the synchronization manifold͒ ͓6͔.Therefore, in a practical problem where the slave systems are approximate replicas, there is no information about the maximum attainable anticipation time.
In this paper, we analyze this problem using neural networks, one of the most popular nonparametric statistical learning techniques, for approximating the nonlinear dynamics from the available data ͑time series͒ ͓7͔.The obtained neural models are used as slaves in the anticipated synchro-nization scheme ͓see Fig. 1͑b͔͒ and the results are compared with the exact-replicas case.Moreover, the practical forecast horizon obtained is compared with an alternative standard forecasting method ͓forward iteration of the neural model from the initial state, Fig. 1͑c͔͒.
This paper is structured as follows.In Sec.II we analyze the mechanism underlying anticipated synchronization and describe its limitations: we show that the maximum attainable anticipation values with a master-slave scheme is close to the linear prediction time given by the autocorrelation ͑Sec.II A͒.Section II B briefly describes the scheme with chain of systems which gives larger anticipation times.Section III presents some basic results about neural networks and their application to chaotic systems forecasts.Section III A shows the prediction performance of anticipated synchronization using neural networks as slave models.Section III B compares the results with a standard technique.Finally, some conclusions are given in Sec.IV. tonomous systems u ˙0͑t͒ = f(u 0 ͑t͒), u ˙1͑t͒ = f(u 1 ͑t͒), several schemes have been presented in the literature ͓8-10͔ to achieve synchronization of their outputs.The simplest synchronization scheme uses a dissipative coupling between them: It might happen that, for particular values of coupling parameters matrix K, the fixed point solution u 0 ͑t͒ = u 1 ͑t͒ is globally asymptotically stable.In this case, the dynamics of both systems will be restricted ͑after some transient͒ to the synchronization manifold u 0 ͑t͒ϵu 1 ͑t͒ and, hence, they will exhibit identical dynamical behavior.In most cases, no analytical results about stability are possible, and synchronization regimes must be obtained numerically.
A different approach, named anticipated synchronization, was proposed recently by Voss ͓4͔ and it has attracted a lot of attention because of its potential applications for predicting the dynamics of chaotic systems.Anticipated synchronization describes the situation when the slave system becomes synchronized with the future output of the master system by appropriately including some delay times in feedback terms.Amongst the different possibilities, we restrict ourselves in this paper to the anticipated synchronization scheme of the form Note the inclusion of the delay time in the equation of the slave.In this case, the synchronization manifold is u 1 ͑t͒ = u 0 ͑t + ͒ and when it is stable, it allows the slave to anticipate by a time the dynamics of the master.This achronal synchronization occurs for some bounded region of coupling parameters K and delay times .Since its proposal by Voss, several papers have analyzed theoretically anticipated synchronization in chaotic dynamical systems ͓11͔ and chaotic maps ͓12͔.In addition, some experiments have been carried out, demonstrating the existence of this phenomenon in electronic chaotic circuits ͓13͔, in semiconductor lasers ͓14͔, and in an electronic circuits implementation of the FitzHugh-Nagumo neuron model ͓15͔.
We illustrate the scheme with two benchmark chaotic systems, the Note that the theoretical prediction horizons of both systems differ by one order of magnitude.We first take two identical Lorenz systems u 0 and u 1 , and couple them by using the scheme of Eq. ͑2͒ with coupling only in the x variable, i.e., K(u 0 ͑t͒ − u 1 ͑t − ͒) = ͕K͓x 0 ͑t͒ − x 1 ͑t − ͔͒ ,0,0͖.Figure 2͑a͒ shows the stability region of the anticipated manifold u 1 ͑t͒ = u 0 ͑t + ͒ by plotting in a gray scale the maximum of the cross-correlation function between x 1 ͑t͒ and x 0 ͑t + ͒.From this figure, we estimate that the maximum anticipation time, = 0.13, is reached for K =19 ͑see Fig. 3͒.A similar analysis ͑not shown͒ for the Rössler system shows a maximum anticipation time = 0.91 for K = 0.5.In both cases, these maximum anticipation times are shorter than the inverse of the largest Lyapunov exponents and, in fact, they are similar to the linear prediction times of the original systems: 0.16 ͑Lorenz͒ and 0.95 ͑Rössler͒.These values are obtained as the horizon where the error of a linear prediction is larger than 5% of the system's range, and agree with those values obtained from the autocorrelation function.This suggests that the anticipated synchronization mechanisms is limited to a neighborhood of t, where u 1 ͑t − ͒ can be linearly approximated in terms of u 1 ͑t͒.In the following section we theoretically derive this relationship.

A. First-order approximation
In order to eliminate the delay term u 1 ͑t − ͒ from Eq. ͑2͒ we consider a linear expansion u 1 ͑t − ͒ = u 1 ͑t͒ − u ˙1͑t͒, and substitute it into the equation.Thus, we get rearranging terms in the derivative, we get Thus, using a first-order approximation, the anticipated synchronization scheme reduces to a nonanticipated one but with different drive and response systems.Then the synchronization manifold is not u 1 ͑t͒ = u 0 ͑t͒, but u 1 ͑t͒ = u 0 ͑t͒ + f(u 0 ͑t͒) in the first-order approximation.This approach has been used to obtain anticipated synchronization in an array of chaotic rf circuits ͓19͔.The time evolution for the response system given by Eq. ͑6͒ corresponds to that of the drive system but with a different time scale tЈ = t / ͑1−K͒.
Anticipation requires tЈ Ͼ t or, ͑at first order͒ 0 Ͻ 1−K Ͻ 1, giving the following two constraints for anticipated synchronization K Ͼ 0 and K Ͻ 1.This simple prediction has been compared with the numerical synchronization diagrams of the Lorenz and Rössler systems in the case that all variables are coupled using a diagonal coupling matrix K = K1 ͓this has been chosen in order to improve the quality of the approximation in Eq. ͑5͔͒.
Figure 4͑a͒ shows the stability region for the Lorenz system together with the simple prediction curve K = 1.For comparison, we have included in Fig. 4͑b͒ the synchronization diagram coming directly from the approximation scheme Eq. ͑6͒.Note the similarity between the two figures so confirming the validity of our simple approximation.The bound K Ͼ 0 is only a necessary condition for the anticipated synchronization, but it turns out that a minimum coupling value is required in order to achieve synchronization.Figures 4͑c͒  and 4͑d͒ show the equivalent results for the Rössler system.
The results shown in Fig. 4 show that in these chaotic systems anticipated synchronization can be effectively considered as the standard synchronization scheme between two nonidentical systems, in which the slave runs at a different time scale than the master.This result complements that in Ref. ͓20͔ showing that the anticipated synchronization in excitable systems is determined by the fact that the delayed coupling term lowers the excitability threshold of the slave through changing the parameters of that system.In both cases the response time of the slave system is shorter due to the effect of the coupling.
It might look deceiving that the anticipated synchronization scheme discussed here cannot forecast longer than the linear prediction time and much less than the inverse of the largest Lyapunov exponent.We will see in the next section that the situation improves dramatically when we consider a chain of coupled systems.In this case, the anticipation time can be made much larger than any of those characteristic times mentioned above.

B. Coupling of systems in a chain
To make anticipated synchronization longer it is necessary to consider a chain of slave systems.Voss ͓5͔ already considered a chain of coupled systems and showed that the stability of the system can be enhanced and larger anticipation times can be obtained.The behavior of a chain of connected systems has been also studied for the FitzHugh-Nagumo neuron model subjected to noise ͓21͔.There it has been shown that coupling of above systems in a chain decreases the number of errors in the response neurons and makes anticipated synchronization more stable.
A chain of identical N + 1 unidirectionally coupled systems is defined in the following way: FIG. 2. Maximum of the cross-correlation function in the K and parameter space for anticipated synchronization scheme with identical copies of ͑a͒ one Lorenz slave, ͑b͒ ten slaves, ͑c͒ twenty slaves.Panels ͑d͒-͑f͒ as in ͑a͒-͑c͒ but for the case of slave neural networks.White ͑black͒ color corresponds to high ͑low͒ correlations.
where subsystem u 0 is the master and subsystems u i , i =1, ... ,N are slaves.In this scheme, convectivelike instabilities introduced by the spatiotemporal character of the chain reduces the maximum delay time in each unit as compared to the case of a single slave ͓6͔.However, since the anticipation time of the Nth slave as compared to the master is N, the total prediction time can be much larger than that of the single slave scheme.Figures 2͑b͒ and 2͑c͒ show the maximum of the cross correlation function in the K vs plane for a chain of N = 10 and N = 20 coupled Lorenz systems, respectively.These figures illustrate how, although the delay time decreases with N, the total anticipation time N increases significantly with N. The set of Figs.5͑a͒-5͑c͒ show a typi-cal time series obtained by coupling a single, twenty, and two hundred slave systems.For the latter we have reached an anticipation time of four time units, much larger than the characteristic times of the Lorenz system for which the inverse largest Lyapunov exponent is 1.1.Similar results ͑not shown͒ were obtained for the Rössler system.
A word of caution is necessary here.The above results consider that the slaves are perfect copies of the master.In other words, that we know exactly the master's equations of motion and parameters.In most practical situations it might not be possible to know the functional form of the chaotic dynamics and only a time series of the system dynamics is available.Then approximate models can be obtained and used as slave systems ͓22͔.Neural networks is one of the most popular learning methods for this task.This is the topic of the next section.

III. MODELING CHAOTIC SYSTEMS WITH NEURAL NETWORKS
The deterministic low-dimensional nature of the chaotic systems considered in this paper allows us to reconstruct its functional structure from a time series using appropriate nonlinear techniques.In recent years new approaches for nonlinear time series modeling have emerged ͑local and global prediction ͓23͔, neural networks ͓24͔, delay embedding reconstruction space ͓25͔, functional networks ͓26͔, etc.͒, providing more powerful methods and giving new insight into the dynamics of these systems ͑see Refs.͓3,7͔, and references therein for an updated survey of this topic͒.Among these techniques, artificial neural networks have been successfully applied in many practical situations ͓27,29͔.Moreover, it has been shown that the neural approximate model and the original system exhibit similar dynamical behavior ͑similar unstable periodic orbits ͓30͔, or even similar Lyapunov exponents or fractal dimensions ͓31͔͒.We will now briefly review the main points of this approach.
Let us assume that we have a time series u n , obtained from a dynamical system u ˙͑t͒ = f(u͑t͒), sampled at equally spaced intervals u n = u͑n⌬t͒, n =0,1,2, ... .We are interested in approximating the functional model which characterizes the short-term evolution of the time series, u n+p = F͑u n ͒, where F is given in terms of f, the sampling time ⌬t, and the prediction step p.To this aim we consider simple feed-forward neural networks with sigmoidal and linear activation functions for hidden and output layers, respectively.This type of network has shown to be an universal approximation for continuous ͑one hidden layer͒ or arbitrary ͑more than one hidden layer͒ functions ͓32͔.The training process is carried out by considering input-output couples of the form ͑u n , u n+p ͒, where p is the prediction step.
In this paper we use time series consisting of 2000 sample points obtained from the Lorenz and Rössler systems, Eqs.͑3͒ and ͑4͒, respectively.The equations were integrated using a fourth-order Runge-Kutta algorithm with a fixed time step of ⌬t = 0.002 and ⌬t = 0.01 for the Lorenz and Rössler systems, respectively.The time series were sampled at steps of t = 0.01; in both cases the prediction step p = 1.The sets were divided in two parts; the first one ͑1500 sample points͒ was used for training purposes whereas the last 500 were reserved for testing the models.We have considered different neural networks with three input neurons ͑x n , y n , z n ͒, three output neurons ͑x n+1 , y n+1 , z n+1 ͒, and a single hidden layer with a number of neurons ranging from one to twenty ͑this type of architecture is usually referred to as a 3 : a : 3 feedforward network, where a is the number of hidden neurons͒.For each of these network structures, ten simulations were performed with different initial network weights, using the Levenberg-Marquardt method ͓28͔ as training algorithm.The best solution in each case was considered as the representative neural approximate model.For instance, the root-meansquare ͑r.m.s.͒ error obtained for predicting the x variable of the Lorenz model with the best neural network with six hidden neurons for the training process was 0.13 ͑less than 0.5% the range of the corresponding variable͒, and 0.15 for the test data, indicating no over-fitting of the model.However, although the above analysis indicates a good accuracy in onestep ahead prediction using a six neuron network, it does not mean necessarily that the obtained neural model can reproduce the dynamics of the Lorenz system when iterated in time ͓29͔.Figure 6 shows the evolution of the above network with a = 6 hidden neurons iterated from two different initial weight configuration; in the first case, the neural system converges to a periodic trajectory ͓Fig.6͑a͔͒, whereas in the second case it converges to a fixed point ͓Fig.6͑b͔͒, neither of them resembling the chaotic behavior of the Lorenz model.When increasing the number of hidden neurons above a = 10, the error decreases and the dynamical behavior of the obtained neural models resembles the original chaotic system.For instance, the training and test rms errors obtained for a = 10 hidden neurons were 2.2ϫ 10 −2 and 2.4 ϫ 10 −2 , respectively, indicating that no over-fitting occurs.In this case, the dynamical behavior of the neural model re-sembles that of the original system.Thus, the neural model can be considered as an approximate replica of the Lorenz model.If we increase the number of hidden neurons above a = 20, the training error decreases even further but the neural models start over-fitting the data.In the following we use the neural model with a = 10 hidden neurons.
Once we have established that the neural network is a good copy of the dynamical system, the next step is to use this copy as the slave in a synchronization scheme.For in-FIG. 4. Maximum of the cross-correlation function in the K and parameter space for anticipated synchronization scheme with an identical copy of a Lorenz slave for ͑a͒ anticipated synchronization scheme ͓Eq.͑2͔͒ and ͑b͒ the firstorder approximation scheme ͓Eq.͑6͔͒.Panels ͑c͒, ͑d͒ as in ͑a͒, ͑b͒ but for the Rössler system.The dashed lines in the top-right corners corresponds to the curves K = 1.White ͑black͒ color corresponds to high ͑low͒ correlations.stance, let us consider the simplest scheme Eq. ͑1͒ where u 0 is the Lorenz system and u 1 is its neural network copy with a = 10 hidden neurons.Figure 7͑c͒ shows the evolution of the synchronization error, measured as the difference x 1 n − x 0 n between the x variables of the master and the slave.Notice that after n = 500 steps, the two systems synchronize rather well although there is a residual error.In the case that the slave is a perfect copy the error vanishes after the transient time, as shown in Fig. 7͑a͒.The residual error observed in the synchronization of the neural network copy can be assimilated to the slave being an imperfect copy of the master.To quantify this imperfection, we have used as slave a Lorenz system with mismatch parameters = 10.1, r = 27.5, and b =8/3, i.e., assuming a certain mismatch in two of the parameters.As shown in Fig. 7͑b͒ the resulting synchronization errors are similar to that of the neural network.From these figures, we could state that the dynamical accuracy of the neural approximate model is roughly equivalent to a 2% mismatch in the system's parameters.
In the case of the Rössler model we performed the same simulations obtaining a similar behavior.A neural network with 12 hidden neurons was found to be appropriate for approximating the system dynamics.In this case, the obtained rms errors were 2.4ϫ 10 −4 and 3.2ϫ 10 −4 for the training and test data.

A. Anticipated synchronization with neural network replicas
In this subsection we present the results of the general anticipated synchronization scheme Eq. ͑7͒ using as master u 0 the Lorenz system, and as a chain of slaves u 1 , ... ,u N , N identical neural network models as discussed previously.Figure 2͑d͒ shows the maximum of the cross correlation function in K vs plane obtained when coupling the Lorenz system to N = 1 neural network with two hidden layers each containing four neurons.Comparing ͑a͒ and ͑d͒ we observe the reduction of the stability region when the slave is a neural network with respect to the identical slaves case.In fact, the performance of the chain of neural network slaves worsens as the number N increases, see Figs. 2͑e͒ and 2͑f͒.Therefore, we get a maximum anticipation time of 0.33 for N = 20, which has to be considered as the maximum prediction horizon for the Lorenz system using the anticipated synchronization scheme with neural network slaves ͑see Fig. 3͒.A similar analysis in the Rössler model using this technique yields a maximum anticipation time of 4.1 for N = 20.These prediction horizons are much shorter than those obtained with anticipated synchronization when the slaves are perfect copies of the master, as discussed in the previous section.

B. Standard prediction with neural networks
We now compare the use of the anticipated synchronization scheme with neural networks as slave systems with the standard prediction technique consisting on iterating forward in time the neural approximate model from an initial condition.Whereas in the former method the maximum anticipation time does not depend on the initial point, it turns out that in the latter case, the attainable forecast horizon depends on the precise location of the initial condition within the attractor.
The forecast horizon T is defined as the time it takes for the absolute value of the difference between the neural network reconstructed orbit and the actual orbit to be larger than 2%.As shown in Figs.8͑a͒ and 8͑b͒, lower horizon values correspond to initial conditions in the unstable regions of the attractor where transitions are more likely to occur.More detailed information is obtained from the histogram of horizon times, as shown in Figs.9͑a͒ for the Lorenz and Fig. 9͑b͒ for the Rössler systems.The distributions are quite broad and the mean values are ͗T͘ = 1.12 and ͗T͘ = 11.8, respectively.These values are close to the inverse of the largest Lyapunov exponent.In the same figures we have indicated the location of the maximum prediction times using anticipated synchronization combined with neural networks.It is worth mentioning that the anticipation times using perfect replicas of the master system are beyond the shown scale for the x axis.Note that the anticipating horizon obtained using standard prediction with neural networks is larger ͑in average͒ than the values obtained with anticipated synchronization ͑dashed lines in Fig. 9͒.In the latter, the horizon is constant over the attractor whereas the standard prediction approach only pro-vides an average forecast value.Thus, depending on the type of application, one method would be better than the other.

IV. CONCLUSIONS
In this paper, we have considered two alternative practical techniques to anticipate the dynamics of chaotic systems.On the one hand, a neural network trained to the available data which can be iterated forward in time reproducing the same orbit of the chaotic system up to a given horizon.One shortcoming of this method is that the prediction horizon, close to the inverse of the largest Lyapunov exponent, can be only given in probabilistic terms.The second technique which we have considered is a chain of neural networks replicas of the master system combined with the anticipated synchronization scheme.In this case the horizon anticipation time adopts a fixed value and the numerical simulations show that the maximum horizon is shorter than the one obtained by iterating the neural network.These results are closely related to the differences between the perfect knowledge of the chaotic system and its neural network reconstruction.The mismatch is produced during the, necessarily imperfect, training of the neural network.
The forecast horizon in the anticipated synchronization scheme with a single slave is of the order of the linear cor- relation time of the system.A chain of slaves allows us to overcome this limitation and, when the slaves are perfect copies, we obtained anticipation times which are much longer than the prediction horizon obtained by iterating the neural network.Thus for effective implementation of the combined scheme of neural networks with anticipated synchronization it is essential to have a more accurate reproduction of the dynamics of the master system.It would be interesting to continue the study with such a network and we are convinced that it could exploit the predictive power of the anticipated synchronization scheme applied to a chain of coupled systems ͓33͔.

FIG. 3 .
FIG.3.Time series of variable x͑t͒ of master Lorenz system and ͑a͒ first slave neural network with i = 0.08, K = 0.2, ͑b͒ tenth slave neural network with i = 0.03, K = 0.4, and ͑c͒ twentieth slave neural network with i = 0.01, K = 0.7 for i =1, . . .,N, where N is a number of slave neural networks in a chain ͑slaves are drawn with dotted line and master with solid line͒.

FIG. 5 .
FIG. 5. Time series of the master Lorenz system ͑solid line͒ and ͑a͒ the first slave, ͑b͒ the twentieth slave, and ͑c͒ the two hundredth slave.Time series for slave are drawn with dashed line.

FIG. 6 .
FIG.6.͑Color online͒ Phase space of two different 3:6:3 neural models trained with the same method, but starting from different initial weight configurations.The shadow in the background corresponds to the original chaotic orbit and is shown for illustrative purposes.

FIG. 8 .
FIG. 8. Residuals x n − x ˆn for two neural models with ͑a͒ Histogram produced by 10 000 points on the attractor; ͑b͒ histogram of the averaged prediction horizon obtained with a neural network.