Autoregressive (1) Model

1. AR(1) model

First order autoregression process, AR(1), is defined as \[ X_t = \phi X_{t-1} + \epsilon_t, \hspace10mm \mbox{ where } \epsilon_t \sim WN(0,\sigma^2) \] and $\phi$ is real-valued constant.

If $\phi=.8$, then \[ X_t = (.8) X_{t-1} + \epsilon_t \]

For example, process goes like:

\[ \begin{array}{|c|c|c|c|c|c|c|c|} \hline t & 0 & 1 & 2 & 3 & 4 & 5 \cdots \\ \hline \phi X_{t-1} & & .80 & .91 & 1.26 & 0.28 & \cdots\\ \epsilon_t & & .34 & .66 & -.91 & -.35 & \cdots\\ \hline X_t & 1 & 1.14 & 1.57 & 0.35 & -.07 & \cdots\\ \hline \end{array} \]

Note that $\epsilon_t$ is the only random error (innovation) generated each time.

Inside AR(1)

Questions:

Is this series stationary? Does $\phi$ has anything to do with the stationarity?

What is $E(X_t)$, $V(X_t)$ and $\gamma(h)$ of $\rho(h)$ of this process (ACVF and ACF)?

Stationarity Condition for AR(1)

We need $|\phi|<1$ for AR(1) process to be stationary.

  #- Simulate AR1 with phi > 1
  Y   <- 0.5                     # initial value of Y
  phi <- 1.1              # <--- greater than 1
  e <- rnorm(100, 0, 1)        # generate 100 N(0,1) error
  for (t in 2:100){
    Y[t] = Y[t-1]*phi + e[t]   # compute AR(1) sequence
  }
  plot(Y, type="o")

  #- Simulate AR1 with phi<1
  Y   <- 0.5
  phi <- 0.8            # <--- absolute value less than 1
  e <- rnorm(100, 0, 1)        # generate 100 N(0,1) error
  for (t in 2:100){
    Y[t] = Y[t-1]*phi + e[t]   # compute AR(1) sequence
  }
  plot(Y, type="o")

  #- Simulate AR1 with |phi|<1
  Y <- 0.5
  phi <- -0.8           # <--- absolute value less than 1
  e <- rnorm(100, 0, 1)        # generate 100 N(0,1) error
  for (t in 2:100){
    Y[t] = Y[t-1]*phi + e[t]   # compute AR(1) sequence
  }
  plot(Y, type="o")

AR(1) is Random Walk if $\phi=1$

If $\phi=1$, then \[ X_t = X_{t-1} + \epsilon_t. \] This is called Random Walk.

  #- Simulate AR1 with phi=1
  Y <- 0.5
  phi <- 1
  e <- rnorm(100, 0, 1)        # generate 100 N(0,1) error
  for (t in 2:100){
    Y[t] = Y[t-1]*phi + e[t]   # compute AR(1) sequence
  }
  plot(Y, type="o")

  #- Simulate 50 AR1 with phi=1
  phi <- 1
  for (i in 1:50) {
    Y <- 1
    e <- rnorm(100, 0, 1)        # generate 100 N(0,1) error
    for (t in 2:100){
      Y[t] = Y[t-1]*phi + e[t]   # compute AR(1) sequence
    }
    plot(Y, type="l", ylim=c(-30,30))
    par(new=TRUE)              # overlay all the plot that comes after
  }

Random Walk is not a stationary series, because the variance depends on $t$.

Mean of AR(1)

If $\phi=.8$, then \[ E( X_t ) \hspace3mm = \hspace3mm E( (\phi) X_{t-1} + \epsilon_t ) \hspace3mm = \hspace3mm \phi E( X_{t-1} ) + E(\epsilon_t) \]

It will be shown in the next section that \[ E(X_t) = 0 \]

2. Theoretical ACVF of AR(1)

Assume that the AR(1) is stationary. (i.e. $|\phi|<1$)

What is the theoretical ACVF of AR(1)? Let’s start with variance ($h=0$).

\[ \begin{aligned} \gamma(0) &= \mbox{Var}(X_t) \hspace3mm = \hspace3mm \mbox{Cov}(X_t, X_t) \\\\ &= \mbox{Var}(\phi X_{t-1}+\epsilon_t) \hspace20mm \mbox{ (because $X_{t-1}$ and $\epsilon_t$ are independent, ) } \\\\ &= \mbox{Var}(\phi X_{t-1}) + \mbox{Var}(\epsilon_t) \\ \\ &= \phi^2 \mbox{Var}(X_{t-1}) \hspace3mm + \hspace3mm \sigma^2 \hspace3mm = \hspace3mm \phi^2 \, \gamma(0) + \sigma^2 \end{aligned} \]

Given \[ \gamma(0) \hspace3mm = \hspace3mm \phi^2 \, \gamma(0) + \sigma^2, \] solve for $\gamma(0)$, we get formula for variance of $X_t$, \[ \gamma(0) \hspace3mm = \hspace3mm \frac{ \sigma^2 }{ (1-\phi^2) }. \]

Do you see what’s wrong if we have $|\phi|>1$?

When $h=1$

Still assuming the stationarity, let’s look at when $h$ is not 0, \[ \begin{align} \gamma(1) &= \mbox{Cov}(X_{t+1}, \, X_t) \\ \\ &= \mbox{Cov}\big(\phi \, X_{t} +e_{t+1}, \, \hspace2mm X_t\big) \\ \\ &= \mbox{Cov}\big(\phi \, X_{t}, \, X_t\big) + \mbox{Cov}\big(e_{t+1}, \, X_t\big) \\ \\ &= \phi \, \mbox{Cov}(X_t, X_t) \hspace5mm = \hspace2mm \phi\, \gamma(0) \end{align} \]

When $h=2$

\[ \begin{align} \gamma(2) &= \mbox{Cov}(X_{t+2}, \, X_t) \\ \\ &= \mbox{Cov}\big(\phi\, X_{t+1} +e_{t+2}, \, \hspace2mm X_t\big) \\ \\ &= \mbox{Cov}\big(\phi\, X_{t+1}, \, X_t\big) + \mbox{Cov}\big(e_{t+2}, \, X_t\big) \\ \\ &= \phi\, \mbox{Cov}(X_{t+1}, X_t) \hspace5mm = \hspace2mm \phi\, \gamma(1) \hspace5mm = \hspace2mm \phi^2 \gamma(0) \end{align} \]

ACF and ACVF of AR(1)

So the ACVF of AR(1) looks like \[ \gamma(h) = \left\{ \begin{array}{ll} \frac{ \sigma^2 }{ (1-\phi^2) } & \mbox{ for } h=0 \\\\ \phi^{|h|} \, \gamma(0) & \mbox{ for } h>0\\ \end{array} \right. \]

Then since ACF = ACVF $/\gamma(0)$, \[ \rho(h) = \left\{ \begin{array}{ll} 1 & \mbox{ for } h=0 \\ \phi^{|h|} & \mbox{ for } h>0\\ \end{array} \right. \]

3. Theoretical ACVF in R

Let $\phi=.8$. i.e. $\hspace5mm \Phi(z) = 1 - .8 z$

  #  install.packages("ltsa")        #- if not installed before

  library(ltsa)                      # load the package ltsa
  Th.ACVF  <- tacvfARMA(phi = c(.8), theta= c(0), maxLag=20, sigma2=1)   # get Theoretical ACVF
  Th.ACVF                            # see what's inside

##  [1] 2.77777778 2.22222222 1.77777778 1.42222222 1.13777778 0.91022222 0.72817778
##  [8] 0.58254222 0.46603378 0.37282702 0.29826162 0.23860929 0.19088744 0.15270995
## [15] 0.12216796 0.09773437 0.07818749 0.06254999 0.05004000 0.04003200 0.03202560

  Th.ACF   <- Th.ACVF / Th.ACVF[1]   # get Theo. ACF
  Th.ACF

##  [1] 1.00000000 0.80000000 0.64000000 0.51200000 0.40960000 0.32768000 0.26214400
##  [8] 0.20971520 0.16777216 0.13421773 0.10737418 0.08589935 0.06871948 0.05497558
## [15] 0.04398047 0.03518437 0.02814750 0.02251800 0.01801440 0.01441152 0.01152922

  layout( matrix(1:2, 1,2) )             # two plots side by side
  plot(0:20, Th.ACVF, type="h", col="red")
  abline(h=0)                            # add horizontal line at 0
  plot(0:20, Th.ACF,  type="h", col="red")
  abline(h=0)

  #-  Another function for Theoretical ACF  (ACF only, you don't need ltsa package)
  Th.ACF2  <- ARMAacf(ar=c(.8), ma=c(0), lag.max=20, pacf=FALSE )
  Th.ACF2

##          0          1          2          3          4          5          6 
## 1.00000000 0.80000000 0.64000000 0.51200000 0.40960000 0.32768000 0.26214400 
##          7          8          9         10         11         12         13 
## 0.20971520 0.16777216 0.13421773 0.10737418 0.08589935 0.06871948 0.05497558 
##         14         15         16         17         18         19         20 
## 0.04398047 0.03518437 0.02814750 0.02251800 0.01801440 0.01441152 0.01152922

4. Simulation of AR(1)