Accuracy function in R is misleading

  arima, forecasting, r, split, windows

I use the accuracy function in the forecast package to get model accuracy metrics of a time series I obtained through the following steps:

  1. Simulate a 10 sample of AR(1) series.
  2. Split the series into subseries of size 2 without overlapping.
  3. Resample the subseries 1000 times with replacement.
  4. Form a new series by joining all the resampled subseries
  5. Check the accuracy of the newly formed series.

as follows:

## simulate ARIMA(1,0, 0)
set.seed(289805)
N10_ar0.8_seed289805 <- arima.sim(n=10, model=list(ar=0.8, order=c(1, 0, 0)), sd=1)
auto.arima(N10_ar0.8_seed289805, ic="aicc")

########################################################
l <- 2 # block size OF 2                                # number of blocks
blk <- split(N10_ar0.8_seed289805, ceiling(seq_along(N10_ar0.8_seed289805)/l))  # divides the series 
into blocks of size 2
######################################################
set.seed(5)
res<-sample(blk, replace=T, 1000)        # resamples the blocks
res.unlist <- unlist(res, use.names = FALSE)   # unlist the bootstrap series
mod <- forecast::auto.arima(res.unlist) # obtain the best model
fit <- fitted(mod) # fit the model with the series
ACCURACY <- forecast::accuracy(fit, res.unlist)      # RETURNS ACCURACY
ACCURACY

I get the below result:

#                    ME     RMSE      MAE      MPE     MAPE
# Test set 0.0002838375 1.413514 1.129676 8.073587 205.3605

I followed the same steps BUT break the series into subseries of size 1, 2, 3, 4, 5, 6, 7, 8, 9 and try to obtain the accuracy as I did for the block of size 2 only using the following steps:

  1. Simulate a 10 sample of AR(1) series.
  2. Split the series into subseries of size 1, 2, 3, 4, 5, 6, 7, 8, 9 without overlapping.
  3. Resample the subseries 1000 times each block size with replacement.
  4. Form a new series by joining all the resampled subseries for each block size.
  5. Check the accuracy of the newly formed series for each block size.

as follows:

################################################################################
## Load packages and prepare multicore process
library(forecast)
library(future.apply)
plan(multisession)
library(parallel)
library(foreach)
library(doParallel)
n_cores <- detectCores()
cl <- makeCluster(n_cores)
registerDoParallel(cores = detectCores())

## simulate ARIMA(1,0, 0)
set.seed(289805)
N10_ar0.8_seed289805 <- arima.sim(n=10, model=list(ar=0.8, order=c(1, 0, 0)), sd=1)
auto.arima(N10_ar0.8_seed289805, ic="aicc")

########################################################
## create a vector of block sizes
t <- length(N10_ar0.8_seed289805)    # the length of the time series
lb <- seq(t-1) #seq(t-2)+1   # vector of block sizes to be 1 < l < n (i.e to be between 1 and n exclusively)
########################################################
## This section create matrix to store block means
BOOTSTRAP <- matrix(nrow = 1, ncol = length(lb))
colnames(BOOTSTRAP) <- lb
########################################################
## This section use foreach function to do detail in the brace
#set.seed(123, kind = "L'Ecuyer-CMRG")
BOOTSTRAP <- foreach(b = 1:length(lb), .combine = 'cbind', .packages = c("forecast", "Metrics")) %dopar%{
  l <- lb[b]# block size at each instance 
  m <- ceiling(t / l)                                 # number of blocks
  blk <- split(N10_ar0.8_seed289805, rep(1:m, each=l, length.out = t))  # divides the series into blocks
  ######################################################
  set.seed(5)
  res<-sample(blk, replace=T, 1000)        # resamples the blocks
  res.unlist <- unlist(res, use.names = FALSE)   # unlist the bootstrap series
  mod <- auto.arima(res.unlist)
  fit <- fitted(mod)
  ACCURACY <- forecast::accuracy(fit, res.unlist)      # RETURNS ACCURACY
  BOOTSTRAP[b] <- ACCURACY
}
BOOTSTRAPS <- matrix(BOOTSTRAP, nrow = 1, ncol = length(lb))
colnames(BOOTSTRAPS) <- lb
BOOTSTRAPS

I got:

#                 1        2        3         4        5            6        7        8        9
# [1,] -0.0001474358 1.576266 1.298314 -1.210742 204.8705 0.0002838375 1.413514 1.129676 8.073587

I want to know

What measure of accuracy am I geting for each block: Is is MAPEorRMSEorMPEorMAE`?

Source: Windows Questions

LEAVE A COMMENT