pyhctsa.operations.model_fit.compare_ar

pyhctsa.operations.model_fit.compare_ar(y, orders=array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), test_how='all')

How the out-of-sample error of an AR model changes with its order.

Fits autoregressive (AR) models of a range of orders and compares the loss of each (the mean squared one-step prediction error on the test segment) when the model fitted to a training segment is applied to a test segment. Statistics are taken over the loss as a function of model order, v.

The first max(orders) points of the training and test segments are excluded from the fit and from the mean, so that every order is fitted to and scored on the same points.

With test_how = 'all' the models are tested on the data they were trained on, so the loss measures in-sample fit: it cannot rise with the model order, and features such as minv, propgain1min and where01max mostly describe how fast the fit improves with order. Use a training fraction (e.g. 0.5) for a genuine out-of-sample comparison.

Parameters:
y : array-like

The input time series.

orders : array-like, optional

The model orders to compare. Default is 1 to 10.

test_how : float or str, optional

A fraction of the time series to train on (the model is tested on the remaining portion), or 'all' to train and test on all the data. Default is 'all'.

Returns:

  • maxv, minv, meanv, medianv: the maximum, minimum, mean and median of the loss over orders,

  • propgain1min: the proportion of the first order’s loss removed by the best order, 1 - min(v) / v[0] (between 0 and 1; 1 for a perfectly predictable series; NaN if the first loss is zero or not finite),

  • medonmax: the median loss divided by the maximum loss (in (0, 1], approaching 1 when the loss is insensitive to the order; NaN if the maximum is zero or not finite),

  • meandiff, stddiff, maxdiff, meddiff: the mean, standard deviation, maximum absolute value and median of the change in loss from one order to the next,

  • minstdfromi: the minimum (over starting orders i) of the standard error of the loss over orders i onward, std(v[i:]) / sqrt(len(v) - i), ignoring zeros,

  • where01max: the first position in the list of orders (from 1) from which that standard error is below 10% of its maximum (NaN if none),

  • whereen4: the first position from which it is below 1e-4 (NaN if none),

  • best_n: the order with the smallest loss,

  • aic_n: the order that minimizes Akaike’s Information Criterion, log(loss * (1 + 2 * order / n)) with n the number of test points scored,

  • bestaic: the minimum value of that criterion over orders.

best_n, aic_n and bestaic use the losses floored at 1e-10 times the mean square of the test segment, so that orders whose loss is at the level of rounding error (an exactly predictable series) tie and the lowest of them is chosen.

NaN if the series is too short for the largest order: the training segment must have more than 2 * max(orders) points and the test segment more than max(orders). Otherwise the highest-order models interpolate the training data and the loss is at machine precision, an artifact.

Return type:

dict