pyhctsa.operations.pre_process.preproc_compare

pyhctsa.operations.pre_process.preproc_compare(y, detrend_meth='medianf3')

How time-series properties change after a preprocessing step.

Applies a given preprocessing transformation (detrending, differencing, filtering or resampling) to the time series, z-scores the original and the processed series, and returns the change in each of a set of statistics from its value for the original to its value for the processed series. The statistics compare stationarity measures (StatAv, and the variation of the local mean and of the local standard deviation across windows), distributional fits (a Gaussian fit to the kernel-smoothed distribution, and the discrepancy from a fitted normal distribution) and the effect of trimming outliers.

The change is the difference (processed minus original) for statistics that can be negative or zero, and the normalized difference (processed - original) / (processed + original) for positive statistics. The latter is between -1 and 1, is 0 when nothing changes, and equals tanh(log(processed / original) / 2), a bounded version of the log-ratio that stays finite when the original value is near 0, where a plain ratio is unstable.

Parameters:
y : array-like

Input time series.

detrend_meth : str, optional

The preprocessing to apply:

  • "poly<n>" : remove a polynomial of order n = 1-9, e.g., "poly1" is a linear detrending

  • "sin<n>" : remove a constant plus a sum of n = 1-8 sinusoids a1*sin(2*pi*f1*t + c1) + …, fitted jointly by least squares, with frequencies searched deterministically between 1/N and 1/2 - 1/(2N) cycles per sample (pyhctsa.robust.bf_fit_sinusoids()), e.g., "sin1": this removes oscillations, and a component with less than one full cycle in the series is a trend, not an oscillation (trends are removed by "poly" and "spline")

  • "spline<npieces><order>" : remove a least-squares spline with the given number of polynomial pieces and spline order, e.g., "spline24" is a cubic spline with 2 pieces

  • "diff<n>" : n successive differences, e.g., "diff1"

  • "medianf<n>": running median filter of length n (end values repeated beyond the ends), e.g., "medianf3"

  • "rav<n>" : running mean filter of length n, e.g., "rav5"

  • "resample_<p>_<q>" : resample by the ratio p/q; e.g., "resample_1_2" halves the length and "resample_10_1" multiplies it by 10

  • "logr" : log returns (positive data only; otherwise NaN)

  • "boxcox" : a Box-Cox transformation (positive data only; otherwise NaN)

Default is "medianf3".

Returns:

The change, from the original to the processed series, of each of these statistics (all of the series are z-scored first).

Normalized differences (positive statistics):

  • statav2: StatAv with 2 segments (stat_av)

  • swms2_2, swms5_1, swms10_1: the standard deviation of the window means across windows (sliding_window ‘mean’), with 2 windows overlapping by half, and 5 and 10 non-overlapping windows

  • swss2_1, swss5_1, swss10_1: the same for the window standard deviations (sliding_window ‘std’)

  • kscn_olapint: the overlap integral of the kernel-smoothed distribution with the best-fitting normal (compare_ks_fit)

  • olbt_s5: the standard deviation after trimming the 5% most extreme values at each end, relative to that of the full series (outlier_test)

Differences (statistics that can be negative or zero):

  • gauss1_kd_r2, gauss1_kd_resAC1, gauss1_kd_resrunsz: the R^2, the lag-1 autocorrelation of the residuals, and the runs-test z-statistic of the residuals of a Gaussian fit to the kernel-smoothed distribution

  • kscn_peaksepy, kscn_peaksepx, kscn_relent: the peak separation in height and in position, and the relative entropy, of the kernel-smoothed distribution against the best-fitting normal (compare_ks_fit)

  • olbt_m2, olbt_m5: the mean after trimming the 2% and 5% most extreme values at each end (outlier_test)

A scalar NaN is returned if the processed series has no variation beyond floating-point round-off, as statistics of round-off are not meaningful: this is the case for the residual of an exact fit (e.g., a polynomial or a sum of sinusoids that the series is), or the differences of an exact linear trend. The criterion is a standard deviation of at most 10*N*eps times the root-mean-square of the series the processing acts on (its length N; the log of the series for 'logr', and the transformed series itself for 'boxcox'), a bound on the rounding error of the processing (sums of up to N terms, and the phase of sinusoids evaluated at times up to N). Small but genuine variation, such as the quantization noise left by an exact fit to 24-bit data (about 1e-7 of the amplitude), is computed. NaN is also returned for 'logr' and 'boxcox' if the data are not all positive.

Return type:

dict