pyhctsa.operations.stationarity.local_distributions¶

pyhctsa.operations.stationarity.local_distributions(y, num_segs=5, each_or_par='par', num_bins=10)¶

Compares the distribution in consecutive time-series segments, in excess of chance.

Breaks the time series into num_segs consecutive segments of equal length and measures how different the distributions of values in two segments are as the total variation distance between their histograms: half the sum of the absolute differences between the proportions of values in each bin (0 for identical distributions, 1 for distributions with no bin in common). The bins are common to all segments and equiprobable for the full series (cut at its Hazen quantiles, tied values sharing a bin; pyhctsa.utils._coarse_grain()), so the measure has no smoothing parameter and does not depend on the scale of the data. Either each segment is compared to the parent (full time series) distribution, or each to all other segments.

A histogram of n values differs from the distribution it samples by about 1/sqrt(n) by chance, so the raw mean distance falls with the length of the series. The mean distance is therefore reported in excess of its exact expectation for the same values in random order. In random order, the number of a segment’s n values that fall in bin i is hypergeometric: n draws without replacement from the N values of the series, K_i of which are in the bin. Comparing to the parent, the expected distance is half the sum, over bins and over the possible counts x, of the probability of x times |x/n - K_i/N|. Between two disjoint segments, the count y in the second, given x in the first, is hypergeometric with n draws from the remaining N - n values, K_i - x of them in the bin, and the expected distance is half the sum over bins of the expectation of |x - y|/n. These are exact for any bin counts (including fewer, unequal bins from tied values), and the samples left over after the last full segment count only towards the parent. Every segment, or pair of segments, has the same expectation, so it is also the expected mean distance. The sums leave out the tails of each hypergeometric distribution, with a total mass below 1e-14 (bounded by Bernstein’s inequality), which changes the expectation by at most 1e-14 per bin.

The excess is near 0 for a stationary series without autocorrelation at any length, can be slightly negative, and approaches the distance between the segments’ underlying distributions as the series grows. Autocorrelation reduces the effective number of independent values in a segment, so a correlated stationary series has a positive excess that depends on its timescale.

A second output, ex_meandiv_ess, subtracts an expectation that also allows for autocorrelation, through an effective sample size. Autocorrelation multiplies the random-order variance of a bin’s count in a segment of n values by g = 1 + 2 sum_{k<=M} (1 - k/n) rho(k), where rho is the autocorrelation function of the indicator of a value being in the bin, so each bin’s term in the expected distance is multiplied by sqrt(g) (unchanged where g = 1). rho is estimated within the segments: each segment’s indicator is taken about its own mean and the autocovariances are pooled over segments, so differences in distribution between segments, the change being measured, do not enter g. Each lag of the estimate is shifted by its exact expectation for values in random order, -(n - k) / (n (n - 1)), so that g is about 1 in random order. The sum runs to the first lag M with M >= 5 g(M), a window that grows with the correlation time it estimates. This removes the excess from correlations shorter than a segment; correlations longer than a segment, such as those of a very slow process, cannot be seen within it and are only partly removed.

Parameters:¶

y : array-like¶

The input time series.

num_segs : int, optional¶

The number of segments to break the time series into. Default is 5.

each_or_par : {'par', 'each'}, optional¶

‘par’: compares each local distribution to the parent (full time series) distribution.
’each’: compares each local distribution to all other local distributions.

Default is 'par'.

num_bins : int, optional¶

The number of equiprobable bins (fewer if values are tied at a quantile). Default is 10.

Returns:¶

ex_meandiv: the mean total variation distance across the comparisons, minus its expectation for the values in random order.
ex_meandiv_ess: the mean distance minus its expectation allowing for autocorrelation within the segments (effective sample size).
stddiv: the standard deviation of the distances across the comparisons.

For ‘each’ with two segments, the single distance minus its expectation is returned as a float. NaN if the segments are empty (fewer values than segments).

Return type:¶

dict