fieldpy.core.helper

append_field2struct_arr(struct_ar, name, vec, dtype=None)

Returns a new structured array with the arrays appended. (the arrays cannot be masked)

Parameters:

struct_ar - structured array to be appended to
name (string) - the name of the new datatype
vec (np.ndarray) - the vector to be appended (its length has to be == struct_ar.shape[0])

dtype - the dtype to be used (defaults to dtype of arr)

>>> r = np.zeros((3,), dtype=[('foo', int), ('bar', float)])
>>> toap = np.array([1,2,3])
>>> append_field2struct_arr(r, 'new', toap)
array([(0, 0.0, 1), (0, 0.0, 2), (0, 0.0, 3)], 
      dtype=[('foo', '<i8'), ('bar', '<f8'), ('new', '<i8')])

append_mrec2mrec(mrec, toap_mr)

source code

Return a new masked structured array with a column appended from another masked structured array.

Parameters:

mrec (record np.ma.core.MaskedArray) - the masked record array to which will be appended
toap_mr (record np.ma.core.MaskedArray) - record np.ma.core.MaskedArray of the right size to be appended

Returns: np.ma.core.MaskedArray

The masked record array with appended fields

Note: Modified from http://mail.scipy.org/pipermail/numpy-discussion/2007-September/029357.html

>>> r = np.zeros((3,), dtype=[('foo', int), ('bar', float)])
>>> mr = r.view(np.ma.MaskedArray)
>>> mr[0] = np.ma.masked
>>> toap = np.ones((3,), dtype=[('foobar', int)])
>>> toap_mr = toap.view(np.ma.MaskedArray)
>>> toap_mr[2] = np.ma.masked
>>> append_mrec2mrec(mr, toap_mr)
masked_array(data = [(--, --, 1) (0, 0.0, 1) (0, 0.0, --)],
             mask = [(True, True, False) (False, False, False) (False, False, True)],
       fill_value = (999999, 1e+20, 999999),
            dtype = [('foo', '<i8'), ('bar', '<f8'), ('foobar', '<i8')])
<BLANKLINE>

append_ma2mrec(mrec, toap_ma, name)

source code

Return a new masked record array with a column appended from a masked array. It is assumed that whole records are masked and not individual entires in a record. The mask will the constructed with a locial or operation.

Parameters:

mrec (record np.ma.core.MaskedArray) - the masked record array to which will be appended
toap_ma (np.ma.core.MaskedArray (no-record)) - np.ma.core.MaskedArray of the right size to be appended
name (string) - the name to give to the new colum

Returns: np.ma.core.MaskedArray

The masked record array with appended fields

Note: Modified from http://mail.scipy.org/pipermail/numpy-discussion/2007-September/029357.html

>>> r = np.zeros((3,), dtype=[('foo', int), ('bar', float)])
>>> mr = r.view(np.ma.MaskedArray)
>>> mr[0] = np.ma.masked
>>> toap = np.ones((3,), dtype=int)
>>> toap_ma = toap.view(np.ma.MaskedArray)
>>> toap_ma[2] = np.ma.masked
>>> append_ma2mrec(mr, toap_ma, 'foobar')
masked_array(data = [(--, --, 1) (0, 0.0, 1) (0, 0.0, --)],
             mask = [(True, True, False) (False, False, False) (False, False, True)],
       fill_value = (999999, 1e+20, 999999),
            dtype = [('foo', '<i8'), ('bar', '<f8'), ('foobar', '<i8')])
<BLANKLINE>

append_a2mrec(mrec, toap_a, name)

source code

Return a new masked record array with a column appended from a array. It is assumed that whole records are masked and not individual entires in a record. The mask will the constructed with a locial or operation.

Parameters:

mrec (record np.ma.core.MaskedArray) - the masked record array to which will be appended
toap_a (np.array (no-record)) - np.array of the right size to be appended
name (string) - the name to give to the new colum

Returns: np.ma.core.MaskedArray

The masked record array with appended fields

Note: thhs just wraps append_field2struct_arr

>>> r = np.zeros((3,), dtype=[('foo', int), ('bar', float)])
>>> mr = r.view(np.ma.MaskedArray)
>>> mr[0] = np.ma.masked
>>> toap = np.ones((3,), dtype=int)
>>> append_a2mrec(mr, toap, 'foobar')
masked_array(data = [(--, --, 1) (0, 0.0, 1) (0, 0.0, 1)],
             mask = [(True, True, False) (False, False, False) (False, False, False)],
       fill_value = (999999, 1e+20, 999999),
            dtype = [('foo', '<i8'), ('bar', '<f8'), ('foobar', '<i8')])
<BLANKLINE>

filter_gauss(t, x, delta_t, sigma_factor=2.0)

source code

Filtes a time signal with a weighted running average in a given time intervall +/-delta_t. The weight is a gaussian such that at t+/-delta_t is at sigma_factor sigma, e.g sigma_factor=2 is c.a. 1/8 of the weight of the center. The filter is symmetric, i.e. no phase shift.

Parameters:

t - time vector
x - data vector (or array)
delta_t - filter time interval

Note:

Really slow as it itterates over all elements... maybe a cython case?
Adapted and refined from my old dataprocessing stuff

>>> t = np.linspace(0,100,10)
>>> x = np.sin(t)
>>> filter_gauss(t, x, 10*np.pi)
array([ 0.        , -0.4581598 , -0.03226275,  0.1324937 ,  0.06281049,
       -0.11801212, -0.09001935,  0.09725727,  0.36815581, -0.50636564])
>>> tm = t.view(np.ma.MaskedArray)
>>> xm = x.view(np.ma.MaskedArray)
>>> xm[3] = np.ma.masked
>>> filter_gauss(tm, xm, 10*np.pi)
masked_array(data = [0.0 -0.458159800543 -0.0375228441936 -0.219964900938 0.0730510540235
 -0.386390056324 -0.090019349563 0.0972572677569 0.368155810038
 -0.50636564111],
             mask = [False False False False False False False False False False],
       fill_value = 1e+20)
<BLANKLINE>

filter_gauss_conv(t, x, delta_t, sigma_factor=2.0)

source code

Filters a regularly-sampled signal with a Gaussian-weighted average with half-window length delta_t. The Gaussian bell is chosen such that at +/-delta_t it is valued at sigma_factor, e.g. sigma_factor=2 is c.a. 1/8 of the weight of the center. There is no phase shift.

Warning: If x is masked, then NaN-values will be laundered into masked values in the output. This means that if x contains both a mask and NaNs, the output will only have a mask, extended over the NaNs.

Parameters:

t - time vector
x - data vector (or array)
delta_t - filter time interval

Note:

Performs filtering using a np.convolve.

>>> t = np.linspace(0,100,10)
>>> x = np.sin(t)
>>> filter_gauss_conv(t, x, 10*np.pi)
array([-0.35491283, -0.25778845, -0.06943494,  0.07441436,  0.03527717,
       -0.06628086, -0.05055887,  0.10013494,  0.20257496,  0.20875278])
>>> tm = t.view(np.ma.MaskedArray)
>>> xm = x.view(np.ma.MaskedArray)
>>> xm[3] = np.ma.masked
>>> filter_gauss_conv(tm, xm, 10*np.pi)
masked_array(data = [-- -- -- 0.0380257449674 0.035277166102 -0.066280860855 -0.0505588746925
 0.100134936194 0.202574957573 0.208752775728],
             mask = [ True  True  True False False False False False False False],
       fill_value = 1e+20)
<BLANKLINE>

filter_and_resample_slow(time, data, filter_window, new_sample_int, first_sample=None)

source code

Filters and resamples a time signal with a running average in a given time intervall +/-filter_window/2. The filter has a PHASE SHIFT for a UN-equally sampled signal. Accepts masked arrays but not "structured arrays" or "recarrays".

If you have cython (or just build the modul) use the much faster filter_and_resample in helper_fns_cython.

Having said this, this python implementation is probably really slow!

Parameters:

time - the times of the timeseries
data - the data of the timeseries
filter_window - filter time interval
new_sample_int - the new sampling interval (in units of time)
first_sample - the time of the first sample in the resampled signal. If None (default) it is (approximatly) set to time[0]+filter_window

Returns: list of np.array

Returns [new_time, new_data]

Note: 1) ignores any masks in the time variable. 2) will not work if it finds no values within filter window

To Do: Correct above point 2) (as well in helper_fns_cython.pyx) >>> tim = np.linspace(-3,90,1000); data = tim**2 >>> filter_window = 30; new_sample_int = 30 >>> filter_and_resample_slow(tim, data, filter_window, new_sample_int) (array([ 0., 30., 60., 90.]), array([ 62.98183318, 975.90237234, 3678.22848073, 6826.19355542])) >>> data = np.ma.MaskedArray(data) >>> data[1:10] = np.ma.masked >>> filter_and_resample_slow(tim, data, filter_window, new_sample_int) (array([ 0., 30., 60., 90.]), array([ 65.73049119, 975.90237234, 3678.22848073, 6826.19355542]))

get_ind_closest(vec, value, guess_ind=None)

source code

Finds the index (ind) of vec such that vec[ind]-value is smallest. Only works reliably for monotone vectors. As start value it uses the value found in ind[0].

This function is somewhat slower than get_ind_closest but can be called directly from Python. (When calling from cython use get_ind_closest_faster)

Parameters:

vec - the vector, usually a time
value - the value which is compared to vec
guess_ind - a guess for index around which the search (symmetric) will be started.

Returns:

The number of indices away from

Note: The algorithm is fairly dum, something like bisection might work much faster.

>>> tim =np.linspace(-3,100,1000)
>>> get_ind_closest(tim, 49.2)
506
>>> get_ind_closest(tim, 49.2, 400)
506

spline_interpolation(vecs, t_par, smoothness=None, spline_order=3, n_knots=None, n_knots_estimate=-1)

source code

Finds a B-spline representation of an N-dimensional curve. Uses scipy.interpolate.fitpack.splprep.

Given a list of N rank-1 arrays, vecs, which represent a curve in N-dimensional space parametrized by t_par, find a smooth approximating spline curve g(t_par). Uses the FORTRAN routine parcur from FITPACK.

Example: t = np.hstack((np.linspace(0,4*np.pi, 2000), np.linspace(4*np.pi+0.1, 8*np.pi, 500))) xx = np.sin(t) x = xx + np.random.normal(scale=0.1, size=t.shape) fn,tckp = spline_interpolation([x],t, smoothness=40.0) plot(t,x),hold(True), plot(t, fn(t)[0],'r'), plot(t,xx, 'g')

Parameters:

vecs - A list of sample vector arrays representing the curve.
t_par - An array of parameter values (probably time).
smoothness - smoothness parameter (default determinded by splprep)
spline_order - spline order (default & reccomended 3)

Note: adapted from http://www.scipy.org/Cookbook/Interpolation#head-34818696f8d7066bb3188495567dd776a451cf11

polyval(p, x)

source code


Copied from numpy and adjusted to work for masked arrays.

Evaluate a polynomial at specific values.

If `p` is of length N, this function returns the value:

``p[0]*x**(N-1) + p[1]*x**(N-2) + ... + p[N-2]*x + p[N-1]``

If `x` is a sequence, then `p(x)` is returned for each element of `x`.
If `x` is another polynomial then the composite polynomial `p(x(t))`
is returned.

Parameters
----------
p : array_like or poly1d object
   1D array of polynomial coefficients (including coefficients equal
   to zero) from highest degree to the constant term, or an
   instance of poly1d.
x : array_like or poly1d object
   A number, a 1D array of numbers, or an instance of poly1d, "at"
   which to evaluate `p`.

Returns
-------
values : ndarray or poly1d
   If `x` is a poly1d instance, the result is the composition of the two
   polynomials, i.e., `x` is "substituted" in `p` and the simplified
   result is returned. In addition, the type of `x` - array_like or
   poly1d - governs the type of the output: `x` array_like => `values`
   array_like, `x` a poly1d object => `values` is also.

See Also
--------
poly1d: A polynomial class.

Notes
-----
Horner's scheme [1]_ is used to evaluate the polynomial. Even so,
for polynomials of high degree the values may be inaccurate due to
rounding errors. Use carefully.

References
----------
.. [1] I. N. Bronshtein, K. A. Semendyayev, and K. A. Hirsch (Eng.
   trans. Ed.), *Handbook of Mathematics*, New York, Van Nostrand
   Reinhold Co., 1985, pg. 720.

Examples
--------
>>> np.polyval([3,0,1], 5)  # 3 * 5**2 + 0 * 5**1 + 1
76
>>> np.polyval([3,0,1], np.poly1d(5))
poly1d([ 76.])
>>> np.polyval(np.poly1d([3,0,1]), 5)
76
>>> np.polyval(np.poly1d([3,0,1]), np.poly1d(5))
poly1d([ 76.])

_check_output(*popenargs, **kwargs)

source code

Copied from python 2.7 standard libary as python 2.6 doesn't have it. Remove after Flavien updated to 2.7.

Run command with arguments and return its output as a byte string.

If the exit code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and output in the output attribute.

The arguments are the same as for the Popen constructor. Example:

>>> _check_output(["ls", "-l", "/dev/null"]) # doctest:+ELLIPSIS
'crw-rw-rw- 1 root root 1, ...

The stdout argument is not allowed as it is used internally. To capture standard error in the result, use stderr=STDOUT.

>>> _check_output(["/bin/sh", "-c",
...               "ls -l non_existent_file ; exit 0"],
...              stderr=subprocess.STDOUT)
'ls: cannot access non_existent_file: No such file or directory\n'

Module helper_fns

pickle_(var, filename)

append_field2struct_arr(struct_ar, name, vec, dtype=None)

append_mrec2mrec(mrec, toap_mr)

append_ma2mrec(mrec, toap_ma, name)

append_a2mrec(mrec, toap_a, name)

mrec_keep_fields(mrec, names)

filter_gauss(t, x, delta_t, sigma_factor=2.0)

filter_gauss_conv(t, x, delta_t, sigma_factor=2.0)

filter_and_resample_slow(time, data, filter_window, new_sample_int, first_sample=None)

get_ind_closest(vec, value, guess_ind=None)

spline_interpolation(vecs, t_par, smoothness=None, spline_order=3, n_knots=None, n_knots_estimate=-1)

polyval(p, x)

get_current_fieldpy_git_hash()

get_current_git_hash(dirname)

_check_output(*popenargs, **kwargs)