matcal.core.data_importer

The classes and functions in this module are intended to import data into MatCal from external sources for use in MatCal studies.

Functions

FileData(filename[, state, file_type, ...])

A function used to import a MatCal Data object from a file.

Classes

BatchDataImporter(filenames, **filedata_kwargs)	Class to import multiple data files using a regular expression or a list of filenames.
CSVDataImporter(filename[, import_strings, ...])	Class for reading in data from a CSV file.
CSVProbeImporterCreator()
JSONProbeDataImporter(filename[, ...])
JSONProbeImporterCreator()
MatlabDataImporter(filename[, ...])	Class for reading in data from a Matlab ".mat" file.
MatlabProbeImporterCreator()
NumpyDataImporter(filename[, ...])	Class for reading in data from a numpy ".npy" file.
NumpyProbeImporterCreator()
ProbeDataImporterFactory()

matcal.core.data_importer.FileData(filename: str, state: State = None, file_type: str = None, import_strings: bool = False, drop_NaNs: bool = False, *args, **kwargs) → Data

A function used to import a MatCal Data object from a file. The user needs to use this function to load experimental data from a file into MatCal

Parameters:

• filename (str) – the name of the file to be loaded.

• state (State) – optional state to be assigned to the data being imported

• file_type (str) – optional file type passed by the user. MatCal will attempt to guess the file type based on the file extension. MatCal recognizes “csv”, “npy” and “mat” file types and only accepts these strings as input for this parameter.

• import_strings (bool) – A boolean to allow MatCal to read in string data fields. By default it is set to False and will error out if any data cannot be converted to numeric values.

• drop_NaNs – A boolean to allow MatCal to read in string data fields with NaNs by dropping any rows that contain a NaN.

Returns:

a populated Data object.

class matcal.core.data_importer.DataImporterBase(filename: str, import_strings: bool = False, drop_NaNs: bool = False, **kwargs)

class matcal.core.data_importer.CSVDataImporter(filename: str, import_strings: bool = False, drop_NaNs: bool = False, **kwargs)

Class for reading in data from a CSV file. This uses the NumPy “genfromtxt” function to read data in from CSV files. It assumes that the columns have headers so that MatCal can identify what information is being read in and make appropriate comparisons between simulations and experiments. This is wrapped by FileData().

Note

This accepts the following keyword arguments that are valid in Numpy “genfromtxt”:

1. comments

2. uscols

3. skip_footer

4. converters

5. missing_values

6. filling_values

load()

Loads the CSV data.

Returns:

A data set object built from the CSV file.

Return type:

Data

class matcal.core.data_importer.NumpyDataImporter(filename: str, import_strings: bool = False, drop_NaNs: bool = False, **kwargs)

Class for reading in data from a numpy “.npy” file. This uses the np.load() function to read data in from numpy files. Since field names are required for MatCal, it assumes that a structured array or record is saved in the file. If there are no field names, it will fail to load the file. This is wrapped by FileData().

load()

Loads the Numpy “npy” data.

Raises:

FileNotFoundError – If the data file is not found.

Returns:

A data set object built from the NPY file.

Return type:

Data

class matcal.core.data_importer.MatlabDataImporter(filename: str, import_strings: bool = False, drop_NaNs: bool = False, **kwargs)

Class for reading in data from a Matlab “.mat” file. This uses the scipy.io.loadmat() function to read data in from the files. Since field names are required for MatCal, it assumes that the data are stored in a format of 1d vectors with each variable name being the field name. This is wrapped by FileData().

class matcal.core.data_importer.BatchDataImporter(filenames, **filedata_kwargs)

Class to import multiple data files using a regular expression or a list of filenames.

Keyword arguments (**filedata_kwargs) are forwarded to FileData() for each file.

States parsed from file headers are reconciled across the batch by rounding numeric state parameters to a configurable significant-figure precision (default: 6). States whose parameters match at the specified precision are combined into a common state.

Parameters:

• filenames (list(str) or str) – The names/paths of the files containing the data. This can be a list of strs or a single string that is a glob pattern.

• filedata_kwargs – Keyword arguments forwarded to FileData() for each file (e.g., comments, delimiter, usecols, import_strings, drop_NaNs, file_type).

set_state_precision(precision: int = 6)

Set the number of significant figures used when reconciling unique states across the batch.

Numeric state parameters are rounded to this many significant figures when determining whether two states are the same; states that match for all parameters at this precision are combined.

Parameters:

precision (int) – non-negative integer significant-figure precision

set_fixed_state_parameters(name: str = None, **fixed_states)

Set fixed/additional state variables applied to every imported dataset during reconciliation.

Values must be either strings or numeric real values.

Optional naming: - If no state information is found in any imported file (i.e., all data sets have

SolitaryState), then providing name will cause the fixed-state name to be used as the state name for the batch-created state.

• If state information is found in at least one file, then the provided name is ignored and a warning is logged (the per-file state naming is preserved).

Parameters:

• name (str or None) – Optional name for the fixed-only state when no file states exist.

• fixed_states (dict) – keyword/value pairs to add to the state (e.g., P1=5.0, P2="abc")

property batch

Imports and collects the data into a DataCollection.

If State data is included in the files, the appropriate states are assigned to each Data object along with any fixed state parameters specified using set_fixed_state_parameters().

Data with similar states are combined into single states according to the configured state precision (see set_state_precision()).

class matcal.core.data_importer.JSONProbeDataImporter(filename: str, import_strings: bool = False, drop_NaNs: bool = False, **kwargs)

class matcal.core.data_importer.ProbeDataImporterFactory

class matcal.core.data_importer.CSVProbeImporterCreator

class matcal.core.data_importer.NumpyProbeImporterCreator

class matcal.core.data_importer.MatlabProbeImporterCreator

class matcal.core.data_importer.JSONProbeImporterCreator