matcal.core.study_base

The study base module contains the base class for all MatCal studies. It is not user facing but must be used as the base class for any new studies.

Classes

`ObjectiveInformation`([record_objectives, ...])	Contains the objective information for a study.
`QoiInformation`([record_data, record_qois, ...])	Contains the QoI information for a study.
`StudyBase`(*parameters)	Base class for all MatCal Studies, not intended for users.
`StudyResults`([record_qois, ...])	A class used to store the results of a study and facilitate user processing of results.

class matcal.core.study_base.StudyBase(*parameters)[source]

Base class for all MatCal Studies, not intended for users.

Parameters:: parameters (list(Parameter) or ParameterCollection) – The parameters of interest for the study.
Raises:: StudyTypeError – if parameters is of incorrect type.

property results: Return access to the study’s results. Will return None, if study has not been run.

add_evaluation_set(model, objectives, data=None, states=None, data_conditioner_class=<class 'matcal.core.data.MaxAbsDataConditioner'>)[source]

Adds an evaluation set to the study. An evaluation set is a set of datasets, objectives and states that are applicable to a model. For each evaluation set, the model will be evaluated for every state in the set. The results from each model state will be compared to each dataset its state. This comparison consists of each objective in the passed objectives.

Parameters:

model (valid model type from the models module) – The model that will generate results for comparison to the data in the set.
objectives (Objective or ObjectiveCollection) – The objectives to quantitatively compare the model results to the data.
data (Data or DataCollection) – The data to be evaluated with this evaluation set. Data is not required when this method is called with a SimulationResultsSynchronizer.
states (State or StateCollection) – A subset of states in the data that are of interest for this study.
data_conditioner_class – the class that will be used as a data conditioner for this evaluation set. See data for valid data conditioners.

Raises:

StudyTypeError – if passed arguments are of the incorrect type.
StudyError – if all the passed states are not in the data.

set_results_storage_options(data: bool = True, qois: bool = True, residuals: bool = True, objectives: bool = True, weighted_conditioned: bool = False, results_save_frequency: int = 1)[source]

Set which history information to save and return with the study results. You can also down sample which evaluations to save using results_save_frequency. This is particularly useful if you wish to not store finite difference evaluations for gradient based studies. The total objective is always stored.

Parameters:

data (bool) – Store the raw data for each simulation and the raw experimental data for each objective for each desired evaluation.
qois (bool) – Store the QoIs for each objective for each desired evaluation. This includes both experiment and simulation QoIs
residuals (bool) – Store the residuals for each objective for each desired evaluation.
objectives (bool) – Store the objective by state and evaluation set for each desired evaluation.
weighted_conditioned (bool) – Store the weighted and conditioned values for each desired evaluation. This will save the weighted and conditioned, residuals, simulation qois and experiment qois.
results_save_frequency (int) – Set how the results save interval. For studies where finite difference derivatives are used, an interval of $n+1$ will exclude finite difference results from the saved results history.

plot_progress()[source]: Calling this method will cause matcal to generate automatic plots after each batch of parameter evaluations. These plots are made using the standard plotter and will show things such as objective value evolution.

property final_results_filename

Returns the filename for the final results file for the current study.

return: final results filename as an absolute path rtype: str

launch()[source]

This launches the study. Note that at least one evaluation set must be added with add_evaluation_set().

Returns:: study specific results.
Raises:: StudyError – if no evaluation sets have been added.

set_core_limit(core_limit, override_max_limit=False)[source]

Sets the total number of cores that the study may use.

Parameters:

core_limit (int) – The max number of cores that the study can use at any time.
override_max_limit – Override the default max cores that can be specified for a given study. The current limit of 500 is recommended by the MatCal team but might not be best for all cases.

Raises:

StudyTypeError – if the passed value is not an int.

set_cleanup_mode(new_pruner: DirectoryPrunerBase)[source]: Changes the pruner to the object passed as an argument

add_parameter_preprocessor(parameter_preprocessor)[source]

Add a parameter preprocessor to the study that will operate on the parameters before they are sent to the models. See UserDefinedParameterPreprocessor.

Parameters:: parameter_preprocessor (UserDefinedParameterPreprocessor) – the parameter preprocessor that will modify and update the given model parameters

set_parameters(*parameters)[source]

Parameters:: parameters (Parameter or ParameterCollection) – The parameters of interest for the study.
Raises:: StudyTypeError – if the parameters are of incorrect type.

set_use_threads(always_use_threads=False)[source]

By default, MatCal assumes that the model being run is CPU intensive. As a result, it runs each model in a subprocess which can result in some additional overhead. If running studies cheaper python models, it may be beneficial to use threading instead of a subprocess. Using this method will run the study with threading if only one model can be evaluated at a time. You can optionally run with threads even with concurrent model evaluations with the “always_use_threads” option; however, this can be less reliable. For large memory calibrations, we always recommend using subprocess.

Finally, any external executable is always run using subprocess, but threading can be use to manage that job and return its results.

Parameters:: always_use_threads (bool) – if true, MatCal will use threads over subprocess for concurrent modeling jobs. Defaults to False.

run_in_serial()[source]

Tell MatCal to run evaluations in serial. This is only recommended if the study is serial, like a MCMC Bayes Study, and the model evaluations are fast, like a python model.

Running in serial avoids the overhead of reloading large data sets that are necessary in async studies.

set_working_directory(working_directory, remove_existing=False)[source]

By default, MatCal runs in the current working directory. This method allows the user to specify a subdirectory in the current directory for the study to be run in. This method will create only the last directory in the path. So if the desired subdirectory is under a multiple folders from the current directory MatCal will error if the head of the path does not exist. See os.path.split() for a definition of the path “head”.

Parameters:

working_directory (str) – The desired working directory for the current study. MatCal will only create the last folder if the path is a nested path.
remove_existing – If True, then the directory will be removed if pre-existing at study launch.

class matcal.core.study_base.StudyResults(record_qois: bool = True, record_residuals: bool = True, record_objectives: bool = True, record_data: bool = True, record_weighted_conditioned: bool = False, results_save_frequency: int = 1, **kwargs)[source]

A class used to store the results of a study and facilitate user processing of results.

To get study specific results in a dictionary access them with outcome(). Or access them as attributes on the class it self.

For example, if you store a calibration result as a variable with name “cal_results”, you can access the outcome as follows:

best_params = cal_results.outcome["best"]
best_y = cal_results.outcome["best:y"]
best_params = cal_results.best.to_dict()
best_y = cal_results.best.y

The dictionary “best_params” and variable “best_y” is the same using both methods to access study results.

Users should not need to initialize this class.

Parameters:

record_qois – Indicates if QoI history should be recorded (default is True)
record_residuals – Indicates if residual history should be recorded.
record_objectives – Indicates if the objective history for individual objectives should be recorded.
record_data – Indicates if the raw simulation results history and raw experimental data should be recorded.
record_weighted_conditioned – Indicates if the weighted and conditioned versions of the QoIs are stored.
results_save_frequency (int) – The frequency at which results should be saved.

property outcome: dict

Stores the relevant outcomes of the study performed

Depending upon the type of study, the returned dictionary will have differently named keys for each parameter. For example, a calibration will return a dictionary with keys named ‘best:<parameter_name>’ for each parameter, while a sensitivity study will have keys titled ‘sobol:<parameter_name>’.

All of the study specific results can also be accessed as attributes or nested attributes of the StudyResults object.

Returns:: The outcomes of the study
Return type:: dict
Raises:: RuntimeError – If no outcome is defined

property parameter_history

Stores the history of parameters evaluated during the study.

Returns:: The history of parameters evaluated during the study where keys are the parameter names and the values are a list which include all parameter values in the order they were evaluated.
Return type:: OrderedDict (str, list(float))

property simulation_history

Stores the history of simulations performed during the study

Returns:: The history of simulations performed during the study where keys are the model names and values a DataCollection of simulation results for each model. The repeats in the DataCollection for a given model and state are the simulation results for that model and state in the order of evaluation for the study.
Return type:: OrderedDict(str, DataCollection)

property number_of_evaluations

The number of evaluations performed in the study

Returns:: The number of evaluations performed in the study
Return type:: int

property success: bool

Indicated whether the study completed successfully or not.

Return type:: bool

property exit_message

Stores any termination information from the study Stores any termination message information from the study

Returns:: The termination information from the study
Returns:: The termination message information from the study
Return type:: str

property exit_status

Returns error codes or exit flag status for the algorithm. Value varies by study and options used with the study.

Returns:: exit code from the algorithm
Return type:: int

property evaluation_sets

Stores the names of evaluation set results stored in the results. These names are needed to access results for certain result types such as QoIs from models and experiments.

Returns:: The evaluation set names stored in the results
Return type:: list(str)

property objective_history

Stores the history of objectives evaluated during the study. The keys are the evaluation set names stored in evaluation_sets(). Each value contains a populated ObjectiveInformation object corresponding to the evaluation set name key.

Returns:: The history of objectives evaluated during the study
Return type:: OrderedDict(str, ObjectiveInformation)

property qoi_history

Stores the history of QoI for the objectives evaluated during the study. The keys are the evaluation set names stored in evaluation_sets(). Each value contains a populated QoiInformation object corresponding to the evaluation set name key.

Returns:: The history of QoIs for each evaluation during the study
Return type:: OrderedDict(str, QoiInformation)

property total_objective_history

The history of the overall total objective evaluated during the study in order of evaluation.

Returns:: The overall or total objective history evaluated during the study
Return type:: list(float)

property best_evaluation_index

The index of the best evaluation index based on the total objective history. This is the index of the best stored evaluation in order of the study’s evaluation history.

Returns:: best evaluation index of stored results
Return type:: int

property evaluation_ids: The evaluation id number for each stored result. The id is the evaluation number in order of the study’s evaluation history.

property best_evaluation_id

The id of the best evaluation based on the total objective history. The id is the evaluation number in order of the study’s evaluation history.

Returns:: best evaluation id number
Return type:: int

property best_total_objective

The best total objective value obtained during the study

Returns:: The best total objective value
Return type:: float

best_evaluation_set_objective(model, obj)[source]

Returns the best evaluation set objective value and its evaluation index/number. An evaluation set objective is the total objective value for a given model and objective. It includes a summation of the objective for all states and fields for a give evaluation set.

Parameters:

model (str, ModelBase) – The model or model name of interest
obj (str, Objective) – The objective or objective name of interest

Returns:

The best evaluation set objective value and its index

Return type:

tuple(float,int)

get_evaluation_set_objectives(model, obj)[source]

Returns the history of the evaluation set objectives for a given model and objective. An evaluation set objective is the total objective value for a given model and objective. It includes a summation of the objective for all states and fields for a give evaluation set. They are returned in order of evaluation.

Parameters:

model (str, ModelBase) – The model or model name of interest
obj (str, Objective) – The objective or objective name of interest

Returns:

The summed state objectives

Return type:

np.ndarray

get_objectives_for_model(model)[source]

Returns the list of objective names that are in the results for a given model. This can be used to easily loop over all objectives for the model if processing results for only a given model.

Parameters:: model (str, ModelBase) – The model or model name of interest
Returns:: The list of objectives names for the model
Return type:: list(str)

property models_in_results

Returns the list of unique model names that are in the results.

Returns:: The list of model names included in the results.
Return type:: set(str)

best_simulation_data(model, state)[source]

Returns the best simulation data for a given model and state. It returns a Data object with the the best simulation’s results.

Parameters:

model (str, ModelBase) – The model or model name of interest
state (str, State) – The objective or objective name of interest

Returns:

The best simulation data

Return type:

matcal.core.data.Data

get_experiment_qois(model, obj, state, index=None)[source]

Returns the experiment qois for a given model and state.

Parameters:

model (str, ModelBase) – The model or model name of interest
state (str, State) – The objective or objective name of interest
index (int) – The index of the desired repeat if given

Returns:

The experiment QoIs

Return type:

Data or list(Data)

get_experiment_data(model, obj, state, index=None)[source]

Returns the experiment data for a given model and state.

Parameters:

model (str, ModelBase) – The model or model name of interest
state (str, State) – The objective or objective name of interest
index (int) – The index of the desired repeat if given

Returns:

The experiment data

Return type:

Data or list(Data)

best_simulation_qois(model, obj, state, index=None)[source]

Returns the best simulation QoIs for a given model and state.

Parameters:

model (str, ModelBase) – The model or model name of interest
state (str, State) – The objective or objective name of interest
index (int) – The index of the desired repeat if given

Returns:

The best simulation qois

Return type:

Data or list(Data)

best_residuals(model, obj, state, index=None)[source]

Returns the best objective residual data for a given model, objective and state.

Parameters:

model (str, ModelBase) – The model or model name of interest
state (str, State) – The objective or objective name of interest
index (int) – The index of the desired repeat if given

Returns:

The best objective residuals

Return type:

Data or list(Data)

best_weighted_conditioned_residuals(model, obj, state, index=None)[source]

Returns the best objective weighted and conditioned residual data for a given model, objective and state.

Parameters:

model (str, ModelBase) – The model or model name of interest
state (str, State) – The objective or objective name of interest
index (int) – The index of the desired repeat if given

Returns:

The best objective weighted and conditioned residuals

Return type:

Data or list(Data)

save(filename)[source]

Saves the study results to a file using matcal_save(). You should use a “joblib” filename.

Parameters:: filename (str) – The name of the file to save the results to

class ResultsAttribute[source]

Study specific results are stored as attributes are a special class that have methods aiding in results organization.

to_dict()[source]

Converts a result attribute into a dictionary with the result name as the key and the result value as the value.

Return type:: OrderedDict

to_list(attributes=None)[source]

Returns a result attribute as a list. Can be useful when needing to pass results to other methods of functions that require unnamed values.

Return type:: list

get_eval_set_name(model, objective)[source]

Returns the evaluation set name for a given model and objective which is needed to access certain results.

Parameters:

model (str, ModelBase) – The model or model name of interest
obj (str, Objective) – The objective or objective name of interest

Returns:

The evaluation set name

Return type:

str

decompose_evaluation_name(evaluation_set_name)[source]

Returns the model name and objective name given an evaluation set name.

Parameters:: evaluation_set_name (str) – The model or model name of interest
Returns:: The model name and objective name
Return type:: tuple(str,str)

class matcal.core.study_base.ObjectiveInformation(record_objectives=True, record_residuals=True, record_weighted_conditioned=False)[source]

Contains the objective information for a study. Objective information includes the objectives, the residuals and the weighted and conditioned residuals as list of DataCollection objects.

Variables:

residuals (list(DataCollection)) – A list of residuals in order of parameter evaluation for the study.
weighted_conditioned_residuals (list(DataCollection)) – list(DataCollection) A list of weighted and conditioned residuals in order of parameter evaluation for the study.
objectives (list(DataCollection)) – list(DataCollection) A list of objectives in order of parameter evaluation for the study.

class matcal.core.study_base.QoiInformation(record_data=True, record_qois=True, record_weighted_conditioned=False)[source]

Contains the QoI information for a study. QoI information includes the experiment data, the experiment QoIs, and the weighted and conditioned experiment QoIs as DataCollection objects.

It also contains the simulation QoIs and simulation weighted and conditioned QoIs as lists of DataCollection objects.

Variables:

experiment_data (DataCollection) – The experimental data
experiment_qois (DataCollection) – The experimental QoIs
experiment_weighted_conditioned_qois (DataCollection) – The weighted and conditioned experiment QoIs
simulation_qois (list(DataCollection)) – The simulation QoIs in order of parameter evaluation for the study.
simulation_weighted_conditioned_qois (list(DataCollection)) – list(DataCollection) The weighted and conditioned simulation QoIs in order of parameter evaluation for the study.