Datasets

Sample creation

The espm.datasets module implements the functions that combines the spatial distributions generated from espm.weights and the spectra generated from espm.models into a 3D dataset. This part of the espm package manages the integration into the hyperspy framework : - the datasets and their metadata are stored as hyperspy signals (.hspy). - the espm.eds_spim module implements the EDS_espm class, which is a subclass of the hyperspy.signals.Signal1D class.

Using the EDS_espm class, the user can easily use most of the hyperspy functionalities (e.g. plotting, fitting, decomposition, etc.) as well as the espm functionalites on their experimental and simulated data.

Note

For now espm supports only the signals modeled as EDS data but we aim at implementing the signals corresponding to EELS data too.

The module espm.datasets.base implements the functions that combines a spatial distribution and associated spectra into a 3D dataset. It also implements the functions to convert the dataset into hyperspy compatible objects.

espm.datasets.base.generate_dataset(*args, base_path=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/espm/envs/latest/lib/python3.11/site-packages/generated_datasets'), sample_number=10, base_seed=0, elements=[], **kwargs)[source]

Generate a set of spectrum images files and save them in the generated dataset folder. Each spectrum image is saved in a separate file and was generated using a different seed.

Parameters:

base_pathstr, optional: The path to the folder where the samples will be saved. The default is DATASETS_PATH.
sample_numberint, optional: The number of samples to generate. The default is 10.
base_seedint, optional: The seed used to generate the samples. The default is 0.

Returns:

None.

espm.datasets.base.generate_spim(phases, weights, densities, N, seed=0, continuous=False)[source]

Generate a noiseless spectrum image as tensor product of the phases and weights. Then, if asked for, a noisy spectrum image is generated by drawing from a Poisson distribution.

The noiseless spectrum image is defined as:

\[Y^{nl} = N D \otimes ( Diag(d) A )\]

where \(D\) is the normalized phases, \(A\) is the weights, \(d\) is the density modifier and \(N\) is the number of counts per pixel.

To obtain the noisy spectrum image, the noiseless spectrum image is drawn from a Poisson distribution.

Parameters:

phasesarray_like: The phases of the model. Shape (n, spectral_len).
weightsarray_like: The weights of the model. Shape (shape_2d[0], shape_2d[1], n).
densitiesarray_like: Density modifier of the phases. Shape (n,).
Nint: The number of counts per pixel.
seedint, optional: Seed for the random number generator. The default is 0.
continuousbool, optional: If True, the function returns a noiseless spectrum image. The default is False.

Returns:

numpy.ndarray: The spectrum image. Shape (shape_2d[0], shape_2d[1], spectral_len).

Notes

More details about the spectrum image generation can be found in the contribution: [TPH+23].

espm.datasets.base.generate_spim_sample(phases, weights, model_params, misc_params, seed=0, g_params={})[source]

Generate a dictionary containing: the spectrum image (made with the weights and phases), the ground truth, the model parameters and the misc parameters.

Parameters:

phasesarray_like: The phases of the model. Shape (n, spectral_len).
weightsarray_like: The weights of the model. Shape (shape_2d[0], shape_2d[1], n). The weights should sum to one along axis 2.
model_paramsdict: The parameters of the model. For examples see the default parameters in espm.conf.
misc_paramsdict: The misc parameters of the model. For examples see the default parameters in espm.conf.
seedint, optional: The seed for the random number generator. The default is 0.
g_paramsdict, optional: The parameters for the g matrix. The default is {}. Note that for EDXS data the g matrix is not used during the creation of the data.

Returns:

sampledict: A dictionary containing the spectrum image, the ground truth, the model parameters and the misc parameters.

espm.datasets.base.sample_to_EDS_espm(sample, elements=[])[source]

Convert dataset to a custom hyperspy signal type called EDS_espm containing the noisy spectrum image as data, the ground truth as metadata and other useful information.

Parameters:

sampledict: A dictionary containing the noisy spectrum image as data, the ground truth as metadata and other useful information. See espm.datasets.base.generate_spim_sample() for more details.
elementslist, optional: A list of the elements present in the sample. The default is [].

Returns:

EDS_espm: The hyperspy compatible signal object of the espm.eds_spim module.

espm.datasets.base.sample_to_Signal1D(sample)[source]: Same as espm.datasets.base.sample_to_EDS_espm() but for non-EDS data such as the toy dataset.

The module espm.datasets.built_in_EDXS_datasets implements the functions that generate two built-in datasets: - A dataset of 2 particles embedded in a matrix. - A dataset with a linear local accumulation of Sr.

espm.datasets.built_in_EDXS_datasets.generate_built_in_datasets(seeds_range=10)[source]

Generate the two built-in datasets if they are not already present in the datasets folder.

Parameters:

seeds_rangeint: The number of seeds to use for the generation of the built-in datasets. The built-in datasets are generated with a base_seed, and then the base_seed + 1, base_seed + 2, etc. up to base_seed + seeds_range -1.

Returns:

None

espm.datasets.built_in_EDXS_datasets.load_grain_boundary(sample=0)[source]

Load the built-in dataset of a grain boundary.

Parameters:

sampleint: The sample number to load.

Returns:

spimhyperspy.signals.EDS_espm: The loaded dataset.

espm.datasets.built_in_EDXS_datasets.load_particules(sample=0)[source]

Load the built-in dataset of particles.

Parameters:

sampleint: The sample number to load.

Returns:

spimhyperspy.signals.EDS_espm: The loaded dataset.

The module espm.eds_spim implements the EDS_espm class, which is a subclass of the hyperspy.signals.Signal1D class. The main purpose of this class is to provide an easy and clean interface between the hyperspy framework and the espm package: - The metadata are organised to correspond as much as possible to the typical metadata that can be found in hyperspy EDS_TEM object. - The machine learning algorithms of espm can be easily applied to the EDS_espm object using the standard hyperspy decomposition method. See the notebooks for examples. - The EDS_espm provides a convinient way to:

get the results of espm.estimators.NMFEstimator

access ground truth in case of simulated data

estimate best binning thanks to the method developed by G. Obozinski, N. Perraudin and M. Martinez Ruts.

set fixed W for the espm.estimators.NMFEstimator decomposition

class espm.datasets.eds_spim.EDS_espm(*args, **kwargs)[source]

add_additional_parameters(thickness=2e-05, density=3.5, width_slope=0.01, width_intercept=0.065, xray_db='default_xrays.json')[source]: Helper function to set the metadata that are specific to the espm package so that it does not overwrite experimental metadata. See the documentation of the set_analysis_parameters() function for the meaning of the parameters.

add_analysis_parameters(beam_energy=200, azimuth_angle=0.0, elevation_angle=22.0, tilt_stage=0.0, elements=[], thickness=2e-05, density=3.5, detector_type='SDD_efficiency.txt', width_slope=0.01, width_intercept=0.065, xray_db='default_xrays.json')[source]

Helper function to set the metadata of the EDS_espm object without overwriting the existing metadata.

Parameters:

beam_energyfloat, optional: The energy of the electron beam in keV.
azimuth_anglefloat, optional: The azimuth angle of the EDS detector in degrees.
elevation_anglefloat, optional: The elevation angle of the EDS detector in degrees.
tilt_stagefloat, optional: The tilt angle of the sample stage in degrees (usually it correspond to alpha on FEI instruments).
elementslist, optional: List of the elements to be used in the analysis.
thicknessfloat, optional: The thickness of the sample in centimeters.
densityfloat, optional: The density of the sample in g/cm^3.
detector_typestr, optional: The type of the detector. It is either the name of a text file containing the efficiency or a modeling dictionary. (see the espm.models.absorption_edxs.det_efficiency function)
width_slopefloat, optional: The slope of the linear fit of the detector width as a function of the energy.
width_interceptfloat, optional: The intercept of the linear fit of the detector width as a function of the energy.
xray_dbstr, optional: The name of the X-ray emission cross-section database to be used. The default tables are avalaible in the espm/tables folder. Additional tables can be generated by emtables.

add_elements(*, elements=[])[source]

Add elements to the existing list of elements in the metadata.

Parameters:

elementslist, optional: List of the elements to be added to the existing list of elements in the metadata.
Note
The input can be either atomic numbers or chemical symbols.
For example elements = [‘Si’, ‘26’, 22] will result in [‘Si’, ‘Fe’, ‘Ti’].

add_microscope_parameters(beam_energy=200, azimuth_angle=0.0, elevation_angle=22.0, tilt_stage=0.0, detector_type='SDD_efficiency.txt')[source]: Helper function to set the microscope parameters of the EDS_espm object. Be careful, it will overwrite the microscope parameters of the object. See the documentation of the set_analysis_parameters() function for the meaning of the parameters.

build_G(problem_type='bremsstrahlung', *, elements_dict={})[source]

Build the G matrix of the espm.models.EDXS model corresponding to the metadata of the EDS_espm object and stores it as an attribute.

Parameters:

problem_typestr, optional

Determines the type of the G matrix to build. It can be “bremsstrahlung”, “no_brstlg” or “identity”. The parameters correspond to:

“bremsstrahlung” : the G matrix is a callable with both characteristic X-rays and a bremsstrahlung model.
“no_brstlg” : the G matrix is a matrix with only characteristic X-rays.
“identity” : the G matrix is None which is equivalent to an identity matrix for espm functions.

elements_dictdict, optional

Dictionary containing atomic numbers and a corresponding cut-off energies. It is used to separate the characteristic X-rays of the given elements into two energies ranges and assign them each a column in the G matrix instead of having one column per element. For example elements_dict = {“26”,3.0} will separate the characteristic X-rays of the element Fe into two energies ranges and assign them each a column in the G matrix. This is useful to circumvent issues with the absorption.

Returns

——-

GNone or numpy.ndarray or callable

The G matrix of the espm.models.EDXS model corresponding to the metadata of the EDS_espm object.

build_ground_truth(reshape=True)[source]

Get the ground truth stored in the metadata of the EDS_espm object, if available. The reshape arguments can be used to get the ground truth in a form easier to use for machine learning algorithms.

Parameters:

reshapebool, optional: If False, the ground truth is returned in the form of a 3D array of shape (shape_2d[0],shape_2d[1],n_phases) and a 2D array of shape (n_phases,n_features).

Returns:

phasesnumpy.ndarray: The ground truth of the spectra of the phases.
weightsnumpy.ndarray: The ground truth of the spatial distribution of the phases.

carto_fixed_W(brstlg_comps=1)[source]

Helper function to create a fixed_W matrix for chemical mapping. It will output a matrix It can be used to make a decomposition with as many components as they are chemical elements and then allow each component to have only one of each element. The spectral components are then the characteristic peaks of each element and the spatial components are the associated chemical maps. The bremsstrahlung is calculated separately and added to other components.

Parameters:

brstlg_compsint, optional: Number of bremsstrahlung components to add to the decomposition.

Returns:

Wnumpy.ndarray

concentration_report(selected_elts=[], W_input=None, fit_error=True)[source]

estimate_best_binning(inspect=False)[source]

Estimate the best binning for the dataset based on the method developed by G. Obozinski, N. Perraudin and M. Martinez Ruts. M. Martinez Ruts has designed an estimator that compares the binned and unbinned data and its minimum gives the best binning factor.

Parameters:

bin_samplingint, optional: Number of binning factors to sample for the estimation.
inspectbool, optional: If True, the function will return the values of the estimator for each binning factor and the estimated best binning factor. If False, it will return only the estimated best binning factor.

Returns:

estimated_binningtuple: The estimated binning for the dataset.

print_concentration_report(selected_elts=[], W_input=None, fit_error=True, disclaimer=True)[source]

Print a report of the chemical concentrations from a fitted W.

Parameters:

selected_eltslist, optional: List of the elements to be printed. If empty, all the elements will be printed.
W_inputnumpy.ndarray, optional: If not None, the concentrations will be computed from this W matrix instead of the one fitted during the decomposition.
fit_errorbool, optional: If True, the statistical errors on the concentrations will be printed.
disclaimerbool, optional: If True, a disclaimer will be printed at the end of the report.

Returns:

None

Notes

This function is only available if the learning results contain a decomposition algorithm that has been fitted.

set_additional_parameters(thickness=2e-05, density=3.5, width_slope=0.01, width_intercept=0.065, xray_db='default_xrays.json')[source]: Helper function to set the metadata that are specific to the espm package so that it does not overwrite experimental metadata. See the documentation of the set_analysis_parameters() function for the meaning of the parameters.

set_analysis_parameters(beam_energy=200, azimuth_angle=0.0, elevation_angle=22.0, tilt_stage=0.0, elements=[], thickness=2e-05, density=3.5, detector_type='SDD_efficiency.txt', width_slope=0.01, width_intercept=0.065, xray_db='default_xrays.json')[source]

Helper function to set the metadata of the EDS_espm object. Be careful, it will overwrite the metadata of the object.

Parameters:

beam_energyfloat, optional: The energy of the electron beam in keV.
azimuth_anglefloat, optional: The azimuth angle of the EDS detector in degrees.
elevation_anglefloat, optional: The elevation angle of the EDS detector in degrees.
tilt_stagefloat, optional: The tilt angle of the sample stage in degrees (usually it correspond to alpha on FEI instruments).
elementslist, optional: List of the elements to be used in the analysis.
thicknessfloat, optional: The thickness of the sample in centimeters.
densityfloat, optional: The density of the sample in g/cm^3.
detector_typestr, optional: The type of the detector. It is either the name of a text file containing the efficiency or a modeling dictionary. (see the espm.models.absorption_edxs.det_efficiency function)
width_slopefloat, optional: The slope of the linear fit of the detector width as a function of the energy.
width_interceptfloat, optional: The intercept of the linear fit of the detector width as a function of the energy.
xray_dbstr, optional: The name of the X-ray emission cross-section database to be used. The default tables are avalaible in the espm/tables folder. Additional tables can be generated by emtables.

set_elements(*, elements=[])[source]

Replace elements with the input list in the metadata.

Parameters:

elementslist, optional: List of the elements to be added in the metadata.
Note
The input can be either atomic numbers or chemical symbols.
For example elements = [‘Si’, ‘26’, 22] will result in [‘Si’, ‘Fe’, ‘Ti’].

set_fixed_W(phases_dict)[source]

Helper function to create a fixed_W matrix. The output matrix will have -1 entries except for the elements (and bremsstrahlung parameters) that are present in the phases_dict dictionary. In the output (fixed_W) matrix, the -1 entries will be ignored during the decomposition using espm.estimator.NMFEstimator are normally learned while the non-negative entries will be fixed to the values given in the phases_dict dictionary. Usually, the easiest is to fix some elements to 0.0 in some phases if you want to improve unmixing results. For example, if you have a phase with only Si and O, you can fix the Fe element to 0.0 in this phase.

Parameters:

phases_dictdict: Determines which elements of fixed_W are going to be non-negative. The dictionnary has typically the following structure : phases_dict = {“phase1_name” : {“Fe” : 0.0, “O” : 1.25e23}, “phase2_name” : {“Si” : 0.0, “b0” : 0.05}}.
Returns
——-
Wnumpy.ndarray

set_microscope_parameters(beam_energy=200, azimuth_angle=0.0, elevation_angle=22.0, tilt_stage=0.0, detector_type='SDD_efficiency.txt')[source]: Helper function to set the microscope parameters of the EDS_espm object. Be careful, it will overwrite the microscope parameters of the object. See the documentation of the set_analysis_parameters() function for the meaning of the parameters.

property G: The G matrix of the espm.models.EDXS model corresponding to the metadata of the EDS_espm object.

property X: The data in the form of a 2D array of shape (n_samples, n_features).

property Xdot: The ground truth in the form of a 3D array of shape (shape_2d[0],shape_2d[1],n_features), if available.

property custom_init: Boolean setting whether using the custom_init (see espm.models.EDXS) or not. If True, the custom_init will be used to initialise the decomposition. If False, the default initialisation will be used. If None, the custom_init will be set to False.

property maps: Ground truth of the spatial distribution of the phases in the form of a 3D array of shape (shape_2d[0],shape_2d[1],n_phases), if available.

property maps_2d: Ground truth of the spatial distribution of the phases in the form of a 2D array of shape (shape_2d[0]*shape_2d[1],n_phases), if available.

property model: The espm.models.EDXS model corresponding to the metadata of the EDS_espm object.

property phases: Ground truth of the spectra of the phases in the form of a 2D array of shape (n_phases,n_features), if available.

property shape_2d: Shape of the data in the spatial dimension.

espm.datasets.eds_spim.get_metadata(spim)[source]: Get the metadata of the EDS_espm object and format it as a model parameters dictionary.