aurel.reading

reading.py

This module provides comprehensive functionality for reading and writing numerical relativity simulation data, with specialized support for Einstein Toolkit (ET) simulations. The module handles complex data structures including multiple restart files, chunked data, variable groupings, and different refinement levels.

Key functions relevant for Aurel: read_data and save_data. All other functions are auxiliary utilities for Einstein Toolkit data handling.

Core Einstein Toolkit Functionality

  • Parameter extraction: Read simulation parameters from .par files

  • Iteration management: List and track available simulation iterations

  • Content listing: List available variables and their file associations

  • Data reading: Read 3D simulation data from HDF5 files

  • Data chunking: Join data chunks from distributed file systems

  • Caching: Save/load data in per-iteration formats for faster access

Environment Setup

Users must set the SIMLOC environment variable to the simulation directories:

export SIMLOC="/path/to/simulations/"

For multiple simulation locations, use colon separation:

export SIMLOC="/path1:/path2:/path3"

aurel.reading.bash(command)[source]

Execute a bash command and return its output as a string.

This utility function provides a simple interface for running shell commands from within Python, primarily used for file system operations and parameter file reading in simulation data processing.

Parameters:

command (str) – The bash command to execute.

Returns:

The command output as a UTF-8 decoded string with leading/trailing whitespace stripped. Empty string if command produces no output.

Return type:

str

Notes

This function uses subprocess.check_output() which will raise an exception if the command fails. Consider using appropriate error handling for production code.

aurel.reading.collect_overall_iterations(its_available, verbose)[source]

Merge iteration data across all restarts into overall summary.

This is a helper function for iterations().

This function analyzes the iteration information from individual restart and merges them into a comprehensive overview of what iterations are available across the entire simulation. It handles multiple refinement levels and various iteration patterns: overlapping iteration ranges, different time stepping patterns, single iterations. These are merged when possible, otherwise they are kept as separate entries.

Parameters:

  • its_available (dict) – Dictionary with iterations available for each restart, typically from read_iterations() or iterations().

  • verbose (bool) – If True, print the overall iterations summary to stdout.

Returns:

The input dictionary with an added ‘overall’ key containing merged iteration information across all restarts:

{
    restart_number: {...},  # Original restart data preserved
    'overall': {
        'rl = 0': [[itmin, itmax, dit], ...],
        'rl = 1': [[itmin, itmax, dit], ...],
        ...
    }
}

Return type:

dict

aurel.reading.fixij(f)[source]

Fix the x-z indexing as you read in the data.

aurel.reading.get_content(param, **kwargs)[source]

Analyze Einstein Toolkit output directory and map variables to files.

This function creates a comprehensive mapping between simulation variables and their corresponding HDF5 files, handling both single-variable and grouped-variable file organizations. It implements intelligent caching to avoid expensive file scanning on repeated calls.

The function performs several key operations: 1. Scans all HDF5 files in the directory 2. Determines file organization (single vs. grouped variables) 3. Maps each variable to its containing files 4. Groups variables that share the same file sets 5. Caches results for faster subsequent access

Parameters:

  • param (dict) –

    Simulation parameters dictionary containing:

    • ’simpath’: Path to simulation root directory

    • ’simname’: Simulation name

  • restart (int, optional) – Restart number to analyze. Default 0. Used to construct the path to the ET output directory containing .h5 files. Example: “/simulations/BHB/output-0000/BHB/”

  • overwrite (bool, optional) – Overwrite existing content file. Default False.

  • verbose (bool, optional) – Print progress information during processing. Default True. Shows file scanning progress and caching status.

  • veryverbose (bool, optional) – Print extra progress information during processing. Default False.

Returns:

Dictionary mapping variable groups to their file lists:

{
    ('var1', 'var2'): ['path/to/file1.h5', 'path/to/file2.h5'],
    ('var3',): ['path/to/file3.h5', 'path/to/file4.h5'],
    ...
}

Return type:

dict

Notes

  • Cache file ‘content.txt’ created in the target directory

  • Variable names follow Einstein Toolkit conventions

  • If variables are grouped in files and are not in the known_groups, the variable names will be retrieved from the keys in the file. Depending on the file size, this may take some time.

aurel.reading.iterations(param, **kwargs)[source]

Analyze and catalog available iterations across all simulation restarts.

This function systematically scans Einstein Toolkit simulation output to determine what data is available at each iteration and refinement level across all restart files. It creates a comprehensive catalog saved to simpath/simname/iterations.txt for quick access by other analysis functions.

Parameters:

  • param (dict) –

    Simulation parameters from parameters() function. Must contain:

    • ’simpath’: Path to simulation directory (with trailing slash)

    • ’simname’: Simulation name

  • skip_last (bool, optional) – Skip the last restart directory. Default True. Useful to avoid interrupting running simulations.

  • verbose (bool, optional) – Print detailed progress information. Default True. Shows contents of iterations.txt file and overall summary.

Returns:

Dictionary containing iteration information for all restarts:

{
    restart_number: {
        'var available': [variable names],
        'its available': [itmin, itmax],
        'rl = 0': [itmin, itmax, dit],
        'rl = 1': [itmin, itmax, dit],
        ...
    },
    ...
    'overall': {
        'rl = 0': [[itmin, itmax, dit], ...],
        'rl = 1': [[itmin, itmax, dit], ...],
        ...
    }
}

Return type:

dict

Notes

Creates/appends to ‘{simpath}/{simname}/iterations.txt’ with format:

=== restart 0
3D variables available: ['rho', 'alpha', 'gxx', ...]
it = 0 -> 1000
rl = 0 at it = np.arange(0, 1000, 2)
rl = 1 at it = np.arange(0, 1000, 1)
=== restart 1
...

Also with verbose=True, prints the above and overall iterations from collect_overall_iterations

=== Overall iterations
rl = 0 at it = [np.arange(0, 2000, 2)]
rl = 1 at it = [np.arange(0, 2000, 1)]
aurel.reading.join_chunks(cut_data, **kwargs)[source]

Join the chunks of data together.

Parameters:

  • cut_data (dict of dict) – A dictionary containing the data from the simulation output files. dict.keys() = tuple of the chunks ‘iorigin’ attribute which maps out how the data is to be joined together.

  • veryextraverbose (bool, optional) – If True, print additional information during the joining process. The default is False.

Returns:

uncut_data – The data now joined together.

Return type:

array_like

aurel.reading.parameters(simname)[source]

Read and parse Einstein Toolkit simulation parameters.

Extracts comprehensive simulation metadata from Einstein Toolkit parameter files (.par), including grid configuration, restart information, and all thorn-specific parameters.

The function automatically: - Locates parameter files - Counts restart files to determine simulation state - Calculates derived grid parameters (extents, grid points) - Parses thorn parameters with proper type conversion

Parameters:

simname (str) – Name of the simulation directory to analyze. Must exist in one of the SIMLOC directories.

Returns:

Comprehensive parameter dictionary containing:

Core Information:

’simname’ : str, Simulation name (input parameter)

’simulation’ : str, Data format identifier (‘ET’)

’simpath’ : str, Path to simulation root directory

’datapath’ : str, Path to output-0000 data directory

Grid Parameters:

’xmin’, ‘xmax’, ‘Lx’, ‘Nx’, ‘dx’ : float/int, minimum, maximum, extent, grid points and spacing. Same for Y and Z-direction parameters.

Thorn Information:

’list_of_thorns’ : list of str, All active thorns in the simulation

Parameter Values:

All parameters from the .par file with appropriate types. Conflicting parameter names include thorn prefix.

Return type:

dict

Notes

The SIMLOC environment variable must be set:

export SIMLOC="/path/to/simulations"

For multiple locations:

export SIMLOC="/path1:/path2:/path3"

aurel.reading.parse_h5file(filepath)[source]

Parse HDF5 filename into its components.

Parameters:

filepath (str) –

HDF5 file path or filename. Can be either:

  • Full absolute/relative path: “/path/to/admbase-metric.h5”

  • Just filename: “rho.xyz.h5”

Returns:

Returns None if the filename doesn’t match the expected format. Otherwise returns dictionary with parsed components:

{
    'thorn_group': str or None, # e.g., 'admbase-' or None
    'thorn': str or None,       # e.g., 'admbase' or None
    'variable_or_group': str,   # e.g., 'metric' or 'rho'
    'xyz_prefix': str or None,  # e.g., '.xyz' or None
    'file_number': str or None, # e.g., '.file_123' or None
    'xyz_suffix': str or None,  # e.g., '.xyz' or None
    'is_group_file': bool,      # True -> a grouped variable file
    'chunk_number': int or None # e.g., 123 or None (from file_nbr)
}

Return type:

dict or None

aurel.reading.parse_hdf5_key(key)[source]

Parse HDF5 dataset key into its components.

Parameters:

key (str) – HDF5 dataset key in format like “admbase::gxx it=0 tl=0 m=0 rl=0 c=0”

Returns:

Returns None if the key doesn’t match the expected format. Otherwise returns dictionary with parsed components:

{
    'thorn': str,        # e.g., 'admbase'
    'variable': str,     # e.g., 'gxx'
    'it': int,          # iteration number
    'tl': int,          # time level
    'm': int or None,   # m value (if present)
    'rl': int or None,  # refinement level (if present)
    'c': int or None    # chunk number (if present)
}

Return type:

dict or None

aurel.reading.read_ET_data(param, **kwargs)[source]

Read Einstein Toolkit simulation data with intelligent restart handling.

This is the main Einstein Toolkit data reading function that handles the complexity of multi-restart simulations, variable groupings, and chunking. It provides a unified interface for accessing ET simulation data regardless of the underlying file organization.

Key Features:

  • Restart Management: Automatically finds iterations across restarts

  • Hybrid Reading: Combines cached per-iteration files with ET files

  • Missing Data Handling: Fills gaps by reading from original ET files

  • Variable Mapping: Handles Aurel to Einstein Toolkit name conversions

  • Caching Strategy: Saves data in optimized format for future access

Parameters:

  • param (dict) – Simulation parameters from parameters() function.

  • it (list of int, optional) – Iterations to read. Default [0].

  • vars (list of str, optional.) – Variables in Aurel format. Default [] (all available). Examples: [‘rho’, ‘Kdown3’], [‘gammadown3’, ‘alpha’]

  • restart (int, optional) – Specific restart to read from. Default -1 (auto-detect).

  • split_per_it (bool, optional) – Use cached per-iteration files when available and save read data in per-iteration files. Default True.

  • verbose (bool, optional) – Print progress information. Default True.

  • veryverbose (bool, optional) – Print detailed debugging information. Default False.

Returns:

Simulation data dictionary with structure:

{
    'it': [iteration_numbers],
    't': [time_values],
    'var1': [data_arrays],
    'var2': [data_arrays],
    ...
}

Return type:

dict

Notes

Processing Workflow:

  1. Iteration Location: Determines which restart contains each iteration

  2. Cache Reading: Loads available data from per-iteration files

  3. Gap Detection: Identifies missing iterations/variables

  4. ET Reading: Fills gaps by reading original Einstein Toolkit files

  5. Data Integration: Combines cached and newly-read data

  6. Cache Update: Saves newly-read data for future access

aurel.reading.read_ET_group_or_var(variables, files, cmax, **kwargs)[source]

Read variables from Einstein Toolkit simulation output files.

Parameters:

  • variables (list) – The variables to read from the simulation output files. Each variable is a string that identifies the variable. These should all be found in the same files.

  • files (list) – The list of files to read the variables from. Each file is a string that identifies the file. These should all contain the same variables just at different chunks.

  • cmax (int) – The maximum number of chunks to read from the simulation output files. If ‘in file’, it will be extracted from the file.

  • it (list, optional) – The iterations to save from the data. The default is [0].

  • rl (int, optional) – The refinement level to read from the simulation output files. The default is 0.

Returns:

A dictionary containing the data from the simulation output files.

dict.keys() = [‘it’, ‘t’, var]

Return type:

dict

aurel.reading.read_ET_variables(param, var, vars_and_files, **kwargs)[source]

Read the data from Einstein Toolkit simulation output files.

Parameters:

  • param (dict) – The parameters of the simulation.

  • var (list) – The variables to read from the simulation output files.

  • it (list, optional) – The iterations to save from the data. The default is [0].

  • rl (int, optional) – The refinement level to read from the simulation output files. The default is 0.

  • restart (int, optional) – The restart number to save the data to. The default is 0.

  • veryverbose (bool, optional) – If True, print additional information during the joining process. The default is False.

Returns:

A dictionary containing the data from the simulation output files. dict.keys() = [‘it’, ‘t’, var[0], var[1], …]

Return type:

dict

aurel.reading.read_aurel_data(param, **kwargs)[source]

Read data from Aurel format simulation output files.

Aurel format stores simulation data in per-iteration HDF5 files with the naming convention ‘it_<iteration>.hdf5’. Each file contains variables organized by refinement level with keys like ‘varname rl=0’.

This format is optimized for easy parallelization across iterations.

Parameters:

  • param (dict) – Simulation parameters dictionary. Must contain either: - ‘datapath’: Direct path to directory with it_*.hdf5 files, OR - ‘simulation’, ‘simpath’, ‘simname’: For ET-style directory structure

  • it (list of int, optional) – Iteration numbers to read. Default [0].

  • vars (list of str, optional) – Variable names to read. Default [] (read all available variables).

  • rl (int, optional) – Refinement level to read. Default 0.

  • restart (int, optional) – Restart number (for ET-style directory structure). Default 0.

  • verbose (bool, optional) – Print progress information. Default False.

  • veryverbose (bool, optional) – Print detailed debugging information. Default False.

Returns:

Dictionary with simulation data:

{
    'it': [iteration_numbers], # ints
    't': [time_values],        # floats or None if not available
    'var1': [data_arrays],     # arrays or Nones if not available
    'var2': [data_arrays],
    ...
}

Return type:

dict

aurel.reading.read_data(param, **kwargs)[source]

Read simulation data from Aurel or Einstein Toolkit format files.

This is the main entry point for reading numerical relativity simulation data. It automatically detects the data format and routes to the appropriate reading function. Supports both Einstein Toolkit HDF5 output and Aurel’s optimized per-iteration format.

Parameters:

  • param (dict) – Simulation parameters dictionary containing metadata about the simulation. If ‘simulation’ key is present, treats as Einstein Toolkit data; otherwise treats as Aurel format. In Einstein Toolkit case, you can use the parameters function.

  • it (list of int, optional) – Iteration numbers to read. Default [0].

  • vars (list of str, optional) – Variable names to read. Default [] (read all available).

  • rl (int, optional) – Refinement level to read. Default 0 (coarsest level).

  • restart (int, optional) – Specific restart number to read from. Default -1 (auto-detect). Set to specific value to read from that restart only.

  • split_per_it (bool, optional) – For ET data: whether to use per-iteration files. Default True. When True, reads from cached files and fills missing data from ET files, then saves newly read ET data to per-iteration files. When False, reads exclusively from ET files.

  • verbose (bool, optional) – Print detailed progress information. Default False.

  • veryverbose (bool, optional) – Print very detailed debugging information. Default False.

Returns:

Dictionary containing the simulation data with keys:

{
    'it': [iteration_numbers], # ints
    't': [time_values],        # floats or None if not available
    'var1': [data_arrays],     # arrays or Nones if not available
    'var2': [data_arrays],
    ...
}

Return type:

dict

aurel.reading.read_iterations(param, **kwargs)[source]

Read and parse the iterations.txt file to extract iteration information.

This is a helper function for iterations().

This function loads the iterations catalog created by iterations() and parses it into a structured dictionary format for programmatic access. If the iterations.txt file doesn’t exist, then the iterations() function is called instead.

Parameters:

  • param (dict) –

    Simulation parameters from parameters() function. Must contain:

    • ’simpath’: Path to simulation directory

    • ’simname’: Simulation name

  • skip_last (bool, optional) – Skip the last restart directory when creating iterations.txt. Default True. This is passed to iterations().

  • verbose (bool, optional) – Print detailed parsing information. Default False. Also passed to iterations().

Returns:

Dictionary containing iteration information for all restarts:

{
    restart_number: {
        'var available': [variable names],
        'its available': [itmin, itmax],
        'rl = 0': [itmin, itmax, dit],
        'rl = 1': [itmin, itmax, dit],
        ...
    },
    ...
}

Return type:

dict

aurel.reading.save_data(param, data, **kwargs)[source]

Save simulation data to Aurel format HDF5 files.

Saves simulation data in the per-iteration format used by Aurel, where each iteration is stored in a separate HDF5 file named ‘it_<iteration>.hdf5’. Variables are stored as datasets with keys like ‘varname rl=<level>’.

This format provides several advantages:

  • Fast random access to specific iterations

  • Parallel I/O capabilities

  • Efficient storage for sparse temporal sampling

Parameters:

  • param (dict) – Simulation parameters dictionary containing path information. Should include either: - ‘datapath’: Direct path where files will be saved, OR - ‘simulation’ ‘simpath’, ‘simname’: For ET-style directory structure

  • data (dict) –

    Dictionary containing simulation data to save. Expected structure:

    {
    'it': [iteration_numbers],
    't': [time_values],
    'var1': [data_arrays],
    'var2': [data_arrays],
    ...
}

    Each variable should have one array per iteration.

  • vars (list of str, optional) – Variables to save. Default [] (save all variables in data). Note: ‘it’ and ‘t’ are automatically included if present.

  • it (list of int, optional) – Iterations to save. Default [0].

  • rl (int, optional) – Refinement level for saved data. Default 0. Used in HDF5 dataset keys: ‘varname rl=<rl>’ Can just represent grids of different shapes.

  • restart (int, optional) – Restart number (for ET-style paths). Default 0.

Notes

  • Overwrites existing datasets with the same name

  • Skips variables with None values

aurel.reading.saveprint(it_file, some_str, verbose=True)[source]

Save the string to the file and print it to the screen.

aurel.reading.transform_vars_ET_to_aurel(var)[source]

Transform ET variable name to Aurel variable name.

Parameters:

var (str) – String of variable name to transform.

Returns:

String of transformed variable name.

Return type:

str

aurel.reading.transform_vars_ET_to_aurel_groups(vars)[source]

Transform ET variable names to Aurel variable group names.

Parameters:

var (list of str) – List of variable names to transform.

Returns:

List of transformed variable names.

Return type:

list of str

aurel.reading.transform_vars_aurel_to_ET(var)[source]

Transform ET variable names to Aurel variable names.

Parameters:

var (list of str) – List of variable names to transform.

Returns:

List of transformed variable names.

Return type:

list of str

aurel.reading.transform_vars_tensor_to_scalar(var)[source]

Transform Aurel tensor variable names to their scalar component names.

This function decomposes tensor quantities into their individual scalar components for detailed analysis. For example, the 3-metric ‘gammadown3’ becomes [‘gxx’, ‘gxy’, ‘gxz’, ‘gyy’, ‘gyz’, ‘gzz’].

Parameters:

var (list of str) – List of Aurel tensor variable names to transform.

Returns:

List of scalar variable names. Tensor variables are expanded to their components, while scalar variables are kept as-is.

Return type:

list of str