otoole package#

Subpackages#

Submodules#

otoole.cli module#

Provides a command line interface to the otoole package

The key functions are convert, cplex and viz.

The convert command allows convertion of multiple different OSeMOSYS input formats including from/to csv, an AMPL format datafile, a folder of CSVs, an Excel workbook with one tab per parameter, an SQLite database

The cplex command provides access to scripts which transform and process a CPLEX solution file into a format which is more readily processed - either to CBC or CSV format.

The validate command checks the technology and fuel names against a standard or user defined configuration file.

The viz command allows you to produce a Reference Energy System diagram

Example

Ask for help on the command line:

>>> $ otoole --help
usage: otoole [-h] [--verbose] [--version] {convert,cplex,validate,viz} ...

otoole: Python toolkit of OSeMOSYS users

positional arguments:
{convert,cplex,validate,viz}
    convert             Convert from one input format to another
    cplex               Process a CPLEX solution file
    validate            Validate an OSeMOSYS model
    viz                 Visualise the model

optional arguments:
-h, --help            show this help message and exit
--verbose, -v         Enable debug mode
--version, -V         The version of otoole
class otoole.cli.DeprecateAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]#

Bases: Action

otoole.cli.data2res(args)[source]#

Get input data and call res creation.

otoole.cli.get_parser()[source]#
otoole.cli.main()[source]#
otoole.cli.setup(args)[source]#

Creates template data

otoole.cli.validate_model(args)[source]#

otoole.convert module#

This module implements the public API of the otoole package

Use the otoole convert function to convert between different file formats. Import the convert function from the otoole package:

>>> from otoole import convert
>>> convert('config.yaml', 'excel', 'datafile', 'input.xlsx', 'output.dat')
otoole.convert.convert(config, from_format, to_format, from_path, to_path, write_defaults=False, keep_whitespace=False) bool[source]#

Convert OSeMOSYS data from/to datafile, csv and Excel formats

Parameters
  • config (str) – Path to config file

  • from_format (str) – Available options are ‘datafile’, ‘datapackage’, ‘csv’ and ‘excel’

  • to_format (str) – Available options are ‘datafile’, ‘datapackage’, ‘csv’ and ‘excel’

  • from_path (str) – Path to destination file (if datafile or excel) or folder (csv or datapackage)

  • write_defaults (bool, default: False) – Write default values to CSVs

  • keep_whitespace (bool, default: False) – Keep whitespace in CSVs

Returns

True if conversion was successful, False otherwise

Return type

bool

otoole.convert.convert_results(config: str, from_format: str, to_format: str, from_path: str, to_path: str, input_format: str, input_path: str, write_defaults: bool = False, glpk_model: str = None) bool[source]#

Post-process results from a CBC, CPLEX, Gurobi, or GLPK solution file into CSV format

Parameters
  • config (str) – Path to config file

  • from_format (str) – Available options are ‘cbc’, ‘cplex’ and ‘gurobi’

  • to_format (str) – Available options are ‘csv’

  • from_path (str) – Path to cbc, cplex or gurobi solution file

  • to_path (str) – Path to destination folder

  • input_format (str) – Format of input data. Available options are ‘datafile’, ‘csv’ and ‘excel’

  • input_path (str) – Path to input data

  • write_defaults (bool) – Write default values to CSVs

  • glpk_model (str) – Path to *.glp model file

Returns

True if conversion was successful, False otherwise

Return type

bool

otoole.convert.read(config: str, from_format: str, from_path: str, keep_whitespace: bool = False) Tuple[Dict[str, DataFrame], Dict[str, float]][source]#

Read OSeMOSYS data from datafile, csv or Excel formats

Parameters
  • config (str) – Path to config file

  • from_format (str) – Available options are ‘datafile’, ‘csv’, ‘excel’ and ‘datapackage’ [deprecated]

  • from_path (str) – Path to source file (if datafile or excel) or folder (csv)

  • keep_whitespace (bool, default: False) – Keep whitespace in source files

Returns

Dictionary of parameter and set data and dictionary of default values

Return type

Tuple[dict[str, pd.DataFrame], dict[str, float]]

otoole.convert.read_results(config: str, from_format: str, from_path: str, input_format: str, input_path: str, glpk_model: str = None) Tuple[Dict[str, DataFrame], Dict[str, float]][source]#

Read OSeMOSYS results from CBC, GLPK, Gurobi, or CPLEX results files

Parameters
  • config (str) – Path to config file

  • from_format (str) – Available options are ‘cbc’, ‘gurobi’, ‘cplex’, and ‘glpk’

  • from_path (str) – Path to source file (if datafile or excel) or folder (csv)

  • input_format (str) – Format of input data. Available options are ‘datafile’, ‘csv’ and ‘excel’

  • input_path (str) – Path to input data

  • glpk_model (str) – Path to *.glp model file

Returns

Dictionary of parameter and set data and dictionary of default values

Return type

Tuple[dict[str, pd.DataFrame], dict[str, float]]

otoole.convert.write(config: str, to_format: str, to_path: str, inputs, default_values: Optional[Dict[str, float]] = None) bool[source]#

Write OSeMOSYS data to datafile, csv or Excel formats

Parameters
  • config (str) – Path to config file

  • to_format (str) – Available options are ‘datafile’, ‘csv’, ‘excel’ and ‘datapackage’ [deprecated],

  • to_path (str) – Path to destination file (if datafile or excel) or folder (csv))

  • inputs (dict[str, pd.DataFrame]) – Dictionary of pandas data frames to write

  • default_values (dict[str, float], default: None) – Dictionary of default values to write to datafile

otoole.exceptions module#

exception otoole.exceptions.OtooleConfigFileError(message: str)[source]#

Bases: OtooleException

Config file validation error

Parameters

message (str) – Message to display to users

exception otoole.exceptions.OtooleDeprecationError(resource, message)[source]#

Bases: OtooleException

New version of otoole does drops this feature support

Parameters
  • resource (str) – Name of the resource which is invalid

  • message (str) – Error message

exception otoole.exceptions.OtooleError(resource, message)[source]#

Bases: OtooleException

General purpose error

Parameters
  • resource (str) – Name of the resource which is invalid

  • message (str) – Error message

exception otoole.exceptions.OtooleExcelNameLengthError(name: str, message: str = 'Parameter name must be less than 31 characters when writing to Excel')[source]#

Bases: OtooleException

Invalid tab name for writing to Excel.

exception otoole.exceptions.OtooleException[source]#

Bases: Exception

Base class for all otoole exceptions.

exception otoole.exceptions.OtooleIndexError(resource, config_indices, data_indices)[source]#

Bases: OtooleException

Index data not consistent between data and config file

Parameters
  • resource (str) – Name of the resource which is invalid

  • config_indices (List[str]) – Indices from config file

  • data_indices (List[str]) – Indices from input data

exception otoole.exceptions.OtooleNameMismatchError(name: Union[List, str])[source]#

Bases: OtooleException

Names not consistent between read in data and config file

exception otoole.exceptions.OtooleRelationError(resource, foreign_resource, message)[source]#

Bases: OtooleException

Relations between input data is not correct

Parameters
  • resource (str) – Name of the resource which is invalid

  • foreign_resource (str) – Name of the resource which is invalid

  • message (str) – Error message

exception otoole.exceptions.OtooleSetupError(resource, message='Data already exists. Delete file/directory or pass the --overwrite flag')[source]#

Bases: OtooleException

Setup data already exists

Parameters
  • resource (str) – Name of the resource which is invalid

  • message (str) – Error message

exception otoole.exceptions.OtooleValidationError(resource, message)[source]#

Bases: OtooleException

Input data is invalid

Parameters
  • resource (str) – Name of the resource which is invalid

  • message (str) – Error message

otoole.input module#

The input module allows you to access the conversion routines programmatically

To use the routines, you need to instantiate a ReadStrategy and a WriteStrategy relevant for the format of the input and output data. You then pass these to a Context.

Example

Convert an in-memory dictionary of pandas DataFrames containing OSeMOSYS parameters to an Excel spreadsheet:

>>> from otoole import ReadMemory
>>> from otoole import WriteExcel
>>> from otoole import Context
>>> reader = ReadMemory(parameters)
>>> writer = WriteExcel()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('.', 'osemosys_to_excel.xlsx')

Convert a GNUMathProg datafile to a folder of CSV files:

>>> from otoole import ReadDataFile
>>> from otoole import WriteCsv
>>> from otoole import Context
>>> reader = ReadDataFile()
>>> writer = WriteCsv()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('my_datafile.txt', 'folder_of_csv_files')
class otoole.input.Context(read_strategy: ReadStrategy, write_strategy: WriteStrategy)[source]#

Bases: object

The Context defines the interface of interest to clients.

convert(input_filepath: str, output_filepath: str, **kwargs: Dict)[source]#

Converts from file input_filepath to file output_filepath

Parameters
  • input_filepath (str) –

  • output_filepath (str) –

property read_strategy: ReadStrategy#

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

property write_strategy: WriteStrategy#

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

class otoole.input.ReadStrategy(user_config: Dict[str, Dict])[source]#

Bases: Strategy

The Strategy interface declares operations common to all reading formats.

The Context uses this interface to call the algorithm defined by Concrete Strategies.

abstract read(filepath: Union[str, TextIO], **kwargs) Tuple[Dict[str, DataFrame], Dict[str, Any]][source]#

Reads in data from file

Parameters

filepath (Union[str, TextIO]) –

Returns

tuple of input_data as a dictionary of pandas DataFrames and dictionary of default values

Return type

Tuple[Dict[str, pd.DataFrame], Dict[str, Any]]

class otoole.input.Strategy(user_config: Dict[str, Dict])[source]#

Bases: ABC

Parameters

user_config (dict, default=None) – A user configuration for the input parameters and sets

property user_config: Dict#
class otoole.input.WriteStrategy(user_config: Dict, filepath: Optional[str] = None, default_values: Optional[Dict] = None, write_defaults: bool = False, input_data: Optional[Dict[str, DataFrame]] = None)[source]#

Bases: Strategy

The WriteStrategy interface declares operations common to all writing formats

The Context uses this interface to call the algorithm defined by Concrete Strategies.

Parameters
  • user_config (dict, default=None) –

  • filepath (str, default=None) –

  • default_values (dict, default=None) –

  • write_defaults (bool, default=False) –

  • input_data (dict, default=None) –

write(inputs: Dict[str, DataFrame], filepath: str, default_values: Dict[str, float], **kwargs)[source]#

Perform the conversion from dict of dataframes to destination format

otoole.read_strategies module#

class otoole.read_strategies.ReadCsv(user_config: Dict[str, Dict], keep_whitespace: bool = False)[source]#

Bases: _ReadTabular

Read in a folder of CSV files to a dict of Pandas DataFrames

Parameters
  • user_config (Dict[str, Dict]) – User configuration

  • keep_whitespace (bool) – Whether to keep whitespace in the dataframes

read(filepath, **kwargs) Tuple[Dict[str, DataFrame], Dict[str, Any]][source]#

Reads in data from file

Parameters

filepath (Union[str, TextIO]) –

Returns

tuple of input_data as a dictionary of pandas DataFrames and dictionary of default values

Return type

Tuple[Dict[str, pd.DataFrame], Dict[str, Any]]

class otoole.read_strategies.ReadDatafile(user_config: Dict[str, Dict])[source]#

Bases: ReadStrategy

Read in a datafile to a dict of Pandas DataFrames

Parameters
  • user_config (Dict[str, Dict]) – User configuration

  • keep_whitespace (bool) – Whether to keep whitespace in the dataframes

extract_param(config, name, datafile_parser, dict_of_dataframes) DataFrame[source]#
extract_set(datafile_parser, name, config, dict_of_dataframes) DataFrame[source]#
read(filepath, **kwargs) Tuple[Dict[str, DataFrame], Dict[str, Any]][source]#

Reads in data from file

Parameters

filepath (Union[str, TextIO]) –

Returns

tuple of input_data as a dictionary of pandas DataFrames and dictionary of default values

Return type

Tuple[Dict[str, pd.DataFrame], Dict[str, Any]]

read_in_datafile(path_to_datafile: str, config: Dict) Amply[source]#

Read in a datafile using the Amply parsing class

Parameters
  • path_to_datafile (str) –

  • config (Dict) –

class otoole.read_strategies.ReadExcel(user_config: Dict[str, Dict], keep_whitespace: bool = False)[source]#

Bases: _ReadTabular

Read in an Excel spreadsheet in wide format to a dict of Pandas DataFrames

Parameters
  • user_config (Dict[str, Dict]) – User configuration

  • keep_whitespace (bool) – Whether to keep whitespace in the dataframes

read(filepath: Union[str, TextIO], **kwargs) Tuple[Dict[str, DataFrame], Dict[str, Any]][source]#

Reads in data from file

Parameters

filepath (Union[str, TextIO]) –

Returns

tuple of input_data as a dictionary of pandas DataFrames and dictionary of default values

Return type

Tuple[Dict[str, pd.DataFrame], Dict[str, Any]]

class otoole.read_strategies.ReadMemory(parameters: Dict[str, DataFrame], user_config: Dict[str, Dict])[source]#

Bases: ReadStrategy

Read a dict of OSeMOSYS parameters from memory

Parameters
  • parameters (Dict[str, pd.DataFrame]) – Dictionary of OSeMOSYS parameters

  • user_config (Dict[str, Dict]) – User configuration

read(filepath: Union[str, TextIO] = None, **kwargs) Tuple[Dict[str, DataFrame], Dict[str, Any]][source]#

Reads in data from file

Parameters

filepath (Union[str, TextIO]) –

Returns

tuple of input_data as a dictionary of pandas DataFrames and dictionary of default values

Return type

Tuple[Dict[str, pd.DataFrame], Dict[str, Any]]

otoole.utils module#

class otoole.utils.UniqueKeyLoader(stream)[source]#

Bases: SafeLoader

YALM Loader to find duplicate keys

This loader will treat lowercase and uppercase keys as the same. Meaning, the keys “SampleKey” and “SAMPLEKEY” are considered the same.

Raises
  • ValueError – If a key is defined more than once.

  • Adapted from:

  • https://stackoverflow.com/a/63215043/14961492

construct_mapping(node, deep=False)[source]#
otoole.utils.create_name_mappings(config: Dict[str, Dict[str, Union[str, List]]], map_full_to_short: bool = True) Dict[source]#

Creates name mapping between full name and short name.

Parameters
  • config (Dict[str, Dict[str, Union[str, List]]]) – Parsed user configuration file

  • map_full_to_short (bool) – Map full name to short name if true, else map short name to full name

Returns

Mapping of full name to shortened name

Return type

csv_to_excel Dict[str, str]

otoole.utils.extract_config(schema: Dict, default_values: Dict) Dict[str, Dict[str, Union[str, List[str]]]][source]#
otoole.utils.format_config_for_validation(config_in: Dict) List[source]#

Formats config for validation function.

Flattens dictionary to a list

Parameters

config_in (Dict) – Read in user config yaml file

Returns

config_out

Return type

List

Example

>>> config_in
>>> AccumulatedAnnualDemand:
      indices: [REGION,FUEL,YEAR]
      type: param
      dtype: float
      default: 0
>>> config_out
>>> [{
    name: AccumulatedAnnualDemand
    indices: [REGION,FUEL,YEAR]
    type: param
    dtype: float
    default: 0
    }, ... ]
otoole.utils.get_all_sets(config: Dict) List[source]#

Extracts user defined sets

otoole.utils.get_packaged_resource(input_data: Dict[str, DataFrame], param: str) List[Dict[str, Any]][source]#

Gets input parameter data and formats it as a dictionary

Parameters
  • input_data (Dict[str, pd.DataFrame]) – Internal datastore for otoole input data

  • param (str) – Name of OSeMOSYS parameter

Returns

List of all rows in the df, where each dictionary is the column name, followed by the value in that row

Return type

List[Dict[str,any]]

Example

>>> get_packaged_resource(input_data, "InputActivityRatio")
>>> [{'REGION': 'SIMPLICITY',
    'TECHNOLOGY': 'RIVWATAGR',
    'FUEL': 'WATIN',
    'MODE_OF_OPERATION': 1,
    'YEAR': 2020,
    'VALUE': 1.0}]
otoole.utils.read_deprecated_datapackage(datapackage: str) str[source]#

Checks filepath for CSVs if a datapackage is provided

Parameters

datapackage (str) – Location of input datapackge

Returns

input_csvs – Location of input csv files

Return type

str

Raises

OtooleDeprecationError – If no ‘data/’ directory is found

otoole.utils.read_packaged_file(filename: str, module_name: str = None)[source]#
otoole.utils.validate_config(config: Dict) None[source]#

Validates user input data

Parameters

config (Dict) – Read in user config yaml file

Raises

ValidationError – If the user inputs are not valid

otoole.validate module#

Ensures that technology and fuel names match the convention

For example, to validate the following list of names, you would use the config shown below:

theseUNIQUE_ENTRY1
are__UNIQUE_ENTRY2
all__UNIQUE_ENTRY1
validUNIQUE_ENTRY2
entryUNIQUE_ENTRY1
in__UNIQUE_ENTRY2
a____UNIQUE_ENTRY1
list_UNIQUE_ENTRY2

Create a yaml validation config with the following format:

codes:
  some_valid_codes:
    UNIQUE_ENTRY1: Description of unique entry 1
    UNIQUE_ENTRY2: Description of unique entry 2
schema:
  schema_name:
  - name: first_entry_in_schema
    valid: ['these', 'are__', 'all__', 'valid', 'entry', 'in__', 'a____', 'list_']
    position: (1, 5) # a tuple representing the start and end position
  - name: second_entry_in_schema
    valid: some_valid_codes  # references an entry in the codes section of the
    config
    position: (6, 19) # a tuple representing the start and end position
otoole.validate.check_for_duplicates(codes: Sequence) bool[source]#
otoole.validate.compose_expression(schema: List) str[source]#

Generates a regular expression from a schema

Return type

str

otoole.validate.compose_multi_expression(resource: List) str[source]#

Concatenates multiple expressions using an OR operator

Use to validate elements using an OR operation e.g. the elements must match this expression OR the expression

otoole.validate.create_schema(config: Dict[str, Dict] = None) Dict[source]#

Populate the dict of schema with codes from the validation config

Parameters

config (dict, default=None) – A configuration dictionary containing codes and schema keys

otoole.validate.identify_orphaned_fuels_techs(input_data: Dict[str, DataFrame]) Dict[str, str][source]#

Returns a list of fuels and technologies which are unconnected

Return type

dict

otoole.validate.main(input_data: Dict[str, DataFrame], config=None)[source]#
otoole.validate.read_validation_config()[source]#
otoole.validate.validate(expression: str, name: str) bool[source]#

Determine if name matches the expression

Parameters
  • expression (str) –

  • name (str) –

Return type

bool

otoole.validate.validate_resource(input_data: Dict[str, DataFrame], resource: str, schemas: List[Dict]) None[source]#

Validates a single resource against the validation config.

Parameters
  • input_data (dict[str,pd.DataFrame]) – otoole internal datastore

  • resource (str) –

  • schemas (List[Dict]) – The schema from which to create a validation expression

otoole.write_strategies module#

class otoole.write_strategies.WriteCsv(user_config: Dict, filepath: Optional[str] = None, default_values: Optional[Dict] = None, write_defaults: bool = False, input_data: Optional[Dict[str, DataFrame]] = None)[source]#

Bases: WriteStrategy

Write parameters to comma-separated value files

Parameters
  • filepath (str, default=None) – The path to write a folder of csv files

  • default_values (dict, default=None) –

  • user_config (dict, default=None) –

class otoole.write_strategies.WriteDatafile(user_config: Dict, filepath: Optional[str] = None, default_values: Optional[Dict] = None, write_defaults: bool = False, input_data: Optional[Dict[str, DataFrame]] = None)[source]#

Bases: WriteStrategy

class otoole.write_strategies.WriteExcel(user_config: Dict, filepath: Optional[str] = None, default_values: Optional[Dict] = None, write_defaults: bool = False, input_data: Optional[Dict[str, DataFrame]] = None)[source]#

Bases: WriteStrategy

Module contents#

otoole.read(config: str, from_format: str, from_path: str, keep_whitespace: bool = False) Tuple[Dict[str, DataFrame], Dict[str, float]][source]#

Read OSeMOSYS data from datafile, csv or Excel formats

Parameters
  • config (str) – Path to config file

  • from_format (str) – Available options are ‘datafile’, ‘csv’, ‘excel’ and ‘datapackage’ [deprecated]

  • from_path (str) – Path to source file (if datafile or excel) or folder (csv)

  • keep_whitespace (bool, default: False) – Keep whitespace in source files

Returns

Dictionary of parameter and set data and dictionary of default values

Return type

Tuple[dict[str, pd.DataFrame], dict[str, float]]

otoole.read_results(config: str, from_format: str, from_path: str, input_format: str, input_path: str, glpk_model: str = None) Tuple[Dict[str, DataFrame], Dict[str, float]][source]#

Read OSeMOSYS results from CBC, GLPK, Gurobi, or CPLEX results files

Parameters
  • config (str) – Path to config file

  • from_format (str) – Available options are ‘cbc’, ‘gurobi’, ‘cplex’, and ‘glpk’

  • from_path (str) – Path to source file (if datafile or excel) or folder (csv)

  • input_format (str) – Format of input data. Available options are ‘datafile’, ‘csv’ and ‘excel’

  • input_path (str) – Path to input data

  • glpk_model (str) – Path to *.glp model file

Returns

Dictionary of parameter and set data and dictionary of default values

Return type

Tuple[dict[str, pd.DataFrame], dict[str, float]]

otoole.write(config: str, to_format: str, to_path: str, inputs, default_values: Optional[Dict[str, float]] = None) bool[source]#

Write OSeMOSYS data to datafile, csv or Excel formats

Parameters
  • config (str) – Path to config file

  • to_format (str) – Available options are ‘datafile’, ‘csv’, ‘excel’ and ‘datapackage’ [deprecated],

  • to_path (str) – Path to destination file (if datafile or excel) or folder (csv))

  • inputs (dict[str, pd.DataFrame]) – Dictionary of pandas data frames to write

  • default_values (dict[str, float], default: None) – Dictionary of default values to write to datafile