otoole package

Submodules

otoole.cli module

Provides a command line interface to the otoole package

The key functions are convert, cplex and viz.

The convert command allows convertion of multiple different OSeMOSYS input formats including from/to csv, an AMPL format datafile, a Tabular Data Package, a folder of CSVs, an Excel workbook with one tab per parameter, an SQLite database

The cplex command provides access to scripts which transform and process a CPLEX solution file into a format which is more readily processed - either to CBC or CSV format.

The validate command checks the technology and fuel names of a tabular data package against a standard or user defined configuration file.

The viz command allows you to produce a Reference Energy System diagram from a Tabular Data Package.

Example

Ask for help on the command line:

>>> $ otoole --help
usage: otoole [-h] [--verbose] [--version] {convert,cplex,validate,viz} ...

otoole: Python toolkit of OSeMOSYS users

positional arguments:
{convert,cplex,validate,viz}
    convert             Convert from one input format to another
    cplex               Process a CPLEX solution file
    validate            Validate an OSeMOSYS model
    viz                 Visualise the model

optional arguments:
-h, --help            show this help message and exit
--verbose, -v         Enable debug mode
--version, -V         The version of otoole
otoole.cli.conversion_matrix(args)[source]

Convert from one format to another

Implemented conversion functions:

from    o     ex cs dp df
-----------------------
excel       -- yy -- --
csv         nn -- yy nn
datapackage yy -- -- yy
datafile    nn -- yy --
otoole.cli.cplex2cbc(args)[source]
otoole.cli.datapackage2res(args)[source]
otoole.cli.get_parser()[source]
otoole.cli.main()[source]
otoole.cli.result_matrix(args)[source]

Post-process results from CBC solution file into CSV format

otoole.cli.validate_model(args)[source]

otoole.exceptions module

exception otoole.exceptions.OtooleException[source]

Bases: Exception

Base class for all otoole exceptions.

exception otoole.exceptions.OtooleRelationError(resource, foreign_resource, message)[source]

Bases: otoole.exceptions.OtooleException

Relations between input data is not correct

Parameters:
  • resource (str) – Name of the resource which is invalid
  • foreign_resource (str) – Name of the resource which is invalid
  • message (str) – Error message
exception otoole.exceptions.OtooleValidationError(resource, message)[source]

Bases: otoole.exceptions.OtooleException

Input data is invalid

Parameters:
  • resource (str) – Name of the resource which is invalid
  • message (str) – Error message

otoole.input module

The input module allows you to access the conversion routines programmatically

To use the routines, you need to instanciate a ReadStrategy and a WriteStrategy relevant for the format of the input and output data. You then pass these to a Context.

Example

Convert an in-memory dictionary of pandas DataFrames containing OSeMOSYS parameters to an Excel spreadsheet:

>>> from otoole import ReadMemory
>>> from otoole import WriteExcel
>>> from otoole import Context
>>> reader = ReadMemory(parameters)
>>> writer = WriteExcel()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('.', 'osemosys_to_excel.xlsx')

Convert a GNUMathProg datafile to a folder of CSV files:

>>> from otoole import ReadDataFile
>>> from otoole import WriteCsv
>>> from otoole import Context
>>> reader = ReadDataFile()
>>> writer = WriteCsv()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('my_datafile.txt', 'folder_of_csv_files')

Convert a GNUMathProg datafile to a folder of Tabular DataPackage:

>>> from otoole import ReadDataFile
>>> from otoole import WriteDatapackage
>>> from otoole import Context
>>> reader = ReadDataFile()
>>> writer = WriteDatapackage()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('my_datafile.txt', 'my_datapackage')
class otoole.input.Context(read_strategy: otoole.input.ReadStrategy, write_strategy: otoole.input.WriteStrategy)[source]

Bases: object

The Context defines the interface of interest to clients.

convert(input_filepath: str, output_filepath: str, **kwargs)[source]

Converts from file input_filepath to file output_filepath

Parameters:
  • input_filepath (str) –
  • output_filepath (str) –
read_strategy

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

write_strategy

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

class otoole.input.ReadStrategy(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.Strategy

The Strategy interface declares operations common to all reading formats.

The Context uses this interface to call the algorithm defined by Concrete Strategies.

read(filepath: Union[str, TextIO], **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.input.Strategy(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: abc.ABC

Parameters:
  • input_config (dict, default=None) – A user configuration for the input parameters and sets
  • results_config (dict, default=None) – A user configuration for the results parameters
input_config
results_config
class otoole.input.WriteStrategy(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.Strategy

The WriteStrategy interface declares operations common to all writing formats

The Context uses this interface to call the algorithm defined by Concrete Strategies.

Parameters:
  • filepath (str, default=None) –
  • default_values (dict, default=None) –
  • user_config (dict, default=None) –
write(inputs: Dict[KT, VT], filepath: str, default_values: Dict[KT, VT])[source]

Perform the conversion from dict of dataframes to destination format

otoole.read_strategies module

class otoole.read_strategies.ReadCsv(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.read_strategies._ReadTabular

Read in a folder of CSV files

read(filepath, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.read_strategies.ReadDatafile(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.ReadStrategy

extract_param(config, name, datafile_parser, dict_of_dataframes) → pandas.core.frame.DataFrame[source]
extract_set(datafile_parser, name, config, dict_of_dataframes) → pandas.core.frame.DataFrame[source]
read(filepath, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
read_in_datafile(path_to_datafile: str, config: Dict[KT, VT]) → amply.amply.Amply[source]

Read in a datafile using the Amply parsing class

Parameters:
  • path_to_datafile (str) –
  • config (Dict) –
class otoole.read_strategies.ReadDatapackage(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.ReadStrategy

read(filepath, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.read_strategies.ReadExcel(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.read_strategies._ReadTabular

Read in an Excel spreadsheet in wide format to a dict of Pandas DataFrames

read(filepath: Union[str, TextIO], **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.read_strategies.ReadMemory(parameters: Dict[str, pandas.core.frame.DataFrame], user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.ReadStrategy

Read a dict of OSeMOSYS parameters from memory

read(filepath: Union[str, TextIO] = None, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]

otoole.utils module

otoole.utils.extract_config(schema: Dict[KT, VT], default_values: Dict[KT, VT]) → Dict[str, Dict[str, Union[str, List[str]]]][source]
otoole.utils.read_datapackage(filepath: str, sql: bool = False)[source]

Open an OSeMOSYS datapackage

Parameters:
  • filepath (str) –
  • sql (bool, default=False) –
otoole.utils.read_datapackage_schema_into_config(filepath: str, default_values: Dict[KT, VT]) → Dict[str, Dict[str, Union[str, List[T]]]][source]
otoole.utils.read_packaged_file(filename: str, module_name: str = None)[source]

otoole.validate module

Ensures that technology and fuel names match the convention

For example, to validate the following list of names, you would use the config shown below:

theseUNIQUE_ENTRY1
are__UNIQUE_ENTRY2
all__UNIQUE_ENTRY1
validUNIQUE_ENTRY2
entryUNIQUE_ENTRY1
in__UNIQUE_ENTRY2
a____UNIQUE_ENTRY1
list_UNIQUE_ENTRY2

Create a yaml validation config with the following format:

codes:
  some_valid_codes:
    UNIQUE_ENTRY1: Description of unique entry 1
    UNIQUE_ENTRY2: Description of unique entry 2
schema:
  schema_name:
  - name: first_entry_in_schema
    valid: ['these', 'are__', 'all__', 'valid', 'entry', 'in__', 'a____', 'list_']
    position: (1, 5) # a tuple representing the start and end position
  - name: second_entry_in_schema
    valid: some_valid_codes  # references an entry in the codes section of the
    config
    position: (6, 19) # a tuple representing the start and end position
otoole.validate.check_for_duplicates(codes: Sequence[T_co]) → bool[source]
otoole.validate.compose_expression(schema: List[T]) → str[source]

Generates a regular expression from a schema

Returns:
Return type:str
otoole.validate.compose_multi_expression(resource: List[T]) → str[source]

Concatenates multiple expressions using an OR operator

Use to validate elements using an OR operation e.g. the elements must match this expression OR the expression

otoole.validate.create_schema(config: Dict[str, Dict[KT, VT]] = None) → Dict[KT, VT][source]

Populate the dict of schema with codes from the validation config

Parameters:config (dict, default=None) – A configuration dictionary containing codes and schema keys
otoole.validate.identify_orphaned_fuels_techs(package) → Dict[str, str][source]

Returns a list of fuels and technologies which are unconnected

Returns:
Return type:dict
otoole.validate.main(file_format: str, filepath: str, config=None)[source]
otoole.validate.read_validation_config()[source]
otoole.validate.validate(expression: str, name: str) → bool[source]

Determine if name matches the expression

Parameters:
  • expression (str) –
  • name (str) –
Returns:

Return type:

bool

otoole.validate.validate_resource(package, resource: str, schemas: List[Dict[KT, VT]])[source]
Parameters:
  • package
  • resource (str) –
  • schemas (List[Dict]) – The schema from which to create a validation expression

otoole.write_strategies module

class otoole.write_strategies.WriteCsv(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.WriteStrategy

Write parameters to comma-separated value files

Parameters:
  • filepath (str, default=None) – The path to write a folder of csv files
  • default_values (dict, default=None) –
  • user_config (dict, default=None) –
class otoole.write_strategies.WriteDatafile(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.WriteStrategy

class otoole.write_strategies.WriteDatapackage(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.write_strategies.WriteCsv

class otoole.write_strategies.WriteExcel(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.WriteStrategy

Module contents

class otoole.Context(read_strategy: otoole.input.ReadStrategy, write_strategy: otoole.input.WriteStrategy)[source]

Bases: object

The Context defines the interface of interest to clients.

convert(input_filepath: str, output_filepath: str, **kwargs)[source]

Converts from file input_filepath to file output_filepath

Parameters:
  • input_filepath (str) –
  • output_filepath (str) –
read_strategy

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

write_strategy

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

class otoole.ReadCbc(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.results.results.ReadResultsCBC

Read a CBC solution file into memory

Parameters:
  • user_config
  • results_config
class otoole.ReadCsv(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.read_strategies._ReadTabular

Read in a folder of CSV files

read(filepath, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.ReadCplex(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.results.results.ReadResults

convert_df(data: List[List[str]], variable: str, start_year: int, end_year: int) → pandas.core.frame.DataFrame[source]

Read the cplex lines into a pandas DataFrame

extract_rows(sol_file: TextIO, start_year: int, end_year: int) → Dict[str, List[List[str]]][source]
extract_variable_dimensions_values(data: List[T]) → Tuple[str, Tuple, List[T]][source]

Extracts useful information from a line of a results file

get_results_from_file(filepath: Union[str, TextIO], input_data) → Dict[str, pandas.core.frame.DataFrame][source]
class otoole.ReadDatafile(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.ReadStrategy

extract_param(config, name, datafile_parser, dict_of_dataframes) → pandas.core.frame.DataFrame[source]
extract_set(datafile_parser, name, config, dict_of_dataframes) → pandas.core.frame.DataFrame[source]
read(filepath, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
read_in_datafile(path_to_datafile: str, config: Dict[KT, VT]) → amply.amply.Amply[source]

Read in a datafile using the Amply parsing class

Parameters:
  • path_to_datafile (str) –
  • config (Dict) –
class otoole.ReadDatapackage(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.ReadStrategy

read(filepath, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.ReadExcel(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.read_strategies._ReadTabular

Read in an Excel spreadsheet in wide format to a dict of Pandas DataFrames

read(filepath: Union[str, TextIO], **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.ReadGurobi(user_config: Optional[Dict[KT, VT]] = None, results_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.results.results.ReadResultsCBC

Read a Gurobi solution file into memory

class otoole.ReadMemory(parameters: Dict[str, pandas.core.frame.DataFrame], user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.ReadStrategy

Read a dict of OSeMOSYS parameters from memory

read(filepath: Union[str, TextIO] = None, **kwargs) → Tuple[Dict[str, pandas.core.frame.DataFrame], Dict[str, Any]][source]
class otoole.WriteCsv(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.WriteStrategy

Write parameters to comma-separated value files

Parameters:
  • filepath (str, default=None) – The path to write a folder of csv files
  • default_values (dict, default=None) –
  • user_config (dict, default=None) –
class otoole.WriteDatafile(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.WriteStrategy

class otoole.WriteDatapackage(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.write_strategies.WriteCsv

class otoole.WriteExcel(filepath: Optional[str] = None, default_values: Optional[Dict[KT, VT]] = None, user_config: Optional[Dict[KT, VT]] = None)[source]

Bases: otoole.input.WriteStrategy