otoole package#
Subpackages#
- otoole.preprocess package
- Submodules
- otoole.preprocess.longify_data module
- otoole.preprocess.setup module
- otoole.preprocess.validate_config module
UserDefinedParameter
UserDefinedResult
UserDefinedSet
UserDefinedValue
UserDefinedValue.calculated
UserDefinedValue.check_name_for_numbers()
UserDefinedValue.check_name_for_spaces()
UserDefinedValue.check_name_for_special_chars()
UserDefinedValue.check_name_length()
UserDefinedValue.check_param_type()
UserDefinedValue.default
UserDefinedValue.defined_sets
UserDefinedValue.dtype
UserDefinedValue.indices
UserDefinedValue.model_config
UserDefinedValue.model_fields
UserDefinedValue.name
UserDefinedValue.short_name
UserDefinedValue.type
- Module contents
- otoole.results package
- Submodules
- otoole.results.result_package module
ResultsPackage
ResultsPackage.accumulated_new_capacity()
ResultsPackage.annual_emissions()
ResultsPackage.annual_fixed_operating_cost()
ResultsPackage.annual_technology_emission_by_mode()
ResultsPackage.annual_technology_emissions()
ResultsPackage.annual_variable_operating_cost()
ResultsPackage.capital_investment()
ResultsPackage.data
ResultsPackage.demand()
ResultsPackage.discounted_tech_emis_pen()
ResultsPackage.get_unique_values_from_index()
ResultsPackage.production_by_technology()
ResultsPackage.production_by_technology_annual()
ResultsPackage.rate_of_product_technology()
ResultsPackage.rate_of_production_tech_mode()
ResultsPackage.rate_of_use_by_technology()
ResultsPackage.rate_of_use_by_technology_by_mode()
ResultsPackage.result_cache
ResultsPackage.result_mapper
ResultsPackage.total_annual_tech_activity_mode()
ResultsPackage.total_capacity_annual()
ResultsPackage.total_discounted_cost()
ResultsPackage.total_tech_model_period_activity()
ResultsPackage.total_technology_annual_activity()
ResultsPackage.use_by_technology()
capital_recovery_factor()
discount_factor()
discount_factor_storage()
pv_annuity()
- otoole.results.results module
- Module contents
- otoole.visualise package
Submodules#
otoole.cli module#
Provides a command line interface to the otoole
package
The key functions are convert, cplex and viz.
The convert
command allows convertion of multiple different OSeMOSYS input formats
including from/to csv, an AMPL format datafile, a folder of
CSVs, an Excel workbook with one tab per parameter, an SQLite database
The cplex
command provides access to scripts which transform and process a CPLEX
solution file into a format which is more readily processed - either to CBC or CSV
format.
The validate
command checks the technology and fuel names
against a standard or user defined configuration file.
The viz command allows you to produce a Reference Energy System diagram
Example
Ask for help on the command line:
>>> $ otoole --help
usage: otoole [-h] [--verbose] [--version] {convert,cplex,validate,viz} ...
otoole: Python toolkit of OSeMOSYS users
positional arguments:
{convert,cplex,validate,viz}
convert Convert from one input format to another
cplex Process a CPLEX solution file
validate Validate an OSeMOSYS model
viz Visualise the model
optional arguments:
-h, --help show this help message and exit
--verbose, -v Enable debug mode
--version, -V The version of otoole
otoole.convert module#
This module implements the public API of the otoole package
Use the otoole convert
function to convert between different file formats.
Import the convert function from the otoole package:
>>> from otoole import convert
>>> convert('config.yaml', 'excel', 'datafile', 'input.xlsx', 'output.dat')
- otoole.convert.convert(config, from_format, to_format, from_path, to_path, write_defaults=False, keep_whitespace=False) bool [source]#
Convert OSeMOSYS data from/to datafile, csv and Excel formats
- Parameters
config (str) – Path to config file
from_format (str) – Available options are ‘datafile’, ‘datapackage’, ‘csv’ and ‘excel’
to_format (str) – Available options are ‘datafile’, ‘datapackage’, ‘csv’ and ‘excel’
from_path (str) – Path to destination file (if datafile or excel) or folder (csv or datapackage)
write_defaults (bool, default: False) – Write default values to CSVs
keep_whitespace (bool, default: False) – Keep whitespace in CSVs
- Returns
True if conversion was successful, False otherwise
- Return type
- otoole.convert.convert_results(config: str, from_format: str, to_format: str, from_path: str, to_path: str, input_format: str, input_path: str, write_defaults: bool = False, glpk_model: str = None) bool [source]#
Post-process results from a CBC, CPLEX, Gurobi, or GLPK solution file into CSV format
- Parameters
config (str) – Path to config file
from_format (str) – Available options are ‘cbc’, ‘cplex’ and ‘gurobi’
to_format (str) – Available options are ‘csv’
from_path (str) – Path to cbc, cplex or gurobi solution file
to_path (str) – Path to destination folder
input_format (str) – Format of input data. Available options are ‘datafile’, ‘csv’ and ‘excel’
input_path (str) – Path to input data
write_defaults (bool) – Write default values to CSVs
glpk_model (str) – Path to
*.glp
model file
- Returns
True if conversion was successful, False otherwise
- Return type
- otoole.convert.read(config: str, from_format: str, from_path: str, keep_whitespace: bool = False) Tuple[Dict[str, DataFrame], Dict[str, float]] [source]#
Read OSeMOSYS data from datafile, csv or Excel formats
- Parameters
- Returns
Dictionary of parameter and set data and dictionary of default values
- Return type
- otoole.convert.read_results(config: str, from_format: str, from_path: str, input_format: str, input_path: str, glpk_model: str = None) Tuple[Dict[str, DataFrame], Dict[str, float]] [source]#
Read OSeMOSYS results from CBC, GLPK, Gurobi, or CPLEX results files
- Parameters
config (str) – Path to config file
from_format (str) – Available options are ‘cbc’, ‘gurobi’, ‘cplex’, and ‘glpk’
from_path (str) – Path to source file (if datafile or excel) or folder (csv)
input_format (str) – Format of input data. Available options are ‘datafile’, ‘csv’ and ‘excel’
input_path (str) – Path to input data
glpk_model (str) – Path to
*.glp
model file
- Returns
Dictionary of parameter and set data and dictionary of default values
- Return type
- otoole.convert.write(config: str, to_format: str, to_path: str, inputs, default_values: Optional[Dict[str, float]] = None) bool [source]#
Write OSeMOSYS data to datafile, csv or Excel formats
- Parameters
config (str) – Path to config file
to_format (str) – Available options are ‘datafile’, ‘csv’, ‘excel’ and ‘datapackage’ [deprecated],
to_path (str) – Path to destination file (if datafile or excel) or folder (csv))
inputs (dict[str, pd.DataFrame]) – Dictionary of pandas data frames to write
default_values (dict[str, float], default: None) – Dictionary of default values to write to datafile
otoole.exceptions module#
- exception otoole.exceptions.OtooleConfigFileError(message: str)[source]#
Bases:
OtooleException
Config file validation error
- Parameters
message (str) – Message to display to users
- exception otoole.exceptions.OtooleDeprecationError(resource, message)[source]#
Bases:
OtooleException
New version of otoole does drops this feature support
- exception otoole.exceptions.OtooleError(resource, message)[source]#
Bases:
OtooleException
General purpose error
- exception otoole.exceptions.OtooleExcelNameLengthError(name: str, message: str = 'Parameter name must be less than 31 characters when writing to Excel')[source]#
Bases:
OtooleException
Invalid tab name for writing to Excel.
- exception otoole.exceptions.OtooleException[source]#
Bases:
Exception
Base class for all otoole exceptions.
- exception otoole.exceptions.OtooleIndexError(resource, config_indices, data_indices)[source]#
Bases:
OtooleException
Index data not consistent between data and config file
- exception otoole.exceptions.OtooleNameMismatchError(name: Union[List, str])[source]#
Bases:
OtooleException
Names not consistent between read in data and config file
- exception otoole.exceptions.OtooleRelationError(resource, foreign_resource, message)[source]#
Bases:
OtooleException
Relations between input data is not correct
- exception otoole.exceptions.OtooleSetupError(resource, message='Data already exists. Delete file/directory or pass the --overwrite flag')[source]#
Bases:
OtooleException
Setup data already exists
- exception otoole.exceptions.OtooleValidationError(resource, message)[source]#
Bases:
OtooleException
Input data is invalid
otoole.input module#
The input
module allows you to access the conversion routines programmatically
To use the routines, you need to instantiate a ReadStrategy
and a WriteStrategy
relevant for the format of the input and output data. You then pass these to a
Context
.
Example
Convert an in-memory dictionary of pandas DataFrames containing OSeMOSYS parameters to an Excel spreadsheet:
>>> from otoole import ReadMemory
>>> from otoole import WriteExcel
>>> from otoole import Context
>>> reader = ReadMemory(parameters)
>>> writer = WriteExcel()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('.', 'osemosys_to_excel.xlsx')
Convert a GNUMathProg datafile to a folder of CSV files:
>>> from otoole import ReadDataFile
>>> from otoole import WriteCsv
>>> from otoole import Context
>>> reader = ReadDataFile()
>>> writer = WriteCsv()
>>> converter = Context(read_strategy=reader, write_strategy=writer)
>>> converter.convert('my_datafile.txt', 'folder_of_csv_files')
- class otoole.input.Context(read_strategy: ReadStrategy, write_strategy: WriteStrategy)[source]#
Bases:
object
The Context defines the interface of interest to clients.
- convert(input_filepath: str, output_filepath: str, **kwargs: Dict)[source]#
Converts from file
input_filepath
to fileoutput_filepath
- property read_strategy: ReadStrategy#
The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.
- property write_strategy: WriteStrategy#
The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.
- class otoole.input.ReadStrategy(user_config: Dict[str, Dict])[source]#
Bases:
Strategy
The Strategy interface declares operations common to all reading formats.
The Context uses this interface to call the algorithm defined by Concrete Strategies.
- class otoole.input.Strategy(user_config: Dict[str, Dict])[source]#
Bases:
ABC
- Parameters
user_config (dict, default=None) – A user configuration for the input parameters and sets
- class otoole.input.WriteStrategy(user_config: Dict, filepath: Optional[str] = None, default_values: Optional[Dict] = None, write_defaults: bool = False, input_data: Optional[Dict[str, DataFrame]] = None)[source]#
Bases:
Strategy
The WriteStrategy interface declares operations common to all writing formats
The Context uses this interface to call the algorithm defined by Concrete Strategies.
- Parameters
otoole.read_strategies module#
- class otoole.read_strategies.ReadCsv(user_config: Dict[str, Dict], keep_whitespace: bool = False)[source]#
Bases:
_ReadTabular
Read in a folder of CSV files to a dict of Pandas DataFrames
- Parameters
- class otoole.read_strategies.ReadDatafile(user_config: Dict[str, Dict])[source]#
Bases:
ReadStrategy
Read in a datafile to a dict of Pandas DataFrames
- Parameters
- class otoole.read_strategies.ReadExcel(user_config: Dict[str, Dict], keep_whitespace: bool = False)[source]#
Bases:
_ReadTabular
Read in an Excel spreadsheet in wide format to a dict of Pandas DataFrames
- Parameters
- class otoole.read_strategies.ReadMemory(parameters: Dict[str, DataFrame], user_config: Dict[str, Dict])[source]#
Bases:
ReadStrategy
Read a dict of OSeMOSYS parameters from memory
- Parameters
otoole.utils module#
- class otoole.utils.UniqueKeyLoader(stream)[source]#
Bases:
SafeLoader
YALM Loader to find duplicate keys
This loader will treat lowercase and uppercase keys as the same. Meaning, the keys “SampleKey” and “SAMPLEKEY” are considered the same.
- Raises
ValueError – If a key is defined more than once.
Adapted from: –
https://stackoverflow.com/a/63215043/14961492 –
- otoole.utils.create_name_mappings(config: Dict[str, Dict[str, Union[str, List]]], map_full_to_short: bool = True) Dict [source]#
Creates name mapping between full name and short name.
- otoole.utils.extract_config(schema: Dict, default_values: Dict) Dict[str, Dict[str, Union[str, List[str]]]] [source]#
- otoole.utils.format_config_for_validation(config_in: Dict) List [source]#
Formats config for validation function.
Flattens dictionary to a list
- Parameters
config_in (Dict) – Read in user config yaml file
- Returns
config_out
- Return type
List
Example
>>> config_in >>> AccumulatedAnnualDemand: indices: [REGION,FUEL,YEAR] type: param dtype: float default: 0
>>> config_out >>> [{ name: AccumulatedAnnualDemand indices: [REGION,FUEL,YEAR] type: param dtype: float default: 0 }, ... ]
- otoole.utils.get_packaged_resource(input_data: Dict[str, DataFrame], param: str) List[Dict[str, Any]] [source]#
Gets input parameter data and formats it as a dictionary
- Parameters
- Returns
List of all rows in the df, where each dictionary is the column name, followed by the value in that row
- Return type
List[Dict[str,any]]
Example
>>> get_packaged_resource(input_data, "InputActivityRatio") >>> [{'REGION': 'SIMPLICITY', 'TECHNOLOGY': 'RIVWATAGR', 'FUEL': 'WATIN', 'MODE_OF_OPERATION': 1, 'YEAR': 2020, 'VALUE': 1.0}]
- otoole.utils.read_deprecated_datapackage(datapackage: str) str [source]#
Checks filepath for CSVs if a datapackage is provided
- Parameters
datapackage (str) – Location of input datapackge
- Returns
input_csvs – Location of input csv files
- Return type
- Raises
OtooleDeprecationError – If no ‘data/’ directory is found
otoole.validate module#
Ensures that technology and fuel names match the convention
For example, to validate the following list of names, you would use the config shown below:
theseUNIQUE_ENTRY1
are__UNIQUE_ENTRY2
all__UNIQUE_ENTRY1
validUNIQUE_ENTRY2
entryUNIQUE_ENTRY1
in__UNIQUE_ENTRY2
a____UNIQUE_ENTRY1
list_UNIQUE_ENTRY2
Create a yaml validation config with the following format:
codes:
some_valid_codes:
UNIQUE_ENTRY1: Description of unique entry 1
UNIQUE_ENTRY2: Description of unique entry 2
schema:
schema_name:
- name: first_entry_in_schema
valid: ['these', 'are__', 'all__', 'valid', 'entry', 'in__', 'a____', 'list_']
position: (1, 5) # a tuple representing the start and end position
- name: second_entry_in_schema
valid: some_valid_codes # references an entry in the codes section of the
config
position: (6, 19) # a tuple representing the start and end position
- otoole.validate.compose_expression(schema: List) str [source]#
Generates a regular expression from a schema
- Return type
- otoole.validate.compose_multi_expression(resource: List) str [source]#
Concatenates multiple expressions using an OR operator
Use to validate elements using an OR operation e.g. the elements must match this expression OR the expression
- otoole.validate.create_schema(config: Dict[str, Dict] = None) Dict [source]#
Populate the dict of schema with codes from the validation config
- Parameters
config (dict, default=None) – A configuration dictionary containing
codes
andschema
keys
- otoole.validate.identify_orphaned_fuels_techs(input_data: Dict[str, DataFrame]) Dict[str, str] [source]#
Returns a list of fuels and technologies which are unconnected
- Return type
- otoole.validate.validate(expression: str, name: str) bool [source]#
Determine if
name
matches theexpression
otoole.write_strategies module#
- class otoole.write_strategies.WriteCsv(user_config: Dict, filepath: Optional[str] = None, default_values: Optional[Dict] = None, write_defaults: bool = False, input_data: Optional[Dict[str, DataFrame]] = None)[source]#
Bases:
WriteStrategy
Write parameters to comma-separated value files
Module contents#
- otoole.read(config: str, from_format: str, from_path: str, keep_whitespace: bool = False) Tuple[Dict[str, DataFrame], Dict[str, float]] [source]#
Read OSeMOSYS data from datafile, csv or Excel formats
- Parameters
- Returns
Dictionary of parameter and set data and dictionary of default values
- Return type
- otoole.read_results(config: str, from_format: str, from_path: str, input_format: str, input_path: str, glpk_model: str = None) Tuple[Dict[str, DataFrame], Dict[str, float]] [source]#
Read OSeMOSYS results from CBC, GLPK, Gurobi, or CPLEX results files
- Parameters
config (str) – Path to config file
from_format (str) – Available options are ‘cbc’, ‘gurobi’, ‘cplex’, and ‘glpk’
from_path (str) – Path to source file (if datafile or excel) or folder (csv)
input_format (str) – Format of input data. Available options are ‘datafile’, ‘csv’ and ‘excel’
input_path (str) – Path to input data
glpk_model (str) – Path to
*.glp
model file
- Returns
Dictionary of parameter and set data and dictionary of default values
- Return type
- otoole.write(config: str, to_format: str, to_path: str, inputs, default_values: Optional[Dict[str, float]] = None) bool [source]#
Write OSeMOSYS data to datafile, csv or Excel formats
- Parameters
config (str) – Path to config file
to_format (str) – Available options are ‘datafile’, ‘csv’, ‘excel’ and ‘datapackage’ [deprecated],
to_path (str) – Path to destination file (if datafile or excel) or folder (csv))
inputs (dict[str, pd.DataFrame]) – Dictionary of pandas data frames to write
default_values (dict[str, float], default: None) – Dictionary of default values to write to datafile