otoole.preprocess package¶

Submodules¶

otoole.preprocess.create_datapackage module¶

Creates a datapackage from a collection of CSV files of OSeMOSYS input data

Uses Frictionless Data datapackage concept to build a JSON schema of the dataset
Enforces relations between sets and indices in parameter files

otoole.preprocess.create_datapackage.convert_datapackage_to_sqlite(path_to_datapackage, sqlite)[source]¶: Load and save table to SQLite

otoole.preprocess.create_datapackage.generate_package(path_to_package)[source]¶

Creates a datapackage in folder path_to_package

[{‘fields’: ‘REGION’, ‘reference’: {‘resource’: ‘REGION’, ‘fields’: ‘VALUE’}}]

otoole.preprocess.create_datapackage.main(wide_folder, narrow_folder)[source]¶

otoole.preprocess.create_datapackage.validate_contents(path_to_package)[source]¶

otoole.preprocess.datafile_to_datapackage module¶

otoole.preprocess.excel_to_osemosys module¶

Extract data from spreadsheets and write to an OSeMOSYS datafile

Reads and writes the following OSeMOSYS parameters and sets to and from files on disk.

Notes

Sets¶

These are the standard sets:

set YEAR;
set TECHNOLOGY;
set TIMESLICE;
set FUEL;
set EMISSION;
set MODE_OF_OPERATION;
set REGION;
set SEASON;
set DAYTYPE;
set DAILYTIMEBRACKET;
set FLEXIBLEDEMANDTYPE;
set STORAGE;

All sets are written in a CSV file in one column of values with header of VALUE. For example, the CSV file for set YEAR:

Sets are written into the OSeMOSYS data file using the following syntax:

set YEAR := 2014 2015 2016 2017 2018 2019 2020 ;

param In general, parameters can be written into CSV files in narrow or wide:

param formats. In narrow format, the CSV file should look as follows:::
	REGION,TIMESLICE,YEAR,VALUE SIMPLICITY,ID,2014,0.1667 SIMPLICITY,IN,2014,0.0833 SIMPLICITY,SD,2014,0.1667 SIMPLICITY,SN,2014,0.0833 SIMPLICITY,WD,2014,0.3333 SIMPLICITY,WN,2014,0.1667

In wide format, the final index is transposed:

REGION,TIMESLICE,2014,2015,...
SIMPLICITY,ID,0.1667,0.1667
SIMPLICITY,IN,0.0833,0.0833
SIMPLICITY,SD,0.1667,0.1667
SIMPLICITY,SN,0.0833,0.0833
SIMPLICITY,WD,0.3333,0.3333
SIMPLICITY,WN,0.1667,0.1667

Wide format is a bit nicer to use with spreadsheets, as it allows you to more easily plot graphs of values, but narrow format is a more flexible and easily manipulated data format.

Writing parameteters:

1-dimensional e.g. DiscountRate{r in REGION}

2-dimension e.g. YearSplit{l in TIMESLICE, y in YEAR}

n-dimensional e.g. DaysInDayType{ls in SEASON, ld in DAYTYPE, y in YEAR}

Global parameters:

param YearSplit{l in TIMESLICE, y in YEAR};
param DiscountRate{r in REGION};
param DaySplit{lh in DAILYTIMEBRACKET, y in YEAR};
param Conversionls{l in TIMESLICE, ls in SEASON};
param Conversionld{l in TIMESLICE, ld in DAYTYPE};
param Conversionlh{l in TIMESLICE, lh in DAILYTIMEBRACKET};
param DaysInDayType{ls in SEASON, ld in DAYTYPE, y in YEAR};
param TradeRoute{r in REGION, rr in REGION, f in FUEL, y in YEAR};
param DepreciationMethod{r in REGION};

Demand parameters:

param SpecifiedAnnualDemand{r in REGION, f in FUEL, y in YEAR};
param SpecifiedDemandProfile{r in REGION, f in FUEL, l in TIMESLICE, y in YEAR};
param AccumulatedAnnualDemand{r in REGION, f in FUEL, y in YEAR};

Performance parameters:

param CapacityToActivityUnit{r in REGION, t in TECHNOLOGY};
param TechWithCapacityNeededToMeetPeakTS{r in REGION, t in TECHNOLOGY};
param CapacityFactor{r in REGION, t in TECHNOLOGY, l in TIMESLICE, y in YEAR};
param AvailabilityFactor{r in REGION, t in TECHNOLOGY, y in YEAR};
param OperationalLife{r in REGION, t in TECHNOLOGY};
param ResidualCapacity{r in REGION, t in TECHNOLOGY, y in YEAR};
param InputActivityRatio{r in REGION, t in TECHNOLOGY, f in FUEL,
                         m in MODE_OF_OPERATION, y in YEAR};
param OutputActivityRatio{r in REGION, t in TECHNOLOGY, f in FUEL,
                          m in MODE_OF_OPERATION, y in YEAR};

Technology Costs parameters:

param CapitalCost{r in REGION, t in TECHNOLOGY, y in YEAR};
param VariableCost{r in REGION, t in TECHNOLOGY, m in MODE_OF_OPERATION, y in YEAR};
param FixedCost{r in REGION, t in TECHNOLOGY, y in YEAR};

Storage parameters:

param TechnologyToStorage{r in REGION, t in TECHNOLOGY, s in STORAGE,
                          m in MODE_OF_OPERATION};
param TechnologyFromStorage{r in REGION, t in TECHNOLOGY, s in STORAGE,
                          m in MODE_OF_OPERATION};
param StorageLevelStart{r in REGION, s in STORAGE};
param StorageMaxChargeRate{r in REGION, s in STORAGE};
param StorageMaxDischargeRate{r in REGION, s in STORAGE};
param MinStorageCharge{r in REGION, s in STORAGE, y in YEAR};
param OperationalLifeStorage{r in REGION, s in STORAGE};
param CapitalCostStorage{r in REGION, s in STORAGE, y in YEAR};
param ResidualStorageCapacity{r in REGION, s in STORAGE, y in YEAR};

Capacity Constraints parameters:

param CapacityOfOneTechnologyUnit{r in REGION, t in TECHNOLOGY, y in YEAR};
param TotalAnnualMaxCapacity{r in REGION, t in TECHNOLOGY, y in YEAR};
param TotalAnnualMinCapacity{r in REGION, t in TECHNOLOGY, y in YEAR};

Investment Constraints parameters:

param TotalAnnualMaxCapacityInvestment{r in REGION, t in TECHNOLOGY, y in YEAR};
param TotalAnnualMinCapacityInvestment{r in REGION, t in TECHNOLOGY, y in YEAR};

Activity Constraints parameters:

param TotalTechnologyAnnualActivityUpperLimit{r in REGION, t in TECHNOLOGY,
                                              y in YEAR};
param TotalTechnologyAnnualActivityLowerLimit{r in REGION, t in TECHNOLOGY,
                                              y in YEAR};
param TotalTechnologyModelPeriodActivityUpperLimit{r in REGION, t in TECHNOLOGY};
param TotalTechnologyModelPeriodActivityLowerLimit{r in REGION, t in TECHNOLOGY};

Reserve Margin parameters:

param ReserveMarginTagTechnology{r in REGION, t in TECHNOLOGY, y in YEAR};
param ReserveMarginTagFuel{r in REGION, f in FUEL, y in YEAR};
param ReserveMargin{r in REGION, y in YEAR};

RE Generation Target parameters:

param RETagTechnology{r in REGION, t in TECHNOLOGY, y in YEAR};
param RETagFuel{r in REGION, f in FUEL, y in YEAR};
param REMinProductionTarget{r in REGION, y in YEAR};

Emissions & Penalties parameters:

param EmissionActivityRatio{r in REGION, t in TECHNOLOGY, e in EMISSION,
                            m in MODE_OF_OPERATION, y in YEAR};
param EmissionsPenalty{r in REGION, e in EMISSION, y in YEAR};
param AnnualExogenousEmission{r in REGION, e in EMISSION, y in YEAR};
param AnnualEmissionLimit{r in REGION, e in EMISSION, y in YEAR};
param ModelPeriodExogenousEmission{r in REGION, e in EMISSION};
param ModelPeriodEmissionLimit{r in REGION, e in EMISSION};

otoole.preprocess.excel_to_osemosys.generate_csv_from_excel(input_workbook, output_folder)[source]¶

Generate a folder of CSV files from a spreadsheet

Parameters:	input_workbook (str) – Path to spreadsheet containing OSeMOSYS data output_folder (str) – Path of the folder containing the csv files

otoole.preprocess.excel_to_osemosys.read_config(path_to_user_config: str = None) → Dict[KT, VT][source]¶

Reads the config file holding expected OSeMOSYS set and parameter dimensions

Parameters:	path_to_user_config (str, optional, default=None) – Optional path to a user defined configuration file
Returns:
Return type:	dict

otoole.preprocess.longify_data module¶

Read in a folder of irregular wide-format csv files and write them out as narrow csvs

otoole.preprocess.longify_data.check_datatypes(narrow: pandas.core.frame.DataFrame, config_details: Dict[KT, VT], parameter: str) → pandas.core.frame.DataFrame[source]¶

Checks a parameters datatypes

Parameters:	narrow (pandas.DataFrame) – The parameter data config_details (dict) – The configuration dictionary parameter (str) – The name of the parameter

otoole.preprocess.longify_data.check_parameter(df, config_details, name)[source]¶

otoole.preprocess.longify_data.check_set(df, config_details, name)[source]¶

otoole.preprocess.longify_data.check_set_datatype(narrow: pandas.core.frame.DataFrame, config_details: Dict[KT, VT], set_name: str) → pandas.core.frame.DataFrame[source]¶

Checks the datatypes of a set_name dataframe

Parameters:	narrow (pandas.DataFrame) – The set data config_details (dict) – The configuration dictionary set_name (str) – The name of the set

otoole.preprocess.longify_data.main(output_folder, narrow_folder)[source]¶: Read in a folder of irregular wide-format files and write as narrow csvs

otoole.preprocess.longify_data.write_out_dataframe(folder, parameter, df)[source]¶

Writes out a dataframe as a csv into the data subfolder of a datapackage

Parameters:	folder (str) – parameter (str) – df (pandas.DataFrame) –

otoole.preprocess.narrow_to_datafile module¶

class otoole.preprocess.narrow_to_datafile.DataPackageTo(datapackage: str, datafilepath: str, sql: bool = False)[source]¶

Bases: object

Convert a data package to another format

Parameters:	datapackage (str) – The path to the databackage datafilepath (str) – The path to the destination file or folder sql (bool, default=False) – Flag to set whether the source datapackage is in sqlite format

convert()[source]¶: Perform the conversion from datapackage to destination format

class otoole.preprocess.narrow_to_datafile.DataPackageToCsv(datapackage: str, datafilepath: str, sql: bool = False)[source]¶: Bases: otoole.preprocess.narrow_to_datafile.DataPackageTo

class otoole.preprocess.narrow_to_datafile.DataPackageToExcel(datapackage: str, datafilepath: str, sql: bool = False)[source]¶: Bases: otoole.preprocess.narrow_to_datafile.DataPackageTo

otoole.preprocess.narrow_to_datafile.convert_datapackage_to_datafile(path_to_datapackage, path_to_datafile)[source]¶

otoole.preprocess.narrow_to_datafile.convert_datapackage_to_excel(path_to_datapackage, path_to_excel)[source]¶

otoole.preprocess package¶

Submodules¶

otoole.preprocess.create_datapackage module¶

otoole.preprocess.datafile_to_datapackage module¶

otoole.preprocess.excel_to_osemosys module¶

Sets¶

otoole.preprocess.longify_data module¶

otoole.preprocess.narrow_to_datafile module¶

Module contents¶

otoole

Navigation

Related Topics