Plots from operational simulations

Created on 6 déc. 2018

@author: lafaysse

usage: python postprocess.py [-b YYYYMMDD] [-e YYYYMMDD] [-o diroutput]

1. extracts operational simulation results

2. Plots maps for the Alps, the Pyrenees the Corse, Vosges, Massif Central, Jura

3. Creates spaghetti plots for all massifs and stations

class plots.pearps2m.postprocess.Config(OPTIONS)

Bases: object

Configuration passed to S2MExtractor and to be used in vortex toolboxes.

1. checks and converts dates

2. creates output directories if they don’t exist.

__init__(OPTIONS)

1. checks and converts dates

2. creates output directories if they don’t exist.

alternate_xpid = ['OPER@lafaysse']

Alternative experiment id

diroutput

output directory

diroutput_maps

output directory for maps

diroutput_plots

output directory for other plots

list_geometry = ['alp', 'pyr', 'cor', 'jur', 'mac', 'vog', 'postes']

List of geometries

list_members = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]

35 for determinstic member, 36 for sytron, 0-34 for PEARP members

previ = True

False for analysis, True for forecast

rundate

Date of model run bronx.stdtypes.date.Date

xpid = 'oper'

Operational chain

class plots.pearps2m.postprocess.Ensemble

Bases: object

Describes an ensemble of simulations

__init__()

close()

Close simulation files and remove data from ensemble.

csg_mean_nonorm(a1, a2, a3, a4, ensmean, ensPOP)

Calculate mu of predictive censored shifted gamma distribution from regression coefficients alpha 1 to alpha 4, ensemble mean and probability of non-zero values of the ensemble.

Parameters:

• a1 (float) – regression coefficient alpha 1

• a2 (float) – regression coefficient alpha 2

• a3 (float) – regression coefficient alpha 3

• a4 (float) – regression coefficient alpha 4

• ensmean (float) – ensemble mean

• ensPOP (float) – probability of non-zero values of the ensemble [0., 1.]

Returns:

mu of predictive censored shifted gamma distribution

Return type:

float

csg_std_nonorm(b1, muclim, sigmaclim, csgmean)

Calculate sigma of predictive censored shifted gamma distribution from regression coefficients beta 1, climatological mu and sigma and csg mu parameter.

Parameters:

• b1 (float) – regression coefficient beta 1

• muclim (float) – climatological mu

• sigmaclim (float) – climatological sigma

• csgmean (float) – csg mu parameter

Returns:

sigma of predictive censored shifted gamma distribution

Return type:

float

emos_csg_nonorm_allstations_newsnow(varname, level)

Applies Censored Shifted Gamma (CSG) EMOS trained for 24h new snow variable and all stations in the Alps without normalisation (internship Benoit Gacon 2024) to variable “varname” and returns the quantiles given in “levels” of the CSG distribution.

Parameters:

• varname (str) – variable name

• level (array) – quantiles (range 0 to 100)

Returns:

quantiles of predictive distribution

Return type:

np.array

ensemble

data dict with variable names as keys and np.arrays as values

get_alti()

Get altitude variable from the first simulation file.

Returns:

altitude variable

Return type:

numpy array

get_aspect()

Get aspect variable from the first simulation file.

Returns:

aspect variable

Return type:

numpy array

get_csg_predictive_dist_nonorm_allstations(varname)

get the censored shifted gamma regression coefficients and climatological paramters for each lead time and apply them to the ensemble forecast and return the parameters mu, sigma and delta of the predictive censored shifted gamma distribution.

Parameters:

varname – variable name

Returns:

mu, sigma, delta of the predictive CSG distribution

get_metadata()

Get a tuple with spatial indices.

Returns:

indpoints, indpoints

get_npoints()

Get the number of spatial points.

Returns:

total number of spatial points

Return type:

int

indpoints

Variables:

indpoints – list with spatial indices :vartype indpoints: list

mean(varname)

Calculate ensemble mean for a given variable.

Parameters:

varname (str) – variable name

Returns:

ensemble mean

Return type:

numpy array

nech

number of time steps (forecast steps). type int

nmembers

number of members type int

npoints

Variables:

npoints – total number of spatial points :vartype npoints: int

open(listmembers)

Opens simulation files

Parameters:

listmembers (list) – list of ensemble members (filenames)

Variables:

inddeterministic (int) – index of the deterministic member

probability(varname, seuilinf=-999999999.0, seuilsup=999999999.0)

Calculates probability as the proportion of ensemble members with values larger than seuilinf and smaller than seuilsup.

Parameters:

• varname (str) – name of the variable for which to calculate the probability

• seuilinf – lower threshold

• seuilsup – upper threshold

Returns:

probability field (or np.nan if the ensemble is not defined)

Return type:

numpy array

quantile(varname, level)

Calculates ensemble percentiles for a given variable and given percentile levels.

Parameters:

• varname (str) – Variable name

• level (list of numbers between 0 and 100) – list of percentiles to calculate

Returns:

array of percentiles

Return type:

numpy array

quantiles_CSGD(csgmean, csgstd, csgdelta, quantiles)

get quantiles of the censored shifted gamma (CSG) distribution with given mu, sigma and delta.

Parameters:

• csgmean (array) – mu parameter of CSG

• csgstd (array) – sigma parameter of CSG

• csgdelta (array) – delta parameter of CSG

• quantiles (array of floats) – quantiles wanted

Returns:

array of quantiles

Return type:

array of same length as quantiles.

read(varname)

Read a variable and store it in ensemble

Parameters:

varname (str) – name of the variable to be read (corresponds to the variable name of the simulation NetCDF files)

read_geovar(varname)

Read a variable from the first simulation file.

Parameters:

varname (str) – NetCDF variable name of the variable to read

Returns:

data read

Return type:

numpy array

select_points()

Get a list of spatial indices from the spatial dimension length in the first simulation file.

Returns:

list of spatial points (indices)

Return type:

list

simufiles

list of prosimu objects with simulation file

property spatialdim

Name of spatial dimension

spread(varname)

Calculate ensemble spread (standard deviation) for a given variable.

Parameters:

varname (str) – variable name

Returns:

ensemble spread

Return type:

numpy array

Return type:

numpy array

time

Variables:

~.time – time variable from the first simulation file :vartype ~.time: numpy array

class plots.pearps2m.postprocess.EnsembleDiags

Bases: Ensemble

Probabilities and quantiles from ensembles.

__init__()

close()

Close simulation files and remove data from ensemble, proba and quantiles

diags(list_var, list_quantiles, list_seuils)

Calculate quantiles and exceedance probabilities for a list of variables, quantiles and thresholds. The quantiles are stored in quantiles and the probabilities in proba

Parameters:

• list_var (iterable) – list of variables

• list_quantiles (iterable) – list of percentiles to be calculated

• list_seuils (dict of iterables with variable names as keys) – list of thresholds for each variable

proba

probabilities

quantiles

percentiles

class plots.pearps2m.postprocess.EnsembleFlatMassif

Bases: _EnsembleMassif

Class for ensemble simulations on a massif geometry (SAFRAN like) where all data points are considered on flat terrain (zero slope and no orientation information).

select_points()

Select spatial indices from the first simulation file where aspect=-1 (zero slope, no orientation information).

Returns:

spatial indices to be read from simulation files

Return type:

numpy boolean array

class plots.pearps2m.postprocess.EnsembleHydro(variables, inputfiles, decile=array([10, 20, 30, 40, 50, 60, 70, 80, 90]), outdir='.', outfilename='PRO_post.nc', emosmethod=None)

Bases: EnsemblePostproc

Class to provide a synthesis of ensemble hydrological diagnostics of S2M

Parameters:

• variables – list of variables to process

• inputfiles – list of input files

• decile – list of percentiles

• outdir (str) – Output directory

• outfilename – output file name

• emosmethod – emos method to use. The method is supposed to take a variable name and the wanted quantiles as arguments.

property spatialdim

Name of spatial dimension

property standardvars

variables always written to the output file

class plots.pearps2m.postprocess.EnsembleMassifPoint(massif_num, alti, aspect, slope)

Bases: _EnsembleMassif

Class for extracting one specific point in massif geometry

__init__(massif_num, alti, aspect, slope)

select_points()

Select index of the corresponding point

class plots.pearps2m.postprocess.EnsembleNorthSouthMassif

Bases: _EnsembleMassif

Class for ensemble simulations on a massif geometry (SAFRAN like) where data points are considered at a slope of 40 degrees and two orientations: North and South.

select_points()

Select spatial indices from the first simulation file where slope=40 and aspect either 0 or 180.

Returns:

spatial indices to be read from simulation files

Return type:

a tuple of numpy boolean array, where the first component corresponds to the Northern orientation and the second to the Southern one.

class plots.pearps2m.postprocess.EnsembleOperDiags

Bases: EnsembleDiags

Class for operationally used plots.

__init__()

alldiags()

Calculate percentiles for all variables in list_var_spag and list_var_map.

attributes = {'DSN_T_ISBA': {'convert_unit': 100.0, 'label': 'Hauteur de neige (cm)'}, 'NAT_LEV': {'forcemax': 5.5, 'forcemin': -0.5, 'label': 'Risque naturel', 'ncolors': 6, 'palette': 'YlOrRd', 'ticks': ['Très faible', 'Faible', 'Mod. A', 'Mod. D', 'Fort', 'Très fort']}, 'PP_SD_1DY_ISBA': {'convert_unit': 1.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Epaisseur de neige fraîche en 24h (cm)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'RAINF_ISBA': {'convert_unit': 10800.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Pluie en 3h (kg/m2/3h)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'RAMSOND_ISBA': {'convert_unit': 100.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Epaisseur mobilisable (cm)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'REFRZTH_ISBA': {'convert_unit': 100.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Epaisseur regelée (cm)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'SD_1DY_ISBA': {'convert_unit': 100.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Epaisseur de neige fraîche en 24h (cm)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'SD_3DY_ISBA': {'convert_unit': 100.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Epaisseur de neige fraîche en 72h (cm)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'SNOMLT_ISBA': {'convert_unit': 10800.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Ecoulement en 3h (kg/m2/3h)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'WET_TH_ISBA': {'convert_unit': 100.0, 'forcemax': 60.0, 'forcemin': 0.0, 'label': 'Epaisseur humide (cm)', 'palette': 'YlGnBu', 'seuiltext': 50.0}, 'WSN_T_ISBA': {'label': 'Equivalent en eau (kg/m2)'}, 'naturalIndex': {'forcemax': 8.0, 'forcemin': 0.0, 'format': '%.1f', 'label': 'Indice de risque naturel', 'nolevel': True, 'palette': 'YlOrRd'}}

dict of plot attributes for each variable

formatplot = 'png'

plot format

list_q = [20, 50, 80]

list of quantiles

pack_spaghettis(suptitle, diroutput='.')

Produce spaghetti plots for all variables in list_var_spag.

Parameters:

• suptitle (unicode string) – Suptitle for all plots.

• diroutput – directory to save the plots

pack_spaghettis_multipoints(list_pairs, suptitle, diroutput='.', **kwargs)

Produce spaghetti plots with up to four points per plot for variables in list_var_spag_2points.

Each point gets a different color.

Parameters:

• list_pairs (list of list) – list of indices to plot together on each plot

• suptitle (unicode str) – common suptitle for all plots

• diroutput – output directory

• kwargs – arguments to be passed to plotting routines, notably “labels”

class plots.pearps2m.postprocess.EnsembleOperDiagsFlatMassif

Bases: EnsembleOperDiags, EnsembleFlatMassif

Class for operationally plotting maps and spaghetti plots for ensembles with massif geometry and for the zero slope case.

levelmax = 3900

maximum height level to be treated

levelmin = 0

minimum heigth level to be treated

list_var_map = ['naturalIndex', 'SD_1DY_ISBA', 'SD_3DY_ISBA', 'SNOMLT_ISBA']

list of variables to plot maps for

list_var_spag = ['naturalIndex', 'DSN_T_ISBA', 'WSN_T_ISBA', 'SNOMLT_ISBA']

list of variables to do spaghetti plots for

pack_maps(domain, suptitle, diroutput='.')

Produce maps for the variables given in list_var_map over the regions given in domain for each available altitude level and time step.

Each massif is colored corresponding to the second percentile value defined in list_q and the color map defined in attributes. At the center of each massif the values corresponding to the first three percentiles in list_q are marked.

Parameters:

• domain – list of region identifiers (e.g., “alp”, “pyr”, “cor”)

• suptitle (str) – common suptitle for all plots

• diroutput – output directory to save the maps

class plots.pearps2m.postprocess.EnsembleOperDiagsNorthSouthMassif

Bases: EnsembleOperDiags, EnsembleNorthSouthMassif

Class for operationally plot maps and spaghetti plots distinguishing between northern and southern orientation at each massif.

alldiags()

Calculate percentiles for all variables in list_var_spag_2points and list_var_map.

ensemble = {}

ensemble data

get_pairs_ns()

Get for each massif and altitude the pair of indices corresponding to values for the northern and southern slopes.

Returns:

indices corresponding to northern and southern slopes

Return type:

list of lists

levelmax = 3900

maximum height level to plot

levelmin = 0

minimum height level to plot

list_var_map = ['RAMSOND_ISBA', 'NAT_LEV', 'WET_TH_ISBA', 'REFRZTH_ISBA']

variable list for producing maps

list_var_spag = []

variable list for standard spaghetti plots

list_var_spag_2points = ['RAMSOND_ISBA', 'NAT_LEV', 'WET_TH_ISBA', 'REFRZTH_ISBA']

variable list for spaghetti plots combining northern and southern orientation

pack_maps(domain, suptitle, diroutput)

Produce maps for the variables given in list_var_map over the regions given in domain for each available altitude level and time step.

For each massif a table with background colors and values corresponding to the percentiles in list_q and the color map defined in attributes is plotted near the center of each massif.

Parameters:

• domain – list of region identifiers (e.g., “alp”, “pyr”, “cor”)

• suptitle (str) – common suptitle for all plots

• diroutput – output directory to save the maps

pack_spaghettis_ns(suptitle, diroutput='.')

Do spaghetti plots with values for the northern and southern slope on the same plot.

Parameters:

• suptitle – common suptitle for all plots

• diroutput – directory to save the plots

versants = ['Nord 40°', 'Sud 40°']

labels for the two orientations

class plots.pearps2m.postprocess.EnsembleOperDiagsStations

Bases: EnsembleOperDiags, EnsembleStation

Class for operationally plotting spaghetti plots for station based simulations.

list_var_map = []

list of variables to plot maps for

list_var_spag = ['DSN_T_ISBA', 'WSN_T_ISBA', 'RAMSOND_ISBA', 'WET_TH_ISBA', 'REFRZTH_ISBA', 'SNOMLT_ISBA']

list of variables to plot spaghetti plots for.

class plots.pearps2m.postprocess.EnsemblePostproc(variables, inputfiles, decile=array([10, 20, 30, 40, 50, 60, 70, 80, 90]), outdir='.', outfilename='PRO_post.nc', emosmethod=None)

Bases: _EnsembleMassif

Class for ensemble post-processing.

Parameters:

• variables – list of variables to process

• inputfiles – list of input files

• decile – list of percentiles

• outdir (str) – Output directory

• outfilename – output file name

• emosmethod – emos method to use. The method is supposed to take a variable name and the wanted quantiles as arguments.

__init__(variables, inputfiles, decile=array([10, 20, 30, 40, 50, 60, 70, 80, 90]), outdir='.', outfilename='PRO_post.nc', emosmethod=None)

Parameters:

• variables – list of variables to process

• inputfiles – list of input files

• decile – list of percentiles

• outdir (str) – Output directory

• outfilename – output file name

• emosmethod – emos method to use. The method is supposed to take a variable name and the wanted quantiles as arguments.

create_outfile()

Create output data set.

Variables:

outdataset (utils.prosimu.prosimu) – output data set

decile

list of percentiles

deciles()

Calculates percentiles given in decile for variables in variables and adds them to the output data set including the corresponding dimension, coordinate variable and attributes applying the method in emosmethod to obtain the percentiles.

init_outfile()

Copy global attributes, dimensions and standard variables from the first simulation file to the output file.

median()

Calculates the median for variables in variables and adds the median to the output data set including the corresponding attributes.

outfile

output filename

postprocess()

Do postprocessing

1. create output file

2. copy global attributes and standard variables to the output file

3. calculate deciles for the data variables and put them to the output file

4. close all input and output data sets.

Calls create_outfile(), init_outfile(), deciles() and close methods for all data sets.

property standardvars

variables always written to the output file

variables

list of variables

class plots.pearps2m.postprocess.EnsemblePostprocStation(variables, inputfiles, decile=array([10, 20, 30, 40, 50, 60, 70, 80, 90]), outdir='.', outfilename='PRO_post.nc', emosmethod=None)

Bases: EnsemblePostproc

Class for ensemble postprocessing at station locations.

Parameters:

• variables – list of variables to process

• inputfiles – list of input files

• decile – list of percentiles

• outdir (str) – Output directory

• outfilename – output file name

• emosmethod – emos method to use. The method is supposed to take a variable name and the wanted quantiles as arguments.

property standardvars

variables always written to the output file

class plots.pearps2m.postprocess.EnsembleStation

Bases: Ensemble

Class for ensemble simulations at station locations.

Variables:

InfoMassifs – Information on the Massif the station is situated.

build_filename(station, alti)

Construct a filename from a station number

Parameters:

• station – station number

• alti – altitude level (not used)

Returns:

filename

Return type:

station number as 8digit integer.

build_title(station, alti)

Construct a title from a station number and altitude level.

Parameters:

• station – station number

• alti – station altitude

Returns:

title string composed of the station name and the altitude.

Return type:

unicode str

property geo

Geometry information.

Returns:

“stations”

get_metadata(**kwargs)

Construct filenames and plot titles from altitude and station information

Parameters:

kwargs

Returns:

a list of filenames and a list of plot titles

get_station()

Read station numbers from the first simulation file.

Returns:

station numbers

Return type:

numpy array

class plots.pearps2m.postprocess.PPQuantiles(filename)

Bases: object

Class for handling files containing postprocessed (quantile) forecasts.

__init__(filename)

levelmax = 3900

maximum height level to be treated

levelmin = 0

minimum heigth level to be treated

list_var_map = ['SD_1DY_ISBA', 'SD_12H_ISBA']

list of variables to plot maps for

list_var_spag = ['SD_1DY_ISBA', 'SD_12H_ISBA']

list of variables to do spaghetti plots for

quantilemaps(domain, suptitle, diroutput, mapindex=array([False, True, False, False, True, False, False, True, False]))

Plots maps of quantile forecasts. For example EMOS postprocessed quantiles.

Parameters:

• domain – domain (region) string to determine which map class to use

• suptitle – Plot title highest level

• diroutput – Output directory for the plots

• mapindex – array of bool indicating for with indices of the quantile dimension (last dimension of the data array) maps should be drawn.

spaghetti_plots(suptitle, diroutput='.')

Produce spaghetti plots for all variables in list_var_spag.

Parameters:

• suptitle (unicode string) – Suptitle for all plots.

• diroutput – directory to save the plots

class plots.pearps2m.postprocess._EnsembleMassif

Bases: Ensemble

Metaclass for ensemble simulations on a massif geometry (SAFRAN like).

Variables:

InfoMassifs – Information of Massifs

build_filename(massif, alti)

Construct a filename from massif and altitude information.

Parameters:

• massif – a massif number

• alti – an altitude level

Returns:

filename

Return type:

str

build_title(massif, alti)

Construct a figure title from massif and altitude information.

Parameters:

• massif – a massif number

• alti – an altitude level

Returns:

a title

Return type:

unicode str

property geo

Geometry

Returns:

“massifs”

get_massifdim()

Read massif_num variable from the first simulation file.

Returns:

massif numbers

Return type:

numpy array

get_massifvar()

Read “massif” variable from the first simulation file.

Returns:

massif numbers

Return type:

numpy array

get_metadata(nolevel=False)

Construct filenames and plot titles from massif and altitude variables in the first simulation file.

Parameters:

nolevel – if True the altitude is not included in the filenames and titles.

Returns:

a list of filenames and a list of titles

Return type:

two lists.

read(varname)

Read data for a given variable name into the ensemble instance variable.

Parameters:

varname (str) – variable name

plots.pearps2m.postprocess.build_filename(massif, alti)

Construct a filename from massif and altitude information.

Parameters:

• massif – a massif number

• alti – an altitude level

Returns:

filename

Return type:

str

plots.pearps2m.postprocess.main(c)

Parameters:

c – config

plots.pearps2m.postprocess.pp_plots(c)

downloads files with postprocessed quantiles and plots maps and spaghetti plots of them

Parameters:

c – config