Compare two versions of output NetCDF file

Created on 1 oct 2014 Major update on 13 sept 2019

@author: Matthieu Lafaysse

class utils.operational_reproductibility.compare2versions.ComparisonNetcdf

Bases: object

Class for Netcdf comparison

compare2files(name1, name2, checktime=True)

main comparison method

Parameters:

• name1 – filename of first file

• name2 – filename of second file

• checktime (bool) – check time variable

Returns:

conform

Return type:

bool

getlistvar(afile)

get variable list from given file

Parameters:

afile (SimplifiedProsimu) – file object

Returns:

list of variables

report(info)

Define what should be done with the information message

class utils.operational_reproductibility.compare2versions.ComparisonS2MDbleDev(nmembers)

Bases: ComparisonS2MIntDev

Class for comparing dev chain with “chaine en double”.

xpid_int = 'DBLE'

experiment id

class utils.operational_reproductibility.compare2versions.ComparisonS2MIntDev(nmembers)

Bases: ComparisonNetcdf

This class is not useful for vortex but is temporarily used while netcdf comparison is not implemented in the diff toolbox.

comparealldomains(date)

Comparison loop over all domains (“alp_allslopes”, “pyr_allslopes”, “cor_allslopes”, “postes”) and “cutoffs” (“A”, “P”) = analysis and forecast

Parameters:

date – date to be compared

Returns:

conform

Return type:

bool

Calls:

compareallmembers()

compareallmembers(vconf, date, cutoff)

Comparison loop over members

Parameters:

• vconf – vortex vconf

• date – run date

• cutoff – “A” or “P” (for analysis or forecast)

Returns:

conform

Return type:

bool

compareallruns(date)

Comparison loop over all runs on a given day

Parameters:

date – run date

Returns:

conform

Return type:

bool

Calls:

comparealldomains()

dirdate(date, cutoff)

get date directory name

Parameters:

• date – rundate

• cutoff – “A” or “P” (for analysis or forecast)

Returns:

date directory name

Return type:

str

filename = {'meteo': 'forcing', 'prep': 'prep', 'pro': 'pro'}

filename dict for s2m files

firstassimruntime = Time(6, 0)

runtime for second run of the day

get_period(rundate, cutoff)

get simulation start date and end date

Parameters:

• rundate – simulation reference date

• cutoff – “A” or “P” (for analysis or forecast)

Returns:

datebegin, dateend

getpathdev(vconf, date, cutoff, member, block)

get path of development simulation

Parameters:

• vconf – vortex vconf

• date – rundate

• cutoff – “A” or “P” (for analysis or forecast)

• member – member number

• block – vortex block (“meteo”, “pro”, “prep”)

Returns:

path

getpathint(vconf, date, cutoff, member, block)

get path of operational simulation

Parameters:

• vconf – vortex vconf

• date – rundate

• cutoff – “A” or “P” (for analysis or forecast)

• member – member number

• block – vortex block (“meteo”, “pro”, “prep”)

Returns:

path

monthly_analysis_time = Time(12, 0)

hour of the monthly reanalysis

nightruntime = Time(3, 0)

runtime for first run of the day

nmembers = {'meteo': {'alp': 36, 'cor': 36, 'postes': 35, 'pyr': 36}, 'prep': {'alp': 37, 'cor': 37, 'postes': 35, 'pyr': 37}, 'pro': {'alp': 37, 'cor': 37, 'postes': 35, 'pyr': 37}}

member number dict

nmembers_meteo = {'alp': 36, 'cor': 36, 'postes': 35, 'pyr': 36}

number of members for safran meteo simulation

nmembers_snow = {'alp': 37, 'cor': 37, 'postes': 35, 'pyr': 37}

number of members for snow simulation

pathdev = '/scratch/mtool/lafaysse/cache/vortex/s2m'

path to development chain

pathint = '/chaine/mxpt001/vortex/mtool/cache/vortex/s2m'

path to operational s2m chain

secondassimruntime = Time(9, 0)

runtime of third run of the day

xpid_dev = 'nouveaux_guess@lafaysse'

experiment id of development chain

xpid_int = 'OPER'

experiment id of operational chain

class utils.operational_reproductibility.compare2versions.FastComparisonS2MDbleDev(nmembers)

Bases: FastComparisonS2MIntDev, ComparisonS2MDbleDev

Class for fast comparison dev chain with “chaine en double”.

class utils.operational_reproductibility.compare2versions.FastComparisonS2MIntDev(nmembers)

Bases: ComparisonS2MIntDev

Fast comparison of operational chain with development chain

getlistvar(afile)

get reduced list of variables to compare

Parameters:

afile (SimplifiedProsimu) – file object

class utils.operational_reproductibility.compare2versions.SimplifiedProsimu(filename)

Bases: Dataset

Simplified prosimu class

`__init__(self, filename, mode=”r”, clobber=True, diskless=False, persist=False, keepweakref=False, memory=None, encoding=None, parallel=False, comm=None, info=None, format=’NETCDF4’)`

Dataset constructor.

`filename`: Name of netCDF file to hold dataset. Can also be a python 3 pathlib instance or the URL of an OpenDAP dataset. When memory is set this is just used to set the filepath().

`mode`: access mode. r means read-only; no data can be modified. w means write; a new file is created, an existing file with the same name is deleted. a and r+ mean append (in analogy with serial files); an existing file is opened for reading and writing. Appending s to modes r, w, r+ or a will enable unbuffered shared access to NETCDF3_CLASSIC, NETCDF3_64BIT_OFFSET or NETCDF3_64BIT_DATA formatted files. Unbuffered access may be useful even if you don’t need shared access, since it may be faster for programs that don’t access data sequentially. This option is ignored for NETCDF4 and NETCDF4_CLASSIC formatted files.

`clobber`: if True (default), opening a file with mode=’w’ will clobber an existing file with the same name. if False, an exception will be raised if a file with the same name already exists.

`format`: underlying file format (one of ‘NETCDF4’, ‘NETCDF4_CLASSIC’, ‘NETCDF3_CLASSIC’, ‘NETCDF3_64BIT_OFFSET’ or ‘NETCDF3_64BIT_DATA’. Only relevant if mode = ‘w’ (if mode = ‘r’,’a’ or ‘r+’ the file format is automatically detected). Default ‘NETCDF4’, which means the data is stored in an HDF5 file, using netCDF 4 API features. Setting format=’NETCDF4_CLASSIC’ will create an HDF5 file, using only netCDF 3 compatible API features. netCDF 3 clients must be recompiled and linked against the netCDF 4 library to read files in NETCDF4_CLASSIC format. ‘NETCDF3_CLASSIC’ is the classic netCDF 3 file format that does not handle 2+ Gb files. ‘NETCDF3_64BIT_OFFSET’ is the 64-bit offset version of the netCDF 3 file format, which fully supports 2+ GB files, but is only compatible with clients linked against netCDF version 3.6.0 or later. ‘NETCDF3_64BIT_DATA’ is the 64-bit data version of the netCDF 3 file format, which supports 64-bit dimension sizes plus unsigned and 64 bit integer data types, but is only compatible with clients linked against netCDF version 4.4.0 or later.

`diskless`: If True, create diskless (in-core) file. This is a feature added to the C library after the netcdf-4.2 release. If you need to access the memory buffer directly, use the in-memory feature instead (see memory kwarg).

`persist`: if diskless=True, persist file to disk when closed (default False).

`keepweakref`: if True, child Dimension and Variable instances will keep weak references to the parent Dataset or Group object. Default is False, which means strong references will be kept. Having Dimension and Variable instances keep a strong reference to the parent Dataset instance, which in turn keeps a reference to child Dimension and Variable instances, creates circular references. Circular references complicate garbage collection, which may mean increased memory usage for programs that create may Dataset instances with lots of Variables. It also will result in the Dataset object never being deleted, which means it may keep open files alive as well. Setting keepweakref=True allows Dataset instances to be garbage collected as soon as they go out of scope, potentially reducing memory usage and open file handles. However, in many cases this is not desirable, since the associated Variable instances may still be needed, but are rendered unusable when the parent Dataset instance is garbage collected.

`memory`: if not None, create or open an in-memory Dataset. If mode = ‘r’, the memory kwarg must contain a memory buffer object (an object that supports the python buffer interface). The Dataset will then be created with contents taken from this block of memory. If mode = ‘w’, the memory kwarg should contain the anticipated size of the Dataset in bytes (used only for NETCDF3 files). A memory buffer containing a copy of the Dataset is returned by the Dataset.close method. Requires netcdf-c version 4.4.1 for mode=’r, netcdf-c 4.6.2 for mode=’w’. To persist the file to disk, the raw bytes from the returned buffer can be written into a binary file. The Dataset can also be re-opened using this memory buffer.

`encoding`: encoding used to encode filename string into bytes. Default is None (sys.getdefaultfileencoding() is used).

`parallel`: open for parallel access using MPI (requires mpi4py and parallel-enabled netcdf-c and hdf5 libraries). Default is False. If True, comm and info kwargs may also be specified.

`comm`: MPI_Comm object for parallel access. Default None, which means MPI_COMM_WORLD will be used. Ignored if parallel=False.

`info`: MPI_Info object for parallel access. Default None, which means MPI_INFO_NULL will be used. Ignored if parallel=False.

getattr(varname, attname)

get a specific attribute for a specific variable

Parameters:

• varname (str) – the variable name

• attname (str) – the attribute name

Returns:

the attribute value

listattr(varname)

list of attributes for a variable

Parameters:

varname (str) – the variable name

Returns:

atributes

listdim()

list of dimensions

Returns:

list of dimensions

Return type:

list

listvar()

list of variables

Returns:

list of variables

Return type:

list

read(varname)

read variable from NetCDF file.

Parameters:

varname – variable name

Returns:

data array

readtime()

Get the time dimension of the netCDF file

Returns:

time axis data

Trype:

numpy array