usevortex — Proxies to reach resources using Vortex

Contains proxies to reach resources using Vortex.

Of course, this module need to have a proper Vortex installation !

Functions

epygram.extra.usevortex.set_defaults(**defaults)[source]

Set defaults key/value pairs for get_resource().

epygram.extra.usevortex.quiet_get(loggers=['vortex.data.stores', 'vortex.data.handlers', 'vortex.tools.net'])[source]

Shut off some loggers (set level to ERROR) while executing action, then restore initial level.

Example of use:

with quiet_get():
    get_resources(...)
epygram.extra.usevortex.get_resources(getmode='epygram', uselocalcache=False, **description)[source]

Get resources, given their description.

Parameters:

  • getmode –

    controls the action and returned object:

    • ’check’ return True only if the description is complete

    • ’epygram’ return the epygram resources

    • ’locate’ return the physical resolved location of the resource

    • ’exist’ return the physical resolved location of the resource and its existence

    • ’fetch’ fetches the resource in local, as filename local

    • ’vortex’ return the vortex data handler of the resource

    • ’prestaging’ puts a prestaging request for the resolved resources, on the archive system

  • uselocalcache – if True, store resources in a local cache (not automatically cleaned, take care) defined either (and by priority order) in $MTOOL_STEP_CACHE, $MTOOLDIR, $FTDIR, $WORKDIR, $TMPDIR.

Examples:

  • for the analysis of AROME-france from experiment 864G on 2015/08/15/00, description will look like:

    • experiment=’864G’, # the experiment id

    • model=’arome’,

    • block=’analysis’, # the OLIVE block

    • kind=’analysis’, # the kind of resource

    • date=’2015081500’, # the initial date and time

    • geometry=’franmgsp’, # the name of the model domain

    • local=’analysis_[experiment]’, # the local filename of the resource, once fetched.

  • for the model state at term 18h of ALADIN-reunion oper, production cutoff, on 2015/08/15/00, description will look like:

    • suite=’oper’, # the suite // ‘dble’ = e-suite

    • kind=’historic’, # model state

    • date=’2015081500’, # the initial date and time

    • term=18, # the forecast term

    • geometry=’reunionsp’, # the name of the model domain

    • local=’ICMSHALAD_[term]’, # the local filename of the resource, once fetched.

    • cutoff=’prod’, # type of cutoff

    • vapp=’aladin’, # type of application in operations namespace

    • vconf=’reunion’, # name of config in operation namespace

    • model=’aladin’, # name of the model

  • for the GRIB post-processed output on MASCA025 BDAP domain at terms 22->24h of ALADIN-reunion oper, production cutoff, on 2015/08/15/00, description will look like:

    • suite=’oper’, # the suite // ‘dble’ = e-suite

    • kind=’gridpoint’, # model state

    • date=’2015081500’, # the initial date and time

    • term=[22,23,24], # the forecast terms

    • geometry=’masca025’, # the name of the post-processing domain

    • local=’[geometry]_[term].grb’, # the local filename of the resource, once fetched.

    • nativefmt=’grib’ # to fetch the gribs and not the FA post-processed files

    • cutoff=’prod’, # type of cutoff

    • origin=’hst’, # origin of post-processed files: historic

    • vapp=’aladin’, # type of application in operations namespace

    • vconf=’reunion’, # name of config in operation namespace

    • model=’aladin’, # name of the model

epygram.extra.usevortex.extractor(vortex_description, coords, start_cutoff, end_cutoff, start_term, end_term, points_fields=None, profiles_FAfields=None, everycutoff_in_hours=24, everyterm_in_hours=1, vertical_coordinate=None, extraction_options=None, use_local_cache=False, prestaging=False, outdir='/home/mary/repositories/EPyGrAM/docs/source', progressmode=None, nc_global_attributes={'history': 'Extracted on 2025-01-16 using epygram.', 'institution': 'Météo France'})[source]

Extract temporal series of points (and profiles for FA historical files) to netCDF files.

Series are either:

  • one series == one file per cutoff if start_term* != **end_term

  • one series == one file containing all cutoffs if start_term* == **end_term

Shall be extended to series of profiles for other formats later on.

Parameters:

  • vortex_description –

    vortex description of the resources, incl. providers, excepted dates/terms Example: {‘model’:’arome’,

    ’vapp’:’arome’, ‘vconf’:’3dvarfr’, ‘kind’:’historic’, ‘suite’:’oper’, ‘geometry’:’franmgsp’, ‘cutoff’:’prod’, #’namespace’:’oper.archive.fr’, #’experiment’:’86U8’, #’block’:’forecast’, #’nativefmt’:’fa’, #’member’:8,}

  • coords – either a list of lon/lat coordinates or a dict of, in which case key would be a “name of the point”, used in output filename

  • start_cutoff – first cutoff of the series (datetime.datetime instance)

  • end_cutoff – last cutoff of the series

  • start_term – first term of the series (datetime.datetime instance)

  • end_term – last term of the series

  • points_fields –

    dictionnary of fields to extract with options, e.g.:

    {fid1:{‘nc_name’:nc_fid1,
    ’nc_attributes’:{‘unit’:…,

    ’comment’:…, },

    ’decumulate’:True, ‘operation’:(‘-‘,273.15) or (‘exp’,), ‘reproject_wind_on_lonlat’:False, },

    fid2:… }

    • ’decumulate’ can be absent==False, True, or ‘center’ for centered decumulation (cf. D3Field.decumulate for more details);

    • ’operation’ is done after decumulation, if both present

    • ’reproject_wind_on_lonlat: FA only: reproject winds on lonlat axes (True by default)

  • profiles_FAfields – like points_fields but for profiles, but only for FA historic resources ! fids supposed to be ‘S*PARAMETER’

  • everycutoff_in_hours – step between 2 cutoffs

  • everyterm_in_hours – step between 2 terms

  • vertical_coordinate – vertical coordinate requested for profiles, among (‘pressure’, ‘height’, ‘altitude’, None)

  • extraction_options – options to FA.extractprofile() and H2DField.extract_point() concerning interpolation, as dict…

  • use_local_cache – use a local Vortex cache, in which files are stored

  • prestaging – triggers prestaging for the requested resources (Hendrix)

  • outdir – directory in which to store output files

  • progressmode – sets the verbosity of the task among (None, ‘percentage’, ‘verbose’)

  • nc_global_attributes – given as a dict. Defaults are above in module variable default_nc_global_attributes

Classes

class epygram.extra.usevortex.Extractor(*args, **kwargs)[source]

Bases: Worker

Independant extractor.

Note

This class is managed by footprint.

  • info: Run extractor().

  • priority: PriorityLevel::DEFAULT (rank=1)

Automatic parameters from the footprint:

  • directives (footprints.stdtypes.FPDict) - rxx - Contains all arguments to extractor(…).

  • expected_time (builtins.float) - rxx - How long the worker is expected to run (in s).

    • Optional. Default is 0.0.

  • memory (builtins.float) - rxx - Memory that should be used by the worker (in MiB).

    • Optional. Default is 0.0.

  • name (builtins.str) - rwx - Name of the worker.

    • Optional. Default is None.

  • scheduler_hooks (footprints.stdtypes.FPList) - rxx - List of callbacks before starting effective task work.

    • Optional. Default is FPList::<<as_list:: []>>.

  • scheduler_ticket (builtins.int) - rxx - The slot number given by the scheduler (optional).

    • Optional. Default is None.

directives

Contains all arguments to extractor(…) (see the documentation above for more details).

expected_time

How long the worker is expected to run (in s) (see the documentation above for more details).

memory

Memory that should be used by the worker (in MiB) (see the documentation above for more details).

name

Name of the worker (see the documentation above for more details).

scheduler_hooks

List of callbacks before starting effective task work (see the documentation above for more details).

scheduler_ticket

The slot number given by the scheduler (optional) (see the documentation above for more details).