Algo Components / computational tasks

Specific CEN AlgoComponents.

components.py

Abstract algo components classes for multiprocessing simulations with an external parallelisation (i.e not MPI). The parallelization is typically over the simulation members or independent sub-periods. All processes share the same executable (if any) but at least partially different IO environments or script arguments.

These algo components rely on the taylorism package. The main process (the Boss) acts as a task scheduler to to allocate tasks to a set of independent processes (the Workers) working simultaneously.

Two abstract “Boss” classes are defined here are:

• _CenParaBlindRun: to launch an executable multiple times in parallel

• _CenTaylorRun: to launch a piece of Python code in parallel on several processes

A third abstract “Boss” class is available in the main vortex-nwp package:

• ParaExpresso: to launch a script multiple times in parallel

These abstract classes provide high-level tools such as the identification of the list of workers through the list of sub-directories (via the “role” of the main input resource varying from one simulation to another).

Each “Boss” is associated to a “Worker”, providing the instructions for each individual task / process. Each “Worker” works in its specific sub-directory, which must be filled with the appropriate IOs (when fetching the inputs at the “Task” level). Some IOs can be shared by several or all workers. These IOs must be stored in the main working directory, and a link to these files can be created in the Worker’s working directory by the Worker itself.

Two abstract “Worker” classes are defined here are:

• _CenWorkerBlindRun: launch an executable without MPI parallelization

• _CenTaylorVortexWorker: launch a piece of Python code

A third abstract “Worker” class is available in the main vortex-nwp package:

• VortexWorkerBlindRun: to launch a script multiple times in parallel

Here is the full inheritance diagram of the available meta classes :

class vortex_cen.algo.components._CenParaBlindRun(*kargs, **kwargs)

Bases: ParaBlindRun

This abstract algo component defines common methods for simulations based on an executable without MPI parallelization. The parallelization is typically over the simulation members or independent sub-periods, and all processes share the same executable but different IO environments.

class vortex_cen.algo.components._CenTaylorRun(*kargs, **kwargs)

Bases: TaylorRun

TaylorRun derived algo components are not (necessarily) associated to an executable and can simply launch a piece of python code.

class vortex_cen.algo.components._CenMixIn

Bases: object

class vortex_cen.algo.components._CenWorkerBlindRun(*args, **kwargs)

Bases: _CenMixIn, VortexWorkerBlindRun

This abstract worker is designed to drive the launch of any script or executable without MPI parallelization (deterministic or ensemble-like simulations) in association with an Algo Component inheriting from an _CenParaBlindRun or ParaExpresso Algo Component.

A single worker is thus a deterministic execution of a given binary or script with a specific IO environment and/or a specific set of argument.

class vortex_cen.algo.components._CenTaylorVortexWorker(*args, **kwargs)

Bases: _CenMixIn, TaylorVortexWorker

This abstract worker is designed to drive the launch a python program. The parallelisation is typically over the simulation members or independent sub-periods.

A single worker is thus a deterministic execution of a list of python commands in a specific environment that can be run in parallel with other workers.

forcing.py

Algo Components for the generation or the modification of FORCING files.

class vortex_cen.algo.forcing.ExtractMassifs(*kargs, **kwargs)

Bases: _CenTaylorRun

Algo component to extract a list of points from a set of S2M FORCING files in the “massif” geometry.

class vortex_cen.algo.forcing.ExtractMassifsWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

Worker to extract a list of points from a given S2M FORCING file in the “massif” geometry.

class vortex_cen.algo.forcing.ConcatForcings(*kargs, **kwargs)

Bases: _CenTaylorRun

Concatenation of a set of FORCING files into a single forcing.

class vortex_cen.algo.forcing.ConcatForcingsWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

Concatenation of a set of FORCING files into a single forcing.

ensemble.py

Algo Components for ensemble Surfex simulations.

class vortex_cen.algo.ensemble.S2MComponent(*kargs, **kwargs)

Bases: _CenParaBlindRun

This Algo Component is designed to manage any Surfex execution without MPI parallelization (deterministic or ensemble-like simulations).

Ensemble-like simulations include real ensemble simulations (many executions of the same simulation with different initial conditions or configurations) and multi-year simulations (associating each year to one member) that can run in parallel.

The different members of an ensemble simulation are identified by an input resource that differ between the members (defined by the method role_ref_namebuilder that can be overloaded). For each identified member, a worker object (whose class inherits from _CenWorkerBlindRun) is generated and the different workers run in parallel.

The S2MComponent class (that relies on the taylorism package):

• allocates the different executions of the same binary to workers

• analyses their feedbacks to look for execution errors (thanks to the inherited _default_post_execute() method): should the rc entry of the dictionary returned by the worker’s _CenWorkerBlindRun.vortex_task() method be an Exception, it is captured and stored.

When the execution of all members finishes, the captured exceptions (see above) are wrapped in a DelayedAlgoComponentError exception that is ultimately raised.

Consequently, we can rely on that to:

• filter/ignore some errors,

• send notifications…

For CEN R&D needs, this can be easily implemented in the Task classes provided they inherit from the vortex_cen.layout.nodes.CENTaskMixIn mixin. For example:

• To filter execution errors, just define filter_execution_error in your Task class: filter_execution_error = CENTaskMixIn.s2moper_filter_execution_error

• To send notification emails about fatal and non-fatal errors: report_execution_warning = CENTaskMixIn.s2moper_report_execution_warning and report_execution_errors = CENTaskMixIn.s2moper_report_execution_errors

class vortex_cen.algo.ensemble.SurfexComponent(*kargs, **kwargs)

Bases: S2MComponent

AlgoComponent that runs several SURFEX executions in parallel.

class vortex_cen.algo.ensemble.SurfexComponentMultiDates(*kargs, **kwargs)

Bases: SurfexComponent

AlgoComponent that runs several SURFEX in parallel (including several dates for reforecasts).

class vortex_cen.algo.ensemble.SurfexWorker(*args, **kwargs)

Bases: _CenWorkerBlindRun, SurfexMixIn

This algo component is designed to run a SURFEX experiment without MPI parallelization.

class vortex_cen.algo.ensemble.PrepareForcingComponent(*kargs, **kwargs)

Bases: _CenTaylorRun

AlgoComponent that prepares several forcing files in parallel (changes of geometry).

class vortex_cen.algo.ensemble.PrepareForcingWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

This algo component is designed to prepare a SURFEX Forcing file (change of geometry).

class vortex_cen.algo.ensemble.S2MReanalysis(*kargs, **kwargs)

Bases: S2MComponent

AlgoComponent that runs several SAFRAN reanalyses in parallel.

class vortex_cen.algo.ensemble.S2MReforecast(*kargs, **kwargs)

Bases: S2MComponent

AlgoComponent that runs several SAFRAN reforecasts in parallel.

class vortex_cen.algo.ensemble.ShadowsForcingWorker(*args, **kwargs)

Bases: PrepareForcingWorker

It only applies shadows to a forcing file without any change of geometry (worker for 1 member).

deterministic.py

Algo Components for deterministic Surfex simulations.

class vortex_cen.algo.deterministic.Surfex_PreProcess(*args, **kw)

Bases: AlgoComponent

Algo component to apply the required preprocessing before a SURFEX run (e.g. namelists adjustements) WARNING : this algo should now be used only in the operationnal S2M chain

class vortex_cen.algo.deterministic.Generate_Clim_TG(*args, **kw)

Bases: AlgoComponent

Algocomponent that generates a file of initial temperature to initialize the soil from surfex meteorological forcing files

class vortex_cen.algo.deterministic.Pgd_Parallel_from_Forcing(*args, **kw)

Bases: Parallel, SurfexMixIn

This algo component is designed to run PGD with MPI parallelization and using a FORCING.nc as input for topography.

class vortex_cen.algo.deterministic.Surfex_Parallel(*args, **kwargs)

Bases: Parallel, DrHookDecoMixin, SurfexMixIn

This algo component is designed to run SURFEX experiments over large domains with MPI parallelization.

class vortex_cen.algo.deterministic.Surfex_Xios_Parallel(*args, **kwargs)

Bases: Parallel, ParallelIoServerMixin, SurfexMixIn, DrHookDecoMixin

This algo component is designed to run SURFEX experiments over large domains with MPI parallelization and IO server XIOS

class vortex_cen.algo.deterministic.Interpol_Forcing(*args, **kw)

Bases: Parallel

This algo component is designed to interpolate SAFRAN forcings on regular grid with MPI parallelization.

assim.py

Algo Components for the exectution of SODA-related tasks within a snow data assimilation workflow.

class vortex_cen.algo.assim.Soda(*args, **kw)

Bases: Parallel

worker for a SODA run (designed for Particle filtering for snow) @author: B. Cluzet 2018-05-24

class vortex_cen.algo.assim.SodaPreProcess(*args, **kw)

Bases: AlgoComponent

Prepare SODA namelist according to configuration file WARNING : this algo should not be used anymore (the preprocessing of the namelist is now done in the Soda algo). It is here only for retro-compatibility.

class vortex_cen.algo.assim.PerturbForcingComponent(*kargs, **kwargs)

Bases: _CenTaylorRun

Algo compent that creates an ensemble of forcing files by stochastic perturbations of a time series of deterministic input forcing files. Each worker deals with one single FORCING file as input (parallelisation over the different sub-periods) and one single FORCING file as output (parallelisation over the ensemble members).

class vortex_cen.algo.assim.PerturbForcingWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

Worker that applies stochastic perturbations to a FORCING file.

class vortex_cen.algo.assim.CrocOPostProcess(*kargs, **kwargs)

Bases: _CenTaylorRun

Not documented yet.

class vortex_cen.algo.assim.CrocOPostProcessWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

Worker associated to the SodaPostProcess algo component. Each worker concatenate the different PRO files covering the subperiods between assimilation dates generated by the crocO task into a single PRO file covering the period datebegin –> dateend

scores.py

Algo components for ESCROC scores.

class vortex_cen.algo.scores.Escroc_Score_Member(*args, **kwargs)

Bases: TaylorVortexWorker

AlgoComponent worker designed to run one member of SURFEX-Crocus experiment without MPI parallelization.

class vortex_cen.algo.scores.Escroc_Score_Ensemble(*kargs, **kwargs)

Bases: TaylorRun

AlgoComponent that compute ESCROC scores for the full ensemble.

class vortex_cen.algo.scores.Escroc_Score_Subensemble(*args, **kwargs)

Bases: TaylorVortexWorker

AlgoComponent worker designed to compute ensemble scores for a given subensemble.

class vortex_cen.algo.scores.Escroc_Optim_Ensemble(*kargs, **kwargs)

Bases: TaylorRun

AlgoComponent that compute ESCROC scores for the full ensemble.

postprocessing.py

Algo Components for S2M post processing.

class vortex_cen.algo.postprocessing.S2m_ensemble_postprocessing(*args, **kw)

Bases: AlgoComponent

S2M ensemble forecast postprocessing.

Current use : Get ensemble deciles of “fresh snow” (12-hourly and daily accumulation for the Bulletin 4 saisons)

class vortex_cen.algo.postprocessing.HydroWorker(*args, **kwargs)

Bases: TaylorVortexWorker

Algo that computes hydrological aggregated diagnostics after a surfex run for 1 member

class vortex_cen.algo.postprocessing.HydroComponent(*kargs, **kwargs)

Bases: TaylorRun

Algo that computes hydrological aggregated diagnostics after a surfex run for an ensemble

class vortex_cen.algo.postprocessing.ExtractDates(*kargs, **kwargs)

Bases: _CenTaylorRun

Not documented yet.

class vortex_cen.algo.postprocessing.ExtractDatesWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

class vortex_cen.algo.postprocessing.SnowCoverDuration(*kargs, **kwargs)

Bases: _CenTaylorRun

Not documented yet.

class vortex_cen.algo.postprocessing.SnowCoverDurationWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

Worker associated to the SnowCoverDuration algo component and calling various snowtools methods to compute snow cover duration diagnostics of SURFEX-Crocus simulations : * SMOD (Snow Melt Out Date) * SCD (Snow Cover Duration) * SOD (Snow onset date) * SD (Total number of days with snow)

obs.py

Algo Components for the manipulation of any observation data.

class vortex_cen.algo.obs.ReconstructObservations(*kargs, **kwargs)

Bases: _CenTaylorRun

Generate an ensemble of ReconstructObservationsWorker

class vortex_cen.algo.obs.ReconstructObservationsWorker(*args, **kwargs)

Bases: _CenTaylorVortexWorker

Update SAFRAN-ready observation files (R*, S* and T* files) with reconstructed hourly temperature observations.

prep.py

Algo component to launch the SURFEX PREP executable.

class vortex_cen.algo.prep.Prep(*args, **kw)

Bases: Parallel, SurfexMixIn

Not documented yet.

safran.py

SAFRAN related algo Components.

prosnow.py

Algo Components for the PROSNOW project

class vortex_cen.algo.prosnow.PrepareForcingComponentForecast(*kargs, **kwargs)

Bases: PrepareForcingComponent

It adapts forcing files to a ski resort geometry (several members in parallel).

This class was implemented by C. Carmagnola in May 2019 (PROSNOW project).

class vortex_cen.algo.prosnow.ExtractForcingWorker(*args, **kwargs)

Bases: PrepareForcingWorker

It adapts forcing files to a ski resort geometry (worker for 1 member).

This class was implemented by C. Carmagnola in May 2019 (PROSNOW project).

class vortex_cen.algo.prosnow.ExtractForcingWorkerEnsembleForecast(*args, **kwargs)

Bases: ExtractForcingWorker

It adapts forcing files to a ski resort geometry (worker for 1 member) with specific adaptations for short term forecast

This class was implemented by C. Carmagnola in May 2019 (PROSNOW project).

class vortex_cen.algo.prosnow.ExtractForcingWorkerLTForecast(*args, **kwargs)

Bases: ExtractForcingWorkerEnsembleForecast

It adapts forcing files to a ski resort geometry (worker for 1 member) with specific adaptations for seasonal forecasts

This class was implemented by C. Carmagnola in May 2019 (PROSNOW project).

class vortex_cen.algo.prosnow.Prosnow_Parallel(*args, **kwargs)

Bases: Surfex_Parallel

It adds snow management specificities by ski resorts to standard SURFEX-Crocus algo components.

This class was implemented by C. Carmagnola in April 2019 (PROSNOW project).