Constant data managment with User Environments

The general documentation of the uenv/uget tool is available here:

https://cnrm-gmap.gitlab.meteo.fr/vortex-gco/index.html

There is a list of useful commands at the end of this section.

The main motivations to use User Environments are : * allowing for reliable, standardized data archiving * ensuring the simulations reproductibility * facilitating and optimizing the exchange of data between users * partitioning the data used in different simulations in a clear and secure way * minimizing the number of circulating files

Note

The source code is available on git.meteo.fr : https://git.meteo.fr/cnrm-gmap/vortex/-/tree/main/vortex-gco?ref_type=heads

Install the vortex-gco plugin

On MF HPC:

pip install --find-links /home/verolive/wheels vortex-gco

On any other MF server:

pip install vortex-gco

Definition of a user environment

User environments refer to user-managed collections of coherent static data files. A user environment is the combination of: * an ensemble of static data files, potentially owned by different users * a catalogue text file associating a unique key (named “gvar”) to identify each file. This catalogue file is refered to as “uenv” in this documentation.

Create a User Environment from scratch

Step by step

  1. Put the file(s) you need in your UEnv in $HOME/.vortexrc/hack/uget/your_username/data

Name your file(s) with a distinctive suffix (version number, git commit or tag,…) : AFirstFile.0, ASecondFile.1

Note

Namelists are a special case : serveral namelist files can be added under $HOME/.vortexrc/hack/uget/your_username/data/<namelist_dir_XXX>

  1. Create a text file with a distinctive name which is the name of your UEnv (ex MyFirstUenv.0) in $HOME/.vortexrc/hack/uget/your_username/env

  2. In this text file, associate a key to each file:

FIRST_KEY="uenv:AFirstFile.0@your_username"
SECOND_KEY="uenv:ASecondFile.1@your_username"

Note

For namelists, the line looks like : NAMELISTS_<MODEL>=”uenv:namelist_dir_XXX@your_username”

  1. UEnv is ready, you can access to file(s) with the command:

toolbox.input(genv='uenv:MyFirstUenv.0@your_username', gvar='FIRST_KEY', unknown=True, filename='...')
toolbox.input(genv='uenv:MyFirstUenv.0@your_username', gvar='SECOND_KEY', unknown=True, filename='...')

Note

For namelists, the additional namelist_source entry must be provided to specify the target namelist’s name from the pool of namelists: toolbox.input(genv=’uenv:MyFirstUenv.0@your_username’, gvar=’SECOND_KEY’, unknown=True, filename=’…’, namelist_source=<name_of_namelist>)

  1. In order to archive and share your UEnv :

uget push env MyFirstUenv.0@your_username

NB: On HPC this can only be done from a TRANSFERT node

Note

  • By convention, the “keys” of SURFEX executables are MASTER_SURFEX_<NAME_OF_EXEC>_MPI or MASTER_SURFEX_<NAME_OF_EXEC>_NOMPI.

  • It is possible to use files from another user’s UEnv in your own UEnv : simply copy the corresponding lines from their UEnv file into yours

User Environments for surfex simulations

Constant files for SURFEX simulations can be grouped in 3 different categories:

  • files that do not change frequently, for which a default uenv is commonly used (eccoclimap*, drdt_bst_fit, SandDB, ClayDB). The associated default uenv for these files can be set with the “consts_surfex_uenv” configuration variable.

  • SURFEX executables can be provided in a specific uenv identified by the “surfex_uenv” configuration variable.

  • SURFEX namelists can be provided in a specific uenv identified by the “namelists_surfex_env” configuration variables.

Note

Of course it is possible to group all these constant files into a single user environment and simply use the “uenv” configuration variable, following the instructions provided in the next section

Example : create a UEnv with SURFEX executables and namelists

Modifying an existing UEnv

1. Create a copy of existing UEnv:

uget hack env Existing_UEnv.0@username_uenv_owner into MyNewUenv.0@your_username

Now, A text file MyNewUenv.0 has been creates in $HOME/.vortexrc/hack/uget/your_username/env

NB: On HPC this can only be done from a TRANSFERT node

Example:

ssh belenostransfert
source path/to/venv/with/vortex-gco/bin/activate
uget hack env Existing_UEnv.0@username_uenv_owner into MyNewUenv.0@your_yousername

Note

A copy of an existing UEnv should always have a different name than the original one to avoid overwriting it. The good practice is to increment a version number or give a new explicit name.

2. Modify new UEnv MyNewUenv.0:

  1. Upgrade a file already in UEnv

–> Identify the file and key in MyNewUenv.0 (ex: SSA_PARAMS=”uenv:drdt_bst_fit_60.nc@CONST_CEN”)

—> Place new version of your file in $HOME/.vortexrc/hack/uget/your_username/data (ex: drdt_bst_fit_60_new.nc)

–> Modify corresponding line in file MyNewUenv.0 (ex: SSA_PARAMS=”uenv:drdt_bst_fit_60_new.nc@your_username”)

  1. Add a new file in MyNewUenv.0

–> Place the file New_File.0 in $HOME/.vortexrc/hack/uget/your_username/data

—> Add a line to the file MyNewUenv.0 (ex: NEW_KEY=”uenv:New_File.0@your_username”)

Concatenation of 2 existing UEnv

  1. Get both UEnv:

uget hack env UEnv1@user_who_own_this_uenv into UEnv1_copy@your_username
uget hack env UEnv2@maybe_another_user into UEnv2_copy@your_username

Now, the files UEnv1_copy and UEnv2_copy are copied in $HOME/.vortexrc/hack/uget/your_username/env

  1. Concatenate both files in a new UEnv UEnv1_UEnv2:

cat UEnv1_copy UEnv2_copy > UEnv1_UEnv2

NB: it is of course posible to just pick few lines of each files

Using UEnv TargetUEnv.X owned by another user

  1. Explore TargetUEnv.X:

uget check env TargetUEnv.X@the_other_user

  1. Get the file of interest with his key Key_from_TargetUEnv.X:

toolbox.input(genv='uenv:TargetUEnv.X@the_other_user', gvar='Key_from_TargetUEnv.X', unknown=True, filename='...')

Archiving a User Environment

Once you uenv is ready, check that all associated data files exist with :

uget check env <target_uenv_name>@<username>

The archive your uenv with :

uget push env <target_uenv_name>@<username>

NB : This “freezes” your uenv and makes it available to other users (just as a “git push”), from now on you should not modify anything from this uenv ! You should also remove the local copies to avoid inconsistencies between local and archived versions:

uget clean_hack

Note

To ensure that archived files are not modified, their archive location is voluntarily “encrypted”

If you detect a mistake in one of the files associated to your uenv at this point, create a new version of your uenv with :

uget hack env <former_uenv_name>@<username> into <new_uenv_name>@<your_username>

Note

A coment line at the begining of the <new_uenv_name> catalogue file is added to track its provenance

and create the new version of the faulty file, if necessary by “hacking” the previous one :

uget hack data <faulty_data_name>@<username> into <new_data_name>@<your_username>

Do not forget to change the line corresponding to the faulty data file in the <new_uenv_name> catalogue :

FAULTY_DATA_KEY="uget:<new_data_name>@<username>"

instead of :

FAULTY_DATA_KEY="uget:<faulty_data_name>@<username>"

Note

  • The “FAULTY_DATA_KEY” remains unchanged between the two versions

  • No need to change anything else, all other data files remain properly identified in the <new_uenv_name>

Data tree

When retrieved with Vortex, data from a uenv can come from :

  • The ‘hack’ data tree under $HOME/.vortexrc/hack/uget/<username>. This data tree should be used only during the creation and test of a new uenv.

  • The ‘Archive’ date tree. This is the data tree from which valid and shared uenv data are retrieved, potentially from different users.

  • The ‘local’ data tree. This is a local copy of the ‘Archive’ date tree, which is created when data is first fetched in order to act as a cache for future use.

Note

Fetching data from the “Archive” or “Local” data trees is transparent to the user in most cases. The only exceptions are linked to issues with transfers from the archive (passord updates, call from a login or compute nodes on HPC,…).

Examples of available User Environments (UEnv)

Started by Ange Haddjeri, to be continued…

Digital elevations models available in “uenv:dem.2@vernaym”:

DEM_ALP1KM_EPSG4326
DEM_FRANCE25M_L93
DEM_GRANDESROUSSES25M_L93
DEM_PYR1KM_EPSG4326
RELIEF_FRANGP0025
RELIEF_GRANDESROUSSES250M_4326
RELIEF_GRANDESROUSSES250M_L93

Shapefiles availables in “uenv:shapefiles.1@vernaym”:

FRENCH_CITIES
MASSIFS_SAFRAN
WORLD_BOUNDARIES

Uenv to reproduce simulations from M.Vernay PhD : “edelweiss_gr250_pappus.2@vernaym”

Uenv for latest version of the ANTILOPE post-processing algorithm : “edelweiss.3@vernaym”

UEnv vs GEnv

Constant files used in operationnal simulations are managed by the GCO team with the “genv” equivalent of user environments. To access data from an operationnal “genv”, simply provide the name of the genv without the ‘uenv:’ prefix nor the ‘@<username>’ suffix:

uenv = cen01_cen@s2m-op2.13

identifies an opertional genv, whereas

uenv = uenv:cen01_cen@s2m-op2.13@vernaym

identifies the uenv of user “vernaym”.

Usefull commands

List available user environments for a given user :

uget list env from <username>

Check data availability for a specific uenv :

uget check env <uenv_name>@<username>

This returns the location of the catalogue file (“Hack” and/or “Archive”), as well as the list of associated files and their location.

Note

The location can be “Hack” (developments in progress) and/or Archive (target already pushed).

Cloning an existing uenv catalogue file:

uget hack env <target_uenv_name>@<username> into <new_uenv_name>@<your_username>

Cloning an existing genv catalogue file:

uget hack genv <target_genv_name> into <new_uenv_name>@<your_username>

Note

It is recommended to clone an existing uenv / genv rather than to create a new one from scratch whenever possible because it adds a comment to track its provenance.

Cloning an existing data file:

uget hack data <target_data_name>@<username> into <new_data_name>@<your_username>

Cloning an existing operational data file:

uget hack gdata <target_data_name> into <new_data_name>@<your_username>

Commit / push a uenv and all its associated data files :

uget push env <target_uenv_name>@<username>

NB : once a uenv catalogue file and its associated data files have been pushed, remove the local copies to avoid inconsistencies between local and archived versions:

uget clean_hack

Commit / push a specific data file :

uget push data <target_data_name>@<username>

NB :

Compare two catalogues (uenv and/or genv alike), or a uenv catalogue with its “parent” in case it has been properly “hacked” :

uget diff env <first_uenv_name>@<username> wrt env <second_uenv_name>@<username>
uget diff env <uenv_name>@<username> wrt genv <genv_name>
uget diff env <uenv_name>@<username> wrt parent

Get the list of data files that differ between two catalogues:

uget export env <first_uenv_name>@<username> wrt env <second_uenv_name>@<username>
uget export env <uenv_name>@<username> wrt genv <genv_name>

FAQ - Frequent issues

  • On HPC, fetch/save data from a TRANSFERT node because the uget tool is based on ftget/ftput commands, which are unavailable on login/compute nodes and raise the following error:

# [2026/04/21-12:40:16][vortex.tools.systems][spawn:1120][CRITICAL]: Could not call ['ftput', '-o', 'mkdir', '-q', '-h', 'hendrix.meteo.fr', '-u', 'vernaym', 'myuenv', 'uget/env/6/myuenv']
Traceback (most recent call last):
[...]
FileNotFoundError: [Errno 2] No such file or directory: 'ftput'

NAMELIST_SURFEX="uget:namelists_surfex9_0_crocus3_0_2_std.tar@lafaysse"

whereas

NAMELIST_SURFEX="uget:namelists_surfex9_0_crocus3_0_2_std@lafaysse"

raises the following error :

# [2026/08/17-12:36:48][vortex.data.abstractstores][incacheget:1180][INFO]: incacheget on uget://uget.hack.fr//data/namelists_surfex9_0_crocus3_0_2_std (to: namelists_surfex9_0_crocus3_0_2_std)
# [2026/08/17-12:36:48][vortex.data.abstractstores][incacheget:1199][INFO]: incacheget retrieve rc=True location=/home/cnrm_other/cen/mrns/vernaym/.vortexrc/hack/uget/vernaym/data/namelists_surfex9_0_crocus3_0_2_std
# [2026/08/17-12:36:48][vortex_gco.data.stores][ugetget:1664][ERROR]: 'namelists_surfex9_0_crocus3_0_2_std' should be a tarfile

  • The lines in uenv catalogue file are sensitive to trailing spaces. The following line (containing a final space character) :

MASTER_OFFLINE_MPI="uget:ms_offline.cen.mpi.11.crocus3.0.2@vernaym" |<-- The line ends here

will lead to a non-explicit error:

# [2026/09/14-12:59:32][vortex_gco.data.stores][_gspawn:0627][INFO]: gget command: /home/mf/dp/marp/gco/public/bin/gget -host hendrix.meteo.fr -path uget:ms_offline.cen.nompi.11.crocus3.0.2@vernaym"
# [2026/09/14-12:59:32][vortex.tools.systems][spawn:1160][WARNING]: Bad return code [1] for ['/home/mf/dp/marp/gco/public/bin/gget', '-host', 'hendrix.meteo.fr', '-path', 'uget:ms_offline.cen.nompi.11.crocus3.0.2@vernaym" ']
[ERROR] [GGet::Element::new] ms_offline.cen.nompi.11.crocus3.0.2@vernaym" : can't guess type of element
# [2026/09/14-12:59:32][vortex.layout.dataflow][_fatal_wrap:0248][ERROR]: Resource ???
# [2026/09/14-12:59:32][vortex_gco.data.stores][_gspawn:0627][INFO]: gget command: /home/mf/dp/marp/gco/public/bin/gget -host hendrix.meteo.fr -path uget:ms_offline.cen.nompi.11.crocus3.0.2@vernaym"
# [2026/09/14-12:59:33][vortex.tools.systems][spawn:1160][WARNING]: Bad return code [1] for ['/home/mf/dp/marp/gco/public/bin/gget', '-host', 'hendrix.meteo.fr', '-path', 'uget:ms_offline.cen.nompi.11.crocus3.0.2@vernaym" ']
[ERROR] [GGet::Element::new] ms_offline.cen.nompi.11.crocus3.0.2@vernaym" : can't guess type of element
# [2026/09/14-12:59:33][vortex.layout.dataflow][_fatal_wrap:0250][CRITICAL]: Fatal error with action get on ???

Note

A ticket has be submited on sept. 14 2026 to fix this issue in vortex-gco (https://git.meteo.fr/cnrm-gmap/vortex/-/work_items/4)