Dynamic Load/Propagate/Store for Data Assimilation with Particle Filters on Supercomputers - Laboratoire d'Informatique de Grenoble Accéder directement au contenu
Pré-Publication, Document De Travail (Preprint/Prepublication) Année : 2024

Dynamic Load/Propagate/Store for Data Assimilation with Particle Filters on Supercomputers

Résumé

Several ensemble-based Data Assimilation (DA) methods rely on a prop- agate/update cycle, where a potentially compute intensive simulation code propagates multiple states for several consecutive time steps, that are then analyzed to update the states to be propagated for the next cycle. In this pa- per we focus on DA methods where the update can be computed by gathering only lightweight data obtained independently from each of the propagated states. This encompasses particle filters where one weight is computed from each state, but also methods like Approximate Bayesian Computation (ABC) or Markov Chain Monte Carlo (MCMC). Such methods can be very compute intensive and running efficiently at scale on supercomputers is challenging. This paper proposes a framework based on an elastic and fault-tolerant runner/server architecture minimizing data movements while enabling dynamic load balancing. Our approach relies on runners that load, propagate and store particles from an asynchronously managed distributed particle cache permitting particles to move from one runner to another in the back- ground while particle propagation proceeds. The framework is validated with a bootstrap particle filter with the WRF simulation code. We handle up to 2,555 particles on 20,442 compute cores. Compared to a file-based implementation, our solution spends up to 2.84 less resources (cores×seconds) per particle.
Fichier principal
Vignette du fichier
main.pdf (4.36 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03927612 , version 1 (06-01-2023)
hal-03927612 , version 2 (14-02-2024)

Licence

Paternité

Identifiants

Citer

Sebastian Friedemann, Kai Keller, Yen-Sen Lu, Bruno Raffin, Leonardo Bautista Gomez. Dynamic Load/Propagate/Store for Data Assimilation with Particle Filters on Supercomputers. 2024. ⟨hal-03927612v1⟩
80 Consultations
74 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More