**CEMRACS**

#### Advanced research in scientific calculation or Centre d’Eté Mathématique de Recherche Avancée en Calcul Scientifique

#### July 17 – August 25, 2023

#### Scientific Committee

Comité scientifique

Rémi Abgrall (University of Zurich)

Patrick Galinari (Sorbonne Université)

Jan Hesthaven (École Polytechnique Fédérale de Lausanne)

Gitta Kutyniok (Ludwig Maximilian University of Munich)

Yvon Maday (Sorbonne Université)

Marc Schoenauer (Centre Inria de Saclay)

Aretha Teckentrup (University of Edinburgh)

#### Organizing Committee

Comité d’organisation

Didier Auroux (Université Côte d’Azur)

Konstantin Brenner (Université Côte d’Azur)

Martin Campos Pinto (Max Planck Institute for Plasma Physics)

Bruno Després (Sorbonne Université)

Victorita Dolean (Université Côte d’Azur)

Emmanuel Frénod (Université Bretagne Sud)

Stéphane Lanteri (Université Côte d’Azur)

Victor Michel-Dansac (Université de Strasbourg)

contact: cemracs23@smai.emath.fr

#### Program

The goal of this event is to gather scientists from both the academic and industrial communities to work and discuss on **Scientific Machine Learning**. The program includes:

- 1 week summer school (July 17-21)
- 5 weeks Hackathon on projects proposed by academic scientists and/or industrial partners (July 24 – August 25)

Participants may register to the summer school only or to the entire program. Junior participants may apply for fellowships to cover part or the whole stay.

#### Scientific Machine Learning

Scientific Machine Learning (SciML) is a relatively new research field based on both machine learning (ML) and scientific computing tools. Its aim is the development of new methods to solve several kinds of problems, which can be forward multi-dimensional partial differential equations, identification of parameters, or inverse problems. The methods we seek must be robust, reliable and interpretable. The new SciML tools allow the natural inclusion of data in the numerical simulation in order to generate new results. This new methodology will be at the forefront of the next wave of data-driven scientific discovery in the physical and engineering sciences. More precisely, our goals are:

- to introduce the mathematical concepts lying at the foundation of machine learning as used in scientific computing;
- to give an overview of the different approaches developed to address these problems;
- to give insight on the latest numerical tools in scientific machine learning and data inclusion for the solution of PDEs.

#### Summer school

The summer school is organized around five topics, each topic corresponding to one course. The format of each course is 4 hours of lectures and 2 hours of computer sessions (CS) on Jupyter Notebooks with Python or Julia (in the case of a more theoretical topic, one or two focused talks will complement the main lecture). In the computer session, the goal would be to implement with the participants an elementary working example that could potentially be of help later on in the projects. The lectures range from theoretical (approximation theory) to applied (solution of PDEs, forward and inverse problems) up to and including real-world applications.

#### Lectures

**Abstract**: Parametric PDEs arise in key applications ranging from parameter optimization, inverse state estimation, to uncertainty quantification. Accurately solving these tasks requires an efficient treatment of the resulting sets of parametric PDE solutions that are generated when parameters vary in a certain range. These solution sets are difficult to handle since their are embedded in infinite dimensional spaces, and present a complex structure. They need to be approximated with numerically efficient reduction techniques, usually called Model Order Reduction methods. The techniques need to be adapted both to the nature of the PDE, and to the given application task. In this course, we will give an overview of linear and nonlinear model order reduction methods when applied to forward and inverse problems. We will particularly emphasize on the role played by nonlinear approximation and geometrical PDE properties to address classical bottlenecks.**Tentative schedule**: (4×1.5 = 6 hours)

- Elements of approximation theory:
- Definition of linear, and nonlinear approximation
- The enconder-decoder framework
- Notions of optimal approximation performance
- Neural networks: approximation properties and open questions

- Linear and Nonlinear Schemes for Model Order Reduction:
- Parametric PDEs
- Linear MOR schemes
- Nonlinear schemes

- Linear and Nonlinear Schemes for Inverse State and Parameter Estimation:
- Optimal reconstruction algorithms
- Implementable linear and nonlinear Schemes: constructions and results on optimality

*This course will be followed by a computer session.* Tentative prerequisites: `Python`

, `NumPy`

, `PyTorch`

.

**Abstract**: In this series of lectures, I will report some recent development of the design and analysis of neural network (NN) based method, such as physics-informed neural networks (PINN) and the finite neuron method (FNM), for numerical solution of partial differential equations (PDEs). I will give an overview on convergence analysis of FNM, for error estimates (without or with numerical quadrature) and also for training algorithms for solving the relevant optimization problems. I will present theoretical results that explains the success as well as the challenges of PINN and FNM that are trained by gradient based methods such as SGD and Adam. I will then present some new classes of training algorithms that can theoretically achieve and numerically observe the asymptotic rate of the underlying discretization algorithms (while the gradient based methods cannot). Motivated by our theoretical analysis, I will finally report some competitive numerical results of CNN and MgNet using an activation function with compact support for image classifications.

*This course will be followed by a computer session.*

**Abstract**: Operators are mappings between infinite-dimensional spaces, which arise in the context of differential equations. Learning operators is challenging due to the inherent infinite-dimensional context. In this course, we present different architectures for learning operators from data. These include operator networks such as DeepONets and Neural operators such as Fourier Neural Operators (FNOs) and their variants. We will present theoretical results that show that these architectures learn operators arising from PDEs. A large number of numerical examples will be provided to illustrate them.*This course will be followed by a computer session.*

**Abstract**: High-fidelity numerical simulation of physical systems modeled by time-dependent partial differential equations (PDEs) has been at the center of many technological advances in the last century. However, for engineering applications such as design, control, optimization, data assimilation, and uncertainty quantification, which require repeated model evaluation over a potentially large number of parameters, or initial conditions, these simulations remain prohibitively expensive, even with state-of-art PDE solvers. The necessity of reducing the overall cost for such downstream applications has led to the development of surrogate models, which captures the core behavior of the target system but at a fraction of the cost. In this context, new advances in machine learning provide a new path for developing surrogates models, particularly when the PDEs are not known and the system is advection-dominated. In a nutshell, we seek to find a data-driven latent representation of the state of the system, and then learn the latent-space dynamics. This allows us to compress the information, and evolve in compressed form, therefore, accelerating the models. In this series of lectures, I will present recent advances in two fronts: deterministic and probabilistic modeling latent representations. In particular, I will introduce the notions of hyper-networks, a neural network that outputs another neural network, and diffusion models, a framework that allows us to represent probability distributions of trajectories directly. I will provide the foundation for such methodologies, how they can be adapted to scientific computing, and which physical properties they need to satisfy. Finally, I will provide several examples of applications to scientific computing.**Tentative schedule**:

- Intro to time domain problems:
- time dependent PDEs
- advection dominated problems
- Kolmogorov n-widths
- Failure of linear projections

- Introduction to HyperNetworks:
- Radiance-networks
- Hypernetworks
- Hyper encoders
- Latent Space trajectories
- Smoothness constraints

- Intro to SDEs and Diffusion models:
- Diffusion processes
- Denoising and backwards evolution
- Score-based diffusion models

- Sampling with Constraints:
- Tweedies’ formula
- Linear constraints and probabilistic approximations
- Non-linear constraints and extreme events

*This course will be followed by a computer session.* Tentative prerequisites: `Python`

, `JAX`

, `Flax`

.

**Abstract**: In this talk, I will give an overview of recent successes (and some failures) of combining modern, high order discretization schemes with machine learning submodels and their applications for large scale computations. The primary focus will be on supervised learning strategies, where a multivariate, non-linear function approximation of given data sets is found through a high-dimensional, non-convex optimization problem that is efficiently solved on modern GPUs. This approach can thus for example be employed in cases where current submodels in the discretization schemes currently rely on heuristic data. A prime of example of this is shock detection and shock capturing for high order methods, where essentially all known approaches require some expert user knowledge as guiding input. As an illustrative example, I will show how modern, multiscale neural network architectures originally designed for image segmentation can ameliorate this problem and provide parameter free and grid independent shock front detection on a subelement level. With this information, we can then inform a high order artificial viscosity operator for inner-element shock capturing. In the second part of my talk, I will present data-driven approaches to LES modeling for implicitly filtered high order discretizations. Whereas supervised learning of the Reynolds force tensor based on non-local data can provide highly accurate results that provide higher a priori correlation than any existing closures, a posteriori stability remains an issue. I will give reasons for this and introduce reinforcement learning as an alternative optimization approach. Our experiments with this method suggest that is it much better suited to account for the uncertainties introduced by the numerical scheme and its induced filter form on the modeling task. For this coupled RL-DG framework, I will present discretization-aware model approaches for the LES equations and discuss the future potential of these solver-in-the-loop optimizations.**Tentative schedule**:

- Deep Neural Networks and Supervised Learning for augmenting PDEs:
- Basic and Advanced Architectures for Supervised Learning
- Incorporating constraints
- Examples: Combining CFD and ML for solving real-world problems

- Data-Driven Models for Large Eddy Simulation of Turbulence:
- Large Eddy Simulation, turbulence and numerics
- Subgrid scale modeling with ML
- Model-Data Consistency

- Deep Reinforcement Learning for turbulence:
- Introduction to RL
- Optimizable PDE solvers on HPC systems
- Deep RL for turbulence modelling

*This course will be followed by a practical session*, where we will investigate model training for canonical turbulent flows through supervised learning and explore methods to incorporate physical constraints into the ML-based models. Tentative prerequisites: `Python`

, `NumPy`

, `TensorFlow`

.

Full version in pdf

Andrea Beck’s research works

#### Schedule

Monday 17/07 | Tuesday 18/07 | Wednesday 19/07 | Thursday 20/07 | Friday 21/07 | |
---|---|---|---|---|---|

09:00 – 10:30 | Lecture 1 (O. Mula) | Lecture 2 (J. Xu) | Lecture 3 (S. Mishra) | Lecture 4 (L. Zepeda-Núñez) | Lecture 5 (A. Beck) |

10:30 – 10:45 | coffee break | coffee break | coffee break | coffee break | coffee break |

10:45 – 12:15 | Lecture 1 (O. Mula) | Lecture 2 (J. Xu) | Lecture 3 (S. Mishra) | Lecture 4 (L. Zepeda-Núñez) | Lecture 5 (A. Beck) |

12:15 – 14:00 | lunch break | lunch break | lunch break | lunch break | lunch break |

14:00 – 15:00 | Lecture 1 (O. Mula) | Lecture 2 (J. Xu) | Lecture 3 (S. Mishra) | Lecture 4 (L. Zepeda-Núñez) | Lecture 5 (A. Beck) |

15:00 – 15:30 | coffee break | coffee break | coffee break | coffee break | coffee break |

15:30 – 17:30 | Computer session | Computer session | Computer session | Computer session | Computer session |

#### Slides

- Slides for the first course are available on Olga Mula’s homepage.

- Slides for the second course are available here:

- Slides for the third course are available by following this link.

- Slides for the fourth course are available by following this link.

- Slides for the fifth course are available by following this link.

#### SPONSORS

* *