
\documentclass[11pt]{report}

\usepackage{graphicx}
\usepackage{pgf-pie}
\usepackage{currfile}
\usepackage{tikz}
\usepackage{pgfplots}
\usepackage{pgfplotstable}
\usepackage{underscore}
\usepackage{amsfonts}
\usepackage{hyperref}
\usepackage{xurl}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{float}
\usepackage[parfill]{parskip}

\setlength{\parindent}{0pt}
\usepgfplotslibrary{fillbetween}

\pgfplotsset{compat=newest}

\begin{document}



\chapter{2025-12-11 13:58:08}


\section{Benchmark Execution Summary}


\subsection{Session}


\begin{itemize}
  \item Hostname: gaya

  \item User: gaya

  \item Time Start: 20251211T135811+0100

  \item Time End: 20251212T002724+0100

\end{itemize}





\subsection{Cases}


\begin{itemize}
  \item Total: 14 

  \item Failures: 2

  \item Runs: 1

\end{itemize}





\subsection{Parametrization}




\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{\begin{tabular}{ c l l l l l l }
    \hline
 & Hash & tasks & discretization & Total Time (s) &  &  \\
    \hline
🟢 & 0c0d9679 & 128 & P1 & 2630.907 & link:descriptions/0c0d9679.html[Description] & link:logs/0c0d9679.html[Logs] \\
🟢 & 1aad25f9 & 8 & P1 & 1058.260 & link:descriptions/1aad25f9.html[Description] & link:logs/1aad25f9.html[Logs] \\
🟢 & 1c58710c & 128 & P2 & 2844.602 & link:descriptions/1c58710c.html[Description] & link:logs/1c58710c.html[Logs] \\
🟢 & 2bee4b08 & 64 & P1 & 2353.527 & link:descriptions/2bee4b08.html[Description] & link:logs/2bee4b08.html[Logs] \\
🟢 & 54cc6124 & 32 & P2 & 2305.390 & link:descriptions/54cc6124.html[Description] & link:logs/54cc6124.html[Logs] \\
🟢 & 6016d433 & 256 & P1 & 37474.588 & link:descriptions/6016d433.html[Description] & link:logs/6016d433.html[Logs] \\
🟢 & 71659d70 & 32 & P1 & 1989.690 & link:descriptions/71659d70.html[Description] & link:logs/71659d70.html[Logs] \\
🟢 & 77eb0048 & 16 & P1 & 1457.834 & link:descriptions/77eb0048.html[Description] & link:logs/77eb0048.html[Logs] \\
🟢 & 888f7b72 & 64 & P2 & 2567.229 & link:descriptions/888f7b72.html[Description] & link:logs/888f7b72.html[Logs] \\
🟢 & a285793a & 16 & P2 & 1907.830 & link:descriptions/a285793a.html[Description] & link:logs/a285793a.html[Logs] \\
🟢 & ac8f4cc2 & 4 & P1 & 320.882 & link:descriptions/ac8f4cc2.html[Description] & link:logs/ac8f4cc2.html[Logs] \\
🔴 & bbf416a6 & 4 & P2 & 870.761 & link:descriptions/bbf416a6.html[Description] & link:logs/bbf416a6.html[Logs] \\
🔴 & f4c5efcd & 8 & P2 & 1373.874 & link:descriptions/f4c5efcd.html[Description] & link:logs/f4c5efcd.html[Logs] \\
🟢 & fb46b4da & 256 & P2 & 37751.945 & link:descriptions/fb46b4da.html[Description] & link:logs/fb46b4da.html[Logs] \\
    \hline
    \end{tabular}}
\end{table}







\section{Benchmark: Thermo-Electric Coupling}


\subsection{Description}

This benchmark models the temperature field and electric current distribution in an high field resistive magnet of the Laboratoire National des Champs Magnétiques Intenses. The magnet consist in a set of 14 copper alloys cylindrical tubes connected 2 by 2 into series by rings. In each tube, the current path is defined by 2 helical cuts of 0.2 mm width. The rings are machined to let water flow in between each tube in a channel of 0.8 mm. The magnet is operated at 12 MW with an imosed total current of 31 kA. The water flow in the magnet is about 140 l/s. The water cooling of the magnet is modelling by using Robin boundary conditions with parameters derived from classical correlation in thermo-hydraulics. A more detailled version of the full model is available in (Daversin Catty, 2016; Hild, 2020). The model is run with thermoelectric Feel++ toolbox. The geometry used in this benchmark performance is illustrated in Figure 4.13. This is a complex domain composed of a large number of components, with some very thin parts. 



\begin{figure}[ht!]
\centering

\begin{minipage}[t]{ 0.500\linewidth }

\includegraphics[width=\linewidth]{ ./data/f3a00492834245f984ee90e44787b33c/feelpp-benchmark-HL-31-geo.png }
\subcaption{Figure 4.13: HL-31 benchmarks - geometry}
\end{minipage}%
\hfill
\begin{minipage}[t]{ 0.500\linewidth }

\includegraphics[width=\linewidth]{ ./data/f3a00492834245f984ee90e44787b33c/feelpp-benchmark-HL-31-geo-zoom.png }
\subcaption{Figure 4.13: HL-31 benchmarks - geometry}
\end{minipage}%


\end{figure}





\subsection{Benchmarking Tools Used}

This benchmark was done using feelpp.benchmarking, version \textbf{{feelpp_benchmarking_version}}

The benchmark was performed on the gaya supercomputer (see Section 10.1). The performance tools integrated into the Feel++-toolboxes framework were used to measure the execution time. Moreover, we need to say that we have used several Feel++ installations.


\begin{itemize}
  \item \textbf{Gaya}:native application from Ubuntu packages of Jammy OS.

  \item \textbf{Discoverer}: Apptainer with Feel++ SIF image based on Ubuntu Noble OS.

\end{itemize}


Note: the Feel++ version is identical but the dependencies (like Petsc) which are of course more recent with Noble.

The metrics measured are the execution time of the main components of the simulation. We enumerate these parts in the following:


\begin{itemize}
  \item \textbf{Init}: load mesh from filesystem and initialize solid toolbox (finite element context and algebraic data structure)

  \item \textbf{Assembly}: calculate and assemble the matrix and rhs values obtained using the finite element method

  \item \textbf{Solve}: the linear system by using a preconditioned GMRES.

  \item \textbf{PostProcess}: Export on the filesystem a visualization format (EnsighGold) of the solution and other fields of interest such as current density and electric field.

\end{itemize}





\subsection{Input/Output Dataset Description}


\paragraph{ Input Data }


\begin{itemize}
  \item Meshes: We have generated one mesh called M1. These meshes are stored in GMSH format. The statistics can be found in Table 4.6. We have also prepared for each mesh level a collection of partitioned mesh. The format used is an in-house mesh format of Feel++ based on JSON+HDF5 file type. The Gmsh meshes and the partitioned meshes can be found on our Girder database management, in the Feel++ collections.

  \item Setup: Use standard setup of Feel++ toolboxes. It corresponds to a cfg file and JSON file. These config files are present in the Github of feelpp.

  \item SIF image (Apptainer): feelpp:v0.111.0-preview.10-noble-sif (stored in the Githubregistry of Feel++)

  \item Ubuntu package Jammy (Native) feelpp:v0.111.0-preview.10

\end{itemize}





\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{\begin{tabular}{ l l l l l l l l l }
    \hline
Tag & \# points & \# edges & \# faces & \# elements & P1 (Heat) & P2 (Heat) & P1 (electric) & P2 (electric) \\
    \hline
M1 & 4.31E+06 & 2.79E+07 & 4.60E+07 & 2.24E+07 & 4.31E+06 & 3.22E+07 & 4.30E+06 & 2.95E+07 \\
    \hline
    \end{tabular}}
    \caption{ HL-31 benchmark - Statistics on meshes and number of degrees of freedom with respect to finite element approximation }
\end{table}





\paragraph{ Output Data }

The output includes export visualization files (mesh, partitioning, temperature, elecric potential, current density, electric field, ...), the time taken to perform each simulation step and some integral quantities such a the total power dissipated by the magnet.




\paragraph{ Data Repository }

All inputdatasets are available in a unistra girder repository (collection HiFiMagnet, HL-31).






\subsection{Results Summary}

We can find in fig. 4.14 some examples of 3D visualization that represent fields obtained by solving the thermoelectric problem. We have the temperature solution and other quantities evaluated at PostProcess phase such as current density, electric potential, and electric fields.



\begin{figure}[ht!]
\centering

\begin{minipage}[t]{ 0.500\linewidth }

\includegraphics[width=\linewidth]{ ./data/f3a00492834245f984ee90e44787b33c/feelpp-benchmark-HL-31-temperature.png }
\subcaption{(a) Temperature}
\end{minipage}%
\hfill
\begin{minipage}[t]{ 0.500\linewidth }

\includegraphics[width=\linewidth]{ ./data/f3a00492834245f984ee90e44787b33c/feelpp-benchmark-HL-31-current_density.png }
\subcaption{(b) Current density}
\end{minipage}%


\end{figure}




\begin{figure}[ht!]
\centering

\begin{minipage}[t]{ 0.500\linewidth }

\includegraphics[width=\linewidth]{ ./data/f3a00492834245f984ee90e44787b33c/feelpp-benchmark-HL-31-potential_density_streamines.png }
\subcaption{(c) Electric potential and current density on streamlines of electric field}
\end{minipage}%
\hfill
\begin{minipage}[t]{ 0.500\linewidth }

\includegraphics[width=\linewidth]{ ./data/f3a00492834245f984ee90e44787b33c/feelpp-benchmark-HL-31-temperature-streamlines.png }
\subcaption{(d) Temperature on streamlines of electric field}
\end{minipage}%


\end{figure}


The performance analysis has been done in two supercomputers, Gaya and Discoverer. For each of them, we have run the benchmark with discretizations P1 and P2. About the Feel++ distribution, we use native Ubuntu Jammy packages for Gaya, and Apptainer (with Ubuntu) for Discoverer. The results on these machines are illustrated in figs. 4.15 and 4.16 respectively. The measures presented are the time-consuming of simulation components related to WP1, namely Init, Assembly, and PostProcess. For the Solve component, we refer to section 6.2.3.3.

In each case analyzed, we have very good scalability, in the sense that we reduced significantly the execution time by increasing the number of tasks. The Init phase is the most timeconsuming, with a limit quite high compared to others. This part contains the mesh reading from the disk, which can explain some bumps sometimes when require many HPC resources.

To conclude, we can say that we successfully ran this benchmark case, and we are ready to go up to larger problems and more complex modeling.




\subsection{Challenges Identified}

HPC experiments with other kinds of discretizations


\begin{itemize}
  \item Modelling: nonlinear, Maxwell, ...

  \item HDG method

  \item Geometry complexity

\end{itemize}


Go to extreme HPC scale: Partitioning and IO issues




\subsection{Results}


\paragraph{ HL-31 benchmarks - Execution time of main simulation components }





\DeclareRobustCommand{\plot}[2][]{
  \begin{tikzpicture}
    \begin{axis}[
      width=0.95\textwidth, height=\linewidth,
      xlabel={ Number of CPU core }, ylabel={ execution time (s) },
      xtick=data, xtick align=outside,
      ylabel style={font=\scriptsize},
      xlabel style={font=\scriptsize},
      ymajorgrids=true, yminorgrids=true,
      xticklabels from table={#2}{tasks},
      xticklabel style={rotate=45, anchor=east, font=\scriptsize},
      yticklabel style={font=\scriptsize},
      cycle list name=color list,
      legend to name=customlegend,
      legend style={font=\small}
    ]
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Create Exporters] {#2} ;
      \addlegendentry{ Create Exporters }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Create Mesh] {#2} ;
      \addlegendentry{ Create Mesh }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Export] {#2} ;
      \addlegendentry{ Export }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Init] {#2} ;
      \addlegendentry{ Init }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Solve] {#2} ;
      \addlegendentry{ Solve }


    \end{axis}
  \end{tikzpicture}
}


\pgfplotstableread[col sep=comma]{\detokenize{./data/f3a00492834245f984ee90e44787b33c/88f93b4749c8489cb432d201cb8a9039/scatter/P1.csv}}\dataA
\pgfplotstableread[col sep=comma]{\detokenize{./data/f3a00492834245f984ee90e44787b33c/88f93b4749c8489cb432d201cb8a9039/scatter/P2.csv}}\dataB


\begin{figure}[H]
  \centering

      \begin{subfigure}[t]{ 0.475\textwidth} % adjust width as needed
        \centering
        \plot{\dataA}
        \caption{ Discretization=P1 }
      \end{subfigure}
      \begin{subfigure}[t]{ 0.475\textwidth} % adjust width as needed
        \centering
        \plot{\dataB}
        \caption{ Discretization=P2 }
      \end{subfigure}
        \par\vspace{1em}
  \vspace{1em}
  \centering
  \pgfplotslegendfromname{customlegend}
  \caption{ Performance by CPU tasks and Discretization }
\end{figure}






\DeclareRobustCommand{\plot}[2][]{
  \begin{tikzpicture}
    \begin{axis}[
      width=0.95\textwidth, height=\linewidth,
      xlabel={ Number of CPU core }, ylabel={ Speedup },
      xtick=data, xtick align=outside,
      ylabel style={font=\scriptsize},
      xlabel style={font=\scriptsize},
      ymajorgrids=true, yminorgrids=true,
      xticklabels from table={#2}{tasks},
      xticklabel style={rotate=45, anchor=east, font=\scriptsize},
      yticklabel style={font=\scriptsize},
      cycle list name=color list,
      legend to name=customlegend,
      legend style={font=\small}
    ]
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Create Exporters] {#2} ;
      \addlegendentry{ Create Exporters }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Create Mesh] {#2} ;
      \addlegendentry{ Create Mesh }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Export] {#2} ;
      \addlegendentry{ Export }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Init] {#2} ;
      \addlegendentry{ Init }
      \addplot+[mark=*,mark size=0.7pt] table [x expr=\coordindex, y=Solve] {#2} ;
      \addlegendentry{ Solve }

      \addplot[black, name path=optimal] table [x expr=\coordindex, y=optimal] {#2} ;
      \addlegendentry{ optimal }
      \addplot[black, name path=half-optimal] table [x expr=\coordindex, y=half-optimal] {#2} ;
      \addlegendentry{ half-optimal }
        \addplot[black, fill opacity=0.2] fill between[of=optimal and half-optimal];

    \end{axis}
  \end{tikzpicture}
}


\pgfplotstableread[col sep=comma]{\detokenize{./data/f3a00492834245f984ee90e44787b33c/61f63808b50f490c8799f3a0d4722e6a/scatter/P1.csv}}\dataA
\pgfplotstableread[col sep=comma]{\detokenize{./data/f3a00492834245f984ee90e44787b33c/61f63808b50f490c8799f3a0d4722e6a/scatter/P2.csv}}\dataB


\begin{figure}[H]
  \centering

      \begin{subfigure}[t]{ 0.475\textwidth} % adjust width as needed
        \centering
        \plot{\dataA}
        \caption{ Discretization=P1 }
      \end{subfigure}
      \begin{subfigure}[t]{ 0.475\textwidth} % adjust width as needed
        \centering
        \plot{\dataB}
        \caption{ Discretization=P2 }
      \end{subfigure}
        \par\vspace{1em}
  \vspace{1em}
  \centering
  \pgfplotslegendfromname{customlegend}
  \caption{ Speedup of main simulation components }
\end{figure}





\paragraph{ Solver Metrics }










\end{document}