Merge pull request #1368 from NNPDF/docs_code_structure

Docs code structure

doc/sphinx/source/buildmaster.md

@@ -1,3 +1,7 @@
+```eval_rst
+.. _buildmaster:
+```
+
 ## Handling experimental data: Buildmaster
 
 Buildmaster is the code that allows the user to generate the ``DATA`` and

doc/sphinx/source/get-started/cite.rst

@@ -1,7 +1,7 @@
 .. _cite:
 
 References
-========
+==========
 If you use this code please consider citing at least the following papers:
 
 * NNPDF4.0 :cite:p:`nnpdf40`

doc/sphinx/source/get-started/index.rst

@@ -1,24 +1,18 @@
-.. _getstarted:
-
 Getting started
 ===============
-This section provides an introduction to the NNPDF code. The workflow
-for an NNPDF fit is displayed below. 
 
-.. image:: diagram.png
-  :width: 300
-  :alt: Alternative text
-
-To get started, find below the instructions on how to download and 
-install the code.
+This section provides an introduction to the NNPDF code.
+To get started, find below the instructions on how to download and
+install the code, as well as a description of the different modules that compose
+the NNPDF code.
 
 If you use the public NNPDF code, please cite all references listed on
 the :ref:`cite` page.
 
 .. toctree::
-   :maxdepth: 1
-
-   ./git.md
-   ./installation.rst
-   ./cite.rst
+  :maxdepth: 1
 
+  ./git.md
+  ./installation.rst
+  ./nnpdfmodules.rst
+  ./cite.rst

doc/sphinx/source/get-started/nnpdfmodules.rst

@@ -0,0 +1,99 @@
+Code structure
+================
+Here, we describe the structure of the NNPDF code and we present a
+high-level description of its functionalities. The workflow
+for an NNPDF fit is displayed in the figure below.
+
+.. figure:: diagram.png
+    :align: center
+    :alt: Code structure diagram
+
+    Code structure diagram
+
+
+The :ref:`APFELcomb <apfelcomb>` interpolation table generator
+--------------------------------------------------------------------------------
+This code takes hard-scattering partonic matrix element interpolators
+from  `APPLgrid <https://applgrid.hepforge.org/>`_ and
+`FastNLO <https://fastnlo.hepforge.org/>`_ (for hadronic processes) and
+`APFEL <https://apfel.hepforge.org/>`_ (for DIS structure functions) and
+combines them with the DGLAP evolution kernels provided by ``APFEL`` to
+construct the fast interpolation grids called
+FK-tables (for further instructions
+see :ref:`tutorialfktables`). In this way, physical
+observables can be evaluated in a highly efficient manner as a tensor sum of
+FK-tables with a grid of PDFs at an initial parametrisation scale :math:`Q_0`.
+``APFELcomb`` also handles NNLO QCD and/or NLO electroweak
+K-factors when needed.
+
+Theory predictions can be generated configuring a variety of options,
+such as the perturbative order (currently up to NNLO), the values of the
+heavy quark masses, the electroweak parameters, the maximum number of
+active flavours, and the variable-flavour-number scheme used to account
+for the effects of the heavy quark masses in the DIS structure functions.
+The FK-tables resulting from each choice are associated to a
+database entry trough a theory id, which allows to quickly identify them
+them.
+
+
+The :ref:`buildmaster <buildmaster>` experimental data formatter
+--------------------------------------------------------------------------------
+A C++ code which transforms the original measurements provided
+by the experimental collaborations,
+e.g. via `HepData <https://www.hepdata.net>`_,
+into a standard format that is tailored for PDF fitting.
+
+In particular, the code  allows for a flexible handling of experimental
+systematic uncertainties allowing for different treatments of the correlated
+systematic uncertainties.
+
+
+The :ref:`n3fit <n3fitindex>` fitting code
+--------------------------------------------------------------------------------
+This module implements the core fitting methodology as implemented through
+the ``TensorFlow`` framework. The ``n3fit`` library allows
+for a flexible specification of the neural network model adopted to
+parametrise the PDFs, whose settings can be selected automatically via
+the built-in :ref:`hyperoptimization algorithm <hyperoptimization>`. These
+include the neural network type and architecture, the activation
+functions, and the initialization strategy; the choice of optimizer and
+of its corresponding parameters; and hyperparameters related to the
+implementation in the fit of theoretical constraints such as PDF
+positivity and integrability. The settings for a
+PDF fit are inputted via a declarative runcard. Using these
+settings, ``n3fit`` finds the values of the neural network parameters,
+corresponding to the PDF at initial scale which describe the input data.
+Following a post-fit selection (using the ``postfit`` tool implemented
+in validphys) and PDF evolution step, the final output
+consists of an `LHAPDF <https://lhapdf.hepforge.org/>`_ grid corresponding to
+the best fit PDF as well as metadata on the fit performance.
+
+
+The :ref:`validphys <vp-index>` analysis framework
+--------------------------------------------------------------------------------
+As an implementation of the
+`reportengine <https://github.com/NNPDF/reportengine/>`_, it enables a workflow
+focused on declarative and reproducible runcards. The code implements data
+structures that can interact with those of ``libnnpdf`` and are accessible from
+the runcard. The analysis code makes heavy use of common Python Data Science
+libraries such as ``NumPy``, ``SciPy``, ``Matplotlib`` and ``Pandas``, and
+through its use of ``Pandoc`` it is capable of outputting the final results to
+HTML reports. These can be composed directly by the user or be generated by more
+specialised, downstream applications. The package includes tools to interact
+with online resources such as the results of fits or PDF grids, which, for
+example, are automatically downloaded when they are required by a runcard.
+
+
+The libnnpdf C++ legacy code
+--------------------------------------------------------------------------------
+A C++ library which contains common data structures together with
+the fitting code used to produce the NNPDF3.0 and NNPDF3.1 analyses.
+
+The availability of the ``libnnpdf`` guarantees strict backwards
+compatibility of the NNPDF framework and the ability to benchmark the
+current methodology against the previous one.
+
+To facilitate the interaction between the NNPDF C++ and Python
+codebases, we have developed Python wrappers using the ``SWIG`` library.
+
+For instructions on how to run a legacy fit, see :ref:`nnfit-usage`.

doc/sphinx/source/index.rst

@@ -4,7 +4,7 @@
    contain the root `toctree` directive.
 
 The NNPDF collaboration
-=================
+=======================
 The `NNPDF collaboration <https://nnpdf.mi.infn.it/>`_ performs research in the field of  high-energy physics. The NNPDF
 collaboration determines the structure of the proton using
 contemporary methods of artificial intelligence. A precise knowledge
@@ -15,7 +15,7 @@ the Large Hadron Collider of CERN.
 
 
 The NNPDF code
-=============
+==============
 The scientific output of the collaboration is freely available to the
 publi through the arXiv, journal repositories, and software
 repositories. Along with this online documentation, we release the
@@ -24,9 +24,9 @@ repositories. Along with this online documentation, we release the
 The code can be used to produce the ingredients needed for PDF fits, to run the fits themselves, and to analyse the results. This is the first framework used to produce a global PDF fit made publicly  available, enabling for a detailed external validation and reproducibility of the NNPDF4.0 analysis. Moreover, the code enables the user to explore a number of phenomenological applications, such as the assessment of the impact of new experimental data on PDFs, the effect of changes in theory settings on the resulting PDFs and a fast quantitative comparison between theoretical predictions and experimental data over a broad range of observables.
 
 If you are a new user head along to :ref:`getstarted` and check out the :ref:`tutorials`.
-  
+
 The NNPDF team
-=============
+==============
 The NNPDF collaboration is currently composed by the following
 members:
 
@@ -43,7 +43,7 @@ members:
 * Rabah Abdul Khalek - Nikhef Theory Group and VU University
 * José Ignacio Latorre - Quantum Research Centre, Technology
   Innovation Institute, Abu Dhabi, United Arab Emirates and Center for Quantum Technologies, National University of Singapore
-* Emanuele R. Nocera - University of Edinburgh 
+* Emanuele R. Nocera - University of Edinburgh
 * Rosalyn Pearson - University of Edinburgh
 * Juan Rojo - Nikhef Theory Group and VU University
 * Roy Stegeman - Tif Lab, Dipartimento di Fisica, Università di
@@ -68,7 +68,7 @@ Former members of the NNPDF collaboration include
 
 
 The NNPDF publications
-=====================
+======================
 * *"Future tests of parton distributions"*, Juan Cruz-Martinez, Stefano
   Forte, Emanuele R. Nocera :cite:p:`Cruz-Martinez:2021rgy`
 * *"Deuteron Uncertainties in the Determination of Proton PDFs"*,
@@ -95,7 +95,7 @@ The NNPDF publications
   Voisey and Michael Wilson  :cite:p:`AbdulKhalek:2019bux`
 * *"Nuclear Parton Distributions from Lepton-Nucleus Scattering and
   the Impact of an Electron-Ion Collider"*, Rabah Abdul Khalek,
-  Jacob J. Ethier, Juan Rojo, :cite:p:`AbdulKhalek:2019mzd` 
+  Jacob J. Ethier, Juan Rojo, :cite:p:`AbdulKhalek:2019mzd`
 * *"A First Determination of Parton Distributions with Theoretical
   Uncertainties"*, Rabah Abdul Khalek, Richard D. Ball, Stefano
   Carrazza, Stefano Forte, Tommaso Giani, Zahari Kassabov,
@@ -110,7 +110,7 @@ The NNPDF publications
   Nathan P. Hartland, Zahari Kassabov,  Jose I. Latorre,
   Emanuele R. Nocera,  Juan Rojo, Luca Rottoli, Emma Slade, and Maria Ubiali :cite:p:`Ball:2017nwa`
 
-  
+
 Contents
 ========
 .. toctree::

doc/sphinx/source/n3fit/hyperopt.rst

@@ -1,4 +1,6 @@
-================================ 
+.. _hyperoptimization:
+
+================================
 Hyperoptimization algorithm
 ================================
 
@@ -77,10 +79,10 @@ An example of a DIS fit using this loss function can be found here: [`best worst
   \begin{array}{lr}
          std(\{\chi^{2}\}) & \text{  if } avg(\chi^2) < \text{ threshold } \\
          \infty & \text{otherwise}
-  \end{array} 
+  \end{array}
   \right.
-  
-An example of a DIS fit using this loss function with the threshold :math:`\chi^2` set to 2.0 
+
+An example of a DIS fit using this loss function with the threshold :math:`\chi^2` set to 2.0
 can be found here: [`best std <https://vp.nnpdf.science/vcPtqM8KSXCVB2GheENd8Q==>`_].
 It can be selected in the runcard using the target ``std``.
 

doc/sphinx/source/n3fit/index.rst

@@ -1,23 +1,25 @@
+.. _n3fitindex:
+
 Fitting code: ``n3fit``
-===================
+=======================
 
--  ``n3fit`` is the next generation fitting code for NNPDF developed by the 
+-  ``n3fit`` is the next generation fitting code for NNPDF developed by the
    N3PDF team :cite:p:`Carrazza:2019mzf`
--  ``n3fit`` is responsible for fitting PDFs from NNPDF4.0 onwards. 
+-  ``n3fit`` is responsible for fitting PDFs from NNPDF4.0 onwards.
 -  The code is implemented in python using `Tensorflow <https://www.tensorflow.org>`_
    and `Keras <https://keras.io/>`_.
 -  The sections below are an overview of the ``n3fit`` design.
 
 ``n3fit`` design
-------------
+----------------
 .. toctree::
    :maxdepth: 1
 
    methodology
    hyperopt
    runcard_detailed
 
-.. important::   
+.. important::
     If you just want to know how to run a fit using ``n3fit``, head to :ref:`n3fit-usage`.