Skip to content

Commit

Permalink
Edit feature guide analysis tool pages.
Browse files Browse the repository at this point in the history 
Signed-off-by: Dave Welsch <[email protected]>
  • Loading branch information
@dwelsch-esi
dwelsch-esi committed Jan 22, 2025
1 parent 6f4f408 commit 289dfb0
Show file tree
Hide file tree
Showing 4 changed files with 218 additions and 172 deletions.
25 changes: 11 additions & 14 deletions Docs/featureguide/analysis tools/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,20 @@ Analysis tools
Quantization analyzer <quant_analyzer>
Layer output generation <layer_output_generation>

AIMET offers these tools to view and analyze a model's interal quantization results.

:ref:`Interactive visualization <featureguide-interactive-visualization>`
-------------------------------------------------------------------------
Interactive visualization
-------------------------

Produces an interactive HTML to view the statistics collected by each quantizer during calibration.
:ref:`Interactive visualization <featureguide-interactive-visualization>` produces an interactive HTML console showing the statistics collected by each quantizer during calibration.

:ref:`Quantization analyzer <featureguide-quant-analyzer>`
----------------------------------------------------------
Quantization analyzer
---------------------

QuantAnalyzer analyzes your pre-trained model and points out sensitive layers to quantization
in the model. It checks model sensitivity to weight and activation quantization, performs per
layer sensitivity and MSE analysis. It also exports per layer encodings min and max ranges and
statistics histogram for every layer.
:ref:`Quantization analyzer <featureguide-quant-analyzer>` (QuantAnalyzer) analyzes your pre-trained model and identifies layers sensitive to quantization. It checks model sensitivity to weight and activation quantization, and performs per-layer sensitivity and mean square error analysis. It also exports per-layer encoding min and max ranges and
statistics histograms for every layer.

:ref:`Layer output generation <featureguide-layer-output-generation>`
---------------------------------------------------------------------
Layer output generation
-----------------------

This API captures and saves intermediate layer-outputs of a model. This allows layer-output
comparison between quantization simulated model (QuantSim object) and actually
quantized model on target-device to debug accuracy miss-match issues at the layer level.
:ref:`Layer output generation <featureguide-layer-output-generation>` is an API that captures and saves intermediate layer model outputs. This allows layer-output comparison between a quantization simulated model (QuantSim object) and an actual quantized model on a target device in order to debug accuracy mismatch issues at the layer level.
21 changes: 15 additions & 6 deletions Docs/featureguide/analysis tools/interactive_visualization.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,13 @@ Interactive visualization
Context
=======

Creates an interactive visualization of min and max activations/weights of all quantized modules
in the Quantization simulation :class:`QuantizationSimModel` object.
Interactive visualization displays the range (min and max values) of activations and weights for all quantized modules
in the quantization simulation :class:`QuantizationSimModel` object.

The features include:
Interactive visualization functionality includes:

- Adjustable threshold values to flag layers whose min or max activations/weights exceed the set thresholds

- Tables containing names and ranges for layers exceeding threshold values.
- Adjustable threshold values to flag layers for which min or max activations or weights exceed these values
- Tables containing names and ranges for layers exceeding threshold values


Workflow
Expand All @@ -33,3 +32,13 @@ API

.. include:: ../../apiref/torch/interactive_visualization.rst
:start-after: # start-after

.. tab-item:: TensorFlow
:sync: tf

Interactive visualization does not support TensorFlow.

.. tab-item:: ONNX
:sync: onnx

Interactive visualization does not support ONNX.
125 changes: 78 additions & 47 deletions Docs/featureguide/analysis tools/layer_output_generation.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,33 @@ Layer output generation
Context
=======

This API captures and saves intermediate layer-outputs of your pre-trained model. The model
can be original (FP32) or :class:`QuantizationSimModel`.
Layer output generation is an API that captures and saves intermediate layer-outputs of your pre-trained model. The model
can be original (FP32) or a :class:`QuantizationSimModel`.

The layer-outputs are named according to the exported PyTorch/ONNX/TensorFlow model by the
The layer outputs are named according to the exported model (PyTorch, ONNX, or TensorFlow) by the
QuantSim export API :func:`QuantizationSimModel.export`.

This allows layer-output comparison amongst quantization simulated model (QuantSim)
and actually quantized model on target-runtimes like |qnn|_ to debug accuracy miss-match
This enables layer output comparison between quantization simulated (QuantSim) models
and quantized models on target runtimes like |qnn|_ to debug accuracy mismatch
issues at the layer level (per operation).

Workflow
========

Code example
------------
The layer output generation framework follows the same workflow for all model frameworks:

Step 1 Obtain Original or QuantSim model from AIMET Export Artifacts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Imports
2. Load a model from AIMET
3. Obtain inputs
4. Generate layer outputs


Choose your framework below for code examples.

Step 1: Importing the API
-------------------------

Import the API.

.. tab-set::
:sync-group: platform
Expand All @@ -39,13 +48,6 @@ Step 1 Obtain Original or QuantSim model from AIMET Export Artifacts
:start-after: # Step 0. Import statements
:end-before: # End step 0

**Obtain Original or QuantSim model from AIMET Export Artifacts**

.. literalinclude:: ../../legacy/torch_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 1. Obtain original or quantsim model
:end-before: # End step 1

.. tab-item:: TensorFlow
:sync: tf

Expand All @@ -54,13 +56,6 @@ Step 1 Obtain Original or QuantSim model from AIMET Export Artifacts
:start-after: # Step 0. Import statements
:end-before: # End step 0

**Obtain Original or QuantSim model from AIMET Export Artifacts**

.. literalinclude:: ../../legacy/keras_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 1. Obtain original or quantsim model
:end-before: # End step 1

.. tab-item:: ONNX
:sync: onnx

Expand All @@ -69,70 +64,106 @@ Step 1 Obtain Original or QuantSim model from AIMET Export Artifacts
:start-after: # Step 0. Import statements
:end-before: # End step 0

**Obtain Original or QuantSim model from AIMET Export Artifacts**

Step 2: Loading a model
-----------------------

Export the original or QuantSim model from AIMET.

.. tab-set::
:sync-group: platform

.. tab-item:: PyTorch
:sync: torch

.. literalinclude:: ../../legacy/torch_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 1. Obtain original or quantsim model
:end-before: # End step 1

.. tab-item:: TensorFlow
:sync: tf

.. literalinclude:: ../../legacy/keras_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 1. Obtain original or quantsim model
:end-before: # End step 1

.. tab-item:: ONNX
:sync: onnx

.. literalinclude:: ../../legacy/onnx_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 1. Obtain original or quantsim model
:end-before: # End step 1

Step 2 Generate layer-outputs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Step 3: Obtaining inputs
------------------------

Obtain inputs from which to generate intermediate layer outputs.

.. tab-set::
:sync-group: platform

.. tab-item:: PyTorch
:sync: torch

**Obtain inputs for which we want to generate intermediate layer-outputs**

.. literalinclude:: ../../legacy/torch_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 2. Obtain pre-processed inputs
:end-before: # End step 2

**Generate layer-outputs**

.. literalinclude:: ../../legacy/torch_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 3. Generate outputs
:end-before: # End step 3

.. tab-item:: TensorFlow
:sync: tf

**Obtain inputs for which we want to generate intermediate layer-outputs**

.. literalinclude:: ../../legacy/keras_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 2. Obtain pre-processed inputs
:end-before: # End step 2

**Generate layer-outputs**

.. literalinclude:: ../../legacy/keras_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 3. Generate outputs
:end-before: # End step 3

.. tab-item:: ONNX
:sync: onnx

**Obtain inputs for which we want to generate intermediate layer-outputs**

.. literalinclude:: ../../legacy/onnx_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 2. Obtain pre-processed inputs
:end-before: # End step 2

**Generate layer-outputs**

Step 4: Generating layer outputs
--------------------------------

Generate the specified layer outputs.

.. tab-set::
:sync-group: platform

.. tab-item:: PyTorch
:sync: torch

.. literalinclude:: ../../legacy/torch_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 3. Generate outputs
:end-before: # End step 3

.. tab-item:: TensorFlow
:sync: tf

.. literalinclude:: ../../legacy/keras_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 3. Generate outputs
:end-before: # End step 3

.. tab-item:: ONNX
:sync: onnx

.. literalinclude:: ../../legacy/onnx_code_examples/layer_output_generation_code_example.py
:language: python
:start-after: # Step 3. Generate outputs
:end-before: # End step 3


API
===

Expand Down
Loading

0 comments on commit 289dfb0

Please sign in to comment.