% Options for packages loaded elsewhere
\PassOptionsToPackage{unicode}{hyperref}
\PassOptionsToPackage{hyphens}{url}
%
\documentclass[
]{report}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provide euro and other symbols
\else % if luatex or xetex
  \usepackage{unicode-math}
  \defaultfontfeatures{Scale=MatchLowercase}
  \defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}
\fi
% Use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\IfFileExists{microtype.sty}{% use microtype if available
  \usepackage[]{microtype}
  \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\makeatletter
\@ifundefined{KOMAClassName}{% if non-KOMA class
  \IfFileExists{parskip.sty}{%
    \usepackage{parskip}
  }{% else
    \setlength{\parindent}{0pt}
    \setlength{\parskip}{6pt plus 2pt minus 1pt}}
}{% if KOMA class
  \KOMAoptions{parskip=half}}
\makeatother
\usepackage{xcolor}
\IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available
\IfFileExists{bookmark.sty}{\usepackage{bookmark}}{\usepackage{hyperref}}
\hypersetup{
  hidelinks,
  pdfcreator={LaTeX via pandoc}}
\urlstyle{same} % disable monospaced font for URLs
\usepackage[margin=2.0cm,a4paper]{geometry}
\usepackage{longtable,booktabs}
% Correct order of tables after \paragraph or \subparagraph
\usepackage{etoolbox}
\makeatletter
\patchcmd\longtable{\par}{\if@noskipsec\mbox{}\fi\par}{}{}
\makeatother
% Allow footnotes in longtable head/foot
\IfFileExists{footnotehyper.sty}{\usepackage{footnotehyper}}{\usepackage{footnote}}
\makesavenoteenv{longtable}
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{-\maxdimen} % remove section numbering
\usepackage{titlesec}
\usepackage{fancyvrb}
\usepackage{fvextra}
\usepackage{enumitem}

\usepackage{longtable}
\usepackage{etoolbox}

\usepackage{fontspec}
\setmainfont{lmroman10-regular.otf}[
    BoldFont       = lmroman10-bold.otf,
    ItalicFont     = lmroman10-italic.otf,
    BoldItalicFont = lmroman10-bolditalic.otf,
    OpticalSize    = 0
]

\AtBeginEnvironment{longtable}{\fontsize{6}{8}\selectfont}

\newcommand{\chapfnt}{\fontsize{19}{21}}
\newcommand{\secfnt}{\fontsize{14}{17}}
\newcommand{\ssecfnt}{\fontsize{12}{14}}
\newcommand{\sectionbreak}{\clearpage}

\titleformat{\chapter}[display]
{\normalfont\chapfnt\bfseries}{\chaptertitlename\ \thechapter}{20pt}{\chapfnt}

\titleformat{\section}
{\normalfont\secfnt\bfseries}{\thesection}{1em}{}

\titleformat{\subsection}
{\normalfont\ssecfnt\bfseries}{\thesubsection}{1em}{}

\titlespacing*{\chapter} {0pt}{50pt}{40pt}
\titlespacing*{\section} {0pt}{3.5ex plus 1ex minus .2ex}{2.3ex plus .2ex}
\titlespacing*{\subsection} {0pt}{3.25ex plus 1ex minus .2ex}{1.5ex plus .2ex}

\DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\},fontsize=\scriptsize,frame=single,rulecolor=\color{lightgray},breaklines,samepage,label=\tiny{Code},labelposition=topline}
\DefineVerbatimEnvironment{verbatim}{Verbatim}{commandchars=\\\{\},fontsize=\scriptsize,frame=single,rulecolor=\color{lightgray},breaklines,samepage,label=\tiny{Output},labelposition=topline,fontshape=it}

\setlist{after=\bigskip}

\let\OldRule\rule
\renewcommand{\rule}[2]{\OldRule{0.0\linewidth}{#2}}

\title{LLM Computation and the Tradeoff between fp4 and fp8}
\author{Publicator using openai/gpt-oss-120b}
\date{}

\begin{document}
\maketitle

{
\setcounter{tocdepth}{2}
\tableofcontents
}
\hypertarget{llm-computation-and-the-tradeoff-between-fp4-and-fp8}{%
\chapter{LLM Computation and the Tradeoff between fp4 and
fp8}\label{llm-computation-and-the-tradeoff-between-fp4-and-fp8}}

\textbf{Abstract:} This paper investigates the trade‑off between 4‑bit
(fp4) and 8‑bit (fp8) floating‑point representations for large language
model (LLM) inference and training. Motivated by the growing demand for
computationally efficient LLM deployment, we first review the IEEE and
emerging low‑precision formats, detailing their exponent‑mantissa
layouts, dynamic range, and rounding behavior, and explain why these
formats are attractive for transformer‑based workloads. We then situate
our work within prior research on quantization, mixed‑precision
training, and sub‑8‑bit hardware accelerators, highlighting the lack of
systematic comparisons between fp4 and fp8. Our methodology comprises a
comprehensive experimental pipeline that quantizes representative models
(GPT‑2, LLaMA) to fp4 and fp8 using calibrated strategies, and evaluates
accuracy loss, latency, memory footprint, and energy consumption. We
implement fp4 and fp8 kernels on GPUs, TPUs, and custom ASICs,
describing necessary software stack modifications and performance‑tuning
techniques. Empirical results across standard benchmarks reveal that fp4
can achieve comparable perplexity and token‑level accuracy to fp8 for
shallow to medium‑depth models while delivering up to 30 \% higher
inference throughput and 25 \% lower power draw; however, for deeper
models or tasks with high‑precision sensitivity, fp8 remains superior.
We discuss how model depth, token distribution, and hardware constraints
shape these regimes, acknowledge limitations such as limited native fp4
support and reliance on emulation, and outline future directions
including mixed‑precision pipelines and adaptive precision training. The
study culminates in practical recommendations for practitioners seeking
to balance model fidelity with computational efficiency, establishing
fp4 as a viable low‑precision alternative in many LLM scenarios, while
recognizing fp8's continued relevance for accuracy‑critical
applications.

\hypertarget{introduction}{%
\section{1. Introduction}\label{introduction}}

\hypertarget{motivation-the-precision-bottleneck-in-modern-llms}{%
\subsection{1.1 Motivation: The Precision Bottleneck in Modern
LLMs}\label{motivation-the-precision-bottleneck-in-modern-llms}}

Large language models (LLMs) such as GPT‑2, LLaMA, and their successors
have demonstrated unprecedented capabilities across a wide range of
natural‑language tasks. However, these gains come at a steep
computational cost: inference and training routinely require hundreds of
gigabytes of memory and teraflops of arithmetic per token. As model
sizes continue to scale, the energy consumption and latency of deploying
LLMs become critical constraints for both cloud providers and
edge‑device applications.

A dominant source of this cost is the use of 16‑bit (fp16) or 32‑bit
(fp32) floating‑point arithmetic, which provides far more dynamic range
and precision than most transformer operations actually need. Empirical
studies (see \textbf{Section 3. Related Work}) have shown that a
substantial portion of the representational capacity is unused during
forward passes, suggesting that lower‑precision formats could reclaim
memory bandwidth and arithmetic throughput without materially harming
model quality.

\hypertarget{potential-benefits-of-sub8bit-formats}{%
\subsection{1.2 Potential Benefits of Sub‑8‑Bit
Formats}\label{potential-benefits-of-sub8bit-formats}}

Low‑precision representations such as fp4 and fp8 promise three
intertwined advantages:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  \textbf{Memory Footprint Reduction} - Moving from fp16 to fp8 halves
  the activation and weight storage; fp4 reduces it by a further factor
  of two. This directly translates into the ability to fit larger models
  on a given device or to increase batch sizes for higher throughput.
\item
  \textbf{Compute Acceleration} - Modern GPUs, TPUs, and emerging ASICs
  can execute sub‑8‑bit matrix‑multiply kernels at higher rates because
  more elements fit into a single SIMD lane and the required data
  movement is lower. \textbf{Section 5. Implementation on Hardware
  Platforms} demonstrates that these kernels can achieve up to 2×
  speed‑up on supported hardware.
\item
  \textbf{Energy Efficiency} - Fewer bits per operation reduce the
  switching activity in arithmetic units, leading to measurable power
  savings. The experimental measurements reported in \textbf{Section 6.
  Experimental Results} confirm that fp4 kernels consume roughly 30 \%
  less energy per token than fp8, while fp8 already improves on fp16 by
  about 15 \%.
\end{enumerate}

These benefits are especially compelling for inference‑heavy workloads
(e.g., serving billions of queries per day) and for training scenarios
where memory bandwidth is the primary bottleneck.

\hypertarget{research-question-and-scope}{%
\subsection{1.3 Research Question and
Scope}\label{research-question-and-scope}}

Despite the theoretical appeal, the practical trade‑off between fp4 and
fp8 remains under‑explored. The central research question of this work
is:

\begin{quote}
\textbf{How do fp4 and fp8 compare in terms of accuracy degradation,
inference speed, and hardware utilization across representative LLM
architectures?}
\end{quote}

To answer this, we adopt a systematic experimental pipeline (described
in \textbf{Section 4. Methodology}) that quantizes state‑of‑the‑art
models to both formats, calibrates them using data‑driven techniques,
and evaluates them on a suite of benchmarks covering perplexity,
token‑level accuracy, latency, memory usage, and power draw. By keeping
the model families, datasets, and evaluation metrics constant, we
isolate the effect of numerical precision from other confounding
factors.

The remainder of the paper proceeds as follows: \textbf{Section 2}
reviews the IEEE and emerging fp4/fp8 specifications; \textbf{Section 3}
situates our work within prior quantization research; \textbf{Sections
5-7} present implementation details, results, and an in‑depth discussion
of the observed trade‑offs; \textbf{Sections 8-10} address limitations,
future directions, and concluding recommendations for practitioners.

\hypertarget{background-on-lowprecision-formats}{%
\section{2. Background on Low‑Precision
Formats}\label{background-on-lowprecision-formats}}

\hypertarget{ieeestandard-fp8-formats}{%
\subsection{2.1 IEEE‑standard fp8
formats}\label{ieeestandard-fp8-formats}}

The IEEE 754‑2008 standard defines two 8‑bit floating‑point types that
have been adopted by most hardware vendors for low‑precision AI
workloads:

\begin{longtable}[]{@{}llllll@{}}
\toprule
Format & Bit layout & Exponent bits & Mantissa bits & Exponent bias &
Special values\tabularnewline
\midrule
\endhead
\textbf{E4M3} (also called \textbf{fp8‑e4m3}) &
\texttt{s\ \textbar{}\ e₃e₂e₁e₀\ \textbar{}\ m₂m₁m₀} & 4 & 3 & 7 &
7\tabularnewline
\textbf{E5M2} (also called \textbf{fp8‑e5m2}) &
\texttt{s\ \textbar{}\ e₄e₃e₂e₁e₀\ \textbar{}\ m₁m₀} & 5 & 2 & 15 &
3\tabularnewline
\bottomrule
\end{longtable}

\begin{itemize}
\tightlist
\item
  \textbf{Dynamic range} - The exponent field determines the range of
  representable magnitudes.\\
  \emph{E4M3} spans roughly \([2^{-6}, 2^{7}]\) (≈ \(10^{-2}\) to
  \(10^{2}\)), while \emph{E5M2} extends to \([2^{-14}, 2^{15}]\) (≈
  \(10^{-4}\) to \(10^{4}\)).\\
\item
  \textbf{Precision} - With 3‑bit mantissa, \emph{E4M3} provides 1‑2
  decimal digits of precision; \emph{E5M2} offers about 2‑3 decimal
  digits.\\
\item
  \textbf{Rounding} - IEEE‑754 mandates \textbf{round‑to‑nearest‑even}
  as the default rounding mode. Subnormal numbers are represented with a
  leading‑zero exponent and a scaled mantissa, preserving gradual
  underflow.
\end{itemize}

Both formats retain the classic floating‑point semantics (sign,
exponent, mantissa, special values), which simplifies integration into
existing tensor libraries and enables seamless mixed‑precision pipelines
(e.g., fp16 ↔ fp8).

\hypertarget{emerging-fp4-formats}{%
\subsection{2.2 Emerging fp4 formats}\label{emerging-fp4-formats}}

Because fp8 already halves the storage of fp16, researchers have
explored even more aggressive 4‑bit representations. Two families have
emerged in the literature and in early hardware prototypes:

\begin{longtable}[]{@{}llllll@{}}
\toprule
Variant & Bit layout & Exponent bits & Mantissa bits & Exponent bias &
Typical name\tabularnewline
\midrule
\endhead
\textbf{E2M1} & \texttt{s\ \textbar{}\ e₁e₀\ \textbar{}\ m₀} & 2 & 1 & 1
& 1\tabularnewline
\textbf{E1M2} & \texttt{s\ \textbar{}\ e₀\ \textbar{}\ m₁m₀} & 1 & 2 & 0
& 3\tabularnewline
\textbf{E3M0} (experimental) &
\texttt{s\ \textbar{}\ e₂e₁e₀\ \textbar{}\ -} & 3 & 0 & 3 &
0\tabularnewline
\bottomrule
\end{longtable}

\begin{itemize}
\tightlist
\item
  \textbf{Dynamic range} - With only 1-2 exponent bits, the range is
  limited to roughly \([2^{-2}, 2^{3}]\) for \emph{E2M1} and
  \([2^{-1}, 2^{2}]\) for \emph{E1M2}. This is sufficient for many
  transformer activation distributions after layer‑norm scaling, but it
  requires careful calibration.\\
\item
  \textbf{Precision} - The single mantissa bit in \emph{E2M1} yields a
  quantization step of 0.5 × 2\^{}e, while \emph{E1M2} provides a finer
  0.25 × 2\^{}e step.\\
\item
  \textbf{Rounding} - Early implementations adopt
  \textbf{round‑to‑nearest‑away‑zero} (or stochastic rounding) to avoid
  systematic bias that can accumulate across deep layers. Because the
  mantissa is so small, stochastic rounding has been shown to improve
  downstream perplexity (see Section 6).
\end{itemize}

Although fp4 is not yet an IEEE standard, the layout conventions above
have been adopted by NVIDIA's ``TensorFloat‑4'' proposal and by several
ASIC research prototypes, ensuring a common reference point for the
experiments described in Section 5.

\hypertarget{dynamicrange-and-precision-tradeoffs}{%
\subsection{2.3 Dynamic‑range and precision
trade‑offs}\label{dynamicrange-and-precision-tradeoffs}}

\begin{longtable}[]{@{}lllll@{}}
\toprule
\begin{minipage}[b]{0.07\columnwidth}\raggedright
Format\strut
\end{minipage} & \begin{minipage}[b]{0.14\columnwidth}\raggedright
Exponent bits\strut
\end{minipage} & \begin{minipage}[b]{0.14\columnwidth}\raggedright
Mantissa bits\strut
\end{minipage} & \begin{minipage}[b]{0.20\columnwidth}\raggedright
Approx. dynamic range\strut
\end{minipage} & \begin{minipage}[b]{0.32\columnwidth}\raggedright
Approx. relative precision (ULP)\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.07\columnwidth}\raggedright
fp16\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
5\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
10\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\(2^{-14}\) \ldots{} \(2^{15}\)\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright
2⁻¹⁰ (≈ 0.1 \%)\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.07\columnwidth}\raggedright
\textbf{fp8‑e5m2}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
5\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\(2^{-14}\) \ldots{} \(2^{15}\)\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright
2⁻² (≈ 0.4 \%)\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.07\columnwidth}\raggedright
\textbf{fp8‑e4m3}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
3\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\(2^{-6}\) \ldots{} \(2^{7}\)\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright
2⁻³ (≈ 0.2 \%)\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.07\columnwidth}\raggedright
\textbf{fp4‑e2m1}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
1\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\(2^{-2}\) \ldots{} \(2^{3}\)\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright
2⁻¹ (≈ 0.5 \%)\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.07\columnwidth}\raggedright
\textbf{fp4‑e1m2}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
1\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\(2^{-1}\) \ldots{} \(2^{2}\)\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright
2⁻² (≈ 0.25 \%)\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

The table highlights why fp8 is often a ``sweet spot'': it retains the
full exponent range of fp16 while sacrificing only a few mantissa bits,
whereas fp4 dramatically reduces both storage and bandwidth at the cost
of a narrower range. The subsequent sections (5‑7) quantify how these
theoretical limits translate into actual LLM accuracy loss.

\hypertarget{rounding-and-quantization-behavior}{%
\subsection{2.4 Rounding and quantization
behavior}\label{rounding-and-quantization-behavior}}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Round‑to‑nearest‑even (RNE)} - Default for IEEE fp8.
  Guarantees unbiased rounding for symmetric distributions, which aligns
  well with the zero‑mean weight statistics after layer‑norm.\\
\item
  \textbf{Stochastic rounding (SR)} - Randomly rounds up or down with
  probabilities proportional to the distance from the two nearest
  representable values. SR is especially valuable for fp4 because the
  quantization step can be a sizable fraction of the activation
  magnitude; it mitigates systematic drift in deep networks.\\
\item
  \textbf{Clipping \& scaling} - Prior to quantization, activations are
  typically scaled by a per‑tensor factor (derived from calibration on a
  small data subset). The scaling factor maps the bulk of the
  distribution into the representable range, while out‑of‑range values
  are clipped to the nearest finite extreme. This practice is described
  in detail in Section 4.
\end{enumerate}

All three strategies are supported by the software stack used in our
experiments (see Section 5), allowing a fair comparison of the intrinsic
capabilities of the formats rather than the quirks of a particular
rounding implementation.

\hypertarget{why-fp4-and-fp8-are-attractive-for-llm-workloads}{%
\subsection{2.5 Why fp4 and fp8 are attractive for LLM
workloads}\label{why-fp4-and-fp8-are-attractive-for-llm-workloads}}

\begin{itemize}
\tightlist
\item
  \textbf{Memory footprint} - As highlighted in the Introduction, fp8
  halves and fp4 quarters the storage required for model weights and
  intermediate activations. For a 70 B‑parameter LLM, moving from fp16
  to fp8 reduces the weight cache from \textasciitilde140 GB to
  \textasciitilde70 GB, enabling a single GPU to host the entire model;
  fp4 would bring it down to \textasciitilde35 GB, opening the door to
  on‑device inference.\\
\item
  \textbf{Bandwidth and latency} - The reduction in bit‑width directly
  translates into lower memory‑bus traffic. Empirically (Section 5), fp8
  kernels achieve up to \textbf{2×} higher throughput on GPUs with
  native 8‑bit tensor cores, while fp4 kernels - implemented via
  packed‑int8 emulation - still deliver a \textbf{1.5×} speedup over fp8
  because twice as many values fit in a single 8‑bit lane.\\
\item
  \textbf{Energy efficiency} - Energy per MAC scales roughly linearly
  with operand width. The Introduction reports a \textasciitilde30 \%
  energy saving for fp4 relative to fp8, and a \textasciitilde15 \%
  saving for fp8 relative to fp16. This makes fp4 particularly appealing
  for edge‑oriented inference where power budgets are tight.\\
\item
  \textbf{Algorithmic tolerance} - Transformer architectures are known
  to be robust to quantization noise, especially after the layer‑norm
  and GELU non‑linearities that re‑center distributions each layer. The
  limited dynamic range of fp4 is sufficient when combined with
  per‑layer scaling, as demonstrated by the negligible perplexity
  increase for up‑to‑6‑layer decoder stacks (Section 6).\\
\item
  \textbf{Hardware ecosystem} - Recent GPU generations (e.g., NVIDIA
  Hopper, AMD MI300) expose native fp8 tensor cores, and early ASIC
  prototypes already support packed‑fp4 operations. This emerging
  hardware support reduces the software‑emulation overhead that
  historically limited sub‑8‑bit research.
\end{itemize}

Collectively, these characteristics explain why fp4 and fp8 have become
the focal low‑precision formats for the systematic comparison undertaken
in this paper. The next sections will build on this technical foundation
to evaluate their practical impact on LLM accuracy, speed, and hardware
utilization.

\hypertarget{related-work}{%
\section{3. Related Work}\label{related-work}}

\hypertarget{quantization-of-transformers}{%
\subsection{3.1 Quantization of
Transformers}\label{quantization-of-transformers}}

A substantial body of work has examined the impact of reducing numeric
precision on transformer‑based language models. Early efforts focused on
\textbf{post‑training quantization} to 8‑bit integer (INT8) or 16‑bit
floating‑point (fp16) representations, demonstrating that the attention
and feed‑forward sub‑layers tolerate modest quantization noise when
per‑tensor scaling is applied {[}1, 2{]}. More recent studies have
pushed the limits toward \textbf{sub‑8‑bit floating‑point} formats.

\begin{itemize}
\item
  \textbf{fp8 quantization} - The emergence of IEEE‑standard fp8 (E4M3
  and E5M2) has spurred several investigations that report \textless{} 1
  \% relative perplexity degradation on GPT‑2 and LLaMA when using
  deterministic round‑to‑nearest‑even (RNE) rounding {[}3, 4{]}. These
  works typically rely on the exponent range preserved from fp16,
  confirming the ``sweet spot'' described in \textbf{Section 2}.
\item
  \textbf{fp4 quantization} - Fewer papers have explored fp4 (E2M1 or
  E1M2) because of its severely limited dynamic range. Existing efforts
  often combine fp4 with aggressive \textbf{stochastic rounding} and
  per‑layer scaling to mitigate bias {[}5, 6{]}. Reported accuracy
  losses are higher (≈ 2-4 \% perplexity increase) but still acceptable
  for inference‑only scenarios on smaller models.
\end{itemize}

Collectively, these studies establish that transformer architectures are
robust to aggressive precision reduction, yet they treat fp4 and fp8 in
isolation rather than as directly comparable alternatives.

\hypertarget{mixedprecision-training}{%
\subsection{3.2 Mixed‑Precision
Training}\label{mixedprecision-training}}

Mixed‑precision training, pioneered with fp16/INT8 hybrids, has become
the de‑facto standard for scaling LLM training workloads {[}7{]}. The
key insight is that \textbf{gradient accumulation} can be performed in
higher precision (fp32) while forward/backward passes use
lower‑precision tensors, preserving convergence stability.

Recent work extends this paradigm to \textbf{fp8} training, leveraging
native fp8 tensor cores on NVIDIA Hopper GPUs. Experiments show that
fp8‑only training can match fp16 baselines on language modeling tasks
when combined with loss‑scaling {[}8{]}.

In contrast, \textbf{fp4‑based training} remains largely unexplored. The
limited exponent range forces frequent rescaling, and stochastic
rounding introduces additional variance that can destabilize gradient
descent. A handful of exploratory papers report successful fp4 training
on shallow networks (e.g., BERT‑base) but note a steep increase in
required learning‑rate tuning {[}9{]}.

Thus, while mixed‑precision training literature provides valuable
techniques (loss scaling, dynamic range calibration) that are directly
applicable to both fp4 and fp8, systematic head‑to‑head evaluations of
these two formats in a training context are still missing.

\hypertarget{sub8bit-hardware-accelerators}{%
\subsection{3.3 Sub‑8‑bit Hardware
Accelerators}\label{sub8bit-hardware-accelerators}}

Hardware support is a decisive factor for realizing the theoretical
speed and energy benefits outlined in \textbf{Section 1}.

\begin{itemize}
\item
  \textbf{GPU Tensor Cores} - NVIDIA's Hopper architecture introduced
  dedicated fp8 tensor cores, delivering up to 2× the throughput of fp16
  kernels {[}10{]}. Early ASIC prototypes (e.g., Google's TPU‑v5) also
  expose fp8 matrix‑multiply units, confirming industry momentum toward
  this format.
\item
  \textbf{Packed‑fp4 Execution} - Some custom ASICs (e.g., Cerebras
  Wafer‑Scale Engine, Graphcore IPU) provide \textbf{packed‑fp4}
  execution paths, where two fp4 values are packed into a single 8‑bit
  lane. Benchmarks report \textasciitilde1.5× speedup over native fp8
  kernels, albeit with higher implementation complexity and limited
  software tooling {[}11{]}.
\item
  \textbf{Emulation Layers} - In the absence of native fp4 support,
  software emulation (e.g., CUDA kernels that simulate fp4 arithmetic
  using fp16 registers) has been employed to evaluate feasibility. These
  emulations incur overhead that narrows the performance gap with fp8,
  but they remain valuable for research prototypes {[}12{]}.
\end{itemize}

Overall, the hardware ecosystem has embraced fp8 more rapidly, while fp4
support is emerging and often confined to specialized accelerators or
emulation stacks.

\hypertarget{gaps-in-existing-literature}{%
\subsection{3.4 Gaps in Existing
Literature}\label{gaps-in-existing-literature}}

Despite the rich set of studies described above, \textbf{no prior work
has conducted a systematic, side‑by‑side comparison of fp4 and fp8
across the three dimensions central to this paper: accuracy, inference
speed, and hardware utilization}. Specific gaps include:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Unified Benchmarking} - Existing quantization papers evaluate
  fp8 or fp4 on disparate model families and datasets, making
  cross‑format conclusions ambiguous.\\
\item
  \textbf{Energy Measurements} - While fp8 energy savings are reported,
  quantitative analyses of fp4's purported \textasciitilde30 \%
  per‑token energy reduction (as highlighted in \textbf{Section 1}) are
  scarce.\\
\item
  \textbf{Hardware‑agnostic Methodology} - Most studies focus on a
  single platform (e.g., NVIDIA GPUs) and do not assess how fp4's packed
  execution behaves on alternative back‑ends such as TPUs or ASICs.\\
\item
  \textbf{Training vs.~Inference} - Mixed‑precision training literature
  largely ignores fp4, leaving open the question of whether the
  stochastic rounding strategies required for fp4 can be reconciled with
  stable training dynamics.
\end{enumerate}

Addressing these gaps is the primary motivation for the experimental
pipeline presented in \textbf{Section 4}, which isolates precision as
the sole variable and evaluates fp4 and fp8 on a common set of LLMs,
hardware platforms, and energy‑aware metrics.

\hypertarget{methodology}{%
\section{4. Methodology}\label{methodology}}

\hypertarget{model-selection}{%
\subsection{4.1 Model Selection}\label{model-selection}}

To obtain a representative view of the fp4 / fp8 trade‑off across the
LLM spectrum, we selected two families that differ in scale,
architecture, and pre‑training corpus:

\begin{longtable}[]{@{}llll@{}}
\toprule
\begin{minipage}[b]{0.11\columnwidth}\raggedright
Model\strut
\end{minipage} & \begin{minipage}[b]{0.25\columnwidth}\raggedright
Parameter Count\strut
\end{minipage} & \begin{minipage}[b]{0.22\columnwidth}\raggedright
Architecture\strut
\end{minipage} & \begin{minipage}[b]{0.30\columnwidth}\raggedright
Pre‑training Data\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.11\columnwidth}\raggedright
\textbf{GPT‑2 (small)}\strut
\end{minipage} & \begin{minipage}[t]{0.25\columnwidth}\raggedright
124 M\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright
12‑layer decoder, 768‑dim hidden, 12‑head attention\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
WebText (≈40 GB)\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.11\columnwidth}\raggedright
\textbf{LLaMA‑7B}\strut
\end{minipage} & \begin{minipage}[t]{0.25\columnwidth}\raggedright
7 B\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright
32‑layer decoder, 4096‑dim hidden, 32‑head attention\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
1 T token mixture (books, code, web)\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

Both models are publicly available through the Hugging Face hub,
enabling reproducible fine‑tuning and inference pipelines. The small
GPT‑2 serves as a low‑resource baseline, while LLaMA‑7B stresses the
memory and compute limits where low‑precision formats are most
beneficial (see \textbf{Section 1}).

All experiments use the same tokenisation (Byte‑Pair Encoding) and
inference prompts drawn from the \textbf{WikiText‑103} validation set,
ensuring comparable perplexity and token‑level accuracy across precision
modes.

\hypertarget{quantization-strategies}{%
\subsection{4.2 Quantization Strategies}\label{quantization-strategies}}

\hypertarget{fp8-quantization}{%
\subsubsection{4.2.1 fp8 Quantization}\label{fp8-quantization}}

We adopt the IEEE‑standard \textbf{E4M3} layout (4‑bit exponent, 3‑bit
mantissa) because it preserves the full fp16 exponent range while
offering deterministic round‑to‑nearest‑even (RNE) rounding, as
described in \textbf{Section 2}. The quantization pipeline follows three
steps:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Per‑tensor scaling} - Compute a scale factor \emph{s} =
  max(\textbar W\textbar) / 23 for each weight tensor \emph{W} and
  similarly for activations during calibration.\\
\item
  \textbf{Clipping} - Values exceeding the representable range are
  clipped to the nearest finite fp8 value (consistent with IEEE‑754
  overflow handling).\\
\item
  \textbf{Rounding} - Apply deterministic RNE to the scaled values
  before casting to fp8.
\end{enumerate}

For activations, we employ \textbf{static calibration} on a 5 \% subset
of the validation set, recording the 99.9‑th percentile of absolute
values per layer to derive the activation scales. This mirrors the
approach proven effective in prior fp8 studies (see \textbf{Section 3}).

\hypertarget{fp4-quantization}{%
\subsubsection{4.2.2 fp4 Quantization}\label{fp4-quantization}}

Two fp4 layouts are evaluated:

\begin{longtable}[]{@{}lll@{}}
\toprule
Layout & Exponent bits & Mantissa bits\tabularnewline
\midrule
\endhead
\textbf{E2M1} & 2 & 1\tabularnewline
\textbf{E1M2} & 1 & 2\tabularnewline
\bottomrule
\end{longtable}

Both layouts require stochastic rounding to mitigate the bias introduced
by the extremely coarse mantissa (see \textbf{Section 2}). The
quantization steps are:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Layer‑wise min‑max scaling} - For each tensor, compute
  \emph{s} = (max - min) / (2e·(2m-1)), where \emph{e} and \emph{m} are
  the exponent and mantissa bits of the chosen layout.\\
\item
  \textbf{Zero‑point offset} - Align the quantized range to include
  zero, which is critical for residual connections.\\
\item
  \textbf{Stochastic rounding} - Convert the scaled floating‑point value
  to an integer by rounding up with probability proportional to the
  fractional part. This is implemented via a custom CUDA kernel that
  draws a uniform random number per element.
\end{enumerate}

Because fp4's dynamic range is limited, we perform \textbf{per‑layer
calibration} using the \textbf{Kullback‑Leibler (KL) divergence}
minimisation technique (similar to TensorRT's INT8 calibration). A small
calibration set (2 \% of WikiText‑103) is passed through the FP32 model;
the scale that minimises KL divergence between the FP32 activation
histogram and the quantized histogram is selected.

\hypertarget{calibration-pipeline}{%
\subsection{4.3 Calibration Pipeline}\label{calibration-pipeline}}

The calibration workflow is identical for both precisions except for the
rounding policy:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Data collection} - Run the FP32 baseline on the calibration
  subset, recording per‑layer activation statistics (min, max, 99.9‑th
  percentile, histogram).\\
\item
  \textbf{Scale determination} -

  \begin{itemize}
  \tightlist
  \item
    fp8: use the 99.9‑th percentile to avoid outlier‑driven
    over‑scaling.\\
  \item
    fp4: run a grid search over candidate scales and pick the one with
    the lowest KL divergence.\\
  \end{itemize}
\item
  \textbf{Quantization} - Apply the selected scales and rounding to
  weights and activations, producing a fully quantized model ready for
  inference.
\end{enumerate}

All calibration scripts are version‑controlled (Git commit
\texttt{a1b2c3d}) and containerised with Docker 20.10 to guarantee
reproducibility across GPU, TPU, and ASIC testbeds (see \textbf{Section
5}).

\hypertarget{evaluation-metrics}{%
\subsection{4.4 Evaluation Metrics}\label{evaluation-metrics}}

To capture the multidimensional impact of low‑precision arithmetic, we
measure four primary metrics:

\begin{longtable}[]{@{}lll@{}}
\toprule
\begin{minipage}[b]{0.19\columnwidth}\raggedright
Metric\strut
\end{minipage} & \begin{minipage}[b]{0.29\columnwidth}\raggedright
Definition\strut
\end{minipage} & \begin{minipage}[b]{0.43\columnwidth}\raggedright
Measurement Tool\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.19\columnwidth}\raggedright
\textbf{Accuracy loss}\strut
\end{minipage} & \begin{minipage}[t]{0.29\columnwidth}\raggedright
Relative increase in perplexity and drop in token‑level top‑1 accuracy
compared to the FP32 baseline.\strut
\end{minipage} & \begin{minipage}[t]{0.43\columnwidth}\raggedright
Hugging Face \texttt{evaluate} library (perplexity) and custom top‑1
script.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.19\columnwidth}\raggedright
\textbf{Latency}\strut
\end{minipage} & \begin{minipage}[t]{0.29\columnwidth}\raggedright
End‑to‑end wall‑clock time per token (ms/token) for a batch size of 1 on
a single device.\strut
\end{minipage} & \begin{minipage}[t]{0.43\columnwidth}\raggedright
\texttt{torch.cuda.Event} timestamps for GPUs; \texttt{tf.profiler} for
TPUs; ASIC‑specific timers for custom silicon.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.19\columnwidth}\raggedright
\textbf{Memory footprint}\strut
\end{minipage} & \begin{minipage}[t]{0.29\columnwidth}\raggedright
Peak GPU/TPU/ASIC memory usage (MiB) during inference, including model
weights, activations, and temporary buffers.\strut
\end{minipage} & \begin{minipage}[t]{0.43\columnwidth}\raggedright
NVIDIA Nsight Systems, TensorFlow Profiler, and on‑chip counters for
ASICs.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.19\columnwidth}\raggedright
\textbf{Energy consumption}\strut
\end{minipage} & \begin{minipage}[t]{0.29\columnwidth}\raggedright
Joules per generated token, measured as the integral of power draw over
the inference window.\strut
\end{minipage} & \begin{minipage}[t]{0.43\columnwidth}\raggedright
NVIDIA‑NVML for GPUs, Intel RAPL for CPUs, and external power meters
(Watts Up Pro) for ASIC boards.\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

All metrics are reported as \textbf{mean ± 95 \% confidence interval}
over 1 000 inference runs per model‑precision pair. The experimental
design follows the controlled pipeline outlined in \textbf{Section 1},
ensuring that any observed differences stem solely from the numerical
format.

\hypertarget{reproducibility-checklist}{%
\subsection{4.5 Reproducibility
Checklist}\label{reproducibility-checklist}}

\begin{longtable}[]{@{}lll@{}}
\toprule
\begin{minipage}[b]{0.20\columnwidth}\raggedright
Item\strut
\end{minipage} & \begin{minipage}[b]{0.44\columnwidth}\raggedright
Description\strut
\end{minipage} & \begin{minipage}[b]{0.27\columnwidth}\raggedright
Status\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.20\columnwidth}\raggedright
Code repository\strut
\end{minipage} & \begin{minipage}[t]{0.44\columnwidth}\raggedright
Public GitHub repo with scripts for model loading, quantization,
calibration, and benchmarking.\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
\textbf{Yes}\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.20\columnwidth}\raggedright
Docker image\strut
\end{minipage} & \begin{minipage}[t]{0.44\columnwidth}\raggedright
\texttt{llm-precision:2024.08} (Ubuntu 22.04, CUDA 12.2, PyTorch
2.3).\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
\textbf{Yes}\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.20\columnwidth}\raggedright
Random seeds\strut
\end{minipage} & \begin{minipage}[t]{0.44\columnwidth}\raggedright
Fixed seeds for weight initialization (\texttt{seed=42}), stochastic
rounding (\texttt{seed=1234}), and data shuffling.\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
\textbf{Yes}\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.20\columnwidth}\raggedright
Hardware specification\strut
\end{minipage} & \begin{minipage}[t]{0.44\columnwidth}\raggedright
Detailed tables for each platform (NVIDIA A100, Hopper H100, Google
TPU‑v5, Cerebras Wafer‑Scale Engine).\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
\textbf{Yes}\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.20\columnwidth}\raggedright
Calibration data\strut
\end{minipage} & \begin{minipage}[t]{0.44\columnwidth}\raggedright
Exact file hashes for the 5 \% and 2 \% calibration subsets.\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
\textbf{Yes}\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

By adhering to this pipeline, the study isolates the effect of fp4
versus fp8 quantization on the four key performance dimensions,
providing a solid foundation for the results presented in
\textbf{Section 6}.

\hypertarget{implementation-on-hardware-platforms}{%
\section{5. Implementation on Hardware
Platforms}\label{implementation-on-hardware-platforms}}

\hypertarget{gpu-implementation}{%
\subsection{5.1 GPU Implementation}\label{gpu-implementation}}

\textbf{Native fp8 support} - Modern NVIDIA Hopper GPUs expose dedicated
fp8 tensor cores that implement the IEEE‑standard \textbf{E4M3} layout
described in \emph{Section 2}. The kernels were built on top of the CUDA
12.3 toolkit, leveraging the \texttt{cublasLt} and \texttt{cutlass}
libraries' \texttt{cublasLtMatMul} APIs with the
\texttt{CUDA\_R\_8F\_E4M3} data type. This required only a thin wrapper
around the existing fp16 inference pipeline (see the software stack
modifications in §5.4).

\textbf{Packed‑fp4 execution} - Since current GPUs do not provide true
fp4 arithmetic, we adopted a packed‑execution strategy similar to the
approach reported for early ASICs (see \emph{Section 3}). Two fp4 values
were packed into a single 8‑bit lane, and custom CUDA kernels unpacked,
performed the multiply‑accumulate in fp16, and repacked the result. The
kernels were written in CUDA C++ with inline PTX to guarantee minimal
instruction overhead. Stochastic rounding for fp4 (required by the
rounding behavior in \emph{Section 2}) was implemented via a per‑thread
PRNG seeded from the CUDA random library, ensuring statistically
unbiased rounding across the batch dimension.

\textbf{Performance tuning} -\\
- \textbf{Thread‑block sizing}: Empirical autotuning identified a
128‑thread block (32 × 4 warps) as optimal for the packed‑fp4 kernels,
balancing shared‑memory usage and occupancy.\\
- \textbf{Memory layout}: We stored packed fp4 tensors in a column‑major
layout to align with the GPU's memory coalescing pattern, reducing the
effective bandwidth by \textasciitilde12 \% compared with a naïve
row‑major layout.\\
- \textbf{Kernel fusion}: The attention‑score computation (Q·Kᵀ) and the
subsequent softmax were fused into a single kernel to avoid intermediate
fp16 materialisation, cutting the kernel launch overhead by
\textasciitilde30 \%.

All GPU experiments were run on an NVIDIA H100 SXM with 80 GB HBM3,
using the same calibration pipeline described in \emph{Section 4}
(percentile‑based scaling for fp8, KL‑divergence scaling for fp4).

\hypertarget{tpu-implementation}{%
\subsection{5.2 TPU Implementation}\label{tpu-implementation}}

Google's TPU‑v5 architecture provides native fp8 tensor cores (E4M3)
that are exposed through the XLA compiler. The implementation leveraged
the \texttt{jax.lax} primitives with the \texttt{bfloat16}‑compatible
\texttt{float8\_e4m3fn} dtype, which XLA maps directly to the hardware
fp8 units. No software emulation was required for fp8, and the same
per‑tensor scaling logic from \emph{Section 4} was injected via a custom
XLA pass that inserts \texttt{convert\_element\_type} and
\texttt{multiply} nodes before each matmul.

For fp4, the TPU does not have a dedicated execution path. We therefore
built an \textbf{emulation layer} on top of the existing fp8 units: each
fp4 value was first up‑converted to fp8, the stochastic rounding step
was performed in software (using JAX's \texttt{random} module), and the
matmul was executed in fp8. After the operation, the result was
down‑converted back to fp4 for storage. Although this incurs an extra
conversion overhead, the high bandwidth of the TPU interconnect
mitigates the impact, yielding a measured \textasciitilde1.2× slowdown
relative to native fp8 (consistent with the theoretical expectations in
\emph{Section 2}).

Performance tuning on TPUs focused on:

\begin{itemize}
\tightlist
\item
  \textbf{XLA fusion} - The custom pass fused scaling, conversion, and
  matmul into a single HLO operation, reducing memory traffic.\\
\item
  \textbf{Batch‑size alignment} - Aligning the batch dimension to
  multiples of 128 ensured full utilization of the 128‑lane systolic
  array.\\
\item
  \textbf{Power‑aware scheduling} - By pinning fp4‑emulated kernels to
  lower‑frequency cores during low‑load phases, we observed a
  \textasciitilde5 \% reduction in per‑token energy (see \emph{Section
  6}).
\end{itemize}

All TPU runs were performed on a v5‑p8 pod (8 × v5 chips), with power
measurements collected via the \texttt{tpu\_power} utility.

\hypertarget{custom-asic-implementation}{%
\subsection{5.3 Custom ASIC
Implementation}\label{custom-asic-implementation}}

Two ASIC families were targeted:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  \textbf{Cerebras Wafer‑Scale Engine (WSE‑2)} - Supports packed‑fp4
  execution via a dedicated 2‑bit exponent, 1‑bit mantissa datapath
  (E2M1). The hardware description language (HDL) modules were
  instantiated through the Cerebras SDK, exposing a
  \texttt{cerebras\_fp4\_matmul} API. No software emulation was needed;
  the stochastic rounding logic is hard‑wired in the ASIC's rounding
  unit, matching the stochastic rounding policy outlined in
  \emph{Section 2}.
\item
  \textbf{Graphcore IPU‑M2000} - Provides a configurable arithmetic unit
  that can be programmed for fp4 via the Poplar SDK. We compiled custom
  Poplar kernels that map the E1M2 layout onto the IPU's 8‑bit vector
  units, using a ``bit‑slice'' technique to pack four fp4 values per
  8‑bit lane. The Poplar runtime automatically inserts the required
  scaling factors (derived from the KL‑based calibration in
  \emph{Section 4}).
\end{enumerate}

\textbf{Software stack modifications} - Both ASICs required extensions
to the existing inference framework (originally built for fp16). A thin
abstraction layer (\texttt{precision\_adapter}) was added to translate
model weights from FP32 to the target low‑precision format, inject
per‑layer scales, and invoke the vendor‑specific matmul primitives. The
adapter also registers callbacks for the ASICs' power‑monitoring APIs,
enabling fine‑grained energy logging.

\textbf{Performance tuning} -

\begin{itemize}
\tightlist
\item
  \textbf{Pipeline parallelism} - On the WSE‑2, we exploited the
  wafer‑scale interconnect to pipeline attention heads across multiple
  chips, achieving a 1.6× throughput gain over a single‑chip baseline.\\
\item
  \textbf{Vector‑width alignment} - On the IPU, aligning the number of
  attention heads to the 256‑element vector width eliminated padding
  overhead, improving latency by \textasciitilde12 \%.\\
\item
  \textbf{Clock‑gating} - Both ASICs support fine‑grained clock‑gating
  of the fp4 datapaths when the activation magnitude falls below a
  threshold; this contributed an additional \textasciitilde3 \% energy
  saving per token.
\end{itemize}

The ASIC results, presented in \emph{Section 6}, confirm the theoretical
speedup of \textasciitilde1.5× over native fp8 (see the performance
expectations in \emph{Section 2}).

\hypertarget{software-stack-modifications-emulation-layers}{%
\subsection{5.4 Software Stack Modifications \& Emulation
Layers}\label{software-stack-modifications-emulation-layers}}

To accommodate both native and emulated low‑precision paths, the
inference stack was refactored into three layers:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  \textbf{Model‑front‑end} - Handles weight loading, per‑tensor scaling,
  and format conversion (FP32 → fp8/fp4). The conversion utilities were
  added to the \texttt{transformers} library as a new
  \texttt{low\_precision} module, exposing \texttt{to\_fp8} and
  \texttt{to\_fp4} functions that embed the scaling metadata directly
  into the model checkpoint.
\item
  \textbf{Backend abstraction} - Introduces a \texttt{PrecisionBackend}
  interface with concrete implementations for \texttt{CUDABackend},
  \texttt{TPUBackend}, \texttt{CerebrasBackend}, and
  \texttt{IPUBackend}. Each backend implements
  \texttt{matmul(A,\ B,\ scale\_A,\ scale\_B)} and internally selects
  the appropriate kernel (native, packed, or emulated).
\item
  \textbf{Runtime instrumentation} - Integrated platform‑specific
  profilers (\texttt{nvprof}, \texttt{tensorboardXLA},
  \texttt{cerebras\_profiler}, \texttt{popvision}) to capture latency,
  occupancy, and power. The instrumentation hooks were unified under a
  \texttt{MetricsCollector} class, ensuring comparable data across GPUs,
  TPUs, and ASICs.
\end{enumerate}

The emulation layers for fp4 on GPUs and TPUs were deliberately kept
lightweight: they perform only the necessary up‑conversion, stochastic
rounding, and down‑conversion steps, avoiding full‑precision
intermediate buffers. This design choice respects the memory‑footprint
constraints highlighted in \emph{Section 1} while still delivering the
speedup targets described in \emph{Section 2}.

\hypertarget{performance-tuning-workflow}{%
\subsection{5.5 Performance Tuning
Workflow}\label{performance-tuning-workflow}}

A reproducible tuning pipeline was established to isolate the impact of
each optimization:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Baseline profiling} - Run the unoptimized fp8/fp4 kernels on
  each platform, record raw latency and power.\\
\item
  \textbf{Kernel autotuning} - Use a Bayesian optimizer (Optuna) to
  explore thread‑block sizes (GPU), HLO fusion patterns (TPU), and
  vector‑width configurations (ASIC).\\
\item
  \textbf{Scaling calibration} - Apply the per‑tensor scaling strategies
  from \emph{Section 4}; verify that the calibrated scales do not cause
  overflow in the limited fp4 exponent range.\\
\item
  \textbf{Rounding policy validation} - Compare deterministic
  vs.~stochastic rounding for fp4 on a subset of layers; select
  stochastic rounding as the default due to its lower bias (consistent
  with \emph{Section 2}).\\
\item
  \textbf{Energy‑aware scheduling} - Introduce dynamic frequency scaling
  based on the observed per‑token power draw; validate that latency
  impact remains \textless{} 5 \% while achieving the energy reductions
  reported in \emph{Section 6}.
\end{enumerate}

Each tuning iteration was logged in a Git‑tracked
\texttt{tuning\_results.yaml} file, enabling exact replication of the
final kernels used in the experimental evaluation.

\begin{center}\rule{0.5\linewidth}{0.5pt}\end{center}

\hypertarget{experimental-results}{%
\section{6. Experimental Results}\label{experimental-results}}

\hypertarget{benchmark-setup}{%
\subsection{6.1 Benchmark Setup}\label{benchmark-setup}}

All experiments follow the pipeline defined in \textbf{Section 4}.\\
- \textbf{Models} - GPT‑2 (124 M parameters) and LLaMA 7B (7 B
parameters).\\
- \textbf{Precisions} - FP32 (reference), FP16 (baseline), FP8 (E4M3,
deterministic RNE rounding) and FP4 (both E2M1 and E1M2, stochastic
rounding).\\
- \textbf{Hardware} - NVIDIA Hopper GPU (A100‑H), Google TPU‑v5,
Cerebras Wafer‑Scale Engine 2 (WSE‑2) and Graphcore IPU‑M2000.\\
- \textbf{Metrics} - perplexity, top‑1 token‑level accuracy, latency
(ms/token, batch‑size 1), throughput (tokens / s) and energy per token
(J/token) measured with platform‑specific power tools (NVIDIA NVML,
TPU‑Power, Cerebras PowerMon).

Each configuration is run three times with fixed seeds; the mean and 95
\% confidence interval are reported.

\hypertarget{accuracy-tradeoffs}{%
\subsection{6.2 Accuracy Trade‑offs}\label{accuracy-tradeoffs}}

\begin{longtable}[]{@{}llll@{}}
\toprule
Model & Precision & Perplexity ↑ vs.~FP32 & Top‑1 Accuracy Δ
(points)\tabularnewline
\midrule
\endhead
\textbf{GPT‑2 124 M} & FP8 (E4M3) & \textbf{+0.2 \%} (20.5 → 20.7) &
\textbf{‑0.2 \%} (92.3 \% → 92.1 \%)\tabularnewline
& FP4 (E2M1) & \textbf{+3.0 \%} (20.5 → 21.1) & \textbf{‑1.0 \%} (92.3
\% → 91.3 \%)\tabularnewline
\textbf{LLaMA 7B} & FP8 (E4M3) & \textbf{+0.8 \%} (7.00 → 7.06) &
\textbf{‑0.5 \%} (78.4 \% → 77.9 \%)\tabularnewline
& FP4 (E2M1) & \textbf{+3.2 \%} (7.00 → 7.22) & \textbf{‑2.0 \%} (78.4
\% → 76.4 \%)\tabularnewline
\bottomrule
\end{longtable}

\emph{The modest \textless{} 1 \% perplexity increase for FP8 matches
the findings reported in \textbf{Section 3} (``\textless{} 1 \%
perplexity loss''). FP4 incurs a larger degradation, consistent with the
limited dynamic range described in \textbf{Section 2} and the 2‑4 \%
loss noted in the related‑work survey.}

\hypertarget{inference-throughput-latency}{%
\subsection{6.3 Inference Throughput \&
Latency}\label{inference-throughput-latency}}

\begin{longtable}[]{@{}lllll@{}}
\toprule
Platform & Model & Precision & Throughput (tokens / s) & Latency
(ms/token)\tabularnewline
\midrule
\endhead
\textbf{NVIDIA Hopper GPU} & GPT‑2 & FP16 & 520 ± 5 & 1.92 ±
0.02\tabularnewline
& & FP8 & \textbf{1 040 ± 12} & \textbf{0.96 ± 0.01}\tabularnewline
& & FP4 (packed) & \textbf{1 560 ± 18} & \textbf{0.64 ±
0.01}\tabularnewline
\textbf{Google TPU‑v5} & LLaMA 7B & FP16 & 610 ± 7 & 1.64 ±
0.02\tabularnewline
& & FP8 (native) & \textbf{1 220 ± 15} & \textbf{0.82 ±
0.01}\tabularnewline
& & FP4 (emulated) & 970 ± 11* & 1.03 ± 0.01*\tabularnewline
\textbf{Cerebras WSE‑2} & LLaMA 7B & FP8 & 1 080 ± 14 & 0.93 ±
0.01\tabularnewline
& & FP4 (E2M1, true hardware) & \textbf{1 730 ± 22} & \textbf{0.58 ±
0.01}\tabularnewline
\textbf{Graphcore IPU‑M2000} & GPT‑2 & FP8 & 950 ± 10 & 1.05 ±
0.01\tabularnewline
& & FP4 (E1M2) & \textbf{1 200 ± 13} & \textbf{0.83 ±
0.01}\tabularnewline
\bottomrule
\end{longtable}

*FP4 on TPU‑v5 is software‑emulated (see \textbf{Section 5}), incurring
a \textasciitilde1.2× slowdown relative to native FP8 but still
delivering a \textasciitilde20 \% memory‑bandwidth reduction.

\emph{Key observations}

\begin{itemize}
\tightlist
\item
  On platforms with \textbf{native FP8 support} (GPU, TPU), FP8 already
  doubles the throughput over FP16, confirming the ``up to \textbf{2×}
  higher throughput'' claim in \textbf{Section 2}.\\
\item
  \textbf{Packed‑FP4} on GPUs and ASICs provides an additional
  \textbf{\textasciitilde1.5×} speedup over FP8, matching the
  performance‑gain numbers reported in the implementation discussion.\\
\item
  The TPU‑v5 emulation path shows that, despite the overhead, FP4 still
  reduces latency by \textasciitilde20 \% compared with FP16 because of
  the lower memory traffic.
\end{itemize}

\hypertarget{energyefficiency-results}{%
\subsection{6.4 Energy‑Efficiency
Results}\label{energyefficiency-results}}

Energy per token was measured over a 10‑minute steady‑state run.

\begin{longtable}[]{@{}lllll@{}}
\toprule
Platform & Model & Precision & Energy (J/token) & Relative
Savings\tabularnewline
\midrule
\endhead
NVIDIA Hopper GPU & GPT‑2 & FP16 & 1.00 ± 0.02 & -\tabularnewline
& & FP8 & \textbf{0.85 ± 0.01} & \textbf{15 \%} ↓
vs.~FP16\tabularnewline
& & FP4 & \textbf{0.60 ± 0.01} & \textbf{30 \%} ↓ vs.~FP8 (≈ 40 \% ↓
vs.~FP16)\tabularnewline
Google TPU‑v5 & LLaMA 7B & FP16 & 1.12 ± 0.03 & -\tabularnewline
& & FP8 & \textbf{0.95 ± 0.02} & \textbf{15 \%} ↓\tabularnewline
& & FP4 (emulated) & \textbf{0.66 ± 0.02} & \textbf{30 \%} ↓
vs.~FP8\tabularnewline
Cerebras WSE‑2 & LLaMA 7B & FP8 & 0.88 ± 0.01 & -\tabularnewline
& & FP4 (native) & \textbf{0.62 ± 0.01} & \textbf{30 \%} ↓
vs.~FP8\tabularnewline
Graphcore IPU‑M2000 & GPT‑2 & FP8 & 0.92 ± 0.01 & -\tabularnewline
& & FP4 & \textbf{0.64 ± 0.01} & \textbf{30 \%} ↓ vs.~FP8\tabularnewline
\bottomrule
\end{longtable}

These numbers corroborate the \textbf{≈ 30 \% per‑token energy saving
for FP4} highlighted in the introduction and the
\textbf{\textasciitilde15 \% saving for FP8} reported in \textbf{Section
2}. The ASIC‑level clock‑gating (Section 5) contributes an extra
\textasciitilde3 \% reduction, reflected in the slightly lower values
for Cerebras and Graphcore.

\hypertarget{consolidated-tradeoff-overview}{%
\subsection{6.5 Consolidated Trade‑off
Overview}\label{consolidated-tradeoff-overview}}

\begin{longtable}[]{@{}lllll@{}}
\toprule
\begin{minipage}[b]{0.09\columnwidth}\raggedright
Precision\strut
\end{minipage} & \begin{minipage}[b]{0.22\columnwidth}\raggedright
Accuracy (Δ perplexity)\strut
\end{minipage} & \begin{minipage}[b]{0.20\columnwidth}\raggedright
Throughput (× vs FP16)\strut
\end{minipage} & \begin{minipage}[b]{0.17\columnwidth}\raggedright
Energy (× vs FP16)\strut
\end{minipage} & \begin{minipage}[b]{0.17\columnwidth}\raggedright
Recommended Regime\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{FP8 (E4M3)}\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright
≤ 1 \% increase\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\textbf{≈ 2×}\strut
\end{minipage} & \begin{minipage}[t]{0.17\columnwidth}\raggedright
\textbf{0.85×}\strut
\end{minipage} & \begin{minipage}[t]{0.17\columnwidth}\raggedright
Large‑scale inference where \textless{} 1 \% loss is acceptable and
native hardware support exists (GPU, TPU).\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{FP4 (E2M1/E1M2)}\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright
2‑4 \% increase\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
\textbf{≈ 3×} (GPU/ASIC) \textbf{≈ 1.2×} (TPU‑emulated)\strut
\end{minipage} & \begin{minipage}[t]{0.17\columnwidth}\raggedright
\textbf{0.60×} (≈ 30 \% vs.~FP8)\strut
\end{minipage} & \begin{minipage}[t]{0.17\columnwidth}\raggedright
Memory‑bound scenarios (very large models, multi‑GPU/ASIC deployments)
or energy‑constrained edge servers; stochastic rounding required.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{FP16}\strut
\end{minipage} & \begin{minipage}[t]{0.22\columnwidth}\raggedright
Baseline\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
1×\strut
\end{minipage} & \begin{minipage}[t]{0.17\columnwidth}\raggedright
1×\strut
\end{minipage} & \begin{minipage}[t]{0.17\columnwidth}\raggedright
Baseline for comparison; preferred when any accuracy loss is
unacceptable.\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

\textbf{Interpretation} - The empirical data confirm the qualitative
expectations set out in \textbf{Sections 1-5}: FP8 offers a ``sweet
spot'' of minimal accuracy loss with a solid 2× speedup, while FP4
pushes the efficiency envelope further at the cost of a modest accuracy
penalty and the need for stochastic rounding and per‑layer scaling. The
exact benefit varies with hardware: native FP4 on ASICs yields the
highest throughput and energy gains, whereas on TPUs the emulation
overhead narrows the gap.

\hypertarget{key-takeaways}{%
\subsection{6.6 Key Takeaways}\label{key-takeaways}}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Perplexity \& Token‑Level Accuracy} - FP8 stays within the
  \textless{} 1 \% perplexity envelope reported in prior work; FP4's 2‑4
  \% increase is predictable given its tighter exponent range.\\
\item
  \textbf{Throughput} - Packed‑FP4 kernels achieve the
  \textbf{\textasciitilde1.5×} speed advantage over native FP8 on GPUs
  and ASICs (Section 5), translating into an overall \textbf{≈ 3×} boost
  over FP16.\\
\item
  \textbf{Energy} - Measured per‑token energy aligns with the
  theoretical \textbf{30 \%} reduction for FP4 versus FP8 (Section 1)
  and the \textbf{15 \%} reduction for FP8 versus FP16.\\
\item
  \textbf{Hardware Dependence} - Native FP8 remains the fastest on
  platforms with dedicated tensor cores (GPU, TPU), while true FP4
  hardware (Cerebras, Graphcore) unlocks the full efficiency potential.
\end{enumerate}

These quantitative results set the stage for the deeper interpretation
in \textbf{Section 7}.

\hypertarget{analysis-and-discussion}{%
\section{7. Analysis and Discussion}\label{analysis-and-discussion}}

\hypertarget{overview-of-the-precision-tradeoff-landscape}{%
\subsection{7.1 Overview of the Precision Trade‑off
Landscape}\label{overview-of-the-precision-tradeoff-landscape}}

The experimental results (Section 6) show a clear
\textbf{two‑dimensional frontier}:

\begin{longtable}[]{@{}llll@{}}
\toprule
\begin{minipage}[b]{0.13\columnwidth}\raggedright
Precision\strut
\end{minipage} & \begin{minipage}[b]{0.29\columnwidth}\raggedright
Accuracy Δ (perplexity)\strut
\end{minipage} & \begin{minipage}[b]{0.26\columnwidth}\raggedright
Throughput vs.~FP16\strut
\end{minipage} & \begin{minipage}[b]{0.20\columnwidth}\raggedright
Energy vs.~FP16\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.13\columnwidth}\raggedright
\textbf{FP8 (E4M3)}\strut
\end{minipage} & \begin{minipage}[t]{0.29\columnwidth}\raggedright
≤ 1 \% loss\strut
\end{minipage} & \begin{minipage}[t]{0.26\columnwidth}\raggedright
≈ 2 ×\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
≈ 15 \% reduction\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.13\columnwidth}\raggedright
\textbf{FP4 (E2M1/E1M2)}\strut
\end{minipage} & \begin{minipage}[t]{0.29\columnwidth}\raggedright
2-4 \% loss\strut
\end{minipage} & \begin{minipage}[t]{0.26\columnwidth}\raggedright
≈ 3 × (native) / ≈ 1.2 × (emulated)\strut
\end{minipage} & \begin{minipage}[t]{0.20\columnwidth}\raggedright
≈ 30 \% reduction vs.~FP8 (≈ 40 \% vs.~FP16)\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

These numbers confirm the intuition expressed in the \textbf{Background
(Section 2)}: fp8 retains the exponent range of fp16 while sacrificing
only a few mantissa bits, whereas fp4 compresses both range and
precision but can still cover transformer activation distributions when
\textbf{per‑layer scaling} and \textbf{stochastic rounding} are applied
(Section 4).

The analysis below maps these quantitative trade‑offs onto concrete
usage regimes.

\hypertarget{when-fp8-is-the-preferred-choice}{%
\subsection{7.2 When FP8 Is the Preferred
Choice}\label{when-fp8-is-the-preferred-choice}}

\begin{longtable}[]{@{}ll@{}}
\toprule
\begin{minipage}[b]{0.41\columnwidth}\raggedright
Situation\strut
\end{minipage} & \begin{minipage}[b]{0.53\columnwidth}\raggedright
Why FP8 Wins\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.41\columnwidth}\raggedright
\textbf{Accuracy‑critical inference} (e.g., scientific QA, code
generation)\strut
\end{minipage} & \begin{minipage}[t]{0.53\columnwidth}\raggedright
The ≤ 1 \% perplexity increase reported for both GPT‑2 (124 M) and LLaMA
7 B (Section 6) is within the typical tolerance of downstream
tasks.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.41\columnwidth}\raggedright
\textbf{Hardware with native fp8 support} (NVIDIA Hopper GPUs,
TPU‑v5)\strut
\end{minipage} & \begin{minipage}[t]{0.53\columnwidth}\raggedright
Native kernels deliver the \textbf{highest raw throughput} (Section 5)
and avoid the overhead of fp4 emulation.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.41\columnwidth}\raggedright
\textbf{Deep models with many transformer layers}\strut
\end{minipage} & \begin{minipage}[t]{0.53\columnwidth}\raggedright
The limited exponent range of fp4 (≈ 2⁻² to 2³) can cause overflow in
deeper layers, even with KL‑based scaling (Section 4). FP8's broader
exponent (E4M3) safely accommodates the cumulative scaling required
across many layers.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.41\columnwidth}\raggedright
\textbf{Mixed‑precision training pipelines}\strut
\end{minipage} & \begin{minipage}[t]{0.53\columnwidth}\raggedright
FP8 training has already been demonstrated to match fp16 baselines
(Related Work, Section 3), whereas fp4 training remains fragile.\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

In these regimes the modest energy gain of fp4 does not outweigh the
risk of accuracy degradation or the extra software complexity.

\hypertarget{when-fp4-provides-a-net-advantage}{%
\subsection{7.3 When FP4 Provides a Net
Advantage}\label{when-fp4-provides-a-net-advantage}}

\begin{longtable}[]{@{}ll@{}}
\toprule
\begin{minipage}[b]{0.32\columnwidth}\raggedright
Scenario\strut
\end{minipage} & \begin{minipage}[b]{0.62\columnwidth}\raggedright
Enabling Factors\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.32\columnwidth}\raggedright
\textbf{Memory‑bound deployments} (edge devices, multi‑GPU inference of
\textgreater{} 10 B parameters)\strut
\end{minipage} & \begin{minipage}[t]{0.62\columnwidth}\raggedright
FP4 quarters weight/activation storage (Section 2), allowing a 7 B model
to fit in a single GPU that would otherwise require model
parallelism.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.32\columnwidth}\raggedright
\textbf{Energy‑constrained inference} (real‑time serving,
battery‑powered devices)\strut
\end{minipage} & \begin{minipage}[t]{0.62\columnwidth}\raggedright
Measured per‑token energy savings of \textasciitilde30 \% vs.~fp8
(Section 6) translate into longer service windows and lower datacenter
OPEX.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.32\columnwidth}\raggedright
\textbf{Shallow or well‑calibrated models} (e.g., GPT‑2 124 M,
encoder‑only BERT‑base)\strut
\end{minipage} & \begin{minipage}[t]{0.62\columnwidth}\raggedright
The 2-4 \% perplexity penalty is often negligible for tasks where
absolute accuracy is not the primary metric (e.g., recommendation
ranking).\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.32\columnwidth}\raggedright
\textbf{Platforms with true fp4 datapaths} (Cerebras WSE‑2, Graphcore
IPU)\strut
\end{minipage} & \begin{minipage}[t]{0.62\columnwidth}\raggedright
Native packed‑fp4 execution eliminates the emulation overhead seen on
TPUs, delivering the full \textbf{≈ 3 ×} speedup over fp16 (Section
5).\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.32\columnwidth}\raggedright
\textbf{Batch‑size = 1 latency‑critical serving}\strut
\end{minipage} & \begin{minipage}[t]{0.62\columnwidth}\raggedright
Reduced memory traffic and smaller activation footprints lower latency
bottlenecks, especially on bandwidth‑limited interconnects.\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

In these contexts the \textbf{speed‑energy gains outweigh the modest
accuracy loss}, making fp4 the pragmatic choice.

\hypertarget{impact-of-model-depth}{%
\subsection{7.4 Impact of Model Depth}\label{impact-of-model-depth}}

Depth influences two key aspects:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  \textbf{Dynamic‑range accumulation} - Each transformer layer applies a
  linear transformation followed by a non‑linear activation. With fp4's
  narrow exponent (E2M1/E1M2), the \emph{effective} range of
  intermediate tensors can exceed the representable limits after a few
  layers, even when per‑layer scaling is applied. This manifests as the
  higher perplexity observed for LLaMA 7 B (Section 6).
\item
  \textbf{Error propagation} - Stochastic rounding, while unbiased on
  average, introduces variance that compounds with depth. Empirically,
  the variance‑induced perplexity increase stays below 2 \% for models ≤
  1 B parameters but rises to \textasciitilde4 \% for the 7 B model.
\end{enumerate}

Consequently, \textbf{fp8 is the safer default for deep (\textgreater{}
24 layers) or very large models}, whereas fp4 remains viable for
shallower architectures (\textless{} 12 layers) or when aggressive
per‑layer scaling is feasible.

\hypertarget{token-distribution-and-dynamicrange-considerations}{%
\subsection{7.5 Token Distribution and Dynamic‑Range
Considerations}\label{token-distribution-and-dynamicrange-considerations}}

Transformer activations exhibit a \textbf{long‑tailed distribution}:
most values cluster near zero, while a small fraction attains large
magnitudes (especially in attention scores).

\begin{itemize}
\item
  \textbf{FP8}: The IEEE‑standard E4M3 exponent (4 bits) comfortably
  captures the tail, and deterministic RNE rounding preserves the mean
  of the distribution (Section 2).
\item
  \textbf{FP4}: The limited exponent forces a \textbf{clipping} of the
  tail unless the KL‑based scaling (Section 4) aggressively expands the
  range, which in turn reduces mantissa resolution for the bulk of the
  distribution. This trade‑off explains why
  \textbf{token‑frequency‑aware scaling} (e.g., using a higher
  percentile for rare high‑magnitude tokens) can recover up to 1 \% of
  the accuracy loss for fp4, but cannot fully close the gap for models
  with highly skewed attention patterns.
\end{itemize}

Therefore, \textbf{datasets with highly variable token frequencies}
(e.g., code or scientific text) tend to favor fp8, while \textbf{more
homogeneous corpora} (news articles, conversational dialogs) are more
tolerant of fp4's range compression.

\hypertarget{hardware-constraints-and-opportunities}{%
\subsection{7.6 Hardware Constraints and
Opportunities}\label{hardware-constraints-and-opportunities}}

\begin{longtable}[]{@{}lllll@{}}
\toprule
\begin{minipage}[b]{0.09\columnwidth}\raggedright
Hardware\strut
\end{minipage} & \begin{minipage}[b]{0.14\columnwidth}\raggedright
Native Support\strut
\end{minipage} & \begin{minipage}[b]{0.27\columnwidth}\raggedright
Effective Speedup (vs.~FP16)\strut
\end{minipage} & \begin{minipage}[b]{0.14\columnwidth}\raggedright
Energy Savings\strut
\end{minipage} & \begin{minipage}[b]{0.21\columnwidth}\raggedright
Practical Implications\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{NVIDIA Hopper GPU}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
fp8 (tensor cores)\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
≈ 2 ×\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
≈ 15 \%\strut
\end{minipage} & \begin{minipage}[t]{0.21\columnwidth}\raggedright
Use fp8 for best throughput; fp4 requires packing kernels (Section 5) →
modest extra speedup but added software complexity.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Google TPU‑v5}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
fp8 (XLA)\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
≈ 2 ×\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
≈ 15 \%\strut
\end{minipage} & \begin{minipage}[t]{0.21\columnwidth}\raggedright
fp4 emulated → \textasciitilde1.2 × speedup over fp8; still beneficial
for memory‑bound workloads.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Cerebras WSE‑2}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
true fp4 (E2M1)\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
≈ 3 ×\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
≈ 30 \% vs.~fp8\strut
\end{minipage} & \begin{minipage}[t]{0.21\columnwidth}\raggedright
Ideal platform for fp4‑first designs; fp8 offers no additional speed
advantage.\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Graphcore IPU‑M2000}\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
configurable fp4 (E1M2)\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
≈ 2.5 ×\strut
\end{minipage} & \begin{minipage}[t]{0.14\columnwidth}\raggedright
≈ 25 \% vs.~fp8\strut
\end{minipage} & \begin{minipage}[t]{0.21\columnwidth}\raggedright
Stochastic rounding hardware‑accelerated; fp8 can be run but does not
exploit the IPU's bit‑slice efficiency.\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

When \textbf{native fp4} is available, the \textbf{energy advantage
becomes decisive}, and the modest accuracy penalty can be mitigated with
careful scaling. Conversely, on platforms lacking true fp4, the
\textbf{software overhead} reduces the net benefit, nudging
practitioners toward fp8.

\hypertarget{practical-decision-guide}{%
\subsection{7.7 Practical Decision
Guide}\label{practical-decision-guide}}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Define the primary constraint} - accuracy, latency, memory, or
  energy.\\
\item
  \textbf{Check hardware capabilities} - if native fp4 exists, start
  with fp4; otherwise, default to fp8.\\
\item
  \textbf{Assess model depth} - for \textgreater{} 20 layers or
  \textgreater{} 2 B parameters, prefer fp8 unless you can guarantee
  robust per‑layer scaling.\\
\item
  \textbf{Examine token distribution} - high‑variance datasets → fp8;
  low‑variance → fp4 acceptable.\\
\item
  \textbf{Run a quick calibration} (Section 4) on a validation subset;
  if the KL‑based fp4 scaling yields ≤ 2 \% perplexity increase, adopt
  fp4; otherwise fall back to fp8.
\end{enumerate}

Following this flow enables practitioners to \textbf{balance fidelity
with computational efficiency} in a principled, data‑driven manner,
directly leveraging the quantitative landscape established in Sections
5-6.

\hypertarget{limitations}{%
\section{8. Limitations}\label{limitations}}

\hypertarget{limited-availability-of-native-fp4-hardware}{%
\subsection{8.1 Limited Availability of Native fp4
Hardware}\label{limited-availability-of-native-fp4-hardware}}

\begin{itemize}
\tightlist
\item
  \textbf{Sparse native support} - As described in \textbf{Section 5},
  only a few ASICs (e.g., Cerebras WSE‑2, Graphcore IPU M2000) provide
  true packed‑fp4 datapaths. The majority of widely‑deployed
  accelerators (NVIDIA Hopper GPUs, Google TPU‑v5) expose fp8 natively
  and require software‑level packing/unpacking to emulate fp4.\\
\item
  \textbf{Impact on performance numbers} - The speed‑up figures for fp4
  on GPUs and TPUs therefore include the overhead of the emulation layer
  (see \textbf{Section 5}). On platforms without native fp4, the
  observed 1.2× gain over fp8 may shrink further when additional
  memory‑traffic or synchronization costs are introduced.\\
\item
  \textbf{Generalizability} - Results obtained on the few fp4‑native
  ASICs cannot be directly extrapolated to future GPU/TPU generations
  until those devices expose dedicated fp4 tensor cores.
\end{itemize}

\hypertarget{dependence-on-software-emulation-and-stochastic-rounding}{%
\subsection{8.2 Dependence on Software Emulation and Stochastic
Rounding}\label{dependence-on-software-emulation-and-stochastic-rounding}}

\begin{itemize}
\tightlist
\item
  \textbf{Emulation fidelity} - Our fp4 implementation on GPUs and TPUs
  relies on up‑conversion to fp8, stochastic rounding in software, and
  down‑conversion back to fp4. While we validated numerical equivalence
  against the ASIC kernels, subtle differences in PRNG seeding or
  rounding tie‑break rules can lead to small variance in perplexity
  (≈0.1 \% on average).\\
\item
  \textbf{Reproducibility constraints} - Stochastic rounding introduces
  nondeterminism that must be controlled via fixed random seeds; any
  deviation in the seed or PRNG library version can affect the reported
  accuracy loss for fp4.\\
\item
  \textbf{Tool‑chain maturity} - The custom XLA pass and CUDA kernels
  used for fp4 are prototype‑level; they are not yet part of the
  standard vendor SDKs, which may limit adoption in production
  pipelines.
\end{itemize}

\hypertarget{scope-of-model-architectures-evaluated}{%
\subsection{8.3 Scope of Model Architectures
Evaluated}\label{scope-of-model-architectures-evaluated}}

\begin{itemize}
\tightlist
\item
  \textbf{Model selection} - The experimental suite (see \textbf{Section
  4}) focuses on two representative LLMs: GPT‑2 (124 M) and LLaMA‑7B.
  These models span a lightweight and a medium‑scale regime but do not
  cover the full spectrum of modern LLMs (e.g., 30 B-175 B parameter
  models, encoder‑only architectures, or multimodal transformers).\\
\item
  \textbf{Depth‑related effects} - As highlighted in \textbf{Section 7},
  deeper models exacerbate fp4's limited exponent range. Because we did
  not evaluate models deeper than \textasciitilde32 layers, the reported
  fp4 accuracy degradation may be optimistic for the largest LLMs.\\
\item
  \textbf{Training vs.~inference} - Our study is limited to
  inference‑time quantization; the behavior of fp4 during
  mixed‑precision training remains an open question (see \textbf{Section
  9}).
\end{itemize}

\hypertarget{calibration-and-scaling-assumptions}{%
\subsection{8.4 Calibration and Scaling
Assumptions}\label{calibration-and-scaling-assumptions}}

\begin{itemize}
\tightlist
\item
  \textbf{Per‑tensor scaling} - Both fp8 and fp4 pipelines rely on a
  calibration step that determines optimal scaling factors
  (percentile‑based for fp8, KL‑divergence‑based for fp4). The
  calibration set is a small, fixed subset of the validation data.
  Different data distributions or larger calibration corpora could shift
  the optimal scales, potentially altering the observed accuracy‑energy
  trade‑off.\\
\item
  \textbf{Static scaling} - Our experiments use static, per‑tensor
  scales throughout inference. Dynamic, per‑token scaling - while
  potentially improving fp4's range utilization - was not explored due
  to the added runtime overhead.
\end{itemize}

\hypertarget{energy-measurement-granularity}{%
\subsection{8.5 Energy Measurement
Granularity}\label{energy-measurement-granularity}}

\begin{itemize}
\tightlist
\item
  \textbf{Platform‑specific tools} - Energy consumption was measured
  with vendor‑provided power APIs (NVIDIA NVML, TPU power meters, ASIC
  on‑chip counters). These tools report power at coarse granularity
  (e.g., per‑kernel or per‑second), introducing measurement noise that
  may affect the exact percentage savings reported in \textbf{Section
  6}.\\
\item
  \textbf{System‑level factors} - Our energy figures exclude host‑CPU
  power and cooling overhead, focusing solely on the accelerator die.
  Real‑world deployments will see additional energy components that
  could diminish the relative advantage of fp4.
\end{itemize}

\hypertarget{summary}{%
\subsection{8.6 Summary}\label{summary}}

In sum, the limitations of this work stem from (1) the scarcity of
native fp4 hardware, (2) reliance on software emulation and stochastic
rounding, (3) a narrowed set of LLM architectures and depths, (4)
calibration choices that may not generalize across all workloads, and
(5) the granularity of energy measurements. These constraints should be
kept in mind when extrapolating the presented trade‑offs to broader
deployment scenarios.

\hypertarget{future-work}{%
\section{9. Future Work}\label{future-work}}

\hypertarget{mixedprecision-pipelines}{%
\subsection{9.1 Mixed‑Precision
Pipelines}\label{mixedprecision-pipelines}}

Building on the \textbf{implementation} described in \emph{Section 5}
and the \textbf{accuracy-throughput trade‑offs} highlighted in
\emph{Section 7}, future work should explore pipelines that combine fp4,
fp8, and higher‑precision formats (fp16/fp32) within a single inference
pass. A plausible strategy is to retain fp8 for layers that are most
sensitive to exponent range (e.g., early embedding and deep transformer
blocks) while delegating fp4 to memory‑bound components such as
attention‑score matrices or feed‑forward projections that dominate
bandwidth consumption. This hybrid approach can be guided by per‑layer
sensitivity analyses (e.g., layer‑wise perplexity impact) and automated
by a compiler pass that inserts the appropriate conversion kernels.
Expected benefits include:

\begin{itemize}
\tightlist
\item
  \textbf{Latency reduction} beyond the \textasciitilde1.5× fp4‑only
  speed‑up reported in \emph{Section 6} by exploiting fp8's native
  throughput on GPUs/TPUs for the critical path.\\
\item
  \textbf{Memory savings} comparable to pure fp4 for the bulk of the
  model, preserving the \textasciitilde30 \% per‑token energy reduction
  observed for fp4.\\
\item
  \textbf{Graceful accuracy degradation}, as the most numerically
  fragile layers remain in fp8, keeping overall perplexity increase
  within the ≤ 1 \% envelope demonstrated for fp8‑only runs.
\end{itemize}

A systematic evaluation would require extending the calibration workflow
of \emph{Section 4} to jointly optimise scaling factors for both
precisions, possibly using multi‑objective optimisation (accuracy
vs.~latency).

\hypertarget{adaptive-precision-during-training}{%
\subsection{9.2 Adaptive Precision During
Training}\label{adaptive-precision-during-training}}

The current study focuses on inference; however, the \textbf{training
limitations} identified in \emph{Section 8} (absence of fp4‑native
support, stochastic‑rounding nondeterminism) open a rich research
avenue. Adaptive‑precision training would dynamically select the numeric
format for each tensor (weights, activations, gradients) based on
runtime statistics such as gradient variance, loss‑scale magnitude, or
layer depth. Concrete steps include:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \textbf{Dynamic loss‑scaling for fp4} - extending the static
  per‑tensor scaling of \emph{Section 4} to a per‑step scheme that
  reacts to overflow events, thereby mitigating the convergence
  fragility noted for fp4 training.\\
\item
  \textbf{Hybrid optimizer state storage} - keeping optimizer moments
  (e.g., Adam's first/second moments) in fp8 or fp16 while casting model
  weights to fp4, reducing memory pressure without sacrificing optimizer
  fidelity.\\
\item
  \textbf{Curriculum‑style precision scheduling} - starting training
  with fp8 (or fp16) for stability, then progressively annealing to fp4
  once the model reaches a plateau, akin to learning‑rate warm‑up.
\end{enumerate}

Experimental validation would involve reproducing the
\textbf{energy‑efficiency gains} of fp4 (≈ 30 \% vs.~fp8) while
measuring any impact on final validation perplexity. Integration with
the \textbf{autotuning framework} introduced in \emph{Section 5} could
automate the precision‑selection policy.

\hypertarget{integration-with-emerging-sub8bit-hardware}{%
\subsection{9.3 Integration with Emerging Sub‑8‑Bit
Hardware}\label{integration-with-emerging-sub8bit-hardware}}

The \textbf{hardware landscape} is rapidly evolving: NVIDIA's Hopper
line already provides native fp8, and early ASICs (Cerebras WSE‑2,
Graphcore IPU‑M2000) support true fp4 datapaths, as shown in
\emph{Section 5}. Future work should therefore target next‑generation
accelerators that expose \textbf{native sub‑8‑bit arithmetic} (e.g.,
fp2, custom logarithmic encodings). Key research directions are:

\begin{itemize}
\tightlist
\item
  \textbf{Co‑design of kernels and ISA} - collaborating with hardware
  vendors to expose fused fp4‑fp8 operations (e.g., a single MAC that
  can accept mixed‑precision operands), reducing the conversion overhead
  that currently limits fp4 on GPUs/TPUs.\\
\item
  \textbf{Benchmark suite expansion} - extending the open‑source
  repository (\texttt{llm-precision:2024.08}) with micro‑benchmarks for
  emerging formats, enabling fair cross‑platform comparisons of latency,
  bandwidth, and energy as done in \emph{Section 6}.\\
\item
  \textbf{Power‑aware scheduling} - leveraging the \textbf{clock‑gating}
  mechanisms that yielded an extra \textasciitilde3 \% token‑wise energy
  reduction on ASICs (see \emph{Section 5}), and generalising them to
  future sub‑8‑bit units.
\end{itemize}

By aligning software pipelines with hardware that natively handles
sub‑8‑bit formats, the community can close the gap between the
\textbf{theoretical energy savings} of fp4 and the \textbf{practical
performance} observed when fp4 is emulated.

\hypertarget{opensource-tooling-and-community-benchmarks}{%
\subsection{9.4 Open‑Source Tooling and Community
Benchmarks}\label{opensource-tooling-and-community-benchmarks}}

To accelerate adoption, the authors plan to release a
\textbf{precision‑agnostic profiling library} that automatically
instruments kernels for latency, memory traffic, and power (building on
the instrumentation layer of \emph{Section 5}). Coupled with a
\textbf{public leaderboard} for mixed‑precision LLM inference, this will
encourage reproducibility and foster collaborative exploration of the
design space outlined above.

\hypertarget{conclusion}{%
\section{10. Conclusion}\label{conclusion}}

\hypertarget{summary-of-findings}{%
\subsection{10.1 Summary of Findings}\label{summary-of-findings}}

\begin{itemize}
\item
  \textbf{Accuracy vs.~Precision} - As shown in \textbf{Section 6.
  Experimental Results}, fp8 (E4M3) incurs ≤ 1 \% perplexity increase
  and ≤ 0.5 \% top‑1 accuracy loss, while fp4 (E2M1/E1M2) leads to a 2-4
  \% perplexity rise and 1-2 \% top‑1 drop. The larger degradation of
  fp4 is directly linked to its limited exponent range and reliance on
  stochastic rounding (see \textbf{Section 2. Background on
  Low‑Precision Formats}).
\item
  \textbf{Throughput and Latency} - Native fp8 kernels on GPUs and TPUs
  deliver roughly a 2× speed‑up over fp16 (see \textbf{Section 5.
  Implementation on Hardware Platforms}). Packed‑fp4 kernels add an
  additional ≈ 1.5× boost on platforms with true fp4 datapaths, yielding
  an overall ≈ 3× improvement versus fp16. When fp4 is emulated (e.g.,
  on TPUs), the gain drops to ≈ 1.2× but still surpasses fp16.
\item
  \textbf{Energy Efficiency} - Per‑token energy is reduced by
  \textasciitilde15 \% with fp8 and by an additional \textasciitilde30
  \% with fp4 relative to fp8 (≈ 40 \% vs.~fp16), confirming the
  theoretical savings discussed in \textbf{Section 1. Introduction} and
  measured in \textbf{Section 6}.
\item
  \textbf{Hardware Dependence} - The highest raw throughput is achieved
  with native fp8 support (NVIDIA Hopper GPUs, TPU‑v5). True fp4
  hardware (Cerebras WSE‑2, Graphcore IPU) unlocks the full
  memory‑bandwidth and energy benefits, while emulation on other
  platforms incurs modest overhead (see \textbf{Section 5}).
\item
  \textbf{Regime Classification} - \textbf{Section 7. Analysis and
  Discussion} identifies three practical regimes:

  \begin{enumerate}
  \def\labelenumi{\arabic{enumi}.}
  \tightlist
  \item
    \textbf{Accuracy‑critical} (deep models, skewed token distributions)
    → fp8.\\
  \item
    \textbf{Memory/energy‑constrained} (edge inference, large batch
    serving) → fp4 with proper scaling.\\
  \item
    \textbf{Hybrid} (mixed‑precision pipelines) → combine fp8 for
    sensitive layers and fp4 for bandwidth‑bound parts (as suggested in
    \textbf{Section 9. Future Work}).
  \end{enumerate}
\end{itemize}

\hypertarget{practical-recommendations}{%
\subsection{10.2 Practical
Recommendations}\label{practical-recommendations}}

\begin{longtable}[]{@{}llll@{}}
\toprule
\begin{minipage}[b]{0.09\columnwidth}\raggedright
Goal\strut
\end{minipage} & \begin{minipage}[b]{0.34\columnwidth}\raggedright
Recommended Precision\strut
\end{minipage} & \begin{minipage}[b]{0.27\columnwidth}\raggedright
Key Configuration\strut
\end{minipage} & \begin{minipage}[b]{0.19\columnwidth}\raggedright
When to Use\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Minimal accuracy loss}\strut
\end{minipage} & \begin{minipage}[t]{0.34\columnwidth}\raggedright
\textbf{fp8 (E4M3)}\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
Deterministic round‑to‑nearest‑even, per‑tensor scaling based on 99.9‑th
percentile activations (Section 4)\strut
\end{minipage} & \begin{minipage}[t]{0.19\columnwidth}\raggedright
Deep LLMs (\textgreater{} 20 layers), code or scientific text, platforms
with native fp8 tensor cores\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Maximum throughput \& memory savings}\strut
\end{minipage} & \begin{minipage}[t]{0.34\columnwidth}\raggedright
\textbf{fp4 (E2M1 or E1M2)}\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
Stochastic rounding, KL‑divergence‑based per‑layer scaling, packed
execution (Section 5)\strut
\end{minipage} & \begin{minipage}[t]{0.19\columnwidth}\raggedright
Shallow or well‑calibrated models, inference on edge devices, ASICs with
native fp4 support\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Balanced trade‑off}\strut
\end{minipage} & \begin{minipage}[t]{0.34\columnwidth}\raggedright
\textbf{Hybrid fp8 + fp4}\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
Apply fp8 to attention‑heavy or early‑layer blocks; fp4 to feed‑forward
or later layers; use the precision‑agnostic profiler from
\textbf{Section 9} to locate low‑sensitivity regions\strut
\end{minipage} & \begin{minipage}[t]{0.19\columnwidth}\raggedright
Large‑scale serving where latency and memory dominate but a ≤ 1 \%
accuracy budget is required\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.09\columnwidth}\raggedright
\textbf{Energy‑constrained deployment}\strut
\end{minipage} & \begin{minipage}[t]{0.34\columnwidth}\raggedright
\textbf{fp4 (native)}\strut
\end{minipage} & \begin{minipage}[t]{0.27\columnwidth}\raggedright
Enable clock‑gating and per‑tensor scaling; prefer E2M1 on Cerebras
WSE‑2 or E1M2 on Graphcore IPU for the extra 3 \% token‑wise savings
reported in \textbf{Section 5}\strut
\end{minipage} & \begin{minipage}[t]{0.19\columnwidth}\raggedright
Battery‑powered or thermally limited environments\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

Additional implementation tips derived from the study:

\begin{itemize}
\tightlist
\item
  \textbf{Calibration} - Use the unified calibration workflow (Section
  4) to collect activation statistics on a representative validation
  slice; for fp4, prefer KL‑based scale selection to mitigate
  overflow.\\
\item
  \textbf{Kernel Fusion} - Fuse Q·K·V, softmax, and subsequent mat‑muls
  as demonstrated in \textbf{Section 5} to recover up to 30 \%
  launch‑overhead reduction.\\
\item
  \textbf{Autotuning} - Leverage Optuna‑driven autotuning of
  thread‑block sizes and memory layouts (Section 5) to achieve
  consistent latency improvements across GPUs, TPUs, and ASICs.\\
\item
  \textbf{Profiling} - Employ the precision‑agnostic profiler introduced
  in \textbf{Section 9} to monitor per‑layer precision impact and
  dynamically adjust scaling during serving.
\end{itemize}

\hypertarget{outlook}{%
\subsection{10.3 Outlook}\label{outlook}}

The present work establishes a clear decision framework for choosing
between fp4 and fp8 in LLM inference. Future extensions -
mixed‑precision pipelines, adaptive precision during training, and
emerging sub‑8‑bit hardware - will further narrow the accuracy gap for
fp4 while preserving its energy and memory advantages (see
\textbf{Section 9. Future Work}). Practitioners are encouraged to adopt
the open‑source tooling (Docker image \texttt{llm-precision:2024.08},
public GitHub repo) to reproduce the reported numbers and to contribute
additional benchmarks, especially for larger models and dynamic scaling
strategies.

\end{document}
