start Cost Chapter

README.md

@@ -22,8 +22,8 @@ Abstract: **The current discourse around Large Language Models (LLMs) tends to f
 | Chapter 4: Safety              | [html](https://www.souzatharsis.com/tamingLLMs/notebooks/safety.html)  |  [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/safety.ipynb) | *Ready for Review*     |
 | Chapter 5: Preference-Based Alignment     | [html](https://www.souzatharsis.com/tamingLLMs/notebooks/alignment.html) | [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/alignment.ipynb) | *Ready for Review*     |
 | Chapter 6: Local LLMs in Practice |   [html](https://www.souzatharsis.com/tamingLLMs/notebooks/local.html) | [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/local.ipynb) | *Ready for Review*     |
-| Chapter 7: The Cost Factor                |              |               |                      |
-| Chapter 8: Frontiers                |              |               |     WIP            |
+| Chapter 7: The Falling Cost Paradox               |              |               |        WIP              |
+| Chapter 8: Frontiers                |              |               |                 |
 | Appendix A: Tools and Resources           |              |               |                      |
 
 

pyproject.toml

@@ -30,7 +30,6 @@ evaluate = "^0.4.3"
 absl-py = "^2.1.0"
 rouge-score = "^0.1.2"
 matplotlib = "^3.9.2"
-lighteval = {extras = ["accelerate"], version = "^0.6.2"}
 outlines = "^0.1.7"
 datasets = "^3.1.0"
 text-generation = "^0.7.0"
@@ -43,6 +42,9 @@ mistralai = "^1.2.5"
 llm-guard = "^0.3.15"
 pygments = "^2.18.0"
 llama-cpp-python = "^0.3.5"
+torch = "^2.5.1"
+gguf = "^0.13.0"
+accelerate = "^1.2.1"
 
 
 [build-system]

tamingllms/_build/html/_sources/markdown/toc.md

@@ -32,7 +32,7 @@ Abstract: *The current discourse around Large Language Models (LLMs) tends to fo
 
 ## [Chapter 6: Local LLMs in Practice](https://www.souzatharsis.com/tamingLLMs/notebooks/local.html)
 
-## Chapter 7: The Cost Factor
+## Chapter 7: The Falling Cost Paradox
 
 ## Chapter 8: Frontiers
 

tamingllms/_build/html/_sources/notebooks/alignment.ipynb

@@ -2537,7 +2537,7 @@
    "source": [
     "## Discussion and Conclusions\n",
     "\n",
-    "LLMs are complex systems and alignment is a challenging problem. In this chapter, we discussed how post-training alignment techniques can be used to align a language model to human preferences. In the case study, we demonstrated how to use DPO to align a language model to a user-provider policy further automating the process via synthetic data generation and LLM-as-judge evaluation. Our approach does serve as a proof of concept, however, several considerations should be taken into account when using this methodology in practice.\n",
+    "LLMs are complex systems and alignment is a challenging problem. In this chapter, we discussed how post-training techniques can be used to align a language model to human preferences. In the case study, we demonstrated how to use DPO to align a language model to a user-provider policy further automating the process via synthetic data generation and LLM-as-judge evaluation. Our approach serves as a proof of concept and several considerations should be taken into account when using this methodology in practice.\n",
     "\n",
     "**Synthetic Data Generation**\n",
     "\n",

tamingllms/_build/html/_sources/notebooks/local.ipynb

@@ -4,6 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "(local)=\n",
     "# Local LLMs in Practice\n",
     "```{epigraph}\n",
     "Freedom is something that dies unless it's used.\n",
@@ -40,7 +41,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Models Considerations\n",
+    "(local-model-selection)=\n",
+    "## Choosing your Model\n",
     "\n",
     "The landscape of open source LLMs is rapidly evolving, with new models emerging by the day. While proprietary LLMs have garnered significant attention, open source LLMs are gaining traction due to their flexibility, customization options, and cost-effectiveness. \n",
     "\n",
@@ -1352,7 +1354,6 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Citation\n",
     "[![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa]\n",
     "\n",
     "[cc-by-nc-sa]: http://creativecommons.org/licenses/by-nc-sa/4.0/\n",

tamingllms/_build/html/_sources/notebooks/structured_output.ipynb

@@ -467,9 +467,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/tobias/src/tamingLLMs/tamingllms/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:654: UserWarning: Can't initialize NVML\n",
+      "  warnings.warn(\"Can't initialize NVML\")\n"
+     ]
+    }
+   ],
    "source": [
     "MODEL_NAME = \"HuggingFaceTB/SmolLM2-1.7B-Instruct\"\n",
     "PROMPT = \"Is Enzo a good name for a baby?\"\n",
@@ -1384,7 +1393,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.3"
+   "version": "3.11.11"
   }
  },
  "nbformat": 4,

tamingllms/_build/html/_static/cost/quantized.tsx

@@ -0,0 +1,58 @@
+import React from 'react';
+import { LineChart, Line, XAxis, YAxis, CartesianGrid, Tooltip, ResponsiveContainer } from 'recharts';
+
+const MemoryUsageChart = () => {
+  const data = [
+    { name: 'F16', value: 141.1 },
+    { name: 'Q8_0', value: 75.0 },
+    { name: 'Q6_K', value: 59.9 },
+    { name: 'Q5_K_M', value: 49.9 },
+    { name: 'Q4_K_M', value: 42.5 },
+    { name: 'Q3_K_M', value: 34.3 },
+    { name: 'Q2_K', value: 26.4 }
+  ];
+
+  return (
+    <div className="w-full h-96 p-4">
+      <ResponsiveContainer width="100%" height="100%">
+        <LineChart data={data} margin={{ top: 20, right: 30, left: 20, bottom: 5 }}>
+          <CartesianGrid strokeDasharray="3 3" />
+          <XAxis 
+            dataKey="name"
+            tick={{ fontSize: 12, fontWeight: 'bold' }}
+          />
+          <YAxis 
+            label={{ 
+              value: 'Model Size (GB)', 
+              angle: -90, 
+              position: 'insideLeft',
+              style: { 
+                textAnchor: 'middle',
+                fontWeight: 'bold'
+              }
+            }}
+            tick={{ fontSize: 12, fontWeight: 'bold' }}
+          />
+          <Tooltip 
+            formatter={(value) => [`${value} GB`, 'Model Size']}
+            contentStyle={{ 
+              backgroundColor: '#fff', 
+              border: '1px solid #ccc',
+              fontWeight: 'bold'
+            }}
+          />
+          <Line 
+            type="monotone"
+            dataKey="value" 
+            stroke="#3eaf7c"
+            strokeWidth={2}
+            dot={{ fill: '#3eaf7c', r: 4 }}
+            activeDot={{ r: 6 }}
+          />
+        </LineChart>
+      </ResponsiveContainer>
+    </div>
+  );
+};
+
+export default MemoryUsageChart;

tamingllms/_build/html/genindex.html

@@ -190,6 +190,15 @@
           </li>
 
 
+          <li class="toctree-l1 ">
+
+              <a href="notebooks/cost.html" class="reference internal ">The Falling Cost Paradox</a>
+
+
+
+          </li>
+
+
       </ul>
     </div>
 

tamingllms/_build/html/markdown/intro.html

@@ -208,6 +208,15 @@
           </li>
 
 
+          <li class="toctree-l1 ">
+
+              <a href="../notebooks/cost.html" class="reference internal ">The Falling Cost Paradox</a>
+
+
+
+          </li>
+
+
       </ul>
     </div>
 

tamingllms/_build/html/markdown/preface.html

@@ -190,6 +190,15 @@
           </li>
 
 
+          <li class="toctree-l1 ">
+
+              <a href="../notebooks/cost.html" class="reference internal ">The Falling Cost Paradox</a>
+
+
+
+          </li>
+
+
       </ul>
     </div>
 
@@ -227,15 +236,15 @@ <h1><span class="section-number">1. </span>Preface<a class="headerlink" href="#p
 <div><p>Models tell you merely what something is like, not what something is.</p>
 <p class="attribution">—Emanuel Derman</p>
 </div></blockquote>
-<p>An alternative title of this book could have been “Language Models Behaving Badly”. If you are coming from a background in financial modeling, you may have noticed the parallel with Emanuel Derman’s seminal work “Models.Behaving.Badly” <span id="id1">[<a class="reference internal" href="#id162" title="E. Derman. Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life. Free Press, 2011. ISBN 9781439165010. URL: https://books.google.co.uk/books?id=lke_cwM4wm8C.">Derman, 2011</a>]</span>. This parallel is not coincidental. Just as Derman cautioned against treating financial models as perfect representations of reality, this book aims to highlight the limitations and pitfalls of Large Language Models (LLMs) in practical applications.</p>
+<p>An alternative title of this book could have been “Language Models Behaving Badly”. If you are coming from a background in financial modeling, you may have noticed the parallel with Emanuel Derman’s seminal work “Models.Behaving.Badly” <span id="id1">[<a class="reference internal" href="#id165" title="E. Derman. Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life. Free Press, 2011. ISBN 9781439165010. URL: https://books.google.co.uk/books?id=lke_cwM4wm8C.">Derman, 2011</a>]</span>. This parallel is not coincidental. Just as Derman cautioned against treating financial models as perfect representations of reality, this book aims to highlight the limitations and pitfalls of Large Language Models (LLMs) in practical applications.</p>
 <p>The book “Models.Behaving.Badly” by Emanuel Derman, a former physicist and Goldman Sachs quant, explores how financial and scientific models can fail when we mistake them for reality rather than treating them as approximations full of assumptions.
 The core premise of his work is that while models can be useful tools for understanding aspects of the world, they inherently involve simplification and assumptions. Derman argues that many financial crises, including the 2008 crash, occurred partly because people put too much faith in mathematical models without recognizing their limitations.</p>
 <p>Like financial models that failed to capture the complexity of human behavior and market dynamics, LLMs have inherent constraints. They can hallucinate facts, struggle with logical reasoning, and fail to maintain consistency across long outputs. Their responses, while often convincing, are probabilistic approximations based on training data rather than true understanding even though humans insist on treating them as “machines that can reason”.</p>
 <p>Today, there is this growing pervasive belief that these models could solve any problem, understand any context, or generate any content as wished by the user. Moreover, language models that were initially designed to be next-token prediction machines and chatbots are now been twisted and wrapped into “reasoning” machines for further integration into technology products and daily-life workflows that control, affect, or decide daily actions of our lives. This technological optimism coupled with lack of understanding of the models’ limitations may pose risks we are still trying to figure out.</p>
 <p>This book serves as an introductory, practical guide for practitioners and technology product builders - software engineers, data scientists, and product managers - who want to create the next generation of GenAI-based products with LLMs while remaining clear-eyed about their limitations and therefore their implications to end-users. Through detailed technical analysis, reproducible Python code examples we explore the gap between LLM capabilities and reliable software product development.</p>
 <p>The goal is not to diminish the transformative potential of LLMs, but rather to promote a more nuanced understanding of their behavior. By acknowledging and working within their constraints, developers can create more reliable and trustworthy applications. After all, as Derman taught us, the first step to using a model effectively is understanding where it breaks down.</p>
 <div class="docutils container" id="id2">
-<div class="citation" id="id162" role="doc-biblioentry">
+<div class="citation" id="id165" role="doc-biblioentry">
 <span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="#id1">Der11</a><span class="fn-bracket">]</span></span>
 <p>E. Derman. <em>Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life</em>. Free Press, 2011. ISBN 9781439165010. URL: <a class="reference external" href="https://books.google.co.uk/books?id=lke_cwM4wm8C">https://books.google.co.uk/books?id=lke_cwM4wm8C</a>.</p>
 </div>

tamingllms/_build/html/markdown/toc.html

@@ -183,6 +183,15 @@
           </li>
 
 
+          <li class="toctree-l1 ">
+
+              <a href="../notebooks/cost.html" class="reference internal ">The Falling Cost Paradox</a>
+
+
+
+          </li>
+
+
       </ul>
     </div>
 
@@ -245,8 +254,8 @@ <h2><a class="reference external" href="https://www.souzatharsis.com/tamingLLMs/
 <section id="chapter-6-local-llms-in-practice">
 <h2><a class="reference external" href="https://www.souzatharsis.com/tamingLLMs/notebooks/local.html">Chapter 6: Local LLMs in Practice</a><a class="headerlink" href="#chapter-6-local-llms-in-practice" title="Permalink to this heading">¶</a></h2>
 </section>
-<section id="chapter-7-the-cost-factor">
-<h2>Chapter 7: The Cost Factor<a class="headerlink" href="#chapter-7-the-cost-factor" title="Permalink to this heading">¶</a></h2>
+<section id="chapter-7-the-falling-cost-paradox">
+<h2>Chapter 7: The Falling Cost Paradox<a class="headerlink" href="#chapter-7-the-falling-cost-paradox" title="Permalink to this heading">¶</a></h2>
 </section>
 <section id="chapter-8-frontiers">
 <h2>Chapter 8: Frontiers<a class="headerlink" href="#chapter-8-frontiers" title="Permalink to this heading">¶</a></h2>