entropy#

  • Series: lecture-python-advanced.myst

  • File: lectures/entropy.md

  • Audit date: 2026-08-26

  • Corpus snapshot: b83d6da399

  • Categories audited: writing, math, figures, references, links (JAX out of scope)

  • Overall score: 6.0 / 10

  • Priority: HIGH

Score breakdown#

Category

Score

One-line note

Writing

3/10

qe-writing-003 ×10; qe-writing-001 ×6; qe-writing-005 ×6, +4 more.

Math

3/10

qe-math-002 ×15; qe-math-011 (proposed) ×13; qe-math-010 (proposed) ×2, +2 more.

Code

N/A

no executable code cells.

JAX

out of scope

JAX rules target lecture-jax.

Figures

7/10

qe-fig-004 ×4; qe-fig-005 ×1; qe-fig-002 ×4.

References

8.5/10

qe-ref-001 ×3.

Links

8.5/10

qe-link-001 ×2.

Admonitions

N/A

no admonitions, exercises or solutions.

Issues#

Critical#

None found.

High severity#

  • [qe-math-002] — Use \top for transpose notation. Count: 15. Lines: 269, 279, 288, 296, 297, 304, 311, 330, 441, 456, …. Example: apostrophe transpose w'.

  • [qe-math-010 (proposed)] — Blackboard \mathbb{P}, \mathbb{E}, \mathbb{V} with braces. Count: 2. Lines: 44, 92. Example: non-blackboard \textrm{Prob}.

  • [qe-math-011 (proposed)] — Distribution names in plain letters, not \mathcal / \mathbb. Count: 13. Lines: 256, 323, 515, 530, 531, 539, 540. Example: decorated distribution {\cal N}.

  • [qe-writing-001] — Use one sentence per paragraph. Count: 6. Lines: 125, 315, 377, 523, 530, 539. Example: 2 sentences in one paragraph.

  • [qe-writing-003] (reviewer) — Maintain logical flow. Count: 10. Lines: 119, 236, 372, 450, 539. Example: both figure captions that carry content are wrong. 112 sends the reader to {numref}`fig1` to see the coin result of 96-110, but the caption at 119 reads “Entropy as a function of \(\hat \pi_1\) when \(\pi_1 = .5\)” - neither \(\pi_1\) nor \(\hat\pi_1\) occurs anywhere in this lecture, which writes the coin probability as \(p\) throughout. And 512-515 introduces {numref}`figure-example4` as the case whose denominator density is \({\mathcal N}(0,.95)\), but its caption at 539-540 names \({\mathcal N}(0,1.5)\) twice, duplicating figure-example3’s caption at 530-531. Four items of algebra are also wrong or stale: 236 defines the cross-entropy as \(H(p,q) = \sum_i p_i \log q_i\), dropping the minus sign that 231-232 requires; 450 writes the Wold representation as \(y_t = D(L)\epsilon_t \equiv \sum_{j=0}^\infty D_j \epsilon_t\), with no lag on \(\epsilon\); 372 asserts \(L_t(p^*_{t+1}/p_{t+1}) = D_{KL,t}(p^*_{t+1}|p_{t+1})\), but with \(E_t\) taken under \(p_{t+1}\) (362) and \(D\) defined as at 224, \(-E_t\log(p^*/p)\) is \(D_{KL,t}(p_{t+1}|p^*_{t+1})\), the other order; and 458-462 writes the one-step-ahead prediction error as \(D(0)D(0)'\) while 484 writes \(\log\det[D(0)'D(0)]\) for the same object. Three smaller slips: 157 has the density as \(p(x_i, y_i)\) where 163 needs \(p(x_i,y_j)\); 217 assumes \(\frac{p_i}{q_t} \in (0,\infty)\) for \(q_i\); 205 calls Boltzmann’s constant \(k_b\) where 200 and 203 write \(k_B\); and the \(-k\) in {eq}`eq:Shannon5` at 331 is the only symbol in the formula that is never defined.

  • [qe-writing-005] (reviewer) — Use bold for definitions, italic for emphasis. Count: 6. Lines: 146, 264, 273, 321, 421. Example: bold is used well for the defined terms - uncertainty (25), surprisal (77, 81), expected surprisal (84), conditional surprisal (166), relative entropy (220), entropy bound (391), frequency domain entropy (472), maximum entropy (490) - which makes the four exceptions stand out: **Claim** (264), **Proof** (273), **Extension** (321) and **Problem:** (421) are structural labels, not definitions, and each stands alone on its own line as a heading. In the other direction, the two properties named at 146-147 are defined in italic - \(H\) is continuous, \(H\) is symmetric - where the same list’s other four items simply state the property, so within five lines the lecture both italicises a definition and leaves definitions unmarked.

  • [qe-writing-008] — Remove excessive whitespace between words. Count: 47. Lines: 40, 48, 64, 77, 78, 110, 125, 128, 156, 180, …. Example: 2 spaces.

Medium severity#

  • [qe-fig-002] — Prefer code-generated figures. Count: 4. Lines: 115, 519, 526, 535. Example: static image .png.

  • [qe-fig-004] — Caption formatting conventions. Count: 4. Lines: 115, 519, 526, 535. Example: caption of 8 words.

  • [qe-link-001] — Use markdown style links for lectures in same lecture series. Count: 2. Lines: 33, 34. Example: full URL to own series (python-advanced.quantecon.org).

  • [qe-math-009] (reviewer) — Choose simplicity in mathematical notation. Count: 3. Lines: 200, 341, 450. Example: a lecture whose subject is that one word names many objects reuses two letters for three objects each. \(S\) is Gibbs thermodynamic entropy at 200-207, von Neumann entropy at 351, and the spectral density at 402, 407, 413, 429, 444, 462 and 475 - so \(S\) appears in {eq}`eq:Gibbs` and in {eq}`eq:Shannon6` meaning unrelated things nine sections apart. \(D\) is Kullback-Leibler divergence at 224-246 and 329, the von Neumann divergence at 341, and the Wold lag polynomial at 450-462 and 484, which puts \(D(p|q)\) and \(D(0)D(0)'\) in the same lecture. Separately the same two operators are spelled four ways: \mathrm{trace} (269, 289, 304, 311), \mathrm {trace} (329), \textrm{trace} (341, 351), and \mathrm {det } with the space inside the braces (330, twice) beside plain \det (268, 310, 330, 462, 475, 484); and the matrix logarithm is \ln at 341 and 351 where every other logarithm in the lecture is \log.

  • [qe-math-014 (proposed)] (reviewer) — Braces {…} for events, parentheses (…) for sets. Count: 2. Lines: 44, 92. Example: both probability statements in the lecture write an event with parentheses: \(p_i = \textrm{Prob}(X = x_i)\) at 44 and \(p = {\rm Prob}(X=1)\) at 92. X = x_i is an event, not a named set, so the rule asks for braces; converting the Roman operator to \mathbb{P} (qe-math-010 (proposed)) and the parentheses to braces gives \(\mathbb{P}\{X = x_i\}\) and fixes both rules in one edit. These are the lecture’s only two probability expressions, so the whole of its use of the notation is affected.

  • [qe-ref-001] — Use correct citation style. Count: 3. Lines: 385, 471, 481. Example: {cite} in author position: ‘{cite}`Backus_Chernov_Zin` note’.

  • [qe-writing-002] (reviewer) — Keep writing clear, concise, and valuable. Count: 4. Lines: 92, 128, 151, 338. Example: 92 defines the same letter twice inside one clause and misuses set membership: “Take a possibly unfair coin, so \(X = \{0,1\}\) with \(p = {\rm Prob}(X=1) = p \in [0,1]\)” - \(X\) takes values in \(\{0,1\}\) rather than equalling the set, and the chain asserts \(p = p\). 151 drops two words, “Entropy is not affected by events zero probability”. 338 reads “A measure of the divergence between two \(P\) and \(Q\)”. 128 has “Among all dies, a fair die maximizes entropy”, where 125-126 has already introduced the die correctly. 344 also leaves the matrix logarithm defined by a bare parenthesised URL, “where the log of a matrix is defined here (https://en.wikipedia.org/wiki/Logarithm_of_a_matrix)”, as does 197 for Gibbs.

  • [qe-writing-004] — Avoid unnecessary capitalization in narrative text. Count: 1. Lines: 33. Example: mid-sentence ‘Model’.

  • [qe-writing-007] (reviewer) — Use visual elements to enhance understanding. Count: 3. Lines: 23, 435, 519. Example: the lecture’s own frame is a list of six senses of entropy (23-30) taken up in eight sections, and it never comes back as a table - one row per sense with its symbol, its formula and the section it lives in would be the single most useful object on the page, and it is exactly what a reader who arrives to reconcile \(H(p)\), \(S\), \(D(p|q)\), \(L_t\), \(\sigma^2_\epsilon\) and the frequency-domain criterion needs. The claim the lecture builds to at 432-436, that a flat spectral density is the time-series analogue of the uniform distribution, is asserted in prose where two small panels - a flat \(S_x(\omega)\) beside a uniform \(p\) - would show it, and the lecture plots nothing at all in its five frequency-domain and prediction sections (399-490). All four figures that do exist are static raster files checked into the repository (115, 519, 526, 535), one of them a .jpg (526), so none can be regenerated when the notation changes - which is how the caption at 119 came to describe a figure in symbols the lecture does not use. There are also no admonitions at all: the Claim, Proof, Extension and Problem at 264-421 would be {prf:} directives, and 62 (“we set \(p \log p = 0\)”) is a convention that belongs in a {note}.

Low severity#

  • [qe-fig-005] — Descriptive figure names for cross-referencing. Count: 1. Lines: 115. Example: non-descriptive name fig1.

Strengths#

  • The lecture does exactly what its opening list promises: the six senses of entropy set out at 23-30 are then taken in that order - information theory (38), thermodynamics (195), Kullback-Leibler divergence (212), Backus-Chernov-Zin volatility of stochastic discount factors (356), Wiener-Kolmogorov prediction error (399) and frequency-domain robust control (469) - so a reader can enter at whichever sense they arrived for.

  • One theme is stated four times in four settings and then named: the fair coin maximises entropy (100-110), the fair die maximises it (125-131), independence maximises conditional entropy (179-191), and a flat spectral density maximises one-step-ahead prediction error (423-433), with 435-436 saying explicitly that the “no-patterns-across-time outcome for a temporally dependent process resembles the no-pattern-across-states outcome” of the coin and the die.

  • Shannon entropy is given a meaning before it is manipulated: the surprisal \(s_i = \log(1/p_i)\) is defined and explained as inversely related to the likelihood of state \(i\) (77-82), which makes \(H(p)\) the expected surprisal (84-88); the same move is then repeated for the conditional case, where \(p(y_j)/p(x_i,y_j)\) is named the conditional surprisal (166).

  • The Gaussian relative-entropy formula is proved rather than quoted (264-315) - the log-likelihood ratio at 277-282, the two expectation computations at 287-305, the combination at 309-313 - and then read for its content: 316-317 points out that the mean distortion \(w\) and the covariance distortion \(\Sigma - I\) appear separately in {eq}`eq:relentropy`.

  • The inequality that makes relative entropy non-negative is both proved and drawn: 508-509 reduces \(\textrm{ent}(g) \geq 0\) to \(g\log g \geq g-1\) together with \(Eg = 1\), and the two remaining figures show the ratio of an \({\mathcal N}(0,1)\) density to a wider and a narrower Gaussian, so the reader sees the bound bind from both sides.

  • The base-of-logarithm question that trips new readers of information theory is disposed of in four lines (64-69): the three usual bases, the names bits, nats and dits, and the note that Shannon used base 2 - and then 133-138 shows what a base-2 count means by evaluating \(\log_2 n\) for \(n = 2\) and \(n = 3\).

  • Every notion arrives attached to where it came from: Shannon (25, 40, 48), Gibbs with the role of Boltzmann’s constant spelled out (197-205), Kullback and Leibler (220), von Neumann and the density matrix (334-352), Backus-Chernov-Zin with their bound offered explicitly as a complement to Hansen-Jagannathan (391-397), Sargent chapter XIV for Wiener-Kolmogorov (410), and Hansen-Sargent chapter 8 for the frequency-domain criterion (471-490).