Motivation
Random partitions of the positive integers N={1,2,…} are the common language of population genetics, Bayesian nonparametrics and the theory of coalescent processes. When a sample of genes, customers or particles is grouped into classes, the grouping is a partition, and in most models its law is exchangeable: it does not depend on the labels. The best-known example is the Ewens sampling formula of population genetics (Ewens 1972); Kingman's representation theory (Kingman 1978) describes all exchangeable random partitions of N.
Pitman's two-parameter family of exchangeable partitions, the (α,θ) partitions (Pitman 1995; Pitman and Yor 1997), contains Ewens' family as α=0 and is the partition counterpart of the two-parameter Poisson–Dirichlet distribution. Theorem 12 of Pitman 1999 shows that two natural random operations act on this family in a dual way: merging blocks of an (α,θ) partition according to an independent exchangeable partition (coagulation) and splitting each block of an (αβ,θ) partition by independent exchangeable partitions (fragmentation) produce the same joint law of a fine and a coarse partition. The paper uses it to describe the Bolthausen–Sznitman coalescent (Bolthausen and Sznitman 1998).
Timeline. 1972: Ewens' sampling formula, the case α=0. 1978: Kingman's correspondence between exchangeable partitions and random mass partitions. 1995: Pitman introduces exchangeable partition probability functions and the (α,θ) formula (15). 1997: Pitman and Yor study the two-parameter Poisson–Dirichlet law. 1999: this paper proves the coagulation–fragmentation duality (Theorem 12) and its sharper converse, by reduction to an identity between explicit formulas. The duality is treated again in Pitman's lecture notes Combinatorial Stochastic Processes (2006).
Setting
A partition of a set is a collection of disjoint nonempty blocks whose union is the set. Write Pn for the partitions of [n]={1,…,n} and P∞ for those of N; the restriction Rnπ of π∈P∞ to [n] keeps the nonempty sets A∩[n]. P∞ carries the σ-algebra generated by the Rn. Writing π={A1,A2,…} lists the blocks in increasing order of least elements.
A random partition Π of N is exchangeable with EPF p if for every n and every partition {B1,…,Bk} of [n],
P(RnΠ={B1,…,Bk})=p(∣B1∣,…,∣Bk∣).
For 0≤α<1 and θ>−α the (α,θ) EPF is
pα,θ(n1,…,nk)=[θ]n[θ/α]ki=1∏k−[−α]ni,[x]m=i=1∏m(x+i−1),n=∑ini,
with factors of α and θ cancelled before evaluation when α=0 or θ=0. An (α,θ) partition is an exchangeable random partition with EPF pα,θ.
Coagulation (Definition 5): for π={A1,A2,…} and γ={B1,B2,…}, the γ-coagulation of π has blocks ⋃j∈BiAj. Π′ is a p-coagulation of Π if, given Π=π, it is the γ-coagulation of π for an independent γ with EPF p. Fragmentation (Definition 11): Π is a p-fragmentation of Π′ if, given Π′=π′, Π restricted to the m-th block of π′ equals Γ(m) restricted to that block, for independent Γ(1),Γ(2),… with EPF p. In Lean these are coag, frag, pdEPF and IsEPFLaw in LambdaCoalescent.CoagFrag.Setting.
Formalization targets
Goal: Theorem 12
For 0<α<1, 0≤β<1, θ>−αβ, the following are equivalent: (i) Π is an (α,θ) partition and Π′ is a (β,θ/α)-coagulation of Π; (ii) Π′ is an (αβ,θ) partition and Π is an (α,−αβ)-fragmentation of Π′. Formally, with laws μ,ν,ρ,κ of EPFs pα,θ,pβ,θ/α,pαβ,θ,pα,−αβ:
(μ⊗ν){(π,γ):(π,coag(π,γ))∈S}=(ρ⊗κ⊗N){(π′,Γ):(frag(π′,Γ),π′)∈S}
for every measurable S⊆P∞×P∞.
Milestones
- Lemma 9 (existence): for 0≤α<1, θ>−α a law with EPF pα,θ exists.
- Lemma 34: Π2 is a p-coagulation of Π1 with EPF p1 iff P(Πn1=π1,Πn2=π2)=p1(a1,…,aK)p(j1,…,jk) for refining pairs.
- Lemma 35: the fragmentation analogue, ∏ip^(ai,1,…,ai,ji)p2(b1,…,bk).
- The EPF identity in the proof of Theorem 12: (62) equals (63) for the four laws of Theorem 12.
- The sharper form: for parameters in PAR={0≤α<1,θ>−α} the two joint laws agree iff αf=α, θf=−α1=−ααc, θc=θ/α, θ1=θ (64).
Significance
Theorem 12 identifies one law of a nested pair of exchangeable partitions from both ends. Read from (i) to (ii), it shows that coagulating an (α,θ) partition by an independent (β,θ/α) partition yields an (αβ,θ) partition; read backwards, that an (αβ,θ) partition fragmented by (α,−αβ) partitions yields an (α,θ) partition. Through Kingman's correspondence it gives Corollary 13 on Poisson–Dirichlet laws, and with β=0 it connects the two-parameter family with Ewens' family. In the paper it is the tool for describing the Bolthausen–Sznitman coalescent through Poisson–Dirichlet laws. The sharper form shows the parameter relations (64) are forced.
The theorem has been proved since 1999. What this mission adds is a machine-checked development: exchangeable random partitions of N with their EPFs, the coagulation and fragmentation kernels, and the reduction of identities between laws on P∞ to identities between finite-dimensional formulas. None of these objects is in Mathlib, and no formalization of the duality is known.
Difficulty
The algebraic heart, the EPF identity, is a finite computation with rising factorials. The difficulty is measure-theoretic and combinatorial. A law on P∞ must be pinned down by its values on the cylinder events {Rnπ=πn}. Computing P(RnΠ=π1,RnΠ′=π2) under the coagulation kernel requires that the blocks of π meeting [n] are exactly the first K blocks in order of least elements, and that the induced partition of [K] has law given by the EPF; under the fragmentation kernel it requires the independence of the Γ(m) and that the restriction of an exchangeable partition to an arbitrary finite set (not an initial segment [b]) is governed by the same EPF. The converse in Lemmas 34 and 35 needs the bookkeeping that the formula on refining pairs exhausts total mass. Lemma 9 is an existence theorem for a projective family of laws, which the paper cites rather than proves.
Formalization scope
A partition of N is an equivalence relation on Lean's N={0,1,…} (PInf := Setoid ℕ), a partition of [n] one on Fin n; labels shift by one, which preserves the order of least elements, and block indices are 0-based. The σ-algebra on PInf is generated by the coordinates i∼πj, equivalently by the Rn. Coagulation is stated with γ∈P∞ (Definition 5's case n=∞). The EPF is the cancelled form
pα,θ(n1,…,nk)=[θ+1]n−1∏i=1k−1(θ+iα)i=1∏k[1−α]ni−1,
equal to (15) for α,θ=0; this is needed because θf=−αβ=0 at β=0. Conditional laws "for all π" are encoded as joint laws, and equalities of laws are stated on preimages of measurable sets, so no Measure.map of an unproved-measurable map appears. Fragmenting partitions are one sample of Measure.infinitePi (fun _ => κ). In the sharper form, θc=θ/α is written as αθc=θ, since PAR allows α=0.
Not formalized: the stick-breaking characterization (12)–(13) in Lemma 9, Corollary 13 and Kingman's correspondence, and the auxiliary fact "∏[−α]ni=cn,k∏[−β]ni⇒α=β", which as printed fails at α=0.
A formalization in which no measure satisfies IsEPFLaw (pdEPF α θ) would make Theorem 12 and Lemmas 34–35 vacuous; Lemma 9 rules this out and must be proved for the parameters used, not assumed.
Infrastructure needed: cylinder-set uniqueness on PInf, the law of the restriction of an exchangeable partition to a finite set, block enumeration by least elements, and rising-factorial algebra (ascPochhammer). The partition, EPF and kernel definitions are reusable for any work on exchangeable partitions, Chinese restaurant processes or Λ-coalescents. Proofs of Lemma 9, of Lemmas 34–35 and of the EPF identity are each welcome as independent contributions.
Selected references
- J. Pitman, Coalescents with multiple collisions, Ann. Probab. 27(4) (1999), 1870–1902. https://doi.org/10.1214/aop/1022874819
- J. Pitman, Exchangeable and partially exchangeable random partitions, Probab. Theory Related Fields 102 (1995), 145–158. https://doi.org/10.1007/BF01213386
- J. Pitman and M. Yor, The two-parameter Poisson–Dirichlet distribution derived from a stable subordinator, Ann. Probab. 25(2) (1997), 855–900. https://doi.org/10.1214/aop/1024404422
- J. F. C. Kingman, The representation of partition structures, J. London Math. Soc. (2) 18 (1978), 374–380. https://doi.org/10.1112/jlms/s2-18.2.374
- W. J. Ewens, The sampling theory of selectively neutral alleles, Theor. Popul. Biol. 3 (1972), 87–112. https://doi.org/10.1016/0040-5809(72)90035-4
- E. Bolthausen and A.-S. Sznitman, On Ruelle's probability cascades and an abstract cavity method, Comm. Math. Phys. 197 (1998), 247–276. https://doi.org/10.1007/s002200050450
- J. Pitman, Combinatorial Stochastic Processes, Lecture Notes in Mathematics 1875, Springer, 2006. https://doi.org/10.1007/b11601500