情報量等式の証明

任意に$\mathit{\theta }\in \mathrm{\Theta }$を固定し, $\mathcal{N}\subset \mathrm{\Theta }$を$\mathit{\theta }$の近傍とする. このとき, 一般性を失わずに$\mathcal{N}$は凸であると仮定してよい. $\mu$-a.e.$x\in \mathcal{X}$について, 平均値の定理より, $\tilde{\mathit{\theta }}\in \mathcal{N}$に対して
$$\begin{array}{r}q(x,\tilde{\mathit{\theta }})=q(x,\mathit{\theta })+{\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }q(x,\mathit{\theta })(\tilde{\mathit{\theta }}-\mathit{\theta })+r(x,\tilde{\mathit{\theta }})\end{array}$$
が成り立つ. ここで,
$$\begin{array}{r}r(x,\tilde{\mathit{\theta }})=\{{\int }_0^1{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta }+u(\tilde{\mathit{\theta }}-\mathit{\theta }))du-{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta }){\}}^{\prime }(\tilde{\mathit{\theta }}-\mathit{\theta })\end{array}$$
である. 今, $\mu$-a.e.$x\in \mathcal{X}$について
$$\begin{array}{r}\frac{\Vert r(x,\tilde{\mathit{\theta }})\Vert }{\parallel \tilde{\mathit{\theta }}-\mathit{\theta }\parallel }\le {\int }_0^1\Vert {\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta }+u(\tilde{\mathit{\theta }}-\mathit{\theta }))-{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })\Vert du\le M(x)\end{array}$$
であるから, Lebesgueの収束定理を適用すると, $\mathit{\theta }\mapsto {\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })$の連続性より
$$\begin{array}{rl}\underset{\tilde{\mathit{\theta }}\to \mathit{\theta }}{lim}{\int }_{\mathcal{X}}\frac{\Vert r(x,\tilde{\mathit{\theta }})\Vert }{\parallel \tilde{\mathit{\theta }}-\mathit{\theta }\parallel }\mu (dx)& ={\int }_{\mathcal{X}}\underset{\tilde{\mathit{\theta }}\to \mathit{\theta }}{lim}\frac{\Vert r(x,\tilde{\mathit{\theta }})\Vert }{\parallel \tilde{\mathit{\theta }}-\mathit{\theta }\parallel }\mu (dx)\\ & \le {\int }_{\mathcal{X}}\underset{\tilde{\mathit{\theta }}\to \mathit{\theta }}{lim}\underset{0\le u\le 1}{max}\Vert {\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta }+u(\tilde{\mathit{\theta }}-\mathit{\theta }))-{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })\Vert \mu (dx)\\ & =0\end{array}$$
となる. したがって,
$$\begin{array}{rl}\Vert {\int }_{\mathcal{X}}q(x,\tilde{\mathit{\theta }})\mu (dx)-{\int }_{\mathcal{X}}q(x,\mathit{\theta })\mu (dx)-({\int }_{\mathcal{X}}{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })\mu (dx){)}^{\prime }(\tilde{\mathit{\theta }}-\mathit{\theta })\Vert & =\Vert {\int }_{\mathcal{X}}r(x,\tilde{\mathit{\theta }})\mu (dx)\Vert \\ & \le {\int }_{\mathcal{X}}\Vert r(x,\tilde{\mathit{\theta }})\Vert \mu (dx)\\ & =o(\parallel \tilde{\mathit{\theta }}-\mathit{\theta }\parallel )\end{array}$$
であるから,
$$\begin{array}{r}{\mathrm{\nabla }}_{\mathit{\theta }}{\int }_{\mathcal{X}}q(x,\mathit{\theta })\mu (dx)={\int }_{\mathcal{X}}{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })\mu (dx)\end{array}$$
が成り立つ. 写像
$$\begin{array}{r}\mathrm{\Theta }\ni \mathit{\theta }\mapsto {\int }_{\mathcal{X}}{\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })\mu (dx)\in {\mathbb{R}}^{k\times p}\end{array}$$
の連続性は$\mathit{\theta }\mapsto {\mathrm{\nabla }}_{\mathit{\theta }}q(x,\mathit{\theta })$の連続性とLebesgueの収束定理による. (証明終)

補題1より微分と積分の順序が交換できて,
$$\begin{array}{rl}\mathbb{E}[{\mathrm{\nabla }}_{\mathit{\theta }}\log f(X;{\mathit{\theta }}_0)]& =\mathbb{E}[\frac{{\mathrm{\nabla }}_{\mathit{\theta }}f(X;{\mathit{\theta }}_0)}{f(X;{\mathit{\theta }}_0)}]\\ & ={\int }_{\mathcal{X}}\frac{{\mathrm{\nabla }}_{\mathit{\theta }}f(x;{\mathit{\theta }}_0)}{f(x;{\mathit{\theta }}_0)}f(x;{\mathit{\theta }}_0)\mu (dx)\\ & ={\int }_{\mathcal{X}}{\mathrm{\nabla }}_{\mathit{\theta }}f(x;{\mathit{\theta }}_0)\mu (dx)\\ & =({\mathrm{\nabla }}_{\mathit{\theta }}{)}_{{\mathit{\theta }}_0}{\int }_{\mathcal{X}}f(x;\mathit{\theta })\mu (dx)\\ & =({\mathrm{\nabla }}_{\mathit{\theta }}{)}_{{\mathit{\theta }}_0}1\\ & =\mathbf{0}\end{array}$$
となる. この結果を踏まえて, 再び補題1を用いることにより
$$\begin{array}{rl}\mathbb{E}[{\mathrm{\nabla }}_{\mathit{\theta }}^2\log f(X;{\mathit{\theta }}_0)]& =\mathbb{E}[({\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }{)}_{{\mathit{\theta }}_0}\frac{{\mathrm{\nabla }}_{\mathit{\theta }}f(X;\mathit{\theta })}{f(X_1;\mathit{\theta })}]\\ & =\mathbb{E}[\frac{{\mathrm{\nabla }}_{\mathit{\theta }}^{2}f(X;{\mathit{\theta }}_0)}{f(X;{\mathit{\theta }}_0)}]-\mathbb{E}[\frac{({\mathrm{\nabla }}_{\mathit{\theta }}f(X;{\mathit{\theta }}_0))({\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }f(X;{\mathit{\theta }}_0))}{f^2(X;{\mathit{\theta }}_0)}]\\ & ={\int }_{\mathcal{X}}{\mathrm{\nabla }}_{\mathit{\theta }}^{2}f(x;{\mathit{\theta }}_0)\mu (dx)-\mathbb{E}[({\mathrm{\nabla }}_{\mathit{\theta }}\log f(X;{\mathit{\theta }}_0))({\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }\log f(X;{\mathit{\theta }}_0))]\\ & =({\mathrm{\nabla }}_{\mathit{\theta }}{)}_{{\mathit{\theta }}_0}{\int }_{\mathcal{X}}{\mathrm{\nabla }}_{\mathit{\theta }}f(x;\mathit{\theta })\mu (dx)-\mathbb{E}[({\mathrm{\nabla }}_{\mathit{\theta }}\log f(X;{\mathit{\theta }}_0))({\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }\log f(X;{\mathit{\theta }}_0))]\\ & =({\mathrm{\nabla }}_{\mathit{\theta }}^2{)}_{{\mathit{\theta }}_0}{\int }_{\mathcal{X}}f(x;\mathit{\theta })\mu (dx)-\mathbb{E}[({\mathrm{\nabla }}_{\mathit{\theta }}\log f(X;{\mathit{\theta }}_0))({\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }\log f(X;{\mathit{\theta }}_0))]\\ & =-\mathbb{E}[({\mathrm{\nabla }}_{\mathit{\theta }}\log f(X;{\mathit{\theta }}_0))({\mathrm{\nabla }}_{\mathit{\theta }}^{\prime }\log f(X;{\mathit{\theta }}_0))]\end{array}$$
を得る. (証明終)