Christopher M. Bishop, PRML, Chapter 1 Introdcution
2.2 Noiseless coding theorem (Shannon, 1948)
The noiseless coding theorem states that the entropy is a lower bound on the number of bits needed to transmit the state of a random variable.
2.3 Alternative view of entropy H(x)
Secondly, let us introduces the concept of entropy in physics in the context of equilibrium thermodynamics and later given a deeper interpretation as a measure of disorder through developments in statistical mechanics.
Consider a set of
identical objects that are to be divided amongst a set of bins, such that there are
objects in the
bin. Consider the number of different ways of allocating the objects to the bins.
- There are
ways to choose the first object,
ways to choose the second object, and so on, leading to a total of
ways to allocate all
objects to the bins.
- However, we don’t wish to distinguish between rearrangements of objects within each bin. In the
bin there are
ways of reordering the objects, and so the total number of ways of allocating the
objects to the bins is given by
which is called the multiplicity.
- The entropy is then defined as the logarithm of the multiplicity scaled by an appropriate constant
![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAn0AAABMCAYAAAAYygHsAAAfU0lEQVR4Ae2dCXgURdrHXzlF8eAWD3A55D7kVMAFRI5wCPgpgstN2CVg8OPjUhQBOQMSDpcQbhfk0oUsEiQo9+HqBlFODbiKgKiEKyBxSSL9Pf/K9tAz6UlmMt2Tmcy/nifp7urqqurfW93zdtVbb92haZomDCRAAiRAAiRAAiRAAvmaQIF8fXe8ORIgARIgARIgARIgAUWASh8bAgmQAAmQAAmQAAmEAAEqfSEgZN4iCZAACZAACZAACVDpYxsgARIgARIgARIggRAgQKUvBITMWyQBEiABEiABEiABKn1sAyRAAiRAAiRAAiQQAgSo9IWAkHmLJEACJEACJEACJEClj22ABEiABEiABEiABEKAAJW+EBAyb5EESIAESIAESIAEqPSxDZAACZAACZAACZBACBCg0hcCQuYtkgAJkAAJkAAJkACVPhvaAJYzjouLk0YNG8nNmzdtKIFZ2kWAsrOLrH/ypfz8w5mlkAAJBCeBQsFZ7cCrdUZGhkRHR0vyhWT59NNP5eTJk6qSN369IUWLFg28CrNGDgKUnQNFUO5QfkEpNlaaBEggDwiwp89C6EWKFJGGDRvKuvXr5N5777UwZ2ZlNwHKzm7C9uZP+dnLl7mTAAnkDwJ3aBgPYbCcQOVKleXatWuSlJQkJUuVtDx/ZmgfAcrOPrb+yJny8wdllkECJBCMBNjTF4xSY51JgARIgARIgARIwEsCVPq8BMbkJEACJEACJEACJBCMBKj0BaPUWGcSIAESIAESIAES8JIAlT4vgTE5CZAACZAACZAACQQjAbpsCUapsc4kQAJ+I9C/X3/ZsmWL38rTCwoPD5fpM6brh9ySAAmQgM8EqPT5jJAZkAAJ5GcCnTp3clL6ChUqJMtXLJdHHnnEq9uGP8HU1FT1B/+dKddS5Ny5c5L0TZIkJiZKcnKyU35r166V18a9RvdPTlR4QAIk4AsBKn2+0OO1JEAC+Z7ACy+8IHv37JV169ape4Xy9s4778imTZukcOHCltw/PGedOnVKli5ZKlD2/vOf/8iNGzdk3dp18ue//NmSMpgJCZAACdCmj22ABEiABHIgMCNqhlSqVMmRKvFfiTJz5kzHsa87d9xxhzz22GMyc9ZMOXzksEQMjVBZLlmyRG7duuVr9ryeBEiABBQBKn1sCCRAAiSQA4G7775bli5bKlj5Qw/z5s6Tffv26YeWbUuWLClvvfWWLF26VM6ePSs7duywLG9mRAIkENoEqPSFtvx59yRAAh4SqFOnjkyYMMGRGkOyEUMi5NKlS444K3e6dusqU6dNlSWLl1iZLfMiARIIYQJU+iwWPux9zp45q2xykPW///1vSUtLs7gUZmcHAcrOmSqUGiwlGCzBH/Ib/OfB0q5dOweSX375RV4e9rLYtZrlgAED1Lvk9OnTjjLzy06wta/8wp33EdoEbF97Fy+rgwcPSuFChaVwkcJSoEAByUjPkPSMdGUE3blzZ7ly5Yps375dihQuIoUKF5KCBQsKXuBIl5aeJt26dXMaVglUkTVt0lR++OEHVf+iRYsK7HRu3ryp7gXDQqe+PSWI9yac+eGM7Ni5Qy5fuiwXL12UXr16Sd26db3JIse0mDmIYSTUDfJJT0uXm2k35fHHH5cqVaqo67dt26ZmHer3lXYzTaV5ps0zjrWFr1+/LgkJCVK0SFEla+RTomQJeeqpp3KsQ14nsEN2uCd/yM/I7tdff1UywLODUKBgATHKyJjWKHe0Vcj0rrvvUkrN77//LmEdwuTo0aMyZ+4c6dmzp/HSgNu3S35mN4pnsWXLlvLzzz87Tk+ePFmGRAxxHPtjx99ty8p7Crb2ZeW9M6+8IYDfYsy8h35hFlJSUuSuu+7K1eQs/PZhJn6pUqWkbNmyZtnnGIcPbHxEVq1aVXUU4Z1s1UQxY+G2z969fPmyHDhwQPV+7dmzR5Vdvnx5adyksVIooPT99ttv8vlnn8tPP/0ku3fvVjd8zz33SIsWLaRcuXLSpUsXY50Ddn//gf2qUUFYxoAvWiixaHDehgvJF2TP7j3y6aefKuUYCpTVSt/x48dl+yfbBYodgi6fCo9UUDKCITnk89nnnwkM2BGgENavX1+eaPqEQ+mDywnkA59meMBQz+effz4olD47ZAdO/pCfEsh//0EGX375pSRfSJa4uDgVi9mnMQtjjMnU/uGvDqsZqJ999plS9qHINGvWTCl9KVdTVD5IeGD/gYBX+uySXxZoIqq9xy6Kle7dujt6+GCD92SzJ6VevXpml9gS5++2ZeVNBFv7svLemZf/CeADafz48WqiFHQKY4CytSh2kSxcuFCgozxSwXNXTJhh/+rYV1WnVc2aNQW/pZWrVJaYmBipWLGisRjH/oULF6RjWMcsE7SgK+3du1elO3nypETNiJLoOdFSpkwZx7WW7Gh+DFUqV9FKlyqtxSyIcVtq506dVZqhEUPdpgnFExMnTFRctmzZYsvtp6ena2XLlFVlxMfHm5aRmJiozkOG586dM02DyF49e2mo761bt9ymCbUTdsvPlWdCQoJWv159Ja/yD5TXLly44JrEcdywQUPNVeYZGRmO9jDhzQmOtNy5TWD69OmO5wHPRONGjbXr16/fTuCnPX+3LStui+3LCorMwxMCO3fs1Nq3a6+dP3/eKTme1TnRc7SqVao6nuMzP5xxSpPdQUpKivZUi6e0ts+01X777TeVFHk+/z/Pa49VfUw7dOiQ6eVTJk9xlIf3hv7Xt09fp/SnTp3SWrdqrSUlJTnF+3rgV5s+DB0i6FszrVU/p2/N0oRiHIbG7Qzohby/xP2qCAwRmgXjFwd68swChrcxXD/u9XFqeNssTSjG2S0/V6boXY8cHimVK1eW9PR0WbVylWsSdQx/cJBlhw4dnM5jCKREiRIqrnSZ0k7neJBJYNSoUdKkaRMHju+//17GjhnrOPbXjr/blhX3xfZlBUXmkRMB9LxFRETI3Hlz1QiWMT38YjZv3lz10hnjPd2fP2++fP3118qB+p133qkuK168uOqdw2/gqJGjsvTmoWdw5cqVyufnJ9s/EePfkqXOE7ZgWjVl6hTp/afeyrzL03rllM6vSl9OleH5vCVQ4v7MH/krl6+YVmTz5s2OeHRFm4WJEyfKpEmTbLFFMCuPceYEMCQLU4BBgwapBO+++65S/lxTw962YcOGpnYuupJfujSVPlduOMaH0qJFi+S+++5znH7//fcFfww5E2D7ypkRU+SeABSvfn37KVvb6tWrZ8kIJkowMyv3gPNwb5aEJhEwSYuNjVVnnnzySacUWKmndu3acuTIEfloy0dO595b9Z60atVK2jzTRplHwURK/zO6g9IvgrkN6jlw4MAsCqSextstlT5vieXj9PAPhnD5SlaF7uLFi/Lhpg8dd48HyjWgdwmTcYy9H65peGw/ARgkw24EX4ov9nxRGSfDXnbrR1uzFA6Z4WvXLOjKXpnSFtuUmBUWpHEPP/ywzJs/z6n2Y0aPke+++84pjgdZCbB9ZWXCGOsIwHk6Jihi8qPVAV45MEKCHj69l89Yhq5k7tq1yxGNERcoipi0it472P158p6Ao3a8p1evXu3Iy5cd72cW+FIar7WUAIw9P/74Y9X1e+nyJRkxYoTqdVi1apWcTDqpDFLbt2+vviQ8KVgfzjPr6Zs+bbqMGTtGuaeAXzJX32SYjTd1ylRx7aL2pNxQTGO17IwMMTHjiSefUMPr9957r/To0UPQ0wdnv892fdaYVPbv2+92ZQl9WFffOl3IAweBTp06qS/x5cuXqzgM4QwOHyxbE7bmmdcBq9oX3E3hxwmG8P3691MTVfCsx2+Ol3/+85+q9zisY5g888wzDh6e7ujtSt96eh3TkUBOBDCbdu2atdL66dZqMmhO6b09r7tQwvOBiY6u5mgPPPCAyvLbb791ZB0fH69m+CICkybxN2niJOnZq6dERUWZKo9Ii8lhDz30kMyJniMvvfSS6aiMoxAPdtjT5wGkQE2CWUcnjp+Q1WtWy5rVa9Si7YMGDpLatWrL2FfHSuqNVGnfrr2aEe3JPbjr6YNdBNxT4MWup3FVDPEV0rp1a9U4PSkr1NNYLTsjT9feu4GDBqrT+JFGe9EDhijw8qpWvZoe5bTVe2L0rdNJHjgRmPTWJKlRs4YjDkM7cOOSV8Gq9jVr1iyBJwUMWXXp3EXNTsQPz7kfzyklEB8RvXr2EgxbeRv0dqVvvb2e6UnAHYH169ertavr1K7jLolP8froBxQ+s966IkUzV+5BT6MemjZtqrwojB07Vho0aKCicT1+u+HlAh4+zAK8gdSoUUP1WlqxAlCe9PThZehu3Up8JXsboC1v2LDB28tU+vvvv18StiWoIbBcZZCHFzVq1EjwN27cOIHX/jfeeEN1A+sv0fDB4eorPWZBjLIjyKmq8KmHYFTo4G4GfLEyAII+2cM4BIwfmJV/Wymb42/b/KnEHv4LRflZLTsjatjz9e7d2xGFFwaGcKEMLlu2TGZHz1bnYM+Herh+peoX6i82vT3p8dxmJYAhHqyTC5+ImByDELswVvnzy00vWNYSvIuxon2hFwMmAXv37RWMHuAjoVPHThL3jzhlZ4QaYRgLvs3ee+896d3ndpsz1hb+y+AGytU/GtuXkRL3rSRwMPGgyi439nqe1KN2ndoC3eHq1auy6R+bZOSokU6XQZlDKH5PcUf8gw8+KHCfhTBq9Ci1cAN++7Zu3apcoq1YvkLg/N0sYF1uDAtjFAc2gb6EPFH64C/nL0P+Ylrvrs92VT7pTE+6iQSoVq1zBwJG2HhpBXOA42uE5557Tow/0GhkeNHqXdE53aM+kcM4SeOjjz5SPUFwGIlQskSm3Z9RMYyeHS3DXh4mxYoVy6kI0/OhLD+rZKeDhT2f7uBTj8M2PDxcKX0ffPCBjH9zvHphqR7BFub2fLgGw26YjeatQ3FjuaG0X61aNZk+fboys9Dve9iwYcr3lz7co8f7a+tL+0L7wGQgfBQcPXJUVTkyMtKh8CECtqOpqalSsJC5w9uvvvpK2rVtJ/PmzZNeLznbVrF9+asVhF458CKB8EC5zGFWqwlgLW4821OnTpXZs2crEypMzoDpw+5du2X9uvWqyEcedu/zD54V/rbybzJ8+HBZt3ad6qBxp/Shpx0h6Zskn28lT5Q+n2vtkgGUG/zZHdA7OWvmLMuLGT1mtIwZM8bnfPGjYwxQ+PDnae+po6fvv5M08KX/zvx3ZP37mQ0Yeetp9J4+dG0fPnxYJky8vSapsQ6e7FN+Ir7KTudstOfT47DtENZB9bZgQgdsXWAcvH//fsHwnbuADwjjR4S7dIjHyw52KVitwpuA9jkofJDgSzanYMfzZ9Wzp9f9T73/JLv37FZf/4gDj399/q8stpR6en9tc9O+4Fz25ciXVRXRK4xhpgEDBzhV+YsvvlDH+nCV00kRefTRRwWOq7s8m9XBvjftyzVfHpNAdgTOnDmjTttpL/rK/74iN1JvyNw5c5XzeqzEgQ8g2NHDDOr8+fM5LkyAZwojn5gkifTuFnHQJ4ugZ9HXkC+UPl8heHo9FDMrlDNPy/M2Hb4+zII7WwHXtHovnt7TB3cU+Do3uqTQ0+g9feiehsLnugqJa96BcBzI8vNVdjpfV3s+PR7uRbCO67Rp09QQb99+fZVxvqsyoKfHFsNvnip9SI/l94re6d0yg/qHibFcd/uBLD+9zngOoqOj1co1eIljRMN18oye1p/b3LSvSpUqqSpipv6JEyeUGwrdplevu75CT6OGjfQopy2GwNwtT+dt+3LKmAckkA0BffUru9bERtF41l9//XVlSvPloS/VEpZ16mTaEOJ5QB06d+mcTS0zT+EZwWQN2FxjgqTriiFIhfckApav9TXkC6UPM8uOHT+WKxaYserqZydXGeWDi/RePNgkwQAVs4s2bdrkdGf6DF/09GHJmvvuv8/j2cFOGRkOKD8DDB93Xe35jNnB5urtt99W60OjV65RY/f2fLiuZq2aagjDmIe7fbyUXO1a3KXN7/Ho+cawJ7744bMy2AOWgMSPp+sa2ojb9OEmNfz/dJunnW4T57bEbxHMIsasX6xJ6hq8aV+u1/KYBLIjgN8lfHTp9rXZpfX1HHrEjUuuwYsGRsnQYQKXTp6EYncVU2YUUADNQvLFZBVdsYL50m5m17iLyxdKHzxcuyon7m7YNR69WB9u/jDo7fpc7ys3x8av+NGjR8urr77q+MLQ89MVw4vJF9XagCveXaGfyvWW8ss1OqcL3dnz6YngDLdr164Cu76FMQtlRtQM/ZTpFr1Dnnypml4copHw8j+g/wA1wWHR4kVZnp9gxILeY4RmzZs5VR/G8mfPnFVtBDN8sToBeivgSHbK5CnKxjfxYKKyJd2wMetEO7YvJ5w8sJAAzEW+PvG1/HjuRwtzzTkrmEHAfAYTl7BQgacBrl3giNmd/TS8ZyD84Q9/8DRLt+nyhdL3xvg3BH8MvhHQe/H0XFq0aKHvOrb68C56MjB0ZdYV7Ujs4Q7l5yGoHJLhxxke5rMbaof9HJQ+BDP5GovYsmWL7Nm9R/XgWSFnY975cR9DM3BfgglNcKPkbkg12O4d7QptqkmT20vO4R7gIxShY8eOagsvAX369FGjBLD1wyzf+vXqS4UKFdR513/u2pe7eNfreUwC7ghg9A6zapNO5jzxQZ9p6y4v9FpjlR34PA0LC3OXTH309O3TV7k4wgQNYycKLoL9O+z+MDnOGD7/7HP5+aef3S6VibS60ofVk3wNfvHTB2hQEjDlX78B125XpEFPxdWUTENFrADhbg1YX2862K4Hm6tXMrnAlg7HesD+lauZq2MgjfEc+MELOFijuzmnoCt9WA4Gw39mQe/pQ3f2kCFDzJIwzoWAO/lZJTu8tDCM9vast9WkHd0m06Ua6hAG97AfwXBbdpMn0F4ihkTIihUrZPGixWZZMc5AAN75+/btq95zUPj8MbEMxbtrW/o5X98NMByHb8datWo51mLWb/vipYtq949//KNyOgvjeXx0YLZv1MwoQa8Hhth6vNhDv8Sxdde+3MU7LuQOCXhAoHv37qrXzOiX1N1lRj97165fy5IME95eHvayQKFLSsqqRGISG2bfhnUIU+9VuC7D0mnGcOjQIWnapKlUe6yawNsFfpcR0Ds+cuRI9XsLcwezgGf8+LHjah31lq1amiXxKu4ODTnaGOLi4mTY0GGqBCgTMG7ECxIPN5aJOvDpAfVSadu2rZoBiO7NwoULq/NIh2tgr2ecTGBjdb3OGopVQkKCZKRnOlYsULCA8tVVslSmaxNjhomJieorGPeIL+e0m2nK+LNdu3bGZE77GzdulKERQxUTsIAChx/5ffv3KWNx2A+AF/LEOTTAbR9vU4s9Hz16VPU6QMRoZPhKgA1OdgFTzKHMvf7G66bJ8BC1bNlSTTXXv/BNEwZJZF7Jb+jQobJgwQKfZQcje7xM8EGFNgA549mC7R5mkpoFfAHv3LVTudEwO6/Hvfbqa6otLYpdpH7M9fhA2dotO0/vE88XFGQ8q/jCz643wNM8PUnnj3cDHLO3atlK+QId8X8jnKqF90v3bt3VUn94LyyIWeCk7OLdtOHvG+T4ieOq98PpYhFx177cxbtez2MSyI7AhDcnyOLFi+XE1yeyfLDgOihw6NyAmQK8GiBUqFhB2ajjN7Bx48YqDgpbx7COShfZu2+vmpGOE8eOHZOPt33s8BHcv39/6T+gv3qnqwsN/zCLH6Y133zzjYrFRzfMbTCCMmXqFGUOYkjutIvJUvithTso+N71OUDpszNkZGRo+HMNt27d0tLS0lT077//bpoGJ9PT0zWkDdTw3XffaePGjdMGhw/WSpcqrf4ihkSYVnfJ4iVa506dVZqyZcpqLzz/gjYneo5pWj0S7MDHGPQ4fWs8h7T4AzdjAEPXOON5ff+DDz7QUlNT9cMsW8hszeo1AS2TLJXOJiKv5AeOrnL1VXbZ3Ga+PGW37DyFFhUVpZ7pmAUxnl5iSTqz51+P07fGgnLTvvDeOHHihNv38+XLl7XExMQs74wbN25oFStU1IYPH67dvHlTGz9+vLEq3CcB2wlcv35da9SwkTb77dk+l/XTTz9ply5ecspn48aN2oYNG9Tz4YmOgufv8OHD2tatW7Xdu3dryNOTEBkZqYWFhannyJP0OaXB8ACDBQQSEhK0+vXqq5d/+QfKaxcuXHCba8MGDbX4+Hi353nC/wQoP/8zt6rEvJQdPpLwsTfy/0bmmw8hK+Syb98+xQU/cCtXrszx49aKMpkHCbgSSEpK0urVraedPn3a9VRQHO/csVOrXau2xwqiJzflF5s+n7sjgyADGDtHDo9U4+4YYlu1cpVprTEEi2HrDh06mJ5nZN4QoPzyhrsVpeaV7GCA/crwV9SySNNnTM92Ao0v94n70w25fcnHn9fCXxlsRpctXSa7du2SocOG+rN4lkUCigDa4Nx5c5X5BSZaBVOAvR8cN8NG2MoVfaj0WdQK4B8NfqwGDRqkcnz33XcdxprGImDcDNs63dmi8Rz3844A5Zd37H0tOS9k9/3336uJG3ChsGz5MlM7Hl/vC9f/+OOPygVMsC0VCRtsrEzy1wV/leXLlyt7KCt4MA8S8JYA1qpVil9EhFqi0tvr8yI97GXnRM+Rv2/4u9StW9fSKlDpswAnZh1jdjImprzY80Xl8w+GoVis3DXgq715c/frnbqm57H9BCg/+xnbVUJeyA4zWl/q9ZKapbp27VrlysGO+8OIwaiRo6RNmza2lWFHvfU8McGMrn50GtzmJQH0+KEjBi5TgiGgvrGLYrO4fbGi7lT6LKBoXO8Uvnx69Mh0UbB06dIsue/ftz9H/2hZLmKErQQoP1vx2pq5v2WHmdFwvgz3JKveWyWPVHC/oLovN47Z2IPDB8v27dulZ6+evmTFa0mABERUZ0x2PkwDCRI8McD1kR3BnlztqGkA5+naezdw0EBVW3inN/oJwov89OnTUq16tQC+m9CrGuUXvDL3p+zgmmXUqFECv10xMTHSqJH5erO+0EQZn3zyifowhJNiLOPkuvyZL/nzWhIggdAmQKXPAvnDpghLqOihRo0ajiHcZcuW6dHKWSl+KOzS4B0FcccrApSfV7gCKrE/ZTd//ny1xBIWWe/aratlHK5du6aWL4Nz7SeaPqGGjrEeNQLMRfi+sAw1MyKBkCdgu3Pm/E4YNkXNmzWXo8eOOs3ei98cLwMGDFDOkY8cPSJYSHnGjBlSunRpCQ+3wMFifgfrp/uj/PwE2oZi/Cm7zR9uVmvKYom17s91d3rWPb01OFWHQ+lrKdfU6kOoP/6yW0EFa9c++uijnhbBdCRAAiSQLYF8sfZutndo80mjTZGxqA5hHdSiy5jQgQWYI4ZGqGGhWbNmGZNxP48JUH55LAAfiveX7OCRPyIiQtUUJhprVq/xodaeX4r1Q6nwec6LKUmABHImQKUvZ0bZpnC1KdITY7k59PRNmzZNMMTbt19fwZBNtWq059MZBcKW8gsEKeSuDv6S3dkzZx2umHJX09xd1bZd29xdyKtIgARIwA0BDu+6AeNpdJun28jC2IWmi9cnJydL/Xr11Vqo6Ok7d+6c8lnlad5MZz8Bys9+xnaVQNnZRZb5kgAJ5FcCnMjhg2Rhj/PLL79I1apVTXPBgspYZBlhYcxCx+QO08SM9DsBys/vyC0rkLKzDCUzIgESCCECVPp8EDaGlxo3aZytUfeg8MwVOlBMixYtfCiNl1pNgPKzmqj/8qPs/MeaJZEACeQfAlT6ciFLzMI7efKkwMXCjRs3sp1916BBA6lXr56UKlXKdAg4F8XzEh8JUH4+AszDyym7PITPokmABIKeAJU+L0V45coVqV6tusCeCF75seh6rZq1ZPV7q01zggfwyMhIad+hfbY9gqYXM9JyApSf5Uj9liFl5zfULIgESCCfEuBEjnwqWN4WCZBA4BDAJK7y5ctLwYIFA6dSrAkJkEDIEWBPX8iJnDdMAiTgTwJfffWVNHi8gby//n1/FsuySIAESCALAfb0ZUHCCBIgARKwjsDVq1dl3dp10rtPbylevLh1GTMnEiABEvCSAJU+L4ExOQmQAAmQAAmQAAkEIwEO7waj1FhnEiCBgCegaZpgDe7o2dFy6dKlgK8vK0gCJJD/CbCnL//LmHdIAiSQBwQmvzVZihUrJokHEyUjPUM2bNyQB7VgkSRAAiRwmwDX3r3NgnskQAIkYAmBs2fPyhdffCFx/4hTSzFWqFDBknyZCQmQAAn4QoDDu77Q47UkQAIkYEKgQIECEjUzSg4ePCjnz5+XHi/2MEnFKBIgARLwLwH29PmXN0sjARIIAQIPPfSQussRI0aoId5u3bqFwF3zFkmABAKdAHv6Al1CrB8JkEBQEkhNTZW4jXHS/bnuUrRoUXnzzTeD8j5YaRIggfxDgEpf/pEl74QESCCACBw6dEitzR0WFibr16+XkiVKBlDtWBUSIIFQJMDZu6Eodd4zCZCA7QRSUlKkY1hHefDBB+Wee++R2NhYKVKkiO3lsgASIAEScEeASp87MownARIgAR8JpKeny+XLl6VcuXI+5sTLSYAESMB3AlT6fGfIHEiABEiABEiABEgg4AnQpi/gRcQKkgAJkAAJkAAJkIDvBKj0+c6QOZAACZAACZAACZBAwBP4fx/poB6pqtr7AAAAAElFTkSuQmCC)
- We now consider the limit
, in which the fractions
are held fixed, and apply Stirling’s approximation ![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAhUAAAA4CAYAAABKSQq/AAAREElEQVR4Ae2dCXQVRRaGL/suOySQgIQ9BJElCAiHoMimwFFkB5WEGc2grCKMyI4iS0gIDqsOKIRNMaAgHGQngMMmmEEJUcImYJCwSTAQ6Tm3Yr3p9173S3fo1wnPv84J3V11a7lfNa9vV92qzqcoikIIIAACIAACIAACIPCABPI/YH5kBwEQAAEQAAEQAAFBAEYFbgQQAAEQAAEQAAFLCMCosAQjCgEBEAABEAABEIBRgXsABEAABEAABEDAEgIwKizBiEJAAARAAARAAARgVOAeAAEQAAEQAAEQsIQAjApLMKIQEAABEAABEAABGBW4B0AABEAABEAABCwhAKPCEowoBARAAARAAARAAEYF7gEQAAEQAAEQAAFLCMCosAQjCgEBEAABEAABEIBRgXsABEAABEAABEDAEgIwKizBiEJAAARAAARAAARgVOAeAAEQAAEQAAEQsIQAjApLMKIQEAABEAABEAABGBW4B0AABEAABEAABCwhAKPCEowoBARAAARAAARAAEaFgXtAURS6efOmAUmIgAAIgAAIgMBfl0BBu1Q/d/Ycbd+xndKuptGvV3+lvn370mOPPeaV6n/77TfasmULZd7LFOXnL5Cf2j/dnsqVL+dW36FDh+j8+fNUpEgRypcvH93NuEvFSxSnDh06CNk//viDOnfqTImJiRQdE019+vRxKyO3I6zUV+piZ39xnd7QQepi99GXdLGbHeoDAV8kkJGRQQULFqQCBQpoqnfjxg0qXrw4FSpUSDPdU+StW7fowoULVL58eapUqZInUc20ixcv0k8//URFixal0NBQunv3rngW5qQtXIFtRkXqlVTavWs37d+/n65du0Zt2rTxmlFx5coV+vbbb+lK6hWKj48XIHv27EnzF8x3g3r82HHasGEDffPNN5Q/f35q27YttWrVymFU3Lh+Q5TFGfcl7MuTRoWV+kpAdvYX1+kNHaQudh99SRe72aE+EPA1AvyCNn78eJo5ayZVrlzZST0eAV+0cBEtWLCAdu/eTYHVAp3SPV3cvn2bxo4ZS9u2baPg4GA6ceIE1axVk+bPn0/Vq1f3lJXYyFn676W0du1aYUS88MIL1KFj1ov0qVOnaMb7M2hO9ByqWLGix3I0ExWbw6SJk5QK5SsomzZt8nrNW7ZsUR5v9Lioz9/PX0lNTdWts2mTpsrGjRvd0jMzM5VKFSuJMiZOmOiWnpcirNDXVR87+4vr9oYOrjrZde1LutjFDPWAgC8R2LF9h9KxQ0fl4sWLTmrdunVLiZ4TrdSuVVs8W/iZeO7sOScZTxc3btxQ2rRuozzT/hnlzp07QpTLfLHHi0qd2nWUo0eP6mbfs2ePeC5Wr1ZdWbt2rXL//n032eTkZKVdWDslKSnJLS27CNt9KgoVNj+8o2kNGYjct28fvTH0DapZsybdu3ePln+yXDPX77//Liy3Tp06uaXzcFXZsmVFfIWKFdzS81KEFfq66mNnf3Hd3tDBVSe7rn1JF7uYoR4Q8BUCPHIQGRlJMXNjyN/f30mt5ORkevLJJ8Uog1OCwYvYubH0ww8/0D/f/qeYtuBsJUuWFKMLPBPw5qg36f79+26lxa2Io54v9qRixYrRjp07iEfwedrfNdSqVYumvTuNBvQfIFwWXNM9XdtuVHhqjNVpPF3B0ywRERGi6GXLlgnjwrWew4cPU9OmTXXnu+QQUIUKedyosEhfVz52XlvVZ3a2Wa8uX9JFT0fEgwAIuBPgB/vLL71Mr0W+RvXq1XMTaNy4MYU2D6XKfs7TIW6CGhF37tyhhQsXipSWLVs6SQQGBlJISAh999139NWmr5zStm/bTiNHjhS+HUuXLaWgoCCndNcLdgPgdoaHh2saKK7y8tpnjQp2fElNTSW2uHr36S2cYC5dukSbv9osdXcc+Y2SrUa9II2JihXMzy/x3FVKSoqmMaNVH8vnJFipb07qtyKPL+ggOfiSLlInHEEABIwRmDlzplgAwAsSrA7sVMnPCXas5D/XII2YnTt3OpLYmXPIkCHCOHh73NtUt25dR5qnk8h/RIrR47i4OE9iTmm5blSwU8gHH3xAUyZPoWHDhtGZM2eEI2dsbCy9PuR1mjFjBh07dsyp0UYu2PGyRcsWYmjnkUceoV69eolsH374oVv2hL0J1Lp1a7d4GSGnPeRRxns68gqAV//+KgXVCKLmoc0puH4wRc2Oyta4GDF8BPENYDZYqW92dT8MfZadDt5Ot7M/vK0LygcBEDBOgH+/V61cRe2eaufmmGm8FH1JfkZy4FUaWlMcfn5+Iv3HH390FMLOoFevXhWjFGzonD59mrZu3SqcO9k1QC80atSIqlatStFzoolXQhoJuW5UsPfr9ye+p7iVcbQybiXxEs+I8AgKaRBCY8aOofTb6dSxQ0fatWuXEX0cMq6jD+ER4SLtwIEDoj4pyENJ3El16+lbbnKkQh5lXr0jd3Svnr3oyJEjNHr0aDHP1bVrV4qJiaEunbuQurPVZfB+GEePHqUSJUqoow2dW6lvdhU+DH2WnQ7eTrezP7ytC8oHARAwTmDNmjXEKzMahjQ0nsmEpBwx5+cMGweuoXCRwiKKt0rgwM+VdevWiXNeffLSwJfoieZPUP9+/SmsbRg1bdKUEhISRLrrP+xvUb9+fTHqsnfvXtdkzetcNyqaNWsmlnr26NFDNPCdd96hxUsW01NPP0U8PzT4b4OFNTb/X+7LQTU1+jOS57N5TkgGBiOnOD766CMZTexPwW3g5aR6QXaiUaPi888/F21O2JdAw0cMp4EDBwrDYs/ePWIdcruwdsQjJmorkzt+TtQcMf/mqS16bbRSX706ZPzD0Geyrbl1tLM/cktH1AsCIOBO4PChwyIyJ/4S7qW5x4Q0DKEyZcqIhA3rN7gJyOdKyVIlRdrly5cdL7IBgQE0/f3pdPHSReLnEY/gs1sAO2/yCLRWqFOnjojm0VcjQf9JaiS3hTKFCmatCuH1suqHd5UqVYQDpRzyMVIlz2f/8ssvVLt2bSfxwYMHi+tPP/2Url+/Ls7FG2VrfX8KFuJpD/as5Q2yjATej0PtlSvz1KhRg7748gsaNWoUTRg/gdq0bkOTJ02m9957T4xg8DTQ+Anjpbjho9X6Gq04L/eZUR28IZdb/eENXVAmCICAOQJnz54VGfwqZ01DmMudvTSPZLN/BIeoqChiB0wOPD3B52tWrxHXgQFZe17w5lYyTJ8+XThy8sZW/KIdOy9WbHiVmZlJ06ZOk2JOR36555B0MskpXu/Cts2v9BrgGu/qQMJLOvmPh5OMBvV8tjpPp86dxNIetsx4zoudUHjYZ9asWWoxt3M2ctSGjpuARkTz5s01YknMafHoRftn2tO7094VXrw8MsHyW7/eKpa/amb0EGm1vh6q0kzKi33m2lD+D8f+Obyjq5nA917E4AiS1rqRvLndH0baCBkQAAHvEDh37pwo2IwPntmWDBs+jG6n36aY6BixISPvpJmenk4dO3akcuXKERsSvPKRg9pngl/S1YF/33r17iXcDvh3i0fMXZeYSmdQ+SKuzq91nmdGKmTj9PwJ2JIyGlzns2U+3iZ10KBB4pKnQNhQ4d3OXB+KUl4eefrDjFExN3auWAcs82sdednPqtWr6PyF85RyJoXi18e7jaxo5dOKs1pfrTo8xeXFPtNqb5HCRahIUfN//B/PTMjt/jDTVsiCAAhYS4CfMxz4Ae2twA/+cePG0eEjh2nJkiXiExL7D+yniZMmimkMbsNzXZ8T1ZcuXdrRDN6TyTU0CG4gongZ7LW0a67Jjq0W7mXqO3SqM+W5kQp143J6zvPZAwYM0Mw+YOAAmj17NvEQFb+5Ngv17E/BhQQ3CHYMN2kW6hJpxidC3oAuRZi6tFpfU5VbJOxtHdgwGPXmKIta67kYb+viuXakggAI5CaB0mVKi5ECrQe41e3i7bjVW3KPGDFCrArp268vBQQEiOp4Pwqe7uARCx6l59Uc6lC2XNbmjhxXoqT7IoErv14R4tWred76W5aZ50YqZMNyetSbz5bl8UZW3bt3F5cL5i9wOG/KdK0jv4lLq08rPTfjvKGv3fr4gg6SmS/pInXCEQRAwDgBOVX684WfjWeyQJIXHfC0Pu/eOWnSJEeJ7AvYokULcZ18KtkRL0/ktAa3W8tvkB09ObBPoJHgc0YFDz3zTmWu80JqGDxHLoOn/SmkzKZNm+it0W8J508Zl92Rp2ty8ic9d7MrX6Z7Q19Ztl1HX9BBsvIlXaROOIIACBgnIHe5TDqVvWNjdr/3PIXCS1Q3b3bftFHdIt6ym5eKlipVij7+5GPhV6FOl8+85SvcP1UhHTBfeeUVdRbHuTQqeNdpI8FWo4IBXb+WteqC527knBMfr13PmsvhdBnPCvAmUjxsw0NJvNmHXuDO4SUxs2fNFr4SaWn6DnlNmjQh3tSDPxUrrUq9crnOyNciaenSpbR40WI9Mad4nuvy9/PP0R/vz6F2rHEqWHXhLX1VVYh+0OovlsnLfabWwa5zO/rDLl1QDwiAQM4JPP/88+KNn/dfyi6o95m4eeummzgvJOBNINlgSEpyN1LYAX31qtXUuVNn8Tz7cuOXYmtt14K6dOlC3bp3o0MHD4ntDGQ6+xXyFg68UrJf/34y2nHk3/kT/z0hFhC0DWvriPd4kt0Xx6xKX7dunVK5UmUloGqAElQjSKniX0Xxq+ynTJk8xRFfM6imUrVKVRF//Phx8QU2lqnxaA3l0eqPirRuXbu5NSktLU187Y3L5jKqBVYTZaxYvsJNVkasj1+vDB06VF56PI4dM1Zp3LixcvA/Bz3KycT09HTl9OnTOfrz9CVVWb639eV69PqLv163Mm5lnu8zycqOox39YYceqAMEQMAaAhPGTxDPIP5t0AonT55UBg0apDQMaej4SmmTJk2U8PBw5eDB/z9njhw5In5rAwMClZSUFEdRiYmJStTsKKVVy1bib/Gixcrdu3cd6VonGRkZCj/L+Nn77LPPKsOHD1dCm4UqfXr3US5fvqyVRTzz+AuqSxYv0UzXiszHkR6tDosS2aLiKQm1EyPH8RseO9Gp4zmOAx/VjozcVM6jjrOoeSjGhYBef3Efcj+49iX6zAUgLkEABP6yBHiEnTc55C2xR44a+UAcePqhcKHCVK58OUc58fHx4neY95rgb314mu53ZPrzhEff+YNjvJ04j1BIh05XOb4eOnSo2Dhr/fr1VLhw1k6dWnLqONuMCnWlOAcBEAABEAABXybA0/H8uYYNX2xwWqHxsOi8c8dOYVR8ve1rkt8TMdJ2W30qjDQIMiAAAiAAAiDwsBNgf72YuTHCJ48/5vUwBXb8nDp1qvgmlxmDgnX0yX0qHqbOQ1tBAARAAAR8k0BYWBjxLpaRkZE0b948r3y11GpyiYmJNC92Hn227jO3VSRG6sL0hxFKkAEBEAABEACBHBLgLbSLFStmyvchh1U9cLaMjAyxWZbaz9FMoTAqzNCCLAiAAAiAAAiAgC4B+FTookECCIAACIAACICAGQIwKszQgiwIgAAIgAAIgIAuARgVumiQAAIgAAIgAAIgYIYAjAoztCALAiAAAiAAAiCgSwBGhS4aJIAACIAACIAACJghAKPCDC3IggAIgAAIgAAI6BKAUaGLBgkgAAIgAAIgAAJmCMCoMEMLsiAAAiAAAiAAAroEYFTookECCIAACIAACICAGQIwKszQgiwIgAAIgAAIgIAuARgVumiQAAIgAAIgAAIgYIYAjAoztCALAiAAAiAAAiCgSwBGhS4aJIAACIAACIAACJghAKPCDC3IggAIgAAIgAAI6BL4H+0yljFId1mJAAAAAElFTkSuQmCC)
- which gives
i.e.,
![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAp4AAABSCAYAAAD9w9R1AAAgAElEQVR4Ae2dB3wU1fbHDx271ND7E5WOSEdCJwGkSe8kPEMVTIDnoxggKom0gAQQ8CFIUxJaUP+PplQpoVfRJ/3xgFClBpj/53fzZt/2nZndnd3ZPffzSWbn1jPfmTt79t5zz80iSZJEHJgAE2ACTIAJMAEmwASYgJcJZPVy/Vw9E2ACTIAJMAEmwASYABMQBFjx5AeBCTABJsAEmAATYAJMQBcCrHjqgpkbYQJMgAkwASbABJgAE2DFk58BJsAEmAATYAJMgAkwAV0IsOKpC2ZuhAkwASbABJgAE2ACTIAVT34GmAATYAJMgAkwASbABHQhwIqnLpi5ESbABJgAE2ACTIAJMAFWPPkZYAJMgAkwASbABJgAE9CFACueumDmRpgAE2ACTIAJMAEmwARY8eRngAkwASbABJgAE2ACTEAXAqx46oKZG2ECTIAJMAEmwASYABNgxZOfASbABJgAE2ACTIAJMAFdCLDiqQtmboQJMAEmwASYABNgAkyAFU9+BpgAE2ACTIAJMAEmwAR0IcCKpy6YuREmwASYABNgAkyACTABVjz5GWACTIAJMAEmwASYABPQhQArnrpg5kaYABNgAkyACTABJsAEWPHkZ4AJuEHgyZMn9OGIDykiIsKNWrQVvZF+g2rVrEXr163XVoEflmKefnhTWCQmwASYgAcJZPdgXVwVEwgqAlCSBg8aTCdPnaT1622Vv8ePH1NSUhKdP3ee+vTtQ1WqVKGnT59S6vpU2r17N2VkZFBYeBg1bdpUE7e8+fJS0pwk6vReJ8qSNQu1bt1aUz3+UsgVT8jpTaaBxtNf7ivLwQSYABMwJ5AtNjY21jyCPzMBJqCMwPjx44USmZqaSgUKFLApNHnyZMqXLx/lzpWbhg0bRs1bNBfHPHnzUPv27alY8WLUpXMXKlK4CFWuUtmmvJKIIkWKUKlSpSjq/SgKbRRKODdqcMUT1+VtpoHE06jPAcvNBJhAgBOQODABJqCaQEpKipQ/X35p/fr1dss+evRIqle3nvT06VNp0aJFIm/JEiWlAwcOmPI/e/ZMKlG8hNSieQtTnNYPERERUpXKVaT06+laq/BpOVc8IZyeTI3O06c3kxtnAkyACTghwDaeAf7Dgi/P8wQuXrxIwz8YTh06dHA4vb1z505q0KABZc2alY4eOSqEGDp0KFWrVs0k0NWrV+n+/fuULXs2U5z1B7SF6XlXIT4+nh4+fEjR0dGusvpduhKeEFpPpkbm6Xc3mAViAkyACZgRYBtPMxj8UT8Cffv0pQ0bNujX4H9bioyMpM8mf+ZWu7BOgTI4bvw4h/WULFmShgwdItL3799PWbJkoX79+1nkT0tLE+fVq1e3iJdPDh06RM2bNafExETq1r2bHG33iCn9uE/iaGDUQNqyeQs1btLYbj5/jFTCE3LrydTIPD19j43cVz3NgutjAkzAfQKseLrPkGvQQKBV61YWimf27Nnpq398RcWLF1dVGxakYNQQf/f+vEe379wmjKCdPnWa9u3bR9euXbOob/ny5fTR3z+il19+2SJe6QkWBa1ds5YwelmsWDGHxcqUKSPSbt68SSdOnKCKFStS3rx5LfLv27tPnNd4q4ZFvHwC282JEydSm3fbyFFOj+3ataNP4j6hjz76iHbu2klg6u9BKU9ch95MjcjTG/fbqH3VGyy4TibABDxAwMk0PCcxAa8SGDJ4iLB9hK0k/sLCwqTHjx97rE3YUJ4+fVoaGTNSKla0mKmteXPnaW6jVatWUkjBEOnixYuK6khNTRXtjh833iI/ZKtWrZpUtEhR6c6dOxZp7pwkzU4S7cFm0ghBLU9ck55MjcbTW/fciH3VWyy4XibABNwjwDaeHlDeuQptBCbHTzaNYqEGjAAmJCRoq8xOKUxvv/baa5TweQIdPnKYBg4aKHLNnz+fnj17ZqeE86iTJ0/Snl/2UHircCpatKjzzP9NhV0iQt16dS3y79+3ny6cv0DNmjejl156iVD3V199JfJIkiRWy0+bOo3S09Mtyrk66dGzB73wwgs0a+YsQj3+HLTwxPXoydRIPL15r43WV73JgutmAkzAPQKseLrHj0u7QQAK0oKFCyhnzpymWhJnJNL27dtN5576gGluTFsvWLCALly4QJs3b1Zd9eKvF4synTp1UlwWShIU4Jo1a1qU+ec//ynOw8PDxTFpdhJVrFBRfI6bFEenTp2iPXv30F8H/NWinKsTmBC0bdeWjh49SgcPHnSV3afpWnhCYD2ZGomnN2+m0fqqN1lw3UyACbhHgBVP9/hxaTcJVKpUiT7++GNTLRilwwIZtSN9pgpcfIBS9smnn9D8L+e7yGmZ/ODBA1q5ciXlzp2bQkNDLRMdnN26dYtOHD9BFSpUoDx58ljkup5+XZy/8847wib1/Pnz9HbNt4VSjEVH0THRdOrkKeEw3aKggpNGoY1EruTkZAW5fZNFC09I6gumRuCpx100Sl/VgwW3wQSYgHYCuq4+OHv2LGGFb47sOShHzhzC1cyTjCeU8SSDcuTIIVzTYDHGpk2bKGeOnJQ9R3bKli0bYQEJ8j3OeEww+DcfIdN+6cYuid1wNm/ZTNg2EUpMt27dqHLl/zkhl6dZMdrm72HAXwfQzz//TPIo4H/+8x8aMngILVu+TIwWelr+fv36iQVCeB6xgEdJ2LVzF929e5datmxJzz33nJIidOnSJZHv3Xfftcnfv39/sdXlzJkzhXI6Z+4cca1wvxSfEC/6yeXLlylmZIxNWVcRUGYRsAgqLi5OEUNXz5OrNtWma+GJNnzBVAtPtTyMkt8IfdVTLPXuE56Sm+sxFoFHjx6JhaDQdeyF27dv0/PPPy90JHvpzuLwnYXFtvDSUbBgQWdZFaVh5zjoFNDX3ArumYiqK52WliYNHz5c6tiho2mhR6WKlaT+/ftLn376qajs0qVLUvSH0VL3bt2lIoWLiHylS5WWevXsJcVEx0j3799X12iA5t63b5/Up3cf6S/l/iIYbdiwwXSlWKzSvFlzqX69+lJ6ujEcisPxecUKFU3PBRYbzUmaY7omX38YM2aMkG3WrFmKRcECohMnTkhPnjyxW+bGjRsS7qO9Zxr9pHix4poXHjUKbSTkPXf2nN22rSOdPU/WeT1xroUn2vUVU7U8PcHIX+vw977qKW569wlPyc31GIcA3s+9e/WWrly5YiP07du3pYT4BAn6z/lz523SnUX8+eefEhYEvl7+dalD+w5S+dfKS+Hh4dLZs2dtim3ZskWqVrWa07+PPvpIlDt69KjUs0dP6erVqzb1qInAAgSfhHJly4kvRqwadRRat2ot8gwaOMhRlqCPj/04VjAyVzx37Ngh4qC8bd602TCMIHeB/AVMshcuVFg6dOiQX8gf2jBUyLV9+3avy3Pv3j0JuxwNGzZM7NYzbtw41W3Kit3qlNWqytp7nlRVoDCznjwhkrtMtfJUiMNw2fy5r3oapl59wtNyc33+TWDL5i1i17rLly9bCHr37l1p+rTppkElfI+rUTyhsDao30Bq1rSZ9ODBA1E36nyv43vSa395zWL3PCRGRkaavnPRlr2/b5Z8Y5LxzJkzEn6Iw2OM1uAzG09MKSLIR3vDtnKafLSXJ9jjYLJgHerUqUOxE2Jp9OjR1DC0oXWy357Xq1ePPoz+0CRfRkYGDYgcQH/++acpzhcfIMfp06dF01WqVPG6CAcOHKB79+5RWFiYsCvNm8fS/6cSAcqWLSuyHTh4QEl2Ux57z5Mp0UMf9OYJsd1lqpWnh5D5XTX+2le9AUqPPuENublO/yVw/PhxGjhwIM1InEGFCxe2EPTMmTOE/gWTQy1hZuJM4SUF/qqxJgHhxRdfpGnTpxFMGWOiY0xeXa5cuUK7d+0WHlXwXXH8xHGLv0GDBglzR3wXyaFcuXJis5KePXoKUz85Xs3RZ4qnGiE5rzoCUNQHDx4s7AMd2Y2oq1G/3DExMVSz1v9WgP/xxx80etRo/QSw09Jvv/1GUJZCQkKE6yM7WTwahUUccAO1cMFC2rp1Kw0aPEh1/bLtKhQufwt688T1u8vUn3n66v76Y1/1FQtulwkoJQDlr0/vPhQ1MIpef/11m2LYVhkLTUMKhdikuYrAos25c+eKbBiAMg/YnAUbmRw5coS+3/C9SFqzeo3YGQ+blCAddqDmfz/9/BPVrVuX8uazHPxAHOTEWgUtrglZ8TS/M/zZ5wSw2868efPolVdeMcny7bffEv58FaD8Iij13emunLh2dPgvZn8hfolqWUwn7/Jz5PARsTjPXZk8WV5vnpDdXab+zNOT90ZNXf7YV9XIz3mZgC8IwFc1XPphQbCnw++//05YrISRTnm007wNWdHFgAbC+1HvU5OmTcyzmD6jLnhlad2mtSnO/AP8YsO13dKlS82jFX3WdVW7Iok4k1sEsBJ69erVlH49Xax279u3L8l7gf/6669i5ThWwqffSKcRI0aIL+QlS5YI1z35C+Snzp07i19FEAJbTm5I3UC3bt8SPia7dusqhuzdElBBYWxFmTgzkbBHtBxGjRxFNWrUsHA4L6d5+3jr5i3RRNFiypzGe0IerBrECKvWACUZo934BQwXRPnz59dUlbNn5tfTv1LxEsWpRYsWVLVqVcX1+4InhHOHqad4KoZkkIz+0FefPn0qRnmwerdjx47iPYFnHu81TFvivdG9e3ePbiHrqX7hC9kN8mgFpJhYZb582XJq1LiRW+93R3DgqQUBq88xEmltplioUCGRjlknBGdeb9atXSfSW7VqJfJa/4PZGd6L06dNF/1Lzewqj3ha0zT4ORQNOB9fsWKFeMBhwyGHO3fuiF8wS5ctpWVLlwnFMqJ/hFAqYQ+ShbJQ82bNxfQuHqY1a9YQlM3IyEgxChkeFk54UeoR8LBjGF8OsHmEvSc6lN7h5q2bosmQgtoVQb1lxmgUdkRCgDsOrcHZMzP6b6Pp/r371KJ5C/rpp58UNxHMPBVDMlBGX/fVOXPmiC/ZUiVLUetWrcXsSM+ePalE8RI0MmYkYTqxd6/eHt3Jy1P9wheyG+jRCjhR4Qsa32WVKlbyyrUVyF9A1Aul81//+pdNGzlzZW7WghFXV2Ht2rWZU/4OBkCgtL7xxhti9Fbtpi8+VzwnTZpEZcuUtfu3Z88eV2xs0ifETqDKlSpr+nunwTt0//59mzqNFIFFELNmzaKuXbvaiI1f/klzksSoABLHjh1LX87/kho3aSzsOyIHRApbxqioKIIvzU8++UTYoMAuBF8u2OJQq8GzjTAKIiZMnEBvvPmGKSdsU/C86B3kETr4UjNSkOXF6I/WoOSZwUsOOy8pDcHIM9DfS77qq/BXnJKcQnhnFSlSRPwwnjRxkvihjM0iMCLfrn072rhxI23YsEHpI+oynyf6ha9kd3lxnMFrBLBVMoIW+00lQlWsVJFeffVVkRV+nK2DbI/54ksvWidZnENpxQKoNq3bWMRbn2AtAsIvv/xineT03OdT7ePGjRN2BvakbPtuW9q1a5e9JIdxcHAc2kjZzjLWlcAOTP6ytk4z2jmc7zsKcOCP0KFDB4spWLy4MVyOqXh5X3O5DrzAEc6dPSdHef0IGxXsq960SVN6+PChaG/unLnUsGFDatq0qdfblxuQFTd7NjNyHn88ys/y7VvaRzzl63L1zMhTPHJ+Z8dg5Bno7yVf9VUsnqtQsYIwo8BMD8Ko0aMs7LFlM5Md23eITUrMn01Mz2NVsZppQvPy7vQLX8tufh38WR8C585lfn8WCsmc8vZ0q9jaFguLMWg0depUYQIFG07MVP609SdauWKlaLJ4sczvc0ftY5odoVVr+9PscjksSEI4fSrT64sc7+roWDtxVdJP06E84c+bATcxPj5etSsBvNwiIiPEimVvyqe07vLly1tkhXz4wzSt/EDJGXLlzCU+YppAzwAZP/vsM2GPKreLjoWdjmR7FTneW0d5ajj3c5muKbzVjqfrNSmed9xXPGXZHD0zap6LYOQZDO8lX/TVggUKElbXI2ChA4L1j1LYeSJcvXZVHOV/hw4dEqZFiYmJ1K27ews9tPQLf5Fd5sFH7xPA1sgIWE/hrfDB8A/o3v17NGP6DDHziVXqmMmFLX7evHkJ60AaNGjgtHmY2cFu31oPsC4kD8bIgwnW6Y7OA07xdHShno6HIpYrd6YyprRuWbFTmt/b+fDryF5AvLVRspwP25vqHXr07CFWectTBxiR3btnL73b1nYrSm/I9vhRpl2pIyb22sTKxc8TPreX5FbcyFEjadSoUYrqkLc186RdrKNnBtvaKg3MUykp9fl8/V7Su6/KMzFYyYvFkPA+YO0XUR6NsR7lgYusiRMnElzJuBu09At/kd3da+fyyglgUAdB3tJaeUnlOWF7OWbMGIKd88EDB+n5F54X7uRQQ423aoiBJUcr1ZFHnmYfO26sy0ah0yCo1QsCTvHE/rrHjh9zCcxehjx58pC17yt7+QA7OibaXhLHeYEAOtK0adNo39594tcaXEDopXTicuQvlQf3Hyi+OiiHShVExZWqzPjgYaa8eK79KQQjz2B5L/mqr+7fv18sPITjbfMAm7Zt27aJKKwkNg+whYMvRV8HI8vua3ZGa/+VV18R32Gy6Zg35S9ZsiThTw7wYoNBCIzuwxuFoyBPs7dp4/oH2bXr10Q1JUv8rx1H9ZrHB5ziuXjxYsJqLC0BNp7r1q8LGDtPLQz8tczhw4fp6tWrYrpgwoQJuoqJXR8Q4DHASOHhg0y72Dyv+pfiGYw8g+m95Iu+Kk+z16tvqXhiMQcWStaqXUvYhqP/YrQJbuLgEqlP3z6UL18+n3ZrI8vuU3AGbByLcU6eOEmXLl7SVXr8uIEbJ8wGxMbGOm0b+hMW9cq+i51llr3mlC5d2lk2m7SAUzwxPKxkiNiGBEf4LQHYaPXr20+ssJ/35TzNCwG0XqCsKBnN44H8qxq/sv0pBCPPYHkv+aqvYuEQwptvvmnxqC9cuFC8LxLiE0w+C+MmxdFzzz1H+/bvE3ahySnJFmX0PjGy7HqzMnp7mFGFydjpX10vxpFXoDu6ZvyAwsYqL7/8sthe2VE+eKOBOzG41/t68dfCztNRXmzucezYMcWzdbLi+dZbbzmq0m687u6UAAsjV/LoEQSXvyBlCZEHvgfhuBzh+vXrPt+vW5bNn47gJLumuXnjpsluxDre/AFGmry4A2VxLgfsiY6tIaFgyfcHaeZl4L9Oz5Cenk7dunYTXxTwPypP0+opAzo2gq/3jFdzzeKe3cz0P6p0qh1lHD1Prp4Z9GGltqTBwlPN/QqEvL7qq3j20tLSBEJ59BAn8N+ZmpoqXMi9WSFTIYX/QuSFqdSpk6dcPrOO+gTqN38vOnqXuuoX3pQ9EJ6pQLuG9u3bU65cuYQ/bVfXZu6H885d2+/dHTt20JDBQ4RSefq0rSKLRdArlq+gsJZhYlR/fep6sc2ls3bl2WJnNqByeTz/x48dJ7hwbBjaUI5WdpR0DCkpKVLhQoXFX8kSJaWyZcpKxYoWkwoWKCjVrVNXSHL82HGpSOEiUkjBEKlE8RIiT/FixUUe5L1165aOEvtvU8nJyYIRmJQpXUYwKxRSSIqPjzfFgy9Ygh/CsqXLLNKKFikqoczhw4elZk2bic+lS5WWSpUsJfK1btVa2rtnr/iMvPL9Qptr16z1OpyHDx9K4eHhQn7I6KuwZvUaKX++/FL7du19JYLqdtPT04XMkPvRo0cuyzt6niZOmKj4mcEz8m6bd122FQw8XUIIsAy+7Kvbtm0Tz/rAqIFS7169pZExI6Wo96Ok2rVqS3t+2WNB+uLFi9KpU6ekvXv3ijKLFy+2SDc/cdQnzpw5o+pd6qxfeEt28+vgz/5FYPy48eK79saNG3YFw/PZr18/qVLFSqZ3ePXq1aX+/fuL51YulJaWJt7N+H7/448/5Gjp6NGj0tQpU4VOBb3qy3lfSo8fPzalO/vQKLSRVKtmLenZs2fOsok06Ab4fpn/5XyXea0zZEGEMhXV/VzQwBHklVByjRABq2KxChejczi3zoO8yIN4GLAHewBLcDBfaY048AMj63jE2Ssjj4biKK+4A1vcA+SX65GPSHN2jzx1X9D+wKiBlJKSIqYHwsLCPFW16npgC1avbj1hkH3w0EHV5VEAo6U//vgjPcnIXP2dNVtW4Z80b768NvVhdS5GZvDLGPcYq8CxMrF58+Y2eR1FYLqkUWgjYdNz5OgRR9lM8faeDUfPk6tnxvw5MjVg9sETPFGdnkzV8jS73ID/6Ou+OnnyZJo6ZarYOKNzl87C8XX2bNmp/OvlTe8v65uAhRbJq5Lp+Injph2+rPM46hPok7hm6/evln7hLdmtr4XP/YcA3lt4N2Ov9g+jP3RLMMwY58yRk8y/R7BlNp5P7CqEvdnxnCoJKAOn8a+8/IrYeMFVmWHDhhG23oTrpZw5M3dEclVGTtfVxtOeMglBAEZ2/WKu4MhCykdXX2hyvmA42mOJOEfxYGIvTeYtH2V2uCeOeFvnlct48vj5559TcnKycHfiS6UT1wTDaXSsS5cuEdy2QCFUG65du0YHDx6ka1evEV4MCJ06dRLTgNZ1HT50WCyQw24QYA2H+XXr1lWleMr+4mq8XcO6ervn9p4NR8+TfP/lo1yhs2dGzoOjJ3iiHj2ZquVpfr2B/tnXfRVTjgh16tYR/aVSJefbEcKUaHXKamrfIXPac/z48eI9Y32fHPUJ63zyudwf5KMc76xfeEt2uW0++h8B2LjDbKxzp87U8b2OFivP1Uprz5c1pvO1BDyn2KVQSdi6ZSvhb+OmjaqVTlG/9RAonzMBXxP47rvvxBB+9IfRiob89ZC34TsNhUynT592q7kff/xRqlqlqqgLZidXr151WN9b1d+SUlNTHaY7S5g8ebJoI2l2krNsPkvzFE9cgB5M/Z2nr26kr/vqvXv3hOkWpiWVTA+C0/bt20Xf+OGHHyRMtU+fNt0n+Iwsu0+ABVijW7dulcJahknXr1831JWdOHFCwpS8O+Zvui8uUqJNc57gJbDnlz30wbAPKDQ0lD6b/JniaQK1xLAIQV6Rp6Rs9erVRTbsduJOQLtDhw0VBtlYyLVk8RK71WHRAUZXW7ZsaTfdVSRGTRGUjni6qs/T6Z7iCbn0YOrvPD19f5TU5+u+ivZjP44VCyKxYA0rfLEQ1VXAiCjc2ixcsJC2bt1KgwYPclXE4+lGlt3jMIK0QnzHzUicQQMHDhQuv4yA4ejRozR92nRalbyKKleurFlkXW08NUsZhAWxrdXs2bPp6ZOnlC17NoqKirLYvurf//43ffHFF5Q7V25hz5EzV04aPXq01xQ1PW4BXDm0bNGSChQoQN//8L1wE+GNdjFlDhub/Wn7FbexZfMW6tKli/j7YvYXmsVq0rgJzZ03V+yb+/e//13YYKYdSDOZmsgVYwpuwfwFtOjrRXKU4iNsdd58402CB4I/zv6hbSpEcWvaMnqKJ1r3NlMj8NR2F7SX8oe+Om/uPLpw8YJ4Bz7OeEyPHj5SvCUxfvTduHGDQkJCtENwo6SRZXfjsrmoHQIw/YB7L0x1+3vAYAjMIq3NSdTKrauNp1rhgjk/XOA0btSYsC853JRcv3ad4MNSDth1o1bNWjRo0CBq27YtdercyRAPriy/9RF7vXbv1l080MuXL1esEFrX4+ocXzgx0THUpEkTVW3Ub1CfsMEA9oiHIqLlJQEXYXAlVq5cOfGFFxcXR/gB8cP3P9jsxIRRPOtdWFxdm5x+5MgRMfJTu3Ztv1Q6IacneKIePZgagad87/U4+ktfxQ5mWgO+PH2ldEJmI8uulTmXs0/g+eeft5/gh7Fa1jfYuwyeardHxQ/i8Auodp3aYjVz1apVxcqx33//3SQZ0ps2a0pQQDFcj2F7owb4f4SDeCzgWPLNEkUr6rRcK3yTDogcQJs2baKu3bqqqgKLi8LCw8T0PFZlawlYLIR7CqUVU4OdO3cW1SxYsMCmOjiVrl+/vk28kgisnkfo3r27kuw+yeMJnhBcD6ZG4KnXTTRCX9WLBbfDBJiANgKseGrjpksp2AHVrlVbODuGq44Z02dYtLt3716xFZzsEcAi0SAnGD2MiYkhTC0nJSVRjRrKVmGruTy0sXHjRqHIbdiwQbhFatCggZoqRF6MLCOsX7dedVkUsB7F7B/RX9Sze/duC4fCUJDPnj0r3MGobQjXmro+VSi27dq3U1tc1/zu8oSw3mZqJJ7evnlG6qveZsH1MwEmoJ0AK57a2Xm9pPyl2qJFC6pQoQJ99913dO7cOVO7SNc6KiZXgqln+OJyFjDCt2jRImG8bz7qal4G8StWrKCl3yylC+cvmCc5/Txz5kyxh+yYMWOobbtMxc5pAYWJsG+EQjfl8ylCecc0/vlz50XpLl27aLJRwahyqVKlCNvwwdZFbdi5Y6dwiySXg581eToddcoB++pCAddiR4MRwFOnThH8GWJU3J+Duzxxbd5maiSe3r7XRuqr3mbB9TMBJqCdAC8u0s7O6yVbNG8h7Dqh7GCUrX///tSrVy+aNn2aaDs8PJxmzJghVmhqFQarp0MbhhL2Ky5atKhNNfHx8ZSSnEIVK1UUo3JQUjt27EhTpk4h+CPD6NyokaOE41n4JEPcpo2bCLJ1697Npj7zCPmaoCDBp54Wu0mMBMMh753bd4S9H2z+8IeFA44C9mgGUy0BivXw4cMF9x49eyiuAjLBCf3RY0ctrhOjk/369RNKIhy9w3QCTqXz589PkZGRiuuXM+IZ+b8f/4927d7lln84uT5vH7XyhFx6MDUaT2/dLyP2VW+x4HqZABNwk4ChHEgFkbB37tyRsE2W7Jvu6dOnUr269YTPOmz7Bh9w8Acpp7uDZufOnVK7tu0k1Gsevv/+e2nwoMGm7bYgw8yZM4UPvLCwMOnatQDri5MAAAcfSURBVGtSxw4dpbhJcRLS5ACZsH0dtv5yFLDdF7aSw5Zbev61ad3GkUiK4rH9ZJ3adcS9ML9mV4XhazIiIsImW0ZGhmlrNNnnZqtWrST4SlMbjh07JhXIX0CaNHGS2qI+y6+VJwT2NlMj8vTGjTRqX/UGC66TCTAB9wnwqnY3FXdvFd+zJ9O+Ux4FxLQrttd6/6/v06yZs8RCl7ffftti9AyyJCQkEGxD1QZMz9atU1e4cGrdprUovnzZckqcmWhy9QMZhg4dKtyWYCS0erXqYqRyzNgxFs1B5sgBkbRq1SrCFLq9gOn4iIgIe0lejWvWvJlb9WNRTHxCPHVo34GWLFlCffr0UVSfbDZhnRm7Q2HE89NPPxVT+L379BYmAeXLl7fO6vQc9ncfj/9YuKIaPmK407z+lKiVJ67Bm0yNytMb99aofdUbLLhOJsAE3CfAiqf7DL1SA2zXZPs/uQEsxkiITxAKz4OHD6he/XpykukIX5PW5UyJDj48fPBQKIiY7jav8+7duwS3TtYhOiaaTp46SevWrqP06+n05MkTm+01CxYsSJcvXbYuajqHPacnbTpNFevwAQuTuvfoLhS9xo0bW/hXddQ87mfPnj3tJvfs1ZOmTJki7Heh0MPpu1r7TuyXC1dP3yz9Rpg72G3ITyO18MSleJOpkXl6+jYbua96mgXXxwSYgAcIuD9oyjV4g0Czps2ks2fP2lS9fPly09T0b7/9ZpOuJWLo0KHS0m+W2hQdMniIdP7ceZv4y5cvS5UrVZYw3Y5p8hHDR9hM+a9atUqaPXu2TdlAibh//76EbR9hauDK3OHWrVtShTcrOM0H0wTZ5GDBggWqMF24cEEqU7qMFBcXp6qcP2VWwxNye5NpIPD0p3vLsjABJsAEzAnwqnYPKO+ergIOq7GIp3jx4jZVY2FPiZIlqFChQlSmTBmbdLURWJiDaUV7C4GGfTCMxo4dS9hZQQ4HDx6kdm3bEZyfr127Vjg+x5TzkMFDxLZfWOyzbds2mpM0RyyEkssF2hELor5e/DWdOHFCrJx3dn2YEn67pq1ZhHmZiMj/mR2o8VSAexPRP4LgLP5vf/ubeZWG+qyGJy7MW0wDhaehbj4LywSYQFAR4FXtfna7YTsI/5yYasWOBuvWr7NZtQ7byV92/yJWlntb/F27dgn7w1dfeVXshpM1W1axP3LNWjVF05hm/2rhVzRr1izhXB02e9jDFdtKli1b1tvi+bz+kydPUqf3Oglfq7DVNA9QwvEDIur9KMpfID/NnTuX8ubNa57F9BnKf7OmzejixYvCjEG27TVlsPMB7Pv26St+OPxj0T/8dpciO6I7jHLGE4W8yTQQeToEzQlMgAkwAR8RYMXTR+CN1CyUoitXrgin5C+88IJD0a9fv07ZsmWzaxfqsFAAJFy+fJn27d1nYbN68+ZNsaUp3E1hmzH4S8WuL7DldOSGae2atbRl6xZKTExURAVbF277eRu1DGsZEEqnfNH2eCLN20wDlafMlY9MgAkwAX8gwIqnP9wFlkEzAYwQFi5cWCi8mivhgkyACXidAPdVryPmBpiAIQiwjachbhMLaY/AoUOHhEunb1d+ay+Z45gAE/ATAtxX/eRGsBhMwA8IZIuNjY31AzlYBCagmkDu3LmF38pOnTsF1FSzahBcgAn4OQHuq35+g1g8JqAjAZ5q1xE2N8UEmAATYAJMgAkwgWAmwFPtwXz3DXrtWOyEPc6nTZ1G6enpBr0KFpsJBD4B7quBf4/5CpmAWgI84qmWGOf3OYFJEycR/D7u27+PnmQ8oeSUZJ/LxAIwASZgS4D7qi0TjmECwU6At8wM9ifAYNd/4cIFSktLo9VrVlPVKlWpRIkSBrsCFpcJBAcB7qvBcZ/5KpmAWgI81a6WGOf3KQE41o9PiKf9+/cT/D127tLZp/Jw40yACdgnwH3VPheOZQLBToBHPIP9CTDY9RctWlRIPGLECDHd3q5dO4NdAYvLBIKDAPfV4LjPfJVMQC0BHvFUS4zz+5wA9tNenbKa2ndoL3YFGj9+vM9lYgGYABOwJcB91ZYJxzCBYCfAimewPwEGvP4DBw7QvXv3KCwsjFauXEl589jf/9yAl8YiM4GAIsB9NaBuJ18ME/AIAV7V7hGMXImeBG7fvk3hYeFUpEgReunll2ju3LnsQF7PG8BtMQGFBLivKgTF2ZhAEBFgxTOIbnYgXWpGRgbduHGDQkJCAumy+FqYQMAR4L4acLeUL4gJuEWAFU+38HFhJsAEmAATYAJMgAkwAaUE2MZTKSnOxwSYABNgAkyACTABJuAWAVY83cLHhZkAE2ACTIAJMAEmwASUEmDFUykpzscEmAATYAJMgAkwASbgFgFWPN3Cx4WZABNgAkyACTABJsAElBJgxVMpKc7HBJgAE2ACTIAJMAEm4BaB/wd5P2ikiMtpcAAAAABJRU5ErkJggg==)
- where we have used
, and
is the probability of an object being assigned to the
bin.
- microstate: In physics terminology, the specific arrangements of objects in the bins is called a microstate,
- macrostate: the overall distribution of occupation numbers, expressed through the ratios
, is called macrostate.
- The multiplicity
is also known as the weight of the macrostate.
- We can interpret the bins as the states
of a discrete random variable
, where
. The entropy of the random variable X is then ![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjMAAABJCAYAAADIdKdMAAAaaElEQVR4Ae2dCXxNVx7H/1pLp3RFFbW1Hd2i1tKpmtJSglZCbbV1LCXUTC3FtKVVbUmMhFRjq2mHkmibBBU62qJFR4nUICo6FBFVJARJEdz5/E7mvnnv5b4177285ff/fF7efeece5bvubn3f8/5n/8pp2maJhQSIAESIAESIAESCFAC1wVovVltEiABEiABEiABElAEqMzwQiABEiABEiABEghoAlRmArr7WHkSIAESIAESIAEqM7wGSIAESIAESIAEApoAlZmA7j5WngRIgARIgARIgMoMrwESIAESIAESIIGAJkBlJqC7j5UnARIgARIgARKgMsNrgARIgARIgARIIKAJUJkJ6O5j5UmABEiABEiABKjM8BogARIgARIgARIIaAJUZgK6+1h5EiABEiABEiABKjO8BkiABEiABEiABAKaAJWZgO4+Vp4ESIAESIAESKA8EQQXgRcGvSBpaWk+b9TQoUNl+ozpPi+XBZIACZAACZBAOU3TNGIIHgKffvqpjIwaaWpQ+fLl5e8f/l3q1KljCnPm4MqVK1JYWKg+BRcKJP9cvhw7dkyy9mfJjh075NSpUxbZVK5cWXbv2S0333yzRTh/kAAJkAAJkIC3CVCZ8TbhMsh/9EujJSkpyVTyIy0fkVWrVkmFChVMYaU5gP77008/yQeLPpDExES5ePGiyu6dd96RF4e/WJqseS4JkAAJkAAJuEyAyozLyPz/hIKCAnmy3ZNy6NAhU2VfHvOyvPbaa6bfnjrIy8uT2bNny7yEeVK/fn35fvv3ct11NMXyFF/mQwIkQAIk4JgAlRnHjAIyxZ49e6RTx05y+fJlVf9y5cpJckqytGnTxivtWbVylQwfPlyWfrxUOnTo4JUymCkJkAAJkAAJGBHw21foz1d/LvXq1jN9/vOf/xjV32bYuLHjTOe2fqy1zXTBGtGoUSN54403TM3D1FDUiCjJzc01hXnyoFtEN3nn3Xdk0cJFnsy2RF4///yzTJo4qUS4pwNm/W2WbP9+u6ezLdP8yK5M8bNwEiABLxLw2MjM4cOHJT09XSqUryAVKlZQUw1Xiq5I0ZUiZavRtWtXOXPmjHz11VdSsUJFKV+hvFx//fUCQ1Oku1x0WSIiIqRixYqquampqfLGlDdkydIl6vf9998vN9xwg9Mojh45Knln8uTbb7+Vjz78SDJ+yHD63GBJCAWmf7/+sn79elOT2rdvL8sTlwtGajwt165dk4huERL/XryacvJ0/vv375eez/WUxKRECQsL83T2Fvnl5eZJp06dJDomWtq1a2cRZ/QD19vXG74WnHc697T07dtXHn74YaOkZRLmz+zKBAgLJYEgJXDp0iXBwg88X40kPz9fbrzxRrdsKM+fP68WglStWlXuuOMOo+wtwq5evSrZ2dnq2V2jRo0Szx3MHOBZ5BF7Tqxm8oTs3LlTe/nll7Ue3Xto1apWU59GYY20wYMHa++++64qIicnRxs3dpz2fN/ntVo1a6k0Deo30Ab0H6CNHzdeKywsNFUlJSVFa9G8hem3uwerV63WmjZp6u7pAX9e7ulcLeyhMFOfoG/mJcwLuHadOXNGa96subZ0yVKf1T09PV27/777tUOHDjksc8eOHdqggYO039/7e8U6LS3N4Tm+SuDv7HzFgeWQQLATOHL4iDZwwEDtxIkTJZqan5+vxUTHaHjmHj1ytES8vYALFy5oL416Sd0Pu0d21+5reJ/WuXNn7fDhw4annT9/Xnt72tvq2QudoO0TbdU5ixYu0q5evWo6Z8+ePVr/fv21kydPmsLcPRB3T7R33r333Ktu6AnvJ9hM1rVLV5VmZNRIwzRUZgyxuBW4ZcsWrXq16iaFpuadNbVdu3a5lVdZnTThlQlar169tGvXrvm0Cq+++qrW87meTpf75htv+p0yEyjsfNqxLIwEgozAhq83aB2f7qgdP37comVQLOJi40wvWnihdUWZgRLU5vE2Wof2HbTffvtN5Y08n+vxnNbw9w21jIwMi/IuXbqkRUZEqvRnz541xWFgAWXPnTvXFIaDn376SWvXtp2WlZVlEe7qD6/YzOirWfRvi3Gn//3Q4/RvozQM8wyB1q1by9hxY02ZFRUVybChw+TChQumMH8++HHfj/LRRx9JVFRUiWFKb9d7xIgRaqryiy++cKooTLH6kwQSO3/ixrqQQCARyMzMVPfH2XNmS82aNS2qDjcaeAbAxMMdiZ8TLz/++KP89dW/mkw9qlSpIrFxscp0ZPy48QITA12+WPeFbN68WUZEjZBbbrlFD5Znnn1GmjRpInPfm4tBFFP4vffeK2+/87YyicA0vbviFWXG3crwPO8RGD9+vLRs1dJUAIxBJ06YaPrtzweLFi2Se+65R5544gmfVxPOBjt36ax86vi8cA8USHYegMgsSMCPCcAWddDAQUp5gG2ptTRt2lTga6zGnTWsoxz+/u2332T+/Pkq3R/+8AeL9Lg3wnZx9+7dsjZtrSluR/oOdXzu3DlTmH4Ap6qoL16ozeWxxx4T1HPw4MEWipF5GkfHVGYcEQqSeBiELViwwEJT/uSTTwQffxYYs61cuVK6dO3i81EZnQuUKBiSZx/N1oMC4pvsAqKbWEkSKBWBmJgYZWSLRQeeloMHDwruI1h8Y7QAR1eeNm7caCq66u1V1fGyj5eZXIMgAMa+ezP3SuPGjU0LfUwniUjUyCjZunWrLFu2zDzY6eOA2psJltHQEuFWv0ePHtKiRQs5e/asLF26VHmkxe/nn39eWXI7TSCEEt51110yJ36OYP8mXSa8MkFxvPvuu/Ugr327039YIQcL+mbNmtmtF1YTYSoKbwNPPvWkhIeHC5wHYql4Tk6OGtXp+kxXtxSiRx99VJUNhaZf/35262EdeeDAAbWaDMOnuXm5MmbMGKVQ4po9kHVA6tStIx07dlTDr9bnWv92lZ+z7FCOt/iVhp11+/mbBEjAkgDujYnLE6Xdk+0Eq4U8LVilDIEigqkka7OQO++8U8Wbu07Bs3nmzJmya9cuwTR9fHy8Wj311ltvCbbGmRE9w7CaUHJq164tcbFx6jluazWW4ckiElAjM/PmzVNA69erL127dFWjCv3795e6derKK+NfkZWpK2XggIEW83G2Gh6q4V26dFFDeXr78cCH/YzuXE8P98a3O/2HfaAgGIK0JXhoT5w4UcI7h6u2QWGYMWOGDOg/QE1PNW/eXIXjTcEdadiwodx0003Ku7Gr50O52pe5T5YtXybLly1X+1oNGTxEwh4Kk4mTJkphQaF0fLqjbNq0yWHWrvJzhh0K9Sa/0rBzCIQJSCDECaxYsUK9tDUKa+QVEtWrVVf5QpEx9yivF1axUrErFSy/1gUvaAsXLVSDCvAXB2/0GGSA366NmzbavJdjifYDDzygRplgc+OqeFWZmTZtmtxz9z2Gn++//96lusJgKCU5RWl6tWrVErylTntrmpo6gcM2AIyIjJAvv/yyVLtGT31zqjzc6GG3Pn9s80e1MaNLDSuDxFPfmioPPPiAqWTMeaKvvCnu9t/xnONqSFJ/A7CuI7ZTmDJ5iixYuEAeeeQRefChBwX/2LGzYtUIzbPdnpXPPvtMnYb5XyPBSB+uJ1uCf7Lq1atLzrEcW0lshmO0MGFeghpJRKLXX39d/aNj9AhzzkOHDVUKesL7CTbzQIQ7/ByxQ76l5edNdnaBMJIESEDSd6QrCu7YwziDL6xRmNx6660qKby8W4tu+FvlpioWUXhpXvz3xUqhgX3m1199rTzDw9jXnuDlB7Jt2zZ7yQzjvDrNNHnyZBk+Yrhhwd2e7SbfffedYZxRYEZGhjwU9pByrgMHYJAJEyeoYSk9fbVq1dThls1bBE763JFhLw6Ttu3aunOqmj6AMyJ/F8x9wjC0/VPtTZtEzp83X03FwKmeN8Td/jt1+pQaFbFVJ+wJ1a9fP4vduvW3hO7du6vTYCmPKR3Y3VgLhkKf7vC0zJkzR/o+b3vO+bbbbhPUxV2BM0kI6qRfp/gNxRzDqfpwrq383eHniB3KKg0/X7GzxYThJBDqBI4cOaIQ3FmjeLrH0zwqV64so0aNEmwiPGvWLDUd/lT7p9TL36aNm2RF0gpVZJ276pQoGg5w4VwPL4qnT58W2PZgdGfu+3NtOvTDCx4ka39WifwcBXhVmXFUuCvxd1S/Q7AiBwIjIYj1gxdL0CAnT51U3+78wcMFH28KOjc6Olp5i3WlHDz0hgwdIrr26sq51mnvu+8+mT59urLh0ONw0X7zzTdiaxRET+fOt7v9d/rUaQtFxbrsjp0s7U0wbQbFAIww/wrB0CU+RoLNMTGXi2WD9gTKjH7jsJfOURy4mwv6FB/U2564w88RO5RXGn6+ZmePD+NIIBQJHD16VDW7WvXiF3lvMPjLy3+RgsICmR03W/r06aM8/xYWFipbv9tvv12OHz9eYs+/j5d+LFOnTpWkFUnqeYqpdUx7Y5QcnoMxO2AkupExbGFdlYBRZjCNBIFlNaDAYNV6Pb2uzRlpia6C8Xb6ShUrSaUbKrlUjP7gc+kkO4lhzLrpm02iDx/CSBXzmpia8bS423+Otl3ANI65bPvXNjVtg6V+zgiGUOEPwRlxVBdn8sCbjpFgWw974g4/Z+pbGn6+ZmePD+NIIBQJYJUqxNxvi6c54D7y2muvCexTf8j4QW6sfKNg7z9Ii+Yt1FQSFlfogiki2C3CtxnsFSErV61U5gCLFy+WhIQEtZDC6KUczzgItkFyVQJGmdEbBmNFGKvCCZC5YO4Oq00gsOx2V7CqA8vH3BG8vVuvxTfKBx02bvw4oyifhuEijY2NlR3bdyjtGlOC3lBkzBvlav9VrVZVrVQzz8PesW449lhr+8oM/vnT1qQJVhsNemGQGg61l++58+fEm28/9so2j3OFn6vsUI4z/AKVnTlHHpNAMBC45dZb1L374sWLXm9OvXr1BB9doLDgWYzpeayU1WX58uXq0PyFElNO02dMV/db3GPgl8ZImdGn8rHJtKsScMqMPsXU+nFLZQaGUL/++qu0erRVqZyrLVmyRFatKmno5AxYeDtc/flqtQzNmfT+kObf//63nDx5Ug0ZYljQ2+Jq/8GaHiuC8AB1ZqRBfxibjzhg+SKWCmI6SZe3p70tv/vd7wQOnlCn5JRkPcrw+1z+Oac2VjM82YOBrvBzlR2q6Qy/QGXnwW5gViTgFwSgEMDLtzuLE0rTALxUYUk4ZkfefPNNi6xOnSy2LbS2H8X9Gw5IcY/BZpdGcuLECRXcoEEDo2i7YQGnzMC4F/Lggw9aNAzDVxjxiImOceqhZ3Gy2Y/XJ78u+ISCwMboTy/8SeD4CKuB9CE+b7bd1f6rfVdtpf1j1YxuHKbXD/O2o0ePFjhpipkZI7Ca37Nnj2Ae1/xNAfO3t95SbJGPc2EgvHPnTkldmSpNGjeRunXr6lkafsPGCec0adrEMN6Xga7ws8cOdXaHnyfZpaWlyTebvlGjlN7wkeHLfmFZJFAWBDATADOBrAOODWb1lUe26okXRjhRhZde+OmyJdjaAC5Q4K7iH0v+oe635mmxAgpbJ+BejBWm5lJ0uXj6CD5ljERXZvTpKaM0tsI8ujQbMPCWry+BRcWsh7+QBlrZ2fxiAx9YOTu7RxDywkMIor+h4hj+ZdasWaOWwGJpLsUxgdzcXOnbp68anYAPFFu2HI5zcj6FO/3XsmXxFgxYPm4t69atk9WrVivnTIh7f+776h/LfAQnKytLkG7kqJGm0+H4KTomWvlXgfFar969THFGB1D6YKDbqmUro2hTGK7ts2eKr+szeWdM89gIP3P2jEqHePzWBdc+XHuDjSNfP67ys8cO5bvDz1Ps0NaoEVHy4YcfysIFC3Uc/CYBEnCBQGRkpFSqVEn5snJ0mrmfGEybW8uWLVvkpVEvKUUF901rwUtdUmKShHcKV9Pyn6/53NBnDBzlwch3zuw5Fl7T8fKUmJio3GcYLbjAfTFzb2bx1jVtXd+6xmMjM6mpqTJq5CjVfsyPYcrlgw8+UMY+WFu+9butCniHDh3UUi10ANJAKYEvGpwDWxXzjamsYcLwFzfBnj17yuZvN6vltphCwBJRlG++95D1ufz9fwIwoh44cKBSPNekrfH66i29ZHf6D55/8Qaw64ddAt8F5tKqVSs1h4v9QXDtYf54w8YNArfeo18arc479PMh5RlYt5LH+foqJ8z5YqopIiLCPNsSxzB6gzxh5x8sJSVFRkaNVK4D8GYzYcIEtfpu0qRJatVYhQoV1LWNqbwpU6bIP9f/U7BBG95esGkb/pHvbnC3Mphbtdp4mtNVfvbYoT3u8PMUO/y/Y0k9OHTq1KkEcwaQAAk4JoBR6CFDhsjChQvVnkew27QWKCZYPav7pEE87v/Y9BGKhz56ghdajM7jfxPPZ1327t0r6/+5XpKTi6fiYQz8wp9eUPc6PY35N5Zj4x42dsxYZfIBVye4J2Ipd7PmzSQuLs5wFgD1g8KFVbbWnobN87d57Oo227bSX7lyRcPHWq5du6ZdvnxZBV+9etUwDSKLioo0pNUlJSVFa9G8hf5TfU+fPl1tIZ64PFFDXrt379b2Ze5TxxYJzX5g2/GmTZqahYT2IRgPf3G4Vr1adW3t2rU+heFO/6GCr4x/RWvWrJlhP6M9Bw4c0LKzs01twbWEayNzb6bFNWVKoGlaQUGBVq9uPe3Pf/6zhi3rJ0+ebB5tcdyndx+te2R3izDrH7j2cU2aC8Jw7VuH4zc+qKe5oC3WYebx7vCzxw55u8PP0+zM28hjEiAB1wicP39ePStn/W2WaycapP7ll1+03NO5FjF4FicnJ2v79u2zeT+1OOF/P3BvQX4bN27UNmzYoOXk5BglM4WNHj1aCw8PV/djU6ALB3gj9EsxUma6dOmilJnDhw87XWcqM5aooqOjFcOE9xMsI3zwy53+Q7X279+v1bijhrbh6w0eq+XmzZsVh3Xr1mlLlizR4mLjDPM+ePCgSrd+/XrDeF8GusOP7HzZQyyLBMqGQFZWltb44caaK8/Gsqmpcam4t4c9FKaUH+MUjkM9ajNjc/jHAxGYb8vYmaGspx0ZbHqguKDMAg6LZsbMlEGDireL92UjS9N/cDQHZ4Hz5s+zsDcpTf3hJwErARZ/sFiw46u5TY15vgvmL1DbIlg7aDRP44tjd/mRnS96h2WQQNkSwL1s9pzZyg4N9pCBJDAoxnY6sN0slcNWx/pO2aQwH5nZ9q9taqqhWtVqWuvHWmtJSUnaqVOnnKoYR2aKMYFhrZq1tOd6PGea9nMKoIuJtmzZUkK7Lk3/6cXn5+drLR9pqS1evFgPKvU3poBOnDhhMx8MjzZs2LDM33ZKyy+U2dnsXEaQQBASwAhNz5497d7X/KnZMAcYNnSYlptrObXlTh3L4SR/1OJg0IsNBOcvmC/YeVPKiVSpXEUuF12WSxcvOXTrjxUoWFn13dbvlAV1xg8Z/thMn9QJS5Y7deykNktcu26t3e0BSlOhnJwcade2naTvTLcoA6Mb2cey5YZKNzjdf0b1OHjwoHSP7K40eBj9elOwEg9W+/HvxZdw1e3Nco3y9gS/UGVnxJNhJBDMBDCKi4UN5qs6/bW9WIyCxRFuGfxaNcpvlRkstR439v9ectd9sU5tYWBVf5s/J02cpFY4IQH2WsLW46Eo2OMCD2V8r1+/Xu0u7g0OWF4M3wNwcY/pIG9J9tFsteu0t3f5jouNUxuONm3a1FtN8Xm+ZOdz5CyQBEjARwT8VpnxUfuDuhgsY+/dq7ds375dLZUz94rryYbDrxB8hsAJ2mfJn5XKA7Mn68W8SIAESIAEQoNAwBgAh0Z3eK6VmD3ELuNwhISNvbyhyKCML7/8Uh5//HGlyMDrbps2bTzXCOZEAiRAAiRAAk4Q8JjTPCfKYhIfEoiPj1d7Z8DBUbeIbh4rGfskZWZmytYtW+XTTz9VTo70zHv36e2RuU89P36TAAmQAAmQgDMEOM3kDKUASwOD6cGDBysjsMjukW4ZgmEfD7jaxwaL2H5C/+Tl5dmkgU0b69evbzOeESRAAiRAAiTgDQJUZrxBtQzzzMjIkGefeVZgJe5LwYZn2DGcQgIkQAIkQAK+JsBpJl8T93J5WLGCvTp8LR2e7uDrIlkeCZAACZAACSgCHJnhhUACJEACJEACJBDQBLiaKaC7j5UnARIgARIgARKgMsNrgARIgARIgARIIKAJUJkJ6O5j5UmABEiABEiABKjM8BogARIgARIgARIIaAJUZgK6+7xT+WPHjsnVq1e9kzlzJQESIAESIAEPE6Ay42GggZ7drl27pFnTZvLJik8CvSmsPwmQAAmQQIgQ4NLsEOloZ5uJ3bWTEpOk/4D+UqVKFWdPYzoSIAESIAESKDMCVGbKDD0LJgESIAESIAES8AQBTjN5gmIQ5IEdsNd8vkZiZ8VKbm5uELSITSABEiABEggVAhyZCZWedtDOaW9NUxtTYrPIK0VXJDkl2cEZjCYBEiABEiAB/yDAvZn8ox/KtBbZ2dmyc+dOSV2ZKk0aN5G6deuWaX1YOAmQAAmQAAm4QoDTTK7QCtK01113nUTHREt6erocP35cevXuFaQtZbNIgARIgASCkQBHZoKxV11sU+3atdUZY8aMUVNNERERLubA5CRAAiRAAiRQdgQ4MlN27P2q5MLCQklNSZXI7pFSqVIlmTJlil/Vj5UhARIgARIgAVsEqMzYIhNi4RkZGVJQUCDh4eGyYsUKuf2220OMAJtLAiRAAiQQqAS4milQe87D9c7Pz5fO4Z2lVq1actPNN8n8+fOlYsWKHi6F2ZEACZAACZCA5wlQmfE804DNsaioSPLy8qRGjRoB2wZWnARIgARIIPQIUJkJvT5ni0mABEiABEggqAjQZiaoupONIQESIAESIIHQI0BlJvT6nC0mARIgARIggaAiQGUmqLqTjSEBEiABEiCB0CNAZSb0+pwtJgESIAESIIGgIkBlJqi6k40hARIgARIggdAj8F9Wdx4/VRrydwAAAABJRU5ErkJggg==)
2.4 Comparison between discrete entropy and continuous entropy
H(x) | Discrete Distribution X | Continuous Distribution X |
Maximum |
Uniform X |
Gaussian X |
+/- |
![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJAAAAAkCAYAAABmHbPbAAALuElEQVR4Xu2cBaxtVxGGv+LBCe7uWlyLFysUSnF3Le4Ed4K7uxR3dy0tbsXdJTgNXvK9rHmZu7r3Xuvsu+/NPU0naV7z9vI1a+aff+a8PVgfOR7w94WXe3zgbwuPue7DHRc4AXA04M/AYVMb2mPk4+kBL6wl/wO+WzU6IXCaVkfg18CfOtqdCHgIcHTgfh3te5ucDXgW8ATgU72djsTtzgPcCTg7cEbAu/0J8FXgZcAPhvY+pEAqwHuAcwMnbRzYj4DLAz9L7R4E3B047URftfr1wO0b418EeB5wcmBv4PsLX6Bj3wJ4GvCIhcdep+EeB9wGODXwiXL/Jyt3e4liJJ4IvLze1JgFUnFODFwUeApwuqrjq8vL/Svw8+rbsUv7SwNOmq3RX8qrfx2g8v1j4pSvCTyjvIRbAQdv0Y2oOI8Eng/cdYvm2MnDPjft+4XA3YD/lgWfBHgFcB3Au7sL4N3tljEFigYq0seA86c+bwBu3HEiupwDgf1T2zcDN+joe23gtcBvgJsBh3T02UyTZxer6Z/32MxAa9b3geWRu2zveZ8BzKNVeh9wQUCDca3s8lsKpC/8UPGLcTaXAz7dcVAq30eBC6S2KtNbGn0dXxeqbFhsx5xzm3gObwKuDzweeOjcgdaonxDlI8VtuezrAW8fWb8WOlz8x0vbXfi1pUBXBj6cBv06cNViGVpnpdX6TEH0thU0X3YMjJXBzgR8sCis4FbwvF1yVuD9BW/drkPRe9alO78w8EXgXz0dtrHNCwpodkoB8hUrLJuX4l1+FjBqVe4IvLhHgZ4EPCCNpP/TpfSIYFolCPkkcLUJ3GPY+FZgX0BFvTrwy56JFmzz2GJ9vgZcY4H5zwLo8n8KvKa4ginct+BWJoc6c3ncuidFr5ChRt3ZwOoDwCXLB+/S8zlsygKpbe8uSDwGvDMg0GqJ+Oe9JXKKtgLqB090FBu9sXw3XH9qa5It+K7L/hxwKmAJPKQCeTkXSgcvdaClW5rTWuU4blTwae/d2E735uNWxEIq06FTCnROQH/nYSoSbhcHvtWxUi9C/6pbCNFEOt6QyPW4wCuUeS4GfLtjnrqJihsRxFB393t4Y1yt7E2AXxWz/p0Z64gucmkvKlFMmH+/eQ5GfYJTL2O7xXD81mlS//+VjUXcH3hyanNf6Y8pBbpD2Xz0cdNqoOFcSwSjRlwhuiTdl5cyJLm90YDuqwczeEHiJtv7Io5RLKQvvBZf3X7APwExzr9H1iI3tcu/L2SFPOO9gPsAV0o4IhRJi+56e0jV1rn3fJdp9m50QSE+2C80OqswUjohUjm3nFIgfXbGO5JucgQ9Yls5g5B3ANed6GjIftPyvYePES/5agRz560uRbJR8C/uCMlch38nt3HbEWskYBQDKbozlW5M8XvOItp41kaYXkStSD6alxRF+sMqg85oKy+nwmZq5nzANxtjSeSKg0KEKPuNKZDM77sAWcgQyTz/riUCLi9IdxSiMon6h0SS0gM0taDcuxCIrXlUAK2hl631ySZYCyP9rjynsM0/TpSCOOSeIxOYxhH7Bf2gdRTcLykqkjivViTJUumEVwG/W3LCNJbnbOomoInRsetpsfwRTcZQKtxeYwok6PtytYFDO/CDXbQOpjFUJEXs5AWPabjpimw+L1NCxlXOTwzlq4ooQQWWmtdtqLwqi3NcCjgl8NIJrOS6pRLi8SwBpsf2MqZIunw9gI9CMnVJMeeV78Jcl1b2h41J9gS+lNq4xr3HFKgOwTezAVlk8c+Yj699q5peK2/P/FqVA0pD8zmPBnSdWjTdQ68cp7DgHqqi2ZbK78FkvXPU7aYUSfcu3liK0tBdefkhXyn7y/nMoX1okVW23G/fIQUSiMoE6/NCvJwjJNJGTkukLtkYouvKeKjuJvObw3sZ0jkRmHgoKAZfk9GNlQI3nHGrupHgRTw0AecSOKi1lMBIYrgctemmTQtplVoX3ZqjViD3Z3Bk5n1KMja0nWvaZ0iBhkLwXqug65Jkkv8Ikd/JEVm9SHmSeO1+m6tANVsqu+ojaJnmoUPLnEeviW9d3CrfxxTJtajcKlLrwsfmk57JD7R3f90urEbb3yhRzW87TkBQKP8Tov8Wd5h5HxMBqnmYEOtRWoBuaKxTlLl9YYr5m0d1rHmoyduqqHGuUs+cfnc3Uz/moK5SDaQ1FJtlpr93LoMEgyGTo4qWWuUYrPdJgw4p0CCIrkPeVdIXss1meEN68EN+7fY7FzCXvBPziFeUXTxF76lW7bIL85PAs4dAnTndYDdJWPGbUWBETDY0KDE/aYrpezMmtEzHe5EUDlGZgroYG1Icm/k1723/2oVZyvjOKgTvTV9IUMmsSpqF9GS2axe2mcuS6BKUK3I4EoyWZa4q2QJpDR2n9UJXnWOsvYpj9Khbz4pjuO0DMcJ0b3PFOzdllHNfRq+teqs62JEeOaBWIF+/nMyc9IWuR/yTNy0XZEQ0JXXEJzEoZbCqWBmpafYRKJp5XeocQJ7dqhjB+qTNgtfWfuRntDhDiqNCG8jM2cvQvAYtPu4Q2XepjSmJRHO0Md1zYK1AOZKx4SrpC4vMLFMN6S39qBN7mtbPt067+i7+0XKa6dbHB4h3TWbDV5FjlhcazLlnoFvcqpyVD+9eA4ojfjT4sCpzDiac2nONZ1pZhlwp4bi7ycdager0RU9aIRYq5rh5WnUvdqqJrZ6is/pwnFuwbkXd04vLsU29/mN18Dm1G9fXZ5C/ijJOtQ3Fcewoq7C9rtI0gfuYCj42s456jz4SMc4Y12VmQtwV7LwWUWx2eFagupELbIXgsQkLsKX/c+qjFztZdys4C1BnEdkq0YVpC82rrlDmNgN53adVjVE6YUQjy3zQxOlrzazCjMPylwpm1JcSFUdXpYXLiqPF1oJLHOY83lLz1uO4r0gvtSoPjAI9kxDdrEq0oSLRcgtLUEM0m3bs4RsMNyUfc/rCFEX9k5+xwxCQRaLWhWVeKPdR4U0G/qL8pXNIsKkU0T/TELK3pkbMg5kYtlTVi5talyG77LlEnqbaoGBOtFPv9RzFVQ0pztJsc4/Smf7x3OLhiot8fEOSAx3xoYbFn/1sUCBBlAnKEF+EwPSPHasxVaAlCFm1os9LNoLTtBoN6Ip+X81rlKc70doJ1v1/uZ7/FIwSP4DLRWEOoUWTTPSAxBNGMVMiKxu1wYbzgsWpGqPW8VhuIrXhuNniyK+ZNPW/pfNdrTXFd3OeJlZ9LD4S3VJOc9jOX9d4NxoHXaoPavcvcXzREm/iDoGcWhli+GtVoIfpC65BpEDV9rqOoXSBbsULMGNu/ylxHVqeqHhzkfWP/WoewvEc17Y5QjIVI6DO9S62fWbZY+twA8u5X/FJJkZbfevvRqSuxXqbEKNcgb2vun4kq46/RHutyWMAGWoVxTJmk60GE3qlhxVuTmzmmW6wxl6cGuefQ6/MCj8HEt8Ev+KifUn+nQVdFmgNSfQVf4hDWrkkFcExtUJD+bMzlKjQel7FJKDuYMjFGmVopSKfNFW+kdeef8Ki8uv2xgrPei4vl7QKVLe7eKxnjbYxANFKSnuoC56pf3rmPiShjWH/EegVG2liW6Ibqwk5w2WVpCUqZi+Hojn3l6Iqm26tjkIsxfBSzezLpgYWGlqDpleCzBKEsVLaul9wUmIfQ/dV6YR6PBXSlIMWx0Bhq6iA1h30fFcXzCcaCBlIWPpr+kpPMJoZaP2sp2fiJdsIkAXj+mYpBZVpu0TsZKgqober3nehiS0P2Qm/xFhoOxuH2WkK5Oq0HBJogmZzWWKIrRaJMul9QWSLVNvqtazV+DtRgTxAUwcSkboyS2mneJslDjwSyP7pPwxxlHSewE5VIJev+zIiMIrbyn9cIZTnKMvTqTS52U5WINdpdCALrDszI74EoZf3b7RnJPfwhdnmGVexnl12ugJ5qkYE5oUMLZf8rby8hwy47PRW/+sf66kdHateBwWKbfQkQju2vKHJkTpCWvUw5rT/PyhUc9pPszFFAAAAAElFTkSuQmCC) |
could be negative |
- Maximum entropy H(x) :
- In the case of discrete distributions, the maximum entropy configuration corresponded to an equal distribution of probabilities across the possible states of the variable.
- For a continuous variable, the distribution that maximizes the differential entropy is the Gaussian [see Page 54 in PRML].
- Is Negative or Positive?
- The discrete entropy in (1.93) is always
, because
. It will equal its minimum value of
when one of the
and all other
.
- The differential entropy can be negative, because
in (1.110) for
.
If we evaluate the differential entropy of the Gaussian, we obtain
This result also shows that the differential entropy, unlike the discrete entropy, can be negative, because
in (1.110) for
.
2.5 Conditional entropy H(y|x)
- Conditional entropy:
Suppose we have a joint distribution
from which we draw pairs of values of
and
. If a value of
is already known, then the additional information needed to specify the corresponding value of
is given by
. Thus the average additional information needed to specify y can be written as
which is called the conditional entropy of y given x.
- It is easily seen, using the product rule, that the conditional entropy satisfies the relation
where
is the differential entropy (i.e., continuous entropy) of
, and
is the differential entropy of the marginal distribution
.
- From (1.112) we get to know that
the information needed to describe
and
is given by the sum of the information needed to describe
alone plus the additional information required to specify
given
.
3. Relative entropy or KL divergence
The “extra number of bits” interpretation should make it clear that
, and that the KL is only equal to zero i.f.f.
. We now give a proof of this important result. ![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArsAAAA2CAYAAADH0lGtAAAgAElEQVR4Ae1dB5gUxRIukoqgoICK+pScj5xzkowgOaPkIHgEkZxzzhlFCY+cs2QPJEoGyaCSw6HkOO/7a6/nZudmdmf37uDOV/V9dzPbobrn7zDV3VU1MTRN00joX4nA+nXr6PixY9S4aVN6++23X8ozHjp0iObN/a9bWY2bNKZUqVO7hckPQUAQ+PchgNfJnt27adevv9Lt28H0/vvvUeEiRShTQEC0ftjxY8fRpUuX9GeIGTMGDRg0SP8tN/9+BKQPRO82jiHCbvRuQKm9ICAICAKCgCAgCAgCgoA9AjHtoyRGEBAEBAFBQBAQBAQBQUAQiN4IiLAbvdtPai8ICAKCgCAgCAgCgoAg4AEBEXY9gCNRgoAgIAgIAoKAICAICALRGwERdqN3+0ntBQFBQBAQBAQBQUAQEAQ8ICDCrgdwJEoQEAQEAUFAEBAEBAFBIHojIMJu9G4/qb0gIAgIAoKAICAICAKCgAcERNj1AI5ECQKCgCAgCAgCgoAgIAhEbwRE2I3e7Se1FwQEAUFAEBAEBAFBQBDwgIAIux7AkShBQBAQBAQBQUAQEAQEgeiNgAi70bv9pPaCgCAgCAgCgoAgIAgIAh4QEGHXAzgSJQgIAoKAICAICAKCgCAQvREQYTd6t5/UXhAQBAQBQUAQEAQEAUHAAwIi7HoAR6IEAUFAEBAEBAFBQBAQBKI3AiLsRu/2k9oLAoKAICAICAKCgCAgCHhAQIRdD+BIlCAgCAgCgoAgIAgIAoJA9EZAhN3o3X5Se0FAEBAEBAFBQBAQBAQBDwiIsOsBHIkSBAQBQUAQEAQEAUFAEIjeCLwUYffu3bv0/Pnz6I2U1D5KIvDw4UO6d+9elKybVCr6IyBzl/M2fP7sOa1audJ5BkPK48eO0Z3gYEOI69YuPExCi4Dly5bRixcvLGIkyBsCmqZxe+D6bySM62fPnkWbR/vnn39Ehgpna0WasIuJpkbVapQnR07KFpCZzp45G86qSnZBIBSB27dvU7PGjSlrpgD+Q1+7dvVqaAK5EwT8RCCqz10QKoNvB1v+qU2Fx48fh4m/f/8+I2IVB34QAPylp0+fUrMmTShmDP9eKYMHDqI1a9aEKd4uPExCiwBg8U2bNiIkWGBjF7R71y6qWqky5ciSlfLmyk1PnjyxSxrtwv+4eJFqVa9OeXPmYpnkwvnzUeoZIHzPmT2bOgS2o4njx9OjR4/oxIkTVLFceSpeuAgVzJuPDvz2W5Sqc3SqTGxfK7tv7z46cuSw12y3g4MpRowYdOvWLa9p/98TYPWMSebY0aP04MFDSpEyBX1WqhS99tprjqC5eOEi/bpzJ125cpk++uhjyl+wAH388ceO8mJAnTl9mn7//Xe6cP4Cdez0raN8vib6ecMGghARK1Ysypotm579q0aN9Htfbrp36Uq7du2ibUFBVKNqVfpt/36aPm06devR3Rc20SLtiePHKV68ePTJp5++0vpGlXpENghp06XzOHdFJg4Yg8eOHaUyZcpSrNixLB911aqV9NPMH+nQwYNUvERxun//Ae3ds4fHVYuWLal4yRI0bsxYmjJpEuXNl49fmA/u36eChQvRhEmTaMqkybR65Uq6cvUK5c9fgM6fP083rl+nXHly08jRo7mvWRbsIbB/n770yaefULkK5T2kso/Cix5CvJnsws3prH5XqVqV9u/bR6NHjqIO33a0ShImDHPxi+cvbLEPk8GHAMy106ZMpf379tKLFxqlTJWS2n4TSO+8+44PXPxP+teff9LjJ08oZcqUtkxy5MxJg4cNpbKlSlP2HDno9ddft00b3SI++vhjGjdxIn1erjwlfOcdSpkqlf4ITrDRE4fjxq4c9P1WzZvTa6+9zuN48MCBPC43/byR2gYGUtlyZalY4SK0dPESypY9ezhq8H+cVfORhg4erKX8NJmWK1t2rX6dunzFb/zVqVlLK1+6DN9nzRSgjR09hu8Rd/L3kz6W9P+RPPj2ba3aF1UYp8/LV9CqVKrM94Xy5dcunL/gFYSpk6doaVKm1DJnyKi1aNpMy5AmrZYuVWptzuzZXvMuWbRYS508hd5GaVKk9JrHnwQHDxzQ0B+yBmTWy8qSMZPWqUNHf9hpt2/dZj7pU6fRnj17pm3ZvJkxPHb0qF/8okqmUSNGaNeuXXOrztrVa/hZ0cYnTpxwi3uZP6JKPV7WM9vNXd5wsGpDX+p88+ZNrVb1GlqfXr28ZsueOYv25x9/aHt279ZKlSgZJn3DevW1qpW/0OrVrqNdvXLFLf6Pixe1HFmyctiQQYO0gf0HuMX78gPl58+TV3v8+LEv2dzS1q5RU/tp5o9uYfhhFx4moU3AvXv3+DmPHz9uk8I9eOeOHVrxIkU0zKt37951jwzHrydPnvAc1b1LV+3Z02faixcveI4umDefduPGjXBwdpb10aNHWo6s2bRM6TNoDx48cMu0ds0arpMK3LtnL885eNf/2+jOnTtaqmTJtZbNmuuP5gs2eiY/bjyVM3niJH4XX758mftGz27dtRJFi/K7PTg4mEs7cviwhvoL+YeAX2dOSZMmpU3bttJPc2ZTlqxZ9aXCkOHDaNW6tdS0eTMOixnTL/Y6v/+Hm549evDRBHbtFi9bSgsWLaKECRPS5cuXqct333mE4MjhIzR08GDeEenWowdNmjqFV4E4Uuzbqzf9+ccfHvNXrPQ5LV62jP7zySce04Un8vq1a9SkUSOqWbsWTZ46RWdVtFgxQn/xhy5ccB0/4Yjtrz//IvBauGQxZciY0R92USLPlStXaPzYcXQn+I5bfYKCfuHfWPnv2vmrW9zL/BFV6vGyntlu7vKEg10b+lLnRIkS0YjRo3jn9tTJUx6zvhE3Lp8EPXnylOLGjeuW9u+//6ZTJ09S0g+T0vc/zqT3P/jALd6V9wGHYRz5u4OHndAB/fpT3Xp1HZ9EuVUkkn/gRARzz+ABAxyVlC9/fp4TYQtQqUIFGjViBEFlKrw0Y/p0Onz4EH3b+TveNcapZ+06dejNePEIu+KRTWin1+LEoU8++Y/e1ggbPmQoDRk02G0n+9DBA1ydHDlzRXa1Xjr//Xv3EZ4bpxiKfMFG5fHnalUO+Dx88ICmTp5MOXLmIMhW6Budu3ah69evU9as2VgeQLpMAQGUIEECf4qWPETklzRar0EDeuutt2wBbNSkCcWJE8c2XiJCEdi/bz//qPzFFxQ7dmyedJTQtn//Po86Uzt37OCBCwY5cuVkPjiGAuH4DyonngjlBWQOoDRp03hKFq44CHDvvfcedercmd58M57OK+6bb+r3vt5A1UPR1atX1G20vm76+WfL+leoWJH7BCa5z0qXskzzMgKjSj1exrN6KsMTDnZt6ImfVdyHH35IGTNlotWr/DP2gupYvdp1qEjRojR67NhInYux4Ib6Vc3ata0eJUqEVatRg3YE7SDobDohbDa0DfyGVq5ew4IG1KT69u5Nly5dcpLdMs2KZcsoY8aM9Pbbb+vxEGqgKgAVLwjXkUlvvPEGb1AtX7WK1EIO6m+TJ02ifPnzuRV98OBB/g3h699Ge/bs5kfKnTtU2PUFm/DgYVUO+G3atImwOM2Xv4DOfuuWrfTg/gPKkzevHiY34UPAZ2EXuqRYKXuiJEmS0LCRI8IkwcoGuxVBv/zCjRsmARErZR8+dJg2/vwz65IijxVB/wmWups3bqKzZ85YWlYiL3YWd/36K8EoAwLg3j17+V7xBB9P5Rl5IC1+Y5LYsnkzG4AoPjDwQNjRI0d0AVTFebp27d6N9epq1qqpJ1NGJtjNgxWmHT1+/EiPevzIdY86KnpkiFdhVtcYFMMqONxhf/31Fy2YP5/adeigT7BWTM0YIw12trdu2cLtZ8yDVfDt26F64OhPV69cJexmK4qIvmGuU3jaHS9JvGzRP86dO+fWP2AUhD4zfKhrl/vGjRv8POpZMNmt3bCBRo8bS7FjhdXh9Pas5ucAXztsVZlWV7t6+Msf9fY07lQdsOuINobuGnRU0c4Yf8gLgrU9+tkv27e7CQzgj3wY71bkqU2s0qswKxzs2hA7M+ibxj/ghZeYMcxKVxU692fP+m7UC751atSk3Hly08Ahg1lHXtU9Mq6rV62iVKlTE3akzYS22bN7t9sYvnnzJtsXKGM5cx5ff9+5c4ffJ8AaZOWdJVmyZFy/tWvW+sT+zXhv0leNG/P4S58+PTX5qhF16tCR3ze+MMI7CH3RaoMofrx4/D6CzYY3Qp+GbcXa1Wvc5giMZ+RX/QhzJOYa9R7Bu+/0qVM8Ri795RLYlyxezEZQKPP+vfs068efCLrooEMHDvIGCBbY6K9YzCxfuoz7rbc6GuPR/hh/Cxcs4LFrjFP3wAbC/qKFC3ke7Ni+Pa1ZtZrQP1DuyhUr3N6B2GXfsH49ywaKh/GK9zBkghXLl7O+9uZNm9289ezdvYfix49P6dKn52y+YoN+BmyNBC8isBnxRFbl4FmA+9zZczgr9Obxm8PmhITduMG/8d4w08sYX+Yyff3tZHz6ytPf9D4bqBmNizwVil2Fo0eO6kn27N7FlrEYdKB3332Xfpo7h9KlS8e/MahgdDFowAA+kkOHxLFgseLFaeSY0fpEgXRz58yhwQMG8osvWfLkPPkkTpyYBg0dwkfaYIiXY/vAQB40+A2ViwH9+tHJ30+yEcWYceO8lmfm0ad/P1YQP3jAdcyDIzIcEe4ICmKjD3RoUKnSpWnC5El8HMEBHv6Vr1CB8KcIgxwvddB7779v+RJRaUuVKUMTxo3nSW3Txk18jL91i2sgYhVZrFhxlfSVXBctWMDHLsVLlLAt34xx/4EDaf26dTwxIxOOV2FUAEMc0MoVK6lr5846vz69ehH+lq5YQZkCMkVI36hUqbJb3/G33bGT9N23ndh46KOPPuIFHtyk5S9QgKZ9P4OfDQYrsLxV1LBePb49c+E89+EO7drpkzUEXuwsgpyMA1+xVXUwX8HHqh7+8Hc6zlEHvLAG9O1HwcHBlDpNavrj4h/0+htvEBZ2OAkJ+mU793817jZt3UqfJvuUrckPHz7MGEFIWbk21MrfSZuYn1/9tsPBrg0xt0E4UQThYc/+/dy/O3/XiYUTzCEbNm0Mo2aA+e/ChQsqq6MrBPi6tWrTW/HjU/eePR3NP44Ye0i0b88eypIli2UKqFjFifMatWregtp17MDC1NMnTylZ8mTUtXMXmjhlMqF9/KX/zp1LP0yfQVWqV6M5s2ZT+YoV6McfZlK7Du15jCm+agcVRnzNW7ZQwY6vOKWsXrMmValWjTasW08QyD74ICm1aNXSTY3PjqFqR6iOmClefNdpF4wEoY5lR1jEfNWwIZ07d5b7Dfq6opbNmtPVK1do1z7Xwm7M6NE0feo0mjnrJxbq6tepS+fPnePNnhWrV3F+qLjs3etKn+S9JLxgxEIOwhyE57r16rGQ2aXTdyx0YkGJE93effuoYj1e8Z4PbNuWkiR5j956+y0CHxgPY/EAwpgd2H8ALV64kEqULEmJkySmoYMGs8pIwUKFqHKFimyghdPHAwaD+O+nTefd6EVLl7iVD8Fv/NixNH3aNCpUqDAlee896talKz158pgOHnYtjLEIOHrsKMfDUBoLL6fYwFgc7x2cZEDWUFgDq5JFi3GfXrN+vVud1A+rchAHrLGgxfsefSxW7Nj6AhcyBgxU8f5DmsJFiih2+jWyx5dekJ83Tsenn+x9z+afqm9oriZfNdKNjv7888/QCE3Txo8dp8cVK1xEmz1rlta1c2c9rE3r1nr6RQsXcjgMrK5cvqJBmRtGDzBu69i+vZ5u+bJlev59e/dyOAwbkA6K58oQ7v69+9qwIUP1tJnSpdfvYfTgpDwzDxhVzZg2TRs3ZqzOC+V+Wb+B9t85c7VypUvr4TuCgvQ6+3IzaMAAnceyJUu9Zl26eImePl/uPPr99m3bvOZVCZo3acr5ItpArcrnlbT2gYGqGO3wocN6/Tp3+o7DzRhnC8jMGI8eOZLbE/iWLVVK5wHjDhikIRx/6FPoK8+fP9ciqm+Y6+RPu8MApcxnpbiOMDxE/TZt3KTXe+WKFfxMT58+1fCs6nlghIDnAZnrofIgzsmzmvN7w5YLtfhn5qPqYQ53wt/JuEMVYCikMFm1ciXXyhgGQ59bt25pTRuFzj/KoPPUyZNsEIn8FcqU1Z/IaZsgg3HusptTFA52bYg2b1C3rv4caFtF06dO1fLmzKX9/fffKsjt+lWDhhr+PBHmR9Qt6JcgDYZOMGpFX2rU8EsN/D3R9evXNRh4gvr16aONGDbcU3LbOBg9TZowIUz8P//8w/VAhBoHq1eu0tMp4zkVYGeIZhc+ZtQoNsY9d+4cs8C8kDNbNi1tylRhDLCQoG/v3lrJosVUceG6oh/9sn07G//BSNtsVGpmvunnjdwHAtu0NUexIRz66ZhRo8PEWQUUKVCQnxN1AMF4Ce89vIcV/TDjey5PjRuEYw6FQSP6pCK8k4sWKqx+8nXD+vWcd+GCBVqzJk20/fv2aejfqCMMJ50QxiH6Rf8+fTk5+jjyV65YkX+DH4wmYUgN/opgkBWQIQMbHWMOhFFu9SpVVTRfS5f8jI3swEMRsOjQrh3jgDkWhPwwxjPmxzhBPWAMZiQn2MBY8eKFC4wBjLphGA1C2ahzz+49jCwt763KgUEq6lSzWjU9D4zWEQbjUjvydXxZ8QFmMIj35W9Av/5WrMKE+To+wzCIhACf1Rh8F6ddOXr27sWrxb79++uuVnC0o2jKxEl8mzZdWvog6Qe8osFOHWjl8hX6MfWEseM4DO5TlH5qlWpV+bgOu0aTJ07keBxBZckauuPwZaNGNG/hQq7DN+0CyUl5Zh5YyUMf+eu2bejj//yHy8GKbOyE8VSrTm36uk1bDsO/k4YdHT3Qyw2OcrAix67soCGDqdIXlT3mgF/ZoUOGcBro3TZu0livV/++/WxVRTwyjaBIHO1gd61w4bArUmMRZoxbt/maMf6mXTtW2Eda+GjGyh2E1S52ixTFiR2H+wr00CKqb5jr5G+7K6MfuHND/bC7q0j5BMbORezYofrtcD2j8pnrofLi6uRZzfm9YWvkb7w381Fx5nAn/J2MO/BXah0w3CxX3uXOKnUad91ynA6Zw5AXYVClsiKFrac2scqHMPPzqnR2bYg2b/jlVyoZrVu7Tr//bf9vVLd+fTcdTj2SiE+1zEZlxnjzPU7BHjx4QEWLF6cx48cRjqlxhBuZhGNzjHOr43kcJWOnEmon2E0v+dlnbm7JEiVORNi98se3L/KNHT2G2gR+Q8mTJ+dHxLzwZtw3KXOWLGGM9ZAgQYKEvJMXEXjg+Hn3rt3MD3qt0PH1RMpXbew4YQ9SY8Z0qZAZVdLseKFc7LDC9RR2q0FQX8B7LyBzZj0bdi1B2KkFQR/09KnTlDtPHl2dDHrdaJfsOdzdWEGFAYT2a9ykCesUq3cZXPI5oWFDh1DMGDGofYi7N7jFBGXKFMBXuL5DWM8+vZk/AuFyD39ZsmTldzl2O9G/jLq1qC/cZObMmZPtW5gZEeEEcdmSpYR3hjoBBH/s5CoMkHav0tc1GKc5xQYnLZiLMmXKxO8iHM+Dzp09y+odtevWUdWxvFqVg4T7QnbXc+XOo+dTdjxQRbKjiBhfjRo3ptZt2vj0Z1S3tKubP+PTjldEhocdfRHJ3cAradIP+RcG4gfvf8D6rmoSQKdUOilXLl+hFk2bkUYaqUaHri06C5T7VTocVyrCUWCixIlZN+z330+oYLcrhMGcuXLyn9PycFxhpA8NwgqMSOAz75133tEneyUAIw/8GfpC+PJQ1+86U67cuWjw0GF8HOst/+hRo3V9OBz1YwGAyb52jZqs2gHB2al/SW9l+Rp/JER3OWMm3zwkQHVDEXwGw8gOumfaC82jOSX6UkT0DVW28epPu+NlBA8RJ46foBcvnrOqxeJFi3S2NqroerynG3+f1V9sPdXFGOeNv9NxhzGl9AdTpU6lv9hjxQyrs2ws39t9ZLaJXdlQ5wIu0E1ctWIFj0cIpdu3b6fuvXpaZoPwgmPppOXKWsZbBabPkIH69u9H37T+mhf1k6dO5SNa6P5GlgHqnb9dL/x48eKHqVK+Avl5Xjx08BAfWZuP6CEkgJ778RWrfn36cp+oWq2aXi6EQKhxVKxUSQ8z3kCFBII1hG9/jadRxoxp02j7tm1Up25dWr1urSOfxPFDjLnhv9dMSncZ7zBvdDBEEM1u8LMK2xaQcWMHAiEEYvX+gt40+hQ8TShSHycw8kKcMk4DRhCOQUG/BPG1cOHCfPX0D2qC69asJfhPh4cQ1K9b5y4sdOJdBJ3qH77/nt9vtQxGjZs2up5D+ZDFYgJk9JqwaeNGDsuTN1QwhGwAI2jojBtVVHCEDnIXdvfyRhK8GijyBRvkSZTEJRPcvHGTy8ScXrnKF7o6puJrvlqVgzT7QmwKjILt3r17OHsugxGdmV9EjC8YwitjeDP/8Pz2Z3yGpzyneV+asKtWolYVM354AgM0Zy6Xy5NcuVwrGwiWCIfeEgYtKG5cd2v+eCHW/dev37AqQn9hItJpeWZGyorVGB4jAtyrsS5gYDvC7vN3XTo7NirZtmULVwXuSpSjcOx2Y2cYxgxbN29+ZcIuXtagpB+6FjlGzDzde+onnvLdvnU7QvqGVRn+tjtW7b2692AhHG6GJk6eTF/Wb2BVhE9h/j6rv9g6rZw3/k7HHfTg1EI47hth9Ryd1scqXWS1iVVZCMOOY/Ua1Vm3GMISdj1wxY4Vxq0VQccTgpl68VulMYdhEwHpoRuLL5nNX7iAho4YTq1btKD5ixexjYQ5j9Pf+EAFjJLhVcVIqm2ePgs1DlXxKi2Mg0H4uIUiCJxnTp/hUyg49/eFsOmBXT/oCRt370PLsbZeR5muHXjfX3nQP50yeTLboGA3rEu3bj65WUuY0OUuysrjwr17rq/aQRj3RsolmOoX2OldvXIVZ1M7u3g/AosevXvr7JTxm7ENlEEVvEEowqYC3KMBJ+Mmyc4dQRxmzK/ymK9ICzpz5gxVKFOWXavBE0+1GtV5Vxn2NjgNaNKsqf5OBi4zps/gfNmyuz44hDpj3lX1g44v9LFBRgEWp75Y5OBjKmoRAz15vFMx9nKE7FxjPsHYAz+VDrycYsMFE+kLCOjhfvroU7YrWLjEXX9YpTVercpB/J49e/g5VZsiDEZ0mEvVybWRj7qPrPGl+Pt79Xd8+lueL/l8H/m+cHeYFruyihIkTMgDQf3GVQm4UGLHAMCRNnZHjATDH1BSkz9JYxp177Q8lT4yr9jx6dSxIxUpWoS6dOuqTwAoEy99WH7CWABK6tgZgjcMtRq7d9/1zG8nCHVnA3xwrAhh924IJuBlld/bc/mTR/G8evUa18PJjoXKE55rRPWN8NTBmBdeNJo1acpHaVBH6TdwAHsRMKbx9z6qPavT53A67hImCD0WxkvFKb3QQnfO1JxhzBuZbWIsx3wP46aJ4yfwPAajOyzaq9esYU6m/96+dRursnja2dETm24qVa7Maj/fdujI6lVNmzfnLzP9NGeOTwKaYovTkjGjRlGlymF3TOO+GZcX1mruVXmMVwhe2Nk2nsTxEfPDh1SsuL1BlpGH8R4eeEBwzWYkCBMQ+LNnDxXejPFoe/Q/bwsyYx4IR5MnTqJr165S8xYtaciwYY43Iox8sBGBuhm9yKj4+yFzuBLqVLjVFUI+5vfMISoLC+bN40UhNoOg0gOCp6N3EyWiAgVD3VjBP7dL5Se1zhY7u1DLSZMmrR4GdTF4CoGBmPpaI4RMnK5hAwrpvdHp06c5CdqnwZcN9QWSGo8Q5EB5DS61fpw5Uz+hzJYtGz8T6ofTCqUiAw8FWCRit1gJ9uCj/F4XLupSl4NsMCLEsw1UIpSLyyOHD/MJg3lMOcWGK+0m7N6g2T/NospfVLFdtKo8uFqVg3GDEyy8z6EmAcLpF4zoYNxqdFNn5GW8D8/4CmzT1mfPIthp72lYSBnrgnt/x6eZT2T8jhLCLjqwslzGFj50c9Rggx5Prx7daeasWby6hMsQAIqVoyKsJtSukfGIQsWbr07L82ViNJfh9PewoUNZRSNWrNjsLULle/r0Ge0MCqIvqlbhl2Tjhl/yYF+8cBFt2xHEkzb8NsK1y7Vr13kBgIkQq+TbwS4n6IgHYaKxyq/Ksrr6k8fI59GjhwS9vJdF2I2IiL4RUfVFH8bEBVKr8KdPQ1VbjPch6necFp989nbsHNWe1SlmTscdXqr40Ak+ioIdKAi8ONm5dv1amKKMn9S+fOky63Ci78KFm5l8aRNzXm+/PbUhVAkKFCzIggg+9xn/rfgsiNrx3LZ1Kx8h44TGHwps347atGrNQirc/mFnskfXbvwZWLs5TQkjxvIwr7Zt1ZrnGqgUmQm80C5YsFsRBCUILdDXNZa7ZNFi/lzrN4GBVtk8hqEPgZKncOnq4h67tjt37GSn+3YC2a1bN912gj0VgmN7ZfvRolUrFhyN9feU1yoOAlfefHnp9OkzPBcbeWHhAwy96cOifeBuL23atCzAYTczVNBJxsVi7odnl/6DBul4wxUXdjrLli9Hv584QQcOHKDqNWowLwjYUBeEp4QBgweR2jk2qpxA0EU7whsAXIjBe4andyzSguA9Rc17eDfDkwFUa9R7W33ICJ9zPvjbAX6mD95/n/sFFhngo/z8on9hMYNTAOjM3r51i73uBLZvTydPuLydJE+RgsudMW06ywn4gR1guDXDOIIbM1dYHv6QxldNGlPMGDEdY4N3KwhtBbpw/jzvqi9evox/e/pn1QZQg8HYgHBuFMDRPtBVNqpv2PEO7/hq0bqV7di1K9OoqmmVxpfxib6PDTXjwsyKZ0SF+W2ghsGH44grly/rdcGkqnz9Id64kr171+UvFkcl9+67jm6w2lbpW83jz5QAAAyvSURBVH7dmvngd81q1Qnfhu7VoyfVql6ddzPVBNG+YwdOhxchjipQDo430GlwPNEqhA8S3boZ6o8VEzfSKnJanpHHvbuunVSUpXYzINBAbwh0795dxZ6MR816oOkGdVq1wuU4HoNy5vc/6H9zZs1i1ytYxWKXFqtaEHZ7kQ+k3LigHRbOX8AYfD99BmOKnYT6DV1H5nb5uQ3v3KHrIUIE2gaLC7SBXR4u2ME/uJz55+9QH8HgadSnxmSBXQSQFcaom+oznMbgW9fYryDUqHaNqL5hVydf2h2LNSWszJk9m/r37ctfu+MHJmK9P6WXqhZ2iBs1YiQbKaoVshEb9ClFTp/VmF/1X0/YKv7mq5GPsR7GcCf8nY47HHOCIMjgAwk4SofAZibjzlTP7t1o9MiRBFdManwG3wnW/Y06bRPgY+xjxnvj8xpx8NSGqHPNWi7f5KhXufIV3I5Sjc8EH8JwZYhdMW8EF2xwXRU/fjw3H8N4MQ8fOYL1JeHfuXPXriz8T5nkMgIGX5UX9xirEAYVAXMY/NSoWo3SZUhPKVKk4CNhFW+8ZsocoO/mGMNxj51IvJDhB12NUfhQhR7nsBHDWXgx5/H2GwtaGDArAyGMSQhSMPg06nKa+cANptoRNccZf0MAmTd3Ln/pbPZ/51LBQgV1wdGYztf7Vl9/TfCjCvdnirCIgxAHIyElTKk48xX9Bu9LLJQgJHTr3Jl69e3L7QghF/YjHQLbUcvWrXWVNvDA84BSp07Duq1wcwm3Y2gXbDANGzKUipcsweUrfd0ixYrqxe8PcWf20ccf0c6gHWF21PWEITcBAS5DuW1btvJ7BC7VYINTv0EDxjF2bJfePeqwbu1aGj1yFDVq0pg3BtRR/sWQj39AsDx06BB1+a4zNW/ZkuUNLAoGDxxEVapW5RKVGiEEYiwkISgr/Vd8LQ7+gfPnL0BK3xluDB9iIyZRIp+wUc+ZOLHL8BXuBus3bKjvPKt4q6tVGyCd8gGu6oswZbCW22CwZsUTYeEdX2h/LGJ8+cNc4Imcjk/MBw3q1iW42jQ6KvDEO9xx/np4mDN7NrvHgIsM459y9QPXUsZwuEfBN8DhcsQYrtzfoB7Lli7VjO6z4NJjyqTJ7N7DWE+4cilSsBDzgbsj8K5UoaJ29MgRPdm0KVPdykGZpYqX0OOdlGfF49edO7VWzVu48YY7lL/++kt3laWeD+6RPBFcEam0dtfbt24zC7iBQRqzO6IVy5ez+xjEAUtcUZ+NG352K9oqP9wNWZULVyQgqzxuTD38gJsouGiBixQQsLAqyw5juHIxpkcbw/XL/P/OcwtHGqPblojoG3Z18rXdlyxarLdJrmzZtZnf/8BuXtRzwW0R6PHjx1qdmrX054IbodOnTmlW9Zg0cSLnwT9vz2qVH/3XDludsenGig/qYRXuhL+TcQ4XScOHDmP3RMALc8HE8eN1jNTYgssp1U+RDu634LIPbnsUzlkzBehP5KRNzHMX+Kxft87yeVV72LWhKvjJkyca+gB4oW3tqP03gVx35VrKKh2eAW60wAvPibkP95jfft6wgbPA/RNcS2E+rVG1mla+dBlOA9zgkjFfrtxaxrTpOA684PKx2hdVmBfmVPBDms+KFddQJzuaO2cOu7RSrpiM6eBOC3yGDBrEcyZcb8H90qGDh4zJ+N7OxZhV+N49e/n54TJNuWhDOXBJaEVwQwWXZGjDV0mzZ83i9xbmMLhhQ7vBlZPRHZhd/dAf0I7cziVKasePH+ek3bp04TC4FTO6GlN88I7BPAz3ib9s/4WDMY/CzSQwwRhWhD4CV2TGvof2RZnFixTR/rh4USW1vWIcwK0c8oA/+hbeBYrg0gxx+INbSrhOWzDPNaejLBBcmKp3GeZCyA1w7wZ+6LNGF3bK9SbienbrrmGcLV60iPnDTRvcEIJ69ejJYbWqV9ffSb5gw0w0TXv48CHzgZs0o/szFW91tSoH6eDKDTiodzzClHyEZ/ZGvowvb7wiMt7J+EQfg0vCzBkyanC19jIIK+4oR2jos2fPcsf1VDn478OL48GDB56SeY1zWp5XRpGYAC91DFwMZjNhsrx29ap27OhRnhiMk5VK6ym/SmO++pNH8cDLr0DevOzPWIW9zGtE9Y3w1hl98/y58/oLDbicOX06jH9VtCH6PPw5+kpR5Vl9rbeTcQdB5cyZM7wgwEtBvSiVsIsy0d/xgoSAZyV0mevltE3M+bz99taG8BcOgdKOIAThJX/wwAG7JFEuHH0PAoiVoAlBFb5v0T4QVtCOdoKdlVCLh7ULxzyIcYQ5qvO3nVigsxs72BBAPazmzpcNKObpBfPnaxDsMC/4QqqfG+d39Hf4Gvb0bPCpbH5HIsyMF/hAmDPTpUuXGGdzuN1v1A8CNf6s2hv8IOQqUr73T5w4oYI0+J/Fn5HwG769zQRMzc+CzSf0DUVW2CHOKTaKDzbUMAepRaUK93Y1l4NFAcY6FjyKIDzjewBqw1CF2119GV92PCIr3Mn4RJ+8c+dOZFUhDN8oobNr3p7G8YXSjTHHGX9DgduJErcxj9W90/Ks8r6sMFiWWvkURfk4AoMRiNH1k7lenvKb06rf/uTR88aKRUOHD6eO7Tvw50TtvrKk0kf0NaL6RnjrBR0mfDFKEdRLUqZKpX7qV7ShtyMiPbHpJqo8q6laXn86GXfwi6s8jdwnl/qTmTFUnKAX65SctolTfiqdVRtCzQk6i9B1w9GxUi1SeXCFLjG+hIjjfXx5MUvWrMboKH2PvlenXj2CqyejrieeFUfK+Hoi2gf6m0qHMyIeCCprahzt2rWL4OJQGfmY+ePLnE2bNbdVHTGnh1oAuzo0R1j8jhkrpq6uZBEdJghzNHRm/SGrfo75RPkatuNp9Fqh0liF2fGBAZwvhHoa1XrMec38Dh88xMZvqVOHGtBZfX7aKgy8rd57Rp/mSGOFHcKtcLAKU88Ad6jw5YuvvvlCZp4wmIOHiEIGd27QK0bfK1TEu4u3yB5fvjybVVon4xPzsNLxteIR0WFRUtiN6IcUfq8GAXwWd96C+br+5KuphZT6b0EAxjSKjPcqLCpee/fsRfAPWqpMaf7U6IhRo8JU8++//6EChQqy311lfR4mURQOaBv4DZX57DPWR1XGNtBHxMtc/Y6s6sPABfYbJUs2sixi65YtFBx8m75sFPpxD8uEIYGod78+fUgL+YiNp7SIixkrFo0cPUoXvL2ll3h3BLAYPHXqFBuTQXCPigSDUejHft22LS9IBw0d4rceN+xyPq9ciT/VDIGwTr26bHNUrkJ51uHFBhMWj97oZY0vb/XwFu9tfHrLH5HxIuxGJJrCKwwCnlb4YRJLgCBggwB2P+fPm6fHYrKHmyW4SYrKdOrUSTamWjBvPo0aO8ZydzEgcwDhL7oSdlTHjh9PHdq1pyXLltHkSZNo8cKF/KXMSRMm0MmTv9PAwYMj9PFgTFulUmW6fPkSl7N82TI6ffoUe+1RBeGrcvBzPXXGDMdu1/BRnxWrXb5rFR+5Rh4CR48eZQPSrNlc/nUjryT/Of8wYwZ/WGPB/PnUslUrty9h+sIVvrOHDh5Ma9es4VMefMk1wdsJaNiQIbRimcurA77C6u2UatCAgZE+vnx5Lqu0TsanVb7IDIsBxYbILEB4CwKCgCAQXgQg3OITuEZKlSolNW7q8tpgDI9K9/iaFb6yhA/kwOr930zw/LBuzRr2Ke3rc8KDRpmyZcN8It0u3Al/eCeoW7+e/mECJ3kkzctFAN4o8JXJsuXKki+fx36ZtYSv6SULF1GOXDmpWPHi4Soa89jmTRv5gxFwyQeCe0V4Y4I3ilKlS4eLv2S2R0CEXXtsJEYQEAQEAUHABwTghhF+oH0l6CDCZzJ0K41kF25MY3fvb13s+Em4ICAIRF8ERNiNvm0nNRcEBAFBQBAQBAQBQUAQ8IKA3x+V8MJXogUBQUAQEAQEAUFAEBAEBIFXjoAIu6+8CaQCgoAgIAgIAoKAICAICAKRhYAIu5GFrPAVBAQBQUAQEAQEAUFAEHjlCIiw+8qbQCogCAgCgoAgIAgIAoKAIBBZCIiwG1nICl9BQBAQBAQBQUAQEAQEgVeOgAi7r7wJpAKCgCAgCAgCgoAgIAgIApGFgAi7kYWs8BUEBAFBQBAQBAQBQUAQeOUIiLD7yptAKiAICAKCgCAgCAgCgoAgEFkIiLAbWcgKX0FAEBAEBAFBQBAQBASBV46ACLuvvAmkAoKAICAICAKCgCAgCAgCkYWACLuRhazwFQQEAUFAEBAEBAFBQBB45QiIsPvKm0AqIAgIAoKAICAICAKCgCAQWQj8D6ar6jjkSWZDAAAAAElFTkSuQmCC)
Proof:
- 1) Convex functions: To do this we first introduce the concept of convex functions. A function
is said to be convex if it has the property that every chord lies on or above the function, as shown in Figure 1.31.
- Convexity then implies
![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoMAAAA6CAYAAADLNFMaAAAW/UlEQVR4Ae2dCXQUVbrHPxQE9ShbWIYAGgVE9kUWwXmyjOz6BiUJIOCBAYZFkPdY5UDEAVQUjJ43wpMMD3nzkBFc2DlPRRggLGHxSBIgIAQSQJAEhLAHuO98l1ehUl3dVdV9013d/b/n9Omqe2999d3f/b6ur++9VVVKCCEICQRAAARAAARAAARAICoJ3BeVrUajQQAEQAAEQAAEQAAEJAEEgzAEEAABEAABEAABEIhiAggGo7jz0XQQAAEQAAEQAAEQQDAIGwABEAABEAABEACBKCaAYDCKOx9NBwEQAAEQAAEQAAEEg7ABEAABEAABEAABEIhiAggGo7jz0XQQAAEQAAEQAAEQQDAIGwABEAABEAABEACBKCaAYDCKOx9NBwEQAAEQAAEQAAEEg7ABEAABEAABEAABEIhiAggGo7jz0XQQAAEQAAEQAAEQQDAIGwABEAABEAABEACBKCaAYDCKOx9NBwEQAAEQAAEQAAEEg7ABEAABEAABEAABEIhiAggGo7jz0XQQAAEQAAEQAAEQQDAIGwABEAABEAABEACBKCaAYDCKOx9NBwEQAAEQAAEQAAEEg7ABEAABEAABEAABEIhiAggGo7jz0XQQAAEQAAEQAAEQQDAIGwABEAABEAABEACBKCaAYDCKOx9NBwEQAAEQAAEQAAEEg7ABEAABEAABEAABEIhiAggGo7jz0XQQAAEQAAEQAAEQiKhgMDs7m6ZMnhLVvRotDObNnUdpu9KU9bVqecoUc5EgFbYVTpxVtNdF3edTlWD3SyBsg62rT3AoBIEIIXD/jBkzZri5LV9++SVNnjSZNm/aTKXLlKa6deuaqnvo0CF6uffL9JeZf6GqVat61Jn7wVw6cPAAxVSOofIVynuUR0KGFQO3t9FJH9V/qj7179+f6tStQ3FxcaZNE0LQV199Rd/+77e0bv06On78ODVv3ty0rh15pgeGINMJJyv1VPmX1Xm08nDhHO6+xLyd2ImKfgnEliLVVzW7x3foCdy4cYNKlSpF991nPgZ28eJFuv/+++UnEG2dynFaPz8/n3JzcykmJqZIzZs3b9KdO3cC0124OC39n6Wixu9qiOzsbPHKy6+IqlWqivy8fA+NL1y4IFq2aCn+/t9/9yjTMo4ePSratmkrateqLfbv369lR8y3HQZ37twRX3/9tWR1/fp117XdaR/t2bNH1H+qvjh27JhpW27cuCGmTZsmunbpKmIqx4hJEyeZ1tMyreRp9VR93759W3CfOE1OOXmTr9K/vJ3DLD/YnM108JUXCb7E7XNqJ4H0S6C25HZf9WUvKHM/gRPHT4hBAweJM2fOeCh78eJF8f6c90Xc43Ei50SOR7ndDKdynNZnPW7evCn+0PkPonOnzsXUSk9PFwNeHSB+/fXXYvlOdshJ5WDW5R+Hhg0aim5du4kjR47Ii3m9uvXE5cuXPdTgi3xCQoLlhXXJkiVSTmJiooeMcM/wxqCwsFDMmTNHTBg/QbR7tp1sPwdGZkG1Gxg47aOpU6eK+D7xPvt+1sxZst0rv1lp2UQ78iyFWFS4evWqSFmYIpo1bSaWL19uUdu82Ckno5SS8C/jOXztB4Ozr/P7KosUX+I2OrUTf/pFpS25zVd92QnKwoPADxt/kAMCp0+fLqZwQUGBSP4wWdStU7fouuhPMOhUjtP6eqVnz54tdTUGg1yH46SOHTqKrKws/SG2t10bDG7dulU2evr06bIxvJ+Xl+fRsAOZB+SI4aZNmzzKjBmXLl0SsTViRbWq1cS5c+eMxUHd37x5sxg1cpSSc/piwMFgcnKyWPb5MpGTkyOeiHtCcnVrMOi0j7hN3J/r16/3yrJHjx6yzWfPnvVaRyuwI0+r6/SbR5zmzZ0nGjzdQAwfNlxkZmQ6FVFU3ymnogP/f6Mk/Mt4Dl/7KjnDl7yTdmon/vSLSltyi696J4qScCKQkZEhnqr3lDh48KCH2vv27RNpu9IEjxryAAl//AkGncpxWl9TfOeOnXImjPU0Cwa5Xmpqqmj1TCu/BnvMJ8+LZqJDt7Fv3z558qZNm8rv5557jipXruyhUEpKCj355JP0/PPPe5QZMx555BHq0aMH3b59m1Z+s9JYHNR9XkCdlZWl5Jy+GJQuXZrGjRtHffv1pVq1aik5X0kKcdpH3KYePXvQ31L+ZqrW1atX6cd9P1KdOnVM15IaD7KSZ6xvZ//s2bP09oy3qU2bNnT69Glav2E9fbrwU2rQsIGdw03rOOVkFFIS/mU8h699lZzhS95JO7UTf/pFlS25wVe9k0RJuBG4cOECvTboNRoxcgTVr1/fQ31eP96qdSuqVr2aR5mTDKdynNZnXQoKCuiNN96gd959x6dq7dq1k+vihwwZItcQ+qxsKHRtMMgXcE4NGzY0qHxvlxeErly5knr26ikXht4r8b7VJ76PLFyxYoX3SmFU4g8DtzfPaR/xH4EtW7ZQbk6uR9PS0tKosLCQ2EnsJl/y7MrgehykjP/38fT8vzwv7XPr1q00d95ceuyxx5yI8VrXKSe9oJLyL/05rLZVcbY6j93ySPQlbrtTO3HaL6psKZS+atdGUC98CLz//vvyRot+/fqFj9JeNOWnpLw64FVq1KiRlxr3skeOGkmpqam0dOnSe5k2tlwVDPKP8T+W/YM+++wz2rlzp1R/27ZttGTJEjp69KhHc/bs2SMj5hYtWniUecvo2LEjVapUifjfrJlM43H872LB/AX0+ujXaf78+TKwOH/+vPw21g3Fvj8MQqGnk3M67aO2bdtK8RwQGhM7BafWbVrTmtVraML4CfRR8kfEfegt+ZLn7Rh9fmZmJg0fNpxe7PWiHI3dlbaLkt5KsjUyqZdjte2UU0n7l1NfCZSzFR+n5ZHoS8zAqZ3Y6ZeSsKVQ+KpTG0H98CDAI2nLPl9GHTt1pGrVAhv5C3WLV61cJYPaUaNG2VKFZ1NjY2Mp+cNkOQtq6yAiclUwePnyZco+nk0HDxykvLw8qlmzJv1y+hc54nPt2jWPNu3evVvmeXtciMcBRFSmTBnq3bu3LOJHIfhK+/fvp98/93s6d+4cTZw0kcqVK0fDhw+nZk2b0Xvvvufr0KCV+cMgaMr5eSInfcSnqFevHvF0GAddxqRdYD79z08pJzeHRr8+mvhW/hdeeIH4Fn2z5EueWX0tb9fOXdSvbz8aOHCgDD5379lN4/5tHJUvXzKPMnLKqST9yx9f8Zezxlv1dyT6EjNyaid2+qUkbCmYvqradiDPXQS++OILunLlCjVu1NhdijnU5tSpU/T222/TJ598YvuxMfz4nKeffloGkDwbZTe5KhjkNYFvvvkmvfSvL0n9O3XqRNOTptO06dNMh0dPnzpNDzzwAFWvXt1ue2W9ommT5Sv4BhrTY7dv3069evaiTp07yVEdntrjeXgODDkwja0Za3pcsDP9ZRBsPZ2ez04faTLZ+KtUqUKnTp7SsuQ3/xjwFBavm+SRudGjR8tnEg4bPoxyTuRQcnJysfrajjd5Wrnxm5/xlBCfQIMHD6Y/9v4j7dq1i4YOHUoPPvigsaryfSecSsq//PUVp5yVwzMIjFRf4mY6sRM7/aLaloLlq4Yux26EEtize49sWaDrAUOJh58byDOSk6dMplq1na335z90nLQZVjvtcFUwqCmckZ4hNxs38R3Vn8s7J0eEtOPsfmsB4IkTJ4inhoyJp0DGjhkrL+Zz5swpVlz+0bujPO3bty+WH6odfxmESl+757XqI6OcihUrErPQJ16DdOvWLTkS3KFDh6IiHinh5Otfk5m8IgGGDf5DMmjQIPkQ0OVfLCcOjjT9DVWV72rn8WbLZidU6V+B+ooTzmZtUZkXqb7EjJzaid1+UWVLwfJVlfYCWe4lwL+HnKpXczZQ5KYWLViwgCpWqkgJCQmO1eIbwThlHbJ/k2ppx2cJwgHp6enyLFaLJfPO5dGjjz7qSKPdabvlCE6vXr1o7dq19OWKL6lVq1bFZPz1P/5KbEwDBg4oNrrDdyFzpM2jUFrkXexAww7P9XsLOA5lHaLck7lyDZvhMLnLU4v8j4ADDV/JHwa+5JmVcbs5KD6f732dndlx/DT3Pw39ky1W+uPt9JG+Pm/zxUv7AdDKtqdul5s8uqtPhw8flrvMzlsyk+etLuf3erGXvKv5u+++oznvzaFZM2fRG+PeoO7du9se3vcl36zMH04sR6V/BeordjnDl/zzJe5vf+zEbr+osqVg+qqZLyEvsgjk5OTIBsVUufeWjnBqIa87X7RoEX3//fe2b47Vt4+XtHH67bff9Nk+t90ZDGaky1fG+LqTmFvF0xlOEr/Llu8smr9gvnytHQeDfDfyrNmz5LoalsX/oPkGFk58A4A+8Q/fpUuX5EiTnXPz6/PKliurF1G0XaZ0GSpFpbyW83F2zmGnTtFJA9go+0BZr7p6E+vPq33s9JG38xlZ/Pjj3TvStQXx2nH8qjFOtWvX1rJMv43yTCvpMvk1R127dqUuXbrIPw0fffQRvTP7HRozZgy90ucVy8BeJ8pyMxBO6Yr8S5Wv2OEMX/LvNVmB2ImdflFlS8H2VUsHQ4WwJsBLgzhpI+Lh1pi3kt6icmXL0fjx44upfrngstznV6vysiRO06ZNk4/X01fkay+nwluF+myf264LBnn91eGsw/K5cFZrrirHVKYjR474bKBWyIv7ExMTacqUKfKCzfl8hyn/WG7atElewDmPF2yeOXNGHtbymZba4fJbG+Vr/5y9KeKePXsSf8wSB5x8x9Ps2bPNim3nOWFgW6ihIhvW+AnFjdJQRcmu3T4yO9mlgktk/BfItlGhQgV5Z5X+mC3/vHvXcefOnfXZxbbN5BWrYNjh6WjtB4gvos8++6z8ZGRk0Mcff0wffPABjRgxQo42P/TQQ4ajne0Gwkmlf6nwFbuc4UvObIRrB2IndvpFpS0F01edk8QR4UagfIXy8pmu169fDzfVpb58gxb7hLf4hm+CXLtmrazLa+GNSVsy9Vht+48xc92aQR614Qur1RQxN75KTBU5UmcV/fPULs+79+nTh/484s9F3Pr36y+39c8c5E7gxLejG+8C1YJBfmYdL+58c8qbRbJCtWGXQaj0s3teJ31kJvPSxUvSHrQy7h8O6uvWrVtshJX7d+PGjVSpciU5ha3VN34b5RnL9ft8Q1H7du1p7NixHs7LdswPBf/q66/kQ8bbtG5Dcz+YS/wYFn9SoJxU+pcKX3HC2R9eTo6JFF/iNgdqJ3b6RZUtBdNXndgD6oYvAW0Zl/GmwnBp0Tcrv6HjJ457fDb/c7NsAl9XtHKzp6loA1pxcXG2m+y6YDB9/931glY3j3AL+Y5e/nd68uRJrw3esWMHJSYkynWB7773brHAgO9a5rn1Des3yFE6FvL444/LKWrjHcp8jtRtqXINIb/xZO/evcSL50Od7DAItY5W53faR0Z5vKYxNzeXasTWKCriKVtmY3xrDa/D4H5Lmp4k1xkWHaDbMJOnK/bY5BFsdtImjZvI0WcevtemvbTK7JTzPpxH32/8Xj7ygB9ZlJSUJB+hpNWx+g6UE8tX6V+B+opTzlZ8Ai2PBF9iBoHaid1+UWVLJeWr69ato0kTJxG/AQgpugjwzAynrMPWN1DwnxGrxANO/LiaDRs2eK1qR47+YF/1+Zry8MMPe3y0WSWerdPKtSlhvWwtGGzZsvjspr6Ocdt1wSBPq3Eyi3aNyrdu3Vpm8TPOzBJPAXMgyA9gXPRfi4rWBWp1+dl0L730kgwOVq9aLbM5OORRxGPHjskRSs7k6TB+rQ2XxT0RJ4PFNWvWyBsGNFmh+rZioOnFo638hg5t2JwfuM2BdKiTP31k1JmH0vnRFDzqpk/x8fFyNE4bOeYAjZ8Pya/n69ff+1PpvcnTyzZus/MOHTZUPlamW7duNOb1MdTnlT7yBiLt/HwMjzi/NeMtSt2eShUrVCRt/aJRnnFfBSeWqdK/AvUVfzgbuajcD3dfYhYq7MRuv6i0JdW+yr9tI0eMpMWLF9PCTxeqNBPICgMC/CzhsmXL0oHMA5ba8rVeS7w8wizxyy/4MS+DBg7y+hpZO3L0sp3W1x/ra5uvN5kZmXdf09vB+jW9RbK0lyC75ZtfVF69WnVx5coVS5WuX78u4h6PE7NmzjKt+2KvF0W9evVEdna2aTln8sufq1apKtq2aStu3bol6+Xn54uXe78sEhISxORJk0XXLl3Fjh07xOpVq0WdJ+uISRMniaFDh4rbt297lWtVsHjxYq8vm7Y6Vl9uxYDrtm7VWlSrWk3U+F0NyeuJuCdEbI1YmVerZi3BMkKV/O0jvb6fL/387kvGc3L02dKGBg4YKF4b9JqYOnWqaNGiheC6VsmbPKvj9OVsG+vXrxfdunYTXV7oItatWxeQvajgxPqp9C+WF4ivqODMOsCX7lmeCjux2y8qbYl/71X76pTJU0Tz5s1F2q60e4CwFTUEkqYnyVji/Pnzpm0+dOiQGDx4sGjcqLG8fsRUjpHXiCFDhoi0tOI2s3fv3qLrpTGecCKHFXFaX6/8sWPHpK6dO3XWZxfbZnvntqQsTCmWb7XDd9u4Jt28eVPUjK0pEhMTbes0ccJE2YFmgVnOiRxx8uRJS1lHjhzx+MG4c+eODCIzMjJEYWFhkYwzZ86IzIxMweWBJFUXMNbBFwMuZ65m+nKet7JA2ubk2ED6SDtP38S+MnjX9vXf3Mbjx4+Lffv22Q56fcnTy7azzefftm2biO8TL9o9204sW7ZMMrdzrL6OCk6q/UvTj9vIP5BOfUUVZ/iS1hNCqLATO/1SErYUal+9RxFbkUCgoKBAPNPyGTFv7jwlzfnll19Efl6+ElklKWTMmDGie/fu4saNG45OE/Jg8Nq1a3LUhDvup59+khEtXzDtJo6yedTrh40/2D3EFfW2bNkixo4dq0SXcGWgovFHjx6VNvPtt9+qECdUy9MrxfbN/zp5tIL/tV29elVfXCLbbvUvlZzhS+pMx1e/uM2WfOmqjggkhTOBrKws0bRJUzkgEM7tsKs7x0GNGjYSHLg6TSEPBge8OkBezBfMXyBSUlLkNOzFixcdtYOnAOPj401HvxwJCuPK0cqAp+x5Op9HFVQk1fLMdOKL2Lhx4+T0REn/iXGrfwWDsxl7O3nR6kvMxle/uM2WfOlqp59RJzoIbNq0SXTv1l3k5eVFdIMPHDggOnboKAfV/GloyINBXlPVpHETOY3H02irVq5y3A4OHnld3KJFixwfGykHRCMDdnJeE8rTwCqSanlWOvGSA6d/fKxkGsvd6F/B5mxkYrUfjb7ETKz6xU22ZKWrVR+jPLoI8AghDxjxb24kpv3794thQ4fJNdz+ti/kwSCPkvBNGrymasWKFf62Q/z8888yqExPT/dbRrgfGE0MeBi8WdNmgqcIVSTV8lTopEKG2/wrXDhHky+xndnpF7fYkh1dVfgOZEQWAb5JSdUMktvI8E2gZvdNONGzFFcuurU4zDf40SkLUxbSzJkzw7wl/qsfLQySP0ymDh072HoEkR2aquXZOWe41VFhW+HEWUV7w6WPg90vgbANtq7h0ofQEwQCIRBRwWAgIHAsCIAACIAACIAACEQjAdc9dDoaOwFtBgEQAAEQAAEQAIFQEUAwGCryOC8IgAAIgAAIgAAIuIAAgkEXdAJUAAEQAAEQAAEQAIFQEUAwGCryOC8IgAAIgAAIgAAIuIAAgkEXdAJUAAEQAAEQAAEQAIFQEUAwGCryOC8IgAAIgAAIgAAIuIAAgkEXdAJUAAEQAAEQAAEQAIFQEUAwGCryOC8IgAAIgAAIgAAIuIAAgkEXdAJUAAEQAAEQAAEQAIFQEfg/CgaB6C56BQYAAAAASUVORK5CYII=)
- 2) Jensen’s inequality:
- Using the technique of proof by induction(数学归纳法), we can show from (1.114) that a convex function
satisfies
where
and
, for any set of points
. The result (1.115) is known as Jensen’s inequality.
- If we interpret the
as the probability distribution over a discrete variable
taking the values
, then (1.115) can be written
For continuous variables, Jensen’s inequality takes the form ![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmUAAABUCAYAAADDNkAGAAAgAElEQVR4Ae2dB3xVRfbHD1J1ZWHpLUEU6b0FKYYmLeDSEQj4IdSwIKKC4Ir7B1n4g1RdBZK1AcKKQBIIdTWshCIQghCSEHov0gRROnc/v4n37cvLK7fm3fty5vOBe9+9M2fOfOfcvPPmzpzJI0mSRJyYABNgAkyACTABJsAE/ErgCb/WzpUzASbABJgAE2ACTIAJCALslLEhMAEmwASYABNgAkzAAgTYKbNAJ7AKTIAJMAEmwASYABNgp4xtgAkwASbABJgAE2ACFiDATpkFOoFVYAJMgAkwASbABJgAO2VsA0yACTABJsAEmAATsAABdsos0AmsAhNgAkyACTABJsAE2CljG2ACTIAJMAEmwASYgAUIsFNmgU5gFZgAE2ACTIAJMAEmwE4Z2wATYAJMgAkwASbABCxAgJ0yC3QCq8AEmAATYAJMgAkwAXbK2AaYABNgAkyACTABJmABAuyUWaATWAUmwASYABNgAkyACbBTxjbABJgAE2ACTIAJMAELEGCnzAKdwCowASbABJgAE2ACTICdMrYBJsAEmAATYAJMgAlYgECudcp27NhBdevUpcOHD1ugG1gFswhcuXKFGtRvQLExsWZVwXKZABNgAkyACRhCIJ8hUmwmJCkpiQYNHEQLPlxA1apVs5n2rK4aAiVLlqSo6Cjq07sP5cmTh/7c7c9qinNeJsAEmAATYAI5RiDXOWWpqan0St9XaMqUKdSlSxdDQd+/f5/emfQO/frrr5Q3b17q1r0btWvXztA6WJh6Ao0aNRKOGRxx9EuXrsb2u3qN/FeCbdR/7LlmJsAEmIAvArnq9eX58+epZ4+e1KtXLwofGO6Ljar7kiTRa2Neo6NHj4oRuDt379DA8IF0/PhxVXI4szkE4ByPHDmSIiMjKT093ZxKLC6VbdTiHcTqMQEmkOsJ5Bqn7NGjRzQqchSVLVuWpkydYnjHJ3yXQKtXr6aIiAjxpb82bi09fPiQ56wZTlq7wImTJlL16tVpSMQQMZqpXZI9S7KN2rPfWGsmwARyD4E8En4+54I0f/58mjF9Bm359xaqW7eu4S3Gq9C9e/ZSxpEMKlKkCMEpu337NvV9pS/ly5fr3hIbztcogadOnaLQF0Pp5Zdfpo/+8ZFRYm0hh23UFt3ESjIBJpCLCeQKp+zAgQPUsUNHGhwxmKZPn254d0N+u7btCHOXNm7aaLh8Fmgsgblz5tKMGTMoNi6Wmjdvbqxwi0pjG7Vox7BaTIAJMAEnAgH/+hIDgZPfnUxPPvkkTZgwwanpxp3Gr4sXwpo1a2acUJZkGoHIUZHiNfakiZPEK2bTKrKQYLZRC3UGq8IEmAAT8EAg4J2y7777jnbt2kXDRwynokWLesCg73J8fKZT1qhxI32CuHSOEICD/uZbb4q5f59/9nmO1OnvSthG/d0DXD8TYAJMwDeBgH59icn9bdq0odOnTlPy/mQqVqyYbyIqc2C1ZbMXMkfI0tLTCHGxOFmfwN27d6le3XqUP39+2pe8jwoUKGB9pTVqyDaqERwXYwJMgAnkMIGAHinbtGkTpaWmiRAYZjhk6KvExETRZcEVg9khy2Hj1VNdoUKFaMiQIXTp0iVas2aNHlGWL8s2avkuYgWZABNgAoJAQDtlS5YsEY0cED7AtO7euXOnkN24UWPT6mDB5hDo1buXEPzJx59QIC9CZhs1x35YKhNgAkzAaAIB65SdPXuWtiZspapVq1K9evWM5ibk4Yt8545MpwwrLznZi0ClSpVE3DIEk03am2Qv5RVqyzaqEBRnYwJMgAlYgEDAOmXLly8Xox/t27cXex6awRrR+rHhNZIZsc/M0JllZiXQOayzuLAmJjBfYbKNZu1v/sQEmAATsDKBgHXK1q1dJ7i3aNHCNP7yayFUUL1GddPqYcHmEejcOdMpi4uNIywMCbTENhpoPcrtYQJMIJAJBKRTdu7cOcrIyBCR9EOahpjWf/Kry4oVK9LTTz9tWj0s2DwCtWvXpgoVKogRz927d5tXkZ8ks436CTxXywSYABPQQCAgnbKtW7cKFA0aNqA//OEPGrD4LiLm6vw+yb9WrVq+C3AOSxLIkycP4RU30o7tOyypo1al2Ea1kuNyTIAJMAH/EAhIpywhIUHQNPPV5enTp+nixYuinpo1a/qn97hWQwjUrJXZfzt2BJZTxjZqiHmwECbABJhAjhEISKcseV+yAGjmtkc//PCDo5Nq1KzhOLfLycmTJ2ni2xN1qztn9hzas3uPbjn+FFC1SlVR/d69e+n+/fv+VMXQuq1ko0bZGwDllM3ZUWdDDYiFMQEmkOMELOuUrVq1irp26UrDhw2njRuVb/J9/fp1unDhggBZuXJl04DKjh8qsNvry8OHD9PLXV+m8IHhuvkMHjyYRo8eTfIrYz0CV69eTR9//DFNe38aLV60WI8oVWWrVK0i8sMhO3LkiKqyVs5sto0qfUaNtDfwNtLmPPWflXX213PiiRVfZwJaCdy7d8/rAqubN2/SgwcPtIp3lFMrR21+R0VOJ/g+0aS7ZMH01bKvpHJly0knT56UevboKZUqWUq6dvWaIk23bdsmlSheQipbpqz08OFDRWW0ZGrdqrWo55mKz0iPHj3SIsIvZW7cuCE1bNBQWrpkqWH1JyUlSdWqVpNOnDihS+aE8ROktm3aCq49uvfQJUtt4RrVa4h6v/nmG7VFdeWH7Tx+/FiXDE+FzbRRpc+oGfaG9hplc+7YWV1nfz4n7njxNSaghcDpU6elQQMHSZcuXcpW/ObNm9KsmbOkSs9Uks6cPpPtvtILauX4yp+QkCDVr1ff679JkyYJ9VJSUqTwAeHSTz/9pFRdkQ+xvCyV7t27J9WsUVPq2KGjdPToUfFFWeX5KtLt27cV6blo4SJRpnGjxorya8n022+/SaVLlRb1hIWFaRHhtzL4g96nTx/DHYF33nlH6t2rt265Fy9eFFxz2inr3q27qHfatGk50jewoeioaKle3XrSypUrDa/TTBtV84yaZW8AZpTNucK3g87+ek5cWfFnJqCFQMJ3CVKH9h2kCxcuZCn+yy+/SPPmzpOer/y8+HuMARYtTplaOUrzDx061KEXdHP3b9nSZY42wYfBj+OMjAzHNV8nlnt9uWfPHrp8+TI1btKY8PoxJjaGdu7aqXgV5ZmzZ8QAIsJUmJUOHDjgGHKtXau2WdUYLjc9LZ2++OILioyMNDyg7siRI2nbtm2E/Ub1pAL5/bMxOHZ+QDp29Jge9X2W/fnnn2nunLnUqGEjwhy2r776inr37u2znNoMZtqo0mfUTHsDD6NszpmtXXT213PizIrPmYAWAqmpqeI7aP6C+VS2bNksIo4ePUrNmzenb7/9Nst1tR/UylGSH/sk79q5iz777DNK3p9MqWmpWf6NGjWK8ubNS506dXKoCx9m2t+nUfiAcLp+7brjureTfN5u+uNecnLmJH05Qr7aFZTyisjg4GDT1N+3b59Dtp1WXkZHR9Nzzz1HoaGhDv2NOgkKCiJEx/9n9D+zGKVR8s2WU7x4cVEFfhCYkSB30cJFtHzFcjFXcsPGDWTmDwczbVTpM2qmvaGPzLA5O+pshr2yTCZgBoEbN27Qq4NepZGRI6latWrZqqhfv764hrlmepJaOUryx8bE0oIFC6htu7ZuVfvP9/8hLC4sVrxYlvu4BvkRERGEnWOeeML7WJj3u1lE58yH/cn7RUVanZ1LFy+J8uXKlTNNYecvvFq17RGjDEYeGxtLYV3CDB8lk0HD2cNo2dkzZ+VLtjnK8ewu/2SsU4YVfG++8SaFvhgquCcmJtLsObNNdcgA3UwbVfKM5oS9oZ1G2pwddbbNA8aKMgEimjVrFmFf6n79+tmOx4iRIzw6ZNjOLi01jbp07eK2XZGjIgkhl/BmxFeyxEgZ/hjGrImhu/fukryMf/v27YQI6xgpw+iO0nTpcqZT9tRTTyktojrfvqTMkTIMVbrz9n0JxBdz1OIoKlioID2R5wmxQgMMPvrHR4QRG3TujBkzHPfv3r1Lw4YPo5YtWwrR2A5o0aJFhJ0LevbsSdgMHa/Fli5dShiGxef+/fuLHQ1kXZKSkuiXX36hBg0ayJccR736yIKaNm0qTuGYDQgfIF/2eES7Nm3cRPv376cHDx+I9jVu3NhtfqN0dCuciJ4unLkjw+VLl8WeqQgqqydhiH7B/AWEbY6GDh1Ku/fspiJFiugRqaqsXht1rUztM+rN3iDbqP70ZHNWfUbQdk86uzKXPyt9ToxiKtfLRyZgJAF8/6xYvoJat2lNpUuXNlJ0jsjy9p2wNm6t+NEdFhbmVhe8+StfvjzNmztPfDfDd/CULDFSdvv2bTp56iRhPsfVq1fFtjcXL1wUIy537tzxpLvb67/++qu4XujJQm7v672I98rOITcKFVJfD8oUL1GcThw/QTExMRQfHy+2+ZGHNfPlzydCM2A/xmPHjlH+AvmpUMH/1bNw4UJ6/PgxPVPxGeoS1oVWrlxJ4eHhFBwUTOPfGk8YZh00cJBwLuT2Yv4SkjxMK1/HUa8+sqwqVapQ4cKFhQMiX/N0xCuwNq3b0Lr4ddS3b18aMmQIHUo5REMihrgtYpSOboXDKft9myw4H5qWMf8uePcPu6nfK/1o4MCB1CSkCe1N2kuvj3s9Rx0yI2zUlZPaZ9SbvUG2Uf3pyeas+oyg7Z50dmWOz2qeE6OYutODrzEBvQS+/vprwvezneZhK21zXFycmAfvydmEQ1e9enUxSogfT16Tr5UAOXl/+/btYjXDG+Pe0FxtcFCwkLFi+QrNMrwV3Lhxo2PFxYjhI7xl9XkPS++bhjQV8rDCFCFAkHBEiImJb0/MtpoR4ROwmuP+/fvS2ri1omytmrWkc+fOOer78ssvxfV169Y5ro1/a7wIM+K44OZEiz6uYpo0biL5Wjl56tQpqWJwRal/v/7ZwpZ8+OGHQndPMozQ0VVnfN6yZYuoF6tpsCxabcKKRKw+rV6tuvSvf/1L9I9aGUblN9JGXXVS+owqsTfINqI/XW3O6s8I2u2qsytnfNb6nBjB1J0+fI0J6CGA70v8fY2OjvYp5u7du46/x1pWX8oVqJWjNj/qOX78uNB14ScL5WrdHt+b/J7IN2PGDLf35YuWGCmTvUaMlCDVrqN9RSNGOpCefPJJcTT6P1lHyNU7n6xo0aJi4vef/vQnQtBbjLBgZKx3r970QrMXxKoN1yFT/HLGtkD58+cnBLhEmvD2BDE0Kre1RIkS4nR74nb5El25ekWMYjkuuDnRoo+rGLQFdXlLY18bK34xYbKn6zDuSy+95K0oGaGjuwrkOWW4J9uQu3yerhUoUIAGDRpEYL/y65XitSX2nvRHMtJGXfWXZft6RpXYG2Qb0Z+uNmf1ZwTtdtXZlTM+a31OjGDqTh++xgT0EMC2b0hlSpfRI8ZyZfHqEgnztb0lLExCyjic4S0bWcopS0lJEcpqjZCPeST4h1SgoDmhFQ4dynQcUYfWxQhCwd//q1SpEn255EvhZMEha9miJWGRAl6/uDosKFKqZCl66623RGl5r8Z27do5ixTzynDhpys/Oa5fvXKV/vjHPzo+ezpRq4+rHPFl85Nnpwyrb6A3nBh322C5a7NrHXp1dJWHz85zELU4ZZCBSZ5YgTN8xHCa+f8zqf1L7cWradkm3dVrxjWjbdRZR6XPqFJ7g2y9/elqc1Z/RtBmV52dGeNc73Oil6mrPvyZCeglcOZMZriqEiUzBw30yrNKeSygq1evnlgN7k0nTC9Awvxvb8laTtmhFLFcVKuzgzlZ8sjSvbv6ltR6gub8hafVeXSV/cILL9AHH3wgLj98+FDEZ4PT4i4FBQeJlXtwHDBv59lnn80W60X2xIMqZHrmkCNzcSfT9ZoafVzL+qrrxx9/FEUw90yJA+ZOPq7p1dFVrvM8Mk/sXcu4+wwb7NChA63fsJ6mvj9VLL6Ao40Jrjm1r6YZNiq3NUXhM6rG3iBbb38612eHZwRtdtZZ5isfjXhO9DKVdeEjEzCCQL58mesK/fUGwYg2uMo4ceIEYVGXp1WXzvnl7zssavOWLOOUiX0HM44Ih0Trq0f8kZO/UOUJ/94ar/YeVo/IQ7CY0FeyZEm1Itzmh5FilV7BggXF/SVLllB0VLTbvPJFrG4DMwTac05YAIDVj0hY5SInLCyA/kqSFn1kubd+uUXefgnJKxD1jh7p0VHW1fnovKBE7gfn+0rO4VDLCbaIL0VMbo2KjqKErQnUNKSpWHX722+/ydkMP5plo1BUzTOqxt4gW09/erI5qz4jaK8nnWWDMOI50cNU1oOPTMAoAkWKZq4+x2riQEnyq8uuXbv6bJI8radisPfA9pZxyjA/Cl9qekef5CFCM5wyhKqQU506deRT3cfpf59OW7ZsoX9/+2/q3LmzkDd58mSvUY3lV5fNW2R1ypL2JokdEUKahmQJEluyREm6detWlhWZnhTXoo8s69bNW4S6PCWMguIXE3TR00d6dHSnm7NTpuVHAco3b9acXnvtNcfrY7ke2DSCkq5es5oyMjIopEkIzf5gtnhFJecx6miWjUI/Nc+oGnuDbD396cnmrPqMoL2edJbtwIjnRA9TWQ8+MgGjCGDVMdL5c+eNEul3OVh1Wb1GdfHGypcyWBWPhKkF3pJlnLKUg5nzyXxNIPbWGNyTJ2xjCb/RCa9u5NSkSRP5VNfx008/pU8++YSWLVsmlswuXLSQ4PBhxGvY0GGUnp7uVr48ib9GjRpZ7kMehklnzZyV5fVI+QrlxUgHYpt5S1r1gUyMfiEwYLnyngP3YhQKMdfQPndbMsn64Ve+p6RHR08y797J/PWG149YRKE2wZHDfLI6teuIEB+DBw8W8dec5eBhnDN3Dn373bfCIcVrzffee0+EgXHOp+fcDBuV9VHzjCq1N8jW05/ebM6Kzwja601nmbXe50QL0/Xr19OE8RPEjzpZDz4yAaMI4M0BUsYR7xPdkQffD74SviPwJmLjxo0esyqR41xYTX4EBsdUka5dfI+SoQ7ZKWvYsKFzldnP5WWY/j4i/AOWy2LJvZ7Urm07IWfqlKl6xLgtO3bsWCEbeiYmJrrNo+Ti4cOHpVWrVomQFyVLlBShIbAhqpwQ0qBUyVKiLuxIHxsTK2EDVznduXNHhLdwXV4csyZGXF+9erWc1XHcuXOnkBcfH++4Jp/o1UeWk56eLupYumSpfMntEZtlh74YKjVs0FC6dOmSI8/58+cltBftat6suQiXgPAGSEbp6KjM5QQhVFAvwovoTQhXgrAYaEPPHj2lbdu2ZQttgjp+/vlnae6cubpsyVVXo2zUVS4+q3lGvdkbZBnVn55szqrPCNruSWdX5mqfEz1MEdIlqEKQeAbM+Nvp2jb+nPsIXLt2TSpfrrzPkEkgc+jQIWGL+JuMc3cJf1dxH/9g++6SEjnO5dTknzdvnqg7LS3NWYTbc3yPIQxOSJMQ6dGjR27zyBctEdEfruLBlIPitZa74KbZXUnPV+TgbWaMlKUeShUVYzRFj57r49fTnDlzRNgOrPpLSEgQwWPlAKabN28WLDA/Dq/5Ro8eTdjLc8fOHaJ+TPDH/B5sZJ24LZGOZBwR88UwORjBaBG01DUhkj8m1/+4/0dyjTqsVx+5Lnn7ndBW3vfWxKjSps2baMr/TRGva1uFthIR/bEZ+KR3JtGoyFHiNd/zlZ+nocOGit0NjNJR1tX1KG+v5LpBrms+JZ8x0oaAuOgf9CVeI+EX2NjXx1LHjh0de59h3tC4N8YpEak4j1E26q5CNc+oN3uDbKP605PNWfUZQds96ezKXO1zglfGWv+u4G/NgAEDaPOWzcJGXXXhz0xAL4FixYqJIOFRUVFi6gZWILsmTO+YOXMmYRqOnBBqCKsbR44cSc47vuCtGN4KwXZd5wGrkYN61OZHGcwnw25DSnb1QXuwKAA79cB/8Jpk78yfR4wsVChfQerbt69uNca9Pk54r8OHDdcty1nAgwcPHKNTrUJbOd/K8XMEn8OvA4zuwOs+ePCglJaa5tMDR0DPBg0a+MyntUGv9H1F0a8gZ/no+wMHDkiph1JFINnbt29LP+z6QcrIyJCuXLkigXtOJAQsBtNBAwcZXh1+JWEEGMFlm73QTFqxYoUpwWXNtFEtz6jZ9oaO8mRzVn1GvOnszfCs8px405HvMQFfBPBGqFHDRtKc2XN8ZVV0/+LFi9K1q9cU5TUyE/6mp6SkSEoD244ZM0bq1KmThBFpXwkTv/2S8Hph/fr1EjoJX8rCyVihPwq/HBG+bZu2hrYLw6PQEf/wZePPFBYWJvRAxG81CW0oXap0llehasp7yytHNUZkfDumXj17CaYTJ040VX3YekREhFS/fn0pOipawisqo5LRNqr3GTXT3sDMm81Z8RnxpbNRdsBymICVCeAHd906dcWOFVbW0yjdMPUI02LgQCpJPsbRvA6y6bqJSeyvDnqVli1dRnv27BGRveWVh3oEV61aVRQ/cuSIosmCSutyjv3UuIn7TbOVytKTD+EUkvcli9hkeKWpJoHNkKFDCIsJvE2kVyNTzrt40WJq07YNuQayle9b/SiHOnFdOGG03ljEgUnYq1atotS0VLESc2vCVkOqMdpG9T6jZtobgHmyOas+I950NsQAWAgTsAEBrMKcv2A+RY6MpGvXrtlAY+0qYqHe+++/T18t/4rKlFG4k4ESz82MPB07dJTq1K4jJScni1c6cbFxhlRz+tRpx4gWzo1Kf3vvbw658h6VRslWKgev9TBKh9E6TCLHZHK84lOTsK8jJhx++umnaop5zbt161apSpUqtv3l8/DhQ6lsmbKCK+wxJxMWOmjZa9OdjkbbqBHPqBn2hrZ7sjmrPiPedHbXl3yNCQQ6AYyY9e7dO8tir0BqM6YVDRs6TMICBzUpDzJr9wO1l8Skt6jFUWLyW5++fahXr17ahTmVxITq5559jjDRH95p+/btne5qP+3Zo6cIylq8eHFKP5yeJdyEdqnqSmJk4Oy5s1SoYCG6/+A+YdcCjHzJ8V+USjt+/Dj16N5D8NEbFw7LfDt17EQffvQhtWzZUqkKlsqHMB4N6jcQEzBPnzlNcqw7SympQBmjbdSoZ9RIewMGbzZnxWfEl84KupazMIGAJIBRbSxo8ba7hV0bjl13sOjL58R+1waq8eDskheTfzGahPllRiRM6qv8XGUhM3xAuBEi/S4DExTfffdd3XogpENOjy7pVtpFAMKTwF5atmjpcsc+H61uo0bZG3okp2zOjjrbx2JZUybABNwRsExIDFdnUc/nFi1biGj4h9MP6xHjKIuApvImov6cT+ZQyIAT7A+Id916k9EhHfTqo6V8YmKiKNaqdSstxS1Rxuo2apS9AXZO2ZwddbaEMbISTIAJaCbgt4n+mjVWUFDeD/L77783ZLJ/Ssr/Ivk7x0lRoApnsQEB2Slr3fp/e4XaQO0sKrKNZsHBH5gAE2ACtiQQkE4ZVrghCOjly5cpOTlZd8fIX3gIUoegmJwChwDmk505fUbMI5O3AbFj69hG7dhrrDMTYAJMICuBgHTKMLGuW/duoqUbN3jeFysrCs+f5D3/QkJCskUO9lyK79iBgLw/IkbJ7DrBH5zZRu1gbawjE2ACTMA7gYB0ytDk7t27i5Zv2LDBOwEFd+VRiBdffFFBbs5iJwIJWxOEuuEDw+2kdjZd2UazIeELTIAJMAHbEQhYpwx7ZSF45bFjx+jo0aOaO+b6tet04cIFUb7li/YM+aC58QFe8ObNm4SR1HLlylGbNm1s21q2Udt2HSvOBJgAE8hCIGCdMsQ9+ctf/iIaGxsTm6XRaj5gE2YkbBZet25dNUU5r8UJrFmzhhBLpv+A/mIDeIur61E9tlGPaPgGE2ACTMBWBALWKUMv9OzVk0qXLk2ff/65+PLV0jMHD2Y6ZR07drT1F7eWtgdyGcRM/uKLL+ipp56iwYMH27qpbKO27j5WngkwASbgIBDQThlWS2K07MqVK/TNym8cjVZzkrgtM4YVHDxOgUNg8+bNlJaaRpGjIqlUqVK2bhjbqK27j5VnAkyACTgI+G2bJYcGJp/cv3+fWrZoSQ8ePqBdu3apWj2JLSCer/w8PV34acJmz9gygZP9CWCU7KV2L4ktq5KSkqhw4cK2bRTbqG27jhVnAkyACWQjENAjZWgtRsumT59OZ8+cpcWLF2cD4O0CgorCqev2527skHkDZbN7a+PW0oEDB2j8+PG2dsiAnW3UZsbH6jIBJsAEvBAIeKcMbW/bri2FhYXRvLnzHCspXZlgxGTMmDFieyb53pbNWyhv3rxi02/5Gh/tTeDWrVv017/+lUJDQykiIsJWjWEbtVV3sbJMgAkwAdUEAv71pUzk+vXr1LZNWwoODqY1MWuEsyXfw7HfK/2EQ1a7dm1C7Krz589Tk8ZNKDw8nGbOmumclc9tTGDC+AkUFxdH32/7nsqUKWOrlrCN2qq7WFkmwASYgGoCuWKkDFSKFStG0f+Mpj179tCC+QuygerQsQMVL16cpv19Gj169IgmT55M2K5p6vtTs+XlC/YkEB8fL1bizl8w33YOGYizjdrT7lhrJsAEmIBSArlmpEwGgjAIE9+eKEZLQpqGyJcJk79XrVpFu3buIokkCqoQRKPHjBZz0hyZ+MS2BDIyMqhzp840YcIEGjFyhC3bwTZqy25jpZkAE2ACignkOqcMZKIWR9Hs2bNp7bq1VK1aNcWwOKM9CWDT8bDOYSIe2bg3xtmzEaw1E2ACTIAJBDyBfAHfQjcNHD5iOD1+/Jh69+pNGzZuoKCgIDe5+FIgELhx44bo5z59+tDr414PhCZxG5gAE2ACTCBACeTKkTK5L48fPy4i/mMLJU6BSQAhTU6ePCn2QQ3MFnKrmAATYAJMIFAI5GqnLFA6kdvBBJgAE2ACTIAJ2PLKw2oAAAB9SURBVJ9Arll9af+u4hYwASbABJgAE2ACgUyAnbJA7l1uGxNgAkyACTABJmAbAuyU2aarWFEmwASYABNgAkwgkAmwUxbIvcttYwJMgAkwASbABGxDgJ0y23QVK8oEmAATYAJMgAkEMgF2ygK5d7ltTIAJMAEmwASYgG0I/BfKz9INvtfhPAAAAABJRU5ErkJggg==)
- 3) Apply Jensen’s inequality in the form (1.117) to the KL divergence (1.113) to give
where we have used the fact that
is a convex function (In fact,
is a strictly convex function, so the equality will hold if, and only if,
for all
), together with the normalization condition
.
- 4) Similarly, Let
be the support of
, and apply Jensen’s inequality in the form (1.115) to the discrete form KL divergence (2.110) to get [see Ref-1]
where the first inequality follows from Jensen’s. Since
is a strictly concave (i.e., the inverse of convex) function, we have equality in Equation (2.115) iff
for some
. We have equality in Equation (2.116) iff
, which implies
.
- 5) Hence
iff
for all
.
3.3 How to use KL divergence
Note that:
- we can interpret the KL divergence as a measure of the dissimilarity of the two distributions
and
.
- If we use a distribution that is different from the true one, then we must necessarily have a less efficient coding, and on average the additional information that must be transmitted is (at least) equal to the Kullback-Leibler divergence between the two distributions.
Problem description:
- Suppose that data is being generated from an unknown distribution
that we wish to model.
- We can try to approximate this distribution using some parametric distribution
, governed by a set of adjustable parameters
, for example a multivariate Gaussian.
- One way to determine
is to minimize the KL divergence between
and
with respect to
.
- We cannot do this directly because we don’t know
. Suppose, however, that we have observed a finite set of training points
, for
, drawn from
. Then the expectation with respect to
can be approximated by a finite sum over these points, using
(1.35), so that (???don’t know how to derive it)![技术分享](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAApEAAABkCAYAAADe62tIAAAgAElEQVR4Ae2dB5gUxfb2DxnRqyIgORgIkkxEiUuQJSclSFJEWZScEUQMwAXJUcL9e5WowBIEvAIi4gISRZSkILCwgASRsAhL6O95a78eemZnZrsnbc/uW8+zO93VVdVVv6rqPl116lQ6TdM0oSMBEiABEiABEiABEiABCwTSWwjLoCRAAiRAAiRAAiRAAiSgCFCIZEMgARIgARIgARIgARKwTIBCpGVkjEACJEACJEACJEACJEAhkm2ABEiABEiABEiABEjAMgEKkZaRMQIJkAAJkAAJkAAJkACFSLYBEiABEiABEiABEiABywQoRFpGxggkQAIkQAIkQAIkQAIUItkGSIAESIAESIAESIAELBOgEGkZGSOQAAmQAAmQAAmQAAlQiGQbIAESIAESIAESIAESsEyAQqRlZIxAAiRAAiRAAiRAAiRAIZJtgARIgARIgARIgARIwDIBCpGWkTECCZAACZAACZAACZAAhUi2ARIgARIgARIgARIgAcsEKERaRsYIJEACJEACJEACJEACFCLZBkiABEiABEiABEiABCwToBBpGRkjkAAJkAAJkAAJkAAJUIhkGyABEiABEiABEiABErBMIKPlGIxAAn4QOH36tEyfPl3u3L4jGTJmkKioKClYsKAjxTNnzsi0adMka5asommaZM6SWQYNGiTp0qVzhOEBCZAACZAACZBAyhPgSGTK10GaykH27NmlVkQtWbFihcyeNVs++vAjp/I//PDDUrFCRZk1a5b8+eefUqlSJQqQToR4QgIkQAIkQAL2IJBOw3APHQmEkEB8fLxUq1ZNcjySQ/bt2ydbt22VJ554wpGD69evS4XyFeSnvT9JpkyZHP48IAESIAESIAESsA8BjkTapy7STE62/7hdKlWsJP3695O7d+/KpImTnMq+Y8cOqVipIgVIJyo8IQESIAESIAF7EaAQaa/6SBO52bJli1SpUkXq1asnpUqVkiVLlsiJEyccZcf1qlWrOs55QAIkQAIkQAIkYD8CFCLtVyepPkcxMTFSpWoVpevYr18/uXPnjkyeNNlRbl3IdHjwgARIgARIgARIwHYEKETarkpSd4auXr0qFy5ekMKFC6uCNmzUUIoXLy6LFy+WuLg4gT7kmdNnpGjRoqkbBEtHAiRAAiRAAmFOgEJkmFdguGV/+/ZEfUjdZE/69Omlb7++cuvWLZk6Zars3LlTypcvzxXZ4VaxzC8JkAAJkECaI0AhMs1VecoWeEtMoj6kMRdNmzZVq7PnzZsn0dHRaqrbeJ3HJEACJEACJEAC9iNAIdJ+dZKqc6T0HatWcSpjhgwZpHef3pKQkCALFyzkohonOjwhARIgARIgAXsSoJ1Ie9ZLqswVbEI2a9pMjhw9IpjGNjpMZ8OweMLNBNn3yz5OZxvh8JgESIAESIAEbEjA+U1uwwwyS6mDQIvmLaRB/QZy+/ZtKflUSfntt9+cCgaj4kOGDFFmf3R9SacAPCEBEiABEiABErAVAY5E2qo6mBkSIAESIAESIAESCA8CHIkMj3piLkmABEiABEiABEjAVgQoRNqqOpgZEiABEiABEiABEggPAhQiw6OemEsSIAESIAESIAESsBUBCpG2qg5mhgRIgARIgARIgATCgwCFyPCoJ+aSBEiABEiABEiABGxFgEKkraqDmSEBEiABEiABEiCB8CCQMTyyyVzalcCrnV6VNWvWhDx7Xbp0kdH/Hh3y+/KGJEACJEACJEACiQRoJ5ItwS8CS5Yskbe6veVII2PGjPJ/n/6fFCxY0OFn5gBGyK9fv67+4q/Fy+Url+XUqVNy+NBh2blzp5w/f94pmfvvv1/tbPPggw86+fOEBEiABEiABEggNAQoRIaGc6q+S4/uPWTx4sWOMpavUF5Wrlwp2IUmEE7TNPn9999l7py5smjRIrlx44ZKduTIkfJm1zcDcQumQQIkQAIkQAIkYJEAhUiLwBg8KYH4+HipFVFL/vjjD8fF3n16y9ChQx3ngTr466+/ZNKkSTJzxkwpUqSIbN+xPck+3IG6F9MhARIgARIgARLwTIBCpGc2vGKBwC+//CKR9SIlISFBxcL+18uil0m1atUspGI+6MoVK6Vr164yb/48qVu3rvmIDEkCJEACJEACJBAQAml+dfaxY8dk8KDBfsNs26atXLt2zZHOTz/9JMOHD3ec+3owftx42bF9h6/RQxavTJky8t577znuhynoblHd5OLFiw6/QB40bdZURo4aKXNmz7Gc7Llz5wRT4XXr1JX6kfXln3/+sZxGaosQqH7gDxeMZPfo0cOfJBxxw6XfODJso4OUbAtm6u3ggYPSsGFDadK4ifznP/+RK1eu2Iges0ICaYtASEYijx8/Lrt27ZJMGTNJpsyZ1PTj7Vu35dbtW0pvrlGjRg7qN2/elFWrVknmTJklc5bMAmEEOnA1a9SUK1evONLJmCmjZMiQQe7cuSN6WqVKlpKnSj7lSCu5g0OHDsnLL70sixYvktKlSycX3Ov1woUKy7Yft0m+fPlUuK9WfSX//e9/1Wic14jJXPzr4l8SGRkpY8aOkYiIiGRCp+xl1FX7du1l3bp1jozUqVNHFi5aKBiZDLS7e/euNGvaTKZMnaKmts2kjxHTl1q+JJUqVZJRo0dJ/vz5zUTzK0zsiVj5duO3grq8cPGCtG3bVsqWLetXmoGMHMh+4E++tmzZIkPfGSqbvt/kNRl8mMydO1d279otaAP16tWT1zq/JljUpTs79hu7twOwS+m2YKXejh49KlFdowQfhavXrLa8mE9vK/xNOwQgX+A5AdnBnbt8+bJky5bNL31+q2mYDX/16lW12DRHjhzy6KOPJsk+ZgHxng3UWoQkN/DkoYXA7d69W+vdu7fWskVLLWeOnOqvTOkyWufOnbVRo0Y55eDatWtao4aNHOFKPlVS69ihoxYbG6u5S6d4seJa2zZttb59+mobN250SsvbyaVLl7Tnn3tem/f5PG/BTF8rVLCQFhcX5wi/auUqrUXzFo5zfw527dqllSheQvvjjz/8SSYkcS9euKiVLlXaUX+o75kzZobk3mZugrZVrGgxLT4+3kzwgITZuXOn1qljJ63ok0UVlzVr1gQk3UAkEuh+4E+eYmJitBrVa3hNYnn0cg197Zmnn9EWzF+gzZg+Q8uVM5d6Bty9e9cprt36jZ3bAcDZpS1YqbcjR46oPjVs2DCnuucJCbgSOHH8hJIlzp4963pJu3z5sjZ2zFjtsSKPabEnYpNcN+NhNQ2z4SETdX+7u5IBIFNA5mnQoIF2/Phxp2z98ssvWvt27bVz5845+Qf7BCN9IXVPPvGk6vR4+Htyp06d0vLkzqNNmzZNu3nzpttgejrTp093ez05z4EDBmqtWrXSXF88ycXzdD2YQiTu+c4772gvv/RywPLrqRyB8IcwgBe7/sGQN09ebe/evYFI2u808uXNp9WPrO93Or4kMOK9EbYTIgPdD3zhosdJToj87LPPVLtC3zc+6CF4oq0tXbpUT8rxa8d+Y8d2AGB2agtm6w3P70dzPaq92ulVR53zgARcCWz8dqNW78V62unTp50uXb16VZs4YaLjAx/PEeOzxSmwhxOraVgJD0GzWtVqWt06dbV//vlH5QDxX2r5khoM2bNnj1Oufv/9dy2iZoR2+PBhJ/9gnoRcJzJ9+sRb6r+uI6SnT5+Wdq+0k8/nfS5vv/22ZM6c2TWIOtfjZ0jvfljabaT/7wmdGkw1d+vWLSjTrN7u7eu1qKgo2bx5s/zvf//zNYmQxatSpYr07dfXcb9bt27JG13ecNIZdVxMgYMsWbKkwF1FqXKkyI093DSc+sGvv/6qdJehMgHVjoKF7tkhfeqpRBWWL774IklJA91vvv/+e3n7rbeT3MeKB1R67Obs1hbM1hum7zA9Sb1Iu7Uo++Rn//796l0/afIkyZs3r1PGYDoO76sNGzY4+Vs5sZqGlfBTJk+RgwcPypB3hkjWrFlVth544AGZMHGCXLp0Sfr3669UevT8Pvnkk/LRyI+UWhlUQ0LhQi5EeisUdIZat2ot73/wflBX3M6ZM0eeeOIJqVGjhrfs2OoajHc3aNhA2Uq0VcY8ZKZ///5SoWIFx1Uo6w8aOMhxnpIH+gdISubBDvcOl34AQ/QwaI+PkQIFCkiTJk2c8OkCxJEjR5z8cRLofoN2fPjw4ST3CXcPu7WFQNdbuNcP8+8bAQhanTp2kqhuUVKiRIkkiTz77LMCu8a58+ROcs2sh9U0zIbHgs9PPvlEZaNy5cpO2UH/wDqOffv2ydo1a52uvfDCC4J7dO7c2UnAdAoUwBPbCJGQzrHgYOzHY4Mq3EGxdsWKFdKwUcOwGYXU6xtCL0YjT8ae1L1M/6LceAHiRWzGIbw/DqMDs2bNkoceesiRzJdffin4o0t5AuHUD1Z/tVp9jYNap1c7OS2ggd+puFMK6PX4627B+tNv3CaYyjzt2hZYb6msoaVAccaOHSsnT55UskUK3N6vW2LhGPomRiD1UUhjgrpQ/N133xm91XG3t7oJFiouWLAgybVAe9xbzhjolC2kh+HmLq93kWnTp8nzzz9vIab1oFgljlVOzz33nNvIWO0N6R9b7rVs2VLKlSsnf//9t8ybN0/tmoLzV155JcmLzG1iyXjiK2nxosVy4OABVW6kC8Pd3/zvG2nTtk2S2FhRDAdBsl37dkmuu/OA2aF+ffvJ6tWrlQ3Hhx9+WDBV1LNXT6+ruPr07qOmDf/1r3+5S9aUH0aNJk+ZLNhfW3cDBwxUTB9//HHdK6S/mA61k/vtt9/UanZMPVz866L06dNHCd5ob78d/k1N22L18TPPPGM622balbd+8MMPP8jsWbMlS9Yskj5devXhgYfZ1GlTBSsDD+w/IKNHj3Zch/WEN958I2g2QefMvWfGqXHjxk4c8FH0x9FEI/ePPPKI0zX9xJd+o8cN1W+g2oEvzy+7tgWz9RYMyw+hqnfeJ3gE8J5ftHCRRNSKkNy5fR9pDF4OvacMqzZwWHUNKxSuM2h58uRR193NwDz99NPK8sjECROVvOJpNbr3HJi7muIjkXv27JGmTZoqW31FixY1l2s/QmEfZjgM97pzM2fOVBVWpHARadSwkRo5a9++vRQqWEgG9B8gK5avkI4dOirTQ+7im/XDMHT1atXVHtGY+oWZou5vd5eWLVoqU0Hu0ilWrJhAqMMuLWYcGl6rl1vJ7t27ZcCAAUqPAi9h7PjSoH4Dcdf4kC4ELdQL9qf218GeG4bVdQchGfqRulFy3T8Uv+BhdiQ2FPnBPTAVC6FswcIFsnDBQrVP+OudX5fSpUrLoMGDBKNr9V6sJ5s2eTd7o+fXbLvy1g/w1ZsjZw4lnC1fvlx9gGDvcv0hBvNaEHpg8B1tCDp+WbMk6uvo+QjUL8y36HZSCxUuJK4fHzDZpG+D+dhjj7m9rdV+4zaRIHsGqh348vyya1swW2/Qm0ffpiMBIwHoSON9U6Z0GaN32BznyplL5RVt27gbnF4AmECEw0irq8OHFXTFcQ2DAsF0KSpEbt26VbDvMl5acXFxMmzYsGCWVaV9Ou60WqyjS/HGG0J4il4WrUbqYO8RX/UffvChmpaFcWso8zdr3kzWr18va9asMUa1dLxt2zZp3KixtG7dWgYPHiyFCxeWLm90URWOlyJ0Gtw5NIxcuXJJ3Kk4d5eT+EVHR6uHa8yWGME2hB06dFCC5OYfNqtRyIiaEcrenvEBDAYTxk9Q+iO60JAkYYse0HE12u+EoPPhhx9aTMX/4BhRhsuZM6f/iQUoBYxsz5g5Q416I0n0gdlzZkut2rWUPh/aBepnxvQZyd7RSrvy1g/Kly+vPjSil0cLFLXhYmNjBfbM4PDShtDTpUsXZf8SBp+hVxQMt3HjRkey0JnOmyev5Mmdx/GHXZJ052m01mq/0dML5W8g2oGvzy+7tgWz9fZw9ocFU38oPx0J6AR27dylDv3Rd9TTSonf0mVKC2YO4fDB7ur09/YD/3rA9ZI6x0cY3I8//uj2eqA8U0yIhHQ8ZMgQWfzFYsGqKTgMPX/zzTeBKpvbdM5fOK9G89xdxOhbqdKllIAFo7twAwcNdDJIrQsgMT/EuEsiWT9MC/bs0VPuu+8+pxXMiKjrPXgSIhEme/bsgjKYcd9v+t5pVZceByM2q75aJf369ZPh7w6XalWryfsj3pdRo0apEcpp06bJu8Pf1YP7/YtyQXFfLx8S/GTmJ36tiLOaKbxgxo0bp4zMdujYwWr0oIeHIX64Fi1aOAm5+JjBVIQ+teEpI1bblbd+oN8DDzAYikebw57l2JUJI48w0F/5hcpqFSBe9MF0mNLX3VtvvaWm1DGtrv8ZRyarVK2iB03ya6XfJIkcQg9/2oGvzy87twUz9fb666/LmTNnBBs80JGATuDEiRPqEB+d4egwEwgLNXDjx4+Xbzd8q44xuIXjLxYnWqMoWOCepQpjObH4Bu7woeAuBEwxncjYk7GybNkyNbKGwr7S7hU1nQddPIycedJvMkJK7hiCw6bvNimdCD3shfMX5MEHH9RPnX4fzfWoYGoZDkqpcNhxxeiwAAju3PlzRm/TxxhRgkDQqVMnZRlfj4jpXUwrYcpOr3z9mvEXD1W9cxj9PR1XqHBvhbQxDBa+YHSyTt06MvKjkUoPFCOPCL9u/Tq1et0Y3t/j4sWLKz066PzpDh0EJlPcjQrrYfz9PXv2rHz66aeyYf0GNbI8f8F8qVq1qsdk0UHHjBmjdpfxGMjNBQh6r3d5XfSvPzdBTHmBk9EhXfxhWsabs9quvPUD433wwfHZ558pNQsIkPjgQBvBtCnyFWxnnMZ5s+ubTh900Pft3au3ygJ0nipWrOgxO1b6Db76PU0BHTp8SE6eOqlMa7i7GRaSQQ3Bk2kyd3Hc+fnSDnx9ftm5LZipN3xcQM1nxIgRsnLVSmWyBSpI/taBu3qhX/gQwOwJXM5c9pl5skqvV+9eEn89XiZNnCRt2rRRO9Vcv35d7dIFGQkmEatVq+Y2WX3QRp+BcxsoAJ4pJkTCFiSmZnWH6U0IfIAyaNAgNXKlX/P199Jfl2TE+yOchEhvIye67TmM6kCgwyiHq10pXar3JP17yyuEWgg0cPUi6zkFxSgC7uttFFKP4K0Mehj8YlFLclPSMBOAbR9hRgXD48F88GIxELa004fmsZgE+m5NmjqbbDGWwd9jCMsQMNDhMFoBITE5lyVzFrVoJLlwxuu6sGf08+XYkx4q6seT86VdmW1DuCfMS3z88cfSu3dv1U4wxR3MdmIs5x/HEhfNYEGPvqWofn3jtxsdurVYBJecUGu2zND5xKIidw4jhekkncfriGf2Pu7S1/18aQe+Pr+s5Dcl2oKZ/GERBXRjbyWYsz6hc+Zv6iWAZz8cno/h6tD2hw4dKvgo+mnPT5Lt/mxSpkyijme558upBb6NGt/bNtpYTv15iO2lg+lSTIh0LRRGBzGtjYUgWLyCRS3QQ/THQU8mX97Evaz1dLBgQB9N1P1cf7FaESODMEJqdBCysDIaDiu+rDoIyBBk4FxHCLdu2ar8Xe/peg/sH272yyo5AdKYtt7hjH6BPkaHmDBhguzcsVN9LHSN6hpUARL5h/oBFvbgb8R7I6RD+w5qP3NPX2/oeP369wt00YOani/tykw/0DONhzD0l2GkHR86n3/+uWCkDCODwXY3bySamipSpEgS4UzXS4bA1aNnD69ZsdJvsBgMf+4cNimAwDJy5Eh3l23hZ/X5Zee2YKbeMEoDVRzo5gbzg9QWlctMmCbw0MMPqfeMvvDOdEQbBsS6CfzpDjN6kFHavtJW2c7V/Y2/utpb4UL34hmvB+o4xXQi3RUgIiJCOnbsqC5hNTFWZvrjvv766ySjF1jxhEUB3r5O9KlsVx0rKOr++eefUrFSRZ9sWeLlA4eRMaP9RPg57vmCs+DqWv4rl6+IvmrL9Vo4nP/888+qXmG25v333w9plqE7hXqf9/m8kN432DfzpV2Z6Qd6vkeNHKXMEK3fsF4aNGigvN99912POq0XL16UqVOnKr0d8L5w4YIyGQS1CatK3oWLJD4A8+V3/hiECa61axON7Pbo0cNJj1TPt/E33PuNsSzJHTueJS46op6eX8FqC4FoB2bqDbM7sOzRuImz+afkOPF66iagqxaZXYgaLjTwkYj1I5glhQqHJwdVLjhPVis8xbPqbyshEpnH9HP+/PnVlj59+/T1Kux5KyweYIsXL04iROYvkF9J8HgJeXL6opmSJUs6BcGXLkaqxo4Zm2RUxCmghxNUJkYHXaflsOp0+/btShdSn5JylwSmYrFk3/WF6i6s7odpUF/+9JVfejqB+MUI8GuvvqZWfs+aPSvZ6cdA3NOYRoGCBdQ0LEY3UpPzpV2Z6QdghDY/Y8YMmT9/vjIZMfOTmVK2bFml+gBTTdiSy9X16tlLmSjC5gHDhw+XqK5RAmEQqhpNGjdxfDC5xnN3XqF8ok6vPiKph4EQixGG6tWrK5unur+7X1/6jbt0wsXP6vMrWG3B33Zgtt7wrM/+SHZLz2SMYsNmLQYF6FInAX2Xl8O/Jb+wxMz7Dh/EMBuEwSl3zkwaxnhWwyMunrcwMQgdYOiqe1s7oguRwba9HTIhEhWAkUVs5QOHArobZgYc/YsSK7UHDxrsNHLoms7fl/92CJq4hlFGGNZu2KChwLadq8CmTyPDzIw7hzzBriKc/kWPY0yxI12YYylZylm4dJeOOz9MB7788stKENTtFWLVK/S5zOhDQgjDAouKFTwvIDDeF7oUMIniyx/sFOp5NKbp6zEe9Fjdi1XpsInoSefL1/TNxNN1q2CTM9QObfPvS4kmhqCri3Pd4fjS35fUKcIYr2HxCOoB7dKTbU1f2pW3foBt/bDobcjgIeqvZs2aymoBMpgtWzYZMHCA+hhC3qDbDB3X7zYm7pqAxTdZ78uq+gjKgd1m8MGAkefnyz2vyrZt6za96Mn+YsocD0qMYEIVBB9EGNFEujBGPfc/c70azccNrPabZDPlRwAwCVY7QLZ8eX4Foy0Eoh1YqbfMmRJt5pmpGvSjblHdlH46jOrTpU4CzZs3Vyo4sMObnDMu4PM0yBATE6NsOUOIc7f1qZk0jPmwEh4fVNiYpH5kfbXhw1erv/Jo6xr3wHNm/6/7E7d3rhnc7Z1DohMJg8Vvv5W4VB0K+ZjKnTt3rhrhgJL+lq2JK6FR+Nq1asuvv/6qzMHg5bho0SLV2Xv16qVeTEjHKNyM+3icYJNy6PThBeP6os2T13l5P3aqgaC696e9bvWesKAGaUDY+2HzD2rXEEwX7t27V1AO437QxgZh9viDDz9QL0O8fIsWK6oMTZcpW0at2E5OHxKKtXA1TDYK2ByELT9fHDZ5z5Qp0eyML/GNcSAgQ00BHxGr16xOItgbw4bi2CikheJ+sNeJvZ/BE7q/AwcOVFYAfoj5QemHQr8F19AvMMWP0btv1n2jVgDDbijqAnl+/LHH1c5GWIHq6qy2K2/9YM3qNcqkBAR+CI2w1YgPMuQDDh936G/oy/ho6969uxQqVEj1Y8RB+bBQDA5mpLAoBg59CE7frkudJPMPvGCOquubXQWjkhjJh9/w94ZLt27dTO0cZbXfJJMlny+Hoh348vwKRlv4csmXfrcDK/VmRf8b7bZdu3aqj0VG3rMz6nPFMqItCeDjEypMs2fPVjObWOnv6iAMwhqHblMS1/Gugs1Z7OwGm7m6w8AHZiLRfiCb6M5KGohjJTxkoXXfrFMf9YiLgaFXX3s12XczygMhFTuLWekbepks/WohcLdv39bw5+ru3r2rJSQkOHnjHP5Gh3N38Y1hrBwP6D9Ae+6557Q7d+4kiTZ69GgtZ46c2qKFi9T1ffv2aQf2H3Ab1hi5UMFCWlxcnMNr1cpVWovmLRznxgOU5+jRo9qRI0dUWVu2aKnuefz4cWOwJMdtWrfxmGaSwDbxQFm7vtlVy5Uzl7Z27doUz1W+vPlCzhBt17Wt6X76rxEMwuLv1q1bRm/VVlz9jAGstitv/cCYri/Hw98druXJnUe7du2aI/qI90ZouR/NrZ0/f97hZzyIiYnRalSvYfRyOka82NjYJCydArk5CWS/+fTTT7XatWq7uUvyXu7qWvfTf42p+NIOfH1+Bast+NIOdAZm6y0l+rSeR/7am8DVq1e1cs+X08aPGx+QjJ45c0a7eOFiQNIyk0h0dLS2bNky7cCBA0nkIm/xe/ToodWvX1+7efOmt2ABuYYRjjTnDh06pF5mG7/dmKTsDRs2NCXQuUa0IkQa46KSC+QvoD1d9mmvjQRCJ4TbdevWGaPb/njMmDEq3zOmz7BFXtPKC8dMu/LWD/ytrDq162iR9SIdyUDALVumrNaqVSvVzgcPHpykvScnRDoSs3AQ6H6zefNmrWfPnhZyENqgvj6/gtUWfGkHIGal3vLmyRvyD8PQ1irv5g+Bw4cPq/drcoM0/tzDTnEh15QuVVqDwBsKFzKdSEvDo0EODPMkMAyNRQLGqU0Y8dyze49a9YTpuVA4TNlAjwmLDnSdPXf3nfXJLLUVnqvxc3dh7eK3dOlS+Xjsx8qwelS3KFtkC1OhNxMSzcbYIkNByoSZduWpH/ibJUxxQ+fYaPMUBvJhiujFF19Uq7rv3L7jtb37mwc9fqD7DUxDTZ48WU/eVr/+PL+C0Rb8aQdm6w2LE4zqTbaqEGbGFgSwShvmA6EHC9381Oyw8AY2t7HuIJibeBgZpkkhEgBg0PzE8RMO49/bf9yu7AjigQRB48svv1SmSYywAnmMVdbz581XOgtIF3qXWLzjqtOJa5s2bZIVK1fI2LG+rQoPZL7NpgWeWJ2JRRmj/z06aAIDFj/pq9DM5A22to4eOeqWs5n4dg9jpV2hLK79IBDlO3bsmNLDMRrBxQI3rKRetXKVzKPGHxYAAAdASURBVJk9R4YOGxqIW3lNIxz7jdcCebkYiOdXoNuCr+3ASr3t379fUfG0b7oXZLyUhgjgPaQEyW7dUu2KfOjPT5wwUZYuW6osaISqetNhuDNUN7PbfWCMvEXzFkpq3xKzRW1nljVLVkm4lSAwKWJlGzsY9Nz24zbHohHs4wrDxMuil7ktNva+3LBhg1LQxSKFGzdvqN1UsBjGuHIZAhJWZE2ZOsXj9kZub5CCnnh5RNaLVDsSrf16rcdtJv3NYlxcnETUjJBdu3eZvgeETqwSr127tvx7zL+VzU5/82Gn+GbblTHPxn6AHYwC4S5fvpzEFirSxUgAFN7djbqjboa+M1TtauRvHsKx3/hTZozcYTtGX59f+r0D3RastgOz9YYRSIx29+jeQ9KlT+fYQlcvB39JwB0BjNZj8Z+754+78OHkhwWsWKAZ9IU0rlBCMWdu53vEnojVhg0b5ncWO3bo6LSIYO/evdoH73/gd7oTxk/Q9uzZ43c6oUrg0qVLWqWKlbQSxUtoYBsshwVYULyP6hpl+RZnz57Vxn08TqsfWV/Doibj4g/LiaWSCIHqB/7gOHbsmNandx9/knDEDbd+48i4DQ5Ssi2YqbeDBw+qBVjNmjbT5syZo8XHx9uAGrNAAmmTQJoeiXQVqHnuHwFMxbdu1Vp27NghMENTrlw5/xL0EBu2RqHfAoPBGLqvUSO4drA8ZIPeJEACJEACJJCmCaRZncg0XetBKDy0Ivr37y8wyIodToIhQOIe69evl6pVqyoBskCBAmEzxR8E5EySBEiABEiABFKUQEiMjadoCXnzkBCYMmWK2s8TxlCbNmsasHtihSeU56GzumTJEmVAVU+8dZvWodf/0G/OXxIgARIgARJI4wQ4nZ3GG0Agio9FRJ07d1YKy81bNPdJaRmK8thG78rlKwJlfP0P20J6cjt37ZQiRYp4ukx/EiABEiABEiCBIBKgEBlEuGkhaWxv16RxE7X3dyjLW7lyZbUdXijvyXuRAAmQAAmQAAncI8Dp7HsseOQDgZOxJ9X+pD5E9StK3Rfr+hWfkUmABEiABEiABPwjwJFI//gxNgmQAAmQAAmQAAmkSQJcnZ0mq52FJgESIAESIAESIAH/CFCI9I8fY5MACZAACZAACZBAmiRAITJNVjsLTQIkQAIkQAIkQAL+EaAQ6R8/xiYBEiABEiABEiCBNEmAQmSarHYWmgRIgARIgARIgAT8I0Ah0j9+jB1iAvHx8fLRhx/Jzp07Q3xn3o4ESIAESIAESMBIgHYijTR4bFsCa9askfHjxsuNGzfk999/lwIFC0j58uVtm19mjARIgARIgARSOwEKkam9hlNJ+SIiIqR69epy4cIFqVC+QiopFYtBAiRAAiRAAuFLgEJk+NadbXJ+8eJFWbhwoZR8qqTUql1LcB69LFrOnz8vtevUlkqVKvmd12zZsqk0IETSkQAJkAAJkAAJpDwBCpEpXwdhn4NePXuprQ9bt24tXaO6ysEDB+WNN9+QokWLqn21l69YLlWqVHGUc8uWLXLu3DnHubeDYsWKSalSpbwF4TUSIAESIAESIIEUIEAhMgWgp6ZbHjlyRLLel1VKliopmqbJ6q9Wy4ZvN0iOHDnkypUrym/b1m0OIRJhli1dJmfPnjWFofILlSlEmiLFQCRAAiRAAiQQWgIUIkPLO9Xd7b777pOBAwfKnj17VNn69eunBEic7N27V/mVKFHCUe506dLJhIkTHOc8IAESIAESIAESCE8CFCLDs95sk+v8+fOrvCyYv0AyZswozVs0d+Ttu43fSYYMGaRSZf91Ih2J8oAESIAESIAESMAWBChE2qIawj8TW7dulWeeeUbuv/9+VRhMW0dHR0uNmjXUyOSQIUNk1KhR6lrzZs0lNjbWVKHrRdaT0aNHmwrLQCRAAiRAAiRAAqEjQCEydKxT7Z2g+7hv3z7p3r27o4wnTpyQ06dPS89ePWXDhg1y5/YdwVQ23IyZM+Ty5cuOsN4OcufO7XT55s2b6jzhZoKTP09IgARIgARIgARCSyCdhiEjOhLwg8DPP/8skfUiZe3Xa+XZZ59VKSUkJEjbNm3l9u3bkiVLFpkzd4489NBDPt9l8+bNMmTwECV8Xrt2TaX5yCOPSKtWraRP3z4+p8uIJEACJEACJEACvhGgEOkbN8ZyIYCRRXdCImxGQtjTRyFdopk+1b91jOnAD3/p03P3TtMgGZAESIAESIAEAkSAQmSAQDIZEiABEiABEiABEkhLBDiEk5Zqm2UlARIgARIgARIggQARoBAZIJBMhgRIgARIgARIgATSEgEKkWmptllWEiABEiABEiABEggQAQqRAQLJZEiABEiABEiABEggLRGgEJmWaptlJQESIAESIAESIIEAEaAQGSCQTIYESIAESIAESIAE0hIBCpFpqbZZVhIgARIgARIgARIIEAEKkQECyWRIgARIgARIgARIIC0R+H+7iDnjiQNL7AAAAABJRU5ErkJggg==)
- the first term is the negative log likelihood function for
under the distribution
evaluated using the training set.
- Thus we see that minimizing this KL divergence is equivalent to maximizing the likelihood function.
Reference
[1]: Section 2.8.2, Page 57, Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. The MIT Press.