Elements Of Information Theory Essay Example

Elements Of Information Theory Paper At first the homework problems and exam problems were generated each week. After a few years of this double duty, the homework problems were rolled forward from previous years and only the exam problems were fresh. So each year, the midterm and final exam problems became candidates for addition to the body to homework problems that you see in the text. The exam problems are necessarily brief, with a point, and reasonable free from time consuming calculation, so the problems in the text for the most part share these properties. The solutions to the problems were generated by the teaching assistants and readers for the weekly homework assignments and handed back with the graded homework in the class immediately following the date the assignment was due. Homework ever optional and did not enter into the course grade. Nonetheless most students did the homework. A list of the many students who contributed to the solutions is given in the book acknowledgment. In particular, eve would like to thank Laura Cheroot, Will Equity, Don Kimberly, Mitchell Trot, Andrew Nobel, Jim Ruche, Vitriol Castillo, Mitchell Slick, Chine-Went These Michael Morel, Marc Goldberg George Smells, Nadia Hazardous, Young-Han Kim, Charles Mathis, Stormier Crisscrossing, Jon Yard, Michael Beer, Mug Aching, Squash Diagram, Else Riskier, Paul gain, Guard lounger, David Julian, Hyannis Assassinations, Amos Lapidated, Erik Orthodontic, Sandmen Pomona. Ark Stunting. Josh Sweetened-Singer and Safe Kiev. We would like to thank Proof. John Gill and Proof. We will write a custom essay sample on Elements Of Information Theory specifically for you for only $16.38 $13.9/page Order now We will write a custom essay sample on Elements Of Information Theory specifically for you FOR ONLY $16.38 $13.9/page Hire Writer We will write a custom essay sample on Elements Of Information Theory specifically for you FOR ONLY $16.38 $13.9/page Hire Writer Baas El Gamma for many interesting problems and solutions. The solutions therefore show a wide range of personalities and styles, although some of them have been smoothed out over the years by the authors, The best way to look at the solutions is that they offer more than you need to solve the problems. And the solutions in some cases may be awkward or inefficient. We view that as a plus. An instructor can see the extent tooth problem by examining the solution but can still improve his or her own version, The solution manual comes to some 400 pages. We are making electronic copies available to course instructors in PDP We hope that all the solutions are not put up on an insecure website-?it will not be useful to use the problems in the book for homework and exams if the solutions can be obtained immediately with a quick Google search. Instead, we will put up a small selected subset of problem solutions on our Bessie, http://www_elementsofinformationtheory. Com, available to all These will be problems that have particularly elegant or long solutions that would not be suitable homework or exam problems. We have also seen some people trying to sell the solutions manual on Amazon or ABA;y. Please note that the Solutions Manual for Elements Of Information Theory is copyrighted and any sale or distribution without the permission of the authors is not permitted. We would appreciate any comments, suggestions and corrections to this solutions manual. Tom Cover Duran 121, Information Systems Lab Stanford University Stanford, CA 94305. Ph. 50-723-4505 FAX: 650-723-8473 Email: [emailprotected] Due Joy Thomas Stratify 701 N Shoreline Avenue Mountain View, CA 94043. Ph. 650-210-2722 FAX: 650-988-2159 Email: [emailprotected] Org Chapter I Introduction Chapter 2 Entropy, Relative Entropy and Mutual Information 1. Coin flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips required, (a) Find the entropy H(X) in bits. The following expressions may be useful: nor= (b) A random variable X is drawn according to this distribution. Find an efficient sequence of yes-no questions of the form, Is X contained in the set S Compare H(X) to the expected number of questions required to determine X Solution: (a) The number X of tosses till the first head appears has the geometric distribution with parameter p = 1/2 , where p (X = n) = PC , n E (l, 2, . Hence the entropy of X is PC n-l log(PC n-?l ) PC n log p + -p log p PC log q -p log p -q logo = H(pop bits. If p = 1/2 , then H(X) NP n logo = 2 bits. Entropy, Relative Entropy and Mutual Information (b) Intuitively, it seems clear that the best questions are those that have equally likely chances of receiving a yes or a no answer. Consequently, one possible uses is that the most efficient series of questions is: Is X= 1 ? If not, is X = 2 ? If not, is X ? With a resulting expected number of questions equal to n-l n(1/2 ) 2 This should reinforce the intuition that H(X) sis measure of the uncertainty of X Indeed in this case, the entropy is exactly the same as the average number of questions needed to define X , and in general E(# of questions) NIX) This problem has an interpretation as a source coding problem. Let 0 = no, 1 = yes, X = Source, and Y = Encoded Source. Then the set of questions in the above procedure can be written as a collection of (X, Y ) pairs: 1, 1) , (2, 01), (3, 001) etc. In tact, this intuitively derived code is the optimal (Huffman) code minimizing the expected number of questions. 2 Entropy of functions. Let X be a random variable taking on a finite number of values. What is the (general) inequality relationship Of H(X) and ) if (a) Y = XX ? (b) Y = coos X? Solution: Let y = g(x) . Then p(x). X: y-g(x) Consider any set of x s that map onto a single y .