Source and channel coding : an algorithmic approach / by John B. Anderson, Seshadri Mohan. p. cm. -- (The K1uwer international series in engineering and.
SOURCE AND CHANNEL CODING An Algorithmic Approach
THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE COMMUNICATIONS AND INFORMATION THEORY
Consulting Editor: Robert Gallager Other books in the series:
Digital Communication. Edward A. Lee, David G. Messerschmitl ISBN: 0-89838-274-2 An Introduction to CryptoloiJ)'. Henk c.A. van Tilborg ISBN: 0-89838-271-8 Finite Fields for Computer Scientists and Engineers. Robert J. McEliece ISBN: 0-89838-191-6 An Introduction to Error Correcting Codes With Applications. Scott A. Vanstone and Paul C. van Oorschot ISBN: 0-7923-9017-2 Source Coding Theory. Robert M. Gray ISBN: 0-7923-9048-2 Switching and Traffic Theory for Integrated Broadband Networks. Joscph Y. Hui ISBN: 0-7923-906I-X Advances in Speech Coding, Bishnu Atal, Vladim ir Cuperman and A lien Gersho ISBN: 0-7923-9091-1
SOURCE AND CHANNEL CODING An Algorithmic Approach
by
J ohn B. Anderson Rensselaer Polytechnic Institute
Seshadri Mohan Bel! Communications Research
Springer Science+Business Media, LLC
Library of Congress Cataloging-in-Publication Data Anderson, John B., 1945Source and channel coding : an algorithmic approach / by John B. Anderson, Seshadri Mohan. p. cm. -- (The K1uwer international series in engineering and computer science ; SECS 150. Communications and information theory) Includes bibliographical references and index. ISBN 978-1-4613-6787-1 ISBN 978-1-4615-3998-8 (eBook)
DOI 10.1007/978-1-4615-3998-8 1. Coding theory. 1. Mohan, Seshadri. II. Title. III. Series: K1uwer international series in engineering and computer science ; SECS 150. IV. Series: K1uwer international series in engineering and computer science. Communications and information theory. QA268.A5 1991
003' .54--dc20
91-4966 CIP
Copyright © 1991 by Springer Science+Business Media New York Originally published by K1uwer Academic Publishers in 1991 Softcover reprint ofthe hardcover lst edition 1991 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system ar transmitted in any form ar by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC.
Printed on acid-free paper.
Contents
Preface
1
Introduction 1.1 The Historical Context 1.2 Information and Coding 1.3 Bandwidth and Energy References
2
Source Coding Algorithms 2.1 Source Coding Preliminaries 2.2 Applications of Noiseless Source Coding 2.3 Rate-Distortion Theory 2.4 Predictive Source Coders 2.5 Some Other Rate-Distortion Approaches Problems References
.
IX
1 1 5 11 16
19 20 27 35 50 63 71 73
vi
3
Contents
77
Error Control Coding
3.1 FUndamentals of Coding 78 3.2 Linear Block Codes 89 3.3 Syndrome Decoding 96 3.4 Bounds on Performance 107 3.5 Cyclic Codes 113 3.6 Decoding Cyclic Codes 128 3.7 Algebra of Finite Fields 139 3.8 The BCR Codes 146 3.9 Decoding BCR Codes 152 3.10 Reed-Solomon Codes: Algebra in Non-Binary Fields 165 3.11 Codes for Burst Error Correction 175 3.12 Automatic Repeat Request Protocols 181 Problems 188 References 195
4
Trellis Codes 4.1 Convolutional Codes 4.2 Trees, Trellises and the Viterbi Algorithm 4.3 Continuous Phase Modulation Codes 4.4 Set-Partition Coding 4.5 Intersymbol Interference Decoding Problems References
5
Red need Trellises 5.1 Truncation of the Impulse Response 5.2 Partition of the Path Map Alphabet 5.3 Pulse Response Simplification Problems References
199
'"
200 209 225 241 253 268 270
275 276 282 285 287 288
vii
Contents
6
Sequential Coding Algorithms 6.1 Basics of Sequential Coding 6.2 Important Channel Decoding Algorithms 6.3 Calculation of the Search Region BOlUldaries .. , 6.4 Number of Paths Searched 6.5 Other Analyses of Sequential Decoding 6.6 Algorithm Steps 6.7 Source Coding Revisited Problems References
7
289 290 297 315 327 336 345 '" . 349 359 361
Architectures for Coding Algorithms 365 7.1 Parallel Architectures and Algorithms 7.2 Architectures for the M-Algorithm 7.3 Architectures for the Viterbi Algorithm Problems References
366 375 381 390 390
APPENDICES
A
Signal Space Theory
393
B
Introduction to Algebra
403
C
Mathematical Details of the Stop and Drop Lines
415
Index
419
Preface
oW should coded communication be approached? Is it about prob-
H ability theorems and bounds, or about algorithms and structures? The traditional course in information theory and coding teaches these together in one course in which the Shannon theory, a probabilistic theory of information, dominates. The theory's predictions and bounds to performance are valuable to the coding engineer, but coding today is mostly about structures and algorithms and their size, speed and error performance. While coding has a theoretical basis, it has a practical side as well, an engineering side in which costs and benefits matter. It is safe to say that most of the recent advances in information theory and coding are in the engineering of coding. These thoughts motivate the present text book: A coded communication book based on methods and algorithms, with information theory in a necessary but supporting role. There has been much recent progress in coding, both in the theory and the practice, and these pages report many new advances. Chapter 2 covers traditional source coding, but also the coding ofreal one-dimensional sources like speech and new techniques like vector quantization. Chapter 4 is a unified treatment of trellis codes, beginning with binary convolutional codes and passing to the new trellis modulation codes. There is much confusion today about what terms like modulation code and trellis mean and about what is old and what is new, and we especially hope to
x
Preface
shed some light on such matters. Chapters 5 and 6 discuss the important practical topic of how much of a code book a decoder needs to search in order to find a good decoding; Chapter 7 is a brief look at VLSI structures for this job. Most of this material has not previously appeared in a text book. Our aim has been to produce a first-year graduate coding text book of moderate size, and this has meant explaining a limited number of topics fully rather than many topics briefly. Many important but smaller topics are left out of this book, and we mean no slight to the talented researchers who work on them. Particularly, we emphasize channel coding over source coding, and trellis and tree coding over block coding. Our rationale is that this bias reflects advances in recent years and advances to come. About the references cited in the text, we follow a priority that has meant injustice to some. Our first aim has been to refer to a source of further reading, and so readability and information often take precedence in our citations. Establishing the discoverer of an idea has taken second priority and listing recent contributions a third. A one-term introduction to coding and information theory for senior undergraduate or first year graduate students can be based on Chapters 1-4, with topics from later chapters added to suit the instructor. This yields a course that stresses channel coding over source coding in a 2:1 ratio. While the text does not cover information theory in detail, all the basics are present, including information measures, mutual information and capacity, and the basic source and channel coding theorems. Particular attention has been paid to the problems, exercises and examples in Chapters 2-4. The prerequisites for these chapters are good courses in probability theory, conunumcations systems, and linear systems. The book depends to some degree on signal space theory and on modern algebra, but we review the needed material in Appendices A and B. Although no further background is required, a strong mathematical aptitude is always a plus for those who would learn coding. The ideas behind this book, both pedagogical and technical, evolved over many years and we owe a great debt to our students and colleagues over that time. It is a particular pleasure to acknowledge the encouragement and support of Prof. Robert Gallager, the consulting editor of this series, of our publishers at Kluwer, Robert Holland and Carl Harris, and of our editorial assistant, Rose Luongo. Our institutions, Rensselaer Polytechnic Institute and Bellcore, Inc., provided time and technical support. We owe a particular debt to Virginia Palmer, who typeset most of the initial pages. Her cheerful persistence with the job was appreciated by both of us. Several colleagues read and made suggestions on parts of the text; these include Sandeep Chennakeshu, William Pearlman, Stan-
xi
Preface
ley Simmons, Ivonete Markman, Amir Said, Kumar Balachandran and Frank Gozzo. Finally, we recognize the enduring support of our families, who felt as we do, that the writing of a book like this one is a worthwhile effort.
John B. Anderson Troy, New York
Seshadri Mohan Red Bank, New Jersey
SOURCE AND CHANNEL CODING An Algorithmic Approach
