(Ebook) Hacking Secret Ciphers with Python: A beginner's guide to cryptography and computer programming with Python by Al Sweigart ISBN 9781482614374, 1482614375
(Ebook) The Mathematics of Secrets: Cryptography from Caesar Ciphers to Digital Encryption by Joshua Holden ISBN 9780691141756, 9780691183312, 9780691184555, 9781400885626, 0691141754
(Ebook) Understanding Cryptography: From Established Symmetric and Asymmetric Ciphers to Post-Quantum Algorithms 2nd Edition by Christof Paar • Jan Pelzl • Tim Güneysu ISBN 9783662690079, 3662690071
For more information on this and other Manning titles go to www.manning.com
Copyright
For online information and ordering of these and other Manning books, please visit www.manning.com. The publisher offers discounts on these books when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co. 20 Baldwin Road PO Box 761
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.
♾ Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to
that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.
Manning Publications Co.
20 Baldwin Road Technical
PO Box 761
Shelter Island, NY 11964
Development editor: Marina Michaels
Review editor: Aleksandar Dragosavljević
Production editor: Keri Hales
Copy editor: Tiffany Taylor
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor
ISBN: 9781633439795
contents
frontmatter foreword preface acknowledgments about this book about the author about the cover illustration
1Introduction
2Whatiscryptography?
2.1 Unbreakable ciphers
2.2 Types of cryptography
2.3 Symmetric vs. asymmetric cryptography
2.4 Block ciphers vs. stream ciphers
2.5 Mechanical vs. digital
2.6 Why choose Secret Key?
2.7 Why build your own?
3Preliminaryconcepts
3.1 Bits and bytes
3.2 Functions and operators
3.3 Boolean operators
3.4 Number bases
3.5 Prime numbers
3.6 Modular arithmetic
4Cryptographer’stoolbox
4.1 Rating system
4.2 Substitution
Huffman codes
4.3 Transposition
4.4 Fractionation
4.5 Random number generators
Chained digit generator
4.6 Useful combinations, wasteful combinations
Bazeries type 4 cipher
5Substitutionciphers
5.1 Simple substitution
5.2 Mixing the alphabet
5.3 Nomenclators
5.4 Polyalphabetic substitution
5.5 The Belaso cipher
5.6 The Kasiski method
5.7 Index of Coincidence
5.8 Index of Coincidence, again
5.9 Solving a polyalphabetic cipher
Solving a Belaso cipher
Solving a Vigenère cipher
Solving a general polyalphabetic cipher
5.10 Autokey
5.11 Running key
5.12 Simulated rotor machines
Single-rotor machine
Three-rotor machine
Eight-rotor machine
6Countermeasures
6.1 Double encipherment
6.2 Null characters
6.3 Interrupted key
6.4 Homophonic substitution Cipher
6.5 Bigram and trigram substitution
6.6 Hiding messages in images
6.7 Adding null bits
6.8 Merging multiple messages
6.9 Embedding a message in a file
7Transposition
7.1 Route transposition
7.2 Columnar transposition Cysquare
Word transposition
7.3 Double columnar transposition
7.4 Cycling columnar transposition
7.5 Random number transposition
7.6 Selector transposition
7.7 Key transposition
7.8 Halving transposition
7.9 Multiple anagramming
8JeffersonWheelCypher
8.1 Known-word solution
8.2 Ciphertext-only solution
9Fractionation
9.1 Polybius square
9.2 Playfair
Solving a Playfair cipher
Strengthening a Playfair cipher
9.3 Two Square
9.4 Three Square
9.5 Four Square
9.6 Bifid
Conjugated matrix bifid
9.7 Diagonal bifid
9.8 6×6 squares
9.9 Trifid
9.10 Three Cube
9.11 Rectangular grids
9.12 Hexadecimal fractionation
9.13 Bitwise fractionation
Cyclic 8×N
9.14 Other fractionation
9.15 Stronger blocks
10Variable-lengthfractionation
10.1 Morse
10.2 Monom-Binom
10.3 Periodic lengths
10.4 Huffman Substitution
10.5 Post tag systems
Same-length tags
Different-length tags
Multiple alphabets
Short and long moves
10.6 Fractionation in other bases
10.7 Text compression
Lempel-Ziv
Arithmetic coding
Adaptive arithmetic coding
11Blockciphers
11.1 Substitution-permutation network
11.2 Data Encryption Standard (DES)
Double DES
Triple DES
Fast bit transposition
Short blocks
11.3 Matrix multiplication
11.4 Matrix multiplication
11.5 Advanced Encryption Standard (AES)
11.6 Fixed vs. keyed substitution
11.7 Involutory ciphers
Involutory substitution
Involutory polyalphabetic substitution
Involutory transposition
Involutory block cipher
Example, poly triple flip
11.8 Variable-length substitutions
11.9 Ripple ciphers
11.10 Block chaining
Polyalphabetic chaining
Enciphered chaining
Lagged chaining
Interior taps
Key chaining
Chaining mode summary
Chaining short blocks
Chaining variable-length blocks
11.11 Strengthening a block cipher
12Principlesforsecureencryption
12.1 Large blocks
12.2 Long keys
Redundant keys
12.3 Confusion
Correlation coefficient
Base-26 linearity
Base-256 linearity
Adding a backdoor
Condensed linearity
Hybrid linearity
Constructing an S-box S-box with a key
12.4 Diffusion
12.5 Saturation
13Streamciphers
13.1 Combining functions
13.2 Random numbers
13.3 Multiplicative congruential generator
13.4 Linear congruential generator
13.5 Chained exclusive-OR generator
13.6 Chained addition generator
13.7 Shift and XOR generator
13.8 FRand
13.9 Mersenne Twister
13.10 Linear feedback shift registers
13.11 Estimating the period
13.12 Strengthening a generator
13.13 Combining generators
13.14 True random numbers
Lagged linear addition
Layering images
13.15 Refreshing the random bytes
13.16 Synchronized key streams
13.17 Hash functions
14One-timepad
14.1 The Vernam cipher
14.2 Key supply
Circulating key
Combined key
Selection key
14.3 Indicators
14.4 Diffie-Hellman key exchange
Constructing large primes, old
Constructing large primes, new
15Matrixmethods
15.1 Inverting a matrix
15.2 Transposition matrix
15.3 The Hill cipher
15.4 Hill cipher, computer versions
15.5 Large integer multiplication
Multiplying and dividing congruences
15.6 Solving a linear congruence
Reducing a congruence
Half-and-Half Rule
Laddering
Continued fractions
15.7 Large integer ciphers
15.8 Small integer multiplication
15.9 Multiplication modulo P
15.10 Change of base
15.11 Rings
15.12 Matrices over a ring
15.13 Constructing a ring
Gaussian integers
Quaternions
15.14 Finding an invertible matrix
16Threepassprotocol
16.1 Shamir’s method
16.2 Massey-Omura
16.3 Discrete logarithm
Logarithms
Powers of primes
Crash
Factoring
Estimates
16.4 Matrix three pass protocol
Commutative family of matrices
Multiplicative order
Maximum order
Emily attacks
Non-commutative ring
Solving bilinear equations
Weaklings
Making it fast
16.5 Two-sided three pass protocol
17Codes
17.1 The Joker
18Quantumcomputers
18.1 Superposition
18.2 Entanglement
18.3 Error correction
18.4 Measurement
18.5 Quantum 3-stage protocol
18.6 Quantum key exchange
18.7 Grover’s algorithm
18.8 Equations
Transpositions
Substitutions
Karnaugh maps
Intermediate variables
Known plaintext
18.9 Minimization
Hill climbing
Mille sommets
Simulated annealing
18.10 Quantum simulated annealing
18.11 Quantum factoring
18.12 Ultracomputers
Substitution
Random numbers
Ultrasubstitution cipher US-A
Ultrastream cipher US-B
front matter foreword
From secret decoder rings to government policy statements, the challenges of hiding and discovering information within other information have long compelled the intellect. Cryptology is a fascinating subject with which almost every schoolchild has some hands-on familiarity. And yet, for good reasons, it is a discipline that throughout time has been shrouded in the deepest levels of secrecy and used by governments to protect their most sensitive weapons. Cryptography’s role in military and diplomatic affairs has always been deadly serious. It is no exaggeration to declare that successes and failures of cryptography have shaped the outcome of wars and the course of history; nor is it an exaggeration to state that the successes and failures of cryptography are setting our current course of history.
Consider the American Civil war battle of Antietam in September of 1862, when George McClellan commanded the Union forces against Robert E. Lee’s Confederate forces near Sharpsburg, Maryland. A few days earlier, two Union soldiers had found a piece of paper near their camp, which turned out to be a copy of an order issued by Lee detailing his plans for the invasion of Maryland. The order had not been encrypted. With the information it contained, McClellan precisely knew the location of the commands of
Lee’s scattered army and was able to destroy Lee’s army before they reunited.
Cryptographic successes and failures have shaped more recent history as well. The terrible Russian failure at Tannenberg in August 1914 was the direct result of the German army’s intercept of Russian communications. Amazingly, the Russian communications were totally in the clear because the Russians had not equipped their field commanders with ciphers and keys. The Russians were thus unable to securely coordinate the activities of neighboring units within each army.
What was to become 50 years of Cold War following WWII was also set up by a cryptographic failure, this time on the Japanese at the Battle of Midway in 1942. American cryptanalysts broke the Japanese codes and were reading many of the messages of the Combined Fleet. Stories like these are within the purview of classical cryptography. Secret Key Cryptography plays in this sandbox.
No one is more capable of enlightening an interested reader in all of the dimensions of recreational classical cryptology, from its mathematical heritage to its sociological implications, than Dr. Frank Rubin. Dr. Rubin’s education is in mathematics and computer science. He worked for 30 years at IBM in the Design Automation field and did cryptography for over 50 years. Dr. Rubin served as an editor for Cryptologia and other publications. He has written dozens of mathematics and computer algorithms and has created thousands of mathematical puzzles.
Secret Key Cryptography is more than an update to the classic Elementary Cryptanalysis by Helen F. Gaines. It covers the field from ancient times through the era of quantum computers. Secret Key Cryptography presents new methods and “cracking” technologies. Lastly, it explains a unique method to measure the strength of a cipher.1,2
The book comes at a strategic point in this evolving history. It provides a timely and important contribution to understanding this critical technology. Whether the reader is seeking edification about cryptology itself or is a practitioner of information security, the depth, and breadth of knowledge included in these pages will be a welcome source of useful information and valuable addition to a library.
—Randall K. Nichols, DTM
Randall K. Nichols is a former president, aristocrat, and book review editor for the American Cryptogram Association (ACA); the director of the Unmanned Aircraft Systems Cybersecurity Certificate Program at Kansas State University, Salina; and professor emeritus of Graduate Cybersecurity and Forensics at Utica College.
REFERENCES
Gaines, H. F. (1956). Cryptanalysis: A Study of Ciphers and their Solution. NYC: Dover.
LANAKI. (1998). Classical Cryptography Course Vol. I. Laguna Hills, CA: Aegean Park Press.
LANAKI. (1999). Classical Cryptography Course Vol. II. Laguna Hills, CA: Aegean Park Press.
Nichols, R. K. (1999). ICSA Guide to Cryptography. New York City: McGraw Hill.
Rubin, F. (2022). Secret Key Cryptography. Shelter Island, New York: Manning Books.
Schneier, B. (1995). Applied Cryptography: Protocols, Algorithms and Source Code in C. New York: John Wiley & Sons.
preface
There are several threads that led to the writing of this book. Let’s begin with my high school friend Charlie Rose. Charlie worked in the school bookstore. One day, while ordering books for the store, he noticed the book Cryptanalysis by Helen F. Gaines. Charlie wanted the book, and he also wanted the employee discount. But there was a hitch. The minimum order the store could place was three copies.
Charlie needed to get two other people to buy the book. He promised that we would all read the book together, then make up cryptograms that the others would solve. I bought the book, read it, and made up cryptograms, but Charlie had lost interest.
The back cover of Cryptanalysis had a long-outdated street address for the American Cryptogram Association (www.cryptogram.org) but I tracked them down and joined. I started solving the many types of cryptograms they published in their hobbyist newsletter, The Cryptogram, and after a few years I became an assistant editor. I remained a member for over 40 years.
In 1977 a more professional journal of cryptography, called Cryptologia, began. You can find it at https://www.tandfonline.com/toc/ucry20/current. I started reading the articles, then contributing articles, and then became an editor. Somehow I became the “crackpot handler.” Those articles all came to me, and I had to find my way through the illogical logic to see if a good idea was hidden inside. In just one case there was. I turned that into an article for The Cryptogram. The author was so grateful he planted a tree in Israel in my honor.
This experience taught me how to separate those articles that were just badly written or where the author had simply overestimated the strength of the cipher from those that were truly off the wall. This is what I learned: the amateur with a weak cipher can describe the cipher and write out the steps. The true crackpots cannot get their vague and grandiose imaginings onto paper. They can write reams about how wonderful their ciphers are, but they cannot write out the steps. They cannot turn their inchoate thoughts into a concrete algorithm.
Starting around 2005, I started taking courses at Marist College CLS, Continuing Life Studies. Soon I was giving lectures on Sudoku, SumSum and other puzzles (I have written three books of Sudoku puzzles); my travels in Tanzania and Mongolia; the construction of the Empire State Building; the life of Alan Turing; and other subjects. I became part of the curriculum committee.
In 2018 I volunteered to give a two-semester course on cryptography. While creating the nearly 450 slides I needed for the course, I realized I had enough material for a book. Fortunately, I discovered that a year earlier I had started writing just such a book. This one.
acknowledgments
The other day I overheard my wife, Miriam, speaking to a friend on the phone, “It is like I am in a ménage à trois, me, Frank and the book.” Thank you, Miriam, for your forbearance during the 18 months it took to write the book, the year searching for a publisher, the 6 months hunting for a literary agent, the year watching the literary agent get no results, and finally the single month to find a home for this book at Manning. Plus over 18 months of reviewing, revising, editing, revising, typesetting, revising, indexing, writing marketing copy, and more.
I thank all of the people at Manning Publications who helped with this book, especially Michael Stephens who took a chance and offered me a contract, and who helped at every
stage of the process; Marina Michaels for her many editorial improvements; Rebecca Rinehart for smoothing the path; Jen Houle and Susan Honeywell for their work on the illustrations; Tiffany Taylor for her many valuable suggestions on: grammar and punctuation; Paul Wells and Keri Hales for their work on the production of the book; Sam Wood for the marketing copy; Dennis Dalinnik for the typesetting; and, of course, Marjan Bace, the publisher.
Special thanks to Prof. Randall K. Nichols for writing the book’s foreword and a review in The Cryptogram on very short notice. Thanks also to Prof. Thomas Perera of the Enigma Museum for providing the Fialka images.
Thanks to the reviewers who read the manuscript and made numerous suggestions and useful criticisms: Christopher Kardell, Alex Lucas, Gabor Hajba, Michal Rutka, Jason Taylor, Roy Prins, Matthew Harvell, Riccardo Marotti, and Paul Love. Your suggestions helped make this a better book.
Finally, I must acknowledge the unwitting role of Lee Harvey Oswald, whose heinous assassination of Pres. John F. Kennedy prevented me from taking a security interview at FBI headquarters, which prevented me from joining the NSA, which would have made it a felony for me to write this book.
about this book
Who should read this book?
This book is aimed at a broad range of readers: general readers, cryptography hobbyists, history buffs, computer science students, electrical engineers, mathematicians, and professional cryptographers. This makes my job harder, because it is impossible to make every part of the book suitable for every type of reader. Some parts of the book may need too much math for some readers. Some parts may be too elementary for some readers. In this section I attempt to guide readers to what I think is the most appropriate material for them.
General readers can read straight through to the end of chapter 8. Simply skip anything where the math is too hard, or the exposition is too technical. From chapter 9 on it starts to get sticky. They can skim from this point on, and pick out the topics of interest. They may want to read chapter 12 to get the general gist, without getting into the details.
Cryptography hobbyists will probably want to read the entire book, then come back for a more detailed look at sections 4.2 to 5.11, 6.1 to 6.5, 6.7, most of chapter 7, and sections 9.1 to 9.9, plus the Fun Pages and the Challenge page.
History buffs can read the entire book, ignoring the math, to get the timeline of when each method was developed, and by whom.
Computer science students may put special emphasis on sections 5.6 to 5.11, chapter 8, and chapters 11 to 16.
Electrical engineers will be looking for practical methods. They should first read chapters 2 and 4 for a basic grounding, then read sections 7.2 to 7.8, chapter 9, and chapters 11 to 16, with special emphasis on chapter 12.
Mathematicians will be most interested in section 4.5, sections 5.6 to 5.12, sections 10.4 to 10.7, sections 11.7 to 11.10, sections 12.3 to 12.6, chapters 13 to 16, especially section 16.4.6, and chapter 18.
Professional cryptographers will be most interested in sections 7.8, 8.2, 10.5, 10.7, 11.4, 12.3 to 12.6, 13.8, 13.15, 14.2, 14.4, 15.4 to 15.14, 16.4, 16.5, and 18.12.
About the ciphers
I have included a number of Fun Ciphers and Challenge Ciphers for readers who want to try their hand at solving. The Fun Ciphers use standard methods described in the book.
The Challenge Ciphers use methods that I have invented myself. They are simple enough that an amateur hobbyist
could both guess the methods, and solve them. I have tried to be fair so that interested readers can solve them. Nothing bizarre or complex. No weird words or distorted letter frequencies. And sufficient material for solving them.
You may notice some sections that begin with a bold * and end with **. These are optional sections that may contain computer algorithms or deeper math. Some readers may choose to skip these.
liveBook discussion forum
Purchase of Secret Key Cryptography includes free access to liveBook, Manning’s online reading platform. Using liveBook’s exclusive discussion features, you can attach comments to the book globally or to specific sections or paragraphs. It’s a snap to make notes for yourself, ask and answer technical questions, and receive help from the author and other users. To access the forum, go to https://livebook.manning.com/book/secret-keycryptography/discussion. You can also learn more about Manning’s forums and the rules of conduct at https://livebook.manning.com/discussion.
Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). The forum and the
archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.
Other online resources
You can find the author's cryptographic products at his website, www.mastersoftware.biz.
about the author
Frank Rubin holds a BS and MS in mathematics and a PhD in computer science. He worked for 28 years at IBM in the design automation field, where he designed and wrote
specialized software that IBM engineers used to design computers and circuits. He is the owner of Master Software Corp. which produces cryptographic software. Frank has been issued four U.S. patents on cryptographic methods. Frank has about 50 papers published in refereed journals on cryptography, computer circuits, graph theory and pure mathematics, plus several books (user manuals and project specifications) published internally at IBM. In cryptography he is best known for solving the Jefferson Cypher Wheel. In computer science Frank is best known for arithmetic coding, now one of the standard methods for text compression, and for his algorithm for finding Hamilton paths. In pure mathematics he is probably best known for introducing the concept of a finite-state recognizer to measure theory.
Frank has three published books of Sudoku puzzles and two self-published books of SumSum puzzles. He is the author of more than 3,500 puzzles published in The Cryptogram, Technology Review, and Journal of Recreational Mathematics, and he is the only person ever honored by having a special issue of JRM dedicated entirely to his own puzzles.
about the cover illustration
The figure on the cover of Secret Key Cryptography is “Le Garçon de Bureau,” or “Office Assistant,” taken from a book edited by Louis Curmer, published in 1841. Each illustration is finely drawn and colored by hand.
In those days, it was easy to identify where people lived and what their trade or station in life was just by their dress. Manning celebrates the inventiveness and initiative of today’s computer business with book covers based on the rich diversity of regional culture centuries ago, brought back to life by pictures from collections such as this one.
1. Both ICSA Guide to Cryptography by R. K. Nichols and Applied Cryptography by Bruce Schneier present cipher strength and randomness methods. The former concentrates on classical cryptography, and the latter concentrates on modern ciphers (Nichols, 1999; Schneier, 1995).
2. Secret Key Cryptography is better defined and written than my first two books on classical cryptography, namely, Classical Cryptography Course Vols. I & II (LANAKI, 1998; 1999).
