Deep learning for the life sciences applying deep learning to genomics microscopy drug discovery and
Microscopy Drug Discovery and More 1st Edition Bharath Ramsundar
Visit to download the full and correct content document: https://textbookfull.com/product/deep-learning-for-the-life-sciences-applying-deep-lear ning-to-genomics-microscopy-drug-discovery-and-more-1st-edition-bharath-ramsund ar/
More products digital (pdf, epub, mobi) instant download maybe you interests ...
Programming PyTorch for Deep Learning Creating and Deploying Deep Learning Applications 1st Edition Ian Pointer
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.
DevelopmentEditor: Nicole Tache
AcquisitionsEditor: Mike Loukides
ProductionEditor: Katherine Tozer
Copyeditor: Rachel Head
Proofreader: Zachary Corleissen
April 2019: First Edition
RevisionHistoryfortheFirstEdition 2019-03-27: First Release
Indexer: Ellen Troutman-Zaig
InteriorDesigner: David Futato
CoverDesigner: Karen Montgomery
Illustrator: Rebecca Demarest
See http://bit.ly/deep-learning-life-science for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Deep Learning for the Life Sciences, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
978-1-492-03983-9
[LSI]
5. Biophysical Machine Learning.
7.
6. Deep Learning for Genomics.
10. Interpretation of Deep Models.
11. A Virtual Screening Work ow Example.
12. Prospects and Perspectives.
Preface
In recent years, life science and data science have converged. Advances in robotics and automation have enabled chemists and biologists to generate enormous amounts of data. Scientists today are capable of generating more data in a day than their prede‐cessors 20 years ago could have generated in an entire career. This ability to rapidly generate data has also created a number of new scientific challenges. We are no longer in an era where data can be processed by loading it into a spreadsheet and making a couple of graphs. In order to distill scientific knowledge from these datasets, we must be able to identify and extract nonobvious relationships.
One technique that has emerged over the last few years as a powerful tool for identi‐fying patterns and relationships in data is deep learning, a class of algorithms that have revolutionized approaches to problems such as image analysis, language transla‐tion, and speech recognition. Deep learning algorithms excel at identifying and exploiting patterns in large datasets. For these reasons, deep learning has broad appli‐cations across life science disciplines. This book provides an overview of how deep learning has been applied in a number of areas including genetics, drug discovery, and medical diagnosis. Many of the examples we describe are accompanied by code examples that provide a practical introduction to the methods and give the reader a starting point for future research and exploration.
Conventions Used in This Book
The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program ele‐ments such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐mined by context.
This element signifies a tip or suggestion.
This element signifies a general note.
This element indicates a warning or caution.
Using Code Examples
Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/deepchem/DeepLearningLifeSciences.
This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a signifi‐cant amount of example code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Deep Learning for the Life Sciences by Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande (O’Reilly).
Copyright 2019 Bharath Ramsundar, Karl Leswing, Peter Eastman, and Vijay Pande, 978-1-492-03983-9.”
viii | Preface
If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.
O’Reilly Online Learning
For almost 40 years, O’Reilly has provided technology and business training, knowledge, and insight to help companies succeed.
Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, indepth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com.
How to Contact Us
Please address comments and questions concerning this book to the publisher:
O’Reilly Media, Inc.
1005 Gravenstein Highway North Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/deep-lrng-for-life-science.
To comment or ask technical questions about this book, send email to bookques‐tions@oreilly.com.
For more information about our books, courses, conferences, and news, see our web‐site at http://www.oreilly.com
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Acknowledgments
We would like to thank Nicole Tache, our editor at O’Reilly, as well as the tech review‐ers and beta reviewers for their valuable contributions to the book. In addtion, we would like to thank Karl Leswing and Zhenqin (Michael) Wu for their contributions to the code and Johnny Israeli for valuable advice on the genomics chapter.
Bharath would like to thank his family for their support and encouragement during many long weekends and nights working on this book.
Peter would like to thank his wife for her constant support, as well as the many col‐leagues from whom he has learned so much about machine learning.
Pat would like to thank his wife Andrea, and his daughters Alee and Maddy, for their love and support. He would also like to acknowledge past and present colleagues at Vertex Pharmaceuticals and Relay Therapeutics, from whom he has learned so much.
Finally, we want to thank the DeepChem open source community for their encour‐agement and support throughout this project.
x | Preface
CHAPTER 1
Why Life Science?
While there are many directions that those with a technical inclination and a passion for data can pursue, few areas can match the fundamental impact of biomedical research. The advent of modern medicine has fundamentally changed the nature of human existence. Over the last 20 years, we have seen innovations that have trans‐formed the lives of countless individuals. When it first appeared in 1981, HIV/AIDS was a largely fatal disease. Continued development of antiretroviral therapies has dra‐matically extended the life expectancy for patients in the developed world. Other dis‐eases, such as hepatitis C, which was considered largely untreatable a decade ago, can now be cured. Advances in genetics are enabling the identification and, hopefully soon, the treatment of a wide array of diseases. Innovations in diagnostics and instru‐mentation have enabled physicians to specifically identify and target disease in the human body. Many of these breakthroughs have benefited from and will continue to be advanced by computational methods.
Why Deep Learning?
Machine learning algorithms are now a key component of everything from online shopping to social media. Teams of computer scientists are developing algorithms that enable digital assistants such as the Amazon Echo or Google Home to under‐stand speech. Advances in machine learning have enabled routine on-the-fly transla‐tion of web pages between spoken languages. In addition to machine learning’s impact on everyday life, it has impacted many areas of the physical and life sciences. Algorithms are being applied to everything from the detection of new galaxies from telescope images to the classification of subatomic interactions at the Large Hadron Collider.
One of the drivers of these technological advances has been the development of a class of machine learning methods known as deep neural networks. While the tech‐
nological underpinnings of artificial neural networks were developed in the 1950s and refined in the 1980s, the true power of the technique wasn’t fully realized until advances in computer hardware became available over the last 10 years. We will pro‐vide a more complete overview of deep neural networks in the next chapter, but it is important to acknowledge some of the advances that have occurred through the application of deep learning:
• Many of the developments in speech recognition that have become ubiquitous in cell phones, computers, televisions, and other internet-connected devices have been driven by deep learning.
• Image recognition is a key component of self-driving cars, internet search, and other applications. Many of the same developments in deep learning that drove consumer applications are now being used in biomedical research, for example, to classify tumor cells into different types.
• Recommender systems have become a key component of the online experience. Companies like Amazon use deep learning to drive their “customers who bought this also bought” approach to encouraging additional purchases. Netflix uses a similar approach to recommend movies that an individual may want to watch. Many of the ideas behind these recommender systems are being used to identify new molecules that may provide starting points for drug discovery efforts.
• Language translation was once the domain of very complex rule-based systems. Over the last few years, systems driven by deep learning have outperformed sys‐tems that had undergone years of manual curation. Many of the same ideas are now being used to extract concepts from the scientific literature and alert scien‐tists to journal articles that they may have missed.
These are just a few of the innovations that have come about through the application of deep learning methods. We are at an interesting time when we have a convergence of widely available scientific data and methods for processing that data. Those with the ability to combine data with new methods for learning from patterns in that data can make significant scientific advances.
Contemporary Life Science Is About Data
As mentioned previously, the fundamental nature of life science has changed. The availability of robotics and miniaturized experiments has brought about dramatic increases in the amount of experimental data that can be generated. In the 1980s a biologist would perform a single experiment and generate a single result. This sort of data could typically be manipulated by hand with the possible assistance of a pocket calculator. If we fast-forward to today’s biology, we have instrumentation that is capa‐ble of generating millions of experimental data points in a day or two. Experiments
2 | Chapter1:WhyLifeScience?
like gene sequencing, which can generate huge datasets, have become inexpensive and routine.
The advances in gene sequencing have led to the construction of databases that link an individual’s genetic code to a multitude of health-related outcomes, including dia‐betes, cancer, and genetic diseases such as cystic fibrosis. By using computational techniques to analyze and mine this data, scientists are developing an understanding of the causes of these diseases and using this understanding to develop new treat‐ments.
Disciplines that once relied primarily on human observation are now utilizing data‐sets that simply could not be analyzed manually. Machine learning is now routinely used to classify images of cells. The output of these machine learning models is used to identify and classify cancerous tumors and to evaluate the effects of potential dis‐ease treatments.
Advances in experimental techniques have led to the development of several data‐bases that catalog the structures of chemicals and the effects that these chemicals have on a wide range of biological processes or activities. These structure–activity relation‐ships (SARs) form the basis of a field known as chemical informatics, or cheminfor‐matics. Scientists mine these large datasets and use the data to build predictive models that will drive the next generation of drug development.
With these large amounts of data comes a need for a new breed of scientist who is comfortable in both the scientific and computational domains. Those with these hybrid capabilities have the potential to unlock structure and trends in large datasets and to make the scientific discoveries of tomorrow.
What Will You Learn?
In the first few chapters of this book, we provide an overview of deep learning and how it can be applied in the life sciences. We begin with machine learning, which has been defined as “the science (and art) of programming computers so that they can learn from data.”1
Chapter 2 provides a brief introduction to deep learning. We begin with an example of how this type of machine learning can be used to perform a simple task like linear regression, and progress to more sophisticated models that are commonly used to solve real-world problems in the life sciences. Machine learning typically proceeds by initially splitting a dataset into a training set that is used to generate a model and a test set that is used to assess the performance of the model. In Chapter 2 we discuss
1 Furbush, James. “Machine Learning: A Quick and Simple Definition.” https://www.oreilly.com/ideas/machinelearning-a-quick-and-simple-definition. 2018.
some of the details surrounding the training and validation of predictive models. Once a model has been generated, its performance can typically be optimized by varying a number of characteristics known as hyperparameters. The chapter provides an overview of this process. Deep learning is not a single technique, but a set of related methods. Chapter 2 concludes with an introduction to a few of the most important deep learning variants.
In Chapter 3, we introduce DeepChem, an open source programming library that has been specifically designed to simplify the creation of deep learning models for a vari‐ety of life science applications. After providing an overview of DeepChem, we intro‐duce our first programming example, which demonstrates how the DeepChem library can be used to generate a model for predicting the toxicity of molecules. In a second programming example, we show how DeepChem can be used to classify images, a common task in modern biology. As briefly mentioned earlier, deep learn‐ing is used in a variety of imaging applications, ranging from cancer diagnosis to the detection of glaucoma. This discussion of specific applications then motivates an explanation of some of the inner workings of deep learning methods.
Chapter 4 provides an overview of how machine learning can be applied to mole‐cules. We begin by introducing molecules, the building blocks of everything around us. Although molecules can be considered analogous to building blocks, they are not rigid. Molecules are flexible and exhibit dynamic behavior. In order to characterize molecules using a computational method like deep learning, we need to find a way to represent molecules in a computer. These encodings can be thought of as similar to the way in which an image can be represented as a set of pixels. In the second half of Chapter 4, we describe a number of ways that molecules can be represented and how these representations can be used to build deep learning models.
Chapter 5 provides an introduction to the field of biophysics, which applies the laws of physics to biological phenomena. We start with a discussion of proteins, the molec‐ular machines that make life possible. A key component of predicting the effects of drugs on the body is understanding their interactions with proteins. In order to understand these effects, we begin with an overview of how proteins are constructed and how protein structures differ. Proteins are entities whose 3D structure dictates their biological function. For a machine learning model to predict the impact of a drug molecule on a protein’s function, we need to represent that 3D structure in a form that can be processed by a machine learning program. In the second half of Chapter 5, we explore a number of ways that protein structures can be represented. With this knowledge in hand, we then review another code example where we use deep learning to predict the degree to which a drug molecule will interact with a pro‐tein.
Genetics has become a key component of contemporary medicine. The genetic sequencing of tumors has enabled the personalized treatment of cancer and has the 4 | Chapter1:WhyLifeScience?
potential to revolutionize medicine. Gene sequencing, which used to be a complex process requiring huge investments, has now become commonplace and can be rou‐tinely carried out. We have even reached the point where dog owners can get inex‐pensive genetic tests to determine their pets’ lineage. In Chapter 6, we provide an overview of genetics and genomics, beginning with an introduction to DNA and RNA, the templates that are used to produce proteins. Recent discoveries have revealed that the interactions of DNA and RNA are much more complex than origi‐nally believed. In the second half of Chapter 6, we present several code examples that demonstrate how deep learning can be used to predict a number of factors that influ‐ence the interactions of DNA and RNA.
Earlier in this chapter, we alluded to the many advances that have come about through the application of deep learning to the analysis of biological and medical images. Many of the phenomena studied in these experiments are too small to be observed by the human eye. In order to obtain the images used with deep learning methods, we need to utilize a microscope. Chapter 7 provides an overview of micro‐scopy in its myriad forms, ranging from the simple light microscope we all used in school to sophisticated instruments that are capable of obtaining images at atomic resolution. This chapter also covers some of the limitations of current approaches, and provides information on the experimental pipelines used to obtain the images that drive deep learning models.
One area that offers tremendous promise is the application of deep learning to medi‐cal diagnosis. Medicine is incredibly complex, and no physician can personally embody all of the available medical knowledge. In an ideal situation, a machine learn‐ing model could digest the medical literature and aid medical professionals in making diagnoses. While we have yet to reach this point, a number of positive steps have been made. Chapter 8 begins with a history of machine learning methods for medical diag‐nosis and charts the transition from hand-encoded rules to statistical analysis of med‐ical outcomes. As with many of the topics we’ve discussed, a key component is representing medical information in a format that can be processed by a machine learning program. In this chapter, we provide an introduction to electronic health records and some of the issues surrounding these records. In many cases, medical images can be very complex and the analysis and interpretation of these images can be difficult for even skilled human specialists. In these cases, deep learning can aug‐ment the skills of a human analyst by classifying images and identifying key features. Chapter 8 concludes with a number of examples of how deep learning is used to ana‐lyze medical images from a variety of areas.
As we mentioned earlier, machine learning is becoming a key component of drug dis‐covery efforts. Scientists use deep learning models to evaluate the interactions between drug molecules and proteins. These interactions can elicit a biological response that has a therapeutic impact on a patient. The models we’ve discussed so far are discriminative models. Given a set of characteristics of a molecule, the model gen‐
erates a prediction of some property. These predictions require an input molecule, which may be derived from a large database of available molecules or may come from the imagination of a scientist. What if, rather than relying on what currently exists, or what we can imagine, we had a computer program that could “invent” new mole‐cules? Chapter 9 presents a type of deep learning program called a generative model. A generative model is initially trained on a set of existing molecules, then used to generate new molecules. The deep learning program that generates these molecules can also be influenced by other models that predict the activity of the new molecules.
Up to now, we have discussed deep learning models as “black boxes.” We present the model with a set of input data and the model generates a prediction, with no explana‐tion of how or why the prediction was generated. This type of prediction can be less than optimal in many situations. If we have a deep learning model for medical diag‐nosis, we often need to understand the reasoning behind the diagnosis. An explana‐tion of the reasons for the diagnosis will provide a physician with more confidence in the prediction and may also influence treatment decisions. One historic drawback to deep learning has been the fact that the models, while often reliable, can be difficult to interpret. A number of techniques are currently being developed to enable users to better understand the factors that led to a prediction. Chapter 10 provides an over‐view of some of these techniques used to enable human understanding of model pre‐dictions. Another important aspect of predictive models is the accuracy of a model’s predictions. An understanding of a model’s accuracy can help us determine how much to rely on that model. Given that machine learning can be used to potentially make life-saving diagnoses, an understanding of model accuracy is critical. The final section of Chapter 10 provides an overview of some of the techniques that can be used to assess the accuracy of model predictions.
In Chapter 11 we present a real-world case study using DeepChem. In this example, we use a technique called virtual screening to identify potential starting points for the discovery of new drugs. Drug discovery is a complex process that often begins with a technique known as screening. Screening is used to identify molecules that can be optimized to eventually generate drugs. Screening can be carried out experimentally, where millions of molecules are tested in miniaturized biological tests known as assays, or in a computer using virtual screening. In virtual screening, a set of known drugs or other biologically active molecules is used to train a machine learning model. This machine learning model is then used to predict the activity of a large set of molecules. Because of the speed of machine learning methods, hundreds of mil‐lions of molecules can typically be processed in a few days of computer time.
The final chapter of the book examines the current impact and future potential of deep learning in the life sciences. A number of challenges for current efforts, includ‐ing the availability and quality of datasets, are discussed. We also highlight opportuni‐ties and potential pitfalls in a number of other areas including diagnostics, personalized medicine, pharmaceutical development, and biology research.
6 | Chapter1:WhyLifeScience?
CHAPTER
2
Introduction to Deep Learning
The goal of this chapter is to introduce the basic principles of deep learning. If you already have lots of experience with deep learning, you should feel free to skim this chapter and then go on to the next. If you have less experience, you should study this chapter carefully as the material it covers will be essential to understanding the rest of the book.
In most of the problems we will discuss, our task will be to create a mathematical function:
y = f x
Notice that x and y are written in bold. This indicates they are vectors. The function might take many numbers as input, perhaps thousands or even millions, and it might produce many numbers as outputs. Here are some examples of functions you might want to create:
• x contains the colors of all the pixels in an image. fx should equal 1 if the image contains a cat and 0 if it does not.
• The same as above, except fx should be a vector of numbers. The first element indicates whether the image contains a cat, the second whether it contains a dog, the third whether it contains an airplane, and so on for thousands of types of objects.
• x contains the DNA sequence for a chromosome. y should be a vector whose length equals the number of bases in the chromosome. Each element should equal 1 if that base is part of a region that codes for a protein, or 0 if not.
• x describes the structure of a molecule. (We will discuss various ways of repre‐senting molecules in later chapters.) y should be a vector where each element
describes some physical property of the molecule: how easily it dissolves in water, how strongly it binds to some other molecule, and so on.
As you can see, fx could be a very, very complicated function! It usually takes a long vector as input and tries to extract information from it that is not at all obvious just from looking at the input numbers.
The traditional approach to solving this problem is to design a function by hand. You would start by analyzing the problem. What patterns of pixels tend to indicate the presence of a cat? What patterns of DNA tend to distinguish coding regions from noncoding ones? You would write computer code to recognize particular types of fea‐tures, then try to identify combinations of features that reliably produce the result you want. This process is slow and labor-intensive, and depends heavily on the exper‐tise of the person carrying it out.
Machine learning takes a totally different approach. Instead of designing a function by hand, you allow the computer to learn its own function based on data. You collect thousands or millions of images, each labeled to indicate whether it includes a cat. You present all of this training data to the computer, and let it search for a function that is consistently close to 1 for the images with cats and close to 0 for the ones without.
What does it mean to “let the computer search for a function”? Generally speaking, you create a model that defines some large class of functions. The model includes parameters, variables that can take on any value. By choosing the values of the param‐eters, you select a particular function out of all the many functions in the class defined by the model. The computer’s job is to select values for the parameters. It tries to find values such that, when your training data is used as input, the output is as close as possible to the corresponding targets.
Linear Models
One of the simplest models you might consider trying is a linear model: y = Mx + b
In this equation, M is a matrix (sometimes referred to as the “weights”) and b is a vector (referred to as the “biases”). Their sizes are determined by the numbers of input and output values. If x has length T and you want y to have length S, then M will be an S × T matrix and b will be a vector of length S. Together, they make up the parameters of the model. This equation simply says that each output component is a linear combination of the input components. By setting the parameters (M and b), you can choose any linear combination you want for each component. 8 | Chapter2:IntroductiontoDeepLearning
This was one of the very earliest machine learning models. It was introduced back in 1957 and was called a perceptron. The name is an amazing piece of marketing: it has a science fiction sound to it and seems to promise wonderful things, when in fact it is nothing more than a linear transform. In any case, the name has managed to stick for more than half a century.
The linear model is very easy to formulate in a completely generic way. It has exactly the same form no matter what problem you apply it to. The only differences between linear models are the lengths of the input and output vectors. From there, it is just a matter of choosing the parameter values, which can be done in a straightforward way with generic algorithms. That is exactly what we want for machine learning: a model and algorithms that are independent of what problem you are trying to solve. Just provide the training data, and parameters are automatically determined that trans‐form the generic model into a function that solves your problem.
Unfortunately, linear models are also very limited. As demonstrated in Figure 2-1, a linear model (in one dimension, that means a straight line) simply cannot fit most real datasets. The problem becomes even worse when you move to very highdimensional data. No linear combination of pixel values in an image will reliably identify whether the image contains a cat. The task requires a much more compli‐cated nonlinear model. In fact, any model that solves that problem will necessarily be very complicated and very nonlinear. But how can we formulate it in a generic way? The space of all possible nonlinear functions is infinitely complex. How can we define a model such that, just by choosing values of parameters, we can create almost any nonlinear function we are ever likely to want?
Figure 2-1. A linear model cannot fit data points that follow a curve. This requires a nonlinear model.
Multilayer Perceptrons
A simple approach is to stack multiple linear transforms, one after another. For example, we could write:
y = M2 M1x + b1 + b2
Look carefully at what we have done here. We start with an ordinary linear transform, M1x + b1. We then pass the result through a nonlinear function x , and then apply a second linear transform to the result. The function x , which is known as the activation function, is an essential part of what makes this work. Without it, the model would still be linear, and no more powerful than the previous one. A linear combina‐tion of linear combinations is itself nothing more than a linear combination of the original inputs! By inserting a nonlinearity, we enable the model to learn a much wider range of functions.
We don’t need to stop at two linear transforms. We can stack as many as we want on top of each other:
This model is called a multilayer perceptron, or MLP for short. The middle steps hi are called hidden layers. The name refers to the fact that they are neither inputs nor out‐puts, just intermediate values used in the process of calculating the result. Also notice that we have added a subscript to each x . This indicates that different layers might use different nonlinearities.
You can visualize this calculation as a stack of layers, as shown in Figure 2-2. Each layer corresponds to a linear transformation followed by a nonlinearity. Information flows from one layer to another, the output of one layer becoming the input to the next. Each layer has its own set of parameters that determine how its output is calcu‐lated from its input.
10 | Chapter2:IntroductiontoDeepLearning
Figure 2-2. A multilayer perceptron, viewed as a stack of layers with information flowing from one layer to the next.
Multilayer perceptrons and their variants are also sometimes called neural networks. The name reflects the parallels between machine learning and neurobiology. A bio‐logical neuron connects to many other neurons. It receives signals from them, adds the signals together, and then sends out its own signals based on the result. As a very rough approximation, you can think of MLPs as working the same way as the neu‐rons in your brain!
What should the activation function x be? The surprising answer is that it mostly doesn’t matter. Of course, that is not entirely true. It obviously does matter, but not as much as you might expect. Nearly any reasonable function (monotonic, reasonably smooth) can work. Lots of different functions have been tried over the years, and although some work better than others, nearly all of them can produce decent results.
The most popular activation function today is probably the rectified linear unit (ReLU), x =max 0,x . If you aren’t sure what function to use, this is probably a good default. Other common choices include the hyperbolic tangent, tanhx , and the logistic sigmoid, x =1/ 1+ e x . All of these functions are shown in Figure 2-3
Figure 2-3. Three common activation functions: the rectified linear unit, hyperbolic tan‐gent, and logistic sigmoid.
We also must choose two other properties for an MLP: its width and its depth. With the simple linear model, we had no choices to make. Given the lengths of x and y, the
sizes of M and b were completely determined. Not so with hidden layers. Width refers to the size of the hidden layers. We can choose each hi to have any length we want. Depending on the problem, you might want them to be much larger or much smaller than the input and output vectors.
Depth refers to the number of layers in the model. A model with only one hidden layer is described as shallow. A model with many hidden layers is described as deep. This is, in fact, the origin of the term “deep learning”; it simply means “machine learning using models with lots of layers.”
Choosing the number and widths of layers in your model involves as much art as sci‐ence. Or, to put it more formally, “This is still an active field of research.” Often it just comes down to trying lots of combinations and seeing what works. There are a few principles that may provide guidance, however, or at least help you understand your results in hindsight:
1. An MLP with one hidden layer is a universal approximator. This means it can approximate any function at all (within certain fairly reason‐able limits). In a sense, you never need more than one hidden layer. That is already enough to reproduce any function you are ever likely to want. Unfortu‐nately, this result comes with a major caveat: the accuracy of the approximation depends on the width of the hidden layer, and you may need a very wide layer to get sufficient accuracy for a given problem. This brings us to the second princi‐ple.
2. Deep models tend to require fewer parameters than shallow ones.
This statement is intentionally somewhat vague. More rigorous statements can be proven for particular special cases, but it does still apply as a general guideline. Here is perhaps a better way of stating it: every problem requires a model with a certain depth to efficiently achieve acceptable accuracy. At shallower depths, the required widths of the layers (and hence the total number of parameters) increase rapidly. This makes it sound like you should always prefer deep models over shallow ones. Unfortunately, it is partly contradicted by the third principle.
3. Deep models tend to be harder to train than shallow ones.
Until about 2007, most machine learning models were shallow. The theoretical advantages of deep models were known, but researchers were usually unsuccess‐ful at training them. Since then, a series of advances has gradually improved the usefulness of deep models. These include better training algorithms, new types of models that are easier to train, and of course faster computers combined with larger datasets on which to train the models. These advances gave rise to “deep learning” as a field. Yet despite the improvements, the general principle remains true: deeper models tend to be harder to train than shallower ones. 12 | Chapter2:IntroductiontoDeepLearning
Training Models
This brings us to the next subject: just how do we train a model anyway? MLPs pro‐vide us with a (mostly) generic model that can be used for any problem. (We will dis‐cuss other, more specialized types of models a little later.) Now we want a similarly generic algorithm to find the optimal values of the model’s parameters for a given problem. How do we do that?
The first thing you need, of course, is a collection of data to train it on. This dataset is known as the training set. It should consist of a large number of (x,y) pairs, also known as samples. Each sample specifies an input to the model, and what you want the model’s output to be when given that input. For example, the training set could be a collection of images, along with labels indicating whether or not each image con‐tains a cat.
Next you need to define a loss function Ly,y , where y is the actual output from the model and y is the target value specified in the training set. This is how you measure whether the model is doing a good job of reproducing the training data. It is then averaged over every sample in the training set:
average loss= 1 N Σ i =1 N L yi, yi
Ly,y should be small when its arguments are close together and large when they are far apart. In other words, we take every sample in the training set, try using each one as an input to the model, and see how close the output is to the target value. Then we average this over the whole training set.
An appropriate loss function needs to be chosen for each problem. A common choice is the Euclidean distance (also known as the L2 distance), L y, y = Σi yi yi 2. (In this expression, yi means the i‘th component of the vector y.) When y represents a probability distribution, a popular choice is the cross entropy, L y, y =−Σi yi log yi. Other choices are also possible, and there is no uni‐versal “best” choice. It depends on the details of your problem.
Now that we have a way to measure how well the model works, we need a way to improve it. We want to search for the parameter values that minimize the average loss over the training set. There are many ways to do this, but most work in deep learning uses some variant of the gradient descent algorithm. Let θ represent the set of all parameters in the model. Gradient descent involves taking a series of small steps:
where L is the average loss over the training set. Each step moves a tiny distance in the “downhill” direction. It changes each of the model’s parameters by a little bit, with the goal of causing the average loss to decrease. If all the stars align and the phase of the moon is just right, this will eventually produce parameters that do a good job of solving your problem. is called the learning rate, and it determines how much the parameters change on each step. It needs to be chosen very carefully: too small a value will cause learning to be very slow, while too large a value will prevent the algo‐rithm from learning at all.
This algorithm really does work, but it has a serious problem. For every step of gradi‐ent descent, we need to loop over every sample in the training set. That means the time required to train the model is proportional to the size of the training set! Sup‐pose that you have one million samples in the training set, that computing the gradi‐ent of the loss for one sample requires one million operations, and that it takes one million steps to find a good model. (All of these numbers are fairly typical of real deep learning applications.) Training will then require one quintillion operations. That takes quite a long time, even on a fast computer.
Fortunately, there is a better solution: estimate L by averaging over a much smaller number of samples. This is the basis of the stochastic gradient descent (SGD) algo‐rithm. For every step, we take a small set of samples (known as a batch) from the training set and compute the gradient of the loss function, averaged over only the samples in the batch. We can view this as an estimate of what we would have gotten if we had averaged over the entire training set, although it may be a very noisy estimate. We perform a single step of gradient descent, then select a new batch of samples for the next step.
This algorithm tends to be much faster. The time required for each step depends only on the size of each batch, which can be quite small (often on the order of 100 sam‐ples) and is independent of the size of the training set. The disadvantage is that each step does a less good job of reducing the loss, because it is based on a noisy estimate of the gradient rather than the true gradient. Still, it leads to a much shorter training time overall.
Most optimization algorithms used in deep learning are based on SGD, but there are many variations that improve on it in different ways. Fortunately, you can usually treat these algorithms as black boxes and trust them to do the right thing without understanding all the details of how they work. Two of the most popular algorithms used today are called Adam and RMSProp. If you are in doubt about what algorithm to use, either one of those will probably be a reasonable choice.
Validation
Suppose you have done everything described so far. You collected a large set of train‐ing data. You selected a model, then ran a training algorithm until the loss became very small. Congratulations, you now have a function that solves your problem!
Right?
Sorry, it’s not that simple! All you really know for sure is that the function works well on the training data. You might hope it will also work well on other data, but you cer‐tainly can’t count on it. Now you need to validate the model to see whether it works on data that it hasn’t been specifically trained on.
To do this you need a second dataset, called the test set. It has exactly the same form as the training set, a collection of x,y pairs, but the two should have no samples in common. You train the model on the training set, then test it on the test set. This brings us to one of the most important principles in machine learning:
• You must not use the test set in any way while designing or training the model.
In fact, it is best if you never even look at the data in the test set. Test set data is only for testing the fully trained model to find out how well it works. If you allow the test set to influence the model in any way, you risk getting a model that works better on the test set than on other data that was not involved in creating the model. It ceases to be a true test set, and becomes just another type of training set.
This is connected to the mathematical concept of overfitting The training data is sup‐posed to be representative of a much larger data distribution, the set of all inputs you might ever want to use the model on. But you can’t train it on all possible inputs. You can only create a finite set of training samples, train the model on those, and hope it learns general strategies that work equally well on other samples. Overfitting is what happens when the training picks up on specific features of the training samples, such that the model works better on them than it does on other samples.
Regularization
Overfitting is a major problem for anyone who uses machine learning. Given that, you won’t be surprised to learn that lots of techniques have been developed for avoid‐ing it. These techniques are collectively known as regularization. The goal of any reg‐ularization technique is to avoid overfitting and produce a trained model that works well on any input, not just the particular inputs that were used for training.
Before we discuss particular regularization techniques, there are two very important points to understand about it.
First, the best way to avoid overfitting is almost always to get more training data. The bigger your training set, the better it represents the “true” data distribution, and the less likely the learning algorithm is to overfit. Of course, that is sometimes impossi‐ble: maybe you simply have no way to get more data, or the data may be very expen‐sive to collect. In that case, you just have to do the best you can with the data you have, and if overfitting is a problem, you will have to use regularization to avoid it. But more data will probably lead to a better result than regularization.
Second, there is no universally “best” way to do regularization. It all depends on the problem. After all, the training algorithm doesn’t know that it’s overfitting. All it knows about is the training data. It doesn’t know how the true data distribution dif‐fers from the training data, so the best it can do is produce a model that works well on the training set. If that isn’t what you want, it’s up to you to tell it.
That is the essence of any regularization method: biasing the training process to pre‐fer certain types of models over others. You make assumptions about what properties a “good” model should have, and how it differs from an overfit one, and then you tell the training algorithm to prefer models with those properties. Of course, those assumptions are often implicit rather than explicit. It may not be obvious what assumptions you are making by choosing a particular regularization method. But they are always there.
One of the simplest regularization methods is just to train the model for fewer steps. Early in training, it tends to pick up on coarse properties of the training data that likely apply to the true distribution. The longer it runs, the more likely it is to start picking up on fine details of particular training samples. By limiting the number of training steps, you give it less opportunity to overfit. More formally, you are really assuming that “good” parameter values should not be too different from whatever values you start training from.
Another method is to restrict the magnitude of the parameters in the model. For example, you might add a term to the loss function that is proportional to θ2, where θ is a vector containing all of the model’s parameters. By doing this, you are assuming that “good” parameter values should not be any larger than necessary. It reflects the fact that overfitting often (though not always) involves some parameters becoming very large.
A very popular method of regularization is called dropout. It involves doing some‐thing that at first seems ridiculous, but actually works surprisingly well. For each hid‐den layer in the model, you randomly select a subset of elements in the output vector hi and set them to 0. On every step of gradient descent, you pick a different random subset of elements. This might seem like it would just break the model: how can you expect it to work when internal calculations keep randomly getting set to 0? The mathematical theory for why dropout works is a bit complicated. Very roughly speak‐ing, by using dropout you are assuming that no individual calculation within the
Another random document with no related content on Scribd:
CHAPTER XIII
REIGN OF AL-HAKEM II.
961–976
Splendid Ceremonial at the Accession of Al-Hakem II His Wise and Prudent Measures—Ordoño seeks an Audience—His Baseness —Successful Expedition against the Christians—Disturbances in Africa—Army of the Khalif Defeated—The Berber Chieftains are corrupted, and their Forces disband Importance of Cordova as a Religious Centre Description of the Great Mosque Death of AlHakem His Literary Attainments His Patronage of Letters The Library Institutions of Learning General Prevalence of Education Public Improvements The Khalif the Exemplar of the Highest Culture of his Age Prosperity of the Empire.
At the death of Abd-al-Rahman III. the Hispano-Arab empire seemed, to all unfamiliar with the defects of the Moslem constitution, invulnerable to the attacks of foreign or domestic enemies. The wise dispositions of that accomplished ruler had, for a time, reconciled the differences arising from tribal antipathy and religious discord. The employment of mercenaries, constituting an army which could not be corrupted, and whose isolation from the seditious populace was the most effectual guaranty of its fidelity, apparently assured the perpetuity of a system which a profound and statesmanlike policy had established. The administration was directed by capable and experienced ministers. The public revenues far exceeded in amount those of the wealthiest contemporaneous nations. Obedient to the law of political attraction, which like gravity in the material world draws the weaker to the stronger power, neighboring kingdoms, although separated from the khalifate by the most powerful motives that can influence humanity—by the antagonism of race, by the prejudices of religion, by the memory of generations of incessant warfare, by hostile traditions which involved the loss of an empire and the subjection of its people—had acknowledged the supremacy of the Ommeyade princes. In the humiliating character of suppliants
for the favor of an hereditary foe, the sovereigns of Leon and Navarre had implored the aid of the infidel, and the former of these had regained possession of his dominions under a treaty which implied, if it did not actually express, conditions of vassalage reflecting little credit upon the successor of the haughty Visigoths. Every consideration which contributes to inspire the respect and admiration of mankind lent its assistance to exalt the fame and greatness of the court of Cordova. The most distinguished monarchs solicited the friendship of the Khalif. His capital was the literary centre of the Western world. The intellectual activity there displayed had never been equalled since the glorious days when Grecian genius immortalized the schools of Ionia and Attica. The Moslem fleets controlled the Mediterranean. The mechanical arts, the science of agriculture, the various branches of foreign commerce and domestic traffic, had, under a well-grounded feeling of public security, received a prodigious and unexampled impulse. It was, therefore, under the most happy auspices that Al-Hakem, at the age of forty-eight, with a character long considered the embodiment of all princely virtues, assumed the supreme direction of affairs.
On the day following the death of Abd-al-Rahman, the accession of his son was celebrated with greater splendor than had yet distinguished this important function since the foundation of the Western empire. Magnificently attired, the great officers of the khalifate attended to swear allegiance to the new sovereign. After the members of the royal family, the chief officials, the ministers of, state, the kadis, and the chamberlains had taken the oath, they administered it, in turn, to the host of retainers and subordinates of the royal household. The pomp displayed on this occasion was worthy of the most powerful, the most opulent, and the most luxurious monarchy in Europe. The ceremony was held under the central pavilion of the palace of Medina-al-Zahrâ, whose decorations, sparkling in the sunlight, seemed to realize the gorgeous dreams of Oriental enchantment. The eight brothers of AlHakem were escorted to the palace with all the honors due to their rank; but it could not escape the notice of those princes that the strong guard which surrounded them, and lined the entrance of their apartments, was less a manifestation of respect than a menacing
precaution against treason. The viziers, invested with their robes and insignia of office, and the nobles took up their positions behind the members of the Khalif’s family, who stood on either side of the throne. Beyond them, crowding the ample space of the rotunda and even filling the adjacent apartments, were the imperial magistrates, the generals of the army, the provincial governors, and the vast crowd of public functionaries summoned from the capital as well as from the neighboring cities of Andalusia to render homage to and assist at the inauguration of the Commander of the Faithful. The eunuchs, that class of monsters which has always sought indemnity for its degradation by the acquisition of power and wealth, which has never failed to revenge itself upon society by the ruin of the government which tolerated its institution, and whose preponderance in the political system of the khalifate had become portentous and appalling, were present by thousands. They were marshalled according to nationality, rank, and the nature of the service to which they were assigned. Most of them were clad in white tunics embroidered with gold, white being at the same time the distinctive color and the badge of mourning of the Ommeyades. The servitors of the harem were drawn up in the hall leading to the audiencechamber; they were both white and black; all were sheathed in shining armor inlaid with gold and silver, and the hilts and scabbards of their scimetars sparkled with costly gems. The marble terrace by which the pavilion was approached was guarded by Sclavonians splendidly mounted, whose arms and equipments were not inferior in richness to those of the magnificent array within. Outside, and reaching to the gate of the capital, were ranged, in exact and motionless order, the royal archers, the slaves, and the various divisions of the garrison.
The imposing ceremonial concluded, the remains of Abd-alRahman were committed to the tomb. The active mind of Al-Hakem was at once applied to the consideration of the details which concerned the administration of every department of his empire. The ministers of his father were confirmed, without exception. Officials were despatched to exact the allegiance of the walis of distant provinces. The troops were reviewed, and a general inspection of the army ordered. A hajib, or prime minister, was appointed, for the
tastes of Al-Hakem were inclined rather to the quiet and refining pursuits of literature than to official drudgery and the responsibilities incident to the administration of a great and turbulent monarchy. In this respect he ignored the oft-repeated counsels of his father, who, conscious of the vast power wielded by a hajib, and which, in the hands of an ambitious and unscrupulous statesman, might be attended with disastrous consequences, had endeavored to impress upon his heir the policy, and even the necessity, of retaining unimpaired the authority conferred by the exalted office of Commander of Believers. The individual selected for this important post was Giafar-al-Asklabi, a Slave, whose caste and nationality seemed to Al-Hakem a sufficient warrant for his good behavior, an opinion subsequently justified by the wisdom and tact he displayed in the discharge of his official duties. His appointment was followed, according to the custom of the court, by the delivery of a magnificent present to the Khalif, consisting, in this instance, of richly apparelled slaves, and arms and armor inlaid with gold.
The news of the death of Abd-al-Rahman had been received with pretended grief and secret delight by the sovereigns of Leon and Navarre. The prestige of the former Khalif, and his ability to enforce his demands, had been repeatedly demonstrated by the eventful transactions of his long and glorious reign. The blackened fields and dismantled castles of the frontier, the heaps of bones bleaching on many a battle-field, bore silent but conclusive testimony to his ruthless hostility and to the prowess of his armies. But the character of his successor was, as yet, undeveloped. A life already advanced beyond middle age had been distinguished by none of those martial deeds which elicit the applause of a subject and awaken the respect and the apprehensions of an enemy. The predilections of the new Khalif for a sedentary life, and his intimate relations with the learned, were viewed with contempt by the barbarous Christians, who considered war as the peculiar calling of a man of spirit, and the acquisition of knowledge as only fit for monks, an order whose pacific occupations did not, nevertheless, exclude even its members from the profession of arms. In those early times, while not as yet expressly sanctioned by the authority of the Papacy, the violation of an engagement made with the enemies of the Church was already
considered a meritorious action, to be governed rather by motives of expediency than by any considerations of morality and national honor. The treaty, by which Sancho had stipulated to deliver to the Khalif a certain number of fortresses in return for the restoration of his crown, had not been carried out by the King of Leon. The temptation to violate it, or at least to elude its performance, now became irresistible. To the demands of Al-Hakem, Sancho returned evasive and temporizing answers. The King of Navarre, doubtless acting in collusion with his neighbor, was even less tractable. Ferdinand Gonzalez, who had fallen into the hands of the latter, was held a close prisoner at Pampeluna, and the Khalif, who was anxious, at all hazards, to obtain possession of the person of this formidable adversary, promised to abate a portion of his demands in case Garcia would agree to send his illustrious captive to Cordova. The King of Navarre not only refused, but actually liberated the Count of Castile, with the understanding that he should expel his son-in-law, Ordoño IV., who had sought a refuge at Burgos, and should at once declare war against the Khalif. Ferdinand promptly fulfilled his engagements. The unfortunate Ordoño was torn from his family and ignominiously driven across the border; at his summons to arms the old warriors of the Count flocked to his standard, and the settlements of Estremadura and Andalusia, long accustomed to peace, once more experienced the deplorable evils of partisan hostility. Accompanied by only a score of retainers, who, of all his numerous retinue, were all that were willing to share his uncertain fortunes, Ordoño, journeying through the enemy’s country, reached the city of Medina-Celi. Here, as in all the empire, the streets resounded with the din of arms, for Al-Hakem, convinced that the respect of the Christians could only be preserved by a display of force, was everywhere making active preparations for war. The spirit of the royal fugitive rose at the inspiring sight, and, hoping that by proper management the military successes of Al-Hakem might be made to enure to his own advantage, he requested of the governor of the city a safe-conduct to Cordova and permission to place himself under the protection of the Khalif. His wish was readily acceded to, and a troop of cavalry was detailed to escort him to the capital. The route passing near the royal cemetery, Ordoño asked to
be shown the tomb of Abd-al-Rahman III. This having been done, he dismounted, uncovered his head, and, kneeling by the grave of the monarch who in life had been his most inveterate enemy, he prayed long and fervently for the repose and welfare of his soul. A few days afterwards he was received by the Khalif in the palace of Medina-alZahrâ. As a token of respect and an evidence of vassalage, he was required to assume the white robes of the Ommeyades. ObeydallahIbn-Kasim, Archbishop of Toledo, and Walid-Ibn-Khaizoran, Judge of the Christians of Cordova, whose Arabic names seem strangely at variance with the offices they administered, escorted him to the audience-chamber; their presence being necessary, both as interpreters and to afford information on points of etiquette rigorously exacted by the punctilious court of Cordova, and concerning which it was justly conjectured the former associations of a barbarian ruler had left him entirely unacquainted.
The introduction of the Christian prince was attended with the pomp ordinarily displayed at the reception of the greatest kings and their ambassadors. The body-guard of the Khalif was drawn up in all the panoply of gleaming weapons and costly armor. The officials were present in their robes of state. Around the throne were ranged the princes and the officers of the royal household. An innumerable army of subordinates filled the halls and lined the terraces. Through the gilded lattices of the harem, which overlooked the hall of audience, the sheen of jewels and the sparkle of bright eyes occasionally revealed the presence of the beauties whom Oriental jealousy only on rare occasions permitted to lend the lustre of their charms to an important ceremony.
The Leonese nobles and their king moved with downcast heads through the grim and lowering ranks of the soldiery, whose fierce looks so terrified them that they endeavored to fortify themselves by the recitation of prayers and frequent repetitions of the sign of the cross. After numerous genuflexions Ordoño approached the Khalif, who gave him his hand to kiss. He was conducted to a seat at some distance from the throne, and received from Al-Hakem the flattering assurance that even more favors than he desired would be accorded to him. It was then that the unspeakable baseness of Ordoño’s
character disclosed itself. He had frequently since his entrance into the Moorish dominions given evidence of a lack of royal dignity and of a total absence of those manly qualities which elicit sympathy for greatness in misfortune. He had courted the favor of the commander of the escort which had accompanied him to the capital, by flattery, by gifts, by the most ignoble concessions. With an hypocrisy that did not impose upon the keen-witted Andalusians, he had bowed in wellfeigned grief before the tomb of Abd-al-Rahman. With a sycophancy that disgusted the objects of it, he had fawned upon the officials of the court. But the time was now come when these disgraceful exhibitions of voluntary self-abasement were to be eclipsed, and the depth of his degradation attained. As soon as the speech of AlHakem had been translated to him, without making allowance for the courtly exaggeration it implied, he arose and said: “I am the slave of the Commander of the Faithful; I trust in his magnanimity; I seek my support in his lofty virtues; I give him full power over me and mine!” He then, in language worthy only of the meanest vassal, begged the aid of the Khalif in the recovery of his throne, and, in conclusion, contrasted his own voluntary submission with the conduct of his cousin Sancho, who had been compelled by Abd-al-Rahman to render him homage as a condition of the treaty. Al-Hakem listened with ill-concealed contempt to his harangue, promised that all his wishes should be fulfilled, and that he should be guaranteed. against molestation from his enemies. The interview terminated, the eunuchs escorted the Leonese from the pavilion. Ordoño displayed little less servility towards the Vizier Giafar than he had shown to his master, and was with difficulty prevented from kissing the hand of that dignitary. Having perceived in an antechamber a seat which he was informed was sometimes used by the Khalif, he grovelled before it with as much apparent reverence as if it had been a sacred reliquary of the most undoubted virtue.
A splendid palace was assigned as a residence to the King and his suite; they were all clothed with robes of honor,—a mark of the highest distinction among Orientals,—and a number of valuable presents sent from the court further assured them of the generous sympathy of their benefactor. In a few days a treaty was concluded, by which Ordoño pledged himself to maintain peace with Al-Hakem
and perpetual war with the Count of Castile. The flower of the Moorish troops, commanded by Ghalib, the ablest of the imperial generals, was ordered to make the campaign, and the Archbishop of Toledo and the Bishop and the Judge of Cordova were designated to accompany the army, not so much to assist the pusillanimous monarch by their counsels as to carefully note his behavior, and see that he was guilty of nothing that could contribute to the injury of the Khalif, or even remotely affect in an unfavorable manner the objects of his ambition.
In the mean time Sancho had taken the alarm. When he learned of the success of Ordoño, he realized that he had presumed too much upon the peaceful disposition and epicurean tastes of AlHakem. It was no secret that a powerful army was mustering to invade his dominions. By the artifices of the enemy, its numbers were purposely multiplied. His popularity had never been great, and his restoration had been accomplished by means abhorrent to a large number of his subjects. Many of his vassals could not be depended upon. The great province of Galicia, a fief of the crown of Leon, refused to acknowledge his title, and its count held language that indicated that he only sought a favorable opportunity to declare himself independent. Under the circumstances but one course was open to Sancho, and he sent an embassy to Cordova to state that he was willing to perform, without delay, the conditions of the treaty he had concluded with Abd-al-Rahman.
The duplicity of Al-Hakem, a defect happily rare in the annals of his race and which reflects such discredit upon his name, now became apparent. He violated, without compunction, the compact he had made with Ordoño. The latter, overcome with disappointment and mortification at the failure of his hopes, gave himself up to melancholy, and died the victim of his own abasement and credulity in the gilded prison which had been set apart as his abode in the environs of the Moslem capital.
His rival removed, and the alliance of Ferdinand Gonzalez assured, Sancho concluded that the occasion was most opportune for a further repudiation of his engagements. He accordingly defied the Khalif, and the latter at once proclaimed the Holy War. The army
of invasion was commanded by Ghalib. The Count of Castile was defeated in a great battle. The governor of Saragossa, Yahya-IbnMohammed, easily worsted the King of Navarre. The important fortresses of San Estevan de Gormaz and Calahorra, which had been stormed and destroyed, were rebuilt, provided with Moorish garrisons, and added to the dominions of the khalifate. The Arab army wasted the borders of Catalonia with fire and sword; for two counts of that principality, Miron and Borel, endowed with more hardihood than discretion, had been prevailed upon by the King of Navarre to join the confederacy, and were now condemned to expiate their breach of faith by the pillage of their cities and the misery of their subjects. Everywhere the arms of the Moslems were triumphant. The enemies who had confidently reckoned on the incapacity and inertness supposed to attach to a literary life, and the well-known aversion of Al-Hakem to the military profession, were compelled to acknowledge the wisdom of his dispositions and the energy with which he carried them into execution. One after another sued for peace. Even from distant Galicia came an embassy, headed by a noble matron, the mother of a count, who was entertained with the distinguished courtesy for which the court of Cordova was famous, and was dismissed with gifts whose splendor dazzled the eyes of a barbarian princess born and bred amidst the barren slopes and poverty-stricken hamlets of the Pyrenees.
This successful campaign closed the military operations of AlHakem in the North. He had taught his perfidious adversaries a severe but salutary lesson. Whatever inclination to renew hostilities they might have entertained, their own dissensions precluded its indulgence. War soon broke out between Galicia and Leon. The rebellious province, which aspired to independence, was in a fair way to be conquered, when a resort to treachery accomplished an end unattainable by the expedients of honorable warfare. Poison was administered to Sancho at a conference on the banks of the Douro. The King of Leon died after some days in excruciating agony; his son Ramiro, who succeeded him, was a child of five years, and the martial aristocracy declined to recognize the authority of an infant, controlled by his aunt, the nun Elvira, whose doubtful qualifications for the government of a kingdom had been acquired in
the solitude of a cloister Elsewhere, also, in the North, matters were propitious to the security and prosperity of the khalifate. A great army of Danes, who had served under the standard of the Duke of Normandy, poured down upon and devastated the plains and valleys of Galicia. Finally, the death of Ferdinand Gonzalez freed Al-Hakem from his most dangerous enemy, and during the remainder of his life the inroads of the Christians ceased to excite the alarm of his subjects or to disturb the peace of his empire.
While permanent safety had been secured in the North, on the side of Africa the danger was constant and menacing. The wild tribes of the Desert had never forgotten with what facility their forefathers had traversed the strait and subjugated a populous and extensive monarchy. The covetous eye of the half-naked Mauritanian robber, whose prowess had at times prevailed over the discipline of the Roman legions, was ever turned towards the beautiful cities of Andalusia, with their teeming bazaars, their prodigious wealth, their palaces furnished with every appliance of luxury, their lovely and fascinating women. The instinct of conquest, the presentiment that one day the exploit of Tarik would be repeated, were inspired by hope and encouraged by tradition. It was not the masses alone who cherished these ominous aspirations. The princes of the various dynasties who, at different periods of its history, swayed the destinies of Al-Maghreb, had, without exception, regarded the riches of the Peninsula as lawful spoil, if not actually as a part of their patrimony. Its condition—political, social, religious, commercial—was as familiar to them as the domestic polity of their own dominions. Their spies were to be found in the great emporiums of trade, in the most sequestered hamlets, in the ranks of the army, in the corridors of the palace, even in the bed-chamber of the khalif. Unknown and often unsuspected, their influence had been felt in many a bloody insurrection, in the decisive moment of many an eventful day. Never for a moment did they abandon the long-nourished project of conquest; never did they renounce the ambition—destined unhappily to be realized—of planting their victorious banners and erecting their throne on the banks of the famous Guadalquivir Since the assumption of the suzerainty of the African provinces bordering on the Mediterranean by Abd-al-Rahman, the maintenance of that
dignity had caused no inconsiderable drain on the treasury of the khalifate. Immense sums were annually transmitted to maintain troops, to support the pretensions of feeble vassals, and to bribe barbarian chieftains to refrain from ravaging the lands of their neighbors. No compensation was offered for the expense incurred except the negative and uncertain one implied by the temporary restraint of Berber invasion.
The attention of the Fatimites had been some years before directed towards the East, and, after a short and victorious struggle, the princes of that dynasty were enabled to remove the seat of their empire from the sandy plains of Mauritania to the inexhaustible Valley of the Nile. A great danger to the Ommeyade Empire of Spain was therefore apparently removed. At this time, Hassan-Ibn-Kenun, the last survivor of the Edrisites, exercised a precarious sovereignty over that portion of the African coast of which Tangier was the capital. A nominal vassal of Al-Hakem, his loyalty was largely dependent upon the fears excited by the encroachments of his neighbors, and when Abu-al-Fotuh, the representative of the Fatimites, invaded his dominions, the allegiance of Ibn-Kenun was, without hesitation, transferred to the Khalif of Egypt.
The revolt of Ibn-Kenun, and the alarming progress made by the Fatimite viceroy, impressed upon Al-Hakem the necessity for immediate and decisive action. With his customary diligence, he issued orders for the departure of a strong military and naval force to punish the treason of his vassal and overawe the fickle and perfidious chieftains of Africa. The object of the expedition was Tangier, the seat of the court and the residence of Ibn-Kenun. The Ommeyade fleet blockaded the harbor, and the troops, having encountered the enemy near the city, after a sharp engagement gained a decisive victory. But this success was of short duration. In the land of Al-Maghreb, swarming with active and warlike barbarians, the recruiting of an army was a matter of trifling difficulty. The Desert hordes, allured by the expectation of plunder and the excitement of arms, crowded to the camp of the Edrisite prince, who soon found himself once more able to tempt the fortunes of war. Another battle was fought; the troops of the Khalif sustained a demoralizing defeat;
their general, Ibn-Tomlos, was left dead on the field, and the survivors who escaped the spears of the Mauritanian cavalry sought security behind the battlements of Tangier. The effect produced by this victory on the venal and inconstant people of Africa was serious. The reputation and the power of Ibn-Kenun received an extraordinary impulse. The petty vassals who for years had enjoyed the bounty of the Khalif hastened to renounce their allegiance. From far and near, along the sandy highways towards the camp of IbnKenun, trooped the ferocious tribesmen whom no prince had yet been able to conciliate, and no government been able to civilize. No territory, except that occupied by a few fortified towns, remained loyal to Al-Hakem; even these places were in a state of siege; and it was evident that, unless energetic measures were taken to retrieve the disaster, the war-cry of the Berbers would soon be heard on the plains of Andalusia.
The Khalif was not unconscious of his danger. From every province of his dominions he summoned his bravest troops and his most experienced generals. The supreme command was entrusted to Ghalib, whose skill and valor had been signalized in the recent campaign against the Christians, and who was solemnly admonished, on peril of his life, to return victorious. It was not, however, to the uncertain event of battle that the Khalif unreservedly committed the destinies of his empire. The mercenary character of the Berber sheiks, always the partisans of him who bribed them most liberally, or who bribed them last, was what Al-Hakem depended upon, far more than upon either the tactics of his general or the courage of his soldiers. A great treasure was placed at the disposal of Ghalib, and he was instructed to spare no pains to detach from the army of Ibn-Kenun every chief of influence, without regard to the numbers of his following or the extravagance of his demands. In case he succeeded, he was ordered to conduct the family of IbnKenun to Cordova. Having landed in safety, Ghalib studiously avoided a general engagement. His advance was impeded by the flying squadrons of Ibn-Kenun, who used every artifice to bring on a battle; but the cautious Ommeyade general, knowing that the fate of his sovereign, as well as his own life, depended on the issue of a conflict, had decided to trust to the secret and more certain means of
corruption. Through the medium of trusty messengers, magnificent weapons, costly garments, and heaps of gold were clandestinely displayed before the greedy eyes of the Berber chieftains. Their constancy was not proof against this exhibition of wealth. They even competed for the infamous distinction of first deserting the standard of their commander; and, in a few days, Ibn-Kenun, abandoned by all but a handful of his old retainers, saw himself compelled to take refuge in a strong castle built on the summit of an isolated mountain called the Eagle’s Rock, where he had, in prudent anticipation of a reverse of fortune, already conveyed his harem and his treasures.
The drafts of Ghalib on the royal treasury excited the astonishment and consternation of the Khalif. Such enormous expenses had never before been incurred in the conduct of a campaign; and Al-Hakem, suspecting that all of the public money had not been used to corrupt the Berbers, and that much of it had been diverted into private hands, determined, for the purpose of investigation, to send an officer experienced in matters of finance and clothed with almost despotic authority, who should not be confined to the mere duties of treasurer, but should also exercise the high and responsible functions of a general and a councillor of state. The individual selected for this delicate mission was Ibn-abi-Amir, a name of both glorious and sinister associations, which now appears for the first time in the annals of the Hispano-Arab domination. He was accompanied by a select body of troops commanded by YahyaIbn-Mohammed, Viceroy of the Northern Frontier, whom Al-Hakem, still fearful of the issue of the African campaign, had sent to reinforce the army of Ghalib.
The siege of the rebel fortress was pushed with energy, but its defences were so formidable that four months elapsed before the garrison could be brought to terms. Then the most favorable conditions were granted; the personal safety of the soldiers was guaranteed and their property kept inviolate; and the main article of the instructions of Al-Hakem, touching the conveyance of the Edrisites to Cordova, was acceded to, though not without manifest reluctance, which, however, was of little moment under circumstances where protest and resistance were equally unavailing.
With the capture of the Eagle’s Rock terminated the campaign in Africa. The remaining Edrisite princes, who had seen with dismay the sudden disappearance of the host of Ibn-Kenun, lost no time in making terms with the representatives of the Khalif. The entire region of Mauritania enjoyed a profound but delusive peace. The Berbers had retired to their solitudes to enjoy the reward of their treason, and to watch for another opportunity to dispose of their services to the highest bidder. The Fatimites, content with their recent acquisitions, left the administration of their African possessions to a viceroy, and, fascinated with the attractions of their new home on the Nile, did not seem, for the moment, desirous of prosecuting any further schemes of imperial aggrandizement.
Ghalib, escorting his illustrious prisoners, more than seventy in number, now returned to Cordova. From the hour of his landing at Algeziras, his march assumed the appearance of a military triumph. The line of march was obstructed by the throngs that, attracted by the novelty of the spectacle, had assembled from far and near. Every town and hamlet through which the cavalcade passed rang with the exultant shouts of a vast and excited multitude. The rich attire of the captives; their noble and dignified bearing; the romance attaching to the story of the foundation of their house; their descent from the family of the Prophet, certainly remote and probably fictitious, provoked the curiosity, if it did not excite the sympathy, of the elated populace, who only saw in their humiliation another addition to the glory of their sovereign. When the capital was approached, the Khalif came forth to meet the guard and their prisoners. As soon as his presence was known, Ibn-Kenun dismounted, knelt before him, and kissed his hand. Ghalib and his officers were received with all the honors due to men who under arduous circumstances have achieved success; the princes were conducted to a fortified palace in the city, and their attendants distributed in different localities, where their safe-keeping could be assured; the body-guard of the chief, consisting of seven hundred warriors of approved valor, was incorporated into a division of the army, and Al-Hakem could now congratulate himself that his decision and energy had compelled the respect of his enemies and had procured for his dominions the benefits of universal peace.
The inconstancy of fortune did not, however, permit the Khalif long to enjoy the leisure and the satisfaction to be derived from the indulgence of his literary tastes and the triumph of his arms. His health, impaired by intense and constant application to study, broke down. An attack of apoplexy admonished him that he must renounce the cares of state, and the conduct of the government was henceforth committed to his vizier Moshafi. The latter, a veteran and accomplished statesman, signalized his advent to uncontrolled power by the institution of many radical and greatly needed reforms. The department of finance, which had become the seat of corruption, was reorganized and administered with prudence and economy. The discipline of the army was improved. The former viceroy of the northern frontier, the gallant Yahya-Ibn-Mohammed, was recalled from command in Africa and reinstated in his former office, a measure that insured the tranquillity of the Christian states, which had recently developed symptoms of agitation, in several instances culminating in acts of open and destructive hostility. The interests of the Khalif in Mauritania were confirmed by the appointment of two native princes whose fidelity could be depended upon; and, by this stroke of policy, the vizier was enabled to remove all the Arab troops, except the garrisons of a few cities, to points where their services would be far more advantageous to the government, and, at the same time, less expensive to the treasury. The entertainment of the Edrisites, who maintained a pomp little inferior to that of the royal family itself, was a source of constant perplexity and annoyance to the economical minister. At length, profiting by the discontent induced by a sedentary life and a vigilant espionage ill-disguised under an appearance of liberty, he succeeded in persuading IbnKenun to allow himself and his followers to be transported to Tunis, with the understanding that they should never again set foot on the soil of Mauritania. When the preparations for departure had been completed, the thrifty vizier partially indemnified the treasury for the expense caused by the involuntary guests of the Khalif by unceremoniously appropriating a large piece of ambergris, of immense value, which Ibn-Kenun considered the most precious object of all his possessions. After landing at Tunis, the exiles proceeded to Alexandria, where for many years they enjoyed the
hospitality and protection of the Fatimite Khalif. From this time the princes of the Edrisite family are no longer prominent in the revolutions of Northern Africa.
While Cordova was, by far, the most populous and magnificent city of Moorish Spain, it was indebted, in no small degree, for its superiority to the prestige derived from its position as a place of pilgrimage and the religious centre of the khalifate. Even in the eyes of the most implacable enemies of the Ommeyade dynasty, a sacred character invested the Western metropolis as the seat of one of the most famous shrines of Islam. Far more intense was the feeling of reverence in the minds of the devoted adherents of that dynasty. There was the throne of the Commander of the Faithful, in whose person the pious believer recognized not only the representative of a line of princes whose genius had added vast provinces to the Moslem empire, but also the venerated successor of the Arabian Prophet. From its gates had gone forth armies which had traversed the natural boundaries of the Peninsula, had occupied the fairest portion of France, and had repeatedly abased the pride of the scoffing infidel. Everywhere were visible significant tokens of Mussulman triumph and Christian humiliation. Its warehouses were filled with the plunder of churches. The inmates of monasteries were exposed by hundreds for sale in its markets. Its mosques had been raised by the labor of Christian captives. From the ceiling of the Djalma, the pride of all true believers, were suspended the bells of the Cathedral of Santiago, which even the vaunted power of the patron saint of the Asturias had been unable to save from the sacrilegious hand of the Moslem. No capital in Africa or Asia enjoyed a larger measure of military glory. No city in all the wide realm of Islam was so renowned for the munificence of its rulers, the wealth of its religious foundations, the learning and eloquence of its theologians, and the pomp of its worship.
On the right bank of the Guadalquivir stood the Great Mosque, in the eyes of the Mohammedans of Africa and Spain superior in sanctity to all other temples, save only the Kaaba. Founded by the first Abd-al-Rahman, fully as much from political motives as through religious zeal, it had been embellished by the wealth, the taste, the
rivalry, and the enthusiasm of nine generations of sovereigns, having at their command the resources of one of the richest and most flourishing countries of the globe. By its erection, the power of the Western Khalifate had been established upon a firm and enduring basis, and the permanent independence of its dynasty assured. The more or less intimate relations hitherto maintained with the Moslem empires of Asia were severed; the nominal allegiance due to the monarchs of Damascus and Bagdad, persistently asserted by peremptory edicts and occasionally conceded by the rendition of a precarious tribute, was forever renounced. The aims of the founder, dictated by a political sagacity savoring of almost superhuman wisdom, were finally realized; and among the subjects and tributaries of the House of Ommeyah, proscribed by the rulers of Arabia, the sanctuary of the Mosque of his capital usurped the place once occupied by the temple of Mecca, the scene of the humiliations, the perils, and the triumphs of the Prophet. Thus, by a masterstroke of policy, which appealed alike to the worldly ambition and the religious pride of the people, was consolidated the authority of a new and progressive race of kings. Its associations especially struck the imagination of the pious Mohammedan. It occupied the site of the principal church of Gothic Cordova, which, in its turn, had been built upon the ruins of a Pagan temple. The stones of its foundation had been transported from Narbonne. The earth in which they were embedded had been stained with infidel blood. Christian prisoners, chained together, had painfully borne these materials for a distance of two hundred leagues, and others had for generations labored upon its walls. The spoil of many a successful campaign had contributed to enrich its interior. On all sides, golden inscriptions and trophies of conquest attested the glory of the princes of Islam and the invincible prowess of their armies. The Great Mosque is not merely an epitome of architecture, wherein are disclosed adaptations of the artistic ideas of many widely separated nations,—the manifestations of a spirit which could appropriate and combine in exquisite harmony the columns of the Roman, the capitals of the African, the arch of the Syrian, the mosaic of the Byzantine, the battlements of the Persian,—it is an eloquent testimonial to the genius which could enlist the ordinarily baneful influence of
superstition in the cultivation of literature and the diffusion of knowledge. For this splendid temple played no unimportant part in the intellectual advancement of Mohammedan Spain, as well as in the civilization of barbarian Europe.
The multitudes of pilgrims and scholars who resorted to Cordova hastened, without delay, to pay their devotions at its shrine. The Arab recognized in the sweep of its arches, the graceful curves of the palm groves of Nejd and Yemen, mementos of the Desert immortalized by the conceptions of the architect, ever mindful of the life and habits of his Bedouin ancestry. The polished Syrian viewed with admiring rapture the rich stuccoes, whose complex and gorgeous patterns surpassed in beauty the brocades of Damascus and the decorations which covered the palaces of the monarchs of Asia. In the carvings of its lattices was to be traced the peculiar form of the Indian cross, a symbol whose origin is unknown to the most ancient tradition, and which appears sculptured upon the venerable altars of Ceylon and Hindustan. Even the emblem of a sect most obnoxious to Islam was appropriated, and, by a singular inconsistency, compelled to assist in the adornment of the most gorgeous mosque of the Moslem world. The cresting of the walls, originally painted scarlet, is typical of flame, and, brought from Persia, symbolized the faith of the Ghebers, the detested worshippers of fire. Thus were concentrated in this unique structure the ideas, the materials, the devices, the ornamentation of many epochs and of many races. Each visit to its hallowed precincts imparted fresh inspiration to the theologian, to the artist, to the poet, to the student, to the antiquary. The reverence it claimed as a seat of pilgrimage invested its shrine with attributes possessed by none of the famous oracles of antiquity, and shared by few of the fanes of any contemporaneous religious faith. It was, moreover, justly regarded as the peculiar creation of a people whom its erection had greatly contributed to form and amalgamate, and who were entitled to credit for the admiration which its magnificence and its beauty elicited. Every city and province of the empire had contributed to the pious undertaking. Cordova paid the army of laborers employed. Merida furnished columns and other materials, ready for the mason, from the temples and the amphitheatre which had embellished the
seat of Roman power in ancient Lusitania. From the quarries of Almeria and Granada came great quantities of jasper, marble, and alabaster. From the forests of the Sierras was obtained the larch for the ceiling, whose remarkable preservation in buildings not subjected to the destructive consequences of ecclesiastical avarice attest its extraordinary exemption from the attacks of insects and the ravages of decay. The princes of Mauritania and the Byzantine vassals or allies of the khalifs, prompted by feelings of piety or friendship, bestowed upon the rising temple the most valuable relics of ancient art to be found in their dominions. A fifth of the spoils of battle—in a single instance amounting to the sum of forty-five thousand pieces of gold—was appropriated to defray the enormous expense which, notwithstanding the drafts on the treasury and the generous donations of the people, was constantly increasing. In the successive enlargements of the building demanded by the growing population, the owners of adjacent property, the purchase of which became indispensable, were rewarded for the sacrifice of their homes with unstinted generosity. Arab estimates have placed the entire cost of the Djalma—whose construction and alterations embraced, from first to last, a period of more than two centuries, under nine princes of the House of Ommeyah—at fifteen million pieces of gold. The Mosque, as completed, comprised an area of six hundred and twenty by four hundred and forty feet, running with the cardinal points of the compass. About one-third of this enclosure was occupied by a spacious court surrounded by arcades, planted with oranges, pomegranates, and palms, and refreshed with the spray of many fountains. The walls, thirty feet high on the northern side, increased in altitude with the approach to the river—the land rapidly descending in that direction—until they rose to the commanding height of seventy feet above the banks of the Guadalquivir. The roof was protected by plates of lead nearly an inch in thickness, whose sale in subsequent times yielded a magnificent sum to priestly depredators. The building, massive and imposing in its exterior, presented a strong resemblance to a fortress, a resemblance not inappropriate when the martial traditions of the religion to which it was dedicated are recalled. Immense buttresses, necessitated by the weight of the walls and the pressure of the arcades, were placed
at frequent intervals, like flanking towers in the defences of a citadel. The summits of wall and tower were fringed with battlements. Access was obtained to the interior by means of twenty-one horseshoe archways, three of which opened into the court-yard, and nine on the east and west sides respectively, three, in all, being reserved for the especial use of women. These archways were decorated with terra-cotta mosaics in red and yellow, relieved by inscriptions in gold on a ground of blue and scarlet. The doors were covered with plates of burnished brass, and provided with rings and knockers of huge dimensions and curious workmanship.
No church in Christendom could offer to the eyes of the worshipper such a scene of beauty as that enjoyed by the Moslem as he passed from the thronged and dusty streets of the city into the spacious Court of the Oranges. The latter bore the fascinating and voluptuous aspect of a tropical garden. The atmosphere was fragrant with the perfume of orange, rose, and jasmine. The foliage of the palm, recalling the famous groves of Medina and transporting the pilgrim in imagination to scenes in the distant Orient, rose majestically above the smaller but not less attractive orange-trees, with their glossy leaves, golden fruit, and snowy blossoms. Exquisite flowers, arranged in beds of fantastic patterns, bloomed along the borders of the arcades. Four great basins, each a monolith, supplied the water for the ceremonial lustration enjoined by the law of Islam. From the fountains the vast throng, clad in white robes, moved silently towards the temple and into the doors, which, looking upon the court, were closed by curtains of stamped and gilded leather Within, the eye was bewildered by the forest of columns,—more than fourteen hundred in number,—stretching far away to an apparently interminable distance. They were destitute of bases, and their capitals were entirely covered with gold. Above, tiers of double arches, in red and white, sustained the ceiling glittering with arabesques entwined with texts from the Koran. The divisions of the latter were formed by medallions oval, hexagonal, and circular, bearing a general resemblance to each other, yet widely differing in distribution of colors and details of ornamentation. The floor was composed of many-colored marble, arranged in designs of simple but pleasing character. Lattices of alabaster, carved in patterns no
