Speech Synthesis Presentation - A short Introduction and Historical Review (19.10.2021)

Hello. In this short presentation I would like to introduce the topic of “speech synthesis”. The slides are in German. However, I am holding this talk in English, because I am a native English speaking speech synthesiser! In fact, I am the standard Apple IOS13 reader. First, an overview: After some usecases, definitions and historical background, I will touch on the challenges involved in transforming written thoughts to speech. Then, I will describe the basic architecture of various systems and compare methods. The talk will conclude with ideas on infusing artificial speech with emotion and a singing voice emulation culled from the Internet.

What is speech synthesis? Essentially, it is the imitation of natural speech by artificial means. It requires a functional model of the human voice organ, be it mechanical, analogue or digital. Target groups include blind people, who can have a text read out to them, or people with a speech impediment, who can learn to express themselves through an artificial voice. Other speech applications are for situations, in which full visual acuteness is required of the able-bodied human, such as when simultaneously driving a car and navigating. In addition, speech synthesis and recognition are increasingly employed in human-robot interactions, as info-tainment and as art.

Many of these uses are non-trivial, since misunderstandings can have real negative social or even life threatening consequences. Insight is required into the nature of human social interaction, in addition to access to appropriate technical resources.

Over the last few years, the market for speech synthesis has boomed thanks to leaps in affordable computing power. We have become accustomed to hearing a computer voice when we call up a helpline, when we listen to announcements in public spaces, or interact with our personal digital assistants, home automation, media and navigation devices. We also increasingly make use of speech-synthesis for e-learning, in automated simultaneous translation and in dialogue systems that do not require a display device. Ultimately, we as customers and consumers are the judge of how successful a particular implementation is.

At this point, I want to distinguish between text-to-speech, or TTS, and concept-to-speech, or CTS. The former type of system is used when pre-composed text, usually originated by humans, is the input. A TTS system needs to apply correct pronunciation and enunciation to the script. Traditionally, this requires a two-step approach: the linguistic meaning of the input text is interpreted before it is rendered by a synthetic voice. Naturally, the result significantly depends on the weakest link in the component chain: errors early in the pipeline ripple through the system uncorrected and are amplified, usually making them easy for a native speaker to detect. In contrast, a concept-to-speech or CTS system directly triggers a pre-recorded voice signal based on a generation component that selects the expression based on semantic, pragmatic and discourse-based knowledge. This approach has traditionally been employed in information systems for acoustic feedback or as the output component of an automatic program. In such cases, errors can initially be hard to detect based on the acoustic signature - even when the phrase memory is actually quite limited.

A few words on the historic development of speech synthesis. In 1003, Gerbert Aurrilac built a bronze “talking head” that could respond with “yes” or “no” to a question. After a gap of several centuries, Christian Katzenstein built the first artificial speech organ in 1779. Only a few years later, Wolfgang von Kempelen built a similar device, albeit of different mechanical construction. In the 1930s, Bell Labs of the US developed the keyboard-controlled Vocoder, i.e. a device that emulates the human voice electronically. In the 1950s they presented the first true speech synthesiser based on an IBM 704 computer. Here we see the mechanical implementation of a modern voice organ, presented in 2011 by Prof. Sawada at Waseda university in Tokyo, Japan. We see the articulated mouth tract and imitation of a nose. Air is supplied by means of a blower. Using artificial intelligence,

the device can learn to imitate sounds previously unknown to it. However, the absence of teeth limits the range of expression to vowel sounds.

Let us turn to the principal challenges of speech synthesis. First, verbal and written forms of expression share a common intent: to convey information. However: we write differently from how we speak. Human dialogue is spontaneous, betrayed by emotions, tends to “jump” and have omissions. We use body language which our partners interpret and mirror back, interactively influencing the natural flow of a conversation.

Here is a German example of this, together with the linear representation of the acoustic signal based on fundamental frequency against time: “Der Mai ist ja noch ein – hat ja noch ein bisschen Zeit und ,, denke, dass wir den April vorziehen sollten.” And here is an example of a written German sentence: “Theresa May konnte für den sog. “Brexit” im Parlament keine Mehrheit gewinnen – trotz der Volksabstimmung vom 23.6.2016 in der 51,9% der Wähler für den Austritt Großbritannies aus der E.U. stimmten.” Truth be told – as in the footnote to the graphic - only 37.4% of those eligible chose to vote in the referendum. Moreover, fundamental constitutional changes usually require a two-thirds and not just a simple majority, at least in Parliament.

Now for some other common challenges. Take the English sentence: “The nights that we met you were awful”. It is evident from the presentation of the sentence on the page that we are talking about a bad lover. Without the visual rendering, we might have misunderstood that it was just the weather that was awful whenever they met. Another example: “He makes a record of everything they record.” Are we talking about an artist who insists on vinyl or a conscientious person taking notes? In this case, the pronunciation is non-consequential. But not here: “She became number as she waited in room number 101.” Clearly, “number” - as a feeling – is pronounced differently from the mathematical figure.

We can ask questions to ascertain whether the pre-requisites for effective machine communication are in place. Questions such as: “Do we have sufficient contextual information for the correct lingual rendering of the statement?”. And: “What is the appropriate choice of words, given the relationship between speaker and addressee?” And: “What are the listeners expectations towards the sound of the synthetic human voice?”

A key challenge in natural language processing is the severely skewed probability distribution for the occurrence of words in sentences, as well as of syllables or morphemes in a word. A limited number feature often - or even very often - BUT there is also a high probability that a particular, rare one occurs. So, unfortunately, we require quite a large lexicon to include the many rare occurrences, in order to sound intelligent. A phonetic lexicon includes the means to convert from graphem to phonem. By phonem we mean the sound of word syllables, or more exactly: of mora. The graphem is the corresponding written rendering accepted by convention. The phonetic lexicon needs to include exceptions to the normal rules of pronunciation as well as the multiple meanings and pronunciations of homonyms.

Here is a list of challenges in linguistic processing. We need syntactic analysis to disambiguate the structure of the sentence or phrase. We need a means to interpret abbreviations, numbers, mathematical formulae, various data expressions, given names etc. We also need to provide for phonetic exception handling of words that are not in the lexicon, perhaps because they come from a different language. As noted already, the machine needs to know how to correctly deal with homonyms. And – last but by no means least in this non-exhaustive list – the machine needs a means to interpret context in order to apply a natural-sounding syllable duration.

So how can we build a text-to-speech system? Simple traditional architecture is based on a threeblock pipeline: the input text is subjected to linguistic analysis, prosody is determined, and the voice synthesised. By linguistic analysis we mean the semiotic interpretation and normalisation of the text.

Prosody refers to speech melody and rhythm, or, more specifically, pitch intonation, phrasing and accentuation. Acoustic synthesis adds the timbre, the sound quality or colour of the artificial voice.

This slide presents the “common form model” of speech synthesis developed by Taylor. First, the ASCII input text is broken down into sentences. The sentences are transformed into token sequences – a process we already referred to as “text normalisation”. A token might be a word, but could also be a number or a date, for example. Next, the semiotic class of each token is determined. Here, special notation might be employed, as required. The software uses algorithms to determine phrasing, accentuation and intonation for encoding the graphemes as phonemes. These are then matched against a sample database of human voice recordings. The fragments are synthesised to form a continuous sound output signal.

Let us listen to some different types of acoustic synthesis, as taken from the historical database of Felix Burkhardt from the university of Cologne, some years ago. We will hear the same German sentence “An den Wochenenden bin ich jetzt immer nach Hause gefahren und habe Agnes besucht.

Dabei war eigentlich immer sehr schönes Wetter gewesen.” First, formant synthesis, that I will explain in more detail in a moment: Next, two different forms of concatenated synthesis. We start with diphon synthesis: And now, unit selection synthesis: What do you think? What synthesis method sounded most natural? Not featured in the Cologne database is a fourth, more recently popular form of synthesis using so-called Hidden-Markov Models, for short, HMM Synthesis, which will be explained later.

I also skipped over articulative synthesis. There is an inspiring TED Talk that explains the medical applications of articulative synthesis for digitally modelling the sound of the human voice tract based solely on explicitly measurable physical parameters. People who have lost the ability to speak can in this way regain an anatomically correct individual electronic voice.

The starting point for electronic speech synthesis is the analysis of samples of natural human speech.

Here we see an example of how a sentence might be chunked, or cut into phonemes, and matched against morphemes. We see several representations vs time: an oscillogram of amplitude or sound energy, a spectrogram of the frequency distribution and a decoding of the fundamental pitch track.

This second slide provides insight into how phonemes can be uniquely detected and synthesised.

Essentially, disambiguation requires analysis of the relative pitch and amplitude of the 1 st, 2nd and 3rd formants over the fundamental pitch. The left graph gives an idea of the natural variability of first and second formant pitch for just a few phoneme transcriptions – a key challenge in speech synthesis using this method.

What are the pros and cons of various synthesis techniques? We have already seen that formant synthesis is an additive method based on an acoustic model. Since fundamental pitch, amplitude and voicing are separately modulated, intonation and thus emotions can be programmed. Formant synthesis is economical in terms of memory and CPU requirements and has been traditionally used in embedded systems. Unfortunately, whilst pronunciation is exact and suffers no sonic glitches, this synthesis method results in an unnatural sounding, metallic robot voice.

Diphone synthesis uses building blocks of speech that begin in the middle of one phonem and end in the middle of the next phonem. Hence, diphons include the transition between two sounds. A speech database for the German language based on diphone synthesis requires approximately 2500 diphones. Exactly one example of each diphone is stored to ensure a unique match. Prosody is applied to the diphones by means of digital signal processing, or DSP for short. Whilst the memory requirements of this technique are modest and the standard database is freely available, diphone synthesis suffers from sonic glitches whilst still sounding robotic.

Unit selection synthesis is a more memory intensive form of speech synthesis. It is based on a large database of segmented speech recordings that have been force-aligned and indexed against acoustic parameters such as pitch, sound length, position in a syllable and neighbouring phonemes. The program dynamically selects the most appropriate unit sample based on weighted decision tree algorithms with error correction to minimise unnatural selections. The advantages of this type of synthesis are its good sound quality with minimal DSP. The database can be tuned for specific applications to maximise realism.

Now let’s briefly look at a different approach based on a mathematical concept – the aforementioned Hidden Markov Model, or HMM, synthesis. Importantly, voice, melody and prosody are simultaneously determined based on statistical concepts of maximum likelihood. The downside to the quite natural sounding voice is a higher CPU load.

I said at the outset that I would conclude this talk with a brief look at how emotion can be injected into synthetic speech. This might increase peoples’ acceptance of synthetic voices in real-world applications and have a greater effect on human behaviour. We know that communication is more than just the transmission of semantic content – we betray anger, sympathy or apathy by the tone of our voice. The appropriate acoustic correlation of emotions has been the focus of scientific study for over 75 years. But only now that we have reached a certain realism in basic voice modelling and have increased computational resources at our disposal has real progress in emotional speech emulation become a feasible proposition. We already saw that natural speech has variability and is imbued by irregularities. Situational information needs to be included and considered in the speech synthesis process. There is no such thing as neutral speech, and setting the wrong tone can lead to undesirable effects. Ideally, tone and tempo should evolve over longer text passages to keep grabbing the attention of the listener, as well as adjust with the complexity of the content. As in the case of a real human reader, a holistic understanding of the text and of the audience is needed.

In this context, I could point to cultural differences in the use of emotional tone. For example, the Japanese language differs from German and English by exhibiting much-increased congruence of graphem and morphem. Japanese is essentially spoken as it is written in kana, with a far smaller number of mora and a more exacting tempo. The prosody is primarily word, not sentence-bound and there are few inflections of stem forms. Intimacy in the Japanese language appears to be more bound to the choice of words and indeed use of pauses and omissions. Is Japanese therefore more straightforward to synthesise because one can dispense with emotional modelling? Or is emotion conveyed differently in the Japanese voice and thus needs a different technical implementation?

Without dwelling on these questions, I want to end by plugging a commercial software package called Vocaloid from a well-known Japanese manufacturer. This can be used to replace a human singer on an anime recording. Currently, the program is distributed in its 4 th reincarnation. I recommend viewing the explanation video. I have included some references and further links on the topics touched upon in this talk. Thanks for listening, and have fun!