Audio Onset Detection: A Brief History and Current Techniques

AUDIO ONSET DETECTION: A BRIEF HISTORY AND CURRENT TECHNIQUES

Rachel Locke Literature Review January 2021

Table of Contents I.

INTRODUCTION ....................................................................................................................................... 3

II.

LITERATURE REVIEW ........................................................................................................................... 4

III.

CONCLUSION ....................................................................................................................................... 6

IV.

REFERENCES........................................................................................................................................ 8

V.

APPENDIX................................................................................................................................................. 10

List of Figures Figure 1. Onset on the Envelope of an Audio Signal Figure 2. Stages of an Onset Detection Algorithm Figure 3. Time-Frequency Analysis via the Short-Time Fourier Transform

Abstract — Audio onset detection is a requirement for a wide range of Music Information Retrieval processes. This report presents a recent history of onset detection techniques and the advancements that led to the current state-of-the-art algorithms. Some background is given on the significant characteristics of generalised onset detection algorithms. Methods that have performed highly over the past fifteen years are introduced and discussed with a key focus being algorithms that achieved first place at the Music Information Retrieval Evaluation Exchange (MIREX 2005 – 2018) in the onset detection category. Spectral techniques and the emergence of deep learning methodology for onset detection is reviewed in greater detail.

I.

INTRODUCTION

At first glance onset detection appears to be a straightforward task: identify the beginning of notes or new musical events, specifically the earliest point in which a transient can be detected (fig 1). Though, in reality, there are varying types of onsets and the literature shows that a challenge of this research area has been developing an algorithm that produces satisfactory results for all onset types. This paper traces the development of onset detection algorithms, specifically focusing on the dominant techniques of the past fifteen years. Onset detection is a sub-category within the wider active research area of Music Information Retrieval (MIR) and is at the heart of our understanding of beattracking, tempo estimation, and automatic music transcription (Chen, Jang, Liu, & Weng, 2016). This time scale is significant to the research as it follows the top-performing submissions of the Music Information Retrieval Evaluation eXchange (MIREX) in the Audio Onset Detection Category from the time when it was founded in 2005 (Appendix I). In the same year (Bello et al., 2005) published a seminal text ‘A Tutorial on Onset Detection in Musical Signals’, which is referenced to this day in the majority of literature pertaining to onset detection. Over the past decade, deep learning techniques have become the standard for many MIR workflows and considerable literature has grown around the use of Neural Networks for onset detection algorithms (Schindler, Lidy, & Böck, 2020).

Figure 1. Onset on the Envelope of an Audio Signal (Mottaghi, Behdin, Esmaeili, Heydari, & Marvasti, 2017) There are three stages to a generalised onset detection algorithm. These are: 1. Pre-processing A first and optional step involving the transformation of the original signal into a format more appropriate for onset detection (Lindqvist, 2019). A conventional choice would be performing the Short-Time Fourier Transform (STFT) on the original signal in order to access its spectral properties. 2. Reduction Reduction describes the process of converting the signal into a downsampled ‘detection function’ in order for the transient incidents to be more easily expressed and detected (Bello et al., 2005). As is common in the literature, this paper focuses mainly on the reduction stage as it is the core of all onset detection algorithms. In some literature, this may be referred to as a ‘novelty function’ (Eyben, Böck, Schuller, & Graves, 2010). 3. Peak Picking Peak Picking is the final step in the onset detection process. It is essentially identifying the peaks in the detection function in order to locate the onsets (Bello et al., 2005). If a suitable method has been used to achieve the detection function these maxima (i.e., peaks) should be easily recognised (fig 2).

Figure 2. Stages of an Onset Detection Algorithm (Bello et al., 2005) Onsets are often categorised into two types. These are: 1) Hard onsets i.e., drums or percussion that exhibit sudden increases in energy 2) Soft onsets i.e., strings that exhibit a gradual change in energy over time. II.

LITERATURE REVIEW

Early detection functions were based on observing the signal in the time-domain and recording fluctuations in the amplitude envelope. This often-involved low pass filtering to reduce the effect of noise (Bello et al., 2005). Though important to acknowledge the methods based on time-domain signal features, these are not discussed further in this paper. Instead, spectral-based and deep learning-based methods are examined as these methods better reflect the trends of high-performing algorithms within the timeframe of interest (Böck, Arzt, Krebs, & Schedl, 2012; Eck, Douglas & Lacoste, 2007; Eyben et al., 2010; Roebel, 2005; Roebel, 2009). The majority of spectral onset detection algorithms discussed in this paper are based on the ShortTime Fourier transform (STFT). The STFT is used in order to analyse the signal’s spectral properties at specific time frames (Müller, 2016). A moving window function (often a Hamming window) multiplies the signal at various points in time and an FFT is performed on each of the windowed frames. This technique reveals which frequencies are present and at what times (fig 3).

Figure 3. Time-Frequency Analysis via the Short-Time Fourier Transform (Gao & Yan, 2006) When observing the signal in the time-frequency domain, increases in energy across frequency bins suggest the presence of an onset. This is known as energy-based onset detection and has shown to be highly effective for non-pitched percussive or ‘hard’ onsets (Reiss & Zhou, 2010). Reiss and Zhou (2007) presented a new technique that combined energy-based and pitch-based detection that received the best overall score in the MIREX 2007 onset detection category. The algorithm made a classification between soft and hard onset types and applied a pitch-based method for soft onsets and an energy-based method for hard onsets (Reiss & Zhou, 2007). When the sum of energy across frequency bins is taken, only the positive flux gives a good indication of onsets. The energy from a previous note will often decrease as energy for the following note ascends. If all changes in energy were considered there is potential for changes in total energy to be masked at points when positive and negative energy crossover (Reiss & Zhou, 2010). The sum of positive energy across frequency bins is referred to as ‘spectral flux’. In 2010, Eyben et al described methods that had harnessed spectral flux as “some of the best performing so far” (Eyben et al., 2010). Onset detection methods involving spectral flux have been shown to perform well on both hard and soft onsets (Böck & Widmer, 2013). For this reason, spectral methods offer an alternative to the algorithm suggested by Bello and colleagues whereby, separate methods were used for soft (pitch-based) and hard (energy-based) onsets (Reiss & Zhou, 2007). Pitch-based methods also rely on the time-frequency decomposition of the signal, whereby the energy changes of two consecutive notes with different pitches will fall into separate frequency channels (Dixon, 2006; Reiss & Zhou, 2007). Whilst energy-based methods make use of the magnitude information in the time-frequency domain, phase-based methods consider the phase information across frequency bins. In this case, it can be assumed that a new musical event will have a different phase to a previous event. Phase-based methods have been shown to perform better than energy-based methods when detecting soft onsets (Dixon, 2006). In short, energy increases and phase irregularities in the frequency domain are good indicators of onsets. In 2007, Eck and Lactose commented that there were few bodies of work combining onset detection and machine learning at the time. The improvement they presented involved feeding the frames of a spectrogram as inputs to a feed-forward neural network (FNN) in order to classify onsets and nononsets. A single-net and a multi-net approach were presented and submitted as entries to the very first onset detection algorithm contest at MIREX 2005. The multi-net variation performed best in the category winning them first place (Eck & Lacoste, 2007). In the ‘further work’ section of the paper accompanying this entry, Convolutional Neural Networks (CNN) are suggested as an improvement upon their work.

2010 saw a paradigm shift for onset detection methods. Inspired by the work of Lacoste and Eck, Böck et al presented an algorithm for onset detection based on a Bidirectional Recurrent Neural Network (BRNN) with Long Short-Term Memory (LSTM). This new development defined the then state-of-the-art for onset detection algorithms. Eyben et al publicised that their novel, data-driven methodology showed superior performance and precision for both pitched and percussive onsets (2010). RNNs have a looping mechanism in the hidden layer, called the hidden state, which makes them great for capturing sequential data like signals or time-series and making predictions (Phi, 2018). A weakness of RNNs is that they suffer from short term memory due to the vanishing gradient effect (Phi, 2018). This problem was solved for onset detection in this case by introducing Long Short-Term Memory to the process, LSTM functions similarly to RNN but they are capable of learning long term dependencies (Phi, 2018). Neural Networks had originally been explored for onset detection in 2002 when Marlot et al used a Multi-Layer Perceptron (MLP) on top of a spectrogram but the study was limited to the onsets of piano music (Marolt & Kavčič, 2003). Another historically significant breakthrough took place in the same year when Long Short-Term Memory (LSTM) was first applied to Music Information Retrieval for the detection of temporal structure in Blues improvisation (Eck, D. & Schmidhuber, 2002). Modern, datadriven onset detection algorithms still use spectral representations of the signal, but instead of calculating spectral flux manually, a spectrogram is fed to a Neural Network (Salamon & Bello, 2017). Inspired by the use of CNN for edge detection in image processing, Böck et al were the first to realise the use of CNN for onset detection (Schlüter & Böck, 2013). When comparing their previous use of RNN for onset detection (Eyben et al., 2010) with the CNN method, a trade-off between pre-processing and computational load was identified, the methods performed similarly overall, however, the CNN required less pre-processing with the trade-off of higher computational power (Schlüter & Böck, 2013). In 2014 a study by Böck and Schlüter saw that CNN yielded improved results when compared to the then state-of-the-art onset detection techniques and superseded their previous use of RNN and CNN for the same task. The then state-of-the-art was ‘OnsetDetector’ also the work of Böck et al, delivered in the 2011 and 2012 MIREX Onset Detection algorithm awards, performing best both years, Böck would go on to author or co-author the winning submissions to the present day. Böck and Schlüter began their study by stating that up until that point, onset detection had been only partially solved for polyphonic signals (Schluter & Bock, 2014). Their implementation of CNN for onset detection was able to overcome this long-standing weakness as CNN are able to distinguish between two musical events or onsets when they are masking each other in time and/or frequency (Salamon & Bello, 2017). Similarly, to their previous data-driven methodology this CNN was trained on a spectrogram. The grid-like representation of frequency as a function of time is ideal for a CNN which efficiently handles spectral, temporal data (Schluter & Bock, 2014).

III.

CONCLUSION

Looking at the literature of the past decade it emerges that Böck has been the most prominent researcher in the field of onset detection and along with Schlüter and Eyben has really championed deep learning for onset detection, consistently defining or contributing to the state-of-the-art algorithms (Bello et al., 2005; Eyben et al., 2010; Schlüter & Böck, 2013). The deep-learning-based approaches discussed in this paper have seemingly managed to overcome the main complications inherent to onset detection that researchers have faced time and again. These are: § § §

Soft onsets with slow transients where the onset is harder to detect Vague or quiet onsets Onsets in polyphonic music

The CNN based onset detection algorithm ‘CNNOnsetDetector’ that Böck and Schlüter presented at MIREX 2018 continues to hold the title as the best performing algorithm for onset detection (Roebel, Jacques, & Aknin, 2018). However, the results have been shown to be impossible to replicate thus far.

Reproduction of the results was attempted most recently by Björn Lindqvist (Lindqvist, 2019). They concluded that they tried and failed to replicate the results achieved by Böck and Schlüter. Gong and Serra (2018) were also unable to reproduce these results. This raises the question of how useful an algorithm is in the wider context if it is not able to be reproduced to the same quality. Lindqvist (2019) proposes raw waveforms as an alternative to spectrograms for training neural networks and suggests that this might improve the performance. Over the past two years, MIR papers suggesting transformers as an improvement upon LSTM based methods have been emerging (Agrawal, Ravi Shanker, & Alluri, 2021; Park, Choi, Jeon, Kim, & Park, 2019). Transformers could potentially be the next step for onset detection algorithms.

IV.

REFERENCES

Agrawal, Y., Ravi Shanker, R. G. and Alluri, V. (2021) ‘Transformer-based Approach Towards Music Emotion Recognition from Lyrics’. Bello, J. P. et al. (2005) ‘A tutorial on onset detection in music signals’, 13(5), pp. 1035–1047. doi: 10.1109/TSA.2005.851998. Böck, S., Krebs, F. and Schedl, M. (2012) ‘Evaluating the Online Capabilities of Onset Detection Methods’. ISMIR. doi: 10.5281/zenodo.1416035. Böck, S. and Widmer, G. (2013) ‘Maximum Filter Vibrato Suppression for Onset Detection’. Dixon, S. (2006a) ‘Onset Detection Revisited’. Dixon, S. (2006b) ‘Simple Spectrum-Based Onset Detection’. Eck, D. and Lacoste, A. (2007) ‘A Supervised Classification Algorithm for Note Onset Detection’. Eck, D. and Schmidhuber, J. (2002) ‘Finding temporal structure in music: blues improvisation with LSTM recurrent networks’, in. IEEE (Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing), pp. 747–756. doi: 10.1109/NNSP.2002.1030094. Eyben, F. et al. (2010) ‘Universal Onset Detection with Bidirectional Long-Short Term Memory Neural Networks’. (11th International Society for Music Information Retrieval Conference). Available at: https://explore.openaire.eu/search/publication?articleId=od_______518::efb4c46cd113df0d8d82e393 eed27bd4. Gao, R. . and Yan, R. (2006) ‘Non-stationary signal processing for bearing health monitoring’, 1(1). doi: 10.1504/IJMR.2006.010701. Gong, R. and Serra, X. (2018) ‘Towards an efficient deep learning model for musical onset detection’. Available at: https://arxiv.org/abs/1806.06773. Lindqvist, B. (2019) Reproducing the State of the Art in Onset Detection Using Neural Networks. KTH, Skolan för elektroteknik och datavetenskap (EECS). Available at: http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-255267. Marolt, M. and Kavčič, A. (2003) ‘Neural Networks for Note Onset Detection in Piano Music ’. Mottaghi, A. et al. (2017) ‘OBTAIN: Real-Time Beat Tracking in Audio Signals’, 5(4). doi: 10.18178/ijsps.5.4.123-129. Müller, M. (2016) Fundamentals of Music Processing : Audio, Analysis, Algorithms, Applications. Springer International Publishing, pp. 303–310. Available at: https://library.biblioboard.com/viewer/0f4841da-d67b-4a2a-822f-6fa532cef1b3. Park, Jonggwon et al. (2019) ‘A Bi-directional Transformer for Musical Chord Recognition’. Available at: https://arxiv.org/abs/1907.02698. Phi, M. (2018) Illustrated Guide to Recurrent Neural Networks. (Towards Data Science). Available at: https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9 (Accessed: 2021).

Reiss, J. D. and Zhou, R. (2007) ‘Music Onset Detection Combining Energy-Based and Pitch-Based Approaches’. Reiss, J. D. and Zhou, R. (2010) Music Onset Detection. (Machine Audition: Principles, Algorithms and Systems). Roebel, A. (2005) ‘Onset Detection in Polyphonic Signals by means of Transient Peak Classification’. Roebel, A. (2009) ‘Onset detection by means of transient peak classification in harmonic bands’, in, p. 1. Available at: https://hal.archives-ouvertes.fr/hal-01161234. Rudolph, G. et al. (2016) Music Data Analysis. CRC Press (Chapman & Hall/CRC Computer Science & Data Analysis), pp. 412–420. doi: 10.1201/9781315370996. Salamon, J. and Bello, J. P. (2017) ‘Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification’, 24(3). doi: 10.1109/LSP.2017.2657381. Schedl, M., Gómez, E. and Urbano, J. (2014) Music Information Retrieval: Recent Developments and Applications . Hanover, Massachusetts: now (Foundations and Trends® in Information Retrieval, 8), pp. 127–261. doi: 10.1561/1500000042. Schindler, A., Lidy, T. and Böck, S. (2020) ‘Deep Learning for MIR Tutorial’. Available at: https://arxiv.org/abs/2001.05266. Schlüter, J. and Böck, S. (2013) ‘Musical Onset Detection with Convolutional Neural Networks’. Available at: http://84.89.139.82/system/files/publications/Schlueter_MML13.pdf. Schluter, J. and Bock, S. (2014) ‘Improved musical onset detection with Convolutional Neural Networks’, in. IEEE (2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)), pp. 6979–6983. doi: 10.1109/ICASSP.2014.6854953.

V. I. Year 2005 2006 2007 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

APPENDIX Onset Detection Category Winners MIREX 2005 - 2018 Name Lacoste, Eck Roebel Reiss, Zhou Roebel Böck, Eyben, Schuller Böck Böck Böck Böck Böck Schlüter, Böck Böck Schlüter, Böck

Algorithm MULTI-NET roebel_3 energy and pitch-based detection roebel_3 OnsetDetector OnsetDetector.2011 OnsetDetector.2012 OnsetDetector.2013 OnsetDetector.2014 OnsetDetector.2015 CNNOnsetDetector CNNOnsetDetector.2017 CNNOnsetDetector

F-measure N/A 0.788 0.808 0.796 0.79 0.83 0.85 0.87 0.86 0.859 0.873 0.874 0.874