Automatic Determination of Vocal Percussive Classes Using Unsupervised Learning

AUTOMATIC DETERMINATION OF VOCAL PERCUSSIVE CLASSES USING UNSUPERVISED LEARNING by Rachel Locke

Supervised by Dr Apostolos Georgakis

MSc Digital Audio Engineering School of Computing and Engineering University of West London United Kingdom

September 2021

2

Abstract Cluster analysis involves the partitioning of a feature space into class-based regions. The potential applications of real-time clustering of vocal percussion hold interesting possibilities for human computer interaction, relating to the control of music production tools. The performance of four feature sets were investigated for the automatic, unsupervised classification of vocal percussive classes, evaluated for three durations of vocal percussion, namely, baseline (23ms), real-time low latency (5.8ms), and real-time ultra-low latency (2.9ms). The highest similarity was gathered using deltaaccelerated Mel Frequency Cepstral Coefficients (MFCC) and MFCC+ feature sets. This study has shown promising experimental results including 100% accuracy when applying the k-means clustering algorithm to a standardised, Principle Component Analysis (PCA) reduced, MFCC+ feature set of three vocal percussive classes, >94% accuracy on five vocal percussive classes, and >93% accuracy on six vocal percussive classes.

Acknowledgments I would like to thank Nino Auricchio, Christos Chousidis, Apostolos Georgakis and Kevin Munisami for your support throughout this MSc. A huge thank you to my colleague and mentor Cameron Thomas. I couldn’t have perused this project without your patience, support, and vast subject knowledge. Finally, thank you to the late Lynsey Dart my dearest friend and collaborator, you’ve inspired all my academic pursuits, knowing you has been my proudest achievement by far, this is for you.

Google Collab Project Code Available at: https://drive.google.com/file/d/1pqc5vxiqxU3RIwEehle0s5ZsX0rltgg/view?usp=sharing

3

Table of Contents Abstract..................................................................................................................................................... 2 1. Introduction ........................................................................................................................................... 5 1.1 Motivation .....................................................................................................................................................5 1.2 Research Questions and Approach ................................................................................................................6 1.3 Structure ........................................................................................................................................................6 2. Background and Related Work ................................................................................................................ 6 2.1 Unsupervised Learning ..................................................................................................................................6 2.1.1 Cluster Analysis .................................................................................................................................7 2.1.2 𝒌-means ............................................................................................................................................7 2.1.3 Gaussian Mixture Models (GMM) ....................................................................................................8 2.1.4 Self-Organising Maps (SOM) ............................................................................................................9 2.2 Vocal Percussion ............................................................................................................................................9 2.2.1 Vocal Imitation..................................................................................................................................9 2.2.2 Query by Percussive Vocalisation .....................................................................................................9 3. Methodology ....................................................................................................................................... 10 3.1 Pipeline ........................................................................................................................................................10 3.2 Datasets .......................................................................................................................................................10 3.3 Audio Pre-processing ...................................................................................................................................11 3.4 Feature Sets .................................................................................................................................................11 3.5 Feature Extraction .......................................................................................................................................15 3.6 Feature Engineering ....................................................................................................................................16 3.7 Performance Metrics ...................................................................................................................................19 3.8 The k-elbow Method....................................................................................................................................19 3.9 Clustering .....................................................................................................................................................20 3.10 Hyperparameter Optimization ..................................................................................................................21 4. Results ................................................................................................................................................. 24 5. Conclusion ........................................................................................................................................... 32 References............................................................................................................................................... 34

4

List of Figures Figure 1. The Potential Effects of Centroid Initialisation on Final Clustering 𝑘-means++ (left) and randomised (right) Figure 2. Incorrect Cluster Assignments Caused by 𝑘-means Circular Cluster Models Figure 3. Project Pipeline Diagram Figure 4. Vocalised Kick Drum for Control (left), Low-Latency (centre), and Ultra-Low Latency (right) Figure 5. Mel Filter Banks Figure 6. Relationship Between the Mel (blue), Bark (green), and Hertz Frequency Scales Figure 7. Scree Plot for three_perc Dataset Principle Components 1 − 10 Figure 8. An Example of a Feature Set Before (top) and After Principle Component Analysis (bottom) Figure 9. Clustering Visualised on a Two-Dimensional Principle Subspace Figure 10. Clustering Visualised on a Three-Dimensional Principle Subspace Figure 11. Spectral Centroids for Three Vocal Percussive Elements (Ultra-low Latency) Figure 12. Zero-Crossing Rate for Three Vocal Percussive Elements (Ultra-low Latency) Figure 13. Spectral Centroids for Five Vocal Percussive Elements (Ultra-low Latency) Figure 14. Zero-Crossing Rate for Five Vocal Percussive Elements (Ultra-low Latency) Figure 15. Spectral Centroids for Six Vocal Percussive Elements (Ultra-low Latency) Figure 16. Zero-Crossing Rate for Six Vocal Percussive Elements (Ultra-low Latency) Figure 17. k-means Clustering for Six Vocal Percussive Elements in Two-Dimensional Principle Subspace (Ultra-Low Latency) Figure 18. k-means Clustering for Six Vocal Percussive Elements in Three-Dimensional Principle Subspace (Ultra-Low Latency) Figure 19. Example of Distinct Cluster Formations (top) Upon Which the k-elbow Metric Performs Well (bottom) Figure 20. Example of Cluster Formations that are not Clearly Defined (top) Upon Which the k-elbow Metric Performs Poorly (bottom)

5

1. Introduction Vocal percussion, more commonly referred to as Human Beat Box (HBB) is a form of vocal imitation that has its origins in early 1980s Hip Hop culture. The term ‘beat box’ refers to the drum machines the art form seeks to imitate (Stowell and Plumbley, 2008). The human voice is an information-rich instrument with far-reaching potential as a medium of control. The applications of Machine Learning (ML) for HBB datasets are largely unexplored in the audio classification field when compared with the body of work regarding speech and singing. Very few studies have concerned the classification of HBB. Even fewer have considered unsupervised learning techniques and how these might assist the use of HBB as a music production tool (Tzanetakis and Kapur, 2004). The unsupervised nature of the learning algorithm is imperative to the proposed application as it is expected that users engaging with music production tools will be supplying vocal data in real-time. This results in unlabelled input data and the need for a ML algorithm that is adaptive for instantaneous input. Voice-controlled audio software is a rapidly growing research field (Tzanetakis and Kapur, 2004; Stowell, 2010; Mehrabi, 2018). HBB has shown potential as an interesting medium for query and retrieval (Tzanetakis and Kapur, 2004; Mehrabi, 2018) and, more importantly for this work, as a performance tool for browsing and triggering samples or effects (Stowell, 2010; Stowell and Plumbley, 2010). Clustering is a fundamental task in data mining, involving the partitioning of data into groups of similar observations. Cluster analysis fundamentally consists of two stages; the first stage in any automatic classification system is to select and extract abstractions of the data known as features. These features should describe the idiosyncrasies of the data across multiple classes (Gonzalez, 2012). As the underlying articulators of HBB are phonemes, (the same articulators that characterise speech) features that perform well for speech recognition tasks are predicted to reliably segment HBB classes (Evain et al., 2021). The second requirement is the application of an appropriate clustering technique. From the vernacular it may seem that the latter of the two stages has the strongest impact on the outcome of the final clustering, however, optimisation of machine learning algorithms is more commonly attributed to the selection and engineering of features that are robust to non-perceptual variations (Gonzalez, 2012). The key to efficient segmentation is to establish which audio features best model the similarity (or perhaps, more importantly, the dissimilarity) across all classes of the vocal percussion datasets.

1.1 Motivation In a recent conference debate, an esteemed machine learning solutions consultant, Valardo (2021) spoke of the value of approaching a problem with a simpler heuristic-based algorithm as an alternative to finding ways to apply cutting-edge deep learning methods. They continue, the rationale behind this statement is that, in many cases, known machine learning models or heuristic-based solutions can get you 80% of the way towards a solution, at which stage, if necessary, a deep-learning pipeline can be built up around it. Using this process, data may be gathered that is instrumental to the building of reliable machine learning solutions. At the time of writing this paper, the state-of-art audio classification methods generally involve Convolutional Neural Networks (CNN) (Brahnam et al, 2020) or Convolutional Deep Belief Networks (Largman et al, 2020; Ciancia et al, 2021). For reasons related to scope and purpose, the motivation of this project is not to recreate or attempt to redefine state-of-the-art classification methods. Instead, the aim of this work is to apply pre-existing, partitional and model-based, unsupervised machine learning solutions, and feature sets that have consistently proven to be reliable to the task of segmenting vocal

6 percussive data. An important consideration underpinning this work is the trade-off between real-time low latency and classification accuracy. To achieve lower latency, a lighter weight classification solution (i.e., computationally light) is favourable (Valardo, 2021). This work explores three separate unsupervised learning solutions in combination with four audio feature sets. It evaluates the reliability of their performance on the clustering of three vocal percussive datasets for three audio durations, namely, baseline (23ms), real-time low latency (5.8ms) and real-time ultra-low latency (2.9ms). There is no conceivable evidence of a publication to date that combines dimensionality reduction and clustering for vocal percussive datasets that documents the procedure or results of hyperparameter tuning for real-time durations.

1.2 Research Questions and Approach This paper aims to add to the pool of research concerning the classification of drum pattern vocalisation, for the purposes of query by vocalisation and the live sketching of rhythmic ideas. The aim of this work is to develop a proof-of-concept for an unsupervised learning based, vocal percussion segmentation pipeline that performs reliably for both offline and online (real-time) audio durations. A secondary aim is to establish which audio features or combinations thereof can be used to accurately represent the similarities and dissimilarities in vocal percussive datasets. Some further objectives of this study are as follows: Explore vocal percussion as a vehicle for real-time control of music production software Determine best performing hyperparameters for the model via hyperparameter optimisation Assess the performance of feature sets that have outperformed MFCC for similar applications

1.3 Structure Chapter 2: Introduces the main concepts that underpin this work including: unsupervised learning, cluster analysis, vocal percussion, and vocal imitation. These main concepts are presented in the context of previous exploration in the field.

Chapter 3: Illustrates the structure of the proposed pipeline and methodology taken throughout the implementation of the various pipeline stages. Focuses heavily on the Python implementation, real-time considerations and hyperparameter tuning.

Chapter 4: Presents and evaluates the results for all combinations of dataset, feature set, audio duration and hyperparameters. Results, scoring metrics and best performing hyperparameters are displayed as visualisations and tables.

2. Background and Related Work 2.1 Unsupervised Learning Many cases exist where it is beneficial to discover underlying patterns (often groups) within a collection of unlabelled data for which there are no target outputs. Unsupervised learning is a Machine Learning technique that involves the classification of these unlabelled datasets (Ghahramani, 2004). The only information the unsupervised learning algorithm possesses is the underlying patterns of the input data from which it must determine potential output classes. For this reason, clever feature selection and feature engineering are critical to the success of many unsupervised learning techniques (Gonzalez, 2012).

7

2.1.1 Cluster Analysis Cluster analysis is an unsupervised learning technique that involves the segmentation of multivariate data into subgroups or ‘clusters’ in which data points have higher similarity within their group than they do with data points belonging to other clusters (Everitt et al., 2011). The multivariate data is generally described by a set of qualities or characteristics, in this case, these are the reduced audio features and, henceforth, post feature extraction, the dataspace will be referred to as a feature space.

2.1.2 𝑘-means A central concept of cluster analysis is the idea of similarity and dissimilarity between the datapoints in a feature space. For partitional clustering algorithms such as k-means, the metric of similarity (or dissimilarity) is distance, whereby, intra-cluster cohesion and inter-cluster separation denotes optimal clustering (Bishop, 2010). The fundamental goal of clustering is to assess the relationships between data. When visualising a feature space, closeness of data points is associated with similarity whilst farness indicates dissimilarity. The dissimilarity measure for the popular 𝑘-means clustering algorithm is squared Euclidian distance. This partitional technique involves the clustering of data points around centroids. 𝑘-means requires a pre-specified value for 𝑘 which dictates the number of centroids and resulting clusters (Hastie, Tibshirani and Friedman, 2009). 𝑘-means is an iterative algorithm that repeats until its centroids converge on a final partitioning (Algorithm 1).

Algorithm 1. 𝒌-means 1. Randomly assign initial 𝒌 centroids (𝒄𝟏 , 𝒄𝟐 … 𝒄𝒌 ) 2. Assign each point in data X to its closest centroid 𝒄𝒊 according to Euclidian distance 3. Calculate the sum of the Euclidian distance between centroid 𝒄𝒊 and the data points in its cluster 4. Divide by number of data points belonging to cluster with centroid 𝒄𝒊 to obtain the mean 5. Move centroid 𝒄𝒊 to reflect the new mean 6. Repeat Steps 2, 3, 4 & 5 until 𝒄𝒊 no longer changes

Traditionally, the initial positions of the centroids would be randomly chosen data points, however, this method can prove unreliable as the initial position influences both the resulting clustering (fig 1) and the convergence time (Syakur et al., 2018). To address this, k-means++ was introduced to overcome the drawbacks of standard k-means reliance on randomised centroid initialisation, whilst remaining 𝛰(𝑙𝑜𝑔 𝑘) 𝑐𝑜𝑚𝑝𝑒𝑡𝑖𝑡𝑖𝑣𝑒 (Bachem et al, 2017).

8 Figure 1. The Potential Effects of Centroid Initialisation on Final Clustering 𝑘-means++ (left) randomised (right) 𝑘-means++ finds optimal centroid locations by endeavouring to cover as much of the feature space as possible with the probabilistic spreading out of centroids. The first centroid’s position is randomly chosen whilst the positions of the following centroids are initialised at the farthest data point 𝑓𝑖 from their nearest preceding centroid 𝑐𝑖 (Arthur and Vassilvitskii, 2007).

Algorithm 2. 𝒌-means++ centroid initialisation 1. 2. 3. 4.

Randomly assign the first centroid 𝒄𝟏 Compute distance of all data points 𝒙𝒊 from the nearest previously chosen centroid 𝒄𝒊 Assign new centroid to the farthest point 𝒇𝒊 from the nearest previously chosen centroid 𝒄𝒊 Repeat steps 2 & 3 until all k centroids are initialised

Thiruvengatanadhan (2015) evaluated the performance of 𝑘-means clustering on Power Normalised Cepstral Coefficients (PNCC) features for the separation of speech and music. The study achieved 87% cluster accuracy for 10-second audio durations. Also in 2015, Bello and Salamon applied the spherical 𝑘-means algorithm to feature learning on an urban sound dataset, yielding improved results compared against Mel Frequency Cepstral Coefficients (MFCC) features when configured to capture the temporal dynamics of urban sounds.

2.1.3 Gaussian Mixture Models (GMM) Although k-means is a common choice for multivariate data segmentation, it is not without its limitations, most notably, the fixed circular cluster model (spherical in higher dimensions) relies on circular cluster allocations and can cause data points to be incorrectly clustered if the clusters present in non-circular distributions (fig 2).

Figure 2. Incorrect Cluster Assignments Caused by 𝑘-means Circular Cluster Models Gaussian Mixture Models (GMM) are presented in the literature as a more flexible alternative to 𝑘means due to their probabilistic nature and ability to model arbitrary shaped clusters. The underlying algorithm of GMM is very similar to k-means, however, due to small differences in the way cluster probability is quantified, GMM can tackle the drawbacks of the k-means algorithm (Everitt et al., 2011). Whilst k-means performs what is referred to as ‘hard classification’ meaning it makes final specifications regarding which data belong to which cluster, it doesn’t achieve what GMM ‘soft

9 clustering’ accomplishes, which is provide information on the probability of each data member belonging to a certain cluster (Kaur and Kumar, 2017). GMM requires the number of Gaussian distributions to be specified (this number should be the same as the number of clusters that are expected) and is synonymous with the value for 𝑘 when implementing 𝑘-means. Methods for predicting number of distributions will be discussed in chapter 3.8. GMM has been used for automatic music genre classification, exhibiting a classification of 94% for categorisation between pop, rock, and classical when clustering on tempogram features (Thiruvengatanadhan, 2018), but showing a decreased performance of 74% accuracy in a similar genre classification study on Mel Frequency Cepstral Coefficients (MFCC), Zero Crossing Rate (ZCR), Spectral Roll-off (SR) and Flux (F) features (Kaur and Kumar, 2017). GMM has also been used in audio feature engineering to model features across frames by means of a probabilistic model (Li et al, 1999).

2.1.4 Self-Organising Maps (SOM) Self-Organising Maps (SOM) are a type of Artificial Neural Network (ANN) for unsupervised learning, frequently used for clustering by remapping high dimensional datasets onto a lower-dimensional (typically 2D) plane whilst preserving their innate nonlinear structure. In the work of Merkl et al, 2001 SOM were utilised for large music libraries to cluster similar songs based on rhythmic features. One of the authors (Pampalk, 2001) extended this technique in their master’s thesis, during which they developed a visualisation tool for exploring large, unlabelled music collections that displayed the output clusters of the SOM as ‘islands’ on a Graphical User Interface (GUI), thus, coining the term ‘Island of Music’. In a more recent study Jiang et al, 2020 present an improved k-means algorithm for speaker segmentation based on a SOM. Here the optimal number for 𝑘 is determined by the winning situation of the competitive neurons in the two-dimensional output layer.

2.2 Vocal Percussion 2.2.1 Vocal Imitation Aura et al (2011) conducted a study that presented participants with a collection of sounds and their corresponding vocal imitations and asked them to group the vocal imitations into classes. The purpose of this study was to conclude whether a vocal imitation of a sound had the same perceptual acoustic properties as the sound being imitated, and therefore, whether participants could successfully classify these imitations. The classes used in the study were ‘gaseous’, ‘liquid’, and ‘solid’. The participants correctly classified the ‘gaseous’ and ‘solid’ classes but were unable to appropriately classify the ‘liquid’ imitations. This was attributed to the human incapability to successfully imitate the high chirps associated with the production of bubbles in liquid sounds. The study ultimately concluded that vocal imitations proved useful in the study of sound identification (within vocal constraints). A similar study conducted by Dixon et al (2019) required participants to decide whether they perceived a vocal imitation of a percussive sound to be more ‘like’ a target sound than other sounds in its category. They found that imitations received the highest similarity ratings for 16 out of 30 percussive sounds. Better results were achieved when matching the imitations and sounds with an auditory image-based measure. The study revealed that query by vocalisation still had some way to go when it comes to producing high accuracy results, but ultimately that feature-based machine learning techniques produced better similarity results than human listeners.

2.2.2 Query by Percussive Vocalisation In recent years, the classification of vocal percussion has been identified as having importance for Music Information Retrieval (MIR) tasks such as query by vocalisation (QBV), automatic transcription, and automatic dataset annotation (Stowell and Plumbley, 2010; Sinha and Soni, 2021). QBV is a form of query-by-example that references a person’s imitation of a sound they wish to retrieve. Typically, QBV systems involve the extraction of acoustic features from the vocal imitation

10 of a target sound and the comparing of these against the features of sounds in a sample library, to find the closest feature match (Mehrabi, Dixon and Sandler, 2019). Identifying how many distinct sounds are present in a stream of vocal percussion can be useful for several music production workflows (Mehrabi, 2018). For example, conceive that a musician has an idea for a percussive loop that they can communicate rhythmically. Through onset detection and splitting of the audio stream based on these onsets, the hits associated with them could then be clustered and matched based on feature similarity to percussive samples in sample libraries. The outcome of this could be a custom drum rack containing samples that most closely match the imitations provided in the vocalised audio stream. The previous example describes onset-wise analysis, a technique explored by Hazan (2005). Their paper documented a three-stage task of the transcription of expressive vocal percussive performances. These are: a) onset detection to separate the individual percussive elements, b) feature extraction to obtain a feature set for each of the percussive elements, c) a machine learning component to segment the separated sounds into classes. The pipeline described is very similar to that which is proposed in this paper, though stage a) is omitted in this work. In 2004 Tzanetakis and Kapur designed and developed a query-by-beatbox system capable of classifying human beatbox, namely, kick drum, snare drum, and hi-hat, and mapping them to corresponding drum samples. They reported classification accuracy of > 95% for zero-crossing rate and Linear Prediction Coefficient (LPC) features.

3. Methodology 3.1 Pipeline Audio samples are loaded from vocal percussive datasets and propagated through the pipeline pictured in fig 3. The stages in the pipeline were implemented in the Python programming language, adhering to Object Oriented Programming (OOP) principles.

Figure 3. Project Pipeline Diagram

3.2 Datasets The amateur HBB elements were recorded, edited, and annotated manually to obtain ground truth references. To assess the performance of clustering on the proposed feature sets, three varying datasets were curated. The ‘three_perc’ dataset consists of 108 vocal percussive utterances, in the classes kick, snare and closed hi-hat. The ‘five_perc’ extends the ‘three_perc’ set to 180 vocal percussive utterances to include clap and cowbell. The ‘six_perc’ dataset extends the ‘five_perc’ set to 216 utterances to include open hi-hat (table I).

Label 0 1 2 3 4 5

Description Kick Snare Closed Hi-Hat Clap Cowbell Open Hi-Hat Table I. Dataset Elements

Phoneme b k t n/a dn tss

11 The baseline recordings in the dataset are mono, fixed duration at 4096 samples per recording (2048 samples either side of the audio onset) and with a fixed sample rate of 44100𝐻𝑧. Each dataset has an associated CSV file that allows the interfacing between the dataset files and the dataset loading class. Each CSV file contains columns for filename, class, folder, and participant.

Audio Durations Justification The standard duration of each audio datum in the three datasets curated for this work is 46ms (4096 samples) as standard (2048 samples pre- and post-onset). This duration is what will be referred to as the control or baseline. The vocal percussive data that populate the datasets are transient and brief, often resulting in leading and trailing zero samples in the control duration (fig 4). In real-time applications, it is the number of trailing samples (after the onset) that will dictate the latency. In the baseline case, 2048 samples are taken after the onset, which translates to 23ms latency (table I). Ideally, the aim is to minimise the latency (samples after the onset) whilst retaining enough samples to provide a reasonable classification result (Stowell and Plumbley, 2010). It follows that there is a trade-off between low latency and classification accuracy. The motivation for the real-time audio durations comes from a study by Stowell and Plumbley (2010) on real-time beatbox percussion classification and a separate study conducted by Jack et al (2018) on the acceptable delay between tactile and auditory feedback for percussionists. The precise durations are taken from a Sound-on-Sound article that gives acceptable audio durations in miliseconds (ms), that states that vocals require < 3𝑚𝑠 and percussion requires < 6𝑚𝑠 (Walker, 2005). Accordingly, the low latency duration is 5.8ms after onset and the ultra-low latency is 2.9ms after onset.

Figure 4. Vocalised Kick Drum for Control (left), Low-Latency (centre) and Ultra-Low Latency (right)

3.3 Audio Pre-processing The indexing and loading of the data are achieved within the VocalPercDataset class. The class inherits from the PyTorch Dataset class and overrides the __len__ and __getitem__ methods for creating a custom PyTorch dataset. This indexable class allows the lookup, loading, pre-processing, and returning of data elements from the associated CSV file path passed into the class’ constructor. The dataset class handles the pre-processing of the audio files in the dataset. Its method’s responsibilities include a) the setting of a baseline audio duration (in samples), b) target sample rate, and c) the number of channels. These methods are called routinely if the loaded audio does not already subscribe to the specified target parameters (i.e., if the conditions are not met). The magic method __getitem__ returns the audio file and label associated with the index specified when indexing an instance of the class. The __len__ method returns the size of the class, synonymous with the size of the dataset (amount of datum in the dataset).

3.4 Feature Sets This study measures the reliability of clustering on vocal percussive datasets for three audio durations. To present a comparative study, four varying feature sets are evaluated.

12 Set 1: Mel Frequency Cepstral Coefficients (MFCC) The MFCC feature vector consists of 9 frames and 12 MFCC totalling 108 features when stacked or 12 when averaged. Recent work from Jiang and Liu (2020) utilised a normalised MFCC feature set for SOM based 𝑘-means clustering.

Set 2: MFCC + ∆ + ∆∆ For consistency 12 MFCC are generated for all MFCC based features. The delta ∆ and delta delta ∆∆ MFCC retain the same shape as the original MFCC. As these are the inter-cepstral coefficients first ∆ and second ∆∆ differences. There will be 12 differences within each of the 9 frames, keeping the shape (9, 12). When stacked this feature set will total 324 features or 36 when averaged. Implemented via librosa.features Python package.

Set 3: MFCC+ MFCC+ is an extended MFCC feature set proposed by Gonzalez (2013) comprised of MFCC, 4 additional temporal features and 6 additional spectral features. For this work, a reduced number of features are included for the MFCC+ feature set. These choices are based on features that well represent percussive sounds in the literature (McDonald and Tsang, 1997; Lakatos, 2000; Brent, 2010), These are: Cepstral MFCC Temporal Zero Crossing Rate (ZCR) Root-Means-Squared Value (RMS) Spectral Spectral Centroid (SC)

Set 4: PSTC Principle Spectral Temporal Components (PSTC) were also proposed by Gonzalez (2013) and were presented in the paper ‘Better Than MFCC Classification Features’ as a feature set that outperform both raw MFCC and MFCC+ feature sets for the classification of a ‘Four Audio’ dataset, consisting of recordings of (1) speech, (2) music, (3) laughter, and (4) applause, and an ‘Insect’ dataset consisting of recording of (1) Katydid, (2) Cricket, (3) Cicada, and (4) Other. MFCCs The features and combinations thereof explored in this work are frame-level features, in that they are calculated for individual frames rather than over an entire signal. Frame-level features allow the statistical averaging across frames (i.e., mean, max, and median) (Stowell, 2010). In terms of levels of abstraction, the features used fall into mid-level (MFCC, ∆ MFCC and ∆∆ MFCC) and low-level (spectral centroid, zero-crossing rate, root-means-squared) audio feature categories. Low-level features are commonly purely statistical and do not have inherent perceptual relevance to humans, whereas midlevel features cross over into the realm of human auditory perception (Knees and Schedl, 2016).

13 One such mid-level descriptor featured heavily in this work are MFCC. MFCC are perceptually motivated representations that have proven to be efficient in decoupling the shape of the human vocal tract (including tongue and teeth) and the initial glottal pulses of speech (Evain et al., 2021).When producing vocal percussive sounds, after generation in the glottis, initial glottal pulses are filtered by the vocal tract. Approximating the shape of the vocal tract will give an accurate impression of the phonemes being formed. The shape of the vocal tract can be approximated by the spectral envelope (i.e., the smoothed log-amplitude spectrum) of an audio signal, able to capture the resonant frequencies of the oral cavity known as formants, that exist at the peaks of a speech signal’s spectral envelope (Dusan and Deng, 1998).

Figure 5. Mel Filter Banks The ‘Mel Frequency’ describes the adherence of MFCC to the auditory perceptual Mel Scale that illustrates the relationship between frequency and pitch. This relationship is linear at low frequencies and logarithmic at high frequencies (fig 6). The psychoacoustic relationship between Hertz and Mels is an approximation gathered from listening tests (Umesh, Cohen and Nelson, 2002). To obtain these coefficients, the frequency axis of the log-amplitude spectrum of an audio signal is warped to the Melfrequency scale by applying triangular Mel filter banks (fig 5). The relationship between Hertz and Mels is given by (1) 𝑀𝑒𝑙(𝑓) = 2595 log10 (1 +

𝑓 ) 100

1

The Discrete Cosine Transform (DCT) is then applied to the Mel-frequency, log-amplitude spectrum. This transformation moves from the frequency domain to the quefrency domain, transforming the spectrum into a cepstrum (2), the name given by early MFCC researchers to a spectrum-of-a-spectrum (Knees and Schedl, 2016).

𝐶(𝑘) = 𝐷𝐶𝑇[log 𝐷𝐹𝑇(𝑥(𝑛)]

2

14

Figure 6. Relationship Between the Mel (blue), Bark (green) and Hertz Frequency Scales In terms of stages in the clustering pipeline, audio feature selection and engineering have the most significant effect on the result of Machine Learning, the choice of machine learning algorithm proving trivial when comparing performance accuracy (Gonzalez, 2012). This is especially significant when it comes to partitional clustering methods, where capturing salient characteristics of the individual vocal percussive elements is fundamental for achieving adequate inter-cluster separation. The better the intercluster separation and intra-cluster cohesion achieved by feature selection, the more efficiently the clustering algorithm will be able to ascertain the individual classes. The adequate separation will additionally improve the accuracy of the k-elbow method for optimal cluster prediction, particularly when scoring with the silhouette metric where the system of measurement is based on the clear definition of distinct clusters or lack thereof (discussed further in chapter 3.8). In the literature, there is overwhelming evidence for MFCC outperforming alternative spectral and temporal audio features for supervised and unsupervised machine learning tasks involving vocal datasets (Mitrović, Zeppelzauer and Breiteneder, 2010; Mehrabi, 2018; Cabral, Fukai and Tamura, 2019; Jiang and Liu, 2020). In addition to capturing the frame-based static MFCC, the first and second derivatives can be calculated across frames. These dynamic trajectories of MFCC between frames are known as delta MFCC (∆) and delta delta MFCC (∆∆). They can be appended to enhance the static MFCC feature vector to give a better representation of the evolution of the dynamics of the MFCC features across frames (Mehrabi, 2018). Delta accelerated MFCC are prominent in numerous speech recognition use cases. An example with relevance to this work is the Brognaux et al paper (2015) that saw the application of delta accelerated MFCC to automatic HBB recognition. The best performance in the case of individual vocalised drum sounds was given by 22 MFCC + ∆ + ∆∆. Encouraged by the Doctorate thesis of Mehrabi (2018) this work evaluates the performance of delta accelerated MFCC to determine whether the temporal inter-frame differences are useful for this application. In a 2019 study by Cabral et al concerning feature extraction for speech recognition and their effectivity for road condition monitoring on smartphones, delta accelerated MFCC were found to produce the highest rates of accuracy at 98.07% classification accuracy. More recently Bernardoni et al (2021) found MFCCs to significantly outperform Perceptual Linear Prediction (PLP) and logarithmic Mel-filter bank coefficients (FBANK) features for HBB recognition by automatic speech recognition toolkit.

Extended Feature Sets Gonzalez (2013) evaluated five feature sets; the latter four sets address the limitations of MFCC. The first being purely MFCC, the second, MFCC+ extends MFCC features to capture additional spectral and temporal information. The extended features aggregated to create the MFCC+ feature set included (1) Zero Crossing Rate, (2) Energy Flux, (3) Short-Term Energy (4) Root-Means-Squared Value, (5) Spectral Centroid, (6) Signal Bandwidth, (7) Pitch and Harmonicity via Bregman’s Method (Micheyl

15 and Oxenham, 2010), (8) Pitch gathered by subharmonic summation and (9) Skew. The remaining three are novel feature sets proposed by Gonzalez as a way of approximating the Karhunen-Loeve theorem (KLT) (commonly referred to as Principal Component Analysis (PCA)) for obtaining Principle Components (PC). They propose approximating the KLT as a Discrete Cosine Transform (DCT) to obtain the Principle Spectral Components (PSC) of a signal. Gonzalez reasoning was the guarantee of PC to optimally characterise the underlying structure of data. The remaining two feature sets are extensions of PSC, Principle Cepstral Components (PCC) and Principle Spectral Temporal Components (PSTC). The PSTC features were found to produce the best overall results (for adequate data) whilst PSC and PCC obtained better results than MFCC and MFCC+ for smaller datasets. First applied to supervised audio classification these feature sets have been explored in this work for unsupervised learning on vocal percussion datasets. The justification and formulas for the proposed features will be introduced in greater detail below. Principle Spectral Coefficients (PSC) – The first few DCT coefficients of the spectrum obtained via a short-time Fourier transform, given by (3) 𝑃𝑆𝐶(𝑘) = 𝐷𝐶𝑇{|𝐷𝐹𝑇{𝑥(𝑛)}|}

3

Principle Cepstral Coefficients (PCC) – A variation on PSC that provides more even scaling of the feature set by whitening the spectrum, given by (4) 𝑃𝐶𝐶(𝑘) = 𝐷𝐶𝑇{log|𝐷𝐹𝑇{𝑥(𝑛)}|}

4

Principle Spectral-Temporal Coefficients (PSTC) – Captures the principle information contained in the time-frequency distribution of energy in the audio signal, given by (5) 𝑃𝑆𝑇𝐶(𝑘, 𝑙) = 2𝑑𝐷𝐶𝑇{𝑆𝑇𝐹𝑇{𝑥(𝑛)}}

5

At 1025 features per frame, the PSTC result in a much larger feature set to describe the underlying audio samples when compared to the alternate features explored during the project. Stacking the 9 frames yield a feature vector of shape (1, 9225) per data member. Gonzalez (2013) proposed the previously introduced features as “better performing” and “simpler to calculate” than MFCCs and presented results that showed this to be true for the specific classification problem to which the feature sets were applied.

3.5 Feature Extraction In the initial stages of this work, features were gathered only from the first frame of each audio sample. Combining features across frames using a technique commonly referred to as ‘stacking’ is described by Meng, 2006 in their PhD thesis as ‘temporal feature integration’ and involves the aggregation of framebased features into a single feature vector. This technique can result in more reliable clusters by making use of the information gathered from features in other frames. Another frame-based aggregation method would be that of averaging across frames to find a mean, median, min or max, though some information is discarded during this process (Knees and Schedl, 2016). Naturally, stacking features creates a larger feature vector. When stacking 12 MFCC, 12 ∆ & 12 ∆∆ (36 features) across 9 frames the result is a 1 𝑏𝑦 324 feature vector to represent each instance of vocal percussion. The shape of a feature vector (in the above example [1, 324]) is known as its dimensionality (Stowell, 2010). Within the six_perc dataset, there are 216 vocal percussive recordings, resulting in a feature space of [216, 324] for the delta accelerated MFCC features. There exists a difficulty that can arise in feature spaces containing many dimensions, a phenomenon referred to in the literature as ‘the curse of dimensionality’. To address this, the dimensions of the feature space were reduced from 324 to 5 prior to clustering, in a process called dimensionality reduction (Winursito, Hidayat and Bejo, 2018).

16 Feature extraction and aggregation are performed within the FeatureExtractor class. The name associated with each audio feature (i.e., ‘mfcc’) and its method of aggregation (i.e., ‘stack’) are stored as key-value pairs in a Python dictionary feature_dict. feature_dict = { "mfcc": "stack", "delta_mfcc": "stack", "delta_delta_mfcc": "stack", }

The dictionary is passed as a parameter to the FeatureExtractor constructor and stored as a member variable self.features for use in the _perform_extraction method. The extracted and aggregated features are appended to an overall feature vector, returned from _perform_extraction as a NumPy array. The shape of this feature vector is known as its dimensionality.

3.6 Feature Engineering Audio is considered as high dimensional data due to its many features, therefore, it is often beneficial to apply dimensionality reduction to the feature vector prior to clustering. Principle Component Analysis (PCA) (also known as the Karhunen-Loève transform) projects the data onto a lowerdimensional subspace, referred to in the literature as a principle subspace (Baniya, Joonwhoan Lee and Ze-Nian Li, 2014). Another consideration is the scaling of the feature set to achieve a standard range and variance (Choi et al., 2018). The following feature handling steps were selected according to the work of Stowell, 2010 with reference to engineering audio features for QBV. Following extraction, features were either standardised to have a mean of 0 and a standard deviation of 1 or normalised between 0 - 1 across all vocal percussive elements. The scaled feature vector was then reduced in dimensionality via a dimensionality reduction technique. PCA has been the most widely used data reduction technique during this work but other options for dimension reduction were explored, including Factor Analysis (FA) and Independent Component Analysis (ICA). PCA is employed in this project as a method of dimensionality reduction to reveal principle components (PC). These are latent axes that best represent the variance of the data (fig 8). Number of components (i.e., the number of dimensions to reduce to) was chosen through a combination of manual hyperparameter testing, scree plot analysis, and automated hyperparameter tuning (this process is described at length in chapter 3.10). A scree plot (fig 7) demonstrates how much of the feature set’s variance is captured by each PC in the principle subspace. This can assist with the selection of an appropriate value for number of components when creating a PCA model.

17

Figure 7. Scree Plot for three_perc Dataset Principle Components 1 − 10 The first PC will always capture the highest percentage of the data’s variance and as the number of PC increases the amount of variance each PC represents decreases. PCA is often used in combination with clustering to reduce a higher-dimensional feature set to a lower-dimensional representation, often giving a better visual segmentation of classes described by a set of audio features (Baniya, Joonwhoan Lee and Ze-Nian Li, 2014). It can be seen in (fig 7) that the contribution of the principle components becomes linear at around 𝑃𝐶 = 5, this, combined with the results presented in chapter 4 (table III) for hyperparameter tuning of number of dimensions determines five to be an ideal number of PCs to capture the variance of the data. The following stage in the pipeline implementation allows both feature scaling and dimensionality reduction.

18

Fig 8. An Example of a Feature Set Before (top) and After Principle Component Analysis (bottom) The FeatureHandling class has two significant methods. These are _scale_features and _get_reduction. This allows the user to specify scaling type, with a choice between standardisation and normalisation. A user may also specify the dimensionality reduction technique and the number of dimensions to which they wish to reduce the feature set. The input feature set, target number of dimensions, dimensionality reduction type, and scaling type are passed as parameters to the FeatureHandling constructor upon object creation. In the case that 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑠 > 3 it is not possible to visualise the dimensions that exceed the third. If the output of a higher dimensional principle subspace is visualised it will be limited to a two-dimensional visualisation (fig 9), or threedimensional visualisation with a 3D plot (fig 10).

Figure 9. Clustering Visualised on a Two-Dimensional Principle Subspace

19

Figure 10. Clustering Visualised on a Three-Dimensional Principle Subspace

3.7 Performance Metrics As introduced previously, unsupervised learning concerns the classification of unlabelled data. For the purposes of this work the datum labels have been recorded within the CSV file of each dataset and loaded directly from the VocalPercDataset class as an array. These labels are necessary only in the pipeline configuration stages, as the true classes can be compared against the predicted classes to score accuracy. This is the basis of the Rand Index performance metric (Somashekara and Manjunatha, 2014). The Rand Index was chosen as the overall scoring metric for this work; however, the labels are only used here to determine the optimal configuration regarding audio features sets, clustering techniques and so on. If the proof-of-concept were to go to production, the data received would be unlabelled. Hence the need to determine the optimal configuration beforehand. In the case that the model is receiving unlabelled data there are other performance metrics that can be explored, though, due to the absence of labels these metrics use systems of measurement that do not rely on the knowledge of the true classes of data. One such metric is the Silhouette Score where the metric is the ‘goodness’ of clustering, determined by the degree of inter-cluster separation (Shutaywi and Kachouie, 2021).

3.8 The k-elbow Method The reliance of 𝑘-means on a known, specified value of 𝑘 can be a limitation. In a real-time application, where it is not known beforehand how many distinct sounds a user will create, it is not possible to hard code a fixed value for 𝑘. Instead, there are methods that can be applied to determine an optimal value for 𝑘. The 𝑘-elbow technique is one such method that involves the fitting of a range of candidates for 𝑘 to the principle subspace (Syakur et al., 2018). The 𝑘-elbow method is implemented from the Yellowbrick machine learning visualization Python library. To determine the optimal number of clusters, the model is fitted with values of k in a specified range. The optimal number of clusters determined by this method will present itself at the ‘elbow’ or ‘knee’ of the graph. The implementation of the 𝑘-elbow method is achieved by first instantiating the cluster model and the visualizer. model = KMeans() visualizer = KElbowVisualizer(model, k=(3,12), metric='calinski_harabasz')

20 When instantiating the visualizer, the model, range of k to fit to the model, and the scoring metric are passed into the constructor. Next, the data is fitted to the visualizer. In this case, the principle subspace data is passed in. Finally, the visualizer show method is called to finalise and render the figure. visualizer.fit(pcas) visualizer.show()

There are a few scoring metrics upon which the k-elbow can be evaluated. These are: – Distortion – The default metric computes the sum of squared distances from each point to its assigned centre. This metric has performed sufficiently for a larger range of 𝑘 i.e., 𝑥 ∈ ℝ | 3 ≤ 𝑘 ≤ 12 Silhouette – The Silhouette Score calculates the ‘goodness’ of a clustering technique. To determine the optimal number of clusters based on this metric, the mean silhouette coefficient of all samples is calculated. The silhouette coefficient is a value between −1 and 1 where a value of 1 indicates clearly defined, distinct clusters and a value of −1 indicates poor separation (Syakur et al., 2018). Based on this system of measurement, employing the silhouette coefficients for determining optimal number of clusters will be most effective when the data points are very clearly segmented into groups. For this specific application this metric has proven to be inconsistent especially within reduced ranges i.e., 𝑥 ∈ ℝ | 2 ≤ 𝑘 ≤ 8. Calinski Harabasz – This metric for optimal k validation considers the levels of inter-cluster and intra-cluster dispersion and has shown promising, consistent results for 10 candidates of k. In the range 𝑥 ∈ ℝ | 3 ≤ 𝐾 ≤ 12 the Calinski Harabasz index metric performs accurately on datasets of 3, 5, and 6 vocal percussive elements without depending on distinct cluster formations. There are many factors that influence the accuracy of the 𝑘-elbow method. The situations in which this method performs consistently well are documented in further detail in chapter 4.

3.9 Clustering The following cluster analysis functionality is encapsulated within the ClusterAnalysis class. The constructor for the ClusterAnalysis class takes the argument cluster_alg which specifies the type of clustering algorithm which will be applied when the _perform_clustering method is called. The _perform_clustering method takes the arguments reduced_features and n_clusters. This is the principle subspace that is passed as input to the clustering algorithm (the data to be clustered) and the number of output clusters.

K-Means Implementation The k-means model used in this work is implemented with the sklearn.cluster module from the scikit-learn Python library. First, a k-means model is instantiated and a value for 𝑘 (i.e., number of clusters) is passed into the constructor, where the model’s default centroid initialisation is ‘kmeans++’. The principle subspace is then fitted to the model. kmeans = KMeans(n_clusters = 3) kmeans.fit(reduced_features)

Once the model is fitted the output classes can be predicted with the fit_predict method. The kmeans model has a member variable called cluster_centers_ which is an array that holds the final centroid placements, there are useful to plot the centroids when visualising the final clustering. y = kmeans.fit_predict(reduced_features)

21 cluster_label = kmeans.cluster_centers_

Gaussian Mixture Model Implementation GMM is implemented in a very similar way to k-means, from the sci-kit learn sklearn.mixture module. Once again, the model is instantiated, this time with a value for number of Gaussian distributions (which will dictate number of clusters) is passed into the constructor and the model is fitted with the principle subspace. gmm = GaussianMixture(n_components = 3).fit(reduced_features) y = gmm.predict(reduced_features)

In GMM each cluster is based upon a Gaussian distribution, therefore the equivalent of the k-means centroid is the mean of each distribution in the model. The gmm object has a member variable called means_ which stores the resulting mean of each Gaussian distribution, which can again be used to plot cluster centres on visualisation. cluster_label = gmm.means_

The probabilistic properties of GMM allow gmm.predict_proba(pcas) which returns the probability of each datapoint belonging to a certain cluster. GMM is more computationally expensive than k-means, therefore, if the results obtained with GMM aren’t considerably more accurate than those obtained with the k-means model, there is no real advantage of choosing GMM over k-means (Everitt et al., 2011). This is especially the case when applied to real-time applications where computational time is costly and low latency is a requirement.

3.10 Hyperparameter Optimisation Hyperparameters are human specified, application specific variables that influence the outcome of Machine Learning models (Du, Xu and Zhu, 2021). The most basic method of finding an optimal outcome for any specific application is through trial and error i.e., changing the variables by hand and recording which combinations of variables produce the best results. This approach may produce some interesting results but requires the expense of time, recording of the results and, if there are many variables, it is possible that there are optimal combinations that will be missed due to human error. Optuna is a powerful, open source hyperparameter optimisation framework that removes the need for manual auditioning of myriad variable combinations by automating this process. The outcome of each trial is recorded in history as Optuna tests all combinations of hyperparameters in a specified range, attempting to either ‘minimise’ or ‘maximise’ a user defined metric. Examples of hyperparameters are, number_of_mfcc, scaling_type, feature_type and number_of_dimensions. The Optuna objective function takes an Optuna trial as an argument to give objective() access to the trial.suggest methods and encases the machine learning pipeline and its suggested hyperparameters within. In this case, the body of the objective function contains code to create objects of (and pass data between) the classes that capture the full clustering pipeline. These classes are class VocalPercDataset(Dataset) class FeatureExtractor() class FeatureHandling() class ClusterAnalysis() #Optuna script to tune the hyperparameter that maximize rand-index score def objective(trial):

22

#Hyperparameters------------------------------------------------------------------feature_select_1 = trial.suggest_categorical("feature_select_1",["mfcc", "delta_mfcc", "delta_delta_mfcc", "pstc", "sc", "zcr"]) feature_select_2 = trial.suggest_categorical("feature_select_2",["mfcc", "delta_mfcc", "delta_delta_mfcc", "pstc", "sc", "zcr"]) feature_select_3 = trial.suggest_categorical("feature_select_3",["mfcc", "delta_mfcc", "delta_delta_mfcc", "pstc", "sc", "zcr"]) parameter_dict = {} parameter_dict[feature_select_1] = "stack" parameter_dict[feature_select_2] = "stack" parameter_dict[feature_select_3] = "stack" n_mfcc = trial.suggest_int("n_mfcc", 12, 40) feature_scale = trial.suggest_categorical("feature_scale", ["standard", "normal"]) n_dims = trial.suggest_int("n_dims", 2, 8) #Objects--------------------------------------------------------------------------#Dataset Loading ANNOTATIONS_FILE_ = "/Desktop/rachels_dataset/metadata/three_perc.csv" AUDIO_DIR_ = "/Users/philfasan/Desktop/rachels_dataset/audio/" SAMPLE_RATE_ = 44100 vocal_percussion_dataset = VocalPercDataset(ANNOTATIONS_FILE_, AUDIO_DIR_, SAMPLE_RATE_) #create a vocal percussion dataset object #Empty containers for audio, labels, and features audios_ = [] labels_ = [] features_ = [] #Loop to extract audio samples and labels from the VPD class for i in range(len(vocal_percussion_dataset)): audio_,label_ = vocal_percussion_dataset[i] audio_ = audio_.squeeze().numpy() audios_.append(audio_) labels_.append(label_) audios_ = np.array(audios_) labels_ = np.array(labels_) #Feature Extraction #Loop to extract audio features form FE class for x in audios_: feature_extractor_ = FeatureExtractor(x, parameter_dict, n_mfcc) temp_extract_ = feature_extractor_._perform_extraction()

23 features_.append(temp_extract_) features_ = np.array(features_) #Feature Handling feature_handler_ = FeatureHandling(features_, n_dims, "PCA", feature_scale) #create a feature handling object feature_handler_._scale_feature() #call scale features on object feature_handler_._get_reduction() #call get reduction on object pcas_ = feature_handler_.scaled_feature.squeeze() #squeeze scaled/reduced features #Clustering clustering_ = ClusterAnalysis("kmeans") #create a cluster analysis object cluster_predict_, centroids_ = clustering_._perform_clustering(pcas_, 3) #call perform clustering on object return rand_score(labels_, cluster_predict_) study = optuna.create_study(direction="maximize") #maximise the rand_score (best score is 1, worst is 0) study.optimize(objective, n_trials=300)

For the assignment of the above hyperparameter suggestions, two methods of the Optuna trail module are utilised. These are trial.suggest_categorical and trial.suggest_int. The former allows the suggestion of a range of choices for the categorial parameter e.g., feature_select. The latter allows the specification of a low and high threshold for a range of ints, for an integer parameter e.g., n_mfcc. The metric used here (and throughout this work) to score the accuracy of the clustering is the rand_score. A score of 1.0 translates to 100% cluster accuracy when comparing ground truth classes and predicted classes. Accordingly, the objective function’s return seeks to ‘maximise’ the rand_score between the ground truth classes, labels_ (gathered from the dataset’s associated csv file) and the predicted classes, cluster_predict_ (generated within the objective function via a call to the _perform_clustering method of the ClusterAnalysis class).

Real-time Considerations Whilst the fixed duration length of 4096 samples has performed well for proof-of-concept, this research has been undertaken with a real-time application in mind. A worthwhile consideration is the durations that are appropriate when classifying in real-time. Accordingly, experiments were performed for various sample durations each side of the audio onset. In a real-time application it is the trailing audio (after the onset) that is costly in terms of latency. The leading audio (before the onset) is free in this regard. It follows that the goal is to find a suitable balance between latency and clustering accuracy. This trade off will be application specific. For example, a performance tool needs to feel instantaneous, the latency must not reach an audible level (3ms for vocals, 6ms for percussion) (Walker, 2005). With instant results less of a requirement, an application like query-by-vocalisation might favour classification accuracy over low latency. With shorter durations it can be beneficial to re-evaluate the feature sets to better represent the fewer samples the machine learning model will now have at its disposal. Below, the hyperparameter optimisation results are discussed for the three durations of audio, presented in table II.

24 Identifier

Pre-onset Post-onset duration Post-onset duration (samples) duration (ms) (samples) @44.1kHz Control/Baseline 2048 2048 23 Real-time – Low Latency 512 256 5.8 Real-time – Ultra Low Latency 256 128 2.9 Table II. Audio Duration Inputs for Hyperparameter Tuning

4. Results The following results demonstrate the feasibility of the unsupervised classification of vocal percussion for real-time audio durations. The best performing model achieved cluster purity of 100% for the control audio duration, > 94% for real-time low latency and > 93% for real-time ultra-low latency. Complete results with best performing parameters are given in table III. The conversion between accuracy and samples correctly classified for best performing results are presented in table IV. three_perc Baseline

five_perc

six_perc

accuracy

parameters

accuracy

parameters

accuracy

parameters

Kmeans 100%

feature1: 𝑚𝑓𝑐𝑐 feature2: 𝑚𝑓𝑐𝑐∆ feature3:𝑚𝑓𝑐𝑐∆∆ n_mfcc: 12 scaling: standard dimensions: 5 feature1: 𝑚𝑓𝑐𝑐 n_mfcc: 12 scaling: standard dimensions: 5

Kmeans 100%

feature1: 𝑚𝑓𝑐𝑐 feature2: 𝑚𝑓𝑐𝑐∆ feature3:𝑚𝑓𝑐𝑐∆∆ n_mfcc: 12 scaling: standard dimensions: 5 feature1: 𝑚𝑓𝑐𝑐 feature2: sc n_mfcc: 12 scaling: standard dimensions: 5

Kmeans 100%

feature1: 𝑚𝑓𝑐𝑐 feature2: zcr feature3: sc n_mfcc: 15 scaling: standard dimensions: 5

Kmeans 98%

feature1: mfcc feature2: 𝑧𝑐𝑟 feature3: sc n_mfcc: 15 scaling: standard dimensions: 5

Kmeans 92%

feature1: 𝑚𝑓𝑐𝑐 feature2: 𝑚𝑓𝑐𝑐∆ feature3:𝑚𝑓𝑐𝑐∆∆ n_mfcc: 12 scaling: standard dimensions: 5 feature1: 𝑚𝑓𝑐𝑐 feature2: sc feature2: zcr n_mfcc: 12 scaling: standard dimensions: 5 feature1: mfcc feature2: sc n_mfcc: 15 scaling: standard n_dims: 5

GMM 100% RealTime – Low Latency

Kmeans 100%

RealTime – Ultra Low Latency

Kmeans 98%

GMM 100%

GMM 100%

GMM 100% Kmeans 97% GMM 97%

GMM 97%

GMM 100% Kmeans 94% GMM 94%

GMM 93%

Table III. Overall best performing results across three datasets and audio durations

Control/Baseline Low Latency Ultra-Low Latency

three_perc

five_perc

six_perc

108 of 108 samples correctly classified 108 of 108 samples correctly classified 108 of 108 samples correctly classified

180 of 180 samples correctly classified 175 of 180 samples correctly classified 177 of 180 samples correctly classified

216 of 216 samples correctly classified 203 of 216 samples correctly classified 201 of 216 Samples correctly classified

Table IV. Conversion from Accuracy scores to Number of Samples Correctly Classified A combination of Mel Frequency Cepstral Coefficients (MFCC), Spectral Centroid (SC) and ZeroCrossing Rate (ZCR) have shown overwhelming success, particularly for the real-time audio durations. For longer audio durations, delta accelerated features that model the trajectories of the MFCC over frames have performed highly. This can be attributed to the higher number of samples and therefore

25 frames gathered from a longer audio duration. The audio features that perform well for the real-time durations appear to do so owing to the silence/zeroed samples that are discarded in the process of obtaining a shorter audio duration. Variations of low-level noise or silence can greatly affect the correctness of ZCR and SC calculations. For ultra-low latency, figures 11, 14, and 16 demonstrate the ability of SC features to separate vocal percussive elements. Good separation is exhibited across all elements, except for the snare and closed hi-hat, whose spectral centroids are closely related and show less of a partition. The addition of the ZCR features gives an observable separation for the snare and closed hi-hat (fig 12).

Figure 11. Spectral Centroids for Three Vocal Percussive Elements (Ultra-low Latency)

Figure 12. Zero-Crossing Rate for Three Vocal Percussive Elements (Ultra-low Latency)

26

Figure 13. Spectral Centroids for Five Vocal Percussive Elements (Ultra-low Latency)

Figure 14. Zero-Crossing Rate for Five Vocal Percussive Elements (Ultra-low Latency)

Figure 15. Spectral Centroids for Six Vocal Percussive Elements (Ultra-low Latency)

27

Fig 16. Zero-Crossing Rate for Six Vocal Percussive Elements (Ultra-low Latency)

Hyperparameter Tuning for K-Means Clustering This round of trials determines the best performing hyperparameters for the k-means clustering model. The audio durations in table II were investigated for each vocal percussive dataset. Having already performed thorough manual hyperparameter testing prior to the implementation of Optuna, and having found PCA to produce consistently optimal results, the decision was made to hard code the reduction type category but tune the number of dimensions. Additionally, the SOM implementation performed inconsistently throughout the project, especially for real-time durations. For this reason, the hyperparameter tuning stage focuses on optimising only k-means and GMM. Previously, a metric upon which to measure the objective function was introduced. For this work, success is measured by a RandIndex score. The objective function is therefore set up to find the combination of hyperparameters that maximise the Rand-Index score. A full overview of best performing hyperparameters for k-means clustering and the corresponding Rand-Index scores can be found in table V. three_perc

Baseline

Real-Time – Low Latency

Real-Time – Ultra Low Latency

five_perc

RandIndex Score

hyperparameter Settings

accuracy

1.0

feature1: mfcc feature2: ∆mfcc feature3: ∆∆mfcc n_mfcc: 24 scaling: standard dimensions: 5 feature1: mfcc feature2:∆∆mfcc n_mfcc:18 scaling: standard dimensions: 5

1.0

hyperparameter Settings

six_perc accuracy

feature1: mfcc 1.0 feature2:∆∆mfcc feature3: sc n_mfcc: 21 scaling: standard dimensions: 5 1.0 0.96 feature1: mfcc 0.94 feature2:∆∆mfcc feature3: sc n_mfcc: 33 scaling: standard dimensions: 5 1.0 feature1: mfcc 0.97 feature1: mfcc 0.92 feature2: zcr feature2: zcr feature3: sc feature3: sc n_mfcc: 15 n_mfcc: 15 scaling: standard scaling: standard dimensions: 4 dimensions: 5 Table V. Hyperparameter Tuning Results for Singular Features

hyperparameter settings

feature1: mfcc feature2: ∆mfcc feature2: zcr n_mfcc: 16 scaling: standard dimensions: 8 feature1: mfcc feature2: zcr n_mfcc: 32 scaling: standard dimensions: 5 feature1: mfcc feature2: zcr n_mfcc: 14 scaling: standard dimensions: 5

28

Hyperparameter Tuning for Gaussian Mixture Modelling This round of trials determines the best performing Optuna hyperparameters for Gaussian Mixture Modelling (GMM). Once again, the audio durations in table II where investigated for each vocal percussive dataset. Several of the feature sets introduced in chapter 3.4 consist of the aggregation of multiple features. For this reason it is useful to assess whether Optuna can derive an optimal combination of features that have not yet been considered. three_perc

Baseline

five_perc

six_perc

RandIndex Score

hyperparameter Settings

accuracy

hyperparameter Settings

accuracy

1.0

feature1: mfcc feature2: ∆mfcc feature3: 𝑠𝑐 n_mfcc: 19 scaling: standard dimensions: 5 feature1: mfcc n_mfcc: 25 scaling: standard dimensions: 5

1.0

feature1: mfcc feature2: ∆mfcc feature3:∆∆mfcc n_mfcc: 14 scaling: standard dimensions: 2 feature1: mfcc feature3: sc n_mfcc: 23 scaling: normal dimensions: 5

1.0

hyperparameter settings

feature1: mfcc feature2: ∆∆mfcc feature3: sc n_mfcc: 15 scaling: standard dimensions: 5 Real-Time 1.0 0.97 0.94 feature1: mfcc – Low feature2: ∆∆mfcc Latency feature3: zcr n_mfcc: 33 scaling: standard dimensions: 5 Real-Time 1.0 feature1: mfcc 0.97 feature1: mfcc 0.93 feature1: mfcc – Ultra feature2: zcr feature2: zcr feature2: zcr Low feature3: sc n_mfcc: 15 feature3: sc Latency n_mfcc: 15 scaling: standard n_mfcc: 18 scaling: standard dimensions: 5 scaling: standard dimensions: 4 dimensions: 5 Table VI. Hyperparameter Tuning Results for Gaussian Mixture Modelling

The perceptually motivated MFCC features have consistently produced the most accurate results across the three datasets and audio durations. A further insight gleaned from the hyperparameter optimisation results is that, in most cases, the MFCC features performance is increased by the aggregation of cherry picked spectral and temporal features or MFCC delta coefficients. Interpreting the Results The clustering accuracy achieved in this work on real-time audio durations gives an impression of how reliably an implementation of this pipeline would perform in a real-world application. The Rand-Index scoring metric ranges between 0 – 1.0, with a score of 1.0 translating to 100% cluster purity. Comparing the rand index scores presented in table VI and table V to the accuracy results in table III, it can be concluded that the hyperparameter tuning results were able to be successfully replicated as manual trials, without loss of accuracy. Standardising the features to have mean 0 and unit variance has drastically outperformed its alternative of normalisation between 0 – 1. Five dimensions of principle components have proved adequate throughout to achieve data separation conducive with k-elbow and clustering accuracy. Though it is not possible to visualise the principle subspace in five dimensions, the clustering results can be visualised in both two and three dimensions to give an impression of inter-dimensional separation. Complete clustering visualisations for all models are given in appendix I, II & III.

29

Figure 17. k-means Clustering for Six Vocal Percussive Elements in Two-Dimensional Principle Subspace (Ultra-Low Latency)

Figure 18. k-means Clustering for Six Vocal Percussive Elements in Three-Dimensional Principle Subspace (Ultra-Low Latency)

Feature Performance PSTC This work saw the implementation of PSTC audio features. These were found not to perform as consistently well when applied to the task of segmenting vocal percussive data. This is due in part to their inability to well represent the vocal percussive datasets across all classes, resulting in poor separation in the principle subspace (separation in the feature space being a key indicator that the outcome of the clustering stage will yield reliable results). For this reason, the introduction of the PSTC

30 features appeared to confuse the k-elbow metric which had previously been working reliably for MFCC and MFCC+ feature sets. K-Elbow Method The k-elbow method, coupled with the Calinski Harabasz metric, correctly predicted a value for k for all best performing parameters detailed in table III, aside from the case of ultra-low latency for six percussive elements. The case at which the k-elbow method was unable to identify the correct value for k (six elements, ultralow latency) is designed to challenge the limits of the capability of the clustering pipeline. Here the model is required to successfully classify six classes on a dataset comprised of a significantly reduced (when compared with the control) number of audio samples. When quantifying the trade-off between classification accuracy and latency reduction, it can be observed in table IV that for a dataset of six vocal percussive elements, the reduction of samples upon which to classify the data from 2048 (23ms after onset) to 128 (2.9ms after onset) results in 15 fewer correctly classified samples, a reduction in clustering accuracy of 7%. This is a 7% reduction in classification accuracy for an 87% decrease in latency. For five vocal percussive elements this translates to a 2% reduction in accuracy for an 87% decrease in latency. Both trade-offs appear highly advantageous in pursuit to lower latency but striking a balance between the two ultimately depends on the application and size of the dataset i.e., 7% reduction in accuracy on a 1000 element dataset results in 70 misclassified data points.

31

Figure 19. Example of Distinct Cluster Formations (top) Upon Which the k-elbow Metric Performs Well (bottom)

32

Figure 20. Example of Cluster Formations that are not Clearly Defined (top) Upon Which the kelbow Metric Performs Poorly (bottom)

5. Conclusion This paper focused on the analysis and automatic class determination of vocal percussive sounds for offline and online audio durations. First, three vocal percussive datasets were curated containing three, five and six vocal percussive classes. Ground truth class tags were provided in the CSV files associated with the datasets to enable the use of the Rand-Index performance metric, crucial in the configuration of the clustering pipeline. Frame-based features were then extracted, engineered, and reduced in

33 dimensionality to obtain a lower-dimensional principle subspace upon which three unsupervised learning techniques were explored for clustering the data. Three audio durations were also compared, with a focus on real-time low latency. Finally, the implementation of hyperparameter tuning via the Optuna framework allowed for the optimisation of hyperparameters to maximise the Rand-Index score (i.e., identify hyperparameters that achieve the highest cluster purity). As far as is known, this is the first comparative study to evaluate model and centroid-based, partitional clustering for classification of HBB data. The planned next steps for this work focus on dataset testing and real-time implementation. The main priority of the testing phase will be the introduction of datasets with multiple participants, and the extension of current datasets with non-linguistic, inhaled and exhaled vocal events common in HBB. The aim of this testing phase will be to incrementally generalise the algorithm through exposure to multiple datasets and vocal percussion both semi-professional and amateur to account for a userbase with varying ability. The real-time implementation will require a development phase of fine tuning the pipeline and translation to a real-time language (i.e., C++). Considering the unpredictable results of the k-elbow method for ultra-low latency durations there is an opportunity to explore alternative methods for predicting number of components. The high performance of the model-based GMM clustering in this work makes a case for the implementation of GMM over k-means in a real-time application. In this case there exists an alternative, high accuracy method for the prediction of number of components for the GMM algorithm named Bayesian Information Criterion (BIC), a method used in linear regression (McLachlan and Rathnayake, 2014). If the currently used models prove to be too computationally heavy for real-time, Olukanmi, Nelwamondo and Marwala (2018) suggest a scalable adaptation of classic k-means entitled ‘k-means lite’. This model is based on the extension of Central Limit Theory (CLT) and prioritises large datasets in real-time applications. In the case that the next dataset testing phase proves the current pipeline to be considerably less accurate for generalised cases there is an opportunity to explore unsupervised, deeplearning based alternatives i.e., a pipeline involving the segmentation of vocal percussive classes based on sematic meaning, extracted employing autoencoder architecture (Chung, et al., 2016).

34

References Acero, A., Huang, X. and Hon, H. (2001) Spoken Language Processing: A Guide to Theory, Algorithm, and System Development. 1st edn. New Jersey: Prentice Hall. Bachem, O., Lucic, M. and Krause, A. (2017) ‘Distributed and Provably Good Seedings for k-Means in Constant Rounds’. PMLR. Baniya, B. K., Lee, J. and Li, Z.-N. (2014) ‘Audio feature reduction and analysis for automatic music genre classification’, in. IEEE (2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC)). doi: 10.1109/SMC.2014.6973950. Bishop, C. M. (2010) Pattern recognition and machine learning. 10th edn. New York: Springer. Brent, W. (2010) Physical and Perceptual Aspects of Percussive Timbre. University of California. Cabral, F. S., Fukai, H. and Tamura, S. (2019) ‘Feature Extraction Methods Proposed for Speech Recognition Are Effective on Road Condition Monitoring Using Smartphone Inertial Sensors’, 19(16). doi: 10.3390/s19163481. Choi, K. et al. (2018) ‘A Comparison of Audio Signal Preprocessing Methods for Deep Neural Networks on Music Tagging’, in. EURASIP (2018 26th European Signal Processing Conference (EUSIPCO)). doi: 10.23919/EUSIPCO.2018.8553106. Chung, Y.-A. et al. (2016) ‘Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder’. Du, X., Xu, H. and Zhu, F. (2021) ‘Understanding the Effect of Hyperparameter Optimization on Machine Learning Models for Structure Design Problems’, 135. doi: 10.1016/j.cad.2021.103013. Dusan, S. and Deng, L. (1998) ‘Recovering Vocal Tract Shapes from MFCC Parameters’. Evain, S. et al. (2021) ‘Human beatbox sound recognition using an automatic speech recognition toolkit’, 67, p. 102468. doi: 10.1016/j.bspc.2021.102468. Everitt, B. et al. (2011) Cluster Analysis. Chichester: Wiley (Wiley series in probability and statistics). Ghahramani, Z. (2004) Unsupervised Learning. Gonzalez, R. (2012) Better Than MFCC Audio Classification Features. New York, NY: Springer (The Era of Interactive Media). doi: 10.1007/978-1-4614-3501-3_24. Hastie, T., Tibshirani, R. and Friedman, J. H. (2009) The elements of statistical learning. 2nd edn. New York: Springer (Springer series in statistics). Hazan, A. (2005) ‘Towards Automatic Transcription of Expressive Oral Percussive Performances’, in. ACM (IUI ’05), pp. 296–298. doi: 10.1145/1040830.1040904. Jiang, N. and Liu, T. (2020) ‘An Improved Speech Segmentation and Clustering Algorithm Based on SOM and K-Means’, 2020. doi: 10.1155/2020/3608286.

35 Kapil, S. and Chawla, M. (2016) ‘Performance Evaluation of K-means Clustering Algorithm with Various Distance Metrics’, in. Piscataway: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) (The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Conference Proceedings). Kaur, C. and Kumar, R. (2017) ‘Study and Analysis of Feature Based Automatic Music Genre Classification Using Gaussian Mixture Model’, in. Piscataway: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) (The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Conference Proceedings). Knees, P. and Schedl, M. (2016) Music Similarity and Retrieval. Berlin, Heidelberg: Springer Berlin (The information retrieval series). Lakatos, S. (2000) ‘A Common Perceptual Space for Harmonic and Percussive Timbres’, 62(7). doi: 10.3758/BF03212144. Likas, A., Vlassis, N. and J. Verbeek, J. (2003) ‘The global k-means clustering algorithm’, 36(2). doi: 10.1016/S0031-3203(02)00060-2. McDonald, S. and Tsang, C.P. (1997) ‘Percussive Sound Identification Using Spectral Centre Trajectories’, Proceedings of 1997 Postgraduate Research Conference. McLachlan, G. J. and Rathnayake, S. (2014) ‘On the Number of Components in a Gaussian Mixture Model’, 4(5). doi: 10.1002/widm.1135. Mehrabi, A. (2018) Vocal Imitation for Query by Vocalisation. Queen Mary University of London. Mehrabi, A., Dixon, S. and Sandler, M. (2019) ‘Vocal Imitation of Percussion Sounds: On the Perceptual Similarity Between Imitations and Imitated Sounds’, 14(7). doi: 10.1371/journal.pone.0219955. Micheyl, C. and Oxenham, A. J. (2010) ‘Pitch, harmonicity and concurrent sound segregation: Psychoacoustical and neurophysiological findings’, 266(1). doi: 10.1016/j.heares.2009.09.012. Mitrović, D., Zeppelzauer, M. and Breiteneder, C. (2010) Features for Content-Based Audio Retrieval. Elsevier Science & Technology (Advances In Computers). doi: 10.1016/S00652458(10)78003-7. Olukanmi, P. O., Nelwamondo, F. and Marwala, T. (2018) ‘k-Means-Lite: Real Time Clustering for Large Datasets’, in. IEEE (2018 5th International Conference on Soft Computing & Machine Intelligence (ISCMI)). doi: 10.1109/ISCMI.2018.8703210. Picart, B., Brognaux, S. and Dupont, S. (2015) ‘Analysis and Automatic Recognition of Human Beatbox sounds: A Comparative Study’, in. IEEE (2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)), pp. 4255–4259. doi: 10.1109/ICASSP.2015.7178773. Shutaywi, M. and Kachouie, N. N. (2021) ‘Silhouette Analysis for Performance Evaluation in Machine Learning with Applications to Clustering’, 23(6). doi: 10.3390/e23060759. Sinha, A. and Soni, H. (2021) ‘Segmentation and Classification of Beatboxing Acoustic Voice Tract Variations in MRI through Image Processing Technique’. MDPI AG. doi: 10.20944/preprints202105.0127.v1. Somashekara, M. T. and Manjunatha, D. (2014) ‘Performance Evaluation of Spectral Clustering Algorithm using Various Clustering Validity Indices’, 5(6).

36 Stowell, D. (2010) Making Music Through Real-time Voice Timbre Analysis: Machine Learning and Timbral Control. Queen Mary University of London. Stowell, D. and Plumbley, M. D. (2008) ‘Characteristics of the beatboxing vocal style’. Stowell, D. and Plumbley, M. D. (2010) ‘Delayed Decision-making in Real-time Beatbox Percussion Classification’, 39(3). doi: 10.1080/09298215.2010.512979. Sunu, J. and Percus, A. G. (2018) ‘Dimensionality Reduction for Acoustic Vehicle Classification with Spectral Embedding’, in. IEEE (2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC)), pp. 1–5. doi: 10.1109/ICNSC.2018.8361290. Syakur, M. A. et al. (2018) ‘Integration K-Means Clustering Method and Elbow Method for Identification of The Best Customer Profile Cluster’, 336(1). doi: 10.1088/1757-899X/336/1/012017. Tzanetakis, G. and Kapur, A. (2004) ‘Query-By-Beat-Boxing: Music Retrieval for The DJ’. Zenodo. doi: 10.5281/zenodo.1418032. Umesh, S., Cohen, L. and Nelson, D. (2002) ‘Frequency Warping and the Mel Scale’, 9(3). doi: 10.1109/97.995829. Vassilvitskii, S. and Arthur, D. (2007) ‘k-means++: The Advantages of Careful Seeding’, in. Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. doi: 10.1145/1283383.1283494. Velardo, V 2021, ‘Beyond the Code’, The Audio Programmer Virtual Meetup. Available from: https://www.youtube.com/watch?v=mlRXBzFwQEs [14 September 2021] Winursito, A., Hidayat, R. and Bejo, A. (2018) ‘Improvement of MFCC feature extraction accuracy using PCA in Indonesian speech recognition’, in. IEEE (International Conference on Information and Communications Technology (ICOIACT)). doi: 10.1109/ICOIACT.2018.8350748.

Appendix I.

Baseline/Control

37

38

39

II.

Low Latency

40

41

III.

Ultra-Low Latency

42

43

44