No part of this publication may be reproduced or transmi ed in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher's permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a ma er of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.
Library of Congress Cataloging-in-Publication Data
A catalog record for this book is available from the Library of Congress
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
ISBN: 978-0-12-822688-9
For information on all Academic Press publications visit our website at h ps://www.elsevier.com/books-and-journals
Publisher: Mara Conner
Acquisitions Editor: Tim Pi s
Editorial Project Manager: Fernanda A. Oliveira
Production Project Manager: Kamesh Ramajogi
Designer: Vicky Pearson Esser
Typeset by VTeX
Dedication
To God, my wife, Daiane, and my sons, Bento and Mateo (coming soon). Last but not least, to my father, Mauro (In memoriam), to my mother, Maria, to my sisters, and the whole family.
João Paulo Papa
I would like to dedicate this work to my mother, Darcy, wife Paula, daughter, Clara, and son, Pedro.
Alexandre Xavier Falcão
List of contributors
Luis C.S. Afonso UNESP – São Paulo State University, School of Sciences, Bauru, Brazil
David Aparco-Cardenas Institute of Computing, University of Campinas (UNICAMP), Campinas, São Paulo, Brazil
Hamid Bostani
Young Researchers and Elite Club, South Tehran Branch, Islamic Azad University, Tehran, Iran
Digital Security Group, Radboud University, Nijmegen, The Netherlands
Rafael S. Bressan Department of Computing, Federal University of Technology – Parana, Cornelio Procopio, Brazil
Pedro H. Buga i Department of Computing, Federal University of Technology – Parana, Cornelio Procopio, Brazil
Kelton Costa São Paulo State University, Department of Computing, Bauru, Brazil
Pedro Jussieu de Rezende Institute of Computing, University of Campinas (UNICAMP), Campinas, São Paulo, Brazil
Gustavo H. de Rosa Department of Computing, São Paulo State University, Bauru, Brazil
Luis A. de Souza Jr. Department of Computing, São Carlos Federal University, São Carlos, Brazil
Alexandre Xavier Falcão Institute of Computing, University of Campinas (UNICAMP), Campinas, São Paulo, Brazil
Danilo Samuel Jodas Department of Computing, São Paulo State University, Bauru, Brazil
João Paulo Papa
UNESP – São Paulo State University, School of Sciences, Bauru, Brazil
Department of Computing, São Paulo State University, Bauru, Brazil
Leandro Aparecido Passos Department of Computing, São Paulo State University, Bauru, Brazil
Rafael Pires Department of Computing, São Paulo State University, Bauru, Brazil
Mateus Roder Department of Computing, São Paulo State University, Bauru, Brazil
Priscila T.M. Saito Department of Computing, Federal University of Technology – Parana, Cornelio Procopio, Brazil
Rafał Scherer Czestochowa University of Technology, Department of Computing, Częstochowa, Poland
Mansour Sheikhan Department of Electrical Engineering, South Tehran Branch, Islamic Azad University, Tehran, Iran
Marcos Cleison Silva Santana Department of Computing, São Paulo State University, Bauru, Brazil
Luan Utimura São Paulo State University, Department of Computing, Bauru, Brazil
Biography of the editors
Alexandre
Xavier Falcão
Alexandre Xavier Falcão (lids.ic.unicamp.br) is a full professor at the Institute of Computing (IC), University of Campinas (Unicamp), where he has worked since 1998.
He a ended the Federal University of Pernambuco from 1984–1988, where he received a B.Sc. in Electrical Engineering. He then a ended Unicamp, where he got an M.Sc. (1993), and a Ph.D. (1996), in Electrical Engineering, by working on volumetric data visualization and medical image segmentation. During his Ph.D., he worked with the Medical Image Processing Group at the University of Pennsylvania from 1994–1996. In 1997, he developed video quality assessment methods for Globo TV. In 2011–2012, he spent a one-year sabbatical at the Robert W. Holley Center for Agriculture and Health (USDA, Cornell University), working on image analysis applied to plant biology.
He served as Associate Director of IC-Unicamp (2006–2007), Coordinator of its Post-Graduation Program (2009–2011), and Senior Area Editor of IEEE Signal Processing Le ers (2016–2020). He is currently a research fellow at the top level for the Brazilian National Council for Scientific and Technological Development (CNPq), President of the Special Commission of Computer Graphics and Image Processing (CEGRAPI) for the Brazilian Computer Society (SBC), and Area Coordinator of Computer Science for the Sao Paulo Research Foundation (FAPESP).
Among several awards, it is worth mentioning three Unicamp inventor awards at the category ”License Technology” (2011, 2012, and 2020), three awards of academic excellence (2006, 2011, 2016)
from IC-Unicamp, one award of academic recognition ”Zeferino Vaz” from Unicamp (2014), and the best paper award in the year of 2012 from the journal “Pa ern Recognition” (received at Stockholm, Sweden, during the conference ICPR 2014).
His research work aims at computational models to learn and interpret the semantic content of images in the domain of several applications. The areas of interest include image and video processing, data visualization, medical image analysis, remote sensing, graph algorithms, image annotation, organization, and retrieval, and (interactive) machine learning and pa ern recognition (h ps://scholar.google.com/citations? user=HTFEUaUAAAAJ&hl=en).
João Paulo Papa
João Paulo Papa received his B.Sc. in Information Systems from the São Paulo State University, SP, Brazil. In 2005, he received his M.Sc. in Computer Science from the Federal University of São Carlos, SP, Brazil. In 2008, he received his Ph.D. in Computer Science from the University of Campinas, SP, Brazil. During 2008–2009, he had worked as a post-doctorate researcher at the same institute, and during 2014–2015 he worked as a visiting scholar at Harvard University. He has been an Associate Professor at the Computer Science Department, São Paulo, State University, since 2016, and his research interests include machine learning, pa ern recognition, and image processing. He is also the recipient of the Alexander von Humboldt fellowship and an IEEE Senior Member.
Among several awards, it is worth mentioning the best paper award in the year of 2012 from the journal “Pa ern Recognition” (received at Stockholm, Sweden, during the conference ICPR 2014, and the Unicamp inventor award at the category ”License Technology” in 2019.
He is currently a research fellow for the Brazilian National Council for Scientific and Technological Development (CNPq), Senior Area Editor of IEEE Signal Processing Le ers, Brazilian representative at the International Association for Pa ern Recognition, and Associate
Editor for the following journals: Computers in Biology and Medicine, SN Computer Science. He is also member of the advisory board of the Integrated Computer-Aided Engineering.
Preface
The Optimum-Path Forest (OPF) story dates back to 2004 with a paper called “The image foresting transform: Theory, algorithms, and applications” published on the well-known IEEE Transactions on Pa ern Analysis and Machine Intelligence, IEEE PAMI for short. The Image Foresting Transform (IFT) came up to deal with image analysis based on the comprehensive framework provided by Graph Theory. By modeling pixels as graph nodes and establishing a proper adjacency relation, IFT became a powerful tool for multiscale skeletonization, distance transforms, morphological reconstructions, and image segmentation.
A mindful reader may be aware that a fine line separates image segmentation from image classification. Pixels become feature vectors and segmenting an object turns out to be classifying (labeling) its pixels accordingly. Half a decade later, in 2009, one of the most prominent papers concerning the OPF was published. What followed then was many papers that either tried to apply OPF on a different application or to improve its learning procedure. Remote sensing, medicine, speech recognition, and engineering are just a few applications that benefited from the OPF framework.
Now the family has grown. The reader can find supervised, semisupervised, and unsupervised variants of the OPF classifier. For sure, deep learning has not been forgo en. Papers have shown how to replace softmax layers with an OPF classifier with considerably higher accuracies. We tried, in this book, to provide a complete workflow of OPF history, i.e., we refreshed our minds with an exciting survey of works and a brief theoretical background to open the reader's mind to the OPF world. We hope this book will serve as
introductory and advanced material so that newcomers and experienced researchers can take advantage of the OPF capabilities.
As we usually say, OPF is not a classifier but a framework for designing classifiers based on optimum-path forest. By just changing some pieces, you can design your own OPF classifier and enjoy it! Which one are you going to pick?
The authors
Chapter 1: Introduction
Alexandre Xavier Falcãoa; João Paulo Papab aInstitute of Computing, University of Campinas (UNICAMP), Campinas, São Paulo, Brazil bUNESP – São Paulo State University, School of Sciences, Bauru, Brazil
Abstract
Pa ern recognition techniques have been consistently applied in various domains, ranging from remote sensing and medicine to engineering, among others. The literature is vast and dominated mainly by Neural Networks in the past, followed by Support Vector Machines, and recently by the so-called “Deep Learning” approaches. However, there is always room for improvements and novel techniques, for there is no approach that can lead to the best results in all situations. This chapter introduces this book, which concerns the Optimum-Path Forest, a framework for designing graph-based classifiers based on optimum connectivity among samples. We highlight new ideas and applications throughout the book, and future trends will foster the related literature in the following years.
Pa ern recognition techniques have been consistently applied in a broad range of domains, varying from remote sensing and medicine to engineering, among others. The literature is vast and dominated
mainly by Neural Networks [1] in the past, followed by Support Vector Machines [2], and recently by the so-called “Deep Learning” approaches [3]. These la er techniques shifted the way we think about problem engineering. Instead of handcrafting features, we can use raw data to feed models that learn the best information that the data describes. On the other hand, we need considerably large data sets to train the models, which are not always available.
According to the well-known “No Free Lunch Paradigm,” there is no single technique that shall fit be er in all situations and data sets. Motivated by such assumption, Papa et al. [4] presented the first version of the supervised Optimum-Path Forest (OPF) classifier, which models data classification as a graph partition problem. The nodes denote samples (i.e., feature vectors) while edges encode the strength of connectivity among them. The idea is to partition the graph into optimum-path trees (OPTs) such that examples from the same class belong to the same tree. Representative samples called “prototypes” are the roots of such OPTs, and they rule the competition process that ends up partitioning the data set. The approach proposed in 2017 showed to be robust and fast for training, besides being parameterless.1
We prefer to understand OPF as a framework for designing classifiers based on optimum-path forests rather than a sole classifier. The rationale behind that concerns the fact that OPF is customizable and, therefore, different versions are obtained by choosing a few hyperparameters. The first one is related to the adjacency relation, which is usually set to a complete graph or a k-nn model. Usually, the la er approach provides an intrinsic representation of the data set as a whole, but it comes with the price of computing the k-nearest neighbors for each example. In 2008, Papa and Falcão [5] proposed the first supervised OPF based on such neighborhood formulation, with results that outperformed standard OPF in some situations. The second hyperparameter stands for the methodology to estimate prototypes. A simple and not effective approach is to pick some at random, but initial experiments have shown us that it may lead to poor generalization over unseen
examples. Throughout the book, the reader can observe that different approaches can be used to find prototypes, each depending on the preliminary information we have from the data set. Last but not least, the competition process is, essentially, an optimization problem. Therefore, a path-cost function must be adopted to guide the OPF training algorithm into finding a suitable partition of the feature space.
New problems require adaptations. In 2009, Rocha et al. [6] proposed the unsupervised OPF. In the absence of labels, the approach proposed by Papa et al. [7] in the very same year is no longer possible since it requires information form labels beforehand. On the contrary, unsupervised OPF aims at finding regions with the highest density such that prototypes are then picked from these places. In this version, both nodes and arcs are weighted, and a kneighborhood relation is used to compute a probability density function over each training sample. Semisupervised learning has also been addressed in work by Amorim et al. [8], in which unlabeled training samples are employed to improve classification.
After a considerable number of publications since its first publication, we observed that supervised OPF with a complete graph could improve during training. In 2012, Papa et al. [9] published such an improvement that sorts training samples according to their costs, i.e., examples with the lowest costs are placed on the first positions of the list. When a test sample appears to be classified, we start evaluating the training samples placed in the first positions first.2 From beyond this point, we could observe OPF has gained popularity around the globe.
The k-neighborhood adjacency relation was further explored by Papa et al. [10]. Inspired by the unsupervised OPF [6], this version also weights nodes and arcs, and prototypes placed at the top of regions with maximum density begin the competition process. This approach's restriction is to estimate the value of k, which is usually accomplished over a validating set [11]. Besides, we observed that ensembles of OPF classifiers composed of versions with different
modelings for the adjacency relation could improve effectiveness [12].
Speaking about ensembles, we can refer to some interesting works. Ponti and Papa [13] showed that both OPF training time and accuracy could be enhanced when trained over disjoint training sets. Later on, Ponti and Rossi [14] demonstrated that one could reduce training sets using ensembles of OPF-based classifiers. The subject of reducing training sets has been of considerable importance. In 2010, Papa et al. [15] proposed an approach aiming to obtain reduced but relevant training sets. The idea is to use a validation set to mark training samples that did not participate in any classification process. These samples are thought to be irrelevant, and thus they can be discarded from the learning step. Years later, Fernandes and Papa [16] proposed a similar approach, but to deal with tie zones, i.e., regions in the feature space that contain test samples that have been offered equal costs from training samples during the competition process. Let and be the cost that training samples A and B offered to a given test sample x, respectively, such that . We addressed the following question in this work: if , is there any way to measure both samples' confidence A and B such that B is preferable to A when conquering x? In other words, we aim to learn a confidence measure for each training sample such that the most reliable samples are prone to conquer others, not only the ones that offer the best (i.e., minimum) cost.
There are many other interesting works related to OPF-based classifiers we can refer to. Obviously, there are quite a few shortcomings that are not yet addressed, such as dealing with sparse and high-dimensional feature spaces, among others. On the other hand, OPF has shown to guide Convolutional Neural Networks into be er results using semisupervised learning in work by Amorim et al. [17], which aimed at coping with the problem of soybean leaf and herbivorous pest identification.
The main idea of this book is to shed light on recent advances in the context of optimum-path forest-based classification and to provide a concise review of the literature from the past years. We
p p y hope this book can also serve as a document to undergraduate and graduate levels, as well as senior researchers. The book comprises applications on several domains, such as intrusion detection in computer networks, active and metric learning, and medicine-driven problems.
References
[1] S. Haykin, Neural Networks: A Comprehensive Foundation. 3rd edition Upper Saddle River, NJ, USA: Prentice-Hall, Inc.; 2007.
[2] C. Cortes, V. Vapnik, Support vector networks, Machine Learning 1995;20:273–297.
[3] Y. LeCun, Y. Bengio, G.E. Hinton, Deep learning, Nature 2015;521(7553):436–444.
[4] J. Papa, A. Falcão, P. Miranda, C. Suzuki, N. Mascarenhas, Design of robust pa ern classifiers based on optimum-path forests, Mathematical Morphology and Its Applications to Signal and Image Processing (ISMM). MCT/INPE; 2007:337–348.
[5] J.P. Papa, A.X. Falcão, A new variant of the optimum-path forest classifier, G. Bebis, R. Boyle, B. Parvin, D. Koracin, P. Remagnino, F. Porikli, J. Peters, J. Klosowski, L. Arns, Y. Chun, T.-M. Rhyne, L. Monroe, eds. Advances in Visual Computing. Lecture Notes in Computer Science. Springer Berlin Heidelberg; 2008;vol. 5358:935–944.
[6] L.M. Rocha, F.A.M. Cappabianco, A.X. Falcão, Data clustering as an optimum-path forest problem with applications in image analysis, International Journal of Imaging Systems and Technology 2009;19(2):50–68.
[7] J.P. Papa, A.X. Falcão, C.T.N. Suzuki, Supervised pa ern classification based on optimum-path forest, International Journal of Imaging Systems and Technology 2009;19(2):120–131.
[8] W.P. Amorim, A.X. Falcão, J.P. Papa, M.H. Carvalho, Improving semi-supervised learning through optimum connectivity, Pa ern Recognition 2016;60:72–85.
[9] J.P. Papa, A.X. Falcão, V.H.C. Albuquerque, J.M.R.S. Tavares, Efficient supervised optimum-path forest classification for large datasets, Pa ern Recognition 2012;45(1):512–520.
[10] J.P. Papa, S.E.N. Fernandes, A.X. Falcão, Optimum-path forest based on k-connectivity: theory and applications, Pa ern Recognition Le ers 2017;87:117–126.
[11] J.P. Papa, A.X. Falcão, A learning algorithm for the optimum-path forest classifier, A. Torsello, F. Escolano, L. Brun, eds. Graph-Based Representations in Pa ern Recognition. Lecture Notes in Computer Science. Springer Berlin Heidelberg; 2009;vol. 5534:195–204.
[12] P.B. Ribeiro, J.P. Papa, R.A.F. Romero, An ensemble-based approach for breast mass classification in mammography images, SPIE Medical Imaging. 2017:101342N-1–101342N-8.
[13] M.P. Ponti, J.P. Papa, Improving accuracy and speed of optimum-path forest classifier using combination of disjoint training subsets, C. Sansone, J. Ki ler, F. Roli, eds. Multiple Classifier Systems. Berlin, Heidelberg: Springer Berlin Heidelberg; 2011:237–248.
[14] M.P. Ponti, I. Rossi, Ensembles of optimum-path forest classifiers using input data manipulation and undersampling, Z.-H. Zhou, F. Roli, J. Ki ler, eds. Multiple Classifier Systems. Berlin, Heidelberg: Springer Berlin Heidelberg; 2013:236–246.
[15] J. Papa, A. Falcão, G. de Freitas, A. Avila, Robust pruning of training pa erns for optimum-path forest classification applied to satellite-based rainfall occurrence estimation, IEEE Geoscience and Remote Sensing Le ers 2010;7(2):396–400.
[16] S.E.N. Fernandes, J.P. Papa, Improving optimum-path forest learning using bag-of-classifiers and confidence measures, Pa ern Analysis & Applications 2019;22:703–716.
[17] W. Amorim, E. Tetila, H. Pistori, J. Papa, Semi-supervised learning with convolutional neural networks for uav images automatic recognition, Computers and Electronics in Agriculture 2019;164, 104932.
1 “The authors are grateful to Celso Tetsuo Nagase Suzuki for his former implementation of the OPF classifier.”
2 “The authors are grateful to Thiago Spina for his ideas and insightful comments on this work.”
Chapter 2: Theoretical background and related works
Luis C.S.
Afonsoa; Alexandre Xavier Falcãob; João Paulo Papaa aUNESP –São Paulo State University, School of Sciences, Bauru, Brazil bInstitute of Computing, University of Campinas (UNICAMP), Campinas, São Paulo, Brazil
Abstract
The Optimum-Path Forest (OPF) is a framework for the design of graph-based classifiers, which covers supervised, semisupervised, and unsupervised applications. The OPF is mainly characterized by its low training and classification times as well as competitive results against well-established machine learning techniques, such as Support Vector Machine and Artificial Neural Networks. Besides, the framework allows the design of different approaches based on the problem itself, which means a specific OPF-based classifier can be built for a given particular task. This paper surveyed several works published in the past years concerning OPF-based classifiers and sheds light on future trends concerning such a framework in the context of the deep learning era.
