Data labeling is a critical and costly process, thus accessing large amounts of labeled data is not always feasible. Transfer Learning (TL) and Semi-Supervised Learning (SSL) are two promising approaches to leverage both labeled and unlabeled samples. In this work, we first study TL methods for classification based on unsupervised pre-training strategies with Autoencoder (AE) networks. Then, we focus on clustering in the Semi-Supervised scenario. Previous works introduced the β-VAE, an AE that learns a disentangled data representation from the unlabeled samples. We conduct an initial study of unsupervised pre-training with AEs to assess its impact on image classification tasks. We also design a new training method for the β-VAE based on cyclical annealing. The results show that annealing β during pre-training favours the learning of the target task. However, the best results on the target classification problem are obtained with a ResNet architecture with random initialization, trained only on labeled samples. Empirical evidence suggests that a deep network designed to learn complex patterns can achieve better results than a simpler pre-trained one. It is known that the quality of the data representation also affects the clustering algorithms. Deep Clustering leverages the strengths of Deep Learning to find the representation that better supports clustering. Hence, we introduce the β-VAE with cyclical annealing in the training process of several methods based on Deep Clustering. With respect to a Denoising Autoencoder (DAE), the β-VAE with annealing increases the Clustering Accuracy of the Deep Embedded Clustering (DEC) algorithm of 1% in the unsupervised scenario for the CIFAR-10 dataset. A new learning approach is also designed for clustering in the Semi-Supervised setting. We add an auxiliary supervised fine-tuning phase on the labeled samples. If 20% of the available examples are labeled, and the auxiliary task is executed, the Clustering Accuracy improves of 3.5% when the DAE is replaced by the β-VAE on the Fashion-MNIST dataset. Experiments also show improvements over previous works in the literature.
Il data labeling è un processo critico e costoso, quindi non sempre è possibile accedere a grandi quantità di dati labeled. Transfer Learning (TL) e Semi-Supervised Learning (SSL) sono metodologie promettenti per sfruttare congiuntamente dati labeled e unlabeled. In questo lavoro studiamo tecniche di TL basate sul pre-training non supervisionato tramite Autoencoders (AEs). Inoltre, ci concentriamo sulla loro applicazione al clustering in scenari semi-supervisionati. Il β-VAE è un AE recentemente proposto nella letteratura e in grado di trovare una rappresentazione disentangled dei dati a partire da campioni unlabeled. Inizialmente studiamo il pre-training non supervisionato con AEs per valutarne l’impatto su problemi di classificazione di immagini. Presentiamo anche un metodo di training per il β-VAE basato sull’annealing ciclico del parametro β: gli esperimenti mostrano che questa metodologia favorisce l’apprendimento del modello finale. I migliori risultati sul task di classificazione vengono tuttavia ottenuti con una rete ResNet inizializzata in modo randomico e senza alcun pre-training. L’esperienza empirica suggerisce che una rete profonda, disegnata per scoprire patterns complessi, può ottenere risultati migliori di una più semplice architettura cui è stata applicata una strategia di pre-training. La qualità della rappresentazione dei dati influenza anche le prestazioni delle varie tecniche di clustering. Il Deep Clustering sfrutta le potenzialità del Deep Learning per trovare la rappresentazione che meglio supporta il clustering. Introduciamo quindi il β-VAE con annealing ciclico nel processo di training di diversi metodi basati su Deep Clustering. Rispetto a un Denoising Autoencoder (DAE), il pre-training con il β-VAE consente di incrementare la Clustering Accuracy di Deep Embedded Clustering (DEC) dell’1% in uno scenario puramente non supervisionato per il dataset CIFAR-10. Un nuovo approccio di apprendimento viene proposto anche per il clustering in contesto semi-supervisionato. Introduciamo un task supervisionato ausiliario, ovvero una fase di fine-tuning coinvolgendo gli esempi labeled disponibili. Se il 20% degli esempi sono labeled, e il task ausiliario viene conseguito, la Clustering Accuracy ha un incremento del 3.5% se si usa il β-VAE al posto del DAE per il dataset Fashion-MNIST. I risultati sperimentali mostrano miglioramenti anche rispetto a precedenti approcci riportati nella letteratura.
On the effectiveness of β-VAEs for image classification and clustering. Using a disentangled representation for transfer learning and semi-supervised learning
Denti, Vittorio Maria Enrico
2019/2020
Abstract
Data labeling is a critical and costly process, thus accessing large amounts of labeled data is not always feasible. Transfer Learning (TL) and Semi-Supervised Learning (SSL) are two promising approaches to leverage both labeled and unlabeled samples. In this work, we first study TL methods for classification based on unsupervised pre-training strategies with Autoencoder (AE) networks. Then, we focus on clustering in the Semi-Supervised scenario. Previous works introduced the β-VAE, an AE that learns a disentangled data representation from the unlabeled samples. We conduct an initial study of unsupervised pre-training with AEs to assess its impact on image classification tasks. We also design a new training method for the β-VAE based on cyclical annealing. The results show that annealing β during pre-training favours the learning of the target task. However, the best results on the target classification problem are obtained with a ResNet architecture with random initialization, trained only on labeled samples. Empirical evidence suggests that a deep network designed to learn complex patterns can achieve better results than a simpler pre-trained one. It is known that the quality of the data representation also affects the clustering algorithms. Deep Clustering leverages the strengths of Deep Learning to find the representation that better supports clustering. Hence, we introduce the β-VAE with cyclical annealing in the training process of several methods based on Deep Clustering. With respect to a Denoising Autoencoder (DAE), the β-VAE with annealing increases the Clustering Accuracy of the Deep Embedded Clustering (DEC) algorithm of 1% in the unsupervised scenario for the CIFAR-10 dataset. A new learning approach is also designed for clustering in the Semi-Supervised setting. We add an auxiliary supervised fine-tuning phase on the labeled samples. If 20% of the available examples are labeled, and the auxiliary task is executed, the Clustering Accuracy improves of 3.5% when the DAE is replaced by the β-VAE on the Fashion-MNIST dataset. Experiments also show improvements over previous works in the literature.| File | Dimensione | Formato | |
|---|---|---|---|
|
MSc_Thesis_Vittorio_Denti.pdf
accessibile in internet solo dagli utenti autorizzati
Dimensione
9.47 MB
Formato
Adobe PDF
|
9.47 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/164603