Since it is difficult to annotate the gaze direction in the task of gaze estimation, it is not easy to prepare sufficient data. In situations like this where data are scarce, it is hoped that a method will be developed that will enable successful learning even with a small amount of supervised data.
Therefore, in this blog, we will provide an overview and recent research examples of semi-supervised learning, which is a method of learning using unsupervised data as well as supervised data, limited to the case of classification problems using deep learning. to introduce.
Contents
- Overview of semi-supervised learning
- Examples of recent research
- in conclusion
1. Overview of semi-supervised learning
There are two basic ideas for semi-supervised learning methods, called consistency regularization and entropy minimization , and many methods can be understood as either or a combination of the two.
consistency regularization
The first, consistency regularization, is a method based on the idea that even if the input is noisy, the output should be the same as when the input is free of noise. It is a method of changing the input by a method such as data augmentation, and regularizing it so as to minimize the difference between the output for each input before and after the change.
It is called consistency regularization because regularization is performed so that the output is consistent before and after noise is added to the input.
Since this regularization term does not require a label, it can be calculated even for unsupervised data. At the same time, the weights of the neural network are updated by error backpropagation. Since this consistency regularization was first proposed by Sajjadi et al.
entropy minimization
The second, entropy minimization, is a method based on the idea that “the decision boundary should not pass through areas where data are dense on the feature space.” It is a method of learning to become distant. Moving the output of a neural net away from the decision boundary is called entropy minimization because it is achieved by reducing the entropy of the output. As shown in Figure 1, lower entropy means more confidence in the model’s predictions.
Pseudo-Label, proposed by Lee et al. in 2013, is a typical entropy minimization method, and it has been used as a component technology in various methods since then. In the method called Pseudo-Label, among the outputs when unsupervised data is input to a model that has been pre-trained with supervised data, the output of the neural network is converted to 1-hot for those that predict with a certain degree of confidence. Create pseudo-labels and use these pseudo-labels to learn in the same way as normal supervised learning. As shown in Fig. 2, by learning so that the output approaches the 1-hot pseudo-label, the entropy of the output decreases, so it is classified as an entropy minimization method.
2. Recent research cases
As a recent research example, we will introduce a method called FixMatch proposed by Sohn et al. in 2020.
FixMatch is a method that combines consistency regularization and entropy minimization, and has been shown to achieve high accuracy even in cases with very little supervised data.
The overview of the method is shown in Fig. 1, and you will learn it in the following flow.
- Two types of data augmentation are performed on the input image: simple weak data augmentation such as inversion and strong data augmentation using reinforcement learning.
- create pseudolabels from output for input with weak data augmentation
- Perform consistency regularization between the output for the input with strong data augmentation and the pseudo label created in 2.
In the paper, it is possible to achieve an accuracy rate of close to 90% by learning using a total of 40 supervised data, 4 for each class, and a large amount of unsupervised data for a 10-class classification benchmark data set called CIFAR-10. is shown, and it can be seen that high accuracy can be achieved even with a small amount of supervised data.
3. Conclusion
At Skill Up AI, we are currently offering a related course called ” Basic Deep Learning Course that can be used in the field “.
The goal of this course is to understand the basics and principles of deep learning, and to master the cutting-edge technologies that support deep learning at the programming level. We practice active learning through group work and hands-on activities.
In addition, we hold a practical AI study session ” Skill Up AI Camp ” every Wednesday. At the study sessions, we will cover various practical themes and provide hints that will lead to improved practical skills in data analysis and AI development. There is also a corner where the instructor answers questions and concerns from the participants.
