Issuu

A STUDY COMPARING SUPERVISED VS. SELF-SUPERVISED LEARNING FOR LOW-DATA ENVIRONMENTS

Divya Vishwakarma1 , Mrs. Arifa Khan2

1Master of Technology, Computer Science and Engineering, Lucknow Institute of Technology, Lucknow, India

2Assistant Professor, Department of Computer Science and Engineering, Lucknow Institute of Technology, Lucknow, India

Abstract - The rapid advancement of deep learning has led to remarkable performance improvements across domains such as computer vision and natural language processing; however, these gains are heavily dependent on theavailabilityoflarge-scalelabeleddatasets.Inreal-world scenarios,obtainingannotateddataisoftenexpensive,timeconsuming, and sometimes infeasible, resulting in significant performance degradation for supervised learning models in low-data environments. This study presents a systematic comparative analysis of supervised and self-supervised learning paradigms under controlled data-scarce conditions. The research employs a rigorous experimental framework using benchmark datasets from both vision and language domains, with labeled data fractions restricted to 1%, 5%, and 10% to simulate lowresource settings. Supervised models are trained from scratch, while self-supervised models undergo representation pretraining on unlabeled data followed by fine-tuning. Performance is evaluated using standard metrics including accuracy, F1-score, precision, recall, and ROC-AUC, along with statistical significance testing to ensure reliability. Experimental results demonstrate that self-supervisedlearningconsistentlyoutperformssupervised learning in extreme low-data regimes by improving sample efficiency and generalization capability. However, the performance gap diminishes as labeled data availability increases. The study also highlights the trade-off between computational cost and annotation effort, providing practical insights for selecting appropriate learning strategiesinresource-constrainedenvironments.

Key Words: Supervised Learning, Self-Supervised Learning , Low-Data Environments , Representation Learning , Sample Efficiency , Deep Learning

1. INTRODUCTION

1.1 Background

1.1.1 Evolution of Machine Learning and Deep Learning Dominance

Machine Learning (ML) has evolved significantly from early rule-based and statistical models to modern deep learning architectures capable of learning complex hierarchical representations. Early approaches such as linearmodelsandSupportVectorMachinesreliedheavily

on handcrafted features and domain expertise. The breakthrough in deep learning, particularly with Convolutional Neural Networks (CNNs) and Transformerbased architectures, enabled end-to-end learning directly from raw data, eliminating the need for manual feature engineering. Landmark developments such as deep CNNs for image recognition and attention-based models for natural language processing have established deep learningasthedominantparadigminartificialintelligence (LeCun et al., 2015). This transition has led to unprecedented improvements in accuracy and scalability acrossmultipledomains.

1.1.2

Dependency on Large Labeled Datasets

Despite its success, deep learning is inherently dataintensive and relies heavily on large-scale annotated datasets. Models such as ResNet and Transformer-based architectures achieve state-of-the- art performance primarily due to training on millions of labeled samples This reliance is grounded in empirical risk minimization, where sufficient labeled data is required to approximate the underlying data distribution effectively. Large benchmark datasets like ImageNet and GLUE have played a crucialrole in advancing model performance; however, their scale is difficult to replicate in practical scenarios. Consequently, the requirement for extensive labeled data becomes a major bottleneck in real-world deployments (Dengetal.,2009).

1.1.3

Gap Between Research Benchmarks and RealWorld Constraints

A critical gap exists between controlled research environments and real-world applications. Benchmark datasets are often clean, balanced, and well-annotated, whereas real-world data is noisy,imbalanced, andlimited in quantity. Domains such as healthcare, remote sensing, and low-resource languages face significant challenges in data annotation due to cost and expertise requirements. This disparity results in a mismatch between laboratoryscale success and practical applicability, highlighting the need for learning paradigms that can perform effectively underconstraineddataconditions(Johnsonetal.,2019).

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

1.2 Problem Statement

1.2.1

Limitations of Supervised Learning in LowData Environments

Supervised learning models exhibit strong performance when trained on large datasets but suffer from severe performance degradation in low-data regimes. With limited labeled samples, models are prone to overfitting, leading to poor generalization on unseen data. This issue isparticularlypronouncedindeepneuralnetworksdueto their high parameter complexity, which requires substantial data to learn robust representations. As a result, supervised learning becomes less reliable in scenarioswheredataavailabilityisrestricted(Goodfellow etal.,2016).

1.2.2 High Annotation Cost and Time Constraints

The creation of labeled datasets involves significant financial, temporal, and human resource investments. Annotation tasks often require domain-specific expertise, such as medical professionals for healthcare data or linguists for language processing tasks. Even with crowdsourcing, large-scale labeling efforts are timeconsuming and costly. These constraints limit the scalability of supervised learning approaches, particularly in specialized domains where expert knowledge is essential(Snowetal.,2008).

1.2.3 Need for Alternative Learning Paradigms

Given the limitations of supervised learning, there is a growing need for alternative approaches that reduce dependencyonlabeleddata.Self-supervisedlearning(SSL) has emerged as a promising paradigm that leverages unlabeled data to learn meaningful representations. By generating supervisory signals from the data itself, SSL reduces the reliance on manual annotation while maintaining strong performance. This shift toward dataefficient learning strategies is essential for addressing real-worldchallengesinlow-dataenvironments(Devlinet al.,2019).

1.3 Motivation

1.3.1

Real-World Low-Data Domains

Severalreal-worlddomainsinherentlyoperateunderdata scarcity. In medical imaging, obtaining labeled data requires expert diagnosis, making large-scale annotation expensive and slow. Similarly, in natural language processing, many languages lack sufficient annotated corpora, limiting the development of robust language models. Cybersecurity applications also face challenges duetotherarityandevolvingnatureofthreatdata.These scenarios highlight the necessity for models that can

performeffectivelywithlimitedlabeleddata(Estevaetal., 2017).

1.3.2 Availability of Unlabeled Data vs. Scarcity of Labels

While labeled data is scarce, unlabeled data is abundant acrossmostdomains.Massivevolumesofrawtext,images, and sensor data are continuously generated, providing an opportunity for representation learning without manual annotation. Self-supervised learning capitalizes on this abundance byextracting meaningful patterns from unlabeled data. This contrast between label scarcity and data abundance motivates the exploration of SSL as a viable solution for improving learning efficiency (Chen et al.,2020).

1.4 Research Objectives

 To analyze generalization capability under varying datasizes

 Tomeasuresampleefficiency

 Toevaluatetransferlearningbenefits

 Toexaminerobustnesstooverfitting

 Toassesscomputationalcostandscalability

1.5 Contributions of the Paper

1.5.1

Controlled Experimental Framework

This study introduces a controlled experimental framework that ensures fair comparison between supervised and self-supervised learning. By maintaining consistent architectures, datasets, and evaluation metrics, the framework eliminates confounding variables and enhancesthevalidityofresults.

1.5.2

Cross-Domain Evaluation

A key contribution is the cross-domain evaluation spanning both computer vision and natural language processing tasks. This approach ensures that findings are not domain-specific and enhances the generalizability of conclusionsacrossdifferentdatamodalities.

1.5.3

Statistical Validation of Results

The study incorporates rigorous statistical validation techniques, including repeated experiments and significance testing, to ensure the reliability of findings. This strengthens the scientific rigor of the comparative analysis.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

1.5.4 Practical Recommendations

Finally, the paper provides practical insights for researchersand practitioners by outlining when to prefer supervised or self-supervised learning. These recommendations are particularly valuable for applications operating under resource constraints, where efficientutilizationofdataandcomputationiscritical.

2. RELATED WORK

2.1 Supervised Learning Advances

2.1.1

CNNs (ResNet) and Transformer Architectures

Supervised learning has witnessed remarkable progress with the introduction of deep neural network architectures,particularlyConvolutional Neural Networks (CNNs) and Transformers. CNN-based models such as ResNet introduced residual connections that enabled the training of very deep networks by mitigating vanishing gradientproblems.Thisinnovationsignificantlyimproved performanceinimageclassificationandothervisiontasks. In parallel, Transformer architectures revolutionized natural language processing by replacing recurrent structures with self-attention mechanisms, enabling efficient modeling of long-range dependencies. These advancements have established supervised deep learning asthedominantapproachacrossmultipledomains(He et al.,2016).

2.1.2

Success with Large-Scale Datasets

The effectiveness of supervised learning is largely attributedtotheavailabilityoflarge-scalelabeleddatasets. Models trained on datasets such as ImageNet and large textcorporahavedemonstratedexceptionalperformance, oftensurpassinghuman-levelbenchmarksinspecifictasks. Thescalabilityofdeeparchitectures,combinedwithhighcapacity datasets, has enabled models to learn highly discriminativeandtransferablefeatures.Empiricalstudies haveconsistentlyshownthatincreasingdatasetsizeleads to improved model accuracy and generalization, reinforcing the data-driven nature of supervised learning (Krizhevskyetal.,2012).

2.1.3

Limitations in Low-Data Regimes

Despite its success, supervised learning exhibits significant limitations when applied to low-data environments. Deep models tend to overfit when trained onsmall datasetsduetotheirhighparameter complexity. Furthermore, insufficient data leads to poor representation learning, reducing model robustness and generalization. These challenges are particularly evident in specialized domains where collecting large labeled datasets is impractical, highlighting a critical weakness of purelysupervisedapproaches(Zhangetal.,2017).

2.2 Self-Supervised Learning

2.2.1

Contrastive Learning Approaches (SimCLR, MoCo)

Self-supervised learning has emerged as a powerful alternativetosupervisedlearningbyleveragingunlabeled data for representation learning. Contrastive learning methods, such as SimCLR and MoCo, learn feature representations by maximizing agreement between augmented views of the same data instance while distinguishing them from other instances. These approaches have demonstrated strong performance in visual representation learning, often rivaling supervised pretraining when fine-tuned on downstream tasks. The ability to learn meaningful embeddings without manual labels makes contrastive learning particularly effective in data-scarcescenarios(Heetal.,2020).

2.2.2 Masked Modeling Approaches (BERT, MIM)

Anothermajorclassofself-supervisedmethodsismasked modeling, where parts of the input are intentionally hidden and predicted by the model. BERT introduced maskedlanguagemodeling,enablingbidirectionalcontext learning in NLP tasks. Similarly, masked image modeling techniques extend this idea to vision tasks by reconstructing missing image patches. These approaches encourage models to capture contextual and semantic relationships within data, leading to highly transferable representationsacrosstasks(Devlinetal.,2019).

2.2.3

Representation Learning Advantages

Self-supervised learning excels in learning generalpurpose representations that can be fine-tuned with minimallabeleddata.Byexploitingintrinsicdatastructure, SSL reduces dependency on annotated datasets while improving sample efficiency. Empirical evidence suggests thatSSL-pretrainedmodelsachievesuperiorperformance compared to supervised models in low-data regimes, particularlywhenlabeleddataislimitedtosmallfractions oftheoriginaldataset(Chenetal.,2020).

2.3 Learning Under Data Scarcity

2.3.1

Transfer Learning

Transfer learning addresses data scarcity by leveraging knowledge from models pretrained on large datasets. In thisapproach,pretrainedmodelsserveasinitializationfor downstream tasks, significantly reducing the amount of labeled data required. This paradigm has been widely adoptedinbothvisionandNLP,wheremodelspretrained on large corpora are fine-tuned for specific applications. Transferlearningimprovesgeneralizationandaccelerates convergence, making it a practical solution for low- data environments(PanandYang,2010).

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

2.3.2 Few-Shot Learning

Few-shot learning focuses on enabling models to generalize froma verysmall number oflabeled examples. Techniques such as meta-learning train models to adapt quickly to new tasks by learning transferable knowledge across multiple tasks. Approaches like Model-Agnostic Meta-Learning (MAML) and Prototypical Networks have demonstrated promising results in scenarios with extremely limited data. These methods emphasize adaptability and generalization rather than task-specific optimization(Finnetal.,2017).

2.3.3 Data Augmentation Techniques

Data augmentation is another widely used strategy to mitigate data scarcity by artificially increasing dataset diversity. Techniques such as Mixup, CutMix, and random transformations generate new training samples by modifying existing data. These methods help prevent overfittingandimprovemodelrobustnessbyexposingthe model to a wider range of variations. Augmentation plays a crucial role in enhancing performance in low-data regimes without requiring additional labeled data (Zhang etal.,2018).

2.4 Existing Comparative Studies

2.4.1

Empirical Comparisons Using Benchmark Datasets

Several studies have conducted empirical comparisons between supervised and self-supervised learning using benchmark datasets such as ImageNet. These studies generally show that self-supervised pretraining followed by fine-tuning outperforms supervised training from scratch when labeled data is limited. Performance gains are particularly evident in scenarios where only a small fractionoflabeleddataisavailable.Suchfindingshighlight theeffectivenessofSSLinimprovingsampleefficiencyand representationquality(Goyaletal.,2019).

2.4.2 Lack of Standardized Experimental Protocols

Despite promising results, existing comparative studies oftenlackstandardizedexperimentalprotocols.Variations in architectures, training procedures, and dataset preprocessing make it difficult to draw consistent conclusions across studies. In many cases, differences in computational resources and hyperparameter tuning further complicate comparisons. This lack of uniformity limits the reproducibility and generalizability of findings, indicating the need for controlled and systematic evaluationframeworks(Oliveretal.,2018).

2.5 Research Gap

Current literature has not sufficiently explored extreme low-data scenarios, particularly when labeled data is

restricted to 1–5% of the original dataset. Most studies focus on moderate reductions, leaving a gap in understanding model behavior under severe data constraints. This limitation restricts the applicability of existingfindingstoreal-worldlow-resourcesettings.

Another significant gap is the lack of cross-domain analysis. Many studies focus exclusively on either computer vision or natural language processing, without evaluating whether findings generalize across different data modalities. This limits the broader applicability of conclusionsregardinglearningparadigms.

Finally, many comparative studies do not incorporate rigorous statistical validation. Results are often reported based on single runs without confidence intervals or hypothesis testing, reducing their reliability. The absence of statistical rigor makes it difficult to determine whether observedperformancedifferencesaresignificantordueto random variation. Addressing this gap is essential for establishingscientificallysoundconclusions.

3. METHODOLOGY

3.1 Experimental Design

3.1.1 Controlled Comparison Framework

Themethodologyadoptsacontrolledexperimentaldesign to ensure a fair and unbiased comparison between supervised and self-supervised learning paradigms. Both approaches are evaluated under identical conditions, including the same datasets, model architectures, optimizationstrategies,andcomputationalresources.The only varying factor is the learning paradigm itself. This controlled setup eliminates confounding variables and ensuresthatany observed differencesin performance are attributable solely to the training strategy rather than implementationinconsistencies.

3.1.2 Independent Variables

Thestudydefinestwoprimaryindependentvariables:the learning paradigm and the labeled data fraction. The learning paradigm is categorized into supervised learning (training from scratch) and self- supervised learning (pretraining followed by fine-tuning). The labeled data fraction is systematically varied to simulate different levelsofdatascarcity,specifically1%,5%,10%,and100% oftheoriginaldataset.Thesevariablesenableastructured analysis of model performance across progressively constraineddataenvironments.

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

Table 1: Independent Variables in Experimental Design

Variable Type Description Levels/Values

Learning Paradigm Training strategyused Supervised, Self-Supervised

DataFraction Percentage of labeled data usedfor training 1%, 5%, 10%, 100%

3.2

Dataset Selection

3.2.1

Vision and NLP Datasets

To ensure cross-domain validity, the study employs benchmark datasets from both computer vision and natural language processing domains. CIFAR-10 and ImageNet are selected for vision tasks due to their standardized structure and widespread use in representation learning research. For NLP tasks, datasets such as AG News and GLUE are utilized, offering diverse classificationtasksandlinguisticcomplexity.Theinclusion of multiple domains enhances the generalizability of findingsandensuresthatresultsarenotdomain-specific.

Table 2: Dataset Overview

3.3.2 Multiple Random Seeds

To improve statistical robustness, multiple random seeds areusedduringdatasetsamplingandmodeltraining.This approach reduces the impact of randomness in data selection and weight initialization.Final results are reported as averages across multiple runs, ensuring that conclusions are reliable and not influenced by a single favorableorunfavorableexperiment.

3.4 Model Architectures

3.4.1

Vision Model: ResNet-50

For computer vision tasks, the ResNet-50 architecture is employed due to its proven effectiveness in deep feature extraction. With residual connections enabling stable gradient flow, ResNet-50 serves as a strong baseline for both supervised and self-supervised experiments. Its moderate complexity makes it suitable for controlled academicexperimentation.

3.4.2 NLP Model: BERT Base

In the NLP domain, BERT Base is utilized as the primary architecture. This Transformer-based model captures contextual relationships using self-attention mechanisms. Itiswidelyusedfortextclassificationtasksandprovidesa consistent backbone for evaluating both supervised and self-supervisedlearningpipelines.

3.5 Training Pipelines

3.5.1

Supervised Learning

In the supervised pipeline, models are initialized with random weights and trained directly on labeled data subsets. The optimization processusescross-entropyloss for classification tasks, enabling the model to learn mappings between inputs and target labels. Training is conducted independently for each data fraction, ensuring thatperformancedifferencesreflectdataavailability.

3.5.2 Self-Supervised Learning

3.3 Low-Data Simulation

3.3.1 Stratified Sampling and Balanced Subsets

To simulate low-data environments, stratified sampling is appliedtocreatereducedlabeledsubsetswhilepreserving theoriginalclassdistribution.Thisensuresthateachclass isproportionallyrepresented,preventingbiasduetoclass imbalance. Balanced subsets are critical for maintaining the integrity of classification performance, especially in extremelow-datascenariossuchas1%and5%.

The self-supervised pipeline consists of two stages. First, models undergo pretraining on unlabeled data using representation learning objectives such as contrastive learning or masked modeling. This stage enables the model to learn general features without manual labels. In the second stage, the pretrained model is fine-tuned on labeled subsets using the same classification objective as in supervised learning. This two-step process enhances feature quality and improves performance in low-data regimes.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

Table 3: Training Pipeline Comparison

Stage Supervised Learning SelfSupervised Learning

Initialization Random Pretrained weights

Pretraining Notapplicable

Unlabeleddata (SSL objective)

Fine-tuning Directon labeled data After pretraining

LossFunction Cross-entropy

3.6 Evaluation Metrics

3.6.1 Performance Metrics

SSLloss+ Cross-entropy

Model performance is evaluated using a comprehensive set of classification metrics. Accuracy measures overall correctness, while precision and recall provide insights into class-specific performance. The F1-score balances precision and recall, making it particularly useful in imbalanced datasets. ROC-AUC is used to evaluate model discrimination ability independent of classification thresholds.

3.6.2 Representation Quality (Linear Probing)

To assess the quality of learned representations, linear probing is employed. In this approach, the pretrained model is frozen, and only a linear classifier is trained on top of its features. This method isolates representation qualityfromtask-specificfine-tuningandprovidesadirect measureoffeatureseparability.

Table 4: Evaluation Metrics

Metric Description

Purpose

Accuracy Overall prediction correctness General performance

Precision Correct positive predictions ratio Falsepositive control

Recall Coverageof actualpositives Sensitivity measurement

F1-score Harmonic Balanced

meanof precision and recall evaluation

ROC-AUC Areaunder ROCcurve Thresholdindependent measure

LinearProbe Accuracyusing frozen features Representation quality

4. EXPERIMENTAL SETUP

4.1 Environment Configuration

4.1.1

Deep Learning Frameworks

The experimental implementation is carried out using widely adopted deep learning frameworks, namely PyTorch and TensorFlow. These frameworks provide flexible computational graphs, efficient GPU acceleration, and extensive libraries for model development, training, and evaluation. PyTorch is primarily utilized for its dynamic computation graph and ease of experimentation, while TensorFlow is leveraged for its scalability and deployment capabilities. The use of these frameworks ensures compatibility with state-of-the-art architectures andfacilitatesreproducibleexperimentation.

4.1.2 Hardware Configuration (GPU Setup)

All experiments are conducted on high-performance GPU hardware to ensure efficient training of deep neural networks. GPUs such as NVIDIA V100 and RTX 3090 are employedduetotheirhighcomputationalthroughputand memory capacity, which are essential for training largescale models like ResNet-50 and BERT Base. GPU acceleration significantly reduces training time and enables the execution of multiple experimental runs requiredforstatisticalvalidation.

4.2 Data Preprocessing

4.2.1

Vision Data Preprocessing

For computer vision tasks, preprocessing involves normalization and data augmentation techniques to improve model generalization. Input images are normalized using dataset-specific mean and standard deviation values to stabilize training. Data augmentation techniques such as random cropping, horizontal flipping, and color jittering are applied to artificially increase dataset diversity. These transformations help the model become invariant to minor variations and reduce over fitting,particularlyinlow-dataregimes.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

4.2.2 NLP Data Preprocessing

In natural language processing tasks, preprocessing includestokenizationandsequencepadding.Tokenization converts raw text into meaningful subword units using pretrained vocabularies, enabling efficient representation learning. Padding ensures that all input sequences have uniformlength,whichisnecessaryforbatchprocessingin Transformer-based models. Additionally, attention masks are generated to differentiate between actual tokens and padded positions, ensuring accurate computation during training.

Table 5: Data Preprocessing Techniques

Domain Technique Purpose

Vision Normalization Stabilizeinput distribution

Vision Augmentation Improve generalization

NLP Tokenization Convert text to model-readable format

NLP Padding Ensureuniform sequence length

4.3 Training Details

4.3.1

Optimization Strategies

Different optimization algorithms are selected based on the nature of the task and model architecture. For vision tasks,StochasticGradient Descent(SGD) withmomentum is used due to its effectiveness in training CNNs and its abilitytogeneralizewell.ForNLPtasks,AdamWoptimizer is employed, as it combines adaptive learning rates with weight decay regularization, making it suitable for Transformer- based architectures like BERT. These optimizers ensure stable convergence and efficient parameterupdates.

4.3.2 Learning Rate Scheduling and Early Stopping

Learning rate scheduling is implemented to dynamically adjust the learning rate during training, improving convergence and preventing overshooting of optimal solutions. Techniques such as step decay or cosine annealing are used depending on the experiment. Early stopping is applied to terminate training when validation performance ceases to improve, thereby preventing overfitting and reducing unnecessary computational cost.

These strategies enhance both efficiency and model performance.

Table 6: Training Configuration

Parameter

Optimizer SGD AdamW

LearningRate 0.01(typical) 2e-5 (typical)

Batch Size 128 16–32

Scheduler Step/Cosine Decay LinearWarmup

Regularization Weightdecay Weightdecay

Early Stopping Enabled Enabled

4.4 Reproducibility

4.4.1

Fixed Random Seeds

Toensureconsistencyandreproducibilityofresults, fixed random seeds are used across all experiments. Randomness in weight initialization, data shuffling, and samplingproceduresiscontrolledbysettingdeterministic seeds. This approach minimizes variability across runs and allows for fair comparison between supervised and self-supervisedlearningmethods.

4.4.2

Controlled Hardware and Software Setup

In addition to controlling randomness, experiments are conducted under a consistent hardware and software environment. The same GPU configurations, framework versions, and dependency libraries are maintained throughout all experimental runs. This controlled setup eliminates external variability and ensures that performance differences arise solely from methodological changesratherthanenvironmentalfactors.

5. RESULTS5.1 Performance Comparison

5.1.1

Comparative Analysis of Supervised

vs. SelfSupervised Learning

This section presents a comprehensive comparison between supervised learning (SL) and self- supervised learning (SSL) across varying data fractions. The evaluation is conducted using standard classification metrics, with results averaged over multiple runs to ensure robustness. Empirical findings indicate that SSL consistentlyoutperformsSLinlow-dataregimes(1%,5%, 10%), while both approaches achieve comparable

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

performancewhentrainedonthefulldataset(100%).The performance gap is most pronounced at extremely low data availability, demonstrating the superior sample efficiencyofSSL.

Graph 1: Performance Comparison Across Data Fractions.

5.2 Sample Efficiency Analysis

5.2.1 Performance vs. Data Fraction Trends

Sample efficiency is evaluated by analyzing how model performance scales with increasing labeled data. The results reveal that SSL achieves higher accuracy with significantly fewer labeled samples compared to SL. The performance curve for SSL demonstrates a steeper improvement at lower data fractions, indicating its ability toleveragelearnedrepresentationseffectively.Incontrast, SLshowsgradualimprovement,heavilydependentonthe availability of labeled data. As the dataset size increases, the performance curves of both methods begin to converge.

Table 7: Sample Efficiency Analysis

5.3 Representation Quality

5.3.1 Linear Probing Evaluation

To assess representation quality independently of taskspecific fine-tuning, linear probing is conducted on pretrained models. In this setup, feature extractors are frozen, and only a linear classifier is trained. Results indicate that SSL-trained models produce more discriminative and generalizable features, particularly in low-data settings. This demonstrates that SSL captures underlying data structures more effectively, leading to improveddownstreamperformance.

Table 8: Linear Probing Results

6. DISCUSSION

6.1 Key Findings

8.1.1 Performance Trends Across Data Regimes

The experimental results clearly demonstrate that selfsupervised learning outperforms supervised learning in low-data environments. The advantage of SSL is most prominent when labeled data is extremely limited (1%–5%), where it significantly boosts model performance. However, as the amount of labeled data increases, the performancegapgraduallynarrows,andbothapproaches achievesimilarresultsunderfull-dataconditions.

6.2 Interpretation

6.2.1

Improved Feature Representation in SSL

The superior performance of SSL can be attributed to its ability to learn rich and generalized feature representations during the pretraining phase. By leveraging unlabeled data, SSL captures intrinsic patterns and semantic relationships, which are not accessible to supervisedmodelstrainedfromscratchonlimiteddata.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

6.2.2 Reduced Overfitting in Low-Data Settings

Another key observation is the reduced tendency of SSL models to overfit. Since SSL models start with pretrained representations, they require fewer labeled samples to generalize effectively. In contrast, supervised models trained on small datasets often memorize training data, leadingtopoorperformanceonunseensamples.

6.3 Computational Trade-Offs

6.3.1

SSL vs. Supervised Learning Costs

While SSL reduces dependency on labeled data, it introduces higher computational costs due to the pretrainingphase.SSLrequireslarge-scaleunlabeleddata processing and longer training times. On the other hand, supervised learning is computationally less expensive but demands extensive labeled datasets, which are costly to obtain.

Table 9: Computational Trade-Off Comparison

Aspect Supervised Learning SelfSupervised Learning

Annotation Cost High Low

Computational Cost Low High

TrainingTime Moderate High(dueto pretraining)

Data Requirement Labeleddata Mostly unlabeleddata

6.4 Practical Implications

6.4.1 Choosing Between SSL and Supervised Learning

The findings suggest that self-supervised learning should bepreferredinscenarioswherelabeleddataisscarcebut unlabeled data is abundant. It is particularly suitable for domainssuchashealthcare, low resourcelanguages,and cybersecurity. Conversely, supervised learning remains effectivewhen largelabeled datasetsare readilyavailable andcomputationalresourcesarelimited.

6.4.2 Industry Applications

In practical applications, SSL can significantly reduce annotation costs while maintaining high performance. Industriesdealingwithlargevolumesofrawdata,suchas social media analytics, medical imaging, and autonomous systems, can benefit from SSL-based approaches. Meanwhile, traditional supervised learning continues to be relevant in well-established domains with extensive labeleddatasets.

7. CONCLUSION

This study presented a comprehensive comparative analysis of supervised learning (SL) and self- supervised learning (SSL) in low-data environments, with the objective of evaluating their effectiveness under constrained labeled data conditions. The experimental results demonstrate that SSL consistently outperforms SL when labeled data is limited, particularly in extreme lowdata regimes such as 1% and 5% of the dataset. This performance advantage is primarily attributed to the ability of SSL to learn rich and transferable feature representations from large volumes of unlabeled data during the pretraining phase. In contrast, SL models trained from scratch struggle to generalize due to insufficient training samples and are more prone to overfitting.

As the amount of labeled data increases, the performance gap between the two paradigms gradually diminishes, withbothapproachesachievingcomparableresultsunder full-data conditions. This highlights that while SL remains effective in data-rich scenarios, SSL provides a more robust and scalable solution in resource-constrained settings. Furthermore, the study emphasizes the trade-off between computational cost and annotation effort, where SSLreducesdependencyonlabeleddataattheexpense of increasedcomputationalrequirements.

Overall, this research underscores the importance of adopting data-efficient learning strategies and provides practical insights for selecting appropriate methodologies based on data availability and resource constraints, thereby contributing to the advancement of machine learninginreal-worldapplications.

8. LIMITATIONS OF THE STUDY

Despiteitscontributions,thisstudyhasseverallimitations. First, the experiments are conducted on benchmark datasets, which may not fully capture the complexity and noise present in real-world data. Second, the analysis is limited to specific model architectures, namely ResNet-50 and BERT Base, which may restrict the generalizability of thefindingstootherarchitecturesorlargermodels.Third, the study focuses on moderate-scale self-supervised techniques and does not explore large-scalepretraining

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

approaches that could further enhance performance. Additionally, computational constraints limit the explorationofmoreextensivehyperparametertuningand longer training schedules. Finally, while statistical validation is performed, the number of experimental repetitions could be increased for greater robustness. These limitations provide opportunities for future researchandimprovement.

REFERENCES

1. Baevski, A., Zhou, H., Mohamed, A. and Auli, M. (2020) ‘wav2vec2.0:Aframeworkforself-supervisedlearningof speech representations’, Advances in Neural Information ProcessingSystems(NeurIPS),33,pp.12449–12460.

2. Bao, H., Dong, L. and Wei, F. (2022) ‘BEiT: BERT pretraining of image transformers’, International Conference onLearningRepresentations(ICLR).

3.Brown,T.B.etal.(2020)‘Languagemodelsarefew-shot learners’, Advances in Neural Information Processing Systems(NeurIPS),33,pp.1877–1901.

4. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P. and Joulin, A. (2020) ‘Unsupervised learning of visual features by contrasting cluster assignments’, Advances in Neural Information Processing Systems(NeurIPS), 33, pp. 9912–9924.

5. Chen, X. and He, K. (2021) ‘Exploring simple Siamese representation learning’, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),pp.15750–15758.

6. Chen, X., Fan, H., Girshick, R. and He, K. (2020) ‘Improvedbaselineswithmomentumcontrastivelearning’, arXivpreprintarXiv:2003.04297.

7. Dosovitskiy, A. et al. (2021) ‘An image is worth 16x16 words: Transformers for image recognition at scale’, International Conference on Learning Representations (ICLR).

8.Grill,J.B.etal.(2020)‘Bootstrapyourownlatent:Anew approach to self-supervised learning’, Advances in Neural InformationProcessingSystems(NeurIPS),33,pp.21271–21284.

9. He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R. (2022)‘Maskedautoencodersarescalablevisionlearners’, Proceedings of the IEEE/CVF Conference on Computer VisionandPatternRecognition(CVPR),pp.16000–16009.

10.Hinton,G.,Vinyals,O.andDean,J.(2015)‘Distillingthe knowledge in a neural network’, arXiv preprint arXiv:1503.02531.

11. Jing, L. and Tian, Y. (2020) ‘Self-supervised visual feature learning with deep neural networks: A survey’, IEEE Transactions on Pattern Analysis and Machine Intelligence,43(11),pp.4037–4058

12.Kolesnikov,A.,Zhai,X.andBeyer,L.(2019)‘Revisiting self-supervised visual representation learning’, Proceedings of the IEEE/CVF Conference on Computer VisionandPatternRecognition(CVPR),pp.1920–1929.

13. Liu, Y. et al. (2019) ‘RoBERTa: A robustly optimized BERT pretraining approach’, arXiv preprint arXiv:1907.11692.

14. Noroozi, M. and Favaro, P. (2016) ‘Unsupervised learning of visual representations by solving jigsaw puzzles’,EuropeanConferenceonComputerVision(ECCV), pp.69–84.

15. Oord, A.V.D., Li, Y. and Vinyals, O. (2018) ‘Representation learning with contrastive predictive coding’,arXivpreprintarXiv:1807.03748.

16. Radford, A. et al. (2021) ‘Learning transferable visual modelsfromnaturallanguagesupervision’,Proceedingsof theInternationalConferenceonMachineLearning(ICML), pp.8748–8763.

17. Ramesh, A. et al. (2021) ‘Zero-shot text-to-image generation’, Proceedings of the International Conference onMachineLearning(ICML),pp.8821–8831.

18.Raschka,S.(2020) ‘Model evaluation,model selection, and algorithm selection in machine learning’, arXiv preprintarXiv:1811.12808.

19. Sohn, K. et al. (2020) ‘FixMatch: Simplifying semisupervised learning with consistency and confidence’, Advances in Neural Information Processing Systems (NeurIPS),33,pp.596–608.

20.Touvron,H.etal.(2021)‘Trainingdata-efficientimage transformers & distillation through attention’, InternationalConferenceonMachineLearning(ICML),pp. 10347–10357.

21. Van Engelen, J.E. and Hoos, H.H. (2020) ‘A survey on semi-supervised learning’, Machine Learning, 109, pp. 373–440.

22. Xie, Q. et al. (2020) ‘Self-training with noisy student improves ImageNet classification’, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),pp.10687–10698.

23. Zhai, X. et al. (2022) ‘Scaling vision transformers’, Proceedings of the IEEE/CVF Conference on Computer VisionandPatternRecognition(CVPR),pp.12104–12113.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

24.Zoph,B.etal.(2020)‘Rethinkingpre-trainingandselftraining’,AdvancesinNeuralInformation ProcessingSystems(NeurIPS),33,pp.3833–3845.

25. Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A.S., Shekhar, S., Goldblum, M., Goldstein, T. and Baraniuk, R. (2023)‘Acookbookofself-supervisedlearning’,Journalof MachineLearningResearch,24(1),pp.1–56.

26.Chen,T.,Kornblith,S.,Norouzi,M.andHinton,G.(2020) ‘A simple framework for contrastive learning of visual representations’, Proceedings of the International ConferenceonMachineLearning(ICML),pp.1597–1607.

27.Deng,J.,Dong,W.,Socher,R.,Li,L.J.,Li,K.andFei-Fei,L. (2009) ‘ImageNet: A large-scale hierarchical image database’, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255.

28.Devlin,J.,Chang,M.W.,Lee,K.andToutanova,K.(2019) ‘BERT:Pre-training ofdeepbidirectional transformersfor language understanding’, Proceedings of the North American Chapter of the Association for Computational Linguistics(NAACL),pp.4171–4186.

29. Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M. and Thrun, S. (2017) ‘Dermatologist-level classification of skin cancer with deep neural networks’, Nature,542(7639),pp.115–118.

30. Finn, C., Abbeel, P. and Levine, S. (2017) ‘Modelagnostic meta-learning for fast adaptation of deep networks’,ProceedingsoftheInternationalConferenceon MachineLearning(ICML),pp.1126–1135.

31.Goodfellow,I.,Bengio,Y.andCourville,A.(2016)Deep Learning.Cambridge,MA:MITPress.

32. Goyal, P., Mahajan, D., Gupta, A. and Misra, I. (2019) ‘Scaling and benchmarking self- supervised visual representation learning’, Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 6391–6400.

33. He, K., Zhang, X., Ren, S. and Sun, J. (2016) ‘Deep residual learning for image recognition’, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR),pp.770–778.

34. He, K., Fan, H., Wu, Y., Xie, S. and Girshick, R. (2020) ‘Momentum contrast for unsupervised visual representation learning’, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),pp.9729–9738.

35. Johnson, J., Gupta, A. and Fei-Fei, L. (2019) ‘Survey on deep learning with limited data’, IEEE Transactions on PatternAnalysisandMachineIntelligence,pp.1–20.

36. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) ‘ImageNet classification with deep convolutional neural networks’, Advances in Neural Information Processing Systems(NeurIPS),25,pp.1097–1105.

37. LeCun, Y., Bengio, Y. and Hinton, G. (2015) ‘Deep learning’,Nature,521(7553),pp.436–444.

38. Oliver, A., Odena, A., Raffel, C.A., Cubuk, E.D. and Goodfellow, I. (2018) ‘Realistic evaluation of deep semisupervised learning algorithms’, Advances in Neural Information Processing Systems (NeurIPS), 31, pp. 3235–3246.

39. Pan, S.J. and Yang, Q. (2010) ‘A survey on transfer learning’, IEEE Transactions on Knowledge and Data Engineering,22(10),pp.1345–1359.

40.Snow,R.,O’Connor,B.,Jurafsky,D.andNg,A.Y.(2008) ‘Cheap and fast but is it good? Evaluating non-expert annotationsfornaturallanguagetasks’,Proceedingsofthe Conference on Empirical Methods in Natural Language Processing(EMNLP),pp.254–263.