QBoard » Artificial Intelligence & ML » AI and ML - PyTorch » What is the difference between transfer learning and fine-tuning?
User Dashboard

What is the difference between transfer learning and fine-tuning?

Back To Topics
Tags : ml data datasciecne leraning
  • 131 2
    What is the difference between transfer learning and fine-tuning?
      September 2, 2021 4:54 PM IST
    0
  • 259
    Transfer Learning:
    Transferring or using the learnt parameters (weights, bias) of a pre-trained network to a new task. Usually in the new task, we keep the network's layers and the learned parameters of the pre-trained network unchanged and we modify the last few layers (e.g. Fully connected layer, Classification layer) which depends upon the application.
     
    Fine tuning
    Fine tuning is like optimization. We optimize the network to achieve the optimal results. May be we can change the number of layers used, no of filters, learning rate and we have many parameters of the model to optimize.
     
    Learning from scratch
    Learning from scratch means building a network like AlexNet, GoogleNet etc. of your own. Every pre-trained network like AlexNet has some fixed or standard number of layers. If you build a network from scratch, you should decide the no of layers, no of filters etc. You should build a model that produces the maximum result for your application. Like trail and error method you should keep on working with different layers, increasing or decreasing the no of layers, filters, deciding about which normalization technique to use and deciding which activation function to use etc.
     
    You could read my survey paper, where i have discussed about different deep learning architectures and techniques that have been used for Medical domain applications.
      September 3, 2021 1:23 PM IST
    0
  • 346 3
    Transfer learning is process of using already developed and trained model on your dataset to get insights.

    while Fine Tuning is like Tuning parameters of already developed/trained model. hope this may help!
      September 7, 2021 1:34 PM IST
    0
  • 317
    Transfer Learning and Fine-tuning are used interchangeably and are defined as the process of training a neural network on new data but initialising it with pre-trained weights obtained from training it on a different, mostly much larger dataset, for a new task which is somewhat related to the data and task the network was previously trained on. In transfer learning, usually the last few layers of the network is replaced by new layers and initialised with random weights, the unchanged layers can either be frozen ie made un-trainable or be kept trainable.
    Learning from scratch simply means initialising the weights of a NN to random values and starting the training on the main dataset and task.
      January 13, 2022 1:21 PM IST
    0
  • 319 1

    Transfer learning is about “transferring” the learnt representations to another problem. For example one can use features from a pre-trained convolutional neural network (convNet) to power a linear support vector machine (SVM). In such a case the pre-trained model can be held fixed while the linear SVM weights can be updated.

    Fine tuning on the other hand is just about making some fine adjustments to further improve performance. For example, during transfer learning, you can unfreeze the pre-trained model and let it adapt more to the task at hand.

    Thus

    • Transfer learning
      • Is about projecting all new inputs through a pre-trained model. Like if we have a pre-trained model function http://www.w3.org/1998/Math/MathML"><mi>f</mi><mo stretchy="false">(</mo><mo stretchy="false">)</mo></math>">f()f() and wish to learn a new function http://www.w3.org/1998/Math/MathML"><mi>g</mi><mo stretchy="false">(</mo><mo stretchy="false">)</mo></math>">g()g(), we can simplify http://www.w3.org/1998/Math/MathML"><mi>g</mi><mo stretchy="false">(</mo><mo stretchy="false">)</mo></math>">g()g() by http://www.w3.org/1998/Math/MathML"><mi>g</mi><mo stretchy="false">(</mo><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo></math>">g(f(x))g(f(x)). This way http://www.w3.org/1998/Math/MathML"><mi>g</mi><mo stretchy="false">(</mo><mo stretchy="false">)</mo></math>">g()g() sees all the data through http://www.w3.org/1998/Math/MathML"><mi>f</mi><mo stretchy="false">(</mo><mo stretchy="false">)</mo></math>">f()f().
      • Can involve fine-tuning the pre-trained model. We can fine-tune http://www.w3.org/1998/Math/MathML"><mi>f</mi><mo stretchy="false">(</mo><mo stretchy="false">)</mo></math>">f()f() as well during the learning process.
    • In machine learning (ML) the learning process itself can qualify as just fine-tuning because the update process takes very tiny steps in the opposite direction of the gradient. But fine-tuning is normally reserved as the final stage that involves setting the learning rate very low so as to fine adjust the weights. The learning process normally starts with a very poor initial state with a fairly large learning rate and then may involve a fine tuning phase with a small learning rate. You can in fact split the learning phase into multiple fine-tuning phases each with a smaller learning rate than the last.

    Thus there is a difference between the two.

      October 27, 2021 2:04 PM IST
    0