QBoard » Artificial Intelligence & ML » AI and ML - Python » Convolutional Neural Networks - Multiple Channels

Convolutional Neural Networks - Multiple Channels

  • How is the convolution operation carried out when multiple channels are present at the input layer? (e.g. RGB)

    After doing some reading on the architecture/implementation of a CNN I understand that each neuron in a feature map references NxM pixels of an image as defined by the kernel size. Each pixel is then factored by the feature maps learned NxM weight set (the kernel/filter), summed, and input into an activation function. For a simple grey scale image, I imagine the operation would be something adhere to the following pseudo code:

    for i in range(0, image_width-kernel_width+1):
        for j in range(0, image_height-kernel_height+1):
            for x in range(0, kernel_width):
                for y in range(0, kernel_height):
                    sum += kernel[x,y] * image
    
            feature_map = act_func(sum)
            sum = 0.0

    However I don't understand how to extend this model to handle multiple channels. Are three separate weight sets required per feature map, shared between each colour?

    Referencing this tutorial's 'Shared Weights' section: http://deeplearning.net/tutorial/lenet.html Each neuron in a feature map references layer m-1 with colours being referenced from separate neurons. I don't understand the relationship they are expressing here. Are the neurons kernels or pixels and why do they reference separate parts of the image?

    Based on my example, it would seem that a single neurons kernel is exclusive to a particular region in an image. Why have they split the RGB component over several regions?

      September 4, 2021 12:58 AM IST
    0
  • Max doesn't quite make sense, since channels are supposed to be independent. Taking max of the results from different filters on different channels is mixing different aspects together.

    For combining outputs from different channels, basically we need a func to add the output together. The choice of the addition func here in my opinion can vary depending on the use cases. One implementation is just to do a summation, according to pytorch conv2d implementation. see https://pytorch.org/docs/stable/nn.html for details

      September 22, 2021 12:06 AM IST
    0
  • How is the convolution operation carried out when multiple channels are present at the input layer? (e.g. RGB)

    In such a case you have one 2D kernel per input channel (a.k.a plane).

    So you perform each convolution (2D Input, 2D kernel) separately and you sum the contributions which gives the final output feature map.

    Please refer to the slide 64 of this CVPR 2014 tutorial by Marc'Aurelio Ranzato:

    enter image description here

    Are three separate weight sets required per feature map, shared between each colour?

    If you consider a given output feature map, you have 3 x 2D kernels (i.e one kernel per input channel). Each 2D kernel shares the same weights along the whole input channel (R, G, or B here).

    So the whole convolutional layer is a 4D-tensor (nb. input planes x nb. output planes x kernel width x kernel height).

    Why have they split the RGB component over several regions?

    As detailed above think of each R, G and B channel as a separate input plane with its dedicated 2D kernel.

      September 22, 2021 2:21 PM IST
    0
  • The first step of 2D convolution for multi-channels: each of the kernels in the filter are applied to three channels in the input layer, separately. The image is adopted from this link. Then these three channels are summed together (element-wise addition) to form one single channel (3 x 3 x 1).
      November 29, 2021 11:57 AM IST
    0
  • For example, if your input image is of size W x H x C where W, H, and C represent the length of width, height, and the size of channels. The dimension of the filter (aka kernel) would be K x K x C where K denotes the length of the dimension of the kernel. Using max to aggregate the results of different channels fails to distinguish the nuances across channels, which is not what we want. As illustrated in the figure below (source), the input data is of size 6 x 6 x 3. The number of units (filters) is 2, each of which has the dimensions 3 x 3 x 3. The output is 4 x 4 x 2. So in general channels need to be treated separately under each filter.
    enter image description here
      September 9, 2021 4:30 PM IST
    0