QBoard » Artificial Intelligence & ML » AI and ML - Conceptual » What is the role of the bias in neural networks?

What is the role of the bias in neural networks?

  • I'm aware of the gradient descent and the back-propagation algorithm. What I don't get is: when is using a bias important and how do you use it?

    For example, when mapping the AND function, when I use 2 inputs and 1 output, it does not give the correct weights, however, when I use 3 inputs (1 of which is a bias), it gives the correct weights.

      August 21, 2020 4:32 PM IST
    0
  • Just to add my two cents.

    A simpler way to understand what the bias is: it is somehow similar to the constant b of a linear function

    y = ax + b

    It allows you to move the line up and down to fit the prediction with the data better. Without b the line always goes through the origin (0, 0) and you may get a poorer fit.

      August 21, 2020 4:37 PM IST
    1

  • Its an additional parameter in NN..adjust
    Output with weighted sum of inputs to neurons.
    For the best model fit  bias helps to provide the best possible way .
      October 6, 2020 3:44 PM IST
    0
  • The activation function in Neural Networks takes an input 'x' multiplied by a weight 'w'. Bias allows you to shift the activation function by adding a constant (i.e. the given bias) to the input. Bias in Neural Networks can be thought of as analogous to the role of a constant in a linear function, whereby the line is effectively transposed by the constant value.

    In a scenario with no bias, the input to the activation function is 'x' multiplied by the connection weight 'w0'.

    In a scenario with bias, the input to the activation function is 'x' times the connection weight 'w0' plus the bias times the connection weight for the bias 'w1'. This has the effect of shifting the activation function by a constant amount (b * w1).

      September 18, 2020 5:19 PM IST
    0
  • A layer in a neural network without a bias is nothing more than the multiplication of an input vector with a matrix. (The output vector might be passed through a sigmoid function for normalisation and for use in multi-layered ANN afterwards but that’s not important.)

    This means that you’re using a linear function and thus an input of all zeros will always be mapped to an output of all zeros. This might be a reasonable solution for some systems but in general it is too restrictive.

    Using a bias, you’re effectively adding another dimension to your input space, which always takes the value one, so you’re avoiding an input vector of all zeros. You don’t lose any generality by this because your trained weight matrix needs not be surjective, so it still can map to all values previously possible.

    2d ANN:

    For a ANN mapping two dimensions to one dimension, as in reproducing the AND or the OR (or XOR) functions, you can think of a neuronal network as doing the following:

    On the 2d plane mark all positions of input vectors. So, for boolean values, you’d want to mark (-1,-1), (1,1), (-1,1), (1,-1). What your ANN now does is drawing a straight line on the 2d plane, separating the positive output from the negative output values.

    Without bias, this straight line has to go through zero, whereas with bias, you’re free to put it anywhere. So, you’ll see that without bias you’re facing a problem with the AND function, since you can’t put both (1,-1) and (-1,1) to the negative side. (They are not allowed to be on the line.) The problem is equal for the OR function. With a bias, however, it’s easy to draw the line.

    Note that the XOR function in that situation can’t be solved even with bias

      September 18, 2020 5:21 PM IST
    0
  • When you use ANNs, you rarely know about the internals of the systems you want to learn. Some things cannot be learned without a bias. E.g., have a look at the following data: (0, 1), (1, 1), (2, 1), basically a function that maps any x to 1.

    If you have a one layered network (or a linear mapping), you cannot find a solution. However, if you have a bias it's trivial!

    In an ideal setting, a bias could also map all points to the mean of the target points and let the hidden neurons model the differences from that point.

      September 18, 2020 5:22 PM IST
    0
  • Modification of neuron WEIGHTS alone only serves to manipulate the shape/curvature of your transfer function, and not its equilibrium/zero crossing point.

    The introduction of bias neurons allows you to shift the transfer function curve horizontally (left/right) along the input axis while leaving the shape/curvature unaltered. This will allow the network to produce arbitrary outputs different from the defaults and hence you can customize/shift the input-to-output mapping to suit your particular needs.

    See here for graphical explanation: http://www.heatonresearch.com/wiki/Bias

      September 18, 2020 5:45 PM IST
    0
  • For all the ML books I studied, the W is always defined as the connectivity index between two neurons , which means the higher connectivity between two neurons , the stronger the signals will be transmitted from the firing neuron to the target neuron or Y= w * X as a result to maintain the biological character of neurons, we need to keep the 1 >=W >= -1 , but in the real regression, the W will end up with |W| >=1 which contradict with how the Neurons are working, as a result I propose W= cos(theta) , while 1 >=| cos( theta)| , and Y= a * X = W * X + b while a = b + W = b + cos( theta) , b is an integer
      December 29, 2020 3:21 PM IST
    0