Neutral Network
Categories:
patience in the face of such frustration is the only way to truly understand and internalize a subject. FIND YOUR PROJECT
Overview
recognize hand digits
Let’s start from a basic task, recognize those digits we don’t usually appreciate how tough a problem our visual systems solve. write a computer program to recognize digits like those above try to make such rules precise will got tou confuse
Neural networks
The idea is to take a large number of handwritten digits
develop a system which can learn from those training examples
artificial neuron called a perceptron. the main neuron model used is one called the sigmoid neuron
write a computer program implementing a neural network
ocusing on handwriting recognition because it’s an excellent prototype problem for learning about neural networks in general.
improve accuracy
Perceptrons
a perceptrons is a device to add evidence to make decisions
perceptron is that it’s a device that makes decisions by weighing up evidence
###### how do perceptrons work? ##### algebraic terms A perceptron takes several binary inputs, x1,x2,…x1,x2,…, and produces a single binary output:
Rosenblatt use weights the importance of the respective inputs to the output
The neuron’s output, 0 or 1, is determined by whether the weighted sum is less than or greater than some threshold value.
A way you can think about the perceptron is that it’s a device that makes decisions by weighing up evidence.
By varying the weights and the threshold, we can get different models of decision-making.
we’ll call the first layer of perceptrons - is making three very simple decisions, by weighing the input evidence.
In this way a perceptron in the second layer can make a decision at a more complex and more abstract level than perceptrons in the first layer.
In this way, a many-layer network of perceptrons can engage in sophisticated decision making.
In fact, they’re still single output. The multiple output arrows are merely a useful way of indicating that the output from a perceptron is being used as the input to several other perceptrons.
simplify one The first change is to write
as a dot product
w and x are vectors whose components are the weights and inputs
move the threshold to the other side of the inequality, and to replace it by what’s known as the perceptron’s bias
bias can be seen as a way to measure how easy it is to get the perceptron to output a 1
the bias is a measure of how easy it is to get the perceptron to fire.
it leads to further notational simplifications. Because of this, in the remainder of the book we won’t use the threshold, we’ll always use the bias.
###### compute the elementary logical functions
suppose we have a perceptron with two inputs, each with weight −2, and an overall bias of 3.
the input 1 produces output 0,
nputs 1 and 0 produce output 1
NAND gate!
we can use networks of perceptrons to compute any logical function at all.
Actually, in neural networks, a neuron has only one output, which is then broadcasted to all its outgoing connections.
for encoding input, we define a input layer, This notation for input perceptrons, in which we have an output, but no inputs. better to consider special units which are simply defined to output the desired values.
It turns out that we can devise learning algorithms which can automatically tune the weights and biases of a network of artificial neurons.
Sigmoid neurons
we already know that our neural networks can simply learn to solve problems, automatically tune the weights and biases of a network of artificial neurons.
This tuning happens in response to external stimuli, without direct intervention by a programmer.
how can we devise such algorithms for a neural network?
before design ourself, let’s see how network behave when a small change happen
a small change will cause a small corresponding change in the output
this property will make learning possible what we need is to make small change and let output closer to actual value.
But the problem is, when small change happens. output from perceptron more likely to flip like 0 -> 1, That flip may then cause the output overcontrol and unpredictable
That’s where sigmoid neuron comes
We can overcome this problem by introducing a new type of artificial neuron called a sigmoid neuron
Sigmoid neurons are similar to perceptrons, but modified so that small changes in their weights and bias cause only a small change in their output.
output instead of 0 and 1 is 
where σ is called the sigmoid function or logistic neurons
we put it explicaitly
we can take the algebraic form as a technical detail than barrier of understatnding
the similarity to the perceptron model
when z is large and positive, the output from the sigmoid neuron is approximately 1, On the Other hand the output is approximately 0
the behaviour of a sigmoid neuron also closely approximates a perceptron. Only z has a modest size that there’s much deviation from the perceptron model.
Indeed, it’s the smoothness of the σσ function that is the crucial fact, not its detailed form.
and we have another version
In that case, the the sigmoid neuron would be a perceptron since the output would be 1 or 0 (we ignore modest value) so sigmoid is basicly smoother version of perceptron
Indeed, it’s the smoothness of the σσ function that is the crucial fact
we already know 
can be approximated by
Don’t panic !!! Only the shape of fuction matters, actually we will talk more for other activation function
where the output is f(w⋅x+b) for some other activation function f(⋅)
Δoutput is a linear function of the changes Δwj and Δb in the weights and bias
This linearity makes it easy to choose small changes in the weights and biases to achieve any desired small change in the output.
So sigmod simulating perceptrons make it much easier to figure out how changing the weights and biases will change the output.
Anyway, one big difference between perceptrons and sigmoid neurons is that sigmoid neurons don’t just output 0 or 1.
Exercises
Sigmoid neurons simulating perceptrons, part I
Suppose we take all the weights and biases in a network of perceptrons, and multiply them by a positive constant, c>0. Show that the behaviour of the network doesn’t change.
Sigmoid neurons simulating perceptrons, part II
Suppose we have the same setup as the last problem - a network of perceptrons. Suppose also that the overall input to the network of perceptrons has been chosen. We won’t need the actual input value, we just need the input to have been fixed.
Suppose the weights and biases are such that 
for the input x to any particular perceptron in the network.
Now replace all the perceptrons in the network by sigmoid neurons, and multiply the weights and biases by a positive constant c>0.
Show that in the limit as c→∞ the behaviour of this network of sigmoid neurons is exactly the same as the network of perceptrons.
How can this fail when w⋅x+b=0 for one of the perceptrons?
