Trying to Understand Python

[admiralk]

I have been getting into machine learning lately and ell the examples or descriptions use python. The one intro I am trying to use has a line in the code that simply will not work for me, also I am unable to figure out why.

I get the same error writing the entire code, minimalist as it is, but I have narrowed it down to this.

Code: [Select]

    def sigmoid(z):
        return 1.0/(1.0+np.exp(-z))
     
    def feedforward(self, a):
        for b, w in zip(self.biases, self.weights):
            a = sigmoid(np.dot(w, a)+b)
        return a
The error is undefined name 'sigmoid'. I was able to figure out other syntax errors, but this one has me stumped. There are other uses of the function that do not throw an error. and I only moved the function to where it is to disprove a linear problem.

If anyone has an idea what I am missing, it would be greatly appreciated.

[ataradov]

Is sigmoid() defined inside the class? Alignment seems to suggest so, but then it should be called as self.sigmoid() and should take "self" as a first argument.

Can you attach a complete minimal example as a text file? It would be much easier to figure out, there is not enough context here.

[admiralk]

Yes, it is all one class.  Adding self did remove the error, but I still do not understand why.

The code comes from this series of videos
The full code is here https://github.com/stephencwelch/Neural-Networks-Demystified

The particular file is

Code: [Select]

"""
network.py
~~~~~~~~~~

A module to implement the stochastic gradient descent learning
algorithm for a feedforward neural network.  Gradients are calculated
using backpropagation.  Note that I have focused on making the code
simple, easily readable, and easily modifiable.  It is not optimized,
and omits many desirable features.
"""

#### Libraries
# Standard library
import random

# Third-party libraries
import numpy as np

class Network(object):

    def __init__(self, sizes):
        """The list ``sizes`` contains the number of neurons in the
        respective layers of the network.  For example, if the list
        was [2, 3, 1] then it would be a three-layer network, with the
        first layer containing 2 neurons, the second layer 3 neurons,
        and the third layer 1 neuron.  The biases and weights for the
        network are initialized randomly, using a Gaussian
        distribution with mean 0, and variance 1.  Note that the first
        layer is assumed to be an input layer, and by convention we
        won't set any biases for those neurons, since biases are only
        ever used in computing the outputs from later layers."""
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
        self.weights = [np.random.randn(y, x)
                        for x, y in zip(sizes[:-1], sizes[1:])]

    def feedforward(self, a):
        """Return the output of the network if ``a`` is input."""
        for b, w in zip(self.biases, self.weights):
            a = sigmoid(np.dot(w, a)+b)
        return a

    def SGD(self, training_data, epochs, mini_batch_size, eta,
            test_data=None):
        """Train the neural network using mini-batch stochastic
        gradient descent.  The ``training_data`` is a list of tuples
        ``(x, y)`` representing the training inputs and the desired
        outputs.  The other non-optional parameters are
        self-explanatory.  If ``test_data`` is provided then the
        network will be evaluated against the test data after each
        epoch, and partial progress printed out.  This is useful for
        tracking progress, but slows things down substantially."""
        if test_data: n_test = len(test_data)
        n = len(training_data)
        for j in xrange(epochs):
            random.shuffle(training_data)
            mini_batches = [
                training_data[k:k+mini_batch_size]
                for k in xrange(0, n, mini_batch_size)]
            for mini_batch in mini_batches:
                self.update_mini_batch(mini_batch, eta)
            if test_data:
                print "Epoch {0}: {1} / {2}".format(
                    j, self.evaluate(test_data), n_test)
            else:
                print "Epoch {0} complete".format(j)

    def update_mini_batch(self, mini_batch, eta):
        """Update the network's weights and biases by applying
        gradient descent using backpropagation to a single mini batch.
        The ``mini_batch`` is a list of tuples ``(x, y)``, and ``eta``
        is the learning rate."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        for x, y in mini_batch:
            delta_nabla_b, delta_nabla_w = self.backprop(x, y)
            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
        self.weights = [w-(eta/len(mini_batch))*nw
                        for w, nw in zip(self.weights, nabla_w)]
        self.biases = [b-(eta/len(mini_batch))*nb
                       for b, nb in zip(self.biases, nabla_b)]

    def backprop(self, x, y):
        """Return a tuple ``(nabla_b, nabla_w)`` representing the
        gradient for the cost function C_x.  ``nabla_b`` and
        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
        to ``self.biases`` and ``self.weights``."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        # feedforward
        activation = x
        activations = [x] # list to store all the activations, layer by layer
        zs = [] # list to store all the z vectors, layer by layer
        for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)
        # backward pass
        delta = self.cost_derivative(activations[-1], y) * \
            sigmoid_prime(zs[-1])
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, activations[-2].transpose())
        # Note that the variable l in the loop below is used a little
        # differently to the notation in Chapter 2 of the book.  Here,
        # l = 1 means the last layer of neurons, l = 2 is the
        # second-last layer, and so on.  It's a renumbering of the
        # scheme in the book, used here to take advantage of the fact
        # that Python can use negative indices in lists.
        for l in xrange(2, self.num_layers):
            z = zs[-l]
            sp = sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
        return (nabla_b, nabla_w)

    def evaluate(self, test_data):
        """Return the number of test inputs for which the neural
        network outputs the correct result. Note that the neural
        network's output is assumed to be the index of whichever
        neuron in the final layer has the highest activation."""
        test_results = [(np.argmax(self.feedforward(x)), y)
                        for (x, y) in test_data]
        return sum(int(x == y) for (x, y) in test_results)

    def cost_derivative(self, output_activations, y):
        """Return the vector of partial derivatives \partial C_x /
        \partial a for the output activations."""
        return (output_activations-y)

#### Miscellaneous functions
def sigmoid(z):
    """The sigmoid function."""
    return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    """Derivative of the sigmoid function."""
    return sigmoid(z)*(1-sigmoid(z))


I do not get errors on the other calls to sigmoid(), although that might just be because Python stops looking after the first one?

I will not get into why VSCode  does not work but spider works fine since I really do not care, as long as I can get something to work.
Thanks for looking ataradov

[ataradov]

This code has sigmoid() outside of the class as a helper function. Either is fine, but the semantics are different and the call differs accordingly.

[admiralk]

OK, I see that now. Never worked with Python before and I find it confusing. I assume I need to add self to sigmoid_prime() as well to get it to work, or get rid of the white space?

Hopefully I can get things working this time. Do not care if it works right, just that it does something at this point.

Thanks.

[ataradov]

Python has significant white spaces, alignment matters. What you need to do depends on your goal. If you want those functions inside the class, they need to take self as a first argument, if you are fine with them being outside, you don't need to do anything special for the calls, but then they need to be located outside of the class.

[admiralk]

I am just trying to understand the algorithms. I was able to figure out the other errors I made on my own, but this one eluded me, I am used to C/C++. White spaces are for clarity in there and not part of the syntax. 

It is also frustrating that variables do not need types and even more so that every example I have seen so far uses horrendous names for them.

[IanB]

It's really important to recognize that whitespace (indentation) is significant in Python, and takes the place of braces {} in languages like C.

Also Python is a scripting (rapid development, prototyping) language, and the lack of enforced types is a part of that.

[ataradov]

And in recent versions there is optional strong typing, but it kind of defeats the purpose.

But also, most of those basic NN algorithms are trivial matrix operations, you should be able to just implement them in C++ if you are more familiar with it.

[admiralk]

That is the idea, but I need to understand the algorithm first. Every AI tutorial/example I have found uses Python. It has been a while, but I could probably dig up the C++ matrix operations from school work. Pretty basic unless you are going for ultimate optimization. I am just trying to figure out how it al works.

[jeremy]

Depending on what you preferences are, you might be better off using an IDE like PyCharm Community. It has some static analysis tools which would have given you a warning that "self" was missing. I think maybe VSCode can also do a similar thing too? I just tried your code now in PyCharm and it automatically added the self while I was typing, but even when I manually deleted it there were warnings like "Usually first parameter to a method is named self".

I think the terrible variable names is just a bad research-ism habit which has escaped from academia now that machine learning has become a lot more popular. Everyone can name variables poorly, but if you've ever tried to understand a complex MATLAB library written by an academic, you'll know what I mean.

I think the main issue is that in the mathematical notation that comes from the research articles everything is i, j, k, n, phi, etc but there are other indicators like fonts, greek symbols, subscript/superscript etc that get lost when converting to ASCII. It gets worse when people do things like represent lowercase "eta" (the greek letter) as "n", rather than calling the variable "efficiency" or something like that...

[tszaboo]

That is the idea, but I need to understand the algorithm first. Every AI tutorial/example I have found uses Python. It has been a while, but I could probably dig up the C++ matrix operations from school work. Pretty basic unless you are going for ultimate optimization. I am just trying to figure out how it al works.

It just provides a function, which returns a value between 0 and 1, non linear, based on the input value.
It has been shown that the function actually doesn't really matter, because all that matters is that it is not linear. The learning and the coefficients of each neuron takes care of the rest.

[janoc]

Is sigmoid() defined inside the class? Alignment seems to suggest so, but then it should be called as self.sigmoid() and should take "self" as a first argument.

Not necessarily - adding "self" makes it into a member function (method). Without self it is a so called "static" function and needs to be called as "Classname.sigmoid(...)" where Classname is the name of the class it was defined in. This is useful for scoping (polluting the global/module namespace is bad style) and typically used for various factory functions or utilities that don't require an object instance to work.

There is a third option, a so called class method - then the first argument isn't "self" (instance/object) but the class itself it was called on. That has sometimes uses too.

However, in modern Python one is supposed to annotate both static and class methods using appropriate decorators - @classmethod and @staticmethod to make this explicit. It will work without but it is a bad style - and a class method can't be really declared otherwise. Python's "self" is not a special keyword, only a conventional name for the first argument - so without the decorator Python wouldn't know if a normal method or a class method was intended.

And in recent versions there is optional strong typing, but it kind of defeats the purpose.

One isn't required to use it. The entire point of it is that one can catch many errors that would otherwise only manifest themselves at runtime by throwing exceptions because some unexpected type/object has been passed into a function, or perhaps the object has the right type but someone forgot to update some part of the code after refactoring - and now some attribute is missing/has been renamed, resulting in AttributeError exceptions.

Unless you have a 100% test coverage (and even if you do, it doesn't ensure that one is actually testing meaningful things) this is bound to happen to you sooner or later, especially for various corner cases on code paths that don't get exercised often. This is a major pain with dynamic languages like Python or Javascript and that's why in both cases the optional types are available (Typescript in the case of Javascript).  It is one more tool to make production Python code more robust without sacrificing flexibility of the language.

There are also some other applications of it - e.g. the IDEs use the typing information where available to both lint your code checking for such mistakes and also to provide better suggestions/completions. That's what VSCode does, for example - Microsoft's Pylance plugin is all based around static analysis and typing. If the typing information isn't available, one gets only very basic functionality from the IDE. Also JITs and the language virtual machines can use the typing info for optimizations (Python doesn't do that yet).

Adding optional strong typing to a dynamically typed language for these reasons is nothing new - e.g. Common Lisp had optional types for decades. There it helps both with generating diagnostics and also with compiler optimizations.

[admiralk]

Thanks for all the tips.

I finally got it all working, after fixing several typos. I am still pretty shaky on how Python works. I would like to be able to use VSCode, but it does not seem to be able to find numpy for some reason. If I could find a tutorial on Python for someone that already knows how to program it would be helpful. It is just too painful going through all the basics in what I have found so far, just to try and pick out what is specific to Python.

I did not post the correct source for this though. What I am actually working on now is http://neuralnetworksanddeeplearning.com/.

I want to be able to use data from these sensors https://www.st.com/en/imaging-and-photonics-solutions/vl53l5cx.html initially, and eventually some sort of camera. 

[IanB]

Thanks for all the tips.

I finally got it all working, after fixing several typos. I am still pretty shaky on how Python works. I would like to be able to use VSCode, but it does not seem to be able to find numpy for some reason. If I could find a tutorial on Python for someone that already knows how to program it would be helpful. It is just too painful going through all the basics in what I have found so far, just to try and pick out what is specific to Python.

A good way to use Python for ML and other things is to use Jupyter Notebook, it is very interactive and shows you plots and graphs interleaved with your input.

I haven't really used Python with VSCode, but it is not Code that has to find a package like numpy. Numpy is installed as a Python extension using pip, and then an IDE like Code just uses the version of Python you tell it to. If numpy is not recognized, you are probably not pointing Code at the right version of Python on your computer.

A good hint for using Python for ML and visualization is to use the Anaconda distribution. Anaconda comes with all the important numerical and graphing packages included out of the box, so it simplifies things tremendously.

[IanB]

If I could find a tutorial on Python for someone that already knows how to program it would be helpful.

Just go to https://www.coursera.org/ and search for Python, machine learning, or anything that interests you. There are many free courses, and you can usually view (or "audit") paid courses for free, so you can access a lot of learning without buying a paid subscription.

[admiralk]

A good way to use Python for ML and other things is to use Jupyter Notebook, it is very interactive and shows you plots and graphs interleaved with your input.

I haven't really used Python with VSCode, but it is not Code that has to find a package like numpy. Numpy is installed as a Python extension using pip, and then an IDE like Code just uses the version of Python you tell it to. If numpy is not recognized, you are probably not pointing Code at the right version of Python on your computer.

A good hint for using Python for ML and visualization is to use the Anaconda distribution. Anaconda comes with all the important numerical and graphing packages included out of the box, so it simplifies things tremendously.

I think you are right about VSCode not seeing the right version. It is supposed to be 2.7.15, the Anaconda version, and that is what it says in the status bar, but checking the version tells me 2.7.16, which is the version I have associated with GNAT studio. I am getting used to Spyder though, so I probably will not worry about it too much.

I dislike cloud/web applications, so aside from taking a look at notebooks I pretty much ignored it.

[jeremy]

You probably should change over to python 3, python2.7 is very obsolete and you’ll probably find that a lot of tutorials and libraries simply don’t work with it.

[IanB]

I dislike cloud/web applications, so aside from taking a look at notebooks I pretty much ignored it.

Jupyter is not a cloud/web app, it installs and runs locally on your computer. It just uses a browser window for its UI.

You probably should change over to python 3, python2.7 is very obsolete and you’ll probably find that a lot of tutorials and libraries simply don’t work with it.

This. Python 3.10 is the current shipping version.

[admiralk]

That could be, but all the ML examples I have seen so far use 2.7, so I would think using 3.x would be more problematic. Also since I am on a win7 machine, the latest versions simply will not work to begin with.  I have plenty of VMs; Mac, Linux, and windows from DOS to 10., but I prefer to work natively when I can.

@IanB I should take another look in that case. Maybe I misunderstood what I was looking at?

[janoc]

That could be, but all the ML examples I have seen so far use 2.7, so I would think using 3.x would be more problematic. Also since I am on a win7 machine, the latest versions simply will not work to begin with.  I have plenty of VMs; Mac, Linux, and windows from DOS to 10., but I prefer to work natively when I can.

Then time to find more up to date/recent examples. Python 2 has been EOL for more than 10 years now. 

Pretty much no current versions of major libraries support Python 2.x anymore. That includes stuff like Pytorch, Numpy, Scipy, Tensorflow ...
So you are wasting your time building on obsolete tools that will give you no end of grief when you will need to update something - and everything breaks because it isn't working in Python 2 anymore.

Windows 7 is supported up to Python 3.9 so you should have no issues with finding a more recent toolset.

Really, don't write new Python 2 code. You are setting yourself up for a ton of completely avoidable pain.

[admiralk]

Well, I am not planning on writing very much new Python code. I will be using C/C++ when I start writing for myself. I just need to understand the Python so I can understand, and work thought, the examples I find. Which reminds me, I need to look up tuples as I do not think they are what I am thinking they are.

[janoc]

Well, I am not planning on writing very much new Python code. I will be using C/C++ when I start writing for myself. I just need to understand the Python so I can understand, and work thought, the examples I find. Which reminds me, I need to look up tuples as I do not think they are what I am thinking they are.

Then for your own sake, do not learn Python 2! Python 3 is sufficiently different from Python 2 that you will struggle trying to figure out why things aren't working. I mean, it is still Python but it is like learning C++98 today because you have found some tutorial for it - all the while everyone else uses C++11 as a minimum and a lot of new code simply expects at least C++17 or newer. And then you are stumped seeing black magic like lambdas and people talk about "universal" references for whatever reason.

[admiralk]

Thanks for the warning, but as far as I understand it Python 2 was never the same as Python 3. 3 was not just an updated version like C++98 vs C++17. Therefore I never expected code written in one to work flawlessly with the other, anymore than I would expect the same from code written in any two different languages, regardless of the similarities.

I mentioned C++ out of hand, but I have been looking to give Ada a try. I would certainly prefer learning modern Ada over modern Python.

[SiliconWizard]

Well, I am not planning on writing very much new Python code. I will be using C/C++ when I start writing for myself. I just need to understand the Python so I can understand, and work thought, the examples I find. Which reminds me, I need to look up tuples as I do not think they are what I am thinking they are.

Then for your own sake, do not learn Python 2!