EEVblog Electronics Community Forum
General => General Technical Chat => Topic started by: RJSV on September 16, 2022, 08:55:43 pm
-
Hi, I'm building some simple structures to do binary decode, classic EE 101 style, starting with the 4 bits input, then having the explicit output set; 0 through 15.
By analogy, that could resemble the 'expansion' of terms that happens in layers of an AI Network.
Those are all the single connections, to next layer, which I assume have the weights assigned.
With binary, you get a perfect set of permutations or 16 possible states. So, I'm guessing, 'finite state' definition applies...( ??? ).
Then, moving down the signal path, those 16 possible states get interpreted...; meaning that some 4 bit combinations don't make sense, in context. For example, I'm doing Motion Detection hardware, so some combinations of (binary) sensors don't represent any clear movement, left to right.
But, that 'reduction', is being done by design, that being, again as example, motion from rt. to left, or from left to right, forming a 'smaller' set of conclusions.
The very simple set, could conclude; No Action, Motion to Left occurred, or Motion to the right occurred.
Notice that's a REDUCTION, from 16 terms, to more like 3 conclusive terms. So, of course, a Neural Net is going to do this process, between 'Reloe' and 'Pooling',
but I'm curious, how close these two processes are ?
In other words, going from 4 (encoded) bits, to 16 terms, to 3 'summary' terms but without Analog 'weights'... Is this a simplified Network ? Or just some logic custom-wired to a few tasks ? Thanks.
- - Rick B.
-
Convolution is used for downsizing, typically in image recognition. It is also used for modifying an image to find edges and other features. Blurring is another use for convolution. I don't see where it is applicable to this problem. https://setosa.io/ev/image-kernels/ (https://setosa.io/ev/image-kernels/)
ReLU, sigmoid and tanh are from a set of possible activation functions and force the output of a layer to constrain values in some range. ReLU eliminates values less than 0 by simply setting the output to 0 when x < 0 or equal to x for x > 0. Sigmoid constrains the values to 0..+1 and tanh constrains the values to -1..+1. You can experiment with this selection but the results will probably be the same regardless of your choice. But it's easy to experiment...
For this problem, the hidden layer(s) will probably use ReLU and the output layer (with 16 outputs) will probably use softmax which will force the outputs to a percentage of probability (the total over all 16 outputs will be 1.0) and the highest probability wins.
https://machinelearningmastery.com/softmax-activation-function-with-python (https://machinelearningmastery.com/softmax-activation-function-with-python)
You can get some ideas from the XOR problem.
https://blog.thoughtram.io/machine-learning/2016/11/02/understanding-XOR-with-keras-and-tensorlow.html (https://blog.thoughtram.io/machine-learning/2016/11/02/understanding-XOR-with-keras-and-tensorlow.html)
Your problem will have 4 bits of input (the low 4 bits of a 32 bit value) and the hidden and output layers will have 16 outputs. If you only want 3 discrete outputs, make the softmax output layer output 3 values instead of 16. I think the first hidden layer should still have 16 outputs and that might be the only hidden layer. The output layer would just have 3 outputs and still use softmax. Your label data, matching the training/test data, determines what the output will be. It will be a 16 bit representation of the 4 bit input.
You can make training/test data with the numpy randint() function.
If I were doing this, I would use tensorflow and keras. I would get very close to the book "Deep Learning with Python" by Francois Chollet. I like this book a lot. I have several others but this one works for me. And the code just works, no typesetting problems where zero and O or 1 and l get scrambled.
https://www.amazon.com/Learning-Python-Second-Fran%C3%A7ois-Chollet/dp/1617296864 (https://www.amazon.com/Learning-Python-Second-Fran%C3%A7ois-Chollet/dp/1617296864)
Or I would use the Machine Learning toolbox in MATLAB.
It is very easy to experiment with various models and activation functions when you use something like tensorflow/keras.
I wouldn't use a DNN to solve this problem unless it was an assignment. A simple switch/case statement in C would do the job better.
-
Thank you, for careful response. (I'm a big fan, of lower level logic, D to A converters, etc.)
I believe the convolution aspect is done manually, in my crude, caracature of a network, that meaning 'staring' at a very few patterns, on notebook there, while deciding which ones imply motion...I'm thinking motion dynamics is very similar to edge detect.
But I'm seeing a similar 'expansion, reduction' cycling, while the data is actually just one bit. That bit originates at the sensor. In the video lecture, Professor Fan discussed a 'real-time Processing Network'. I'm doing 2 bits from sensors, with another 2 bits holding older sample (20 mSec older).
All of this really, for education, on some of the recently popular network theory!
-
From what little I know, if you want to detect a sequence (sensors changing state), it is a slightly different process than simply decoding the inputs. Here's a paper on the subject: https://arxiv.org/pdf/1802.02046.pdf
I know absolutely nothing about the process but Google turns up hits for 'neural network sequence detection' - no quotes
-
Thanks, I'm just reasoning things out. Considering having a pair of opto-sensors, 4 meters apart, off on sidewalk. Those could make pulses that you then decode. At 30 meters per second, those two impulses from car motion would be some 100 milli-seconds apart, but also containing info, by which sensor got the (shadow) first.
For example, the classic 'falling edge' for the left side sensor, would be a 'zero', currently, with a 'one' state just recently; That along with a rising edge, on the right side sensor, (similar pattern of older=0, current=1),
that's makes for 2 compelling arguments, to conclude that motion from left to right just occurred.
My point about the 4 bits, 2 of a new input, 1 from each digital sensor, and then another 2 a saved older copy, is that, combinatorially, there are exactly 16 ways that can happen.
Of course, with bigger inputs, that number gets colossal pretty fast. Just with 4 corners, and 2 older copies (shifted), that's 12 bits..., So that's 4096 variations that would need to be checked, if MANUAL pre- determinations are done.
Oh, and also it's the case, where each point relates to 11 others, in space and in the back-time copies, so that there's A LOT of ambiguous patterns, that aren't clearly confused, but don't perfectly indicate some outcome, in this case, some perfectly indicated direction of travel.
Plus, most real-life sensing has the input pulses overlapping, in time.
As for 'POOLING', I've done an approximation, just by dropping half of the sensors, although losing some data. You probably can appreciate, having TWO sensing elements, allows some (crude) estimate, of motion object's speed, at least in terms like:
'Moving slow to left direction'.
I'm also thinking about, using weights after the first stages are digital (digital output from sensors).
-
So, part of the series connected Network could also, keep in mind; could pass audio or music, down the line-up, although of bad quality. You're hearing basic beat, in hf hiss, plus some very muffled vocals, all very quiet there, thru the headphones.
I see that series transfer as almost like a single bit of DtoA converted and output, via each inverter gate in turn. Supplementing that (string), you add some means of 'Escapement' gating, for synchronous access, to each down the line.
I think it's called a 'Computational Network', (although sounds like double-speak), for having the series bit state progress, in start-stop fashion, or shift-register style action, down the simple line-up of binary inverters.
-
Thanks rstofer, for that reference to XOR function (network), here is some more info, to be clear on manual decode efforts:
For the 16 possible combinations, of the 4 bits, there was only 2 places where there was a clearly defined motion direction. Figuring through the others, the likely motion to leftward would be '1001' where the upper 2 bits there are the currently read pair. Motion rightward would be '0110' which is; '01'=located on right side,
'10'= was on left side.
In 5 of the cases, things are either both blank now, or both tenetive so those cases are concluded
'NO ACTION' for the current cycle.
To illustrate, suppose an additional two copies are maintained, for a total 8 bits, and suppose one of those comes 'ON',...a stray, pattern '0100 1001' ; that could almost be ignored, for a moment, by concluding that leftward motion is likely.
At any rate, some of the other (manually scanned) patterns include 2 tentative motions, such as '0100'= new on left, and also '1000' = new on right. So those two have no action, as the 'history' bit pair are '00'.
Hopefully, I've started to illustrate how a system having 12 bits or more, of state and memory, can get collosal but also subtle, making manual evaluation, or pre-calculation too burdensome.
-
MANY years ago, in a compiler construction class, we were introduced to state transition matrices (instead of state diagrams). Draw a 16x16 matrix, label the rows 'Present State', label the columns 'Inputs' and fill in each cell with the next state given the present state and the new inputs. You can have a parallel matrix with actions to take upon entering a new state (if any) or just a single 3 dimensional matrix. If you need more history than just the present state, I suppose you could put the preceding state on a stack or something before changing to the new state. This would give you a backtrace but somehow you need to limit depth.
DNNs are easy to create with Python-TensorFlow-Keras, just code something up and experiment.
Recurrent Neural Networks seem more applicable and are supported by TensorFlow/Keras
https://www.kdnuggets.com/2020/07/rnn-deep-learning-sequential-data.html (https://www.kdnuggets.com/2020/07/rnn-deep-learning-sequential-data.html)
https://www.tensorflow.org/guide/keras/rnn (https://www.tensorflow.org/guide/keras/rnn)
I haven't gotten far enough to comment on RNNs