Optical Bench REDUX: Digital Switching can have Analog Functions!

[RJSV]

   Making minor mods helps, the photo showing can use a tiny slice, taken off a solderless breadboard; that provides the LED Output with 4 pairs of pin-sockets right at place on housing that originally had it's LED.  That way, I've designated a 'Type M' gate, having output splits, usually two-way.  Meanwhile, a type M would also have at least one LED premounted to fire into the input (solar cell surface).  Eventually, all these MODs make for an IC with connectors interfacing, but of course, it's always been ELECTRO-optronics, so I can tolerate limited (wire) connections (some of which could be done with fiber optic cables.
The protoboard is remarkably easy to cut, using a old WOOD SAW...The row of 5 springy clips gets pulled up and out (the bottom) easily.  I've cut the little strips with having 3 lanes, for cases where LED resistor may be added in.

[RJSV]

This is my 'Chia PETs' moment...those little solar garden lights need feeding (sun) every day (or they go into SPAZZ mode.
   The LED output distributor helped packaging, by not requiring various twists and turns, creating a 'branch-like' or tree like split; and having one of the two (or more) inputs be a dedicated LED (wire pair input) helps keep the assemblies as 'in-line' and parallel housings.
   Photo shows, the one deviation, from direct 'in-line' placements, has the 2 gate pair as pointing slightly off-line, that way for placement in bright light, or SUN direct, to charge.
The 2/3 size 'AAA' batteries are limited; to 100 mA hrs.

[RJSV]

   I was going to take a sec, for tribute to the new Space Telescope...AWESOME!
   Forgot it was '6-sided' (mirrors).  But, (Pls see photo), the 5-sided variation does not lead to a flat assembly; the pentagon shapes, assembled, make for a curved upwards set of sides...plus, the top piece center of weight is NOT directly above, the center of the bottom piece.  The Chemistry / Chrystalography folks know about these things.
The Space Telescope, with 6-sided pieces, can be assembled into bigger shapes, flat, without being forced into a 3-D solid shape, like happens with 5-sided components.
   Notice, in my photo of 5-sided assembly, that a 6th piece can be added, to be a center, plus that gets things into 3-D territory, and looks nice.  Other up/down variations can be done, further out from center.

[RJSV]

   Adding gates, into existing indents, you can add another 5 gates, (notice how the original pentagon shape gets repeated, at larger scale).  That addition makes for 11 total gates; 1+5+5 = 11.
   Also, though, these 5-sided shapes will never go together, to solid plane, as there are always gaps.

[RJSV]

Placing yet another 5 gates, you get yet another pentagon, almost back where started; that's now
at 1+5+5+5= 16.  Etc etc, but this will never produce a gap free plane, like a 6-sided configuration would.
   Maybe NASA could fly this 'telescope' design...reminds me of the tiny satellites, as these could rely on solar energy, individually, (although not vacume-ready for the batteries, etc.).

[RJSV]

    For a two-stage flipflop, having input f/f separate from main f/f, that way any feedback actions are not actuated, until the COPY process completes, so any feedback won't cause a flakey response that might affect the input set/reset (until f/f has fully switched).
 
   Picture shows, this is getting into the shape needed, including that oval shaped clear plastic wrap, having 4 sets of pairs, in series order.  Notice, some bias 'lean' for capture of Sunlight, during recharge, while protected enough from nighttime moisture, if left outside.  This way, can avoid major dismantling efforts at end of each day. 
   This is 2 lanes wide, by 4 gates deep, and structured generally as 'SET' and 'RESET' corridors, meaning that usually can be activated, using a hand held flashlight, shining on either input.
   (Please also see schematic, posted next).  That schematic showing the 2-stage copy process, that also requires a 2-phase clock of separate phases acting to copy from input f/f to main f/f.

[RJSV]

   Here is approximate schematic, for that 'oval' or
 2-lanes clear enclosure.

[RJSV]

This view shows, the flipflop assembly SET and RESET output LEDs.

[RJSV]

ebastler, here is some more response, to your question(s), about Patent, and about motivation, for the optical gate organizing.

   You are exactly right, it was 18 months after initial filing and the examiner had done a few minutes 'skim' reading, to determine 'class and suitability, for publish the application.  I had spent, virtually 3 YEARS, up to the original mail-in date...September 9th, 2001.
   But with a couple more days prep, and it was September 11th, (2001), the morning of the attacks, on New York City.  I almost got that (big) mail packet, jumbo sized Application document (some 436 pages),
almost would have been mailed, like Sept. 10th...
   Big mess ensued, luckily I held off filing, until the whole scare with powder poisons in various government mail.  The application got published, on Jan. 11th, 2003, just like you said, at 18 months.
So, in my case, the prep time, and care, (some 88 drawings!), were huge, and that's not counting R&D time.  I'm fairly decent, at writing, and did all the spell-check and grammer, to the utmost quality. (Examiner can reject a crappy application; basically requesting a rewrite, by professional.). So I missed most of that, 911 stress / drama.  Actually, now that I'm reminding myself, the Examiner had made the comment about "Novelty likely present", on our brief phone consultation.
   My strategy, was to include literally everything, related to the 'Mechanical System', so to prevent some other party/industry player from filing, and thus frustrate (my) ability to operate (my own invention).
Kind of an amatuer lawyer dopey 'strategy'.  But you should hear my 'Lawyer Story', around that Patent.  Short story: My 'Patent Attorney' drank...er I mean, worked, from home...not so productive, always, I'm hearing lately...

   Now, back to motivations for this thread.  Stanford University's EE researcher is doing optical systems, that do, essentially a Fourier Transform, down thru layers, in a block of CMOS logic.  That way, the device can do some incredible parallel operations...some 1K by 1K block of pixels, for some 1 million operations, all with light beams as signal. Massively parallel, and plus that project illustrates the application right there.  Sorry, can't remember the Stanford Researcher's name.

[RJSV]

   Here is an example of using an 'inbetween' medium, useful to process some general signal, where the transmit section is essentially a discrete number line, and, in this example, a discrete receiver section takes the output value.  It's very similar to a look-up table.
   Kind of schetchy and general, but you can see, in diagram the yellow color highlighted area is a light diffracting element, prebuilt to perform division.  For better accuracy, an interpolation method can also be applied, to the 'receiver' array.
   The transmit and receive portions can be standardized (especially for what is considered full scale).  Of course you could get silly, and try putting a 'dead possum', into that medium processing space, but that comment really meant to indicate the processing method is all about what is placed there, such as a set of 'filters' that can perform AI 'Pooling', all in one big parallel step.
   For those kinds of processes, you might generally want the configuration to be 2-dimensional...where this example is a simpler 1-dimensional number line.
An example of 2-D filter could be a 2-axis SINC Filter, (useful in convolution)

[RJSV]

Working my way through various mechanical issues, right now playing 'herd Management', on that messy pile of gates (and internal AAA batteries).
Trying to weed out the flakes; you can see in picture, that 'Yellow #302' unit has been 'going spazz', (blinking or strobing), but yet battery checks out OK, at 1.26 V.
So today the strategy for tracking that, is to replace with another battery, (also starting out, in charged state at 1.26 V).  Let that sit in SUN and then go overnight, (for another 'spazz mode' trial...this time I will probably put the IC / PC board assembly on QC 'bad' status...(I have, approx 80 of these solar lights, so it helps to serialize for tracking).

   Eventually, as all this shakes out, I will have a (larger) set of 'gates', with many in sub-assembly forms that can be simply placed in sunlight, during the day, without disassembling (to excess).
Thanks.

[ebastler]

   Now, back to motivations for this thread.  Stanford University's EE researcher is doing optical systems, that do, essentially a Fourier Transform, down thru layers, in a block of CMOS logic.  That way, the device can do some incredible parallel operations...some 1K by 1K block of pixels, for some 1 million operations, all with light beams as signal. Massively parallel, and plus that project illustrates the application right there.  Sorry, can't remember the Stanford Researcher's name.

Hmm, not quite sure what you are describing there.

Multilayer, massively parallel FFT implementations have been done, of course, e.g. on FPGAs. To do an FFT on 2ⁿ points, you need n layers, each of them combining pairs of data via a "butterfly" operation. But I am only aware of purely electronic implementations, and don't see a benefit from using optical technology there.



On the other hand, the realization that optical imaging can be described by Fourier transformations has been around for a long time. (Joseph Goodman published a book about it in 1968, "Fourier Optics".) By simply imaging an object plane to infinity, with a single lens, you have a done a 2D Fourier transform -- massively parallel, no "multiple layers" neeed, just a single operation at light speed. I believe 1D arrangements have been implemented on CMOS chips, although probably as research projects only.



(Nice and simple illustration of a Fourier transform and then transforming back into real space. But they mislabelled it; should be "Fourier Plane" in the middle, not Focal Plane.)

In the latter case, using light obviously makes sense: The imaging operation does the whole FFT job. But you do not get a similar benefit in your concept, where light is just used to transport scalar analog or digital signals.

If you can find the Stanford reference, I'd be curious. Maybe it is a combination of the digital, multi-layer approach and an optical FFT "core"?

[RJSV]

   ebastler: I've been looking back, at my notes and previous videos, on that Stanford researcher who commented on using CMOS as a coincidence, because the material qualities were good.  I believe it was, maybe, at infrared wavelengths, a silicon 'disk' made good optical processing qualities, (not sure if it was diffraction grating structures, or whatever, sorry).
   So, by coincidence, CMOS manufacturing processes lent to good results...otherwise the researchers couldn't justify the 'millions' of dollars that have already been expended, for building CMOS electronics...
I believe it was a 'core', as you've surmised, put in between array of optical sending sources, and corresponding receiving sensors.  Their research in that aspect MIGHT have been less clear, if not for the fact that the various (micro) fabrication stuff has gone through such history, of progress.
  I'll keep trying various searches, but I did verify that it wasn't the 'CS-231' Computer Vision lectures, (although that in itself is fascinating).
Some of those video materials have also mentioned using Fourier transform, in some filters, as easier to code, vs full convolution.
   My Android smartphone doesn't maintain a full history, so not possible to go back, that 4 or 5 weeks, to that interesting material.
More persistence I'll find it...That kind of research helps frame the math, in a practical example, rather than simply stating that massively parallel computation is useful!

[RJSV]

   OK, found it! (Essentially).
   Found the reference, Stanford University E.E. Department doing research on (CMOS) Optical and Wave physics, as mass computational element, or 'core' as a flat disk.
The 10 inch diameter disk serves as a (X,Y) array doing implementation of Neural Networks, having more conventional electronics interfaces, in and out, but based on optical and wave physics in the (middle) 'core', which appears like a plastic disk, with clear surface.
   Still haven't found original video, but the following info is very likely the same Department:

   Stanford University 'Initiative for the Theoretical study of Neural Networks'  video, October 21, 2021.
Lecture presentation by Professor Shanhui Fan, of Ginzton Laboratory.  It's still very relevant to similar 'Deep Learning' methods being researched using FPGA logic, etc.
   The photo (video paused at approx 7:00 minute), shows the round silicon 'wafer' from foundry, that I'm assuming is the CMOS fab, mentioned here a few posts back.  They are calling that a fabrication of waveguide arrays.
Introduction by Professor Vnod Menon (sorry if mis-spelled the first name).
   Lecture starts by mention of energy footprint, of mass computation for Deep Learning networks, being problematic.
Sorry if cannot pin down the exact time (7:00), as YouTube format change doesn't give elapsed time, in video.  But, at any rate, got the right academic department there.  I've noticed that many universities today are CHOCKABLOCK FULL, of required social media, If you want any kind of reasonable communication...that means no published EMAIL contact(s), even....Ridiculous.
Thanks, - - Rick

[ebastler]

Thank you for digging that up! Here's the actual video you refer to, I think:


So they use a waveguide (interferometer) structure on a CMOS chip to perform matrix*vector multiplications. It's meant to be used in neural network (deep learning) systems, where one would typically have multiple stages of such multipliers on one chip.

The main selling point is that this is much more energy-efficient than an implementation in digital electronics. The fact that it's also faster, especially for larger matrices, is a nice extra benefit. 

By the way, there is also a renewed interest in analog computing, for much the same reasons: Fast due to fully parallel processing (more like an FPGA than a CPU), and using much less power at comparable computing speeds. Here's one startup company which aims to integrate analog computers on single chips, both for portable low-power uses and for high-power computing: https://anabrid.com/

[RJSV]

This 56 Gate monster (see photo), you could term as a crude pantomime of a neural network (visual processor).  It brings a general stack of gates into a form that can be re-charged in-place simply by placing the entire module in a sunny or bright spot.
   As configured, each stack of 7 gates has a sensor (layer 1), various inverters and flip/flops as 1 bit for each of the four corners.  Extended bit storage, to 12, is accomplished via shift register function.
   As in edge detection strategies, this 'cube' can be used to experiment.  For example, a moving light scanned across the top (layer 1) can cause an effect where 'old' status but was present, in older sample, but not now, while current sample (corner) signal is new. That gets to be quite a specific criteria!  (I think a simplified version, that sort of specific motion detection was one-of-seven variations, of conditions.

[RJSV]

This photo shows, the line-up from pixel sensing down, is organized in pairs, and further simplified by only doing half of the pixels, out of 16 array, (of 4 by 4).
Each pair is Logically OR'ed from 2 pixels, resulting in one bit per corner, in a 2 by 2 array.
After inverter (layer 2), and then Logical OR, (layer 3), that result is inverted, then used as a potential 'SET' signal, for Layer 5 flip/flop.
A first copy, at (t-1) time, is made, to layer 7 flip/flops.
With another layer (not shown), you get a current status, along with (t-1), and (t-2) histories,... at suggested variable rate, around 20 milliseconds per sample.
   (That's an 8-foot stack, of gates, if put all together!).

[RJSV]

This latest (photo) is the most advanced so far, as it simply allows for decent 'sunning', for charging up each (of 56) elements.
A helpful feature has the base come apart as two sections; each reinforced against weight of all those lawn lights. Plus I'm testing to assure that all those are getting full charge, each morning, enough to run normal at dusk.
   Free running, a signal quickly runs down a column, about 8 mSec.  This system is designed for synchronized operation, stage to stage.  Each pixel 'pair' is quickly combined, into a 'corner' signal, by Logical OR.  Then, this corner column has one latch, while adjacent column in the pair has another two bits latched, by synchronizing gates.  So all told this 'cube' module contains 8 pixel sensors, combined into 4, and has 3 stages latching those corner bits as older copies.  Very similar to bucket brigade logic, current state isn't latched until Q7 copies Q5, and Q5 copied current value, shift register clocked at 10 mSec per sample.  This info can then be interpreted for motion sensing, etc.

[RJSV]

This design has more open air, support columns reduced from 25 to just 5.  That wall of 28 gates
(4 by 7) contains the 4 pixels, for two corners, each of 3 bits storage.
  In front, you can see the partially stacked pixel pair for other corner in front.

[RJSV]

Here's an example, using various edge-detect methods, while keeping two 'history' copies.  The two copies, of the corner detect bits, are each timed at 10 mSec, so first copy at 10, and second copy is at 20 mSec. after current (4-bit) copy.
   An event, such as with a sweep of a flashlight,  can cause an 'ON' indication first on the left, then, moving rightward, a short pulse on the right side sensor.  Nothing this, just for the front two sensors, you have a bit sequence like; '00,10,01...' as that beam is swept rapidly across.
   While this perfect case seems practical, in use there would be, actually something like 4096 combinations, some that make sense, as to detecting direction of motion, many of the permutations or combinations of signal don't make sense, or represent a confused multiple sources.
   I'm figuring a simple, rapid hand movement in the range of 10 mSec, to cross a gate-sensor.

[RJSV]

Viewing that last diagram, representing stored samples of 1 bit for each corner (pixel pair).  That way you get into interesting territory, when discussing MOTION DETECTION mechanisms.  This set-up involves keeping a first and second (in time) copy, for the mainly 'edge-detect' type operations.
   So, with the 4 corners, the bit storage comes to 12 bits...; that's going to be 4096 different ways to express.  Labels are T0: (Current sample), T1 is (at 10 mSec. past), and T2 is (at 20 mSec.).

   Tried various light intensity weights, by simple distance from sensing cell: Consider T0 sample at
80 %, T1 at 40%, and, oldest, T2 at 20 % weight.
These being based on 100 % representing the internal comparator trigger (at PV cell=0.5 Volts).
You could have T0 and T1 'ON' and that would exceed the comparator threshold.  OR,... You have T0 'ON' (now), and T2 'was ON', for a total 100% this time, again exceeding threshold.
This way expresses how to approach using 'imperfect' or 'almost correct' data, for identifying simple motion directions.

[RJSV]

Thanks to ebastler (July 31) for digging up that video by Professor Shanhui Fan, of Oct 21, 2021.
(Ginzton Laboratory).
  Prof Fan talks a bit about edge detect mechanisms, which is part of the convolution and image recognition concepts for AI network.
My setup provides various copies, of single-bit status at each corner, or called 'Q1, Q2, 'etc as 'quarters' of the visual field. One point to make here is that, whether a full set of data points going back over (at least) several discrete time intervals.  Realize: those quarter portion corners simplify the data by logical 'OR'ing whatever pixels get active, so you are losing a bit of separation details, lumping all 4 pixel positions into one, per corner.
   But even that, simplified has a lot of permutations, which I have noted (see diagram showing 2 front corners and a single older copy that is 10 millisec older.). You could work through each possible state, of those 4 total bits, and 6 of them are clear to interpret.  Other combos not so much; consider the 'all bits ON' scenerio or the 'all off' variation, that make logical sense.  A couple of the variations are contradictory in interpretation so the whole set, of 16 possible combinations of present and past samples becomes burdensome.  This is the environment tackled by some 'deep AI', as trendy as that may sound (lol).
  In my musings here, I'm seeing some parallels with the 'Pooling' concepts, to reduce data to more 'symbolic' or 'stereotype', i.e
 identifying a mouth, nose or just a simple curve of some sort, in an image convolution.  Plus, here, with this 4 by 4 pixel array I'm encountering 'expansion' of data that needs to be analyzed due to inclusion of earlier bit copies.

[RJSV]

   In this picture, the columns lean a lot,...that's OK they will be secured as straight up.  You can see this 'first layer' (light sensing), having 8 gates, in capacity for 16 gates, if fully populated (a typical 4 by 4 array).
To reduce the data load, while experimenting, each pair of sensing elements gets 'ORed' into one pixel.  Spacing variations (non diagonal for this example), COULD be 1, 2, ,3, or 4 pixels apart...but choice is made to ultimately use just one separation variety, ending up with a smaller 2 by 2 array, (while experimenting with edge detect and with Pooling and Rectification.
That seems appropriate as this little 'network' has, also, some data bulk reduction, by crudely dropping or consolidating pixels (into 2 X 2, ultimately).
Then, similar (or, merely resembling) to neural nets, supplementing current (bit) states, by inclusion of 'older' copies, for edge detect; that dynamic causing expansion, of data needing to be analyzed.
   I think I have that correct (?)... That the neural system goes through expansions and contractions, along the paths from input to output, through the various layers.
   I'm wondering if in cases of contradictory inputs, if there is a mechanism for catagorizing in cases of 'Invalid input combination'.  One example of that, invalid case, could be with both front corners occupied (by user's flashlight beam), followed by NO pixels ON.  That does not indicate any kind of particular 'direction' resolved, while user sweeps the (top of array) with a flashlight.  Of course, there is, also, some valid results, but they just don't indicate a particular direction trend.

   By the way, the real-time action, of the digital portion, is to shift older samples over, (through 2 older copies), and then take current sample, of each of the corner 'Q' values.  When triggered, this continuous storage action, (every 10 milliSeconds seemed about right), the storage action will freeze or halt, allowing examination of each bit, ON our OFF.

[RJSV]

...AND, as appropriate for the base that opens up during component assembly, you can see; the right hand front pair (of sensing gates) can slide open and clear.  That's why that top plate is two pieces.

[RJSV]

   Picture here shows the top layer #1, (out of 7 layers).
I left a little space under that top layer, just for now to show the usual electrical interfacing is done in layers 2 to 7, by including an extra LED up top, actually piggy-back with the ordinary gate (#1) LED, so effectively the second gate receives light signal from both sources.
The way this works is for each rotary selector, user can dial in which source they wish; for a column (of sensor/gates).  While this only is the 4 'main' columns, of the 8, the other, 'Auxillary' column has other switches, for chaining both.  This way a user can dial in a 'CHAIN' Mode extending or encompassing ALL of the 56 gates.
   If you look at the top, see a wire pair protruding from the upright column; This brings voltage from the rotaries up, to cause the column top LED to be driven.
This electrical rotary selection is done for both LED lead wires, as many components are electrically separate; No ground or shared common (negative).

   One casual principal I like to employ, (especially when that's not a lot of extra work), is concept called:
   'MIB', which means 'Make It (your) Best', a simple marketing concept I've seen.  For that, I liked to spend a little extra time, cutting that plexiglass  top sheet, as neat and square as possible.  Heck, it's only a prototype, but...
  'Never underestimate the Marketing forces...they rule'.