APDS9960 Pro - making better gesture recognition module for Arduino and others

[Georgy.Moshkin]

I am working on a new gesture recognition module for Arduino. My module provides two advantages over existing generic modules:
1) It measures speed and angle of hand motion (not limited to 45°!) very reliably, watch my two-minute-long video and see for yourself.
2) It has separate output pins for all gestures - single module easily can replace few mechanical buttons (only i2c in mini version).
3) USB Type-C port for direct PC connection - can used as HID device for computer games / music creation / no programming

UPDATE (November 18, 2023)
Starting to release an algorithm


UPDATE (August 9, 2023)
- Project is postponed. I am out of money for prototyping, maybe next time
- No luck with money prize from HICOOL either. Shortly after my pitch deck I discovered that my project has "failed" status.

UPDATE (June 20, 2023)
- Added 3.3v/5v GPIO and breadboard compatibility
- Entered 2023 HICOOL Global Startup Competition (plus two other modules)

UPDATE (June 18, 2023)
A good progress on compatibility with Arduino. Added USB-C connector. Bought cheapest foldable photo studio box (little box with LED strips) and rotating stand. Tight on money already. I have almost everything ready for payment processing and shipment. Just need to do few more iterations to make product good enough for shipping.

[Georgy.Moshkin]

Started to work on PCB design, here is preliminary pin arrangement:

UPDATE (July 11, 2023)
Correct pinout will be available in first message.

Basic mode pinout.
By default,  module will work in BASIC mode. This mode provides 12 separate outputs (10 gesture events + proximity event + fast motion event). It is especially convenient for novice Arduino users: choose pins for gestures you like and connect them to Arduino inputs.

A1 - Up
A2 - Down
A3 - Left
A4 - Right
A5 - Up-left (45°)
A6 - Up-right (45°)
A7 - Down-left (45°)
A8 - Down-right (45°)

B1 - 5v
B2 - Ground
B3 - Proximity detected
B4 - Fast motion detected
B5 - I2C_SDA
B6 - I2C_SCL
B7 - Clockwise rotation
B8 - Counter-clockwise rotation

Advanced mode pinout.
Module can be configured to work in ADVANCED mode, with different pin arrangement:

A1 - Up
A2 - Down
A3 - Left
A4 - Right
A5 - Clockwise rotation
A6 - Counter-clockwise rotation
A7 - Motion speed (bit 0)
A8 - Motion speed (bit 1)

B1 - 5v
B2 - Ground
B3 - Proximity (bit 0)
B4 - Proximity (bit 1)
B5 - I2C_SDA
B6 - I2C_SCL
B7 - TX
B8 - RX

In advanced mode,  45° gesture events are indicated by simultaneous activation of Up/Down/Left/Right pin pairs:
A1 & A3 - Up-left (45°)
A1 & A4 - Up-right (45°)
A2 & A3 - Down-left (45°)
A2 & A4 - Down-right (45°)


Idea is to make it usable even with simple GPIO logic using Arduino/Raspberry PI without using any libraries.
I would like to hear any suggestions, especially if you are planning on using such module.

[ebastler]

It's hard to understand the meaning of the various GPIOs and serial buses unless you tell us about the actual sensors you intend to use.  
Two general recommendations:

(a) Don't overdo it an try to be all things to everyone. You end up with a complex board, taking many I/Os, and it is unlikely that many users will want the exact same complement of sensors. Modular is better, I'd say, especially when serving a relatively small niche.

(b) Not sure the various dedicated gesture GPIOs are needed. This will be connected to a microcontroller, right? My preference would be to use SPI (or maybe I²C) only, and minimize the number of hardware connections and pins required.

[Georgy.Moshkin]

Sensors are apds9960+vl53l0x+additional color sensor. Idea is to make long range proximity effects possible when user approaches sensor from 2 meters (vl53l0x), an invite for gesture interactions and standard proximity stuff. GPIO outputs are for those who need simplistic outputs, like mechanical buttons. Thank for your comments, I am actually start thinking about covering advanved pins with some plastic cap to make it more friendly for beginners, just plug gpios to Arduino, no need to configure , no need for libraries. Plus advanced mode through i2c.

[Georgy.Moshkin]

Working on PCB layout of gesture processing module. Below is current layout with pins description.

Currently I use four 2.54mm pitched single row 8-pin headers for user connections and
four 1.27mm pitched 2x2 headers for inter-pcb connection between processing board and sensor board.
I am trying to keep module small, around 30x30x10mm. Rotational fingertip gesture is usually below 30mm in diameter. Also, it is possible to put several modules in a row or matrix.

Next steps:
- need to find a room for USB type C connector and linear regulator
- need to add level shifters for 5v compatible outputs
- start routing of sensor board. It will be three different sensors (apds9960+vl53l0x+...) or four sensors (3 x APDS9960 + single vl53l0x)
- order small batch of modules and try start to sell them
- start work on more detailed usage examples

[ebastler]

I am a bit lost. What MCU do you intend to use? Where are the sensors? What are the four 2*2 pin things in the corners?

Edit: Ah, hang on. This will be a sandwich of two boards, with off-pitch interconnect in the corners? Is that necessary? Can't you get away with a single PCB, with the MCU on the bottom and the sensors on top?

[tooki]

Why make a board with its own MCU and GPIOs? Why not just a small sensor board and a library so that people can integrate it into their own hardware?

I wouldn’t expect to make money on this. And that’s even if gesture interfaces were popular, which they’re not (for good reason).

[Georgy.Moshkin]

Current MCU is stm32h7 family in lqfp100 package. I've bought som stm32l4 and stm32g0 samples and breakout boards. Cheaper 2-3 times but less RAM and performance. I'll start with H7, more place for maneuver on firmware updates. Always can lower frequency or put it to sleep mode.

Yes, sandwich of two boards. All breakout modules I've bought for experimenting made single-board, and as you've said, components soldered on both sides. I never ordered such double-sided smt assembly. Maybe it is more expensive and routing will look good only if routed forb4 copper layers (now 2) . I just came up with a clever reason for two separate boards: theoretically, I can reuse MCU board in other modules, but not entirely. If I start making machine vsion module, need to reroute for DCMI/PSSI interfaces. Main deawback of sandwich arrangement is increased thickness of the module. The good thing is that current arrangement is cheaper to order, and I can do several prototype iterations for less money.

[Georgy.Moshkin]

Why make a board with its own MCU and GPIOs? Why not just a small sensor board and a library so that people can integrate it into their own hardware?

I wouldn’t expect to make money on this. And that’s even if gesture interfaces were popular, which they’re not (for good reason).

Those GPIOS are outputs. Separate pin for each gesture can simplify things for some users. No need to configure, no need to tune sensitivity. It can be used without programming at all. For people with programming skills I prepared usart, u2c and i2s i2c and spi interfaces. USB port for audible entertainment and as HID input device, e.g. for gaming controller.

Yes, this might be difficult. I have a list of projects. I'll try my best and then move to the next one.

[Georgy.Moshkin]

Why make a board with its own MCU and GPIOs? Why not just a small sensor board and a library so that people can integrate it into their own hardware?

EDIT: that's exactly what I am trying to make, a module that people can use in their own design.

I thought about your comment and now thinking about making stripped down i2c version with a small MCU. MCU is still important to include my algorithms. My goal is to make it pretty small, current full version is around 30 x 30 x 15mm. Stripped down version may be much smaller.

Here is current progress:


Idea is that sensor module may be used several ways.
1) 4-pin power+i2c (Qwiic/Stemma/Grove style), with user friendly communication protocol, can be used without library or with minimal
or
2) GPIO outputs (Mechanical button style); each gesture assigned to separate GPIO, some parameters (speed) may be assigned to GPIO, Easy to use with Arduino even without adding libraries.
or
3) USART port to read gestures using computer USB-to-RS232 adapter or some development board, like Arduino
or
4) USB port connected to computer as input device (HID device like keyboard, mouse, joystick) or as musical device (guitar pedal, gesture sound effect, DJ "scratch" effect, etc.)

Currently I think that USB port is more important than SPI.
Currently only two sensors VL53L0X+APDS9960. Main purpose of VL53L0X is to increase proximity range to make it more interactive-capable, and maybe more features for entertainment and as HID device.

Next steps: ordering prototype, adding mounting holes, making better demo videos and a website.

EDIT: I've re-read comment by ebastler, and it resonates with what tooki said. It seems that 32 pins is too much and maybe unnecessary?  I need to think on this more. Eliminating all those pins reduces module height by 8.5mm, makes module look more simplistic.  All those pins probably may scare some people away, but at the same time those pins allow to make very simple usage examples.  I think it's time to make physical prototype and try it out with various development boards. I have few stm32 boards, rp2040 pico and Arduino, and probably start with Arduino.

[tooki]

Why make a board with its own MCU and GPIOs? Why not just a small sensor board and a library so that people can integrate it into their own hardware?

EDIT: that's exactly what I am trying to make, a module that people can use in their own design.

Well sorta… if it’s got its own MCU, then it’s way beyond what I mean.

Put your algorithm in the library and distribute that.

If your concern is that you don’t want people using your algorithm on their own MCU because you want money for it… well, you could theoretically sell licenses for it. It doesn’t have to be open source. Though that would be the nicest thing.

If someone really likes your algorithm and then wants to integrate it into a commercial device, they’re not going to want modules, they’ll want code and specs.

[Georgy.Moshkin]

First of all, ebastler was right about using single PCB with MCU on the bottom and the sensors on top.
With current sandwich arrangement 1.27mm connectors between boards take 5.6mm. With two 1.6mm thick PCBs, total thickness is around 1.6 x 2 + 5.6 = 8.8 [mm]. Adding 2.54mm interface connectors gives total thickness of 8.8+8.5 = 17 [mm], plus few millimeters from components on sensor side, which is pretty thick (30 x 30 x 20 mm).

Thinner module provides user with more freedom. E.g., one can find a room for the module inside old laptop enclosure and connect it to USB pins on minipcie interface.

EDIT: tooki, I think I understand your idea about selling source code. In my opinion, it is not what most potential users of such product would benefit from. Selling source code adds a layer of unnecessary complexity: end user needs to write source code for optimal readout of sensor data, deal with performance and memory constraints of MCU they are using. Moreover, I do not see a way for reasonable pricing for such software library,  to cover all the time spent solving "it does not compile" type of problems.  I see much more potential in more straightforward plug-it-in solution, which can be used with or without programming. Moreover, I'll opensource all the examples for Arduino, stm32, Arduino and PC. What your noted very well is physical dimensions and feeling of complexity, it must look like a tiny sensor and not as some bulky development board.

Below is a photo of prototype PCBs I've ordered for testing. I'll solder them later and make a few videos.

[Georgy.Moshkin]

Started to write product datasheet:  current version
Added photos of first assembled prototype.
My current plan is to re-route for single board two-sided component placement, then make some better examples and demo videos. not anymore

[Georgy.Moshkin]

Current progress:
1. Working on new prototype, removed 16 pins, making it breadboard compatible
2. Added 5v output compatibility (3.3v / 5v switch)
3. Preparing first Arduino example smart light color/brightness control example
4. Learned kdenlive, a nice program for editing videos
5. Going to show this prototype during my pitch deck next week in attempt to get some funding (I have 2 more modules planned: 3d radar and machine vision camera for Arduino)

A small video update: moved to first message.

[Georgy.Moshkin]

Prototype v2 manufactured and assembled! Still need to make one more iteration before first batch production.
version 1 - green PCB
version 2 - black PCB with USB Type-C connector
Main changes: added USB port, proximity and speed are now analog pins. Added third sensor (revealed later).

[Georgy.Moshkin]

Why make a board with its own MCU and GPIOs? Why not just a small sensor board and a library so that people can integrate it into their own hardware?

I wouldn’t expect to make money on this. And that’s even if gesture interfaces were popular, which they’re not (for good reason).

So tooki was right. I haven't sold any. I had a tracker on a "buy now" button and there was only 10 clicks since summer. I am going to release current version of algorithm. I am thinking about creating newer smaller version with I2C-port only and better recognition.

[tooki]

Thanks for the update. While I don't celebrate it not working out for you as a business, I am not surprised. From the start it wasn't clear who the target audience for this is.

A frequent way products get invented is because a client has needed something, you make it for them, and you then realize you can sell it to a lot more customers. (Look at the history of the ICL7106 multimeter chip and of the Intel 4004 microprocessor, for great examples of this. Memory foam is another: invented for NASA, now used by millions of people for bedding.) That's because you're developing something you know there's a market for. In contrast, developing something based on hypothetical market demand is much riskier.

All the same, good luck with your endeavors!