Shimano Di2 modulation

[Radiohead]

Hello All,

I've been busy tearing down a Shimano Di2 eletronic shifter handle to see if I could make my own controller for operating the shifter. Specificly I'm tearing down the E-tube version, so no can-bus interface. The data between the devices is AC coupled onto the DC powerlines that is connected to all other parts of the Di2 system. No transceiver IC or anything, just an dual diode to limit the AC amplitude and some AC coupling between the power input and the microcontroller with a 16MHz oscillator. The C4N part seems to be an 3.3V LDO.
PCB:

The main frequency is 1MHz and seems to use single period amplitude modulation. Every millisecond the message is repeated and sometimes seems to be acked. The same message is repeated over and over.

The reason I've posted this is that I don't recognize the modulation used. I've begun to wonder weither it's not just a waveform created by a DAC. Anyone has any idea what kind of modulation might be used?

[amspire]

Looks like Phase Shift Keying. The phase shifts 180 degrees to indicates the 1's or 0's. Some or all of the first six pulses may be there to establish the phase.

I have never used electric derailleurs, but looking at a teardown

, it looks like the rear derailleur has a high speed DC motor driving the output shaft through a lot of gearing. So it is not a stepping motor, and there is probably no DAC and linear motion sensor used. There is a magnet involved, so the motor or a geared shaft revolutions may be counted via the magnet and a hall effect sensor. The code may be transmitting a revolution count based on some calibration point. Does the Di2 do any kind of initialisation when it starts such as going to the 1st or 12th gear?

Richard

[amspire]

Your best bet may be to look at the Shimano Patents. You can track down the patent numbers here: https://patents.justia.com/patent/10773773.

This site does not have the drawings, but with the numbers, you can probably get the patents from Google.

[Radiohead]

Thanks amspire/Richard!

I had interpreted the change in the signal as some kind of amplitude modulation and not a phase shift, which explains why it didn't match any amplitude modulation I could find. I had compared the scope plots with different modulation waveforms found on the web, but all the drawings of course show no signal aberrations as the do in reality.

Do you have any suggestion what I could use to decode/encode the PSK messages? I have an Analog Discovery 2, but there might be something better suited for this task.

[CoteRotie]

I'm a Campy only guy, so no help here 

[bson]

I have two bikes with R8000 Di2, but that's CANbus, and on those versions the derailleurs are smart.  The shifters only send up/down commands, and the derailleurs go do it, then transmit the position.  The position is picked up by the wireless unit, which sends it to the computer head unit (Garmin Edge 530 or Wahoo Elemnt in my case) which updates its notion of which gear is used.  The gear is in terms of 0-(n-1) which is displayed as 1-n.  Not sure how full synchro shifting is done.  There is no calibration, except Wahoo head units can lose track of which gear you're in - shifting to the extremes will recalibrate it, while the Garmin Edge never loses track even if moved between bikes.  So my guess is the Garmins pick up absolute gear indexes, while the Wahoo picks up on the up/down shifts.  This is just my WAG based on observing how it works along with some very scant, not very technical information my brain has collected.

[amspire]

Do you have any suggestion what I could use to decode/encode the PSK messages? I have an Analog Discovery 2, but there might be something better suited for this task.

bson is right. Apparently Shimano has been using CANbus since 2009 for Di2 and since I think Di2 was released in 2009, that means all Di2 should be CANBus. The best bet is probably to get a CAN compatible interface chip or microcontroller.

It would be good news. The amount of data needed to send a shift position would only need to be 8 bits, but since it is a premier product, Shimano may go over the top and use 16 bits. It could just be sending the up/down shift commands, but I think I have seen a Shimano Di2 app that allows you to tweak the exact position of each shift. Is this correct? The data packet looks much longer then 16 bits, so I was thinking there is probably some error detecting bits as well that would be hard to work out. CRC bits are a standard part of the CAN protocol, so it solves that issue.

Richard.

[KE5FX]

Crazy that derailleurs are still a thing in 2020.  Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

[amspire]

Looks like the hard work may have been done at least for the ANT+ communications: https://hackaday.com/2019/03/26/reverse-engineering-shimano-bike-electronics/

[amspire]

Crazy that derailleurs are still a thing in 2020.  Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

If you want cranks, you basically need gears or derailleurs. Derailleurs and cassettes can be very cheap and they work very well.

If you want to make a fully electric bike with no pedalling, then you start to need a lot of batteries for a decent range and you will probably have to license the bike as a motor vehicle. At the moment, to get regeneration, you usually need the motor built into the hub of the rear wheel and the current rear wheel motors with regeneration usually do not have enough torque to get an average person up a 20% slope. They also will not have the rating to fully recover the energy to the battery pack when you are going down the same 20% slope. Putting a lot of weight into the rear wheel will affect handling, and to get regeneration with a center mounted motor, you have to eliminate the rear freewheel hub. This will be less efficient when coasting.

I do agree that at some point when we have motor assist with regeneration that actually works well and with only a few kilograms of extra weight, then ebikes will probably become the norm. I think it still needs a lot of new technology to get there.

[Someone]

Crazy that derailleurs are still a thing in 2020.  Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

Tough to beat gears when it comes to efficiency.
https://www.cyclingabout.com/drivetrain-efficiency-difference-speed-between-1x-2x/
Motor-generaotrs have relatively narrow bands of efficient operation, and aren't going to be competitive in any of cost/weight/performance.

[KE5FX]

Crazy that derailleurs are still a thing in 2020.  Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

Tough to beat gears when it comes to efficiency.
https://www.cyclingabout.com/drivetrain-efficiency-difference-speed-between-1x-2x/
Motor-generaotrs have relatively narrow bands of efficient operation, and aren't going to be competitive in any of cost/weight/performance.

But chain-driven derailleurs are mechanically-annoying objects that require constant maintenance and really only work well the day they come back from the bike shop.  Meanwhile, PM motor/generator efficiency can exceed 90%.  (Yes, they would both need to be geared appropriately, but those gearsets could be lightweight sealed units.)

Note that I'm not asking for batteries, I just don't see why the pedals have to turn a chain drive instead of a generator that drives a motor in the wheel hub.

[Jeroen3]

Crazy that derailleurs are still a thing in 2020.  Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

There are variable transmissions with belts on the market for bikes. It's all expensive stuff. Chains are cheap.

Above modulation doesn't like like CAN at all. So something is going on.

[Someone]

Crazy that derailleurs are still a thing in 2020.  Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

Tough to beat gears when it comes to efficiency.
https://www.cyclingabout.com/drivetrain-efficiency-difference-speed-between-1x-2x/
Motor-generaotrs have relatively narrow bands of efficient operation, and aren't going to be competitive in any of cost/weight/performance.

But chain-driven derailleurs are mechanically-annoying objects that require constant maintenance and really only work well the day they come back from the bike shop.  Meanwhile, PM motor/generator efficiency can exceed 90%.  (Yes, they would both need to be geared appropriately, but those gearsets could be lightweight sealed units.)

Note that I'm not asking for batteries, I just don't see why the pedals have to turn a chain drive instead of a generator that drives a motor in the wheel hub.

You're doing it wrong if derailleurs aren't working smoothly, sounds like the common complaint from people who have only ever used toy-store/"disposable" quality bikes and haven't ridden something built to last. When do I do maintenance on chain drives? When the chain has stretched and gets replaced, the rest of the time it just works. Want something even more robust? Internal geared hubs and belt drives are available (with a weight penalty).

The weight and cost kills an electric transmission, before any issues of efficiency. Efficiency which is not going to be peaking at 90% (citation needed!) over anything more than a tiny part of the operating envelope even for the best technology available (which won't be the lightest). The weight for a chain drive is:
https://weightweenies.starbike.com/listings.php
Cassette 200g
Chainrings 150g
Chain 300g
Front Derailleur 100g
Rear Derailleur 200g
Shifters 250g

So the parts you can strip off total 1200g, for a transmission that carries peak powers up past 2000W and has very little flex or slip.
Swap out to a geared hub and its under 3000g. There is a reason beyond safety that most power steering still has a mechanical linkage between the driver and the road, substituting for a slipping/delayed drive (just as a loose chain would do) kills the connection and feel.

P.S. just the hub motor for a typical electric bicycle is 3kg, 3000g, before considering a generator, power electronics, and wiring.

Its that sort of thinking that ends up with toy bicycles that weigh 15-20kg but look nice and have all the marketing "check boxes".

[KE5FX]

   Sure, this wouldn't result in any sort of racing bike, but it would certainly yield a maintenance-free commuter bike.

In my book, if there's any surface with oil and grease that's exposed to the environment, it's a maintenance hassle.  I'd be OK if my bike weighed 20% more but never needed maintenance beyond tires, and was always in the right 'gear.'

Anyway, back to the original topic, which is making a temperamental mechanical kludge even better by adding what amounts to consumer IoT technology.

[Someone]

  Sure, this wouldn't result in any sort of racing bike, but it would certainly yield a maintenance-free commuter bike.

It doesn't meet the requirements of any cycling market, other than your hypothetical "cool tech" in an inappropriate application. Fails in all departments:
Price,
Weight,
Performance,
Usability,

...with no upside compared to existing options. Even electric bicycles that already have more than half the system in place don't try to put a generator between the pedals.

In my book, if there's any surface with oil and grease that's exposed to the environment, it's a maintenance hassle.  I'd be OK if my bike weighed 20% more but never needed maintenance beyond tires, and was always in the right 'gear.'

They exist, IKEA made an attempt at it, and there are a range of utility/commuter bicycles with internal hub gears that are effectively zero customer maintenance. The Netherlands and surrounding countries are full of those everyday bicycles. "Automatic" gearing has been tried several times with little customer acceptance.

Electric drivetrain does nothing to address that market, yet electric assist is growing rapidly.

Anyway, back to the original topic, which is making a temperamental mechanical kludge even better by adding what amounts to consumer IoT technology.

c.f. "an exquisitely optimised system refined over a century of continuous use"

[amspire]

Above modulation doesn't like like CAN at all. So something is going on.

I think it probably is CAN compatible if the amplitude is 5V p-p rather then 0.5V p-p.

CANBus is differential and only one side was captured, so the differential signalling looks a bit like a phase reversal. It looks like a sine wave, but if the CANBus sync causes a sampling at the sine wave peaks, then it would work fine.

[Jeroen3]

CAN bus has bit stuffing after 5 highs. I count 16 highs.

[Radiohead]

I can find a lot of references which say that the Di2 uses a CAN bus system, but nowhere is it supported by any information.

Do you have any suggestion what I could use to decode/encode the PSK messages? I have an Analog Discovery 2, but there might be something better suited for this task.

bson is right. Apparently Shimano has been using CANbus since 2009 for Di2 and since I think Di2 was released in 2009, that means all Di2 should be CANBus. The best bet is probably to get a CAN compatible interface chip or microcontroller.

It would be good news. The amount of data needed to send a shift position would only need to be 8 bits, but since it is a premier product, Shimano may go over the top and use 16 bits. It could just be sending the up/down shift commands, but I think I have seen a Shimano Di2 app that allows you to tweak the exact position of each shift. Is this correct? The data packet looks much longer then 16 bits, so I was thinking there is probably some error detecting bits as well that would be hard to work out. CRC bits are a standard part of the CAN protocol, so it solves that issue.

Richard.

I did find the appnote slla336 of TI which describes using a CAN bus with power-over-data. However, I do not find any differential CAN bus signals on the PCB, nor does it seem there's any data on the negative supply wire. Only on the positive supply input is there AC decoupling and a diode voltage clamp with the dual package diode. If there was differential signalling, I would expect the same kind of input protection on the negative input. The capacitors that are of the same size as the one next to the dual package diode are all connected to the 3.3V or 1.8V supplies. Only the diode under the mounting holes functionality is unclear. Might just be some reset pin circuitry. There's no communication signal on it and on both pads I measure 3.3V.

Above modulation doesn't like like CAN at all. So something is going on.

   
I guess it's a bit of both. according: https://www.bicycles.net.au/forums/viewtopic.php?t=84327&start=36

The Di2 system uses a variation of CAN bus with a physical layer that can encode the data onto the power wires. Every connected node on the bus sees every message, but they ignore the messages they are not interested. There is no splitting of the signals at the lower junction. The lower junction is just a dumb junction that connects all the cables coming into it together. The FD sees messages meant for the RD, and the RD sees messages meant for the FD. They just ignore the ones that make no difference to them.

So the shifters are quite smart, and it is a lot more complex then just using a single pole switch or similar.

There are some single wire CAN bus IC's like the MC33897 for example, but I would be surprised to find that in a micro controller as a peripheral.

Above modulation doesn't like like CAN at all. So something is going on.

I think it probably is CAN compatible if the amplitude is 5V p-p rather then 0.5V p-p.

CANBus is differential and only one side was captured, so the differential signalling looks a bit like a phase reversal. It looks like a sine wave, but if the CANBus sync causes a sampling at the sine wave peaks, then it would work fine.

My probe and oscilloscope are both on x1.0, so the amplitude is not 5V. The TI app note also discusses the problem with AC coupled CAN bus, where due to the AC decoupling the differential amplitude is not large enough for the receiver to reach the thresholds. In order to fix this they propose a solution with two CAN bus transceivers which work together in order to transmit the same data and are connected invertedly on the CAN bus in order to create the necessary signal amplitude.

Thanks for all your replies so far!

[bson]

Above modulation doesn't like like CAN at all. So something is going on.

Yeah, that's exactly what I was thinking...  Definitely looks like PSK.  But it could be the shifters just pulse at some rate and don't really talk CANbus at all.

It does seem like early Dura-Ace Di2 was maybe 4-wire?

[amspire]

Above modulation doesn't like like CAN at all. So something is going on.

Yeah, that's exactly what I was thinking...  Definitely looks like PSK.  But it could be the shifters just pulse at some rate and don't really talk CANbus at all.

It does seem like early Dura-Ace Di2 was maybe 4-wire?

The initial Di2 was 5 wire and I think they then went to 4 wire. These had power and data separated The newest Di2 now have 2 wire with the wires carrying both power and data simultaneously.

Apparently they have taken CANBus but have used their own communications protocols. Seems like they have also modified CAN voltage levels and waveforms as well.

Shimano has done a really terrible job keeping compatibility even between its own Di2 models. The derailleurs are basically very simple devices, and it would have been pretty trivial at keeping compatibility, even if it meant cable converters to go from 5 to 2 wire. They also do not let you use an 11 speed Di2 derailleur with a 10 speed rear cassette, even though the 11 speed derailleur should be able to get the correct positions for all the 10 speed gears.

[bson]

Di2 moved the indexing from the shifters to the derailleurs, so yeah if you want to change your indexing you need to replace the rear derailleur.  And since the derailleurs talk they need to be mutually compatible and you need to replace the front to match the rear.  But you can keep your shifters.  This is different from mechanical where indexing was in the shifters and the derailleurs only had to implement a specific pull ratio and range of motion.

[Radiohead]

Hello all,

I've gotten a bit further, but in order to determine the Power Line Communication protocol I woud like to digitize the analog signal in an automated repeatable manner. The attached csv file was created with an oscilloscope.

The attached csv(changed the extension to .txt in order to upload on the forum) file shows the communication to the connection hub in order to read its firmware version with a command and two replies. I know the firmware version is 3.0.4, which is possibly transmitted as 0x3004 or a variation of. Both replies 1 and 2 are different, so there might be some sort of packet counter and with maybe a 2's complement 8-bit checksum. In different messages I got quite long periods of a continuous wave-form, so I think bit-stuffing might not be used or limited.

I've installed Phyton, but have only a limited experience with Matlab (and currently no license or active trial) and all most no experience with Phyton. Is there something, a Phyton library or module etc anyone can recommend to read the following csv file and digitize the following wave forms? I would like to create a binary/hexadecimal output from the wave forms according to different interpretations of what the different waveforms might represent bit-wise. Then it will be a matter of trying to interpret the binary data with some expected values.

Thanks in advance!

[qosch]

Hello Radiohead,
have you gotten any further than this? I've also taken (=dremeled) apart an XTR trigger shifter in hope they'd have used an IC that I could just buy to talk to the shifter but no luck. Concerning the modulation I am afraid I can't contribute anything useful. Since I only want to receive the button presses from the shifter I thought a brute force method could be to digitize the AC coupled signal on the other side and see if it matches a reference signal (assuming the same button will always trigger the same signal waveform).
Cheers

[Teknotronix]

Perhaps this transceiver would allow us to interface with the Di2 version of CAN?

https://yamar.com/product/dcan500/

Eval board:
https://yamar.com/product/dcan500-evb/