Synchronized SPI communication between three FPGAs using VHDL

[eds]

Hello everyone,

I'm relatively new to working with FPGAs, and while I've managed projects without real-time communication needs, I'm now faced with a task that requires communication between three FPGAs. One of these FPGAs will act as the master, controlling the clock and other essential functions of the system. The other two FPGAs will essentially duplicate tasks simultaneously, allowing one to take over if the other fails.

Each FPGA has three connection lines available for communication. In my setup, I've designated one line for the clock, one for SDO (Serial Data Out), and the remaining one for SDI (Serial Data In), thus enabling full-duplex communication. To check whether I receive messages from both FPGAs, I've implemented a test pin that I monitor using an oscilloscope. However, I've noticed that there's a half-cycle phase difference between the test pins of the two FPGAs.

I'm seeking guidance on resolving this issue and ensuring proper synchronization between the FPGAs for reliable communication. Any help or suggestions would be greatly appreciated.

This is my controller code for both of the slave FPGAs

Code: [Select]

library IEEE;
use IEEE.STD_LOGIC_1164.ALL;

entity FPGAKONE_CONTROLLER is
    Port ( reset: in STD_LOGIC;
           CLK : in STD_LOGIC;       -- Clock line
           SDI : in STD_LOGIC;       -- Serial Data In
           SDO : out STD_LOGIC;      -- Serial Data Out
           SCK : out STD_LOGIC;       -- Clock output
shift_reg_in :  out STD_LOGIC_VECTOR(7 DOWNTO 0)
    );
end FPGAKONE_CONTROLLER;

architecture Behavioral of FPGAKONE_CONTROLLER is
    signal shift_reg_out : std_logic_vector(7 downto 0):= "10101010";
    --signal shift_reg_in : std_logic_vector(7 downto 0);  -- Input shift register
    signal bit_counter : integer range 0 to 15 := 0;       -- Bit counter
    type state_type is (IDLE, TRANSFER);
    signal state : state_type := IDLE;
   
begin

    -- Process for FSM
process(CLK)
begin
    if CLK'event and CLK = '1' then
             case state is
                when IDLE =>
                    if reset = '0' then
                        state <= TRANSFER;                           
                    else
                        shift_reg_in <= "00000000";
                        state <= IDLE;
                    end if;   
                when TRANSFER =>
                if bit_counter = 0 then
                    SDO <= shift_reg_out(7);
                    SCK <= '1';
                    shift_reg_in(7) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 1 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 2 then
                    SDO <= shift_reg_out(6);
                    SCK <= '1';
                    shift_reg_in(6) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 3 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 4 then
                    SDO <= shift_reg_out(5);
                    SCK <= '1';
                    shift_reg_in(5) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 5 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;                                               
                elsif bit_counter = 6 then
                    SDO <= shift_reg_out(4);
                    SCK <= '1';
                    shift_reg_in(4) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 7 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;                                               
                elsif bit_counter = 8 then
                    SDO <= shift_reg_out(3);
                    SCK <= '1';
                    shift_reg_in(3) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 9 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;                                               
                elsif bit_counter = 10 then
                    SDO <= shift_reg_out(2);
                    SCK <= '1';
                    shift_reg_in(2) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 11 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;                                               
                elsif bit_counter = 12 then
                    SDO <= shift_reg_out(1);
                    SCK <= '1';
                    shift_reg_in(1) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 13 then
                    SCK <= '0';
                    bit_counter <= bit_counter + 1;                                               
                elsif bit_counter = 14 then
                    SDO <= shift_reg_out(0);
                    SCK <= '1';
                    shift_reg_in(0) <= SDI;
                    bit_counter <= bit_counter + 1;
                elsif bit_counter = 15 then
                    SCK <= '0';
                    bit_counter <= 0;
                    state <= IDLE;                                                                   
                else
                    SDO <= '0';
                    bit_counter <= 0;
                    state <= IDLE;
                end if; 
      end case;     
    end if;
end process;

end Behavioral;
and this is my code for the slaves, which is also the same for both of the FPGAs:

Code: [Select]

library IEEE;
use IEEE.STD_LOGIC_1164.ALL;

entity FPGAH_SLAVE is
    Port ( reset: in STD_LOGIC;
           OSC_K : in STD_LOGIC;
SDI : in STD_LOGIC;       -- Serial Data In
           SDO : out STD_LOGIC;      -- Serial Data Out
           SCK : in STD_LOGIC;       -- Clock output
           shift_reg_in : out std_logic_vector(7 downto 0)
    );
end FPGAH_SLAVE;

architecture Behavioral of FPGAH_SLAVE is
    type state_type is (IDLE, TRANSFER);
    signal state : state_type := IDLE;
    signal shift_reg_out : std_logic_vector(7 downto 0):="01010101"; -- Output shift register
    signal bit_counter : integer range 0 to 7 := 0;       -- Bit counter

begin
    -- Process for FSM
    process(OSC_K)
    begin
        if rising_edge(OSC_K) then
            case state is
                when IDLE =>
                    if reset = '0' then
                        state <= TRANSFER;                           
                    else
                        shift_reg_in <= "00000000";
                        state <= IDLE;
                    end if;   
                when TRANSFER =>
      if SCK = '1' then
                    if bit_counter = 0 then
                        SDO <= shift_reg_out(7);
                        shift_reg_in(7) <= SDI;
                    elsif bit_counter = 1 then
                        SDO <= shift_reg_out(6);
                        shift_reg_in(6) <= SDI;
                    elsif bit_counter = 2 then
                        SDO <= shift_reg_out(5);
                        shift_reg_in(5) <= SDI;
                    elsif bit_counter = 3 then
                        SDO <= shift_reg_out(4);
                        shift_reg_in(4) <= SDI;
                    elsif bit_counter = 4 then
                        SDO <= shift_reg_out(3);
                        shift_reg_in(3) <= SDI;
                    elsif bit_counter = 5 then
                        SDO <= shift_reg_out(2);
                        shift_reg_in(2) <= SDI;
                    elsif bit_counter = 6 then
                        SDO <= shift_reg_out(1);
                        shift_reg_in(1) <= SDI;
                    elsif bit_counter = 7 then
                        SDO <= shift_reg_out(0);
                        shift_reg_in(0) <= SDI;

                    else
                        SDO <= '0';
                        state <= IDLE;
                        bit_counter <= 0;   
                    end if;
      else
bit_counter <= bit_counter + 1;
      end if;      
             end case;
        end if;
     end process;
 end Behavioral;   

[Daixiwen]

Are all FPGAs sharing the same clock (CLK, not SCK)?
Did you specify timing constraints for the SDI, SDO and SCK signals? You need to specify setup and hold constraints relative to the CLK clock, and if the frequency or the distance between the FPGAs i high, taking into account the propagation time on the PCB.

[pcprogrammer]

On a side note, how did you tie the SDO lines together or has the master separate inputs for each SDO line.

I don't see a setup in your outside world connection to make up an open drain output, and when having the signals tied together it might result in nasty collisions.

[eds]

No, all of the FPGAs have their own oscilllator clock but I use the same oscillator for all three which generates 12 MHz clock. I did not specify any timing constraints for the SDI, SDO and SCK signals. I don't know if the distance is considered high as one of the FPGAs signals are 1-2 cm away from the master and the other around 3-4 cms. I also did not tie the SDO lines together.

[ejeffrey]

It looks like SCK is only CLK/2.  This is not enough to get reliable transfer in asynchronous transfers like this.  Because data needs to be either shifted in or out on every edge of CLK, of there is any rate difference or phase drift you will eventually have to drop bits.

In principle since your clocks are derived from the same oscillator you can match timing, but I really don't recommend that.  In fact if you do that, the SCK signal is essentially superfluous.

There are basically two ways to do this. One is to directly clock the slave logic from SCK.  This is common in ASICs, since it ends up  doing a simple shift register but is not how it is normally done in an FPGA.  In an FPGA you normally would treat e a larger divider and keep SCK at or below CLK/4 or CLK/8.

[davep238]

Your assignments to SDO and SCK will create latches, since they are not assigned in every "then" selection. They are not assigned in "if reset..". Assign them a value prior to the "case" statement. The state should be reset to IDLE before the case statement. BTW, using "rising_edge(clk)", instead of "clk'event.." has been the preferred way of doing things for several years, although textbooks may still show "clk'event..".

if rising_edge(clk) then
  if reset = '1' then
     state <= IDLE;
     SDO <= '0';  -- whatever is appropriate
     SCK <= '0';  -- etc
  else
    -- default values:
    SDO <= '0';  -- whatever is appropriate
    SCK <= '0';  -- etc
    case
     when...
     :
    end case;
  end if;
end if;

[glenenglish]

don't forget, the SPI MODE matters alot to how hard the slave has to work the critical path to put data on the bus.

consider after issuing a slave register access command, say 8 bits slave address then 8 bits by the master , then the next 8 clocks on the bus will be the register output from the slave.

there is a critical path from slave address decoder and  register address decode to presenting the first data bit  - - you will have half a clock cycle to present the data and this is usually the limit to bus speed- lots of sequential logic in action.

also, as a slave, important to de-bounce the SCLK  input , use a shift register with say 3 bits and majority vote on the transistion 011, 100 etc as if you get 101, then you have a clock bounce. This applies to Asynchronous SPI interfaces which is what most of them are. I run my async SPI blocks FPGA fabric  at 4x to 8x  max serial bus speed so there is plenty of time to do good debounce

If the clocks with all the FPGAs that are talking SPI together are synchronous, you can buy a few favors from the hardware config.

WRT constraints. if it is an Asynchronous interface, you dont really need to constrain anything except specify the tool not to care about IO pin timings relative to internal clock
If synchronous, you MUST constrain IO delays to clock.

[Daixiwen]

Your assignments to SDO and SCK will create latches, since they are not assigned in every "then" selection.

They won't in this case because it is a clocked process. Not assigning them will just make them keep their current value during the next clock cycle. Latches are created if signals are not assigned in every case of a combinational process.
That said I agree that assigning default values before the state machine is a good practice, it makes the code easier to read because you don't have to wonder what the state of the outputs were on the previous cycle. So if you don't explicitly need to keep a signal level from one clock cycle to the other, use default assignments.
As Dave said it also good practice need to assign values to your signals at reset, so that you begin in a defined state.