+ Post New Thread
Results 1 to 15 of 15
  1. #1
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Multiple users of a DDR interface

    Hello,

    I have a DDR3 controller with a 128 bit wide data interface.

    Currently there're 5 users connected to using the following architecture:

    Each "user" has a 16 to 128 width converting FIFO. The application write data to the 16 bit side.
    Each of the 5 128 bit FIFO outputs is connected via a MUX to a custom arbiter I wrote.
    With every clock that the DDR3 controller is ready (wait signal inactive), the arbiter issues a select signal to the MUX, strobes a read signal to the appropriate FIFO and writes the data to a predetermined address range (that matches the FIFO number).

    The design works well at ~120 MHz but I doubt that it'll scale up well.
    I expect the 128 bit wide MUX to become a performance bottleneck as the number of users increases...

    How did you solve similar problems?

  2. #2
    Super Moderator
    Points: 27,821, Level: 40
    ads-ee's Avatar
    Join Date
    Sep 2013
    Location
    USA
    Posts
    6,355
    Helped
    1540 / 1540
    Points
    27,821
    Level
    40

    Re: Multiple users of a DDR interface

    Pipeline the multiplexer (and address/control) for maximum performance and maximum resource usage, peel off bits of the select and multiplex that subset of inputs and register the output, then multiplex the next set of inputs using the next set of select bits. Assuming a binary select and 6-input LUTs you should end up with two stages of 4-to-1 multiplexers for a 16-to-1 mux, with registers on the output of each stage.


    1 members found this post helpful.

  3. #3
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Re: Multiple users of a DDR interface

    This was my first thought a while ago...and it's probably what I'll do eventually...
    https://alteraforum.com/forum/showthread.php?t=54643

    But I'm asking about other approaches to this issue...
    Obviously anyone who tried to connect multiple users to a shared medium with a very wide bus - faced similar design challenges.
    So I'm thinking perhaps someone here implemented a different solution than the one I'm considering.



    •   Alt30th November 2017, 15:10

      advertising

        
       

  4. #4
    Advanced Member level 3
    Points: 6,882, Level: 19
    Achievements:
    7 years registered Created Blog entry
    dpaul's Avatar
    Join Date
    Jan 2008
    Location
    Germay
    Posts
    999
    Helped
    221 / 221
    Points
    6,882
    Level
    19
    Blog Entries
    1

    Re: Multiple users of a DDR interface

    I had come across a design in which multiple FIFOs were to write data to an external DDR3 memory (in your case it would be 5 FIFOs of 5 users ).
    The FIFO interface signals were AXIS.
    So we had used the AXI Virtual FIFO Controller IP core. I am talking about Xilinx based designs.

    You may read here a bit and get some idea : https://www.xilinx.com/products/inte...ontroller.html

    If you look at the spec, PG038 November 18, 2015, Pg. 8 shows the top-level block diagram, you will get an idea as to what is happening. You don't need any custom Arbiter or MUX design, everything is saleable.
    Last edited by dpaul; 30th November 2017 at 16:06.
    .....yes, I do this for fun!


    1 members found this post helpful.

  5. #5
    Advanced Member level 5
    Points: 35,071, Level: 45
    Achievements:
    7 years registered

    Join Date
    Jun 2010
    Posts
    6,424
    Helped
    1872 / 1872
    Points
    35,071
    Level
    45

    Re: Multiple users of a DDR interface

    You dont specify what interface you're using?
    The Xilinx memory controller offers an AXI4 slave interface, and there is an axi interconnect IP for multiple masters to a Memory slave. THe interconnect has several configuration options, like buffer sizes and packet fifos for buffering data. Different masters can have different dwidths and all transactions will be converted inside the interconnect.
    Make your life easy, use existing IP for existing interfaces.

    Good thing about AXI4 is that it's pretty simple to plug AXIS into it.


    1 members found this post helpful.

  6. #6
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Re: Multiple users of a DDR interface

    The Xilinx memory controller offers an AXI4 slave interface, and there is an axi interconnect IP for multiple masters to a Memory slave.
    I designed the custom arbiter for an Altera Cyclone V SOC implementation.
    At first, I tried to use the native QSYS interconnect but the timing performance and resource usage were VERY VERY POOR. Fmax was under 95MHz.
    I was surprised at first - but our ALTERA FAE confirmed that my Qsys design was optimal...

    This led me to try and design something of my own. The Fmax I got with the architecture I described is 124MHz with 1/5 the logic usage of the Qsys equivalent!

    Now, I'm working on a new design that will be implemented on a much bigger Xilinx Kintex Ultrascale FPGA.
    So I'm thinking whether or not to reuse my custom arbiter or to give Xilinx interconnect a try...



    •   Alt30th November 2017, 22:13

      advertising

        
       

  7. #7
    Advanced Member level 5
    Points: 35,071, Level: 45
    Achievements:
    7 years registered

    Join Date
    Jun 2010
    Posts
    6,424
    Helped
    1872 / 1872
    Points
    35,071
    Level
    45

    Re: Multiple users of a DDR interface

    Give the xilinx stuff a go. We use it at 200 MHz.


    1 members found this post helpful.

    •   Alt30th November 2017, 23:12

      advertising

        
       

  8. #8
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Re: Multiple users of a DDR interface

    Give the xilinx stuff a go. We use it at 200 MHz.
    On what device?



  9. #9
    Advanced Member level 5
    Points: 35,071, Level: 45
    Achievements:
    7 years registered

    Join Date
    Jun 2010
    Posts
    6,424
    Helped
    1872 / 1872
    Points
    35,071
    Level
    45

    Re: Multiple users of a DDR interface

    Ultrascale.
    One thing to bare in mind that Xilinx stuff is all AXI - I dont know what your custom IP speaks.



  10. #10
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Re: Multiple users of a DDR interface

    I dont know what your custom IP speaks.
    It speaks Avalon which is almost the same...

    Do you know what arbitration scheme the AXI interconnect implements in case of multiple users?
    I know that Qsys uses weighted round robin...somewhat inefficient.



  11. #11
    Advanced Member level 5
    Points: 35,071, Level: 45
    Achievements:
    7 years registered

    Join Date
    Jun 2010
    Posts
    6,424
    Helped
    1872 / 1872
    Points
    35,071
    Level
    45

    Re: Multiple users of a DDR interface

    Almost the same, but you need to ensure that you set readyLatency to 0 to work with AXI
    The Xilinx AXI Interconnect can be configured to work with either round robin or weighted round robin. But it is fully pipelined (if you configure it that way).

    - - - Updated - - -

    Read the interconnect documentation:
    https://www.xilinx.com/support/docum...terconnect.pdf


    1 members found this post helpful.

  12. #12
    Advanced Member level 3
    Points: 4,845, Level: 16

    Join Date
    Feb 2015
    Posts
    808
    Helped
    235 / 235
    Points
    4,845
    Level
    16

    Re: Multiple users of a DDR interface

    Arbitration scheme and clock rate -- does this matter that much? You should design any algorithms to avoid small external memory accesses. If short accesses are needed, you should include a caching layer similar to the L1/L2 cache from modern CPUs. External memory interfaces like DDR* favor kilobyte sized accesses when possible.



    •   Alt1st December 2017, 12:13

      advertising

        
       

  13. #13
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Re: Multiple users of a DDR interface

    Arbitration scheme and clock rate -- does this matter that much?
    The Arbitration algorithm does start to matter when you approach the memory bandwidth limits...



  14. #14
    Advanced Member level 5
    Points: 35,071, Level: 45
    Achievements:
    7 years registered

    Join Date
    Jun 2010
    Posts
    6,424
    Helped
    1872 / 1872
    Points
    35,071
    Level
    45

    Re: Multiple users of a DDR interface

    Quote Originally Posted by shaiko View Post
    The Arbitration algorithm does start to matter when you approach the memory bandwidth limits...
    But why are you approaching the limits? is it because of small transfers? lots of row changes? Much easier to get the access sequence fixed (if possible) to save bandwidth that the complicated arbitration.



  15. #15
    Advanced Member level 5
    Points: 12,132, Level: 26

    Join Date
    Aug 2011
    Posts
    2,467
    Helped
    285 / 285
    Points
    12,132
    Level
    26

    Re: Multiple users of a DDR interface

    But why are you approaching the limits? is it because of small transfers? lots of row changes?
    No. I use the memory very efficiently...long bursts and as much spatial locality as possible.
    With 5 users, I consume ~40% of the bandwidth - but as the design grows, more users will be added, so I'd like to work as efficiently as possible...



--[[ ]]--