Author Topic: Open bus architecture for instrumentation/control (VME, VXI). Feedback wanted  (Read 967 times)

0 Members and 1 Guest are viewing this topic.

Offline msat

  • Contributor
  • Posts: 12
  • Country: us
Hello, everyone! Occasional lurker, but first time poster.

There's been something that's been in the back of my mind for several months that occasionally springs to the front, which I've been gestating a bit more as of late. This has come about after finding myself creating bespoke circuits for instrumenting and controlling experiments. Sometimes I have to add additional capabilities which becomes somewhat unwieldy, or requires redesigns and/or firmware hackery. Worse still, these are often not reusable for other projects. Sure, it can in part be blamed by lack of foresight or good practices/discipline. This is partly the goal of this project. Yes, VME, VXI, etc already exists but there's reasons I don't want to go that route which I'll elaborate more on later. Currently, this project is nothing more than a collection of thoughts, and may very well stay that way, but in any case, I'd like to get feedback on the design choices that I'm currently considering.

First things first: On my part, I intend for this to be completely open source. I have no intention of making money off this, but really just want to benefit from people creating open source modules that are useful to me.


So, it shouldn't take much imagination to get a visual idea of what this project is. Either it'll consist of stackable modules with board-to-board connectors, or modules with something like card edge connectors that slide in a rack and connect to a backplane (though it certainly could be both, though I'm leaning towards the latter for tidiness reasons).

There are 3 items that are the core of the system which I see every unit containing:
1) A global voltage reference module. This provides a Vref for all additional modules that require it. Since this is a module, it can be as fancy or cheap as you need.
2) Global clock ref module for sync and time keeping/stamping
3) Bus to USB bridge module, likely by was of the Cypress FX3 USB 3.0 peripheral controller.

That last one requires the most elaboration because it's in a way the most critical component, and inherently differentiates this from the other bus architectures. The FX3 offers quite a bit of flexibility in terms of interfacing via a customizable parallel bus, good bandwidth, some inter bus-to-usb processing using the 200MHz ARM9 core, as well as SPI and I2C controllers, which are all things that make it amenable to low cost and DIY-able rack modules. I see a lot more drawbacks than benefits in creating a bus that is designed to support computing modules, while the FX3 potentially allows for much more flexibility and future proofing by giving someone the ability to interface with the computer of their choice. No, it can't compete with PCI-Express in terms of bandwidth and latency, but it's not meant to. This is really targeting the home/small/school lab. It's just one of those engineering trade-offs and I have no illusions of it being a one size fits all type of thing.

Current thoughts on a parallel bus design: The FX3 allows for an up to 100MHz 8/16/32-bit wide bus with configurable control lines, which I don't think is intended to be reconfigured in real-time (but possibly might be). In terms of bandwidth, 32-bit would be the most desirable, but I personally think chews up too many IO pins of a uC each module is likely to contain. 16-bit is probably a happy medium while simpler modules could use SPI/I2C. The FX3 parallel interface can be driven from an external clock up to 100MHz and best I can tell, no lower limit. The idea here is to let each module control the clock to set the bus speed so that even relatively slow microcontrollers can be used. That's not great for maximum bus throughput, but the ability to use potentially simple low-cost modules is more important, IMO. Again, this isn't mean to compete with VXI in terms of performance, but should still be plenty fast for many uses.

PC-side drivers: The FX3 has 15 in + 15 out usable USB endpoints. For a rack that probably would support ~8 modules, that would allow at least one in+out endpoint per module. While that could certainly be handy, I'm personally leaning towards it enumerating as a mass storage device and interfacing taking place via virtual files (inspired by other projects that used this method). The main reason for this is that it allows for the use of OS-agnostic driver libraries and the benefits that brings. That said, since this will be fully open source (and hopefully everyone's custom modules as well), endpoints can be enumerated however a person wants.

Since the current vision is that all communication takes place only between a module and the FX3, I don't think it's necessary that trace lengths between each slot and FX3 be the same, especially since each module will drive the bus clock. However, I do intend that the trace lengths match for the global ref sync clock.

Originally I was thinking about additional bus lines to be used for inter-module communication but I've since shied away from it. That requires a bit of forethought that I really don't want to spend time on, especially since I'm having a hard time seeing the value in it, and I definitely don't want to try to accommodate every edge case (which is hopeless). Best I can do is either spec a card edge connector with plenty of extra pins for potential future use, or place the used card edge in a location where another one can be added down the road.

Ultimately, I would like to see people not only developing a sharing their module designs, but potentially seeing them being picked up and produced from places like China, similar to what has been done with other useful gadgets (the LCR and component testers come to mind).

I don't want this to eat up too much of my time. I'm hoping that my initial investment offsets the time I'd spend down the road making custom and non-reusable circuits for my experiments. Though I don't want this to end up being completely hampered by poor design choices either, which is why I'm seeking feedback. There's probably a handful of details I've considered that I forgot to mention, but I think I outlined the bulk of them. Any illumination of major flaws would be appreciated.

-Mark
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 16066
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Nice thing about standards, there's so many of them...

What would you really be using this for?

Also, why parallel?  Peripherals are in ever shorter supply, and it's such a pain to breadboard a half a hundred connections (or if we're talking 32 bits here, maybe hundreds).

Assuming this gets advertised enough for people to be aware of it -- what would they find in it, that isn't offered by, say:
- PCI: thousands of expansion cards to choose from (some still being made..?); require drivers (which are mostly x86/Windows??)
- ISA, in slot or PC104 form: still standard in industry, I think?  Low bandwidth, may require various compatibility strategies (setting port/memory windows, drivers, etc.).
- USB: just toss a bunch of hubs on it, she'll be right.  Plug-and-play, may need drivers but also many things can just be used as character-mode devices, and structures can be built on top of that (e.g. mounting ttyusbN via some filesystem driver in Linux must be a thing?).
- PCIe: like USB, it's serial and it's fast, just not (usually) cable-oriented.  Tons of peripherals and expansion cards, more bandwidth than you can shake a lane at; plug-and-play; drivers compatible with current computers; etc.
- VME and all those: unless you're supporting ancient cards or doing vintage computing sorts of things, I don't get it, and again don't see how you're differentiating from it..?  (You mentioned VME once but didn't refer back to it later; not sure if we're supposed to understand VME and draw that conclusion ourselves?)
- Or also from the vintage era, GPIB: slow, moderately addressable, cable-oriented.
- Or if bandwidth really isn't much, then any of the embedded serials buses, like I2C, or SPI with multi-master addressing and arbitration, or RS-485 with arbitration.
- Or Ethernet for a similar experience (lots of serial point-to-point links) but way more bandwidth, and again, standard hardware.  VREFs can't be shared, but time is regularly synchronized over it!
- Or for an emphasis on analog, whatever Eurorack synths use..?  Maybe that's just power, I never looked it up...

I guess what you're really after, is some manner of metrology playground, given the emphasis on VREF and CLK?  That already puts it into a very narrow space, and like, if all you're after is some community adoption of something, I guess that could work?  But I don't see much reason for industry to consider it.

Note that most of these buses include a clock already, which may not be all that high quality in the original application, but there's no reason you can't make it so.  You could take a 10MHz disciplined ref and PLL it to the 8MHz of ISA, or 33 or 66MHz of PCI, nicely buffered at each card connector; or just abuse it for your own purposes and dump 10MHz through the pin regardless of the standard.  Likewise, the -5 or -12V supplies (which at a glance, VME has as well?) could be used for REF, equipped with modest current capacity to remain compliant, or who cares just make it a modest -5 or -10V ref and don't use any cards that draw much current from it (which is most of them, AFAIK?).

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline Doctorandus_P

  • Super Contributor
  • ***
  • Posts: 1222
  • Country: nl
This is also a long standing idea of mine, but it never got much further then that.
Join this thread:
https://www.eevblog.com/forum/projects/diy-instrumentation-bus-(or-dib)/

I think it's mostly fizzled out over the years, but you may find it interesting to read.

Here's a list:
http://www.interfacebus.com/Interface_BackPlane_Buses.html

There are also a few open source bus based systems you can re-use. Some time ago I even found an open source project which made it into a real-life product and you can buy chassis with power supply and several plugin boards commercially for reasonable prices.

=============
My latest idea is to use a bus architecture that is no bus architecture at all.
The idea a box with:
* Front accepts pieces of angle aluminum, which can have connectors, switches, etc, and also a PCB can be screwed to it.
* The boards with angle aluminum, are all connected to some frame to keep relative positions solid.
* The whole frame can be lifted out of the box, so 3 sides of all PCB's are accessible.
* All PCB's have connectors that fit their function, such as USB, Ethernet, CAN, Whatever.
* I think I'd define a fixed location for an IDC connector for power and some common signals such as I2C, SPI, UART, and then use it either with flatcable or a ridgid PCB.

The main idea though is to have simple way to connect different future projects together in a flexible way so any sort of connector / standard connection can be used for your future projects.

In some ways it's similar to the idea of: "Use DIN rails", on which you can fit almost anything, but then for a relatively neat and flat front panel.

If you look closer at a system like:
https://linuxgizmos.com/modular-open-source-automation-controller-runs-on-raspberry-pi-cm3/
Then (from memory, it was some time ago) it is not much more then a small Linux computer, with a power supply and an USB3 hub, and a fancy box which makes it incompatible with anything else. Is that worth USD350?

I'd rather grab an USD10 USB hub, glue it to a piece of angle aluminum and then use it with any small linux computer that is not based on broadcom chips. But whatever board you want. Olinuxino, Odroid, BBB, etc, or some other board such as some form of microcontroller development board. As long as it's got some mounting holes it's easy to fix to a piece of angle aluminium to stow it away in a box so cables etc do not get dislodged if you want to clean your desk and stow away your project for some time.
And even if it does not have mounting holes, you can still glue it to the aluminum profile :)

Those low volume projects can never compete on price with mass produced standard products.
« Last Edit: November 09, 2020, 10:27:22 am by Doctorandus_P »
 

Offline msat

  • Contributor
  • Posts: 12
  • Country: us
Nice thing about standards, there's so many of them...

What would you really be using this for?

Also, why parallel?  Peripherals are in ever shorter supply, and it's such a pain to breadboard a half a hundred connections (or if we're talking 32 bits here, maybe hundreds).

You're right, there's no real shortage of standards, but it's a reasonable question to ask why there are so many. Sometimes the most ideal standard to use is one that doesn't exist yet, perhaps unfortunately.

Just to be clear, when I say parallel, I don't mean the PC-type parallel port - I just mean the bus is more than 1-bit wide. The reason is for bandwidth purposes for a given clock rate. Plus, it's often fewer instructions on a uC to bit-bang 8 or 16 bits in parallel than it is to shift in/out one bit at a time. Every uC can do this as long as it has sufficient pins. The current thought is 16 bits + few control lines. But this is all tentative and you have me questioning it, which is actually helpful. So, thanks!

Quote
Assuming this gets advertised enough for people to be aware of it -- what would they find in it, that isn't offered by, say:
- PCI: thousands of expansion cards to choose from (some still being made..?); require drivers (which are mostly x86/Windows??)
- ISA, in slot or PC104 form: still standard in industry, I think?  Low bandwidth, may require various compatibility strategies (setting port/memory windows, drivers, etc.).
- USB: just toss a bunch of hubs on it, she'll be right.  Plug-and-play, may need drivers but also many things can just be used as character-mode devices, and structures can be built on top of that (e.g. mounting ttyusbN via some filesystem driver in Linux must be a thing?).
- PCIe: like USB, it's serial and it's fast, just not (usually) cable-oriented.  Tons of peripherals and expansion cards, more bandwidth than you can shake a lane at; plug-and-play; drivers compatible with current computers; etc.
- VME and all those: unless you're supporting ancient cards or doing vintage computing sorts of things, I don't get it, and again don't see how you're differentiating from it..?  (You mentioned VME once but didn't refer back to it later; not sure if we're supposed to understand VME and draw that conclusion ourselves?)
- Or also from the vintage era, GPIB: slow, moderately addressable, cable-oriented.
- Or if bandwidth really isn't much, then any of the embedded serials buses, like I2C, or SPI with multi-master addressing and arbitration, or RS-485 with arbitration.
- Or Ethernet for a similar experience (lots of serial point-to-point links) but way more bandwidth, and again, standard hardware.  VREFs can't be shared, but time is regularly synchronized over it!
- Or for an emphasis on analog, whatever Eurorack synths use..?  Maybe that's just power, I never looked it up...

While I'd be happy if others adopted it, I don't expect it to be the "next big thing". My hope is that perhaps enough other people would use it to create modules that I would find useful for my own needs, but mainly I want something to conveniently control and instrument my experiments. If I'm the only user, that's fine too. After all, I've already done that with previous experiments - and those would probably be almost entirely useless to anyone else.

Yes, plenty of devices exist with various interfaces, but with a home lab budget, you'll mainly be scouring ebay and you'd have to resort to settling with a hodge-podge assortment of equipment requiring a variety of different interfaces, many probably dated (and drivers not supported by newer OSes). That's one of the big things I'm trying to avoid. I'd prefer to have as many tools in one box as possible. I also want the interfacing to be simple and flexible for cost reduction reasons. If you want to make your own PCI compliant card, that's Big Boy stuff. If you want to make a PCIe card, then you're in grey beard territory.

As for mention of VME and VXI in the thread title, I was just using that as an example of what I mean by instrument bus architecture, and not that this necessarily shares any technical specifications with them. I couldn't make that more clear in the title since I reached the character limit.

When I first read your reply last night, you had me thinking of just using USB w/hub as a data bus (though still on a backplane to include some of the other things I want), and it certainly could be a workable idea. After all, plenty of microcontrollers across the performance spectrum support it. There is something about it that I don't like, though, and it's all related to PC-side drivers and enumeration, and the hassle you might have pointing the software to the correct module. I did just have a thought of a potential acceptable solution. I need to ruminate on this some more.

Quote
I guess what you're really after, is some manner of metrology playground, given the emphasis on VREF and CLK?  That already puts it into a very narrow space, and like, if all you're after is some community adoption of something, I guess that could work?  But I don't see much reason for industry to consider it.

Note that most of these buses include a clock already, which may not be all that high quality in the original application, but there's no reason you can't make it so.  You could take a 10MHz disciplined ref and PLL it to the 8MHz of ISA, or 33 or 66MHz of PCI, nicely buffered at each card connector; or just abuse it for your own purposes and dump 10MHz through the pin regardless of the standard.  Likewise, the -5 or -12V supplies (which at a glance, VME has as well?) could be used for REF, equipped with modest current capacity to remain compliant, or who cares just make it a modest -5 or -10V ref and don't use any cards that draw much current from it (which is most of them, AFAIK?).

Tim

I suppose you can consider it a metrology playground in some ways. A global vRef just seems like a worthwhile thing to add in which all modules utilizing ADCs/DACs/comparators/etc can use. It also means you only need to calibrate one reference voltage in a manner potentially much easier than finding offsets for all those devices using internal references. I see no good reason not to have one, especially since a simple one is cheap (but that doesn't stop someone from going fancy if they need/want it)

The global clock is mainly for synchronization, which modules can reference for things like sequencing and timestamping. As with everything else, I haven't completely thought it through, but the idea is to not just be a pulse train, but also have a data line transmitting the current counter value. Once again, the quality of the clock is up to the user.

-Mark
 

Offline msat

  • Contributor
  • Posts: 12
  • Country: us
This is also a long standing idea of mine, but it never got much further then that.
Join this thread:
https://www.eevblog.com/forum/projects/diy-instrumentation-bus-(or-dib)/

I think it's mostly fizzled out over the years, but you may find it interesting to read.

Thanks for bringing this to my attention! I suspected that I wasn't the only one interested in something like this. I will be looking through that thread for ideas and inspiration, or perhaps even try to revive someone else's good ideas.

Quote
Here's a list:
http://www.interfacebus.com/Interface_BackPlane_Buses.html

There are also a few open source bus based systems you can re-use. Some time ago I even found an open source project which made it into a real-life product and you can buy chassis with power supply and several plugin boards commercially for reasonable prices.

I didn't come across a bus that I was satisfied with, mainly because I figured I'd end up creating the modules I wanted anyways, so I figured speccing my own bus would end up making my life easier and thriftier in the future.

Quote
=============
My latest idea is to use a bus architecture that is no bus architecture at all.
The idea a box with:
* Front accepts pieces of angle aluminum, which can have connectors, switches, etc, and also a PCB can be screwed to it.
* The boards with angle aluminum, are all connected to some frame to keep relative positions solid.
* The whole frame can be lifted out of the box, so 3 sides of all PCB's are accessible.
* All PCB's have connectors that fit their function, such as USB, Ethernet, CAN, Whatever.
* I think I'd define a fixed location for an IDC connector for power and some common signals such as I2C, SPI, UART, and then use it either with flatcable or a ridgid PCB.

The main idea though is to have simple way to connect different future projects together in a flexible way so any sort of connector / standard connection can be used for your future projects.

In some ways it's similar to the idea of: "Use DIN rails", on which you can fit almost anything, but then for a relatively neat and flat front panel.

That's certainly an effective way of making things more tidy, but it's limited in what it can do about cable chaos. A data bus doesn't have to be super complex, but then obviously you're limited to equipment that supports it. Tade-offs, trade-offs, trade-offs.

Quote
If you look closer at a system like:
https://linuxgizmos.com/modular-open-source-automation-controller-runs-on-raspberry-pi-cm3/
Then (from memory, it was some time ago) it is not much more then a small Linux computer, with a power supply and an USB3 hub, and a fancy box which makes it incompatible with anything else. Is that worth USD350?

I think you summed it up. I can't say it doesn't look cool, but otherwise it's a lot of things I don't particularly want. There's one low-performance interface (I2C) while the rest are vastly more high performance (and thus complex). Plus, while I don't have anything against a RasPI, I strongly dislike being anchored to any particular computer [architecture] or OS. It's also one of the things I dislike about some of the other buses that are intended to utilize a computing module. Why waste time designing a compatible SBC when you can plug in almost anything you want via ethernet/USB (unless you require the performance a higher bandwidth interface provides?)

Quote
I'd rather grab an USD10 USB hub, glue it to a piece of angle aluminum and then use it with any small linux computer that is not based on broadcom chips. But whatever board you want. Olinuxino, Odroid, BBB, etc, or some other board such as some form of microcontroller development board. As long as it's got some mounting holes it's easy to fix to a piece of angle aluminium to stow it away in a box so cables etc do not get dislodged if you want to clean your desk and stow away your project for some time.
And even if it does not have mounting holes, you can still glue it to the aluminum profile :)

Those low volume projects can never compete on price with mass produced standard products.

Yep! It's both simpler and cheaper!

So I'm going to go through that thread you linked, and if it's appropriate to continue this discussion there, then I'll do so.
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 16066
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Ahh, so instrumentation, generic peripherals, that sort of thing.

You might take the ad-hoc approach that many others have taken: just pin out a whole ass MCU, and use whatever you need.  Arduino Shield, TI Launchpad, ST Nucleo, etc. -- whichever flavor of MCU you like.  Breadboard something with jumpers and breakouts, and if you find yourself reusing something often, print a board for it (or deadbug it or whatever).

Arduino is a good example of flexible libraries, too; the drivers and headers are all there with support for many configurations, from MCU platform to peripherals.  Perhaps a downside is it's statically linked (compiled), and you could build an interpreter on top of that but only after quite some effort.  This is easier on platforms with dynamic code (most ARMs?) than restricted or Harvard architectures (AVR).

Using a common bus, like USB, doesn't make much sense for one-off peripherals like you'd use this way.  You could make a USB-SPI bridge and whack a bunch on a hub (yes, with suitable identification to select them; there could be position index pins in the backplane for example, which show up in the PID or whatever), but you still have to program for each one, so it's as much effort as putting them all on one or several SPI ports on an MCU and programming it instead.  The lower-level option is usually better, for sure you don't have to jump through as many hoops to manage the high level OS stuff.

A little bit more on PCIe -- don't be so hasty I think.  As far as I know, it's as plug-and-play as USB, the OS (you'll be using something like Linux on these platforms -- SoCs and SoMs, often multicore, lots of CPU cycles available) enumerates it and puts it in an address space.  As root, you can poke it directly and implement drivers from bash if you want, or given suitable drivers (ARM/Linux is very common these days, after all) you can do... whatever you're supposed to do with them.  PCIe lanes aren't troublesome to route, the main thing is they're low voltage, so sensitive to common mode interference -- routing through headers or edge connectors is about the worst you should do with them, and surround the pins with lots of grounds when you do.

I don't know that there's all that much general-purpose functionality available in PCIe directly, like what you'd be looking to do here -- aside from glue of course (bridges to lower bandwidth buses?), but as far as I know it's not as hard to use as it sounds, probably not worse than anything else here.

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 20137
  • Country: nl
    • NCT Developments
Nice thing about standards, there's so many of them...

What would you really be using this for?

Also, why parallel?  Peripherals are in ever shorter supply, and it's such a pain to breadboard a half a hundred connections (or if we're talking 32 bits here, maybe hundreds).

You're right, there's no real shortage of standards, but it's a reasonable question to ask why there are so many. Sometimes the most ideal standard to use is one that doesn't exist yet, perhaps unfortunately.

Just to be clear, when I say parallel, I don't mean the PC-type parallel port - I just mean the bus is more than 1-bit wide. The reason is for bandwidth purposes for a given clock rate. Plus, it's often fewer instructions on a uC to bit-bang 8 or 16 bits in parallel than it is to shift in/out one bit at a time. Every uC can do this as long as it has sufficient pins. The current thought is 16 bits + few control lines. But this is all tentative and you have me questioning it, which is actually helpful. So, thanks!
Somehow I2C pops into my mind when reading this. You can drive this at speeds of 1MHz nowadays so it is in now way slow and every reasonably modern uC has a hardware I2C interface nowadays.
If you insist on parallel then the good old GPIB could be a candidate. But if there is one bus master you might as well go for USB and use a Raspberry Pi to control the instruments.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline msat

  • Contributor
  • Posts: 12
  • Country: us
Ahh, so instrumentation, generic peripherals, that sort of thing.

Yeah, instrumentation and process control of experiments that aren't purely electronic in nature.

Quote
You might take the ad-hoc approach that many others have taken: just pin out a whole ass MCU, and use whatever you need.  Arduino Shield, TI Launchpad, ST Nucleo, etc. -- whichever flavor of MCU you like.  Breadboard something with jumpers and breakouts, and if you find yourself reusing something often, print a board for it (or deadbug it or whatever).

Arduino is a good example of flexible libraries, too; the drivers and headers are all there with support for many configurations, from MCU platform to peripherals.  Perhaps a downside is it's statically linked (compiled), and you could build an interpreter on top of that but only after quite some effort.  This is easier on platforms with dynamic code (most ARMs?) than restricted or Harvard architectures (AVR).

So this is what I've been doing and what I want to get away from! Once you start asking the microcontroller to do a lot of stuff, it can potentially be difficult to maintain performance (or should I say, more hassle than it's necessarily worth), and if you find yourself adding more and more functionality as project requirements change, you might be doing a lot of code refactoring, and it gets tedious. If you just pipe the data to and from the PC, you can do all the heavy lifting there where there's plenty of performance to spare. I've never done it that way, but I'm thinking it'd end up being a lot more convenient. After all, the code isn't the project, it's just the means for running and measuring/testing it.

Then there's the whole issue of finding the right microcontroller. Usually what happens to me is that I find a part that has almost every feature that I want except one or two things, which ends up making the project a lot more difficult. And if you can find a part that does everything you want, it often doesn't have a demo board available (or not useful enough if it does) and it's in a package that's miserable to work with for a hand assembled one-off.


Quote
Using a common bus, like USB, doesn't make much sense for one-off peripherals like you'd use this way.  You could make a USB-SPI bridge and whack a bunch on a hub (yes, with suitable identification to select them; there could be position index pins in the backplane for example, which show up in the PID or whatever), but you still have to program for each one, so it's as much effort as putting them all on one or several SPI ports on an MCU and programming it instead.  The lower-level option is usually better, for sure you don't have to jump through as many hoops to manage the high level OS stuff.

I'm starting to suspect that you're a mind reader! If I go the USB-on-backplane route, I was thinking of electrically keying the module slots so that the modules can report which slot they're in to the software, thus automating that part of the process. Of course USB is appealing for many reasons. It's robust, very well supported, and would make this whole thing a lot easier to develop. Pretty much all that's necessary are some USB hub ICs. That said, there's also things that I don't like about it. For one, isolation (to avoid ground loops, or worse) becomes tricky and/or expensive. Might have to do it all on the uC peripheral side. Ugh. The other thing is that the USB stack on a lot of these microcontrollers eat up quite a bit of processing cycles. I suppose this isn't a huge deal, considering that each module probably won't have that much variety in functionality to be a cause for concern, but it's worth at least thinking about.

You mention a USB to SPI bridge, which is also something I've been thinking about. Some of the dedicated parts that I've seen after a brief search I didn't find to be suitable, but I think there's some microcontrollers out there that would have the performance and flexibility I'd want. The benefit with SPI is that the majority of microcontrollers support it in hardware, and if the USB<->SPI bridge just transmits the data extracted from USB over SPI, there's very little overhead in terms of data transmission required on part of the individual module's uC. Also, SPI - and especially Quad SPI - @ 50MHz has substantially more bandwidth than USB Full Speed which is the extent of what a lot of microcontrollers support. I suspect isolation would be easier, too. The biggest potential downside is that the module's uC might often need two hardware SPI peripherals since it might use that to interface with ADCs etc.

 
Quote
A little bit more on PCIe -- don't be so hasty I think.  As far as I know, it's as plug-and-play as USB, the OS (you'll be using something like Linux on these platforms -- SoCs and SoMs, often multicore, lots of CPU cycles available) enumerates it and puts it in an address space.  As root, you can poke it directly and implement drivers from bash if you want, or given suitable drivers (ARM/Linux is very common these days, after all) you can do... whatever you're supposed to do with them.  PCIe lanes aren't troublesome to route, the main thing is they're low voltage, so sensitive to common mode interference -- routing through headers or edge connectors is about the worst you should do with them, and surround the pins with lots of grounds when you do.

I don't know that there's all that much general-purpose functionality available in PCIe directly, like what you'd be looking to do here -- aside from glue of course (bridges to lower bandwidth buses?), but as far as I know it's not as hard to use as it sounds, probably not worse than anything else here.

Tim

I just think that for most devices, a PCIe interface is excessive and is more difficult to develop for than everything else discussed. I also want to be able to use a computer that doesn't necessarily have PCIe available (laptop, SBC, etc), and I absolutely don't want a computer to have to sit on the bus. For what I want, I think it's the wrong way to go.

I have to do some part hunting to get a better idea of what's out there to help with decision making. I wouldn't want to decide on something just to find that there's a limited number of suitable microcontrollers.
 

Offline msat

  • Contributor
  • Posts: 12
  • Country: us

Somehow I2C pops into my mind when reading this. You can drive this at speeds of 1MHz nowadays so it is in now way slow and every reasonably modern uC has a hardware I2C interface nowadays.
If you insist on parallel then the good old GPIB could be a candidate. But if there is one bus master you might as well go for USB and use a Raspberry Pi to control the instruments.

I2C is convenient in many ways, and offers sufficient bandwidth for many things, but it's towards the lower end in terms of performance. At <1MBit data bandwidth, that's far too low for a high-res/sample-rate multi-channel ADC. So what happens when you want a bunch of those in your rack? More performance is needed.
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 16066
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
So this is what I've been doing and what I want to get away from! Once you start asking the microcontroller to do a lot of stuff, it can potentially be difficult to maintain performance (or should I say, more hassle than it's necessarily worth), and if you find yourself adding more and more functionality as project requirements change, you might be doing a lot of code refactoring, and it gets tedious. If you just pipe the data to and from the PC, you can do all the heavy lifting there where there's plenty of performance to spare. I've never done it that way, but I'm thinking it'd end up being a lot more convenient. After all, the code isn't the project, it's just the means for running and measuring/testing it.

Then there's the whole issue of finding the right microcontroller. Usually what happens to me is that I find a part that has almost every feature that I want except one or two things, which ends up making the project a lot more difficult. And if you can find a part that does everything you want, it often doesn't have a demo board available (or not useful enough if it does) and it's in a package that's miserable to work with for a hand assembled one-off.

So get a bigger one!  SoM, rPi, etc.  Or any kind of PC, from the same sorts of things to mini motherboards to full boxes. ;D

Obviously as things get bigger, you'll need to manage the higher bandwidth buses and OS stuff.  Which is why PCIe looks attractive, even if just used to route between hubs (more PCIe lanes) and bridges or peripherals (e.g. USB ports, and more hubs from there?).

It's a logistical problem, no matter how you slice it.  I'm not sure what kind of scale you're talking -- how many, and how many kinds of, things you want to use at once, but it starts to sound like coordinating all of them together is your challenge, and the platform and the ratio of hardware to software work required, is just a symptom of that, not the core problem.

Also if you're working with a lot of mechanical stuff, that's a very much solved problem, in everything from industrial automation to hobby CNC.  PLCs are plenty smart these days!  (Again, the amount of solution being in relative terms...)

And there's always the go-to lab options, like National Instruments' DAQ systems.  They've solved it top down bottom up, from GUI interface to hardware.  The price is a tiny fraction of the labor required to recreate it -- very much worthwhile, if you're doing as much work as it sounds like you are.


Quote
For one, isolation (to avoid ground loops, or worse) becomes tricky and/or expensive. Might have to do it all on the uC peripheral side. Ugh. The other thing is that the USB stack on a lot of these microcontrollers eat up quite a bit of processing cycles. I suppose this isn't a huge deal, considering that each module probably won't have that much variety in functionality to be a cause for concern, but it's worth at least thinking about.

Meh, isolation doesn't bother me anymore.  Avoiding loops is a system level problem, you can only mask it with local solutions.

That said, isolation does happen to be a local solution with unusually high efficacy...  But indeed, it's not the simplest thing to do, in terms of parts required, or cost of modules.

Taking differential measurements is the easiest partial solution, and just avoiding ground loops in the system in general.

Something a PC can be a PITA with, an experience I've already had on my bench.

Most recent example: PC is plugged in and grounded.  USB goes to a hub, to various peripherals, including a serial port and programmer.  These connect to the EUT, a DSP project.  The audio output from which also goes to the PC (sound card).  It's powered by the bench supply (floating), and probed by the scope (grounded).  Well, something about these three paths (USB, audio, scope) leaves a lot of noise, and it's not actually RF interference being detected, it's low frequency (including DC) offsets.  Noise goes away when any of the three is disconnected.  (Best I can tell, it's actually coming from the PC's USB port, but it's weird.)

I solved the noise issue, for testing purposes, by building a differential amplifier on the breadboard.  This reads the audio input's ground reference, and translates the DSP's output level to the new ground reference.  Voila, noise gone.  It happens to be a local solution, but driven all the same by system-level understanding.

And laptops can be even worse, sometimes a buzz or whine comes through the charger so they're quiet when unplugged, but you only get some hours of runtime before you have to plug it back in again.


Quote
You mention a USB to SPI bridge, which is also something I've been thinking about. Some of the dedicated parts that I've seen after a brief search I didn't find to be suitable, but I think there's some microcontrollers out there that would have the performance and flexibility I'd want. The benefit with SPI is that the majority of microcontrollers support it in hardware, and if the USB<->SPI bridge just transmits the data extracted from USB over SPI, there's very little overhead in terms of data transmission required on part of the individual module's uC. Also, SPI - and especially Quad SPI - @ 50MHz has substantially more bandwidth than USB Full Speed which is the extent of what a lot of microcontrollers support. I suspect isolation would be easier, too. The biggest potential downside is that the module's uC might often need two hardware SPI peripherals since it might use that to interface with ADCs etc.

You could easily make (or get) a little dev board that's an MCU and a few ports pinned out, and send e.g. serial commands to it, via RS-232, USB, whatever; more generally, a character-mode interface with internal state.  You have commands to set and read register states, and in turn affect the state of whatever's attached to it.

Then it's just a networking and data collection problem, managing a plurality of such interfaces.  ("Just", he says :) )

Wouldn't be conducive to 50MHz QSPI, no, but... do you really need that?  If you're going to blast a bunch of SD cards or something, just plug 'em into the poor computer in the first place..?!!

I did this exercise in GPIB, adapting a popular codebase to the platform/interface I opted for; the generalization of that is SCPI, as I understand it?

This is generally how I approach one-offs, demo new peripherals, etc.  I have a serial command line interface I drop into the project, then patch in whatever specific commands I need to play with the device in question.  Then build up more elaborate commands to automate e.g. initialization, read/write cycles, drawing graphics for displays, etc.  Most recently I did this with an emulated (via GPIOs) parallel bus, but also SPI and etc.  Or just twiddling I/O registers directly (or doing some light debugging of the program itself, by viewing/twiddling RAM).

Oh hey I've even got a released example: https://github.com/T3sl4co1l/Reverb

Maybe you'd have to do this anyway, just for starters.  I don't know that you'd be able to avoid this, aside from dropping in someone else's drivers or whatever for some particular device -- in any case, that of course still leaves the next layer up, coordinating everything.


Quote
I just think that for most devices, a PCIe interface is excessive and is more difficult to develop for than everything else discussed. I also want to be able to use a computer that doesn't necessarily have PCIe available (laptop, SBC, etc), and I absolutely don't want a computer to have to sit on the bus. For what I want, I think it's the wrong way to go.

Right, a laptop might not have it available externally, although if you can hack into a docking station, that might provide some access.  SBCs vary, maybe?  If it's got a slot, you can always put an extension cord in one.  Some SoMs definitely have it available.

SATA and USB3 would also be options?  I'm not sure how general-purpose SATA is, it's almost always used for drives.  But USB is definitely attractive, and offered by most everything, yeah.

Also, IEEE 1394?  Not even sure offhand if that's an enumerated bus or point-to-point only... I never have to use it. :-DD

Tim
« Last Edit: November 11, 2020, 09:16:37 am by T3sl4co1l »
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline msat

  • Contributor
  • Posts: 12
  • Country: us

So get a bigger one!  SoM, rPi, etc.  Or any kind of PC, from the same sorts of things to mini motherboards to full boxes. ;D

Obviously as things get bigger, you'll need to manage the higher bandwidth buses and OS stuff.  Which is why PCIe looks attractive, even if just used to route between hubs (more PCIe lanes) and bridges or peripherals (e.g. USB ports, and more hubs from there?).

It's a logistical problem, no matter how you slice it.  I'm not sure what kind of scale you're talking -- how many, and how many kinds of, things you want to use at once, but it starts to sound like coordinating all of them together is your challenge, and the platform and the ratio of hardware to software work required, is just a symptom of that, not the core problem.

Also if you're working with a lot of mechanical stuff, that's a very much solved problem, in everything from industrial automation to hobby CNC.  PLCs are plenty smart these days!  (Again, the amount of solution being in relative terms...)

And there's always the go-to lab options, like National Instruments' DAQ systems.  They've solved it top down bottom up, from GUI interface to hardware.  The price is a tiny fraction of the labor required to recreate it -- very much worthwhile, if you're doing as much work as it sounds like you are.

From my understanding, a RasPI (I got one of those tiny ones when they were $5, but I never even powered it up) are not all that great at twiddling bits on their GPIO pins compared to many "lesser" micros. Of course, there are some beefy-ish SoCs from the likes of TI and ST that have additional cores meant for real-time IO stuff.

Don't get me wrong, I think PCIe is pretty amazing. It can support a metric buttload of bandwidth at very low latencies. If you need maximum performance, you won't find it anywhere else, especially considering the hardware is baked in on all modern CPUs, unlike when PCI, AGP etc used to be implemented in a northbridge chip, and translated to whatever bus sat between that and the CPU.

I'm not sure what scale I'm talking about either, lol. What I do know is that I want something that can easily scale with growing requirements for experiments. I think others could find value in that too. I could probably continue to get by with the way I've been doing things, but it does get messy and it bugs me. There's been times when I hacked together a circuit which did the job just fine (if maybe a little delicate) but looking at the damn thing drove me nuts so I went ahead and created an PCB for it. Maybe a waste of time, but it sure does make me feel better!

There is a lot of nice equipment out there that already exists, but I have it stuck in my head that for a variety of functions it would be cheaper to build much of it myself. I'm probably wrong, but I like building stuff!


Quote

Meh, isolation doesn't bother me anymore.  Avoiding loops is a system level problem, you can only mask it with local solutions.

That said, isolation does happen to be a local solution with unusually high efficacy...  But indeed, it's not the simplest thing to do, in terms of parts required, or cost of modules.

Taking differential measurements is the easiest partial solution, and just avoiding ground loops in the system in general.

Something a PC can be a PITA with, an experience I've already had on my bench.

Most recent example: PC is plugged in and grounded.  USB goes to a hub, to various peripherals, including a serial port and programmer.  These connect to the EUT, a DSP project.  The audio output from which also goes to the PC (sound card).  It's powered by the bench supply (floating), and probed by the scope (grounded).  Well, something about these three paths (USB, audio, scope) leaves a lot of noise, and it's not actually RF interference being detected, it's low frequency (including DC) offsets.  Noise goes away when any of the three is disconnected.  (Best I can tell, it's actually coming from the PC's USB port, but it's weird.)

I solved the noise issue, for testing purposes, by building a differential amplifier on the breadboard.  This reads the audio input's ground reference, and translates the DSP's output level to the new ground reference.  Voila, noise gone.  It happens to be a local solution, but driven all the same by system-level understanding.

And laptops can be even worse, sometimes a buzz or whine comes through the charger so they're quiet when unplugged, but you only get some hours of runtime before you have to plug it back in again.

Your method of choice for isolation is certainly valid, but if we're talking about a bus architecture that's intended to communicate back to a computer via USB, then you pretty much have to consider it at the very least on the USB side, especially if you can't have the hardware powered by it. Now, USB3 is differential dual simplex. Looking at the FX3 chip's tech ref manual, it appears that the device attempts initialization via USB3, and if that fails, falls back to USB2 (controllable via software). That makes me wonder whether I could disable all USB2 functionality and also avoid sharing the grounds. Perhaps that's not the best practice, but the question is, how bad of an idea is that?

Quote


You could easily make (or get) a little dev board that's an MCU and a few ports pinned out, and send e.g. serial commands to it, via RS-232, USB, whatever; more generally, a character-mode interface with internal state.  You have commands to set and read register states, and in turn affect the state of whatever's attached to it.

Then it's just a networking and data collection problem, managing a plurality of such interfaces.  ("Just", he says :) )

Wouldn't be conducive to 50MHz QSPI, no, but... do you really need that?  If you're going to blast a bunch of SD cards or something, just plug 'em into the poor computer in the first place..?!!

Not everything would need 50MHz QSPI, no, but some things certainly might! The nice thing is, a device that can do QSPI can also do regular SPI at a variety of frequencies. It's there if you need it, but you don't have to use it. If I could find one, I'd seriously consider a uC that can either do USB2 Hi-Speed or USB3 and have two QSPI hardware peripherals. If it was USB2, I'd power the chip off the bus, and then isolate the SPI side.

Quote
I did this exercise in GPIB, adapting a popular codebase to the platform/interface I opted for; the generalization of that is SCPI, as I understand it?

I briefly started looking into SCPI after if got mentioned in the thread Doctorandus_P linked too. I haven't made my way through the whole thread yet, but a lot of my thoughts/concerns had already been brought up there, so I feel like I'm kind of on the right track.

Quote
This is generally how I approach one-offs, demo new peripherals, etc.  I have a serial command line interface I drop into the project, then patch in whatever specific commands I need to play with the device in question.  Then build up more elaborate commands to automate e.g. initialization, read/write cycles, drawing graphics for displays, etc.  Most recently I did this with an emulated (via GPIOs) parallel bus, but also SPI and etc.  Or just twiddling I/O registers directly (or doing some light debugging of the program itself, by viewing/twiddling RAM).

Oh hey I've even got a released example: https://github.com/T3sl4co1l/Reverb

Maybe you'd have to do this anyway, just for starters.  I don't know that you'd be able to avoid this, aside from dropping in someone else's drivers or whatever for some particular device -- in any case, that of course still leaves the next layer up, coordinating everything.

I have no real idea how the software side (not related to firmware) is going to work, yet. Obviously, it would be wisest to stick with existing standards where applicable. So perhaps other than standardized USB APIs, I don't want to specify any particular software tools or programming languages.


Quote


SATA and USB3 would also be options?  I'm not sure how general-purpose SATA is, it's almost always used for drives.  But USB is definitely attractive, and offered by most everything, yeah.

Also, IEEE 1394?  Not even sure offhand if that's an enumerated bus or point-to-point only... I never have to use it. :-DD

Tim

So I'm all but dead set on going with USB3. It's widely supported by computers (not so much on micros - just fancy SoCs), and offers a good amount of bandwidth with pretty low latency. At the very least, USB2 Hi-Speed. Avoiding everything else, including ethernet.

Honestly, 1394 wouldn't be that bad. However, it's no longer supported by Windows, meaning there's probably not a single new computer out there that has a 1394 port, nevermind an SBC.
 

Offline msat

  • Contributor
  • Posts: 12
  • Country: us
After a lot of back and forth consideration, and not wanting to back myself into many corners, I realized I can accomplish much of what I want with minimal standardization and do things on an as-needed basis in a way that's most convenient at the time. Really, what this will boil down to is visible cleanliness. So I'll merely be stuffing everything in a 3U subrack with no backplane (at least not a global one). Other than limiting modules to a maximum 100mm height, depth will be variable (up to some maximum, of course). Connections to modules will be made with suitable cables (USB, etc) as needed. COTS parts will be used when possible (USB hubs, dev boards, etc) and either they can be attached to 100mm carrier board, or rails can be 3D printed to accept smaller boards. Faceplates can be 3d printed as well. Not only is this the simplest/quickest, but also the most flexible since probably all of the modules would at least in part be designed by myself anyways. I ended up realizing that the main benefit of standardization is compatibility which is really only important if you're buying rather than making hardware. Since any standard bus I'd create would not have much uptake (and I had no appetite to try to commercialize it), what compatible hardware would there be to buy? So I've come up with a name for my "new" "standard": Low Standards  ;D
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf