sometimes the open-source sucks

[legacy]

It usually takes about fifteen minutes for me to get a good handle on an open source project, by looking at the mailing list

We are having problems with the router node now. If observed within a time window of a few hours it appears "stable" and "solid" like a stone.

However, within 48h, the probability of a crash is also solid true, and this means there is some "bastard" bug somewhere.

Now, the latest component added is a pair of Oxford Semiconductor Ltd OX16PCI954 UARTs, which are capable of DMA, therefore it's conceivable they could corrupt something, but it's difficult to confirm because the doc is unclear (yes, it might support, but yes/no/implemented?configured?WTF?), and the code ... is weird, even the PCI enum looks weird.

Weeks later, digging deeper finally we observed that the OX16PCI954 doesn't have DMA enabled, hence it cannot be the guilty. There the problem seems now related to the VIA-Rhine chip, which is DMA capable, and stress-tested during our burn-in.

I don't know how accurate Microtik's tests were , but I know that the board is still mainstream with Linux, and nobody noticed that starting from kernel 4.4.* the router does not even bootstrap kernels.

I do find it funny 

Do you happen to know to whom I should write? I am in the MIPS mailing list, but ... they seem to be not very interested because kernel 4.4 is about for years old, and, in general, people are not very interested in debugging kernels that old.

[bd139]

Ahh microtik. Have you tried using a cricket bat to hit it over a fence?

[legacy]

I re-wrote the section, & submitted it to them via email. By the very next day, they emailed me back,
confirmed my diagnosis/rewrite, thanking me for my efforts, and informed me they have now included it
in the very next revision Update coming out shortly !   

We spent two months testing PCI_SATA controllers for HPPA and PPC. In the end my team found a bug with the HPPA Linux support, hence we informed the mailing list, and we worked together to get the bug fixed. Then a dude merged the patch mainstream so it was included directly in Linux.

I remember I wrote to a retired engineer who worked for HP during the gold age of PA-RISC, and he said that he didn't remember details regarding the hardware but he had his personal notes in a drawer and he would have given an eye for us.

His help contributed to bug-fix, and it took only a couple of days this way.

Great!!!! That was a super very great experience 

[legacy]

Ahh microtik. Have you tried using a cricket bat to hit it over a fence?

... that board has an evil story on its shoulder.

I bought it online years ago from a company. The board arrived perfectly packaged and brand new.

Something like two years after the purchase I received a legal letter from a lawyer claiming that from the business budget resulted that my router not paid. He said there was an open business invoice, and he intimated me to give him the money.

At that point, since the router was paid online, for sure it was paid via Paypal, hence I wrote to them in order to have written confirmation, but I had to hire a lawyer to validate the document and to talk with the other lawyer in order to close this story.

In the end, we understood what happened: the person from whom I bought was an employee of a company that was selling Mikrotik products, but he tried to sell a product like if it was of his personal property. Therefore there was an open business invoce ... to me.

Hence he got the money, but not the company. This is something like if you steal something from the company where you are employed.


I spent 200 euro to fix this story, the router cost me 140 euro, therefore now, it's a personal matter, and IT MUST work 

[bd139]

Oh nice. Had one of those nice people before. Issue turned out to be a SAP bug in their system but I had to diagnose it remotely from their idiocy 

I understand the position now

[Nominal Animal]

However, within 48h, the probability of a crash is also solid true, and this means there is some "bastard" bug somewhere.

Kernel panic? Silent lockup? Anything in the logs? Do you monitor the key component temperatures to verify they stay within sane values?

I don't know how accurate Microtik's tests were

532A is already quite old, so even thermal issues are possible (bad caps?).  Plus, Mikrotik is Mikrotik; I would not expect any support from them.

Do you happen to know to whom I should write? I am in the MIPS mailing list, but ... they seem to be not very interested because kernel 4.4 is about for years old, and, in general, people are not very interested in debugging kernels that old.

No, unfortunately not.  OpenWRT folks might know more, but they seem to hang out on IRC, which I don't use.

[Siwastaja]

And this is one of the reasons why I'm never going to bother using complex free software for anything productive if there's a commercial alternative.

That is fairly idiotic.  You are associating "open source == badly managed project" and "commercial == well managed project".

Indeed. Altium Designer, for example, is a very classical example of a commercial project where the management refuses to have bugs fixed, despite ever-growing bug tracker records with thousands of unfixed issues, with reports of customer reps fighting against the customers, claiming that the well-known bugs are caused by the customer's hardware. Instead, they focus on ticking more feature boxes. Fixing trivial bugs is reported to take years, even a decade.

It actually sounds very much like a typical badly managed open source project, and has many similarities to KiCad community.

Which is unsuprising - it's people, after all, who make the management decisions. It's to be expected to see similar management and support problems in both open source, and commercial software.

[Nominal Animal]

Exactly.  It is not a coincidence that people-managers call managing developers cat herding.

[legacy]

Kernel panic? Silent lockup? Anything in the logs? Do you monitor the key component temperatures to verify they stay within sane values?

Code: [Select]

do_page_fault(): sending SIGSEGV to minicom for invalid read access from 00000000
epc = 77b867b8 in libc-2.9.so[77ad4000+160000]
ra  = 00414b18 in minicom[400000+2f000]


Code: [Select]

CPU 0 Unable to handle kernel paging request at virtual address 1040ff60, epc == 8049543c, ra == 80495fe0
Oops[#1]:
CPU: 0 PID: 6395 Comm: in.tftpd Not tainted 4.4.197-BlurryFishButt-rb532 #2

Events like these, logs are silent, but we see symptoms of memory corruption in the tty structure, and ethernet structures.

During the burn-in, minicom is used to monitor one of the serial channel bombarded by high-streams, while the TFTP server is also bombarded by requests.

And these two randomly crash, but only under heavy load, probably because there is a concurrency in the PCI bus regarding the two ethernet units and the two uart modules.

I don't know 532A is already quite old, so even thermal issues are possible

Anyway, we have two RB532A with the same behavior. They are running in our computer room, with ventilation and 14 C controlled temperature.

Not sure, but it looks more a software problem; kernel 2.6 are all unable to operate when uart-modules are installed (the kernel crashes immediately), kernel <=4.4 are OK, but it also maybe caps, which haven't been investigated yet.

[Nominal Animal]

And these two randomly crash, but only under heavy load, probably because there is a concurrency in the PCI bus regarding the two ethernet units and the two uart modules.

Is this RB532A, a Mikrotik board that allows one to stack "daughtercards", each providing mini-PCI slots?
If so, this could just be a Mikrotik hardware design issue also.

But yeah, I do agree it is most likely a software issue.

Have you ran some sort of memtest on it?  If not, that might be a good idea.  (I would personally run it as init, allocating suitable amount of RAM (most of it), mlock() the RAM, an then repeatedly fill it with a Xorshift generator verifying with the same initial data that the data is unchanged.  I do not believe the RAM is at fault here, but making sure, excluding it as a possible reason, could be useful.)

The minicom SIGSEGV looks like a NULL pointer dereference, though.

Also, do you have two mini-PCI UART modules installed?  The driver may not be designed to do that correctly.  A lot of PCI devices assume the system has exactly one of those.  If you have an online link to the sources (gitweb or similar, I'm lazy and don't want to untar tarballs) of that particular driver, I could take a look and make an educated guess.

Not sure, but it looks more a software problem; kernel 2.6 are all unable to operate when uart-modules are installed (the kernel crashes immediately), kernel <=4.4 are OK, but it also maybe caps, which haven't been investigated yet.

No, I agree; I am just pointing out some of the possible causes just in case.

Me, I'd verify the RAM using some sort of quickly written memory test, so I could exclude the RAM from the list of possible reasons.  Then, I'd modify it a bit, to use a few rounds of the Xorshoft PRNG per word of memory, to also pin the CPU, to get a RAM+CPU stress test.  If that does not trigger any crashes or unexpected memory changes, you can basically exclude userspace from the list of possible reasons, leaving just the kernel drivers.  While it can seem like superfluous work, being able to subdivide the problem space like this, excluding possible by unlikely causes one by one, works.

That said, if this indeed is the multi-miniPCI-daughter-card board, and you have more than one daughter card installed, you could test them one-by-one, using the same kernel configuration if possible.  This too takes time, but if they are stable alone, and only unstable when both in use -- perhaps the stack/slot order matters too? --, then you have pinpointed the problem to the interaction between the two drivers.

Then, adding some kind of PCI spinlock in the driver (as you said, the devices do not use DMA), to ensure "logical transactions" to the two devices are not interspersed, might "fix" the issue.  If it does not, it is a hardware issue.  If it does, then one can contact the Linux PCI people, and ask for advice on how to properly implement that feature.

Or this is how I would go about it.  There is probably a much better way; I just don't know it.

[legacy]

Is this RB532A, a Mikrotik board that allows one to stack "daughtercards", each providing mini-PCI slots?
If so, this could just be a Mikrotik hardware design issue also.

description and a few pics are here. It's a rb532A + rb502-daughtercards.

Have you ran some sort of memtest on it?

The firmware comes with a diagnost that test the physical ram: done, all passed.

Besides as a "wild" test, we compiled "cmake" (which strongly uses C++ on large chunks) directly on the board. It took something like two weeks no stop, and it used all the physical ram (64Mbyte) plus 200Mbyte of the stack on the microdrive plus 200Mbyte of the stack over NFS.

This way we tested
- the ram
- the Microdive
- the built-in Ethernet

This test was performed without any additional miniPCI hardware, just the router compiling stuff in a corner.

Insanely crazy, but greatly funny 

Also, do you have two mini-PCI UART modules installed?  The driver may not be designed to do that correctly. 

Yup, your plan about testing only one quad-uart sounds good 

[legacy]

The minicom SIGSEGV looks like a NULL pointer dereference, though.

yes, but an application in userspace usually doesn't crash the kernel like it happened a few seconds later. I do believe it was a synthom that something corrupted both the user and kernel space.

I mean
If something corrupts the memory in userspace -> the app crashes, the kernel does not crash
If something corrupte the memory in kernel space -> weird behaviors manifest followed by kernel crash.

[legacy]

multi-miniPCI-daughter-card board, and you have more than one daughter card installed

You can only have one daugther-card "rb502" installed on the top of the router "rb532A", and it gives two extra miniPCI slots.

Currently we have
-A- one quad-uart miniPCI module installed on the router rb532A
-B- one quad-uart miniPCI module installed on the daugther-card rb502

Crashes randomly happen the same way, no matter if we test a uart channel on A or B. This probably means the daugther-card rb502 is OK. More tests are need to confirm.

[Nominal Animal]

Yup, agreed; the stress testing you've done clearly indicates the issue is probably the kernel driver (some PCI detail overlooked, maybe tiny race window), or (unlikely but possible) the PCI hardware implementation on that board.

I would definitely test with just A, and RB502 disconnected, for say eight hours (or until crash).
Then, with just A, but RB502 connected (B not connected), for another say eight hours (or until crash).
Then, without A, but RB502 connected and B connected, for another say eight hours (or until crash).

[legacy]

If you have an online link to the sources (gitweb or similar, I'm lazy and don't want to untar tarballs) of that particular driver, I could take a look and make an educated guess.

Thanks for your help. Mure more than appreciated 

I have just created some pages here and uploaded text files, only those involved with the serial driver, so you can consult them online.

Code: [Select]

00:0a.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) (rev 01) (prog-if 06 [$
        Subsystem: Oxford Semiconductor Ltd Device 0000
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 140
        Region 0: I/O ports at 18800840 [size=32]
        Region 1: Memory at 50012000 (32-bit, non-prefetchable) [size=4K]
        Region 2: I/O ports at 18800860 [size=32]
        Region 3: Memory at 50013000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0+,D1-,D2+,D3hot+,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: serial


[Nominal Animal]

I'm still just looking the source over, but something poked me in the eye, so although very unlikely (to be a fix), here's my first theory:

The quirk in fixup-rc32434.c disables prefetch and sets PCI cache line size for bus 0, slot 6. It does not have an IRQ, so it is not an active device; perhaps it is the RB502 daughter card with the miniPCI slots?  The quirk looks suspiciously like a lazy dev making a minimal tweak to make their test machine work, seeing as there is no comment explaining the quirk at all.  This makes my paranoia itch.

Granted, the lspci listing shows that the two 4K memory regions on the quad-uart shows that it has those exact settings (prefetch disabled, 32-bit PCI cache lines), so the quirk seems to work (assuming the other quad-UART device has the same memory region features), and I am probably wrong... but I still have that itch.  Best to verify, methinks.

The existence of that quirk is odd to me, considering the fact that it only targets one exact slot in such a modular hardware design.  It is an early fixup too..

Okay, so my first suggestion is to modify the fixup to cover basically all PCI devices.  Essentially, replace arch/mips/pci/fixup-rc32434.c:rc32434_pci_early_fixup() with something like

Code: [Select]

static void rc32434_pci_early_fixup(struct pci_dev *dev)
{
const int  slot = PCI_SLOT(dev->devfn);
const int  bus = dev->bus->number;

/* Force no prefetch and 32-bit cache line size for all external PCI devices.
   The reason for the necessity of this quirk is unknown. */
if ((bus == 0 || bus == 1) && (slot >= 0 && slot < 12)) {
/* disable prefetched memory range */
pci_write_config_word(dev, PCI_PREF_MEMORY_LIMIT, 0);
pci_write_config_word(dev, PCI_PREF_MEMORY_BASE, 0x10);
/* set 32-bit PCI accesses */
pci_write_config_byte(dev, PCI_CACHE_LINE_SIZE, 4);
}
}
I don't recall seeing the entire lspci output, so you might wish to change the bus and slot selection, though.  The idea is to cover all PCI devices outside the processor and PCI bus chip.

Recompile, and stress-test with both quad-UART modules installed to see if it fixes the issue.  If not, revert.

If this did fix the issue (which I only give a small chance), it would indicate a bug/failure/weakness in the RB502 design, or perhaps the PCI bus chip on the main RB532A board.  Perhaps an early quirk is needed to ensure ALL mini-PCI devices on the RB502 won't do PCI prefetches -- remember, the Oxford Semiconductor chip is just a PCI-UART bridge; it is rather obvious why it might go bonkers if prefetches occur, although the 0:a PCI device seems to have the quirked properties  --, or perhaps the entire PCI subsystem on the RB532A tolerates no PCI prefetches.  I think the latter is more likely, and the dev writing the quirk was just too lazy to think about it: the absence of a comment supports this.  Usually devs are rather cross with hardware needing quirks, and like to leave comments explaining why a quirk is needed.  Not having the comment might be an indicative detail.

In particular, if the PCI subsystem hardware architecture is such that PCI prefetches interleaved with driver access can yield the incorrect results, it would certainly explain the random kernel memory corruption.  Prefetches occur very soon before/after an actual/intended access, so such a hardware bug would essentially be like a short race window, during which separate PCI accesses may "swap" their data, or something similar.  Again, I consider this unlikely to yield a fix, but it is certainly something that I would want to exclude before trying anything else.

[legacy]

In particular, if the PCI subsystem hardware architecture is such that PCI prefetches interleaved with driver access can yield the incorrect results, it would certainly explain the random kernel memory corruption. 

Code: [Select]

--- .config     2019-10-30 21:54:05.297488563 -0000
+++ k4.4.197-rb532.config       2019-10-29 00:11:45.102878924 -0000
@@ -1105,7 +1105,7 @@
 # CONFIG_NET_VENDOR_TOSHIBA is not set
 CONFIG_NET_VENDOR_VIA=y
 CONFIG_VIA_RHINE=y
+# CONFIG_VIA_RHINE_MMIO is not set
-CONFIG_VIA_RHINE_MMIO=y
 # CONFIG_VIA_VELOCITY is not set
 # CONFIG_NET_VENDOR_WIZNET is not set
 # CONFIG_FDDI is not set

On our second board under testing, my friend The Halloween unset the MMIO mode of the VIA-Rhine Ethernet modules, and this "seems" to be an improvement: not yet seen any memory corruption, not yet a panic in the last 36 hours, and the machine seems stable without any glitch 

A synthom of memory curruption is perceived as glitch, I mean you suddenly lost the ssh connection to the board, and/or see garbage on the serial console. This usually anticipated a crash.


I will report your message to him, and try to prepare a new patch. Thanks for the hint.

[Nominal Animal]

In particular, if the PCI subsystem hardware architecture is such that PCI prefetches interleaved with driver access can yield the incorrect results, it would certainly explain the random kernel memory corruption.

[Disable VIA Rhine memory-mapped IO]

Yes, that patch is extremely interesting, because it stops the driver from using memory-mapped access to the driver, essentially avoiding the PCI prefetch issues as well.  (They are only done for memory and memory-mapped I/O.)  If you do encounter a crash, my suggested patch is not going to work either, because the error is not caused by PCI accesses.
(Unless, of course, the underlying issue is with memory-mapped I/O accesses only.  Could be, but very, very unlikely.)

If disabling the memory-mapped IO works, I am not sure whether it makes sense to try my patch, unless you can measure a significant drop in throughput with memory-mapped IO disabled.  I believe the difference between the two should be small, since no DMA is involved in any case.

I will report your message to him, and try to prepare a new patch. Thanks for the hint.

Do let us know here after testing.

And, to get back to the topic of this thread, hopefully we are in agreement that it is not exactly the open source software that often sucks, but more that development communities often suck, because people in general often suck.  The hardest part is not the software itself, but dealing with the developers, IMHO.