EEVblog Electronics Community Forum

Products => Computers => Programming => Topic started by: DiTBho on August 22, 2022, 11:56:52 pm

Title: altermatives to vt100? neater protocols?
Post by: DiTBho on August 22, 2022, 11:56:52 pm
so, I put my money into the purchase of several vt-terms, a VT220 and a Digital DEC VT525.

They are nice machines based on 8080 CPUs, they are nice when you just use them with Linux, and they are very useful when you program on the text-console, especially the VT525 with a faster serial.

With VT100, ANSI escape codes for cursor control and other tasks became the de facto standard for hardware video terminals and later terminal emulators, and when you look at the protocol ... well, it's rather patchy, especially if you consider all the DEC addictions.

I'd like to re-design the terminal in modern terms, supporting blocks like IBM 3.27K (but not too IBM-ish) to minimize the number of I/O interrupts required by transferring large blocks of data known as data streams with high an a 48Mbps optical fiber interface, udp/ip over Ethernet connection optionally available.

So, a bit smarter than a dumb-terminal, not so much, but, absolutely of primary importance, the protocol must be designed in a super polished and neat way.

I don't want to see complex ESC sequences with the quality of being so illogical, inconsistent, and unclear, that trying to implement it's fsm in HDL is practically more annoying than heaping and stacking stones in the mine,



Suggestion, advice, all welcome  :D
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 23, 2022, 01:57:14 am
It seems to me that you are conflating two entirely different issues:

1) the commands recognised / sequences sent by special keys, and

2) the communications channel, whether 20 mA current loop, RS-232, SDLC, X.25 etc, synchronous or async, "line mode" or not.

Waaay back 40 years ago when I was using such things, I seem to recall that the VT100 by default interpreted both ANSI ("VT100") commands and VT52 commands in the same mode. You could switch to ANSI-only using I think ESC < and to the mode that did both using ... uhhhh ..  ESC [?2l ??????  I don't think VT100 had a pure VT52 emulation mode, that I can recall.

When we wrote games on the PDP-11 or VAX, we used the VT52 codes to the maximum extent possible, because they are shorter and that made a huge difference at 9600 BPS. Especially the random cursor positioning using ESC Y row col (encoded in binary) was much shorter than the ANSI  ESC [23;75H or whatever.

Regarding communication channel: I remember at some point fairly early on a PDP-11/34 was re-purposed as a "LAT" to manage the async terminals and service their interrupts, and provide the VAX with nice packetised input and output. And to let you connect your terminal to any one of several VAXen too. That protocol is probably documented somewhere.

Yeah.

http://www.bitsavers.org/pdf/dec/ethernet/lat/AA-NL26A-TE_LAT_Specification_Jun89.pdf (http://www.bitsavers.org/pdf/dec/ethernet/lat/AA-NL26A-TE_LAT_Specification_Jun89.pdf)

That's dated 1989 but I was definitely accessing VAXen via this protocol by ... 1983 maybe.
Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on August 23, 2022, 02:06:14 am
Why would you want to torture yourself implementing that in HDL (so I suppose using an FPGA, unless you actually plan on making silicon out of that), when even a small modern MCU would allow you to implement this with ease? Just wondering. (And yes, even with a VGA output without any additional ICs if you're using something as cheap as a RP2040!)

As to the protocol, I've never looked at it closely enough to figure if it was really that bad. But sure if you don't need to be compatible with anything, you can devise a clean protocol, not based on clunky ESC sequences. And if you're targetting much faster interfaces than what was standard at the time, you can encapsulate everything in neat packets with nice headers and CRCs and whatnot.
Title: Re: altermatives to vt100? neater protocols?
Post by: westfw on August 23, 2022, 07:15:47 am
x11?
At one point “X terminals” (tektronix and NCD were two vendors) were a wonderful thing.
They even supported a compressed Async protocol (XRemote) that made them useful over 9600bps dialup (and Cisco supported it in their terminal servers!)
X runs over tcp/ip, of course.


I think Stanford’s “V kernel” did similar things, but didn’t catch on anywhere.  I suspect SUN siphoned away all of the developers.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 23, 2022, 08:59:41 am
x11?

I own an original Tektronix xp-2xx-line with its netboot OS and license  :D

It's MIPS-R3000 based @ 50Mhz with 16Mbyte of fastpage 72pin ram (must be <=50nsec), fully net-booting, VxWorks v5-based, built-in mwm Windows manager, telnet, a built-in superfast Nescape browser and a font client.

First problem: it's pseudo color ---> forget everything GTK-based, only OpenMotif apps will work.
Second problem: only HTML-v1 (1991) and HTML-v2 (1995) are supported ---> forget 90% modern web sites

However, it also adds two serial lines in case you need vt100 an vt200 emulation.

And you don't need any "multi window" extension to operate with two sessions (you need >= vt420 for this), just open two of them, and connect each on a dedicated serial line, you have two physical serial lines, you can have up two sessions.

Do you want more? Use telenet, connect to a trusted telent-to-ssh bridge (WR703? a router?) and you will have many many more.

---

I like it, I like the "thin terminal" concept, and I recently replicated with a modern PPC GNU/Linux + uclibc + nanoX + DWM + (Invisible Island version) xterm solution ...

... just X11 and nanoX are toooooooooo big and complex to be implemented from scratch, and I won't for sure try to build my own.


The funny here is: the PPC@200Mhz with GNU/Linux and X11 is 1.2x slower than X11 on VxWorks on a MIPS-R3000@50Mhz

Kind of LOL
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 23, 2022, 09:14:24 am
when even a small modern MCU would allow you to implement this with ease?

Well, Ania implemented a minimal vt100 in vHDL, and not only is it feasible but it's also cool and neat!

Once defined the new protocol, the plan is to meet Ania to implement new things in HDL.

Kind of interesting collaboration, you shouldn't underestimate this aspect.
Social Engineering is always awesome.

Plus, this project could even win an award for the next 2023 hack camping  :D :D :D
We are talking about a cool premium badge!!! and a case of Bel-cola!!!

- -

All the MPU-solutions I have ever seen around suck. Full of tricks and horrid things.

When people have an MPU, the design tends to be illogical, inconsistent, and unclear, especially for those who suck at software design, and, worse still, have an horrid C-style programming.


Yes, they write "sorry, I will polish and uncrustify in the next version"
Which 99.999999% times means gnegne ... forget it, it will never happen  ;D

It would be interesting to write the whole project in myC, but it only supports MIPS4++ and for sure I won't put a MIPS R18200 into a vt-term ...

... I mean ... it would really be like sending a Grendizer robot to squash mosquitoes  :o :o :o

[attachimg=1]
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 23, 2022, 09:28:35 am
It seems to me that you are conflating two entirely different issues

there are two different needs

VT52, VT100, etc are char-oriented-only, and uses a char-device as communications channel, while I want to support a block-device like 48MBps fiber optic.

The new protocol needs to minimize the amount of data transmitted, and minimize the frequency of interrupts by ensuring the CPU is not interrupted at every keystroke. I won't for sure send one character at a time like in VT100, worse still with a lot of ESC sequences to address row and column, and optionally also the char attributes.
Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on August 23, 2022, 07:23:10 pm
when even a small modern MCU would allow you to implement this with ease?

Well, Ania implemented a minimal vt100 in vHDL, and not only is it feasible but it's also cool and neat!

Well, everyone has a different concept of what is cool, but certainly it IS doable. My point was that it probably made little sense and would be harder to maintain.

All the MPU-solutions I have ever seen around suck. Full of tricks and horrid things.

Maybe, I dunno. Pretty much all commercial terminals have been implemented with MCUs/MPUs. That's certainly a workable approach.
The "let's throw a FPGA at this project and see if it sticks" sounds like a weird engineering approach, but whatever floats your boat.

Just because you have seen sucky implementations doesn't mean that you can't write a clean one (that stamtent sounded a bit like magical thinking), and certainly a VT100 terminal with say VGA output could be neatly implemented with just a RP2040 with even a lot of processing power to spare (yes you can implement a clean VGA output with the PIO). If you needed HDMI output, then sure you'd need additional parts though.

Just a thought. This is ultimately down to your preferences and that's alright. But just don't necessarily convince yourself and others that this is the only right way of doing it. :)
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 23, 2022, 08:15:30 pm
let's throw a FPGA at this project and see if it sticks

no, it's let define a finite state machine and see how we can make the protocol the neatest possible to minimize the finite state machine and get all the constrains satisfied.


Which is better than throw a MPU at this project, write some shitty C code, and see if it sticks
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 23, 2022, 10:49:07 pm
let's throw a FPGA at this project and see if it sticks

no, it's let define a finite state machine and see how we can make the protocol the neatest possible to minimize the finite state machine and get all the constrains satisfied.


Which is better than throw a MPU at this project, write some shitty C code, and see if it sticks

All MPU+program combinations ARE finite state machines.  And there's no reason to write shitty C code if you don't want to. Writing shitty asm code would be a lot easier than creating a huge FSM by hand. Or, you could write non-shitty C or asm code. That would be even better.

It only takes about 200 LUT4s and 200 FF on an iCE40 and one RAM block to make an RV32I soft core that would be 100 times faster than the 8080 in a real terminal. On Xilinx it's about 130 LUTs I think.

Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 12:40:13 am
All MPU+program combinations ARE finite state machines.

A program on a MPU is NOT the minimal finite state machine possible.
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 24, 2022, 01:47:46 am
All MPU+program combinations ARE finite state machines.

A program on a MPU is NOT the minimal finite state machine possible.

You're not going to get the minimal FSM no matter what you do.

Just the 24x80 screen with 8 bit chars and 4 attribute bits is 223040 states already. before you even start counting interpreting incoming control sequences.
Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on August 24, 2022, 03:49:14 am
Not sure I get DiTBho's point exactly - I think we can almost all agree that writing this in HDL will be harder than in software (so will take longer), and since software programming is very often discussed here, I think we can also all agree with the fact we can write decent and robust software. Especially for something like this which is not rocket science.

But as I said, whatever floats everyone's boat.

As I mentioned, I'd find this fun to implement on a RP2040, supporting multiple interfaces and using the PIO for a VGA (or similar) output. But if someone is more comfortable doing this in HDL, why not. I'm just pretty sure it will take more time.

Now one could argue that implementing it in HDL might have more chances of being more robust - if just because you need to be extra careful when using an HDL, and you also are more likely to test your code using test benches and simulation than you are for pure software - in that sense, HDLs may be better to force you to do things right. But that's also debatable.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 11:09:44 am
Just the 24x80 screen with 8 bit chars and 4 attribute bits is 223040 states already. before you even start counting interpreting incoming control sequences.

Not the way you interpret it! You are not working at gate level but rather above the rtl level, you have vHDL and you can describe a finite state machines in the VDU stage that perfectly knows the concept of "row" and "column" and EA to correctly address the video ram  :D

In Ania's vt100-lapop you have hardware two-fish encryption directly connected to the optical fiber circuits (Manchester encoded) and to the VDU.

What arrives from the optical cable enters the protocol link engines, gets accepted or rejected (corrupted, go back n), gets payload extracted, enters the encrypt unit, gets decrypted, enters the VT100 engine (minimal set), gets interpreted, decomposed to operations for the VDU unit which address character to the video ram, and, on refresh event, you see it on the LCD.

Very clean! No software on no Softcore implemented, all modules in pipelining, 100% vHDL

Ania showed that all these units work in parallel and there are many bypassing circuits that would consume many more resources if implemented in software on general purpose CPUs, and commented that it's not because CPUs suck but rather because they are *generic* and filled with other things like pipelining, caching, etc. to go faster. Also, parallelism on a single core needs interrupts at least and context save/restore, which ... is good but not so much when compared to modules hardware pipeling.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 12:05:09 pm
Meanwhile, talking about "multiprocessor programming"

I just ordered this book
                    The Art of Multiprocessor Programming, Revised Reprint
                    by Maurice Herlihy, Nir Shavit
Title: Re: altermatives to vt100? neater protocols?
Post by: rstofer on August 24, 2022, 05:29:49 pm
The new protocol needs to minimize the amount of data transmitted, and minimize the frequency of interrupts by ensuring the CPU is not interrupted at every keystroke. I won't for sure send one character at a time like in VT100, worse still with a lot of ESC sequences to address row and column, and optionally also the char attributes.

Actual keystroke interrupts occur at something less than 10 per second.  100 ms is a very long time.  A 600 MHz Teensy 4.1 won't even notice the handler running.  ARM Cortex M7 https://www.pjrc.com/store/teensy41.html (https://www.pjrc.com/store/teensy41.html)

Telnet usually operates one character at a time (Character Mode) unless Line Mode is negotiated.  Operator input obviously uses character mode since a single character response is often used (thinking of the response to 'y/N').  I don't see how you get around one character at a time when the message is only one character long.  In terms of keyboard input, you have, at the driver level, no idea how long the message will be.

There's a reason standards survive.  They are useful and very nearly optimal in their application.

Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on August 24, 2022, 07:02:34 pm
To each their own really! That's all we can say. But all this can clearly be written cleanly in pure software on a modest MCU. Only the encryption part may require a beefier one, but even this is probably not that much of a problem due to the typical data throughput of a text-based terminal. For a graphics terminal, things would be another story.
Title: Re: altermatives to vt100? neater protocols?
Post by: Jr460 on August 24, 2022, 08:08:27 pm
I'm really not getting what you are trying to do and why, but I'll try to play along.

In terms of less interrupts, while that is comm/serial controller's problem on the computer side, not the terminal's problem.   Someone said LAT, well that was just the transport between the CPU and and the terminal server over Ethernet.   You still came out of the terminal server at 19.2k bps or so to a terminal.   LAT is going to wait for data until it has more to fill up an ethernet packet.   I think the timer was something like 80 milliseconds, and then it had to send what not already had.   The idea was to keep the delay below the time that the human eye / brain could detect the delay.

How about this, go find an old VT1000.   That is not a typo.   I tw as monochrome X-term, so it had builtin ethernet, was based on on a 68K processor.   And if you didn't want to do full blow X, you could pop open a bunch of vt100 compatible windows that natively did LAT over the network.   Problem solved.

As far as moving less data, look at the differences between VT100, 101, 102, 200 and 220, 240, 300 code.   The later terminals allowed for 8 bit escape sequences which shorten many commands, also local line editing,, a character could be added or deleted mid line without re-painting the whole line.   Also scrolling regions on the screen a set of lines or a block.   If your program uses the right commands to the terminal, you can save a lot of output to refresh a screen.


Or go real old school.   have the terminal "screen" as part of system memoryand the video generator just cycle steals or does DMA from the screen buffer.   I had an old S100 based board that did this.   And also had a ROM on it that could be called passing it data, and it would emulate a vt100 and write to memory on the card for you.

How fast do you really need data to come to fill a screen that is 80x24.   What serial speed does it take to write the whole screen in under 80 ms?   And faster than that and you will not see the difference.   If you are blasting lines to screen just for it to scroll by and that is the delay in your program, then I'd say fix you program to only put out what need to see at the end.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 09:09:57 pm
Actual keystroke interrupts

yeah, not the best example, except when a keystoke causes a full page update, e.g. software vertical or horizontal scrolling.

You have them with a text-editor

let's say, worst case, you have to update the whole screen, 80x25x2 (1 char, 1 meta), one page is 4K byte of ram(1), 4kbyte/s.


but here you may want larger screen, and also transport other stuff, so you want to compress and maximize the payload and get rid of all the extra things you have to transport for your terminal


The vt100-procol is ASCII based, not good for four points

You appreciate the last point especially with IBM 3.2K terminals, which are block-oriented, and whole page updating is one shot!



(1) Ania's VDU text ram is 32Kbyte at the moment. It's a true dual port ram.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 09:14:24 pm
In terms of less interrupts, while that is comm/serial controller's problem on the computer side, not the terminal's problem

yes, but starting from scratch I want to solve this too since who knows? I will probably use the terminal with super fast GNU/Linux MIPS64 board @ 1.4Ghz, perhaps with a 68hc11 @ 4Mhz ...

let's make life easy on the server side too, if possible  :D
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 09:17:07 pm
Someone said LAT, well that was just the transport between the CPU and and the terminal server over Ethernet.   You still came out of the terminal server at 19.2k bps or so to a terminal.   LAT is going to wait for data until it has more to fill up an ethernet packet.   I think the timer was something like 80 milliseconds, and then it had to send what not already had.   The idea was to keep the delay below the time that the human eye / brain could detect the delay.

Yup, I bought a LAT terminal, it's here in my lab. I'd like to *copy* the best form it  ;D
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 24, 2022, 09:34:27 pm
As far as moving less data, look at the differences between VT100, 101, 102, 200 and 220, 240, 300 code.   The later terminals allowed for 8 bit escape sequences which shorten many commands, also local line editing,, a character could be added or deleted mid line without re-painting the whole line.   Also scrolling regions on the screen a set of lines or a block.   If your program uses the right commands to the terminal, you can save a lot of output to refresh a screen.

Yeah, and it's not supported by Ania's vt100-hdl, they are extensions, while Ania's one is a very basic vt100, like the first model appeared for the market.

Nice feature addons, I can play with them with my VT525 and see how they are pretty and nice ... just, well they made the vt-protocol more spotted than a cow, and fist thing you have to do is probing your vt-term "which version are you?" and "which feature o you support?"

flexible for different products that evolve year by year, but not good when you try to synthesize the minimal protocol for full features.
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 24, 2022, 10:38:17 pm
The new protocol needs to minimize the amount of data transmitted, and minimize the frequency of interrupts by ensuring the CPU is not interrupted at every keystroke. I won't for sure send one character at a time like in VT100, worse still with a lot of ESC sequences to address row and column, and optionally also the char attributes.

Actual keystroke interrupts occur at something less than 10 per second.  100 ms is a very long time.  A 600 MHz Teensy 4.1 won't even notice the handler running.  ARM Cortex M7 https://www.pjrc.com/store/teensy41.html (https://www.pjrc.com/store/teensy41.html)

Well, yeah, but we used to have 50+ VT100s attached to a 1 MIPS VAX 11/780.  600 MHz Teensy is about 1000 MIPS. So the relative interrupt load on the VAX was something like a factor of 50,000 higher. That's why they started using old PDP-11s as terminal servers, to take that interrupt load off the VAX.
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 24, 2022, 10:56:26 pm
Nice feature addons, I can play with them with my VT525 and see how they are pretty and nice ... just, well they made the vt-protocol more spotted than a cow, and fist thing you have to do is probing your vt-term "which version are you?" and "which feature o you support?"

You haven't lived until you've programmed graphics using ReGIS / GiGi.

Code: [Select]
<ESC>P0p
S(E)(C1)
P[100,440]
V(B),[+100,+0],[+0,-10],[-100,+0],(E)
P[500,300],F(C[+100])
<ESC>\

That clears the screen then draws an outlined 100x10 rectangle, then a filled circle with radius 100.
Title: Re: altermatives to vt100? neater protocols?
Post by: Jr460 on August 24, 2022, 10:59:49 pm
Nice feature addons, I can play with them with my VT525 and see how they are pretty and nice ... just, well they made the vt-protocol more spotted than a cow, and fist thing you have to do is probing your vt-term "which version are you?" and "which feature o you support?"

You haven't lived until you've programmed graphics using ReGIS / GiGi.

Code: [Select]
<ESC>P0p
S(E)(C1)
P[100,440]
V(B),[+100,+0],[+0,-10],[-100,+0],(E)
P[500,300],F(C[+100])
<ESC>\

That clears the screen then draws an outlined 100x10 rectangle, then a filled circle with radius 100.

Yep loved doing things with those graphics.   Wrote tuns of stuff to take files in HP-GL or Tex 41xx format and output it as Regis.   
Title: Re: altermatives to vt100? neater protocols?
Post by: Jr460 on August 24, 2022, 11:08:21 pm
The new protocol needs to minimize the amount of data transmitted, and minimize the frequency of interrupts by ensuring the CPU is not interrupted at every keystroke. I won't for sure send one character at a time like in VT100, worse still with a lot of ESC sequences to address row and column, and optionally also the char attributes.

Actual keystroke interrupts occur at something less than 10 per second.  100 ms is a very long time.  A 600 MHz Teensy 4.1 won't even notice the handler running.  ARM Cortex M7 https://www.pjrc.com/store/teensy41.html (https://www.pjrc.com/store/teensy41.html)

Well, yeah, but we used to have 50+ VT100s attached to a 1 MIPS VAX 11/780.  600 MHz Teensy is about 1000 MIPS. So the relative interrupt load on the VAX was something like a factor of 50,000 higher. That's why they started using old PDP-11s as terminal servers, to take that interrupt load off the VAX.

First you needed a better serial line controller, DMZ rather than a DMF, that will cut down on interrupts. 

I never saw the TT/TX driver or such imparting much load on a 785 or 8600.   Also had people dial in at 19.2K and use Kermit to move data from mobile test stations why out in the great white north where they were luck to have a phone line.   If you had the SPM+ package you could profile the time the CPU spent running code in system space.   The TX/TT drivers never showed much of a load.   Cluster comms and MSCP to HSCs, yea that could make a bump.   Worst thing for interrupt time was page faults.   Keep those under control and the machine ran great.
Title: Re: altermatives to vt100? neater protocols?
Post by: IanB on August 24, 2022, 11:17:32 pm
Back in the 80's I programmed a whole set of complex colour text/graphics displays using a combination of ANSI and Tek 4014 graphics commands over a 19200 serial line running on a VAX minicomputer. I was able to get pretty good performance with nearly instant screen refresh. The key to it was to create an abstract graphics library to hide away the escape sequences and optimize the data stream sent to the terminal.

VMS also had various WYSIWYG text editors running on VT terminals, such as EDT or TPU.

One thing you would never do for good performance was to send one character at a time. You would always set up the terminal and then send a block of data.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 25, 2022, 12:34:49 am
Unfortunately impossible but it would be super cool... supporting svg in hardware.
It can be limited to primitives, lines, arc, circles, ... Nothing more.
Forget more complex curves.

Anyway, enought to graph math 2D functions, I mean like a graphing calculator, which is something I'd like to have.

Otherwise you can transport bitmaps. Ania has her personal logo encoded into the terminal, when she powers on and inserts the phassprede, the terminal shows a tiny image at the center of the screen.

Nice bitmap ram, pre initialized Bram, it can be used to show things in the background.






Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 25, 2022, 12:39:24 am
The key to it was to create an abstract graphics library to hide away the escape sequences and optimize the data stream sent to the terminal.

Cleever idea, now I want an ESC compositor library  ;D
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 25, 2022, 12:52:53 am
VMS also had various WYSIWYG text editors running on VT terminals, such as EDT or TPU.

One thing you would never do for good performance was to send one character at a time. You would always set up the terminal and then send a block of data.

Yup!

Years ago I wrote a text-editor. It runs on Linux and its interface uses ncurses.
I think I will change its interface, so it will be the first application for the new-terminal protocol.
It can be the ideal stressing-test application.

Search, Go-To-Page, etc, are good examples of why you should send a block of data  :D
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 25, 2022, 01:10:09 am
The iconic DEC VT100 has an optional video card called "Selanar".
It's as big as the motherboard, and full of chips  :o :o :o
Probably it can help with graphics, but I don't exactly how much  :-//

[attachimg=1]
[attachimg=2]
[attachimg=3]

Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on August 25, 2022, 01:37:50 am
Well sure, with TTL logic of the 70s. The same kind of logic could be implemented with probably a few tens of LUTs on a very small FPGA.
And as I said, even a $1 RP2040 with its PIO used cleverly can get you much, much better graphics than this thing ever could. ;D

https://www.youtube.com/watch?v=KSYjGul84aU (https://www.youtube.com/watch?v=KSYjGul84aU)
Title: Re: altermatives to vt100? neater protocols?
Post by: westfw on August 25, 2022, 07:48:47 am
Tn3270?  Telnet linemode option?


I don’t understand what you plan to have this “improved” protocol talk to, since it’s been a long time since most operating systems have had anything other than a character-oriented “console” interface.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 25, 2022, 07:53:29 am
as I said, even a $1 RP2040 with its PIO used cleverly can get you much, much better graphics than this thing ever could

Nice on YouTube, then when you look at sources ... meh, C && cleverly use don't match with neat, especially with OpenSource.

Technically I could even use the Timer Processing Unit of machines that have it aboard to implement softVGA@pixelclock 25Mhz  :o :o :o

Already done for spi, i2c, uart, PWM, etc, and it's not bit-banging, it's a set of dirty tricks on the cleverly use of a special piece of hardware called TPU that can be programmed to control PIOs, so nothing special under the bridge for me, I heard about it in 2001, and read lot lot of pages of manuals, experimented lot of compromises, spent lot of weeks to master the new toy.

The $ 1 RP2040 probably means you'd better spend your weeks not uncrustifying other people's things than getting too bored of wasting time on unclear manuals ;D

For me HDL is the neatest solution here. No tricks. Just modules.

You want to change the PHY? You don't need to hammer your head on the wall to get enlightened for new tricks to cleverly exploit the PIO in a different way, just change a module.

I remember ... with Motorola' TPU, there was a manual of 500 pages, and still today there are some so unclear things that in going from PWM to sigma delta TPU channels ... similar functionality, just a bit different timing, and I haven't yet understood what I did, and why? it works  :o :o :o
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 25, 2022, 08:29:51 am
I don’t understand what you plan to have this “improved” protocol talk to

good point, especially for Linux ;D

It doesn't need to support already existing stuff, like telnet.
Likely, it will be this way:

Protocol development and initial testing
Code: [Select]
.    server side                                          client side
 _________________                                    _________________
|                 |                                  |                 |
|    Linux/R64    |                                  |    Linux/R64    |
|    arm32/le     |                                  |    arm32/le     |
|                 |                                  |                 |
|      myTE       |                                  |      Hterm      |
|                 |                                  |     ncurses     |
|_________________|                                  |_________________|
         \____________________________________________________/
                               optical fiber
Optical fiber here is achieved with a miniPCIe to PCI bridge + PCI optic adapter.
R64 is a router based on MediaTek SoC. Two miniPCIe lanes.
I already have to support it, I take the opportunity.

Hterm here will work similarly to "minicom" or "screen", with its native support for ncurses so I won't need to write that stuff.

Final application
Code: [Select]
.    server side                                          client side
 _________________                                    _________________
|                 |                                  |                 |
|    Linux/R64    |                                  |       fpga      |
|    arm32/le     |                                  |                 |
|                 |                                  |                 |
|      myTE       |                                  |      Hterm      |
|                 |                                  |     VGA/ps2     |
|_________________|                                  |_________________|
         \____________________________________________________/
                               optical fiber
Optical fiber here is achieved with a dedicated module.
Ania already wrote the vHDL driver, so I take the opportunity.


myTE will be the first application exploiting Hterm, it's my text editor.
Then I will write a remote-shell server for Hterm.
Kind of stdin/out/err handler for Bash, but it will be only usable to issue line commands and see their line responses.

forget things like htop and cmatrix for now.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 26, 2022, 01:50:39 pm
Today I am looking at tekonivel's DEC VT420 over a null-modem cable as Linux terminal.

The VT420 terminal is used in VT220 emulation mode, because the “vt420” termcap entry doesn’t happen to be installed on my machine, anyway it's is used in that mode with 132 columns and 48 rows, page-length also set to 48 to get full-screen pages, and the cable is simple, and doesn’t provide hardware flow-control, and it works *just fine* for bash and text-line programs, but the speed is set quite low, to 9600bps, for text-editor, so in practice forget Nano, forget Joe, forget Vim+NerdTree, only Emacs is decently usable since it wisely updates only the parts of screen that have changed.

That's interesting :D
Title: Re: altermatives to vt100? neater protocols?
Post by: IanB on August 26, 2022, 06:37:51 pm
the cable is simple, and doesn’t provide hardware flow-control

What about ^S/^Q flow control? You always needed that enabled in my experience for reliable operation.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 26, 2022, 07:12:04 pm
the cable is simple, and doesn’t provide hardware flow-control

What about ^S/^Q flow control? You always needed that enabled in my experience for reliable operation.

Good idea! soft flow control! Yup, the vtern supports it, thanks
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 27, 2022, 12:59:15 am
Modifications for hterm: get rid of terminfo and termcap

The purpose of a terminfo and termcap files (one file for each terminal/mode) is to let programs (e.g. Nano, Vi, Vim, Emacs, Joe, htop, bmon, etc ...) running in the terminal know the capabilities of the terminal.

Code: [Select]
toe -a
will show many files, one for each supported terminal/mode

Code: [Select]
toe - table of (terminfo) entries

SYNOPSIS
       toe [-v[n]] [-ahuUV] file...

DESCRIPTION
       With  no  options,  toe  lists  all available terminal types by primary name with descriptions.  File
       arguments specify the directories to be scanned; if no such arguments are given,  your  default  ter-
       minfo  directory is scanned.  If you also specify the -h option, a directory header will be issued as
       each directory is

       There are other options intended for use by terminfo file maintainers:

       -a     report on all of the terminal databases which ncurses would search, rather than only the first
              one that it finds.


Termcap and terminfo, basically answer questions like:

- Does it support colors? How many?
- How many column? How many rows?
- How special keys are mapped?
etc...

A VT100 terminfo should make programs assume an 80 column monotone screen and a keyboard with 10 function keys

VT100 compliance is why the F11 .. F12 keys are traditionally not used by console programs, which is really annoying when programs using "ten" function-keys (like mc) on a terminal lacking ten usable function-keys and YOU have to accept an alternate such as escape-0, escape-1, etc because there's no intuitive "escape-11".

- no way, this has to go! -

The protocol of Hterm is fully auto-probing. When a program needs to know something it has to ask directly to the terminal instead of looking for some file in /etc/term... or /usr/share/term...

And you don't have to compile any terminfo file with program like "Tic"  :D
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 30, 2022, 07:25:46 am
Compression

So, yesterday I was considering string compression for large cluster of chars, which can be useful during page_update or page_scroll to minimize the effective number of data to be transmitted.

Hterm supports this kind of things  :o

Ania has designed a very simple, very small and self-contained string compression algorithm, Zeda(1) has provided us with a dictionary which contains what she thinks is the best combination of two letters used in English, and we analyzed how the compression ratio behaves with real text pages.

Ratio = length(compressed string) / length(original string) * 100

Average Ratio: 59 of 100
Best Ratio: 55 of 100 <------------ theoretically it is 50 of 100
Worst Ratio 79 of 100 <----------- theoretically it can be 100 of 100, ZERO compression
 
not as good as the compression you will get out of a zip file or something similar to that, the maximum compression that can possibly be achieved is 50%, and her classmate Zeda said that, depending on the *nature* of what you are compressing, you may find that a different dictionary that may give better performance.

The dictionary Zeda gave us seems to work reasonably well with pages of "C source" text and pages of "English articles".


just ... is string-compression worth it on a 48 Mbps link? big question :o :o :o
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 30, 2022, 08:10:07 am
just ... is string-compression worth it on a 48 Mbps link? big question :o :o :o

48 Mbps? 6 MB/s?

lz4 with default settings gets 2.1:1 compression on the "Silesa Corpus" at a speed of 625 MB/s on one core of an i7-3930K @ 4.5 GHz.

That's on a 2011 CPU that does memcpy() at 7300 MB/s.

If you only need 1.8:1 compression it will do that at 911 MB/s.

If you want 2.7:1 compression (-9) that slows lz4 down to compressing only 34 MB/s. At that point you're better off using zstd -3 which gets 3.16:1 compression at 200 MB/s.

lz4 decompression is at over 3200 MB/s on the same CPU, regardless.


lz4 is a small and simple algorithm. It's hard to think of a reason to NOT give it a quick run over data rather than sending raw data.

I don't have figures to hand for 2022 CPUs, but they will be better.
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 30, 2022, 08:35:27 am
So, I just tried on my 2020 M1 Mac Mini, using lz4 from homebrew

211938580 Silesa corpus total size
100934427 lz4 default, saved 52.37% in 0.402s CPU = 502 MB/s.

oh.

Oops. That was an x86_64 lz4. So that was being emulated.

Let me build a native one.

ok, now it's 0.378s CPU time, identical output size. So that's 534 MB/s.

Your comms link was 6 MB/s, right?
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 30, 2022, 11:31:57 am
48 Mbps? 6 MB/s?

yup, and it's 5Mbyte/s payload due to transport overhead and synchronization.

lz4 is a small and simple algorithm. It's hard to think of a reason to NOT give it a quick run over data rather than sending raw data.

yup, but it needs to be implemented in hardware, and it should be simple and small.

lz4 is very nice, more complex than our simple string compressor, but more performative
Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on August 30, 2022, 07:20:35 pm
I have reimplemented this one (in a cleaner way) based on LZ77: https://github.com/banebyte5115/xlz
It's very fast (faster than LZ4) and gives good compression ratios.

Implementing it in HDL shouldn't be too hard, it's a relatively simple algorithm. I might do it one of these days.
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 30, 2022, 09:36:31 pm
lz4 is a small and simple algorithm. It's hard to think of a reason to NOT give it a quick run over data rather than sending raw data.

yup, but it needs to be implemented in hardware, and it should be simple and small.

If it's implemented in hardware then it's obviously going to be much faster. If you can do VT100 in pure state machine hardware without a CPU then lz4 is a doddle.

After all, lz77 and lz78 were originally implement in hardware, in IBM disk drives.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 31, 2022, 12:54:42 am
If you can do VT100 in pure state machine hardware without a CPU then lz4 is a doddle.

yup, so Hterm will have the choice to negotiate the compression.

query, do you have Zeda's compression? yes/no          (here we need to find a nice name for her algorithm)
query, do you have lz4 compression? yes/no

none of them means -> no compression
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 31, 2022, 10:40:22 am
Code: [Select]
msg Original=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]
msg Restored=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]
     Original msg Length=122
   Compressed msg Length=74
     Restored msg Length=122
      Compression  Ratio=60 of 100
(Zeda's algorithm v1, dictionary ULK-11)

Code: [Select]
msg Original=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]
msg Restored=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]
     Original msg Length=123
   Compressed msg Length=100
     Restored msg Length=123
      Compression  Ratio=81 of 100
(xlz algorithm)


Ummm, the above is just a short example-string, I have to run more real full screen-page and check average compression rate, min, max, but it seems Zeda's wins for simplicity.

No doubt about.

Then we also have binary-pages, containing meta attributes for each char.
Color foreground
Color background
Blinking?
Underlined?
etc
Title: Re: altermatives to vt100? neater protocols?
Post by: brucehoult on August 31, 2022, 11:06:27 am
Your "Zeda's algorithm" is using a pre-loaded dictionary which must be previously shared/transmitted between the two ends.

Other algorithms are starting from zero shared knowledge.

Other algorithms (e.g. lz4 "-D file" flag) can also be initialised with a dictionary known to both sender and receiver, and will obviously do much better on short texts if they are.

For example if I save this entire discussion thread as a .html file (193793 bytes, at present), and then use it as a dictionary for lz4, then your 122 byte message is compressed to 63 bytes with default settings, or 29 bytes with "-9".

Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on August 31, 2022, 10:20:12 pm
Your "Zeda's algorithm" is using a pre-loaded dictionary which must be previously shared/transmitted between the two ends.

Other algorithms are starting from zero shared knowledge.

Zeda is Ania's classmate. She decided to join our hacker team, so now I have a team of two girls, like James Bond with a blonde and a brunette  :o :o :o

Yes, her algorithm needs a pre-loaded dictionary 320 byte sized.
It should be loaded into BRAM.

It's designed to map this charset
Code: [Select]
charset
{
      ' ','!','"','#','$','%','&',''','(',')','*','+',',','-','.','/',
      '0','1','2','3','4','5','6','7','8','9',':',';','<','=','>','?',
      '@','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O',
      'P','Q','R','S','T','U','V','W','X','Y','Z','[','\',']','^','_',
      '`','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o',
      'p','q','r','s','t','u','v','w','x','y','z','{','|','}','~','\n'
};
So, she reduced a subset of ASCII-8bit into 96 possibilities, and '\0' is not included.


I grabbed this topic' page to make a test with a bigger stream (16Kbyte):

Code: [Select]
   Compressed msg Length=11074
     Original msg Length=16804
     Restored msg Length=16804
      Compression  Ratio=65 of 100
(Zeda's algorithm)

Code: [Select]
   Compressed msg Length=9470
     Original msg Length=16804
     Restored msg Length=16804
      Compression  Ratio=56 of 100
(xlz)

Not bad, considering its simplicity  :o :o :o

I will also implement xlz  ;D
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on September 01, 2022, 12:22:57 am
with the same testing file

Code: [Select]
./lz77 -c -i guinea-pig.txt -o guinea-pig.lz -s 2048
lsprettysize guinea-pig.*
       10. 4 Kbyte guinea-pig.lz
       16. 8 Kbyte guinea-pig.txt
(lz77, Git Repo here (https://github.com/cstdvd/lz77))
Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on September 01, 2022, 02:03:17 am
(...)
Code: [Select]
   Compressed msg Length=9470
     Original msg Length=16804
     Restored msg Length=16804
      Compression  Ratio=56 of 100
(xlz)

Not bad, considering its simplicity  :o :o :o

Yes! I was also impressed when I tried it. Pretty cool.
Title: Re: altermatives to vt100? neater protocols?
Post by: westfw on September 04, 2022, 01:28:18 am
I don't think I understand the point of speeding up an interface/protol that already goes faster than the data can be displayed on most displays, and MUCH faster than it can be consumed by a user...

I guess it's fun.  But...
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on September 04, 2022, 02:34:33 am
I don't think I understand the point of speeding up an interface/protol that already goes faster than the data can be displayed on most displays, and MUCH faster than it can be consumed by a user...

It also needs to transport other stuff.
Including bitmaps.
Title: Re: altermatives to vt100? neater protocols?
Post by: DiTBho on September 05, 2022, 04:08:09 pm
I don't think I understand the point of speeding up an interface/protol that already goes faster than the data can be displayed on most displays, and MUCH faster than it can be consumed by a user...

I guess it's fun.  But...

reason2, today a new member joined the team and proposed to make a vintage CPU version of hterm.
His hardware has 10Mbit/s ethernet, hence it will use a subset of the hterm protocol and see built-in compression helps.
Title: Re: altermatives to vt100? neater protocols?
Post by: SiliconWizard on September 05, 2022, 08:32:26 pm
Odd vintage world. ;D