The Imperium programming language - IPL

[Sherlock Holmes]

As part of the rationalization of the language, I've replaced "auto" / "automatic" with "stack", were all grown ups, time we stopped using these arcane terms.

Hardly anyone uses 'auto' in C, and in C++11 and later it is a placeholder type specifier, meaning that the declared variable derives its type from the initializer (e.g. 'auto f = 3.0;' declares variable 'f' with type 'double').

Good Lord! really? I guess it's so hard to add new keywords to the language that it's now become necessary to give the same keyword different meanings in different contexts!

[cfbsoftware]

Here's some actual real life CORAL 66 source, it's very odd that the language seems to need single quotes around keywords, never seen that before, I wonder why they did that...

Another way to differentiate between keywords and other identifiers to allow new keywords to be added later without causing problems?

I've seen this in some Algol 60 programs, also discussed here:

http://cs.ecs.baylor.edu/~maurer/SieveE/Algol60.htm

Some compilers forced programmers to put all keywords in quotes, which was permitted by the language definition.

[Sherlock Holmes]

Here's some actual real life CORAL 66 source, it's very odd that the language seems to need single quotes around keywords, never seen that before, I wonder why they did that...

Another way to differentiate between keywords and other identifiers to allow new keywords to be added later without causing problems?

I've seen this in some Algol 60 programs, also discussed here:

http://cs.ecs.baylor.edu/~maurer/SieveE/Algol60.htm

Some compilers forced programmers to put all keywords in quotes, which was permitted by the language definition.

Thanks, that was an interesting little page. I do have some faint recollection of the quotes around keywords, very distant memory but where, why and what I have no idea, but I do think I encountered it somewhere once... I suspect you're right, it was a way of allowing new ones without breaking old source code.

Those early languages and compilers deserve a lot of respect, even sitting at a terminal was a rare privilege for most programmers in the 1960s.  The guys who wrote PL/I were kind of forced to write the compiler mostly in assembler. IBM were not interested in stuff that they didn't own, so using Algol to write it was not an option, that kind of closed mindedness is mainly why PL/I never became more popular or more established outside of IBM environments, but that's not the only mistake IBM made as history shows!

[Sherlock Holmes]

Well it's been two months pretty much to the day since this thread was started. A lot has happened, the language has a name "IPL", the grammar is now very solid and stable and arbitrarily complex - legal - source code can be parsed reliably and a parse tree created.

The adaptation from a base PL/I syntax has gone rather well, and the language now supports name spaces, internationalization, type definitions, multiline string literals, enums, free form numeric literals in multiple bases and embedded spaces, new label syntax, Unicode, various shift related operators and more.

The current state of play is summarized here for those interested.

There are several posters here who have been truly helpful and constructive, and I'd like to thank them for their valuable insights and ideas.

I'm stepping back from this work for a time now, but will be actively monitoring the thread, thanks to all.

[JPortici]

As part of the rationalization of the language, I've replaced "auto" / "automatic" with "stack", were all grown ups, time we stopped using these arcane terms.

Hardly anyone uses 'auto' in C, and in C++11 and later it is a placeholder type specifier, meaning that the declared variable derives its type from the initializer (e.g. 'auto f = 3.0;' declares variable 'f' with type 'double').

Good Lord! really? I guess it's so hard to add new keywords to the language that it's now become necessary to give the same keyword different meanings in different contexts!

no it's because a local variable that isn't "static" is implicitly "auto", the same way it's redundant to add "extern" to a definition in a header file
Incidentally i was re-reading "Expert C programming" yesterday, there they explain how auto is actually from when they thought a developer would decide between "auto"matic storage (on the stack) or "register" storage (on of course a register)
soon they realized that compilers produced better code if neither would be used.

I guess in C++ they decided to reuse a word that was already reserved, but abandoned

[westfw]

the language has a name "IPL", the grammar is now very solid and stable and arbitrarily complex - legal - source code can be parsed reliably and a parse tree created ...
...
I'm stepping back from this work for a time now...

This is the sort of thing that drove me crazy back in my college "compilers" class.
I learned a hell of a lot about lexical analysis, parsing, and grammars, but the class essentially STOPPED and hand-waved away the actual code-generation (including any optimization steps that might or might not occur.)  At the time I was a heavy-duty assembly-language programmer, and most of what I cared about was whether a compiler could come close to duplicating the efficiency of code I could write in assembly.

So I used to think that C owed its success and longevity to the small footprint of the require "runtime environment."  But now I want to add that "code generation" for C code is "obvious."  You can look at C code and picture what the code generated ought to look like.  Perhaps the compiler optimizes that weirdly, perhaps not.  Maybe this is what people OUGHT to mean when they say "C is just a high level assembler."
As languages get more complex, the code generation gets considerably LESS obvious.
I always figured that the failure of PL/1, even with IBM behind it, was that the compiler ended up being too complicated to port (in a meaningful way) to new architectures.  I don't know if that was run-time environment related, code generation related, compiler size related, or some combination.

So - how is IPL doing in the code generation department?  What's the first target?

[ataradov]

This is how 99% of "I want to make my own compiler" threads end up being. Grammar is like 1% of the work and the easiest part at that.

I can sort of understand this for the education though. Code gen past the primitive stack machine with AST traversal and direct generation is really hard work, you would have to dedicate multiple semesters to that, and nobody has time for that.

And a lot of this work is not purely algorithmic as academics like, but more practical. There is no real theories, you just need to sit down and code. And the code gets messy and hard. This is the part where most new languages give up.

[Sherlock Holmes]

the language has a name "IPL", the grammar is now very solid and stable and arbitrarily complex - legal - source code can be parsed reliably and a parse tree created ...
...
I'm stepping back from this work for a time now...

This is the sort of thing that drove me crazy back in my college "compilers" class.
I learned a hell of a lot about lexical analysis, parsing, and grammars, but the class essentially STOPPED and hand-waved away the actual code-generation (including any optimization steps that might or might not occur.)  At the time I was a heavy-duty assembly-language programmer, and most of what I cared about was whether a compiler could come close to duplicating the efficiency of code I could write in assembly.

So I used to think that C owed its success and longevity to the small footprint of the require "runtime environment."  But now I want to add that "code generation" for C code is "obvious."  You can look at C code and picture what the code generated ought to look like.  Perhaps the compiler optimizes that weirdly, perhaps not.  Maybe this is what people OUGHT to mean when they say "C is just a high level assembler."
As languages get more complex, the code generation gets considerably LESS obvious.
I always figured that the failure of PL/1, even with IBM behind it, was that the compiler ended up being too complicated to port (in a meaningful way) to new architectures.  I don't know if that was run-time environment related, code generation related, compiler size related, or some combination.

So - how is IPL doing in the code generation department?  What's the first target?

Hi,

I want to recommend a great (if a little dated) book on compilers. It was the best of five books I used in the 90s for compiler implementation and explained the code generation far better than any of the other books.

Here's the PDF, I have a paper back copy, you can find them around if you prefer that (I do).

The writer understands hardware, assembler and I suspect you'd like the book, it was like a missing piece of the puzzle for me back then, frankly I learned more about compilers from that book than I did from the "Dragon" or the (huge) "Compiler Design in C".

These were all good books but the Bornat book showed the author's passion for the subject, he knew his stuff (it contains one if the very few, earliest discussions of coroutines).

I also tend to agree, the gulf between the front end of a compiler and the raw code bytes that get generated is covered poorly in the literature (when compared to parsing etc) and this was a big struggle for me, Bornat's book was a breath of fresh air frankly.

Yes PL/I faded because of several reasons, one of them was absolutely the complexity of the compiler. Most of the IBM compiler was written in assembler, by very experienced mainframe assembler gurus. It was regarded (in the literature) as the most complex language compiler ever produced.

Even today with the powerful tools we have, the "full IBM PL/I" language is daunting, it included real multitasking and I/O as an integral part of the language.

But it also faded because IBM were IBM, there was zero interest from them in sharing a language or agreeing to standardize it, it was - like much of IBM's products - proprietary, a means to and end, dominate the computer market basically.

For various marketing reasons it was used far more in Europe, particularly Eastern Europe, than it was in the US.

So you're right it was as non-portable as it could get! But it did catch the eye of various software engineers at the time and steps were taken to try and standardize it because many of the concepts it introduced were original and the language had clearly been truly designed, with a list of requirements that were addressed professionally over many months by a host of participants.

But enough of that! As for IPL, the project (if I can call it that) is still in its infancy, the code generation aspects are something I am looking forward too. There's a lot out there that did not exist in the 1990s when I created my own PL/I implementation. It seems to me that LLVM is where the action will be. That offers a powerful abstract CPU that one can generate code for and then, leverage LLVM code generators that can translate that abstract "assembler" to very specific targets.

I know only a little about LLVM, but from what I read that does for the back end what tools like Antlr4 do for the front end. Having created a real working compiler that generated Win32 COFF object and dll files, I'm very relaxed about the code generation, I've done it and seen source code compile, link and run and I'm comfortable with the problem domain.

So I'm not sure if I'm making sense, but I'm not fazed with any aspect of this undertaking, I'm fortunate having done it before and it's a total accident that I'm getting interested in it again.

My focus the past few months has been (to the irritation of some here) "the grammar" and that's because too few programming languages really, truly design the grammar. They seem (and this is true of the vast majority) to adopt some existing language (usually C, Java, Javascript) and then hang on the coat tails.

The problem here though is that a grammar is like clay, once it dries and the shape is formed, it is what it is and there's no going back. What I've been doing the past few months is like a potter at his wheel getting interesting forms but always keeping the clay wet, not letting it set just yet.

I don't want the clay to set until it reaches a point where all the possible innovations and changes have been made, the bad ones cast aside and the good ones nurtured, at some point the grammar will get to the point of "That's it boys, she's ready, lets build her" so to speak!

That point is now close, only this afternoon I reviewed Verilog's syntax and encountered their "reduction" operators. Would a language "need" these? an MCU oriented language? I don't know, but they are neat and if the have a use in FPGA design so that's pretty close to hardware.

Not sure how hardware "oriented" you are but here's a guy I'd have loved to have met, I like his style.

Later!

[Sherlock Holmes]

This is how 99% of "I want to make my own compiler" threads end up being. Grammar is like 1% of the work and the easiest part at that.

With your never ending put downs and defeatism, this comes as no surprise. But please support the bizarre unscientific claim that the grammar design is 1% of the effort in designing a programming language? do you have any evidence? any data?

I can sort of understand this for the education though. Code gen past the primitive stack machine with AST traversal and direct generation is really hard work, you would have to dedicate multiple semesters to that, and nobody has time for that.

Don't assume everybody else has the same shortcomings as you do, I guess you're just not cut out for this subject.

And a lot of this work is not purely algorithmic as academics like, but more practical. There is no real theories, you just need to sit down and code. And the code gets messy and hard. This is the part where most new languages give up.

How many languages have you designed? how many compilers have you built and delivered? Why should anyone listen to your opinions on compilers when you've never written one? Why do you think your opinion should be taken seriously when you apparently struggle so much to even understand the basics of the subject?

If you find it hard to write code, if it gets messy, then take up another vocation, one that doesn't make you so bitter and grumpy.

The more you post here the more foolish you make yourself look, you do realize that? don't you?

[MIS42N]

But now I want to add that "code generation" for C code is "obvious."  You can look at C code and picture what the code generated ought to look like.  Perhaps the compiler optimizes that weirdly, perhaps not.  Maybe this is what people OUGHT to mean when they say "C is just a high level assembler."

I agree.

Any messy algorithm I code in VB script then translate it to assembler when the logic is right. I am then just a human compiler/optimiser.

The advantage of a higher level language is it doesn't make mistakes. Testing A < B for a multi byte quantity on an 8-bit CPU requires a few assembler instructions and remembering how to carry forward or get out of the comparison. Easy to accidentally say BCS when should have said BCC. The compiler gets it right every time. I am lazy and just throw in what I guess it to be then run an emulator to sort it out. Once a function is worked out, just cut and paste. Something that gets used more than a couple of times, make a macro. It takes longer to write, but the payoff comes in speed.

[DC1MC]

"Parturient montes, nascetur ridiculus mus" 

[Siwastaja]

This is how 99% of "I want to make my own compiler" threads end up being. Grammar is like 1% of the work and the easiest part at that.

With your never ending put downs and defeatism, this comes as no surprise. But please support the bizarre unscientific claim that the grammar design is 1% of the effort in designing a programming language? do you have any evidence? any data?

We will soon* have yet another data point. Now that you have finished your perfect grammar within two months of forum shitposting and ignoring all the experts, and had a bit of experimentation with off-the-shelf lexical tools, maybe a few hours of actual work, let's see how long it takes for you to have actual code generation which works for non-trivial pieces of code. Let's say, something you can compile the compiler itself with. Any single actual CPU architecture, maybe x86, or maybe ARMV7-M or RISCV. You should also have some kind of standard library with basic IO functionality, even for a single OS (POSIX or Windows).

*) or maybe not so soon. I'm not holding my breath

Then we can estimate the percentage that went into the design of grammar.

[magic]

For the final project on my compilers course at the uni I wanted to deliver something that I wouldn't be too embarrassed about.

So definitely not a simple stack machine, but real register allocation with SSA, I also recall some rescheduling instruction to reduce the amount of data values alive at a time, constant propagation, elimination of pointless arithmetic (think x+0), dead code, tail call optimizations...

The humble goal was to write simple, recursive implementations of factorial, fibonacci and similar things and have them compiled down to a few lines of clean assembly.

This took me three weeks, several hours per day. My conclusion was that compilers can be done, but they are not exactly trivial

edit
That being said, it is my understanding that nowadays one can save a lot of time by writing a frontend for LLVM/GCC. These are supposed to have a lot of the hard stuff and all the low level code generation handled for you. Although when it comes to portability, I seem to recall that some of their CPU abstractions are still somewhat leaky...

[Nominal Animal]

My interpretation of the essential content of this thread:

"I have now done the hard but brilliant work of defining this new, superior programming language.
As my own time is better spent elsewhere, I shall leave the straightforward task of implementation to the lesser, more mechanically oriented minds, those better suited for such tasks."

I'm sure everyone working in software has met the type: "Here's my million dollar idea: make a new Facebook, but cheaper!  If you implement it, I'll cut you in on 5% of the profits.  Think of the visibility you'll get!"

[tggzzz]

I want to recommend a great (if a little dated) book on compilers. It was the best of five books I used in the 90s for compiler implementation and explained the code generation far better than any of the other books.
...
These were all good books but the Bornat book showed the author's passion for the subject, he knew his stuff (it contains one if the very few, earliest discussions of coroutines).

Based on a quick skim, that book is very dated - first written in 1979 with examples in BCPL. There are, apparently some updates, but I don't know what they are.

I'm sure that it represents a good exposition of part of the state of the art back then, but now it is woefully inadequate.

A few observations:

• the discussion of heaps and garbage collection is primitive. Unfortunately that's the level at which they are taught on most courses
• co-routines are limited to those in the compiler, not the runtime concept
• there's nothing related to parallel processing
• there's nothing about non-uniform memories
• there's nothing about I/O and "variables" that can change behind the compiler's back

So, while it might be a reasonable exposition of compiler front ends, it doesn't consider what's happening at runtime in modern MCUs or processors.

My focus the past few months has been (to the irritation of some here) "the grammar" and that's because too few programming languages really, truly design the grammar. They seem (and this is true of the vast majority) to adopt some existing language (usually C, Java, Javascript) and then hang on the coat tails.

The problem here though is that a grammar is like clay, once it dries and the shape is formed, it is what it is and there's no going back. What I've been doing the past few months is like a potter at his wheel getting interesting forms but always keeping the clay wet, not letting it set just yet.

I agree.

But I'm not sure you understand how much the compiler backend and runtime behaviour on modern hardware will affect the language concepts.

[DiTBho]

Reading the readme, op wants to discuss, but none of the points we have commented (e.g. Babylon is stupid idea, rot@what is stupid idea, etc) are there in the readme.

So what's the point here? The op daily requests attention for nothing but his "grammar",  and with aggressive arrogance it's nothing but all "write mode".

[DiTBho]

This is how 99% of "I want to make my own compiler" threads end up being. Grammar is like 1% of the work and the easiest part at that.

I can sort of understand this for the education though. Code gen past the primitive stack machine with AST traversal and direct generation is really hard work, you would have to dedicate multiple semesters to that, and nobody has time for that.

And a lot of this work is not purely algorithmic as academics like, but more practical. There is no real theories, you just need to sit down and code. And the code gets messy and hard. This is the part where most new languages give up.

I can confirm this 

my-c experience is exactly like this: one man project, 8 years (6 hours per week ) to have a *decent* machine layer; while, according to my task-planner, adapting the c89 grammar only took 78 hours.

[DiTBho]

My focus the past few months has been (to the irritation of some here) "the grammar" and that's because too few programming languages really, truly design the grammar. They seem (and this is true of the vast majority) to adopt some existing language (usually C, Java, Javascript) and then hang on the coat tails.

The problem here though is that a grammar is like clay, once it dries and the shape is formed, it is what it is and there's no going back. What I've been doing the past few months is like a potter at his wheel getting interesting forms but always keeping the clay wet, not letting it set just yet.

I agree.

... the OP speaks as if he were a luminary university professor with several qualifications with compilers and hardware, worse still, to *justify* the aggressive arrogance he thinks he's right when he wastes people's time with fake endless arguments where we spend the time making comments that HE always ignores.

I agree to one point.

Lot Pop Corn

[DiTBho]

@magic
if you want to laugh I can try to find where I parked my old "L interpreter" and post its C code somewhere here.

It's a "complete" toy, well... just ... I emmm emmm "forgot" to implement function calls because I thought the project would take less time, and at some point I realized it would miss the deadline assigned by my teacher for the project.

Teacher: oh, well, it looks like Pascal
Me: yup (  with this face -> ), I perfected my grammar a bit...
Teacher: hey wait… I see examples, BUT no function calls!? !? !?
Me: ... ups, I think *the polish* took me so long that I think I forgot to implement ...  Next commit.
Teacher: sure, next semester!
Me ... umm?
Teacher: I can give you 24/30 now, but you can always re-take the course and re-commit next semester.



lesson learned: the polish is nice to show off but it's not a priority

[magic]

My compiler also had a curious property: compilation time increased exponentially with the number of nested loops, an unintended consequence of my laziness implementing some optimization

Thankfully, the teacher had no tests for it and I only realized my mistake later

[Siwastaja]

And finally, only time will test the true usability of a programming language. If no one ends up using it, there probably is something wrong with it. Simple languages which have simple workarounds to problems that always and inevitably occur even to most brilliantly designed languages, have advantage over complex languages which require complex workarounds.

[DC1MC]

"A language that doesn't have everything is actually easier to program in than some that do." -- Dennis M. Ritchie

[Sherlock Holmes]

For the final project on my compilers course at the uni I wanted to deliver something that I wouldn't be too embarrassed about.

So definitely not a simple stack machine, but real register allocation with SSA, I also recall some rescheduling instruction to reduce the amount of data values alive at a time, constant propagation, elimination of pointless arithmetic (think x+0), dead code, tail call optimizations...

The humble goal was to write simple, recursive implementations of factorial, fibonacci and similar things and have them compiled down to a few lines of clean assembly.

This took me three weeks, several hours per day. My conclusion was that compilers can be done, but they are not exactly trivial

edit
That being said, it is my understanding that nowadays one can save a lot of time by writing a frontend for LLVM/GCC. These are supposed to have a lot of the hard stuff and all the low level code generation handled for you. Although when it comes to portability, I seem to recall that some of their CPU abstractions are still somewhat leaky...

I know little yet about LLVM, I believe it's certainly a sensible way to go though, over a hand crafted code generator. Despite the incessant prattling from the usual self appointed compiler Gestapo, I've shown some of them evidence of my experience, I mentioned this in my first post.

I've designed and written a Pentium code generator for the PL/I language, I've revealed the source code. That was a lot of work, it includes an abstract code generator, an instruction analyzer and opcode generator, a peep hole optimizer, a COFF object file output generator.

So yes that was a lot of work, LLVM promises to reduce that but it will (like Antlr) require that those tool be fully understood and that takes time.

I don't think anybody here has ever designed a grammar yet the prattlers here glibly dismiss its significance. This is likely because the beginner books they've likely read never talk about it, they almost always use an existing language as their grammar.

Grammars need to be designed, until it is, writing code that assumes a grammar is a waste of time because until the grammar is deemed stable, changes to it will be needed, rework to reams of code needed, so why do it? I think many inexperienced developers are like this, the never design, they just "code", I guess its the way they were taught.

[tggzzz]

I don't think anybody here has ever designed a grammar yet the prattlers here glibly dismiss its significance.

Probably, and probably not. The latter because it is a solved, and more importantly because other issues are far more problematic. (Examples: memory models, possibility of aliasing, fundamental concepts embodied in the language, optimisation)

This is likely because the beginner books they've likely read never talk about it, they almost always use an existing language as their grammar.

Possibly.

If in order to complete a task you have to do either A or B, then start with the easier.
If in order to complete a task you have to do either A and B, then start with the more difficult.

You have to define the concepts/behaviour and grammar, and do the compilation and optimisation and code generation. You are starting with the the easiest. Shame.

[Sherlock Holmes]

I don't think anybody here has ever designed a grammar yet the prattlers here glibly dismiss its significance.

Probably, and probably not. The latter because it is a solved, and more importantly because other issues are far more problematic. (Examples: memory models, possibility of aliasing, fundamental concepts embodied in the language, optimisation)

So that's not a yes then is it? have you ever designed a programming language? have you ever designed anything? I'm assuming the answer is "No".

It isn't solved, a design isn't done until the goals of the design are met, until the objectives defined for the grammar met, until then it is not "done", do you really not understand what design is?

One of the reasons for this thread existing was simply to gather input in order to define those requirements and goals, but rather than participate you prefer to complain about "progress" and "code generation" things that cannot be progressed efficiently when the goals are still being defined.

Whether some phase of a project is or is not the largest source of challenges, is not what you were discussing, it is the order of the phases you've been lecturing me on.

Building a house generates many more problems than laying the foundations and laying the drainage system, yet if you were building a house you'd do the foundations after building the house I suppose?

What would happen if you had constructed the first twenty floors of a large skyscraper when you discover you didn't design the foundations sufficiently well? let me tell you - you'd be fired as the project manager and replaced by someone who know what "design" means.

This is likely because the beginner books they've likely read never talk about it, they almost always use an existing language as their grammar.

Possibly.

If in order to complete a task you have to do either A or B, then start with the easier.
If in order to complete a task you have to do either A and B, then start with the more difficult.

This is proof positive that you're a troll now. The order of phases in a project is not dictated by their relative complexity, one cannot build a rocket and send men to the moon without doing other stuff first like deciding on a route, deciding the likely fuel consumption, safety of the crew, costs, support staffing, team roles etc etc.

You have to define the concepts/behaviour and grammar, and do the compilation and optimisation and code generation. You are starting with the the easiest. Shame.

Your trolling is nothing to do with language design at all, it really about your naive "understanding" of project planning and scheduling, you think you are an expert yet your own buffoonery above shows us otherwise.

There are a handful of people in this thread who know how to discuss this subject like adults, you are clearly not one of them.

I happen to have decades of experience as a software project manager, I've earned very large bonuses for delivering multi million dollar projects ahead of schedule, managing risk, maintaining progress, you have nothing to teach me about project management or programming language design.

Like so many of the blowhards here you think designing a programming language can be reduced to "coding" and if one isn't "coding" then they are fools, doomed to failure.

Coding, coding coding..

I know all about coding, here's the hand coded 8,500 lines recursive descent parser for PL/I subset G written in C on a DOS PC with no internet resources.

Or how would like to see the 4,500 line semantic analyzer? what? it's irrelevant? really? Oh well there it is anyway.

Perhaps the symbol table manager and reference resolver is of some interest? here it is then sorry it's only 750 lines, I'll try harder next time!

Surely the storage allocator is relevant? no? that too is a waste of time? oh well, for the record here it is, again it's only 700 lines, I am sorry about that.

Now your absolute favorite part at last! yes the code generator ! whoopee ! here's the first part, the abstract Pentium code generator this is 3,000 lines.

And finally the real deal, the stuff that separate the "men from the boys" as it were, the actual instruction opcode generator, generates real binary instruction stream, another 3,000 lines.

Last but by no means least, the COFF output file generator itself, that's right, yes, it's true, code generators aren't enough, we really need to create an output file with a strict structure and format that a linker can understand, so here it is.

Ooh, I almost forgot, for all those lovers of assembler here, this is the 1,200 line 80386 runtime (for MS-DOS) that got linked in to every build of a PL/I app.

To cap it all the proof of the pudding as they say! This code ran on DOS.

Now please, pretty please with a cherry on top, stop trolling and let the adults here talk in peace.