Why not an embedded script interpreter rather than C# assemblies ???

I keep asking me the same question and I would like to understand what is going on...

Unity's core is written in C++, and users have the possibility to add functionalities which are coded in C#, compiled and then loaded in the engine.

C# is a very good language in terms of productivity, there is not arguing about that...

But why, god why, using compiled libraries rather and just embed an interpreter in the C++, such as python, lua, javascript V8, go, or whatever ?

It would make so much more sense, one would not need to care about portability of the added functionalities because the script would be interpreted directly in the compiled C++ app...
As long as the core app runs on the platform the script would run too, there would be no need for il2cpp insanity...

Also it would be much easier to reload changes, since no assemblies would have to be reloaded, just the scripts would have to be reloaded...which means that we could make changes in games that are already compiled...

I understand that performance might be an issue, but where to we stand for with old mono performance compared to lets say last generation V8 javascript engine...

So the only reason to use C# instead, except for performance reasons (that would need to be benchmarked), would be the extensive support for third party libraries, however due to the outdated .NET profile most of the interesting libraries that I'd like to use are not available anyways...

Could somebody explain me what happened in their mind for making such decision ?

This is not a hate message, I am actually interested in game engine development, and I would like to know what would have been the best approach if they had a chance to start over again.

 

It would be much slower and some platforms do not allow it.

 

mat_muze, I read your post twice, looking for a reason why an embedded interpreter would be preferable, but I can't find it. What exactly would be the point? Embedding python would limit portability to those platforms where python is supported, so no improvement there.

 

Hi guys it looks like you might missing a bit of technical knowledge, here is a explanation how and embedded interpreter works:

An embedded interpreter is a C++ program that read "text" instructions and execute them internally.
It also allows to expose function in the scripting language to functions that are coded in C++ and would be executed in the host program (like a pointer to a function for instance)...

Usually the interpreter is shipped as a C++ compiled library and is therefore very portable, as long as the library can be compiled on your platform you are good to go (much easier to port than C#)

So the python instruction print("hello") would be a string which would interpreted directly in the C++ program.

More details here: https://docs.python.org/2/extending/embedding.html

But python is just an example, there are also other C++ interpreters which offer better performances such as JavaScript interpreters...

 

That still doesn't explain why you think it is an advantage. Sure it works differently, how does that make life better for us?

I can think of positives for both approaches, by the way. I'm not knocking interpreted languages. It's just that you're citing differences rather than benefits.

They made that decision over 10 years ago, so the "outdated" nature which you cite against it simply wasn't the case at the time. Similarly, plenty of the alternatives you mention as potentially being superior didn't exist at the time, so they weren't options at all.

 

The main advantage is that you do not need to compile C# at all.

C# ---> must be compiled (problem for supporting multiple platforms)
Script ---> simply interpreted (no problem)

Scripts that are written will work seamlessly on any platform, no need to rely on Mono/.NET/IL2CPP for compiling the code since it is interpreted...

Which would also mean that performances would only depend on how fast the native core app runs, while right now there are some discrepancies between the technologies that one is using to compile C#.

This would make a lot of sense for UT, because it does look like that one of the major issue/challenge that Unity Tech. is facing is making sure that the code written in C# should run seamlessly on any platform.

That's why they decided to develop IL2CPP, however this technology is still not very mature, add extra build steps which make the development/debugging more cumbersome and also increase building times...

Well using an interpreter this would guaranty portability out of the box... even for WebGL.

tl;dr:

Advantage would be better portability, no need to develop and maintain complex programs such as IL2CPP.

The only problem I could think of, could be in terms of performance, but without proper benchmarks it is hard to say what would actually runs faster, and it seems like modern JavaScript engines like V8 have became very fast these days.

 

I suspect that they went with it because of things like the following:
- Performance compared to scripting systems available at the time
- Cross-platform availability
- When in the Editor it still has many of the benefits of an interpreted language, eg: not requiring an explicit recompilation step when changing the code
- Access to C# as a programming language
- Ability to create their own languages for it. I suspect JavaScript was important for their marketing, for instance (otherwise why call it "JavaScript" at all, when it is not actually JavaScript?).
- For C# in particular, they could take advantage of the excellent C# development tools already available. I'm not aware of any similarly good IDEs available for interpreted languages. (Probably they do exist today? But, again, the decision was made over a decade ago.)

I understand that you think this. I don't understand why that is an advantage. As in, how would that make my day to day life as a programmer better? Significantly better?

As it is, if I want to change code I hit alt-tab, change my code, hit alt-tab again, and hit the Play button. How would the use of an interpreted language change that?

One thing that I would see as an advantage is if we wanted to change code in live applications in our development. I've seen people show that with Lua developing for games consoles and it's sweet. With C# we don't have quite that level of flexibility, in that we have to stop and re-start our game to see the changes.

Also this. I'm not sure about right now, but I believe Apple's agreements ruled out the use of exactly the type of system you're talking about. (Makes sense given how far they go to QA software deployed to their devices. Not much point doing that then giving people the ability to easily change it after the fact.)

Except where they don't. I mean, it's great that one thing can compile for all platforms, but compiling isn't the whole story. There still needs to be per-platform support for the areas where the platforms are different.

Check out something as simple and common as saving a file. The code might compile for all supported platforms, but that doesn't mean it'll work.


Edit: Also, just want to say, pretty decent discussion.

 

Forgive my lack of familiarity with JavaScript... does it support multi-threading yet?

 

It would not affect your experience much expect the that you would need to use another language. (btw, IDE are improving greatly these days mostly due to the large catalogue which is offered by JetBrains, check it out)

The main advantage would be for UT which would not have to struggle to make their game engine portable.

Again, embedding an interpreter is a very simple operation which does not really depend on the system that you are using, please read more information about it before commenting on this thread.

If the scripts is calling a functionality which is coded is the C++, and works for any platform/system, it will work not matter what platform/system. In essence, the script is just a way to make calls to C++ functions, just like when you press a GUI button and a function get called for that action...

 

A standalone JavaScript engine embedded in a C++ application, can be different from the JavaScript engine which you get in a Web Browser and which you might probably be familiar with. It would resemble more to what you would get with Python, and would therefore offer similar functionalities, multi-threading included.

 

So... the main effects are making me do extra work in changing language, and losing access to the incredible amounts of my prior work and work available from others (eg: the Asset Store)?

Hmm... you don't seem to have read what I wrote, on this or some of the other points.

Have you ever written file IO code for, say, PC + Web Browser + iOS? If you had it would be very clear that what you're saying here simply isn't correct. Sure, the code might interpret correctly for each system (as it already compiles correctly for each system) but it won't actually work on them all. One of them will run as expected, the others will simply return a bunch of errors and fail unless you write platform-specific code that adheres to the appropriate rules.

 

I'm also confused as to why this would be a benefit. Here are some thoughts to consider:

• Anything resembling building code at runtime (compiling or interpreting) is specifically disallowed on apples systems. That portion of the market would still require precompiled code.
• Compile time errors are a developers friend. Eliminating a compile step eliminates an opportunity to catch errors
• Interpreted languages are typically slower

 

Oh the days before web standards, when IE 6 had to be supported.

 

No idea, why would anyone want their code to run 10-100 times slower and getting no benefits in return.

 

What make you say that, have you bench-marked the execution of instructions for interpreted languages vs Mono ?

Do you have any numbers that would be helpful for that discussion ?

 

An interpreter is precompiled... *sigh*...the parsing of the script just trigger functions that are precompiled...

Just imagine a script as an list of command/function calls that are executed on the C++ side sequentially, just like batched commands

True that, however Mono/IL2CPP are still not the super duper fast either, I would be curious to see the execution speed of modern interpreted languages vs IL2CPP/Mono, just curious...

 

On the one hand, the script would call C++ functions, which you exposed manually, and that are guaranteed to work on any machine because that's how the core of the engine in built.

On the other hand, the script would call functions from the script engine's (I/O functionalities for instance) which are usually multi-platform, see python for instance.

Those instructions would then be broken down to low-level calls by the interpreter directly inside the core C++ application.

But the main problem would be that you would also need to include those scripts or libraries as well (I/O scripts, etc...) just like how you need to include a VM

 

My guess is that you're coming from some sort of modding background and not a serious programming background. No serious programmer would ever want an interpreted language over a compile language. The benefits of compilation are far too numerous to list. There are plenty of engines that use LUA or Python or Lisp, go use those. I only use unity because of its C# integration.

 

That hurts a little I do have a solid background in programming tho...

Is that just a matter of taste or have you witnessed significant performance discrepancies between using scripting-based engines and C# ?

Have you ever used panda3D for instance ? Where you disappointed with how fast your game logic code was executed from the interpreted scripts ?

 

A serious programmer would never even get started with a VM like Mono anyways... since we are already doing a huge compromise in performance using Mono, why not envisaging a more comfortable scenario in terms of multi-platform capabilities...

I am not here to attack Unity at all I am rather trying to get insights on how a game engine should be built in the best scenario, knowing how technology has evolved nowadays...

 

I don't "like" things like LUA, python, or LISP. I've used them, the other day I wrote a blender plugin in python. I didn't like doing it. I've never used panda3D.

I LIKE C#. I like it a lot. Its my favorite language and I've known dozens of languages through the years. If I cared about performance (a lot) then I would be using C++, but I care about the balance of performance and RAD. C# strikes a perfect balance in that.

Now, as for my own experience with interpreted languages, I can only compare something with javascript vs c#. I hate putting javascript in my webapps. Its a necessary evil, but if it were up to me, everything would be C# on the backend that rendered straight html to the client.

If you really want something interpreted then I believe that amazon's new lumberyard engine is LUA based.

 

You're suggesting that Mono brings huge compromises in performance... only to suggest they use an interpreted language instead? How would that be an improvement in performance?

 

I don't see how what we currently have is an "uncomfortable scenario". It seems to me that Unity's chose path is extremely flexible for cross platform support, considering that, if I so chose, I could write an app for my tizen smartwatch, or samsung tv.

Maybe I don't understand your objections. Unity has supported javascript and BOO in the past. Do you just not like compiled assemblies for some reason?

 

Well, with all due respect, I am not that interested with what your personal preference may be, that is not the aim of that thread.

C# is a very nice language indeed, but as a compiled languages it has a few short comings as well, portability being the main one.

What I would like to discuss here is whether one would get more advantage using interpreted languages rather that compiled one, both in terms of portability and performance, not in terms of end-users experience.

 

Well, if the performance is similar you gain portability...

If performance is that bad well it's not so interesting of course.

The thing is no one really knows how modern interpreted languages would perform against Mono/IL2CPP so I guess it is not so interesting to discuss about it in that case...

 

Uh...

You were interested in my personal preference, otherwise I wouldn't have stated it.

You're not really interested in discussing anything because you haven't actually said anything. You talked about how an interpreted language system might work, and how you don't like assemblies, but you haven't actually said anything on the topic of why an interpreted language would be better for performance, RAD, or portability.

 

We already have portability.

One of the Unity developers posted a while back (I'm searching for the post now) that Mono C# was approximately half the performance of C++. I'm not aware of any interpreted-only scripting languages that come close to that.

Edit: Apparently I misremembered the contents of the post. It was over a year ago though. Surprised I remembered it at all.

Sorry, but I have to disagree. I'm confident the engineers at Unity actually do know.

 

Nobody's arguing about whether the functions will execute. As far as the computer is concerned it will be correct and everything is fine and dandy. Unfortunately, in many cases, as far as the computer is concerned in many cases "working properly" means aborting a task and giving an error code instead.

Which is exactly what will happen if you write your save game function for desktop PC and then just assume it will work on iOS because "interpreted".

Once again, though, Mono / .NET already provides this. (Yes, Unity had to do work for some platforms, but that's already done so no sense throwing it away now. If they were starting from scratch it might be different.)

Furthermore, it actually has nothing to do with whether the language in use is compiled or interpreted. Either way, someone has to write code somewhere that takes the different rules of each platform into account.

With this in mind, consider that Mono / .NET VMs have also evolved in the last decade. So comparing Unity's old Mono to a current JS VM isn't exactly fair. If we're going to discuss opportunities with current technology then we need to consider the productivity gains allowed by modern .NET languages, toolchains and VMs as well.

 

I would counter this by saying: If python gives us portability, then why can't blender game engine games run on my tizen smartwatch?

 

If I knew it I would not have started this thread

I know that... interpreters can be embedded and are highly portable...

I know that... interpreters are slower that native code, but I don't know how well it would compare to a VM like Mono with exacts numbers...

The rest I hoped somebody could answer...

 

Huh... because Unity is aiming at making profit and therefore has more money to invest in developers while Blender is open-source and developed by people for free...

 

Yes, but at a very expensive cost which is maintaining Mono VMs for each platforms and developing and maintaining IL2CPP for each platform too, that's a huge amount of work and money too...

 

It's not Mono that's slow, it's the version they have that needs a walking stick that's slow.. Chief, if were going to go down this path.. Might as well chuck in a GC Alloc / Dyn Mem system for native C++ and just use C++ as a scripting language. It'd always be far faster (if done right) than any interpreted language.

If you wanted to use LUA for example, sure use something like GCC it'll work on any platform.. Here's the thing, Unity isn't exactly C# (well not strictly) it's a component based subset of C#.. Which is what everyone loves..

It's one of the things that made Unity popular in the first place, if / when they sort out the issues with ILCPO? Whatever, then issue gone.. Everyone gets there beloved component based C# and all is right with the world.

 

Yet there are hundreds of Linux distros and only a relative few are backed by companies.

Choice of scripting language is only one factor though. Otherwise Blender's game engine wouldn't be so restricted.

 

There are a lot of game engines that are meant to make money. Take leadwerks for example. Its a pretty great engine, its built with C++ and scripted with LUA. It has practically no cross platform support. You continue to claim that it would be easier to port stuff using interpreted languages, but I can't find any evidence of that. What I know is that the most feature complete, modern, platform that supports the most platforms in unity. Nothing else comes close (that i know of). That would seem to indicate that unity's approach is either superior in design, or unity has refined it to the point that it is relatively inconsequential.

Panda3d which uses python only supports windows/linux/osx.

Amazon lumberyard uses lua and only supports desktop/console.

 

Actually Lumberyard will support mobile as well, lets not confuse being able to support platforms with lack of caring about some platforms..

 

Until it does, it does not. When/If it does, there will be exactly one modern cross platform game engine built using interpreted languages as a base.

 

You mean besides from AD Stingray right? If Amazon say, Amazon do..! They have mucho dollar!.

 

OK, 2, and there may be others too that noone cares about (shiva3d). Amazon may do, but they don't always do right. Firephone?

My point is that with the vast amount of available engines, and the relatively few that are cross platform, an extremely small amount of those use interpreted languages. If, as the OP proposes, interpreted=portability, then you would think these numbers wouldn't be skewed as such.

 

Interpreted does not instantly provide portability. That's straight out wrong.

At some point you have to get from code the dev writes in engine to code that will run on the target platform. Unity currently uses mono/IL2CPP for this.

An interpreted system still requires a seperate interpreter for each platform. So switching to using an interpreted system will give no extra portability.

 

Oh I agree, I don't see any point in it.. Unity already supports tons of platforms, so meh!.

 

"Because reasons."
It makes as much sense as epic's team obsession with Blueprints.

No. They just decided that this makes sense and did it this way. That decision may not have been the "best possible one".

Interpreters are usually written in more-or-less portable fashion. Meaning you'll have same codebase for all platforms with ocuple of ifdefs here and there for the bottom level OS function. Lua traditionally has been quite popular as interpreted language, and it didn't have much problem running anywhere.

 

I really have trouble believing something like this. Surely there is to ton of differences between how graphics, sound, networking, IAP ect is handled on a tiny handheld ios and on a monolith windows gaming computer are different. There is more to it then just a couple of ifdefs.

I could totally be wrong on this. But this

If true cross platform portability was as simple as throwing in an interpreted language, then someone would have done it years ago. (Either that or its a great opportunity for someone to exploit a hole in the market, go do and make millions).

 

+1.

In general whether we're talking about an interpreter, IL and .net/mono, or a compiler (JIT or not), everything always has a path to assembly and machine code. The very reason interpreters tend to be the most portable is they take the crudest path to that machine code. The interpreter generally takes a very narrow view on what might get the job done to keep it "portable". Note that this doesn't really reduce the amount of work that Unity would have to do it would just shift it because a game engine has to perform quick not just work.

Let's say a brand new platform appears (like the 3ds). Perhaps there is already a LUA intepreter for it (although I don't know of one). If there is one it will be awfully generic and map to the underlying OS/Machine calls in a not performant way. So Unity would fork the interpreter and start building "smarts" into it and make a better interpreter that manages the underlying resources more effeciently. In the end the amount of "cross platform" work that they have to do unique on each platform would be the same and basically starts to look like the work of a compiler. I don't think it saves Unity any time.

The "generic assembly" that is in IL of .NET/Mono's runtime VM is just of a good place to start as writing your own custom interpreter for lua or other scripting languages custom to your C++ engine. In general, because the cross platform conversion in .net happens at a lower level it should always be more performant. Obviously, going the c++ compiler route will be more efficient than either, just as writing assembly by hand is better than compiling c++ but takes more time for the developer to create.

Also earlier it was brought up the javascript/boo support as a note that Unity did support some interpreted languages. They did not. What they did was write a compiler to IL for them. At the end of the day it was still .NET IL the same as any other like via C#. It wasn't interpreted at all.

 

I think you're lumping two very different subsystems together.

There is standard libaray for pretty much everything. When there is no standard library, someone wrote an abstraction layer that can be used instead.

Interpreted language usually serve as embedded subsystem that handles scripting only. That subsystem has nearly no connection to the operating system - it talks to the engine instead. You can keep entirety of that code portable relying only on runtime libraries. That's what lua does. In similar fashion you can embed python into your app.

Interaction with the operating system and handling graphical api differences is not interpreter's job. It is handled by underlying abstraction layer. The layer can be provided by something like libSDL, GLUT, Allegro, Qt, and whatever else. Heck, even standard C/C++ library is made with the purpose of abstracting away the operating system itself, there's OpenGL for graphics, several different apis for sound, etc. Generally speaking you don't really need to write your own wrapper unless you really, really want to. Frankly, writing your own wrapper requires a good reason, and if you don't have one, that's a big warning flag.

Then we have frameworks like PyGame and PyOpenGL. The code written with use of those will remain unchanged for platforms where they're running.

Not sure what you meant speaking about TRUE cross-platform portability (reminds me of one true scottsman), but that was indeed done years ago.

You have qt framework, and libSDL. Write code once, compile for any platform without changes. It has been around for quite a while. Python has PyGame library written for the same purpose.

The decision to use C# of all things as a scripting language is very strange. There were established and older language before that, and something like Python would've been simpler and friendlier to the newbies than C#. But, I guess you can always chalk that up as one of the unity quirks. Like, imaigine that you have a friend that is a nice and completely normal person overall, except that he insists on using his shoes instead of drinking glass at all times. That kind of thing.

 

So to surmise you points:

• Using an interpreter would be just as much work for Unity devs as the current system
• Some people would like a different language then C# for scripting

I don't really see this as an issue worth anyone pursuing.

 

Most interpreted languages do actually drift into things like file io and other tasks that in the end are clearly in the OS and low level end of the spectrum. If the interpreted language had no connections to at least a few low level subsystems it wouldn't be very powerful and you'd end doing anything important on the C++ side anyways at which point you probably wouldn't bother with it. It's the fact that it can do some things for you that is some of the appeal (albeit you generally give up performance as the cost).

You'd have to ask the Unity founders why for sure but I suspect that it had just as much to do with all the .NET libraries available as it did the C# language itself. Want to do threads, sockets, file IO, serialization, etc. well there is already a library for that battle tested and already written for you. C# is just the icing on the cake. The fact that all the translation to native happened from the IL via the runtime also meant it was going to have better performance than an interpreted language.

 

That's incorrect.

• Abstraction layer used by the engine would be maintained by someone else.
• The interpreter would be maintained by someone else.
• And C# by itself does not not provide significant benefit.

FYI OpenGL is an abstraction layer. OpenGL app would run on several platform and you'd only need to ifdef window creation.

Basically, when any developer team decide to "write their own thing", it is a big red warning flag which may indicate overly enthusiastic person on the project that can ultimately deal a lot of damage to the project, because the best idea is reusing and combining existing tech, instead of writing your own (because it creates less bugs).

By using established language that is not C# or getting rid of C# completely you could've freed few years worth of work from the project. However, if unity decides to stick with C# and then even roll out their own C#-related tech, that's their decision, that makes roughly as much sense as Epic's Blueprint fetish. Or drinking tea from your shoe. You get the picture.

 

-_-
File io is provided by C standard library and essentially abstracted away from the code.
Unless you specifically want OS-specific function, and want to mess with something like CFFI, a script running on linux will be identical to the script running on windows.

And C++ provides even more libraries than that.
And python may beat even C++.

The decision doesn't make much sense, so I just chalk it up as a mistake that unity team decided to keep.

-----

P.S. The discussion is taking a strange turn though, because I'm explaining things I'd expect to be known by default.