How much of Unity now relies upon the Python programming language?

It looks like Python is turning up in lots of areas within Unity in build processes and AI toolkits is this a good thing or a bad thing?

Pros: Python is a go to quick scripting language.
Cons: Python is inherently slow compared to native compilation languages.

1. Should Unity be writing everything in DOTS Burst Jobs for maximum performance?
2. Or could Unity adopt Python and boost it's performance with it's DOTS Burst and Jobs technologies into a Python compiler?

Personally I think performance should be key once a working solution is found therefore moving from python to native should be a key step for all Unity technologies.

For instance in this blog post regarding the ML agents toolkit...

It sounds like python is heavily used to link the technology together yet Python is known to be about 3x slower than native code.



So in AI training sessions that can last for days how much of that time could be saved by using more performant native code?

 

Here we go again...

 

You choose the tool for the problem. If the problem is better suited for Python, then use Python. In the Machine Learning/Data Science world, Python is a widely used language. They should not rewrite already proven solutions just because you don't like the language.

 

The major ML frameworks all have gpu/compute shader support built in for training.

 

Neither.

https://en.wikipedia.org/wiki/False_dilemma

 

https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/gpp-python3.html

TLDR Python vs Native (C++ g++)

• 2x to 100x of times slower.
  
• 1.1x to 374x more memory.

If you build to WebGL you will experience this slowdown as emscripten is triggered in the wasm build process.

Unity is working on making the engine and buildprocess faster and more streamlined as well as building an amazing compiler technologies in Burst Jobs and DOTS.

I'm suggesting that they look at how they can use their in house optimisation technologies to enhance and boost the slowest elements of their toolset/pipeline.

It could be that they can burst compile python to super fast native executables or rebuild and replace the python components with C# Burst DOTS components that are way more performant.

 

Those numbers are running on a chip from over a decade ago. Python also has other advantages for ML stuff in that it's extremely easy to do on the fly adjustments to.

Once again you have read something without doing your due diligence.

 

This. The Core 2 Quad Q6600 is missing many of the modern features we expect of a processor that needs to perform heavy math operations. Starting with the AVX instructions which were first introduced with Sandy Bridge and Bulldozer which came out in Q1 2011 and Q3 2011 respectively.

https://ark.intel.com/content/www/u...sor-q6600-8m-cache-2-40-ghz-1066-mhz-fsb.html
https://en.wikipedia.org/wiki/Advanced_Vector_Extensions

Using those benchmarks as evidence against Python would be like pointing at the old raytracing demos from the early days of GPGPU as evidence against modern raytracing.

http://gpurt.sourceforge.net/DA07_0405_Ray_Tracing_on_GPU-1.0.5.pdf (2005)
http://www.icare3d.org/codes-and-projects/codes/raytracer_gpu_full_1-0.html (2007)

 

Also python is just an interface like every language, they can tweak the compiler to make further optimization. Python is slow because it doesn't force you to specify stuff and push that responsability to the compiled program, which then as to make extra check for you in the background. For domain specific stuff like ML, where there is tighter assumption, we can generally compile domain specific optimization, since ML mostly only need declarative statement, they can just use python to pipe data and structure to a tighter compiled code from a library (like tensorflow), and we know unity has written his own. We have precedent with compiling to webassembly to get almost native speed. Also if you are not satisfied by that take a look at cython, it's c + python for speed.

 

Found this github programming benchmark it looks much more recent but does have a rather rude name https://github.com/kostya/benchmarks as you can see in every benchmark Python3 is much slower than C++ and tends to use more memory.

 

And what's the point? You got the answer why Unity does not support Python as a scripting language. This does not mean it's useless for algorithms and hosting ML/DM applications.

 

How much time do Unity's tools spend executing Python code?

Unless the answer is a significant proportion of overall code execution time then there's little to gain from optimising this, which is rule #1 for any optimisation work you might do. If I can make 1% of my code* even 1000x faster, I've still gained less than one percent performance improvement.

Plus, as others have said, it's quite possible that the Python code isn't as slow as a benchmark might make it look, and/or there could be other reasons that it might not practically help a lot.

* By execution time.

 

It's like you would want to rewrite your game's launcher in C++ because that is faster in general than the Mono-driven C# code.
True, and all (usually), but carry no weight if you really think about it.

 

You don't use a second bottle when the first hasn't fill yet, even though it mean you can transport more water.

 

Those areas are about tools, advanced and complex graphics games don't use python as programming language.

 

Python turns up when it makes sense for Python to be used.

It's not a good or bad thing, it's just a thing.

 

As someone who writes python code for work, and absolutely hates the language, I really wish you were wrong.

 

Well, my first paying gig was in dBase III, I have worked for years using pure Javascript and PHP. I develop in Java nowadays and I even use GWT. I used Python for work only once, we needed some extra components for our task manager back in the days (we used Trac at the time) and I won this task.

 

HISSS! Heathen! I <3 Python its so easy to code with.

Out of curiosity why don't you like it?

I love python because it is so abstract that its useful for putting together high level functionality. Ie. instead of writing C code directly for the platform you need and having to deal with that mess Python lets you focus on just getting things done. This makes it sort of slow but I think it has its uses. I think it would be great to b able to code in Unity with Python BTW.

Python is essentially pseudo-code that actually runs. <3

 

Security Issues: OK so Unity needs Python, they why not compile the python it needs to native and get around these 10 common security issues with Python or fix the security issues with it's python usage.

1. Input Injection - attacker can change filepath so external files are injected into pythons command processing.
2. Parsing XML - the Billion laughs XML using referntial entities in XML.or external entity expansion.
3. Assert Statments - allows python to jump to secure code regarless of admin rights.
4. Timing Attacks - these allow hackers to break your security enrypted passwords.
5. A polluted site-packages or import path - one of the largest security holes in Python allowning execution of arbitrary code.
6. Temporary files - allowing a file to be changed and trick your Python app into loading or exposing data.
7. Using yaml.load - loading YAML files leaves your system wide open to attack.
8. Pickles - as bad as YAML allows attacker to use include references to subprocess modules to run commands on host.
9. Using the system PYthon runtime and not patching it - keeping Python up to date to ensure security holes are patched.
10. Not patching your dependencis - keeping all of the Python stack updated and patched to ensure securtiy holes are patched.

The article that lists these Python security issues also lists a range of ways to secure them, does Unity use/provide any of these e.g. some invovle using tools to scan the integrity of your python stack?

Or is your system most vulnerable to attack when you are working with Unity and a Python script is running?

 

@Arowx, are you trolling now, or have nothing better to write?

First you write about performance, now about security. You common C# Unity game can be manipulated with cheatengine at ease. Not to mention it can be recompiled. And you are concern about some injections in python .... or other xyz language. Do server side, problem solved.

 

Making a case for Unity to migrate Python code to native, for performance and security reasons.

I've just noticed that Python is also in Unity Packages, it's turning up everywhere in Unity so it should at least be secure and safe to use could Unity Sandbox Python?

 

Please elaborate on "Security Issues", with examples.
Like if someone hacks into your Unity on desktop?
Did it happened already?
By now, there will be tons of reports, if that would be the issue indeed.
I haven't searched, maybe they are. But for now I assume, they are not.

Is your compiled and published game running on Python?
Sorry, I don't get your "security" point.

 

Can you state what the specific instances of these security issues are in Unity?

 

Because it's a micro-optimization. A build tool doesn't spend most of its time running its own code. It spends most of its time sitting there idle while it waits for each job to complete.

Ideally with an explanation of why he believes that compiling to native would make it more secure.

 

This is a web issue.

Only applicable if an XML parser is being used and you load untrusted files.

Non-applicable when it comes to things like Unity's implementation.

ALSO non-applicable when it comes to Unity's implementation.

This ALSO has nothing to do with Unity's implementation because, if you haven't noticed, executing code is kind of a thing we're all dealing with all day.

I'm starting to wonder if you've actually even read the full description of these issues and considered the way they could impact Unity itself because, once again, this has nothing to really do with Python or Unity or the combination of the two.

So don't load YAML files from untrusted sources. What, do you just not practice other basic infosec either?

See above.

Unity's ML requires at least Python 3.6. This is an end user issue, nothing to do with Unity.

See above.

You really have this problem where you can't see past the nose on your face. The issues you're pointing at are completely irrelevant, but I guess you've got some bone to pick with... python I guess?

 

Stop everyone, the Unity editor is running some html and stuff sometimes. World is ending.

 

But Hippocoder, what if I leave my computer unlocked and go to the bathroom and somebody sneaks a picture of a boob into my game? WHAT THEN?

 

I would be worry more about boob (boo?) and python together.
You will just stick PEGI18 label on your title
And blame python .... alternatively OP.

 

Primarily it is white space used for controlling the flow of code that I hate.

 

I don't understand why Unity does not require finger print authentication for every change in the project folder either.

Yeah, that's super weird, especially when you practically grow up on curly brackets and begin-end pairs. (I did quite a few years Pascal as well, when it was cool)

 

+ for increased safety measures.
And add yet ear shape for check.

 

Unity assumes everyone has that implemented when connecting to git

The weirdness isn't the issue. The number of times I've encountered bugs in other people's code which are caused by mixing tabs and spaces (against internal coding standards) is unacceptable, and this type of bug simply doesn't exist in C style curly brace and semicolon languages.

 

What about security issues with Visual Studio? Anyone who has access to your computer can just open your code files and type whatever they want in there.

 

Well then I'll download it.

 

Then you have a sure fire winning formula for launching your game on steam.

 

I grew up with them, still hates them with a passion

Yeah, that's true, instead they are replaced by misplaced/missing curly brace and semi colon 90% of bug I help people with in my training is just that. There is no lesser evil, just taste (also use ide that reveal the hidden character in subdued color, then that problem go away and never come back)/

 

Sure, but at least they're visible mistakes. I much prefer "this thing is in the wrong place" over "this invisible thing is the wrong invisible thing". To be fair, that's more a gripe with the implementation of tabs than with the concept of white-space having meaning. If the meaning were consistent then it would be fine.

Also, having characters which explicitly delimit scope means that your tools can auto-format your code's white-space. If the white space is the scope delimiter then... nope.

 

I guess the people who designed Python were thinking the same as you. But they made a mistake, they have replaced a problematic, but visual delimiter system with a problematic hidden one. Until you permanently turn on the 'show whitespace character' option which makes your code bloated with a bunch of arrows and points and whatnot.

The curly brackets are less intrusive. Also the begin-end. Even the C64 Basic's For-Next.

 

Curly brace are so evil I don't even begin to understand how they were able to brainwash you into a cult

Anyway, My current hypothetical solution is to come back to a prompt based IDE, like with chat. It solves all problem because you only input one expression at a time and the validation step (submit the expression) is the delimiter, inside the prompt area you can spread the symbol however you want for readability (on chat it's crtl+enter to make non submit line break). It can auto format, and you can probably just switch between auto formating and custom formating on the fly because you would store the code into itemized format.

 

Braces add more than they remove, you can do code that's not possible otherwise with shorthand, and you'd need to borrow { } for that purpose regardless... plus they do force you to think about scope a lot.

 

I agree with HIppo.
At least with braces is hard to make mistake, which scope should be indented to Level1, 2, 3 etc, if formatting get screwed for any reason.

 

That's the IDE interface jobs, not mine. BY experiences, that's literally 90% of petty bug, and I don't have time for that, no rationalization can help. Let's agree that both python and c like are bad and we should enter the future where interface design is actually meaningful.

Edit: also curly brace only help if they are correctly formatted and indented, that mean that the code has been frozen and you won't work on it ever again, even though they are basically visual noize by that point so YAGNI. And scope/indent/block is already visualized by vertical bar in all basic IDE so that's redundant.

 

only time I do not use braces are for early exits

Code (CSharp):

1. if(something) return;

Or if I'm in a tight loop and cant use collection.ForEach(Action<T> action) (Allocates). I do

Code (CSharp):

1. foreach(var item in items) item.OperateOnItem();

 

Curly braces don't care about formatting. You can write all software in one line, if you like. That is one of the points of that introduced "noise". If you do web programming, you will notice, JavaScript uses this aspect to format scripts (remove indentations and break lines (also comments) ), to minimize number of characters, hence minimize JS files, to allow faster load over net. Hence you got often dev js and minimal js, doing exactly the same.

 

Curly brace don't care, sure, using them to read scope does I mean unless you use them to read minified javascript code, because that's how we work on code right? Minification is as good as compiling, you don't want to deal with it. It's just more parseable easily back to good formatting, but that's what decompiler do anyway, and we are moving toward compiling webassembly anyway. AND we still keep a dev version that isn't minified.

I understand why curly brace are there, they reduce ambiguity (especially for the parser), it's a trade off, I'm saying it's old tech, let's move on. We can hide them like we hide format tags in forum post.

 

I'd rather keep the braces and the freedom of use even notepad.exe to write my code than be forced to use a "high-tech" IDE with vague chatty coding style (which only works if it's linear development, as soon as you need to insert code somewhere, you're f***ed, your thought process is out of the window), but thanks.

 

Here's a question: Why do we even need braces/tabs for our programming? Their purpose is to make reading code easier so we can make better edits.

I think a decent solution may be to widen the braces font signature. I sometimes have trouble picking out the braces because they are thin. The other thing that bothers me is the obtuse error messages one gets when a brace is missing. They are side effect errors instead of direct. I think it would be very easy to do a simple count of the number of opening versus closing braces and flag an error if they are not 1:1 ...

 

https://docs.microsoft.com/en-us/dotnet/csharp/misc/cs1513

It does it whenever possible. There are circumstances when the missing bracket cause other problems and those problems may be found first, then the compiler will report those (as well).


Edit: BTW, I'm pretty sure that multiple teams are working on some kind AI-supported compiler, which can analyze the code closer to a human being and can report the real root cause of a problem. We will have those in a couple of years.

 

Another advantage of curly brackets is pasting existing scoped codes inside other scopes.

Code (CSharp):

1. if ( a == b )
2. {
3.   if ( b > c )
4.   {
5.     if ( c < d )
6.     { }
7.   }
8. }

Now if I paste unformatted code into line 6

Code (CSharp):

1. if ( d == e )
2. {
3. }

I got

Code (CSharp):

1. if ( a == b )
2. {
3.   if ( b > c )
4.   {
5.     if ( c < d )
6.     {
7. if ( d == e )
8. {
9. }
10.     }
11.   }
12. }

Compiler knows exactly where code sits, doe to brackets.

Now do exactly the same but remove brackets.

Code (CSharp):

1. if ( a == b )
2.   if ( b > c )
3.     if ( c < d )

Now if I paste unformatted code into line 4

Code (CSharp):

1. if ( d == e )

I got

Code (CSharp):

1. if ( a == b )
2.   if ( b > c )
3.     if ( c < d )
4. if ( d == e )

Now context of the code execution has completely different meaning. Good luck.
In this sense, curly brackets, or even endif are superior, to prevent many type of logical errors.