[LEGO A.I.] I've got many questions for LEGO. Lego microgame license, LEGO(?) "Speech Finder" etc

Dear LEGO,
I see there are LEGO tutorials (LEGO microgame etc) are included in Unity Hub.
I've just tried two first tutorials of MicroGame, and see that there is LEGO minifigure included, and almost 50 animations for character.
I am really happy that such tutorial appeared officially. I am fan of LEGO (won on TV LEGO constructor, and my father is architect, and I made him a small program to fix lengthes of walls according to bricks sizes, in LEOG style). I've bought one LEGO character
https://free3d.com/3d-model/lego-rigged-minifigure-2580.html
for 29$ maybe around year ago or more,
as I really wanted to use LEGO character/minifigure to express my ideas.
I am studying programming now in computer academy (1,5 already, 1 year ahead),
and I feel now it's best time to ask you, dear LEGO, my questions.

In short: I want a program to automatically convert books - into some [Audio]Visual stories.
I mean those books for children mostly. As Alice in WonderLand said: "what is the use of books without conversations and pictures?" Book should have many talks - direct speeches, maybe indirect speeches also. In short, these might be children's books like fairy tales (which usually Disney most amazingly can screen on form of cartoons (2d/3d graphics) or maybe movies with real human actors).
Sure, it will require A.I..

Do you know about program "Spinnkers Story Machine"?

in this program child can type sentences, and program visualizes them.

I want something like "LEGO Story Machine" (or instead it maybe MineCraft or StickMan or any other similar character whichis very general for all humanoids).
I know LEGO has Lego Digital Designer - https://www.lego.com/en-us/ldd
but that's not what I really want.

I think that people, children who play LEGO, they first inspire themselves by reading books (or parents read to them, or that's Dsiney movies, or LEGO constructor which is schemed like Disney's fairy tales etc), and then they find most inpiring characters and play like in theatre - but with LEGO minifigures etc.

Problem is that ... there are so many stories (I mean first of all those fairy tales, children's stories with lot's of conversations and usually common characters, which are humanoids, and common animals, common birds, common fishes, and other comon species, be they even some demigods or other creatures like in Heroes of Might and Magic 2/3 etc cool game) in world. If we will nly take those in English (or translated into English) - we quickly see that not all of them are visualized.

I want to propose a solution. We need a program to analyze "raw" books, which are not pre-edited by some screen-writers. If we can make this process more automatic, then many children's books would be visualized. I hope in LEGO style. I think LEGO is the company #1 which can make this possible easier than any other company.

LEGO has many games, i mean both board games (constructors of course), and e-games, in Unity.
I do realize that you've got copyright to to make your LEGO games, sell them, etc.
But I want something like LEGO STORY MACHINE

But to make it possible, we need to make some original, unique program (one and only of it's kind, as I don't know anythng of similar existing programs nowadays).

I want to call this program "LEGO ([In]Direct) SPEECH FINDER". I recently wrote small program to find direct speeches in a book "Alice in WonderLand" - see http://www.gutenberg.org/files/11/11-h/11-h.htm

In my small C# program ( I wrote it just in half-day or so),
I just look for quote marks " (paired and in all forms, like we do when parsing quotes in JSON files or in other programming language which requires parameter/value or variable or whatever to be put in quotes) - and replace them into paired quotes, so I find Direct speeches.

It is quite simple, if book is "Normalized" (yes, I use this word, we need normalized data to put that into DataBases like MS-SQL etc, so here it is same, we need A.I. to normalize any book with story. yes, usually scenarist\editor does it, but why not try make it more automatic? I know how! this is algorythm)...

Then, those quotes can be put int 'LEGO Microgame" speech module. In those Microgame Tutorial I see that some brick (?) is talking or of course minifigures also can talk.
So, to make this conversion tool possible ("LEGO story machine"),
we need to look for actors\characters names near direct speeches. I think this can also become more automatic. Once user has to confirm that such and such name can be attached to some minifigure (humanoid creature, or non-humanoid - that can be some abstract smily face\ LEGO head which will be a "template" replacement for any specie which has no 3d model in LEGO and still it is ok, isn't it? consider it as 'Chesihre Cat" or whatever alive SMILY actor you like) ...
This is similar to Story Machine - player inputs the names of actors.

If we make search in text for same names all over the text -
then we can find that one and same character repeats appearing again and again. He may do common events/actions (that's verbs): talk (find direct speech in quotes is quite easy, but indirect speech finder is a much more troublesome task, but very interesting It can be then converted into direct-speech )..

I don't think that making text-to-speech into childrens program like this would nbe a good idea - it doesn't sound so nice (like robot, right?). But if that looks like comix - it's great.

I think LEGO may create a program "LEGO STORYBOARD MAKER", which will automatically detects these common verbs. i know there is a big free database of 2500 animations -
https://forum.unity.com/threads/huge-free-fbx-mocap-library-released.261401/

And I'ld be happy to apply them all to LEGO minifigures -
to be able to play all those 2500 animations with smily faces for all those characters
which speak auto-detected direct speeches
and whose names are just near the speeches they speak
(usualy it is "[any_LEGO_actor_name_here] said",
but sometimes books omit the speaker's name, and simply write -- - long dash before and after the speec - they also must be replaced into Direct speeches in quotes. automatically, in this "LEGO story machine").

I made this simple program, and was amazed how fast it is and how good it is to look for speeches first!

If we want to make a tool to automatically converting any story (childrens' book, like Disney fairy tale etc) -
into "LEGO story" or LEGO story-board / comix
using "LEGO story machine"
then we need always start with Direct speeches. then find InDirect speeches,
and then find actors\speakers near be - we may use dictionaries
(approved by user like in "Spinnakers Story Machine",
and confirm each action like it is done in program like MS Word which doesn't replace automatically the word if it suspects there is an error, but suggests in a pop-up dialog all possible variants - and user has to choose what is best for this exact Fairy-Tale \ Children's story with conversations)...

Purpose is to visualize in LEGO style all possible children's stories (all conversations with direct\indirect speeches) which are availalbe now in all libraries in the world in English (at least) and (hopefully, then in all) other laguages as well...

At present, I suggest using SharpNLP
https://archive.codeplex.com/?p=sharpnlp
program - it can analyze only English texts, but can make simple Part-of-spech tagging and detect sentence structure.

Also we may try BookNLP
https://github.com/dbamman/book-nlp etc

Really, I don't think such programs are good enough for the purose which I describe here...
We need programs of very diferent kind. It may be "data-mining" software.

But... I believe, that if we want "MVP" - Minimum Viable Product (it means it saves time for customers - children who want to read all the books in LEGO style fat, even if they are not visualized by anyone) -
then we need to begin with Speech Module - "LEGO Direct Speech Finder"
Once this done - LEGO InDirectr Speech Finder can be done,

and only then we can try detect who is talking
(well, yes, you can think that it is "a brick" is talking,
if we only detect Direct Seech, put it automatically into LEGO scene
(instantiate "LEGO speech" control, similarly to "Microsoft Agent charracter")...

But well, it is rather "boring", if a child opens a book file (txt, rtf, doc, docx, html etc),
and sees only "LEGO Speaking Bricks" ;-)

but it is fun. It can be achieved ;-)

Then, what is to be done. I want that a child must be a collaborator in making visualization of all books possible in LEGO style ...
I 've made a little program ( you can make it on your own),
which has a purpose to make LEGO characters automatically - before I put them into Unity3d automatically.

i use that character which I've bought for 29$ (link is in beginning this of message).
But before going to animation, we need to make 2d design.

i suggest, that child who opens any story-book TXT file, then make step-by-step confirmation for each direct\indirect speech (we don't need 100% precision now, like all those programs whch automatically convert English into Chinese etc - they are not 100% perfect, but it is fast, and it is cool for those who don't know language at all!) - then child must be adviced to find the speakers - near "[IN]Direct speeches"...

In SharpNLP - I know, the speakers - those words starting with capital letters - they have unique tags, which mean they are nouns and some actors\speaker etc - Proper Nouns, so they should be converted automatically into LEGO minifigures (like in this free LEGO game tutorial at GitHub)...

So, speakers are found, what is next? It is fun!
We need to allow a child to upload any picture ... it's best if there is "crop" tool to select only face, but other part of body also can be allowed to include - this can be done semi-automatically, - just add a "medicine dropper" tool, to pich the color from Lego character,
and - simply paste the color into LEGO minifigure.

What I did in C# - I've got very amazing tool, it looks great, this tool simply takes a picture, and converts it into 256-color image (like for Microsoft Agent, which is speaking ActiveX control, cool for Web, i want this "LEGO Story Machine" to be online service. if you want to copy-protect those all animations which you have in LEGO, then you can make them AJAX-supplied to user, so he will not be able to misuse its copyright etc / pirate it. But I'ld like such program, lifetime license, if this would be done by LEGO or whatever company. Or i'll have to make it on my own, but I want it faster, so i write it here. I'll make it free, don't worry, non-commercial).

So, child would get TOP 256 colors from the uploaded image, and that palette can be used to draw over LEGO texture of minifigure (well, it's done in a bit "draft" way now, as I don't access to LEGO source, and Unity3d does it too slow, and those textures are usually very hard to program automatically... so we need separate textures for for head, torso, legs, hands at least, and let child take colors from top colors using "eye-dropper" and paste color into any part of body, so generate textures... But it can be done automatically, just colors would be imperfectly applied from original picture, still it's fun! ;-) and most important - very quick. That's task for designer, who knows how to make art-therapy, how to match colors harmoniously (and that also applies to sounds\notes, 7 colors -> 7 notes, that can be done in SOL-RE-SOL language, maybe LEGO would make LEGO SOL-RE-SOL also? ;-) )...

So, user may draw whatever s\he wants on the doby of LEGO minifigure texture (I talk about humanoid now). Then in my program i press "generate texture" - and it makes a file, which is taken as texture in LEGO. from Winforms I run process with Unity test tool, which opens LEGO character, file of texture is loaded aways same, but as it is new one - old version is backuped, and user sees new character.

So, problem here is that user may generate so many characters, but they all must be put into databases. Of course it is all optional - to set colors manually. it is not necessary, if there are already pre-defined presets for all most common LEGO characters, and if they are found in some new text (maybe same characters but in new scenarios) - so they may be applied...
What to do if there is no LEGO 3d character for some noun\actor\character (specie of life) word? no problem - that would be "smily face", "LEGO brick" talking instead. Idea is to draw this in some nice colors, harmoniously, so that it will not make borind feeling. i know LEGO and Disney can do this well, so I ask you to do it... And I know you can make it if you really want it (having millions of fans, like me, all over the world, and millions of minifigures already done, and huge database of 3d-characters, their animations etc)... I'ld like to be able to subscribe to some animations/LEGO-3d-creatures - let me pick - and if you (or we, all AI programmers all from over the world) make this program - please make it as online service. It will solve so many problems (of course you can't make it open-source, at least it would be any terabytes, and it is a lot of your mony, but you can sell parts)...

Actually, I am talking now a bit as a customer, but also as programmer. You know, programmers make programs to solve their problems. But as for LEGO - you have not only programs (game is not a big "deal" - it can be done in Unity3d - there are already many free 3d models and animations), but great, amazing, cool design - LEGO minifigures. They are smily - and probably it is a key, secret why people love LEGO so much. It makes one more hapy when he simple looks at LEGO smile - and he wants to play with it... if one is happy (especially a child) - then what he needs more than that?

But time goes, as Alice in Wonderland said: "to remain in smae place, position - we have to run very fast"... So as I grow up, i need programs to analyze texts, read books, but I don't need so much visualization like LEGO does... Say, some companies spend millions of dollars to make one movie, which is just 1,5 hour length... and so many people make it... and... part of such movies fail. lot of time and money wasted...

But what people really need (you can call it MVP - Minimum Viable \ Valuable product - for which customer is ready to pay, and which is interesting to be done by companies like LEGO/Disney thus, I hope - and it is where real economic problem is: "customer\client <-vs-> producer/product-service maker")...

So, take a look at those "so-called" visualization software... What do they visualize? That are tabls, simple graphs etc... Is that good for people who live today? When one can just open Unity3d and make some cool LEGO game in "no time"? There are so many resources available now - tools to do part-of-speech-tagging, tools to translate from one language to another (and I belive we need to tool to convert texts of fairy tales ("Alice's "books with conversations" ), game engines, text-to-speech, many 3d models\characters\minifigures, animations, dictionaries, databases, AI tools - but it all must gotogether to solve one simple problem of people: they read books. They read books, so they get knowledge from them... And most simpe books- for children, they have converstaions (well, even Bible have converstaions, Direct-Speeches, and you would be amazed how cool it is to try to find text between quotes, mark it in some other color, and put it qutomatically into LEGO Story Machine or whatever way you may call it)...

I think many people fail to make such programs, because they don't want to make "[In]Direct speech finder", and that is the key, first thing, and MVP, minimum viable product, for which customers may even pay... That which will motivate both - developers of "friendly LEGO AI" products (programmers, designers etc), and will make end-clients also happier... Everyone loves LEGO. then why not make one step forward, and make this "silent revolution" - make "LEGO AI for audio-visualization of children's books"?

I believe it was me who "pushed" "Bazelevs Innovations" towards their idea to make FilmLanguage, and they now look for "Talented programmer in Unity3d" to make core of similar program (they got millions funding for that already, and used that money only for russian texts which are like scenarios in Story machine, not so impressive). I was contacted recently by another man, who raised almost million euros for his product at fundraiser, and he makes a Story-telling program, and he invites me to work with him (now I cant say his name). One another millionaire also 'blessed' me ("maybe you will become richest man in the world, and i't what I wish to you. Try with something small")... And yes, ane man from Vatican (theology advisor of Pope of Vatican) also blessed me "money is good for you". No, maybe it is just nothing, we need work very hard to make it done. Talent is nothing without hard work...

By the way, I was talking somewhere on official LEGO site about this idea, they liked it, but I didn't manage to talk to their people who are responsible for development of products which may be related to what I describe here...

I am not after money, million, and ready to make this program just for free. many hapy people who may be happy using this program is enough for me (if they want, they can donate me).

I wanted to make such program using that 29$ model from site above. I think I may make it as Android app, Windows app, Ubuntu \ Linux app, or even as Browser app.
it should be some simple program which takes text from user (maybe even big book, but more preferably children's book with many conversations, [in]direct-speeches etc). And then when user logs in, he should be able to do edits to text (it is "software as service"), use some "friendly AI" to find direct\indirct speech, who is speaker, find some simple animations like walk\run\go\fly\swim etc. but that would be next part. First we need to be assured that such program is required/wanted by end-users, and they are ready to pay for it. Well, I'm gonna make this rogram even just for me - to use it myself. But I'ld be happy if people from LEGO or anyone else to cooperate with me in this. I have to thank LEGO for providing officially amazing "LEGO MicroGame" tutorial.

But I just want to know if those LEGO minifigures compatible with common (humanoid rig) animations of "2500 Carnegie Mellon University"? If yes, can I apply them for "fun" purpose, say, educational, non-commercial, and publish maybe somewhere on Andoird market (google play) or anywhere else (Win\Linux\Browser) - does this idea sounds good? i mean I will set 0$, "FREE" price - and want to pubhish program like this somewhere - to find more fans, and then get feedbacks, and develop program which is better. Well, if that will become too cool, maybe we wil think of "presenting" it to LEGO... and is it possible to work (distantly) in LEGO on program like this? for Story-Telling?

I've got some modules already, they are very simple, but I want to convert them all into one LEGO-style (or if you don't like, I'll have to use Minecraft style, or stick-man style or invent something else, but unfortunately I don't have time, oney, and also designer skills for that, I'm not designer, and just learning programming)...
I think I have a cool idea, so it would be too bad if LEGO refuses to cooperate in this interesting project. But of course it's your last word, as it's your copyright.

In short, again: purpose of this project is automatically visualize all children's stories in the world (in English for now) - in LEGO style. Find Direct\Indirect speeches, speakers, visualize them in story-boards (I think without Text-to-spech synthesys, but if one wants, he may turn it on), and apply some common animations out of those 2500 from Carnegie Melon or similar. I know LEGO has more animations, so if they can provide them - it could be cool. Just look at site - https://www.wordseye.com - these people try invent something really great, but if you can buy them and make same in LEGO style, and do something like "Spinnakers Story Machine" in LEGO style, wihc speech-finders etc (to generate comixes etc) - it would be really cool. Only if you will do this program alone (wthout me), then please provide me a free (i ready to pay also, if that's good) version of this service...

I think if LEGO will buy (or make contracts with them to use) all the technologies which analize texts, do Part-of-speech-tagging, translate from one languages to other (and usualy Google Translate is here, so Google+LEGO must be friends, for analysis of texts), for visualization words (pictorial dictionaries), programs like Symbol Writer ( https://www.widgit.com/products/symwriter/index.htm ) - this would be cool... LEGO is the key between all these technologies, they lok very unconnected, so I dare to ask you to cooperate with all those mentioned companies, or similar technologies to make product/service which I describe here in much details...


P.S. Sorry for long letter. But hope this all sounds very interesting

Thank you in advance

 

You're in the wrong place. This is the Unity forum, not the LEGO forum. (And I doubt anybody from LEGO actually reads the LEGO forums, either.)

Also, please take the time to make your messages shorter. I am very interested in the topics you're talking about, and even I only made it about 1/3 through your wall of text.

 

But is there a way to contact those people from LEGO who create those LEGO MicroGame Tutorials on Unity Hub?
https://learn.unity.com/project/lego-template

And I see Unity themselves also publish some assets in asstes store which use LEGO assets
https://assetstore.unity.com/packag...publisher=Unity Technologies&q=lego&orderBy=1

So this makes me think that Unity3d and LEGO work together on developing those LEGO tutorials, LEGO assets in assets store... Am I wrong?

Do you mean to say LEGO are just among many other developers of Unity-courses, and Unity is not really to connected to them like to so many other companies, who also provide their tutorials/assets etc? But Unity themselves make LEGO assets in assets store. So there must be (it seems) much closer connection between Unity and LEGO

and, are there any people from LEGO on this forum? Do them use this forum? And how do those companies which develop Unity3d courses/tutorials/assets support their users? I see that some of them use this forum to present their assets/courses/tutorials etc, and talk to people like me. Is it not the case with LEGO?

And if probbly there is no connection between LEGO and Unity, then do I also have right to publish FREE projects which use LEGO assets from LEGO Microsgame tutorial?

 

My understanding is that when you open a LEGO microgame project from the Hub, you get a notice you have to agree to that indicates you cannot use the assets from that project in a commercial (for-profit) product. (Source here)

It also seems you can publish anything you make using these assets (as long as you're not selling it or making any revenue, like from ads or IAP), but you cannot distribute the assets yourself. Nor can you distribute content YOU create that utilizes anything that's part of an intellectual property you do not have the rights to. So you can, for example, make your own custom minifigs that work with the assets in the microgame project, but you can't share them via the Asset Store or anywhere else.

 

Yep. Clearly some deal was made between Unity and LEGO to enable those LEGO tutorials, but that's as far as it goes. They are still separate companies and to do what you want, you'd have to work with LEGO directly.