Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

[Released] LLM integration in Unity!

Discussion in 'Assets and Asset Store' started by undreamAI, Jan 10, 2024.

  1. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Happy new year !
    We have just released LLMUnity, a package that allows to integrate LLMs (Large Language Models) in Unity !

    At a glance
    • Cross-platform! Supports Windows, Linux and macOS
    • Runs locally without internet access but also supports remote servers
    • Fast inference on CPU and GPU
    • Support of the major LLM models
    • Easy to setup, call with a single line code
    • Free to use for both personal and commercial purposes

    Setup
    You can install it as a custom package with the GitHub url (how to):
    https://github.com/undreamai/LLMUnity
    We have also submitted it to the Asset store and hope to have it there soon!

    How to use
    You can find our simple tutorial at Medium: How to Use LLMs in Unity
    and samples that you can try as part of the package (simple setup with package manager): Samples



    Let us know if you have any comments / questions / suggestions!
     
    Last edited: Jan 10, 2024
    Diegar, biscito, MS80 and 4 others like this.
  2. SpecsyGeek

    SpecsyGeek

    Joined:
    Jul 5, 2019
    Posts:
    6
    Cool stuff, is it possible for android?
     
  3. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    I haven't tested it yet on android, I have added it to the backlog!
     
    Mark_01 likes this.
  4. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,509
    tested in win10 + 2022.3.7f1,
    unity hangs when press Play.. (no errors)

    model is loaded.
     
  5. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    hi @mgear thank you for testing!
    I have sent you a message to understand more about the issue.
     
    Mark_01 likes this.
  6. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,509
    Ok, it was solved. Just posting here for info,
    I had space in project name so it messed things up. (and it will be fixed in coming updates)

    This is amazing:
    upload_2024-1-11_19-19-35.png
     
  7. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    thanks a lot @mgear for the debugging!!
    I just released 1.0.1 that solves this issue.
     
    Last edited: Jan 12, 2024
  8. Tubbritt

    Tubbritt

    Joined:
    Nov 30, 2015
    Posts:
    49
    Hello.
    Thank you for the work going into this. It’s really good. However I’ve noticed a problem I was wondering if you could help with.

    In the Inspector Prompt field I’ve set a persona for the AI character to act as. This works really great, but I’ve noticed if we talk for a while, it forgets everything I set in the persona.

    If I were to guess why, I would think that the contents of the Prompt field in the inspector only get sent to the AI on run time once and when the context size is exceeded due to continues conversations, it forgets who it is.

    Is that how it’s programmed right now to work?

    Traditionally, if you are working with GPT to do the exact same kind of role-playing AI chatbot, you have to send the persona every single time a message is being sent to the AI.

    Can LLLUnity be updated to work the same way? This would also help with injecting additional important information into the Prompt during run time that can carry on through our interactions. Such as current variable values.

    Kind regards
    James
     
  9. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Hi James, thank you for the post!
    I'll look into that and get back to you.
     
  10. Tubbritt

    Tubbritt

    Joined:
    Nov 30, 2015
    Posts:
    49
    Thank you for that.
    Sincerely, I appreciate the effort gone into putting this together. Having the AI run locally is quite a considerable advantage in countless ways. Not only $$$ cost savings on API calls, but quite simply the speed in response time. There’s also some very entertaining LLM’s out there that are fully open sourced and perfect for role-playing chats. Makes everything much more fun. Thank you.

    Kind regards
    James.
     
  11. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Thank you for the nice words!
    Yes that's exactly how I see it as well:).
    Do you have any examples of good role-playing LLMs? I'd be very interested!

    Regarding the issue:
    The default behavior of LLMUnity is to always keep the initial character prompt when the context is exceeded.
    In more technical terms:
    LLMUnity is built on top of llama.cpp that handles the prompts as follows:
    https://github.com/ggerganov/llama.cpp/discussions/1838#discussioncomment-6190738
    The "keep" variable of llama.cpp is set by calculating the number of tokens of the initial prompt (here).
     
    Mark_01 likes this.
  12. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,509
    is it possible to "embed" information (like those askpdf sites, where you upload file and then can ask about it?)

    this ones pretty interesting,
    https://play.aidungeon.com/
     
  13. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    You can add information as part of the chat message, like the following:
    upload_2024-1-18_15-9-52.png
    If you do something like this, you can also disable the history so that each message is independent from the other:
    _ = llm.Chat(message, HandleReply, ReplyCompleted, false);

    I'm not sure if I understood correctly, does this answer your question?

    Thanks for the link, I have seen this game it's very cool!
     
  14. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Hi all, we just released v1.0.2!

    Features
    - AMD support (switch to llamafile 0.6)
    - Start server on Awake instead of OnEnable
    - Code auto-formatting and automatic releases

    Fixes
    - Support Unity 2021 LTS
    - Fix macOS command
     
    Mark_01 likes this.
  15. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,509
    undreamAI likes this.
  16. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    I am implementing embeddings this week :)!
     
    JasonsFreeTime likes this.
  17. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    @Tubbritt I'm also working on a long-term memory for the model based on this solution (embeddings).
    It should help the model not forget when it has a large discussion / context.
     
    Mark_01 and DragonCoder like this.
  18. Tubbritt

    Tubbritt

    Joined:
    Nov 30, 2015
    Posts:
    49
    Sorry for the delay in replying. I was out of the country for a few days. Thanks for the information and continued work on this. Having the model never forget would be great as the role playing appeal to all this is amazing. It adds an excellent dimension to interactions with the LLM.

    Yes sure, if you want to try a role-playing LMM, check out the Deepseek LLM.
    https://huggingface.co/TheBloke/deepseek-llm-7B-chat-GGUF/tree/main

    I find this specific version is excellent for my needs.
    deepseek-llm-7b-chat.Q4_K_M.gguf

    It’s really good and it has a wicked sense of humour when you get the prompt / persona locked in.
     
    biscito and undreamAI like this.
  19. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    thank you, I didn't know that model!
     
  20. Aghawa

    Aghawa

    Joined:
    Apr 17, 2019
    Posts:
    6
    Amazing job!! Please allow me to use this on Android builds as well - life changing!
     
  21. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Thank you for the nice words!
    I did some testing on Android and it is a large feature.
    It is added in the backlog for the upcoming month(s).
     
    Mark_01 likes this.
  22. biscito

    biscito

    Joined:
    Apr 3, 2013
    Posts:
    138
    super cool asset !! I was looking for something like that !!!
    working fine in Mac
     
    Last edited: Feb 9, 2024
  23. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Hi! Can you install the Xcode command line tools and try again?
    If that doesn't work could you join our discord to resolve it?
    https://discord.gg/RwXKQb6zdv
    I'm trying to figure out why it works on some Macs and not others :/
     
    Mark_01 likes this.
  24. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Hi, The issue should be solved with the latest release!
     
    Mark_01, biscito and hopeful like this.
  25. JasonsFreeTime

    JasonsFreeTime

    Joined:
    May 23, 2015
    Posts:
    36
    This is an awesome asset!
    I made this quick chat UI using UI Toolkit and if you wanted to add this to your samples, it would be really easy.
    upload_2024-2-14_20-58-40.png

    These are the only files needed for the sample.
    upload_2024-2-14_21-9-58.png

    Very simple hierarchy:
    upload_2024-2-14_21-12-56.png

    And pretty simple script:

    Code (CSharp):
    1. using LLMUnity;
    2. using UnityEngine;
    3. using UnityEngine.UIElements;
    4.  
    5. public class ChatWithAI : MonoBehaviour
    6. {
    7.     LLMClient llm;
    8.  
    9.     UIDocument doc;
    10.     VisualElement root;
    11.  
    12.     ScrollView chatScrollView;
    13.     VisualTreeAsset promptTextFieldTemplate;
    14.     VisualTreeAsset responseTextFieldTemplate;
    15.     TextField currentResponse;
    16.     bool triggerUpdateScrollView;
    17.  
    18.     void OnEnable()
    19.     {
    20.         llm = GetComponent<LLM>();
    21.         doc = GetComponent<UIDocument>();
    22.         root = doc.rootVisualElement;
    23.  
    24.         promptTextFieldTemplate = Resources.Load<VisualTreeAsset>("PromptTemplate");
    25.         responseTextFieldTemplate = Resources.Load<VisualTreeAsset>("ResponseTemplate");
    26.  
    27.         chatScrollView = root.Q<ScrollView>(name: "ChatScrollView");
    28.         chatScrollView.Clear();
    29.         DisplayPrompt();
    30.     }
    31.  
    32.     void Update()
    33.     {
    34.         if (triggerUpdateScrollView)
    35.         {
    36.             UpdateScrollView();
    37.             triggerUpdateScrollView = false;
    38.         }
    39.     }
    40.  
    41.     void DisplayPrompt()
    42.     {
    43.         if (currentResponse != null)
    44.         {
    45.             currentResponse.isReadOnly = true;
    46.         }
    47.         var promptInstance = promptTextFieldTemplate.Instantiate();
    48.         var prompt = promptInstance.Q<TextField>(name: "Prompt");
    49.         prompt.RegisterCallback<KeyDownEvent>(ev => TriggerResponse(ev, prompt));
    50.         chatScrollView.Add(prompt);
    51.         triggerUpdateScrollView = true;
    52.         prompt.Focus();
    53.     }
    54.  
    55.     void TriggerResponse(KeyDownEvent evt, TextField prompt)
    56.     {
    57.         if (evt.keyCode == KeyCode.Return || evt.keyCode == KeyCode.KeypadEnter)
    58.         {
    59.             prompt.isReadOnly = true;
    60.             string userPrompt = prompt.text.Replace("\v", "\n");
    61.             var responseInstance = responseTextFieldTemplate.Instantiate();
    62.             currentResponse = responseInstance.Q<TextField>(name: "Response");
    63.             chatScrollView.Add(currentResponse);
    64.             triggerUpdateScrollView = true;
    65.             _ = llm.Chat(userPrompt, ResponseCallback, DisplayPrompt, true);
    66.         }
    67.     }
    68.  
    69.     void ResponseCallback(string message)
    70.     {
    71.         currentResponse.SetValueWithoutNotify(message);
    72.         triggerUpdateScrollView = true;
    73.     }
    74.  
    75.     void UpdateScrollView()
    76.     {
    77.         chatScrollView.scrollOffset = new Vector2(0, chatScrollView.contentContainer.layout.size.y - chatScrollView.contentViewport.layout.size.y);
    78.     }
    79. }
    80.  
    Let me know and I can send this to you.
     

    Attached Files:

    Mark_01 and mgear like this.
  26. JasonsFreeTime

    JasonsFreeTime

    Joined:
    May 23, 2015
    Posts:
    36
  27. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    wow thanks!
    This looks more elegant than the ChatBot sample :cool:!
    You can send me on Discord or you could create a PR and add it inside a folder in the Samples~.
     
    Mark_01 likes this.
  28. michael_unity387

    michael_unity387

    Joined:
    Feb 2, 2021
    Posts:
    8
    First off, thank you! This is so awesome and I'm ready to get playing with it!

    Second, I'm doing a webGL build, and it's not loading. I see the web build is about 4.42GB, so I'm guessing that's what's holding it up? Is there another way to speed that up?

    Follow-up: Cannot deploy on Android either. Seems to time out.
     
    Last edited: Mar 11, 2024
  29. michael_unity387

    michael_unity387

    Joined:
    Feb 2, 2021
    Posts:
    8
    In a macOS build (not webGL), why is it asking for access to my Documents folder?
     

    Attached Files:

  30. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    Hi, it doesn't work with Android / iOS / webGl at the moment.

    I don't know, where do you see this screen?
    Do you have the build stored inside the Documents folder?
    .
     
  31. michael_unity387

    michael_unity387

    Joined:
    Feb 2, 2021
    Posts:
    8
    Sorry, I was excited about trying it out!

    This screen pops up when I build and run on macOS. I do have the build folder in /Users/me/Documents/Unity/LLM/Builds .
     
  32. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    That's ok then, it runs the binary of the package.
     
  33. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    hopeful, Mark_01 and mgear like this.
  34. Mark_01

    Mark_01

    Joined:
    Mar 31, 2016
    Posts:
    646
    Thank you so much for making this and more over, making it free!! Even tho I can't code, I want to try this, when / if I ever get back to Unity in a few months. I hope this does well fro you and things are very bright for you in the future.
     
  35. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,509
    yeah, add some ko-fi or github sponsor support links too!
     
  36. undreamAI

    undreamAI

    Joined:
    Jan 2, 2024
    Posts:
    19
    thank you both :)!!
     
    Mark_01 likes this.
  37. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,509
  38. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    21,511
    Based on the size I'm thinking it's a typo and they meant Q5.
     
    undreamAI likes this.
  39. Glockenbeat

    Glockenbeat

    Joined:
    Apr 24, 2012
    Posts:
    670
    This looks fantastic, thank you so much for this. One question - you mentioned embeddings support earlier. I think this is a crucial part to give the LLM game specific context and abilities. Is this still something you are working on?