Search Unity

Showcase Unity as a Library rendered on partial screen with SwiftUI

Discussion in 'iOS and tvOS' started by Neonlyte, Feb 12, 2023.

  1. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    I tweaked some of the Unity iOS support code to make it work with SwiftUI as a UIViewRepresentable. Will share the process of getting there if people are interested.

     
  2. SKArctop

    SKArctop

    Joined:
    Feb 12, 2018
    Posts:
    38
    Yes Please!
    Can you put this into a github and share it? I've done a bunch of work to get this to work as a window above a swift UI, but it has it's issues. This seems much more seamless.
     
    Last edited: Mar 28, 2023
  3. SKArctop

    SKArctop

    Joined:
    Feb 12, 2018
    Posts:
    38
    @Neonlyte maybe also post it into the UAAL forum page. But please show us a github for it!
     
  4. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    I checked my changes and it's bits and pieces everywhere that I don't think I can just share a git repo and it'll be clear by itself. I'll do a write-up here instead and update the main post.

    The anatomy of an iOS app is essential to understand what we need to do. An iOS app would consist of one or more UIWindow, in each window, one or more UIViewController will be presented, and in each view controller, one or more UIView will be presented.

    Unity's UaaL initialization for iOS is essentially the same as if it was running as a standalone application, where on initialization, Unity's iOS "trampoline" code would create one of each UI components internally so that it can have exclusive control over those components, rather than reusing any of those from the host application.

    What we need to do is to hijack the creation of these UI components, and along the way, we will also need to move around some code to make things visible under Swift, so that we can embed the Unity View into SwiftUI.

    1. Prevent Unity creating its own window and view controller. A newly UIWindow will cause it to be placed on top of the host application, and you would have to manually place the SwiftUI app window in front of it again, or all touch events will be blocked by that new window. The UnityViewControllerBase contains code related to auto screen orientation, status bar and home bar, which we want to avoid letting Unity changing those.

    The following changes also removes snapshots VC and splash screen usage. My anecdotal observation is that removing the splash screen code actually makes the app launches perceivably faster even for non-UaaL purpose, as it seems that Unity would manually present the splash screen VC, even though iOS would present those automatically.

    Screenshot 2023-03-29 at 12.40.39 AM.png Screenshot 2023-03-29 at 12.40.55 AM.png Screenshot 2023-03-29 at 12.41.12 AM.png Screenshot 2023-03-29 at 12.44.30 AM.png Screenshot 2023-03-29 at 12.45.08 AM.png
     
    Last edited: Mar 29, 2023
    AlejMC and SKArctop like this.
  5. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    Screenshot 2023-03-29 at 12.53.18 AM.png

    This flag enabled auto-rotation which would try to access Unity's own view controller that would not exist, plus we automatically get auto-rotation when embedded into other view controllers.
    Screenshot 2023-03-29 at 12.59.10 AM.png

    This change may be optional as it's part of the auto-rotation code.
    Screenshot 2023-03-29 at 1.03.21 AM.png
     
    AlejMC and SKArctop like this.
  6. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    2. Make Unity View accessible in Swift. By default, Unity does not export the UnityView class, and its header file structure prevents Swift from importing the relevant methods and symbols.
    Screenshot 2023-03-29 at 1.07.01 AM.png

    Create a new header file called "UnityView+Private.h", and move the 3 private fields deleted in the previous image into here.
    Screenshot 2023-03-29 at 1.08.24 AM.png

    Then, update header includes
    Screenshot 2023-03-29 at 1.10.23 AM.png Screenshot 2023-03-29 at 1.10.14 AM.png
     
    AlejMC and SKArctop like this.
  7. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    Finally, export the header file in the framework umbrella header, and configure Xcode project to copy the header into the framework:
    Screenshot 2023-03-29 at 1.15.20 AM.png WX20230329-011455@2x.png
     
    AlejMC likes this.
  8. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    3. Integrate with SwiftUI in a way that SwiftUI Preview still works.
    Code (Swift):
    1. import SwiftUI
    2.  
    3. #if targetEnvironment(simulator)
    4. #else
    5. import UnityFramework
    6. import MachO
    7.  
    8. private var isUnityStarted: Bool = false
    9. func initializeUnityIfNeeded() {
    10.     guard !isUnityStarted else {
    11.         return
    12.     }
    13.  
    14.     UnityInstance.setDataBundleId("com.unity3d.framework")
    15.     UnityInstance.runEmbedded(withArgc: CommandLine.argc, argv: CommandLine.unsafeArgv, appLaunchOpts: [:])
    16.  
    17.     isUnityStarted = true
    18. }
    19.  
    20. private let UnityBundle = Bundle(path: Bundle.main.privateFrameworksPath! + "/UnityFramework.framework")!
    21. let UnityInstance = {
    22.     guard UnityBundle.load() else {
    23.         fatalError("Unity Bundle failed to load")
    24.     }
    25.  
    26.     let unityFrameworkInstance = UnityBundle.principalClass!.getInstance()!
    27.  
    28.     if unityFrameworkInstance.appController() == nil {
    29.         let machineHeader = UnsafeMutablePointer<MachHeader>.allocate(capacity: 1)
    30.         machineHeader.pointee = _mh_execute_header
    31.         unityFrameworkInstance.setExecuteHeader(machineHeader)
    32.     }
    33.  
    34.     return unityFrameworkInstance
    35. }()
    36. #endif
    37.  
    38. @main
    39. struct Partial_View_TestApp: App {
    40.     var body: some Scene {
    41.         WindowGroup {
    42.             ContentView()
    43.         }
    44.     }
    45. }
    46.  
    Code (Swift):
    1.  
    2. import SwiftUI
    3.  
    4. #if targetEnvironment(simulator)
    5. // Does not import UnityFramework for simulator
    6. #else
    7. import UnityFramework
    8. #endif
    9.  
    10. struct ContentView: View {
    11.     var body: some View {
    12.         VStack {
    13.             Text("Hello from Swift UI!")
    14.                 .padding()
    15.             UnityWrapperView()
    16.                 .aspectRatio(16/9, contentMode: .fit)
    17.             Button("Render") {
    18. #if targetEnvironment(simulator)
    19.                 // Does nothing if simulator
    20. #else
    21.                 UnityInstance.sendMessageToGO(
    22.                     withName: "Message Receiver",
    23.                     functionName: "Receive",
    24.                     message: "C"
    25.                 )
    26. #endif
    27.             }
    28.             .padding()
    29.         }
    30.         .padding()
    31.     }
    32. }
    33.  
    34. #if targetEnvironment(simulator)
    35. struct UnityWrapperView: View {
    36.     var body: some View {
    37.         Rectangle()
    38.             .foregroundColor(.blue)
    39.     }
    40. }
    41. #else
    42. struct UnityWrapperView: UIViewRepresentable {
    43.     func makeUIView(context: Context) -> UIView {
    44.         initializeUnityIfNeeded()
    45.      
    46.         let view = UnityInstance.appController().unityView!
    47.         view.removeFromSuperview()
    48.      
    49.         return view
    50.     }
    51.  
    52.     func updateUIView(_ uiView: UIView, context: Context) {
    53.     }
    54. }
    55. #endif
    56.  
    57. struct ContentView_Previews: PreviewProvider {
    58.     static var previews: some View {
    59.         ContentView()
    60.     }
    61. }
    62.  

    Make sure that the framework is not linked via the build phase.
    Screenshot 2023-03-29 at 1.23.51 AM.png

    Linker flag:
    Screenshot 2023-03-29 at 1.20.16 AM.png
     
    AlejMC likes this.
  9. SKArctop

    SKArctop

    Joined:
    Feb 12, 2018
    Posts:
    38
    @Neonlyte ! There's some great info there!
    This is a completely different setup than how I have it. I'll give it a spin soon and see if that helps solving some things.
    One question though, how does your UnityInstance object / file looks like? I'm using a version of the UnityBridge idea from the other thread. or is UnityInstance at that point imported from the library?
    I feel just by reading this that I'm missing something.
    Thank you for taking the time to write this up!
     
    Last edited: Apr 2, 2023
  10. SKArctop

    SKArctop

    Joined:
    Feb 12, 2018
    Posts:
    38
    I've tried replicating this in my own project, but i'm getting some odd error in an unrelated class (lifecycle).
    There are also some lines that weren't in my project, so my questions are:
    1. Can you please share this as a repo? This will be easier to compare where discrepencies are.
    2. Which version of Unity is this based off? I'm using 2021.3
     
  11. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    This is based off Unity 2022.2.10.

    I cannot share this as a repository because of the changes I have made is very fragmented, and since I can't reasonably bundle the IL2CPP sources and Unity library, you won't be able to build it to run anyway. All the screenshots above contained all of my changes from a clean export.

    The UnityInstance global variable is an instance of UnityFramework. You get that by calling [UnityFramework getInstance] in Objective-C. The UnityFramework class is the principal class of the UnityFramework.framework. See the first code chunk of this post on Wednesday at 6:26 AM.
     
    AlejMC likes this.
  12. SKArctop

    SKArctop

    Joined:
    Feb 12, 2018
    Posts:
    38
    Ok thanks. Might have to try a clean export out into a new project or something. I got all the changes down yet it won't compile due to some wierd error happening on the LifeCycleListener protocol. This happens after I add the last parts, of the view / view+private. All of a sudden I'm getting some errors in that class.

    upload_2023-4-4_12-23-13.png

    But, unfortunatly those errors don't make any sense, especially as this is a class that hasn't even been edited, and compiles correctly without the other changes...

    @Neonlyte. Thanks again for this write up and your help. hopefully this will benefit other users in the future. I'll post updates if I manage to get this working.

    Update : Did a clean XCode 14.3 project, with a clean Unity Project 2022.2.11, getting the same error. Looks like a dead cause atm unfortunately.
     
    Last edited: Apr 4, 2023
  13. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    You may have messed up a header file somewhere which can cause this kind of bogus errors. These changes are indeed not easy to replicate due to all the moving pieces and I was only comfortable doing it because my day job is iOS Development.
     
    AlejMC likes this.
  14. SKArctop

    SKArctop

    Joined:
    Feb 12, 2018
    Posts:
    38
    Yeah we use this in an IOS app ourselves however I'm new to it and never worked with ObjC. What irks me is that it happens with a file that isn't releated and I've double and triple checked. It has to be some odd order of includes that breaking it or something along those lines.

    A clean project and build also didn't work and i've gone through this multiple times, commiting each step and seeing where it breaks.

    Could be some config thing, xcode issue or god knows what, and at this point, I can't spend anymore time troubleshooting it. Unity really needs to step up and update their codebase at this point so we aren't relying on the community to solve these issues.

    In any case, @Neonlyte, thanks for the effort, maybe someone else will have better success and will be able to get it working.
     
  15. antol

    antol

    Joined:
    Feb 8, 2014
    Posts:
    4
    It's alive! It is working. @Neonlyte Thank you many times.
    One thing is that you missed, you have to add UnityFramework in General settings Frameworks, Libraries, and Embedded Content.
    And probably there a bit more things that can be cleaned up before making Framework, I will try and maybe post a github link with changes on empty project
    Screenshot 2023-05-23 at 12.31.20.png
     
    Agent0023 likes this.
  16. antol

    antol

    Joined:
    Feb 8, 2014
    Posts:
    4
    Oh. I forgot to mention that I have a small problem, probably it is related to headers as well. When I follow exact steps from your instruction I am getting an error (on the screen) Screenshot 2023-05-23 at 16.38.18.png

    If I comment that line - everything builds fine.
    BUT in Native iOS project where I am trying to use resulted framework I am getting an opposite error in that file that Screenorientation can’t be found. I have to go inside the framework (fortunately it's just a folder basically) and uncomment that line manually.

    May be the order of headers or quirks of compiler idk. But for future followers - worth mentioning
     
    Agent0023 likes this.
  17. Agent0023

    Agent0023

    Joined:
    Jul 31, 2017
    Posts:
    11
    Hello, I followed along and got stuck on the LifeCycleListener.h warnings identical to @SKArctop, how were you able to get past those?
     
  18. Neonlyte

    Neonlyte

    Joined:
    Oct 17, 2013
    Posts:
    516
    Glad you were able to make it work. I hope my posts have served the necessary inspiration.

    One comment on “Frameworks, Libraries, and Embedded Content”. The reason that I did not set it there is because I manually configured the build settings to achieve the same effect. That setting is usually convenience, but in my case it interferes with building SwiftUI in Xcode Preview, since the Unity Xcode project can only link with either Simulator or Device SDK at one time, so if I export the framework with Device SDK, it can’t link with Simulator targets, which is what Xcode Preview is underneath.
     
  19. antol

    antol

    Joined:
    Feb 8, 2014
    Posts:
    4
    I am not sure. I just followed the steps ¯\_(ツ)_/¯
    The only difference is that I made a universal framework (xcframework) which includes a build for simulator and device.
    I made two separate builds and after that, I joined the results via command:
    xcodebuild -create-xcframework -framework <deviceFrameworkPath> -framework <simulatorFrameworkPath> -output <xcframeworkPath>
     
  20. sheenarn

    sheenarn

    Joined:
    Nov 20, 2023
    Posts:
    3
    Hi, Can you please share the github link for the working copy of this development
     
  21. Agent0023

    Agent0023

    Joined:
    Jul 31, 2017
    Posts:
    11
    I was able to get it working this past summer using a combination of fixes I found from online several online sources - even get simulator previews working (using compile flags). Feel free to reach out if you need more assistance, but the crux of getting Unity to work along with SwiftUI is this custom Unity bridge class that exposes the Unity window as a regular SwiftUI view.

    Code (CSharp):
    1. //
    2. //  Created by Adellar Irankunda on 05/26/2023.
    3. //
    4. import Foundation
    5.  
    6. #if targetEnvironment(simulator)
    7. #else
    8. import UnityFramework
    9.  
    10. class UnityBridge: UIResponder, UnityFrameworkListener {
    11.     private static var instance : UnityBridge?
    12.     private let ufw: UnityFramework
    13.     private var observation: NSKeyValueObservation?
    14.  
    15.     public var api: UnityAPI
    16.     public var onReady: () -> () = {}
    17.     public var superview: UIView? {
    18.         didSet {
    19.             // remove old observation
    20.             observation?.invalidate()
    21.  
    22.             if superview == nil {
    23.                 ufw.appController().window.rootViewController?.view.removeFromSuperview()
    24.             } else {
    25.                 // register new observation; it fires on register and on new value at .rootViewController
    26.                 observation = ufw.appController().window.observe(\.rootViewController, options: [.initial], changeHandler: { [weak self] (window, _) in
    27.                     if let superview = self?.superview, let view = window.rootViewController?.view {
    28.                         // the rootViewController of Unity's window has been assigned
    29.                         // now is the proper moment to apply our superview if we have one
    30.                         superview.addSubview(view)
    31.                         view.frame = superview.frame
    32.                     }
    33.                 })
    34.             }
    35.         }
    36.     }
    37.  
    38.     public static func getInstance() -> UnityBridge {
    39.         if UnityBridge.instance == nil {
    40.             UnityBridge.instance = UnityBridge()
    41.         }
    42.         return UnityBridge.instance!
    43.     }
    44.  
    45.     private static func loadUnityFramework() -> UnityFramework? {
    46.         let bundlePath: String = Bundle.main.bundlePath + "/Frameworks/UnityFramework.framework"
    47.         let bundle = Bundle(path: bundlePath)
    48.         bundle?.load()
    49.  
    50.         let ufw = bundle?.principalClass?.getInstance()
    51.         if ufw?.appController() == nil {
    52.             let machineHeader = UnsafeMutablePointer<MachHeader>.allocate(capacity: 1)
    53.             machineHeader.pointee = _mh_execute_header
    54.             ufw!.setExecuteHeader(machineHeader)
    55.         }
    56.         return ufw
    57.     }
    58.  
    59.     internal override init() {
    60.         self.ufw = UnityBridge.loadUnityFramework()!
    61.         self.ufw.setDataBundleId("com.unity3d.framework")
    62.         self.api = UnityAPI()
    63.         super.init()
    64.         self.api.communicator = self
    65.         //self.api.bridge = self
    66.         self.ufw.register(self)
    67.         FrameworkLibAPI.registerAPIforNativeCalls(self.api)
    68.  
    69.         // runEmbedded will call the framework's showUnityWindow method internally
    70.         self.ufw.runEmbedded(withArgc: CommandLine.argc, argv: CommandLine.unsafeArgv, appLaunchOpts: nil)
    71.  
    72.         // Unity claims the key window, so let user interactions passthrough to our window
    73.         self.ufw.appController().window.isUserInteractionEnabled = false
    74.     }
    75.  
    76.     public func unload() {
    77.         ufw.unloadApplication()
    78.     }
    79.  
    80.     internal func unityDidUnload(_ notification: Notification!) {
    81.         ufw.unregisterFrameworkListener(self)
    82.         UnityBridge.instance = nil
    83.     }
    84. }
    85.  
    86. extension UnityBridge: UnityCommunicationProtocol {
    87.  
    88.     public func sendMessageToGameObject(go: String, function: String, message: String) {
    89.         self.ufw.sendMessageToGO(withName: go, functionName: function, message: message)
    90.     }
    91.  
    92. }
    93. #endif
    94.  
    You can then call the view and load it using
    Code (CSharp):
    1.  
    2. struct UnityView: UIViewControllerRepresentable {
    3.  
    4.  
    5.  
    6.     func makeUIViewController(context: Context) -> UIViewController {
    7.  
    8.         let vc = UIViewController()
    9.  
    10.  
    11.  
    12.         UnityBridge.getInstance().superview = vc.view
    13.  
    14.         UnityBridge.getInstance().onReady = {
    15.  
    16.             print("Unity is now ready!")
    17.  
    18.             //UnityBridge.getInstance().api.test("This string travels far, far away toward Unity")
    19.  
    20.         }
    21.  
    22.  
    23.  
    24.         return vc
    25.  
    26.     }
    27.  
    28.  
    29.  
    30.     func updateUIViewController(_ viewController: UIViewController, context: Context) {}
    31.  
    32. }
    33.  
    34.  
     
  22. soodonim

    soodonim

    Joined:
    Dec 6, 2023
    Posts:
    1
    EDIT: I figured it out, but will leave this problem here along with how I solved it in case anyone runs into the same issue.
    Solution:
    1) Tap on the "Data" folder underneath your Unity-iPhone project and make sure to tap the checkbox next to UnityFramework for Target Membership.
    2) Copy the UnityFramework folder underneath your Unity-iPhone project (it should have an Info.plist and a UnityFramework.h header file) and also put it in your Swift project, and make sure to add it to your Swift project Target Membership.

    This should now let you use UnityView() to open your Unity project from within your Swift project.

    -----------

    Hey mate! I am having some trouble getting this to work, but I think it might be because I'm missing some settings.

    What I did was create an .xcworkspace that includes both the .xcodeproj that I exported from Unity as an iOS project and my actual Swift iOS project. That was also the only way I could get UnityFramework.framework to even show up as a framework option to add in my main .xcodeproj, and then the "import UnityFramework" did work afterwards so I assume that part is correct.
    Screenshot 2023-12-20 at 11.58.55 AM.png Screenshot 2023-12-20 at 12.07.33 PM.png

    When I open the UnityView() from my app though after bringing it in like that in a ZStack, I get a this error:

    -> applicationDidFinishLaunching()
    [libil2cpp] ERROR: Could not open /var/containers/Bundle/Application/DAB3F643-3EEB-4ED2-B9FA-FC23CDBC80A2/Pawsitive.app/Frameworks/UnityFramework.framework/Data/Managed/Metadata/global-metadata.dat
    IL2CPP initialization failed
    2023-12-20 11:55:58.091949+0900 Pawsitive[845:72234] [client] No error handler for XPC error: Connection invalid

    I'm thinking that maybe the way I added the UnityFramework is incorrect because it seems to not have some crucial metadata along with it... So maybe what I'm not clear on is how to properly add in UnityFramework into the project?

    I don't have a Unity API key on Unity Personal rn, so I commented out those lines... so maybe that's part of the issue?
     
    Last edited: Dec 20, 2023
  23. avorna

    avorna

    Joined:
    Dec 6, 2023
    Posts:
    1
    Where is the `UnityAPI` coming from

     
  24. raviIOS

    raviIOS

    Joined:
    Mar 27, 2019
    Posts:
    13
    iam able to load the unity with the window. But with this there is a limitation that iam not able to customize the view frame. We have a requirement to load a custom frame (like CGRect(0,200, 300, 300)) of the unity view(UnityFramework.getInstance().appController().unityView).

    Issue that i face is:
    1. UnityFramework.getInstance().appController().unityView is always nil.
    2. Even UnityFramework.getInstance().appController().rootViewController is also nil.

    Can someone able to do this with unity v2021.3.8f1 or higher?