Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

continue running app in background for audio generation (iOS) ?

Discussion in 'iOS and tvOS' started by clinton_d, Nov 3, 2011.

  1. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Hi.

    An important feature of my Unity app is to continue running in the background after the person has pressed the HOME button.

    iOS allows this to happen, even with OpenGL apps, provided:

    - The app is designated as requiring background execution (eg, audio app).
    - The app does not make OpenGL calls whilst in the background.

    However Unity does not allow this to happen, as pushing the HOME button gracefully suspends the app. This occurs regardless of how the Info.plist is configured (eg, designating the app as an audio app that should continue in the background).

    This poses a question.

    Does anyone have thoughts on how Unity could be forced to continue background execution, given that it appears to be intentionally preventing background execution at present?

    I have spent a week trying to modify the XCode project to this effect, although I have not managed to keep Unity executing.

    It should be noted that my app uses the physics side of Unity to generate Audio; customers are constantly requesting the ability to play the app in the background, although my hands are tied by what Unity allows.

    Any thoughts/clues/comments aimed at making Unity run in the background would be extremely appreciated.

    Thanks :)
     
  2. OneThree

    OneThree

    Joined:
    Oct 28, 2011
    Posts:
    181
    Has there been any luck on this front? I would love to have background audio keep running in an app I'm currently working on.
     
  3. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,851
    Hi,
    this is quite complicated case. It should be quite possible to get native part of the app running in background and having unity runtime suspended. You just need to add appropriate info.plist keys and modify required callbacks on the AppController to execute your code instead of Unity player loop. Display Link should be disabled and some other main loop mode should be used.
    Your case is more extreme, because by default Unity runtime tries to render to OpenGL and is not appropriate for background applications. You might try one trick:
    a) disable DisplayLink, choose other main loop mode.
    b) when entering background instead of calling UnityPlayerLoop() function declare and call following one:
    bool PlayerLoop (bool, bool);
    Call it with first and second parameters set to true: PlayerLoop(true, true);
     
  4. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Hi Mantas

    Well, PlayerLoop() was the magic ingredient! I can confirm that background execution takes place.

    There seems to be one problem though, the audio is muted when the app enters the background. I imagine that Unity is handling this as part of its graceful 'enter background state' process.

    Is this the case, or should audio continue? Perhaps there is an audio function I need to call (to increase volume?).

    Thanks for your help thus far. Once I get it all working I will post example code for others to us.

    :)
     
  5. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,851
    I believe this might be related to the way how Audio Session is set-up. Have you tried checking "Override iPod Music" checkbox in Player Settings?
     
    A_never_kill likes this.
  6. J_P_

    J_P_

    Joined:
    Jan 9, 2010
    Posts:
    1,021
    I'd love more details about how you pulled it off using PlayerLoop. I'm also trying to get audio to continue when the user goes back to the home screen, but it's been hard since I'm not familiar with Objective C.

    I'm not sure what audio session category "Override iPod Music" uses exactly, but if it doesn't work you'll probably have to change it to Playback manually. But maybe we'll get lucky and "Override iPod Music" already does that (I haven't been able to test because as you found, Unity pauses itself -- which is why it'd be great if you could give me a bit more info on your solution).

    See: http://developer.apple.com/library/...SessionCategories/AudioSessionCategories.html
     
    Last edited: Jan 31, 2012
  7. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    @mantasp

    I've tried the 'Override iPod Music' checkbox (and confirmed the ipod was being overridden), but still no luck.

    So right now I can still confirm that with PlayerLoop unity continues to bubble away in the background once the app has been sent to the background, but sound is gracefully muted when entering the background state; regardless of the 'Override iPod' setting.

    @JTown

    As things stand, it's not fully working and from the looks of things to get it working I need pointers from Mantas, as he knows the hidden functions I need to call :)

    Once it's all sorted I will post my complete solution, but the jist of it is:

    - Designate the app as an 'audio' app within Info.plist.
    - Declare PlayerLoop (as outlined earlier in the thread).
    - Within 'applicationDidEnterBackground', call PlayerLoop within a managed loop (for testing, this can simply be a 'while' loop).

    The first two are easy. The third one is far from easy, and im still working on making my approach robust.



    Hopefully getting the audio volume/continuation sorted is not too tricky, as everything else is pretty much working!
     
  8. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,851
    Then I guess it requires MEDIAPLAYBACK audio session category, which we currently don't support.
     
  9. J_P_

    J_P_

    Joined:
    Jan 9, 2010
    Posts:
    1,021
    Is there a possible workaround? Some way we could manually force Unity to use the correct category for its audio session?
     
  10. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,851
    No, unless you implement your native plugin, which would initialize audio session and play sounds directly via Apple framework.
     
  11. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Hi. Reigniting this thread in the hope that there has been some further consideration regarding the audio session types which Unity supports.

    Surely adding the ability for developers to choose between 1-2 audio categories is not such a big deal? Because I have to say, this limitation is really starting to hurt me ...

    And I'm really amazed this is not such an issue for many developers - perhaps they are yet to think of the awesome apps you can create with Unity running in the background ;)
     
  12. Ostwind

    Ostwind

    Joined:
    Mar 22, 2011
    Posts:
    2,775
    Its a big deal because Unity is not designed for such thing and background audio needs to be as light weighted as possible. Running Unity audio on background means Unity running too. Also such feature is really rare among game developers and its platform feature which makes it really more native plugin than core feature.
     
  13. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Hi Ostwind.

    I totally understand that Unity running in the background is a big deal, as the CPU/memory overhead 'could' be huge. We see that in many other apps however, so the idea of a heavy bg app is not unique to Unity (the Inception App is a good example).

    In my experience though, I have seen Unity run quite efficiently in the background, using only around 10-20% CPU (on an iPad first gen). This is less then what the app consumes in the foreground (30-50% CPU). Once you cease drawing to the OpenGLES buffer the load drops significantly.

    I guess I have a question: Do the Unity dev guys think this limitation can be circumnavigated with a custom plugin? I have no idea on how I would implement such a thing (I'm a Unity based developer, not an XCode one). If it's possible, some pointers on how to achieve this would be appreciated.

    cheers.
     
  14. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Some how this thread has been forked into two separate threads ... Anyway, some responses to the forked thread:

    AdbC99 wrote @ 04:37 AM 12 Hours Ago
    Am another developer needing this, where you are incorrect above is that unity does not gracefully suspend the app when audio is playing, frequently it crashes in such a way as to indicate time has stopped in the engine. As a result I use th kill on suspen option. Anyway would b good if unity could fix this; bu for now I haven't seen a fix to an audio bug in a long time
    M

    Ostwind wrote @ 05:19 AM 12 Hours Ago
    For what task you need the app to be in the background? only the audio? Apple has its own requirements for apps running in the background and what functions they need to use when backgrounded. If I recall correctly its already been said that the way Unity is done and to maintain decent cross-platform stuff its not possible to use Unity audio engine for any background tasks and you must roll out your own code via objc to do things way Apple wants and just pause Unity.
     
  15. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    @AdbC99 - Not quite sure what exactly you are referring to, though it sounds like an aside to the core issue of this thread.

    But I've got an update on my adventures into getting this to work.

    So, with some assistance from an objective C developer, I have managed to run Unity in the background, with Audio. Yep. As Mantas mentioned, it required a native audio plugin to be created, and for ALL audio calls within my app to be handled by the native player. This requires an immense amount of re-engineering of code that is written within Unity, but it can be done ... To a point.

    The one issue I have encountered, is that after 10 minutes of the app commencing in the background, the Audio crashes (no sound can be heard any longer), yet the Unity timeline continues.

    I have a question for @Mantas: To avoid people like me spending thousands of dollars on additional development work, chasing an end-goal which might not actually be possible at present, can Unity simply be made to handle a different AudioSessionCategory ?

    I can't imagine having a switch which allows developers to toggle between different audio sessions would be that hard for Unity to implement (Im sure there are other complexities it introduces, but still...).

    Thoughts always appreciated.
     
  16. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Giving this a nudge so someone from the dev team might chime in.

    What are the issues with Unity allowing dev's the ability to control the AudioSession category ?
     
  17. AdrienTwnkls

    AdrienTwnkls

    Joined:
    Nov 13, 2012
    Posts:
    3
    @Mantas @clinton_d

    Hi guys, I'm trying the method mentioned above to keep my app running in the background on iOS but can't seem to declare and call the PlayerLoop method as described. I get the following Apple Match-O Linker error:

    Undefined symbols for architecture i386:
    "PlayerLoop(bool, bool)", referenced from:
    ___47-[AppController applicationDidEnterBackground:]_block_invoke_0224 in AppController.o
    (maybe you meant: __Z10PlayerLoopbbP10IHookEvent)

    It looks like PlayerLoop has a third parameter of type IHookEvent but have no idea how to get this Interface in objective-c.

    Some info on this app; it's a utility app for personal use in my Unity workflow. It will not need to be submitted to the App Store and no audio will need to be playing in the background either. I will however need to use the gyro attitude and RPC.

    Any help would be greatly appreciated.

    Cheers,
    Adrien
     
  18. emperor1412

    emperor1412

    Joined:
    Jan 30, 2012
    Posts:
    26
    Is there a way for Unity networking to run in background, my game is using Unity built-in networking and when I press HOME button and come back then the connection get lost and lose my current match.
     
  19. Hotshot10101

    Hotshot10101

    Joined:
    Jun 23, 2012
    Posts:
    112
    I would like more information about this PlayerLoop call and disabling display. My app gets Bluetooth data and right now I just queue up the incoming data until the app is not paused anymore.

    If possible I would like to have the incoming data go all the way up to Unity and get processed. Right now Unity is paused, so I just queue up the serial data.

    Since the protocol I am using is pretty complex I don't want to have try to process it in ObjectiveC. This is why I just queue it up for now.

    Then I read this posting and thought maybe there is hope. All I need to be able to do is send the data up to Unity (which I just use the UnitySendMessage with Base64 encoded data) and have a way to unpause the serial processing of the data I just sent.

    Where can I get documentation on PlayerLoop or some other way to do this?
     
  20. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    811
    there is none
    running in backgroud is simply not supported as of now; the PlayerLoop bit is kind of hackish and they even seem to change [it's] internal API in 4.x ;/
     
  21. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,851
    Yes, PlayerLoop now has three parameters, the third one is pointer to IHookEvent class (default value is NULL); You can declare method signature as follows:
    Code (csharp):
    1.  
    2. class IHookEvent;
    3. bool PlayerLoop (bool, bool, IHookEvent*);
    4.  
     
  22. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    811
    Thanks for the update!
    Just tested it, behaves as expected :)
     
  23. Dsp

    Dsp

    Joined:
    Feb 20, 2013
    Posts:
    2
    hai..
    im new in this ios development, but my problem is same, now i dont know how to implement the playerloop(bool,bool)
    could ya tell me the detail?
    thanks much... :)
     
  24. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Hi Mantas

    Any update on when we can control the AudioSessionCategory ? There is another limitation with the way Unity currently handles audio - if the device is MUTE, Unity does not generate sound. This is pretty much inconsistent with every other app on the app store. As MUTE is designed to silence the phone functions, not apps.

    Additionally, given I've managed to get the app working in the background for up to 10 minutes, having control over the audio session type would probably get me across the line.

    thanks.
     
  25. J_P_

    J_P_

    Joined:
    Jan 9, 2010
    Posts:
    1,021
    I expect apps to be muted if I have my phone on silent and I think most apps are like this. Nevertheless, one of my own Unity apps is NOT like this -- Cake Day uses the audio session that allows mic input and audio output simultaneously and it ignores the mute switch (annoyingly). So it's definitely possible.

    http://forum.unity3d.com/threads/30194-iPhone-microphone?p=476394&viewfull=1#post476394 is what I based my implementation on, though I have no idea if it works with other audio categories.
     
    Last edited: Feb 23, 2013
  26. AdrienTwnkls

    AdrienTwnkls

    Joined:
    Nov 13, 2012
    Posts:
    3
    @Mantas; Wow, completely missed your reply. Thanks for the help!

    Cheers,
    Adrien
     
  27. AdbC99

    AdbC99

    Joined:
    Jan 18, 2012
    Posts:
    52
    Has anyone on here tried doing an Audiobus integration yet? I'm doing synthesizers in Unity3d, I know I'm mad, but I wanted particle generators in the synths and for them to be cross-platform so used Unity.
     
  28. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,804
    We're constantly hassled to get audiobus support into Physynth and I really, really would love to update Physynth with this support + loads of new sounds for our loyal fanbase so I will keep watching this thread for news on Audiobus.
     
  29. jonlab

    jonlab

    Joined:
    Apr 6, 2010
    Posts:
    182
    I'm interested in an Audiobus support in Unity as well !
    I'm afraid it would require to completely bypass Unity audio engine and roll a new native one (with iOS low level Audio Units).
    http://developer.audiob.us/
     
  30. Ippokratis

    Ippokratis

    Joined:
    Oct 13, 2008
    Posts:
    1,514
    I am very interested in finding a way to support audiobus too.
     
  31. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,027
    Me 3 on audiobus support
     
  32. ugur

    ugur

    Joined:
    Jun 3, 2008
    Posts:
    681
    I have a sorta related issue, i'd like to run unity onwards on iOS when pushed to the bg (but without rendering side) to still be able to get and react to location updates.

    I know i should add
    <key>UIBackgroundModes</key>
    <array>
    <string>location</string>
    to the Info.plist, where it gets all merky to me is how to change things in the xcode project to make unity run onwards without rendering when getting the pushed to the bg event.
    I couldn't find any documentation on how to change the UnityPlayerLoop or use the other PlayerLoop really.
    Could someone who has done that already shed some light on those parts?
    Like where you add:
    class IHookEvent;
    bool PlayerLoop (bool, bool, IHookEvent*);
    what are the params for that method, how and where to implement it; where and how to switch forth and back between the usual Unity Loop and a version not running the render side?
     
  33. ugur

    ugur

    Joined:
    Jun 3, 2008
    Posts:
    681
  34. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    811
    you have to start a timer which updates player loop when entering background AND don't pause unity
    it has to be done in app delegate ( AppController.mm or whatever controller scheme currently Unity employs :)

    Code (csharp):
    1. class IHookEvent;
    2. bool PlayerLoop (bool, bool, IHookEvent*);
    3.  
    are just prototypes, implemented elsewhere; after declaring them like that, you can call them
    e.g.

    Code (csharp):
    1. class IHookEvent;
    2. bool PlayerLoop (bool, bool, IHookEvent*);
    3.  
    4. NSTimer *aTimer;
    5.  
    6. @implementation AppController
    7.  
    8. ...
    9.  
    10. - (void) applicationDidBecomeActive:(UIApplication*)application
    11. {
    12.     printf_console("-> applicationDidBecomeActive()\n");
    13.     if (_didResignActive)
    14.         //UnityPause(false);
    15.         [aTimer invalidate];
    16.  
    17.     _didResignActive = NO;
    18. }
    19.  
    20.  
    21. - (void) applicationWillResignActive:(UIApplication*)application
    22. {
    23.     printf_console("-> applicationWillResignActive()\n");
    24.     // UnityPause(true);
    25.     aTimer = [NSTimer scheduledTimerWithTimeInterval:1.0 target:self selector:@selector(UpdateUnityInTheBackground) userInfo:nil repeats:YES];
    26.  
    27.     _didResignActive = YES;
    28. }
    29.  
    30. - (void) UpdateUnityInTheBackground
    31. {
    32.     PlayerLoop(true, true, NULL);
    33. }
    34.  

    now, in case of audio - since this is an audio thread - things get bit more complicated

    I wanted to act as remote controller - in order to be notified of playback events such as when user uses audio player controls from the lock screen;
    problem is Unity uses / or used at the time / audio category which is / wasn't suitable for this kind of usage ( things might have changed since then )
    So I had to manage the whole audio playback by myself and apart from required Apple setup I also had practically completely unplug unity audio via
    Code (csharp):
    1. UnitySetAudioSessionActive(false);
    in the end of startUnity method I believe, in order to actually receive the events

    [ that wasn't a problem since I did it for an audio ( mod -) player, but I can imagine is completely unusable for common applications/games using Unity audio ( again, things might have changed... ) ]

    You might run into similar various issues with location services, but the basic idea of player loop updating outlined above works ( or, at least, worked at the time.. )

    edit: you also have to disable display link as Mantas mentioned earlier in the thread
     
    Last edited: Mar 29, 2014
  35. ugur

    ugur

    Joined:
    Jun 3, 2008
    Posts:
    681
    hello there,
    thanks a lot for this, that helped me a lot =)

    for anyone else trying this:
    -slight issue when you pasted the code, in your method names application was somehow altered to applicatio n
    -one still has to change the Info.plist, too of course to add something like
    <key>UIBackgroundModes</key>
    <array>
    <string>location</string>
    </array>
    in case of wanting location updates while in bg.

    Now while it works for me now and that is very cool, i'd like to better understand it.
    The part that is not clear to me is the PlayerLoop (bool, bool, IHookEvent*); part. I mean what are the two bool params for?
    I read somewhere that it would be PlayerLoop(batchMode=false, performRendering=true, pHookEvt=0x00000000)
    is this correct?
    If so, why should one pass true as value when wanting to run in the bg but with unity not doing the render update while in bg?
    Also what is batchmode about?
     
  36. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,851
    That's probably my mistake. I think you should be calling it with PlayerLoop(false, false, NULL) while in background..
    As for location tracking in background, most likely you should find location service callback in iPhone_Sensors.mm:
    Code (csharp):
    1.  
    2. // Delegate method from the CLLocationManagerDelegate protocol.
    3. - (void)locationManager:(CLLocationManager *)manager
    4.     didUpdateToLocation:(CLLocation *)newLocation
    5.                    fromLocation:(CLLocation *)oldLocation
    6. {
    7.  
    8.  
    and put PlayerLoop call here. (I'm assuming you have outcommented pausing of unity while entering background).
     
  37. ugur

    ugur

    Joined:
    Jun 3, 2008
    Posts:
    681
    Hello Mantas,
    i got it working in the meantime, but cool, thanks, i'll improve the implementation based on your input =)
    our app basically allows the user to use gps constantly for some stuff or not use it in that moment when in a different mode.

    So based on that, if the user was in the mode of constantly tracking gps (like a navigation system) while the app is pushed to the bg, i let it running onwards then, but if the user was not in that mode of constantly using the gps i do location.Stop()
    this then makes the app paused while in background (since no background activity is in use then).

    So that all worked cool, besides one thing:

    When stopping the gps (when going to the background and user not needing gps constantly in that mode) and then restarting it when coming to the foreground again, we receive crap unprecise gps info for several seconds/gps updates (despite having set a short range and high precision target).

    Do you know any way to get around that and make it deliver good values right away again when restarting gps?
     
  38. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    811
    would have to try it out, sometime, doh !
    I guess that's what we have dealing with not-so-straightforwardly-supported APIs
     
  39. uniphonic

    uniphonic

    Joined:
    Jun 24, 2012
    Posts:
    130
  40. F.Salka

    F.Salka

    Joined:
    Dec 13, 2013
    Posts:
    43
    Is it possible to integrate inter-app audio? As far as i understand it doesn't requier the app to support background audio..so it should be possible with unity3d?
     
  41. uniphonic

    uniphonic

    Joined:
    Jun 24, 2012
    Posts:
    130
    F.Salka, why wouldn't it require background audio? Did you hear that somewhere?
     
  42. F.Salka

    F.Salka

    Joined:
    Dec 13, 2013
    Posts:
    43
    The documentation regarding inter-app audio are vague, the only resource available is the wwdc lecture about IAA, i'm guessing it could be possible to send audio streams from unity3d through IAA but will only possible if the node app (unity) is running in the foreground, and the host app (Garageband) is running in the background (sort of limited foreground only IAA support), the IAA examples Apple provides on its website provide functions to handle certain situations where the app can't run in the background (and it will still be handled by IAA)...
     
  43. clinton_d

    clinton_d

    Joined:
    Feb 11, 2011
    Posts:
    17
    Jumping back in to this thread. Can't believe it's been 2.5 years since I started it... A little disappointing that's it's still not solved to be honest.

    @Mantas Puida

    Has there been any consideration inside the dev team regarding the ability to officially support 'play in background with audio' functionality ?

    There is a whole world of unexplored apps that unity developers could be creating with this feature. It would move unity devs past games and into really creative places (I know as I have many ideas for such apps).

    How do we go about making this happen? The endless stream of emails from users of my app asking 'why doesn't it play in the background' actually keeps me up at night (around 100 emails this week alone).

    Please please please help (see, I'm begging now!).

    Clinton
     
  44. Batigol

    Batigol

    Joined:
    Oct 17, 2012
    Posts:
    233
    I also need it!
     
  45. Moghul

    Moghul

    Joined:
    Sep 8, 2013
    Posts:
    2
    Hello, I also need this for a running game that I am developing. It needs to be able to play sounds, get gps signals and read pedometer data after pressing the home button or the button at the top of the phone.

    We had the hackish workaround mentioned here until yesterday when, after "upgrading" to iOS 8/xcode 6.0.1/unity 4.5.4, the app would no longer pass the Splash Screen and would eventually crash at the PlayerLoop(true, true, NULL); line.

    The person who wrote the workaround is not available for consultation so I'm asking you guys: how did you trick iOS/unity into letting you do all that in the background?

    Thank you very much for your time.

    PS we're 99% done and this is practically the only thing plaguing us. Redesigning the whole sound system from scratch is NOT an option.
     
    Last edited: Sep 24, 2014
  46. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    811
    PlayerLoop should not be called at startup / splash screen in the first place - it''s supposed to keep unity alive while the app is in the background, i.e. in completely different situation
    There might be a bug in iOS 8 or so I've heard, but I'm not sure - dont have iOS 8 device right now
     
  47. menevse

    menevse

    Joined:
    Jul 18, 2014
    Posts:
    2
    Hello @r618

    I tried to do what you have said but still could not figure out to solve audio part. My whole app based on audio. I need to figure out how to keep running audio in the background.

    Is there anyone to help around, it would be appreciated.
    Thanks.
     
  48. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    811
    it's not possible, unity doesn't play audio in the background, i used independent audio playback libraries
     
  49. Zocker1996

    Zocker1996

    Joined:
    Jan 12, 2015
    Posts:
    20
    so, whats this libary called?
     
  50. RP1

    RP1

    Joined:
    Oct 12, 2012
    Posts:
    2
    Hi guys,
    Does anyone know if this was addressed? I am trying to build an audio heavy app with unity where I need to be able to continue playing the audio in the background. I set the player settings to Audio/Airplay but it still stops when I press the home button.