AR Remote Tool - Alpha Preview

I get the following error when importing the package.

C:/Users/Pedro/AppData/Local/Unity/cache/packages/staging-packages.unity.com/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Runtime/RemoteScreenComponent.cs(11,16): error CS0246: The type or namespace name `AsyncGPUReadbackRequest' could not be found. Are you missing `UnityEngine.Experimental.Rendering' using directive?

Was there another step I am missing?

 

2018.1.2f1

 

Lukasz, after successfully importing the package via your instructions above, when I switch my project's build platform to "iOS" in Build Settings I get the following compile error from one of the plug-in scripts:

/Users/kschultz/Library/Unity/cache/packages/staging-packages.unity.com/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/EditorRemoting.Utils.cs(47,17): error CS0103: The name `ipAddress' does not exist in the current context

 

Lukasz can you give little more info on how to get thos going please?
My android is connected to my pc running Unity 2018.2.13f1
I added the stuff in the manifest and I see the "AR Tools" in the menu. Opening the "Remote Window" show in status "Disconnected" Clicking "Connect" doesnt do anything

Trying to open the "RemotingApp" scene doesnt do anything (right click, open) and also (double click), seems to do nothing

I can't build it if I cant add it. What am I missing?

EDIT: Trying to open the RemotingApp scene gives me a warning: "Invalid AssetDatabase path", Use path relative to the Project folder.
No idea what that means

 

Hi Lukasz,

I'm using Unity 2018.2.0f2 (64-bit). The AR plane in Editor is look different from the actual phone.
please look at the attached image. the left side is from Editor, compare to the right one from phone. the plane suppose to be transparent. I'm not sure if I'm doing something wrong?

and the code I'm using to place prefab on Ar plane got error "NullReferenceException: Object reference not set to an instance of an object" in Editor. but can running perfectly on actual phone.

This is full error message.

NullReferenceException: Object reference not set to an instance of an object​

EditorRemoting.MockProxy.RaycastHandler (System.Single x, System.Single y, System.Byte flag) (at C:/Users/bodin/AppData/Local/Unity/cache/packages/staging-packages.unity.com/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/MockProxy.cs:201)
UnityEngine.Experimental.XR.XRRaycastSubsystem.Raycast (UnityEngine.Vector3 screenPoint, System.Collections.Generic.List`1[T] hitResults, UnityEngine.Experimental.XR.TrackableType trackableTypeMask) (at C:/buildslave/unity/build/Modules/XR/Subsystems/Raycast/XRRaycastSubsystem.bindings.cs:59)
UnityEngine.XR.ARFoundation.ARSessionOrigin.Raycast (UnityEngine.Vector3 screenPoint, System.Collections.Generic.List`1[T] hitResults, UnityEngine.Experimental.XR.TrackableType trackableTypeMask) (at C:/Users/bodin/AppData/Local/Unity/cache/packages/staging-packages.unity.com/com.unity.xr.arfoundation@1.0.0-preview.17/Runtime/AR/ARSessionOrigin.cs:202)
PlaceOnPlane.Update () (at Assets/Scripts/PlaceOnPlane.cs:75)

here this the code from PlaceOnPlane.cs line 75

Code (CSharp):

1. if (Input.GetMouseButtonDown(0))
2.         {
3.          
4.             if (m_SessionOrigin.Raycast(Input.mousePosition, s_Hits, TrackableType.PlaneWithinPolygon))
5.             {
6.                 Pose hitPose = s_Hits[0].pose;
7.  
8.                 if (spawnedObject == null)
9.                 {
10.                     spawnedObject = Instantiate(m_PlacedPrefab, hitPose.position, hitPose.rotation);
11.                 }
12.                 else
13.                 {
14.                     spawnedObject.transform.position = hitPose.position;
15.                 }
16.             }
17.         }

another question, is it possible to transfer "touch input" from phone to editor as well?

Thanks in advance & Best regards

 

Hi lukasz_pasek

Thank you for your fast response! I just sent the project via "Report a bug" from Unity.

can't wait for the new input system.It should help me save a lot of time for testing.

 

Hi Lukasz, again, I'm a newbie and need some more info to get this going. What I meant by the scene for remoting app was that I want to build the APK myself and not use the provided APK from your Google Drive.

Anyway, I didnt understand what to do so I used your provided APK, installed it on my phone via USB cable and ran the app. I get a green screen with an IP address and "Waiting for Connection". This just sits there and I get no option to change the IP. Back in Unity 2018.2.14f1 I use the menu "AR Tools" > "Remote Window". Type ID is empty with no option to change it. Clicking "Connect" does nothing. Status says Disconnected. I don't see anything here to type an IP address so not sure where I'm suppose to add the localhost address 127.0.0.1. Under [DEBUG] Helpers I click "Start Remote Device - Local, it opens a dos window and runs some command, then closes. Basically nothing seem to happen. No change ont he display on my phone, still says waiting for connection

Clicking play in the editor does nothing to the phone, and in Unity gives me a black game window. If I skip this remote stuff and just build my app it runs fine on my phone. Just dont understand to skip the build and get this remoting to work

 

@lukasz_pasek ?

 

@lukasz_pasek Any updates on the error posted by @krisschultz :

/Users/xxxx/Library/Unity/cache/packages/staging-packages.unity.com/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/EditorRemoting.Utils.cs(47,17): error CS0103: The name `ipAddress' does not exist in the current context

I'm seeing the same thing.

Thanks,
Sam

 

Hi @lukasz_pasek no I'm afraid it still doesnt work. My phone is conncted via usb cable and running the remote app on my mobile tells me waiting for connection and ip 127.0.0.1

I do everything as you do in the video, except I add 127.0.0.1 as my ip in the "Type ID Address" (by they way shouldent that say type IP address, not ID address?) Anyway, I press play in the editor then press connect in the remote window, but nothing happens, phone just stays with that green screen.

If I disconnect the USB cable, and switch on my mobile's wifi and then run the app, I get a different IP, in the range of my local network. If I use that address instead in the remote tool, I get same result, nothing happens

 

I'm seeing the same issue as krisschultz for the iOS platform, "The name `ipAddress' does not exist in the current context". For now, I just added a var before the line in question, and it seems to work OK.

I also had to change EditorRemoting.Connection.cs

private void RefreshCamera()
{
if (cameraToStream == null)
{
Debug.LogWarning("No camera to stream.");
}
else
{
cameraToStream.backgroundColor = Color.clear;
}
}


The video is displayed in the Unity Editor viewport, but not on the phone, and the image itself seems narrow, as if the image is cropped.

With regards anyone wanting to open the RemotingApp scene, it can't be opened from within the Packages folder, you can copy it into the Assets folder and load it from there. It seems to require having "Development build" checked.

When you're running the apps in the Editor, remember to load the other scenes, and you currently need to change the Platform so it's not iOS ( eg back to Windows )

 

I have an iphone SE and iphone XS. When I connect to iphone SE, it just freeze. But at least it is showing the camera. But on iphone XS, when I connect. It stayed at "waiting for connection" page.

 

@lukasz_pasek I am getting Connection Timeout error when trying to connect over Wi-Fi to Android:

SocketException: Connection timed out
System.Net.Sockets.TcpClient..ctor (System.String hostname, System.Int32 port) (at <3845a180c26b4889bc2d47593a665814>:0)
EditorRemoting.DirectConnectionProvider.StartListening () (at Library/PackageCache/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/Providers/DirectConnectionProvider.cs:37)
EditorRemoting.EditorRemoting.Connect () (at Library/PackageCache/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/EditorRemoting.Connection.cs:293)

When I try to connect through usb cable and input 127.0.0.1 ip I get:

SocketException: Could not resolve host '127.0.0.1 '
System.Net.Sockets.TcpClient..ctor (System.String hostname, System.Int32 port) (at <3845a180c26b4889bc2d47593a665814>:0)
EditorRemoting.DirectConnectionProvider.StartListening () (at Library/PackageCache/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/Providers/DirectConnectionProvider.cs:37)

 

When I try it on iPhone 7 I see this only this screen:


This is after applying fixes mentioned by @Mal_Duffin

 

I've had to make other changes here and there to the packages to try to fix some issues, I'm assuming that the next drop will be tested with iOS ( the errors to fix will show up fairly quickly ).

Is there any ETA on when the next drop might happen? It's been a month since the last drop, which is a long time to wait for people who are running into issues which may have been fixed in-house weeks ago.

It would be great if the packages could be checked out via a repo, so that fixes could be submitted quickly by developers keen to use this tool ( and those fixes downloaded by other devs ). Bugs and issues are understandable with new features like this, but having the ability for external devs to find, fix and share issues would be great.

 

Same issue as @krisschultz @shaddadjr85 got. Tried on 2018.2.8f1 and 2018.2.17f1, same problem. MacOS, iOS platform, preview 14

Code (CSharp):

1. /Library/Unity/cache/packages/staging-packages.unity.com/com.unity.xr.mock@0.0.1-preview.14/com.unity.xr.remoting/Editor/EditorRemoting.Utils.cs(47,17): error CS0103: The name `ipAddress' does not exist in the current context

 

@lukasz_pasek I'm still not able to get this to work. A reply would be appreciated

 

it works just with Foundation, right?
i'm trying to make the "old" ARKitRemote work.. but no way.. and i can't update my AR project to Foundation right now

 

Lukasz, that sounds great, I'm looking forward to trying it out!

Has it been tested on iOS devices?

 

i'm using the ImageTracker of ARKit plugin 2.0 (from bitbucket) so i can't use the AR Foundation (since image tracking isn't already integrated, right?

i don't want to mix "pure arkit plugin" with "AR Foundation".. unluckily the current ARKit Remote Kit doesn't work at all (Unity 2018.2.18 and ipad 2018/iPhone SE at least)

that's why i wanted to try this new alpha versions!

 

Thanks for the amazing update @lukasz_pasek .

Should we expect it by the end of the year or Q12019?

Thanks once again for the amazing work

 

but does the Remote tool (the otiginal one or this alpha one) really works for anybody?
i really can't make it work..

 

ahhh with 0.0.1-preview.10 something starts to work!
now i can stream from ipad to editor and see things...
with Foundation, yes (it didn't work before)

i'll try with ARKit plugin from Bitbucket.. do you think it'll work?

 

Hi,

I tried to get this working on my LG V30 and it somewhat worked. One thing worth mentioning to those who get compiler errors when starting i.e. arfoundation-samples after adding Remote Tool package - switch .NET Runtime vertion to 4.x instead of default 3.x.

Let me briefly summarize what I have done and what the outcome is so someone can hopefully comment whether I messed things up or is this tool just not playing well with my device, so:
1. Added RemotingApp into my app as scene at index 0.
2. Deployed via USB and launched my app to view my device IP
3. Opened an AR scene of my app.
4. Opened Remote Window
5. Pressed play in the Editor
6. Some cmd windows prompted, stuff happened, and kaboom I can see my camera preview in the Editor window.

What doesn't work:
1. I can't see anything on the device (the green background turned white/light grey and that's it).
2. Camera frame resolution/frame rate in Editor window is really bad. (get max 240p and 5fps).
3. Actual AR tracking (i.e. plane detection) does not work (dunno if that's even supposed to work?).
4. Get some runtime errors related to raw sockets and permissions so I ran Unity as Admin to get rid of those errors (nothing else has improved, unfortunately).

BTW, I am running Unity 2018.3.0f2, and RemotingApp scene runs without USB cable attached.

Thanks,
Łukasz

 

I didn't get the idea, what that remote actually does... I can't track any plane, so what the reason to use it?

 

Followed the instructions.
Implemented the packages into the manifest:

{
"dependencies": {
...
"com.unity.xr.arcore": "1.0.0-preview.18",
"com.unity.xr.arfoundation": "1.0.0-preview.17",
"com.unity.xr.arkit": "1.0.0-preview.14",
"com.unity.xr.remoting": "0.0.1-preview.10"
...

"com.unity.modules.xr": "1.0.0"
[OTHER DEPENDENCIES]
},
"registry": "https://staging-packages.unity.com"
}

I get an error while launching the project: "Unexpected token "O" at 43:3
[OTHER DEPENDENCIES]. No packages loaded."

If I get rid of [OTHER DEPENDENCIES], it loads the project but gives an error and I cannot access AR Tools. What Am I doing wrong? Having the latest Unity version.

"com.unity.xr.remoting@0.0.1-preview.10/com.unity.xr.remoting/Editor/Providers/DirectConnectionProvider.cs(44,17): error CS0246: The type or namespace name `CancellationToken' could not be found. Are you missing an assembly reference?"


UPDATE:
Was able to run it on ARexamples on my Android device. The package works but frame rate is VERY low.

 

Hello, I tried adding the necessary lines to the manifest, and Unity throws this error:

Code (CSharp):

1. /Unity/cache/packages/staging-packages.unity.com/com.unity.xr.remoting@0.0.1-preview.10/com.unity.xr.remoting/Editor/Providers/DirectConnectionProvider.cs(44,17): error CS0246: The type or namespace name `CancellationToken' could not be found. Are you missing an assembly reference?

I'm using 2018.2.11. Thoughts? Thanks.

 

getting this bug in the editor when the connection to the remote happens

Setting anti-aliasing of already created render texture is not supported!
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)

 

I'm also getting an error same as the person above:

/Users/Edwon/Projects/Puppetoon/Library/PackageCache/com.unity.xr.remoting@0.0.1-preview.10/com.unity.xr.remoting/Editor/Providers/DirectConnectionProvider.cs(44,17): error CS0246: The type or namespace name `CancellationToken' could not be found. Are you missing an assembly reference?

 

@lukasz_pasek would be great to get some response to questions above...

Just a note for dev team - imaging remote work as expected and will detect the planes (I believe that's the main idea of any ar remote tool) we strongly need some flag or method to define, is actually remote session is running or not to set logic dependency for tests

 

Just tested this on Android with Galaxy S9. While I am able to get a video to stream, not much else happens.

• no tracked planes
• resulting output only visible on Desktop, not on the mobile - that doesn't help much
• no input transmitted (no touches)
• approx. 1fps, approx. 3fps with downscale = 2
• if I go to downscale = 3 the Game View goes to Scale 30 and I see what looks like a 10x10px video stream... This is only resolved when restarting Unity. After looking further it seems the GameView is set to a resolution of 10x10px with "Remote Unknown". Manually setting the scene view to "Free Aspect" gives me an image again.
• blending of the camera image and the scene is totally wrong and doesn't seem to respect Alpha at all
• Unity crashes reliably (every single time) after I turned Autoconnect off and pressed play

That being said, Google's remote (in a different project, using their toolchain instead of AR Foundation) works pretty perfect, to the point where I nearly can't distinguish between Remote and Deployed when looking at my phone. Do you expect to get to that quality in a foreseeable timespan?

 

If you get the CancellationToken error that helios and edwon got, change the Scripting Runtime Version in Player Settings to .NET 4.x instead of 3.5

 

Is accessing camera image on CPU supported?
This code fails always to get latest camera image:

Code (CSharp):

1.  
2. private void OnEnable()
3. {
4.     ARSubsystemManager.cameraFrameReceived += ARSubsystemManager_OnCameraFrameReceived;
5. }
6.  
7. private void OnDisable()
8. {
9.     ARSubsystemManager.cameraFrameReceived -= ARSubsystemManager_OnCameraFrameReceived;
10. }
11.  
12. private void ARSubsystemManager_OnCameraFrameReceived(ARCameraFrameEventArgs e)
13. {
14.     CameraImage cameraImage;
15.     if(!ARSubsystemManager.cameraSubsystem.TryGetLatestImage(out cameraImage))
16.     {
17.         Debug.Log("TryGetLatestImage failed");
18.     }
19. }

 

I'm also having issues. Client displaying "waiting for connection" always. iPhoneXR, client built w/ Xcode v10.1.
Tried both USB and Wifi addresses.

Please help. Thanks.

 

Appreciate you taking the time to create and share this, really great you guys are pushing the boundaries of AR and unity development, cant wait to see what 2019 brings thanks @lukasz_pasek

For any beginners interested, I made a tutorial showing you how to set this up in Unity.

Its not perfect, but I hope the tutorial helps anyone new to AR remote and Unity.

 

Connecting either via localhost with USB cable or via IP over my wifi, on the Android phone I just get a fully white screen when connecting the app, and in the Unity Console I get an error:

Setting anti-aliasing of already created render texture is not supported!

UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

 

On Windows, and using an iOS device, I also need to take this this step to connect via the USB cable ( 127.0.0.1 ).

https://issuetracker.unity3d.com/is...missing-from-playbackengines-slash-iossupport

 

This problem ( from a previous post ) seems to still exist. It would be great to be able to use mouse clicks within the editor using the Remote tool.

 

Great tool @lukasz_pasek ! Thanks for making it available for us!

One tip for those who got blank/white screen. I saw that one too and got it solved but disabling "Development Build" setting. From what I see, the performance of this solution might be limiting the functionality and anything that can slow the app down, should be disabled.

Also, set scripting runtime version to 4.x and try to connect with WiFi instead of USB. I saw great improvement when I switched form USB to WiFi. I am on Unity 2018.3, working with Android Pixel 2 XL device.

It would be great to know what else can be disabled during debugging to speed it up.

 

Hey guys,
I have been able to connect the device to the editor via the 'AR Tools' menu, but even though it says connected the camera on my phone is not activated and there is no AR. Any thoughts or possible solutions?
Thanks, Joel

 

I'm having the same issue as @jatreutlein . When the device is connected via Wifi, the device stays on the "waiting to connect" green screen even though in the ARTools Remoting window it claims it has connected. If connected via USB, The green screen turns to white but I can't see anything. I've tried changing to .NET 4.0 as well.

I'm running Unity 2018.3 with LWRP 3.3