Lenovo/Disney "Jedi Challenges" — how to develop for it?

Just today came across this cool new Jedi Challenges product from Lenovo and Disney.

It uses what appears to be a custom headset. Like a Cardboard-style headset, it uses your phone as the computer and display; but a little like Hololens, it is see-through. They use a glowing ball you place on the floor in front of you as a tracking marker to keep the view stable; apparently there are a couple of cameras with fisheye lenses in the headset to locate this (and the lightsaber input device).

EDIT: This review has more detail than any others I've seen (and more detail than the official website, too).

All in all, a very slick system for $200.

Anybody know how/whether we can write more games for this hardware?

 

I checked the Lenovo Developer portal and forum and didn't see any information for "Developers". I'm guessing Lenovo will not be providing a SDK for the headset.

For $200, the Nolo VR Controllers look like the same technology, with A LOT more games (And you can develop for it, they have a SDK):
https://www.nolovr.com/

 

Hey, thanks for pointing that out, I hadn't seen it before! I agree, that looks pretty slick.

Of course it's not a complete system; you have to combine it with your own headset. But I guess those are pretty common.

My guess is, though, that the Disney kit will probably sell better. For tech geeks like us, the Nolo (plus some decent Cardboard clone) is the clear winner; it's more capable, and more open. But for the casual consumer, an all-in-one system that works right out of the box, with three cool officially licensed Star Wars games to boot, is an easier sell. Also, the see-through display of the Jedi Challenges headset is likely to be preferred by most casual users. (There's something rather intimidating about completely blocking your vision — you can't see what people around you are doing, you worry about bumping into walls and furniture, etc.)

But I do think the Nolo fits very neat niche; the world of Cardboard-style viewers needs better head tracking, and it definitely needs input devices. So I'll keep my fingers crossed for their wild success, and get ready to jump on board!

 

...But actually, it appears that Nolo only tracks head position, not rotation.

That makes it substantially more useless than I thought. Certainly you couldn't use a system like that with a see-through display... your images would be swimming all over the place. Inertial tracking alone just isn't good enough.

 

It's worth asking over here about developing for Lenovo Mirage AR, I'm wondering if it will ever be just 1 app "Star Wars" or if they'll open it up.
https://lenovodev.com/develop/Forum

It seems like Lenovo Mirage AR just doing VR/AR on the smartphone screen which means they are using camera based AR (Vuforia / ARcore / ARkit), and the controller (lightsaber) uses same tech as Nolo VR for Controller. So I would assume you can create the same experience using NoloVR and Vuforia / ARcore / ARkit.

Perhaps it has Google Tango or Intel Realsense some other AR sensors, but I doubt it as $200.

 

Fascinating stuff. Let us all know if you find out anything more!

 

Today I got the stuff from Best Buys, it's actually in stores.

I tested it and it is actually the best for the money under $200 one can buy; AR Headset, but what does it use?

I used it w/an IOS 11 device and it worked.

I am sure it works w/ Android and other devices too, clever clamping.

So how do I develop for it in Unity3D?

 

@Germanus4711, I assume you're talking about the Jedi Challenges kit.

By experimenting with it, can you tell whether it uses any other sorts of tracking sensors other than the camera phone? Does it even use the camera phone, or does it have its own camera(s)?

I still don't know how to develop for it, but if we can figure out the hardware, that's a good first step. I certainly agree that for $200 seems like a pretty awesome AR kit.

 

Any updates on this? I'm dying to know gents...

 

In this thread on the Lenovo forums, someone was told at CES 2018 that a developer SDK is in the works.

I can't wait! This thing is no Magic Leap... but for sub-$200, it looks like an amazing bit of AR tech. And it's great that (unlike some AR kits — I'm looking at you again, Magic Leap) they have designed it for gaming from the beginning.

I may just invest in a kit in the next few weeks, even if we can't develop for it yet, to get those creative juices flowing.

 

I've learned that the Jedi Challenges kit is based on tech from Ximmerse. Ximmerse also makes their own headset (the "Viewer"), though it's not nearly as polished as Lenovo's, nor is it actually in production as far as I can tell.

I wonder, though, whether the Ximmerse SDK might allow us to access their hardware in the Jedi Challenges system?

As far as I can tell, this hardware still occupies a unique and compelling place in the AR landscape. There are a few other systems out there with see-through AR, but they all rely on the phone itself to do image-based tracking. This is the only one with high-speed, reliable tracking over a wide FOV. If you want to play something like Beat Saber in AR (and I really, really do), you need a system like this.

Still no official word from Lenovo on an SDK, though.

 

Any word on this?

 

Nope. To be honest, I've kinda given up on these guys. They had a chance to beat Magic Leap to market. They failed at that, but with ML1 being so expensive and underwhelming, they still have a chance to carve out a decent niche — but not if this is how fast they move.

 

I got the Lenovo Jedi Challenge set on black Friday for $50 on Amazon. I was very impressed by the experiences. I had a demo of the MS hololense and was very disappointed by the range of vision. This set gave me almost a better range and a very similar experience. It would be amazing to have an SDK to develop for this set. Is very well build and it has great potential. I'm using it with my Galaxy S8, and the app performs well for up to 20 minutes continuously. I was afraid the phone will shut down due to overheating but it has not happened. Best the best 50 bucks I've spent so far. If Lenovo develops an app market for this, I would be the first one to develop and buy apps from it.

 

Yep. But they don't appear to be pursuing that, more's the pity.

 

no SDK?

 

Nope.

 

Hi Any new about this?

 

Still no.

 

I will try to see if i can get the camera listed somewhere on ARfoundation or Vuforia configuration. And maybe force a connection with one of those cameras, i really dont know whats in the middle. But i will keep you informed.

 

Any updates on sdk or anything else?

 

I was decompiling an APK. They apparently use Vuforia

 

That's interesting! But we'd still need access to the hardware (particularly cameras). Have you managed to figure out a way to get that?

 

Has anyone tried accessing the cameras through a pc?

 

I am joining the effort to develop for the Lenovo Mirage platform. Is there any more to this... Vuforia?
What about Webm playback?
Did anyone connect to the cameras?
I need to know more about the three beacons.

Help.. I must crack this thing and deploy to it!

 

Hi,
the headset (Lenovo Mirage AR) is basically an Ximmerse XHawk device, using onboard FPGA to process camera images and is sending only position data over USB and the lightsaber is some customized XCobra device using custom protocol over BLE. For the Jedi challenges app, the Ximmerse provided libxdevice.so and Java + Unity SDK, so without it, there is no chance to use the hardware.

Ximmerse development kit - XHawk with cameras + 2x XCobras:

 

Check out the latest posts on https://forums.lenovo.com/t5/Star-Wars™-Jedi-Challenges/Mirage-AR-Headset-SDK/m-p/3888638 - interesting stuff

 

I found the SDK Here, it's all there, JichyDev was correct:
https://unitylist.com/p/aan/Ximmerse-Unity-SDK
Video Tutorial Here:

I looked through the files and there seems to be a port to OSX, IOS, Google, and PC.
Disclaimer, I suck at unity. I'm still struggling with the basics.
I purchased four of these things and I would love to get a simple app running.

 

News on this? Did you make any Progress?

 

I couldn't just leave the headset to cover with dust, so I did few modifications to have the barebone SDK working in Unity 2021 LTS:

https://github.com/jichydev/LenovoMirageAR-Unity2021SDK

Maybe someone will find that usefull for lowcost AR.