Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Custom Input & Output Source Integration

Discussion in 'Formats & External Tools' started by Max_Bol, Jan 24, 2018.

  1. Max_Bol

    Max_Bol

    Joined:
    May 12, 2014
    Posts:
    168
    First, for the sake of making things simple, whenever I'm mentioning Custom Input & Output Source Integration, I will use the acronym CIOSI. It will make discussion simpler.

    What is CIOSI? It's the name I have given to the concept of having an external device that isn't just a keyboard and/or a mouse. You could call it a "custom controller" and it would fit right into the subject, but it can be much more than that as it could also fall into something like VR and AR. To put it simple, it's about either simplified or more complex controlling setup being developed for and in Unity.

    It can be a simple controller like an Atari's old stick controller:


    Or it could be a controller like Steel Battalion:


    It's nothing new, but it's still kinda underground where you see something pop-up now and then.
    For example, in 2012, this popped out at E3 2012:



    Now, why I don't just call it "a custom controller" is for 2 reasons. First, in the future (who knows), using the term controller in Unity's forum will be really bad considered it's already a term used for a component in Unity. (Try searching for "Custom Controller" on the forum search. ;) ) I also though about Custom Gamepad, but as to avoid making it sound like it has to be a pad, I though it missed a little something. Secondly, it's a kick-start to have a nickname for this kind of "ideas" and innovation. While CIOSI might might you think of some sort of sickness and there's already a bit of use for the word (like the name of a company that used to sell on amazon with really bad reviews), it's still better. If you got a better name, feel free to suggest it!

    This isn't the first time I bring this on the Unity forum, but the last time, it wasn't related to the controller itself, but was more about something that could be related to it. What I'm wishing to discuss here is how can we, developers who build up the medium on which those device gets to play, think outside of the usual box with this. Integration is an heavy part of it and there are many ways of handling it which is why I think that this can be a really good discussion.

    Warning: This can involves some talk about really deep technical stuff outside of Unity, so don't feel bad if you get lost at some point. I'm creating this topic in the External Tools because I'm looking up to discuss things about the CIOSI between a custom controllers and Unity.

    Another reason for this topic is the current raise of the DIY trend that has started for about a year. Even Nintendo has caught on, which is why they're releasing the Nintendo LABO. It's one example of how CIOSI works with predetermined hardware: Making use of existing features of a controller to evolve further from its initial setup.

    Personally, I'm currently going more with the other approach which involve creating the controller itself with hardware components. I won't say that "it's as easy", but as I previously mentioned, the trend of DIY has raised and now you can have access to some pretty good stuff for relatively low cost. There are also quite a few You-tubers who dig the stuff and display DIY stuff quite efficiently.

    My own goal is not just to create something similar to the complex controller displayed above, but also 3 things:

    1) Creating a controller with low cost components. Currently setting up the budget of my controller project at $150 all included. This might sounds big if you consider usual controllers are at around $60, but this isn't just a "normal" controller, but more like an complex arcade controller borderline from being a simulator controller that can be modified as I like. I can add or remove buttons from the setup and all.

    2) Creating a controller in a way that even a teenager can do it with just his dad's (or mom's) tools. This means that it involve, at least for a working prototype, that you don't need some specialized studies or Tony Stark's lab to be able to build it yourself.

    3) Create the documentation of both how to build the Controller (think of it like an IKEA build-yourself guide) for anyone and the relevant software/plugin/assets for Unity to allow developers to integrate, at least, my own way of the CIOSI. Of course, this would, essentially, be an optional feature on any game implementing it.

    Now, some of you who read this will think something like "But if you make a complex controller that works with only 1 game that you make, it have huge chances to fails!" That's why I'm doing the CIOSI side of the project as an option with the game I'm preparing to start developing it for. On top of that, the thing I'm planning to make isn't something "fixed" per say, but more of a system that works with multiple kind of custom controllers as long as they are made within a specific ways with specific rules. A bit like how the Lego Brick system involve those brilliant yet simple patterns on top and below to fit together, I'm working on a similar system, but for digital data linked to hardware.

    This is why my own version of CIOSI isn't something you can do, for example, from dismantling a PS3/PS4 or XBox360/XBoxOne Controller and just change some bits and the case of it. It's something that is akin to the idea of allowing lots of buttons and inputs (or fewer) as necessary. It also involve the ability to have a touchscreen which display game's data (like the Wii U gamepad) and things like actual digital counter on your controller if you want. For example, it could even be a special gun-modeled controller with ammo counts and such.

    If you're a developer, have you even wanted to have your game playable on something like an arcade machine you can build yourself? This kind of discussion could, maybe, help you reach that goal!

    I'm opening this discussion up for anyone's personal experience or project around CIOSI.

    Simple Guideline:
    (Those are not official Unity Forum's rules, but more of a guideline to keep the topic on the right track)
    • Don't post about conceptual ideas which you are not interested to make yourself right now.
    I think many of us have this dream of building this futuristic AR or VR or Simulator in which things are perfect. A bit like how a Mobile Suit Gundam fan might really want to try a MS Gundam box-like arcade game in which they are sitting and using the same kind of controller as in the show. If you dream of something like that, it's a dream and this isn't a topic about dreams. If you're actually seriously thinking about making it and are currently in the planning of hardware and software for it, feel free to share your experience, questions and ideas that, if things goes well, will becomes real things in the future!
    • You can share your ideas and opinions if they are constructive and realistic.
    Meaning: Please, don't post just to tell "Cool! I like it! Wanna try it out!" or "This is a bad idea" without giving actual information about why it's a good or bad idea.
    • Please, no suppositions.
    What I mean by that is that if you're uncertain about something that isn't a question oriented toward the subject, don't post about it. Things like "I don't think..." or "I'm not sure..." usually never bring constructive and technical arguments because it's keeps walking on hesitation all the time (It always turn around the "What if?" state of arguments) If you don't think something can work, post it as an rock-solid argument with details as to why it won't work. If you doubt it won't work because you never tried it or because it's foreign to you, don't post about it. Let people experiment in a joyful and positive environment as, maybe, they'll find a solution you haven't though about even if they failed the first time like you though they would. If you have doubts and really want to talk about them, PM the person. Don't flood the topic with doubts. Flood it with only actual results and processes. ;)
    • Disagreement doesn't means hatred, but keep it within the core of the subject.
    If you disagree with someone and have good reason for it, you could share why you disagree, but please keep it within the boundary of the subject. Try to not extend toward things that aren't related to CIOSI. (For example, don't start explaining what's your game is all about in details while all you should write is that the response time is delayed by 200ms when you press the kick button.)

    In the future, with a bit of luck and even more raise in popularity, I would like to see a a tiny part of the forum dedicated to things like CIOSI. Maybe a sub forum in the VR/AR Discussion so that people can share their own CIOSI story, ideas and comments in separate topics. *Wink at any Unity community manager*
     
  2. Max_Bol

    Max_Bol

    Joined:
    May 12, 2014
    Posts:
    168
    I'm posting a 2nd topic because this is my actual current take on the CIOSI. (So that I can separate the topic from my own part of it).

    I'm currently working (on my rare spare time) on a controller made with 3 key components:
    • a RaspBerry Pi 3
    • a Rii Mini X1 Keyboard
    • a set of buttons and digital displays (distributed by Free Nove as a Starter Kit Package)

    The RaspBerry Pi is the device that "manage" the data from both the keyboard and the buttons/digital display.
    As I'm currently building the thing and trying things out, I'm still uncertain about I will format the data to be sent to the PC, but it's still cool that I'm able to display gameplay-like information in the setup. I'm making use of a LOT of stuff out of the Free Nove starter kit package. 1x 8-directional stick, 4x colored arcade button, 4x black squared buttons, 1x single digit screen, 1x 4 digits screen, 1 blue screen that can display a string of text.

    I'm still at the point where I beta-test the input devices and see what kind of data I can "gather" or "send" to it. Once I'll have it down to the core, that's when I'll start looking up for how I can take this data and send it in Unity and make Unity understand that this is "input" data. That's because it will not be recognized as a controller since it's the RaspBerry pie that will be connected and not the an actual chipped controller which is recognized by Unity.

    First, no, I'm not using Unity on the Raspberry itself. For now, I'm most likely coding things. I plan on adding an actual touch screen later, but I'm still not at that point yet as that will involve making an app for the RaspBerry Pie that will have UI interaction and much more complex data being exchanged between the PC and the device.

    My goal is to make a controller that works with a project of mine related to mechs and tanks. The analogue stick controller the "aim" and the player enter the coordinates he/she wish to get to on the micro keyboard. In other words, it's not the usual FPS setup, but more like a tank simulator setup where the movements are controlled like a RTS.
    Another use I plan for the keyboard is to have online property and a chat system. Adding a microphone and a webcam is the last step out of the 3 step of development.

    (Note that all those features will be optional and a player will be able to play with a keyboard + mouse, webcam and mic if he wishes. In other words, the CIOSI side of the project is an option I make because I want to.)

    It's still early so I don't have much to show, but once I get to a point where I get something more game-like out of the controller, I'll share some stuff about it here.

    = EDIT=
    By the way, I forgot to mention. I'm using Processing PDE to script the output controls and reads from the inputs and outputs toward the CIOSI system. You would be surprise as how much it's alike to Unity's version of C#. Once you know how a circuit board works and how to send the signal to the right wire with the right voltage, it's actually quite fun to work on. This also means that it will be a lot more simple to translate the data from the board to Unity.

    The main hard "task" from it will be to set up the ground rules. When you got a circuit board, you're free to set up as many wire and links as you wish, but for a system that, for example, could allow an unknown number of buttons at once. On a board, you have limited output "slot" where the power can be read. For example the GPIO Extension board (the thing that is connected with a large wire to the Raspberry Pi) I'm using has a limit of 17 GPIO pin that can be read. Each of those pins can either be a boolean value (0,1) which is 0 or 100% power being read, a float between 0 and 1 which it's if you want the power to be between 0 and 100% (like if you dim down a light), or, with the right chip, could be a bit more complex in terms of signals.

    I'm glad that I took the initiative of adding a keyboard to my project. By sacrificing 1 USB port (out of the 4 available on the Raspberry Pie 3), it covers all the requirement in terms of chat/text based inputs without using any of the available pins for which I can then use to focus on the more gameplay-oriented buttons and inputs. (It reminds me of the XBox360's Controller's keyboard which you plugged in the jack port on the bottom of the controller.)


    I was also quite surprise at how Linux-ARM can be so "single minded" as an OS. It's not even hard to set it up to automatically launch an app with full focus after startup and, from the app itself, set everything up so that it doesn't even need an actual screen that, normally, would display the "Desktop". (The desktop could still be accessed, but it wouldn't be seen unless a shortcut is done on the keyboard... and yeah you can set up a custom shortcut for that too! No need to fear that the user might press ALT+TAB by mistake!)
     
    Last edited: Jan 25, 2018
  3. skelteria

    skelteria

    Joined:
    Jun 20, 2017
    Posts:
    1
    Gud am,im trying to create a game(im new to unity)which involves simulating an electronic drums.the game should go like this:a drum track is played as an audio source,the player must hit a costum drum pad(a piezo switch)in tune with the drum track to gain score(timing is relevant in this game).i want to use ARDUINO for the switch and of course the interface but how can i incorporate it in my code?my e mail:hekterskelter17@gmail.com
     
  4. Max_Bol

    Max_Bol

    Joined:
    May 12, 2014
    Posts:
    168
    The first thing to understand when you want to use an Arduino, a Raspberry Pie or whatever micro-PC component as an input device is that you got to work with 2 different PC at once to make it works.

    In my own experiment, I find it a lot easier to use the micro-PC (in your case, the Arduino) to just convert the action (hit of the drum in your case) into simplified data (would be an float between 0 and 1 in your case) and then send a packet to the PC that runs Unity.

    To do so, you got to follow some steps (more like guidelines).

    A) You got to make a special App for the Arduino. It will be that app that will read the drum's data through the piezo switch and convert it into something Unity can use. There are LOTS of tutorial online that display how to use a piezo switch and transform its vibration into a float data. This App will have to be launched for the special drum to work with Unity. Unless you use the Arduino for something else as well, it's rather easy to make an app launch by default when the Arduino is powered up. (Again, there are many tutorial online.)

    If you're really not sure how to script in your Arduino nor how to set up the piezo switch to be read by it, I suggest you invest in a starter kit that includes all the parts required as well as a guide for beginners. I bought one for the Raspberry Pie 3 and it teaches you some really good stuff that makes it rather easy, afterward, to build almost anything. (Got to google a bit for specific more complex things, but you get all the examples necessary to make something like creating the data through an input linked to your micro-PC.) Even if it's just to understand how to script an App in your Arduino, it's still worth it.

    Before you start wondering, it's possible to, more or less, wire the piezo switch directly to the Arduino without a circuit board, but I suggest you start by using a circuit board to make your prototype. That way, you avoid the risk of frying either or both your piezo switch and the Arduino. ;)

    B) Once your Arduino is able to read the vibration on its own, from there on, you need to script both in the Arduino and in Unity a way that allows the two to exchange data. It's not simple, but not something really hard and impossible for beginner. You got to think what kind of data is generated from the vibration (from the piezo switch) and translate it into something usable.

    There's an example of a video series that displays how to do it through USB :


    If this video doesn't help you, that means you're still at the starting line and you really should invest in a proper starter kit with the proper tutorials and hardware pieces that will helps you understand how the Arduino works internally. (The video is actually kinda on the beginning level in terms of hardware integration in Unity.)

    There's one thing you got to know. There's no master nor slave when it comes to using a controller made with and micro-pc and a main PC which runs Unity. What I means by that is that both PC can do calculations and even works in tandem. You don't have to make Unity read every bits of data from the Arduino's port and convert it in Unity as you got plenty of processing power in the Arduino itself to manage everything it has to do. That's the major difference between a normal controller and a controller that has its own motherboard unit.

    For example, you could script the Arduino to returns a bool value that switch to true for a fraction of a sec whenever the drum is hit instead of a float that constantly change. Your script in Unity could also register the current framerate of the game and send it to the Arduino so that its know the minimal frequency to check from the drum and avoid having strokes on the drum to not be registered because the game, on the PC, runs at 35 fps while the user strokes 2 times within that time.

    In fact, if you want, you could even use the Arduino to assist the main PC in calculating things. You could add a digit screen to the drum that display the score. You can do whatever you want with it as long as you know how the circuit works and how to read the data from the Arduino's hardware and exchange it with Unity. You could also make the Arduino able to submit score to an external website. The only limit is the limit set by the way the Arduino is connected to the PC. The refresh rate of the data will varies between bluetooth, WIFI, Ethernet and USB as well as the amount of data that can be packed together each time. The cable, if you use a cable, could also be one of the limit. USB can be limited if your device is USB 2.0 instead of USB 3.0. Ethernet cable prior to 3e have limited bandwidth to 50MB/s while anything after can reach as at least or around 2GB/s and the latest models can reach as much as 50GB/s.
     
    Last edited: Apr 12, 2018
  5. cordCladKing

    cordCladKing

    Joined:
    Jul 13, 2023
    Posts:
    1
    I have inherited a project that I think needs rebuilding in unity or a similar dev environment to make work. It has various custom input and output devices (CIODs) talking across multiple communication protocols.
    I need unity to be able to output 4 channels of audio, DMX512 messages, pwm signals and other binary outputs? I also need it to be able to receive 2 binary inputs. I can obviously add more hardware to cope with this but unity would be the base for the software and run on a NUC or similar small scale computer. Any thoughts?
     
    Last edited: Jul 13, 2023
  6. Max_Bol

    Max_Bol

    Joined:
    May 12, 2014
    Posts:
    168
    In your project's case, the 2 struggles I see that could make it works or not is these 2 points:

    First unless it got added in the last few years stealthily and I don't know/ can't find anything about it, Unity doesn't really have any library or drivers natively compatible with the DMX512 protocols required for DMX512 messages.

    Depending on was is communicating via DMX512, you may find some stuff already around that may fits your needs (for example this: https://assetstore.unity.com/packages/tools/integration/u-dmx-230795 . Note: I'm not endorsing if this works or not for your case. It's just something that I found when looking for a solution.)

    The 2nd is your vague mention of "multiple communication protocols". Each comms protocols requires its own piece of a solution which might be found either via raw signal convertion or via the use of some existing libraries. For example, USB and Ethernet are kinda straight forward, but if you use something like a custom 16+ pins flat threaded cable to a custom board inserted into a PCI slot on a PC that manage a substantial level complex data, that's a whole different matter. You might have to rethink / look-up some alternative pieces of equipment that has said comms protocols automatically converted to a simpler output standard if some equipment doesn't have anything on the subject.

    The PWM signals can be managed with an arduino like explained in this doc https://docs.arduino.cc/learn/microcontrollers/analog-output

    Hence, by mixing my previous post on how to make an arduino communicate with Unity (which, yes, can be run on a NUC, but the speed and quality of what may be managed heavily depend on the gen of the NUC and which OS is used on it. Just remember that a NUC is not great on the rendering side, so you got to keep things either simpler (like managing images/stuff to be rendered in byte format and doing some transition/conversions via the CPU prior to caching the result in the renderer. This is kinda like old school console rendering hacks.)