[WIP] Release Manager. (Installer, Patcher, Launcher)

We have put up a short tutorial on configuring Release Manager to work with freehosting.

http://info.celestial-games.com/doku.php?id=releasemanager:tut:freehosting

Hopefully this will assist people who don't have the scale or need for full hosting and also that might need some additional insight into how to set up there FTP and HTTP.

 

A small patch 0.0.8a was released to fix a few issues with FTP and SFTP. If you have been having difficulties please try again.

 

Really excited about this for multiple reasons. It will be nice when this is integrated into the Release Manager so that using the Release Manager isn't limited by download count (5), but instead by a valid user account. This could also help save bandwidth by only pushing patches/updates to subscribers and approved users. So yeah, this is my unselfish way of selfishly asking that this particular feature gets placed even higher on the priority list!

Cheers and thanks for all the awesome work thus far!

 

I have been toying with an account managed process for Release Manager. I don't know how I would integrate that into the Unity asset store though. I would be better for many other reasons. If you need more downloads then 5 mind you let me know.

 

Some answers to a set of questions send in a message.

Yes its stand alone, currently any executable can be launched from the Release Manager but it only has settings support for Unity. We are considering other platforms too. But first we want to get the product feature complete.

Right now it only supports the quality mode, not individual configurable items. We do have that on the roadmap since our own games require the same kind of granularity in configuration.

I have been taking long to answer as I was doing some DropBox research. Looks like we can include drop box. This would require that you are running the DropBox service that does the uploads part but its doable with public folders. So I am starting the design now for this support.

Interesting question, I guess its really about how many people will want that same configuration. Redmine appears to have a pretty straight forward rest api to integrate through. There are also a few project I see that make Redmine use an OpenAuth type authentication. Are you hosting your own redmine or are you using an online provider? I will keep this opetion in mind when we do the account management part.

The compile process takes a long time. It uses the same algorithm as bsdiff (http://www.daemonology.net/bsdiff/). Memory requirements are

n=oldsize, m=new size
max(17*n,9*n+m)+O(1)

How long is not as predictable, it really depends on how many changes. It also compresses the file with bzip2. If memory runs out the solution will turn to the drive, but since right now we only release the 32Bit version its limited to 4gig.

Yes Windows64,Mac64 and Linux32/64 are all planned. I just wanted to get it feature complete before adding all those platforms. I already have the packager for Mac and Linux ready. Linus will be a self executing .sh file. Mac is a normal mac.app but I have to undo Unities files and create my own.

I see myself always having new features to add to Release Manager, so I guess I cant say which ones would make it Prime Time ready as apposed to just ready. Right now we are in Beta. I have a list of about 40 things I want to finish before I consider it out of Beta. (The list does keep growing but some items are already flagged as post Beta )

 

?!

 

Hi Zeblote,

As I said we had not decided a final Price. We have put the product up on our own shop in direct response to some eager customers. The beta price is still under $100.

Since the feature list has grown since we started the product it means we have a lot more work to do to reach completion, this is playing a significant role in our thinking about pricing.

Market dynamics may still play a role ongoing pricing.

 

Just an update. I managed to use the sftp towards godaddy. It was a matter of some weird requirements. I've been very busy with other things I didn't fully test things yet. But one thing I found was I have to manually load the design every time I start the design launcher. Just for your information Travis.

 

That's good news its more secure.

You saying when you load the Release Manager the design is lost or that the Launcher is downloading the Design each time your start the application?

I will investigate both.

 

Yes the design is displayed in red with missing Graphics/content. The I have to load template and browse to the template I have saved. I am NOT using any default path but a another hard drive for everything I produce.

 

I can recreate the issue here. Will try fix it ASAP.

 

Ok for now a work around is to press CTRL-S (save) then click the refresh button. You will only need to do this the first time you load the project. The issue is caused by a missing sync that is only happening in the EXE of the app not in the IDE when testing. I am looking for a solution still.

 

I'm so glad to have this program, it's saving me a ton of time! I've got a question though: is it possible to have the start button become the update button when a new release is available? It would be useful if you want to save space...

 

Thanks! That works forn now.

It would be nice with some documentation telling the process to this whole thing. You know step 1 to finish, like a quick start. Some of these things are obvious to you but not to the user. Not to complain just giving my view as a new user.

I am sure you will make these and other things. I am also sure you might find yourself busy with support rather than developing because this will probably turn out to be one of the most popular assets for unity. So the work flow inside the documentation will help not just the users but you too

I wish you all the luck!

 

Hello HitsuSan, thank you I am glad the product is proving useful for you. We do not plan to make a button that can change from one thing to another. But we do have a plan to add a feature that will help you do exactly what you need. Every component in the display is getting a hide property. That will allow you to make it hide or show under different conditions. So you could make the start button vanish when there is an update and the update vanish when there is no update. So it will solve your problem and also allows allows you to do many other things like make the progress bar vanish. A helpful text display show when an update is available and so on.

I agree with your sentiments. We have a shot Wiki up with some information but its not comprehensive yet. I suspect it will continue to grow forever.

The tutorials are here
http://info.celestial-games.com/doku.php?id=releasemanager:gettingstarted

Some are not yet completed.

Thanks I agree that we need to make the documentation as clear and easy to use as possible. I think that the concepts in the Release Manager can seam daunting. I have found that the part of linking the FTP to HTTP is probably the bit that stumps most people at first. We are creating wizard style helpers to assist with that setup and I worked out how we can use DropBox which I think for many will be a far easier setup.

Thank you, please don't hesitate to list any issues or ideas. We really eager for feedback.

 

That's way better thank you!

 

Something weird is happening, i can't understand what's going on but i've configured everything and it can upload the launcher fine and all but it doesn't want to upload a release... the process is being queued but it doesn't start.

 

Hi Hitsu,

The upload process for both the files and the launcher is the same subsystem. So its very strange that the one should work and the other not. What type of FTP are you using? SFPT, Passive or Active?

 

Yeah it's very weird, i'm using Passive.

 

I have found the issue. Its a glitch in the upload factory for standard FTP. I am repairing it a patch wont take long to get out.

 

Cool thank you

 

The passive and active FTP both working apologies for the issue its about 700kb patch.

The design loosing references on each load has been patched in 0.0.9

 

Nice! Works like a charm now.

I am having problem uploading using sftp. It stops at around 100+ Mb upload. Not sure if it's on the server the problem is. So I uploaded Everything with filezilla, no problem. But I have to guess what to actually upload. Then when I run the launcher it just says downloading manifest without actually downloading anything.

Once I got it to start downloading and then it skipped after half the download. But as I said I am not sure what to actually upload. Might be missing some detail.

Also there is no .msi file when after creating the package and installation. Where is this supposed to be created?

 

Well, now it's uploading but it stops after a while without finish the queue... ^^ i've tried twice and it stops around the same amount... uhm maybe it's a timeout issue?

 

You need to make the release current. There is a button that does that. When you click current it will create the manifest file and upload it.

It should report that in the logs on the create package screen. Can you show me the logs you do get. You did install WiX first?

It may well be a timeout issue. I will artificially cause a few failures and see how to make it more robust. At the very least I need to make sure you get communication for failed files.

 

Maybe a resume queue button can bypass this problem until you can make a more robust solution.

 

I am working right now on restructuring the download screen to give you the opportunity to retry and to cancel downloads sets. Its not perfect since a broken upload would require starting from the beginning again but at least only for that file not the whole set. I expect this will be done before the end of the day.

 

So 0.0.10 has been released.

I have spend some time refactoring the system that manages the upload choices. I am about to start changing the config and reporting screens for that as mentioned earlier.

The other significant change is DropBox Local support. That is if you have DropBox installed you can us it do to all your uploading of files.


Configuring DropBox local is easy.

Specify the Local folder you want to use for your drop box files. It MUST be a subfolder of your Public Folder. Make sure your dropbox public share is enabled. for more information about dropbox share you can see there official page https://www.dropbox.com/en/help/16

Next you must specify the http location of the share folder. You can not right click on a folder to get its location so you will have to create a text file right click on that and make the adjustments to the url.

You can still do the test, it will ask you to wait at one point for DropBox to finish uploading.

 

The new structure will keep the files packaged in the set they were uploaded in. And any failed files can be retried. Resuming uploads after failure though can be pretty tricky. I am looking into that. Most FTP solutions have good ways to resume downloads but not uploads. The best we could do is append to the file where it was last written too and assume everything written was correct.

 

Oh well i didn't mean to resume the upload of a single file but just the upload of the queue because i've noticed whatever it's uploading it finishes but it's like it's running out of threads. That's something i think you should look into it because actually even if you're using 4 threads before it stops you can notice it's just uploading less and less files till it's one and when that one finishes it doesn't start with the others in the queue. I hope it's useful information.

 

Hello. I guess you mean the "Mark as Active" button?

It seems like it's never uploading manifest.

I have WiX installed. But no msi created.

I run one of the exe that starts the launcher and it drops the download in the middle of a file.

package log:

Creating windows 32 package.
Cleaning out D:\Release work\package\Win32\package
Create Icon
Create Executable
Create Config
Built in D:\Release work\package\Win32\package
Create Temp folder D:\Release work\package\Win32\temp
Creating windows 32 package.
Cleaning out D:\Release work\package\Win32\temp
Create Icon
Create Executable
Create Config
Built in D:\Release work\package\Win32\temp
Create Wix file (earlier I got 2 more lines here but no msi file.)

launcher log:

Unexpected end of ZLIB input stream
java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.InflaterInputStream.fill(Unknown Source)
at java.util.zip.InflaterInputStream.read(Unknown Source)
at java.util.zip.ZipInputStream.read(Unknown Source)
at java.io.FilterInputStream.<unknown>(Unknown Source)
at celestial.launcher.UpdateThread.unZipIt(Unknown Source)
at celestial.launcher.UpdateThread.run(Unknown Source)
_________________________________________________

Well, to be honest I didn't have much time digging deeper into this. I am very occupied by current project.

 

I have this in my dropbox how to use the public folder.txt

P.S. You can only link to actual files within your Public folder, not to folders.

 

Correct you cant link to folders. you need to take that link remove the text file name from the end and the add the name of the folder you want in public.

If you look at the URL in my screen shot you will see the part up to the number (That number is your account reference on dropbox it will show in the link to the file) you just need to delete the file name and append the folder part. ReleaseManager will add files to that path for retrieval. I will get to making a wiki for this, I am just chasing down upload stability and the packaging issues you are having as a priority over the help.

 

It seems i've got the same problem as Zubba, it stops here http://take.ms/1d7ss

 

I suspect this issue is cause by missing data. I have tested with a blank project and then added check to inform you whats missing or suitable defaults for some things.

0.0.11 will be out today with fixes for this and also a new uploader screen that lets you retry failed uploads and gives you more information on why they failed.

 

Uhm maybe let us know how the folders should be structured so at least we are not doing some mistakes (which files goes where ecc)!

 

Version 0.0.11 released.

Since it becoming clear that uploading files is one of the main bottlenecks getting a project live. I have restructured the uploads tab to include a lot more information. It not allows you to cancel an upload and also to retry failed files. Additionally it gives you an idea what went wrong with each file that failed to upload.

Also some checking before packaging or installation creating to make sure we wont experience any issues performing the build. This will hopefully resolve the packaging issues.

 

Here is a new page on the wiki

http://info.celestial-games.com/doku.php?id=releasemanager:directories

It gives a high level view of the folders. Let me know if you need more clarity.

 

Inside "versions" the structure should be like this?

versions>"myproject">Win32>game_Data + game.exe

I want to be sure if the structure is correct to avoid problems, what missing data were you talking about earlier? Is it something i should check on my end?


Btw now it's uploading fine, the new patch is working great!

 

I've run some tests, uploading is ok. When i start the launcher it detects the patch and downloads it but when it's completed the game won't start when pushing the start button. After that if you run the launcher you can't click any button.

And i can't manage to make the package application work, if i can provide any useful information let me know.

 

Nice progress!

Some things you neet to know.

1. sftp still drops uploads at around 100mb (passive seems work now)

2. No msi created by packager and log looks like Before. Good though now I know it requires atleast one Icon image.

 

OK so I had success in creating the msi package. The problem was the installation directory. I was using full path eg: d:\xxx\xxx while the packager only expects a name of the subfolder that will be default installation folder under Program(x86)

However after installing the msi and running the launcher my download drops at around 100 mb downloaded, and the progress bar jumps to 100% and nothing more happends. Sometimes the download just stops.

When trying to create a new release selecting "patch", Release Manager crashed 5 times in a row with an out of memory exception.

 

Noted I will test the directory name for validity.

The Launcher (That the application that does the downloding) does product logs. If you look in the programdata folder for your application you will see them. They might give some insight into the issue. I am busy reworking the download code to better handle issues. Hopefully I can even get it to pick up where the download left off. That depends on your hosting provider.

Whats the largest File in your set? And how much ram do you have? The system should switch to HDD when mem is to little. I will check those settings.

 

I will extend the info page I can see whats not clear now.

The folder structure is

versions\{versionname}\Win32\ <---you application root goes here.

If you use the create new version Button it will create the folder structure for you.

Is the button active? If so is the application exe name correctly specified?



Above is the run button for ReleaseManager. The exe is ReleaseManager.exe we only specify ReleaseManager. This is done since each os has a different expectation of what the file name will be.

A few things to check when building your installation.

The first is make sure the Directory is only the directory you want not the full path. So for ReleaseManager the directory is just ReleaseManager not C:\Program File(86)\ReleaseManager

You specifically said Package. I just want to make sure you clicking the correct button.



The panel has a Package button that creates a working folder you can then build your own installer for. (not recommended) Or you can click the Create Installation button below that. That will create the MSI installation if WiX is installed.

 

Ok now with all the information i've managed to make it work properly and i've found you another bug, if you use the check "Only when updated" for the launcher the start button never reactivates again unless you've just received an update (even if you are at the last version available).

This is not a priority by all means but can you make an option to read the alpha channel as solid for buttons? If i use an image like this instead of a solid button i can click on it only if if the pointer is in the actual writing and not the whole area (it's not the best). http://take.ms/x8C39

Oh and another non priority but useful addition you can make is an option to reload all the graphics for the launcher from the disc so you don't have to remove and replace manually something every time you make a modification.

 

Glad you working and thanks for all your effort helping me understand how to make the product better. I have a few projects that use the Launcher that only run when updated. But in all those cases I don't include an Update button, I just auto update. That might be the issue. I will check since I am working on a new version of the Launcher now.

I will need to add an option to the button weather it uses the mask or not. The mask is really handy for complex shaped buttons. I will add the check box to the buttons.

The problem with that is we don't actuly keep the images source location. However, what I should point out is if you keep all your launcher graphics in a single directory you can drag all of them into the resources windows and they will all be updated in one go. (I will keep the idea of linking them in my mind.)

 

launcher log when it fails:

SSL peer shut down incorrectly
javax.net.ssl.SSLException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.readV3Record(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.MeteredStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.<unknown>(Unknown Source)
at java.io.BufferedInputStream.<unknown>(Unknown Source)
at celestial.launcher.UpdateThread.download(Unknown Source)
at celestial.launcher.UpdateThread.run(Unknown Source)

(Once I managed to download the whole project)

About the compile crasching. The largest file is 294mb, I have 16gb RAM and a good amount of hard drive space. But I read somewhere that each file needs 10 times it's size while compiling, and being a 32bit program it can't use more than 4gb and in reality thats usually less.

I have good hopes for you making sure it will start using the swap file as well as a 64bit release manager.

 

Yes i've noticed that It's very useful that way! But for the future if you just have a referenced dir with all the graphics that you collect to build the launcher that's auto reloaded (or just with the pressing of a button) we can work on the graphics itself on the fly without having to go inside RM, delete the modified graphics and then reinsert the new one. But then again it's not a priority, it's just a faster pipeline

Anyway it's a pleasure to help you, you've made a very good program and it does have a ton of potential. I've got a ton of ideas on how to make it better if you want, i'm avoiding spamming this forum if not with debugging for a good all around working release at the moment

 

Thanks for the logs, from what I can see there appears to be many people complaining about ssl downloads of large files breaking from various reasons. So No single solution looks like it will work. I will focus on restarting part way in so I can complete broken downloads. Most servers support restarting into a file.

The 64 bit of the Release Manager is probably the only version I should be recommending in the long run. Almost no one has 32 bit machines anymore. It will still support 32 bit applications of course.

Our game BAD has at it largest a 200mb file. So you pushing the limits further then we are right now. I will try get that 64bit version done ASAP. your 16gig of rams is more then enough.

 

It wont be much to remember the original files location. Of course if you like me your environment changes all the time and the links might break. But you can still fall back on the other way. This has been added to the backlog

I need to focus on the primary features right now. I do welcome the ideas, its useful especially since it will only make the product better. You may find many are already in my backlog.