Publishing large package to custom registry

Hi,

I've been setting up a server running Verdaccio to have my own package registry.
Everything worked well, until I had to put up some asset packages that goes beyond a gigabyte.

Here's the message I get after trying

npm publish --registry {my_registry_address}

:

Code (CSharp):

1. npm notice === Tarball Details ===
2. npm notice name:          com.large.something
3. npm notice version:       1.0.0
4. npm notice package size:  1.6 GB
5. npm notice unpacked size: 2.3 GB
6. npm notice shasum:        932648fd193944bb9dcbbe1899155d7a69b9f058
7. npm notice integrity:     sha512-Ck2S18aGauWmn[...]ycDjg2NI1Yh1w==
8. npm notice total files:   1237
9. npm notice
10. npm ERR! code ERR_STRING_TOO_LONG
11. npm ERR! Cannot create a string longer than 0x3fffffe7 characters

I tried following this suggestion but had no luck.
I tried setting the environment variable by:

1. adding it to Windows system properties (could check via

   printenv

   )
2. adding it from git-bash using

   EXPORT

   (could check via

   printenv

   )
3. adding it using cross-env npm module through added

   "scripts"

   field in package.json, as noted here (I tried

   "prepare"

   too)

and none made any difference.

I want to make use of package manager not only for code modules but also to manage assets, with dependencies on asset related scripts and shaders. And 1GB is little too tight for my use.

Do you have any solution to work around this other than splitting assets to even smaller pieces?
Or do you have any other suggestion to manage long list of large Unity assets that can also handle code and shader dependencies?

Thanks in advance

 

TLDR; you can not publish a package that more than 1GB. The UPM package is a great solution for handling software, but not good for large assets. You may want to move your large assets back to the Assets folder.

Details are here,

0x3fffffe7 characters mean 0x3fffffe7 bytes which are roughly 1GB. The issue is being asked in the npm forum with no response. The error code is documented in nodejs, that you can not create a string longer than 1G. Instead, developers should use bytes and stream.

The current implementation of the npm-cli tries to convert the tarball data into a base64 encoded string that triggered the issue.

Even you can cross the issue by using the curl to directly post the large file. Your backend server may not be ready for such a large package. i.e. verdaccio the popular open-source npm server, fail to handle a 1.5GB package in my test (raise RangeError: Invalid string length).

So if you insist, your best bet is to try the curl hack and test your npm server.

It's an engineering issue the npm-cli and any npm server should solve.

 

Thank you for the definitive answer!

I haven't tried direct curl-ing, and it was very informative that you shared your test. I have some control over the server but I don't wish (more like incapable of) to work on making custom solution ditching npm publish and verdaccio.. So I guess I'm going to try making sense of splitting packages further down for now.