Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Fast ETC2 encoder: ETC2Comp. Unity integration please!

Discussion in 'Android' started by mh114, Sep 19, 2016.

Page 1 of 2

  1. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    294
    mh114, Sep 19, 2016
    #1
    guzzo, NeatWolf, iivo_k and 1 other person like this.

  2. Agent_007

    Agent_007

    Joined:
    Dec 18, 2011
    Posts:
    899
    Agent_007, Sep 22, 2016
    #2
    NeatWolf likes this.

  3. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    294
    Bump! Unity Android devs, any chance of getting this into Unity? @Yury-Habets ? :)
     
    mh114, Jan 16, 2017
    #3

  4. Yury-Habets

    Yury-Habets

    Unity Technologies

    Joined:
    Nov 18, 2013
    Posts:
    1,165
    We'll consider it. Thanks!
     
    Yury-Habets, Jan 17, 2017
    #4
    mh114 likes this.

  5. tripknotix

    tripknotix

    Joined:
    Apr 21, 2011
    Posts:
    744
    That would be great, only "best" has the really long wait times, increases the import time 20x easily.
     
    tripknotix, Jan 23, 2017
    #5

  6. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    294
    I reckon ETC2Comp would help even in lower quality levels than "best", so it'd be a win-win.
     
    mh114, Jan 23, 2017
    #6

  7. Yury-Habets

    Yury-Habets

    Unity Technologies

    Joined:
    Nov 18, 2013
    Posts:
    1,165
    First tests show slower compression on Normal and faster on Best.
    So it's again not as easy as it could be.
    @florianpenzkofer has more details!
     
    Yury-Habets, Jan 24, 2017
    #7
    mh114 likes this.

  8. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    294
    Interesting. Thanks for checking. Maybe use PVRTexTool (or what have you) for Normal and ETC2Comp for Best, then..? :D
     
    mh114, Jan 25, 2017
    #8

  9. tripknotix

    tripknotix

    Joined:
    Apr 21, 2011
    Posts:
    744
    We only need it for best, everything else works fine on normal and fast
     
    tripknotix, Jan 27, 2017
    #9

  10. Yury-Habets

    Yury-Habets

    Unity Technologies

    Joined:
    Nov 18, 2013
    Posts:
    1,165
    PVRTexTool is also not the fastest way to compress for normal.
    We're considering options - your feedback is valuable!
    Are you extensively using Best quality?
    Is Normal quality too slow? Could we trade off quality for speed?
     
    Yury-Habets, Jan 27, 2017
    #10

  11. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    294
    I was considering using Best for my background images (2D game) but opted not to since the compression time was horrible. Normal is alright by me, but if you can make it faster I won't complain!
     
    mh114, Jan 27, 2017
    #11

  12. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    I made some experiments with various ETC compressors. So far it looks like:

    Etc2comp could give us significantly faster 'Best' quality. The actual quality will probably be slightly worse than current 'Best' but better than 'Normal'.

    etcpak could give us slightly slower but much higher quality 'Fast' compression.

    With my test data etcpak actually looks good enough for 'Normal' when compressing to ETC2. So changing the default format for RGB to ETC2 would also be an option.

    I have not found a tool that is faster than our current solution for 'Normal' that has similar quality.
     
    florianpenzkofer, Jan 27, 2017
    #12
    mh114 likes this.

  13. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    Btw, I tried etcpak, ETCPACK, PVRTexTool's ETCPACK, ARM's etcpack, rg-etc1/crunch, ISPCTextureCompressor, etc2comp, ALGCompressionTool
     
    florianpenzkofer, Jan 27, 2017
    #13

  14. tripknotix

    tripknotix

    Joined:
    Apr 21, 2011
    Posts:
    744
    Can you mix and match?

    The problem lies with "best" quality only, thats why i was asking.

    For ETCPack, how is the speed on "Best", because if its similiar quality but faster best, then we its a win win all around.
     
    tripknotix, Feb 2, 2017
    #14

  15. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    Yes we can mix and match. ETCPACK is what we use now (PVRTexTool's flavor of ETCPACK). So I assume you mean etc2comp. ETCPACK/PVRTexTool at the quality setting we use for 'Best' was around 7x slower compared to etc2comp at 'effort 70' setting. But I didn't do that measurement on a lot of texture, mainly because 'Best' takes so long to compress ;).

    Comparing quality is not so easy. To me and in PSNR numbers etc2comp effot 70 seems fine compared to what we have now.

    I'm curious, for what kind of textures do you typically use 'Best' quality?
     
    florianpenzkofer, Feb 2, 2017
    #15

  16. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    No.
     
    AcidArrow, Feb 2, 2017
    #16

  17. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    All of them?

    Why wouldn't I use best?
     
    AcidArrow, Feb 2, 2017
    #17

  18. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    With some textures the difference in quality is very small, 'Best' takes a long time to compress and I would assume that many people stick to the default setting ('Normal') for textures where they are happy with quality. That's why I was asking.
    Otherwise, no reason not to use 'Best'.
     
    florianpenzkofer, Feb 2, 2017
    #18

  19. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    I'm of the opinion that spending extra offline cpu time for better looking compression is generally worth it, even if the improvement is miniscule.

    So I would never agree with making "best" worse looking but faster. I'd actually be okay with better looking but slower.

    Maybe you could add a few more tiers so everyone can be happy? Fast, Normal, Best, Insane? Or something like that.
     
    AcidArrow, Feb 2, 2017
    #19

  20. tripknotix

    tripknotix

    Joined:
    Apr 21, 2011
    Posts:
    744
    Thanks, yeah I meant Etc2comp.

    Could you give us a sample of PVRTextool/ETCPACK at best and then Etc2comp at 70.
    Is there an 80? we could use a sample of that as well.

    If its possible to show a 2048x2048 , and its Normal map compressed, on a 3D Model with an upper strong point light and lower weak point light.

    We should be able to tell pretty easily, a good example is a Framed Picture, that has designs around the edge frame in normal map.

    So the actual model is very flat around 100polys or so and the details lost will be obvious between the 2.

    If they look nearly the same, then we'll know the speed is better.
     
    tripknotix, Feb 4, 2017
    #20

  21. Yury-Habets

    Yury-Habets

    Unity Technologies

    Joined:
    Nov 18, 2013
    Posts:
    1,165
    If you could share an uncompressed texture - we could check the quality and speed of the compressors.
     
    Yury-Habets, Feb 10, 2017
    #21

  22. NotCrazyScientist

    NotCrazyScientist

    Joined:
    Apr 16, 2017
    Posts:
    5
    NotCrazyScientist, Apr 16, 2017
    #22
    Yury-Habets and mh114 like this.

  23. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    Interesting, I'll have a look at this.

    Unfortunately the code is written using SSE 4.1 intrinsics with no reference implementation in plain portable C. At the moment the Unity Editor still works with CPUs that only support SSE2, so that's a problem.
     
    florianpenzkofer, May 4, 2017
    #23

  24. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    General update: we are planning to update the ETC compression tool(s) in Unity 2017.2, with some configurability.
     
    florianpenzkofer, May 4, 2017
    #24
    Yury-Habets and mh114 like this.

  25. NotCrazyScientist

    NotCrazyScientist

    Joined:
    Apr 16, 2017
    Posts:
    5
    Configurability will be a good benefit!
    I still don't known how to integrate fast ETC1,ETC2,PVRTC tools despite on willing...

    For SSE2 conformance some changes are needed:
    1) #include <smmintrin.h> // SSE4.1 goes to #include <emmintrin.h> // SSE2
    2) emulation code for _mm_abs_epi16, _mm_cvtepu8_epi16, _mm_cvtepu8_epi32, _mm_cvtepi16_epi32, _mm_hadd_epi32, _mm_min_epu16, _mm_min_epi32, _mm_minpos_epu16, _mm_mullo_epi32, _mm_packus_epi32.

    For normals compression linear weights should be used and masking should be turned off (OutlineAlpha: `w[x * 4] = 1;`). So, bunch of amd64 compilation units with different defines (image vs data, sse4.vex vs sse4 vs sse2) and run-time code path selection seem efficient.

    I can try to do all of this, but command line syntax is a main question.
     
    NotCrazyScientist, May 5, 2017
    #25

  26. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    @NotCrazyScientist Do you mean the command line syntax how the Unity Editor launches PVRTexTool?
    I can look that up in our source code, but it might change any time.
    For ETC we actually launch multiple PVRTexTool processes per texture.

    Current plan for 2017.2 is a separate executable that combines etc2comp, ETCPACK, etcpak and maybe others.
     
    florianpenzkofer, May 5, 2017
    #26
    NotCrazyScientist likes this.

  27. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    Once Unity 2017.2 beta is out you can find some configuration options for the ETC texture compressor in the EditorSettings.
     
    florianpenzkofer, Jun 19, 2017
    #27
    Yury-Habets and mh114 like this.

  28. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    294
    Great! Thanks for listening!
     
    mh114, Jun 19, 2017
    #28

  29. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    Some more details:
    • behavior for existing projects shouldn't change
    • new projects will use a new default that uses etcpak for "Fast", ETCPACK fast for "Normal", Etc2Comp with effort 70 for "Best"
    • the new default should be significantly faster than current in "Best", look better in "Fast" and have similar speed/quality in "Normal"
    • there is an option to switch back to the old behavior
    • there is another "Custom" option where you can assign different compression libraries that we integrated to "Fast", "Normal", "Best" (but you can't add your own library or tool)
    The ETC compression code is in a separate executable that the Editor is using. At the moment the code is not public, but if someone is interested we could probably change that. It's just some glue code around ETCPACK, etcpak, Etc2Comp.
     
    florianpenzkofer, Jun 22, 2017
    #29
    nirvanajie, JustAnotherDude, AcidArrow and 3 others like this.

  30. tripknotix

    tripknotix

    Joined:
    Apr 21, 2011
    Posts:
    744
    Thank you, i really appreciate the fix =)
     
    tripknotix, Jul 6, 2017
    #30

  31. NotCrazyScientist

    NotCrazyScientist

    Joined:
    Apr 16, 2017
    Posts:
    5
    I did an effort to support SSE2 target (but amd64 only). Also effective optimizations were done. I hope Etc2Compress can compete all other packers in theirs medium and slow modes for RGBA ETC2 textures. It still does an exhaustive search making ETC2 as fast as 2..3 x ETC1.
     
    NotCrazyScientist, Oct 9, 2017
    #31

  32. nhnpa-kichangkim

    nhnpa-kichangkim

    Joined:
    May 23, 2016
    Posts:
    9
    Can I find that how Unity executes PVRTexTool or etccompress (detailed parameters)? That information needs to develop in-house Texture-Compression tool for Unity :)
     
    nhnpa-kichangkim, Oct 13, 2017
    #32

  33. Yury-Habets

    Yury-Habets

    Unity Technologies

    Joined:
    Nov 18, 2013
    Posts:
    1,165
    Yury-Habets, Oct 13, 2017
    #33

  34. NotCrazyScientist

    NotCrazyScientist

    Joined:
    Apr 16, 2017
    Posts:
    5
    NotCrazyScientist, Oct 14, 2017
    #34
    frjtrifork likes this.

  35. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    If you launch the etccompress executable on the command line you can see the options.
    Unity will always run it with all of those, in that order (because etccompress doesn't have proper command line argument parsing). But this is not a public interface, so this might change at any time :).
     
    florianpenzkofer, Oct 19, 2017
    #35

  36. Ice_MJ

    Ice_MJ

    Joined:
    Jul 26, 2016
    Posts:
    15
    Hello @florianpenzkofer, I got a bug when I try to use Etc2Comp to compress texture with RGBA channels.(I am using Unity 2017.4.3f1 (64-bit) and the ETC Texture Compressor Behavior is set to Default)
    You can see it through the picture below(the right most image shows the bug I met). Hope you can see it clearly. :)
    The Alpha channel seems to affect RGB channels and cause a fault compresssion result.
    I use google's etc2comp tool(https://github.com/google/etc2comp),everything works fine.
    Not sure am I the only one who meet this bug.:(:(:(

     
    Ice_MJ, Aug 16, 2018
    #36

  37. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    AcidArrow, Aug 17, 2018
    #37

  38. Ice_MJ

    Ice_MJ

    Joined:
    Jul 26, 2016
    Posts:
    15
    Ice_MJ, Aug 17, 2018
    #38

  39. Evaldas_Unity

    Evaldas_Unity

    Unity Technologies

    Joined:
    Jun 23, 2016
    Posts:
    69
    Hi. : ) The alpha versions are not public. However, it is fixed in 2018.2.4f1 which should be coming out next week I think.
    I have also updated that bug's description for consistency : )
     
    Evaldas_Unity, Aug 17, 2018
    #39

  40. Ice_MJ

    Ice_MJ

    Joined:
    Jul 26, 2016
    Posts:
    15
    WOW, Lucky me. Thanks a lot, I download 2018.2.4f1, and the etccompress.exe works! Thanks again.
    : )
     
    Ice_MJ, Aug 20, 2018
    #40

  41. Ice_MJ

    Ice_MJ

    Joined:
    Jul 26, 2016
    Posts:
    15
    Also want to know that, my company has bought the source code of the Unity Engine 2018, but I cannot find the source code of etccompress.exe , so , is there a way to get the source code ? We want to adjust this tool to our project.:)
     
    Ice_MJ, Aug 20, 2018
    #41

  42. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    I'll try to find out..
     
    florianpenzkofer, Aug 20, 2018
    #42

  43. Ice_MJ

    Ice_MJ

    Joined:
    Jul 26, 2016
    Posts:
    15
    Cool, thanks a lot !
     
    Ice_MJ, Aug 21, 2018
    #43

  44. NotCrazyScientist

    NotCrazyScientist

    Joined:
    Apr 16, 2017
    Posts:
    5
    This behavior seems to be flawed for 3D model textures, but it can significantly reduce the size of 2D graphics.
     
    NotCrazyScientist, Aug 26, 2018
    #44

  45. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    Not really.

    It's a bug.
     
    AcidArrow, Aug 26, 2018
    #45

  46. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    It's been a while, but is there a way to use higher effort than 70 with Etc2Comp?
     
    AcidArrow, May 23, 2020
    #46

  47. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    Not really.

    Do you already know that higher effort would give you a noticeable improvement or would you just like try?
     
    florianpenzkofer, May 23, 2020
    #47

  48. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    I don’t know. I would like to try. Maybe I’ll try it outside of Unity and report my findings back here.

    I’ve switched from using ETCPACK Best to Etc2Comp Best, and there is a noticeable drop in Quality but for much faster compression times. I was wondering if with higher effort I could have a medium between I am happier with.
     
    AcidArrow, May 23, 2020
    #48

  49. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    479
    If it‘s just for testing you could also try to patch or replace etccompress.exe that we ship with Unity Editor. There should be a 70.0 as 32bit float in there somewhere.
     
    florianpenzkofer, May 23, 2020
    #49
    AcidArrow likes this.

  50. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,001
    I think I did it. And after a bit of very sloppy testing, I have concluded that 70 is... fine!

    It's into the range where diminishing returns kick in. I was going to do more thorough testing if I got any indication of quality gains, but I'm giving up, 70's fine.
     
    AcidArrow, May 24, 2020
    #50
    Yury-Habets and florianpenzkofer like this.
Page 1 of 2