Lasso Mask, Brushes, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Hier diskutieren die Betatester von PhotoLine untereinander und mit den Entwicklern
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Lasso Mask, Brushes, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

One of the remaining fundamental limitations in PhotoLine is Brushes, Clipping Soft Edges, Layer Styles and 3D Light Effects that always operate in 8-bit, even if applied in 16-bit or 32-bit Document or Layer. It is really strange why so powerful and technically advanced app as PhotoLine still limits those tools to 8-bit only. This limitation produce a lot of potential low quality banding problems. It is 100% impossible to paint in Linear gamma. Risk to paint in wide color spaces, Lab, and L* gamma due possible banding problems (it is well known that wide color spaces and log gammas require at least 10-bit depth). This is also very bad for retouching, because soft edges of Remove Brush or Healing Brush will damage original 16-bit source and introduce low quality 8 bit areas into original 16-bit image. Later during color correction those areas may generate visible banding artifacts. Same goes to Clipping Soft Edges in Masks and Lassos. The worst thing is that 8-bit limitation in PhotoLine is not documented and those users who didn't done personal tests probably don't realize that their work may be ruined by image editor.

Problem was discussed earlier in beta version threads as well in my "Large List of Suggestions and Requests for PhotoLine Tools" thread. Also i attempt to use 16 bit Stamp instead of brush and can't see any performance issues:
viewtopic.php?p=49567#p49567
viewtopic.php?p=49604#p49604
viewtopic.php?p=50826#p50826
I also emailed this request to Martin Huber, and he seems agreed that 8-bit brushes are bad, but explain to me that changing all these tools to 16-bit is a complex task and due low amount of requests from other users not a priority bugfix yet.

:arrow: So the idea of this thread is to draw more attention to this bug and collect more votes and requests from other users to fix this bug. For my opinion this is a HUGE technical fundamental problem and i really hope that it could be fixed as soon as possible.

Image
Last edited by shijan on Tue 04 Jul 2023 02:00, edited 2 times in total.
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
User avatar
Herbert123
Mitglied
Posts: 2204
Joined: Sat 12 May 2012 21:38

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by Herbert123 »

You are not the first one to identify these 8bit limits in certain tools in PhotoLine. When I first began to use PhotoLine I pointed out a few of these and a number have been fixed - but not all.

I would welcome it if some or most of these would be addressed. I do, however, understand that it does involve quite a bit of work on the side of the Hubers. I am also not sure whether all of the ones you mention deserve the same priority - some I feel are more important to upgrade to 16 bit or even 32bit than others.

Perhaps a poll would be useful?
/*---------------------------------------------*/
System: Win10 64bit - i7 920@3.6Ghz, p6t Deluxe v1, 48gb (6x8gb RipjawsX), Nvidia GTX1080 8GB, Revodrive X2 240gb, e-mu 1820, 2XSamsung SA850 (2560*1440) and 1XHP2408H 1920*1200 portrait
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

Sure, Brushes are the main priority.

Mask/Lasso Soft Edges currently could be replaced by Blur effect filter. Gaussian Blur effect support 16/32 bit and also way more accurate than basic Box Blur currently used by default in all Soft Edges engine. https://www.pl32.com/forum3/viewtopic.p ... 629#p50629

Layer Styles & 3D Light Effects are less important for sure.
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
Jake123
Mitglied
Posts: 23
Joined: Mon 28 Feb 2022 05:12

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by Jake123 »

+1 vote here, from a digital painter and video game artist. Specially about the brush issue. :)

But I understand that it is a huge revamp in code. I can paint in my style with 8 bits brushes in 16 bits images (I lately always work on 16 bits images, as it gives more "room" for many operations), as my realistic style is kind of painterly and it gets disguised with the actual painting, in my special case, but for those hyper realistic painters (and retouch people) using airbrush-like techniques, it's surely a big problem (and also, I could guess, any manga artist, they're all about subtle gradients in their coloring. They can just use gradients, but....).

Also....maybe I'm wrong, but I was doing some tests today (although my beta is yet 23.40b8, I want to install b11 today, sorry if this has been already changed) , to see how to fix an 8 bit (very) soft shadow (a layer style "outside shadow"), by merging the 16 bits layer with another (empty) 16 bits one, and it resulted a merged layer of 8 bits.

Sorry if there's a more advanced way to rasterize a layer, I'm just very newbie to this app (experienced in Photoshop, though) and could not find it (ie, right click on layer, "rasterize"). There's no problem for this test, as I merged them (previously getting the exact object selection, a rectangle) to actually apply a blur to the inverted selection (the shadow only), and see if a gaussian blur made a difference (I didn't see a difference, but it might be how I have my monitor configured (it's an Eizo ColorEdge CS, though, calibrated)).

But it could be a problem in any merging of 16 bits layers for other purposes, as the 16 bits subtlety (smooth gradients and other fine detail) of the 16 bits gradients is lost if merged to one of 8 bits. Please, feel free to correct me in my surely many mistakes, because while I have worked a lot in image editing (several fields), I am absolutely new to this software, and I can see how advanced it is, so, I might just be not using the correct workflows.

Kudos for such an amazing software, I truly love it. 8)
Last edited by Jake123 on Sun 24 Jul 2022 19:55, edited 1 time in total.
Jake123
Mitglied
Posts: 23
Joined: Mon 28 Feb 2022 05:12

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by Jake123 »

Please, totally disregard my mention about manga artists might be having an issue with this... I just remembered CSP is just 8 bit. Clearly they found their workarounds... A huge amount of manga comic artists (and not just manga ones, ie, the creator of The Walking Dead comic (which originated later the TV show) uses Clip Studio) do use Clip Studio (indeed, I use it from time to time, much less now), and I know even very professional mangakas do use it daily, while it does not support 16 bits in any way.

In another famous app used to paint and draw, Paint Tool SAI 2 (massively used for making art commissions) it does not even give you the option (I purchased it, too). PaintStorm Studio does not either have a 16bit mode, as far as I know. Even Corel Painter does not seem to have a 16 bit color mode. The wonderful Rebelle (specialized in mimicking traditional painting) , neither, it's all 8 bit. So, they will all face bad banding in certain situations. The thing is, now I use actual photo editing software for painting because not only these tend to support 16 bits, but also they don't lack many features which the "specialized" painting tools do lack, which are very needed in real illustration and general graphic work. There is a reason why many triple A games studios do absolutely require their concept artists should use and master Photoshop (instead of Corel Painter or etc).

Probably these painting apps do use a lot of very fine dithering in the brushes to avoid the issue, though, among other "tricks".

So, is not a "lack" compared to what many painting artists (and general graphic artists) use now in the market. Indeed, this app is in a better position in that regard, than most. It'd be an amazing advantage, though. And yep, I do know this is an image/photo editing tool (just that a lot of users are working with image editing apps for a wider number of uses than before. I kind of see that as a good thing, as it multiplies the customers base).
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

Jake123 wrote: Sun 24 Jul 2022 18:11 I was doing some tests today, to see how to fix an 8 bit (very) soft shadow (a layer style "outside shadow"), by merging the 16 bits layer with another (empty) 16 bits one, and it resulted a merged layer of 8 bits.
Seems this problem don't depends of Layer Style at all. Merged layers use bit depth of Background layer in "Picture mode", or bit depth of Document in "Document mode".
ICC profiles behave in same way. Merged layers that have different color profiles automatically converted to ICC profile of Background layer in "Picture mode", or to ICC profile of Document in "Document mode".
Last edited by shijan on Mon 25 Jul 2022 11:14, edited 2 times in total.
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

Sure there are plenty of other apps that don't have 16 bit brushes and probably also may not have 16-bit color support at all. Even Photoshop use some sort of simplification and dithering and so it's brushes looks a little bit "fake" in 16 bit mode. In Photoshop it is also impossible to paint in Linear gamma due those fake 16-bit brushes. But same time there are plenty of other image editors and painting tools that support real 16-bit brushes. So why look at lowest possible level instead of attempt to reach highest possible level of image quality?
Image
Image
Image
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
Jake123
Mitglied
Posts: 23
Joined: Mon 28 Feb 2022 05:12

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by Jake123 »

Yep, now that you mention it, I'm currently using A. Photo for painting. It has some inconveniences, though (until I applied some config changes, I had a lot of lag painting with very large images). I kind of use different apps for different projects (depending on each project needs), though.

At least on my monitor, the AP brush stamp looks like the smoothest of the above images.

Well, any path will be good for me (changed or not). If it's possible, it'd be quite exciting. :)
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

Just discovered one more 8-bit situation. Here is how to reproduce:
Create 16-bit document.
Enter to "Mask Mode"
Paint with 16-bit raster Stamp Tool (explained here https://www.pl32.com/forum3/viewtopic.p ... 826#p50826)
Exit to "Mask Mode"
Apply Flood Fill to Lasso Selection
Apply Levels
And it all looks like 8-bit :(

Image
Image
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

And another surprise-surprise:
Switch 16-bit layer to "Clip" mode (it remains 16-bit)
Put it inside Image Layer
Layer Mask now auto converted to 8-bit Gray image :(

Why it happens like this? Is it a bug or designed to make layer somehow compatible with some other 8-bit only Mask/Lasso tools?
Image
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

UPDATE.
Seems it is impossible to keep mask in 16-bit at all. Even if don't put mask inside layer. When layer switched to "Clip" mode it remains in 16-bit in Layer Attributes, but mask renders in 8-bit. This is really bad and confused situation.
Earlier i was thinking that only mask soft edge generator is 8-bit only, bit it seems that all mask engine is limited to 8-bit :(
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
User avatar
russellcottrell
Mitglied
Posts: 254
Joined: Sat 26 Jul 2014 10:13
Location: California

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by russellcottrell »

+1 for 16-bit brushes.

There is no way to mask a color filter; but a workaround would be to mask the adjustment layer, duplicate the layer, invert the mask, then modify or delete the color filter.

But the overlap between the masks has banding, even if the adjustment layers are 16-bit. Example attached. I am guessing that this is due to the 16-bit brush.
You do not have the required permissions to view the files attached to this post.
User avatar
Hoogo
Betatester
Posts: 4033
Joined: Sun 03 Jul 2005 13:35
Location: Mülheim/Ruhr

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by Hoogo »

russellcottrell wrote: Wed 28 Dec 2022 03:49But the overlap between the masks has banding, even if the adjustment layers are 16-bit. Example attached. I am guessing that this is due to the 16-bit brush.
I think it's because of the maths of transparency. 50% opacity + another 50% opacity don't sum up to 100% opacity, but to 75%.
2 half whites.jpg
Same effect happens with clipping layers in groups: They are applied to each layer, not to the result of the group, and that can make the same difference.
If you're used to photography: A 2x filter takes away 50% of light.Adding a 2nd filter won't remove the other 50%.
You do not have the required permissions to view the files attached to this post.
----------------
Herr Doktor, ich bin mir ganz sicher, ich habe Atom! /Doctor, doctor, I'm sure, I've got atoms!
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Brushes, Clipping Soft Edges, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

Unfortunately vector gradient applied as mask (in Luminance mode) to vector layer still produce 8-bit banding in 16 or 32-bit document.
In Hoogo's example we mostly can see additional banding due 8-bit document depth that affect quality of layers blending mode. If switch document to 16-bit it will be no banding between blended layers, but 8-bit banding (mixed with some dithering under magnification) in layer mask will remain.

UPD: Seems it was a partial fix of this problem somewhere in 23.9b12 and 24.00. Great!
"Layer mask and clippings layers with 16-/32 bits
In documents with 16- and 32-bit color depth, 16- and 32-bit images can now be used as layer masks and clipping layers with full bit depth."


UPD: Note from email discussion with Martin Huber about possible improvements:
It works only if convert clipping layer to Gray and turn off the transparency. Also note that menu command from layer attributes bottom bar "Convert Layer to Layer Mask for Previous" or "Create Layer Mask" functions still create 8-bit layers

UPD: Also in PhotoLine 24 Vector gradient clipping layer used in Luminance mode with 16-/32 bits image still looks like 8-bit, but Martin Huber told me that problem will be fixed in future update.

UPD: Vector gradient clipping layer used in Luminance mode with 16-/32 bits image works in 16/32-bit in 24.40b1. Great!
Last edited by shijan on Fri 14 Jul 2023 02:03, edited 1 time in total.
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302
User avatar
shijan
Mitglied
Posts: 1573
Joined: Mon 23 Dec 2019 15:21
Location: Ukraine

Re: Lasso Mask, Brushes, Layer Styles & 3D Light Effects are 8-bit only and it is a HUGE problem

Post by shijan »

Seems accidentally i find a "hack" for true 16-bit Mask Soft Edge :)
If put inactive or invisible Adjustment Layer inside Mask layer, then Mask Soft Edge start to work in 16-bit.

Image
Image
PhotoLine UI Icons Customization Project: https://www.pl32.com/forum3/viewtopic.php?f=3&t=6302