Exp: Selections and Masks

[Alan9940]

Hello Ken,

Thanks for the detailed update! I will play around with your suggestions and see what comes of it; I've had a very difficult time understanding the BM tool and have yet to get anything useful from it. For fact, I've yet to observe these transitional areas you speak of...but, based on your comments maybe I'll at least get to see something happening that I expect to happen!

Best regards,
AlanH

[Alan9940]

Hello Ken,

I played around a little with the suggestions in your previous post, but more than anything I'm coming to the conclusion that I just fundamentally don't understand how the BM tool is supposed to be used or what to expect from its use. And, frankly, I'm really surprised that no one seems to be able to explain its use and that the two of us just poke away at it hoping to stumble across something worthwhile. Somebody...the brothers, maybe...has to understand what this tool does; it frustrates me that we're trying to figure it all out with no help. Anyway, whining over...

Here's what I'm seeing after trying your suggestions:

1. I used Automask with a fairly low tolerance to select a good majority of the hair without hitting too much of the background (soft edge = 0).

2. Zoomed into 100% and switched to the BM tool.

At this point the selection turned to a yellow overlay and it was clear that the edges were hard.

3. Reverted back to the original, re-did step 1 with soft edge = 1.

4. Zoomed into 100%, switched to the BM tool, and like step 2 revealing the yellow I think I saw some orange which represents transitional areas, right?

5. Reverted back to the original, re-did steps 3 & 4 except this time I added Enhance Edge = 1.

I really couldn't see much difference between no use of Enhance Edge vs when used. Anyway, at this point I painted over unmarked strands of hair with the "keep" brush, hit apply, and then Extract Selection to a new layer. Basically, what I wound up with was a yellow/green extraction of the marked hair on transparency. I have no clue what this means nor does it seem useful for anything! Sorry, but as I said the BM tool has baffled me and I don't easily give up on these sorts of problems.

I look forward to your next report and findings.

Best regards,
AlanH

[photoken]

Hey Alan,
I share your frustration with trying to figure out the BM tool. It's maddening because I can get a result really close to being great, but I'm getting lost in trying to perfect it.

Your step 4: Yes, those are the "transitional" areas I was talking about. "Transitional" is probably the wrong term, but those are the edges we want to be semi-transparent in order to get a nice feathered edge to the strands of her hair once the image is extracted.

As you did, I didn't notice much difference between using Enhance Edge or not before using the BM tool. Like I said, that was only one thing I wanted to try as a test to see what the effect, if any, would be.

There's one more thing I learned not to do -- do not use the BM "Apply" more than once before doing the final extraction. In other words, I was making some adjustments, hitting "Apply" and then extracting to test the results of my adjustments. Then I was undo-ing the extraction and making further adjustments, then hitting "Apply" and then extracting again. What was happening is that every time I hit "Apply", the new and the old Working Area sections were being recalculated and giving very ugly results.

At least, I'm learning what not to do....

I did do another series of tests, and I can confirm that for the difficult problem of her hair, it's going to be necessary to use the BM tool because that's the most efficient way to remove that orange colour from the background from around the strands of hair. What I did was to simply use the AutoMask tool to select her hair and then its Enhance Edge to create a selection. After using "Create Mask from Mask" to hide the background, I was left with orange or yellow "halos" surrounding the strands of her hair. So, the BM tool with its "Remove Color Border" is the only way to eliminate those orange edges.

I've got a gut feeling that I'm over-complicating things and that's what is leading me into trouble. So, my approach will be trying to find the simplest method of selecting with the AM tool and extracting with the BM tool as I can.

[photoken]

Basically, what I wound up with was a yellow/green extraction of the marked hair on transparency. I have no clue what this means nor does it seem useful for anything!

Ah, now I see what you're talking about. You're seeing the extracted image as well as the mask used to do the extraction. Simply use Edit...Deselect (Ctrl+D) to remove the selection/mask and you'll see the extracted image only.

[photoken]

Here's where I am, so far.

I used the AutoMask tool at 400% magnification with a Soft Edge of "1" to select strands of hair (and to remove from the selection bits of the background). I didn't worry at all about the "holes" left in the solid areas of her hair and on her face because I knew they would be very easy to correct. And, as a matter of fact, I switched to the Mask Brush to quickly add those areas in. (By the way, I'm really liking the AM tool -- it's a breeze to vary the tolerance by dragging and seeing what will be masked, and it's a snap to switch from adding to subtracting to the mask by using the Shift and Ctrl keys.)

This is the resulting selection:

JA automask.jpg

Then, I switched to the BM tool and didn't do a darned thing except to hit "Extract". (The extracted image is displayed against a separate solid black image layer to better show what's happening along the edges.)

JA simple BM.jpg

Now, what I'm trying to figure out is how to use the brushes in the BM tool to refine the areas that will be in or out and the edges where I want the colour contamination removed. I think the BM tool needs to be the one to do that because I reached a point of diminishing returns with the AM tool in trying to add in the extremely fine and diffuse strands of hair.

[Alan9940]

As you did, I didn't notice much difference between using Enhance Edge or not before using the BM tool. Like I said, that was only one thing I wanted to try as a test to see what the effect, if any, would be.

Yep, I knew you hadn't actually tried the Enhance Edge, yet. I just wanted to post what I found to see if it jived with your tests, once you had a chance to try it. Apparently, we both came to the same conclusion regarding use of Enhance Edge in this particular exercise.

There's one more thing I learned not to do -- do not use the BM "Apply" more than once before doing the final extraction.

Yeah, I came to this same conclusion during all my playing. I'd do the initial selection, switch to the BM tool, paint with the various brushes to define in/out/compute areas, hit Apply, lather-n-repeat. It really, really messed things up!

I've got a gut feeling that I'm over-complicating things and that's what is leading me into trouble. So, my approach will be trying to find the simplest method of selecting with the AM tool and extracting with the BM tool as I can.

I know...that's one of the things that frustrates the heck out of me! I think I understand the principles behind the BM tool, yet, when I apply it it just doesn't work the way I would expect! That's why I keep whining that if someone would jump into this thread and, at least, let us know what to expect we could stop wasting time stabbing at it in the dark. Just sayin'...

Best regards,
AlanH

[Alan9940]

Ah, now I see what you're talking about. You're seeing the extracted image as well as the mask used to do the extraction. Simply use Edit...Deselect (Ctrl+D) to remove the selection/mask and you'll see the extracted image only.

Ah, duhhh... Didn't think of that!

Thanks!

Best regards,
AlanH

[Alan9940]

Hello Ken,

OK, with your help and based on the post with the attachments I think I'm getting closer to an acceptable result, if I really put some time into creating that initial selection; which I didn't do here because...well...we're just learning.

Like you, I made an initial selection with Automask, soft edge "1", varying the radius until I got what I felt was a useful initial selection of the hair. Oh, and I had lowered the tolerance slider to something less than the default of 50%; I think I used something like 36%. And, another useful tidbit that seemed to work for me with this initial selection was to click on a middle value blonde/tan of her hair, and then drag for the radius. I tried many different shades of the hair to test my theory, but that mid-land blondish/tanish color seemed to work best.

Then, still with Automask and zoomed in to 400% I varied the tolerance setting, used Shift/Ctrl to add/remove from the initial selection, and single-clicked to modify the selection. If anything went wonky, I just hit Ctrl/Z to undo. I was pretty happy so far!

And, here's where my technique varied from yours... I, then clicked on Show Mask and used the Masking Brush and/or Eraser to, again, add/remove from the selection. Zoomed in at 400% I found use of these brushes allowed pretty precise control to work on the misc strands of hair not selected during the initial workup. As you can see from the attachment, I didn't bother with every little strand of hair missed; especially under her chin on the left. If I was doing this for real, of course I'd spend the time necessary to pick up every single fly-away hair. Why does she have to have so much hair, anyway!

When I was satisfied with the selection I had, I clicked on the BM tool, set to "layer", and hit Extract. Added the solid black layer below to reveal all my little faux pas and cropped for the attachment.

My gut feeling is that the in/out/compute brushes of the BM tool should do what I did with the Masking Brush/Eraser, and probably with more precision, but I've yet to figure out how to use the BM brushes. It doesn't seem logical to me at all that if our theory is correct that Apply should be hit only once, that anyone would be able to precisely mark all in/out/compute areas for a single pass of Apply. PS's Refine Brush doesn't work that way...I can "paint" as much as I care to until I'm satisfied that tricky areas are properly selected. I want the BM tool to give me that same capability! I don't think that's asking too much!!

Best regards,
AlanH

[photoken]

OK, with your help and based on the post with the attachments I think I'm getting closer to an acceptable result, if I really put some time into creating that initial selection; which I didn't do here because...well...we're just learning.

What I finally did (sometimes I have to hit my head against a wall a couple of times before I smarten up. ) is to do a really good selection with the AM tool, then use the Mask Brush to include all of her face and exclude some things like the dark line where the taxi's door meets the body panel, then save that mask as an external PLD Mask file. Now I can screw things up as much as possible and instantly get back the same (very good) selection.

Like you, I made an initial selection with Automask, soft edge "1", varying the radius until I got what I felt was a useful initial selection of the hair. Oh, and I had lowered the tolerance slider to something less than the default of 50%; I think I used something like 36%. And, another useful tidbit that seemed to work for me with this initial selection was to click on a middle value blonde/tan of her hair, and then drag for the radius. I tried many different shades of the hair to test my theory, but that mid-land blondish/tanish color seemed to work best.

What works for me is to set the initial Tolerance value in the tool's settings to "15". That's what is used when the Left Mouse Button is initially put down on the image, and I can then drag out the "tolerance circle" to suit. I found that it's pretty easy (using the "Add" mode) to then successively select different colours of hair to include those hairs in the selection.

And, here's where my technique varied from yours... I, then clicked on Show Mask and used the Masking Brush and/or Eraser to, again, add/remove from the selection. Zoomed in at 400% I found use of these brushes allowed pretty precise control to work on the misc strands of hair not selected during the initial workup.

Yep. I found that to be the case, too.

My gut feeling is that the in/out/compute brushes of the BM tool should do what I did with the Masking Brush/Eraser, and probably with more precision, but I've yet to figure out how to use the BM brushes. It doesn't seem logical to me at all that if our theory is correct that Apply should be hit only once, that anyone would be able to precisely mark all in/out/compute areas for a single pass of Apply.

I agree. What's also frustrating for me is that if a very fine strand of hair is not part of the initial selection, I can simply paint over it with the BM's Working Area brush and it will be nicely included in the extraction.

[photoken]

Hey Alan,
How about this for a rough cut:

JA select colors result.jpg

I took a new approach and used the Tool...Colors...Select Colors function to select the backgrounds instead of trying to select all her hair.

Here's the settings I used, supplied as a preset for that tool:

Jen Hair 01.zip

Unzip the PLGD file to a convenient directory and use the little icon in the settings to load the preset:

sel col settings.png

After the Select Colors ran, I used the Mask Brush to quickly eliminate all the selected areas except for her head. Then I ran the BM tool and extracted her head.

I have no idea why there still are bizarre areas inside her face, even though I had used the BM "In" brush on it, but since the extracted image is only an intermediate step towards creating a layer mask, those areas will be easy to clean up on the layer mask.

I really like the softness around the top of her head and along the right side -- couldn't get that effect when I made the selections with the AutoMask tool....

I still am very disappointed by the orange halos around the open areas of her hair near her neck. I tried using the BM Working Area brush on them, to no avail. I'm becoming convinced that the BM "Remove Color Border" option needs to be significantly enhanced. We definitely need an "Agressiveness" option for it.

BTW, two things I just discovered:

1. There's no need to use the BM tool for a simple extraction. Use the Tool...Lasso...Extract Selection to directly get to the Extract Selection dialog.
2. That BM Working Area brush is only meant to paint over the edges between the object and the background. In other words, if we had started this Experiment with a simple image such as a soup can on a kitchen counter, we would have quickly painted the Working Area around the four edges of the can, hit "Extract" and been done with this in 75 seconds or less. It's only because we're dealing with superfine hair that we're getting such grief.

I also realize that this technique is only appropriate for images like this one where the background is only one or two colours. If she had been photographed in a park where the background was many small areas of light and dark leaves as well as bits of sky, or if she had been photographed on a city street where the out-of-focus urban background would have been many multi-coloured areas, we'd be forced to select her hair as we've been doing. So I still need to practice the original techniques we've been using....

[photoken]

OK, I think I'm on to something.

1. Used the Select Colors from the previous post to get the initial selection.
2. Used the Mask Brush to remove the selection from all areas except her hair and head.
3. Continued with the Mask Brush to add in all areas of her face and the solid areas of her hair.
4. Switched to the Border Mask tool, zoomed in to 400% and used a 2px Working Area brush to paint over the edges of her hair where there were gaps near her face and the orange halos were occurring. Also used the Object brush to include as "In" any areas of her face and solid areas of hair that were marked as "Transitional" (orange).
5. Extracted the image, removing the color border.
6. De-selected everything.
7. With the extracted image layer as the active layer, used Tool...Mask...Mask from Alpha to get a new selection.
8. Switched to the AutoMask tool just to use its "Enhance Edge" with a value of "1".

Looking good:

JA select colors result 02.jpg

There are still some areas of background showing along the edges of some of the strands of hair, but not too many, and I think I can correct those after creating a layer mask from this extracted image. At any rate, this seems the best I can get out of the "Remove Color Border" -- the orange background still shows in some places even though I re-defined those edges with the Working Area tool....

[photoken]

Got it!

I can get an excellent result by selecting the hair with the AutoMask tool, then defining the areas with the tools in the Border Masking tool:

JA automask result.jpg

I used the AutoMask tool instead of the Select Colors in step 1.
Didn't have to bother with step 2.
Didn't do steps 6 & 7, either -- I simply re-used the existing mask for step 8.

There are still a couple of nits here and there, but they can be easily corrected by either erasing or painting on the layer mask (once it's created).

[Alan9940]

Hello Ken,

Thanks for the update!

IMO, based on this low resolution image and the very similar coloring between background elements and Jen's hair, it looks like you've pushed the limits of what's currently available in PL! A tip-o-the-techie-hat to ya for sure!!

I will definitely play around with your technique and I'm sure a high resolution original would certainly benefit any masking effort in PL, but I'm still finding the whole methodology and, more importantly, results of PL masking on my own images to be lacking. I know I keep sounding like the proverbial scratched record, but every time I do just about any masking in PL (except for straightforward hard well defined edges, etc) I find myself longing for the capability of PS Refine Brush or OnOne's old Perfect Mask program. If OnOne had just implemented the idea of the "perfect brush" (edge aware brushing) in the prior masking program IMO it would have been a beautiful thing! But, they didn't. They chose to incorporate pieces of their masking technology into a couple of the other suite modules in the latest version of the suite, and eliminated PM. Bummer!!

Thanks for all your collaborating on this thread as we worked through these various techniques. To be honest, I think you expended yeoman effort!!

Best regards,
AlanH

[photoken]

Hi Alan,
Thanks for the kind words, and thanks for all your work collaborating on this -- this was a valuable learning experience for me. A real "Voyage of Discovery", for sure....

Now that I know how to use the Border Matting tool, my thoughts about it have completely changed and I really like the darned thing.

The "Aha!" moment for me came after I used the Select Color technique to get a selection. When I opened the BM tool on that selection, I zoomed in to 400% and looked around the edges and saw how the green, orange, and red masking areas were automatically defined. Around the top of her head and the sides of her hair, the orange Working Area was very wide. That taught me how to get a nice, soft diffuse extraction for those edges. I also saw how important it is to pay attention to where the green mask and the orange mask meet -- it's important that the green mask does not include any areas of background to keep. Where the edge between the orange mask and the red mask meet is relatively less important.

Looking forward to seeing a comparison with what you can get using your preferred masking software, if you have the time. I never spent much time with PS and its Refine Edge, so I'll have to leave it to the more experienced PS users like yourself to elaborate on it if a Feature Request is posted. It does sound like a good tool, though.

BTW, I've got an idea about getting something similar to that "Chisel" tool you mentioned from the other software. Will post a message here showing what I tried later tonight for your comments.

[Alan9940]

Hello Ken,

Actually, the very first post of mine on pg 1 of this thread reveals about the best result of any of the third-party tools I currently own for masking; that test was worked up in Topaz Remask 4. I'm sure with a bit more time and effort with Jen in RM 4 I could eliminate the fine orange coloring around her back of arm and lower torso area. The dark hair in shadow under her chin would represent the biggest challenge! But, as you've said before, with any of these techniques it just takes some time...

I've tried repeatedly with OnOne's masking tools with Perfect Photo Suite 9 to get a useful result, but I've yet to be happy with anything generated. I haven't played with this image too much in PS with its Refine Edge, but I will and I'll let ya know what I find vis-a-vis that tool. Looking forward to your post regarding a possible chisel-like mask refinement technique in PL.

Best regards,
AlanH