The Lasso tool Sucks and Pen tool Rules. The lasso tool is a rusty saw and the pen tool is a surgical scalpel. Once you learn how to create a mask with the pen tool you will never look back.

That’s right the lasso tool sucks! Don’t get me wrong the lasso tool has its place, if you want to very quickly select small rough shapes it is fine, but it is not suitable for large tasks or professional quality masking.

In this article, I will demonstrate how to cut out a difficult shape using the pen tool.

What is a Vector Mask?

Basically a vector mask uses vector shapes to selectively mask an area of the image.

If you have created a mask before with either the lasso tools or a grayscale image that would be a pixel mask. A vector mask is a mask created using the vector shape tools. Vector shapes are different because unlike pixel-based shapes you can edit the shape non-destructively. You can also adjust the shape while the mask is active.

Most importantly vector shapes also have unlimited resolution (you can scale them up as big as you like). This is because a vector shape is composed of points and lines.

Why use a Vector Mask?

If I need to do a difficult masking job I almost exclusively use the pen tool and then use the shape as the layer mask, why?

  • You can make smooth curves
  • The pen tool is never finalized – simply put you can always edit the shape at any stage of the process. You can improve the shape of a corner, remove or add anchor points as you go
  • You can start simple and add detail, let’s say you need to cut out something like a car with lots of smooth flowing curves. You can begin by roughly marking out the key landmarks of the car very quickly, you can then take your time and refine the shape in small steps at your own pace
  • You can customize the appearance of the pen tool, if you are cutting out a blue object you can change the colour of the lines to yellow making life easier
  • You can undo the points as you make them this will take you back in steps

Many of the product photos on the top e-commerce sites have been masked using this technique.

How do I do it?

To demonstrate how to cut out a tricky shape I will be creating a mask using the pen tool. The photo I am going to work on is a photo of my hand. This is similar to the type of shapes and angles you might need to cut out if you were cutting out say a T-shirt for example. The lasso tool would not be good enough because it cannot create smooth curves.

The second issue is that the background is of similar hue to my skin color which would make colour selection a bit difficult. This is not an issue with the pen tool.

First things first, make your life easier by rotating and aligning the image. (if needed)

Select the pen tool.

I like to begin by very roughly outlining the shape of the object. I sometimes do this without any curves at all. This bit will feel just like using the polygon lasso tool except you don’t need to worry about it going wrong and making an incomplete selection.

You will notice I have missed out some curves entirely – this is fine because I can come back later and add the necessary points.

If you click once you will create a corner anchor if you click and drag you will create a curved anchor point (bezier). I usually do the whole shape with corner points and then add or convert the points into curves later.

If you have any trouble with clearly seeing the pen tool: maybe it is too thin or the wrong color. You can customize the settings of the tool in the preferences.

Go to EDIT > PREFERENCES > GUIDES, GRIDS & SLICES.

At the bottom, you will find the options for paths here you can adjust the size and the color of the lines.

Top Tip! You may notice the edges of one color seem to bleed into the next color, this is called anti-aliasing. I usually draw my mask on the inside of the object – this prevents the color bleeding into the edges.

So we have a very rough outline of the object

Before we refine the shape it’s important to know about the two types of anchor points: corner and bezier.

Corner: basically point-to-point anchors with a straight line joining each anchor point.

Bezier: this when you have a curve between the anchor points, you can adjust the curve using the handlers.

Next, I edit the shape adding extra anchor points as I need them using the Add Anchor Point Tool. This tool will create a new anchor point on a line. You may notice two little dots on the end of a line – these are the handlers and they control the amount of curve.

Simply click on the part of the line you wish to add a point too.

The second way to refine the shape is by converting the points you already have into bezier (curved) anchor points. This done with the convert point tool simply click on the corner point and drag the mouse to adjust the curve.

If you want to move points around you will need to use the direct selection tool.

I repeat this process on the rest of the image. This takes a bit of getting used too, adjusting the curves can be a bit frustrating at times.

Once I am happy with the mask, I now need to create a layer mask from the shape I have just made.

To do this is I make sure the layer I want to mask is selected and the shape I want to use is also selected. You can select shapes with the path selection tool – it’s below the pen tool.

I select Layer > LAYER MASK > VECTOR MASK > CURRENT PATH

Done! – You can still edit the shape with the pen tool even with the mask applied. You can also decrease the density which is basically the opacity.

Wait my mask is the wrong way around!

This happens from time to time. With the path selected hit the path operation options. This will present a drop-down menu simply pick a different path option – Combine shapes is usually the right one.

If you want to use the traditional method of masking with a pixel mask you can simply right click on the vector mask and hit Rasterize Vector Mask. I advise duplicating the layer beforehand so you have a backup of the vector mask in case things go wrong.

If you lose your shape at any point of the process unless you deleted it, you can find it by going to the paths panel next to the layers & channels (see image above)

The limitations of vector masking.

Vector masking is ideal for solid objects with hard edges.  It is not suitable for objects with soft edges or semi-transparent areas. You can always use a combination of vector and traditional masking.

I recommend vector masking for solid objects it’s perfect for editing product photos. In fact, this is how I do all of my product photos at my job.

The pen tool can be a bit fiddly and frustrating at first but once you nail it you will be able to mask anything. Personally, I think this way of masking is vastly superior due to the fact you can always edit the path.

Happy masking.


David

Works part-time as a Graphic Designer and Website Designer. Studied Animation and has been using and learning design software for over 15 years.