AlphaTest + Stencil Masking in XNA

Our team decided we want a smooth and consistent look for the environment and terrain in Attraction. At the time we were developing our level editor which uses a tile-based system similar to many 2D games. We were looking at ways to avoid a really "repeated" look due to each tile being used over and over to generate a level. Becky suggested we try to "mask" the tiles with a larger, repeating texture. The following explains the process.

The basic idea is as follows, we want this effect:

Here, the purple part is at 50% opacity. The green circle texture is repeatable. The "grass" on the purple tiles still shows through in the final result. Here's how it's done.

First and foremost, set the graphics format like so:

graphics.PreferredDepthStencilFormat = DepthFormat.Depth24Stencil8;

We are going to utilize XNA's stencil masking capabilities. The following are the two stencil masks we need to use.

//This is a stencil. It keeps track of pixels. But only the ones we tell it to.
//In this case, every pixel we pass to this stencil will be written to the buffer because we are using CompareFunction.Always
public static DepthStencilState AlwaysStencilState = new DepthStencilState()
{

       StencilEnable = true,
       StencilFunction = CompareFunction.Always,
       StencilPass = StencilOperation.Replace,
       ReferenceStencil = 1,
       DepthBufferEnable = false,
};


//This one will check what stuff was written to the stencil buffer and will help us to know what pixels to mask.
public static DepthStencilState EqualStencilState = new DepthStencilState()
{
       StencilEnable = true,
       StencilFunction = CompareFunction.Equal,
       StencilPass = StencilOperation.Keep,
       ReferenceStencil = 1,
       DepthBufferEnable = false,
};

These masks will help us "slice out" the portions of a texture that we wish to mask. The AlwaysStencilState will pass every pixel of the texture-to-be-masked to the buffer we are using. The EqualStencilState will then check the pixels that were written and see if their alpha channel passes the alpha test; in this case, if they have an alpha of 127 out of 255 (50%). Great, we have some stencils set up. But how do we use them?

First, we make a buildMask method:

private void buildMask()
{
        _bounds = _camera.Viewport.Bounds;
        _projection = Matrix.CreateOrthographicOffCenter(
             0,
             _spriteBatch.GraphicsDevice.PresentationParameters.BackBufferWidth,
             _spriteBatch.GraphicsDevice.PresentationParameters.BackBufferHeight,
             0,
             0,
             1);
}

This defines a couple of member variables for us. _bounds is for the size of the buffer we will be writing to. Not used in this particular example. _projection sets up the correct view matrix for our masking needs.

Now set up a couple of alpha tests. One to check for 50% opacity and the other to check for more solid values.

       //Create Alpha Test Effect
        _alphaEffect = new AlphaTestEffect(_spriteBatch.GraphicsDevice);
        _alphaEffect.AlphaFunction = CompareFunction.Equal;
        //This value can be 127, 128 or 129 depending on the program used to create the tile. We're shooting for 50% opacity

        //Apparently GIMP gives us 127
        _alphaEffect.ReferenceAlpha = 127;

        //Create next Alpha Test Effect for grass
        _alphaEffect2 = new AlphaTestEffect(_spriteBatch.GraphicsDevice);
        _alphaEffect2.AlphaFunction = CompareFunction.Greater; //Use the "greater than" comparison, the alpha value of the pixel must be greater than the ReferenceAlpha
        _alphaEffect2.ReferenceAlpha = 130;              //The value to test against is 130 out of 255

        //Give the alpha test the correct projection matrix
        _alphaEffect.Projection = _projection;
        _alphaEffect2.Projection = _projection;

Once all that is set up, we can simply draw. Draw in this order 50% opacity parts, opaque parts, mask.

protected override void Draw(GameTime gameTime)
{
        GraphicsDevice.Clear(Color.Transparent);

        //Clear the stencil's buffer
        GraphicsDevice.Clear(ClearOptions.Stencil, Color.Black, 0, 0);

        //Paint the layer to the stencil buffer
        _spriteBatch.Begin(SpriteSortMode.Deferred, null, SamplerState.PointClamp, AlwaysStencilState, null, _alphaEffect);

        //Draw the tile
        _spriteBatch.Draw(_tile, new Vector2(200,200), null, Color.White, 0f, Vector2.Zero, 1f, SpriteEffects.None, 0);

        //End
        _spriteBatch.End();

        //Now we draw the "grass" part of each tile. It is the only part that gets drawn to the rendertarget because of alphaEffect2
        //which says, if you have over 130/255 opacity, get drawn son! We don't use a stencil here because we don't need to keep track
        //of these pixels because we aren't masking them later.

       _spriteBatch.Begin(SpriteSortMode.Immediate, null, SamplerState.PointClamp, null, null, _alphaEffect2);

        //Draw me some grass!
        _spriteBatch.Draw(_tile, new Vector2(200, 200), null, Color.White, 0f, Vector2.Zero, 1f, SpriteEffects.None, 0);
        //Done with grass!
        _spriteBatch.End();

        //Now we paint the mask texture
       _spriteBatch.Begin(SpriteSortMode.Immediate, null, SamplerState.PointClamp, EqualStencilState, null, null);

        //Draw the mask
        _spriteBatch.Draw(_maskTex, new Vector2(200, 200), null, Color.White, 0f, Vector2.Zero, 1f, SpriteEffects.None, 0);

        //End
        _spriteBatch.End();

        base.Draw(gameTime);
}

Take a look at the linked project to see a fully working example. Feel free to use the code however you please but please give me credit somewhere. Thanks!

Click here to download sample project.