Cookbook recipe #4: part 2

In part 1 of this recipe, we saw how to determine if a shader was in a particular channel. But that left a problem: what if the shader was itself contained in a Layer shader? The function IsInChannel() would return false in that case, indicating that our shader was not in the specified channel. So we need some way to check a Layer shader to see if it contains our shader in one of its layers.

This is part 2 of the recipe: with this link you can go back to part 1

If there is a shader in the channel but it's not our shader, instead of returning a null pointer, what we need to do is check if it is a Layer shader. If it isn't we can return a null pointer. If it is, then we need a function to iterate through all the layers in the shader to determine if one of them is an instance of our shader. If so, then the function can return a pointer to that shader which we can compare with the instance of the shader we are testing; if it is not, then return a null pointer.

Iterate through the layers of a Layer shader

Let's assume that in IsInChannel() we have found a Layer shader and we pass this to the new function. This is how it might work:

BaseShader* OurShaderClass::WalkLayers(LayerShader* ls) const
{
BaseShader* ret = nullptr;
LayerShaderLayer* lslayer = nullptr;
GeData gd;

// get the first layer in the Layer shader, if there is one
lslayer = ls->GetFirstLayer();
while (lslayer != nullptr)
{
  // there is a layer, but is it a shader layer - it could be one of the Layer shader effects such as HSL
  if (lslayer->GetType() == LayerType::TypeShader)
  {
    // it is a shader layer but we need to get the shader link then the shader in that link (if any)
    if (lslayer->GetParameter(LAYER_S_PARAM_SHADER_LINK, gd) == true)
    {
      if (gd.GetType() == DA_VOID)
      {
        const BaseLink* const link = reinterpret_cast<const BaseLink*>(gd.GetVoid());
        if (link != nullptr)
        {
          // we have the link, now to find a shader in it
          BaseShader* shd = static_cast<BaseShader*>(link->GetLink(ls->GetDocument()));
          // yes there is a shader but is it our shader?
          if (shd != nullptr && shd->GetType() == OUR_SHADER_ID)
          {
            // it is an instance of our shader, so return it
            return shd;
          }
          else if (shd != nullptr && shd->GetType() == Xlayer)
          {
            // the remaining issue is that a Layer shader can contain another Layer shader
            // so we need to call this function recursively since our shader could be in that nested Layer shader
            shd = WalkLayers(static_cast<LayerShader*>(shd));
            if (shd != nullptr && shd->GetType() == OUR_SHADER_ID)
            {
              // it does contain an instance of our shader, so return it
              return shd;
            }
          }
        }
      }
    }
  }
  // get the next layer
  lslayer = lslayer->GetNext();
}

// return either the shader pointer to our shader, or a null pointer if we didn't find one
return ret;

}

That should handle a single Layer shader and the much less common nested Layer shaders. You will no doubt have spotted one remaining issue: if the function finds an instance of our shader, it returns immediately. What happens if the Layer shader contains more than one instance? Resolving that is, as they say, left as an exercise for the reader.

Using WalkLayers() in the IsInChannel( ) function

The IsInChannel() function from part 1 can now be expanded to check for Layer shaders as well. Here is the finished function, slightly simplified to reduce space:

Bool OurShaderClass::IsInChannel(GeListNode* node, Int32 chnID) const
{
BaseList2D* blist = nullptr;
BaseMaterial* mat = nullptr;
BaseChannel* channel = nullptr;
BaseShader* shd = nullptr;
BaseShader* ret = nullptr;
LayerShader* ls;
Bool success = false;

blist = (static_cast<BaseList2D*>(node))->GetMain();
if (blist != nullptr && blist->GetType() == Mmaterial)
{
  // get material
  mat = static_cast<BaseMaterial*>(blist);
  // get the required channel
  channel = mat->GetChannel(chnID);

  // get the channel shader if there is one
  if (channel != nullptr)
  {
    shd = channel->GetShader();
    if (shd != nullptr)
    {
      // first, is this a shader of our type and if it is, is it the correct instance
      if (shd->GetType() == OUR_SHADER_ID)
      {
        if (shd == node)
          success = true; // it's our shader and it's in this channel
        else
          success = false; // it is our type of shader, but it isn't the correct instance
      }
      else
      {
        // it isn't our shader type at all, but it could be a Layer shader
        if (shd->GetType() == Xlayer)
        {
          ls = static_cast<LayerShader*>(shd);
          // iterate through all the layers, the function will also handle nested layer shaders
          ret = WalkLayers(ls);
        }
        // is the shader returned from WalkLayers() the correct instance of the shader?
        if (ret != nullptr && ret == node)
          success = true;
        else
          success = false;
      }
    }
  }
}

return success;
}

Conclusion

With these two functions you can easily check which channel a shader instance is in, either at render time or outside of rendering. I hope this has been of some use. I've used these functions extensively in my 'Space' series of shader plugins, and they seem to work fine.

Go back to part 1

Page last updated June 1st 2025