Has anybody implemented GPU/vertex shader bones yet?

[Redman]

I am curious to see if anybody has implemented GPU bones Shader yet and if they would be willing to share.  I see it mentioned in past topics, but no solution.  In the meantime, I will try to write one and share it if I'm successful.

I can't continue my project without it as my frame rates drop dramatically if I call animateSkin(...);

Thanks.

[raft]

how many polygons is your model?

as you had said, there was some work on that but nobody published anything as far as I know.

also beware, although HW animation with a shader is possible and will increase the performance, polygon level collision detection will not work in software as software will not be aware of updated mesh.

[Redman]

My model is about 500 polys and there are about 20 bones in the model.  If I add 20 animated models, the framerate drops to about 8-9 fps using a Droid Turbo 2.  Without the animation, the rendering stays at the 60fps cap.

Thank you for the note on the software / hardware, as that obviously would be affected.  Bounding box calculations, etc...  Thankfully I don't need to worry about any of that for what I'm working on.

I started working on moving the processing to the GPU, but it's going to take me a little time as I only get about 1 hour to myself at night and I'm not familiar with direct OpenGL commands.  But I will publish what I create with a "use at your own risk" stipulation.

It looks like I'm going to need to alter the JPCT core code itself for what I want to do.  I'm going to let the bones software handle the skeleton bone matrix transformations, and then pass in transform Matricies to the shader on every frame (what Bones calls the currentPosePalette).  Should just be 1 per every bone used.  The data for the weights and jointIndicies is going to be more troublesome for me as I would like to just have the shader use the weight and jointIndices as an attribute float[4] & int[4] based off the index of the vertex being processed.  I need to read up and do a little research.  Please feel free to drop me suggestions.

As a side-note, I have separated out the processing of the bone matrix transformations on every frame for the 20 models, and it keeps the framerate at 60.

More to follow...

[raft]

you can go for a hybrid solution. i.e. using skeleton animation create jPCT's regular mesh based key frame animations on the fly and use that afterwards. this will increase performance but you may hit memory limit (depends on number/length of your animations)

[Redman]

I have implemented a hybrid solution off-loading all of the per vertex calculations to the GPU, and it runs great.  My device is keeping at the 60-62 fps cap animating 20 models on the screen (as opposed to 8-9 fps).

I have a little clean up to do with my code and then I will post the solution.  You don't need to alter the Bones or JPCT library, but you do need to add a couple helper classes using their namespace in order to access namespace restricted variables/methods.

More to follow.

[raft]

cool, you make quick progress

[EgonOlsen]

but you do need to add a couple helper classes using their namespace in order to access namespace restricted variables/methods.

Can you provide a list of the jPCT releated ones? Maybe I can expose them in a cleaner way.

[Redman]

Egon,
I will do.

I'll try to post the code tonight, but I personally don't think you need to expose anything additional.  I was working on some refactoring/code-clean-up last night, as I wanted to clean it up before posting, and now it really needs to reside in the namespace.

Originally to test the functionality, I was using a separate shader for each 3D model instance and updating the uniforms manually on every frame.  Obviously this is not a good approach, but it was for quick testing of the shader and the passing of the data into shader.

Last night I tied it into the JPCT render pipeline, but in a sort of hacky-approach as I was attempting to add it without changing the JPCT library, which I'll describe below that requires the namespace.  The way the code works is as such:

• com.threed.jpct.GPUAnimated3D extends Animated3D (which extends Object3D).  In this class, I am overriding the void getInternalShader() call.  It needs to be in the JPCT namespace to override the call.  The getInternalShader() gets called once in the JPCT render pipeline on the render visList before it renders the object.  This method shouldn't really be exposed as its meant to be internal (as the name describes  ).  Inside this overwritten method, it fetches the shader and checks the instance name of the shader attached to the object and if it extends the GPUAnimated3DShader class, it will cast it and call a new shader method passing the GPUAnimated3D.  The class constructor fetches the attached mesh and appends skin vertex attributes if they don't already exist (cloning using same mesh)
• com.threed.jpct.GPUAnimated3DShader extends GLSLShader.  Kept in the namespace for uniformity.  This class constructor sets up handlers for the skin vertex attributes.  It also contains a method for setting the skeleton pose matrices as a couple of uniforms from a passed in GPUAnimated3D (called by the GPUAnimated3D).

This ties it into the render pipeline, where if the object added to the world is a GPUAnimated3D, it will update/override the shader uniforms, per Object, of its SkeletalPose before its rendered.  Now you can use 1 shader for all GPUAnimated3D objects and different meshes.

I just have a little refactoring to do tonight of moving the setAnimation() into the GPUAnimated3D (which excludes the code for applying the SkeletalPose to the vertices on the software-side), so that it only updates/creates the bone matrices.

It will be easier to see when I post the code.  Hopefully I will get some time tonight.

NOTE: I only checked the regular render pipeline.  It may not animate in wireframe rendering.

[EgonOlsen]

I'm not sure if I got this correctly, but it sounds to me as if you were actually looking for this: http://www.jpct.net/jpct-ae/doc/com/threed/jpct/IRenderHook.html#setCurrentShader(com.threed.jpct.GLSLShader)

[Redman]

Thanks, Egon!  I didn't know about this class.  I can refactor my code to set the renderhook to tap into the render pipeline.

I didn't get a chance to go on my comp last night, so no updates.

[Redman]

Egon, I think the only JPCT namespaced variable that I won't be able to access without a helper class is VertexAttributes.name

Because objects can reuse mesh data, I want to validate if my new VertexAttributes already exist in the mesh.

When I get on tonight, I will refactor and give you a concrete response of what's namespace inaccessible.

[Redman]

GPUAnimated3D

Code Select Expand


package yournamespacegoeshere;

import yournamespacegoeshere.GPUAnimated3DShader;
import com.threed.jpct.GLSLShader;
import com.threed.jpct.IRenderHook;
import com.threed.jpct.Object3D;

import raft.jpct.bones.Animated3D;
import raft.jpct.bones.BonesNamespaceUtils;
import raft.jpct.bones.SkinClip;

/**
* Created by Dougie on 6/21/16.
*/
public class GPUAnimated3D extends Animated3D implements IRenderHook {
    public GPUAnimated3D(Animated3D object) {
        super(object);
        BonesNamespaceUtils.setSkinAttributes(object);
        setRenderHook(this);
    }

    public void animateSkin(float index, int sequence) {
        if(getSkinClipSequence() != null) {
            if(sequence == 0) {
                BonesNamespaceUtils.animate(this, index*getSkinClipSequence().getTime(), getSkeletonPose());
            } else {
                SkinClip clip = getSkinClipSequence().getClip(sequence - 1);
                clip.applyTo(index * clip.getTime(), getSkeletonPose());
            }
            getSkeletonPose().updateTransforms();
        }
    }
    public void animatePose(float index, int sequence, float weight) {
        BonesNamespaceUtils.animatePoseDontApply(this, index, sequence, weight);
    }

    @Override
    public void beforeRendering(int i) { ; }
    @Override
    public void afterRendering(int i) { ; }
    @Override
    public void setCurrentObject3D(Object3D object3D) { ; }
    @Override
    public void setCurrentShader(GLSLShader glslShader) {
        if(glslShader!=null && glslShader instanceof GPUAnimated3DShader) {
            ((GPUAnimated3DShader)glslShader).updateBeforeRenderingObject(this);
        }
    }
    @Override
    public void setTransparency(float v) { ; }
    @Override
    public void onDispose() { ; }
    @Override
    public boolean repeatRendering() { return false; }
}


GPUAnimated3DShader

Code Select Expand


package yournamespacegoeshere;

import android.opengl.GLES20;

import com.threed.jpct.GLSLShader;
import com.threed.jpct.Matrix;

import raft.jpct.bones.Animated3D;
import raft.jpct.bones.BonesNamespaceUtils;

/**
* Created by Dougie on 6/21/16.
*/
public class GPUAnimated3DShader extends GLSLShader {
    int skinWeightsHandle = -1;
    int jointIndicesHandle = -1;

    public GPUAnimated3DShader(String vertexShaderSource, String fragmentShaderSource) {
        super(vertexShaderSource, fragmentShaderSource);

        skinWeightsHandle = GLES20.glGetAttribLocation(getProgram(), BonesNamespaceUtils.ATTR_SKIN_WEIGHTS);
        jointIndicesHandle = GLES20.glGetAttribLocation(getProgram(), BonesNamespaceUtils.ATTR_JOINT_INDICES);
    }

    public void updateBeforeRenderingObject(Animated3D pObject) {
        if(pObject!=null) {
            Matrix[] skelPose = BonesNamespaceUtils.getSkeletonPosePallete(pObject.getSkeletonPose());
            if(skelPose!=null && skelPose.length>0) {
                setUniform(BonesNamespaceUtils.UNIFORM_SKEL_POSE_SIZE, skelPose.length);
                setUniform(BonesNamespaceUtils.UNIFORM_SKEL_POSE, skelPose);
            } else {
                setUniform(BonesNamespaceUtils.UNIFORM_SKEL_POSE_SIZE, 0);
            }
        }
    }
}


JPCTNamespaceUtils

Code Select Expand


package com.threed.jpct;

/**
* Created by Dougie on 6/18/16.
*/
public class JPCTNamespaceUtils {
    public static boolean VertexAttributesNameIs(VertexAttributes pVertAttrs, String pName) {
        if(pVertAttrs!=null && pName!=null && pVertAttrs.name!=null && pVertAttrs.name.equals(pName))
            return true;
        return false;
    }
}


BonesNamespaceUtils

Code Select Expand


package raft.jpct.bones;

import com.threed.jpct.Matrix;
import com.threed.jpct.JPCTNamespaceUtils;
import com.threed.jpct.VertexAttributes;

/**
* Created by Dougie on 6/17/16.
*/
public class BonesNamespaceUtils {
    public static String ATTR_SKIN_WEIGHTS = "skinWeights";
    public static String ATTR_JOINT_INDICES = "jointIndices";
    public static String UNIFORM_SKEL_POSE = "skelPose";
    public static String UNIFORM_SKEL_POSE_SIZE = "skelPoseSize";

    public static void animate(Animated3D pAnimated3D, float pTime, SkeletonPose pPose) {
        if(pAnimated3D!=null && pAnimated3D.getSkinClipSequence()!=null) {
            pAnimated3D.getSkinClipSequence().animate(pTime, pPose);
        }
    }
    public static void animatePoseDontApply(Animated3D pAnimated3D, float index, int sequence, float weight) {
        if(pAnimated3D!=null) {
            pAnimated3D.animatePoseDontApply(index, sequence, weight);
        }
    }
    public static Matrix[] getSkeletonPosePallete(SkeletonPose pPose) {
        if(pPose!=null) {
            return pPose.palette;
        }
        return null;
    }
    public static void setSkinAttributes(Animated3D pAnimated3D) {
        int i, j, k, len;
        if(pAnimated3D != null && pAnimated3D.getMesh()!=null) {
            boolean hasWeights = false, hasJointIndices = false;
            VertexAttributes[] vertAttrs = pAnimated3D.getMesh().getVertexAttributes();
            if(vertAttrs!=null) {
                //Loop backwards as they should be the last added ones if shared mesh
                for(i=vertAttrs.length-1;i>=0;--i) {
                    if(JPCTNamespaceUtils.VertexAttributesNameIs(vertAttrs[i], ATTR_SKIN_WEIGHTS)) {
                        hasWeights = true;
                    }
                    if(JPCTNamespaceUtils.VertexAttributesNameIs(vertAttrs[i], ATTR_JOINT_INDICES)) {
                        hasJointIndices = true;
                    }
                    if(hasWeights && hasJointIndices) {
                        break;
                    }
                }
            }

            if(!hasWeights) {
                float[][] weights = pAnimated3D.skin.weights;
                if(weights != null && weights.length > 0) {
                    len = weights[0].length;
                    float[] weightsArr = new float[pAnimated3D.getMesh().getUniqueVertexCount() * len];
                    for (j = 0; j < weights.length; ++j) {
                        for (k = 0; k < len; ++k) {
                            weightsArr[j * len + k] = weights[j][k];
                        }
                    }
                    if (weights.length < pAnimated3D.getMesh().getUniqueVertexCount()) {
                        for (j = weights.length; j < pAnimated3D.getMesh().getUniqueVertexCount(); ++j) {
                            for (k = 0; k < len; ++k) {
                                weightsArr[j * len + k] = 0f;
                            }
                        }
                    }
                    pAnimated3D.getMesh().addVertexAttributes(new VertexAttributes(ATTR_SKIN_WEIGHTS, weightsArr, len));
                }
            }
            if(!hasJointIndices) {
                short[][] jointIndices = pAnimated3D.skin.jointIndices;
                if (jointIndices != null && jointIndices.length > 0) {
                    len = jointIndices[0].length;
                    float[] jointIndicesArr = new float[pAnimated3D.getMesh().getUniqueVertexCount() * len];
                    for (j = 0; j < jointIndices.length; ++j) {
                        for (k = 0; k < len; ++k) {
                            jointIndicesArr[j * len + k] = jointIndices[j][k];
                        }
                    }
                    if (jointIndices.length < pAnimated3D.getMesh().getUniqueVertexCount()) {
                        for (j = jointIndices.length; j < pAnimated3D.getMesh().getUniqueVertexCount(); ++j) {
                            for (k = 0; k < len; ++k) {
                                jointIndicesArr[j * len + k] = 0f;
                            }
                        }
                    }
                    pAnimated3D.getMesh().addVertexAttributes(new VertexAttributes(ATTR_JOINT_INDICES, jointIndicesArr, len));
                }
            }
        }
    }
}


Vertex Shader Snippet

Code Select Expand


...
uniform mat4 modelViewProjectionMatrix;
...
uniform mat4 skelPose[50];
uniform int skelPoseSize;

attribute vec4 position;
attribute vec3 normal;
...
attribute vec4 skinWeights;
attribute vec4 jointIndices;

void main(void)
{
vec4 newPosition = vec4(0.0,0.0,0.0, position[3]);
vec3 newNormal = vec3(0.0,0.0,0.0);
if(skelPoseSize>0) {
float weight;
for(int j=0; j<4; ++j) {
weight = skinWeights[j];
//newPosition[j] += weight;
if(weight != 0.0) {
mat4 boneMat = skelPose[int(floor(jointIndices[j]+0.5))];
newPosition.xyz += vec3(position.x*boneMat[0][0] + position.y*boneMat[1][0] + position.z*boneMat[2][0] + boneMat[3][0],
position.x*boneMat[0][1] + position.y*boneMat[1][1] + position.z*boneMat[2][1] + boneMat[3][1],
position.x*boneMat[0][2] + position.y*boneMat[1][2] + position.z*boneMat[2][2] + boneMat[3][2])*weight;
newNormal += vec3(normal.x*boneMat[0][0] + normal.y*boneMat[1][0] + normal.z*boneMat[2][0],
  normal.x*boneMat[0][1] + normal.y*boneMat[1][1] + normal.z*boneMat[2][1],
  normal.x*boneMat[0][2] + normal.y*boneMat[1][2] + normal.z*boneMat[2][2])*weight;
}
}
newPosition.yz *= -1.0;
newNormal.yz *= -1.0;
} else {
newPosition = position;
newNormal = normal;
}

...

gl_Position = modelViewProjectionMatrix * newPosition;
}


Usage:
Use the GPUAnimated3D constructor passing in the Animated3D you want it to be.  It will clone the Animated3D reusing the Mesh.
Afterwards, call GPUAnimated3D.setShader() passing in a GPUAnimated3DShader using the integrated version of the vertex shader above.
Call your Bones animateSkin() methods as you normally would and watch it animate using the GPU instead.

[EgonOlsen]

That's pretty cool. Maybe you want to add that to the wiki as well? If so, please let me know and I'll make an account for you.

Edit: So I guess adding a getter method for VertexAttributes' name property would be a good idea as well?

[Redman]

Sure.  I'll gladly add it to the wiki.

I think it would be useful to be able to read the name of VertexAttributes, but that's up to you.  A helper utility works just as well.

[EgonOlsen]

Here a beta with the getter methods added:

Desktop: http://jpct.de/download/beta/jpct.jar

Android: http://jpct.de/download/beta/jpct_ae.jar