Hardware skinning needed

[kkl]

I can't use SkinClipSequence animate() method as it's a private method. Could you compile a new Bones.jar with the method declared as PUBLIC? This's the best I can think of. Would love to hear if there is better suggestion

[raft]

you have the source code, just change it and recompile. the zip even contains the ant script to build the whole project

[kkl]

Ah~ I missed out the source code in the zip file. I always thought it's close sourced. Ok, i can start working on it now. Thanks alot.

[kkl]

Hi Raft,

I've spending few weeks to implemet hardware skinning, but I encountered few problems and I hope you could help me out on this

I've made these code, but the result seems incorrect:

vertex shader: skinPalette is hardcoded to match the bones file I have (im not sure if the number of matrix palettes is fixed).

Code Select Expand


uniform mat4 modelViewMatrix;
uniform mat4 modelViewProjectionMatrix;
uniform mat4 textureMatrix;

uniform vec4 additionalColor;
uniform vec4 ambientColor;

uniform float alpha;
uniform float shininess;
uniform bool useColors;

uniform float fogStart;
uniform float fogEnd;
uniform vec3 fogColor;

uniform int lightCount;

uniform mat4 skinPalette[7];

uniform vec3 lightPositions[8];
uniform vec3 diffuseColors[8];
uniform vec3 specularColors[8];
uniform float attenuation[8];

attribute vec4 position;
attribute vec3 normal;
attribute vec4 color;
attribute vec2 texture0;
attribute vec2 texture1;
attribute vec2 texture2;
attribute vec2 texture3;

attribute vec4 skinWeights;
attribute vec4 skinJointIndices;

varying vec2 texCoord[4];
varying vec4 vertexColor;
varying vec3 fogVertexColor;
varying float fogWeight;

const vec4 WHITE = vec4(1,1,1,1);

void main() {

texCoord[0] = texture0;
texCoord[1] = (textureMatrix * vec4(texture1, 0, 1)).xy;
texCoord[2] = texture2;
texCoord[3] = texture3;

vec4 myPosition = vec4(position);
vec3 myNormal = vec3(normal);

vec4 vertexTemp;
vec3 normalTemp;
mat4 mat;


for (int i = 0; i < 4; i++) {
mat = skinPalette[int(skinJointIndices[i])];
vertexTemp = mat * position;
vertexTemp *= skinWeights[i];
myPosition += vertexTemp;
normalTemp = normal * mat3(mat);
normalTemp *= skinWeights[i];
myNormal += normalTemp;
}

vec4 vertexPos = modelViewMatrix * myPosition;
vertexColor = ambientColor + additionalColor;

if (lightCount>0) {
// This is correct only if the modelview matrix is orthogonal. In jPCT-AE, it always is...unless you fiddle around with it.
vec3 normalEye   = normalize(modelViewMatrix * vec4(myNormal, 0.0)).xyz;

float angle = dot(normalEye, normalize(lightPositions[0] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[0] * angle + specularColors[0] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[0] - vertexPos.xyz)*attenuation[0])), 1);
}

// Freaky Adreno shader compiler can't handle loops without locking or creating garbage results....this is why the
// loop has been unrolled here. It's faster this way on PowerVR SGX540 too, even if PVRUniSCoEditor says otherwise...

if (lightCount>1) {
angle = dot(normalEye, normalize(lightPositions[1] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[1] * angle + specularColors[1] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[1] - vertexPos.xyz)*attenuation[1])), 1);
}

}
}


if (fogStart != -1.0) {
fogWeight = clamp((-vertexPos.z - fogStart) / (fogEnd - fogStart), 0.0, 1.0);
fogVertexColor = fogColor * fogWeight;
} else {
fogWeight = -1.0;
}

vertexColor=vec4(min(WHITE, vertexColor).xyz, alpha);

if (useColors) {
vertexColor *= color;
}

gl_Position = modelViewProjectionMatrix * myPosition;
}


This's the code to add vertex attributes to shader. linearizeArray is converting float[][] to float[] and linearizeToFloatArray is short[][] to float[]

Code Select Expand


float[] f1 = SkinHelper.linearizeArray(skin.weights);
float[] f2 = SkinHelper.linearizeToFloatArray(skin.jointIndices);
skinWeights = new VertexAttributes(
"skinWeights", f1, VertexAttributes.TYPE_FOUR_FLOATS);
skinJointIndices = new VertexAttributes(
"skinJointIndices", f2, VertexAttributes.TYPE_FOUR_FLOATS);
Mesh mesh = getMesh();
mesh.addVertexAttributes(skinWeights);
mesh.addVertexAttributes(skinJointIndices);


This's where I set the uniform for matrix palette (in array form, Egon created a new GLSLShader to pass matrix array).

Code Select Expand


object3D.setRenderHook(new RenderHookAdapter(){

@Override
public void setCurrentShader(GLSLShader shader) {
super.setCurrentShader(shader);
shader.setUniform("skinPalette", object3D.getSkeletonPose().getPalette());
}
});


I tried to debug glsl and most debugger I tried were not working. Hope you could help me out on this. May be I missed something.

[kkl]

I got it worked!! I fixed the previous code with these:

Code Select Expand


uniform mat4 modelViewMatrix;
uniform mat4 modelViewProjectionMatrix;
uniform mat4 textureMatrix;

uniform vec4 additionalColor;
uniform vec4 ambientColor;

uniform float alpha;
uniform float shininess;
uniform bool useColors;

uniform float fogStart;
uniform float fogEnd;
uniform vec3 fogColor;

uniform int lightCount;

uniform mat4 skinPalette[7];

uniform vec3 lightPositions[8];
uniform vec3 diffuseColors[8];
uniform vec3 specularColors[8];
uniform float attenuation[8];

attribute vec4 position;
attribute vec3 normal;
attribute vec4 color;
attribute vec2 texture0;
attribute vec2 texture1;
attribute vec2 texture2;
attribute vec2 texture3;

attribute vec4 skinWeights;
attribute vec4 skinJointIndices;

varying vec2 texCoord[4];
varying vec4 vertexColor;
varying vec3 fogVertexColor;
varying float fogWeight;

const vec4 WHITE = vec4(1,1,1,1);

void main() {

texCoord[0] = texture0;
texCoord[1] = (textureMatrix * vec4(texture1, 0, 1)).xy;
texCoord[2] = texture2;
texCoord[3] = texture3;

vec4 myPosition = vec4(0,0,0,0);
vec3 myNormal = vec3(0,0,0);

vec4 vertexTemp;
vec3 normalTemp;
mat4 mat;

for (int i = 0; i < 4; i++) {
mat = skinPalette[int(skinJointIndices[i])];
vertexTemp = mat * position;
vertexTemp *= skinWeights[i];
myPosition += vertexTemp;
normalTemp = mat3(mat) * normal;
normalTemp *= skinWeights[i];
myNormal += normalTemp;
}

vec4 vertexPos = modelViewMatrix * myPosition;
vertexColor = ambientColor + additionalColor;

if (lightCount>0) {
// This is correct only if the modelview matrix is orthogonal. In jPCT-AE, it always is...unless you fiddle around with it.
vec3 normalEye   = normalize(modelViewMatrix * vec4(myNormal, 0.0)).xyz;

float angle = dot(normalEye, normalize(lightPositions[0] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[0] * angle + specularColors[0] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[0] - vertexPos.xyz)*attenuation[0])), 1);
}

// Freaky Adreno shader compiler can't handle loops without locking or creating garbage results....this is why the
// loop has been unrolled here. It's faster this way on PowerVR SGX540 too, even if PVRUniSCoEditor says otherwise...

if (lightCount>1) {
angle = dot(normalEye, normalize(lightPositions[1] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[1] * angle + specularColors[1] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[1] - vertexPos.xyz)*attenuation[1])), 1);
}

}
}


if (fogStart != -1.0) {
fogWeight = clamp((-vertexPos.z - fogStart) / (fogEnd - fogStart), 0.0, 1.0);
fogVertexColor = fogColor * fogWeight;
} else {
fogWeight = -1.0;
}

vertexColor=vec4(min(WHITE, vertexColor).xyz, alpha);

if (useColors) {
vertexColor *= color;
}

gl_Position = modelViewProjectionMatrix * myPosition;
}


where myPosition and myNormal are initialized to zero.  But the normal seems to be abit off. I'm looking into it.

[raft]

cool, i'm glad you solved it

[EgonOlsen]

And how much faster is it now?

[kkl]

And how much faster is it now?

I tested it out by just doing a simple check on the time taken for the entire drawframe method. Test case is a model w/o texture with ~1968 vertices and 7 bones. Only 3 bones are moving. Test device is Samsung Galaxy S4 with octa core.

Here's the result:
Software: ~ 11-14 ms
Hardware skinning: ~ 1-4 ms

The test result seems to be quite promising. Anyway the rotation seems to be incorrect compared to the software version. I'll try check it out.

[raft]

The test result seems to be quite promising.

really they are

[kiffa]

Cool~ Would you like to share it when you done? This will give  others(include me  ) a lot of help.

[kkl]

Hi Kiffa,

It's almost done, but there's still some problem wif the initial rotation and I didn't have time to fix that yet. It would be great if you could fix it.

The attachment contains the required java files. You would need to download the Bones source code and replace the java files with the same names.

Example usage:

Code Select Expand


try {
group = BonesIO.loadGroup(context.getAssets().open(
"models/monkey.bones"));
} catch (Exception ex) {
ex.printStackTrace();
}

for (final Animated3D o : group) {

try {
final GLSLShader shader1 = new GLSLShader(
Loader.loadTextFile(context.getAssets().open("shaders/vertex_hardware_skinning.src")),
Loader.loadTextFile(context.getAssets().open("shaders/fragment.src")));
o.setShader(shader1);
o.setRenderHook(new RenderHookAdapter(){

@Override
public void beforeRendering(int arg0) {
shader1.setUniform("skinPalette", o.getSkeletonPose().getPalette());
}

});
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
o.setHardwareEnabled(true);
o.build();
o.discardMeshData();
}


Currently, I hard coded the vertex shader to support up to 10 bones only. If you wanna increase it, edit skinPalette[10] to bigger size array in vertex_hardware_skinning.src. The vertex shader also supports up to 2 light sources only, but you can go ahead and copy part of jpct original vertex shader and paste it to increase light sources.

[attachment deleted by admin]

[kkl]

Hi raft and egon,

Sorry for bringing this up. Previously I have implemented hardware skinning and was having initial rotation issue, in which the object appears different rotation than the one original with software skinning. The incorrect rotation started to cause issue when JPCT detects clipping boundary and object is not in the screen and it's automatically not showing the object, but in actual it should show as it's just having different rotation after processing in shader, which is after clipping process.

I was just wondering if there's any addition rotations applied before vertex is passed to shader. I'm afraid I might have missed some rotation matrices, or it could be something else that cause the issue. Thanks.

[EgonOlsen]

No, actually not. All vertex transformations happen inside the shader.

[kkl]

Not sure where went wrong.. Currently i cant find the root cause, is it possbile if we can have an option to disable clipping object process for temporary solution?

[kkl]

Strangely, I tried negating final position y and z in shader (which i think it's same with rotateX) and it works ok now. I wonder if I missed somewhere in software part.