Apps OpenGL and Camera Preview - blending together gets

[Tobias Reich]

Hello everyone,

I hope you may help me with this problem.
I'm working on a new Galaxy S3 and found a problem that didn't occur on other phones like my old S1 or the Galaxy S2 which has almost the same GPU as the S3 I think.
My problem is hard to describe. When drawing my OpenGL models on top of my camera image like that:

    mGLView = new GLSurfaceView(this);
    mGLView.setOnClickListener(InputManager.getInstance());
    GameRenderer renderer = new GameRenderer(kamera, this);
    InputManager.getInstance().currentActivity = renderer;
                
    mGLView.setEGLContextClientVersion(2);
    mGLView.setEGLConfigChooser(8, 8, 8, 8, 16, 0);
    
    setPreserveEGLContextOnPause
    mGLView.getHolder().setFormat(PixelFormat.TRANSLUCENT);
    mGLView.setZOrderOnTop(true);
        
    mGLView.setRenderer(renderer);
    setContentView(new CameraView(this), new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
    addContentView(mGLView, new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));

So the GLRenderer should be transparent. It actually is. It works fine for all the objects that are opaque as well as the background (where I see the image of the camera)

My problem is, when I make the pixels semi-transparent in the fragment Shader I get a weird result. The brightest pixels seem overexpose when "blending together". And it looks like they got clamped after so the value that brighter than white - loosing probably the first bit - got dark again.
I guess it is an "overflow" for the values but how can I change this? By the way - perhaps it helps - this effect gets stronger the lower the alpha value of the fragment (the gl rendered one) is. So somehow it looks like something is compensating/normalizing the "semi transparent" values.

I use a simple call like:

gl_FragColor = vec4(clamp(textureColor.rgb * lightWeighting.xyz, 0.0, 1.0), 0.5);

So the actual GL-Fragment is clamped and I'm pretty sure it comes from the camera preview itself.
Here is an image that describes my problem:

I hope someone here has seen that problem before. Is there any way of "clamping" both values together since in the Shader I can only clamp the actual rendered fragment somehow. :-(

Thank you for your help,

Tobias

 

[Tobias Reich]

Okay, looks like nobody knows that problem. Maybe there is someone with a Galaxy S3 who might want to help me.
I made a very small test program to show this.
You may download the code here:

https://dl.dropbox.com/u/13527005/CameraGLTest.zip

or if you only want to test it you will find the .apk here:

https://dl.dropbox.com/u/13527005/CameraGLTest.apk

I would be very, very glad for any information about this. Even if you tell me it works perfectly fine on your S3. After all than I would know I have a problem with my phone!
Thank you very much for your help!
Tobias

 

[jonbonazza]

Sorry, didn't see this thread. The answer might not be what you were hoping, but in short, you can't use a translucent pixel format and also create translucency in your fragment shader at the same time.

May I ask why you can't just use translucency in your fragment shader and not change the pixelformat?

 

[Tobias Reich]

Sorry, didn't see this thread. The answer might not be what you were hoping, but in short, you can't use a translucent pixel format and also create translucency in your fragment shader at the same time.

May I ask why you can't just use translucency in your fragment shader and not change the pixelformat?

Hello,
Thank you for your quick reply.
Well, the idea is simple. If I dont set the Format
mGLView.getHolder().setFormat(PixelFormat.TRANSLUCENT);
The OpenGL Background will not be transparent. So even if I clear with
GLES20.glClearColor(0,0,0,0);
I get only a black background instead of the camera image.
On the other hand I want the (at least some) objects in OpenGL transparent aswell.
So there I rasterize the fragment with a certain alpha value.

And by the way, as mentioned earlier, it worked on the S1 and the S2. So was that just a bug then that made it work perfectly??

Thank you anyway for your idea! Maybe we can solve it somehow!

 

[jonbonazza]

Why do you want a transparent background? The background should always be in the back, so even if you can "see through it", you still wouldn't actually see anything. I seem to be confused at what you are trying to accomplish. Could you explain in further detail?

EDIT: Ah, I understand now I think. It seems you have an activity behind your EGL activity that you want to show through. If that's the case, I am really not sure what to tell you. I'd have to think about it some more...

 

[jonbonazza]

What does it look like if you use an opaque background and keep the translucent fragment shader?

Also, what are your texture params and blending func set to?

 

[Tobias Reich]

What does it look like if you use an opaque background and keep the translucent fragment shader?

Also, what are your texture params and blending func set to?

Well, good question. I thought I made a mistake with the openGL Shader stuff. But the next thing I tried was a Gl1.0 Testproject. It happens there as well.
In this example there are no textures at all. Just glColor stuff.
The idea with the opaque background doesn't work because I want to see the camera preview in the background. So I need a TRANSLUCENT GL Layer and - for Augmented Reality it looks much better - transparent objects (OpenGL) in the foreground.

Weird enough, if I make a screenshot by using ddms in the android/tools, everything looks fine. So what ever that means.
Anyway, I made a screenshot for you - compared to the S1:

Hope this helps!
Thank you very much for your assistance!
Tobias

 

[Tobias Reich]

Alright, I got an answer for that.
The problem was the premultiplication for the alpha values.
So in the easiest way it is enough to write:

vec3 color = clamp(textureColor.rgb * lightWeighting.xyz, 0.0, 1.0);
color *= 0.5; // premultiply by alpha
gl_FragColor = vec4(color, 0.5);

Even though I don't see why it worked perfectly fine on older systems and it I'm still not sure how to manage this on Gl1.0 where I can't write my own shader code.
So hope this helps to anyone who has the same problem!
Thanks,
Tobias

 

[jonbonazza]

Interesting... So it was working correctly on the S1 but on later hardware, such as the S3, it was oversaturated... That is definitely odd... Only answer for that (though it's not much of an answer, I admit) is that it is hardware or driver (most likely driver) related. The GPUs in the S1 and S3 are very different, so their drivers are also very likely different.

I am glad you figured it out and I will definitely keep this in mind in case I ever run across this issue. Thanks!

 

[ZeDuS]

Alright, I got an answer for that.
The problem was the premultiplication for the alpha values.
So in the easiest way it is enough to write:

vec3 color = clamp(textureColor.rgb * lightWeighting.xyz, 0.0, 1.0);
color *= 0.5; // premultiply by alpha
gl_FragColor = vec4(color, 0.5);

Even though I don't see why it worked perfectly fine on older systems and it I'm still not sure how to manage this on Gl1.0 where I can't write my own shader code.
So hope this helps to anyone who has the same problem!
Thanks,
Tobias

After encountering the same issue, I don't feel that what you have is a real solution.

All you did was reduce the cases of "overflow" by weakening the colors that you try to blend into the camera preview.

If you'd had a case, for example, that the camera preview is (1.0,1.0,1.0) at some pixel, and you want to blending something on top of it, you'd still encounter overflow.

To sum it up, you are just outputting smaller color values from your fragment shader, and thus reducing the cases of overflow, avoiding the problem.

I wonder what is the real solution for this...