Passos Petits cap a una nova manera de viure

Un lloc sobre permacultura, tecnologia responsable i sostenibilitat

Augmented Reality 3D with JPCT-AE and ARToolkit

4 comentaris

The most used AR frameworks based on markers (ARToolkit and Vuforia) have SDKs for android. And in both cases, they end up in a method that draws every frame, and where you get a matrix for the camera projection and a transformation matrix to place things on each detected marker. Let’s see what does it mean…

In this case we’ll use ARToolkit, but the same may apply to the Vuforia case. The AR framework scans the camera input in each frame, and detects the markers that are configured to be detected. Once it detects a marker, it analyzes its situation relative to the screen (i.e. the position and the orientation), and gives this information back to the program. This is done each frame, as the camera or the marker may move.

Over the camera preview, there is a GLSurfaceView where the program is supposed to draw the 3D augmentations. So there is a Renderer that on each frame can have the information on where the marker is situated in the 3D space. There you can use OpenGL to draw whatever you want.

On the other hand, JPCT-AE is a library to render 3D models and animate them which works on top of OpenGL, making it easier for developers to handle with OpenGL. It loads .obj and .3ds files out-of-the-box, and has an OOP approach that OpenGL has not. So why not use it in an AR application.

Setting up artoolkit

How to set up artoolkit is defined in the Android ARToolkit section, take a look at it. This blog entry is based on importing the ARSimple example and work on it. Simply import the project into Android studio, and run it on your phone. If your camera sees the Hiro Pattern, then a cube will show up on top of it.

Setting up JPCT-AE

This is really easy. Download zip from the downloads section, unzip it and get the library (jpct-ae.jar). Finally add it to the libs folder on the android studio project. Maybe you need to right click on it and select “Add as Library…”. You have the library on your classpath!

Start to code!

All the code to add is on the SimpleRenderer, where the 3D frames are rendered. First of all, we must set up the JPCT world. We do so on a custom method, called in the configureARscene method. There we set up the world, its lights, and our models:

 @Override
    public boolean configureARScene() {
        configureJPCTWorld();
        markerId1=ARToolKit.getInstance().
                addMarker("single;Data/patt.kanji;80");
        markerId2=ARToolKit.getInstance().
                addMarker("single;Data/patt.hiro;80");

        Log.i("RENDERER", "Markers loaded: " + markerId1 + ", " + markerId2);

        if(markerId1<0)return false;
    }

    public void configureJPCTWorld(){
        world=new World();
        world.setAmbientLight(20, 20, 20);

        Light sun=new Light(world);
        sun.setIntensity(250, 250, 250);

        camera=world.getCamera();

        Texture cubeText=
                new Texture(
                        BitmapHelper.rescale(
                                BitmapHelper.convert(
                                       context.
                                               getResources().
                                               getDrawable(R.drawable.texture)
                                ),128,128
                        )
                );

        TextureManager.getInstance().addTexture("cubeTexture",cubeText);

        Texture tankTexture=
                new Texture(
                        BitmapHelper.rescale(
                                BitmapHelper.convert(
                                        context.
                                                getResources().
                                                getDrawable(R.drawable.tanktexture)
                                ),128,128
                        )
                );
        TextureManager.getInstance().addTexture("tankTexture",tankTexture);

        cube = Primitives.getCube(40);
        cube.calcTextureWrapSpherical();
        cube.setTexture("cubeTexture");
        cube.strip();
        cube.build();

        try {
            AssetManager assetManager =
                    context.getAssets();
            Object3D[] objects =
                    Loader.load3DS(assetManager.open("Tank.3DS"), 0.1f);

            tank= Object3D.mergeAll(objects);
            tank.setTexture("tankTexture");
            tank.strip();
            tank.build();

        }catch(IOException e){
            Log.e("RENDERER-JPCT","Error loading tank",e);
        }

        //MemoryHelper.compact();

    }

    @Override
    public void onSurfaceChanged(GL10 unused, int w, int h) {
        super.onSurfaceChanged(unused, w, h);
        if(frameBuffer!=null){
            frameBuffer.dispose();
        }
        frameBuffer=new FrameBuffer(unused,w,h);
    }

As you may see in this example we load a 3DS model, additionally to a Cube. Refer to the JPCT-AE wiki for examples and further explanation on setting up the world.

Then, the last thing is to draw every frame. This is done on the draw method. One thing to note is that there the AR framework will give you 2 things:

  • Information of the camera (field of view, orientation…)
  • Information of each marker detected (position and orientation)

Both things are given as a matrix. Why? because on OpenGL and in a 3D world, matrices are the way to define the transformations to change points of view, and move objects and cameras. It is strongly recommended to have a look at the opengl tutorial for matrix. As you will see here, what is given by the AR framework is, on one hand the Projection matrix (which we’ll apply to the camera object) and on the other hand the transformation matrix related to each frame.

Thus, if we want to display an AR object, first we have to apply the projection matrix to the camera object, and then, for each object to display on a marker, make first the model transformations (rotate and move it to place it as we want), and then translate it to the marker (which is done by multiplying the matrix that has the model transformations with the matrix that has the marker transformations. WARNING: the order is important!):

Matrix projMatrix=new Matrix(); //the matrix having the camera projection
    Matrix markerMatrix=new Matrix(); //the matrix having the transformation to place the tank in the marker
    Matrix modelMatrix=new Matrix(); //the matrix having the model (tank) rotation and translation

    @Override
    public void draw(GL10 gl) {

        world.removeAllObjects();
        markerVisible=false;

        float[] projectionCamera=ARToolKit.getInstance().getProjectionMatrix();


        projMatrix.setIdentity();

        projMatrix.setDump(projectionCamera);
        projMatrix.transformToGL();
        SimpleVector translation=projMatrix.getTranslation();
        SimpleVector dir=projMatrix.getZAxis();
        SimpleVector up=projMatrix.getYAxis();
        camera.setPosition(translation);
        camera.setOrientation(dir, up);

        if(ARToolKit.getInstance().queryMarkerVisible(markerId1)){
            //Matrix markerMatrix=new Matrix();
            markerMatrix.setIdentity();
            //Log.e("RENDERER", "Marker1 detected");

            float[] marker1Transformation=
                    ARToolKit.
                            getInstance().
                            queryMarkerTransformation(markerId1);

            markerMatrix.setDump(marker1Transformation);
            markerMatrix.transformToGL();

            //cube.clearTranslation();
            //cube.clearRotation();

            cube.setRotationMatrix(markerMatrix);
            cube.setTranslationMatrix(markerMatrix);

            world.addObject(cube);

        }
        if(ARToolKit.getInstance().queryMarkerVisible(markerId2)){
            markerVisible=true;

            //Reset the matrix
            markerMatrix.setIdentity();
            modelMatrix.setIdentity();
            tank.clearTranslation();
            tank.clearRotation();

            //do the model transformations (rotate first and translate then)
            //the angle must be negative, or x-y movement swapped, as JPCT has an inverted z-axis

            modelMatrix.rotateZ((float) Math.toRadians(-angle+180));
            modelMatrix.translate(movementX, movementY, 0);

            //the position and rotation of the marker gives us the transformation matrix
            float[] markerTransformation=
                    ARToolKit.
                            getInstance().
                            queryMarkerTransformation(markerId2);

            markerMatrix.setDump(markerTransformation);
            markerMatrix.transformToGL();

            //now multiply trasnformationMat * modelMat

            modelMatrix.matMul(markerMatrix);

            //apply the resulting matrix to the tank.

            tank.setRotationMatrix(modelMatrix);
            tank.setTranslationMatrix(modelMatrix);

            world.addObject(tank);

        }

        frameBuffer.clear();
        world.renderScene(frameBuffer);
        world.draw(frameBuffer);
        frameBuffer.display();

        //if(markerVisible)updateMovement(4,2);

    }

in this example, for the cube, there is no model transformation, so the marker transformation is applied directly. However, for the second marker, there is a rotation and a translation for the model, which makes it necessary to multiply the resulting matrix afterwards with the matrix defined by the marker position.

The last thing to note is that the matrices are not created on each frame, but resseted to their initial state. This is strongly reccomended, as new objects created on each frame mean memory allocation, and are more expensive in terms of cpu and, which is worse, may trigger garbage collections.

Anuncis

4 thoughts on “Augmented Reality 3D with JPCT-AE and ARToolkit

  1. This is not working

    • Hello Javier!
      Wha’ts exaclty the problem? It has been a some time since I wrote this, but it should still work. Have you some compilation issues? Or when running it you get errors? If you give me some feedback, I may try to help you.

  2. Null pointer exepcion when obj 3ds load and texture load

    • Hello Angelo,

      Have you checked that you have the .3ds in your project assets/ folder? And that in drawables folder you have the textures? Have you checked the file names (note the extension in uppercase must match the actual file name!). If so, please post the stacktrace, maybe I can find out something.

Deixa un comentari

Fill in your details below or click an icon to log in:

WordPress.com Logo

Esteu comentant fent servir el compte WordPress.com. Log Out /  Canvia )

Google+ photo

Esteu comentant fent servir el compte Google+. Log Out /  Canvia )

Twitter picture

Esteu comentant fent servir el compte Twitter. Log Out /  Canvia )

Facebook photo

Esteu comentant fent servir el compte Facebook. Log Out /  Canvia )

S'està connectant a %s