Detection of the instantaneous position of the object shown to the camera in the browser

[harundogan01]

Hi dear friends,

I'm trying to detect ID cards using Tensorflow.js' Coco-Ssd model on a web page developed with Vue.js. I put an svg image in the middle of the page as an ID card template. I want the ID card shown manually to the camera to be placed exactly on the svg image, and when the card is hovered over the svg image, I want the user to click on the button to take a photo of the ID card in base64 format and then scan the back of the card. What I want to ask you here is, how can I get the correct coordinate information of the card in the user's hand?

https://i.stack.imgur.com/VTOA3.png --> Screenshot for this subject

I'm trying to reach the closest estimate using bbox information, but these coordinates do not include the current location information of the card on the page. If I can access the correct coordinates, I will try to reach the correct result by comparing it with the coordinates of the svg image. Anyone have experience with this issue before?

[harundogan01]

• Edit: The coordinate data coming from Tensorflow comes from the ObjectDetection.prototype.buildDetectedObjects function in the coco-ssd.js file in the @tensorflow-models npm package.
• Note: The ID card is sometimes guessed as a remote, sometimes as a cell phone, and rarely as a book. It would be great if a prediction class could be defined in the form of cards.
  
  

[harundogan01]

• Note 2: When I want to take a base64 type photo of the card I show to the camera, the image I get is very small. The reason for this is that the width and height values coming from the bbox object I specified do not cover the full size of the card. I would be very happy if work was done on this issue. I need to get a base64 output based on the exact size of the object shown to the camera.

-- Card Size is bigger than base64 output. (339x218)

-- Base64 Output (237x167 or lower than)

[gaikwadrahul8]

Hi, @harundogan01

Thank you for bringing this issue to our attention and if possible could you please help us with your Github repo so i'll try to replicate the same issue from our end if you've trained Tensorflow.js Coco-SSD model with ID card dataset ?

I see Tensorflow.js Coco-SSD model has trained on total 90 classes which are mentioned in this file https://github.com/tensorflow/tfjs-models/blob/master/coco-ssd/src/classes.ts and in that there is no ID card class available so if you're using Tensorflow.js Coco-SSD model without training ID cards dataset then it won't detect ID card with the help of Tensorflow.js Coco-SSD model.

If you've done Tensorflow.js Coco-SSD model fine tuning with your own ID card dataset and trained model and it's detecting ID card wrongly then please help us with your Github repo to replicate the same behavior from our end.

Thank you for your understanding and patience.

[harundogan01]

Hi @gaikwadrahul8,

There is no data on this topic in my own Github repository. I am developing the project locally. Yes, as you stated, there is no special class for ID cards. For this reason, I look at whether it is remote, cell phone or book and gets at least 0.7 points. On the backend side, there is already a special detection method for the ID card. I can say that I do it symbolically on the frontend side. However, it would still be nice if such an improvement was made. If it is not done, it is not a big deal. The other issues I mentioned, canvas size and coordinate information, are more important to me.

Thanks

[harundogan01]

Is there any progress on this issue, guys? @mattsoulanille @gaikwadrahul8