Elde Ne Var? — Görüntü İşleme

I designed this project together with AI to move beyond knowing computer vision at a coursework level and actually be able to apply it, and I am still working on it. It consists of three applications sharing a common core: Hand Detection, which finds the hand in the camera feed and draws its 21 landmarks; Hand Mouse, which drives the Windows cursor with finger gestures (one finger to move, thumb touch to click, two fingers to scroll, three fingers to right-click); and my actual goal, What's in the hand?.

The pipeline of the third application works like this: every frame goes through the MediaPipe HandLandmarker, which extracts the hand skeleton in about 12 milliseconds. A square focus region 2.8 times the size of the hand is cropped around it and scaled to 320×320. That crop first goes to EfficientDet-Lite0, which answers the "where?" question by proposing candidate boxes. Then SigLIP2 steps in and answers "what?". The result passes through a stability window of roughly 0.7 seconds before it is drawn on screen and spoken out loud.

This division of labour was the project's defining decision. In the first version the detector also named the object, and whatever you held came out as a "toothbrush" or a "mobile phone"; when the hand fills most of the frame, the detector's classification head drifts towards the "things held in a hand" classes of the dataset it was trained on. Now the detector only says where the object is, and a separate model gives it a name.

The second defining decision was moving from a closed vocabulary to an open one. The previous model had a thousand fixed classes; objects like a pen, a charging cable or a card were not something a threshold tweak could fix — they were structurally unrecognisable because the vocabulary had no entry for them. Since SigLIP2 embeds images and text into the same space, the vocabulary became a 176-line list: adding a new object means writing one line, not training a model. I also added negative classes such as "empty hand", "finger", "arm" and "background"; when one of those wins, the result is silence, so the decision comes from the model itself rather than from a threshold. If confidence is low but not negligible, the label is written in grey as "maybe: …" and is not spoken.

On the performance side, a few details made the difference. For GPU acceleration I used ONNX Runtime with DirectML instead of CUDA; on the large SigLIP model inference dropped from 2175 milliseconds to 78. Switching the camera backend to Media Foundation took the stream from 20 FPS to 30. Since OpenCV's fonts cannot render Turkish characters, I draw the text with Pillow using a real, cached font. For speech I use the built-in Windows SAPI rather than an extra package; because spawning a new process each time cost half a second, a single process is kept open and the text is sent through standard input. Camera capture and recognition run on separate threads, so the video streams at 30 FPS while recognition runs at around 13 Hz.

The part I learned the most from was moving forward by measurement rather than by guesswork. I collected a labelled set of 184 frames with my own camera and wrote two separate scripts, one to measure accuracy and one to grid-search the thresholds. The measurement reports three groups separately, because the real question is not "what is the accuracy" but "what kind of mistakes are we making". Overall accuracy went from 32% to 65%; but in practice the most important line is this one: the number of wrong names spoken out loud dropped from 19 to 5, and most remaining errors are now silence rather than a wrong name.

Measurement also proved what did not work: checking whether the hand is full using a skin-colour mask, masking out the hand pixels (when a thin object is held in a fist, the object itself was being erased), preprocessing such as CLAHE and gamma, test-time augmentation… none of them helped. The most striking finding was this: with the same object and the same code, changing nothing but the shooting conditions took the phone's accuracy from 20% to 85% — while all the work on the model had gained 5% on that object. Holding the camera properly matters more than improving the model. The project is still under development; I will share the code on GitHub soon.