Sparkles on Fingertips

Sparkles on Fingertips

What is this project about?

This project is a prototype of a 2D platform video game. In the game, I used OpenCV, Mediapipe and ML.NET to create the function of hand/finger recognition and symbol recognition.

Game Idea

The idea behind this game is to let the player really feel that he/she is the one drawing and casting the spells that the avatar is drawing and casting. Therefore, the player was allowed the ability of casting spells by making gestures with his/her hand instead of the mouse.

Game Story

The idea of the protagonist came from Douxie, who is a character in a TV show called Tales of Arcadia. The protagonist of the game is also a mage who has a cool device on her wrist that can contain spells she draws. When needed, she can swipe the interface of the device to choose a spell and then cast it.

Gameplay

In the game, using their index finger to draw in front of the webcam, the player can draw spells, which will be stored and then used to attack enemies later. (See the pic) Then, the player can choose spells by swiping both index and middle fingers in the air. (See the pic) Finally, the player can cast the spell and attack the enemies by holding and then releasing their hand. (See the pic)

Programming

In the project, we first have to be able to detect the player's hands and fingers in Unity. It was fulfilled by using Google's mediapipe, which is a very useful but not yet completed API made with OpenCV that can be used for visual recognition. However, the problem is that mediapipe doesn't support C# and Unity, so I had to embedded a whole Python environment into my Unity project in order to use mediapipe in my project. After that, ML.MET and the training data are used to classify the symbols that the player draws.

Critique

Overall, the outcome of this project in satisfying. I've achieved the mechanic in my mind, but I have to admit that the function of the hand recognition is not smooth enough and the prototype can be polished a little more. Except that, I think the notion itself is great and the building process of the prototype is rewarding.

P.S. Here is a web page of  the programming details.

***The game does not function if the directory it is placed in contains special characters, such as Chinese.***

***You will need a webcam in order to play this game***

***Runs under Window 10 x64 System***