This project uses the Mediapipe library to detect hand gestures in real-time video feed, and Keras with LSTM to train a model to recognize these gestures. The project involves two main parts: data collection and model training.
The collect_data.py script can be used to collect training data. It reads video frames from a camera or a saved video, and uses Mediapipe to detect and extract hand landmarks in each frame. The landmarks are saved to disk as NumPy arrays, which can be used to train the model. The data is organized into directories based on the type of gesture and the sequence number.
The train_model.py script loads the saved NumPy arrays and trains a Keras model with LSTM layers to recognize the gestures. The model is saved to disk as a JSON file and a weights file, which can be loaded and used for prediction.
To collect data, run collect_data.py with appropriate command line arguments. Run the data.py file. To train the model, run train_model.py. The script will automatically load all the saved data and train the model. The training progress is logged to TensorBoard, and the final model is saved to model.json and model.h5 files.
To use the trained model for prediction, load the saved model and weights files using Keras, and use the predict() method on a new set of hand landmarks extracted from a video feed or an image..
Python 3.6+ OpenCV Mediapipe Keras NumPy TensorBoard