Bionic Vision VR Simulation

For my computer science masters research project, I studied the effectiveness of combining different deep-learning based computer vision techniques to highlight more meaningful information when using retinal prostheses – devices used to treat degenerative eye diseases. These devices rely on an external camera to capture the visual scene, which is then translated frame-by-frame into an electrical stimulation pattern that is sent to the implant in the eye. To test the effectiveness of the combination of different simulated prosthetic vision (SPV) modes such as room layout estimation and relative depth extraction, we presented the modes individually (DepthOnly, LayoutOnly), together (DepthAndLayout), and by giving the user the ability to flexibly switch between them (DepthOrLayout). We used SPV in a virtual reality environment to test the effectiveness of the different modes by tasking users with avoiding obstacles and recognizing objects. This study is an important first step towards a retinal prosthesis that uses computer vision to improve a user’s scene understanding. Relevant Technologies: Unity, Python, C#, OpenGL, HTC Vive, SteamVR, Github

• Read the Report
• View the Presentation

Interactive Agent in VR

The interative agent project was a research project where a team and I developed a conversable AI in virtual reality using GPT-3. We created multiple characters with unique personalities – happy, sad, angry, confused, etc. – that were able to each hold conversations that were representative of their respective personalities. The final project was a huge success that allowed for voice conversation and responses from the agent in VR. Built multiple interactive agents to converse with using NLP, each having a unique personality & realistic immersive environment Relevant Technologies: Unity, C#, Python, OpenAI’s GPT-3, NLP, Oculus Quest 2, Oculus VoiceSDK

• Demo the project
• Read the Report

Volleyball Rally Prediction

In this project, a teammate and I created and optimized several Graph Nerual Network (GNN) models – GAT, GNN, GCN, etc – to test the effectiveness of predicting volleyball rally outcomes and set locations. In order to do this, we first collected rally and game data from the UCSB men's volleyball team, and preprocessed it to be best suited to create graphs to training our models. The final outcome of the project was extrememly successful, with rally outcomes being predicted successfully up to 80% of the time. Relevant Technologies: Python, PyTorch, Graph Neural Networks, Machine Learning, Data Colletion & Processing

• View the Presentation

Social Sensor

A video conferencing web application for people with autism spectrum disorder (ASD) that detects real-time sentiment of participants using machine learning. This project was created by a team of five people (including myself) over the course of 6 months, with multiple user studies, using the agile software development principles. The final project was entered into our UCSB Capstone competition and placed fourth overall. Relevant Technologies: Python, Node.js, React, Twilio, Computer Vision, Convolutional Neural Networks, Sentiment Analysis, Machine Learning, Full Stack Development

• GitHub

Image Classification

A personal project where I trained and evaluated Convolutional Neural Networks (CNNs) for image classification on the CIFAR-10 dataset. The goal of the project was to find ways to improve upon baseline models using ML techniques such as regularization, dropout, batching, etc. Overall, I was able to acheive a successful classification rate of 89%. Relevant Technologies: Python, PyTorch, Computer Vision, Convolutional Neural Networks, Machine Learning, Data Colletion & Processing

• Read the Report
• GitHub

Sanskrit to English Machine Translation

This research project was an attempt at training a transformer model to be able to translate between Sanskrit and English. Previously, this had been attempted, but because of the lack of datapoints and difference in grammatric syntax, it is a very difficult problem to solve. Nonetheless, a teammate and I attempted to use a new, vast dataset and a transformer model based approach to solve the problem. Relevant Technologies: Python, PyTorch, Transformer Model, Natural Language Processing, Machine Learning, Data Colletion & Processing

• GitHub

NavSB

NavSB is a tool created by a small group to help lost students and visiting families to navigate their way around UCSB. Often times, it is frustrating to find your way around a new place, especially on a dense, college campus. Using NavSB, people are able to easily navigate around campus and the Isla Vista community and visualize the best route to take. The iOS app provides users an interactive map that allows the user to pick a destination, choose either a 2D or augmented reality view, and seamlessly shows directions to the destination through guiding lines.

• Demo the project

Billiards Game

A fully functional billiards web app simulation with accurate momentum physics forumlas and sleek UI.

• Shoot Some Pool

Face Filter

The face filter is a Java application that recognizes the users face using the OpenCV library and masks it using a dog filter. The project is coded in Java so the user interface is easier to manage and implement. I had to learn about the many different aspects of the OpenCV library and how to integrate it into my program without getting and errors. While making this project, I had to be diligent and responsible with the personal deadlines I set for myself to make sure the project got completed.