Research Projects
Knowledge Distillation for Transformer Based NLP Models (2021 - 2022)
This is a project I am working on under the guidance of Dr. Colin Raffel in his research group at UNC.
Project description: I am working on a deep learning knowledge distillation (KD) project which aims to establish a consistent methodology to map large pre-trained neural network models to smaller, more practical, models while preserving model accuracy both before and after fine tuning for downstream tasks. This project builds upon prior model distillation work but considers using pre-trained student models for KD and hopes to compile the "best practices" for KD. We are largely considering models like BERT (any many others related to or stemming from BERT).
Relevant Papers:
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
PEA-KD: PARAMETER-EFFICIENT AND ACCURATE KNOWLEDGE DISTILLATION ON BERT
WELL-READ STUDENTS LEARN BETTER: ON THE IMPORTANCE OF PRE-TRAINING COMPACT MODELS
TinyBERT: Distilling BERT for Natural Language Understanding
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Deep Learning Pose Prediction on Drone Imaging of Whales (2021)
This is a semester-long project I am working on under the guidance of Dr. David Johnston at the Marine Robotics and Remote Sensing Lab (MaRRS) which is housed at the Duke Marine Lab (DUML) facility on Pivers Island, Morehead City, NC.
Project description: I am creating and training a deep learning framework for the task of pose prediction on drone videos of whales taken around the globe. We are currently looking at Minke whales, Blue whales, and Humpback whales. The goal of this project is to provide behavioral scientists with a quantifiable mechanism to track and predict an animal's movements.
Results: Playlist
Manually labeled whale (right), neural network predicted whale (left); Blue whale:
FLOPs Aware Deep Learning (2021)
This is a project I worked on through the Department of Energy Summer Undergraduate Learning Internship Program. I worked with the Leadership Computing Facility's (ALCF) Data Science team at Argonne National Laboratories (ANL) with Dr. Kyle Felker and Dr. Taylor Childers.
FLOPS_Aware_Deep_Learning__SULI_Summer_Project.pdfComputer Organization Teaching Assistant (2021 Spring and Summer)
I served as a Teaching Assistant for Computer Organization (COMP 311) under Dr. Henry Fuchs at UNC Chapel Hill. While this is not a research project, this TAing experience got me really interested in computer architecture, and it is a field I would love to study further/conduct research in some day.
Responsibilities: Create and grade homeworks, labs, quizzes, and tests. Host office hours to assist students with conceptual and assignment related questions.
FlyBy CNN (2020-2021)
I have been working on this project under the guidance of Dr. Juan Prieto and Dr. Martin Styner at the UNC Neuro Image Research Analysis Laboratories (NIRAL).
"Fly by CNN is an approach to capture 2D views from 3D objects and use the generated images to train deep learning algorithms or make inference once you have a trained model."
Project description: I am working to establish Fly by CNN's role as useful pre-processing tool to generate image sequences of brain MRIs for deep learning purposes. This is an important project because current state-of-the-art deep learning techniques focusing on neuro-imaging are extremely slow an inefficient due to the size of raw MRIs (or other traditional imaging). My project has preliminarily shown that with as little as 16 zoomed-in views of a brain, the age of the patient can be discerned with attention-based neural networks. FlyBy is capable of projecting properties of an image onto the surface of a 3D model before producing an image sequence. This is fascinating because in some cases deep learners using Fly By images are able to perform classification with fewer neuro-morphic features (such as sulcal depth, cortical thickness, and surface area) than was originally hypothesized. Even more interestingly, FlyBy is even able to accurately perform with extremely zoomed in image sequence which do not show the entire brain. Overall, this is an extremely exciting study which is providing a lot of insight into deep learning for the brain surface.
Relevant Papers:
Multi-view Convolutional Neural Networks for 3D Shape Recognition
Dominant Set Clustering and Pooling for Multi-view 3D Object Recognition
Learning Multi-View Representation With LSTM for 3-D Shape Recognition and Retrieval
O-CNN: Octree-based Convolutional Neural Networks for 3D ShapeAnalysis
Neural Machine Translation By Jointly Learning to Align And Translate
SWITCH TRANSFORMERS: SCALING TO TRILLION PARAMETER MODELS WITH SIMPLE AND EFFICIENT SPARSITY
OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
FlyBy CNN: A 3D object segmentation and classification framework
Example of spherical spiral subdivision in FlyBy (on a tooth):
Detection of Atypical Neurodevelopment in Infants Using PCA and KMeans Clustering (2020)
This was a final research project that I completed for my Introduction to Machine Learning Class (COMP 562) at UNC, Chapel Hill. The code we used to uncover these results is available in the GitHub repository below, however, due to HIPPA regulations, the data is not provided. Citations are listed at the bottom of the paper.
COMP562_EACSF_FINAL.pdfGlobal Extra-Axial Cerebrospinal Fluid Investigation (2019 - ongoing)
I have been studying the global extra-axial cerebrospinal fluid space in infants under the guidance of Dr. Martin Styner at the UNC Neuro Image Research Analysis Laboratories (NIRAL).
Project description: Currently my project has branched into two main focuses:
My first study has established a physiological difference in EA-CSF volumes in twins versus non-twin individuals.
My second study is currently establishing the significant adverse role of elevated EA-CSF at infancy in a child’s performance in a battery of behavioral, cognitive, and emotional tests administered at age 6.
Abstract and Poster published and at Perinatal Preterm and Pediatric Image Analysis workshop (PIPPI) at the Medical Image Computing and Computer Assisted Interventions conference (MICCAI) - 2020
Abstract and Poster presented at State of North Carolina Undergraduate Research and Creativity Symposium - 2020
Relevant Papers:
A lymphatic waste-disposal system implicated in Alzheimer’s disease
A Novel Framework for the Local Extraction of Extra-Axial Cerebrospinal Fluid from MR Brain Images
Automatic Measurement of Extra-Axial CSF from Infant MRI Data
Cerebrospinal fluid and the early brain development of autism
Increased Extra-axial Cerebrospinal Fluid in High-Risk Infants Who Later Develop Autism
PIPPI_Poster_Abstract13.pdfRubiks Cube Solver in C (2019)
Fun project to build a Rubik's cube solver from scratch - not very clean code. I came up with a "catch-all-cases" algorithm by tinkering around with a Rubik's cube, so it is definitely not very efficient. Let me know if you find a bug in the algorithm/code.
Graph Theory - Reversed Arcs Investigation (2018)
This was the final project for my high school Introduction to Proofs and Graph Theory class at the North Carolina School of Science and Mathematics where we investigated reversed arcs. See paper below for findings.
reversed_arcs.pdf