AI for visual communication via sketching

Last weekend, Spell sponsored a hackathon at Brown University. Over 500 students from Brown, RISD, and colleges and universities across the US gathered in Providence, Rhode Island for a chilly weekend of app building and coding.

The Hack@Brown event is noted amongst the big college hackathons for it’s inclusive mission. “While many other hackathons accept applicants based on coding experience, Hack@Brown selects participants through a weighted random lottery. Preference is given to applicants who are from underrepresented groups in computer science as well as first-time hackers.”

Spell’s easy-to-use interface and beginner friendly tutorials make it a perfect platform for new hackers.

The prize

We were really pleased with the prize submissions from our HackMIT sponsorship, so we decided to offer the same prize at Hack@Brown.

Since Spell is application agnostic, we kept the Spell prize wide open. We wanted to give students free reign to be creative with Spell.

Our offer to hackers: We want to see your creative applications of deep neural networks. Maybe it’s an art project, a generated book of poetry, or an app that predicts housing prices. As long a it uses Spell’s cloud GPU, it’s fair game!

The winning project

We were really pleased with the projects we saw being built on Spell. After reviewing the submissions carefully, we awarded the Spell prize to a team that built a hand-drawn sketch recognition app.

In 2018, about 1 in 59 children is diagnosed with autism. Treatment are expensive; educational institutions are over-booked; parents couldn’t afford the amount accompany required for early intervention. Automatic assistanceis key to improve the situation, according to many people we talked to who are closely related to autism;

On the other hand, science advancement makes it clear that visual communication is more effective than text for intellectual and social development of children with autism. We want to leverage AI technology to enable visual communication via sketching. This is what we want to accomplish with A.I. (Autism Inspire).

We trained a DenseNet model (a variation of CNN model) on 38 million sketches from the Google Quick!Draw dataset. The sketch classification model recognizes the sketches drawn by our users and uses the label as the key to form a sentence. In the future, we expect to build a “sketch captioning” model where makes more informative and expressive sentences made up of all the details the network is able to capture in it. This will lead to more meaningful and interesting visual conversation.

Our model is served on SPELL. It requires ZERO extra work to publish a TF model and responds at LIGHTNING speed free of concerns about container cold start.

The winning team

All four members of the winning team are MA candidates in the Computer Science department at Brown University.

Fumi Honda: (github.com/fumih) Fumi is pursuing a CS Master’s at Brown, after working for two years as a machine learning engineer. She hopes to combine her computational skills with social science research background and apply them for social good.

Queena Zhang: (github.com/queena-zhang) Queena is a master’s student at Brown University majoring in computer science. She had two internships in the past where she worked as back-end engineer. She’s also interested in augmented reality.

Yang Zhang: (github.com/AllenZYoung)Yang Zhang is a master student at Brown University majored in Computer Science, he’s interested in Machine Learning, Full-Stack Development and Software Engineering. He also enjoys playing the piano and reading social science books.

Ziyin Ma: (github.com/blownhither) Ziyin is an enthusiastic full-stack developer keen on developing AI-centered applications for business and people. He is a master student at CS Brown, a head teaching assistant for distributed system and a researcher at Brown Visual Computing.

Ready to Get Started?

Create an account in minutes or connect with our team to learn how Spell can accelerate your business.