SPADE-COCO

Machine Learning Storyboarding for non artistic people

Timeframe: 1 week | 2021
My Role: UX Design
Tools Used: SPADE-COCO | Adobe Photoshop
Design Challenge
Apply machine learning in an interactivity or image making way.
Challenge Statement
How might we help non artistic people, with creating storyboards that could help visually convey a story?
Problem
Storyboarding is a very effective way to let other people see your vision. It's a great form of storytelling that can be very difficult for non artistic people, leaving them to draw stick figures and hope the message gets across.
Solution
SPADE-COCO machine learning helps with visually conveying a story across to another by generating images through segmentation maps. Creating photorealistic images through the process.
Measurement of Success
A person looking at these 6 images could describe what story is being told with very little confusion.
Story
Using RunwayML with Spade-Coco, which uses segmentation maps. I downloaded the reference labels which help identify what color is associated with what "thing". Using the "things" I had available to me, I wanted to write a quick story involving only 6 frames.


The Story
1. I went to bed one night and started having a dream.
‍
2. In this dream, I was in the center of a room scared and surrounded by different kinds of food that were alive.
‍
3. One of the foods walked up to me and gave me a high five
‍
4. We all became friends and started walking the streets
‍
5. We found a park and started dancing happily
‍
6. I then woke up from my dream surrounded by regular food
Exercise
I presented the story to someone who had little to no artistic background, and challenged them to create 6 separate storyboards in photoshop that reflected their vision of the story, using specific colors only, to identify each element in the frame.
‍

‍
Testing
After exporting the finished images from the participant, I plugged them into RunwayML with Spade-Coco and ran the model. After receiving the results I presented them to different individuals asking them to tell me the story as best they could, with first presenting them with the segmentation portion.

I then asked them to tell me the story with the generated images.
Results
The participants were able to identify the story being told with the generated images very well, with a few exceptions.

Two of the participants thought the whole experience was due to a "weed or other drug trip". This came from the first image dream cloud looking like smoke.

The participants really enjoyed the ability to be able to visually share a story with others without having artistic skills. Especially when it came to pitch meetings, which was mentioned by one participant.‍

↑
BACK TO TOP

Christian Enriquez
LinkedInInstagram