Team
Megan Lin
Alina Fang
Timeline
2 Day
Hackathon
Tools
Figma
Figjam
Writing accurate prompts for AI is difficult.
Given the rise of AI, UX designers must learn to integrate them into their workflow. However, accurately depicting the envisioned design is difficult.
Vague prompts cause AI tools such as Mid-journey to generate non-usable UI components. This challenge calls for an innovative solution to bridge the gap between creative ideas and AI-driven execution.
AI Figma Prompt Optimization Plugin
Utilizes AI to eliminate vague prompts, ensuring accurate mockup generation.
When I first saw the prompt on xhacks, I started brainstorming out-of-the-box ideas, envisioning tools like 3D design VR apps. However, I soon realized that there are already many AI tools that designers can integrate into their workflow. This prompted me to ask: “Why are AI tools not yet widely adopted by UX designers?”
Innovate an AI-powered solution for designers.
Image Generation
Can be used for designs
Require designers to invest time in refining prompts.
Co-pilots
Can be used to prep or handoff
Cannot be directly applied to the design process.
To investigate, I led the team to analyze current AI tools and documented pros and cons. Seeing the gap between AI tools and the design process, we shifted our research to focus on incorporating AI tools into the design flow.
Current AI tools are not tailored to design flow.
To better understand how we can integrate current tools into the design process, we conducted 4 interviews with CMU students and professors. Then, we crafted a user journey map to document major pain points and opportunities to apply AI technologies.
Considering the business aspect, we plotted these solutions in the impact-effort matrix to better gauge the effectiveness of each solution. We agreed that prompt refinement for Midjourney would be the “easy win” given the maturity of the tools and the direct impact it has on generating designs.
Capturing key pain points and opportunity gaps.
When I further researched the challenges of using Midjourney, I came across an article “Is this the end for UX designers?” Affirming the importance of learning AI tools, the author demonstrated how it takes him multiple tries of refining the prompt to finally get the desired screens.
This helped us further narrow down our design problem to focus on prompt engineering.
Final Insights.
Takes multiple tries to refine prompts.
Design Problem
“How might we help users write prompts for AI tools like Midjourney to accurately generate UI components and mockups?”
To start our design process, we sketched out a simple user flow showcasing how users would add suggested descriptors.
Visualize the prompt refinement process.
In the first iteration, we created 10 categories for users to refine their prompts. However, I soon realized that it would cost designers too much time to click into each category and change the descriptors.
Add descriptors & keep track of past prompts:
provides a refined prompt with added descriptors
allows designers to track all the prompt variations
Refine based on 10 categories:
provides guidance on what part of the prompt to refine
provides suggestions to help designers input more descriptive words
Providing a set criteria for designers to enter is limiting and tedious.
To streamline the prompt refinement process, we decided to reduce the categories to “Theme”, “Dimension” and “Tasks”. This way, the users will be able to accurately describe the visual as well as the features of the mockup.
Dilemma: How to limit number of categories while ensuring that the prompt is accurate?
After testing our design on CMU HCI students, we received feedback about automating the refinement process. As we played around with the design of the prompt editor, we finally came up with the idea to automatically highlighting vague terms that might lead to unclear mockups. Taking inspiration from the autocorrect feature, we pivoted to a design that would minimize the clicks needed to view and change to suggested descriptors.
The Pivot: Ditch the manual approach; highlight vague terms automatically.