Doubao
豆包
2023-2025
In April 2023, the Doubao project was launched within ByteDance.
As one of the earliest designers, I took part in building the Mobile, Web, plugin, and PC products from the ground up, with a main focus on iterating the core chat experience on Mobile and designing the voice interaction features.
Part 1
Iterated the core chat experience from 0 to 1
Role
Product Designer
Design Team
2-3 designers
Time Period
2023.06 - 2025.02
1.Basic Chat Experience
By April 2023, products like ChatGPT, Perplexity, POE, and Character.ai had already gained visibility and user traction overseas. In contrast, how to build an AI-native product and establish user awareness remained unproven in the Chinese market. This led us to ask a fundamental question: Why do users come to us?
Despite the rise of large language models, the underlying user needs haven’t changed—they still revolve around information consumption and emotional connection:
Accessing high-quality information in a fast, efficient, and convenient way
Fulfilling emotional companionship, self-expression, and social entertainment needs
In the early stage of the product, I explored a wide range of directions — including chat, in-app browser, PDF reader, notes, community, intimacy features, and AI personas. However, exploring too many tracks before the model was ready led to a loss of focus on the core chat content quality.
After exploring various possibilities, we gradually reduced product complexity and refocused on enhancing the core Doubao chat experience:
>> 🤔 The input box and the AI's response are the most critical input/output channels. We focused on creating differentiation through a clean interface, smooth interaction experience, richer Markdown formatting, and thoughtful topic suggestions.
1.Input Experience
Leveraging the advantage of ByteDance's ASR recognition accuracy, we made voice input a key focus, with the principle of minimizing user distraction:
Users shouldn't need to look at the screen while speaking, and the message that appears immediately after speaking should be correct.
To achieve this, we tested multiple interaction solutions, and ultimately reached a result where — in terms of user perception — there is no noticeable delay in voice recognition appearing on screen.
After the improvement, the voice input consistently accounted for around 35% of all input methods.
As user engagement deepened, needs emerged for editing long texts, uploading images/files, and expanding capabilities. Based on this, I unified the structure of the input box across different forms and defined an extensible input component.
2.Reading Experience
In the early stages of the project, we studied the text reading experiences of various chat apps and document-style apps, conducting detailed research on line height, paragraph spacing, and more. This allowed us to improve screen efficiency while supporting diverse message card formats and ensuring a consistent reading experience across multiple languages.
After Doubao’s response types increased, I unified and optimized the display of Markdown elements such as lists, headers, code blocks, tables, and highlight blocks. I also wrote message card specifications to provide to other collaborators developing agents within Doubao, such as news, weather, Music, and Douyin local lifestyle cards.
3.Searching Experience
With users’ increasing demands for accuracy, timeliness, and richness of information, Doubao’s online retrieval architecture has undergone several iterations, some unified understandings have emerged:
Retrieval Process: It is unnecessary to show the entire search process to users. The technical pipeline can be complex, but users do not need to be aware of it.
Information Sources: Users rely on information sources for confidence and a sense of security. We aim to summarize information so well that users do not need to click on sources unless necessary, thus intentionally controlling the number of citations.
Information Structure: The combination of rich media and text is very important. Key information should be placed at the top, with auxiliary recommendations (usually videos) below, aligning better with reading habits.
4.Actions
The biggest difference between conversational AI and traditional search engines lies in the minimum granularity of information. Search engines treat entire web pages as their smallest unit, whereas LLMs operate at the level of individual information elements—a single word, a sentence, a piece of code, a table, an image, or a message.
We aim to enable users to interact with each of these minimal information units, such as copying a word, expanding a table, or exporting a message.
Regarding operations on messages, the long-press menu had a 10% usage rate. During iterations, the menu content gradually became overloaded, prompting a targeted redesign.
2.Message Sharing, Revisit, and Knowledge Management
As the user engagement deepens, revisiting and managing conversation content has become increasingly important. To facilitate users in saving and sharing on social media, a simple sharing feature was designed.
After launch, it achieved nearly 3% penetration, with a conversion rate exceeding 50%. The most frequently clicked options were saving images and sharing via WeChat, mainly involving content with high information density or strong entertainment value.
A major difference between Doubao and other products is that when opening the app, users do not enter a new topic page but remain in the main conversation. The advantage is a strong sense of asset continuity; however, this also leads to users lacking a clear understanding of the “create new conversation” feature. They continue to ask questions within the main conversation, which increases the difficulty of revisiting and searching messages. As a result, users tend to abandon message management altogether, which is unfavorable for long-term retention.
Within Doubao’s existing framework, we designed features like automatic timestamp summarization and search to make retrospection feasible.
Users with a deep focus on efficiency tend to want their own knowledge assets, including note editing. Thus, we quickly launched an MVP version of the message bookmarking feature. After launch, research showed that 85.71% of users bookmarked messages for easier future reference. The most frequently bookmarked content included generated copy, professional knowledge, search results, and document summaries. Some users also expressed a desire for secondary editing, and we are continuously iterating.
Part 2
Voice Call Iteration Process
Role
Product Designer
Time Period
2023.10 - 2025.03
1.Defining the Basic Interaction of Voice Calls
ByteDance’s ASR and TTS technologies have inherent advantages, so we planned to launch the voice chat feature by the end of 2023.
Building on a dialogue experience focused on efficiency, we aimed to enhance emotional connection with users. During voice chats, we sought to simulate human-to-human conversation as much as possible, emphasizing three key elements:
Doubao’s Persona: Following principles of friendliness and liveliness, we want users to see a dynamic avatar during conversations;
Response Content: Compared to text chat, Doubao’s replies are more conversational, warm, concise, and humanized;
Call Status: The states of Doubao’s responses, thinking process, and the user’s speaking status need to be clearly visible to users.
There were three design directions for Doubao’s persona:
1. Animated avatar
2. Real human images
3. Scenario avatar
After testing, the real human images caused a strong uncanny valley effect, and the scenario avatars felt too specific, limiting the breadth of conversations. Therefore, an animated avatar was chosen as the final solution.
During the iteration process, we replaced “pause call” with “mute” because we found that most users’ needs were to prevent surrounding noise from interfering with their conversation with Doubao, while still wanting to hear Doubao’s replies. Due to limitations in voice interruption accuracy, accidental interruptions occurred frequently. We aimed to create an experience similar to a real call—users can mute themselves, and the call only automatically pauses and enters a holding state when there are other calls or audio interferences.
After launch, 56% of users used the mute function. Next-day retention for calls increased by 2.23%. Call duration increased, while depth slightly decreased, which met our expectations.
Six months after the voice call feature launched, its penetration remained around 3%. Among Doubao’s weekly active users, over 40% were unaware of the call feature. To improve both penetration and user engagement, we not only enhanced the model experience with time awareness, internet connectivity, and memory capabilities, but also increased feature visibility and user outreach.
2.A Wider Range of Voice Call Use Cases
Through feedback from users, we learned that the core scenarios for voice calls are emotional companionship and problem-solving. Many users also use calls to practice their expressive abilities and English speaking skills, as well as to train for presentations and interviews. However, a common cause of user churn is the feeling of “not knowing what to talk about”, which leads to a loss of desire to continue chatting.
1.Scenario Mode
To help users find topics, we provided several scenario options based on frequent user requests and optimized response quality using different models. For example, in "English Learning" scenario, the default conversation language is English, while in the “Tree Hole” mode, the model listens more to the user.
After launching the “Scenario Mode”, users who engaged in calls using scenario mode accounted for 30.56%, a considerable proportion. The next-day retention for both the “English Learning” and “Emotional Tree Hole” modes exceeded 4%.
However, these modes did not affect message depth nor fundamentally solve users’ difficulty in finding topics. Therefore, we experimented with “Topic Recommendations”, which offer more candidates, greater timeliness, and higher flexibility.
2.Subtitle
We received a lot of user feedback, especially regarding English learning and information retrieval in professional fields, indicating that subtitles are a very necessary feature. Providing subtitles not only improves the efficiency and accuracy of information acquisition but also enables users to have deeper conversations.
To ensure the auditory experience is synchronized with the visual, we controlled the subtitle speed to match the speech speed, using a sentence-by-sentence display instead of word-by-word, minimizing visual jumps and information interference for users.
Subtitle experiments showed an enablement rate close to 13%, with the highest conversion rate in the English learning scenario. Call message activity over nearly 7 days increased by 3.13%, and penetration increased by 3.15%. Both user activity and chat depth improved.
3.Visual Understanding During Calls
Whether in software or hardware, large-model products are exploring practical scenarios for visual understanding, such as mock interviews, language learning, office assistants, online problem-solving, and gaming companions.
Through user interviews, we found that high-frequency and strong demands are concentrated in general knowledge scenarios like object recognition. Meanwhile, daily life monitoring and companionship needs typically require other hardware as a carrier to be realized.
1.MVP: Ask as You Shoot
When reviewing high-frequency needs, we found that inputting a one-time static image during calls already meets most scenarios, such as object explanation and text recognition, without needing to keep the camera on for a long time.
In contrast, in person-to-person video calls, everyday life scenarios mostly aim to convey emotional information, while work scenarios mainly involve screen sharing and presentations. Therefore, the MVP version was decided to use a photo-taking method during calls, with internal testing starting at the end of 2024.
When users input both visual and text information simultaneously, the timing design of responses becomes very important — deciding when to answer directly and when to consider the context input.
2.Video Call
Video calls can support scenarios requiring long, continuous questioning, such as visiting museums, traveling, repairing, cooking, and so on. Under current technical conditions, video calls are also implemented by extracting frames at appropriate intervals. Therefore, after the MVP photo-based call, we began designing the video experience.
The final released test version uses a full-screen camera view to enhance immersion and highlight the video feature.
MVP internal testing, integrating video, photo capture, photo upload, and screen sharing ↓
Other Design Tests ↓
4.Trying New Avatars to Overcome Audience Limitations
In the survey of September, we found that Doubao’s image was perceived as friendly and approachable but lacked a sense of technology and advancement.
Therefore, we hope to gradually update Doubao’s brand image to encompass technology, professionalism, and approachability, aiming to break through the limitations of audience and scenarios. This will be combined with the release of the end-to-end s2s voice model before the 2025 Spring Festival.
1.De-personalization
After small-scale experiments, the average call depth decreased by nearly 20% for the 18- and 31-40 age groups, but increased by nearly 20% for the 24-30 age group, which matched our expectations: teenagers love the Doubao cute avatar, and parents use real-time call for their kids' bed-time stories.
Therefore, we decided to launch an avatar switching feature that allows users to instantly change their avatar by clicking, aiming to comprehensively meet the needs of all user groups.
2.Character Companion Avatar
For character-type AI agents, users hope to establish deeper emotional connections. Therefore, we replaced static avatars with dynamic, anthropomorphic full-screen images to provide users with a more immersive conversation experience.
5.Some challenging topics that did not succeed — continuous voice chat
We once tried to lower the barrier to using calls, aiming to meet both quick interaction needs and immersive/emotional needs. Therefore, we experimented with various quick chat methods:
New entry point: ASR (automatic speech recognition) on the conversation page is a highly used input method, and users who frequently use ASR highly overlap with those who use calls.
New form: No need to hold the button to speak; continuous conversation is enabled on the conversation page, reducing the cost of follow-up questions during voice input.
😖 After small-scale testing, we found that users’ demand for long-duration, immersive experiences far exceeded the need for lightweight, short quick voice conversations. Lightweight quick interactions are actually better met by AI headsets and other hardware. Therefore, we abandoned using the mobile app as the carrier to fulfill this demand.
其他尝试方向 ↓