Future Smart Home for
Interactive Data-Enabled Recommendations

Fridge Interface and Mobile App 2021

A product-service innovation project in Miele
that included the entire innovation process from user research, design to testing and iteration.

Oya is a humanized, understandable, and sustainable IoT system, embedded in a smart fridge to build a health-aware lifestyle for you. Balance your diet, sport, and shopping everywhere and anywhere.

Overview

The project has collaborated with Miele Design Center to research and design interactive data-driven recommendations in the context of future smart home IoT. The main goal is to humanize the smart system by utilizing personal data. Furthermore, the project investigated deeply data types and interactive styles for supporting the presentation of personal recommendations on smart products. I worked closely together with the internal stakeholders, synthesized their thoughts, and proposed a data-enabled solution with a hi-fi prototype as the artifact to explore future IoT. The result contributes to the organizational changes within the organization.

Focusing on Humanizing Smart Agents

Tools

Semi-structured interview, Adobe Creative Suite (XD, AI, ID, PS, Premium), Figma, Excel

My Roles

Stakeholder Interviews, Market Research, Concept Sketch, UX Design, UI Design, Tangible Prototype, Modeling, Qualitative and Quantitative Analysis, Evaluation, Filming

Interactive Artefact

The system provides a multimodal interface through the fridge and mobile interfaces with a voice agent, Oya. It adapts to the personal preference to decide when to show what. The flow below shows an example of presented interfaces depending on timing.

When You are Home

Onboarding Process to Morning Routine

When Starting Oya
Let Oya Get to Know You

The onboarding process is to make personalization so everyone can formulate their daily cooking and eating rhythm according to their personalized routine. The onboarding process contains 8 steps, such as food preference (including allergic) etc.

Presented on fridge

Before Leaving Home
Oya recommend shopping time according to weather forecast

In the morning, both mobile and fridge present personalized recipes. The user can save the recipe if they want to see it next time. Meanwhile, the fridge will recommend shopping time according to the weather forecast.

Lunch Review
The User Can Check The Ingredient List, and Order for Picking Up in The Supermarket

During lunchtime, the saved meal will be presented under the wishing list. The hypothesis is that once the user has more time to look at the phone, they have higher chances to check the detailed ingredients, edit, and shop the ingredients.

The delivery process will be shown after showing up in the kitchen.

After Work - Home / On Car

Personalized Recipe Recommendations and Personal Health Data

The recipe is recommended by the weather forecast on the weekend. It pops up on the fridge after the user comes back from work or the supermarket and enters the kitchen; or before starting to cook dinner. The voice agents will interact with the users to know if they want to add the ingredients that are not in the fridge to the shopping list.

Define Challenges after Research

I did grounded research for mundane data in everyday life, humanizing the experience of automation, the modalities of interaction among automotive artifacts, and data visualization to foster interaction to set up a background to further design for the context of the smart kitchen environment.

How can the system be transformed into an interactive style within the whole cooking journey,
and assist the users for
building up a health-aware lifestyle?

How should the interface be performed with the transformative data in an aesthetically, approachable and clear visualization style?

Data-Enabled Design (DeD)

The project applies DeD methodology to use data as the creative material and apply experimental and prototype-centric methods for fast hardware and software design iteration.

Design Based on Personas:
Digital Revolutionary and Responsible Family

Mike represents a new digital revolutionary,

meaning that he is willing to use the digital

ecosystems extensively. Linda and Mark's attitude toward using digital products is very open and very conservative. Therefore, they would be potential users when the technical functionality of the digitalized product is more mature.

Opportunities for mobile and fridge were concluded from the integrated customer journey map

I synthesized the opportunities from 6 participants. 19 activities were drawn out and many interesting possibilities were found, for example, they would like to try out new recipes according to their personal preference; the time they left the company is the most likely time to think about what to eat for dinner and weaken.

Situated prototype and interactive chatbot gave the insight in the context, behavior and experiences

I collected quantitative and qualitative data through the prototypes with the Oya system, which is situated in real life. We find they would like to interact with recipe information and daily mood while using the system.

Installed the prototype for a week and collected their preference by a workshop.

Using the chatbot to collect data, like food intake, feedback system, etc.

Wireframes for mobile and fridge visual interfaces

The interfaces were created according to the flow of the user story. It's based on the touchpoints of the contexts. For instance, in the morning, user might want to look at fridge instead of the mobile.

Wireframe of The Mobile

Wireframe of The Fridge

Explored VUI and planned dialog

Write down what are the chosen touch points to be further developed.

The geometric lines are further sketched through different emotional qualities. We chose one which fits the brand image meanwhile.

Creating the coversation dialog.

Collected data from interactive chatbot to further iterate

I listed the daily steps, workout, burned calories, and moods. Later, I transferred them into the pattern according to the exploration above. Afterward, I interviewed the participants and understood their interpretation of the data visualization.

Evaluation and results

I did evaluations with 6 individuals with 6 personalized flows, and a focus group to answer the questions. The question is, 'Do these system features enhance the smart fridge?'; 'Does the system become more humanized'?

The overall evaluation shows positive feedback, from the seamless experience; personalized setup; pop-up functionalities; the visualization is interpretable. Regarding the e

Combined data on weather and recipes are appreciated; the participants like the data about weather, mood, and recipes.

1/ Voice agent owning polite quality
2/ Pop-up effect with surprising and
personalized meal

3/ People are allowed to have
negative emotion and eating less healthy food

4/ Narrative visualizations presented in an open space
for triggering the users’ curiosity about their personal data

Reflection and next steps

The intimacy of data and the level of data privacy would influence the modalities of interaction.

People especially do not want to see their mood being presented on the interface of the fridge nor do they like if the mood is spoken out or shown on the system.

The pop-up effect is discussable people like the effect presented on the fridge, but don’t like it shown on mobile devices. This might reveal the closeness of humans and devices.

In the project, one of the participants thought should would not like to interact with Oya. However, she was very positive and talked to the smart fridge naturally. Interestingly, she interpreted that it might trigger her interest due to the male sound.

Next step, the smart objects could further adapt themselves to show the right interaction through the understanding of users’ contexts, which is the so-called ‘Contextual AI’. The core value is to be intelligible, adaptive, customizable, controllable, and context-aware. Thus, the ‘semi-automation’ of the smart kitchen system can be further realized