An integrated multi-modal solution for adding synesthesia experience to ordinary food photos. Using natural language processing and visual recognition, SENTHESIA learn from the comments and photos from these websites and more materials online, to extract proper and delicate description from the semantic space, and analysis the key elements of taste and sense experience. Visually impaired users can also benefit from vivid descriptions beyond photos.
What if you see this picture? You may feel quite calm——an ordinary picture of food, isn’t it?
But with the vivid description and sound? You may feel mouthwatering now.
What makes the experience so different?
Most of us share the similar semantic space for describing the taste of food. These vivid descriptions, combining with vision and auditory sense, create a strong experience of gastronomy synesthesia, which means 联觉 or 通感 in Chinese.
We produce SENTHESIA, an integrated solution for adding synesthesia experience to ordinary food photos.
Users are actively engaged in the process of keeping their own multimodal food experience. By uploading photos of food and surroundings position and sound, describing the taste and so on, users contribute to our cloud database. By exploring the database, users could either re-experience the food uploaded by themselves, or try the contents uploaded by others or designed by restaurants.
Except for the food information generated by our users, we can also take advantage of rating websites such as meituan and dazhongdianping.
Using natural language perception and visual recognition, we learn from the comments and photos from these websites and more materials online, to extract proper and delicate description from the semantic space, and analysis the key elements of taste and sense experience.
With the app, there is portable solution with cardboard VR
And comprehensive solution with hololens is supported for better experience.
If olfactory device is provided, users can also experience the corresponding smell.
Olfactory Experiments
My Role
Proposed and designed the concept, now cooperating with an interdisciplinary team to implement prototype.