Design Diagram

Our goal is to build a Mixed-Reality application to enhance the piano learning experience, both remotely and collaboratively . By visualizing the correct hand movement and notes to be played in Hololens, users’ learning curve increases by multiple folds.Because our design is a multi-device collaborative concept, so it’s necessary to define an overall workload for our design concept. we believe this gives us a blueprint to communicate on what we need to do.

Figure1 Design Diagram

Figure2 Teaching scenario sketch

Storyboard

Prototyping

Prototyping for XR applications is different from prototyping for web or mobile applications. There is no standard way to build a digital prototype. We decided to use body storming methodology to create a scene. One thing that we found interesting is that we built a paper prototype of the HoloLens device. This is really helpful for us to consider about the Field of View (FOV) limitation of HoloLens. And during the second iteration, we added ipad to get a taste of how it feels to interact with the menu digitally.

Figure3 1st Version Prototype&2nd Version Prototype

Technological discussion

At first, we decided to use the Live stream and Leap motion to implement the teaching function. However we found out that Leap motion cannot detect trivial finger movement while playing piano, so we finally decided to use Video to demonstrate tutorial.

For exploring the technical possibility for musical bar dropping function, I built a simple demo for the music rhythm game-the rhythm turtel.

Final system Diagram

Figure4 Final system Diagram

UI Design

User testing&iteration

We are fortunate enough to get our design tested by others at the exhibition and get valuable feedbacks. Based on those feedbacks, we improved our project by adding more feedback.

Figure5 User test

• Make sure that people who have never used HoloLens before go through the gesture tutorial first
• Add feedback when user presses the wrong key on the Keyboard
• Add different color(ivory and ebony) to the virtual keyboard for better mapping the real keyboard

Final Output

Figure6 Menu Selection

Figure7 Section Practise

Self-reflection

This is a really unique experience designing for HoloLens. There is no standard progress or theory for design for MR application. We referred to the Microsoft HoloLens developer documentation a lot and also experience from VR development. We found out that playing with physical prototypes and body storming are especially useful to design the system at early phase.

Regarding the User testing, it’s good to have a tutorial about the gestures and visual cues inside the application. Also, in MR design,it’s also worth noticing the trade-off between perfect user experience and technical possibility.