How did Microsoft's HoloLens start?

Hackathon - Microsoft HoloLens

Virtual, augmented and mixed reality are terms that you may have heard before or indirectly dealt with. But what does it mean? As part of a two-day hackathon, we wanted to dedicate ourselves specifically to the topic of ‘Mixed Reality’ and we came across some surprising results.

 

But what is the difference between the individual “forms of reality”?

  • Virtual Reality describes a world that is separate from regular space and is not influenced by our environment.
  • Augmented Reality is the same as Virtual Reality with the difference that the information displayed in it can come from our environment
  • With mixed reality, the boundary between our environment and the virtual world becomes blurred. A Microsoft HoloLens Developer Edition was used for our hackathon.

In order to guarantee a presentable result in the end, some milestones were defined from the beginning:

  • Stage 1: Presentation of a 3D model of Berlin on a table including additional information in an app, which can then be projected onto the wall.
  • Step 2: Representation of a single building instead of the city map from the 3D model and an optically prepared app on the wall.
  • Level 3: Interaction options with the building, e.g. to display dimensions. The additional information can either be displayed directly in the model or in the app on the wall.

The HoloLens

To get started, we dedicated ourselves to the HoloLens. For people who have never had anything to do with this, using it for the first time is a bit strange, but still fantastically easy.

After starting the HoloLens for the first time, it will guide you through the calibration and configuration of the device. You have to alternately close one eye and then the other and cover with your finger six different points that appear in the middle of the room. This is used, among other things, to measure the wearer's eye relief, which has a direct effect on the display of holograms in space.

Gesture control is then explained to the user. There are only three of these:

Ready. If the gesture is performed in the recognition area of ​​the HoloLens, the cursor changes to a circle

Please adjust your cookie settings in order to be able to play the video.

To adjust

Bloom, to open and close the Windows start menu and Air Tap, select the currently viewed object / menu entry.

Please adjust your cookie settings in order to be able to play the video.

To adjust

In addition to gesture control, voice control is also available at any time. Either via Microsoft Cortana or, depending on the context in which you are currently, also individual commands such as Remove, Adjust etc.

As soon as one has mastered the gesture control, standard information of the user is queried: First a Microsoft account with which one would like to log in, also with which WLAN one would like to connect.

A note at this point: If you are currently using a HoloLens Developer Edition, only one person can log on to it. In order to subsequently switch users, the HoloLens must be reset to factory settings. It makes sense to create and use a general account right from the start.

Once you have completed the introduction and configuration, you can start using the HoloLens straight away.

Out of the box apps

To make it easier to get started, Microsoft has already installed some applications for using the HoloLens, including the in-house browser “Edge”.

If you start this, a floating window appears in the field of vision of the HoloLens wearer. If you move your head, the window moves with it. Depending on what you are looking at, the HoloLens tries to identify the area and moves the window along it. If you are satisfied with the position of the window, you can now fix the window at the desired location with a “click”. All parameters such as size, position and alignment can be changed afterwards at any time.

After starting and using the HoloLens apps at the latest, it is noticeable that various aspects are perceived as completely natural and intuitive:

  • At no time could the contrast or the brightness of the apps be criticized.
  • An app also always stayed where it was stored.
  • There was no noticeable delay or other image errors.
  • The all-round view was unrestricted throughout
  • The tint of the HoloLens glass looks much stronger from the outside, but does not restrict the user's view any further and gives the environment a slightly greenish filter, similar to sunglasses.

Hello World - HoloLens variant

To make it as easy as possible to get started with HoloLens app development, Microsoft offers a number of courses in its Academy. With the help of this, the first app could be developed, which in the first step should do nothing other than display a cube in space.

The following software is generally required to get started, regardless of whether you just want to work through the tutorials in the Academy or write your own apps:

If 3D objects are to be displayed in the app to be developed, Unity can be used, a software with which, among other things, games can be developed. Experience with Unity is very valuable in this regard. Alternatively, DirectX is available, which gives you more direct access to the hardware. However, this is bought at the cost of much more effort in development.

Deploying the app is quick and easy. The project opened in Visual Studio, which is also created by Unity, can be deployed from there directly onto the previously registered HoloLens.

The result can be seen in the following video:

Please adjust your cookie settings in order to be able to play the video.

To adjust

Sharxx 4 Hololens

Now that it has been ensured that both the development environments were correctly installed and that the deployment to the HoloLens worked, the actual development can begin.

As mentioned at the beginning, our app should, among other things, display a 3D model of Berlin on a table and an additional window that can be freely positioned with detailed information. However, it became clear very early on that the HoloLens was not intended to display a 3D app and an additional window at the same time. At least not if the 3D app comes from Unity. Said apps run in a so-called “exclusive” mode, among other things in order to provide the app with as much computing capacity as possible. At the same time, this means that other holograms are hidden. Accordingly, an additional window with detailed information was initially discarded.

The implementation took place in the following iteration steps:

  • Day No. 1
    Show Berlin 3D model, including base plate
    Integration of your own cursor
    (Re) positioning of the 3D model is possible with a “click”:
    - In addition, no “click” for positioning could be made.
    - Since the base plate was very thin, the hologram sank into the floor.
    Thicker base plate and the color of the base plate changes when you look at it.
  • Day # 2
    Exchanged soil texture
    When the app is started, the floor slab is displayed without a building and can be moved and positioned directly.
    For performance and usability reasons, all buildings are hidden as long as the floor slab is moved.
    Clicking on the building enlarges / reduces the corresponding building.

Lessons learned

Based on the app created so far, the following findings result:

  • The operating concept of the HoloLens is very simple. In short there is only one “click”. This has a direct impact on actions that can be performed with 3D objects. Instead of making several actions available within the same object, it is advisable to display a menu bar in which the desired action can be selected beforehand.
  • The performance of the HoloLens is usually very good. However, the 3D model we used was too complex due to the conversion from another format, so that a noticeable jerking was visible. The issue of performance should therefore be considered and dealt with at an early stage.
  • While watching or recording a live stream with the HoloLens Windows app, the quality and refresh rate are noticeably reduced for the HoloLens wearer.
© 2021 novaCapta Software & Consulting GmbH