This issue happens when I use VirtualBox 5.2.6 on a “Mac High Sierra” host.
This issue happens in the following conditions:
Install VirtualBox 5.2.6 on MacOS High Sierra
An “Ubuntu 16.04” guest instance is hosted on the VirtualBox
The “Display -> Acceleration -> Enable 3D Acceleration” option of the Ubuntu guest instance is checked
The “VirtualBox 5.2.6 Add-on” is installed on the Ubuntu guest instance
When all the conditions meet, starting the Ubuntu guest instance will get a black screen and no display at all.
To fix this issue, you need to uncheck the “Display -> Acceleration -> Enable 3D Acceleration” option and restart. Then the guest OS can start properly. This looks like a bug of “VirtualBox 5.2.6” since no such issue was found under “VirtualBox 5.2.4” when the other conditions were the same.
Added on 2018-05-16: Today I found on Ubuntu 18.04 if “video memory” is set to “64MB”, this issue happened. However if I set video memory to “16MB” or “32MB”, this issue didn’t happen. I’m not sure if the issue happened on Ubuntu 16.04 is also related to the video memory setting.