This issue happens when I use VirtualBox 5.2.6 on a “Mac High Sierra” host.
This issue happens in the following conditions:
Install VirtualBox 5.2.6 on MacOS High Sierra
An “Ubuntu 16.04” guest instance is hosted on the VirtualBox
The “Display -> Acceleration -> Enable 3D Acceleration” option of the Ubuntu guest instance is checked
The “VirtualBox 5.2.6 Add-on” is installed on the Ubuntu guest instance
When all the conditions meet, starting the Ubuntu guest instance will get a black screen and no display at all.
To fix this issue, you need to uncheck the “Display -> Acceleration -> Enable 3D Acceleration” option and restart. Then the guest OS can start properly. This looks like a bug of “VirtualBox 5.2.6” since no such issue was found under “VirtualBox 5.2.4” when the other conditions were the same.