My SF solution currently has one service, but seems like there is a problem in the reference from the sfproj to the service.
I saw the following question and answer and made sure all target framework is matching: Service Fabric Service reference has yellow triangle
I'm using Visual Studio 2017.
All of these projects are targeting .NET Framework 4.5.2, sfproj version 1.6
I also tried to add new services, and they too have yellow reference.

Problem was with NuGet packages.
My local NuGet server only had
Microsoft.VisualStudio.Azure.Fabric.MSBuildversion 1.4, but apparently, thesfprojdemanded version 1.6. By manually editing thesfproj, problem was resolved.