Elasticsearch: Unable to start Windows service. The system property [es.path.conf] must be set

Question

We're facing a problem coming from several users that they cannot use Elasticsearch on their machines (the windows service cannot be started). The version being used is the 6.6.0.

The information we have from the field is very poor and insufficient to make a good diagnostic of the problem (hopefully we get better info). However, the following is what we know:

• The stdout log file points the following:

2019-08-20 08:38:23 Commons Daemon procrun stdout initialized
ERROR: the system property [es.path.conf] must be set

• The service installation/start batch file points that the service was correctly installed and started. Moreover, the -Des.path.conf="C:\ProgramData\Elastic\Elasticsearch\config" was set during installation

...
C:\Program Files\Elastic\Elasticsearch>set ES_PARAMS=-Delasticsearch;-Des.path.home="C:\Program Files\Elastic\Elasticsearch";-Des.path.conf="C:\ProgramData\Elastic\Elasticsearch\config";-Des.distribution.flavor="default";-Des.distribution.type="zip" 

C:\Program Files\Elastic\Elasticsearch>if "auto" == "" set ES_START_TYPE=manual 

C:\Program Files\Elastic\Elasticsearch>if "" == "" set ES_STOP_TIMEOUT=0 

C:\Program Files\Elastic\Elasticsearch>if "" == "" set SERVICE_DISPLAY_NAME=Elasticsearch 6.6.0 (elasticsearch-service-x64) 

C:\Program Files\Elastic\Elasticsearch>if "" == "" set SERVICE_DESCRIPTION=Elasticsearch 6.6.0 Windows Service - https://elastic.co 

C:\Program Files\Elastic\Elasticsearch>if not "" == "" (if not "" == "" (set SERVICE_PARAMS= --ServiceUser "" --ServicePassword "" ) ) 

C:\Program Files\Elastic\Elasticsearch>"C:\Program Files\Elastic\Elasticsearch\bin\elasticsearch-service-x64.exe" //IS//elasticsearch-service-x64 --Startup auto --StopTimeout 0 --StartClass org.elasticsearch.bootstrap.Elasticsearch --StartMethod main ++StartParams --quiet --StopClass org.elasticsearch.bootstrap.Elasticsearch --StopMethod close --Classpath "C:\Program Files\Elastic\Elasticsearch\lib\*" --JvmMs 256  --JvmMx 256  --JvmSs 1024 --JvmOptions -Xms256m;-Xmx256m;-XX:+UseConcMarkSweepGC;-XX:CMSInitiatingOccupancyFraction=75;-XX:+UseCMSInitiatingOccupancyOnly;-Des.networkaddress.cache.ttl=60;-Des.networkaddress.cache.negative.ttl=10;-XX:+AlwaysPreTouch;-Xss1m;-Djava.awt.headless=true;-Dfile.encoding=UTF-8;-Djna.nosys=true;-XX:-OmitStackTraceInFastThrow;-Dio.netty.noUnsafe=true;-Dio.netty.noKeySetOptimization=true;-Dio.netty.recycler.maxCapacityPerThread=0;-Dlog4j.shutdownHookEnabled=false;-Dlog4j2.disable.jmx=true;-Djava.io.tmpdir=C:\Users\KE'SBU~1\AppData\Local\Temp\elasticsearch;-XX:+HeapDumpOnOutOfMemoryError;-XX:HeapDumpPath=data;-XX:ErrorFile=logs/hs_err_pid%p.log;-XX:+PrintGCDetails;-XX:+PrintGCDateStamps;-XX:+PrintTenuringDistribution;-XX:+PrintGCApplicationStoppedTime;-Xloggc:logs/gc.log;-XX:+UseGCLogFileRotation;-XX:NumberOfGCLogFiles=32;-XX:GCLogFileSize=64m ++JvmOptions -Delasticsearch;-Des.path.home="C:\Program Files\Elastic\Elasticsearch";-Des.path.conf="C:\ProgramData\Elastic\Elasticsearch\config";-Des.distribution.flavor="default";-Des.distribution.type="zip" --LogPath "C:\Program Files\Elastic\Elasticsearch\logs" --LogPrefix "elasticsearch-service-x64" --StdError auto --StdOutput auto --PidFile "elasticsearch-service-x64.pid" --DisplayName "Elasticsearch 6.6.0 (elasticsearch-service-x64)" --Description "Elasticsearch 6.6.0 Windows Service - https://elastic.co" --Jvm "C:\Program Files\Elastic\Elasticsearch\jdk\jre\bin\server\jvm.dll" --StartMode jvm --StopMode jvm --StartPath "C:\Program Files\Elastic\Elasticsearch"  ++Environment HOSTNAME="%COMPUTERNAME%" 

C:\Program Files\Elastic\Elasticsearch>if not errorlevel 1 goto installed 

C:\Program Files\Elastic\Elasticsearch>echo The service 'elasticsearch-service-x64' has been installed. 
The service 'elasticsearch-service-x64' has been installed.

C:\Program Files\Elastic\Elasticsearch>goto:eof

C:\Program Files\Elastic\Elasticsearch>"C:\Program Files\Elastic\Elasticsearch\bin\elasticsearch-service-x64.exe" //ES//elasticsearch-service-x64 --LogPath "C:\Program Files\Elastic\Elasticsearch\logs" --LogPrefix "elasticsearch-service-x64" --StdError auto --StdOutput auto 

C:\Program Files\Elastic\Elasticsearch>if not errorlevel 1 goto started 

C:\Program Files\Elastic\Elasticsearch>echo The service 'elasticsearch-service-x64' has been started 
The service 'elasticsearch-service-x64' has been started

C:\Program Files\Elastic\Elasticsearch>goto:eof

• As far as we know the ES_HOME and ES_PATH_CONF are set:

C:\>echo %ES_HOME%
C:\Program Files\Elastic\Elasticsearch


C:\>echo %ES_PATH_CONF%
C:\ProgramData\Elastic\Elasticsearch\config

From what I saw in the ES source code, the error message seems to come from the "EnviromentAwareComand.java". It seems that at that level, the System.getProperty("es.path.conf") is returning null.

As a summary I see the following:

• Environment variables set
• Windows service installed using the Java option -Des.path.conf="C:\ProgramData\Elastic\Elasticsearch\config"
• Windows service says ERROR: the system property [es.path.conf] must be set

Unfortunately we were not able to have access to the registry nor to the elasticsearch-service-mgr.exe so we could have checked the actual value of the es.path.conf.

What are we missing here? Does anyone know why ES is throwing that error?

Thanks in advance!

UPDATE

We were able to connect to one of the problematic machines. The culprit of the problem is somehow the Window user name.

From the previous log files we can see that the call to the service is the following:

C:\Program Files\Elastic\Elasticsearch>"C:\Program Files\Elastic\Elasticsearch\bin\elasticsearch-service-x64.exe" //IS//elasticsearch-service-x64 --Startup auto --StopTimeout 0 --StartClass org.elasticsearch.bootstrap.Elasticsearch --StartMethod main ++StartParams --quiet --StopClass org.elasticsearch.bootstrap.Elasticsearch --StopMethod close --Classpath "C:\Program Files\Elastic\Elasticsearch\lib\*" --JvmMs 256  --JvmMx 256  --JvmSs 1024 --JvmOptions -Xms256m;-Xmx256m;-XX:+UseConcMarkSweepGC;-XX:CMSInitiatingOccupancyFraction=75;-XX:+UseCMSInitiatingOccupancyOnly;-Des.networkaddress.cache.ttl=60;-Des.networkaddress.cache.negative.ttl=10;-XX:+AlwaysPreTouch;-Xss1m;-Djava.awt.headless=true;-Dfile.encoding=UTF-8;-Djna.nosys=true;-XX:-OmitStackTraceInFastThrow;-Dio.netty.noUnsafe=true;-Dio.netty.noKeySetOptimization=true;-Dio.netty.recycler.maxCapacityPerThread=0;-Dlog4j.shutdownHookEnabled=false;-Dlog4j2.disable.jmx=true;-Djava.io.tmpdir=C:\Users\KE'SBU~1\AppData\Local\Temp\elasticsearch;-XX:+HeapDumpOnOutOfMemoryError;-XX:HeapDumpPath=data;-XX:ErrorFile=logs/hs_err_pid%p.log;-XX:+PrintGCDetails;-XX:+PrintGCDateStamps;-XX:+PrintTenuringDistribution;-XX:+PrintGCApplicationStoppedTime;-Xloggc:logs/gc.log;-XX:+UseGCLogFileRotation;-XX:NumberOfGCLogFiles=32;-XX:GCLogFileSize=64m ++JvmOptions -Delasticsearch;-Des.path.home="C:\Program Files\Elastic\Elasticsearch";-Des.path.conf="C:\ProgramData\Elastic\Elasticsearch\config";-Des.distribution.flavor="default";-Des.distribution.type="zip" --LogPath "C:\Program Files\Elastic\Elasticsearch\logs" --LogPrefix "elasticsearch-service-x64" --StdError auto --StdOutput auto --PidFile "elasticsearch-service-x64.pid" --DisplayName "Elasticsearch 6.6.0 (elasticsearch-service-x64)" --Description "Elasticsearch 6.6.0 Windows Service - https://elastic.co" --Jvm "C:\Program Files\Elastic\Elasticsearch\jdk\jre\bin\server\jvm.dll" --StartMode jvm --StopMode jvm --StartPath "C:\Program Files\Elastic\Elasticsearch"  ++Environment HOSTNAME="%COMPUTERNAME%" 

The interesting part is how the -Djava.io.tmpdir=C:\Users\KE'SBU~1\AppData\Local\Temp\elasticsearch is set. As you can see the path contain a single quote character.

By taking a look to the registry entry where the procrun stores the java parameters we can see that all java options are broken because of this (the single quote is not there and the rest of the JvmOptions after the -Djava.io.tmpdir seem to be not parsed):

-Xms256m
-Xmx256m
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=C:\Users\KESBU~1\AppData\Local\Temp\elasticsearch;-        XX:+HeapDumpOnOutOfMemoryError;-XX:HeapDumpPath=data;-XX:ErrorFile=logs/hs_err_pid%p.log;-XX:+PrintGCDetails;-XX:+PrintGCDateStamps;-XX:+PrintTenuringDistribution;-XX:+PrintGCApplicationStoppedTime;-Xloggc:logs/gc.log;-XX:+UseGCLogFileRotation;-XX:NumberOfGCLogFiles=32;-XX:GCLogFileSize=64m

-Delasticsearch
-Des.path.home=C:\Program Files\Elastic\Elasticsearch
-Des.path.conf=C:\ProgramData\Elastic\Elasticsearch\config
-Des.distribution.flavor=default
-Des.distribution.type=zip

So from my point of view the problem is somewhere where the JvmOptions are parsed, the single quote character is not allowed.

UPDATE 2

From the Procrun documentation:

++JvmOptions List of options in the form of -D or -X that will be passed to the JVM. The options are separated using either # or ; characters. If you need to embed either # or ; character put them inside single quotes. (Not used in exe mode.)

This somehow would imply that single quotes are escape characters for # and ; ... but, how to escape the single quote?

UPDATE 3 An issue was opened for ES, contains a workaround -> ES issue