token_get_all() returns different values on same php version

Question

I'm using PhpCS to detect wrong syntax in my code. This tool use the function token_get_all however I have some inconsistency between PHP version (or even the same PHP version online)

The simplified code is :

var_dump(
    token_get_all('<? ?>')    
);

If I use PHP 7.3 on both this website the results are differents :
https://onlinephp.io/c/c30ec ==> the result is an array of 3 entries
https://3v4l.org/5Ih6Z#v7.4.3 ==> the result is an array of 1 entry

I have the same problem between a php version on docker and my ubuntu.

Does someone have any idea what is the cause of this issue ?
Thank you in advance

Answer 1

Because <? is deprecated tag since php 5.0, most PHP servers don't support them. Use <?php tag.

Answer 2

Interesting question. The magic you're looking for lies in the PHP ini setting short_open_tag(php-ini).

token_get_all(php) relies on this setting, which can not be changed while PHP is already running (as it affects how the code is being tokenized that is currently running, compare include(php), require(php) etc.).

onlinephp.io has it enabled (ref) and 3v4l.org disabled (ref).

---

Pratical Consequences

You also write how you use it:

I'm using PhpCS to detect wrong syntax in my code.

This appears to me you're interested in portable PHP code, portable means that the code robustly executes despite some PHP configuration differences like the version or ini settings.

This requires to only use <?php ... ?> (standard) or <?= ... ?> (echo variant) PHP tags¹.

In your case of <? ... ?> (short open variant, not recommended) replace the opening <? with <?php (or add php directly after it).

Compare with the note in the manual¹:

As short tags can be disabled it is recommended to only use the normal tags (<?php ?> and <?= ?>) to maximise compatibility.

In PhpCS enable the Generic.PHP.DisallowShortOpenTag sniff and execute phpcs with the short_open_tag setting enabled (otherwise PhpCS can't do the work²). IIRC the phpcbf command then also automatically fixes the places -- don't change by hand that you can let the machine with the fine PhpCs tooling do on your fingertips.

The sniff will safe-guard (phpcs) your ongoing development when you occasionally fall back into the habit to type in a short open tage here or there, the fixer (phpcbf) can quickly correct it.

---

Examples executing on the command line where you can control the setting per each invocation:

short open tag disabled (short_open_tag=0)

$ php7.4 -d short_open_tag=0 -r 'var_export(token_get_all("<? ?>")); echo "\n";'
array (
  0 => 
  array (
    0 => 313,
    1 => '<? ?>',
    2 => 1,
  ),
)

short open tag enabled (short_open_tag=1)

$ php7.4 -d short_open_tag=1 -r 'var_export(token_get_all("<? ?>")); echo "\n";'
array (
  0 => 
  array (
    0 => 382,
    1 => '<?',
    2 => 1,
  ),
  1 => 
  array (
    0 => 385,
    1 => ' ',
    2 => 1,
  ),
  2 => 
  array (
    0 => 384,
    1 => '?>',
    2 => 1,
  ),
)

---

1. PHP tags in the PHP manual
2. Generic.PHP.DisallowShortOpenTag doesn't catch <? (squizlabs/PHP_CodeSniffer #1571)
3. PHP RFC: Deprecate PHP Short open tags (by George Peter Banyard; March 2019)