Boost Spirit X3: What is the difference between `_val` and `_attr`?

Question

I am looking into Boost Spirit X3, but am struggling to fully grasp the difference between x3::_val and x3::_attr as used in semantic actions to extract attributes from the passed context. The official docs state

_val: A reference to the attribute of the innermost rule that directly or indirectly invokes the parser p
_attr: A reference to the attribute of the parser p

which doesn't really help me. I researched some and stumbled upon https://stackoverflow.com/a/61703199/3907364, which states

x3::_val(ctx) is like qi::_val
x3::_attr(ctx) is like qi::_0 (or qi::_1 for simple parsers)

but unfortunately, I was unsuccessful at figuring out what qi::_val and qi::_0/qi::_1 are - or more precisely what their difference is.

Finally, I also found https://stackoverflow.com/a/53971888/3907364, where it is stated that

What they [x3::_val and x3::_attr] will be in a particular situation depends on make_attribute/transform_attribute traits. By default they will reference to the same value until you have nested rules with different attribute types

which seems to conform with experiments that I have conducted thus far - except I have not yet managed for _attr and _val to yield different values.

Even though I was unable to understand their difference yet, it seems rather relevant as all examples that I have seen, that use semantic actions to e.g. compute the result of a given calculation (see e.g. here) always seem to use _attr as a more global state, whereas _val seems to be the immediate state of the thing that has just been parsed. E.g.

[](auto& ctx) { _val(ctx) = _val(ctx) * _attr(ctx); }

Yet with all of this, I am still not quite able to point my finger to the exact difference in semantics between the two properties. Could someone perhaps try to rephrase Boost's docs and give an example of where the difference is actually important/visible?

Answer 1

_val: A reference to the attribute of the innermost rule that directly or indirectly invokes the parser p
_attr: A reference to the attribute of the parser p

Since this didn't help you, let's illustrate.

I like to call _attr the (accessor for) the synthesized attribute:

Parsers and generators in Spirit are fully attributed. Spirit.Qi parsers always expose an attribute specific to their type. This is called synthesized attribute as it is returned from a successful match representing the matched input sequence.

For instance, numeric parsers, such as int_ or double_, return the int or double value converted from the matched input sequence. Other primitive parser components have other intuitive attribute types, such as for instance int_ which has int, or ascii::char_ which has char

On the other hand, there is the attribute bound to the rule when invoking a parser, the _val(ctx). This depends on what context you use it in.

Here's a demonstration program illustrating the differences:

Live On Coliru

#include <boost/fusion/adapted.hpp>
#include <boost/fusion/include/as_vector.hpp>
#include <boost/fusion/include/io.hpp>
#include <boost/spirit/home/x3.hpp>
#include <iostream>
using boost::core::demangle;
namespace x3 = boost::spirit::x3;

namespace Parser {
    using boost::fusion::at_c;

    auto action = [](auto& ctx) {
        using Synthesized = decltype(_attr(ctx));
        using Bound       = decltype(_val(ctx));

        std::cout << "synthesized: " << demangle(typeid(Synthesized).name()) << "\n";
        std::cout << "bound:       " << demangle(typeid(Bound).name()) << "\n";

        if constexpr (!std::is_same_v<x3::unused_type, std::decay_t<Bound>>) {
            at_c<0>(_val(ctx)) = at_c<0>(_attr(ctx));
            at_c<1>(_val(ctx)) = at_c<1>(_attr(ctx));
        }
    };

    auto expr = (x3::int_ >> x3::int_)[action];
    template <typename T> auto rule = x3::rule<struct Tag, T>{"rule"} = expr;
} // namespace Parser

template <typename T = x3::unused_type> void test() {
    static constexpr std::string_view input = "123 234";

    if constexpr (std::is_same_v<x3::unused_type, T>) {
        phrase_parse(begin(input), end(input), Parser::rule<T>, x3::space);
        std::cout << " -> (no attribute)\n\n";
    } else {
        T attr;
        phrase_parse(begin(input), end(input), Parser::rule<T>, x3::space, attr);

        std::cout << " -> " << boost::fusion::as_vector(attr) << "\n\n";
    }
}

int main() {
    test();
    test<boost::fusion::deque<int, int>>();
    test<std::pair<int, int>>();
    test<std::tuple<int, int>>();
}

Printing:

synthesized: boost::fusion::deque<int, int>
bound:       boost::spirit::x3::unused_type
 -> (no attribute)

synthesized: boost::fusion::deque<int, int>
bound:       boost::fusion::deque<int, int>
 -> (123 234)

synthesized: boost::fusion::deque<int, int>
bound:       std::pair<int, int>
 -> (123 234)

synthesized: boost::fusion::deque<int, int>
bound:       std::tuple<int, int>
 -> (123 234)

Note that the action is completely redundant here, just used to add output:

Live On Coliru

template <typename T = x3::unused_type> void test() {
    static constexpr std::string_view input = "123 234";

    T attr;
    phrase_parse(begin(input), end(input), x3::int_ >> x3::int_, x3::space, attr);

    if constexpr (std::is_same_v<x3::unused_type, T>)
        std::cout << " -> (no attribute)\n";
    else
        std::cout << " -> " << boost::fusion::as_vector(attr) << "\n";
}

int main() {
    test();
    test<boost::fusion::deque<int, int>>();
    test<std::pair<int, int>>();
    test<std::tuple<int, int>>();
}

Still printing

 -> (no attribute)
 -> (123 234)
 -> (123 234)
 -> (123 234)

Answer 2

TL;DR: _val(ctx) yields the attribute of the rule the parser is contained in (if any), whereas _attr(ctx) yields the attribute of the parser itself. The job of a semantic action (usually) is to propagate

_val(ctx) <- _attr(ctx)

Note: _attr(ctx) will already have been set by the parser itself while/after having parsed the corresponding part in the input.

---

After some more trials and research, I think I have found the answer. It turns out that the official docs are (of course) correct, albeit they have phrased it somewhat more complicated than (I think) it needs to be.

First of all: An example, where the difference between _val and _attr becomes apparent (Godbolt):

#include <boost/spirit/home/x3.hpp>

#include <iostream>
#include <string>

namespace x3 = boost::spirit::x3;

namespace Parser {

auto transform = [](auto &ctx) {
  std::cout << "Type of _val: " << typeid(x3::_val(ctx)).name()
            << "\nType of _attr: " << typeid(x3::_attr(ctx)).name() << "\n";
  x3::_val(ctx) = std::to_string(x3::_attr(ctx));
};

x3::rule<struct literal, std::string> literal{"literal"};

auto literal_def = x3::int_[transform];

BOOST_SPIRIT_DEFINE(literal);

} // namespace Parser

int main() {
  std::string output;
  std::string input = "25";
  x3::phrase_parse(input.begin(), input.end(), Parser::literal, x3::space, output);
  std::cout << "Output: " << output << "\n";
}

Here, we define a rule literal that has a std::string attribute. The rule consists of only a single parser: an x3::int_ parser, whose attribute is of type int.

If we omit the transform semantic action, this snippet will fail to compile since automatic attribute propagation will try to convert the attribute of a rule's parser(s) to the rule's attribute(s) (and int can't implicitly be converted to std::string).

Inside the transform action, we explicitly print the type of _val and _attr, which yields std::string and int respectively. Therefore, we can conclude that _attr(ctx) yields a reference to the attribute of the parser that has parsed the given portion of the input (aka: the parser we added the action to), whereas _val(ctx) yields a reference to the attribute of the rule this parser belongs to (if the corresponding rule doesn't have an attribute or there is no such rule to begin with, this will yield x3::unused_parameter).

Overall, our transform action simply converts the parser's attribute to the rule's one, making everything work.

Without explicitly specifying a semantic action, this conversion is done automatically, as can be seen in this example (Godbolt:

#include <boost/spirit/home/x3.hpp>

#include <iostream>
#include <string>

namespace x3 = boost::spirit::x3;

namespace Parser {

auto dummy = [](auto &ctx) {
};

x3::rule<struct literal, int> literal{"literal"};

auto literal_def = x3::int_;

BOOST_SPIRIT_DEFINE(literal);

} // namespace Parser

int main() {
  int output;
  std::string input = "25";
  x3::phrase_parse(input.begin(), input.end(), Parser::literal, x3::space, output);
  std::cout << "Output: " << output << "\n";
}

However, as soon as we insert a semantic action, this has to be done explicitly, which can be observed by adding the dummy action to the int_ parser like so:

x3::int_[dummy]

While the output variable was set to 25 (as expected) before, it now remains at its default-initialized value of 0, because the parser's attribute has never been propagated to the rule's attribute.

Answer 3

@Raven,

Examining the actual code shows _val and _attr are actually created in 3 lines starting in call.hpp here. That's called from action.hpp here, where it can be seen that attr comes from being synthesized by this->subject here.

But, you might ask, where does rcontext, the value used to make _val come from? Well, you have to look further up the call chain to detail/rule.hpp in call_rule_definition where the rcontext("rule context") is assigned the same value as the "transformed" attribute, attr_. Now, attr_ USUALLY will be the same as the actual rule attribute, attr depending on what the "customization point", transform::pre does. The default transform::pre does make attr_ the same as attr. But is attr actually the rule's attribute? That, again, frustratingly, depends on from where call_rule_definition is called.

There are 2 places where call_rule_definition is called from:

1. directly from rule_definition::parse.

2. indirectly, from rule::parse via the generated parse_rule which directly calls call_rule_definition.

   Now, what may puzzle you, after careful code study, is why transform_attribute pre and post are called twice in this case:

   1. in rule::parse directly

   2. indirectly in call_rule_defintion

      That call is made indirectly, as previously mentioned, through the generated parse_rule.

   Now, @Raven, you might ask yourself, "why not just move the transform_attribute code from call_rule_definition to rule_definition::parse and thereby eliminate the duplicate transform_attribute?" I've repeatedly asked myself that question and can think of no downside. @sehe, can you think of some reason why this change would not be good?