Benthos: How to get variable from processor to input?

Question

i'm new to benthos, hope following configuration to work, i looked at the benthos doc and tried to google, but didn't find an answer, any answer is greatly appreciated

actually, the sign will be the calculated value, but now I'm stuck on the first step, i can't get the sign value to be successfully assigned to the header

input:
  processors:
   - bloblang: |
      meta sign = "1233312312312312"
      meta device_id = "31231312312"
  http_client:
    url: >-
      https://test/${!meta("device_id")}
    verb: GET
    headers: 
      sign: ${!meta("sign")}

after @Mihai Todor helped, now i have new question.

this config below can work.(first)

input:
  http_client:
    url: >-
      https://test/api
    verb: GET
    headers: 
      sign: "signcode"

but this one returned invalid signature error

input:
  mapping: root = {}
  generate:
    count: 1
pipeline:
  processors:
    - http:
        url: >-
          https://test/api
        verb: GET
        headers:
          sign: "signcode"
output:
  stdout: {}

update(more detail screenshot)

first

second

---

Finally i got it work with the @Mihai helped.

the reason why i got 'signure is invaild' is because a space character in paramter headers->stringToSign, for reason i need add paramter 'stringTosign' and the value need to be empty, but somehow i copy an invisible space character in it, this will make benthos to add a Content-Length: 1 to the request header(i don't know why), this Content-Length: 1 cause my request always failed with error 'signure invaild'.

after i deleted the space character, all is ok.

Answer 1

Input processors operate on the messages returned by the input, so you can't set metadata that way. Also, metadata is associated with in-flight messages and doesn't persist across messages (use an in-memory cache if you need that).

One workaround is to combine a generate input with an http processor like so:

input:
  generate:
    mapping: root = ""
    interval: 0s
    count: 1

  processors:
    - mapping: |
        meta sign = "1233312312312312"
        meta device_id = "31231312312"
        
pipeline:
  processors:
    - http:
        url: >-
          https://test/${!meta("device_id")}
        verb: GET
        headers:
          sign: ${!meta("sign")}

output:
  stdout: {}

Note that the mapping processor (the replacement for the soon-to-be deprecated bloblang one) can also reside under pipeline.processors, and, if you just need to set those metadata fields, you can also do it inside the mapping field of the generate input (root = {} is implicit).

Update: Following the comments, I ran two configs and used nc to print the full HTTP request each of them make:

1. generate input with http processor:

input:
  generate:
    mapping: root = ""
    interval: 0s
    count: 1

  processors:
    - mapping: |
        meta sign = "1233312312312312"
        meta device_id = "31231312312"

pipeline:
  processors:
    - http:
        url: >-
          http://localhost:6666/${!meta("device_id")}
        verb: GET
        headers:
          sign: ${!meta("sign")}

output:
  stdout: {}

HTTP request dump:

> nc -l localhost 6666
GET /31231312312 HTTP/1.1
Host: localhost:6666
User-Agent: Go-http-client/1.1
Sign: 1233312312312312
Accept-Encoding: gzip

2. http_client input:

input:
  http_client:
    url: >-
      http://localhost:6666/31231312312
    verb: GET
    headers:
      sign: 1233312312312312

output:
  stdout: {}

HTTP request dump:

> nc -l localhost 6666
GET /31231312312 HTTP/1.1
Host: localhost:6666
User-Agent: Go-http-client/1.1
Sign: 1233312312312312
Accept-Encoding: gzip

I used Benthos v4.5.1 on OSX and, for both configs, the request looks identical.

My best guess is that you're seeing a transitive issue on your end (some rate limiting perhaps).