Logstash sequential thread execution flow

Question

I made two jdbc sections in my config, but I want always execute first actionA and than actionB, it's important for me. How can I do this?

Here is my config:

input {
    jdbc {
        type => "actionA"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        statement => "SELECT ID FROM my_table WHERE delete='Y'"
       }

    jdbc {
        type => "actionB"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        statement => "UPDATE my_table SET delete='T' WHERE delete='Y'"
       }
}
output {
    stdout { codec => rubydebug }
    if [type] == "actionA" {
        elasticsearch {
            action => "delete"
            index => "contacts"
            document_type => "contact"
            document_id => "%{id}"
            hosts => ["http://localhost:9200"]
        }   
    }
    if [type] == "actionB" { }


}

Thanks in advance

Jay

Answer 1

You can leverage the schedule parameter and make actionA always run before actionB, e.g. by making actionA run every even minutes (0, 2, 4, 6, ...) and actionB run every odd minutes (1, 3, 5, ...)

input {
    jdbc {
        type => "actionA"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        schedule => "*/2 * * * *" # run every even minutes
        statement => "SELECT ID FROM my_table WHERE delete='Y'"
       }

    jdbc {
        type => "actionB"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        schedule => "1-59/2 * * * *" # run every odd minutes
        statement => "UPDATE my_table SET delete='T' WHERE delete='Y'"
       }
}

Answer 2

There is a trick for it.

Put your actions in separate files, so each file looks like this:

# action_X.conf
input {
    jdbc {
        # ...
    }
}
output {
    # ...
}

Then execute logstash manually for each file:

logstash -f action_a.conf
logstash -f action_b.conf

To repeat periodically, you could put the above commands in a shell script and add that script to the crontab.