Horseman - clicking multiple buttons then grabbing data - How?

126 Views Asked by At

I want to parse a web-page which could have any number of buttons. I want to click on all buttons and grab some resulting data from each button-press. I don't have any idea how to do this. My horse-man code, so far:

horse
 .on('resourceError', function(err) {
   console.dir(err);
   horse.close();
})
.userAgent('Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0')
.open('https://www.example.com')
.click('#agree')
.click('input[class="button"]')
.waitForSelector('#address')
.type('#address', 'blah blah')
.click('#button-search-address')
.wait(3000)
.evaluate(function() {
  var btns=[];
  $('[data-agency-code]').each(function(i) {
    btns.push({dac: $(this).attr('data-agency-code')});
  });
  return btns;
})
.then(?????) 

So I have all agency codes in the btns array. Now I need to go thru all buttons sort-of like this pseudo-code:

var resData=[]; 
jQuery.each(btns, function(i, val) {
  ... 
  .click('[data-agency-code]'+val.dac)
  .waitForSelector('#data-agency-data')
  .grab data like:
     resData.push({email: $('#agency-email').val(), phone: $('#agency-phone').val()});
});      

Can't figure-out the horseman code to do this loop. Thx.

1

There are 1 best solutions below

0
ImTalkingCode On

Got it working from Issue #85 on the Horseman Github page.

Here's the code to traverse all buttons on the page:

horse
  .on('resourceError', function(err) {
    console.dir(err);
    horse.close();
  })
  .userAgent('Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0')
  .open('https://www.example.com')
  .click('#agree')
  .click('input[class="button"]')
  .waitForSelector('#address')
  .type('#address', 'blah blah')
  .click('#button-search-address')
  .wait(3000)
  .evaluate(function() {
    var btns=[];
    $('[data-agency-code]').each(function(i) {
      btns.push({dac: $(this).attr('data-agency-code')});
    });
    return btns;
  })
  .then(function(btns) {
    if (btns.length == 0)
      horse.close();
    console.log(btns);
    var chain = horse;
    for(var i=0; i < btns.length; i++) {
      chain = chain
        .click('[data-agency-code='+btns[i].dac+']')
        .wait(2000)
        .evaluate(function() {
           return {
             name: $('some selector').text().trim(),
             email: $('some selector').text(),
             www: $('some selector').text() 
           }
         })
         .then(function(aobj) {
           agya.push(aobj);
         })
    }
    return chain;
  })
  .then(function(chain) {
    console.log(agya);
  })
  .close();

Now I have all agencies and info in the agya array.