Not able to scrape linked-in profile in production using Ruby Gems(Ruby on Rails application)

180 Views Asked by At

I am using the following ruby code to scrape LinkedIn public profile using following ruby gems

1) Using 'mechanize' gem


    require 'rubygems'
    require 'mechanize'
    require 'nokogiri'
    require 'open-uri'
    agent = Mechanize.new { |agent| agent.user_agent_alias = 'Mac Safari 4'}
    agent.follow_meta_refresh = true
    page = agent.get("https://www.linkedin.com/login")
    login_form = page.form(:class => 'login__form')
    login_form.session_key = "my_email_id"
    login_form.session_password = "my_password"
    page = agent.submit(login_form, login_form.buttons.first)

2) Using 'watir' gem


    require 'nokogiri'
    require 'open-uri'
    require 'webdrivers'
    require 'watir'
    browser = Watir::Browser.new :chrome, headless: true
    browser.goto 'https://www.linkedin.com/login'
    browser.input(name: 'session_key').send_keys('my_email_id', :return)
    browser.input(name: 'session_password').send_keys('my_password', :return)
    browser.html

When I tried to use this on a local machine(ubuntu), LinkedIn does not send a security code as I had already used the local chrome browser to log in to this account a previous time. so LinkedIn knows its known browser and it sends the proper response and able to scrape the details.

but when I tried these codes on production(Linux ec2 instance), LinkedIn sent the security code to my email as it does not log in to my LinkedIn account as it does not know the browser(installed google chrome & chrome driver on Linux ec2) and does not give right response ad not able to scrape it

Any approach to resolve this issue or bypass security checks as I am using the right linkedin credentials?

0

There are 0 best solutions below