The code I used was
import scrapy
class JobSpider(scrapy.Spider):
name = 'job'
start_urls = [
'https://jobs.goodlifefitness.com/listjobs/'
]
In the scrapy shell I put the following code for the link:
response.css('div.jobTitle a::attr(href)')
and I got a " [ ] "
On
I would highly suggest you to take a look at their backend api. You can do that using the chrome dev tools or a proxy.
This allows you to scrape more data with one request. Most of the time Backend Apis return Json objects which are very nice to work with instead of finding the data within the html file
I've found the backend api for your specific case and wrote a small script that hopefully does what you want. As you can see it scraped 408 Datapoints with one request.
import requests
import json
url = "https://jobsapi-internal.m-cloud.io/api/job?callback=jobsCallback&sortfield=open_date&sortorder=descending&Limit=408&Organization=2239&offset=1"
r = requests.get(url).text
#I know not nice but I was too lazy
r =r.replace("jobsCallback(","")
r =r.replace("}]})","}]}")
json_obj = json.loads(r)
output = list()
for job in json_obj["queryResult"]:
output.append(job["title"])
#Amount of jobs scraped
print(len(output))
#The available data of each job
print(json_obj["queryResult"][0].keys())
#All the jobs in a dictionary
print(output)
Output:
408
dict_keys(['company_name', 'clientid', 'id', 'xc_id', 'sf_id', 'entity_status', 'scout_orgid', 'scout_userid', 'scout_teamid', 'language', 'ats_portalid', 'industry', 'function', 'title', 'ref', 'primary_city', 'primary_state', 'primary_zip', 'primary_country', 'primary_address', 'primary_location', 'addtnl_locations', 'description', 'primary_category', 'addtnl_categories', 'salary', 'job_type', 'travel', 'level', 'relocation', 'education', 'years_experience', 'open_positions', 'brand', 'department', 'shift', 'recruiter', 'parent_category', 'sub_category', 'business_unit', 'is_internal', 'employment_type', 'schedule', 'compliment', 'store_id', 'close_date', 'open_date', 'fndly_url', 'url', 'seo_url', 'location_type', 'importance', 'is_child_job', 'campaign_id', 'campaign_name', 'publish_to_cws', 'hidden', 'job_classifications', 'easy_apply', 'internal_url', 'internal_description', 'multi_select1', 'multi_select2', 'erp_eligible', 'erp_bonus', 'update_date'])
['Customer Service Representative (Motivator) - Prince George River Point Landing', 'Club Attendant - Winnipeg Grant Park Shopping Centre', 'Sales (Fitness Advisor) - North York Dufferin and Finch', 'Club Attendant - North York Dufferin and Finch', 'Personal Trainer - Regina Victoria Square', 'Customer Service Representative (Motivator) - Mississauga Heartland Town Centre', 'Club Attendant - Brampton Kingspoint Plaza', 'Sales (Fitness Advisor) - Vaughan Milani and Highway 27', 'Sales (Fitness Advisor) - Calgary Richmond Square', 'Sales (Fitness Advisor) - Toronto Yonge and St Clair', 'Customer Service Representative (Motivator) - Burlington Appleby Crossing', 'Personal Trainer - North York Yonge and Finch', 'Sales and Service Representative (Fitness Coach) - Whitby Taunton and Brock For Women', 'Senior Fitness Coach - Whitby Taunton and Brock For Women', 'Club Attendant - Brampton McLaughlin Corners West', 'Personal Trainer - Windsor Tecumseh Mall', 'Fit4Less Host - Saskatoon Circle West Plaza', ...]
It is because the entire page is rendered from
javascript. Once you fetch the request, if you were to open a local file and paste the html content, you will see that 99% of the html is<script>tags. Fortunately these types of pages are easy to scrape with therequests-htmllibrary (not to be confused with therequestslibrary).For example:
pip install requests-htmlOUTPUT