Hey, for that you would need to connect the scraper to a database and add the logic which filters out the old job posts. I can't do that on my end. An example with n8n would be here, I have it running every 5 minutes: https://n8n.io/workflows/2834-automated-upwork-job-alerts-with-mongodb-and-slack/

Interesting.. would this be possible using something like Airtable as the database? Also, I use Make.com, not n8n, so, I am looking for some workarounds. Thanks

I'm not that familiar with make and airtable, but i'm quite sure you can do the same in there - this is what chatgpt told me:

Check for Existing Entries in Airtable (Equivalent to "Find Existing Entries") Use the Airtable "Search Records" module: Search for records where the Title and Budget match existing entries. Identify New Entries (Equivalent to "Merge") Use the "Filter" module to keep only records that are not found in Airtable. Store New Entries in Airtable (Equivalent to "Insert into MongoDB") Use the Airtable "Create Record" module to store new job posts.

Thanks man! So helpful. That was my thought exactly. Thanks for your help, I'm loving this actor. Great work!

Last question: is there anyway to include the number of connects required to apply? There's more details in the jobs that may be useful to scrape. Otherwise I imagine a simple HTTP get request will have to do to get all of the content on the page.

Glad to hear it :)

Required connects are only visible when logged in. What the scraper sees is the same as when you open the job search or job post in incognito mode. I do have the upwork job details scraper as well, which includes more details about the job post, but it doesn't include the required connects im afraid

What are the differences between the details scraped by both actors?

This is what the job details scraper might output: [ { "title": "Looking for a Startup Hubspot Sales Ops and CRM Manager", "paymentType": "Hourly", "paymentAmount": "$65.00 - $110.00", "description": "We’re seeking a skilled and proactive freelance ...", "jobPosted": "3 days ago", "lastViewedByClient": "yesterday", "jobType": "Remote Job", "projectType": "Complex project", "estimatedHours": "Less than 30 hrs/week", "duration": "3-6 months", "experienceLevel": "Intermediate", "clientLocation": "United States", "clientTotalSpent": "$154K", "clientTotalHires": "30", "clientActiveHires": "9", "clientHoursBilled": "8,297 hours", "clientMemberSince": "Apr 20, 2022", "proposals": "20 to 50", "interviews": "7", "invitesSent": "5", "skills": [ "HubSpot", "Marketing Automation", "Sales Operations", "Sales" ], "locationRestriction": "Only freelancers located in the U.S. may apply.", "contractToHire": true } ]

Gotcha. I'll stick with this actor because, I've sent my search URLs to have several filters so I don't need necessarily need all of that data to come back to me because I already have the assumption that the jobs that I'm scraping meet a certain criteria.

Thanks for the back and forth man. Blessings!