# Job Listings Aggregator Pro (`assertive_analogy/job-listings-aggregator`)

Job Listings Aggregator – Find Jobs Fast!
Search 8+ top job boards (LinkedIn, Indeed, RemoteOK, Dice, more) in one click. Get Python, tech & remote roles with smart deduplication, keyword filters & instant results. Supercharge your job hunt with this powerful, all-in-one Python scraper!

- **URL**: https://beta-securitybyobscurity.apify.com/assertive\_analogy/job-listings-aggregator.md
- **Developed by:** [Gideon Nesh](https://beta-securitybyobscurity.apify.com/assertive_analogy) (community)
- **Categories:** Jobs, Automation, Developer tools
- **Stats:** 12 total users, 7 monthly users, 100.0% runs succeeded, NaN bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00 / 1,000 results

This Actor is paid per result. You are not charged for the Apify platform usage, but only a fixed price for dataset items the Actor outputs.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-result

# README

## Job Listings Aggregator

A powerful Python application that crawls and aggregates job listings from multiple public job boards, providing a unified interface to search, filter, and manage job opportunities.

### Features

#### Core Features

- **Multi-source scraping**: Fetch jobs from RemoteOK, We Work Remotely, and easily extensible to other job boards
- **Intelligent data processing**: Automatic normalization and deduplication of job listings
- **Flexible storage**: Support for both JSON files and SQLite database storage
- **Advanced filtering**: Filter by keywords, location, job type, company, and more
- **Export capabilities**: Export results to CSV or JSON formats
- **CLI interface**: Easy-to-use command-line interface

#### Advanced Features

- **Automated scheduling**: Set up daily automated scraping
- **Data normalization**: Consistent formatting across different job sources
- **Duplicate detection**: Smart deduplication based on job similarity
- **Comprehensive logging**: Detailed logging for monitoring and debugging
- **Modular architecture**: Easy to extend with new job boards

### Installation

1. **Clone the repository:**

```bash
git clone <repository-url>
cd job-listings-aggregator
```

2. **Install dependencies:**

```bash
pip install -r requirements.txt
```

3. **Install Playwright browsers (optional, for JS-heavy sites):**

```bash
playwright install
```

### Configuration

The application can be configured by modifying `config.py`:

```python
## Storage settings
STORAGE_TYPE = 'json'  ## or 'sqlite'

## Scraping settings
MAX_JOBS_PER_BOARD = 100
REQUEST_DELAY = 1  ## seconds between requests

## Scheduler settings
SCHEDULE_TIME = '09:00'  ## Daily scraping time
ENABLE_SCHEDULER = True
```

### Usage

#### Command Line Interface

##### 1. Scrape Jobs

```bash
## Scrape all jobs from enabled job boards
python main.py scrape

## Scrape with specific keywords
python main.py scrape -k python -k developer -k remote

## Limit jobs per board
python main.py scrape -m 50
```

##### 2. Search Jobs

```bash
## Search by keyword
python main.py search -k python

## Search for remote jobs
python main.py search --remote

## Search by location
python main.py search -l "San Francisco"

## Complex search with multiple filters
python main.py search -k "data scientist" -l remote --company google

## Export search results
python main.py search -k python --export results.csv
python main.py search --remote --export jobs.json
```

##### 3. View Statistics

```bash
## Show comprehensive statistics
python main.py stats
```

##### 4. Manage Data

```bash
## Clear all saved jobs
python main.py clear
```

##### 5. Scheduler Management

```bash
## Start automated daily scraping
python main.py schedule --start

## Stop the scheduler
python main.py schedule --stop

## Check scheduler status
python main.py schedule --status

## Run scraping immediately
python main.py schedule --run-now
```

#### Python API

You can also use the application programmatically:

```python
from main import JobAggregator

## Initialize the aggregator
app = JobAggregator()

## Scrape jobs with keywords
jobs = app.scrape_all_jobs(keywords=['python', 'remote'])

## Process and save jobs
processed_jobs = app.process_jobs(jobs)
app.save_jobs(processed_jobs)

## Search saved jobs
remote_jobs = app.search_jobs(remote_only=True, keyword='developer')

## Print job details
for job in remote_jobs[:5]:
    print(f"{job.title} at {job.company}")
    print(f"Location: {job.location}")
    print(f"Link: {job.application_link}")
    print("---")
```

### Architecture

#### Project Structure

```
job-listings-aggregator/
├── main.py                 ## CLI interface and main application
├── config.py              ## Configuration settings
├── requirements.txt       ## Python dependencies
├── models/
│   ├── __init__.py
│   └── job_listing.py     ## Job data model
├── scrapers/
│   ├── __init__.py
│   ├── base_scraper.py    ## Abstract scraper base class
│   ├── remoteok_scraper.py
│   └── weworkremotely_scraper.py
├── storage/
│   ├── __init__.py
│   ├── base_storage.py    ## Abstract storage interface
│   ├── json_storage.py    ## JSON file storage
│   └── sqlite_storage.py  ## SQLite database storage
├── utils/
│   ├── __init__.py
│   ├── normalizer.py      ## Data normalization utilities
│   └── deduplicator.py    ## Duplicate detection
├── filters/
│   ├── __init__.py
│   └── job_filters.py     ## Advanced filtering
├── scheduler/
│   ├── __init__.py
│   └── job_scheduler.py   ## Automated scheduling
└── data/                  ## Generated data directory
    ├── job_listings.json
    └── job_listings.db
```

#### Data Model

Each job listing contains:

- **title**: Job title
- **company**: Company name
- **location**: Job location
- **job\_type**: Employment type (Full-time, Remote, etc.)
- **date\_posted**: When the job was posted
- **description**: Job description snippet
- **application\_link**: URL to apply
- **source**: Which job board it came from
- **id**: Unique identifier
- **scraped\_at**: When it was scraped

### Adding New Job Boards

To add a new job board, create a new scraper class:

```python
## scrapers/newboard_scraper.py
from .base_scraper import BaseScraper
from models.job_listing import JobListing

class NewBoardScraper(BaseScraper):
    def __init__(self):
        super().__init__('newboard')
    
    def scrape_jobs(self, keywords=None, max_jobs=None):
        ## Implement scraping logic
        jobs = []
        ## ... scraping code ...
        return jobs
    
    def parse_job_element(self, job_element):
        ## Implement parsing logic for individual job elements
        ## Return JobListing object
        pass
```

Then update `config.py` and the main application to include the new scraper.

### Filtering Options

The application supports various filtering criteria:

- **keyword**: Match in title, description, or job type
- **keywords**: Multiple keywords (match any or all)
- **location**: Match job location
- **remote\_only**: Filter for remote jobs only
- **source**: Filter by job board source
- **company**: Filter by company name
- **job\_type**: Filter by employment type
- **date\_range**: Jobs posted within N days
- **exclude\_keywords**: Exclude jobs with certain keywords
- **min\_description\_length**: Minimum description length
- **custom\_filter**: Apply custom filter function

### Storage Options

#### JSON Storage (Default)

- Human-readable format
- Easy to inspect and edit
- Good for smaller datasets
- Portable across systems

#### SQLite Storage

- Better performance for large datasets
- SQL query capabilities
- ACID compliance
- Indexing for faster searches

Switch between storage types by setting `STORAGE_TYPE` in `config.py`.

### Logging

The application provides comprehensive logging:

- Console output for user feedback
- File logging to `job_aggregator.log`
- Different log levels for components
- Structured logging for monitoring

### Best Practices

#### Respectful Scraping

- Built-in delays between requests
- Retry logic with exponential backoff
- User-Agent headers
- Respect robots.txt (implement if needed)

#### Data Quality

- Automatic data normalization
- Duplicate detection and removal
- Input validation and cleaning
- Error handling and logging

#### Performance

- Configurable limits on jobs per board
- Efficient storage options
- Background scheduling
- Memory-conscious processing

### Troubleshooting

#### Common Issues

1. **No jobs found**: Check if job boards have changed their HTML structure
2. **Connection errors**: Verify internet connection and site availability
3. **Import errors**: Ensure all dependencies are installed
4. **Permission errors**: Check file/directory permissions for data storage

#### Debug Mode

Run with verbose logging:

```bash
python main.py -v scrape
```

#### Logs

Check the log file for detailed error information:

```bash
tail -f job_aggregator.log
```

### Contributing

1. Fork the repository
2. Create a feature branch
3. Add new scrapers, filters, or storage backends
4. Include tests for new functionality
5. Submit a pull request

### License

This project is open source. Please use responsibly and respect job board terms of service.

### Disclaimer

This tool is for educational and personal use. Always check and comply with the terms of service of the job boards you're scraping. Be respectful with request rates and consider using official APIs when available.

# Actor input Schema

## `keywords` (type: `array`):

Job search keywords (e.g., \['python', 'developer', 'remote'])

## `location` (type: `string`):

Filter jobs by location (optional)

## `maxJobsPerBoard` (type: `integer`):

Maximum number of jobs to scrape from each job board

## `sources` (type: `array`):

Which job boards to scrape from

## `remoteOnly` (type: `boolean`):

Filter for remote jobs only

## `excludeKeywords` (type: `array`):

Keywords to exclude from results (optional)

## `minDescriptionLength` (type: `integer`):

Minimum length of job description (0 = no limit)

## `outputFormat` (type: `string`):

Format for the output data

## Actor input object example

```json
{
  "keywords": [
    "python",
    "developer"
  ],
  "location": "San Francisco, CA",
  "maxJobsPerBoard": 50,
  "sources": [
    "remoteok",
    "linkedin",
    "ziprecruiter",
    "monster"
  ],
  "remoteOnly": false,
  "excludeKeywords": [],
  "minDescriptionLength": 0,
  "outputFormat": "json"
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("assertive_analogy/job-listings-aggregator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("assertive_analogy/job-listings-aggregator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call assertive_analogy/job-listings-aggregator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=assertive_analogy/job-listings-aggregator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Job Listings Aggregator Pro",
        "description": "Job Listings Aggregator – Find Jobs Fast!\nSearch 8+ top job boards (LinkedIn, Indeed, RemoteOK, Dice, more) in one click. Get Python, tech & remote roles with smart deduplication, keyword filters & instant results. Supercharge your job hunt with this powerful, all-in-one Python scraper!",
        "version": "0.0",
        "x-build-id": "RacdefRVipQcooWGe"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/assertive_analogy~job-listings-aggregator/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-assertive_analogy-job-listings-aggregator",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/assertive_analogy~job-listings-aggregator/runs": {
            "post": {
                "operationId": "runs-sync-assertive_analogy-job-listings-aggregator",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/assertive_analogy~job-listings-aggregator/run-sync": {
            "post": {
                "operationId": "run-sync-assertive_analogy-job-listings-aggregator",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "keywords",
                    "sources"
                ],
                "properties": {
                    "keywords": {
                        "title": "Search Keywords",
                        "minItems": 1,
                        "type": "array",
                        "description": "Job search keywords (e.g., ['python', 'developer', 'remote'])",
                        "items": {
                            "type": "string"
                        },
                        "default": [
                            "python",
                            "developer"
                        ]
                    },
                    "location": {
                        "title": "Location Filter",
                        "type": "string",
                        "description": "Filter jobs by location (optional)",
                        "default": ""
                    },
                    "maxJobsPerBoard": {
                        "title": "Max Jobs Per Board",
                        "minimum": 1,
                        "maximum": 200,
                        "type": "integer",
                        "description": "Maximum number of jobs to scrape from each job board",
                        "default": 50
                    },
                    "sources": {
                        "title": "Job Boards",
                        "uniqueItems": true,
                        "type": "array",
                        "description": "Which job boards to scrape from",
                        "items": {
                            "type": "string",
                            "enum": [
                                "remoteok",
                                "weworkremotely",
                                "linkedin",
                                "glassdoor",
                                "indeed",
                                "dice",
                                "ziprecruiter",
                                "monster"
                            ]
                        },
                        "default": [
                            "remoteok",
                            "linkedin",
                            "ziprecruiter",
                            "monster"
                        ]
                    },
                    "remoteOnly": {
                        "title": "Remote Jobs Only",
                        "type": "boolean",
                        "description": "Filter for remote jobs only",
                        "default": false
                    },
                    "excludeKeywords": {
                        "title": "Exclude Keywords",
                        "type": "array",
                        "description": "Keywords to exclude from results (optional)",
                        "items": {
                            "type": "string"
                        },
                        "default": []
                    },
                    "minDescriptionLength": {
                        "title": "Minimum Description Length",
                        "minimum": 0,
                        "type": "integer",
                        "description": "Minimum length of job description (0 = no limit)",
                        "default": 0
                    },
                    "outputFormat": {
                        "title": "Output Format",
                        "enum": [
                            "json",
                            "csv"
                        ],
                        "type": "string",
                        "description": "Format for the output data",
                        "default": "json"
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```
