Sympla Event Scraper

Overview

This code is a web scraper designed to extract event data from the Sympla event platform (https://www.sympla.com.br). It's built to be used within the Apify platform, which allows for scheduled or on-demand execution of web scraping tasks. The scraper gathers event information, including details like event name, date, location, organizer, and ticket information, for a specified city.

Features

• Event Listing: Fetches lists of events for a given city, with optional filtering by state and date range.
• Date Range Filtering: Supports filtering events by a start and end date.
• Event Details: Optionally fetches detailed information for each event, including descriptions, venue details, and category.
• Ticket Information: Retrieves ticket details for events, such as prices, availability, and sale dates.
• Pagination: Handles pagination of event listings to retrieve events from multiple pages.
• Error Handling: Includes error handling for API requests and data processing.
• Axios for HTTP Requests: Uses Axios for making HTTP requests.
• Apify Integration: Designed to run on the Apify platform.
• Session Management: (Partial) Attempts to use an Axios session to handle cookie-based access to event details and ticket information.
• Input Schema: Defines the expected input parameters for the scraper.

How it Works

The scraper operates in the following general steps:

1. Initialization: The script initializes the Apify actor and retrieves input parameters, including the target city, state, date range, and other options.
2. Event List Fetching:
   • The fetchEventsByCity function is used to retrieve a list of events from Sympla's API. This function constructs a URL with the necessary parameters, including the city, state, and date. It handles pagination to get all events.
   • The d parameter is always included in the request. If a date range is provided, d includes the start and end dates (YYYY-MM-DD,YYYY-MM-DD). If no date range is provided, d defaults to today's date (YYYY-MM-DD).
3. Event Details and Ticket Fetching (Optional):
   • If the fetch_details input parameter is set to true, the scraper fetches detailed information for each event.
   • It uses fetchEventPageDetails to get event descriptions, venue information, etc.
   • It uses fetchEventTickets to get ticket information.
   • A session using Axios is initialized to try to handle cookie requirements for fetching details. If session initialization fails, it falls back to non-session requests.
4. Data Formatting:
   • The formatEventData function is used to structure the event data into a consistent format.
   • The enhanceEventWithPageDetails and enhanceEventWithTicketsOnly functions add the detailed information and ticket data to the formatted event objects.
5. Data Output:
   • The scraper pushes the formatted event data to the Apify dataset, making it available for download or further processing.

Input Parameters

The scraper accepts the following input parameters:

1{
2    "city": {
3        "title": "City",
4        "type": "string",
5        "description": "Name of the city to search for events",
6        "default": "Florianopolis",
7        "editor": "textfield"
8    },
9    "state": {
10        "title": "State",
11        "type": "string",
12        "description": "State code (e.g., SC, SP, RJ)",
13        "default": "SC",
14        "editor": "textfield"
15    },
16    "use_date_range": {
17        "title": "Use date range",
18        "type": "boolean",
19        "description": "Whether to filter events by date range",
20        "default": true,
21        "editor": "checkbox"
22    },
23    "start_date": {
24        "title": "Start date",
25        "type": "string",
26        "description": "Start date in YYYY-MM-DD format (defaults to today if date range is enabled)",
27        "editor": "textfield",
28        "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
29        "example": "2025-04-15"
30    },
31    "end_date": {
32        "title": "End date",
33        "type": "string",
34        "description": "End date in YYYY-MM-DD format (defaults to 30 days from start date if date range is enabled)",
35        "editor": "textfield",
36        "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
37        "example": "2025-04-30"
38    },
39    "fetch_details": {
40        "title": "Fetch detailed information",
41        "type": "boolean",
42        "description": "Whether to fetch detailed information and ticket data for events",
43        "default": true,
44        "editor": "checkbox"
45    },
46    "max_events_for_details": {
47        "title": "Maximum events for details",
48        "type": "integer",
49        "description": "Maximum number of events to fetch detailed information for",
50        "default": 20,
51        "minimum": 1,
52        "maximum": 100,
53        "editor": "number"
54    },
55    "max_pages": {
56        "title": "Maximum pages",
57        "type": "integer",
58        "description": "Maximum number of pages to scrape",
59        "default": 3,
60        "minimum": 1,
61        "maximum": 20,
62        "editor": "number"
63    },
64    "request_delay": {
65        "title": "Request delay (seconds)",
66        "type": "integer",
67        "description": "Delay between requests in seconds to avoid rate limiting",
68        "default": 2,
69        "minimum": 1,
70        "maximum": 10,
71        "editor": "number"
72    },
73     "detail_request_delay": {
74        "title": "Detail request delay (milliseconds)",
75        "type": "integer",
76        "description": "Delay between detail requests in milliseconds",
77        "default": 1000,
78        "minimum": 500,
79        "maximum": 5000,
80        "editor": "number"
81    }
82}
83DisclaimerAll data extracted by this scraper is publicly available on the Sympla website (https://www.sympla.com.br). This scraper only accesses and retrieves information that is already accessible to any internet user. It does not bypass any access restrictions or retrieve any non-public data.
84sageApify Account: You'll need an account on the Apify platform (https://apify.com) to run this scraper.
85Apify Actor: This code is designed to be deployed as an Apify actor. You can create a new actor on the Apify platform and upload this code.Input: Configure the actor's input parameters (city, state, date range, etc.) according to your needs.Run: Run the actor on the Apify platform. The scraped data will be stored in the actor's dataset.Output: You can then download the data from the Apify dataset in various formats (JSON, CSV, etc.).DependenciesaxiosapifyDate Parameter (d) BehaviorThe scraper constructs Sympla API requests, and a key parameter in these requests is d, which specifies the date or date range for events.  Here's how the scraper handles the d parameter:With Date Range: If you provide both start_date and end_date in the input, the scraper will include them in the d parameter in the format YYYY-MM-DD,YYYY-MM-DD (e.g., 2025-04-15,2025-04-30).With Only Start Date: If you provide only start_date, the scraper will include it in the d parameter in the format YYYY-MM-DD.Without Date Range: If you do not provide a date range (either by not setting use_date_range to true, or by not providing start/end dates), the scraper will default to including today's date in the d parameter, using the format YYYY-MM-DD.This ensures that the d parameter is *