# EPC Scraper (`dhrumil/epc-scraper`)

Scrape https://find-energy-certificate.service.gov.uk to crawl millions of EPCs from United Kingdom. Our energy performance certificate scraper also lets you monitor specific post code for new EPC listed. You can provide multiple search result listings to scrape/monitor.

- **URL**: https://beta-securitybyobscurity.apify.com/dhrumil/epc-scraper.md
- **Developed by:** [Dhrumil Bhankhar](https://beta-securitybyobscurity.apify.com/dhrumil) (community)
- **Categories:** Real estate, Automation
- **Stats:** 28 total users, 2 monthly users, 100.0% runs succeeded, 5 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

$30.00/month + usage

To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.
You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#rental-actors

# README

### 🏡 What is EPC Scraper?

This EPC Scraper will enable you scrape any energy performance certificate from [find-energy-certificate.service.gov.uk](https://find-energy-certificate.service.gov.uk/).

You can simply take your listing url from browser and enter it into this actor. This actor will crawl through all pages of particular listing and generate dataset for you.

Listing url is something you get when you perform the search on epc site. Example listing urls :

- https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+3JB

### 🚪 What can this EPC Scraper do?

📈 Extract EPC certificates from official UK epc site

👀 This actor is not just scraper but also has monitoring capability. You can turn on monitoring mode and it will give you only newly added epc compared to your previous scrapes.

📩  This actor also helps yu to identify which epc are expired.

⬇️ Download epc data in Excel, CSV, JSON, and other formats

### 📚 How do I start scraping with this scraper?

1. Register for your free Apify account [here](https://apify.com/pricing?fpr=z2di7)
2. You don't need to provide your credit card details for free acount. Just click on "Get Started" button on above link and complete the registration only.
3. Free account comes with reasonable credits to try out this actor. This actor also comes with free trial of 3 days without any commitment/upfront charge.
4. Run this actor and verify the scraped data. Apify has huge integration possibilities. You can download the data or push the data into any 3rd party platform directly.

### 🌳 What epc data can I extract using this tool?

| 📝                  | 📝                               |
| ------------------ | ------------------------------- |
| Rating             | Full Address                    |
| EPC URL            | Locality                        |
| Postcode           | Property Type                   |
| Floor Area         | Expiry Stats                    |
| Current Score      | Potential Score                 |
| Primary Usage      | Average Bill                    |
| Potential Saving   | Year                            |
| Produced CO2       | Potential CO2                   |
| Assesor Name       | Assesor Phone                   |
| Assesor Email      | Accreditation Details           |
| Assessment Date    | Certificate Date                |
| Assessment Type    | Certificate Expiry Date         |
| Fetaures           | Proposed Changes                |

### ⬇️ Input

For simple usecase, you just need to provide browser url of rightmove search result page & that's all. You can leave other fields as they are to be sensible defaults.

#### Input example

```json
{
    "listUrls": [
        {
            "url": "https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+3JB"
        }
    ],
    "fullScrape": true,
    "monitoringMode": false,
    "addEmptyTrackerRecord" : false
}
```

Understading monitoring mode :

- `fullScrape` : This option is by default turned on. When enabled it always force actor to scrape complete listing from all pagination pages regardless of monitoring is enabled or not.

- `monitoringMode` : This option when turned on will only scrape newly added epc compared to previously scraped epc by this actor. It's important to turn off fullScrape setting if you are using this mode. If you keep fullScrape on, it will re-scrape complete listing again.

- `addEmptyTrackerRecord` : This option when turned on will add empty record having only id of epc to Apify dataset. This helps you identify whether epc is still listed compared to your own database in incremental mode.

### ⬆️ Output

The scraped data is stored in the dataset of each run. The data can be viewed or downloaded in many popular formats, such as *JSON, CSV, Excel, XML, RSS, and HTML*.

#### Output example

The result for scraping a single property like this:

```json
{
	"url": "https://find-energy-certificate.service.gov.uk/energy-certificate/0010-2129-7282-2472-9215",
	"postCode": "BN1 3JB",
	"locality": "BRIGHTON",
	"address": "Flat 6,36 Dyke Road",
	"rating": "C",
	"id": "0010-2129-7282-2472-9215",
	"propertyType": "Mid-floor flat",
	"floorArea": "35 square metres",
	"currentScore": "75 C",
	"potentialScore": "79 C",
	"features": [
		{
			"name": "Wall",
			"description": "Solid brick, with internal insulation",
			"rating": "Good"
		},
		{
			"name": "Window",
			"description": "Partial double glazing",
			"rating": "Poor"
		}
	],
	"primaryUsage": 327,
	"averageBill": 488,
	"potentialSaving": 94,
	"averageCostYear": 2022,
	"co2Produces": 1.9,
	"co2Potential": 1.5,
	"changes": [
		{
			"name": "Heat recovery system for mixer showers",
			"installationCost": "£585 - £725",
			"yearlySaving": "£36",
			"potentialRating": "76 C"
		},
		{
			"name": "Double glazed windows",
			"installationCost": "£3,300 - £6,500",
			"yearlySaving": "£59",
			"potentialRating": "79 C"
		}
	],
	"assessorName": "Paul Cronin",
	"assessorPhone": "01273 977447",
	"assessorEmail": "paul@croninspropertychecks.com",
	"accreditationScheme": "Stroma Certification Ltd",
	"accreditationAssessorID": "STRO033856",
	"accreditationPhone": "0330 124 9660",
	"accreditationEmail": "certification@stroma.com",
	"assessorDeclaration": "No related party",
	"assessmentDate": "31 August 2022",
	"certificateDate": "6 September 2022",
	"assessmentType": "RdSAP",
	"otherCertificates": [
		{
			"id": "0576-2869-6215-9092-1251",
			"expiryDate": "16 September 2022"
		}
	],
	"validtillDate": "5 September 2032",
	"expired": false
}
```

### ❓Limitations

Since epc site allows only search by full post code, you need to provide multiple such post code list urls to cover larger targetted area. Good News is that even if multiple list urls contains overlapping results, they will get deduplicated within same run data.

### 🙋‍♀️ For custom solutions

In case you need some custom solution, you can contact me : <dhrumil@techvasu.com>

Or learn more about me on github : <https://github.com/dhrumil4u360>

# Actor input Schema

## `listUrls` (type: `array`):

Any find-energy-certificate url that has list of certificate listed

## `fullScrape` (type: `boolean`):

If checked, it will perform full scrape including all pagination pages. You might want to tunr this off in case you are running in monitoring mode.

## `monitoringMode` (type: `boolean`):

If checked, it will only scrape newly listed properties compared to what has been scraped in previous runs. Please turn off full scrape setting above in case you are tunring this on.

## `addEmptyTrackerRecord` (type: `boolean`):

If checked, it will add tracker ID record in incremental mode so that you can identify delisted properties if you have custom loader

## `email` (type: `string`):

Email address to send notifications or reports to.

## `threads` (type: `string`):

It will try to run given number of threads in parallel

## `proxy` (type: `object`):

Select proxies to be used by your crawler.

## Actor input object example

```json
{
  "listUrls": [
    {
      "url": "https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+4JB"
    }
  ],
  "fullScrape": true,
  "monitoringMode": false,
  "addEmptyTrackerRecord": false,
  "email": "",
  "threads": "3",
  "proxy": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "listUrls": [
        {
            "url": "https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+4JB"
        }
    ],
    "threads": "3",
    "proxy": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("dhrumil/epc-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "listUrls": [{ "url": "https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+4JB" }],
    "threads": "3",
    "proxy": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("dhrumil/epc-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "listUrls": [
    {
      "url": "https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+4JB"
    }
  ],
  "threads": "3",
  "proxy": {
    "useApifyProxy": true
  }
}' |
apify call dhrumil/epc-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=dhrumil/epc-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "EPC Scraper",
        "description": "Scrape https://find-energy-certificate.service.gov.uk to crawl millions of EPCs from United Kingdom. Our energy performance certificate scraper also lets you monitor specific post code for new EPC listed. You can provide multiple search result listings to scrape/monitor.",
        "version": "0.0",
        "x-build-id": "14vyR5MAHsxKSfIPu"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/dhrumil~epc-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-dhrumil-epc-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/dhrumil~epc-scraper/runs": {
            "post": {
                "operationId": "runs-sync-dhrumil-epc-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/dhrumil~epc-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-dhrumil-epc-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "listUrls"
                ],
                "properties": {
                    "listUrls": {
                        "title": "List URLs",
                        "type": "array",
                        "description": "Any find-energy-certificate url that has list of certificate listed",
                        "default": [
                            {
                                "url": "https://find-energy-certificate.service.gov.uk/find-a-certificate/search-by-postcode?postcode=BN1+4JB"
                            }
                        ],
                        "items": {
                            "type": "object",
                            "required": [
                                "url"
                            ],
                            "properties": {
                                "url": {
                                    "type": "string",
                                    "title": "URL of a web page",
                                    "format": "uri"
                                }
                            }
                        }
                    },
                    "fullScrape": {
                        "title": "Scrape all the properties from all pages available in given listing urls",
                        "type": "boolean",
                        "description": "If checked, it will perform full scrape including all pagination pages. You might want to tunr this off in case you are running in monitoring mode.",
                        "default": true
                    },
                    "monitoringMode": {
                        "title": "Run in monitoring mode where only newly listed properties compared to previous runs will be scraped",
                        "type": "boolean",
                        "description": "If checked, it will only scrape newly listed properties compared to what has been scraped in previous runs. Please turn off full scrape setting above in case you are tunring this on.",
                        "default": false
                    },
                    "addEmptyTrackerRecord": {
                        "title": "Add tracker ID record in incremental mode",
                        "type": "boolean",
                        "description": "If checked, it will add tracker ID record in incremental mode so that you can identify delisted properties if you have custom loader",
                        "default": false
                    },
                    "email": {
                        "title": "Email Address (In case of any error or notification)",
                        "type": "string",
                        "description": "Email address to send notifications or reports to.",
                        "default": ""
                    },
                    "threads": {
                        "title": "Number of parallel thread",
                        "enum": [
                            "3",
                            "5",
                            "10"
                        ],
                        "type": "string",
                        "description": "It will try to run given number of threads in parallel",
                        "default": "3"
                    },
                    "proxy": {
                        "title": "Proxy configuration",
                        "type": "object",
                        "description": "Select proxies to be used by your crawler.",
                        "default": {
                            "useApifyProxy": true
                        }
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```
