PGVector Integration avatar
PGVector Integration

Pricing

Pay per usage

Go to Store
PGVector Integration

PGVector Integration

Developed by

Apify

Maintained by Apify

This integration transfers data from Apify Actors to a Postgres SQL database (with PGVector extension).

4.6 (5)

Pricing

Pay per usage

4

Monthly users

2

Runs succeeded

33%

Response time

19 hours

Last modified

5 days ago

SW

Unable to connect

Open

subadhra-wo opened this issue
6 days ago

I am using neon db and have added the pgvector extension and also shared a connection string that works. I am able to see the following tables in my db i.e. langchain_pg_collection and langchain_pg_embedding I have connected the actor with a dataset i have produced using apify web crawler But I am not able to run the actor

jiri.spilka avatar

Hi,

Thank you for using the PGVector integration!

I tested it with Neon DB—even though I didn’t have prior experience with it, it looks really good.

I’m not sure what went wrong in your run. I tried it myself and it worked — you can see my run here.

Could you please try again?
Maybe start with a smaller dataset just to test.

Also, please double-check that your Postgres settings are correct. Unfortunately, it’s hard to debug on my end since I can’t reproduce the issue :(

Thank you, Jiri

SW

subadhra-wo

5 days ago

Hi,

I just tried it again. No luck.

Failed to connect/get database. Please ensure the following: 1. Database credentials are correct and the database is configure properly. 2. The vector dimension of your embedding model in the Actor input (Embedding settings -> model) matches the one set up in the database. Database error message: Can't load plugin: sqlalchemy.dialects:postgres

My database credentials are correct. Do I need to do anything to configure the database properly? I was assuming that the actor would create the tables needed etc. If not, are there any instructions here? Since I did not create the table, I do not know what the vector dimensions that I need to set are. Can you provide some guidance here.

Thanks

jiri.spilka avatar

Hi, I’m really sorry to hear that you're running into this issue.

I haven’t encountered the exact same error myself:

| Database error message: Can't load plugin: sqlalchemy.dialects:postgres

However, I found a relevant Stack Overflow post that explains the issue. According to it:

| "SQLAlchemy 1.4 removed the deprecated postgres dialect name. You must now use postgresql instead."

Could it be that your postgresSqlConnectionStr starts with postgres:// instead of postgresql://? For reference, my connection string looks like this: postgresql://neondb_owner:npg_****

Let me know if updating the prefix solves it. Jiri

SW

subadhra-wo

4 days ago

Hi,

Thanks for getting back.

After a few attempts (using postgresql and removing the config as suggested in your docs from my embeddings config) I finally got it to work but I don't know what embedding is being used by default and i also do not know what the vector dimensions in the table are. I looked at the docs but was unable to figure it out.

Here is the JSON of my config (which worked) I've snipped the sensitive info

1{
2    "dataUpdatesPrimaryDatasetFields": [
3        "url"
4    ],
5    "dataUpdatesStrategy": "deltaUpdates",
6    "datasetFields": [
7        "text"
8    ],
9    "datasetId": "udz1WeiXraNvbrxMq",
10    "deleteExpiredObjects": true,
11    "deltaUpdatesPrimaryDatasetFields": [
12        "url"
13    ],
14    "embeddingsApiKey": "ENCRYPTED_VALUE:ZjR...snipped",
15    "embeddingsProvider": "OpenAI",
16    "enableDeltaUpdates": true,
17    "performChunking": true,
18    "postgresCollectionName": "apify-collection",
19    "postgresSqlConnectionStr": "ENCRYPTED_VALUE:U5Oy...snipped",
20}

If you could help me understand the following i.e.

  • what embedding model is being used by default (I need this so I can get embeddings of the user query using the same model)
  • what are the vector dimensions that were used by default in the table created (I need this so I can recreate this table as needed)

Thanks

jiri.spilka avatar

Hi,

I’m glad to hear it’s working now! I agree that the configuration could definitely benefit from some simplification.

Based on the information you shared, it looks like you're using OpenAI embeddings without specifying a model.
By default, the model used is text-embedding-3-small, but you can explicitly set it using the embeddingsConfig field:

1"embeddingsConfig": {
2  "model": "text-embedding-3-small"
3}

The vector dimension for text-embedding-3-small is 1536. Jiri

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.