🏯 Instagram Scraper (Pay Per Result)

Hello,

You can do this with our Instagram User Scraper here https://apify.com/apidojo/instagram-user-scraper

Cheers!

Apologies so to clarify, I only want to scrape 1 post, the latest post and the instagram user data, other actors can do both under the 1 actor, its a pity yours is separated into two, this is very restrictive and would be much better having one actor, giving the user the option to pick and choose what they wanted to pull.

Hello,

Unfortunately those are 2 separate things on Instagram and that's how we architectured our Instagram scrapers. This scraper is suitable to fetch posts and it also gives you some data about the user. However if you want to fetch the profile, you need to use the other one.

You can also combine the 2 of them together, that's what most people do. If other actors do both under 1 actor, feel free to use them.

I hope this helps

Cheers!

Ok thanks, can you add an option on this scraper to only scrape the last post ?

Hello,

When you scrape the posts of a user, they come in the order so the first post will be the last post of the user.

Best

I appreciate that, but I only need the latest post, and it gives me lots of data I don't use, also I assume this costs more pulling all posts. Other actors have this option.

Hello,

You can specify maxItems in the INPUT to specify how many items you want to fetch.

Thanks

thank you, I tested this and it works well, but I am not seeing the full instagram url in the excel output ? I will be doing 100's of these and we need to map the image url to the instagram account.

Hello,

You're welcome. It should be there, can you send us your run ID so we can check? You can also contact us via Discord DM to get faster support.

Thanks

https://console.apify.com/actors/culc72xb7MP3EbaeX/runs/MpgbzLtjHdLGHHC0e#output

Hello,

When I check the output I can see the full instagram URL as well as the image URLs. Can you elaborate on what is missing here?

https://api.apify.com/v2/datasets/Gd9lV0LSaUl30LmxC/items?clean=true&format=json

Thanks

The full instagram url ?

https://www.instagram.com/venusetfleur/

When we export the results we would expect the above url to be there.

Hello,

Since the url i just domain + /username, it is not there. However you can add it yourself. You can map your result objects to any object you want by using customMapFunction in the INPUT.

Thanks

Sorry I have absolutely no idea how to do that. Surely when you scrape data, the item you are using to scrape this data is exported in the excel file when its completed. You need this to map your data on import on our side. This is a very basic requirement. I cannot believe nobody has flagged this with you.

Hello,

Unfortunately that is not how our actors are working. We are providing the raw Instagram data without touching it and in the raw data that field doesn't exist.

However, we provide a way to customize this by customMapFunction. If you check the readme of the actor, you can get more information. And it is up to the users to customize this output. You can use the following input to do that:

1{
2  "customMapFunction": "(object) => {  let data = object; data.owner.fullUrl = `https://instagram.com/${data.owner.username}`; return data }",
3  "maxItems": 1,
4  "startUrls": [
5    "https://www.instagram.com/venusetfleur/"
6  ]
7}

Thanks

Huh ? I think there is some mis understanding, all we are asking for is the instagram url to be added to the export file. Every other actor we use from facebook, to instagram , to twitter shows has the url in the output excel file. This is a very normal and basic requirement when scraping data. Without this url, which is added in the input, we cannot match data on our side. You get me ?

Hello,

As I mentioned, this is not possible out of the box with our actor, you need to use customMapFunction as I sent to you. If you use it, it will add a fullUrl field to the owner with the user's full profile URL.

Best

I am attaching some screenshots for reference.

I have absolutely no interest in adding custom parameters to this. Let me know if you can fix this by adding this very fundamental functionality, if not you have just lost a customer. As I mentioned I am looking to run this daily on 100's of urls.

Hello,

Unfortunately we cannot add this to the actor since we are giving you the raw data. As I mentioned it is very easy for you to add it by giving a simple parameter to your run. If our actor is not suitable for your needs, please feel free to use other actors on the platform.

Best

Every other actor on the apify platform that I have used does this, I am very confused as to why you cannot do it.

Also the parameter you gave me see below is specific to 1 url, how does this work with 100's of urls ??

{ "customMapFunction": "(object) => { let data = object; data.owner.fullUrl = https://instagram.com/${data.owner.username}; return data }", "maxItems": 1, "startUrls": [ "https://www.instagram.com/venusetfleur/" ] }

Hello,

This function works for all objects you fetch in that run, if you provide maxItems more than 1 or you provide more URLs, you will see that all data objects will have the new parameter.

Best

Can I ask why you are being so unhelpful ? I want to use this actor, but you are being very rude and abrupt and your support is terrible, you close of the issue without any resolution, do you not want new customers !! Like many users I am not technical and your last answer was zero help to me. As I have already mentioned I want to run this daily, against 100's of instagram urls.

and to top this all off you only have 132 monthly users !! its no wonder with your support. Other actors are more expensive, hence me contacting you and they have over 1000 users per month, their support is on a different level. I ran the function you gave me, added two different instagram urls and it failed.

Hello,

As I mentioned it several times, it is not possible for us to add new fields because of technical reasons, we are really sorry about that. However, we provide a mechanism for everyone so they can add whatever field they want. Our actor is actually the most flexible and the cheapest one on the platform

We try out best to help people about it and we even sent you the customMapFunction you need to use, normally this is up to the users to come up with it.

We are really sorry that you are not happy with the actor and the support you are getting however believe me we are actually trying to help you and doing our best on a Saturday night. When I said you to refer to the other actors, it was actually for your own good. Since we cannot add it and if you really don't want to use the customMapFunction, there is no other way around with our actor.

I hope this clear things out.

Best

Hello again,

I also checked your recent failed run. Seems like you are using the field in a wrong way. Can you please use it like the following?

1{
2  "customMapFunction": "(object) => {  let data = object; data.owner.fullUrl = `https://instagram.com/${data.owner.username}`; return data }",
3  "maxItems": 2,
4  "startUrls": [
5    "https://www.instagram.com/venusetfleur/",
6    "https://www.instagram.com/brooksrunning/"
7  ]
8}

If you are using the visual editor, you can copy paste the following:

(object) => {  let data = object; data.owner.fullUrl = `https://instagram.com/${data.owner.username}`; return data }

Attaching a screenshot as well.

Please don't forget to change the maxItems to something else than 1, or you will get a single post

Best

This is now my 13th message to you, so sorry if you think you are providing great support and I do appreciate its a Sat.

"As I mentioned it several times, it is not possible for us to add new fields because of technical reasons"

This is complete and utter rubbish, you cannot add new fields ? Seriously ! that is not true. I have been chatting to other developers, 1 in particular regarding their facebook actor, and I have been testing it with them and helping them improve their actor.

You need to ask yourself the question, if yours is the cheapest then why are so few user, using it ?? I am pretty sure the answer is the functionality.

What I am asking for as you know, is that the URL you input is added to the Output or excel file you export. You NEED this to map your data. I can absolutely guarantee you, users have not chosen your actor as you do not offer this very needed functionality.

I am not trying to pi** you off here, I am try to give you constructive feedback to improve your actor.

Thank you for sending on, so I used the 2nd one and i think that worked, but I added two different urls and its only doing 1.

Is this because I selected 1 under

Maximum number of items on output (optional)

I thought this was to restrict 1 post per url ?

As an aside the text "Maximum number of items on output" is very vague, it would be worthwhile explaining this to users more.

PS I appreciate its a Sat night, you can come back to me on Monday of course

Hello,

Thanks a lot for your feedback, we will definitely keep these in mind. Also please keep in mind that all actors are built differently. We can customize most stuff in some of them and very limited in others like this one. However we try our best to add some workarounds so our users can leverage them,

If you need to limit for every URL you provide, unfortunately you cannot do it, you need to create several runs for each of them. maxItems input applies to the whole run.

Our actors are mostly targeted to programmatically use via the API and most of our users are running the actors that way to fetch millions of results every day. They are mapping this easily in their end programmatically and this actor is actually optimized for that use case. I can understand the issue you are having but believe me customMapFunction is our solution to this. Our users are free to add whatever field they want with this way, there is no limitation on that.

Let me know if you have more questions.

Best

Hello,

Closing task due to inactivity. Please feel free to re-open in case you have more questions.

Cheers

Hi

Sorry, I was busy finishing off the facebook import, I would really like to get this working. We have resolved the issue with the adding the Instagram url to the output file, but can you tell me how I can only scrape the latest post per profile. Once I get this working I can test it and then create scheduled tasks for all the urls. As I mentioned depending on the cost, we will be doing it daily, and maybe even twice daily.

Kind Regards

Scott

Hello Scott,

As I mentioned, you can set the maxItems to one if you want to get a single result. However if you are going to batch the queries into a single run, it won't work since the limitation works for the whole run. For instance if you add userA, userB, userC and set maxItems as 3, it will most probably fetch you 3 items from userA.

If you want to limit this for every profile, you need to create runs separately for each of them since the scraper doesn't support limitations per query.

Best

When you say

userA, userB, userC

Do you mean 3 different Instagram profiles ?

If so you are saying you can only scrape 1 profile at a time ? and limit it to 1 post, with the filter "Maximum number of items on output (optional)"

surely this is very much limiting your user base and potential customers. Many many people out there want to scrape more than 1 profile. I will be doing 100's of profiles a day.

If this is not something that is working now, can you add it as a feature in the future.

I currently have around 20 saved tasks, each with around 20-50 instagram urls in them. This will grow in the future.

Kind Regards

Scott

This actor has this functionality

https://console.apify.com/actors/shu8hvrXbJbY3Eb9W/input

Max Items

How many posts or comments (max 50 comments per post) you want to scrape from each Instagram URL. If you set this to 1, you will get a single post from each page.

Hello,

Yes you are correct, if you want to scrape several profiles at the same time, it is not possible to limit profile by profile, you can only limit the count per run.

As I mentioned before, since most of our customers are using our scraper programatically, it is not an issue however if you are using Apify Dashboard, it can be a bit of an extra work, I can understand that.

I will add this to our feature board, however I am not sure when we can deliver this, but will definitely keep you in the loop.

Cheers

You mentioned

"As I mentioned before, since most of our customers are using our scraper programatically, it is not an issue "

So they can do this ? if so let me know how and I can pass the details onto my developer, I am assuming though this does not stop the apify run from pulling all the data, if so then its still an issue as its costing more then it should.

Hello,

You can use Apify API/SDKs to trigger the runs separately for each of the user and set a maxItem for each of your run.

You can get more information at: https://docs.apify.com/api

Cheers!

Can you give me some more information on this. There are dozens of pages in this documentation.

In particular limiting the output to post per instagram account.

Can you send me on the link where this is shown.

Hello,

This is general information about Apify API, there is no specific information about how you can use any actor. What you need to do is to create a single run for each of the instagram profiles, setting maxItems as 1. That way, all your runs will have a single output (latest post) of that user. Your developer can do this programmatically very easily.

Does that make sense?

Thanks