494 comments found.
Hi,
Is this license also treated as Aiomatic?
Hello,
No, only Aimogen (formerly Aiomatic) is a plugin which requires weekly updates and maintenance (and also gets new features added weekly), because of this it requires a large amount of work.
Crawlomatic (and all the other plugins I created) belongs a different category plugin, which does not need frequent maintenance and updates.
I hope you understand the reasoning.
Regards, Szabi – CodeRevolution.
1)Can I bulk import products details including pictures, descriptions, attributes just by giving shop url and selecting the next button or infinite scrolling? 2)Does this plugin require a connection to your servers to work? 3)Does it have a protection so that ou IP is not not banned? 4)Where are the new tutorials, all Youtube tutorials are like 4 years or 6 years old.
Hello,
1. Yes this is possible, but it requires some setup from your part, to pinpoint in pages each detail like description, attributes and others, so the plugin knows which info to insert in which part. Please check this tutorial for an example: https://www.youtube.com/watch?v=MJaEFHy5bxQ
2. No connection required.
3. You can add proxies in the plugin settings, which will rotate the IP address the plugin uses. If you don’t add proxies, your server’s IP address will be used.
4. I will create new tutorials also, it is planned.
Regards, Szabi – CodeRevolution
1. I saw you YT video, but when you imported attributes you’ve added them to the description, what I need is to match them with the actual attributes that I have on my Store. 2. Is it possible to download the results as a CSV file instead of injecting them directly to Woo products?
If you mean attribute/variation scraping, this is also possible: https://www.youtube.com/watch?v=FPAnbXU8rRI
Also, scraping to csv is not possible, only to post/woo products.
Regards.
No I mean Attributes, for example for a phone attributes would be Storage, RAM, Color, Screen size, and so one, if you go to Woo Products menu, you’ll find attributes. Are you planning on adding export to csv since this will solve this problem.
Understood. Please send me the url you want to scrape, to my email kisded@yahoo.com Also, noted the csv idea.
check your email
still waiting for your feedback. Thanks
I replied to your email, please check.
I just replied to your email, please check.
Is there a stand alone version of this? I would like to run it on a php site I have
Hello,
Sorry, but a stand alone version of Crawlomatic is not yet available, but I noted this idea, will think on it.
Regards, Szabi – CodeRevolution
Can websites that host audio files, such as MP3s and podcasts, be scraped to extract and share their audio links?
Hello,
Yes, this is possible, you can scrape data from sites and create the same posts on your site also. However, the mp3 files will not be copied to your site, they will be played from their original location. If you need also the mp3 files copied locally, I can do a paid custom update for this. For details, please contact me using my email: kisded@yahoo.com
Regards, Szabi – CodeRevolution
Hi
Is is possible to feed the results of ‘crawls’ into an LLM model for data use?
Hello,
This is not yet possible, but it is an interesting idea, i noted it, will think on it for upcoming updates.
Regards, Szabi – CodeRevolution
I have a problem with the URL on my site, how can I remove part of the unnecessary URL?
Hello,
Please send me details about this issue kisded@yahoo.com
Also, send ke in email your purchase code, so i can check it.
Regards, Szabi – CodeRevolution
Can the software scrape membership course download post or content which is password protected (Login Needed – Email And Password) and the download button which is also protected with (Ultimate Membership Plugin) https://codecanyon.net/item/ultimate-membership-pro-wordpress-plugin/12159253
I HAVE PREMIUM ACCESS TO THE WEBSITE AND ACCESS TO ALL DOWNLOADS.
Hello,
Yes, this is possible, please check details: https://m.youtube.com/watch?v=TqwDyB3y9bk&pp=ygURQ3Jhd2xvbWF0aWMgbG9naW4%3D
Regards, Szabi – CodeRevolution
Great. So What If I want to rewrite the content scraped with AI following a Template Format And Locking The Download Section With My Own Ultimate Membership Lock Is That Also Possible?
Yes, you can rewrite using AI if you combine Crawlomatic with Aiomatic: https://codecanyon.net/item/aiomatic-automatic-ai-content-writer/38877369
As shown in this tutorial: https://youtu.be/G52OMDC6Eig?si=ZMaYt1aLqblv9_-Z
Regards.
Is it compatible with CPT and ACF?
Hello,
Yes, the plugin is able to create custom post types and also custom fields for them. Please check this tutorial: https://www.youtube.com/watch?v=GMIojmlI9fA
Regards, Szabi – CodeRevolution
can it be done with done with a graphic interface? It seems like a little bit tricky all the acf manage
Currently only the interface shown in the video is possible to be used. I noted this idea for upcoming updates.
Regards.
Hello,
The website where my license was active is now lost because I forgot to pay for the hosting, causing me to lose the domain. As a result, I can’t reinstall or reactivate the plugin. Could you please help me resolve this issue?
Thank you!
Hello,
Please send me your purchase code to my email: kisded@yahoo.com
I will help solve this issue.
Regards, Szabi – CodeRevolution
I’m wondering if there is any chance to get integration with the word-spinner.com API.
Hello,
I checked the word-spinner.com website, but I don’t see on it any reference to an API service. Are you sure that they are providing a public API for their services?
Regards, Szabi – CodeRevolution
First of all thanks Devloper you are amazing ^^
I installed nodeJS & puppeteer globally on my server, but the plugins say on the rule “Puppeteer not found! Please install it on your server globally.” and on the log says ” nodeJS not found, please install it on your server” what should I do?
Hello,
First of all, thank you for your purchase, I am glad you like the plugin.
If it is not working for you, please try installing Puppeteer in this folder of the plugin: \wp-content\plugins\crawlomatic-multipage-scraper-post-generator\res\puppeteer
Let me know if this helped.
Regards, Szabi – CodeRevolution
Hi , i installed Puppeteer locally on : \wp-content\plugins\crawlomatic-multipage-scraper-post-generator\res\puppeteer now I got this error from the log :
“puppeteercheck failed to run, error: /home/user/web/website.com/public_html/wp-content/plugins/crawlomatic-multipage-scraper-post-generator/res/puppeteer/puppeteer.js:10 process.on(‘unhandledRejection’, up => { throw up }) ^ Error: Could not find Chrome (ver. 132.0.6834.110). This can occur if either 1. you did not perform an installation before running the script (e.g. `npx puppeteer browsers install chrome`) or 2. your cache path is incorrectly configured (which is: /home/user/.cache/puppeteer). For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration. at ChromeLauncher.resolveExecutablePath (/home/user/web/website.com/public_html/wp-content/plugins/crawlomatic-multipage-scraper-post-generator/res/puppeteer/node_modules/puppeteer-core/lib/cjs/puppeteer/node/BrowserLauncher.js:298:27) at ChromeLauncher.computeLaunchArguments (/home/user/web/website.com/public_html/wp-content/plugins/crawlomatic-multipage-scraper-post-generator/res/puppeteer/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ChromeLauncher.js:93:24) at async ChromeLauncher.launch (/home/user/web/website.com/public_html/wp-content/plugins/crawlomatic-multipage-scraper-post-generator/res/puppeteer/node_modules/puppeteer-core/lib/cjs/puppeteer/node/BrowserLauncher.js:84:28) at async /home/user/web/website.com/public_html/wp-content/plugins/crawlomatic-multipage-scraper-post-generator/res/puppeteer/puppeteer.js:36:19 Node.js v18.20.6”
This means that the plugin detected Puppeteer, however, it is not set up correctly. Please try following this tutorial for install: https://www.youtube.com/watch?v=pRUDcSOe724
Regards.
Hi , I saw in the article of the video that I should install version 17.1.3 of Puppeteer and newer versions have an issue with running.
I checked my Puppeteer version, and I found 24.1.1 . could that be the problem?
Yes, this should be an issue, please install v17.1.3
Unfortunately I was not able to make it work with more recent versions of Puppeteer, I noted this, will do some more research on this.
Regards.
Thanks, it worked with v17.1.3 installed on \wp-content\plugins\crawlomatic-multipage-scraper-post-generator\res\puppeteer but now I have another problem with the captcha, is there any solution to that ?
without HeadlessBrowserAPI , just with puppeteer or phantom on my server
actually is cloudflare captcha to be exact
I am glas to hear that it works. Rwgarding CloudFlare, i was not yet able to find a method around it. It is a really hard nut to crack. Will think more on this.
Regards.
thanks, what do you think about using “captcha *com” service ? It is a Сaptcha solving service that can be integrated with puppeteer
I am not familiar with it, but you can give it a try, it might help.
how did you solve that captcha problem with your HeadlessBrowserAPI ? that should be the same solution to my problem
HeadlessBrowserAPI is integrated with a similar service as you mentioned: 2captcha.com
Regards.
Great, can you tell me which service you integrate with ? and how if that’s possible
Best regards
I use services from 2captcha.com
Puppeteer code script needs to be modified, as shown here: https://www.webshare.io/academy-article/puppeteer-captcha
Regards
Hi Dear Szabi, I hope you are okay. . I managed to bypass Cloudflare with this test Js Puppeteer Code here: https://rustpad.io/#jVbYPH . How can I integrate that into the Crawlomatic /res/puppeteer/Puppeteer.js file without messing up with your original plugin code?
Best regards.
Hello,
For this, you need to edit the \wp-content\plugins\crawlomatic-multipage-scraper-post-generator\res\puppeteer\puppeteer.js from the plugin and add your customizations.
Regards.
Hi , yes exactly i don’t know what & where to add inside your code of puppeteer.js ( I got that code from chatgpt )
please can you help me with that?
Well, you can try feeding in ChatGPT both scripts (your and also the one found in the puppeteer.js file of the plugin) and instructing it to modify the puppeteer.js script, so it uses the method from the first script you have.
If this is not working for you, I can also help as a paid custom job, for details you can contact me at email kisded@yahoo.com
Regards.
i Sent you an email
Does this support gsm*ren4 phone?
Hello,
Yes, you will be able to connect to your WordPress site’s admin page from a Oppo Reno4 phone and will be also able to use the Crawlomatic plugin in it.
I hope this info helps.
Regards, Szabi – CodeRevolution
I mean can grab phone spesification/item detail from web gsm4r*na ?
I understand now.
Yes, the Crawlomatic plugin can scrape the website you mentioned. It will work.
Regards.
How do I get to the original server log files?
Hello,
Please go to the ‘Logs and Translations’ menu of the plugin -> ‘Activity Logs’ tab.
Regards, Szabi – CodeRevolution
I tested it with a demo before I bought it, why can’t I copy this kind of homepage? I can’t import the image normally.
Hello,
Thank you for contacting me.
Please send me the demo URL where you tried the plugin and where the image is not imported, so I can check it.
I tried this on my test site and for me it worked correctly.
Regards, Szabi – CodeRevolution
What I’m talking about is the details images of the page.
Could you please check if I bring all the contents of the URL I sent correctly? I definitely couldn’t bring the detailed image. But the main image is downloaded.
I understand now.
The images on this page are lazy loaded. To import them correctly, please do the following config in Crawlomatic rule settings:
Content Query Type Class Content Query String xans-element- xans-product xans-product-additional section Lazy Loading Images HTML Tag ec-data-src
Tutorial on lazy loaded images: https://www.youtube.com/watch?v=AzadF_dAAco
Update video: https://www.youtube.com/watch?v=BMzJWZdodlo
Let me know if this helped.
Regards.
good
When we delete a post, even if we check to avoid duplicate post, Crawlomatic and Newsomatic will fetch again the same post. Thanks to fix it so even if the post is deleted, the plugin will not fetch again the same post. thanks
Hello,
Currently the plugins will check duplicates by the existing posts, so if posts are deleted, they will be republished. If you need the plugin to not publish posts even if they are deleted, a custom update is needed for the plugin. For details, please contact me at kisded@yahoo.com
Regards, Szabi – CodeRevolution
tell me. if I scrap posts from the category archive, and there is 1. pagination with numbers 2. pagination with loading if I scroll further or click “show more” and I need to download all the posts from the category. how will this plugin behave in situations 1 and 2? very important, waiting for an answer
Hello,
Thank you for contacting me.
This is possible only in the case when the pagination is shown using the numbers (not the scroll for loading more posts). Please check this tutorial for details: https://www.youtube.com/watch?v=WfT5R5Oi8RU
Regards, Szabi – CodeRevolution
Pre-sale: Thanks for this alternative recommendation. Let’s see if I’m understanding the process correctly. When using the google api I can put in keywords (i.e. scented candle companies)? It will then process the scraped info. into posts with images? I’m using the listinghub plugin…is there a way to have the results show up as listings instead of just posts?
I’m ready to purchase, I just want to make sure I’m understanding the process and it can quickly do what I need. Thanks again!
Hello,
Yes, scraping Google Search results is possible, please check this video for details on this: https://www.youtube.com/watch?v=h6fQeH9-X8c
Also, yes, the plugin is able to create also listings instead of posts, you have to select your custom post type which provide listings, in the ‘Item Type’ settings field, in rule settings from the plugin.
Regards, Szabi – CodeRevolution
is it possible to crawl with your plugin and then place the crawled data into a google sheet somehow?
Hello,
Yes, this should be possible if you use a plugin to export the posts scraped using Crawlomatic to Google Sheets. Example: https://wpsheeteditor.com/export-wordpress-posts-google-sheets/
Regards, Szabi – CodeRevolution
Hi CodeRvolutuion! I will work with Faceb. Events? https://www.facebook.com/events/search?q=galati&filters=eyJycF9ldmVudHNfbG9jYXRpb246MCI6IntcIm5hbWVcIjpcImZpbHRlcl9ldmVudHNfbG9jYXRpb25cIixcImFyZ3NcIjpcIjEwOTkyNzg5MjM2OTk1OFwifSJ9 I need to add FB events (titles, photos, date, description, etc) to my page.
Thanks!
Hello,
Thank you for contacting me.
Sorry, but scraping events is not yet possible in Crawlomatic. To scrape Facebook posts, you can try F-omatic: https://codecanyon.net/item/fbomatic-automatic-post-generator-and-facebook-auto-poster-plugin-for-wordpress/19528972
Tutorial on usage: https://www.youtube.com/watch?v=lzapShbFPNs
Regards, Szabi – CodeRevolution
Hi, does the plugin have a visual selector or do I have to map the HTML DOM elements manually by code?
Hello,
Thank you for contacting me.
Yes, the plugin has a visual selector. Please check this tutorial video for details: https://www.youtube.com/watch?v=2ixtS3LQsI4
I hope this info helps.
Regards, Szabi – CodeRevolution
