Discussion on WP Content Crawler - Get content from almost any site, automatically!

turgutsaricam

turgutsaricam supports this item

Supported

This author's response time can be up to 2 business days.

299 comments found.

Hello, i’m facing some issues with another same function plugins, if the crawled website category pages loads from scripts i can’t find the link to the post urls same issue applies with gallery images for crawled sites

can you please check and see if for Maycs, http://wpcontentcrawler.com/demo/wp-admin/post.php?post=65415&action=edit if i can get the Gallery Image URL Selectors for posts

and for Chicme, if i can get the Category Post URL Selectors http://wpcontentcrawler.com/demo/wp-admin/post.php?post=65414&action=edit

update: it looks like the demo is broken and is not saving any fields

Hi,

The demo is not broken. It is not possible for me to provide support without you buying the plugin.

i don’t need support, if your plugin fulfill my needs with a proof i’ll buy it right away i just need to make sure that what i’ve faced with other plugins won’t be the same issue….. everytime i add something to for example Category Post URL Selectors hit save, its gone, thats why i’ve said the demo is broken

It looks like there are too many categories, which takes up all allowed POST data limit. This is a limit that can be set from your server settings. You probably exceed the limit. Hence, the changes you have made are lost. Try to lower the number of categories for the demo.

For your other question, the plugin cannot run JavaScript via PHP. If the target site is manipulated via JavaScript, you cannot crawl it. If an information is created via JavaScript, you cannot get it. If anything is loaded to the page after the page is fully loaded, i.e. via AJAX, you cannot get it. All of the information you need from the page must exist in the unmanipulated source code.

Hi,

Changed the web server, same domain, error with license. Please help http://codscales.info

The database is same as the old one too. I imported it. it is 35.188.179.83

You can save your license settings again. After that, the error should be gone.

Yes it worked :)

ekotheme

ekotheme Purchased

Merhaba dostum, bu eklentiyi satın almak istiyorum fakat birkaçtane soru işareti var.

1. Hedef sitedeki öne çıkarılmış görselleri kendi sunucumuza kayıt alıyormu? 2. Hedef sitede öne çıkarılmış herhangi bir görsel yok ise, varsayılan bir resim belirleyebiliyor muyuz? Sitenin estetiği açısından resim konusu önemli Teşekkürler, harika bir iş çıkarmışsınız. Tebrik ederim

Merhaba,

Teşekkürler.

1. Evet 2. Maalesef eklentinin böyle bir özelliği yok. Temanızdan öne çıkan görseli olmayan yazılar için varsayılan bir öne çıkan görsel belirleyebilirsiniz.

Hi there great plugin, I have some proxies from actproxy, but the plugin does not verify them. I added my server address as stated in cpanel, for auth ip for the proxies. The proxies work fine from my computer i.e they verify on scrapebox, they don’t work on wp content crawler. They fail to verify. My proxies are http/https I tried both, no success.The plugin works fine otherwise.

So what was the culprit with auto crawler, your fixed it instantly

to check the proxies you must tell me the ip from which you are going to chek to add it to auth ips

As I said via email, it was your theme causing CRON tasks not being scheduled properly.

HI,

I can’t activate the plugin.

The error:

Warning: require_once(/data/wwwroot/www.myDomain.cc/wordpress/wordpresswp-admin/includes/update.php): failed to open stream: No such file or directory in /data/wwwroot/www.myDomain.cc/wordpress/wp-content/plugins/wp-content-crawler/app/services/DatabaseService.php on line 458

Fatal error: require_once(): Failed opening required ’/data/wwwroot/www.myDomain.cc/wordpress/wordpresswp-admin/includes/update.php’ (include_path=’.:/usr/local/php/lib/php’) in /data/wwwroot/www.myDomain.cc/wordpress/wp-content/plugins/wp-content-crawler/app/services/DatabaseService.php on line 458

Thanks~

Hi,

Could you please send your FTP and admin login credentials through my profile page so that I can check?

Send to u now. Thanks~

HI,

I can’t activate the plugin.

The error:

Warning: require_once(/data/wwwroot/www.myDomain.cc/wordpress/wordpresswp-admin/includes/update.php): failed to open stream: No such file or directory in /data/wwwroot/www.myDomain.cc/wordpress/wp-content/plugins/wp-content-crawler/app/services/DatabaseService.php on line 458

Fatal error: require_once(): Failed opening required ’/data/wwwroot/www.myDomain.cc/wordpress/wordpresswp-admin/includes/update.php’ (include_path=’.:/usr/local/php/lib/php’) in /data/wwwroot/www.myDomain.cc/wordpress/wp-content/plugins/wp-content-crawler/app/services/DatabaseService.php on line 458

Thanks~

Hi! Is this Plugin work with Wordpress 4.8? (Last update)?! Thanks

I did not understand what you meant. Just follow the instructions written in the FAQ, it’ll be fine.

the site I’m trying to crawl have a meta tag that defines a charset other than UTF-8, it using win-1256. Where i’m change this in your demo?

Read the FAQ.

Hello , i tried your demo site and i cant understand why ther is Queue: 230 and Saved: 2 Why its just 2 ?? i have 1 min for Post URL Collection Interval and 1 min for Post Crawl Interval too !!

Hi,

Which site is yours? There are a lot of sites. If there are, say, three sites active and post crawling interval is 1 minute, every minute a different site is crawled. So, there needs to pass 3 minutes for a site to be crawled again. In addition to that, the plugin can collect hundreds of URLs from a category page. However, that does not mean all of them will be crawled at once. Again, every minute a new post is crawled. So, if there are 230 URLs in queue, there needs to pass 230 minutes for all of them to be crawled. If there are three active sites, 3 * 230 minutes should pass. On the other hand, you can specify how many posts can be saved when post crawling function run from general settings (this cannot be changed on the demo site but you will be able to do so on your own site).

i understand whta i see Queue Saved … and Other , what “other” is refered to ? i want to understand too how the process is going does it start from the new posts every time to check for new posts and then continue with older posts or how ? thnks

The plugin goes to the target category pages and check for new post URLs. If it finds them, it adds them to the queue. After that, the URLs are crawled one by one and new posts are created.

Other shows the number of post URLs that cannot be categorized as “queue”, “saved”, “updated”, or “deleted”. There might be some kind of error on the target site, or the saved posts might be deleted outside of WordPress. In those cases, the number of those URLs are shown as “other”.

Bemirad

Bemirad Purchased

This comment is currently being reviewed.

Bemirad

Bemirad Purchased

merhaba hocam dun eklentiyi satın aldım ama videodaki sorunum bir turlu cözemedim yardımcı olursanız sevinirim https://www.youtube.com/watch?v=bQ5HmlbMNgA php 5.6 wp sürüm 4.7

Merhaba,

PHP’nin 5.6 olduğundan emin olun. mbstring uzantısının da aktif olduğunu teyit edin.

Hello, I would like to be issued a refund as this product does not work as described. After spending 10+ hours trying to set up this thing I still have not been able to capture the article content without issues. I would like a refund for this product please.

Hi,

Please make sure you configured the settings properly. I cannot refund if there is no technical problem causing this. You can send me your admin login credentials through my profile page so that I can check what’s wrong.

doreso

doreso Purchased

Merhaba güzel bir iş çıkartmışsınız. Tebrik ederim. En kısa zamanda alıp deneyeceğim. Kolay gelsin…

Ben böyle bir problem olacağını sanmıyorum fakat gerekirse söylediğiniz gibi eklentiyi kullanırken true yapabilirsiniz.

doreso

doreso Purchased

Okeydir :) Çok teşekkürler. Başka soracaklarım olursa, yine rahatsız edebilirim. Görüşmek üzere… İyi bir hafta olsun.

Teşekkürler, size de.

can i use your plugin to set up a store with products which i add them from other store (dropbox store), and update prices and availability at all time?

or even to compare the prices of product if i want to run a compare site?

thanks for your time!

Hi,

The plugin can save post meta values. If the comparing site will use post meta values, yes, you probably can do that. For the e-commerce site, again, you can use post meta values to save prices. The plugin can update the posts. It looks possible. However, you probably need to be able to write regular expressions with ease so that you can use find and replace options of the plugin effectively to reformat the values that should be stored in post meta fields.

Merhaba,

Az önce satın alma işlemi gerçekleştirdim, lisansı nerede bulabilirim ?

Hi, Great plugin! Presale question: Suppose I’ll be having about 500 RSS feed sites to crawl post every 5/10 minutes, how much RAM or server would you recommend? Or does it really affects the server at all?

Hi,

I’m glad you liked it. Unfortunately, the plugin cannot crawl RSS feeds. In terms of RAM, it depends on what operations you do when crawling a site. You can use the tester to see how much memory the plugin uses to crawl a post or a category page. The plugin can crawl, say, 1000 posts per minute. If you configure your settings to crawl 1000 posts per minute, you will need a lot of memory. As you can see, you need to figure out how much memory you need by testing.

Hi, Thank you for letting me know. Also, Suppose I’ve set post deleted after two days automatically, will the crawler crawls the deleted post again?

As long as the deleted post’s URL on the target site is not changed, the plugin will not crawl the deleted post again.

Pre Question: I would like to import Woo Commerce Products automatically from different URLs – including Text, price, pictures … , but then the client will be directed to the URL to pay directly to the shop owner (external product) ... is this possible with your plugin ?

Hi,

Redirecting is not possible. You need to handle this outside of the plugin.

Merhaba,

Bot harika emeğiniz için teşekkürler. Bir sorum var uğraştım ama yapamadım. Çektiğim sitede ki konu başlığı ne ise ben konu başlığına 3 etiket üretmek istiyorum. Örnek olarak [ konu başlığı yaz ], [ konu başlığı oku ], [ konu başlığı indir ] gbi bunu nasıl ve nereden yapabilirim. teşekkürler.

Merhaba,

Teşekkürler, beğenmenize sevindim. Eklentinin böyle bir özelliği yok fakat “bul ve değiştir” ve HTML manipülasyon seçeneklerini kullanarak bunu başarmanız mümkün gibi görünüyor. Yapmanız gereken başlığı kopyalayıp her bir kopyada tekrar değişiklik yaparak başlığın sonuna “yaz”, “oku” gibi kelimeler eklemek. Bunun yapmak için “bul ve değiştir” seçeneklerini düzenli ifadeler ile kullanabilirsiniz.

merhabalar hocam ben bu eklentinizi alacaktım ama kafamda bir soru var altta gecen sitelerin içeriklerini çekiyorsa alacagım maksat paramız bouşuna gitmesin ben demo da denedim ama hatalar aldım . beceremedim. siz acaba bu sitelerdeki verileri çekebiliyormuyum diye kontrol edebilirmisiniz.

Www.anextour.com.ua

Tui.ua

Anextour.com

Tui.ru

Merhaba,

Maalesef siteleri tek tek test etmem mümkün değil. Demoyu kullanarak test edebilirsiniz.

rttpi

rttpi Purchased

Hi Great plug in ….

I would like to short the content result , lets say 50 words, and then a link to the original post. Is this possible with regex on your plug in ??

You could try something like this ( https://regex101.com/r/edbETU/1 ) to match a number of words. Replacing the match with $1 will leave only the specified number of words and get rid of the remaining part of the text. I normally do not help with the settings. However, since you bought it, I wanted to help. The support does not cover configuring the settings.

rttpi

rttpi Purchased

Thanks ….. i can make it work with your hint …... Thanks again

You are welcome.

Hey mate, very nice crawling plugin,

i’ve a question pre sale and ready to buy if i got what i need.

i’ve got a website setup on your demo iherb.com

how can i change the country from US to DE before crawling,

when i go the website there’s a place to chose my country from it, but it is not working from the demo

i’m in UK and need to crawl DE websites so i’ll face a lot of these stuff, how can i click on change country and save the country i want to crawl?

Then it might not be possible.

then what does the proxy do?, and what is the format for the cookies, maybe i can add my account cookies

The proxy feature works fine. I meant if connecting over a proxy does not work, it might not be possible. Each cookie option requires a cookie name and its value.

by
by
by
by
by
by