This author's response time can be up to 2 business days.
Hello, i’m facing some issues with another same function plugins, if the crawled website category pages loads from scripts i can’t find the link to the post urls
same issue applies with gallery images for crawled sites
can you please check and see if for Maycs, http://wpcontentcrawler.com/demo/wp-admin/post.php?post=65415&action=edit
if i can get the Gallery Image URL Selectors for posts
update: it looks like the demo is broken and is not saving any fields
The demo is not broken. It is not possible for me to provide support without you buying the plugin.
i don’t need support, if your plugin fulfill my needs with a proof i’ll buy it right away
i just need to make sure that what i’ve faced with other plugins won’t be the same issue…..
everytime i add something to for example Category Post URL Selectors hit save, its gone, thats why i’ve said the demo is broken
It looks like there are too many categories, which takes up all allowed POST data limit. This is a limit that can be set from your server settings. You probably exceed the limit. Hence, the changes you have made are lost. Try to lower the number of categories for the demo.
If you have the same domain, the error should not related to it. Make sure your server fulfills the requirements of the plugin. Your problem might be related to cURL.
Maybe I don’t know it was a site copy type migration to the new hardware. Do i need to uninstall it. I am unable to do it manually as it is asking for ftp access which I do not have. The domain name is same. It may have an additional CNAME but the original host name is same. Site is currently down
Your domain does not exist in the license server. So, you can install it to any domain you want. If there is a license error even in this case, there might be something else wrong. What does the error message say?
I have rested the site to genric install but still getting license error
What does the error message say?
Message: This license has reached its domain limit and is not valid for this domain. Registered domains: ...
It shows the IP address of my server
Send it to me.
The database is same as the old one too. I imported it. it is 184.108.40.206
You can save your license settings again. After that, the error should be gone.
Yes it worked
Merhaba dostum, bu eklentiyi satın almak istiyorum fakat birkaçtane soru işareti var.
1. Hedef sitedeki öne çıkarılmış görselleri kendi sunucumuza kayıt alıyormu?
2. Hedef sitede öne çıkarılmış herhangi bir görsel yok ise, varsayılan bir resim belirleyebiliyor muyuz?
Sitenin estetiği açısından resim konusu önemli
Teşekkürler, harika bir iş çıkarmışsınız. Tebrik ederim
1. Evet 2. Maalesef eklentinin böyle bir özelliği yok. Temanızdan öne çıkan görseli olmayan yazılar için varsayılan bir öne çıkan görsel belirleyebilirsiniz.
Hi there great plugin,
I have some proxies from actproxy, but the plugin does not verify them. I added my server address as stated in cpanel, for auth ip for the proxies. The proxies work fine from my computer i.e they verify on scrapebox, they don’t work on wp content crawler. They fail to verify. My proxies are http/https I tried both, no success.The plugin works fine otherwise.
Automatic crawling does not work now also, it worked just great, then I stopped it, and now it does not work anymore. Manual save works. It has nothing to do with cron problem as I am active on my site as admin and visitor.
Then auto-crawling problem might be related to cURL errors. Check this FAQ. It might solve your issue.
For your proxy question, if it fails on the test, then you cannot use it. Follow the instructions of the setting by clicking info button next to the setting name. I do not know what you did on cPanel, but the plugin does not require any proxy to be configured on cPanel.
Hi again, I tried define( ‘ALTERNATE_WP_CRON’, true ); it does not fix it. The weird thing is once I installed I was getting posts and url’s as supposed. It did work and now it doesn’t.
Concerning the proxies I suppose the plugin uses the server ip to crawl. The proxies I use need to have authentication ip in order to work. I have properly added the authentication ip from my server to my actproxy panel, I have tested the ips from my computer in the same spirit. Proxies work fine. Your plugin fails to verify the proxies.
So my first question concerning this, have I got this right, the wp content crawler uses the server ip to crawl as stated in cpanel?
The second is can you try a test for me, I send you my ip’s and I add your server ip as authenticated so you can try in your own test environment see whats the problem.
The plugin does not suddenly stop crawling. Check if the target site has new post URLs on category pages whose URLs are in category map. If there are new post URLs on those category pages, check if your site settings are correct by using the category test in Tester page. If everything seems to work fine, try to clear the URLs waiting in the queue and wait for a while so that the plugin can start collecting the new post URLs. If these do not work, go and check the dashboard for next task dates. If, say, next URL collection’s date says “x mins ago” instead of “x mins later”, your problem is probably related to CRON, not the plugin. If all of these do not work, go to this FAQ and follow the instructions, send me your debug file.
Regarding the proxy, you can make sure that the plugin’s proxy feature works by going to a random free proxy listing site and test a few of the proxies by entering them in the proxy setting and then hitting the test button. You should see that a few of them pass the test (I said a few of them because those proxies do not work for sure). You can say that your proxy uses authentication, and you enter your proxy string with your username and password. The plugin handles authentication as well. I know that because I helped a few customers regarding this. They all followed the instructions written next to the setting and everything worked just fine. If, again, these do not work for you, see the FAQ I mentioned above.
Thanks for fast response, it does say -min ago on the next task just a – not a number. I did all the previous things, even try new site. Manual and tester work fine. So you say it is cron related? what to try next?
Concerning proxies, the feature did seem to work on some free proxies for a while. That seems to work except socket protocol socks:// I could not verify a single one from a huge list I was getting curl error. So there might be some issue with this feature.
I have specific issue which you did not answer. My proxies authenticate by ip. I put the ip of my server as authentication ip. The proxy company says their proxies authenticate just fine. I did try the proxies in my computer they work fine. So, either I need to put another auth ip for the crawler, or the feature does not work properly. Simple as that. My server ip is not a mystery, so im positive I got that pretty much right. So the plugin may have some bug with it.
Pls, give some attention on this, proxy feature is quite important and your plugin has great potential in the market, for #1 spot I know because I’m a many year user of autoblog plugins have tried everything there is. You have done some quite brilliant work here.
My debug file is totally empty so there’s nothing to see there.
I installed wp crontrol to view the crons running the wp crawler cron is there I also hit the run now link and got “Successfully executed the cron event wptslm_9bb72b6ea0cd78f59a693edfeb2a2c5e.”
But nothing. I also checked ht.access, it is clean. I do have alternate cron in wp config.
My host hostgator has a 15 min limit on running any cron, but that would not be a problem.
It just worked when I installed and tried for first time and then dead.
Do not trust Crontrol’s “execute” functionality that much. Can you send me your FTP and admin login credentials through my profile page?
You are right. I checked it and socks is not supported by the plugin. Is there a way that you can use http or https?
By the way, try to save your general settings again before sending me the credentials. After saving, check if next CRON runs are shown properly. If that does not work, then please proceed with sending the credentials.
ok seems to work now, pls take a look at the proxies as well they are http/https with authentication ip of my server they should work but they fail.
Please send the proxies to me via email so that I can check them.
So what was the culprit with auto crawler, your fixed it instantly
to check the proxies you must tell me the ip from which you are going to chek to add it to auth ips
As I said via email, it was your theme causing CRON tasks not being scheduled properly.
I can’t activate the plugin.
Warning: require_once(/data/wwwroot/www.myDomain.cc/wordpress/wordpresswp-admin/includes/update.php): failed to open stream: No such file or directory in /data/wwwroot/www.myDomain.cc/wordpress/wp-content/plugins/wp-content-crawler/app/services/DatabaseService.php on line 458
Fatal error: require_once(): Failed opening required ’/data/wwwroot/www.myDomain.cc/wordpress/wordpresswp-admin/includes/update.php’ (include_path=’.:/usr/local/php/lib/php’) in /data/wwwroot/www.myDomain.cc/wordpress/wp-content/plugins/wp-content-crawler/app/services/DatabaseService.php on line 458
Could you please send your FTP and admin login credentials through my profile page so that I can check?
Send to u now.
Hi! Is this Plugin work with Wordpress 4.8? (Last update)?! Thanks
New WordPress versions always support older versions. So, yes, it should work. But, I’ll check to make sure.
Everything seems to work fine on version 4.8.
Thanks for quickly answer! But, I use your demo and see this, i think that 4.8 don’t work or php, if i’ll buy this Plugin, i’m have many troubles? Any ideas, WTF? My demo in your demo “kommersant” http://s018.radikal.ru/i502/1706/13/99d817e95e52.png
There is nothing wrong with PHP or WordPress 4.8. It is a character encoding problem. You can see this FAQ or try to check “use custom general settings” in your site settings and then check “always use UTF8 encoding” option under Settings tab, which will be shown after you check “use custom general settings” option.
Okay, you’re really right kommersant.ru is using another charset other than UTF-8 (it use windows -1251), but in your demo also 2 ways : UTF and without UTF, where is i must change on windows-1251 ?
I did not understand what you meant. Just follow the instructions written in the FAQ, it’ll be fine.
the site I’m trying to crawl have a meta tag that defines a charset other than UTF-8, it using win-1256. Where i’m change this in your demo?
Read the FAQ.
Hello , i tried your demo site and i cant understand why ther is Queue: 230 and Saved: 2
Why its just 2 ?? i have 1 min for Post URL Collection Interval and 1 min for Post Crawl Interval too !!
Which site is yours? There are a lot of sites. If there are, say, three sites active and post crawling interval is 1 minute, every minute a different site is crawled. So, there needs to pass 3 minutes for a site to be crawled again. In addition to that, the plugin can collect hundreds of URLs from a category page. However, that does not mean all of them will be crawled at once. Again, every minute a new post is crawled. So, if there are 230 URLs in queue, there needs to pass 230 minutes for all of them to be crawled. If there are three active sites, 3 * 230 minutes should pass. On the other hand, you can specify how many posts can be saved when post crawling function run from general settings (this cannot be changed on the demo site but you will be able to do so on your own site).
i understand whta i see Queue Saved … and Other , what “other” is refered to ?
i want to understand too how the process is going does it start from the new posts every time to check for new posts and then continue with older posts or how ? thnks
The plugin goes to the target category pages and check for new post URLs. If it finds them, it adds them to the queue. After that, the URLs are crawled one by one and new posts are created.
Other shows the number of post URLs that cannot be categorized as “queue”, “saved”, “updated”, or “deleted”. There might be some kind of error on the target site, or the saved posts might be deleted outside of WordPress. In those cases, the number of those URLs are shown as “other”.
This comment is currently being reviewed.
merhaba hocam dun eklentiyi satın aldım ama videodaki sorunum bir turlu cözemedim yardımcı olursanız sevinirim https://www.youtube.com/watch?v=bQ5HmlbMNgA
php 5.6 wp sürüm 4.7
PHP’nin 5.6 olduğundan emin olun. mbstring uzantısının da aktif olduğunu teyit edin.
Hello, I would like to be issued a refund as this product does not work as described. After spending 10+ hours trying to set up this thing I still have not been able to capture the article content without issues. I would like a refund for this product please.
Please make sure you configured the settings properly. I cannot refund if there is no technical problem causing this. You can send me your admin login credentials through my profile page so that I can check what’s wrong.
Merhaba güzel bir iş çıkartmışsınız. Tebrik ederim. En kısa zamanda alıp deneyeceğim. Kolay gelsin…
Turgut bey, sizi yakalamışken bir de soru sorayım… Acaba bu eklenti ile başka bir siteden event kopyalama şansımız varmıdır? Etkinlikleri başlangıç ve bitiş tarihi ile beraber alıp, eventon pluginine kopyalamak gibi bir faliyetten bahsediyorum. Sergi duyuruları mesela… Olabiliyorsa, ne şık olur Teşekkürler, selamlar.
Eklenti hedef sitedeki bilgileri post meta değerleri olarak kaydedebiliyor. Ayrıca yazıları istediğiniz yazı türünde de kaydedebiliyorsunuz. Eğer bahsettiğiniz eklenti post meta değerlerini kullanarak eventleri gösteriyorsa yapılabilir gibi görünüyor. Fakat o tip eklentiler taxonomy de kullanabiliyor. O durumda kaydedemezsiniz.
Denemeye değer Satın aldım az önce. Takıldığım yerler olursa haberleşiriz… Çok teşekkürler !
Tamamdır, rica ederim.
Şahane ötesi bir eklenti bu
Yazıları hiç sorunsuz indirebiliyorum.
Yanlız, Plesk panelde WP-Cron normalde çalışmıyor olduğu için define(‘WP_DEBUG’, false); dizinini eklemem gerekti mecburen wp-config içine.
Bunun bir tehlikesi olur mu?
Bazen olur olmaz 500 Error verebiliyor bu satır eklenince diye biliyorum çünkü.
Aslı varmıdır ? Ne dersiniz, fikrinizi merak ediyorum.
Çok teşekkürler !
Beğenmenize sevindim Debug özelliğinin açık ya da kapalı olması bir probleme yol açmaz. Bu özellik ile 500 hatasının veya CRON’un bir bağlantısı olduğunu sanmıyorum. Sizin için kapalı olması uygunsa FALSE olarak belirleyebilirsiniz.
Pardon yanlış yazmışım Şu satırdan bahsediyorum:
define( ‘ALTERNATE_WP_CRON’, true );
True yapınca eklenti çalışıyor.
Ama false yaparsam eklenti cronjobları çalıştıramıyor. Plesk panel ayarları müsade etmiyormuş çünkü.
Şu anda true vaziyetinde, ve eklenti sürekli çalışacağından dolayı hep öyle kalması gerekecek. Bunun 500 Error vermesi gibi bir durum olabilir mi ileride ziyaretçiler sayfayı gezerken falan, olur olmaz zamanlarda ? Onu merak ediyorum.
Yani bir tehlikesi varsa, sadece plugin’i çalıştırırken true yaparım, diğer zamanlarda false olarak bırakayım diye soruyorum.
Yada böyle birşey olmaz derseniz, hep true kalsın.
Ne dersiniz ?
Ben böyle bir problem olacağını sanmıyorum fakat gerekirse söylediğiniz gibi eklentiyi kullanırken true yapabilirsiniz.
Okeydir Çok teşekkürler. Başka soracaklarım olursa, yine rahatsız edebilirim. Görüşmek üzere… İyi bir hafta olsun.
Teşekkürler, size de.
can i use your plugin to set up a store with products which i add them from other store (dropbox store), and update prices and availability at all time?
or even to compare the prices of product if i want to run a compare site?
thanks for your time!
The plugin can save post meta values. If the comparing site will use post meta values, yes, you probably can do that. For the e-commerce site, again, you can use post meta values to save prices. The plugin can update the posts. It looks possible. However, you probably need to be able to write regular expressions with ease so that you can use find and replace options of the plugin effectively to reformat the values that should be stored in post meta fields.
Az önce satın alma işlemi gerçekleştirdim, lisansı nerede bulabilirim ?
Şurada anlatımı var: https://help.market.envato.com/hc/en-us/articles/202822600-Where-Is-My-Purchase-Code-
Presale question: Suppose I’ll be having about 500 RSS feed sites to crawl post every 5/10 minutes, how much RAM or server would you recommend? Or does it really affects the server at all?
I’m glad you liked it. Unfortunately, the plugin cannot crawl RSS feeds. In terms of RAM, it depends on what operations you do when crawling a site. You can use the tester to see how much memory the plugin uses to crawl a post or a category page. The plugin can crawl, say, 1000 posts per minute. If you configure your settings to crawl 1000 posts per minute, you will need a lot of memory. As you can see, you need to figure out how much memory you need by testing.
Thank you for letting me know.
Also, Suppose I’ve set post deleted after two days automatically, will the crawler crawls the deleted post again?
As long as the deleted post’s URL on the target site is not changed, the plugin will not crawl the deleted post again.
Pre Question: I would like to import Woo Commerce Products automatically from different URLs – including Text, price, pictures … , but then the client will be directed to the URL to pay directly to the shop owner (external product) ... is this possible with your plugin ?
Redirecting is not possible. You need to handle this outside of the plugin.
Bot harika emeğiniz için teşekkürler. Bir sorum var uğraştım ama yapamadım. Çektiğim sitede ki konu başlığı ne ise ben konu başlığına 3 etiket üretmek istiyorum. Örnek olarak [ konu başlığı yaz ], [ konu başlığı oku ], [ konu başlığı indir ] gbi bunu nasıl ve nereden yapabilirim. teşekkürler.
Teşekkürler, beğenmenize sevindim. Eklentinin böyle bir özelliği yok fakat “bul ve değiştir” ve HTML manipülasyon seçeneklerini kullanarak bunu başarmanız mümkün gibi görünüyor. Yapmanız gereken başlığı kopyalayıp her bir kopyada tekrar değişiklik yaparak başlığın sonuna “yaz”, “oku” gibi kelimeler eklemek. Bunun yapmak için “bul ve değiştir” seçeneklerini düzenli ifadeler ile kullanabilirsiniz.
ben bu eklentinizi alacaktım ama kafamda bir soru var altta gecen sitelerin içeriklerini çekiyorsa alacagım maksat paramız bouşuna gitmesin
ben demo da denedim ama hatalar aldım . beceremedim.
siz acaba bu sitelerdeki verileri çekebiliyormuyum diye kontrol edebilirmisiniz.
Maalesef siteleri tek tek test etmem mümkün değil. Demoyu kullanarak test edebilirsiniz.
Hi Great plug in ….
I would like to short the content result , lets say 50 words, and then a link to the original post.
Is this possible with regex on your plug in ??
Thanks. Yes, it looks possible. It depends on your regular expression knowledge.
i really like it and i find very usefull for my project but i can’t copy all content, i need to trim it and link it to original.
I’ll buy it so you give an hint ?
You could try something like this ( https://regex101.com/r/edbETU/1 ) to match a number of words. Replacing the match with $1 will leave only the specified number of words and get rid of the remaining part of the text. I normally do not help with the settings. However, since you bought it, I wanted to help. The support does not cover configuring the settings.
Thanks ….. i can make it work with your hint …... Thanks again
You are welcome.
Hey mate, very nice crawling plugin,
i’ve a question pre sale and ready to buy if i got what i need.
i’ve got a website setup on your demo iherb.com
how can i change the country from US to DE before crawling,
when i go the website there’s a place to chose my country from it, but it is not working from the demo
i’m in UK and need to crawl DE websites so i’ll face a lot of these stuff, how can i click on change country and save the country i want to crawl?
It looks like connecting over Germany may cause the site to be loaded in German. So, you can try using a proxy server located in Germany.
i’ve added a german proxy to the this website on the demo, but still from the tester it crawls english version.
i’ve clicked on Use custom general settings then added the proxy
but still all english
Then it might not be possible.
then what does the proxy do?, and what is the format for the cookies, maybe i can add my account cookies
The proxy feature works fine. I meant if connecting over a proxy does not work, it might not be possible. Each cookie option requires a cookie name and its value.
Use, by you or one client, in a single end product which end users are not charged for. The total price includes the item price and a buyer fee.
Use, by you or one client, in a single end product which end users can be charged for. The total price includes the item price and a buyer fee.
View license details
Get it now and save up to $10
Deliver better projects faster. Photos, templates & courses
Unlimited downloads. Only $29/month
Learn almost anything with
Envato Tuts+ for free
9000 free tutorials, 3000 paid courses
Designers matched perfectly to
you on Envato Studio
2000 artists ready to undertake your work