2655 comments found.
Can the crawler skip partifuclar post if condition meets? for example, If the post name contains the words of ABC then skip this post and crawl the next post?thanks
Hi,
No, the plugin currently does not have a feature that can be used for that purpose, unfortunately. The only way might be to remove a post URL from the category page if it matches a regular expression.
Can I use your plugin to automatically sync content on a single page?
e.g.
Create a page in WordPress and Content Crawler will keep that page updated automatically based on a page from another website.
Hi,
The plugin can only update the posts/pages that are crawled by the plugin. So, if you crawl a post from the target site using a URL and enable recrawling, the plugin can update that post using the recrawling options you configured. For example, it can recrawl the post every 5 minutes.
Hello, I have purchased but there is license error. Where is the license key
Hi,
Please see this FAQ.
ben scrapes adlı eklentiyi kullanıyodum, onda direk çekilecek sitenin linkini yazıp bi kaç yere tıklayıp içerikleri çekiyoduk bunda sanırım herşey için chrome konsoldan kod bulmak zorundayız doğrumudur? Uğraşmadan çekmek mümkünmü bu eklenti ile? birde tek lisans kaç domainde kullanılabilir? tek domain ise, bir domainde iptal edip diğerinde kullanabilirmiyiz?
demoya baktımda zamanlama süresi en düşük “dakikada bir” olarak seçiliyor. Wordpress ile çalışan uygulamalarım var bunlara tek resim olarak yüklüyorum, dakikada bir içerik çekmek aşırı yavaş olur. Süre olarak bir içerik çekildikten sonra beklemeden diğerine geçilsin gelirse süper olur.
Merhaba,
Eklentinin nasıl kullanıldığıyla ilgili şu videoyu izleyebilirsiniz: https://www.youtube.com/watch?v=HW23JDj4I68 Dakikada birden fazla yazı kaydetmek istiyorsanız genel ayarlardan “Yazı toplama fonksiyonunun çalışma sayısı” seçeneğinin değerini artırabilirsiniz.
merhaba,türkçe karekter sıkıntısı çekiyorum,kendi tarayıcısında açtığı sitelerdeki tr karekterleri göstermiyor
Merhaba,
“Genel Ayarlar > Gelişmiş > Her zaman UTF8 dil kodlaması kullan” ve “Karakter kodlaması UTF8 değilse UTF8’e dönüştür” seçeneklerini işaretleyerek veya işaretlerini kaldırarak deneyebilirsiniz. Eğer iki seçenek de sorunu çözmüyorsa, hedef sitenin kaynak kodunun karakter kodlamasındaki problemlerden dolayı eklenti karakterleri doğru gösteremiyor demektir.
I am crawling products from websites and in order to make the Product URLs as affiliate links I need to add parameters to the original URLs crawled and store into woocommerce product URL. I tried using find and replace in the category section where now the URLs show the right affiliate format, but in woocommerce section using the shortcode [wcc-source-url] it does not use the modified URL it just puts the original URL. I also tried to create a custom shortcode, but it does not allow me to do it from category page, only from post page. Is there any solution to convert the original URLs using find and replace to affiliate links to store into woocommerce product link?
Hi,
The plugin saves the post URLs from the target category page. Hence, if you successfully replaced the URLs, the plugins saves them. After saving them, it uses the saved URLs to save the posts. The source URL short code points to that URL. The plugin cannot replace the short code with a URL that it does not know about. Hence, please make sure your settings are correct. You can use the tester to see if the settings are correct. If you see the changed URLs in the tester, then please make sure the plugin does not crawl the old URLs it saved before you made the changes.
Oh I see there are old URL in the QUEUE ! How Can I delete these ?
Found it ! In Tools > URLs section 
What happens if I have only activated “recrawling” in the scheduling and also site options and a previously post URL is not existing anymore. Will it be deleted ? If not, I understand that I need to activate Delete option, which will delete all posts and create again the existing ones on crawling. Is this correct ?
Hi,
If the target URL is not found, then the process of saving/updating the post is stopped. The post will not be deleted. If a post is deleted by the plugin, the plugin will not save the post again since its URL exists in the database.
hmmm getting a little confused now. Which would be then the way to remove products that are not appearing anymore on the website at recrawling them, whithout removing the ones that still exist ?
The plugin does not have a feature that can be used for that purpose.
So this is a very important feature if using with woocommerce products. As the should be a possibility to remove automatically products that are not anymore on the crawled website. So for Now I will need to externally delete all woocommerce products with a bulk tool, before recrawling… not very practical.
I noted this as a feature request. Thanks.
Hello, I see still this feature is not implemented. It is very important for all affiliate woocommerce users to be able to recrawl shops and automatically remove products that are not any more in the shop we recrawl. Also I find it very time consuming that automatically deleted products are not able to collect and save again automatically (which would solve above problem). So I have to manually go to “Tools > Clear URLs” in order to be able to again crawl products. So I can not have this task automatically done. So you see there is real need for a checkbox on the recrawl settings “Delete posts/products which are not anymore crawlable” – also this option should clear the URL at the same time, so if the shop puts same product online again, and scheduler is active it can save it again. Another option which could solve the problem, could be adding in the Delete section and Option “Also clear URLS of deleted posts”. So maybe this will be easier to do ? Please consider all these things. It is very important and useful. Any questions, please feel free to ask. Thanks, Daniel
Thanks for reminding the feature request. I am trying to implement as many features as possible but, since there are many requests, it takes time to implement them, unfortunately. I noted your explanation of the feature as well. Thanks.
Hello, thanks for your reply. Is it possible to sponsor/contribute to this feature so it comes in the next update ?
No, we do not provide such a service unfortunately.
Pre-Sales Questions; 1. Will it able to scrape data from other website and store in specific format. for example i want to crawl and scrape data from event website and I just need specific information i.e. events description, event location, start and end date, event organizer, contact person, etc.? 2. Will the plugin able to work with this theme https://templatic.com/wordpress-event-themes/eventum/ ?
Hi,
The plugin can save post meta (custom fields) and post taxonomies. If your theme uses them to show the details of the event, yes, you can use the plugin. I cannot check if the plugin can work with that theme since it is not a free theme.
I have a problem in woocommerce products, writing custom attributes, there is no option for “existing attributes”, so the attribute will be written and even if it is the same as already existing attributes, it will not counted in the woocommerce attributes into the existing ones. Any solution for this ?
No, currently there is no solution for this.
Ohh, then I cannot use the woocommerce attribute function at all – as I can then not group the products with the use of attributes. Are you sure it is this way ? In that case it would more or less useless using attributes…
for example I have created in woocommerce an attribute called “black” and then how could I add the crawled products to that attribute ? currently it writes me black but it appears not counted when showing all porducts of attribute “black”. in Wpallimport I have the option to check “existing attribute” and thenit works fine.
The plugin currently adds attributes as custom product attributes, not global WooCommerce attributes. The next version will have the feature that can be used to add attributes globally.
Great ! When approximately will be next version available ?
There is no exact date, currently.
I did not ask for exact date, I asked for approximate date. Why ? Because for me the plugin is completely useless without that function, and so I will hire somebody to make this feature but if you would say an approximate date I could consider waiting for it. It seem that you also do not accept payment for features to be done as you stated in another post. So I would appreciate and approximate date, lets say: 1 month ? 3 months ? 6 months ? 1 year ? something approximate please
Let’s say 3 months, then.
thankyou very much !
Hello again. I noticed one thing and maybe you have a hint for me… When I set the attribute in the plugin to pa_myname it does not work properly. But if I edit it in woocommerce itself, it changes it really to the taxonomy. Is there no way, to set this manually from your plugin ? I would not need the selector, I just want to set it manually to the existing taxonomy.
Hi,
There is no way to save the WooCommerce attributes as taxonomies currently since WooCommerce stores the values in a structure that cannot be created by using the available settings of the plugin. As I said, the feature will be available in the next version of the plugin.
ok. I understand. In the meantime I use a little php-script to rewrite the attributes into taxonomies. Works fine !
I’m glad you achieved your goal. Thanks for letting me know.
Hello, in the find and replace section, is there a possibility to do a replaceALL ? For example I have a long text, and I find the word “cheap” so I would like to set all the text (replaceAll) to a given word, for example “Sales Offer”
Hi,
Yes, it is possible. You can directly write what to find and all results will be replaced with the text you write into the replace input. Also, you can use regular expressions for case-insensitive replacements by using “i” modifier. There are many tutorials on the Internet about regular expressions if you want to learn more about them.
I have been able to use regex in that part, but find no way to do a replaceAll, as it finds and replaces the same it finds. I tried to put in the replace section something like: .replaceAll (Sales Offer), but it does not work. the regex expression seem only to be effective in the first box (find)
I do not exactly understand what you want to do. If you want to replace “x” with “y”, write “x” to find input, and write “y” to replace input. The plugin replaces all “x”s with “y”s that it finds in the text. It replaces in a case-sensitive way. Hence, if you want to make a replacement in a case-insensitive way, use regular expressions with delimiters and “i” modifier.
If you want to use the things that are found in the replace input, you can use groupings in the find input and use them in the replace input by writing $1, $2, and so on. There are many tutorials on the Internet about regular expressions. It is a very broad topic and I cannot tell you everything you need here in this comment section.
I do not want to do any of these. I know how to do that ones you explained. I need to do as explained above, in regex it is this and working perfectly, but do not know how to put it into the boxes of this pugin: preg_match(”/cheap/i”, $search) ) $return = “Sales Offer”
Configure the find-replace option as follows:
Regex: Checked Find: /cheap/i Replace: Sales Offer
this would not work as it would mantain all the text around the word cheap and replace the word only inside the text. I need to remove all the text and replace it with only the words “Sales Offer”. So if word “cheap” is found inside the long text just remove everything and retunr “Sales Offer”
You can write a regular expression that does what you want. Creating regular expressions is out of the support’s scope, unfortunately.
I can create any regular expression for the find section, no problem. But for the replace section, I do not see how to do it… that´s what would be needed here, and I am pretty sure it does not work with the current functions available
You cannot use regular expressions in the replace section, since it is not logical. I already explained how you can configure the settings in a way that it achieves the same thing as your PHP code snippet does. As I explained before, you can use groupings in the find input and use them as $1, $2, etc. Any other functionality you need does not exist in the plugin.
Thanks I think I got it now !
i have price same as: 1.320.000 VNĐ . How i can remove and get price from it.
Hi,
If you want to save it as a WooCommerce product’s price, after setting Post Type as “product” in General Settings, go to Post > WooCommerce > General > Regular Price Selectors option in site settings, enter the CSS selector that finds the price, and then click Options Box button that is available next to the CSS selector input. In the options box, go to Find-Replace tab and write ” VNĐ” (without quotation marks) to find input, and leave the replace input empty. You can then hit the test button next to the find input and check if the text is removed from the price. If you want to use it in a template, you can define a custom short code just for the price by following the same steps for the options box of Post > Custom Content Selectors for Shortcodes option. After that, you can use the short code you defined in a template available in the site settings page. If you want to directly remove the unwanted part of the price from the source code of the target page, you can use options under Post > Manipulate HTML section.
Turgut hocam merhaba, haber url bilgilerini ekliyorum. kategoriler ve yazı ayarlarını yapıyorum. testçi sayfasında kategoriyi çağırdığım zaman çıkan yazı url leri ile resimleri tutmuyor. bir yerde yanlış mı yapıyorum. yardımcı olursanız sevinirim
Merhaba,
“Kategori > Yazı linkleri öne çıkan görsellerden önce geliyor” seçeneğini işaretleyerek tekrar deneyebilir misiniz?
Teşekkür Ederim.
Hello, just bought and installed. I could not find any documentation about where int integrates with Woocommerce. Could you help me ? I have woocommerce installed and working, already importing products with WPALLImport and now wanted to add the option with this plugin. I tried adding a new product but there do not appear the selectors as in the video on the sales page is shown. How do I activate these ?
Thanks Daniel
Hi,
You can watch the first 3.5 minutes of this video to learn how WooCommerce settings are enabled and used.
Great, this works. However one more question:
As I have product pages where there are absolutely no data about the category of the product (not in breadcrumb or description). Is it possible to set the attributes in Woocommerce product getting these from the category of post ? In my cases the attributes I would like to set them depending from which category page it was linked. Thats because there are same products linked from several category pages, so setting them as attributes would be perfect. But I can see no way to do this ? Thanks for help.
Unfortunately, the plugin currently does not have a feature to achieve what you want. Could you please tell me what type of category information you need to use in the product attributes by elaborating a little bit more?
I am crawling flower shops and each flower is linked from different “category” pages, which could be occasions for which the flowers could be bought or type of flowers as “red roses” or “white roses”, so at the end each flower could have several attributes as “birthday” “Wedding” “roses”, etc.etc. I would like to be able to at least add one attribute, using the post category which I could associate in the post getting it from each category list page. That would be enough. Adding several attributes would be far more complicated I think.
All right. I have added this as a feature request, but please note that there are already more fundamental features that should be added to the plugin before implementing this. Hence, I cannot give a certain time for when it will be implemented into the plugin.
Great ! Is there a way I can contribute (pay) for the urgency of the implementation ?
Unfortunately, there is not. The features are sorted by how fundamental they are and implemented in that order.
Hi,
The license is for the main domain only or including the subdomains as well?
Thanks
Hi,
The plugin does not differentiate subdomains and main domains. The license can be actively used for only one domain, either subdomain or main domain. It cannot be actively used for more than one domain, including subdomains.
Hi is there any way to not publish or to publish as a draft if title/content are empty ??
Thanks
Hi,
You cannot set the post status depending on the existence of title or content, unfortunately.
I hope in the future you can implement these features that would be very interesting for the professional use of this plugin …. that is already very professional…. thanks
Thank you. I noted your feature request. Thanks.
Hello, can you help me with the settings? I will give you the cost.
Hi,
Sorry, I do not provide such services currently.
I’m struggling to test my site on the demo; there’s far too much going on to be able to test the plugin properly.
If the plugin doesn’t work to crawl the sites I need, will you offer a refund? I’m trying to crawl www.harryalbertgroup.co.uk/blog and the property listings on www.zoopla.co.uk ?
Hi,
I do not understand what you mean by “far too much going on”. If you mean there are too many settings, you do not need to configure all of them. A basic configuration needs Main > Site URL, Category > Category Map, Category > Category Post URL Selectors, Post > Post Title Selectors and Post > Post Content Selectors. That’s 5 settings in total. If the target site does not require JavaScript to show its contents, you will probably be fine. You can test the target sites by configuring only the five settings I mentioned.
I mean in terms of other people testing their sites, etc. And the settings yes
If it doesn’t work, will you offer refund? My technical abilities aren’t that great haha
Other people testing the settings should not affect you testing the plugin that much actually. The same account can login from multiple different places.
If you buy the for testing purposes, I do not offer a refund, unfortunately. You can do all the tests in the demo. Once you are sure you can use the plugin for your own purposes, you can buy the plugin by knowing that the plugin will definitely work for you.
Your Rastreador de conteúdo license has expired. Please get a new license until 28/01/2019 03:36 to continue using Rastreador de conteúdo.
Message: This license has reached its domain limit and is not valid for this domain. Registered domains: crawler.localhost
How to solve this?
Hi,
I replied to your email.
Hi, thanks for help!
Just one more question! Scheduling works, but does not save posts to the database. I do not know the reason, could you guide me?
If the plugin can collect URLs from the target site but cannot save them as posts, then the collected post URLs might not be valid. You can go to Dashboard page and check a few of the URLs in the queue against whether they are valid or not. You can also check those URLs using the tester.
thank you very much !!! Great job!!
Thanks
Good afternoon.
I am looking for a product like yours, but before buying it I would need to know if it would be worth my project.
I’m doing an online store (affiliates) on mobiles, and I want to make a scrap from a website that has more than 10,000 mobiles. With this plugin can I scrape all the products at once? Another question, when scraping the products, I have the attributes somewhat changed by copyright issue, can I replace them, or should I add the terms to the attributes that I have?
Regards!
Hi,
I assume the mobiles are the posts you want to retrieve. The plugin does not crawl all posts at once. It crawls them one by one, or a few of them at a time, depending on the scheduling settings. Briefly, the plugin works by crawling post URLs from the target site’s categories. After that, it crawls the collected post URLs one by one, or a few of them at a time. You can define how many posts should be saved at each run. The minimum crawling interval for the plugin is 1 minute. So, you can, for example, get 5 posts every minute using the automatic crawling. There is also a manual crawling tool. For that tool, there is no limitation on the number of posts that can be saved. It can be configured in a way that it saves many posts in parallel, i.e. at the same time. However, you need to insert category URLs or post URLs into the tool. If you insert category URLs, there is no need to enter post URLs since the plugin will find the post URLs from the target categories automatically. In short, if you can enter the category URLs to the manual crawling tool, you can save the target posts as fast as possible, without any limitation. The only problem might be that the target site might prevent your IP from accessing their site if you send too many requests to the target site in a short time. Also note that the plugin cannot retrieve the content that is created by using JavaScript. You can use the demo to test the plugin.
Hello again.
Thanks for your quick respect.
My idea is scrapear to introduce them in woocommerce, the only thing I need is that when scraping, the text of the web to scrap me well enter the terms in the attributes that I have put.
Example: in the web to scrape, they have the title “audio and video” and I have it in two sections, “audio” and “video ”.
Could the terms of attributes be well introduced as well?
Thanks again and best regards
If I understood correctly, you want to define your custom attributes and add values for those attributes from the target site. If this is the case, yes, you can do so. You can enter your custom attributes under Post > WooCommerce > Attributes > Custom Attributes. To define a value that is retrieved from the target site, you can use custom short codes defined in Post > Custom Short Codes > Custom Content Selectors for Shortcodes option. For example, you can define a short code that stores the value for the video attribute, and name it as “video”. Then, for the custom attribute’s value, you can enter “[video]”. The plugin will replace the [video] with the value retrieved from the target page.
Hello
Does this plugin update prices if they are updated from the website they were created on??
Does this plugin update meta, excerpt etc from websites where posts are pulled from?
Hi,
Yes, the plugin has a recrawling feature that you can use to update the posts.
This item was featured on CodeCanyon
Elite Author
Contact the Envato Market Help Team
