Code

Discussion on WP Content Crawler - Get content from almost any site, automatically!

Discussion on WP Content Crawler - Get content from almost any site, automatically!

Cart 3,660 sales

turgutsaricam supports this item

Supported

This author's response time can be up to 5 business days.

2655 comments found.

can i crawl cnn.com for articles with your crawler?

Hi,

Yes, you can. However, you need to remove some text from the page that cause HTML to be parsed incorrectly. You can go to “Manipulate HTML > Find and replace in raw HTML” and configure it as follows:

Regex: Checked
Find: <script[^>]*>.*?<\/script>
Replace: Leave this input empty

Hello; I would like to ask a question, does the plugin allow to extract tags defined example X1, X2, etc. in a page www.website.com/content-page1 and created from these fields a user with fields X1, X2, etc..

Hi,

The plugin does not have a feature that can be used to create users, unfortunately

Ok can you please, give me an example to extract a page content from Wikipedia https://fr.wikipedia.org/wiki/Banco_Comercial_Portugu%C3%AAs without the plugin extracting the side table . Is it possible? I tried the demo but I can’t extract just the text and titles from this page.

If you cannot select elements directly, i.e. without the elements you do not want, you can remove the unwanted elements by entering their selectors into “unnecessary elements selectors” option.

Hi, I bought your plugin! After try to use Scraper en Scrapes, your plugin is the best with the most functions!!

I have 2 questions,

1. i crawl sales products from some webshops. I need an option to get an input for an custom field. The field is an end date off sale coupon.

I need to generate: “Crawl date + 7 days” for example. Do you have any idea for this to fix. Output must be 2019-03-08.

2, I try to make title off product with first letter capital i found next: functx:capitalize-first(‘[wcc-main-title]’)

This is not working, is there an way to get that working also?

There is no function to show date, unfortunately. Could you please tell me what you mean by “last crawl date”? I will add this as a feature request but I need to understand what you mean exactly.

I mean the current date. Date off scraping.

I added this as a feature request. Thanks.

Merhaba 3 ay önce ürünü aldım. Bugün lisans süresinin dolduğunu belirten bir mesaj gördüm. 6 aylık bir destek uzatması aldım fakat aynı uyarı mesajı hala durmakta. Ne yapmam gerekir.

Profil sayfamdaki iletişim formunu kullanarak lisans anahtarınızı gönderebilir misiniz? Kayıtlı domaininizi sileyim, lisans ayarlarınızı tekrar kaydetmeyi deneyin.

Yardımınız için teşekkürler, düzeldi.

Rica ederim, düzelmesine sevindim.

Merhaba, siteyi test et butonuna bastığım zaman

An error occurred. 502 Bad Gateway The server returned an invalid or incomplete response.

hatası alıyorum ne yapmam gerek acil yardımcı olursanız çok sevinirim, teşekkürler.

Size zaten dönüş yaptım. Sürekli hızlı bir dönüş beklediğinizi belirtmenize gerek yok.

Anlıyorum ama size yolladığım siteyi test ederken bende hep 502 Bad getaway hatası verdi

Hem burdan hem de mailden aynı şeyi neden yazdığınızı anlayamıyorum. Size her iki platformdan da aynı şeyleri söylemek tam olarak vakit kaybı. Ya buradan yazın, ya da mail atın lütfen.

hello, pre-purchase question, the plugin contains a field to put an featured image from the media? I want all articles to have a unique and predefined photo (photo media link) and I don’t want to download those presented in the article. is it possible?

Hi,

No, the plugin does not have that feature, unfortunately. You can maybe use find-replace options to add an img element into the target site’s source code and write a CSS selector that finds that image element as featured image URL. However, the plugin will still save it even if it exists in your site.

Hello,

I have an RSS feed for my Deviant Art group, however the feed works fine but sometimes the Mature filter stops all the content from getting to my site I need a cookied session to be recorded to prevent this from happening can your plugin do this?

The plugin’s support does not cover configuration of settings, unfortunately.

ah so what if I can’t get it working..then what?

In that case, if you think that the plugin has a problem, you can ask for a refund. If you do not know how to use a feature, you can ask how to use it.

Hi,

I really think this plugin can solve my problem. However, I am not deep into programming and CSS selectors. Would you be available to help for a few sites in return for an extra donation? I want to crawl articles on 5-6 pages. I want to display the search results on those pages on my website. Much appreciated, Nihad

Hi,

Unfortunately, I do not provide configuration services even if money is involved.

Hello, I need a plugin that allows me to add 100’s of urls for single product pages with the ability to add specific taxonomies to each individual url (product). I need a way to do this in bulk without having to do each url manually. This will be used for Woocommerce products and I would like to have it so that the image urls are not downloaded to my server. Is this possible with this plugin?

Thank you. Lastly, is it possible to import urls via wp all import or with CSV. files?

No, it is not possible to do so using them. However, you can use Tools page to manualy import URLs. You can enter new line separated URLs in manual crawling tool and click “add to database” button. After that, the plugin will crawl those URLs using scheduling options you defined.

That sounds about perfect…. I will be purchasing tonight when I get home from work. thank you

Hello, I’m interested in your plugin.

can you make a tutorial for scrap UDEMY website?

Hi,

You can watch the tutorials available in this playlist to learn how to use the plugin.

turgut hocam bu bot kendını guncellıyor degılmı,extra bırsey yapmaya gerek varmı

turgut hocam guncellemıyor acaba kopya yazı olarak goruyor ondanmı,cunku başlık aynı yazıda

manuel olarak guncellıyor

Lütfen eklentinin genel ayarlar sayfasında bulunan yazı güncelleme ayarlarının doğru ayarlandığından emin olun. Diğer destek talepleriniz için lütfen satın aldığınız desteği güncelleyin.

merhabalar,yazı baslıgının sonuna günün tarihini nasıl eklerım acaba

turgut hocam bir sorun daha olucak, içerik çekilen site yeni yazı eklediği zaman o yazılarda eklenıyormu

Merhaba,

Hedef siteden herhangi bir veriyi kısa kod olarak kullanabilirsiniz. Kısa kod tanımlamak için “Yazı > Özel Kısa Kodlar > Kısa Kodlar için İçerik Seçiciler” seçeneğini kullanabilirsiniz. Tarih değerini bir kısa koda atadıktan sonra, “Şablonlar > Yazı Başlığı Şablonu” seçeneğini kullanarak başlığa ekleyebilirsiniz.

Eklenti hedef sitedeki kategori sayfalarını programlama seçeneklerinde belirtilen zaman aralıklarında kontrol eder. Eğer yeni bir yazı varsa, URL’sini sıraya ekler ve yazı kaydetme zamanı geldiğinde de yazıyı kaydeder.

Hi there I want to replace first 3 letters of an output. can you help me please

Hi,

You can do it by using find-replace options with regular expressions. Here is an example: https://regex101.com/r/QUlfXj/2 To use this, go to the find-replace option, check “regex” checkbox, write the regex into “find” input, and write ”$1” into “replace” input. Configuration of the settings is not covered by the support, unfortunately. You can find many tutorials about regular expressions on the web.

Hello again. Is that possible to remove link text in post source?

For example in this picture: https://drive.google.com/file/d/15AD2sAW2bTDwgHLOmil7BbNgE9177b6X/view?usp=sharing. I would like to remove all links from the text link but keep the text.

To remove links, you can check “Templates > Quick Fixes > Remove links from short codes” option.

Thank you for your help.

Pre purchase question. Can I add a url after product url? So I want to take content from a site and add my affiliate url to the end of each url. Is this possible? I’ll probably have more questions but will start with this :)

Hi,

Yes, it is possible.

Hi, i setup the plugin correctly and started to crawl the pages… but i see that in the first crawl doesnt save the featured imagen and the other images on the post.

If i go tools -> manual recrawling then it saves correctly….

My question is if you wonder why doesn’t save correctly the images the first time… or how can i manually recrawling all the posts ids with a cron job.

Best Regards,

Hi,

You might have configured the settings such that the plugin retrieves the featured images from the category pages as well as from the post pages. If your category settings are not correct, the plugin might have been matching the post URLs and featured images incorrectly. So, please check your category settings using the tester. If the featured images are not matched correctly, check “Category > Featured Image > Post links come before featured images” option and try again. Note that after you change the category options, you should invalidate already-collected URLs so that the plugin crawls them correctly this time. Otherwise, previously-collected post URLs will still be saved using an incorrect featured image. You can invalidate the URLs from “Tools > URLs > Clear URLs” tool.

If you want to manually recrawl posts, you can use “Tools > Manual Recrawling > Manually recrawl (update) a post” tool. You cannot manually crawl the posts with a CRON job since the plugin does not have a feature for that purpose. CRON jobs are used to do things automatically, not manually.

Hi,

I bought your plugin but did not receive the license. It does not exist in the zip file. Please tell me where I can get it.

Hello

can the plugin grabbing from PHP websites ?

Hi,

Language used in the backend does not matter. The plugin can crawl any HTML code.

hi,

i see the description page says your plugin is able to parsing Json. how to achieve this ? i did not find the Json option for css selector .

i ask this is because we plan to use your plugin to crawl a api json feeds.

waiting for your reply

Hi,

You can watch this to learn how to extract JSON values: https://www.youtube.com/watch?v=2MYWzq_oAig&t=727s

Dostum işte alıcılara cevap vermemek için burda script yazmayı bıraktım. gerçekten sabır gerektiren bir iş :))

by
by
by
by
by
by

Tell us what you think!

We'd like to ask you a few questions to help improve CodeCanyon.

Sure, take me to the survey