Discussion on WP Content Crawler - Get content from almost any site, automatically!

turgutsaricam

turgutsaricam supports this item

Supported

This author's response time can be up to 2 business days.

170 comments found.

warlordsp

warlordsp Purchased

Selam Turgut.

Yeni sürümle birlikte türkçe kararkter sıkıntısı yaşıyorum. Çekilen tüm gönderilerde türkçe karakterler yerine saçma sapan işaretler çıkartıyor.

Merhaba,

Yeni sürümde karakter kodlamasıyla ilgili bir değişiklik yapılmadı fakat içerik aldığınız sitenin kodunda böyle bir değişiklik olmuş olabilir. Şurada tarif edildiği şekilde yaptığınızda büyük ihtimalle sorununuz çözülecektir. Bir de burada sadece İngilizce konuşulması isteniyor. Dilerseniz bundan sonraki sorularınızı bana profiliminden mail gönderebilirsiniz.

Hi,

There is no change in character encoding in the last version. However, the target site may have changed its character set. You can follow the instructions here to fix the encoding problem.

yafidok

yafidok Purchased

hello turgutsaricam, can you add feature for proxy tester, because if i typo / wrong input proxy this plugin is still running, thanks for your great plugin

Hi,

I added this to my todo list. Thanks.

Is the recrawling feature added to this update?

Hi,

Unfortunately, it is not in this update. But, it is in my todo list and I’m planning to implement it when I have the time.

seohawks

seohawks Purchased

Hi,

How many website I can use this plugin

Hi,

You can use your license key to actively use the plugin on one domain only. If you want to use the plugin on another domain, you just need to disable it from the previous one.

seohawks

seohawks Purchased

Thanks

sinayes

sinayes Purchased

NEED CSS SELECTOR ON PAGE. Hard with manuel select!!!

hginli

hginli Purchased

hello

Can you Cookie support? Set which cookie values to be sent to source url for every requests for cookie required websites. ?

thank you very much

Hi,

I added this to my todo list. I’ll implement this feature when I have the time. Thank you.

MadeByAIM

MadeByAIM Purchased

Hey Turgut, after installing the update, i receive these errors on Edit Post:

TypeError: _.debounce is not a function

TypeError: _.memoize is not a function

TypeError: wp.media.gallery is undefined

The cause of this is probably that Lodash library the plugin uses overrides another Lodash or similar library that is loaded on the edit page by another plugin. I’ll remove Lodash in the next release. The problem will probably be fixed then. However, even in this case, this is a problem caused by another plugin.

MadeByAIM

MadeByAIM Purchased

Yep, it’s a conflict between the crawler and anther plugin. Any temp fixes until the next release?

In wp-content-crawler/app/AssetManager.php, delete line 135:
$this->addLodash();

In wp-content-crawler/app/public/scripts/dev-tools.js, line 423 – 428

Replace this:
alternatives = _.uniq(alternatives)
    .sort(function(a, b){
        // ASC  -> a.length - b.length
        // DESC -> b.length - a.length
        return a.length - b.length;
    });
With this:
alternatives = alternatives
    .sort(function(a, b){
        // ASC  -> a.length - b.length
        // DESC -> b.length - a.length
        return a.length - b.length;
    });
I would like to be able to automatically crawl a site (https://www.defense.gov/News/Contracts) and pull everything off of that page into individual posts (e.g. https://www.defense.gov/News/Contracts/Contract-View/Article/1051724 would automatically be made into a post on my site). I think I am confused by the Category section, because I can’t seem to figure out how to make it do this. These are my current settings that are not working:
Category List Page URL: https://www.defense.gov/News/Contracts
Category List URL Selectors: #dnn_ContentPaneLeft .alist .item a
Category Map (this is confusing to me) : _my selected category_ https://www.defense.gov/News/Contracts

When I run the category test I get nothing, but when I run individual posts it works perfectly.

Thanks in advance for your help

Hi,

Write https://www.defense.gov/News/Contracts to the Category Map and select a category for it. Then, write

#dnn_ContentPaneLeft .alist .item a
to Category Post URL Selectors.

Category List Url Selectors are used to add the category links automatically to the category map. For example, “News”, “Military Services”, “Resources” are categories. When you write a URL selector in Category List URL Selectors that selects these category links, and then press the plus button, the URLs of “News”,”Military”, “Resources”, etc. will be added to the category map automatically. Category map defines which category URLs correspond to which categories in your site. What gets the URLs of the posts in a category page is Category Post URL Selectors, which should contain URL selectors for posts, as its name indicates.

dear i get the following error :

Parse error: syntax error, unexpected ‘class’ (T_CLASS), expecting identifier (T_STRING) or variable (T_VARIABLE) or ‘{’ or ’$’ in /home/eroticasupplies.nl/public_html/bg-sjop.nl/wp-content/plugins/wp-content-crawler/app/Factory.php on line 27

can you explain how to solve this error ? kindest regards Bart

Hi. Your plugin looks great. Before I purchase I just need to check something with you. I want to pull a timetable of events from a page (https://www.engage.england.nhs.uk/) and have them appear on a page on my site. I don’t want them each on a separate page, the whole table displayed will be good. And I have other websites to add to the same page too showing other timetables. Is this possible with your plugin? Thanks.

Hi,

Thanks. I’m glad that you like it. The plugin can be used to save each event as a post. Showing them in a list or as a timetable cannot be done with the plugin, because it is an entirely different functionality. However, you can use another plugin or a widget to show saved posts (events) in a table. You can also save their dates as post meta, and using those values, you can order the events by their date or organize them the way you like.

by
by
by
by
by
by