Hi,

I think the easiest way to do this is to create another element that contains the tag as its text. Here is how you can do it:

1. Go to Post Tab > Manipulate HTML Section > Find and replace in element HTML setting and configure it like this:

   Regex: Checked
   Selector: script
   Find: \barticleTags\':\'([^;]+)\b
   Replace: </script><span id="my-tag">$1</span><script>
   

   This should convert the script element containing the tag to something like this:

   <script type="text/javascript">var ......
   window.dataLayer=window.dataLayer||[];dataLayer.push({'pageCategory':'1001009'});dataLayer.push({'pageType':'Article'});dataLayer.push({'pagePlatform':'Web'});dataLayer.push({'pageSubcategory':'Thường thức'});dataLayer.push({'articleId':'4510532'});dataLayer.push({'articleTitle':'Những nền văn minh lâu đời nhất hành tinh'});dataLayer.push({'articleAuthor':'1700000248'});dataLayer.push({'articleAuthorName':''});dataLayer.push({'articlePublishDate':'20220914200000'});
   
   ...
   dataLayer.push({'
   </script>
   
   <span id="my-tag">nền văn minh, nền văn minh lâu đời nhất, Ai Cập cổ đại</span>
   
   <script>'});
   ...
   
   dataLayer.push({'tag_id':'51467, 1527186, ...https://vnexpress.net/nhung-nen-van-minh-lau-doi-nhat-hanh-tinh-4510532.html"/>
   ....
   </script>
   

   In other words, by making this change, you add the following element:

   <span id="my-tag">nền văn minh, nền văn minh lâu đời nhất, Ai Cập cổ đại</span>
   

2. Then, go to Post Tab > Post Tag Selectors setting and configure it like this to select the content of the newly created element’s text as a post tag:

   Selector: #my-tag
   Attribute: text
   

Please note that I could not check if this works as intended since you did not provide the URL of the page that contains the script element.

I did not realize that the URL in the script was for the post So, I checked the page’s source code. Actually, the tags exist in other places as well, meaning that you can simply go to Post Tab > Post Tag Selectors setting and configure it like this:

Selector: meta[name="keywords"]
Attribute: content

Like this: https://ibb.co/wsG9s3z

This is the website link https://vnexpress.net/nhung-nen-van-minh-lau-doi-nhat-hanh-tinh-4510532.html

It looks like you missed my previous reply. You can use the meta element that contains the tags.

Thank you for your reply, btw, I set Exchange element attributes for img from data-src to src, but test not work https://ibb.co/tbXMPCj https://ibb.co/mCkKmHy

Tester: https://ibb.co/02TVfmC It just get old src?

There can be two causes of this. The first one is that you might not have saved the settings. The other one is that there might be other elements in the page that already contain the correct image URL in their “src” attributes. You can specifically target the elements whose src attribute contains a data URL via a CSS selector like this in the Exchange element attributes setting:

.fig-picture picture .lazy[src^="data:"]

The CSS selector above should select the elements whose src attribute’s value starts with “data:”.

It’s not work! I have already updated setting every times

I do not see any issues in the demo as it can be seen in this screenshot: https://ibb.co/pxwvQrC Could you please import your settings to the demo and check if the same issue happens there as well?

When I change post content selectors to ”.fck_detail” (first is ”.Normal”), it’s worked! Thanks for your support

All right. Thanks for letting me know.

Your welcome! and why it’s not save the post? Just queue the links? https://ibb.co/DD7cZGg

You can see this page to troubleshoot automatic crawling issues.

Hi,

Thanks, I am glad to hear that.

1) The plugin downloads the images as-is. It does not change the size or quality of the images. If the image URL found in the target site is for a high-quality version of the image, then maybe your theme displays a lower-quality version of the downloaded image, as WordPress might be creating different sizes for the image when it is saved. If that is the case, you probably need to configure your theme to use a high-quality version of the saved image. Otherwise, if the URL of the image is for a low-quality version, then you need to make the plugin find the URL of the high-quality version of the image, so that the plugin can download the high-quality image from the target site.

2) There might be a few reasons for this. I just crawled a post in the demo site of the plugin, and the post can be commented on. This indicates that the issue is either related to your configuration or your theme. If you are using custom general settings for the site, the checkbox of “Allow Comments” setting might not be checked there. If you are not using custom general settings, and the “Allow Comments” checkbox is checked in the general settings, then maybe your site or your theme is configured to not allow comments.

If you previously crawled a post when the “Allow Comments” checkbox was not checked, and you checked the checkbox after that, the previously crawled posts will not be automatically configured to allow comments. You need to recrawl them.

Thank you for your fast reply.

1) The source url is for a high quality image, however the image saved is 300px wide. So guessing it’s something in this step rather than theme template (using Elementor Pro theme builder for this). I have no image compression plugins running… any ideas?

2) “Allow comments” is checked. The post itself is allowed to be commented on. Theme is configured to allow comments. I guess it’s something to do with Elementor theme builder displaying – just very weird it works perfectly on any native posts…

Any thoughts?

Thanks again. M

Just as a further point… on 2) When I edit with Elementor on the Single Post template on one of the freshly crawled posts, it has an error that says: Comments are closed. Switch on comments form the discussion box or the post edit screen… both of these are on and were on when the post was imported.

And, the related posts doesn’t appear, which is normally inserted by Jetpack…

Thanks again for your help on this

As I said, the plugin does not change the size or quality of the downloaded images. It is not capable of doing that. There is no code in the plugin for resizing an image. Hence, it is impossible for the plugin to do that.

About the comments, I checked it in the demo site, and it works without any issues. Maybe it is related to another plugin or your theme. Have you tried disabling other plugins, converting your theme to one of the official WordPress themes, and then checking if the issue still happens?

Hello, just replying to this as I still haven’t found a resolution.

I’ve disabled all plugins, reverted to original wordpress theme, spoken to Elementor support, who say it must be your fault of course… I don’t know what to do…

The featured image that is scraped is low quality on the new site. This is the major issue.

The comments do not work on most of the scraped posts (it seems they do on some).

Everyone just keeps finger pointing and blaming another plugin. I’ve also had to buy Content Crawler for 2 sites, when all i need it to do is crape one post for each site approximately once per month. So I have 2 licenses, both not working properly.

Can you help me?

Could you please import your site settings to the demo site and try to reproduce the issues you are having? Do the same issues happen there as well?

hello, thanks for your speedy reply – appreciate the help.

i imported the scraper settings and on the post it scraped, no featured image at all?

comments are there

no pictures seem to be scraped inside the content either

Could you please tell me the name of the site settings you created in the demo so that I can check it?

MM Test

The target website stores the original image URLs in “data-src” attribute of “img” elements. However, browsers can only display the images if their URL is in the “src” attribute. Also, the plugin can save the images by using the URL defined in their “src” attribute. Therefore, you should put the image URLs existing in the “data-src” attributes into “src” attributes of the “img” elements. You can do it by simply going to Post Tab > Manipulate HTML Section > Exchange element attributes setting and configuring it like this:

Selector: img[data-src]
Attribute 1: src
Attribute 2: data-src

You can also see this screenshot: https://ibb.co/g4QDsbd

Once I do that, the test results are shown like this: https://ibb.co/RcgpXRn As you can see, the image in the post content is shown. Also, the featured image is saved. Its dimension is 2560×1920.

You should always test your site settings by using the Tester page to catch this type of issues and fix them.

great thanks! and what about comments sometimes not working? on this post The Chief Explorer’s Almanac on the demo site they don’t work for example.

I can see that the comments are enabled for that post. Could you please further explain what you mean by that?

By the way, I saw that you also configured the plugin to save the featured images from the category pages. If the featured images are available in the post pages, the recommended way is to save them only from the post pages. If you still want to do that, you should also exchange the “src” and “data-src” attributes’ values in the category pages.

sorry, i’m still not familiar with all these terms. I would prefer to scrape the featured image from the post than the category – what vaules do I change where?

yes, you’re right comments are working. they worked on the last post scraped, so maybe this was settings I had wrong in the begin. i’ll keep an eye on the comments and reach out if I have any issues. really appreciate your help

a great plugin!

You can simply disable the crawling of the featured images under the Category tab by unchecking Category Tab > Featured Images Section > Save featured images? setting’s checkbox.

amazing thank you – you’re a legend!

Thanks. I am glad it is resolved. By the way, since the site settings on your site was configured to crawl featured images from the category pages, you should delete the already-saved post URLs from the database by using the Tools Page > URLs Tab > Clear URLs tool. You should delete all URLs for the site. Please do not forget to check the “I’m sure” checkbox. The reason you should do this is to clear the featured image URLs stored in the database. Otherwise, the plugin will keep using those wrong featured image URLs. The plugin will save the post URLs from the category pages again, that time without storing the featured image URLs.

ok great, thanks. is there a way of asking the plugin to rescrape the existing posts to update the featured image to a new high resolution one?

I see that you already enabled the recrawling. When it is enabled, the plugin updates the posts. However, since you previously saved the featured image URLs from the category pages, it doesn’t update the featured images. So, once you disable crawling of the featured images from the category pages and delete the existing URLs from the database, the plugin will update the featured images by using the settings under the Post Tab.

My old domain has been lost and cannot be deactivated

Could you please send your purchase code via the contact form on my profile page so that I can remove the domain registered to your license?

I wanna crawl title in playlist https://www.youtube.com/playlist?list=PLDYlX8j8IhZJOyLfxg5BF1aVdFMAAujRy can I help me?

The plugin’s support does not cover configuration of the settings, unfortunately.

I see title in the JSON How to pick value?

Maybe you can use the “title” element instead of the JSON, since it is easier to retrieve the “title” element because the post title selectors setting does not have an options box that can be used to retrieve a JSON value.

I tried use a#video-title this isn’t work

That element does not exist in the page when JavaScript is disabled. Maybe you can use meta[property=”og:title”] selector and “content” as its attribute. For more information on what data you can retrieve from a page, you can see this page.

It still error, but I want to list title in the playlist. Not only meta title page. https://ibb.co/N2zB8nC

Could you build a short tutorial for a crawler for this case?

First of all, the double quote character used in your CSS selector looks like it is different from the regular double quotes. Also, fixing the double quotes does not result in the playlist title, as I saw when I tested it in the demo of the plugin. The reason is that YouTube does not include the playlist title when I make a request via the demo. It looks like the site sends different page contents in certain cases. Hence, the plugin cannot get the title that does not exist in the response sent from the target site.

I just got 1 value in script, can I get multiple values at the same time?

I do not understand what you mean, unfortunately.

I take a title in JSON. I want to take many title. Can you help me?https://ibb.co/xXsVPTt

That setting is for finding post URLs in category pages. I do not understand why you configured it to find a post title.

I just want demo, You can ignore it. You would like focus support for many title

Could you please explain what exactly you want to do after finding multiple titles? Also, do you want to find multiple titles in a category page? If so, you cannot save titles from category pages.

I know in a section category need URL. I want to crawl multiple title for content.

I also want to URL for category. By the way, you can support for both cases

If you want to retrieve multiple values from a JSON object, you can create different custom short codes, as it is explained here and include all of them into the post content. For example, if you want to retrieve 10 values from a JSON object, you can define 10 different custom short codes where each finds one of the 10 values you want. Then, you can simply add them into the post content as it is explained in the documentation.

If you want to retrieve the URLs from JSON objects in category pages, you can use the options box of Category Tab > Category Post URL Selectors setting. You need to add a different rule for each of the values.

I want to replace but not work, can you support for me? https://ibb.co/L93c1zJ

For the find input’s value to be considered as a regular expression, you need to check the “regex” checkbox. For more information, you can see this page.

I have retrieve url pages: https://ibb.co/HrZpSXr. When I crawl, tool just crawl first page: https://ibb.co/m8Zpv0z. I can do multiple page?

The manual crawling tool does not automatically retrieve the next pages of the categories. You need to explicitly define the category URLs in the manual crawling tool.

How to retrieve all post of the categories if it have next page?

You can enable automatic crawling.

Can you give me link to the tutorial?

Sure. You can see the video tutorial or the written tutorial. The other guides are available here. The documentation of the plugin is available here. The other video tutorials can be seen here.

when I test and manual crawling post has retrieve iframe but when I run autocrawling post has no iframe. I don’t know if I missed something during setup? What makes manual and autocrawling different?

You can see this FAQ.

I click save but my actions are not saved. I lost all my settings like when I didn’t have license

The plugin is extensively tested via automated browser tests before every release, by using the plugin just like a user does, such as changing inputs, clicking buttons, and saving settings. These tests are run in multiple environments. Hence, I can say, in confidence, that the plugin does not simply lose all the settings. If your settings are partially lost, you can see this troubleshooting item to learn how to avoid it in the future. If your settings are completely lost, then the issue might be related to another plugin, your theme or your browser extensions. You might try to disable other plugins, convert your theme to one of the official WordPress themes, and disable all the browser extensions. Then, once you see that the plugin’s settings are saved without any issues, you can enable the previously disabled third-party software one-by-one and, after you enable each item, try to save the settings of the plugin again. Once you see that the settings are not saved as expected, you can say that the cause of the issue is the last item you enabled.

Error when I add many URL category https://ibb.co/vh1Jk7s . Maybe the reason is my server. Thanks

In HTML have many have class the same. I want to retrieves content in .... of the 2nd. How do i get the content?

You can use the nth-child selector or + operator such as ”.class + .class”. Because the CSS selectors are not a feature that is developed by the plugin, finding CSS selectors is not covered by the support, unfortunately. You can find many tutorials about CSS selectors on the web.

the nth-child selector or + operator => You can give me link tutorial?

I do not know a specific tutorial, unfortunately. You can find many tutorials on YouTube or just by searching on the web via Google or another search engine.

I want add id=”preload” for this script https://ibb.co/ySWGxmh I tried setting as below: https://ibb.co/xftXZXh But not work