WP Content Crawler logo
WP Content Crawler

Get content from almost any site, automatically!

WP Content Crawler is a WordPress plugin that helps you get content from other websites, into your website.

Now with ChatGPT!
Save anything.

30-second* site setup with Config Helper

Config Helper greatly speeds up the initial site setup

*Config Helper is displayed when creating a new site. Its purpose is to speed up the initial setup for a site. Although Config Helper works for many sites, not every site can be configured this easily, meaning that certain sites may require a manual and more elaborate setup. Even for those sites, you can try to create the basic configuration with Config Helper and then modify the site settings later. Additionally, it is possible to exit Config Helper any time or disable Config Helper entirely.

How it works

It is all about CSS selectors

1
Websites contain many sections
2
Sections are defined with HTML elements
3
HTML elements have classes and attributes
4
CSS selectors are used to select specific elements by using their classes and attributes
5
You specify which element to use via CSS selectors
6
CSS selectors can be found with Visual Inspector by clicking to elements
7
After settings are configured, the plugin collects URLs and saves posts automatically via WP-Cron
Websites contain many sections

Visual Inspector

Find CSS selectors of elements by clicking to them

Find unique elements

Find unique elements

Unique CSS selectors select only one element. This is useful, for example, for finding titles and featured images.

Find similar elements

Find similar elements

The "similar" mode finds a CSS selector that selects the items similar to the clicked item. This is useful, for example, for finding post tags and product attributes.

Find containing elements

Find containing elements

The "contains" mode finds a CSS selector that selects the element containing all the clicked elements. This is useful, for example, for finding post content and excerpt.

See the visual inspector in action
This video demonstrates the selection modes of the visual inspector.

Automatic

Configure the settings and let the plugin handle the rest

Auto embed social media posts

Media from 70+ domains are automatically converted to embed short codes

Examples of social media posts embedded by the plugin automatically

HTML code of social media posts existing in the target post page is automatically converted to short codes. By this way, they are displayed on the front end of your website correctly. Additionally, all the iframe elements are converted to a short code. An iframe short code is displayed only if its source is trusted. If the source of an iframe is not trusted by default, you can manually add a trusted domain so that the iframe is displayed. With this method, you can display media from third-party sites securely. Websites whose media are embedded automatically include Instagram, Imgur, YouTube, Vimeo, TikTok, Kickstarter, Twitter, Pinterest, and so on.

Manipulations

Prepare the data in the way you want

Find and replace

Find and replace

Find and replace anything in the target page. You can also use regular expressions. You can use dedicated options to find and replace in the post title, content, tags, post meta, and others…

Learn more about finding and replacing
Advanced text manipulations

Advanced text manipulations

Make text upper case, lower case, title case, camel case, snake case, kebab case, and more… You can also limit the length by character count or word count.

Learn more about advanced text manipulations

Test everything

Easily test your configurations to make sure everything works as expected

Quick tests

Quick tests

Use the magnifier button to quickly test the configurations. You can test find-replace options, CSS selectors, formulas, filter commands, and more… You can also enable caching for the quick tests to make them even faster.

Warnings

Warnings

When a configuration is wrong, you get a warning that tells you why it is wrong. By this way, you can fix it right then and there.

Test the settings altogether

Test the settings altogether

The Site Tester page helps you check if your configurations work as expected. You just enter a category or post page URL and hit the test button. The plugin crawls the page and shows you all the details it finds in the page, along with some performance information and, if any, warnings.

Learn more about testing the settings
See the Site Tester in action
This video demonstrates the features of the Site Tester page.

Transform, translate, and spin

Automatically transform, translate, and spin specific parts of the posts

Transform

Transform

You can transform the parts of the post by using paid OpenAI GPT models, like ChatGPT. Use the cutting-edge AI models to generate SEO-friendly summaries, insert interesting facts, rewrite posts, fix grammar errors, and do anything else that you can imagine.

Learn more about transformation
Spin

Spin

Spinning is like paraphrasing, keeping the same meaning while using different words. You can automatically spin the posts by using one of the following paid services. You can also choose which parts of the post should be spun.

Learn more about spinning

Do things conditionally!

Perform certain operations if a certain condition is met

A filter that assigns the status of post as "draft" if the title has less than 20 characters

The filter above assigns the status of the post as "draft" if the crawled post's title has less than 20 characters. Using many condition commands available in the plugin, you can create complex conditions. You can combine the conditions with "and" and "or" operators as well. When the condition is met, you can do multiple things, such as sending an email notification to yourself, adding the post to a specific category, finding and replacing things, assigning a specific featured image, calculating a value, and many more.

Learn how to use the filters
This video tutorial explains the filters in detail, with examples

Save WooCommerce products

Use the options dedicated to WooCommerce products to easily save the product details

Dedicated options

Dedicated options

The WooCommerce section's design is similar to the design of the original WooCommerce options. Here, instead of entering the values, you define CSS selectors. You can create simple and external products. You can also mark the product as virtual and/or downloadable. You can retrieve the downloadable file from the target page as well.

Learn more about WooCommerce options
Dedicated test results

Dedicated test results

When you test a product page via the Site Tester, the test results contain a dedicated WooCommerce section where you can see all the product details retrieved from the target page, so that you can ensure that the products will be created as expected.

Crawling WooCommerce products
This video tutorial walks you through configuring the site settings to crawl products from eBay

Save custom post types

Save the crawled posts as custom post types instead of regular WordPress posts

Select any post type

Select any post type

Select the type of post you want to create, if you do not want regular WordPress posts. All the post types registered in your WordPress site are available. You can select a different post type for each site or set the post type for all the sites from the general settings.

Learn more about saving custom post types
Save custom fields (post meta)

Save custom fields (post meta)

Custom post types typically keep extra information in their custom fields. You can save the values of the custom fields in plain text format so that the custom post type can use them accordingly.

Learn more about saving custom fields
Save taxonomy values

Save taxonomy values

Custom post types might also keep extra information in specific taxonomies, such as tags, company name, ISBN number, and so on. You can save the values of the taxonomies in plain text format so that the custom post type can use those values in the correct places.

Learn more about saving taxonomy values
Use custom post type categories

Use custom post type categories

Custom post types might have their own category taxonomy. You can save the posts into those categories by defining the category taxonomy in the general settings. Then, you can select those categories in the Category URLs setting. The rest is handled by the plugin.

Learn more about using custom post type categories

Track what's going on

The dashboard page helps you understand what's going on in the background

Track active sites

Track active sites

See the active sites, when they were crawled the latest, how many posts were crawled, how many URLs are in the queue, and more…

Learn more about the active sites
See what's happening

See what's happening

See when the URL collection, crawling, deleting, and recrawling events will be executed and for which site. You can also see the total number of posts in queue, crawled, recrawled, and deleted. While a post is being crawled or recrawled, it is also displayed in this section.

Learn more about seeing what's happening
Last crawled, recrawled, and deleted posts

Last crawled, recrawled, and deleted posts

See the latest posts that were crawled, recrawled, and deleted. You can also make the dashboard page refresh in specific time intervals to follow the events as they happen.

Learn more about recently crawled posts
See the Dashboard page in action
This video observes the dashboard page while the plugin crawls posts

Options box

Manipulates the value found by a CSS selector

Available for many settings

Available for many settings

The options box is displayed by clicking to the "settings" button, shown in the image with a blue border. The button is available for many settings that define a CSS selector.

Learn more about the options box

Handle JSON with ease

Convert JSON to HTML and work with it just like other HTML elements

Convert JSON to HTML via your custom template

Convert JSON to HTML via your custom template

You can selectively retrieve parts from JSON data and put them into your custom HTML template. By this way, you can recreate an entire site defined as JSON. This makes it possible to retrieve data from pages that define their entire page as JSON and convert it to HTML later via JavaScript.

Learn more about converting JSON to HTML via a template

Proxies

Use proxies to access sites that block your server

Define proxies

Define proxies

You can define one or many proxy servers. While crawling, the plugin tries them one by one until it reaches the target website. The plugin uses these proxies every time it makes a request to the target site.

Learn more about using proxies

Manual crawling

Crawl posts manually with a powerful tool

Crawl multiple posts

Crawl multiple posts

Enter a post URL in a new line and hit the "Crawl now" button. The tool will crawl all the posts whose URLs are entered.

Retrieve post URLs from category pages

Retrieve post URLs from category pages

When you do not want to enter the post URLs manually, you can enter the URLs of category pages. The plugin extracts the post URLs from the category pages and crawls each found post URL.

Crawl in parallel

Crawl in parallel

The manual crawling tool is capable of crawling multiple posts in parallel. Just define how many posts you want to be crawled at the same time.

Add post URLs to the queue

Add post URLs to the queue

If you do not feel like crawling right now, you can choose to add the post URLs into the queue manually. Then, the plugin crawls them by using your scheduling options.

See the manual crawling tool in action
This video demonstrates the manual crawling tool by crawling a few posts manually

Notifications

Send yourself an email when something happens

Custom notifications

Custom notifications

By using the filters, you can define your custom conditions and send yourself a custom email when those conditions are met. You can include the URL of the post, HTTP status code of the response, name, ID and edit URL of the site, and current time in your email. You can also leave the email template empty to use the default template provided by the plugin.

Learn more about custom notifications

Interactive guides

See how to do things, step-by-step, interactively

Learn what to do

Learn what to do

The steps of the interactive guides show you exactly which setting to configure and how to configure it. The steps also check if the setting is correctly configured and show you what to do to fix the error when there is an error.

Start any guide whenever you want

Start any guide whenever you want

At the bottom right corner of the plugin's pages, there is a "Guides" button. When you click it, you see the list of interactive guides available in the plugin. From there, you can search for a guide and click its "play" button to start it. You can also start a guide from a specific step.

Watch the quick start guide
This video tutorial follows the interactive quick start guide and explains it along the way

and more…