Adjust the Speed of the Indexer

By default SearchWP’s indexer is configured to run as fast as it can without causing your site to run slowly. Part of this configuration involves sane defaults and system monitoring (e.g. CPU/RAM usage) when applicable.

If you are confident that your server is more capable than most, there are a number of hooks you can use to forcefully cause SearchWP’s indexer to be more aggressive.

Note: there is a checkbox built in to the Advanced tab of the SearchWP settings screen that decreases indexer aggressiveness. This article explains how to make the indexer more aggressive.

Restrict Engine Source Attributes

Before experimenting with the indexing process directly, it is worth evaluating your Engine(s) configuration. The more work the indexer has to do, the longer it will take.

For example: adding Any Meta Key to index all Custom Fields is a convenient thing to do, but it also means all postmeta will be indexed, including the content you have in mind but also the meta implemented/used by all of the code on your site. Very often metadata is used to store data/content that is completely irrelevant to search, so it is worth reviewing.

Further, if there are any Rules you can apply so as to limit the work the indexer is doing, that will be worth considering as well. The less content SearchWP has to process to build its index, the faster the index will build.

Increasing indexer batch size

SearchWP’s indexer runs by batching Entries to index. This avoids timeout and potential memory issues. Increasing the number of Entries processed per batch can help SearchWP’s indexer to run a bit faster.

There is a hook available to control the batch size used by SearchWP’s indexer: searchwp\indexer\batch_size

You can experiment with this batch size to have SearchWP process more data per indexer iteration. Note that increasing the batch size too far can result in indexer process failure.

Disable indexer throttling

SearchWP will monitor server resource usage when possible. There are a few hooks you can use to adjust how SearchWP reacts to the current CPU load during indexing.

Disable load monitoring entirely

You can completely disable load monitoring with the searchwp\background_process\load_monitoring hook.

Adjust the load maximum

When SearchWP monitors CPU load, it checks to see if a maximum CPU load threshold has been breached. The default for this load is 2 but if your server has many cores, this default may be too aggressive. You can adjust the threshold with the searchwp\background_process\load_maximum hook.

Adjust the throttle time when CPU load threshold is breached

When SearchWP has detected that CPU load is too high, it will inject a timeout in seconds. If you’d like to customize the duration of this timeout you can use the searchwp\background_process\load_throttle hook.

Proceed with caution!

Please keep in mind that experimenting with these hooks can cause your server to be fully consumed by SearchWP’s indexing process and potentially interrupt site visitor traffic while the indexer is running. Incrementally adjusting return values for these hooks is recommended.