In version 2.1 of MyCurator we introduced a new background process called Request Processing. How do I know if I need to use it for my site? The short answer is that it will probably benefit every site.
How the New Request Process Works
The previous background process would send each article in your Source feeds to the cloud process one by one. It would wait for the cloud process to grab the full text from the page and process the filters and relevance engine. Because the original web site had to be accessed, this meant that the background process would be waiting for a couple of seconds on average for each article.
While this waiting didn’t take any process time on your site, or add any load, it could take minutes to send out and receive every article. On many hosts, especially shared hosts, the host looks to stop processes that take too long – as little as 30 seconds on some hosts. This could mean that the background process of MyCurator was stopped early, and would have to pick up where it left off on the next run to finish gathering all of your articles. With this start/stop processing, the flow of articles to your site could be reduced as MyCurator was constantly trying to catch up after it was stopped.
The new request process removes this ‘wait time’ for articles to be processed. A new 2 step process starts with MyCurator sending all of the URL’s for all of the articles in your feed to the cloud process every 3,6,12 or 24 hours based on your setting of the schedule option on the Basic tab. This usually takes just a few seconds as it doesn’t need to wait for each article to be processed.
A new second background process kicks off every hour. It looks for any articles that have been processed by the cloud and brings them back to your site and posts them in your Training Posts. Again, it doesn’t have to wait for each article to be processed. They are ready to go and just need to be posted on your site. If you have a lot of articles, this posting may take some time depending on your host. But even if it doesn’t complete the first time, it starts up again an hour later to finish.
How is the Original Process working on My Site?
If you want to check up on how the original background process is running on your site, go to the Logs menu item. Choose Process from the drop down at the top left that says Activity and click Select Filter. Look for a log message that says “Cron Starting MyCurator”. After this entry (reading up the log) you will see entries for each topic and feed that was processed. Eventually you should see an entry that says “End Site Processing”. If this all happens together in the same time period, then the original process is running OK on your site. You could get by without the new request process, but as outlined above, it is still probably the more efficient way to go.
If you see that processing stops and then another “Cron Starting MyCurator” message appears, maybe with a “Restarting with Feed” message, then your background process is not completing on each run and could certainly benefit from the new background process.
Turning on the new Request Process Mode
To switch to the new Process, go to Options and choose the Admin tab. Check the option “Page Request Mode” to turn on the new Request processing for your site.
You will also notice a new option just below to use Inline Site Processing. This is a more reliable way to start the background process of MyCurator, whether in Request mode or the original process. If you aren’t getting any articles or very few, this option may be required. Some hosts block the original way that MyCurator started the background process.
Checking this option will bypass this block and let MyCurator run properly. Even if your background process was running correctly, this option is more reliable in starting up than the old way of kicking off the background process and will benefit most sites.