Improved Import With Power 4.9.2

Power 4.9.2 for our WordPress plugins was released tonight with improved import processing.  Several updates are included as we progress towards improved large location import processing.    All of the upgrades are coming to the MySLP service for our non-WordPress users on services like Wix, Weebly, and Squarespace but we have a bit more testing and integration to ensure 99.999% up time on our SaaS service.

Fixing large location imports

If you were importing thousands of locations and found the import process would get partway through then stop, the Power 4.9.2 patch should help.    If your file could not be processed in one pass by the background import processor, Store Locator Plus would detect the unfinished import and pick up where it left off.  Problem was it would not re-read the header row to match up data to locator fields and no additional locations would come into the location list.    This issue, while leaving the new 4.9 import architecture no better but no worse than the 4.8 and earlier releases, has now been resolved.     The second , and third or fourth passes if necessary, properly pick up where things left off.

The second fix is to prevent that second or third location import pass from starting prematurely when the first pass slows down.    Power 4.9.1 would sometimes think the first import stopped processing and start where the first pass paused.   This slowed EVERYTHING down and if Locations > Import > File Settings > Duplicates Handling was set to Add versus Update it may have duplicated records.

The import issue could manifest for smaller lists of locations if your server is underpowered or your hosting service is severely restricting PHP time slices (processing time) on your account.   A typical moderately-loaded server should be able to easily process 2500 locations within a 60-second PHP time slice with time to spare.     For servers that cannot process imports that quickly this patch will help keep the process going.

Improved import for all list sizes

While traveling the path of fixing the large list imports we also found several ways to improve import performance.  Where we would run through around 600 locations being imported in a 15 second processing time limit ( Settings > General > Server > PHP Time Limit ) with Power 4.9.1 the same file would load all 1,000 locations before the time ran out.     Our 25,000 location test would load nearly 10,000 locations with a typical 60-second PHP time limit.

As an aside, the default time limit setting for SLP is 600 seconds.   We were turning this DOWN to try to force our servers to behave like a severely taxed web server or a severely underpowered server.    Many shared hosting plans will NOT honor this setting and changing it will not have any impact on most server.     MySLP users will not see this setting in Advanced Options as our servers are already tuned to allow maximum PHP processing for large location lists.

Keeping data intact with slashes in CSV files

The most commonly referenced specification for CSV file processing, RFC 4180, dictates what most computer scientist agree is a valid CSV file format.   One part of that specification is how to deal with data that has a quote in it.   Since CSV data that has any special character within is to be wrapped in quotes (“) there has to be a special way to deal with things like a store name My “Special” Store.    RFC 4180 says that the CSV file should use double-quotes to manage this.

Rather than code all of this ourselves we use the built-in PHP CSV file processing functions.    It is far faster than running custom PHP code.

Until our 4.9 releases we had been trying to keep all of our code compatible with PHP 5.2.4 due to the WordPress minimum requirement being 5.2.4 despite their recommendation to use PHP 7.     We finally decided we could not longer support the 11-year old version if we were going to write efficient applications.   Our plugins no longer support PHP 5.2.4 and now require PHP 5.3.

The benefit of this requirement is we can now support the proper RFC 4180 specification instead of the PHP “special Sybase compatible” version that sees the \” character combination as an escape sequence.   This messes up the import with any field that ends with a \ and starts “shifting fields” into the wrong columns.

As of Power 4.9.2 the \ character is treated like any other character which makes the imports a lot happier with files that Excel is mangling.

Power 4.9.2 still requires proper line endings, so for our Mac users running older versions of Excel DO NOT export in Mac compatible mode.   For some reason Microsoft things Mac computers still use the 2003 standard return (\r) line ending instead of the modern standard CRLF (\r\n) line endings that are required for a proper CSV file formatting.    Why Excel allows a CSV export of ANY format that does NOT have a CRLF is a mystery as this is not part of ANY CSV standard including the “special Sybase compatible” version that PHP allows.

Improved import instructions and reporting

One of the obvious changes is some notes are on the Locations > Import tab to help guide you to some of the updates in the 4.9 import process.  This includes explaining the basic 3-step process and the fact that you can find your location CSV file in the Media > Library of WordPress.

The Power 4.9.2 release has simplified the message logging on the bottom of the import page timestamping only the start and end of processing and reporting problem records.   Added, updated, and not updated records are no longer reported on an individual basis in order to improve performance and consume far less memory on large file imports.

PHP Versions
PHP Versions

In addition to the improved import reporting,  the summary with a count of how many records in the CSV file has added, updated, not updated, were malformed and other summary data is now visible on a per-file basis in the Media Library file details for location CSV files.    This is in addition to the Power 4.9 standard file status , file size, current processing offset, and record count (when completed).

Other tweaks and updates

We snuck in a few smaller things into 4.9.2.

The Identifier column has always had special meaning in CSV files; enabled in General > Data > Add On Data Extensions > Enable Contact Fields, this special column override the sl_id column to match up records to be updated.  While useful for aligning Store Locator Plus with third party data systems it does add some extra overhead to the file processing.    Power will no longer add the overhead to check for that column if the Enable Contact Fields setting is not enabled.  It also means updates will no longer work if you DO include that field in your data source but have not ensured the setting is enabled.   Most sites will not need the Identifier column and update feature.

We fixed the Remote File Retrieval settings.  Somewhere during our updates over the past year these settings stopped saving when you changed them.   We also happened to create a better system for marking “quick save” fields like these the auto-save when changed. You may start seeing quick-save options in more places as they reduce button clicks and are faster than saving entire pages of settings to change one thing.

One last improved import addition — we now report on CSV files that look like they have too many columns.  A CSV file with more than 256 columns is a good indicator that the file is corrupt and the line endings (see Excel and MacOS above) are not set properly.   Rather than create a huge string that includes the entire file and may crash the server, Power 4.9.2 flags this potential error and truncates the logging.   The number of columns (aka fields) is shown in the Media Library details as well and provides more insight into what is going on “behind the scenes” should an import fail.

 

Store Locator Plus 4.9.2 Delete Locations

Store Locator Plus 4.9.2 patches several things including the delete locations bug.  Other fixes include several patches to address WordPress 4.8.3 introduced issues with the database query engine; the changes require intelligence in the plugin to support pre-4.8.3 WordPress installations.    We’ve also addressed an issue with memory consumption if logging is enabled and large location lists are being imported.

These patches are available today for WordPress and will be in our year-end update to MySLP.

 

We are continuing to improve the performance and resilience for sites with large lists of locations (10,000+ locations) and have several new upgrades on the way.

We are also working on improving our Style Gallery and adding features to allow users to create more interesting and modern interfaces with better responsive design support.    More updates are coming over the next few months.

If you have questions or comments please contact us.

Why Good JavaScript Code Is Critical To Your Locator

Whether you are using our MySLP SaaS service or our WordPress plugins for Store Locator Plus, having a web page that is loading “clean” JavaScript code is CRITICAL to the functionality of your locator.   Most web pages today, and certainly those including an interactive map, are far removed from static HTML and images.   Today’s web pages are full-on applications.   They are programs that are running locally on your device to draw a page that looks similar to the old-time static HTML pages but with a lot more cool interactions; like maps.

While these new interfaces are cool and certainly make the web easier to use they come at a cost.   It requires that all of the little mini-applets that you are loading on a page to show to your users play nicely together.   Every single interactive component like that Google Ads sidebar, the Recaptcha validation and maps are all separate programs that are playing in the same “pool”.    The problem with this is that it is very easy for any one of those components to “go haywire” and spoil the fun for everyone else.

Part of the issue is that for most web page builders you have ZERO control over WHEN you get your turn to jump in the pool.   Do you get in first?   Or do you get in after someone left their “code turd” in the pool?   How well your app runs can depend a lot on WHEN it gets to run.  To make matters more complicated an interactive page may not “sprout new problems” until AFTER the user starts interacting with the page.   Maybe it happens when a user clicks on an ad that is being tracked which then breaks the code and all subsequent JavaScript is running in a “polluted pool”.

Often the Store Locator Plus map is the most visual an interactive element on the page.  It happens to be what most people notice first when JavaScript is broken.   You won’t notice if your ad-click is not being tracked but you sure as heck notice when you try to look up locations and the map doesn’t change.    Often the cry goes out “you broke the locator” when many times it is something else that broke well before the locator had a chance to do its work.

Check For JavaScript Code Errors

Use your web browser developer tools.   All modern browsers have this functionality and it can give you some clues as to what is really going on.    If you see errors in the console you will want to clean this up and see if it resolves the issues with the locator.

Safari

Go to Safari | Preferences | Advanced and click “Show Develop menu in menu bar”.

Safari Turn On Dev Tools
Safari Turn On Dev Tools

 

To check your JavaScript Console go to Develop | Show JavaScript Console

Look for any red exclamation marks and look at the error logs to suss out JavaScript code issues.

JavaScript Code Errors In Web Console
JavaScript Code Errors In Web Console

Firefox

Older versions of Firefox have the web developer tools enabled by default.  If you do not see the Web Developer menu option check under the Settings menu a this may be disabled by default in future releases.

To view possible JavaScript errors:

Go to Tools | Web Developer | Web Console

 

Learn More In This Video

Check out this video to learn more about the technical details of why JavaScript loading order and errors in other JavaScript code is important.

 

If you are not seeing JavaScript code errors on your page and your Store Locator Plus map is still not working please contact us!