Working with Large Catalogs

Everything below applies to stores with tens of thousands of products or orders. On a store with 500 products, ignore all of it and use the defaults.

Split large files #

A 100,000 row file will import. The problem is what happens when it fails at row 91,000: you are left working out which rows landed and which did not, and re-running the whole thing takes as long again.

Files of 10,000 to 20,000 rows are the sweet spot. Each finishes in a manageable time, each produces its own history record with its own counts, and a failure costs you one chunk rather than the whole job. Split by category, by date range, or just by row number.

Tune the batch size #

Batch size is the main lever you have. Under Settings, the import batch defaults to 10.

• Imports stopping partway or hitting memory errors: drop to 5, or 3 for product imports with many images.
• Comfortable server with headroom: raise to 50 or 100 and the same file finishes considerably faster.
• Exports timing out: lower the export batch, and reduce the column count before anything else.

Change one thing at a time and re-run a sample. Doubling the batch and turning off meta at the same time tells you nothing about which one helped.

Import in dependency order #

Categories and tags, then products, then customers, then coupons, then orders, then reviews. Each stage references the ones before it. Import orders before products and the line items have nothing to attach to.

A bundle import applies this ordering for you, which is the strongest argument for using one on a migration.

Keep files narrow #

A product export with 200 columns is far heavier than one with 20, on both sides of the round trip. For a price update, export ID, SKU, Name, Regular Price and Sale Price and nothing else. The file is small, the import is fast, and there is no chance of overwriting a description because somebody’s spreadsheet mangled it.

Turn off custom meta and hidden meta unless you specifically need those fields.

Turn logging off in production #

Import logging writes a line per operation. On a 50,000 row import that is a great deal of extra writing, and it inflates the history table. Use it while you are getting a job working, then switch it off.

Turn history auto-delete on as well, so old records do not accumulate indefinitely.

Images are usually the bottleneck #

Product images download in background jobs, one per image, from wherever the URL points. A slow or rate-limiting image server will drag out an import that would otherwise take minutes.

• Host the images somewhere fast for the duration of the migration.
• Run a first import without the image columns to get the catalogue in place, then a second pass with only ID, SKU and images.
• Leave image optimization off for the first run. It adds CPU work per image and can be applied later by a dedicated plugin.

Run big jobs when the store is quiet #

A large import competes with your customers for the same database. Schedule heavy work for the small hours, and avoid running a full catalogue import during a sale.

Watch the failed count #

An import can finish with a completed status and hundreds of failed rows. Open the History record and read the four counts rather than trusting the status. If the failed count is not zero, turn on logging, import a sample of the failing rows, and read what the log says.

Use the scheduler for recurring work #

A scheduled job that runs at 4 AM does the work while nobody is watching and mails you the result. This is better than a person remembering to run a large import, and better than that person running it at 11 AM.

Ask your host for headroom #

Before a big migration, ask for memory_limit raised to 256M or 512M, and max_execution_time raised to 300 seconds. Most hosts will do this temporarily. It turns an afternoon of restarts into a single clean run.

A working plan for a 50,000 product migration #

1. Back up both stores.
2. Ask the host to raise memory and execution time.
3. Export categories and tags. Import them. Verify the tree.
4. Export products in chunks of 10,000, without images. Import each chunk, checking the counts between runs.
5. Export products again with only ID, SKU and image columns. Import those, and expect it to take a while.
6. Export and import customers.
7. Export orders one month at a time. Import in date order.
8. Spot-check: a variable product with variations, a multi-line order, a nested category, a customer with orders attached.