Whether it’s to back up your data, transfer product information to your drop shippers, or analyse the data in a specialist application, you need to be able to export your data. However, the more data you have, the longer it can take to compile and export. When working with browser-based systems such as Neto, this can lead to browser timeouts that prevent the data from exporting.
Why do exports timeout?
The basic answer is that the export takes too long and the browser gives up waiting for it to finish.
When you hit the "Download Data File" button on the export template, your browser opens a connection and sends a request to the server to run the export.
The browser then waits for the server to finish, at which time the file created by the server is downloaded by the browser.
Problems occur when the export takes more than a few mins to process. Browsers are set to wait for a specific period; if they do not get a response from the server within that period, they time out.
Factors that increase the time to export events
There’s no “one thing” that affects the time an export can take. Usually, it’s a combination of factors, the most common of which are
- The Number of Columns
- Number of Results
- The Datatype and Size
- Server System Load
Generally, you would be looking for no more than a couple thousand results (rows) to be included in any export. Having a large number of columns adds more data to each result, and can reduce the number of results that can be produced before causing a timeout. This is why you may find you can export 10000 results for Order Payments but products, which have many more fields (and thus more columns), will give you trouble during export with only 5000 products in your systems.
The Maropost infrastructure team has systems in place to balance server load as best as possible. However, there are a few things you can do to affect the data size.
“Simple” and “Complex” Exports
In Neto, there are two types of export. We call these “simple” and “complex”. The names are a little misleading, the complex export isn’t more difficult despite what the name would imply.
The simple export is best used for reporting, as fields can’t be customised as much and filters are limited. As the simple export does not allow for customisation, this means you are exporting all your data. This is why you will most commonly encounter timeout issue during a simple export.
If you do find you are getting timeouts when using a simple export, your best option is to try using a complex export.
A complex export will allow you to select the delivery method of the exported data, as well as offer more customisation of what data is included in the export.
Complex exports are possible for
Exporting to Local URL
The Complex Export Wizard allows you to select where to send the export file that is created. If you get timeouts using a simple export or errors sending to Email, make sure the File Delivery section is set to Local URL (folder on your website).
Clicking the Export Now> button creates a process to generate the export file.
Copy the URL that you’re given as this will be the location of the file once it is created.
You can click the process number to view the process in the process manager. If there is a Date End value then you can go to the provided URL to download the exported file.
If the process is still running (no Date End shown) you can click the refresh button to update the listing.
Breaking Up Larger Exports
With the complex export, you have more control over the data that is included in the export. This can be useful in reducing the size of the export by limiting it to only the specific data you are looking for.
You can segment the data included in the file by selecting filters. Not all export types have the same filter options, but see the following suggestions for each data type:
- Products
- Brand
- Product Supplier
- Active
- Sales Orders
- Customers
- User Group
- Last Purchase date
- Payments
Using the filters to break up data into several exports rather than one can also help to avoid timeouts. For example, if you tried to export your payment data and received a timeout, you could use the filters to export each payment type separately, making several smaller files.
More information about exports (and imports) can be found here. Let us know in the comments if this helps, or if there are any other parts of the Commerce Cloud you’d like to see a future tip on!
