Questions? We have answers.

Best practices: Large files

  • Updated

Summary

Large files (any file greater than 5 million rows) often present time-consuming challenges.

Learn some tips and tricks for how to get large files into MindBridge with ease.

 


Tips

  1. Zip or compress your data file. MindBridge has implemented an 8GB file import limit to reduce the likelihood of import failures due to network connectivity issues. You can zip your file, effectively reducing the file size by a factor of 10-15x, and import the compressed file into MindBridge.
    • If the general ledger is spread across multiple files that need to be combined (for example, one file per month), you will need to zip each file individually before uploading to the File manager, and if no formatting needs to be done, use the merge tool to combine them. You can also use the split tool for workbooks with multiple worksheets.
  2. Check your browser's firewall settings, as there may be restrictions imposed on the browser by your IT team. For example, you will not be able to import a 5GB file into MindBridge if your browser limit is set to 4GB.
  3. Before obtaining an export, identify which columns you need for your analysis, including the columns required in MindBridge. Large files can take a lot of time to import, and even longer if any data formatting is needed. Keeping only the columns you absolutely need will save time and disk space.
  4. Identify a good transaction ID. Although it is always important to identify a good transaction ID, the larger the file, the more important this becomes to successfully running an analysis. Read the following articles for more information:
  5. The first Data Validation step provides high-level information to help you perform quick sanity checks and to spot potential data ingestion and integrity issues before completing the import. Since the full import process can take a few hours for large files, spotting potential issues before proceeding with the import can save you a significant amount of time.
  6. Plan ahead and be patient. If there are no data issues, you might spend a day completing the import process, and then the analysis might need to run overnight or longer.
    • If your GL is exceptionally large (approximately 50 million rows or more), feel free to let us know before running your analysis so we can keep an eye on it and/or let you know if there are other large jobs already running that could cause delays.


Anything else on your mind?

Head over to the community with your thoughts and inquiries! You can also chat with us, or, submit a request for further assistance.


Related articles

Was this article helpful?