Improved
BigQuery import — windowed exports and a 100 GB result limit
7 days ago
BigQuery imports handle large tables more reliably. A large query result is now exported and imported a piece at a time, so a failure retries only the piece that failed, and an import's result is capped at 100 GB.
What's new
- Large results import in windows. When a result is larger than about 10 GiB and the import has a Timestamp Field, it is split into time windows of about 10 GiB, and each run exports and imports one window. A failed export retries only its own window and reuses the query result instead of running your query again.
- Imported files are cleaned up. Each window's export files are deleted as soon as they're imported, and a failed export deletes what it wrote.
- A stop doesn't lose rows. If a deploy or rebalance interrupts a file mid-import, that file is imported again.
What you need to know
- An import whose result is larger than 100 GB now stops without exporting anything, and doesn't retry. Its error says how large the result was.
- To bring in a larger history, narrow what the import reads, for example with a view over a date range, and import older data with separate imports. Lytics Support can raise the limit for your account.
See Google BigQuery for setup.