How can you build a recipe that detects a CSV file has been uploaded to Dropbox (or Box etc.) then process the entire file and rename it to indicate it has been processed?
Using the Dropbox New/Updated File trigger and Dropbox Read CSV File lines action it is possible to detect a new file and rename it after processing, but only 100 records can be processed.
While using the Dropbox New Line Batch in CSV File trigger you can process any number of records, but only one file can be processed once and it is not possible to identify the last batch when the file should ne renamed.
Has the limitation of 100 records been recently implemented?
I understand that Workato may require users to implement batch processing for large files, but it should still be possible to automate the process.
Best Answer
V
Vivek
said
over 6 years ago
Hi,
I have put together a simple recipe that can achieve what you are looking for.
Here is how the logic goes:
The recipe triggers when a new line batch(100) of csv has been uploaded in dropbox.
For each row in the batch, the data will be mapped to Google sheets (or to any application of your preference)
After the batch has been processed, if the batch size is lesser than 100, rename and move the file to a processed folder.
We have limited to 100 records to eliminate possible timeout errors for larger data files.
Hope this answers your question.
I
ITAdmin Deaf Society
said
over 6 years ago
Hi Vivek,
Although a good suggestion this does not allow the process to be automated. Because the recipe can only be run once you would have to copy the recipe and reselect the CSV file each time. Is there a solution that will support automated processing of an uploaded CSV file?
Also there is a (hiddent) problem with the proposal to identify the last batch. It will not work when the number or records is divisible by the batch size. Hence it would be preferable if that information was present in the event.
I
ITAdmin Deaf Society
said
over 6 years ago
For other users reading this be aware that I've also come across some significant issues with the processing of CSV data by the Dropbox New Line Batch in CSV File trigger, e.g. it doesn't handle UTF-8 encoding, quoting of fields, or multiline fields. I'll be logging these issues as bugs, which will hopefully be addressed.
In practice this means that your CSV data cannot contain commas, newlines or non-ascii characters.
Note, I believe the Dropbox New Line in CSV File trigger and Read CSV File lines action handle CSV format correctly.
ITAdmin Deaf Society
Automatically process uploaded CSV file
How can you build a recipe that detects a CSV file has been uploaded to Dropbox (or Box etc.) then process the entire file and rename it to indicate it has been processed?
Using the Dropbox New/Updated File trigger and Dropbox Read CSV File lines action it is possible to detect a new file and rename it after processing, but only 100 records can be processed.
While using the Dropbox New Line Batch in CSV File trigger you can process any number of records, but only one file can be processed once and it is not possible to identify the last batch when the file should ne renamed.
Has the limitation of 100 records been recently implemented?
I understand that Workato may require users to implement batch processing for large files, but it should still be possible to automate the process.
Hi,
I have put together a simple recipe that can achieve what you are looking for.
Here is how the logic goes:
- The recipe triggers when a new line batch(100) of csv has been uploaded in dropbox.
- For each row in the batch, the data will be mapped to Google sheets (or to any application of your preference)
- After the batch has been processed, if the batch size is lesser than 100, rename and move the file to a processed folder.
https://www.workato.com/recipes/536730#recipePlease modify the field mappings accordingly.
We have limited to 100 records to eliminate possible timeout errors for larger data files.
Hope this answers your question.
Vivek
Hi,
I have put together a simple recipe that can achieve what you are looking for.
Here is how the logic goes:
- The recipe triggers when a new line batch(100) of csv has been uploaded in dropbox.
- For each row in the batch, the data will be mapped to Google sheets (or to any application of your preference)
- After the batch has been processed, if the batch size is lesser than 100, rename and move the file to a processed folder.
https://www.workato.com/recipes/536730#recipePlease modify the field mappings accordingly.
We have limited to 100 records to eliminate possible timeout errors for larger data files.
Hope this answers your question.
ITAdmin Deaf Society
Hi Vivek,
Although a good suggestion this does not allow the process to be automated. Because the recipe can only be run once you would have to copy the recipe and reselect the CSV file each time. Is there a solution that will support automated processing of an uploaded CSV file?
Also there is a (hiddent) problem with the proposal to identify the last batch. It will not work when the number or records is divisible by the batch size. Hence it would be preferable if that information was present in the event.
ITAdmin Deaf Society
For other users reading this be aware that I've also come across some significant issues with the processing of CSV data by the Dropbox New Line Batch in CSV File trigger, e.g. it doesn't handle UTF-8 encoding, quoting of fields, or multiline fields. I'll be logging these issues as bugs, which will hopefully be addressed.
In practice this means that your CSV data cannot contain commas, newlines or non-ascii characters.
Note, I believe the Dropbox New Line in CSV File trigger and Read CSV File lines action handle CSV format correctly.