• Documentation
  • Workato Blog
  • Product Blog
  • Return to Workato
  • Login
  • Automation Institute
  • Documentation
  • Community Forums
New Topic
Discussions Recipe Building Recipe Building Questions
I

ITAdmin Deaf Society

started a topic almost 6 years ago
Answered

Automatically process uploaded CSV file

How can you build a recipe that detects a CSV file has been uploaded to Dropbox (or Box etc.) then process the entire file and rename it to indicate it has been processed?


Using the Dropbox New/Updated File trigger and Dropbox Read CSV File lines action it is possible to detect a new file and rename it after processing, but only 100 records can be processed.


While using the Dropbox New Line Batch in CSV File trigger you can process any number of records, but only one file can be processed once and it is not possible to identify the last batch when the file should ne renamed.


Has the limitation of 100 records been recently implemented?


I understand that Workato may require users to implement batch processing for large files, but it should still be possible to automate the process.


Best Answer
V
Vivek said almost 6 years ago

Hi, 


I have put together a simple recipe that can achieve what you are looking for.


Here is how the logic goes:

  • The recipe triggers when a new line batch(100) of csv has been uploaded in dropbox.
  • For each row in the batch, the data will be mapped to Google sheets (or to any application of your preference)
  • After the batch has been processed, if the batch size is lesser than 100, rename and move the file to a processed folder.
https://www.workato.com/recipes/536730#recipe

Please modify the field mappings accordingly. 


We have limited to 100 records to eliminate possible timeout errors for larger data files. 


Hope this answers your question.


Answer

Vivek

said almost 6 years ago

Hi, 


I have put together a simple recipe that can achieve what you are looking for.


Here is how the logic goes:

  • The recipe triggers when a new line batch(100) of csv has been uploaded in dropbox.
  • For each row in the batch, the data will be mapped to Google sheets (or to any application of your preference)
  • After the batch has been processed, if the batch size is lesser than 100, rename and move the file to a processed folder.
https://www.workato.com/recipes/536730#recipe

Please modify the field mappings accordingly. 


We have limited to 100 records to eliminate possible timeout errors for larger data files. 


Hope this answers your question.


I

ITAdmin Deaf Society

said almost 6 years ago

Hi Vivek,


Although a good suggestion this does not allow the process to be automated. Because the recipe can only be run once you would have to copy the recipe and reselect the CSV file each time. Is there a solution that will support automated processing of an uploaded CSV file?


Also there is a (hiddent) problem with the proposal to identify the last batch. It will not work when the number or records is divisible by the batch size. Hence it would be preferable if that information was present in the event.


I

ITAdmin Deaf Society

said almost 6 years ago

For other users reading this be aware that I've also come across some significant issues with the processing of CSV data by the Dropbox New Line Batch in CSV File trigger, e.g. it doesn't handle UTF-8 encoding, quoting of fields, or multiline fields. I'll be logging these issues as bugs, which will hopefully be addressed.


In practice this means that your CSV data cannot contain commas, newlines or non-ascii characters.

 

Note, I believe the Dropbox  New Line in CSV File trigger and Read CSV File lines action handle CSV format correctly.

Login to post a comment

Still can't find your solution?

Send us a ticket, we will try our best to assist you with your problem

Documentation
Developer's Library
Tutorials
eBooks
Product Hour
Product Blog
Workato Blog
Product updates
Customer Stories
© Workato 2022   Privacy   Terms   +1 (844) 469-6752
  • Documentation
  • Workato Blog
  • Product Blog
  • Solutions
  • Forums
  • Tickets
  • Log in
  • Return to Workato
Topic views count