Aws takes a long time to download single file

Do we know how long it takes to upload a 20kb file to S3 the first time? If that doesn't take much time, would it make sense to create a new file each time and delete the old one i. EV3 about the upload time, I would definitely recommend that you do a simple benchmark -- it would be strongly related to where your upload originates from.

I would definitely investigate that further - based on my experience, it's likely that 20kB uploads to S3 will give you good enough performance! Thanks for your advice about DynamoDB. My final design is to create a new S3 document when user saves a step, save this S3 location to dynamoDB and keep a mapping of document identifier and document S3 location , remove the old S3 document to free up space.

EV3: that sounds like a very good design! JayKumar - no, there's no such flag. In CS terms, there can't be such a flag - the existence of such a flag would imply that the system is strongly consistent, violating the hypothesis of eventual consistency.

Show 2 more comments. We can freeze our website when document is being saved to S3 You will not find the exact time anywhere. No there is no any function using which you can calculate this. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Introducing Content Health, a new way to keep the knowledge base up-to-date.

Podcast what if you could invest in your favorite developer? Featured on Meta. And I will explain to you each and every step which will h So, in this blog post, I will share with you some of the best courses that I found. These courses wi About Me. Close Menu. Posts Open Menu. Share this post. Post written by Abhishek Sharma. But wait Open the S3 console Click on the bucket from which you want to download the file Select all the files which you want to download and click on Open.

Look at the picture below. I guess there is a limit in Chrome and it will only download 6 files at once. Download single file To download a single file follow the below steps - Open the S3 console Click on the bucket from which you want to download the file Select the file that you want to download and click on the download button Using the AWS CLI Note - If you are wondering, let me tell you that you do not need to specify any region in the below commands.

Conclusion I believe this post helped you solve your problem. Thank you and keep learning. Learn more. Asked 4 years, 10 months ago. Active 6 months ago. Viewed 15k times. Improve this question. Community Bot 1 1 1 silver badge. King Dedede King Dedede 1 1 gold badge 8 8 silver badges 22 22 bronze badges. That sounds like expected behavior. Syncing mb to a new directory takes almost no time, but syncing to a heavily used directory can take hours - hopefully there is an alternative!

One alternative that works for me rclone rclone. PhilipKirkbride: I don't see why. PhilipKirkbride: what I mean is that, to me, it is unexpected given that: 1. In other words, I don't contest the fact that the current implementation of aws s3 sync is slow in this case, but I am surprised by it.

Bruno Reis Bruno Reis Thanks for the alternative solutions. Do we know how long it takes to upload a 20kb file to S3 the first time? If that doesn't take much time, would it make sense to create a new file each time and delete the old one i. EV3 about the upload time, I would definitely recommend that you do a simple benchmark -- it would be strongly related to where your upload originates from.

I would definitely investigate that further - based on my experience, it's likely that 20kB uploads to S3 will give you good enough performance! Thanks for your advice about DynamoDB. My final design is to create a new S3 document when user saves a step, save this S3 location to dynamoDB and keep a mapping of document identifier and document S3 location , remove the old S3 document to free up space.

EV3: that sounds like a very good design! JayKumar - no, there's no such flag. In CS terms, there can't be such a flag - the existence of such a flag would imply that the system is strongly consistent, violating the hypothesis of eventual consistency. Show 2 more comments. We can freeze our website when document is being saved to S3 You will not find the exact time anywhere. No there is no any function using which you can calculate this.

Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog.