Common File Upload Strategies and Their Pros & Cons

The strategies we’ll be looking at are in no particular order, although it may seem so at first. There’s no silver bullet, no “best” approach to use (even though I do have my favorite…). 

Each approach has both benefits and drawbacks, and both sides of the equation need to be carefully considered when deciding which strategy to use. Factors like available time for implementation, operational costs and functional requirements must all be considered before picking a solution. You should choose wisely.

We’re going to look at these strategies from the backend’s perspective and will use AWS services as a reference to describe the approach — but I believe similar patterns can be extrapolated to any cloud provider. 

But first, let’s establish some nomenclature used throughout the article:

• Client: A website, mobile app or other entity that is used by the person who intends to upload a file.
• Resource: A record in the database, for example a user account, a blog post or other similar entity.
• Backend: Some kind of REST or GraphQL API that provides the functionality to your solution.
• Signed URL: A mechanism used by AWS where a trusted entity (usually your backend) generates a URL in such a way that it can be used by some other (usually unauthorized) entity to perform a one-time operation that they would normally not be able to do. In this article, that is going to be exclusively a file upload via HTTP POST from the client to AWS S3 bucket or to CloudFront.

The “Sign-And-PUT” Strategy

This strategy is useful in situations where the file itself is not mandatory for a resource; in other words, it’s acceptable to have a user account without a profile picture, some article without a banner image, etc. Therefore, clients can create their resources independently and provide the relevant files at any later time. 

It’s a very simple strategy that requires only two requests, but it comes with some potentially serious drawbacks.

Flow

1. A resource (user account, article, etc.) already exists in the backend.
2. Client sends a request to the backend signaling intent to upload and associate a file with this resource.
3. Client receives a signed S3 or CloudFront URL where to send the file.
4. Backend creates a record in the database that indicates the presence of the file in that S3 location and associates the file with the resource.
5. Client sends the file to the designated URL.

Benefits

• Easy to implement. Backend only needs to generate a signed URL and, during this process, it also creates necessary records in the database to serve that file to other clients looking at the resource.
• Only two requests to complete the whole process.

Drawbacks

• Since the backend does not get notified when the file has been actually uploaded, there might be situations where the client requests the signed URL from the backend but never finishes the actual file upload, leading to an inconsistent state (backend thinks the file is there but the file is never uploaded).
• Once backend generates the signed URL, it also starts informing other clients that the resource has a file associated with it — but the file might not yet be uploaded to S3, leading again to a temporarily inconsistent state where other clients will get HTTP 404 from CloudFront until the file upload has been completed.
• This strategy does not allow for a resource to require an associated file at the time of the resource’s creation because the file uploads always happen after a resource already exists.

Some of these drawbacks can be remediated by allowing S3 to trigger file upload events and send them to SQS (a managed job queue from AWS), from where backend can pick them up and mark those files as “actually uploaded” in the database, only serving the files once they are actually uploaded. This complicates this strategy noticeably.

The “3-way PUT” Strategy

This strategy achieves a single file upload using three HTTP requests: one for obtaining the signed URL, one to send the file to S3, and the final one to actually reference the file in some kind of resource-related operation — like creating a user account with a profile picture, publishing a blog post with a banner image, etc. 

It’a a very useful strategy in situations where the file itself is a required part of the resource and the API does not want to allow creating a resource without it (e.g., a user account with a required profile pic, a banking account with a photograph of a passport or other ID, etc.).

Flow

1. Client asks the backend for a signed URL to upload a file. The client might indicate future intent to associate this file with a specific type of resource to help the backend generate a proper signed URL — but at this time, the resource might not exist yet.
2. Client receives a signed S3 or CloudFront URL where to send the file.
3. Client sends the file to the designated URL.
4. Client sends a request to create/mutate a resource and references the uploaded file to associate it with that resource.
5. Backend creates the resource and a record in the database that indicates the presence of the file in that S3 location — potentially moving the file to a more appropriate location within the bucket if needed or performing other validations on the file before accepting the resource operation.

Benefits

• Consistent state at all steps. To actually use the uploaded file, the client needs to reference it during resource creation or mutation. At this moment, the backend can actually check with S3 to make sure the file really exists — or perform any validations on the file metadata — and can either accept or reject the resource’s creation/mutation.
• Allows implementations where a file is required in order for a resource to exist.

Drawbacks

• The whole process requires three requests to complete. Latency becomes a noticeable factor in this setup and the file upload experience might be worse on unstable networks, even for relatively small files.
• If the client uploads a file to S3 but never references that file in any resource operation, then the file will effectively become orphaned — with no knowledge about its existence by the backend. This may incur unnecessary storage costs if remediations are not implemented.

The “Public Dropbox” Strategy

This strategy allows write-only access to a part of an S3 bucket where clients dump the files and then reference those files during resource creation/mutation. The files are then moved by the backend to a secure location within the bucket for permanent storage. 

It’s a strategy that creates the best experience for the end user and provides opportunity for file verification and implementing required files, but it does require more effort to implement in a secure way.

Flow

1. Client dumps the file to a predetermined location in an S3 bucket, usually /public or /dropbox, and uses a random file name to avoid collision with other legitimate users (ideally UUID). This S3 folder allows write-only access for unauthenticated users. Reads are forbidden for everyone.
2. Client sends a request to create/mutate a resource and references the uploaded file to associate it with that resource.
3. Backend moves the file from the public dropbox to a more secure location inside the S3 bucket, potentially also performing validations on the file as necessary, then proceeds with the resource’s creation/mutation as usual.
4. Files that remain in the public dropbox for longer than X time are automatically deleted. This is implemented using the S3 Lifecycle policy.

Benefits

• Consistent state at all steps. To actually use the uploaded file, the client needs to reference it during resource creation or mutation. At this moment, the backend can actually check with S3 to make sure the file really exists — or perform any validations on the file metadata — and can either accept or reject the resource’s creation/mutation.
• Allows implementations where a file is required in order for a resource to exist.
• Unused files are automatically removed by the S3 Lifecycle policy, therefore there is no possibility to introduce orphaned files.
• Only two requests to complete the whole process.

Drawbacks

• Requires more effort to implement. The S3 bucket needs custom ACL to allow write-only access to a part of the bucket and a lifecycle policy to clean up unused files from the public dropbox; the files need to be moved to a different location when used; clients need to know where to put the files (either by hard-coding the CloudFront URL on the client or by exposing the URL via some kind of /meta API on the backend); and they need to generate random enough filenames during uploads to avoid collisions with other users. 
• Be aware that malicious users would not be able to exploit this by uploading files with non-random names (e.g., /public/image.png) because no legitimate client would attempt to use such a file.
• Since the public dropbox allows unauthenticated and unrestricted uploads, a potential malicious user might attempt to upload excessive amounts of data, incurring costs on traffic and storage. This problem is only partially mitigated by the automatic removal of files after X time. Additional restrictions should be considered — like updating the S3 access policy to limit the file size — if technically possible.

The “Don Corleone Backend” Strategy

This strategy puts the backend in charge of absolutely everything — from resource creation all the way to the actual processing of the byte data. This allows the backend to have absolute, undisputed control over the whole flow (hence “Don Corleone”); it completely hides the S3 from the clients and allows file modifications on-the-fly during file uploads at binary level. 

But the above comes at the cost of very high implementation complexity, higher compute costs and somewhat reduced client<->backend interaction ergonomics due to using less-known and less-used HTTP body encodings to achieve this strategy.

Flow

1. Client sends a request to create/mutate a resource and includes both the resource data and the file’s actual contents in the request body. The body must be encoded as multipart/form-data (MDN docs), the resource data must be the first part of the form and the file contents must be last in order for backend to get the resource data first, then proceed with handling the binary contents.
2. Backend decodes the first part of the multipart body which contains the resource itself (in any format, usually JSON-encoded), verifies the resource’s data, then proceeds with reading/streaming the second part of the multipart form to S3 or performs any desired verification/computation on the data on-the-fly.
3. Once the backend is done processing and saving the file to S3, it can return a regular API response back to the client. Since everything has been handled within a single request, there is no further interaction necessary.

Benefits

• Backend owns the whole process of binary file handling. It can modify the data on-the-fly, verify it, even duplicate it and send it to two different locations at the same time if it so pleases.
• Client has no knowledge of how or where the files are actually stored. The whole process is completely hidden from them. This even allows multi-cloud setups where some files reside in AWS and others live in Google Cloud or Azure.
• Only a single request is needed to complete the whole process.

Drawbacks

• Backend becomes a bottleneck for file uploads and needs to be scaled up accordingly. File uploads should be handled as streams to keep the memory usage in check. Compute costs might increase substantially if the project has high file traffic.
• More difficult to implement on backend because multipart form data is not something that works nicely with GraphQL or standard OpenAPI endpoints, and it requires good support from the HTTP web server library used by the backend.
• The strategy is highly dependent on the ordering of the request body. If the binary data is sent first, then the backend must buffer the whole file in memory before it reaches the second part where the actual resource data is located — which is usually required to determine how to treat the file. Also, if the file upload is not handled as a stream but is always buffered in memory before processing, then it opens up an attack vector where a malicious user could easily overload the servers and cause DoS (Denial of Service).
• Degraded client developer experience because each resource that needs a file to be uploaded this way now requires a custom, probably non-standard request-response handling (compared to other endpoints or queries within the project).

Conclusion

These four strategies are the most “usual” we’ve seen on projects. There are many variants of each — the many drawbacks mentioned here can be mitigated with the right approach, and the overall strategy could be made much more robust. However, this also requires additional development effort, so everything must be considered when making your choice.

I hope this article helps you choose the right strategy for file uploads on your next project! Don’t be afraid to experiment and alter these strategies with other approaches to make it even better for you.