Uploading files from Android to AWS S3 using Retrofit2

This is one of those posts that serves as a helping hand to others in need of clear and updated answers (at the time of writing) about how to upload files from Android to a server (in my case to a bucket in AWS S3) using Retrofit2. Note, this post assumes you already have a pre-signed URL and you just need to do the upload part.

Uploading files using retrofit is straight forward… if you’re not in Android, otherwise you need to provide you’re own solution to a simple problem, get a java.io.File or a java.net.URI when you’re forced to use content providers? Either a File or a URI is needed to build a MultipartBody.Part, but in my case I had an Intent to capture some photo from the camera and all I could get back was a content Uri from the android.net package, the thing is that now Android doesn’t allow easy direct access to the file system, since it comes with more granular permissions to access files you need to use the ContentResolver to access and operate on files. The problem arises when the API doesn’t provide any function to obtain a URI from a Uri, and there’s no way to get a File (unless you copy the file to a folder your app can access), however you can get ContentResolver.openInputStream() to get a FileInputStream.

There have been several issues in Okhttp3’s GitHub repo giving the argument that Okhttp3 can’t work with InputStream because such streams can only be consumed once, and the HTTP client need to be able to consume it multiple times in case of redirects or retrials. However they will also not provide integration with Android because then the library (which is not Android specific) will need to include some references to Android classes (such as the ContentResolver or Uri). The solution then, is to create our own RequestBody:

class InputStreamRequestBody(
    private val contentResolver: ContentResolver,
    private val uri: Uri
): RequestBody() {

    override fun contentType(): MediaType? =
        contentResolver.getType(uri)?.toMediaTypeOrNull()

    override fun writeTo(sink: BufferedSink) {
        contentResolver.openInputStream(uri)?.source()?.use(sink::writeAll)
    }

    override fun contentLength(): Long =
        contentResolver.query(uri, null, null, null, null)?.use { cursor ->
            val sizeColumnIndex: Int = cursor.getColumnIndex(OpenableColumns.SIZE)
            cursor.moveToFirst()
            cursor.getLong(sizeColumnIndex)
        } ?: super.contentLength()

}

This way, writeTo can be called several times and every time it will open a fresh FileInputStream to the file and use a Okio Source to write the stream to the given BufferedSink.

Note that to construct a MediaType required to set the Content-Type we now need to use the extension method toMediaTypeOrNull which needs to be explicitly imported, for some reason Android Studio doesn’t do it.

As to the getContentLength method, you might not need to override it, by default okhttp will set it to -1 meaning “unknown size”, but I was experiencing a broken pipe error without it, so we query the ContentResolver to get the file size, more info on that here.

With the above, assuming your Retrofit2 service defines an endpoint like this:

@Multipart
@PUT
suspend fun uploadFile(
    @Header("Content-Type") contentType: String,
    @Url uploadUrl: String,
    @Part file: MultipartBody.Part
): Response<Unit>

You will be able to upload the file to the server like this:

suspend fun uploadFile(url: String, fileUri: Uri) {
    context.contentResolver.getType(fileUri)?.let { mimeType ->
        val file = File(fileUri.path!!)
        val requestBody = InputStreamRequestBody(context.contentResolver, fileUri)
        val filePart = MultipartBody.Part.createFormData("file", file.name, requestBody)

        SomeServerApi.retrofitService.uploadFile(mimeType, url, filePart)
    }
}

Here we create a File from an Android’s Uri and thus it is invalid to treat it as a file, you’ll get an exception regarding invalid scheme if you try to, since it is expected a File’s URI have a file:// scheme but the content Uri has a content:// scheme instead, however, here it is used just to extract the file name from it (which only depends on the parsing the end of the URI), another option to get the File name is to use the ContentResolver.query method, or just give it a random name in the spot if you don’t care.

So, this will work if the server accepts a Multipart form data to upload a file, for AWS S3 this wasn’t working for me, I keep getting a Broken Pipe error:

javax.net.ssl.SSLException: Write error: ssl=0x7cf38d7f88: I/O error during system call, Broken pipe

Instead the solution was to directly send the file in the Request’s body, for that we need to change the Retrofit2 endpoint to not use Multipart and just take a RequestBody:

@PUT
suspend fun uploadFile(
    @Header("Content-Type") contentType: String,
    @Url uploadUrl: String,
    @Body file: RequestBody
): Response<Unit>

Now, creating a Part isn’t necessary, we just set the request body directly:

suspend fun uploadFile(url: String, fileUri: Uri) {
    context.contentResolver.getType(fileUri)?.let { mimeType ->
        val requestBody = InputStreamRequestBody(context.contentResolver, fileUri)
        SomeServerApi.retrofitService.uploadFile(mimeType, url, requestBody)
    }
}

This is all that is needed to make it work. Before closing the post I would say that while looking for answers to this problem some people just solved it by using HttpURLConnection instead, there’s an example how to use it in the AWS S3 documentation, but using retrofit is nicer.

Finally, let’s remember we can leverage extension functions in Kotlin, so we can create the RequestBody more succinctly by extending the ContentResolver:

fun ContentResolver.readAsRequestBody(uri: Uri): RequestBody =
    object: RequestBody() {
        override fun contentType(): MediaType? =
            this@readAsRequestBody.getType(uri)?.toMediaTypeOrNull()

        override fun writeTo(sink: BufferedSink) {
            this@readAsRequestBody.openInputStream(uri)?.source()?.use(sink::writeAll)
        }

        override fun contentLength(): Long =
            this@readAsRequestBody.query(uri, null, null, null, null)?.use { cursor ->
                val sizeColumnIndex: Int = cursor.getColumnIndex(OpenableColumns.SIZE)
                cursor.moveToFirst()
                cursor.getLong(sizeColumnIndex)
            } ?: super.contentLength()
    }

Now we can use it very naturally:

val requestBody = context.contentResolver.readAsRequestBody(fileUri)
SomeServerApi.retrofitService.uploadFile(mimeType, url, requestBody)

That’s it!