Building the Video Maker#

This was the hardest part to figure out. Turns out assembling a video is really hard with all the codecs, video formats and standards out there. My first attempt was with FFmpegKit which is like precompiled ffmpeg binaries already wrapped in JNI. Sadly it was discontinued after I finished making a proof of concept with it. I tried some forks at that time, but they were all crashing with codec errors. It was torturous to get it to assemble a single video and then the video won’t play because of an obscure cryptic codec error.

Since I did not know much about JNI and integrating native binaries into apps back then I decided to give up on it, But one day I randomly asked a Chatbot about ways to make videos from a list of bitmaps, and it came up with an alternate suggestion. Using Mediacodec I can assemble the video using the device’s own codecs, eliminating the need for ffmpeg and JNI altogether.

The API was much more complicated than I expected and understanding it was becoming harder and harder. Thanks to a friend on discord, found bitmap2video which is like a proof of concept app and library built with the Mediacodec API. Initially I just copied the video generation part from this library and made it work.

Later, I customized it to be more efficient and fine-tuned with the requirements of the app. Added fixed aspect ratios, removed a bunch of dead code and made a module out of the localized changes. At the end I had a very convenient and stable API with a bunch of parameters to customize the videos. Learnt what these complex terms actually mean and got a taste of how complicated dealing with videos is.

1
// all the different options available for making videos
2
data class MontageConfig(
3
    val mimeType: String = MediaFormat.MIMETYPE_VIDEO_AVC,
4
    val bitrate: Int = 10_000_000,
5
    val iFrameInterval: Int = 1,
6
    val framesPerImage: Int = 1,
7
    val framesPerSecond: Float = 1f,
8
    val videoQuality: VideoQuality = VideoQuality.SMALL,
9
    val backgroundColor: Color = Color.Black,
10
    val waterMark: Boolean = true,
11
    val showDate: Boolean = true,
12
    val showMessage: Boolean = true,
13
    val font: Fonts = Fonts.FIGTREE,
14
    val dateStyle: DateStyle = DateStyle.FULL,
15
    val stabilizeFaces: Boolean = false,
16
    val censorFaces: Boolean = false,
17
)

Storing Images and Keeping track of them#

The core database of the app is simple. It’s just two tables, one of all the projects and their descriptions other is all the days, which is an entity wrapping the location of the image with additional info like date, face data and the project ID to use as a foreign key. All using Room

I really wanted to use as little permissions as I could. Instead of accesing all the media files, I let the user select the image they want to add for a day using the Photopicker API. I was requesting persistable URI permission for each image that the user selected. Until I started facing some unexpected behaviours in production and discovered it was a ticking time bomb. Not only was this approach really naive but each app is only allowed to have a maximum of 512 persisted URIs at any given moment.

Quickly scrapped that approach and opted for copying the selected images in the app’s files directory. Thankfully it was within 512 days, so I really dodged a bomb there. The surprising thing is, This limit is not mentioned anywhere in the API docs. The only place I found this was in an obscure GitHub issue (which I can’t find anymore) and the android source code 😭

1
  // Maximum number of persisted Uri grants a package is allowed
2
  private static final int MAX_PERSISTED_URI_GRANTS = 512;

Extracting faces from Images#

This was simple, I knew mlkit had an API for extracting faces from images for android. So I implemented it pretty easily. But as the number of images in a project grew, It became clear that the bounding boxes were not that accurate. The images won’t seamlessly transtition over each other and the final result came out jarring. Mlkit is also a closed source library, meaning I could not know how it works, publish my app on FOSS appstores, add features or debug it properly. Also for some reason mlkit processes can’t run inside coroutines? it can only run in Google’s com.google.android.gms.tasks.Tasks. This really bothered me.

Then I came across mediapipe which is also a Google library that is more like a harness to run ml tasks across many devices seamlessly. It was FOSS and was much more accurate and powerful than mlkit. The only downside is it is heavy and requires tasks to be included as assets. Besides face detection it can also be used to analyze face landmarks, pose detection, emotions and much more which will be useful in the future for other features.

One very peculiar thing I noticed with Mediapipe. It has a Google logging library built in, GoogleDataTransport. It is an internal library that they use in their sdks for logging usage. I tried to exclude this from my app by excluding it in buildscripts, which caused crashes. Then I tried stripping it out using proguard, that caused unexpected bugs. Finally, I decided to let it be but remove the network permission for the app entirely in the GitHub release. Without the network permission the app can’t log anything. It was crazy to see the lengths google is going to collect telemetry. When a log attempt failed for some reason, It scheduled a job to try again later! I’m probably naive, but I don’t get why telemetry is that important.

Assembling the Video maker#

To make things fast. Momentum processes and saves the face data for each image the moment they are selected. This avoided flooding the cpu with ml tasks all at once. Also added a button in each Day’s page to view the bounding box around the face. With all the metadata collected for an image and Montage creation preferences, wrote a function to create an edited bitmap from the given image. Face adjustment, scale, watermark and text to be shown in the montage were configured in this step.

Then the processed images are collected in flows to be passed to the video muxer, which takes the images as they are generated and muxes them into a video all in sync

1
var processedCount = 0              // for tracking progress in UI
2
val bitmapFlow = flow {             // flow created
3
    sortedDays.forEach { day ->
4
        processDay(
5
            // function to process each day
6
            day = day,
7
            config = config,
8
            textPaint = textPaint,
9
            censorPaint = censorPaint,
10
        )
11
            ?.let { bitmap -> emit(bitmap) }
12
    }
13
}
14

15
val flowForMuxer =
16
    bitmapFlow.onEach {
17
        processedCount++
18
        emit(MontageState.ProcessingImages(processedCount.toFloat() / total))
19
    }
20

21
when (val result = muxer.mux(flowForMuxer)) {  // muxer consuming the flow
22
    is MuxingResult.MuxingError -> {
23
        emit(MontageState.Error(result.message, result.exception))
24
    }
25

26
    is MuxingResult.MuxingSuccess -> {
27
        emit(MontageState.Success(result.file, config))
28
    }
29
}

The first iterations of this pipeline were a total unoptimized mess. This is the way I am sticking to for now.

The UI#

It’s all Jetpack Compose with MVI, obviously. MVI is the best design pattern, architecture, whatever you name it for Jetpack Compose. All my states are in neat data classes and the UI is just a function that takes the state. Also, a lambda to bubble up events. Tried to make it look more Material 3 Expressive.

1
@Immutable
2
data class HomeState(val projects: List<ProjectListData> = emptyList()) // state
3

4
sealed interface HomeAction { // predefined actions the UI can perform
5
    data class OnChangeProject(val project: Project) : HomeAction
6
    data class OnAddProject(val title: String, val description: String) : HomeAction
7
}
8

9
@Composable
10
fun HomeGraph(   // UI composable that only takes the state and exposes a lambda for the ViewModel
11
    state: HomeState,
12
    onAction: (HomeAction) -> Unit,
13
    onNavigateToSettings: () -> Unit,
14
    onNavigateToProject: () -> Unit,
15
    isPlusUser: Boolean,
16
    onNavigateToPaywall: () -> Unit,
17
    modifier: Modifier = Modifier,
18
) {
19
    // ....
20
}

Dependency Injection (Koin)#

I tried out koin annotations with this project. With the introduction of the new Koin Gradle plugin, it’s now much easier to inject classes. You just need to annotate them with @Single, @KoinViewModel and koin will automatically detect these classes and validate if all the requested injections are satisfied or not during compile time!

Behind the scenes, it’s just an extra step over the regular service locator approach of koin. It just automates the generation of a Module that maps all the dependencies in the koin dsl

1
@Single
2
class MyClass()
3

4
// generated code
5
module = module {
6
    single { MyClass() }
7
}

Navigation 3#

Also tried out the new navigation 3 after many friends suggested it. I am genuinely blown away by how simple it is to implement. The backstack is just a list! and you add or remove objects from it to navigate. You can also animate navigation easily using metadata and android’s new predictive back gestures are supported out of the box.

1
// objects representing different nav destinations
2
@Serializable data object ProjectDetails : NavKey
3
@Serializable data object ProjectCalendarView : NavKey
4
@Serializable data object ProjectMontageView : NavKey
5
@Serializable data class DayInfoView(val selectedDate: Long) : NavKey
6

7
// just a list!
8
val backStack = rememberNavBackStack(ProjectDetails)
9

10
NavDisplay(
11
    modifier = modifier,
12
    backStack = backStack,
13
    entryProvider =
14
        entryProvider { // similar to nav 2
15
            entry<ProjectDetails> {
16
                ProjectDetails(...)
17
            }
18

19
            // metadata specifying transition animation between screens
20
            entry<ProjectCalendarView>(metadata = verticalTransitionMetadata()) {
21
                ProjectCalendar(...)
22
            }
23

24
            entry<DayInfoView>(metadata = verticalTransitionMetadata()) {
25
                DayInfo(...)
26
            }
27

28
            entry<ProjectMontageView>(metadata = horizontalTransitionMetadata()) {
29
                ProjectMontageView(...)
30
            }
31
        },
32
)