Live Android v1
Live Android
Live
Android
Version 1
Home
/
Live
/
Android

Camera filter

Camera filter is a feature that you can use to apply face filters on the host's face. By using various face filters, the host will be able to offer an engaging and entertaining live event experience for the participants. Follow this guide to integrate Banuba Face Filters SDK to Sendbird Live SDK.


Requirements

The minium requirements to implement camera filter for Live SDK for Android are:

  • Android 5.0 (API level 21) or higher
  • Java 8 or higher
  • AndroidX
  • Android Gradle plugin 4.0.1 or higher
  • Sendbird Live SDK 1.0.0-beta.5 or higher
  • Banuba Face Filters SDK

Before you start

Before installing the Live SDK, create a Sendbird account to acquire an application ID which you will need to initialize the Live SDK. Go to Sendbird Dashboard and create an application by selecting Calls+Live in product type. Once you have created an application, go to Overview and you will see the Application ID.


How it works

In this guide, you will be installing Sendbird Live SDK and Banuba Face Filters SDK. After installing and initializing the two SDKs, you will set up a live event by using some essential UI components to integrate face filters to a live event in the quickest manner.

Once you create a user to act as a host, you can start a live event and add face filters on the host's camera feed. You can also enter the live event to watch as a participant using our UIKit sample. Follow along this guide using the sample app for camera filter which you can download here.

If you would like to learn more about how to implement the full-fledged Live UIKit, go to the start your first live page.


Install the Live SDK and Face Filters SDK

First, you can install the Live UIKit for Android through Gradle. If you use Gradle 6.7 or lower, add the following code to your root build.gradle file.

allprojects {
    repositories {
        maven { url "https://jitpack.io" }
        maven { url "https://repo.sendbird.com/public/maven" }
        maven {
            name "GitHubPackages"
            url "https://maven.pkg.github.com/sdk-banuba/banuba-sdk-android"
            credentials {
                username = "sdk-banuba"
                password = "\u0067\u0068\u0070\u005f\u0033\u0057\u006a\u0059\u004a\u0067\u0071\u0054\u0058\u0058\u0068\u0074\u0051\u0033\u0075\u0038\u0051\u0046\u0036\u005a\u0067\u004f\u0041\u0053\u0064\u0046\u0032\u0045\u0046\u006a\u0030\u0036\u006d\u006e\u004a\u004a"
            }
        }
    }
}

Note: Make sure the above code block isn't added to your module build.gradle file.

If you use Gradle 6.8 or higher, add the following code to your settings.gradle file.

dependencyResolutionManagement {
    repositories {
        maven { url "https://jitpack.io" }
        maven { url "https://repo.sendbird.com/public/maven" }
        maven {
            name "GitHubPackages"
            url "https://maven.pkg.github.com/sdk-banuba/banuba-sdk-android"
            credentials {
                username = "sdk-banuba"
                password = "\u0067\u0068\u0070\u005f\u0033\u0057\u006a\u0059\u004a\u0067\u0071\u0054\u0058\u0058\u0068\u0074\u0051\u0033\u0075\u0038\u0051\u0046\u0036\u005a\u0067\u004f\u0041\u0053\u0064\u0046\u0032\u0045\u0046\u006a\u0030\u0036\u006d\u006e\u004a\u004a"
            }
        }
    }
}

Note: To learn more about updates to Gradle, visit this page.

Next, for all Gradle versions, open the build.gradle file at the application level. For both Java and Kotlin, add the following code block and dependencies.

apply plugin: 'com.android.application'

android {
    ...

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8    // Make sure you have JavaVersion 1.8.
        targetCompatibility JavaVersion.VERSION_1_8    // Make sure you have JavaVersion 1.8.
    }
}

dependencies {
    // Sendbird Live SDK
    implementation 'com.sendbird.sdk:sendbird-live:1.0.0-beta.5' //  Implement the mentioned version or higher.

    //Banuba SDK
    implementation "com.banuba.sdk:banuba_sdk:1.4+"

}

Request permission to access camera and microphone

To use camera and microphone on a mobile device, you need to ask the users to grant the media access permission on their devices. This permission is also needed to access the photo library.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.sendbird.liveuikitapplication">

    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
    <uses-permission android:name="android.permission.READ_PHONE_STATE" />

    <!--  To detect bluetooth audio device  -->
    <uses-permission android:name="android.permission.BLUETOOTH_ADMIN" />
    <uses-permission android:name="android.permission.BLUETOOTH" />
    <uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />
    <uses-permission android:name="android.permission.BLUETOOTH_SCAN" />

    <!--  To utilize audio in livestreaming event  -->
    <uses-permission android:name="android.permission.CAMERA" />
    <uses-feature android:name="android.hardware.camera" android:required="false" />
    <uses-feature android:name="android.hardware.camera.autofocus" android:required="false" />

    <!--  To utilize video in livestreaming event for audio  -->
    <uses-permission android:name="android.permission.CALL_PHONE" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
    <uses-permission android:name="android.permission.AUDIO_SESSION_ID_GENERATE" />

    ...
</manifest>

Initialize the Live SDK

To integrate the Live SDK in a client app, initialize the Live SDK with a Sendbird application ID by adding the code below.

// BaseApplication.kt

const val SENDBIRD_APP_ID = "YOUR_SENDBIRD_APPLICATION_ID"
const val BANUBA_TOKEN = "YOUR_BANUBA_TOKEN"

If you already have an application ID, log in to Sendbird Dashboard, go to Overview and you will see the Application ID, or get one by creating an application by selecting Calls+Live in product type.

// BaseApplication.kt

class BaseApplication : Application() {
    ...

    override fun onCreate() {
        super.onCreate()
        val params = InitParams(SENDBIRD_APP_ID, applicationContext, false)
        SendbirdChat.init(params, object : InitResultHandler {
            override fun onInitFailed(e: SendbirdException) {
            }

            override fun onInitSucceed() {
                val result = SendbirdLive.init(applicationContext, SENDBIRD_APP_ID)
                // If Sendbird.init() is successful, result return true and you can proceed to the next step.
            }

            override fun onMigrationStarted() {
            }
        })
    }
}

Next, add BaseApplication to AndroidManifest.xml.

// AndroidManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">

    <application
        android:name=".BaseApplication"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:roundIcon="@mipmap/ic_launcher_round"
        android:theme="@style/AppTheme">

        ...

    </application>
</manifest>

Authenticate a user

To use the interfaces of the Live SDK, you need to authenticate a user. You can authenticate a user by providing their user ID and access token in SendbirdLive.authenticate. This will establish a connection between the Sendbird server and the user. Once you have authenticated and connected the user, the user can act as a host to create, enter, and start a live event.

// SignInActivity.kt

class SignInActivity : AppCompatActivity() {
    ...

    private fun sendbirdAuth(userId: String, accessToken: String? = null) {
        val params = AuthenticateParams(userId, accessToken)
        SendbirdLive.authenticate(params) { user, e ->
            if (user == null || e != null) {
                return@authenticate
            }
            startActivity(Intent(this, StartLiveEventActivity::class.java))
            finish()
        }
    }
}

Set up a live event with UI components

To set up a live event, provide UI views to create, enter, and start a live event. This will allow you to start a live event using the most essential and minimal components which would look like views shown in the image below.

]

Step 1 Create and enter a live event

First, provide credentials to log into the client app. After logging in, select the Go live button to create and enter the live event. When you enter, you will be the host of the live event which allows you to control the live event such as whether to start or stop your media stream.

// StartLiveEventActivity.kt

class StartLiveEventActivity : AppCompatActivity() {
    ...

    private fun goLiveEvent(hostId: String) {
        val params = LiveEventCreateParams(listOf(hostId))
        SendbirdLive.createLiveEvent(params) { liveEvent, e ->
            if (liveEvent == null || e != null) {
                return@createLiveEvent
            }
            liveEvent.enterAsHost(MediaOptions()) {
                if (it != null) return@enterAsHost
                val intent = Intent(this, LiveEventActivity::class.java)
                intent.putExtra(KEY_LIVE_EVENT_ID, liveEvent.liveEventId)
                startActivity(intent)
            }
        }
    }
}

Step 2 Start the live event

When you start the live event, participants can enter your live event, view the host's media stream, and chat. To start the live event, follow the code below.

// LiveEventActivity.kt

class LiveEventActivity : AppCompatActivity() {
    ...

    private fun startLiveEvent(liveEventId: String) {
        SendbirdLive.getLiveEvent(liveEventId) { liveEvent, e ->
            if (liveEvent == null || e != null) {
                return@getLiveEvent
            }
            liveEvent.startEvent {
                if (it != null) return@startEvent
                val hostId = liveEvent.host?.hostId ?: return@startEvent
                liveEvent.setVideoViewForLiveEvent(binding.svvLiveEvent, hostId)
            }
        }
    }
}

If you would like to learn more about using the full-fledged UI features for Sendbird Live, go to Live UIKit Overview page.


Initialize the Face Filters SDK

You need to initialize the Face Filters SDK that you downloaded previously by using Banuba's SDK token. Visit their website to get a valid token and follow the code shown below to initialize the SDK.

// LiveEventActivity.kt

class LiveEventActivity : AppCompatActivity() {
    ...

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        BanubaSdkManager.initialize(this, BANUBA_TOKEN)

        ...
    }
}

Stream with Face Filters SDK

Once the Face Filters SDK is initialized, you can select filters to apply to the host's face. To stream a live event that has filters applied, you need to let the Live SDK know that such external video will be streaming. Then, the Face Filters SDK delivers modified video frames through its handler which the Live SDK can use to stream to the participants. Take the steps below to apply the filters and stream the modified video.

Step 1 Configure the Face Filters SDK

First, set up the Face Filters SDK with the correct configurations then choose the desired face filter. You can learn more about the core feature that applies face filters from here.

// LiveEventActivity.kt

class LiveEventActivity : AppCompatActivity() {
    ...

    private var mOEP: OffscreenEffectPlayer? = null
    private val handler = Handler(Looper.getMainLooper())

    private fun initOEP() {
        val size = getWindowSize()
        val effectPlayerConfig = EffectPlayerConfiguration(
            size.width,
            size.height,
            NnMode.ENABLE,
            FaceSearchMode.MEDIUM,
            false,
            false
        )
        OrientationHelper.getInstance(this).startDeviceOrientationUpdates()

        val effectPlayer = EffectPlayer.create(effectPlayerConfig) ?: return
        effectPlayer.setRenderConsistencyMode(ConsistencyMode.ASYNCHRONOUS_CONSISTENT)
        val oepConfig = OffscreenSimpleConfig.newBuilder(mBuffersQueue).build()
        mOEP = OffscreenEffectPlayer(applicationContext, effectPlayer, size, oepConfig)
    }
}

Step 2 Streaming with external video frames

To receive the external video frames, the enterAsHost() method must be called before adding the listener and streaming with an external video. By implementing the code below, you can pass the video frames from the Live SDK to the Face Filters SDK.

// LiveEventActivity.kt

class LiveEventActivity : AppCompatActivity() {
    ...

    private val videoFrameListener = VideoFrameListener { videoFrame ->
// Send the frames to the Face Filters SDK to apply filters.
// Do not pass videoFrame directly to the SDK without processing the video frames using the Banauba's Face Filters SDK.
    }
}

Then, register addVideoFrameListener to receive events about video frames from the Live SDK.

liveEvent.addVideoFrameListener(videoFrameListener)

To start streaming your live event with external video frames, call startUsingExternalVideo().

liveEvent.startUsingExternalVideo()

Step 3 Apply face filters to external video frames

You can select face filters to apply to the host's face. Once you select filters, the Face Filters SDK modifies the camera feed to apply the filters and returns the processed frames through its handler. Pass the frames to the live event to stream to the participants.

// LiveEventActivity.kt

class LiveEventActivity : AppCompatActivity() {

    private fun initOEPFramesOutputToSendbirdLiveSDK() {
        val mOEP = mOEP ?: return
        mOEP.setImageProcessListener({ oepImageResult: ImageProcessResult ->
            ...

            liveEvent?.enqueueExternalVideoFrame(videoFrame)
        }, handler)
    }
}

Step 4 Stop streaming with external video frames

If you would like to stop streaming with face filters, call stopUsingExternalVideo(). If you would like the host to stream from another camera, specify the device using the selectVideoDevice() method.


Watch the live event as a participant

To watch the ongoing live event as a participant, download Sendbird Live sample app for iOS from The App Store, for Android from Google Play, or from Github. You can also use Live studio from Sendbird Dashboard to test Sendbird Live.