/ Platform API
Platform API
    Chat Platform API v3
    Chat Platform API
    Chat Platform API
    Version 3

    Image moderation

    Copy link

    Sendbird’s image moderation is powered by Google Cloud Vision API. This feature is used to moderate text and file messages with explicit images or inappropriate image URLs. It uses five categories to moderate images: adult, spoof, medical, violence, and racy. Image moderation doesn't apply when uploading channel images or profile images to Sendbird server.

    After an image is uploaded and moderated, the feature returns limit values. These numbers range from one to five for each category and corresponds to how likely the image will be blocked. The following shows what each limit means:

    LimitDescription

    1 (very unlikely)

    The probability of the image getting blocked is very unlikely.

    2 (unlikely)

    The probability of the image getting blocked is unlikely.

    3 (possible)

    The probability of the image getting blocked is possible.

    4 (likely)

    The probability of the image getting blocked is likely.

    5 (very likely)

    The probability of the image getting blocked is very likely.

    You can test different images on the Google Cloud Vision API's try-it tool to see how moderation works and determine which image moderation settings suit your needs.

    Note: This feature may not work on culturally sensitive images such as those related to religion, drugs, or weapons.


    HTTP request

    Copy link
    // To apply the settings application-wide.
    PUT https://api-{application_id}.sendbird.com/v3/applications/settings_global
    
    // To apply the filter to channels with a custom channel type.
    PUT https://api-{application_id}.sendbird.com/v3/applications/settings_by_channel_custom_type/{custom_type}
    

    Parameters

    Copy link

    The following table lists a parameter that this action supports.

    Required
    Parameter nameTypeDescription

    custom_type

    string

    Specifies the custom channel type to apply a set of settings to a channel.


    Request body

    Copy link

    The following table lists the properties of an HTTP request that this action supports.

    Properties
    OptionalTypeDescription

    image_moderation

    nested object

    Specifies a moderation configuration to moderate inappropriate images in the application. This feature is powered by Google Cloud Vision API, which supports various image types.

    image_moderation.type

    int

    Determines which moderation method to apply to images and image URLs in text and file messages. Acceptable values are the following:
    - 0 (none): no moderation is imposed. This is the default value.
    - 1 (normal): messages with images or image URLs that violate content policies are blocked.
    - 2 (strict): file messages with no images are also blocked in addition to messages with explicit images or inappropriate image URLs.

    image_moderation.soft_block

    boolean

    Determines whether to moderate images in messages. If true, the moderation method set by the type property above is ignored and no moderation is imposed on text or file messages containing explicit images or inappropriate image URLS. If false, messages containing images will be moderated according to the set moderation method.

    image_moderation.limits

    nested object

    Specifies a set of values returned after an image has been moderated. These limit numbers range from one to five and indicate the likelihood of the image passing the moderation standard. (Default: adult: 3, spoof: 5, medical: 5, violence: 3, racy: 4)

    image_moderation.limits.adult

    int

    Specifies the likelihood that the image contains adult content.

    image_moderation.limits.spoof

    int

    Specifies the likelihood that the image contains spoof.

    image_moderation.limits.medical

    int

    Specifies the likelihood that the image contains medical content.

    image_moderation.limits.violence

    int

    Specifies the likelihood that the image contains violent content.

    image_moderation.limits.racy

    int

    Specifies the likelihood that the image contains racy content.

    image_moderation.check_urls

    boolean

    Determines whether to check if the image URLs in text and file messages are appropriate. This property can filter URLs of inappropriate images but it can’t moderate URLs of websites containing inappropriate images. For example, image search results of adult images on “google.com” will not be filtered.

    * Moderation for image URLs applies only to the message property of a text or file message resource, not the data property.

    application widechannels with custom type
    {
        "image_moderation": {   // For more information, see the 'Properties' table above.
            "type": 1,  // The 'normal' moderation method.
            "soft_block": false,
            "limits": {
                "adult": 3,
                "spoof": 5,
                "medical": 5,
                "violence": 3,
                "racy": 4
            },
            "check_urls": true  // Check if the image URLs in messages are appropriate.
        }
    }
    

    If you want to turn off the image moderation, send a PUT request with a value of the type property set to 0 as shown below:

    application widechannels with custom type
    {
        "image_moderation": {
            "type": 0
        }
    }
    

    When a message has passed the image moderation, Sendbird server sends back a success response body like the following:

    application widechannels with custom type
    {
        ...
        "moderation_action": 1,
        "moderation_info": {
            "action": 1,
            "message": "Success",
            "scores": { // Image analysis results.
                "adult": false,
                "spoof": false,
                "medical": false,
                "violence": false,
                "racy": false
            },
            "cache_hit": false
        },
        ...
    }
    

    When a message has been blocked by image moderation, Sendbird server sends back an error response containing the 900066 (ERROR_FILE_MOD_BLOCK) or 900065 (ERROR_FILE_URL_BLOCK) code.


    Response

    Copy link

    If successful, this action returns the updated moderation settings or channels with a custom channel type in the response body.

    In the case of an error, an error object is returned. A detailed list of error codes is available here.