Optimizing GitHub security at Sendbird: Challenges & key takeaways
Sendbird uses GitHub to store and interact with all our codebases from a single trusted source. GitHub as a platform has many options to secure access, but doesn’t enable these options by default. GitHub also has some shortcomings that need automations to be built out. We identified areas where we could improve our setup and started an initiative to redo how we consume GitHub internally to enable secure by default ease of access to platform users.
Recognizing opportunities for improvement, we initiated a GitHub security project to enhance our internal usage of Github. Our goal was to create a more secure and user-friendly environment. In this post, we’ll share our journey, detailing the initial state of our GitHub setup (similar to what most startups would have), the steps we took to improve it, the challenges we encountered, and areas where we see room for further enhancement.
Initial state of GitHub security
While our GitHub security configuration wasn’t problematic in the true sense of the term, it had room for improvement in scalability and automation. Here are some key challenges we identified:
1. User authentication
Users were logging in with their GitHub username and password and were added to our Sendbird organization with the requirement that they have already enabled multi-factor authentication.
2. User and team management
All user and team management was done manually, with administrators individually adding and deleting users from the organization and managing team memberships directly.
3. External collaborators
We allowed external collaborators (such as contractors) on our GitHub organization, but a mature process for access management needed to be implemented. This meant there was a possibility of external collaborators who no longer needed the access, but still mistakenly had it.
4. Central organization structure
We had the entirety of our platform and related codebases in a single GitHub organization, which made it difficult to manage permissions and enforce controls. There were instances where few repositories needed more flexible controls than what we wanted to enforce on the organization level. This made it difficult to have uniform controls yet enable the business.
5. Repository ownership
We’ve all been in this situation: Someone created a repository with application code at some point, and now that’s a critical repository; but who actually owns this repo? Who is responsible for it? To answer this question, you need to talk to someone who’s been at the company for a while. This person will then track down someone who knows about that repository.
At Sendbird, many repositories existed for so long that even if we wanted to clean them up, it was challenging to identify which were necessary and which weren’t. There was no clear mapping of repositories to teams, which, in turn, made it difficult to ask teams to take ownership of repositories.
Key improvements made to Sendbird’s GitHub security
To address these challenges, we implemented a series of improvements in our GitHub security setup in close partnership with our engineering teams.
1. Authentication via Okta SSO
The first step was to streamline user authentication to our GitHub organization. Because we use Okta as our SSO user authentication solution, we set up user authentication to our main Sendbird GitHub organization via Okta. GitHub documentation is a great guide for this setup.
Once the Okta integration was set up, we assigned the Okta app to users who needed it through Okta group assignments.
2. Authentication policy
We also implemented Okta authentication policies to allow authentication only from a company-provisioned device, along with a FIDO2-compliant multi-factor authentication (yubikeys or user fingerprints). This was a part of another project in the company to uplift our enterprise security posture.
3. Okta group syncs for teams
Okta groups are among the cleanest ways to manage team-based access to applications and resources. GitHub uses GitHub Teams, which is a grouping of users that can be used to provision access to repositories at a team level, rather than at a user level.
We use BambooHR at Sendbird as our Human Resource Information System (HRIS). To ensure accurate team membership in Okta groups, our IT team synced Okta groups with BambooHR users. We then created Github teams that synced with these Okta groups to ensure that these Github teams reflected the content in our HRIS. Setting up SCIM provisioning and team synchronization are prerequisites for the Okta group syncs to work
4. Need-based GitHub organizations
A lot was happening in our main Sendbird Github org, which served multiple purposes. Because of this, we wanted to clear up the main org so that only codebases pushed to our cloud infrastructure and customer-facing application code repositories would be present. This raised questions such as:
Where should people go if they’re building something for a hackathon or playing around with code?
Where should external collaborators be added? What about all the repos not being used in the current org?
We split our GitHub into organizations based on needs to address these questions. We created multiple orgs under our GitHub enterprise to cater to different use cases. We enabled Okta SSO on each of the orgs and only assigned each org to people who needed it, limiting who had access to different types of repositories. Example Github orgs:
Sendbird-collabs: Repos that needed external collaboration were moved to a new org called sendbird-collabs and the external collaborators were added to this org and only to the requested repositories. This ensured that non-Sendbird employees were limited in what they could access on GitHub.
Sendbird-playground: We created another org called sendbird-playground, which employees could use to collaborate during hackathons and to test things out.
We ended up creating six different organizations based on our collective use cases. This made it easier to manage permissions for users at an org level, which in turn gave us the ability to be firmer or more flexible in terms of controls based on the org’s purpose. (Example: Security controls such as SCA checks, PR +1 requirements are more stringent in our main org compared to the playground org.)
5. Organizing repositories
After splitting up the orgs, we needed to do a full audit of all the repositories in our main GitHub organization. We had teams take ownership of repositories they used regularly (refer to the challenges section for more) and everything else was moved to different organizations based on the repo’s use case. Additionally, there were repos that no one was using. All of these were moved to an org to which no engineer had access. This was to ensure that the repos were not brought back to our main sendbird org and once again ran into the issue of having repositories in the org without team ownership.
6. Team ownership via Backstage
Now that had done a cleanup of all the repositories in our main GitHub org, we needed to ensure that there was a persistent, automated way to have teams own repositories instead of the Security team asking in a Slack channel, Zoom call, or chasing down people in the office. For this purpose, we deployed Backstage, an open-source developer platform that had a built-in catalog-info.yaml file specification designed for a repository-level product taxonomy. We set up this service as a Kubernetes deployment with an AWS RDS backend and had all Engineering teams add the yaml file in a taxonomy specification we had agreed on.
We wanted to ensure that the data being put into these yaml files was valid and reliable. Therefore, we built a GitHub app to run as a status check and validate the contents of the yaml file during every commit and pull request on a repository. This ensured that the data in the yaml file was always current and reliable during times of incidents. It was also useful when we needed immediate ownership info on repositories.
We also requested Engineering to change all CODEOWNERS and repository collaborators' info to use the new GitHub teams, as well as deprecate and delete the old entries.
7. Audit logs for monitoring
The entire Sendbird enterprise audit logs were exported to S3 and then ingested into our SIEM solution, which our DART team used to monitor for malicious behavior. The audit log export to S3 is available natively in GitHub and is a convenient way to consume logs as it is SIEM vendor agnostic.
8. Pull request +1s
We enabled an org-wide enforcement on our main GitHub org such that all pull requests required a peer review before they could be merged to the default branch. This is for three reasons:
To ensure teammates catch any issues with a code change before it’s deployed.
We have code scanning tools for SCA, SAST, and IaC scanning. Having PRs mandated also means that we can run our code scanning tools as part of a PR pipeline and ensure we catch a good portion of security vulnerabilities before the code is merged to the default branch and deployed.
To meet compliance requirements.
You can use rulesets to enforce peer reviews before a PR is mergeable at the org level and ensure every angle repository gets a review on its PRs.
9. Revamp personal access tokens
We discovered that we had numerous personal access tokens in use without a clear understanding of their purposes, and many of these tokens were set to never expire. This was around the time that the CircleCi incident occurred. This was a big catalyst for a cleanup of all access tokens. We then mandated only fine-grained access tokens (restricting the use of PAT or Personal Access Tokens) with more restrictive permission schemes and expiry dates to ensure GitHub API tokens were more secure.
GitHub provides good documentation on restricting access tokens in your organization and would serve as a great guide for your implementation.
10. Restricting member privileges
We decided to restrict member privileges to only what was necessary on a daily basis. We decided to give users base-level read-only permissions on the org with write permissions on repositories. Users also cannot invite new members or external collaborators to the org and they can’t create teams, transfer or delete repositories, and fork our repositories. If someone needs any of the above privileges, then they can reach out to a GitHub admin to act on their behalf. The scale of such requests has been limited, so we are confident that this model caters to our current needs.
11. Code security
We have implemented various security controls within our Github organizations like SAST (with custom rule sets), SCA, secrets scanning, etc. Stay tuned for more details in a separate blog post!
Challenges faced in implementing robust GitHub security measures
All the above changes were made over a period of 1.5 years, with gradual rollouts and grace periods, ensuring we did not disrupt engineers. At times, we needed to reassess our approach to the rollout and make modifications to our plan. However, the following was the biggest challenge we faced during this project:
Repository cleanup and ownership
In our main Github org, we had about 800 repositories with no ownership information, which was why we could not clean up all unused repositories in an automated workflow. After discussing this with our engineering teams, we decided that the best path forward was to do a one-time manual review and cleanup.
We created a spreadsheet listing all the repositories in our organization and had teams review and mark all repositories as either ‘safe to delete’ or ‘still needed’. For repositories that teams marked as ‘still needed’, they were asked to add the Backstage catalog-info.yaml file to reflect team ownership info. This collaborative effort took about a month to complete and was really the biggest and only major challenge for this entire project.
The future of GitHub security at Sendbird
We’ve made a lot of progress since we started this project of cleaning up our GitHub presence and have made it more secure with a better user experience. However, like all other tech, there’s always room for improvement. With all the changes we’ve made, we’re confident in our secure-by-default setup of GitHub at Sendbird and now would like to make it easier to manage. Below are some of the things we want to improve on to make the management of the platform more accessible.
1. Handling team reorgs with Okta group syncs
Handling team reorganizations is one of the biggest challenges with our current setup. Since we rely on BambooHR to Okta group syncs, which we then manually map to GitHub teams, it’s a hassle to redo these team mappings when there’s an internal team reorganization. This doesn’t happen often, but when it does, we need to remap the newly created Okta groups with GitHub teams. If we create new GitHub teams, then engineers have to update their repositories’ settings collaborators and CODEOWNERS files to use the newly created GitHub teams.
We would like to have a cleaner and automatic sync from BambooHR to Okta groups to Github teams so that even reorgs will have minimum disruption to engineering activities.
2. Workflow for repository deletion
Since we restricted who can delete repos, engineers need to contact GitHub admins to perform this action for them. This requires admins to spend time on this. To free up admins from this task, we would like to build a Slackbot that end users can ask to delete repositories. This would then kick off a workflow to get approval from at least one admin via a Slack button/message, and then the bot backend would call the GitHub API to delete the repository. This would also create a Jira ticket to serve as an audit log of why it’s being deleted and make it easier for admins to perform such tasks with minimal touchpoints all through a single app (Slack).
Streamlining workflows & GitHub security at Sendbird
Our journey to optimizing and securing our GitHub environment at Sendbird has been comprehensive and insightful. We hope that by sharing our experience of optimizing our use of GitHub at Sendbird, we inspire other organizations to assess and strengthen their GitHub practices, leveraging available tools and documentation to build a safer, more efficient code management system.
Security is an ever-evolving field at Sendbird. We are committed to maintaining a secure-by-default GitHub environment that supports our current and future engineering goals by continually refining our approach and adapting to evolving needs.
