Shepherding security

In late 2021 OSPOCO was brought in to help with the Magma Project. The project sponsor - originally Meta, later the Magma Core Foundation - quickly identified that security was a core need of the project. We spoke with Lucas Gonze about his experiences working on Magma.

About Magma

Our first assignment was to improve the uptake of Magma by outside users and contributors. We needed to find a way to start making progress while we became accustomed to the project and got to know the code.

Magma is only a mid-size project, but it is complex. It is an implementation of the 3GPP core. That's the technology that allows 3G, 4G, and 5G phones to connect to their networks. There is a lot of system and protocol-level complexity that is mandated by the standards.

We began by working on Dependabot alerts and automated security advisories. (For those who aren't familiar, see our blog post on Dependabot.) Github security advisories are similar, but those are based on automated flagging of patterns that may have security implications.

Security in the supply chain

As we got deeper into these vulnerability alerts, it became clear that a dedicated focus on security was needed. The main developers were busy in the core and this was an area that hadn't received a lot of attention.

Projects like Magma have an interesting spot in the open source ecosystem. For end users, Magma is a deployable product that needs to be responsible for its own security. But Magma is also a consumer of many other open source libraries. It has its own supply chain that needs to be managed.

As Magma was moved from an internal project at Meta into a neutral, open source project, these supply-chain security issues started to accumulate. Over time the growing number of alerts made potential users and contributors wary of even getting started with the project.

Steps toward implementing secure processes

As we stepped into the broader realm of product security, we needed more than just a dedication to decreasing alerts. We needed a security process. We took this in several steps:

Creating a list of dependencies: Our first task was to create an audited list of our third party dependencies. As we wrote on the OSPOCO blog, the core administrative function of an Open Source Program Office is making sure you know what open source software your organization is using. Every other function, including security, relies on this basic knowledge.

We started by looking at several methods of creating a software bill of materials (SBOM) to find ones that would integrate the best with Magma's existing processes. Following our research, we used the tools to create an SBOM and created a pull request to check it into Magma's core repository.

Using the OpenSSF Scorecards: Our next step was getting OpenSSF Scorecard results. The Open Source Security Foundation (OpenSSF) is a multi-vendor effort to create validated security models and practices for open source projects. They created Security Scorecards to give consumers of open-source projects an easy way to judge whether their dependencies are safe.

Scorecards is an automated tool that assesses a number of important heuristics ("checks") associated with software security and assigns each check a score of 0-10. You can use these scores to understand specific areas to improve in order to strengthen the security posture of your project. You can also assess the risks that dependencies introduce, and make informed decisions about accepting these risks, evaluating alternative solutions, or working with the maintainers to make improvements.

We investigated the use of scorecards for Magma, quickly identifying key practices that we could improve.

Establishing a Security Working Group: The third thing we did was create a Magma security team with its own tools. We created an issue tracker specifically for security issues and created policies and methods for Magma to receive security vulnerability reports.

Just-in-time security

We had barely set up our first processes before we started receiving the first reports. It was clear that we received some vulnerability reports just because we were prepared to receive them. If we didn't have a vulnerability disclosure reporting process, we might never have found out about some of the issues that were reported to us.

The single best vulnerability disclosure we received came from an academic researcher who is interested in the 3GPP standards and who was hunting for protocol-level vulnerabilities or weaknesses. A lot of really good thought went into his disclosure. Mitigating it was not easy, but the process was clear.

Code secrets and security

Another concrete thing that we did was that we got our code secrets under control. Code secrets are things like API keys, internal passwords, etc., that are used to manage internal connections securely.

Over the lifetime of the project there have been almost 3000 code secrets potential exposure reports - places where one of these secrets might have been exposed. I read pretty much every single one of those to see which ones were real and which ones were false alarms - and then I had to triage all the items that were real.

After doing our review we found some instances where private keys or API keys were being stored in a public repository. Many people don't realize that the repository stores all history, so even if you delete the key, it is still there in the history. So we went through and revoked all the keys that were vulnerable and issued new ones using proper secrets management.

Results

With a project of this size, we needed to both deal with the backlog of alerts as well as deal with new vulnerabilities that became public. We regularly worked on tracking all open vulnerabilities, prioritizing them to be sure that We were putting our available resources into the most important, most impactful items.

We have a couple security dashboards that we use to track our progress. Here are a couple screenshots showing our progress over the past year.

Over time were able to bring the total number of reported vulnerabilities down from over 5000 to our current count of ninety. And even that number - ninety - is mostly Dependabot alerts that we are working to solve upstream, outside of Magma itself.

The impact of security

When we look at what we have learned, the first lesson is that security occupies a high visibility role. People don't want to use or deploy projects that don't appear to handle their security well - even if they do! Having a visible security process is necessary to build wide adoption.

Thinking about Magma specifically, this software is used for infrastructure. It is the foundation of other products. Because Magma is so fundamental, security isn't just overhead, it is a product benefit. The telecom industry has security as a high priority. They're kind of security consumers. They actively look to see if they have reasons to believe in the security of the software. We have discovered that our active approach to security and our public security record can help "sell" Magma and encourage broader adoption.

We have also observed that security has become a point of pride in the Magma community. When we started, I think you might have said that for some people it was a sore spot. Now it is something that the whole community is passionate about.

Final thoughts

Security is a driving force in today's OSPOs. While compliance has traditionally been the core task of OSPOs, it seems that encouraging good open source security practices are going to be a key part of any OSPO's work going forward.

It is always hard to figure out how to best allocate your engineering labor versus features or security. And it is true that security processes can take time, especially at first. However, we have observed that having a consistent approach to security helps keep code velocity over the long run. The software is less brittle and easier to extend.

These days, open source management and security go hand in hand. We usually think about security from the perspective of open source consumers. But open source projects also need to manage their own security issues as a necessary part of driving adoption and deployment.