Securing an Embedded System by Catching the Oversights

Before we dive in, I would like to thank Gili Yankovitch who was kind enough to spend his time discussing some items and reviewing the text.

Gili, that’s only the first one!

The ever-growing number of IoT, industrial, imaging, and other embedded devices flooding the markets is both exciting and scary. It’s no joke: millions of devices control and affect our lives in different ways. If a major, repetitive problem occurs in just a tiny fraction of the “fleet,” many systems could suffer.

This situation is scary because of robust security — namely, the lack of it. Devices are not well-protected, and the number of risks and remediation options is overwhelming.

Importance of Security

Creating a comprehensive security suite for a device is an art: too heavy and it stumbles over its own complexity, but too light makes for a brittle system. The art is in creating strong security without impeding the device’s functionality.

Many articles bring up the topic of security issues in embedded systems. While most are solid and truthful, they are scattered across the internet and sometimes give too-broad answers, like implementing secure FW upgrade or using Secure Boot mechanics.

Instead of adding yet another to the stack, I’ll touch on things that are either rarely discussed or everyone is silent about. Here are some unusual gaps and obstacles that you might have to deal with.

Process Failures

No Threat Model

The most obvious and equally common missing point is the absence of a threat model, “a structured representation of all the information that affects the security of an application” (definition courtesy of OWASP author Victoria Drake) and patterns/templates that clarify how to deal with unknowns.

Different teams advocate different approaches or techniques, such as Agile or Waterfall or whatever mix suits the situation. Regardless of the domain, security design should be considered before the development process even begins.

Having a threat model early in the product lifecycle won’t provide 100% security coverage, but it will eliminate excuses — any change should be reflected in it.

No Decision-Made Records

Experts agree that Architecture Decision Records (ADRs) or Architecturally-Significant Requirements (ASRs) define the best way to document solutions.

Github offers examples of this flexible format. It would help to learn more about them and use them, in addition to maintaining threat model representations.

Document every decision made and log all action items and points provided. Ideally, each decision should link to the mitigation measures described in the first document.

No CVE Tracking

One important thing engineers rarely think of is maintaining a database of all the critical CVEs related to major hardware components (MPUs, MCUs, memory, etc.) in the hardware BOM.

To build said database:

1. Download all current revisions of application notes (AN) and data sheets (DS) for each component and view the ChangeLog section.
2. Find vendor community forums and look for entries that mention a specific part number.
3. Contact the vendor and ask about problems that have yet to be disclosed. Helpful advice:

• Use a company email address and word the request politely — be nice
• Write on behalf of the company rather as an individual
• Get a contact point or ask for a dedicated service engineer

Choice of Components

Limited selection of components

Not all products suitable for the project are available within the project’s timeframe; they may be announced but only in development or unavailable for another reason.

Sometimes a business or startup doesn’t fit the expectations or requirements of a chosen vendor:

• low estimated volume
• geopolitical circumstances (take into consideration political situation)
• possibility of dual-use

These criteria, coupled with the pandemic-generated chip shortage, make ordering parts difficult for many companies.

With security MCUs, even for simple documentation, a purchaser must prove to a vendor the firm intention to use the solution in the future. And that future must be bright and beneficial for the vendor.

Suboptimal hardware choices

Limited selection of components and poor hardware choices result in less-than-ideal decisions:

• Not being able to get an EVK in time to test all the features prevents reasonable choices. Full access to documentation is not enough to make the best judgment.
• Time constraints or deadlines contribute to confusion and missteps
• Reconsidering decisions also means updating the threat model, weighing the pros and cons of the new hardware, talking with vendors and finding alternatives.

Lack of hardware support for latest crypto algorithms

Advances in software come faster than in hardware. Why? because it’s not an easy task to update existing hardware blocks with new and fancy algorithms. The situation gets complicated when we talk about new algorithms in cryptography (hashing or a new type of signature).

Advances and new things in the security field must undergo extensive analysis and continuous testing to ensure adequate security level before everyone believes they work (be time-tested).

Do not forget about certification that is usually time-consuming and costly, such as FIPS 140 process and steps required to pass the validation.

For example, I needed a secure MCU that inherently supported ECC curves other than the default (e.g. Montgomery). I spent a lot of time pushing emails and checking part numbers before finding an appropriate solution (NXP, Infineon). Then I had to wait until things started rolling up.

Often, you must compromise: either sacrifice product security with a simpler solution or wait for new hardware and pray that the MVP can make its delivery deadline.

Security Audit

ideally, every security solution, and every firmware design as a whole, should be created by dedicated experts with solid knowledge in that particular area. Unfortunately, that is not always the case, as often all pieces are designed and implemented by Jack of all trades.

The good thing is that thinking about the need for security architecture is becoming prevalent these days. But how far does that thinking go?

Lack of security review

It’s OK for the security architecture to be designed and implemented by project engineers without security experts. A senior architecture engineer should be able to do that kind of job.

But arranging a review with a security expert is a must. All decisions and reasons for them should be documented (see first section) for reference.

Expert knowledge shades affordable tradeoffs

Although third-party experts are good for reviewing designs, they are not necessarily best at creating them. They may also cast shade on certain options to highlight preferred choices. Having engineers who oppose an expert’s opinion makes it easier to analyze potential tradeoffs.

Absence of the security audit as a service

• D grade: discussing security architecture
• C grade: creating threat model and appropriate documentation
• B grade: mandating security audit
• A+ grade: rolling out a CI/CD pipeline specifically for security

Discrepancies in expert opinions regarding foundational mathematics

Security deals with cryptography and the latter is all about mathematics. Knowing the subject is a must, but practical experience is also very important. Experts will have varying depths of knowledge and experience, creating different views on the same problem that may confuse engineers.

Topics in cryptography may involve complex math concepts with few practical experiments and tests, or risks mitigation measures with unclear or unproven results.

Over-designing

Excess enciphering

Being paranoid is a must for a security expert. Indulging in the urge to scatter crypto primitives here and there (encrypting each piece of data stored on MCUs or protocols used to talk with peripherals) may introduce new bugs and errors because of the increased codebase.

Try to design simple things, as they are easy to review, discuss and scale.

Too complex functionality breaks simplicity

Sometimes hardware lacks the required capabilities and this issue only comes to light after the BOM list is approved, schematics are designed and development is in progress, so no hardware changes are allowed.

What happens next? Firmware accepts the challenge by implementing the missing parts! As a result the risk of introducing new vulnerabilities and errors increases.

Complex requirements cause complex, error-prone solutions.

Conclusion

I started thinking about the key ideas of this article before I began writing it. I didn’t want to elaborate on yet another bullet list of security-related items, even though they seem practical. Instead, I wanted to delineate what would be the focal point of the article:

Security gaps are unpleasant things, but it’s OK. We all make mistakes and we learn from them. And the more we know, the more chill we can be when a crisis strikes.

By Sergey Danielian, Head of Embedded Systems, QuantumSoft