Test techniques and security evaluation approach
Security Evaluation approach
A security evaluation typically consists of a Vulnerability Analysis and a Penetration Testing phase. Both phases have their benefits, and often both of them are executed to evaluate the security of a product.
The Vulnerability Analysis is used to study the product and its context. The study is documentation based, so a physical product is not needed. A review of the design and code is in scope if the evaluation is done ‘white box’ style. The vulnerability analysis results in a report describing theoretical strengths and weaknesses of the product, within its context. The context is very important as this may amplify or mitigate potential weaknesses. As this phase does not include testing, it may not yet be possible to claim that a weakness can be exploited (and will become a problem in practice). Often the vulnerability analysis report includes a proposal for tests to be executed that should verify the exploitability of identified vulnerabilities. When only software security is in scope, and code review findings are conclusive, there is no need for additional testing. Otherwise, Penetration Testing is a valuable next step.
Security weaknesses can relate to the design, implementation, or configuration of a device. A Vulnerability Analysis is a white box review to identify those weaknesses. This evaluation phase is relevant for all vendors who wish to efficiently identify risks for their product. A Vulnerability Analysis can include:
A paper based evaluation of the product architecture (hardware / software). This focuses on methods to protect the security of the device, and identification of the attack surface. This activity is a first step in the evaluation. It helps the evaluator to understand the product and may already reveal design issues.
An exploration of attacks and study of their applicability to the product. This is an important step to scope further evaluation work, and focus on realistic risks.
This activity is about the relation of a product with its environment. Typically developers have a limited understanding of system components outside their responsibility, such as third-party code and underlying platform technology. Incorrect assumptions about the integration context may lead to vulnerabilities not visible to a developer. We check the correct interaction with other components according to their security guidance and configuration options.
Both software and hardware implementations can be reviewed as code. This is partially manual and partially automated work. Here we focus on verifying that the security design has been correctly implemented, countermeasures against known threats are in place, and common vulnerabilities are avoided. Typically the software is the weakest link in a product, and almost any software product has software vulnerabilities. The review can make a very significant contribution to improving the product security.
The Penetration Testing focuses on physical products. It is meant to verify if potential weaknesses exist, and that known weaknesses can be exploited. Testing can be done on hardware and software. Products that reside in a secure environment may need only software testing (which can typically be executed remotely), but products that operate in a potentially hostile environment (as most consumer products do) will need local tests that require physical access. This includes Hardware Testing, Side Channel Testing, and Fault Testing. Penetration Testing results in a report detailing test results, including a rating per exploitable vulnerability. This can be used by the customer to prioritize resolving issues.
Device Level Hardware Testing
We test hardware at chip and device level. This is a physical activity that requires local access, and can be destructive. It is a relevant activity for products that rely on the strength of their hardware. Device level tests have medium complexity and are typically in scope for consumer products. At device level this includes:
By reading out a memory chip we can reverse engineer sensitive code and data. This makes sense if the product code or data is confidential, and demonstrates the complexity of attackers to get this information.
By (temporary) replacing components or data connections we alter the device behavior leading to compromise of it functionality. In this test it may be possible to break assumptions in software about the trustworthiness of data, and as such these tests challenge the robustness of the software against basic hardware manipulation.
Abuse of test ports
We use JTAG and serial ports to extract sensitive information or change the behavior of the device. Almost any product has (internal) test ports, which may give low-level access to code and data. We verify that these test ports are sufficiently protected and do not provide a backdoor into the product.
Chip Level Hardware Testing
Chip level tests have high complexity and are typically not scalable. An attacker would use these methods to extract global information, such as sensitive code or keys. The tests are typically relevant for high-value products. Chip Level testing includes:
We open a chip and use microscopes to reverse engineer the chip. We aim for identification of security features and sensitive information (e.g. ROM content).
Focused Ion Beam modifications
We use a FIB to test the strength of security features. This includes repairing fuses, changing OTP cells, accessing test logic, and accessing buried data lines.
We use probe needles to access hidden functionality or exploit changes made through FIB edits.
Side Channel Testing
These tests verify that confidential information is well protected by software and hardware in a chip, and exploit information leakage through channels not meant for communication. Attackers use these methods to extract secrets with medium effort, but this does almost always require physical access. We collect side channel information to retrieve secrets (keys and pass codes) from chips. We use the following side channels to collect data:
We measure power consumed by a chip by observing current consumption;
We measure EM radiation emitted by a chip by using sensitive probes that cover part of a chip and zoom in to leakage from specific chip functions;
We measure infra-red light radiated by a chip using cameras and other optical measurement devices to identify the location of sensitive functions, and extract secrets;
We measure execution time of chips and devices to extract passwords and secret keys.
We use a variety of analysis methods to extract secrets from measurement data:
- Simple Analysis
A method where observation of a side channel directly reveals secrets due to observable variations;
- Correlation Analysis
By repeating a sensitive process many times and correlating measurements we use statistical methods to extract secrets;
- Template Analysis
We use a large measurement set to learn about expected behavior for specific confidential information and build corresponding templates. Next we use a single measurement, or small set thereof, to match with the established templates and retrieve the secret information.
With our tests a customer learns about the strength of their product, and how hard it is for an attacker to extract secrets.
These tests use excessive environmental conditions to manipulate the behavior of a chip. These tests have medium complexity and are powerful tools for attackers who wish to gain control of a device. Such attacks can also be used to extract binary code and attempt to find scalable software vulnerabilities.
We introduce faults in the processing of a chip in order to change its behavior and extract sensitive information. We use the following perturbation mechanisms:
- Voltage perturbation
By creating a brief glitch in the power supply we corrupt intermediate data during processing, resulting in a wrong decision or output;
- Electro-Magnetic perturbation
A strong EM pulse creates an internal spike on chip data lines resulting in a fault;
- Optical perturbation
Laser pulses on and exposed chip surface forces individual transistors, or groups thereof to switch and trigger a fault.
Faults can be exploited for the following goals:
- Extract secret information directly by comparing correct and faulty results using Differential Fault Analysis;
- Privilege escalation by bypassing a security check;
- Data dump when an I/O routine is affected leading to extraction of code or sensitive data.
With our tests a customer learns about the robustness of their product and how hard it is for an attacker to take control.
In a white-box context the weaknesses of a product are identified by a code review, but may need verification by testing. In a black-box context the code may not be available, and tests would be needed to find issues without prior knowledge. Through this test, a customer gets certainty about exploitability of their product. Absence of exploitable issues provides assurance. Software exploits are relatively cheap, and remotely executable (and thus scalable). That makes this type of testing very relevant.
We evaluate software security by:
We exhaust communication variations to identify unspecified behavior;
- Model Based Testing
We use malicious test cases to explore corner cases in device communication and expose confidential information. These tests are based on product security requirements, or on tester assumptions on the implementation of the product;
- Reverse Engineering
We use binary code dumps and analyze this to reconstruct the software structure, and identify weaknesses;
We use identified weaknesses (through code review or reverse engineering) to demonstrate that these can be used to attack the product.