Question

In the context of formal verification of neural networks, what does reachability analysis specifically calculate to ensure system safety?

Accepted Answer

In the formal verification of neural networks, reachability analysis calculates the set of all possible output values that a network can produce given a defined set of possible input values. This process establishes the forward mapping from an input region, known as the input constraint, through every hidden layer of the network until it reaches the final output layer. The primary goal is to determine if this calculated set of reachable outputs overlaps with a region defined as unsafe. An unsafe region represents system states that must be avoided, such as a self-driving car steering off the road or a controller sending a dangerous command. To calculate these sets, researchers typically use abstract interpretation, which involves over-approximating the reachable sets using geometric shapes like boxes, zonotopes, or polytopes. These shapes are easier to compute than the exact output set, which is often non-convex and mathematically intractable due to the non-linear activation functions in the network, such as ReLU. If the entire over-approximated reachable set is contained within the safe region, the system is mathematically guaranteed to be safe for all inputs within the specified range. If the reachable set intersects with the unsafe region, the analysis indicates a potential safety violation, meaning either the network is unsafe or the over-approximation was too loose to prove safety.

Home → All Courses → Engineering and Technology Courses → AI Safety and Responsible AI Engineering → Flashcard

In the context of formal verification of neural networks, what does reachability analysis specifically calculate to ensure system safety?