These updates include major improvements but did not yet advance the final stage of the pipeline, flow analysis, which I teased in the previous post. That’s because our work on flow analysis so far has been groundwork: figuring out how to adapt static-analysis techniques to expl3 before moving on to implementation.
In this post, I outline the flow graph structure currently used to represent expl3 code and describe our adaptation of the reaching definitions algorithm for dynamically scoped languages like expl3.
Chunks, edges, and flow graphs
In the previous post, we introduced segments as the units of top-level code in expl3 parts and nested code in T- and F-branches of conditionals, function definitions, and others. Each segment contains zero or more statements found during semantic analysis.
Flow analysis links statements using directed edges to form a flow
graph. It also partitions each segment into chunks: sequences of recognized
statements separated by OTHER_TOKENS_COMPLEX statements, which contain
arbitrary TeX code that may have side effects.
Static edges and confidence
Some edges arise directly from the static structure of an expl3 program. These are known without extra analysis and are called static edges.
Following statements and chunks, conditionals
Within a chunk, statements typically follow one another deterministically. Explcheck does not record these next-statement edges explicitly, but the analysis implicitly assumes them to be present.
The most common explicitly recorded static edge is NEXT_CHUNK, which
represents moving to the following chunk. We add NEXT_CHUNK edges as follows:
- Connect the last statement of each chunk in an expl3 part to the first statement of the first chunk in the next expl3 part.
- Connect the last statement of each chunk to the first statement of the next chunk in the same segment.
Another common static edge is NEXT_FILE, which represents inputting a different
file from the current file group:
- Connect the last statement of every file to the first statement of every other file.
Other common static edges are TF_BRANCH and TF_BRANCH_RETURN:
- For each conditional function, add a
TF_BRANCHedge to the first statements of the first chunks in its T- and F-branches. - For each T- and F-branch, add a
TF_BRANCH_RETURNedge from the last statement of its last chunk to the statement following its conditional function.
Confidence
All edges carry varying degrees of confidence: statements do not always follow one another deterministically, TeX code between chunks can alter the execution state in arbitrary ways, and transitions between files can be speculative.
For implicit next-statement edges, the confidence is usually DEFINITELY. It is
weakened to MAYBE when the current statement may be a call to a user-defined
function. For confirmed calls, the edge is removed entirely, because control flow
enters the function and returns later rather than advancing directly to the
following statement. We will discuss how function calls are represented and
resolved in the following section.
For NEXT_CHUNK and NEXT_FILE edges, the confidence is usually MAYBE; it
becomes DEFINITELY only for NEXT_CHUNK edges between immediately adjacent
expl3 parts.
For TF_BRANCH and TF_BRANCH_RETURN edges, the confidence is always
DEFINITELY, because exactly one branch will execute for any given call and
control will always return from it.
Weakening confidence
Although TF_BRANCH and TF_BRANCH_RETURN edges are individually marked as
DEFINITELY, the combination of multiple incoming or outgoing edges can still
introduce ambiguity about which path was actually taken: if both T- and
F-branches are present, only one will execute, and if a conditional function is
called from multiple places, the origin of control flow becomes statically
ambiguous within the function body.
To address this, later analyses typically weaken confidence based on edge multiplicity.
For forward analyses such as reaching definitions, edges whose endpoint has
in-degree greater than one are weakened to MAYBE. For example, this ensures
that function definitions coming from only one of the T- or F-branches are
appropriately weakened before control returns to the caller, indicating that the
definition may not exist depending on which branch was taken.
For backward analyses such as live variable analysis, edges whose
starting point has out-degree greater than one are weakened to MAYBE. For
example, this ensures that variable liveness is weakened before entering a
function with several possible definitions, reflecting the fact that only some
of those definitions may use the variable.
We may also skip the weakening if the definitions reach both the T- and
F-branches, or if the variable is used in all possible definitions of the
called function, even when the in- or out-degree is greater than one.
Dynamic edges and reaching definitions
Other edges depend on runtime behavior and dynamic scoping. These require estimation and are called dynamic edges.
Function definitions and calls
A key example is function calls and returns: FUNCTION_CALL and
FUNCTION_CALL_RETURN. Determining them requires knowing which function
definitions may reach each call.
If a function name has only one definition, we might assume that definition is used. But that definition may not reach the call. And often, multiple definitions share the same name, so we must determine which ones are used.
We start by building only the static edges. Then, we run the reaching definitions algorithm. Afterwards, we insert the dynamic edges:
- For each function call, add a
FUNCTION_CALLedge to the first statement of the first chunk in every definition that reaches it. - For each definition, add a
FUNCTION_CALL_RETURNedge from the last statement of its last chunk to the statement following any call it can reach.
For FUNCTION_CALL and FUNCTION_CALL_RETURN edges, the confidence is the lower
of the minimum confidence along the path from the definition to the call, and the
confidence of the definition statement itself.
Dynamic function definitions
The reaching definitions algorithm does not capture definitions created dynamically inside function calls. Consider the following example:
\cs_new:Nn
\example_bar:
{ bar }
\cs_new:Nn
\example_foo:
{
\cs_set:Nn
\example_bar:
{ baz }
}
\example_foo:
\example_bar:
The final call to \example_bar: should use the redefinition inside
\example_foo: (expands to baz), not the original definition on the first
three lines (expands to bar). A naive pass would get this wrong.
To handle cases like this, we iterate:
- Add dynamic edges based on the current reaching definitions.
- Run reaching definitions again.
- Replace the old dynamic edges with the new ones.
- Repeat until nothing changes.
Intuitively, the process converges because each round finishes some calls (their reaching definitions no longer depend on other unfinished calls), and the set of unfinished calls strictly decreases. When no edges change, we have reached a fixed point.
In the above example, both calls start out as unfinished. After the first
iteration, the call to \example_foo: finishes, but the call to
\example_bar: remains unfinished. After the second iteration, the
\example_bar: call also finishes because its reaching definition
shifts to the redefinition inside \example_foo:. No unfinished calls remain,
no edges change, and the algorithm stabilizes.
Limitations
While the above algorithm works for straightforward expl3 programs, it cannot detect function definitions or calls that arise only through dynamic expansion and are not obvious from the surface text of the program. This limitation is inherent to any static analyzer of a highly dynamic language such as TeX.
While some simple cases may become supported in the future, any attempt to model complex dynamic behavior is constrained both by our incomplete knowledge of the execution environment and by the requirement that the analysis must always terminate in a predictable number of steps.
Get involved!
Your feedback is invaluable. Feel free to contribute or share your thoughts by visiting the project repository. More improvements are on the way as explcheck continues to grow.
]]>
