Skip to content

Add a DataFlowAnalyzer soundness property test with a Yul program generator - #17142

Open
msooseth wants to merge 22 commits into
developfrom
only-knowledge-fuzzer
Open

msooseth wants to merge 22 commits into
developfrom
only-knowledge-fuzzer

Conversation

@msooseth

Copy link
Copy Markdown
Contributor

Adds a FuzzTest property test, DataFlowAnalyzerProperty.AnalyzerFactsHoldAtRuntime, that checks the facts the DataFlowAnalyzer / KnowledgeBase claim are true at runtime.

It generates a Yul program, runs a random optimizer step sequence on it, turns every claimed fact (known storage/memory contents, variable values, constant offsets, disequalities) into a runtime check, and executes the result in the Yul interpreter. Checking the claims directly catches unsound facts even when no step exploits them observably. It would have caught the Knowledge Base Stale Bookkeeping bug fixed in 0.8.38.

Supporting changes

  • test/tools/yulFactAuditor: the instrumentation library.
  • yulopti --audit prints the instrumented code; yulrun --calldata <hex> runs it on a failing input.
  • *Solo variants of CSE, EqualStoreEliminator, Rematerialiser, LiteralRematerialiser and ExpressionSimplifier in yulOptimizerTests, with regression tests for that bug.
  • Missing <vector> includes in MultiUseYulFunctionCollector.h and SSACFGTypes.h.

PS: Two upcoming PRs build on this Yul program generator: one checks that optimizer steps leave the code in the shape they promise, the other checks what SideEffectsCollector and ControlFlowSideEffectsCollector claim against execution.

@blishko blishko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still have to look at the program generator, but this looks interesting!

Comment thread test/tools/yulFactAuditor/FactAuditor.h Resolved

@blishko blishko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am slowly getting through this PR.
I have looked at YulProgramGenerator and did not notice anything suspicious.
I wish the code would be simpler, but not sure if that is possible.

In any case, it seems to me a few things could be separated out of this PR, like the extra yulOptimizerTests and the missing headers.
Keeping the PR focused is better for reviewing.

Comment on lines +33 to +42
libyul/DataFlowAnalyzerPropertyTest.cpp
libyul/DominatorTreePropertyTest.cpp
libyul/OptimizerStepSequence.cpp
libyul/OptimizerStepSequence.h
libyul/StackLimitEvaderPropertyTest.cpp
libyul/StackShufflerPropertyTest.cpp
libyul/TarjanSCCPropertyTest.cpp
libyul/TLSFFreeListPropertyTest.cpp
libyul/YulProgramGenerator.cpp
libyul/YulProgramGenerator.h

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to organize these tests a bit more. Not necessarily here, more likely as a follow-up.

);
static_assert(
smallValueBound > 1,
"The calldata of all ones must be made of small values, otherwise it is not what the programs expect to read."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't quite parse this statement. "calldata of all ones" What does that mean?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was about the calldatawords that are fixed 0,0,0,0 or 1,1,1,1 or rnd(0,smallValueBound-1),rnd(0,smallValueBound-1),rnd(0,smallValueBound-1),rnd(0,smallValueBound-1).

It's now less obscure I hope...


/// @returns a number below @a _count.
/// Draw into named variables: the evaluation order of function arguments is unspecified.
std::size_t pick(std::size_t const _count)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
std::size_t pick(std::size_t const _count)
std::size_t pick(std::uint8_t const _count)

maybe?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what we'd gain with this? I see some potential issues, but I don't see the gain. Maybe I'm blind. Can you explain?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember anymore why I thought it would be a good idea.
Please ignore.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you had the right idea. This is only meaningful for _count <= 256 if the input is a single byte. I'd add an assert instead of changing the signature though:

yulAssert(0 < _count && _count <= 256, "pick draws from a single byte");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, fair, and the return type can be uint8_t. Fixing.

@msooseth
msooseth force-pushed the only-knowledge-fuzzer branch from 79b1fc5 to a26a03f Compare November 16, 2026 16:04
@msooseth
msooseth force-pushed the only-knowledge-fuzzer branch from a26a03f to b0dad71 Compare November 16, 2026 16:29
@msooseth

Copy link
Copy Markdown
Contributor Author

OK, I removed the ProgramProfile and moved the solo optimizer test steps to Draft PR #17232 as requested by @clonker

I also made a little tweak to the generation, so on 12 cores we hit the DataFlowAnalyzer bug in median time of 5s when running with 12 cores.

@clonker clonker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some smaller nits, I think this is in good shape!

{
Operation operation;
char const* name;
/// How often the operation is drawn, relative to the others

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that describing a discrete distribution where the probability to draw it is the weight over the sum of all weights?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. But not great. So I am now adjusting it to be precentages, and adding a static_assert that they must add up to 100. Hopefully more readable? I like relative distrib, but I maybe these percentages are easier on the eye? Had its downside, but anyway.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean it was just a question out of curiosity :) I am fine keeping it like it is as well. Perhaps with a tiny comment that this indeed what it does.

Comment on lines +90 to +93
static_assert(
std::size(operationInfos) == numOperations,
"Every operation needs a row in operationInfos."
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to derive numOperations directly from operationInfos? then this assert could go. although it also doesn't really hurt, just one more thing to keep in sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd need reflections? C++ has no way to count an enum's enumerators, so numOperations has to name the last one. Counting them automatically needs C++26 reflection, right? I am not sure how else we could do it. I think I'm not a C++ magician enough to do this right without reflection.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe i'm missing something but how does numOperations = std::size(operationInfos) need reflection? :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The catch is that GeneratorStatistics needs the count in the header, so the table would have to move there as inline constexpr. I am completely fine to leave as is if you'd rather keep the table in the .cpp.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, true. I did it now :)


/// @returns a number below @a _count.
/// Draw into named variables: the evaluation order of function arguments is unspecified.
std::size_t pick(std::size_t const _count)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you had the right idea. This is only meaningful for _count <= 256 if the input is a single byte. I'd add an assert instead of changing the signature though:

yulAssert(0 < _count && _count <= 256, "pick draws from a single byte");

}

/// Chains of calls multiply the execution time. A function can therefore only call functions that
/// call none

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this contradicts itself with maxCallDepth being 2 right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this remained from an old version of this system. It's been fixed now to read:

A function makes at most maxCallsPerFunction calls, and no chain of calls is longer than maxCallDepth

/// so what is known about them survives the stores through locations.
/// 2**256 - 3 .. a location minus 1 to 3, wrapped around below zero
///
/// Limitations:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the limitations list also say that the checked facts are those of a standalone analysis of the final code, not of the analyses inside the steps?
I think that is worth adding since the stale bookkeeping bug class depends on the order of queries.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true of course. It's actually kinda annoying. We'd need a property analysis for the facts while the data flow analyzer is running... that's a different beast. Anyway, yes, added it to the DataFlowAnalyzerPropertyTest.cpp's Limitations part.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants