# Deterministic Simulation Testing ## Mindmap - The DST technique - Mindset - Think of tests at a higher level than low level value assertions, brittle, code-state specific - Define high-level targets for how you want the system to behave - Let a search algorithm explore the state space to validate those targets are met - Requirements - a concurrency model that is - single threaded (as soon as we involve the OS scheduler, determinism is lost) - async - all side effects abstracted and driven by a simulator - a search algorithm + high level assertions to guide the search - Benefits - When property violations are found, deterministic, reproducible - High degree of confidence that the properties are not violated, unlocks dev workflows with reduced cycle time - Costs - Entire system must be designed for DST - Sweat equity in the search algorithm - Approaches - Antithesis - Build the universe out of deterministic material - a deterministic hypervisor - Comprehensive, an accelerator to reach DST benefits - Not local, slow response times - only one company doing this, costs - Designed - Go through the exercise of building your system with - A concurrency model compatible with DST - Abstracted side effects - Possible if you control the entire system. - Difficult for libraries as integration tests will need to exercise third party code - Significant engineering investment and discipline ## Resources - [Datadog PoC Recap](https://docs.google.com/presentation/d/1cXm3qFk2R2GplHWo6S0SlPQpWFlmi8GbT5vY4PC_OKQ/edit?slide=id.g33d6445f4a1_0_49&pli=1#slide=id.g33d6445f4a1_0_49) (deck) - [technical notes](https://docs.google.com/document/d/1ZtKYSnupmR1o9qA-07widVDb6MLFeQoGj7URRawIImM/edit?tab=t.0) - [Datadog Antithesis PoC](https://datadoghq.atlassian.net/wiki/spaces/~109860613/pages/5163845907/Antithesis+PoC) (notes dump) - What is DST - [Let's All Write Good Software, BugBash 2025, Will Wilson](https://www.youtube.com/watch?v=eZ1mmqlq-mY&t=1s) (video) - [Testing Distributed Systems With Deterministic Simulation](https://www.youtube.com/watch?v=4fFDFbi3toc) Strange Loop 2013, Will Wilson - [Simulation Testing With Simulant, Clojure West 2013, Stuart Halloway](https://www.infoq.com/presentations/Simulation-Testing/) - [Let The Computer Do The Work](https://datadoghq.atlassian.net/wiki/spaces/VEC/pages/2126381424/Let+The+Computer+Do+the+Work) by [@blt](https://dd.enterprise.slack.com/team/U01NEAU8S72) - [How Amazon Tests Their Systems](https://cacm.acm.org/practice/systems-correctness-practices-at-amazon-web-services/) - ![[marc brooker dst quote.png]] ### What is Antithesis? - [Is Something Bugging You](https://antithesis.com/blog/is_something_bugging_you/)? (blog post Antithesis wrote coming out of Stealth) - [How is Antithesis Different?](https://antithesis.com/product/how_is_antithesis_different/) - [Building a Deterministic Hypervisor with bhyve, FreeBSD Dev Summit 2024, Alex Pshenichkin](https://www.youtube.com/watch?v=0E6GBg13P60) (video) ### Getting Started with Antithesis - [etcd test cluster example](https://antithesis.com/docs/tutorials/cluster-setup/) - [docker compose setup guide](https://antithesis.com/docs/getting_started/setup/) - [SDK](https://antithesis.com/docs/using_antithesis/sdk/) - driving workloads - can use any container entrypoint or - have a dedicated test driver container and use [Test Composer](https://antithesis.com/docs/test_templates/) - [basics](https://antithesis.com/docs/test_templates/first_test/) - [reference](https://antithesis.com/docs/test_templates/test_composer_reference/) - [an entrypoint](https://github.com/DataDog/dd-profiling-antithesis/blob/main/runner/resources/entrypoint.sh) that does `sleep infinity` inside antithesis and automatically runs the test composer tests itself is useful for local testing.