%% date:: [[2022-04-29]] %% rel:: [[Software Engineering|Software Engineering MOC]] rel:: [[Databases MOC]] # Have You Tried Rubbing A Database On It 2022 ## Reference - [videos](https://www.hytradboi.com/) - [chat](https://chat.hytradboi.com/) - [conference home page](https://www.hytradboi.com) ## Incremental stream processing with DDlog by [Leonid Ryzhyk](http://ryzhyk.net/) - [Backup video](https://archive.org/details/hytradboi2022/hytradboi-2022+-+Leonid+Ryzhyk.mp4) - [More info](https://github.com/vmware/differential-datalog) > DDlog is a programming language for _incremental computation_. It is well suited for writing programs that continuously update their output in response to input changes. With DDlog, the programmer does not need to worry about writing incremental algorithms. Instead they specify the desired input-output mapping in a declarative manner, using a dialect of Datalog. The DDlog compiler then synthesizes an efficient incremental implementation. DDlog is based on [Frank McSherry's](https://github.com/frankmcsherry/) excellent [differential dataflow](https://github.com/frankmcsherry/differential-dataflow) library. ## Ultorg: a user interface for relational databases by [Eirik Bakke](https://people.csail.mit.edu/ebakke/index.html) - [Backup video](https://archive.org/details/hytradboi2022/HYTRADBOI+Ultorg+-+Eirik+Bakke.mp4) - [More info](https://www.ultorg.com/) Lets you quickly craft CRUD views on your database, as easy to use as a spreadsheet. Impressive demo. > AirTable started with CRUD and creating data from scratch within the interface. Ultorg started from the other end--assuming you already have data in some external database, and now it's slowly working its way towards supporting editing scenarios. ## Baking in time at the bottom of the database by [Jeremy Taylor](https://github.com/refset) - [Backup video](https://archive.org/details/hytradboi-refset-bitatbotd-v-2-jeremy-taylor) - [More info](https://xtdb.com/) > [XTDB](https://xtdb.com/) is a general purpose database with graph-oriented bitemporal indexes. Datalog, SQL & EQL queries are supported, and Java, HTTP & Clojure APIs are provided. > > XTDB follows an _unbundled_ architectural approach, which means that it is assembled from decoupled components through the use of an immutable log and document store at the core of its design. A range of storage options are available for embedded usage and cloud native scaling. ![[xtdb-node-1.svg]] - [what is bitemporality](https://github.com/xtdb/xtdb/blob/master/docs/concepts/modules/ROOT/pages/bitemporality.adoc)? - `transaction-time` - used for audit, event sourcing, relates to `system time`. Time baked into the engine (a la [[Datomic]]) - SQL:2011 spec has support for querying system time `SELECT... AS OF SYSTEM TIME` - But, this time doesn't correlate to all domains where the system time isn't the source-of-truth. Think streaming applications, batch loading data from another upstream system, etc. - Bitemporal systems introduction a second time axis, called `valid time`. - `valid time` defaults to `system time`, but can be provided to the transaction on update. - `valid time` has a fundamental difference to `transaction time`, in that you can insert into the past. ## UIs are streaming DAGs by [Dustin Getz](https://www.dustingetz.com/) (hyperfiddle) - [Backup video](https://archive.org/details/hytradboi2022/UIs+are+streaming+DAGs+-+Dustin+Getz+-+Dustin+Getz.mp4) - [More info](https://www.hyperfiddle.net/) key idea: Hyperfiddle ASTs compile into a DAG which the macro will slice and partition over client/server system, shaking out dependencies to produce optimal network effects. See also their [RCF](https://github.com/hyperfiddle/rcf) test library for [[Clojure]] ^20154a ## Debugging by querying a database of all program state by [Kyle Huey](https://pernos.co/) - [Backup video](https://archive.org/details/hytradboi2022/HYTRADBOI+-+Kyle+Huey.mp4) - [More info](https://pernos.co/) **Record** failures anywhere: locally, QA, CI, or in production - Record your x86 Linux program using [rr](https://rr-project.org/). - [Submit](https://github.com/Pernosco/pernosco-submit) those recordings to Pernosco for processing and Pernosco sends you a link to access the debugger that you can use from anywhere. - Once you've captured a recording (e.g. of an intermittent test failure) you don't need to reproduce the bug again. **Rapidly debug** recordings using the [revolutionary](https://pernos.co/about/overview) Pernosco omniscient debugger - gives you instant access to the full details of any program state at any point in time. - search - visualizes control flow and data flow history so you can quickly track effects back to causes. Implementation - delta-encoding to make state storage tractable - re-uses instruction stream for encoding - they are effectively re-executing instruction fragments to step through debugger. **Security Concerns?** ## Simple Graph: [[SQLite]] as (probably) the only graph database you'll ever need by Denis Papathanasiou - [Backup video](https://archive.org/details/hytradboi2022/simple-graph-10min+-+denis.papathanasiou%40banrai.com.mp4) - [More info](https://github.com/dpapathanasiou/simple-graph) - two tables - simple, indexed, directional relation between nodes ```sql CREATE TABLE IF NOT EXISTS nodes ( body TEXT, id TEXT GENERATED ALWAYS AS (json_extract(body, '$.id')) VIRTUAL NOT NULL UNIQUE ); CREATE INDEX IF NOT EXISTS id_idx ON nodes(id); CREATE TABLE IF NOT EXISTS edges ( source TEXT, target TEXT, properties TEXT, UNIQUE(source, target, properties) ON CONFLICT REPLACE, FOREIGN KEY(source) REFERENCES nodes(id), FOREIGN KEY(target) REFERENCES nodes(id) ); CREATE INDEX IF NOT EXISTS source_idx ON edges(source); CREATE INDEX IF NOT EXISTS target_idx ON edges(target); ``` ## Asynchronous replication without bloat by [Paul Khuong](https://pvk.ca/) - [Backup video](https://archive.org/details/hytradboi2022/pkhuong-async+repl+without+bloat+-+Paul+Khuong.mp4) - [More info](https://github.com/backtrace-labs/verneuil/blob/main/doc/REPLICATION_BUFFER.md) - [[SQLite#Verneuil]] key idea: "replication buffer" - [[SQLite#^ae3b74]] ## [[Datasette]]: a big bag of tricks for solving interesting problems using [[SQLite]] by [Simon Willison](https://simonwillison.net/) - [Backup video](https://archive.org/details/hytradboi2022/HYTRADBOI-Simon-Willison+-+Simon+Willison+-+faster.mp4) - [More info](https://datasette.io/) - "SQL injection vulnerability as a feature" - but okay because - Datasette opens in read-only mode - queries timeout at `1s` - shows off [[SQLite#^1d51eb|sqlite-utils]] - [[Datasette]] supports publishing directly to [[Google Cloud Run]] ## Cambria: schema translations in distributed systems using bidirectional lenses by [Geoffrey Litt](https://geoffreylitt.com/) - [Backup video](https://archive.org/details/hytradboi2022/Cambria+at+HYTRADBOI+-+Geoffrey+Litt.mp4) - [Cambria](x-devonthink-item://1DCD5CAA-928B-409D-BBC7-513C71330A30) is an [[Ink and Switch]] project. - Cambria schema changes for local-first data stores, [[CRDT]]s - yet-another layer of indirection - graph of bidirectional schemas, declarative transformations to translate date between representations - each client reads and writes a document in its native schema - data translations at runtime - graph of lenses translates data between clients - Version-Tagged Edit Log ^9cdbda - documents materialized by playing log - only do translation at read time - ![[Screen Shot 2022-05-05 at 3.37.09 PM.png]] - Papers - [Edit Lenses](x-devonthink-item://6FD950ED-36C6-4D83-9825-D09FD0F04483) - [Combinations for Bi-Directional Tree Transformations](x-devonthink-item://3AB9F1A6-6736-4968-B807-9F02FF036164) - Ideas - Tag data with versions - put data translation and validation in an isolated layer - persist documents as [[#^9cdbda|Version Tagged Edit Log]] ## Viewing collaborative editing through a databases lens by [Martin Kleppmann](https://martin.kleppmann.com/) - [More info](https://github.com/automerge/automerge-perf/blob/master/columnar/README.md) - using delta and run length encoding techniques used by columnar-store databases, can efficiently compress [[CRDT]] edit histories. [[CRDT]]s, automerge - This relates to [[Ink and Switch#^745fc7]]