Introduction to Memoria

Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.

– Rob Pike in “Notes on Programming in C”, 1989.

Memoria is …

  • a “full-stack” data-centric framework for latency-sensitive class of computations covering data- and knowledge bases, logical reasoners, constraint solving for both exact and approximate (including Any-Time) types of inference.
  • based in a complexity-minded design philosophy, derived from theorems of Algorithmic Information Theory (AIT).
  • focusing on data structures across all levels of design as a main “center of complexity”.
  • spanning not only classical multicore CPU-centric software-only part of the design space, but also relevant hardware design space via co-design methodologies: RISC-V accelerators and direct hardware implementation of essential algorithms, data structures and communication protocols.

Memoria Framework has the following components:

  1. Hermes - arbitrarily structured object graphs allocated in relocatable contiguous memory segments, suitable for memory mapping and inter-process communication, with focus on data storage purposes. Hermes objects and containers are string-externalizable and may look like “Json with types”. GRPC-style services and related tooling (IDL) is provided (HRPC).
  2. Extensible and customizable set of Data Containers, internally based on B+Trees crafted from reusable building blocks by the metaprogramming framework. The spectrum of possible containers is from plain dynamic arrays and sets, via row-wise/column-wise tables, multitude of graphs, to compressed spatial indexes and beyond. Everything that maps well to B+Trees can be a first-class citizen of data containers framework. Containers and Hermes data objects are deeply integrated with each other.
  3. Pluggable Storage Engines based on Copy-on-Write principles. Storage is completely separated from containers via simple but efficient contracts. Out of the box, OLTP-optimized and HTAP-optimized storage, as well as In-Memory storage options, are provided, supporting advanced features like serializable transactions and Git-like branching.
  4. DSL execution engine (DSL Engine). Lightweight embeddable VM with Hermes-backed code model (classes, byte-code, resources) natively supporting Hermes data types. Direct Interpreter and AOT compilation to optimized C++.
  5. Acceleration architecture.
  6. Runtime Environments. Single thread per CPU core, non-migrating fibers, high-performance IO on top of io-uring and hardware accelerators.
  7. Development Automation tools. Clang-based build tools to extract metadata directly from C++ sources and generate boilerplate code.

These components are organized into the following logical architecture:

The purpose of the project is to integrate all aspects and components described above into a single vertical framework, starting from bare silicon up to networking and human-level interfaces. The framework may eventually grow up into a fully-featured metaprogramming platform.

Philosophy →