An open and extensible framework for building observability toolkits and instruments that emit observations over a circuit of contexts (sources) and conduits (nodes) via currents (flows). The Substrates project is the reference implementation for the Humainary Substrates API, an API employed across all observability software libraries under the scope of the Humainary initiative.
- From Many to One Observability ModelToday the approach to observability at various stages within the data pipeline, from Application to Console, has been to create different models in terms of concepts, structure, features, and values. But what if the model employed were the same across all stages in a data pipeline?
- Multi-Event-Casting with InletsThe typical flow of execution for an observability Instrument is for instrumentation within the application code to make a single call to a method defined in the Instrument interface. But there are cases where a single call into an Instrument interface causes the dispatching of multiple events.
- Scaling Observability ResourcesA significant challenge in building observability agents or heavily instrumented applications or frameworks is in scaling, both up and down, the resources consumed. There is a trade-off here in terms of time and cost.
- Composing Instrument CircuitsIn this post, we walk through one of the showcases in the project’s code repository that demonstrates how the complexity of hooking up components in a Circuit is greatly simplified.
- Designing for ExtensibilityThe interfaces defined within Substrates API are designed with extensibility and evolution in mind, both from a client library and provider implementation perspective.
- Pipelines and ProjectionsAn objective of the Substrates API is that it should be possible for developers to be location independent in that code can execute without change within the local process itself or a remote server that is being served the data via a Relay.
- Simplifying Instrument SynthesisThree overriding principles are applied in the design of the Substrates API – consistency (standardizing), conciseness (simplifying), and correctness (sequencing).
- API Design – A Moving TargetGood design takes time, over many iterations (of converging and diverging design paths), in developing, discovering, discussing, discounting, and occasionally destroying.
- Data Pipeline SegmentationThe stages within a local data pipeline managed by the Substrates runtime are detailed, from a client call to an Instrument interface method to dispatching an Event to an Outlet registered by a Subscriber.
- Playing with PipelinesA walk through of one of the Substrates showcase examples hosted on GitHub, demonstrating two of the most critical aspects of pipelines – the production and consumption of captured data and published events.
- Pipelines ReimaginedUsing Substrates, the fusion of multiple streams of data from multiple sources, an essential process of any monitoring and management solution, can be done in-process and in real-time.
- Circuits, not Pipelines!The first official release of the Substrates API is nearing completion. With that, it is time to explore one of the toolkit’s most significant differences compared to other approaches, such as OpenTelemetry.