Caching roadmap

[zilto]

Here's a place to discuss the caching roadmap. Hamilton 1.79.0 introduces the core caching feature, but we have a lot more planned.

How this thread works

• comment with feedback on listed items or suggest new ideas
• ideas will be added to this first comment to catalog everything
• this is not a commitment to implementing everything listed.

Roadmap

1. async support: work with the AsyncDriver
2. integrate with remote execution: allow to selectively execute nodes on Ray, Modal, Skypilot, Runhouse, etc.
3. distributed execution and caching: current backends are designed for a machine. Make it work across multiple workers and nodes
4. in-memory stores: cache metadata and results in-memory for long-lasting Python sections (e.g., notebook, interpreter). Could have an option to "persist" the in-memory store by converting it to another store type.
5. more backends: support S3, Redis, etc.
6. cache eviction: automatically manage cache storage (expiration date, max n items, max storage size)
7. Hamilton UI integration: have a "result catalog" and see which artifact (keyed by data_version) is used in which execution
8. manual cache storage management: provide user-facing utilities to "delete everything associated with node X", "delete everything not in this DAG", etc.
9. store partitioning and eager loading (optimization): the backend could partition metadata by code_version. When creating the DAG, only load metadata for matching code_version. Associated results could be loaded in-memory or Redis for lightning fast access. This is relevant for production web services with: a set DAG and a large number of entries
10. DAG optimization and tuning (tangential to caching): Let's say you have a machine learning pipeline built with Hamilton (or a RAG pipeline). It involves data cleaning, feature engineering, feature selection, model training, evaluation. Build tooling to optimize the DAG as a whole with "dag hyperparameter". Caching can supercharge that and make it very efficient

[mitstake]

https://github.com/mitstake/darl

Hi I posted this elsewhere in the discussions but check out how my library darl does caching to get some ideas for how you can implement some of your ideas above. Darl already provides 2, 3, 4, 5, 6, 7, 10. (Surprisingly you might find number 8 can be the hardest to implement when dealing with persistent caching unless you implement an indexing system that can get rather complex). Implementing some of the caching ideas from darl can enable Hamilton to become a formidable incremental computing engine as well.

[mitstake]

In general the goal of the API UX is to make certain things easier (or harder) to accomplish and trading off what is required

To preemptively address @skrawcz comment above from the other thread: this might be another case where the design of how this is done in darl might not fit into the Hamilton philosophy, but will leave that to the Hamilton team to decide. The reason being that darl caching is based off of dependency aware code hashing of functions in the dag. This only works if the functions are perfectly deterministic, and there are no guardrails to ensure/enforce determinism. There is a mechanism to handle some non-determinism but it’s meant to be used sparingly and is not a substitute for careful deterministic definition of all code. This introduces an easy way to shoot yourself in the foot and introduce bad stale cached values. In my experience with a well trained user base it’s not hard to keep all your code deterministic, but for a wide audience and guaranteed accuracy tool this methodology might not fit the Hamilton philosophy. But again I’ll leave it here for the team to take a look and decide for yourselves. (I’m not even sure it’s worth leaning heavily into cross-process caching without expecting all code to be deterministic. But interested to hear your thoughts on this)