Compresh — Episodic Memory for LLM APIs

Developers looking to reduce costs on large language model API calls can route their requests through Pith, a proxy service that optimizes prompts in real-time before they reach OpenAI, Anthropic, or OpenRouter. The tool promises token savings of up to 50% while maintaining output quality, requiring only two lines of code to implement—users simply swap their API key and base URL. Pith employs a two-tier optimization system that processes every prompt automatically. The first tier uses deterministic pattern matching that runs instantly on each request without involving machine learning models. The second tier leverages the LLMLingua-2 model to identify which tokens can be safely removed without altering meaning. This layer runs in the background and caches results for identical future prompts. If neither optimization layer can reduce tokens without risk, Pith forwards the original prompt unchanged, ensuring zero quality degradation. The service integrates natively with existing development workflows across multiple languages and frameworks. Supported environments include the Python OpenAI SDK, Anthropic SDK, Node.js and TypeScript implementations, LangChain, LlamaIndex, VS Code extensions like Continue, and AI coding assistants including Cursor and Windsurf. Developers can also use Pith through REST APIs, cURL, or any HTTP client by simply changing the target URL. Beyond cost optimization, Pith includes built-in prompt injection detection at no additional charge. Every request passes through a three-layer machine learning defense system that scans for attacks automatically, requiring no setup or SDK modifications. The service was benchmarked against 100,000 real conversations from the LMSYS-Chat-1M dataset, which represents actual user interactions rather than synthetic test data. During this testing, Pith saved 10.7 million tokens. Repeated prompts benefit from caching, achieving up to 100% savings on cache hits, while short prompts pass through unmodified to avoid unnecessary processing. Pith operates on a freemium model. The free tier includes $30 in credits upon signup, allowing developers to test token savings on their own prompts without providing credit card information. An enterprise tier exists for organizations operating at scale, though specific pricing details for that tier are not detailed in the public-facing content. The optimization database continuously learns new patterns across different industries and use cases, meaning savings can increase over time as the system processes more data.