Kong AI Gateway Demo: Streamlining Multi-Provider LLM Usage, Security & Monitoring
Try Kong AI Gateway: https://bit.ly/3xhKw49
Timestamps:
0:00 Introduction
0:25 Recap of Kong 3.6 features
1:35 Unified inference format across multiple providers (OpenAI, Anthropic, etc.)
3:51 Plugins for enhancing and restricting usage of language models
6:02 Detailed contextual usage statistics
7:02 Augmenting existing APIs with LLM features
10:26 Response streaming and token streaming support
11:40 OpenAI SDK support
12:52 Token-based rate limiting (Enterprise feature)
15:12 Azure Content Safety introspection (Enterprise feature)
16:54 Upcoming features and ideas
This demo showcases the latest features and enhancements in Kong's AI Gateway, as of the June 2024 release. This demo covers:
Unified inference format across multiple providers (OpenAI, Anthropic, etc.) for easy compatibility and developer experience.
Plugins for enhancing and restricting usage of language models: Decorator for appending or prepending context to chats, Templator for creating APIs with static parameters for controlled usage, and Prompt Guardian for allowing or blocking prompts based on regex patterns.
Detailed contextual usage statistics, including token counts, latency information, and logging of input/output chats.
Augmenting existing APIs with LLM features using request and response transformers.
Response streaming and token streaming support for various providers.
OpenAI SDK support with transparent model selection delegation to the Gateway.
Token-based rate limiting for restricting access to specific models or providers based on token limits.
Azure Content Safety introspection for moderation and safety checks across providers.
Upcoming features and ideas, such as cooldown periods for users breaching safety limits and attaching specific roles to consumers for granular access control.
Смотрите видео Kong AI Gateway Demo: Streamlining Multi-Provider LLM Usage, Security & Monitoring онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь Kong 14 Июнь 2024, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 203 раз и оно понравилось 2 людям.