Kong AI Gateway Demo: Streamlining Multi-Provider LLM Usage, Security & Monitoring
Try Kong AI Gateway: https://bit.ly/3xhKw49
Timestamps:
0:00 Introduction
0:25 Recap of Kong 3.6 features
1:35 Unified inference format across multiple providers (OpenAI, Anthropic, etc.)
3:51 Plugins for enhancing and restricting usage of language models
6:02 Detailed contextual usage statistics
7:02 Augmenting existing APIs with LLM features
10:26 Response streaming and token streaming support
11:40 OpenAI SDK support
12:52 Token-based rate limiting (Enterprise feature)
15:12 Azure Content Safety introspection (Enterprise feature)
16:54 Upcoming features and ideas
This demo showcases the latest features and enhancements in Kong's AI Gateway, as of the June 2024 release. This demo covers:
Unified inference format across multiple providers (OpenAI, Anthropic, etc.) for easy compatibility and developer experience.
Plugins for enhancing and restricting usage of language models: Decorator for appending or prepending context to chats, Templator for creating APIs with static parameters for controlled usage, and Prompt Guardian for allowing or blocking prompts based on regex patterns.
Detailed contextual usage statistics, including token counts, latency information, and logging of input/output chats.
Augmenting existing APIs with LLM features using request and response transformers.
Response streaming and token streaming support for various providers.
OpenAI SDK support with transparent model selection delegation to the Gateway.
Token-based rate limiting for restricting access to specific models or providers based on token limits.
Azure Content Safety introspection for moderation and safety checks across providers.
Upcoming features and ideas, such as cooldown periods for users breaching safety limits and attaching specific roles to consumers for granular access control.
Watch video Kong AI Gateway Demo: Streamlining Multi-Provider LLM Usage, Security & Monitoring online without registration, duration hours minute second in high quality. This video was added by user Kong 14 June 2024, don't forget to share it with your friends and acquaintances, it has been viewed on our site 203 once and liked it 2 people.