Changelog
🔄 Test Improvements and Infrastructure Updates
- 📊 Added display of pass rate by each configuration in test results
- 🛠️ Fixed tool result focus behavior when there are multiple tool calls in playground
- 🐛 Fixed test run detail sidebar not updating when switching between cells in test table
- 🐛 Fixed o1-preview and o1-mini which dont support reasoning effort level - Langtail is no longer showing it
- 🔀 Added support for changing thread message list order direction in the API (
order=desc)
🤖 New Gemini Models and UI Improvements
- ✨ Added new Google models:
- Gemini Flash 2
- Gemini Flash 2 Lite
- Gemini Pro 2 (Experimental)
- 🔄 Added New Flash 2 model and Sonnet 3.5 to OpenRouter models
- 🎛️ Replaced temperature setting with reasoning effort level for OpenAI reasoning models (o1/o3)
- 💾 Test run details now persist when switching between playground, tests and test run tabs
🎉 OpenAI o3-mini Launch and Performance Updates
- 🚀 Added support for OpenAI’s new o3-mini model
- More powerful and cost-effective compared to o1-mini
- Lower latency responses
- Full support for function calling, structured outputs, and developer messages
- Note: No vision capabilities, requires temperature 1.0
- 💲 Fixed request cost calculations in logs & metrics to account for cached prompt tokens
- 🗃️ Added caching support for Anthropic models
🔄 Anthropic Caching and Usage Metrics Improvements
- 🔄 Added caching support for Anthropic models
- Enable via UI on template messages
- 🐛 Improved token count and pricing calculations for Anthropic models and Gemini models
- 💰 Implemented cached token price calculations for OpenAI and Anthropic providers
🤖 Fixed bugs and added support for Deepseek-R1 model
- 🤖 Added OpenRouter Deepseek-R1 model
- 🔄 Fixed bug: OpenAI compatible providers (including OpenAI) didn’t return a proper error response message - only “Whoops! We couldn’t send your message. Let us know if this keeps happening.”
- 🔄 Fixed bug: Test inputs (rows) could not be deleted if some columns were hidden
🧪 Test Duplication and UI Improvements
- ✨ Added test duplication dialog to copy tests between prompts
- 🗑️ Removed model column from test runs list for cleaner UI
- 🖱️ Fixed scroll behavior on message removal in playground
- 📊 Improved test column actions and styling
- 🔧 Fixed various UI issues with prompt and test form lists
- 🛠️ Fixed tool calls styling in tests - support when LLM responds with both text and tool call
- 🐛 Fixed test runs failing with exceptions in rare cases
- 🔒 Fixed 404 issue after logging out and in as a different user
🤖 DeepSeek-V3 Support Added
- 🧠 Added support for DeepSeek-V3, the latest model from the DeepSeek team
- 📚 Pre-trained on nearly 15 trillion tokens
- ⌨️ Excels at instruction following and coding tasks
- 🏆 Outperforms other open-source models and rivals leading closed-source models
🧠 OpenAI o1 Models Now Available
- 🤖 Added support for OpenAI’s new o1 models
- 🎯 Features internal reasoning chains for complex problem-solving
- 🤔 Models demonstrate improved reasoning capabilities by thinking before answering
🔍 Gemini 2.0 Flash (Experimental) Integration
- 🆕 Added Gemini 2.0 Flash (Experimental) model
- 🎯 First model with precise object bounds detection in images
- 🖼️ Eliminates guesswork in image analysis tasks
- ✅ Extensively tested with impressive results
🔧 Improved Test UX and Bug Fixes
- 🚀 It’s much easier to switch between prompt editing (Playground) and Tests now using tab navigation
- 📊 New users now get limited free usage for Anthropic & Google without providing their own API keys
- 🧪 Added Gemini experimental model 1206
- 🐛 Fixed index & id in response for multiple tool calls from Anthropic
- 🐛 Fixed variables list rendering in prompt detail view
- 🐛 Fixed missing user message in assistant playground
- 🐛 Fixed copy-pasting prompts from playground to include new lines correctly
- 🐛 Fixed dev console error when viewing non-JSON test results
🎨 Test design updates and new onboarding experience
- 🎨 Redesigned test interface to match playground styling and UX
- 📚 Introduced new user onboarding flow for better getting started experience
- ✨ Magic prompt enhancer with diff view and accept functionality
- 🐞 Fixed assistant deployment snippet generation
- 🐞 Added support for video URLs without file extensions
🔗 Improved test response UX and other fixes
- 📏 You can now enable test response text wrapping to easily see longer responses
- 📝 Added support for markdown in the test response drawer
- 🔧 Fixed selection of content in the test response drawer
- 🎨 Fixed test variable icon color on theme switch without reload
- 🔧 Adjusted LLM assertion to allow shorter Yes/No descriptions
- 🔄 Fixed thread messages not saving with assistant API in Anthropic/Google
- 🐛 Fixed a rare bug where a second main test config could be created
🔗 Shareable Prompts and Assistants, and UX Enhancements
- 🔗 Added OG tags to shareable prompt assistants for better visibility and discoverability.
- 🐞 When copy-pasting multiple rows into Tests, the order is now fixed.
- ⌨️ Enhanced tab navigation in the messages drawer for better accessibility.
- 🔄 Implemented proper handling of multiple test runs.
- 📋 Formatted the test output preview and added a copy button for improved readability.
- 🐞 Resolved issues with Gemini tool calls in tests and Anthropic model behavior.
- 📝 Added undo/redo functionality to the test table for easier editing.
- 🤖 Introduced a prototype for LLM assertion generation.
- 🤖 Added support for the xAI grok-beta model.
- ✨ Implemented magic to generate descriptions and titles for assistants.
🤖 New Model Support, Magic buttons, Testing Improvements, and Onboarding
- 🤖 Added support for the Claude 3.5 Haiku model from Anthropic.
- ✨ Introduced several “magic” buttons throughout the app to improve your experience with LLMs. These include prompts for enhancing prompts, generating test cases, and more! Let us know which ones you find and how useful they are - it’s like an Easter egg hunt!
- 🐞 Fixed an issue where the test data grid couldn’t delete repeated test cases.
- ⚙️ Introduced a “none” option for tool choice.
- 🔍 Added a history viewer for better visibility into past events.
- 📱 Improved onboarding experience for mobile devices.
🐞 Post 1.0 Fixes and UI Improvements
- 🎨 Implemented various small UI fixes and improvements.
- 🔢 Fixed an issue with ordering items in the testing table when they have the same creation date.
- 📝 Improved test generation by passing test_inputs to the magic function for better test data.
- 📚 Added templates and documentation links to the sidebar for easier access.
🚀 Langtail 1.0: Brand New Testing Interface, Assistants, and More!
We’re thrilled to announce the release of Langtail 1.0, packed with exciting new features and improvements!
📊 Brand New Testing Interface
- 📝 At the core of Langtail 1.0 is our new spreadsheet-like interface for testing LLM applications, designed to feel natural for anyone used to working with Excel or Google Sheets.
🔄 Test Configurations
- 🎚️ Introducing test configurations, allowing you to easily create and compare tests across different models with just a few clicks.
🛠️ Tools with Hosted Code
- 💻 Your prompts can now call tools and have the code run directly within Langtail, making prototyping and testing much easier without the need for mocking responses.
- 🔒 Enjoy secure and fast code execution in a sandboxed environment powered by QuickJS.
🧠 Assistants: Memory Management for Prompts
- 🧠 Langtail 1.0 introduces Assistants – stateful entities that automatically manage memory and conversation history, reducing the need for additional code.
- 🔄 Assistants can be used across models, in tests, deployed as APIs, or integrated with tools.
🔒 Additional Features
- 🌞 Light Mode: Introduced a new Light mode for the Langtail app, providing a refreshing alternative to the default Dark mode.
- 🛡️ Protect your AI apps from prompt injection attacks and other security vulnerabilities with our built-in AI Firewall.
- 🔐 Execute LLM-generated code securely in a sandboxed container, thanks to our partnership with E2B.
- 📚 Explore a gallery of templates showcasing different ways Langtail can help your team.
- 🤖 Added support for Google Gemini models, including the ability to attach videos to prompts.
- 🏢 Self-hosting: Langtail can now be self-hosted entirely on your infrastructure (contact us for more information).
🛠️ Tool Calls in Assistant Playground and Gemini Support
- 🛠️ Added the ability to handle tool calls in the Assistant Playground, allowing you to provide responses for tool calls not handled by Langtail function handlers.
- 🤖 Introduced native support for Gemini models, leveraging the @google/generative-ai library under the hood, with support for tools, streaming, and various media formats as input.
📋 Other Improvements
- 🔍 Fixed an issue with opening logs with prompts in the playground.
- 💰 Set Gemini model prices.
🌐 Assistant Sharing Redesign, LLM Leaderboard, and Hosted Functions
🤖 Assistant Sharing Redesign
- 🌐 Redesigned the assistant sharing feature, allowing you to make your AI assistants publicly accessible and share them as standalone apps.
📊 LLM Leaderboard
- 📈 Introduced a new LLM leaderboard page to compare the performance of different language models.
🛠️ Hosted Code Support
- 🔌 Added support for Hosted Code, enabling you to test your tools right inside Langtail by directly calling external APIs.
🔒 Security and Stability Improvements
- 🔒 Ensured that only team members can edit shared assistants.
- 🤖 Added support for new OpenAI models: o1-mini and o1-preview.
- 🔧 Fixed issues with tool usage for Azure models and display names.
🔍 Shared Assistant Indexes
- 🔍 Implemented shared assistant indexes for improved discoverability and searchability.
📝 Text Assertion Enhancements and LLaMa 3.1 Support
- 📝 Grouped all text operation assertions under “Text” for better organization.
- 🔖 Added new text assertion options: not equals, does not contain, and does not icontain.
- 📝 Introduced a text editor for the “Text” assertion modal form, with support for variables.
- 🤖 Added support for new LLaMa 3.1 models from groq.
💰 Usage Limits, Test Improvements, and GPT-4o Mini Support
- 🤖 Added support for GPT-4o Mini, OpenAI’s cost-efficient model offering advanced text and multimodal performance
- 💰 Implemented a banner to notify free users when they exceed usage limits.
- 🐞 Fixed an issue with the test ID for the assertion popup in test results.
- 🎚️ Improved the layout styles for project logs.
🌐 Expanded Message Types and Provider Visibility
- 📨 Added support for other message types, tools, and images in the “messages” column, allowing you to simulate user and assistant messages, as well as mock tools and upload images for Vision testing.
- 🔓 Unlocked providers are now displayed at the top for better visibility.
🎨 Layout Redesign and Test Table Improvements
🔍 Enhanced Project-Level Visibility
We’ve redesigned the layout to provide better visibility and access to key project-level components:
- Playground
- Tests
- Deployments
- Assistants
🐞 Test Table Fixes and Enhancements
- 🎥 Fixed an issue with the test table width when the drawer is open.
- 🔘 Added the ability to close the cell edit drawer by pressing the ESC key.
- 🐛 Resolved issues with HTML pasting in the data table.
- 📋 Improved the display of relevant information in the test history.
🔒 Security Enhancements
- 🔒 Fixed issues related to system_fingerprint and model value for Anthropic models.
- 🔼 Updated to the latest SDK, resolving an OpenAI issue with images.
🤖 Introducing Anthropic’s Claude 3.5 Sonnet Model
- 🆕 Added support for Anthropic’s Claude 3.5 Sonnet model, offering faster and cheaper performance compared to Claude 3 Opus, with enhanced reasoning, knowledge, and coding abilities.
🐞 Test Table UX Enhancements and Bug Fixes
We’ve made several improvements to the test table user experience and addressed some pesky bugs:
- 🖼️ Text table UX improvements
- 🎯 Streamlined the handling of messages in the test table for a smoother workflow.
- 🔍 Enhanced the behavior of the edit drawer, ensuring a more intuitive experience.
- ⌨️ Improved keyboard navigation and selection within the test table.
- 🐛 Fixed issues related to arrow key navigation and drawer behavior.
🎯 Testing Enhancements: Keyboard-Friendly Table, Helper Variables, and Improved LLM Assertions
⌨️ Keyboard-Friendly Testing Table ⌨️
We’ve made our testing table more keyboard-friendly, allowing you to bulk delete rows and copy multiple values at once.
🧮 Helper Variables for Assertions 🧮
You can now add additional columns that act as helper variables for assertions (JavaScript, LLM assertions). Place the correct answer in these helper variables and use them in assertions for more efficient and accurate testing.
📚 Improved LLM Assertions 📚
Based on user feedback, we’ve improved LLM assertions. Now, the entire conversation history is included in the context, providing a more comprehensive view for better testing.
📨 Multiple Messages in the “Messages” Column 📨
You can now use multiple messages in the “messages” column. Currently, it’s possible to simulate user and assistant messages. We’re working on the ability to mock Tools and upload images for Vision testing.
Tests are a big focus for us, and we’re committed to continuously improving them. Stay tuned for more updates!
⚡ Major Upgrade to Langtail’s Testing Table Component and Upcoming Features
⚡ Langtail’s Testing Table Component Upgrade ⚡
We’re excited to announce significant updates to Langtail’s table component aimed at improving your test-writing experience:
- 🚀 Lightning-fast row addition and cell content deletion for efficient test setup
- ➡️ Tab navigation between cells for seamless navigation
- 🗑️ Bulk row deletion for quick test cleanup
- 🎯 Selective test case execution for focused testing
- 🎁 Test result display with LLM and assertion outcomes for comprehensive insights
🔜 Upcoming Feature: LLM as a Judge Assertion 🔜
Stay tuned! We’re in the process of developing an “LLM as a judge assertion” feature, set to take our testing capabilities to the next level. More updates coming your way soon!
New Dashboard View and Launch of Langtail SDK 🚀
📊 New Dashboard View 📊
We’re thrilled to introduce our new dashboard view! Now serving as the main hub for each project, the dashboard provides a snapshot of your project’s activity and performance at a glance. With recent logs and quick statistics, you can stay updated the moment you log in.
🎯 Langtail SDK Release 🎯
The first version of the Langtail SDK is now live! 🥳 The SDK can function as an OpenAI proxy and is also compatible with prompts directly deployed within Langtail. Dive into the details over at our GitHub docs.
Currently, we’ve rolled out the JS SDK. If you’re interested in a Python SDK, we’d love to hear from you.
New Homepage Video, About Us Page, and Embeddable Prompts
🎞️ New Homepage Video 🎞️
We’ve released a new video that’s now featured on our homepage. Get a quick overview of what Langtail offers by watching our latest video.
🧑💻 About Us Page 🧑💻
Curious about the team behind Langtail? Visit our newly updated About Us page to learn more about our mission and the people who make Langtail possible. Learn more
🛣️ Roadmap, Changelog, and Feedback 🛣️
Your feedback is crucial in helping us improve. Visit our feedback portal to see our roadmap, changelog, and to share your thoughts. Give feedback
🖼️ Embeddable Prompts 🖼️
We’re happy to announce that you can now embed publicly shared prompts into iframes! Here’s a quick example:
📚 Docs Expansion, 🖼️ Dynamic OG Images, and 🌍 Project-Level Env Variables
📚 Expanded Documentation for Seamless Integration 📚
We’ve significantly expanded our documentation to help you hit the ground running:
- 🚀 Deploying Prompts: Learn how to deploy your prompts effortlessly. Read more
- 🌐 REST API Usage: Discover how to leverage our powerful REST API. Learn how
🖼️ Dynamic OG Images for Shared Prompts 🖼️
Sharing is caring, and we’ve made it even better! When you share a prompt publicly, we’ll automatically generate an OG image for you.
🌍 Project-Level Environment Variables 🌍
Tired of hardcoding API endpoints and keys directly in code? We’ve got you covered! Introducing project-level environment variables:
- 🔒 Secure: Keep your sensitive information safe and separate from the code
- 🎛️ Centralized: Set your variables once at the project level
- 🌐 Accessible: Use your env variables in the JavaScript editor for Test Assertions and soon in the Playground too
Langtail.com updates, Proxy Integration, and Quality of Life Improvements 🚀
✨ Landing page updates ✨
Our brand new website is live, showcasing the four key features that make Langtail the ultimate tool for prompt engineering:
- 🐛 Debug prompts effortlessly in the playground
- ✅ Run tests to ensure your prompts are top-notch
- 🚀 Deploy prompts in a flash
- 📊 Monitor production with comprehensive logs and metrics
💸 Pricing Preview 💸
We’ve got a pricing plan for everyone! Choose from three tiers:
- 🆓 Free Tier: Perfect for getting started
- 💰 Mid Tier: Unleash more power at just $99 per month
- 🏢 Enterprise Tier: Tailored solutions for your business needs
📚 Docs Page Expansion 📚
We’ve been busy expanding our documentation to help you get the most out of Langtail. Head over to our docs page to explore the latest information and guides.
🌉 Langtail Proxy: Seamless Integration 🌉
Already building your application using the OpenAI SDK? No problem! With Langtail Proxy, you can experiment with Langtail without migrating your prompts. Here’s how:
- Create your Langtail account
- Generate your project API key
- Modify your code by updating the base URL to
"https://proxy.langtail.com/v1and passing the Langtail API key - Test it out and witness the magic!
🎨 Quality of Life Improvements 🎨
We’ve also released some handy enhancements to make your life easier:
- 🔍 Log Filtering: Easily filter logs by categories like proxy
- 🎯 Single Test Run: Run individual tests instead of the entire suite
- ❌ Test Cancellation: Cancel test runs whenever you need
- 📅 Current Date in System Prompt: Add the current date using handlebars
📆 $Date Helper, 🔍 Enhanced Log Filters, and 🏠 Webpage Updates
📆 $Date Helper for Chatbots 📆
Chatbots can now stay up-to-date with the current date! Introducing the $date helper, a new feature designed to keep your chatbots in sync with real-time.
🔍 Log Filtering 🔍
Say hello to granular control over your logs. You can now filter logs by any column, even custom-send metadata. Find what you need, when you need it.
💰 Pricing Page is Live 💰
Ever wondered about our pricing? We’ve just launched our pricing page, making it easier for you to find the perfect plan for your needs. Check it out!
🚀 Introducing Public Sharing of Prompts and Read-Only Playground!
🎡 Public Sharing of Prompts 🎡
Good news, everyone! We’ve just launched a new feature that lets you publicly share your prompts. Previously, users needed an account to view a shared prompt, but now, anyone can view it thanks to our new read-only Playground view. 🎉
🏞️ Read-Only Playground 🏞️
In the read-only Playground, you can see all messages, variables, and Tools (functions). It’s an open window to the world of your prompts. Currently, only a fixed version of the prompt can be shared, but we’re already working on updates that will let you share and always view the most recent version of a prompt.
🚀 Major DX Improvements for JavaScript Assertions in Tests and Upcoming Playground Enhancements 🛠️
🧪 Enhanced JavaScript Assertions in Tests 🧪
We’ve made some significant improvements to our developer experience for JavaScript assertions in Tests:
- 📝 Run functions with example input: Simply fill in your example input in the left column and run the function.
- ✅ Auto Input Addition: If any input already exists, we’ll automatically add it for you.
- 🐞 Debugging Support: Use
console.logto debug your tests effectively. - 🌐 API Calls: For advanced use cases, you can now call external APIs using the Fetch API.
- ⚡️ Ultra-fast API requests: Thanks to our robust infrastructure, your API requests will execute swiftly.
🚀 New Templating Engine Based on Handlebars for Enhanced Prompt Handling
🎨 Templating Engine Support 🎨
We’re excited to announce that we’ve released support for a templating engine, based on Handlebars! With this new feature, you can now tackle more complex use cases, including:
- Conditional parts of prompts
- Variations of prompts
Simply deploy a prompt with this template, send the correct variable, and watch as the final version of the prompt resolves. You can then view this in the Logs. This added flexibility opens up a world of possibilities for your prompt handling.
🚀 Project Settings, Workspace Refinement, Redesigned Logs, and OpenAI Tools Support!
🔧 New Project Settings 🔧
Managing your projects just got easier! We’ve added a new item in the project-level sidebar that lets you perform common actions like renaming or deleting a project.
🛠️ Enhanced Workspace Settings 🛠️
Your workspace is now even more flexible! We’ve refined the settings to allow you to rename your workspace and made the UI more consistent across the different sections for a smoother experience.
📜 Redesigned Logs 📜
We’ve completely redesigned our Logs for a cleaner, more user-friendly interface. Plus, Logs now also display the JSON representation of what’s being sent from Langtail to end providers like OpenAI. We think you’ll love this upgrade as much as we do!
🔍 OpenAI Tools Support 🔍
We’re thrilled to announce that support for the new OpenAI Tools is here! Functions now visually resemble TypeScript functions and are even typesafe. The types are generated based on the JSON schema. We’re confident you’ll find them incredibly useful.
🌟 GPT-4 Vision is Live, Plus New Workspace Settings and Refined UX 🔍
🌟 GPT-4 Vision Now Live 🌟
We’re thrilled to announce that GPT-4 Vision is now live in production! You can now input images along with text, opening up a whole new realm of creative use cases. For instance, take a photo of an invoice and ask for the cost of each item – Vision has you covered.
🎨 Experimenting with Vision in Playground 🎨
Want to give Vision a spin? Head over to our Playground and upload an image or insert an image URL. Please note that Vision takes a few seconds to process, so a bit of patience is required.
🚀 Deploy Prompts with Vision 🚀
Great news for developers – deploying prompts with Vision in Langtail is now supported!
🛠️ Workspace Settings and UX Updates 🛠️
We’ve also rolled out a couple of key updates for improved user experience:
-
New Workspace Settings: Now, you can manage your provider API keys and team members all in one place.
-
Refined UX for Per API Budget: Based on your feedback, we’ve refined the UX and terminology for the per API budget. You must now enable the budget before setting the total for your API key and specific user allowances.