Character.AI has made significant breakthroughs in the field of prompt engineering, recognizing its crucial role in business operations. The company's approach to prompt construction is comprehensive, taking into account various factors such as dialogue patterns, ongoing experiments, character profiles, chat types, user attributes, fixed memories, user roles, and the entire conversation history. This meticulous design is necessary because they generate a massive number of prompts every day - billions of them - and need to maximize the potential of expanding the LLM context window. In the face of various use cases, Character.AI advocates for a shift from traditional "prompt engineering" to "prompt design". This evolution aims to go beyond mundane string operations and create precise and engaging prompts. To achieve this goal, they have developed an innovative library called Prompt Poet. Character.AI's innovative prompt design approach is embodied in their newly developed tool, Prompt Poet. While Python's f-strings have become the industry standard for prompt engineers, allowing for simple query insertion to complex string operations, this approach often requires programming skills, limiting accessibility for non-technical users. Prompt Poet addresses this challenge by providing a more intuitive and efficient solution for designing and managing production prompts, catering to both developers and non-technical personnel. This tool significantly reduces the time spent on engineering string operations, allowing users to focus on creating the best prompts. Drawing inspiration from UI design principles, Prompt Poet conceptualizes prompts as functions with runtime states, encompassing elements such as prompt templates, data, token constraints, and other relevant factors. This approach represents a significant step towards making prompt design more accessible and efficient for a wider user base. Prompt Poet revolutionizes the prompt creation process by shifting the focus from engineering to design. This innovative tool combines YAML and Jinja2 for templating, providing flexibility and composability. Template processing in Prompt Poet is divided into two key stages: rendering and loading. During the rendering stage, Jinja2 handles input data, executes control flow logic, validates and binds data to variables, and evaluates template functions. The loading stage then converts the rendered output into structured YAML files, which are subsequently transformed into Python data structures. Each prompt has specific attributes: a readable name, the actual content string, an optional role specifier to differentiate between user or system components, and an optional truncation priority. This structured approach allows for more efficient prompt management, enabling both technical and non-technical users to create and iterate prompts with ease and accuracy. Prompt Poet combines Jinja2 and YAML to create a powerful and flexible template system. Jinja2 brings dynamic capabilities to templates, allowing for direct data binding, arbitrary function calls, and basic control flow structures. This flexibility empowers users to create complex, context-aware prompts that adapt to various scenarios. YAML, on the other hand, provides a structured format for templates, with a depth of one level, which is crucial for implementing complex truncation strategies when token limits are reached. This structured approach ensures that prompts remain coherent and effective even when shortening is necessary. Character.AI is committed to continuously improving its model alignment methods with user preferences. By using Prompt Poet, they have created a system that seamlessly reconstructs production prompts in offline processes such as evaluation and post-training workloads. This templated approach brings significant advantages to their workflow. It enables easy sharing of template files among different teams within the organization, eliminating the need to piece together various parts of their ever-changing codebase. This streamlined approach not only enhances collaboration but also ensures consistency in prompt design across different development and deployment stages. The ability of Jinja2 to call arbitrary Python functions at runtime within templates is a key feature of Prompt Poet. This functionality enables real-time data retrieval, manipulation, and validation, simplifying prompt construction. For example, an `extract_user_query_topic` function can handle user queries for template control flow, potentially involving a round trip to a topic classifier. This feature significantly enhances the dynamic capabilities of prompt design. Prompt Poet defaults to using the "o200k_base" tokenizer from TikToken but allows for alternative encoding names through the `tiktoken_encoding_name` parameter. Users can also provide their own encoding function using the `encode_func` parameter, which should be a callable object that accepts a string and returns a list of integers. This flexibility allows for custom tokenization processes based on specific requirements. For LLM providers that support GPU affinity and prefix caching, Character.AI's truncation algorithm maximizes the prefix cache rate. The cache rate represents the proportion of prompt tokens retrieved from the cache to the total prompt tokens. Users should find the optimal truncation step size and token limit for their usage. Increasing the truncation step size improves the prefix cache rate but also increases the number of truncated tokens in prompts. Character.AI's truncation strategy achieves a significant 95% cache rate through optimized message truncation. The strategy involves truncating to a fixed point, moving that point every k steps on average. This approach maximizes the utilization of GPU prefix caching, as described in "Optimizing Inference". While this method often truncates more than strictly necessary, it outperforms simple token limit truncation in terms of cache utilization. In a typical chat scenario, messages from M1 to M10, a simple truncation to the token limit would result in the truncation point moving every time. This would lead to a small prefix portion retrievable from the cache, resulting in a significant recomputation cost. This approach fails to fully leverage the GPU prefix cache. Character.AI's cache-aware truncation algorithm keeps the truncation point fixed every k steps. This approach preserves an uninterrupted sequence of tokens until the most recent message, allowing for reusing the previous computation stored in the GPU prefix cache. The value of k is determined by the truncation step size and the average number of tokens per truncated message. Prompt Poet revolutionizes prompt engineering by shifting the focus from manual string operations to intuitive design. It simplifies the creation of complex prompts, enhancing AI interaction with users. By empowering both technical and non-technical users to prioritize design over engineering, Prompt Poet has the potential to transform AI interaction, making it more efficient and user-centric. As large language models continue to evolve, tools like Prompt Poet will become crucial in maximizing their potential in a user-friendly manner.