The clipboard, a foundational component of graphical user interfaces, has long served as a simple, transient buffer for transferring data between applications. Its core function— copy, cut, and paste — has remained largely unchanged for decades. Useful as it is, it often slows things down: users end up reformatting code, extracting data from unstructured text, or manually converting between formats.

As Microsoft’s trusted development partner, Janea Systems helped transform Advanced Paste, the clipboard tool within PowerToys. We led the integration of Semantic Kernel, turning a simple copy-paste function into a customizable AI agent. Continue reading to learn how we did it, the challenges we solved, and what it means for the future of any software product.

From a Toolbox to an AI Agent: Semantic Kernel Integration

The baseline version of Advanced Paste works as a locally-run clipboard manager without any reliance on AI. Its capabilities focus on structured format conversions:

• Invoking shortcuts to paste clipboard content as plain text, JSON, or Markdown.
• Creating files directly from clipboard content (saving text to a .txt file, HTML to a .html file, or a bitmap to a .png file).
• Extracting text from an image on the clipboard with the Optical Character Recognition (OCR) feature.

A practical guide on how to Convert Clipboard with PowerToys Advanced Paste is available for users who want to get hands-on with the feature and explore its text transformation capabilities.

The initial "Paste with AI" feature leveraged a direct call to an OpenAI model to perform a single, discrete transformation on clipboard text. Use cases include summarizing a copied article, translating text, generating code based on a description, or stylizing content, such as rewriting a paragraph in a Shakespearean voice. While powerful, this functionality was limited to one action at a time on text-based input.

Microsoft aimed to create a system that enabled a new class of user interaction with the following capabilities:

• Multi-modal Input: Operating on non-text clipboard data, such as images, which would require an initial OCR step.
• Complex, Chained Actions: Executing a sequence of transformations (OCR, translation, and file saving) in response to a single command.
• Implicit and Explicit Instructions: Handling not just explicit instructions but also inferring necessary intermediate steps. For example, if a user has an image on the clipboard and issues the command, "Translate to French and save as a TXT file," the system must implicitly understand that an OCR operation is the required first step.

To achieve this vision, Semantic Kernel was chosen as the foundation of "Paste with AI." It is an open-source SDK designed as a lightweight orchestration layer to bridge the gap between natural language commands and the executable code needed to accomplish them.

The integration is built on two concepts**:**

• Plugins: Each plugin is a group of C# functions with a clear, descriptive name that LLMs can understand. For example, the ‘Image to Text’ is given the description: "Takes an image in the clipboard and extracts all text from it using OCR."
• Planners: When a user makes a request, the Planner analyzes the goal and chains together the necessary plugins in the correct order to produce a result.

This plugin-based architecture was a critical design choice, making the system highly modular and extensible.

To add a new capability, developers simply need to build a new plugin, and the Planner will automatically incorporate it into its skillset. On top of that, Janea Systems’ engineers designed it to be model-agnostic, compatible with any LLM of choice.

Generative AI Integration Challenges & Solutions

Integrating a sophisticated AI framework like Semantic Kernel presented unique engineering challenges.

Challenge 1: Model Complexity, Latency, and API Costs

State-of-the-art reasoning LLMs are powerful but also have higher latency and API costs. Using a reasoning model for every task would create a poor user experience. This trade-off was explicitly acknowledged in the project's public GitHub issue tracker, which notes that Semantic Kernel queries "consume more OpenAI API credits" than simpler transformations.

Solution: Dual-Model Strategy. The team implemented a two-tiered model architecture. The reasoning model was reserved for the complex task of planning. For simpler, single-step transformations, the system routes requests to the faster, more cost-effective model. Furthermore, to avoid regenerating plans for frequently used custom actions, our engineers implemented intelligent caching. As stated in the design document, "When Semantic Kernel is used for Custom Actions, the chain of actions executed is cached on disk," dramatically improving responsiveness and eliminating redundant API calls for saved workflows.

Challenge 2: Backward Compatibility and User Trust

Introducing a powerful new AI behavior risked disrupting existing user workflows and eroding their trust. The main concern was ensuring transparency around costs, so that users wouldn’t face a hidden increase in token usage without understanding why.

Solution: User-Centric, Opt-In Deployment: The solution was a user-centric deployment strategy. A new, separate toggle switch labeled ‘Enable Advanced AI’ was added to the PowerToys settings. The design stipulated that this toggle would be off when 'Enable Paste with AI' is already enabled (so there is no change for existing users). This opt-in approach for the existing user base respected established workflows while allowing new users to benefit from the more powerful feature by default.

The Engineering Force Behind PowerToys

Microsoft selected Janea Systems as the primary engineering force to transition PowerToys from legacy software to the Windows feature sandbox. The engagement encompassed foundational development and ongoing stewardship of the entire project. Our engineering team was instrumental in developing core modules like FancyZones and PowerToys Run, demonstrating deep systems-level mastery.

If you're interested in a higher-level perspective on the PowerToys project and its various utilities, an interview with Jaime Bernardo, a PowerToys development lead, offers valuable insights into the project's philosophy and its most popular tools.

Beyond engineering, Janea Systems embraced the role of open-source project maintainer, managing over 25,000 issues and reviewing community pull requests. This dual expertise in technical implementation and open-source stewardship solidified Janea Systems' position as a strategic partner of Microsoft.

Take a detailed look into the collaborative development process and learn more about the importance of open-source projects for the development community in our blog.

As Janea Systems continues to drive innovation, our expertise extends beyond system-level engineering into gen AI integration and AI agent development services. As a trusted generative AI integration company, Janea Systems helps enterprises embed gen AI capabilities into their products and mature their custom AI and LLMs.

Contact us to explore how we can bring the same engineering excellence to your AI initiative.