Improve User Experience with Natural Language Commands

How you can revolutionise the way users interact with your digital system using LLMs and Tool Calling.

Have you ever wished your software could just understand and execute your requests without the need for manual inputs, programming, or navigating through multiple screens?

This is where tool calling comes in. Tool calling allows you to interact with complex software systems using natural language commands, transforming the way we interact with digital systems. This innovation simplifies user experiences and makes technology more accessible.

What is Tool Calling?

Tool calling — also known as function calling -enables Large Language Models (LLMs) to determine and execute specific software functions based on user input. This technique revolutionises interactions with systems and databases, allowing seamless task execution through natural language queries.

To understand tool calling better, refer to the diagram below which outlines the process flow from user input to tool execution.

The diagram shows how the AI assistant selects and runs the relevant tool, processes the output, and generates a response. This seamless interaction is the core of tool calling.

Main categories of Tools

• Data Retrieval: Fetching data from databases based on user queries, such as retrieving order statuses.
• Data Update: Modifying existing records in a database, like updating delivery addresses.
• Analysis and Computation: Performing calculations or data analysis tasks, such as computing delivery estimates.

How are Tools Used in Practice?

Let’s consider a customer interacting with an e-commerce chatbot. This chatbot can fetch order statuses, update delivery information, and more using tools like check_order_status and update_address.

For instance, if a user wants to know the status of their latest order, the chatbot can call the check_order_status tool with parameters such as order ID and user ID, as shown in the JSON output below. Here’s what the structured output might look like:

{
  "function": "check_order_status",
  "parameters": {
    "order_id": "latest",
    "user_id": 9443
  }
}

In this example, we see that the model wants to call a function called check_order_status, and set the required input parameters order_id=latest and user_id=9443.

Whenever we see this type of structured output, we can call this function with the provided parameters and then pass the outputs back to the LLM. This allows the LLM to generate a textual answer based on the function’s response, seamlessly integrating backend processes with natural language interactions.

Then, at this stage, the LLM will have access to the output of the function and hence will be able to generate a textual answer to the user.

You can imagine a whole conversation between the user and the assistant, with the AI assistant remembering the outputs of functions called previously (e.g., order_id=12345). For example:

And so on. You can imagine any other action as long as the LLM has access to a tool that can execute that action! Basically, with tools, you can give the ability to interact with the world whenever needed.

How to Get Started?

This paradigm shift allows us to focus less on the LLM logic and more on what can bring value to the user.

Instead of pre-designing an AI assistant for all use cases, ask yourself: “If the user had a human assistant, what actions would the assistant do?” Your goal should be to improve communication between users and your software by implementing functions or APIs to execute these actions, such as fetching user data or updating records.

Step-by-Step Guide

1. Identify Key Actions: Determine the primary tasks that an assistant would need to do when helping a customer, such as checking order statuses or updating delivery information. Think about the common user requests and how a human assistant would handle them.
2. Develop Functions/APIs: Implement the necessary functions or APIs that can perform these tasks. Ensure they are well-documented and easily accessible. Consider breaking down complex tasks into simpler, more manageable functions.
3. Integrate with LLM: Connect these functions to your LLM. This process is straightforward — just a model that supports function calling, a predefined list of tools, and some code to integrate them.
4. Testing and Refinement: Test the integration thoroughly to ensure the LLM correctly calls the functions and handles the responses. Refine as needed to improve accuracy and reliability.

For a detailed coding example, stay tuned for our future post. In the meantime, you can check out this OpenAI Cookbook for practical guidance.

Add Speech to Get the Magic!

Integrate this with models like OpenAI’s Whisper for speech-to-text functionality, and you can record your queries and receive responses directly on your phone. This seamless integration is groundbreaking:

• Logistics: Warehouse workers can update inventory, track shipments, and manage logistics through voice commands. Imagine a worker saying, “Check inventory for item 123,” and getting an immediate response.
• Manufacturing: Technicians can perform equipment checks, log maintenance activities, and request parts without stopping their work. For example, a technician could say, “Log maintenance for machine A,” and continue their task seamlessly.
• Field Services: Field technicians can access client information, update job statuses, and request real-time support, all while on the move. Picture a technician asking, “Update job status to complete for client XYZ,” and the system handling it instantly.

Conclusion

Function calling changes how we interact with software by letting LLMs perform specific tasks based on user queries. By using function calling, businesses can simplify operations, save money, and provide customers with more personalised experiences.

Ready to revolutionise your workflow? Start exploring tool calling today by integrating it into your systems.

Hi! I’m Louis, an AI-Engineer specialising in custom, privacy-focused LLM solutions for startups. Follow for updates and insights. Have questions or need help? Connect on LinkedIn or chat in the comments!