Multi AI Agent Methods 101. Automating Routine Duties in Knowledge Supply… | by Mariya Mansurova | Jun, 2024

Initially, when ChatGPT simply appeared, we used easy prompts to get solutions to our questions. Then, we encountered points with hallucinations and commenced utilizing RAG (Retrieval Augmented Era) to offer extra context to LLMs. After that, we began experimenting with AI brokers, the place LLMs act as a reasoning engine and might determine what to do subsequent, which instruments to make use of, and when to return the ultimate reply.

The subsequent evolutionary step is to create groups of such brokers that may collaborate with one another. This method is logical because it mirrors human interactions. We work in groups the place every member has a selected position:

• The product supervisor proposes the following venture to work on.
• The designer creates its feel and look.
• The software program engineer develops the answer.
• The analyst examines the information to make sure it performs as anticipated and identifies methods to enhance the product for patrons.

Equally, we will create a workforce of AI brokers, every specializing in one area. They’ll collaborate and attain a closing conclusion collectively. Simply as specialization enhances efficiency in actual life, it might additionally profit the efficiency of AI brokers.

One other benefit of this method is elevated flexibility. Every agent can function with its personal immediate, set of instruments and even LLM. As an example, we will use completely different fashions for various elements of our system. You should utilize GPT-4 for the agent that wants extra reasoning and GPT-3.5 for the one which does solely easy extraction. We are able to even fine-tune the mannequin for small particular duties and use it in our crew of brokers.

The potential drawbacks of this method are time and value. A number of interactions and data sharing between brokers require extra calls to LLM and devour further tokens. This might lead to longer wait instances and elevated bills.

There are a number of frameworks out there for multi-agent programs at this time.
Listed below are among the hottest ones:

• AutoGen: Developed by Microsoft, AutoGen makes use of a conversational method and was one of many earliest frameworks for multi-agent programs,
• LangGraph: Whereas not strictly a multi-agent framework, LangGraph permits for outlining advanced interactions between actors utilizing a graph construction. So, it will also be tailored to create multi-agent programs.
• CrewAI: Positioned as a high-level framework, CrewAI facilitates the creation of “crews” consisting of role-playing brokers able to collaborating in varied methods.

I’ve determined to begin experimenting with multi-agent frameworks from CrewAI because it’s fairly extensively common and consumer pleasant. So, it appears to be like like a very good choice to start with.

On this article, I’ll stroll you thru how you can use CrewAI. As analysts, we’re the area consultants accountable for documenting varied information sources and addressing associated questions. We’ll discover how you can automate these duties utilizing multi-agent frameworks.

Let’s begin with organising the atmosphere. First, we have to set up the CrewAI important bundle and an extension to work with instruments.

pip set up crewai
pip set up 'crewai[tools]'

CrewAI was developed to work primarily with OpenAI API, however I’d additionally wish to attempt it with an area mannequin. In line with the ChatBot Arena Leaderboard, the perfect mannequin you may run in your laptop computer is Llama 3 (8b parameters). It will likely be probably the most possible choice for our use case.

We are able to entry Llama fashions utilizing Ollama. Set up is fairly simple. You should obtain Ollama from the website after which undergo the set up course of. That’s it.

Now, you may check the mannequin in CLI by working the next command.

ollama run llama3

For instance, you may ask one thing like this.

Let’s create a customized Ollama mannequin to make use of later in CrewAI.

We’ll begin with a ModelFile (documentation). I solely specified the bottom mannequin (llama3), temperature and cease sequence. Nonetheless, you may add extra options. For instance, you may decide the system message utilizing SYSTEM key phrase.

FROM llama3
# set parameters
PARAMETER temperature 0.5
PARAMETER cease Consequence

I’ve saved it right into a Llama3ModelFile file.

Let’s create a bash script to load the bottom mannequin for Ollama and create the customized mannequin we outlined in ModelFile.

#!/bin/zsh
# outline variables
model_name="llama3"
custom_model_name="crewai-llama3"
# load the bottom mannequin
ollama pull $model_name
# create the mannequin file
ollama create $custom_model_name -f ./Llama3ModelFile

Let’s execute this file.

chmod +x ./llama3_setup.sh
./llama3_setup.sh

Yow will discover each information on GitHub: Llama3ModelFile and llama3_setup.sh

We have to initialise the next environmental variables to make use of the native Llama mannequin with CrewAI.

os.environ["OPENAI_API_BASE"]='http://localhost:11434/v1'
os.environ["OPENAI_MODEL_NAME"]='crewai-llama3' 
# custom_model_name from the bash script
os.environ["OPENAI_API_KEY"] = "NA"

We’ve completed the setup and are able to proceed our journey.

As analysts, we regularly play the position of subject material consultants for information and a few data-related instruments. In my earlier workforce, we used to have a channel with virtually 1K individuals, the place we have been answering plenty of questions on our information and the ClickHouse database we used as storage. It took us numerous time to handle this channel. It could be attention-grabbing to see whether or not such duties could be automated with LLMs.

For this instance, I’ll use the ClickHouse database. In case you’re , You may be taught extra about ClickHouse and how you can set it up domestically in my previous article. Nonetheless, we received’t utilise any ClickHouse-specific options, so be at liberty to stay to the database you understand.

I’ve created a reasonably easy information mannequin to work with. There are simply two tables in our DWH (Knowledge Warehouse): ecommerce_db.customers and ecommerce_db.classes. As you may guess, the primary desk incorporates details about the customers of our service.

The ecommerce_db.classes desk shops details about consumer classes.

Relating to information supply administration, analysts sometimes deal with duties like writing and updating documentation and answering questions on this information. So, we are going to use LLM to jot down documentation for the desk within the database and train it to reply questions on information or ClickHouse.

However earlier than transferring on to the implementation, let’s be taught extra concerning the CrewAI framework and its core ideas.

The cornerstone of a multi-agent framework is an agent idea. In CrewAI, brokers are powered by role-playing. Function-playing is a tactic while you ask an agent to undertake a persona and behave like a top-notch backend engineer or useful buyer help agent. So, when making a CrewAI agent, you want to specify every agent’s position, objective, and backstory in order that LLM is aware of sufficient to play this position.

The brokers’ capabilities are restricted with out tools (features that brokers can execute and get outcomes). With CrewAI, you should use one of many predefined instruments (for instance, to go looking the Web, parse an internet site, or do RAG on a doc), create a customized device your self or use LangChain instruments. So, it’s fairly straightforward to create a strong agent.

Let’s transfer on from brokers to the work they’re doing. Brokers are engaged on tasks (particular assignments). For every process, we have to outline an outline, anticipated output (definition of completed), set of obtainable instruments and assigned agent. I actually like that these frameworks observe the managerial greatest practices like a transparent definition of completed for the duties.

The subsequent query is how you can outline the execution order for duties: which one to work on first, which of them can run in parallel, and so forth. CrewAI applied processes to orchestrate the duties. It supplies a few choices:

• Sequential —probably the most simple method when duties are known as one after one other.
• Hierarchical — when there’s a supervisor (specified as LLM mannequin) that creates and delegates duties to the brokers.

Additionally, CrewAI is engaged on a consensual course of. In such a course of, brokers will be capable of make choices collaboratively with a democratic method.

There are different levers you should use to tweak the method of duties’ execution:

• You may mark duties as “asynchronous”, then they are going to be executed in parallel, so it is possible for you to to get a solution sooner.
• You should utilize the “human enter” flag on a process, after which the agent will ask for human approval earlier than finalising the output of this process. It might let you add an oversight to the method.

We’ve outlined all the first constructing blocks and might focus on the holly grail of CrewAI — crew idea. The crew represents the workforce of brokers and the set of duties they are going to be engaged on. The method for collaboration (processes we mentioned above) will also be outlined on the crew stage.

Additionally, we will arrange the memory for a crew. Reminiscence is essential for environment friendly collaboration between the brokers. CrewAI helps three ranges of reminiscence:

• Brief-term reminiscence shops data associated to the present execution. It helps brokers to work collectively on the present process.
• Lengthy-term reminiscence is information concerning the earlier executions saved within the native database. This kind of reminiscence permits brokers to be taught from earlier iterations and enhance over time.
• Entity reminiscence captures and constructions details about entities (like personas, cities, and so forth.)

Proper now, you may solely swap on all sorts of reminiscence for a crew with none additional customisation. Nonetheless, it doesn’t work with the Llama fashions.

We’ve realized sufficient concerning the CrewAI framework, so it’s time to begin utilizing this information in follow.

Let’s begin with a easy process: placing collectively the documentation for our DWH. As we mentioned earlier than, there are two tables in our DWH, and I want to create an in depth description for them utilizing LLMs.

First method

At first, we’d like to consider the workforce construction. Consider this as a typical managerial process. Who would you rent for such a job?

I’d break this process into two elements: retrieving information from a database and writing documentation. So, we’d like a database specialist and a technical author. The database specialist wants entry to a database, whereas the author received’t want any particular instruments.

Now, we have now a high-level plan. Let’s create the brokers.

For every agent, I’ve specified the position, objective and backstory. I’ve tried my greatest to offer brokers with all of the wanted context.

database_specialist_agent = Agent(
position = "Database specialist",
objective = "Present information to reply enterprise questions utilizing SQL",
backstory = '''You might be an knowledgeable in SQL, so you may assist the workforce 
to assemble wanted information to energy their choices. 
You might be very correct and have in mind all of the nuances in information.''',
allow_delegation = False,
verbose = True
)
tech_writer_agent = Agent(
position = "Technical author",
objective = '''Write partaking and factually correct technical documentation 
for information sources or instruments''',
backstory = ''' 
You might be an knowledgeable in each expertise and communications, so you may simply clarify even refined ideas.
You base your work on the factual data supplied by your colleagues.
Your texts are concise and could be simply understood by a large viewers. 
You employ skilled however moderately an off-the-cuff type in your communication.
''',
allow_delegation = False,
verbose = True
)

We’ll use a easy sequential course of, so there’s no want for brokers to delegate duties to one another. That’s why I specified allow_delegation = False.

The subsequent step is setting the duties for brokers. However earlier than transferring to them, we have to create a customized device to hook up with the database.

First, I put collectively a operate to execute ClickHouse queries utilizing HTTP API.

CH_HOST = 'http://localhost:8123' # default deal with 
def get_clickhouse_data(question, host = CH_HOST, connection_timeout = 1500):
r = requests.publish(host, params = {'question': question}, 
timeout = connection_timeout)
if r.status_code == 200:
return r.textual content
else: 
return 'Database returned the next error:n' + r.textual content

When working with LLM brokers, it’s vital to make instruments fault-tolerant. For instance, if the database returns an error (status_code != 200), my code received’t throw an exception. As an alternative, it can return the error description to the LLM so it may try and resolve the difficulty.

To create a CrewAI customized device, we have to derive our class from crewai_tools.BaseTool, implement the _run technique after which create an occasion of this class.

from crewai_tools import BaseTool
class DatabaseQuery(BaseTool):
title: str = "Database Question"
description: str = "Returns the results of SQL question execution"
def _run(self, sql_query: str) -> str:
# Implementation goes right here
return get_clickhouse_data(sql_query)
database_query_tool = DatabaseQuery()

Now, we will set the duties for the brokers. Once more, offering clear directions and all of the context to LLM is essential.

table_description_task = Process(
description = '''Present the excellent overview for the information 
in desk {desk}, in order that it is simple to know the construction 
of the information. This process is essential to place collectively the documentation 
for our database''',
expected_output = '''The excellent overview of {desk} within the md format. 
Embrace 2 sections: columns (checklist of columns with their varieties) 
and examples (the primary 30 rows from desk).''',
instruments = [database_query_tool],
agent = database_specialist_agent
)
table_documentation_task = Process(
description = '''Utilizing supplied details about the desk, 
put collectively the detailed documentation for this desk in order that 
folks can use it in follow''',
expected_output = '''Nicely-written detailed documentation describing 
the information scheme for the desk {desk} in markdown format, 
that provides the desk overview in 1-2 sentences then then 
describes every columm. Construction the columns description 
as a markdown desk with column title, kind and outline.''',
instruments = [],
output_file="table_documentation.md",
agent = tech_writer_agent
)

You may need seen that I’ve used {desk} placeholder within the duties’ descriptions. We’ll use desk as an enter variable when executing the crew, and this worth might be inserted into all placeholders.

Additionally, I’ve specified the output file for the desk documentation process to avoid wasting the ultimate outcome domestically.

We’ve got all we’d like. Now, it’s time to create a crew and execute the method, specifying the desk we’re curious about. Let’s attempt it with the customers desk.

crew = Crew(
brokers = [database_specialist_agent, tech_writer_agent],
duties = [table_description_task,  table_documentation_task],
verbose = 2
)
outcome = crew.kickoff({'desk': 'ecommerce_db.customers'})

It’s an thrilling second, and I’m actually trying ahead to seeing the outcome. Don’t fear if execution takes a while. Brokers make a number of LLM calls, so it’s completely regular for it to take a couple of minutes. It took 2.5 minutes on my laptop computer.

We requested LLM to return the documentation in markdown format. We are able to use the next code to see the formatted lead to Jupyter Pocket book.

from IPython.show import Markdown
Markdown(outcome)

At first look, it appears to be like nice. We’ve acquired the legitimate markdown file describing the customers’ desk.

However wait, it’s incorrect. Let’s see what information we have now in our desk.

The columns listed within the documentation are utterly completely different from what we have now within the database. It’s a case of LLM hallucinations.

We’ve set verbose = 2 to get the detailed logs from CrewAI. Let’s learn by the execution logs to determine the foundation reason for the issue.

First, the database specialist couldn’t question the database as a result of issues with quotes.

The specialist didn’t handle to resolve this drawback. Lastly, this chain has been terminated by CrewAI with the next output: Agent stopped as a result of iteration restrict or time restrict.

This implies the technical author didn’t obtain any factual details about the information. Nonetheless, the agent continued and produced utterly pretend outcomes. That’s how we ended up with incorrect documentation.

Fixing the problems

Despite the fact that our first iteration wasn’t profitable, we’ve realized so much. We’ve got (at the very least) two areas for enchancment:

• Our database device is simply too troublesome for the mannequin, and the agent struggles to make use of it. We are able to make the device extra tolerant by eradicating quotes from the start and finish of the queries. This resolution shouldn’t be excellent since legitimate SQL can finish with a quote, however let’s attempt it.
• Our technical author isn’t basing its output on the enter from the database specialist. We have to tweak the immediate to focus on the significance of offering solely factual data.

So, let’s attempt to repair these issues. First, we are going to repair the device — we will leverage strip to get rid of quotes.

CH_HOST = 'http://localhost:8123' # default deal with 
def get_clickhouse_data(question, host = CH_HOST, connection_timeout = 1500):
r = requests.publish(host, params = {'question': question.strip('"').strip("'")}, 
timeout = connection_timeout)
if r.status_code == 200:
return r.textual content
else: 
return 'Database returned the next error:n' + r.textual content

Then, it’s time to replace the immediate. I’ve included statements emphasizing the significance of sticking to the info in each the agent and process definitions.

tech_writer_agent = Agent(
position = "Technical author",
objective = '''Write partaking and factually correct technical documentation 
for information sources or instruments''',
backstory = ''' 
You might be an knowledgeable in each expertise and communications, so that you 
can simply clarify even refined ideas.
Your texts are concise and could be simply understood by vast viewers. 
You employ skilled however moderately casual type in your communication.
You base your work on the factual data supplied by your colleagues. 
You stick with the info within the documentation and use ONLY 
data supplied by the colleagues not including something.''',
allow_delegation = False,
verbose = True
)
table_documentation_task = Process(
description = '''Utilizing supplied details about the desk, 
put collectively the detailed documentation for this desk in order that 
folks can use it in follow''',
expected_output = '''Nicely-written detailed documentation describing 
the information scheme for the desk {desk} in markdown format, 
that provides the desk overview in 1-2 sentences then then 
describes every columm. Construction the columns description 
as a markdown desk with column title, kind and outline.
The documentation is predicated ONLY on the knowledge supplied 
by the database specialist with none additions.''',
instruments = [],
output_file = "table_documentation.md",
agent = tech_writer_agent
)

Let’s execute our crew as soon as once more and see the outcomes.

We’ve achieved a bit higher outcome. Our database specialist was capable of execute queries and think about the information, which is a big win for us. Moreover, we will see all of the related fields within the outcome desk, although there are many different fields as properly. So, it’s nonetheless not totally appropriate.

I as soon as once more appeared by the CrewAI execution log to determine what went improper. The problem lies in getting the checklist of columns. There’s no filter by database, so it returns some unrelated columns that seem within the outcome.

SELECT column_name 
FROM information_schema.columns 
WHERE table_name = 'customers'

Additionally, after a number of makes an attempt, I seen that the database specialist, on occasion, executes choose * from <desk> question. It would trigger some points in manufacturing as it’d generate plenty of information and ship it to LLM.

Extra specialised instruments

We are able to present our agent with extra specialised instruments to enhance our resolution. At the moment, the agent has a device to execute any SQL question, which is versatile and highly effective however liable to errors. We are able to create extra centered instruments, corresponding to getting desk construction and top-N rows from the desk. Hopefully, it can cut back the variety of errors.

class TableStructure(BaseTool):
title: str = "Desk construction"
description: str = "Returns the checklist of columns and their varieties"
def _run(self, desk: str) -> str:
desk = desk.strip('"').strip("'")
return get_clickhouse_data(
'describe {desk} format TabSeparatedWithNames'
.format(desk = desk)
)
class TableExamples(BaseTool):
title: str = "Desk examples"
description: str = "Returns the primary N rows from the desk"
def _run(self, desk: str, n: int = 30) -> str:
desk = desk.strip('"').strip("'")
return get_clickhouse_data(
'choose * from {desk} restrict {n} format TabSeparatedWithNames'
.format(desk = desk, n = n)
)
table_structure_tool = TableStructure()
table_examples_tool = TableExamples()

Now, we have to specify these instruments within the process and re-run our script. After the primary try, I acquired the next output from the Technical Author.

Process output: This closing reply supplies an in depth and factual description 
of the ecommerce_db.customers desk construction, together with column names, varieties, 
and descriptions. The documentation adheres to the supplied data 
from the database specialist with none additions or modifications.

Extra centered instruments helped the database specialist retrieve the right desk data. Nonetheless, although the author had all the required data, we didn’t get the anticipated outcome.

As we all know, LLMs are probabilistic, so I gave it one other attempt. And hooray, this time, the outcome was fairly good.

It’s not good because it nonetheless consists of some irrelevant feedback and lacks the general description of the desk. Nonetheless, offering extra specialised instruments has undoubtedly paid off. It additionally helped to forestall points when the agent tried to load all the information from the desk.

High quality assurance specialist

We’ve achieved fairly good outcomes, however let’s see if we will enhance them additional. A standard follow in multi-agent setups is high quality assurance, which provides the ultimate assessment stage earlier than finalising the outcomes.

Let’s create a brand new agent — a High quality Assurance Specialist, who might be in command of assessment.

qa_specialist_agent = Agent(
position = "High quality Assurance specialist",
objective = """Guarantee the very best high quality of the documentation we offer 
(that it is appropriate and simple to know)""",
backstory = '''
You're employed as a High quality Assurance specialist, checking the work 
from the technical author and guaranteeing that it is inline 
with our highest requirements.
You should verify that the technical author supplies the complete full 
solutions and make no assumptions. 
Additionally, you want to make it possible for the documentation addresses 
all of the questions and is simple to know.
''',
allow_delegation = False,
verbose = True
)

Now, it’s time to explain the assessment process. I’ve used the context parameter to specify that this process requires outputs from each table_description_task and table_documentation_task.

qa_review_task = Process(
description = '''
Overview the draft documentation supplied by the technical author.
Be sure that the documentation totally solutions all of the questions: 
the aim of the desk and its construction within the type of desk. 
Make it possible for the documentation is in keeping with the knowledge 
supplied by the database specialist. 
Double verify that there aren't any irrelevant feedback within the closing model 
of documentation.
''',
expected_output = '''
The ultimate model of the documentation in markdown format 
that may be printed. 
The documentation ought to totally deal with all of the questions, be constant 
and observe our skilled however casual tone of voice.
''',
instruments = [],
context = [table_description_task, table_documentation_task],
output_file="checked_table_documentation.md",
agent = qa_specialist_agent
)

Let’s replace our crew and run it.

full_crew = Crew(
brokers=[database_specialist_agent, tech_writer_agent, qa_specialist_agent],
duties=[table_description_task,  table_documentation_task, qa_review_task],
verbose = 2,
reminiscence = False # do not work with Llama
)
full_result = full_crew.kickoff({'desk': 'ecommerce_db.customers'})

We now have extra structured and detailed documentation due to the addition of the QA stage.

Delegation

With the addition of the QA specialist, it might be attention-grabbing to check the delegation mechanism. The QA specialist agent may need questions or requests that it might delegate to different brokers.

I attempted utilizing the delegation with Llama 3, but it surely didn’t go properly. Llama 3 struggled to name the co-worker device appropriately. It couldn’t specify the right co-worker’s title.

We achieved fairly good outcomes with an area mannequin that may run on any laptop computer, however now it’s time to change gears and use a far more highly effective mannequin — GPT-4o.

To do it, we simply have to replace the next atmosphere variables.

os.environ["OPENAI_MODEL_NAME"] = 'gpt-4o'  
os.environ["OPENAI_API_KEY"] = config['OPENAI_API_KEY'] # your OpenAI key

To change on the delegation, we should always specify allow_delegation = True for the QA specialist agent.

Additionally, we will use useful reminiscence performance for OpenAI fashions, as talked about above. The reminiscence will permit our brokers to share their data with one another throughout execution and leverage long-term reminiscence to get data from earlier executions.

full_crew = Crew(
brokers = [database_specialist_agent, tech_writer_agent, qa_specialist_agent],
duties = [table_description_task,  table_documentation_task, qa_review_task],
verbose = 2,
reminiscence = True 
)
full_result = full_crew.kickoff({'desk': 'ecommerce_db.customers'})

Let’s see the CrewAI execution logs to know how delegation works. Listed below are all of the logs for the QA specialist. We are able to see that it reached out to the database specialist to double-check the knowledge.

Thought: I have to assessment the supplied draft documentation to make sure it 
totally solutions all of the questions and is full, appropriate, and simple 
to know. Particularly, I want to make sure that it explains the aim 
of the desk, its construction, and consists of related examples with out 
any irrelevant feedback.
First, I'll examine the 2 drafts to determine any discrepancies and 
lacking data. Then, I'll confirm the accuracy and completeness of 
the knowledge supplied.
**Comparability:**
1. **Introduction:**
- First draft: Consists of an introduction and examples.
- Second draft: Provides a extra detailed introduction concerning the goal and significance of the desk.
2. **Columns:**
- Each drafts have the identical column construction and descriptions.
3. **Further Info:**
- Each drafts embrace related further data, however the second draft is extra concise.
4. **Examples:**
- The primary draft consists of examples, that are lacking within the second draft.
5. **Remaining Checks:**
- Guarantee there aren't any irrelevant feedback.
- Confirm consistency with data supplied by the database specialist.
Motion: Ask query to co-worker
Motion Enter:  25  
<-- omitted logs from the database specialist -->
Based mostly on the supplied context and examples for the `ecommerce_db.customers` desk, 
the column descriptions and examples seem correct. Listed below are the main points:
- `user_id`: This column incorporates a singular identifier for every consumer. 
The supplied examples (e.g., 1000001, 1000002) align with this description 
as they're distinctive numeric identifiers.
- `nation`: This column specifies the nation of the consumer. The supplied 
examples (e.g., United Kingdom, France, Germany, Netherlands) are 
all legitimate nation names.
- `is_active`: This column signifies whether or not the consumer is energetic (1) or not (0). The supplied examples (e.g., 0, 1) appropriately characterize this binary standing.
- `age`: This column reveals the age of the consumer. The supplied examples (e.g., 70, 
87, 88, 25, 48, 78, 65, 31, 66, 73) are all legitimate ages.
Subsequently, the column descriptions and examples supplied for the 
`ecommerce_db.customers` desk are correct and meet the anticipated standards.
Thought: Based mostly on the statement from the database specialist, 
the column descriptions and examples supplied for the `ecommerce_db.customers` 
desk are correct. Now, I'll consolidate the perfect parts from each drafts,
guaranteeing the documentation is full, clear, and freed from irrelevant feedback.
Motion: Finalize the documentation in markdown format, incorporating 
the detailed introduction, examples, and guaranteeing the tone is skilled 
however casual.

After I tried the delegation for the primary time, I didn’t allow reminiscence, which led to incorrect outcomes. The info specialist and the technical author initially returned the right data. Nonetheless, when the QA specialist returned with the follow-up questions, they began to hallucinate. So, it appears to be like like delegation works higher when reminiscence is enabled.

Right here’s the ultimate output from GPT-4o. The outcome appears to be like fairly good now. We undoubtedly can use LLMs to automate documentation.

So, the primary process has been solved!

I used the identical script to generate documentation for the ecommerce_db.classes desk as properly. It will likely be useful for our subsequent process. So, let’s not waste any time and transfer on.

Our subsequent process is answering questions primarily based on the documentation because it’s frequent for a lot of information analysts (and different specialists).

We’ll begin easy and can create simply two brokers:

• The documentation help specialist might be answering questions primarily based on the docs,
• The help QA agent will assessment the reply earlier than sharing it with the client.

We might want to empower the documentation specialist with a few instruments that may permit them to see all of the information saved within the listing and skim the information. It’s fairly simple since CrewAI has applied such instruments.

from crewai_tools import DirectoryReadTool, FileReadTool
documentation_directory_tool = DirectoryReadTool(
listing = '~/crewai_project/ecommerce_documentation')
base_file_read_tool = FileReadTool()

Nonetheless, since Llama 3 retains fighting quotes when calling instruments, I needed to create a customized device on high of the FileReaderTool to beat this problem.

from crewai_tools import BaseTool
class FileReadToolUPD(BaseTool):
title: str = "Learn a file's content material"
description: str = "A device that can be utilized to learn a file's content material."
def _run(self, file_path: str) -> str:
# Implementation goes right here
return base_file_read_tool._run(file_path = file_path.strip('"').strip("'"))
file_read_tool = FileReadToolUPD()

Subsequent, as we did earlier than, we have to create brokers, duties and crew.

data_support_agent = Agent(
position = "Senior Knowledge Help Agent",
objective = "Be probably the most useful help for you colleagues",
backstory = '''You're employed as a help for data-related questions 
within the firm. 
Despite the fact that you are a giant knowledgeable in our information warehouse, you double verify 
all of the info in documentation. 
Our documentation is totally up-to-date, so you may totally depend on it 
when answering questions (you need not verify the precise information 
in database).
Your work is essential for the workforce success. Nonetheless, bear in mind 
that examples of desk rows do not present all of the potential values. 
You should be certain that you present the absolute best help: answering 
all of the questions, making no assumptions and sharing solely the factual information.
Be artistic attempt your greatest to unravel the client drawback. 
''',
allow_delegation = False,
verbose = True
)
qa_support_agent = Agent(
position = "Help High quality Assurance Agent",
objective = """Guarantee the very best high quality of the solutions we offer 
to the shoppers""",
backstory = '''You're employed as a High quality Assurance specialist, checking the work 
from help brokers and guaranteeing that it is inline with our highest requirements.
You should verify that the agent supplies the complete full solutions 
and make no assumptions. 
Additionally, you want to make it possible for the documentation addresses all 
the questions and is simple to know.
''',
allow_delegation = False,
verbose = True
)
draft_data_answer = Process(
description = '''Essential buyer {buyer} reached out to you 
with the next query:
```
{query}
```
Your process is to offer the perfect reply to all of the factors within the query 
utilizing all out there data and never making any assumprions. 
If you do not have sufficient data to reply the query, simply say 
that you do not know.''',
expected_output = '''The detailed informative reply to the client's 
query that addresses all the purpose talked about. 
Make it possible for reply is full and stict to info 
(with none further data not primarily based on the factual information)''',
instruments = [documentation_directory_tool, file_read_tool], 
agent = data_support_agent
)
answer_review = Process(
description = '''
Overview the draft reply supplied by the help agent.
Be sure that the it totally solutions all of the questions talked about 
within the preliminary inquiry. 
Make it possible for the reply is constant and would not embrace any assumptions.
''',
expected_output = '''
The ultimate model of the reply in markdown format that may be shared 
with the client. 
The reply ought to totally deal with all of the questions, be constant 
and observe our skilled however casual tone of voice. 
We're very chill and pleasant firm, so do not forget to incorporate 
all of the well mannered phrases.
''',
instruments = [], 
agent = qa_support_agent
)
qna_crew = Crew(
brokers = [data_support_agent, qa_support_agent],
duties = [draft_data_answer,  answer_review],
verbose = 2,
reminiscence = False # do not work with Llama
)

Let’s see the way it works in follow.

outcome = qna_crew.kickoff(
{'buyer': "Max", 
'query': """Hey workforce, I hope you are doing properly. I want to search out 
the numbers earlier than our CEO presentation tomorrow, so I'll actually 
admire your assist.
I have to calculate the variety of classes from our Home windows customers in 2023. I've tried to search out the desk with such information in our information warehouse, however wasn't capable of. 
Do you may have any concepts whether or not we retailer the wanted information someplace, 
in order that I can question it? """
}
)

We’ve acquired a well mannered, sensible and useful reply in return. That’s actually nice.

**Hiya Max,**
Thanks for reaching out along with your query! I am glad that can assist you 
discover the variety of classes from Home windows customers in 2023. 
After reviewing our documentation, I discovered that we do retailer information 
associated to classes and customers in our ecommerce database, particularly in 
the `ecommerce_db.classes` desk.
To reply your query, I can give you a step-by-step information 
on how you can question this desk utilizing SQL. First, you should use the `session_id` 
column together with the `os` column filtering for "Home windows" and 
the `action_date` column filtering for dates in 2023. 
Then, you may group the outcomes by `os` utilizing the `GROUP BY` clause 
to depend the variety of classes that meet these circumstances.
This is a pattern SQL question that ought to provide the desired output:
```sql
SELECT COUNT(*) 
FROM ecommerce_db.classes 
WHERE os = 'Home windows' 
AND action_date BETWEEN '2023-01-01' AND '2023-12-31'
GROUP BY os;
```
This question will return the overall variety of classes from Home windows 
customers in 2023. I hope this helps! When you have any additional questions or 
want extra help, please do not hesitate to ask.

Let’s complicate the duty a bit. Suppose we will get not solely questions on our information but additionally about our device (ClickHouse). So, we could have one other agent within the crew — ClickHouse Guru. To offer our CH agent some data, I’ll share a documentation web site with it.

from crewai_tools import ScrapeWebsiteTool, WebsiteSearchTool
ch_documenation_tool = ScrapeWebsiteTool(
'https://clickhouse.com/docs/en/guides/creating-tables')

If you want to work with a prolonged doc, you may attempt utilizing RAG (Retrieval Augmented era) — WebsiteSearchTool. It’s going to calculate embeddings and retailer them domestically in ChromaDB. In our case, we are going to stick with a easy web site scraper device.

Now that we have now two subject material consultants, we have to determine who might be engaged on the questions. So, it’s time to make use of a hierarchical course of and add a supervisor to orchestrate all of the duties.

CrewAI supplies the supervisor implementation, so we solely have to specify the LLM mannequin. I’ve picked the GPT-4o.

from langchain_openai import ChatOpenAI
from crewai import Course of
complext_qna_crew = Crew(
brokers = [ch_support_agent, data_support_agent, qa_support_agent],
duties = [draft_ch_answer, draft_data_answer, answer_review],
verbose = 2,
manager_llm = ChatOpenAI(mannequin='gpt-4o', temperature=0),  
course of = Course of.hierarchical,  
reminiscence = False 
)

At this level, I needed to swap from Llama 3 to OpenAI fashions once more to run a hierarchical course of because it hasn’t labored for me with Llama (much like this issue).

Now, we will attempt our new crew with various kinds of questions (both associated to our information or ClickHouse database).

ch_result = complext_qna_crew.kickoff(
{'buyer': "Maria", 
'query': """Good morning, workforce. I am utilizing ClickHouse to calculate 
the variety of clients. 
May you please remind whether or not there's an choice so as to add totals 
in ClickHouse?"""
}
)
doc_result = complext_qna_crew.kickoff(
{'buyer': "Max", 
'query': """Hey workforce, I hope you are doing properly. I want to search out 
the numbers earlier than our CEO presentation tomorrow, so I'll actually 
admire your assist.
I have to calculate the variety of classes from our Home windows customers 
in 2023. I've tried to search out the desk with such information 
in our information warehouse, however wasn't capable of. 
Do you may have any concepts whether or not we retailer the wanted information someplace, 
in order that I can question it. """
}
)

If we take a look at the ultimate solutions and logs (I’ve omitted them right here since they’re fairly prolonged, but you could find them and full logs on GitHub), we are going to see that the supervisor was capable of orchestrate appropriately and delegate duties to co-workers with related data to deal with the client’s query. For the primary (ClickHouse-related) query, we acquired an in depth reply with examples and potential implications of utilizing WITH TOTALS performance. For the data-related query, fashions returned roughly the identical data as we’ve seen above.

So, we’ve constructed a crew that may reply varied sorts of questions primarily based on the documentation, whether or not from an area file or an internet site. I believe it’s a superb outcome.

Yow will discover all of the code on GitHub.

On this article, we’ve explored utilizing the CrewAI multi-agent framework to create an answer for writing documentation primarily based on tables and answering associated questions.

Given the in depth performance we’ve utilised, it’s time to summarise the strengths and weaknesses of this framework.

General, I discover CrewAI to be an extremely helpful framework for multi-agent programs:

• It’s simple, and you may construct your first prototype rapidly.
• Its flexibility permits to unravel fairly refined enterprise issues.
• It encourages good practices like role-playing.
• It supplies many useful instruments out of the field, corresponding to RAG and an internet site parser.
• The help of various kinds of reminiscence enhances the brokers’ collaboration.
• Constructed-in guardrails assist stop brokers from getting caught in repetitive loops.

Nonetheless, there are areas that could possibly be improved:

• Whereas the framework is straightforward and simple to make use of, it’s not very customisable. As an example, you presently can’t create your personal LLM supervisor to orchestrate the processes.
• Typically, it’s fairly difficult to get the complete detailed data from the documentation. For instance, it’s clear that CrewAI applied some guardrails to forestall repetitive operate calls, however the documentation doesn’t totally clarify the way it works.
• One other enchancment space is transparency. I like to know how frameworks work below the hood. For instance, in Langchain, you should use langchain.debug = True to see all of the LLM calls. Nonetheless, I haven’t found out how you can get the identical stage of element with CrewAI.
• The total help for the native fashions could be an excellent addition, as the present implementation both lacks some options or is troublesome to get working correctly.

The area and instruments for LLMs are evolving quickly, so I’m hopeful that we’ll see a number of progress within the close to future.

Thank you a large number for studying this text. I hope this text was insightful for you. When you have any follow-up questions or feedback, please depart them within the feedback part.

This text is impressed by the “Multi AI Agent Systems with CrewAI” brief course from DeepLearning.AI.