Create A Hackathon Judge Committee with Workforce#
Workforce is a system where multiple agents collaborate to solve a given task. In this notebook, we will walk through it with a demo of a hackathon judge committee, where judges with different personas collaborate together to give scores to hackathon projects.
You can also check this notebook in colab here.
Dependency Installation#
To get started, make sure you have camel-ai[all]==0.2.1 installed.
[ ]:
%pip install "camel-ai[all]==0.2.1"
Workforce employs an asynchronous design with coroutines. However, since coroutines cannot directly run in notebooks, we need to do specific handlings in this demo. Note that, under most normal cases (not inside notebook environment), we don’t need to do this.
[ ]:
%pip install nest_asyncio
import nest_asyncio
nest_asyncio.apply()
Key Configuration#
In this demo, we will use tools related to web searching. Therefore, we need to configure the OpenAI API key, along with the Google API keys beforeheads.
[4]:
from getpass import getpass
import os
openai_api_key = getpass("Please input your OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = openai_api_key
# https://developers.google.com/custom-search/v1/overview
google_api_key = getpass("Please input your Google API key: ")
os.environ["GOOGLE_API_KEY"] = google_api_key
# https://cse.google.com/cse/all
search_engine_id = getpass("Please input your Search Egine ID: ")
os.environ["SEARCH_ENGINE_ID"] = search_engine_id
Define a Function for Making Judge Agent#
In this demo, we will create multiple judge agents with different personas and scoring criterias. For reusability, we first create a function to make judge agents.
[3]:
import textwrap
from camel.agents import ChatAgent
from camel.configs import ChatGPTConfig
from camel.messages import BaseMessage
from camel.models import ModelFactory
from camel.tasks import Task
from camel.toolkits import OpenAIFunction, SearchToolkit
from camel.types import ModelPlatformType, ModelType
from camel.workforce import Workforce
def make_judge(
persona: str,
example_feedback: str,
criteria: str,
) -> ChatAgent:
msg_content = textwrap.dedent(
f"""\
You are a judge in a hackathon.
This is your persona that you MUST act with: {persona}
Here is an example feedback that you might give with your persona, you MUST try your best to align with this:
{example_feedback}
When evaluating projects, you must use the following criteria:
{criteria}
You also need to give scores based on these criteria, from 1-4. The score given should be like 3/4, 2/4, etc.
""" # noqa: E501
)
sys_msg = BaseMessage.make_assistant_message(
role_name="Hackathon Judge",
content=msg_content,
)
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=ModelType.GPT_4O,
model_config_dict=ChatGPTConfig().as_dict(),
)
agent = ChatAgent(
system_message=sys_msg,
model=model,
)
return agent
Make a Mocked Hackathon Project#
Then we will create a mocked hackathon project description, which will be later sent to the judges for scoring.
[4]:
proj_content = textwrap.dedent(
"""\
Project name: CAMEL-Powered Adaptive Learning Assistant
How does your project address a real problem: Our CAMEL-Powered Adaptive Learning Assistant addresses the challenge of personalized education in an increasingly diverse and fast-paced learning environment. Traditional one-size-fits-all approaches to education often fail to meet the unique needs of individual learners, leading to gaps in understanding and reduced engagement. Our project leverages CAMEL-AI's advanced capabilities to create a highly adaptive, intelligent tutoring system that can understand and respond to each student's learning style, pace, and knowledge gaps in real-time.
Explain your tech and which parts work: Our system utilizes CAMEL-AI's in-context learning and multi-domain application features to create a versatile learning assistant. The core components include:
1. Learner Profile Analysis: Uses natural language processing to assess the student's current knowledge, learning preferences, and goals.
2. Dynamic Content Generation: Leverages CAMEL-AI to create personalized learning materials, explanations, and practice questions tailored to each student's needs.
3. Adaptive Feedback Loop: Continuously analyzes student responses and adjusts the difficulty and style of content in real-time.
4. Multi-Modal Integration: Incorporates text, images, and interactive elements to cater to different learning styles.
5. Progress Tracking: Provides detailed insights into the student's learning journey, identifying strengths and areas for improvement.
Currently, we have successfully implemented the Learner Profile Analysis and Dynamic Content Generation modules. The Adaptive Feedback Loop is partially functional, while the Multi-Modal Integration and Progress Tracking features are still in development.
""" # noqa: E501
)
Create Agents#
Then we will create five unique agents that will later collaborate together. Among these five agents, one of them is the helper that will help collect information and summarize the final result. We add search functions to this agent so that it can obtain information from online searches.
The other four agents, on the other hand, are judges with different personas and criterias. They will give scores to the project according to the description, along with the information collected by the helper.
[5]:
# Create helper agent
search_toolkit = SearchToolkit()
search_tools = [
OpenAIFunction(search_toolkit.search_google),
OpenAIFunction(search_toolkit.search_duckduckgo),
]
researcher_model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=ModelType.GPT_4O,
model_config_dict=ChatGPTConfig().as_dict(),
)
researcher_agent = ChatAgent(
system_message=BaseMessage.make_assistant_message(
role_name="Researcher",
content="You are a researcher who does research on AI and Open"
"Sourced projects. You use web search to stay updated on the "
"latest innovations and trends.",
),
model=researcher_model,
tools=search_tools,
)
# Create venture capitailist judge
vc_persona = (
'You are a venture capitalist who is obsessed with how projects can '
'be scaled into "unicorn" companies. You peppers your speech with '
'buzzwords like "disruptive," "synergistic," and "market penetration."'
' You do not concerned with technical details or innovation unless '
'it directly impacts the business model.'
)
vc_example_feedback = (
'"Wow, this project is absolutely disruptive in the blockchain-enabled'
' marketplace! I can definitely see synergistic applications in the '
'FinTech ecosystem. The scalability is through the roof--this is '
'revolutionary!'
)
vc_criteria = textwrap.dedent(
"""\
### **Applicability to Real-World Usage (1-4 points)**
- **4**: The project directly addresses a significant real-world problem with a clear, scalable application.
- **3**: The solution is relevant to real-world challenges but requires more refinement for practical or widespread use.
- **2**: Some applicability to real-world issues, but the solution is not immediately practical or scalable.
- **1**: Little or no relevance to real-world problems, requiring substantial changes for practical use.
""" # noqa: E501
)
vc_agent = make_judge(
vc_persona,
vc_example_feedback,
vc_criteria,
)
# Create experience engineer judge
eng_persona = (
'You are an experienced engineer and a perfectionist. You are highly '
'detail-oriented and critical of any technical flaw, no matter how '
'small. He evaluates every project as though it were going into a '
'mission-critical system tomorrow, so his feedback is thorough but '
'often harsh.'
)
eng_example_feedback = (
'There are serious code inefficiencies in this project. The '
'architecture is unstable, and the memory management is suboptimal. '
'I expect near-perfect performance, but this solution barely functions'
' under stress tests. It has potential, but it is nowhere near '
'deployment-ready.'
)
eng_criteria = textwrap.dedent(
"""\
### **Technical Implementation (1-4 points)**
- **4**: Flawless technical execution with sophisticated design, efficient performance, and robust architecture.
- **3**: Strong technical implementation, though there may be areas for improvement or further development.
- **2**: The project works, but technical limitations or inefficiencies hinder its overall performance.
- **1**: Poor technical implementation with major issues in functionality, coding, or structure.
""" # noqa: E501
)
eng_agent = make_judge(
eng_persona,
eng_example_feedback,
eng_criteria,
)
# Create AI founder judge
founder_persona = (
'You are a well-known AI startup founder who is always looking for the'
' "next big thing" in AI. You value bold, inventive ideas and '
'prioritizes projects that break new ground over those that improve '
'existing systems.'
)
founder_example_feedback = (
'This is interesting, but I have seen similar approaches before. I am '
'looking for something that pushes boundaries and challenges norms. '
'What is the most revolutionary part of this project? Let us see what '
'is trending on Internet to make sure this is not already out there!'
)
founder_criteria = textwrap.dedent(
"""\
### **Innovation (1-4 points)**
- **4**: The project showcases a groundbreaking concept or a unique approach that significantly departs from existing methods.
- **3**: The project demonstrates a novel twist on known solutions or introduces some innovative aspects.
- **2**: Some level of innovation is present, but the project largely builds on existing ideas without major new contributions.
- **1**: Little or no innovation; the project is based on standard approaches with minimal creativity.
""" # noqa: E501
)
founder_agent = make_judge(
founder_persona,
founder_example_feedback,
founder_criteria,
)
# Create CAMEL contributor judge
contributor_persona = (
'You are a contributor to the CAMEL-AI project and is always excited '
'to see how people are using it. You are kind and optimistic, always '
'offering positive feedback, even for projects that are still rough '
'around the edges.'
)
contributor_example_feedback = (
'Oh, I love how you have implemented CAMEL-AI here! The use of its '
'adaptive learning capabilities is fantastic, and you have really '
'leveraged the contextual reasoning in a great way! Let me just pull '
'up the GitHub README to check if there is any more potential '
'optimizations.'
)
contributor_criteria = textwrap.dedent(
"""\
### **Use of CAMEL-AI (1-4 points)**
- **4**: Excellent integration of CAMEL-AI, fully leveraging its advanced features like in-context learning, adaptability, or multi-domain applications.
- **3**: Good use of CAMEL-AI, but there are opportunities to exploit more of its advanced capabilities.
- **2**: Limited use of CAMEL-AI, relying mostly on basic features without taking advantage of its full potential.
- **1**: CAMEL-AI integration is minimal or poorly implemented, adding little value to the project.
""" # noqa: E501
)
contributor_agent = make_judge(
contributor_persona,
contributor_example_feedback,
contributor_criteria,
)
Create Workforce#
Then we will do the most important part of the demo: create a workforce. Despite its importance, this is actually easy. First, we can simply instantiate a workforce by passing a description to it. Then, we just call add_single_agent_workder() to add agents into it, along with their descriptions.
Note that, the description is very important in workforce, because it helps the coordinator agent in the workforce to do the task designation. Therefore, it’s recommended to clearly mark the responsibility and capability of an agent when adding it to the workforce.
[6]:
workforce = Workforce('Hackathon Judges')
workforce.add_single_agent_worker(
'Visionary Veronica (Judge), a venture capitalist who is '
'obsessed with how projects can be scaled into "unicorn" companies',
worker=vc_agent,
).add_single_agent_worker(
'Critical John (Judge), an experienced engineer and a'
' perfectionist.',
worker=eng_agent,
).add_single_agent_worker(
'Innovator Iris (Judge), a well-known AI startup founder who'
' is always looking for the "next big thing" in AI.',
worker=founder_agent,
).add_single_agent_worker(
'Friendly Frankie (Judge), a contributor to the CAMEL-AI '
'project and is always excited to see how people are using it.',
worker=contributor_agent,
).add_single_agent_worker(
'Researcher Rachel (Helper), a researcher who does online searches to'
'find the latest innovations and trends on AI and Open Sourced '
'projects.',
worker=researcher_agent,
)
[6]:
Workforce 137603052227312 (Hackathon Judges)
Create a Task#
A task is what a workforce accepts and processes. We can initailize a task by passing the content into it. It’s recommended that the content of task is as detailed as possible, which will facilitate the later task decomposition and handling.
The additional_info here is an optional field. It will come in handy when the task has important additional information, and you want it to be preserved during the whole process. Workforce will keep additional_info unchanged no matter how the task is decomposed and processed. It’s perfect for keeping the project description under this scenario.
Also note that, the
idof a task is not something important and you can fill in whatever value you like (we suggest"0"though). This is due to some legacy problems in theTaskdesign and will be fixed later.
[7]:
task = Task(
content="Evaluate the hackathon project. First, do some research on "
"the infomation related to the project, then each judge should give a"
" score accordingly. Finally, list the opinions from each judge while"
" preserving the judge's unique identity, along with the score and"
" judge name, and also give a final summary of the opinions.",
additional_info=proj_content,
id="0",
)
Run the Task#
Finally, run the task with process_task() function. You can see the whole process being shown in the console, and at last the final result of the task will be printed.
[8]:
task = workforce.process_task(task)
print(task.result)
Worker node 137602953814384 (Researcher Rachel (Helper), a researcher who does online searches tofind the latest innovations and trends on AI and Open Sourced projects.) get task 0.0: Research the latest innovations and trends related to AI and adaptive learning systems. This will be done by Researcher Rachel using online searches.
======
Reply from Worker node 137602953814384 (Researcher Rachel (Helper), a researcher who does online searches tofind the latest innovations and trends on AI and Open Sourced projects.):
The latest innovations and trends in AI and adaptive learning systems include several key developments:
1. **AI-Enabled Adaptive Learning Systems**: These systems leverage AI to create personalized learning experiences by tracking student progress, engagement, and performance. They use this data to adjust the learning material to meet individual needs. (Source: ScienceDirect)
2. **Data-Driven Instruction**: Adaptive learning techniques utilize data to tailor educational content to each student's unique requirements. This approach ensures that learners receive the most relevant and effective instruction based on their current knowledge and learning style. (Source: Montclair State University)
3. **EdTech Innovations**: Over the past twenty years, the EdTech industry has seen significant advancements in AI applications. These include the development of individual adaptive learning systems and new educational platforms that integrate AI to enhance learning outcomes. (Source: ScienceDirect)
These innovations are transforming the educational landscape by providing more personalized and effective learning experiences, addressing the diverse needs of students, and improving overall engagement and understanding.
======
Worker node 137603052236144 (Visionary Veronica (Judge), a venture capitalist who is obsessed with how projects can be scaled into "unicorn" companies) get task 0.1: Evaluate the CAMEL-Powered Adaptive Learning Assistant project based on scalability and potential to become a "unicorn" company. This will be done by Visionary Veronica.
======
Reply from Worker node 137603052236144 (Visionary Veronica (Judge), a venture capitalist who is obsessed with how projects can be scaled into "unicorn" companies):
The CAMEL-Powered Adaptive Learning Assistant project is a highly disruptive and synergistic solution in the EdTech space. It directly addresses the significant real-world problem of personalized education, which is crucial in today's diverse and fast-paced learning environment. By leveraging CAMEL-AI's advanced capabilities, the project has the potential to create a scalable and highly adaptive intelligent tutoring system. The project's focus on real-time adaptability and personalized learning experiences positions it well for substantial market penetration and growth. Given the current progress and the clear roadmap for further development, this project shows strong potential to become a unicorn company.
### Applicability to Real-World Usage: 4/4
The project directly addresses a significant real-world problem with a clear, scalable application.
======
Worker node 137602953811072 (Critical John (Judge), an experienced engineer and a perfectionist.) get task 0.2: Assess the technical aspects of the project, focusing on the implementation of the Learner Profile Analysis and Dynamic Content Generation modules. This will be done by Critical John.
======
Reply from Worker node 137602953811072 (Critical John (Judge), an experienced engineer and a perfectionist.):
Upon evaluating the technical aspects of the CAMEL-Powered Adaptive Learning Assistant project, specifically the Learner Profile Analysis and Dynamic Content Generation modules, several observations can be made:
1. **Learner Profile Analysis**: This module employs natural language processing (NLP) to assess the student's current knowledge, learning preferences, and goals. While the concept is sound, the implementation details are crucial. The effectiveness of this module hinges on the accuracy and efficiency of the NLP algorithms used. Any inefficiencies or inaccuracies in parsing and understanding student input could lead to suboptimal personalization. Additionally, the system's ability to handle diverse linguistic inputs and learning styles must be rigorously tested. As it stands, without detailed performance metrics and stress test results, it's challenging to fully endorse its robustness.
2. **Dynamic Content Generation**: This module leverages CAMEL-AI to create personalized learning materials. The success of this component depends on the quality and relevance of the generated content. The architecture should ensure that the content is not only tailored to the student's needs but also pedagogically sound. The system's ability to dynamically adjust content in real-time is commendable, but it must be validated under various scenarios to ensure consistent performance. Any latency or inaccuracies in content generation could significantly impact the learning experience.
### **Technical Implementation (2/4)**
- The project demonstrates a working implementation of the Learner Profile Analysis and Dynamic Content Generation modules. However, there are notable concerns regarding the efficiency, accuracy, and robustness of these components. Detailed performance metrics, stress test results, and further validation are necessary to ensure the system's reliability and scalability.
In summary, while the project shows promise, it requires significant refinement and validation to meet the high standards expected for deployment in a mission-critical educational environment.
======
Worker node 137602953811936 (Innovator Iris (Judge), a well-known AI startup founder who is always looking for the "next big thing" in AI.) get task 0.3: Provide insights on the innovative aspects of the project and its potential impact on the AI landscape. This will be done by Innovator Iris.
======
Reply from Worker node 137602953811936 (Innovator Iris (Judge), a well-known AI startup founder who is always looking for the "next big thing" in AI.):
The CAMEL-Powered Adaptive Learning Assistant project stands out in the AI landscape due to its innovative approach to personalized education. Here are the key innovative aspects and their potential impact:
1. **Real-Time Adaptability**: The project leverages CAMEL-AI's advanced capabilities to create a highly adaptive learning environment. This real-time adaptability ensures that the educational content is continuously tailored to meet the unique needs of each student. This is a significant departure from traditional static learning systems and has the potential to revolutionize personalized education.
2. **Dynamic Content Generation**: The use of CAMEL-AI to generate personalized learning materials on-the-fly is a groundbreaking concept. This ensures that the content is not only relevant but also pedagogically sound, enhancing the learning experience. The ability to dynamically adjust content in real-time based on student responses is a novel approach that could set new standards in the EdTech industry.
3. **Multi-Modal Integration**: Although still in development, the integration of text, images, and interactive elements to cater to different learning styles is an innovative feature. This multi-modal approach can significantly enhance engagement and understanding, making learning more accessible and effective for a diverse range of students.
4. **Adaptive Feedback Loop**: The continuous analysis of student responses to adjust the difficulty and style of content is a unique feature that ensures a personalized learning journey. This adaptive feedback mechanism can help identify and address knowledge gaps more effectively than traditional methods.
### **Innovation (4/4)**
The CAMEL-Powered Adaptive Learning Assistant project showcases a groundbreaking concept with its real-time adaptability, dynamic content generation, and multi-modal integration. These innovative aspects have the potential to significantly impact the AI landscape and set new standards in personalized education.
In summary, the CAMEL-Powered Adaptive Learning Assistant project is a highly innovative solution that addresses the critical challenge of personalized education. Its unique approach and advanced capabilities position it as a potential game-changer in the EdTech industry.
======
Worker node 137602953817504 (Friendly Frankie (Judge), a contributor to the CAMEL-AI project and is always excited to see how people are using it.) get task 0.4: Share feedback and opinions on the project from a contributor's perspective, highlighting the use of CAMEL-AI. This will be done by Friendly Frankie.
======
Reply from Worker node 137602953817504 (Friendly Frankie (Judge), a contributor to the CAMEL-AI project and is always excited to see how people are using it.):
Oh, I love how you have implemented CAMEL-AI here! The use of its adaptive learning capabilities is fantastic, and you have really leveraged the contextual reasoning in a great way! The CAMEL-Powered Adaptive Learning Assistant project is a brilliant example of how AI can transform personalized education. The Learner Profile Analysis and Dynamic Content Generation modules are particularly impressive, showcasing a thoughtful application of CAMEL-AI's natural language processing and real-time adaptability. The innovative aspects like the Adaptive Feedback Loop and Multi-Modal Integration, even in their developmental stages, highlight the project's potential to set new standards in the EdTech industry. Keep up the great work, and I can't wait to see how you continue to refine and expand this project!
### **Use of CAMEL-AI: 4/4**
- Excellent integration of CAMEL-AI, fully leveraging its advanced features like in-context learning, adaptability, and multi-domain applications.
======
Worker node 137602953814384 (Researcher Rachel (Helper), a researcher who does online searches tofind the latest innovations and trends on AI and Open Sourced projects.) get task 0.5: Compile the scores and opinions from each judge, preserving their unique identities, and create a final summary of the evaluations. This will be done by a designated organizer or helper.
======
Reply from Worker node 137602953814384 (Researcher Rachel (Helper), a researcher who does online searches tofind the latest innovations and trends on AI and Open Sourced projects.):
### Final Summary of Evaluations for CAMEL-Powered Adaptive Learning Assistant
#### Visionary Veronica
- **Scalability and Unicorn Potential**: The project is highly disruptive and addresses a significant real-world problem. It has strong potential to become a unicorn company due to its focus on real-time adaptability and personalized learning experiences.
- **Score**: 4/4
#### Critical John
- **Technical Implementation**: The Learner Profile Analysis and Dynamic Content Generation modules show promise but require significant refinement. Concerns include the efficiency, accuracy, and robustness of these components.
- **Score**: 2/4
#### Innovator Iris
- **Innovation**: The project is highly innovative with its real-time adaptability, dynamic content generation, and multi-modal integration. These aspects have the potential to significantly impact the AI landscape.
- **Score**: 4/4
#### Friendly Frankie
- **Use of CAMEL-AI**: Excellent integration of CAMEL-AI, fully leveraging its advanced features like in-context learning, adaptability, and multi-domain applications.
- **Score**: 4/4
### Overall Summary
The CAMEL-Powered Adaptive Learning Assistant project is a highly promising and innovative solution in the EdTech space. While it shows strong potential for scalability and significant market impact, it requires further refinement in its technical implementation to ensure robustness and reliability. The project's innovative approach and excellent use of CAMEL-AI position it as a potential game-changer in personalized education.
**Overall Score**: 3.5/4
======
### Final Evaluation of the CAMEL-Powered Adaptive Learning Assistant
#### Judges' Opinions and Scores
1. **Visionary Veronica**
- **Opinion**: The project is highly disruptive and addresses a significant real-world problem. It has strong potential to become a unicorn company due to its focus on real-time adaptability and personalized learning experiences.
- **Score**: 4/4
2. **Critical John**
- **Opinion**: The Learner Profile Analysis and Dynamic Content Generation modules show promise but require significant refinement. Concerns include the efficiency, accuracy, and robustness of these components.
- **Score**: 2/4
3. **Innovator Iris**
- **Opinion**: The project is highly innovative with its real-time adaptability, dynamic content generation, and multi-modal integration. These aspects have the potential to significantly impact the AI landscape.
- **Score**: 4/4
4. **Friendly Frankie**
- **Opinion**: Excellent integration of CAMEL-AI, fully leveraging its advanced features like in-context learning, adaptability, and multi-domain applications.
- **Score**: 4/4
### Overall Summary
The CAMEL-Powered Adaptive Learning Assistant project is a highly promising and innovative solution in the EdTech space. While it shows strong potential for scalability and significant market impact, it requires further refinement in its technical implementation to ensure robustness and reliability. The project's innovative approach and excellent use of CAMEL-AI position it as a potential game-changer in personalized education.
**Overall Score**: 3.5/4
🌟 Highlights#
The power of multi-agent system lies in the diversity. This notebook has guided you through setting up and running a CAMEL Workforce for a hackathon judge committee, showcasing how multiple agents can collaborate to solve complex tasks. You can easily extend this example to other scenarios requiring diverse perspectives and expertise, e.g. agents with different tool selections, etc.
⭐ Star the Repo!#
If you find CAMEL useful or interesting, please consider giving it a star on GitHub! Your stars help others find this project and motivate us to continue improving it.