Agentic Data Generation, Evaluation & Filtering with Reward Models#
You can also check this cookbook in colab here
This notebook demonstrates how to set up and leverage CAMEL’s reward model to evaluate and filter synthetic data.
In this notebook, you’ll explore:
CAMEL: A powerful multi-agent framework that enables data synthesis, evaluation, and model training, as well as multi-agent role-playing scenarios, allowing for sophisticated AI-driven tasks.
CAMEL FireCrawl Reader: The Firecrawl loader encapsulated in CAMEL allows users to retrieve web information through Firecrawl.
Reward Model Module: A critical component designed to score and evaluate the quality of generated data based on predefined criteria. It supports fine-tuning of the evaluation process and ensures alignment with desired outcomes, making it an essential tool for filtering synthetic data effectively.
This cookbook demonstrates CAMEL serves as a flexible framework that can be adapted for various scenarios requiring evaluation, filtering, and optimization of AI-generated content.
📦 Installation#
First, install the CAMEL package with its dependencies
[ ]:
!pip install "camel-ai[web_tools]==0.2.16"
Next, we need to securely input and store the required API keys for accessing OpenAI, Firecrawl, and NVIDIA services.
[ ]:
from getpass import getpass
import os
openai_api_key = getpass('Enter your OpenAI API key: ')
os.environ["OPENAI_API_KEY"] = openai_api_key
# Generate an API key at https://www.firecrawl.dev/app/api-keys
firecrawl_api_key = getpass('Enter your Firecrawl API key: ')
os.environ["FIRECRAWL_API_KEY"] = firecrawl_api_key
# Generate an API key at https://build.nvidia.com/nvidia/nemotron-4-340b-reward
nvidia_api_key = getpass('Enter your NVIDIA API key: ')
os.environ["NVIDIA_API_KEY"] = nvidia_api_key
Enter your OpenAI API key: ··········
Enter your Firecrawl API key: ··········
Enter your NVIDIA API key: ··········
To work effectively with the Alpaca format and manage items systematically, we define a set of models using Pydantic. These models ensure that the data is well-structured, type-safe, and validated.
[ ]:
from pydantic import BaseModel
from camel.messages.conversion import AlpacaItem
class NumberedAlpacaItem(BaseModel):
number: int
item: AlpacaItem
class AlpacaItemResponse(BaseModel):
"""
Represents an instruction-response item in the Alpaca format.
"""
items: list[NumberedAlpacaItem]
🚀 Data Generation#
Next, we define our data generation function. It takes a source content and generates a list of instruction-input-response triplets based on it.
Later, we will use a reward model to filter this list.
[ ]:
from typing import List
from camel.models import ModelFactory
from camel.types import ModelPlatformType, ModelType
from camel.configs import ChatGPTConfig
from camel.agents import ChatAgent
def generate_alpaca_items(content: str, n_items: int, start_num: int = 1, examples: List[AlpacaItem] = None) -> List[AlpacaItem]:
system_msg = """
You are an AI assistant generating detailed, accurate responses based on the provided content.
You will be given a reference content, and you must generate a specific number of AlpacaItems.
These are instruction-input-response triplets, where the input is the context or examples.
Add a number to the items to keep track of the order. Generate exactly that many.
For each instruction, imagine but do not include a real world scenario and real user in that scenario to inform realistic and varied instructions. Avoid common sense questions and answers.
Include multiple lines in the output as appropriate to provide sufficient detail. Cite the most relevant context verbatim in output fields, do not omit anything important.
Leave the input field blank.
Ensure all of the most significant parts of the context are covered.
Start with open ended instructions, then move to more specific ones. Consider the starting number for an impression of what has already been generated.
"""
examples_str = ""
if examples:
examples_str = "\n\nHere are some example items for reference:\n" + \
"\n".join(ex.model_dump_json() for ex in examples)
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=ModelType.GPT_4O_MINI,
model_config_dict=ChatGPTConfig(
temperature=0.6, response_format=AlpacaItemResponse
).as_dict(),
)
agent = ChatAgent(
system_message=system_msg,
model=model,
)
prompt = f"Content reference:\n{content}{examples_str}\n\n Generate {n_items} AlpacaItems. The first should start numbering at {start_num}."
response = agent.step(prompt)
# Parse the generated JSON to our wrapper class
alpaca_items = [n_item.item for n_item in
AlpacaItemResponse.
model_validate_json(response.msgs[0].content).items]
return alpaca_items
# Few shot examples to ensure the right amount of detail
examples = [
AlpacaItem(
instruction="Explain the process for sprint planning and review in CAMEL.",
input="",
output="The process for sprint planning and review in CAMEL includes:\n1. **Sprint Duration**: Each sprint lasts two weeks for development and one week for review.\n2. **Planning Meeting**: Conducted biweekly, where the founder highlights the sprint goal and developers select items for the sprint.\n3. **Review Meeting**: Stakeholders review the delivered features and provide feedback on the work completed during the sprint."
)
]
📊 Point to content and generate data!#
Now we point to the content that we wish to generate SFT data around and use CAMEL’s Firecrawl integration to get this content in a nice markdown format.
[ ]:
import random
from camel.loaders.firecrawl_reader import Firecrawl
firecrawl = Firecrawl()
# Scrape and clean content from a specified URL
response = firecrawl.scrape(
url="https://github.com/camel-ai/camel/blob/master/CONTRIBUTING.md"
)
# Generate the items 50 a time up to 50
alpaca_entries = []
for start in range(1, 51, 50):
# Combine default examples with random samples from previous generations
current_examples = examples + (random.sample(alpaca_entries,
min(5, len(alpaca_entries)))
if alpaca_entries else [])
batch = generate_alpaca_items(
content=response["markdown"],
n_items=50,
start_num=start,
examples=current_examples
)
print(f"Generated {len(batch)} items")
alpaca_entries.extend(batch)
print(alpaca_entries)
Generated 50 items
[AlpacaItem(instruction='Summarize the contributing guidelines for the CAMEL project.', input='', output='The contributing guidelines for the CAMEL project include:\n1. **Types of Contributions**: Contributions can include new features, infrastructure enhancements, documentation improvements, issue reporting, and bug fixes.\n2. **Workflow**: Community contributors should follow the Fork-and-Pull-Request workflow, while members of CAMEL-AI.org should use the Checkout-and-Pull-Request workflow.\n3. **Pull Requests**: Mention related issues and tag relevant maintainers. Pull requests must pass formatting, linting, and testing checks before merging.'), AlpacaItem(instruction='What are the steps for writing a cookbook entry for CAMEL?', input='', output='The steps for writing a cookbook entry for CAMEL are:\n1. **Template Usage**: Use the provided template for consistency.\n2. **Writing in Colab**: Write your cookbook in Google Colab, ensuring that LLM-generated responses are correct and verified.\n3. **Clear Explanations**: Provide clear explanations of the code and its purpose.\n4. **Review and Submit**: After writing, review in Colab, then submit by downloading the notebook and creating a pull request.'), AlpacaItem(instruction='Describe the principles for writing docstrings in CAMEL.', input='', output='The principles for writing docstrings in CAMEL include:\n1. **Use Raw Strings**: Start with `r"""` to prevent issues with special characters.\n2. **Brief Descriptions**: Provide a concise summary of the functionality at the beginning.\n3. **Document Parameters**: Use an `Args` section to document parameters, including type and default values.'), AlpacaItem(instruction='What is the process for conducting code reviews in CAMEL?', input='', output='The process for conducting code reviews in CAMEL includes:\n1. **Reviewers Check**: Reviewers assess code for functionality, readability, and compliance with standards.\n2. **Feedback**: Leave constructive feedback where changes are necessary.\n3. **Approval**: The code must be approved by at least two reviewers before merging.'), AlpacaItem(instruction='Outline the steps for submitting a cookbook to CAMEL.', input='', output='The steps for submitting a cookbook to CAMEL are:\n1. **Finalize in Colab**: Ensure that the Colab notebook is complete and reviewed.\n2. **Download Notebook**: Download the finalized notebook as a .ipynb file.\n3. **Create Pull Request**: Open a pull request to add the .ipynb file to the repository, including any necessary documentation.'), AlpacaItem(instruction='Explain the importance of code coverage in CAMEL.', input='', output='Code coverage is important in CAMEL because it measures how much of the code is tested by unit tests. This helps in identifying robust areas of the codebase and areas that may need more testing, ensuring better reliability and maintainability of the code.'), AlpacaItem(instruction='What are the guidelines for labeling pull requests in CAMEL?', input='', output='The guidelines for labeling pull requests in CAMEL include:\n- **feat**: For new features.\n- **fix**: For bug fixes.\n- **docs**: For documentation updates.\n- **style**: For code style changes.\n- **refactor**: For code refactoring.\n- **test**: For adding or updating tests.\n- **chore**: For maintenance tasks.'), AlpacaItem(instruction='What communication channels are available for CAMEL contributors?', input='', output='The communication channels available for CAMEL contributors include:\n- **Discord**: For real-time communication.\n- **WeChat**: For Chinese-speaking contributors.\n- **Slack**: For team collaboration.'), AlpacaItem(instruction='Describe the purpose of the sprint planning process in CAMEL.', input='', output='The purpose of the sprint planning process in CAMEL is to define what can be delivered in the upcoming sprint and how those goals will be achieved. It ensures that the team has a clear understanding of the sprint objectives and tasks to focus on.'), AlpacaItem(instruction='What are the key components of the CAMEL project license?', input='', output='The CAMEL project is licensed under Apache 2.0, which means that contributions will also be licensed under the same terms by default. Contributors can add the license manually or use a script to automate the process.'), AlpacaItem(instruction='Summarize the guidelines for contributing to the CAMEL documentation.', input='', output='The guidelines for contributing to the CAMEL documentation include:\n1. **Comprehensive Coverage**: Provide thorough documentation for all classes and methods.\n2. **Sphinx Usage**: Documentation is generated automatically using Sphinx from the codebase.'), AlpacaItem(instruction='What steps should be taken to ensure reproducibility in the CAMEL cookbook?', input='', output='To ensure reproducibility in the CAMEL cookbook, contributors should:\n1. Write code that can be run and produces consistent results.\n2. Provide clear instructions for running the code.\n3. Validate all examples with real code execution.'), AlpacaItem(instruction='Explain the significance of logging over using print statements in CAMEL.', input='', output='Using logging instead of print statements in CAMEL is significant because it allows for more professional, configurable, and consistent logging. It helps in maintaining clean code and provides better control over the logging output.'), AlpacaItem(instruction='What is the process for updating dependencies in CAMEL?', input='', output='The process for updating dependencies in CAMEL includes:\n1. Modify the `pyproject.toml` file to add, update, or delete dependencies.\n2. Run `poetry lock` to synchronize the dependencies with the lock file.'), AlpacaItem(instruction='What are the responsibilities of a code reviewer in CAMEL?', input='', output='The responsibilities of a code reviewer in CAMEL include:\n1. Conduct timely reviews to keep the project moving.\n2. Provide clear and constructive feedback.\n3. Collaborate with contributors to ensure code meets project standards.'), AlpacaItem(instruction='Outline the steps to build documentation locally for CAMEL.', input='', output='To build documentation locally for CAMEL, follow these steps:\n1. Navigate to the `docs` directory.\n2. Run the command `make html` to generate the HTML documentation.'), AlpacaItem(instruction='What are the key principles to follow when writing entries for the CAMEL cookbook?', input='', output='The key principles for writing entries for the CAMEL cookbook include:\n1. High Quality: Ensure accurate code and clear explanations.\n2. Correctness: Validate LLM-generated responses with real code execution.\n3. Accessibility: Make content understandable for all users, from beginners to advanced.'), AlpacaItem(instruction='Describe how to run tests in CAMEL.', input='', output='To run tests in CAMEL, use the command `pytest .` in the project root directory. This will execute all tests, including those that require an OpenAI API key. For local isolated unit tests, use `pytest --fast-test-mode .`.'), AlpacaItem(instruction='What is the purpose of the code review checklist in CAMEL?', input='', output='The purpose of the code review checklist in CAMEL is to ensure that all contributions maintain high quality by evaluating functionality, code quality, design consistency, and documentation before merging into the main branch.'), AlpacaItem(instruction='Explain the process for scheduling an introduction call for new contributors in CAMEL.', input='', output='To schedule an introduction call for new contributors in CAMEL, visit the provided Calendly links:\n- For English speakers: Use the designated link for English.\n- For Chinese speakers: Use the designated link for Chinese.'), AlpacaItem(instruction='What are the main goals of the CAMEL project?', input='', output='The main goals of the CAMEL project include developing a robust open-source framework for AI applications, fostering community contributions, and ensuring high-quality documentation and testing.'), AlpacaItem(instruction="What is the significance of the 'Fork-and-Pull-Request' workflow in CAMEL?", input='', output="The 'Fork-and-Pull-Request' workflow in CAMEL is significant because it allows contributors to work on their own copy of the project and submit changes for review, ensuring that contributions are managed efficiently and collaboratively."), AlpacaItem(instruction='What should contributors do before their pull request can be merged?', input='', output='Before a pull request can be merged in CAMEL, it must pass formatting, linting, and testing checks. Contributors should also ensure that they have addressed any feedback from reviewers.'), AlpacaItem(instruction='Summarize the guidelines for writing clear and concise explanations in the CAMEL cookbook.', input='', output='The guidelines for writing clear and concise explanations in the CAMEL cookbook include:\n1. Use simple language to describe code functionality.\n2. Break down complex logic into understandable steps.\n3. Provide context for technical terms.'), AlpacaItem(instruction='What is the recommended way to provide feedback during code reviews in CAMEL?', input='', output='The recommended way to provide feedback during code reviews in CAMEL is to leave constructive comments that are clear and aimed at helping the contributor improve their code.'), AlpacaItem(instruction='What are the key components of the CAMEL community guidelines?', input='', output='The key components of the CAMEL community guidelines include promoting an inclusive environment, encouraging collaboration, and respecting diverse perspectives among contributors.'), AlpacaItem(instruction='Explain the significance of ensuring high-quality documentation in CAMEL.', input='', output='Ensuring high-quality documentation in CAMEL is significant because it helps users understand how to use the framework effectively, facilitates onboarding for new contributors, and promotes better collaboration within the community.'), AlpacaItem(instruction='What should contributors do if they encounter challenges during the setup process?', input='', output='If contributors encounter challenges during the setup process in CAMEL, they should reach out to a maintainer for assistance to ensure a smooth experience.'), AlpacaItem(instruction='Describe the process for merging pull requests in CAMEL.', input='', output='The process for merging pull requests in CAMEL involves:\n1. Ensuring the pull request has been approved by at least two reviewers.\n2. The merging should be done by a maintainer or an authorized contributor.'), AlpacaItem(instruction='What are the best practices for conducting a sprint review in CAMEL?', input='', output='Best practices for conducting a sprint review in CAMEL include:\n1. Gathering stakeholder feedback on delivered features.\n2. Identifying areas for improvement based on the work completed.'), AlpacaItem(instruction='What is the significance of using Google Colab for cookbook contributions?', input='', output='Using Google Colab for cookbook contributions is significant because it allows for interactive code execution alongside explanations, making it easier for users to understand and experiment with the code.'), AlpacaItem(instruction='What are the steps to create a new issue on the CAMEL GitHub repository?', input='', output="To create a new issue on the CAMEL GitHub repository, follow these steps:\n1. Go to the issues page on GitHub.\n2. Click on 'New Issue' and fill in the required information, including a proper title and description.\n3. Assign labels and update the assignees and milestones as necessary."), AlpacaItem(instruction="What should contributors do to maintain the project's coding standards?", input='', output="To maintain the project's coding standards, contributors should:\n1. Follow the established style guidelines.\n2. Use tools like Ruff for formatting and linting checks.\n3. Ensure documentation is up-to-date and comprehensive."), AlpacaItem(instruction='Explain the role of the CAMEL project maintainers.', input='', output='The role of the CAMEL project maintainers includes overseeing contributions, ensuring code quality, managing project direction, and facilitating communication within the community.'), AlpacaItem(instruction='What are the guidelines for submitting documentation updates in CAMEL?', input='', output='The guidelines for submitting documentation updates in CAMEL include:\n1. Ensure that updates are comprehensive and improve clarity.\n2. Follow the automated documentation generation process using Sphinx.'), AlpacaItem(instruction='What are the requirements for submitting a feature request in CAMEL?', input='', output='The requirements for submitting a feature request in CAMEL include providing a clear description of the feature, explaining its benefits, and linking to any relevant issues or discussions.'), AlpacaItem(instruction='Describe the importance of unit tests in the CAMEL project.', input='', output='Unit tests are important in the CAMEL project because they verify that individual components of the code work as expected, helping to catch bugs early and ensuring code reliability.'), AlpacaItem(instruction='What is the process for addressing feedback received during code reviews?', input='', output='The process for addressing feedback received during code reviews involves:\n1. Carefully reviewing the comments and suggestions provided by reviewers.\n2. Making necessary changes to the code based on the feedback.\n3. Updating the pull request to reflect the changes made.'), AlpacaItem(instruction='Summarize the steps for conducting a developer meeting in CAMEL.', input='', output='The steps for conducting a developer meeting in CAMEL include:\n1. Scheduling the meeting in advance and sharing the agenda.\n2. Discussing ongoing projects, challenges, and upcoming tasks.\n3. Gathering feedback and suggestions from team members.'), AlpacaItem(instruction='What should contributors include in their pull requests?', input='', output='Contributors should include the following in their pull requests:\n1. A clear title and description of the changes.\n2. Links to any related issues.\n3. Any necessary documentation or references.'), AlpacaItem(instruction='Explain the significance of following the pull request review process.', input='', output='Following the pull request review process is significant because it ensures that all contributions are evaluated for quality, functionality, and adherence to project standards, which helps maintain the integrity of the codebase.'), AlpacaItem(instruction='What are the best practices for writing unit tests in CAMEL?', input='', output='Best practices for writing unit tests in CAMEL include:\n1. Ensure tests cover all edge cases and expected outcomes.\n2. Keep tests isolated to avoid dependencies on other parts of the code.\n3. Use descriptive names for test functions to clarify their purpose.'), AlpacaItem(instruction='What is the process for integrating new features into the CAMEL project?', input='', output='The process for integrating new features into the CAMEL project involves:\n1. Submitting a feature request or proposal.\n2. Developing the feature in a separate branch.\n3. Creating a pull request for review once development is complete.'), AlpacaItem(instruction='Describe the importance of maintaining a clean and organized project structure in CAMEL.', input='', output='Maintaining a clean and organized project structure in CAMEL is important because it enhances code readability, facilitates easier navigation for contributors, and makes it simpler to manage and scale the project.'), AlpacaItem(instruction='What are the steps for conducting a retrospective after a sprint in CAMEL?', input='', output='The steps for conducting a retrospective after a sprint in CAMEL include:\n1. Gathering team feedback on what went well and what could be improved.\n2. Discussing challenges faced during the sprint.\n3. Identifying actionable items to implement in the next sprint.'), AlpacaItem(instruction='What should contributors do to ensure their code is secure?', input='', output='To ensure their code is secure, contributors should:\n1. Follow best practices for security in coding.\n2. Review code for potential vulnerabilities.\n3. Conduct security testing as part of the development process.'), AlpacaItem(instruction="What is the significance of the CAMEL community's code of conduct?", input='', output="The significance of the CAMEL community's code of conduct is to promote a respectful and inclusive environment for all contributors, ensuring that everyone feels welcome and valued while participating in the project."), AlpacaItem(instruction='Explain the relationship between CAMEL and its contributors.', input='', output='The relationship between CAMEL and its contributors is collaborative; contributors provide enhancements, fixes, and documentation, while CAMEL offers a platform for sharing knowledge and fostering community engagement.'), AlpacaItem(instruction='What are the benefits of participating in the CAMEL community?', input='', output='The benefits of participating in the CAMEL community include:\n1. Gaining experience in open-source development.\n2. Networking with other developers and contributors.\n3. Contributing to a project that aims to advance AI capabilities.'), AlpacaItem(instruction='Summarize the guidelines for writing clear commit messages in CAMEL.', input='', output="The guidelines for writing clear commit messages in CAMEL include:\n1. Use the imperative mood to describe the changes (e.g., 'Add feature' instead of 'Added feature').\n2. Keep messages concise and focused on the changes made.\n3. Include references to related issues when applicable.")]
[ ]:
len(alpaca_entries)
50
🔄 Code for Conversion to Reward Model Format#
Next, we transform the Alpaca-style entries into a format compatible with the reward model. Each entry will be converted into a structured list of instruction-input-response pairs that the reward model can evaluate.
[ ]:
messages_lists=[]
for item in alpaca_entries:
messages_list =[]
user_content = item.instruction
if item.input:
user_content += f"\nInput: {item.input}"
messages_list.append({"role": "user", "content": user_content})
messages_list.append({"role": "assistant", "content": item.output})
messages_lists.append(messages_list)
✨Test Reward Model#
Then, we can test the reward model to check its output format and use it as a reference to set the filtering criteria.
[ ]:
from camel.models.reward import Evaluator, NemotronRewardModel
from camel.types import ModelType
reward_model = NemotronRewardModel(
model_type=ModelType.NVIDIA_NEMOTRON_340B_REWARD,
url="https://integrate.api.nvidia.com/v1",
)
evaluator = Evaluator(reward_model=reward_model)
results = [] # To store results for comparison
for i in range(min(10, len(messages_lists))):
print(f"Evaluating message list {i+1}:")
print(messages_lists[i]) # Display the message list
scores = evaluator.evaluate(messages_lists[i])
print(f"Scores: {scores}\n") # Print the evaluation scores
results.append((i + 1, messages_lists[i], scores))
# Print a summary of the results
print("\nSummary of evaluations:")
for index, messages, scores in results:
print(f"Message List {index}:")
print(messages)
print(f"Scores: {scores}\n")
Evaluating message list 1:
[{'role': 'user', 'content': 'Summarize the contributing guidelines for the CAMEL project.'}, {'role': 'assistant', 'content': 'The contributing guidelines for the CAMEL project include:\n1. **Types of Contributions**: Contributions can include new features, infrastructure enhancements, documentation improvements, issue reporting, and bug fixes.\n2. **Workflow**: Community contributors should follow the Fork-and-Pull-Request workflow, while members of CAMEL-AI.org should use the Checkout-and-Pull-Request workflow.\n3. **Pull Requests**: Mention related issues and tag relevant maintainers. Pull requests must pass formatting, linting, and testing checks before merging.'}]
Scores: {'helpfulness': 2.21875, 'correctness': 2.40625, 'coherence': 3.453125, 'complexity': 1.6640625, 'verbosity': 1.078125}
Evaluating message list 2:
[{'role': 'user', 'content': 'What are the steps for writing a cookbook entry for CAMEL?'}, {'role': 'assistant', 'content': 'The steps for writing a cookbook entry for CAMEL are:\n1. **Template Usage**: Use the provided template for consistency.\n2. **Writing in Colab**: Write your cookbook in Google Colab, ensuring that LLM-generated responses are correct and verified.\n3. **Clear Explanations**: Provide clear explanations of the code and its purpose.\n4. **Review and Submit**: After writing, review in Colab, then submit by downloading the notebook and creating a pull request.'}]
Scores: {'helpfulness': 2.71875, 'correctness': 2.734375, 'coherence': 3.46875, 'complexity': 1.7109375, 'verbosity': 1.203125}
Evaluating message list 3:
[{'role': 'user', 'content': 'Describe the principles for writing docstrings in CAMEL.'}, {'role': 'assistant', 'content': 'The principles for writing docstrings in CAMEL include:\n1. **Use Raw Strings**: Start with `r"""` to prevent issues with special characters.\n2. **Brief Descriptions**: Provide a concise summary of the functionality at the beginning.\n3. **Document Parameters**: Use an `Args` section to document parameters, including type and default values.'}]
Scores: {'helpfulness': 1.5546875, 'correctness': 1.890625, 'coherence': 3.015625, 'complexity': 1.65625, 'verbosity': 0.6484375}
Evaluating message list 4:
[{'role': 'user', 'content': 'What is the process for conducting code reviews in CAMEL?'}, {'role': 'assistant', 'content': 'The process for conducting code reviews in CAMEL includes:\n1. **Reviewers Check**: Reviewers assess code for functionality, readability, and compliance with standards.\n2. **Feedback**: Leave constructive feedback where changes are necessary.\n3. **Approval**: The code must be approved by at least two reviewers before merging.'}]
Scores: {'helpfulness': 2.34375, 'correctness': 2.546875, 'coherence': 3.59375, 'complexity': 1.3984375, 'verbosity': 1.0625}
Evaluating message list 5:
[{'role': 'user', 'content': 'Outline the steps for submitting a cookbook to CAMEL.'}, {'role': 'assistant', 'content': 'The steps for submitting a cookbook to CAMEL are:\n1. **Finalize in Colab**: Ensure that the Colab notebook is complete and reviewed.\n2. **Download Notebook**: Download the finalized notebook as a .ipynb file.\n3. **Create Pull Request**: Open a pull request to add the .ipynb file to the repository, including any necessary documentation.'}]
Scores: {'helpfulness': 2.34375, 'correctness': 2.625, 'coherence': 3.546875, 'complexity': 1.46875, 'verbosity': 1.046875}
Evaluating message list 6:
[{'role': 'user', 'content': 'Explain the importance of code coverage in CAMEL.'}, {'role': 'assistant', 'content': 'Code coverage is important in CAMEL because it measures how much of the code is tested by unit tests. This helps in identifying robust areas of the codebase and areas that may need more testing, ensuring better reliability and maintainability of the code.'}]
Scores: {'helpfulness': 2.78125, 'correctness': 2.875, 'coherence': 3.8125, 'complexity': 1.7734375, 'verbosity': 1.0234375}
Evaluating message list 7:
[{'role': 'user', 'content': 'What are the guidelines for labeling pull requests in CAMEL?'}, {'role': 'assistant', 'content': 'The guidelines for labeling pull requests in CAMEL include:\n- **feat**: For new features.\n- **fix**: For bug fixes.\n- **docs**: For documentation updates.\n- **style**: For code style changes.\n- **refactor**: For code refactoring.\n- **test**: For adding or updating tests.\n- **chore**: For maintenance tasks.'}]
Scores: {'helpfulness': 2.859375, 'correctness': 2.984375, 'coherence': 3.78125, 'complexity': 1.6171875, 'verbosity': 1.1328125}
Evaluating message list 8:
[{'role': 'user', 'content': 'What communication channels are available for CAMEL contributors?'}, {'role': 'assistant', 'content': 'The communication channels available for CAMEL contributors include:\n- **Discord**: For real-time communication.\n- **WeChat**: For Chinese-speaking contributors.\n- **Slack**: For team collaboration.'}]
Scores: {'helpfulness': 1.4296875, 'correctness': 1.6796875, 'coherence': 3.59375, 'complexity': 0.796875, 'verbosity': 0.546875}
Evaluating message list 9:
[{'role': 'user', 'content': 'Describe the purpose of the sprint planning process in CAMEL.'}, {'role': 'assistant', 'content': 'The purpose of the sprint planning process in CAMEL is to define what can be delivered in the upcoming sprint and how those goals will be achieved. It ensures that the team has a clear understanding of the sprint objectives and tasks to focus on.'}]
Scores: {'helpfulness': 2.84375, 'correctness': 2.984375, 'coherence': 3.890625, 'complexity': 1.6015625, 'verbosity': 0.95703125}
Evaluating message list 10:
[{'role': 'user', 'content': 'What are the key components of the CAMEL project license?'}, {'role': 'assistant', 'content': 'The CAMEL project is licensed under Apache 2.0, which means that contributions will also be licensed under the same terms by default. Contributors can add the license manually or use a script to automate the process.'}]
Scores: {'helpfulness': 1.890625, 'correctness': 1.828125, 'coherence': 3.5625, 'complexity': 1.3515625, 'verbosity': 0.65625}
Summary of evaluations:
Message List 1:
[{'role': 'user', 'content': 'Summarize the contributing guidelines for the CAMEL project.'}, {'role': 'assistant', 'content': 'The contributing guidelines for the CAMEL project include:\n1. **Types of Contributions**: Contributions can include new features, infrastructure enhancements, documentation improvements, issue reporting, and bug fixes.\n2. **Workflow**: Community contributors should follow the Fork-and-Pull-Request workflow, while members of CAMEL-AI.org should use the Checkout-and-Pull-Request workflow.\n3. **Pull Requests**: Mention related issues and tag relevant maintainers. Pull requests must pass formatting, linting, and testing checks before merging.'}]
Scores: {'helpfulness': 2.21875, 'correctness': 2.40625, 'coherence': 3.453125, 'complexity': 1.6640625, 'verbosity': 1.078125}
Message List 2:
[{'role': 'user', 'content': 'What are the steps for writing a cookbook entry for CAMEL?'}, {'role': 'assistant', 'content': 'The steps for writing a cookbook entry for CAMEL are:\n1. **Template Usage**: Use the provided template for consistency.\n2. **Writing in Colab**: Write your cookbook in Google Colab, ensuring that LLM-generated responses are correct and verified.\n3. **Clear Explanations**: Provide clear explanations of the code and its purpose.\n4. **Review and Submit**: After writing, review in Colab, then submit by downloading the notebook and creating a pull request.'}]
Scores: {'helpfulness': 2.71875, 'correctness': 2.734375, 'coherence': 3.46875, 'complexity': 1.7109375, 'verbosity': 1.203125}
Message List 3:
[{'role': 'user', 'content': 'Describe the principles for writing docstrings in CAMEL.'}, {'role': 'assistant', 'content': 'The principles for writing docstrings in CAMEL include:\n1. **Use Raw Strings**: Start with `r"""` to prevent issues with special characters.\n2. **Brief Descriptions**: Provide a concise summary of the functionality at the beginning.\n3. **Document Parameters**: Use an `Args` section to document parameters, including type and default values.'}]
Scores: {'helpfulness': 1.5546875, 'correctness': 1.890625, 'coherence': 3.015625, 'complexity': 1.65625, 'verbosity': 0.6484375}
Message List 4:
[{'role': 'user', 'content': 'What is the process for conducting code reviews in CAMEL?'}, {'role': 'assistant', 'content': 'The process for conducting code reviews in CAMEL includes:\n1. **Reviewers Check**: Reviewers assess code for functionality, readability, and compliance with standards.\n2. **Feedback**: Leave constructive feedback where changes are necessary.\n3. **Approval**: The code must be approved by at least two reviewers before merging.'}]
Scores: {'helpfulness': 2.34375, 'correctness': 2.546875, 'coherence': 3.59375, 'complexity': 1.3984375, 'verbosity': 1.0625}
Message List 5:
[{'role': 'user', 'content': 'Outline the steps for submitting a cookbook to CAMEL.'}, {'role': 'assistant', 'content': 'The steps for submitting a cookbook to CAMEL are:\n1. **Finalize in Colab**: Ensure that the Colab notebook is complete and reviewed.\n2. **Download Notebook**: Download the finalized notebook as a .ipynb file.\n3. **Create Pull Request**: Open a pull request to add the .ipynb file to the repository, including any necessary documentation.'}]
Scores: {'helpfulness': 2.34375, 'correctness': 2.625, 'coherence': 3.546875, 'complexity': 1.46875, 'verbosity': 1.046875}
Message List 6:
[{'role': 'user', 'content': 'Explain the importance of code coverage in CAMEL.'}, {'role': 'assistant', 'content': 'Code coverage is important in CAMEL because it measures how much of the code is tested by unit tests. This helps in identifying robust areas of the codebase and areas that may need more testing, ensuring better reliability and maintainability of the code.'}]
Scores: {'helpfulness': 2.78125, 'correctness': 2.875, 'coherence': 3.8125, 'complexity': 1.7734375, 'verbosity': 1.0234375}
Message List 7:
[{'role': 'user', 'content': 'What are the guidelines for labeling pull requests in CAMEL?'}, {'role': 'assistant', 'content': 'The guidelines for labeling pull requests in CAMEL include:\n- **feat**: For new features.\n- **fix**: For bug fixes.\n- **docs**: For documentation updates.\n- **style**: For code style changes.\n- **refactor**: For code refactoring.\n- **test**: For adding or updating tests.\n- **chore**: For maintenance tasks.'}]
Scores: {'helpfulness': 2.859375, 'correctness': 2.984375, 'coherence': 3.78125, 'complexity': 1.6171875, 'verbosity': 1.1328125}
Message List 8:
[{'role': 'user', 'content': 'What communication channels are available for CAMEL contributors?'}, {'role': 'assistant', 'content': 'The communication channels available for CAMEL contributors include:\n- **Discord**: For real-time communication.\n- **WeChat**: For Chinese-speaking contributors.\n- **Slack**: For team collaboration.'}]
Scores: {'helpfulness': 1.4296875, 'correctness': 1.6796875, 'coherence': 3.59375, 'complexity': 0.796875, 'verbosity': 0.546875}
Message List 9:
[{'role': 'user', 'content': 'Describe the purpose of the sprint planning process in CAMEL.'}, {'role': 'assistant', 'content': 'The purpose of the sprint planning process in CAMEL is to define what can be delivered in the upcoming sprint and how those goals will be achieved. It ensures that the team has a clear understanding of the sprint objectives and tasks to focus on.'}]
Scores: {'helpfulness': 2.84375, 'correctness': 2.984375, 'coherence': 3.890625, 'complexity': 1.6015625, 'verbosity': 0.95703125}
Message List 10:
[{'role': 'user', 'content': 'What are the key components of the CAMEL project license?'}, {'role': 'assistant', 'content': 'The CAMEL project is licensed under Apache 2.0, which means that contributions will also be licensed under the same terms by default. Contributors can add the license manually or use a script to automate the process.'}]
Scores: {'helpfulness': 1.890625, 'correctness': 1.828125, 'coherence': 3.5625, 'complexity': 1.3515625, 'verbosity': 0.65625}
# 🎯Filtering the Generated Data Using the Reward Model Finally, we utilize NVIDIA’s Nemotron Reward Model to filter out low-quality instruction-input-response triplets. The model evaluates each response based on defined thresholds for metrics such as helpfulness and correctness.
Let’s use thresholds = {“helpfulness”: 2.5, “correctness”: 2.5} as an example of filter parameters. After filtering, some high-quality triplets are retained.
[ ]:
thresholds = {"helpfulness": 2.5, "correctness": 2.5}
filtered_messages_lists = []
for messages_list in messages_lists:
response = evaluator.filter_data(messages_list, thresholds)
if response:
filtered_messages_lists.append(messages_list)
print(len(filtered_messages_lists))
18
[ ]:
filtered_messages_lists
[[{'role': 'user',
'content': 'What are the steps for writing a cookbook entry for CAMEL?'},
{'role': 'assistant',
'content': 'The steps for writing a cookbook entry for CAMEL are:\n1. **Template Usage**: Use the provided template for consistency.\n2. **Writing in Colab**: Write your cookbook in Google Colab, ensuring that LLM-generated responses are correct and verified.\n3. **Clear Explanations**: Provide clear explanations of the code and its purpose.\n4. **Review and Submit**: After writing, review in Colab, then submit by downloading the notebook and creating a pull request.'}],
[{'role': 'user',
'content': 'Explain the importance of code coverage in CAMEL.'},
{'role': 'assistant',
'content': 'Code coverage is important in CAMEL because it measures how much of the code is tested by unit tests. This helps in identifying robust areas of the codebase and areas that may need more testing, ensuring better reliability and maintainability of the code.'}],
[{'role': 'user',
'content': 'What are the guidelines for labeling pull requests in CAMEL?'},
{'role': 'assistant',
'content': 'The guidelines for labeling pull requests in CAMEL include:\n- **feat**: For new features.\n- **fix**: For bug fixes.\n- **docs**: For documentation updates.\n- **style**: For code style changes.\n- **refactor**: For code refactoring.\n- **test**: For adding or updating tests.\n- **chore**: For maintenance tasks.'}],
[{'role': 'user',
'content': 'Describe the purpose of the sprint planning process in CAMEL.'},
{'role': 'assistant',
'content': 'The purpose of the sprint planning process in CAMEL is to define what can be delivered in the upcoming sprint and how those goals will be achieved. It ensures that the team has a clear understanding of the sprint objectives and tasks to focus on.'}],
[{'role': 'user',
'content': 'What are the key principles to follow when writing entries for the CAMEL cookbook?'},
{'role': 'assistant',
'content': 'The key principles for writing entries for the CAMEL cookbook include:\n1. High Quality: Ensure accurate code and clear explanations.\n2. Correctness: Validate LLM-generated responses with real code execution.\n3. Accessibility: Make content understandable for all users, from beginners to advanced.'}],
[{'role': 'user',
'content': 'What is the purpose of the code review checklist in CAMEL?'},
{'role': 'assistant',
'content': 'The purpose of the code review checklist in CAMEL is to ensure that all contributions maintain high quality by evaluating functionality, code quality, design consistency, and documentation before merging into the main branch.'}],
[{'role': 'user',
'content': "What is the significance of the 'Fork-and-Pull-Request' workflow in CAMEL?"},
{'role': 'assistant',
'content': "The 'Fork-and-Pull-Request' workflow in CAMEL is significant because it allows contributors to work on their own copy of the project and submit changes for review, ensuring that contributions are managed efficiently and collaboratively."}],
[{'role': 'user',
'content': 'What should contributors do if they encounter challenges during the setup process?'},
{'role': 'assistant',
'content': 'If contributors encounter challenges during the setup process in CAMEL, they should reach out to a maintainer for assistance to ensure a smooth experience.'}],
[{'role': 'user',
'content': 'What are the steps to create a new issue on the CAMEL GitHub repository?'},
{'role': 'assistant',
'content': "To create a new issue on the CAMEL GitHub repository, follow these steps:\n1. Go to the issues page on GitHub.\n2. Click on 'New Issue' and fill in the required information, including a proper title and description.\n3. Assign labels and update the assignees and milestones as necessary."}],
[{'role': 'user',
'content': "What should contributors do to maintain the project's coding standards?"},
{'role': 'assistant',
'content': "To maintain the project's coding standards, contributors should:\n1. Follow the established style guidelines.\n2. Use tools like Ruff for formatting and linting checks.\n3. Ensure documentation is up-to-date and comprehensive."}],
[{'role': 'user',
'content': 'Describe the importance of unit tests in the CAMEL project.'},
{'role': 'assistant',
'content': 'Unit tests are important in the CAMEL project because they verify that individual components of the code work as expected, helping to catch bugs early and ensuring code reliability.'}],
[{'role': 'user',
'content': 'What should contributors include in their pull requests?'},
{'role': 'assistant',
'content': 'Contributors should include the following in their pull requests:\n1. A clear title and description of the changes.\n2. Links to any related issues.\n3. Any necessary documentation or references.'}],
[{'role': 'user',
'content': 'Explain the significance of following the pull request review process.'},
{'role': 'assistant',
'content': 'Following the pull request review process is significant because it ensures that all contributions are evaluated for quality, functionality, and adherence to project standards, which helps maintain the integrity of the codebase.'}],
[{'role': 'user',
'content': 'Describe the importance of maintaining a clean and organized project structure in CAMEL.'},
{'role': 'assistant',
'content': 'Maintaining a clean and organized project structure in CAMEL is important because it enhances code readability, facilitates easier navigation for contributors, and makes it simpler to manage and scale the project.'}],
[{'role': 'user',
'content': "What is the significance of the CAMEL community's code of conduct?"},
{'role': 'assistant',
'content': "The significance of the CAMEL community's code of conduct is to promote a respectful and inclusive environment for all contributors, ensuring that everyone feels welcome and valued while participating in the project."}],
[{'role': 'user',
'content': 'Explain the relationship between CAMEL and its contributors.'},
{'role': 'assistant',
'content': 'The relationship between CAMEL and its contributors is collaborative; contributors provide enhancements, fixes, and documentation, while CAMEL offers a platform for sharing knowledge and fostering community engagement.'}],
[{'role': 'user',
'content': 'What are the benefits of participating in the CAMEL community?'},
{'role': 'assistant',
'content': 'The benefits of participating in the CAMEL community include:\n1. Gaining experience in open-source development.\n2. Networking with other developers and contributors.\n3. Contributing to a project that aims to advance AI capabilities.'}],
[{'role': 'user',
'content': 'Summarize the guidelines for writing clear commit messages in CAMEL.'},
{'role': 'assistant',
'content': "The guidelines for writing clear commit messages in CAMEL include:\n1. Use the imperative mood to describe the changes (e.g., 'Add feature' instead of 'Added feature').\n2. Keep messages concise and focused on the changes made.\n3. Include references to related issues when applicable."}]]
🌟 Highlights#
That’s everything: Got questions about 🐫 CAMEL-AI? Join us on Discord! Whether you want to share feedback, explore the latest in multi-agent systems, get support, or connect with others on exciting projects, we’d love to have you in the community! 🤝
Summary of This Cookbook: In this cookbook, we demonstrated how to leverage CAMEL-AI to filter generate data. This practical guide helps you efficiently evaluate synthetic data.
Check out some of our other work:
🐫 Creating Your First CAMEL Agent free Colab
🔥 3 ways to ingest data from websites with Firecrawl & CAMEL free Colab
🦥 Agentic SFT Data Generation with CAMEL and Mistral Models, Fine-Tuned with Unsloth free Colab
Thanks from everyone at 🐫 CAMEL-AI