Skip to content

Datapool

Go to Datapool

The Datapool is a module of the BotCity Orchestrator that allows for efficient management of batch item processing.

Diagram comparing automation with and without Datapool. Without Datapool: a single runner processes items sequentially with hardcoded error handling and no parallel execution. With Datapool: the BotCity Orchestrator distributes items across multiple runners in parallel, with dynamic settings, auto retry, and auto stop.

Through the Datapool, you can have control and granularity over the queue of items that need to be processed, making it an essential resource for automations that process a large volume of data.

Screenshot of a Datapool overview. At the top, general information: Datapool details including its status, total number of items, average time per item; Items in Processing including Pending Items, Processed Items, and Completed Items; Exception cases including Error Items, Items on hold, and Cancelled Items. Below, search bar, filters, and Action buttons: Add Items, Import CSV, and bulk actions. At the bottom is the Datapool Items section, displaying a table with the fields Entry, Priority, Status, Termination Message, Error Type, Auto-retry, Processing Time, Lifecycle of each item.

Overview

Key Features

  • Parallel Item Processing: Allows multiple tasks to concurrently consume items from a single queue.
  • Automatic Reprocessing: Automatically re-queues items processed with errors for reprocessing.
  • Data Standardization: Schema Possibility of creating a pre-defined structure for items, defining the expected fields for each item added.
  • Task Triggering: Automatic task triggering for execution, based on the creation of new items in the queue.

Key Benefits

  • Complexity Abstraction: Eliminates the need to implement complex logic in the code, as it has native handling for concurrent consumption and reprocessing scenarios.
  • Centralized Management: Offers centralized management directly in the BotCity Orchestrator, facilitating control and monitoring, and is already integrated with other platform resources. Eliminates the need to depend on external databases and auxiliary tools.
  • Simplified Integrations: You can read from any database and transform the data into items in the Datapool, adding them to the processing queue through APIs, SDKs, and .csv files. Easily extract processing data and integrate with your data visualization platforms.

Diagram showing the Datapool workflow: items are sent one by one to an Automation runner for processing, then tracked individually with Done or Error status. Each item generates Logs, Alerts, Status, Result Files, and Errors.

Use Cases

  • Batch Item Processing: Ideal for automations that need to process a large volume of data or batches of items, such as registrations, updates, and queries.
  • Producer/Consumer Scenarios: An excellent alternative for scenarios where one automation generates demand by creating new items in the Datapool, and another automation consumes the queue and performs the processing flow of these items.