Merge pull request #69 from powerloom/feat/snapshotter-cli

Add Snapshotter CLI Documentation

Author: SwaroopH

docs/Protocol/Protocol_v2/Sequencer/EventCollector.md

@@ -45,7 +45,7 @@ When a match is identified, it indicates the end of the submission window, promp
 ### Batch Submission
 After the batches are organized, they are iterated over, and the relevant data is transformed into submission details. These details are then placed into the Finalizer Queue for further processing.
 
-The [**Finalizer**](https://github.com/PowerLoom/submission-sequencer-finalizer), an auto-scaled component that follows, retrieves and processes data from the Finalizer Queue, completing the batch processing pipeline. This architecture enables the system to scale effectively, managing submission tasks in parallel and optimizing both performance and throughput.
+The [**Finalizer**](https://github.com/powerloom/submission-sequencer-finalizer), an auto-scaled component that follows, retrieves and processes data from the Finalizer Queue, completing the batch processing pipeline. This architecture enables the system to scale effectively, managing submission tasks in parallel and optimizing both performance and throughput.
 
 ## On-Chain Updates via Relayer
 

docs/Protocol/Protocol_v2/relay.md

@@ -9,7 +9,7 @@ Circuit relayers are currently not being supported for the pre-mainnet and mainn
 
 The trusted list of sequeuncer interfaces can be found on our Github repository in this JSON file:
 
-https://github.com/PowerLoom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json
+https://github.com/powerloom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json
 :::
 
 ## Recap

docs/Protocol/Protocol_v2/sequencer.md

@@ -8,7 +8,7 @@ sidebar_position: 1
 :::warning
 The sequencer listening interfaces for specific data markets are listed in the following trusted sequencer JSON file hosted on the Powerloom Github repository:
 
-https://github.com/PowerLoom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json
+https://github.com/powerloom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json
 
 DO NOT attempt to connect to any other sequencer interfaces supplied by anyone claiming to represent Powerloom if they are not listed in the above file.
 :::
@@ -100,7 +100,7 @@ The libp2p listening interface is loadbalanced on Layer 4 of the networking stac
 
 :::info
 * Read more: [Sequencer: Libp2p Listener](/Protocol/Protocol_v2/Sequencer/Listener.md)
-* [Github](https://github.com/PowerLoom/libp2p-submission-sequencer-listener)
+* [Github](https://github.com/powerloom/libp2p-submission-sequencer-listener)
 :::
 
 ### Autoscaled dequeuers
@@ -109,7 +109,7 @@ These dequeuers are autoscaled by an event driven architecture and intermediated
 
 :::info
 * Read more: [Sequencer: Dequeuer](/Protocol/Protocol_v2/Sequencer/Dequeuer.md)
-* [Github](https://github.com/PowerLoom/sequencer-dequeuer)
+* [Github](https://github.com/powerloom/sequencer-dequeuer)
 :::
 
 ### Event collector
@@ -126,7 +126,7 @@ This works as the system clock of the sequencer setup as a whole.
 
 :::info
 * Read more: [Sequencer: Event Collector](/Protocol/Protocol_v2/Sequencer/EventCollector.md)
-* [Github](https://github.com/PowerLoom/submission-sequencer-event-collector/)
+* [Github](https://github.com/powerloom/submission-sequencer-event-collector/)
 :::
 
 ### Autsocaled Batch Finalizers
@@ -138,5 +138,5 @@ These are another set of autoscaled pods that are responsible for
 
 :::info
 * Read more: [Sequencer: Batch Finalizer](/Protocol/Protocol_v2/Sequencer/Finalizer.md)
-* [Github](https://github.com/PowerLoom/submission-sequencer-batch-finalizer/)
+* [Github](https://github.com/powerloom/submission-sequencer-batch-finalizer/)
 :::

docs/Protocol/Specifications/Epoch.md

@@ -65,4 +65,4 @@ The Force Consensus service operates slightly differently than the Epoch Generat
 
 ![Force consensus mechanism](https://raw.githubusercontent.com/PowerLoom/onchain-consensus/feat/force_consensus_only_relevant_projects/docs/images/force_consensus.png)
 
-- [Epoch Generator Source Code on Github](https://github.com/Powerloom/onchain-consensus/blob/63d09aa9ab1d98a2fed55e05b7760c12692fea83/epoch_generator.py)
+- [Epoch Generator Source Code on Github](https://github.com/powerloom/onchain-consensus/blob/63d09aa9ab1d98a2fed55e05b7760c12692fea83/epoch_generator.py)

docs/Protocol/Specifications/Snapshotter/components.md

@@ -12,19 +12,19 @@ The System Event Detector tracks events triggered on the protocol state contract
 
 ## Process Hub Core
 
-The Process Hub Core, defined in [`process_hub_core.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/process_hub_core.py), serves as the primary process manager in the snapshotter.
-- Operated by the CLI tool [`processhub_cmd.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processhub_cmd.py), it is responsible for starting and managing the `SystemEventDetector` and `ProcessorDistributor` processes.
-- Additionally, it spawns the base snapshot and aggregator workers required for processing tasks from the `powerloom-backend-callback` queue. The number of workers and their configuration path can be adjusted in [`config/settings.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json).
+The Process Hub Core, defined in [`process_hub_core.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/process_hub_core.py), serves as the primary process manager in the snapshotter.
+- Operated by the CLI tool [`processhub_cmd.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processhub_cmd.py), it is responsible for starting and managing the `SystemEventDetector` and `ProcessorDistributor` processes.
+- Additionally, it spawns the base snapshot and aggregator workers required for processing tasks from the `powerloom-backend-callback` queue. The number of workers and their configuration path can be adjusted in [`config/settings.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json).
 
 ## Processor Distributor
-The Processor Distributor, defined in [`processor_distributor.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py), is initiated using the `processhub_cmd.py` CLI.
+The Processor Distributor, defined in [`processor_distributor.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py), is initiated using the `processhub_cmd.py` CLI.
 - It loads the preloader, base snapshotting, and aggregator config information from the settings file.
-- It reads the events forwarded by the event detector to the `f'powerloom-event-detector:{settings.namespace}:{settings.instance_id}'` RabbitMQ queue bound to a topic exchange as configured in `settings.rabbitmq.setup.event_detector.exchange`([code-ref: RabbitMQ exchanges and queue setup in pooler](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py)).
-- It creates and distributes processing messages based on the preloader configuration present in `config/preloader.json`, the project configuration present in [`config/projects.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json), and [`config/aggregator.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json).
+- It reads the events forwarded by the event detector to the `f'powerloom-event-detector:{settings.namespace}:{settings.instance_id}'` RabbitMQ queue bound to a topic exchange as configured in `settings.rabbitmq.setup.event_detector.exchange`([code-ref: RabbitMQ exchanges and queue setup in pooler](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py)).
+- It creates and distributes processing messages based on the preloader configuration present in `config/preloader.json`, the project configuration present in [`config/projects.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json), and [`config/aggregator.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json).
   - For [`EpochReleased` events](/Protocol/Specifications/Epoch#1-epoch_released), it forwards such messages to base snapshot builders for data source contracts as configured in `config/projects.json` for the current epoch information contained in the event.
 
 ```python reference
-https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L1077-L1115
+https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L1077-L1115
 ```
 
 
@@ -34,44 +34,44 @@ https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d0
 
 Preloaders often fetch and cache large volumes of data, such as all the transaction receipts for a block on the data source blockchain. In such cases, a single worker is often insufficient to feasibly fetch the data for timely base snapshot generation and subsequent aggregate snapshot generations to reach a consensus.
 
-To address this, workers are defined as `delegate_tasks` in [`config/preloader.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json). The [Process Hub Core](#process-hub-core) then launches a specific number of workers, as defined in the primary settings file, [`config/settings.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json), under the key `callback_worker_config.num_delegate_workers`.
+To address this, workers are defined as `delegate_tasks` in [`config/preloader.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json). The [Process Hub Core](#process-hub-core) then launches a specific number of workers, as defined in the primary settings file, [`config/settings.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json), under the key `callback_worker_config.num_delegate_workers`.
 
 
 ```python reference
-https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json#L19-L25
+https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json#L19-L25
 ```
 
 ```python reference
-https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L86-L90
+https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L86-L90
 ```
 
 Delegation workers operate over a simple request-response queue architecture over RabbitMQ.
 
 ```python reference
-https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L243-L254
+https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L243-L254
 ```
 
 One of the preloaders bundled with this snapshotter peer is tasked with fetching all the transaction receipts within a given epoch's block range and because of the volume of data to be fetched it delegates this work to a bunch of delegation worker
 
-* The Preloader: [snapshotter/utils/preloaders/tx_receipts/preloader.py](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/preloader.py).
-* The Delegation Workers: [snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py)
+* The Preloader: [snapshotter/utils/preloaders/tx_receipts/preloader.py](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/preloader.py).
+* The Delegation Workers: [snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py)
 
 As a common functionality shared by all preloaders that utilize delegate workers, this logic is present in the generic class `DelegatorPreloaderAsyncWorker` that all such preloaders inherit. Here you can observe the workload is sent to the delegation workers
 
 ```python reference
-https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L191-L210
+https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L191-L210
 ```
 
 Upon sending out the workloads tagged by unique request IDs, the delegator sets up a temporary exclusive queue to which only the delegation workers meant for the task type push their responses.
 
 ```python reference
-https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L159-L186
+https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L159-L186
 ```
 
 The corresponding response being pushed by the delegation workers can be found here in the generic class `DelegateAsyncWorker` that all such workers should inherit from:
 
 ```python reference
-https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/delegate_worker.py#L74-L84
+https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/delegate_worker.py#L74-L84
 ```
 
 ## Callback Workers
@@ -81,11 +81,11 @@ The callback workers are the ones that build the base snapshot and aggregation s
 They listen to new messages on the RabbitMQ topic exchange as described in the following configuration, and the topic queue's initialization is as follows.
 
 ```python reference
-https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L33-L55
+https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L33-L55
 ```
 
 ```python reference
-https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L182-L213
+https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L182-L213
 ```
 
 Upon receiving a message from the processor distributor after preloading is complete, the workers do most of the heavy lifting along with some sanity checks and then call the `compute()` callback function on the project's configured snapshot worker class to transform the dependent data points as cached by the preloaders to finally generate the base snapshots.
@@ -96,12 +96,12 @@ Upon receiving a message from the processor distributor after preloading is comp
 
 ## RPC Helper
 
-Extracting data from the blockchain state and generating the snapshot can be a complex task. The `RpcHelper`, defined in [`utils/rpc.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/rpc.py), has a bunch of helper functions to make this process easier. It handles all the `retry` and `caching` logic so that developers can focus on efficiently building their use cases.
+Extracting data from the blockchain state and generating the snapshot can be a complex task. The `RpcHelper`, defined in [`utils/rpc.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/rpc.py), has a bunch of helper functions to make this process easier. It handles all the `retry` and `caching` logic so that developers can focus on efficiently building their use cases.
 
 
 ## Core API
 
-This component is one of the most important and allows you to access the finalized protocol state on the smart contract running on the anchor chain. Find it in [`core_api.py`](https://github.com/PowerLoom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/core_api.py).
+This component is one of the most important and allows you to access the finalized protocol state on the smart contract running on the anchor chain. Find it in [`core_api.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/core_api.py).
 
 In the end,