Skip to content

Concurrency manager system#507

Closed
adityachoudhari26 wants to merge 27 commits intomainfrom
concurrency-manager-system
Closed

Concurrency manager system#507
adityachoudhari26 wants to merge 27 commits intomainfrom
concurrency-manager-system

Conversation

@adityachoudhari26
Copy link
Copy Markdown
Member

@adityachoudhari26 adityachoudhari26 commented Apr 24, 2025

Summary by CodeRabbit

  • New Features

    • Introduced a new API endpoint to fetch release targets for a specific resource.
    • Added support for managing and computing release targets and resource associations via new worker processes.
    • Enhanced OpenAPI schemas and API endpoints to provide more detailed status and error responses for deployments and deployment versions.
  • Bug Fixes

    • Improved error handling and response consistency for deployment and deployment version update endpoints.
  • Refactor

    • Reworked resource, environment, deployment, and policy selector computation to use new worker-based architecture, removing previous in-line and builder-based logic.
    • Simplified and streamlined worker event handling and job queuing for resource and deployment updates.
  • Documentation

    • Updated OpenAPI specifications to reflect new endpoints, improved error responses, and schema refinements.
  • Tests

    • Added comprehensive end-to-end tests for deployments, deployment versions, environments, and release targets.
    • Introduced YAML fixtures for robust test data setup and validation.
  • Chores

    • Added new npm scripts for targeted test execution.
    • Removed obsolete selector builder and mutex implementation files.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 24, 2025

Caution

Review failed

The pull request is closed.

Walkthrough

This update introduces a major refactor of resource selector computation and release target management across the backend, API, and end-to-end test suites. The core change is the removal of the previous selector compute builder classes and mutex-based concurrency logic from the database package, replaced by new worker-based event-driven computation pipelines in the event-worker app. New workers are introduced for computing environment and deployment resource selectors, policy target release target selectors, and system release targets, all orchestrated via Redis-based concurrency primitives. The API is updated to support new endpoints and improved OpenAPI schemas, including a new endpoint for fetching release targets by resource. End-to-end tests are expanded and refactored to align with the new computation flows and API contracts.

Changes

File(s) / Path(s) Change Summary
apps/event-worker/src/system-release-targets.ts, apps/event-worker/src/utils/with-mutex.ts Introduced new concurrency management utilities/classes for system release target computations using Redis mutexes and semaphores.
apps/event-worker/src/workers/compute-*.ts Added new worker event handlers for computing environment resource selectors, deployment resource selectors, policy target release target selectors, and system release targets.
apps/event-worker/src/workers/index.ts Registered new workers in the worker registry for event-driven computation.
apps/event-worker/src/workers/new-*.ts, apps/event-worker/src/workers/update-*.ts, apps/event-worker/src/workers/updated-resources/index.ts Refactored workers to enqueue computation jobs instead of performing inline resource selector and release target computations.
packages/events/src/types.ts Extended the Channel enum and ChannelMap type with new channels for the added computation workers.
packages/db/src/selectors/compute/*.ts, packages/db/src/selectors/index.ts Removed all selector compute builder classes, mutex logic, and related exports; only query builders remain.
packages/rule-engine/src/db/create-policy.ts, packages/rule-engine/src/db/update-policy.ts Removed side-effect calls to compute and release target selectors after policy creation/update.
apps/webservice/src/app/api/v1/deployment-versions/[deploymentVersionId]/openapi.ts, .../route.ts, .../openapi.ts Improved OpenAPI schemas and error handling for deployment version PATCH/POST endpoints.
apps/webservice/src/app/api/v1/deployments/[deploymentId]/openapi.ts, .../resources/openapi.ts, .../route.ts Refined OpenAPI schemas for PATCH and GET endpoints, improved validation, and updated handler logic to use new computation queues.
apps/webservice/src/app/api/v1/environments/[environmentId]/resources/openapi.ts, .../route.ts Updated OpenAPI response schemas to use reusable components; adjusted handler logic for new computation flow.
apps/webservice/src/app/api/v1/resources/[resourceId]/release-targets/openapi.ts, .../route.ts Added new endpoint and OpenAPI schema for fetching release targets by resource.
apps/webservice/src/app/api/v1/resources/[resourceId]/route.ts, apps/webservice/src/app/api/v1/resources/route.ts Removed selector compute logic; now only enqueues update events for resource changes.
apps/webservice/src/app/api/v1/systems/[systemId]/route.ts Minor internal refactor to parameter extraction for authorization.
apps/webservice/src/app/api/v1/openapi.ts Refined DeploymentVersion schema: restricted metadata to string values, added status enum.
e2e/api/schema.ts, openapi.v1.json Updated API schema: added new endpoint, refined resource and deployment version schemas, improved error responses.
e2e/tests/api/deployment-version.spec.ts, e2e/tests/api/deployments.spec.ts, e2e/tests/api/environments.spec.ts, e2e/tests/api/release-targets.spec.ts Added and refactored E2E tests to cover new resource selector, deployment version, and release target flows.
e2e/tests/api/resource-selectors.spec.ts, e2e/tests/api/resource-selectors.spec.yaml Removed obsolete resource selector E2E tests and fixtures.
e2e/tests/api/environments.spec.yaml, e2e/tests/api/deployments.spec.yaml, e2e/tests/api/deployment-version.spec.yaml, e2e/tests/api/release-targets.spec.yaml Added or updated YAML fixtures for new/updated E2E tests.
e2e/package.json Added script for running release target E2E tests.

Sequence Diagram(s)

sequenceDiagram
    participant API
    participant Worker
    participant DB
    participant Redis

    API->>Worker: Enqueue ComputeDeploymentResourceSelector job
    Worker->>Redis: Acquire Mutex for deployment
    alt Mutex Acquired
        Worker->>DB: Fetch deployment and system
        Worker->>DB: Delete old computed deployment resources
        Worker->>DB: Query matching resources
        Worker->>DB: Insert new computed deployment resources
        Worker->>Worker: Enqueue ComputeSystemsReleaseTargets job
    else Mutex Not Acquired
        Worker->>Worker: Re-enqueue job with deduplication
    end
Loading
sequenceDiagram
    participant API
    participant Worker
    participant DB
    participant Redis

    API->>Worker: Enqueue ComputeEnvironmentResourceSelector job
    Worker->>Redis: Acquire Mutex for environment
    alt Mutex Acquired
        Worker->>DB: Fetch environment and system
        Worker->>DB: Delete old computed environment resources
        Worker->>DB: Query matching resources
        Worker->>DB: Insert new computed environment resources
        Worker->>Worker: Enqueue ComputeSystemsReleaseTargets job
    else Mutex Not Acquired
        Worker->>Worker: Re-enqueue job with deduplication
    end
Loading

Possibly related PRs

  • ctrlplanedev/ctrlplane#448: Removes Redis and redis-semaphore dependencies and related mutex implementations from the event-worker package, directly related to the new concurrency management introduced in this PR.

Poem

In the warren of code, selectors once hopped,
Now queues and workers keep computations on top.
With mutexes guarding, and Redis in tow,
Release targets are matched in a synchronous flow.
The API’s grown wiser, its schemas refined,
And rabbits rejoice—no resource left behind!
🐇✨


📜 Recent review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6f16093 and 2aca50e.

📒 Files selected for processing (53)
  • apps/event-worker/src/system-release-targets.ts (1 hunks)
  • apps/event-worker/src/utils/with-mutex.ts (1 hunks)
  • apps/event-worker/src/workers/compute-deployment-resource-selector.ts (1 hunks)
  • apps/event-worker/src/workers/compute-environment-resource-selector.ts (1 hunks)
  • apps/event-worker/src/workers/compute-policy-target-release-target-selector.ts (1 hunks)
  • apps/event-worker/src/workers/compute-systems-release-targets.ts (1 hunks)
  • apps/event-worker/src/workers/index.ts (2 hunks)
  • apps/event-worker/src/workers/new-deployment.ts (1 hunks)
  • apps/event-worker/src/workers/new-environment.ts (1 hunks)
  • apps/event-worker/src/workers/new-policy.ts (1 hunks)
  • apps/event-worker/src/workers/new-resource.ts (2 hunks)
  • apps/event-worker/src/workers/update-deployment.ts (2 hunks)
  • apps/event-worker/src/workers/update-environment.ts (2 hunks)
  • apps/event-worker/src/workers/update-resource-variable.ts (1 hunks)
  • apps/event-worker/src/workers/updated-resources/index.ts (2 hunks)
  • apps/webservice/src/app/api/v1/deployment-versions/[deploymentVersionId]/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/deployment-versions/[deploymentVersionId]/route.ts (4 hunks)
  • apps/webservice/src/app/api/v1/deployment-versions/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/deployments/[deploymentId]/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/deployments/[deploymentId]/resources/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/deployments/[deploymentId]/route.ts (3 hunks)
  • apps/webservice/src/app/api/v1/deployments/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/environments/[environmentId]/resources/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/environments/route.ts (1 hunks)
  • apps/webservice/src/app/api/v1/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/resources/[resourceId]/release-targets/openapi.ts (1 hunks)
  • apps/webservice/src/app/api/v1/resources/[resourceId]/release-targets/route.ts (1 hunks)
  • apps/webservice/src/app/api/v1/resources/[resourceId]/route.ts (1 hunks)
  • apps/webservice/src/app/api/v1/resources/route.ts (2 hunks)
  • apps/webservice/src/app/api/v1/systems/[systemId]/route.ts (1 hunks)
  • e2e/api/schema.ts (10 hunks)
  • e2e/package.json (1 hunks)
  • e2e/tests/api/deployment-version.spec.ts (1 hunks)
  • e2e/tests/api/deployment-version.spec.yaml (1 hunks)
  • e2e/tests/api/deployments.spec.ts (1 hunks)
  • e2e/tests/api/deployments.spec.yaml (1 hunks)
  • e2e/tests/api/environments.spec.ts (5 hunks)
  • e2e/tests/api/environments.spec.yaml (1 hunks)
  • e2e/tests/api/release-targets.spec.ts (1 hunks)
  • e2e/tests/api/release-targets.spec.yaml (1 hunks)
  • e2e/tests/api/resource-selectors.spec.ts (0 hunks)
  • e2e/tests/api/resource-selectors.spec.yaml (0 hunks)
  • openapi.v1.json (10 hunks)
  • packages/db/src/selectors/compute/compute.ts (0 hunks)
  • packages/db/src/selectors/compute/deployment-builder.ts (0 hunks)
  • packages/db/src/selectors/compute/environment-builder.ts (0 hunks)
  • packages/db/src/selectors/compute/mutex.ts (0 hunks)
  • packages/db/src/selectors/compute/policy-builder.ts (0 hunks)
  • packages/db/src/selectors/compute/resource-builder.ts (0 hunks)
  • packages/db/src/selectors/index.ts (0 hunks)
  • packages/events/src/types.ts (2 hunks)
  • packages/rule-engine/src/db/create-policy.ts (1 hunks)
  • packages/rule-engine/src/db/update-policy.ts (1 hunks)
✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@adityachoudhari26 adityachoudhari26 deleted the concurrency-manager-system branch May 5, 2025 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants