Add SQS integration and report forwarder for filtered transactions by diegoximenes · Pull Request #4620 · OffchainLabs/nitro

diegoximenes · 2026-04-09T16:51:16Z

Resolves NIT-4672

Attention to the base branch

Summary

Add ReportFilteredTransactions RPC endpoint that publishes filtered transaction reports to SQS
Add reusable SQS client package (util/sqsclient) with interface, config, and mock client for testing
Add HTTPClientConfig to genericconf for configurable HTTP client settings
Add report forwarder service that consumes messages from SQS and forwards them to an external HTTP endpoint

Future work

Dead Letter Queue (DLQ) support
Exponential backoff retries to the external endpoint
Metrics to be used by alert rules

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Serialize FilteredTxReport as JSON and enqueue to SQS. The SQS client is defined as an interface in util/sqsclient for easy mocking in tests. Sensitive SQS credentials are masked in config dumps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…point Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tests exercise the full flow: API enqueues to mock SQS, forwarder consumes and forwards to httptest server. Covers success, endpoint failure, and empty queue scenarios. Also removes DLQ support for now. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ing-report Separates report concerns from the transaction filterer service into the dedicated filtering-report service. Adds FilteringReportRPCClient using the filteringreport namespace for future execution engine integration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace the plain ExternalEndpoint string with a reusable HTTPClientConfig (URL + Timeout) in cmd/genericconf, making the HTTP client timeout configurable instead of hardcoded at 30s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Extract the forwarder into cmd/filtering-report/forwarder/ as a separate package. Bundle Workers and PollInterval into a WorkersConfig struct for cleaner configuration grouping. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove the nested WorkersConfig struct and move its fields directly into Config: Count is renamed to Workers, PollInterval is promoted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…nt field Add WaitTimeSeconds as a config option (default 5) instead of hardcoding it, and remove the externalEndpoint field from Forwarder in favor of reading directly from config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Real AWS SQS rejects MaxNumberOfMessages outside 1-10. The mock was silently defaulting 0 to 1, which could hide caller bugs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Replace per-report Info log with single Debug log with count - Drain response body before close to enable HTTP connection reuse - Include response body in non-2xx error messages for debugging - Derive DefaultFilteringReportRPCClientConfig from DefaultClientConfig - Fix Forwarder struct field alignment Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Introduce sqsclient.QueueClient that wraps Client with a QueueURL field. NewClient now returns *QueueClient directly. All consumers (API, forwarder) take *QueueClient instead of separate (Client, queueURL) parameters. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Fail early if sqs.queue-url or external-endpoint.url are empty - Truncate error response body to 512 bytes to avoid huge error strings - Use strings.NewReader instead of bytes.NewBufferString for request body Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

FilteredTxReport is tagged with lint:require-exhaustive-initialization but the test was only setting TxHash. Initialize all fields to satisfy the structinit linter. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Align Workers koanf tag and flag name with field name, and add Enable guard in Start() so the field is referenced within its own package. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tests were passing nil as the SQS client, causing ReportFilteredTransactions to fail with "SQS client not configured". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

diegoximenes · 2026-04-10T17:12:17Z

+	}
+	if err := r.forwardToEndpoint(ctx, *msg.Body); err != nil {
+		log.Warn("Failed to forward report to external endpoint", "err", err, "messageId", msgID)
+		return 0


In a future PR there will be an exponential back off retry mechanism.
We are also going to have a DLQ setup.

Rename the Queue interface to QueueClient and unexport the concrete struct (QueueClient → queueClient). NewQueueClient now returns the interface so consumers depend on the abstraction, not the implementation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

MishkaRogachev

A few minor questions

MishkaRogachev · 2026-04-13T10:10:35Z

+	sqstypes "github.com/aws/aws-sdk-go-v2/service/sqs/types"
+)
+
+type MockQueueClient struct {


suggestion: add error injection to test error paths in forwarder's pollAndForward.

Can you describe an example on how this error injection can be used here?

MishkaRogachev · 2026-04-13T10:12:46Z

+	}
+	if msg.Body == nil {
+		log.Warn("Received SQS message with nil body, deleting", "messageId", msgID)
+		if err = r.queueClient.Delete(ctx, *msg.ReceiptHandle); err != nil {


Same as msg.Body, msg.ReceiptHandle could be nil

AWS SDK guarantees that Body, RequestId and MessageId are not nil.
So I removed the over protective nil checks here.

MishkaRogachev · 2026-04-13T10:13:37Z

+}
+
+func (c *Config) Validate() error {
+	return c.ExternalEndpoint.Validate()


nit: add Workers > 0 check

I updated some types in this config here, using uint instead of int for Workers for example.
Zero is valid though, which can be used to disable the forwarding feature.

MishkaRogachev · 2026-04-13T10:20:36Z

+
+type QueueClient interface {
+	Send(ctx context.Context, body string) error
+	Receive(ctx context.Context, waitTimeSecs, maxMessages int32) ([]sqstypes.Message, error)


Can you add package-local QueueMessage type to keep sqstypes.Message only in implementation?

The goal of QueueClient is not to abstract SQS away, e.g., enable another message queue other than SQS to be used under the hood.
We could definitely do that, but the ROI of doing that right now is pretty low, it is an early optimization.
sqs.Client is not attached to a single QueueURL, and our current use case is to use it with a single queue url.
The current goal of QueueClient is just to encapsulate the QueueURL and the sqs.Client in the same object.
Seems OK for exposing the SQS types here right now.

Makes sense?

mahdy-nasr · 2026-04-13T15:27:28Z

LGTM overall. very minor comments. I also think we need to validate workers == 1 for now till we support mutliple workers in future.

Body, MessageId, and ReceiptHandle are always populated by SQS ReceiveMessage, so the defensive nil checks in pollAndForward were over-protective and inconsistent (ReceiptHandle was already dereferenced unguarded). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Guarantees nonnegativity at the type level instead of requiring runtime validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

A negative PollInterval would make the retry backoff meaningless, and a negative WaitTimeSeconds is rejected by SQS on every ReceiveMessage call. Catch both at config load. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Clarifies that this config controls the SQS ReceiveMessage long-polling wait time, distinct from any HTTP or scheduling timeout in the forwarder. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Log response-body drain and read errors at Warn instead of silently dropping them, and stop reassigning the outer err from inside the defer so the control flow is obvious. resp.Body.Close() is left as-is since net/http effectively never returns a non-nil error there. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

diegoximenes · 2026-04-13T18:08:56Z

LGTM overall. very minor comments. I also think we need to validate workers == 1 for now till we support mutliple workers in future.

Multiple workers are already supported

MishkaRogachev

LGTM

MishkaRogachev · 2026-04-14T10:10:26Z

+			return err
 		}
+		bodyStr := string(body)
+		err = a.queueClient.Send(ctx, bodyStr)


I'm wondering if it is possible to use sqs's SendMessageBatch here instead of loop + Send

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

diegoximenes and others added 29 commits April 9, 2026 12:50

Add ReportFilteredTransactions API endpoint and client method

577b89a

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add SQS consumer workers that forward reports to an external HTTP end…

a1838d6

…point Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Improve report forwarder test

0b96fbe

Handle error from io.ReadAll in report forwarder test

1fbba4a

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Extract MockClient to util/sqsclient for reuse across tests

c68f2d4

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Verify forwarded message bodies match original reports in test

242105e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Early return pollAndForward in case of error

1114cf2

Flatten WorkersConfig fields into forwarder.Config

0c6e5fd

Remove the nested WorkersConfig struct and move its fields directly into Config: Count is renamed to Workers, PollInterval is promoted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Limit pollAndForward to receive at most 1 SQS message per poll

6867365

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

File formatting and change log error to warn

59d585e

Validate MaxNumberOfMessages in MockClient to match real SQS behavior

c4ca1f3

Real AWS SQS rejects MaxNumberOfMessages outside 1-10. The mock was silently defaulting 0 to 1, which could hide caller bugs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rename max to limit and use builtin min in MockClient.ReceiveMessage

0a3c80d

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Fix field alignment and minor formatting issues

4ea1313

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

New log.Debug line. Fix body draining call

f75541b

Revert change made by claude

248ddba

Remove redundant ReportFilteredTransactions

666ad3b

Removes dummy FilteredTxReport

ec3ee4e

Fix koanf lint errors in forwarder config

0fe5f67

Align Workers koanf tag and flag name with field name, and add Enable guard in Start() so the field is referenced within its own package. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Fix API tests by providing mock SQS client

5af7dae

Tests were passing nil as the SQS client, causing ReportFilteredTransactions to fail with "SQS client not configured". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

diegoximenes requested a review from Copilot April 9, 2026 16:53

Rename sqsClient to queueClient in FilteringReportAPI

38262e7

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

diegoximenes commented Apr 10, 2026

View reviewed changes

Merge branch 'add-report-filtered-txn-api-2' into report_filtered_tx_api

c0e85dd

diegoximenes requested review from MishkaRogachev and mahdy-nasr April 10, 2026 17:59

diegoximenes assigned MishkaRogachev Apr 10, 2026

diegoximenes marked this pull request as ready for review April 10, 2026 17:59

MishkaRogachev reviewed Apr 13, 2026

View reviewed changes

MishkaRogachev assigned diegoximenes and unassigned MishkaRogachev Apr 13, 2026

mahdy-nasr reviewed Apr 13, 2026

View reviewed changes

Comment thread cmd/filtering-report/forwarder/forwarder.go

mahdy-nasr reviewed Apr 13, 2026

View reviewed changes

Comment thread cmd/filtering-report/forwarder/forwarder.go Outdated

mahdy-nasr reviewed Apr 13, 2026

View reviewed changes

Comment thread cmd/filtering-report/forwarder/forwarder.go Outdated

mahdy-nasr reviewed Apr 13, 2026

View reviewed changes

Comment thread cmd/filtering-report/main.go

mahdy-nasr reviewed Apr 13, 2026

View reviewed changes

Comment thread cmd/filtering-report/forwarder/forwarder.go Outdated

diegoximenes and others added 5 commits April 13, 2026 14:38

Use uint for forwarder Workers config

4dcffa5

Guarantees nonnegativity at the type level instead of requiring runtime validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rename forwarder WaitTimeSeconds to SQSWaitTimeSeconds

e7ece65

Clarifies that this config controls the SQS ReceiveMessage long-polling wait time, distinct from any HTTP or scheduling timeout in the forwarder. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

diegoximenes requested review from MishkaRogachev and mahdy-nasr April 13, 2026 19:16

diegoximenes assigned MishkaRogachev and unassigned diegoximenes Apr 13, 2026

MishkaRogachev approved these changes Apr 14, 2026

View reviewed changes

mahdy-nasr approved these changes Apr 14, 2026

View reviewed changes

Use port 0 in forwarder tests to avoid port conflicts

cb2ce2d

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Conversation

diegoximenes commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Future work

Uh oh!

diegoximenes Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MishkaRogachev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mahdy-nasr commented Apr 13, 2026

Uh oh!

Uh oh!

diegoximenes commented Apr 13, 2026

Uh oh!

MishkaRogachev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

diegoximenes commented Apr 9, 2026 •

edited

Loading

diegoximenes Apr 10, 2026 •

edited

Loading