Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/common-aws/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@paradoxical-io/common-aws",
"version": "2.6.0",
"version": "2.7.0",
"description": "Common aws for paradox services",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@ import {
PutParameterCommand,
SSMClient,
} from '@aws-sdk/client-ssm';
import { Arrays, Limiter } from '@paradoxical-io/common';
import { log, retry as autoRetry } from '@paradoxical-io/common-server';
import { Arrays, Limiter, retry as autoRetry } from '@paradoxical-io/common';
import { log } from '@paradoxical-io/common-server';
import { ErrorCode, ErrorWithCode, isErrorWithCode } from '@paradoxical-io/types';

import retry = require('async-retry');

export class ParameterStoreApi {
Expand Down
60 changes: 34 additions & 26 deletions packages/common-aws/src/sqs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,20 +37,18 @@ interface UserEvent {
}

// Create a publisher
const publisher = new SQSPublisher<UserEvent>(
'https://sqs.us-west-2.amazonaws.com/123456789/my-queue' as QueueUrl
);
const publisher = new SQSPublisher<UserEvent>('https://sqs.us-west-2.amazonaws.com/123456789/my-queue' as QueueUrl);

// Publish a single message
await publisher.publish({
userId: 'user-123',
action: 'login'
action: 'login',
});

// Publish multiple messages (automatically batched in chunks of 10)
await publisher.publish([
{ userId: 'user-123', action: 'login' },
{ userId: 'user-456', action: 'logout' }
{ userId: 'user-456', action: 'logout' },
]);
```

Expand All @@ -65,8 +63,8 @@ await publisher.publish(
{
delay: {
type: 'invisible',
seconds: 30 as Seconds
}
seconds: 30 as Seconds,
},
}
);
```
Expand All @@ -78,15 +76,15 @@ Schedule messages to be processed at a specific time, even beyond SQS's 15-minut
```typescript
import { EpochMS } from '@paradoxical-io/types';

const processAt = Date.now() + (3 * 24 * 60 * 60 * 1000); // 3 days from now
const processAt = Date.now() + 3 * 24 * 60 * 60 * 1000; // 3 days from now

await publisher.publish(
{ userId: 'user-123', action: 'reminder' },
{
delay: {
type: 'processAfter',
epoch: processAt as EpochMS
}
epoch: processAt as EpochMS,
},
}
);
```
Expand All @@ -108,7 +106,7 @@ class OrderProcessor extends SQSConsumer<OrderEvent> {
constructor() {
super('https://sqs.us-west-2.amazonaws.com/123456789/orders' as QueueUrl, {
maxNumberOfMessages: 5,
longPollWaitTimeSeconds: 20 as Seconds
longPollWaitTimeSeconds: 20 as Seconds,
});
}

Expand All @@ -122,7 +120,7 @@ class OrderProcessor extends SQSConsumer<OrderEvent> {
return {
type: 'retry-later',
reason: 'Database temporarily unavailable',
retryInSeconds: 60 as Seconds
retryInSeconds: 60 as Seconds,
};
}
throw error; // Non-retryable errors let the message go to DLQ
Expand All @@ -147,10 +145,10 @@ import { newConsumer, SQSConfig } from '@paradoxical-io/common-aws/sqs';

const config: SQSConfig = {
queueUrl: 'https://sqs.us-west-2.amazonaws.com/123456789/orders' as QueueUrl,
maxNumberOfMessages: 10
maxNumberOfMessages: 10,
};

const consumer = newConsumer<OrderEvent>(async (event) => {
const consumer = newConsumer<OrderEvent>(async event => {
console.log('Processing:', event);
// Returning void acknowledges the message
}, config);
Expand Down Expand Up @@ -209,6 +207,7 @@ async process(data: OrderEvent): Promise<MessageProcessorResult> {
The retry delay will be calculated as: `min + 2^(publishCount)`

Examples:

- 1st retry: 1 + 2^1 = 3 seconds
- 2nd retry: 1 + 2^2 = 5 seconds
- 3rd retry: 1 + 2^3 = 9 seconds
Expand Down Expand Up @@ -245,6 +244,7 @@ await runSQS([consumer1, consumer2]);
```

The `runSQS` function:

- Starts all consumers concurrently
- Registers shutdown signal handlers (SIGTERM, SIGINT)
- Ensures graceful shutdown with proper cleanup
Expand Down Expand Up @@ -310,40 +310,46 @@ await docker.container.stop();
Main class for publishing messages to SQS.

**Constructor:**

```typescript
new SQSPublisher<T>(queueUrl: string, sqs?: SQSClient)
```

**Methods:**

- `publish(data: T | T[], opts?: PublishOptions): Promise<void>` - Publish one or more messages

### SQSConsumer<T>

Abstract base class for consuming messages.

**Constructor:**

```typescript
new SQSConsumer<T>(queueUrl: QueueUrl, opts?: Partial<Options>)
```

**Abstract Methods:**

- `process(data: T): Promise<MessageProcessorResult>` - Implement to handle messages

**Methods:**

- `start(): Promise<void>` - Start consuming messages
- `stop(opts?: { timeoutMilli?: number; flush?: boolean }): void` - Stop the consumer
- `adhoc(message: Message): Promise<void>` - Process a single message ad-hoc

**Options:**

```typescript
interface Options {
longPollWaitTimeSeconds: Seconds; // Default: 20
maxNumberOfMessages: Max10; // Default: 10
sqs: SQSClient; // Default: new SQSClient()
makeAvailableOnError: boolean; // Default: false
timeProvider: TimeProvider; // Default: defaultTimeProvider()
longPollWaitTimeSeconds: Seconds; // Default: 20
maxNumberOfMessages: Max10; // Default: 10
sqs: SQSClient; // Default: new SQSClient()
makeAvailableOnError: boolean; // Default: false
timeProvider: TimeProvider; // Default: defaultTimeProvider()
maxVisibilityTimeoutSeconds?: Seconds; // Optional
proxyProvider?: ProxyQueueProvider; // Optional
proxyProvider?: ProxyQueueProvider; // Optional
}
```

Expand Down Expand Up @@ -384,11 +390,13 @@ interface RetryMessageLater {
interface RepublishMessage {
type: 'republish-later';
reason: string;
retryInSeconds: Seconds | {
type: 'exponential-backoff';
max: Seconds;
min: Seconds;
};
retryInSeconds:
| Seconds
| {
type: 'exponential-backoff';
max: Seconds;
min: Seconds;
};
expireFromFirstPublishMS?: Milliseconds;
}
```
Expand Down Expand Up @@ -417,4 +425,4 @@ interface RepublishMessage {
- **Trace Propagation**: Trace IDs are automatically generated and propagated through message lifecycle for distributed tracing
- **Visibility Management**: The module handles message visibility for both retry strategies appropriately
- **Error Handling**: Unhandled errors are logged with trace context and metrics but don't affect other messages in the batch
- **Development Proxy**: In non-production environments, messages can be proxied to additional queues for testing
- **Development Proxy**: In non-production environments, messages can be proxied to additional queues for testing
2 changes: 1 addition & 1 deletion packages/common-hapi/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@paradoxical-io/common-hapi",
"version": "2.6.0",
"version": "2.7.0",
"description": "Common hapi code for paradoxical services",
"files": [
"dist/**/*.js",
Expand Down
26 changes: 15 additions & 11 deletions packages/common-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,26 +130,26 @@ const configShape = (env: Env) => ({
doc: 'Server port',
format: 'port',
default: 3000,
env: 'PORT'
env: 'PORT',
},
database: {
host: {
doc: 'Database host',
format: String,
default: 'localhost'
default: 'localhost',
},
port: {
doc: 'Database port',
format: 'port',
default: 5432
}
default: 5432,
},
},
apiKey: {
doc: 'API key',
format: String,
default: '',
sensitive: true
}
sensitive: true,
},
});

// Load from config/local.json or config/prod.json
Expand Down Expand Up @@ -248,12 +248,12 @@ const users = await reader.read<User>('/path/to/users.csv');
const csv = new Csv<User>('/path/to/output.csv', [
{ id: 'name', title: 'Name' },
{ id: 'email', title: 'Email' },
{ id: 'age', title: 'Age' }
{ id: 'age', title: 'Age' },
]);

await csv.write([
{ name: 'John', email: '[email protected]', age: '30' },
{ name: 'Jane', email: '[email protected]', age: '25' }
{ name: 'Jane', email: '[email protected]', age: '25' },
]);
```

Expand All @@ -268,13 +268,13 @@ import { spawnPromise, runShell } from '@paradoxical-io/common-server';
const { code, result } = await spawnPromise('npm', ['install'], {
verbose: true,
cwd: '/path/to/project',
acceptableErrorCodes: [0]
acceptableErrorCodes: [0],
});

// Run shell command
const exitCode = await runShell('git status', {
verbose: true,
cwd: process.cwd()
cwd: process.cwd(),
});
```

Expand Down Expand Up @@ -347,6 +347,7 @@ The library respects several environment variables for configuration:
## API

### Logger

- `log.info(msg)` - Log info message
- `log.error(msg, error?)` - Log error with optional error object
- `log.warn(msg, error?)` - Log warning
Expand All @@ -356,17 +357,20 @@ The library respects several environment variables for configuration:
- `log.alarm(msg)` - Log alertable message

### Metrics

- `Metrics.instance.increment(name, tags?)` - Increment counter
- `Metrics.instance.timing(name, ms, tags?)` - Record timing
- `Metrics.instance.gauge(name, value, tags?)` - Set gauge value

### Decorators

- `@logMethod(options?)` - Log method calls with arguments
- `@timed(options?)` - Track method execution time
- `@retry(options?)` - Add retry logic to methods
- `@sensitive(redaction?)` - Mark parameters as sensitive

### Tracing

- `withNewTrace(fn, trace?, context?)` - Create new trace context
- `traceID()` - Get current trace ID
- `setCurrentUserId(userId)` - Set user ID in trace context
Expand All @@ -378,4 +382,4 @@ MIT

## Author

Anton Kropp
Anton Kropp
2 changes: 1 addition & 1 deletion packages/common-server/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@paradoxical-io/common-server",
"version": "2.6.0",
"version": "2.7.0",
"description": "Common code for paradox services",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
Expand Down
1 change: 0 additions & 1 deletion packages/common-server/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ export * from './logger';
export * from './metrics';
export * from './path';
export * from './process';
export * from './retry/retryDecorator';
export * from './sftp';
export * from './test';
export * from './trace';
Expand Down
1 change: 1 addition & 0 deletions packages/common-server/src/metrics/metrics.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/* eslint-disable no-console,@typescript-eslint/no-explicit-any */
// make sure we allow all logging (even if the test runner says otherwise)

process.env.PARADOX_LOG_LEVEL = 'info';

// ensure log decorators always run
Expand Down
Loading