Implementing a Custom Consumer Assignment Strategy using TypeScript

My service runs in Kubernetes and can scale to thousands during peak times. Each instance of the service consumes events from Kafka, with Managed Streaming Kafka (AWS MSK) as the broker. To avoid performance overhead from having numerous topics with metadata, I am considering using a single topic with multiple partitions, each attached to a specific service instance. Our backend is built in TypeScript and we utilize kafkajs for Kafka integration. However, it appears that the library does not currently support custom assignment strategies, where consumers in a consumer group are sequentially allocated to partitions as they increase. Is there a JavaScript Kafka library available that supports this particular feature?

typescriptapache-kafkakafkajs

Answer №1

KafkaJS offers support for individual partition assignment through the use of partitionAssigners consumer option.

For stateless applications in Kubernetes, utilizing a StatefulSet with an emptyDir volume allows for predicting the instance id of a pod and targeting a specific partition accordingly.

It is important to note that having more replicas than partitions is not efficient. Opting for KEDA autoscaler instead of traditional CPU/MEM load balancing can help ensure each partition is treated equally without associating partition values with process orchestration.

Answer №2

If Kafkajs doesn't seem to be an option, you can always turn to node-rdkafka.

import {KafkaConsumer} from 'node-rdkafka';

const consumer = new KafkaConsumer({
    'group.id': 'secondGP',
    'metadata.broker.list': 'localhost:9092',
}, {
    'auto.offset.reset': 'earliest',
});

consumer.connect();

consumer.on('ready', () => {
    console.log('Consumer is ready');
    consumer.assign([{topic: 'testTopic', partition: 1}]);
    consumer.consume();
    consumer.on('data', (message) => {
        console.log(`Received message: key: ${message.key.toString()} value: ${message.value.toString()} Partition: ${message.partition}`);
        consumer.commit(message);
    });
});

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The function executes one loop and then outputs the result

This function is designed to remove duplicate items from an array: removeDuplicates(item: String, allForToday: Array) { let temp = allForToday; for (let i = 0; i < temp.length; i++) { if (temp[i].indexOf(item) > -1) { tem ...

javascriptarraysloopsfor-looptypescript

Vue.js is unable to recognize this type when used with TypeScript

In my code snippet, I am trying to set a new value for this.msg but it results in an error saying Type '"asdasd"' is not assignable to type 'Function'. This issue persists both in Visual Studio and during webpack build. It seems like Ty ...

visual-studiotypescriptwebpackvue.js

Simplify deeply nested JSON structures with TypeScript generics

After receiving the JSON data shown below, I have a specific model in mind for deserialization: {"results": {"data": { "id":"93cba9bd-2969-3214-b26f-7f42d20241a4", "first_name":"PSU", "last_name":"Bot", "profile": { "data":{"abou ...

typescriptgenericsinterface

Using props in React can be declared either as a `type` or an `interface`

I am working with React code export default class MyComponent extends Component<Props,State> I'm trying to figure out whether I should define my props using a type or an interface. type Props = { isActive: Boolean, onClick: Function } ...

reactjstypescriptflowtype

Webpack is failing to recognize certain CSS files

My development stack includes Vue.js 2.5.15, Webpack 4.12.0, css-loader 0.28.11, ASP.Net Core 2.1 in Visual Studio 2017. Starting with the Visual Studio asp.net core template project for Vue and Typescript, I prefer to have individual small CSS files with ...

csstypescriptwebpackvue.jscss-loader

How to resolve TypeScript error TS2307 when importing a text file: "Module not found"

I am trying to import a text file into my TypeScript file and then print its contents. index.ts file: import d from "./a.txt"; console.log(d); txt.d.ts file: declare module "*.txt" { const value: string; export default value; } tsconfig.json file: ...

typescriptimportmodule

Manipulate values within an array when a checkbox is selected or deselected

I am developing an Angular application where I have created a checkbox that captures the change event and adds the checked value to an array. The issue I am facing is that even if the checkbox is unchecked, the object is still being added to the array. D ...

javascriptangulartypescriptcheckboxangular6

Enable child classes to overwrite using either a function attribute or a method

class Foo { foo: () => void } class Bar extends Foo { foo() {} } Is there a way in which TypeScript can be configured to allow the scenario described above? Playground The 'Foo' class defines a property 'foo' as an instance ...

typescript

The system encountered an issue while trying to access the property 'play' of null

I'm attempting to create a customized audio player, but I've encountered an error: "cannot read property 'play' of null". After some research, I discovered that this could be due to the function being called before the ID exists. Howeve ...

javascriptangulartypescripthtml5-audio

Is there a way to retrieve the chosen value from an ion-alert radio alert?

async showAlertRadio(heading:string){ const alert = await this.alertCtrl.create({ header: heading, inputs :[ { name : 'Radio 1', type: 'radio', label: 'Radio 1', ...

angulartypescriptionic-frameworkionic4

Tips for removing the Y-Axis entirely from a CanavasJS chart

I'm working with a canvasJS chart and my goal is to completely hide the Y-Axis. I've managed to remove the Y-Axis line and title in this JSFiddle example. I came across these properties to hide the Y-Axis title and line, but I'm struggling t ...

javascripthtmltypescriptjsfiddlecanvasjs

The process of switching from timestamp to date varies depending on whether you are working locally or on a server

Currently, I am in the process of developing an app using Firebase as the server and Flutter for the frontend. The situation I am facing is that when I upload a new document with a timestamp containing the current time in this format: "timestamp": DateTim ...

typescriptfirebasefluttergoogle-cloud-firestoretimestamp

What's wrong with the current longitude and latitude bounding box algorithm used for geolocation searches?

I am currently working on a piece of code that calculates a bounding box for a specific location to search for user profiles within a given radius. The code is mostly functional, but I am encountering a slight distortion in the final values. When I input 5 ...

javascripttypescriptmathgeolocationcalculation

Angular: sending the user input to the component

I'm trying to link the value of an HTML input field to a variable 'numToAdd' in my component, and then increment the 'numToAdd' variable by 1. However, I'm facing difficulties passing the input value to the component variable. ...

htmlangulartypescript

Creating a parameter type in TypeScript by referencing another parameter type

Attempting to articulate our situation the best I can, given the lack of a specific title. We have developed a package that facilitates communication between our various services and cloud functions. This package includes functions to call each service (e ...

typescript

Tips for incorporating promise/async within a function that returns an Observable

I am a user of nestjs and I am currently trying to create a function that returns an Observable (rxjs) with cache functionality. import { HttpService } from '@nestjs/axios'; import { CACHE_MANAGER, Inject, Injectable } from '@nestjs/common&a ...

typescriptpromiserxjsnestjs

Angular error: "No directive was found with exportAs 'ngModel'." Ensure that FomsModule has already been imported

I'm encountering an issue where I am being advised to import "FomsModule", but it is already imported in my code. I attempted to include "ReactiveFormsModule" as well, but the problem persists. Here is the complete error message: src/app/components/ ...

javascriptangulartypescript

Exploring LocalStorage Monitoring in Vue.js 2

How can I stay informed about any changes in Banana.vue? I have tried using addEventListener, @watch but it doesn't seem to be working... The index.vue file is importing both Apple.vue and Banana.vue In Apple.vue: localStorage.setItem('fruit ...

typescriptvue.jsaddeventlistenerwatchvue-property-decorator

What leads to the inability to utilize environment variables in this TypeScript app built with Vue 3?

Currently, I am developing a single page application utilizing Vue 3 and TypeScript. The main purpose of this app is to interact with an API. All the necessary information including the API's URL and key are stored in the 'src\env.js' f ...

typescriptvue.jsvuejs3

The parameter cannot accept a value of type 'number | null' even if it is not null

I'm puzzled about the error I'm encountering in TypeScript: "Argument of type 'number | null' is not assignable to parameter of type 'number'", even though I have already checked if my variable is not null. Is there any other ...

angulartypescript