Skip to content

Distributed System in Docker with Apache Kafka and Spark for big data streaming and visualisation (NodeJS, TypeScript, React, NestJS, Java)

Notifications You must be signed in to change notification settings

zoltan-nz/kafka-spark-project

Repository files navigation

Apache Kafka and Apache Spark - A Distributed Streaming Project

For detailed project report, please check out The Project Report. ;)

Running the project

You can try out this project running all components in a docker composed cluster, each component will run in a separated container and they will be connected with the default network inside the compose cluster.

Prerequisites:

  • Docker
  • Port 80 should be available or change the port mapping of the frontend app in docker-compose.yml

Run the project:

  • Clone this repository on your computer
  • Fire up the docker compose cluster:
$ docker-compose up
  • Open the frontend application in your browser: $ open http://localhost:80

Shutting down docker-compose (use an other terminal window for running this command):

$ docker-compose down

Notes:

  • Building the maven based projects at the first time takes a while, please be patient.
  • Kafka generated files are mapped to ./kafka/volumes folder. If the streaming does not start when the project is launched at first time, shut down the docker-compose cluster and start it up again. All mapped volumes and folders should be available at the second launch and Kafka can start properly.

Run the project in developer mode

You can run all components locally. In this way you can easily debug and add new features.

Prerequisites:

The setup npm script will install individual packages and prepare the project for you. The start:dev will run all component's development script concurrently in the same terminal.

$ npm run setup
$ npm run start:dev

Documents

Component's README files

Useful links

Reading

About

Distributed System in Docker with Apache Kafka and Spark for big data streaming and visualisation (NodeJS, TypeScript, React, NestJS, Java)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published