Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updater script, asset files and also spark.binproto file added. #448

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

VickyTheViking
Copy link

Hi, dear tsunami team.
Apache Spark has different Web UI based on the way which it is ran. After hours of search I found a way to run it in the way that I can access all web UIs. So in this pull request we have:

1- Master web UI
2- Worker web UI
3- Web interface (Runs only when a SparkContext is running)

Which are extracted in one run of each version . I used apache/spark as base docker image, because it covered more versions than the official _/spark docker repo which naturally do not differ from each other. and versions without docker image were ignored.

@lokiuox
Copy link

lokiuox commented May 24, 2024

Hey @VickyTheViking, thanks for your contribution!

I'm reviewing your plugin but I've found that it's not working properly. Specifically, it looks like the fingerprinting for the main 8080 port is working, but not for the other two ports. I tried to do some quick troubleshooting, but I'm not familiar with Spark.

Here are the issues to fix:

  • update.sh should be in be in the spark/ folder, not spark/app
  • update.sh should have the executable bit set
  • The spark-worker container fails to start. Docker logs show the issue ERROR Utils: Failed to create directory /opt/spark/work which appears to be a permission issue. I was able to bypass it by adding user: root in the docker compose file, but I don't know if this is the correct way to fix such an issue. This seems to fix the issue with port 8081.
  • Nonetheless, the docker exec command fails, so the Python script is never executed and port 4040 remains unreachable.
  • Also the selected Spark docker images do not seem to have Python at all, ensure they are the correct images.

Feel free to reach out.

~ Savio (Doyensec)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants