Add exec subcommand to CLI #17797

twalluxio · 2023-07-18T08:16:02Z

Add exec commands to golang CLI as part of #17522

bin/alluxio runClass -> bin/cli exec class
bin/alluxio runTests -> bin/cli exec testRun
bin/alluxio runUfsTests -> bin/cli exec testUfs
bin/alluxio runUfsIOTest -> bin/cli exec testUfsIO
bin/alluxio runHdfsMountTests -> bin/cli exec testHdfsMount
bin/alluxio runHmsTests -> bin/cli exec testHms
bin/alluxio runJournalCrashTest -> bin/cli exec testJournalCrash

For command exec test, using different --name flags can lead to huge difference in other flags and options.
To manage flags and options, currently using different commands for different test types.

alluxio-bot · 2023-07-18T08:16:06Z

Thank you for your pull request.
In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement (CLA).
It's all electronic and will take just a few minutes. Please download CLA form here, sign, and e-mail back to cla@alluxio.org

cli/src/alluxio.org/cli/cmd/exec/class.go

Xenorith · 2023-07-19T00:54:58Z

For command exec test, using different --name flags can lead to huge difference in other flags and options.
To manage flags and options, currently using different commands for different test types.

agree with this

cli/src/alluxio.org/cli/cmd/exec/class.go

cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

cli/src/alluxio.org/cli/cmd/exec/test_hms.go

cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

cli/src/alluxio.org/cli/cmd/exec/test_run.go

cli/src/alluxio.org/cli/cmd/exec/class.go

cli/src/alluxio.org/cli/cmd/exec/test_hms.go

cli/src/alluxio.org/cli/cmd/exec/test_journal_crash.go

cli/src/alluxio.org/cli/cmd/exec/test_ufs.go

cli/src/alluxio.org/cli/cmd/exec/test_ufsio.go

Xenorith

okay, left suggestions for the command names to be more like xxxTest

but note that the file names should remain the same. if you change the file name to xxx_test.go, go will treat it as a test file

cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

cli/src/alluxio.org/cli/cmd/exec/test_hms.go

cli/src/alluxio.org/cli/cmd/exec/test_journal_crash.go

cli/src/alluxio.org/cli/cmd/exec/test_ufs.go

Co-authored-by: Kai <cs.zhangkai@outlook.com>

cli/src/alluxio.org/cli/cmd/exec/class.go

cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

cli/src/alluxio.org/cli/cmd/exec/test_hms.go

cli/src/alluxio.org/cli/cmd/exec/test_journal_crash.go

cli/src/alluxio.org/cli/cmd/exec/test_ufs.go

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Xenorith

more cleanups, can assume i approve after making these changes

also add reminder about checking user inputs. i want to share @jenoudet 's comment about using the validator library as a convenient way to define valid input values: #17570 (comment)
this is also another area to help improve this code

cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

cli/src/alluxio.org/cli/cmd/exec/test_hms.go

cli/src/alluxio.org/cli/cmd/exec/test_ufs.go

cli/src/alluxio.org/cli/cmd/exec/test_journal_crash.go

Xenorith · 2023-07-27T04:18:22Z

cli/src/alluxio.org/cli/cmd/exec/test_journal_crash.go

+	if c.testDir != "" {
+		javaArgs = append(javaArgs, "-testDir", c.testDir)
+	}
+	javaArgs = append(javaArgs, "-totalTime", strconv.Itoa(c.totalTime))


set with group of other flags above since the order of flags doesn't matter

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Kai-Zhang · 2023-08-01T02:52:30Z

alluxio-bot, merge this please

Add exec commands to golang CLI as part of #17522 `bin/alluxio runClass` -> `bin/cli exec class` `bin/alluxio runTests` -> `bin/cli exec testRun` `bin/alluxio runUfsTests` -> `bin/cli exec testUfs` `bin/alluxio runUfsIOTest` -> `bin/cli exec testUfsIO` `bin/alluxio runHdfsMountTests` -> `bin/cli exec testHdfsMount` `bin/alluxio runHmsTests` -> `bin/cli exec testHms` `bin/alluxio runJournalCrashTest` -> `bin/cli exec testJournalCrash` For command `exec test`, using different `--name` flags can lead to huge difference in other flags and options. To manage flags and options, currently using different commands for different test types. pr-link: #17797 change-id: cid-eccbfaa37239ec8d567bcd89393df93ea60288f2

Add exec commands to golang CLI as part of Alluxio#17522 `bin/alluxio runClass` -> `bin/cli exec class` `bin/alluxio runTests` -> `bin/cli exec testRun` `bin/alluxio runUfsTests` -> `bin/cli exec testUfs` `bin/alluxio runUfsIOTest` -> `bin/cli exec testUfsIO` `bin/alluxio runHdfsMountTests` -> `bin/cli exec testHdfsMount` `bin/alluxio runHmsTests` -> `bin/cli exec testHms` `bin/alluxio runJournalCrashTest` -> `bin/cli exec testJournalCrash` For command `exec test`, using different `--name` flags can lead to huge difference in other flags and options. To manage flags and options, currently using different commands for different test types. pr-link: Alluxio#17797 change-id: cid-eccbfaa37239ec8d567bcd89393df93ea60288f2

- Add cli/ folder containing golang code - Add script to compile code to executable in build/cli/build-cli.sh - Add entrypoint script for development in bin/cli.sh - Add build profile to build the CLI as part of tha maven build via `-PgoCli` - Add `conf` service, consisting of - `bin/cli.sh conf get` as an example command, equivalent to `bin/alluxio getConf` - `bin/cli.sh conf log --name fully.qualified.class.path`, equivalent to `bin/alluxio fsadmin logLevel --logName fully.qualified.class.path` See #17522 for more background info pr-link: #17532 dev docs for defining conventions for the new alluxio cli in golang pr-link: #17530 - Add commands to start/stop individual processes, ex. `bin/cli.sh process start master` - Define process interface and registry - process specific java opts are defined with the process and are dynamically added to env - Add journal format command as part of starting master process - Mount options for worker are not ported because they will be deprecated in the near future pr-link: #17561 Adds `info` subcommand, which contains: - `cache`: worker capacity information, calls the existing `bin/alluxio fsadmin report capacity` command - `collect`: collects cluster information into a single tarball, calls the existing `bin/alluxio collectInfo` command - `master`: master quorum information, calls the existing `bin/alluxio fs masterInfo` command - `report`: cluster information, calls the existing `bin/alluxio fsadmin report` command, excluding the `capacity` category pr-link: #17566 Add journal commands to CLI as part of #17522 pr-link: #17569 Add quorum/HA related commands as part of #17522 pr-link: #17570 Add fs commands to golang CLI as part of #17522 unlike other CLI commands, utilize arguments to maintain the existing structure of filesystem commands (ex. cp, du, ls, mv, rm, etc) pr-link: #17580 Add generate commands to golang CLI as part of #17522 bin/alluxio docGen -> bin/cli generate docs bin/alluxio bootstrapConf -> bin/cli generate template to read ALLUXIO_CONF_DIR, also conducted a minor change on env variables. pr-link: #17790 Add exec commands to golang CLI as part of #17522 `bin/alluxio runClass` -> `bin/cli exec class` `bin/alluxio runTests` -> `bin/cli exec testRun` `bin/alluxio runUfsTests` -> `bin/cli exec testUfs` `bin/alluxio runUfsIOTest` -> `bin/cli exec testUfsIO` `bin/alluxio runHdfsMountTests` -> `bin/cli exec testHdfsMount` `bin/alluxio runHmsTests` -> `bin/cli exec testHms` `bin/alluxio runJournalCrashTest` -> `bin/cli exec testJournalCrash` For command `exec test`, using different `--name` flags can lead to huge difference in other flags and options. To manage flags and options, currently using different commands for different test types. pr-link: #17797 Prepare golang cli for tarball build and replace bin/alluxio allow java 8 or 11 add cli binaries to other tarball profiles Add `process` commands with multiple nodes to golang CLI as part of #17522 `bin/alluxio-start.sh masters` -> `bin/cli.sh process start masters` `bin/alluxio-start.sh job_masters` -> `bin/cli.sh process start job_masters` `bin/alluxio-start.sh workers` -> `bin/cli.sh process start workers` `bin/alluxio-start.sh job_workers` -> `bin/cli.sh process start job_workers` `bin/alluxio-start.sh proxies` -> `bin/cli.sh process start proxies` `bin/alluxio-start.sh all` -> `bin/cli.sh process start all` `bin/alluxio-start.sh local` -> `bin/cli.sh process start local` `bin/alluxio-stop.sh masters` -> `bin/cli.sh process stop masters` `bin/alluxio-stop.sh job_masters` -> `bin/cli.sh process stop job_masters` `bin/alluxio-stop.sh workers` -> `bin/cli.sh process stop workers` `bin/alluxio-stop.sh job_workers` -> `bin/cli.sh process stop job_workers` `bin/alluxio-stop.sh proxies` -> `bin/cli.sh process stop proxies` `bin/alluxio-stop.sh all` -> `bin/cli.sh process stop all` `bin/alluxio-stop.sh local` -> `bin/cli.sh process stop local` For command `process start/stop` on multiple nodes, using crypto's `ssh` package to create an SSH session, connect to masters, workers or all nodes, then send according subcommand on single nodes to these nodes. pr-link: #17887 Add job commands to golang CLI as part of #17522 `bin/alluxio-bash job cancel id` -> `bin/alluxio job cancel --id` `bin/alluxio-bash job leader` -> `bin/alluxio job leader` `bin/alluxio-bash job ls` -> `bin/alluxio job list` `bin/alluxio-bash job getCmdStatus jobControlId` -> `bin/alluxio job cmdStatus --id` `bin/alluxio-bash job stat [-v] id` -> `bin/alluxio job jobStatus [-v] --id` `bin/alluxio-bash fs distributedCp src dst` -> `bin/alluxio job submit --type cp --src --dst` `bin/alluxio-bash fs distributedMv src dst` -> `bin/alluxio job submit --type mv --src --dst` `bin/alluxio-bash fs load path --submit` -> `bin/alluxio job load --path` pr-link: #17931 Clean up multiprocess code - refactor commands launching the same process on multiple hosts into a single struct - simplify the logic to ssh into different hosts with more straightforward handling of error channel - move names.go under its own package to avoid import cycles pr-link: #17947 Add an `info version` subcommand to golang CLI as part of #17522 `bin/alluxio-bash version` -> `bin/alluxio info version` pr-link: #17943 update calls to old bash scripts to refer to alluxio-bash Add `cache` commands to golang CLI as part of #17522 `bin/alluxio-bash formatWorker` -> `bin/cli cache format` `bin/alluxio-bash fs freeWorker` -> `bin/cli cache free --worker` `bin/alluxio-bash fs free` -> `bin/cli cache free [-f] --path` `bin/alluxio-bash fs persist` -> `bin/cli cache persist --path` `bin/alluxio-bash fs setTtl` -> `bin/cli cache ttl --set --duration --action` `bin/alluxio-bash fs unsetTtl` -> `bin/cli cache ttl --unset` pr-link: #17937 backcompat support for start/stop scripts skip mount args in start script backcompat remove unsupported cmds

twalluxio added 8 commits July 18, 2023 16:06

Add exec/testRun command to CLI

fbe7d6b

Add exec/testJournalCrash command to CLI

a935eb8

Add exec/testHms command to CLI

fe74f5d

Add exec/testHdfsMount command to CLI

9fc2b11

Add exec/testUfsIO command to CLI

769a48c

Add exec/testUfs command to CLI

947b6b1

Add exec/class command to CLI

fb4e105

Add exec service to CLI

49bd1a8

alluxio-bot added the cla-missing label Jul 18, 2023

Kai-Zhang requested a review from Xenorith July 18, 2023 08:39

Allow custom flags to input and read for class.go

87bd2bb

Kai-Zhang requested changes Jul 18, 2023

View reviewed changes

Kai-Zhang reviewed Jul 18, 2023

View reviewed changes

cli/src/alluxio.org/cli/cmd/exec/class.go Outdated Show resolved Hide resolved

cli/src/alluxio.org/cli/cmd/exec/class.go Show resolved Hide resolved

Updated class.go, solved requested change & minor updates

0c6f4ec

Zijian-Zhu removed the cla-missing label Jul 18, 2023

Xenorith requested changes Jul 19, 2023

View reviewed changes

Xenorith mentioned this pull request Jul 19, 2023

Add generate subcommand to CLI #17790

Merged

twalluxio added 3 commits July 19, 2023 09:37

removed unnecessary JavaClassName definition

f1ad788

removed fetchValue function

1fa7e3b

format update on imports

232e6d8

Xenorith reviewed Jul 19, 2023

View reviewed changes

cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go Outdated Show resolved Hide resolved

Update cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

85e453c

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Kai-Zhang reviewed Jul 25, 2023

View reviewed changes

Kai-Zhang added the type-devops Build, deploy, and maintain devops questions label Jul 25, 2023

Kai-Zhang reviewed Jul 25, 2023

View reviewed changes

cli/src/alluxio.org/cli/cmd/exec/test_ufsio.go Outdated Show resolved Hide resolved

Xenorith reviewed Jul 26, 2023

View reviewed changes

twalluxio added 2 commits July 26, 2023 11:32

omit flags

5bc78df

Implementing MarkFlagRequired

1bbc8a1

Update cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

c2323dd

Co-authored-by: Kai <cs.zhangkai@outlook.com>

Xenorith reviewed Jul 26, 2023

View reviewed changes

twalluxio and others added 6 commits July 27, 2023 09:30

remove error checks on cmd.MarkFlagRequired

6892de0

simplify logic in class.go

7282fa4

remove checking empty for required flags

122f0ba

rearrange import order

a07ab77

edit numeric flags to integer

b5a15e3

Update cli/src/alluxio.org/cli/cmd/exec/test_ufs.go

6a38ac7

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Xenorith requested changes Jul 27, 2023

View reviewed changes

twalluxio and others added 10 commits July 27, 2023 16:56

Update cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

fbf12e5

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Update cli/src/alluxio.org/cli/cmd/exec/test_hdfs_mount.go

822a93e

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Update cli/src/alluxio.org/cli/cmd/exec/test_hms.go

ac1b53f

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Update cli/src/alluxio.org/cli/cmd/exec/test_ufs.go

41c23f5

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

Update cli/src/alluxio.org/cli/cmd/exec/test_journal_crash.go

da09f6f

Co-authored-by: Rico Chiu <Xenorith@users.noreply.github.com>

fix bug in test_hdfs_mount.go

5b369f6

remove if not expecting arguments

997d157

add number check

9a7d443

move up flag --totalTime

012ba22

enforce no args

a3b691e

twalluxio requested review from Kai-Zhang and Xenorith July 31, 2023 09:11

Xenorith approved these changes Jul 31, 2023

View reviewed changes

Kai-Zhang approved these changes Aug 1, 2023

View reviewed changes

alluxio-bot merged commit 48dd042 into Alluxio:golangCli Aug 1, 2023
17 checks passed

twalluxio deleted the golangCli-exec branch August 1, 2023 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add exec subcommand to CLI #17797

Add exec subcommand to CLI #17797

twalluxio commented Jul 18, 2023

alluxio-bot commented Jul 18, 2023

Xenorith commented Jul 19, 2023

Xenorith left a comment

Xenorith left a comment

Xenorith Jul 27, 2023

Kai-Zhang commented Aug 1, 2023

Add exec subcommand to CLI #17797

Add exec subcommand to CLI #17797

Conversation

twalluxio commented Jul 18, 2023

alluxio-bot commented Jul 18, 2023

Xenorith commented Jul 19, 2023

Xenorith left a comment

Choose a reason for hiding this comment

Xenorith left a comment

Choose a reason for hiding this comment

Xenorith Jul 27, 2023

Choose a reason for hiding this comment

Kai-Zhang commented Aug 1, 2023