SNOW-828186 Remove code duplication in transaction.go #807
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-dheyman-1
requested review from
sfc-gh-pbulawa,
sfc-gh-pfus and
sfc-gh-igarish
sfc-gh-dheyman-1
force-pushed
the
SNOW-828186-transaction-code-duplication
branch
from
ba0564a
to
087bfd6
Compare
sfc-gh-dheyman-1
force-pushed
the
SNOW-828186-transaction-code-duplication
branch
from
087bfd6
to
6b408db
Compare
sfc-gh-pfus
reviewed
May 29, 2023
sfc-gh-pbulawa
approved these changes
May 29, 2023
sfc-gh-pfus
approved these changes
May 29, 2023
sureshmula2
added a commit
to sigmacomputing/gosnowflake
that referenced
this pull request
Aug 30, 2023
* SNOW-824297: Bumped up GoLang connector PATCH version from 1.6.20 to 1.6.21 (snowflakedb#800) * use the context object supplied by BeginTx instead of from the parent connection (snowflakedb#804) * Make port/portStr variables in cmd more consistent (snowflakedb#803) * Add tests to increase code coverage (snowflakedb#784) add tests to increase code coverage to baseline 80% * SNOW-828186 Remove code duplication in transaction.go (snowflakedb#807) * SNOW-829489: Fix CLA Assistant Github Action (snowflakedb#811) * SNOW-829187 Add multistatement query example (snowflakedb#810) * Add missing makefiles for example apps (snowflakedb#809) * Add multistatement exec example (snowflakedb#812) * SNOW-829454: Async API Example (snowflakedb#813) * SNOW-830268: Add PUT/GET feature example (snowflakedb#814) * SNOW-838450: Bumped up GoLang connector PATCH version from 1.6.21 to 1.6.22 (snowflakedb#820) @noreview - This is an automated process. No review is required Co-authored-by: Jenkins User <snowflake-provisioner> * SNOW-832816: Unify and remove duplicates of getDsn() function (snowflakedb#818) * SNOW-832885 Add arrow_batches example (snowflakedb#819) * Add null checks before accessing connection config during chunk downloading (snowflakedb#821) * SNOW-832816: Unify and remove duplicates of getDsn() function (snowflakedb#823) * Fix multistatement demo omitting first result set (snowflakedb#827) * SNOW-840123 Upgrade arrow 12.0.0 -> 12.0.1 (snowflakedb#829) * retry async request if still in progress (snowflakedb#824) update comment * SNOW-723750: Panic with readonly file system (snowflakedb#828) * add jenkins user for cla assistant check (snowflakedb#830) * add mutex lock to prevent race condition in exec() function (snowflakedb#834) defer mutex unlock * SNOW-847338: Add timeout for authentication in external browser (snowflakedb#835) * SNOW-852381: Add documentation and DSN parameter for external browser timeout (snowflakedb#837) * SNOW-853914 Add distributed fetch example (snowflakedb#840) * SNOW-832825: Fix async documentation (snowflakedb#839) * SNOW-856850: Add missing parameter check (snowflakedb#842) * SNOW-833537 Separate JWT http client to specify timeout (snowflakedb#841) * SNOW-847417: Added support for sql.Null types for query bind mapping (snowflakedb#844) * SNOW-833537 Each time retry keypair auth with new token (snowflakedb#845) * SNOW-857829 Fix username and password requiredness (snowflakedb#846) * fix(arrow): handle non-arrow result sets (snowflakedb#851) * SNOW-645253 Handle binding named parameters (snowflakedb#850) * SNOW-870818: Add Snyk to CLA Assistant allowlist (snowflakedb#854) * SNOW-870818: Add snyk-bot to CLA Assistant allowlist (snowflakedb#855) * SNOW-871839: Fix Snyk permissions (snowflakedb#857) * SNOW-859548 Reuse connection in tests (snowflakedb#856) * SNOW-859548 Replace test table in each test (snowflakedb#858) * SNOW-875425: Bumped up GoLang connector PATCH version from 1.6.22 to 1.6.23 (snowflakedb#861) * Enable procedure calling test (snowflakedb#863) * cover critical areas test code coverage for logger, dsn and connection (snowflakedb#866) * SNOW-833537 Add JWT retry docs (snowflakedb#868) * SNOW-880396: Do not cancel other matrix builds when one fails (snowflakedb#870) * SNOW-880396: Do not cancel other matrix builds when one fails * SNOW-880396: Cancel builds when new code in branch appears * increase code coverage on coverter.go (snowflakedb#867) * Remove executable bits in file open/creation Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> * SNOW-859548 Refactor runTests to runDBTest (snowflakedb#869) * SNOW-880442: Add pass context to arrow fetch function (snowflakedb#871) * refactor download files (snowflakedb#875) * SNOW-540086: Missing critical areas for code coverage (snowflakedb#873) * SNOW-870356: Integrate code coverage with CodeCov (snowflakedb#876) * SNOW-845282: Allow configuring tmpdir in DSN (snowflakedb#874) * SNOW-857660: Init rows location once (snowflakedb#882) * SNOW-845282: Add docs about tmpDirPath (snowflakedb#884) * SNOW-859547: Refactor tests to create a new test for each test case (snowflakedb#877) * SNOW-894815: Disable TestConcurrentReadOnParams (snowflakedb#886) * [SNOW-892549] Remove references to sfcdev1 (snowflakedb#885) * SNOW-897024: Bumped up GoLang connector PATCH version from 1.6.23 to 1.6.24 (snowflakedb#891) @noreview - This is an automated process. No review is required Co-authored-by: Jenkins User <snowflake-provisioner> * SNOW-895534: Add HTAP query context struct (snowflakedb#888) * SNOW-726742 Remove queryID from snowflakeConn (snowflakedb#892) * SNOW-898353 Fix snyk permissions (snowflakedb#893) * SNOW-889572 Refactored snowflake type implementation to map (snowflakedb#890) * SNOW-895534: Add HTAP query context entries to cache (snowflakedb#889) * SNOW-848019 Change global error objects to instantiated by function (snowflakedb#897) * SNOW-726742: Implement GetQueryId for statements (snowflakedb#899) * [Maintenance] Create Fork [Maintenance] enable CI tests in our repo (#20) add README note describing how CI secrets are set up change yaml to reference environment remove tests for platforms other than Ubuntu + AWS disable staticcheck add encrypted parameters file add private key disable tests that fail [Maintenance] Ignore vendored libs * [Fix] arrowToValue handling of timestamp_ltz at non-nanosecond scales (#24) * [Fix] SIG-18794: Fix handling of results response for canceled queries (#58) * [Feature] Add fetchers for result, monitoring, query-graph. * add hook to fetch "monitoring" data and query-graph, in goroutines * add support to fetching monitoring info * add monitoring threshold * move to Duration Lint/Build fixes nit: Goimports fixes - [Feature] [monitoring] fetch the query graph for slow queries (#26) * fetch query graph * fix * more fixes update interface move monitoring to a goroutine; client decides how long it'll wait (#27) add MonitoringResultFetcher to fetch monitoring (#48) - not use qid from SFconnection when fetch monitoring data (#52) * [Feature] allow clients to set QUERY_TAG parameter via context * [Fix] SIG-17456: Trap chunk-downloader errors with Sentry at Multiplex. (#44) * nit: Move "send error on channel" within the `recover` condition. * Improvement: don't panic _indiscriminately_, have a separate error-type that signals the internal panics we want to re-propagate. Co-authored-by: Agam Brahma <agam@sigmacomputing.com> * [Feature] Allow clients to specify any/null DataType explicitly - Allow clients to specify DataTypeNull explicitly (#50) * first cut of explict DataTypeNull * make lint * more tests - Allow client to specify any data type explicitly (#28) * dataTypeMode should work for all types * fix test * allow explicit binaryType declaration even if we're already using binaryType * silly attempt with *SnowflakeDataType * switch from *SnowflakeDataType --> SnowflakeDataType * bindings_test.go passes * add test for corner case * fix lint * fix null-handling tests * only need connection.CheckNamedValue * use checked cast instead of switch on type * [Feature] add a cache to the FetchResult codepath to avoid API calls (#45) lint: fix `ConnectionId` Fix CI for current master (#47) * lint: fix `ConnectionId` * Test fixes for previous change (`41c90a09`) * nit: tweaks to dsn-test Co-authored-by: Agam Brahma <agam@sigmacomputing.com> fix var inits (#46) * [Fix] SIG-18794: Fix getAsync() to not panic on context exceeded + test. (#56) * [Fix] shutdown race over access of the connection's "restful" structure. * [Feature] Improvements to Async handling SIG-18794: Refactor the monitoring fetcher & auto-cancel async (#57) * SIG-18794: Refactor the monitoring fetcher & auto-cancel async * better configuration * wait until the monitoring data refers to completed queries SIG-16907: Error out early when invalid state for multi-statement requests. (#43) Remove the debug-tracing added earlier. Co-authored-by: Agam Brahma <agam@sigmacomputing.com> SIG-18794: Make getAsync wait until the query completes or times out. (#59) * SIG-18794: Make getAsync wait until the query completes or times out. * make the getStatus for an async made query block until the ctx timeout SIG-18794: Fix a bug when looping to complete an async query (#60) * discovered a new error code the needs looping when async, enhanced the test * [Maintenance] Rebase: 6/6/22 reconcile with origin/master nit: fix build and gitignore dead code Lint fix nit: fix erroneous deletion nit: s/uuid/UUID nit: refactore use of monitoring fetcher for multi-stmt Refactor out older version of `getMonitoringResult` that didn't have the `endpoint` parameter * [Feature] Enable fetching raw Arrow records directly * [Feature][SIG-24903] Prototype for Snowflake SubmitSync (#81) Adds a SubmitSync method to the Snowflake driver that directly calls Snowflake's POST /queries/v1/query-request API endpoint synchronously, waiting for up to 45 seconds (the current, fixed timeout) but does not enter the "ping pong" phase of fetching query results for long-running queries. Instead, the caller is responsible for using the query ID to wait for and fetch query results. This prototype mainly exists right now so that we can test the relevant functionality and compare performance with the fully asynchronous mode of execution. * [Feature] Snowflake driver to support `WaitForQueryCompletion()`; wait for results to finish without returning result rows (#84) * [Feature] Enable extracting tokens from snowflake conn and constructing TokenAccessor (#86) * Add TokenGetter interface * Add public constructor for TokenAccessor * Expose FillMissingConfigParameters * Add comment * Add comments * Fix comment * [Maintenance] Rebase with upstream 2022-09-19 * If FuncGet() returns no code dont convert to str (#89) * test with context deadline (#92) * add panic Log whether or not we use the cache here (#95) * Log whether or not we use the cache here * Update monitoring.go * experiment * Update connection.go * make status error thing more clear an explicit * more logging * Don't assume error type * add more logging if request fails * pass qid * also tag code * Retry one more time & log response * log one more thing * skipCache * add the proper logging * retry on bad response * retry at most 2 times * lint * Dont check status before block * Don't cache non success responses (#109) Don't cache non success responses and debug the body response * Fix atoi error. (#110) * the result can return just Data field, so we parse a code as an error only if it is set as an error. * Revert "Don't cache non success responses" (#111) Revert "Don't cache non success responses (#109)" This reverts commit ff7d6b9. * fix incorrect error sending (#114) * fix res.Success in async.go (#115) * Add StateDuration to QueryMonitoringData struct (#116) * add logging for sf cache bug (#117) * add logging for sf cache bug * add log without data to see if prod is seeing this issue * Protect against nil and raise error instead (#118) * [Feature] Add support for client log context hooks (#119) * [Feature] Add support for client log context hooks * Pin honnef.co/go/stools/cmd/staticcheck dep version * Add comment * save changes * fix up * bump down go * put back build test to what it was * just test on linux like we used to * fix status * update test back to what it was * Add a nil check for snowflakeRows.Close (#123) * Only cache successful API responses (#124) * Only cache success responses * Add another if * [Maint] squash 8 commits from my last rebase (#125) * rebase master with head * Add a nil check for snowflakeRows.Close (#123) * Only cache successful API responses (#124) * Only cache success responses * Add another if --------- Co-authored-by: Eric Bannatyne <aldld@users.noreply.github.com> * fmt * use unsupported type * use TimestampNtz as default when the client does not specify data type explicitly * [SIG-35025] Add a nil check in snowflake driver heartbeat function (#133) * Add a nil check in snowflake driver heartbeat function * Update heartbeat.go * [SIG-36502]add more visibility into chunk downloader (#132) * Add status on snowflakes status result so we know if the query is queued in the warehouse (#139) * proof of concept: materialization more info * add status to status response * Don't require username & password if using token accessor (#144) * dont require user & pass for token accessor type * undo rows commit * undo connection commit * Adding Changes of sf_stable_master to master branch (#147) * Add more visibility in chunk downloader * Added changes that are on stable branch but not on master * Added comments to pass lint * remove dependency on arrow v0 (#148) * [SIG-40301] fix integer value not in range error (#149) * use bigFloat when convert int64 to float (#150) * [SIG-40455] fix arrow batch int64 to ntz/ltz (#151) * add arrowToRecord tz tests (#152) * [SIG-40738] convert binary array to string array (#153) * [SIG-40820]don't panic if type mismatch is decode arrow (#154) * Revert "[SIG-40738] convert binary array to string array" (#155) * [SIG-41137] remove decode fixed types from arrowToRecord (#156) * fix recordToSchema and not do conversion for fixedType (#157) * [Security] Remove logging sql text from the snowflake driver (part 2 of 2) (#158) Remove logging sql text from the snowflake driver * Update connection.go * reconcile merge changes * Update bindings_test.go * Increase test timeout to 60 mins to see if it helps with timeout * Fix a hanging test. Change timeout to 30s again. * Fix DSN Test * Fix DSN TESTS * Update submit_sync_test.go * fix TestFunctionParameters and TestValueToString * Revert "SNOW-645253 Handle binding named parameters (snowflakedb#850)" This reverts commit 67ec6cf. * Revert "Revert "SNOW-645253 Handle binding named parameters (snowflakedb#850)"" This reverts commit a6a8a01. * Fix DSN for external Browser * dont break things * Final Fix --------- Co-authored-by: Kiran Dama <69480841+sfc-gh-kdama@users.noreply.github.com> Co-authored-by: Lorna <115649563+sfc-gh-ext-simba-lb@users.noreply.github.com> Co-authored-by: Piotr Fus <piotr.fus@snowflake.com> Co-authored-by: Dawid Heyman <dawid.heyman@snowflake.com> Co-authored-by: Piotr Bulawa <piotr.bulawa@snowflake.com> Co-authored-by: Angel Antonio Avalos Cisneros <angel.avalos@snowflake.com> Co-authored-by: Matt Topol <zotthewizard@gmail.com> Co-authored-by: Dominik Przybysz <132913826+sfc-gh-dprzybysz@users.noreply.github.com> Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> Co-authored-by: Srikanth Reddy Kumbham <s.kumbham@snowflake.com> Co-authored-by: Agam Brahma <agam@sigmacomputing.com> Co-authored-by: Luke Paulsen <luke@sigmacomputing.com> Co-authored-by: Mihai Claudiu Toader <mihai@sigmacomputing.com> Co-authored-by: Yifeng-Sigma <yifeng@sigmacomputing.com> Co-authored-by: Jack Qian <jsjqian@users.noreply.github.com> Co-authored-by: Agam Brahma <agam.brahma+git@gmail.com> Co-authored-by: Greg Owen <GregOwen@users.noreply.github.com> Co-authored-by: Max Seiden <max@sigmacomputing.com> Co-authored-by: fengqingthu <qing.feng@sigmacomputing.com> Co-authored-by: Eric Bannatyne <aldld@users.noreply.github.com> Co-authored-by: Madison Chamberlain <46542378+madisonchamberlain@users.noreply.github.com> Co-authored-by: Eric Bannatyne <eric@sigmacomputing.com> Co-authored-by: mansap22 <110418152+mansap22@users.noreply.github.com> Co-authored-by: Ayman Elkfrawy <120422207+ayman-sigma@users.noreply.github.com> Co-authored-by: GregOwen <greg@sigmacomputing.com> Co-authored-by: sureshmula2 <126016714+sureshmula2@users.noreply.github.com> Co-authored-by: Ryan Kwong <ryankwong@ryank01-sigma.lan> Co-authored-by: Madison Chamberlain <madisonchamberlain@madisonc01-sigma.local>
rajsigma
added a commit
to sigmacomputing/gosnowflake
that referenced
this pull request
Sep 8, 2023
* SNOW-824297: Bumped up GoLang connector PATCH version from 1.6.20 to 1.6.21 (snowflakedb#800) * use the context object supplied by BeginTx instead of from the parent connection (snowflakedb#804) * Make port/portStr variables in cmd more consistent (snowflakedb#803) * Add tests to increase code coverage (snowflakedb#784) add tests to increase code coverage to baseline 80% * SNOW-828186 Remove code duplication in transaction.go (snowflakedb#807) * SNOW-829489: Fix CLA Assistant Github Action (snowflakedb#811) * SNOW-829187 Add multistatement query example (snowflakedb#810) * Add missing makefiles for example apps (snowflakedb#809) * Add multistatement exec example (snowflakedb#812) * SNOW-829454: Async API Example (snowflakedb#813) * SNOW-830268: Add PUT/GET feature example (snowflakedb#814) * SNOW-838450: Bumped up GoLang connector PATCH version from 1.6.21 to 1.6.22 (snowflakedb#820) @noreview - This is an automated process. No review is required Co-authored-by: Jenkins User <snowflake-provisioner> * SNOW-832816: Unify and remove duplicates of getDsn() function (snowflakedb#818) * SNOW-832885 Add arrow_batches example (snowflakedb#819) * Add null checks before accessing connection config during chunk downloading (snowflakedb#821) * SNOW-832816: Unify and remove duplicates of getDsn() function (snowflakedb#823) * Fix multistatement demo omitting first result set (snowflakedb#827) * SNOW-840123 Upgrade arrow 12.0.0 -> 12.0.1 (snowflakedb#829) * retry async request if still in progress (snowflakedb#824) update comment * SNOW-723750: Panic with readonly file system (snowflakedb#828) * add jenkins user for cla assistant check (snowflakedb#830) * add mutex lock to prevent race condition in exec() function (snowflakedb#834) defer mutex unlock * SNOW-847338: Add timeout for authentication in external browser (snowflakedb#835) * SNOW-852381: Add documentation and DSN parameter for external browser timeout (snowflakedb#837) * SNOW-853914 Add distributed fetch example (snowflakedb#840) * SNOW-832825: Fix async documentation (snowflakedb#839) * SNOW-856850: Add missing parameter check (snowflakedb#842) * SNOW-833537 Separate JWT http client to specify timeout (snowflakedb#841) * SNOW-847417: Added support for sql.Null types for query bind mapping (snowflakedb#844) * SNOW-833537 Each time retry keypair auth with new token (snowflakedb#845) * SNOW-857829 Fix username and password requiredness (snowflakedb#846) * fix(arrow): handle non-arrow result sets (snowflakedb#851) * SNOW-645253 Handle binding named parameters (snowflakedb#850) * SNOW-870818: Add Snyk to CLA Assistant allowlist (snowflakedb#854) * SNOW-870818: Add snyk-bot to CLA Assistant allowlist (snowflakedb#855) * SNOW-871839: Fix Snyk permissions (snowflakedb#857) * SNOW-859548 Reuse connection in tests (snowflakedb#856) * SNOW-859548 Replace test table in each test (snowflakedb#858) * SNOW-875425: Bumped up GoLang connector PATCH version from 1.6.22 to 1.6.23 (snowflakedb#861) * Enable procedure calling test (snowflakedb#863) * cover critical areas test code coverage for logger, dsn and connection (snowflakedb#866) * SNOW-833537 Add JWT retry docs (snowflakedb#868) * SNOW-880396: Do not cancel other matrix builds when one fails (snowflakedb#870) * SNOW-880396: Do not cancel other matrix builds when one fails * SNOW-880396: Cancel builds when new code in branch appears * increase code coverage on coverter.go (snowflakedb#867) * Remove executable bits in file open/creation Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> * SNOW-859548 Refactor runTests to runDBTest (snowflakedb#869) * SNOW-880442: Add pass context to arrow fetch function (snowflakedb#871) * refactor download files (snowflakedb#875) * SNOW-540086: Missing critical areas for code coverage (snowflakedb#873) * SNOW-870356: Integrate code coverage with CodeCov (snowflakedb#876) * SNOW-845282: Allow configuring tmpdir in DSN (snowflakedb#874) * SNOW-857660: Init rows location once (snowflakedb#882) * SNOW-845282: Add docs about tmpDirPath (snowflakedb#884) * SNOW-859547: Refactor tests to create a new test for each test case (snowflakedb#877) * SNOW-894815: Disable TestConcurrentReadOnParams (snowflakedb#886) * [SNOW-892549] Remove references to sfcdev1 (snowflakedb#885) * SNOW-897024: Bumped up GoLang connector PATCH version from 1.6.23 to 1.6.24 (snowflakedb#891) @noreview - This is an automated process. No review is required Co-authored-by: Jenkins User <snowflake-provisioner> * SNOW-895534: Add HTAP query context struct (snowflakedb#888) * SNOW-726742 Remove queryID from snowflakeConn (snowflakedb#892) * SNOW-898353 Fix snyk permissions (snowflakedb#893) * SNOW-889572 Refactored snowflake type implementation to map (snowflakedb#890) * SNOW-895534: Add HTAP query context entries to cache (snowflakedb#889) * SNOW-848019 Change global error objects to instantiated by function (snowflakedb#897) * SNOW-726742: Implement GetQueryId for statements (snowflakedb#899) * [Maintenance] Create Fork [Maintenance] enable CI tests in our repo (#20) add README note describing how CI secrets are set up change yaml to reference environment remove tests for platforms other than Ubuntu + AWS disable staticcheck add encrypted parameters file add private key disable tests that fail [Maintenance] Ignore vendored libs * [Fix] arrowToValue handling of timestamp_ltz at non-nanosecond scales (#24) * [Fix] SIG-18794: Fix handling of results response for canceled queries (#58) * [Feature] Add fetchers for result, monitoring, query-graph. * add hook to fetch "monitoring" data and query-graph, in goroutines * add support to fetching monitoring info * add monitoring threshold * move to Duration Lint/Build fixes nit: Goimports fixes - [Feature] [monitoring] fetch the query graph for slow queries (#26) * fetch query graph * fix * more fixes update interface move monitoring to a goroutine; client decides how long it'll wait (#27) add MonitoringResultFetcher to fetch monitoring (#48) - not use qid from SFconnection when fetch monitoring data (#52) * [Feature] allow clients to set QUERY_TAG parameter via context * [Fix] SIG-17456: Trap chunk-downloader errors with Sentry at Multiplex. (#44) * nit: Move "send error on channel" within the `recover` condition. * Improvement: don't panic _indiscriminately_, have a separate error-type that signals the internal panics we want to re-propagate. Co-authored-by: Agam Brahma <agam@sigmacomputing.com> * [Feature] Allow clients to specify any/null DataType explicitly - Allow clients to specify DataTypeNull explicitly (#50) * first cut of explict DataTypeNull * make lint * more tests - Allow client to specify any data type explicitly (#28) * dataTypeMode should work for all types * fix test * allow explicit binaryType declaration even if we're already using binaryType * silly attempt with *SnowflakeDataType * switch from *SnowflakeDataType --> SnowflakeDataType * bindings_test.go passes * add test for corner case * fix lint * fix null-handling tests * only need connection.CheckNamedValue * use checked cast instead of switch on type * [Feature] add a cache to the FetchResult codepath to avoid API calls (#45) lint: fix `ConnectionId` Fix CI for current master (#47) * lint: fix `ConnectionId` * Test fixes for previous change (`41c90a09`) * nit: tweaks to dsn-test Co-authored-by: Agam Brahma <agam@sigmacomputing.com> fix var inits (#46) * [Fix] SIG-18794: Fix getAsync() to not panic on context exceeded + test. (#56) * [Fix] shutdown race over access of the connection's "restful" structure. * [Feature] Improvements to Async handling SIG-18794: Refactor the monitoring fetcher & auto-cancel async (#57) * SIG-18794: Refactor the monitoring fetcher & auto-cancel async * better configuration * wait until the monitoring data refers to completed queries SIG-16907: Error out early when invalid state for multi-statement requests. (#43) Remove the debug-tracing added earlier. Co-authored-by: Agam Brahma <agam@sigmacomputing.com> SIG-18794: Make getAsync wait until the query completes or times out. (#59) * SIG-18794: Make getAsync wait until the query completes or times out. * make the getStatus for an async made query block until the ctx timeout SIG-18794: Fix a bug when looping to complete an async query (#60) * discovered a new error code the needs looping when async, enhanced the test * [Maintenance] Rebase: 6/6/22 reconcile with origin/master nit: fix build and gitignore dead code Lint fix nit: fix erroneous deletion nit: s/uuid/UUID nit: refactore use of monitoring fetcher for multi-stmt Refactor out older version of `getMonitoringResult` that didn't have the `endpoint` parameter * [Feature] Enable fetching raw Arrow records directly * [Feature][SIG-24903] Prototype for Snowflake SubmitSync (#81) Adds a SubmitSync method to the Snowflake driver that directly calls Snowflake's POST /queries/v1/query-request API endpoint synchronously, waiting for up to 45 seconds (the current, fixed timeout) but does not enter the "ping pong" phase of fetching query results for long-running queries. Instead, the caller is responsible for using the query ID to wait for and fetch query results. This prototype mainly exists right now so that we can test the relevant functionality and compare performance with the fully asynchronous mode of execution. * [Feature] Snowflake driver to support `WaitForQueryCompletion()`; wait for results to finish without returning result rows (#84) * [Feature] Enable extracting tokens from snowflake conn and constructing TokenAccessor (#86) * Add TokenGetter interface * Add public constructor for TokenAccessor * Expose FillMissingConfigParameters * Add comment * Add comments * Fix comment * [Maintenance] Rebase with upstream 2022-09-19 * If FuncGet() returns no code dont convert to str (#89) * test with context deadline (#92) * add panic Log whether or not we use the cache here (#95) * Log whether or not we use the cache here * Update monitoring.go * experiment * Update connection.go * make status error thing more clear an explicit * more logging * Don't assume error type * add more logging if request fails * pass qid * also tag code * Retry one more time & log response * log one more thing * skipCache * add the proper logging * retry on bad response * retry at most 2 times * lint * Dont check status before block * Don't cache non success responses (#109) Don't cache non success responses and debug the body response * Fix atoi error. (#110) * the result can return just Data field, so we parse a code as an error only if it is set as an error. * Revert "Don't cache non success responses" (#111) Revert "Don't cache non success responses (#109)" This reverts commit ff7d6b9. * fix incorrect error sending (#114) * fix res.Success in async.go (#115) * Add StateDuration to QueryMonitoringData struct (#116) * add logging for sf cache bug (#117) * add logging for sf cache bug * add log without data to see if prod is seeing this issue * Protect against nil and raise error instead (#118) * [Feature] Add support for client log context hooks (#119) * [Feature] Add support for client log context hooks * Pin honnef.co/go/stools/cmd/staticcheck dep version * Add comment * save changes * fix up * bump down go * put back build test to what it was * just test on linux like we used to * fix status * update test back to what it was * Add a nil check for snowflakeRows.Close (#123) * Only cache successful API responses (#124) * Only cache success responses * Add another if * [Maint] squash 8 commits from my last rebase (#125) * rebase master with head * Add a nil check for snowflakeRows.Close (#123) * Only cache successful API responses (#124) * Only cache success responses * Add another if --------- Co-authored-by: Eric Bannatyne <aldld@users.noreply.github.com> * fmt * use unsupported type * use TimestampNtz as default when the client does not specify data type explicitly * [SIG-35025] Add a nil check in snowflake driver heartbeat function (#133) * Add a nil check in snowflake driver heartbeat function * Update heartbeat.go * [SIG-36502]add more visibility into chunk downloader (#132) * Add status on snowflakes status result so we know if the query is queued in the warehouse (#139) * proof of concept: materialization more info * add status to status response * Don't require username & password if using token accessor (#144) * dont require user & pass for token accessor type * undo rows commit * undo connection commit * Adding Changes of sf_stable_master to master branch (#147) * Add more visibility in chunk downloader * Added changes that are on stable branch but not on master * Added comments to pass lint * remove dependency on arrow v0 (#148) * [SIG-40301] fix integer value not in range error (#149) * use bigFloat when convert int64 to float (#150) * [SIG-40455] fix arrow batch int64 to ntz/ltz (#151) * add arrowToRecord tz tests (#152) * [SIG-40738] convert binary array to string array (#153) * [SIG-40820]don't panic if type mismatch is decode arrow (#154) * Revert "[SIG-40738] convert binary array to string array" (#155) * [SIG-41137] remove decode fixed types from arrowToRecord (#156) * fix recordToSchema and not do conversion for fixedType (#157) * [Security] Remove logging sql text from the snowflake driver (part 2 of 2) (#158) Remove logging sql text from the snowflake driver * Update connection.go * reconcile merge changes * Update bindings_test.go * Increase test timeout to 60 mins to see if it helps with timeout * Fix a hanging test. Change timeout to 30s again. * Fix DSN Test * Fix DSN TESTS * Update submit_sync_test.go * fix TestFunctionParameters and TestValueToString * Revert "SNOW-645253 Handle binding named parameters (snowflakedb#850)" This reverts commit 67ec6cf. * Revert "Revert "SNOW-645253 Handle binding named parameters (snowflakedb#850)"" This reverts commit a6a8a01. * Fix DSN for external Browser * dont break things * Final Fix --------- Co-authored-by: Kiran Dama <69480841+sfc-gh-kdama@users.noreply.github.com> Co-authored-by: Lorna <115649563+sfc-gh-ext-simba-lb@users.noreply.github.com> Co-authored-by: Piotr Fus <piotr.fus@snowflake.com> Co-authored-by: Dawid Heyman <dawid.heyman@snowflake.com> Co-authored-by: Piotr Bulawa <piotr.bulawa@snowflake.com> Co-authored-by: Angel Antonio Avalos Cisneros <angel.avalos@snowflake.com> Co-authored-by: Matt Topol <zotthewizard@gmail.com> Co-authored-by: Dominik Przybysz <132913826+sfc-gh-dprzybysz@users.noreply.github.com> Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> Co-authored-by: Srikanth Reddy Kumbham <s.kumbham@snowflake.com> Co-authored-by: Agam Brahma <agam@sigmacomputing.com> Co-authored-by: Luke Paulsen <luke@sigmacomputing.com> Co-authored-by: Mihai Claudiu Toader <mihai@sigmacomputing.com> Co-authored-by: Yifeng-Sigma <yifeng@sigmacomputing.com> Co-authored-by: Jack Qian <jsjqian@users.noreply.github.com> Co-authored-by: Agam Brahma <agam.brahma+git@gmail.com> Co-authored-by: Greg Owen <GregOwen@users.noreply.github.com> Co-authored-by: Max Seiden <max@sigmacomputing.com> Co-authored-by: fengqingthu <qing.feng@sigmacomputing.com> Co-authored-by: Eric Bannatyne <aldld@users.noreply.github.com> Co-authored-by: Madison Chamberlain <46542378+madisonchamberlain@users.noreply.github.com> Co-authored-by: Eric Bannatyne <eric@sigmacomputing.com> Co-authored-by: mansap22 <110418152+mansap22@users.noreply.github.com> Co-authored-by: Ayman Elkfrawy <120422207+ayman-sigma@users.noreply.github.com> Co-authored-by: GregOwen <greg@sigmacomputing.com> Co-authored-by: sureshmula2 <126016714+sureshmula2@users.noreply.github.com> Co-authored-by: Ryan Kwong <ryankwong@ryank01-sigma.lan> Co-authored-by: Madison Chamberlain <madisonchamberlain@madisonc01-sigma.local>
rajsigma
added a commit
to sigmacomputing/gosnowflake
that referenced
this pull request
Sep 8, 2023
* SNOW-824297: Bumped up GoLang connector PATCH version from 1.6.20 to 1.6.21 (snowflakedb#800) * use the context object supplied by BeginTx instead of from the parent connection (snowflakedb#804) * Make port/portStr variables in cmd more consistent (snowflakedb#803) * Add tests to increase code coverage (snowflakedb#784) add tests to increase code coverage to baseline 80% * SNOW-828186 Remove code duplication in transaction.go (snowflakedb#807) * SNOW-829489: Fix CLA Assistant Github Action (snowflakedb#811) * SNOW-829187 Add multistatement query example (snowflakedb#810) * Add missing makefiles for example apps (snowflakedb#809) * Add multistatement exec example (snowflakedb#812) * SNOW-829454: Async API Example (snowflakedb#813) * SNOW-830268: Add PUT/GET feature example (snowflakedb#814) * SNOW-838450: Bumped up GoLang connector PATCH version from 1.6.21 to 1.6.22 (snowflakedb#820) @noreview - This is an automated process. No review is required Co-authored-by: Jenkins User <snowflake-provisioner> * SNOW-832816: Unify and remove duplicates of getDsn() function (snowflakedb#818) * SNOW-832885 Add arrow_batches example (snowflakedb#819) * Add null checks before accessing connection config during chunk downloading (snowflakedb#821) * SNOW-832816: Unify and remove duplicates of getDsn() function (snowflakedb#823) * Fix multistatement demo omitting first result set (snowflakedb#827) * SNOW-840123 Upgrade arrow 12.0.0 -> 12.0.1 (snowflakedb#829) * retry async request if still in progress (snowflakedb#824) update comment * SNOW-723750: Panic with readonly file system (snowflakedb#828) * add jenkins user for cla assistant check (snowflakedb#830) * add mutex lock to prevent race condition in exec() function (snowflakedb#834) defer mutex unlock * SNOW-847338: Add timeout for authentication in external browser (snowflakedb#835) * SNOW-852381: Add documentation and DSN parameter for external browser timeout (snowflakedb#837) * SNOW-853914 Add distributed fetch example (snowflakedb#840) * SNOW-832825: Fix async documentation (snowflakedb#839) * SNOW-856850: Add missing parameter check (snowflakedb#842) * SNOW-833537 Separate JWT http client to specify timeout (snowflakedb#841) * SNOW-847417: Added support for sql.Null types for query bind mapping (snowflakedb#844) * SNOW-833537 Each time retry keypair auth with new token (snowflakedb#845) * SNOW-857829 Fix username and password requiredness (snowflakedb#846) * fix(arrow): handle non-arrow result sets (snowflakedb#851) * SNOW-645253 Handle binding named parameters (snowflakedb#850) * SNOW-870818: Add Snyk to CLA Assistant allowlist (snowflakedb#854) * SNOW-870818: Add snyk-bot to CLA Assistant allowlist (snowflakedb#855) * SNOW-871839: Fix Snyk permissions (snowflakedb#857) * SNOW-859548 Reuse connection in tests (snowflakedb#856) * SNOW-859548 Replace test table in each test (snowflakedb#858) * SNOW-875425: Bumped up GoLang connector PATCH version from 1.6.22 to 1.6.23 (snowflakedb#861) * Enable procedure calling test (snowflakedb#863) * cover critical areas test code coverage for logger, dsn and connection (snowflakedb#866) * SNOW-833537 Add JWT retry docs (snowflakedb#868) * SNOW-880396: Do not cancel other matrix builds when one fails (snowflakedb#870) * SNOW-880396: Do not cancel other matrix builds when one fails * SNOW-880396: Cancel builds when new code in branch appears * increase code coverage on coverter.go (snowflakedb#867) * Remove executable bits in file open/creation Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> * SNOW-859548 Refactor runTests to runDBTest (snowflakedb#869) * SNOW-880442: Add pass context to arrow fetch function (snowflakedb#871) * refactor download files (snowflakedb#875) * SNOW-540086: Missing critical areas for code coverage (snowflakedb#873) * SNOW-870356: Integrate code coverage with CodeCov (snowflakedb#876) * SNOW-845282: Allow configuring tmpdir in DSN (snowflakedb#874) * SNOW-857660: Init rows location once (snowflakedb#882) * SNOW-845282: Add docs about tmpDirPath (snowflakedb#884) * SNOW-859547: Refactor tests to create a new test for each test case (snowflakedb#877) * SNOW-894815: Disable TestConcurrentReadOnParams (snowflakedb#886) * [SNOW-892549] Remove references to sfcdev1 (snowflakedb#885) * SNOW-897024: Bumped up GoLang connector PATCH version from 1.6.23 to 1.6.24 (snowflakedb#891) @noreview - This is an automated process. No review is required Co-authored-by: Jenkins User <snowflake-provisioner> * SNOW-895534: Add HTAP query context struct (snowflakedb#888) * SNOW-726742 Remove queryID from snowflakeConn (snowflakedb#892) * SNOW-898353 Fix snyk permissions (snowflakedb#893) * SNOW-889572 Refactored snowflake type implementation to map (snowflakedb#890) * SNOW-895534: Add HTAP query context entries to cache (snowflakedb#889) * SNOW-848019 Change global error objects to instantiated by function (snowflakedb#897) * SNOW-726742: Implement GetQueryId for statements (snowflakedb#899) * [Maintenance] Create Fork [Maintenance] enable CI tests in our repo (#20) add README note describing how CI secrets are set up change yaml to reference environment remove tests for platforms other than Ubuntu + AWS disable staticcheck add encrypted parameters file add private key disable tests that fail [Maintenance] Ignore vendored libs * [Fix] arrowToValue handling of timestamp_ltz at non-nanosecond scales (#24) * [Fix] SIG-18794: Fix handling of results response for canceled queries (#58) * [Feature] Add fetchers for result, monitoring, query-graph. * add hook to fetch "monitoring" data and query-graph, in goroutines * add support to fetching monitoring info * add monitoring threshold * move to Duration Lint/Build fixes nit: Goimports fixes - [Feature] [monitoring] fetch the query graph for slow queries (#26) * fetch query graph * fix * more fixes update interface move monitoring to a goroutine; client decides how long it'll wait (#27) add MonitoringResultFetcher to fetch monitoring (#48) - not use qid from SFconnection when fetch monitoring data (#52) * [Feature] allow clients to set QUERY_TAG parameter via context * [Fix] SIG-17456: Trap chunk-downloader errors with Sentry at Multiplex. (#44) * nit: Move "send error on channel" within the `recover` condition. * Improvement: don't panic _indiscriminately_, have a separate error-type that signals the internal panics we want to re-propagate. Co-authored-by: Agam Brahma <agam@sigmacomputing.com> * [Feature] Allow clients to specify any/null DataType explicitly - Allow clients to specify DataTypeNull explicitly (#50) * first cut of explict DataTypeNull * make lint * more tests - Allow client to specify any data type explicitly (#28) * dataTypeMode should work for all types * fix test * allow explicit binaryType declaration even if we're already using binaryType * silly attempt with *SnowflakeDataType * switch from *SnowflakeDataType --> SnowflakeDataType * bindings_test.go passes * add test for corner case * fix lint * fix null-handling tests * only need connection.CheckNamedValue * use checked cast instead of switch on type * [Feature] add a cache to the FetchResult codepath to avoid API calls (#45) lint: fix `ConnectionId` Fix CI for current master (#47) * lint: fix `ConnectionId` * Test fixes for previous change (`41c90a09`) * nit: tweaks to dsn-test Co-authored-by: Agam Brahma <agam@sigmacomputing.com> fix var inits (#46) * [Fix] SIG-18794: Fix getAsync() to not panic on context exceeded + test. (#56) * [Fix] shutdown race over access of the connection's "restful" structure. * [Feature] Improvements to Async handling SIG-18794: Refactor the monitoring fetcher & auto-cancel async (#57) * SIG-18794: Refactor the monitoring fetcher & auto-cancel async * better configuration * wait until the monitoring data refers to completed queries SIG-16907: Error out early when invalid state for multi-statement requests. (#43) Remove the debug-tracing added earlier. Co-authored-by: Agam Brahma <agam@sigmacomputing.com> SIG-18794: Make getAsync wait until the query completes or times out. (#59) * SIG-18794: Make getAsync wait until the query completes or times out. * make the getStatus for an async made query block until the ctx timeout SIG-18794: Fix a bug when looping to complete an async query (#60) * discovered a new error code the needs looping when async, enhanced the test * [Maintenance] Rebase: 6/6/22 reconcile with origin/master nit: fix build and gitignore dead code Lint fix nit: fix erroneous deletion nit: s/uuid/UUID nit: refactore use of monitoring fetcher for multi-stmt Refactor out older version of `getMonitoringResult` that didn't have the `endpoint` parameter * [Feature] Enable fetching raw Arrow records directly * [Feature][SIG-24903] Prototype for Snowflake SubmitSync (#81) Adds a SubmitSync method to the Snowflake driver that directly calls Snowflake's POST /queries/v1/query-request API endpoint synchronously, waiting for up to 45 seconds (the current, fixed timeout) but does not enter the "ping pong" phase of fetching query results for long-running queries. Instead, the caller is responsible for using the query ID to wait for and fetch query results. This prototype mainly exists right now so that we can test the relevant functionality and compare performance with the fully asynchronous mode of execution. * [Feature] Snowflake driver to support `WaitForQueryCompletion()`; wait for results to finish without returning result rows (#84) * [Feature] Enable extracting tokens from snowflake conn and constructing TokenAccessor (#86) * Add TokenGetter interface * Add public constructor for TokenAccessor * Expose FillMissingConfigParameters * Add comment * Add comments * Fix comment * [Maintenance] Rebase with upstream 2022-09-19 * If FuncGet() returns no code dont convert to str (#89) * test with context deadline (#92) * add panic Log whether or not we use the cache here (#95) * Log whether or not we use the cache here * Update monitoring.go * experiment * Update connection.go * make status error thing more clear an explicit * more logging * Don't assume error type * add more logging if request fails * pass qid * also tag code * Retry one more time & log response * log one more thing * skipCache * add the proper logging * retry on bad response * retry at most 2 times * lint * Dont check status before block * Don't cache non success responses (#109) Don't cache non success responses and debug the body response * Fix atoi error. (#110) * the result can return just Data field, so we parse a code as an error only if it is set as an error. * Revert "Don't cache non success responses" (#111) Revert "Don't cache non success responses (#109)" This reverts commit ff7d6b9. * fix incorrect error sending (#114) * fix res.Success in async.go (#115) * Add StateDuration to QueryMonitoringData struct (#116) * add logging for sf cache bug (#117) * add logging for sf cache bug * add log without data to see if prod is seeing this issue * Protect against nil and raise error instead (#118) * [Feature] Add support for client log context hooks (#119) * [Feature] Add support for client log context hooks * Pin honnef.co/go/stools/cmd/staticcheck dep version * Add comment * save changes * fix up * bump down go * put back build test to what it was * just test on linux like we used to * fix status * update test back to what it was * Add a nil check for snowflakeRows.Close (#123) * Only cache successful API responses (#124) * Only cache success responses * Add another if * [Maint] squash 8 commits from my last rebase (#125) * rebase master with head * Add a nil check for snowflakeRows.Close (#123) * Only cache successful API responses (#124) * Only cache success responses * Add another if --------- Co-authored-by: Eric Bannatyne <aldld@users.noreply.github.com> * fmt * use unsupported type * use TimestampNtz as default when the client does not specify data type explicitly * [SIG-35025] Add a nil check in snowflake driver heartbeat function (#133) * Add a nil check in snowflake driver heartbeat function * Update heartbeat.go * [SIG-36502]add more visibility into chunk downloader (#132) * Add status on snowflakes status result so we know if the query is queued in the warehouse (#139) * proof of concept: materialization more info * add status to status response * Don't require username & password if using token accessor (#144) * dont require user & pass for token accessor type * undo rows commit * undo connection commit * Adding Changes of sf_stable_master to master branch (#147) * Add more visibility in chunk downloader * Added changes that are on stable branch but not on master * Added comments to pass lint * remove dependency on arrow v0 (#148) * [SIG-40301] fix integer value not in range error (#149) * use bigFloat when convert int64 to float (#150) * [SIG-40455] fix arrow batch int64 to ntz/ltz (#151) * add arrowToRecord tz tests (#152) * [SIG-40738] convert binary array to string array (#153) * [SIG-40820]don't panic if type mismatch is decode arrow (#154) * Revert "[SIG-40738] convert binary array to string array" (#155) * [SIG-41137] remove decode fixed types from arrowToRecord (#156) * fix recordToSchema and not do conversion for fixedType (#157) * [Security] Remove logging sql text from the snowflake driver (part 2 of 2) (#158) Remove logging sql text from the snowflake driver * Update connection.go * reconcile merge changes * Update bindings_test.go * Increase test timeout to 60 mins to see if it helps with timeout * Fix a hanging test. Change timeout to 30s again. * Fix DSN Test * Fix DSN TESTS * Update submit_sync_test.go * fix TestFunctionParameters and TestValueToString * Revert "SNOW-645253 Handle binding named parameters (snowflakedb#850)" This reverts commit 67ec6cf. * Revert "Revert "SNOW-645253 Handle binding named parameters (snowflakedb#850)"" This reverts commit a6a8a01. * Fix DSN for external Browser * dont break things * Final Fix --------- Co-authored-by: Kiran Dama <69480841+sfc-gh-kdama@users.noreply.github.com> Co-authored-by: Lorna <115649563+sfc-gh-ext-simba-lb@users.noreply.github.com> Co-authored-by: Piotr Fus <piotr.fus@snowflake.com> Co-authored-by: Dawid Heyman <dawid.heyman@snowflake.com> Co-authored-by: Piotr Bulawa <piotr.bulawa@snowflake.com> Co-authored-by: Angel Antonio Avalos Cisneros <angel.avalos@snowflake.com> Co-authored-by: Matt Topol <zotthewizard@gmail.com> Co-authored-by: Dominik Przybysz <132913826+sfc-gh-dprzybysz@users.noreply.github.com> Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> Co-authored-by: Srikanth Reddy Kumbham <s.kumbham@snowflake.com> Co-authored-by: Agam Brahma <agam@sigmacomputing.com> Co-authored-by: Luke Paulsen <luke@sigmacomputing.com> Co-authored-by: Mihai Claudiu Toader <mihai@sigmacomputing.com> Co-authored-by: Yifeng-Sigma <yifeng@sigmacomputing.com> Co-authored-by: Jack Qian <jsjqian@users.noreply.github.com> Co-authored-by: Agam Brahma <agam.brahma+git@gmail.com> Co-authored-by: Greg Owen <GregOwen@users.noreply.github.com> Co-authored-by: Max Seiden <max@sigmacomputing.com> Co-authored-by: fengqingthu <qing.feng@sigmacomputing.com> Co-authored-by: Eric Bannatyne <aldld@users.noreply.github.com> Co-authored-by: Madison Chamberlain <46542378+madisonchamberlain@users.noreply.github.com> Co-authored-by: Eric Bannatyne <eric@sigmacomputing.com> Co-authored-by: mansap22 <110418152+mansap22@users.noreply.github.com> Co-authored-by: Ayman Elkfrawy <120422207+ayman-sigma@users.noreply.github.com> Co-authored-by: GregOwen <greg@sigmacomputing.com> Co-authored-by: sureshmula2 <126016714+sureshmula2@users.noreply.github.com> Co-authored-by: Ryan Kwong <ryankwong@ryank01-sigma.lan> Co-authored-by: Madison Chamberlain <madisonchamberlain@madisonc01-sigma.local>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Common code was extracted to separate function. In addition to that, an enum representation of transaction command was added to disallow any string being passed as a transaction command.
Checklist
make fmtto fix inconsistent formatsmake lintto get lint errors and fix all of them