Commit Graph

114 Commits

Author SHA1 Message Date
Erik Ritter
fa5dab85c4 feat: more specific presto error messages (#11099) 2020-09-29 21:50:58 -07:00
Erik Ritter
0f7e90f1f7 feat: improve presto query perf (#11069) 2020-09-28 08:52:00 -07:00
Kasia Kucharczyk
af1e8e8839 fix: enable several pylint rules partially in db_engines_specs module (#11000)
* Enabled mssql

* Enabling pylint rules in `mysql.py` from `db_engine_specs`:
- added variable to shorten line and enable `line-too-long`
- `import-error` seems not prompting anymore
- specified exception instead broad-except

* Refactored too long line in `oracle.py` from `db_engine_specs`. Enabled too-long-line pylint rule.

* Changed fstring into string `pinot.py` to reenable `f-string-without-interpolation` rule.

* Enabled `unused-import` rule and changed line to adjust to correct number of characters in `db_engine_specs/postgres.py`.

* Enabled unused-import pylint check in `db_engine_specs/presto.py`

* Enabled unused-import pylint check in `db_engine_specs/snowflake.py`

* Enabled unused-import pylint check in `db_engine_specs/sqlite.py`
2020-09-23 10:47:56 -07:00
Kasia Kucharczyk
a6f258747b fix: enable pylint rules in db_engine_specs module (#10998)
* Athena.py: cretaed variable for too long line, removed disabled pylint rule.

* Enabled `line-too-long` in `drill.py`. Added variable for too long string.

* Reformatted hana.py:
- changed return statement into two lines to keep line lenght limit

* Enabling pylint rule in hive.py:
- `no-name-in-module` for pyhive in several places
- `line-too-long` and adding new lines
- `unused-import` is not invalid anymore

* Enabled `line-too-long` in `kylin.py`. Added variable for too long string.

* Enabled `unused_import` in `base.py` from db_engine_specs module.

* Enabled `unused_import` in `bigquery.py` from db_engine_specs module.

* Enabled `unused_import` in `druid.py` from db_engine_specs module.

* fix: athena datetime string converting
2020-09-22 13:44:45 -07:00
Beto Dealmeida
0554c8918e Enable subqueries in gsheetsdb (#10991) 2020-09-22 08:51:29 -07:00
Ville Brofeldt
bd140e018a fix(presto): default unknown types to string type (#10753)
* fix(presto): default unknown types to string type

* lint
2020-09-22 13:16:54 +03:00
Yongjie Zhao
38edb69d95 Feat: Adding table comment and columns comment for SQLLab (#10844)
* Adding table comment and columns comment for backend

* fix mypy

* Fix CI

* adding wider catch

* use logger

* fix lint
2020-09-14 21:11:48 -07:00
Ville Brofeldt
4392c4608f chore: bump pyarrow (#10733)
* chore: bump pandas an pyarrow

* remove unsupported kwargs

* fix types

* move pandas back to 1.0 branch due to possible MultiIndex regression
2020-08-31 19:55:53 +03:00
chuancy
234b6bbba9 fix(db-engine-spec): execute oracle DML statement bug in sqllab (#10706)
* fix execute oracle DML statement bug in sqllab

when i execute oracle sql statements like update in SQLLAB, get "oracle error: not a query" error. 

Refer https://www.python.org/dev/peps/pep-0249/, superset old version use
`cursor.description` ,because this attribute will be None for operations that do not return rows or if the cursor has not had an operation invoked via the .execute*() method yet.

* Apply suggestions from code review

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* Update oracle.py

* Update oracle.py

* Update oracle.py

* Apply suggestions from code review

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* Update oracle.py

* Update superset/db_engine_specs/oracle.py

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-08-31 08:03:07 +03:00
Ville Brofeldt
dd0bc472e3 refactor(database): use SupersetResultSet on SqlaTable.get_df() (#10707)
* refactor(database): use SupersetResultSet on SqlaTable.get_df()

* lint

* change cypress test
2020-08-28 21:12:03 +03:00
Bogdan
19a9bcc9c5 chore: ci Initial hive support (#10593)
* Initial hive support

* Clone hive setup

* Make hive tests work locally

* Debugging presto failure

* sleep in dataset test

* Address comments

* Address comments

* Pin ipython, exclude new pylint rules

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-08-27 09:49:18 -07:00
Ville Brofeldt
9461f9c1e0 fix(db_engine_specs): improve Presto column type matching (#10658)
* fix: improve Presto column type matching

* add optional callback to type map and add tests

* lint

* change private to public
2020-08-24 22:42:07 +03:00
Ville Brofeldt
0177c2f591 fix(db_engine_specs): mysql longtext type should not be numeric (#10661) 2020-08-24 21:24:43 +03:00
Ville Brofeldt
96b9ba3364 ci: bump pre-commit py36 to py37 (#10541)
* ci: bump pre-commit py36 to py37

* add 3rd party for py37

* lint

* yet more lint
2020-08-07 06:54:22 +03:00
Erik Ritter
2055ecc1ba feat: refactor error components and add database issue code (#10473)
* feat: refactor error components and add database issue code

* Apply suggestions from code review

Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com>

Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com>
2020-08-06 13:22:24 -07:00
John Bodley
cda764fd4b chore: Adding pip-compile-multi et al. (#10499)
* chore: Adding pip-compile-multi et al

* Specify requirements.txt path for fossa

* [ci] Fixing CI

Co-authored-by: John Bodley <john.bodley@airbnb.com>
Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
2020-08-04 19:31:52 -07:00
pphszx
473fe1003a fix: excel sheet upload is not working (#10450)
* remove conflicts with csv upload

* revert StringField

* change description

* remove redundant space

* apply string approach
2020-07-29 10:20:44 +03:00
Beto Dealmeida
4f678272d6 fix(presto): Handle ROW data stored as string (#10456)
* Handle ROW data stored as string

* Use destringify

* Fix mypy

* Fix mypy with cast

* Bypass pylint
2020-07-28 16:05:58 -07:00
Ville Brofeldt
ac85aebe4a chore: prefer allow/deny terminology (#10320)
* chore: prefer allow/deny terminology

* fix tests

* add PR reference
2020-07-15 19:22:46 +03:00
serenajiang
c67b1abfd8 fix(csv upload): hive params typo (#10273)
* fix(csv upload): fix :params typo

* fix tests
2020-07-09 11:25:35 -07:00
Nilanjan Sarkar
300b2bbf2d fix: Fixed bug for issue #9967 (#10269)
* Added formatted file setup.cfg

* lint

Co-authored-by: Nilanjan1.Sarkar <nilanjan1.sarkar@ril.com>
2020-07-09 16:15:07 +03:00
Bogdan
84f8a51458 feat: support nulls in the csv uploads (#10208)
* Support more table properties for the hive upload

Refactor

Add tests, and refactor them to be pytest friendly

Use lowercase table names

Ignore isort

* Use sql params

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-07-06 13:26:43 -07:00
Maxime Beauchemin
318e5347bc feat: Alerts! allowing users to set SQL-based email alerts with screenshots (#9944)
* feat: add support for alerts

* Add ModelViews

* First pass at adding scheduled Alert support

* Fix syntax errors

* Add log_retention and database to model

* Improving screenshots

* Still refactoring screenshots

* Pass down thumb_size properly

* Progress on screenshot endpoint

* Add alerts.last_eval_dttm and alert query logic

* Getting ready to split compute_screenshot and screenshot/{SHA}

* split the endpoints

* Improving alerting loop

* empty methods

* Add CLI command 'superset alert' that runs a sched loop

* Getting things to work

* email delivery

* A working email\!

* Add feature flag

* Add some description_columns to AlertModelView

* Little tweaks

* Use database.get_df, eval cells for truthiness

* Migrate thumbnail/screenshot functions to use new arg pattern

* Addressing PR feedback

* Update alerts DB migration down_revision

* Resolve _deliver_email arg conflict

* Make mypy happy

* Make isort happy

* Make pylint happy

Co-authored-by: Rob DiCiuccio <rob.diciuccio@gmail.com>
2020-07-06 13:12:27 -07:00
Hossein Torabi
fdd28c1a5e Upload excel (#9825) 2020-07-03 08:28:30 +03:00
Erik Ritter
b686004294 feat: add configuration for Presto cursor poll interval (#10191) 2020-06-30 08:56:22 -07:00
Hendrik Schultze
6a8f441d54 fix: make auth for hive optional (#10172)
* fix: make auth for hive optional

This will make allow the admin to setup a root connection which can be impersonated.

* style: black lint conform

* style: clean up unnecessary condition expansion
2020-06-29 07:40:24 +03:00
Ville Brofeldt
b205ce32b0 feat: Add new timegrains and convert_dttm to Druid engine spec (#10160)
* feat: Add new timegrains and convert_dttm to Druid engine spec

* Add TemporalType enum and fix test case

* Remove DATETIME for athena (original spec)
2020-06-25 12:18:37 +03:00
Will Barrett
8e23d4f369 chore: Upgrade pylint to 2.5.3 and fix most new rules (#10101)
* Bump pylint version to 2.5.3

* Add a global disable for the most common new pylint error

* Fix a bunch of files containing very few errors

* More pylint tweakage, low-hanging fruit

* More easy stuff...

* Fix more erroring files

* Fix the last couple of errors, clean pylint!

* Black

* Fix mypy issue in connectors/druid/models.py
2020-06-18 14:03:42 -07:00
Maxime Beauchemin
280ade826c fix: dashboard export raises schema issues (#10060)
* fix: dashboard export raises schema issues

* Fix isort
2020-06-15 21:43:05 -07:00
serenajiang
8744dadca8 [csv upload][hive] support other delimiters (#9971)
Co-authored-by: serena-jiang <serena.jiang@airbnb.com>
2020-06-10 17:08:02 -07:00
Erik Ritter
e17da58a39 feat: add replace option to hive csv upload (#9764) 2020-06-10 08:57:33 -07:00
John Bodley
91517a56a3 style(mypy): Spit-and-polish pass (#10001)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-06-07 08:53:46 -07:00
Xiang Fu
1d9dbcd9dc fix: pinot select query logic (#9954)
* Update how pinot handles selection logic

* Change DATETIMECONVERT argument to use single quote for literals
2020-06-02 23:18:35 -07:00
John Bodley
7f6dbf838e style: enforcing mypy typing for connectors (#9824)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-05-25 12:32:49 -07:00
Maxime Beauchemin
ee991963c7 fix: don't create examples db on 'superset init' (#9881)
* chore: don't create examples db on 'superset init'

* fix isort
2020-05-21 21:34:53 -07:00
Bogdan
3e8e441bfc feat: implement csv upload configuration func for the schema enforcement (#9734)
* Implement csv upload func for schema enforcement

Implement function controlled csv upload schema

Refactor + fix tests

Fixing hive as well

* Add explore_db to the extras

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-05-21 13:49:53 -07:00
Daniel Vaz Gaspar
3cc540019f fix(mssql): reverts #9644 and displays a better error msg (#9752) 2020-05-14 17:00:02 +01:00
Erik Ritter
83ec7365a7 feat: convert backend chart errors to the new error type (#9753) 2020-05-12 18:15:53 -07:00
serenajiang
5d167afb94 [hive][csv upload] make INTs BIGINTs (#9716) 2020-05-03 09:28:31 -07:00
Daniel Vaz Gaspar
2b59075d57 tests(engine_specs): full bigquery engine coverage (#9702) 2020-04-30 18:31:15 +01:00
Daniel Vaz Gaspar
527bee5051 tests(engine_specs): full postgres engine coverage (#9682) 2020-04-29 18:20:25 +01:00
Daniel Vaz Gaspar
516bdf6db1 fix(mssql): apply limit and set alias for functions (#9644) 2020-04-27 09:23:08 +01:00
Ville Brofeldt
980dd2fd41 pylint: accept specific 2 character names by default (#9460)
* lint: accept 2 letter names by default

* Address review comments

* Remove e and d from good-names
2020-04-08 20:32:26 +03:00
Ville Brofeldt
499f9c8fca feat: add SSL certificate validation for Druid (#9396)
* feat: add SSL certificate feature

* Address comments

* don't mutate extras

* Address comments and add polish

* Add further polish
2020-03-27 19:07:07 +02:00
Ville Brofeldt
982c234a50 fix: remove character set and collate column info by default (#9316)
* fix: remove character set and collate column info by default

* lint

* remove collation and charset info before compile
2020-03-17 21:34:39 +02:00
Ville Brofeldt
3682702e91 fix: add connection testing params for snowflake (#9272)
* fix: add connection testingt params for snowflake

* Linting
2020-03-11 06:51:57 +02:00
Ville Brofeldt
ef2ebbd570 Add option to specify type specific date truncation functions (#9238) 2020-03-05 07:25:50 +02:00
Ville Brofeldt
969bc87431 fix: Oracle fetch_query and datetime conversion (#9240) 2020-03-04 23:43:28 +02:00
Ville Brofeldt
7a91498cf1 fix: share column type matching between model and result set (#9161)
* Share column type matching between model and result set

* Address comments
2020-03-04 16:33:22 +02:00
Bogdan
4e1fa95035 Make schema name for the CTA queries and limit configurable (#8867)
* Make schema name configurable

Fixing unit tests

Fix table quoting

Mypy

Split tests out for sqlite

Grant more permissions for mysql user

Postgres doesn't support if not exists

More logging

Commit for table creation

Priviliges for postgres

Update tests

Resolve comments

Lint

No limits for the CTA queries if configures

* CTA -> CTAS and dict -> {}

* Move database creation to the .travis file

* Black

* Move tweaks to travis db setup

* Remove left over version

* Address comments

* Quote table names in the CTAS queries

* Pass tmp_schema_name for the query execution

* Rebase alembic migration

* Switch to python3 mypy

* SQLLAB_CTA_SCHEMA_NAME_FUNC -> SQLLAB_CTAS_SCHEMA_NAME_FUNC

* Black
2020-03-03 19:52:20 +02:00