Commit Graph

183 Commits

Author SHA1 Message Date
Daniel Vaz Gaspar
1e3aaab590 fix(reports): validator_config, report state machine, working_timeout (#11890)
* fix(reports): expect more exceptions and fix validator config

* use a state pattern on command reports

* use a state pattern on command reports continue

* fix multiple heads

* fix unittests

* add more tests

* fix api tests after enum rename

* fix alembic multiple heads

* fix tests

* fix fixture cleanup

* fix mysql tests

* fix initial and not found state

* fix schema, and private public methods, addressing comments

* add new col to the API
2020-12-09 18:19:07 +00:00
Erik Ritter
77d362d306 fix: Remove expensive logs table migration (#11920) 2020-12-04 07:59:28 -08:00
Daniel Vaz Gaspar
71c567bf3a feat(css templates): security perm simplification (#11856)
* feat: security converge css templates

* fix security tests

* fix JS test

* add migration

* black and fix migration
2020-12-03 08:45:32 +00:00
Daniel Vaz Gaspar
ac9761c730 fix: dashboard iframe to markdown db migration (#11845)
* fix: dashboard iframe to markdown db migration

* Update superset/migrations/versions/978245563a02_migrate_iframe_to_dash_markdown.py

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-12-01 20:50:35 +00:00
Daniel Vaz Gaspar
b1a963b93f fix: security converge downgrade procedure (#11852) 2020-11-30 17:50:12 +00:00
Daniel Vaz Gaspar
9dd33d5566 feat(saved queries): security perm simplification (#11764)
* feat(saved queries): security perm simplification

* migration script and frontend

* add downgrade procedure

* downgrade procedure, inferred from the upgrade data

* fix JS test

* improve code

* add tests for role migration

* more tests and a short description for each one

* Improve readability

* simplify dataclass creation

* fix dataclass
2020-11-30 13:07:39 +00:00
Maxime Beauchemin
0504cf1a00 chore: improve analytics (#11714)
* chore: improve analytics

* lint

* log more events, add note in UPDATING.md

* handling base class

* more events\!

* get ref through

* right before @expose

* fix context

* touchups
2020-11-25 08:45:02 -08:00
Daniel Vaz Gaspar
bd79bd2a54 feat: new report schedule models (#11550)
* feat: new report schedule models

* lint and unique constraint

* support sqlite

* fix sqlite

* add audit mixin and minor fixes

* fix FK's

* address comments

* lint
2020-11-06 09:33:32 +00:00
Beto Dealmeida
98daf0a435 feat: add UUID column to saved_query for export/import (#11397)
* Add UUID to saved_query

* Reuse function from previous migration

* Point to new head
2020-10-27 08:49:09 -07:00
Daniel Vaz Gaspar
144b279aa2 feat: saved queries with execution info (#11391)
* feat: add rows and last_run info to saved queries

* feat: add rows to saved query

* refactor and tests

* lint

* fix tests
2020-10-26 11:20:07 -07:00
Jesse Yang
d7eb1d476f perf: speed up uuid column generation (#11209) 2020-10-13 13:29:49 -07:00
Bogdan
cb3f649a7f chore: simplify alerting data model to leverage a single class (#11179)
* Collapse alerting models into a single one

Fixing upgrade migration & tests

* Address feedback

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-10-09 19:20:12 -07:00
Beto Dealmeida
9785667a0d feat: add UUID column to ImportMixin (#11098)
* Add UUID column to ImportMixin

* Fix default value

* Fix lint

* Fix order of downgrade

* Add logging when downgrade fails

* Migrate position_json to contain UUIDs, and add schedule tables

* Save UUID when adding charts to dashboard

* Fix heads

* Rename migration file

* Fix dashboard serialization

* Fix migration script with Postgres

* Fix unique contraint name

* Handle UUID when exporting dashboard

* Fix Dataset PUT

* Add UUID JSON serialization

* Fix tests

* Simplify logic

* Try binary=True
2020-10-07 09:00:55 -07:00
John Bodley
e90c63fc7b fix: Alembic migration 18532d70ab98 (#11136) 2020-10-02 09:12:03 +01:00
Daniel Vaz Gaspar
50d80405a9 fix: alembic migration error msg trying to delete constraint on tables (#11115)
* fix: alembic migration fails by deleting non existent constraint on tables

* Revert "fix: alembic migration fails by deleting non existent constraint on tables"

This reverts commit 3a359b0577.

* mantain migration but just for MySQL and add downgrade procedure

* skip the downgrade
2020-10-01 12:35:12 +01:00
Bogdan
94d4d55908 Fix alembic migration (#11110)
Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-09-29 16:35:00 -07:00
Bogdan
03eebd3019 fix: fix table existence validation function (#11066)
* Fix table existance validation function

* Drop left over table name index in mysql db

* Do not modify model

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-09-29 15:01:01 -07:00
Daniel Vaz Gaspar
5d08a426d3 fix(api): unable to delete virtual dataset, wrong permission name (#11019)
* fix(api): unable to delete virtual dataset because of wrong permission name

* Still delete the dataset even when no permission was found

* migration script to fix possible existing faulty permissions on the db

* black

* fix db migration and one more test

* add more comments to the migration script

* freeze a partial schema of the model on the migration step

* fix mig script

* Update superset/migrations/versions/3fbbc6e8d654_fix_data_access_permissions_for_virtual_.py

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-09-29 12:33:07 +01:00
Ville Brofeldt
448a41a4e7 feat(row-level-security): add base filter type and filter grouping (#10946)
* feat(row-level-security): add filter type and group key

* simplify tests and add custom list widget

* address comments

* use enum value to ensure case sensitive value is used
2020-09-22 08:54:10 +03:00
Jason Davis
c1f2214d7c merge db migrations (#10758)
Co-authored-by: Jason Davis <@dropbox.com>
2020-09-01 17:59:02 -07:00
Jason Davis
b59f6b1833 feat: refractored SQL-based alerting framework (#10605)
* added new tables for alerting refractor

* reformatted inheritance structure

* added workflow for updated framework

* added suggested changes

* cleaned up changes

* added obervations to alert table to enable view

* added comments

* added requested changes

* fix tests

* added styling changes

* mypy

* added requested changes

* updated operator logic

* requested changes, 1 validator, styling changes

* refactored tests

* fix test alert workflow

* fixed create_alert in test

Co-authored-by: Jason Davis <@dropbox.com>
2020-09-01 13:36:02 -07:00
Bogdan
4572ebb600 chore: log cache keys to the logs (#10678)
* Log cache keys to the logs

* Add tests

* Use separate table for the cache keys

* Add migration for the cache lookup table

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-09-01 09:41:25 -07:00
Grace Guo
ca9ca99510 refactor: [migration] convert iframe chart into dashboard markdown component (#10590)
* refactor: [migration] convert iframe chart into dashboard markdown component

* remove 3 viz_types

* fix comments
2020-08-16 14:43:30 -07:00
Erik Ritter
bd88e12b8b feat: add extra column to tables and sql_metrics (#10592) 2020-08-12 15:27:29 -07:00
Jason Davis
c0bb86d0d6 feat: slack integration for SQL-based alerts (#10566)
* add slack functionality

* deleted unused variable

* updated test

* black

* fix rebase

* added nits

* added slack no screenshot integration

* isort

* added namedtuple for screenshot

* added test

* fix precommit

Co-authored-by: Jason Davis <@dropbox.com>
2020-08-11 11:15:31 -07:00
John Bodley
cda764fd4b chore: Adding pip-compile-multi et al. (#10499)
* chore: Adding pip-compile-multi et al

* Specify requirements.txt path for fossa

* [ci] Fixing CI

Co-authored-by: John Bodley <john.bodley@airbnb.com>
Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
2020-08-04 19:31:52 -07:00
John Bodley
821916a681 chore: pre-commit run --all-files (#10500)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-08-02 14:32:17 -07:00
Maxime Beauchemin
318e5347bc feat: Alerts! allowing users to set SQL-based email alerts with screenshots (#9944)
* feat: add support for alerts

* Add ModelViews

* First pass at adding scheduled Alert support

* Fix syntax errors

* Add log_retention and database to model

* Improving screenshots

* Still refactoring screenshots

* Pass down thumb_size properly

* Progress on screenshot endpoint

* Add alerts.last_eval_dttm and alert query logic

* Getting ready to split compute_screenshot and screenshot/{SHA}

* split the endpoints

* Improving alerting loop

* empty methods

* Add CLI command 'superset alert' that runs a sched loop

* Getting things to work

* email delivery

* A working email\!

* Add feature flag

* Add some description_columns to AlertModelView

* Little tweaks

* Use database.get_df, eval cells for truthiness

* Migrate thumbnail/screenshot functions to use new arg pattern

* Addressing PR feedback

* Update alerts DB migration down_revision

* Resolve _deliver_email arg conflict

* Make mypy happy

* Make isort happy

* Make pylint happy

Co-authored-by: Rob DiCiuccio <rob.diciuccio@gmail.com>
2020-07-06 13:12:27 -07:00
Will Barrett
df71fac1e0 refactor: Fix lint on superset/utils/core.py (#10120)
* Fix lint on superset/utils/core.py

* black

* mypy

* Fix some missing renames
2020-06-26 08:49:12 -07:00
Bogdan
42a56e29a1 Fix alembic migrations order (#10156)
Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-06-24 11:41:47 -07:00
John Bodley
37777f382f fix(security): dbs/clusters perm (#10130)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-06-24 10:06:25 -07:00
Bogdan
3db76c6fdc Implement create view as functionality (#9794)
Implement create view as button in sqllab

Make CVAS configurable

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-06-24 09:50:41 -07:00
Ville Brofeldt
d615949820 chore: Bump sqlalchemy-utils and cachelib and refresh requirements.txt (#10117)
* chore: Bump sqlalchemy-utils and cachelib and refresh requirements.txt

* undeprecate EncryptedType

* black

* reference correct class

Co-authored-by: Maxime Beauchemin <maximebeauchemin@gmail.com>
2020-06-22 16:01:46 +03:00
Aliaksei Kushniarevich
550e78ff7c feat: Add multiple table filters for Row Level Security (#9751)
* Add multiple table filters for Row Level Security

* Set ENABLE_ROW_LEVEL_SECURITY back to False (default)

* Merge DB migrations

* Drop table_id column and foreign key on PostgreSQL, MySQL, SQLite

* Support db records migration also

* Support downgrading from the new-fashioned formatted records

* Straighten up migrations

* Update migration's down_revision to comply master branch
2020-06-22 12:51:08 +03:00
Bogdan
29e9f2c70b feat: superset report slack integration (#9810)
* First draft for the slack integration

Fix slack

another typo

another typo

Fix slack

Add channels to the form

another typo

Another set of changes

Make code more transparent

Fix tests

Add logging

logging

use logger

import logging

import logging

import logging

add assert

more logging

Fix channels

Fix channels

* Address comments

* Move slack into a separate module

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-06-17 11:01:25 -07:00
Erik Ritter
e95af7f783 fix: migrate all slices off the old time grain format (#9684) 2020-04-30 08:38:13 -07:00
Lily Kuang
b27a81e0da [logging] deprecation notices for SQLite (#9662)
* [logging] deprecation notices for SQLite

* [logging] use warning for SQLite notice

* [logging] update grammar
2020-04-28 14:28:09 -07:00
John Bodley
636ee5d19a [mypy] Enforcing typing for superset.migrations (#9583)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-18 23:24:32 -07:00
Ville Brofeldt
980dd2fd41 pylint: accept specific 2 character names by default (#9460)
* lint: accept 2 letter names by default

* Address review comments

* Remove e and d from good-names
2020-04-08 20:32:26 +03:00
John Bodley
e63cb52630 [fix] Fixing cache key inconsistencies (#9382)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-03-28 12:33:16 -07:00
Ville Brofeldt
499f9c8fca feat: add SSL certificate validation for Druid (#9396)
* feat: add SSL certificate feature

* Address comments

* don't mutate extras

* Address comments and add polish

* Add further polish
2020-03-27 19:07:07 +02:00
Bogdan
4e1fa95035 Make schema name for the CTA queries and limit configurable (#8867)
* Make schema name configurable

Fixing unit tests

Fix table quoting

Mypy

Split tests out for sqlite

Grant more permissions for mysql user

Postgres doesn't support if not exists

More logging

Commit for table creation

Priviliges for postgres

Update tests

Resolve comments

Lint

No limits for the CTA queries if configures

* CTA -> CTAS and dict -> {}

* Move database creation to the .travis file

* Black

* Move tweaks to travis db setup

* Remove left over version

* Address comments

* Quote table names in the CTAS queries

* Pass tmp_schema_name for the query execution

* Rebase alembic migration

* Switch to python3 mypy

* SQLLAB_CTA_SCHEMA_NAME_FUNC -> SQLLAB_CTAS_SCHEMA_NAME_FUNC

* Black
2020-03-03 19:52:20 +02:00
altef
dee16de03e [SIP-29] Add support for row-level security (#8699)
* Support and apply filters.

* Added the UI for row level security, and moved it all under SQLA in order to access the Table model more easily.

* Added a row level security filter documentation entry.

* Accidentally added two new lines to this file.

* Blacked and iSorted, hopefully.  Also, sometimes g.user may not be set.

* Another isort, and handling g not having a user attribute another way.

* Let's try this again #CI tests.

* Adjusted import order for isort; I was sure I'd already done this..

* Row level filters should be wrapped in parentheses in case one contains an OR.

* Oops, did not think that would change Black's formatting.

* Changes as per @mistercrunch.

* RLS filters are now many-to-many with Roles.

* Updated documentation to reflect RLS filters supporting multiple rows.

* Let's see what happens when I set it to the previous revision ID

* Updated from upstream.

* There was a pylint error.

* Added RLS ids to the cache keys; modified documentation; added template processing to RLS filters.

* A new migration was merged in.

* Removed RLS cache key from query_object.

* RLS added to the cache_key from query_context.

* Changes as per @etr2460.

* Updating entry for RLS pull request.

* Another migration to skip.

* Changes as per @serenajiang.

* Blacked.

* Blacked and added some attributes to check for.

* Changed to a manual query as per @mistercrunch.

* Blacked.

* Another migration in the meantime.

* Black wanted some whitespace changes.

* AttributeError: 'AnonymousUserMixin' object has no attribute 'id'.

* Oops, did hasattr backwards.

* Changes as per @mistercrunch.

* Doesn't look like text us required here anymore.

* Changes as per @dpgaspar

* Two RLS tests.

* Row level security is now disabled by default via the feature flag ENABLE_ROW_LEVEL_SECURITY.

* New head to revise.

* Changed the comment.
2020-02-22 11:21:31 +02:00
Grace Guo
f4ad15e644 [migration] metadata for dashboard filters (#9109)
* [migration] metadata for dashboard filters

* fix comments

* fix extra code review comments
2020-02-14 13:26:04 -08:00
serenajiang
1f6f4ed879 [db migration] change datasources-clusters foreign key to cluster_id (#8576)
* [db migration] change datasources foreign key to cluster_id

* address pr comments

* address pr comment, fix ci
2020-01-13 11:02:36 -08:00
Ville Brofeldt
5b690f9411 chore: refactor, add typing and fix uncovered errors (#8900)
* Add type annotations and fix inconsistencies

* Address review comments

* Remove incorrect typing of jsonable obj
2019-12-31 09:26:23 +02:00
Will Barrett
ed54f6ec8d Require the SQLAlchemy URI on the database model (#8720)
* Require the SQLAlchemy URI when creating a database

* Add migration to make dbs.sqlalchemy_uri not-nullable

* Fixes for black, isort, tests

* Alter migration to use current revision from master as downgrade target

* Update tests to support new db constraint

* black
2019-12-11 14:05:23 -08:00
Will Barrett
68aca88c54 Re-enable pylint on some configuration files (#8767)
* re-enable pylint for superset/config.py

* re-enable pylint on superset/migrations/env.py

* re-enable pylint for superset/legacy.py

* re-enable pylint for superset/forms.py

* re-enable pylint for superset/stats_logger.py

* Tweaks to make mypy and pylint happy

* black
2019-12-06 14:30:11 -08:00
Beto Dealmeida
5b934bb377 Change column to longtext (#8739)
* Change column to longtext

* Manual merge heads
2019-12-04 11:24:23 -08:00
Bogdan
003e98c231 Support schema permissions (#8219)
* Build support for schema access in Superset and SQLab

* Lint
* Test for old and new flask
* Black formatting
* Cleanup modified files
* Support schema permissions
* Add migration
* Make it work for slices as well
* TODO and remove unused field
* Apply schema access check to dashboards as well

Cleaner schema permissions check

Fetch schema permissions in 1 query

Bugfix

Better datasource check

Use security manager for permissions checks and remove dead code

Properly handle anon user

Add database permissions on the database updates

Fix schema filtering

Lint / flake changes

Increase cypress request timeout

Run black

Resolve multiple alembic heads

* Use set comprehensions

* Fixes for the pylint
2019-12-03 14:57:24 -08:00