* [sql lab] a better approach at limiting queries
Currently there are two mechanisms that we use to enforce the row
limiting constraints, depending on the database engine:
1. use dbapi's `cursor.fetchmany()`
2. wrap the SQL into a limiting subquery
Method 1 isn't great as it can result in the database server storing
larger than required result sets in memory expecting another fetch
command while we know we don't need that.
Method 2 has a positive side of working with all database engines,
whether they use LIMIT, ROWNUM, TOP or whatever else since sqlalchemy
does the work as specified for the dialect. On the downside though
the query optimizer might not be able to optimize this as much as an
approach that doesn't use a subquery.
Since most modern DBs use the LIMIT syntax, this adds a regex approach
to modify the query and force a LIMIT clause without using a subquery
for the database that support this syntax and uses method 2 for all
others.
* Fixing build
* Fix lint
* Added more tests
* Fix tests
* Add ISO duration to time grains
* Use ISO duration
* Remove debugging code
* Add module to yarn.lock
* Remove autolint
* Druid granularity as ISO
* Remove dangling comma
* [bugfix] convert metrics to numeric in dataframe
It appears sometimes the dbapi driver and pandas's read_sql fail at
returning the proper numeric types for metrics and they show up as
`object` in the dataframe. This results in "No numeric types to
aggregate" errors when trying to perform aggregations or pivoting in
pandas.
This PR looks for metrics in dataframes that are typed as "object"
and uses pandas' to_numeric to convert.
* Fix tests
* Remove all iteritems
* move access permissions methods to security manager
* consolidate all security methods into SupersetSecurityManager
* update security method calls
* update calls from tests
* move get_or_create_main_db to utils
* raise if supersetsecuritymanager is not extended
* rename sm to security_manager
* [sql lab] disable cross schema search
This is killing our metastore as people type it emits large
all-table-dump as they hit the keystroke. It never returns as it times
out and hammers the poor metastore.
Also some improvements around the disabling the table select on the left
panel and having the table name not be sticky.
* typo
* Expose hook to inject database connection logic on the fly
This environment configuration setting hook allows administrators to
alter the database connection parameters on the fly based on user
information. This can be use for a variety of purposes:
* rewire a subset of users to use different database user accounts
* pass user related information to the database for logging or QoS
purposes
* ...
* Fixes
* [Explore view] Use POST method for charting requests
* fix per code review comments
* more code review fixes
* code review fix: remove duplicated calls for getting values from request
* [Explore view] Use POST method for charting requests
* fix per code review comments
* more code review fixes
* code review fix: remove duplicated calls for getting values from request
Currently, even though `get_sqla_engine` calls get memoized, engines are
still short lived since they are attached to an models.Database ORM
object. All engines created through this method have the scope of a web
request.
Knowing that the SQLAlchemy objects are short lived means that
a related connection pool would also be short lived and mostly useless.
I think it's pretty rare that connections get reused within the context
of a view or Celery worker task.
We've noticed on Redshift that Superset was leaving many connections
opened (hundreds). This is probably due to a combination of the current
process not garbage collecting connections properly, and perhaps the
absence of connection timeout on the redshift side of things. This
could also be related to the fact that we experience web requests timeouts
(enforced by gunicorn) and that process-killing may not allow SQLAlchemy
to clean up connections as they occur (which this PR may not help
fixing...)
For all these reasons, it seems like the right thing to do to use
NullPool for external connection (but not for our connection to the metadata
db!).
Opening the PR for conversation. Putting this query into our staging
today to run some tests.
* Make Welcome page into a simple React app
This removes a dependency on datatables, we should be able to get rid
of it as we re-write the Table and PivotTable viz
* tests/lint
* Bump node version to latest
* Adding full Annotation Framework
* Viz types
* Re organizing native annotations
* liniting
* Bug fix
* Handle no data
* Cleanup
* Refactor slice form_data to data
fixes https://github.com/apache/incubator-superset/issues/3507
This prevents malformed JSON from getting saved in a slice's params. It
also prevents the issue described in #3507 from happening though as a
result malformed slices will render using default control values.
When the database impersonation flag is on, a query using a template
fails. It has to do with templating using a database connection without
a username being specified by the caller, along with the fact that the
work is taking place on a worker, outside a web request, where
referencing g.user raises this exception.
* Adding annotations to backend
* Auto fetching Annotations on the backend
* Closing the loop
* Adding missing files
* annotation layers UI
for https://github.com/apache/incubator-superset/issues/3502
* a few fixes per code review.
- add annotation input sanity check before add and before update.
- make SelectAsyncControl component statelesis, and generic
- add annotation description in d3 tool tip
- use less variable to replace hard-coded color
* Add "Impersonate user" setting to Datasource
* Add tests
* Use g.user.username for all the sync cases
* use uri.username instead of uri.user
* Small refactoring
