update changelog

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
(cherry picked from commit 42980a69a7)

.github/workflows/superset-python-unittest.yml

@@ -50,6 +50,8 @@ jobs:
             mkdir ${{ github.workspace }}/.temp
       - name: Python unit tests
         if: steps.check.outcome == 'failure'
+        env:
+          SUPERSET_TESTENV: true
         run: |
           pytest --durations-min=0.5 --cov-report= --cov=superset ./tests/common ./tests/unit_tests --cache-clear
       - name: Upload code coverage

Dockerfile

@@ -114,7 +114,14 @@ COPY ./requirements/*.txt ./docker/requirements-*.txt/ /app/requirements/
 USER root
 
 RUN apt-get update -y \
-    && apt-get install -y --no-install-recommends libnss3 libdbus-glib-1-2 libgtk-3-0 libx11-xcb1 wget
+    && apt-get install -y --no-install-recommends \
+          libnss3 \
+          libdbus-glib-1-2 \
+          libgtk-3-0 \
+          libx11-xcb1 \
+          libasound2 \
+          libxtst6 \
+          wget
 
 # Install GeckoDriver WebDriver
 RUN wget https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VERSION}/geckodriver-${GECKODRIVER_VERSION}-linux64.tar.gz -O /tmp/geckodriver.tar.gz && \

RELEASING/Dockerfile.from_local_tarball

@@ -61,6 +61,7 @@ RUN pip install --upgrade setuptools pip \
 RUN flask fab babel-compile --target superset/translations
 
 ENV PATH=/home/superset/superset/bin:$PATH \
-    PYTHONPATH=/home/superset/superset/:$PYTHONPATH
+    PYTHONPATH=/home/superset/superset/:$PYTHONPATH \
+    SUPERSET_TESTENV=true
 COPY from_tarball_entrypoint.sh /entrypoint.sh
 ENTRYPOINT ["/entrypoint.sh"]

RELEASING/from_tarball_entrypoint.sh

@@ -19,6 +19,10 @@ set -ex
 
 echo "[WARNING] this entrypoint creates an admin/admin user"
 echo "[WARNING] it should only be used for lightweight testing/validation"
+if [ "$SUPERSET_TESTENV" = "true" ]
+then
+  echo "SUPERSET IS RUNNING IN TEST MODE"
+fi
 
 # Create an admin user (you will be prompted to set username, first and last name before setting a password)
 superset fab create-admin \

RESOURCES/FEATURE_FLAGS.md

@@ -61,6 +61,7 @@ These features are **finished** but currently being tested. They are usable, but
 - GENERIC_CHART_AXES
 - GLOBAL_ASYNC_QUERIES [(docs)](https://github.com/apache/superset/blob/master/CONTRIBUTING.md#async-chart-queries)
 - RLS_IN_SQLLAB
+- SSH_TUNNELING [(docs)](https://superset.apache.org/docs/installation/setup-ssh-tunneling)
 - USE_ANALAGOUS_COLORS
 - UX_BETA
 - VERSIONED_EXPORT

UPDATING.md

@@ -22,8 +22,20 @@ under the License.
 This file documents any backwards-incompatible changes in Superset and
 assists people when migrating to a new version.
 
-## Next
+## 2.1.0
 
+- [22809](https://github.com/apache/superset/pull/22809): Migrated endpoint `/superset/sql_json` and `/superset/results/` to `/api/v1/sqllab/execute/` and `/api/v1/sqllab/results/` respectively. Corresponding permissions are `can sql_json on Superset` to `can execute on SQLLab`, `can results on Superset` to `can results on SQLLab`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22931](https://github.com/apache/superset/pull/22931): Migrated endpoint `/superset/get_or_create_table/` to `/api/v1/dataset/get_or_create/`. Corresponding permissions are `can get or create table on Superset` to `can get or create dataset on Dataset`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22882](https://github.com/apache/superset/pull/22882): Migrated endpoint `/superset/filter/<datasource_type>/<int:datasource_id>/<column>/` to `/api/v1/datasource/<datasource_type>/<datasource_id>/column/<column_name>/values/`. Corresponding permissions are `can filter on Superset` to `can get column values on Datasource`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22789](https://github.com/apache/superset/pull/22789): Migrated endpoint `/superset/recent_activity/<user_id>/` to `/api/v1/log/recent_activity/<user_id>/`. Corresponding permissions are `can recent activity on Superset` to `can recent activity on Log`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22913](https://github.com/apache/superset/pull/22913): Migrated endpoint `/superset/csv` to `/api/v1/sqllab/export/`. Corresponding permissions are `can csv on Superset` to `can export csv on SQLLab`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22496](https://github.com/apache/superset/pull/22496): Migrated endpoint `/superset/slice_json/<int:layer_id>` to `/api/v1/chart/<int:id>/data/`. Corresponding permissions are `can slice json on Superset` to `can read on Chart`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22496](https://github.com/apache/superset/pull/22496): Migrated endpoint `/superset/annotation_json/<int:layer_id>` to `/api/v1/chart/<int:id>/data/`. Corresponding permissions are `can annotation json on Superset` to `can read on Chart`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22624](https://github.com/apache/superset/pull/22624): Migrated endpoint `/superset/stop_query/` to `/api/v1/query/stop`. Corresponding permissions are `can stop query on Superset` to `can read on Query`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22579](https://github.com/apache/superset/pull/22579): Migrated endpoint `/superset/search_queries/` to `/api/v1/query/`. Corresponding permissions are `can search queries on Superset` to `can read on Query`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22501](https://github.com/apache/superset/pull/22501): Migrated endpoint `/superset/tables/<int:db_id>/<schema>/` to `/api/v1/database/<int:id>/tables/`. Corresponding permissions are `can tables on Superset` to `can read on Database`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22611](https://github.com/apache/superset/pull/22611): Migrated endpoint `/superset/queries/` to `api/v1/query/updated_since`. Corresponding permissions are `can queries on Superset` to `can read on Query`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [23186](https://github.com/apache/superset/pull/23186): Superset will refuse to start if a default `SECRET_KEY` is detected on a non Flask debug setting.
 - [22022](https://github.com/apache/superset/pull/22022): HTTP API endpoints `/superset/approve` and `/superset/request_access` have been deprecated and their HTTP methods were changed from GET to POST
 - [20606](https://github.com/apache/superset/pull/20606): When user clicks on chart title or "Edit chart" button in Dashboard page, Explore opens in the same tab. Clicking while holding cmd/ctrl opens Explore in a new tab. To bring back the old behaviour (always opening Explore in a new tab), flip feature flag `DASHBOARD_EDIT_CHART_IN_NEW_TAB` to `True`.
 - [20799](https://github.com/apache/superset/pull/20799): Presto and Trino engine will now display tracking URL for running queries in SQL Lab. If for some reason you don't want to show the tracking URL (for example, when your data warehouse hasn't enabled access for to Presto or Trino UI), update `TRACKING_URL_TRANSFORMER` in `config.py` to return `None`.
@@ -31,19 +43,20 @@ assists people when migrating to a new version.
 - [21163](https://github.com/apache/superset/pull/21163): The time grain will be decoupled from the time filter column and the time grain control will move below the X-Axis control when `GENERIC_CHART_AXES` feature flags set to `True`. The time grain will be applied on the time column in the column-like controls(x axis, dimensions) instead of the time column in the time section.
 - [21284](https://github.com/apache/superset/pull/21284): The non-functional `MAX_TABLE_NAMES` config key has been removed.
 - [21794](https://github.com/apache/superset/pull/21794): Deprecates the undocumented `PRESTO_SPLIT_VIEWS_FROM_TABLES` feature flag. Now for Presto, like other engines, only physical tables are treated as tables.
-
-### Breaking Changes
-
 - [22798](https://github.com/apache/superset/pull/22798): To make the welcome page more relevant in production environments, the last tab on the welcome page has been changed from to feature all charts/dashboards the user has access to (previously only examples were shown). To keep current behavior unchanged, add the following to your `superset_config.py`: `WELCOME_PAGE_LAST_TAB = "examples"`
 - [22328](https://github.com/apache/superset/pull/22328): For deployments that have enabled the "THUMBNAILS" feature flag, the function that calculates dashboard digests has been updated to consider additional properties to more accurately identify changes in the dashboard metadata. This change will invalidate all currently cached dashboard thumbnails.
 - [21765](https://github.com/apache/superset/pull/21765): For deployments that have enabled the "ALERT_REPORTS" feature flag, Gamma users will no longer have read and write access to Alerts & Reports by default. To give Gamma users the ability to schedule reports from the Dashboard and Explore view like before, create an additional role with "can read on ReportSchedule" and "can write on ReportSchedule" permissions. To further give Gamma users access to the "Alerts & Reports" menu and CRUD view, add "menu access on Manage" and "menu access on Alerts & Report" permissions to the role.
 
+### Breaking Changes
+
 ### Potential Downtime
 
 - [21284](https://github.com/apache/superset/pull/21284): A change which drops the unused `dbs.allow_multi_schema_metadata_fetch` column via a (potentially locking) DDL operation.
 
 ### Other
 
+- [23118](https://github.com/apache/superset/pull/23118): Previously the "database access on <database>" permission granted access to all datasets on the underlying database, but they didn't show up on the list views. Now all dashboards, charts and datasets that are accessible via this permission will also show up on their respective list views.
+
 ## 2.0.1
 
 - [21895](https://github.com/apache/superset/pull/21895): Markdown components had their security increased by adhering to the same sanitization process enforced by Github. This means that some HTML elements found in markdowns are not allowed anymore due to the security risks they impose. If you're deploying Superset in a trusted environment and wish to use some of the blocked elements, then you can use the HTML_SANITIZATION_SCHEMA_EXTENSIONS configuration to extend the default sanitization schema. There's also the option to disable HTML sanitization using the HTML_SANITIZATION configuration but we do not recommend this approach because of the security risks. Given the provided configurations, we don't view the improved sanitization as a breaking change but as a security patch.

docker/.env-non-dev

@@ -42,6 +42,7 @@ REDIS_PORT=6379
 FLASK_ENV=production
 SUPERSET_ENV=production
 SUPERSET_LOAD_EXAMPLES=yes
+SUPERSET_SECRET_KEY=TEST_NON_DEV_SECRET
 CYPRESS_CONFIG=false
 SUPERSET_PORT=8088
 MAPBOX_API_KEY=''

docs/docs/installation/configuring-superset.mdx

@@ -23,8 +23,8 @@ SUPERSET_WEBSERVER_PORT = 8088
 # Your App secret key will be used for securely signing the session cookie
 # and encrypting sensitive information on the database
 # Make sure you are changing this key for your deployment with a strong key.
-# You can generate a strong key using `openssl rand -base64 42`
-
+# You can generate a strong key using `openssl rand -base64 42`.
+# Alternatively you can set it with `SUPERSET_SECRET_KEY` environment variable.
 SECRET_KEY = 'YOUR_OWN_RANDOM_GENERATED_SECRET_KEY'
 
 # The SQLAlchemy connection string to your database backend

docs/docs/installation/installing-superset-from-scratch.mdx

@@ -138,6 +138,12 @@ superset load_examples
 # Create default roles and permissions
 superset init
 
+# Build javascript assets
+cd superset-frontend
+npm ci
+npm run build
+cd ..
+
 # To start a development web server on port 8088, use -p to bind to another port
 superset run -p 8088 --with-threads --reload --debugger
 ```

docs/docs/installation/setup-ssh-tunneling.mdx

@@ -0,0 +1,21 @@
+---
+title: Setup SSH Tunneling
+hide_title: true
+sidebar_position: 13
+version: 1
+---
+
+## SSH Tunneling
+
+1. Turn on feature flag
+    - Change [`SSH_TUNNELING`](https://github.com/apache/superset/blob/eb8386e3f0647df6d1bbde8b42073850796cc16f/superset/config.py#L489) to `True`
+    - If you want to add more security when establishing the tunnel we allow users to overwrite the `SSHTunnelManager` class (here)[https://github.com/apache/superset/blob/eb8386e3f0647df6d1bbde8b42073850796cc16f/superset/config.py#L507]
+    - You can also set the [`SSH_TUNNEL_LOCAL_BIND_ADDRESS`](https://github.com/apache/superset/blob/eb8386e3f0647df6d1bbde8b42073850796cc16f/superset/config.py#L508) this the host address where the tunnel will be accessible on your VPC
+
+2. Create database w/ ssh tunnel enabled
+    - With the feature flag enabled you should now see ssh tunnel toggle.
+    - Click the toggle to enables ssh tunneling and add your credentials accordingly.
+        - Superset allows for 2 different type authenticaion (Basic + Private Key). These credentials should come from your service provider.
+
+3. Verify data is flowing
+    - Once SSH tunneling has been enabled, go to SQL Lab and write a query to verify data is properly flowing.

requirements/base.txt

@@ -82,7 +82,7 @@ flask==2.1.3
     #   flask-migrate
     #   flask-sqlalchemy
     #   flask-wtf
-flask-appbuilder==4.2.0
+flask-appbuilder==4.3.0
     # via apache-superset
 flask-babel==1.0.0
     # via flask-appbuilder

setup.py

@@ -83,7 +83,7 @@ setup(
         "cryptography>=39.0.0,<40",
         "deprecation>=2.1.0, <2.2.0",
         "flask>=2.1.3, <2.2",
-        "flask-appbuilder>=4.2.0, <5.0.0",
+        "flask-appbuilder>=4.3.0, <5.0.0",
         "flask-caching>=1.10.1, <1.11",
         "flask-compress>=1.13, <2.0",
         "flask-talisman>=1.0.0, <2.0",

superset-frontend/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "superset",
-  "version": "0.0.0-dev",
+  "version": "2.1.0",
   "lockfileVersion": 2,
   "requires": true,
   "packages": {
     "": {
       "name": "superset",
-      "version": "0.0.0-dev",
+      "version": "2.1.0",
       "license": "Apache-2.0",
       "workspaces": [
         "packages/*",

superset-frontend/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superset",
-  "version": "0.0.0-dev",
+  "version": "2.1.0",
   "description": "Superset is a data exploration platform designed to be visual, intuitive, and interactive.",
   "keywords": [
     "big",

superset-frontend/src/dashboard/components/DashboardBuilder/DashboardBuilder.tsx

@@ -442,8 +442,6 @@ const DashboardBuilder: FC<DashboardBuilderProps> = () => {
   const dashboardIsSaving = useSelector<RootState, boolean>(
     ({ dashboardState }) => dashboardState.dashboardIsSaving,
   );
-  const nativeFilters = useSelector((state: RootState) => state.nativeFilters);
-  const focusedFilterId = nativeFilters?.focusedFilterId;
   const fullSizeChartId = useSelector<RootState, number | null>(
     state => state.dashboardState.fullSizeChartId,
   );
@@ -580,10 +578,7 @@ const DashboardBuilder: FC<DashboardBuilderProps> = () => {
         {!hideDashboardHeader && <DashboardHeader />}
         {showFilterBar &&
           filterBarOrientation === FilterBarOrientation.HORIZONTAL && (
-            <FilterBar
-              focusedFilterId={focusedFilterId}
-              orientation={FilterBarOrientation.HORIZONTAL}
-            />
+            <FilterBar orientation={FilterBarOrientation.HORIZONTAL} />
           )}
         {dropIndicatorProps && <div {...dropIndicatorProps} />}
         {!isReport && topLevelTabs && !uiConfig.hideNav && (
@@ -613,7 +608,6 @@ const DashboardBuilder: FC<DashboardBuilderProps> = () => {
       </div>
     ),
     [
-      focusedFilterId,
       nativeFiltersEnabled,
       filterBarOrientation,
       editMode,
@@ -658,7 +652,6 @@ const DashboardBuilder: FC<DashboardBuilderProps> = () => {
                     <ErrorBoundary>
                       {!isReport && (
                         <FilterBar
-                          focusedFilterId={focusedFilterId}
                           orientation={FilterBarOrientation.VERTICAL}
                           verticalConfig={{
                             filtersOpen: dashboardFiltersOpen,

superset-frontend/src/dashboard/components/FiltersBadge/index.tsx

@@ -23,7 +23,13 @@ import cx from 'classnames';
 import { DataMaskStateWithId, Filters } from '@superset-ui/core';
 import Icons from 'src/components/Icons';
 import { usePrevious } from 'src/hooks/usePrevious';
-import { setFocusedNativeFilter } from 'src/dashboard/actions/nativeFilters';
+import { setDirectPathToChild } from 'src/dashboard/actions/dashboardState';
+import {
+  ChartsState,
+  DashboardInfo,
+  DashboardLayout,
+  RootState,
+} from 'src/dashboard/types';
 import DetailsPanelPopover from './DetailsPanel';
 import { Pill } from './Styles';
 import {
@@ -32,12 +38,6 @@ import {
   selectIndicatorsForChart,
   selectNativeIndicatorsForChart,
 } from './selectors';
-import {
-  ChartsState,
-  DashboardInfo,
-  DashboardLayout,
-  RootState,
-} from '../../types';
 
 export interface FiltersBadgeProps {
   chartId: number;
@@ -87,7 +87,7 @@ export const FiltersBadge = ({ chartId }: FiltersBadgeProps) => {
 
   const onHighlightFilterSource = useCallback(
     (path: string[]) => {
-      dispatch(setFocusedNativeFilter(path[0]));
+      dispatch(setDirectPathToChild(path));
     },
     [dispatch],
   );

superset-frontend/src/dashboard/components/FiltersBadge/selectors.ts

@@ -23,6 +23,7 @@ import {
   FeatureFlag,
   Filters,
   FilterState,
+  getColumnLabel,
   isFeatureEnabled,
   NativeFilterType,
   NO_TIME_RANGE,
@@ -145,8 +146,8 @@ const getAppliedColumns = (chart: any): Set<string> =>
 
 const getRejectedColumns = (chart: any): Set<string> =>
   new Set(
-    (chart?.queriesResponse?.[0]?.rejected_filters || []).map(
-      (filter: any) => filter.column,
+    (chart?.queriesResponse?.[0]?.rejected_filters || []).map((filter: any) =>
+      getColumnLabel(filter.column),
     ),
   );
 

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/FilterBarSettings/index.tsx

@@ -144,7 +144,7 @@ const FilterBarSettings = () => {
   );
   const menuItems: DropDownSelectableProps['menuItems'] = [];
 
-  if (isCrossFiltersFeatureEnabled) {
+  if (isCrossFiltersFeatureEnabled && canEdit) {
     menuItems.unshift({
       key: crossFiltersMenuKey,
       label: crossFiltersMenuItem,

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/FilterControls/FilterControl.tsx

@@ -222,7 +222,6 @@ const FilterControl = ({
   filter,
   icon,
   onFilterSelectionChange,
-  focusedFilterId,
   inView,
   showOverflow,
   parentRef,
@@ -288,7 +287,6 @@ const FilterControl = ({
           dataMaskSelected={dataMaskSelected}
           filter={filter}
           showOverflow={showOverflow}
-          focusedFilterId={focusedFilterId}
           onFilterSelectionChange={onFilterSelectionChange}
           inView={inView}
           parentRef={parentRef}

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/FilterControls/FilterControls.tsx

@@ -54,15 +54,14 @@ import Icons from 'src/components/Icons';
 import { FiltersOutOfScopeCollapsible } from '../FiltersOutOfScopeCollapsible';
 import { useFilterControlFactory } from '../useFilterControlFactory';
 import { FiltersDropdownContent } from '../FiltersDropdownContent';
+import { useFilterOutlined } from '../useFilterOutlined';
 
 type FilterControlsProps = {
-  focusedFilterId?: string;
   dataMaskSelected: DataMaskStateWithId;
   onFilterSelectionChange: (filter: Filter, dataMask: DataMask) => void;
 };
 
 const FilterControls: FC<FilterControlsProps> = ({
-  focusedFilterId,
   dataMaskSelected,
   onFilterSelectionChange,
 }) => {
@@ -73,12 +72,13 @@ const FilterControls: FC<FilterControlsProps> = ({
         : FilterBarOrientation.VERTICAL,
   );
 
+  const { outlinedFilterId, lastUpdated } = useFilterOutlined();
+
   const [overflowedIds, setOverflowedIds] = useState<string[]>([]);
   const popoverRef = useRef<DropdownContainerRef>(null);
 
   const { filterControlFactory, filtersWithValues } = useFilterControlFactory(
     dataMaskSelected,
-    focusedFilterId,
     onFilterSelectionChange,
   );
   const portalNodes = useMemo(() => {
@@ -94,6 +94,11 @@ const FilterControls: FC<FilterControlsProps> = ({
   const [filtersInScope, filtersOutOfScope] =
     useSelectFiltersInScope(filtersWithValues);
 
+  const hasRequiredFirst = useMemo(
+    () => filtersWithValues.some(filter => filter.requiredFirst),
+    [filtersWithValues],
+  );
+
   const dashboardHasTabs = useDashboardHasTabs();
   const showCollapsePanel = dashboardHasTabs && filtersWithValues.length > 0;
 
@@ -119,6 +124,7 @@ const FilterControls: FC<FilterControlsProps> = ({
       {showCollapsePanel && (
         <FiltersOutOfScopeCollapsible
           filtersOutOfScope={filtersOutOfScope}
+          forceRender={hasRequiredFirst}
           hasTopMargin={filtersInScope.length > 0}
           renderer={renderer}
         />
@@ -200,6 +206,7 @@ const FilterControls: FC<FilterControlsProps> = ({
                   filtersOutOfScope={filtersOutOfScope}
                   renderer={renderer}
                   showCollapsePanel={showCollapsePanel}
+                  forceRenderOutOfScope={hasRequiredFirst}
                 />
               )
             : undefined
@@ -234,10 +241,10 @@ const FilterControls: FC<FilterControlsProps> = ({
   }, [filtersOutOfScope, filtersWithValues, overflowedFiltersInScope]);
 
   useEffect(() => {
-    if (focusedFilterId && overflowedIds.includes(focusedFilterId)) {
+    if (outlinedFilterId && overflowedIds.includes(outlinedFilterId)) {
       popoverRef?.current?.open();
     }
-  }, [focusedFilterId, popoverRef, overflowedIds]);
+  }, [outlinedFilterId, lastUpdated, popoverRef, overflowedIds]);
 
   return (
     <>

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/FilterControls/FilterValue.tsx

@@ -43,13 +43,17 @@ import { FeatureFlag, isFeatureEnabled } from 'src/featureFlags';
 import { waitForAsyncData } from 'src/middleware/asyncEvent';
 import { ClientErrorObject } from 'src/utils/getClientErrorObject';
 import { FilterBarOrientation, RootState } from 'src/dashboard/types';
-import { onFiltersRefreshSuccess } from 'src/dashboard/actions/dashboardState';
+import {
+  onFiltersRefreshSuccess,
+  setDirectPathToChild,
+} from 'src/dashboard/actions/dashboardState';
 import { FAST_DEBOUNCE } from 'src/constants';
 import { dispatchHoverAction, dispatchFocusAction } from './utils';
 import { FilterControlProps } from './types';
 import { getFormData } from '../../utils';
 import { useFilterDependencies } from './state';
 import { checkIsMissingRequiredValue } from '../utils';
+import { useFilterOutlined } from '../useFilterOutlined';
 
 const HEIGHT = 32;
 
@@ -79,7 +83,6 @@ const useShouldFilterRefresh = () => {
 const FilterValue: React.FC<FilterControlProps> = ({
   dataMaskSelected,
   filter,
-  focusedFilterId,
   onFilterSelectionChange,
   inView = true,
   showOverflow,
@@ -111,6 +114,8 @@ const FilterValue: React.FC<FilterControlProps> = ({
   const [isRefreshing, setIsRefreshing] = useState(false);
   const dispatch = useDispatch();
 
+  const { outlinedFilterId, lastUpdated } = useFilterOutlined();
+
   const handleFilterLoadFinish = useCallback(() => {
     setIsRefreshing(false);
     setIsLoading(false);
@@ -212,26 +217,34 @@ const FilterValue: React.FC<FilterControlProps> = ({
   ]);
 
   useEffect(() => {
-    if (focusedFilterId && focusedFilterId === filter.id) {
-      setTimeout(() => {
-        inputRef?.current?.focus();
-      }, FAST_DEBOUNCE);
+    if (outlinedFilterId && outlinedFilterId === filter.id) {
+      setTimeout(
+        () => {
+          inputRef?.current?.focus();
+        },
+        overflow ? FAST_DEBOUNCE : 0,
+      );
     }
-  }, [inputRef, focusedFilterId, filter.id]);
+  }, [inputRef, outlinedFilterId, lastUpdated, filter.id, overflow]);
 
   const setDataMask = useCallback(
     (dataMask: DataMask) => onFilterSelectionChange(filter, dataMask),
     [filter, onFilterSelectionChange],
   );
 
-  const setFocusedFilter = useCallback(
-    () => dispatchFocusAction(dispatch, id),
-    [dispatch, id],
-  );
-  const unsetFocusedFilter = useCallback(
-    () => dispatchFocusAction(dispatch),
-    [dispatch],
-  );
+  const setFocusedFilter = useCallback(() => {
+    // don't highlight charts in scope if filter was focused programmatically
+    if (outlinedFilterId !== id) {
+      dispatchFocusAction(dispatch, id);
+    }
+  }, [dispatch, id, outlinedFilterId]);
+
+  const unsetFocusedFilter = useCallback(() => {
+    dispatchFocusAction(dispatch);
+    if (outlinedFilterId === id) {
+      dispatch(setDirectPathToChild([]));
+    }
+  }, [dispatch, id, outlinedFilterId]);
 
   const setHoveredFilter = useCallback(
     () => dispatchHoverAction(dispatch, id),

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/FiltersDropdownContent/index.tsx

@@ -26,6 +26,7 @@ export interface FiltersDropdownContentProps {
   filtersOutOfScope: (Filter | Divider)[];
   renderer: (filter: Filter | Divider, index: number) => ReactNode;
   showCollapsePanel?: boolean;
+  forceRenderOutOfScope?: boolean;
 }
 
 export const FiltersDropdownContent = ({
@@ -33,6 +34,7 @@ export const FiltersDropdownContent = ({
   filtersOutOfScope,
   renderer,
   showCollapsePanel,
+  forceRenderOutOfScope,
 }: FiltersDropdownContentProps) => (
   <div
     css={(theme: SupersetTheme) =>
@@ -47,6 +49,7 @@ export const FiltersDropdownContent = ({
       <FiltersOutOfScopeCollapsible
         filtersOutOfScope={filtersOutOfScope}
         renderer={renderer}
+        forceRender={forceRenderOutOfScope}
         horizontalOverflow
       />
     )}

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/FiltersOutOfScopeCollapsible/index.tsx

@@ -26,6 +26,7 @@ export interface FiltersOutOfScopeCollapsibleProps {
   renderer: (filter: Filter | Divider, index: number) => ReactNode;
   hasTopMargin?: boolean;
   horizontalOverflow?: boolean;
+  forceRender?: boolean;
 }
 
 export const FiltersOutOfScopeCollapsible = ({
@@ -33,6 +34,7 @@ export const FiltersOutOfScopeCollapsible = ({
   renderer,
   hasTopMargin,
   horizontalOverflow,
+  forceRender = false,
 }: FiltersOutOfScopeCollapsibleProps) => (
   <AntdCollapse
     ghost
@@ -80,6 +82,7 @@ export const FiltersOutOfScopeCollapsible = ({
     }
   >
     <AntdCollapse.Panel
+      forceRender={forceRender}
       header={t('Filters out of scope (%d)', filtersOutOfScope.length)}
       key="1"
     >

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/Horizontal.tsx

@@ -93,7 +93,6 @@ const HorizontalFilterBar: React.FC<HorizontalBarProps> = ({
   dataMaskSelected,
   filterValues,
   isInitialized,
-  focusedFilterId,
   onSelectionChange,
 }) => {
   const hasFilters = filterValues.length > 0;
@@ -124,7 +123,6 @@ const HorizontalFilterBar: React.FC<HorizontalBarProps> = ({
             {hasFilters && (
               <FilterControls
                 dataMaskSelected={dataMaskSelected}
-                focusedFilterId={focusedFilterId}
                 onFilterSelectionChange={onSelectionChange}
               />
             )}

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/Vertical.tsx

@@ -141,7 +141,6 @@ const VerticalFilterBar: React.FC<VerticalBarProps> = ({
   actions,
   canEdit,
   dataMaskSelected,
-  focusedFilterId,
   filtersOpen,
   filterValues,
   height,
@@ -258,7 +257,6 @@ const VerticalFilterBar: React.FC<VerticalBarProps> = ({
                   <FilterControlsWrapper>
                     <FilterControls
                       dataMaskSelected={dataMaskSelected}
-                      focusedFilterId={focusedFilterId}
                       onFilterSelectionChange={onSelectionChange}
                     />
                   </FilterControlsWrapper>
@@ -300,7 +298,6 @@ const VerticalFilterBar: React.FC<VerticalBarProps> = ({
                 <FilterControlsWrapper>
                   <FilterControls
                     dataMaskSelected={dataMaskSelected}
-                    focusedFilterId={focusedFilterId}
                     onFilterSelectionChange={onSelectionChange}
                   />
                 </FilterControlsWrapper>

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/index.tsx

@@ -111,7 +111,6 @@ const publishDataMask = debounce(
 
 export const FilterBarScrollContext = createContext(false);
 const FilterBar: React.FC<FiltersBarProps> = ({
-  focusedFilterId,
   orientation = FilterBarOrientation.VERTICAL,
   verticalConfig,
 }) => {
@@ -254,7 +253,6 @@ const FilterBar: React.FC<FiltersBarProps> = ({
       canEdit={canEdit}
       dashboardId={dashboardId}
       dataMaskSelected={dataMaskSelected}
-      focusedFilterId={focusedFilterId}
       filterValues={filterValues}
       isInitialized={isInitialized}
       onSelectionChange={handleFilterSelectionChange}
@@ -264,7 +262,6 @@ const FilterBar: React.FC<FiltersBarProps> = ({
       actions={actions}
       canEdit={canEdit}
       dataMaskSelected={dataMaskSelected}
-      focusedFilterId={focusedFilterId}
       filtersOpen={verticalConfig.filtersOpen}
       filterValues={filterValues}
       isInitialized={isInitialized}

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/types.ts

@@ -16,6 +16,8 @@
  * specific language governing permissions and limitations
  * under the License.
  */
+
+import { ReactNode } from 'react';
 import {
   DataMask,
   DataMaskStateWithId,
@@ -25,10 +27,9 @@ import {
 import { FilterBarOrientation } from 'src/dashboard/types';
 
 interface CommonFiltersBarProps {
-  actions: React.ReactNode;
+  actions: ReactNode;
   canEdit: boolean;
   dataMaskSelected: DataMaskStateWithId;
-  focusedFilterId?: string;
   filterValues: (Filter | Divider)[];
   isInitialized: boolean;
   onSelectionChange: (
@@ -45,8 +46,7 @@ interface VerticalBarConfig {
   width: number;
 }
 
-export interface FiltersBarProps
-  extends Pick<CommonFiltersBarProps, 'focusedFilterId'> {
+export interface FiltersBarProps {
   orientation: FilterBarOrientation;
   verticalConfig?: VerticalBarConfig;
 }

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/useFilterControlFactory.tsx

@@ -32,7 +32,6 @@ import FilterDivider from './FilterControls/FilterDivider';
 
 export const useFilterControlFactory = (
   dataMaskSelected: DataMaskStateWithId,
-  focusedFilterId: string | undefined,
   onFilterSelectionChange: (filter: Filter, dataMask: DataMask) => void,
 ) => {
   const filters = useFilters();
@@ -67,7 +66,6 @@ export const useFilterControlFactory = (
         <FilterControl
           dataMaskSelected={dataMaskSelected}
           filter={filter}
-          focusedFilterId={focusedFilterId}
           onFilterSelectionChange={onFilterSelectionChange}
           inView={false}
           orientation={filterBarOrientation}
@@ -75,12 +73,7 @@ export const useFilterControlFactory = (
         />
       );
     },
-    [
-      filtersWithValues,
-      dataMaskSelected,
-      focusedFilterId,
-      onFilterSelectionChange,
-    ],
+    [filtersWithValues, dataMaskSelected, onFilterSelectionChange],
   );
 
   return { filterControlFactory, filtersWithValues };

superset-frontend/src/dashboard/components/nativeFilters/FilterBar/useFilterOutlined.ts

@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { useSelector } from 'react-redux';
+import { RootState } from 'src/dashboard/types';
+import getChartAndLabelComponentIdFromPath from 'src/dashboard/util/getChartAndLabelComponentIdFromPath';
+
+export const useFilterOutlined = () =>
+  useSelector<RootState, { outlinedFilterId: string; lastUpdated: number }>(
+    state => ({
+      outlinedFilterId: (
+        getChartAndLabelComponentIdFromPath(
+          state.dashboardState.directPathToChild || [],
+        ) as Record<string, string>
+      )?.native_filter,
+      lastUpdated: state.dashboardState.directPathLastUpdated,
+    }),
+  );

superset-frontend/src/dashboard/components/nativeFilters/FilterCard/DependenciesRow.tsx

@@ -21,7 +21,7 @@ import { useDispatch } from 'react-redux';
 import { css, t, useTheme } from '@superset-ui/core';
 import Icons from 'src/components/Icons';
 import { useTruncation } from 'src/hooks/useTruncation';
-import { setFocusedNativeFilter } from 'src/dashboard/actions/nativeFilters';
+import { setDirectPathToChild } from 'src/dashboard/actions/dashboardState';
 import {
   DependencyItem,
   Row,
@@ -40,7 +40,7 @@ const DependencyValue = ({
 }: DependencyValueProps) => {
   const dispatch = useDispatch();
   const handleClick = useCallback(() => {
-    dispatch(setFocusedNativeFilter(dependency.id));
+    dispatch(setDirectPathToChild([dependency.id]));
   }, [dependency.id, dispatch]);
   return (
     <span>

superset-frontend/src/dashboard/components/nativeFilters/FilterCard/FilterCard.test.tsx

@@ -23,7 +23,7 @@ import { Filter, NativeFilterType } from '@superset-ui/core';
 import userEvent from '@testing-library/user-event';
 import { render, screen } from 'spec/helpers/testing-library';
 import { DASHBOARD_ROOT_ID } from 'src/dashboard/util/constants';
-import { SET_FOCUSED_NATIVE_FILTER } from 'src/dashboard/actions/nativeFilters';
+import { SET_DIRECT_PATH } from 'src/dashboard/actions/dashboardState';
 import { FilterCardContent } from './FilterCardContent';
 
 const baseInitialState = {
@@ -304,8 +304,8 @@ test('focus filter on filter card dependency click', () => {
 
   userEvent.click(screen.getByText('Native filter 2'));
   expect(dummyDispatch).toHaveBeenCalledWith({
-    type: SET_FOCUSED_NATIVE_FILTER,
-    id: 'NATIVE_FILTER-2',
+    type: SET_DIRECT_PATH,
+    path: ['NATIVE_FILTER-2'],
   });
 });
 

superset-frontend/src/explore/components/ControlPanelsContainer.test.tsx

@@ -17,6 +17,7 @@
  * under the License.
  */
 import React from 'react';
+import userEvent from '@testing-library/user-event';
 import { render, screen } from 'spec/helpers/testing-library';
 import {
   DatasourceType,
@@ -104,5 +105,43 @@ describe('ControlPanelsContainer', () => {
     expect(
       await screen.findAllByTestId('collapsible-control-panel-header'),
     ).toHaveLength(4);
+    expect(screen.getByRole('tab', { name: /customize/i })).toBeInTheDocument();
+    userEvent.click(screen.getByRole('tab', { name: /customize/i }));
+    expect(
+      await screen.findAllByTestId('collapsible-control-panel-header'),
+    ).toHaveLength(5);
+  });
+
+  test('renders ControlPanelSections no Customize Tab', async () => {
+    getChartControlPanelRegistry().registerValue('table', {
+      controlPanelSections: [
+        {
+          label: t('GROUP BY'),
+          description: t(
+            'Use this section if you want a query that aggregates',
+          ),
+          expanded: true,
+          controlSetRows: [
+            ['groupby'],
+            ['metrics'],
+            ['percent_metrics'],
+            ['timeseries_limit_metric', 'row_limit'],
+            ['include_time', 'order_desc'],
+          ],
+        },
+        {
+          label: t('Options'),
+          expanded: true,
+          controlSetRows: [],
+        },
+      ],
+    });
+    render(<ControlPanelsContainer {...getDefaultProps()} />, {
+      useRedux: true,
+    });
+    expect(screen.queryByText(/customize/i)).not.toBeInTheDocument();
+    expect(
+      await screen.findAllByTestId('collapsible-control-panel-header'),
+    ).toHaveLength(2);
   });
 });

superset-frontend/src/explore/components/ControlPanelsContainer.tsx

@@ -235,7 +235,7 @@ function getState(
       )
     ) {
       querySections.push(section);
-    } else {
+    } else if (section.controlSetRows.length > 0) {
       customizeSections.push(section);
     }
   });

superset-frontend/src/views/CRUD/data/database/DatabaseModal/index.tsx

@@ -1385,6 +1385,69 @@ const DatabaseModal: FunctionComponent<DatabaseModalProps> = ({
     </StyledBtns>
   );
 
+  const renderDatabaseConnectionForm = () => (
+    <>
+      <DatabaseConnectionForm
+        isEditMode={isEditMode}
+        db={db as DatabaseObject}
+        sslForced={sslForced}
+        dbModel={dbModel}
+        onAddTableCatalog={() => {
+          setDB({ type: ActionType.addTableCatalogSheet });
+        }}
+        onQueryChange={({ target }: { target: HTMLInputElement }) =>
+          onChange(ActionType.queryChange, {
+            name: target.name,
+            value: target.value,
+          })
+        }
+        onExtraInputChange={({ target }: { target: HTMLInputElement }) =>
+          onChange(ActionType.extraInputChange, {
+            name: target.name,
+            value: target.value,
+          })
+        }
+        onRemoveTableCatalog={(idx: number) => {
+          setDB({
+            type: ActionType.removeTableCatalogSheet,
+            payload: { indexToDelete: idx },
+          });
+        }}
+        onParametersChange={({ target }: { target: HTMLInputElement }) =>
+          onChange(ActionType.parametersChange, {
+            type: target.type,
+            name: target.name,
+            checked: target.checked,
+            value: target.value,
+          })
+        }
+        onChange={({ target }: { target: HTMLInputElement }) =>
+          onChange(ActionType.textChange, {
+            name: target.name,
+            value: target.value,
+          })
+        }
+        getValidation={() => getValidation(db)}
+        validationErrors={validationErrors}
+        getPlaceholder={getPlaceholder}
+      />
+      <SSHTunnelContainer>
+        <SSHTunnelSwitchComponent
+          isEditMode={isEditMode}
+          dbFetched={dbFetched}
+          disableSSHTunnelingForEngine={disableSSHTunnelingForEngine}
+          useSSHTunneling={useSSHTunneling}
+          setUseSSHTunneling={setUseSSHTunneling}
+          setDB={setDB}
+          isSSHTunneling={isSSHTunneling}
+        />
+      </SSHTunnelContainer>
+      {useSSHTunneling && (
+        <SSHTunnelContainer>{renderSSHTunnelForm()}</SSHTunnelContainer>
+      )}
+    </>
+  );
+
   const renderFinishState = () => {
     if (!editNewDb) {
       return (
@@ -1421,51 +1484,7 @@ const DatabaseModal: FunctionComponent<DatabaseModalProps> = ({
         />
       );
     }
-    return (
-      <DatabaseConnectionForm
-        isEditMode
-        sslForced={sslForced}
-        dbModel={dbModel}
-        db={db as DatabaseObject}
-        onParametersChange={({ target }: { target: HTMLInputElement }) =>
-          onChange(ActionType.parametersChange, {
-            type: target.type,
-            name: target.name,
-            checked: target.checked,
-            value: target.value,
-          })
-        }
-        onExtraInputChange={({ target }: { target: HTMLInputElement }) =>
-          onChange(ActionType.extraInputChange, {
-            name: target.name,
-            value: target.value,
-          })
-        }
-        onChange={({ target }: { target: HTMLInputElement }) =>
-          onChange(ActionType.textChange, {
-            name: target.name,
-            value: target.value,
-          })
-        }
-        onQueryChange={({ target }: { target: HTMLInputElement }) =>
-          onChange(ActionType.queryChange, {
-            name: target.name,
-            value: target.value,
-          })
-        }
-        onAddTableCatalog={() =>
-          setDB({ type: ActionType.addTableCatalogSheet })
-        }
-        onRemoveTableCatalog={(idx: number) =>
-          setDB({
-            type: ActionType.removeTableCatalogSheet,
-            payload: { indexToDelete: idx },
-          })
-        }
-        getValidation={() => getValidation(db)}
-        validationErrors={validationErrors}
-      />
-    );
+    return renderDatabaseConnectionForm();
   };
 
   if (fileList.length > 0 && (alreadyExists.length || passwordFields.length)) {
@@ -1604,49 +1623,7 @@ const DatabaseModal: FunctionComponent<DatabaseModalProps> = ({
               )}
             </StyledAlignment>
           ) : (
-            <DatabaseConnectionForm
-              isEditMode
-              sslForced={sslForced}
-              dbModel={dbModel}
-              db={db as DatabaseObject}
-              onParametersChange={({ target }: { target: HTMLInputElement }) =>
-                onChange(ActionType.parametersChange, {
-                  type: target.type,
-                  name: target.name,
-                  checked: target.checked,
-                  value: target.value,
-                })
-              }
-              onExtraInputChange={({ target }: { target: HTMLInputElement }) =>
-                onChange(ActionType.extraInputChange, {
-                  name: target.name,
-                  value: target.value,
-                })
-              }
-              onChange={({ target }: { target: HTMLInputElement }) =>
-                onChange(ActionType.textChange, {
-                  name: target.name,
-                  value: target.value,
-                })
-              }
-              onQueryChange={({ target }: { target: HTMLInputElement }) =>
-                onChange(ActionType.queryChange, {
-                  name: target.name,
-                  value: target.value,
-                })
-              }
-              onAddTableCatalog={() =>
-                setDB({ type: ActionType.addTableCatalogSheet })
-              }
-              onRemoveTableCatalog={(idx: number) =>
-                setDB({
-                  type: ActionType.removeTableCatalogSheet,
-                  payload: { indexToDelete: idx },
-                })
-              }
-              getValidation={() => getValidation(db)}
-              validationErrors={validationErrors}
-            />
+            renderDatabaseConnectionForm()
           )}
           {!isEditMode && (
             <StyledAlertMargin>
@@ -1796,73 +1773,7 @@ const DatabaseModal: FunctionComponent<DatabaseModalProps> = ({
                   dbModel={dbModel}
                 />
                 {hasAlert && renderStepTwoAlert()}
-                <DatabaseConnectionForm
-                  db={db}
-                  sslForced={sslForced}
-                  dbModel={dbModel}
-                  onAddTableCatalog={() => {
-                    setDB({ type: ActionType.addTableCatalogSheet });
-                  }}
-                  onQueryChange={({ target }: { target: HTMLInputElement }) =>
-                    onChange(ActionType.queryChange, {
-                      name: target.name,
-                      value: target.value,
-                    })
-                  }
-                  onExtraInputChange={({
-                    target,
-                  }: {
-                    target: HTMLInputElement;
-                  }) =>
-                    onChange(ActionType.extraInputChange, {
-                      name: target.name,
-                      value: target.value,
-                    })
-                  }
-                  onRemoveTableCatalog={(idx: number) => {
-                    setDB({
-                      type: ActionType.removeTableCatalogSheet,
-                      payload: { indexToDelete: idx },
-                    });
-                  }}
-                  onParametersChange={({
-                    target,
-                  }: {
-                    target: HTMLInputElement;
-                  }) =>
-                    onChange(ActionType.parametersChange, {
-                      type: target.type,
-                      name: target.name,
-                      checked: target.checked,
-                      value: target.value,
-                    })
-                  }
-                  onChange={({ target }: { target: HTMLInputElement }) =>
-                    onChange(ActionType.textChange, {
-                      name: target.name,
-                      value: target.value,
-                    })
-                  }
-                  getValidation={() => getValidation(db)}
-                  validationErrors={validationErrors}
-                  getPlaceholder={getPlaceholder}
-                />
-                <SSHTunnelContainer>
-                  <SSHTunnelSwitchComponent
-                    isEditMode={isEditMode}
-                    dbFetched={dbFetched}
-                    disableSSHTunnelingForEngine={disableSSHTunnelingForEngine}
-                    useSSHTunneling={useSSHTunneling}
-                    setUseSSHTunneling={setUseSSHTunneling}
-                    setDB={setDB}
-                    isSSHTunneling={isSSHTunneling}
-                  />
-                </SSHTunnelContainer>
-                {useSSHTunneling && (
-                  <SSHTunnelContainer>
-                    {renderSSHTunnelForm()}
-                  </SSHTunnelContainer>
-                )}
+                {renderDatabaseConnectionForm()}
                 <div css={(theme: SupersetTheme) => infoTooltip(theme)}>
                   {dbModel.engine !== Engines.GSheet && (
                     <>

superset/charts/filters.py

@@ -18,13 +18,15 @@ from typing import Any
 
 from flask_babel import lazy_gettext as _
 from sqlalchemy import and_, or_
+from sqlalchemy.orm import aliased
 from sqlalchemy.orm.query import Query
 
-from superset import db, security_manager
+from superset import security_manager
 from superset.connectors.sqla import models
 from superset.connectors.sqla.models import SqlaTable
 from superset.models.slice import Slice
 from superset.utils.core import get_user_id
+from superset.utils.filters import get_dataset_access_filters
 from superset.views.base import BaseFilter
 from superset.views.base_api import BaseFavoriteFilter
 
@@ -77,23 +79,13 @@ class ChartFilter(BaseFilter):  # pylint: disable=too-few-public-methods
     def apply(self, query: Query, value: Any) -> Query:
         if security_manager.can_access_all_datasources():
             return query
-        perms = security_manager.user_view_menu_names("datasource_access")
-        schema_perms = security_manager.user_view_menu_names("schema_access")
-        owner_ids_query = (
-            db.session.query(models.SqlaTable.id)
-            .join(models.SqlaTable.owners)
-            .filter(
-                security_manager.user_model.id
-                == security_manager.user_model.get_user_id()
-            )
-        )
-        return query.filter(
-            or_(
-                self.model.perm.in_(perms),
-                self.model.schema_perm.in_(schema_perms),
-                models.SqlaTable.id.in_(owner_ids_query),
-            )
+
+        table_alias = aliased(SqlaTable)
+        query = query.join(table_alias, self.model.datasource_id == table_alias.id)
+        query = query.join(
+            models.Database, table_alias.database_id == models.Database.id
         )
+        return query.filter(get_dataset_access_filters(self.model))
 
 
 class ChartHasCreatedByFilter(BaseFilter):  # pylint: disable=too-few-public-methods

superset/common/query_actions.py

@@ -17,7 +17,7 @@
 from __future__ import annotations
 
 import copy
-from typing import Any, Callable, cast, Dict, List, Optional, TYPE_CHECKING
+from typing import Any, Callable, Dict, Optional, TYPE_CHECKING
 
 from flask_babel import _
 
@@ -32,7 +32,6 @@ from superset.utils.core import (
     ExtraFiltersReasonType,
     get_column_name,
     get_time_filter_status,
-    is_adhoc_column,
 )
 
 if TYPE_CHECKING:
@@ -102,7 +101,6 @@ def _get_full(
     datasource = _get_datasource(query_context, query_obj)
     result_type = query_obj.result_type or query_context.result_type
     payload = query_context.get_df_payload(query_obj, force_cached=force_cached)
-    applied_template_filters = payload.get("applied_template_filters", [])
     df = payload["df"]
     status = payload["status"]
     if status != QueryStatus.FAILED:
@@ -113,23 +111,23 @@ def _get_full(
         payload["result_format"] = query_context.result_format
     del payload["df"]
 
-    filters = query_obj.filter
-    filter_columns = cast(List[str], [flt.get("col") for flt in filters])
-    columns = set(datasource.column_names)
     applied_time_columns, rejected_time_columns = get_time_filter_status(
         datasource, query_obj.applied_time_extras
     )
+
+    applied_filter_columns = payload.get("applied_filter_columns", [])
+    rejected_filter_columns = payload.get("rejected_filter_columns", [])
+    del payload["applied_filter_columns"]
+    del payload["rejected_filter_columns"]
     payload["applied_filters"] = [
-        {"column": get_column_name(col)}
-        for col in filter_columns
-        if is_adhoc_column(col) or col in columns or col in applied_template_filters
+        {"column": get_column_name(col)} for col in applied_filter_columns
     ] + applied_time_columns
     payload["rejected_filters"] = [
-        {"reason": ExtraFiltersReasonType.COL_NOT_IN_DATASOURCE, "column": col}
-        for col in filter_columns
-        if not is_adhoc_column(col)
-        and col not in columns
-        and col not in applied_template_filters
+        {
+            "reason": ExtraFiltersReasonType.COL_NOT_IN_DATASOURCE,
+            "column": get_column_name(col),
+        }
+        for col in rejected_filter_columns
     ] + rejected_time_columns
 
     if result_type == ChartDataResultType.RESULTS and status != QueryStatus.FAILED:

superset/common/query_context_processor.py

@@ -165,6 +165,8 @@ class QueryContextProcessor:
             "cache_timeout": self.get_cache_timeout(),
             "df": cache.df,
             "applied_template_filters": cache.applied_template_filters,
+            "applied_filter_columns": cache.applied_filter_columns,
+            "rejected_filter_columns": cache.rejected_filter_columns,
             "annotation_data": cache.annotation_data,
             "error": cache.error_message,
             "is_cached": cache.is_cached,

superset/common/utils/query_cache_manager.py

@@ -29,6 +29,7 @@ from superset.exceptions import CacheLoadError
 from superset.extensions import cache_manager
 from superset.models.helpers import QueryResult
 from superset.stats_logger import BaseStatsLogger
+from superset.superset_typing import Column
 from superset.utils.cache import set_and_log_cache
 from superset.utils.core import error_msg_from_exception, get_stacktrace
 
@@ -54,6 +55,8 @@ class QueryCacheManager:
         query: str = "",
         annotation_data: Optional[Dict[str, Any]] = None,
         applied_template_filters: Optional[List[str]] = None,
+        applied_filter_columns: Optional[List[Column]] = None,
+        rejected_filter_columns: Optional[List[Column]] = None,
         status: Optional[str] = None,
         error_message: Optional[str] = None,
         is_loaded: bool = False,
@@ -66,6 +69,8 @@ class QueryCacheManager:
         self.query = query
         self.annotation_data = {} if annotation_data is None else annotation_data
         self.applied_template_filters = applied_template_filters or []
+        self.applied_filter_columns = applied_filter_columns or []
+        self.rejected_filter_columns = rejected_filter_columns or []
         self.status = status
         self.error_message = error_message
 
@@ -93,6 +98,8 @@ class QueryCacheManager:
             self.status = query_result.status
             self.query = query_result.query
             self.applied_template_filters = query_result.applied_template_filters
+            self.applied_filter_columns = query_result.applied_filter_columns
+            self.rejected_filter_columns = query_result.rejected_filter_columns
             self.error_message = query_result.error_message
             self.df = query_result.df
             self.annotation_data = {} if annotation_data is None else annotation_data
@@ -107,6 +114,8 @@ class QueryCacheManager:
                 "df": self.df,
                 "query": self.query,
                 "applied_template_filters": self.applied_template_filters,
+                "applied_filter_columns": self.applied_filter_columns,
+                "rejected_filter_columns": self.rejected_filter_columns,
                 "annotation_data": self.annotation_data,
             }
             if self.is_loaded and key and self.status != QueryStatus.FAILED:
@@ -150,6 +159,12 @@ class QueryCacheManager:
                 query_cache.applied_template_filters = cache_value.get(
                     "applied_template_filters", []
                 )
+                query_cache.applied_filter_columns = cache_value.get(
+                    "applied_filter_columns", []
+                )
+                query_cache.rejected_filter_columns = cache_value.get(
+                    "rejected_filter_columns", []
+                )
                 query_cache.status = QueryStatus.SUCCESS
                 query_cache.is_loaded = True
                 query_cache.is_cached = cache_value is not None

superset/config.py

@@ -188,10 +188,11 @@ CUSTOM_SECURITY_MANAGER = None
 SQLALCHEMY_TRACK_MODIFICATIONS = False
 # ---------------------------------------------------------
 
-# Your App secret key. Make sure you override it on superset_config.py.
+# Your App secret key. Make sure you override it on superset_config.py
+# or use `SUPERSET_SECRET_KEY` environment variable.
 # Use a strong complex alphanumeric string and use a tool to help you generate
 # a sufficiently random sequence, ex: openssl rand -base64 42"
-SECRET_KEY = CHANGE_ME_SECRET_KEY
+SECRET_KEY = os.environ.get("SUPERSET_SECRET_KEY") or CHANGE_ME_SECRET_KEY
 
 # The SQLAlchemy connection string.
 SQLALCHEMY_DATABASE_URI = "sqlite:///" + os.path.join(DATA_DIR, "superset.db")
@@ -1301,9 +1302,8 @@ WEBDRIVER_AUTH_FUNC = None
 WEBDRIVER_CONFIGURATION: Dict[Any, Any] = {"service_log_path": "/dev/null"}
 
 # Additional args to be passed as arguments to the config object
-# Note: these options are Chrome-specific. For FF, these should
-# only include the "--headless" arg
-WEBDRIVER_OPTION_ARGS = ["--headless", "--marionette"]
+# Note: If using Chrome, you'll want to add the "--marionette" arg.
+WEBDRIVER_OPTION_ARGS = ["--headless"]
 
 # The base URL to query for accessing the user interface
 WEBDRIVER_BASEURL = "http://0.0.0.0:8080/"

superset/connectors/sqla/models.py

@@ -99,9 +99,11 @@ from superset.datasets.models import Dataset as NewDataset
 from superset.db_engine_specs.base import BaseEngineSpec, TimestampExpression
 from superset.exceptions import (
     AdvancedDataTypeResponseError,
+    ColumnNotFoundException,
     DatasetInvalidPermissionEvaluationException,
     QueryClauseValidationException,
     QueryObjectValidationError,
+    SupersetGenericDBErrorException,
     SupersetSecurityException,
 )
 from superset.extensions import feature_flag_manager
@@ -150,6 +152,8 @@ ADDITIVE_METRIC_TYPES_LOWER = {op.lower() for op in ADDITIVE_METRIC_TYPES}
 
 class SqlaQuery(NamedTuple):
     applied_template_filters: List[str]
+    applied_filter_columns: List[ColumnTyping]
+    rejected_filter_columns: List[ColumnTyping]
     cte: Optional[str]
     extra_cache_keys: List[Any]
     labels_expected: List[str]
@@ -159,6 +163,8 @@ class SqlaQuery(NamedTuple):
 
 class QueryStringExtended(NamedTuple):
     applied_template_filters: Optional[List[str]]
+    applied_filter_columns: List[ColumnTyping]
+    rejected_filter_columns: List[ColumnTyping]
     labels_expected: List[str]
     prequeries: List[str]
     sql: str
@@ -882,6 +888,8 @@ class SqlaTable(Model, BaseDatasource):  # pylint: disable=too-many-public-metho
         sql = self.mutate_query_from_config(sql)
         return QueryStringExtended(
             applied_template_filters=sqlaq.applied_template_filters,
+            applied_filter_columns=sqlaq.applied_filter_columns,
+            rejected_filter_columns=sqlaq.rejected_filter_columns,
             labels_expected=sqlaq.labels_expected,
             prequeries=sqlaq.prequeries,
             sql=sql,
@@ -1024,13 +1032,16 @@ class SqlaTable(Model, BaseDatasource):  # pylint: disable=too-many-public-metho
             )
             is_dttm = col_in_metadata.is_temporal
         else:
-            sqla_column = literal_column(expression)
-            # probe adhoc column type
-            tbl, _ = self.get_from_clause(template_processor)
-            qry = sa.select([sqla_column]).limit(1).select_from(tbl)
-            sql = self.database.compile_sqla_query(qry)
-            col_desc = get_columns_description(self.database, sql)
-            is_dttm = col_desc[0]["is_dttm"]
+            try:
+                sqla_column = literal_column(expression)
+                # probe adhoc column type
+                tbl, _ = self.get_from_clause(template_processor)
+                qry = sa.select([sqla_column]).limit(1).select_from(tbl)
+                sql = self.database.compile_sqla_query(qry)
+                col_desc = get_columns_description(self.database, sql)
+                is_dttm = col_desc[0]["is_dttm"]
+            except SupersetGenericDBErrorException as ex:
+                raise ColumnNotFoundException(message=str(ex)) from ex
 
         if (
             is_dttm
@@ -1185,6 +1196,8 @@ class SqlaTable(Model, BaseDatasource):  # pylint: disable=too-many-public-metho
         }
         columns = columns or []
         groupby = groupby or []
+        rejected_adhoc_filters_columns: List[Union[str, ColumnTyping]] = []
+        applied_adhoc_filters_columns: List[Union[str, ColumnTyping]] = []
         series_column_names = utils.get_column_names(series_columns or [])
         # deprecated, to be removed in 2.0
         if is_timeseries and timeseries_limit:
@@ -1443,9 +1456,14 @@ class SqlaTable(Model, BaseDatasource):  # pylint: disable=too-many-public-metho
             if flt_col == utils.DTTM_ALIAS and is_timeseries and dttm_col:
                 col_obj = dttm_col
             elif is_adhoc_column(flt_col):
-                sqla_col = self.adhoc_column_to_sqla(flt_col)
+                try:
+                    sqla_col = self.adhoc_column_to_sqla(flt_col)
+                    applied_adhoc_filters_columns.append(flt_col)
+                except ColumnNotFoundException:
+                    rejected_adhoc_filters_columns.append(flt_col)
+                    continue
             else:
-                col_obj = columns_by_name.get(flt_col)
+                col_obj = columns_by_name.get(cast(str, flt_col))
             filter_grain = flt.get("grain")
 
             if is_feature_enabled("ENABLE_TEMPLATE_REMOVE_FILTERS"):
@@ -1770,8 +1788,27 @@ class SqlaTable(Model, BaseDatasource):  # pylint: disable=too-many-public-metho
             qry = select([col]).select_from(qry.alias("rowcount_qry"))
             labels_expected = [label]
 
+        filter_columns = [flt.get("col") for flt in filter] if filter else []
+        rejected_filter_columns = [
+            col
+            for col in filter_columns
+            if col
+            and not is_adhoc_column(col)
+            and col not in self.column_names
+            and col not in applied_template_filters
+        ] + rejected_adhoc_filters_columns
+        applied_filter_columns = [
+            col
+            for col in filter_columns
+            if col
+            and not is_adhoc_column(col)
+            and (col in self.column_names or col in applied_template_filters)
+        ] + applied_adhoc_filters_columns
+
         return SqlaQuery(
             applied_template_filters=applied_template_filters,
+            rejected_filter_columns=rejected_filter_columns,
+            applied_filter_columns=applied_filter_columns,
             cte=cte,
             extra_cache_keys=extra_cache_keys,
             labels_expected=labels_expected,
@@ -1910,6 +1947,8 @@ class SqlaTable(Model, BaseDatasource):  # pylint: disable=too-many-public-metho
 
         return QueryResult(
             applied_template_filters=query_str_ext.applied_template_filters,
+            applied_filter_columns=query_str_ext.applied_filter_columns,
+            rejected_filter_columns=query_str_ext.rejected_filter_columns,
             status=status,
             df=df,
             duration=datetime.now() - qry_start_dttm,

superset/connectors/sqla/utils.py

@@ -17,6 +17,7 @@
 from __future__ import annotations
 
 import logging
+from functools import lru_cache
 from typing import (
     Any,
     Callable,
@@ -40,6 +41,7 @@ from sqlalchemy.orm import Session
 from sqlalchemy.orm.exc import ObjectDeletedError
 from sqlalchemy.sql.type_api import TypeEngine
 
+from superset.constants import LRU_CACHE_MAX_SIZE
 from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
 from superset.exceptions import (
     SupersetGenericDBErrorException,
@@ -49,7 +51,6 @@ from superset.models.core import Database
 from superset.result_set import SupersetResultSet
 from superset.sql_parse import has_table_query, insert_rls, ParsedQuery
 from superset.superset_typing import ResultSetColumnType
-from superset.utils.memoized import memoized
 
 if TYPE_CHECKING:
     from superset.connectors.sqla.models import SqlaTable
@@ -200,12 +201,12 @@ def validate_adhoc_subquery(
     return ";\n".join(str(statement) for statement in statements)
 
 
-@memoized
+@lru_cache(maxsize=LRU_CACHE_MAX_SIZE)
 def get_dialect_name(drivername: str) -> str:
     return SqlaURL.create(drivername).get_dialect().name
 
 
-@memoized
+@lru_cache(maxsize=LRU_CACHE_MAX_SIZE)
 def get_identifier_quoter(drivername: str) -> Dict[str, Callable[[str], str]]:
     return SqlaURL.create(drivername).get_dialect()().identifier_preparer.quote
 

superset/constants.py

@@ -37,6 +37,8 @@ NO_TIME_RANGE = "No filter"
 QUERY_CANCEL_KEY = "cancel_query"
 QUERY_EARLY_CANCEL_KEY = "early_cancel_query"
 
+LRU_CACHE_MAX_SIZE = 256
+
 
 class RouteMethod:  # pylint: disable=too-few-public-methods
     """

superset/dao/base.py

@@ -65,9 +65,9 @@ class BaseDAO:
             query = cls.base_filter(  # pylint: disable=not-callable
                 cls.id_column_name, data_model
             ).apply(query, None)
-        id_filter = {cls.id_column_name: model_id}
+        id_column = getattr(cls.model_cls, cls.id_column_name)
         try:
-            return query.filter_by(**id_filter).one_or_none()
+            return query.filter(id_column == model_id).one_or_none()
         except StatementError:
             # can happen if int is passed instead of a string or similar
             return None

superset/dashboards/filters.py

@@ -24,12 +24,14 @@ from sqlalchemy import and_, or_
 from sqlalchemy.orm.query import Query
 
 from superset import db, is_feature_enabled, security_manager
-from superset.models.core import FavStar
+from superset.connectors.sqla.models import SqlaTable
+from superset.models.core import Database, FavStar
 from superset.models.dashboard import Dashboard
 from superset.models.embedded_dashboard import EmbeddedDashboard
 from superset.models.slice import Slice
 from superset.security.guest_token import GuestTokenResourceType, GuestUser
 from superset.utils.core import get_user_id
+from superset.utils.filters import get_dataset_access_filters
 from superset.views.base import BaseFilter
 from superset.views.base_api import BaseFavoriteFilter
 
@@ -101,9 +103,6 @@ class DashboardAccessFilter(BaseFilter):  # pylint: disable=too-few-public-metho
         if security_manager.is_admin():
             return query
 
-        datasource_perms = security_manager.user_view_menu_names("datasource_access")
-        schema_perms = security_manager.user_view_menu_names("schema_access")
-
         is_rbac_disabled_filter = []
         dashboard_has_roles = Dashboard.roles.any()
         if is_feature_enabled("DASHBOARD_RBAC"):
@@ -112,13 +111,14 @@ class DashboardAccessFilter(BaseFilter):  # pylint: disable=too-few-public-metho
         datasource_perm_query = (
             db.session.query(Dashboard.id)
             .join(Dashboard.slices, isouter=True)
+            .join(SqlaTable, Slice.datasource_id == SqlaTable.id)
+            .join(Database, SqlaTable.database_id == Database.id)
             .filter(
                 and_(
                     Dashboard.published.is_(True),
                     *is_rbac_disabled_filter,
-                    or_(
-                        Slice.perm.in_(datasource_perms),
-                        Slice.schema_perm.in_(schema_perms),
+                    get_dataset_access_filters(
+                        Slice,
                         security_manager.can_access_all_datasources(),
                     ),
                 )

superset/db_engine_specs/base.py

@@ -72,7 +72,6 @@ from superset.utils.hashing import md5_sha_from_str
 from superset.utils.network import is_hostname_valid, is_port_open
 
 if TYPE_CHECKING:
-    # prevent circular imports
     from superset.connectors.sqla.models import TableColumn
     from superset.models.core import Database
     from superset.models.sql_lab import Query
@@ -355,6 +354,8 @@ class BaseEngineSpec:  # pylint: disable=too-many-public-methods
     # This set will give the keywords for data limit statements
     # to consider for the engines with TOP SQL parsing
     top_keywords: Set[str] = {"TOP"}
+    # A set of disallowed connection query parameters
+    disallow_uri_query_params: Set[str] = set()
 
     force_column_alias_quotes = False
     arraysize = 0
@@ -1725,6 +1726,19 @@ class BaseEngineSpec:  # pylint: disable=too-many-public-methods
             "disable_ssh_tunneling": cls.disable_ssh_tunneling,
         }
 
+    @classmethod
+    def validate_database_uri(cls, sqlalchemy_uri: URL) -> None:
+        """
+        Validates a database SQLAlchemy URI per engine spec.
+        Use this to implement a final validation for unwanted connection configuration
+
+        :param sqlalchemy_uri:
+        """
+        if existing_disallowed := cls.disallow_uri_query_params.intersection(
+            sqlalchemy_uri.query
+        ):
+            raise ValueError(f"Forbidden query parameter(s): {existing_disallowed}")
+
 
 # schema for adding a database by providing parameters instead of the
 # full SQLAlchemy URI

superset/db_engine_specs/clickhouse.py

@@ -14,29 +14,43 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-import logging
-from datetime import datetime
-from typing import Any, Dict, List, Optional, Type, TYPE_CHECKING
+from __future__ import annotations
 
+import logging
+import re
+from datetime import datetime
+from typing import Any, cast, Dict, List, Optional, Type, TYPE_CHECKING
+
+from flask import current_app
+from flask_babel import gettext as __
+from marshmallow import fields, Schema
+from marshmallow.validate import Range
 from sqlalchemy import types
+from sqlalchemy.engine.url import URL
 from urllib3.exceptions import NewConnectionError
 
-from superset.db_engine_specs.base import BaseEngineSpec
+from superset.databases.utils import make_url_safe
+from superset.db_engine_specs.base import (
+    BaseEngineSpec,
+    BasicParametersMixin,
+    BasicParametersType,
+    BasicPropertiesType,
+)
 from superset.db_engine_specs.exceptions import SupersetDBAPIDatabaseError
+from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
 from superset.extensions import cache_manager
+from superset.utils.core import GenericDataType
+from superset.utils.hashing import md5_sha_from_str
+from superset.utils.network import is_hostname_valid, is_port_open
 
 if TYPE_CHECKING:
-    # prevent circular imports
     from superset.models.core import Database
 
 logger = logging.getLogger(__name__)
 
 
-class ClickHouseEngineSpec(BaseEngineSpec):  # pylint: disable=abstract-method
-    """Dialect for ClickHouse analytical DB."""
-
-    engine = "clickhouse"
-    engine_name = "ClickHouse"
+class ClickHouseBaseEngineSpec(BaseEngineSpec):
+    """Shared engine spec for ClickHouse."""
 
     time_secondary_columns = True
     time_groupby_inline = True
@@ -56,8 +70,78 @@ class ClickHouseEngineSpec(BaseEngineSpec):  # pylint: disable=abstract-method
         "P1Y": "toStartOfYear(toDateTime({col}))",
     }
 
-    _show_functions_column = "name"
+    column_type_mappings = (
+        (
+            re.compile(r".*Enum.*", re.IGNORECASE),
+            types.String(),
+            GenericDataType.STRING,
+        ),
+        (
+            re.compile(r".*Array.*", re.IGNORECASE),
+            types.String(),
+            GenericDataType.STRING,
+        ),
+        (
+            re.compile(r".*UUID.*", re.IGNORECASE),
+            types.String(),
+            GenericDataType.STRING,
+        ),
+        (
+            re.compile(r".*Bool.*", re.IGNORECASE),
+            types.Boolean(),
+            GenericDataType.BOOLEAN,
+        ),
+        (
+            re.compile(r".*String.*", re.IGNORECASE),
+            types.String(),
+            GenericDataType.STRING,
+        ),
+        (
+            re.compile(r".*Int\d+.*", re.IGNORECASE),
+            types.INTEGER(),
+            GenericDataType.NUMERIC,
+        ),
+        (
+            re.compile(r".*Decimal.*", re.IGNORECASE),
+            types.DECIMAL(),
+            GenericDataType.NUMERIC,
+        ),
+        (
+            re.compile(r".*DateTime.*", re.IGNORECASE),
+            types.DateTime(),
+            GenericDataType.TEMPORAL,
+        ),
+        (
+            re.compile(r".*Date.*", re.IGNORECASE),
+            types.Date(),
+            GenericDataType.TEMPORAL,
+        ),
+    )
 
+    @classmethod
+    def epoch_to_dttm(cls) -> str:
+        return "{col}"
+
+    @classmethod
+    def convert_dttm(
+        cls, target_type: str, dttm: datetime, db_extra: Optional[Dict[str, Any]] = None
+    ) -> Optional[str]:
+        sqla_type = cls.get_sqla_column_type(target_type)
+
+        if isinstance(sqla_type, types.Date):
+            return f"toDate('{dttm.date().isoformat()}')"
+        if isinstance(sqla_type, types.DateTime):
+            return f"""toDateTime('{dttm.isoformat(sep=" ", timespec="seconds")}')"""
+        return None
+
+
+class ClickHouseEngineSpec(ClickHouseBaseEngineSpec):
+    """Engine spec for clickhouse_sqlalchemy connector"""
+
+    engine = "clickhouse"
+    engine_name = "ClickHouse"
+
+    _show_functions_column = "name"
     supports_file_upload = False
 
     @classmethod
@@ -73,21 +157,9 @@ class ClickHouseEngineSpec(BaseEngineSpec):  # pylint: disable=abstract-method
             return exception
         return new_exception(str(exception))
 
-    @classmethod
-    def convert_dttm(
-        cls, target_type: str, dttm: datetime, db_extra: Optional[Dict[str, Any]] = None
-    ) -> Optional[str]:
-        sqla_type = cls.get_sqla_column_type(target_type)
-
-        if isinstance(sqla_type, types.Date):
-            return f"toDate('{dttm.date().isoformat()}')"
-        if isinstance(sqla_type, types.DateTime):
-            return f"""toDateTime('{dttm.isoformat(sep=" ", timespec="seconds")}')"""
-        return None
-
     @classmethod
     @cache_manager.cache.memoize()
-    def get_function_names(cls, database: "Database") -> List[str]:
+    def get_function_names(cls, database: Database) -> List[str]:
         """
         Get a list of function names that are able to be called on the database.
         Used for SQL Lab autocomplete.
@@ -123,3 +195,202 @@ class ClickHouseEngineSpec(BaseEngineSpec):  # pylint: disable=abstract-method
 
         # otherwise, return no function names to prevent errors
         return []
+
+
+class ClickHouseParametersSchema(Schema):
+    username = fields.String(allow_none=True, description=__("Username"))
+    password = fields.String(allow_none=True, description=__("Password"))
+    host = fields.String(required=True, description=__("Hostname or IP address"))
+    port = fields.Integer(
+        allow_none=True,
+        description=__("Database port"),
+        validate=Range(min=0, max=65535),
+    )
+    database = fields.String(allow_none=True, description=__("Database name"))
+    encryption = fields.Boolean(
+        default=True, description=__("Use an encrypted connection to the database")
+    )
+    query = fields.Dict(
+        keys=fields.Str(), values=fields.Raw(), description=__("Additional parameters")
+    )
+
+
+try:
+    from clickhouse_connect.common import set_setting
+    from clickhouse_connect.datatypes.format import set_default_formats
+
+    # override default formats for compatibility
+    set_default_formats(
+        "FixedString",
+        "string",
+        "IPv*",
+        "string",
+        "UInt64",
+        "signed",
+        "UUID",
+        "string",
+        "*Int256",
+        "string",
+        "*Int128",
+        "string",
+    )
+    set_setting(
+        "product_name",
+        f"superset/{current_app.config.get('VERSION_STRING', 'dev')}",
+    )
+except ImportError:  # ClickHouse Connect not installed, do nothing
+    pass
+
+
+class ClickHouseConnectEngineSpec(ClickHouseEngineSpec, BasicParametersMixin):
+    """Engine spec for clickhouse-connect connector"""
+
+    engine = "clickhousedb"
+    engine_name = "ClickHouse Connect"
+
+    default_driver = "connect"
+    _function_names: List[str] = []
+
+    sqlalchemy_uri_placeholder = (
+        "clickhousedb://user:password@host[:port][/dbname][?secure=value&=value...]"
+    )
+    parameters_schema = ClickHouseParametersSchema()
+    encryption_parameters = {"secure": "true"}
+
+    @classmethod
+    def get_dbapi_exception_mapping(cls) -> Dict[Type[Exception], Type[Exception]]:
+        return {}
+
+    @classmethod
+    def get_dbapi_mapped_exception(cls, exception: Exception) -> Exception:
+        new_exception = cls.get_dbapi_exception_mapping().get(type(exception))
+        if new_exception == SupersetDBAPIDatabaseError:
+            return SupersetDBAPIDatabaseError("Connection failed")
+        if not new_exception:
+            return exception
+        return new_exception(str(exception))
+
+    @classmethod
+    def get_function_names(cls, database: Database) -> List[str]:
+        # pylint: disable=import-outside-toplevel,import-error
+        from clickhouse_connect.driver.exceptions import ClickHouseError
+
+        if cls._function_names:
+            return cls._function_names
+        try:
+            names = database.get_df(
+                "SELECT name FROM system.functions UNION ALL "
+                + "SELECT name FROM system.table_functions LIMIT 10000"
+            )["name"].tolist()
+            cls._function_names = names
+            return names
+        except ClickHouseError:
+            logger.exception("Error retrieving system.functions")
+            return []
+
+    @classmethod
+    def get_datatype(cls, type_code: str) -> str:
+        # keep it lowercase, as ClickHouse types aren't typical SHOUTCASE ANSI SQL
+        return type_code
+
+    @classmethod
+    def build_sqlalchemy_uri(
+        cls,
+        parameters: BasicParametersType,
+        encrypted_extra: Optional[Dict[str, str]] = None,
+    ) -> str:
+        url_params = parameters.copy()
+        if url_params.get("encryption"):
+            query = parameters.get("query", {}).copy()
+            query.update(cls.encryption_parameters)
+            url_params["query"] = query
+        if not url_params.get("database"):
+            url_params["database"] = "__default__"
+        url_params.pop("encryption", None)
+        return str(URL(f"{cls.engine}+{cls.default_driver}", **url_params))
+
+    @classmethod
+    def get_parameters_from_uri(
+        cls, uri: str, encrypted_extra: Optional[Dict[str, Any]] = None
+    ) -> BasicParametersType:
+        url = make_url_safe(uri)
+        query = url.query
+        if "secure" in query:
+            encryption = url.query.get("secure") == "true"
+            query.pop("secure")
+        else:
+            encryption = False
+        return BasicParametersType(
+            username=url.username,
+            password=url.password,
+            host=url.host,
+            port=url.port,
+            database="" if url.database == "__default__" else cast(str, url.database),
+            query=dict(query),
+            encryption=encryption,
+        )
+
+    @classmethod
+    def validate_parameters(
+        cls, properties: BasicPropertiesType
+    ) -> List[SupersetError]:
+        # pylint: disable=import-outside-toplevel,import-error
+        from clickhouse_connect.driver import default_port
+
+        parameters = properties.get("parameters", {})
+        host = parameters.get("host", None)
+        if not host:
+            return [
+                SupersetError(
+                    "Hostname is required",
+                    SupersetErrorType.CONNECTION_MISSING_PARAMETERS_ERROR,
+                    ErrorLevel.WARNING,
+                    {"missing": ["host"]},
+                )
+            ]
+        if not is_hostname_valid(host):
+            return [
+                SupersetError(
+                    "The hostname provided can't be resolved.",
+                    SupersetErrorType.CONNECTION_INVALID_HOSTNAME_ERROR,
+                    ErrorLevel.ERROR,
+                    {"invalid": ["host"]},
+                )
+            ]
+        port = parameters.get("port")
+        if port is None:
+            port = default_port("http", parameters.get("encryption", False))
+        try:
+            port = int(port)
+        except (ValueError, TypeError):
+            port = -1
+        if port <= 0 or port >= 65535:
+            return [
+                SupersetError(
+                    "Port must be a valid integer between 0 and 65535 (inclusive).",
+                    SupersetErrorType.CONNECTION_INVALID_PORT_ERROR,
+                    ErrorLevel.ERROR,
+                    {"invalid": ["port"]},
+                )
+            ]
+        if not is_port_open(host, port):
+            return [
+                SupersetError(
+                    "The port is closed.",
+                    SupersetErrorType.CONNECTION_PORT_CLOSED_ERROR,
+                    ErrorLevel.ERROR,
+                    {"invalid": ["port"]},
+                )
+            ]
+        return []
+
+    @staticmethod
+    def _mutate_label(label: str) -> str:
+        """
+        Suffix with the first six characters from the md5 of the label to avoid
+        collisions with original column names
+
+        :param label: Expected expression label
+        :return: Conditionally mutated label
+        """
+        return f"{label}_{md5_sha_from_str(label)[:6]}"

superset/db_engine_specs/mysql.py

@@ -173,6 +173,7 @@ class MySQLEngineSpec(BaseEngineSpec, BasicParametersMixin):
             {},
         ),
     }
+    disallow_uri_query_params = {"local_infile"}
 
     @classmethod
     def convert_dttm(

superset/exceptions.py

@@ -270,3 +270,7 @@ class SupersetCancelQueryException(SupersetException):
 
 class QueryNotFoundException(SupersetException):
     status = 404
+
+
+class ColumnNotFoundException(SupersetException):
+    status = 404

superset/extensions/ssh.py

@@ -51,8 +51,7 @@ class SSHManager:
     ) -> SSHTunnelForwarder:
         url = make_url_safe(sqlalchemy_database_uri)
         params = {
-            "ssh_address_or_host": ssh_tunnel.server_address,
-            "ssh_port": ssh_tunnel.server_port,
+            "ssh_address_or_host": (ssh_tunnel.server_address, ssh_tunnel.server_port),
             "ssh_username": ssh_tunnel.username,
             "remote_bind_address": (url.host, url.port),  # bind_port, bind_host
             "local_bind_address": (self.local_bind_address,),
@@ -62,9 +61,10 @@ class SSHManager:
             params["ssh_password"] = ssh_tunnel.password
         elif ssh_tunnel.private_key:
             private_key_file = StringIO(ssh_tunnel.private_key)
-            private_key = RSAKey.from_private_key(private_key_file)
+            private_key = RSAKey.from_private_key(
+                private_key_file, ssh_tunnel.private_key_password
+            )
             params["ssh_pkey"] = private_key
-            params["ssh_private_key_password"] = ssh_tunnel.private_key_password
 
         return open_tunnel(**params)
 

superset/initialization/__init__.py

@@ -18,6 +18,7 @@ from __future__ import annotations
 
 import logging
 import os
+import sys
 from typing import Any, Callable, Dict, TYPE_CHECKING
 
 import wtforms_json
@@ -52,7 +53,7 @@ from superset.extensions import (
 from superset.security import SupersetSecurityManager
 from superset.superset_typing import FlaskResponse
 from superset.tags.core import register_sqla_event_listeners
-from superset.utils.core import pessimistic_connection_handling
+from superset.utils.core import is_test, pessimistic_connection_handling
 from superset.utils.log import DBEventLogger, get_event_logger_from_cfg_value
 
 if TYPE_CHECKING:
@@ -439,7 +440,7 @@ class SupersetAppInitializer:  # pylint: disable=too-many-public-methods
         self.init_views()
 
     def check_secret_key(self) -> None:
-        if self.config["SECRET_KEY"] == CHANGE_ME_SECRET_KEY:
+        def log_default_secret_key_warning() -> None:
             top_banner = 80 * "-" + "\n" + 36 * " " + "WARNING\n" + 80 * "-"
             bottom_banner = 80 * "-" + "\n" + 80 * "-"
             logger.warning(top_banner)
@@ -452,6 +453,19 @@ class SupersetAppInitializer:  # pylint: disable=too-many-public-methods
             )
             logger.warning(bottom_banner)
 
+        if self.config["SECRET_KEY"] == CHANGE_ME_SECRET_KEY:
+            if (
+                self.superset_app.debug
+                or self.superset_app.config["TESTING"]
+                or is_test()
+            ):
+                logger.warning("Debug mode identified with default secret key")
+                log_default_secret_key_warning()
+                return
+            log_default_secret_key_warning()
+            logger.error("Refusing to start due to insecure SECRET_KEY")
+            sys.exit(1)
+
     def init_app(self) -> None:
         """
         Main entry point which will delegate to other methods in

superset/jinja_context.py

@@ -17,7 +17,7 @@
 """Defines the templating context for SQL Lab"""
 import json
 import re
-from functools import partial
+from functools import lru_cache, partial
 from typing import (
     Any,
     Callable,
@@ -38,6 +38,7 @@ from sqlalchemy.engine.interfaces import Dialect
 from sqlalchemy.types import String
 from typing_extensions import TypedDict
 
+from superset.constants import LRU_CACHE_MAX_SIZE
 from superset.datasets.commands.exceptions import DatasetNotFoundError
 from superset.exceptions import SupersetTemplateException
 from superset.extensions import feature_flag_manager
@@ -46,7 +47,6 @@ from superset.utils.core import (
     get_user_id,
     merge_extra_filters,
 )
-from superset.utils.memoized import memoized
 
 if TYPE_CHECKING:
     from superset.connectors.sqla.models import SqlaTable
@@ -70,7 +70,7 @@ ALLOWED_TYPES = (
 COLLECTION_TYPES = ("list", "dict", "tuple", "set")
 
 
-@memoized
+@lru_cache(maxsize=LRU_CACHE_MAX_SIZE)
 def context_addons() -> Dict[str, Any]:
     return current_app.config.get("JINJA_CONTEXT_ADDONS", {})
 
@@ -602,7 +602,7 @@ DEFAULT_PROCESSORS = {
 }
 
 
-@memoized
+@lru_cache(maxsize=LRU_CACHE_MAX_SIZE)
 def get_template_processors() -> Dict[str, Any]:
     processors = current_app.config.get("CUSTOM_TEMPLATE_PROCESSORS", {})
     for engine, processor in DEFAULT_PROCESSORS.items():

superset/migrations/versions/2020-04-29_09-24_620241d1153f_update_time_grain_sqla.py

@@ -34,7 +34,6 @@ from sqlalchemy.ext.declarative import declarative_base
 
 from superset import db, db_engine_specs
 from superset.databases.utils import make_url_safe
-from superset.utils.memoized import memoized
 
 Base = declarative_base()
 
@@ -70,7 +69,6 @@ class Slice(Base):
     datasource_id = Column(Integer)
 
 
-@memoized
 def duration_by_name(database: Database):
     return {grain.name: grain.duration for grain in database.grains()}
 

superset/models/core.py

@@ -24,6 +24,7 @@ from ast import literal_eval
 from contextlib import closing, contextmanager, nullcontext
 from copy import deepcopy
 from datetime import datetime
+from functools import lru_cache
 from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Type, TYPE_CHECKING
 
 import numpy
@@ -54,7 +55,7 @@ from sqlalchemy.schema import UniqueConstraint
 from sqlalchemy.sql import expression, Select
 
 from superset import app, db_engine_specs
-from superset.constants import PASSWORD_MASK
+from superset.constants import LRU_CACHE_MAX_SIZE, PASSWORD_MASK
 from superset.databases.utils import make_url_safe
 from superset.db_engine_specs.base import MetricType, TimeGrain
 from superset.extensions import (
@@ -67,7 +68,6 @@ from superset.models.helpers import AuditMixinNullable, ImportExportMixin
 from superset.result_set import SupersetResultSet
 from superset.utils import cache as cache_util, core as utils
 from superset.utils.core import get_username
-from superset.utils.memoized import memoized
 
 config = app.config
 custom_password_store = config["SQLALCHEMY_CUSTOM_PASSWORD_STORE"]
@@ -424,6 +424,8 @@ class Database(
         sqlalchemy_url = make_url_safe(
             sqlalchemy_uri if sqlalchemy_uri else self.sqlalchemy_uri_decrypted
         )
+        self.db_engine_spec.validate_database_uri(sqlalchemy_url)
+
         sqlalchemy_url = self.db_engine_spec.adjust_database_uri(sqlalchemy_url, schema)
         effective_username = self.get_effective_user(sqlalchemy_url)
         # If using MySQL or Presto for example, will set url.username
@@ -723,7 +725,7 @@ class Database(
         return self.get_db_engine_spec(url)
 
     @classmethod
-    @memoized
+    @lru_cache(maxsize=LRU_CACHE_MAX_SIZE)
     def get_db_engine_spec(cls, url: URL) -> Type[db_engine_specs.BaseEngineSpec]:
         backend = url.get_backend_name()
         try:
@@ -897,7 +899,6 @@ class Database(
     def has_view_by_name(self, view_name: str, schema: Optional[str] = None) -> bool:
         return self.has_view(view_name=view_name, schema=schema)
 
-    @memoized
     def get_dialect(self) -> Dialect:
         sqla_url = make_url_safe(self.sqlalchemy_uri_decrypted)
         return sqla_url.get_dialect()()

superset/models/datasource_access_request.py

@@ -22,7 +22,6 @@ from sqlalchemy import Column, Integer, String
 
 from superset import app, db, security_manager
 from superset.models.helpers import AuditMixinNullable
-from superset.utils.memoized import memoized
 
 if TYPE_CHECKING:
     from superset.connectors.base.models import BaseDatasource
@@ -57,7 +56,6 @@ class DatasourceAccessRequest(Model, AuditMixinNullable):
         return self.get_datasource
 
     @datasource.getter  # type: ignore
-    @memoized
     def get_datasource(self) -> "BaseDatasource":
         ds = db.session.query(self.cls_model).filter_by(id=self.datasource_id).first()
         return ds

superset/models/helpers.py

@@ -80,6 +80,7 @@ from superset.jinja_context import BaseTemplateProcessor
 from superset.sql_parse import has_table_query, insert_rls, ParsedQuery, sanitize_clause
 from superset.superset_typing import (
     AdhocMetric,
+    Column as ColumnTyping,
     FilterValue,
     FilterValues,
     Metric,
@@ -545,6 +546,8 @@ class QueryResult:  # pylint: disable=too-few-public-methods
         query: str,
         duration: timedelta,
         applied_template_filters: Optional[List[str]] = None,
+        applied_filter_columns: Optional[List[ColumnTyping]] = None,
+        rejected_filter_columns: Optional[List[ColumnTyping]] = None,
         status: str = QueryStatus.SUCCESS,
         error_message: Optional[str] = None,
         errors: Optional[List[Dict[str, Any]]] = None,
@@ -555,6 +558,8 @@ class QueryResult:  # pylint: disable=too-few-public-methods
         self.query = query
         self.duration = duration
         self.applied_template_filters = applied_template_filters or []
+        self.applied_filter_columns = applied_filter_columns or []
+        self.rejected_filter_columns = rejected_filter_columns or []
         self.status = status
         self.error_message = error_message
         self.errors = errors or []
@@ -1646,7 +1651,7 @@ class ExploreMixin:  # pylint: disable=too-many-public-methods
             elif utils.is_adhoc_column(flt_col):
                 sqla_col = self.adhoc_column_to_sqla(flt_col)  # type: ignore
             else:
-                col_obj = columns_by_name.get(flt_col)
+                col_obj = columns_by_name.get(cast(str, flt_col))
             filter_grain = flt.get("grain")
 
             if is_feature_enabled("ENABLE_TEMPLATE_REMOVE_FILTERS"):

superset/models/slice.py

@@ -46,7 +46,6 @@ from superset.tasks.thumbnails import cache_chart_thumbnail
 from superset.tasks.utils import get_current_user
 from superset.thumbnails.digest import get_chart_digest
 from superset.utils import core as utils
-from superset.utils.memoized import memoized
 from superset.viz import BaseViz, viz_types
 
 if TYPE_CHECKING:
@@ -151,9 +150,12 @@ class Slice(  # pylint: disable=too-many-public-methods
 
     # pylint: disable=using-constant-test
     @datasource.getter  # type: ignore
-    @memoized
     def get_datasource(self) -> Optional["BaseDatasource"]:
-        return db.session.query(self.cls_model).filter_by(id=self.datasource_id).first()
+        return (
+            db.session.query(self.cls_model)
+            .filter_by(id=self.datasource_id)
+            .one_or_none()
+        )
 
     @renders("datasource_name")
     def datasource_link(self) -> Optional[Markup]:
@@ -189,8 +191,7 @@ class Slice(  # pylint: disable=too-many-public-methods
 
     # pylint: enable=using-constant-test
 
-    @property  # type: ignore
-    @memoized
+    @property
     def viz(self) -> Optional[BaseViz]:
         form_data = json.loads(self.params)
         viz_class = viz_types.get(self.viz_type)

superset/queries/api.py

@@ -137,6 +137,7 @@ class QueryRestApi(BaseSupersetModelRestApi):
     base_related_field_filters = {
         "created_by": [["id", BaseFilterRelatedUsers, lambda: []]],
         "user": [["id", BaseFilterRelatedUsers, lambda: []]],
+        "database": [["id", DatabaseFilter, lambda: []]],
     }
     related_field_filters = {
         "created_by": RelatedFieldFilter("first_name", FilterRelatedOwners),
@@ -145,7 +146,6 @@ class QueryRestApi(BaseSupersetModelRestApi):
 
     search_columns = ["changed_on", "database", "sql", "status", "user", "start_time"]
 
-    base_related_field_filters = {"database": [["id", DatabaseFilter, lambda: []]]}
     allowed_rel_fields = {"database", "user"}
     allowed_distinct_fields = {"status"}
 

superset/security/manager.py

@@ -84,6 +84,7 @@ from superset.utils.core import (
     get_user_id,
     RowLevelSecurityFilterType,
 )
+from superset.utils.filters import get_dataset_access_filters
 from superset.utils.urls import get_url_host
 
 if TYPE_CHECKING:
@@ -98,6 +99,8 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
+DATABASE_PERM_REGEX = re.compile(r"^\[.+\]\.\(id\:(?P<id>\d+)\)$")
+
 
 class DatabaseAndSchema(NamedTuple):
     database: str
@@ -154,40 +157,53 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
     READ_ONLY_MODEL_VIEWS = {"Database", "DruidClusterModelView", "DynamicPlugin"}
 
     USER_MODEL_VIEWS = {
+        "RegisterUserModelView",
         "UserDBModelView",
         "UserLDAPModelView",
+        "UserInfoEditView",
         "UserOAuthModelView",
         "UserOIDModelView",
         "UserRemoteUserModelView",
     }
 
     GAMMA_READ_ONLY_MODEL_VIEWS = {
+        "Annotation",
+        "CssTemplate",
         "Dataset",
         "Datasource",
-        "CssTemplate",
     } | READ_ONLY_MODEL_VIEWS
 
     ADMIN_ONLY_VIEW_MENUS = {
+        "Access Requests",
         "AccessRequestsModelView",
-        "SQL Lab",
+        "Action Log",
+        "Log",
+        "List Users",
+        "List Roles",
         "Refresh Druid Metadata",
         "ResetPasswordView",
         "RoleModelView",
-        "Log",
-        "Security",
         "Row Level Security",
         "Row Level Security Filters",
         "RowLevelSecurityFiltersModelView",
+        "Security",
+        "SQL Lab",
     } | USER_MODEL_VIEWS
 
     ALPHA_ONLY_VIEW_MENUS = {
         "Manage",
         "CSS Templates",
+        "Annotation Layers",
         "Queries",
         "Import dashboards",
         "Upload a CSV",
         "ReportSchedule",
         "Alerts & Report",
+        "TableSchemaView",
+        "CsvToDatabaseView",
+        "ColumnarToDatabaseView",
+        "ExcelToDatabaseView",
+        "ImportExportRestApi",
     }
 
     ADMIN_ONLY_PERMISSIONS = {
@@ -199,6 +215,7 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
         "all_query_access",
         "can_grant_guest_token",
         "can_set_embedded",
+        "can_warm_up_cache",
     }
 
     READ_ONLY_PERMISSION = {
@@ -222,16 +239,33 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
         "datasource_access",
     }
 
-    ACCESSIBLE_PERMS = {"can_userinfo", "resetmypassword"}
+    ACCESSIBLE_PERMS = {"can_userinfo", "resetmypassword", "can_recent_activity"}
 
     SQLLAB_ONLY_PERMISSIONS = {
         ("can_my_queries", "SqlLab"),
         ("can_read", "SavedQuery"),
-        ("can_sql_json", "Superset"),
+        ("can_write", "SavedQuery"),
+        ("can_export", "SavedQuery"),
+        ("can_read", "Query"),
+        ("can_export_csv", "Query"),
+        ("can_get_results", "SQLLab"),
+        ("can_execute_sql_query", "SQLLab"),
+        ("can_export_csv", "SQLLab"),
+        ("can_sql_json", "Superset"),  # Deprecated permission remove on 3.0.0
         ("can_sqllab_history", "Superset"),
         ("can_sqllab_viz", "Superset"),
-        ("can_sqllab_table_viz", "Superset"),
+        ("can_sqllab_table_viz", "Superset"),  # Deprecated permission remove on 3.0.0
         ("can_sqllab", "Superset"),
+        ("can_stop_query", "Superset"),  # Deprecated permission remove on 3.0.0
+        ("can_test_conn", "Superset"),  # Deprecated permission remove on 3.0.0
+        ("can_search_queries", "Superset"),  # Deprecated permission remove on 3.0.0
+        ("can_activate", "TabStateView"),
+        ("can_get", "TabStateView"),
+        ("can_delete_query", "TabStateView"),
+        ("can_post", "TabStateView"),
+        ("can_delete", "TabStateView"),
+        ("can_put", "TabStateView"),
+        ("can_migrate_query", "TabStateView"),
         ("menu_access", "SQL Lab"),
         ("menu_access", "SQL Editor"),
         ("menu_access", "Saved Queries"),
@@ -239,7 +273,7 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
     }
 
     SQLLAB_EXTRA_PERMISSION_VIEWS = {
-        ("can_csv", "Superset"),
+        ("can_csv", "Superset"),  # Deprecated permission remove on 3.0.0
         ("can_read", "Superset"),
         ("can_read", "Database"),
     }
@@ -494,8 +528,6 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
         :returns: The list of datasources
         """
 
-        user_perms = self.user_view_menu_names("datasource_access")
-        schema_perms = self.user_view_menu_names("schema_access")
         user_datasources = set()
 
         # pylint: disable=import-outside-toplevel
@@ -503,12 +535,7 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
 
         user_datasources.update(
             self.get_session.query(SqlaTable)
-            .filter(
-                or_(
-                    SqlaTable.perm.in_(user_perms),
-                    SqlaTable.schema_perm.in_(schema_perms),
-                )
-            )
+            .filter(get_dataset_access_filters(SqlaTable))
             .all()
         )
 
@@ -573,6 +600,19 @@ class SupersetSecurityManager(  # pylint: disable=too-many-public-methods
             return {s.name for s in view_menu_names}
         return set()
 
+    def get_accessible_databases(self) -> List[int]:
+        """
+        Return the list of databases accessible by the user.
+
+        :return: The list of accessible Databases
+        """
+        perms = self.user_view_menu_names("database_access")
+        return [
+            int(match.group("id"))
+            for perm in perms
+            if (match := DATABASE_PERM_REGEX.match(perm))
+        ]
+
     def get_schemas_accessible_by_user(
         self, database: "Database", schemas: List[str], hierarchical: bool = True
     ) -> List[str]:

superset/sqllab/api.py

@@ -68,7 +68,7 @@ class SqlLabRestApi(BaseSupersetApi):
     resource_name = "sqllab"
     allow_browser_login = True
 
-    class_permission_name = "Query"
+    class_permission_name = "SQLLab"
 
     execute_model_schema = ExecutePayloadSchema()
 

superset/utils/core.py

@@ -221,7 +221,7 @@ class AdhocFilterClause(TypedDict, total=False):
 
 
 class QueryObjectFilterClause(TypedDict, total=False):
-    col: str
+    col: Column
     op: str  # pylint: disable=invalid-name
     val: Optional[FilterValues]
     grain: Optional[str]
@@ -1089,7 +1089,7 @@ def simple_filter_to_adhoc(
         "expressionType": "SIMPLE",
         "comparator": filter_clause.get("val"),
         "operator": filter_clause["op"],
-        "subject": filter_clause["col"],
+        "subject": cast(str, filter_clause["col"]),
     }
     if filter_clause.get("isExtra"):
         result["isExtra"] = True

superset/utils/date_parser.py

@@ -18,6 +18,7 @@ import calendar
 import logging
 import re
 from datetime import datetime, timedelta
+from functools import lru_cache
 from time import struct_time
 from typing import Dict, List, Optional, Tuple
 
@@ -45,8 +46,7 @@ from superset.charts.commands.exceptions import (
     TimeRangeAmbiguousError,
     TimeRangeParseFailError,
 )
-from superset.constants import NO_TIME_RANGE
-from superset.utils.memoized import memoized
+from superset.constants import LRU_CACHE_MAX_SIZE, NO_TIME_RANGE
 
 ParserElement.enablePackrat()
 
@@ -394,7 +394,7 @@ class EvalHolidayFunc:  # pylint: disable=too-few-public-methods
         )
 
 
-@memoized
+@lru_cache(maxsize=LRU_CACHE_MAX_SIZE)
 def datetime_parser() -> ParseResults:  # pylint: disable=too-many-locals
     (  # pylint: disable=invalid-name
         DATETIME,

superset/utils/filters.py

@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Type
+
+from flask_appbuilder import Model
+from sqlalchemy import or_
+from sqlalchemy.sql.elements import BooleanClauseList
+
+
+def get_dataset_access_filters(
+    base_model: Type[Model],
+    *args: Any,
+) -> BooleanClauseList:
+    # pylint: disable=import-outside-toplevel
+    from superset import security_manager
+    from superset.connectors.sqla.models import Database
+
+    database_ids = security_manager.get_accessible_databases()
+    perms = security_manager.user_view_menu_names("datasource_access")
+    schema_perms = security_manager.user_view_menu_names("schema_access")
+
+    return or_(
+        Database.id.in_(database_ids),
+        base_model.perm.in_(perms),
+        base_model.schema_perm.in_(schema_perms),
+        *args,
+    )

superset/utils/memoized.py

@@ -1,81 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-import functools
-from typing import Any, Callable, Dict, Optional, Tuple, Type
-
-
-class _memoized:
-    """Decorator that caches a function's return value each time it is called
-
-    If called later with the same arguments, the cached value is returned, and
-    not re-evaluated.
-
-    Define ``watch`` as a tuple of attribute names if this Decorator
-    should account for instance variable changes.
-    """
-
-    def __init__(
-        self, func: Callable[..., Any], watch: Optional[Tuple[str, ...]] = None
-    ) -> None:
-        self.func = func
-        self.cache: Dict[Any, Any] = {}
-        self.is_method = False
-        self.watch = watch or ()
-
-    def __call__(self, *args: Any, **kwargs: Any) -> Any:
-        key = [args, frozenset(kwargs.items())]
-        if self.is_method:
-            key.append(tuple(getattr(args[0], v, None) for v in self.watch))
-        key = tuple(key)  # type: ignore
-        try:
-            if key in self.cache:
-                return self.cache[key]
-        except TypeError as ex:
-            # Uncachable -- for instance, passing a list as an argument.
-            raise TypeError("Function cannot be memoized") from ex
-        value = self.func(*args, **kwargs)
-        try:
-            self.cache[key] = value
-        except TypeError as ex:
-            raise TypeError("Function cannot be memoized") from ex
-        return value
-
-    def __repr__(self) -> str:
-        """Return the function's docstring."""
-        return self.func.__doc__ or ""
-
-    def __get__(
-        self, obj: Any, objtype: Type[Any]
-    ) -> functools.partial:  # type: ignore
-        if not self.is_method:
-            self.is_method = True
-        # Support instance methods.
-        func = functools.partial(self.__call__, obj)
-        func.__func__ = self.func  # type: ignore
-        return func
-
-
-def memoized(
-    func: Optional[Callable[..., Any]] = None, watch: Optional[Tuple[str, ...]] = None
-) -> Callable[..., Any]:
-    if func:
-        return _memoized(func)
-
-    def wrapper(f: Callable[..., Any]) -> Callable[..., Any]:
-        return _memoized(f, watch)
-
-    return wrapper

superset/views/base.py

@@ -46,7 +46,7 @@ from flask_jwt_extended.exceptions import NoAuthorizationError
 from flask_wtf.csrf import CSRFError
 from flask_wtf.form import FlaskForm
 from pkg_resources import resource_filename
-from sqlalchemy import exc, or_
+from sqlalchemy import exc
 from sqlalchemy.orm import Query
 from werkzeug.exceptions import HTTPException
 from wtforms import Form
@@ -78,7 +78,7 @@ from superset.reports.models import ReportRecipientType
 from superset.superset_typing import FlaskResponse
 from superset.translations.utils import get_language_pack
 from superset.utils import core as utils
-from superset.utils.core import get_user_id
+from superset.utils.filters import get_dataset_access_filters
 
 from .utils import bootstrap_user_data
 
@@ -188,6 +188,7 @@ def generate_download_headers(
 
 def deprecated(
     eol_version: str = "3.0.0",
+    new_target: Optional[str] = None,
 ) -> Callable[[Callable[..., FlaskResponse]], Callable[..., FlaskResponse]]:
     """
     A decorator to set an API endpoint from SupersetView has deprecated.
@@ -196,13 +197,19 @@ def deprecated(
 
     def _deprecated(f: Callable[..., FlaskResponse]) -> Callable[..., FlaskResponse]:
         def wraps(self: "BaseSupersetView", *args: Any, **kwargs: Any) -> FlaskResponse:
-            logger.warning(
+            messsage = (
                 "%s.%s "
-                "This API endpoint is deprecated and will be removed in version %s",
+                "This API endpoint is deprecated and will be removed in version %s"
+            )
+            logger_args = [
                 self.__class__.__name__,
                 f.__name__,
                 eol_version,
-            )
+            ]
+            if new_target:
+                messsage += " . Use the following API endpoint instead: %s"
+                logger_args.append(new_target)
+            logger.warning(messsage, *logger_args)
             return f(self, *args, **kwargs)
 
         return functools.update_wrapper(wraps, f)
@@ -670,20 +677,11 @@ class DatasourceFilter(BaseFilter):  # pylint: disable=too-few-public-methods
     def apply(self, query: Query, value: Any) -> Query:
         if security_manager.can_access_all_datasources():
             return query
-        datasource_perms = security_manager.user_view_menu_names("datasource_access")
-        schema_perms = security_manager.user_view_menu_names("schema_access")
-        owner_ids_query = (
-            db.session.query(models.SqlaTable.id)
-            .join(models.SqlaTable.owners)
-            .filter(security_manager.user_model.id == get_user_id())
-        )
-        return query.filter(
-            or_(
-                self.model.perm.in_(datasource_perms),
-                self.model.schema_perm.in_(schema_perms),
-                models.SqlaTable.id.in_(owner_ids_query),
-            )
+        query = query.join(
+            models.Database,
+            models.Database.id == self.model.database_id,
         )
+        return query.filter(get_dataset_access_filters(self.model))
 
 
 class CsvResponse(Response):

superset/views/chart/filters.py

@@ -16,10 +16,10 @@
 # under the License.
 from typing import Any
 
-from sqlalchemy import or_
 from sqlalchemy.orm.query import Query
 
 from superset import security_manager
+from superset.utils.filters import get_dataset_access_filters
 from superset.views.base import BaseFilter
 
 
@@ -27,8 +27,5 @@ class SliceFilter(BaseFilter):  # pylint: disable=too-few-public-methods
     def apply(self, query: Query, value: Any) -> Query:
         if security_manager.can_access_all_datasources():
             return query
-        perms = security_manager.user_view_menu_names("datasource_access")
-        schema_perms = security_manager.user_view_menu_names("schema_access")
-        return query.filter(
-            or_(self.model.perm.in_(perms), self.model.schema_perm.in_(schema_perms))
-        )
+
+        return query.filter(get_dataset_access_filters(self.model))

superset/views/core.py

@@ -211,7 +211,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/datasources/")
-    @deprecated()
+    @deprecated(new_target="api/v1/dataset/")
     def datasources(self) -> FlaskResponse:
         return self.json_response(
             sorted(
@@ -505,7 +505,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @expose("/slice_json/<int:slice_id>")
     @etag_cache()
     @check_resource_permissions(check_slice_perms)
-    @deprecated()
+    @deprecated(new_target="/api/v1/chart/<int:id>/data/")
     def slice_json(self, slice_id: int) -> FlaskResponse:
         form_data, slc = get_form_data(slice_id, use_slice_data=True)
         if not slc:
@@ -529,7 +529,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/annotation_json/<int:layer_id>")
-    @deprecated()
+    @deprecated(new_target="/api/v1/chart/<int:id>/data/")
     def annotation_json(  # pylint: disable=no-self-use
         self, layer_id: int
     ) -> FlaskResponse:
@@ -999,7 +999,10 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/filter/<datasource_type>/<int:datasource_id>/<column>/")
-    @deprecated()
+    @deprecated(
+        new_target="/api/v1/datasource/<datasource_type>/"
+        "<datasource_id>/column/<column_name>/values/"
+    )
     def filter(  # pylint: disable=no-self-use
         self, datasource_type: str, datasource_id: int, column: str
     ) -> FlaskResponse:
@@ -1143,7 +1146,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @event_logger.log_this
     @expose("/tables/<int:db_id>/<schema>/")
     @expose("/tables/<int:db_id>/<schema>/<force_refresh>/")
-    @deprecated()
+    @deprecated(new_target="api/v1/database/<int:pk>/tables/")
     def tables(  # pylint: disable=no-self-use
         self,
         db_id: int,
@@ -1352,7 +1355,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/testconn", methods=["POST", "GET"])
-    @deprecated()
+    @deprecated(new_target="/api/v1/database/test_connection/")
     def testconn(self) -> FlaskResponse:  # pylint: disable=no-self-use
         """Tests a sqla connection"""
         db_name = request.json.get("name")
@@ -1441,7 +1444,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/recent_activity/<int:user_id>/", methods=["GET"])
-    @deprecated()
+    @deprecated(new_target="/api/v1/log/recent_activity/<user_id>/")
     def recent_activity(self, user_id: int) -> FlaskResponse:
         """Recent activity (actions) for a given user"""
         error_obj = self.get_user_activity_access_error(user_id)
@@ -1462,7 +1465,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/available_domains/", methods=["GET"])
-    @deprecated()
+    @deprecated(new_target="/api/v1/available_domains/")
     def available_domains(self) -> FlaskResponse:  # pylint: disable=no-self-use
         """
         Returns the list of available Superset Webserver domains (if any)
@@ -1477,7 +1480,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/fave_dashboards_by_username/<username>/", methods=["GET"])
-    @deprecated()
+    @deprecated(new_target="api/v1/dashboard/favorite_status/")
     def fave_dashboards_by_username(self, username: str) -> FlaskResponse:
         """This lets us use a user's username to pull favourite dashboards"""
         user = security_manager.find_user(username=username)
@@ -1487,7 +1490,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/fave_dashboards/<int:user_id>/", methods=["GET"])
-    @deprecated()
+    @deprecated(new_target="api/v1/dashboard/favorite_status/")
     def fave_dashboards(self, user_id: int) -> FlaskResponse:
         error_obj = self.get_user_activity_access_error(user_id)
         if error_obj:
@@ -1524,7 +1527,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/created_dashboards/<int:user_id>/", methods=["GET"])
-    @deprecated()
+    @deprecated(new_target="api/v1/dashboard/")
     def created_dashboards(self, user_id: int) -> FlaskResponse:
         error_obj = self.get_user_activity_access_error(user_id)
         if error_obj:
@@ -1609,7 +1612,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @event_logger.log_this
     @expose("/created_slices", methods=["GET"])
     @expose("/created_slices/<int:user_id>/", methods=["GET"])
-    @deprecated()
+    @deprecated(new_target="api/v1/chart/")
     def created_slices(self, user_id: Optional[int] = None) -> FlaskResponse:
         """List of slices created by this user"""
         if not user_id:
@@ -1926,7 +1929,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access
     @expose("/get_or_create_table/", methods=["POST"])
     @event_logger.log_this
-    @deprecated()
+    @deprecated(new_target="api/v1/dataset/get_or_create/")
     def sqllab_table_viz(self) -> FlaskResponse:  # pylint: disable=no-self-use
         """Gets or creates a table object with attributes passed to the API.
 
@@ -2041,7 +2044,9 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access
     @expose("/extra_table_metadata/<int:database_id>/<table_name>/<schema>/")
     @event_logger.log_this
-    @deprecated()
+    @deprecated(
+        new_target="api/v1/database/<int:pk>/table_extra/<table_name>/<schema_name>/"
+    )
     def extra_table_metadata(  # pylint: disable=no-self-use
         self, database_id: int, table_name: str, schema: str
     ) -> FlaskResponse:
@@ -2099,7 +2104,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @expose("/results/<key>/")
     @event_logger.log_this
-    @deprecated()
+    @deprecated(new_target="/api/v1/sqllab/results/")
     def results(self, key: str) -> FlaskResponse:
         return self.results_exec(key)
 
@@ -2221,7 +2226,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
         on_giveup=lambda details: db.session.rollback(),
         max_tries=5,
     )
-    @deprecated()
+    @deprecated(new_target="/api/v1/query/stop")
     def stop_query(self) -> FlaskResponse:
         client_id = request.form.get("client_id")
         query = db.session.query(Query).filter_by(client_id=client_id).one()
@@ -2250,7 +2255,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access_api
     @event_logger.log_this
     @expose("/validate_sql_json/", methods=["POST", "GET"])
-    @deprecated()
+    @deprecated(new_target="/api/v1/database/<pk>/validate_sql/")
     def validate_sql_json(
         # pylint: disable=too-many-locals,no-self-use
         self,
@@ -2323,7 +2328,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @handle_api_exception
     @event_logger.log_this
     @expose("/sql_json/", methods=["POST"])
-    @deprecated()
+    @deprecated(new_target="/api/v1/sqllab/execute/")
     def sql_json(self) -> FlaskResponse:
         errors = SqlJsonPayloadSchema().validate(request.json)
         if errors:
@@ -2401,7 +2406,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access
     @event_logger.log_this
     @expose("/csv/<client_id>")
-    @deprecated()
+    @deprecated(new_target="/api/v1/sqllab/export/")
     def csv(self, client_id: str) -> FlaskResponse:  # pylint: disable=no-self-use
         """Download the query results as csv."""
         logger.info("Exporting CSV file [%s]", client_id)
@@ -2475,7 +2480,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access
     @event_logger.log_this
     @expose("/fetch_datasource_metadata")
-    @deprecated()
+    @deprecated(new_target="api/v1/database/<int:pk>/table/<table_name>/<schema_name>/")
     def fetch_datasource_metadata(self) -> FlaskResponse:  # pylint: disable=no-self-use
         """
         Fetch the datasource metadata.
@@ -2498,7 +2503,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @event_logger.log_this
     @expose("/queries/<float:last_updated_ms>")
     @expose("/queries/<int:last_updated_ms>")
-    @deprecated()
+    @deprecated(new_target="api/v1/query/updated_since")
     def queries(self, last_updated_ms: Union[float, int]) -> FlaskResponse:
         """
         Get the updated queries.
@@ -2530,7 +2535,7 @@ class Superset(BaseSupersetView):  # pylint: disable=too-many-public-methods
     @has_access
     @event_logger.log_this
     @expose("/search_queries")
-    @deprecated()
+    @deprecated(new_target="api/v1/query/")
     def search_queries(self) -> FlaskResponse:  # pylint: disable=no-self-use
         """
         Search for previously run sqllab queries. Used for Sqllab Query Search

superset/viz.py

@@ -154,7 +154,8 @@ class BaseViz:  # pylint: disable=too-many-public-methods
         self.status: Optional[str] = None
         self.error_msg = ""
         self.results: Optional[QueryResult] = None
-        self.applied_template_filters: List[str] = []
+        self.applied_filter_columns: List[Column] = []
+        self.rejected_filter_columns: List[Column] = []
         self.errors: List[Dict[str, Any]] = []
         self.force = force
         self._force_cached = force_cached
@@ -288,7 +289,8 @@ class BaseViz:  # pylint: disable=too-many-public-methods
 
         # The datasource here can be different backend but the interface is common
         self.results = self.datasource.query(query_obj)
-        self.applied_template_filters = self.results.applied_template_filters or []
+        self.applied_filter_columns = self.results.applied_filter_columns or []
+        self.rejected_filter_columns = self.results.rejected_filter_columns or []
         self.query = self.results.query
         self.status = self.results.status
         self.errors = self.results.errors
@@ -492,25 +494,21 @@ class BaseViz:  # pylint: disable=too-many-public-methods
         if "df" in payload:
             del payload["df"]
 
-        filters = self.form_data.get("filters", [])
-        filter_columns = [flt.get("col") for flt in filters]
-        columns = set(self.datasource.column_names)
-        applied_template_filters = self.applied_template_filters or []
+        applied_filter_columns = self.applied_filter_columns or []
+        rejected_filter_columns = self.rejected_filter_columns or []
         applied_time_extras = self.form_data.get("applied_time_extras", {})
         applied_time_columns, rejected_time_columns = utils.get_time_filter_status(
             self.datasource, applied_time_extras
         )
         payload["applied_filters"] = [
-            {"column": get_column_name(col)}
-            for col in filter_columns
-            if is_adhoc_column(col) or col in columns or col in applied_template_filters
+            {"column": get_column_name(col)} for col in applied_filter_columns
         ] + applied_time_columns
         payload["rejected_filters"] = [
-            {"reason": ExtraFiltersReasonType.COL_NOT_IN_DATASOURCE, "column": col}
-            for col in filter_columns
-            if not is_adhoc_column(col)
-            and col not in columns
-            and col not in applied_template_filters
+            {
+                "reason": ExtraFiltersReasonType.COL_NOT_IN_DATASOURCE,
+                "column": get_column_name(col),
+            }
+            for col in rejected_filter_columns
         ] + rejected_time_columns
         if df is not None:
             payload["colnames"] = list(df.columns)
@@ -535,8 +533,11 @@ class BaseViz:  # pylint: disable=too-many-public-methods
                 try:
                     df = cache_value["df"]
                     self.query = cache_value["query"]
-                    self.applied_template_filters = cache_value.get(
-                        "applied_template_filters", []
+                    self.applied_filter_columns = cache_value.get(
+                        "applied_filter_columns", []
+                    )
+                    self.rejected_filter_columns = cache_value.get(
+                        "rejected_filter_columns", []
                     )
                     self.status = QueryStatus.SUCCESS
                     is_loaded = True

tests/integration_tests/charts/api_tests.py

@@ -609,17 +609,21 @@ class TestChartApi(SupersetTestCase, ApiOwnersTestCaseMixin, InsertChartMixin):
         """
         Chart API: Test update set new owner implicitly adds logged in owner
         """
-        gamma = self.get_user("gamma")
+        gamma = self.get_user("gamma_no_csv")
         alpha = self.get_user("alpha")
-        chart_id = self.insert_chart("title", [alpha.id], 1).id
-        chart_data = {"slice_name": "title1_changed", "owners": [gamma.id]}
-        self.login(username="alpha")
+        chart_id = self.insert_chart("title", [gamma.id], 1).id
+        chart_data = {
+            "slice_name": (new_name := "title1_changed"),
+            "owners": [alpha.id],
+        }
+        self.login(username=gamma.username)
         uri = f"api/v1/chart/{chart_id}"
         rv = self.put_assert_metric(uri, chart_data, "put")
-        self.assertEqual(rv.status_code, 200)
+        assert rv.status_code == 200
         model = db.session.query(Slice).get(chart_id)
-        self.assertIn(alpha, model.owners)
-        self.assertIn(gamma, model.owners)
+        assert model.slice_name == new_name
+        assert alpha in model.owners
+        assert gamma in model.owners
         db.session.delete(model)
         db.session.commit()
 

tests/integration_tests/charts/data/api_tests.py

@@ -56,6 +56,7 @@ from superset.utils.core import (
     AnnotationType,
     get_example_default_schema,
     AdhocMetricExpressionType,
+    ExtraFiltersReasonType,
 )
 from superset.utils.database import get_example_database, get_main_database
 from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
@@ -73,6 +74,12 @@ ADHOC_COLUMN_FIXTURE: AdhocColumn = {
     "when gender = 'girl' then 'female' else 'other' end",
 }
 
+INCOMPATIBLE_ADHOC_COLUMN_FIXTURE: AdhocColumn = {
+    "hasCustomLabel": True,
+    "label": "exciting_or_boring",
+    "sqlExpression": "case when genre = 'Action' then 'Exciting' else 'Boring' end",
+}
+
 
 class BaseTestChartDataApi(SupersetTestCase):
     query_context_payload_template = None
@@ -1059,6 +1066,33 @@ class TestGetChartDataApi(BaseTestChartDataApi):
         assert unique_genders == {"male", "female"}
         assert result["applied_filters"] == [{"column": "male_or_female"}]
 
+    @pytest.mark.usefixtures("load_birth_names_dashboard_with_slices")
+    def test_chart_data_with_incompatible_adhoc_column(self):
+        """
+        Chart data API: Test query with adhoc column that fails to run on this dataset
+        """
+        self.login(username="admin")
+        request_payload = get_query_context("birth_names")
+        request_payload["queries"][0]["columns"] = [ADHOC_COLUMN_FIXTURE]
+        request_payload["queries"][0]["filters"] = [
+            {"col": INCOMPATIBLE_ADHOC_COLUMN_FIXTURE, "op": "IN", "val": ["Exciting"]},
+            {"col": ADHOC_COLUMN_FIXTURE, "op": "IN", "val": ["male", "female"]},
+        ]
+        rv = self.post_assert_metric(CHART_DATA_URI, request_payload, "data")
+        response_payload = json.loads(rv.data.decode("utf-8"))
+        result = response_payload["result"][0]
+        data = result["data"]
+        assert {column for column in data[0].keys()} == {"male_or_female", "sum__num"}
+        unique_genders = {row["male_or_female"] for row in data}
+        assert unique_genders == {"male", "female"}
+        assert result["applied_filters"] == [{"column": "male_or_female"}]
+        assert result["rejected_filters"] == [
+            {
+                "column": "exciting_or_boring",
+                "reason": ExtraFiltersReasonType.COL_NOT_IN_DATASOURCE,
+            }
+        ]
+
 
 @pytest.fixture()
 def physical_query_context(physical_dataset) -> Dict[str, Any]:

tests/integration_tests/datasets/api_tests.py

@@ -239,28 +239,47 @@ class TestDatasetApi(SupersetTestCase):
         response = json.loads(rv.data.decode("utf-8"))
         assert response["result"] == []
 
-    def test_get_dataset_list_gamma_owned(self):
+    def test_get_dataset_list_gamma_has_database_access(self):
         """
-        Dataset API: Test get dataset list owned by gamma
+        Dataset API: Test get dataset list with database access
         """
         if backend() == "sqlite":
             return
 
-        main_db = get_main_database()
-        owned_dataset = self.insert_dataset(
-            "ab_user", [self.get_user("gamma").id], main_db
-        )
-
         self.login(username="gamma")
+
+        # create new dataset
+        main_db = get_main_database()
+        dataset = self.insert_dataset("ab_user", [], main_db)
+
+        # make sure dataset is not visible due to missing perms
         uri = "api/v1/dataset/"
         rv = self.get_assert_metric(uri, "get_list")
         assert rv.status_code == 200
         response = json.loads(rv.data.decode("utf-8"))
 
-        assert response["count"] == 1
-        assert response["result"][0]["table_name"] == "ab_user"
+        assert response["count"] == 0
 
-        db.session.delete(owned_dataset)
+        # give database access to main db
+        main_db_pvm = security_manager.find_permission_view_menu(
+            "database_access", main_db.perm
+        )
+        gamma_role = security_manager.find_role("Gamma")
+        gamma_role.permissions.append(main_db_pvm)
+        db.session.commit()
+
+        # make sure dataset is now visible
+        uri = "api/v1/dataset/"
+        rv = self.get_assert_metric(uri, "get_list")
+        assert rv.status_code == 200
+        response = json.loads(rv.data.decode("utf-8"))
+
+        tables = {tbl["table_name"] for tbl in response["result"]}
+        assert tables == {"ab_user"}
+
+        # revert gamma permission
+        gamma_role.permissions.remove(main_db_pvm)
+        db.session.delete(dataset)
         db.session.commit()
 
     def test_get_dataset_related_database_gamma(self):
@@ -2255,6 +2274,8 @@ class TestDatasetApi(SupersetTestCase):
         assert len(new_dataset.columns) == 2
         assert new_dataset.columns[0].column_name == "id"
         assert new_dataset.columns[1].column_name == "name"
+        db.session.delete(new_dataset)
+        db.session.commit()
 
     @pytest.mark.usefixtures("create_datasets")
     def test_duplicate_physical_dataset(self):

tests/integration_tests/queries/api_tests.py

@@ -202,6 +202,10 @@ class TestQueryApi(SupersetTestCase):
         gamma2 = self.create_user(
             "gamma_2", "password", "Gamma", email="gamma2@superset.org"
         )
+        # Add SQLLab role to these gamma users, so they have access to queries
+        sqllab_role = self.get_role("sql_lab")
+        gamma1.roles.append(sqllab_role)
+        gamma2.roles.append(sqllab_role)
 
         gamma1_client_id = self.get_random_string()
         gamma2_client_id = self.get_random_string()
@@ -383,7 +387,7 @@ class TestQueryApi(SupersetTestCase):
             sql="SELECT col1, col2 from table1",
         )
 
-        self.login(username="gamma")
+        self.login(username="gamma_sqllab")
         arguments = {"filters": [{"col": "sql", "opr": "sw", "value": "SELECT col1"}]}
         uri = f"api/v1/query/?q={prison.dumps(arguments)}"
         rv = self.client.get(uri)

tests/integration_tests/queries/saved_queries/api_tests.py

@@ -748,7 +748,7 @@ class TestSavedQueryApi(SupersetTestCase):
             db.session.query(SavedQuery).filter(SavedQuery.created_by == admin).first()
         )
 
-        self.login(username="gamma")
+        self.login(username="gamma_sqllab")
         argument = [sample_query.id]
         uri = f"api/v1/saved_query/export/?q={prison.dumps(argument)}"
         rv = self.client.get(uri)

tests/integration_tests/security_tests.py

@@ -1332,8 +1332,11 @@ class TestRolePermission(SupersetTestCase):
         self.assertNotIn(("menu_access", view_menu), permissions_set)
 
     def assert_cannot_gamma(self, perm_set):
+        self.assert_cannot_write("Annotation", perm_set)
         self.assert_cannot_write("CssTemplate", perm_set)
+        self.assert_cannot_menu("SQL Lab", perm_set)
         self.assert_cannot_menu("CSS Templates", perm_set)
+        self.assert_cannot_menu("Annotation Layers", perm_set)
         self.assert_cannot_menu("Manage", perm_set)
         self.assert_cannot_menu("Queries", perm_set)
         self.assert_cannot_menu("Import dashboards", perm_set)
@@ -1374,7 +1377,6 @@ class TestRolePermission(SupersetTestCase):
         self.assert_can_all("Annotation", perm_set)
         self.assert_can_all("CssTemplate", perm_set)
         self.assert_can_all("Dataset", perm_set)
-        self.assert_can_read("Query", perm_set)
         self.assert_can_read("Database", perm_set)
         self.assertIn(("can_import_dashboards", "Superset"), perm_set)
         self.assertIn(("can_this_form_post", "CsvToDatabaseView"), perm_set)
@@ -1504,6 +1506,8 @@ class TestRolePermission(SupersetTestCase):
         self.assert_can_gamma(alpha_perm_tuples)
         self.assert_can_alpha(alpha_perm_tuples)
         self.assert_cannot_alpha(alpha_perm_tuples)
+        self.assertNotIn(("can_this_form_get", "UserInfoEditView"), alpha_perm_tuples)
+        self.assertNotIn(("can_this_form_post", "UserInfoEditView"), alpha_perm_tuples)
 
     @pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
     def test_admin_permissions(self):
@@ -1548,6 +1552,8 @@ class TestRolePermission(SupersetTestCase):
         # make sure that user can create slices and dashboards
         self.assert_can_all("Dashboard", gamma_perm_set)
         self.assert_can_read("Dataset", gamma_perm_set)
+        self.assert_can_read("Annotation", gamma_perm_set)
+        self.assert_can_read("CssTemplate", gamma_perm_set)
 
         # make sure that user can create slices and dashboards
         self.assert_can_all("Chart", gamma_perm_set)

tests/integration_tests/utils_tests.py

@@ -991,6 +991,7 @@ class TestUtils(SupersetTestCase):
         slc = self.get_slice("Girls", db.session)
         dashboard_id = 1
 
+        assert slc.viz is not None
         resp = self.get_json_resp(
             f"/superset/explore_json/{slc.datasource_type}/{slc.datasource_id}/"
             + f'?form_data={{"slice_id": {slc.id}}}&dashboard_id={dashboard_id}',

tests/unit_tests/db_engine_specs/test_clickhouse.py

@@ -16,12 +16,26 @@
 # under the License.
 
 from datetime import datetime
-from typing import Optional
+from typing import Any, Dict, Optional, Type
 from unittest.mock import Mock
 
 import pytest
+from sqlalchemy.types import (
+    Boolean,
+    Date,
+    DateTime,
+    DECIMAL,
+    Float,
+    Integer,
+    String,
+    TypeEngine,
+)
 
-from tests.unit_tests.db_engine_specs.utils import assert_convert_dttm
+from superset.utils.core import GenericDataType
+from tests.unit_tests.db_engine_specs.utils import (
+    assert_column_spec,
+    assert_convert_dttm,
+)
 from tests.unit_tests.fixtures.common import dttm
 
 
@@ -53,3 +67,147 @@ def test_execute_connection_error() -> None:
     )
     with pytest.raises(SupersetDBAPIDatabaseError) as ex:
         ClickHouseEngineSpec.execute(cursor, "SELECT col1 from table1")
+
+
+@pytest.mark.parametrize(
+    "target_type,expected_result",
+    [
+        ("Date", "toDate('2019-01-02')"),
+        ("DateTime", "toDateTime('2019-01-02 03:04:05')"),
+        ("UnknownType", None),
+    ],
+)
+def test_connect_convert_dttm(
+    target_type: str, expected_result: Optional[str], dttm: datetime
+) -> None:
+    from superset.db_engine_specs.clickhouse import ClickHouseEngineSpec as spec
+
+    assert_convert_dttm(spec, target_type, expected_result, dttm)
+
+
+@pytest.mark.parametrize(
+    "native_type,sqla_type,attrs,generic_type,is_dttm",
+    [
+        ("String", String, None, GenericDataType.STRING, False),
+        ("LowCardinality(String)", String, None, GenericDataType.STRING, False),
+        ("Nullable(String)", String, None, GenericDataType.STRING, False),
+        (
+            "LowCardinality(Nullable(String))",
+            String,
+            None,
+            GenericDataType.STRING,
+            False,
+        ),
+        ("Array(UInt8)", String, None, GenericDataType.STRING, False),
+        ("Enum('hello', 'world')", String, None, GenericDataType.STRING, False),
+        ("Enum('UInt32', 'Bool')", String, None, GenericDataType.STRING, False),
+        (
+            "LowCardinality(Enum('hello', 'world'))",
+            String,
+            None,
+            GenericDataType.STRING,
+            False,
+        ),
+        (
+            "Nullable(Enum('hello', 'world'))",
+            String,
+            None,
+            GenericDataType.STRING,
+            False,
+        ),
+        (
+            "LowCardinality(Nullable(Enum('hello', 'world')))",
+            String,
+            None,
+            GenericDataType.STRING,
+            False,
+        ),
+        ("FixedString(16)", String, None, GenericDataType.STRING, False),
+        ("Nullable(FixedString(16))", String, None, GenericDataType.STRING, False),
+        (
+            "LowCardinality(Nullable(FixedString(16)))",
+            String,
+            None,
+            GenericDataType.STRING,
+            False,
+        ),
+        ("UUID", String, None, GenericDataType.STRING, False),
+        ("Int8", Integer, None, GenericDataType.NUMERIC, False),
+        ("Int16", Integer, None, GenericDataType.NUMERIC, False),
+        ("Int32", Integer, None, GenericDataType.NUMERIC, False),
+        ("Int64", Integer, None, GenericDataType.NUMERIC, False),
+        ("Int128", Integer, None, GenericDataType.NUMERIC, False),
+        ("Int256", Integer, None, GenericDataType.NUMERIC, False),
+        ("Nullable(Int256)", Integer, None, GenericDataType.NUMERIC, False),
+        (
+            "LowCardinality(Nullable(Int256))",
+            Integer,
+            None,
+            GenericDataType.NUMERIC,
+            False,
+        ),
+        ("UInt8", Integer, None, GenericDataType.NUMERIC, False),
+        ("UInt16", Integer, None, GenericDataType.NUMERIC, False),
+        ("UInt32", Integer, None, GenericDataType.NUMERIC, False),
+        ("UInt64", Integer, None, GenericDataType.NUMERIC, False),
+        ("UInt128", Integer, None, GenericDataType.NUMERIC, False),
+        ("UInt256", Integer, None, GenericDataType.NUMERIC, False),
+        ("Nullable(UInt256)", Integer, None, GenericDataType.NUMERIC, False),
+        (
+            "LowCardinality(Nullable(UInt256))",
+            Integer,
+            None,
+            GenericDataType.NUMERIC,
+            False,
+        ),
+        ("Float32", Float, None, GenericDataType.NUMERIC, False),
+        ("Float64", Float, None, GenericDataType.NUMERIC, False),
+        ("Decimal(1, 2)", DECIMAL, None, GenericDataType.NUMERIC, False),
+        ("Decimal32(2)", DECIMAL, None, GenericDataType.NUMERIC, False),
+        ("Decimal64(2)", DECIMAL, None, GenericDataType.NUMERIC, False),
+        ("Decimal128(2)", DECIMAL, None, GenericDataType.NUMERIC, False),
+        ("Decimal256(2)", DECIMAL, None, GenericDataType.NUMERIC, False),
+        ("Bool", Boolean, None, GenericDataType.BOOLEAN, False),
+        ("Nullable(Bool)", Boolean, None, GenericDataType.BOOLEAN, False),
+        ("Date", Date, None, GenericDataType.TEMPORAL, True),
+        ("Nullable(Date)", Date, None, GenericDataType.TEMPORAL, True),
+        ("LowCardinality(Nullable(Date))", Date, None, GenericDataType.TEMPORAL, True),
+        ("Date32", Date, None, GenericDataType.TEMPORAL, True),
+        ("Datetime", DateTime, None, GenericDataType.TEMPORAL, True),
+        ("Nullable(Datetime)", DateTime, None, GenericDataType.TEMPORAL, True),
+        (
+            "LowCardinality(Nullable(Datetime))",
+            DateTime,
+            None,
+            GenericDataType.TEMPORAL,
+            True,
+        ),
+        ("Datetime('UTC')", DateTime, None, GenericDataType.TEMPORAL, True),
+        ("Datetime64(3)", DateTime, None, GenericDataType.TEMPORAL, True),
+        ("Datetime64(3, 'UTC')", DateTime, None, GenericDataType.TEMPORAL, True),
+    ],
+)
+def test_connect_get_column_spec(
+    native_type: str,
+    sqla_type: Type[TypeEngine],
+    attrs: Optional[Dict[str, Any]],
+    generic_type: GenericDataType,
+    is_dttm: bool,
+) -> None:
+    from superset.db_engine_specs.clickhouse import ClickHouseConnectEngineSpec as spec
+
+    assert_column_spec(spec, native_type, sqla_type, attrs, generic_type, is_dttm)
+
+
+@pytest.mark.parametrize(
+    "column_name,expected_result",
+    [
+        ("time", "time_07cc69"),
+        ("count", "count_e2942a"),
+    ],
+)
+def test_connect_make_label_compatible(column_name: str, expected_result: str) -> None:
+    from superset.db_engine_specs.clickhouse import ClickHouseConnectEngineSpec as spec
+
+    label = spec.make_label_compatible(column_name)
+    assert label == expected_result

tests/unit_tests/db_engine_specs/test_mssql.py

@@ -20,7 +20,7 @@ from textwrap import dedent
 from typing import Any, Dict, Optional, Type
 
 import pytest
-from sqlalchemy import column, table, types
+from sqlalchemy import column, table
 from sqlalchemy.dialects import mssql
 from sqlalchemy.dialects.mssql import DATE, NTEXT, NVARCHAR, TEXT, VARCHAR
 from sqlalchemy.sql import select
@@ -50,7 +50,7 @@ from tests.unit_tests.fixtures.common import dttm
 )
 def test_get_column_spec(
     native_type: str,
-    sqla_type: Type[types.TypeEngine],
+    sqla_type: Type[TypeEngine],
     attrs: Optional[Dict[str, Any]],
     generic_type: GenericDataType,
     is_dttm: bool,

tests/unit_tests/db_engine_specs/test_mysql.py

@@ -33,6 +33,7 @@ from sqlalchemy.dialects.mysql import (
     TINYINT,
     TINYTEXT,
 )
+from sqlalchemy.engine.url import make_url
 
 from superset.utils.core import GenericDataType
 from tests.unit_tests.db_engine_specs.utils import (
@@ -99,6 +100,25 @@ def test_convert_dttm(
     assert_convert_dttm(spec, target_type, expected_result, dttm)
 
 
+@pytest.mark.parametrize(
+    "sqlalchemy_uri,error",
+    [
+        ("mysql://user:password@host/db1?local_infile=1", True),
+        ("mysql://user:password@host/db1?local_infile=0", True),
+        ("mysql://user:password@host/db1", False),
+    ],
+)
+def test_validate_database_uri(sqlalchemy_uri: str, error: bool) -> None:
+    from superset.db_engine_specs.mysql import MySQLEngineSpec
+
+    url = make_url(sqlalchemy_uri)
+    if error:
+        with pytest.raises(ValueError):
+            MySQLEngineSpec.validate_database_uri(url)
+        return
+    MySQLEngineSpec.validate_database_uri(url)
+
+
 @patch("sqlalchemy.engine.Engine.connect")
 def test_get_cancel_query_id(engine_mock: Mock) -> None:
     from superset.db_engine_specs.mysql import MySQLEngineSpec

tests/unit_tests/memoized_tests.py

@@ -1,96 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-from pytest import mark
-
-from superset.utils.memoized import memoized
-
-
-@mark.unittest
-class TestMemoized:
-    def test_memoized_on_functions(self):
-        watcher = {"val": 0}
-
-        @memoized
-        def test_function(a, b, c):
-            watcher["val"] += 1
-            return a * b * c
-
-        result1 = test_function(1, 2, 3)
-        result2 = test_function(1, 2, 3)
-        assert result1 == result2
-        assert watcher["val"] == 1
-
-    def test_memoized_on_methods(self):
-        class test_class:
-            def __init__(self, num):
-                self.num = num
-                self.watcher = 0
-
-            @memoized
-            def test_method(self, a, b, c):
-                self.watcher += 1
-                return a * b * c * self.num
-
-        instance = test_class(5)
-        result1 = instance.test_method(1, 2, 3)
-        result2 = instance.test_method(1, 2, 3)
-        assert result1 == result2
-        assert instance.watcher == 1
-        instance.num = 10
-        assert result2 == instance.test_method(1, 2, 3)
-
-    def test_memoized_on_methods_with_watches(self):
-        class test_class:
-            def __init__(self, x, y):
-                self.x = x
-                self.y = y
-                self.watcher = 0
-
-            @memoized(watch=("x", "y"))
-            def test_method(self, a, b, c):
-                self.watcher += 1
-                return a * b * c * self.x * self.y
-
-        instance = test_class(3, 12)
-        result1 = instance.test_method(1, 2, 3)
-        result2 = instance.test_method(1, 2, 3)
-        assert result1 == result2
-        assert instance.watcher == 1
-        result3 = instance.test_method(2, 3, 4)
-        assert instance.watcher == 2
-        result4 = instance.test_method(2, 3, 4)
-        assert instance.watcher == 2
-        assert result3 == result4
-        assert result3 != result1
-        instance.x = 1
-        result5 = instance.test_method(2, 3, 4)
-        assert instance.watcher == 3
-        assert result5 != result4
-        result6 = instance.test_method(2, 3, 4)
-        assert instance.watcher == 3
-        assert result6 == result5
-        instance.x = 10
-        instance.y = 10
-        result7 = instance.test_method(2, 3, 4)
-        assert instance.watcher == 4
-        assert result7 != result6
-        instance.x = 3
-        instance.y = 12
-        result8 = instance.test_method(1, 2, 3)
-        assert instance.watcher == 4
-        assert result1 == result8

tests/unit_tests/models/core_test.py

@@ -59,9 +59,7 @@ def test_get_metrics(mocker: MockFixture) -> None:
                 },
             ]
 
-    database.get_db_engine_spec = mocker.MagicMock(  # type: ignore
-        return_value=CustomSqliteEngineSpec
-    )
+    database.get_db_engine_spec = mocker.MagicMock(return_value=CustomSqliteEngineSpec)
     assert database.get_metrics("table") == [
         {
             "expression": "COUNT(DISTINCT user_id)",