Compare commits

..

30 Commits

Author SHA1 Message Date
Beto Dealmeida
60781a05d6 fix: SQL Lab vertical space (#12187)
(cherry picked from commit de61859e98)
2021-01-11 12:32:20 -08:00
Ville Brofeldt
617d6e51a9 silence two tests 2021-01-07 22:12:05 +02:00
Ville Brofeldt
aa9622c272 fix black linting error 2021-01-07 21:38:53 +02:00
Ville Brofeldt
70e83e811c feat(chart-data-api): ignore unknown fields on QueryObject (#12118) 2021-01-07 21:35:17 +02:00
Kamil Gabryjelski
722e2ea204 fix(explore): bugs in Custom SQL editor in filter popover (#12278)
* Fix Save button disabled until clause is switched

* Fix 'undefined undefined' when sql query is empty

* fix test

Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>
2021-01-07 20:56:16 +02:00
Maxwell Amante
0b0891cf1c chore(home): add ErrorBoundary to Charts section (#12239) 2021-01-07 20:55:58 +02:00
Ville Brofeldt
cd8906c181 fix(explore): bump superset-ui to 0.16.3 and fix DeckGL bug (#12242) 2021-01-07 20:55:44 +02:00
Jesse Yang
58d577989f refactor: remove queryFields in QueryObject and update chart control configs (#12091)
* Clean up queryFields

* Clean up unused vars

* Bump chart plugins

* Bringing changes in #12147
2021-01-07 20:54:34 +02:00
Yongjie Zhao
d6d4eb7644 feat(explore): Time picker enhancement follow up (#12208)
* refactor layout

* WIP

* fix typing

* styling

* fix lint

* frontend IT

* rename variable

* added quarter

* typos

* refine code structure
2021-01-07 20:50:12 +02:00
Kamil Gabryjelski
33bfa90086 fix: Altered button crashing and title not updating on Explore view (#11960)
* Fix chart name not updating

* Fix AlteredSliceTag crashing

* Update name after it's changed in another modal
2021-01-07 20:50:02 +02:00
Kamil Gabryjelski
2085b4c6dc Refetch samples only when filters change (#11999) 2021-01-07 20:49:50 +02:00
Kamil Gabryjelski
9ac25898fb feat: Implement drag & drop for metrics and filters labels (#12184) 2021-01-07 20:49:43 +02:00
Evan Rusackas
098ea0c850 feat: Native filters fast-follow (#12138)
* wip: filter create modal

* add a feature flag

* automatic changes to package lock

* wip

* filter sidebar and basic state management

* move create button to the sidebar

* first step for edit filterconfig

* partially fix tests...

* edits to types and comments

* respect feature flag on the filter sidebar

* add filterconfig form

* get input state working

* feat: tree filter scopes UI

* fix: turn on flag

* sticky filter bar

* stop preferring default export

* feat: finish filter scoping

* fix: under toggle

* fix: title

* fix: add licence

* refactor: update TS

* fix: fix on reopen modal + validation

* new filter bar menu

* adding, but commenting out, bulk scoping action

* adding some placeholder buttons and styles therefor

* feat: add filter chart

* add relative path to package.json

* update modal

* a little input styling... just getting warmed up

* Revert "feat: add filter chart"

This reverts commit b1302d35b6.

* Revert "add relative path to package.json"

This reverts commit 26a7b40e18.

* https package lock idk

* feat: add filter chart

* add relative path to package.json

* flexboxes all the way down

* dynamically generate groupby and datasource in select control

* big wip

* fix target column name

* no importing nonexistent things

* styles and name editing

* Add hook for retrieval of all filter states

* start with a new filter when clicking add filter

* handle removed filters gracefully

* fix incorrect default filter configuration

* add fields to useAllFilterState

* add redux for filterconfigs

* add support for native_filters

* remove consoles

* improve filter removal

* unbreak infinite loop

* basic sidebar toggling working!

* collapsing and menu working more smoothly

* linting

* make dataset and column inputs work

* save filter values properly

* add dashboard event for filter updates

* guarded

* apply filters properly

* fix schema

* making New Filter button a link

* gridunits ftw

* centering modal

* tis not a button anymore! nixing type.

* plus and collapse buttons instead of "more" menu

* updating full size filter icons

* adding icons to filter collapsing/expanding

* turning off animation, but leaving class-based animation css

* fix linting error

* fix native filters for legacy charts

* updates test

* no individual apply buttons

* fix bugs with filter config modal

* remove redundant code

* switch to the filter with validation errors on submit

* separate form validation

* switch config button from add to edit

* switch to the filter with validation errors on submit

* separate form validation

* switch config button from add to edit

* update tests

* oops forgot to add the fancy new useChangeEffect hook

* comments and code reorganization

* rename native_filters to extr_form_data and move hook

* disable native filters in viz selector

* add cascading

* implement new extra form data api

* cleanup

* updates tests

* bump npm packages

* fix bad merge on package.json + lock

* lint

* replace in and not in with uppercase

* lint

* lint

* lint

* lint

* bulk test fix

* Sort select input alphabetically

* Change type for sorting elements

* sleeker filter removal UX

* fix rest of unit tests

* make filter operators all uppercase

* Hide Filter bar when there are no filters

* Show edit button for dashboard owners only

* Add visible argument to filters toggle function to avoid future regression

* Improve Toggle filters bar function

* lint

* fix js lint + set createNewOnOpen

* Handle setting extra form data in Filter Bar instead of Filter Control

* Add Handle apply filter function to Apply button

* Allow applying changes instantly

* Fix types

* remove console logs

* fix package

* Add Error Boundary component to Filter bar and Filter Config Modal

* fix jest tests

* update native filters tests to pass

* reset cypress baseUrl

* remove unnecessary field

* Add Parent Filter input field to Config Modal

* Create Cascade Filter & display children filters

* Add Cascade Popover

* Display Filter value both in Filter Bar and in Cascade Popover

* Display the youngest filter value label in the Filter bar

* Add styles to Cascade Popover and filters

* Force to apply changes instantly for parent filters and refactor styles

* Show error for no cyclical hierarchy and refactor

* Add validation for parent filter to be applied instantly

* Add Error Boundary to Filter Config Modal

* cleanup: remove unused state fields

* move unrelated types to an appropriate location

* remove misplaced resource fetch error logic

* fix cascadeParentIds error

* fix cypress password

* initial attempt at fixing scope issue

* fix bad merge

* fix lint

* trying out makeApi for saving filters

* remove unused import

* fix test

* silence bad test

* Improve styling of Filter Config Modal

* Improve styles for whole native filters feature

* Add styles for active filter tab

* Fix text for scoping

* Clean up Filter Bar and Config Modal styles

* Remove fractional gridUnits. Change name for CheckboxFormItem. Add placeholder to Parent Filter select.

* Remove unnecessary button size for Config Modal

* add native-filter feat flag config

* oops fix here

* remove space

* Update superset-frontend/src/common/components/index.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/cypress-base/cypress/integration/dashboard/nativeFilters.test.ts

Co-authored-by: Evan Rusackas <evan@preset.io>

* Add Cache Wrapper helper to avoid datasets requests deduplication

* Add license to new Cache Wrapper helper

* Add Cache Wrapper tests

* Fix expanding Filter Bar

* use styledMount in tests

* comment

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterBar.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterConfigForm.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterConfigModal.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterConfigurationLink.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* address PR feedback

* fix package lock

* null guards

* Fix charts resizing

* removing emotion/react and going old school on css animation

* fxing css glitch on scoping disclaimer

* src paths.

* using gridUnits

* nixing unnecessary diamonds

* linting

* fix type errors

* Inverting collapsed icons... closer to data src selector design

* restoring feature flag to proper default setting

* missing condition

* fix tests

* patching test

* just a button

* flaky tests

Co-authored-by: David Aaron Suddjian <aasuddjian@gmail.com>
Co-authored-by: Phillip Kelley-Dotson <pkelleydotson@yahoo.com>
Co-authored-by: Simcha Shats <simcha.shats@nielsen.com>
Co-authored-by: amitNielsen <amit.miran@nielsen.com>
Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>
Co-authored-by: David Aaron Suddjian <1858430+suddjian@users.noreply.github.com>
Co-authored-by: Agata Stawarz-Pastewska <agata.stawarz-pastewska@polidea.com>
Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2021-01-07 20:49:32 +02:00
David Aaron Suddjian
c69b517bfc feat(dashboard): Dashboard-Native Filters (#11814)
* wip: filter create modal

* add a feature flag

* automatic changes to package lock

* wip

* filter sidebar and basic state management

* move create button to the sidebar

* first step for edit filterconfig

* partially fix tests...

* edits to types and comments

* respect feature flag on the filter sidebar

* add filterconfig form

* get input state working

* feat: tree filter scopes UI

* fix: turn on flag

* sticky filter bar

* stop preferring default export

* feat: finish filter scoping

* fix: under toggle

* fix: title

* fix: add licence

* refactor: update TS

* fix: fix on reopen modal + validation

* new filter bar menu

* adding, but commenting out, bulk scoping action

* adding some placeholder buttons and styles therefor

* feat: add filter chart

* add relative path to package.json

* update modal

* a little input styling... just getting warmed up

* Revert "feat: add filter chart"

This reverts commit b1302d35b6.

* Revert "add relative path to package.json"

This reverts commit 26a7b40e18.

* https package lock idk

* feat: add filter chart

* add relative path to package.json

* flexboxes all the way down

* dynamically generate groupby and datasource in select control

* big wip

* fix target column name

* no importing nonexistent things

* styles and name editing

* Add hook for retrieval of all filter states

* start with a new filter when clicking add filter

* handle removed filters gracefully

* fix incorrect default filter configuration

* add fields to useAllFilterState

* add redux for filterconfigs

* add support for native_filters

* remove consoles

* improve filter removal

* unbreak infinite loop

* basic sidebar toggling working!

* collapsing and menu working more smoothly

* linting

* make dataset and column inputs work

* save filter values properly

* add dashboard event for filter updates

* guarded

* apply filters properly

* fix schema

* making New Filter button a link

* gridunits ftw

* centering modal

* tis not a button anymore! nixing type.

* plus and collapse buttons instead of "more" menu

* updating full size filter icons

* adding icons to filter collapsing/expanding

* turning off animation, but leaving class-based animation css

* fix linting error

* fix native filters for legacy charts

* updates test

* no individual apply buttons

* fix bugs with filter config modal

* remove redundant code

* switch to the filter with validation errors on submit

* separate form validation

* switch config button from add to edit

* update tests

* oops forgot to add the fancy new useChangeEffect hook

* comments and code reorganization

* rename native_filters to extr_form_data and move hook

* disable native filters in viz selector

* add cascading

* implement new extra form data api

* cleanup

* updates tests

* bump npm packages

* fix bad merge on package.json + lock

* lint

* replace in and not in with uppercase

* lint

* lint

* lint

* lint

* bulk test fix

* Sort select input alphabetically

* Change type for sorting elements

* fix rest of unit tests

* make filter operators all uppercase

* Hide Filter bar when there are no filters

* Show edit button for dashboard owners only

* Add visible argument to filters toggle function to avoid future regression

* Improve Toggle filters bar function

* lint

* fix js lint + set createNewOnOpen

* Handle setting extra form data in Filter Bar instead of Filter Control

* Add Handle apply filter function to Apply button

* Allow applying changes instantly

* Fix types

* remove console logs

* Add Error Boundary component to Filter bar and Filter Config Modal

* fix jest tests

* update native filters tests to pass

* reset cypress baseUrl

* remove unnecessary field

* cleanup: remove unused state fields

* move unrelated types to an appropriate location

* remove misplaced resource fetch error logic

* fix cascadeParentIds error

* fix cypress password

* initial attempt at fixing scope issue

* fix bad merge

* fix lint

* trying out makeApi for saving filters

* remove unused import

* fix test

* silence bad test

* add native-filter feat flag config

* oops fix here

* remove space

* Update superset-frontend/src/common/components/index.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/cypress-base/cypress/integration/dashboard/nativeFilters.test.ts

Co-authored-by: Evan Rusackas <evan@preset.io>

* use styledMount in tests

* comment

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterBar.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterConfigForm.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterConfigModal.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* Update superset-frontend/src/dashboard/components/nativeFilters/FilterConfigurationLink.tsx

Co-authored-by: Evan Rusackas <evan@preset.io>

* address PR feedback

* fix package lock

* null guards

* Fix charts resizing

* fix cypress tests

* add in nativefilters to form data

* fix lint and test

Co-authored-by: Phillip Kelley-Dotson <pkelleydotson@yahoo.com>
Co-authored-by: Simcha Shats <simcha.shats@nielsen.com>
Co-authored-by: amitNielsen <amit.miran@nielsen.com>
Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>
Co-authored-by: Evan Rusackas <evan@preset.io>
Co-authored-by: Agata Stawarz-Pastewska <agata.stawarz-pastewska@polidea.com>
Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2021-01-07 20:49:12 +02:00
Jesse Yang
cb4c79c499 fix: fix the flaky FilterBox test re-introduced by #12114 (#12124) 2021-01-07 20:47:56 +02:00
Yongjie Zhao
7a9488fef7 feat(explore): time picker enhancement (#11418) 2021-01-07 20:47:20 +02:00
Evan Rusackas
bc5b815976 fix(explore): overflow issue with metric options (#12129)
* fixing overflow issue

* lint!
2021-01-07 20:09:04 +02:00
Michael S. Molina
16743c3541 Improves chart save and go button (#12125)
Resolves #11748
2021-01-07 20:08:58 +02:00
Kamil Gabryjelski
04d3559d88 feat: Add Saved Metrics tab to metrics popover (#12123)
* Implement saved metrics

* Fix bug in sql editor

* Fix unit tests

* Fix outlines in popovers

* Add types for saved metrics

* Add translations

* Move savedMetricType to a separate file
2021-01-07 20:08:53 +02:00
Kamil Gabryjelski
88ae1455a4 feat(explore): metrics and filters controls redesign (#12095)
* Redesign metrics control

* Redesign filters control

* Bugfixes

* Fix unit tests

* Fix tests

* Code review fixes
2021-01-07 20:08:46 +02:00
simchaNielsen
4253edd91a feat: Support multiple queries per request (#11880)
* refactor: add queriesData fields for multiple queries

* feat: support multi queries request

* lint: fix lint

* lint: fix lint

* lint: fix lint

* fix: fix CR notes

* fix: fix CR notes

* fix: fix CR notes

* fix: fix error case for multi queries

* feat: change queryResponse to queriesResponse

* fix: revert webpack

* test: fix tests

* chore: lint

* chore: adjust asyncEvent to multiple results

* fix: lint

* fix: eslint

* fix: another eslint rule

Co-authored-by: Amit Miran <47772523+amitmiran137@users.noreply.github.com>
Co-authored-by: amitmiran137 <amit.miran@nielsen.com>
2021-01-07 20:08:39 +02:00
Kamil Gabryjelski
b744308cba fix(explore): bugs in Custom SQL editor in filter popover (#12278)
* Fix Save button disabled until clause is switched

* Fix 'undefined undefined' when sql query is empty

* fix test

Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>
(cherry picked from commit e660723a2f)
2021-01-06 12:15:11 -08:00
Kamil Gabryjelski
5f8cf8ce11 Bump superset-ui deps (#12280)
(cherry picked from commit d0668fa7e4)
2021-01-06 12:14:47 -08:00
Beto Dealmeida
08136cffc9 fix: load example data into correct DB (#12292)
* fix: load example data into correct DB

* Fix force_data

* Fix lint

(cherry picked from commit 6b2b208b3b)
2021-01-06 12:11:29 -08:00
Beto Dealmeida
e25e869b50 fix: CTAS on multiple statements (#12188)
* WIP

* Add unit tests for sql_parse

* Add unit tests for sql_lab

(cherry picked from commit 164db3e5a1)
2021-01-06 11:51:34 -08:00
Henry Yeh
d973bd8602 superset-ui: bump version to 0.16.4 (https://github.com/apache-superset/superset-ui/pull/886) 2021-01-06 11:35:15 -08:00
Daniel Vaz Gaspar
ce05a3dde3 fix: database alpha permissions (#12136)
* fix: database alpha permissions

* add test

(cherry picked from commit 2f0add3aec)
2020-12-21 15:18:19 -08:00
Beto Dealmeida
c6a08fdc46 fix: make example table name safe (#12135)
(cherry picked from commit 81f5631fe5)
2020-12-21 15:14:02 -08:00
Jesse Yang
09589059a1 fix: properly render booleans in FilterBox and explore page data preview (#12116)
(cherry picked from commit 8e625e0a64)
2020-12-21 15:13:49 -08:00
Jesse Yang
666e08b819 fix: FilterBox select lose focus when focused (#12114)
(cherry picked from commit af130ea5e9)
2020-12-21 15:13:36 -08:00
5048 changed files with 247349 additions and 853533 deletions

View File

@@ -19,7 +19,7 @@
---
github:
description: "Apache Superset is a Data Visualization and Data Exploration Platform"
homepage: https://superset.apache.org/
homepage: https://superset.incubator.apache.org/
labels:
- superset
- apache
@@ -52,30 +52,3 @@ github:
squash: true
merge: false
rebase: false
protected_branches:
master:
required_status_checks:
# strict means "Require branches to be up to date before merging".
strict: false
# contexts are the names of checks that must pass
contexts:
- check
- cypress-matrix (1, chrome)
- cypress-matrix (2, chrome)
- cypress-matrix (3, chrome)
- docker-build
- frontend-build
- pre-commit (3.8)
- python-lint (3.8)
- test-mysql (3.8)
- test-postgres (3.8)
- test-postgres (3.9)
- test-sqlite (3.8)
required_pull_request_reviews:
dismiss_stale_reviews: false
require_code_owner_reviews: true
required_approving_review_count: 1
required_signatures: false

View File

@@ -1,35 +1,11 @@
codecov:
notify:
after_n_builds: 4
ignore:
- "superset/migrations/versions/*.py"
- "superset-frontend/packages/superset-ui-demo/**/*"
- "**/*.stories.tsx"
- "**/*.stories.jsx"
coverage:
status:
project:
default:
informational: true
# Commits pushed to master should not make the overall
# project coverage decrease:
target: auto
threshold: 0%
core-packages-ts:
target: 100%
paths:
- 'superset-frontend/packages'
- '!superset-frontend/packages/**/*.jsx'
- '!superset-frontend/packages/**/*.tsx'
core-packages-tsx:
target: 50%
paths:
- 'superset-frontend/packages/**/*.jsx'
- 'superset-frontend/packages/**/*.tsx'
patch:
default:
informational: true
threshold: 0%
flag_management:
default_rules:
carryforward: true

View File

@@ -15,9 +15,6 @@
# limitations under the License.
#
**/__pycache__/
**/.git
**/.apache_superset.egg-info
**/.github
**/.mypy_cache
**/.pytest_cache
**/.tox
@@ -33,15 +30,11 @@
**/*.pyc
**/*.sqllite
**/*.swp
**/.terser-plugin-cache/
**/.storybook/
**/node_modules/
tests/
docs/
install/
superset-frontend/cypress-base/
superset-frontend/node_modules/
superset-frontend/cypress/
superset-frontend/coverage/
superset/static/assets/
superset-websocket/dist/
venv

1
.gitattributes vendored
View File

@@ -1 +0,0 @@
docker/**/*.sh text eol=lf

14
.github/CODEOWNERS vendored
View File

@@ -1,14 +0,0 @@
# Notify all committers of DB migration changes, per SIP-59
# https://github.com/apache/superset/issues/13351
/superset/migrations/ @apache/superset-committers
# Notify Preset team when ephemeral env settings are changed
.github/workflows/ecs-task-definition.json @robdiciuccio @craig-rueda @rusackas @eschutho @dpgaspar @nytai @mistercrunch
.github/workflows/docker-ephemeral-env.yml @robdiciuccio @craig-rueda @rusackas @eschutho @dpgaspar @nytai @mistercrunch
.github/workflows/ephemeral*.yml @robdiciuccio @craig-rueda @rusackas @eschutho @dpgaspar @nytai @mistercrunch
# Notify some committers of changes in the Select component
/superset-frontend/src/components/Select/ @michael-s-molina @geido
# Notify Helm Chart maintainers about changes in it
/helm/superset/ @craig-rueda @dpgaspar @villebro

View File

@@ -7,13 +7,6 @@ labels: "#bug"
A clear and concise description of what the bug is.
#### How to reproduce the bug
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
### Expected results
what you expected to happen.
@@ -26,16 +19,20 @@ what actually happens.
If applicable, add screenshots to help explain your problem.
#### How to reproduce the bug
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
### Environment
(please complete the following information):
- browser type and version:
- superset version: `superset version`
- python version: `python --version`
- node.js version: `node -v`
- any feature flags active:
### Checklist

View File

@@ -6,9 +6,9 @@ labels: "#SIP"
---
*Please make sure you are familiar with the SIP process documented*
(here)[https://github.com/apache/superset/issues/5602]. The SIP number should be the next number after the latest SIP listed [here](https://github.com/apache/superset/issues?q=is%3Aissue+label%3Asip).
(here)[https://github.com/apache/incubator-superset/issues/5602]
## [SIP-\<number>] Proposal for <title>
## [SIP] Proposal for XXX
### Motivation

View File

@@ -1,27 +1,18 @@
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### TEST PLAN
<!--- What steps should be taken to verify the changes -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Requires DB Migration.
- [ ] Confirm DB Migration upgrade and downgrade tested.
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API

View File

@@ -15,15 +15,3 @@ updates:
labels:
- pip
- dependabot
- package-ecosystem: "npm"
directory: ".github/actions"
schedule:
interval: "daily"
open-pull-requests-limit: 0
- package-ecosystem: "npm"
directory: "/docs/"
schedule:
interval: "daily"
open-pull-requests-limit: 0

View File

@@ -55,6 +55,7 @@ npm-install() {
cd "$GITHUB_WORKSPACE/superset-frontend"
# cache-restore npm
say "::group::Install npm packages"
echo "npm: $(npm --version)"
echo "node: $(node --version)"
@@ -68,7 +69,7 @@ build-assets() {
cd "$GITHUB_WORKSPACE/superset-frontend"
say "::group::Build static assets"
npm run build
npm run build -- --no-progress
say "::endgroup::"
}
@@ -80,7 +81,7 @@ build-instrumented-assets() {
if [[ -f "$ASSETS_MANIFEST" ]]; then
echo 'Skip frontend build because instrumented static assets already exist.'
else
npm run build-instrumented
npm run build-instrumented -- --no-progress
cache-save instrumented-assets
fi
say "::endgroup::"
@@ -130,7 +131,12 @@ testdata() {
codecov() {
say "::group::Upload code coverage"
bash ".github/workflows/codecov.sh" "$@"
local codecovScript="${HOME}/codecov.sh"
# download bash script if needed
if [[ ! -f "$codecovScript" ]]; then
curl -s https://codecov.io/bash >"$codecovScript"
fi
bash "$codecovScript" "$@"
say "::endgroup::"
}
@@ -158,10 +164,9 @@ cypress-run() {
export TERM="xterm"
say "::group::Run Cypress for [$page]"
if [[ -z $CYPRESS_KEY ]]; then
if [[ -z $CYPRESS_RECORD_KEY ]]; then
$cypress --spec "cypress/integration/$page" --browser "$browser"
else
export CYPRESS_RECORD_KEY=`echo $CYPRESS_KEY | base64 --decode`
# additional flags for Cypress dashboard recording
$cypress --spec "cypress/integration/$page" --browser "$browser" \
--record --group "$group" --tag "${GITHUB_REPOSITORY},${GITHUB_EVENT_NAME}" \
@@ -185,13 +190,18 @@ cypress-run-all() {
cypress-run "*/**/*"
# Upload code coverage separately so each page can have separate flags
# -c will clean existing coverage reports, -F means add flags
# || true to prevent CI failure on codecov upload
codecov -cF "cypress" || true
# After job is done, print out Flask log for debugging
say "::group::Flask log for default run"
cat "$flasklog"
say "::endgroup::"
# Rerun SQL Lab tests with backend persist enabled
export SUPERSET_CONFIG=tests.integration_tests.superset_test_config_sqllab_backend_persist
export SUPERSET_CONFIG=tests.superset_test_config_sqllab_backend_persist
# Restart Flask with new configs
kill $flaskProcessId
@@ -200,10 +210,8 @@ cypress-run-all() {
cypress-run "sqllab/*" "Backend persist"
# Upload code coverage separately so each page can have separate flags
# -c will clean existing coverage reports, -F means add flags
# || true to prevent CI failure on codecov upload
codecov -c -F "cypress" || true
codecov -cF "cypress" || true
say "::group::Flask log for backend persist"
cat "$flasklog"

View File

@@ -1,40 +0,0 @@
name: Cancel Duplicates
on:
workflow_run:
workflows:
- "Miscellaneous"
types:
- requested
jobs:
cancel-duplicate-runs:
name: Cancel duplicate workflow runs
runs-on: ubuntu-20.04
steps:
- name: Check number of queued tasks
id: check_queued
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPO: ${{ github.repository }}
run: |
get_count() {
echo $(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$GITHUB_REPO/actions/runs?status=$1" | \
jq ".total_count")
}
count=$(( `get_count queued` + `get_count in_progress` ))
echo "Found $count unfinished jobs."
echo "::set-output name=count::$count"
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
if: steps.check_queued.outputs.count >= 20
uses: actions/checkout@v2
- name: Cancel duplicate workflow runs
if: steps.check_queued.outputs.count >= 20
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
pip install click requests typing_extensions python-dateutil
python ./scripts/cancel_github_workflows.py

View File

@@ -1,59 +0,0 @@
name: Check DB migration conflict
on:
push:
paths:
- "superset/migrations/**"
jobs:
check_db_migration_conflict:
name: Check DB migration conflict
runs-on: ubuntu-20.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Check and notify
uses: actions/github-script@v3
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
// API reference: https://octokit.github.io/rest.js
const currentBranch = context.ref.replace('refs/heads/', '');
// Find all pull requests to current branch
const opts = github.pulls.list.endpoint.merge({
owner: context.repo.owner,
repo: context.repo.repo,
base: context.ref,
state: 'open',
sort: 'updated',
per_page: 100,
});
const pulls = await github.paginate(opts);
if (pulls.length > 0) {
console.log(`Found ${pulls.length} open PRs for base branch "${currentBranch}"`)
}
for (const pull of pulls) {
const listFilesOpts = await github.pulls.listFiles.endpoint.merge({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: pull.number,
});
const files = await github.paginate(listFilesOpts);
if (
files.some(x => x.contents_url.includes('/contents/superset/migrations'))
) {
console.log(`PR #${pull.number} "${pull.title}" also added db migration`)
await github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pull.number,
body:
`⚠️ @${pull.user.login} Your base branch \`${currentBranch}\` has just ` +
'also updated `superset/migrations`.\n' +
'\n' +
'❗ **Please consider rebasing your branch to avoid db migration conflicts.**',
});
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,78 +0,0 @@
name: Push ephmereral env image
on:
workflow_run:
workflows: ["Docker"]
types:
- completed
jobs:
docker_ephemeral_env:
name: Push ephemeral env Docker image to ECR
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
steps:
- name: 'Download artifact'
uses: actions/github-script@v3.1.0
with:
script: |
const artifacts = await github.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: ${{ github.event.workflow_run.id }},
});
core.info('*** artifacts')
core.info(JSON.stringify(artifacts))
const matchArtifact = artifacts.data.artifacts.filter((artifact) => {
return artifact.name == "build"
})[0];
if(!matchArtifact) return core.setFailed("Build artifacts not found")
const download = await github.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: matchArtifact.id,
archive_format: 'zip',
});
var fs = require('fs');
fs.writeFileSync('${{github.workspace}}/build.zip', Buffer.from(download.data));
- run: unzip build.zip
- name: Display downloaded files (debug)
run: ls -la
- name: Get SHA
id: get-sha
run: echo "::set-output name=sha::$(cat ./SHA)"
- name: Get PR
id: get-pr
run: echo "::set-output name=num::$(cat ./PR-NUM)"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Load, tag and push image to ECR
id: push-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: superset-ci
SHA: ${{ steps.get-sha.outputs.sha }}
IMAGE_TAG: pr-${{ steps.get-pr.outputs.num }}
run: |
docker load < $SHA.tar.gz
docker tag $SHA $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
docker tag $SHA $ECR_REGISTRY/$ECR_REPOSITORY:$SHA
docker push -a $ECR_REGISTRY/$ECR_REPOSITORY

View File

@@ -1,22 +0,0 @@
name: Docker
on:
release:
types: [published]
jobs:
docker-release:
name: docker-release
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.ref }}
- shell: bash
env:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
.github/workflows/docker_build_push.sh

View File

@@ -5,18 +5,17 @@ on:
branches:
- 'master'
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
types: [opened, synchronize, reopened]
release:
types: [published]
jobs:
docker-build:
if: github.event.pull_request.draft == false
name: docker-build
build:
name: build
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- name: Checkout
uses: actions/checkout@v2
with:
persist-credentials: false
- shell: bash
env:
@@ -24,19 +23,3 @@ jobs:
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
.github/workflows/docker_build_push.sh
- name: Build ephemeral env image
if: github.event_name == 'pull_request'
run: |
mkdir -p ./build
echo ${{ github.sha }} > ./build/SHA
echo ${{ github.event.pull_request.number }} > ./build/PR-NUM
docker build --target ci -t ${{ github.sha }} -t "pr-${{ github.event.pull_request.number }}" .
docker save ${{ github.sha }} | gzip > ./build/${{ github.sha }}.tar.gz
- name: Upload build artifacts
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@v2
with:
name: build
path: build/

View File

@@ -18,17 +18,17 @@
set -eo pipefail
SHA=$(git rev-parse HEAD)
REPO_NAME="apache/superset"
REPO_NAME="apache/incubator-superset"
if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then
REFSPEC=$(echo "${GITHUB_HEAD_REF}" | sed 's/[^a-zA-Z0-9]/-/g' | head -c 40)
REFSPEC=$(echo "${GITHUB_HEAD_REF}" | sed 's/[^a-zA-Z0-9]/-/' | head -c 20)
PR_NUM=$(echo "${GITHUB_REF}" | sed 's:refs/pull/::' | sed 's:/merge::')
LATEST_TAG="pr-${PR_NUM}"
elif [[ "${GITHUB_EVENT_NAME}" == "release" ]]; then
REFSPEC=$(echo "${GITHUB_REF}" | sed 's:refs/tags/::' | head -c 40)
REFSPEC="${GITHUB_REF}"
LATEST_TAG="${REFSPEC}"
else
REFSPEC=$(echo "${GITHUB_REF}" | sed 's:refs/heads/::' | sed 's/[^a-zA-Z0-9]/-/g' | head -c 40)
REFSPEC=$(echo "${GITHUB_REF}" | sed 's:refs/heads/::' | sed 's/[^a-zA-Z0-9]/-/' | head -c 20)
LATEST_TAG="${REFSPEC}"
fi
@@ -76,5 +76,5 @@ else
# Login and push
docker logout
docker login --username "${DOCKERHUB_USER}" --password "${DOCKERHUB_TOKEN}"
docker push --all-tags "${REPO_NAME}"
docker push "${REPO_NAME}"
fi

View File

@@ -1,51 +0,0 @@
{
"containerDefinitions": [
{
"name": "superset-ci",
"image": "apache/superset:latest",
"cpu": 0,
"links": [],
"portMappings": [
{
"containerPort": 8080,
"hostPort": 8080,
"protocol": "tcp"
}
],
"essential": true,
"entryPoint": [],
"command": [],
"environment": [
{
"name": "SUPERSET_LOAD_EXAMPLES",
"value": "yes"
},
{
"name": "SUPERSET_PORT",
"value": "8080"
}
],
"mountPoints": [],
"volumesFrom": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/superset-ci",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
}
}
],
"family": "superset-ci",
"taskRoleArn": "ecsTaskExecutionRole",
"executionRoleArn": "ecsTaskExecutionRole",
"networkMode": "awsvpc",
"volumes": [],
"placementConstraints": [],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "512",
"memory": "1024"
}

View File

@@ -1,23 +0,0 @@
name: Embedded SDK Release
on:
push:
branches:
- 'master'
jobs:
build:
runs-on: ubuntu-20.04
defaults:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: "16"
registry-url: 'https://registry.npmjs.org'
- run: npm ci
- run: npm run ci:release
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

View File

@@ -1,24 +0,0 @@
name: Embedded SDK PR Checks
on:
pull_request:
paths:
- "superset-embedded-sdk/**"
types: [synchronize, opened, reopened, ready_for_review]
jobs:
embedded-sdk-test:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
defaults:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: "16"
registry-url: 'https://registry.npmjs.org'
- run: npm ci
- run: npm test
- run: npm run build

View File

@@ -1,58 +0,0 @@
name: Cleanup ephemeral envs (PR close)
on:
pull_request_target:
types: [closed]
jobs:
ephemeral-env-cleanup:
name: Cleanup ephemeral envs
runs-on: ubuntu-latest
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Describe ECS service
id: describe-services
run: |
echo "::set-output name=active::$(aws ecs describe-services --cluster superset-ci --services pr-${{ github.event.number }}-service | jq '.services[] | select(.status == "ACTIVE") | any')"
- name: Delete ECS service
if: steps.describe-services.outputs.active == 'true'
id: delete-service
run: |
aws ecs delete-service \
--cluster superset-ci \
--service pr-${{ github.event.number }}-service \
--force
- name: Login to Amazon ECR
if: steps.describe-services.outputs.active == 'true'
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Delete ECR image tag
if: steps.describe-services.outputs.active == 'true'
id: delete-image-tag
run: |
aws ecr batch-delete-image \
--registry-id $(echo "${{ steps.login-ecr.outputs.registry }}" | grep -Eo "^[0-9]+") \
--repository-name superset-ci \
--image-ids imageTag=pr-${{ github.event.number }}
- name: Comment (success)
if: steps.describe-services.outputs.active == 'true'
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
github.issues.createComment({
issue_number: ${{ github.event.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: 'Ephemeral environment shutdown and build artifacts deleted.'
})

View File

@@ -1,194 +0,0 @@
name: Ephemeral env workflow
on:
issue_comment:
types: [created]
jobs:
ephemeral_env_comment:
if: github.event.issue.pull_request
name: Evaluate ephemeral env comment trigger (/testenv)
runs-on: ubuntu-latest
outputs:
slash-command: ${{ steps.eval-body.outputs.result }}
feature-flags: ${{ steps.eval-feature-flags.outputs.result }}
steps:
- name: Debug
run: |
echo "Comment on PR #${{ github.event.issue.number }} by ${{ github.event.issue.user.login }}, ${{ github.event.comment.author_association }}"
- name: Eval comment body for /testenv slash command
uses: actions/github-script@v3
id: eval-body
with:
result-encoding: string
script: |
const pattern = /^\/testenv (up|down)/
const result = pattern.exec(context.payload.comment.body)
return result === null ? 'noop' : result[1]
- name: Eval comment body for feature flags
uses: actions/github-script@v3
id: eval-feature-flags
with:
script: |
const pattern = /FEATURE_(\w+)=(\w+)/g;
let results = [];
[...context.payload.comment.body.matchAll(pattern)].forEach(match => {
const config = {
name: `SUPERSET_FEATURE_${match[1]}`,
value: match[2],
};
results.push(config);
});
return results;
- name: Limit to committers
if: >
steps.eval-body.outputs.result != 'noop' &&
github.event.comment.author_association != 'MEMBER' &&
github.event.comment.author_association != 'OWNER'
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const errMsg = '@${{ github.event.comment.user.login }} Ephemeral environment creation is currently limited to committers.'
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: errMsg
})
core.setFailed(errMsg)
ephemeral_env_up:
needs: ephemeral_env_comment
if: needs.ephemeral_env_comment.outputs.slash-command == 'up'
name: Spin up an ephemeral environment
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
persist-credentials: false
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Check target image exists in ECR
id: check-image
continue-on-error: true
run: |
aws ecr describe-images \
--registry-id $(echo "${{ steps.login-ecr.outputs.registry }}" | grep -Eo "^[0-9]+") \
--repository-name superset-ci \
--image-ids imageTag=pr-${{ github.event.issue.number }}
- name: Fail on missing container image
if: steps.check-image.outcome == 'failure'
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const errMsg = '@${{ github.event.comment.user.login }} Container image not yet published for this PR. Please try again when build is complete.'
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: errMsg
})
core.setFailed(errMsg)
- name: Fill in the new image ID in the Amazon ECS task definition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@v1
with:
task-definition: .github/workflows/ecs-task-definition.json
container-name: superset-ci
image: ${{ steps.login-ecr.outputs.registry }}/superset-ci:pr-${{ github.event.issue.number }}
- name: Update env vars in the Amazon ECS task definition
run: |
cat <<< "$(jq '.containerDefinitions[0].environment += ${{ needs.ephemeral_env_comment.outputs.feature-flags }}' < ${{ steps.task-def.outputs.task-definition }})" > ${{ steps.task-def.outputs.task-definition }}
- name: Describe ECS service
id: describe-services
run: |
echo "::set-output name=active::$(aws ecs describe-services --cluster superset-ci --services pr-${{ github.event.issue.number }}-service | jq '.services[] | select(.status == "ACTIVE") | any')"
- name: Create ECS service
if: steps.describe-services.outputs.active != 'true'
id: create-service
env:
ECR_SUBNETS: subnet-0e15a5034b4121710,subnet-0e8efef4a72224974
ECR_SECURITY_GROUP: sg-092ff3a6ae0574d91
run: |
aws ecs create-service \
--cluster superset-ci \
--service-name pr-${{ github.event.issue.number }}-service \
--task-definition superset-ci \
--launch-type FARGATE \
--desired-count 1 \
--platform-version LATEST \
--network-configuration "awsvpcConfiguration={subnets=[$ECR_SUBNETS],securityGroups=[$ECR_SECURITY_GROUP],assignPublicIp=ENABLED}" \
--tags key=pr,value=${{ github.event.issue.number }} key=github_user,value=${{ github.actor }}
- name: Deploy Amazon ECS task definition
id: deploy-task
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: pr-${{ github.event.issue.number }}-service
cluster: superset-ci
wait-for-service-stability: true
wait-for-minutes: 10
- name: List tasks
id: list-tasks
run: |
echo "::set-output name=task::$(aws ecs list-tasks --cluster superset-ci --service-name pr-${{ github.event.issue.number }}-service | jq '.taskArns | first')"
- name: Get network interface
id: get-eni
run: |
echo "::set-output name=eni::$(aws ecs describe-tasks --cluster superset-ci --tasks ${{ steps.list-tasks.outputs.task }} | jq '.tasks | .[0] | .attachments | .[0] | .details | map(select(.name=="networkInterfaceId")) | .[0] | .value')"
- name: Get public IP
id: get-ip
run: |
echo "::set-output name=ip::$(aws ec2 describe-network-interfaces --network-interface-ids ${{ steps.get-eni.outputs.eni }} | jq -r '.NetworkInterfaces | first | .Association.PublicIp')"
- name: Comment (success)
if: ${{ success() }}
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: '@${{ github.event.comment.user.login }} Ephemeral environment spinning up at http://${{ steps.get-ip.outputs.ip }}:8080. Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping and startup.'
})
- name: Comment (failure)
if: ${{ failure() }}
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: '@${{ github.event.comment.user.login }} Ephemeral environment creation failed. Please check the Actions logs for details.'
})

View File

@@ -1,30 +0,0 @@
name: Tags
on:
release:
types: [published] # This makes it run only when a new released is published
jobs:
latest-release:
name: Add/update tag to new release
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Check for latest tag
id: latest-tag
run: |
source ./scripts/tag_latest_release.sh $(echo ${{ github.event.release.tag_name }}) --dry-run
- name: Run latest-tag
uses: ./.github/actions/latest-tag
if: (! ${{ steps.latest-tag.outputs.SKIP_TAG }} )
with:
description: Superset latest release
tag-name: latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

45
.github/workflows/license-check.yml vendored Normal file
View File

@@ -0,0 +1,45 @@
name: License
on:
push:
branches: [ master ]
pull_request:
jobs:
check:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v2
- name: Setup Java
uses: actions/setup-java@v1
with:
java-version: 8
- name: Generate fossa report
env:
FOSSA_API_KEY: ${{ secrets.FOSSA_API_KEY }}
run: |
set -eo pipefail
if [[ "${{github.event_name}}" != "pull_request" ]]; then
./scripts/fossa.sh
exit 0
fi
URL="https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files"
FILES=$(curl -s -X GET -G $URL | jq -r '.[] | .filename')
cat<<EOF
CHANGED FILES:
$FILES
EOF
if [[ "${FILES}" =~ (.*package*\.json|requirements\/[a-z_-]+\.txt|setup\.py) ]]; then
echo "Detected dependency changes... running fossa check"
./scripts/fossa.sh
else
echo "No dependency changes... skiping fossa check"
fi
shell: bash
- name: Run license check
run: ./scripts/check_license.sh

View File

@@ -1,99 +0,0 @@
name: Miscellaneous
on:
push:
branches-ignore:
- "dependabot/**"
pull_request:
jobs:
license_check:
name: License Check
runs-on: ubuntu-20.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@v1
with:
java-version: 8
- name: Generate fossa report
env:
FOSSA_API_KEY: ${{ secrets.FOSSA_API_KEY }}
run: |
set -eo pipefail
if [[ "${{github.event_name}}" != "pull_request" ]]; then
./scripts/fossa.sh
exit 0
fi
URL="https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files"
FILES=$(curl -s -X GET -G $URL | jq -r '.[] | .filename')
cat<<EOF
CHANGED FILES:
$FILES
EOF
if [[ "${FILES}" =~ (.*package*\.json|requirements\/[a-z_-]+\.txt|setup\.py) ]]; then
echo "Detected dependency changes... running fossa check"
./scripts/fossa.sh
else
echo "No dependency changes... skiping fossa check"
fi
shell: bash
- name: Run license check
run: ./scripts/check_license.sh
prefer_typescript:
if: github.ref == 'ref/heads/master' && github.event_name == 'pull_request'
name: Prefer Typescript
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Get changed files
id: changed
uses: ./.github/actions/file-changes-action
with:
githubToken: ${{ secrets.GITHUB_TOKEN }}
- name: Determine if a .js or .jsx file was added
id: check
run: |
js_files_added() {
jq -r '
map(
select(
endswith(".js") or endswith(".jsx")
)
) | join("\n")
' ${HOME}/files_added.json
}
echo ::set-output name=js_files_added::$(js_files_added)
- if: steps.check.outputs.js_files_added
name: Add Comment to PR
uses: ./.github/actions/comment-on-pr
continue-on-error: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
msg: |
### WARNING: Prefer TypeScript
Looks like your PR contains new `.js` or `.jsx` files:
```
${{steps.check.outputs.js_files_added}}
```
As decided in [SIP-36](https://github.com/apache/superset/issues/9101), all new frontend code should be written in TypeScript. Please convert above files to TypeScript then re-request review.

View File

@@ -12,14 +12,9 @@ jobs:
check:
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- uses: morrisoncole/pr-lint-action@v1.4.1
with:
persist-credentials: false
submodules: recursive
- uses: ./.github/actions/pr-lint-action
with:
title-regex: "^(build|chore|ci|docs|feat|fix|perf|refactor|style|test|other)(\\(.+\\))?(\\!)?:\\s.+"
title-regex: "^(build|chore|ci|docs|feat|fix|perf|refactor|style|test|other)((.+))?:\\s.+"
on-failed-regex-fail-action: true
on-failed-regex-request-changes: false
on-failed-regex-create-review: false

48
.github/workflows/prefer-typescript.yml vendored Normal file
View File

@@ -0,0 +1,48 @@
name: Prefer TypeScript
on:
pull_request:
branches:
- master
jobs:
check:
runs-on: ubuntu-latest
steps:
- name: Get changed files
id: changed
uses: trilom/file-changes-action@v1.2.4
with:
githubToken: ${{ secrets.GITHUB_TOKEN }}
- name: Determine if a .js or .jsx file was added
id: check
run: |
js_files_added() {
jq -r '
map(
select(
endswith(".js") or endswith(".jsx")
)
) | join("\n")
' ${HOME}/files_added.json
}
echo ::set-output name=js_files_added::$(js_files_added)
- if: steps.check.outputs.js_files_added
name: Add Comment to PR
uses: unsplash/comment-on-pr@master
continue-on-error: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
msg: |
### WARNING: Prefer TypeScript
Looks like your PR contains new `.js` or `.jsx` files:
```
${{steps.check.outputs.js_files_added}}
```
As decided in [SIP-36](https://github.com/apache/incubator-superset/issues/9101), all new frontend code should be written in TypeScript. Please convert above files to TypeScript then re-request review.

View File

@@ -1,86 +0,0 @@
name: release-workflow
on:
push:
branches:
- 'master'
jobs:
build:
name: Bump version and publish package(s)
runs-on: ubuntu-20.04
strategy:
matrix:
node-version: [16]
steps:
- uses: actions/checkout@v2
with:
# pulls all commits (needed for lerna / semantic release to correctly version)
fetch-depth: 0
- name: Get tags and filter trigger tags
run: |
git fetch --depth=1 origin "+refs/tags/*:refs/tags/*"
git fetch --prune --unshallow
git tag -d `git tag | grep -E '^trigger-'`
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
- name: Cache npm
uses: actions/cache@v1
with:
path: ~/.npm # npm cache files are stored in `~/.npm` on Linux/macOS
key: ${{ runner.OS }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.OS }}-node-
${{ runner.OS }}-
- name: Get npm cache directory path
id: npm-cache-dir-path
run: echo "::set-output name=dir::$(npm config get cache)"
- name: Cache npm
uses: actions/cache@v1
id: npm-cache # use this to check for `cache-hit` (`steps.npm-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.npm-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-npm-
- name: Install dependencies
working-directory: ./superset-frontend
run: npm ci
- name: Run unit tests
working-directory: ./superset-frontend
run: npm run test -- plugins packages
- name: Build packages
working-directory: ./superset-frontend
run: npm run plugins:build
- name: Configure npm and git
run: |
echo "@superset-ui:registry=https://registry.npmjs.org/" > .npmrc
echo "registry=https://registry.npmjs.org/" >> .npmrc
echo "//registry.npmjs.org/:_authToken=\${NPM_TOKEN}" >> $HOME/.npmrc 2> /dev/null
npm whoami
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
git remote set-url origin "https://${GITHUB_TOKEN}@github.com/apache-superset/superset-ui.git" > /dev/null 2>&1
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Bump version and publish package(s)
working-directory: ./superset-frontend
run: |
git tag -d `git tag | grep -E '^trigger-'`
npm run plugins:release-from-tag
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}

View File

@@ -9,33 +9,25 @@ on:
- "docs/**"
jobs:
build-deploy:
name: Build & Deploy
runs-on: ubuntu-20.04
defaults:
run:
working-directory: docs
docs:
name: build
runs-on: ubuntu-18.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: yarn install
- name: npm install
working-directory: ./docs
run: |
yarn install --check-cache
- name: yarn build
npm install
- name: lint
working-directory: ./docs
run: |
yarn build
- name: deploy docs
if: github.ref == 'refs/heads/master'
uses: ./.github/actions/github-action-push-to-another-repository
env:
API_TOKEN_GITHUB: ${{ secrets.SUPERSET_SITE_BUILD }}
with:
source-directory: './docs/build'
destination-github-username: 'apache'
destination-repository-name: 'superset-site'
target-branch: 'asf-site'
commit-message: "deploying docs: ${{ github.event.head_commit.message }} (apache/superset@${{ github.sha }})"
user-email: dev@superset.apache.org
npm run lint
- name: gatsby build
working-directory: ./docs
run: |
npm run build

View File

@@ -1,18 +1,10 @@
name: E2E
on:
push:
branches-ignore:
- "dependabot/**/docs/**"
paths-ignore:
- "docs/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
on: [push, pull_request_target]
jobs:
cypress-matrix:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
strategy:
# when one test fails, DO NOT cancel the other
# containers, because this will kill Cypress processes
@@ -25,7 +17,7 @@ jobs:
env:
FLASK_ENV: development
ENABLE_REACT_CRUD_VIEWS: true
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET_CONFIG: tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
@@ -43,78 +35,63 @@ jobs:
ports:
- 16379:6379
steps:
- name: "Checkout (pull) ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- name: Checkout code (push)
if: github.event_name == 'push'
with:
persist-credentials: false
submodules: recursive
- name: "Checkout (pull_request) ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Checkout code (pull_request)
if: github.event_name == 'pull_request' || github.event_name == 'pull_request_target'
uses: actions/checkout@v2
with:
ref: "refs/pull/${{ github.event.number }}/merge"
persist-credentials: false
submodules: recursive
- name: Check if python or frontend changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python frontend
ref: 'refs/pull/${{ github.event.number }}/merge'
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: "3.8"
python-version: "3.7"
- name: OS dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: apt-get-install
run: |
apt-get-install
- name: Install python dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
pip-upgrade
pip install -r requirements/testing.txt
- name: Setup postgres
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: setup-postgres
run: |
setup-postgres
- name: Import test data
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: testdata
- name: Setup Node.js
uses: actions/setup-node@v2
with:
node-version: '16'
run: |
testdata
- name: Install npm dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: npm-install
run: |
npm-install
- name: Build javascript packages
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: build-instrumented-assets
run: |
build-instrumented-assets
- name: Install cypress
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: cypress-install
run: |
cypress-install
- name: Run Cypress
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
env:
CYPRESS_BROWSER: ${{ matrix.browser }}
CYPRESS_KEY: YjljODE2MzAtODcwOC00NTA3LWE4NmMtMTU3YmFmMjIzOTRhCg==
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
with:
run: cypress-run-all
- name: Upload Artifacts
@@ -123,3 +100,12 @@ jobs:
with:
name: screenshots
path: ${{ github.workspace }}/superset-frontend/cypress-base/cypress/screenshots
Cypress:
if: ${{ always() }}
name: Cypress (chrome)
runs-on: ubuntu-18.04
needs: cypress-matrix
steps:
- name: Check build matrix status
if: ${{ needs.cypress-matrix.result != 'success' }}
run: exit 1

View File

@@ -1,67 +1,31 @@
name: Frontend
on:
push:
branches-ignore:
- "dependabot/**/docs/**"
- "dependabot/**/cypress-base/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
on: [push, pull_request]
jobs:
frontend-build:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
build:
runs-on: ubuntu-18.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Check npm lock file version
run: ./scripts/ci_check_npm_lock_version.sh ./superset-frontend/package-lock.json
- name: Check if frontend changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh frontend
- name: Setup Node.js
if: steps.check.outcome == 'failure'
uses: actions/setup-node@v2
with:
node-version: "16"
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: npm-install
- name: lint
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: |
npm run lint
npm run prettier-check
- name: Build plugins packages
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: npm run plugins:build
- name: Build plugins Storybook
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: npm run plugins:build-storybook
- name: unit tests
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: |
npm run test -- --coverage
# todo: remove this step when fix generator as a project in root jest.config.js
- name: generator-superset unit tests
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend/packages/generator-superset
run: npx jest
- name: Upload code coverage
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: ../.github/workflows/codecov.sh -c -F javascript
run: |
bash <(curl -s https://codecov.io/bash) -cF javascript

View File

@@ -1,46 +0,0 @@
name: Lint and Test Charts
on:
pull_request:
types: [opened, edited, reopened, synchronize]
jobs:
lint-test:
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
fetch-depth: 0
- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: v3.5.4
- uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Set up chart-testing
uses: ./.github/actions/chart-testing-action
- name: Run chart-testing (list-changed)
id: list-changed
run: |
changed=$(ct list-changed --print-config)
if [[ -n "$changed" ]]; then
echo "::set-output name=changed::true"
fi
env:
CT_CHART_DIRS: helm
CT_SINCE: HEAD
- name: Run chart-testing (lint)
run: ct lint --print-config
env:
CT_CHART_DIRS: helm
CT_LINT_CONF: lintconf.yaml
CT_SINCE: HEAD

View File

@@ -1,37 +0,0 @@
name: Release Charts
on:
push:
branches:
- 'master'
paths:
- 'helm/**'
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
fetch-depth: 0
- name: Configure Git
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Install Helm
uses: azure/setup-helm@v1
with:
version: v3.5.4
- name: Run chart-releaser
uses: ./.github/actions/chart-releaser-action
with:
charts_dir: helm
env:
CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
CR_RELEASE_NAME_TEMPLATE: "superset-helm-chart-{{ .Version }}"

View File

@@ -1,202 +0,0 @@
# Python integration tests
name: Python-Integration
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
jobs:
test-mysql:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: [3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: |
mysql+mysqldb://superset:superset@127.0.0.1:13306/superset?charset=utf8mb4&binary_prefix=true
services:
mysql:
image: mysql:5.7
env:
MYSQL_ROOT_PASSWORD: root
ports:
- 13306:3306
redis:
image: redis:5-alpine
options: --entrypoint redis-server
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-mysql
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python integration tests (MySQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh
- name: Upload code coverage
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F mysql
test-postgres:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: [3.8, 3.9]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
services:
postgres:
image: postgres:10-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15432:5432
redis:
image: redis:5-alpine
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python integration tests (PostgreSQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh
- name: Upload code coverage
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F postgres
test-sqlite:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: [3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: |
sqlite:///${{ github.workspace }}/.temp/unittest.db
services:
redis:
image: redis:5-alpine
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
mkdir ${{ github.workspace }}/.temp
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python integration tests (SQLite)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh
- name: Upload code coverage
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F sqlite

View File

@@ -1,111 +1,84 @@
# Python Misc unit tests
name: Python Misc
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
on: [push, pull_request]
jobs:
python-lint:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
lint:
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.8]
python-version: [3.7]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
- name: pylint
if: steps.check.outcome == 'failure'
# `-j 0` run Pylint in parallel
run: pylint -j 0 superset
pre-commit:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.8]
python-version: [3.7]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
persist-credentials: false
submodules: recursive
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
requirements/base.txt
requirements/integration.txt
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/base.txt
pip install -r requirements/integration.txt
- name: pre-commit
run: pre-commit run --all-files
babel-extract:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.8]
python-version: [3.7]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
persist-credentials: false
submodules: recursive
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/base.txt'
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/base.txt
- name: Test babel extraction
run: flask fab babel-extract --target superset/translations --output superset/translations/messages.pot --config superset/translations/babel.cfg -k _,__,t,tn,tct

View File

@@ -0,0 +1,55 @@
# Python MySQL unit tests
name: Python MySQL
on: [push, pull_request]
jobs:
test-mysql:
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.7]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: |
mysql+mysqldb://superset:superset@127.0.0.1:13306/superset?charset=utf8mb4&binary_prefix=true
services:
mysql:
image: mysql:5.7
env:
MYSQL_ROOT_PASSWORD: root
ports:
- 13306:3306
redis:
image: redis:5-alpine
options: --entrypoint redis-server
ports:
- 16379:6379
steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install -r requirements/testing.txt
setup-mysql
- name: Run celery
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (MySQL)
run: |
./scripts/python_tests.sh
- name: Upload code coverage
run: |
bash <(curl -s https://codecov.io/bash) -cF python

View File

@@ -0,0 +1,56 @@
# Python Postgres unit tests
name: Python Postgres
on: [push, pull_request]
jobs:
test-postgres:
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.7, 3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
services:
postgres:
image: postgres:10-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15432:5432
redis:
image: redis:5-alpine
ports:
- 16379:6379
steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
run: |
./scripts/python_tests.sh
- name: Upload code coverage
run: |
bash <(curl -s https://codecov.io/bash) -cF python

View File

@@ -1,23 +1,18 @@
# Python Presto/Hive unit tests
name: Python Presto/Hive
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
on: [push, pull_request]
jobs:
test-postgres-presto:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
strategy:
matrix:
# run unit tests in multiple version just for fun
python-version: [3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
SUPERSET__SQLALCHEMY_EXAMPLES_URI: presto://localhost:15433/memory/default
@@ -32,7 +27,7 @@ jobs:
# GitHub action runner's default installations
- 15432:5432
presto:
image: starburstdata/presto:350-e.6
image: prestosql/presto:339
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
@@ -45,56 +40,41 @@ jobs:
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
access_token: ${{ github.token }}
- uses: actions/checkout@v2
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh -m 'chart_data_flow or sql_json_flow'
./scripts/python_tests.sh
- name: Upload code coverage
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F presto
bash <(curl -s https://codecov.io/bash) -cF python
test-postgres-hive:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.8]
# run unit tests in multiple version just for fun
python-version: [3.7, 3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
SUPERSET__SQLALCHEMY_EXAMPLES_URI: hive://localhost:10000/default
@@ -114,52 +94,34 @@ jobs:
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
access_token: ${{ github.token }}
- uses: actions/checkout@v2
- name: Create csv upload directory
if: steps.check.outcome == 'failure'
run: sudo mkdir -p /tmp/.superset/uploads
- name: Give write access to the csv upload directory
if: steps.check.outcome == 'failure'
run: sudo chown -R $USER:$USER /tmp/.superset
- name: Start hadoop and hive
if: steps.check.outcome == 'failure'
run: docker-compose -f scripts/databases/hive/docker-compose.yml up -d
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh -m 'chart_data_flow or sql_json_flow'
./scripts/python_tests.sh
- name: Upload code coverage
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F hive
bash <(curl -s https://codecov.io/bash) -cF python

View File

@@ -0,0 +1,48 @@
# Python Sqlite unit tests
name: Python Sqlite
on: [push, pull_request]
jobs:
test-sqlite:
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.7]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: |
sqlite:///${{ github.workspace }}/.temp/unittest.db
services:
redis:
image: redis:5-alpine
ports:
- 16379:6379
steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install -r requirements/testing.txt
mkdir ${{ github.workspace }}/.temp
- name: Run celery
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (SQLite)
run: |
./scripts/python_tests.sh
- name: Upload code coverage
run: |
bash <(curl -s https://codecov.io/bash) -cF python

View File

@@ -1,58 +0,0 @@
# Python unit tests
name: Python-Unit
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
jobs:
unit-tests:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: [3.8, 3.9]
env:
PYTHONPATH: ${{ github.workspace }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
# TODO: separated requiermentes.txt file just for unit tests
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
mkdir ${{ github.workspace }}/.temp
- name: Python unit tests
if: steps.check.outcome == 'failure'
run: |
pytest --durations=0 ./tests/common ./tests/unit_tests --cache-clear
- name: Upload code coverage
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F unit

View File

@@ -1,28 +1,19 @@
name: Translations
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
on: [push, pull_request]
jobs:
frontend-check:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
with:
persist-credentials: false
submodules: recursive
- name: Setup Node.js
uses: actions/setup-node@v2
with:
node-version: '16'
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: npm-install
- name: lint
@@ -31,27 +22,30 @@ jobs:
npm run check-translation
babel-extract:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
runs-on: ubuntu-18.04
strategy:
matrix:
python-version: [3.8]
python-version: [3.7]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.6.0
with:
persist-credentials: false
submodules: recursive
access_token: ${{ github.token }}
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install -r requirements/base.txt
- name: Test babel extraction
run: ./scripts/babel_update.sh
run: |
flask fab babel-extract --target superset/translations \
--output superset/translations/messages.pot \
--config superset/translations/babel.cfg -k _,__,t,tn,tct

View File

@@ -1,33 +0,0 @@
name: WebSocket server
on:
push:
paths:
- "superset-websocket/**"
pull_request:
paths:
- "superset-websocket/**"
jobs:
app-checks:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
- name: Install dependencies
working-directory: ./superset-websocket
run: npm install
- name: lint
working-directory: ./superset-websocket
run: npm run lint
- name: prettier
working-directory: ./superset-websocket
run: npm run prettier-check
- name: unit tests
working-directory: ./superset-websocket
run: npm run test
- name: build
working-directory: ./superset-websocket
run: npm run build

61
.github/workflows/test-hive.yml vendored Normal file
View File

@@ -0,0 +1,61 @@
name: Hive
on: [push, pull_request]
jobs:
test-postgres-hive:
runs-on: ubuntu-18.04
strategy:
matrix:
# run unit tests in multiple version just for fun
python-version: [3.7, 3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI:
postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
SUPERSET__SQLALCHEMY_EXAMPLES_URI: hive://localhost:10000/default
UPLOAD_FOLDER: /tmp/.superset/uploads/
services:
postgres:
image: postgres:10-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15432:5432
redis:
image: redis:5-alpine
ports:
- 16379:6379
steps:
- uses: actions/checkout@v2
- name: Create csv upload directory
run: sudo mkdir -p /tmp/.superset/uploads
- name: Give write access to the csv upload directory
run: sudo chown -R $USER:$USER /tmp/.superset
- name: Start hadoop and hive
run: docker-compose -f scripts/databases/hive/docker-compose.yml up -d
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
run: |
./scripts/python_tests.sh
- name: Upload code coverage
run: |
bash <(curl -s https://codecov.io/bash) -cF python

64
.github/workflows/test-presto.yml vendored Normal file
View File

@@ -0,0 +1,64 @@
name: Presto
on: [push, pull_request]
jobs:
test-postgres-presto:
runs-on: ubuntu-18.04
strategy:
matrix:
# run unit tests in multiple version just for fun
python-version: [3.8]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI:
postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
SUPERSET__SQLALCHEMY_EXAMPLES_URI:
presto://localhost:15433/memory/default
services:
postgres:
image: postgres:10-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15432:5432
presto:
image: prestosql/presto:339
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15433:8080
redis:
image: redis:5-alpine
ports:
- 16379:6379
steps:
- uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: apache-superset/cached-dependencies@b90713b
with:
run: |
apt-get-install
pip-upgrade
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
run: |
./scripts/python_tests.sh
- name: Upload code coverage
run: |
bash <(curl -s https://codecov.io/bash) -cF python

28
.gitignore vendored
View File

@@ -14,6 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
*.ipynb
*.bak
*.db
*.pyc
@@ -21,8 +22,6 @@
*.swp
__pycache__
.local
.cache
.bento*
.cache-loader
.coverage
@@ -53,45 +52,25 @@ envpy3
env36
local_config.py
/superset_config.py
/superset_text.yml
superset.egg-info/
superset/bin/supersetc
tmp
rat-results.txt
superset/app/
# Node.js, webpack artifacts, storybook
# Node.js, webpack artifacts
*.entry.js
*.js.map
node_modules
npm-debug.log*
superset/static/assets
superset/static/version_info.json
superset-frontend/**/esm/*
superset-frontend/**/lib/*
superset-frontend/**/storybook-static/*
yarn-error.log
*.map
*.min.js
test-changelog.md
*.tsbuildinfo
# Ignore package-lock in packages
plugins/*/package-lock.json
packages/*/package-lock.json
# For country map geojson conversion script
.ipynb_checkpoints/
scripts/*.zip
# IntelliJ
*.iml
venv
@eaDir/
# PyCharm
.run
# Test data
celery_results.sqlite
celerybeat-schedule
@@ -102,11 +81,8 @@ ghostdriver.log
testCSV.csv
.terser-plugin-cache/
apache-superset-*.tar.gz*
release.json
# Translation binaries
messages.mo
docker/requirements-local.txt
cache/

41
.gitmodules vendored
View File

@@ -1,41 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
[submodule ".github/actions/latest-tag"]
path = .github/actions/latest-tag
url = https://github.com/EndBug/latest-tag
[submodule ".github/actions/pr-lint-action"]
path = .github/actions/pr-lint-action
url = https://github.com/morrisoncole/pr-lint-action
[submodule ".github/actions/file-changes-action"]
path = .github/actions/file-changes-action
url = https://github.com/trilom/file-changes-action
[submodule ".github/actions/cached-dependencies"]
path = .github/actions/cached-dependencies
url = https://github.com/apache-superset/cached-dependencies
[submodule ".github/actions/comment-on-pr"]
path = .github/actions/comment-on-pr
url = https://github.com/unsplash/comment-on-pr
[submodule ".github/actions/chart-testing-action"]
path = .github/actions/chart-testing-action
url = https://github.com/helm/chart-testing-action
[submodule ".github/actions/chart-releaser-action"]
path = .github/actions/chart-releaser-action
url = https://github.com/helm/chart-releaser-action
[submodule ".github/actions/github-action-push-to-another-repository"]
path = .github/actions/github-action-push-to-another-repository
url = https://github.com/cpina/github-action-push-to-another-repository

View File

@@ -15,17 +15,20 @@
# limitations under the License.
#
repos:
- repo: https://github.com/PyCQA/isort
rev: 5.9.3
- repo: https://github.com/asottile/seed-isort-config
rev: v1.9.3
hooks:
- id: seed-isort-config
- repo: https://github.com/pre-commit/mirrors-isort
rev: v4.3.21
hooks:
- id: isort
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.941
rev: v0.790
hooks:
- id: mypy
additional_dependencies: [types-all]
- repo: https://github.com/peterdemin/pip-compile-multi
rev: v2.4.1
rev: v1.5.8
hooks:
- id: pip-compile-multi-verify
- repo: https://github.com/pre-commit/pre-commit-hooks
@@ -33,7 +36,6 @@ repos:
hooks:
- id: check-docstring-first
- id: check-added-large-files
exclude: \.(geojson)$
- id: check-yaml
exclude: ^helm/superset/templates/
- id: debug-statements
@@ -41,13 +43,7 @@ repos:
- id: trailing-whitespace
args: ["--markdown-linebreak-ext=md"]
- repo: https://github.com/psf/black
rev: 22.3.0
rev: 19.10b0
hooks:
- id: black
language_version: python3
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v2.4.1 # Use the sha or tag you want to point at
hooks:
- id: prettier
args: ['--ignore-path=./superset-frontend/.prettierignore']
files: 'superset-frontend'

View File

@@ -70,8 +70,7 @@ confidence=
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=
useless-suppression,
#enable=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
@@ -82,12 +81,8 @@ enable=
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=
missing-docstring,
duplicate-code,
unspecified-encoding,
# re-enable once this no longer raises false positives
too-many-instance-attributes
disable=long-builtin,dict-view-method,intern-builtin,suppressed-message,no-absolute-import,unpacking-in-except,apply-builtin,delslice-method,indexing-exception,old-raise-syntax,print-statement,cmp-builtin,reduce-builtin,useless-suppression,coerce-method,input-builtin,cmp-method,raw_input-builtin,nonzero-method,backtick,basestring-builtin,setslice-method,reload-builtin,oct-method,map-builtin-not-iterating,execfile-builtin,old-octal-literal,zip-builtin-not-iterating,buffer-builtin,getslice-method,metaclass-assignment,xrange-builtin,long-suffix,round-builtin,range-builtin-not-iterating,next-method-called,parameter-unpacking,unicode-builtin,unichr-builtin,import-star-module-level,raising-string,filter-builtin-not-iterating,using-cmp-argument,coerce-builtin,file-builtin,old-division,hex-method,missing-docstring,too-many-lines,ungrouped-imports,import-outside-toplevel,raise-missing-from,super-with-arguments,bad-option-value
[REPORTS]
@@ -120,7 +115,7 @@ evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / stateme
[BASIC]
# Good variable names which should always be accepted, separated by a comma
good-names=_,df,ex,f,i,id,j,k,l,o,pk,Run,ts,v,x,y
good-names=_,df,ex,f,i,id,j,k,l,o,pk,Run,ts,v,x
# Bad variable names which should always be refused, separated by a comma
bad-names=fd,foo,bar,baz,toto,tutu,tata
@@ -214,7 +209,7 @@ max-nested-blocks=5
[FORMAT]
# Maximum number of characters on a single line.
max-line-length=90
max-line-length=88
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
@@ -313,7 +308,7 @@ generated-members=
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
contextmanager-decorators=contextlib.contextmanager,contextlib2.contextmanager
[VARIABLES]
@@ -367,10 +362,10 @@ ignored-argument-names=_.*
max-locals=15
# Maximum number of return / yield for function / method body
max-returns=10
max-returns=6
# Maximum number of branch for function / method body
max-branches=15
max-branches=12
# Maximum number of statements in function / method body
max-statements=50
@@ -379,7 +374,7 @@ max-statements=50
max-parents=7
# Maximum number of attributes for a class (see R0902).
max-attributes=8
max-attributes=7
# Minimum number of public methods for a class (see R0903).
min-public-methods=2

View File

@@ -35,9 +35,8 @@ apache_superset.egg-info
.*csv
# Generated doc files
env/*
docs/README.md
docs/.htaccess*
docs-v2/.htaccess*
.nojekyll
_build/*
_static/*
.buildinfo
@@ -49,16 +48,3 @@ vendor/*
# github configuration
.github/*
.*mdx
# skip license check in superset-ui
tmp/*
lib/*
esm/*
tsconfig.tsbuildinfo
.*ipynb
.*yml
.*iml
.esprintrc
.prettierignore
generator-superset/*
temporary_superset_ui/*

File diff suppressed because it is too large Load Diff

View File

@@ -106,12 +106,12 @@ This statement thanks the following, on which it draws for content and inspirati
# Slack Community Guidelines
If you decide to join the [Community Slack](https://join.slack.com/t/apache-superset/shared_invite/zt-16jvzmoi8-sI7jKWp~xc2zYRe~NqiY9Q), please adhere to the following rules:
If you decide to join the [Community Slack](https://join.slack.com/t/apache-superset/shared_invite/enQtNDMxMDY5NjM4MDU0LWJmOTcxYjlhZTRhYmEyYTMzOWYxOWEwMjcwZDZiNWRiNDY2NDUwNzcwMDFhNzE1ZmMxZTZlZWY0ZTQ2MzMyNTU), please adhere to the following rules:
**1. Treat everyone in the community with respect.**
- We strive to make this community a warm place for people from all industries, use cases, geographies, and backgrounds. Harassment of any kind is not acceptable and wont be tolerated.
- Please follow the guidelines as outlined in the Superset Community [code of conduct here](https://github.com/apache/superset/blob/master/CODE_OF_CONDUCT.md).
- Please follow the guidelines as outlined in the Superset Community [code of conduct here](https://github.com/apache/incubator-superset/blob/master/CODE_OF_CONDUCT.md).
**2. Use the right channel.**
@@ -119,7 +119,7 @@ If you decide to join the [Community Slack](https://join.slack.com/t/apache-supe
**3. Ask thoughtful questions.**
- Were all here to help each other out. The best way to get help is by investing effort into your questions. First check and see if your question is answered in [the Superset documentation](https://superset.apache.org/faq.html) or on [Stack Overflow](https://stackoverflow.com/search?q=apache+superset). You can also check [Github issues](https://github.com/apache/superset/issues) to see if your question or feature request has been submitted before. Then, use Slack search to see if your question has already been asked and answered in the past. If you still feel the need to ask a question, make sure you include:
- Were all here to help each other out. The best way to get help is by investing effort into your questions. First check and see if your question is answered in [the Superset documentation](https://superset.incubator.apache.org/faq.html) or on [Stack Overflow](https://stackoverflow.com/search?q=apache+superset). You can also check [Github issues](https://github.com/apache/incubator-superset/issues) to see if your question or feature request has been submitted before. Then, use Slack search to see if your question has already been asked and answered in the past. If you still feel the need to ask a question, make sure you include:
- The steps youve already taken
- Relevant details presented cleanly (text stacktraces, formatted markdown, or screenshots. Please dont paste large blocks of code unformatted or post photos of your screen from your phone)

File diff suppressed because it is too large Load Diff

11
DISCLAIMER Normal file
View File

@@ -0,0 +1,11 @@
DISCLAIMER
Apache Superset (incubating) is an effort undergoing incubation at the Apache
Software Foundation (ASF), sponsored by the Apache Incubator PMC.
Incubation is required of all newly accepted projects until a further review
indicates that the infrastructure, communications, and decision making process
have stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the completeness or
stability of the code, it does indicate that the project has yet to be fully
endorsed by the ASF.

View File

@@ -18,7 +18,7 @@
######################################################################
# PY stage that simply does a pip install on our requirements
######################################################################
ARG PY_VER=3.8.13
ARG PY_VER=3.7.9
FROM python:${PY_VER} AS superset-py
RUN mkdir /app \
@@ -45,10 +45,7 @@ RUN cd /app \
######################################################################
# Node stage to deal with static asset construction
######################################################################
FROM node:16 AS superset-node
ARG NPM_VER=7
RUN npm install -g npm@${NPM_VER}
FROM node:12 AS superset-node
ARG NPM_BUILD_CMD="build"
ENV BUILD_CMD=${NPM_BUILD_CMD}
@@ -57,12 +54,14 @@ ENV BUILD_CMD=${NPM_BUILD_CMD}
RUN mkdir -p /app/superset-frontend
RUN mkdir -p /app/superset/assets
COPY ./docker/frontend-mem-nag.sh /
COPY ./superset-frontend /app/superset-frontend
COPY ./superset-frontend/package* /app/superset-frontend/
RUN /frontend-mem-nag.sh \
&& cd /app/superset-frontend \
&& npm ci
# This seems to be the most expensive step
# Next, copy in the rest and let webpack do its thing
COPY ./superset-frontend /app/superset-frontend
# This is BY FAR the most expensive step (thanks Terser!)
RUN cd /app/superset-frontend \
&& npm run ${BUILD_CMD} \
&& rm -rf node_modules
@@ -71,7 +70,7 @@ RUN cd /app/superset-frontend \
######################################################################
# Final lean image...
######################################################################
ARG PY_VER=3.8.13
ARG PY_VER=3.7.9
FROM python:${PY_VER} AS lean
ENV LANG=C.UTF-8 \
@@ -80,20 +79,18 @@ ENV LANG=C.UTF-8 \
FLASK_APP="superset.app:create_app()" \
PYTHONPATH="/app/pythonpath" \
SUPERSET_HOME="/app/superset_home" \
SUPERSET_PORT=8088
SUPERSET_PORT=8080
RUN mkdir -p ${PYTHONPATH} \
&& useradd --user-group -d ${SUPERSET_HOME} -m --no-log-init --shell /bin/bash superset \
RUN useradd --user-group --no-create-home --no-log-init --shell /bin/bash superset \
&& mkdir -p ${SUPERSET_HOME} ${PYTHONPATH} \
&& apt-get update -y \
&& apt-get install -y --no-install-recommends \
build-essential \
default-libmysqlclient-dev \
libsasl2-modules-gssapi-mit \
libpq-dev \
libecpg-dev \
&& rm -rf /var/lib/apt/lists/*
COPY --from=superset-py /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/
COPY --from=superset-py /usr/local/lib/python3.7/site-packages/ /usr/local/lib/python3.7/site-packages/
# Copying site-packages doesn't move the CLIs, so let's copy them one by one
COPY --from=superset-py /usr/local/bin/gunicorn /usr/local/bin/celery /usr/local/bin/flask /usr/bin/
COPY --from=superset-node /app/superset/static/assets /app/superset/static/assets
@@ -104,12 +101,9 @@ COPY superset /app/superset
COPY setup.py MANIFEST.in README.md /app/
RUN cd /app \
&& chown -R superset:superset * \
&& pip install -e . \
&& flask fab babel-compile --target superset/translations
&& pip install -e .
COPY ./docker/run-server.sh /usr/bin/
RUN chmod a+x /usr/bin/run-server.sh
COPY ./docker/docker-entrypoint.sh /usr/bin/
WORKDIR /app
@@ -119,49 +113,18 @@ HEALTHCHECK CMD curl -f "http://localhost:$SUPERSET_PORT/health"
EXPOSE ${SUPERSET_PORT}
CMD /usr/bin/run-server.sh
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
######################################################################
# Dev image...
######################################################################
FROM lean AS dev
ARG GECKODRIVER_VERSION=v0.28.0
ARG FIREFOX_VERSION=88.0
COPY ./requirements/*.txt ./docker/requirements-*.txt/ /app/requirements/
USER root
RUN apt-get update -y \
&& apt-get install -y --no-install-recommends libnss3 libdbus-glib-1-2 libgtk-3-0 libx11-xcb1
# Install GeckoDriver WebDriver
RUN wget https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VERSION}/geckodriver-${GECKODRIVER_VERSION}-linux64.tar.gz -O /tmp/geckodriver.tar.gz && \
tar xvfz /tmp/geckodriver.tar.gz -C /tmp && \
mv /tmp/geckodriver /usr/local/bin/geckodriver && \
rm /tmp/geckodriver.tar.gz
# Install Firefox
RUN wget https://download-installer.cdn.mozilla.net/pub/firefox/releases/${FIREFOX_VERSION}/linux-x86_64/en-US/firefox-${FIREFOX_VERSION}.tar.bz2 -O /opt/firefox.tar.bz2 && \
tar xvf /opt/firefox.tar.bz2 -C /opt && \
ln -s /opt/firefox/firefox /usr/local/bin/firefox
# Cache everything for dev purposes...
RUN cd /app \
&& pip install --no-cache -r requirements/docker.txt \
&& pip install --no-cache -r requirements/requirements-local.txt || true
USER superset
######################################################################
# CI image...
######################################################################
FROM lean AS ci
COPY --chown=superset ./docker/docker-bootstrap.sh /app/docker/
COPY --chown=superset ./docker/docker-init.sh /app/docker/
COPY --chown=superset ./docker/docker-ci.sh /app/docker/
RUN chmod a+x /app/docker/*.sh
CMD /app/docker/docker-ci.sh

View File

@@ -16,8 +16,8 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# INSTALL / BUILD instructions for Apache Superset
# INSTALL / BUILD instructions for Apache Superset (incubating)
At this time, the docker file at RELEASING/Dockerfile.from_local_tarball
At this time, the docker file at RELEASING/Dockerfile.from_tarball
constitutes the recipe on how to get to a working release from a source
release tarball.

View File

@@ -17,11 +17,11 @@ specific language governing permissions and limitations
under the License.
-->
## Superset Users in the Wild
## Superset users in the wild
Here's a list of organizations, broken down into broad industry categories, that have taken the time to send a PR to let
the world know they are using Apache Superset. If you are a user and want to be recognized,
all you have to do is file a simple PR [like this one](https://github.com/apache/superset/pull/10122). If you think
all you have to do is file a simple PR [like this one](https://github.com/apache/incubator-superset/pull/10122). If you think
the categorization is inaccurate, please file a PR with your correction as well.
Join our growing community!
@@ -32,12 +32,10 @@ Join our growing community!
- [Hostnfly](https://www.hostnfly.com/) [@alexisrosuel]
- [Lime](https://www.limebike.com/) [@cxmcc]
- [Lyft](https://www.lyft.com/)
- [Ontruck](https://www.ontruck.com/)
### Financial Services
- [Aktia Bank plc](https://www.aktia.com) [@villebro]
- [American Express](https://www.americanexpress.com) [@TheLastSultan]
- [Cape Crypto](https://capecrypto.com)
- [Capital Service S.A.](http://capitalservice.pl) [@pkonarzewski]
- [Clark.de](http://clark.de/)
- [Xendit](http://xendit.co/) [@LieAlbertTriAdrian]
@@ -61,12 +59,10 @@ Join our growing community!
- [Tails.com](https://tails.com) [@alanmcruickshank]
- [THE ICONIC](http://theiconic.com.au/) [@ksaagariconic]
- [Utair](https://www.utair.ru) [@utair-digital]
- [VkusVill](https://www.vkusvill.ru) [@ETselikov]
- [Zalando](https://www.zalando.com) [@dmigo]
- [Zalora](https://www.zalora.com) [@ksaagariconic]
### Enterprise Technology
- [A3Data](https://a3data.com.br) [@neylsoncrepalde]
- [Apollo GraphQL](https://www.apollographql.com/) [@evans]
- [Astronomer](https://www.astronomer.io) [@ryw]
- [Avesta Technologies](https://avestatechnologies.com/) [@TheRum]
@@ -80,11 +76,9 @@ Join our growing community!
- [FBK - ICT center](http://ict.fbk.eu)
- [GfK Data Lab](http://datalab.gfk.com) [@mherr]
- [GrowthSimple](https://growthsimple.ai/)
- [Hydrolix](https://www.hydrolix.io/)
- [Intercom](https://www.intercom.com/) [@kate-gallo]
- [jampp](https://jampp.com/)
- [Konfío](http://konfio.mx) [@uis-rodriguez]
- [mishmash io](https://mishmash.io/)[@mishmash-io]
- [Myra Labs](http://www.myralabs.com/) [@viksit]
- [Nielsen](http://www.nielsen.com/) [@amitNielsen]
- [Ona](https://ona.io) [@pld]
@@ -98,23 +92,22 @@ Join our growing community!
- [Showmax](https://tech.showmax.com) [@bobek]
- [source{d}](https://www.sourced.tech) [@marnovo]
- [Steamroot](https://streamroot.io/)
- [TechAudit](https://www.techaudit.info) [@ETselikov]
- [Tenable](https://www.tenable.com) [@dflionis]
- [timbr.ai](https://timbr.ai/) [@semantiDan]
- [Tobii](http://www.tobii.com/) [@dwa]
- [Tooploox](https://www.tooploox.com/) [@jakubczaplicki]
- [Unvired](https://unvired.com)[@srinisubramanian]
- [Whale](http://whale.im)
- [Windsor.ai](https://www.windsor.ai/) [@octaviancorlade]
- [Zeta](https://www.zeta.tech/) [@shaikidris]
### Entertainment
- [6play](https://www.6play.fr) [@CoryChaplin]
- [bilibili](https://www.bilibili.com) [@Moinheart]
- [Douban](https://www.douban.com/) [@luchuan]
- [Kuaishou](https://www.kuaishou.com/) [@zhaoyu89730105]
- [Netflix](https://www.netflix.com/)
- [TME QQMUSIC/WESING](https://www.tencentmusic.com/)[@shenyuanli,@marklaw]
- [TME QQMUSIC/WESING](https://www.tencentmusic.com/)
- [Xite](https://xite.com/) [@shashankkoppar]
- [Zaihang](http://www.zaih.com/)
@@ -122,7 +115,6 @@ Join our growing community!
- [Brilliant.org](https://brilliant.org/)
- [Udemy](https://www.udemy.com/) [@sungjuly]
- [VIPKID](https://www.vipkid.com.cn/) [@illpanda]
- [Sunbird](https://www.sunbird.org/) [@eksteporg]
### Energy
- [Airboxlab](https://foobot.io) [@antoine-galataud]
@@ -132,18 +124,13 @@ Join our growing community!
### Healthcare
- [Amino](https://amino.com) [@shkr]
- [Care](https://www.getcare.io/)[@alandao2021]
- [Living Goods](https://www.livinggoods.org) [@chelule]
- [Maieutical Labs](https://maieuticallabs.it) [@xrmx]
- [QPID Health](http://www.qpidhealth.com/)
- [TrustMedis](https://trustmedis.com) [@famasya]
- [WeSure](https://www.wesure.cn/)
### HR / Staffing
- [Symmetrics](https://www.symmetrics.fyi)
### Others
- [Dropbox](https://www.dropbox.com/) [@bkyryliuk]
- [Grassroot](https://www.grassrootinstitute.org/)
- [komoot](https://www.komoot.com/) [@christophlingg]
- [Let's Roam](https://www.letsroam.com/)

111
Makefile
View File

@@ -1,111 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Python version installed; we need 3.7-3.9
PYTHON=`command -v python3.9 || command -v python3.8 || command -v python3.7`
.PHONY: install superset venv pre-commit
install: superset pre-commit
superset:
# Install external dependencies
pip install -r requirements/local.txt
# Install Superset in editable (development) mode
pip install -e .
# Create an admin user in your metadata database
superset fab create-admin \
--username admin \
--firstname "Admin I."\
--lastname Strator \
--email admin@superset.io \
--password general
# Initialize the database
superset db upgrade
# Create default roles and permissions
superset init
# Load some data to play with
superset load-examples
# Install node packages
cd superset-frontend; npm install
update: update-py update-js
update-py:
# Install external dependencies
pip install -r requirements/local.txt
# Install Superset in editable (development) mode
pip install -e .
# Initialize the database
superset db upgrade
# Create default roles and permissions
superset init
update-js:
# Install js packages
cd superset-frontend; npm ci
venv:
# Create a virtual environment and activate it (recommended)
if ! [ -x "${PYTHON}" ]; then echo "You need Python 3.7, 3.8 or 3.9 installed"; exit 1; fi
test -d venv || ${PYTHON} -m venv venv # setup a python3 virtualenv
. venv/bin/activate
activate:
. venv/bin/activate
pre-commit:
# setup pre commit dependencies
pip3 install -r requirements/integration.txt
pre-commit install
format: py-format js-format
py-format: pre-commit
pre-commit run black --all-files
py-lint: pre-commit
pylint -j 0 superset
js-format:
cd superset-frontend; npm run prettier
flask-app:
flask run -p 8088 --with-threads --reload --debugger
node-app:
cd superset-frontend; npm run dev-server
build-cypress:
cd superset-frontend; npm run build-instrumented
cd superset-frontend/cypress-base; npm install
open-cypress:
if ! [ $(port) ]; then cd superset-frontend/cypress-base; CYPRESS_BASE_URL=http://localhost:9000 npm run cypress open; fi
cd superset-frontend/cypress-base; CYPRESS_BASE_URL=http://localhost:$(port) npm run cypress open
admin-user:
superset fab create-admin

4
NOTICE
View File

@@ -1,5 +1,5 @@
Apache Superset
Copyright 2016-2021 The Apache Software Foundation
Apache Superset (incubating)
Copyright 2016-2020 The Apache Software Foundation
This product includes software developed at The Apache Software
Foundation (http://www.apache.org/).

192
README.md
View File

@@ -17,121 +17,129 @@ specific language governing permissions and limitations
under the License.
-->
# Superset
Superset
=========
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/apache/superset?sort=semver)](https://github.com/apache/superset/tree/latest)
[![Build Status](https://github.com/apache/superset/workflows/Python/badge.svg)](https://github.com/apache/superset/actions)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/apache/incubator-superset)
[![Build Status](https://travis-ci.org/apache/incubator-superset.svg?branch=master)](https://travis-ci.org/apache/incubator-superset)
[![PyPI version](https://badge.fury.io/py/apache-superset.svg)](https://badge.fury.io/py/apache-superset)
[![Coverage Status](https://codecov.io/github/apache/superset/coverage.svg?branch=master)](https://codecov.io/github/apache/superset)
[![Coverage Status](https://codecov.io/github/apache/incubator-superset/coverage.svg?branch=master)](https://codecov.io/github/apache/incubator-superset)
[![PyPI](https://img.shields.io/pypi/pyversions/apache-superset.svg?maxAge=2592000)](https://pypi.python.org/pypi/apache-superset)
[![Get on Slack](https://img.shields.io/badge/slack-join-orange.svg)](https://join.slack.com/t/apache-superset/shared_invite/zt-16jvzmoi8-sI7jKWp~xc2zYRe~NqiY9Q)
[![Get on Slack](https://img.shields.io/badge/slack-join-orange.svg)](https://join.slack.com/t/apache-superset/shared_invite/zt-g8lpruog-HeqpgYrwdfrD5OYhlU7hPQ)
[![Documentation](https://img.shields.io/badge/docs-apache.org-blue.svg)](https://superset.apache.org)
[![Dependencies Status](https://david-dm.org/apache/incubator-superset/status.svg?path=superset-frontend)](https://david-dm.org/apache/incubator-superset?path=superset-frontend)
<img
src="https://github.com/apache/superset/raw/master/superset-frontend/src/assets/branding/superset-logo-horiz-apache.png"
src="https://github.com/apache/incubator-superset/raw/master/superset-frontend/branding/superset-logo-horiz-apache.png"
alt="Superset"
width="500"
/>
A modern, enterprise-ready business intelligence web application.
[**Why Superset?**](#why-superset) |
[**Why Superset**](#why-superset) |
[**Supported Databases**](#supported-databases) |
[**Installation and Configuration**](#installation-and-configuration) |
[**Release Notes**](RELEASING/README.md#release-notes-for-recent-releases) |
[**Get Involved**](#get-involved) |
[**Get Help**](#get-help) |
[**Contributor Guide**](#contributor-guide) |
[**Resources**](#resources) |
[**Organizations Using Superset**](RESOURCES/INTHEWILD.md)
[**Superset Users**](INTHEWILD.md)
## Why Superset?
Superset is a modern data exploration and data visualization platform. Superset can replace or augment proprietary business intelligence tools for many teams. Superset integrates well with a variety of data sources.
Superset provides:
- A **no-code interface** for building charts quickly
- A powerful, web-based **SQL Editor** for advanced querying
- A **lightweight semantic layer** for quickly defining custom dimensions and metrics
- Out of the box support for **nearly any SQL** database or data engine
- A wide array of **beautiful visualizations** to showcase your data, ranging from simple bar charts to geospatial visualizations
- Lightweight, configurable **caching layer** to help ease database load
- Highly extensible **security roles and authentication** options
- An **API** for programmatic customization
- A **cloud-native architecture** designed from the ground up for scale
## Screenshots & Gifs
**Large Gallery of Visualizations**
**Gallery**
<kbd><a href="https://superset.apache.org/gallery"><img title="Gallery" src="superset-frontend/src/assets/images/screenshots/gallery.jpg"/></a></kbd><br/>
<kbd><a href="https://superset.apache.org/gallery"><img title="Gallery" src="https://raw.githubusercontent.com/apache/incubator-superset/master/superset-frontend/images/screenshots/gallery.png"></a></kbd><br/>
**Craft Beautiful, Dynamic Dashboards**
**View Dashboards**
<kbd><img title="View Dashboards" src="superset-frontend/src/assets/images/screenshots/slack_dash.jpg"/></kbd><br/>
<kbd><img title="View Dashboards" src="https://raw.githubusercontent.com/apache/incubator-superset/master/superset-frontend/images/screenshots/bank_dash.png"></kbd><br/>
**No-Code Chart Builder**
**Slice & dice your data**
<kbd><img title="Slice & dice your data" src="superset-frontend/src/assets/images/screenshots/explore.jpg"/></kbd><br/>
<kbd><img title="Slice & dice your data" src="https://raw.githubusercontent.com/apache/incubator-superset/master/superset-frontend/images/screenshots/explore.png"></kbd><br/>
**Powerful SQL Editor**
**Query and visualize your data with SQL Lab**
<kbd><img title="SQL Lab" src="https://raw.githubusercontent.com/apache/incubator-superset/master/superset-frontend/images/screenshots/sqllab.png"></kbd><br/>
**Visualize geospatial data with deck.gl**
<kbd><img title="Geospatial" src="https://raw.githubusercontent.com/apache/incubator-superset/master/superset-frontend/images/screenshots/deckgl_dash.png"></kbd><br/>
**Choose from a wide array of visualizations**
<kbd><img title="Visualizations" src="https://raw.githubusercontent.com/apache/incubator-superset/master/superset-frontend/images/screenshots/visualizations.png"></kbd><br/>
## Why Superset
Superset provides:
* An intuitive interface to explore and visualize datasets, and
create interactive dashboards
* A wide array of beautiful visualizations to showcase your data.
* Easy, code-free, user flows to drill down and slice and dice the data
underlying exposed dashboards. The dashboards and charts act as a starting
point for deeper analysis
* A state of the art SQL editor/IDE exposing a rich metadata browser, and
an easy workflow to create visualizations out of any result set.
* An extensible, high granularity security model allowing intricate rules
on who can access which product features and datasets.
Integration with major
authentication backends (database, OpenID, LDAP, OAuth, REMOTE_USER, ...)
* A lightweight semantic layer, allowing to control how data sources are
exposed to the user by defining dimensions and metrics
* Out of the box support for most SQL-speaking databases
* Deep integration with Druid allows for Superset to stay blazing fast while
slicing and dicing large, realtime datasets
* Fast loading dashboards with configurable caching
<kbd><img title="SQL Lab" src="superset-frontend/src/assets/images/screenshots/sql_lab.jpg"/></kbd><br/>
## Supported Databases
Superset can query data from any SQL-speaking datastore or data engine (Presto, Trino, Athena, [and more](https://superset.apache.org/docs/databases/installing-database-drivers/)) that has a Python DB-API driver and a SQLAlchemy dialect.
Here are some of the major database solutions that are supported:
Superset speaks many SQL dialects through SQLAlchemy - a Python
SQL toolkit that is compatible with most databases. Here are some of the major database solutions that are supported:
<p align="center">
<img src="superset-frontend/src/assets/images/redshift.png" alt="redshift" border="0" width="106" height="41"/>
<img src="superset-frontend/src/assets/images/google-biquery.png" alt="google-biquery" border="0" width="114" height="43"/>
<img src="superset-frontend/src/assets/images/snowflake.png" alt="snowflake" border="0" width="152" height="46"/>
<img src="superset-frontend/src/assets/images/trino.png" alt="trino" border="0" width="46" height="46"/>
<img src="superset-frontend/src/assets/images/presto.png" alt="presto" border="0" width="152" height="46"/>
<img src="superset-frontend/src/assets/images/druid.png" alt="druid" border="0" width="135" height="37" />
<img src="superset-frontend/src/assets/images/firebolt.png" alt="firebolt" border="0" width="133" height="21.5" />
<img src="superset-frontend/src/assets/images/timescale.png" alt="timescale" border="0" width="102" height="26.8" />
<img src="superset-frontend/src/assets/images/rockset.png" alt="rockset" border="0" width="125" height="51" />
<img src="superset-frontend/src/assets/images/postgresql.png" alt="postgresql" border="0" width="132" height="81" />
<img src="superset-frontend/src/assets/images/mysql.png" alt="mysql" border="0" width="119" height="62" />
<img src="superset-frontend/src/assets/images/mssql-server.png" alt="mssql-server" border="0" width="93" height="74" />
<img src="superset-frontend/src/assets/images/db2.png" alt="db2" border="0" width="62" height="62" />
<img src="superset-frontend/src/assets/images/sqlite.png" alt="sqlite" border="0" width="102" height="45" />
<img src="superset-frontend/src/assets/images/sybase.png" alt="sybase" border="0" width="128" height="47" />
<img src="superset-frontend/src/assets/images/mariadb.png" alt="mariadb" border="0" width="83" height="63" />
<img src="superset-frontend/src/assets/images/vertica.png" alt="vertica" border="0" width="128" height="40" />
<img src="superset-frontend/src/assets/images/oracle.png" alt="oracle" border="0" width="121" height="66" />
<img src="superset-frontend/src/assets/images/firebird.png" alt="firebird" border="0" width="86" height="56" />
<img src="superset-frontend/src/assets/images/greenplum.png" alt="greenplum" border="0" width="140" height="45" />
<img src="superset-frontend/src/assets/images/clickhouse.png" alt="clickhouse" border="0" width="133" height="34" />
<img src="superset-frontend/src/assets/images/exasol.png" alt="exasol" border="0" width="106" height="59" />
<img src="superset-frontend/src/assets/images/monet-db.png" alt="monet-db" border="0" width="106" height="46" />
<img src="superset-frontend/src/assets/images/apache-kylin.png" alt="apache-kylin" border="0" width="56" height="64"/>
<img src="superset-frontend/src/assets/images/hologres.png" alt="hologres" border="0" width="71" height="64"/>
<img src="superset-frontend/src/assets/images/netezza.png" alt="netezza" border="0" width="64" height="64"/>
<img src="superset-frontend/src/assets/images/pinot.png" alt="pinot" border="0" width="165" height="64"/>
<img src="superset-frontend/src/assets/images/teradata.png" alt="teradata" border="0" width="165" height="64"/>
<img src="superset-frontend/src/assets/images/yugabyte.png" alt="yugabyte" border="0" width="180" height="31"/>
<img src="superset-frontend/images/redshift.png" alt="redshift" border="0" width="106" height="41"/>
<img src="superset-frontend/images/google-biquery.png" alt="google-biquery" border="0" width="114" height="43"/>
<img src="superset-frontend/images/snowflake.png" alt="snowflake" border="0" width="152" height="46"/>
<img src="superset-frontend/images/presto.png" alt="presto" border="0" width="152" height="46"/>
<img src="superset-frontend/images/druid.png" alt="druid" border="0" width="135" height="37" />
<img src="superset-frontend/images/postgresql.png" alt="postgresql" border="0" width="132" height="81" />
<img src="superset-frontend/images/mysql.png" alt="mysql" border="0" width="119" height="62" />
<img src="superset-frontend/images/mssql-server.png" alt="mssql-server" border="0" width="93" height="74" />
<img src="superset-frontend/images/db2.png" alt="db2" border="0" width="62" height="62" />
<img src="superset-frontend/images/sqlite.png" alt="sqlite" border="0" width="102" height="45" />
<img src="superset-frontend/images/sybase.png" alt="sybase" border="0" width="128" height="47" />
<img src="superset-frontend/images/mariadb.png" alt="mariadb" border="0" width="83" height="63" />
<img src="superset-frontend/images/vertica.png" alt="vertica" border="0" width="128" height="40" />
<img src="superset-frontend/images/oracle.png" alt="oracle" border="0" width="121" height="66" />
<img src="superset-frontend/images/firebird.png" alt="firebird" border="0" width="86" height="56" />
<img src="superset-frontend/images/greenplum.png" alt="greenplum" border="0" width="140" height="45" />
<img src="superset-frontend/images/clickhouse.png" alt="clickhouse" border="0" width="133" height="34" />
<img src="superset-frontend/images/exasol.png" alt="exasol" border="0" width="106" height="59" />
<img src="superset-frontend/images/monet-db.png" alt="monet-db" border="0" width="106" height="46" />
<img src="superset-frontend/images/apache-kylin.png" alt="apache-kylin" border="0" width="56" height="64"/>
</p>
**A more comprehensive list of supported databases** along with the configuration instructions can be found
[here](https://superset.apache.org/docs/databases/installing-database-drivers).
A complete list of supported databases can be found
[here](https://superset.incubator.apache.org/docs/databases/installing-database-drivers).
Want to add support for your datastore or data engine? Read more [here](https://superset.apache.org/docs/frequently-asked-questions#does-superset-work-with-insert-database-engine-here) about the technical requirements.
## Installation and Configuration
[Extended documentation for Superset](https://superset.apache.org/docs/installation/installing-superset-using-docker-compose)
[See in the documentation](https://superset.incubator.apache.org/docs/installation/installing-superset-using-docker-compose)
## Get Involved
- Ask and answer questions on [StackOverflow](https://stackoverflow.com/questions/tagged/apache-superset) using the **apache-superset** tag
- [Join our community's Slack](https://join.slack.com/t/apache-superset/shared_invite/zt-16jvzmoi8-sI7jKWp~xc2zYRe~NqiY9Q)
and please read our [Slack Community Guidelines](https://github.com/apache/superset/blob/master/CODE_OF_CONDUCT.md#slack-community-guidelines)
- [Join our dev@superset.apache.org Mailing list](https://lists.apache.org/list.html?dev@superset.apache.org)
* Ask and answer questions on [StackOverflow](https://stackoverflow.com/questions/tagged/apache-superset)
* [Join our community's Slack](https://join.slack.com/t/apache-superset/shared_invite/zt-g8lpruog-HeqpgYrwdfrD5OYhlU7hPQ)
and please read our [Slack Community Guidelines](CODE_OF_CONDUCT.md#slack-community-guidelines)
* [Join our dev@superset.apache.org Mailing list](https://lists.apache.org/list.html?dev@superset.apache.org)
## Contributor Guide
@@ -140,29 +148,19 @@ Interested in contributing? Check out our
to find resources around contributing along with a detailed guide on
how to set up a development environment.
## Resources
- Getting Started with Superset
- [Superset in 2 Minutes using Docker Compose](https://superset.apache.org/docs/installation/installing-superset-using-docker-compose#installing-superset-locally-using-docker-compose)
- [Installing Database Drivers](https://superset.apache.org/docs/databases/docker-add-drivers/)
- [Building New Database Connectors](https://preset.io/blog/building-database-connector/)
- [Create Your First Dashboard](https://superset.apache.org/docs/creating-charts-dashboards/first-dashboard)
- [Comprehensive Tutorial for Contributing Code to Apache Superset
](https://preset.io/blog/tutorial-contributing-code-to-apache-superset/)
- [Documentation for Superset End-Users (by Preset)](https://docs.preset.io/docs/terminology)
- Deploying Superset
- [Official Docker image](https://hub.docker.com/r/apache/superset)
- [Helm Chart](https://github.com/apache/superset/tree/master/helm/superset)
- Recordings of Past [Superset Community Events](https://preset.io/events)
- [Live Demo: Interactive Time-series Analysis with Druid and Superset](https://preset.io/events/2021-03-02-interactive-time-series-analysis-with-druid-and-superset/)
- [Live Demo: Visualizing MongoDB and Pinot Data using Trino](https://preset.io/events/2021-04-13-visualizing-mongodb-and-pinot-data-using-trino/)
- [Superset Contributor Bootcamp](https://preset.io/events/superset-contributor-bootcamp-dec-21/)
- [Introduction to the Superset API](https://preset.io/events/introduction-to-the-superset-api/)
- [Apache Superset 1.3 Meetup](https://preset.io/events/apache-superset-1-3/)
- [Building a Database Connector for Superset](https://preset.io/events/2021-02-16-building-a-database-connector-for-superset/)
- Visualizations
- [Building Custom Viz Plugins](https://superset.apache.org/docs/installation/building-custom-viz-plugins)
- [Managing and Deploying Custom Viz Plugins](https://medium.com/nmc-techblog/apache-superset-manage-custom-viz-plugins-in-production-9fde1a708e55)
- [Why Apache Superset is Betting on Apache ECharts](https://preset.io/blog/2021-4-1-why-echarts/)
- [Superset API](https://superset.apache.org/docs/rest-api)
* Superset 101 -- Getting Started Guide (From [Preset Blog](https://preset.io/blog/))
* [Installing Apache Superset Locally](https://preset.io/blog/2020-05-11-getting-started-installing-superset/)
* [Installing Database Drivers](https://preset.io/blog/2020-05-18-install-db-drivers/)
* [Connect Superset To Google Sheets](https://preset.io/blog/2020-06-01-connect-superset-google-sheets/)
* [Create Your First Chart](https://preset.io/blog/2020-06-08-first-chart/)
* [Create Time Series Charts](https://preset.io/blog/2020-06-26-timeseries-chart/)
* [Docker image](https://hub.docker.com/r/apache/incubator-superset)
* [Youtube Channel](https://www.youtube.com/channel/UCMuwrvBsg_jjI2gLcm04R0g)
* [May 2020: Virtual Meetup Recording. Topics: 0.36 Overview, Committers Self-Intro, Roadmap](https://www.youtube.com/watch?v=tXGDmqjmcTs&t=20s)
* [July 2020: Virtual Meetup Recording. Topics: Visualization Plugins, 0.37 Preview, Demo](https://www.youtube.com/watch?v=f6up5x_iRbI)
* [November 2020: Superset 1.0](https://www.youtube.com/watch?v=GwtWRUSEjk4)
* [Building Custom Viz Plugins](https://superset.apache.org/docs/installation/building-custom-viz-plugins)
* [Superset API](https://superset.apache.org/docs/rest-api)

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.8-buster
FROM python:3.7-buster
RUN useradd --user-group --create-home --no-log-init --shell /bin/bash superset
@@ -28,13 +28,14 @@ RUN apt-get update -y
RUN apt-get install -y apt-transport-https apt-utils
# Install superset dependencies
# https://superset.apache.org/docs/installation/installing-superset-from-scratch
# https://superset.incubator.apache.org/installation.html#os-dependencies
RUN apt-get install -y build-essential libssl-dev \
libffi-dev python3-dev libsasl2-dev libldap2-dev libxi-dev
# Install nodejs for custom build
# https://superset.incubator.apache.org/installation.html#making-your-own-build
# https://nodejs.org/en/download/package-manager/
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - \
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash - \
&& apt-get install -y nodejs
RUN mkdir -p /home/superset
@@ -47,13 +48,13 @@ ARG SUPERSET_RELEASE_RC_TARBALL
# Can fetch source from svn or copy tarball from local mounted directory
COPY $SUPERSET_RELEASE_RC_TARBALL ./
RUN tar -xvf *.tar.gz
WORKDIR /home/superset/apache-superset-$VERSION/superset-frontend
WORKDIR /home/superset/apache-superset-incubating-$VERSION/superset-frontend
RUN npm ci \
&& npm run build \
&& rm -rf node_modules
WORKDIR /home/superset/apache-superset-$VERSION
WORKDIR /home/superset/apache-superset-incubating-$VERSION
RUN pip install --upgrade setuptools pip \
&& pip install -r requirements/base.txt \
&& pip install --no-cache-dir .

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.8-buster
FROM python:3.7-buster
RUN useradd --user-group --create-home --no-log-init --shell /bin/bash superset
@@ -28,13 +28,14 @@ RUN apt-get update -y
RUN apt-get install -y apt-transport-https apt-utils
# Install superset dependencies
# https://superset.apache.org/docs/installation/installing-superset-from-scratch
# https://superset.incubator.apache.org/installation.html#os-dependencies
RUN apt-get install -y build-essential libssl-dev \
libffi-dev python3-dev libsasl2-dev libldap2-dev libxi-dev
# Install nodejs for custom build
# https://superset.incubator.apache.org/installation.html#making-your-own-build
# https://nodejs.org/en/download/package-manager/
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - \
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash - \
&& apt-get install -y nodejs
RUN mkdir -p /home/superset
@@ -44,9 +45,9 @@ WORKDIR /home/superset
ARG VERSION
# Can fetch source from svn or copy tarball from local mounted directory
RUN svn co https://dist.apache.org/repos/dist/dev/superset/$VERSION ./
RUN svn co https://dist.apache.org/repos/dist/dev/incubator/superset/$VERSION ./
RUN tar -xvf *.tar.gz
WORKDIR apache-superset-$VERSION
WORKDIR apache-superset-incubating-$VERSION
RUN cd superset-frontend \
&& npm ci \
@@ -54,9 +55,9 @@ RUN cd superset-frontend \
&& rm -rf node_modules
WORKDIR /home/superset/apache-superset-$VERSION
WORKDIR /home/superset/apache-superset-incubating-$VERSION
RUN pip install --upgrade setuptools pip \
&& pip install -r requirements/base.txt \
&& pip install -r requirements.txt \
&& pip install --no-cache-dir .
RUN flask fab babel-compile --target superset/translations

View File

@@ -14,10 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.8-buster
FROM python:3.7-buster
ARG VERSION
RUN git clone --depth 1 --branch ${VERSION} https://github.com/apache/superset.git /superset
RUN git clone --depth 1 --branch ${VERSION} https://github.com/apache/incubator-superset.git /superset
WORKDIR /superset
# install doc dependencies
RUN pip install -r requirements/base.txt

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.8-buster
FROM python:3.7-buster
RUN apt-get update -y
RUN apt-get install -y jq

View File

@@ -28,16 +28,6 @@ synchronous communications, we recommend using the `#apache-releases` channel
on the Superset Slack. People crafting releases and those interested in
partaking in the process should join the channel.
## Release notes for recent releases
- [1.5](release-notes-1-5/README.md)
- [1.4](release-notes-1-4/README.md)
- [1.3](release-notes-1-3/README.md)
- [1.2](release-notes-1-2/README.md)
- [1.1](release-notes-1-1/README.md)
- [1.0](release-notes-1-0/README.md)
- [0.38](release-notes-0-38/README.md)
## Release setup (First Time Only)
First you need to setup a few things. This is a one-off and doesn't
@@ -49,9 +39,9 @@ need to be done at every release.
# Checkout ASF dist repo
svn checkout https://dist.apache.org/repos/dist/dev/superset/ ~/svn/superset_dev
svn checkout https://dist.apache.org/repos/dist/dev/incubator/superset/ ~/svn/superset_dev
svn checkout https://dist.apache.org/repos/dist/release/superset/ ~/svn/superset
svn checkout https://dist.apache.org/repos/dist/release/incubator/superset/ ~/svn/superset
cd ~/svn/superset
@@ -62,15 +52,12 @@ need to be done at every release.
# Commit the changes
svn commit -m "Add PGP keys of new Superset committer"
# push the changes
svn update
```
## Setting up the release environment (do every time)
As the vote process takes a minimum of 72h, sometimes stretching over several weeks
of calendar time if votes don't pass, chances are
As the vote process takes a minimum of 72h (community vote) + 72h (IPMC) vote,
often stretching over several weeks calendar time if votes don't pass, chances are
the same terminal session won't be used for crafting the release candidate and the
final release. Therefore, it's a good idea to do the following every time you
work on a new phase of the release process to make sure you aren't releasing
@@ -80,33 +67,33 @@ and execute the `set_release_env.sh` script with the relevant parameters:
Usage (BASH):
```bash
. set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
. set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
```
Usage (ZSH):
```bash
source set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
source set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
```
Example:
```bash
source set_release_env.sh 0.38.0rc1 myid@apache.org
source set_release_env.sh 0.38.0rc1 myid@apache.org
```
The script will output the exported variables. Here's example for 0.38.0rc1:
```
Set Release env variables
SUPERSET_VERSION=0.38.0
SUPERSET_RC=1
SUPERSET_GITHUB_BRANCH=0.38
SUPERSET_PGP_FULLNAME=myid@apache.org
SUPERSET_VERSION_RC=0.38.0rc1
SUPERSET_RELEASE=apache-superset-0.38.0
SUPERSET_RELEASE_RC=apache-superset-0.38.0rc1
SUPERSET_RELEASE_TARBALL=apache-superset-0.38.0-source.tar.gz
SUPERSET_RELEASE_RC_TARBALL=apache-superset-0.38.0rc1-source.tar.gz
SUPERSET_TMP_ASF_SITE_PATH=/tmp/superset-site-0.38.0
Set Release env variables
SUPERSET_VERSION=0.38.0
SUPERSET_RC=1
SUPERSET_GITHUB_BRANCH=0.38
SUPERSET_PGP_FULLNAME=myid@apache.org
SUPERSET_VERSION_RC=0.38.0rc1
SUPERSET_RELEASE=apache-superset-incubating-0.38.0
SUPERSET_RELEASE_RC=apache-superset-incubating-0.38.0rc1
SUPERSET_RELEASE_TARBALL=apache-superset-incubating-0.38.0-source.tar.gz
SUPERSET_RELEASE_RC_TARBALL=apache-superset-incubating-0.38.0rc1-source.tar.gz
SUPERSET_TMP_ASF_SITE_PATH=/tmp/incubator-superset-site-0.38.0
```
## Crafting a source release
@@ -127,20 +114,12 @@ git push upstream $SUPERSET_GITHUB_BRANCH
Next, update the `CHANGELOG.md` with all the changes that are included in the release.
Make sure the branch has been pushed to `upstream` to ensure the changelog generator
can pick up changes since the previous release.
Change log script requires a github token and will try to use your env var GITHUB_TOKEN.
you can also pass the token using the parameter `--access_token`.
Example:
```bash
python changelog.py --previous_version 0.37 --current_version 0.38 changelog
```
You can get a list of pull requests with labels started with blocking, risk, hold, revert and security by using the parameter `--risk`.
Example:
```bash
python changelog.py --previous_version 0.37 --current_version 0.38 changelog --access_token {GITHUB_TOKEN} --risk
```
The script will checkout both branches and compare all the PR's, copy the output and paste it on the `CHANGELOG.md`
Then, in `UPDATING.md`, a file that contains a list of notifications around
@@ -151,18 +130,18 @@ section for the new release.
Finally bump the version number on `superset-frontend/package.json` (replace with whichever version is being released excluding the RC version):
```json
"version": "0.38.0"
"version": "0.38.0"
```
Commit the change with the version number, then git tag the version with the release candidate and push to the branch:
```
# add changed files and commit
git add ...
git commit ...
# push new tag
git tag ${SUPERSET_VERSION_RC}
git push upstream ${SUPERSET_VERSION_RC}
# add changed files and commit
git add ...
git commit ...
# push new tag
git tag ${SUPERSET_VERSION_RC}
git push upstream ${SUPERSET_VERSION_RC}
```
## Preparing the release candidate
@@ -173,15 +152,15 @@ release on Superset's repo (MAJOR.MINOR branch), the following script will clone
the tag and create a signed source tarball from it:
```bash
# make_tarball will use the previously set environment variables
# you can override by passing arguments: make_tarball.sh <SUPERSET_VERSION> <SUPERSET_VERSION_RC> "<PGP_KEY_FULLNAME>"
./make_tarball.sh
# make_tarball will use the previously set environment variables
# you can override by passing arguments: make_tarball.sh <SUPERSET_VERSION> <SUPERSET_VERSION_RC> "<PGP_KEY_FULLNAME>"
./make_tarball.sh
```
Note that `make_tarball.sh`:
- By default assumes you have already executed an SVN checkout to `$HOME/svn/superset_dev`.
This can be overridden by setting `SUPERSET_SVN_DEV_PATH` environment var to a different svn dev directory
This can be overriden by setting `SUPERSET_SVN_DEV_PATH` environment var to a different svn dev directory
- Will refuse to craft a new release candidate if a release already exists on your local svn dev directory
- Will check `package.json` version number and fails if it's not correctly set
@@ -189,10 +168,10 @@ This can be overridden by setting `SUPERSET_SVN_DEV_PATH` environment var to a d
To build and run the **local copy** of the recently created tarball:
```bash
# Build and run a release candidate tarball
./test_run_tarball.sh local
# you should be able to access localhost:5001 on your browser
# login using admin/admin
# Build and run a release candidate tarball
./test_run_tarball.sh local
# you should be able to access localhost:5001 on your browser
# login using admin/admin
```
### Shipping to SVN
@@ -200,20 +179,19 @@ To build and run the **local copy** of the recently created tarball:
Now let's ship this RC into svn's dev folder
```bash
cd ~/svn/superset_dev/
svn add ${SUPERSET_VERSION_RC}
svn commit -m "Release ${SUPERSET_VERSION_RC}"
svn update
cd ~/svn/superset_dev/
svn add ${SUPERSET_VERSION_RC}
svn commit -m "Release ${SUPERSET_VERSION_RC}"
```
### Build and test from SVN source tarball
To build and run the recently created tarball **from SVN**:
```bash
# Build and run a release candidate tarball
./test_run_tarball.sh
# you should be able to access localhost:5001 on your browser
# login using admin/admin
# Build and run a release candidate tarball
./test_run_tarball.sh
# you should be able to access localhost:5001 on your browser
# login using admin/admin
```
### Voting
@@ -224,17 +202,17 @@ https://lists.apache.org/thread.html/e60f080ebdda26896214f7d3d5be1ccadfab95d48fb
To easily send a voting request to Superset community, still on the `superset/RELEASING` directory:
```bash
# Note: use Superset's virtualenv
(venv)$ python send_email.py vote_pmc
# Note: use Superset's virtualenv
(venv)$ python send_email.py vote_pmc
```
The script will interactively ask for extra information so it can authenticate on the Apache Email Relay.
The release version and release candidate number are fetched from the previously set environment variables.
```
Sender email (ex: user@apache.org): your_apache_email@apache.org
Apache username: your_apache_user
Apache password: your_apache_password
```bash
Sender email (ex: user@apache.org): your_apache_email@apache.org
Apache username: your_apache_user
Apache password: your_apache_password
```
Once 3+ binding votes (by PMC members) have been cast and at
@@ -244,24 +222,56 @@ https://lists.apache.org/thread.html/50a6b134d66b86b237d5d7bc89df1b567246d125a71
To easily send the result email, still on the `superset/RELEASING` directory:
```bash
# Note: use Superset's virtualenv
python send_email.py result_pmc
# Note: use Superset's virtualenv
(venv)$ python send_email.py result_pmc
```
The script will interactively ask for extra information needed to fill out the email template. Based on the
voting description, it will generate a passing, non passing or non conclusive email.
here's an example:
```
Sender email (ex: user@apache.org): your_apache_email@apache.org
Apache username: your_apache_user
Apache password: your_apache_password
A List of people with +1 binding vote (ex: Max,Grace,Krist): Daniel,Alan,Max,Grace
A List of people with +1 non binding vote (ex: Ville): Ville
A List of people with -1 vote (ex: John):
```bash
Sender email (ex: user@apache.org): your_apache_email@apache.org
Apache username: your_apache_user
Apache password: your_apache_password
A List of people with +1 binding vote (ex: Max,Grace,Krist): Daniel,Alan,Max,Grace
A List of people with +1 non binding vote (ex: Ville): Ville
A List of people with -1 vote (ex: John):
```
Following the result thread, yet another [VOTE] thread should be
started at general@incubator.apache.org.
To easily send the voting request to Apache community, still on the `superset/RELEASING` directory:
```bash
# Note: use Superset's virtualenv
(venv)$ python send_email.py vote_ipmc
```
Once 3+ binding votes (by IPMC members) have been cast and at
least 72 hours have past, you can post a [RESULT] thread
To easily send the result email, still on the `superset/RELEASING` directory:
```bash
# Note: use Superset's virtualenv
(venv)$ python send_email.py result_ipmc
```
Again, the script will interactively ask for extra information needed to fill out the email template. Based on the
voting description, it will generate a passing, non passing or non conclusive email.
here's an example:
```bash
Sender email (ex: user@apache.org): your_apache_email@apache.org
Apache username: your_apache_user
Apache password: your_apache_password
A List of people with +1 binding vote (ex: Alan, Justin): Alan,Jeff,
A List of people with +1 non binding vote (ex: Ville):
A List of people with -1 vote (ex: John):
```
### Validating a release
@@ -269,27 +279,25 @@ https://www.apache.org/info/verification.html
## Publishing a successful release
Upon a successful vote, you'll have to copy the folder into the non-"dev/" folder.
Upon a successful vote (community AND IPMC), you'll have to copy the folder
into the non-"dev/" folder.
```bash
cp -r ~/svn/superset_dev/${SUPERSET_VERSION_RC}/ ~/svn/superset/${SUPERSET_VERSION}/
cd ~/svn/superset/
# Rename the RC (0.34.1rc1) to the actual version being released (0.34.1)
for f in ${SUPERSET_VERSION}/*; do mv "$f" "${f/${SUPERSET_VERSION_RC}/${SUPERSET_VERSION}}"; done
svn add ${SUPERSET_VERSION}
svn commit -m "Release ${SUPERSET_VERSION}"
svn update
cp -r ~/svn/superset_dev/${SUPERSET_VERSION_RC}/ ~/svn/superset/${SUPERSET_VERSION}/
cd ~/svn/superset/
# Rename the RC (0.34.1rc1) to the actual version being released (0.34.1)
for f in ${SUPERSET_VERSION}/*; do mv "$f" "${f/${SUPERSET_VERSION_RC}/${SUPERSET_VERSION}}"; done
svn add ${SUPERSET_VERSION}
svn commit -m "Release ${SUPERSET_VERSION}"
```
Then tag the final release:
```bash
# Go to the root directory of the repo, e.g. `~/src/superset`
cd ~/src/superset/
# make sure you're on the correct branch (e.g. 0.34)
git branch
# Create the release tag
git tag -f ${SUPERSET_VERSION}
# push the tag to the remote
git push upstream ${SUPERSET_VERSION}
# Go to the root directory of the repo, e.g. `~/src/incubator-superset`
cd ~/src/incubator-superset/
# make sure you're on the correct branch (e.g. 0.34)
git branch
# Create the release tag
git tag -f ${SUPERSET_VERSION}
```
### Update CHANGELOG and UPDATING on superset
@@ -311,15 +319,13 @@ while requesting access to push packages.
Once it's all done, an [ANNOUNCE] thread announcing the release to the dev@ mailing list is the final step.
```bash
# Note use Superset's virtualenv
python send_email.py announce
# Note use Superset's virtualenv
(venv)$ python send_email.py announce
```
### Github Release
Finally, so the Github UI reflects the latest release, you should create a release from the
tag corresponding with the new version. Go to https://github.com/apache/superset/tags,
tag corresponding with the new version. Go to https://github.com/apache/incubator-superset/tags,
click the 3-dot icon and select `Create Release`, paste the content of the ANNOUNCE thread in the
release notes, and publish the new release.
At this point, a GitHub action will run that will check whether this release's version number is higher than the current 'latest' release. If that condition is true, this release sha will automatically be tagged as `latest` so that the most recent release can be referenced simply by using the 'latest' tag instead of looking up the version number. The existing version number tag will still exist, and can also be used for reference.

View File

@@ -13,25 +13,21 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# pylint: disable=no-value-for-parameter
import csv as lib_csv
import json
import os
import re
import sys
from dataclasses import dataclass
from time import sleep
from typing import Any, Dict, Iterator, List, Optional, Union
from urllib import request
from urllib.error import HTTPError
import click
from click.core import Context
try:
from github import BadCredentialsException, Github, PullRequest, Repository
except ModuleNotFoundError:
print("PyGithub is a required package for this script")
exit(1)
SUPERSET_REPO = "apache/superset"
SUPERSET_PULL_REQUEST_TYPES = r"^(fix|feat|chore|refactor|docs|build|ci|/gmi)"
SUPERSET_RISKY_LABELS = r"^(blocking|risk|hold|revert|security vulnerability)"
@dataclass
@@ -48,7 +44,7 @@ class GitLog:
author_email: str = ""
def __eq__(self, other: object) -> bool:
"""A log entry is considered equal if it has the same PR number"""
""" A log entry is considered equal if it has the same PR number """
if isinstance(other, self.__class__):
return other.pr_number == self.pr_number
return False
@@ -64,43 +60,48 @@ class GitChangeLog:
We want to map a git author to a github login, for that we call github's API
"""
def __init__(
self,
version: str,
logs: List[GitLog],
access_token: Optional[str] = None,
risk: Optional[bool] = False,
) -> None:
def __init__(self, version: str, logs: List[GitLog]) -> None:
self._version = version
self._logs = logs
self._pr_logs_with_details: Dict[int, Dict[str, Any]] = {}
self._github_login_cache: Dict[str, Optional[str]] = {}
self._github_prs: Dict[int, Any] = {}
self._wait = 10
github_token = access_token or os.environ.get("GITHUB_TOKEN")
self._github = Github(github_token)
self._show_risk = risk
self._superset_repo: Repository = None
def _fetch_github_pr(self, pr_number: int) -> PullRequest:
def _wait_github_rate_limit(self) -> None:
"""
Waits for available rate limit slots on the github API
"""
while True:
rate_limit_payload = self._fetch_github_rate_limit()
if rate_limit_payload["rate"]["remaining"] > 1:
break
print(".", end="", flush=True)
sleep(self._wait)
print()
@staticmethod
def _fetch_github_rate_limit() -> Dict[str, Any]:
"""
Fetches current github rate limit info
"""
with request.urlopen("https://api.github.com/rate_limit") as response:
payload = json.loads(response.read())
return payload
def _fetch_github_pr(self, pr_number: int) -> Dict[str, Any]:
"""
Fetches a github PR info
"""
payload = {}
try:
github_repo = self._github.get_repo(SUPERSET_REPO)
self._superset_repo = github_repo
pull_request = self._github_prs.get(pr_number)
if not pull_request:
pull_request = github_repo.get_pull(pr_number)
self._github_prs[pr_number] = pull_request
except BadCredentialsException as ex:
print(
f"Bad credentials to github provided"
f" use access_token parameter or set GITHUB_TOKEN"
)
sys.exit(1)
return pull_request
self._wait_github_rate_limit()
with request.urlopen(
"https://api.github.com/repos/apache/incubator-superset/pulls/"
f"{pr_number}"
) as response:
payload = json.loads(response.read())
except HTTPError as ex:
print(f"{ex}", flush=True)
return payload
def _get_github_login(self, git_log: GitLog) -> Optional[str]:
"""
@@ -113,119 +114,35 @@ class GitChangeLog:
if git_log.pr_number:
pr_info = self._fetch_github_pr(git_log.pr_number)
if pr_info:
github_login = pr_info.user.login
github_login = pr_info["user"]["login"]
else:
github_login = author_name
# set cache
self._github_login_cache[author_name] = github_login
return github_login
def _has_commit_migrations(self, git_sha: str) -> bool:
commit = self._superset_repo.get_commit(sha=git_sha)
return any(
"superset/migrations/versions/" in file.filename for file in commit.files
)
def _get_pull_request_details(self, git_log: GitLog) -> Dict[str, Any]:
pr_number = git_log.pr_number
if pr_number:
detail = self._pr_logs_with_details.get(pr_number)
if detail:
return detail
pr_info = self._fetch_github_pr(pr_number)
has_migrations = self._has_commit_migrations(git_log.sha)
title = pr_info.title if pr_info else git_log.message
pr_type = re.match(SUPERSET_PULL_REQUEST_TYPES, title)
if pr_type:
pr_type = pr_type.group().strip('"')
labels = (" | ").join([label.name for label in pr_info.labels])
is_risky = self._is_risk_pull_request(pr_info.labels)
detail = {
"id": pr_number,
"has_migrations": has_migrations,
"labels": labels,
"title": title,
"type": pr_type,
"is_risky": is_risky or has_migrations,
}
if pr_number:
self._pr_logs_with_details[pr_number] = detail
return detail
def _is_risk_pull_request(self, labels: List[Any]) -> bool:
for label in labels:
risk_label = re.match(SUPERSET_RISKY_LABELS, label.name)
if risk_label is not None:
return True
return False
def _get_changelog_version_head(self) -> str:
return f"### {self._version} ({self._logs[0].time})"
def _parse_change_log(
self,
changelog: Dict[str, str],
pr_info: Dict[str, str],
github_login: str,
) -> None:
formatted_pr = (
f"- [#{pr_info.get('id')}]"
f"(https://github.com/{SUPERSET_REPO}/pull/{pr_info.get('id')}) "
f"{pr_info.get('title')} (@{github_login})\n"
)
if pr_info.get("has_migrations"):
changelog["Database Migrations"] += formatted_pr
elif pr_info.get("type") == "fix":
changelog["Fixes"] += formatted_pr
elif pr_info.get("type") == "feat":
changelog["Features"] += formatted_pr
else:
changelog["Others"] += formatted_pr
def __repr__(self) -> str:
result = f"\n{self._get_changelog_version_head()}\n"
changelog = {
"Database Migrations": "\n",
"Features": "\n",
"Fixes": "\n",
"Others": "\n",
}
for i, log in enumerate(self._logs):
github_login = self._get_github_login(log)
pr_info = self._get_pull_request_details(log)
if not github_login:
github_login = log.author
if self._show_risk:
if pr_info.get("is_risky"):
result += (
f"- [#{log.pr_number}]"
f"(https://github.com/{SUPERSET_REPO}/pull/{log.pr_number}) "
f"{pr_info.get('title')} (@{github_login}) "
f"{pr_info.get('labels')} \n"
)
else:
self._parse_change_log(changelog, pr_info, github_login)
result = result + (
f"- [#{log.pr_number}]"
f"(https://github.com/apache/incubator-superset/pull/{log.pr_number}) "
f"{log.message} (@{github_login})\n"
)
print(f"\r {i}/{len(self._logs)}", end="", flush=True)
if self._show_risk:
return result
for key in changelog:
result += f"**{key}** {changelog[key]}\n"
return result
def __iter__(self) -> Iterator[Dict[str, Any]]:
for log in self._logs:
yield {
"pr_number": log.pr_number,
"pr_link": f"https://github.com/{SUPERSET_REPO}/pull/"
"pr_link": f"https://github.com/apache/incubator-superset/pull/"
f"{log.pr_number}",
"message": log.message,
"time": log.time,
@@ -325,8 +242,8 @@ def print_title(message: str) -> None:
@click.pass_context
@click.option("--previous_version", help="The previous release version", required=True)
@click.option("--current_version", help="The current release version", required=True)
def cli(ctx: Context, previous_version: str, current_version: str) -> None:
"""Welcome to change log generator"""
def cli(ctx, previous_version: str, current_version: str) -> None:
""" Welcome to change log generator """
previous_logs = GitLogs(previous_version)
current_logs = GitLogs(current_version)
previous_logs.fetch()
@@ -338,7 +255,7 @@ def cli(ctx: Context, previous_version: str, current_version: str) -> None:
@cli.command("compare")
@click.pass_obj
def compare(base_parameters: BaseParameters) -> None:
"""Compares both versions (by PR)"""
""" Compares both versions (by PR) """
previous_logs = base_parameters.previous_logs
current_logs = base_parameters.current_logs
print_title(
@@ -358,29 +275,15 @@ def compare(base_parameters: BaseParameters) -> None:
@cli.command("changelog")
@click.option(
"--csv",
help="The csv filename to export the changelog to",
"--csv", help="The csv filename to export the changelog to",
)
@click.option(
"--access_token",
help="The github access token,"
" if not provided will try to fetch from GITHUB_TOKEN env var",
)
@click.option("--risk", is_flag=True, help="show all pull requests with risky labels")
@click.pass_obj
def change_log(
base_parameters: BaseParameters, csv: str, access_token: str, risk: bool
) -> None:
"""Outputs a changelog (by PR)"""
def change_log(base_parameters: BaseParameters, csv: str) -> None:
""" Outputs a changelog (by PR) """
previous_logs = base_parameters.previous_logs
current_logs = base_parameters.current_logs
previous_diff_logs = previous_logs.diff(current_logs)
logs = GitChangeLog(
current_logs.git_ref,
previous_diff_logs[::-1],
access_token=access_token,
risk=risk,
)
logs = GitChangeLog(current_logs.git_ref, previous_diff_logs[::-1])
if csv:
with open(csv, "w") as csv_file:
log_items = list(logs)

View File

@@ -18,18 +18,18 @@
-#}
To: {{ receiver_email }}
From: {{ sender_email }}
Subject: [ANNOUNCE] Apache {{ project_name }} version {{ version }} Released
Subject: [ANNOUNCE] Apache {{ project_name }} (Incubating) version {{ version }} Released
Hello Community,
The Apache {{ project_name }} team is pleased to announce that {{ project_name }}
The Apache {{ project_name }} (incubating) team is pleased to announce that {{ project_name }}
{{ version }} has just been released.
{{ project_description }}
The official source release:
https://www.apache.org/dist/{{ project_module }}/{{ version }}
https://www.apache.org/dist/incubator/{{ project_module }}/{{ version }}
The Pypi package:
@@ -40,6 +40,18 @@ find any problems about enhancements included in this release, please
don't hesitate to let us know by sending feedback to this mailing
list.
=====
*Disclaimer*
Apache {{ project_name }} is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the Incubator. Incubation is required of all
newly accepted projects until a further review indicates that the
infrastructure, communications, and decision making process have stabilized
in a manner consistent with other successful ASF projects. While incubation
status is not necessarily a reflection of the completeness or stability of
the code, it does indicate that the project has yet to be fully endorsed by
the ASF.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@apache.org
For additional commands, e-mail: general-help@apache.org
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

View File

@@ -0,0 +1,58 @@
{#
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-#}
To: {{ receiver_email }}
From: {{ sender_email }}
Subject: [RESULT] [VOTE] Release Apache {{ project_name }} (incubating) version {{ version }}
Thanks to everyone that participated. The vote to release
Apache {{ project_name }} (incubating) version {{ version }} is now closed.
{% if vote_negatives|length > 0 %}
The vote did not PASS with {{vote_bindings|length}} binding +1, {{ vote_nonbindings|length}} non binding +1 and {{vote_negatives|length}} -1 votes:
{% elif vote_bindings|length > 2 %}
The vote PASSED with {{vote_bindings|length}} binding +1, {{ vote_nonbindings|length}} non binding +1 and {{vote_negatives|length}} -1 votes:
{% else %}
The vote is non conclusive with {{vote_bindings|length}} binding +1, {{ vote_nonbindings|length}} non binding -1 and {{vote_negatives|length}} -1 votes:
{% endif %}
{% if vote_bindings|length > 0 -%}
Binding votes:
{% for voter in vote_bindings -%}
- {{ voter }}
{% endfor %}
{%- endif %}
{% if vote_nonbindings|length > 0 -%}
Non binding votes:
{% for voter in vote_nonbindings -%}
- {{ voter }}
{% endfor %}
{%- endif %}
{% if vote_negatives|length > 0 -%}
Negative votes:
{% for voter in vote_negatives -%}
- {{ voter }}
{% endfor %}
{%- endif %}
We will work to complete the release process.
Thanks,
The Apache {{ project_name }} (Incubating) Team

View File

@@ -18,10 +18,10 @@
-#}
To: {{ receiver_email }}
From: {{ sender_email }}
Subject: [RESULT] [VOTE] Release Apache {{ project_name }} {{ version }} based on Superset {{ version_rc }}
Subject: [RESULT] [VOTE] Release Apache {{ project_name }} (incubating) {{ version }} based on Superset {{ version_rc }}
Thanks to everyone that participated. The vote to release
Apache {{ project_name }} version {{ version }} based on {{ version_rc }} is now closed.
Apache {{ project_name }} (incubating) version {{ version }} based on {{ version_rc }} is now closed.
{% if vote_negatives|length > 0 -%}
The vote did NOT PASS with {{vote_bindings|length}} binding +1, {{ vote_nonbindings|length}} non binding +1 and {{vote_negatives|length}} -1 votes:
@@ -52,9 +52,7 @@ Negative votes:
{% endfor -%}
{%- endif %}
Link to vote thread: {{ vote_thread }}
We will work to complete the release process.
Thanks,
The Apache {{ project_name }} Team
The Apache {{ project_name }} (Incubating) Team

View File

@@ -0,0 +1,63 @@
{#
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-#}
To: {{ receiver_email }}
From: {{ sender_email }}
Subject: [VOTE] Release Apache {{ project_name }} (incubating) version {{ version }}
Hello IPMC,
The Apache {{ project_name }} (incubating) community has voted on and approved a proposal to
release Apache {{ project_name }} (incubating) version {{ version }}.
The voting thread can be found here: {{ voting_thread }}
{% if vote_mentors|length > 0 %}
Here are the binding +1 votes from mentors, carrying over from the podling vote:
{% for voter in vote_mentors -%}
- {{ voter }}
{% endfor %}
{% endif -%}
We now kindly request the Incubator PMC members review and vote on this
incubator release.
{{ project_description }}
The release candidate:
https://dist.apache.org/repos/dist/dev/incubator/{{ project_module }}/{{ version_rc }}/
Git tag for the release:
https://github.com/apache/incubator-{{ project_module }}/tree/{{ version_rc }}
The Change Log for the release:
https://github.com/apache/incubator-{{ project_module }}/blob/{{ version_rc }}/CHANGELOG.md
public keys are available at:
https://www.apache.org/dist/incubator/{{ project_module }}/KEYS
The vote will be open for at least 72 hours or until the necessary number
of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Thanks,
The Apache {{ project_name }} (Incubating) Team

View File

@@ -18,27 +18,28 @@
-#}
To: {{ receiver_email }}
From: {{ sender_email }}
Subject: [VOTE] Release Apache {{ project_name }} {{ version }} based on Superset {{ version_rc }}
Subject: [VOTE] Release Apache {{ project_name }} (incubating) {{ version }} based on Superset {{ version_rc }}
Hello {{ project_name }} Community,
This is a call for the vote to release Apache {{ project_name }} version {{ version }}.
This is a call for the vote to release Apache {{ project_name }}
(incubating) version {{ version }}.
The release candidate:
https://dist.apache.org/repos/dist/dev/{{ project_module }}/{{ version_rc }}/
https://dist.apache.org/repos/dist/dev/incubator/{{ project_module }}/{{ version_rc }}/
Git tag for the release:
https://github.com/apache/{{ project_module }}/tree/{{ version_rc }}
https://github.com/apache/incubator-{{ project_module }}/tree/{{ version_rc }}
The Change Log for the release:
https://github.com/apache/{{ project_module }}/blob/{{ version_rc }}/CHANGELOG.md
https://github.com/apache/incubator-{{ project_module }}/blob/{{ version_rc }}/CHANGELOG.md
The Updating instructions for the release:
https://github.com/apache/{{ project_module }}/blob/{{ version_rc }}/UPDATING.md
https://github.com/apache/incubator-{{ project_module }}/blob/{{ version_rc }}/UPDATING.md
public keys are available at:
https://www.apache.org/dist/{{ project_module }}/KEYS
https://www.apache.org/dist/incubator/{{ project_module }}/KEYS
The vote will be open for at least 72 hours or until the necessary number
of votes are reached.
@@ -50,4 +51,4 @@ Please vote accordingly:
[ ] -1 disapprove with the reason
Thanks,
The Apache {{ project_name }} Team
The Apache {{ project_name }} (Incubating) Team

View File

@@ -32,7 +32,7 @@ else
SUPERSET_VERSION="${1}"
SUPERSET_RC="${2}"
SUPERSET_PGP_FULLNAME="${3}"
SUPERSET_RELEASE_RC_TARBALL="apache-superset-${SUPERSET_VERSION_RC}-source.tar.gz"
SUPERSET_RELEASE_RC_TARBALL="apache-superset-incubating-${SUPERSET_VERSION_RC}-source.tar.gz"
fi
SUPERSET_VERSION_RC="${SUPERSET_VERSION}rc${SUPERSET_RC}"

View File

@@ -22,7 +22,7 @@ if [ -z "${SUPERSET_VERSION_RC}" ] || [ -z "${SUPERSET_SVN_DEV_PATH}" ] || [ -z
exit 1
fi
SUPERSET_RELEASE_RC=apache-superset-"${SUPERSET_VERSION_RC}"
SUPERSET_RELEASE_RC=apache-superset-incubating-"${SUPERSET_VERSION_RC}"
SUPERSET_RELEASE_RC_TARBALL="${SUPERSET_RELEASE_RC}"-source.tar.gz
SUPERSET_RELEASE_RC_BASE_PATH="${SUPERSET_SVN_DEV_PATH}"/"${SUPERSET_VERSION_RC}"
SUPERSET_RELEASE_RC_TARBALL_PATH="${SUPERSET_RELEASE_RC_BASE_PATH}"/"${SUPERSET_RELEASE_RC_TARBALL}"
@@ -32,9 +32,9 @@ mkdir -p "${SUPERSET_SVN_DEV_PATH}"/"${SUPERSET_VERSION_RC}"
# Clone superset from tag to /tmp
cd /tmp
git clone --depth 1 --branch ${SUPERSET_VERSION_RC} https://github.com/apache/superset.git
git clone --depth 1 --branch ${SUPERSET_VERSION_RC} https://github.com/apache/incubator-superset.git
mkdir -p "${HOME}/${SUPERSET_VERSION_RC}"
cd superset && \
cd incubator-superset && \
# Check RC version
if ! jq -e --arg SUPERSET_VERSION $SUPERSET_VERSION '.version == $SUPERSET_VERSION' superset-frontend/package.json

View File

@@ -1,318 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Release Notes for Superset 0.38
## What's new
- [New features](#new-features)
- [Bugfixes](#bugfixes)
- [Breaking Changes](#breaking-changes)
- [Complete Changelog](#complete-changelog)
### New features
**DATABASES, DATASETS, QUERIES**
[SIP-40] Proposal for Custom Error Messages #9194 ([SIP](https://github.com/apache/superset/issues/9194))
- feat: improve presto query perf (#[11069](https://github.com/apache/superset/pull/11069))
- feat: more specific presto error messages (#[11099](https://github.com/apache/superset/pull/11099))
- feat: refactor error components and add database issue code (#[10473](https://github.com/apache/superset/pull/10473))
- feat: welcome presto to the suite of tested databases (#[10498](https://github.com/apache/superset/pull/10498))
Certification of Data Entities ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/73))
- feat: add certification to metrics (#[10630](https://github.com/apache/superset/pull/10630))
- feat: add extra column to tables and sql_metrics (#[10592](https://github.com/apache/superset/pull/10592))
- feat: bump superset-ui for certified tag (#[10650](https://github.com/apache/superset/pull/10650))
Database CRUD screen refactor/redesign ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/14))
- feat: add/edit database modal form sections UI (#[10745](https://github.com/apache/superset/pull/10745))
- feat: base tabbed modal for new database CRUD UI (#[10668](https://github.com/apache/superset/pull/10668))
Database list view refactor/redesign ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/55))
- feat: database delete warning (#[10800](https://github.com/apache/superset/pull/10800))\
<kbd><img alt="10800" src="media/10800.gif" width="400"/></kbd>
- feat: filters for database list view (#[10772](https://github.com/apache/superset/pull/10772))
- feat: SIP-34 table list view for databases (#[10705](https://github.com/apache/superset/pull/10705))
Database list view refactor/redesign #55 ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/55))
- feat(api): database schemas migration to new API (#[10436](https://github.com/apache/superset/pull/10436))
- feat(database): POST, PUT, DELETE API endpoints (#[10741](https://github.com/apache/superset/pull/10741))
- feat(databases): test connection api (#[10723](https://github.com/apache/superset/pull/10723))
Datasets CRUD screen refactor/redesign ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/13))
- feat: dataset editor improvements (#[10444](https://github.com/apache/superset/pull/10444))\
<kbd><img alt="10444" src="media/10444.gif" width="400"/></kbd>
Datasets CRUD screen refactor/redesign #13 ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/13))
- feat(datasource): remove deleted columns and update column type on metadata refresh (#[10619](https://github.com/apache/superset/pull/10619))\
<kbd><img alt="10619" src="media/10619.png" width="400"/></kbd>
Datasets list view refactor/redesign #12 ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/12))
- feat: update dataset editor modal (#[10347](https://github.com/apache/superset/pull/10347))\
<kbd><img alt="10347" src="media/10347.gif" width="400"/></kbd>
- feat(datasets): REST API bulk delete (#[11237](https://github.com/apache/superset/pull/11237))
Saved queries CRUD screen + list view refactor/redesign #15 ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/15))
- feat: CRUD REST API for saved queries (#[10777](https://github.com/apache/superset/pull/10777))
- feat: saved query list actions (#[11109](https://github.com/apache/superset/pull/11109))\
<kbd><img alt="11109" src="media/11109.gif" width="400"/></kbd>
- feat: saved query list view + sort/filters (#[11005](https://github.com/apache/superset/pull/11005))
- feat: SavedQuery REST API for bulk delete and new API fields (#[10793](https://github.com/apache/superset/pull/10793))
- feat: update saved query backend routing + add savedquery list (#[10922](https://github.com/apache/superset/pull/10922))
- feat(saved_queries): add custom api filter for all string & text fields (#[11031](https://github.com/apache/superset/pull/11031))
Other features
- feat: dataset REST API for distinct values (#[10595](https://github.com/apache/superset/pull/10595))
**EXPLORE, CHARTS, DASHBOARDS**
[SIP-40] Proposal for Custom Error Messages #9194 ([SIP](https://github.com/apache/superset/issues/9194))
- feat: add download as image button to explore (#[10297](https://github.com/apache/superset/pull/10297))\
<kbd><img alt="10297" src="media/10297.gif" width="400"/></kbd>
[SIP-34] Proposal to establish a new design direction, system, and process for Superset ([SIP](https://github.com/apache/superset/issues/8976))
- feat: SIP-34 explore save modal (#[10355](https://github.com/apache/superset/pull/10355))\
<kbd><img alt="10355" src="media/10355.gif" width="400"/></kbd>
Charts list view refactor/redesign ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/47))
- feat(charts): modify custom api filter to include more fields (#[11054](https://github.com/apache/superset/pull/11054))
Echarts integration ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/48))
- feat: add linear color scale to sunburst chart (#[10474](https://github.com/apache/superset/pull/10474))\
<kbd><img alt="10474" src="media/10474.gif" width="400"/></kbd>
- feat: add optional prophet forecasting functionality to chart data api (#[10324](https://github.com/apache/superset/pull/10324))
- feat(viz): add ECharts Timeseries chart (#[10752](https://github.com/apache/superset/pull/10752))\
<kbd><img alt="10752" src="media/10752.gif" width="400"/></kbd>
Improvements to cache handling #74 ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/74))
- feat: add ECharts Pie chart (#[10966](https://github.com/apache/superset/pull/10966))\
<kbd><img alt="10966" src="media/10966.gif" width="400"/></kbd>
- feat: implement cache invalidation api (#[10761](https://github.com/apache/superset/pull/10761))
- feat: Adding extra_filters to warm_up_cache (#[10675](https://github.com/apache/superset/pull/10675))
[SIP-34] Proposal to establish a new design direction, system, and process for Superset ([SIP](https://github.com/apache/superset/issues/8976))
- feat: add favorite star to dashboard and chart lists (#[10510](https://github.com/apache/superset/pull/10510))\
<kbd><img alt="10510" src="media/10510.png" width="400"/></kbd>
- feat: use svg for checkbox component (#[10799](https://github.com/apache/superset/pull/10799))\
<kbd><img alt="10799" src="media/10799.gif" width="400"/></kbd>
Other features
- feat(table-viz): translation and metric column header align right (#[10549](https://github.com/apache/superset/pull/10549))
- feat: adding dashboard toggle fullscreen button (#[10840](https://github.com/apache/superset/pull/10840))\
<kbd><img alt="10840" src="media/10840.png" width="400"/></kbd>
- feat: enable ETag header for dashboard GET requests (#[10963](https://github.com/apache/superset/pull/10963))
- feat: move ace-editor and mathjs to async modules (#[10837](https://github.com/apache/superset/pull/10837))
- feat: server side dashboard css for less repaint (#[10850](https://github.com/apache/superset/pull/10850))
- feat: use shorten url in standalone iframe (#[10651](https://github.com/apache/superset/pull/10651))
**SQL LAB**
- feat: Adding table comment and columns comment for SQLLab (#[10844](https://github.com/apache/superset/pull/10844))
**SYSTEM, OTHER**
- feat: add TXT as default CSV extension (#[10371](https://github.com/apache/superset/pull/10371))
[SIP-34] Proposal to establish a new design direction, system, and process for Superset ([SIP](https://github.com/apache/superset/issues/8976))
- feat: card view bulk select (#[10607](https://github.com/apache/superset/pull/10607))
- feat: custom favorite filter for dashboards, charts and saved queries (#[11083](https://github.com/apache/superset/pull/11083))
- feat: SIP-34 card/grid views for dashboards and charts (#[10526](https://github.com/apache/superset/pull/10526))
- feat(listviews): SIP-34 Bulk Select (#[10298](https://github.com/apache/superset/pull/10298))
- feat(listviews): SIP-34 filters for charts, dashboards, datasets (#[10335](https://github.com/apache/superset/pull/10335))
[SIP-40] Proposal for Custom Error Messages #9194 ([SIP](https://github.com/apache/superset/issues/9194))
- feat: update timeout error UX (#[10274](https://github.com/apache/superset/pull/10274))
Alerts (send notification when a condition is met) ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/54))
- feat: add test email functionality to SQL-based email alerts (#[10476](https://github.com/apache/superset/pull/10476))
- feat: refractored SQL-based alerting framework (#[10605](https://github.com/apache/superset/pull/10605))
[SIP-34] Proposal to establish a new design direction, system, and process for Superset ([SIP](https://github.com/apache/superset/issues/8976))
- feat: adding all icons from the design system to the codebase (#[11033](https://github.com/apache/superset/pull/11033))
- feat: storybook for Icon component (#[10515](https://github.com/apache/superset/pull/10515))
[SIP-48] Using Ant Design as our primary component library ([SIP](https://github.com/apache/superset/issues/10254))
- feat: Add antd to the codebase (#[10508](https://github.com/apache/superset/pull/10508))
Alerts (send notification when a condition is met) ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/54))
- feat: updated email format for SQL-based email alerts (#[10512](https://github.com/apache/superset/pull/10512))
Superset Component library - Phase 1 ([Roadmap](https://github.com/apache-superset/superset-roadmap/issues/23))
- feat: adding Storybook to Superset (#[10383](https://github.com/apache/superset/pull/10383))
Other
- feat: Allow tests files in /src (plus Label component tests) (#[10634](https://github.com/apache/superset/pull/10634))
- feat: Getting fancier with Storybook (#[10647](https://github.com/apache/superset/pull/10647))
### Bugfixes
- fix(permissions): alpha role has all full features (#[10241](https://github.com/apache/superset/pull/10241))
- fix: broken glyphicons used in react-json-schema (#[10267](https://github.com/apache/superset/pull/10267))
- fix: add additional ui tweaks (#[10275](https://github.com/apache/superset/pull/10275))
- fix: saving custom CSS correctly (#[10289](https://github.com/apache/superset/pull/10289))
- fix: fetch datasets list after dataset created successfully (#[10290](https://github.com/apache/superset/pull/10290))
- fix: update community Slack link (#[10360](https://github.com/apache/superset/pull/10360))
- fix: allow creating table option and remove schema requirement in dataset add modal (#[10369](https://github.com/apache/superset/pull/10369))
- fix(datasets): sort and humanized modified by (#[10380](https://github.com/apache/superset/pull/10380))
- fix(api): fixes perf on charts and introduces sorting by database on datasets (#[10392](https://github.com/apache/superset/pull/10392))
- fix(api): fixes openapi spec errors and adds a test to validate all spec (#[10393](https://github.com/apache/superset/pull/10393))
- fix(charts): disable CSRF for chart data endpoint (#[10397](https://github.com/apache/superset/pull/10397))
- fix: dataset list filters bug (#[10398](https://github.com/apache/superset/pull/10398))
- fix: remove FAB rendered menu in favor of react based one (#[10401](https://github.com/apache/superset/pull/10401))
- fix: show label for filters in filter box in explore (#[10412](https://github.com/apache/superset/pull/10412))
- fix: Implement updates to SQL-based email alerts (#[10454](https://github.com/apache/superset/pull/10454))
- fix(presto): Handle ROW data stored as string (#[10456](https://github.com/apache/superset/pull/10456))
- fix: change "add new slice" copy to "add new chart" (#[10457](https://github.com/apache/superset/pull/10457))
- fix(sqllab): button width isn't wide enough for 'Run Selection' (#[10461](https://github.com/apache/superset/pull/10461))
- fix: timeout error message (#[10478](https://github.com/apache/superset/pull/10478))
- fix: enforce mandatory chart name on save and edit (#[10482](https://github.com/apache/superset/pull/10482))
- fix: More tweaks needed after adding Doctype tag (#[10504](https://github.com/apache/superset/pull/10504))
- fix: explore panel missing padding (#[10505](https://github.com/apache/superset/pull/10505))
- fix: refactored SQL-based alerts to not pass sqlalchemy objects as args (#[10506](https://github.com/apache/superset/pull/10506))
- fix(sqllab): Handle long table names in SQL Lab (#[10518](https://github.com/apache/superset/pull/10518))
- fix: make SQL-based alert email links user friendly (#[10519](https://github.com/apache/superset/pull/10519))
- fix(dashboard): changing the chart title, except not (#[10527](https://github.com/apache/superset/pull/10527))
- fix: misaligned LimitControl buttons and port jsx->tsx (#[10529](https://github.com/apache/superset/pull/10529))
- fix: Resolves #10535 (#[10536](https://github.com/apache/superset/pull/10536))
- fix: add retry to SQL-based alerting celery task (#[10542](https://github.com/apache/superset/pull/10542))
- fix: Updating Dockerfile to work with updated python requirements. (#[10550](https://github.com/apache/superset/pull/10550))
- fix(thumbnails): missing field, logging and new config var (#[10562](https://github.com/apache/superset/pull/10562))
- fix: add translate for dropdown menu (#[10573](https://github.com/apache/superset/pull/10573))
- fix: error message modal overflow (#[10580](https://github.com/apache/superset/pull/10580))
- fix: add None checking to cast_to_num (#[10584](https://github.com/apache/superset/pull/10584))
- fix: removing unsupported modal sizes (#[10625](https://github.com/apache/superset/pull/10625))
- fix: remove duplicated params and cache_timeout from list_columns; add viz_type to list_columns (#[10643](https://github.com/apache/superset/pull/10643))
- fix: controls scroll issue (#[10644](https://github.com/apache/superset/pull/10644))
- fix(db_engine_specs): improve Presto column type matching (#[10658](https://github.com/apache/superset/pull/10658))
- fix(db_engine_specs): mysql longtext type should not be numeric (#[10661](https://github.com/apache/superset/pull/10661))
- fix: change public role like gamma procedure (#[10674](https://github.com/apache/superset/pull/10674))
- fix(sqllab): log exceptions caused by the user as debug and not error (#[10676](https://github.com/apache/superset/pull/10676))
- fix: only call signal if executing on the main thread (#[10677](https://github.com/apache/superset/pull/10677))
- fix: layout flexiness (#[10681](https://github.com/apache/superset/pull/10681))
- fix: SubMenu css (#[10682](https://github.com/apache/superset/pull/10682))
- fix: dashboard extra filters (#[10692](https://github.com/apache/superset/pull/10692))
- fix: shorten url with extra request parameters (#[10693](https://github.com/apache/superset/pull/10693))
- fix: card view failed cypress tests (#[10699](https://github.com/apache/superset/pull/10699))
- fix: deprecation warnings due to invalid escape sequences. (#[10710](https://github.com/apache/superset/pull/10710))
- fix: move menu reorg logic from crud app into Menu component (#[10717](https://github.com/apache/superset/pull/10717))
- fix: local docker deployment (#[10738](https://github.com/apache/superset/pull/10738))
- Fix: Rejiggering some dependencies, trying to get CI to pass (#[10747](https://github.com/apache/superset/pull/10747))
- fix(presto): default unknown types to string type (#[10753](https://github.com/apache/superset/pull/10753))
- fix: add validator information to email/slack alerts (#[10762](https://github.com/apache/superset/pull/10762))
- fix: re-installing local superset in cache image (#[10766](https://github.com/apache/superset/pull/10766))
- fix: can not type `0.05` in `TextControl` (#[10778](https://github.com/apache/superset/pull/10778))
- fix: MVC show saved query (#[10781](https://github.com/apache/superset/pull/10781))
- fix: disable domain sharding on explore view (#[10787](https://github.com/apache/superset/pull/10787))
- fix: Database API missing allow none on fields (#[10795](https://github.com/apache/superset/pull/10795))
- fix: bump node version on Dockerfile to be on par with docker-compose (#[10813](https://github.com/apache/superset/pull/10813))
- fix(tests): export dataset tests fails with presto (#[10818](https://github.com/apache/superset/pull/10818))
- fix: use nullpool in the celery workers (#[10819](https://github.com/apache/superset/pull/10819))
- fix: Making the database read-only (#[10823](https://github.com/apache/superset/pull/10823))
- fix(databases): test connection api endpoint (#[10824](https://github.com/apache/superset/pull/10824))
- fix: update the time filter for 'Last Year' option in explore (#[10829](https://github.com/apache/superset/pull/10829))
- fix(test): missing auth on tests (#[10842](https://github.com/apache/superset/pull/10842))
- fix(cypress): wait for filterValues request (#[10884](https://github.com/apache/superset/pull/10884))
- fix: superset alerting misc fixes (#[10891](https://github.com/apache/superset/pull/10891))
- fix(cypress): prevent CI failure on codecov failure (#[10892](https://github.com/apache/superset/pull/10892))
- fix: front end CI tests and test runner (#[10897](https://github.com/apache/superset/pull/10897))
- fix: babel script broken by format string (#[10902](https://github.com/apache/superset/pull/10902))
- fix: several disabled pylint rules in models/helpers.py (#[10909](https://github.com/apache/superset/pull/10909))
- fix: spelling in docs homepage (#[10912](https://github.com/apache/superset/pull/10912))
- fix: address all disabled pylint checks in charts/api.py (#[10932](https://github.com/apache/superset/pull/10932))
- fix: use nullpool even for user lookup in the celery (#[10938](https://github.com/apache/superset/pull/10938))
- fix: update pylint disabled checks in common/query_context.py (#[10941](https://github.com/apache/superset/pull/10941))
- fix: setting specific exceptions common/query_context.py (#[10942](https://github.com/apache/superset/pull/10942))
- fix: re-enabling several globally disabled lint rules (#[10957](https://github.com/apache/superset/pull/10957))
- fix: removed disabled lint rule `too-many-locals` in connectors/base/models.py (#[10958](https://github.com/apache/superset/pull/10958))
- fix: typo in prefer typescript (#[10959](https://github.com/apache/superset/pull/10959))
- fix: pylint checks in connectors/sqla/models.py (#[10974](https://github.com/apache/superset/pull/10974))
- fix: pylint disabled rules in dashboard/api.py (#[10976](https://github.com/apache/superset/pull/10976))
- fix: changes a pylint check in dashboard module (#[10978](https://github.com/apache/superset/pull/10978))
- fix: changed disabled rules in datasets module (#[10979](https://github.com/apache/superset/pull/10979))
- fix: Add Item Overflow on Dataset Editor (#[10983](https://github.com/apache/superset/pull/10983))
- fix: enable pylint rules in db_engine_specs module (#[10998](https://github.com/apache/superset/pull/10998))
- fix: enable several pylint rules partially in db_engines_specs module (#[11000](https://github.com/apache/superset/pull/11000))
- fix: unbreak ci (#[11003](https://github.com/apache/superset/pull/11003))
- fix: timer component, fixes #10849, closes #11002 (#[11004](https://github.com/apache/superset/pull/11004))
- fix: menu shows a 0 when there are not settings (#[11009](https://github.com/apache/superset/pull/11009))
- fix: reenable pylint rule `unused-import` in charts and connectors modules (#[11014](https://github.com/apache/superset/pull/11014))
- fix: query search low privileged user search access denied (#[11017](https://github.com/apache/superset/pull/11017))
- fix(api): unable to delete virtual dataset, wrong permission name (#[11019](https://github.com/apache/superset/pull/11019))
- fix: [dashboard] should not show edit button when user has no edit permit (#[11024](https://github.com/apache/superset/pull/11024))
- fix: dashboard edit button (again) (#[11029](https://github.com/apache/superset/pull/11029))
- fix: sql lab autocomplete width (#[11063](https://github.com/apache/superset/pull/11063))
- fix: fix table existence validation function (#[11066](https://github.com/apache/superset/pull/11066))
- fix: database list checkboxes (#[11068](https://github.com/apache/superset/pull/11068))
- fix: Adding timeout to flaky cypress test, to wait for animation to complete (#[11074](https://github.com/apache/superset/pull/11074))
- fix: surface connection error messages on the client (#[11077](https://github.com/apache/superset/pull/11077))
- fix(jest): using UTC mock date (#[11079](https://github.com/apache/superset/pull/11079))
- fix: double scroll bars on dataset editor (#[11095](https://github.com/apache/superset/pull/11095))
- fix: echarts timeseries groupby (#[11103](https://github.com/apache/superset/pull/11103))
- fix: Disabling timezone of dataframe before passing Prophet (#[11107](https://github.com/apache/superset/pull/11107))
- fix(chart-data-api): ignore missing filters (#[11112](https://github.com/apache/superset/pull/11112))
- fix: alembic migration error msg trying to delete constraint on tables (#[11115](https://github.com/apache/superset/pull/11115))
- fix: remove extra flash import (#[11121](https://github.com/apache/superset/pull/11121))
- fix: Revert "Replace reactable with DataTable from superset-ui in QueryTable (#10981)" (#[11125](https://github.com/apache/superset/pull/11125))
- fix: SpatialControl popover won't open (#[11127](https://github.com/apache/superset/pull/11127))
- fix: Alembic migration 18532d70ab98 (#[11136](https://github.com/apache/superset/pull/11136))
- fix(examples): missing expressions in birth_names (#[11141](https://github.com/apache/superset/pull/11141))
- fix: Fix Time Column dropdown for date filter (#[11149](https://github.com/apache/superset/pull/11149))
- fix(dataset): update user list endpoint (#[11221](https://github.com/apache/superset/pull/11221))
- fix(crud): set default extra value (#[11262](https://github.com/apache/superset/pull/11262))
- fix(sqla): allow 'unknown' type queries in explore view (#[11365](https://github.com/apache/superset/pull/11365))
- fix: prior npm font source had a glitch (#[11724](https://github.com/apache/superset/pull/11724))
## Breaking Changes
[List of backwards incompatible changes](https://github.com/apache/superset/blob/master/UPDATING.md#0380)
## Complete Changelog
For the complete changelog please see [apache/superset/CHANGELOG.md](https://github.com/apache/superset/blob/master/CHANGELOG.md)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.7 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.1 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.8 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 131 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.9 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 491 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 887 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 414 KiB

View File

@@ -1,226 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Release Notes for Superset 1.0.0
Superset 1.0 is a ***huge milestone*** for Apache Superset. This release holds a higher quality standard than any previous version and raises the bar for releases to come. Superset 1.0 improves on usability and delivers a new set of long awaited features focused on five major themes:
- [**User Experience**](#user-experience)
- [**Developer Experience**](#developer-experience)
- [**Performance**](#performance)
- [**New Features**](#new-features)
- [**Feature Flags**](#feature-flags)
- [**Stability and Bugfixes**](#stability-and-bugfixes)
# User Experience
We have taken the Superset user experience to the next level with a much simpler, more intuitive UI.
Lists of Charts and Dashboards can be shown in a new format using a **thumbnail grid**. This makes it easier to discover and find Charts and Dashboards, especially when there are a lot of them in your Superset instance.
<kbd><img alt="card view" src="media/dashboard_card_view.jpg" width="600"/></kbd>
Visualization controls have been updated to create a more consistent and modernized interface. Refactoring includes updates to the content/labeling/ordering of these controls as well as updating their styling and simplifying their interactions and layout. This will line us up for features like drag-and-drop controls, and dynamically populated control inputs in future releases.
<kbd><img alt="explore ui" src="media/explore_ui.png" width="600"/></kbd>
Users also now have the ability to create and name a new dataset or update an existing dataset when moving from **SQL Lab** to **Explore**. When going back to SQL Lab, the underlying query for the dataset will be shown, allowing users to easily make changes and update their dataset when moving between SQL Lab and Explore.
The [Superset 1.0 documentation](https://superset.apache.org/docs/intro) has been updated to reflect the current design and functionality.
See related PR highlights [here](#user-experience-1).
# Developer Experience
Superset 1.0 makes it easier for developers to build, deploy, and maintain Superset functionality. This release is a major milestone in an ongoing effort to **modernize**, **consolidate**, and **simplify** the interface elements of Superset. Highly-used components have been visually updated, refactored to modern component libraries, and integrated with React Storybook for immediate visibility with consolidated test/styles. This improves the developer experience and creates a more consistent and modern aesthetic for the user.
The **REST API** continues to see improvement as well, with new endpoints and improvements to existing endpoints. More details [here](https://superset.apache.org/docs/rest-api).
<kbd><img alt="rest api" src="media/rest_api.png" width="600"/></kbd>
The Swagger interface can be used to explore and try out these changes.
<kbd><img alt="rest api swagger" src="media/rest_api_swagger.png" width="600"/></kbd>
See related PR highlights [here](#developer-experience-1).
# Performance
This the most performant Superset release to date, with enhanced scalability and efficiency. In addition to many small tweaks, **Charts** and **SQL Lab** both now support **asynchronous data loading**. Users will **feel** this improvement when loading query results, especially when there are many charts in a Dashboard or when queries are long running.
See related PR highlights [here](#performance-1).
# New Features
Superset 1.0 includes a slew of new features and other enhancements. **New visualization plugin architecture** will make it easier to build, test, style and configure custom visualizations for Superset. **Dynamic viz plugin** imports will allow Superset to load data viz plugins, allowing developers to more easily use or share their custom plugins and **load custom plugins on-demand** from anywhere on the web. Superset has also adopted [**Apache ECharts**](https://echarts.apache.org/en/index.html) as the core library for new visualizations.
<kbd><img alt="timeseries prophet" src="media/echarts_timeseries_prophet.png" width="600"/></kbd>
Along with the **Echarts integration**, we are also introducing a better set of annotation features. **Formula annotation**, which allows users to plot any mathematical function on the chart; **Interval and Event annotations**, which allow users to add context to the trend in Time-Series; and **Line annotation**, which use a predefined chart as the source of annotation.
<kbd><img alt="annotations" src="media/annotations.png" width="600"/></kbd>
A redesigned home screen provides a **personalized landing page** that the user sees when they log into Superset, improving discoverability. It shows items relevant to the user (Charts, Dashboards, Saved Queries, etc) and serves as a hub to discover content and access recent items quickly.
<kbd><img alt="home screen" src="media/home_screen.png" width="600"/></kbd>
Charts on dashboards have been updated to concisely show which **filters** are in scope, have been applied, or are in error. Context around filter changes has been increased by highlighting scoped charts when a filter is selected and increasing visibility of charts with incompatible filters applied.
<kbd><img alt="filter status" src="media/filter_status.png" width="600"/></kbd>
**Alerts and reporting** have received a robust backend and UI overhaul.
<kbd><img alt="alerts reports" src="media/alerts_reports_2.png" width="600"/></kbd>
<kbd><img alt="alerts reports edit" src="media/alerts_reports_3.png" width="600"/></kbd>
See related PR highlights [here](#new-features-1).
# Feature Flags
Some of the new features in this release are disabled by default. Each has a feature flag in `config.py` and some require configuration of additional backend dependencies (e.g. celery, SMTP server, etc). The following table will help you get started on the list of features and their associated flags & dependencies. Feel free to contribute and expand this list.
| Feature | Feature Flag | Dependencies | Documentation
| --- | --- | --- | --- |
| Global Async Queries | `GLOBAL_ASYNC_QUERIES: True` | Redis 5.0+, celery workers configured and running | [Extra documentation](https://github.com/apache/superset/blob/master/CONTRIBUTING.md#async-chart-queries )
| Dashboard Native Filters | `DASHBOARD_NATIVE_FILTERS: True` | |
| Alerts & Reporting | `ALERT_REPORTS: True` | [Celery workers configured & celery beat process](https://superset.apache.org/docs/installation/async-queries-celery) |
| Homescreen Thumbnails | `THUMBNAILS: TRUE, THUMBNAIL_CACHE_CONFIG: CacheConfig = { "CACHE_TYPE": "null", "CACHE_NO_NULL_WARNING": True}`| selenium, pillow 7, celery |
| Row Level Security | `ROW_LEVEL_SECURITY` | | [Extra Documentation](https://superset.apache.org/docs/security#row-level-security)
| Dynamic Viz Plugin Import | `DYNAMIC_PLUGINS: True` | |
# Stability and Bugfixes
This release includes **hundreds** of bugfixes and stability enhancements. Future major releases will have a continued emphasis on providing a stable and bug-free experience for the user.
# PR Highlights
Below is a highlight of the PRs included in this update. The full list is much longer, and can be found [here](apache/incubator-superset/CHANGELOG.md).
## User Experience
- Revert "refactor: Remove usages of reactable from TimeTable (#11046)" (#[11150](https://github.com/apache/incubator-superset/pull/11150))
- style: Restoring menu highlights (#[12024](https://github.com/apache/incubator-superset/pull/12024))
- style: eslint curly rule (#[11913](https://github.com/apache/incubator-superset/pull/11913))
- style: remove react bootstrap fade component (#[11843](https://github.com/apache/incubator-superset/pull/11843))
- style: dark filter popover background (#[11611](https://github.com/apache/incubator-superset/pull/11611))
<!--### Card Layout-->
### Explore Controls
- feat: Global nav menus open on hover (#[12025](https://github.com/apache/incubator-superset/pull/12025))
- feat: add hook for dataset health check (#[11970](https://github.com/apache/incubator-superset/pull/11970))
- feat: add modal to import dashboards (#[11924](https://github.com/apache/incubator-superset/pull/11924))
- feat: add modal to import charts (#[11956](https://github.com/apache/incubator-superset/pull/11956))
- feat: add modal to import datasets (#[11910](https://github.com/apache/incubator-superset/pull/11910))
- feat: add modal to import databases (#[11884](https://github.com/apache/incubator-superset/pull/11884))
- feat: Results table on Explore view (#[11854](https://github.com/apache/incubator-superset/pull/11854))
- feat: report/alert list CRUD view (#[11802](https://github.com/apache/incubator-superset/pull/11802))
- feat: filter by me on CRUD list view (#[11683](https://github.com/apache/incubator-superset/pull/11683))
- feat(explore-datasource): add new datasource tab to explore view (#[12008](https://github.com/apache/incubator-superset/pull/12008))
- feat(explore): time picker enhancement (#[11418](https://github.com/apache/incubator-superset/pull/11418))
- feat(explore): metrics and filters controls redesign (#[12095](https://github.com/apache/incubator-superset/pull/12095))
- feat(explore): change save button text when users cannot override (#[11281](https://github.com/apache/incubator-superset/pull/11281))
- feat(explore): update the order of datasource dropdown menu (#[11424](https://github.com/apache/incubator-superset/pull/11424))
### SQL Lab
- feat: initial custom error messages for SQL Lab (#[12080](https://github.com/apache/incubator-superset/pull/12080))
- feat: show missing parameters in query (#[12049](https://github.com/apache/incubator-superset/pull/12049))
- feat: Sqllab to Explore UX improvements (#[11755](https://github.com/apache/incubator-superset/pull/11755))
- feat: add Postgres SQL validator (#[11538](https://github.com/apache/incubator-superset/pull/11538))
- feat: run BQ queries as single statement (#[11904](https://github.com/apache/incubator-superset/pull/11904))
- feat: Adding SaveDatasetModal component (#[11861](https://github.com/apache/incubator-superset/pull/11861))
- feat: list view filters for Query History (#[11702](https://github.com/apache/incubator-superset/pull/11702))
- feat: Sqllab to Explore UX improvements api changes (#[11836](https://github.com/apache/incubator-superset/pull/11836))
- feat: SQL preview modal for Query History (#[11634](https://github.com/apache/incubator-superset/pull/11634))
- feat: Query History CRUD list view (#[11574](https://github.com/apache/incubator-superset/pull/11574))
- feat: add UUID column to saved_query for export/import (#[11397](https://github.com/apache/incubator-superset/pull/11397))
- feat: saved queries with execution info (#[11391](https://github.com/apache/incubator-superset/pull/11391))
- feat: saved query preview modal (#[11135](https://github.com/apache/incubator-superset/pull/11135))
- feat(saved queries): security perm simplification (#[11764](https://github.com/apache/incubator-superset/pull/11764))
- feat(welcome): add SQL snippets to saved queries card (#[11678](https://github.com/apache/incubator-superset/pull/11678))
### Documentation
- docs: restored page on roles in security, executed other touchups, and… (#[11978](https://github.com/apache/incubator-superset/pull/11978))
- docs: 0.38.0 CHANGELOG and UPDATING (#[11809](https://github.com/apache/incubator-superset/pull/11809))
- docs: add Xite to users list (#[11775](https://github.com/apache/incubator-superset/pull/11775))
- docs: fix docker config paths (#[11703](https://github.com/apache/incubator-superset/pull/11703))
- docs: add support for Google Analytics on superset.apache.org (#[11613](https://github.com/apache/incubator-superset/pull/11613))
- docs: add Avesta to users list (#[11551](https://github.com/apache/incubator-superset/pull/11551))
- docs: add Xendit to users list (#[11361](https://github.com/apache/incubator-superset/pull/11361))
- docs: fix db drivers PG and ES (#[11504](https://github.com/apache/incubator-superset/pull/11504))
- docs: improve docs and update apache release scripts (#[11457](https://github.com/apache/incubator-superset/pull/11457))
- docs: Update Dremio doc with Arrow Flight connection details (#[11422](https://github.com/apache/incubator-superset/pull/11422))
- docs: Add semantic prefix info for PR titles (#[11398](https://github.com/apache/incubator-superset/pull/11398))
- docs: fix typo (#[11129](https://github.com/apache/incubator-superset/pull/11129))
- docs: add VLMedia to users list (#[11386](https://github.com/apache/incubator-superset/pull/11386))
- docs: Fix typo in installing from scratch (#[11026](https://github.com/apache/incubator-superset/pull/11026))
- docs: add a roadmap page pointing to SIP-53 (#[11169](https://github.com/apache/incubator-superset/pull/11169))
- docs: add note about migration in #11098 to Changelog (#[11256](https://github.com/apache/incubator-superset/pull/11256))
- docs: Add utair to list of companies using Apache Superset (#[11261](https://github.com/apache/incubator-superset/pull/11261))
- docs: don't link to database engine pages (#[11242](https://github.com/apache/incubator-superset/pull/11242))
- docs: Added Dremio to superset home page (#[11231](https://github.com/apache/incubator-superset/pull/11231))
- docs: typo (#[11152](https://github.com/apache/incubator-superset/pull/11152))
- docs: add API page to docs using Swagger UI (#[11154](https://github.com/apache/incubator-superset/pull/11154))
- docs(release): small fixes and deprecate apache docs (#[11827](https://github.com/apache/incubator-superset/pull/11827))
- docs(releasing): update release automation and docs for Apache graduation (#[12117](https://github.com/apache/incubator-superset/pull/12117))
- feat: Add new default dashboard (#[12044](https://github.com/apache/incubator-superset/pull/12044))
## Developer Experience
- feat: add endpoint to export saved queries using new format (#[11447](https://github.com/apache/incubator-superset/pull/11447))
### Component Library
- build(deps-dev): update cypress to 5.5.0, improvements for running locally (#[11603](https://github.com/apache/incubator-superset/pull/11603))
- build(deps-dev): bump less from 3.9.0 to 3.12.2 in /superset-frontend (#[11597](https://github.com/apache/incubator-superset/pull/11597))
- build(deps-dev): bump @storybook/addon-essentials in /superset-frontend (#[11573](https://github.com/apache/incubator-superset/pull/11573))
- build(deps-dev): bump node-fetch in /superset-frontend (#[11485](https://github.com/apache/incubator-superset/pull/11485))
- build(deps): bump @emotion/core in /superset-frontend (#[11518](https://github.com/apache/incubator-superset/pull/11518))
- build(deps): bump core-js from 3.2.1 to 3.6.5 in /superset-frontend (#[11522](https://github.com/apache/incubator-superset/pull/11522))
- build(deps): bump object-path from 0.11.4 to 0.11.5 in /docs (#[11484](https://github.com/apache/incubator-superset/pull/11484))
- build(deps): bump lodash in /superset-frontend/cypress-base (#[11483](https://github.com/apache/incubator-superset/pull/11483))
- chore: bump nvd3 plugin version (#[11947](https://github.com/apache/incubator-superset/pull/11947))
- chore: bump pypi cryptography from 2.4.2 to 3.2.1 (#[11511](https://github.com/apache/incubator-superset/pull/11511))
## Performance
- perf: cache dashboard bootstrap data (#[11234](https://github.com/apache/incubator-superset/pull/11234))
- perf: speed up uuid column generation (#[11209](https://github.com/apache/incubator-superset/pull/11209))
- perf(api): improve API info performance (#[11346](https://github.com/apache/incubator-superset/pull/11346))
### Global Async Query Support
- feat(SIP-39): Async query support for charts (#[11499](https://github.com/apache/incubator-superset/pull/11499))
###New Features
### Improved Data Visualizations
- feat: add event and interval annotation support to chart data ep (#[11665](https://github.com/apache/incubator-superset/pull/11665))
- feat: add ECharts BoxPlot chart (#[11199](https://github.com/apache/incubator-superset/pull/11199))
- feat(sankey-viz): allow sort by metric and bump superset-ui packages (#[11626](https://github.com/apache/incubator-superset/pull/11626))
### Discovery and Navigation
- chore: Add home link to navbar (#[11851](https://github.com/apache/incubator-superset/pull/11851))
- chore(home-screen): fixes for loading states, flicker issue, and reduction of api calls (#[11557](https://github.com/apache/incubator-superset/pull/11557))
- feat: alert/report execution log list view (#[11937](https://github.com/apache/incubator-superset/pull/11937))
- feat: home screen mvp (#[11206](https://github.com/apache/incubator-superset/pull/11206))
- style: various little tweaks to home screen stylin' (#[11650](https://github.com/apache/incubator-superset/pull/11650))
### Alerts and Reporting
- feat: update alert/report icons and column order (#[12081](https://github.com/apache/incubator-superset/pull/12081))
- feat: add cron picker to AlertReportModal (#[12032](https://github.com/apache/incubator-superset/pull/12032))
- feat: alerts/reports add/edit modal (#[11770](https://github.com/apache/incubator-superset/pull/11770))
- feat: filters for alerts and reports list view (#[11900](https://github.com/apache/incubator-superset/pull/11900))
- feat(alert/report): add 'not null' condition option to modal (#[12077](https://github.com/apache/incubator-superset/pull/12077))
- feat(alerts/reports): remove SIP_34_UI feature flag (#[12085](https://github.com/apache/incubator-superset/pull/12085))
- feat(alerts/reports): add refresh action (#[12071](https://github.com/apache/incubator-superset/pull/12071))
- feat(alerts/reports): delete and bulk delete actions (#[12053](https://github.com/apache/incubator-superset/pull/12053))
- feat(releasing): support changelog csv export (#[11893](https://github.com/apache/incubator-superset/pull/11893))
## Complete Changelog
Backwards incompatible changes and can be found [here](../../UPDATING.md).
To see the complete changelog, see [apache/incubator-superset/CHANGELOG.md](https://github.com/apache/superset/blob/master/CHANGELOG.md)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 283 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 263 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 316 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 320 KiB

Some files were not shown because too many files have changed in this diff Show More