diff --git a/docs/docs/contributing/pkg-resources-migration.md b/docs/docs/contributing/pkg-resources-migration.md new file mode 100644 index 00000000000..6a0648a966e --- /dev/null +++ b/docs/docs/contributing/pkg-resources-migration.md @@ -0,0 +1,101 @@ + + +# pkg_resources Deprecation and Migration Guide + +## Background + +As of setuptools 81.0.0 (scheduled for removal around 2025-11-30), the `pkg_resources` API is deprecated and will be removed. This affects several packages in the Python ecosystem. + +## Current Status + +### Superset Codebase ✅ +The Superset codebase has already migrated away from `pkg_resources` to the modern `importlib.metadata` API: + +- `superset/db_engine_specs/__init__.py:36` - Uses `from importlib.metadata import entry_points` +- All entry point loading uses the modern API + +### Production Dependencies ⚠️ +Some third-party dependencies may still use `pkg_resources`: + +- **`clients` package** (Preset-specific): Uses `pkg_resources` in `const.py` +- Error path: `/usr/local/lib/python3.10/site-packages/clients/const.py:1` + +## Migration Path + +### Short-term Solution (Current) +Pin setuptools to version 80.x to prevent breaking changes: + +```python +# requirements/base.in +setuptools<81 +``` + +This prevents the removal of `pkg_resources` while dependent packages are updated. + +### Long-term Solution +Update all dependencies to use `importlib.metadata` instead of `pkg_resources`: + +#### Migration Example +**Old (deprecated):** +```python +import pkg_resources + +version = pkg_resources.get_distribution("package_name").version +entry_points = pkg_resources.iter_entry_points("group_name") +``` + +**New (recommended):** +```python +from importlib.metadata import version, entry_points + +pkg_version = version("package_name") +eps = entry_points(group="group_name") +``` + +## Action Items + +### For Preset Team +1. **Update `clients` package** to use `importlib.metadata` instead of `pkg_resources` +2. **Review other internal packages** for `pkg_resources` usage +3. **Test with setuptools >= 81.0.0** once all packages are migrated +4. **Monitor Datadog logs** for similar deprecation warnings + +### For Superset Maintainers +1. ✅ Already using `importlib.metadata` +2. Monitor third-party dependencies for updates +3. Update setuptools pin once ecosystem is ready + +## Timeline + +- **2025-11-30**: Expected removal of `pkg_resources` from setuptools +- **Before then**: All dependencies must migrate to `importlib.metadata` + +## References + +- [setuptools pkg_resources deprecation notice](https://setuptools.pypa.io/en/latest/pkg_resources.html) +- [importlib.metadata documentation](https://docs.python.org/3/library/importlib.metadata.html) +- [Migration guide](https://setuptools.pypa.io/en/latest/deprecated/pkg_resources.html) + +## Monitoring + +Track this issue in production using Datadog: +- Warning pattern: `pkg_resources is deprecated as an API` +- Component: `@component:app` +- Environment: `environment:production` diff --git a/requirements/base.in b/requirements/base.in index d110fa89314..9d527fba593 100644 --- a/requirements/base.in +++ b/requirements/base.in @@ -36,3 +36,9 @@ marshmallow-sqlalchemy>=1.3.0,<1.4.1 # needed for python 3.12 support openapi-schema-validator>=0.6.3 + +# Pin setuptools <81 until all dependencies migrate from pkg_resources to importlib.metadata +# pkg_resources is deprecated and will be removed in setuptools 81+ (around 2025-11-30) +# Known affected packages: Preset's 'clients' package +# See docs/docs/contributing/pkg-resources-migration.md for details +setuptools<81 diff --git a/requirements/base.txt b/requirements/base.txt index 076404726a4..c3abfaf43df 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -160,7 +160,6 @@ greenlet==3.1.1 # via # apache-superset (pyproject.toml) # shillelagh - # sqlalchemy gunicorn==23.0.0 # via apache-superset (pyproject.toml) h11==0.16.0 @@ -364,6 +363,8 @@ rsa==4.9.1 # via google-auth selenium==4.32.0 # via apache-superset (pyproject.toml) +setuptools==80.9.0 + # via -r requirements/base.in shillelagh==1.4.3 # via apache-superset (pyproject.toml) simplejson==3.20.1 diff --git a/requirements/development.txt b/requirements/development.txt index b479423133e..7543c18139d 100644 --- a/requirements/development.txt +++ b/requirements/development.txt @@ -368,7 +368,6 @@ greenlet==3.1.1 # apache-superset # gevent # shillelagh - # sqlalchemy grpcio==1.71.0 # via # apache-superset @@ -884,7 +883,7 @@ rsa==4.9.1 # google-auth ruff==0.8.0 # via apache-superset -secretstorage==3.4.0 +secretstorage==3.4.1 # via keyring selenium==4.32.0 # via @@ -892,8 +891,9 @@ selenium==4.32.0 # apache-superset semver==3.0.4 # via apache-superset-extensions-cli -setuptools==80.7.1 +setuptools==80.9.0 # via + # -c requirements/base-constraint.txt # nodeenv # pandas-gbq # pydata-google-auth