gh-91539: improve performance of get_proxies_environment #91566

eendebakpt · 2022-04-15T11:27:27Z

Improve performance of get_proxies_environment when there are many environment variables. Improvements depend on the number of environment variables, but the method is several times faster in this PR.

Fixes #91539

Performance test details

Results:

0.1634683609008789
0.024187326431274414

with

import os
import time
import urllib.request

if 0:
    os.environ={ f'{ii}': 1 for ii in range(8000)}
    os.environ.update({ f'{ii}_proxy': 1 for ii in range(30)})

def getproxies_environment():
    """Return a dictionary of scheme -> proxy server URL mappings.
    Scan the environment for variables named <scheme>_proxy;
    this seems to be the standard convention.  If you need a
    different way, you can pass a proxies dictionary to the
    [Fancy]URLopener constructor.
    """
    # in order to prefer lowercase variables, process environment in
    # two passes: first matches any, second pass matches lowercase only

    # select only environment variables which end in (after making lowercase) _proxy 
    candidate_names = [name for name in os.environ.keys() if len(name)>5 and name[-6]=='_'] # fast selection of candidates
    environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] 

    proxies = {}
    for name, value, name_lower in environment:
        if value and name_lower[-6:] == '_proxy':
            proxies[name_lower[:-6]] = value
    # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
    # (non-all-lowercase) as it may be set from the web server by a "Proxy:"
    # header from the client
    # If "proxy" is lowercase, it will still be used thanks to the next block
    if 'REQUEST_METHOD' in os.environ:
        proxies.pop('http', None)
    for name, value, name_lower in environment:
        if name[-6:] == '_proxy':
            if value:
                proxies[name_lower[:-6]] = value
            else:
                proxies.pop(name_lower[:-6], None)
    return proxies

nn=400
t0=time.time()
for ii in range(nn):
    urllib.request.getproxies_environment()
dt=time.time()-t0
print(dt)

t0=time.time()
for ii in range(nn):
    getproxies_environment()
dt=time.time()-t0
print(dt)

Issue: speed up urllib.request.getproxies_environment #91539

…vironment variables

…:eendebakpt/cpython into performance/getproxies_environment

Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

eendebakpt · 2022-05-18T08:53:49Z

@carljm Thanks for the suggestion. Benchmarks show it is just as fast, and much cleaner code.

carljm

Sorry I missed this in first review, but the name of the function has a typo (extra underscore) in the NEWS entry.

Also suggested an added comment. (Wouldn't have bothered with this if it was the only thing, but may be worth it if you are doing one more update to fix NEWS anyway.)

Thanks for the improvements to this function!

Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst

Lib/urllib/request.py

…gVuA.rst Co-authored-by: Carl Meyer <[email protected]>

Co-authored-by: Carl Meyer <[email protected]>

eendebakpt · 2022-08-10T20:49:48Z

@ambv As the latest core dev touching Lib/urllib/request.py, would you be able to review this PR?

orsenthil

LGTM

miss-islington · 2022-10-05T17:57:55Z

Thanks @eendebakpt for the PR, and @orsenthil for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

bedevere-bot · 2022-10-05T17:58:02Z

GH-97918 is a backport of this pull request to the 3.11 branch.

…nGH-91566) * improve performance of get_proxies_environment when there are many environment variables * 📜🤖 Added by blurb_it. * fix case of short env name * fix formatting * fix whitespace * whitespace * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * whitespace * Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Carl Meyer <[email protected]> (cherry picked from commit aeb28f5) Co-authored-by: Pieter Eendebak <[email protected]>

bedevere-bot · 2022-10-05T17:58:06Z

GH-97919 is a backport of this pull request to the 3.10 branch.

…nGH-91566) * improve performance of get_proxies_environment when there are many environment variables * 📜🤖 Added by blurb_it. * fix case of short env name * fix formatting * fix whitespace * whitespace * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * whitespace * Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Carl Meyer <[email protected]> (cherry picked from commit aeb28f5) Co-authored-by: Pieter Eendebak <[email protected]>

* improve performance of get_proxies_environment when there are many environment variables * 📜🤖 Added by blurb_it. * fix case of short env name * fix formatting * fix whitespace * whitespace * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * whitespace * Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Carl Meyer <[email protected]> (cherry picked from commit aeb28f5) Co-authored-by: Pieter Eendebak <[email protected]>

* main: (66 commits) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896) pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383) pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759) pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251) pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662) pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644) pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756) pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778) pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826) Add re.VERBOSE flag documentation example (python#97678) ...

* main: (53 commits) pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926) pythonGH-88050: fix race in closing subprocess pipe in asyncio (python#97951) pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962) pythongh-95986: Fix the example using match keyword (python#95989) pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944) pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929) pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610) pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703) pythonGH-88968: Add notes about socket ownership transfers (python#97936) pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) ...

…n#91566) * improve performance of get_proxies_environment when there are many environment variables * 📜🤖 Added by blurb_it. * fix case of short env name * fix formatting * fix whitespace * whitespace * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> * whitespace * Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Co-authored-by: Carl Meyer <[email protected]> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <[email protected]> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Carl Meyer <[email protected]>

improve performance of get_proxies_environment when there are many en…

42118e3

…vironment variables

bedevere-bot added the awaiting review label Apr 15, 2022

📜🤖 Added by blurb_it.

37ff8d9

eendebakpt marked this pull request as draft April 15, 2022 11:31

eendebakpt added 4 commits April 15, 2022 13:34

fix case of short env name

b8de962

Merge branch 'performance/getproxies_environment' of githubeendebakpt…

0f90945

…:eendebakpt/cpython into performance/getproxies_environment

fix formatting

336da07

fix whitespace

cabeb47

eendebakpt marked this pull request as ready for review April 15, 2022 12:18

eendebakpt added 4 commits April 15, 2022 14:41

whitespace

c581748

Merge branch 'main' into performance/getproxies_environment

4217396

Merge branch 'main' into performance/getproxies_environment

0055604

Merge branch 'main' into performance/getproxies_environment

869fa9a

carljm reviewed May 17, 2022

View reviewed changes

Lib/urllib/request.py Outdated Show resolved Hide resolved

Lib/urllib/request.py Outdated Show resolved Hide resolved

Lib/urllib/request.py Outdated Show resolved Hide resolved

Lib/urllib/request.py Outdated Show resolved Hide resolved

eendebakpt and others added 4 commits May 18, 2022 10:03

Update Lib/urllib/request.py

dba0414

Co-authored-by: Carl Meyer <[email protected]>

Update Lib/urllib/request.py

76f16ca

Co-authored-by: Carl Meyer <[email protected]>

Update Lib/urllib/request.py

d3feb07

Co-authored-by: Carl Meyer <[email protected]>

Update Lib/urllib/request.py

b9ab037

Co-authored-by: Carl Meyer <[email protected]>

eendebakpt force-pushed the performance/getproxies_environment branch 2 times, most recently from 0f694c9 to aeb96ea Compare May 18, 2022 08:14

whitespace

f961505

eendebakpt force-pushed the performance/getproxies_environment branch from aeb96ea to f961505 Compare May 18, 2022 08:52

carljm requested changes May 18, 2022

View reviewed changes

Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Outdated Show resolved Hide resolved

Lib/urllib/request.py Show resolved Hide resolved

bedevere-bot added awaiting core review and removed awaiting review labels May 18, 2022

eendebakpt and others added 2 commits May 18, 2022 18:48

Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7W…

36e5497

…gVuA.rst Co-authored-by: Carl Meyer <[email protected]>

Update Lib/urllib/request.py

ca4fbd4

Co-authored-by: Carl Meyer <[email protected]>

carljm approved these changes May 18, 2022

View reviewed changes

Merge branch 'main' into performance/getproxies_environment

77cceab

eendebakpt mentioned this pull request Aug 31, 2022

Triage the open "performance" issues on cpython repo faster-cpython/ideas#440

Closed

iritkatriel requested a review from orsenthil August 31, 2022 20:47

iritkatriel added performance Performance or resource usage 3.12 bugs and security fixes labels Aug 31, 2022

eendebakpt added 2 commits September 11, 2022 14:04

Merge branch 'main' into performance/getproxies_environment

41b72c6

Merge branch 'main' into performance/getproxies_environment

9db1103

orsenthil self-assigned this Oct 5, 2022

orsenthil added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Oct 5, 2022

orsenthil approved these changes Oct 5, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 5, 2022

orsenthil merged commit aeb28f5 into python:main Oct 5, 2022

bedevere-bot removed the awaiting merge label Oct 5, 2022

bedevere-bot removed the needs backport to 3.11 only security fixes label Oct 5, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 5, 2022

eendebakpt deleted the performance/getproxies_environment branch October 5, 2022 19:23

eendebakpt restored the performance/getproxies_environment branch October 5, 2022 19:23

eendebakpt deleted the performance/getproxies_environment branch October 5, 2022 19:23

eendebakpt restored the performance/getproxies_environment branch October 5, 2022 19:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-91539: improve performance of get_proxies_environment #91566

gh-91539: improve performance of get_proxies_environment #91566

eendebakpt commented Apr 15, 2022 •

edited by bedevere-bot

Loading

eendebakpt commented May 18, 2022

carljm left a comment

eendebakpt commented Aug 10, 2022

orsenthil left a comment

miss-islington commented Oct 5, 2022

bedevere-bot commented Oct 5, 2022

bedevere-bot commented Oct 5, 2022

gh-91539: improve performance of get_proxies_environment #91566

gh-91539: improve performance of get_proxies_environment #91566

Conversation

eendebakpt commented Apr 15, 2022 • edited by bedevere-bot Loading

eendebakpt commented May 18, 2022

carljm left a comment

Choose a reason for hiding this comment

eendebakpt commented Aug 10, 2022

orsenthil left a comment

Choose a reason for hiding this comment

miss-islington commented Oct 5, 2022

bedevere-bot commented Oct 5, 2022

bedevere-bot commented Oct 5, 2022

eendebakpt commented Apr 15, 2022 •

edited by bedevere-bot

Loading