|
1 | 1 | # Python Proxy Headers |
2 | 2 |
|
3 | | -The `python-proxy-headers` package provides support for handling custom proxy headers when making HTTPS requests in various python modules. |
| 3 | +The `python-proxy-headers` package provides support for handling custom proxy headers when making HTTPS requests in various Python modules. |
4 | 4 |
|
5 | 5 | We currently provide extensions to the following packages: |
6 | 6 |
|
7 | | -* [urllib3](https://urllib3.readthedocs.io/en/stable/) |
8 | | -* [requests](https://docs.python-requests.org/en/latest/index.html) |
9 | | -* [aiohttp](https://docs.aiohttp.org/en/stable/index.html) |
10 | | -* [httpx](https://www.python-httpx.org/) |
| 7 | +* [urllib3](https://python-proxy-headers.readthedocs.io/en/latest/urllib3.html) - a user-friendly HTTP client library for Python |
| 8 | +* [requests](https://python-proxy-headers.readthedocs.io/en/latest/requests.html) - a simple, yet elegant, HTTP library |
| 9 | +* [aiohttp](https://python-proxy-headers.readthedocs.io/en/latest/aiohttp.html) - asynchronous HTTP client/server framework for asyncio and Python |
| 10 | +* [httpx](https://python-proxy-headers.readthedocs.io/en/latest/httpx.html) - a next generation HTTP client for Python |
| 11 | + |
| 12 | +## Purpose |
11 | 13 |
|
12 | 14 | None of these modules provide good support for parsing custom response headers from proxy servers. And some of them make it hard to send custom headers to proxy servers. So we at [ProxyMesh](https://proxymesh.com) made these extension modules to support our customers that use Python and want to use custom headers to control our proxy behavior. But these modules can work for handling custom headers with any proxy. |
13 | 15 |
|
14 | 16 | *If you are looking for [Scrapy](https://scrapy.org/) support, please see our [scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers) project.* |
15 | 17 |
|
16 | 18 | ## Installation |
17 | 19 |
|
18 | | -Examples for how to use these extension modules are described below. You must first do the following: |
| 20 | +To use these extension modules, you must first do the following: |
19 | 21 |
|
20 | 22 | 1. `pip install python-proxy-headers` |
21 | | -2. Install the appropriate package based on the python module you want to use. |
22 | | - |
23 | | -This package does not have any dependencies because we don't know which module you want to use. |
24 | | - |
25 | | -You can also find more example code in our [proxy-examples for python](https://github.com/proxymesh/proxy-examples/tree/main/python). |
26 | | - |
27 | | -## urllib3 |
28 | | - |
29 | | -If you just want to send custom proxy headers, but don't need to receive proxy response headers, then you can [urllib3.ProxyManager](https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html#urllib3.ProxyManager), like so: |
30 | | - |
31 | | -``` python |
32 | | -import urllib3 |
33 | | -proxy = urllib3.ProxyManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-Country': 'US'}) |
34 | | -r = proxy.request('GET', 'https://api.ipify.org?format=json') |
35 | | -``` |
36 | | - |
37 | | -Note that when using this method, if you keep reusing the same `ProxyManager` instance, you may be re-using the proxy connection, which may have different behavior than if you create a new proxy connection for each request. For example, with ProxyMesh you may keep getting the same IP address if you reuse the proxy connection. |
38 | | - |
39 | | -To get proxy response headers, use our extension module like this: |
40 | | - |
41 | | -``` python |
42 | | -from python_proxy_headers import urllib3_proxy_manager |
43 | | -proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT') |
44 | | -r = proxy.request('GET', 'https://api.ipify.org?format=json') |
45 | | -r.headers['X-ProxyMesh-IP'] |
46 | | -``` |
47 | | - |
48 | | -You can also pass `proxy_headers` into our `ProxyHeaderManager` as well. For example, you can pass back the same `X-ProxyMesh-IP` header to ensure you get the same IP address on subsequent requests. |
49 | | - |
50 | | -## requests |
51 | | - |
52 | | -The requests adapter builds on our `urllib3_proxy_manager` module to make it easy to pass in proxy headers and receive proxy response headers. |
53 | | - |
54 | | -``` python |
55 | | -from python_proxy_headers import requests_adapter |
56 | | -r = requests_adapter.get('https://api.ipify.org?format=json', proxies={'http': 'http://PROXYHOST:PORT', 'https': 'http://PROXYHOST:PORT'}, proxy_headers={'X-ProxyMesh-Country': 'US'}) |
57 | | -r.headers['X-ProxyMesh-IP'] |
58 | | -``` |
59 | | - |
60 | | -The `requests_adapter` module supports all the standard requests methods: `get`, `post`, `put`, `delete`, etc. |
61 | | - |
62 | | -## aiohttp |
63 | | - |
64 | | -While it's not documented, aiohttp does support passing in custom proxy headers by default. |
65 | | - |
66 | | -``` python |
67 | | -import aiohttp |
68 | | -async with aiohttp.ClientSession() as session: |
69 | | - async with session.get('https://api.ipify.org?format=json', proxy="http://PROXYHOST:PORT", proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: |
70 | | - await r.text() |
71 | | -``` |
72 | | - |
73 | | -However, if you want to get proxy response, you should use our extension module: |
74 | | - |
75 | | -``` python |
76 | | -from python_proxy_headers import aiohttp_proxy |
77 | | -async with aiohttp_proxy.ProxyClientSession() as session: |
78 | | - async with session.get('https://api.ipify.org?format=json', proxy="http://PROXYHOST:PORT", proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: |
79 | | - await r.text() |
80 | | - |
81 | | -r.headers['X-ProxyMesh-IP'] |
82 | | -``` |
83 | | - |
84 | | -## httpx |
85 | | - |
86 | | -httpx also supports proxy headers by default, though it's not documented: |
87 | | - |
88 | | -``` python |
89 | | -import httpx |
90 | | -proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) |
91 | | -transport = HTTPProxyTransport(proxy=proxy) |
92 | | -with httpx.Client(mounts={'http://': transort, 'https://': transport}) as client: |
93 | | - r = client.get('https://api.ipify.org?format=json') |
94 | | -``` |
95 | | - |
96 | | -But to get the response headers, you need to use our extension module: |
97 | | - |
98 | | -``` python |
99 | | -import httpx |
100 | | -from python_proxy_headers.httpx_proxy import HTTPProxyTransport |
101 | | -proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) |
102 | | -transport = HTTPProxyTransport(proxy=proxy) |
103 | | -with httpx.Client(mounts={'http://': transort, 'https://': transport}) as client: |
104 | | - r = client.get('https://api.ipify.org?format=json') |
105 | | - |
106 | | -r.headers['X-ProxyMesh-IP'] |
107 | | -``` |
108 | | - |
109 | | -This module also provide helper methods similar to requests: |
110 | | - |
111 | | -``` python |
112 | | -import httpx |
113 | | -from python_proxy_headers import httpx_proxy |
114 | | -proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) |
115 | | -r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy) |
116 | | -r.headers['X-ProxyMesh-IP'] |
117 | | -``` |
| 23 | +2. Install the appropriate package based on the Python library you want to use. |
118 | 24 |
|
119 | | -And finally, httpx supports async requests, so we provide an async extension too: |
| 25 | +This package does not have any dependencies because we don't know which library you want to use. |
120 | 26 |
|
121 | | -``` python |
122 | | -import httpx |
123 | | -from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport |
124 | | -proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) |
125 | | -transport = AsyncHTTPProxyTransport(proxy=proxy) |
126 | | -async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client: |
127 | | - r = await client.get('https://api.ipify.org?format=json') |
| 27 | +## Documentation |
128 | 28 |
|
129 | | -r.headers['X-ProxyMesh-IP'] |
130 | | -``` |
| 29 | +For detailed documentation, examples, and usage instructions, please see the [full documentation](https://python-proxy-headers.readthedocs.io/en/latest/). |
131 | 30 |
|
132 | | -Our httpx helper module internally provides extension classes for [httpcore](https://www.encode.io/httpcore/), for handling proxy headers over tunnel connections. |
133 | | -You can use those classes if you're building on top of httpcore. |
| 31 | +You can also find more example code in our [proxy-examples for python](https://github.com/proxymesh/proxy-examples/tree/main/python). |
0 commit comments