Skip to content

Commit 5928773

Browse files
proxymeshproxymesh
committed
initial docs
1 parent 7aaf580 commit 5928773

5 files changed

Lines changed: 502 additions & 1 deletion

File tree

‎docs/aiohttp.rst‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
aiohttp
2+
=======
3+
4+
The `aiohttp <https://docs.aiohttp.org/en/stable/index.html>`_ library is an async HTTP client/server framework for Python. This page describes how to use aiohttp with proxies and how to interact with proxy headers.
5+
6+
Using Proxies with aiohttp
7+
---------------------------
8+
9+
aiohttp provides built-in support for proxies through the ``proxy`` parameter in request methods. You can specify a proxy URL for each request.
10+
11+
Basic Proxy Usage
12+
~~~~~~~~~~~~~~~~~
13+
14+
To use a proxy with aiohttp, you can pass the ``proxy`` parameter to any request method:
15+
16+
.. code-block:: python
17+
18+
import aiohttp
19+
async with aiohttp.ClientSession() as session:
20+
async with session.get('https://api.ipify.org?format=json',
21+
proxy="http://PROXYHOST:PORT") as r:
22+
text = await r.text()
23+
24+
This routes the request through the specified proxy server.
25+
26+
Sending Custom Proxy Headers
27+
-----------------------------
28+
29+
While it's not documented, aiohttp does support passing in custom proxy headers by default using the ``proxy_headers`` parameter:
30+
31+
.. code-block:: python
32+
33+
import aiohttp
34+
async with aiohttp.ClientSession() as session:
35+
async with session.get('https://api.ipify.org?format=json',
36+
proxy="http://PROXYHOST:PORT",
37+
proxy_headers={'X-ProxyMesh-Country': 'US'}) as r:
38+
text = await r.text()
39+
40+
The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address.
41+
42+
Receiving Proxy Response Headers
43+
---------------------------------
44+
45+
However, if you want to get proxy response headers, you should use our extension module ``python_proxy_headers.aiohttp_proxy``:
46+
47+
.. code-block:: python
48+
49+
from python_proxy_headers import aiohttp_proxy
50+
async with aiohttp_proxy.ProxyClientSession() as session:
51+
async with session.get('https://api.ipify.org?format=json',
52+
proxy="http://PROXYHOST:PORT",
53+
proxy_headers={'X-ProxyMesh-Country': 'US'}) as r:
54+
text = await r.text()
55+
proxy_ip = r.headers['X-ProxyMesh-IP']
56+
57+
The ``ProxyClientSession`` extends the standard ``ClientSession`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request.
58+
59+
Complete Example
60+
----------------
61+
62+
Here's a complete example showing how to use aiohttp with proxy headers:
63+
64+
.. code-block:: python
65+
66+
import asyncio
67+
from python_proxy_headers import aiohttp_proxy
68+
69+
async def main():
70+
async with aiohttp_proxy.ProxyClientSession() as session:
71+
async with session.get('https://api.ipify.org?format=json',
72+
proxy="http://PROXYHOST:PORT",
73+
proxy_headers={'X-ProxyMesh-Country': 'US'}) as r:
74+
data = await r.json()
75+
proxy_ip = r.headers.get('X-ProxyMesh-IP')
76+
print(f"Your IP: {data['ip']}")
77+
print(f"Proxy IP: {proxy_ip}")
78+
79+
asyncio.run(main())
80+
81+
Proxy Headers Overview
82+
----------------------
83+
84+
Proxy headers are custom HTTP headers that can be used to communicate with proxy servers. They allow you to:
85+
86+
* **Control proxy behavior**: Send headers like ``X-ProxyMesh-Country`` to select a specific country for your proxy connection
87+
* **Receive proxy information**: Get headers like ``X-ProxyMesh-IP`` to know which IP address was assigned to your request
88+
* **Maintain session consistency**: Use headers like ``X-ProxyMesh-IP`` to ensure you get the same IP address across multiple requests
89+
90+
The exact headers available depend on your proxy provider. Check your proxy provider's documentation for the specific headers they support.
91+
92+
Async Context Managers
93+
-----------------------
94+
95+
The ``ProxyClientSession`` works just like the standard ``ClientSession`` and supports all the same features, including:
96+
97+
* Connection pooling
98+
* Cookie handling
99+
* Timeout configuration
100+
* SSL verification settings
101+
102+
All standard aiohttp request methods are supported: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``.
103+

‎docs/httpx.rst‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
httpx
2+
=====
3+
4+
The `httpx <https://www.python-httpx.org/>`_ library is a modern HTTP client for Python with support for both sync and async requests. This page describes how to use httpx with proxies and how to interact with proxy headers.
5+
6+
Using Proxies with httpx
7+
------------------------
8+
9+
httpx provides built-in support for proxies through the ``httpx.Proxy`` class. You can create a proxy object and use it with httpx clients.
10+
11+
Basic Proxy Usage
12+
~~~~~~~~~~~~~~~~~
13+
14+
httpx also supports proxy headers by default, though it's not documented. You can use the ``httpx.Proxy`` class with custom headers:
15+
16+
.. code-block:: python
17+
18+
import httpx
19+
from httpx import HTTPProxyTransport
20+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
21+
transport = HTTPProxyTransport(proxy=proxy)
22+
with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client:
23+
r = client.get('https://api.ipify.org?format=json')
24+
25+
This creates a proxy with custom headers and uses it with an httpx client.
26+
27+
Receiving Proxy Response Headers
28+
---------------------------------
29+
30+
But to get the response headers, you need to use our extension module ``python_proxy_headers.httpx_proxy``:
31+
32+
.. code-block:: python
33+
34+
import httpx
35+
from python_proxy_headers.httpx_proxy import HTTPProxyTransport
36+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
37+
transport = HTTPProxyTransport(proxy=proxy)
38+
with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client:
39+
r = client.get('https://api.ipify.org?format=json')
40+
41+
r.headers['X-ProxyMesh-IP']
42+
43+
The ``HTTPProxyTransport`` from our extension module extends the standard transport to make proxy response headers available in the response headers.
44+
45+
Helper Methods
46+
--------------
47+
48+
This module also provides helper methods similar to requests for convenience:
49+
50+
.. code-block:: python
51+
52+
import httpx
53+
from python_proxy_headers import httpx_proxy
54+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
55+
r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy)
56+
r.headers['X-ProxyMesh-IP']
57+
58+
The helper module supports all standard HTTP methods: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``.
59+
60+
Async Support
61+
-------------
62+
63+
httpx supports async requests, so we provide an async extension too:
64+
65+
.. code-block:: python
66+
67+
import httpx
68+
from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport
69+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
70+
transport = AsyncHTTPProxyTransport(proxy=proxy)
71+
async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client:
72+
r = await client.get('https://api.ipify.org?format=json')
73+
74+
r.headers['X-ProxyMesh-IP']
75+
76+
The ``AsyncHTTPProxyTransport`` works just like the sync version but for async clients.
77+
78+
Complete Examples
79+
-----------------
80+
81+
Synchronous Example
82+
~~~~~~~~~~~~~~~~~~~
83+
84+
.. code-block:: python
85+
86+
import httpx
87+
from python_proxy_headers.httpx_proxy import HTTPProxyTransport
88+
89+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
90+
transport = HTTPProxyTransport(proxy=proxy)
91+
92+
with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client:
93+
r = client.get('https://api.ipify.org?format=json')
94+
data = r.json()
95+
proxy_ip = r.headers.get('X-ProxyMesh-IP')
96+
print(f"Your IP: {data['ip']}")
97+
print(f"Proxy IP: {proxy_ip}")
98+
99+
Asynchronous Example
100+
~~~~~~~~~~~~~~~~~~~~
101+
102+
.. code-block:: python
103+
104+
import httpx
105+
from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport
106+
107+
async def main():
108+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
109+
transport = AsyncHTTPProxyTransport(proxy=proxy)
110+
111+
async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client:
112+
r = await client.get('https://api.ipify.org?format=json')
113+
data = r.json()
114+
proxy_ip = r.headers.get('X-ProxyMesh-IP')
115+
print(f"Your IP: {data['ip']}")
116+
print(f"Proxy IP: {proxy_ip}")
117+
118+
import asyncio
119+
asyncio.run(main())
120+
121+
Using Helper Methods
122+
~~~~~~~~~~~~~~~~~~~~
123+
124+
For simpler use cases, you can use the helper methods:
125+
126+
.. code-block:: python
127+
128+
import httpx
129+
from python_proxy_headers import httpx_proxy
130+
131+
proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'})
132+
133+
# GET request
134+
r = httpx_proxy.get('https://api.example.com', proxy=proxy)
135+
136+
# POST request
137+
r = httpx_proxy.post('https://api.example.com', json={'key': 'value'}, proxy=proxy)
138+
139+
# Access proxy response headers
140+
proxy_ip = r.headers.get('X-ProxyMesh-IP')
141+
142+
Proxy Headers Overview
143+
----------------------
144+
145+
Proxy headers are custom HTTP headers that can be used to communicate with proxy servers. They allow you to:
146+
147+
* **Control proxy behavior**: Send headers like ``X-ProxyMesh-Country`` to select a specific country for your proxy connection
148+
* **Receive proxy information**: Get headers like ``X-ProxyMesh-IP`` to know which IP address was assigned to your request
149+
* **Maintain session consistency**: Use headers like ``X-ProxyMesh-IP`` to ensure you get the same IP address across multiple requests
150+
151+
The exact headers available depend on your proxy provider. Check your proxy provider's documentation for the specific headers they support.
152+
153+
httpcore Integration
154+
--------------------
155+
156+
Our httpx helper module internally provides extension classes for `httpcore <https://www.encode.io/httpcore/>`_, for handling proxy headers over tunnel connections. You can use those classes if you're building on top of httpcore directly.
157+

‎docs/index.rst‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,48 @@
66
Welcome to python-proxy-headers's documentation!
77
================================================
88

9+
The ``python-proxy-headers`` package provides support for handling custom proxy headers when making HTTPS requests in various Python modules.
10+
11+
We currently provide extensions to the following packages:
12+
13+
* :doc:`urllib3 <urllib3>` - HTTP client library
14+
* :doc:`requests <requests>` - Simple HTTP library for Python
15+
* :doc:`aiohttp <aiohttp>` - Async HTTP client/server framework
16+
* :doc:`httpx <httpx>` - Modern HTTP client library
17+
18+
Purpose
19+
-------
20+
21+
None of these modules provide good support for parsing custom response headers from proxy servers. And some of them make it hard to send custom headers to proxy servers. So we at `ProxyMesh <https://proxymesh.com>`_ made these extension modules to support our customers that use Python and want to use custom headers to control our proxy behavior. But these modules can work for handling custom headers with any proxy.
22+
23+
If you are looking for `Scrapy <https://scrapy.org/>`_ support, please see our `scrapy-proxy-headers <https://github.com/proxymesh/scrapy-proxy-headers>`_ project.
24+
25+
Installation
26+
------------
27+
28+
To use these extension modules, you must first do the following:
29+
30+
1. Install the package::
31+
32+
``pip install python-proxy-headers``
33+
34+
2. Install the appropriate package based on the Python module you want to use.
35+
36+
This package does not have any dependencies because we don't know which module you want to use.
37+
38+
You can also find more example code in our `proxy-examples for python <https://github.com/proxymesh/proxy-examples/tree/main/python>`_.
39+
40+
Contents
41+
--------
42+
943
.. toctree::
1044
:maxdepth: 2
1145
:caption: Contents:
1246

13-
47+
urllib3
48+
requests
49+
aiohttp
50+
httpx
1451

1552
Indices and tables
1653
==================

0 commit comments

Comments
 (0)