Apache-HttpClient

Apache HttpComponents/HttpClient 笔记

Apache HttpComponents
http://hc.apache.org/

---

HttpClient 4.5 超时配置

Apache HttpClient 不同版本的超时参数配置不同，4.5 版本的超时配置如下：

private CloseableHttpClient httpClient() {
    try {
        RequestConfig config = RequestConfig.custom()
                .setConnectTimeout(connectTimeout)
                .setConnectionRequestTimeout(connectTimeout)
                .setSocketTimeout(readTimeout)
                .build();
        return HttpClientBuilder
                .create()
                .setDefaultRequestConfig(config)
                .build();
    } catch (Exception e) {
        throw new RuntimeException();
    }
}

connectionRequestTimeout 获取连接超时时间

connectionRequestTimeout 设置从连接池获取连接的超时时间，单位毫秒。这个属性是 4.5 新加的，因为目前版本是可以共享连接池的。
超过该时间未拿到可用连接，会抛出 org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

connectTimeout 建立连接超时时间

connectTimeout 连接超时时间，单位毫秒。
连接上服务器(握手成功)的时间，超出该时间抛出 connect timeout

socketTimeout 读超时时间

socketTimeout 请求获取数据的超时时间(即响应时间)，单位毫秒。 如果访问一个接口，多少时间内无法返回数据，就直接放弃此次调用。
服务器返回数据(response)的时间，超过该时间抛出 read timeout

---

HttpClient 线程池

HttpComponents HttpClient连接池(1)-结构
https://cloud.tencent.com/developer/article/1607662

HttpComponents HttpClient连接池(2)-连接的申请
https://cloud.tencent.com/developer/article/1607744

HttpClient 中每个 host:port 有独立的连接池

httpclient 连接池对于每一个 host + port 定义为唯一 route, 参考 DefaultHttpRoutePlanner.determineRoute() 方法。
对于每一个 route 都有一个 RouteSpecificPool 类型对象对应。这个对象也是一个连接池，既在 httpclient 连接池里，对每一个 route 访问都独立建立各自的连接池，从而实现不同 route 访问连接池隔离。在连接池对象 Cpool 里由 Map 存储，key 为 route，value 为RouteSpecificPool 对象。

http 连接的申请主要调用 Cpool 对象(在父类 AbstractConnPool )的 lease() 方法

AbstractHttpClient.doExecute() 调用
connManager.requestConnection() 调用
PoolingClientConnectionManager.requestConnection() 调用
this.pool.lease(route, state) 返回一个 Future 调用起 get() 里面会调用 getPoolEntryBlocking() 方法，其中调用
getPool(route) 获取连接池

final RouteSpecificPool<T, C, E> pool = getPool(route);

所以每个 route 间的线程池是隔离的。

maxConnPerRoute 每个路由(host:port)的最大连接数(默认2)

maxConnPerRoute 就是上面说的每个 host:port 连接池的大小的，即允许同一个 host:port 最多有几个活跃连接。当设置为2时，意味着同时最多有两个到host:port的连接，如果有空闲连接，本次请求会使用已经建立的空闲连接，但如果没有，就会进入等待队列，等待到该host:port的连接可用。

maxConnTotal 总的最大连接数(默认20)

总共允许 HttpClient 建立多少个连接，超出以后所有的连接都会进入等待队列。

keepAliveStrategy

在 keep-alive 时间内，可以使用同一个 tcp 连接发起多次 http 请求。
HTTP/1.1 中默认是持久连接

HTTP/1.1 200
Connection: keep-alive
Keep-Alive: timeout=60

validateAfterInactivity

连接池中连接不活跃多长时间后需要进行一次验证，默认为2s

connManagerShared

连接池是否共享模式，这个共享是指与其它 httpClient 是否共享，某些中间件内部可能也使用 httpClient

---

AsyncHttpClient

AsyncHttpClient / async-http-client
https://github.com/AsyncHttpClient/async-http-client

---

Cookie rejected Illegal ‘domain’ attribute

日志中不断重复的报warn：
00:00:00.665 [http-nio-8380-exec-454] WARN o.a.h.c.p.ResponseProcessCookies - Cookie rejected [BAIDUID=”C303600CB4B1BEDDCC9551003BD54DD5:FG=1”, version:0, domain:baidu.com, path:/, expiry:Wed Jan 12 00:00:46 CST 2022] Illegal ‘domain’ attribute “baidu.com”. Domain of origin: “10.123.123.21”

原因：服务器端 cookie 的 domain 是 10.123.123.21, 但客户端给传了个 baidu.com，两者不一致导致Cookie rejected。
domain是用来在浏览器端限制哪个域名可以访问此cookie内容的。我们使用httpclient一般来说是不需要用到cookie的。
为了不让日志不停的打印，可以设置HttpClient的Cookie策略:

RequestConfig requestConfig = = RequestConfig.custom()
                    // 设置Cookie策略为忽略
                    .setCookieSpec(CookieSpecs.STANDARD_STRICT)
                    .build();

---

httpClient 开启 debug 日志

打开 HttpClient 的 debug 日志

<logger name="org.apache.http" level="debug" />

一次完整的 http 请求 debug 日志

2022-08-29 10:15:05.571 [configWatchTaskScheduler-1] DEBUG o.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
2022-08-29 10:15:05.571 [configWatchTaskScheduler-1] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
2022-08-29 10:15:05.571 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://consul-service:8500][total available: 1; route allocated: 2 of 500; total allocated: 2 of 1000]
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://consul-service:8500][total available: 0; route allocated: 2 of 500; total allocated: 2 of 1000]
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 600000
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request GET /v1/kv/config/tc,online/?recurse&token=my-token&wait=55s&index=546983 HTTP/1.1
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /v1/kv/config/tc,online/?recurse&token=my-token&wait=55s&index=546983 HTTP/1.1
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: consul-service:8500
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
2022-08-29 10:15:05.572 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.13 (Java/1.8.0_101)
2022-08-29 10:15:05.573 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 404 Not Found
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << Vary: Accept-Encoding
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << X-Consul-Index: 675782
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << X-Consul-Knownleader: true
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << X-Consul-Lastcontact: 0
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Mon, 29 Aug 2022 02:15:13 GMT
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Length: 0
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://consul-service:8500] can be kept alive indefinitely
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0
2022-08-29 10:15:05.583 [configWatchTaskScheduler-1] DEBUG o.a.h.i.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://consul-service:8500][total available: 1; route allocated: 2 of 500; total allocated: 2 of 1000]

---

从HTTPResponse中读取Json

利用 BufferedReader.readLine() 从 HTTPResponse.getEntity().getContent() 流中逐行读取返回内容：

HttpResponse response; // some response object
BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
StringBuilder builder = new StringBuilder();
for (String line = null; (line = reader.readLine()) != null;) {
    builder.append(line).append("\n");
}
JSONTokener tokener = new JSONTokener(builder.toString());
JSONArray finalResult = new JSONArray(tokener);

如果 JSON 实际上是一行，那么你也可以删除循环和 builder：

HttpResponse response; // some response object
BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
String json = reader.readLine();
JSONTokener tokener = new JSONTokener(json);
JSONArray finalResult = new JSONArray(tokener);

使用gson：

void getJson() throws IOException {
    HttpClient  httpClient = new DefaultHttpClient();
    HttpGet httpGet = new HttpGet("some url of json");
    HttpResponse httpResponse = httpClient.execute(httpGet);
    String response = EntityUtils.toString(httpResponse.getEntity());

    Gson gson = new Gson();
    MyClass myClassObj = gson.fromJson(response, MyClass.class);
}

使用实例：

HttpResponse httpResponse = UdsHttpClient.postForRawResponseWithHeader(server + QUERY_SO_ORDER_LIST_API, null, content, false, headersMap, logger);
checkState(httpResponse != null, "Artemis returned empty response");
checkState(httpResponse.getStatusLine() != null, "Artemis return error response");
checkState(httpResponse.getEntity() != null, "Artemis returned empty response entity");
checkState(httpResponse.getStatusLine().getStatusCode() != ARTEMIS_TOKEN_ERROR_CODE, "Artemis return token error code");
BufferedReader reader = new BufferedReader(new InputStreamReader(httpResponse.getEntity().getContent(), "UTF-8"));
String result = reader.readLine();
QuerySOListResponse response = JSONUtils.readValue(result, QuerySOListResponse.class);

How do I parse JSON from a Java HTTPResponse?
https://stackoverflow.com/questions/2845599/how-do-i-parse-json-from-a-java-httpresponse

如何从Java HTTPResponse解析JSON？
https://codeday.me/bug/20170629/34297.html

---