代码之家 › 专栏 › 技术社区 › Karthikeyan

org.apache.http.contenttoolongexception:对于配置的缓冲区限制[104857600],实体内容太长[105539255]

elastic-stack kibana elasticsearch java

1

Karthikeyan · 技术社区 · 7 年前

正在尝试从我的索引(ElasticSearch)中获取索引的PDF文档。我已经使用摄取附件处理器插件索引了我的PDF文档。其2500份文件已与PDF附件一起编入索引。

现在,我通过搜索PDF的内容来获取这些PDF,并得到下面的错误。

org.apache.http.ContentTooLongException: entity content is too long [105539255] for the configured buffer limit [104857600]
    at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76)
    at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:131)
    at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:147)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:303)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.NullPointerException
    at com.es.utility.DocumentSearch.main(DocumentSearch.java:88)

请找到我的Java API代码,从弹性搜索中获取文档

private final static String ATTACHMENT = "document_attachment";
private final static String TYPE = "doc";

public static void main(String args[])
{
    RestHighLevelClient restHighLevelClient = null;

    try {
        restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http"),
                new HttpHost("localhost", 9201, "http")));

    } catch (Exception e) {
        System.out.println(e.getMessage());
    }



    SearchRequest contentSearchRequest = new SearchRequest(ATTACHMENT); 
    SearchSourceBuilder contentSearchSourceBuilder = new SearchSourceBuilder();
    contentSearchRequest.types(TYPE);
    QueryBuilder attachmentQB = QueryBuilders.matchQuery("attachment.content", "activa");
    contentSearchSourceBuilder.query(attachmentQB);
    contentSearchSourceBuilder.size(50);
    contentSearchRequest.source(contentSearchSourceBuilder);
    SearchResponse contentSearchResponse = null;
    System.out.println("Request --->"+contentSearchRequest.toString());
    try {
        contentSearchResponse = restHighLevelClient.search(contentSearchRequest);
    } catch (IOException e) {
        e.getLocalizedMessage();
    }

    try {
        System.out.println("Response --->"+restHighLevelClient.search(contentSearchRequest)); // am printing the mentioned error from this line.
    } catch (IOException e) {
        e.printStackTrace();
    }
    SearchHit[] contentSearchHits = contentSearchResponse.getHits().getHits();
    long contenttotalHits=contentSearchResponse.getHits().totalHits;
    System.out.println("condition Total Hits --->"+contenttotalHits);

AM使用ElasticSearch版本6.2.3

1 回复 | 直到 7 年前

1

2

Val 7 年前

你需要增加 http.max_content_length 在你的 elasticsearch.yml 配置文件。

默认情况下,它被设置为100MB(100*1024*1024=104857600),因此您可能需要将其设置得稍高一点。

更新

这实际上是一个不同的问题,可以解释 here .基本上,违约 HttpAsyncResponseConsumerFactory 缓冲堆内存中的整个响应主体,但默认情况下最多只能缓冲100MB。解决方法是为该缓冲区配置另一个大小,但您唯一的选择是使用低级REST客户机。在ES7中,您可以使用一个名为 RequestOptions 但还没有发布。

long BUFFER_SIZE = 120 * 1024 * 1024;     <---- set buffer to 120MB instead of 100MB
Map<String, String> params = Collections.emptyMap();
HttpEntity entity = new NStringEntity(contentSearchSourceBuilder.toString(), ContentType.APPLICATION_JSON);
HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory consumerFactory =
        new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(BUFFER_SIZE);
Response response = restClient.performRequest("GET", "/document_attachment/doc/_search", params, entity, consumerFactory);