Elastic Search Lengthy Payload and Recurring Fields Index and Search

2 min readOct 11, 2022

Elastic is a distributed search engine with analytic capability for all types of data such as textual, numeric, structured and unstructured.

Setup

First of all we need to setup elastic in our environment. I chose elastic docker for my future steps.

After that you can connect http://localhost:9200 to test elastic rest endpoints.

Data Format

Let’s take JSON object to our scenario which has thousand lines of payload. Therefore entire object index will take more time than we expected and gain more CPU with higher memory consumption.

amount of data is increased.
field mapping incompatible (different data objects) ex : take two payload first one has time as a long and second payload has time as a string.

{
 "name": "first_name",
 "session_id": "0d3712e9-9873-441d-955c-19d22460bbd8",
 "time": 1664289403933,
.
.
.
}

Specific Field Index

In my example, I don’t need to index entire object just need to index name and the time to retrieve search results. Therefore we need to change the mapping of the data to avoid memory consumption and incompatible filed mapping. Let’s take our index name as elastic_index,

Create Index

Create index with mapping configuration,

http://localhost:9200/elastic_index

PUT request with below body,

{
   "mappings":{
      "dynamic":"false",
      "properties":{
         "time":{
            "type":"long"
         },
         "name":{
            "type":"text",
            "fields":{
               "keyword":{
                  "type":"keyword",
                  "ignore_above":256
               }
            }
         }
      }
   }
}

Mapping dynamic filed is default true so mapping will be change according to the payload. Now you can index any data type with field of name and time as above payload.

Search Data

I will guide you through simple data search mechanism which will help you to write query in elastic. Simplest way to write a search query is boolean query.

As a example, Let’s search documents which has name field as bob,

http://localhost:9200/elastic_index

POST request with below body,

{
  "query": {
    "bool": {
      "must": [
        {"term":{"name.keyword":"bob"}}
      ]
    }
  },
  "from": 0,
  "size": 100
}

Summary

This is only an overview of elastic search indexing. I’d recommended to use documentation of Elastic. Understand notations, operators and underline architecture.

Thank you for reading this article. I hope you enjoyed it!