How to search in nested data of index in Elasticsearch PHP Laravel

Question

My JSON is like

{
  "myData": [
    {
      "id": "4rr9a74a-d7cc-11e8-824f-0242ac160002",
      "uri": "http://google.com/fagdge",
      "statement": "Distinguish between random and non-random sampling methods, identify possible sources of bias in sampling, describe how such bias can be controlled and reduced, evaluate the characteristics of a good survey and well-designed experiment, design simple experiments or investigations to collect data to answer questions of interest, and make inferences from sample results.",
      "dateTime": "2021-06-17T18:41:58+00:00",
      "type": "Standard",
      "hcs": "AII.DSP.1",
      "docUri": {
        "id": "c339d471-d7cb-11e8-824f-0242ac160002",
        "uri": "https://api.test.com/24124",
        "title": "Test Title"
      },
      "edLevels": ["09", "10", "11", "12"],
      "itemUri": {
        "title": "Test Title 1",
        "identifier": "3397fbd1-7ad2-4236-bd5c-68b77f67b144",
        "uri": "https://api.test.com/231221"
      }
    },
    {
      "id": "4tt9a74a-d7cc-11e8-824f-0243ac160002",
      "uri": "http://google.com/fagdge",
      "statement": "sources of bias in sampling, describe how such bias can be controlled and reduced, evaluate the characteristics of a good survey and well-designed experiment, design simple experiments or investigations to collect data to answer questions of interest, and make inferences from sample results.",
      "dateTime": "2021-06-17T18:41:58+00:00",
      "type": "Standard",
      "hcs": "AII.DSP.2",
      "docUri": {
        "id": "c559d471-d7cb-11e8-824f-0242ac160002",
        "uri": "https://api.test.com/24124",
        "title": "Test Title 22"
      },
      "edLevels": ["09", "10", "11", "12"],
      "itemUri": {
        "title": "Test Title 2",
        "identifier": "6697fbd1-7ad2-4236-bd5c-68b77f67b144",
        "uri": "https://api.test.com/231221"
      }
    }
  ]
}

and I want to search hcs and matched value in elastic search using php and laravel?

I have tried below code:

$client = ClientBuilder::create()
        ->setHosts(['http://127.0.0.1:9200'])
        ->setBasicAuthentication('<user>', '<pass>')
        ->build();
        
        try {
            $params = [
                'body' => [
                    'query' => [
                        'nested' => [
                            "path" => "myData",
                            "query" => [
                                'match' => [
                                    'myData.hcs' => 'AII.DSP.1'
                                ]
                            ]
                        ]
                        
                    ]
                ]
            ];
            $response = $client->search($params);
            dd($response);
        } catch (NoNodesAvailableException $e) {
            printf ("NoNodesAvailableException: %s\n", $e->getMessage());
        }
    exit;

But getting below error

failed to create query: [nested] nested object under path

Edit:

Mapping is:

        {
      "myData" : {
        "aliases" : { },
        "mappings" : {
          "properties" : {
            "event" : {
              "properties" : {
                "duri" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "type" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "nUri" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "identifier" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "lastChangeDateTime" : {
                  "type" : "date"
                },
                "onUri" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "sequence" : {
                  "type" : "long"
                },
                "uri" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "event2" : {
              "properties" : {
                "iTypes" : {
                  "properties" : {
                    "description" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "code" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "lastChangeDateTime" : {
                      "type" : "date"
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "tCode" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "event3" : {
                  "properties" : {
                    "code" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "lastChangeDateTime" : {
                      "type" : "date"
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                }
              }
            },
            "event4" : {
              "properties" : {
                "pUri" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "status" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "creator" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "identifier" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "language" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "lastChangeDateTime" : {
                  "type" : "date"
                },
                "sUrl" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "subject" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "subjectURI" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "title" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "uri" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "version" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "event5" : {
              "properties" : {
                "docUri" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "event45" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "itemUri" : {
                  "properties" : {
                    "identifier" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "title" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "uri" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },
                "edLevels" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "statement" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "hcs" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "identifier" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "language" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "lastChangeDateTime" : {
                  "type" : "date"
                },
                "list" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "notes" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "uri" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "testField" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "settings" : {
          "index" : {
            "routing" : {
              "allocation" : {
                "include" : {
                  "_tier_preference" : "data_content"
                }
              }
            },
            "number_of_shards" : "1",
            "provided_name" : "taxonomies",
            "creation_date" : "1654273882675",
            "number_of_replicas" : "1",
            "uuid" : "ychbqd9GTzC-arQgLib_RQ",
            "version" : {
              "created" : "7130099"
            }
          }
        }
      }
    }

Answer 1

Tldr;

You are not dealing with an elasticsearch nested field.

Although you json has indeed nested object. Your elasticsearch mapping does not.

To reproduce

In order to get the same error as you did:

POST /72497503/_doc
{
  "data": {
    "plop": "plop"
  }
}

GET /72497503/_mapping

{
  "72497503" : {
    "mappings" : {
      "properties" : {
        "data" : {
          "properties" : {
            "other" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}

Has you can see nothing nested in here.

If i were to do a nested query against this index

GET /72497503/_search
{
  "query": {
    "nested": {
      "path": "data",
      "query": {
        "match_all": {}
      }
    }
  }
}

I would get this error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "query_shard_exception",
        "reason" : "failed to create query: [nested] failed to find nested object under path [data]",
        "index_uuid" : "NjMzpVgnTRiEtH6_Ly0f5w",
        "index" : "72497503"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "72497503",
        "node" : "sgyLLHFrRO2GUk9mYZWN-Q",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "failed to create query: [nested] failed to find nested object under path [data]",
          "index_uuid" : "NjMzpVgnTRiEtH6_Ly0f5w",
          "index" : "72497503",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "[nested] failed to find nested object under path [data]"
          }
        }
      }
    ]
  },
  "status" : 400
}

Which sounds just like yours

To fix

You will first need to declare the mapping, with the nested fields. The reindex the data. And only then you will be able to query it.

PUT /72497503-2/
{
  "mappings": {
    "properties": {
      "data": {
        "type": "nested",
        "properties": {
          "other": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

POST _reindex
{
  "source": {
    "index": "72497503"
  },
  "dest": {
    "index": "72497503-2"
  }
}

GET /72497503-2/_search
{
  "query": {
    "nested": {
      "path": "data",
      "query": {
        "match_all": {}
      }
    }
  }
}

The query gives:

{
  "took" : 810,
  "timed_out" : false,
  "_shards" : {
    ...
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "72497503-2",
        "_id" : "I6O_OoEBOgujegeQtip-",
        "_score" : 1.0,
        "_source" : {
          "data" : {
            "other" : "data"
          }
        }
      }
    ]
  }
}