A few days ago, I came across an issue in a Magento 2 project where the client complained that no products were being displayed when a customer searched for anything on the site. The most curious part is that this only happened in their production environment. Everything was working fine in their staging environment, which led me to believe the issue was related to something in their production environment or in one of the services running there.
Right below is an illustration of what was happening on the client’s website.
Note: for obvious reasons, I replicated the same error on a vanilla Magento 2 installation.
I started this investigation by doing some trivial tasks just to discard the most common reasons for this to happen and start digging into the problem.
I started by:
- Performing a full indexation of the whole site;
- Clearing all the caches, including the 3rd-party cache service system, like Fastly.
How to start debugging it
Well, nothing happened and the investigation went on with the understanding that probably the reason was something wrong with the data in the database or ElasticSearch.
After taking the above steps and confirming it didn’t fix the issue, I decided to check the logs. After a few minutes of looking at the logs, I found something interesting in one of them:
{
"error":{
"root_cause":[
{
"type":"illegal_argument_exception",
"reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_description] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
],
"type":"search_phase_execution_exception",
"reason":"all shards failed",
"phase":"query",
"grouped":true,
"failed_shards":[
{
"shard":0,
"index":"magento2_product_3_v193",
"node":"xxxxxxx",
"reason":{
"type":"illegal_argument_exception",
"reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_description] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
],
"caused_by":{
"type":"illegal_argument_exception",
"reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_description] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
"caused_by":{
"type":"illegal_argument_exception",
"reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_description] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
},
"status":400
}
Well, it definitely says that something is wrong with the data indexed to the ElasticSearch, and I started digging into the ElasticSearch data and queries performed by Magento to understand the root reasons of the problem.
I have some experience with debugging how Magento 2 uses ElasticSearch as a search engine under the hood and how I could start debugging it. The first thing is to see exactly what query Magento 2 is building to perform product searches in ElasticSearch. To do so, we need to start by checking which search adapter Magento 2 is using in the project. If you’re using the ElasticSearch bundled modules that come with the vanilla Magento 2 installation, you probably want to check the following class, which was my case:
\Magento\Elasticsearch7\SearchAdapter\Adapter
If you’re not 100% sure of what class you should look at, keep in mind that all the Search Adapters may implement the following interface in Magento 2.
\Magento\Framework\Search\AdapterInterface
Now, it’s only a matter of checking what are the classes that implement this above interface and try one by one. For doing so, you can use PhpStorm by opening the interface and clicking on the left-side green arrow. It will open a pop-up with all the classes that implement the interface (it works with any kind of class in PHP) so you can easily navigate to these classes and make your tests to find the right class.
Well, once you find the proper search adapter your Magento 2 is using, check the method called query in the class.
Pay attention to the line where the variable $query is built. This variable contains the query that you can use for performing a query in ElasticSearch.
Tip: the query itself is the body index in the array.
In my case, the query looked like the following:
{
"from":0,
"size":12,
"stored_fields":"_none_",
"docvalue_fields":[
"_id",
"_score"
],
"sort":[
{
"_score":{
"order":"desc"
}
}
],
"query":{
"bool":{
"must":[
{
"terms":{
"visibility":[
"3",
"4"
]
}
}
],
"should":[
{
"match":{
"_search":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"name":{
"query":"bag",
"boost":6
}
}
},
{
"match":{
"sku":{
"query":"bag",
"boost":7
}
}
},
{
"match":{
"description":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"short_description":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"manufacturer_value":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"status_value":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"url_key":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"tax_class_id_value":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"custom_description":{
"query":"bag",
"boost":2
}
}
},
{
"match":{
"_search":{
"query":"bag",
"boost":2
}
}
},
{
"match_phrase_prefix":{
"name":{
"query":"bag",
"boost":2,
"analyzer":"prefix_search"
}
}
},
{
"match_phrase_prefix":{
"sku":{
"query":"bag",
"boost":2,
"analyzer":"sku_prefix_search"
}
}
}
],
"must_not":[
{
"term":{
"category_permission_1_0":"-2"
}
}
],
"minimum_should_match":1
}
},
"aggregations":{
"price_bucket":{
"extended_stats":{
"field":"price_0_1"
}
},
"category_bucket":{
"terms":{
"include":[
"38",
"20",
"11",
"3",
"9",
"37",
"41"
],
"field":"category_ids",
"size":500
}
},
"custom_description_bucket":{
"terms":{
"field":"custom_description",
"size":500
}
}
}
}
With this on hand, you can use Postman or cURL to perform some queries to ElasticSearch directly and check the results.
This is very useful when you need to identify the part of the query that is breaking the search in ElasticSearch so, by using reverse engineering, you can understand how this piece of query is built by debugging the highlighted line above.
Well, these are the first steps I use to debug ElasticSearch. I won’t get into the details of how to do it because it’s out of the scope of this blog post. If you want a more detailed post about how to debug ElasticSearch in Magento 2, just let me know in the comments.
The solution for the issue
If you came across the same issue and want to understand why it happens, your answer is on this line of the error:
"reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_description] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
Basically, it says that the custom_description attribute is being used for data aggregations and sorting in ElasticSearch, however, it’s a text field and is not optimized for such operations in the mapping that Magento 2 uses in ElasticSearch by default.
If we check our attribute on the admin we can see the following:
It’s really a text field.
It’s being used in the Layer Navigation, which is not allowed by default for text fields in Magento 2 by the way.
For this blog post, I have created this attribute and changed it directly in the database since Magento doesn’t allow the user to change it via the admin panel, but how the client’s attribute ended up like this is still a mystery to me. Maybe an integration, maybe a custom module created this attribute like this, I don’t know, there are plenty of possibilities of how the attribute ended up like this in the client’s database.
The simplest fix for this is to go to the catalog_eav_attribute table and set the field is_filterable_in_search to 0 for this attribute.
After changing this, run a full indexation via the terminal.
Clear all the caches and try searching on the website again.
The search issue is now gone and your client is happy again.
I appreciate your reading up here and hope this post helped you and saved you a ton of debugging time.
Tiago Sampaio 