Apache Solr Ref Guide 5.1

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 578

DownloadApache-solr-ref-guide-5.1
Open PDF In BrowserView PDF
TM

Apache Solr Reference Guide
Covering Apache Solr 5.1

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Apache Lucene, Apache
Solr and their respective logos are trademarks of the Apache Software Foundation. Please see the Apache
Trademark Policy for more information.
Fonts used in the Solr Reference Guide include Raleway, licensed under the SIL Open Font License, 1.1.

Apache Solr Reference Guide
This reference guide describes Apache Solr, the open source solution for search. You can download Apache
Solr from the Solr website at http://lucene.apache.org/solr/.
This Guide contains the following sections:
Getting Started: This section guides you through the installation and setup of Solr.
Using the Solr Administration User Interface: This section introduces the Solr Web-based user interface.
From your browser you can view configuration files, submit queries, view logfile settings and Java environment
settings, and monitor and control distributed configurations.
Documents, Fields, and Schema Design: This section describes how Solr organizes its data for indexing. It
explains how a Solr schema defines the fields and field types which Solr uses to organize data within the
document files it indexes.
Understanding Analyzers, Tokenizers, and Filters: This section explains how Solr prepares text for indexing
and searching. Analyzers parse text and produce a stream of tokens, lexical units used for indexing and
searching. Tokenizers break field data down into tokens. Filters perform other transformational or selective work
on token streams.
Indexing and Basic Data Operations: This section describes the indexing process and basic index operations,
such as commit, optimize, and rollback.
Searching: This section presents an overview of the search process in Solr. It describes the main components
used in searches, including request handlers, query parsers, and response writers. It lists the query parameters
that can be passed to Solr, and it describes features such as boosting and faceting, which can be used to
fine-tune search results.
The Well-Configured Solr Instance: This section discusses performance tuning for Solr. It begins with an
overview of the solrconfig.xml file, then tells you how to configure cores with solr.xml, how to configure
the Lucene index writer, and more.
Managing Solr: This section discusses important topics for running and monitoring Solr. Other topics include
how to back up a Solr instance, and how to run Solr with Java Management Extensions (JMX).
SolrCloud: This section describes the newest and most exciting of Solr's new features, SolrCloud, which
provides comprehensive distributed capabilities.
Legacy Scaling and Distribution: This section tells you how to grow a Solr distribution by dividing a large index
into sections called shards, which are then distributed across multiple servers, or by replicating a single index
across multiple services.
Client APIs: This section tells you how to access Solr through various client APIs, including JavaScript, JSON,
and Ruby.

Apache Solr Reference Guide 5.1

2

About This Guide
This guide describes all of the important features and functions of Apache Solr. It is free to download from http://l
ucene.apache.org/solr/.
Designed to provide high-level documentation, this guide is intended to be more encyclopedic and less of a
cookbook. It is structured to address a broad spectrum of needs, ranging from new developers getting started to
well-experienced developers extending their application or troubleshooting. It will be of use at any point in the
application life cycle, for whenever you need authoritative information about Solr.
The material as presented assumes that you are familiar with some basic search concepts and that you can read
XML. It does not assume that you are a Java programmer, although knowledge of Java is helpful when working
directly with Lucene or when developing custom extensions to a Lucene/Solr installation.

Special Inline Notes
Special notes are included throughout these pages.
Note Type
Information

Notes

Tip

Warning

Look & Description
Notes with a blue background are used for information that is important for you to
know.

Yellow notes are further clarifications of important points to keep in mind while using
Solr.

Notes with a green background are Helpful Tips.

Notes with a red background are warning messages.

Hosts and Port Examples
The default port when running Solr is 8983. The samples, URLs and screenshots in this guide may show
different ports, because the port number that Solr uses is configurable. If you have not customized your
installation of Solr, please make sure that you use port 8983 when following the examples, or configure your own
installation to use the port numbers shown in the examples. For information about configuring port numbers, see
Managing Solr.
Similarly, URL examples use 'localhost' throughout; if you are accessing Solr from a location remote to the server
hosting Solr, replace 'localhost' with the proper domain or IP where Solr is running.

Paths
Path information is given relative to solr.home, which is the location under the main Solr installation where
Solr's collections and their conf and data directories are stored. When running the various examples
mentioned through out this tutorial (i.e., bin/solr -e techproducts) the solr.home will be a sub directory
of example/ created for you automatically.

Apache Solr Reference Guide 5.1

3

Getting Started
Solr makes it easy for programmers to develop sophisticated, high-performance search applications with
advanced features such as faceting (arranging search results in columns with numerical counts of key terms).
Solr builds on another open source search technology: Lucene, a Java library that provides indexing and search
technology, as well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities. Both Solr
and Lucene are managed by the Apache Software Foundation (www.apache.org).
The Lucene search library currently ranks among the top 15 open source projects and is one of the top 5 Apache
projects, with installations at over 4,000 companies. Lucene/Solr downloads have grown nearly ten times over
the past three years, with a current run-rate of over 6,000 downloads a day. The Solr search server, which
provides application builders a ready-to-use search platform on top of the Lucene search library, is the fastest
growing Lucene sub-project. Apache Lucene/Solr offers an attractive alternative to the proprietary licensed
search and discovery software vendors.
This section helps you get Solr up and running quickly, and introduces you to the basic Solr architecture and
features. It covers the following topics:
Installing Solr: A walkthrough of the Solr installation process.
Running Solr: An introduction to running Solr. Includes information on starting up the servers, adding documents,
and running queries.
A Quick Overview: A high-level overview of how Solr works.
A Step Closer: An introduction to Solr's home directory and configuration options.
Solr Start Script Reference: a complete reference of all of the commands and options available with the bin/solr
script.

Installing Solr
This section describes how to install Solr. You can install Solr in any system where a suitable Java Runtime
Environment (JRE) is available, as detailed below. Currently this includes Linux, OS X, and Microsoft Windows.
The instructions in this section should work for any platform, with a few exceptions for Windows as noted.

Got Java?
You will need the Java Runtime Environment (JRE) version 1.7 or higher. At a command line, check your Java
version like this:
$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

The output will vary, but you need to make sure you have version 1.7 or higher. If you don't have the required
version, or if the java command is not found, download and install the latest version from Oracle at http://www.or
acle.com/technetwork/java/javase/downloads/index.html.

Installing Solr
Apache Solr Reference Guide 5.1

4

Solr is available from the Solr website at http://lucene.apache.org/solr/.
For Linux/Unix/OSX systems, download the .tgz file. For Microsoft Windows systems, download the .zip file.
When getting started, all you need to do is extract the Solr distribution archive to a directory of your choosing.
When you're ready to setup Solr for a production environment, please refer to the instructions provided on the Ta
king Solr to Production page. To keep things simple for now, extract the Solr distribution archive to your local
home directory, for instance on Linux, do:
$ cd ~/
$ tar zxf solr-5.0.0.tgz

Once extracted, you are now ready to run Solr using the instructions provided in the Running Solr section.

Running Solr
This section describes how to run Solr with an example schema, how to add documents, and how to run queries.

Start the Server
If you didn't start Solr after installing it, you can start it by running bin/solr from the Solr directory.
$ bin/solr start

If you are running Windows, you can start Solr by running bin\solr.cmd instead.
bin\solr.cmd start

This will start Solr in the background, listening on port 8983.
When you start Solr in the background, the script will wait to make sure Solr starts correctly before returning to
the command line prompt.
The bin/solr and bin\solr.cmd scripts allow you to customize how you start Solr. Let's work through a few
examples of using the bin/solr script (if you're running Solr on Windows, the bin\solr.cmd works the same
as what is shown in the examples below):

Solr Script Options
The bin/solr script has several options.

Script Help
To see how to use the bin/solr script, execute:
$ bin/solr -help

For specific usage instructions for the start command, do:

Apache Solr Reference Guide 5.1

5

$ bin/solr start -help

Start Solr in the Foreground
Since Solr is a server, it is more common to run it in the background, especially on Unix/Linux. However, to start
Solr in the foreground, simply do:
$ bin/solr start -f

If you are running windows, you can run:
bin\solr.cmd start -f

Start Solr with a Different Port
To change the port Solr listens on, you can use the -p parameter when starting, such as:
$ bin/solr start -p 8984

Stop Solr
When running Solr in the foreground (using -f), then you can stop it using Ctrl-c. However, when running in the
background, you should use the stop command, such as:
$ bin/solr stop -p 8983

The stop command requires you to specify the port Solr is listening on or you can use the -all parameter to
stop all running Solr instances.

Start Solr with a Specific Example Configuration
Solr also provides a number of useful examples to help you learn about key features. You can launch the
examples using the -e flag. For instance, to launch the "techproducts" example, you would do:
$ bin/solr -e techproducts

Currently, the available examples you can run are: techproducts, dih, schemaless, and cloud. See the section Ru
nning with Example Configurations for details on each example.
Getting Started with SolrCloud
Running the cloud example starts Solr in SolrCloud mode. For more information on starting Solr in
cloud mode, see the section Getting Started with SolrCloud.

Check if Solr is Running
If you're not sure if Solr is running locally, you can use the status command:

Apache Solr Reference Guide 5.1

6

$ bin/solr status

This will search for running Solr instances on your computer and then gather basic information about them, such
as the version and memory usage.
That's it! Solr is running. If you need convincing, use a Web browser to see the Admin Console.
http://localhost:8983/solr/

The Solr Admin interface.
If Solr is not running, your browser will complain that it cannot connect to the server. Check your port number
and try again.

Create a Core
If you did not start Solr with an example configuration, you would need to create a core in order to be able to
index and search. You can do so by running:
$ bin/solr create -c 

This will create a core that uses a data-driven schema which tries to guess the correct field type when you add
documents to the index.
To see all available options for creating a new core, execute:
$ bin/solr create -help

Add Documents
Solr is built to find documents that match queries. Solr's schema provides an idea of how content is structured
(more on the schema later), but without documents there is nothing to find. Solr needs input before it can do
much.

Apache Solr Reference Guide 5.1

7

You may want to add a few sample documents before trying to index your own content. The Solr installation
comes with different types of example documents located under the sub-directories of the example/ directory of
your installation.
In the bin/ directory is the post script, a command line tool which can be used to index different types of
documents. Do not worry too much about the details for now. The Indexing and Basic Data Operations section
has all the details on indexing.
To see some information about the usage of bin/post, use the -help option.
bin/post can post various types of content to Solr, including files in Solr's native XML and JSON formats, CSV
files, a directory tree of rich documents, or even a simple short web crawl. See the examples at the end of
`bin/post -help` for various commands to easily get started posting your content into Solr.
Go ahead and add all the documents in some example XML files:
$ bin/post -c gettingstarted example/exampledocs/*.xml
SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
Entering auto mode. File endings considered are
xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt
,log
POSTing file gb18030-example.xml (application/xml) to [base]
POSTing file hd.xml (application/xml) to [base]
POSTing file ipod_other.xml (application/xml) to [base]
POSTing file ipod_video.xml (application/xml) to [base]
POSTing file manufacturers.xml (application/xml) to [base]
POSTing file mem.xml (application/xml) to [base]
POSTing file money.xml (application/xml) to [base]
POSTing file monitor.xml (application/xml) to [base]
POSTing file monitor2.xml (application/xml) to [base]
POSTing file mp500.xml (application/xml) to [base]
POSTing file sd500.xml (application/xml) to [base]
POSTing file solr.xml (application/xml) to [base]
POSTing file utf8-example.xml (application/xml) to [base]
POSTing file vidcard.xml (application/xml) to [base]
14 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
Time spent: 0:00:00.153

That's it! Solr has indexed the documents contained in those files.

Ask Questions
Now that you have indexed documents, you can perform queries. The simplest way is by building a URL that
includes the query parameters. This is exactly the same as building any other HTTP URL.
For example, the following query searches all document fields for "video":
http://localhost:8983/solr/gettingstarted/select?q=video
Notice how the URL includes the host name (localhost), the port number where the server is listening (8983),
the application name (solr), the request handler for queries (select), and finally, the query itself (q=video).
The results are contained in an XML document, which you can examine directly by clicking on the link above.
The document contains two parts. The first part is the responseHeader, which contains information about the

Apache Solr Reference Guide 5.1

8

response itself. The main part of the reply is in the result tag, which contains one or more doc tags, each of
which contains fields from documents that match the query. You can use standard XML transformation
techniques to mold Solr's results into a form that is suitable for displaying to users. Alternatively, Solr can output
the results in JSON, PHP, Ruby and even user-defined formats.
Just in case you are not running Solr as you read, the following screen shot shows the result of a query (the next
example, actually) as viewed in Mozilla Firefox. The top-level response contains a lst named responseHeade
r and a result named response. Inside result, you can see the three docs that represent the search results.

An XML response to a query.
Once you have mastered the basic idea of a query, it is easy to add enhancements to explore the query syntax.
This one is the same as before but the results only contain the ID, name, and price for each returned document.
If you don't specify which fields you want, all of them are returned.
http://localhost:8983/solr/gettingstarted/select?q=video&fl=id,name,price
Here is another example which searches for "black" in the name field only. If you do not tell Solr which field to

Apache Solr Reference Guide 5.1

9

search, it will search default fields, as specified in the schema.
http://localhost:8983/solr/gettingstarted/select?q=name:black
You can provide ranges for fields. The following query finds every document whose price is between $0 and
$400.
http://localhost:8983/solr/gettingstarted/select?q=price:[0%20TO%20400]&fl=id,name
,price
Faceted browsing is one of Solr's key features. It allows users to narrow search results in ways that are
meaningful to your application. For example, a shopping site could provide facets to narrow search results by
manufacturer or price.
Faceting information is returned as a third part of Solr's query response. To get a taste of this power, take a look
at the following query. It adds facet=true and facet.field=cat.
http://localhost:8983/solr/gettingstarted/select?q=price:[0%20TO%20400]&fl=id,name
,price&facet=true&facet.field=cat
In addition to the familiar responseHeader and response from Solr, a facet_counts element is also present.
Here is a view with the responseHeader and response collapsed so you can see the faceting information
clearly.

Apache Solr Reference Guide 5.1

10

An XML Response with faceting


...



SOLR1000
Solr, the Enterprise Search Server
0.0
...





6
3
2
2
1
1
1
1
1
1
0
0
0
0
0







The facet information shows how many of the query results have each possible value of the cat field. You could
easily use this information to provide users with a quick way to narrow their query results. You can filter results
by adding one or more filter queries to the Solr request. Here is a request further constraining the request to
documents with a category of "software".
http://localhost:8983/solr/gettingstarted/select?q=price:0%20TO%20400&fl=id,name,p
rice&facet=true&facet.field=cat&fq=cat:software

A Quick Overview
Having had some fun with Solr, you will now learn about all the cool things it can do.
Here is a example of how Solr might be integrated into an application:

Apache Solr Reference Guide 5.1

11

In the scenario above, Solr runs along side other server applications. For example, an online store application
would provide a user interface, a shopping cart, and a way to make purchases for end users; while an inventory
management application would allow store employees to edit product information. The product metadata would
be kept in some kind of database, as well as in Solr.
Solr makes it easy to add the capability to search through the online store through the following steps:
1. Define a schema. The schema tells Solr about the contents of documents it will be indexing. In the online
store example, the schema would define fields for the product name, description, price, manufacturer, and
so on. Solr's schema is powerful and flexible and allows you to tailor Solr's behavior to your application.
See Documents, Fields, and Schema Design for all the details.
2. Deploy Solr to your application server.
3. Feed Solr the document for which your users will search.
4. Expose search functionality in your application.
Because Solr is based on open standards, it is highly extensible. Solr queries are RESTful, which means, in
essence, that a query is a simple HTTP request URL and the response is a structured document: mainly XML,
but it could also be JSON, CSV, or some other format. This means that a wide variety of clients will be able to
use Solr, from other web applications to browser clients, rich client applications, and mobile devices. Any
platform capable of HTTP can talk to Solr. See Client APIs for details on client APIs.
Solr is based on the Apache Lucene project, a high-performance, full-featured search engine. Solr offers support
for the simplest keyword searching through to complex queries on multiple fields and faceted search results. Sea
rching has more information about searching and queries.
If Solr's capabilities are not impressive enough, its ability to handle very high-volume applications should do the
trick.
A relatively common scenario is that you have so much data, or so many queries, that a single Solr server is
unable to handle your entire workload. In this case, you can scale up the capabilities of your application using So

Apache Solr Reference Guide 5.1

12

lrCloud to better distribute the data, and the processing of requests, across many servers. Multiple options can
be mixed and matched depending on the type of scalability you need.
For example: "Sharding" is a scaling technique in which a collection is split into multiple logical pieces called
"shards" in in order to scale up the number of documents in a collection beyond what could physically fit on a
single server. Incoming queries are distributed to every shard in the collection, which respond with merged
results. Another technique available is to increase the "Replication Factor" of your collection, which allows you to
add servers with additional copies of your collection to handle higher concurrent query load by spreading the
requests around to multiple machines. Sharding and Replication are not mutually exclusive, and together make
Solr an extremely powerful and scalable platform.
Best of all, this talk about high-volume applications is not just hypothetical: some of the famous Internet sites that
use Solr today are Macy's, EBay, and Zappo's.
For more information, take a look at https://wiki.apache.org/solr/PublicServers.

A Step Closer
You already have some idea of Solr's schema. This section describes Solr's home directory and other
configuration options.
When Solr runs in an application server, it needs access to a home directory. The home directory contains
important configuration information and is the place where Solr will store its index. The layout of the home
directory will look a little different when you are running Solr in standalone mode vs when you are running in
SolrCloud mode.
The crucial parts of the Solr home directory are shown in these examples:
Standalone Mode
/
solr.xml
core_name1/
core.properties
conf/
solrconfig.xml
schema.xml
data/
core_name2/
core.properties
conf/
solrconfig.xml
schema.xml
data/

Apache Solr Reference Guide 5.1

13

SolrCloud Mode
/
solr.xml
core_name1/
core.properties
data/
core_name2/
core.properties
data/

You may see other files, but the main ones you need to know are:
solr.xml specifies configuration options for your Solr server instance. For more information on solr.xm
l see Solr Cores and solr.xml.
Per Solr Core:
core.properties defines specific properties for each core such as its name, the collection the
core belongs to, the location of the schema, and other parameters. For more details on core.pro
perties, see the section Defining core.properties.
solrconfig.xml controls high-level behavior. You can, for example, specify an alternate location
for the data directory. For more information on solrconfig.xml, see Configuring solrconfig.xml.
schema.xml describes the documents you will ask Solr to index. Inside schema.xml, you define
a document as a collection of fields. You get to define both the field types and the fields
themselves. Field type definitions are powerful and include information about how Solr processes
incoming field values and query values. For more information on schema.xml, see Documents,
Fields, and Schema Design.
data/ The directory containing the low level index files.
Note that the SolrCloud example does not include a conf directory for each Solr Core (so there is no solrconf
ig.xml or schema.xml). This is because the configuration files usually found in the conf directory are stored
in ZooKeeper so they can be propagated across the cluster.
If you are using SolrCloud with the embedded ZooKeeper instance, you may also see zoo.cfg and zoo.data
which are ZooKeeper configuration and data files. However, if you are running your own ZooKeeper ensemble,
you would supply your own ZooKeeper configuration file when you start it and the copies in Solr would be
unused. For more information about ZooKeeper and SolrCloud, see the section SolrCloud.

Solr Start Script Reference
Solr includes a script known as "bin/solr" that allows you to start and stop Solr, create and delete collections
or cores, and check the status of Solr and configured shards. You can find the script in the bin/ directory of your
Solr installation. The bin/solr script makes Solr easier to work with by providing simple commands and
options to quickly accomplish common goals.
In this section, the headings below correspond to available commands. For each command, the available options
are described with examples.
More examples of bin/solr in use are available throughout the Solr Reference Guide, but particularly in the
sections Running Solr and Getting Started with SolrCloud.

Apache Solr Reference Guide 5.1

14

Starting and Stopping
Start and Restart
Status
Stop
Healthcheck
Collections and Cores
Create
Delete

Starting and Stopping
Start and Restart
The start command starts Solr. The restart command allows you to restart Solr while it is already running or if it
has been stopped already.
The start and restart commands have several options to allow you to run in SolrCloud mode, use an example
configuration set, start with a hostname or port that is not the default and point to a local ZooKeeper ensemble.
bin/solr start [options]
bin/solr start -help
bin/solr restart [options]
bin/solr restart -help
When using the restart command, you must pass all of the parameters you initially passed when you started
Solr. Behind the scenes, a stop request is initiated, so Solr will be stopped before being started again. If no
nodes are already running, restart will skip the step to stop and proceed to starting Solr.

Available Parameters
The bin/solr script provides many options to allow you to customize the server in common ways, such as
changing the listening port. However, most of the defaults are adequate for most Solr installations, especially
when just getting started.

Parameter

Description

Example

-a ""

Start Solr with additional JVM
parameters, such as those starting with
-X. If you are passing JVM parameters
that begin with "-D", you can omit the -a
option.

bin/solr start -a
"-Xdebug -Xrunjdwp:transport=dt_socket,
server=y,suspend=n,address=1044"

Apache Solr Reference Guide 5.1

15

-cloud

Start Solr in SolrCloud mode, which will
also launch the embedded ZooKeeper
instance included with Solr.

bin/solr start -c

This option can be shortened to simply
-c.
If you are already running a ZooKeeper
ensemble that you want to use instead
of the embedded (single-node)
ZooKeeper, you should also pass the -z
parameter.
For more details, see the section SolrCl
oud Mode below.
-d 

Define a server directory, defaults to se

bin/solr start -d newServerDir

rver (as in, $SOLR_HOME/server). It
is uncommon to override this option.
When running multiple instances of Solr
on the same host, it is more common to
use the same server directory for each
instance and use a unique Solr home
directory using the -s option.
-e 

Start Solr with an example
configuration. These examples are
provided to help you get started faster
with Solr generally, or just try a specific
feature.

bin/solr start -e schemaless

The available options are:
cloud
techproducts
dih
schemaless
See the section Running with Example
Configurations below for more details
on the example configurations.
-f

Start Solr in the foreground; you cannot
use this option when running examples
with the -e option.

bin/solr start -f

-h


Start Solr with the defined hostname. If
this is not specified, 'localhost' will be
assumed.

bin/solr start -h search.mysolr.com

Apache Solr Reference Guide 5.1

16

-m


Start Solr with the defined value as the
min (-Xms) and max (-Xmx) heap size
for the JVM.

bin/solr start -m 1g

-noprompt

Start Solr and suppress any prompts
that may be seen with another option.
This would have the side effect of
accepting all defaults implicitly.

bin/solr start -e cloud -noprompt

For example, when using the "cloud"
example, an interactive session guides
you through several options for your
SolrCloud cluster. If you want to accept
all of the defaults, you can simply add
the -noprompt option to your request.
-p 

Start Solr on the defined port. If this is
not specified, '8983' will be used.

bin/solr start -p 8655

-s 

Sets the solr.solr.home system
property; Solr will create core
directories under this directory. This
allows you to run multiple Solr
instances on the same host while
reusing the same server directory set
using the -d parameter. If set, the
specified directory should contain a
solr.xml file. The default value is serve

bin/solr start -s newHome

r/solr.
This parameter is ignored when running
examples (-e), as the solr.solr.home
depends on which example is run.
-V

Start Solr with verbose messages from
the start script.

bin/solr start -V

-z 

Start Solr with the defined ZooKeeper
connection string. This option is only
used with the -c option, to start Solr in
SolrCloud mode. If this option is not
provided, Solr will start the embedded
ZooKeeper instance and use that
instance for SolrCloud operations.

bin/solr start -c -z
server1:2181,server2:2181

To emphasize how the default settings work take a moment to understand that the following commands are
equivalent:
bin/solr start
bin/solr start -h localhost -p 8983 -d server -s solr -m 512m

Apache Solr Reference Guide 5.1

17

It is not necessary to define all of the options when starting if the defaults are fine for your needs.

Setting Java System Properties
The bin/solr script will pass any additional parameters that begin with -D to the JVM, which allows you to set
arbitrary Java system properties. For example, to set the auto soft-commit frequency to 3 seconds, you can do:
bin/solr start -Dsolr.autoSoftCommit.maxTime=3000

SolrCloud Mode
The -c and -cloud options are equivalent:
bin/solr start -c
bin/solr start -cloud
If you specify a ZooKeeper connection string, such as -z 192.168.1.4:2181, then Solr will connect to
ZooKeeper and join the cluster. If you do not specify the -z option when starting Solr in cloud mode, then Solr will
launch an embedded ZooKeeper server listening on the Solr port + 1000, i.e., if Solr is running on port 8983,
then the embedded ZooKeeper will be listening on port 9983.
IMPORTANT: If your ZooKeeper connection string uses a chroot, such as localhost:2181/solr, then you
need to bootstrap the /solr znode before launching SolrCloud using the bin/solr script. To do this, you need to
use the zkcli.sh script shipped with Solr, such as:
server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd bootstrap
-solrhome server/solr
When starting in SolrCloud mode, the interactive script session will prompt you to choose a configset to use.
For more information about starting Solr in SolrCloud mode, see also the section Getting Started with SolrCloud.

Running with Example Configurations
bin/solr start -e 
The example configurations allow you to get started quickly with a configuration that mirrors what you hope to
accomplish with Solr. The following examples are provided:
cloud: This example starts a 1-4 node SolrCloud cluster on a single machine. When chosen, an
interactive session will start to guide you through options to select the number of nodes for your example
cluster, the ports to use, and name of the collection to be created.
techproducts: This example starts Solr in standalone mode with a schema designed for the sample
documents included in the $SOLR_HOME/example/exampledocs directory. The configset used does
not have SolrCloud or schemaless modes enabled, so fields must be explicitly defined in schema.xmlin
order for documents including those fields to be added to the index. The configset used can be found in $
SOLR_HOME/server/solr/configsets/sample_techproducts_configs.
dih: This example starts Solr in standalone mode with the DataImportHandler (DIH) enabled and several
example dataconfig.xml files pre-configured for different types of data supported with DIH (such as,
database contents, email, RSS feeds, etc.). For more information about DIH, see the section Uploading
Structured Data Store Data with the Data Import Handler.
schemaless: This example starts Solr in standalone mode using a managed schema, as described in the
section Managed Schema Definition in SolrConfig, and provides a very minimal pre-defined schema. Solr

Apache Solr Reference Guide 5.1

18

will run in Schemaless Mode with this configuration, where Solr will create fields in the schema on the fly
and will guess field types used in incoming documents. The configset used can be found in $SOLR_HOME
/server/solr/configsets/data_driven_schema_configs.
Note: The run-in-foreground option (-f) does not work with the -e option since the script needs to perform
additional tasks after starting the Solr server.

Status
The status command displays basic JSON-formatted information for any Solr nodes found running on the local
system. The status command uses the SOLR_PID_DIR environment variable to locate Solr process ID files to
find running Solr instances; the SOLR_PID_DIR variable defaults to the bin directory.
bin/solr status
The output will include a status of each node of the cluster, as in this example:
Found 2 Solr nodes:
Solr process 39920 running on port 7574
{
"solr_home":"/Applications/Solr/solr-5.0.0/example/cloud/node2/solr/",
"version":"5.0.0 1658469 - anshumgupta - 2015-02-09 09:54:36",
"startTime":"2015-02-10T17:19:54.739Z",
"uptime":"1 days, 23 hours, 55 minutes, 48 seconds",
"memory":"77.2 MB (%15.7) of 490.7 MB",
"cloud":{
"ZooKeeper":"localhost:9865",
"liveNodes":"2",
"collections":"2"}}
Solr process 39827 running on port 8865
{
"solr_home":"/Applications/Solr/solr-5.0.0/example/cloud/node1/solr/",
"version":"5.0.0 1658469 - anshumgupta - 2015-02-09 09:54:36",
"startTime":"2015-02-10T17:19:49.057Z",
"uptime":"1 days, 23 hours, 55 minutes, 54 seconds",
"memory":"94.2 MB (%19.2) of 490.7 MB",
"cloud":{
"ZooKeeper":"localhost:9865",
"liveNodes":"2",
"collections":"2"}}

Stop
The stop command sends a STOP request to a running Solr node, which allows it to shutdown gracefully. The
command will wait up to 5 seconds for Solr to stop gracefully and then will forcefully kill the process (kill -9).
bin/solr stop [options]
bin/solr stop -help

Available Parameters
Parameter

Apache Solr Reference Guide 5.1

Description

Example

19

-p 

Stop Solr running on the given port. If you are running more than one instance,
or are running in SolrCloud mode, you either need to specify the ports in
separate requests or use the -all option.

bin/solr
stop -p
8983

-all

Stop all running Solr instances that have a valid PID.

bin/solr
stop -all

-k 

Stop key used to protect from stopping Solr inadvertently; default is "solrrocks".

bin/solr
stop -k
solrrocks

Healthcheck
The healthcheck command generates a JSON-formatted health report for a collection when running in SolrCloud
mode. The health report provides information about the state of every replica for all shards in a collection,
including the number of committed documents and its current state.
bin/solr healthcheck [options]
bin/solr healthcheck -help

Available Parameters
Parameter

Description

Example

-c


Name of the collection to run a healthcheck against (required).

bin/solr
healthcheck
-c
gettingstarted

-z 

ZooKeeper connection string, defaults to localhost:9983. If you are
running Solr on a port other than 8983, you will have to specify the
ZooKeeper connection string. By default, this will be the Solr port + 1000.

bin/solr
healthcheck
-z
localhost:2181

Below is an example healthcheck request and response using a non-standard ZooKeeper connect string, with 2
nodes running:

Apache Solr Reference Guide 5.1

20

$ bin/solr healthcheck -c gettingstarted -z localhost:9865
{
"collection":"gettingstarted",
"status":"healthy",
"numDocs":0,
"numShards":2,
"shards":[
{
"shard":"shard1",
"status":"healthy",
"replicas":[
{
"name":"core_node1",
"url":"http://10.0.1.10:8865/solr/gettingstarted_shard1_replica2/",
"numDocs":0,
"status":"active",
"uptime":"2 days, 1 hours, 18 minutes, 48 seconds",
"memory":"25.6 MB (%5.2) of 490.7 MB",
"leader":true},
{
"name":"core_node4",
"url":"http://10.0.1.10:7574/solr/gettingstarted_shard1_replica1/",
"numDocs":0,
"status":"active",
"uptime":"2 days, 1 hours, 18 minutes, 42 seconds",
"memory":"95.3 MB (%19.4) of 490.7 MB"}]},
{
"shard":"shard2",
"status":"healthy",
"replicas":[
{
"name":"core_node2",
"url":"http://10.0.1.10:8865/solr/gettingstarted_shard2_replica2/",
"numDocs":0,
"status":"active",
"uptime":"2 days, 1 hours, 18 minutes, 48 seconds",
"memory":"25.8 MB (%5.3) of 490.7 MB"},
{
"name":"core_node3",
"url":"http://10.0.1.10:7574/solr/gettingstarted_shard2_replica1/",
"numDocs":0,
"status":"active",
"uptime":"2 days, 1 hours, 18 minutes, 42 seconds",
"memory":"95.4 MB (%19.4) of 490.7 MB",
"leader":true}]}]}

Collections and Cores
The bin/solr script can also help you create new collections (in SolrCloud mode) or cores (in standalone mode),
or delete collections.

Create
The create command detects the mode that Solr is running in (standalone or SolrCloud) and then creates a core
or collection depending on the mode.

Apache Solr Reference Guide 5.1

21

bin/solr create options
bin/solr create -help

Available Parameters
Parameter

Description

Example

-c 

Name of the core or collection to create (required).

bin/solr
create -c
mycollection

-d 

The configuration directory. This defaults to data_driven_schema_

bin/solr
create -d
basic_configs

configs.
See the section Configuration Directories and SolrCloud below for
more details about this option when running in SolrCloud mode.
-n 

The configuration name. This defaults to the same name as the core
or collection.

bin/solr
create -n
basic

-p 

Port of a local Solr instance to send the create command to; by
default the script tries to detect the port by looking for running Solr
instances.

bin/solr
create -p
8983

This option is useful if you are running multiple standalone Solr
instances on the same host, thus requiring you to be specific about
which instance to create the core in.
-s 
-shards
-rf 
-replicationFactor

Number of shards to split a collection into, default is 1; only applies
when Solr is running in SolrCloud mode.

bin/solr
create -s 2

Number of copies of each document in the collection. The default is 1
(no replication).

bin/solr -rf
2

Configuration Directories and SolrCloud
Before creating a collection in SolrCloud, the configuration directory used by the collection must be uploaded to
ZooKeeper. The create command supports several use cases for how collections and configuration directories
work. The main decision you need to make is whether a configuration directory in ZooKeeper should be shared
across multiple collections. Let's work through a few examples to illustrate how configuration directories work in
SolrCloud.
First, if you don't provide the -d or -n options, then the default configuration ($SOLR_HOME/server/solr/con
figsets/data_driven_schema_configs/conf) is uploaded to ZooKeeper using the same name as the
collection. For example, the following command will result in the data_driven_schema_configs configuration
being uploaded to /configs/contacts in ZooKeeper: bin/solr create -c contacts. If you create
another collection, by doing bin/solr create -c contacts2, then another copy of the data_driven_sch

Apache Solr Reference Guide 5.1

22

ema_configs directory will be uploaded to ZooKeeper under /configs/contacts2. Any changes you make
to the configuration for the contacts collection will not affect the contacts2 collection. Put simply, the default
behavior creates a unique copy of the configuration directory for each collection you create.
You can override the name given to the configuration directory in ZooKeeper by using the -n option. For
instance, the command bin/solr create -c logs -d basic_configs -n basic will upload the serve
r/solr/configsets/basic_configs/conf directory to ZooKeeper as /configs/basic.
Notice that we used the -d option to specify a different configuration than the default. Solr provides several
built-in configurations under server/solr/configsets. However you can also provide the path to your own
configuration directory using the -d option. For instance, the command bin/solr create -c mycoll -d
/tmp/myconfigs, will upload /tmp/myconfigs into ZooKeeper under /configs/mycoll . To reiterate, the
configuration directory is named after the collection unless you override it using the -n option.
Other collections can share the same configuration by specifying the name of the shared configuration using the
-n option. For instance, the following command will create a new collection that shares the basic configuration
created previously: bin/solr create -c logs2 -n basic.

Data-driven schema and shared configurations
The data_driven_schema_configs schema can mutate as data is indexed. Consequently, we recommend that
you do not share data-driven configurations between collections unless you are certain that all collections should
inherit the changes made when indexing data into one of the collections.

Delete
The delete command detects the mode that Solr is running in (standalone or SolrCloud) and then deletes the
specified core (standalone) or collection (SolrCloud) as appropriate.
bin/solr delete [options]
bin/solr delete -help
If running in SolrCloud mode, the delete command checks if the configuration directory used by the collection
you are deleting is being used by other collections. If not, then the configuration directory is also deleted from
ZooKeeper. For example, if you created a collection by doing bin/solr create -c contacts, then the
delete command bin/solr delete -c contacts will check to see if the /configs/contacts configuratio
n directory is being used by any other collections. If not, then the /configs/contacts directory is removed
from ZooKeeper.

Available Parameters
Parameter

Description

Example

-c 

Name of the core / collection to delete (required).

bin/solr
delete -c
mycoll

-deleteConfig


Delete the configuration directory from ZooKeeper. The default is true.

bin/solr
delete
-deleteConfig
false

If the configuration directory is being used by another collection, then it
will not be deleted even if you pass -deleteConfig true.

Apache Solr Reference Guide 5.1

23

-p 

The port of a local Solr instance to send the delete command to. By
default the script tries to detect the port by looking for running Solr
instances.

bin/solr
delete -p 8983

This option is useful if you are running multiple standalone Solr
instances on the same host, thus requiring you to be specific about
which instance to delete the core from.

Apache Solr Reference Guide 5.1

24

Upgrading Solr
If you are already using Solr 5.0, Solr 5.1 should not present any major problems. However, you should review
the CHANGES.txt file found in your Solr package for changes and updates that may effect your existing
implementation.

Upgrading from 5.0.x
SolrClient query functions now declare themselves as throwing IOException in addition to SolrSer
verException, to bring them in line with the update functions.
SolrRequest.process() is now final. Subclasses should instead be parameterized by their
corresponding SolrResponse type, and implement createResponse().
The signature of SolrDispatchFilter.createCoreContainer() has changed to take (String,P
roperties) arguments.
Deprecated the 'lib' option added to create-requesthandler as part of SOLR-6801 in 5.0 release.
Please use the add-runtimelib command.
Tika's runtime dependency of 'jhighlight' was removed as the latter was found to contain some LGPL-only
code. Until that's resolved by Tika, you can download the jar yourself and place it under contrib/extra
ction/lib.
The _text catch-all field in data_driven_schema_configs has been renamed to _text_.

Upgrading from Older Versions of Solr
Users upgrading from older versions are strongly encouraged to consult CHANGES.txt for the details of all cha
nges since the version they are upgrading from
For users upgrading from Solr 4.x, a summary of the significant changes can be found in the Major Changes
from Solr 4 to Solr 5 section.

Apache Solr Reference Guide 5.1

25

Using the Solr Administration User Interface
This section discusses the Solr Administration User Interface ("Admin UI").
The Overview of the Solr Admin UI explains the basic features of the user interface, what's on the initial Admin UI
page, and how to configure the interface. In addition, there are pages describing each screen of the Admin UI:
Getting Assistance shows you how to get more information about the UI.
Logging explains the various logging levels available and how to invoke them.
Cloud Screens display information about nodes when running in SolrCloud mode.
Core Admin explains how to get management information about each core.
Java Properties shows the Java information about each core.
Thread Dump lets you see detailed information about each thread, along with state information.
Core-Specific Tools is a section explaining additional screens available for each named core.
Analysis - lets you analyze the data found in specific fields.
Dataimport - shows you information about the current status of the Data Import Handler.
Documents - provides a simple form allowing you to execute various Solr indexing commands
directly from the browser.
Files - shows the current core configuration files such as solrconfig.xml and schema.xml.
Ping - lets you ping a named core and determine whether the core is active.
Plugins/Stats - shows statistics for plugins and other installed components.
Query - lets you submit a structured query about various elements of a core.
Replication - shows you the current replication status for the core, and lets you enable/disable
replication.
Schema Browser - displays schema data in a browser window.
Segments Info - Provides a visualization of the underlying Lucene index segments.

Overview of the Solr Admin UI
Solr features a Web interface that makes it easy for Solr administrators and programmers to view Solr
configuration details, run queries and analyze document fields in order to fine-tune a Solr configuration and
access online documentation and other help.

Apache Solr Reference Guide 5.1

26

Accessing the URL http://hostname:8983/solr/ will show the main dashboard, which is divided into two
parts.
A left-side of the screen is a menu under the Solr logo that provides the navigation through the screens of the UI.
The first set of links are for system-level information and configuration and provide access to Logging, Core
Admin and Java Properties, among other things. At the end of this information is a list of Solr cores configured
for this instance. Clicking on a core name shows a secondary menu of information and configuration options for
the core specifically. Items in this list include the Schema, Config, Plugins, and an ability to perform Queries on
indexed data.
The center of the screen shows the detail of the option selected. This may include a sub-navigation for the option
or text or graphical representation of the requested data. See the sections in this guide for each screen for more
details.
Under the covers, the Solr Admin UI re-uses the same HTTP APIs available to all clients to access Solr-related
data to drive an external interface.
The path to the Solr Admin UI given above is http://hostname:port/solr, which redirects to http
://hostname:port/solr/#/ in the current version. A convenience redirect is also supported, so
simply accessing the Admin UI at http://hostname:port/ will also redirect to http://hostname:
port/solr/#/.

Configuring the Admin UI in solrconfig.xml
You can configure the Solr Admin UI by editing the file solrconfig.xml.
The  block in the solrconfig.xml file determines the default query to be displayed in the Query
section of the core-specific pages. The default is *:*, which is to find all documents. In this example, we have
changed the default to the term solr.

solr


Related Topics
Configuring solrconfig.xml

Getting Assistance
At the bottom of each screen of the Admin UI is a set of links that can be used to get more assistance with
configuring and using Solr.

Assistance icons
These icons include the following links.

Apache Solr Reference Guide 5.1

27

Link

Description

Documentation

Navigates to the Apache Solr documentation hosted on http://lucene.apache.org/solr/.

Issue Tracker

Navigates to the JIRA issue tracking server for the Apache Solr project. This server resides
at http://issues.apache.org/jira/browse/SOLR.

IRC Channel

Navigates to an Apache Wiki page describing how to join Solr's IRClive-chat room: https://wi
ki.apache.org/solr/IRCChannels.

Community
forum

Navigates to the Apache Solr web page http://lucene.apache.org/solr/resources.html#comm
unity which has further information about ways to engage.

Solr Query
Syntax

Navigates to the section "Query Syntax and Parsing" in this reference guide.

These links cannot be modified without editing the admin.html in the solr.war that contains the Admin UI
files.

Logging
The Logging page shows messages from Solr's log files.
When you click the link for "Logging", a page similar to the one below will be displayed:

The Main Logging Screen, including an example of an error due to a bad document sent by a client
While this example shows logged messages for only one core, if you have multiple cores in a single instance,
they will each be listed, with the level for each.

Selecting a Logging Level

Apache Solr Reference Guide 5.1

28

When you select the Level link on the left, you see the hierarchy of classpaths and classnames for your
instance. A row highlighted in yellow indicates that the class has logging capabilities. Click on a highlighted row,
and a menu will appear to allow you to change the log level for that class. Characters in boldface indicate that
the class will not be affected by level changes to root.
For an explanation of the various logging levels, see Configuring Logging.

Cloud Screens
When running in SolrCloud mode, an option will appear in the Admin UI between Logging and Core Admin for
Cloud. It's not possible at the current time to manage the nodes of the SolrCloud cluster from the Admin UI, but
you can view them and open the Solr Admin UI on each node to view the status and statistics for the node and
each core on each node.
Only Visible When using SolrCloud
The "Cloud" menu option is only available on Solr instances running in SolrCloud mode. Single node or
master/slave replication instances of Solr will not display this option.

Click on the Cloud option in the left-hand navigation, and a small sub-menu appears with options called "Tree",
"Graph", "Graph (Radial)" and "Dump". The default view ("Graph") shows a graph of each collection, the shards
that make up those collections, and the addresses of each replica for each shard. This example shows the very
simple two-node, two-shard, two-replica cluster you can get running the "bin/solr -e cloud" example:

Apache Solr Reference Guide 5.1

29

The "Graph (Radial)" option provides a different visual view of each node. Using the same example cluster, the
radial graph view looks like:

The "Tree" option shows a directory structure of the files in ZooKeeper, including clusterstate.json,
configuration files, and other status and information files. In this example, we show the leader definition for
"shard1" in the "gettingstarted" collection:

Apache Solr Reference Guide 5.1

30

The final option is "Dump", which allows you to download an XML file with all the ZooKeeper configuration files.

Core Admin
The Core Admin screen lets you manage your cores.
The buttons at the top of the screen let you add a new core, unload the core displayed, rename the currently
displayed core, swap the existing core with one that you specify in a drop-down box, reload the current core, and
optimize the current core.
The main display and available actions correspond to the commands used with the CoreAdminHandler, but
provide another way of working with your cores.

Java Properties
The Java Properties screen provides easy access to one of the most essential components of a top-performing
Solr systems With the Java Properties screen, you can see all the properties of the JVM running Solr, including
the class paths, file encodings, JVM memory settings, operating system, and more.

Apache Solr Reference Guide 5.1

31

Thread Dump
The Thread Dump screen lets you inspect the currently active threads on your server. Each thread is listed and
access to the stacktraces is available where applicable. Icons to the left indicate the state of the thread: for
example, threads with a green check-mark in a green circle are in a "RUNNABLE" state. On the right of the
thread name, a down-arrow means you can expand to see the stacktrace for that thread.

When you move your cursor over a thread name, a box floats over the name with the state for that thread.
Thread states can be:
State

Meaning

NEW

A thread that has not yet started.

RUNNABLE

A thread executing in the Java virtual machine.

BLOCKED

A thread that is blocked waiting for a monitor lock.

Apache Solr Reference Guide 5.1

32

WAITING

A thread that is waiting indefinitely for another thread to perform a particular action.

TIMED_WAITING

A thread that is waiting for another thread to perform an action for up to a specified
waiting time.

TERMINATED

A thread that has exited.

When you click on one of the threads that can be expanded, you'll see the stacktrace, as in the example below:

Inspecting a thread
You can also check the Show all Stacktraces button to automatically enable expansion for all threads.

Core-Specific Tools
In the left-hand navigation bar, you will see a pull-down menu titled "Core Selector". Clicking on the menu will
show a list of Solr cores, with a search box that can be used to find a specific core (handy if you have a lot of
cores). When you select a core, a secondary menu opens under the core name with the administration options
available for that particular core.

After selecting the core, the central part of the screen shows Statistics and other information about the core you

Apache Solr Reference Guide 5.1

33

chose. You can define a file called admin-extra.html that includes links or other information you would like to
display in the "Admin Extra" part of this main screen.
On the left side, under the core name, are links to other screens that display information or provide options for
the specific core chosen. The core-specific options are listed below, with a link to the section of this Guide to find
out more:
Analysis - lets you analyze the data found in specific fields.
Dataimport - shows you information about the current status of the Data Import Handler.
Documents - provides a simple form allowing you to execute various Solr indexing commands directly
from the browser.
Files - shows the current core configuration files such as solrconfig.xml and schema.xml.
Ping - lets you ping a named core and determine whether the core is active.
Plugins/Stats - shows statistics for plugins and other installed components.
Query - lets you submit a structured query about various elements of a core.
Replication - shows you the current replication status for the core, and lets you enable/disable replication.
Schema Browser - displays schema data in a browser window.
Segments Info - Provides a visualization of the underlying Lucene index segments.

Analysis Screen
The Analysis screen lets you inspect how data will be handled according to the field, field type and dynamic rule
configurations found in schema.xml. You can analyze how content would be handled during indexing or during
query processing and view the results separately or at the same time. Ideally, you would want content to be
handled consistently, and this screen allows you to validate the settings in the field type or field analysis chains.
Enter content in one or both boxes at the top of the screen, and then choose the field or field type definitions to
use for analysis.

If you click the Verbose Output check box, you see more information, including transformations to the input
(such as, convert to lower case, strip extra characters, etc.) and the bytes, type and detailed position information.
The information displayed will vary depending on the settings of the field or field type. Each step of the process is
displayed in a separate section, with an abbreviation for the tokenizer or filter that is applied in that step. Hover or
click on the abbreviation, and you'll see the name and path of the tokenizer or filter.

Apache Solr Reference Guide 5.1

34

In example screenshot above, several transformations are applied to the input "Running is a sport." The words
"is" and "a" have been removed and the word "running" has been changed to its basic form, "run". This is
because we are using the field type text_en in this scenario, which is configured to remove stop words (small
words that usually do not provide a great deal of context) and "stem" terms when possible to find more possible
matches (this is particularly helpful with plural forms of words). If you click the question mark next to the Analyze
Fieldname/Field Type pull-down menu, the Schema Browser window will open, showing you the settings for the
field specified.
The section Understanding Analyzers, Tokenizers, and Filters describes in detail what each option is and how it
may transform your data and the section Running Your Analyzer has specific examples for using the Analysis
screen.

Dataimport Screen
The Dataimport screen shows the configuration of the DataImportHandler (DIH) and allows you start, and
monitor the status of, import commands as defined by the options selected on the screen and defined in the
configuration file.

Apache Solr Reference Guide 5.1

35

This screen also lets you adjust various options to control how the data is imported to Solr, and view the data
import configuration file that controls the import. For more information about data importing with DIH, see the
section on Uploading Structured Data Store Data with the Data Import Handler.

Documents Screen
The Documents screen provides a simple form allowing you to execute various Solr indexing commands in a
variety of formats directly from the browser.

The screen allows you to:
Copy documents in JSON, CSV or XML and submit them to the index
Upload documents (in JSON, CSV or XML)
Construct documents by selecting fields and field values
The first step is to define the RequestHandler to use (aka, 'qt'). By default /update will be defined. To use Solr
Cell, for example, change the request handler to /update/extract.

Apache Solr Reference Guide 5.1

36

Then choose the Document Type to define the type of document to load. The remaining parameters will change
depending on the document type selected.

JSON
When using the JSON document type, the functionality is similar to using a requestHandler on the command line.
Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The
document structure should still be in proper JSON format.
Then you can choose when documents should be added to the index (Commit Within), whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped), and, finally, if a document boost should be applied.
This option will only add or overwrite documents to the index; for other update tasks, see the Solr Command opti
on.

CSV
When using the CSV document type, the functionality is similar to using a requestHandler on the command line.
Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The
document structure should still be in proper CSV format, with columns delimited and one row per document.
Then you can choose when documents should be added to the index (Commit Within), and whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped).

Document Builder
The Document Builder provides a wizard-like interface to enter fields of a document

File Upload
The File Upload option allows choosing a prepared file and uploading it. If using only /update for the
Request-Handler option, you will be limited to XML, CSV, and JSON.
However, to use the ExtractingRequestHandler (aka Solr Cell), you can modify the Request-Handler to /update
/extract. You must have this defined in your solrconfig.xml file, with your desired defaults. You should
also update the &literal.id shown in the Extracting Req. Handler Params so the file chosen is given a
unique id.
Then you can choose when documents should be added to the index (Commit Within), and whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped).

Solr Command
The Solr Command option allows you use XML or JSON to perform specific actions on documents, such as
defining documents to be added or deleted, updating only certain fields of documents, or commit and optimize
commands on the index.
The documents should be structured as they would be if using /update on the command line.

XML

Apache Solr Reference Guide 5.1

37

When using the XML document type, the functionality is similar to using a requestHandler on the command line.
Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The
document structure should still be in proper Solr XML format, with each document separated by  tags and
each field defined.
Then you can choose when documents should be added to the index (Commit Within), and whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped).
This option will only add or overwrite documents to the index; for other update tasks, see the Solr Command opti
on.

Related Topics
Uploading Data with Index Handlers
Uploading Data with Solr Cell using Apache Tika

Files Screen
The Files screen lets you browse & view the various configuration files (such solrconfig.xml and schema.x
ml) for the core you selected.

While the solrconfig.xml defines the behaviour of Solr as it indexes content and responds to queries, the sc
hema.xml allows you to define the types of data in your content (field types), the fields your documents will be
broken into, and any dynamic fields that should be generated based on patterns of field names in the incoming
documents. Any other configuration files are used depending on how they are referenced in either solrconfig
.xml or schema.xml.
Configuration files cannot be edited with this screen, so a text editor of some kind must be used.
This screen is related to the Schema Browser Screen, in that they both can display information from the schema,
but the Schema Browser provides a way to drill into the analysis chain and displays linkages between field types,

Apache Solr Reference Guide 5.1

38

fields, and dynamic field rules.
Many of the options defined in solrconfig.xml and schema.xml are described throughout the rest of this
Guide. In particular, you will want to review these sections:
Indexing and Basic Data Operations
Searching
The Well-Configured Solr Instance
Documents, Fields, and Schema Design

Ping
Choosing Ping under a core name issues a ping request to check whether a server is up.
Ping is configured using a requestHandler in the solrconfig.xml file:



solrpingquery


all





The Ping option doesn't open a page, but the status of the request can be seen on the core overview page
shown when clicking on a collection name. The length of time the request has taken is displayed next to the Ping
option, in milliseconds.

Plugins & Stats Screen
The Plugins screen shows information and statistics about Solr's status and performance. You can find
information about the performance of Solr's caches, the state of Solr's searchers, and the configuration of
searchHandlers and requestHandlers.
Choose an area of interest on the right, and then drill down into more specifics by clicking on one of the names
that appear in the central part of the window. In this example, we've chosen to look at the Searcher stats, from
the Core area:

Apache Solr Reference Guide 5.1

39

Searcher Statistics
The display is a snapshot taken when the page is loaded. You can get updated status by choosing to either Wat
ch Changes or Refresh Values. Watching the changes will highlight those areas that have changed, while
refreshing the values will reload the page with updated information.

Query Screen
You can use Query, shown under the name of each core, to submit a search query to a Solr server and analyze
the results. In the example in the screenshot, a query has been submitted, and the screen shows the query
results sent to the browser as JSON.

The query was sent to a core named "techproducts". We used Solr's default query for this screen (as defined in s

Apache Solr Reference Guide 5.1

40

olrconfig.xml), which is *:*. This query will find all records in the index for this core. We kept the other
defaults, but the table below explains these options, which are also covered in detail in later parts of this Guide.
The response is shown to the right of the form. Requests to Solr are simply HTTP requests, and the query
submitted is shown in light type above the results; if you click on this it will open a new browser window with just
this request and response (without the rest of the Solr Admin UI). The rest of the response is shown in JSON,
which is part of the request (see the wt=json part at the end).
The response has at least two sections, but may have several more depending on the options chosen. The two
sections it always has are the responseHeader and the response. The responseHeader includes the status
of the search (status), the processing time (QTime), and the parameters (params) that were used to process
the query.
The response includes the documents that matched the query, in doc sub-sections. The fields return depend
on the parameters of the query (and the defaults of the request handler used). The number of results is also
included in this section.
This screen allows you to experiment with different query options, and inspect how your documents were
indexed. The query parameters available on the form are some basic options that most users want to have
available, but there are dozens more available which could be simply added to the basic request by hand (if
opened in a browser). The table below explains the parameters available:
Field

Description

Request-handler
(qt)

Specifies the query handler for the request. If a query handler is not specified, Solr
processes the response with the standard query handler.

q

The query event. See Searching for an explanation of this parameter.

fq

The filter queries. See Common Query Parameters for more information on this parameter.

sort

Sorts the response to a query in either ascending or descending order based on the
response's score or another specified characteristic.

start, rows

start is the offset into the query result starting at which documents should be returned.
The default value is 0, meaning that the query should return results starting with the first
document that matches. This field accepts the same syntax as the start query parameter,
which is described in Searching. rows is the number of rows to return.

fl

Defines the fields to return for each document. You can explicitly list the stored fields, functi
ons, and doc transformers you want to have returned by separating them with either a
comma or a space.

wt

Specifies the Response Writer to be used to format the query response. Defaults to XML if
not specified.

indent

Click this button to request that the Response Writer use indentation to make the
responses more readable.

debugQuery

Click this button to augment the query response with debugging information, including
"explain info" for each document returned. This debugging information is intended to be
intelligible to the administrator or programmer.

Apache Solr Reference Guide 5.1

41

dismax

Click this button to enable the Dismax query parser. See The DisMax Query Parser for
further information.

edismax

Click this button to enable the Extended query parser. See The Extended DisMax Query
Parser for further information.

hl

Click this button to enable highlighting in the query response. See Highlighting for more
information.

facet

Enables faceting, the arrangement of search results into categories based on indexed
terms. See Faceting for more information.

spatial

Click to enable using location data for use in spatial or geospatial searches. See Spatial
Search for more information.

spellcheck

Click this button to enable the Spellchecker, which provides inline query suggestions based
on other, similar, terms. See Spell Checking for more information.

Related Topics
Searching

Replication Screen
The Replication screen shows you the current replication state for the named core you have specified. SolrCloud
has supplanted much of this functionality, but if you are still using Master-Slave index replication, you can use
this screen to:
1. View the replicatable index state. (on a master node)
2. View the current replication status (on a slave node)
3. Disable replication. (on a master node)
Caution When Using SolrCloud
When using SolrCloud, do not attempt to disable replication via this screen.
More details on how to configure replication is available in the section called Index Replication.

Schema Browser Screen
The Schema Browser screen lets you see schema data in a browser window. If you have accessed this window
from the Analysis screen, it will be opened to a specific field, dynamic field rule or field type. If there is nothing
chosen, use the pull-down menu to choose the field or field type.

Apache Solr Reference Guide 5.1

42

The screen provides a great deal of useful information about each particular field. In the example above, we
have chosen the text field. On the right side of the center window, we see the field name, and a list of fields that
populate this field because they are defined to be copied to the text field. Click on one of those field names,
and you can see the definitions for that field. We can also see the field type, which would allow us to inspect the
type definitions as well.
In the left part of the center window, we see the field type again, and the defined properties for the field. We can
also see how many documents have populated this field. Then we see the analyzer used for indexing and query
processing. Click the icon to the left of either of those, and you'll see the definitions for the tokenizers and/or
filters that are used. The output of these processes is the information you see when testing how content is
handled for a particular field with the Analysis Screen.
Under the analyzer information is a button to Load Term Info. Clicking that button will show the top N terms that
are in the index for that field. Click on a term, and you will be taken to the Query Screen to see the results of a
query of that term in that field. If you want to always see the term information for a field, choose Autoload and it
will always appear when there are terms for a field. A histogram shows the number of terms with a given
frequency in the field.

Segments Info
The Segments Info screen lets you see a visualization of the various segments in the underlying Lucene index,
with information about the size of each segment – both bytes and in number of documents – as well as other
basic metadata about those segments, notably the number of deleted documents.

Apache Solr Reference Guide 5.1

43

This information may be useful for people to help make decisions about the optimal merge settings for their data.

Apache Solr Reference Guide 5.1

44

Documents, Fields, and Schema Design
This section discusses how Solr organizes its data into documents and fields, as well as how to work with the
Solr schema file, schema.xml. It includes the following topics:
Overview of Documents, Fields, and Schema Design: An introduction to the concepts covered in this section.
Solr Field Types: Detailed information about field types in Solr, including the field types in the default Solr
schema.
Defining Fields: Describes how to define fields in Solr.
Copying Fields: Describes how to populate fields with data copied from another field.
Dynamic Fields: Information about using dynamic fields in order to catch and index fields that do not exactly
conform to other field definitions in your schema.
Schema API: Use curl commands to read various parts of a schema or create new fields and copyField rules.
Other Schema Elements: Describes other important elements in the Solr schema.
Putting the Pieces Together: A higher-level view of the Solr schema and how its elements work together.
DocValues: Describes how to create a docValues index for faster lookups.
Schemaless Mode: Automatically add previously unknown schema fields using value-based field type guessing.

Overview of Documents, Fields, and Schema Design
The fundamental premise of Solr is simple. You give it a lot of information, then later you can ask it questions
and find the piece of information you want. The part where you feed in all the information is called indexing or up
dating. When you ask a question, it's called a query.
One way to understand how Solr works is to think of a loose-leaf book of recipes. Every time you add a recipe to
the book, you update the index at the back. You list each ingredient and the page number of the recipe you just
added. Suppose you add one hundred recipes. Using the index, you can very quickly find all the recipes that use
garbanzo beans, or artichokes, or coffee, as an ingredient. Using the index is much faster than looking through
each recipe one by one. Imagine a book of one thousand recipes, or one million.
Solr allows you to build an index with many different fields, or types of entries. The example above shows how to
build an index with just one field, ingredients. You could have other fields in the index for the recipe's cooking
style, like Asian, Cajun, or vegan, and you could have an index field for preparation times. Solr can answer
questions like "What Cajun-style recipes that have blood oranges as an ingredient can be prepared in fewer than
30 minutes?"
The schema is the place where you tell Solr how it should build indexes from input documents.

How Solr Sees the World
Solr's basic unit of information is a document, which is a set of data that describes something. A recipe document
would contain the ingredients, the instructions, the preparation time, the cooking time, the tools needed, and so
on. A document about a person, for example, might contain the person's name, biography, favorite color, and
shoe size. A document about a book could contain the title, author, year of publication, number of pages, and so
on.

Apache Solr Reference Guide 5.1

45

In the Solr universe, documents are composed of fields, which are more specific pieces of information. Shoe size
could be a field. First name and last name could be fields.
Fields can contain different kinds of data. A name field, for example, is text (character data). A shoe size field
might be a floating point number so that it could contain values like 6 and 9.5. Obviously, the definition of fields is
flexible (you could define a shoe size field as a text field rather than a floating point number, for example), but if
you define your fields correctly, Solr will be able to interpret them correctly and your users will get better results
when they perform a query.
You can tell Solr about the kind of data a field contains by specifying its field type. The field type tells Solr how to
interpret the field and how it can be queried.
When you add a document, Solr takes the information in the document's fields and adds that information to an
index. When you perform a query, Solr can quickly consult the index and return the matching documents.

Field Analysis
Field analysis tells Solr what to do with incoming data when building an index. A more accurate name for this
process would be processing or even digestion, but the official name is analysis.
Consider, for example, a biography field in a person document. Every word of the biography must be indexed so
that you can quickly find people whose lives have had anything to do with ketchup, or dragonflies, or
cryptography.
However, a biography will likely contains lots of words you don't care about and don't want clogging up your
index—words like "the", "a", "to", and so forth. Furthermore, suppose the biography contains the word "Ketchup",
capitalized at the beginning of a sentence. If a user makes a query for "ketchup", you want Solr to tell you about
the person even though the biography contains the capitalized word.
The solution to both these problems is field analysis. For the biography field, you can tell Solr how to break apart
the biography into words. You can tell Solr that you want to make all the words lower case, and you can tell Solr
to remove accents marks.
Field analysis is an important part of a field type. Understanding Analyzers, Tokenizers, and Filters is a detailed
description of field analysis.

Solr Field Types
The field type defines how Solr should interpret data in a field and how the field can be queried. There are many
field types included with Solr by default, and they can also be defined locally.
Topics covered in this section:
Field Type Definitions and Properties
Field Types Included with Solr
Working with Currencies and Exchange Rates
Working with Dates
Working with Enum Fields
Working with External Files and Processes
Field Properties by Use Case

Apache Solr Reference Guide 5.1

46

Related Topics
SchemaXML-DataTypes
FieldType Javadoc

Field Type Definitions and Properties
A field type definition can include four types of information:
The name of the field type (mandatory)
An implementation class name (mandatory)
If the field type is TextField, a description of the field analysis for the field type
Field type properties - depending on the implementation class, some properties may be mandatory.

Field Type Definitions in schema.xml
Field types are defined in schema.xml. Each field type is defined between fieldType elements. They can
optionally be grouped within a types element. Here is an example of a field type definition for a type called tex
t_general:















The first line in the example above contains the field type name, text_general, and the name of the
implementing class, solr.TextField. The rest of the definition is about field analysis, described in Understand
ing Analyzers, Tokenizers, and Filters.
The implementing class is responsible for making sure the field is handled correctly. In the class names in schem
a.xml, the string solr is shorthand for org.apache.solr.schema or org.apache.solr.analysis.
Therefore, solr.TextField is really org.apache.solr.schema.TextField..

Field Type Properties
The field type class determines most of the behavior of a field type, but optional properties can also be defined.
For example, the following definition of a date field type defines two properties, sortMissingLast and omitNo

Apache Solr Reference Guide 5.1

47

rms.


The properties that can be specified for a given field type fall into three major categories:
Properties specific to the field type's class.
General Properties Solr supports for any field type.
Field Default Properties that can be specified on the field type that will be inherited by fields that use this
type instead of the default behavior.

General Properties
Property

Description

Values

name

The name of the fieldType. This value gets used in field definitions,
in the "type" attribute. It is strongly recommended that names
consist of alphanumeric or underscore characters only and not start
with a digit. This is not currently strictly enforced.

class

The class name that gets used to store and index the data for this
type. Note that you may prefix included class names with "solr." and
Solr will automatically figure out which packages to search for the
class - so "solr.TextField" will work. If you are using a third-party
class, you will probably need to have a fully qualified class name.
The fully qualified equivalent for "solr.TextField" is
"org.apache.solr.schema.TextField".

positionIncrementGap

For multivalued fields, specifies a distance between multiple values,
which prevents spurious phrase matches

integer

autoGeneratePhraseQueries

For text fields. If true, Solr automatically generates phrase queries for
adjacent terms. If false, terms must be enclosed in double-quotes to
be treated as phrases.

true or
false

docValuesFormat

Defines a custom DocValuesFormat to use for fields of this type.

n/a

This requires that a schema-aware codec, such as the SchemaCode
cFactory has been configured in solrconfig.xml.
postingsFormat

Defines a custom PostingsFormat to use for fields of this type.

n/a

This requires that a schema-aware codec, such as the SchemaCode
cFactory has been configured in solrconfig.xml.

Lucene index back-compatibility is only supported for the default codec. If you choose to customize the p
ostingsFormat or docValuesFormat in your schema.xml, upgrading to a future version of Solr may
require you to either switch back to the default codec and optimize your index to rewrite it into the default
codec before upgrading, or re-build your entire index from scratch after upgrading.

Field Default Properties

Apache Solr Reference Guide 5.1

48

Property

Description

Values

indexed

If true, the value of the field can be used in queries to retrieve
matching documents

true or
false

stored

If true, the actual value of the field can be retrieved by queries

true or
false

docValues

If true, the value of the field will be put in a column-oriented DocValues
structure

true or
false

sortMissingFirst
sortMissingLast

Control the placement of documents when a sort field is not present.
As of Solr 3.5, these work for all numeric fields, including Trie and date
fields.

true or
false

multiValued

If true, indicates that a single document might contain multiple values
for this field type

true or
false

omitNorms

If true, omits the norms associated with this field (this disables length
normalization and index-time boosting for the field, and saves some
memory). Defaults to true for all primitive (non-analyzed) field types,
such as int, float, data, bool, and string. Only full-text fields or fields
that need an index-time boost need norms.

true or
false

omitTermFreqAndPositions

If true, omits term frequency, positions, and payloads from postings for
this field. This can be a performance boost for fields that don't require
that information. It also reduces the storage space required for the
index. Queries that rely on position that are issued on a field with this
option will silently fail to find documents. This property defaults to true
for all fields that are not text fields.

true or
false

omitPositions

Similar to omitTermFreqAndPositions but preserves term

true or
false

frequency information
termVectors
termPositions
termOffsets
termPayloads

These options instruct Solr to maintain full term vectors for each
document, optionally including position, offset and payload information
for each term occurrence in those vectors. These can be used to
accelerate highlighting and other ancillary functionality, but impose a
substantial cost in terms of index size. They are not necessary for
typical uses of Solr.

true or
false

required

Instructs Solr to reject any attempts to add a document which does not
have a value for this field. This property defaults to false.

true or
false

Field Types Included with Solr
The following table lists the field types that are available in Solr. The org.apache.solr.schema package
includes all the classes listed in this table.
Class
BinaryField

Apache Solr Reference Guide 5.1

Description
Binary data.

49

BoolField

Contains either true or false. Values of "1", "t", or "T" in the first
character are interpreted as true. Any other values in the first
character are interpreted as false.

CollationField

Supports Unicode collation for sorting and range queries.
ICUCollationField is a better choice if you can use ICU4J. See the
section Unicode Collation.

CurrencyField

Supports currencies and exchange rates. See the section Working
with Currencies and Exchange Rates.

DateRangeField

Supports indexing date ranges, to include point in time date instances
as well (single-millisecond durations). See the section Working with
Dates. Consider using this even if it's just for date instances,
particularly when the queries typically fall on UTC
year/month/day/hour etc. boundaries.

ExternalFileField

Pulls values from a file on disk. See the section Working with External
Files and Processes.

EnumField

Allows defining an enumerated set of values which may not be easily
sorted by either alphabetic or numeric order (such as a list of
severities, for example). This field type takes a configuration file, which
lists the proper order of the field values. See the section Working with
Enum Fields for more information.

ICUCollationField

Supports Unicode collation for sorting and range queries. See the
section Unicode Collation.

LatLonType

Spatial Search: a latitude/longitude coordinate pair. The latitude is
specified first in the pair.

PointType

Spatial Search: An arbitrary n-dimensional point, useful for searching
sources such as blueprints or CAD drawings.

PreAnalyzedField

Provides a way to send to Solr serialized token streams, optionally
with independent stored values of a field, and have this information
stored and indexed without any additional text processing. Useful if
you want to submit field content that was already processed by some
existing external text processing pipeline (e.g. tokenized, annotated,
stemmed, inserted synonyms, etc.), while using all the rich attributes
that Lucene's TokenStream provides via token attributes.

RandomSortField

Does not contain a value. Queries that sort on this field type will return
results in random order. Use a dynamic field to use this feature.

SpatialRecursivePrefixTreeFieldType

(RPT for short) Spatial Search: Accepts latitude comma longitude
strings or other shapes in WKT format.

StrField

String (UTF-8 encoded string or Unicode).

TextField

Text, usually multiple words or tokens.

Apache Solr Reference Guide 5.1

50

TrieDateField

Date field. Represents a point in time with millisecond precision. See
the section Working with Dates. precisionStep="0" enables
efficient date sorting and minimizes index size; precisionStep="8"
(the default) enables efficient range queries.

TrieDoubleField

Double field (64-bit IEEE floating point). precisionStep="0" enabl
es efficient numeric sorting and minimizes index size; precisionSte
p="8" (the default) enables efficient range queries.

TrieField

If this field type is used, a "type" attribute must also be specified, valid
values are: integer, long, float, double, date. Using this field is
the same as using any of the Trie fields. precisionStep="0" enabl
es efficient numeric sorting and minimizes index size; precisionSte
p="8" (the default) enables efficient range queries.

TrieFloatField

Floating point field (32-bit IEEE floating point). precisionStep="0"
enables efficient numeric sorting and minimizes index size; precisio
nStep="8" (the default) enables efficient range queries.

TrieIntField

Integer field (32-bit signed integer). precisionStep="0" enables
efficient numeric sorting and minimizes index size; precisionStep=
"8" (the default) enables efficient range queries.

TrieLongField

Long field (64-bit signed integer). precisionStep="0" enables
efficient numeric sorting and minimizes index size; precisionStep=
"8" (the default) enables efficient range queries.

UUIDField

Universally Unique Identifier (UUID). Pass in a value of "NEW" and
Solr will create a new UUID. Note: configuring a UUIDField instance
with a default value of "NEW" is not advisable for most users when
using SolrCloud (and not possible if the UUID value is configured as
the unique key field) since the result will be that each replica of each
document will get a unique UUID value. Using
UUIDUpdateProcessorFactory to generate UUID values when
documents are added is recommended instead.

The MultiTermAwareComponent has been added to relevant solr.TextField entries in schema.xml (e.g.
, wildcards, regex, prefix, range, etc.) to allow automatic lowercasing for multi-term queries.
Further, you can optionally specify a multi-term analyzer in field types in your schema: ; if you don't do this, analyzer will process the fields according to their specific
attributes.

Working with Currencies and Exchange Rates
The currency FieldType provides support for monetary values to Solr/Lucene with query-time currency
conversion and exchange rates. The following features are supported:
Point queries
Range queries
Function range queries

Apache Solr Reference Guide 5.1

51

Sorting
Currency parsing by either currency code or symbol
Symmetric & asymmetric exchange rates (asymmetric exchange rates are useful if there are fees
associated with exchanging the currency)

Configuring Currencies
The currency field type is defined in schema.xml. This is the default configuration of this type:


In this example, we have defined the name and class of the field type, and defined the defaultCurrency as
"USD", for U.S. Dollars. We have also defined a currencyConfig to use a file called "currency.xml". This is a
file of exchange rates between our default currency to other currencies. There is an alternate implementation that
would allow regular downloading of currency data. See Exchange Rates below for more.
At indexing time, money fields can be indexed in a native currency. For example, if a product on an e-commerce
site is listed in Euros, indexing the price field as "1000,EUR" will index it appropriately. The price should be
separated from the currency by a comma, and the price must be encoded with a floating point value (a decimal
point).
During query processing, range and point queries are both supported.

Exchange Rates
You configure exchange rates by specifying a provider. Natively, two provider types are supported: FileExchan
geRateProvider or OpenExchangeRatesOrgProvider.

FileExchangeRateProvider
This provider requires you to provide a file of exchange rates. It is the default, meaning that to use this provider
you only need to specify the file path and name as a value for currencyConfig in the definition for this type.
There is a sample currency.xml file included with Solr, found in the same directory as the schema.xml file.
Here is a small snippet from this file:

Apache Solr Reference Guide 5.1

52









rate="0.869914" />
rate="7.800095" />
rate="8.966508" />






OpenExchangeRatesOrgProvider
You can configure Solr to download exchange rates from OpenExchangeRates.Org, with updates rates between
USD and 158 currencies hourly. These rates are symmetrical only.
In this case, you need to specify the providerClass in the definitions for the field type. Here is an example:


The refreshInterval is minutes, so the above example will download the newest rates every 60 minutes.

Working with Dates
Date Formatting
Solr's TrieDateField (and deprecated DateField) represents a point in time with millisecond precision. The
format used is a restricted form of the canonical representation of dateTime in the XML Schema specification:
YYYY-MM-DDThh:mm:ssZ
YYYY is the year.
MM is the month.
DD is the day of the month.
hh is the hour of the day as on a 24-hour clock.
mm is minutes.
ss is seconds.
Z is a literal 'Z' character indicating that this string representation of the date is in UTC
Note that no time zone can be specified; the String representations of dates is always expressed in Coordinated
Universal Time (UTC). Here is an example value:

Apache Solr Reference Guide 5.1

53

1972-05-20T17:33:18Z
You can optionally include fractional seconds if you wish, although any precision beyond milliseconds will be
ignored. Here are examples value with sub-seconds include:
1972-05-20T17:33:18.772Z
1972-05-20T17:33:18.77Z
1972-05-20T17:33:18.7Z

Date Range Formatting
Solr's DateRangeField supports the same point in time date syntax described above (with date math describe
d below) and more to express date ranges. One class of examples is truncated dates, which represent the entire
date span to the precision indicated. The other class uses the range syntax ( [ TO ]). Here are some
examples:
2000-11 – The entire month of November, 2000.
2000-11T13 – Likewise but for the 13th hour of the day (1pm-2pm).
-0009 – The year 10 BC. A 0 in the year position is 0 AD, and is also considered 1 BC.
[2000-11-01 TO 2014-12-01] – The specified date range at a day resolution.
[2014 TO 2014-12-01] – From the start of 2014 till the end of the first day of December.
[* TO 2014-12-01] – From the earliest representable time thru till the end of 2014-12-01.
Limitations: The range syntax doesn't support embedded date math. If you specify a date instance supported by
TrieDateField with date math truncating it, like NOW/DAY, you still get the first millisecond of that day, not the
entire day's range. Exclusive ranges (using { & }) work in queries but not for indexing ranges.

Date Math
Solr's date field types also supports date math expressions, which makes it easy to create times relative to fixed
moments in time, include the current time which can be represented using the special value of " NOW".

Date Math Syntax
Date math expressions consist either adding some quantity of time in a specified unit, or rounding the current
time by a specified unit. expressions can be chained and are evaluated left to right.
For example: this represents a point in time two months from now:
NOW+2MONTHS
This is one day ago:
NOW-1DAY
A slash is used to indicate rounding. This represents the beginning of the current hour:
NOW/HOUR
The following example computes (with millisecond precision) the point in time six months and three days into the
future and then rounds that time to the beginning of that day:
NOW+6MONTHS+3DAYS/DAY
Note that while date math is most commonly used relative to NOW it can be applied to any fixed moment in time

Apache Solr Reference Guide 5.1

54

as well:
1972-05-20T17:33:18.772Z+6MONTHS+3DAYS/DAY

Request Parameters That Affect Date Math
NOW
The NOW parameter is used internally by Solr to ensure consistent date math expression parsing across multiple
nodes in a distributed request. But it can be specified to instruct Solr to use an arbitrary moment in time (past or
future) to override for all situations where the the special value of " NOW" would impact date math expressions.
It must be specified as a (long valued) milliseconds since epoch
Example:
q=solr&fq=start_date:[* TO NOW]&NOW=1384387200000
TZ
By default, all date math expressions are evaluated relative to the UTC TimeZone, but the TZ parameter can be
specified to override this behaviour, by forcing all date based addition and rounding to be relative to the specified
time zone.
For example, the following request will use range faceting to facet over the current month, "per day" relative
UTC:
http://localhost:8983/solr/my_collection/select?q=*:*&facet.range=my_date_field&face
t=true&facet.range.start=NOW/MONTH&facet.range.end=NOW/MONTH%2B1MONTH&facet.range.ga
p=%2B1DAY

0
name="2013-11-02T00:00:00Z">0
name="2013-11-03T00:00:00Z">0
name="2013-11-04T00:00:00Z">0
name="2013-11-05T00:00:00Z">0
name="2013-11-06T00:00:00Z">0
name="2013-11-07T00:00:00Z">0

While in this example, the "days" will be computed relative to the specified time zone - including any applicable
Daylight Savings Time adjustments:
http://localhost:8983/solr/my_collection/select?q=*:*&facet.range=my_date_field&face
t=true&facet.range.start=NOW/MONTH&facet.range.end=NOW/MONTH%2B1MONTH&facet.range.ga
p=%2B1DAY&TZ=America/Los_Angeles

Apache Solr Reference Guide 5.1

55

0
name="2013-11-02T07:00:00Z">0
name="2013-11-03T07:00:00Z">0
name="2013-11-04T08:00:00Z">0
name="2013-11-05T08:00:00Z">0
name="2013-11-06T08:00:00Z">0
name="2013-11-07T08:00:00Z">0

More DateRangeField Details
DateRangeField is almost a drop-in replacement for places where TrieDateField is used. The only
difference is that Solr's XML or SolrJ response formats will expose the stored data as a String instead of a Date.
The underlying index data for this field will be a bit larger. Queries that align to units of time a second on up
should be faster than TrieDateField, especially if it's in UTC. But the main point of DateRangeField as it's name
suggests is to allow indexing date ranges. To do that, simply supply strings in the format shown above. It also
supports specifying 3 different relational predicates between the indexed data, and the query range: Intersect
s (default), Contains, Within. You can specify the predicate by querying using the op local-params
parameter like so:
fq={!field f=dateRange op=Contains}[2013 TO 2018]

In that example, it would find documents with indexed ranges that contain (or equals) the range 2013 thru 2018.
Multi-valued overlapping indexed ranges in a document are effectively coalesced.

Working with Enum Fields
The EnumField type allows defining a field whose values are a closed set, and the sort order is pre-determined
but is not alphabetic nor numeric. Examples of this are severity lists, or risk definitions.

Defining an EnumField in schema.xml
The EnumField type definition is quite simple, as in this example defining field types for "priorityLevel" and
"riskLevel" enumerations:



Besides the name and the class, which are common to all field types, this type also takes two additional
parameters:
enumsConfig: the name of a configuration file that contains the  list of field values and their
order that you wish to use with this field type. If a path to the file is not defined specified, the file should be
in the conf directory for the collection.
enumName: the name of the specific enumeration in the enumsConfig file to use for this type.

Apache Solr Reference Guide 5.1

56

Defining the EnumField configuration file
The file named with the enumsConfig parameter can contain multiple enumeration value lists with different
names if there are multiple uses for enumerations in your Solr schema.
In this example, there are two value lists defined. Each list is between enum opening and closing tags:



Not Available
Low
Medium
High
Urgent


Unknown
Very Low
Low
Medium
High
Critical



Changing Values
You cannot change the order, or remove, existing values in an  without reindexing.
You can however add new values to the end.

Working with External Files and Processes
The ExternalFileField Type
The ExternalFileField type makes it possible to specify the values for a field in a file outside the Solr index.
For such a field, the file contains mappings from a key field to the field value. Another way to think of this is that,
instead of specifying the field in documents as they are indexed, Solr finds values for this field in the external file.
External fields are not searchable. They can be used only for function queries or display. For more
information on function queries, see the section on Function Queries.
The ExternalFileField type is handy for cases where you want to update a particular field in many
documents more often than you want to update the rest of the documents. For example, suppose you have
implemented a document rank based on the number of views. You might want to update the rank of all the
documents daily or hourly, while the rest of the contents of the documents might be updated much less
frequently. Without ExternalFileField, you would need to update each document just to change the rank.
Using ExternalFileField is much more efficient because all document values for a particular field are stored
in an external file that can be updated as frequently as you wish.
In schema.xml, the definition of this field type might look like this:

Apache Solr Reference Guide 5.1

57



The keyField attribute defines the key that will be defined in the external file. It is usually the unique key for the
index, but it doesn't need to be as long as the keyField can be used to identify documents in the index. A defV
al defines a default value that will be used if there is no entry in the external file for a particular document.
The valType attribute specifies the actual type of values that will be found in the file. The type specified must be
either a float field type, so valid values for this attribute are pfloat, float or tfloat. This attribute can be
omitted.

Format of the External File
The file itself is located in Solr's index directory, which by default is $SOLR_HOME/data. The name of the file
should be external_fieldname or external_fieldname.*. For the example above, then, the file could be
named external_entryRankFile or external_entryRankFile.txt.
If any files using the name pattern .* (such as .txt) appear, the last (after being sorted by name) will
be used and previous versions will be deleted. This behavior supports implementations on systems
where one may not be able to overwrite a file (for example, on Windows, if the file is in use).
The file contains entries that map a key field, on the left of the equals sign, to a value, on the right. Here are a
few example entries:
doc33=1.414
doc34=3.14159
doc40=42
The keys listed in this file do not need to be unique. The file does not need to be sorted, but Solr will be able to
perform the lookup faster if it is.

Reloading an External File
It's possible to define an event listener to reload an external file when either a searcher is reloaded or when a
new searcher is started. See the section Query-Related Listeners for more information, but a sample definition in
solrconfig.xml might look like this:



Pre-Analyzing a Field Type
The PreAnalyzedField type provides a way to send to Solr serialized token streams, optionally with
independent stored values of a field, and have this information stored and indexed without any additional text
processing applied in Solr. This is useful if user wants to submit field content that was already processed by
some existing external text processing pipeline (e.g., it has been tokenized, annotated, stemmed, synonyms
inserted, etc.), while using all the rich attributes that Lucene's TokenStream provides (per-token attributes).

Apache Solr Reference Guide 5.1

58

The serialization format is pluggable using implementations of PreAnalyzedParser interface. There are two
out-of-the-box implementations:
JsonPreAnalyzedParser: as the name suggests, it parses content that uses JSON to represent field's
content. This is the default parser to use if the field type is not configured otherwise.
SimplePreAnalyzedParser: uses a simple strict plain text format, which in some situations may be easier
to create than JSON.
There is only one configuration parameter, parserImpl. The value of this parameter should be a fully qualified
class name of a class that implements PreAnalyzedParser interface. The default value of this parameter is org.
apache.solr.schema.JsonPreAnalyzedParser.

Field Properties by Use Case
Here is a summary of common use cases, and the attributes the fields or field types should have to support the
case. An entry of true or false in the table indicates that the option must be set to the given value for the use
case to function correctly. If no entry is provided, the setting of that attribute has no impact on the case.
Use Case
search within
field

indexed

stored

multiValued

omitNorms

termVectors

termPositions

docValues

true

retrieve
contents

true

use as
unique key

true

false

sort on field

true7

false

use field

true 1

true7

false

boosts 5
document
boosts affect
searches
within field

false

highlighting

true 4

faceting 5

true7

true2

true

add multiple
values,
maintaining
order
field length
affects doc
score
MoreLikeThis

true 3
true7

true

false

true 6

5

Apache Solr Reference Guide 5.1

59

Notes:
1

Recommended but not necessary.

2

Will be used if present, but not necessary.

3

(if termVectors=true)

4

A tokenizer must be defined for the field, but it doesn't need to be indexed.

5

Described in Understanding Analyzers, Tokenizers, and Filters.

6

Term vectors are not mandatory here. If not true, then a stored field is analyzed. So term vectors are
recommended, but only required if stored=false.
7

Either indexed or docValues must be true, but both are not required. DocValues can be more efficient in

many cases.

Defining Fields
Fields are defined in the fields element of schema.xml. Once you have the field types set up, defining the fields
themselves is simple.

Example
The following example defines a field named price with a type named float and a default value of 0.0; the i
ndexed and stored properties are explicitly set to true, while any other properties specified on the float fiel
d type are inherited.


Field Properties
Property

Description

name

The name of the field. Field names should consist of alphanumeric or underscore characters only
and not start with a digit. This is not currently strictly enforced, but other field names will not have
first class support from all components and back compatibility is not guaranteed. Names with both
leading and trailing underscores (e.g. _version_) are reserved. Every field must have a name.

type

The name of the fieldType for this field. This will be found in the "name" attribute on the field
Type definition. Every field must have a type.

default

A default value that will be added automatically to any document that does not have a value in this
field when it is indexed. If this property is not specified, there is no default.

Optional Field Type Override Properties
Fields can have the same options as field types. The field type options serve as defaults which can be
overridden by options defined per field. Included below is the table of field type properties from the section Field
Type Definitions and Properties:
Property
indexed

Apache Solr Reference Guide 5.1

Description
If true, the value of the field can be used in queries to retrieve
matching documents

Values
true or
false

60

stored

If true, the actual value of the field can be retrieved by queries

true or
false

docValues

If true, the value of the field will be put in a column-oriented DocValues
structure

true or
false

sortMissingFirst
sortMissingLast

Control the placement of documents when a sort field is not present.
As of Solr 3.5, these work for all numeric fields, including Trie and date
fields.

true or
false

multiValued

If true, indicates that a single document might contain multiple values
for this field type

true or
false

omitNorms

If true, omits the norms associated with this field (this disables length
normalization and index-time boosting for the field, and saves some
memory). Defaults to true for all primitive (non-analyzed) field types,
such as int, float, data, bool, and string. Only full-text fields or fields
that need an index-time boost need norms.

true or
false

omitTermFreqAndPositions

If true, omits term frequency, positions, and payloads from postings for
this field. This can be a performance boost for fields that don't require
that information. It also reduces the storage space required for the
index. Queries that rely on position that are issued on a field with this
option will silently fail to find documents. This property defaults to true
for all fields that are not text fields.

true or
false

omitPositions

Similar to omitTermFreqAndPositions but preserves term

true or
false

frequency information
termVectors
termPositions
termOffsets
termPayloads

These options instruct Solr to maintain full term vectors for each
document, optionally including position, offset and payload information
for each term occurrence in those vectors. These can be used to
accelerate highlighting and other ancillary functionality, but impose a
substantial cost in terms of index size. They are not necessary for
typical uses of Solr.

true or
false

required

Instructs Solr to reject any attempts to add a document which does not
have a value for this field. This property defaults to false.

true or
false

Related Topics
SchemaXML-Fields
Field Options by Use Case

Copying Fields
You might want to interpret some document fields in more than one way. Solr has a mechanism for making
copies of fields so that you can apply several distinct field types to a single piece of incoming information.
The name of the field you want to copy is the source, and the name of the copy is the destination. In schema.xm
l, it's very simple to make copies of fields:

Apache Solr Reference Guide 5.1

61



If the text destination field has data of its own in the input documents, the contents of the cat field will be
added as additional values – just as if all of the values had originally been specified by the client. Remember to
configure your fields as multivalued="true" if they will ultimately get multiple values (either from a
multivalued source, or multiple copyField directives, etc...)
The maxChars parameter, an int parameter, establishes an upper limit for the number of characters to be
copied from the source value when constructing the value added to the destination field. This limit is useful for
situations in which you want to copy some data from the source field, but also control the size of index files.
Both the source and the destination of copyField can contain either leading or trailing asterisks, which will
match anything. For example, the following line will copy the contents of all incoming fields that match the
wildcard pattern *_t to the text field.:


The copyField command can use a wildcard (*) character in the dest parameter only if the source p
arameter contains one as well. copyField uses the matching glob from the source field for the dest fie
ld name into which the source content is copied.

Related Topics
SchemaXML-Copy Fields

Dynamic Fields
Dynamic fields allow Solr to index fields that you did not explicitly define in your schema. This is useful if you
discover you have forgotten to define one or more fields. Dynamic fields can make your application less brittle by
providing some flexibility in the documents you can add to Solr.
A dynamic field is just like a regular field except it has a name with a wildcard in it. When you are indexing
documents, a field that does not match any explicitly defined fields can be matched with a dynamic field.
For example, suppose your schema includes a dynamic field with a name of *_i. If you attempt to index a
document with a cost_i field, but no explicit cost_i field is defined in the schema, then the cost_i field will
have the field type and analysis defined for *_i.
Like regular fields, dynamic fields have a name, a field type, and options.


It is recommended that you include basic dynamic field mappings (like that shown above) in your schema.xml.
The mappings can be very useful.

Related Topics
SchemaXML-Dynamic Fields

Apache Solr Reference Guide 5.1

62

Other Schema Elements
This section describes several other important elements of schema.xml.

Unique Key
The uniqueKey element specifies which field is a unique identifier for documents. Although uniqueKey is not
required, it is nearly always warranted by your application design. For example, uniqueKey should be used if
you will ever update a document in the index.
You can define the unique key field by naming it:
id

Schema defaults and copyFields cannot be used to populate the uniqueKey field. You also can't use UUIDU
pdateProcessorFactory to have uniqueKey values generated automatically.
Further, the operation will fail if the uniqueKey field is used, but is multivalued (or inherits the multivalueness
from the fieldtype). However, uniqueKey will continue to work, as long as the field is properly used.

Default Search Field
If you are using the Lucene query parser, queries that don't specify a field name will use the defaultSearchField.
The DisMax and Extended DisMax query parsers will also fallback to this if qf is not specified.
Use of the defaultSearchField element is deprecated in Solr versions 3.6 and higher. Instead, you
should use the df request parameter. At some point, the defaultSearchField element may be
removed.
For more information about query parsers, see the section on Query Syntax and Parsing.

Query Parser Default Operator
In queries with multiple terms, Solr can either return results where all conditions are met or where one or more
conditions are met. The operator controls this behavior. An operator of AND means that all conditions must be
fulfilled, while an operator of OR means that one or more conditions must be true.
In schema.xml, the solrQueryParser element controls what operator is used if an operator is not specified in
the query. The default operator setting only applies to the Lucene query parser, not the DisMax or Extended
DisMax query parsers, which internally hard-code their operators to OR.
The query parser default operator parameter has been deprecated in Solr versions 3.6 and higher. You
are instead encouraged to specify the query parser q.op parameter in your request handler.

Similarity
Similarity is a Lucene class used to score a document in searching.
A global  declaration can be used to specify a custom similarity implementation that you want
Solr to use when dealing with your index. A similarity can be specified either by referring directly to the name of a

Apache Solr Reference Guide 5.1

63

class with a no-argument constructor:


or by referencing a SimilarityFactory implementation, which may take optional initialization parameters:

P
L
H2
7


A special SchemaSimilarityFactory is available, which allows individual field types to be configured with a
similarity to override the default behavior:




SPL
DF
H2



If a field type does not have an explicitly configured similarity factory, then a DefaultSimilarity instance is
used.
This examples above show DFRSimilarityFactory (divergence from random) and IBSimilarityFactory
(using the Information-Based model), but there are several similarity implementations that can be used such as
the SweetSpotSimilarityFactory, BM25SimilarityFactory, etc.... For details, see the Solr Javadocs
for the similarity factories.

Related Topics
SchemaXML-Miscellaneous Settings
UniqueKey

Schema API
The Schema API provides read and write access to the Solr schema for each collection (or core, when using
standalone Solr). Read access to all schema elements is supported. Fields, dynamic fields, field types and
copyField rules may be added, removed or replaced. Future Solr releases will extend write access to allow more
schema elements to be modified.
Re-index after schema modifications!
If you modify your schema, you will likely need to re-index all documents. If you do not, you may lose
access to documents, or not be able to interpret them properly, e.g. after replacing a field type.
To enable schema modification with this API, the schema will need to be managed and mutable. See the section

Apache Solr Reference Guide 5.1

64

Managed Schema Definition in SolrConfig for more information.
The API allows two output modes for all calls: JSON or XML. When requesting the complete schema, there is
another output mode which is XML modeled after the schema.xml file itself.
When modifying the schema with the API, a core reload will automatically occur in order for the changes to be
available immediately.
The base address for the API is http://:/solr/. If for example you
run Solr's "cloud" example (via the bin/solr command shown below), which creates a "gettingstarted"
collection, then the base URL (as in all the sample URLs in this section) would be: http://localhost:8983/
solr/gettingstarted .
bin/solr -e cloud -noprompt

API Entry Points
Modify the Schema
Add a New Field
Delete a Field
Replace a Field
Add a Dynamic Field Rule
Delete a Dynamic Field Rule
Replace a Dynamic Field Rule
Add a New Field Type
Delete a Field Type
Replace a Field Type
Add a New Copy Field Rule
Delete a Copy Field Rule
Multiple Commands in a Single POST
Schema Changes among Replicas
Retrieve Schema Information
Retrieve the Entire Schema
List Fields
List Dynamic Fields
List Field Types
List Copy Fields
Show Schema Name
Show the Schema Version
List UniqueKey
Show Global Similarity
Get the Default Query Operator
Manage Resource Data

API Entry Points
/schema: retrieve the schema, or modify the schema to add, remove, or replace fields, dynamic fields, copy
fields, or field types
/schema/fields: retrieve information about all defined fields or a specific named field

Apache Solr Reference Guide 5.1

65

/schema/dynamicfields: retrieve information about all dynamic field rules or a specific named dynamic rule
/schema/fieldtypes: retrieve information about all field types or a specific field type
/schema/copyfields: retrieve information about copy fields
/schema/name: retrieve the schema name
/schema/version: retrieve the schema version
/schema/uniquekey: retrieve the defined uniqueKey
/schema/similarity: retrieve the global similarity definition
/schema/solrqueryparser/defaultoperator: retrieve the default operator

Modify the Schema
POST /collection/schema
To add, remove or replace fields, dynamic field rules, copy field rules, or new field types, you can send a POST
request to the /collection/schema/ endpoint with a sequence of commands to perform the requested
actions. The following commands are supported:
add-field: add a new field with parameters you provide.
delete-field: delete a field.
replace-field: replace an existing field with one that is differently configured.
add-dynamic-field: add a new dynamic field rule with parameters you provide.
delete-dynamic-field: delete a dynamic field rule.
replace-dynamic-field: replace an existing dynamic field rule with one that is differently configured.
add-field-type: add a new field type with parameters you provide.
delete-field-type: delete a field type.
replace-field-type: replace an existing field type with one that is differently configured.
add-copy-field: add a new copy field rule.
delete-copy-field: delete a copy field rule.
These commands can be issued in separate POST requests or in the same POST request. Commands are
executed in the order in which they are specified.
In each case, the response will include the status and the time to process the request, but will not include the
entire schema.
When modifying the schema with the REST API, a core reload will automatically occur in order for the changes
to be available immediately.

Add a New Field
The add-field command adds a new field definition to your schema. If a field with the same name exists an
error is thrown.
All of the properties available when defining a field with manual schema.xml edits can be passed via the API.
These request attributes are described in detail in the section Defining Fields.
For example, to define a new stored field named "sell-by", of type "tdate", you would POST the following request:

Apache Solr Reference Guide 5.1

66

curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"sell-by",
"type":"tdate",
"stored":true }
}' http://localhost:8983/solr/gettingstarted/schema

Delete a Field
The delete-field command removes a field definition from your schema. If the field does not exist in the
schema, or if the field is the source or destination of a copy field rule, an error is thrown.
For example, to delete a field named "sell-by", you would POST the following request:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"delete-field" : { "name":"sell-by" }
}' http://localhost:8983/solr/gettingstarted/schema

Replace a Field
The replace-field command replaces a field's definition. Note that you must supply the full definition for a
field - this command will not partially modify a field's definition. If the field does not exist in the schema an error
is thrown.
All of the properties available when defining a field with manual schema.xml edits can be passed via the API.
These request attributes are described in detail in the section Defining Fields.
For example, to replace the definition of an existing field "sell-by", to make it be of type "date" and to not be
stored, you would POST the following request:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"replace-field":{
"name":"sell-by",
"type":"date",
"stored":false }
}' http://localhost:8983/solr/gettingstarted/schema

Add a Dynamic Field Rule
The add-dynamic-field command adds a new dynamic field rule to your schema.
All of the properties available when editing schema.xml can be passed with the POST request. The section Dyn
amic Fields has details on all of the attributes that can be defined for a dynamic field rule.
For example, to create a new dynamic field rule where all incoming fields ending with "_s" would be stored and
have field type "string", you can POST a request like this:

Apache Solr Reference Guide 5.1

67

curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-dynamic-field":{
"name":"*_s",
"type":"string",
"stored":true }
}' http://localhost:8983/solr/gettingstarted/schema

Delete a Dynamic Field Rule
The delete-dynamic-field command deletes a dynamic field rule from your schema. If the dynamic field
rule does not exist in the schema, or if the schema contains a copy field rule with a target or destination that
matches only this dynamic field rule, an error is thrown.

For example, to delete a dynamic field rule matching "*_s", you can POST a request like this:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"delete-dynamic-field":{ "name":"*_s" }
}' http://localhost:8983/solr/gettingstarted/schema

Replace a Dynamic Field Rule
The replace-dynamic-field command replaces a dynamic field rule in your schema. Note that you must
supply the full definition for a dynamic field rule - this command will not partially modify a dynamic field rule's
definition. If the dynamic field rule does not exist in the schema an error is thrown.
All of the properties available when editing schema.xml can be passed with the POST request. The section Dyn
amic Fields has details on all of the attributes that can be defined for a dynamic field rule.
For example, to replace the definition of the "*_s" dynamic field rule with one where the field type is
"text_general" and it's not stored, you can POST a request like this:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"replace-dynamic-field":{
"name":"*_s",
"type":"text_general",
"stored":false }
}' http://localhost:8983/solr/gettingstarted/schema

Add a New Field Type
The add-field-type command adds a new field type to your schema.
All of the field type properties available when editing schema.xml by hand are available for use in a POST
request. The structure of the command is a json mapping of the standard field type definition, including the
name, class, index and query analyzer definitions, etc. Details of all of the available options are described in the
section Solr Field Types.
For example, to create a new field type named "myNewTxtField", you can POST a request as follows:

Apache Solr Reference Guide 5.1

68

curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field-type" : {
"name":"myNewTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer" : {
"charFilters":[{
"class":"solr.PatternReplaceCharFilterFactory",
"replacement":"$1$1",
"pattern":"([a-zA-Z])\\\\1+" }],
"tokenizer":{
"class":"solr.WhitespaceTokenizerFactory" },
"filters":[{
"class":"solr.WordDelimiterFilterFactory",
"preserveOriginal":"0" }]}}
}' http://localhost:8983/solr/gettingstarted/schema

Note in this example that we have only defined a single analyzer section that will apply to index analysis and
query analysis. If we wanted to define separate analysis, we would replace the analyzer section in the above
example with separate sections for indexAnalyzer and queryAnalyzer. As in this example:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field-type":{
"name":"myNewTextField",
"class":"solr.TextField",
"indexAnalyzer":{
"tokenizer":{
"class":"solr.PathHierarchyTokenizerFactory",
"delimiter":"/" }},
"queryAnalyzer":{
"tokenizer":{
"class":"solr.KeywordTokenizerFactory" }}}
}' http://localhost:8983/solr/gettingstarted/schema

Delete a Field Type
The delete-field-type command removes a field type from your schema. If the field type does not exist in
the schema, or if any field or dynamic field rule in the schema uses the field type, an error is thrown.

For example, to delete the field type named "myNewTxtField", you can make a POST request as
follows:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"delete-field-type":{ "name":"myNewTxtField" }
}' http://localhost:8983/solr/gettingstarted/schema

Replace a Field Type
The replace-field-type command replaces a field type in your schema. Note that you must supply the full
definition for a field type - this command will not partially modify a field type's definition. If the field type does not
exist in the schema an error is thrown.
All of the field type properties available when editing schema.xml by hand are available for use in a POST

Apache Solr Reference Guide 5.1

69

request. The structure of the command is a json mapping of the standard field type definition, including the
name, class, index and query analyzer definitions, etc. Details of all of the available options are described in the
section Solr Field Types.
For example, to replace the definition of a field type named "myNewTxtField", you can make a POST request as
follows:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"replace-field-type":{
"name":"myNewTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" }}}
}' http://localhost:8983/solr/gettingstarted/schema

Add a New Copy Field Rule
The add-copy-field command adds a new copy field rule to your schema.
The attributes supported by the command are the same as when creating copy field rules by manually editing
the schema.xml, as below:
Name

Required

Description

source

Yes

The source field.

dest

Yes

A field or an array of fields to which the source field will be copied.

maxChars

No

The upper limit for the number of characters to be copied. The section Copying Fields
has more details.

For example, to define a rule to copy the field "shelf" to the "location" and "catchall" fields, you would POST the
following request:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-copy-field":{
"source":"shelf",
"dest":[ "location", "catchall" ]}
}' http://localhost:8983/solr/gettingstarted/schema

Delete a Copy Field Rule
The delete-copy-field command deletes a copy field rule from your schema. If the copy field rule does not
exist in the schema an error is thrown.
The source and dest attributes are required by this command.
For example, to delete a rule to copy the field "shelf" to the "location" field, you would POST the following
request:

Apache Solr Reference Guide 5.1

70

curl -X POST -H 'Content-type:application/json' --data-binary '{
"delete-copy-field":{ "source":"shelf", "dest":"location" }
}' http://localhost:8983/solr/gettingstarted/schema

Multiple Commands in a Single POST
It is possible to perform one or more add requests in a single command. The API is transactional and all
commands in a single call either succeed or fail together.
The commands are executed in the order in which they are specified. This means that if you want to create a
new field type and in the same request use the field type on a new field, the section of the request that creates
the field type must come before the section that creates the new field. Similarly, since a field must exist for it to
be used in a copy field rule, a request to add a field must come before a request for the field to be used as either
the source or the destination for a copy field rule.
The syntax for making multiple requests supports several approaches. First, the commands can simply be made
serially, as in this request to create a new field type and then a field that uses that type:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field-type":{
"name":"myNewTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer":{
"charFilters":[{
"class":"solr.PatternReplaceCharFilterFactory",
"replacement":"$1$1",
"pattern":"([a-zA-Z])\\\\1+" }],
"tokenizer":{
"class":"solr.WhitespaceTokenizerFactory" },
"filters":[{
"class":"solr.WordDelimiterFilterFactory",
"preserveOriginal":"0" }]}},
"add-field" : {
"name":"sell-by",
"type":"myNewTxtField",
"stored":true }
}' http://localhost:8983/solr/gettingstarted/schema

Or, the same command can be repeated, as in this example:

Apache Solr Reference Guide 5.1

71

curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"shelf",
"type":"myNewTxtField",
"stored":true },
"add-field":{
"name":"location",
"type":"myNewTxtField",
"stored":true },
"add-copy-field":{
"source":"shelf",
"dest":[ "location", "catchall" ]}
}' http://localhost:8983/solr/gettingstarted/schema

Finally, repeated commands can be sent as an array:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":[
{ "name":"shelf",
"type":"myNewTxtField",
"stored":true },
{ "name":"location",
"type":"myNewTxtField",
"stored":true }]
}' http://localhost:8983/solr/gettingstarted/schema

Schema Changes among Replicas
When running in SolrCloud mode, changes made to the schema on one node will propagate to all replicas in the
collection. You can pass the updateTimeoutSecs parameter with your request to set the number of seconds to
wait until all replicas confirm they applied the schema updates. This helps your client application be more robust
in that you can be sure that all replicas have a given schema change within a defined amount of time. If
agreement is not reached by all replicas in the specified time, then the request fails and the error message will
include information about which replicas had trouble. In most cases, the only option is to re-try the change after
waiting a brief amount of time. If the problem persists, then you'll likely need to investigate the server logs on the
replicas that had trouble applying the changes. If you do not supply an updateTimeoutSecs parameter, the
default behavior is for the receiving node to return immediately after persisting the updates to ZooKeeper. All
other replicas will apply the updates asynchronously. Consequently, without supplying a timeout, your client
application cannot be sure that all replicas have applied the changes.

Retrieve Schema Information
The following endpoints allow you to read how your schema has been defined. You can GET the entire schema,
or only portions of it as needed.
To modify the schema, see the previous section Modify the Schema.

Retrieve the Entire Schema
GET /collection/schema

INPUT

Apache Solr Reference Guide 5.1

72

Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters should be added to the API request after '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json, xml or schem
a.xml. If not specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include all fields, field types, dynamic rules and copy field rules, in the format requested (JSON or
XML). The schema name and version are also included.

EXAMPLES
Get the entire schema in JSON.
curl http://localhost:8983/solr/gettingstarted/schema?wt=json

Apache Solr Reference Guide 5.1

73

{
"responseHeader":{
"status":0,
"QTime":5},
"schema":{
"name":"example",
"version":1.5,
"uniqueKey":"id",
"fieldTypes":[{
"name":"alphaOnlySort",
"class":"solr.TextField",
"sortMissingLast":true,
"omitNorms":true,
"analyzer":{
"tokenizer":{
"class":"solr.KeywordTokenizerFactory"},
"filters":[{
"class":"solr.LowerCaseFilterFactory"},
{
"class":"solr.TrimFilterFactory"},
{
"class":"solr.PatternReplaceFilterFactory",
"replace":"all",
"replacement":"",
"pattern":"([^a-z])"}]}},
...
"fields":[{
"name":"_version_",
"type":"long",
"indexed":true,
"stored":true},
{
"name":"author",
"type":"text_general",
"indexed":true,
"stored":true},
{
"name":"cat",
"type":"string",
"multiValued":true,
"indexed":true,
"stored":true},
...
"copyFields":[{
"source":"author",
"dest":"text"},
{
"source":"cat",
"dest":"text"},
{
"source":"content",
"dest":"text"},
...
{
"source":"author",
"dest":"author_s"}]}}

Apache Solr Reference Guide 5.1

74

Get the entire schema in XML.
curl http://localhost:8983/solr/gettingstarted/schema?wt=xml



0
5


example
1.5
id


alphaOnlySort
solr.TextField
true
true


solr.KeywordTokenizerFactory



solr.LowerCaseFilterFactory


solr.TrimFilterFactory


solr.PatternReplaceFilterFactory
all

([^a-z])




...

author
author_s





Get the entire schema in "schema.xml" format.
curl http://localhost:8983/solr/gettingstarted/schema?wt=schema.xml

Apache Solr Reference Guide 5.1

75


id









...





List Fields
GET /collection/schema/fields
GET /collection/schema/fields/fieldname

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

fieldname

The specific fieldname (if limiting request to a single field).

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include each field and any defined configuration for each field. The defined configuration can vary
for each field, but will minimally include the field name, the type, if it is indexed and if it is stored. If multiVa
lued is defined as either true or false (most likely true), that will also be shown. See the section Defining Fields f
or more information about each parameter.

EXAMPLES

Apache Solr Reference Guide 5.1

76

Get a list of all fields.
curl http://localhost:8983/solr/gettingstarted/schema/fields?wt=json

The sample output below has been truncated to only show a few fields.
{
"fields": [
{
"indexed": true,
"name": "_version_",
"stored": true,
"type": "long"
},
{
"indexed": true,
"name": "author",
"stored": true,
"type": "text_general"
},
{
"indexed": true,
"multiValued": true,
"name": "cat",
"stored": true,
"type": "string"
},
...
],
"responseHeader": {
"QTime": 1,
"status": 0
}
}

List Dynamic Fields
GET /collection/schema/dynamicfields
GET /collection/schema/dynamicfields/name

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

name

The name of the dynamic field rule (if limiting request to a single dynamic field rule).

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Apache Solr Reference Guide 5.1

Description

77

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include each dynamic field rule and the defined configuration for each rule. The defined
configuration can vary for each rule, but will minimally include the dynamic field name, the type, if it is indexed
and if it is stored. See the section Dynamic Fields for more information about each parameter.

EXAMPLES
Get a list of all dynamic field declarations:
curl http://localhost:8983/solr/gettingstarted/schema/dynamicfields?wt=json

The sample output below has been truncated.

Apache Solr Reference Guide 5.1

78

{
"dynamicFields": [
{
"indexed": true,
"name": "*_coordinate",
"stored": false,
"type": "tdouble"
},
{
"multiValued": true,
"name": "ignored_*",
"type": "ignored"
},
{
"name": "random_*",
"type": "random"
},
{
"indexed": true,
"multiValued": true,
"name": "attr_*",
"stored": true,
"type": "text_general"
},
{
"indexed": true,
"multiValued": true,
"name": "*_txt",
"stored": true,
"type": "text_general"
}
...
],
"responseHeader": {
"QTime": 1,
"status": 0
}
}

List Field Types
GET /collection/schema/fieldtypes
GET /collection/schema/fieldtypes/name

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

name

The name of the field type (if limiting request to a single field type).

Query Parameters

Apache Solr Reference Guide 5.1

79

The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include each field type and any defined configuration for the type. The defined configuration can
vary for each type, but will minimally include the field type name and the class. If query or index analyzers,
tokenizers, or filters are defined, those will also be shown with other defined parameters. See the section Solr
Field Types for more information about how to configure various types of fields.

EXAMPLES
Get a list of all field types.
curl http://localhost:8983/solr/gettingstarted/schema/fieldtypes?wt=json

The sample output below has been truncated to show a few different field types from different parts of the list.

Apache Solr Reference Guide 5.1

80

{
"fieldTypes": [
{
"analyzer": {
"class": "solr.TokenizerChain",
"filters": [
{
"class": "solr.LowerCaseFilterFactory"
},
{
"class": "solr.TrimFilterFactory"
},
{
"class": "solr.PatternReplaceFilterFactory",
"pattern": "([^a-z])",
"replace": "all",
"replacement": ""
}
],
"tokenizer": {
"class": "solr.KeywordTokenizerFactory"
}
},
"class": "solr.TextField",
"dynamicFields": [],
"fields": [],
"name": "alphaOnlySort",
"omitNorms": true,
"sortMissingLast": true
},
...
{
"class": "solr.TrieFloatField",
"dynamicFields": [
"*_fs",
"*_f"
],
"fields": [
"price",
"weight"
],
"name": "float",
"positionIncrementGap": "0",
"precisionStep": "0"
},
...
}

List Copy Fields
GET /collection/schema/copyfields

INPUT
Path Parameters

Apache Solr Reference Guide 5.1

81

Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include the source and destination of each copy field rule defined in schema.xml. For more
information about copying fields, see the section Copying Fields.

EXAMPLES
Get a list of all copyfields.
curl http://localhost:8983/solr/gettingstarted/schema/copyfields?wt=json

The sample output below has been truncated to the first few copy definitions.
{
"copyFields": [
{
"dest": "text",
"source": "author"
},
{
"dest": "text",
"source": "cat"
},
{
"dest": "text",
"source": "content"
},
{
"dest": "text",
"source": "content_type"
},
...
],
"responseHeader": {
"QTime": 3,
"status": 0
}
}

Show Schema Name
Apache Solr Reference Guide 5.1

82

GET /collection/schema/name

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will be simply the name given to the schema.

EXAMPLES
Get the schema name.
curl http://localhost:8983/solr/gettingstarted/schema/name?wt=json

{
"responseHeader":{
"status":0,
"QTime":1},
"name":"example"}

Show the Schema Version
GET /collection/schema/version

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Apache Solr Reference Guide 5.1

Description

83

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will simply be the schema version in use.

EXAMPLES
Get the schema version
curl http://localhost:8983/solr/gettingstarted/schema/version?wt=json

{
"responseHeader":{
"status":0,
"QTime":2},
"version":1.5}

List UniqueKey
GET /collection/schema/uniquekey

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include simply the field name that is defined as the uniqueKey for the index.

EXAMPLES
List the uniqueKey.

Apache Solr Reference Guide 5.1

84

curl http://localhost:8983/solr/gettingstarted/schema/uniquekey?wt=json

{
"responseHeader":{
"status":0,
"QTime":2},
"uniqueKey":"id"}

Show Global Similarity
GET /collection/schema/similarity

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include the class name of the global similarity defined (if any).

EXAMPLES
Get the similarity implementation.
curl http://localhost:8983/solr/gettingstarted/schema/similarity?wt=json

{
"responseHeader":{
"status":0,
"QTime":1},
"similarity":{
"class":"org.apache.solr.search.similarities.DefaultSimilarityFactory"}}

Get the Default Query Operator

Apache Solr Reference Guide 5.1

85

GET /collection/schema/solrqueryparser/defaultoperator

INPUT
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

wt

string

No

json

Description
Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

OUTPUT
Output Content
The output will include simply the default operator if none is defined by the user.

EXAMPLES
Get the default operator.
curl
http://localhost:8983/solr/gettingstarted/schema/solrqueryparser/defaultoperator?wt=
json

{
"responseHeader":{
"status":0,
"QTime":2},
"defaultOperator":"OR"}

Manage Resource Data
The Managed Resources REST API provides a mechanism for any Solr plugin to expose resources that should
support CRUD (Create, Read, Update, Delete) operations. Depending on what Field Types and Analyzers are
configured in your Schema, additional /schema/ REST API paths may exist. See the Managed Resources secti
on for more information and examples.

Putting the Pieces Together
At the highest level, schema.xml is structured as follows. This example is not real XML, but it gives you an idea
of the structure of the file.

Apache Solr Reference Guide 5.1

86








Obviously, most of the excitement is in types and fields, where the field types and the actual field definitions
live. These are supplemented by copyFields. The uniqueKey must always be defined. In older Solr versions
you would find defaultSearchField and solrQueryParser tags as well, but although these still work they
are deprecated and discouraged, see Other Schema Elements.
Types and fields are optional tags
Note that the types and fields sections are optional, meaning you are free to mix field, dynamiFi
eld, copyField and fieldType definitions on the top level. This allows for a more logical grouping of
related tags in your schema.

Choosing Appropriate Numeric Types
For general numeric needs, use TrieIntField, TrieLongField, TrieFloatField, and TrieDoubleFiel
d with precisionStep="0".
If you expect users to make frequent range queries on numeric types, use the default precisionStep (by not
specifying it) or specify it as precisionStep="8" (which is the default). This offers faster speed for range
queries at the expense of increasing index size.

Working With Text
Handling text properly will make your users happy by providing them with the best possible results for text
searches.
One technique is using a text field as a catch-all for keyword searching. Most users are not sophisticated about
their searches and the most common search is likely to be a simple keyword search. You can use copyField to
take a variety of fields and funnel them all into a single text field for keyword searches. In the schema.xml file for
the "techproducts" example included with Solr, copyField declarations are used to dump the contents of ca
t, name, manu, features, and includes into a single field, text. In addition, it could be a good idea to copy
ID into text in case users wanted to search for a particular product by passing its product number to a keyword
search.
Another technique is using copyField to use the same field in different ways. Suppose you have a field that is
a list of authors, like this:
Schildt, Herbert; Wolpert, Lewis; Davies, P.
For searching by author, you could tokenize the field, convert to lower case, and strip out punctuation:
schildt / herbert / wolpert / lewis / davies / p
For sorting, just use an untokenized field, converted to lower case, with punctuation stripped:

Apache Solr Reference Guide 5.1

87

schildt herbert wolpert lewis davies p
Finally, for faceting, use the primary author only via a StringField:
Schildt, Herbert

Related Topics
SchemaXML

DocValues
DocValues are a way of recording field values internally that is more efficient for some purposes, such as sorting
and faceting, then traditional indexing.

Why DocValues?
The standard way that Solr builds the index is with an inverted index. This style builds a list of terms found in all
the documents in the index and next to each term is a list of documents that the term appears in (as well as how
many times the term appears in that document). This makes search very fast - since users search by terms,
having a ready list of term-to-document values makes the query process faster.
For other features that we now commonly associate with search, such as sorting, faceting, and highlighting, this
approach is not very efficient. The faceting engine, for example, must look up each term that appears in each
document that will make up the result set and pull the document IDs in order to build the facet list. In Solr, this is
maintained in memory, and can be slow to load (depending on the number of documents, terms, etc.).
In Lucene 4.0, a new approach was introduced. DocValue fields are now column-oriented fields with a
document-to-value mapping built at index time. This approach promises to relieve some of the memory
requirements of the fieldCache and make lookups for faceting, sorting, and grouping much faster.

How to Use DocValues
To use docValues, you only need to enable it for a field that you will use it with. As with all schema design, you
need to define a field type and then define fields of that type with docValues enabled. All of these actions are
done in schema.xml.
Enabling a field for docValues only requires adding docValues="true" to the field (or field type) definition, as
in this example from the schema.xml of Solr's sample_techproducts_configs config set:


If you have already indexed data into your Solr index, you will need to completely re-index your content
after changing your field definitions in schema.xml in order to successfully use docValues.
DocValues are only available for specific field types. The types chosen determine the underlying Lucene
docValue type that will be used. The available Solr field types are:
StrField and UUIDField.
If the field is single-valued (i.e., multi-valued is false), Lucene will use the SORTED type.
If the field is multi-valued, Lucene will use the SORTED_SET type.

Apache Solr Reference Guide 5.1

88

Any Trie* numeric fields and EnumField.
If the field is single-valued (i.e., multi-valued is false), Lucene will use the NUMERIC type.
If the field is multi-valued, Lucene will use the SORTED_SET type.
These Lucene types are related to how the values are sorted and stored.
There is an additional configuration option available, which is to modify the docValuesFormat used by the field
type. The default implementation employs a mixture of loading some things into memory and keeping some on
disk. In some cases, however, you may choose to specify an alternative DocValuesFormat implementation. For
example, you could choose to keep everything in memory by specifying docValuesFormat="Memory" on a
field type:


Please note that the docValuesFormat option may change in future releases.
Lucene index back-compatibility is only supported for the default codec. If you choose to customize the d
ocValuesFormat in your schema.xml, upgrading to a future version of Solr may require you to either
switch back to the default codec and optimize your index to rewrite it into the default codec before
upgrading, or re-build your entire index from scratch after upgrading.

Schemaless Mode
Schemaless Mode is a set of Solr features that, when used together, allow users to rapidly construct an effective
schema by simply indexing sample data, without having to manually edit the schema. These Solr features, all
specified in solrconfig.xml, are:
1. Managed schema: Schema modifications are made through Solr APIs rather than manual edits - see Man
aged Schema Definition in SolrConfig.
2. Field value class guessing: Previously unseen fields are run through a cascading set of value-based
parsers, which guess the Java class of field values - parsers for Boolean, Integer, Long, Float, Double,
and Date are currently available.
3. Automatic schema field addition, based on field value class(es): Previously unseen fields are added to the
schema, based on field value Java classes, which are mapped to schema field types - see Solr Field
Types.

Using the Schemaless Example
The three features of schemaless mode are pre-configured in the data_driven_schema_configs config set i
n the Solr distribution. To start an example instance of Solr using these configs, run the following command:
bin/solr start -e schemaless

This will launch a Solr server, and automatically create a collection (named " gettingstarted") that contains
only three fields in the initial schema: id, _version_, and _text_.
You can use the /schema/fields Schema API to confirm this: curl http://localhost:8983/solr/get
tingstarted/schema/fields will output:

Apache Solr Reference Guide 5.1

89

{
"responseHeader":{
"status":0,
"QTime":1},
"fields":[{
"name":"_text_",
"type":"text_general",
"multiValued":true,
"indexed":true,
"stored":false},
{
"name":"_version_",
"type":"long",
"indexed":true,
"stored":true},
{
"name":"id",
"type":"string",
"multiValued":false,
"indexed":true,
"required":true,
"stored":true,
"uniqueKey":true}]}

Because the data_driven_schema_configs config set includes a copyField directive that causes
all content to be indexed in a predefined "catch-all" _text_ field, to enable single-field search that
includes all fields' content, the index will be larger than it would be without the copyField. When you
nail down your schema, consider removing the _text_ field and the corresponding copyField directiv
e if you don't need it.

Configuring Schemaless Mode
As described above, there are three configuration elements that need to be in place to use Solr in schemaless
mode. In the data_driven_schema_configs config set included with Solr these are already configured. If,
however, you would like to implement schemaless on your own, you should make the following changes.

Enable Managed Schema
As described in the section Managed Schema Definition in SolrConfig, changing the schemaFactory will allow
the schema to be modified by the Schema API. Your solrconfig.xml should have a section like the one
below (and the ClassicIndexSchemaFactory should be commented out or removed).

true
managed-schema


Define an UpdateRequestProcessorChain
The UpdateRequestProcessorChain allows Solr to guess field types, and you can define the default field type
classes to use. To start, you should define it as follows (see the javadoc links below for update processor factory

Apache Solr Reference Guide 5.1

90

documentation):

Apache Solr Reference Guide 5.1

91








[^\w-\.]
_






yyyy-MM-dd'T'HH:mm:ss.SSSZ
yyyy-MM-dd'T'HH:mm:ss,SSSZ
yyyy-MM-dd'T'HH:mm:ss.SSS
yyyy-MM-dd'T'HH:mm:ss,SSS
yyyy-MM-dd'T'HH:mm:ssZ
yyyy-MM-dd'T'HH:mm:ss
yyyy-MM-dd'T'HH:mmZ
yyyy-MM-dd'T'HH:mm
yyyy-MM-dd HH:mm:ss.SSSZ
yyyy-MM-dd HH:mm:ss,SSSZ
yyyy-MM-dd HH:mm:ss.SSS
yyyy-MM-dd HH:mm:ss,SSS
yyyy-MM-dd HH:mm:ssZ
yyyy-MM-dd HH:mm:ss
yyyy-MM-dd HH:mmZ
yyyy-MM-dd HH:mm
yyyy-MM-dd



strings

java.lang.Boolean
booleans


java.util.Date
tdates


java.lang.Long
java.lang.Integer
tlongs


java.lang.Number
tdoubles





Apache Solr Reference Guide 5.1

92

Javadocs for update processor factories mentioned above:
UUIDUpdateProcessorFactory
RemoveBlankFieldUpdateProcessorFactory
FieldNameMutatingUpdateProcessorFactory
ParseBooleanFieldUpdateProcessorFactory
ParseLongFieldUpdateProcessorFactory
ParseDoubleFieldUpdateProcessorFactory
ParseDateFieldUpdateProcessorFactory
AddSchemaFieldsUpdateProcessorFactory

Make the UpdateRequestProcessorChain the Default for the UpdateRequestHandler
Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to
use it when working with index updates (i.e., adding, removing, replacing documents). Here is an example using
InitParams to set the defaults on all /update request handlers:


add-unknown-fields-to-the-schema



After each of these changes have been made, Solr should be restarted (or, you can reload the cores to
load the new solrconfig.xml definitions).

Examples of Indexed Documents
Once the schemaless mode has been enabled (whether you configured it manually or are using data_driven_
schema_configs ), documents that include fields that are not defined in your schema should be added to the
index, and the new fields added to the schema.
For example, adding a CSV document will cause its fields that are not in the schema to be added, with
fieldTypes based on values:
curl "http://localhost:8983/solr/gettingstarted/update?commit=true" -H
"Content-type:application/csv" -d '
id,Artist,Album,Released,Rating,FromDistributor,Sold
44C,Old Shews,Mead for Walking,1988-08-13,0.01,14,0'

Output indicating success:

0106


The fields now in the schema (output from curl http://localhost:8983/solr/gettingstarted/sche
ma/fields ):

Apache Solr Reference Guide 5.1

93

{
"responseHeader":{
"status":0,
"QTime":1},
"fields":[{
"name":"Album",
"type":"strings"},
//
{
"name":"Artist",
"type":"strings"},
//
{
"name":"FromDistributor",
"type":"tlongs"},
//
{
"name":"Rating",
"type":"tdoubles"},
//
{
"name":"Released",
"type":"tdates"},
//
{
"name":"Sold",
"type":"tlongs"},
//
{
"name":"_text_",
...
},
{
"name":"_version_",
...
},
{
"name":"id",
...
}]}

Field value guessed as String -> strings fieldType

Field value guessed as String -> strings fieldType

Field value guessed as Long -> tlongs fieldType

Field value guessed as Double -> tdoubles fieldType

Field value guessed as Date -> tdates fieldType

Field value guessed as Long -> tlongs fieldType

You Can Still Be Explicit
Even if you want to use schemaless mode for most fields, you can still use the Schema API to
pre-emptively create some fields, with explicit types, before you index documents that use them.
Internally, the Schema REST API and the Schemaless Update Processors both use the same Managed
Schema functionality.
Once a field has been added to the schema, its field type is fixed. As a consequence, adding documents with
field value(s) that conflict with the previously guessed field type will fail. For example, after adding the above
document, the "Sold" field has the fieldType tlongs, but the document below has a non-integral decimal
value in this field:
curl "http://localhost:8983/solr/gettingstarted/update?commit=true" -H
"Content-type:application/csv" -d '
id,Description,Sold
19F,Cassettes by the pound,4.93'

This document will fail, as shown in this output:

Apache Solr Reference Guide 5.1

94



400
7


ERROR: [doc=19F] Error adding field 'Sold'='4.93' msg=For input
string: "4.93"
400



Apache Solr Reference Guide 5.1

95

Understanding Analyzers, Tokenizers, and Filters
The following sections describe how Solr breaks down and works with textual data. There are three main
concepts to understand: analyzers, tokenizers, and filters.
Field analyzers are used both during ingestion, when a document is indexed, and at query time. An analyzer
examines the text of fields and generates a token stream. Analyzers may be a single class or they may be
composed of a series of tokenizer and filter classes.
Tokenizers break field data into lexical units, or tokens.
Filters examine a stream of tokens and keep them, transform or discard them, or create new ones. Tokenizers
and filters may be combined to form pipelines, or chains, where the output of one is input to the next. Such a
sequence of tokenizers and filters is called an analyzer and the resulting output of an analyzer is used to match
query results or build indices.

Using Analyzers, Tokenizers, and Filters
Although the analysis process is used for both indexing and querying, the same analysis process need not be
used for both operations. For indexing, you often want to simplify, or normalize, words. For example, setting all
letters to lowercase, eliminating punctuation and accents, mapping words to their stems, and so on. Doing so
can increase recall because, for example, "ram", "Ram" and "RAM" would all match a query for "ram". To
increase query-time precision, a filter could be employed to narrow the matches by, for example, ignoring all-cap
acronyms if you're interested in male sheep, but not Random Access Memory.
The tokens output by the analysis process define the values, or terms, of that field and are used either to build an
index of those terms when a new document is added, or to identify which documents contain the terms you are
querying for.

For More Information
These sections will show you how to configure field analyzers and also serves as a reference for the details of
configuring each of the available tokenizer and filter classes. It also serves as a guide so that you can configure
your own analysis classes if you have special needs that cannot be met with the included filters or tokenizers.
For Analyzers, see:
Analyzers: Detailed conceptual information about Solr analyzers.
Running Your Analyzer: Detailed information about testing and running your Solr analyzer.
For Tokenizers, see:
About Tokenizers: Detailed conceptual information about Solr tokenizers.
Tokenizers: Information about configuring tokenizers, and about the tokenizer factory classes included in
this distribution of Solr.
For Filters, see:
About Filters: Detailed conceptual information about Solr filters.
Filter Descriptions: Information about configuring filters, and about the filter factory classes included in this
distribution of Solr.
CharFilterFactories: Information about filters for pre-processing input characters.
To find out how to use Tokenizers and Filters with various languages, see:

Apache Solr Reference Guide 5.1

96

Language Analysis: Information about tokenizers and filters for character set conversion or for use with
specific languages.

Analyzers
An analyzer examines the text of fields and generates a token stream. Analyzers are specified as a child of the <
fieldType> element in the schema.xml configuration file (in the same conf/ directory as solrconfig.xml)
.
In normal usage, only fields of type solr.TextField will specify an analyzer. The simplest way to configure an
analyzer is with a single  element whose class attribute is a fully qualified Java class name. The
named class must derive from org.apache.lucene.analysis.Analyzer. For example:




In this case a single class, WhitespaceAnalyzer, is responsible for analyzing the content of the named text
field and emitting the corresponding tokens. For simple cases, such as plain English prose, a single analyzer
class like this may be sufficient. But it's often necessary to do more complex analysis of the field content.
Even the most complex analysis requirements can usually be decomposed into a series of discrete, relatively
simple processing steps. As you will soon discover, the Solr distribution comes with a large selection of
tokenizers and filters that covers most scenarios you are likely to encounter. Setting up an analyzer chain is very
straightforward; you specify a simple  element (no class attribute) with child elements that name
factory classes for the tokenizer and filters to use, in the order you want them to run.
For example:










Note that classes in the org.apache.solr.analysis package may be referred to here with the shorthand so
lr. prefix.
In this case, no Analyzer class was specified on the  element. Rather, a sequence of more
specialized classes are wired together and collectively act as the Analyzer for the field. The text of the field is
passed to the first item in the list (solr.StandardTokenizerFactory), and the tokens that emerge from the
last one (solr.EnglishPorterFilterFactory) are the terms that are used for indexing or querying any
fields that use the "nametext" fieldType.

Apache Solr Reference Guide 5.1

97

Field Values versus Indexed Terms
The output of an Analyzer affects the terms indexed in a given field (and the terms used when parsing
queries against those fields) but it has no impact on the stored value for the fields. For example: an
analyzer might split "Brown Cow" into two indexed terms "brown" and "cow", but the stored value will still
be a single String: "Brown Cow"

Analysis Phases
Analysis takes place in two contexts. At index time, when a field is being created, the token stream that results
from analysis is added to an index and defines the set of terms (including positions, sizes, and so on) for the
field. At query time, the values being searched for are analyzed and the terms that result are matched against
those that are stored in the field's index.
In many cases, the same analysis should be applied to both phases. This is desirable when you want to query
for exact string matches, possibly with case-insensitivity, for example. In other cases, you may want to apply
slightly different analysis steps during indexing than those used at query time.
If you provide a simple  definition for a field type, as in the examples above, then it will be used for
both indexing and queries. If you want distinct analyzers for each phase, you may include two  defi
nitions distinguished with a type attribute. For example:













In this theoretical example, at index time the text is tokenized, the tokens are set to lowercase, any that are not
listed in keepwords.txt are discarded and those that remain are mapped to alternate values as defined by the
synonym rules in the file syns.txt. This essentially builds an index from a restricted set of possible values and
then normalizes them to values that may not even occur in the original text.
At query time, the only normalization that happens is to convert the query terms to lowercase. The filtering and
mapping steps that occur at index time are not applied to the query terms. Queries must then, in this example, be
very precise, using only the normalized terms that were stored at index time.

About Tokenizers
The job of a tokenizer is to break up a stream of text into tokens, where each token is (usually) a sub-sequence
of the characters in the text. An analyzer is aware of the field it is configured for, but a tokenizer is not.
Tokenizers read from a character stream (a Reader) and produce a sequence of Token objects (a
TokenStream).

Apache Solr Reference Guide 5.1

98

Characters in the input stream may be discarded, such as whitespace or other delimiters. They may also be
added to or replaced, such as mapping aliases or abbreviations to normalized forms. A token contains various
metadata in addition to its text value, such as the location at which the token occurs in the field. Because a
tokenizer may produce tokens that diverge from the input text, you should not assume that the text of the token is
the same text that occurs in the field, or that its length is the same as the original text. It's also possible for more
than one token to have the same position or refer to the same offset in the original text. Keep this in mind if you
use token metadata for things like highlighting search results in the field text.






The class named in the tokenizer element is not the actual tokenizer, but rather a class that implements the org
.apache.solr.analysis.TokenizerFactory interface. This factory class will be called upon to create new
tokenizer instances as needed. Objects created by the factory must derive from org.apache.lucene.analys
is.TokenStream, which indicates that they produce sequences of tokens. If the tokenizer produces tokens that
are usable as is, it may be the only component of the analyzer. Otherwise, the tokenizer's output tokens will
serve as input to the first filter stage in the pipeline.
A TypeTokenFilterFactory is available that creates a TypeTokenFilter that filters tokens based on their
TypeAttribute, which is set in factory.getStopTypes.
For a complete list of the available TokenFilters, see the section Tokenizers.

When To use a CharFilter vs. a TokenFilter
There are several pairs of CharFilters and TokenFilters that have related (ie: MappingCharFilter and ASCIIF
oldingFilter) or nearly identical (ie: PatternReplaceCharFilterFactory and PatternReplaceFilte
rFactory) functionality and it may not always be obvious which is the best choice.
The decision about which to use depends largely on which Tokenizer you are using, and whether you need to
preprocess the stream of characters.
For example, suppose you have a tokenizer such as StandardTokenizer and although you are pretty happy
with how it works overall, you want to customize how some specific characters behave. You could modify the
rules and re-build your own tokenizer with JFlex, but it might be easier to simply map some of the characters
before tokenization with a CharFilter.

About Filters
Like tokenizers, filters consume input and produce a stream of tokens. Filters also derive from org.apache.lu
cene.analysis.TokenStream. Unlike tokenizers, a filter's input is another TokenStream. The job of a filter is
usually easier than that of a tokenizer since in most cases a filter looks at each token in the stream sequentially
and decides whether to pass it along, replace it or discard it.
A filter may also do more complex analysis by looking ahead to consider multiple tokens at once, although this is
less common. One hypothetical use for such a filter might be to normalize state names that would be tokenized
as two words. For example, the single token "california" would be replaced with "CA", while the token pair
"rhode" followed by "island" would become the single token "RI".

Apache Solr Reference Guide 5.1

99

Because filters consume one TokenStream and produce a new TokenStream, they can be chained one after
another indefinitely. Each filter in the chain in turn processes the tokens produced by its predecessor. The order
in which you specify the filters is therefore significant. Typically, the most general filtering is done first, and later
filtering stages are more specialized.









This example starts with Solr's standard tokenizer, which breaks the field's text into tokens. Those tokens then
pass through Solr's standard filter, which removes dots from acronyms, and performs a few other common
operations. All the tokens are then set to lowercase, which will facilitate case-insensitive matching at query time.
The last filter in the above example is a stemmer filter that uses the Porter stemming algorithm. A stemmer is
basically a set of mapping rules that maps the various forms of a word back to the base, or stem, word from
which they derive. For example, in English the words "hugs", "hugging" and "hugged" are all forms of the stem
word "hug". The stemmer will replace all of these terms with "hug", which is what will be indexed. This means
that a query for "hug" will match the term "hugged", but not "huge".
Conversely, applying a stemmer to your query terms will allow queries containing non stem terms, like "hugging",
to match documents with different variations of the same stem word, such as "hugged". This works because both
the indexer and the query will map to the same stem ("hug").
Word stemming is, obviously, very language specific. Solr includes several language-specific stemmers created
by the Snowball generator that are based on the Porter stemming algorithm. The generic Snowball Porter
Stemmer Filter can be used to configure any of these language stemmers. Solr also includes a convenience
wrapper for the English Snowball stemmer. There are also several purpose-built stemmers for non-English
languages. These stemmers are described in Language Analysis.

Tokenizers
You configure the tokenizer for a text field type in schema.xml with a  element, as a child of :







The class attribute names a factory class that will instantiate a tokenizer object when needed. Tokenizer factory
classes implement the org.apache.solr.analysis.TokenizerFactory. A TokenizerFactory's create()
method accepts a Reader and returns a TokenStream. When Solr creates the tokenizer it passes a Reader
object that provides the content of the text field.
Tokenizers discussed in this section:
Standard Tokenizer

Apache Solr Reference Guide 5.1

100

Classic Tokenizer
Keyword Tokenizer
Letter Tokenizer
Lower Case Tokenizer
N-Gram Tokenizer
Edge N-Gram Tokenizer
ICU Tokenizer
Path Hierarchy Tokenizer
Regular Expression Pattern Tokenizer
UAX29 URL Email Tokenizer
White Space Tokenizer
Related Topics
Arguments may be passed to tokenizer factories by setting attributes on the  element.






The following sections describe the tokenizer factory classes included in this release of Solr.
For more information about Solr's tokenizers, see http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

Standard Tokenizer
This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter
characters are discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token, including Internet domain
names.
The "@" character is among the set of token-splitting punctuation, so email addresses are not preserved
as single tokens.
Note that words are split at hyphens.
The Standard Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following token
types: , , , , and .
Factory class: solr.StandardTokenizerFactory
Arguments:
maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by
maxTokenLength.
Example:




Apache Solr Reference Guide 5.1

101

In: "Please, email john.doe@foo.com by 03-09, re: m37-xq."
Out: "Please", "email", "john.doe", "foo.com", "by", "03", "09", "re", "m37", "xq"

Classic Tokenizer
The Classic Tokenizer preserves the same behavior as the Standard Tokenizer of Solr versions 3.1 and
previous. It does not use the Unicode standard annex UAX#29 word boundary rules that the Standard Tokenizer
uses. This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter
characters are discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token.
Words are split at hyphens, unless there is a number in the word, in which case the token is not split and
the numbers and hyphen(s) are preserved.
Recognizes Internet domain names and email addresses and preserves them as a single token.
Factory class: solr.ClassicTokenizerFactory
Arguments:
maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by
maxTokenLength.
Example:




In: "Please, email john.doe@foo.com by 03-09, re: m37-xq."
Out: "Please", "email", "john.doe@foo.com", "by", "03-09", "re", "m37-xq"

Keyword Tokenizer
This tokenizer treats the entire text field as a single token.
Factory class: solr.KeywordTokenizerFactory
Arguments: None
Example:




In: "Please, email john.doe@foo.com by 03-09, re: m37-xq."
Out: "Please, email john.doe@foo.com by 03-09, re: m37-xq."

Letter Tokenizer
This tokenizer creates tokens from strings of contiguous letters, discarding all non-letter characters.

Apache Solr Reference Guide 5.1

102

Factory class: solr.LetterTokenizerFactory
Arguments: None
Example:




In: "I can't."
Out: "I", "can", "t"

Lower Case Tokenizer
Tokenizes the input stream by delimiting at non-letters and then converting all letters to lowercase. Whitespace
and non-letters are discarded.
Factory class: solr.LowerCaseTokenizerFactory
Arguments: None
Example:




In: "I just LOVE my iPhone!"
Out: "i", "just", "love", "my", "iphone"

N-Gram Tokenizer
Reads the field text and generates n-gram tokens of sizes in the given range.
Factory class: solr.NGramTokenizerFactory
Arguments:
minGramSize: (integer, default 1) The minimum n-gram size, must be > 0.
maxGramSize: (integer, default 2) The maximum n-gram size, must be >= minGramSize.
Example:
Default behavior. Note that this tokenizer operates over the whole field. It does not break the field at whitespace.
As a result, the space character is included in the encoding.




In: "hey man"

Apache Solr Reference Guide 5.1

103

Out: "h", "e", "y", " ", "m", "a", "n", "he", "ey", "y ", " m", "ma", "an"
Example:
With an n-gram size range of 4 to 5:




In: "bicycle"
Out: "bicy", "bicyc", "icyc", "icycl", "cycl", "cycle", "ycle"

Edge N-Gram Tokenizer
Reads the field text and generates edge n-gram tokens of sizes in the given range.
Factory class: solr.EdgeNGramTokenizerFactory
Arguments:
minGramSize: (integer, default is 1) The minimum n-gram size, must be > 0.
maxGramSize: (integer, default is 1) The maximum n-gram size, must be >= minGramSize.
side: ("front" or "back", default is "front") Whether to compute the n-grams from the beginning (front) of the text
or from the end (back).
Example:
Default behavior (min and max default to 1):




In: "babaloo"
Out: "b"
Example:
Edge n-gram range of 2 to 5




In: "babaloo"
Out:"ba", "bab", "baba", "babal"
Example:
Edge n-gram range of 2 to 5, from the back side:

Apache Solr Reference Guide 5.1

104





In: "babaloo"
Out: "oo", "loo", "aloo", "baloo"

ICU Tokenizer
This tokenizer processes multilingual text and tokenizes it appropriately based on its script attribute.
You can customize this tokenizer's behavior by specifying per-script rule files. To add per-script rules, add a rul
efiles argument, which should contain a comma-separated list of code:rulefile pairs in the following
format: four-letter ISO 15924 script code, followed by a colon, then a resource path. For example, to specify rules
for Latin (script code "Latn") and Cyrillic (script code "Cyrl"), you would enter Latn:my.Latin.rules.rbbi,C
yrl:my.Cyrillic.rules.rbbi.
The default solr.ICUTokenizerFactory provides UAX#29 word break rules tokenization (like solr.Stand
ardTokenizer), but also includes custom tailorings for Hebrew (specializing handling of double and single
quotation marks), and for syllable tokenization for Khmer, Lao, and Myanmar.
Factory class: solr.ICUTokenizerFactory
Arguments:
rulefile: a comma-separated list of code:rulefile pairs in the following format: four-letter ISO 15924 script
code, followed by a colon, then a resource path.
Example:









Path Hierarchy Tokenizer
This tokenizer creates synonyms from file path hierarchies.
Factory class: solr.PathHierarchyTokenizerFactory
Arguments:
delimiter: (character, no default) You can specify the file path delimiter and replace it with a delimiter you
provide. This can be useful for working with backslash delimiters.

Apache Solr Reference Guide 5.1

105

replace: (character, no default) Specifies the delimiter character Solr uses in the tokenized output.
Example:






In: "c:\usr\local\apache"
Out: "c:", "c:/usr", "c:/usr/local", "c:/usr/local/apache"

Regular Expression Pattern Tokenizer
This tokenizer uses a Java regular expression to break the input text stream into tokens. The expression
provided by the pattern argument can be interpreted either as a delimiter that separates tokens, or to match
patterns that should be extracted from the text as tokens.
See the Javadocs for java.util.regex.Pattern for more information on Java regular expression syntax.
Factory class: solr.PatternTokenizerFactory
Arguments:
pattern: (Required) The regular expression, as defined by in java.util.regex.Pattern.
group: (Optional, default -1) Specifies which regex group to extract as the token(s). The value -1 means the
regex should be treated as a delimiter that separates tokens. Non-negative group numbers (>= 0) indicate that
character sequences matching that regex group should be converted to tokens. Group zero refers to the entire
regex, groups greater than zero refer to parenthesized sub-expressions of the regex, counted from left to right.
Example:
A comma separated list. Tokens are separated by a sequence of zero or more spaces, a comma, and zero or
more spaces.




In: "fee,fie, foe , fum, foo"
Out: "fee", "fie", "foe", "fum", "foo"
Example:
Extract simple, capitalized words. A sequence of at least one capital letter followed by zero or more letters of
either case is extracted as a token.

Apache Solr Reference Guide 5.1

106





In: "Hello. My name is Inigo Montoya. You killed my father. Prepare to die."
Out: "Hello", "My", "Inigo", "Montoya", "You", "Prepare"
Example:
Extract part numbers which are preceded by "SKU", "Part" or "Part Number", case sensitive, with an optional
semi-colon separator. Part numbers must be all numeric digits, with an optional hyphen. Regex capture groups
are numbered by counting left parenthesis from left to right. Group 3 is the subexpression "[0-9-]+", which
matches one or more digits or hyphens.




In: "SKU: 1234, Part Number 5678, Part: 126-987"
Out: "1234", "5678", "126-987"

UAX29 URL Email Tokenizer
This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter
characters are discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token.
Words are split at hyphens, unless there is a number in the word, in which case the token is not split and
the numbers and hyphen(s) are preserved.
Recognizes top-level Internet domain names (validated against the white list in the IANA Root Zone
Database when the tokenizer was generated); email addresses; file : //, http(s)://, and ftp:// a
ddresses; IPv4 and IPv6 addresses; and preserves them as a single token.
The UAX29 URL Email Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following
token types: , , , , , , and .
Factory class: solr.UAX29URLEmailTokenizerFactory
Arguments:
maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by
maxTokenLength.
Example:

Apache Solr Reference Guide 5.1

107





In: "Visit http://accarol.com/contact.htm?from=external&a=10 or e-mail bob.cratchet@accarol.com"
Out: "Visit", "http://accarol.com/contact.htm?from=external&a=10", "or", "e", "mail", "bob.cratchet@accarol.com"

White Space Tokenizer
Simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters
as tokens. Note that any punctuation will be included in the tokenization.
Factory class: solr.WhitespaceTokenizerFactory
Arguments: None
Example:




In: "To be, or what?"
Out: "To", "be,", "or", "what?"

Related Topics
TokenizerFactories

Filter Descriptions
You configure each filter with a  element in schema.xml as a child of , following the  element. Filter definitions should follow a tokenizer or another filter definition because they take a T
okenStream as input. For example.



...



The class attribute names a factory class that will instantiate a filter object as needed. Filter factory classes must
implement the org.apache.solr.analysis.TokenFilterFactory interface. Like tokenizers, filters are
also instances of TokenStream and thus are producers of tokens. Unlike tokenizers, filters also consume tokens
from a TokenStream. This allows you to mix and match filters, in any order you prefer, downstream of a
tokenizer.
Arguments may be passed to tokenizer factories to modify their behavior by setting attributes on the  e
lement. For example:

Apache Solr Reference Guide 5.1

108








The following sections describe the filter factories that are included in this release of Solr.
For more information about Solr's filters, see http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.
Filters discussed in this section:
ASCII Folding Filter
Beider-Morse Filter
Classic Filter
Common Grams Filter
Collation Key Filter
Daitch-Mokotoff Soundex Filter
Double Metaphone Filter
Edge N-Gram Filter
English Minimal Stem Filter
Hunspell Stem Filter
Hyphenated Words Filter
ICU Folding Filter
ICU Normalizer 2 Filter
ICU Transform Filter
Keep Words Filter
KStem Filter
Length Filter
Lower Case Filter
Managed Stop Filter
Managed Synonym Filter
N-Gram Filter
Numeric Payload Token Filter
Pattern Replace Filter
Phonetic Filter
Porter Stem Filter
Remove Duplicates Token Filter
Reversed Wildcard Filter
Shingle Filter
Snowball Porter Stemmer Filter
Standard Filter
Stop Filter
Suggest Stop Filter
Synonym Filter
Token Offset Payload Filter
Trim Filter
Type As Payload Filter
Type Token Filter
Word Delimiter Filter

Apache Solr Reference Guide 5.1

109

Related Topics

ASCII Folding Filter
This filter converts alphabetic, numeric, and symbolic Unicode characters which are not in the Basic Latin
Unicode block (the first 127 ASCII characters) to their ASCII equivalents, if one exists. This filter converts
characters from the following Unicode blocks:
C1 Controls and Latin-1 Supplement (PDF)
Latin Extended-A (PDF)
Latin Extended-B (PDF)
Latin Extended Additional (PDF)
Latin Extended-C (PDF)
Latin Extended-D (PDF)
IPA Extensions (PDF)
Phonetic Extensions (PDF)
Phonetic Extensions Supplement (PDF)
General Punctuation (PDF)
Superscripts and Subscripts (PDF)
Enclosed Alphanumerics (PDF)
Dingbats (PDF)
Supplemental Punctuation (PDF)
Alphabetic Presentation Forms (PDF)
Halfwidth and Fullwidth Forms (PDF)
Factory class: solr.ASCIIFoldingFilterFactory
Arguments: None
Example:




In: "á" (Unicode character 00E1)
Out: "a" (ASCII character 97)

Beider-Morse Filter
Implements the Beider-Morse Phonetic Matching (BMPM) algorithm, which allows identification of similar names,
even if they are spelled differently or in different languages. More information about how this works is available in
the section on Phonetic Matching.
BeiderMorseFilter changed its behavior in Solr 5.0 (version 3.04 of the BMPM algorithm is implemented,
while previous Solr versions implemented BMPM version 3.00 - see http://stevemorse.org/phoneticinfo.h
tm), so any index built using this filter with earlier versions of Solr will need to be rebuilt.
Factory class: solr.BeiderMorseFilterFactory
Arguments:

Apache Solr Reference Guide 5.1

110

nameType: Types of names. Valid values are GENERIC, ASHKENAZI, or SEPHARDIC. If not processing
Ashkenazi or Sephardic names, use GENERIC.
ruleType: Types of rules to apply. Valid values are APPROX or EXACT.
concat: Defines if multiple possible matches should be combined with a pipe ("|").
languageSet: The language set to use. The value "auto" will allow the Filter to identify the language, or a
comma-separated list can be supplied.
Example:






Classic Filter
This filter takes the output of the Classic Tokenizer and strips periods from acronyms and "'s" from possessives.
Factory class: solr.ClassicFilterFactory
Arguments: None
Example:





In: "I.B.M. cat's can't"
Tokenizer to Filter: "I.B.M", "cat's", "can't"
Out: "IBM", "cat", "can't"

Common Grams Filter
This filter creates word shingles by combining common tokens such as stop words with regular tokens. This is
useful for creating phrase queries containing common words, such as "the cat." Solr normally ignores stop words
in queried phrases, so searching for "the cat" would return all matches for the word "cat."
Factory class: solr.CommonGramsFilterFactory
Arguments:
words: (a common word file in .txt format) Provide the name of a common word file, such as stopwords.txt.
format: (optional) If the stopwords list has been formatted for Snowball, you can specify format="snowball"
so Solr can read the stopwords file.

Apache Solr Reference Guide 5.1

111

ignoreCase: (boolean) If true, the filter ignores the case of words when comparing them to the common word
file. The default is false.
Example:





In: "the Cat"
Tokenizer to Filter: "the", "Cat"
Out: "the_cat"

Collation Key Filter
Collation allows sorting of text in a language-sensitive way. It is usually used for sorting, but can also be used
with advanced searches. We've covered this in much more detail in the section on Unicode Collation.

Daitch-Mokotoff Soundex Filter
Implements the Daitch-Mokotoff Soundex algorithm, which allows identification of similar names, even if they are
spelled differently. More information about how this works is available in the section on Phonetic Matching.
Factory class: solr.DaitchMokotoffSoundexFilterFactory
Arguments:
inject : (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens
are replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact
spelling of the target word may not match.
Example:





Double Metaphone Filter
This filter creates tokens using the

DoubleMetaphone encoding algorithm from commons-codec. For more

information, see the Phonetic Matching section.
Factory class: solr.DoubleMetaphoneFilterFactory
Arguments:
inject: (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens
are replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact
spelling of the target word may not match.

Apache Solr Reference Guide 5.1

112

maxCodeLength: (integer) The maximum length of the code to be generated.
Example:
Default behavior for inject (true): keep the original token and add phonetic token(s) at the same position.





In: "four score and Kuczewski"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "Kuczewski"(4)
Out: "four"(1), "FR"(1), "score"(2), "SKR"(2), "and"(3), "ANT"(3), "Kuczewski"(4), "KSSK"(4), "KXFS"(4)
The phonetic tokens have a position increment of 0, which indicates that they are at the same position as the
token they were derived from (immediately preceding). Note that "Kuczewski" has two encodings, which are
added at the same position.
Example:
Discard original token (inject="false").





In: "four score and Kuczewski"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "Kuczewski"(4)
Out: "FR"(1), "SKR"(2), "ANT"(3), "KSSK"(4), "KXFS"(4)
Note that "Kuczewski" has two encodings, which are added at the same position.

Edge N-Gram Filter
This filter generates edge n-gram tokens of sizes within the given range.
Factory class: solr.EdgeNGramFilterFactory
Arguments:
minGramSize: (integer, default 1) The minimum gram size.
maxGramSize: (integer, default 1) The maximum gram size.
Example:
Default behavior.

Apache Solr Reference Guide 5.1

113






In: "four score and twenty"
Tokenizer to Filter: "four", "score", "and", "twenty"
Out: "f", "s", "a", "t"
Example:
A range of 1 to 4.





In: "four score"
Tokenizer to Filter: "four", "score"
Out: "f", "fo", "fou", "four", "s", "sc", "sco", "scor"
Example:
A range of 4 to 6.





In: "four score and twenty"
Tokenizer to Filter: "four", "score", "and", "twenty"
Out: "four", "scor", "score", "twen", "twent", "twenty"

English Minimal Stem Filter
This filter stems plural English words to their singular form.
Factory class: solr.EnglishMinimalStemFilterFactory
Arguments: None
Example:





Apache Solr Reference Guide 5.1

114

In: "dogs cats"
Tokenizer to Filter: "dogs", "cats"
Out: "dog", "cat"

Hunspell Stem Filter
The Hunspell Stem Filter provides support for several languages. You must provide the dictionary (.dic) and
rules (.aff) files for each language you wish to use with the Hunspell Stem Filter. You can download those
language files here. Be aware that your results will vary widely based on the quality of the provided dictionary
and rules files. For example, some languages have only a minimal word list with no morphological information.
On the other hand, for languages that have no stemmer but do have an extensive dictionary file, the Hunspell
stemmer may be a good choice.
Factory class: solr.HunspellStemFilterFactory
Arguments:
dictionary: (required) The path of a dictionary file.
affix: (required) The path of a rules file.
ignoreCase: (boolean) controls whether matching is case sensitive or not. The default is false.
strictAffixParsing: (boolean) controls whether the affix parsing is strict or not. If true, an error while
reading an affix rule causes a ParseException, otherwise is ignored. The default is true.
Example:





In: "jump jumping jumped"
Tokenizer to Filter: "jump", "jumping", "jumped"
Out: "jump", "jump", "jump"

Hyphenated Words Filter
This filter reconstructs hyphenated words that have been tokenized as two tokens because of a line break or
other intervening whitespace in the field test. If a token ends with a hyphen, it is joined with the following token
and the hyphen is discarded. Note that for this filter to work properly, the upstream tokenizer must not remove
trailing hyphen characters. This filter is generally only useful at index time.
Factory class: solr.HyphenatedWordsFilterFactory
Arguments: None
Example:

Apache Solr Reference Guide 5.1

115






In: "A hyphen- ated word"
Tokenizer to Filter: "A", "hyphen-", "ated", "word"
Out: "A", "hyphenated", "word"

ICU Folding Filter
This filter is a custom Unicode normalization form that applies the foldings specified in Unicode Technical Report
30 in addition to the NFKC_Casefold normalization form as described in ICU Normalizer 2 Filter. This filter is a
better substitute for the combined behavior of the ASCII Folding Filter, Lower Case Filter, and ICU Normalizer 2
Filter.
To use this filter, see solr/contrib/analysis-extras/README.txt for instructions on which jars you
need to add to your solr_home/lib.
Factory class: solr.ICUFoldingFilterFactory
Arguments: None
Example:





For detailed information on this normalization form, see http://www.unicode.org/reports/tr30/tr30-4.html.

ICU Normalizer 2 Filter
This filter factory normalizes text according to one of five Unicode Normalization Forms as described in Unicode
Standard Annex #15:
NFC: (name="nfc" mode="compose") Normalization Form C, canonical decomposition
NFD: (name="nfc" mode="decompose") Normalization Form D, canonical decomposition, followed by
canonical composition
NFKC: (name="nfkc" mode="compose") Normalization Form KC, compatibility decomposition
NFKD: (name="nfkc" mode="decompose") Normalization Form KD, compatibility decomposition, followed
by canonical composition
NFKC_Casefold: (name="nfkc_cf" mode="compose") Normalization Form KC, with additional Unicode
case folding. Using the ICU Normalizer 2 Filter is a better-performing substitution for the Lower Case Filter
and NFKC normalization.
Factory class: solr.ICUNormalizer2FilterFactory
Arguments:

Apache Solr Reference Guide 5.1

116

name: (string) The name of the normalization form; nfc, nfd, nfkc, nfkd, nfkc_cf
mode: (string) The mode of Unicode character composition and decomposition; compose or decompose
Example:





For detailed information about these Unicode Normalization Forms, see http://unicode.org/reports/tr15/.
To use this filter, see solr/contrib/analysis-extras/README.txt for instructions on which jars you
need to add to your solr_home/lib.

ICU Transform Filter
This filter applies ICU Tranforms to text. This filter supports only ICU System Transforms. Custom rule sets are
not supported.
Factory class: solr.ICUTransformFilterFactory
Arguments:
id: (string) The identifier for the ICU System Transform you wish to apply with this filter. For a full list of ICU
System Transforms, see http://demo.icu-project.org/icu-bin/translit?TEMPLATE_FILE=data/translit_rule_main.ht
ml.
Example:





For detailed information about ICU Transforms, see http://userguide.icu-project.org/transforms/general.
To use this filter, see solr/contrib/analysis-extras/README.txt for instructions on which jars you
need to add to your solr_home/lib.

Keep Words Filter
This filter discards all tokens except those that are listed in the given word list. This is the inverse of the Stop
Words Filter. This filter can be useful for building specialized indices for a constrained set of terms.
Factory class: solr.KeepWordFilterFactory
Arguments:
words: (required) Path of a text file containing the list of keep words, one per line. Blank lines and lines that
begin with "#" are ignored. This may be an absolute path, or a simple filename in the Solr config directory.
ignoreCase: (true/false) If true then comparisons are done case-insensitively. If this argument is true, then the
words file is assumed to contain only lowercase words. The default is false.

Apache Solr Reference Guide 5.1

117

enablePositionIncrements: if luceneMatchVersion is 4.3 or earlier and enablePositionIncrement
s="false", no position holes will be left by this filter when it removes tokens. This argument is invalid if luc
eneMatchVersion is 5.0 or later.
Example:
Where keepwords.txt contains:
happy
funny
silly





In: "Happy, sad or funny"
Tokenizer to Filter: "Happy", "sad", "or", "funny"
Out: "funny"
Example:
Same keepwords.txt, case insensitive:





In: "Happy, sad or funny"
Tokenizer to Filter: "Happy", "sad", "or", "funny"
Out: "Happy", "funny"
Example:
Using LowerCaseFilterFactory before filtering for keep words, no ignoreCase flag.






In: "Happy, sad or funny"
Tokenizer to Filter: "Happy", "sad", "or", "funny"
Filter to Filter: "happy", "sad", "or", "funny"

Apache Solr Reference Guide 5.1

118

Out: "happy", "funny"

KStem Filter
KStem is an alternative to the Porter Stem Filter for developers looking for a less aggressive stemmer. KStem
was written by Bob Krovetz, ported to Lucene by Sergio Guzman-Lara (UMASS Amherst). This stemmer is only
appropriate for English language text.
Factory class: solr.KStemFilterFactory
Arguments: None
Example:





In: "jump jumping jumped"
Tokenizer to Filter: "jump", "jumping", "jumped"
Out: "jump", "jump", "jump"

Length Filter
This filter passes tokens whose length falls within the min/max limit specified. All other tokens are discarded.
Factory class: solr.LengthFilterFactory
Arguments:
min: (integer, required) Minimum token length. Tokens shorter than this are discarded.
max: (integer, required, must be >= min) Maximum token length. Tokens longer than this are discarded.
enablePositionIncrements: if luceneMatchVersion is 4.3 or earlier and enablePositionIncrement
s="false", no position holes will be left by this filter when it removes tokens. This argument is invalid if luc
eneMatchVersion is 5.0 or later.
Example:





In: "turn right at Albuquerque"
Tokenizer to Filter: "turn", "right", "at", "Albuquerque"
Out: "turn", "right"

Lower Case Filter

Apache Solr Reference Guide 5.1

119

Converts any uppercase letters in a token to the equivalent lowercase token. All other characters are left
unchanged.
Factory class: solr.LowerCaseFilterFactory
Arguments: None
Example:





In: "Down With CamelCase"
Tokenizer to Filter: "Down", "With", "CamelCase"
Out: "down", "with", "camelcase"

Managed Stop Filter
This is specialized version of the Stop Words Filter Factory that uses a set of stop words that are managed from
a REST API.
Arguments:
managed: The name that should be used for this set of stop words in the managed REST API.
Example:
With this configuration the set of words is named "english" and can be managed via /solr/collection_name
/schema/analysis/stopwords/english





See Stop Filter for example input/output.

Managed Synonym Filter
This is specialized version of the Synonym Filter Factory that uses a mapping on synonyms that is managed
from a REST API.
Arguments:
managed: The name that should be used for this mapping on synonyms in the managed REST API.
Example:
With this configuration the set of mappings is named "english" and can be managed via /solr/collection_n
ame/schema/analysis/synonyms/english

Apache Solr Reference Guide 5.1

120






See Synonym Filter for example input/output.

N-Gram Filter
Generates n-gram tokens of sizes in the given range. Note that tokens are ordered by position and then by gram
size.
Factory class: solr.NGramFilterFactory
Arguments:
minGramSize: (integer, default 1) The minimum gram size.
maxGramSize: (integer, default 2) The maximum gram size.
Example:
Default behavior.





In: "four score"
Tokenizer to Filter: "four", "score"
Out: "f", "o", "u", "r", "fo", "ou", "ur", "s", "c", "o", "r", "e", "sc", "co", "or", "re"
Example:
A range of 1 to 4.





In: "four score"
Tokenizer to Filter: "four", "score"
Out: "f", "fo", "fou", "four", "s", "sc", "sco", "scor"
Example:
A range of 3 to 5.

Apache Solr Reference Guide 5.1

121






In: "four score"
Tokenizer to Filter: "four", "score"
Out: "fou", "four", "our", "sco", "scor", "score", "cor", "core", "ore"

Numeric Payload Token Filter
This filter adds a numeric floating point payload value to tokens that match a given type. Refer to the Javadoc for
the org.apache.lucene.analysis.Token class for more information about token types and payloads.
Factory class: solr.NumericPayloadTokenFilterFactory
Arguments:
payload: (required) A floating point value that will be added to all matching tokens.
typeMatch: (required) A token type name string. Tokens with a matching type name will have their payload set
to the above floating point value.
Example:





In: "bing bang boom"
Tokenizer to Filter: "bing", "bang", "boom"
Out: "bing"[0.75], "bang"[0.75], "boom"[0.75]

Pattern Replace Filter
This filter applies a regular expression to each token and, for those that match, substitutes the given replacement
string in place of the matched pattern. Tokens which do not match are passed though unchanged.
Factory class: solr.PatternReplaceFilterFactory
Arguments:
pattern: (required) The regular expression to test against each token, as per java.util.regex.Pattern.
replacement: (required) A string to substitute in place of the matched pattern. This string may contain
references to capture groups in the regex pattern. See the Javadoc for java.util.regex.Matcher.
replace: ("all" or "first", default "all") Indicates whether all occurrences of the pattern in the token should be
replaced, or only the first.

Apache Solr Reference Guide 5.1

122

Example:
Simple string replace:





In: "cat concatenate catycat"
Tokenizer to Filter: "cat", "concatenate", "catycat"
Out: "dog", "condogenate", "dogydog"
Example:
String replacement, first occurrence only:





In: "cat concatenate catycat"
Tokenizer to Filter: "cat", "concatenate", "catycat"
Out: "dog", "condogenate", "dogycat"
Example:
More complex pattern with capture group reference in the replacement. Tokens that start with non-numeric
characters and end with digits will have an underscore inserted before the numbers. Otherwise the token is
passed through.





In: "cat foo1234 9987 blah1234foo"
Tokenizer to Filter: "cat", "foo1234", "9987", "blah1234foo"
Out: "cat", "foo_1234", "9987", "blah1234foo"

Phonetic Filter
This filter creates tokens using one of the phonetic encoding algorithms in the org.apache.commons.codec.
language package. For more information, see the section on Phonetic Matching.
Factory class: solr.PhoneticFilterFactory

Apache Solr Reference Guide 5.1

123

Arguments:
encoder: (required) The name of the encoder to use. The encoder name must be one of the following (case
insensitive): "DoubleMetaphone", "Metaphone", "Soundex", "RefinedSoundex", "Caverphone" (v2.0), "CologneP
honetic", or "Nysiis".
inject: (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens
are replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact
spelling of the target word may not match.
maxCodeLength: (integer) The maximum length of the code to be generated by the Metaphone or Double
Metaphone encoders.
Example:
Default behavior for DoubleMetaphone encoding.





In: "four score and twenty"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "twenty"(4)
Out: "four"(1), "FR"(1), "score"(2), "SKR"(2), "and"(3), "ANT"(3), "twenty"(4), "TNT"(4)
The phonetic tokens have a position increment of 0, which indicates that they are at the same position as the
token they were derived from (immediately preceding).
Example:
Discard original token.





In: "four score and twenty"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "twenty"(4)
Out: "FR"(1), "SKR"(2), "ANT"(3), "TWNT"(4)
Example:
Default Soundex encoder.





Apache Solr Reference Guide 5.1

124

In: "four score and twenty"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "twenty"(4)
Out: "four"(1), "F600"(1), "score"(2), "S600"(2), "and"(3), "A530"(3), "twenty"(4), "T530"(4)

Porter Stem Filter
This filter applies the Porter Stemming Algorithm for English. The results are similar to using the Snowball Porter
Stemmer with the language="English" argument. But this stemmer is coded directly in Java and is not based
on Snowball. It does not accept a list of protected words and is only appropriate for English language text.
However, it has been benchmarked as four times faster than the English Snowball stemmer, so can provide a
performance enhancement.
Factory class: solr.PorterStemFilterFactory
Arguments: None
Example:





In: "jump jumping jumped"
Tokenizer to Filter: "jump", "jumping", "jumped"
Out: "jump", "jump", "jump"

Remove Duplicates Token Filter
The filter removes duplicate tokens in the stream. Tokens are considered to be duplicates if they have the same
text and position values.
Factory class: solr.RemoveDuplicatesTokenFilterFactory
Arguments: None
Example:
One example of where RemoveDuplicatesTokenFilterFactory is in situations where a synonym file is
being used in conjuntion with a stemmer causes some synonyms to be reduced to the same stem. Consider the
following entry from a synonyms.txt file:
Television, Televisions, TV, TVs

When used in the following configuration:

Apache Solr Reference Guide 5.1

125








In: "Watch TV"
Tokenizer to Synonym Filter: "Watch"(1) "TV"(2)
Synonym Filter to Stem Filter: "Watch"(1) "Television"(2) "Televisions"(2) "TV"(2) "TVs"(2)
Stem Filter to Remove Dups Filter: "Watch"(1) "Television"(2) "Television"(2) "TV"(2) "TV"(2)
Out: "Watch"(1) "Television"(2) "TV"(2)

Reversed Wildcard Filter
This filter reverses tokens to provide faster leading wildcard and prefix queries. Tokens without wildcards are not
reversed.
Factory class: solr.ReveresedWildcardFilterFactory
Arguments:
withOriginal (boolean) If true, the filter produces both original and reversed tokens at the same positions. If
false, produces only reversed tokens.
maxPosAsterisk (integer, default = 2) The maximum position of the asterisk wildcard ('*') that triggers the
reversal of the query term. Terms with asterisks at positions above this value are not reversed.
maxPosQuestion (integer, default = 1) The maximum position of the question mark wildcard ('?') that triggers
the reversal of query term. To reverse only pure suffix queries (queries with a single leading asterisk), set this to
0 and maxPosAsterisk to 1.
maxFractionAsterisk (float, default = 0.0) An additional parameter that triggers the reversal if asterisk ('*')
position is less than this fraction of the query token length.
minTrailing (integer, default = 2) The minimum number of trailing characters in a query token after the last
wildcard character. For good performance this should be set to a value larger than 1.
Example:





In: "*foo *bar"
Tokenizer to Filter: "*foo", "*bar"
Out: "oof*", "rab*"

Apache Solr Reference Guide 5.1

126

Shingle Filter
This filter constructs shingles, which are token n-grams, from the token stream. It combines runs of tokens into a
single token.
Factory class: solr.ShingleFilterFactory
Arguments:
minShingleSize: (integer, default 2) The minimum number of tokens per shingle.
maxShingleSize: (integer, must be >= 2, default 2) The maximum number of tokens per shingle.
outputUnigrams: (true/false) If true (the default), then each individual token is also included at its original
position.
outputUnigramsIfNoShingles: (true/false) If false (the default), then individual tokens will be output if no
shingles are possible.
tokenSeparator: (string, default is " ") The default string to use when joining adjacent tokens to form a shingle.
Example:
Default behavior.





In: "To be, or what?"
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4)
Out: "To"(1), "To be"(1), "be"(2), "be or"(2), "or"(3), "or what"(3), "what"(4)
Example:
A shingle size of four, do not include original token.





In: "To be, or not to be."
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "not"(4), "to"(5), "be"(6)
Out: "To be"(1), "To be or"(1), "To be or not"(1), "be or"(2), "be or not"(2), "be or not to"(2), "or not"(3), "or not
to"(3), "or not to be"(3), "not to"(4), "not to be"(4), "to be"(5)

Snowball Porter Stemmer Filter
This filter factory instantiates a language-specific stemmer generated by Snowball. Snowball is a software

Apache Solr Reference Guide 5.1

127

package that generates pattern-based word stemmers. This type of stemmer is not as accurate as a table-based
stemmer, but is faster and less complex. Table-driven stemmers are labor intensive to create and maintain and
so are typically commercial products.
Solr contains Snowball stemmers for Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French,
German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish. For
more information on Snowball, visit http://snowball.tartarus.org/.
StopFilterFactory, CommonGramsFilterFactory, and CommonGramsQueryFilterFactory can
optionally read stopwords in Snowball format (specify format="snowball" in the configuration of those
FilterFactories).
Factory class: solr.SnowballPorterFilterFactory
Arguments:
language: (default "English") The name of a language, used to select the appropriate Porter stemmer to use.
Case is significant. This string is used to select a package name in the "org.tartarus.snowball.ext" class
hierarchy.
protected: Path of a text file containing a list of protected words, one per line. Protected words will not be
stemmed. Blank lines and lines that begin with "#" are ignored. This may be an absolute path, or a simple file
name in the Solr config directory.
Example:
Default behavior:





In: "flip flipped flipping"
Tokenizer to Filter: "flip", "flipped", "flipping"
Out: "flip", "flip", "flip"
Example:
French stemmer, English words:





In: "flip flipped flipping"
Tokenizer to Filter: "flip", "flipped", "flipping"
Out: "flip", "flipped", "flipping"
Example:

Apache Solr Reference Guide 5.1

128

Spanish stemmer, Spanish words:





In: "cante canta"
Tokenizer to Filter: "cante", "canta"
Out: "cant", "cant"

Standard Filter
This filter removes dots from acronyms and the substring "'s" from the end of tokens. This filter depends on the
tokens being tagged with the appropriate term-type to recognize acronyms and words with apostrophes.
Factory class: solr.StandardFilterFactory
Arguments: None
This filter is no longer operational in Solr when the luceneMatchVersion (in solrconfig.xml) is
higher than "3.1".

Stop Filter
This filter discards, or stops analysis of, tokens that are on the given stop words list. A standard stop words list is
included in the Solr config directory, named stopwords.txt, which is appropriate for typical English language
text.
Factory class: solr.StopFilterFactory
Arguments:
words: (optional) The path to a file that contains a list of stop words, one per line. Blank lines and lines that
begin with "#" are ignored. This may be an absolute path, or path relative to the Solr config directory.
format: (optional) If the stopwords list has been formatted for Snowball, you can specify format="snowball"
so Solr can read the stopwords file.
ignoreCase: (true/false, default false) Ignore case when testing for stop words. If true, the stop list should
contain lowercase words.
enablePositionIncrements: if luceneMatchVersion is 4.4 or earlier and enablePositionIncrement
s="false", no position holes will be left by this filter when it removes tokens. This argument is invalid if luc
eneMatchVersion is 5.0 or later.
Example:
Case-sensitive matching, capitalized words not stopped. Token positions skip stopped words.

Apache Solr Reference Guide 5.1

129






In: "To be or what?"
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4)
Out: "To"(1), "what"(4)
Example:





In: "To be or what?"
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4)
Out: "what"(4)

Suggest Stop Filter
Like Stop Filter, this filter discards, or stops analysis of, tokens that are on the given stop words list. Suggest
Stop Filter differs from Stop Filter in that it will not remove the last token unless it is followed by a token
separator. For example, a query "find the" would preserve the 'the' since it was not followed by a space,
punctuation etc., and mark it as a KEYWORD so that following filters will not change or remove it. By contrast, a
query like "find the popsicle" would remove "the" as a stopword, since it's followed by a space. When
using one of the analyzing suggesters, you would normally use the ordinary StopFilterFactory in your index
analyzer and then SuggestStopFilter in your query analyzer.
Factory class: solr.SuggestStopFilterFactory
Arguments:
words: (optional; default: StopAnalyzer#ENGLISH_STOP_WORDS_SET ) The name of a stopwords file to
parse.
format: (optional; default: wordset) Defines how the words file will be parsed. If words is not specified, then f
ormat must not be specified. The valid values for the format option are:
wordset: This is the default format, which supports one word per line (including any intra-word
whitespace) and allows whole line comments begining with the "#" character. Blank lines are ignored.
snowball: This format allows for multiple words specified on each line, and trailing comments may be
specified using the vertical line ("|"). Blank lines are ignored.
ignoreCase: (optional; default: false) If true, matching is case-insensitive.

Example:

Apache Solr Reference Guide 5.1

130







In: "The The"
Tokenizer to Filter: "the"(1), "the"(2)
Out: "the"(2)

Synonym Filter
This filter does synonym mapping. Each token is looked up in the list of synonyms and if a match is found, then
the synonym is emitted in place of the token. The position value of the new tokens are set such they all occur at
the same position as the original token.
Factory class: solr.SynonymFilterFactory
Arguments:
synonyms: (required) The path of a file that contains a list of synonyms, one per line. In the (default) solr forma
t - see the format argument below for alternatives - blank lines and lines that begin with " #" are ignored. This
may be an absolute path, or path relative to the Solr config directory. There are two ways to specify synonym
mappings:
A comma-separated list of words. If the token matches any of the words, then all the words in the list are
substituted, which will include the original token.
Two comma-separated lists of words with the symbol "=>" between them. If the token matches any word
on the left, then the list on the right is substituted. The original token will not be included unless it is also in
the list on the right.
ignoreCase: (optional; default: false) If true, synonyms will be matched case-insensitively.
expand: (optional; default: true) If true, a synonym will be expanded to all equivalent synonyms. If false, all
equivalent synonyms will be reduced to the first in the list.
format: (optional; default: solr) Controls how the synonyms will be parsed. The short names solr (for SolrS
ynonymParser) and wordnet (for WordnetSynonymParser ) are supported, or you may alternatively supply
the name of your own SynonymMap.Builder subclass.
tokenizerFactory: (optional; default: WhitespaceTokenizerFactory) The name of the tokenizer factory
to use when parsing the synonyms file. Arguments with the name prefix " tokenizerFactory." will be
supplied as init params to the specified tokenizer factory. Any arguments not consumed by the synonym filter
factory, including those without the "tokenizerFactory." prefix, will also be supplied as init params to the
tokenizer factory. If tokenizerFactory is specified, then analyzer may not be, and vice versa.
analyzer: (optional; default: WhitespaceTokenizerFactory) The name of the analyzer class to use when
parsing the synonyms file. If analyzer is specified, then tokenizerFactory may not be, and vice versa.
For the following examples, assume a synonyms file named mysynonyms.txt:

Apache Solr Reference Guide 5.1

131

couch,sofa,divan
teh => the
huge,ginormous,humungous => large
small => tiny,teeny,weeny

Example:





In: "teh small couch"
Tokenizer to Filter: "teh"(1), "small"(2), "couch"(3)
Out: "the"(1), "tiny"(2), "teeny"(2), "weeny"(2), "couch"(3), "sofa"(3), "divan"(3)
Example:





In: "teh ginormous, humungous sofa"
Tokenizer to Filter: "teh"(1), "ginormous"(2), "humungous"(3), "sofa"(4)
Out: "the"(1), "large"(2), "large"(3), "couch"(4), "sofa"(4), "divan"(4)

Token Offset Payload Filter
This filter adds the numeric character offsets of the token as a payload value for that token.
Factory class: solr.TokenOffsetPayloadTokenFilterFactory
Arguments: None
Example:





In: "bing bang boom"
Tokenizer to Filter: "bing", "bang", "boom"
Out: "bing"[0,4], "bang"[5,9], "boom"[10,14]

Trim Filter

Apache Solr Reference Guide 5.1

132

This filter trims leading and/or trailing whitespace from tokens. Most tokenizers break tokens at whitespace, so
this filter is most often used for special situations.
Factory class: solr.TrimFilterFactory
Arguments:
updateOffsets: if luceneMatchVersion is 4.3 or earlier and updateOffsets="true", trimmed tokens'
start and end offsets will be updated to those of the first and last characters (plus one) remaining in the token. T
his argument is invalid if luceneMatchVersion is 5.0 or later.
Example:
The PatternTokenizerFactory configuration used here splits the input on simple commas, it does not remove
whitespace.





In: "one, two , three ,four "
Tokenizer to Filter: "one", " two ", " three ", "four "
Out: "one", "two", "three", "four"

Type As Payload Filter
This filter adds the token's type, as an encoded byte sequence, as its payload.
Factory class: solr.TypeAsPayloadTokenFilterFactory
Arguments: None
Example:





In: "Pay Bob's I.O.U."
Tokenizer to Filter: "Pay", "Bob's", "I.O.U."
Out: "Pay"[], "Bob's"[], "I.O.U."[]

Type Token Filter
This filter blacklists or whitelists a specified list of token types, assuming the tokens have type metadata
associated with them. For example, the UAX29 URL Email Tokenizer emits "" and "" typed
tokens, as well as other types. This filter would allow you to pull out only e-mail addresses from text as tokens, if
you wish.
Factory class: solr.TypeTokenFilterFactory

Apache Solr Reference Guide 5.1

133

Arguments:
types: Defines the location of a file of types to filter.
useWhitelist: If true, the file defined in types should be used as include list. If false, or undefined, the file
defined in types is used as a blacklist.
enablePositionIncrements: if luceneMatchVersion is 4.3 or earlier and enablePositionIncrement
s="false", no position holes will be left by this filter when it removes tokens. This argument is invalid if luc
eneMatchVersion is 5.0 or later.
Example:




Word Delimiter Filter
This filter splits tokens at word delimiters. The rules for determining delimiters are determined as follows:
A change in case within a word: "CamelCase" -> "Camel", "Case". This can be disabled by setting split
OnCaseChange="0".
A transition from alpha to numeric characters or vice versa: "Gonzo5000" -> "Gonzo", "5000" "4500XL" ->
"4500", "XL". This can be disabled by setting splitOnNumerics="0".
Non-alphanumeric characters (discarded): "hot-spot" -> "hot", "spot"
A trailing "'s" is removed: "O'Reilly's" -> "O", "Reilly"
Any leading or trailing delimiters are discarded: "--hot-spot--" -> "hot", "spot"
Factory class: solr.WordDelimiterFilterFactory
Arguments:
generateWordParts: (integer, default 1) If non-zero, splits words at delimiters. For example:"CamelCase",
"hot-spot" -> "Camel", "Case", "hot", "spot"
generateNumberParts: (integer, default 1) If non-zero, splits numeric strings at delimiters:"1947-32" ->"1947",
"32"
splitOnCaseChange: (integer, default 1) If 0, words are not split on camel-case changes:"BugBlaster-XL" -> "B
ugBlaster", "XL". Example 1 below illustrates the default (non-zero) splitting behavior.
splitOnNumerics: (integer, default 1) If 0, don't split words on transitions from alpha to numeric:"FemBot3000"
-> "Fem", "Bot3000"
catenateWords: (integer, default 0) If non-zero, maximal runs of word parts will be joined: "hot-spot-sensor's" > "hotspotsensor"
catenateNumbers: (integer, default 0) If non-zero, maximal runs of number parts will be joined: 1947-32" -> "1
94732"

Apache Solr Reference Guide 5.1

134

catenateAll: (0/1, default 0) If non-zero, runs of word and number parts will be joined: "Zap-Master-9000" -> "
ZapMaster9000"
preserveOriginal: (integer, default 0) If non-zero, the original token is preserved: "Zap-Master-9000" -> "Zap
-Master-9000", "Zap", "Master", "9000"
protected: (optional) The pathname of a file that contains a list of protected words that should be passed
through without splitting.
stemEnglishPossessive: (integer, default 1) If 1, strips the possessive "'s" from each subword.
Example:
Default behavior. The whitespace tokenizer is used here to preserve non-alphanumeric characters.





In: "hot-spot RoboBlaster/9000 100XL"
Tokenizer to Filter: "hot-spot", "RoboBlaster/9000", "100XL"
Out: "hot", "spot", "Robo", "Blaster", "9000", "100", "XL"
Example:
Do not split on case changes, and do not generate number parts. Note that by not generating number parts,
tokens containing only numeric parts are ultimately discarded.





In: "hot-spot RoboBlaster/9000 100-42"
Tokenizer to Filter: "hot-spot", "RoboBlaster/9000", "100-42"
Out: "hot", "spot", "RoboBlaster", "9000"
Example:
Concatenate word parts and number parts, but not word and number parts that occur in the same token.





In: "hot-spot 100+42 XL40"
Tokenizer to Filter: "hot-spot"(1), "100+42"(2), "XL40"(3)

Apache Solr Reference Guide 5.1

135

Out: "hot"(1), "spot"(2), "hotspot"(2), "100"(3), "42"(4), "10042"(4), "XL"(5), "40"(6)
Example:
Concatenate all. Word and/or number parts are joined together.





In: "XL-4000/ES"
Tokenizer to Filter: "XL-4000/ES"(1)
Out: "XL"(1), "4000"(2), "ES"(3), "XL4000ES"(3)
Example:
Using a protected words list that contains "AstroBlaster" and "XL-5000" (among others).





In: "FooBar AstroBlaster XL-5000 ==ES-34-"
Tokenizer to Filter: "FooBar", "AstroBlaster", "XL-5000", "==ES-34-"
Out: "FooBar", "FooBar", "AstroBlaster", "XL-5000", "ES", "34"

Related Topics
TokenFilterFactories

CharFilterFactories
Char Filter is a component that pre-processes input characters. Char Filters can be chained like Token Filters
and placed in front of a Tokenizer. Char Filters can add, change, or remove characters while preserving the
original character offsets to support features like highlighting.
Topics discussed in this section:
solr.MappingCharFilterFactory
solr.HTMLStripCharFilterFactory
solr.ICUNormalizer2CharFilterFactory
solr.PatternReplaceCharFilterFactory
Related Topics

solr.MappingCharFilterFactory
This filter creates org.apache.lucene.analysis.MappingCharFilter, which can be used for changing
one string to another (for example, for normalizing é to e.).
This filter requires specifying a mapping argument, which is the path and name of a file containing the mappings

Apache Solr Reference Guide 5.1

136

to perform.
Example:



[...]


Mapping file syntax:
Comment lines beginning with a hash mark (#), as well as blank lines, are ignored.
Each non-comment, non-blank line consists of a mapping of the form: "source" => "target"
Double-quoted source string, optional whitespace, an arrow (=>), optional whitespace,
double-quoted target string.
Trailing comments on mapping lines are not allowed.
The source string must contain at least one character, but the target string may be empty.
The following character escape sequences are recognized within source and target strings:
Escape
sequence

Resulting character (ECMA-48 alia
s)

Unicode
character

Example mapping line

\\

\

U+005C

"\\" => "/"

\"

"

U+0022

"\"and\"" => "'and'"

\b

backspace (BS)

U+0008

"\b" => " "

\t

tab (HT)

U+0009

"\t" => ","

\n

newline (LF)

U+000A

"\n" => "
" \f form feed (FF) U+000C "\f" => "\n" \r carriage return (CR) U+000D "\r" => "/carriage-return/" \uXXXX Unicode char referenced by the 4 hex digits U+XXXX "\uFEFF" => "" A backslash followed by any other character is interpreted as if the character were present without the backslash. solr.HTMLStripCharFilterFactory This filter creates org.apache.solr.analysis.HTMLStripCharFilter. This Char Filter strips HTML from the input stream and passes the result to another Char Filter or a Tokenizer. This filter: Removes HTML/XML tags while preserving other content. Removes attributes within tags and supports optional attribute quoting. Apache Solr Reference Guide 5.1 137 Removes XML processing instructions, such as: Removes XML comments. Removes XML elements starting with . Removes contents of '); --> hello if a hello a [...]
solr.PatternReplaceCharFilterFactory This filter uses regular expressions to replace or change character patterns. Arguments: pattern: the regular expression pattern to apply to the incoming text. replacement: the text to use to replace matching patterns. You can configure this filter in schema.xml like this: [...] The table below presents examples of regex-based pattern replacement: Input pattern replacement Output Description see-ing looking (\w+)(ing) $1 see-ing look Removes "ing" from the end of word. see-ing looking (\w+)ing $1 see-ing look Same as above. 2nd parentheses can be omitted. No.1 NO. no. 543 [nN][oO]\.\s*(\d+) #$1 #1 NO. #543 Replace some string literals abc=1234=5678 (\w+)=(\d+)=(\d+) $3=$1=$2 5678=abc=1234 Change the order of the groups. Related Topics CharFilterFactories Language Analysis This section contains information about tokenizers and filters related to character set conversion or for use with specific languages. For the European languages, tokenization is fairly straightforward. Tokens are delimited by white space and/or a relatively small set of punctuation characters. In other languages the tokenization rules are often not so simple. Some European languages may require special tokenization rules as well, such as rules for decompounding German words. Apache Solr Reference Guide 5.1 139 For information about language detection at index time, see Detecting Languages During Indexing. Topics discussed in this section: KeywordMarkerFilterFactory StemmerOverrideFilterFactory Dictionary Compound Word Token Filter Unicode Collation ASCII Folding Filter Language-Specific Factories Related Topics KeywordMarkerFilterFactory Protects words from being modified by stemmers. A customized protected word list may be specified with the "protected" attribute in the schema. Any words in the protected word list will not be modified by any stemmer in Solr. A sample Solr protwords.txt with comments can be found in the sample_techproducts_configs config set directory: StemmerOverrideFilterFactory Overrides stemming algorithms by applying a custom mapping, then protecting these terms from being modified by stemmers. A customized mapping of words to stems, in a tab-separated file, can be specified to the "dictionary" attribute in the schema. Words in this mapping will be stemmed to the stems from the file, and will not be further changed by any stemmer. A sample stemdict.txt with comments can be found in the Source Repository. Dictionary Compound Word Token Filter This filter splits, or decompounds, compound words into individual words using a dictionary of the component words. Each input token is passed through unchanged. If it can also be decompounded into subwords, each subword is also added to the stream at the same logical position. Apache Solr Reference Guide 5.1 140 Compound words are most commonly found in Germanic languages. Factory class: solr.DictionaryCompoundWordTokenFilterFactory Arguments: dictionary: (required) The path of a file that contains a list of simple words, one per line. Blank lines and lines that begin with "#" are ignored. This path may be an absolute path, or path relative to the Solr config directory. minWordSize: (integer, default 5) Any token shorter than this is not decompounded. minSubwordSize: (integer, default 2) Subwords shorter than this are not emitted as tokens. maxSubwordSize: (integer, default 15) Subwords longer than this are not emitted as tokens. onlyLongestMatch: (true/false) If true (the default), only the longest matching subwords will generate new tokens. Example: Assume that germanwords.txt contains at least the following words: dumm kopf donau dampf schiff In: "Donaudampfschiff dummkopf" Tokenizer to Filter: "Donaudampfschiff"(1), "dummkopf"(2), Out: "Donaudampfschiff"(1), "Donau"(1), "dampf"(1), "schiff"(1), "dummkopf"(2), "dumm"(2), "kopf"(2) Unicode Collation Unicode Collation is a language-sensitive method of sorting text that can also be used for advanced search purposes. Unicode Collation in Solr is fast, because all the work is done at index time. Rather than specifying an analyzer within , the solr.Colla tionField and solr.ICUCollationField field type classes provide this functionality. solr.ICUCollatio nField, which is backed by the ICU4J library, provides more flexible configuration, has more locales, is significantly faster, and requires less memory and less index space, since its keys are smaller than those produced by the JDK implementation that backs solr.CollationField. solr.ICUCollationField is included in the Solr analysis-extras contrib - see solr/contrib/analys is-extras/README.txt for instructions on which jars you need to add to your SOLR_HOME/lib in order to use it. solr.ICUCollationField and solr.CollationField fields can be created in two ways: Based upon a system collator associated with a Locale. Based upon a tailored RuleBasedCollator ruleset. Apache Solr Reference Guide 5.1 141 Arguments for solr.ICUCollationField, specified as attributes within the element: Using a System collator: locale: (required) RFC 3066 locale ID. See the ICU locale explorer for a list of supported locales. strength: Valid values are primary, secondary, tertiary, quaternary, or identical. See Comparison Levels in ICU Collation Concepts for more information. decomposition: Valid values are no or canonical. See Normalization in ICU Collation Concepts for more information. Using a Tailored ruleset: custom: (required) Path to a UTF-8 text file containing rules supported by the ICU RuleBasedCollator strength: Valid values are primary, secondary, tertiary, quaternary, or identical. See Comparison Levels in ICU Collation Concepts for more information. decomposition: Valid values are no or canonical. See Normalization in ICU Collation Concepts for more information. Expert options: alternate: Valid values are shifted or non-ignorable. Can be used to ignore punctuation/whitespace. caseLevel: (true/false) If true, in combination with strength="primary", accents are ignored but case is taken into account. The default is false. See CaseLevel in ICU Collation Concepts for more information. caseFirst: Valid values are lower or upper. Useful to control which is sorted first when case is not ignored. numeric: (true/false) If true, digits are sorted according to numeric value, e.g. foobar-9 sorts before foobar-10. The default is false. variableTop: Single character or contraction. Controls what is variable for alternate Sorting Text for a Specific Language In this example, text is sorted according to the default German rules provided by ICU4J. Locales are typically defined as a combination of language and country, but you can specify just the language if you want. For example, if you specify "de" as the language, you will get sorting that works well for the German language. If you specify "de" as the language and "CH" as the country, you will get German sorting specifically tailored for Switzerland. Apache Solr Reference Guide 5.1 142 ... ... In the example above, we defined the strength as "primary". The strength of the collation determines how strict the sort order will be, but it also depends upon the language. For example, in English, "primary" strength ignores differences in case and accents. Another example: ... ... ... The type will be used for the fields where the data contains Polish text. The "secondary" strength will ignore case differences, but, unlike "primary" strength, a letter with diacritic(s) will be sorted differently from the same base letter without diacritics. An example using the "city_sort" field to sort: q=*:*&fl=city&sort=city_sort+asc Sorting Text for Multiple Languages There are two approaches to supporting multiple languages: if there is a small list of languages you wish to support, consider defining collated fields for each language and using copyField. However, adding a large number of sort fields can increase disk and indexing costs. An alternative approach is to use the Unicode defau lt collator. The Unicode default or ROOT locale has rules that are designed to work well for most languages. To use the d efault locale, simply define the locale as the empty string. This Unicode default sort is still significantly more advanced than the standard Solr sort. Apache Solr Reference Guide 5.1 143 Sorting Text with Custom Rules You can define your own set of sorting rules. It's easiest to take existing rules that are close to what you want and customize them. In the example below, we create a custom rule set for German called DIN 5007-2. This rule set treats umlauts in German differently: it treats ö as equivalent to oe, ä as equivalent to ae, and ü as equivalent to ue. For more information, see the ICU RuleBasedCollator javadocs. This example shows how to create a custom rule set for solr.ICUCollationField and dump it to a file: // get the default rules for Germany // these are called DIN 5007-1 sorting RuleBasedCollator baseCollator = (RuleBasedCollator) Collator.getInstance(new ULocale("de", "DE")); // define some tailorings, to make it DIN 5007-2 sorting. // For example, this makes ö equivalent to oe String DIN5007_2_tailorings = "& ae , a\u0308 & AE , A\u0308"+ "& oe , o\u0308 & OE , O\u0308"+ "& ue , u\u0308 & UE , u\u0308"; // concatenate the default rules to the tailorings, and dump it to a String RuleBasedCollator tailoredCollator = new RuleBasedCollator(baseCollator.getRules() + DIN5007_2_tailorings); String tailoredRules = tailoredCollator.getRules(); // write these to a file, be sure to use UTF-8 encoding!!! FileOutputStream os = new FileOutputStream(new File("/solr_home/conf/customRules.dat")); IOUtils.write(tailoredRules, os, "UTF-8"); This rule set can now be used for custom collation in Solr: JDK Collation As mentioned above, ICU Unicode Collation is better in several ways than JDK Collation, but if you cannot use ICU4J for some reason, you can use solr.CollationField. The principles of JDK Collation are the same as those of ICU Collation; you just specify language, country an d variant arguments instead of the combined locale argument. Arguments for solr.CollationField, specified as attributes within the element: Apache Solr Reference Guide 5.1 144 Using a System collator (see Oracle's list of locales supported in Java 7): language: (required) ISO-639 language code country: ISO-3166 country code variant: Vendor or browser-specific code strength: Valid values are primary, secondary, tertiary or identical. See Oracle Java 7 Collator javadocs for more information. decomposition: Valid values are no, canonical, or full. See Oracle Java 7 Collator javadocs for more information. Using a Tailored ruleset: custom: (required) Path to a UTF-8 text file containing rules supported by the JDK RuleBasedCollator strength: Valid values are primary, secondary, tertiary or identical. See Oracle Java 7 Collator javadocs for more information. decomposition: Valid values are no, canonical, or full. See Oracle Java 7 Collator javadocs for more information. A solr.CollationField example: ... ... ASCII Folding Filter This filter converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists. Only those characters with reasonable ASCII alternatives are converted: This can increase recall by causing more matches. On the other hand, it can reduce precision because language-specific character differences may be lost. Factory class: solr.ASCIIFoldingFilterFactory Arguments: None Example: Apache Solr Reference Guide 5.1 145 In: "Björn Ångström" Tokenizer to Filter: "Björn", "Ångström" Out: "Bjorn", "Angstrom" Language-Specific Factories These factories are each designed to work with specific languages. The languages covered here are: Arabic Brazilian Portuguese Bulgarian Catalan Chinese Simplified Chinese CJK Czech Danish Dutch Finnish French Galician German Greek Hebrew, Lao, Myanmar, Khmer Hindi Indonesian Italian Irish Japanese Latvian Norwegian Persian Polish Portuguese Romanian Russian Scandinavian Serbian Spanish Swedish Thai Turkish Arabic Solr provides support for the Light-10 (PDF) stemming algorithm, and Lucene includes an example stopword list. This algorithm defines both character normalization and stemming, so these are split into two filters to provide more flexibility. Apache Solr Reference Guide 5.1 146 Factory classes: solr.ArabicStemFilterFactory, solr.ArabicNormalizationFilterFactory Arguments: None Example: Brazilian Portuguese This is a Java filter written specifically for stemming the Brazilian dialect of the Portuguese language. It uses the Lucene class org.apache.lucene.analysis.br.BrazilianStemmer. Although that stemmer can be configured to use a list of protected words (which should not be stemmed), this factory does not accept any arguments to specify such a list. Factory class: solr.BrazilianStemFilterFactory Arguments: None Example: In: "praia praias" Tokenizer to Filter: "praia", "praias" Out: "pra", "pra" Bulgarian Solr includes a light stemmer for Bulgarian, following this algorithm (PDF), and Lucene includes an example stopword list. Factory class: solr.BulgarianStemFilterFactory Arguments: None Example: Catalan Apache Solr Reference Guide 5.1 147 Solr can stem Catalan using the Snowball Porter Stemmer with an argument of language="Catalan". Solr includes a set of contractions for Catalan, which can be stripped using solr.ElisionFilterFactory. Factory class: solr.SnowballPorterFilterFactory Arguments: language: (required) stemmer language, "Catalan" in this case Example: In: "llengües llengua" Tokenizer to Filter: "llengües"(1) "llengua"(2), Out: "llengu"(1), "llengu"(2) Chinese Chinese Tokenizer The Chinese Tokenizer is deprecated as of Solr 3.4. Use the solr.StandardTokenizerFactory instead. Factory class: solr.ChineseTokenizerFactory Arguments: None Example: Chinese Filter Factory The Chinese Filter Factory is deprecated as of Solr 3.4. Use the solr.StopFilterFactory instead. Factory class: solr.ChineseFilterFactory Arguments: None Example: Apache Solr Reference Guide 5.1 148 Simplified Chinese For Simplified Chinese, Solr provides support for Chinese sentence and word segmentation with the solr.HMMC hineseTokenizerFactory in the analysis-extras contrib module. This component includes a large dictionary and segments Chinese text into words with the Hidden Markov Model. To use this filter, see solr/co ntrib/analysis-extras/README.txt for instructions on which jars you need to add to your solr_home/l ib. Factory class: solr.HMMChineseTokenizerFactory Arguments: None Examples: To use the default setup with fallback to English Porter stemmer for English words, use: Or to configure your own analysis setup, use the solr.HMMChineseTokenizerFactory along with your custom filter setup. CJK This tokenizer breaks Chinese, Japanese and Korean language text into tokens. These are not whitespace delimited languages. The tokens generated by this tokenizer are "doubles", overlapping pairs of CJK characters found in the field text. Factory class: solr.CJKTokenizerFactory Arguments: None Example: Czech Solr includes a light stemmer for Czech, following this algorithm, and Lucene includes an example stopword list. Factory class: solr.CzechStemFilterFactory Arguments: None Example: Apache Solr Reference Guide 5.1 149 In: "prezidenští, prezidenta, prezidentského" Tokenizer to Filter: "prezidenští", "prezidenta", "prezidentského" Out: "preziden", "preziden", "preziden" Danish Solr can stem Danish using the Snowball Porter Stemmer with an argument of language="Danish". Also relevant are the Scandinavian normalization filters. Factory class: solr.SnowballPorterFilterFactory Arguments: language: (required) stemmer language, "Danish" in this case Example: In: "undersøg undersøgelse" Tokenizer to Filter: "undersøg"(1) "undersøgelse"(2), Out: "undersøg"(1), "undersøg"(2) Dutch Solr can stem Dutch using the Snowball Porter Stemmer with an argument of language="Dutch". Factory class: solr.SnowballPorterFilterFactory Arguments: language: (required) stemmer language, "Dutch" in this case Example: Apache Solr Reference Guide 5.1 150 In: "kanaal kanalen" Tokenizer to Filter: "kanaal", "kanalen" Out: "kanal", "kanal" Finnish Solr includes support for stemming Finnish, and Lucene includes an example stopword list. Factory class: solr.FinnishLightStemFilterFactory Arguments: None Example: In: "kala kalat" Tokenizer to Filter: "kala", "kalat" Out: "kala", "kala" French Elision Filter Removes article elisions from a token stream. This filter can be useful for languages such as French, Catalan, Italian, and Irish. Factory class: solr.ElisionFilterFactory Arguments: articles: The pathname of a file that contains a list of articles, one per line, to be stripped. Articles are words such as "le", which are commonly abbreviated, such as in l'avion (the plane). This file should include the abbreviated form, which precedes the apostrophe. In this case, simply " l". If no articles attribute is specified, a default set of French articles is used. ignoreCase: (boolean) If true, the filter ignores the case of words when comparing them to the common word file. Defaults to false Example: In: "L'histoire d'art" Apache Solr Reference Guide 5.1 151 Tokenizer to Filter: "L'histoire", "d'art" Out: "histoire", "art" French Light Stem Filter Solr includes three stemmers for French: one in the solr.SnowballPorterFilterFactory, a lighter stemmer called solr.FrenchLightStemFilterFactory, and an even less aggressive stemmer called solr .FrenchMinimalStemFilterFactory. Lucene includes an example stopword list. Factory classes: solr.FrenchLightStemFilterFactory, solr.FrenchMinimalStemFilterFactory Arguments: None Examples: In: "le chat, les chats" Tokenizer to Filter: "le", "chat", "les", "chats" Out: "le", "chat", "le", "chat" Galician Solr includes a stemmer for Galician following this algorithm, and Lucene includes an example stopword list. Factory class: solr.GalicianStemFilterFactory Arguments: None Example: In: "felizmente Luzes" Apache Solr Reference Guide 5.1 152 Tokenizer to Filter: "felizmente", "luzes" Out: "feliz", "luz" German Solr includes four stemmers for German: one in the solr.SnowballPorterFilterFactory language="German", a stemmer called solr.GermanStemFilterFactory, a lighter stemmer called solr. GermanLightStemFilterFactory, and an even less aggressive stemmer called solr.GermanMinimalSt emFilterFactory. Lucene includes an example stopword list. Factory classes: solr.GermanStemFilterFactory, solr.LightGermanStemFilterFactory, solr.M inimalGermanStemFilterFactory Arguments: None Examples: In: "haus häuser" Tokenizer to Filter: "haus", "häuser" Out: "haus", "haus" Greek This filter converts uppercase letters in the Greek character set to the equivalent lowercase character. Factory class: solr.GreekLowerCaseFilterFactory Arguments: None Use of custom charsets is not longer supported as of Solr 3.1. If you need to index text in these encodings, please use Java's character set conversion facilities (InputStreamReader, and so on.) during I/O, so that Lucene can analyze this text as Unicode instead. Example: Apache Solr Reference Guide 5.1 153 Hindi Solr includes support for stemming Hindi following this algorithm (PDF), support for common spelling differences through the solr.HindiNormalizationFilterFactory, support for encoding differences through the solr .IndicNormalizationFilterFactory following this algorithm, and Lucene includes an example stopword list. Factory classes: solr.IndicNormalizationFilterFactory, solr.HindiNormalizationFilterFac tory, solr.HindiStemFilterFactory Arguments: None Example: Indonesian Solr includes support for stemming Indonesian (Bahasa Indonesia) following this algorithm (PDF), and Lucene includes an example stopword list. Factory class: solr.IndonesianStemFilterFactory Arguments: None Example: In: "sebagai sebagainya" Tokenizer to Filter: "sebagai", "sebagainya" Out: "bagai", "bagai" Italian Solr includes two stemmers for Italian: one in the solr.SnowballPorterFilterFactory language="Italian", and a lighter stemmer called solr.ItalianLightStemFilterFactory. Lucene Apache Solr Reference Guide 5.1 154 includes an example stopword list. Factory class: solr.ItalianStemFilterFactory Arguments: None Example: In: "propaga propagare propagamento" Tokenizer to Filter: "propaga", "propagare", "propagamento" Out: "propag", "propag", "propag" Irish Solr can stem Irish using the Snowball Porter Stemmer with an argument of language="Irish". Solr includes solr.IrishLowerCaseFilterFactory, which can handle Irish-specific constructs. Solr also includes a set of contractions for Irish which can be stripped using solr.ElisionFilterFactory. Factory class: solr.SnowballPorterFilterFactory Arguments: language: (required) stemmer language, "Irish" in this case Example: In: "siopadóireacht síceapatacha b'fhearr m'athair" Tokenizer to Filter: "siopadóireacht", "síceapatacha", "b'fhearr", "m'athair" Out: "siopadóir", "síceapaite", "fearr", "athair" Japanese Solr includes support for analyzing Japanese, via the Lucene Kuromoji morphological analyzer, which includes several analysis components - more details on each below: JapaneseIterationMarkCharFilter normalizes Japanese horizontal iteration marks (odoriji) to their Apache Solr Reference Guide 5.1 155 expanded form. JapaneseTokenizer tokenizes Japanese using morphological analysis, and annotates each term with part-of-speech, base form (a.k.a. lemma), reading and pronunciation. JapaneseBaseFormFilter replaces original terms with their base forms (a.k.a. lemmas). JapanesePartOfSpeechStopFilter removes terms that have one of the configured parts-of-speech. JapaneseKatakanaStemFilter normalizes common katakana spelling variations ending in a long sound character (U+30FC) by removing the long sound character. Also useful for Japanese analysis, from lucene-analyzers-common: CJKWidthFilter folds fullwidth ASCII variants into the equivalent Basic Latin forms, and folds halfwidth Katakana variants into their equivalent fullwidth forms. Japanese Iteration Mark CharFilter Normalizes horizontal Japanese iteration marks (odoriji) to their expanded form. Vertical iteration marks are not supported. Factory class: JapaneseIterationMarkCharFilterFactory Arguments: normalizeKanji: set to false to not normalize kanji iteration marks (default is true) normalizeKana: set to false to not normalize kana iteration marks (default is true) Japanese Tokenizer Tokenizer for Japanese that uses morphological analysis, and annotates each term with part-of-speech, base form (a.k.a. lemma), reading and pronunciation. JapaneseTokenizer has a search mode (the default) that does segmentation useful for search: a heuristic is used to segment compound terms into their constituent parts while also keeping the original compound terms as synonyms. Factory class: solr.JapaneseTokenizerFactory Arguments: mode: Use search mode to get a noun-decompounding effect useful for search. search mode improves segmentation for search at the expense of part-of-speech accuracy. Valid values for mode are: normal: default segmentation search: segmentation useful for search (extra compound splitting) extended: search mode plus unigramming of unknown words (experimental) For some applications it might be good to use search mode for indexing and normal mode for queries to increase precision and prevent parts of compounds from being matched and highlighted. userDictionary: filename for a user dictionary, which allows overriding the statistical model with your own entries for segmentation, part-of-speech tags and readings without a need to specify weights. See lang/userd ict_ja.txt for a sample user dictionary file. userDictionaryEncoding: user dictionary encoding (default is UTF-8) Apache Solr Reference Guide 5.1 156 discardPunctuation: set to false to keep punctuation, true to discard (the default) Japanese Base Form Filter Replaces original terms' text with the corresponding base form (lemma). (JapaneseTokenizer annotates each term with its base form.) Factory class: JapaneseBaseFormFilterFactory (no arguments) Japanese Part Of Speech Stop Filter Removes terms with one of the configured parts-of-speech. JapaneseTokenizer annotates terms with parts-of-speech. Factory class : JapanesePartOfSpeechStopFilterFactory Arguments: tags: filename for a list of parts-of-speech for which to remove terms; see conf/lang/stoptags_ja.txt in the sample_techproducts_config config set for an example. enablePositionIncrements: if luceneMatchVersion is 4.3 or earlier and enablePositionIncrement s="false", no position holes will be left by this filter when it removes tokens. This argument is invalid if luc eneMatchVersion is 5.0 or later. Japanese Katakana Stem Filter Normalizes common katakana spelling variations ending in a long sound character (U+30FC) by removing the long sound character. CJKWidthFilterFactory should be specified prior to this filter to normalize half-width katakana to full-width. Factory class: JapaneseKatakanaStemFilterFactory Arguments: minimumLength: terms below this length will not be stemmed. Default is 4, value must be 2 or more. CJK Width Filter Folds fullwidth ASCII variants into the equivalent Basic Latin forms, and folds halfwidth Katakana variants into their equivalent fullwidth forms. Factory class: CJKWidthFilterFactory (no arguments) Example: Apache Solr Reference Guide 5.1 157 Hebrew, Lao, Myanmar, Khmer Lucene provides support, in addition to UAX#29 word break rules, for Hebrew's use of the double and single quote characters, and for segmenting Lao, Myanmar, and Khmer into syllables with the solr.ICUTokenizerF actory in the analysis-extras contrib module. To use this tokenizer, see solr/contrib/analysis-ext ras/README.txt for instructions on which jars you need to add to your solr_home/lib. See the ICUTokenizer for more information. Latvian Solr includes support for stemming Latvian, and Lucene includes an example stopword list. Factory class: solr.LatvianStemFilterFactory Arguments: None Example: In: "tirgiem tirgus" Tokenizer to Filter: "tirgiem", "tirgus" Out: "tirg", "tirg" Norwegian Solr includes two classes for stemming Norwegian, NorwegianLightStemFilterFactory and NorwegianM inimalStemFilterFactory. Lucene includes an example stopword list. Apache Solr Reference Guide 5.1 158 Another option is to use the Snowball Porter Stemmer with an argument of language="Norwegian". Also relevant are the Scandinavian normalization filters. Norwegian Light Stemmer The NorwegianLightStemFilterFactory requires a "two-pass" sort for the -dom and -het endings. This means that in the first pass the word "kristendom" is stemmed to "kristen", and then all the general rules apply so it will be further stemmed to "krist". The effect of this is that "kristen," "kristendom," "kristendommen," and "kristendommens" will all be stemmed to "krist." The second pass is to pick up -dom and -het endings. Consider this example: One pass Two passes Before After Before After forlegen forleg forlegen forleg forlegenhet forlegen forlegenhet forleg forlegenheten forlegen forlegenheten forleg forlegenhetens forlegen forlegenhetens forleg firkantet firkant firkantet firkant firkantethet firkantet firkantethet firkant firkantetheten firkantet firkantetheten firkant Factory class: solr.NorwegianLightStemFilterFactory Arguments: variant: Choose the Norwegian language variant to use. Valid values are: nb: Bokmål (default) nn: Nynorsk no: both Example: In: "Forelskelsen" Tokenizer to Filter: "forelskelsen" Out: "forelske" Apache Solr Reference Guide 5.1 159 Norwegian Minimal Stemmer The NorwegianMinimalStemFilterFactory stems plural forms of Norwegian nouns only. Factory class: solr.NorwegianMinimalStemFilterFactory Arguments: variant: Choose the Norwegian language variant to use. Valid values are: nb: Bokmål (default) nn: Nynorsk no: both Example: In: "Bilens" Tokenizer to Filter: "bilens" Out: "bil" Persian Persian Filter Factories Solr includes support for normalizing Persian, and Lucene includes an example stopword list. Factory class: solr.PersianNormalizationFilterFactory Arguments: None Example: Polish Solr provides support for Polish stemming with the solr.StempelPolishStemFilterFactory, and solr.M orphologikFilterFactory for lemmatization, in the contrib/analysis-extras module. The solr.Ste mpelPolishStemFilterFactory component includes an algorithmic stemmer with tables for Polish. To use either of these filters, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to add to your solr_home/lib. Apache Solr Reference Guide 5.1 160 Factory class: solr.StempelPolishStemFilterFactory and solr.MorfologikFilterFactory Arguments: None Example: In: ""studenta studenci" Tokenizer to Filter: "studenta", "studenci" Out: "student", "student" More information about the Stempel stemmer is available in the Lucene javadocs. The Morfologik dictionary-resource param value is a constant specifying which dictionary to choose. The dictionary resource must be named morfologik/dictionaries/{dictionaryResource}.dict and have an associated .info metadata file. See the Morfologik project for details. Portuguese Solr includes four stemmers for Portuguese: one in the solr.SnowballPorterFilterFactory, an alternative stemmer called solr.PortugueseStemFilterFactory, a lighter stemmer called solr.Portugu eseLightStemFilterFactory, and an even less aggressive stemmer called solr.PortugueseMinimalS temFilterFactory. Lucene includes an example stopword list. Factory classes: solr.PortugueseStemFilterFactory, solr.PortugueseLightStemFilterFactor y, solr.PortugueseMinimalStemFilterFactory Arguments: None Example: Apache Solr Reference Guide 5.1 161 In: "praia praias" Tokenizer to Filter: "praia", "praias" Out: "pra", "pra" Romanian Solr can stem Romanian using the Snowball Porter Stemmer with an argument of language="Romanian". Factory class: solr.SnowballPorterFilterFactory Arguments: language: (required) stemmer language, "Romanian" in this case Example: Russian Russian Stem Filter Solr includes two stemmers for Russian: one in the solr.SnowballPorterFilterFactory language="Russian", and a lighter stemmer called solr.RussianLightStemFilterFactory. Lucene includes an example stopword list. Factory class: solr.RussianLightStemFilterFactory Arguments: None Use of custom charsets is no longer supported as of Solr 3.4. If you need to index text in these encodings, please use Java's character set conversion facilities (InputStreamReader, and so on.) during I/O, so that Lucene can analyze this text as Unicode instead. Apache Solr Reference Guide 5.1 162 Example: Scandinavian Scandinavian is a language group spanning three languages Norwegian, Swedish and Danish which are very similar. Swedish å,ä,ö are in fact the same letters as Norwegian and Danish å,æ,ø and thus interchangeable when used between these languages. They are however folded differently when people type them on a keyboard lacking these characters. In that situation almost all Swedish people use a, a, o instead of å, ä, ö. Norwegians and Danes on the other hand usually type aa, ae and oe instead of å, æ and ø. Some do however use a, a, o, oo, ao and sometimes permutations of everything above. There are two filters for helping with normalization between Scandinavian languages: one is solr.Scandinavi anNormalizationFilterFactory trying to preserve the special characters (æäöå) and another solr.Scan dinavianFoldingFilterFactory which folds these to the more broad ø/ö->o etc. See also each language section for other relevant filters. Scandinavian Normalization Filter This filter normalize use of the interchangeable Scandinavian characters æÆäÄöÖøØ and folded variants (aa, ao, ae, oe and oo) by transforming them to åÅæÆøØ. It's a semantically less destructive solution than ScandinavianFoldingFilter, most useful when a person with a Norwegian or Danish keyboard queries a Swedish index and vice versa. This filter does not perform the common Swedish folds of å and ä to a nor ö to o. Factory class: solr.ScandinavianNormalizationFilterFactory Arguments: None Example: In: "blåbærsyltetøj blåbärsyltetöj blaabaarsyltetoej blabarsyltetoj" Tokenizer to Filter: "blåbærsyltetøj", "blåbärsyltetöj", "blaabaersyltetoej", "blabarsyltetoj" Out: "blåbærsyltetøj", "blåbærsyltetøj", "blåbærsyltetøj", "blabarsyltetoj" Scandinavian Folding Filter Apache Solr Reference Guide 5.1 163 This filter folds Scandinavian characters åÅäæÄÆ->a and öÖøØ->o. It also discriminate against use of double vowels aa, ae, ao, oe and oo, leaving just the first one. It's is a semantically more destructive solution than ScandinavianNormalizationFilter, but can in addition help with matching raksmorgas as räksmörgås. Factory class: solr.ScandinavianFoldingFilterFactory Arguments: None Example: In: "blåbærsyltetøj blåbärsyltetöj blaabaarsyltetoej blabarsyltetoj" Tokenizer to Filter: "blåbærsyltetøj", "blåbärsyltetöj", "blaabaersyltetoej", "blabarsyltetoj" Out: "blabarsyltetoj", "blabarsyltetoj", "blabarsyltetoj", "blabarsyltetoj" Serbian Serbian Normalization Filter Solr includes a filter that normalizes Serbian Cyrillic and Latin characters to "bald" Latin. Cyrillic characters are first converted to Latin; then, Latin characters have their diacritics removed, with the exception of "" which is converted to "dj". Note that this filter expects lowercased input. Factory class: solr.SerbianNormalizationFilterFactory Arguments: None Example: In: "ura () " Tokenizer to Filter: "ura" "" "" Out: "djura", "djura", "srpski" Spanish Solr includes two stemmers for Spanish: one in the solr.SnowballPorterFilterFactory language="Spanish", and a lighter stemmer called solr.SpanishLightStemFilterFactory. Lucene includes an example stopword list. Apache Solr Reference Guide 5.1 164 Factory class: solr.SpanishStemFilterFactory Arguments: None Example: In: "torear toreara torearlo" Tokenizer to Filter: "torear", "toreara", "torearlo" Out: "tor", "tor", "tor" Swedish Swedish Stem Filter Solr includes two stemmers for Swedish: one in the solr.SnowballPorterFilterFactory language="Swedish", and a lighter stemmer called solr.SwedishLightStemFilterFactory. Lucene includes an example stopword list. Also relevant are the Scandinavian normalization filters. Factory class: solr.SwedishStemFilterFactory Arguments: None Example: In: "kloke klokhet klokheten" Tokenizer to Filter: "kloke", "klokhet", "klokheten" Out: "klok", "klok", "klok" Thai This filter converts sequences of Thai characters into individual Thai words. Unlike European languages, Thai does not use whitespace to delimit words. Factory class: solr.ThaiTokenizerFactory Arguments: None Example: Apache Solr Reference Guide 5.1 165 Turkish Solr includes support for stemming Turkish through the solr.SnowballPorterFilterFactory; support for case-insensitive search through the solr.TurkishLowerCaseFilterFactory; support for stripping apostrophes and following suffixes through solr.ApostropheFilterFactory (see Role of Apostrophes in Turkish Information Retrieval); support for a form of stemming that truncating tokens at a configurable maximum length through the solr.TruncateTokenFilterFactory (see Information Retrieval on Turkish Texts); and Lucene includes an example stopword list. Factory class: solr.TurkishLowerCaseFilterFactory Arguments: None Example: Another example, illustrating diacritics-insensitive search: Related Topics LanguageAnalysis Phonetic Matching Phonetic matching algorithms may be used to encode tokens so that two different spellings that are pronounced similarly will match. For overviews of and comparisons between algorithms, see http://en.wikipedia.org/wiki/Phonetic_algorithm and h ttp://ntz-develop.blogspot.com/2011/03/phonetic-algorithms.html Algorithms discussed in this section: Beider-Morse Phonetic Matching (BMPM) Apache Solr Reference Guide 5.1 166 Daitch-Mokotoff Soundex Double Metaphone Metaphone Soundex Refined Soundex Caverphone Kölner Phonetik a.k.a. Cologne Phonetic NYSIIS Beider-Morse Phonetic Matching (BMPM) To use this encoding in your analyzer, see Beider Morse Filter in the Filter Descriptions section. Beider-Morse Phonetic Matching (BMPM) is a "soundalike" tool that lets you search using a new phonetic matching system. BMPM helps you search for personal names (or just surnames) in a Solr/Lucene index, and is far superior to the existing phonetic codecs, such as regular soundex, metaphone, caverphone, etc. In general, phonetic matching lets you search a name list for names that are phonetically equivalent to the desired name. BMPM is similar to a soundex search in that an exact spelling is not required. Unlike soundex, it does not generate a large quantity of false hits. From the spelling of the name, BMPM attempts to determine the language. It then applies phonetic rules for that particular language to transliterate the name into a phonetic alphabet. If it is not possible to determine the language with a fair degree of certainty, it uses generic phonetic instead. Finally, it applies language-independent rules regarding such things as voiced and unvoiced consonants and vowels to further insure the reliability of the matches. For example, assume that the matches found when searching for Stephen in a database are "Stefan", "Steph", "Stephen", "Steve", "Steven", "Stove", and "Stuffin". "Stefan", "Stephen", and "Steven" are probably relevant, and are names that you want to see. "Stuffin", however, is probably not relevant. Also rejected were "Steph", "Steve", and "Stove". Of those, "Stove" is probably not one that we would have wanted. But "Steph" and "Steve" are possibly ones that you might be interested in. For Solr, BMPM searching is available for the following languages: English French German Greek Hebrew written in Hebrew letters Hungarian Italian Polish Romanian Russian written in Cyrillic letters Russian transliterated into English letters Spanish Turkish The name matching is also applicable to non-Jewish surnames from the countries in which those languages are spoken. Apache Solr Reference Guide 5.1 167 For more information, see here: http://stevemorse.org/phoneticinfo.htm and http://stevemorse.org/phonetics/bmp m.htm. Daitch-Mokotoff Soundex To use this encoding in your analyzer, see Daitch-Mokotoff Soundex Filter in the Filter Descriptions section. The Daitch-Mokotoff Soundex algorithm is a refinement of the Russel and American Soundex algorithms, yielding greater accuracy in matching especially Slavish and Yiddish surnames with similar pronunciation but differences in spelling. The main differences compared to the other soundex variants are: coded names are 6 digits long initial character of the name is coded rules to encoded multi-character n-grams multiple possible encodings for the same name (branching) Note: the implementation used by Solr (commons-codec's DaitchMokotoffSoundex ) has additional branching rules compared to the original description of the algorithm. For more information, see http://en.wikipedia.org/wiki/Daitch%E2%80%93Mokotoff_Soundex and http://www.avo taynu.com/soundex.htm Double Metaphone To use this encoding in your analyzer, see Double Metaphone Filter in the Filter Descriptions section. Alternatively, you may specify encoding="DoubleMetaphone" with the Phonetic Filter, but note that the Phonetic Filter version will not provide the second ("alternate") encoding that is generated by the Double Metaphone Filter for some tokens. Encodes tokens using the double metaphone algorithm by Lawrence Philips. See the original article at http://w ww.drdobbs.com/the-double-metaphone-search-algorithm/184401251?pgno=2 Metaphone To use this encoding in your analyzer, specify encoding="Metaphone" with the Phonetic Filter. Encodes tokens using the Metaphone algorithm by Lawrence Philips, described in "Hanging on the Metaphone" in Computer Language, Dec. 1990. See http://en.wikipedia.org/wiki/Metaphone Soundex To use this encoding in your analyzer, specify encoding="Soundex" with the Phonetic Filter. Encodes tokens using the Soundex algorithm, which is used to relate similar names, but can also be used as a general purpose scheme to find words with similar phonemes. See http://en.wikipedia.org/wiki/Soundex Refined Soundex Apache Solr Reference Guide 5.1 168 To use this encoding in your analyzer, specify encoding="RefinedSoundex" with the Phonetic Filter. Encodes tokens using an improved version of the Soundex algorithm. See http://en.wikipedia.org/wiki/Soundex Caverphone To use this encoding in your analyzer, specify encoding="Caverphone" with the Phonetic Filter. Caverphone is an algorithm created by the Caversham Project at the University of Otago. The algorithm is optimised for accents present in the southern part of the city of Dunedin, New Zealand. See http://en.wikipedia.org/wiki/Caverphone and the Caverphone 2.0 specification at http://caversham.otago.ac. nz/files/working/ctp150804.pdf Kölner Phonetik a.k.a. Cologne Phonetic To use this encoding in your analyzer, specify encoding="ColognePhonetic" with the Phonetic Filter. The Kölner Phonetik, an algorithm published by Hans Joachim Postel in 1969, is optimized for the German language. See http://de.wikipedia.org/wiki/K%C3%B6lner_Phonetik NYSIIS To use this encoding in your analyzer, specify encoding="Nysiis" with the Phonetic Filter. NYSIIS is an encoding used to relate similar names, but can also be used as a general purpose scheme to find words with similar phonemes. See http://en.wikipedia.org/wiki/NYSIIS and http://www.dropby.com/NYSIIS.html Running Your Analyzer Once you've defined a field type in schema.xml and specified the analysis steps that you want applied to it, you should test it out to make sure that it behaves the way you expect it to. Luckily, there is a very handy page in the Solr admin interface that lets you do just that. You can invoke the analyzer for any text field, provide sample input, and display the resulting token stream. For example, let's look at some of the "Text" field types available in the " bin/solr -e techproducts" example configuration, and use the Analysis Screen (http://localhost:8983/solr/#/techproducts/analysis) to compare how the tokens produced at index time for the sentence "Running an Analyzer" match up with a slightly different query text of "run my analyzers" We can begin with "text_ws" - one of the most simplified Text field types available: Apache Solr Reference Guide 5.1 169 We can see very clearly that the only thing this field type does is tokenize text on whitespace. If our objective is to allow queries like "run my analyzer" to match indexed text like "Running an Analyzer" then we will evidently need to pick a different field type with index & query time text analysis that does more processing of the inputs. In particular we will want: Case insensitivity, so "Analyzer" and "analyzer" match. Stemming, so words like "Run" and "Running" are considered equivalent terms. Stop Word Pruning, so small words like "an" and "my" don't affect the query. For our next attempt, let's try the "text_general" field type: With the verbose output enabled, we can see how each stage of our new analyzers modify the tokens they receive before passing them on to the next stage. As we scroll down to the final output, we can see that we do start to get a match on "analyzer" from each input string, thanks to the "LCF" stage -- which if you hover over with your mouse, you'll see is the "LowerCaseFilter": Apache Solr Reference Guide 5.1 170 "text_general" is designed to be generally useful for any language, and it has definitely gotten us closer to our objective then "text_ws" by solving the problem of case sensitivity, by but it's still not quite what we are looking for. So now let us try the "text_en" field type: Now we can see the "SF" (StopFilter) stage of the analyzers solving the problem of removing Stop Words, and as we scroll down, we also see the "PSF" ( PorterStemFilter) stage apply stemming rules suitable for our English language input, such that the terms produced by our "index analyzer" and the terms produced by our "query analyzer" match the way we expect. Apache Solr Reference Guide 5.1 171 At this point, we can continue to experiment with additional inputs, verifying that our analyzers produce matching tokens when we expect them to match, and disparate tokens when we do not expect them to match, as we iterate and tweak our field type configuration. Apache Solr Reference Guide 5.1 172 Indexing and Basic Data Operations This section describes how Solr adds data to its index. It covers the following topics: Introduction to Solr Indexing: An overview of Solr's indexing process. Post Tool: Information about using post.jar to quickly upload some content to your system. Uploading Data with Index Handlers: Information about using Solr's Index Handlers to upload XML/XSLT, JSON and CSV data. Uploading Data with Solr Cell using Apache Tika: Information about using the Solr Cell framework to upload data for indexing. Uploading Structured Data Store Data with the Data Import Handler: Information about uploading and indexing data from a structured data store. Updating Parts of Documents: Information about how to use atomic updates and optimistic concurrency with Solr. Detecting Languages During Indexing: Information about using language identification during the indexing process. De-Duplication: Information about configuring Solr to mark duplicate documents as they are indexed. Content Streams: Information about streaming content to Solr Request Handlers. UIMA Integration: Information about integrating Solr with Apache's Unstructured Information Management Architecture (UIMA). UIMA lets you define custom pipelines of Analysis Engines that incrementally add metadata to your documents as annotations. Indexing Using Client APIs Using client APIs, such as SolrJ, from your applications is an important option for updating Solr indexes. See the Client APIs section for more information. Introduction to Solr Indexing This section describes the process of indexing: adding content to a Solr index and, if necessary, modifying that content or deleting it. By adding content to an index, we make it searchable by Solr. A Solr index can accept data from many different sources, including XML files, comma-separated value (CSV) files, data extracted from tables in a database, and files in common file formats such as Microsoft Word or PDF. Here are the three most common ways of loading data into a Solr index: Using the Solr Cell framework built on Apache Tika for ingesting binary files or structured files such as Office, Word, PDF, and other proprietary formats. Uploading XML files by sending HTTP requests to the Solr server from any environment where such requests can be generated. Writing a custom Java application to ingest data through Solr's Java Client API (which is described in more detail in Client APIs. Using the Java API may be the best choice if you're working with an application, such as a Content Management System (CMS), that offers a Java API. Apache Solr Reference Guide 5.1 173 Regardless of the method used to ingest data, there is a common basic data structure for data being fed into a Solr index: a document containing multiple fields, each with a name and containing content, which may be empty. One of the fields is usually designated as a unique ID field (analogous to a primary key in a database), although the use of a unique ID field is not strictly required by Solr. If the field name is defined in the schema.xml file that is associated with the index, then the analysis steps associated with that field will be applied to its content when the content is tokenized. Fields that are not explicitly defined in the schema will either be ignored or mapped to a dynamic field definition (see Documents, Fields, and Schema Design), if one matching the field name exists. For more information on indexing in Solr, see the Solr Wiki. The Solr Example Directory When starting Solr with the "-e" option, the example/ directory will be used as base directory for the example Solr instances that are created. This directory also includes an example/exampledocs/ subdirectory containing sample documents in a variety of formats that you can use to experiment with indexing into the various examples. The curl Utility for Transferring Files Many of the instructions and examples in this section make use of the curl utility for transferring content through a URL. curl posts and retrieves data over HTTP, FTP, and many other protocols. Most Linux distributions include a copy of curl. You'll find curl downloads for Linux, Windows, and many other operating systems at http://curl.haxx.se/download.html. Documentation for curl is available here: http://curl.haxx.se/docs/ manpage.html. Using curl or other command line tools for posting data is just fine for examples or tests, but it's not the recommended method for achieving the best performance for updates in production environments. You will achieve better performance with Solr Cell or the other methods described in this section. Instead of curl, you can use utilities such as GNU wget (http://www.gnu.org/software/wget/) or manage GETs and POSTS with Perl, although the command line options will differ. Post Tool Solr includes a simple command line tool for POSTing various types of content to a Solr server. The tool is bin/ post. The bin/post tool is a Unix shell script; for Windows (non-Cygwin) usage, see the Windows section below. To run it, open a window and enter: bin/post -c gettingstarted example/films/films.json This will contact the server at localhost:8983. Specifying the collection/core name is mandatory. The '-help' (or simply '-h') option will output information on its usage (i.e., bin/post -help). Using the bin/post Tool Specifying either the collection/core name or the full update url is mandatory when using bin/post. Apache Solr Reference Guide 5.1 174 The basic usage of bin/post is: $ bin/post -help Usage: post -c [OPTIONS] or post -help collection name defaults to DEFAULT_SOLR_COLLECTION if not specified OPTIONS ======= Solr options: -url (overrides collection, host, and port) -host (default: localhost) -port (default: 8983) -commit yes|no (default: yes) Web crawl options: -recursive (default: 1) -delay (default: 10) Directory crawl options: -delay (default: 0) stdin/args options: -type (default: application/xml) Other options: -filetypes [,,...] (default: xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt ,log) -params "=[&=...]" (values must be URL-encoded; these pass through to Solr update request) -out yes|no (default: no; yes outputs Solr response to console) ... Examples There are several ways to use bin/post. This section presents several examples. Indexing XML Add all documents with file extension .xml to collection or core named gettingstarted. bin/post -c gettingstarted *.xml Add all documents with file extension .xml to the gettingstarted collection/core on Solr running on port 898 4. bin/post -c gettingstarted -port 8984 *.xml Send XML arguments to delete a document from gettingstarted. bin/post -c gettingstarted -d '42' Indexing CSV Apache Solr Reference Guide 5.1 175 Index all CSV files into gettingstarted: bin/post -c gettingstarted *.csv Index a tab-separated file into gettingstarted: bin/post -c signals -params "separator=%09" -type text/csv data.tsv The content type (-type) parameter is required to treat the file as the proper type, otherwise it will be ignored and a WARNING logged as it does not know what type of content a .tsv file is. The CSV handler supports the se parator parameter, and is passed through using the -params setting. Indexing JSON Index all JSON files into gettingstarted. bin/post -c gettingstarted *.json Indexing rich documents (PDF, Word, HTML, etc) Index a PDF file into gettingstarted. bin/post -c gettingstarted a.pdf Automatically detect content types in a folder, and recursively scan it for documents for indexing into gettingst arted. bin/post -c gettingstarted afolder/ Automatically detect content types in a folder, but limit it to PPT and HTML files and index into gettingstarte d. bin/post -c gettingstarted -filetypes ppt,html afolder/ Windows support bin/post exists currently only as a Unix shell script, however it delegates its work to a cross-platform capable Java program. The SimplePostTool can be run directly in supported environments, including Windows. SimplePostTool The bin/post script currently delegates to a standalone Java program called SimplePostTool. This tool, bundled into a executable JAR, can be run directly using java -jar example/exampledocs/post.jar. See the help output and take it from there to post files, recurse a website or file system folder, or send direct commands to a Solr server. Apache Solr Reference Guide 5.1 176 $ java -jar example/exampledocs/post.jar -h SimplePostTool version 5.0.0 Usage: java [SystemProperties] -jar post.jar [-h|-] [ [...]] . . . Uploading Data with Index Handlers Index Handlers are Request Handlers designed to add, delete and update documents to the index. In addition to having plugins for importing rich documents using Tika or from structured data sources using the Data Import Handler, Solr natively supports indexing structured documents in XML, CSV and JSON. The recommended way to configure and use request handlers is with path based names that map to paths in the request url. However, request handlers can also be specified with the qt (query type) parameter if the request Dispatcher is appropriately configured. It is possible to access the same handler using more than one name, which can be useful if you wish to specify different sets of default options. A single unified update request handler supports XML, CSV, JSON, and javabin update requests, delegating to the appropriate ContentStreamLoader based on the Content-Type of the ContentStream. Topics covered in this section: UpdateRequestHandler Configuration XML Formatted Index Updates Adding Documents XML Update Commands Using curl to Perform Updates Using XSLT to Transform XML Index Updates JSON Formatted Index Updates Solr-Style JSON JSON Update Convenience Paths Transforming and Indexing Custom JSON CSV Formatted Index Updates CSV Update Parameters Indexing Tab-Delimited files CSV Update Convenience Paths Nested Child Documents UpdateRequestHandler Configuration The default configuration file has the update request handler configured by default. XML Formatted Index Updates Apache Solr Reference Guide 5.1 177 Index update commands can be sent as XML message to the update handler using Content-type: application/xml or Content-type: text/xml. Adding Documents The XML schema recognized by the update handler for adding documents is very straightforward: The element introduces one more documents to be added. The element introduces the fields making up a document. The element presents the content for a specific field. For example: Patrick Eagar Sports 796.35 128 12.40 Summer of the all-rounder: Test and championship cricket in England 1982 0002166313 1982 Collins ... Each element has certain optional attributes which may be specified. Command Optional Parameter Parameter Description commitWithin= number Add the document within the specified number of milliseconds overwrite=bool ean Default is true. Indicates if the unique key constraints should be checked to overwrite previous versions of the same document (see below) boost=float Default is 1.0. Sets a boost value for the document.To learn more about boosting, see Searching. boost=float Default is 1.0. Sets a boost value for the field. If the document schema defines a unique key, then by default an /update operation to add a document will overwrite (ie: replace) any document in the index with the same unique key. If no unique key has been defined, indexing performance is somewhat faster, as no check has to be made for an existing documents to replace. If you have a unique key field, but you feel confident that you can safely bypass the uniqueness check (eg: you build your indexes in batch, and your indexing code guarantees it never adds the same document more then once) you can specify the overwrite="false" option when adding your documents. Apache Solr Reference Guide 5.1 178 XML Update Commands Commit and Optimize Operations The operation writes all documents loaded since the last commit to one or more segment files on the disk. Before a commit has been issued, newly indexed content is not visible to searches. The commit operation opens a new searcher, and triggers any event listeners that have been configured. Commits may be issued explicitly with a message, and can also be triggered from parameters in solrconfig.xml. The operation requests Solr to merge internal data structures in order to improve search performance. For a large index, optimization will take some time to complete, but by merging many small segment files into a larger one, search performance will improve. If you are using Solr's replication mechanism to distribute searches across many systems, be aware that after an optimize, a complete index will need to be transferred. In contrast, post-commit transfers are usually much smaller. The and elements accept these optional attributes: Optional Attribute Description waitSearcher Default is true. Blocks until a new searcher is opened and registered as the main query searcher, making the changes visible. expungeDeletes (commit only) Default is false. Merges segments that have more than 10% deleted docs, expunging them in the process. maxSegments (optimize only) Default is 1. Merges the segments down to no more than this number of segments. Here are examples of and using optional attributes: Delete Operations Documents can be deleted from the index in two ways. "Delete by ID" deletes the document with the specified ID, and can be used only if a UniqueID field has been defined in the schema. "Delete by Query" deletes all documents matching a specified query, although commitWithin is ignored for a Delete by Query. A single delete message can contain multiple delete operations. 0002166313 0031745983 subject:sport publisher:penguin Rollback Operations Apache Solr Reference Guide 5.1 179 The rollback command rolls back all add and deletes made to the index since the last commit. It neither calls any event listeners nor creates a new searcher. Its syntax is simple: . Using curl to Perform Updates You can use the curl utility to perform any of the above commands, using its --data-binary option to append the XML message to the curl command, and generating a HTTP POST request. For example: curl http://localhost:8983/solr/my_collection/update -H "Content-Type: text/xml" --data-binary ' Patrick Eagar Sports 796.35 0002166313 1982 Collins ' For posting XML messages contained in a file, you can use the alternative form: curl http://localhost:8983/solr/my_collection/update -H "Content-Type: text/xml" --data-binary @myfile.xml Short requests can also be sent using a HTTP GET command, URL-encoding the request, as in the following. Note the escaping of "<" and ">": curl http://localhost:8983/solr/my_collection/update?stream.body=%3Ccommit/%3E Responses from Solr take the form shown here: 0 127 The status field will be non-zero in case of failure. Using XSLT to Transform XML Index Updates The UpdateRequestHandler allows you to index any arbitrary XML using the parameter to apply an XSL transformation. You must have an XSLT stylesheet in the conf/xslt directory of your config set that can transform the incoming data to the expected format, and use the tr parameter to specify the name of that stylesheet. Here is an example XSLT stylesheet: Apache Solr Reference Guide 5.1 180 This stylesheet transforms Solr's XML search result format into Solr's Update XML syntax. One example usage would be to copy a Solr 1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file (provided that all fields are stored): http://localhost:8983/solr/my_collection/select?q=*:*&wt=xslt&tr=updateXml.xsl&rows= 1000 You can also use the stylesheet in XsltUpdateRequestHandler to transform an index when updating: curl "http://localhost:8983/solr/my_collection/update?commit=true&tr=updateXml.xsl" -H "Content-Type: text/xml" --data-binary @myexporteddata.xml For more information about the XML Update Request Handler, see https://wiki.apache.org/solr/UpdateXmlMessa ges. Apache Solr Reference Guide 5.1 181 JSON Formatted Index Updates Solr can accept JSON that conforms to a defined structure, or can accept arbitrary JSON-formatted documents. If sending arbitrarily formatted JSON, there are some additional parameters that need to be sent with the update request, described below in the section Transforming and Indexing Custom JSON. Solr-Style JSON JSON formatted update requests may be sent to Solr's /update handler using Content-Type: application/json or Content-Type: text/json. JSON formatted updates can take 3 basic forms, described in depth below: A single document to add, expressed as a top level JSON Object. To differentiate this from a set of commands, the json.command=false request parameter is required. A list of documents to add, expressed as a top level JSON Array containing a JSON Object per document. A sequence of update commands, expressed as a top level JSON Object (aka: Map). Adding a Single JSON Document The simplest way to add Documents via JSON is to send each document individually as a JSON Object, using the /update/json/docs path: curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/my_collection/update/json/docs' --data-binary ' { "id": "1", "title": "Doc 1" }' Adding Multiple JSON Documents Adding multiple documents at one time via JSON can be done via a JSON Array of JSON Objects, where each object represents a document: curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/my_collection/update' --data-binary ' [ { "id": "1", "title": "Doc 1" }, { "id": "2", "title": "Doc 2" } ]' A sample JSON file is provided at example/exampledocs/books.json and contains an array of objects that you can add to the Solr techproducts example: Apache Solr Reference Guide 5.1 182 curl 'http://localhost:8983/solr/techproducts/update?commit=true' --data-binary @example/exampledocs/books.json -H 'Content-type:application/json' Sending JSON Update Commands In general, the JSON update syntax supports accepts all of the update commands that the XML update handler supports, through a straightforward mapping. Multiple commands, adding and deleting documents, may be contained in one message: curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/my_collection/update' --data-binary ' { "add": { "doc": { "id": "DOC1", "my_boosted_field": { /* use a map with boost/value for a boosted field */ "boost": 2.3, "value": "test" }, "my_multivalued_field": [ "aaa", "bbb" ] /* Can use an array for a multi-valued field */ } }, "add": { "commitWithin": 5000, /* commit this document within 5 seconds */ "overwrite": false, /* don't check for existing documents with the same uniqueKey */ "boost": 3.45, /* a document boost */ "doc": { "f1": "v1", /* Can use repeated keys for a multi-valued field */ "f1": "v2" } }, "commit": {}, "optimize": { "waitSearcher":false }, "delete": { "id":"ID" }, "delete": { "query":"QUERY" } /* delete by ID */ /* delete by query */ }' Comments are not allowed in JSON, but duplicate names are. The comments in the above example are for illustrative purposes only, and can not be included in actual commands sent to Solr. As with other update handlers, parameters such as commit, commitWithin, optimize, and overwrite may be specified in the URL instead of in the body of the message. The JSON update format allows for a simple delete-by-id. The value of a delete can be an array which contains a list of zero or more specific document id's (not a range) to be deleted. For example, a single document: Apache Solr Reference Guide 5.1 183 { "delete":"myid" } Or a list of document IDs: { "delete":["id1","id2"] } The value of a "delete" can be an array which contains a list of zero or more id's to be deleted. It is not a range (start and end). You can also specify _version_ with each "delete": { "delete":"id":50, "_version_":12345 } You can specify the version of deletes in the body of the update request as well. JSON Update Convenience Paths In addition to the /update handler, there are a few additional JSON specific request handler paths available by default in Solr, that implicitly override the behavior of some request parameters: Path Default Parameters /update/json stream.contentType=application/json /update/json/docs stream.contentType=application/json json.command=false The /update/json path may be useful for clients sending in JSON formatted update commands from applications where setting the Content-Type proves difficult, while the /update/json/docs path can be particularly convenient for clients that always want to send in documents – either individually or as a list – with out needing to worry about the full JSON command syntax. Transforming and Indexing Custom JSON If you have JSON documents that you would like to index without transforming them into Solr's structure, you can add them to Solr by including some parameters with the update request. These parameters provide information on how to split a single JSON file into multiple Solr documents and how to map fields to Solr's schema. One or more valid JSON documents can be sent to the /update/json/docs path with the configuration params. Mapping Parameters These parameters allow you to define how a JSON file should be read for multiple Solr documents. split: Defines the path at which to split the input JSON into multiple Solr documents and is required if you have multiple documents in a single JSON file. If the entire JSON makes a single solr document, the path must be “/”. f: This is a multivalued mapping parameter. At least one field mapping must be provided. The format of Apache Solr Reference Guide 5.1 184 the parameter is target-field-name:json-path. The json-path is required. The target-field -name is the Solr document field name, and is optional. If not specified, it is automatically derived from the input JSON. Wildcards can be used here, see the section Wildcards below for more information. mapUniqueKeyOnly (boolean): This parameter is particularly convenient when the fields in the input JSON are not available in the schema and schemaless mode is not enabled. This will index all the fields into the default search field (using the df parameter, below) and only the uniqueKey field is mapped to the corresponding field in the schema. If the input JSON does not have a value for the uniqueKey field then a UUID is generated for the same. df: If the mapUniqueKeyOnly flag is used, the update handler needs a field where the data should be indexed to. This is the same field that other handlers use as a default search field. srcField: This is the name of the field to which the JSON source will be stored into. This can only be used if split=/ (i.e., you want your JSON input file to be indexed as a single Solr document). Note that atomic updates will cause the field to be out-of-sync with the document. echo: This is for debugging purpose only. Set it to true if you want the docs to be returned as a response. Nothing will be indexed. For example, if we have a JSON file that includes two documents, we could define an update request like this: curl 'http://localhost:8983/solr/my_collection/update/json/docs'\ '?split=/exams'\ '&f=first:/first'\ '&f=last:/last'\ '&f=grade:/grade'\ '&f=subject:/exams/subject'\ '&f=test:/exams/test'\ '&f=marks:/exams/marks'\ -H 'Content-type:application/json' -d ' { "first": "John", "last": "Doe", "grade": 8, "exams": [ { "subject": "Maths", "test" : "term1", "marks" : 90}, { "subject": "Biology", "test" : "term1", "marks" : 86} ] }' With this request, we have defined that "exams" contains multiple documents. In addition, we have mapped several fields from the input document to Solr fields. When the update request is complete, the following two documents will be added to the index: Apache Solr Reference Guide 5.1 185 { "first":"John", "last":"Doe", "marks":90, "test":"term1", "subject":"Maths", "grade":8 } { "first":"John", "last":"Doe", "marks":86, "test":"term1", "subject":"Biology", "grade":8 } In the prior example, all of the fields we wanted to use in Solr had the same names as they did in the input JSON. When that is the case, we can simplify the request as follows: curl 'http://localhost:8983/solr/my_collection/update/json/docs'\ '?split=/exams'\ '&f=/first'\ '&f=/last'\ '&f=/grade'\ '&f=/exams/subject'\ '&f=/exams/test'\ '&f=/exams/marks'\ -H 'Content-type:application/json' -d ' { "first": "John", "last": "Doe", "grade": 8, "exams": [ { "subject": "Maths", "test" : "term1", "marks" : 90}, { "subject": "Biology", "test" : "term1", "marks" : 86} ] }' In this example, we simply named the field paths (such as /exams/test). Solr will automatically attempt to add the content of the field from the JSON input to the index in a field with the same name. Note that if you are not working in Schemaless Mode, where fields that don't exist will be created on the fly with Solr's best guess for the field type, documents may get rejected if the fields do not exist in the schema before indexing. Wildcards Apache Solr Reference Guide 5.1 186 Instead of specifying all the field names explicitly, it is possible to specify wildcards to map fields automatically. There are two restrictions: wildcards can only be used at the end of the json-path, and the split path cannot use wildcards. A single asterisk "*" maps only to direct children, and a double asterisk "**" maps recursively to all descendants. The following are example wildcard path mappings: f=/docs/*: maps all the fields under docs and in the name as given in json f=/docs/**: maps all the fields under docs and its children in the name as given in json f=searchField:/docs/*: maps all fields under /docs to a single field called ‘searchField’ f=searchField:/docs/**: maps all fields under /docs and its children to searchField f=$FQN:/**: maps all fields to the fully qualified name ($FQN) of the JSON field. The fully qualified name is obtained by concatenating all the keys in the hierarchy with a period ( .) as a delimiter. (Note: The default value of f is '$FQN:/**' fom Solr 5.0 . It used to be '/**' in 4.10.x releases. This breaks backward compatibility. If you wish to have the old behavior please specify f=/** explicitly.) With wildcards we can further simplify our previous example as follows: curl 'http://localhost:8983/solr/my_collection/update/json/docs'\ '?split=/exams'\ '&f=/**'\ -H 'Content-type:application/json' -d ' { "first": "John", "last": "Doe", "grade": 8, "exams": [ { "subject": "Maths", "test" : "term1", "marks" : 90}, { "subject": "Biology", "test" : "term1", "marks" : 86} ] }' Because we want the fields to be indexed with the field names as they are found in the JSON input, the double wildcard in f=/** will map all fields and their descendants to the same fields in Solr. It is also possible to send all the values to a single field and do a full text search on that. This is a good option to blindly index and query JSON documents without worrying about fields and schema. Apache Solr Reference Guide 5.1 187 curl 'http://localhost:8983/solr/my_collection/update/json/docs'\ '?split=/'\ '&f=txt:/**'\ -H 'Content-type:application/json' -d ' { "first": "John", "last": "Doe", "grade": 8, "exams": [ { "subject": "Maths", "test" : "term1", "marks" : 90}, { "subject": "Biology", "test" : "term1", "marks" : 86} ] }' In the above example, we've said all of the fields should be added to a field in Solr named 'txt'. This will add multiple fields to a single field, so whatever field you choose should be multi-valued. The default behavior is to use the fully qualified name (FQN) of the node. So, if we don't define any field mappings, like this: curl 'http://localhost:8983/solr/my_collection/update/json/docs?split=/exams'\ -H 'Content-type:application/json' -d ' { "first": "John", "last": "Doe", "grade": 8, "exams": [ { "subject": "Maths", "test" : "term1", "marks" : 90}, { "subject": "Biology", "test" : "term1", "marks" : 86} ] }' The indexed documents would be added to the index with fields that look like this: Apache Solr Reference Guide 5.1 188 { "first":"John", "last":"Doe", "grade":8, "exams.subject":"Maths", "exams.test":"term1", "exams.marks":90}, { "first":"John", "last":"Doe", "grade":8, "exams.subject":"Biology", "exams.test":"term1", "exams.marks":86} Setting JSON Defaults If It is possible to send any json to the /update/json/docs endpoint and the default configuration of the component is as follows: _src_ true text So, if no params are passed, the entire json file would get indexed to the _src_ field and all the values in the input JSON would go to a field named text. If there is a value for the uniqueKey it is stored and if no value could be obtained from the input JSON, a UUID is created and used as the uniqueKey field value. CSV Formatted Index Updates CSV formatted update requests may be sent to Solr's /update handler using Content-Type: application/csv or Content-Type: text/csv. A sample CSV file is provided at example/exampledocs/books.csv that you can use to add some documents to the Solr techproducts example: curl 'http://localhost:8983/solr/techproducts/update?commit=true' --data-binary @example/exampledocs/books.csv -H 'Content-type:application/csv' Apache Solr Reference Guide 5.1 189 CSV Update Parameters The CSV handler allows the specification of many parameters in the URL in the form: f.parameter.optional _fieldname=value. The table below describes the parameters for the update handler. Parameter Usage Global (g) or Per Field (f) Example separator Character used as field separator; default is "," g,(f: see split) separator=% trim If true, remove leading and trailing whitespace from values. Default=false. g,f f.isbn.trim=true trim=false header Set to true if first line of input contains field names. These will be used if the fieldnames pa rameter is absent. g fieldnames Comma separated list of field names to use when adding documents. g fieldnames=isbn,price,title literal. Comma separated list of field names to use when processing literal values. g literal.color=red,blue,black skip Comma separated list of field names to skip. g skip=uninteresting,shoesize skipLines Number of lines to discard in the input stream before the CSV data starts, including the header, if present. Default=0. g skipLines=5 encapsulator The character optionally used to surround values to preserve characters such as the CSV separator or whitespace. This standard CSV format handles the encapsulator itself appearing in an encapsulated value by doubling the encapsulator. g,(f: see split) encapsulator=" escape The character used for escaping CSV separators or other reserved characters. If an escape is specified, the encapsulator is not used unless also explicitly specified since most formats use either encapsulation or escaping, not both g escape=\ keepEmpty Keep and index zero length (empty) fields. Default=false. g,f f.price.keepEmpty=true Apache Solr Reference Guide 5.1 190 map Map one value to another. Format is value:replacement (which can be empty.) g,f map=left:right f.subject.map=history:bunk split If true, split a field into multiple values by a separate parser. f overwrite If true (the default), check for and overwrite duplicate documents, based on the uniqueKey field declared in the Solr schema. If you know the documents you are indexing do not contain any duplicates then you may see a considerable speed up setting this to false. g commit Issues a commit after the data has been ingested. g commitWithin Add the document within the specified number of milliseconds. g commitWithin=10000 rowid Map the rowid (line number) to a field specified by the value of the parameter, for instance if your CSV doesn't have a unique key and you want to use the row id as such. g rowid=id rowidOffset Add the given offset (as an int) to the rowid before adding it to the document. Default is 0 g rowidOffset=10 Indexing Tab-Delimited files The same feature used to index CSV documents can also be easily used to index tab-delimited files (TSV files) and even handle backslash escaping rather than CSV encapsulation. For example, one can dump a MySQL table to a tab delimited file with: SELECT * INTO OUTFILE '/tmp/result.txt' FROM mytable; This file could then be imported into Solr by setting the separator to tab (%09) and the escape to backslash (%5c). curl 'http://localhost:8983/solr/update/csv?commit=true&separator=%09&escape=%5c' --data-binary @/tmp/result.txt CSV Update Convenience Paths In addition to the /update handler, there is an additional CSV specific request handler path available by default in Solr, that implicitly override the behavior of some request parameters: Path Default Parameters /update/csv stream.contentType=application/csv The /update/csv path may be useful for clients sending in CSV formatted update commands from applications Apache Solr Reference Guide 5.1 191 where setting the Content-Type proves difficult. For more information on the CSV Update Request Handler, see https://wiki.apache.org/solr/UpdateCSV. Nested Child Documents Solr nested documents using a "Block Join" when indexing as a way to model documents containing other documents, such as a blog post parent document and comments as child documents -- or products as parent documents and sizes, colors, or other variations as child documents. At query time, the Block Join Query Parsers can be used search against these relationships. In terms of performance, indexing the relationships between documents may be more efficient than attempting to do joins only at query time, since the relationships are already stored in the index and do not need to be computed. Nested documents may be indexed via either the XML or JSON data syntax (or using SolrJ) - but regardless of syntax, you must include a field that identifies the parent document as a parent; it can be any field that suits this purpose, and it will be used as input for the block join query parsers. XML Examples For example, here are two documents and their child documents: 1 Solr adds block join support parentDocument 2 SolrCloud supports it too! 3 New Lucene and Solr release is out parentDocument 4 Lots of new features In this example, we have indexed the parent documents with the field content_type, which has the value "parentDocument". We could have also used a boolean field, such as isParent, with a value of "true", or any other similar approach. JSON Examples This example is equivalent to the XML example above, note the special _childDocuments_ key need to indicate the nested documents in JSON. Apache Solr Reference Guide 5.1 192 [ { "id": "1", "title": "Solr adds block join support", "content_type": "parentDocument", "_childDocuments_": [ { "id": "2", "comments": "SolrCloud supports it too!" } ] }, { "id": "3", "title": "New Lucene and Solr release is out", "content_type": "parentDocument", "_childDocuments_": [ { "id": "4", "comments": "Lots of new features" } ] } ] Note One limitation of indexing nested documents is that the whole block of parent-children documents must be updated together whenever any changes are required. In other words, even if a single child document or the parent document is changed, the whole block of parent-child documents must be indexed together. Uploading Data with Solr Cell using Apache Tika Solr uses code from the Apache Tika project to provide a framework for incorporating many different file-format parsers such as Apache PDFBox and Apache POI into Solr itself. Working with this framework, Solr's Extracti ngRequestHandler can use Tika to support uploading binary files, including files in popular formats such as Word and PDF, for data extraction and indexing. When this framework was under development, it was called the Solr Content Extraction Library or CEL; from that abbreviation came this framework's name: Solr Cell. If you want to supply your own ContentHandler for Solr to use, you can extend the ExtractingRequestHan dler and override the createFactory() method. This factory is responsible for constructing the SolrConten tHandler that interacts with Tika, and allows literals to override Tika-parsed values. Set the parameter litera lsOverride, which normally defaults to *true, to *false" to append Tika-parsed values to literal values. For more information on Solr's Extracting Request Handler, see https://wiki.apache.org/solr/ExtractingRequestH Apache Solr Reference Guide 5.1 193 andler. Topics covered in this section: Key Concepts Trying out Tika with the Solr techproducts Example Input Parameters Order of Operations Configuring the Solr ExtractingRequestHandler Indexing Encrypted Documents with the ExtractingUpdateRequestHandler Examples Sending Documents to Solr with a POST Sending Documents to Solr with Solr Cell and SolrJ Related Topics Key Concepts When using the Solr Cell framework, it is helpful to keep the following in mind: Tika will automatically attempt to determine the input document type (Word, PDF, HTML) and extract the content appropriately. If you like, you can explicitly specify a MIME type for Tika with the stream.type p arameter. Tika works by producing an XHTML stream that it feeds to a SAX ContentHandler. SAX is a common interface implemented for many different XML parsers. For more information, see http://www.saxproject.or g/quickstart.html. Solr then responds to Tika's SAX events and creates the fields to index. Tika produces metadata such as Title, Subject, and Author according to specifications such as the DublinCore. See http://tika.apache.org/1.7/formats.html for the file types supported. Tika adds all the extracted text to the content field. You can map Tika's metadata fields to Solr fields. You can also boost these fields. You can pass in literals for field values. Literals will override Tika-parsed values, including fields in the Tika metadata object, the Tika content field, and any "captured content" fields. You can apply an XPath expression to the Tika XHTML to restrict the content that is produced. While Apache Tika is quite powerful, it is not perfect and fails on some files. PDF files are particularly problematic, mostly due to the PDF format itself. In case of a failure processing any file, the Extractin gRequestHandler does not have a secondary mechanism to try to extract some text from the file; it will throw an exception and fail. Trying out Tika with the Solr techproducts Example You can try out the Tika framework using the techproducts example included in Solr. Start the example: bin/solr -e techproducts You can now use curl to send a sample PDF file via HTTP POST: Apache Solr Reference Guide 5.1 194 curl 'http://localhost:8983/solr/techproducts/update/extract?literal.id=doc1&commit=true' -F "myfile=@example/exampledocs/solr-word.pdf" The URL above calls the Extracting Request Handler, uploads the file solr-word.pdf and assigns it the unique ID doc1. Here's a closer look at the components of this command: The literal.id=doc1 parameter provides the necessary unique ID for the document being indexed. The commit=true parameter causes Solr to perform a commit after indexing the document, making it immediately searchable. For optimum performance when loading many documents, don't call the commit command until you are done. The -F flag instructs curl to POST data using the Content-Type multipart/form-data and supports the uploading of binary files. The @ symbol instructs curl to upload the attached file. The argument myfile=@tutorial.html needs a valid path, which can be absolute or relative. You can also use bin/post to send a PDF file into Solr (without the params, the literal.id parameter would be set to the absolute path to the file): bin/post -c techproducts example/exampledocs/solr-word.pdf -params "literal.id=a" Now you should be able to execute a query and find that document. You can make a request like http://loc alhost:8983/solr/techproducts/select?q=pdf . You may notice that although the content of the sample document has been indexed and stored, there are not a lot of metadata fields associated with this document. This is because unknown fields are ignored according to the default parameters configured for the /update/extract handler in solrconfig.xml, and this behavior can be easily changed or overridden. For example, to store and see all metadata and content, execute the following: bin/post -c techproducts example/exampledocs/solr-word.pdf -params "literal.id=doc1&uprefix=attr_" In this command, the uprefix=attr_ parameter causes all generated fields that aren't defined in the schema to be prefixed with attr_, which is a dynamic field that is stored and indexed. This command allows you to query the document using an attribute, as in: http://localhost:8983/solr/t echproducts/select?q=attr_meta:microsoft. Input Parameters The table below describes the parameters accepted by the Extracting Request Handler. Parameter boost. Description Boosts the specified field by the defined float amount. (Boosting a field alters its importance in a query response. To learn about boosting fields, see Searching.) Apache Solr Reference Guide 5.1 195 capture Captures XHTML elements with the specified name for a supplementary addition to the Solr document. This parameter can be useful for copying chunks of the XHTML into a separate field. For instance, it could be used to grab paragraphs (< p>) and index them into a separate field. Note that content is still also captured into the overall "content" field. captureAttr Indexes attributes of the Tika XHTML elements into separate fields, named after the element. If set to true, for example, when extracting from HTML, Tika can return the href attributes in tags as fields named "a". See the examples below. commitWithin Add the document within the specified number of milliseconds. date.formats Defines the date format patterns to identify in the documents. defaultField If the uprefix parameter (see below) is not specified and a field cannot be determined, the default field will be used. extractOnly Default is false. If true, returns the extracted content from Tika without indexing the document. This literally includes the extracted XHTML as a string in the response. When viewing manually, it may be useful to use a response format other than XML to aid in viewing the embedded XHTML tags.For an example, see http://wiki.apache.org/solr/TikaExtractOnlyExampleOutput. extractFormat Default is "xml", but the other option is "text". Controls the serialization format of the extract content. The xml format is actually XHTML, the same format that results from passing the -x command to the Tika command line application, while the text format is like that produced by Tika's -t command. This parameter is valid only if extractOnly is set to true. fmap. Maps (moves) one field name to another. The source_field must be a field in incoming documents, and the value is the Solr field to map to. Example: fmap.co ntent=text causes the data in the content field generated by Tika to be moved to the Solr's text field. literal. Populates a field with the name supplied with the specified value for each document. The data can be multivalued if the field is multivalued. literalsOverride If true (the default), literal field values will override other values with the same field name. If false, literal values defined with literal. will be appended to data already in the fields extracted from Tika. If setting literalsOv erride to "false", the field must be multivalued. lowernames Values are "true" or "false". If true, all field names will be mapped to lowercase with underscores, if needed. For example, "Content-Type" would be mapped to "content_type." multipartUploadLimitInKB Useful if uploading very large documents, this defines the KB size of documents to allow. passwordsFile Defines a file path and name for a file of file name to password mappings. Apache Solr Reference Guide 5.1 196 resource.name Specifies the optional name of the file. Tika can use it as a hint for detecting a file's MIME type. resource.password Defines a password to use for a password-protected PDF or OOXML file tika.config Defines a file path and name to a customized Tika configuration file. This is only required if you have customized your Tika implementation. uprefix Prefixes all fields that are not defined in the schema with the given prefix. This is very useful when combined with dynamic field definitions. Example: uprefix=ig nored_ would effectively ignore all unknown fields generated by Tika given the example schema contains xpath When extracting, only return Tika XHTML content that satisfies the given XPath expression. See http://tika.apache.org/1.7/index.html for details on the format of Tika XHTML. See also http://wiki.apache.org/solr/TikaExtractOnlyExampleOutput. Order of Operations Here is the order in which the Solr Cell framework, using the Extracting Request Handler and Tika, processes its input. 1. Tika generates fields or passes them in as literals specified by literal.=. If li teralsOverride=false, literals will be appended as multi-value to the Tika-generated field. 2. If lowernames=true, Tika maps fields to lowercase. 3. Tika applies the mapping rules specified by fmap. source = target parameters. 4. If uprefix is specified, any unknown field names are prefixed with that value, else if defaultField is specified, any unknown fields are copied to the default field. Configuring the Solr ExtractingRequestHandler If you are not working with the supplied sample_techproducts_configs or data_driven_schema_conf igs config set, you must configure your own solrconfig.xml to know about the Jar's containing the Extract ingRequestHandler and it's dependencies: You can then configure the ExtractingRequestHandler in solrconfig.xml. Apache Solr Reference Guide 5.1 197 last_modified ignored_ /my/path/to/tika.config yyyy-MM-dd In the defaults section, we are mapping Tika's Last-Modified Metadata attribute to a field named last_modifie d. We are also telling it to ignore undeclared fields. These are all overridden parameters. The tika.config entry points to a file containing a Tika configuration. The date.formats allows you to specify various java.text.SimpleDateFormats date formats for working with transforming extracted input to a Date. Solr comes configured with the following date formats (see the DateUtil in Solr): yyyy-MM-dd'T'HH:mm:ss'Z' yyyy-MM-dd'T'HH:mm:ss yyyy-MM-dd yyyy-MM-dd hh:mm:ss yyyy-MM-dd HH:mm:ss EEE MMM d hh:mm:ss z yyyy EEE, dd MMM yyyy HH:mm:ss zzz EEEE, dd-MMM-yy HH:mm:ss zzz EEE MMM d HH:mm:ss yyyy You may also need to adjust the multipartUploadLimitInKB attribute as follows if you are submitting very large documents. ... Multi-Core Configuration For a multi-core configuration, you can specify sharedLib='lib' in the section of solr.xml and place the necessary jar files there. For more information about Solr cores, see The Well-Configured Solr Instance. Indexing Encrypted Documents with the ExtractingUpdateRequestHandler The ExtractingRequestHandler will decrypt encrypted files and index their content if you supply a password in either resource.password on the request, or in a passwordsFile file. Apache Solr Reference Guide 5.1 198 In the case of passwordsFile, the file supplied must be formatted so there is one line per rule. Each rule contains a file name regular expression, followed by "=", then the password in clear-text. Because the passwords are in clear-text, the file should have strict access restrictions. # This is a comment myFileName = myPassword .*\.docx$ = myWordPassword .*\.pdf$ = myPdfPassword Examples Metadata As mentioned before, Tika produces metadata about the document. Metadata describes different aspects of a document, such as the author's name, the number of pages, the file size, and so on. The metadata produced depends on the type of document submitted. For instance, PDFs have different metadata than Word documents do. In addition to Tika's metadata, Solr adds the following metadata (defined in ExtractingMetadataConstants) : Solr Metadata Description stream_name The name of the Content Stream as uploaded to Solr. Depending on how the file is uploaded, this may or may not be set stream_source_info Any source info about the stream. (See the section on Content Streams later in this section.) stream_size The size of the stream in bytes. stream_content_type The content type of the stream, if available. We recommend that you try using the extractOnly option to discover which values Solr is setting for these metadata elements. Examples of Uploads Using the Extracting Request Handler Capture and Mapping The command below captures
tags separately, and then maps all the instances of that field to a dynamic field named foo_t. bin/post -c techproducts example/exampledocs/sample.html -params "literal.id=doc2&captureAttr=true&defaultField=_text_&fmap.div=foo_t&capture=div" Capture, Mapping, and Boosting The command below captures
tags separately, maps the field to a dynamic field named foo_t, then boosts foo_t by 3. Apache Solr Reference Guide 5.1 199 bin/post -c techproducts example/exampledocs/sample.html -params "literal.id=doc3&captureAttr=true&defaultField=_text_&capture=div&fmap.div=foo_t&boo st.foo_t=3" Using Literals to Define Your Own Metadata To add in your own metadata, pass in the literal parameter along with the file: bin/post -c techproducts -params "literal.id=doc4&captureAttr=true&defaultField=text&capture=div&fmap.div=foo_t&boost .foo_t=3&literal.blah_s=Bah" example/exampledocs/sample.html XPath The example below passes in an XPath expression to restrict the XHTML returned by Tika: bin/post -c techproducts -params "literal.id=doc5&captureAttr=true&defaultField=text&capture=div&fmap.div=foo_t&boost .foo_t=3&xpath=/xhtml:html/xhtml:body/xhtml:div//node()" example/exampledocs/sample.html Extracting Data without Indexing It Solr allows you to extract data without indexing. You might want to do this if you're using Solr solely as an extraction server or if you're interested in testing Solr extraction. The example below sets the extractOnly=true parameter to extract data without indexing it. curl "http://localhost:8983/solr/techproducts/update/extract?&extractOnly=true" --data-binary @example/exampledocs/sample.html -H 'Content-type:text/html' The output includes XML generated by Tika (and further escaped by Solr's XML) using a different output format to make it more readable (`-out yes` instructs the tool to echo Solr's output to the console): bin/post -c techproducts -params "extractOnly=true&wt=ruby&indent=true" -out yes example/exampledocs/sample.html Sending Documents to Solr with a POST The example below streams the file as the body of the POST, which does not, then, provide information to Solr about the name of the file. curl "http://localhost:8983/solr/techproducts/update/extract?literal.id=doc6&defaultField =text&commit=true" --data-binary @example/exampledocs/sample.html -H 'Content-type:text/html' Sending Documents to Solr with Solr Cell and SolrJ Apache Solr Reference Guide 5.1 200 SolrJ is a Java client that you can use to add documents to the index, update the index, or query the index. You'll find more information on SolrJ in Client APIs. Here's an example of using Solr Cell and SolrJ to add documents to a Solr index. First, let's use SolrJ to create a new SolrClient, then we'll construct a request containing a ContentStream (essentially a wrapper around a file) and sent it to Solr: public class SolrCellRequestDemo { public static void main (String[] args) throws IOException, SolrServerException { SolrClient server = new HttpSolrClient("http://localhost:8983/solr/my_collection"); ContentStreamUpdateRequest req = new ContentStreamUpdateRequest("/update/extract"); req.addFile(new File("my-file.pdf")); req.setParam(ExtractingParams.EXTRACT_ONLY, "true"); NamedList result = server.request(req); System.out.println("Result: " + result); } This operation streams the file my-file.pdf into the Solr index for my_collection. The sample code above calls the extract command, but you can easily substitute other commands that are supported by Solr Cell. The key class to use is the ContentStreamUpdateRequest, which makes sure the ContentStreams are set properly. SolrJ takes care of the rest. Note that the ContentStreamUpdateRequest is not just specific to Solr Cell. You can send CSV to the CSV Update handler and to any other Request Handler that works with Content Streams for updates. Related Topics ExtractingRequestHandler Uploading Structured Data Store Data with the Data Import Handler Many search applications store the content to be indexed in a structured data store, such as a relational database. The Data Import Handler (DIH) provides a mechanism for importing content from a data store and indexing it. In addition to relational databases, DIH can index content from HTTP based data sources such as RSS and ATOM feeds, e-mail repositories, and structured XML where an XPath processor is used to generate fields. The example/example-DIH directory contains several collections many of the features of the data import handler. To run this "dih" example: bin/solr -e dih For more information about the Data Import Handler, see https://wiki.apache.org/solr/DataImportHandler. Topics covered in this section: Apache Solr Reference Guide 5.1 201 Concepts and Terminology Configuration Data Import Handler Commands Property Writer Data Sources Entity Processors Transformers Special Commands for the Data Import Handler Concepts and Terminology Descriptions of the Data Import Handler use several familiar terms, such as entity and processor, in specific ways, as explained in the table below. Term Definition Datasource As its name suggests, a datasource defines the location of the data of interest. For a database, it's a DSN. For an HTTP datasource, it's the base URL. Entity Conceptually, an entity is processed to generate a set of documents, containing multiple fields, which (after optionally being transformed in various ways) are sent to Solr for indexing. For a RDBMS data source, an entity is a view or table, which would be processed by one or more SQL statements to generate a set of rows (documents) with one or more columns (fields). Processor An entity processor does the work of extracting content from a data source, transforming it, and adding it to the index. Custom entity processors can be written to extend or replace the ones supplied. Transformer Each set of fields fetched by the entity may optionally be transformed. This process can modify the fields, create new fields, or generate multiple rows/documents form a single row. There are several built-in transformers in the DIH, which perform functions such as modifying dates and stripping HTML. It is possible to write custom transformers using the publicly available interface. Configuration Configuring solrconfig.xml The Data Import Handler has to be registered in solrconfig.xml. For example: /path/to/my/DIHconfigfile.xml The only required parameter is the config parameter, which specifies the location of the DIH configuration file that contains specifications for the data source, how to fetch data, what data to fetch, and how to process it to generate the Solr documents to be posted to the index. You can have multiple DIH configuration files. Each file would require a separate definition in the solrconfig. xml file, specifying a path to the file. Apache Solr Reference Guide 5.1 202 Configuring the DIH Configuration File An annotated configuration file, based on the "db" collection in the dih example server, is shown below (exampl e/example-DIH/solr/db/conf/db-data-config.xml). It extracts fields from the four tables defining a simple product database, with this schema. More information about the parameters and options shown here are described in the sections following. Apache Solr Reference Guide 5.1 203 Apache Solr Reference Guide 5.1 204 Datasources can still be specified in solrconfig.xml. These must be specified in the defaults section of the handler in solrconfig.xml. However, these are not parsed until the main configuration is loaded. The entire configuration itself can be passed as a request parameter using the dataConfig parameter rather than using a file. When configuration errors are encountered, the error message is returned in XML format. A reload-config command is also supported, which is useful for validating a new configuration file, or if you want to specify a file, load it, and not have it reloaded again on import. If there is an xml mistake in the configuration a user-friendly message is returned in xml format. You can then fix the problem and do a reloadconfig. You can also view the DIH configuration in the Solr Admin UI and there is an interface to import content. Data Import Handler Commands DIH commands are sent to Solr via an HTTP request. The following operations are supported. Command abort Description Aborts an ongoing operation. The URL is http://:/ solr/ / dataimport? command=abort. delta-import For incremental imports and change detection. The command is of the form http://:/ solr/ / dataimport? command=delta-impor t. It supports the same clean, commit, optimize and debug parameters as full-import command. full-import A Full Import operation can be started with a URL of the form http://:/ solr/ / dataimport? command=full-import. The command returns immediately. The operation will be started in a new thread and the status attribute in the response should be shown as busy. The operation may take some time depending on the size of dataset. Queries to Solr are not blocked during full-imports. When a full-import command is executed, it stores the start time of the operation in a file located at conf/dataimport.properties. This stored timestamp is used when a delta-import operation is executed. For a list of parameters that can be passed to this command, see below. reload-config If the configuration file has been changed and you wish to reload it without restarting Solr, run the command http://:/ solr/ / dataimp ort? command=reload-config. status The URL is http://:/ solr/ / dataimport? command=status. It returns statistics on the number of documents created, deleted, queries run, rows fetched, status, and so on. Parameters for the full-import Command The full-import command accepts the following parameters: Parameter Apache Solr Reference Guide 5.1 Description 205 clean Default is true. Tells whether to clean up the index before the indexing is started. commit Default is true. Tells whether to commit after the operation. debug Default is false Runs the command in debug mode. It is used by the interactive development mode. Note that in debug mode, documents are never committed automatically. If you want to run debug mode and commit the results too, add commit=true as a request parameter. entity The name of an entity directly under the tag in the configuration file. Use this to execute one or more entities selectively. Multiple "entity" parameters can be passed on to run multiple entities at once. If nothing is passed, all entities are executed. optimize Default is true. Tells Solr whether to optimize after the operation. Property Writer The propertyWriter element defines the date format and locale for use with delta queries. It is an optional configuration. Add the element to the DIH configuration file, directly under the dataConfig element. The parameters available are: Parameter Description dateFormat A java.text.SimpleDateFormat to use when converting the date to text. The default is "yyyy-MM-dd HH:mm:ss". type The implementation class. Use SimplePropertiesWriter for non-SolrCloud installations. If using SolrCloud, use ZKPropertiesWriter. If this is not specified, it will default to the appropriate class depending on if SolrCloud mode is enabled. directory Used with the SimplePropertiesWriter only). The directory for the properties file. If not specified, the default is "conf". filename Used with the SimplePropertiesWriter only). The name of the properties file. If not specified, the default is the requestHandler name (as defined in solrconfig.xml, appended by ".properties" (i.e., "dataimport.properties"). locale The locale. If not defined, the ROOT locale is used. It must be specified as language-country. For example, en-US. Data Sources A data source specifies the origin of data and its type. Somewhat confusingly, some data sources are configured within the associated entity processor. Data sources can also be specified in solrconfig.xml, which is useful when you have multiple environments (for example, development, QA, and production) differing only in their data sources. You can create a custom data source by writing a class that extends org.apache.solr.handler.dataimpo rt.DataSource. Apache Solr Reference Guide 5.1 206 The mandatory attributes for a data source definition are its name and type. The name identifies the data source to an Entity element. The types of data sources available are described below. ContentStreamDataSource This takes the POST data as the data source. This can be used with any EntityProcessor that uses a DataSour ce. FieldReaderDataSource This can be used where a database field contains XML which you wish to process using the XPathEntityProcessor. You would set up a configuration with both JDBC and FieldReader data sources, and two entities, as follows: /> ... The FieldReaderDataSource can take an encoding parameter, which will default to "UTF-8" if not specified.It must be specified as language-country. For example, en-US. FileDataSource This can be used like an URLDataSource, but is used to fetch content from files on disk. The only difference from URLDataSource, when accessing disk files, is how a pathname is specified. This data source accepts these optional attributes. Optional Attribute Description basePath The base path relative to which the value is evaluated if it is not absolute. encoding Defines the character encoding to use. If not defined, UTF-8 is used. JdbcDataSource Apache Solr Reference Guide 5.1 207 This is the default datasource. It's used with the SqlEntityProcessor. See the example in the FieldReaderDataSo urce section for details on configuration. URLDataSource This data source is often used with XPathEntityProcessor to fetch content from an underlying file:// or http :// location. Here's an example: The URLDataSource type accepts these optional parameters: Optional Parameter Description baseURL Specifies a new baseURL for pathnames. You can use this to specify host/port changes between Dev/QA/Prod environments. Using this attribute isolates the changes to be made to the solrconfig.xml connectionTimeout Specifies the length of time in milliseconds after which the connection should time out. The default value is 5000ms. encoding By default the encoding in the response header is used. You can use this property to override the default encoding. readTimeout Specifies the length of time in milliseconds after which a read operation should time out. The default value is 10000ms. Entity Processors Entity processors extract data, transform it, and add it to a Solr index. Examples of entities include views or tables in a data store. Each processor has its own set of attributes, described in its own section below. In addition, there are non-specific attributes common to all entities which may be specified. Attribute Use dataSource The name of a data source. If there are multiple data sources defined, use this attribute with the name of the data source for this entity. name Required. The unique name used to identify an entity. pk The primary key for the entity. It is optional, and required only when using delta-imports. It has no relation to the uniqueKey defined in schema.xml but they can both be the same. It is mandatory if you do delta-imports and then refers to the column name in ${dataimporter.delta.} which is used as the primary key. Apache Solr Reference Guide 5.1 208 processor Default is SqlEntityProcessor. Required only if the datasource is not RDBMS. onError Permissible values are (abort|skip|continue) . The default value is 'abort'. 'Skip' skips the current document. 'Continue' ignores the error and processing continues. preImportDeleteQuery Before a full-import command, use this query this to cleanup the index instead of using '*:*'. This is honored only on an entity that is an immediate sub-child of . postImportDeleteQuery Similar to the above, but executed after the import has completed. rootEntity By default the entities immediately under the are root entities. If this attribute is set to false, the entity directly falling under that entity will be treated as the root entity (and so on). For every row returned by the root entity, a document is created in Solr. transformer Optional. One or more transformers to be applied on this entity. cacheImpl Optional. A class (which must implement DIHCache) to use for caching this entity when doing lookups from an entity which wraps it. Provided implementation is "Sort edMapBackedCache". cacheKey The name of a property of this entity to use as a cache key if cacheImpl is specified. cacheLookup An entity + property name that will be used to lookup cached instances of this entity if cacheImpl is specified. Caching of entities in DIH is provided to avoid repeated lookups for same entities again and again. The default S ortedMapBackedCache is a HashMap where a key is a field in the row and the value is a bunch of rows for that same key. In the example below, each manufacturer entity is cached using the 'id' property as a cache key. Cache lookups will be performed for each product entity based on the product's "manu" property. When the cache has no data for a particular key, the query is run and the cache is populated The SQL Entity Processor The SqlEntityProcessor is the default processor. The associated data source should be a JDBC URL. The entity attributes specific to this processor are shown in the table below. Attribute query Use Required. The SQL query used to select rows. Apache Solr Reference Guide 5.1 209 deltaQuery SQL query used if the operation is delta-import. This query selects the primary keys of the rows which will be parts of the delta-update. The pks will be available to the deltaImportQuery through the variable ${dataimporter.delta.}. parentDeltaQuery SQL query used if the operation is delta-import. deletedPkQuery SQL query used if the operation is delta-import. deltaImportQuery SQL query used if the operation is delta-import. If this is not present, DIH tries to construct the import query by(after identifying the delta) modifying the 'query' (this is error prone). There is a namespace ${dataimporter.delta.} which can be used in this query. For example, select * from tbl where id=${dataimporter.delta.id}. The XPathEntityProcessor This processor is used when indexing XML formatted data. The data source is typically URLDataSource or FileD ataSource. Xpath can also be used with the FileListEntityProcessor described below, to generate a document from each file. The entity attributes unique to this processor are shown below. Attribute Use Processor Required. Must be set to "XpathEntityProcessor". url Required. HTTP URL or file location. stream Optional: Set to true for a large file or download. forEach Required unless you define useSolrAddSchema. The Xpath expression which demarcates each record. This will be used to set up the processing loop. xsl Optional: Its value (a URL or filesystem path) is the name of a resource used as a preprocessor for applying the XSL transformation. useSolrAddSchema Set this to true if the content is in the form of the standard Solr update XML schema. flatten Optional: If set true, then text from under all the tags is extracted into one field. Each field element in the entity can have the following attributes as well as the default ones. Attribute Use xpath Required. The XPath expression which will extract the content from the record for this field. Only a subset of Xpath syntax is supported. commonField Optional. If true, then when this field is encountered in a record it will be copied to future records when creating a Solr document. Here is an example from the "rss" collection in the dih example (example/example-DIH/solr/rss/conf/ rss-data-config.xml): Apache Solr Reference Guide 5.1 210 column="source" xpath="/RDF/channel/title" commonField="true" /> column="source-link" xpath="/RDF/channel/link" commonField="true"/> column="subject" xpath="/RDF/channel/subject" commonField="true" /> column="title" xpath="/RDF/item/title" /> column="link" xpath="/RDF/item/link" /> column="description" xpath="/RDF/item/description" /> column="creator" xpath="/RDF/item/creator" /> column="item-subject" xpath="/RDF/item/subject" /> column="date" xpath="/RDF/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" /> column="slash-department" xpath="/RDF/item/department" /> column="slash-section" xpath="/RDF/item/section" /> column="slash-comments" xpath="/RDF/item/comments" /> The MailEntityProcessor The MailEntityProcessor uses the Java Mail API to index email messages using the IMAP protocol. The MailEntityProcessor works by connecting to a specified mailbox using a username and password, fetching the email headers for each message, and then fetching the full email contents to construct a document (one document for each mail message). Here is an example from the "mail" collection of the dih example (example/example-DIH/mail/conf/mai l-data-config.xml): Apache Solr Reference Guide 5.1 211 The entity attributes unique to the MailEntityProcessor are shown below. Attribute Use processor Required. Must be set to "MailEntityProcessor". user Required. Username for authenticating to the IMAP server; this is typically the email address of the mailbox owner password Required. Password for authenticating to the IMAP server host Required. The IMAP server to connect to protocol Required. The IMAP protocol to use, valid values are: imap, imaps, gimap, and gimaps fetchMailsSince Optional. Date/time used to set a filter to import messages that occur after the specified date; expected format is: yyyy-MM-dd HH:mm:ss folders Required. Comma-delimited list of folder names to pull messages from, such as "inbox" recurse Optional (default is true). Flag to indicate if the processor should recurse all child folders when looking for messages to import include Optional. Comma-delimited list of folder patterns to include when processing folders (can be a literal value or regular expression) exclude Optional. Comma-delimited list of folder patterns to exclude when processing folders (can be a literal value or regular expression); excluded folder patterns take precedent over include folder patterns. processAttachement Optional (default is true). Use Tika to process message attachments. or processAttachments includeContent Optional (default is true). Include the message body when constructing Solr documents for indexing Importing New Emails Only After running a full import, the MailEntityProcessor keeps track of the timestamp of the previous import so that subsequent imports can use the fetchMailsSince filter to only pull new messages from the mail server. This occurs automatically using the Data Import Handler dataimport.properties file (stored in conf). For instance, if you set fetchMailsSince=2014-08-22 00:00:00 in your mail-data-config.xml, then all mail messages that occur after this date will be imported on the first run of the importer. Subsequent imports will use the date of the previous import as the fetchMailsSince filter, so that only new emails since the last import are indexed each time. GMail Extensions When connecting to a GMail account, you can improve the efficiency of the MailEntityProcessor by setting the protocol to gimap or gimaps. This allows the processor to send the fetchMailsSince filter to the GMail server to have the date filter applied on the server, which means the processor only receives new messages from the server. However, GMail only supports date granularity, so the server-side filter may return previously seen messages if run more than once a day. Apache Solr Reference Guide 5.1 212 The TikaEntityProcessor The TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to Uploading Data with Solr Cell using Apache Tika, but using the DataImportHandler options instead. Here is an example from the "tika" collection of the dih example (example/example-DIH/tika/conf/tik a-data-config.xml): The parameters for this processor are described in the table below: Attribute dataSource Use This parameter defines the data source and an optional name which can be referred to in later parts of the configuration if needed. This is the same dataSource explained in the description of general entity processor attributes above. The available data source types for this processor are: BinURLDataSource: used for HTTP resources, but can also be used for files. BinContentStreamDataSource: used for uploading content as a stream. BinFileDataSource: used for content on the local filesystem. url The path to the source file(s), as a file path or a traditional internet URL. This parameter is required. htmlMapper Allows control of how Tika parses HTML. The "default" mapper strips much of the HTML from documents while the "identity" mapper passes all HTML as-is with no modifications. If this parameter is defined, it must be either default or identity; if it is absent, "default" is assumed. format The output format. The options are text, xml, html or none. The default is "text" if not defined. The format "none" can be used if metadata only should be indexed and not the body of the documents. parser The default parser is org.apache.tika.parser.AutoDetectParser. If a custom or other parser should be used, it should be entered as a fully-qualified name of the class and path. Apache Solr Reference Guide 5.1 213 fields The list of fields from the input documents and how they should be mapped to Solr fields. If the attribute meta is defined as "true", the field will be obtained from the metadata of the document and not parsed from the body of the main text. extractEmbedded Instructs the TikaEntityProcessor to extract embedded documents or attachments when tr ue. If false, embedded documents and attachments will be ignored. onError By default, the TikaEntityProcessor will stop processing documents if it finds one that generates an error. If you define onError to "skip", the TikaEntityProcessor will instead skip documents that fail processing and log a message that the document was skipped. The FileListEntityProcessor This processor is basically a wrapper, and is designed to generate a set of files satisfying conditions specified in the attributes which can then be passed to another processor, such as the XPathEntityProcessor. The entity information for this processor would be nested within the FileListEnitity entry. It generates four implicit fields: fil eAbsolutePath, fileSize, fileLastModified, fileName which can be used in the nested processor. This processor does not use a data source. The attributes specific to this processor are described in the table below: Attribute Use fileName Required. A regular expression pattern to identify files to be included. basedir Required. The base directory (absolute path). recursive Whether to search directories recursively. Default is 'false'. excludes A regular expression pattern to identify files which will be excluded. newerThan A date in the format yyyy-MM-ddHH:mm:ss or a date math expression (NOW - 2YEARS). olderThan A date, using the same formats as newerThan. rootEntity This should be set to false. This ensures that each row (filepath) emitted by this processor is considered to be a document. dataSource Must be set to null. The example below shows the combination of the FileListEntityProcessor with another processor which will generate a set of fields from each file found. Apache Solr Reference Guide 5.1 214 LineEntityProcessor This EntityProcessor reads all content from the data source on a line by line basis and returns a field called rawL ine for each line read. The content is not parsed in any way; however, you may add transformers to manipulate the data within the rawLine field, or to create other additional fields. The lines read can be filtered by two regular expressions specified with the acceptLineRegex and omitLineR egex attributes. The table below describes the LineEntityProcessor's attributes: Attribute Description url A required attribute that specifies the location of the input file in a way that is compatible with the configured data source. If this value is relative and you are using FileDataSource or URLDataSource, it assumed to be relative to baseLoc. acceptLineRegex An optional attribute that if present discards any line which does not match the regExp. omitLineRegex An optional attribute that is applied after any acceptLineRegex and that discards any line which matches this regExp. For example: Apache Solr Reference Guide 5.1 215 ... While there are use cases where you might need to create a Solr document for each line read from a file, it is expected that in most cases that the lines read by this processor will consist of a pathname, which in turn will be consumed by another EntityProcessor, such as XPathEntityProcessor. PlainTextEntityProcessor This EntityProcessor reads all content from the data source into an single implicit field called plainText. The content is not parsed in any way, however you may add transformers to manipulate the data within the plainTe xt as needed, or to create other additional fields. For example: Ensure that the dataSource is of type DataSource (FileDataSource, URLDataSource). Transformers Transformers manipulate the fields in a document returned by an entity. A transformer can create new fields or modify existing ones. You must tell the entity which transformers your import operation will be using, by adding an attribute containing a comma separated list to the element. Specific transformation rules are then added to the attributes of a element, as shown in the examples below. The transformers are applied in the order in which they are specified in the transformer attribute. The Data Import Handler contains several built-in transformers. You can also write your own custom transformers, as described in the Solr Wiki (see http://wiki.apache.org/solr/DIHCustomTransformer). The ScriptTransformer (described below) offers an alternative method for writing your own transformers. Solr includes the following built-in transformers: Transformer Name Use ClobTransformer Used to create a String out of a Clob type in database. DateFormatTransformer Parse date/time instances. Apache Solr Reference Guide 5.1 216 HTMLStripTransformer Strip HTML from a field. LogTransformer Used to log data to log files or a console. NumberFormatTransformer Uses the NumberFormat class in java to parse a string into a number. RegexTransformer Use regular expressions to manipulate fields. ScriptTransformer Write transformers in Javascript or any other scripting language supported by Java. TemplateTransformer Transform a field using a template. These transformers are described below. ClobTransformer You can use the ClobTransformer to create a string out of a CLOB in a database. A CLOB is a character large object: a collection of character data typically stored in a separate location that is referenced in the database. See http://en.wikipedia.org/wiki/Character_large_object. Here's an example of invoking the ClobTransformer. ... The ClobTransformer accepts these attributes: Attribute Description clob Boolean value to signal if ClobTransformer should process this field or not. If this attribute is omitted, then the corresponding field is not transformed. sourceColName The source column to be used as input. If this is absent source and target are same The DateFormatTransformer This transformer converts dates from one format to another. This would be useful, for example, in a situation where you wanted to convert a field with a fully specified date/time into a less precise date format, for use in faceting. DateFormatTransformer applies only on the fields with an attribute dateTimeFormat. Other fields are not modified. This transformer recognizes the following attributes: Attribute Description dateTimeFormat The format used for parsing this field. This must comply with the syntax of the Java SimpleDateFormat class. sourceColName The column on which the dateFormat is to be applied. If this is absent source and target are same. Apache Solr Reference Guide 5.1 217 locale The locale to use for date transformations. If not specified, the ROOT locale will be used. It must be specified as language-country. For example, en-US. Here is example code that returns the date rounded up to the month "2007-JUL": ... The HTMLStripTransformer You can use this transformer to strip HTML out of a field. For example: ... There is one attribute for this transformer, stripHTML, which is a boolean value (true/false) to signal if the HTMLStripTransformer should process the field or not. The LogTransformer You can use this transformer to log data to the console or log files. For example: .... Unlike other transformers, the LogTransformer does not apply to any field, so the attributes are applied on the entity itself. The NumberFormatTransformer Use this transformer to parse a number from a string, converting it into the specified format, and optionally using a different locale. NumberFormatTransformer will be applied only to fields with an attribute formatStyle. This transformer recognizes the following attributes: Attribute formatStyle Description The format used for parsing this field. The value of the attribute must be one of (number|p ercent|integer|currency). This uses the semantics of the Java NumberFormat class. sourceColName The column on which the NumberFormat is to be applied. This is attribute is absent. The source column and the target column are the same. Apache Solr Reference Guide 5.1 218 locale The locale to be used for parsing the strings. If this is absent, the ROOT locale is used. It must be specified as language-country. For example, en-US. For example: ... The RegexTransformer The regex transformer helps in extracting or manipulating values from fields (from the source) using Regular Expressions. The actual class name is org.apache.solr.handler.dataimport.RegexTransformer. But as it belongs to the default package the package-name can be omitted. The table below describes the attributes recognized by the regex transformer. Attribute Description regex The regular expression that is used to match against the column or sourceColName's value(s). If replaceWith is absent, each regex group is taken as a value and a list of values is returned. sourceColName The column on which the regex is to be applied. If not present, then the source and target are identical. splitBy Used to split a string. It returns a list of values. groupNames A comma separated list of field column names, used where the regex contains groups and each group is to be saved to a different field. If some groups are not to be named leave a space between commas. replaceWith Used along with regex . It is equivalent to the method new String().replaceAll(, ). Here is an example of configuring the regex transformer: In this example, regex and sourceColName are custom attributes used by the transformer. The transformer Apache Solr Reference Guide 5.1 219 reads the field full_name from the resultset and transforms it to two new target fields, firstName and lastNa me. Even though the query returned only one column, full_name, in the result set, the Solr document gets two extra fields firstName and lastName which are "derived" fields. These new fields are only created if the regexp matches. The emailids field in the table can be a comma-separated value. It ends up producing one or more email IDs, and we expect the mailId to be a multivalued field in Solr. Note that this transformer can either be used to split a string into tokens based on a splitBy pattern, or to perform a string substitution as per replaceWith, or it can assign groups within a pattern to a list of groupNames. It decides what it is to do based upon the above attributes splitBy, replaceWith and groupNames which are looked for in order. This first one found is acted upon and other unrelated attributes are ignored. The ScriptTransformer The script transformer allows arbitrary transformer functions to be written in any scripting language supported by Java, such as Javascript, JRuby, Jython, Groovy, or BeanShell. Javascript is integrated into Java 7; you'll need to integrate other languages yourself. Each function you write must accept a row variable (which corresponds to a Java Map, thus permitting get,put,remove operations). Thus you can modify the value of an existing field or add new fields. The return value of the function is the returned object. The script is inserted into the DIH configuration file at the top level and is called once for each row. Here is a simple example. .... The TemplateTransformer Apache Solr Reference Guide 5.1 220 You can use the template transformer to construct or modify a field value, perhaps using the value of other fields. You can insert extra text into the template. ... Special Commands for the Data Import Handler You can pass special commands to the DIH by adding any of the variables listed below to any row returned by any component: Variable $skipDoc Description Skip the current document; that is, do not add it to Solr. The value can be the string tr ue|false. $skipRow Skip the current row. The document will be added with rows from other entities. The value can be the string true|false $docBoost Boost the current document. The boost value can be a number or the toString conve rsion of a number. $deleteDocById Delete a document from Solr with this ID. The value has to be the uniqueKey value of the document. $deleteDocByQuery Delete documents from Solr using this query. The value must be a Solr Query. Updating Parts of Documents Once you have indexed the content you need in your Solr index, you will want to start thinking about your strategy for dealing with changes to those documents. Solr supports two approaches to updating documents that have only partially changed. The first is atomic updates. This approach allows changing only one or more fields of a document without having to re-index the entire document. The second approach is known as optimistic concurrency or optimistic locking. It is a feature of many NoSQL databases, and allows conditional updating a document based on its version. This approach includes semantics and rules for how to deal with version matches or mis-matches. Atomic Updates and Optimistic Concurrency may be used as independent strategies for managing changes to documents, or they may be combined: you can use optimistic concurrency to conditionally apply an atomic update. Atomic Updates Solr supports several modifiers that atomically update values of a document. This allows updating only specific fields, which can help speed indexing processes in an environment where speed of index additions is critical to the application. Apache Solr Reference Guide 5.1 221 To use atomic updates, add a modifier to the field that needs to be updated. The content can be updated, added to, or incrementally increased if a number. Modifier set Usage Set or replace the field value(s) with the specified value(s), or remove the values if 'null' or empty list is specified as the new value. May be specified as a single value, or as a list for multivalued fields add Adds the specified values to a multivalued field. May be specified as a single value, or as a list. remove Removes (all occurrences of) the specified values from a multivalued field. May be specified as a single value, or as a list. removeregex Removes all occurrences of the specified regex from a multiValued field. May be specified as a single value, or as a list. inc Increments a numeric value by a specific amount. Must be specified as a single numeric value. All original source fields must be stored for field modifiers to work correctly, which is the Solr default. For example, if the following document exists in our collection: {"id":"mydoc", "price":10, "popularity":42, "categories":["kids"], "promo_ids":["a123x"], "tags":["free_to_try","buy_now","clearance","on_sale"] } And we apply the following update command: {"id":"mydoc", "price":{"set":99}, "popularity":{"inc":20}, "categories":{"add":["toys","games"]}, "promo_ids":{"remove":"a123x"}, "tags":{"remove":["free_to_try","on_sale"]} } The resulting document in our collection will be: Apache Solr Reference Guide 5.1 222 {"id":"mydoc", "price":99, "popularity":62, "categories":["kids","toys","games"], "tags":["buy_now","clearance"] } Optimistic Concurrency Optimistic Concurrency is a feature of Solr that can be used by client applications which update/replace documents to ensure that the document they are replacing/updating has not been concurrently modified by another client application. This feature works by requiring a _version_ field on all documents in the index, and comparing that to a _version_ specified as part of the update command. By default, Solr's schema.xml includ es a _version_ field, and this field is automatically added to each new document. In general, using optimistic concurrency involves the following work flow: 1. A client reads a document. In Solr, one might retrieve the document with the /get handler to be sure to have the latest version. 2. A client changes the document locally. 3. The client resubmits the changed document to Solr, for example, perhaps with the /update handler. 4. If there is a version conflict (HTTP error code 409), the client starts the process over. When the client resubmits a changed document to Solr, the _version_ can be included with the update to invoke optimistic concurrency control. Specific semantics are used to define when the document should be updated or when to report a conflict. If the content in the _version_ field is greater than '1' (i.e., '12345'), then the _version_ in the document must match the _version_ in the index. If the content in the _version_ field is equal to '1', then the document must simply exist. In this case, no version matching occurs, but if the document does not exist, the updates will be rejected. If the content in the _version_ field is less than '0' (i.e., '-1'), then the document must not exist. In this case, no version matching occurs, but if the document exists, the updates will be rejected. If the content in the _version_ field is equal to '0', then it doesn't matter if the versions match or if the document exists or not. If it exists, it will be overwritten; if it does not exist, it will be added. If the document being updated does not include the _version_ field, and atomic updates are not being used, the document will be treated by normal Solr rules, which is usually to discard the previous version. When using Optimistic Concurrency, clients can include an optional versions=true request parameter to indicate that the new versions of the documents being added should be included in the response. This allows clients to immediately know what the _version_ is of every documented added with out needing to make a redundant /get request. For example... Apache Solr Reference Guide 5.1 223 $ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?versions=true' --data-binary ' [ { "id" : "aaa" }, { "id" : "bbb" } ]' {"responseHeader":{"status":0,"QTime":6}, "adds":["aaa",1498562471222312960, "bbb",1498562471225458688]} $ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?_version_=999999&versions=true' --data-binary ' [{ "id" : "aaa", "foo_s" : "update attempt with wrong existing version" }]' {"responseHeader":{"status":409,"QTime":3}, "error":{"msg":"version conflict for aaa expected=999999 actual=1498562471222312960", "code":409}} $ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?_version_=1498562471222312960&versio ns=true&commit=true' --data-binary ' [{ "id" : "aaa", "foo_s" : "update attempt with correct existing version" }]' {"responseHeader":{"status":0,"QTime":5}, "adds":["aaa",1498562624496861184]} $ curl 'http://localhost:8983/solr/techproducts/query?q=*:*&fl=id,_version_' { "responseHeader":{ "status":0, "QTime":5, "params":{ "fl":"id,_version_", "q":"*:*"}}, "response":{"numFound":2,"start":0,"docs":[ { "id":"bbb", "_version_":1498562471225458688}, { "id":"aaa", "_version_":1498562624496861184}] }} For more information, please also see Yonik Seeley's presentation on NoSQL features in Solr 4 from Apache Lucene EuroCon 2012. Apache Solr Reference Guide 5.1 224 Power Tip The _version_ field is by default stored in the inverted index (indexed="true"). However, for some systems with a very large number of documents, the increase in FieldCache memory requirements may be too costly. A solution can be to declare the _version_ field as DocValues: Sample field definition Document Centric Versioning Constraints Optimistic Concurrency is extremely powerful, and works very efficiently because it uses an internally assigned, globally unique values for the _version_ field. However, In some situations users may want to configure their own document specific version field, where the version values are assigned on a per-document basis by an external system, and have Solr reject updates that attempt to replace a document with an "older" version. In situations like this the DocBasedVersionConstraintsProcessorFactory can be useful. The basic usage of DocBasedVersionConstraintsProcessorFactory is to configure it in solrconfig.x ml as part of the UpdateRequestProcessorChain and specify the name of your custom versionField in your schema that should be checked when validating updates: my_version_l Once configured, this update processor will reject (HTTP error code 409) any attempt to update an existing document where the value of the my_version_l field in the "new" document is not greater then the value of that field in the existing document. versionField vs _version_ The _version_ field used by Solr for its normal optimistic concurrency also has important semantics in how updates are distributed to replicas in SolrCloud, and MUST be assigned internally by Solr. Users can not re-purpose that field and specify it as the versionField for use in the DocBasedVersionCo nstraintsProcessorFactory configuration. DocBasedVersionConstraintsProcessorFactory supports two additional configuration params which are optional: ignoreOldUpdates - A boolean option which defaults to false. If set to true then instead of rejecting updates where the versionField is too low, the update will be silently ignored (and return a status 200 to the client). deleteVersionParam - A String parameter that can be specified to indicate that this processor should also inspect Delete By Id commands. The value of this configuration option should be the name of a request parameter that the processor will now consider mandatory for all attempts to Delete By Id, and Apache Solr Reference Guide 5.1 225 must be be used by clients to specify a value for the versionField which is greater then the existing value of the document to be deleted. When using this request param, any Delete By Id command with a high enough document version number to succeed will be internally converted into an Add Document command that replaces the existing document with a new one which is empty except for the Unique Key and versionField to keeping a record of the deleted version so future Add Document commands will fail if their "new" version is not high enough. Please consult the processor javadocs and test configs for additional information and example usages. De-Duplication Preventing duplicate or near duplicate documents from entering an index or tagging documents with a signature/fingerprint for duplicate field collapsing can be efficiently achieved with a low collision or fuzzy hash algorithm. Solr natively supports de-duplication techniques of this type via the class and allows for the easy addition of new hash/signature implementations. A Signature can be implemented several ways: Method Description MD5Signature 128 bit hash used for exact duplicate detection. Lookup3Signature 64 bit hash used for exact duplicate detection, much faster than MD5 and smaller to index TextProfileSignature Fuzzy hashing implementation from nutch for near duplicate detection. It's tunable but works best on longer text. Other, more sophisticated algorithms for fuzzy/near hashing can be added later. Adding in the de-duplication process will change the allowDups setting so that it applies to an update Term (with signatureField in this case) rather than the unique field Term. Of course the signature Field could be the unique field, but generally you want the unique field to be unique. When a document is added, a signature will automatically be generated and attached to the document in the specified sign atureField. Configuration Options There are two places in Solr to configure de-duplication: in solrconfig.xml and in schema.xml. In solrconfig.xml The SignatureUpdateProcessorFactory has to be registered in the solrconfig.xml as part of the UpdateRe questProcessorChain: Apache Solr Reference Guide 5.1 226 true id false name,features,cat solr.processor.Lookup3Signature Setting Default Description signatureClass org.apache.solr.update.processor.Lookup3Signature A Signature implementation for generating a signature hash. fields all fields The fields to use to generate the signature hash in a comma separated list. By default, all fields on the document will be used. signatureField signatureField The name of the field used to hold the fingerprint/signature. Be sure the field is defined in schema.xml. enabled true Enable/disable de-duplication processing. In schema.xml If you are using a separate field for storing the signature you must have it indexed: Be sure to change your update handlers to use the defined chain, i.e. dedupe The update processor can also be specified per request with a parameter of update.chain=dedupe. Detecting Languages During Indexing Solr can identify languages and map text to language-specific fields during indexing using the langid UpdateRe questProcessor. Solr supports two implementations of this feature: Tika's language detection feature: http://tika.apache.org/0.10/detection.html Apache Solr Reference Guide 5.1 227 LangDetect language detection: http://code.google.com/p/language-detection/ You can see a comparison between the two implementations here: http://blog.mikemccandless.com/2011/10/acc uracy-and-performance-of-googles.html. In general, the LangDetect implementation supports more languages with higher performance. For specific information on each of these language identification implementations, including a list of supported languages for each, see the relevant project websites. For more information about the langid UpdateRequestP rocessor, see the Solr wiki: http://wiki.apache.org/solr/LanguageDetection. For more information about language analysis in Solr, see Language Analysis. Configuring Language Detection You can configure the langid UpdateRequestProcessor in solrconfig.xml. Both implementations take the same parameters, which are described in the following section. At a minimum, you must specify the fields for language identification and a field for the resulting language code. Configuring Tika Language Detection Here is an example of a minimal Tika langid configuration in solrconfig.xml: title,subject,text,keywords language_s Configuring LangDetect Language Detection Here is an example of a minimal LangDetect langid configuration in solrconfig.xml: title,subject,text,keywords language_s langid Parameters As previously mentioned, both implementations of the langid UpdateRequestProcessor take the same parameters. Parameter langid Type Boolean Apache Solr Reference Guide 5.1 Default true Required no Description Enables and disables language detection. 228 langid.fl string none yes A comma- or space-delimited list of fields to be processed by langid. langid.langField string none yes Specifies the field for the returned language code. langid.langsField multivalued string none no Specifies the field for a list of returned language codes. If you use langid.ma p.individual, each detected language will be added to this field. langid.overwrite Boolean false no Specifies whether the content of the la ngField and langsField fields will be overwritten if they already contain values. langid.lcmap string none false A space-separated list specifying colon delimited language code mappings to apply to the detected languages. For example, you might use this to map Chinese, Japanese, and Korean to a common cjk code, and map both American and British English to a single en code by using langid.lcmap=ja: cjk zh:cjk ko:cjk en_GB:en en_US:en. This affects both the values put into the langField and langsFie ld fields, as well as the field suffixes when using langid.map, unless overridden by langid.map.lcmap langid.threshold float 0.5 no Specifies a threshold value between 0 and 1 that the language identification score must reach before langid accep ts it. With longer text fields, a high threshold such at 0.8 will give good results. For shorter text fields, you may need to lower the threshold for language identification, though you will be risking somewhat lower quality results. We recommend experimenting with your data to tune your results. langid.whitelist string none no Specifies a list of allowed language identification codes. Use this in combination with langid.map to ensure that you only index documents into fields that are in your schema. Apache Solr Reference Guide 5.1 229 langid.map Boolean false no Enables field name mapping. If true, Solr will map field names for all fields listed in langid.fl. langid.map.fl string none no A comma-separated list of fields for la ngid.map that is different than the fields specified in langid.fl. langid.map.keepOrig Boolean false no If true, Solr will copy the field during the field name mapping process, leaving the original field in place. langid.map.individual Boolean false no If true, Solr will detect and map languages for each field individually. langid.map.individual.fl string none no A comma-separated list of fields for use with langid.map.individual that is different than the fields specified in lan gid.fl. langid.fallbackFields string none no If no language is detected that meets the langid.threshold score, or if the detected language is not on the la ngid.whitelist, this field specifies language codes to be used as fallback values. If no appropriate fallback languages are found, Solr will use the language code specified in langid.fa llback. langid.fallback string none no Specifies a language code to use if no language is detected or specified in la ngid.fallbackFields. langid.map.lcmap string determined by langid.lcmap no A space-separated list specifying colon delimited language code mappings to use when mapping field names. For example, you might use this to make Chinese, Japanese, and Korean language fields use a common *_cjk s uffix, and map both American and British English fields to a single *_en b y using langid.map.lcmap=ja:cjk zh:cjk ko:cjk en_GB:en en_US:en. langid.map.pattern Java regular expression Apache Solr Reference Guide 5.1 none no By default, fields are mapped as _. To change this pattern, you can specify a Java regular expression in this parameter. 230 langid.map.replace Java replace none no By default, fields are mapped as _. To change this pattern, you can specify a Java replace in this parameter. langid.enforceSchema Boolean true no If false, the langid processor does not validate field names against your schema. This may be useful if you plan to rename or delete fields later in the UpdateChain. Content Streams When Solr RequestHandlers are accessed using path based URLs, the SolrQueryRequest object containing the parameters of the request may also contain a list of ContentStreams containing bulk data for the request. (The name SolrQueryRequest is a bit misleading: it is involved in all requests, regardless of whether it is a query request or an update request.) Stream Sources Currently RequestHandlers can get content streams in a variety of ways: For multipart file uploads, each file is passed as a stream. For POST requests where the content-type is not application/x-www-form-urlencoded, the raw POST body is passed as a stream. The full POST body is parsed as parameters and included in the Solr parameters. The contents of parameter stream.body is passed as a stream. If remote streaming is enabled and URL content is called for during request handling, the contents of each stream.url and stream.file parameters are fetched and passed as a stream. By default, curl sends a contentType="application/x-www-form-urlencoded" header. If you need to test a SolrContentHeader content stream, you will need to set the content type with the "-H" flag. RemoteStreaming Remote streaming lets you send the contents of a URL as a stream to a given SolrRequestHandler. You could use remote streaming to send a remote or local file to an update plugin. For convenience, remote streaming is enabled in most of the example solrconfig.xml files included with Solr, however it is not recommended in a production situation with out additional security between you and untrusted remote clients. The default behavior, when enableRemoteStreaming is not specified in solrconfig.xml is to not allow remote streaming (i.e., enableRemoteStreaming="false"). Apache Solr Reference Guide 5.1 231 If you enableRemoteStreaming="true" is used, be aware that this allows anyone to send a request to any URL or local file. If DumpRequestHandler is enabled, it will allow anyone to view any file on your system. Debugging Requests The example solrconfig.xml files include a "dump" RequestHandler: This handler simply outputs the contents of the SolrQueryRequest using the specified writer type wt. This is a useful tool to help understand what streams are available to the RequestHandlers. UIMA Integration You can integrate the Apache Unstructured Information Management Architecture (UIMA) with Solr. UIMA lets you define custom pipelines of Analysis Engines that incrementally add metadata to your documents as annotations. For more information about Solr UIMA integration, see https://wiki.apache.org/solr/SolrUIMA. Configuring UIMA The SolrUIMA UpdateRequestProcessor is a custom update request processor that takes documents being indexed, sends them to a UIMA pipeline, and then returns the documents enriched with the specified metadata. To configure UIMA for Solr, follow these steps: 1. Copy solr-uima-4.x.y.jar (under /solr-4.x.y/dist/) and its libraries (under contrib/uima/l ib) to a Solr libraries directory, or set tags in solrconfig.xml appropriately to point to those jar files: 2. Modify schema.xml, adding your desired metadata fields specifying proper values for type, indexed, stored, and multiValued options. For example: 3. Add the following snippet to solrconfig.xml: Apache Solr Reference Guide 5.1 232 VALID_ALCHEMYAPI_KEY VALID_ALCHEMYAPI_KEY VALID_ALCHEMYAPI_KEY VALID_ALCHEMYAPI_KEY VALID_ALCHEMYAPI_KEY VALID_OPENCALAIS_KEY /org/apache/uima/desc/OverridingParamsExtServicesAE.xml< /str> true false text org.apache.uima.alchemy.ts.concept.ConceptFS text concept org.apache.uima.alchemy.ts.language.LanguageFS language language org.apache.uima.SentenceAnnotation coveredText sentence Apache Solr Reference Guide 5.1 233 VALID_ALCHEMYAPI_KEY is your AlchemyAPI Access Key. You need to register an AlchemyAPI Access key to use AlchemyAPI services: http://www.alchemyapi.com/api/register.html. VALID_OPENCALAIS_KEY is your Calais Service Key. You need to register a Calais Service key to use the Calais services: http://www.opencalais.com/apikey. analysisEngine must contain an AE descriptor inside the specified path in the classpath. analyzeFields must contain the input fields that need to be analyzed by UIMA. If merge=true then their content will be merged and analyzed only once. Field mapping describes which features of which types should go in a field. 4. In your solrconfig.xml replace the existing default UpdateRequestHandler or create a new UpdateRequestHandler: uima Once you are done with the configuration your documents will be automatically enriched with the specified fields when you index them. Apache Solr Reference Guide 5.1 234 Searching This section describes how Solr works with search requests. It covers the following topics: Overview of Searching in Solr: An introduction to searching with Solr. Velocity Search UI: A simple search UI using the VelocityResponseWriter. Relevance: Conceptual information about understanding relevance in search results. Query Syntax and Parsing: A brief conceptual overview of query syntax and parsing. It also contains the following sub-sections: Common Query Parameters: No matter the query parser, there are several parameters that are common to all of them. The Standard Query Parser: Detailed information about the standard Lucene query parser. The DisMax Query Parser: Detailed information about Solr's DisMax query parser. The Extended DisMax Query Parser: Detailed information about Solr's Extended DisMax (eDisMax) Query Parser. Function Queries: Detailed information about parameters for generating relevancy scores using values from one or more numeric fields. Local Parameters in Queries: How to add local arguments to queries. Other Parsers: More parsers designed for use in specific situations. Faceting: Detailed information about categorizing search results based on indexed terms. Highlighting: Detailed information about Solr's highlighting utilities. Sub-sections cover the different types of highlighters: Standard Highlighter: Uses the most sophisticated and fine-grained query representation of the three highlighters. FastVector Highlighter: Optimized for term vector options on fields, and good for large documents and multiple languages. Postings Highlighter: Uses similar options as the FastVector highlighter, but is more compact and efficient. Spell Checking: Detailed information about Solr's spelling checker. Query Re-Ranking: Detailed information about re-ranking top scoring documents from simple queries using more complex scores. Transforming Result Documents: Detailed information about using DocTransformers to add computed information to individual documents Suggester: Detailed information about Solr's powerful autosuggest component. MoreLikeThis: Detailed information about Solr's similar results query component. Pagination of Results: Detailed information about fetching paginated results for display in a UI, or for fetching all documents matching a query. Result Grouping: Detailed information about grouping results based on common field values. Apache Solr Reference Guide 5.1 235 Result Clustering: Detailed information about grouping search results based on cluster analysis applied to text fields. A bit like "unsupervised" faceting. Spatial Search: How to use Solr's spatial search capabilities. The Terms Component: Detailed information about accessing indexed terms and the documents that include them. The Term Vector Component: How to get term information about specific documents. The Stats Component: How to return information from numeric fields within a document set. The Query Elevation Component: How to force documents to the top of the results for certain queries. Response Writers: Detailed information about configuring and using Solr's response writers. Near Real Time Searching: How to include documents in search results nearly immediately after they are indexed. RealTime Get: How to get the latest version of a document without opening a searcher. Overview of Searching in Solr Solr offers a rich, flexible set of features for search. To understand the extent of this flexibility, it's helpful to begin with an overview of the steps and components involved in a Solr search. When a user runs a search in Solr, the search query is processed by a request handler. A request handler is a Solr plug-in that defines the logic to be used when Solr processes a request. Solr supports a variety of request handlers. Some are designed for processing search queries, while others manage tasks such as index replication. Search applications select a particular request handler by default. In addition, applications can be configured to allow users to override the default selection in preference of a different request handler. To process a search query, a request handler calls a query parser, which interprets the terms and parameters of a query. Different query parsers support different syntax. The default query parser is the DisMax query parser. Solr also includes an earlier "standard" (Lucene) query parser, and an Extended DisMax (eDisMax) query parser. The standard query parser's syntax allows for greater precision in searches, but the DisMax query parser is much more tolerant of errors. The DisMax query parser is designed to provide an experience similar to that of popular search engines such as Google, which rarely display syntax errors to users. The Extended DisMax query parser is an improved version of DisMax that handles the full Lucene query syntax while still tolerating syntax errors. It also includes several additional features. In addition, there are common query parameters that are accepted by all query parsers. Input to a query parser can include: search strings---that is, terms to search for in the index parameters for fine-tuning the query by increasing the importance of particular strings or fields, by applying Boolean logic among the search terms, or by excluding content from the search results Apache Solr Reference Guide 5.1 236 parameters for controlling the presentation of the query response, such as specifying the order in which results are to be presented or limiting the response to particular fields of the search application's schema. Search parameters may also specify a query filter. As part of a search response, a query filter runs a query against the entire index and caches the results. Because Solr allocates a separate cache for filter queries, the strategic use of filter queries can improve search performance. (Despite their similar names, query filters are not related to analysis filters. Query filters perform queries at search time against data already in the index, while analysis filters, such as Tokenizers, parse content for indexing, following specified rules). A search query can request that certain terms be highlighted in the search response; that is, the selected terms will be displayed in colored boxes so that they "jump out" on the screen of search results. Highlighting can make it easier to find relevant passages in long documents returned in a search. Solr supports multi-term highlighting. Solr includes a rich set of search parameters for controlling how terms are highlighted. Search responses can also be configured to include snippets (document excerpts) featuring highlighted text. Popular search engines such as Google and Yahoo! return snippets in their search results: 3-4 lines of text offering a description of a search result. To help users zero in on the content they're looking for, Solr supports two special ways of grouping search results to aid further exploration: faceting and clustering. Faceting is the arrangement of search results into categories (which are based on indexed terms). Within each category, Solr reports on the number of hits for relevant term, which is called a facet constraint. Faceting makes it easy for users to explore search results on sites such as movie sites and product review sites, where there are many categories and many items within a category. The image below shows an example of faceting from the CNET Web site, which was the first site to use Solr. Faceting makes use of fields defined when the search applications were indexed. In the example above, these fields include categories of information that are useful for describing digital cameras: manufacturer, resolution, and zoom range. Clustering groups search results by similarities discovered when a search is executed, rather than when content is indexed. The results of clustering often lack the neat hierarchical organization found in faceted search results, but clustering can be useful nonetheless. It can reveal unexpected commonalities among search results, and it can help users rule out content that isn't pertinent to what they're really searching for. Apache Solr Reference Guide 5.1 237 Solr also supports a feature called MoreLikeThis, which enables users to submit new queries that focus on particular terms returned in an earlier query. MoreLikeThis queries can make use of faceting or clustering to provide additional aid to users. A Solr component called a response writer manages the final presentation of the query response. Solr includes a variety of response writers, including an XML Response Writer and a JSON Response Writer. The diagram below summarizes some key elements of the search process. Velocity Search UI Solr includes a sample search UI based on the VelocityResponseWriter (also known as Solritas) that demonstrates several useful features, such as searching, faceting, highlighting, autocomplete, and geospatial searching. When using the sample_techproducts_configs config set, you can access the Velocity sample Search UI here: http://localhost:8983/solr/techproducts/browse Apache Solr Reference Guide 5.1 238 The Velocity Search UI For more information about the Velocity Response Writer, see the Response Writer page. Relevance Relevance is the degree to which a query response satisfies a user who is searching for information. The relevance of a query response depends on the context in which the query was performed. A single search application may be used in different contexts by users with different needs and expectations. For example, a search engine of climate data might be used by a university researcher studying long-term climate trends, a farmer interested in calculating the likely date of the last frost of spring, a civil engineer interested in rainfall patterns and the frequency of floods, and a college student planning a vacation to a region and wondering what to pack. Because the motivations of these users vary, the relevance of any particular response to a query will vary as well. How comprehensive should query responses be? Like relevance in general, the answer to this question depends on the context of a search. The cost of not finding a particular document in response to a query is high in some contexts, such as a legal e-discovery search in response to a subpoena, and quite low in others, such as a search for a cake recipe on a Web site with dozens or hundreds of cake recipes. When configuring Solr, you should weigh comprehensiveness against other factors such as timeliness and ease-of-use. The e-discovery and recipe examples demonstrate the importance of two concepts related to relevance: Precision is the percentage of documents in the returned results that are relevant. Apache Solr Reference Guide 5.1 239 Recall is the percentage of relevant results returned out of all relevant results in the system. Obtaining perfect recall is trivial: simply return every document in the collection for every query. Returning to the examples above, it's important for an e-discovery search application to have 100% recall returning all the documents that are relevant to a subpoena. It's far less important that a recipe application offer this degree of precision, however. In some cases, returning too many results in casual contexts could overwhelm users. In some contexts, returning fewer results that have a higher likelihood of relevance may be the best approach. Using the concepts of precision and recall, it's possible to quantify relevance across users and queries for a collection of documents. A perfect system would have 100% precision and 100% recall for every user and every query. In other words, it would retrieve all the relevant documents and nothing else. In practical terms, when talking about precision and recall in real systems, it is common to focus on precision and recall at a certain number of results, the most common (and useful) being ten results. Through faceting, query filters, and other search components, a Solr application can be configured with the flexibility to help users fine-tune their searches in order to return the most relevant results for users. That is, Solr can be configured to balance precision and recall to meet the needs of a particular user community. The configuration of a Solr application should take into account: the needs of the application's various users (which can include ease of use and speed of response, in addition to strictly informational needs) the categories that are meaningful to these users in their various contexts (e.g., dates, product categories, or regions) any inherent relevance of documents (e.g., it might make sense to ensure that an official product description or FAQ is always returned near the top of the search results) whether or not the age of documents matters significantly (in some contexts, the most recent documents might always be the most important) Keeping all these factors in mind, it's often helpful in the planning stages of a Solr deployment to sketch out the types of responses you think the search application should return for sample queries. Once the application is up and running, you can employ a series of testing methodologies, such as focus groups, in-house testing, TREC te sts and A/B testing to fine tune the configuration of the application to best meet the needs of its users. For more information about relevance, see Grant Ingersoll's tech article Debugging Search Application Relevance Issues which is available on SearchHub.org. Query Syntax and Parsing Solr supports several query parsers, offering search application designers great flexibility in controlling how queries are parsed. This section explains how to specify the query parser to be used. It also describes the syntax and features supported by the main query parsers included with Solr and describes some other parsers that may be useful for particular situations. There are some query parameters common to all Solr parsers; these are discussed in the section Common Query Parameters. The parsers discussed in this Guide are: The Standard Query Parser The DisMax Query Parser The Extended DisMax Query Parser Apache Solr Reference Guide 5.1 240 Other Parsers The query parser plugins are all subclasses of QParserPlugin. If you have custom parsing needs, you may want to extend that class to create your own query parser. For more detailed information about the many query parsers available in Solr, see https://wiki.apache.org/solr/So lrQuerySyntax. Common Query Parameters The table below summarizes Solr's common query parameters, which are supported by the Standard, DisMax, and eDisMax Request Handlers. Parameter Description defType Selects the query parser to be used to process the query. sort Sorts the response to a query in either ascending or descending order based on the response's score or another specified characteristic. start Specifies an offset (by default, 0) into the responses at which Solr should begin displaying content. rows Controls how many rows of responses are displayed at a time (default value: 10) fq Applies a filter query to the search results. fl Limits the information included in a query response to a specified list of fields. The fields need to have been indexed as stored for this parameter to work correctly. debug Request additional debugging information in the response. Specifying the debug=timing pa rameter returns just the timing information; specifying the debug=results parameter returns "explain" information for each of the documents returned; specifying the debug=quer y parameter returns all of the debug information. explainOther Allows clients to specify a Lucene query to identify a set of documents. If non-blank, the explain info of each document which matches this query, relative to the main query (specified by the q parameter) will be returned along with the rest of the debugging information. timeAllowed Defines the time allowed for the query to be processed. If the time elapses before the query response is complete, partial information may be returned. omitHeader Excludes the header from the returned results, if set to true. The header contains information about the request, such as the time the request took to complete. The default is false. wt Specifies the Response Writer to be used to format the query response. logParamsList By default, Solr logs all parameters. Set this parameter to restrict which parameters are logged. Valid entries are the parameters to be logged, separated by commas (i.e., logParam sList=param1,param2). An empty list will log no parameters, so if logging all parameters is desired, do not define this additional parameter at all. The following sections describe these parameters in detail. The defType Parameter Apache Solr Reference Guide 5.1 241 The defType parameter selects the query parser that Solr should use to process the main query parameter ( q) in the request. For example: defType=dismax If no defType param is specified, then by default, the The Standard Query Parser is used. (eg: defType=lucen e) The sort Parameter The sort parameter arranges search results in either ascending (asc) or descending (desc) order. The parameter can be used with either numerical or alphabetical content. The directions can be entered in either all lowercase or all uppercase letters (i.e., both asc or ASC). Solr can sort query responses according to document scores or the value of any indexed field with a single value (that is, any field whose attributes in schema.xml include multiValued="false" and indexed="true"), provided that: the field is non-tokenized (that is, the field has no analyzer and its contents have been parsed into tokens, which would make the sorting inconsistent), or the field uses an analyzer (such as the KeywordTokenizer) that produces only a single term. If you want to be able to sort on a field whose contents you want to tokenize to facilitate searching, use the directive in the schema.xml file to clone the field. Then search on the field and sort on its clone. The table explains how Solr responds to various settings of the sort parameter. Example Result If the sort parameter is omitted, sorting is performed as though the parameter were set to score desc. score desc Sorts in descending order from the highest score to the lowest score. price asc Sorts in ascending order of the price field inStock desc, price asc Sorts by the contents of the inStock field in descending order, then within those results sorts in ascending order by the contents of the price field. Regarding the sort parameter's arguments: A sort ordering must include a field name (or score as a pseudo field), followed by whitespace (escaped as + or %20 in URL strings), followed by a sort direction (asc or desc). Multiple sort orderings can be separated by a comma, using this syntax: sort=+,+],... When more than one sort criteria is provided, the second entry will only be used if the first entry results in a tie. If there is a third entry, it will only be used if the first AND second entries are tied. This pattern continues with further entries. The start Parameter When specified, the start parameter specifies an offset into a query's result set and instructs Solr to begin displaying results from this offset. Apache Solr Reference Guide 5.1 242 The default value is "0". In other words, by default, Solr returns results without an offset, beginning where the results themselves begin. Setting the start parameter to some other number, such as 3, causes Solr to skip over the preceding records and start at the document identified by the offset. You can use the start parameter this way for paging. For example, if the rows parameter is set to 10, you could display three successive pages of results by setting start to 0, then re-issuing the same query and setting start to 10, then issuing the query again and setting start to 20. The rows Parameter You can use the rows parameter to paginate results from a query. The parameter specifies the maximum number of documents from the complete result set that Solr should return to the client at one time. The default value is 10. That is, by default, Solr returns 10 documents at a time in response to a query. The fq (Filter Query) Parameter The fq parameter defines a query that can be used to restrict the superset of documents that can be returned, without influencing score. It can be very useful for speeding up complex queries, since the queries specified with fq are cached independently of the main query. When a later query uses the same filter, there's a cache hit, and filter results are returned quickly from the cache. When using the fq parameter, keep in mind the following: The fq parameter can be specified multiple times in a query. Documents will only be included in the result if they are in the intersection of the document sets resulting from each instance of the parameter. In the example below, only documents which have a popularity greater then 10 and have a section of 0 will match. fq=popularity:[10 TO *]&fq=section:0 Filter queries can involve complicated Boolean queries. The above example could also be written as a single fq with two mandatory clauses like so: fq=+popularity:[10 TO *] +section:0 The document sets from each filter query are cached independently. Thus, concerning the previous examples: use a single fq containing two mandatory clauses if those clauses appear together often, and use two separate fq parameters if they are relatively independent. (To learn about tuning cache sizes and making sure a filter cache actually exists, see The Well-Configured Solr Instance.) As with all parameters: special characters in an URL need to be properly escaped and encoded as hex values. Online tools are available to help you with URL-encoding. For example: http://meyerweb.com/eric/t ools/dencoder/. The fl (Field List) Parameter The fl parameter limits the information included in a query response to a specified list of fields. The fields need to have been indexed as stored for this parameter to work correctly. Apache Solr Reference Guide 5.1 243 The field list can be specified as a space-separated or comma-separated list of field names. The string "score" can be used to indicate that the score of each document for the particular query should be returned as a field. The wildcard character "*" selects all the stored fields in a document. You can also add psuedo-fields, functions and transformers to the field list request. This table shows some basic examples of how to use fl: Field List Result id name price Return only the id, name, and price fields. id,name,price Return only the id, name, and price fields. id name, price Return only the id, name, and price fields. id score Return the id field and the score. * Return all the fields in each document. This is the default value of the fl parameter. * score Return all the fields in each document, along with each field's score. Function Values Functions can be computed for each document in the result and returned as a psuedo-field: fl=id,title,product(price,popularity) Document Transformers Document Transformers can be used to modify the information returned about each documents in the results of a query: fl=id,title,[explain] Field Name Aliases You can change the key used to in the response for a field, function, or transformer by prefixing it with a "displ ayName:". For example: fl=id,sales_price:price,secret_sauce:prod(price,popularity),why_score:[explain style=nl] Apache Solr Reference Guide 5.1 244 "response":{"numFound":2,"start":0,"docs":[ { "id":"6H500F0", "secret_sauce":2100.0, "sales_price":350.0, "why_score":{ "match":true, "value":1.052226, "description":"weight(features:cache in 2) [DefaultSimilarity], result of:", "details":[{ ... The debug Parameter The debug parameter can be specified multiple times and supports the following arguments: debug=query: return debug information about the query only. debug=timing: return debug information about how long the query took to process. debug=results: return debug information about the score results (also known as "explain") debug=all: return all available debug information about the request request. (alternatively usage: debug =true) For backwards compatibility with older versions of Solr, debugQuery=true may instead be specified as an alternative way to indicate debug=all The default behavior is not to include debugging information. The explainOther Parameter The explainOther parameter specifies a Lucene query in order to identify a set of documents. If this parameter is included and is set to a non-blank value, the query will return debugging information, along with the "explain info" of each document that matches the Lucene query, relative to the main query (which is specified by the q parameter). For example: q=supervillians&debugQuery=on&explainOther=id:juggernaut The query above allows you to examine the scoring explain info of the top matching documents, compare it to the explain info for documents matching id:juggernaut, and determine why the rankings are not as you expect. The default value of this parameter is blank, which causes no extra "explain info" to be returned. The timeAllowed Parameter This parameter specifies the amount of time, in milliseconds, allowed for a search to complete. If this time expires before the search is complete, any partial results will be returned. The omitHeader Parameter This parameter may be set to either true or false. Apache Solr Reference Guide 5.1 245 If set to true, this parameter excludes the header from the returned results. The header contains information about the request, such as the time it took to complete. The default value for this parameter is false. The wt Parameter The wt parameter selects the Response Writer that Solr should use to format the query's response. For detailed descriptions of Response Writers, see Response Writers. The cache=false Parameter Solr caches the results of all queries and filter queries by default. To disable result caching, set the cache=fals e parameter. You can also use the cost option to control the order in which non-cached filter queries are evaluated. This allows you to order less expensive non-cached filters before expensive non-cached filters. For very high cost filters, if cache=false and cost>=100 and the query implements the PostFilter interfac e, a Collector will be requested from that query and used to filter documents after they have matched the main query and all other filter queries. There can be multiple post filters; they are also ordered by cost. For example: // normal function range query used as a filter, all matching documents // generated up front and cached fq={!frange l=10 u=100}mul(popularity,price) // function range query run in parallel with the main query like a traditional // lucene filter fq={!frange l=10 u=100 cache=false}mul(popularity,price) // function range query checked after each document that already matches the query // and all other filters. Good for really expensive function queries. fq={!frange l=10 u=100 cache=false cost=100}mul(popularity,price) The logParamsList Parameter By default, Solr logs all parameters of requests. From version 4.7, set this parameter to restrict which parameters of a request are logged. This may help control logging to only those parameters considered important to your organization. For example, you could define this like: logParamsList=q,fq And only the 'q' and 'fq' parameters will be logged. If no parameters should be logged, you can send logParamsList as empty (i.e., logParamsList=). This parameter does not only apply to query requests, but to any kind of request to Solr. The Standard Query Parser Solr's default Query Parser is also known as the "lucene" parser. The key advantage of the standard query parser is that it supports a robust and fairly intuitive syntax allowing Apache Solr Reference Guide 5.1 246 you to create a variety of structured queries. The largest disadvantage is that it's very intolerant of syntax errors, as compared with something like the DisMax query parser which is designed to throw as few errors as possible. Topics covered in this section: Standard Query Parser Parameters The Standard Query Parser's Response Specifying Terms for the Standard Query Parser Specifying Fields in a Query to the Standard Query Parser Boolean Operators Supported by the Standard Query Parser Grouping Terms to Form Sub-Queries Differences between Lucene Query Parser and the Solr Standard Query Parser Related Topics Standard Query Parser Parameters In addition to the Common Query Parameters, Faceting Parameters, Highlighting Parameters, and MoreLikeThis Parameters, the standard query parser supports the parameters described in the table below. Parameter Description q Defines a query using standard query syntax. This parameter is mandatory. q.op Specifies the default operator for query expressions, overriding the default operator specified in the schema.xml file. Possible values are "AND" or "OR". df Specifies a default field, overriding the definition of a default field in the schema.xml file. Default parameter values are specified in solrconfig.xml, or overridden by query-time values in the request. The Standard Query Parser's Response By default, the response from the standard query parser contains one block, which is unnamed. If the debug parameter is used, then an additional block will be returned, using the name "debug". This will contain useful debugging info, including the original query string, the parsed query string, and explain info for each document in the block. If the explainOther parameter is also used, then additional explain info will be provided for all the documents matching that query. Sample Responses This section presents examples of responses from the standard query parser. The URL below submits a simple query and requests the XML Response Writer to use indentation to make the XML response more readable. http://localhost:8983/solr/techproducts/select?q=id:SP2514N Results: Apache Solr Reference Guide 5.1 247 01 electronicshard drive 7200RPM, 8MB cache, IDE Ultra ATA-133 NoiseGuard, SilentSeek technology, Fluid Dynamic Bearing (FDB) motor SP2514N true Samsung Electronics Co. Ltd. Samsung SpinPoint P120 SP2514N - hard drive - 250 GB ATA-133 6 92.0 SP2514N Here's an example of a query with a limited field list. http://localhost:8983/solr/techproducts/select?q=id:SP2514N&fl=id+name Results: 02 SP2514N Samsung SpinPoint P120 SP2514N - hard drive - 250 GB ATA-133 Specifying Terms for the Standard Query Parser A query to the standard query parser is broken up into terms and operators. There are two types of terms: single terms and phrases. A single term is a single word such as "test" or "hello" A phrase is a group of words surrounded by double quotes such as "hello dolly" Multiple terms can be combined together with Boolean operators to form more complex queries (as described below). Apache Solr Reference Guide 5.1 248 It is important that the analyzer used for queries parses terms and phrases in a way that is consistent with the way the analyzer used for indexing parses terms and phrases; otherwise, searches may produce unexpected results. Term Modifiers Solr supports a variety of term modifiers that add flexibility or precision, as needed, to searches. These modifiers include wildcard characters, characters for making a search "fuzzy" or more general, and so on. The sections below describe these modifiers in detail. Wildcard Searches Solr's standard query parser supports single and multiple character wildcard searches within single terms. Wildcard characters can be applied to single terms, but not to search phrases. Wildcard Search Type Special Character Single character (matches a single character) ? Multiple characters (matches zero or more sequential characters) * Example The search string te?t would match both test and text. The wildcard search: tes* would match test, testing, and tester. You can also use wildcard characters in the middle of a term. For example: te*t would match test and text. *est would match pest and test. Fuzzy Searches Solr's standard query parser supports fuzzy searches based on the Damerau-Levenshtein Distance or Edit Distance algorithm. Fuzzy searches discover terms that are similar to a specified term without necessarily being an exact match. To perform a fuzzy search, use the tilde ~ symbol at the end of a single-word term. For example, to search for a term similar in spelling to "roam," use the fuzzy search: roam~ This search will match terms like roams, foam, & foams. It will also match the word "roam" itself. An optional distance parameter specifies the maximum number of edits allowed, between 0 and 2, defaulting to Apache Solr Reference Guide 5.1 249 2. For example: roam~1 This will match terms like roams & foam - but not foams since it has an edit distance of "2". In many cases, stemming (reducing terms to a common stem) can produce similar effects to fuzzy searches and wildcard searches. Proximity Searches A proximity search looks for terms that are within a specific distance from one another. To perform a proximity search, add the tilde character ~ and a numeric value to the end of a search phrase. For example, to search for a "apache" and "jakarta" within 10 words of each other in a document, use the search: "jakarta apache"~10 The distance referred to here is the number of term movements needed to match the specified phrase. In the example above, if "apache" and "jakarta" were 10 spaces apart in a field, but "apache" appeared before "jakarta", more than 10 term movements would be required to move the terms together and position "apache" to the right of "jakarta" with a space in between. Range Searches A range search specifies a range of values for a field (a range with an upper bound and a lower bound). The query matches documents whose values for the specified field or fields fall within the range. Range queries can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexicographically, except on numeric fields. For example, the range query below matches all documents whose mod_date field has a value between 20020101 and 20030101, inclusive. mod_date:[20020101 TO 20030101] Range queries are not limited to date fields or even numerical fields. You could also use range queries with non-date fields: title:{Aida TO Carmen} This will find all documents whose titles are between Aida and Carmen, but not including Aida and Carmen. The brackets around a query determine its inclusiveness. Square brackets [ ] denote an inclusive range query that matches values including the upper and lower bound. Curly brackets { } denote an exclusive range query that matches values between the upper and lower bounds, but excluding the upper and lower bounds themselves. You can mix these types so one end of the range is inclusive and the other is exclusive. Here's an example: count:{1 TO 10] Boosting a Term with ^ Lucene/Solr provides the relevance level of matching documents based on the terms found. To boost a term use the caret symbol ^ with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be. Apache Solr Reference Guide 5.1 250 Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for "jakarta apache" and you want the term "jakarta" to be more relevant, you can boost it by adding the ^ symbol along with the boost factor immediately after the term. For example, you could type: jakarta^4 apache This will make documents with the term jakarta appear more relevant. You can also boost Phrase Terms as in the example: "jakarta apache"^4 "Apache Lucene" By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (for example, it could be 0.2). Constant Score with ^= Constant score queries are created with ^=, which sets the entire clause to the specified score for any documents matching that clause. This is desirable when you only care about matches for a particular clause and don't want other relevancy factors such as term frequency (the number of times the term appears in the field) or inverse document frequency (a measure across the whole index for how rare a term is in a field). Example: (description:blue OR color:blue)^=1.0 text:shoes Specifying Fields in a Query to the Standard Query Parser Data indexed in Solr is organized in fields, which are defined in the Solr schema.xml file. Searches can take advantage of fields to add precision to queries. For example, you can search for a term only in a specific field, such as a title field. The schema.xml file defines one field as a default field. If you do not specify a field in a query, Solr searches only the default field. Alternatively, you can specify a different field or a combination of fields in a query. To specify a field, type the field name followed by a colon ":" and then the term you are searching for within the field. For example, suppose an index contains two fields, title and text,and that text is the default field. If you want to find a document called "The Right Way" which contains the text "don't go this way," you could include either of the following terms in your search query: title:"The Right Way" AND text:go title:"Do it right" AND go Since text is the default field, the field indicator is not required; hence the second query above omits it. The field is only valid for the term that it directly precedes, so the query title:Do it right will find only "Do" in the title field. It will find "it" and "right" in the default field (in this case the text field). Boolean Operators Supported by the Standard Query Parser Boolean operators allow you to apply Boolean logic to queries, requiring the presence or absence of specific Apache Solr Reference Guide 5.1 251 terms or conditions in fields in order to match documents. The table below summarizes the Boolean operators supported by the standard query parser. Boolean Operator Alternative Symbol Description AND && Requires both terms on either side of the Boolean operator to be present for a match. NOT ! Requires that the following term not be present. OR || Requires that either term (or both terms) be present for a match. + Requires that the following term be present. - Prohibits the following term (that is, matches on fields or documents that do not include that term). The - operator is functionally similar to the Boolean operator !. Because it's used by popular search engines such as Google, it may be more familiar to some user communities. Boolean operators allow terms to be combined through logic operators. Lucene supports AND, " +", OR, NOT and "-" as Boolean operators. When specifying Boolean operators with keywords such as AND or NOT, the keywords must appear in all uppercase. The standard query parser supports all the Boolean operators listed in the table above. The DisMax query parser supports only + and -. The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR. In the schema.xml file, you can specify which symbols can take the place of Boolean operators such as OR. To search for documents that contain either "jakarta apache" or just "jakarta," use the query: "jakarta apache" jakarta or "jakarta apache" OR jakarta The Boolean Operator + The + symbol (also known as the "required" operator) requires that the term after the + symbol exist somewhere in a field in at least one document in order for the query to return a match. For example, to search for documents that must contain "jakarta" and that may or may not contain "lucene," use the following query: +jakarta lucene Apache Solr Reference Guide 5.1 252 This operator is supported by both the standard query parser and the DisMax query parser. The Boolean Operator AND (&&) The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND. To search for documents that contain "jakarta apache" and "Apache Lucene," use either of the following queries: "jakarta apache" AND "Apache Lucene" "jakarta apache" && "Apache Lucene" The Boolean Operator NOT (!) The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT. The following queries search for documents that contain the phrase "jakarta apache" but do not contain the phrase "Apache Lucene": "jakarta apache" NOT "Apache Lucene" "jakarta apache" ! "Apache Lucene" The Boolean Operator The - symbol or "prohibit" operator excludes documents that contain the term after the - symbol. For example, to search for documents that contain "jakarta apache" but not "Apache Lucene," use the following query: "jakarta apache" -"Apache Lucene" Escaping Special Characters Solr gives the following characters special meaning when they appear in a query: + - && || ! ( ) { } [ ] ^ " ~ * ? : / To make Solr interpret any of these characters literally, rather as a special character, precede the character with a backslash character \. For example, to search for (1+1):2 without having Solr interpret the plus sign and parentheses as special characters for formulating a sub-query with two terms, escape the characters by preceding each one with a backslash: \(1\+1\)\:2 Grouping Terms to Form Sub-Queries Lucene/Solr supports using parentheses to group clauses to form sub-queries. This can be very useful if you want to control the Boolean logic for a query. The query below searches for either "jakarta" or "apache" and "website": Apache Solr Reference Guide 5.1 253 (jakarta OR apache) AND website This adds precision to the query, requiring that the term "website" exist, along with either term "jakarta" and "apache." Grouping Clauses within a Field To apply two or more Boolean operators to a single field in a search, group the Boolean clauses within parentheses. For example, the query below searches for a title field that contains both the word "return" and the phrase "pink panther": title:(+return +"pink panther") Differences between Lucene Query Parser and the Solr Standard Query Parser Solr's standard query parser differs from the Lucene Query Parser in the following ways: A * may be used for either or both endpoints to specify an open-ended range query field:[* TO 100] finds all field values less than or equal to 100 field:[100 TO *] finds all field values greater than or equal to 100 field:[* TO *] matches all documents with the field Pure negative queries (all clauses prohibited) are allowed (only as a top-level clause) -inStock:false finds all field values where inStock is not false -field:[* TO *] finds all documents without a value for field A hook into FunctionQuery syntax. You'll need to use quotes to encapsulate the function if it includes parentheses, as shown in the second example below: _val_:myfield _val_:"recip(rord(myfield),1,2,3)" Support for using any type of query parser as a nested clause. inStock:true OR {!dismax qf='name manu' v='ipod'} Range queries ("[a TO z]"), prefix queries ("a*"), and wildcard queries ("a*b") are constant-scoring (all matching documents get an equal score). The scoring factors TF, IDF, index boost, and "coord" are not used. There is no limitation on the number of terms that match (as there was in past versions of Lucene). Specifying Dates and Times Queries against fields using the TrieDateField type (typically range queries) should use the appropriate date syntax: timestamp:[* TO NOW] createdate:[1976-03-06T23:59:59.999Z TO *] createdate:[1995-12-31T23:59:59.999Z TO 2007-03-06T00:00:00Z] pubdate:[NOW-1YEAR/DAY TO NOW/DAY+1DAY] createdate:[1976-03-06T23:59:59.999Z TO 1976-03-06T23:59:59.999Z+1YEAR] createdate:[1976-03-06T23:59:59.999Z/YEAR TO 1976-03-06T23:59:59.999Z] Related Topics Local Parameters in Queries Other Parsers Apache Solr Reference Guide 5.1 254 The DisMax Query Parser The DisMax query parser is designed to process simple phrases (without complex syntax) entered by users and to search for individual terms across several fields using different weighting (boosts) based on the significance of each field. Additional options enable users to influence the score based on rules specific to each use case (independent of user input). In general, the DisMax query parser's interface is more like that of Google than the interface of the 'standard' Solr request handler. This similarity makes DisMax the appropriate query parser for many consumer applications. It accepts a simple syntax, and it rarely produces error messages. The DisMax query parser supports an extremely simplified subset of the Lucene QueryParser syntax. As in Lucene, quotes can be used to group phrases, and +/- can be used to denote mandatory and optional clauses. All other Lucene query parser special characters (except AND and OR) are escaped to simplify the user experience. The DisMax query parser takes responsibility for building a good query from the user's input using Boolean clauses containing DisMax queries across fields and boosts specified by the user. It also lets the Solr administrator provide additional boosting queries, boosting functions, and filtering queries to artificially affect the outcome of all searches. These options can all be specified as default parameters for the handler in the solrco nfig.xml file or overridden in the Solr query URL. Interested in the technical concept behind the DisMax name? DisMax stands for Maximum Disjunction. Here's a definition of a Maximum Disjunction or "DisMax" query: A query that generates the union of documents produced by its subqueries, and that scores each document with the maximum score for that document as produced by any subquery, plus a tie breaking increment for any additional matching subqueries. Whether or not you remember this explanation, do remember that the DisMax request handler was primarily designed to be easy to use and to accept almost any input without returning an error. DisMax Parameters In addition to the common request parameter, highlighting parameters, and simple facet parameters, the DisMax query parser supports the parameters described below. Like the standard query parser, the DisMax query parser allows default parameter values to be specified in solrconfig.xml, or overridden by query-time values in the request. Parameter Description q Defines the raw input strings for the query. q.alt Calls the standard query parser and defines query input strings, when the q parameter is not used. qf Query Fields: specifies the fields in the index on which to perform the query. If absent, defaults to df. Apache Solr Reference Guide 5.1 255 mm Minimum "Should" Match: specifies a minimum number of clauses that must match in a query. If no 'mm' parameter is specified in the query, or as a default in solrconfig.xml, the effective value of the q.op parameter (either in the query, as a default in solrconfig.xml, or from the 'defaultOperator' option in schema.xml) is used to influence the behavior. If q.op is effectively AND'ed, then mm=100%; if q.op is OR'ed, then mm=1. Users who want to force the legacy behavior should set a default value for the 'mm' parameter in their solrconfig.xml file. Users should add this as a configured default for their request handlers. This parameter tolerates miscellaneous white spaces in expressions (e.g., " 3 < -25% 10 < -3\n", " \n-25%\n ", " \n3\n "). pf Phrase Fields: boosts the score of documents in cases where all of the terms in the q parameter appear in close proximity. ps Phrase Slop: specifies the number of positions two terms can be apart in order to match the specified phrase. qs Query Phrase Slop: specifies the number of positions two terms can be apart in order to match the specified phrase. Used specifically with the qf parameter. tie Tie Breaker: specifies a float value (which should be something much less than 1) to use as tiebreaker in DisMax queries. bq Boost Query: specifies a factor by which a term or phrase should be "boosted" in importance when considering a match. bf Boost Functions: specifies functions to be applied to boosts. (See for details about function queries.) The sections below explain these parameters in detail. The q Parameter The q parameter defines the main "query" constituting the essence of the search. The parameter supports raw input strings provided by users with no special escaping. The + and - characters are treated as "mandatory" and "prohibited" modifiers for terms. Text wrapped in balanced quote characters (for example, "San Jose") is treated as a phrase. Any query containing an odd number of quote characters is evaluated as if there were no quote characters at all. The q parameter does not support wildcard characters such as *. The q.alt Parameter If specified, the q.alt parameter defines a query (which by default will be parsed using standard query parsing syntax) when the main q parameter is not specified or is blank. The q.alt parameter comes in handy when you need something like a query to match all documents (don't forget &rows=0 for that one!) in order to get collection-wise faceting counts. The qf (Query Fields) Parameter The qf parameter introduces a list of fields, each of which is assigned a boost factor to increase or decrease that particular field's importance in the query. For example, the query below: Apache Solr Reference Guide 5.1 256 qf="fieldOne^2.3 fieldTwo fieldThree^0.4" assigns fieldOne a boost of 2.3, leaves fieldTwo with the default boost (because no boost factor is specified), and fieldThree a boost of 0.4. These boost factors make matches in fieldOne much more significant than matches in fieldTwo, which in turn are much more significant than matches in fieldThree. The mm (Minimum Should Match) Parameter When processing queries, Lucene/Solr recognizes three types of clauses: mandatory, prohibited, and "optional" (also known as "should" clauses). By default, all words or phrases specified in the q parameter are treated as "optional" clauses unless they are preceded by a "+" or a "-". When dealing with these "optional" clauses, the mm parameter makes it possible to say that a certain minimum number of those clauses must match. The DisMax query parser offers great flexibility in how the minimum number can be specified. The table below explains the various ways that mm values can be specified. Syntax Example Description Positive integer 3 Defines the minimum number of clauses that must match, regardless of how many clauses there are in total. Negative integer -2 Sets the minimum number of matching clauses to the total number of optional clauses, minus this value. Percentage 75% Sets the minimum number of matching clauses to this percentage of the total number of optional clauses. The number computed from the percentage is rounded down and used as the minimum. Negative percentage -25% Indicates that this percent of the total number of optional clauses can be missing. The number computed from the percentage is rounded down, before being subtracted from the total to determine the minimum number. An expression beginning with a positive integer followed by a > or < sign and another value 3<90% Defines a conditional expression indicating that if the number of optional clauses is equal to (or less than) the integer, they are all required, but if it's greater than the integer, the specification applies. In this example: if there are 1 to 3 clauses they are all required, but for 4 or more clauses only 90% are required. Multiple conditional expressions involving > or < signs 2<-25% 9<-3 Defines multiple conditions, each one being valid only for numbers greater than the one before it. In the example at left, if there are 1 or 2 clauses, then both are required. If there are 3-9 clauses all but 25% are required. If there are more then 9 clauses, all but three are required. When specifying mm values, keep in mind the following: When dealing with percentages, negative values can be used to get different behavior in edge cases. 75% and -25% mean the same thing when dealing with 4 clauses, but when dealing with 5 clauses 75% means 3 are required, but -25% means 4 are required. If the calculations based on the parameter arguments determine that no optional clauses are needed, the usual rules about Boolean queries still apply at search time. (That is, a Boolean query containing no required clauses must still match at least one optional clause). No matter what number the calculation arrives at, Solr will never use a value greater than the number of Apache Solr Reference Guide 5.1 257 optional clauses, or a value less than 1. In other words, no matter how low or how high the calculated result, the minimum number of required matches will never be less than 1 or greater than the number of clauses. When searching across multiple fields that are configured with different query analyzers, the number of optional clauses may differ between the fields. In such a case, the value specified by mm applies to the maximum number of optional clauses. For example, if a query clause is treated as stopword for one of the fields, the number of optional clauses for that field will be smaller than for the other fields. A query with such a stopword clause would not return a match in that field if mm is set to 100% because the removed clause does not count as matched. The default value of mm is 100% (meaning that all clauses must match). The pf (Phrase Fields) Parameter Once the list of matching documents has been identified using the fq and qf parameters, the pf parameter can be used to "boost" the score of documents in cases where all of the terms in the q parameter appear in close proximity. The format is the same as that used by the qf parameter: a list of fields and "boosts" to associate with each of them when making phrase queries out of the entire q parameter. The ps (Phrase Slop) Parameter The ps parameter specifies the amount of "phrase slop" to apply to queries specified with the pf parameter. Phrase slop is the number of positions one token needs to be moved in relation to another token in order to match a phrase specified in a query. The qs (Query Phrase Slop) Parameter The qs parameter specifies the amount of slop permitted on phrase queries explicitly included in the user's query string with the qf parameter. As explained above, slop refers to the number of positions one token needs to be moved in relation to another token in order to match a phrase specified in a query. The tie (Tie Breaker) Parameter The tie parameter specifies a float value (which should be something much less than 1) to use as tiebreaker in DisMax queries. When a term from the user's input is tested against multiple fields, more than one field may match. If so, each field will generate a different score based on how common that word is in that field (for each document relative to all other documents). The tie parameter lets you control how much the final score of the query will be influenced by the scores of the lower scoring fields compared to the highest scoring field. A value of "0.0" makes the query a pure "disjunction max query": that is, only the maximum scoring subquery contributes to the final score. A value of "1.0" makes the query a pure "disjunction sum query" where it doesn't matter what the maximum scoring sub query is, because the final score will be the sum of the subquery scores. Typically a low value, such as 0.1, is useful. The bq (Boost Query) Parameter The bq parameter specifies an additional, optional, query clause that will be added to the user's main query to influence the score. For example, if you wanted to add a relevancy boost for recent documents: Apache Solr Reference Guide 5.1 258 q=cheese bq=date:[NOW/DAY-1YEAR TO NOW/DAY] You can specify multiple bq parameters. If you want your query to be parsed as separate clauses with separate boosts, use multiple bq parameters. The bf (Boost Functions) Parameter The bf parameter specifies functions (with optional boosts) that will be used to construct FunctionQueries which will be added to the user's main query as optional clauses that will influence the score. Any function supported natively by Solr can be used, along with a boost value. For example: recip(rord(myfield),1,2,3)^1.5 Specifying functions with the bf parameter is essentially just shorthand for using the bq param combined with the {!func} parser. For example, if you want to show the most recent documents first, you could use either of the following: bf=recip(rord(creationDate),1,1000,1000) ...or... bq={!func}recip(rord(creationDate),1,1000,1000) Examples of Queries Submitted to the DisMax Query Parser All of the sample URLs in this section assume you are running Solr's "techproducts" example: bin/solr -e techproducts Normal results for the word "video" using the StandardRequestHandler with the default search field: http://localhost:8983/solr/techproducts/select?q=video&fl=name+score The "dismax" handler is configured to search across the text, features, name, sku, id, manu, and cat fields all with varying boosts designed to ensure that "better" matches appear first, specifically: documents which match on the name and cat fields get higher scores. http://localhost:8983/solr/techproducts/select?defType=dismax&q=video Note that this instance is also configured with a default field list, which can be overridden in the URL. http://localhost:8983/solr/techproducts/select?defType=dismax&q=video&fl=*,scor e You can also override which fields are searched on and how much boost each field gets. http://localhost:8983/solr/techproducts/select?defType=dismax&q=video&qf=featur es^20.0+text^0.3 You can boost results that have a field that matches a specific value. http://localhost:8983/solr/techproducts/select?defType=dismax&q=video&bq=cat:el Apache Solr Reference Guide 5.1 259 ectronics^5.0 Another instance of the handler is registered using the qt "instock" and has slightly different configuration options, notably: a filter for (you guessed it) inStock:true). http://localhost:8983/solr/techproducts/select?defType=dismax&q=video&fl=name,s core,inStock http://localhost:8983/solr/techproducts/select?defType=dismax&q=video&qt=instoc k&fl=name,score,inStock One of the other really cool features in this handler is robust support for specifying the "BooleanQuery.minimumNumberShouldMatch" you want to be used based on how many terms are in your user's query. These allows flexibility for typos and partial matches. For the dismax handler, one and two word queries require that all of the optional clauses match, but for three to five word queries one missing word is allowed. http://localhost:8983/solr/techproducts/select?defType=dismax&q=belkin+ipod http://localhost:8983/solr/techproducts/select?defType=dismax&q=belkin+ipod+gib berish http://localhost:8983/solr/techproducts/select?defType=dismax&q=belkin+ipod+app le Just like the StandardRequestHandler, it supports the debugQuery option to viewing the parsed query, and the score explanations for each document. http://localhost:8983/solr/techproducts/select?defType=dismax&q=belkin+ipod+gib berish&debugQuery=true http://localhost:8983/solr/techproducts/select?defType=dismax&q=video+card&debu gQuery=true The Extended DisMax Query Parser The Extended DisMax (eDisMax) query parser is an improved version of the DisMax query parser. In addition to supporting all the DisMax query parser parameters, Extended Dismax: supports the full Lucene query parser syntax. supports queries such as AND, OR, NOT, -, and +. treats "and" and "or" as "AND" and "OR" in Lucene syntax mode. respects the 'magic field' names _val_ and _query_. These are not a real fields in schema.xml, but if used it helps do special things (like a function query in the case of _val_ or a nested query in the case of _query_). If _val_ is used in a term or phrase query, the value is parsed as a function. includes improved smart partial escaping in the case of syntax errors; fielded queries, +/-, and phrase queries are still supported in this mode. improves proximity boosting by using word shingles; you do not need the query to match all words in the document before proximity boosting is applied. includes advanced stopword handling: stopwords are not required in the mandatory part of the query but are still used in the proximity boosting part. If a query consists of all stopwords, such as "to be or not to be", then all words are required. includes improved boost function: in Extended DisMax, the boost function is a multiplier rather than an addend, improving your boost results; the additive boost functions of DisMax ( bf and bq) are also supported. Apache Solr Reference Guide 5.1 260 supports pure negative nested queries: queries such as +foo (-foo) will match all documents. lets you specify which fields the end user is allowed to query, and to disallow direct fielded searches. Extended DisMax Parameters In addition to all the DisMax parameters, Extended DisMax includes these query parameters: The boost Parameter A multivalued list of strings parsed as queries with scores multiplied by the score from the main query for all matching documents. This parameter is shorthand for wrapping the query produced by eDisMax using the Boos tQParserPlugin The lowercaseOperators Parameter A Boolean parameter indicating if lowercase "and" and "or" should be treated the same as operators "AND" and "OR". The ps Parameter Default amount of slop on phrase queries built with pf, pf2 and/or pf3 fields (affects boosting). The pf2 Parameter A multivalued list of fields with optional weights, based on pairs of word shingles. The ps2 Parameter This is similar to ps but overrides the slop factor used for pf2. If not specified, ps is used. The pf3 Parameter A multivalued list of fields with optional weights, based on triplets of word shingles. Similar to pf, except that instead of building a phrase per field out of all the words in the input, it builds a set of phrases for each field out of each triplet of word shingles. The ps3 Parameter This is similar to ps but overrides the slop factor used for pf3. If not specified, ps is used. The stopwords Parameter A Boolean parameter indicating if the StopFilterFactory configured in the query analyzer should be respected when parsing the query: if it is false, then the StopFilterFactory in the query analyzer is ignored. The uf Parameter Specifies which schema fields the end user is allowed to explicitly query. This parameter supports wildcards. The default is to allow all fields, equivalent to uf=*. To allow only title field, use uf=title. To allow title and all fields ending with _s, use uf=title,*_s. To allow all fields except title, use uf=*-title. To disallow all fielded searches, use uf=-*. Field aliasing using per-field qf overrides Apache Solr Reference Guide 5.1 261 Per-field overrides of the qf parameter may be specified to provide 1-to-many aliasing from field names specified in the query string, to field names used in the underlying query. By default, no aliasing is used and field names specified in the query string are treated as literal field names in the index. Examples of Queries Submitted to the Extended DisMax Query Parser All of the sample URLs in this section assume you are running Solr's " techproducts" example: bin/solr -e techproducts Boost the result of the query term "hello" based on the document's popularity: http://localhost:8983/solr/techproducts/select?defType=edismax&q=hello&pf=text&qf=te xt&boost=popularity Search for iPods OR video: http://localhost:8983/solr/techproducts/select?defType=edismax&q=ipod+OR+video Search across multiple fields, specifying (via boosts) how important each field is relative each other: http://localhost:8983/solr/techproducts/select?q=video&defType=edismax&qf=features^2 0.0+text^0.3 You can boost results that have a field that matches a specific value: http://localhost:8983/solr/techproducts/select?q=video&defType=edismax&qf=features^2 0.0+text^0.3&bq=cat:electronics^5.0 Using the "mm" param, 1 and 2 word queries require that all of the optional clauses match, but for queries with three or more clauses one missing clause is allowed: http://localhost:8983/solr/techproducts/select?q=belkin+ipod&defType=edismax&mm=2 http://localhost:8983/solr/techproducts/select?q=belkin+ipod+gibberish&defType=edism ax&mm=2 http://localhost:8983/solr/techproducts/select?q=belkin+ipod+apple&defType=edismax&m m=2 In the example below, we see a per-field override of the qf parameter being used to alias "name" in the query string to either the "last_name" and "first_name" fields: defType=edismax q=sysadmin name:Mike qf=title text last_name first_name f.name.qf=last_name first_name Using negative boost Negative query boosts have been supported at the "Query" object level for a long time (resulting in negative Apache Solr Reference Guide 5.1 262 scores for matching documents). Now the QueryParsers have been updated to handle this too. Using 'slop' Dismax and Edismax can run queries against all query fields, and also run a query in the form of a phrase against the phrase fields. (This will work only for boosting documents, not actually for matching.) However, that phrase query can have a 'slop,' which is the distance between the terms of the query while still considering it a phrase match. For example: q=foo bar qf=field1^5 field2^10 pf=field1^50 field2^20 defType=dismax With these parameters, the Dismax Query Parser generates a query that looks something like this: (+(field1:foo^5 OR field2:bar^10) AND (field1:bar^5 OR field2:bar^10)) But it also generates another query that will only be used for boosting results: field1:"foo bar"^50 OR field2:"foo bar"^20 Thus, any document that has the terms "foo" and "bar" will match; however if some of those documents have both of the terms as a phrase, it will score much higher because it's more relevant. If you add the parameter ps (phrase slop), the second query will instead be: ps=10 field1:"foo bar"~10^50 OR field2:"foo bar"~10^20 This means that if the terms "foo" and "bar" appear in the document with less than 10 terms between each other, the phrase will match. For example the doc that says: *Foo* term1 term2 term3 *bar* will match the phrase query. How does one use phrase slop? Usually it is configured in the request handler (in solrconfig). With query slop (qs) the concept is similar, but it applies to explicit phrase queries from the user. For example, if you want to search for a name, you could enter: q="Hans Anderson" A document that contains "Hans Anderson" will match, but a document that contains the middle name "Christian" or where the name is written with the last name first ("Anderson, Hans") won't. For those cases one could configure the query field qs, so that even if the user searches for an explicit phrase query, a slop is applied. Finally, edismax contains not only a phrase fields (pf) parameters, but also phrase and query fields 2 and 3. You can use those fields for setting different fields or boosts. Each of those can use a different phrase slop. Apache Solr Reference Guide 5.1 263 Using the 'magic fields' _val_ and _query_ If the 'magic field' name _val_ is used in a term or phrase query, the value is parsed as a function. The Solr Query Parser's use of _val_ and _query_ differs from the Lucene Query Parser in the following ways: If the magic field name _val_ is used in a term or phrase query, the value is parsed as a function. It provides a hook into FunctionQuery syntax. Quotes are necessary to encapsulate the function when it includes parentheses. For example: _val_:myfield _val_:"recip(rord(myfield),1,2,3)" The Solr Query Parser offers nested query support for any type of query parser (via QParserPlugin). Quotes are often necessary to encapsulate the nested query if it contains reserved characters. For example: _query_:"{!dismax qf=myfield}how now brown cow" Although not technically a syntax difference, note that if you use the Solr TrieDateField type (or the deprecated DateField type), any queries on those fields (typically range queries) should use either the Complete ISO 8601 Date syntax that field supports, or the DateMath Syntax to get relative dates. For example: timestamp:[* TO NOW] createdate:[1976-03-06T23:59:59.999Z TO *] createdate:[1995-12-31T23:59:59.999Z TO 2007-03-06T00:00:00Z] pubdate:[NOW-1YEAR/DAY TO NOW/DAY+1DAY] createdate:[1976-03-06T23:59:59.999Z TO 1976-03-06T23:59:59.999Z+1YEAR] createdate:[1976-03-06T23:59:59.999Z/YEAR TO 1976-03-06T23:59:59.999Z] TO must be uppercase, or Solr will report a 'Range Group' error. Function Queries Function queries enable you to generate a relevancy score using the actual value of one or more numeric fields. Function queries are supported by the DisMax, Extended DisMax, and standard query parsers. Function queries use functions. The functions can be a constant (numeric or string literal), a field, another function or a parameter substitution argument. You can use these functions to modify the ranking of results for users. These could be used to change the ranking of results based on a user's location, or some other calculation. Function query topics covered in this section: Using Function Query Available Functions Example Function Queries Sort By Function Apache Solr Reference Guide 5.1 264 Related Topics Using Function Query Functions must be expressed as function calls (for example, sum(a,b) instead of simply a+b). There are several ways of using function queries in a Solr query: Via an explicit QParser that expects function arguments, such func or frange . For example: q={!func}div(popularity,price)&fq={!frange l=1000}customer_ratings In a Sort expression. For example: sort=div(popularity,price) desc, score desc Add the results of functions as psuedo-fields to documents in query results. For instance, for: &fl=sum(x, y),id,a,b,c,score the output would be: ... foo 40 0.343 ... Use in a parameter that is explicitly for specifying functions, such as the EDisMax query parser's boost p aram, or DisMax query parser's bf (boost function) parameter. (Note that the bf parameter actually takes a list of function queries separated by white space and each with an optional boost. Make sure you eliminate any internal white space in single function queries when using bf). For example: q=dismax&bf="ord(popularity)^0.5 recip(rord(price),1,1000,1000)^0.3" Introduce a function query inline in the lucene QParser with the _val_ keyword. For example: q=_val_:mynumericfield _val_:"recip(rord(myfield),1,2,3)" Only functions with fast random access are recommended. Available Functions The table below summarizes the functions available for function queries. Function abs Description Returns the absolute value of the specified value or function. Apache Solr Reference Guide 5.1 Syntax Examples abs(x) abs(-5) 265 and Returns a value of true if and only if all of its operands evaluate to true. and(not(exists(popularity)),exi sts(price)): returns true for any document which has a value in the price f ield, but does not have a value in the popu larity field "constant" Specifies a floating point constant. 1.5 def def is short for default. Returns the value of def(rating,5): This def() function field "field", or if the field does not exist, returns the default value specified. and yields the first value where exists()==tru returns the rating, or if no rating specified in the doc, returns 5 def(myfield, 1.0): equivalent to if( exists(myfield),myfield,1.0) e.) div dist Divides one value or function by another. div(x,y) divides x by y. div(1,y) Return the distance between two vectors (points) in an n-dimensional space. Takes in the power, plus two or more ValueSource instances and calculates the distances between the two vectors. Each ValueSource must be a number. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector. dist(2, x, y, 0, 0): calculates the div(sum(x,100),max(y,1)) Euclidean distance between (0,0) and (x,y) for each document dist(1, x, y, 0, 0): calculates the Manhattan (taxicab) distance between (0,0) and (x,y) for each document dist(2, x,y,z,0,0,0): Euclidean distance between (0,0,0) and (x,y,z) for each document. dist(1,x,y,z,e,f,g): Euclidean distance between (x,y,z) and (e,f,g) where each letter is a field name docfreq(field,val) Returns the number of documents that contain the term in the field. This is a constant (the same value for all documents in the index). docfreq(text,'solr') ...&defType=func &q=docfreq(text,$myterm) &myterm=solr You can quote the term if it's more complex, or do parameter substitution for the term value. exists Returns TRUE if any member of the field exists. exists(author) returns TRUE for any document has a value in the "author" field. exists(query(price:5.00)) returns TRUE if "price" matches "5.00". Apache Solr Reference Guide 5.1 266 field Returns the numeric field value of an indexed (not multi-valued) field with a maximum of one value per document. The f ield() function can be called using the myFloatFieldName field("my complex float fieldName") name of the field as a string, or for most conventional field names simply use the field name by itself. 0 is returned for documents without a value in the field. hsin The Haversine distance calculates the distance between two points on a sphere when traveling along the sphere. The values must be in radians. hsin also take a hsin(2, true, x, y, 0, 0) Boolean argument to specify whether the function should convert its output to radians. idf if Inverse document frequency; a measure of whether the term is common or rare across all documents. Obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient. See also tf. idf(fieldName,'solr'): measures Enables conditional function queries. In if( if(termfreq(cat,'electronics'), popularity,42) : test,value1,value2): test is or refers to a logical value or expression that returns a logical value (TRUE or FALSE). value1 is the value that is returned by the function if test yields TRUE. the inverse of the frequency of the occurrence of the term 'solr' in fieldN ame. This function checks each document for the to see if it contains the term "electro nics" in the cat field. If it does, then the value of the popularity field is returned, otherwise the value of 42 is returned. value2 is the value that is returned by the function if test yields FALSE. An expression can be any function which outputs boolean values, or even functions returning numeric values, in which case value 0 will be interpreted as false, or strings, in which case empty string is interpreted as false. linear Implements m*x+c where m and c are linear(x,m,c) constants and x is an arbitrary function. This linear(x,2,4) returns 2*x+4 is equivalent to sum(product(m,x),c), but slightly more efficient as it is implemented as a single function. Apache Solr Reference Guide 5.1 267 log map Returns the log base 10 of the specified function. log(x) Maps any values of an input function x that fall within min and max inclusive to the specified target. The arguments min and max must be constants. The arguments tar map(x,min,max,target) get and default can be constants or functions. If the value of x does not fall between min and max, then either the value of x is returned, or a default value is returned if specified as a 5th argument. log(sum(x,100)) map(x,0,0,1) - changes any values of 0 to 1. This can be useful in handling default 0 values. map(x,min,max,target,default) map(x,0,100,1,-1) - changes any values between 0 and 100 to 1, and all other values to -1. map(x,0,100,sum(x,599),docfreq( text,solr)) - changes any values between 0 and 100 to x+599, and all other values to frequency of the term 'solr' in the field text. max Returns the max of another function and a constant, which are specified as arguments: max(x,c). The max function is useful for max(myfield,0) "bottoming out" another function at some constant. maxdoc Returns the number of documents in the index, including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index). maxdoc() ms Returns milliseconds of difference between its arguments. Dates are relative to the Unix or POSIX time epoch, midnight, January 1, 1970 UTC. Arguments may be the name of an indexed TrieDateField, or date math ms(NOW/DAY) based on a constant date or NOW . :00Z) ms(): Equivalent to ms(NOW), number of ms(2000-01-01T00:00:00Z) ms(mydatefield) ms(NOW,mydatefield) ms(mydatefield,2000-01-01T00:00 ms(datefield1,datefield2) milliseconds since the epoch. ms(a): Returns the number of milliseconds since the epoch that the argument represents. ms(a,b) : Returns the number of milliseconds that b occurs before a (that is, a - b) Apache Solr Reference Guide 5.1 268 norm(field) Returns the "norm" stored in the index for the specified field. This is the product of the index time boost and the length normalization factor, according to the Similar ity for the field. norm(fieldName) not The logically negated value of the wrapped function. not(exists(author)): TRUE only numdocs Returns the number of documents in the index, not including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index). numdocs() or A logical disjunction. or(value1,value2): TRUE if either va when exists(author) is false. lue1 or value2 is true. ord Returns the ordinal of the indexed field value within the indexed list of terms for that field in Lucene index order (lexicographically ordered by unicode value), starting at 1. In other words, for a given field, all values are ordered lexicographically; this function then returns the offset of a particular value in that ordering. The field must have a maximum of one value per document (not multi-valued). 0 is returned for documents without a value in the field. ord(myIndexedField) Example: If there were only three values ("apple","banana","pear") for a particular field X, then: ord(X) would be 1 for documents containing "apple", 2 for documnts containing "banana", etc... ord() depends on the position in an index and can change when other documents are inserted or deleted. See also rord below. pow Raises the specified base to the specified power. pow(x,y) raises x to the power of y. pow(x,y) pow(x,log(y)) pow(x,0.5): the same as sqrt product Returns the product of multiple values or functions, which are specified in a comma-separated list. mul(...) may also be used as an alias for this function. Apache Solr Reference Guide 5.1 product(x,y,...) product(x,2) product(x,y) mul(x,y) 269 query Returns the score for the given subquery, or the default value for documents not matching the query. Any type of subquery is supported through either parameter de-referencing $otherparam or direct specification of the query string in the Local Parameters through the v key. query(subquery, default) q=product(popularity, query({!d ismax v='solr rocks'}): returns the product of the popularity and the score of the DisMax query. q=product(popularity, query($qq ))&qq={!dismax}solr rocks: equivalent to the previous query, using parameter de-referencing. q=product(popularity, query($qq ,0.1))&qq={!dismax}solr rocks: specifies a default score of 0.1 for documents that don't match the DisMax query. recip Performs a reciprocal function with recip(x recip(myfield,m,a,b) ,m,a,b) implementing a/(m*x+b) where recip(rord(creationDate),1,1000 ,1000) m,a,b are constants, and x is any arbitrarily complex function. When a and b are equal, and x>=0, this function has a maximum value of 1 that drops as x increases. Increasing the value of a and b together results in a movement of the entire function to a flatter part of the curve. These properties can make this an ideal function for boosting more recent documents when x is rord(datefield). rord Returns the reverse ordering of that returned by ord. Apache Solr Reference Guide 5.1 rord(myDateField) 270 scale Scales values of the function x such that they fall between the specified minTarget a scale(x,minTarget,maxTarget) nd maxTarget inclusive. The current such that all values will be between 1 and 2 inclusive. implementation traverses all of the function values to obtain the min and max, so it can pick the correct scale. scale(x,1,2): scales the values of x The current implementation cannot distinguish when documents have been deleted or documents that have no value. It uses 0.0 values for these cases. This means that if values are normally all greater than 0.0, one can still end up with 0.0 as the min value to map from. In these cases, an appropriate map() function could be used as a workaround to change 0.0 to a value in the real range, as shown here: scale(map(x,0,0,5),1,2) sqedist The Square Euclidean distance calculates the 2-norm (Euclidean distance) but does not take the square root, thus saving a fairly expensive operation. It is often the case that applications that care about Euclidean distance do not need the actual distance, but instead can use the square of the distance. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector. sqedist(x_td, y_td, 0, 0) sqrt Returns the square root of the specified value or function. sqrt(x)sqrt(100)sqrt(sum(x,100)) Apache Solr Reference Guide 5.1 271 strdist Calculate the distance between two strings. Uses the Lucene spell checker StringDist strdist("SOLR",id,edit) ance interface and supports all of the implementations available in that package, plus allows applications to plug in their own via Solr's resource loading capabilities. strd ist takes (string1, string2, distance measure). Possible values for distance measure are: jw: Jaro-Winkler edit: Levenstein or Edit distance ngram: The NGramDistance, if specified, can optionally pass in the ngram size too. Default is 2. FQN: Fully Qualified class Name for an implementation of the StringDistance interface. Must have a no-arg constructor. sub Returns x-y from sub(x,y). sub(myfield,myfield2) sub(100,sqrt(myfield)) sum Returns the sum of multiple values or functions, which are specified in a comma-separated list. add(...) may be used as an alias for this function sumtotaltermfreq Returns the sum of totaltermfreq values for all terms in the field in the entire index (i.e., the number of indexed tokens for that field). (Aliases sumtotaltermfreq to sttf .) sum(x,y,...) sum(x,1) sum(x,y) sum(sqrt(x),log(y),z,0.5) add(x,y) If doc1:(fieldX:A B C) and doc2:(fieldX:A A A A): docFreq(fieldX:A) = 2 (A appears in 2 docs) freq(doc1, fieldX:A) = 4 (A appears 4 times in doc 2) totalTermFreq(fieldX:A) = 5 (A appears 5 times across all docs) sumTotalTermFreq(fieldX) = 7 in fi eldX, there are 5 As, 1 B, 1 C termfreq Returns the number of times the term appears in the field for that document. Apache Solr Reference Guide 5.1 termfreq(text,'memory') 272 tf Term frequency; returns the term frequency factor for the given term, using the Similarity for the field. The tf-idf value increases tf(text,'solr') proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the document, which helps to control for the fact that some words are generally more common than others. See also idf. top Causes the function query argument to derive its values from the top-level IndexReader containing all parts of an index. For example, the ordinal of a value in a single segment will be different from the ordinal of that same value in the complete index. The ord() and rord() functions implicitly use top(), and hence ord(foo) is equivalent to top(ord(foo)). totaltermfreq Returns the number of times the term appears in the field in the entire index. (Aliases totaltermfreq to ttf.) ttf(text,'memory') xor() Logical exclusive disjunction, or one or the other but not both. xor(field1,field2) returns TRUE if either field1 or field2 is true; FALSE if both are true. Example Function Queries To give you a better understanding of how function queries can be used in Solr, suppose an index stores the dimensions in meters x,y,z of some hypothetical boxes with arbitrary names stored in field boxname. Suppose we want to search for box matching name findbox but ranked according to volumes of boxes. The query parameters would be: q=boxname:findbox _val_:"product(x,y,z)" This query will rank the results based on volumes. In order to get the computed volume, you will need to request the score, which will contain the resultant volume: &fl=*, score Suppose that you also have a field storing the weight of the box as weight. To sort by the density of the box and return the value of the density in score, you would submit the following query: http://localhost:8983/solr/collection_name/select?q=boxname:findbox _val_:"div(weight,product(x,y,z))"&fl=boxname x y z weight score Sort By Function Apache Solr Reference Guide 5.1 273 You can sort your query results by the output of a function. For example, to sort results by distance, you could enter: http://localhost:8983/solr/collection_name/select?q=*:*&sort=dist(2, point1, point2) desc Sort by function also supports pseudo-fields: fields can be generated dynamically and return results as though it was normal field in the index. For example, &fl=id,sum(x, y),score would return: foo 40 0.343 Related Topics FunctionQuery Local Parameters in Queries Local parameters are arguments in a Solr request that are specific to a query parameter. Local parameters provide a way to add meta-data to certain argument types such as query strings. (In Solr documentation, local parameters are sometimes referred to as LocalParams.) Local parameters are specified as prefixes to arguments. Take the following query argument, for example: q=solr rocks We can prefix this query string with local parameters to provide more information to the Standard Query Parser. For example, we can change the default operator type to "AND" and the default field to "title": q={!q.op=AND df=title}solr rocks These local parameters would change the query to require a match on both "solr" and "rocks" while searching the "title" field by default. Basic Syntax of Local Parameters To specify a local parameter, insert the following before the argument to be modified: Begin with {! Insert any number of key=value pairs separated by white space End with } and immediately follow with the query argument You may specify only one local parameters prefix per argument. Values in the key-value pairs may be quoted via single or double quotes, and backslash escaping works within quoted strings. Query Type Short Form If a local parameter value appears without a name, it is given the implicit name of "type". This allows short-form Apache Solr Reference Guide 5.1 274 representation for the type of query parser to use when parsing a query string. Thus q={!dismax qf=myfield}solr rocks is equivalent to: q={!type=dismax qf=myfield}solr rocks If no "type" is specified (either explicitly or implicitly) then the lucene parser is used by default. Thus fq={!df=summary}solr rocks is equivilent to: fq={!type=lucene df=summary}solr rocks Specifying the Parameter Value with the 'v' Key A special key of v within local parameters is an alternate way to specify the value of that parameter. q={!dismax qf=myfield}solr rocks is equivalent to q={!type=dismax qf=myfield v='solr rocks'} Parameter Dereferencing Parameter dereferencing or indirection lets you use the value of another argument rather than specifying it directly. This can be used to simplify queries, decouple user input from query parameters, or decouple front-end GUI parameters from defaults set in solrconfig.xml. q={!dismax qf=myfield}solr rocks is equivalent to: q={!type=dismax qf=myfield v=$qq}&qq=solr rocks Other Parsers In addition to the main query parsers discussed earlier, there are several other query parsers that can be used instead of or in conjunction with the main parsers for specific purposes. This section details the other parsers, and gives examples for how they might be used. Many of these parsers are expressed the same way as Local Parameters in Queries. Query parsers discussed in this section: Block Join Query Parsers Boost Query Parser Collapsing Query Parser Complex Phrase Query Parser Field Query Parser Function Query Parser Function Range Query Parser Join Query Parser Lucene Query Parser Max Score Query Parser Apache Solr Reference Guide 5.1 275 More Like This Query Parser Nested Query Parser Old Lucene Query Parser Prefix Query Parser Raw Query Parser Re-Ranking Query Parser Simple Query Parser Spatial Filter Query Parser Surround Query Parser Switch Query Parser Term Query Parser Terms Query Parser Block Join Query Parsers There are two query parsers that support block joins. These parsers allow indexing and searching for relational content that has been indexed as nested documents. The example usage of the query parsers below assumes these two documents and each of their child documents have been indexed: 1 Solr has block join support parentDocument 2 SolrCloud supports it too! 3 New Lucene and Solr release parentDocument 4 Lots of new features Block Join Children Query Parser This parser takes a query that matches some parent documents and returns their children. The syntax for this parser is: q={!child of=}. The parameter allParents is a filter that matches only parent documents; here you would define the field and value that you used to identify a document as a parent. The parameter someParents identifies a query that will match some or all of the parent documents. The output is the children. Using the example documents above, we can construct a query such as q={!child of="content_type:parentDocument"}title:lucene. We only get one document in response: Apache Solr Reference Guide 5.1 276 4 Lots of new features Block Join Parent Query Parser This parser takes a query that matches child documents and returns their parents. The syntax for this parser is similar: q={!parent which=}. Again the parameter The parameter allPa rents is a filter that matches only parent documents; here you would define the field and value that you used to identify a document as a parent. The parameter someChildren is a query that matches some or all of the child documents. Note that the query for someChildren should match only child documents or you may get an exception. Again using the example documents above, we can construct a query such as q={!parent which="content_type:parentDocument"}comments:SolrCloud. We get this document in response: 1 Solr has block join support parentDocument Boost Query Parser BoostQParser extends the QParserPlugin and creates a boosted query from the input value. The main value is the query to be boosted. Parameter b is the function query to use as the boost. The query to be boosted may be of any type. Examples: Creates a query "foo" which is boosted (scores are multiplied) by the function query log(popularity): {!boost b=log(popularity)}foo Creates a query "foo" which is boosted by the date boosting function referenced in ReciprocalFloatFunctio n: {!boost b=recip(ms(NOW,mydatefield),3.16e-11,1,1)}foo Collapsing Query Parser The CollapsingQParser is really a post filter that provides more performant field collapsing than Solr's standard approach when the number of distinct groups in the result set is high. This parser collapses the result set to a single document per group before it forwards the result set to the rest of the search components. So all downstream components (faceting, highlighting, etc...) will work with the collapsed result set. Apache Solr Reference Guide 5.1 277 Details about using the CollapsingQParser can be found in the section Collapse and Expand Results. Complex Phrase Query Parser The ComplexPhraseQParser provides support for wildcards, ORs, etc., inside phrase queries using Lucene's ComplexPhraseQueryParser . Under the covers, this query parser makes use of the Span group of queries, e.g., spanNear, spanOr, etc., and is subject to the same limitations as that family or parsers. Parameter Description inOrder Set to true to force phrase queries to match terms in the order specified. Default: true df The default search field. Examples: {!complexphrase inOrder=true}name:"Jo* Smith" {!complexphrase inOrder=false}name:"(john jon jonathan~) peters*" A mix of ordered and unordered complex phrase queries: +_query_:"{!complexphrase inOrder=true}manu:\"a* c*\"" +_query_:"{!complexphrase inOrder=false df=name}\"bla* pla*\"" Limitations Performance is sensitive to the number of unique terms that are associated with a pattern. For instance, searching for "a*" will form a large OR clause (technically a SpanOr with many terms) for all of the terms in your index for the indicated field that start with the single letter 'a'. It may be prudent to restrict wildcards to at least two or preferably three letters as a prefix. Allowing very short prefixes may result in to many low-quality documents being returned. MaxBooleanClauses You may need to increase MaxBooleanClauses in solrconfig.xml as a result of the term expansion above: 4096 This property is described in more detail in the section Query Sizing and Warming. Stopwords It is recommended not to use stopword elimination with this query parser. Lets say we add the, up, to to stopwo rds.txt for your collection, and index a document containing the text "Stores up to 15,000 songs, 25,00 photos, or 150 yours of video" in a field named "features". While the query below does not use this parser: Apache Solr Reference Guide 5.1 278 q=features:"Stores up to 15,000" the document is returned. The next query that does use the Complex Phrase Query Parser, as in this query: q=features:"sto* up to 15*"&defType=complexphrase does not return that document because SpanNearQuery has no good way to handle stopwords in a way analogous to PhraseQuery. If you must remove stopwords for your use case, use a custom filter factory or perhaps a customized synonyms filter that reduces given stopwords to some impossible token. Field Query Parser The FieldQParser extends the QParserPlugin and creates a field query from the input value, applying text analysis and constructing a phrase query if appropriate. The parameter f is the field to be queried. Example: {!field f=myfield}Foo Bar This example creates a phrase query with "foo" followed by "bar" (assuming the analyzer for myfield is a text field with an analyzer that splits on whitespace and lowercase terms). This is generally equivalent to the Lucene query parser expression myfield:"Foo Bar". Function Query Parser The FunctionQParser extends the QParserPlugin and creates a function query from the input value. This is only one way to use function queries in Solr; for another, more integrated, approach, see the section on Function Queries. Example: {!func}log(foo) Function Range Query Parser The FunctionRangeQParser extends the QParserPlugin and creates a range query over a function. This is also referred to as frange, as seen in the examples below. Other parameters: Parameter Description l The lower bound, optional u The upper bound, optional incl Include the lower bound: true/false, optional, default=true incu Include the upper bound: true/false, optional, default=true Examples: Apache Solr Reference Guide 5.1 279 {!frange l=1000 u=50000}myfield fq={!frange l=0 u=2.2} sum(user_ranking,editor_ranking) Both of these examples are restricting the results by a range of values found in a declared field or a function query. In the second example, we're doing a sum calculation, and then defining only values between 0 and 2.2 should be returned to the user. For more information about range queries over functions, see Yonik Seeley's introductory blog post Ranges over Functions in Solr 1.4, hosted at SearchHub.org. Join Query Parser JoinQParser extends the QParserPlugin. It allows normalizing relationships between documents with a join operation. This is different from the concept of a join in a relational database because no information is being truly joined. An appropriate SQL analogy would be an "inner query". Examples: Find all products containing the word "ipod", join them against manufacturer docs and return the list of manufacturers: {!join from=manu_id_s to=id}ipod Find all manufacturer docs named "belkin", join them against product docs, and filter the list to only products with a price less than $12: q = {!join from=id to=manu_id_s}compName_s:Belkin fq = price:[* TO 12] Joining Across Collections You can also specify a fromIndex parameter to join with a field from another core or collection. If running in SolrCloud mode, then the collection specified in the fromIndex parameter must have a single shard and a replica on all Solr nodes where the collection you're joining to has a replica. Let's consider an example where you want to use a Solr join query to filter movies by directors that have won an Oscar. Specifically, imagine we have two collections with the following fields: movies: id, title, director_id, ... movie_directors: id, name, has_oscar, ... To filter movies by directors that have won an Oscar using a Solr join on the movie_directors collection, you can send the following filter query to the movies collection: fq={!join from=id fromIndex=movie_directors to=director_id}has_oscar:true Notice that the query criteria of the filter (has_oscar:true) is based on a field in the collection specified using fromIndex. Keep in mind that you cannot return fields from the fromIndex collection using join queries, you Apache Solr Reference Guide 5.1 280 can only use the fields for filtering results in the "to" collection (movies). Next, let's understand how these collections need to be deployed in your cluster. Imagine the movies collection is deployed to a four node SolrCloud cluster and has two shards with a replication factor of two. Specifically, the movies collection has replicas on the following four nodes: node 1: movies_shard1_replica1 node 2: movies_shard1_replica2 node 3: movies_shard2_replica1 node 4: movies_shard2_replica2 To use the movie_directors collection in Solr join queries with the movies collection, it needs to have a replica on each of the four nodes. In other words, movie_directors must have one shard and replication factor of four: node 1: movie_directors_shard1_replica1 node 2: movie_directors_shard1_replica2 node 3: movie_directors_shard1_replica3 node 4: movie_directors_shard1_replica4 At query time, the JoinQParser will access the local replica of the movie_directors collection to perform the join. If a local replica is not available or active, then the query will fail. At this point, it should be clear that since you're limited to a single shard and the data must be replicated across all nodes where it is needed, this approach works better with smaller data sets where there is a one-to-many relationship between the from collection and the to collection. Moreover, if you add a replica to the to collection, then you also need to add a replica for the from collection. For more information about join queries, see the Solr Wiki page on Joins. Erick Erickson has also written a blog post about join performance called Solr and Joins, hosted by SearchHub.org. Lucene Query Parser The LuceneQParser extends the QParserPlugin by parsing Solr's variant on the Lucene QueryParser syntax. This is effectively the same query parser that is used in Lucene. It uses the operators q.op, the default operator ("OR" or "AND") and df, the default field name. Example: {!lucene q.op=AND df=text}myfield:foo +bar -baz For more information about the syntax for the Lucene Query Parser, see the Classic QueryParser javadocs. Max Score Query Parser The MaxScoreQParser extends the LuceneQParser but returns the Max score from the clauses. It does this by wrapping all SHOULD clauses in a DisjunctionMaxQuery with tie=1.0. Any MUST or PROHIBITED clauses are passed through as-is. Non-boolean queries, e.g. NumericRange falls-through to the LuceneQParser parser behavior. Example: Apache Solr Reference Guide 5.1 281 {!maxscore tie=0.01}C OR (D AND E) More Like This Query Parser MLTQParser enables retrieving documents that are similar to a given document. It uses Lucene's existing More LikeThis logic and also works in SolrCloud mode. The document identifier used here is the unique id value and not the Lucene internal document id. This query parser takes the following parameters: Parameter Description qf Specifies the fields to use for similarity. mintf Specifies the Minimum Term Frequency, the frequency below which terms will be ignored in the source document. mindf Specifies the Minimum Document Frequency, the frequency at which words will be ignored which do not occur in at least this many documents. minwl Sets the minimum word length below which words will be ignored. maxwl Sets the maximum word length above which words will be ignored. Examples: Find documents like the document with id=1 and using the name field for similarity. {!mlt qf=name}1 Adding more constraints to what qualifies as similar using mintf and mindf. {!mlt qf=name mintf=2 mindf=3}1 Nested Query Parser The NestedParser extends the QParserPlugin and creates a nested query, with the ability for that query to redefine its type via local parameters. This is useful in specifying defaults in configuration and letting clients indirectly reference them. Example: {!query defType=func v=$q1} If the q1 parameter is price, then the query would be a function query on the price field. If the q1 parameter is {!lucene}inStock:true}} then a term query is created from the Lucene syntax string that matches documents with inStock=true. These parameters would be defined in solrconfig.xml, in the defaults section: Apache Solr Reference Guide 5.1 282 {!lucene}inStock:true For more information about the possibilities of nested queries, see Yonik Seeley's blog post Nested Queries in Solr, hosted by SearchHub.org. Old Lucene Query Parser OldLuceneQParser extends the QParserPlugin by parsing Solr's variant of Lucene's QueryParser syntax, including the deprecated sort specification after the query. Example: {!lucenePlusSort} myfield:foo +bar -baz;price asc Prefix Query Parser PrefixQParser extends the QParserPlugin by creating a prefix query from the input value. Currently no analysis or value transformation is done to create this prefix query. The parameter is f, the field. The string after the prefix declaration is treated as a wildcard query. Example: {!prefix f=myfield}foo This would be generally equivalent to the Lucene query parser expression myfield:foo*. Raw Query Parser RawQParser extends the QParserPlugin by creating a term query from the input value without any text analysis or transformation. This is useful in debugging, or when raw terms are returned from the terms component (this is not the default). The only parameter is f, which defines the field to search. Example: {!raw f=myfield}Foo Bar This example constructs the query: TermQuery(Term("myfield","Foo Bar")). For easy filter construction to drill down in faceting, the TermQParserPlugin is recommended. For full analysis on all fields, including text fields, you may want to use the FieldQParserPlugin. Re-Ranking Query Parser The ReRankQParserPlugin is a special purpose parser for Re-Ranking the top result of a simple query using a more complex ranking query. Details about using the ReRankQParserPlugin can be found in the Other Parsers section. Simple Query Parser Apache Solr Reference Guide 5.1 283 The Simple query parser in Solr is based on Lucene's SimpleQueryParser. This query parser is designed to allow users to enter queries however they want, and it will do its best to interpret the query and return results. This parser takes the following parameters: Parameter q.operators Description Enables specific operations for parsing. By default, all operations are enabled, and this can be used to disable specific operations as needed. Passing an empty string with this parameter disables all operations. Operator Description Example + Specifies AND token1+token2 | Specifies OR token1|token2 - Specifies NOT -token3 " Creates a phrase "term1 term2" * Specifies a prefix query term* ~N At the end of terms, specifies a fuzzy query term~1 ~N At the end of phrases, specifies a NEAR query "term1 term2"~5 () Specifies precedence; tokens inside the parenthesis will be analyzed first. Otherwise, normal order is left to right. token1 + (token2 | token3) If needed, operators can be escaped with the \ character. q.op Defines an operator to use by default if none are defined by the user. By default, OR is defined; an alternative option is AND. qf A list of query fields and boosts to use when building the query. df Defines the default field if none is defined in schema.xml, or overrides the default field if it is already defined. Any errors in syntax are ignored and the query parser will interpret as best it can. This can mean, however, odd results in some cases. Spatial Filter Query Parser SpatialFilterQParser extends the QParserPlugin by creating a spatial Filter based on the type of spatial point used. The field must implement SpatialQueryable. All units are in Kilometers. This query parser takes the following parameters: Parameter sfield Description The field on which to filter. Required. Apache Solr Reference Guide 5.1 284 pt The point to use as a reference. Must match the dimension of the field. Required. d The distance in km. Required. The distance measure used currently depends on the FieldType. LatLonType defaults to using haversine, Poin tType defaults to Euclidean (2-norm). This example shows the syntax: {!geofilt sfield= pt= d=} Here are some examples with values configured: fq={!geofilt sfield=store pt=10.312,-20.556 d=3.5} fq={!geofilt sfield=store}&pt=10.312,-20&d=3.5 fq={!geofilt}&sfield=store&pt=10.312,-20&d=3.5 If using geofilt with LatLonType, it is capable of producing scores equal to the computed distance from the point to the field, making it useful as a component of the main query or a boosting query. There is more information about spatial searches available in the section Spatial Search. Surround Query Parser SurroundQParser extends the QParserPlugin. This provides support for the Surround query syntax, which provides proximity search functionality. There are two operators: w creates an ordered span query and n creates an unordered one. Both operators take a numeric value to indicate distance between two terms. The default is 1, and the maximum is 99. Note that the query string is not analyzed in any way. Example: {!surround} 3w(foo, bar) This example would find documents where the terms "foo" and "bar" were no more than 3 terms away from each other (i.e., no more than 2 terms between them). This query parser will also accept boolean operators (AND, OR, and NOT, in either upper- or lowercase), wildcards, quoting for phrase searches, and boosting. The w and n operators can also be expressed in upper- or lowercase. More information about Surround queries can be found at http://wiki.apache.org/solr/SurroundQueryParser. Switch Query Parser SwitchQParser is a QParserPlugin that acts like a "switch" or "case" statement. The primary input string is trimmed and then prefixed with case. for use as a key to lookup a "switch case" in the parser's local params. If a matching local param is found the resulting param value will then be parsed as a Apache Solr Reference Guide 5.1 285 subquery, and returned as the parse result. The case local param can be optionally be specified as a switch case to match missing (or blank) input strings. The default local param can optionally be specified as a default case to use if the input string does not match any other switch case local params. If default is not specified, then any input which does not match a switch case local param will result in a syntax error. In the examples below, the result of each query is "XXX": {!switch case.foo=XXX case.bar=zzz case.yak=qqq}foo {!switch case.foo=qqq case.bar=XXX case.yak=zzz} bar // extra whitespace is trimmed {!switch case.foo=qqq case.bar=zzz default=XXX}asdf // fallback to the default {!switch case=XXX case.bar=zzz case.yak=qqq} // blank input uses 'case' A practical usage of this QParsePlugin, is in specifying appends fq params in the configuration of a SearchHandler, to provide a fixed set of filter options for clients using custom parameter names. Using the example configuration below, clients can optionally specify the custom parameters in_stock and shipping to override the default filtering behavior, but are limited to the specific set of legal values (shipping=any|free, in_stock=yes|no|all). yes any {!switch case.all='*:*' case.yes='inStock:true' case.no='inStock:false' v=$in_stock} {!switch case.any='*:*' case.free='shipping_cost:0.0' v=$shipping} Term Query Parser TermQParser extends the QParserPlugin by creating a single term query from the input value equivalent to r eadableToIndexed(). This is useful for generating filter queries from the external human readable terms returned by the faceting or terms components. The only parameter is f, for the field. Example: {!term f=weight}1.5 For text fields, no analysis is done since raw terms are already returned from the faceting and terms Apache Solr Reference Guide 5.1 286 components. To apply analysis to text fields as well, see the Field Query Parser, above. If no analysis or transformation is desired for any type of field, see the Raw Query Parser, above. Terms Query Parser TermsQParser functions similarly to the Term Query Parser but takes in multiple values separated by commas and returns documents matching any of the specified values. This can be useful for generating filter queries from the external human readable terms returned by the faceting or terms components, and may be more efficient in some cases than using the Standard Query Parser to generate an boolean query since the default implementation "method" avoids scoring. This query parser takes the following parameters: Parameter Description f The field on which to search. Required. separator Separator to use when parsing the input. If set to " " (a single blank space), will trim additional white space from the input terms. Defaults to ",". method The internal implementation to requested for building the query: termsFilter, booleanQuery , automaton, or docValuesTermsFilter. Defaults to "termsFilter". Examples: {!terms f=tags}software,apache,solr,lucene {!terms f=categoryId method=booleanQuery separator=" "}8 6 7 5309 Faceting As described in the section Overview of Searching in Solr, faceting is the arrangement of search results into categories based on indexed terms. Searchers are presented with the indexed terms, along with numerical counts of how many matching documents were found were each term. Faceting makes it easy for users to explore search results, narrowing in on exactly the results they are looking for. Topics covered in this section: General Parameters Field-Value Faceting Parameters Range Faceting Date Faceting Parameters Pivot (Decision Tree) Faceting Interval Faceting Local Parameters for Faceting Related Topics General Parameters The table below summarizes the general parameters for controlling faceting. Parameter Apache Solr Reference Guide 5.1 Description 287 facet If set to true, enables faceting. facet.query Specifies a Lucene query to generate a facet count. These parameters are described in the sections below. The facet Parameter If set to "true," this parameter enables facet counts in the query response. If set to "false" to a blank or missing value, this parameter disables faceting. None of the other parameters listed below will have any effect unless this parameter is set to "true." The default value is blank. The facet.query Parameter This parameter allows you to specify an arbitrary query in the Lucene default syntax to generate a facet count. By default, Solr's faceting feature automatically determines the unique terms for a field and returns a count for each of those terms. Using facet.query, you can override this default behavior and select exactly which terms or expressions you would like to see counted. In a typical implementation of faceting, you will specify a number of facet.query parameters. This parameter can be particularly useful for numeric-range-based facets or prefix-based facets. You can set the facet.query parameter multiple times to indicate that multiple queries should be used as separate facet constraints. To use facet queries in a syntax other than the default syntax, prefix the facet query with the name of the query notation. For example, to use the hypothetical myfunc query parser, you could set the facet.query parameter like so: facet.query={!myfunc}name~fred Field-Value Faceting Parameters Several parameters can be used to trigger faceting based on the indexed terms in a field. When using this parameter, it is important to remember that "term" is a very specific concept in Lucene: it relates to the literal field/value pairs that are indexed after any analysis occurs. For text fields that include stemming, lowercasing, or word splitting, the resulting terms may not be what you expect. If you want Solr to perform both analysis (for searching) and faceting on the full literal strings, use the copyField directive in the schema.xml fi le to create two versions of the field: one Text and one String. Make sure both are indexed="true". (For more information about the copyField directive, see Documents, Fields, and Schema Design.) The table below summarizes Solr's field value faceting parameters. Parameter Description facet.field Identifies a field to be treated as a facet. facet.prefix Limits the terms used for faceting to those that begin with the specified prefix. facet.contains Limits the terms used for faceting to those that contain the specified substring. facet.contains.ignoreCase If facet.contains is used, ignore case when searching for the specified substring. Apache Solr Reference Guide 5.1 288 facet.sort Controls how faceted results are sorted. facet.limit Controls how many constraints should be returned for each facet. facet.offset Specifies an offset into the facet results at which to begin displaying facets. facet.mincount Specifies the minimum counts required for a facet field to be included in the response. facet.missing Controls whether Solr should compute a count of all matching results which have no value for the field, in addition to the term-based constraints of a facet field. facet.method Selects the algorithm or method Solr should use when faceting a field. facet.enum.cache.minDF (Advanced) Specifies the minimum document frequency (the number of documents matching a term) for which the filterCache should be used when determining the constraint count for that term. facet.overrequest.count (Advanced) A number of documents, beyond the effective facet.limit to request from each shard in a distributed search facet.overrequest.ratio (Advanced) A multiplier of the effective facet.limit to request from each shard in a distributed search facet.threads (Advanced) Controls parallel execution of field faceting These parameters are described in the sections below. The facet.field Parameter The facet.field parameter identifies a field that should be treated as a facet. It iterates over each Term in the field and generate a facet count using that Term as the constraint. This parameter can be specified multiple times in a query to select multiple facet fields. If you do not set this parameter to at least one field in the schema, none of the other parameters described in this section will have any effect. The facet.prefix Parameter The facet.prefix parameter limits the terms on which to facet to those starting with the given string prefix. This does not limit the query in any way, only the facets that would be returned in response to the query. This parameter can be specified on a per-field basis with the syntax of f..facet.prefix. The facet.contains Parameter The facet.contains parameter limits the terms on which to facet to those containing the given substring. This does not limit the query in any way, only the facets that would be returned in response to the query. This parameter can be specified on a per-field basis with the syntax of f..facet.contains. The facet.contains.ignoreCase Parameter If facet.contains is used, the facet.contains.ignoreCase parameter causes case to be ignored when Apache Solr Reference Guide 5.1 289 matching the given substring against candidate facet terms. This parameter can be specified on a per-field basis with the syntax of f..facet.contains.ig noreCase. The facet.sort Parameter This parameter determines the ordering of the facet field constraints. Results facet.sort Setting count Sort the constraints by count (highest count first). index Return the constraints sorted in their index order (lexicographic by indexed term). For terms in the ASCII range, this will be alphabetically sorted. The default is count if facet.limit is greater than 0, otherwise, the default is index. This parameter can be specified on a per-field basis with the syntax of f..facet.sort. The facet.limit Parameter This parameter specifies the maximum number of constraint counts (essentially, the number of facets for a field that are returned) that should be returned for the facet fields. A negative value means that Solr will return unlimited number of constraint counts. The default value is 100. This parameter can be specified on a per-field basis to apply a distinct limit to each field with the syntax of f..facet.limit. The facet.offset Parameter The facet.offset parameter indicates an offset into the list of constraints to allow paging. The default value is 0. This parameter can be specified on a per-field basis with the syntax of f..facet.offset. The facet.mincount Parameter The facet.mincount parameter specifies the minimum counts required for a facet field to be included in the response. If a field's counts are below the minimum, the field's facet is not returned. The default value is 0. This parameter can be specified on a per-field basis with the syntax of f..facet.mincount. The facet.missing Parameter If set to true, this parameter indicates that, in addition to the Term-based constraints of a facet field, a count of all results that match the query but which have no facet value for the field should be computed and returned in the response. The default value is false. Apache Solr Reference Guide 5.1 290 This parameter can be specified on a per-field basis with the syntax of f..facet.missing. The facet.method Parameter The facet.method parameter selects the type of algorithm or method Solr should use when faceting a field. Setting enum Results Enumerates all terms in a field, calculating the set intersection of documents that match the term with documents that match the query. This method is recommended for faceting multi-valued fields that have only a few distinct values. The average number of values per document does not matter. For example, faceting on a field with U.S. States such as Alabama, Alaska, ... Wyoming wou ld lead to fifty cached filters which would be used over and over again. The filterCache should be large enough to hold all the cached filters. fc Calculates facet counts by iterating over documents that match the query and summing the terms that appear in each document. This is currently implemented using an UnInvertedField cache if the field either is multi-valued or is tokenized (according to FieldType.isTokened()). Each document is looked up in the cache to see what terms/values it contains, and a tally is incremented for each value. This method is excellent for situations where the number of indexed values for the field is high, but the number of values per document is low. For multi-valued fields, a hybrid approach is used that uses term filters from the filterCache for terms that match many documents. The letters fc stand for field cache. fcs Per-segment field faceting for single-valued string fields. Enable with facet.method=fcs and control the number of threads used with the threads local parameter. This parameter allows faceting to be faster in the presence of rapid index changes. The default value is fc (except for fields using the BoolField field type) since it tends to use less memory and is faster when a field has many unique terms in the index. This parameter can be specified on a per-field basis with the syntax of f..facet.method. The facet.enum.cache.minDf Parameter This parameter indicates the minimum document frequency (the number of documents matching a term) for which the filterCache should be used when determining the constraint count for that term. This is only used with the facet.method=enum method of faceting. A value greater than zero decreases the filterCache's memory usage, but increases the time required for the query to be processed. If you are faceting on a field with a very large number of terms, and you wish to decrease memory usage, try setting this parameter to a value between 25 and 50, and run a few tests. Then, optimize the parameter setting as necessary. The default value is 0, causing the filterCache to be used for all terms in the field. This parameter can be specified on a per-field basis with the syntax of f..facet.enum.cache. minDF. Over-Request Parameters In some situations, the accuracy in selecting the "top" constraints returned for a facet in a distributed Solr query Apache Solr Reference Guide 5.1 291 can be improved by "Over Requesting" the number of desired constraints (ie: facet.limit) from each of the individual Shards. In these situations, each shard is by default asked for the top "10 + (1.5 * facet.limit)" constraints. In some situations, depending on how your docs are partitioned across your shards, and what facet.limit val ue you used, you may find it advantageous to increase or decrease the amount of over-requesting Solr does. This can be achieved by setting the facet.overrequest.count (defaults to 10) and facet.overrequest. ratio (defaults to 1.5) parameters. The facet.threads Parameter This param will cause loading the underlying fields used in faceting to be executed in parallel with the number of threads specified. Specify as facet.threads=N where N is the maximum number of threads used. Omitting this parameter or specifying the thread count as 0 will not spawn any threads, and only the main request thread will be used. Specifying a negative number of threads will create up to Integer.MAX_VALUE threads. Range Faceting You can use Range Faceting on any date field or any numeric field that supports range queries. This is particularly useful for stitching together a series of range queries (as facet by query) for things like prices. As of Solr 3.1, Range Faceting is preferred over Date Faceting (described below). Parameter Description facet.range Specifies the field to facet by range. facet.range.start Specifies the start of the facet range. facet.range.end Specifies the end of the facet range. facet.range.gap Specifies the span of the range as a value to be added to the lower bound. facet.range.hardend A boolean parameter that specifies how Solr handles a range gap that cannot be evenly divided between the range start and end values. If true, the last range constraint will have the facet.range.end value an upper bound. If false, the last range will have the smallest possible upper bound greater then facet.range.end such that the range is the exact width of the specified range gap. The default value for this parameter is false. facet.range.include Specifies inclusion and exclusion preferences for the upper and lower bounds of the range. See the facet.range.include topic for more detailed information. facet.range.other Specifies counts for Solr to compute in addition to the counts for each facet range constraint. The facet.range Parameter The facet.range parameter defines the field for which Solr should create range facets. For example: facet.range=price&facet.range=age The facet.range.start Parameter Apache Solr Reference Guide 5.1 292 The facet.range.start parameter specifies the lower bound of the ranges. You can specify this parameter on a per field basis with the syntax of f..facet.range.start. For example: f.price.facet.range.start=0.0&f.age.facet.range.start=10 The facet.range.end Parameter The facet.range.end specifies the upper bound of the ranges. You can specify this parameter on a per field basis with the syntax of f..facet.range.end. For example: f.price.facet.range.end=1000.0&f.age.facet.range.start=99 The facet.range.gap Parameter The span of each range expressed as a value to be added to the lower bound. For date fields, this should be expressed using the DateMathParser syntax (such as, facet.range.gap=%2B1DAY ... '+1DAY'). You can specify this parameter on a per-field basis with the syntax of f..facet.range.gap. For example: f.price.facet.range.gap=100&f.age.facet.range.gap=10 The facet.range.hardend Parameter The facet.range.hardend parameter is a Boolean parameter that specifies how Solr should handle cases where the facet.range.gap does not divide evenly between facet.range.start and facet.range.end. If true, the last range constraint will have the facet.range.end value as an upper bound. If false, the last range will have the smallest possible upper bound greater then facet.range.end such that the range is the exact width of the specified range gap. The default value for this parameter is false. This parameter can be specified on a per field basis with the syntax f..facet.range.hardend . The facet.range.include Parameter By default, the ranges used to compute range faceting between facet.range.start and facet.range.end are inclusive of their lower bounds and exclusive of the upper bounds. The "before" range defined with the face t.range.other parameter is exclusive and the "after" range is inclusive. This default, equivalent to "lower" below, will not result in double counting at the boundaries. You can use the facet.range.include parameter to modify this behavior using the following options: Option Description lower All gap-based ranges include their lower bound. upper All gap-based ranges include their upper bound. edge The first and last gap ranges include their edge bounds (lower for the first one, upper for the last one) even if the corresponding upper/lower option is not specified. outer The "before" and "after" ranges will be inclusive of their bounds, even if the first or last ranges already include those boundaries. all Includes all options: lower, upper, edge, outer. Apache Solr Reference Guide 5.1 293 You can specify this parameter on a per field basis with the syntax of f..facet.range.includ e, and you can specify it multiple times to indicate multiple choices. To ensure you avoid double-counting, do not choose both lower and upper, do not choose outer, and do not choose all. The facet.range.other Parameter The facet.range.other parameter specifies that in addition to the counts for each range constraint between facet.range.start and facet.range.end, counts should also be computed for these options: Option Description before All records with field values lower then lower bound of the first range. after All records with field values greater then the upper bound of the last range. between All records with field values between the start and end bounds of all ranges. none Do not compute any counts. all Compute counts for before, between, and after. This parameter can be specified on a per field basis with the syntax of f..facet.range.other . In addition to the all option, this parameter can be specified multiple times to indicate multiple choices, but no ne will override all other options. The facet.mincount Parameter in Range Faceting The facet.mincount parameter, the same one as used in field faceting is also applied to range faceting. When used, no ranges with a count below the minimum will be included in the response. Date Ranges & Time Zones Range faceting on date fields is a common situation where the TZ parameter can be useful to ensure that the "facet counts per day" or "facet counts per month" are based on a meaningful definition of when a given day/month "starts" relative to a particular TimeZone. For more information, see the examples in the Working with Dates section. Date Faceting Parameters Date faceting using the type specific facet.date parameters has been deprecated since Solr 3.1. Existing users are encouraged to switch to using the more general Range Faceting, which provides the same features for date fields, but can also work with any numeric field. The response format is slightly different, but the request parameters are virtually identical. Pivot (Decision Tree) Faceting Pivoting is a summarization tool that lets you automatically sort, count, total or average data stored in a table. The results are typically displayed in a second table showing the summarized data. Pivot faceting lets you create a summary table of the results from a faceting documents by multiple fields. Apache Solr Reference Guide 5.1 294 Another way to look at it is that the query produces a Decision Tree, in that Solr tells you "for facet A, the constraints/counts are X/N, Y/M, etc. If you were to constrain A by X, then the constraint counts for B would be S/P, T/Q, etc.". In other words, it tells you in advance what the "next" set of facet results would be for a field if you apply a constraint from the current facet results. facet.pivot The facet.pivot parameter defines the fields to use for the pivot. Multiple facet.pivot values will create multiple "facet_pivot" sections in the response. Separate each list of fields with a comma. facet.pivot.mincount The facet.pivot.mincount parameter defines the minimum number of documents that need to match in order for the facet to be included in results. The default is 1. Using the "bin/solr -e techproducts" example, A query URL like this one will returns the data below, with the pivot faceting results found in the section "facet_pivot": http://localhost:8983/solr/techproducts/select?q=*:*&facet.pivot=cat,popularity,inSt ock &facet.pivot=popularity,cat&facet=true&facet.field=cat&facet.limit=5 &rows=0&wt=json&indent=true&facet.pivot.mincount=2 "facet_counts":{ "facet_queries":{}, "facet_fields":{ "cat":[ "electronics",14, "currency",4, "memory",3, "connector",2, "graphics card",2]}, "facet_dates":{}, "facet_ranges":{}, "facet_pivot":{ "cat,popularity,inStock":[{ "field":"cat", "value":"electronics", "count":14, "pivot":[{ "field":"popularity", "value":6, "count":5, "pivot":[{ "field":"inStock", "value":true, "count":5}]}, ... Combining Stats Component With Pivots In addition to some of the general local parameters supported by other types of faceting, a stats local parameters can be used with facet.pivot to refer to stats.field instances (by tag) that you would like to Apache Solr Reference Guide 5.1 295 have computed for each Pivot Constraint. In the example below, two different (overlapping) sets of statistics are computed for each of the facet.pivot result hierarchies: stats=true stats.field={!tag=piv1,piv2 min=true max=true}price stats.field={!tag=piv2 mean=true}popularity facet=true facet.pivot={!stats=piv1}cat,inStock facet.pivot={!stats=piv2}manu,inStock Results: "facet_pivot":{ "cat,inStock":[{ "field":"cat", "value":"electronics", "count":12, "pivot":[{ "field":"inStock", "value":true, "count":8, "stats":{ "stats_fields":{ "price":{ "min":74.98999786376953, "max":399.0}}}}, { "field":"inStock", "value":false, "count":4, "stats":{ "stats_fields":{ "price":{ "min":11.5, "max":649.989990234375}}}}], "stats":{ "stats_fields":{ "price":{ "min":11.5, "max":649.989990234375}}}}, { "field":"cat", "value":"currency", "count":4, "pivot":[{ "field":"inStock", "value":true, "count":4, "stats":{ "stats_fields":{ "price":{ ... "manu,inStock":[{ Apache Solr Reference Guide 5.1 296 "field":"manu", "value":"inc", "count":8, "pivot":[{ "field":"inStock", "value":true, "count":7, "stats":{ "stats_fields":{ "price":{ "min":74.98999786376953, "max":2199.0}, "popularity":{ "mean":5.857142857142857}}}}, { "field":"inStock", "value":false, "count":1, "stats":{ "stats_fields":{ "price":{ "min":479.95001220703125, "max":479.95001220703125}, "popularity":{ Apache Solr Reference Guide 5.1 297 "mean":7.0}}}}], ... Additional Pivot Parameters Although facet.pivot.mincount deviates in name from the facet.mincount parameter used by field faceting, many other Field faceting parameters described above can also be used with pivot faceting: facet.limit facet.offset facet.sort facet.overrequest.count facet.overrequest.ratio Interval Faceting Another supported form of faceting is interval faceting. This sounds similar to range faceting, but the functionality is really closer to doing facet queries with range queries. Interval faceting allows you to set variable intervals and count the number of documents that have values within those intervals in the specified field. In order to use interval faceting on a field, it is required that the field has docValues enabled. Even though the same functionality can be achieved by using a facet query with range queries, the implementation of these two methods is very different and will provide different performance depending on the context. If you are concerned about the performance of your searches you should test with both options. Interval faceting tends to be better with multiple intervals for the same fields, while facet query tend to be better in environments where cache is more effective (static indexes for example). Name What it does facet.interval Specifies the field to facet by interval. facet.interval.set Sets the intervals for the field. The facet.interval parameter This parameter Indicates the field where interval faceting must be applied. It can be used multiple times in the same request to indicate multiple fields. All the interval fields must have docValues=“true” in the schema. facet.interval=price&facet.interval=size The facet.interval.set parameter This parameter is used to set the intervals for the field, it can be specified multiple times to indicate multiple intervals. This parameter is global, which means that it will be used for all fields indicated with facet.interval unless there is an override for a specific field. To override this parameter on a specific field you can use: f..facet.interval.set, for example: f.price.facet.interval.set=[0,10]&f.price.facet.interval.set=(10,100] Interval Syntax Intervals must begin with either '(' or '[', be followed by the start value, then a comma (','), the end value, and finally a closing ')' or ']’. Apache Solr Reference Guide 5.1 298 For example: (1,10) -> will include values greater than 1 and lower than 10 [1,10) -> will include values greater or equal to 1 and lower than 10 [1,10] -> will include values greater or equal to 1 and lower or equal to 10 The initial and end values cannot be empty. If the interval needs to be unbounded, the special character '*' can be used for both, start and end limit. When using '*', '(' and '[', and ')' and ']' will be treated equal. [*,*] will include all documents with a value in the field. The interval limits may be strings but there is no need to add quotes. All the text until the comma will be treated as the start limit, and the text after that will be the end limit. For example: [Buenos Aires,New York]. Keep in mind that a string-like comparison will be done to match documents in string intervals (case-sensitive). The comparator can't be changed. Commas, brackets and square brackets can be escaped by using '\' in front of them. Whitespaces before and after the values will be omitted. The start limit can't be grater than the end limit. Equal limits are allowed, this allows you to indicate the specific values that you want to count, like [A,A], [B,B] and [C,Z]. Interval faceting supports output key replacement described below. Output keys can be replaced in both the fac et.interval parameter and in the facet.interval.set parameter. For example: &facet.interval={!key=popularity}some_field &facet.interval.set={!key=bad}[0,5] &facet.interval.set={!key=good}[5,*] &facet=true Local Parameters for Faceting The LocalParams syntax allows overriding global settings. It can also provide a method of adding metadata to other parameter values, much like XML attributes. Tagging and Excluding Filters You can tag specific filters and exclude those filters when faceting. This is useful when doing multi-select faceting. Consider the following example query with faceting: q=mainquery&fq=status:public&fq=doctype:pdf&facet=true&facet.field=doctype Because everything is already constrained by the filter doctype:pdf, the facet.field=doctype facet command is currently redundant and will return 0 counts for everything except doctype:pdf. To implement a multi-select facet for doctype, a GUI may want to still display the other doctype values and their associated counts, as if the doctype:pdf constraint had not yet been applied. For example: === Document Type === [ ] Word (42) [x] PDF (96) [ ] Excel(11) [ ] HTML (63) To return counts for doctype values that are currently not selected, tag filters that directly constrain doctype, and exclude those filters when faceting on doctype. Apache Solr Reference Guide 5.1 299 q=mainquery&fq=status:public&fq={!tag=dt}doctype:pdf&facet=true&facet.field ={!ex=dt}doctype Filter exclusion is supported for all types of facets. Both the tag and ex local parameters may specify multiple values by separating them with commas. Changing the Output Key To change the output key for a faceting command, specify a new name with the key local parameter. For example: facet.field={!ex=dt key=mylabel}doctype The parameter setting above causes the field facet results for the "doctype" field to be returned using the key "mylabel" rather than "doctype" in the response. This can be helpful when faceting on the same field multiple times with different exclusions. Related Topics SimpleFacetParameters from the Solr Wiki. Heatmap Faceting (Spatial) Highlighting Highlighting in Solr allows fragments of documents that match the user's query to be included with the query response. The fragments are included in a special section of the response (the highlighting section), and the client uses the formatting clues also included to determine how to present the snippets to users. Solr provides a collection of highlighting utilities which allow a great deal of control over the fields fragments are taken from, the size of fragments, and how they are formatted. The highlighting utilities can be called by various Request Handlers and can be used with the DisMax, Extended DisMax, or standard query parsers. There are three highlighting implementations available: Standard Highlighter: The Standard Highlighter is the swiss-army knife of the highlighters. It has the most sophisticated and fine-grained query representation of the three highlighters. For example, this highlighter is capable of providing precise matches even for advanced queryparsers such as the surroun d parser. It does not require any special datastructures such as termVectors, although it will use them if they are present. If they are not, this highlighter will re-analyze the document on-the-fly to highlight it. This highlighter is a good choice for a wide variety of search use-cases. FastVector Highlighter: The FastVector Highlighter requires term vector options (termVectors, termP ositions, and termOffsets) on the field, and is optimized with that in mind. It tends to work better for more languages than the Standard Highlighter, because it supports Unicode breakiterators. On the other hand, its query-representation is less advanced than the Standard Highlighter: for example it will not work well with the surround parser. This highlighter is a good choice for large documents and highlighting text in a variety of languages. Postings Highlighter: The Postings Highlighter requires storeOffsetsWithPositions to be configured on the field. This is a much more compact and efficient structure than term vectors, but is not appropriate for huge numbers of query terms (e.g. wildcard queries). Like the FastVector Highlighter, it Apache Solr Reference Guide 5.1 300 supports Unicode algorithms for dividing up the document. On the other hand, it has the most coarse query-representation: it focuses on summary quality and ignores the structure of the query completely, ranking passages based solely on query terms and statistics. This highlighter a good choice for classic full-text keyword search. Configuring Highlighting The configuration for highlighting, whichever implementation is chosen, is first to configure a search component and then reference the component in one or more request handlers. The exact parameters for the search component vary depending on the implementation, but there is a robust example in the solrconfig.xml used in the "techproducts" example which shows how to configure both the Standard Highlighter and the FastVector Highlighter (see the Postings Highlighter section for details on how to configure that implementation). Standard Highlighter The standard highlighter (AKA the default highlighter) doesn't require any special indexing parameters on the fields to highlight. However you can optionally turn on termVectors, termPositions, and termOffsets for any field to be highlighted. This will avoid having to run documents through the analysis chain at query-time and will make highlighting significantly faster and use less memory, particularly for large text fields, and even more so when hl.usePhraseHighlighter is enabled. Standard Highlighting Parameters The table below describes Solr's parameters for the Standard highlighter. These parameters can be defined in the highlight search component, as defaults for the specific request handler, or passed to the request handler with the query. Parameter Default Value Description hl blank (no highlight) When set to true, enables highlighted snippets to be generated in the query response. If set to false or to a blank or missing value, disables highlighting. hl.q blank Specifies an overriding query term for highlighting. If hl.q is specified, the highlighter will use that term rather than the main query term. hl.qparser Apache Solr Reference Guide 5.1 blank Specifies a qparser to use for the hl.q query. If blank, will use the defType of the overall query. 301 hl.fl blank Specifies a list of fields to highlight. Accepts a comma- or space-delimited list of fields for which Solr should generate highlighted snippets. If left blank, highlights the defaultSearchField (or the field specified the df parameter if used) for the StandardRequestHandler. For the DisMaxRequestHandler, the qf fields are used as defaults. A '*' can be used to match field globs, such as 'text_*' or even '*' to highlight on all fields where highlighting is possible. When using '*', consider adding hl.requireFieldMatch=true. hl.snippets 1 Specifies maximum number of highlighted snippets to generate per field. It is possible for any number of snippets from zero to this value to be generated. This parameter accepts per-field overrides. hl.fragsize 100 Specifies the size, in characters, of fragments to consider for highlighting. 0 indicates that no fragmenting should be considered and the whole field value should be used. This parameter accepts per-field overrides. hl.mergeContiguous false Instructs Solr to collapse contiguous fragments into a single fragment. A value of true indicates contiguous fragments will be collapsed into single fragment. This parameter accepts per-field overrides. The default value, false, is also the backward-compatible setting. hl.requireFieldMatch false If set to true, highlights terms only if they appear in the specified field. If false, terms are highlighted in all requested fields regardless of which field matched the query. hl.maxAnalyzedChars 51200 Specifies the number of characters into a document that Solr should look for suitable snippets. hl.maxMultiValuedToExamine integer.MAX_VALUE Specifies the maximum number of entries in a multi-valued field to examine before stopping. This can potentially return zero results if the limit is reached before any matches are found. If used with the maxMultiValuedToMatch, whichever limit is reached first will determine when to stop looking. Apache Solr Reference Guide 5.1 302 hl.maxMultiValuedToMatch integer.MAX_VALUE Specifies the maximum number of matches in a multi-valued field that are found before stopping. If h l.maxMultiValuedToExamine is also defined, whichever limit is reached first will determine when to stop looking. hl.alternateField blank Specifies a field to be used as a backup default summary if Solr cannot generate a snippet (i.e., because no terms match). This parameter accepts per-field overrides. hl.maxAlternateFieldLength unlimited Specifies the maximum number of characters of the field to return. Any value less than or equal to 0 means the field's length is unlimited. This parameter is only used in conjunction with the hl.alternate Field parameter. hl.formatter simple Selects a formatter for the highlighted output. Currently the only legal value is simple, which surrounds a highlighted term with a customizable pre- and post-text snippet. This parameter accepts per-field overrides. hl.simple.pre hl.simple.post and Specifies the text that should appear before (hl.si mple.pre) and after (hl.simple.post) a highlighted term, when using the simple formatter. This parameter accepts per-field overrides. hl.fragmenter gap Specifies a text snippet generator for highlighted text. The standard fragmenter is gap, which creates fixed-sized fragments with gaps for multi-valued fields. Another option is regex, which tries to create fragments that resemble a specified regular expression. This parameter accepts per-field overrides. hl.usePhraseHighlighter true If set to true, Solr will use the Lucene SpanScorer class to highlight phrase terms only when they appear within the query phrase in the document. hl.highlightMultiTerm true If set to true, Solr will use highlight phrase terms that appear in multi-term queries. Apache Solr Reference Guide 5.1 303 hl.regex.slop 0.6 When using the regex fragmenter (hl.fragmente r=regex), this parameter defines the factor by which the fragmenter can stray from the ideal fragment size (given by hl.fragsize) to accommodate a regular expression. For instance, a slop of 0.2 with hl.fragsize=100 should yield fragments between 80 and 120 characters in length. It is usually good to provide a slightly smaller hl.fr agsize value when using the regex fragmenter. hl.regex.pattern blank Specifies the regular expression for fragmenting. This could be used to extract sentences. hl.regex.maxAnalyzedChars 10000 Instructs Solr to analyze only this many characters from a field when using the regex fragmenter (after which, the fragmenter produces fixed-sized fragments). Applying a complicated regex to a huge field is computationally expensive. hl.preserveMulti false If true, multi-valued fields will return all values in the order they were saved in the index. If false, only values that match the highlight request will be returned. hl.payloads (automatic) When hl.usePhraseHighlighter=true and the indexed field has payloads but not term vectors (generally quite rare), the index's payloads will be read into the highlighter's memory index along with the postings. If this may happen and you know you don't need them for highlighting (i.e. your queries don't filter by payload) then you can save a little memory by setting this to false. Related Content HighlightingParameters from the Solr wiki Highlighting javadocs FastVector Highlighter The FastVectorHighlighter is a TermVector-based highlighter that offers higher performance than the standard highlighter in many cases. To use the FastVectorHighlighter, set the hl.useFastVectorHighlighter param eter to true. You must also turn on termVectors, termPositions, and termOffsets for each field that will be highlighted. Lastly, you should use a boundary scanner to prevent the FastVectorHighlighter from truncating your terms. In most cases, using the breakIterator boundary scanner will give you excellent results. See the section Using Boundary Scanners with the Fast Vector Highlighter for more details about boundary scanners. Apache Solr Reference Guide 5.1 304 FastVector Highlighter Parameters The table below describes Solr's parameters for this highlighter, many of which overlap with the standard highlighter. These parameters can be defined in the highlight search component, as defaults for the specific request handler, or passed to the request handler with the query. Parameter Default Description hl blank (no highlighting) When set to true, enables highlighted snippets to be generated in the query response. A false or blank value disables highlighting. hl.useFastVectorHighligter false When set to true, enables the FastVector Highlighter. hl.q blank Specifies an overriding query term for highlighting. If hl.q is specified, the highlighter will use that term rather than the main query term. hl.fl blank Specifies a list of fields to highlight. Accepts a comma- or space-delimited list of fields for which Solr should generate highlighted snippets. If left blank, highlights the defaultSearchField (or the field specified the df parameter if used) for the StandardRequestHandler. For the DisMaxRequestHandler, the qf fields are used as defaults. A '*' can be used to match field globs, such as 'text_*' or even '*' to highlight on all fields where highlighting is possible. When using '*', consider adding hl.requireFieldMatch=true. hl.snippets 1 Specifies maximum number of highlighted snippets to generate per field. It is possible for any number of snippets from zero to this value to be generated. This parameter accepts per-field overrides. hl.fragsize 100 Specifies the size, in characters, of fragments to consider for highlighting. 0 indicates that no fragmenting should be considered and the whole field value should be used. This parameter accepts per-field overrides. hl.requireFieldMatch false If set to true, highlights terms only if they appear in the specified field. If false, terms are highlighted in all requested fields regardless of which field matched the query. Apache Solr Reference Guide 5.1 305 hl.maxMultiValuedToExamine integer.MAX_VALUE Specifies the maximum number of entries in a multi-valued field to examine before stopping. This can potentially return zero results if the limit is reached before any matches are found. If used with the maxMultiValuedToMatch, whichever limit is reached first will determine when to stop looking. hl.maxMultiValuedToMatch integer.MAX_VALUE Specifies the maximum number of matches in a multi-valued field that are found before stopping. If h l.maxMultiValuedToExamine is also defined, whichever limit is reached first will determine when to stop looking. hl.alternateField blank Specifies a field to be used as a backup default summary if Solr cannot generate a snippet (i.e., because no terms match). This parameter accepts per-field overrides. hl.maxAlternateFieldLength unlimited Specifies the maximum number of characters of the field to return. Any value less than or equal to 0 means the field's length is unlimited. This parameter is only used in conjunction with the hl.alternate Field parameter. hl.tag.pre hl.tag.post and Specifies the text that should appear before (hl.ta g.pre) and after (hl.tag.post) a highlighted term. This parameter accepts per-field overrides. hl.phraseLimit integer.MAX_VALUE To improve the performance of the FastVectorHighlighter, you can set a limit on the number (int) of phrases to be analyzed for highlighting. hl.usePhraseHighlighter true If set to true, Solr will use the Lucene SpanScorer class to highlight phrase terms only when they appear within the query phrase in the document. hl.preserveMulti false If true, multi-valued fields will return all values in the order they were saved in the index. If false, the default, only values that match the highlight request will be returned. hl.fragListBuilder weighted The snippet fragmenting algorithm. The weighted fr agListBuilder uses IDF-weights to order fragments. Other options are single, which returns the entire field contents as one snippet, or simple. You can select a fragListBuilder with this parameter, or modify an existing implementation in solrconfig. xml to be the default by adding "default=true". Apache Solr Reference Guide 5.1 306 hl.fragmentsBuilder default The fragments builder is responsible for formatting the fragments, which uses and markup (if hl.tag.pre and hl.tag.post are not defined). Another pre-configured choice is colored, which is an example of how to use the fragments builder to insert HTML into the snippets for colored highlights if you choose. You can also implement your own if you'd like. You can select a fragments builder with this parameter, or modify an existing implementation in solrconfig.xml to be the default by adding "default=true". Using Boundary Scanners with the Fast Vector Highlighter The Fast Vector Highlighter will occasionally truncate highlighted words. To prevent this, implement a boundary scanner in solrconfig.xml, then use the hl.boundaryScanner parameter to specify the boundary scanner for highlighting. Solr supports two boundary scanners: breakIterator and simple. The breakIterator Boundary Scanner The breakIterator boundary scanner offers excellent performance right out of the box by taking locale and boundary type into account. In most cases you will want to use the breakIterator boundary scanner. To implement the breakIterator boundary scanner, add this code to the highlighting section of your solrc onfig.xml file, adjusting the type, language, and country values as appropriate to your application: WORD en US Possible values for the hl.bs.type parameter are WORD, LINE, SENTENCE, and CHARACTER. The simple Boundary Scanner The simple boundary scanner scans term boundaries for a specified maximum character value ( hl.bs.maxSc an) and for common delimiters such as punctuation marks (hl.bs.chars). The simple boundary scanner may be useful for some custom To implement the simple boundary scanner, add this code to the highlighting se ction of your solrconfig.xml file, adjusting the values as appropriate to your application: Apache Solr Reference Guide 5.1 307 10 .,!?\t\n Related Content HighlightingParameters from the Solr wiki Highlighting javadocs Postings Highlighter PostingsHighlighter focuses on good document summarizes and efficiency, but is less flexible than the other highlighters. It uses significantly less disk space, focuses on good document summaries, and provides a performant approach if queries have a low number of terms relative to the number of results per page. However, the drawbacks are that it is not a query matching debugger (it focuses on fast highlighting for full-text search) and it does not allow broken analysis chains. To use this highlighter, you must turn on storeOffsetsWithPositions for the field. There is no need to turn on termVectors, termPositions, or termOffsets in fields since this highlighter does not make use of term vectors. Configuring Postings Highlighter The configuration for the Postings Highlighter is done in solrconfig.xml. First, define the search component: Note in this example, we have named the search component "highlight". If you started with a default solrconfi g.xml file, then you already have a component with that name. You should either replace the default with this example, or rename the search component that is already there so there is no confusion about which search component implementation Solr should use. Then in the request handler, you can define the defaults, as in this example: Apache Solr Reference Guide 5.1 308 1 <em> </em> ... true simple 1.2 0.75 87 SENTENCE 10000 This example shows all of the defaults for each parameter. If you intend to keep all of the defaults, you would not need to add anything to the request handler and could override the default values at query time as needed. Postings Highlighter Parameters The table below describes Solr's parameters for this highlighter. These parameters can be set as defaults (as in the examples), or the default values can be changed in the request handler or at query time. Most of the parameters can be specified per-field (exceptions noted below). Parameter Default Description hl blank (no highlight) When set to true, enables highlighted snippets to be generated in the query response. If set to false or to a blank or missing value, disables highlighting. hl.q blank Specifies an overriding query term for highlighting. If hl.q is specified, the highlighter will use that term rather than the main query term. hl.fl blank Specifies a list of fields to highlight. Accepts a comma- or space-delimited list of fields for which Solr should generate highlighted snippets. If left blank, highlights the defaultSearchField (or the field specified the df parameter if used) for the StandardRequestHandler. For the DisMaxRequestHandler, the qf fields are used as defaults. A '*' can be used to match field globs, such as 'text_*' or even '*' to highlight on all fields where highlighting is possible. When using '*', consider adding hl.requireFieldMatch=true. Apache Solr Reference Guide 5.1 309 hl.snippets 1 Specifies maximum number of highlighted snippets to generate per field. It is possible for any number of snippets from zero to this value to be generated. This parameter accepts per-field overrides. hl.tag.pre Specifies the text that should appear before a highlighted term. hl.tag.post Specifies the text that should appear after a highlighted term. hl.tag.ellipsis "... " Specifies the text that should join two unconnected passages in the resulting snippet. hl.maxAnalyzedChars 10000 Specifies the number of characters into a document that Solr should look for suitable snippets. This parameter does not accept per-field overrides. hl.multiValuedSeparatorChar " " (space) Specifies the logical separator between multi-valued fields. hl.defaultSummary true If true, a field should have a default summary if highlighting finds no matching passages. hl.encoder simple Defines the encoding for the resulting snippet. The value simpl e applies no escaping, while html will escape HTML characters in the text. hl.score.k1 1.2 Specifies BM25 term frequency normalization parameter 'k1'. For example, it can be set to "0" to rank passages solely based on the number of query terms that match. hl.score.b 0.75 Specifies BM25 length normalization parameter 'b'. For example, it can be set to "0" to ignore the length of passages entirely when ranking. hl.score.pivot 87 Specifies BM25 average passage length in characters. hl.bs.language blank Specifies the breakiterator language for dividing the document into passages. hl.bs.country blank Specifies the breakiterator country for dividing the document into passages. hl.bs.variant blank Specifies the breakiterator variant for dividing the document into passages. hl.bs.type SENTENCE Specifies the breakiterator type for dividing the document into passages. Can be SENTENCE, WORD, CHARACTER, LINE, or WHOLE. Related Content PostingsHighlighter from the Solr wiki PostingsSolrHighlighter javadoc Spell Checking Apache Solr Reference Guide 5.1 310 The SpellCheck component is designed to provide inline query suggestions based on other, similar, terms. The basis for these suggestions can be terms in a field in Solr, externally created text files, or fields in other Lucene indexes. Topics covered in this section: Configuring the SpellCheckComponent Spell Check Parameters Distributed SpellCheck Configuring the SpellCheckComponent Define Spell Check in solrconfig.xml The first step is to specify the source of terms in solrconfig.xml. There are three approaches to spell checking in Solr, discussed below. IndexBasedSpellChecker The IndexBasedSpellChecker uses a Solr index as the basis for a parallel index used for spell checking. It requires defining a field as the basis for the index terms; a common practice is to copy terms from some fields (such as title, body, etc.) to another field created for spell checking. Here is a simple example of configuring s olrconfig.xml with the IndexBasedSpellChecker: solr.IndexBasedSpellChecker ./spellchecker content true The first element defines the searchComponent to use the solr.SpellCheckComponent. The classname i s the specific implementation of the SpellCheckComponent, in this case solr.IndexBasedSpellChecker. Defining the classname is optional; if not defined, it will default to IndexBasedSpellChecker. The spellcheckIndexDir defines the location of the directory that holds the spellcheck index, while the fiel d defines the source field (defined in schema.xml) for spell check terms. When choosing a field for the spellcheck index, it's best to avoid a heavily processed field to get more accurate results. If the field has many word variations from processing synonyms and/or stemming, the dictionary will be created with those variations in addition to more valid spelling data. Finally, buildOnCommit defines whether to build the spell check index at every commit (that is, every time new documents are added to the index). It is optional, and can be omitted if you would rather set it to false. DirectSolrSpellChecker The DirectSolrSpellChecker uses terms from the Solr index without building a parallel index like the Index BasedSpellChecker. This spell checker has the benefit of not having to be built regularly, meaning that the terms are always up-to-date with terms in the index. Here is how this might be configured in solrconfig.xml Apache Solr Reference Guide 5.1 311 default name solr.DirectSolrSpellChecker internal 0.5 2 1 5 4 0.01 .01 When choosing a field to query for this spell checker, you want one which has relatively little analysis performed on it (particularly analysis such as stemming). Note that you need to specify a field to use for the suggestions, so like the IndexBasedSpellChecker, you may want to copy data from fields like title, body, etc., to a field dedicated to providing spelling suggestions. Many of the parameters relate to how this spell checker should query the index for term suggestions. The dista nceMeasure defines the metric to use during the spell check query. The value "internal" uses the default Levenshtein metric, which is the same metric used with the other spell checker implementations. Because this spell checker is querying the main index, you may want to limit how often it queries the index to be sure to avoid any performance conflicts with user queries. The accuracy setting defines the threshold for a valid suggestion, while maxEdits defines the number of changes to the term to allow. Since most spelling mistakes are only 1 letter off, setting this to 1 will reduce the number of possible suggestions (the default, however, is 2); the value can only be 1 or 2. minPrefix defines the minimum number of characters the terms should share. Setting this to 1 means that the spelling suggestions will all start with the same letter, for example. The maxInspections parameter defines the maximum number of possible matches to review before returning results; the default is 5. minQueryLength defines how many characters must be in the query before suggestions are provided; the default is 4. maxQueryFrequency sets the maximum threshold for the number of documents a term must appear in before being considered as a suggestion. This can be a percentage (such as .01, or 1%) or an absolute value (such as 4). A lower threshold is better for small indexes. Finally, tresholdTok enFrequency sets the minimum number of documents a term must appear in, and can also be expressed as a percentage or an absolute value. FileBasedSpellChecker The FileBasedSpellChecker uses an external file as a spelling dictionary. This can be useful if using Solr as a spelling server, or if spelling suggestions don't need to be based on actual terms in the index. In solrconfig .xml, you would define the searchComponent as so: Apache Solr Reference Guide 5.1 312 solr.FileBasedSpellChecker file spellings.txt UTF-8 ./spellcheckerFile The differences here are the use of the sourceLocation to define the location of the file of terms and the use of characterEncoding to define the encoding of the terms file. In the previous example, name is used to name this specific definition of the spellchecker. Multiple definitions can co-exist in a single solrconfig.xml, and the name helps to differentiate them when they are defined in the schema.xml. If only defining one spellchecker, no name is required. WordBreakSolrSpellChecker WordBreakSolrSpellChecker offers suggestions by combining adjacent query terms and/or breaking terms into multiple words. It is a SpellCheckComponent enhancement, leveraging Lucene's WordBreakSpellChec ker. It can detect spelling errors resulting from misplaced whitespace without the use of shingle-based dictionaries and provides collation support for word-break errors, including cases where the user has a mix of single-word spelling errors and word-break errors in the same query. It also provides shard support. Here is how it might be configured in solrconfig.xml: wordbreak solr.WordBreakSolrSpellChecker lowerfilt true true 10 Some of the parameters will be familiar from the discussion of the other spell checkers, such as name, classna me, and field. New for this spell checker is combineWords, which defines whether words should be combined in a dictionary search (default is true); breakWords, which defines if words should be broken during a dictionary search (default is true); and maxChanges, an integer which defines how many times the spell checker should check collation possibilities against the index (default is 10). The spellchecker can be configured with a traditional checker (ie: DirectSolrSpellChecker). The results are combined and collations can contain a mix of corrections from both spellcheckers. Add It to a Request Handler Queries will be sent to a RequestHandler. If every request should generate a suggestion, then you would add the following to the requestHandler that you are using: Apache Solr Reference Guide 5.1 313 true One of the possible parameters is the spellcheck.dictionary to use, and multiples can be defined. With multiple dictionaries, all specified dictionaries are consulted and results are interleaved. Collations are created with combinations from the different spellcheckers, with care taken that multiple overlapping corrections do not occur in the same collation. Here is an example with multiple dictionaries: default wordbreak 20 spellcheck Spell Check Parameters The SpellCheck component accepts the parameters described in the table below. Parameter Description spellcheck Turns on or off SpellCheck suggestions for the request. If true, then spelling suggestions will be generated. spellcheck.q or q Selects the query to be spellchecked. spellcheck.build Instructs Solr to build a dictionary for use in spellchecking. spellcheck.collate Causes Solr to build a new query based on the best suggestion for each term in the submitted query. spellcheck.maxCollations This parameter specifies the maximum number of collations to return. spellcheck.maxCollationTries This parameter specifies the number of collation possibilities for Solr to try before giving up. spellcheck.maxCollationEvaluations This parameter specifies the maximum number of word correction combinations to rank and evaluate prior to deciding which collation candidates to test against the index. spellcheck.collateExtendedResult If true, returns an expanded response detailing the collations found. If s pellcheck.collate is false, this parameter will be ignored. spellcheck.collateMaxCollectDocs Apache Solr Reference Guide 5.1 The maximum number of documents to collect when testing potential Collations 314 spellcheck.collateParam.* Specifies param=value pairs that can be used to override normal query params when validating collations spellcheck.count Specifies the maximum number of spelling suggestions to be returned. spellcheck.dictionary Specifies the dictionary that should be used for spellchecking. spellcheck.extendedResults Causes Solr to return additional information about spellcheck results, such as the frequency of each original term in the index (origFreq) as well as the frequency of each suggestion in the index (frequency). Note that this result format differs from the non-extended one as the returned suggestion for a word is actually an array of lists, where each list holds the suggested term and its frequency. spellcheck.onlyMorePopular Limits spellcheck responses to queries that are more popular than the original query. spellcheck.maxResultsForSuggest The maximum number of hits the request can return in order to both generate spelling suggestions and set the "correctlySpelled" element to "false". spellcheck.alternativeTermCount The count of suggestions to return for each query term existing in the index and/or dictionary. spellcheck.reload Reloads the spellchecker. spellcheck.accuracy Specifies an accuracy value to help decide whether a result is worthwhile. spellcheck..key Specifies a key/value pair for the implementation handling a given dictionary. The spellcheck Parameter This parameter turns on SpellCheck suggestions for the request. If true, then spelling suggestions will be generated. The spellcheck.q or q Parameter This parameter specifies the query to spellcheck. If spellcheck.q is defined, then it is used; otherwise the original input query is used. The spellcheck.q parameter is intended to be the original query, minus any extra markup like field names, boosts, and so on. If the q parameter is specified, then the SpellingQueryConverte r class is used to parse it into tokens; otherwise the WhitespaceTokenizer is used. The choice of which one to use is up to the application. Essentially, if you have a spelling "ready" version in your application, then it is probably better to use spellcheck.q. Otherwise, if you just want Solr to do the job, use the q parameter. The SpellingQueryConverter class does not deal properly with non-ASCII characters. In this case, you have either to use spellcheck.q, or implement your own QueryConverter. The spellcheck.build Parameter If set to true, this parameter creates the dictionary that the SolrSpellChecker will use for spell-checking. In a Apache Solr Reference Guide 5.1 315 typical search application, you will need to build the dictionary before using the SolrSpellChecker. However, it's not always necessary to build a dictionary first. For example, you can configure the spellchecker to use a dictionary that already exists. The dictionary will take some time to build, so this parameter should not be sent with every request. The spellcheck.reload Parameter If set to true, this parameter reloads the spellchecker. The results depend on the implementation of SolrSpellC hecker.reload(). In a typical implementation, reloading the spellchecker means reloading the dictionary. The spellcheck.count Parameter This parameter specifies the maximum number of suggestions that the spellchecker should return for a term. If this parameter isn't set, the value defaults to 1. If the parameter is set but not assigned a number, the value defaults to 5. If the parameter is set to a positive integer, that number becomes the maximum number of suggestions returned by the spellchecker. The spellcheck.onlyMorePopular Parameter If true, Solr will to return suggestions that result in more hits for the query than the existing query. Note that this will return more popular suggestions even when the given query term is present in the index and considered "correct". The spellcheck.maxResultsForSuggest Parameter For example, if this is set to 5 and the user's query returns 5 or fewer results, the spellchecker will report "correctlySpelled=false" and also offer suggestions (and collations if requested). Setting this greater than zero is useful for creating "did-you-mean?" suggestions for queries that return a low number of hits. The spellcheck.alternativeTermCount Parameter Specify the number of suggestions to return for each query term existing in the index and/or dictionary. Presumably, users will want fewer suggestions for words with docFrequency>0. Also setting this value turns "on" context-sensitive spell suggestions. The spellcheck.extendedResults Parameter This parameter causes to Solr to include additional information about the suggestion, such as the frequency in the index. The spellcheck.collate Parameter If true, this parameter directs Solr to take the best suggestion for each token (if one exists) and construct a new query from the suggestions. For example, if the input query was "jawa class lording" and the best suggestion for "jawa" was "java" and "lording" was "loading", then the resulting collation would be "java class loading". The spellcheck.collate parameter only returns collations that are guaranteed to result in hits if re-queried, even when applying original fq parameters. This is especially helpful when there is more than one correction per query. This only returns a query to be used. It does not actually run the suggested query. Apache Solr Reference Guide 5.1 316 The spellcheck.maxCollations Parameter The maximum number of collations to return. The default is 1. This parameter is ignored if spellcheck.colla te is false. The spellcheck.maxCollationTries Parameter This parameter specifies the number of collation possibilities for Solr to try before giving up. Lower values ensure better performance. Higher values may be necessary to find a collation that can return results. The default value is 0, which maintains backwards-compatible (Solr 1.4) behavior (do not check collations). This parameter is ignored if spellcheck.collate is false. The spellcheck.maxCollationEvaluations Parameter This parameter specifies the maximum number of word correction combinations to rank and evaluate prior to deciding which collation candidates to test against the index. This is a performance safety-net in case a user enters a query with many misspelled words. The default is 10,000 combinations, which should work well in most situations. The spellcheck.collateExtendedResult Parameter If true, this parameter returns an expanded response format detailing the collations Solr found. The default value is false and this is ignored if spellcheck.collate is false. The spellcheck.collateMaxCollectDocs Parameter This parameter specifies the maximum number of documents that should be collect when testing potential collations against the index. A value of 0 indicates that all documents should be collected, resulting in exact hit-counts. Otherwise an estimation is provided as a performance optimization in cases where exact hit-counts are unnecessary – the higher the value specified, the more precise the estimation. The default value for this parameter is 0, but when spellcheck.collateExtendedResults is false, the optimization is always used as if a 1 had been specified. The spellcheck.collateParam.* Parameter Prefix This parameter prefix can be used to specify any additional parameters that you wish to the Spellchecker to use when internally validating collation queries. For example, even if your regular search results allow for loose matching of one or more query terms via parameters like "q.op=OR&mm=20%" you can specify override params such as "spellcheck.collateParam.q.op=AND&spellcheck.collateParam.mm=100%" to require that only collations consisting of words that are all found in at least one document may be returned. The spellcheck.dictionary Parameter This parameter causes Solr to use the dictionary named in the parameter's argument. The default setting is "default". This parameter can be used to invoke a specific spellchecker on a per request basis. The spellcheck.accuracy Parameter Specifies an accuracy value to be used by the spell checking implementation to decide whether a result is worthwhile or not. The value is a float between 0 and 1. Defaults to Float.MIN_VALUE. Apache Solr Reference Guide 5.1 317 The spellcheck..key Parameter Specifies a key/value pair for the implementation handling a given dictionary. The value that is passed through is just key=value (spellcheck.. is stripped off. For example, given a dictionary called foo, spellcheck.foo.myKey=myValue would result in myKey=myVal ue being passed through to the implementation handling the dictionary foo. Example Using Solr's "bin/solr -e techproducts" example, this query shows the results of a simple request that defines a query using the spellcheck.q parameter, and forces the collations to require all input terms must match: http://localhost:8983/solr/techproducts/spell?df=text&spellcheck.q=delll+ultra+sha rp&spellcheck=true&spellcheck.collateParam.q.op=AND Results: Apache Solr Reference Guide 5.1 318 1 0 5 0 dell 1 1 6 17 0 ultrasharp 1 false dell ultrasharp 1 dell ultrasharp Distributed SpellCheck The SpellCheckComponent also supports spellchecking on distributed indexes. If you are using the SpellCheckComponent on a request handler other than "/select", you must provide the following two parameters: Parameter Description shards Specifies the shards in your distributed indexing configuration. For more information about distributed indexing, see Distributed Search with Index Sharding shards.qt Specifies the request handler Solr uses for requests to shards. This parameter is not required for the /select request handler. For example: http://localhost:8983/solr/techproducts/spell?spellcheck=true&spellcheck. build=true&spellcheck.q=toyata&shards.qt=/spell&shards=solr-shard1:8983/solr/techpr Apache Solr Reference Guide 5.1 319 oducts,solr-shard2:8983/solr/techproducts In case of a distributed request to the SpellCheckComponent, the shards are requested for at least five suggestions even if the spellcheck.count parameter value is less than five. Once the suggestions are collected, they are ranked by the configured distance measure (Levenstein Distance by default) and then by aggregate frequency. Query Re-Ranking Query Re-Ranking allows you to run a simple query (A) for matching documents and then re-rank the top N documents using the scores from a more complex query (B). Since the more costly ranking from query B is only applied to the top N documents it will have less impact on performance then just using the complex query B by itself – the trade off is that documents which score very low using the simple query A may not be considered during the re-ranking phase, even if they would score very highly using query B. Specifying A Ranking Query A Ranking query can be specified using the "rq" request parameter. The "rq" parameter must specify a query string that when parsed, produces a RankQuery. This could also be done with a custom QParserPlugin you have written as a plugin, but most users can just use the "rerank" parser provided with Solr. The "rerank" parser wraps a query specified by an local parameter, along with additional parameters indicating how many documents should be re-ranked, and how the final scores should be computed: Parameter Default Description reRankQuery (Mandatory) The query string for your complex ranking query - in most cases a variable w ill be used to refer to another request parameter. reRankDocs 200 The number of top N documents from the original query that should be re-ranked. This number will be treated as a minimum, and may be increased internally automatically in order to rank enough documents to satisfy the query (ie: start+rows) reRankWeight 2.0 A multiplicative factor that will be applied to the score from the reRankQuery for each of the top matching documents, before that score is added to the original score In the example below, the top 1000 documents matching the query "greetings" will be re-ranked using the query "(hi hello hey hiya)". The resulting scores for each of those 1000 documents will be 3 times their score from the "(hi hello hey hiya)", plus the score from the original "gretings" query: q=greetings&rq={!rerank reRankQuery=$rqq reRankDocs=1000 reRankWeight=3}&rqq=(hi+hello+hey+hiya) If a document matches the original query, but does not match the re-ranking query, the document's original score will remain. Combining Ranking Queries With Other Solr Features The "rq" parameter and the re-ranking feature in general works well with other Solr features. For example, it can Apache Solr Reference Guide 5.1 320 be used in conjunction with the collapse parser to re-rank the group heads after they've been collapsed. It also preserves the order of documents elevated by the elevation component. And it even has it's own custom explain so you can see how the re-ranking scores were derived when looking at debug information. Transforming Result Documents Document Transformers can be used to modify the information returned about each documents in the results of a query. Using Document Transformers When executing a request, a document transformer can be used by including it in the fl parameter using square brackets, for example: fl=id,name,score,[shard] Some transformers allow, or require, local parameters which can be specified as key value pairs inside the brackets: fl=id,name,score,[explain style=nl] As with regular fields, you can change the key used when a Transformer adds a field to a document via a prefix: fl=id,name,score,my_val_a:[value v=42 t=int],my_val_b:[value v=7 t=float] The sections below discuss exactly what these various transformers do. Available Transformers [value] - ValueAugmenterFactory Modifies every document to include the exact same value, as if it were a stored field in every document: q=*:*&fl=id,greeting:[value v='hello'] The above query would produce results like the following: 1 hello ... By default, values are returned as a String, but a "t" parameter can be specified using a value of int, float, double, or date to force a specific return type: q=*:*&fl=id,my_number:[value v=42 t=int],my_string:[value v=42] In addition to using these request parameters, you can configure additional named instances of Apache Solr Reference Guide 5.1 321 ValueAugmenterFactory, or override the default behavior of the existing [value] transformer in your solrconfig.xml file: 5 5 The "value" option forces an explicit value to always be used, while the " defaultValue" option provides a default that can still be overridden using the "v" and "t" local parameters. [explain] - ExplainAugmenterFactory Augments each document with an inline explanation of it's score exactly like the information available about each document in the debug section: q=features:cache&wt=json&fl=id,[explain style=nl] Supported values for "style" are "text", and "html", and "nl" which returns the information as structured data: "response":{"numFound":2,"start":0,"docs":[ { "id":"6H500F0", "[explain]":{ "match":true, "value":1.052226, "description":"weight(features:cache in 2) [DefaultSimilarity], result of:", "details":[{ ... A default style can be configured by specifying an "args" parameter in your configuration: nl [child] - ChildDocTransformerFactory This transformer returns all descendant documents of each parent document matching your query in a flat list nested inside the matching parent document. This is useful when you have indexed nested child documents and want to retrieve the child documents for the relevant parent documents for any type of search query. fl=id,[child parentFilter=doc_type:book childFilter=doc_type:chapter limit=100] Note that this transformer can be used even though the query itself is not a Block Join query. Apache Solr Reference Guide 5.1 322 When using this transformer, the parentFilter parameter must be specified, and works the same as in all Block Join Queries, additional optional parameters are: childFilter - query to filter which child documents should be included, this can be particularly useful when you have multiple levels of hierarchical documents (default: all children) limit - the maximum number of child documents to be returned per parent document (default: 10) [shard] - ShardAugmenterFactory This transformer adds information about what shard each individual document came from in a distributed request. ShardAugmenterFactory does not support any request parameters, or configuration options. [docid] - DocIdAugmenterFactory This transformer adds the internal Lucene document id to each document – this is primarily only useful for debugging purposes. DocIdAugmenterFactory does not support any request parameters, or configuration options. [elevated] and [excluded] These transformers are available only when using the Query Elevation Component. [elevated] annotates each document to indicate if it was elevated or not. [excluded] annotates each document to indicate if it would have been excluded - this is only supported if you also use the markExcludes parameter. fl=id,[elevated],[excluded]&excludeIds=GB18030TEST&elevateIds=6H500F0&markExcludes=t rue "response":{"numFound":32,"start":0,"docs":[ { "id":"6H500F0", "[elevated]":true, "[excluded]":false}, { "id":"GB18030TEST", "[elevated]":false, "[excluded]":true}, { "id":"SP2514N", "[elevated]":false, "[excluded]":false}, ... Suggester The SuggestComponent in Solr provides users with automatic suggestions for query terms. You can use this to implement a powerful auto-suggest feature in your search application. Apache Solr Reference Guide 5.1 323 Although it is possible to use the Spell Checking functionality to power autosuggest behavior, Solr has a dedicated SuggestComponent designed for this functionality. This approach utilizes Lucene's Suggester implementation and supports all of the lookup implementations available in Lucene. The main features of this Suggester are: Lookup implementation pluggability Term dictionary pluggability, giving you the flexibility to choose the dictionary implementation Distributed support The solrconfig.xml found in Solr's "techproducts" example has the new Suggester implementation configured already. For more on search components, see the section RequestHandlers and SearchComponents in SolrConfig. Covered in this section: Configuring Suggester in solrconfig.xml Adding the Suggest Search Component Adding the Suggest Request Handler Example Usages Get Suggestions with Weights Multiple Dictionaries Configuring Suggester in solrconfig.xml The "techproducts" example solrconfig.xml has a suggest search component and a /suggest request handler already configured. You can use that as the basis for your configuration, or create it from scratch, as detailed below. Adding the Suggest Search Component The first step is to add a search component to solrconfig.xml to extend the SpellChecker. Here is some sample code that could be used. mySuggester FuzzyLookupFactory DocumentDictionaryFactory cat price string false Suggester Search Component Parameters The Suggester search component takes several configuration parameters. The choice of the lookup implementation (lookupImpl, how terms are found in the suggestion dictionary) and the dictionary implementation (dictionaryImpl, how terms are stored in the suggestion dictionary) will dictate some of the parameters required. Below are the main parameters that can be used no matter what lookup or dictionary implementation is used. In the following sections additional parameters are provided for each implementation. Apache Solr Reference Guide 5.1 324 Parameter Description searchComponent name Arbitrary name for the search component. name A symbolic name for this suggester. You can refer to this name in the URL parameters and in the SearchHandler configuration. It is possible to have mutiples of these lookupImpl Lookup implementation. There are several possible implementations, described below in the section Lookup Implementations. If not set, the default lookup is JaspellLookupFactory. dictionaryImpl The dictionary implementation to use. There are several possible implementations, described below in the section Dictionary Implementations . If not set, the default dictionary implementation is HighFrequencyDictionaryFactory unless a sourceLocatio n is used, in which case, the dictionary implementation will be FileDictionaryFactory field A field from the index to use as the basis of suggestion terms. If sourceLocation is empty (meaning any dictionary implementation other than FileDictionaryFactory) then terms from this field in the index will be used. To be used as the basis for a suggestion, the field must be stored. You may want to use copyField rules to create a special 'suggest' field comprised of terms from other fields in documents. In any event, you likely want a minimal amount of analysis on the field, so an additional option is to create a field type in your schema that only uses basic tokenizers or filters. One option for such a field type is shown here: However, this minimal analysis is not required if you want more analysis to occur on terms. If using the AnalyzingLookupFactory as your lookupImpl, however, you have the option of defining the field type rules to use for index and query time analysis. sourceLocation The path to the dictionary file if using the FileDictionaryFactory. If this value is empty then the main index will be used as a source of terms and weights. storeDir The location to store the dictionary file. Apache Solr Reference Guide 5.1 325 buildOnCommit or buildOnOptimize If true then the lookup data structure will be rebuilt after soft-commit. If false, the default, then the lookup data will be built only when requested by URL parameter suggest.bui ld=true. Use buildOnCommit to rebuild the dictionary with every soft-commit, or buil dOnOptimize to build the dictionary only when the index is optimized. Some lookup implementations may take a long time to build, specially with large indexes, in such cases, using buildOnCommit or buildOnOptimize, particularly with a high frequency of softCommits is not recommended, and it's recommended instead build the suggester manually issuing requests with suggest.build=true at a lower frequency. buildOnStartup If true then the lookup data structure will be built when Solr starts or when the core is reloaded. If this parameter is not specified, the suggester will check if the lookup data structure is present on disk and build it if not found. Enabling this to true could lead to the core talking longer to load (or reload) as the suggester data structure needs to be built, which can sometimes take a long time. It’s usually preferred to have this setting set to 'false' and build suggesters manually issuing requests with suggest.build=true. Lookup Implementations The lookupImpl parameter defines the algorithms used to look up terms in the suggest index. There are several possible implementations to choose from, and some require additional parameters to be configured. AnalyzingLookupFactory A lookup that first analyzes the incoming text and adds the analyzed form to a weighted FST, and then does the same thing at lookup time. This implementation uses the following additional properties: suggestAnalyzerFieldType: The field type to use for the query-time and build-time term suggestion analysis. exactMatchFirst: If true, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights. preserveSep: If true, the default, then a separator between tokens is preserved. This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball). preservePositionIncrements: If true, the suggester will preserve position increments. This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester. The default is false. FuzzyLookupFactory This is a suggester which is an extension of the AnalyzingSuggester but is fuzzy in nature. The similarity is measured by the Levenshtein algorithm. This implementation uses the following additional properties: exactMatchFirst: If true, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights. preserveSep: If true, the default, then a separator between tokens is preserved. This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball). maxSurfaceFormsPerAnalyzedForm: Maximum number of surface forms to keep for a single analyzed form. When there are too many surface forms we discard the lowest weighted ones. maxGraphExpansions: When building the FST ("index-time"), we add each path through the tokenstream Apache Solr Reference Guide 5.1 326 graph as an individual entry. This places an upper-bound on how many expansions will be added for a single suggestion. The default is -1 which means there is no limit. preservePositionIncrements: If true, the suggester will preserve position increments. This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester. The default is false. maxEdits: The maximum number of string edits allowed. The systems hard limit is 2. The default is 1. transpositions: If true, the default, transpositions should be treated as a primitive edit operation. nonFuzzyPrefix: The length of the common non fuzzy prefix match which must match a suggestion. The default is 1. minFuzzyLength: The minimum length of query before which any string edits will be allowed. The default is 3. unicodeAware: If true, maxEdits, minFuzzyLength, transpositions and nonFuzzyPrefix parameters will be measured in unicode code points (actual letters) instead of bytes. The default is false. AnalyzingInfixLookupFactory Analyzes the input text and then suggests matches based on prefix matches to any tokens in the indexed text. This uses a Lucene index for its dictionary. This implementation uses the following additional properties. indexPath: When using AnalyzingInfixSuggester you can provide your own path where the index will get built. The default is analyzingInfixSuggesterIndexDir and will be created in your collections data directory. minPrefixChars: Minimum number of leading characters before PrefixQuery is used (default is 4). Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster). allTermsRequired: Boolean option for multiple terms. Default is true - all terms required. highlight: Highlight suggest terms. Default is true. BlendedInfixLookupFactory An extension of the AnalyzingInfixSuggester which provides additional functionality to weight prefix matches across the matched documents. You can tell it to score higher if a hit is closer to the start of the suggestion or vice versa. This implementation uses the following additional properties: blenderType: used to calculate weight coefficient using the position of the first matching word. Can be one of: linear: weightFieldValue*(1 - 0.10*position): Matches to the start will be given a higher score (Default) reciprocal: weightFieldValue/(1+position): Matches to the end will be given a higher score. numFactor: The factor to multiply the number of searched elements from which results will be pruned. Default is 10. indexPath: When using BlendedInfixSuggester you can provide your own path where the index will get built. The default directory name is blendedInfixSuggesterIndexDir and will be created in your collections data directory. minPrefixChars: Minimum number of leading characters before PrefixQuery is used (default 4). Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster). FreeTextLookupFactory It looks at the last tokens plus the prefix of whatever final token the user is typing, if present, to predict the most Apache Solr Reference Guide 5.1 327 likely next token. The number of previous tokens that need to be considered can also be specified. This suggester would only be used as a fallback, when the primary suggester fails to find any suggestions. This implementation uses the following additional properties: suggestFreeTextAnalyzerFieldType: The analyzer used at "query-time" and "build-time" to analyze suggestions. This field is required. ngrams: The max number of tokens out of which singles will be make the dictionary. The default value is 2. Increasing this would mean you want more than the previous 2 tokens to be taken into consideration when making the suggestions. FSTLookupFactory An automaton-based lookup. This implementation is slower to build, but provides the lowest memory cost. We recommend using this implementation unless you need more sophisticated matching results, in which case you should use the Jaspell implementation. This implementation uses the following additional properties: exactMatchFirst: If true, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights. weightBuckets: The number of separate buckets for weights which the suggester will use while building its dictionary. TSTLookupFactory A simple compact ternary trie based lookup. WFSTLookupFactory A weighted automaton representation which is an alternative to FSTLookup for more fine-grained ranking. WFSTLookup does not use buckets, but instead a shortest path algorithm. Note that it expects weights to be whole numbers. If weight is missing it's assumed to be 1.0. Weights affect the sorting of matching suggestions when spellcheck.onlyMorePopular=true is selected: weights are treated as "popularity" score, with higher weights preferred over suggestions with lower weights. JaspellLookupFactory A more complex lookup based on a ternary trie from the JaSpell project. Use this implementation if you need more sophisticated matching results. Dictionary Implementations The dictionary implementations define how terms are stored. There are several options, and multiple dictionaries can be used in a single request if necessary. DocumentDictionaryFactory A dictionary with terms, weights, and an optional payload taken from the index. This dictionary implementation takes the following parameters in addition to parameters described for the Suggester generally and for the lookup implementation: weightField: A field that is stored or a numeric DocValue field. This field is optional. payloadField: The payloadField should be a field that is stored. This field is optional. Apache Solr Reference Guide 5.1 328 DocumentExpressionDictionaryFactory This dictionary implementation is the same as the DocumentDictionaryFactory but allows users to specify an arbitrary expression into the 'weightExpression' tag. This dictionary implementation takes the following parameters in addition to parameters described for the Suggester generally and for the lookup implementation: payloadField: The payloadField should be a field that is stored. This field is optional. weightExpression: An arbitrary expression used for scoring the suggestions. The fields used must be numeric fields. This field is required. HighFrequencyDictionaryFactory This dictionary implementation allows adding a threshold to prune out less frequent terms in cases where very common terms may overwhelm other terms. This dictionary implementation takes one parameter in addition to parameters described for the Suggester generally and for the lookup implementation: threshold: A value between zero and one representing the minimum fraction of the total documents where a term should appear in order to be added to the lookup dictionary. FileDictionaryFactory This dictionary implementation allows using an external file that contains suggest entries. Weights and payloads can also be used. If using a dictionary file, it should be a plain text file in UTF-8 encoding. Blank lines and lines that start with a '#' are ignored. You can use both single terms and phrases in the dictionary file. If adding weights or payloads, those should be separated from terms using the delimiter defined with the fieldDelimiter property (the default is '\t', the tab representation). This dictionary implementation takes one parameter in addition to parameters described for the Suggester generally and for the lookup implementation: fieldDelimiter: Specify the delimiter to be used separating the entries, weights and payloads. The default is tab ('\t'). # This is a sample dictionary file. acquire accidentally\t2.0 accommodate\t3.0 Multiple Dictionaries It is possible to include multiple dictionaryImpl definitions in a single SuggestComponent definition. To do this, simply define separate suggesters, as in this example: Apache Solr Reference Guide 5.1 329 mySuggester FuzzyLookupFactory DocumentDictionaryFactory cat price string altSuggester DocumentExpressionDictionaryFactory FuzzyLookupFactory product_name ((price * 2) + ln(popularity)) weight price suggest_fuzzy_doc_expr_dict text_en When using these Suggesters in a query, you would define multiple 'suggest.dictionary' parameters in the request, referring to the names given for each Suggester in the search component definition. The response will include the terms in sections for each Suggester. See the Examples section below for an example request and response. Adding the Suggest Request Handler After adding the search component, a request handler must be added to solrconfig.xml. This request handler works the same as any other request handler, and allows you to configure default parameters for serving suggestion requests. The request handler definition must incorporate the "suggest" search component defined previously. true 10 suggest Suggest Request Handler Parameters The following parameters allow you to set defaults for the Suggest request handler: Parameter suggest=true Description This parameter should always be true, because we always want to run the Suggester for queries submitted to this handler. Apache Solr Reference Guide 5.1 330 suggest.dictionary The name of the dictionary component configured in the search component. This is a mandatory parameter. It can be set in the request handler, or sent as a parameter at query time. suggest.q The query to use for suggestion lookups. suggest.count Specifies the number of suggestions for Solr to return. suggest.build If true, it will build the suggester index. This is likely useful only for initial requests; you would probably not want to build the dictionary on every request, particularly in a production system. If you would like to keep your dictionary up to date, you should use the buildOnCommit or buildOnOptimize parameter for the search component. suggest.reload If true, it will reload the suggester index. suggest.buildAll If true, it will build all suggester indexes. suggest.reloadAll If true, it will reload all suggester indexes. These properties can also be overridden at query time, or not set in the request handler at all and always sent at query time. Example Usages Get Suggestions with Weights This is the basic suggestion using a single dictionary and a single Solr core. Example query: http://localhost:8983/solr/techproducts/suggest?suggest=true&suggest.build=true&sugg est.dictionary=mySuggester&wt=json&suggest.q=elec In this example, we've simply requested the string 'elec' with the suggest.q parameter and requested that the suggestion dictionary be built with suggest.build (note, however, that you would likely not want to build the index on every query - instead you should use buildOnCommit or buildOnOptimize if you have regularly changing documents). Example response: Apache Solr Reference Guide 5.1 331 { "responseHeader": { "status": 0, "QTime": 35 }, "command": "build", "suggest": { "mySuggester": { "elec": { "numFound": 3, "suggestions": [ { "term": "electronics and computer1", "weight": 2199, "payload": "" }, { "term": "electronics", "weight": 649, "payload": "" }, { "term": "electronics and stuff2", "weight": 279, "payload": "" } ] } } } } Multiple Dictionaries If you have defined multiple dictionaries, you can use them in queries. Example query: http://localhost:8983/solr/techproducts/suggest?suggest=true&suggest.dictionary=mySu ggester&suggest.dictionary=altSuggester&wt=json&suggest.q=elec In this example we have sent the string 'elec' as the suggest.q parameter and named two suggest.dictionary definitions to be used. Example response: Apache Solr Reference Guide 5.1 332 { "responseHeader": { "status": 0, "QTime": 3 }, "suggest": { "mySuggester": { "elec": { "numFound": 1, "suggestions": [ { "term": "electronics and computer1", "weight": 100, "payload": "" } ] } }, "altSuggester": { "elec": { "numFound": 1, "suggestions": [ { "term": "electronics and computer1", "weight": 10, "payload": "" } ] } } } } MoreLikeThis The MoreLikeThis search component enables users to query for documents similar to a document in their result list. It does this by using terms from the original document to find similar documents in the index. There are three ways to use MoreLikeThis. The first, and most common, is to use it as a request handler. In this case, you would send text to the MoreLikeThis request handler as needed (as in when a user clicked on a "similar documents" link). The second is to use it as a search component. This is less desirable since it performs the MoreLikeThis analysis on every document returned. This may slow search results. The final approach is to use it as a request handler but with externally supplied text. This case, also referred to as the MoreLikeThisHandler, will supply information about similar documents in the index based on the text of the input document. Covered in this section: How MoreLikeThis Works Common Parameters for MoreLikeThis Parameters for the MoreLikeThisComponent Parameters for the MoreLikeThisHandler Apache Solr Reference Guide 5.1 333 Related Topics How MoreLikeThis Works MoreLikeThis constructs a Lucene query based on terms in a document. It does this by pulling terms from the defined list of fields ( see the mlt.fl parameter, below). For best results, the fields should have stored term vectors in schema.xml. For example: If term vectors are not stored, MoreLikeThis will generate terms from stored fields. A uniqueKey must also be stored in order for MoreLikeThis to work properly. The next phase filters terms from the original document using thresholds defined with the MoreLikeThis parameters. Finally, a query is run with these terms, and any other query parameters that have been defined (see the mlt.qf parameter, below) and a new document set is returned. Common Parameters for MoreLikeThis The table below summarizes the MoreLikeThis parameters supported by Lucene/Solr. These parameters can be used with any of the three possible MoreLikeThis approaches. Parameter Description mlt.fl Specifies the fields to use for similarity. If possible, these should have stored termVectors. mlt.mintf Specifies the Minimum Term Frequency, the frequency below which terms will be ignored in the source document. mlt.mindf Specifies the Minimum Document Frequency, the frequency at which words will be ignored which do not occur in at least this many documents. mlt.maxdf Specifies the Maximum Document Frequency, the frequency at which words will be ignored which occur in more than this many documents. mlt.minwl Sets the minimum word length below which words will be ignored. mlt.maxwl Sets the maximum word length above which words will be ignored. mlt.maxqt Sets the maximum number of query terms that will be included in any generated query. mlt.maxntp Sets the maximum number of tokens to parse in each example document field that is not stored with TermVector support. mlt.boost Specifies if the query will be boosted by the interesting term relevance. It can be either "true" or "false". mlt.qf Query fields and their boosts using the same format as that used by the DisMaxRequestHandler. These fields must also be specified in mlt.fl. Parameters for the MoreLikeThisComponent Using MoreLikeThis as a search component returns similar documents for each document in the response set. In Apache Solr Reference Guide 5.1 334 addition to the common parameters, these additional options are available: Parameter mlt Description If set to true, activates the MoreLikeThis component and enables Solr to return MoreLikeThi s results. mlt.count Specifies the number of similar documents to be returned for each result. The default value is 5. Parameters for the MoreLikeThisHandler The table below summarizes parameters accessible through the MoreLikeThisHandler. It supports faceting, paging, and filtering using common query parameters, but does not work well with alternate query parsers. Parameter Description mlt.match.include Specifies whether or not the response should include the matched document. If set to false, the response will look like a normal select response. mlt.match.offset Specifies an offset into the main query search results to locate the document on which the MoreLikeThis query should operate. By default, the query operates on the first result for the q parameter. mlt.interestingTerms Controls how the MoreLikeThis component presents the "interesting" terms (the top TF/IDF terms) for the query. Supports three settings. The setting list lists the terms. The setting none lists no terms. The setting details lists the terms along with the boost value used for each term. Unless mlt.boost=true, all terms will have boost=1.0. Related Topics RequestHandlers and SearchComponents in SolrConfig Pagination of Results Basic Pagination In most search application usage, the "top" matching results (sorted by score, or some other criteria) are then displayed to some human user. In many applications the UI for these sorted results are displayed to the user in "pages" containing a fixed number of matching results, and users don't typically look at results past the first few pages worth of results. In Solr, this basic paginated searching is supported using the start and rows parameters, and performance of this common behaviour can be tuned by utilizing the queryResultCache and adjusting the queryResultWin dowSize configuration options based on your expected page sizes. Basic Pagination Examples The easiest way to think about simple pagination, is to simply multiply the page number you want (treating the "first" page number as "0") by the number of rows per page; such as in the following psuedo-code: Apache Solr Reference Guide 5.1 335 function fetch_solr_page($page_number, $rows_per_page) { $start = $page_number * $rows_per_page $params = [ q = $some_query, rows = $rows_per_page, start = $start ] return fetch_solr($params) } How Basic Pagination is Affected by Index Updates The start param specified in a request to Solr indicates an absolute "offset" in the complete sorted list of matches that the client wants Solr to use as the beginning of the current "page". If an index modification (such as adding or removing documents) which affects the sequence of ordered documents matching a query occurs in between two requests from a client for subsequent pages of results, then it is possible that these modifications can result in the same document being returned on multiple pages, or documents being "skipped" as the result set shrinks or grows. For example: consider an index containing 26 documents like so: id name 1 A 2 B ... 26 Z Followed by the following requests & index modifications interleaved: A client requests q=*:*&rows=5&start=0&sort=name asc documents with the ids 1-5 will be returned to the client Document id 3 is deleted The client requests "page #2" using q=*:*&rows=5&start=5&sort=name asc Documents 7-11 will be returned Document 6 has been skipped, since it is now the 5th document in the sorted set of all matching results – it would be returned on a new request for "page #1" 3 new documents are now added with the ids 90, 91, and 92; All three documents have a name of A The client requests "page #3" using q=*:*&rows=5&start=10&sort=name asc Documents 9-13 will be returned Documents 9, 10, and 11 have now been returned on both page #2 and page #3 since they moved farther back in the list of sorted results In typical situations these impacts from index changes on paginated searching don't significantly affect user experience -- either because they happen extremely infrequently in fairly static collections, or because the users recognize that the collection of data is constantly evolving and expect to see documents shift up and down in the result sets. Performance Problems with "Deep Paging" In some situations, the results of a Solr search are not destined for a simple paginated user interface. When you Apache Solr Reference Guide 5.1 336 wish to fetch a very large number of sorted results from Solr to feed into an external system, using very large values for the start or rows parameters can be very inefficient. Pagination using start and rows not only require Solr to compute (and sort) in memory all of the matching documents that should be fetched for the current page, but also all of the documents that would have appeared on previous pages. So while a request for start=0&rows=1000000 may be obviously inefficient because it requires Solr to maintain & sort in memory a set of 1 million documents, likewise a request for start=999000&rows=1000 is equally inefficient for the same reasons. Solr can't compute which matching document is the 999001st result in sorted order, without first determining what the first 999000 matching sorted results are. Fetching A Large Number of Sorted Results: Cursors As an alternative to increasing the "start" parameter to request subsequent pages of sorted results, Solr supports using a "Cursor" to scan through results. Cursors in Solr are a logical concept, that doesn't involve caching any state information on the server. Instead the sort values of the last document returned to the client are used to compute a "mark" representing a logical point in the ordered space of sort values. That "mark" can be specified in the parameters of subsequent requests to tell Solr where to continue. Using Cursors To use a cursor with Solr, specify a cursorMark parameter with the value of "*". You can think of this being analogous to start=0 as a way to tell Solr "start at the beginning of my sorted results" except that it also informs Solr that you want to use a Cursor. So in addition to returning the top N sorted results (where you can control N using the rows parameter) the Solr response will also include an encoded String named nextCursor Mark. You then take the nextCursorMark String value from the response, and pass it back to Solr as the cur sorMark parameter for your next request. You can repeat this process until you've fetched as many docs as you want, or until the nextCursorMark returned matches the cursorMark you've already specified -indicating that there are no more results. Constraints when using Cursors There are a few important constraints to be aware of when using cursorMark parameter in a Solr request 1. cursorMark and start are mutually exclusive parameters Your requests must either not include a start parameter, or it must be specified with a value of "0 ". 2. sort clauses must include the uniqueKey field (either "asc" or "desc") If id is your uniqueKey field, then sort params like id asc and name asc, id desc would both work fine, but name asc by itself would not Cursor mark values are computed based on the sort values of each document in the result, which means multiple documents with identical sort values will produce identical Cursor mark values if one of them is the last document on a page of results. In that situation, the subsequent request using that cursorMark would not know which of the documents with the identical mark values should be skipped. Requiring that the uniqueKey field be used as a clause in the sort criteria guarantees that a deterministic ordering will be returned, and that every cursorMark value will identify a unique point in the sequence of documents. Cursor Examples Fetch All Docs Apache Solr Reference Guide 5.1 337 The psuedo-code shown here shows the basic logic involved in fetching all documents matching a query using a cursor: // when fetching all docs, you might as well use a simple id sort // unless you really need the docs to come back in a specific order $params = [ q => $some_query, sort => 'id asc', rows => $r, cursorMark => '*' ] $done = false while (not $done) { $results = fetch_solr($params) // do something with $results if ($params[cursorMark] == $results[nextCursorMark]) { $done = true } $params[cursorMark] = $results[nextCursorMark] } Using SolrJ, this psuedo-code would be: SolrQuery q = (new SolrQuery(some_query)).setRows(r).setSort(SortClause.asc("id")); String cursorMark = CursorMarkParams.CURSOR_MARK_START; boolean done = false; while (! done) { q.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark); QueryResponse rsp = solrServer.query(q); String nextCursorMark = rsp.getNextCursorMark(); doCustomProcessingOfResults(rsp); if (cursorMark.equals(nextCursorMark)) { done = true; } cursorMark = nextCursorMark; } If you wanted to do this by hand using curl, the sequence of requests would look something like this: Apache Solr Reference Guide 5.1 338 $ curl '...&rows=10&sort=id+asc&cursorMark=*' { "response":{"numFound":32,"start":0,"docs":[ // ... 10 docs here ... ]}, "nextCursorMark":"AoEjR0JQ"} $ curl '...&rows=10&sort=id+asc&cursorMark=AoEjR0JQ' { "response":{"numFound":32,"start":0,"docs":[ // ... 10 more docs here ... ]}, "nextCursorMark":"AoEpVkRCREIxQTE2"} $ curl '...&rows=10&sort=id+asc&cursorMark=AoEpVkRCREIxQTE2' { "response":{"numFound":32,"start":0,"docs":[ // ... 10 more docs here ... ]}, "nextCursorMark":"AoEmbWF4dG9y"} $ curl '...&rows=10&sort=id+asc&cursorMark=AoEmbWF4dG9y' { "response":{"numFound":32,"start":0,"docs":[ // ... 2 docs here because we've reached the end. ]}, "nextCursorMark":"AoEpdmlld3Nvbmlj"} $ curl '...&rows=10&sort=id+asc&cursorMark=AoEpdmlld3Nvbmlj' { "response":{"numFound":32,"start":0,"docs":[ // no more docs here, and note that the nextCursorMark // matches the cursorMark param we used ]}, "nextCursorMark":"AoEpdmlld3Nvbmlj"} Fetch first N docs, Based on Post Processing Since the cursor is stateless from Solr's perspective, your client code can stop fetching additional results as soon as you have decided you have enough information: while (! done) { q.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark); QueryResponse rsp = solrServer.query(q); String nextCursorMark = rsp.getNextCursorMark(); boolean hadEnough = doCustomProcessingOfResults(rsp); if (hadEnough || cursorMark.equals(nextCursorMark)) { done = true; } cursorMark = nextCursorMark; } How cursors are Affected by Index Updates Unlike basic pagination, Cursor pagination does not rely on using an absolute "offset" into the completed sorted list of matching documents. Instead, the cursorMark specified in a request encapsulates information about the relative position of the last document returned, based on the absolute sort values of that document. This means that the impact of index modifications is much smaller when using a cursor compared to basic pagination. Apache Solr Reference Guide 5.1 339 Consider the same example index described when discussing basic pagination: id name 1 A 2 B ... 26 Z A client requests q=*:*&rows=5&start=0&sort=name asc, id asc&cursorMark=* Documents with the ids 1-5 will be returned to the client in order Document id 3 is deleted The client requests 5 more documents using the nextCursorMark from the previous response Documents 6-10 will be returned -- the deletion of a document that's already been returned doesn't affect the relative position of the cursor 3 new documents are now added with the ids 90, 91, and 92; All three documents have a name of A The client requests 5 more documents using the nextCursorMark from the previous response Documents 11-15 will be returned -- the addition of new documents with sort values already past does not affect the relative position of the cursor Document id 1 is updated to change it's 'name' to Q Document id 17 is updated to change it's 'name' to A The client requests 5 more documents using the nextCursorMark from the previous response The resulting documents are 16,1,18,19,20 in that order Because the sort value of document 1 changed so that it is after the cursor position, the document is returned to the client twice Because the sort value of document 17 changed so that it is before the cursor position, the document has been "skipped" and will not be returned to the client as the cursor continues to progress In a nutshell: When fetching all results matching a query using cursorMark, the only way index modifications can result in a document being skipped, or returned twice, is if the sort value of the document changes. One way to ensure that a document will never be returned more then once, is to use the uniqueKey field as the primary (and therefore: only significant) sort criterion. In this situation, you will be guaranteed that each document is only returned once, no matter how it may be be modified during the use of the cursor. "Tailing" a Cursor Because Cursor requests are stateless, and the cursorMark values encapsulate the absolute sort values of the last document returned from a search, it's possible to "continue" fetching additional results from a cursor that has already reached its end -- if new documents are added (or existing documents are updated) to the end of the results. You can think of this as similar to using something like " tail -f" in Unix. The most common examples of how this can be useful is when you have a "timestamp" field recording when a Apache Solr Reference Guide 5.1 340 document has been added/updated in your index. Client applications can continuously poll a cursor using a sor t=timestamp asc, id asc for documents matching a query, and always be notified when a document is added or updated matching the request criteria. Another common example is when you have uniqueKey values that always increase as new documents are created, and you can continuously poll a cursor using sort=id asc to be notified about new documents. The psuedo-code for tailing a cursor is only a slight modification from our early example for processing all docs matching a query: while (true) { $doneForNow = false while (not $doneForNow) { $results = fetch_solr($params) // do something with $results if ($params[cursorMark] == $results[nextCursorMark]) { $doneForNow = true } $params[cursorMark] = $results[nextCursorMark] } sleep($some_configured_delay) } Result Grouping Result Grouping groups documents with a common field value into groups and returns the top documents for each group. For example, if you searched for "DVD" on an electronic retailer's e-commerce site, you might be returned three categories such as "TV and Video," "Movies," and "Computers," with three results per category. In this case, the query term "DVD" appeared in all three categories, so Solr groups them together in order to increase relevancy for the user. Result Grouping is separate from Faceting. Though it is conceptually similar, faceting returns all relevant results and allows the user to refine the results based on the facet category. For example, if you searched for "shoes" on a footwear retailer's e-commerce site, you would be returned all results for that query term, along with selectable facets such as "size," "color," "brand," and so on. You can however combined grouping with faceting: Grouped faceting supports facet.field and facet.rang e but currently doesn't support date and pivot faceting. The facet counts are computed based on the first group .field parameter, and other group.field parameters are ignored. Grouped faceting differs from non grouped facets (sum of all facets) == (total of products with that property) as shown in the following example: Object 1 name: Phaser 4620a ppm: 62 product_range: 6 Object 2 name: Phaser 4620i ppm: 65 product_range: 6 Apache Solr Reference Guide 5.1 341 Object 3 name: ML6512 ppm: 62 product_range: 7 If you ask Solr to group these documents by "product_range", then the total amount of groups is 2, but the facets for ppm are 2 for 62 and 1 for 65. Request Parameters Result Grouping takes the following request parameters. Any number of these request parameters can be included in a single request: Parameter Type Description group Boolean If true, query results will be grouped. group.field string The name of the field by which to group results. The field must be single-valued, and either be indexed or a field type that has a value source and works in a function query, such as ExternalFileFiel d. It must also be a string-based field, such as StrField or TextFi eld group.func query Group based on the unique values of a function query. NOTE: This option does not work with distributed searches. group.query query Return a single group of documents that match the given query. rows integer The number of groups to return. The default value is 10. start integer Specifies an initial offset for the list of groups. group.limit integer Specifies the number of results to return for each group. The default value is 1. group.offset integer Specifies an initial offset for the document list of each group. sort sortspec Specifies how Solr sorts the groups relative to each other. For example, sort=popularity desc will cause the groups to be sorted according to the highest popularity document in each group. The default value is score desc. group.sort sortspec Specifies how Solr sorts documents within a single group. The default value is score desc. group.format grouped/simple If this parameter is set to simple, the grouped documents are presented in a single flat list, and the start and rows parameters affect the numbers of documents instead of groups. group.main Boolean Apache Solr Reference Guide 5.1 If true, the result of the first field grouping command is used as the main result list in the response, using group.format=simple. 342 group.ngroups Boolean If true, Solr includes the number of groups that have matched the query in the results. The default value is false. See below for Distributed Result Grouping Caveats when using sharded indexes group.truncate Boolean If true, facet counts are based on the most relevant document of each group matching the query. The default value is false. group.facet Boolean Determines whether to compute grouped facets for the field facets specified in facet.field parameters. Grouped facets are computed based on the first specified group. As with normal field faceting, fields shouldn't be tokenized (otherwise counts are computed for each token). Grouped faceting supports single and multivalued fields. Default is false. See below for Distributed Result Grouping Caveats when using sharded indexes group.cache.percent integer between 0 and 100 Setting this parameter to a number greater than 0 enables caching for result grouping. Result Grouping executes two searches; this option caches the second search. The default value is 0. Testing has shown that group caching only improves search time with Boolean, wildcard, and fuzzy queries. For simple queries like term or "match all" queries, group caching degrades performance. Any number of group commands (group.field, group.func, group.query) may be specified in a single request. Examples All of the following sample queries work with Solr's "bin/solr -e techproducts" example. Grouping Results by Field In this example, we will group results based on the manu_exact field, which specifies the manufacturer of the items in the sample dataset. http://localhost:8983/solr/techproducts/select?wt=json&indent=true&fl=id,name&q=so lr+memory&group=true&group.field=manu_exact Apache Solr Reference Guide 5.1 343 { ... "grouped":{ "manu_exact":{ "matches":6, "groups":[{ "groupValue":"Apache Software Foundation", "doclist":{"numFound":1,"start":0,"docs":[ { "id":"SOLR1000", "name":"Solr, the Enterprise Search Server"}] }}, { "groupValue":"Corsair Microsystems Inc.", "doclist":{"numFound":2,"start":0,"docs":[ { "id":"VS1GB400C3", "name":"CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail"}] }}, { "groupValue":"A-DATA Technology Inc.", "doclist":{"numFound":1,"start":0,"docs":[ { "id":"VDBDB1A16", "name":"A-DATA V-Series 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - OEM"}] }}, { "groupValue":"Canon Inc.", "doclist":{"numFound":1,"start":0,"docs":[ { "id":"0579B002", "name":"Canon PIXMA MP500 All-In-One Photo Printer"}] }}, { "groupValue":"ASUS Computer Inc.", "doclist":{"numFound":1,"start":0,"docs":[ { "id":"EN7800GTX/2DHTV/256M", "name":"ASUS Extreme N7800GTX/2DHTV (256 MB)"}] } } ] } } The response indicates that there are six total matches for our query. For each unique value of group.field, Solr returns a docList with the top scoring document. The docList also includes the total number of matches in that group as the numFound value. The groups are sorted by the score of the top document within each group. We can run the same query with the request parameter group.main=true. This will format the results as a single flat document list. This flat format does not include as much information as the normal result grouping query results, but it may be easier for existing Solr clients to parse. http://localhost:8983/solr/techproducts/select?wt=json&indent=true&fl=id,name,manu Apache Solr Reference Guide 5.1 344 facturer&q=solr+memory&group=true&group.field=manu_exact&group.main=true { "responseHeader":{ "status":0, "QTime":1, "params":{ "fl":"id,name,manufacturer", "indent":"true", "q":"solr memory", "group.field":"manu_exact", "group.main":"true", "group":"true", "wt":"json"}}, "grouped":{}, "response":{"numFound":6,"start":0,"docs":[ { "id":"SOLR1000", "name":"Solr, the Enterprise Search Server"}, { "id":"VS1GB400C3", "name":"CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail"}, { "id":"VDBDB1A16", "name":"A-DATA V-Series 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - OEM"}, { "id":"0579B002", "name":"Canon PIXMA MP500 All-In-One Photo Printer"}, { "id":"EN7800GTX/2DHTV/256M", "name":"ASUS Extreme N7800GTX/2DHTV (256 MB)"}] } } Grouping by Query In this example, we will use the group.query parameter to find the top three results for "memory" in two different price ranges: 0.00 to 99.99, and over 100. http://localhost:8983/solr/techproducts/select?wt=json&indent=true&fl=name,price&q =memory&group=true&group.query=price:[0+TO+99.99]&group.query=price:[100+TO+*]&gro up.limit=3 Apache Solr Reference Guide 5.1 345 { "responseHeader":{ "status":0, "QTime":42, "params":{ "fl":"name,price", "indent":"true", "q":"memory", "group.limit":"3", "group.query":["price:[0 TO 99.99]", "price:[100 TO *]"], "group":"true", "wt":"json"}}, "grouped":{ "price:[0 TO 99.99]":{ "matches":5, "doclist":{"numFound":1,"start":0,"docs":[ { "name":"CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail", "price":74.99}] }}, "price:[100 TO *]":{ "matches":5, "doclist":{"numFound":3,"start":0,"docs":[ { "name":"CORSAIR XMS 2GB (2 x 1GB) 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) Dual Channel Kit System Memory - Retail", "price":185.0}, { "name":"Canon PIXMA MP500 All-In-One Photo Printer", "price":179.99}, { "name":"ASUS Extreme N7800GTX/2DHTV (256 MB)", "price":479.95}] } } } } In this case, Solr found five matches for "memory," but only returns four results grouped by price. This is because one result for "memory" did not have a price assigned to it. Distributed Result Grouping Caveats Grouping is supported for distributed searches, with some caveats: Currently group.func is is not supported in any distributed searches group.ngroups and group.facet require that all documents in each group must be co-located on the same shard in order for accurate counts to be returned. Document routing via composite keys can be a useful solution in many situations. Apache Solr Reference Guide 5.1 346 Collapse and Expand Results The collapsing query parser and the expand component combine to form an approach to grouping documents for field collapsing in search results. Collapsing Query Parser The CollapsingQParser is really a post filter that provides more performant field collapsing than Solr's standard approach when the number of distinct groups in the result set is high. This parser collapses the result set to a single document per group before it forwards the result set to the rest of the search components. So all downstream components (faceting, highlighting, etc...) will work with the collapsed result set. The CollapsingQParser accepts the following local parameters: Parameter Description Default field The field that is being collapsed on. The field must be a single valued String, Int or Float none min | max Selects the group heads by the min or max value of a numeric field or function query. none If not specified group heads will be selected based on the highest scoring documents in the result set. nullPolicy There are three null policies: ignore ignore: removes documents with a null value in the collapse field. This is the default. expand: treats each document with a null value in the collapse field as a separate group. collapse: collapses all documents with a null value into a single group using either highest score, or minimum/maximum. hint Currently there is only one hint available "top_fc", which stands for top level FieldCache. The top_fc hint is only available when collapsing on String fields. top_fc provides the best query time speed but takes the longest to warm on startup or following a commit. top_fc also will result in having the collapsed field cached in memory twice if the it's used for faceting or sorting. none size Sets the initial size of the collapse data structures when collapsing on a numeric field only. The data structures used for collapsing grow dynamically when collapsing on numeric fields. Setting the size above the number of results expected in the result set will eliminate the resizing cost. 100,000 Sample Syntax: Collapse based on the highest scoring document: fq={!collapse field=field_name} Collapse based on the minimum value of a numeric field: Apache Solr Reference Guide 5.1 347 fq={!collapse field=field_name min=field_name} Collapse based on the maximum value of a numeric field: fq={!collapse field=field_name max=field_name} Collapse based on the min/max value of a function. The cscore() function can be used with the CollapsingQParserPlugin to return the score of the current document being collapsed. fq={!collapse field=field_name max=sum(cscore(),field(A))} Collapse with a null policy: fq={!collapse field=field_name nullPolicy=nullPolicy} Collapse with a hint: fq={!collapse field=field_name hint=top_fc} The CollapsingQParserPlugin fully supports the QueryElevationComponent. Expand Component The ExpandComponent can be used to expand the groups that were collapsed by the CollapsingQParserPlugin. Example usage with the CollapsingQParserPlugin: q=foo&fq={!collapse field=ISBN} In the query above, the CollapsingQParserPlugin will collapse the search results on the ISBN field. The main search results will contain the highest ranking document from each book. The ExpandComponent can now be used to expand the results so you can see the documents grouped by ISBN. For example: q=foo&fq={!collapse field=ISBN}&expand=true The “expand=true” parameter turns on the ExpandComponent. The ExpandComponent adds a new section to the search output labeled “expanded”. Inside the expanded section there is a map with each group head pointing to the expanded documents that are within the group. As applications iterate the main collapsed result set, they can access the expanded map to retrieve the expanded groups. The ExpandComponent has the following parameters: Parameter Apache Solr Reference Guide 5.1 Description Default 348 expand.sort Orders the documents within the expanded groups score desc expand.rows The number of rows to display in each group 5 expand.q Overrides the main q parameter, determines which documents to include in the main group. main q expand.fq Overrides main fq's, determines which documents to include in the main group. main fq's Result Clustering The clustering (or cluster analysis) plugin attempts to automatically discover groups of related search hits (documents) and assign human-readable labels to these groups. By default in Solr, the clustering algorithm is applied to the search result of each single query—this is called an on-line clustering. While Solr contains an extension for for full-index clustering (off-line clustering) this section will focus on discussing on-line clustering only. Clusters discovered for a given query can be perceived as dynamic facets. This is beneficial when regular faceting is difficult (field values are not known in advance) or when the queries are exploratory in nature. Take a look at the Carrot2 project's demo page to see an example of search results clustering in action (the groups in the visualization have been discovered automatically in search results to the right, there is no external information involved). The query issued to the system was Solr. It seems clear that faceting could not yield a similar set of groups, although the goals of both techniques are similar—to let the user explore the set of search results and either rephrase the query or narrow the focus to a subset of current documents. Clustering is also similar to Result Grouping in that it can help to look deeper into search results, beyond the top few hits. Topics covered in this section: Preliminary Concepts Quick Start Example Installation Configuration Tweaking Algorithm Settings Apache Solr Reference Guide 5.1 349 Performance Considerations Additional Resources Preliminary Concepts Each document passed to the clustering component is composed of several logical parts: a unique identifier, origin URL, the title, the main content, a language code of the title and content. The identifier part is mandatory, everything else is optional but at least one of the text fields (title or content) will be required to make the clustering process reasonable. It is important to remember that logical document parts must be mapped to a particular schema and its fields. The content (text) for clustering can be sourced from either a stored text field or context-filtered using a highlighter, all these options are explained below in the configuration section. A clustering algorithm is the actual logic (implementation) that discovers relationships among the documents in the search result and forms human-readable cluster labels. Depending on the choice of the algorithm the clusters may (and probably will) vary. Solr comes with several algorithms implemented in the open source Carrot2 project , commercial alternatives also exist. Quick Start Example The "techproducts" example included with Solr is pre-configured with all the necessary components for result clustering - but they are disabled by default. To enable the clustering component contrib and a dedicated search handler configured to use it, specify the " -a" option to set a JVM System Property when running the example: bin/solr start -e techproducts -a '-Dsolr.clustering.enabled=true' You can now try out the clustering handler by opening the following URL in a browser: http://localhost:89 83/solr/techproducts/clustering?q=*:*&rows=100 The output XML should include search hits and an array of automatically discovered clusters at the end, resembling the output shown here: 0 299 GB18030TEST Test with some GB18030 encoded characters No accents here This is a feature (translated) Apache Solr Reference Guide 5.1 350 This document is very shiny (translated) 0.0 0,USD true 1448955395025403904 1.0 DDR 3.9599865057283354 TWINX2048-3200PRO VS1GB400C3 VDBDB1A16 iPod 11.959228467119022 F8V7067-APL-KIT IW-02 MA147LL/A Other Topics 0.0 true adata apple asus ati Apache Solr Reference Guide 5.1 351 There were a few clusters discovered for this query (*:*), separating search hits into various categories: DDR, iPod, Hard Drive, etc. Each cluster has a label and score that indicates the "goodness" of the cluster. The score is algorithm-specific and is meaningful only in relation to the scores of other clusters in the same set. In other words, if cluster A has a higher score than cluster B, cluster A should be of better quality (have a better label and/or more coherent document set). Each cluster has an array of identifiers of documents belonging to it. These identifiers correspond to the uniqueKey field declared in the schema. Depending on the quality of input documents, some clusters may not make much sense. Some documents may be left out and not be clustered at all; these will be assigned to the synthetic Other Topics group, marked with the other-topics property set to true (see the XML dump above for an example). The score of the other topics group is zero. Installation The clustering contrib extension requires dist/solr-clustering-*.jar and all JARs under contrib/clu stering/lib. Configuration Declaration of the Search Component and Request Handler Clustering extension is a search component and must be declared in solrconfig.xml. Such a component can be then appended to a request handler as the last component in the chain (because it requires search results which must be previously fetched by the search component). An example configuration could look as shown below. 1. Include the required contrib JARs. Note that by default paths are relative to the Solr core so they may need adjustments to your configuration, or an explicit specification of the $solr.install.dir. 2. Declaration of the search component. Each component can also declare multiple clustering pipelines ("engines"), which can be selected at runtime. Apache Solr Reference Guide 5.1 352 lingo org.carrot2.clustering.lingo.LingoClusteringAlgorithm< /str> stc org.carrot2.clustering.stc.STCClusteringAlgorithm 3. A request handler to which we append the clustering component declared above. true true name="carrot.url">id name="carrot.title">doctitle name="carrot.snippet">content 100 *,score clustering Configuration Parameters of the Clustering Component The table below summarizes parameters of each clustering engine or the entire clustering component (depending where they are declared). Parameter clustering Apache Solr Reference Guide 5.1 Description When true, clustering component is enabled. 353 clustering.engine Declares which clustering engine to use. If not present, the first declared engine will become the default one. clustering.results When true, the component will perform clustering of search results (this should be enabled). clustering.collection When true, the component will perform clustering of the whole document index (this section does not cover full-index clustering). At the engine declaration level, the following parameters are supported. Parameter Description carrot.algorithm The algorithm class. carrot.resourcesDir Algorithm-specific resources and configuration files (stop words, other lexical resources, default settings). By default points to conf/clusterin g/carrot2/ carrot.outputSubClusters If true and the algorithm supports hierarchical clustering, sub-clusters will also be emitted. carrot.numDescriptions Maximum number of per-cluster labels to return (if the algorithm assigns more than one label to a cluster). The carrot.algorithm parameter should contain a fully qualified class name of an algorithm supported by the Carrot2 framework. Currently, the following algorithms are available: org.carrot2.clustering.lingo.LingoClusteringAlgorithm (open source) org.carrot2.clustering.stc.STCClusteringAlgorithm (open source) org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithm (open source) com.carrotsearch.lingo3g.Lingo3GClusteringAlgorithm (commercial) For a comparison of characteristics of these algorithms see the following links: http://doc.carrot2.org/#section.advanced-topics.fine-tuning.choosing-algorithm http://project.carrot2.org/algorithms.html http://carrotsearch.com/lingo3g-comparison.html The question of which algorithm to choose depends on the amount of traffic (STC is faster than Lingo, but arguably produces less intuitive clusters, Lingo3G is the fastest algorithm but is not free or open source), expected result (Lingo3G provides hierarchical clusters, Lingo and STC provide flat clusters), and the input data (each algorithm will cluster the input slightly differently). There is no one answer which algorithm is "the best". Contextual and Full Field Clustering The clustering engine can apply clustering to the full content of (stored) fields or it can run an internal highlighter pass to extract context-snippets before clustering. Highlighting is recommended when the logical snippet field contains a lot of content (this would affect clustering performance). Highlighting can also increase the quality of clustering because the content passed to the algorithm will be more focused around the query (it will be query-specific context). The following parameters control the internal highlighter. Parameter Apache Solr Reference Guide 5.1 Description 354 carrot.produceSummary When true the clustering component will run a highlighter pass on the content of logical fields pointed to by carrot.title and carrot.snippet. Otherwise full content of those fields will be clustered. The size, in characters, of the snippets (aka fragments) created by the highlighter. If not specified, the default highlighting fragsize (hl.fragsize) carrot.fragSize will be used. carrot.summarySnippets The number of summary snippets to generate for clustering. If not specified, the default highlighting snippet count (hl.snippets) will be used. Logical to Document Field Mapping As already mentioned in Preliminary Concepts, the clustering component clusters "documents" consisting of logical parts that need to be mapped onto physical schema of data stored in Solr. The field mapping attributes provide a connection between fields and logical document parts. Note that the content of title and snippet fields must be stored so that it can be retrieved at search time. Parameter Description carrot.title The field (alternatively comma- or space-separated list of fields) that should be mapped to the logical document's title. The clustering algorithms typically give more weight to the content of the title field compared to the content (snippet). For best results, the field should contain concise, noise-free content. If there is no clear title in your data, you can leave this parameter blank. carrot.snippet The field (alternatively comma- or space-separated list of fields) that should be mapped to the logical document's main content. If this mapping points to very large content fields the performance of clustering may drop significantly. An alternative then is to use query-context snippets for clustering instead of full field content. See the description of the carrot.produceSummary parameter for details. carrot.url The field that should be mapped to the logical document's content URL. Leave blank if not required. Clustering Multilingual Content The field mapping specification can include a carrot.lang parameter, which defines the field that stores ISO 639-1 code of the language in which the title and content of the document are written. This information can be stored in the index based on apriori knowledge of the documents' source or a language detection filter applied at indexing time. All algorithms inside the Carrot2 framework will accept ISO codes of languages defined in Langua geCode enum. The language hint makes it easier for clustering algorithms to separate documents from different languages on input and to pick the right language resources for clustering. If you do have multi-lingual query results (or query results in a language different than English), it is strongly advised to map the language field appropriately. Parameter carrot.lang Description The field that stores ISO 639-1 code of the language of the document's text fields. Apache Solr Reference Guide 5.1 355 carrot.lcmap A mapping of arbitrary strings into ISO 639 two-letter codes used by carrot.lang. The syntax of this parameter is the same as langid.map.lcmap, for example: langid.map. lcmap=japanese:ja polish:pl english:en The default language can also be set using Carrot2-specific algorithm attributes (in this case the MultilingualClust ering.defaultLanguage attribute). Tweaking Algorithm Settings The algorithms that come with Solr are using their default settings which may be inadequate for all data sets. All algorithms have lexical resources and resources (stop words, stemmers, parameters) that may require tweaking to get better clusters (and cluster labels). For Carrot2-based algorithms it is probably best to refer to a dedicated tuning application called Carrot2 Workbench (screenshot below). From this application one can export a set of algorithm attributes as an XML file, which can be then placed under the location pointed to by carrot.resourc esDir. Providing Defaults The default attributes for all engines (algorithms) declared in the clustering component are placed under carrot .resourcesDir and with an expected file name of engineName-attributes.xml. So for an engine named lingo and the default value of carrot.resourcesDir, the attributes would be read from a file in conf/clus tering/carrot2/lingo-attributes.xml. An example XML file changing the default language of documents to Polish is shown below. Apache Solr Reference Guide 5.1 356 Tweaking at Query-Time The clustering component and Carrot2 clustering algorithms can accept query-time attribute overrides. Note that certain things (for example lexical resources) can only be initialized once (at startup, via the XML configuration files). An example query that changes the LingoClusteringAlgorithm.desiredClusterCountBase parameter for the Lingo algorithm: http://localhost:8983/solr/techproducts/clustering?q=*:*&rows=100 &LingoClusteringAlgorithm.desiredClusterCountBase=20 Performance Considerations Dynamic clustering of search results comes with two major performance penalties: Increased cost of fetching a larger-than-usual number of search results (50, 100 or more documents), Additional computational cost of the clustering itself. For simple queries, the clustering time will usually dominate the fetch time. If the document content is very long the retrieval of stored content can become a bottleneck. The performance impact of clustering can be lowered in several ways: feed less content to the clustering algorithm by enabling carrot.produceSummary attribute, perform clustering on selected fields (titles only) to make the input smaller, use a faster algorithm (STC instead of Lingo, Lingo3G instead of STC), tune the performance attributes related directly to a specific algorithm. Some of these techniques are described in Apache SOLR and Carrot2 integration strategies document, available at http://carrot2.github.io/solr-integration-strategies. The topic of improving performance is also included in the Carrot2 manual at http://doc.carrot2.org/#section.advanced-topics.fine-tuning.performance. Additional Resources The following resources provide additional information about the clustering component in Solr and its potential applications. Apache Solr and Carrot2 integration strategies: http://carrot2.github.io/solr-integration-strategies Apache Solr Wiki (covers previous Solr versions, may be inaccurate): http://carrot2.github.io/solr-integratio n-strategies Clustering and Visualization of Solr search results (video from Berlin BuzzWords conference, 2011): http:// vimeo.com/26616444 Apache Solr Reference Guide 5.1 357 Spatial Search Solr supports location data for use in spatial/geospatial searches. Using spatial search, you can: Index points or other shapes Filter search results by a bounding box or circle or by other shapes Sort or boost scoring by distance between points, or relative area between rectangles Generate a 2D grid of facet count numbers for heatmap generation or point-plotting. There are three main field types available for spatial search: LatLonType (and its non-geodetic twin PointType) SpatialRecursivePrefixTreeFieldType (RPT for short) BBoxField RPT offers more features than LatLonType and fast filter performance, although LatLonType is more appropriate when efficient distance sorting/boosting is desired. They can both be used simultaneously for what each does best – LatLonType for sorting/boosting, RPT for filtering. BBoxField is for indexing bounding boxes, querying by a box, specifying a search predicate (Intersects,Within,Contains,Disjoint,Equals), and a relevancy sort/boost like overlapRatio or simply the area. For more information on Solr spatial search, see http://wiki.apache.org/solr/SpatialSearch. Indexing and Configuration For indexing geodetic points (latitude and longitude), supply the pair of numbers as a string with a comma separating them in latitude then longitude order. For non-geodetic points, the order is x,y for PointType, and for RPT you must use a space instead of a comma, or use WKT. See the section SpatialRecursivePrefixTreeFieldType below for RPT configuration specifics. Spatial Filters There are 2 types of Spatial filters, which both support the following parameters: Parameter d Description the radial distance, usually in kilometers. (RPT & BBoxField can set other units via the setting di stanceUnits) pt the center point using the format "lat,lon" if latitude & longitude. Otherwise, "x,y" for PointType or "x y" for RPT field types. sfield a spatial indexed field Apache Solr Reference Guide 5.1 358 score (Advanced option; RPT and BBoxField field types only) If the query is used in a scoring context (e.g. as the main query in q), this local-param determines what scores will be produced. Valid values are: none - A fixed score of 1.0. (the default) kilometers - distance in kilometers between the field value and the specified center point miles - distance in miles between the field value and the specified center point degrees - distance in degrees between the field value and the specified center point distance - distance between the field value and the specified center point in the distance Units configured for this field recipDistance - 1 / the distance When used with BBoxField,additional options are supported: overlapRatio - The relative overlap between the indexed shape & query shape. area - haversine based area of the overlapping shapes expressed in terms of the distanc eUnits configured for this field area2D - cartesian coordinates based area of the overlapping shapes expressed in terms of the distanceUnits configured for this field filter (Advanced option; RPT and BBoxField field types only) If you only want the query to score (with the above score local-param), not filter, then set this local-param to false. geofilt The geofilt filter allows you to retrieve results based on the geospatial distance (AKA the "great circle distance") from a given point. Another way of looking at it is that it creates a circular shape filter. For example, to find all documents within five kilometers of a given lat/lon point, you could enter &q=*:*&fq={!geofilt sfield=store}&pt=45.15,-93.85&d=5. This filter returns all results within a circle of the given radius around the initial point: bbox The bbox filter is very similar to geofilt except it uses the bounding box of the calculated circle. See the blue box in the diagram below. It takes the same parameters as geofilt. Here's a sample query: &q=*:*&fq={!bbox sfield=store}&pt=45.15,-93.85&d=5. The rectangular shape is faster to compute and so it's sometimes used as an alternative to geofilt when it's acceptable to return points outside of the radius. However, if the ideal goal is a circle but you want it to run faster, then instead consider using the RPT field and try a large "distErrPct" value like 0.1 (10% radius). This will return results outside the radius but it will do so somewhat uniformly around the shape. Apache Solr Reference Guide 5.1 359 When a bounding box includes a pole, the bounding box ends up being a "bounding bowl" (a spherical cap) that includes all values north of the lowest latitude of the circle if it touches the north pole (or south of the highest latitude if it touches the south pole). Filtering by an arbitrary rectangle Sometimes the spatial search requirement calls for finding everything in a rectangular area, such as the area covered by a map the user is looking at. For this case, geofilt and bbox won't cut it. This is somewhat of a trick, but you can use Solr's range query syntax for this by supplying the lower-left corner as the start of the range and the upper-right corner as the end of the range. Here's an example: &q=*:*&fq=store:[45,-94 TO 46,-9 3]. LatLonType does not support rectangles that cross the dateline, but RPT does. If you are using RPT with non-geospatial coordinates (geo="false") then you must quote the points due to the space, e.g. "x y". Optimization: Solr Post Filtering Most likely, the fastest spatial filters will be to simply use the RPT field type. However, sometimes it may be faster to use LatLonType with Solr post filtering in circumstances when both the spatial query isn't worth caching and there aren't many matching documents that match the non-spatial filters (e.g. keyword queries and other filters). To use Solr post filtering with LatLonType, use the bbox or geofilt query parsers in a filter query but specify cache=false and cost=100 (or greater) as local-params. Here's a short example: &q=...mykeywords...&fq=...someotherfilters...&fq={!geofilt cache=false cost=100}&sfield=store&pt=45.15,-93.85&d=5 Distance Function Queries There are four distance function queries: geodist, see below, usually the most appropriate; dist, to calculate the p-norm distance between multi-dimensional vectors; hsin, to calculate the distance between two points on a sphere; and sqedist, to calculate the squared Euclidean distance between two points. For more information about these function queries, see the section on Function Queries. geodist geodist is a distance function that takes three optional parameters: (sfield,latitude,longitude). You can use the geodist function to sort results by distance or score return results. For example, to sort your results by ascending distance, enter ...&q=*:*&fq={!geofilt}&sfield=store& pt=45.15,-93.85&d=50&sort=geodist() asc. To return the distance as the document score, enter ...&q={!func}geodist()&sfield=store&pt=45.15 ,-93.85&sort=score+asc. Apache Solr Reference Guide 5.1 360 More Examples Here are a few more useful examples of what you can do with spatial search in Solr. Use as a Sub-Query to Expand Search Results Here we will query for results in Jacksonville, Florida, or within 50 kilometers of 45.15,-93.85 (near Buffalo, Minnesota): &q=*:*&fq=(state:"FL" AND city:"Jacksonville") OR {!geofilt}&sfield=store&pt=45.15,-93.85&d=50&sort=geodist()+asc Facet by Distance To facet by distance, you can use the Frange query parser: &q=*:*&sfield=store&pt=45.15,-93.85&facet.query={!frange l=0 u=5}geodist()&facet.query={!frange l=5.001 u=3000}geodist() There are other ways to do it too, like using a {!geofilt} in each facet.query. Boost Nearest Results Using the DisMax or Extended DisMax, you can combine spatial search with the boost function to boost the nearest results: &q.alt=*:*&fq={!geofilt}&sfield=store&pt=45.15,-93.85&d=50&bf=recip(geodist(),2,20 0,20)&sort=score desc SpatialRecursivePrefixTreeFieldType (abbreviated as RPT) This field type offers several functional improvements over LatLonType: Query by polygons and other complex shapes, in addition to circles & rectangles Multi-valued indexed fields Ability to index non-point shapes (e.g. polygons) as well as point shapes Rectangles with user-specified corners that can cross the dateline Multi-value distance sort and score boosting (warning: non-optimized) Well-Known-Text (WKT) shape syntax (required for specifying polygons & other complex shapes) RPT incorporates the basic features of LatLonType and PointType, such as lat-lon bounding boxes and circles, in addition to supporting geofilt, bbox, geodist, and a range-queries. Schema configuration To use RPT, the field type must be registered and configured in schema.xml. There are many options for this field type. Setting name Description The name of the field type. Apache Solr Reference Guide 5.1 361 class This should be solr.SpatialRecursivePrefixTreeFieldType. But be aware that the Lucene spatial module includes some other so-called "spatial strategies" other than RPT, notably TermQueryPT*, BBox, PointVector*, and SerializedDV. Solr requires a field type to parallel these in order to use them. The asterisked ones have them. spatialContextFactory If polygons or linestrings are required, then JTS Topology Suite is a needed to implement them. It's a JAR file that you need to put on Solr's classpath (but not via the standard solrconfig.xml mechanisms). If you intend to use those shapes, set this attribute to com.spatial4j.core.context.jts.JtsSpatialContextFactory . Furthermore, the context factory has its own options which are directly configurable on the Solr field type here; follow the link to the Javadocs, and remember to look at the superclass's options in SpatialContextFactory as well. One option in particular you should most likely enable is autoIndex (i.e. use PreparedGeometry) as it's been shown to be a major performance boost for polygons. Further details about specifying polygons to index or query are at Solr's Wiki linked below. geo If true, the default, latitude and longitude coordinates will be used and the mathematical model will generally be a sphere. If false, the coordinates will be generic X & Y on a 2D plane using Euclidean/Cartesian geometry. distanceUnits This is used to specify the units for distance measurements used throughout the use of this field. This can be degrees, kilometers or miles. It is applied to nearly all distance measurements involving the field: maxDistErr, distErr, d, geodist and the score when score is distance, area, or area2d. However, it doesn't affect distances embedded in WKT strings, (eg: "BUFFER(POINT(200 10),0.2))"), which are still in degrees. distanceUnits defaults to either "kilometers" if geo is "true", or "degress" if g eo is "false". distanceUnits replaces the units attribute; which is now deprecated and mutually exclusive with this attribute. distErrPct Defines the default precision of non-point shapes (both index & query), as a fraction between 0.0 (fully precise) to 0.5. The closer this number is to zero, the more accurate the shape will be. However, more precise indexed shapes use more disk space and take longer to index. Bigger distErrPct values will make queries faster but less accurate. maxDistErr Defines the highest level of detail required for indexed data. If left blank, the default is one meter – just a bit less than 0.000009 degrees. This setting is used internally to compute an appropriate maxLevels (see below). worldBounds Defines the valid numerical ranges for x and y, in the format of ENVELOPE(minX, maxX, maxY, minY). If geo="true", the standard lat-lon world boundaries are assumed. If geo=false, you should define your boundaries. distCalculator Defines the distance calculation algorithm. If geo=true, "haversine" is the default. If geo=false, "cartesian" will be the default. Other possible values are "lawOfCosines", "vincentySphere" and "cartesian^2". Apache Solr Reference Guide 5.1 362 prefixTree Defines the spatial grid implementation. Since a PrefixTree (such as RecursivePrefixTree) maps the world as a grid, each grid cell is decomposed to another set of grid cells at the next level. If geo=true then the default prefix tree is "geohash", otherwise it's "quad". Geohash has 32 children at each level, quad has 4. Geohash can only be used for geo=true as it's strictly geospatial. maxLevels Sets the maximum grid depth for indexed data. Instead, it's usually more intuitive to compute an appropriate maxLevels by specifying maxDistErr . Once the field type has been defined, use it to define a field that uses it. Because RPT has more advanced features, some of which are new and experimental, please review the Solr Wiki at http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4 for more information about using this field type. Heatmap Faceting The RPT field supports generating a 2D grid of facet counts for documents having spatial data in each grid cell. For high-detail grids, this can be used to plot points, and for lesser detail it can be used for heatmap generation. The grid cells are determined at index-time based on RPT's configuration. At facet counting time, the indexed cells in the region of interest are traversed and a grid of counters corresponding to each cell are incremented. Solr can return the data in a straight-forward 2D array of integers or in a PNG which compresses better for larger data sets but must be decoded. The heatmap feature is accessed from Solr's faceting feature. As a part of faceting, it supports the key local-par am as well as excluding tagged filter queries, just like other types of faceting do. This allows multiple heatmaps to be returned on the same field with different filters. Parameter Description facet Set to true to enable faceting facet.heatmap The field name of type RPT facet.heatmap.geom The region to compute the heatmap on, specified using the rectangle-range syntax or WKT. It defaults to the world. ex: ["-180 -90" TO "180 90"] facet.heatmap.gridLevel A specific grid level, which determines how big each grid cell is. Defaults to being computed via distErrPct (or distErr) facet.heatmap.distErrPct A fraction of the size of geom used to compute gridLevel. Defaults to 0.15. It's computed the same as a similarly named parameter for RPT. facet.heatmap.distErr A cell error distance used to pick the grid level indirectly. It's computed the same as a similarly named parameter for RPT. Apache Solr Reference Guide 5.1 363 facet.heatmap.format The format, either ints2D (default) or png. Tip For heatmap purposes, you'll experiment with different distErrPct values (probably 0.10 - 0.20) with various input geometries till the default size is what you're looking for. The specific details of how it's computed isn't important. For high-detail grids used in point-plotting (loosely one cell per pixel), set distErr to be the number of decimal-degrees of several pixels or so of the map being displayed. Here's some sample output in JSON (with some ..... inserted for brevity): {gridLevel=6,columns=64,rows=64,minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0, counts_ints2D=[[0, 0, 2, 1, ....],[1, 1, 3, 2, ...],...]} The output shows the gridLevel which is interesting since it's often computed from other parameters. If an interface being developed allows an explicit resolution increase/decrease feature then subsequent requests can specify the gridLevel explicitly. The minX, maxX, minY, maxY reports the region where the counts are. This is the minimally enclosing bounding rectangle of the input geom at the target grid level. This may wrap the dateline. The columns and rows values are how many columns and rows that the output rectangle is to be divided by evenly. Note: Don't divide an on-screen projected map rectangle evenly to plot these rectangles/points since the cell data is in the coordinate space of decimal degrees if geo=true or whatever units were given if geo=false. This could be arranged to be the same as an on-screen map but won't necessarily be. The counts_ints2D key has a 2D array of integers. The initial outer level is in row order (top-down), then the inner arrays are the columns (left-right). If any array would be all zeros, a null is returned instead for efficiency reasons. The entire value is null if there is no matching spatial data. If format=png then the output key is counts_png. It's a base-64 encoded string of a 4-byte PNG. The PNG logically holds the same data that the ints2D does. Note that the alpha channel byte is flipped to make it easier to view the PNG for diagnostic purposes, since otherwise counts would have to exceed 2^24 before it becomes non-opague. Thus counts greater than this value will become opaque. BBoxField The BBoxField field type indexes a single rectangle (bounding box) per document field and supports searching via a bounding box. It supports most spatial search predicates, it has enhanced relevancy modes based on the overlap or area between the search rectangle and the indexed rectangle. It's particularly useful for its relevancy modes. To configure it in the schema, use a configuration like this: BBoxField is actually based off of 4 instances of another field type referred to by numberType. It also uses a boolean to flag a dateline cross. Assuming you want to use the relevancy feature, docValues is required. Some of the attributes are in common with the RPT field like geo, units, worldBounds, and spatialContextFactory Apache Solr Reference Guide 5.1 364 because they share some of the same spatial infrastructure. To index a box, add a field value to a bbox field that's a string in the WKT/CQL ENVELOPE syntax. Example: EN VELOPE(-10, 20, 15, 10) which is minX, maxX, maxY, minY order. The parameter ordering is unintuitive but that's what the spec calls for. To search, you can use the {!bbox} query parser, or the range syntax e.g. [10,-10 TO 15,20], or the ENVELOPE syntax wrapped in parenthesis with a leading search predicate. The latter is the only way to choose a predicate other than Intersects. For example: &q={!field f=bbox}Contains(ENVELOPE(-10, 20, 15, 10)) Now to sort the results by one of the relevancy modes, use it like this: &q={!field f=bbox score=overlapRatio}Intersects(ENVELOPE(-10, 20, 15, 10)) The score local-param can be one of overlapRatio, area, and area2D. area scores by the document area using surface-of-a-sphere (assuming geo=true) math, area2D uses simple width * height. overlapRatio computes a [0-1] ranged score based on how much overlap exists relative to the document's area and the query area. The javadocs of BBoxOverlapRatioValueSource have more info on the formula, if you're really curious. There is an additional parameter queryTargetProportion that allows you to weight the query side of the formula to the index (target) side of the formula. You can also use &debug=results to see useful score computation info. The Terms Component The Terms Component provides access to the indexed terms in a field and the number of documents that match each term. This can be useful for building an auto-suggest feature or any other feature that operates at the term level instead of the search or document level. Retrieving terms in index order is very fast since the implementation directly uses Lucene's TermEnum to iterate over the term dictionary. In a sense, this search component provides fast field-faceting over the whole index, not restricted by the base query or any filters. The document frequencies returned are the number of documents that match the term, including any documents that have been marked for deletion but not yet removed from the index. Configuring the Terms Component By default, the Terms Component is already configured in solrconfig.xml for each collection. Defining the Terms Component Defining the Terms search component is straightforward: simply give it a name and use the class solr.TermsC omponent. This makes the component available for use, but by itself will not be useable until included with a request handler. Using the Terms Component in a Request Handler Apache Solr Reference Guide 5.1 365 The /terms request handler is also defined in solrConfig.xml by default. true false terms Note that the defaults for the this request handler set the parameter "terms" to true, which allows terms to be returned on request. The parameter "distrib" is set to false, which allows this handler to be used only on a single Solr core. To finish out the configuration, he Terms Component is included as an available component to this request handler. You could add this component to another handler if you wanted to, and pass "terms=true" in the HTTP request in order to get terms back. If it is only defined in a separate handler, you must use that handler when querying in order to get terms and not regular documents as results. Terms Component Parameters The parameters below allow you to control what terms are returned. You can also any of these to the request handler if you'd like to set them permanently. Or, you can add them to the query request. These parameters are: Parameter terms Required Default No false Description If set to true, enables the Terms Component. By default, the Terms Component is off. Example: terms=true terms.fl Yes null Specifies the field from which to retrieve terms. Example: terms.fl=title terms.limit No 10 Specifies the maximum number of terms to return. The default is 10. If the limit is set to a number less than 0, then no maximum limit is enforced. Although this is not required, either this parameter or term s.upper must be defined. Example: terms.limit=20 terms.lower No empty string Specifies the term at which to start. If not specified, the empty string is used, causing Solr to start at the beginning of the field. Example: terms.lower=orange terms.lower.incl No true If set to true, includes the lower-bound term (specified with terms.l ower in the result set. Example: terms.lower.incl=false Apache Solr Reference Guide 5.1 366 terms.mincount No null Specifies the minimum document frequency to return in order for a term to be included in a query response. Results are inclusive of the mincount (that is, >= mincount). Example: terms.mincount=5 terms.maxcount No null Specifies the maximum document frequency a term must have in order to be included in a query response. The default setting is -1, which sets no upper bound. Results are inclusive of the maxcount (that is, <= maxcount). Example: terms.maxcount=25 terms.prefix No null Restricts matches to terms that begin with the specified string. Example: terms.prefix=inter terms.raw No false If set to true, returns the raw characters of the indexed term, regardless of whether it is human-readable. For instance, the indexed form of numeric numbers is not human-readable. Example: terms.raw=true terms.regex No null Restricts matches to terms that match the regular expression. Example: terms.regex=*pedist terms.regex.flag No null Defines a Java regex flag to use when evaluating the regular expression defined with terms.regex. See http://docs.oracle.com/j avase/tutorial/essential/regex/pattern.html for details of each flag. Valid options are: case_insensitive comments multiline literal dotall unicode_case canon_eq unix_lines Example: terms.regex.flag=case_insensitive terms.sort No count Defines how to sort the terms returned. Valid options are count, which sorts by the term frequency, with the highest term frequency first, or index, which sorts in index order. Example: terms.sort=index terms.upper No null Specifies the term to stop at. Although this parameter is not required, either this parameter or terms.limit must be defined. Example: terms.upper=plum Apache Solr Reference Guide 5.1 367 terms.upper.incl No false If set to true, the upper bound term is included in the result set. The default is false. Example: terms.upper.incl=true The output is a list of the terms and their document frequency values. See below for examples. Examples All of the following sample queries work with Solr's "bin/solr -e techproducts" example. Get Top 10 Terms This query requests the first ten terms in the name field: http://localhost:8983/solr/techproducts/t erms?terms.fl=name Results: 0 2 5 3 3 3 3 3 3 3 3 3 Get First 10 Terms Starting with Letter 'a' This query requests the first ten terms in the name field, in index order (instead of the top 10 results by document count): http://localhost:8983/solr/techproducts/terms?terms.fl=name&terms.lower=a&ter ms.sort=index Results: Apache Solr Reference Guide 5.1 368 0 0 1 1 1 1 1 1 1 1 1 1 Using the Terms Component for an Auto-Suggest Feature If the Suggester doesn't suit your needs, you can use the Terms component in Solr to build a similar feature for your own search application. Simply submit a query specifying whatever characters the user has typed so far as a prefix. For example, if the user has typed "at", the search engine's interface would submit the following query: http://localhost:8983/solr/techproducts/terms?terms.fl=name&terms.prefix=at Result: 0 1 1 1 You can use the parameter omitHeader=true to omit the response header from the query response, like in this example, which also returns the response in JSON format: http://localhost:8983/solr/techprodu cts/terms?terms.fl=name&terms.prefix=at&indent=true&wt=json&omitHeader=true Result: Apache Solr Reference Guide 5.1 369 { "terms": { "name": [ "ata", 1, "ati", 1 ] } } Distributed Search Support The TermsComponent also supports distributed indexes. For the /terms request handler, you must provide the following two parameters: Parameter Description shards Specifies the shards in your distributed indexing configuration. For more information about distributed indexing, see Distributed Search with Index Sharding. shards.qt Specifies the request handler Solr uses for requests to shards. More Resources TermsComponent wiki page TermsComponent javadoc The Term Vector Component The TermVectorComponent is a search component designed to return additional information about documents matching your search. For each document in the response, the TermVectorCcomponent can return the term vector, the term frequency, inverse document frequency, position, and offset information. Configuration The TermVectorComponent is not enabled implicitly in Solr - it must be explicitly configured in your solrconfig .xml file. The examples on this page show how it is configured in Solr's "techproducts" example: bin/solr -e techproducts To enable the this component, you need to configure it using a searchComponent element: A request handler must then be configured to use this component name. In this example, the component is associated with a special request handler named /tvrh, that enables term vectors by default using the tv=tru e parameter; but you can associate it with any request handler: Apache Solr Reference Guide 5.1 370 true tvComponent Once your handler is defined, you may use it to fetch term vectors for any fields configured with the termVecto r attribute in your schema.xml, for example: Invoking the Term Vector Component The example below shows an invocation of this component using the above configuration: http://localhost:8983/solr/techproducts/tvrh?q=*%3A*&start=0&rows=10&fl=id,includes Apache Solr Reference Guide 5.1 371 ... id MA147LL/A id 3007WFP id 9885A004 id Request Parameters The example below shows the available request parameters for this component: http://localhost:8983/solr/techproducts/tvrh?q=includes:[* TO *]&rows=10&indent=true&tv=true&tv.tf=true&tv.df=true&tv.positions=true&tv.offsets= true&tv.payloads=true&tv.fl=includes Boolean Parameters tv Description Should the component run or not Apache Solr Reference Guide 5.1 Type boolean 372 tv.docIds Returns term vectors for the specified list of Lucene document IDs (not the Solr Unique Key). comma seperated integers tv.fl Returns term vectors for the specified list of fields. If not specified, the fl para meter is used. comma seperated list of field names tv.all A shortcut that invokes all the boolean parameters listed below. boolean tv.df Returns the Document Frequency (DF) of the term in the collection. This can be computationally expensive. boolean tv.offsets Returns offset information for each term in the document. boolean tv.positions Returns position information. boolean tv.payloads Returns payload information. boolean tv.tf Returns document term frequency info per term in the document. boolean tv.tf_idf Calculates TF*IDF for each term. Requires the parameters tv.tf and tv.df boolean to be "true". This can be computationally expensive. (The results are not shown in example output) To learn more about TermVector component output, see the Wiki page: http://wiki.apache.org/solr/TermVectorC omponentExampleOptions For schema requirements, see the Wiki page: http://wiki.apache.org/solr/FieldOptionsByUseCase SolrJ and the Term Vector Component Neither the SolrQuery class nor the QueryResponse class offer specific method calls to set Term Vector Component parameters or get the "termVectors" output. However, there is a patch for it: SOLR-949. The Stats Component The Stats component returns simple statistics for numeric, string, and date fields within the document set. The sample queries in this section assume you are running the "techproducts" example included with Solr: bin/solr -e techproducts Stats Component Parameters The Stats Component accepts the following parameters: Parameter stats Description If true, then invokes the Stats component. Apache Solr Reference Guide 5.1 373 stats.field Specifies a field for which statistics should be generated. This parameter may be invoked multiple times in a query in order to request statistics on multiple fields. Local Parameters may be used to indicate which subset of the supported statistics should be computed, and/or that statistics should be computed over the results of an arbitrary numeric function (or query) instead of a simple field name. See the examples below. stats.facet Returns sub-results for values within the specified facet. This legacy parameter is not recommended for new users - instead please consider combin ing stats.field with facet.pivot stats.calcdistinct If true, distinct values will be calculated and returned as "countDistinct" and "distinctValues" in the response. This calculation can be very expensive for fields that do not have a tiny cardinality, so it is false by default. This parameter can be specified using per-filed override (ie: f..stats.calcdis tinct=true) but users are encouraged to instead specify it as a Local Parameter - As a top level request parameter, this option is deprecated. Example The query below demonstrates computing stats against two different fields numeric fields, as well as stats over the results of a a 'termfreq()' function call using the 'text' field: http://localhost:8983/solr/techproducts/select?q=*:*&stats=true&stats.field={!func }termfreq('text','memory')&stats.field=price&stats.field=popularity&rows=0&indent= true Apache Solr Reference Guide 5.1 374 0.0 3.0 32 0 10.0 22.0 0.3125 0.7803018439949604 0.0 2199.0 16 16 5251.270030975342 6038619.175900028 328.20437693595886 536.3536996709846 0.0 10.0 15 17 85.0 603.0 5.666666666666667 2.943920288775949 Statistics Supported The table below explains the statistics supported by the Stats component. Not all statistics are supported for all field types, and not all statistics are computed by default (See Local Parameters below for details) Local Param Sample Input Output Name Description Supported Computed Types by Default min true min The minimum value of the field/function in all documents in the set. All Yes max true max The maximum value of the field/function in all documents in the set. All Yes Apache Solr Reference Guide 5.1 375 sum true sum The sum of all values of the field/function in all documents in the set. Numeric & Date Yes count true count The number of values found in all documents in the set for this field/function. All Yes missing true missing The number of documents in the set which do not have a value for this field/function. All Yes sumOfSquares true sumOfSquares Sum of all values squared (a by product of computing stddev) Numeric & Date Yes mean true mean The average (v1 + v2 .... + vN)/N Numeric & Date Yes stddev true stddev Standard deviation, measuring how widely spread the values in the data set are. Numeric & Date Yes percentiles "1,99,99.9" percentiles A list of percentile values based on cut-off points specified by the param value. These values are an approximation, using the t-dige st algorithm. Numeric No calcdistinct true distinctValues The set of all distinct values for the field/function in all of the documents in the set. This calculation can be very expensive for fields that do not have a tiny cardinality. All No calcdistinct true countDistinct The number of distinct values in the field/function in all of the documents in the set. This calculation can be very expensive for fields that do not have a tiny cardinality. All No Local Parameters Similar to the Facet Component, the stats.field parameter supports local parameters for: Tagging & Excluding Filters: stats.field={!ex=filterA}price Changing the Output Key: stats.field={!key=my_price_stats}price Tagging stats for use with facet.pivot: stats.field={!tag=my_pivot_stats}price Local parameters can also be used to specify individual statistics by name, overriding the set of statistics computed by default, eg: stats.field={min=true max=true percentiles='99,99.9,99.99'}price Apache Solr Reference Guide 5.1 376 If any supported statistics are specified via local parameters, then the entire set of default statistics is overridden and only the requested statistics are computed. Additional "Expert" local params are supported in some cases for affecting the behavior of some statistics: tdigestCompression - a positive numeric value defaulting to 100.0 controlling the compression factor of the T-Digest. Larger values means more accuracy, but also uses more memory. Examples Here we compute some statistics for the price field. The min, max, mean, 90th, and 99th percentile price values are computed against all products that are in stock (q=*:* and fq=inStock:true), and independently all of the default statistics are computed against all products regardless of whether they are in stock or not (by excluding that filter). http://localhost:8983/solr/techproducts/select?q=*:*&fq={!tag=stock_check}inStock: true&stats=true&stats.field={!ex=stock_check+key=instock_prices+min=true+max=true+ mean=true+percentiles='90,99'}price&stats.field={!key=all_prices}price&rows=0&inde nt=true 0.0 2199.0 328.20437693595886 564.9700012207031 1966.6484985351556 0.0 2199.0 12 5 4089.880027770996 5385249.921747174 340.823335647583 602.3683083752779 The Stats Component and Faceting Although the stats.facet parameter is no longer recommended, sets of stats.field parameters can be referenced by 'tag' when using Pivot Faceting to compute multiple statistics at every level (i.e.: field) in the tree of pivot constraints. For more information and a detailed example, please see Combining Stats Component With Pivots. Apache Solr Reference Guide 5.1 377 The Query Elevation Component The Query Elevation Component lets you configure the top results for a given query regardless of the normal Lucene scoring. This is sometimes called "sponsored search," "editorial boosting," or "best bets." This component matches the user query text to a configured map of top results. The text can be any string or non-string IDs, as long as it's indexed. Although this component will work with any QueryParser, it makes the most sense to use with DisMax or eDisMax. The Query Elevation Component is supported by distributed searching. All of the sample configuration and queries used in this section assume you are running Solr's " techproducts" example: bin/solr -e techproducts Configuring the Query Elevation Component You can configure the Query Elevation Component in the solrconfig.xml file. string elevate.xml explicit elevator Optionally, in the Query Elevation Component configuration you can also specify the following to distinguish editorial results from "normal" results: foo The Query Elevation Search Component takes the following arguments: Argument Description queryFieldType Specifies which fieldType should be used to analyze the incoming text. For example, it may be appropriate to use a fieldType with a LowerCaseFilter. config-file Path to the file that defines query elevation. This file must exist in /con f/ or /. If the file exists in the /conf/ directory it will be loaded once at startup. If it exists in the data directory, it will be reloaded for each IndexReader. Apache Solr Reference Guide 5.1 378 forceElevation By default, this component respects the requested sort parameter: if the request asks to sort by date, it will order the results by date. If forceElevation=true (the default), results will first return the boosted docs, then order by date. elevate.xml Elevated query results are configured in an external XML file specified in the config-file argument. An elev ate.xml file might look like this: In this example, the query "foo bar" would first return documents 1, 2 and 3, then whatever normally appears for the same query. For the query "ipod", it would first return "MA147LL/A", and would make sure that "IW-02" is not in the result set. Using the Query Elevation Component The enableElevation Parameter For debugging it may be useful to see results with and without the elevated docs. To hide results, use enableEl evation=false: http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&debugQuery=true&ena bleElevation=true http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&debugQuery=true&ena bleElevation=false The forceElevation Parameter You can force elevation during runtime by adding forceElevation=true to the query URL: http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&debugQuery=true&ena bleElevation=true&forceElevation=true The exclusive Parameter You can force Solr to return only the results specified in the elevation file by adding exclusive=true to the URL: http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&debugQuery=true&exc lusive=true Apache Solr Reference Guide 5.1 379 Document Transformers and the markExcludes Parameter The [elevated] Document Transformer can be used to annotate each document with information about whether or not it was elevated: http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&fl=id,[elevated] Likewise, it can be helpful when troubleshooting to see all matching documents – including documents that the elevation configuration would normally exclude. This is possible by using the markExcludes=true parameter, and then using the [excluded] transformer: http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&markExcludes=true&f l=id,[elevated],[excluded] The elevateIds and excludeIds Parameters When the elevation component is in use, the pre-configured list of elevations for a query can be overridden at request time to use the unique keys specified in these request parameters. For example, in the request below documents 3007WFP and 9885A004 will be elevated, and document IW-02 will be excluded -- regardless of what elevations or exclusions are configured for the query "cable" in elevate.xml: http://localhost:8983/solr/techproducts/elevate?q=cable&df=text&excludeIds=IW-02&e levateIds=3007WFP,9885A004 If either one of these parameters is specified at request time, the the entire elevation configuration for the query is ignored. For example, in the request below documents IW-02 and F8V7067-APL-KIT will be elevated, and no documents will be excluded – regardless of what elevations or exclusions are configured for the query "ipod" in elevate.xml: http://localhost:8983/solr/techproducts/elevate?q=ipod&df=text&elevateIds=IW-02,F8 V7067-APL-KIT The fq Parameter Query elevation respects the standard filter query (fq) parameter. That is, if the query contains the fq parameter , all results will be within that filter even if elevate.xml adds other documents to the result set. Response Writers A Response Writer generates the formatted response of a search. Solr supports a variety of Response Writers to ensure that query responses can be parsed by the appropriate language or application. The wt parameter selects the Response Writer to be used. The table below lists the most common settings for the wt parameter. wt Parameter Setting Response Writer Selected csv CSVResponseWriter json JSONResponseWriter php PHPResponseWriter Apache Solr Reference Guide 5.1 380 phps PHPSerializedResponseWriter python PythonResponseWriter ruby RubyResponseWriter velocity VelocityResponseWriter xml XMLResponseWriter xslt XSLTResponseWriter The Standard XML Response Writer The XML Response Writer is the most general purpose and reusable Response Writer currently included with Solr. It is the format used in most discussions and documentation about the response of Solr queries. Note that the XSLT Response Writer can be used to convert the XML produced by this writer to other vocabularies or text-based formats. The behavior of the XML Response Writer can be driven by the following query parameters. The version Parameter The version parameter determines the XML protocol used in the response. Clients are strongly encouraged to always specify the protocol version, so as to ensure that the format of the response they receive does not change unexpectedly if the Solr server is upgraded and a new default format is introduced. Currently supported version values are: XML Version 2.2 Notes The format of the responseHeader changed to use the same structure as the rest of the response. The default value is the latest supported. The stylesheet Parameter The stylesheet parameter can be used to direct Solr to include a declaration in the XML response it returns. The default behavior is not to return any stylesheet declaration at all. Use of the stylesheet parameter is discouraged, as there is currently no way to specify external stylesheets, and no stylesheets are provided in the Solr distributions. This is a legacy parameter, which may be developed further in a future release. The indent Parameter If the indent parameter is used, and has a non-blank value, then Solr will make some attempts at indenting its XML response to make it more readable by humans. Apache Solr Reference Guide 5.1 381 The default behavior is not to indent. The XSLT Response Writer The XSLT Response Writer applies an XML stylesheet to output. It can be used for tasks such as formatting results for an RSS feed. tr Parameter The XSLT Response Writer accepts one parameter: the tr parameter, which identifies the XML transformation to use. The transformation must be found in the Solr conf/xslt directory. The Content-Type of the response is set according to the statement in the XSLT transform, for example: Configuration The example below, from the sample_techproducts_configs config set in the Solr distribution , shows how the XSLT Response Writer is configured. 5 A value of 5 for xsltCacheLifetimeSeconds is good for development, to see XSLT changes quickly. For production you probably want a much higher value. JSON Response Writer A very commonly used Response Writer is the JsonResponseWriter, which formats output in JavaScript Object Notation (JSON), a lightweight data interchange format specified in specified in RFC 4627. Setting the wt parameter to json invokes this Response Writer. The default mime type for the JSON writer is application/json, however this can be overridden in the solr config.xml - such as in this example from the "techproducts" configuration: text/plain Python Response Writer Solr has an optional Python response format that extends its JSON output in the following ways to allow the Apache Solr Reference Guide 5.1 382 response to be safely evaluated by the python interpreter: true and false changed to True and False Python unicode strings are used where needed ASCII output (with unicode escapes) is used for less error-prone interoperability newlines are escaped null changed to None PHP Response Writer and PHP Serialized Response Writer Solr has a PHP response format that outputs an array (as PHP code) which can be evaluated. Setting the wt par ameter to php invokes the PHP Response Writer. Example usage: $code = file_get_contents('http://localhost:8983/solr/techproducts/select?q=iPod&wt=php'); eval("$result = " . $code . ";"); print_r($result); Solr also includes a PHP Serialized Response Writer that formats output in a serialized array. Setting the wt par ameter to phps invokes the PHP Serialized Response Writer. Example usage: $serializedResult = file_get_contents('http://localhost:8983/solr/techproducts/select?q=iPod&wt=phps'); $result = unserialize($serializedResult); print_r($result); Ruby Response Writer Solr has an optional Ruby response format that extends its JSON output in the following ways to allow the response to be safely evaluated by Ruby's interpreter: Ruby's single quoted strings are used to prevent possible string exploits. \ and ' are the only two characters escaped. Unicode escapes are not used. Data is written as raw UTF-8. nil used for null. => is used as the key/value separator in maps. Here is a simple example of how one may query Solr using the Ruby response format: require 'net/http' h = Net::HTTP.new('localhost', 8983) hresp, data = h.get('/solr/techproducts/select?q=iPod&wt=ruby', nil) rsp = eval(data) puts 'number of matches = ' + rsp['response']['numFound'].to_s #print out the name field for each returned document rsp['response']['docs'].each { |doc| puts 'name field = ' + doc['name'\] } CSV Response Writer Apache Solr Reference Guide 5.1 383 The CSV response writer returns a list of documents in comma-separated values (CSV) format. Other information that would normally be included in a response, such as facet information, is excluded. The CSV response writer supports multi-valued fields, as well as psuedo-fields, and the output of this CSV format is compatible with Solr's CSV update format. CSV Parameters These parameters specify the CSV format that will be returned. You can accept the default values or specify your own. Parameter Default Value csv.encapsulator " csv.escape None csv.separator , csv.header Defaults to true. If false, Solr does not print the column headers csv.newline \n csv.null Defaults to a zero length string. Use this parameter when a document has no value for a particular field. Multi-Valued Field CSV Parameters These parameters specify how multi-valued fields are encoded. Per-field overrides for these values can be done using f..csv.separator=|. Parameter Default Value csv.mv.encapsulator None csv.mv.escape \ csv.mv.separator Defaults to the csv.separator value Example http://localhost:8983/solr/techproducts/select?q=ipod&fl=id,cat,name,popularity,pr ice,score&wt=csv returns: id,cat,name,popularity,price,score IW-02,"electronics,connector",iPod & iPod Mini USB 2.0 Cable,1,11.5,0.98867977 F8V7067-APL-KIT,"electronics,connector",Belkin Mobile Power Cord for iPod w/ Dock,1,19.95,0.6523595 MA147LL/A,"electronics,music",Apple 60 GB iPod with Video Playback Black,10,399.0,0.2446348 Velocity Response Writer The VelocityResponseWriter is an optional plugin available in the contrib/velocity directory. It is used to Apache Solr Reference Guide 5.1 384 power the Velocity Search UI in the example configuration. Its JAR and dependencies must be added (via or solr/home lib inclusion), and must be registered in solrc onfig.xml like this: ${velocity.template.base.dir:} velocity-init.properties true false --> The above example shows the optional initialization parameters used by VelocityResponseWriter and detailed in the following table. These initialization parameters are only specified in the writer registration in solrconfig.xml, not as request-time parameters. See further below for request-time parameters. VelocityResponseWriter initialization parameters Parameter Description template.base.dir If specified and exists as a file system directory, a file resource loader will be added for this directory. Templates in this directory will override "solr" resource loader templates. init.properties.file Specifies a properties file name which must exist in the Solr conf/ directory (not under a velocity/ subdirectory) or root of a JAR file in a . params.resource.loader.enabled The "params" resource loader allows templates to be specified in Solr request parameters. For example: http://localhost:8 Default value false 983/solr/gettingstarted/select?q=*:*&wt=velocity &v.template=custom&v.template.custom=CUSTOM%3A% 20%23core_name where v.template=custom says to render a template called "custom" and v.template.custom's value is the actual custom template. This is disabled by default; it'd be a niche, unusual use case to need this enabled. solr.resource.loader.enabled The "solr" resource loader is the only template loader registered by default. Templates are served from resources visible to the SolrResourceLoader under a velocity/ subdirectory. The true VelocityResponseWriter itself has some built-in templates (in its JAR file, under velocity/) that are available automatically through this loader. These built-in templates can be overridden when the same template name is in conf/velocity/ or by using the templat e.base.dir option. Apache Solr Reference Guide 5.1 385 VelocityResponseWriter request parameters Parameter Description v.template Specifies the name of the template to render. v.layout Specifies a template name to use as the layout around the main, v.template, Default value specified template. The main template is rendered into a string value included into the layout rendering as $c ontent. v.layout.enabled Determines if the main template should have a layout wrapped around it. True by default, but requires v.layout to specified as well. true v.contentType Specifies the content type used in the HTTP response. If not specified, the default will depend on whether v.json is specified or not. without json.wrf: text/html;charset=UTF-8 v.json with json.wrf: application/json;c harset=UTF-8 Specifies a function name to wrap around the response rendered as JSON. If specified, the content type used in the response will be "application/json;charset=UTF-8", unless overridden by v.contentType. Output will be in this format (with v.json=wrf): wrf("result":"") Apache Solr Reference Guide 5.1 386 v.locale Locale to use with the $resource tool. The default locale is Locale.ROOT. Localized resources are loaded from standard Java resource bundles named resources[_locale-code].prop erties. Resource bundles can be added by providing a JAR file visible by the SolrResourceLoader with resource bundles under a velocity sub-directory. Resource bundles are not loadable under conf/, as only the class loader aspect of SolrResourceLoader can be used here. v.template. When the "params" resource loader is enabled, templates can be specified as part of the Solr request. VelocityResponseWriter context objects Context reference Description request SolrQueryRequest javadocs response QueryResponse most of the time, but in some cases where QueryResponse doesn't like the request handlers output (AnalysisRequestHandler, for example, causes a ClassCastException parsing "response") the response will be a SolrRespo nseBase object. esc A Velocity EscapeTool instance date A Velocity ComparisonDateTool instance list A Velocity ListTool instance math A Velocity MathTool instance number A Velocity NumberTool instance sort A Velocity SortTool instance display A Velocity DisplayTool instance resource A Velocity ResourceTool instance engine The current VelocityEngine instance page An instance of Solr's PageTool (only included if the response is a QueryResponse where paging makes sense) Apache Solr Reference Guide 5.1 387 debug A shortcut to the debug part of the response, or null if debug is not on. This is handy for having debug-only sections in a template using #if($debug)...#end content The rendered output of the main template, when rendering the layout (v.layout.enabled=true and v.layout=