Apache Solr Ref Guide 4.10

apache-solr-ref-guide-4.10

User Manual:

Open the PDF directly: View PDF .
Page Count: 511

Download
Open PDF In Browser	View PDF

TM

Apache Solr Reference Guide
Covering Apache Solr 4.10

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Apache Lucene, Apache
Solr and their respective logos are trademarks of the Apache Software Foundation. Please see the Apache
Trademark Policy for more information.

Apache Solr Reference Guide
This reference guide describes Apache Solr, the open source solution for search. You can download Apache Solr
from the Solr website at http://lucene.apache.org/solr/.
This Guide contains the following sections:
Getting Started: This section guides you through the
installation and setup of Solr.

Searching: This section presents an overview of the
search process in Solr. It describes the main components
used in searches, including request handlers, query
parsers, and response writers. It lists the query parameters
that can be passed to Solr, and it describes features such
as boosting and faceting, which can be used to fine-tune
search results.

Using the Solr Administration User Interface: This
section introduces the Solr Web-based user interface.
From your browser you can view configuration files,
submit queries, view logfile settings and Java
environment settings, and monitor and control distributed
configurations.
The Well-Configured Solr Instance: This section
discusses performance tuning for Solr. It begins with an
Documents, Fields, and Schema Design: This section
overview of the solrconfig.xml file, then tells you how
describes how Solr organizes its data for indexing. It
to configure cores with solr.xml, how to configure the
explains how a Solr schema defines the fields and field
Lucene index writer, and more.
types which Solr uses to organize data within the
document files it indexes.
Managing Solr: This section discusses important topics for
running and monitoring Solr. It describes running Solr in
Understanding Analyzers, Tokenizers, and Filters:
the Apache Tomcat servlet runner and Web server. Other
This section explains how Solr prepares text for indexing
topics include how to back up a Solr instance, and how to
and searching. Analyzers parse text and produce a
run Solr with Java Management Extensions (JMX).
stream of tokens, lexical units used for indexing and
searching. Tokenizers break field data down into tokens. SolrCloud: This section describes the newest and most
Filters perform other transformational or selective work exciting of Solr's new features, SolrCloud, which provides
on token streams.
comprehensive distributed capabilities.
Indexing and Basic Data Operations: This section
describes the indexing process and basic index
operations, such as commit, optimize, and rollback.

Legacy Scaling and Distribution: This section tells you
how to grow a Solr distribution by dividing a large index
into sections called shards, which are then distributed
across multiple servers, or by replicating a single index
across multiple services.
Client APIs: This section tells you how to access Solr
through various client APIs, including JavaScript, JSON,
and Ruby.

Apache Solr Reference Guide 4.10

2

About This Guide
This guide describes all of the important features and functions of Apache Solr. It is free to download from http://luce
ne.apache.org/solr/.
Designed to provide high-level documentation, this guide is intended to be more encyclopedic and less of a
cookbook. It is structured to address a broad spectrum of needs, ranging from new developers getting started to
well-experienced developers extending their application or troubleshooting. It will be of use at any point in the
application life cycle, for whenever you need authoritative information about Solr.
The material as presented assumes that you are familiar with some basic search concepts and that you can read
XML. It does not assume that you are a Java programmer, although knowledge of Java is helpful when working
directly with Lucene or when developing custom extensions to a Lucene/Solr installation.

Special Inline Notes
Special notes are included throughout these pages.
Note Type
Information

Notes

Tip

Warning

Look & Description
Notes with a blue background are used for information that is important for you to know.

Yellow notes are further clarifications of important points to keep in mind while using Solr.

Notes with a green background are Helpful Tips.

Notes with a red background are warning messages.

Hosts and Port Examples
The default port configured for Solr during the install process is 8983. The samples, URLs and screenshots in this
guide may show different ports, because the port number that Solr uses is configurable. If you have not customized
your installation of Solr, please make sure that you use port 8983 when following the examples, or configure your
own installation to use the port numbers shown in the examples. For information about configuring port numbers
used by Tomcat or Jetty, see Managing Solr.
Similarly, URL examples use 'localhost' throughout; if you are accessing Solr from a location remote to the server
hosting Solr, replace 'localhost' with the proper domain or IP where Solr is running.

Paths
Path information is given relative to solr.home, which is the location under the main Solr installation where Solr's
collections and their conf and data directories are stored. In the default Solr package, solr.home is example/s
olr, which is itself relative to where you unpackaged the application; if you have moved this location for your servlet
container or for another reason, the path to solr.home may be different than the default.

Apache Solr Reference Guide 4.10

3

Getting Started
Solr makes it easy for programmers to develop sophisticated, high-performance search applications with advanced
features such as faceting (arranging search results in columns with numerical counts of key terms). Solr builds on
another open source search technology: Lucene, a Java library that provides indexing and search technology, as
well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities. Both Solr and Lucene are
managed by the Apache Software Foundation (www.apache.org).
The Lucene search library currently ranks among the top 15 open source projects and is one of the top 5 Apache
projects, with installations at over 4,000 companies. Lucene/Solr downloads have grown nearly ten times over the
past three years, with a current run-rate of over 6,000 downloads a day. The Solr search server, which provides
application builders a ready-to-use search platform on top of the Lucene search library, is the fastest growing
Lucene sub-project. Apache Lucene/Solr offers an attractive alternative to the proprietary licensed search and
discovery software vendors.
This section helps you get Solr up and running quickly, and introduces you to the basic Solr architecture and
features. It covers the following topics:
Installing Solr: A walkthrough of the Solr installation process.
Running Solr: An introduction to running Solr. Includes information on starting up the servers, adding documents,
and running queries.
A Quick Overview: A high-level overview of how Solr works.
A Step Closer: An introduction to Solr's home directory and configuration options.

Installing Solr
This section describes how to install Solr. You can install Solr anywhere that a suitable Java Runtime Environment
(JRE) is available, as detailed below. Currently this includes Linux, OS X, and Microsoft Windows. The instructions
in this section should work for any platform, with a few exceptions for Windows as noted.

Got Java?
You will need the Java Runtime Environment (JRE) version 1.7 or higher. At a command line, check your Java
version like this:
$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

The output will vary, but you need to make sure you have version 1.7 or higher. If you don't have the required
version, or if the java command is not found, download and install the latest version from Oracle at http://www.oracle
.com/technetwork/java/javase/downloads/index.html.

Installing Solr
Solr is available from the Solr website at http://lucene.apache.org/solr/.
For Linux/Unix/OSX systems, download the .tgz file. For Microsoft Windows systems, download the .zip file.
Solr runs inside a Java servlet container such as Tomcat, Jetty, or Resin. The Solr distribution includes a working
demonstration server in the Example directory that runs in Jetty. You can use the example server as a template for

Apache Solr Reference Guide 4.10

4

your own installation, whether or not you are using Jetty as your servlet container. For more information about the
demonstration server, see the Solr Tutorial.
Solr ships with a working Jetty server, with optimized settings for Solr, inside the example directory. It is
recommended that you use the provided Jetty server for optimal performance. If you absolutely must use a
different servlet container then continue to the next section on how to install Solr.
To install Solr

1. Unpack the Solr distribution to your desired location.
2. Stop your Java servlet container.
3. Copy the solr.war file from the Solr distribution to the webapps directory of your servlet container. Do not
change the name of this file: it must be named solr.war.
4. Copy the Solr Home directory solr-4.x.0/example/solr/ from the distribution to your desired Solr
Home location.
5. Start your servlet container, passing to it the location of your Solr Home in one of these ways:
Set the Java system property solr.solr.home to your Solr Home. (for example, using the example
jetty setup: java -Dsolr.solr.home=/some/dir -jar start.jar).
Configure the servlet container so that a JNDI lookup of java:comp/env/solr/home by the Solr
webapp will point to your Solr Home.
Start the servlet container in the directory containing ./solr: the default Solr Home is solr under the
JVM's current working directory ($CWD/solr).
To confirm your installation, go to the Solr Admin page at http://localhost:8983/solr/ . Note that your
servlet container may have started on a different port: check the documentation for your servlet container to
troubleshoot that issue. Also note that if that port is already in use, Solr will not start. In that case, shut down the
servlet container running on that port, or change your Solr port.
For more information about installing and running Solr on different Java servlet containers, see the SolrInstall page
on the Solr Wiki.

Related Topics
SolrInstall

Running Solr
This section describes how to run Solr with an example schema, how to add documents, and how to run queries.

Start the Server
If you didn't start Solr after installing it, you can start it by running bin/solr from the Solr directory.
$ bin/solr -f

If you are running Windows, you can start the Web server by running bin\solr.cmd instead.
C:\Applications\Solr\bin\solr.cmd -f

This will start Solr in the foreground, listening on port 8983. The bin/solr and bin\solr.cmd scripts allow you to
customize how you start Solr. Let's work through a few examples of using the bin/solr script (if you're running

Apache Solr Reference Guide 4.10

5

Solr on Windows, the bin\solr.cmd works the same as what is shown in the examples below):
Solr Script Options

The bin/solr script has several options.
Script Help

To see how to use the bin/solr script, execute:
$ bin/solr -help

For specific usage instructions for the start command, do:
$ bin/solr start -help

Start Solr in the Background

As you saw above, the -f flag will start Solr running in the foreground. Since Solr is a server, it is more common to
run it in the background, especially on Unix/Linux. To start Solr running in the background, simply do:
$ bin/solr start

When you start Solr in the background, the script will wait to make sure Solr starts correctly before returning to the
command line prompt.
Start Solr with a Different Port

To change the port Solr listens on, you can use the -p parameter, such as:
$ bin/solr start -p 8984

Stop Solr

When running Solr in the foreground (using -f), then you can stop it using Ctrl-c. However, when running in the
background, you should use the stop command, such as:
$ bin/solr stop

Start Solr with a Specific Example Configuration

Solr also provides a number of useful examples to help you learn about key features. You can launch the examples
using the -e flag. For instance, to launch the Data Import Handler example, you would do:
$ bin/solr -e dih

Currently, the available examples you can run are: default, dih, schemaless, and cloud.
Check if Solr is Running

If you're not sure if Solr is running locally, you can send the info flag ( -i):

Apache Solr Reference Guide 4.10

6

$ bin/solr -i

This will search for running Solr instances on your computer and then gather basic information about them, such as
the version and memory usage.
For more information on starting Solr in cloud mode, see: Getting Started with SolrCloud.
That's it! Solr is running. If you need convincing, use a Web browser to see the Admin Console.
http://localhost:8983/solr/

The Solr Admin interface.
If Solr is not running, your browser will complain that it cannot connect to the server. Check your port number and try
again.

Add Documents
Solr is built to find documents that match queries. Solr's schema provides an idea of how content is structured (more
on the schema later), but without documents there is nothing to find. Solr needs input before it can do anything.
You may want to add a few sample documents before trying to index your own content. The Solr installation comes
with example documents located in the example/exampledocs directory of your installation.
In the exampledocs directory is the SimplePostTool, a Java-based command line tool, post.jar, which can be
used to index the documents. Do not worry too much about the details for now. The Indexing and Basic Data
Operations section has all the details on indexing.
To see some information about the usage of post.jar, use the -help option.
$ java -jar post.jar -help

The SimplePostTool is a simple command line tool for POSTing raw XML to a Solr port. XML data can be read from
files specified as command line arguments, as raw command line arg strings, or via STDIN.
Examples:

Apache Solr Reference Guide 4.10

7

java -Ddata=files -jar post.jar *.xml
java -Ddata=args -jar post.jar '42'
java -Ddata=stdin -jar post.jar < hd.xml

Other options controlled by System Properties include the Solr URL to POST to, and whether a commit should be
executed. These are the defaults for all System Properties:
-Ddata=files
-Durl=http://localhost:8983/solr/update
-Dcommit=yes

Go ahead and add all the documents in the directory as follows:
$ java -Durl=http://localhost:8983/solr/update -jar post.jar *.xml
SimplePostTool: version 1.2
SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, other
encodings are not currently supported
SimplePostTool: POSTing files to http://10.211.55.8:8983/solr/update..
SimplePostTool: POSTing file hd.xml
SimplePostTool: POSTing file ipod_other.xml
SimplePostTool: POSTing file ipod_video.xml
SimplePostTool: POSTing file mem.xml
SimplePostTool: POSTing file monitor.xml
SimplePostTool: POSTing file monitor2.xml
SimplePostTool: POSTing file mp500.xml
SimplePostTool: POSTing file sd500.xml
SimplePostTool: POSTing file solr.xml
SimplePostTool: POSTing file spellchecker.xml
SimplePostTool: POSTing file utf8-example.xml
SimplePostTool: POSTing file vidcard.xml
SimplePostTool: COMMITting Solr index changes..
Time spent: 0:00:00.633
$

That's it! Solr has indexed the documents contained in the files.

Ask Questions
Now that you have indexed documents, you can perform queries. The simplest way is by building a URL that
includes the query parameters. This is exactly the same as building any other HTTP URL.
For example, the following query searches all document fields for "video":
http://localhost:8983/solr/select?q=video
Notice how the URL includes the host name (localhost), the port number where the server is listening (8983), the
application name (solr), the request handler for queries (select), and finally, the query itself (q=video).
The results are contained in an XML document, which you can examine directly by clicking on the link above. The
document contains two parts. The first part is the responseHeader, which contains information about the response
itself. The main part of the reply is in the result tag, which contains one or more doc tags, each of which contains
fields from documents that match the query. You can use standard XML transformation techniques to mold Solr's
results into a form that is suitable for displaying to users. Alternatively, Solr can output the results in JSON, PHP,
Ruby and even user-defined formats.

Apache Solr Reference Guide 4.10

8

Just in case you are not running Solr as you read, the following screen shot shows the result of a query (the next
example, actually) as viewed in Mozilla Firefox. The top-level response contains a lst named responseHeader a
nd a result named response. Inside result, you can see the three docs that represent the search results.

An XML response to a query.
Once you have mastered the basic idea of a query, it is easy to add enhancements to explore the query syntax. This
one is the same as before but the results only contain the ID, name, and price for each returned document. If you
don't specify which fields you want, all of them are returned.
http://localhost:8983/solr/select?q=video&fl=id,name,price
Here is another example which searches for "black" in the name field only. If you do not tell Solr which field to
search, it will search default fields, as specified in the schema.
http://localhost:8983/solr/select?q=name:black
You can provide ranges for fields. The following query finds every document whose price is between $0 and $400.

Apache Solr Reference Guide 4.10

9

http://localhost:8983/solr/select?q=price:[0%20TO%20400]&fl=id,name,price
Faceted browsing is one of Solr's key features. It allows users to narrow search results in ways that are meaningful
to your application. For example, a shopping site could provide facets to narrow search results by manufacturer or
price.
Faceting information is returned as a third part of Solr's query response. To get a taste of this power, take a look at
the following query. It adds facet=true and facet.field=cat.
http://localhost:8983/solr/select?q=price:[0%20TO%20400]&fl=id,name,price&facet=true&
facet.field=cat
In addition to the familiar responseHeader and response from Solr, a facet_counts element is also present.
Here is a view with the responseHeader and response collapsed so you can see the faceting information clearly.

An XML Response with faceting


...



SOLR1000
Solr, the Enterprise Search Server
0.0
...





6
3
2
2
1
1
1
1
1
1
0
0
0
0
0







The facet information shows how many of the query results have each possible value of the cat field. You could
easily use this information to provide users with a quick way to narrow their query results. You can filter results by

Apache Solr Reference Guide 4.10

10

adding one or more filter queries to the Solr request. Here is a request further constraining the request to documents
with a category of "software".
http://localhost:8983/solr/select?q=price:[0%20TO%20400]&fl=id,name,price&facet=true&
facet.field=cat&fq=cat:software

A Quick Overview
Having had some fun with Solr, you will now learn about all the cool things it can do.
Here is a typical configuration:

In the scenario above, Solr runs alongside another application in a Web server. For example, an online store
application would provide a user interface, a shopping cart, and a way to make purchases. The store items would be
kept in some kind of database.
Solr makes it easy to add the capability to search through the online store through the following steps:
1. Define a schema. The schema tells Solr about the contents of documents it will be indexing. In the online
store example, the schema would define fields for the product name, description, price, manufacturer, and so
on. Solr's schema is powerful and flexible and allows you to tailor Solr's behavior to your application. See Doc
uments, Fields, and Schema Design for all the details.
2. Deploy Solr to your application server.
3. Feed Solr the document for which your users will search.
4. Expose search functionality in your application.
Because Solr is based on open standards, it is highly extensible. Solr queries are RESTful, which means, in

Apache Solr Reference Guide 4.10

11

essence, that a query is a simple HTTP request URL and the response is a structured document: mainly XML, but it
could also be JSON, CSV, or some other format. This means that a wide variety of clients will be able to use Solr,
from other web applications to browser clients, rich client applications, and mobile devices. Any platform capable of
HTTP can talk to Solr. See Client APIs for details on client APIs.
Solr is based on the Apache Lucene project, a high-performance, full-featured search engine. Solr offers support for
the simplest keyword searching through to complex queries on multiple fields and faceted search results. Searching
has more information about searching and queries.
If Solr's capabilities are not impressive enough, its ability to handle very high-volume applications should do the trick.
A relatively common scenario is that you have so many queries that the server is unable to respond fast enough to
each one. In this case, you can make copies of an index. This is called replication. Then you can distribute incoming
queries among the copies in any way you see fit. A round-robin mechanism is one simple way to do this.

Another useful technique is sharding. If you have so many documents that you simply cannot fit them all on a single
box for RAM or index size reasons, you can split an index into multiple pieces, called shards. Each shard lives on its
own physical server. An incoming query is sent to all the shard servers, which respond with matching results.

If you have huge numbers of documents and users, you might need to combine the techniques of sharding and
replication. In this case, Solr's new SolrCloud functionality may be more effective for your needs. SolrCloud includes
a number of features to simplify the process of distributing the index and the queries, and manage the resulting
nodes.

Apache Solr Reference Guide 4.10

12

For full details on sharding and replication, see Legacy Scaling and Distribution. We've split the SolrCloud
information into it's own section, called SolrCloud.
Best of all, this talk about high-volume applications is not just hypothetical: some of the famous Internet sites that
use Solr today are Macy's, EBay, and Zappo's.
For more information, take a look at https://wiki.apache.org/solr/PublicServers.

A Step Closer
You already have some idea of Solr's schema. This section describes Solr's home directory and other configuration
options.
When Solr runs in an application server, it needs access to a home directory. The home directory contains important
configuration information and is the place where Solr will store its index.
The crucial parts of the Solr home directory are shown here:
/
solr.xml
conf/
solrconfig.xml
schema.xml
data/

You supply solr.xml, solrconfig.xml, and schema.xml to tell Solr how to behave. By default, Solr stores its
index inside data.
solr.xml specifies configuration options for your Solr core, and also allows you to configure multiple cores. For
more information on solr.xml see The Well-Configured Solr Instance.
solrconfig.xml controls high-level behavior. You can, for example, specify an alternate location for the data
directory. For more information on solrconfig.xml, see The Well-Configured Solr Instance.
schema.xml describes the documents you will ask Solr to index. Inside schema.xml, you define a document as a

Apache Solr Reference Guide 4.10

13

collection of fields. You get to define both the field types and the fields themselves. Field type definitions are
powerful and include information about how Solr processes incoming field values and query values. For more
information on schema.xml, see Documents, Fields, and Schema Design.

Apache Solr Reference Guide 4.10

14

Upgrading Solr
If you are already using Solr 4.9, Solr 4.10 should not present any major problems. However, you should review the
CHANGES.txt file found in your Solr package for changes and updates that may effect your existing
implementation.

Upgrading from 4.9.x
In Solr 3.6, all primitive field types were changed to omit norms by default when the schema version is 1.5 or
greater (SOLR-3140), but TrieDateField's default was mistakenly not changed. As of Solr 4.10, TrieDat
eField omits norms by default (see SOLR-6211).
Creating a SolrCore via CoreContainer.create() no longer requires an additional call to CoreContai
ner.register() to make it available to clients (see SOLR-6170).
CoreContainer.remove() has been removed. You should now use CoreContainer.unload() to
delete a SolrCore (see SOLR-6232).
solr.xml parsing has been improved to better account for the expected data types of various options. As
part of this fix, additional error checking has also been added to provide errors in the event of duplicated
options, or unknown option names that may indicate a typo. Users who have modified their solr.xml in the
past and now upgrade may get errors on startup if they have typos or unexpected options specified in their s
olr.xml file. (See SOLR-5746 for more information.)

Upgrading from Older Versions of Solr
This is a summary of some of the key issues related to upgrading in previous versions of Solr. Users upgrading from
older versions are strongly encouraged to consult CHANGES.txt for the details of all changes since the version they
are upgrading from.
In Solr 4.9, Support for DiskDocValuesFormat (i.e., fieldTypes configured with docValuesFormat="Disk")
was removed due to poor performance. If you have existing fieldTypes using DiskDocValuesFormat please
modify your schema.xml to remove the 'docValuesFormat' attribute, and optimize your index to rewrite it into
the default codec prior to upgrading to 4.9 or later. See LUCENE-5761 for more details.
Begining with Solr 4.8, Java 7 or greater is required. When using Oracle Java 7 or OpenJDK 7, be sure to not
use the GA build 147 or update versions u40, u45 and u51! We recommend using u55 or later. An overview
of known JVM bugs can be found on http://wiki.apache.org/lucene-java/JavaBugs
Prior to Solr 4.8, terms that exceeded Lucene's MAX_TERM_LENGTH were silently ignored when indexing
documents. Begining with Solr 4.8, a document an error will be generated when attempting to index a
document with a term that is too large. If you wish to continue to have large terms ignored, use solr.Lengt
hFilterFactory in all of your Analyzers. See LUCENE-5472 for more details.
The  and  tags in schema.xml was deprecated in Solr 4.8. There is no longer any
reason to keep them in the schema file, they may be safely removed. This allows intermixing of ,  and  definitions if desired. Currently, these tags are supported so either style may
be implemented. They may be deprecated formally in 5.0. See SOLR-5228 for more details.
In Solr 4.7, due to a bug in previous versions the default value of the discountOverlap property of Defaul
tSimilarity was not being set appropriately if you were using the implicit DefaultSimilarityFactory
instead of explicitly configuring it. To preserve consistent behavior for people who upgrade, the implicit
behavior is now contingent on the  -- discountOverlap=false for 4.6 and
below, discountOverlap=true for 4.7 and above. See SOLR-5561 for more information.
In Solr 4.6, The "file" attribute of infoStream in solrconfig.xml was removed. Control this via your logging

Apache Solr Reference Guide 4.10

15

configuration (org.apache.solr.update.LoggingInfoStream) instead.
In Solr 4.5, XML configuration parsing was made more strict about situations where a single setting is allowed
but multiple values are found. Configuration parsing now fails with an error in situations like this. Also, schem
a.xml parsing was also made more strict: "default" or "required" options specified on  de
clarations will cause an init error. You can safely remove these attributes.
In Solr 4.5, CloudSolrServer can now use multiple threads to add documents by default. This is a small
change in runtime semantics when using the bulk add method - you will still end up with the same exception
on a failure, but some documents beyond the one that failed may have made it in. To get the old, single
threaded behavior, set parallel updates to false on the CloudSolrServer instance.
Beginning with 4.4, the use of the Compound File Format is determined by IndexWriter configuration, and not
the Merge Policy. If you have explicitly configured a  with the setUseCompoundFile confi
guration option, you should change this to use the useCompoundFile configuration option directly in the  block. Specifying setUseCompoundFile on the Merge Policy will no longer work in Solr 5.0.
In Solr 4.4, ByteField and ShortField were deprecated, and will be removed in 5.0. Please switch to
using TrieIntField
The pre-4.3.0 solr.xml "legacy" mode and format will no longer be supported in Solr 5.0. Users are
encouraged to migrate from "legacy" to "discovery" solr.xml configurations, see Solr Cores and solr.xml.
As of Solr 4.3 the slf4j/logging jars are no longer included in the Solr webapp to allow for more flexibility in
logging.
Minor changes were made to the Schema API response format in Solr 4.3
In Solr 4.1 the method Solr uses to identify node names for SolrCloud was changed. If you are using
SolrCloud and upgrading from Solr 4.0, you may have issues with unknown or lost nodes. If this occurs, you
can manually set the host parameter either in solr.xml or as a system variable. More information can be
found in the section on SolrCloud.
If you are upgrading from Solr 3.x, you should familiarize yourself with the Major Changes from Solr 3 to Solr
4.

Apache Solr Reference Guide 4.10

16

Using the Solr Administration User Interface
This section discusses the Solr Administration User Interface ("Admin UI").
The Overview of the Solr Admin UI explains how the features of the user interface that are new with Solr 4, what's
on the initial Admin UI page, and how to configure the interface. In addition, there are pages describing each screen
of the Admin UI:
Getting Assistance shows you how to get
Core-Specific Tools is a section explaining additional
more information about the UI.
screens available for each named core.
Analysis - lets you analyze the data found in
Logging explains the various logging levels
specific fields.
available and how to invoke them.
Dataimport - shows you information about the
Cloud Screens display information about
current status of the Data Import Handler.
nodes when running in SolrCloud mode.
Documents - provides a simple form allowing you to
Core Admin explains how to get
execute various Solr indexing commands directly
management information about each core.
from the browser.
Java Properties shows the Java
Files - shows the current core configuration files
information about each core.
such as solrconfig.xml and schema.xml.
Thread Dump lets you see detailed
information about each thread, along with
state information.

Ping - lets you ping a named core and determine
whether the core is active.
Plugins/Stats - shows statistics for plugins and other
installed components.
Query - lets you submit a structured query about
various elements of a core.
Replication - shows you the current replication
status for the core, and lets you enable/disable
replication.
Schema Browser - displays schema data in a
browser window.

Overview of the Solr Admin UI
Solr features a Web interface that makes it easy for Solr administrators and programmers to view Solr configuration
details, run queries and analyze document fields in order to fine-tune a Solr configuration and access online
documentation and other help.

Apache Solr Reference Guide 4.10

17

With Solr 4, the Solr Admin has been completely redesigned. The redesign was completed with these benefits in
mind:
load pages quicker
access and control functionality from the Dashboard
re-use the same servlets that access Solr-related data from an external interface, and
ignore any differences between working with one or multiple cores.
Accessing the URL http://hostname:8983/solr/ (if running Jetty on the default port of 8983), will show the
main dashboard, which is divided into two parts.
A left-side of the screen is a menu under the Solr logo that provides the navigation through the screens of the UI.
The first set of links are for system-level information and configuration and provide access to Logging, Core Admin
and Java Properties, among other things. At the end of this information is a list of Solr cores configured for this
instance. Clicking on a core name shows a secondary menu of information and configuration options for the core
specifically. Items in this list include the Schema, Config, Plugins, and an ability to perform Queries on indexed data.
The center of the screen shows the detail of the option selected. This may include a sub-navigation for the option or
text or graphical representation of the requested data. See the sections in this guide for each screen for more
details.

Configuring the Admin UI in solrconfig.xml
You can configure the Solr Admin UI by editing the file solrconfig.xml.
The  block in the solrconfig.xml file determines the default query to be displayed in the Query section
of the core-specific pages. The default is *:*, which is to find all documents. In this example, we have changed the
default to the term solr.

solr


Related Topics
Configuring solrconfig.xml

Apache Solr Reference Guide 4.10

18

Getting Assistance
At the bottom of each screen of the Admin UI is a set of links that can be used to get more assistance with
configuring and using Solr.

Assistance icons
These icons include the following links.
Link

Description

Documentation

Navigates to the Apache Solr documentation hosted on http://lucene.apache.org/solr/.

Issue Tracker

Navigates to the JIRA issue tracking server for the Apache Solr project. This server resides at ht
tp://issues.apache.org/jira/browse/SOLR.

IRC Channel

Connects you to the web interface for Solr's IRC channel. This channel is found on irc.freen
ode.net, Port 7000, #solr channel.

Community
forum

Connects you to the Solr community forum, which at the current time is a set of mailing lists and
their archives.

Solr Query
Syntax

Navigates to the Apache Wiki page describing the Solr query syntax: http://wiki.apache.org/solr/
SolrQuerySyntax.

These links cannot be modified without editing the admin.html in the solr.war that contains the Admin UI files.

Logging
The Logging page shows messages from Solr's log files.
When you click the link for "Logging", a page similar to the one below will be displayed:

The Main Logging Screen
While this example shows logged messages for only one core, if you have multiple cores in a single instance, they
will each be listed, with the level for each.

Apache Solr Reference Guide 4.10

19

Selecting a Logging Level

When you select the Level link on the left, you see the hierarchy of classpaths and classnames for your instance. A
row highlighted in yellow indicates that the class has logging capabilities. Click on a highlighted row, and a menu will
appear to allow you to change the log level for that class. Characters in boldface indicate that the class will not be
affected by level changes to root.
For an explanation of the various logging levels, see Configuring Logging.

Cloud Screens
When running in SolrCloud mode, an option will appear in the Admin UI between Logging and Core Admin for
Cloud. It's not possible at the current time to manage the nodes of the SolrCloud cluster, but you can view them and
open the Solr Admin UI on each node to view the status and statistics for the node and each core on each node.
Click on the Cloud option in the left-hand navigation, and a small sub-menu appears with options called "Tree",
"Graph", "Graph (Radial)" and "Dump". The default view (which is "Tree") shows a graph of each core and the
addresses of each node. This example shows a very simple two-node cluster with a single core:

The "Graph (Radial)" option provides a different visual view of each node. Using the same simple two-node cluster,

Apache Solr Reference Guide 4.10

20

the radial graph view looks like:

The "Tree" option shows a directory structure of the files in ZooKeeper, including clusterstate.json,
configuration files, and other status and information files. In this example, we show the leader definition files for the
core named "collection1":

The final option is "Dump", which allows you to download an XML file with all the ZooKeeper configuration files.

Core Admin
The Core Admin screen lets you manage your cores.
The buttons at the top of the screen let you add a new core, unload the core displayed, rename the currently
displayed core, swap the existing core with one that you specify in a drop-down box, reload the current core, and
optimize the current core.
The main display and available actions correspond to the commands used with the CoreAdminHandler, but provide
another way of working with your cores.

Apache Solr Reference Guide 4.10

21

Java Properties
The Java Properties screen provides easy access to one of the most essential components of a top-performing Solr
systems With the Java Properties screen, you can see all the properties of the JVM running Solr, including the class
paths, file encodings, JVM memory settings, operating system, and more.

Thread Dump
The Thread Dump screen lets you inspect the currently active threads on your server. Each thread is listed and
access to the stacktraces is available where applicable. Icons to the left indicate the state of the thread: for example,
threads with a green check-mark in a green circle are in a "RUNNABLE" state. On the right of the thread name, a
down-arrow means you can expand to see the stacktrace for that thread.

Apache Solr Reference Guide 4.10

22

When you move your cursor over a thread name, a box floats over the name with the state for that thread. Thread
states can be:
State

Meaning

NEW

A thread that has not yet started.

RUNNABLE

A thread executing in the Java virtual machine.

BLOCKED

A thread that is blocked waiting for a monitor lock.

WAITING

A thread that is waiting indefinitely for another thread to perform a particular action.

TIMED_WAITING

A thread that is waiting for another thread to perform an action for up to a specified waiting
time.

TERMINATED

A thread that has exited.

When you click on one of the threads that can be expanded, you'll see the stacktrace, as in the example below:

Apache Solr Reference Guide 4.10

23

Inspecting a thread
You can also check the Show all Stacktraces button to automatically enable expansion for all threads.

Core-Specific Tools
In the left-hand navigation bar, you will see a pull-down menu titled "Core Selector". Clicking on the menu will show
a list of Solr cores, with a search box that can be used to find a specific core (handy if you have a lot of cores).
When you select a core, such as collection1 in the example, a secondary menu opens under the core name with
the administration options available for that particular core.

After selecting the core, the central part of the screen shows Statistics and other information about the core you
chose. You can define a file called admin-extra.html that includes links or other information you would like to
display in the "Admin Extra" part of this main screen.
On the left side, under the core name, are links to other screens that display information or provide options for the
specific core chosen. The core-specific options are listed below, with a link to the section of this Guide to find out

Apache Solr Reference Guide 4.10

24

more:
Analysis - lets you analyze the data found in specific fields.
Dataimport - shows you information about the current status of the Data Import Handler.
Documents - provides a simple form allowing you to execute various Solr indexing commands directly from
the browser.
Files - shows the current core configuration files such as solrconfig.xml and schema.xml.
Ping - lets you ping a named core and determine whether the core is active.
Plugins/Stats - shows statistics for plugins and other installed components.
Query - lets you submit a structured query about various elements of a core.
Replication - shows you the current replication status for the core, and lets you enable/disable replication.
Schema Browser - displays schema data in a browser window.

Analysis Screen
The Analysis screen lets you inspect how data will be handled according to the field, field type and dynamic rule
configurations found in schema.xml. You can analyze how content would be handled during indexing or during
query processing and view the results separately or at the same time. Ideally, you would want content to be handled
consistently, and this screen allows you to validate the settings in the field type or field analysis chains.
Enter content in one or both boxes at the top of the screen, and then choose the field or field type definitions to use
for analysis.
The standard output (shown if "Verbose Output" is not checked) will display the step of the analysis and the output
based on the current settings. If you click the Verbose Output check box, you see more information, including
transformations to the input (such as, convert to lower case, strip extra characters, etc.) and the bytes, type and
detailed position information. The information displayed will vary depending on the settings of the field or field type.
Each step of the process is displayed in a separate section, with an abbreviation for the tokenizer or filter that is
applied in that step. Hover or click on the abbreviation, and you'll see the name and path of the tokenizer or filter.

In example screenshot above, several transformations are applied to the text string "Running is a sport." We've used
the field "text", which has rules that remove the "is" and "a" and the word "running" has been changed to its basic
form, "run". This is because we have defined the field type, text_en in this scenario, to remove stop words (small
words that usually do not provide a great deal of context) and "stem" terms when possible to find more possible

Apache Solr Reference Guide 4.10

25

matches (this is particularly helpful with plural forms of words). If you click the question mark next to the Analyze
Fieldname/Field Type pull-down menu, the Schema Browser window will open, showing you the settings for the
field specified.

The section Understanding Analyzers, Tokenizers, and Filters describes in detail what each option is and how it may
transform your data and the section Running Your Analyzer has specific examples for using the Analysis screen.

Dataimport Screen
The Dataimport screen shows the configuration of the DataImportHandler (DIH) and allows you to start indexing
data, as defined by the options selected on the screen and defined in the configuration file.

The configuration file defines the location of the data and how to perform the SQL queries for the data you want. The
options on the screen control how the data is imported to Solr. For more information about data importing with DIH,
see the section on Uploading Structured Data Store Data with the Data Import Handler.

Documents Screen
The Documents screen provides a simple form allowing you to execute various Solr indexing commands in a variety
of formats directly from the browser.

Apache Solr Reference Guide 4.10

26

The screen allows you to:
Copy documents in JSON, CSV or XML and submit them to the index
Upload documents (in JSON, CSV or XML)
Construct documents by selecting fields and field values
The first step is to define the RequestHandler to use (aka, 'qt'). By default /update will be defined. To use Solr Cell,
for example, change the request handler to /update/extract.
Then choose the Document Type to define the type of document to load. The remaining parameters will change
depending on the document type selected.
JSON

When using the JSON document type, the functionality is similar to using a requestHandler on the command line.
Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The
document structure should still be in proper JSON format.
Then you can choose when documents should be added to the index (Commit Within), whether existing documents
should be overwritten with incoming documents with the same id (if this is not true, then the incoming documents
will be dropped), and, finally, if a document boost should be applied.
This option will only add or overwrite documents to the index; for other update tasks, see the #Solr Command option
.
CSV

When using the CSV document type, the functionality is similar to using a requestHandler on the command line.
Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The
document structure should still be in proper CSV format, with columns delimited and one row per document.
Then you can choose when documents should be added to the index (Commit Within), and whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped).

Apache Solr Reference Guide 4.10

27

Document Builder

The Document Builder provides a wizard-like interface to enter fields of a document
File Upload

The File Upload option allows choosing a prepared file and uploading it. If using only /update for the
Request-Handler option, you will be limited to XML, CSV, and JSON.
However, to use the ExtractingRequestHandler (aka Solr Cell), you can modify the Request-Handler to /update/e
xtract. You must have this defined in your solrconfig.xml file, with your desired defaults. You should also
update the &literal.id shown in the Extracting Req. Handler Params so the file chosen is given a unique id.
Then you can choose when documents should be added to the index (Commit Within), and whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped).
Solr Command

The Solr Command option allows you use XML or JSON to perform specific actions on documents, such as defining
documents to be added or deleted, updating only certain fields of documents, or commit and optimize commands on
the index.
The documents should be structured as they would be if using /update on the command line.
XML

When using the XML document type, the functionality is similar to using a requestHandler on the command line.
Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The
document structure should still be in proper Solr XML format, with each document separated by  tags and
each field defined.
Then you can choose when documents should be added to the index (Commit Within), and whether existing
documents should be overwritten with incoming documents with the same id (if this is not true, then the incoming
documents will be dropped).
This option will only add or overwrite documents to the index; for other update tasks, see the #Solr Command option
.
Related Topics

Uploading Data with Index Handlers
Uploading Data with Solr Cell using Apache Tika

Files Screen

The Files screen lets you browse & view the various configuration files (such solrconfig.xml and schema.xml)
for the core you selected.

Apache Solr Reference Guide 4.10

28

While the solrconfig.xml defines the behaviour of Solr as it indexes content and responds to queries, the schem
a.xml allows you to define the types of data in your content (field types), the fields your documents will be broken
into, and any dynamic fields that should be generated based on patterns of field names in the incoming documents.
Any other configuration files are used depending on how they are referenced in either solrconfig.xml or schema
.xml.
Configuration files cannot be edited with this screen, so a text editor of some kind must be used.
This screen is related to the Schema Browser Screen, in that they both can display information from the schema, but
the Schema Browser provides a way to drill into the analysis chain and displays linkages between field types, fields,
and dynamic field rules.
Many of the options defined in solrconfig.xml and schema.xml are described throughout the rest of this Guide.
In particular, you will want to review these sections:
Indexing and Basic Data Operations
Searching
The Well-Configured Solr Instance
Documents, Fields, and Schema Design

Ping
Choosing Ping under a core name issues a ping request to check whether a server is up.
Ping is configured using a requestHandler in the solrconfig.xml file:

Apache Solr Reference Guide 4.10

29




solrpingquery


all





The Ping option doesn't open a page, but the status of the request can be seen on the core overview page shown
when clicking on a collection name. The length of time the request has taken is displayed next to the Ping option, in
milliseconds.

Plugins & Stats Screen
The Plugins screen shows information and statistics about Solr's status and performance. You can find information
about the performance of Solr's caches, the state of Solr's searchers, and the configuration of searchHandlers and
requestHandlers.
Choose an area of interest on the right, and then drill down into more specifics by clicking on one of the names that
appear in the central part of the window. In this example, we've chosen to look at the Searcher stats, from the Core
area:

Searcher Statistics
The display is a snapshot taken when the page is loaded. You can get updated status by choosing to either Watch
Changes or Refresh Values. Watching the changes will highlight those areas that have changed, while refreshing
the values will reload the page with updated information.

Query Screen

Apache Solr Reference Guide 4.10

30

You can use Query, shown under the name of each core, to submit a search query to a Solr server and analyze the
results. In the example in the screenshot, a query has been submitted, and the screen shows the query results sent
to the browser as JSON.

The query was sent to a core named "collection1". We used Solr's default query for this screen (as defined in solrc
onfig.xml), which is *:*. This query will find all records in the index for this core. We kept the other defaults, but
the table below explains these options, which are also covered in detail in later parts of this Guide.
The response is shown to the right of the form. Requests to Solr are simply HTTP requests, and the query submitted
is shown in light type above the results; if you click on this it will open a new browser window with just this request
and response (without the rest of the Solr Admin UI). The rest of the response is shown in JSON, which is part of the
request (see the wt=json part at the end).
The response has at least two sections, but may have several more depending on the options chosen. The two
sections it always has are the responseHeader and the response. The responseHeader includes the status of
the search (status), the processing time (QTime), and the parameters (params) that were used to process the
query.
The response includes the documents that matched the query, in doc sub-sections. The fields return depend on
the parameters of the query (and the defaults of the request handler used). The number of results is also included in
this section.
This screen allows you to experiment with different query options, and inspect how your documents were indexed.
The query parameters available on the form are some basic options that most users want to have available, but
there are dozens more available which could be simply added to the basic request by hand (if opened in a browser).
The table below explains the parameters available:

Apache Solr Reference Guide 4.10

31

Field

Description

Request-handler
(qt)

Specifies the query handler for the request. If a query handler is not specified, Solr processes
the response with the standard query handler.

q

The query event. See Searching for an explanation of this parameter.

fq

The filter queries. See Common Query Parameters for more information on this parameter.

sort

Sorts the response to a query in either ascending or descending order based on the
response's score or another specified characteristic.

start, rows

start is the offset into the query result starting at which documents should be returned. The
default value is 0, meaning that the query should return results starting with the first document
that matches. This field accepts the same syntax as the start query parameter, which is
described in Searching. rows is the number of rows to return.

fl

Defines the fields to return for each document. You can explicitly list the stored fields you want
to have returned by separating them with either a comma or a space. In Solr 4, the results of
functions can also be included in the fl list.

wt

Specifies the Response Writer to be used to format the query response. Defaults to XML if not
specified.

indent

Click this button to request that the Response Writer use indentation to make the responses
more readable.

debugQuery

Click this button to augment the query response with debugging information, including "explain
info" for each document returned. This debugging information is intended to be intelligible to
the administrator or programmer.

dismax

Click this button to enable the Dismax query parser. See The DisMax Query Parser for further
information.

edismax

Click this button to enable the Extended query parser. See The Extended DisMax Query
Parser for further information.

hl

Click this button to enable highlighting in the query response. See Highlighting for more
information.

facet

Enables faceting, the arrangement of search results into categories based on indexed terms.
See Faceting for more information.

spatial

Click to enable using location data for use in spatial or geospatial searches. See Spatial
Search for more information.

spellcheck

Click this button to enable the Spellchecker, which provides inline query suggestions based on
other, similar, terms. See Spell Checking for more information.

Related Topics

Searching

Replication Screen
The Replication screen shows you the current replication state for the named core you have specified. In Solr,

Apache Solr Reference Guide 4.10

32

replication is for the index only. SolrCloud has supplanted much of this functionality, but if you are still using index
replication, you can use this screen to see the replication state:

In this example, replication is enabled and will be done after each commit. Because this server is the Master, it is
showing only the config settings for the master. On the master, you can disable replication by clicking the Disable
Replication button.
In Solr, the replication is initiated by the slave servers so there is more value by looking at the Replication screen on
the slave nodes. This screenshot shows the Replication screen for a slave:

You can click the Refresh Status button to show the most current replication status, or choose to get a new
snapshot from the master server.
More details on how to configure replication is available in the section called Index Replication.

Schema Browser Screen
The Schema Browser screen lets you see schema data in a browser window. If you have accessed this window
from the Analysis screen, it will be opened to a specific field, dynamic field rule or field type. If there is nothing
chosen, use the pull-down menu to choose the field or field type.

Apache Solr Reference Guide 4.10

33

The screen provides a great deal of useful information about each particular field. In the example above, we have
chosen the text field. On the right side of the center window, we see the field name, and a list of fields that
populate this field because they are defined to be copied to the text field. Click on one of those field names, and
you can see the definitions for that field. We can also see the field type, which would allow us to inspect the type
definitions as well.
In the left part of the center window, we see the field type again, and the defined properties for the field. We can also
see how many documents have populated this field. Then we see the analyzer used for indexing and query
processing. Click the icon to the left of either of those, and you'll see the definitions for the tokenizers and/or filters
that are used. The output of these processes is the information you see when testing how content is handled for a
particular field with the Analysis Screen.
Under the analyzer information is a button to Load Term Info. Clicking that button will show the top N terms that are
in the index for that field. Click on a term, and you will be taken to the Query Screen to see the results of a query of
that term in that field. If you want to always see the term information for a field, choose Autoload and it will always
appear when there are terms for a field. A histogram shows the number of terms with a given frequency in the field.

Apache Solr Reference Guide 4.10

34

Documents, Fields, and Schema Design
This section discusses how Solr organizes its data into documents and fields, as well as how to work with the Solr
schema file, schema.xml. It includes the following topics:
Overview of Documents, Fields, and Schema Design: An introduction to the concepts covered in this section.
Solr Field Types: Detailed information about field types in Solr, including the field types in the default Solr schema.
Defining Fields: Describes how to define fields in Solr.
Copying Fields: Describes how to populate fields with data copied from another field.
Dynamic Fields: Information about using dynamic fields in order to catch and index fields that do not exactly conform
to other field definitions in your schema.
Schema API: Use curl commands to read various parts of a schema or create new fields and copyField rules.
Other Schema Elements: Describes other important elements in the Solr schema: Unique Key, Default Search Field,
and the Query Parser Operator.
Putting the Pieces Together: A higher-level view of the Solr schema and how its elements work together.
DocValues: Describes how to create a docValues index for faster lookups.
Schemaless Mode: Automatically add previously unknown schema fields using value-based field type guessing.

Overview of Documents, Fields, and Schema Design
The fundamental premise of Solr is simple. You give it a lot of information, then later you can ask it questions and
find the piece of information you want. The part where you feed in all the information is called indexing or updating.
When you ask a question, it's called a query.
One way to understand how Solr works is to think of a loose-leaf book of recipes. Every time you add a recipe to the
book, you update the index at the back. You list each ingredient and the page number of the recipe you just added.
Suppose you add one hundred recipes. Using the index, you can very quickly find all the recipes that use garbanzo
beans, or artichokes, or coffee, as an ingredient. Using the index is much faster than looking through each recipe
one by one. Imagine a book of one thousand recipes, or one million.
Solr allows you to build an index with many different fields, or types of entries. The example above shows how to
build an index with just one field, ingredients. You could have other fields in the index for the recipe's cooking
style, like Asian, Cajun, or vegan, and you could have an index field for preparation times. Solr can answer
questions like "What Cajun-style recipes that have blood oranges as an ingredient can be prepared in fewer than 30
minutes?"
The schema is the place where you tell Solr how it should build indexes from input documents.

How Solr Sees the World
Solr's basic unit of information is a document, which is a set of data that describes something. A recipe document
would contain the ingredients, the instructions, the preparation time, the cooking time, the tools needed, and so on.
A document about a person, for example, might contain the person's name, biography, favorite color, and shoe size.
A document about a book could contain the title, author, year of publication, number of pages, and so on.
In the Solr universe, documents are composed of fields, which are more specific pieces of information. Shoe size
could be a field. First name and last name could be fields.

Apache Solr Reference Guide 4.10

35

Fields can contain different kinds of data. A name field, for example, is text (character data). A shoe size field might
be a floating point number so that it could contain values like 6 and 9.5. Obviously, the definition of fields is flexible
(you could define a shoe size field as a text field rather than a floating point number, for example), but if you define
your fields correctly, Solr will be able to interpret them correctly and your users will get better results when they
perform a query.
You can tell Solr about the kind of data a field contains by specifying its field type. The field type tells Solr how to
interpret the field and how it can be queried.
When you add a document, Solr takes the information in the document's fields and adds that information to an
index. When you perform a query, Solr can quickly consult the index and return the matching documents.

Field Analysis
Field analysis tells Solr what to do with incoming data when building an index. A more accurate name for this
process would be processing or even digestion, but the official name is analysis.
Consider, for example, a biography field in a person document. Every word of the biography must be indexed so that
you can quickly find people whose lives have had anything to do with ketchup, or dragonflies, or cryptography.
However, a biography will likely contains lots of words you don't care about and don't want clogging up your
index—words like "the", "a", "to", and so forth. Furthermore, suppose the biography contains the word "Ketchup",
capitalized at the beginning of a sentence. If a user makes a query for "ketchup", you want Solr to tell you about the
person even though the biography contains the capitalized word.
The solution to both these problems is field analysis. For the biography field, you can tell Solr how to break apart the
biography into words. You can tell Solr that you want to make all the words lower case, and you can tell Solr to
remove accents marks.
Field analysis is an important part of a field type. Understanding Analyzers, Tokenizers, and Filters is a detailed
description of field analysis.

Solr Field Types
The field type defines how Solr should interpret data in a field and how the field can be queried. There are many
field types included with Solr by default, and they can also be defined locally.
Topics covered in this section:
Field Type Definitions and Properties
Field Types Included with Solr
Working with Currencies and Exchange Rates
Working with Dates
Working with Enum Fields
Working with External Files and Processes
Field Properties by Use Case

Related Topics
SchemaXML-DataTypes
FieldType Javadoc

Apache Solr Reference Guide 4.10

36

Field Type Definitions and Properties
A field type definition can include four types of information:
The name of the field type (mandatory)
An implementation class name (mandatory)
If the field type is TextField, a description of the field analysis for the field type
Field type properties - depending on the implementation class, some properties may be mandatory.
Field Type Definitions in schema.xml

Field types are defined in schema.xml, with the types element. Each field type is defined between fieldType el
ements. Here is an example of a field type definition for a type called text_general:















The first line in the example above contains the field type name, text_general, and the name of the implementing
class, solr.TextField. The rest of the definition is about field analysis, described in Understanding Analyzers,
Tokenizers, and Filters.
The implementing class is responsible for making sure the field is handled correctly. In the class names in schema.
xml, the string solr is shorthand for org.apache.solr.schema or org.apache.solr.analysis. Therefore,
solr.TextField is really org.apache.solr.schema.TextField..
Field Type Properties

The field type class determines most of the behavior of a field type, but optional properties can also be defined.
For example, the following definition of a date field type defines two properties, sortMissingLast and omitNorm
s.


The properties that can be specified for a given field type fall into three major categories:
Properties specific to the field type's class.

Apache Solr Reference Guide 4.10

37

General Properties Solr supports for any field type.
Field Default Properties that can be specified on the field type that will be inherited by fields that use this type
instead of the default behavior.
General Properties

Property

Description

Values

name

The name of the fieldType. This value gets used in field definitions, in
the "type" attribute. It is strongly recommended that names consist of
alphanumeric or underscore characters only and not start with a digit.
This is not currently strictly enforced.

class

The class name that gets used to store and index the data for this type.
Note that you may prefix included class names with "solr." and Solr will
automatically figure out which packages to search for the class - so
"solr.TextField" will work. If you are using a third-party class, you will
probably need to have a fully qualified class name. The fully qualified
equivalent for "solr.TextField" is "org.apache.solr.schema.TextField".

positionIncrementGap

For multivalued fields, specifies a distance between multiple values,
which prevents spurious phrase matches

integer

autoGeneratePhraseQueries

For text fields. If true, Solr automatically generates phrase queries for
adjacent terms. If false, terms must be enclosed in double-quotes to be
treated as phrases.

true or
false

docValuesFormat

Defines a custom DocValuesFormat to use for fields of this type. This

n/a

requires that a schema-aware codec, such as the SchemaCodecFacto
ry has been configured in solrconfig.xml.
postingsFormat

Defines a custom PostingsFormat to use for fields of this type. This

n/a

requires that a schema-aware codec, such as the SchemaCodecFacto
ry has been configured in solrconfig.xml.

Lucene index back-compatibility is only supported for the default codec. If you choose to customize the pos
tingsFormat or docValuesFormat in your schema.xml, upgrading to a future version of Solr may
require you to either switch back to the default codec and optimize your index to rewrite it into the default
codec before upgrading, or re-build your entire index from scratch after upgrading.
Field Default Properties

Property

Description

Values

indexed

If true, the value of the field can be used in queries to retrieve matching
documents

true or
false

stored

If true, the actual value of the field can be retrieved by queries

true or
false

Apache Solr Reference Guide 4.10

38

docValues

If true, the value of the field will be put in a column-oriented DocValues str
ucture

true or
false

sortMissingFirst
sortMissingLast

Control the placement of documents when a sort field is not present. As of
Solr 3.5, these work for all numeric fields, including Trie and date fields.

true or
false

multiValued

If true, indicates that a single document might contain multiple values for
this field type

true or
false

omitNorms

If true, omits the norms associated with this field (this disables length
normalization and index-time boosting for the field, and saves some
memory). Defaults to true for all primitive (non-analyzed) field types, such
as int, float, data, bool, and string. Only full-text fields or fields that need
an index-time boost need norms.

true or
false

omitTermFreqAndPositions

If true, omits term frequency, positions, and payloads from postings for
this field. This can be a performance boost for fields that don't require that
information. It also reduces the storage space required for the index.
Queries that rely on position that are issued on a field with this option will
silently fail to find documents. This property defaults to true for all fields
that are not text fields.

true or
false

omitPositions

Similar to omitTermFreqAndPositions but preserves term frequency

true or
false

information
termVectors
termPositions
termOffsets

These options instruct Solr to maintain full term vectors for each
document, optionally including the position and offset information for each
term occurrence in those vectors. These can be used to accelerate
highlighting and other ancillary functionality, but impose a substantial cost
in terms of index size. They are not necessary for typical uses of Solr

true or
false

required

Instructs Solr to reject any attempts to add a document which does not
have a value for this field. This property defaults to false.

true or
false

Field Types Included with Solr
The following table lists the field types that are available in Solr. The org.apache.solr.schema package
includes all the classes listed in this table.
Class

Description

BCDIntField

Binary-coded decimal (BCD) integer. BCD is a relatively inefficient
encoding that offers the benefits of quick decimal calculations and quick
conversion to a string. This field has been deprecated and will be
removed in Solr 5.0, use TrieIntField instead.

BCDLongField

Binary-coded decimal long integer. This field has been deprecated and
will be removed in Solr 5.0, use TrieLongField instead.

BCDStrField

Binary-coded decimal string. This field has been deprecated and will be
removed in Solr 5.0, use TrieIntField instead.

BinaryField

Binary data.

Apache Solr Reference Guide 4.10

39

BoolField

Contains either true or false. Values of "1", "t", or "T" in the first character
are interpreted as true. Any other values in the first character are
interpreted as false.

ByteField

Contains a byte (an 8-bit signed integer). This field has been deprecated
and will be removed in Solr 5.0, use TrieIntField instead.

CollationField

Supports Unicode collation for sorting and range queries.
ICUCollationField is a better choice if you can use ICU4J. See the section
Unicode Collation.

CurrencyField

Supports currencies and exchange rates. See the section Working with
Currencies and Exchange Rates.

DateField

Represents a point in time with millisecond precision. See the section Wor
king with Dates. This field has been deprecated and will be removed in
Solr 5.0, use TrieDateField instead.

DoubleField

Double (64-bit IEEE floating point). This field has been deprecated and
will be removed in Solr 5.0, use TrieDoubleField instead.

ExternalFileField

Pulls values from a file on disk. See the section Working with External
Files and Processes.

EnumField

Allows defining an enumerated set of values which may not be easily
sorted by either alphabetic or numeric order (such as a list of severities,
for example). This field type takes a configuration file, which lists the
proper order of the field values. See the section Working with Enum
Fields for more information.

FloatField

Floating point (32-bit IEEE floating point). This field has been deprecated
and will be removed in Solr 5.0, use TrieFloatField instead.

ICUCollationField

Supports Unicode collation for sorting and range queries. See the section
Unicode Collation.

IntField

Integer (32-bit signed integer). This field has been deprecated and will be
removed in Solr 5.0, use TrieIntField instead.

LatLonType

Spatial Search: a latitude/longitude coordinate pair. The latitude is
specified first in the pair.

LongField

Long integer (64-bit signed integer). This field has been deprecated and
will be removed in Solr 5.0, use TrieLongField instead.

PointType

Spatial Search: An arbitrary n-dimensional point, useful for searching
sources such as blueprints or CAD drawings.

Apache Solr Reference Guide 4.10

40

PreAnalyzedField

Provides a way to send to Solr serialized token streams, optionally with
independent stored values of a field, and have this information stored and
indexed without any additional text processing. Useful if you want to
submit field content that was already processed by some existing external
text processing pipeline (e.g. tokenized, annotated, stemmed, inserted
synonyms, etc.), while using all the rich attributes that Lucene's TokenSt
ream provides via token attributes.

RandomSortField

Does not contain a value. Queries that sort on this field type will return
results in random order. Use a dynamic field to use this feature.

ShortField

Short integer. This field has been deprecated and will be removed in Solr
5.0, use TrieIntField instead.

SortableDoubleField

The Sortable fields provide correct numeric sorting. This field has been
deprecated and will be removed in Solr 5.0, use TrieDoubleField instead.

SortableFloatField

Numerically sorted floating point. This field has been deprecated and will
be removed in Solr 5.0, use TrieFloatField instead.

SortableIntField

Numerically sorted integer. This field has been deprecated and will be
removed in Solr 5.0, use TrieIntField instead.

SortableLongField

Numerically sorted long integer. This field has been deprecated and will
be removed in Solr 5.0, use TrieLongField instead.

SpatialRecursivePrefixTreeFieldType

(RPT for short) Spatial Search: Accepts latitude comma longitude strings
or other shapes in WKT format.

StrField

String (UTF-8 encoded string or Unicode).

TextField

Text, usually multiple words or tokens.

TrieDateField

Date field. Represents a point in time with millisecond precision. See the
section Working with Dates. precisionStep="0" enables efficient date
sorting and minimizes index size; precisionStep="8" (the default)
enables efficient range queries.

TrieDoubleField

Double field (64-bit IEEE floating point). precisionStep="0" enables
efficient numeric sorting and minimizes index size; precisionStep="8"
(the default) enables efficient range queries.

TrieField

If this field type is used, a "type" attribute must also be specified, valid
values are: integer, long, float, double, date. Using this field is the
same as using any of the Trie fields. precisionStep="0" enables
efficient numeric sorting and minimizes index size; precisionStep="8"
(the default) enables efficient range queries.

TrieFloatField

Floating point field (32-bit IEEE floating point). precisionStep="0" en
ables efficient numeric sorting and minimizes index size; precisionSte
p="8" (the default) enables efficient range queries.

Apache Solr Reference Guide 4.10

41

TrieIntField

Integer field (32-bit signed integer). precisionStep="0" enables
efficient numeric sorting and minimizes index size; precisionStep="8"
(the default) enables efficient range queries.

TrieLongField

Long field (64-bit signed integer). precisionStep="0" enables efficient
numeric sorting and minimizes index size; precisionStep="8" (the
default) enables efficient range queries.

UUIDField

Universally Unique Identifier (UUID). Pass in a value of "NEW" and Solr
will create a new UUID. Note: configuring a UUIDField instance with a
default value of "NEW" is not advisable for most users when using
SolrCloud (and not possible if the UUID value is configured as the unique
key field) since the result will be that each replica of each document will
get a unique UUID value. Using UUIDUpdateProcessorFactory to
generate UUID values when documents are added is recommended
instead.

The MultiTermAwareComponent has been added to relevant solr.TextField entries in schema.xml (e.g.,
wildcards, regex, prefix, range, etc.) to allow automatic lowercasing for multi-term queries.
Further, you can optionally specify a multi-term analyzer in field types in your schema: ; if you don't do this, analyzer will process the fields according to their specific attributes.

Working with Currencies and Exchange Rates
The currency FieldType provides support for monetary values to Solr/Lucene with query-time currency conversion
and exchange rates. The following features are supported:
Point queries
Range queries
Function range queries (new in Solr 4.2)
Sorting
Currency parsing by either currency code or symbol
Symmetric & asymmetric exchange rates (asymmetric exchange rates are useful if there are fees associated
with exchanging the currency)
Configuring Currencies

The currency field type is defined in schema.xml. This is the default configuration of this type:


In this example, we have defined the name and class of the field type, and defined the defaultCurrency as
"USD", for U.S. Dollars. We have also defined a currencyConfig to use a file called "currency.xml". This is a file
of exchange rates between our default currency to other currencies. There is an alternate implementation that would
allow regular downloading of currency data. See #Exchange Rates below for more.
At indexing time, money fields can be indexed in a native currency. For example, if a product on an e-commerce site
is listed in Euros, indexing the price field as "1000,EUR" will index it appropriately. The price should be separated
from the currency by a comma, and the price must be encoded with a floating point value (a decimal point).

Apache Solr Reference Guide 4.10

42

During query processing, range and point queries are both supported.
Exchange Rates

You configure exchange rates by specifying a provider. Natively, two provider types are supported: FileExchange
RateProvider or OpenExchangeRatesOrgProvider.
FileExchangeRateProvider

This provider requires you to provide a file of exchange rates. It is the default, meaning that to use this provider you
only need to specify the file path and name as a value for currencyConfig in the definition for this type.
There is a sample currency.xml file included with Solr, found in the same directory as the schema.xml file. Here
is a small snippet from this file:








rate="0.869914" />
rate="7.800095" />
rate="8.966508" />






OpenExchangeRatesOrgProvider

With Solr 4, you can configure Solr to download exchange rates from OpenExchangeRates.Org, with updates rates
between USD and 158 currencies hourly. These rates are symmetrical only.
In this case, you need to specify the providerClass in the definitions for the field type. Here is an example:


The refreshInterval is minutes, so the above example will download the newest rates every 60 minutes.

Working with Dates
Date Formatting

Solr's TrieDateField (and deprecated DateField) represents a point in time with millisecond precision. The
format used is a restricted form of the canonical representation of dateTime in the XML Schema specification:
YYYY-MM-DDThh:mm:ssZ

Apache Solr Reference Guide 4.10

43

YYYY is the year.
MM is the month.
DD is the day of the month.
hh is the hour of the day as on a 24-hour clock.
mm is minutes.
ss is seconds.
Z is a literal 'Z' character indicating that this string representation of the date is in UTC
Note that no time zone can be specified; the String representations of dates is always expressed in Coordinated
Universal Time (UTC). Here is an example value:
1972-05-20T17:33:18Z
You can optionally include fractional seconds if you wish, although any precision beyond milliseconds will be
ignored. Here are examples value with sub-seconds include:
1972-05-20T17:33:18.772Z
1972-05-20T17:33:18.77Z
1972-05-20T17:33:18.7Z
Date Math

Solr's date field types also supports date math expressions, which makes it easy to create times relative to fixed
moments in time, include the current time which can be represented using the special value of " NOW".
Date Math Syntax

Date math expressions consist either adding some quantity of time in a specified unit, or rounding the current time
by a specified unit. expressions can be chained and are evaluated left to right.
For example: this represents a point in time two months from now:
NOW+2MONTHS
This is one day ago:
NOW-1DAY
A slash is used to indicate rounding. This represents the beginning of the current hour:
NOW/HOUR
The following example computes (with millisecond precision) the point in time six months and three days into the
future and then rounds that time to the beginning of that day:
NOW+6MONTHS+3DAYS/DAY
Note that while date math is most commonly used relative to NOW it can be applied to any fixed moment in time as
well:
1972-05-20T17:33:18.772Z+6MONTHS+3DAYS/DAY
Request Parameters That Affect Date Math

NOW
The NOW parameter is used internally by Solr to ensure consistent date math expression parsing across multiple

Apache Solr Reference Guide 4.10

44

nodes in a distributed request. But it can be specified to instruct Solr to use an arbitrary moment in time (past or
future) to override for all situations where the the special value of " NOW" would impact date math expressions.
It must be specified as a (long valued) milliseconds since epoch
Example:
q=solr&fq=start_date:[* TO NOW]&NOW=1384387200000
TZ
By default, all date math expressions are evaluated relative to the UTC TimeZone, but the TZ parameter can be
specified to override this behaviour, by forcing all date based addition and rounding to be relative to the specified tim
e zone.
For example, the following request will use range faceting to facet over the current month, "per day" relative UTC:
http://localhost:8983/solr/select?q=*:*&facet.range=my_date_field&facet=true&facet.ran
ge.start=NOW/MONTH&facet.range.end=NOW/MONTH%2B1MONTH&facet.range.gap=%2B1DAY

0
name="2013-11-02T00:00:00Z">0
name="2013-11-03T00:00:00Z">0
name="2013-11-04T00:00:00Z">0
name="2013-11-05T00:00:00Z">0
name="2013-11-06T00:00:00Z">0
name="2013-11-07T00:00:00Z">0

While in this example, the "days" will be computed relative to the specified time zone - including any applicable
Daylight Savings Time adjustments:
http://localhost:8983/solr/select?q=*:*&facet.range=my_date_field&facet=true&facet.ran
ge.start=NOW/MONTH&facet.range.end=NOW/MONTH%2B1MONTH&facet.range.gap=%2B1DAY&TZ=Ameri
ca/Los_Angeles

0
name="2013-11-02T07:00:00Z">0
name="2013-11-03T07:00:00Z">0
name="2013-11-04T08:00:00Z">0
name="2013-11-05T08:00:00Z">0
name="2013-11-06T08:00:00Z">0
name="2013-11-07T08:00:00Z">0

Working with Enum Fields
The EnumField type allows defining a field whose values are a closed set, and the sort order is pre-determined but
is not alphabetic nor numeric. Examples of this are severity lists, or risk definitions.
Defining an EnumField in schema.xml

The EnumField type definition is quite simple, as in this example defining field types for "priorityLevel" and
"riskLevel" enumerations:

Apache Solr Reference Guide 4.10

45




Besides the name and the class, which are common to all field types, this type also takes two additional
parameters:
enumsConfig: the name of a configuration file that contains the  list of field values and their order
that you wish to use with this field type. If a path to the file is not defined specified, the file should be in the co
nf directory for the collection.
enumName: the name of the specific enumeration in the enumsConfig file to use for this type.
Defining the EnumField configuration file

The file named with the enumsConfig parameter can contain multiple enumeration value lists with different names
if there are multiple uses for enumerations in your Solr schema.
In this example, there are two value lists defined. Each list is between enum opening and closing tags:



Not Available
Low
Medium
High"
Urgent


Unknown
Very Low
Low
Medium
High
Critical



Changing Values
You cannot change the order, or remove, existing values in an  without reindexing.
You can however add new values to the end.

Working with External Files and Processes
The ExternalFileField Type

The ExternalFileField type makes it possible to specify the values for a field in a file outside the Solr index. For
such a field, the file contains mappings from a key field to the field value. Another way to think of this is that, instead
of specifying the field in documents as they are indexed, Solr finds values for this field in the external file.
External fields are not searchable. They can be used only for function queries or display. For more

Apache Solr Reference Guide 4.10

46

information on function queries, see the section on Function Queries.
The ExternalFileField type is handy for cases where you want to update a particular field in many documents
more often than you want to update the rest of the documents. For example, suppose you have implemented a
document rank based on the number of views. You might want to update the rank of all the documents daily or
hourly, while the rest of the contents of the documents might be updated much less frequently. Without ExternalF
ileField, you would need to update each document just to change the rank. Using ExternalFileField is
much more efficient because all document values for a particular field are stored in an external file that can be
updated as frequently as you wish.
In schema.xml, the definition of this field type might look like this:


The keyField attribute defines the key that will be defined in the external file. It is usually the unique key for the
index, but it doesn't need to be as long as the keyField can be used to identify documents in the index. A defVal
defines a default value that will be used if there is no entry in the external file for a particular document.
The valType attribute specifies the actual type of values that will be found in the file. The type specified must be
either a float field type, so valid values for this attribute are pfloat, float or tfloat. This attribute can be
omitted.
Format of the External File

The file itself is located in Solr's index directory, which by default is $SOLR_HOME/data. The name of the file should
be external_fieldname or external_fieldname.*. For the example above, then, the file could be named ex
ternal_entryRankFile or external_entryRankFile.txt.
If any files using the name pattern .* (such as .txt) appear, the last (after being sorted by name) will be
used and previous versions will be deleted. This behavior supports implementations on systems where one
may not be able to overwrite a file (for example, on Windows, if the file is in use).
The file contains entries that map a key field, on the left of the equals sign, to a value, on the right. Here are a few
example entries:
doc33=1.414
doc34=3.14159
doc40=42
The keys listed in this file do not need to be unique. The file does not need to be sorted, but Solr will be able to
perform the lookup faster if it is.
Reloading an External File

As of Solr 4.1, it's possible to define an event listener to reload an external file when either a searcher is reloaded or
when a new searcher is started. See the section Query-Related Listeners for more information, but a sample
definition in solrconfig.xml might look like this:

Apache Solr Reference Guide 4.10

47




Pre-Analyzing a Field Type

The PreAnalyzedField type provides a way to send to Solr serialized token streams, optionally with independent
stored values of a field, and have this information stored and indexed without any additional text processing applied
in Solr. This is useful if user wants to submit field content that was already processed by some existing external text
processing pipeline (e.g., it has been tokenized, annotated, stemmed, synonyms inserted, etc.), while using all the
rich attributes that Lucene's TokenStream provides (per-token attributes).
The serialization format is pluggable using implementations of PreAnalyzedParser interface. There are two
out-of-the-box implementations:
JsonPreAnalyzedParser: as the name suggests, it parses content that uses JSON to represent field's content.
This is the default parser to use if the field type is not configured otherwise.
SimplePreAnalyzedParser: uses a simple strict plain text format, which in some situations may be easier to
create than JSON.
There is only one configuration parameter, parserImpl. The value of this parameter should be a fully qualified
class name of a class that implements PreAnalyzedParser interface. The default value of this parameter is org.apa
che.solr.schema.JsonPreAnalyzedParser.

Field Properties by Use Case
Here is a summary of common use cases, and the attributes the fields or field types should have to support the
case. An entry of true or false in the table indicates that the option must be set to the given value for the use case to
function correctly. If no entry is provided, the setting of that attribute has no impact on the case.
Use Case

indexed

search within
field

true

retrieve
contents

stored

multiValued

omitNorms

termVectors

termPositions

docValues

true

use as unique
key

true

false

sort on field

true7

false

use field boosts

true 1

true7

false

5

document
boosts affect
searches within
field
highlighting

false

true 4

true

Apache Solr Reference Guide 4.10

true2

true 3

48

faceting 5

true7

add multiple
values,
maintaining
order

true7
true

field length
affects doc
score

false

MoreLikeThis 5

true 6

Notes:
1

Recommended but not necessary.

2

Will be used if present, but not necessary.

3

(if termVectors=true)

4

A tokenizer must be defined for the field, but it doesn't need to be indexed.

5

Described in Understanding Analyzers, Tokenizers, and Filters.

6

Term vectors are not mandatory here. If not true, then a stored field is analyzed. So term vectors are
recommended, but only required if stored=false.
7

Either indexed or docValues must be true, but both are not required. DocValues can be more efficient in many

cases.

Defining Fields
Fields are defined in the fields element of schema.xml. Once you have the field types set up, defining the fields
themselves is simple.
Example

The following example defines a field named price with a type named float and a default value of 0.0; the inde
xed and stored properties are explicitly set to true, while any other properties specified on the float field type
are inherited.


Field Properties

Property

Description

name

The name of the field. Field names should consist of alphanumeric or underscore characters only and
not start with a digit. This is not currently strictly enforced, but other field names will not have first
class support from all components and back compatibility is not guaranteed. Names with both leading
and trailing underscores (e.g. _version_) are reserved. Every field must have a name.

type

The name of the fieldType for this field. This will be found in the "name" attribute on the fieldTyp
e definition. Every field must have a type.

default

A default value that will be added automatically to any document that does not have a value in this
field when it is indexed. If this property is not specified, there is no default.

Apache Solr Reference Guide 4.10

49

Optional Field Type Override Properties

Fields can have the same options as field types. The field type options serve as defaults which can be overridden by
options defined per field. Included below is the table of field type properties from the section Field Type Definitions
and Properties:
Property

Description

Values

indexed

If true, the value of the field can be used in queries to retrieve matching
documents

true or
false

stored

If true, the actual value of the field can be retrieved by queries

true or
false

docValues

If true, the value of the field will be put in a column-oriented DocValues str
ucture

true or
false

sortMissingFirst
sortMissingLast

Control the placement of documents when a sort field is not present. As of
Solr 3.5, these work for all numeric fields, including Trie and date fields.

true or
false

multiValued

If true, indicates that a single document might contain multiple values for
this field type

true or
false

omitNorms

If true, omits the norms associated with this field (this disables length
normalization and index-time boosting for the field, and saves some
memory). Defaults to true for all primitive (non-analyzed) field types, such
as int, float, data, bool, and string. Only full-text fields or fields that need
an index-time boost need norms.

true or
false

omitTermFreqAndPositions

If true, omits term frequency, positions, and payloads from postings for
this field. This can be a performance boost for fields that don't require that
information. It also reduces the storage space required for the index.
Queries that rely on position that are issued on a field with this option will
silently fail to find documents. This property defaults to true for all fields
that are not text fields.

true or
false

omitPositions

Similar to omitTermFreqAndPositions but preserves term frequency

true or
false

information
termVectors
termPositions
termOffsets

These options instruct Solr to maintain full term vectors for each
document, optionally including the position and offset information for each
term occurrence in those vectors. These can be used to accelerate
highlighting and other ancillary functionality, but impose a substantial cost
in terms of index size. They are not necessary for typical uses of Solr

true or
false

required

Instructs Solr to reject any attempts to add a document which does not
have a value for this field. This property defaults to false.

true or
false

Related Topics
SchemaXML-Fields
Field Options by Use Case

Copying Fields
Apache Solr Reference Guide 4.10

50

You might want to interpret some document fields in more than one way. Solr has a mechanism for making copies of
fields so that you can apply several distinct field types to a single piece of incoming information.
The name of the field you want to copy is the source, and the name of the copy is the destination. In schema.xml,
it's very simple to make copies of fields:


If the text destination field has data of its own in the input documents, the contents of the cat field will be added
as additional values – just as if all of the values had originally been specified by the client. Remember to configure
your fields as multivalued="true" if they will ultimately get multiple values (either from a multivalued source, or
multiple copyField directives, etc...)
The maxChars parameter, an int parameter, establishes an upper limit for the number of characters to be copied
from the source value when constructing the value added to the destination field. This limit is useful for situations in
which you want to copy some data from the source field, but also control the size of index files.
Both the source and the destination of copyField can contain either leading or trailing asterisks, which will match
anything. For example, the following line will copy the contents of all incoming fields that match the wildcard pattern
*_t to the text field.:


The copyField command can use a wildcard (*) character in the dest parameter only if the source para
meter contains one as well. copyField uses the matching glob from the source field for the dest field
name into which the source content is copied.

Related Topics
SchemaXML-Copy Fields

Dynamic Fields
Dynamic fields allow Solr to index fields that you did not explicitly define in your schema. This is useful if you
discover you have forgotten to define one or more fields. Dynamic fields can make your application less brittle by
providing some flexibility in the documents you can add to Solr.
A dynamic field is just like a regular field except it has a name with a wildcard in it. When you are indexing
documents, a field that does not match any explicitly defined fields can be matched with a dynamic field.
For example, suppose your schema includes a dynamic field with a name of *_i. If you attempt to index a
document with a cost_i field, but no explicit cost_i field is defined in the schema, then the cost_i field will have
the field type and analysis defined for *_i.
Dynamic fields are also defined in the fields element of schema.xml. Like fields, they have a name, a field type,
and options.


It is recommended that you include basic dynamic field mappings (like that shown above) in your schema.xml. The

Apache Solr Reference Guide 4.10

51

mappings can be very useful.

Related Topics
SchemaXML-Dynamic Fields

Other Schema Elements
This section describes several other important elements of schema.xml.

Unique Key
The uniqueKey element specifies which field is a unique identifier for documents. Although uniqueKey is not
required, it is nearly always warranted by your application design. For example, uniqueKey should be used if you
will ever update a document in the index.
You can define the unique key field by naming it:
id

Starting with Solr 4, schema defaults and copyFields cannot be used to populate the uniqueKey field. You also
can't use UUIDUpdateProcessorFactory to have uniqueKey values generated automatically.
Further, the operation will fail if the uniqueKey field is used, but is multivalued (or inherits the multivalueness from
the fieldtype). However, uniqueKey will continue to work, as long as the field is properly used.

Default Search Field
If you are using the Lucene query parser, queries that don't specify a field name will use the defaultSearchFiel
d. The DisMax and Extended DisMax query parsers do not use this value.
Use of the defaultSearchField element is deprecated in Solr versions 3.6 and higher. Instead, you
should use the df request parameter. At some point, the defaultSearchField element may be
removed.
For more information about query parsers, see the section on Query Syntax and Parsing.

Query Parser Default Operator
In queries with multiple terms, Solr can either return results where all conditions are met or where one or more
conditions are met. The operator controls this behavior. An operator of AND means that all conditions must be
fulfilled, while an operator of OR means that one or more conditions must be true.
In schema.xml, the solrQueryParser element controls what operator is used if an operator is not specified in
the query. The default operator setting only applies to the Lucene query parser, not the DisMax or Extended DisMax
query parsers, which internally hard-code their operators to OR.
The query parser default operator parameter has been deprecated in Solr versions 3.6 and higher. You are
instead encouraged to specify the query parser q.op parameter in your request handler.

Similarity

Apache Solr Reference Guide 4.10

52

Similarity is a Lucene class used to score a document in searching. This class can be changed in order to provide a
more custom sorting. With Solr 4, you can configure a different similarity for each field, meaning that scoring a
document will differ depending on what's in each field. However, you can still configure a global similarity is
configured in the schema.xml file, where an implicit instance of DefaultSimilarityFactory is used.
A global  declaration can be used to specify a custom similarity implementation that you want Solr to
use when dealing with your index. A similarity can be specified either by referring directly to the name of a class with
a no-argument constructor:


or by referencing a SimilarityFactory implementation, which may take optional initialization parameters:

P
L
H2
7


Beginning with Solr 4, similarity factories can be specified on individual field types:



SPL
DF
H2



This example uses IBSimilarityFactory (using the Information-Based model), but there are several similarity
implementations that can be used. For Solr 4.2, SweetSpotSimilarityFactory has been added. Other options
include BM25SimilarityFactory, DFRSimilarityFactory, SchemaSimilarityFactory and others. For
details, see the Solr Javadocs for the similarity factories.

Related Topics
SchemaXML-Miscellaneous Settings
UniqueKey

Schema API
The Solr schema API allows using a REST API to get information about the schema.xml for each collection (or
core for standalone Solr), including defined field types, fields, dynamic fields, and copy field declarations. In Solr 4.2
and 4.3, it only allows GET (read-only) access, but in Solr 4.4, new fields and copyField directives may be added to
the schema. Future Solr releases will extend this functionality to allow more schema elements to be updated.
To enable schema modification with this API, the schema will need to be managed and mutable. See the section Ma
naged Schema Definition in SolrConfig for more information.
The API allows two output modes for all calls: JSON or XML. When requesting the complete schema, there is
another output mode which is XML modeled after the schema.xml file itself.

Apache Solr Reference Guide 4.10

53

The base address for the API is http://:/, where  is
usually solr, though you may have configured it differently. Example base address: http://localhost:8983/s
olr .
In the API entry points and example URLs below, you may alternatively specify a Solr core name where it says colle
ction.
API Entry Points
Retrieve schema information
Retrieve the Entire Schema
List Fields
List a Specific Field
List Dynamic Fields
List a Specific Dynamic Field Rule
List Field Types
List a Specific Field Type
List Copy Fields
Show Schema Name
Show the Schema Version
List UniqueKey
Show Global Similarity
Get the Default Query Operator
Modify the schema
Create new schema fields
Create one new schema field
Create new copyField directives
Manage Resource Data
Related Topics

API Entry Points
/collection/schema: retrieve the entire schema
/collection/schema/fields: retrieve information about all defined fields, or create new fields with optional
copyField directives
/collection/schema/fields/name : retrieve information about a named field, or create a new named field
with optional copyField directives
/collection/schema/dynamicfields: retrieve information about dynamic field rules
/collection/schema/dynamicfields/name : retrieve information about a named dynamic rule
/collection/schema/fieldtypes: retrieve information about field types
/collection/schema/fieldtypes/name : retrieve information about a named field type
/collection/schema/copyfields: retrieve information about copy fields, or create new copyField directives
/collection/schema/name: retrieve the schema name
/collection/schema/version: retrieve the schema version
/collection/schema/uniquekey: retrieve the defined uniqueKey
/collection/schema/similarity: retrieve the global similarity definition
/collection/schema/solrqueryparser/defaultoperator: retrieve the default operator
/collection/schema/managed/resource/paths: Manipulate managed resource data

Retrieve schema information

Apache Solr Reference Guide 4.10

54

Retrieve the Entire Schema

GET /collection/schema
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json, xml or schema.x
ml. If not specified, JSON will be returned by default.

Output
Output Content
The output will include all fields, field types, dynamic rules and copy field rules. The schema name and version are
also included.
Examples
Input
Get the entire schema in JSON.
curl http://localhost:8983/solr/collection1/schema?wt=json

Get the entire schema in XML.
curl http://localhost:8983/solr/collection1/schema?wt=xml

Get the entire schema in "schema.xml" format.
curl http://localhost:8983/solr/collection1/schema?wt=schema.xml

Output
The samples below have been truncated to only show a few snippets of the output.
Example output in JSON:

Apache Solr Reference Guide 4.10

55

{
"responseHeader":{
"status":0,
"QTime":5},
"schema":{
"name":"example",
"version":1.5,
"uniqueKey":"id",
"fieldTypes":[{
"name":"alphaOnlySort",
"class":"solr.TextField",
"sortMissingLast":true,
"omitNorms":true,
"analyzer":{
"tokenizer":{
"class":"solr.KeywordTokenizerFactory"},
"filters":[{
"class":"solr.LowerCaseFilterFactory"},
{
"class":"solr.TrimFilterFactory"},
{
"class":"solr.PatternReplaceFilterFactory",
"replace":"all",
"replacement":"",
"pattern":"([^a-z])"}]}},
...
"fields":[{
"name":"_version_",
"type":"long",
"indexed":true,
"stored":true},
{
"name":"author",
"type":"text_general",
"indexed":true,
"stored":true},
{
"name":"cat",
"type":"string",
"multiValued":true,
"indexed":true,
"stored":true},
...
"copyFields":[{
"source":"author",
"dest":"text"},
{
"source":"cat",
"dest":"text"},
{
"source":"content",
"dest":"text"},
...
{
"source":"author",
"dest":"author_s"}]}}

Apache Solr Reference Guide 4.10

56

Example output in XML:


0
5


example
1.5
id


alphaOnlySort
solr.TextField
true
true


solr.KeywordTokenizerFactory



solr.LowerCaseFilterFactory


solr.TrimFilterFactory


solr.PatternReplaceFilterFactory
all

([^a-z])




...

author
author_s





Example output in schema.xml format:

Apache Solr Reference Guide 4.10

57


id









...





List Fields

GET /collection/schema/fields
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include each field and any defined configuration for each field. The defined configuration can vary for
each field, but will minimally include the field name, the type, if it is indexed and if it is stored. If multiValued i
s defined as either true or false (most likely true), that will also be shown. See the section Defining Fields for more
information about each parameter.
Examples
Input
Get a list of all fields.

Apache Solr Reference Guide 4.10

58

curl http://localhost:8983/solr/collection1/schema/fields?wt=json

Output
The sample output below has been truncated to only show a few fields.
{
"fields": [
{
"indexed": true,
"name": "_version_",
"stored": true,
"type": "long"
},
{
"indexed": true,
"name": "author",
"stored": true,
"type": "text_general"
},
{
"indexed": true,
"multiValued": true,
"name": "cat",
"stored": true,
"type": "string"
},
...
],
"responseHeader": {
"QTime": 1,
"status": 0
}
}

List a Specific Field

GET /collection/schema/fields/fieldname
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

fieldname

The specific field name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Apache Solr Reference Guide 4.10

Description

59

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include each field and any defined configuration for the field. The defined configuration can vary for a
field, but will minimally include the field name, the type, if it is indexed and if it is stored. If multiValued is
defined as either true or false (most likely true), that will also be shown. See the section Defining Fields for more
information about each parameter.
Examples
Input
Get the 'author' field.
curl http://localhost:8983/solr/collection1/schema/fields/author?wt=json

Output
{
"field": {
"indexed": true,
"name": "author",
"stored": true,
"type": "text_general"
},
"responseHeader": {
"QTime": 2,
"status": 0
}
}

List Dynamic Fields

GET /collection/schema/dynamicfields
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Apache Solr Reference Guide 4.10

60

Output
Output Content
The output will include each dynamic field rule and the defined configuration for each rule. The defined configuration
can vary for each rule, but will minimally include the dynamic field name, the type, if it is indexed and if it is store
d. See the section Dynamic Fields for more information about each parameter.
Examples
Input
Get a list of all dynamic field declarations
curl http://localhost:8983/solr/collection1/schema/dynamicfields?wt=json

Output
The sample output below has been truncated.
{
"dynamicFields": [
{
"indexed": true,
"name": "*_coordinate",
"stored": false,
"type": "tdouble"
},
{
"multiValued": true,
"name": "ignored_*",
"type": "ignored"
},
{
"name": "random_*",
"type": "random"
},
{
"indexed": true,
"multiValued": true,
"name": "attr_*",
"stored": true,
"type": "text_general"
},
{
"indexed": true,
"multiValued": true,
"name": "*_txt",
"stored": true,
"type": "text_general"
}
...
],
"responseHeader": {
"QTime": 1,
"status": 0
}
}

Apache Solr Reference Guide 4.10

61

List a Specific Dynamic Field Rule

GET /collection/schema/dynamicfields/name
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

name

The name of the dynamic field rule.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include the requested dynamic field rule and any defined configuration for the rule. The defined
configuration can vary for each rule, but will minimally include the dynamic field name, the type, if it is indexed and
if it is stored. See the section Dynamic Fields for more information about each parameter.
Examples
Input
Get the details of the "*_s" rule.
curl http://localhost:8983/solr/collection1/schema/dynamicfields/*_s?wt=json

Output
{
"dynamicfield": {
"indexed": true,
"name": "*_s",
"stored": true,
"type": "string"
},
"responseHeader": {
"QTime": 1,
"status": 0
}
}

List Field Types

Apache Solr Reference Guide 4.10

62

GET /collection/schema/fieldtypes
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include each field type and any defined configuration for the type. The defined configuration can vary
for each type, but will minimally include the field type name and the class. If query or index analyzers, tokenizers,
or filters are defined, those will also be shown with other defined parameters. See the section Solr Field Types for
more information about how to configure various types of fields.
Examples
Input
Get a list of all field types.
curl http://localhost:8983/solr/collection1/schema/fieldtypes?wt=json

Output
The sample output below has been truncated to show a few different field types from different parts of the list.

Apache Solr Reference Guide 4.10

63

{
"fieldTypes": [
{
"analyzer": {
"class": "solr.TokenizerChain",
"filters": [
{
"class": "solr.LowerCaseFilterFactory"
},
{
"class": "solr.TrimFilterFactory"
},
{
"class": "solr.PatternReplaceFilterFactory",
"pattern": "([^a-z])",
"replace": "all",
"replacement": ""
}
],
"tokenizer": {
"class": "solr.KeywordTokenizerFactory"
}
},
"class": "solr.TextField",
"dynamicFields": [],
"fields": [],
"name": "alphaOnlySort",
"omitNorms": true,
"sortMissingLast": true
},
...
{
"class": "solr.TrieFloatField",
"dynamicFields": [
"*_fs",
"*_f"
],
"fields": [
"price",
"weight"
],
"name": "float",
"positionIncrementGap": "0",
"precisionStep": "0"
},
...
}

List a Specific Field Type

GET /collection/schema/fieldtypes/name
Input
Path Parameters

Apache Solr Reference Guide 4.10

64

Key

Description

collection

The collection (or core) name.

name

The name of the field type.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include each field type and any defined configuration for the type. The defined configuration can vary
for each type, but will minimally include the field type name and the class. If query and/or index analyzers,
tokenizers, or filters are defined, those will be shown with other defined parameters. See the section Solr Field
Types for more information about how to configure various types of fields.
Examples
Input
Get details of the "date" field type.
curl http://localhost:8983/solr/collection1/schema/fieldtypes/date?wt=json

Output
The sample output below has been truncated.
{
"fieldType": {
"class": "solr.TrieDateField",
"dynamicFields": [
"*_dts",
"*_dt"
],
"fields": [
"last_modified"
],
"name": "date",
"positionIncrementGap": "0",
"precisionStep": "0"
},
"responseHeader": {
"QTime": 2,
"status": 0
}
}

List Copy Fields

Apache Solr Reference Guide 4.10

65

GET /collection/schema/copyfields
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include the source and destination of each copy field rule defined in schema.xml. For more
information about copying fields, see the section Copying Fields.
Examples
Input
Get a list of all copyfields.
curl http://localhost:8983/solr/collection1/schema/copyfields?wt=json

Output
The sample output below has been truncated to the first few copy definitions.

Apache Solr Reference Guide 4.10

66

{
"copyFields": [
{
"dest": "text",
"source": "author"
},
{
"dest": "text",
"source": "cat"
},
{
"dest": "text",
"source": "content"
},
{
"dest": "text",
"source": "content_type"
},
...
],
"responseHeader": {
"QTime": 3,
"status": 0
}
}

Show Schema Name

GET /collection/schema/name
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will be simply the name given to the schema.
Examples
Input
Get the schema name.

Apache Solr Reference Guide 4.10

67

curl http://localhost:8983/solr/collection1/schema/name?wt=json

Output
{
"responseHeader":{
"status":0,
"QTime":1},
"name":"example"}

Show the Schema Version

GET /collection/schema/version
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will simply be the schema version in use.
Examples
Input
Get the schema version
curl http://localhost:8983/solr/collection1/schema/version?wt=json

Output
{
"responseHeader":{
"status":0,
"QTime":2},
"version":1.5}

List UniqueKey

Apache Solr Reference Guide 4.10

68

GET /collection/schema/uniquekey
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include simply the field name that is defined as the uniqueKey for the index.
Examples
Input
List the uniqueKey.
curl http://localhost:8983/solr/collection1/schema/uniquekey?wt=json

Output
The sample output below has been truncated to the first few copy definitions.
{
"responseHeader":{
"status":0,
"QTime":2},
"uniqueKey":"id"}

Show Global Similarity

GET /collection/schema/similarity
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters

Apache Solr Reference Guide 4.10

69

The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include the class name of the global similarity defined (if any).
Examples
Input
Get the similarity implementation.
curl http://localhost:8983/solr/collection1/schema/similarity?wt=json

Output
{
"responseHeader":{
"status":0,
"QTime":1},
"similarity":{
"class":"org.apache.solr.search.similarities.DefaultSimilarityFactory"}}

Get the Default Query Operator

GET /collection/schema/solrqueryparser/defaultoperator
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, JSON will be returned by default.

Output
Output Content
The output will include simply the default operator if none is defined by the user.

Apache Solr Reference Guide 4.10

70

Examples
Input
Get the default operator.
curl
http://localhost:8983/solr/collection1/schema/solrqueryparser/defaultoperator?wt=json

Output
{
"responseHeader":{
"status":0,
"QTime":2},
"defaultOperator":"OR"}

Modify the schema
Create new schema fields

POST /collection/schema/fields
To enable schema modification, the schema will need to be managed and mutable. See the section Managed
Schema Definition in SolrConfig for more information.
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, json will be returned by default.

Request body
Only JSON format is supported in the request body. The JSON must contain an array of one or more new field
specifications, each of which must include mappings for the new field's name and type. All attributes specifiable on
a schema  declaration may be specified here - see Defining Fields.
Additionally, copyField destination(s) may optionally be specified. Note that each specified copyField destination
must be an existing schema field (and not a dynamic field). In particular, since the new fields specified in a new field
creation request are defined all at once, you cannot specify a copyField that targets another new field in the same
request - instead, you have to make two requests, defining the copyField destination in the first new field creation
request, then specifying that field as a copyField destination in the second new field creation request.

Apache Solr Reference Guide 4.10

71

The curl utility can provide the request body via its --data-binary option.
Output
Output Content
The output will be the response header, containing a status code, and if there was a problem, an associated error
message.
Example output in the default JSON format:
{
"responseHeader":{
"status":0,
"QTime":8}}

Examples
Input
Add two new fields:
curl http://localhost:8983/solr/collection1/schema/fields -X POST -H
'Content-type:application/json' --data-binary '
[
{
"name":"sell-by",
"type":"tdate",
"stored":true
},
{
"name":"catchall",
"type":"text_general",
"stored":false
}
]'

Add a third new field and copy it to the "catchall" field created above:
curl http://localhost:8983/solr/collection1/schema/fields -X POST -H
'Content-type:application/json' --data-binary '
[
{
"name":"department",
"type":"string",
"docValues":"true",
"default":"no department",
"copyFields": [ "catchall" ]
}
]'

Create one new schema field

PUT /collection/schema/fields/name
To enable schema modification, the schema will need to be managed and mutable. See the section Managed

Apache Solr Reference Guide 4.10

72

Schema Definition in SolrConfig for more information.
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

name

The new field name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, json will be returned by default.

Request body
Only JSON format is supported in the request body. The body must include a set of mappings, minimally for the new
field's name and type. All attributes specifiable on a schema  declaration may be
specified here - see Defining Fields.
Additionally, copyField destination(s) may optionally be specified. Note that each specified copyField destination
must be an existing schema field (and not a dynamic field).
The curl utility can provide the request body via its --data-binary option.
Output
Output Content
The output will be the response header, containing a status code, and if there was a problem, an associated error
message.
Example output in the default JSON format:
{
"responseHeader":{
"status":0,
"QTime":4}}

Examples
Input
Add a new field named "narrative":

Apache Solr Reference Guide 4.10

73

curl http://localhost:8983/solr/collection1/schema/fields/narrative -X PUT -H
'Content-type:application/json' --data-binary '
{
"type":"text_general",
"stored":true,
"termVectors":true,
"termPositions":true,
"termOffsets":true
}'

Add a new field named "color" and copy it to two fields, named "narrative" and "catchall", which must already exist in
the schema:
curl http://localhost:8983/solr/collection1/schema/fields/color -X PUT -H
'Content-type:application/json' --data-binary '
{
"type":"string",
"stored":true,
"copyFields": [
"narrative",
"catchall"
]
}'

Create new copyField directives

POST /collection/schema/copyfields
To enable schema modification, the schema will need to be managed and mutable. See the section Managed
Schema Definition in SolrConfig for more information.
Input
Path Parameters
Key

Description

collection

The collection (or core) name.

Query Parameters
The query parameters can be added to the API request after a '?'.
Key

Type

Required

Default

Description

wt

string

No

json

Defines the format of the response. The options are json or xml. If not
specified, json will be returned by default.

Request body
Only JSON format is supported in the request body. The body must contain an array of zero or more copyField
directives, each containing a mapping from source to the source field name, and from dest to an array of
destination field name(s).

Apache Solr Reference Guide 4.10

74

source field names must either be an existing field, or be a field name glob (with an asterisk either at the beginning
or the end, or consist entirely of a single asterisk). dest field names must either be existing fields, or, if source is a
glob, dest fields may be globs that match an existing dynamic field.
The curl utility can provide the request body via its --data-binary option.
Output
Output Content
The output will be the response header, containing a status code, and if there was a problem, an associated error
message.
Example output in the default JSON format:
{
"responseHeader":{
"status":0,
"QTime":2}}

Examples
Input
Copy the "affiliations" field to the "relations" field, and the "shelf" field to the "location" and "catchall" fields:
curl http://localhost:8983/solr/collection1/schema/copyfields -X POST -H
'Content-type:application/json' --data-binary '
[
{
"source":"affiliations",
"dest": [
"relations"
]
},
{
"source":"shelf",
"dest": [
"location",
"catchall"
]
}
]'

Copy all fields names matching "finance_*" to the "*_s" dynamic field:
curl http://localhost:8983/solr/collection1/schema/copyfields -X POST -H
'Content-type:application/json' --data-binary '
[
{
"source":"finance_*",
"dest": [
"*_s"
]
}
]'

Apache Solr Reference Guide 4.10

75

Manage Resource Data

The Managed Resources REST API provides a mechanism for any Solr plugin to expose resources that should
support CRUD (Create, Read, Update, Delete) operations. Depending on what Field Types and Analyzers are
configured in your Schema, additional /schema/ REST API paths may exist. See the Managed Resources section
for more information and examples.

Related Topics
Managed Schema Definition in SolrConfig

Putting the Pieces Together
At the highest level, schema.xml is structured as follows. This example is not real XML, but it gives you an idea of
the structure of the file.









Obviously, most of the excitement is in types and fields, where the field types and the actual field definitions live.
These are supplemented by copyFields. Sandwiched between fields and the copyField section are the unique
key, default search field, and the default query operator.

Choosing Appropriate Numeric Types
For general numeric needs, use TrieIntField, TrieLongField, TrieFloatField, and TrieDoubleField w
ith precisionStep="0".
If you expect users to make frequent range queries on numeric types, use the default precisionStep (by not
specifying it) or specify it as precisionStep="8" (which is the default). This offers faster speed for range queries
at the expense of increasing index size.

Working With Text
Handling text properly will make your users happy by providing them with the best possible results for text searches.
One technique is using a text field as a catch-all for keyword searching. Most users are not sophisticated about their
searches and the most common search is likely to be a simple keyword search. You can use copyField to take a
variety of fields and funnel them all into a single text field for keyword searches. In the example schema
representing a store, copyField is used to dump the contents of cat, name, manu, features, and includes int
o a single field, text. In addition, it could be a good idea to copy ID into text in case users wanted to search for a
particular product by passing its product number to a keyword search.
Another technique is using copyField to use the same field in different ways. Suppose you have a field that is a
list of authors, like this:
Schildt, Herbert; Wolpert, Lewis; Davies, P.

Apache Solr Reference Guide 4.10

76

For searching by author, you could tokenize the field, convert to lower case, and strip out punctuation:
schildt / herbert / wolpert / lewis / davies / p
For sorting, just use an untokenized field, converted to lower case, with punctuation stripped:
schildt herbert wolpert lewis davies p
Finally, for faceting, use the primary author only via a StringField:
Schildt, Herbert

Related Topics
SchemaXML

DocValues
An exciting addition to Solr functionality was introduced in Solr 4.2. This functionality has been around in Lucene for
a while, but is now available to Solr users.
DocValues are a way of building the index that is more efficient for some purposes.

Why DocValues?
The standard way that Solr builds the index is with an inverted index. This style builds a list of terms found in all the
documents in the index and next to each term is a list of documents that the term appears in (as well as how many
times the term appears in that document). This makes search very fast - since users search by terms, having a
ready list of term-to-document values makes the query process faster.
For other features that we now commonly associate with search, such as sorting, faceting, and highlighting, this
approach is not very efficient. The faceting engine, for example, must look up each term that appears in each
document that will make up the result set and pull the document IDs in order to build the facet list. In Solr, this is
maintained in memory, and can be slow to load (depending on the number of documents, terms, etc.).
In Lucene 4.0, a new approach was introduced. DocValue fields are now column-oriented fields with a
document-to-value mapping built at index time. This approach promises to relieve some of the memory
requirements of the fieldCache and make lookups for faceting, sorting, and grouping much faster.

How to Use DocValues
To use docValues, you only need to enable it for a field that you will use it with. As with all schema design, you need
to define a field type and then define fields of that type with docValues enabled. All of these actions are done in sch
ema.xml.
Enabling a field for docValues only requires adding docValues="true" to the field definition, as in this example
(from Solr's default schema.xml):


Prior to Solr 4.5, a field could not be empty to be used with docValues; in Solr 4.5, that restriction is removed.
If you have already indexed data into your Solr index, you will need to completely re-index your content after
changing your field definitions in schema.xml in order to successfully use docValues.

Apache Solr Reference Guide 4.10

77

DocValues are only available for specific field types. The types chosen determine the underlying Lucene docValue
type that will be used. The available Solr field types are:
String fields of type StrField.
If the field is single-valued (i.e., multi-valued is false), Lucene will use the SORTED type.
If the field is multi-valued, Lucene will use the SORTED_SET type.
Any Trie* fields.
If the field is single-valued (i.e., multi-valued is false), Lucene will use the NUMERIC type.
If the field is multi-valued, Lucene will use the SORTED_SET type.
UUID fields
These Lucene types are related to how the values are sorted and stored.
There is an additional configuration option available, which is to modify the docValuesFormat used by the field
type. The default implementation employs a mixture of loading some things into memory and keeping some on disk.
In some cases, however, you may choose to specify an alternative DocValuesFormat implementation. For example,
you could choose to keep everything in memory by specifying docValuesFormat="Memory" on a field type:


Please note that the docValuesFormat option may change in future releases.
Lucene index back-compatibility is only supported for the default codec. If you choose to customize the doc
ValuesFormat in your schema.xml, upgrading to a future version of Solr may require you to either switch
back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or
re-build your entire index from scratch after upgrading.

Related Topics
DocValues are quite new to Solr. For more background see:
Introducing Lucene Index Doc Values, by Simon Willnauer, at SearchWorkings.org
Fun with DocValues in Solr 4.2, by David Arthur, at SearchHub.org
The old wiki page on DocValues (note, that page is now obsoleted by this one)

Schemaless Mode
Schemaless Mode is a set of Solr features that, when used together, allow users to rapidly construct an effective
schema by simply indexing sample data, without having to manually edit the schema. These Solr features, all
specified in solrconfig.xml, are:
1. Managed schema: Schema modifications are made through Solr APIs rather than manual edits - see Manage
d Schema Definition in SolrConfig.
2. Field value class guessing: Previously unseen fields are run through a cascading set of value-based parsers,
which guess the Java class of field values - parsers for Boolean, Integer, Long, Float, Double, and Date are
currently available.
3. Automatic schema field addition, based on field value class(es): Previously unseen fields are added to the
schema, based on field value Java classes, which are mapped to schema field types - see Solr Field Types.

Using the Schemaless Example

Apache Solr Reference Guide 4.10

78

The three features of schemaless mode are pre-configured in the example/example-schemaless/solr/ direct
ory in the Solr distribution. To start Solr in this pre-configured schemaless mode, go to the example/ directory and
start up Solr, setting the solr.solr.home system property to this directory on the command line:
java -Dsolr.solr.home=example-schemaless/solr -jar start.jar

The schema in example-schemaless/solr/collection1/conf/ is shipped with only two fields, id and _ver
sion_, as can be seen from calling the /schema/fields Schema API - curl http://localhost:8983/solr
/schema/fields outputs:
{
"responseHeader":{
"status":0,
"QTime":1},
"fields":[{
"name":"_version_",
"type":"long",
"indexed":true,
"stored":true},
{
"name":"id",
"type":"string",
"multiValued":false,
"indexed":true,
"required":true,
"stored":true,
"uniqueKey":true}]}

Configuring Schemaless Mode
As described above, there are three configuration elements that need to be in place to use Solr in schemaless
mode. If you use the solrconfig.xml from example/example-schemaless these elements are configured
already. If, however, you would like to implement schemaless on your own, you should make the following changes.
Enable Managed Schema

As described in the section Managed Schema Definition in SolrConfig, changing the schemaFactory will allow the
schema to be modified by the Schema API. Your solrconfig.xml should have a section like the one below (and
the ClassicIndexSchemaFactory should be commented out or removed).

true
managed-schema


Define an UpdateRequestProcessorChain

The UpdateRequestProcessorChain allows Solr to guess field types, and you can define the default field type
classes to use. To start, you should define it as follows (see the javadoc links below for update processor factory
documentation):

Apache Solr Reference Guide 4.10

79












yyyy-MM-dd'T'HH:mm:ss.SSSZ
yyyy-MM-dd'T'HH:mm:ss,SSSZ
yyyy-MM-dd'T'HH:mm:ss.SSS
yyyy-MM-dd'T'HH:mm:ss,SSS
yyyy-MM-dd'T'HH:mm:ssZ
yyyy-MM-dd'T'HH:mm:ss
yyyy-MM-dd'T'HH:mmZ
yyyy-MM-dd'T'HH:mm
yyyy-MM-dd HH:mm:ss.SSSZ
yyyy-MM-dd HH:mm:ss,SSSZ
yyyy-MM-dd HH:mm:ss.SSS
yyyy-MM-dd HH:mm:ss,SSS
yyyy-MM-dd HH:mm:ssZ
yyyy-MM-dd HH:mm:ss
yyyy-MM-dd HH:mmZ
yyyy-MM-dd HH:mm
yyyy-MM-dd



text_general

java.lang.Boolean
booleans


java.util.Date
tdates


java.lang.Long
java.lang.Integer
tlongs


java.lang.Number
tdoubles





Javadocs for update processor factories mentioned above:
UUIDUpdateProcessorFactory

Apache Solr Reference Guide 4.10

80

ParseBooleanFieldUpdateProcessorFactory

ParseLongFieldUpdateProcessorFactory
ParseDoubleFieldUpdateProcessorFactory
ParseDateFieldUpdateProcessorFactory
AddSchemaFieldsUpdateProcessorFactory
Make the UpdateRequestProcessorChain the Default for the UpdateRequestHandler

Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandler to use it
when working with index updates (i.e., adding, removing, replacing documents). Here is an example using the /upd
ate requestHandler:


add-unknown-fields-to-the-schema



After each of these changes have been made, Solr should be restarted (or, you can reload the cores to load
the new solrconfig.xml definitions).

Examples of Indexed Documents
Once the schemaless mode has been enabled (whether you configured it manually or are using example-schema
less), documents that include fields that are not defined in your schema should be added to the index, and the new
fields added to the schema.
For example, adding a CSV document will cause its fields that are not in the schema to be added, with fieldTypes
based on values:
curl "http://localhost:8983/solr/update?commit=true" -H "Content-type:application/csv"
-d '
id,Artist,Album,Released,Rating,FromDistributor,Sold
44C,Old Shews,Mead for Walking,1988-08-13,0.01,14,0'

Output indicating success:

0106


The fields now in the schema (output from curl http://localhost:8983/solr/schema/fields ):

Apache Solr Reference Guide 4.10

81

{
"responseHeader":{
"status":0,
"QTime":1},
"fields":[{
"name":"Album",
"type":"text_general"}, //
fieldType
{
"name":"Artist",
"type":"text_general"}, //
fieldType
{
"name":"FromDistributor",
"type":"tlongs"},
//
{
"name":"Rating",
"type":"tdoubles"},
//
{
"name":"Released",
"type":"tdates"},
//
{
"name":"Sold",
"type":"tlongs"},
//
{
"name":"_version_",
...
},
{
"name":"id",
...
}]}

Field value guessed as String -> text_general

Field value guessed as String -> text_general

Field value guessed as Long -> tlongs fieldType

Field value guessed as Double -> tdoubles fieldType

Field value guessed as Date -> tdates fieldType

Field value guessed as Long -> tlongs fieldType

You Can Still Be Explicit
Even if you want to use schemaless mode for most fields, you can still use the Schema API to pre-emptively
create some fields, with explicit types, before you index documents that use them.
Internally, the Schema REST API and the Schemaless Update Processors both use the same Managed
Schema functionality.

Once a field has been added to the schema, its field type is fixed. As a consequence, adding documents with field
value(s) that conflict with the previously guessed field type will fail. For example, after adding the above document,
the Sold field has the fieldType tlongs, but the document below has a non-integral decimal value in this field:
curl "http://localhost:8983/solr/update?commit=true" -H "Content-type:application/csv"
-d '
id,Description,Sold
19F,Cassettes by the pound,4.93'

This document will fail, as shown in this output:

Apache Solr Reference Guide 4.10

82



400
7


ERROR: [doc=19F] Error adding field 'Sold'='4.93' msg=For input
string: "4.93"
400



Apache Solr Reference Guide 4.10

83

Understanding Analyzers, Tokenizers, and Filters
The following sections describe how Solr breaks down and works with textual data. There are three main concepts
to understand: analyzers, tokenizers, and filters.
Field analyzers are used both during ingestion, when a document is indexed, and at query time. An analyzer
examines the text of fields and generates a token stream. Analyzers may be a single class or they may be
composed of a series of tokenizer and filter classes.
Tokenizers break field data into lexical units, or tokens.
Filters examine a stream of tokens and keep them, transform or discard them, or create new ones. Tokenizers and
filters may be combined to form pipelines, or chains, where the output of one is input to the next. Such a sequence
of tokenizers and filters is called an analyzer and the resulting output of an analyzer is used to match query results
or build indices.

Using Analyzers, Tokenizers and Filters
Although the analysis process is used for both indexing and querying, the same analysis process need not be used
for both operations. For indexing, you often want to simplify, or normalize, words. For example, setting all letters to
lowercase, eliminating punctuation and accents, mapping words to their stems, and so on. Doing so can increase
recall because, for example, "ram", "Ram" and "RAM" would all match a query for "ram". To increase query-time
precision, a filter could be employed to narrow the matches by, for example, ignoring all-cap acronyms if you're
interested in male sheep, but not Random Access Memory.
The tokens output by the analysis process define the values, or terms, of that field and are used either to build an
index of those terms when a new document is added, or to identify which documents contain the terms your are
querying for.

For More Information
These sections will show you how to configure field analyzers and also serves as a reference for the details of
configuring each of the available tokenizer and filter classes. It also serves as a guide so that you can configure your
own analysis classes if you have special needs that cannot be met with the included filters or tokenizers.
For Analyzers, see:
Analyzers: Detailed conceptual information about Solr analyzers.
Running Your Analyzer: Detailed information about testing and running your Solr analyzer.
For Tokenizers, see:
About Tokenizers: Detailed conceptual information about Solr tokenizers.
Tokenizers: Information about configuring tokenizers, and about the tokenizer factory classes included in this
distribution of Solr.
For Filters, see:
About Filters: Detailed conceptual information about Solr filters.
Filter Descriptions: Information about configuring filters, and about the filter factory classes included in this
distribution of Solr.
CharFilterFactories: Information about filters for pre-processing input characters.
To find out how to use Tokenizers and Filters with various languages, see:

Apache Solr Reference Guide 4.10

84

Language Analysis: Information about tokenizers and filters for character set conversion or for use with
specific languages.

Analyzers
An analyzer examines the text of fields and generates a token stream. Analyzers are specified as a child of the  element in the schema.xml configuration file that can be found in the solr/conf directory, or
wherever solrconfig.xml is located.
In normal usage, only fields of type solr.TextField will specify an analyzer. The simplest way to configure an
analyzer is with a single  element whose class attribute is a fully qualified Java class name. The
named class must derive from org.apache.lucene.analysis.Analyzer. For example:




In this case a single class, WhitespaceAnalyzer, is responsible for analyzing the content of the named text field
and emitting the corresponding tokens. For simple cases, such as plain English prose, a single analyzer class like
this may be sufficient. But it's often necessary to do more complex analysis of the field content.
Even the most complex analysis requirements can usually be decomposed into a series of discrete, relatively simple
processing steps. As you will soon discover, the Solr distribution comes with a large selection of tokenizers and
filters that covers most scenarios you are likely to encounter. Setting up an analyzer chain is very straightforward;
you specify a simple  element (no class attribute) with child elements that name factory classes for the
tokenizer and filters to use, in the order you want them to run.
For example:










Note that classes in the org.apache.solr.analysis package may be referred to here with the shorthand solr
. prefix.
In this case, no Analyzer class was specified on the  element. Rather, a sequence of more specialized
classes are wired together and collectively act as the Analyzer for the field. The text of the field is passed to the first
item in the list (solr.StandardTokenizerFactory), and the tokens that emerge from the last one (solr.Engl
ishPorterFilterFactory) are the terms that are used for indexing or querying any fields that use the
"nametext" fieldType.

Analysis Phases
Analysis takes place in two contexts. At index time, when a field is being created, the token stream that results from

Apache Solr Reference Guide 4.10

85

analysis is added to an index and defines the set of terms (including positions, sizes, and so on) for the field. At
query time, the values being searched for are analyzed and the terms that result are matched against those that are
stored in the field's index.
In many cases, the same analysis should be applied to both phases. This is desirable when you want to query for
exact string matches, possibly with case-insensitivity, for example. In other cases, you may want to apply slightly
different analysis steps during indexing than those used at query time.
If you provide a simple  definition for a field type, as in the examples above, then it will be used for both
indexing and queries. If you want distinct analyzers for each phase, you may include two  definitions
distinguished with a type attribute. For example:













In this theoretical example, at index time the text is tokenized, the tokens are set to lowercase, any that are not listed
in keepwords.txt are discarded and those that remain are mapped to alternate values as defined by the synonym
rules in the file syns.txt. This essentially builds an index from a restricted set of possible values and then
normalizes them to values that may not even occur in the original text.
At query time, the only normalization that happens is to convert the query terms to lowercase. The filtering and
mapping steps that occur at index time are not applied to the query terms. Queries must then, in this example, be
very precise, using only the normalized terms that were stored at index time.

About Tokenizers
The job of a tokenizer is to break up a stream of text into tokens, where each token is (usually) a sub-sequence of
the characters in the text. An analyzer is aware of the field it is configured for, but a tokenizer is not. Tokenizers read
from a character stream (a Reader) and produce a sequence of Token objects (a TokenStream).
Characters in the input stream may be discarded, such as whitespace or other delimiters. They may also be added
to or replaced, such as mapping aliases or abbreviations to normalized forms. A token contains various metadata in
addition to its text value, such as the location at which the token occurs in the field. Because a tokenizer may
produce tokens that diverge from the input text, you should not assume that the text of the token is the same text
that occurs in the field, or that its length is the same as the original text. It's also possible for more than one token to
have the same position or refer to the same offset in the original text. Keep this in mind if you use token metadata
for things like highlighting search results in the field text.






Apache Solr Reference Guide 4.10

86

The class named in the tokenizer element is not the actual tokenizer, but rather a class that implements the org.ap
ache.solr.analysis.TokenizerFactory interface. This factory class will be called upon to create new
tokenizer instances as needed. Objects created by the factory must derive from org.apache.lucene.analysis
.TokenStream, which indicates that they produce sequences of tokens. If the tokenizer produces tokens that are
usable as is, it may be the only component of the analyzer. Otherwise, the tokenizer's output tokens will serve as
input to the first filter stage in the pipeline.
A TypeTokenFilterFactory is available that creates a TypeTokenFilter that filters tokens based on their
TypeAttribute, which is set in factory.getStopTypes.
For a complete list of the available TokenFilters, see the section Tokenizers.

When To use a CharFilter vs. a TokenFilter
There are several pairs of CharFilters and TokenFilters that have related (ie: MappingCharFilter and ASCIIFol
dingFilter) or nearly identical (ie: PatternReplaceCharFilterFactory and PatternReplaceFilterFac
tory) functionality and it may not always be obvious which is the best choice.
The decision about which to use depends largely on which Tokenizer you are using, and whether you need to
preprocess the stream of characters.
For example, suppose you have a tokenizer such as StandardTokenizer and although you are pretty happy with
how it works overall, you want to customize how some specific characters behave. You could modify the rules and
re-build your own tokenizer with JFlex, but it might be easier to simply map some of the characters before
tokenization with a CharFilter.

About Filters
Like tokenizers, filters consume input and produce a stream of tokens. Filters also derive from org.apache.lucen
e.analysis.TokenStream. Unlike tokenizers, a filter's input is another TokenStream. The job of a filter is usually
easier than that of a tokenizer since in most cases a filter looks at each token in the stream sequentially and decides
whether to pass it along, replace it or discard it.
A filter may also do more complex analysis by looking ahead to consider multiple tokens at once, although this is
less common. One hypothetical use for such a filter might be to normalize state names that would be tokenized as
two words. For example, the single token "california" would be replaced with "CA", while the token pair "rhode"
followed by "island" would become the single token "RI".
Because filters consume one TokenStream and produce a new TokenStream, they can be chained one after
another indefinitely. Each filter in the chain in turn processes the tokens produced by its predecessor. The order in
which you specify the filters is therefore significant. Typically, the most general filtering is done first, and later filtering
stages are more specialized.









This example starts with Solr's standard tokenizer, which breaks the field's text into tokens. Those tokens then pass

Apache Solr Reference Guide 4.10

87

through Solr's standard filter, which removes dots from acronyms, and performs a few other common operations. All
the tokens are then set to lowercase, which will facilitate case-insensitive matching at query time.
The last filter in the above example is a stemmer filter that uses the Porter stemming algorithm. A stemmer is
basically a set of mapping rules that maps the various forms of a word back to the base, or stem, word from which
they derive. For example, in English the words "hugs", "hugging" and "hugged" are all forms of the stem word "hug".
The stemmer will replace all of these terms with "hug", which is what will be indexed. This means that a query for
"hug" will match the term "hugged", but not "huge".
Conversely, applying a stemmer to your query terms will allow queries containing non stem terms, like "hugging", to
match documents with different variations of the same stem word, such as "hugged". This works because both the
indexer and the query will map to the same stem ("hug").
Word stemming is, obviously, very language specific. Solr includes several language-specific stemmers created by
the Snowball generator that are based on the Porter stemming algorithm. The generic Snowball Porter Stemmer
Filter can be used to configure any of these language stemmers. Solr also includes a convenience wrapper for the
English Snowball stemmer. There are also several purpose-built stemmers for non-English languages. These
stemmers are described in Language Analysis.

Tokenizers
You configure the tokenizer for a text field type in schema.xml with a  element, as a child of :







The class attribute names a factory class that will instantiate a tokenizer object
when needed. Tokenizer factory classes implement the org.apache.solr.an
alysis.TokenizerFactory. A TokenizerFactory's create() method
accepts a Reader and returns a TokenStream. When Solr creates the tokenizer it
passes a Reader object that provides the content of the text field.

Apache Solr Reference Guide 4.10

88

Tokenizers
discussed in this
section:
Standard
Tokenizer
Classic
Tokenizer
Keyword
Tokenizer
Letter
Tokenizer
Lower Case
Tokenizer
N-Gram
Tokenizer
Edge N-Gram
Tokenizer
ICU
Tokenizer
Path
Hierarchy
Tokenizer
Regular
Expression
Pattern
Tokenizer
UAX29 URL
Email
Tokenizer
White Space
Tokenizer
Related
Topics

Arguments may be passed to tokenizer factories by setting attributes on the  element.






The following sections describe the tokenizer factory classes included in this release of Solr.
For more information about Solr's tokenizers, see http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

Standard Tokenizer

Apache Solr Reference Guide 4.10

89

This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters
are discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token, including Internet domain
names.
The "@" character is among the set of token-splitting punctuation, so email addresses are not preserved as
single tokens.
Note that words are split at hyphens.
The Standard Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following token types:
, , , , and .
Factory class: solr.StandardTokenizerFactory
Arguments:
maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by max
TokenLength.
Example:




In: "Please, email john.doe@foo.com by 03-09, re: m37-xq."
Out: "Please", "email", "john.doe", "foo.com", "by", "03", "09", "re", "m37", "xq"

Classic Tokenizer
The Classic Tokenizer preserves the same behavior as the Standard Tokenizer of Solr versions 3.1 and previous. It
does not use the Unicode standard annex UAX#29 word boundary rules that the Standard Tokenizer uses. This
tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are
discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token.
Words are split at hyphens, unless there is a number in the word, in which case the token is not split and the
numbers and hyphen(s) are preserved.
Recognizes Internet domain names and email addresses and preserves them as a single token.
Factory class: solr.ClassicTokenizerFactory
Arguments:
maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by max
TokenLength.
Example:




Apache Solr Reference Guide 4.10

90

In: "Please, email john.doe@foo.com by 03-09, re: m37-xq."
Out: "Please", "email", "john.doe@foo.com", "by", "03-09", "re", "m37-xq"

Keyword Tokenizer
This tokenizer treats the entire text field as a single token.
Factory class: solr.KeywordTokenizerFactory
Arguments: None
Example:




In: "Please, email john.doe@foo.com by 03-09, re: m37-xq."
Out: "Please, email john.doe@foo.com by 03-09, re: m37-xq."

Letter Tokenizer
This tokenizer creates tokens from strings of contiguous letters, discarding all non-letter characters.
Factory class: solr.LetterTokenizerFactory
Arguments: None
Example:




In: "I can't."
Out: "I", "can", "t"

Lower Case Tokenizer
Tokenizes the input stream by delimiting at non-letters and then converting all letters to lowercase. Whitespace and
non-letters are discarded.
Factory class: solr.LowerCaseTokenizerFactory
Arguments: None
Example:




In: "I just LOVE my iPhone!"

Apache Solr Reference Guide 4.10

91

Out: "i", "just", "love", "my", "iphone"

N-Gram Tokenizer
Reads the field text and generates n-gram tokens of sizes in the given range.
Factory class: solr.NGramTokenizerFactory
Arguments:
minGramSize: (integer, default 1) The minimum n-gram size, must be > 0.
maxGramSize: (integer, default 2) The maximum n-gram size, must be >= minGramSize.
Example:
Default behavior. Note that this tokenizer operates over the whole field. It does not break the field at whitespace. As
a result, the space character is included in the encoding.




In: "hey man"
Out: "h", "e", "y", " ", "m", "a", "n", "he", "ey", "y ", " m", "ma", "an"
Example:
With an n-gram size range of 4 to 5:




In: "bicycle"
Out: "bicy", "bicyc", "icyc", "icycl", "cycl", "cycle", "ycle"

Edge N-Gram Tokenizer
Reads the field text and generates edge n-gram tokens of sizes in the given range.
Factory class: solr.EdgeNGramTokenizerFactory
Arguments:
minGramSize: (integer, default is 1) The minimum n-gram size, must be > 0.
maxGramSize: (integer, default is 1) The maximum n-gram size, must be >= minGramSize.
side: ("front" or "back", default is "front") Whether to compute the n-grams from the beginning (front) of the text or
from the end (back).
Example:
Default behavior (min and max default to 1):

Apache Solr Reference Guide 4.10

92





In: "babaloo"
Out: "b"
Example:
Edge n-gram range of 2 to 5




In: "babaloo"
Out:"ba", "bab", "baba", "babal"
Example:
Edge n-gram range of 2 to 5, from the back side:




In: "babaloo"
Out: "oo", "loo", "aloo", "baloo"

ICU Tokenizer
This tokenizer processes multilingual text and tokenizes it appropriately based on its script attribute.
You can customize this tokenizer's behavior by specifying per-script rule files. To add per-script rules, add a rulefi
les argument, which should contain a comma-separated list of code:rulefile pairs in the following format:
four-letter ISO 15924 script code, followed by a colon, then a resource path. For example, to specify rules for Latin
(script code "Latn") and Cyrillic (script code "Cyrl"), you would enter Latn:my.Latin.rules.rbbi,Cyrl:my.Cy
rillic.rules.rbbi.
The default solr.ICUTokenizerFactory provides UAX#29 word break rules tokenization (like solr.Standard
Tokenizer), but also includes custom tailorings for Hebrew (specializing handling of double and single quotation
marks), and for syllable tokenization for Khmer, Lao, and Myanmar.
Factory class: solr.ICUTokenizerFactory
Arguments:
rulefile: a comma-separated list of code:rulefile pairs in the following format: four-letter ISO 15924 script
code, followed by a colon, then a resource path.
Example:

Apache Solr Reference Guide 4.10

93










Path Hierarchy Tokenizer
This tokenizer creates synonyms from file path hierarchies.
Factory class: solr.PathHierarchyTokenizerFactory
Arguments:
delimiter: (character, no default) You can specify the file path delimiter and replace it with a delimiter you
provide. This can be useful for working with backslash delimiters.
replace: (character, no default) Specifies the delimiter character Solr uses in the tokenized output.
Example:






In: "c:\usr\local\apache"
Out: "c:", "c:/usr", "c:/usr/local", "c:/usr/local/apache"

Regular Expression Pattern Tokenizer
This tokenizer uses a Java regular expression to break the input text stream into tokens. The expression provided
by the pattern argument can be interpreted either as a delimiter that separates tokens, or to match patterns that
should be extracted from the text as tokens.
See the Javadocs for java.util.regex.Pattern for more information on Java regular expression syntax.
Factory class: solr.PatternTokenizerFactory
Arguments:
pattern: (Required) The regular expression, as defined by in java.util.regex.Pattern.
group: (Optional, default -1) Specifies which regex group to extract as the token(s). The value -1 means the regex
should be treated as a delimiter that separates tokens. Non-negative group numbers (>= 0) indicate that character
sequences matching that regex group should be converted to tokens. Group zero refers to the entire regex, groups
greater than zero refer to parenthesized sub-expressions of the regex, counted from left to right.

Apache Solr Reference Guide 4.10

94

Example:
A comma separated list. Tokens are separated by a sequence of zero or more spaces, a comma, and zero or more
spaces.




In: "fee,fie, foe , fum, foo"
Out: "fee", "fie", "foe", "fum", "foo"
Example:
Extract simple, capitalized words. A sequence of at least one capital letter followed by zero or more letters of either
case is extracted as a token.




In: "Hello. My name is Inigo Montoya. You killed my father. Prepare to die."
Out: "Hello", "My", "Inigo", "Montoya", "You", "Prepare"
Example:
Extract part numbers which are preceded by "SKU", "Part" or "Part Number", case sensitive, with an optional
semi-colon separator. Part numbers must be all numeric digits, with an optional hyphen. Regex capture groups are
numbered by counting left parenthesis from left to right. Group 3 is the subexpression "[0-9-]+", which matches one
or more digits or hyphens.




In: "SKU: 1234, Part Number 5678, Part: 126-987"
Out: "1234", "5678", "126-987"

UAX29 URL Email Tokenizer
This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters
are discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token.
Words are split at hyphens, unless there is a number in the word, in which case the token is not split and the
numbers and hyphen(s) are preserved.
Recognizes top-level Internet domain names (validated against the white list in the IANA Root Zone
Database when the tokenizer was generated); email addresses; file : //, http(s)://, and ftp:// addr

Apache Solr Reference Guide 4.10

95

esses; IPv4 and IPv6 addresses; and preserves them as a single token.
The UAX29 URL Email Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following
token types: , , , , , , and .
Factory class: solr.UAX29URLEmailTokenizerFactory
Arguments:
maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by max
TokenLength.
Example:




In: "Visit http://accarol.com/contact.htm?from=external&a=10 or e-mail bob.cratchet@accarol.com"
Out: "Visit", "http://accarol.com/contact.htm?from=external&a=10", "or", "email", "bob.cratchet@accarol.com"

White Space Tokenizer
Simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters as
tokens. Note that any punctuation will be included in the tokenization.
Factory class: solr.WhitespaceTokenizerFactory
Arguments: None
Example:




In: "To be, or what?"
Out: "To", "be,", "or", "what?"

Related Topics
TokenizerFactories

Filter Descriptions
You configure each filter with a  element in schema.xml as a
child of , following the  element. Filter
definitions should follow a tokenizer or another filter definition because they
take a TokenStream as input. For example.

Apache Solr Reference Guide 4.10

Filters discussed in this
section:
ASCII Folding Filter
Beider-Morse Filter
Classic Filter
Common Grams
Filter
Collation Key Filter

96




...



The class attribute names a factory class that will instantiate a filter object
as needed. Filter factory classes must implement the org.apache.solr.
analysis.TokenFilterFactory interface. Like tokenizers, filters are
also instances of TokenStream and thus are producers of tokens. Unlike
tokenizers, filters also consume tokens from a TokenStream. This allows
you to mix and match filters, in any order you prefer, downstream of a
tokenizer.
Arguments may be passed to tokenizer factories to modify their behavior by
setting attributes on the  element. For example:







The following sections describe the filter factories that are included in this
release of Solr.
For more information about Solr's filters, see http://wiki.apache.org/solr/Ana
lyzersTokenizersTokenFilters.

Apache Solr Reference Guide 4.10

Edge N-Gram Filter
English Minimal
Stem Filter
Hunspell Stem
Filter
Hyphenated Words
Filter
ICU Folding Filter
ICU Normalizer 2
Filter
ICU Transform
Filter
Keep Words Filter
KStem Filter
Length Filter
Lower Case Filter
Managed Stop
Filter
Managed Synonym
Filter
N-Gram Filter
Numeric Payload
Token Filter
Pattern Replace
Filter
Phonetic Filter
Porter Stem Filter
Position Filter
Factory
Remove Duplicates
Token Filter
Reversed Wildcard
Filter
Shingle Filter
Snowball Porter
Stemmer Filter
Standard Filter
Stop Filter
Synonym Filter
Token Offset
Payload Filter
Trim Filter
Type As Payload
Filter
Type Token Filter
Word Delimiter
Filter
Related Topics

97

ASCII Folding Filter
This filter converts alphabetic, numeric, and symbolic Unicode characters which are not in the Basic Latin Unicode
block (the first 127 ASCII characters) to their ASCII equivalents, if one exists. This filter converts characters from the
following Unicode blocks:
C1 Controls and Latin-1 Supplement (PDF)
Latin Extended-A (PDF)
Latin Extended-B (PDF)
Latin Extended Additional (PDF)
Latin Extended-C (PDF)
Latin Extended-D (PDF)
IPA Extensions (PDF)
Phonetic Extensions (PDF)
Phonetic Extensions Supplement (PDF)
General Punctuation (PDF)
Superscripts and Subscripts (PDF)
Enclosed Alphanumerics (PDF)
Dingbats (PDF)
Supplemental Punctuation (PDF)
Alphabetic Presentation Forms (PDF)
Halfwidth and Fullwidth Forms (PDF)
Factory class: solr.ASCIIFoldingFilterFactory
Arguments: None
Example:




In: "á" (Unicode character 00E1)
Out: "a" (ASCII character 97)

Beider-Morse Filter
Implements the Beider-Morse Phonetic Matching (BMPM) algorithm, which allows identification of similar names,
even if they are spelled differently or in different languages. More information about how this works is available in
the section on Phonetic Matching.
Factory class: solr.BeiderMorseFilterFactory
Arguments:
nameType: Types of names. Valid values are GENERIC, ASHKENAZI, or SEPHARDIC. If not processing
Ashkenazi or Sephardic names, use GENERIC.
ruleType: Types of rules to apply. Valid values are APPROX or EXACT.

Apache Solr Reference Guide 4.10

98

concat: Defines if multiple possible matches should be combined with a pipe ("|").
languageSet: The language set to use. The value "auto" will allow the Filter to identify the language, or a
comma-separated list can be supplied.
Example:






Classic Filter
This filter takes the output of the Classic Tokenizer and strips periods from acronyms and "'s" from possessives.
Factory class: solr.ClassicFilterFactory
Arguments: None
Example:





In: "I.B.M. cat's can't"
Tokenizer to Filter: "I.B.M", "cat's", "can't"
Out: "IBM", "cat", "can't"

Common Grams Filter
This filter creates word shingles by combining common tokens such as stop words with regular tokens. This is useful
for creating phrase queries containing common words, such as "the cat." Solr normally ignores stop words in queried
phrases, so searching for "the cat" would return all matches for the word "cat."
Factory class: solr.CommonGramsFilterFactory
Arguments:
words: (a common word file in .txt format) Provide the name of a common word file, such as stopwords.txt.
format: (optional) If the stopwords list has been formatted for Snowball, you can specify format="snowball" so
Solr can read the stopwords file.
ignoreCase: (boolean) If true, the filter ignores the case of words when comparing them to the common word file.
The default is false.
Example:

Apache Solr Reference Guide 4.10

99






In: "the Cat"
Tokenizer to Filter: "the", "Cat"
Out: "the_cat"

Collation Key Filter
Collation allows sorting of text in a language-sensitive way. It is usually used for sorting, but can also be used with
advanced searches. We've covered this in much more detail in the section on Unicode Collation.

Edge N-Gram Filter
This filter generates edge n-gram tokens of sizes within the given range.
Factory class: solr.EdgeNGramFilterFactory
Arguments:
minGramSize: (integer, default 1) The minimum gram size.
maxGramSize: (integer, default 1) The maximum gram size.
Example:
Default behavior.





In: "four score and twenty"
Tokenizer to Filter: "four", "score", "and", "twenty"
Out: "f", "s", "a", "t"
Example:
A range of 1 to 4.





In: "four score"
Tokenizer to Filter: "four", "score"

Apache Solr Reference Guide 4.10

100

Out: "f", "fo", "fou", "four", "s", "sc", "sco", "scor"
Example:
A range of 4 to 6.





In: "four score and twenty"
Tokenizer to Filter: "four", "score", "and", "twenty"
Out: "four", "scor", "score", "twen", "twent", "twenty"

English Minimal Stem Filter
This filter stems plural English words to their singular form.
Factory class: solr.EnglishMinimalStemFilterFactory
Arguments: None
Example:





In: "dogs cats"
Tokenizer to Filter: "dogs", "cats"
Out: "dog", "cat"

Hunspell Stem Filter
The Hunspell Stem Filter provides support for several languages. You must provide the dictionary (.dic) and rules (
.aff) files for each language you wish to use with the Hunspell Stem Filter. You can download those language files
here. Be aware that your results will vary widely based on the quality of the provided dictionary and rules files. For
example, some languages have only a minimal word list with no morphological information. On the other hand, for
languages that have no stemmer but do have an extensive dictionary file, the Hunspell stemmer may be a good
choice.
Factory class: solr.HunspellStemFilterFactory
Arguments:
dictionary: (required) The path of a dictionary file.
affix: (required) The path of a rules file.
ignoreCase: (boolean) controls whether matching is case sensitive or not. The default is false.
strictAffixParsing: (boolean) controls whether the affix parsing is strict or not. If true, an error while reading an
affix rule causes a ParseException, otherwise is ignored. The default is true.

Apache Solr Reference Guide 4.10

101

Example:





In: "jump jumping jumped"
Tokenizer to Filter: "jump", "jumping", "jumped"
Out: "jump", "jump", "jump"

Hyphenated Words Filter
This filter reconstructs hyphenated words that have been tokenized as two tokens because of a line break or other
intervening whitespace in the field test. If a token ends with a hyphen, it is joined with the following token and the
hyphen is discarded. Note that for this filter to work properly, the upstream tokenizer must not remove trailing
hyphen characters. This filter is generally only useful at index time.
Factory class: solr.HyphenatedWordsFilterFactory
Arguments: None
Example:





In: "A hyphen- ated word"
Tokenizer to Filter: "A", "hyphen-", "ated", "word"
Out: "A", "hyphenated", "word"

ICU Folding Filter
This filter is a custom Unicode normalization form that applies the foldings specified in Unicode Technical Report 30
in addition to the NFKC_Casefold normalization form as described in ICU Normalizer 2 Filter. This filter is a better
substitute for the combined behavior of the ASCII Folding Filter, Lower Case Filter, and ICU Normalizer 2 Filter.
To use this filter, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to
add to your solr_home/lib.
Factory class: solr.ICUFoldingFilterFactory
Arguments: None
Example:

Apache Solr Reference Guide 4.10

102






For detailed information on this normalization form, see http://www.unicode.org/reports/tr30/tr30-4.html.

ICU Normalizer 2 Filter
This filter factory normalizes text according to one of five Unicode Normalization Forms as described in Unicode
Standard Annex #15:
NFC: (name="nfc" mode="compose") Normalization Form C, canonical decomposition
NFD: (name="nfc" mode="decompose") Normalization Form D, canonical decomposition, followed by
canonical composition
NFKC: (name="nfkc" mode="compose") Normalization Form KC, compatibility decomposition
NFKD: (name="nfkc" mode="decompose") Normalization Form KD, compatibility decomposition, followed by
canonical composition
NFKC_Casefold: (name="nfkc_cf" mode="compose") Normalization Form KC, with additional Unicode case
folding. Using the ICU Normalizer 2 Filter is a better-performing substitution for the Lower Case Filter and
NFKC normalization.
Factory class: solr.ICUNormalizer2FilterFactory
Arguments:
name: (string) The name of the normalization form; nfc, nfd, nfkc, nfkd, nfkc_cf
mode: (string) The mode of Unicode character composition and decomposition; compose or decompose
Example:





For detailed information about these Unicode Normalization Forms, see http://unicode.org/reports/tr15/.
To use this filter, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to
add to your solr_home/lib.

ICU Transform Filter
This filter applies ICU Tranforms to text. This filter supports only ICU System Transforms. Custom rule sets are not
supported.
Factory class: solr.ICUTransformFilterFactory
Arguments:
id: (string) The identifier for the ICU System Transform you wish to apply with this filter. For a full list of ICU System
Transforms, see http://demo.icu-project.org/icu-bin/translit?TEMPLATE_FILE=data/translit_rule_main.html.

Apache Solr Reference Guide 4.10

103

Example:





For detailed information about ICU Transforms, see http://userguide.icu-project.org/transforms/general.
To use this filter, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to
add to your solr_home/lib.

Keep Words Filter
This filter discards all tokens except those that are listed in the given word list. This is the inverse of the Stop Words
Filter. This filter can be useful for building specialized indices for a constrained set of terms.
Factory class: solr.KeepWordFilterFactory
Arguments:
words: (required) Path of a text file containing the list of keep words, one per line. Blank lines and lines that begin
with "#" are ignored. This may be an absolute path, or a simple filename in the Solr config directory.
ignoreCase: (true/false) If true then comparisons are done case-insensitively. If this argument is true, then the
words file is assumed to contain only lowercase words. The default is false.
Example:
Where keepwords.txt contains:
happy
funny
silly





In: "Happy, sad or funny"
Tokenizer to Filter: "Happy", "sad", "or", "funny"
Out: "funny"
Example:
Same keepwords.txt, case insensitive:





Apache Solr Reference Guide 4.10

104

In: "Happy, sad or funny"
Tokenizer to Filter: "Happy", "sad", "or", "funny"
Out: "Happy", "funny"
Example:
Using LowerCaseFilterFactory before filtering for keep words, no ignoreCase flag.






In: "Happy, sad or funny"
Tokenizer to Filter: "Happy", "sad", "or", "funny"
Filter to Filter: "happy", "sad", "or", "funny"
Out: "happy", "funny"

KStem Filter
KStem is an alternative to the Porter Stem Filter for developers looking for a less aggressive stemmer. KStem was
written by Bob Krovetz, ported to Lucene by Sergio Guzman-Lara (UMASS Amherst). This stemmer is only
appropriate for English language text.
Factory class: solr.KStemFilterFactory
Arguments: None
Example:





In: "jump jumping jumped"
Tokenizer to Filter: "jump", "jumping", "jumped"
Out: "jump", "jump", "jump"

Length Filter
This filter passes tokens whose length falls within the min/max limit specified. All other tokens are discarded.
Factory class: solr.LengthFilterFactory
Arguments:
min: (integer, required) Minimum token length. Tokens shorter than this are discarded.
max: (integer, required, must be >= min) Maximum token length. Tokens longer than this are discarded.

Apache Solr Reference Guide 4.10

105

Example:





In: "turn right at Albuquerque"
Tokenizer to Filter: "turn", "right", "at", "Albuquerque"
Out: "turn", "right"

Lower Case Filter
Converts any uppercase letters in a token to the equivalent lowercase token. All other characters are left
unchanged.
Factory class: solr.LowerCaseFilterFactory
Arguments: None
Example:





In: "Down With CamelCase"
Tokenizer to Filter: "Down", "With", "CamelCase"
Out: "down", "with", "camelcase"

Managed Stop Filter
This is specialized version of the Stop Words Filter Factory that uses a set of stop words that are managed from a
REST API.
Arguments:
managed: The name that should be used for this set of stop words in the managed REST API.
Example:
With this configuration the set of words is named "english" and can be managed via /solr/[collection]/sche
ma/analysis/stopwords/english





See Stop Filter for example input/output.

Apache Solr Reference Guide 4.10

106

Managed Synonym Filter
This is specialized version of the Synonym Filter Factory that uses a mapping on synonyms that is managed from a
REST API.
Arguments:
managed: The name that should be used for this mapping on synonyms in the managed REST API.
Example:
With this configuration the set of mappings is named "english" and can be managed via /solr/[collection]/sc
hema/analysis/synonyms/english





See Synonym Filter for example input/output.

N-Gram Filter
Generates n-gram tokens of sizes in the given range. Note that tokens are ordered by position and then by gram
size.
Factory class: solr.NGramFilterFactory
Arguments:
minGramSize: (integer, default 1) The minimum gram size.
maxGramSize: (integer, default 2) The maximum gram size.
Example:
Default behavior.





In: "four score"
Tokenizer to Filter: "four", "score"
Out: "f", "o", "u", "r", "fo", "ou", "ur", "s", "c", "o", "r", "e", "sc", "co", "or", "re"
Example:
A range of 1 to 4.

Apache Solr Reference Guide 4.10

107






In: "four score"
Tokenizer to Filter: "four", "score"
Out: "f", "fo", "fou", "four", "s", "sc", "sco", "scor"
Example:
A range of 3 to 5.





In: "four score"
Tokenizer to Filter: "four", "score"
Out: "fou", "four", "our", "sco", "scor", "score", "cor", "core", "ore"

Numeric Payload Token Filter
This filter adds a numeric floating point payload value to tokens that match a given type. Refer to the Javadoc for the
org.apache.lucene.analysis.Token class for more information about token types and payloads.
Factory class: solr.NumericPayloadTokenFilterFactory
Arguments:
payload: (required) A floating point value that will be added to all matching tokens.
typeMatch: (required) A token type name string. Tokens with a matching type name will have their payload set to
the above floating point value.
Example:





In: "bing bang boom"
Tokenizer to Filter: "bing", "bang", "boom"
Out: "bing"[0.75], "bang"[0.75], "boom"[0.75]

Pattern Replace Filter

Apache Solr Reference Guide 4.10

108

This filter applies a regular expression to each token and, for those that match, substitutes the given replacement
string in place of the matched pattern. Tokens which do not match are passed though unchanged.
Factory class: solr.PatternReplaceFilterFactory
Arguments:
pattern: (required) The regular expression to test against each token, as per java.util.regex.Pattern.
replacement: (required) A string to substitute in place of the matched pattern. This string may contain references
to capture groups in the regex pattern. See the Javadoc for java.util.regex.Matcher.
replace: ("all" or "first", default "all") Indicates whether all occurrences of the pattern in the token should be
replaced, or only the first.
Example:
Simple string replace:





In: "cat concatenate catycat"
Tokenizer to Filter: "cat", "concatenate", "catycat"
Out: "dog", "condogenate", "dogydog"
Example:
String replacement, first occurrence only:





In: "cat concatenate catycat"
Tokenizer to Filter: "cat", "concatenate", "catycat"
Out: "dog", "condogenate", "dogycat"
Example:
More complex pattern with capture group reference in the replacement. Tokens that start with non-numeric
characters and end with digits will have an underscore inserted before the numbers. Otherwise the token is passed
through.

Apache Solr Reference Guide 4.10

109






In: "cat foo1234 9987 blah1234foo"
Tokenizer to Filter: "cat", "foo1234", "9987", "blah1234foo"
Out: "cat", "foo_1234", "9987", "blah1234foo"

Phonetic Filter
This filter creates tokens using one of the phonetic encoding algorithms in the org.apache.commons.codec.lang
uage package.
Factory class: solr.PhoneticFilterFactory
Arguments:
encoder: (required) The name of the encoder to use. The encoder name must be one of the following (case
insensitive): "DoubleMetaphone", "Metaphone", "Soundex", "RefinedSoundex", "Caverphone", or "ColognePhonetic"
inject: (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens are
replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact spelling of
the target word may not match.
maxCodeLength: (integer) The maximum length of the code to be generated by the Metaphone or Double
Metaphone encoders.
Example:
Default behavior for DoubleMetaphone encoding.





In: "four score and twenty"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "twenty"(4)
Out: "four"(1), "FR"(1), "score"(2), "SKR"(2), "and"(3), "ANT"(3), "twenty"(4), "TNT"(4)
The phonetic tokens have a position increment of 0, which indicates that they are at the same position as the token
they were derived from (immediately preceding).
Example:
Discard original token.

Apache Solr Reference Guide 4.10

110






In: "four score and twenty"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "twenty"(4)
Out: "FR"(1), "SKR"(2), "ANT"(3), "TWNT"(4)
Example:
Default Soundex encoder.





In: "four score and twenty"
Tokenizer to Filter: "four"(1), "score"(2), "and"(3), "twenty"(4)
Out: "four"(1), "F600"(1), "score"(2), "S600"(2), "and"(3), "A530"(3), "twenty"(4), "T530"(4)

Porter Stem Filter
This filter applies the Porter Stemming Algorithm for English. The results are similar to using the Snowball Porter
Stemmer with the language="English" argument. But this stemmer is coded directly in Java and is not based on
Snowball. It does not accept a list of protected words and is only appropriate for English language text. However, it
has been benchmarked as four times faster than the English Snowball stemmer, so can provide a performance
enhancement.
Factory class: solr.PorterStemFilterFactory
Arguments: None
Example:





In: "jump jumping jumped"
Tokenizer to Filter: "jump", "jumping", "jumped"
Out: "jump", "jump", "jump"

Position Filter Factory
This filter sets the position increment values of all tokens in a token stream except the first, which retains its original

Apache Solr Reference Guide 4.10

111

position increment value. This filter has been deprecated and will be removed in Solr 5.
Factory class: solr.PositionIncrementFilterFactory
Arguments:
positionIncrement: (integer, default = 0) The position increment value to apply to all tokens in a token stream
except the first.
Example:





In: "hello world"
Tokenizer to Filter: "hello", "world"
Out: "hello" (token position 1), "world" (token position 2)

Remove Duplicates Token Filter
The filter removes duplicate tokens in the stream. Tokens are considered to be duplicates if they have the same text
and position values.
Factory class: solr.RemoveDuplicatesTokenFilterFactory
Arguments: None
Example:
One example of where RemoveDuplicatesTokenFilterFactory is in situations where a synonym file is being
used in conjuntion with a stemmer causes some synonyms to be reduced to the same stem. Consider the following
entry from a synonyms.txt file:

Television, Televisions, TV, TVs

When used in the following configuration:







In: "Watch TV"
Tokenizer to Synonym Filter: "Watch"(1) "TV"(2)
Synonym Filter to Stem Filter: "Watch"(1) "Television"(2) "Televisions"(2) "TV"(2) "TVs"(2)

Apache Solr Reference Guide 4.10

112

Stem Filter to Remove Dups Filter: "Watch"(1) "Television"(2) "Television"(2) "TV"(2) "TV"(2)
Out: "Watch"(1) "Television"(2) "TV"(2)

Reversed Wildcard Filter
This filter reverses tokens to provide faster leading wildcard and prefix queries. Tokens without wildcards are not
reversed.
Factory class: solr.ReveresedWildcardFilterFactory
Arguments:
withOriginal (boolean) If true, the filter produces both original and reversed tokens at the same positions. If
false, produces only reversed tokens.
maxPosAsterisk (integer, default = 2) The maximum position of the asterisk wildcard ('*') that triggers the reversal
of the query term. Terms with asterisks at positions above this value are not reversed.
maxPosQuestion (integer, default = 1) The maximum position of the question mark wildcard ('?') that triggers the
reversal of query term. To reverse only pure suffix queries (queries with a single leading asterisk), set this to 0 and m
axPosAsterisk to 1.
maxFractionAsterisk (float, default = 0.0) An additional parameter that triggers the reversal if asterisk ('*')
position is less than this fraction of the query token length.
minTrailing (integer, default = 2) The minimum number of trailing characters in a query token after the last
wildcard character. For good performance this should be set to a value larger than 1.
Example:





In: "*foo *bar"
Tokenizer to Filter: "*foo", "*bar"
Out: "oof*", "rab*"

Shingle Filter
This filter constructs shingles, which are token n-grams, from the token stream. It combines runs of tokens into a
single token.
Factory class: solr.ShingleFilterFactory
Arguments:
minShingleSize: (integer, default 2) The minimum number of tokens per shingle.
maxShingleSize: (integer, must be >= 2, default 2) The maximum number of tokens per shingle.
outputUnigrams: (true/false) If true (the default), then each individual token is also included at its original position.

Apache Solr Reference Guide 4.10

113

outputUnigramsIfNoShingles: (true/false) If false (the default), then individual tokens will be output if no
shingles are possible.
tokenSeparator: (string, default is " ") The default string to use when joining adjacent tokens to form a shingle.
Example:
Default behavior.





In: "To be, or what?"
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4)
Out: "To"(1), "To be"(1), "be"(2), "be or"(2), "or"(3), "or what"(3), "what"(4)
Example:
A shingle size of four, do not include original token.





In: "To be, or not to be."
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "not"(4), "to"(5), "be"(6)
Out: "To be"(1), "To be or"(1), "To be or not"(1), "be or"(2), "be or not"(2), "be or not to"(2), "or not"(3), "or not to"(3),
"or not to be"(3), "not to"(4), "not to be"(4), "to be"(5)

Snowball Porter Stemmer Filter
This filter factory instantiates a language-specific stemmer generated by Snowball. Snowball is a software package
that generates pattern-based word stemmers. This type of stemmer is not as accurate as a table-based stemmer,
but is faster and less complex. Table-driven stemmers are labor intensive to create and maintain and so are typically
commercial products.
Solr contains Snowball stemmers for Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German,
Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish. For more
information on Snowball, visit http://snowball.tartarus.org/.
StopFilterFactory, CommonGramsFilterFactory, and CommonGramsQueryFilterFactory can optionally
read stopwords in Snowball format (specify format="snowball" in the configuration of those FilterFactories).
Factory class: solr.SnowballPorterFilterFactory
Arguments:
language: (default "English") The name of a language, used to select the appropriate Porter stemmer to use. Case

Apache Solr Reference Guide 4.10

114

is significant. This string is used to select a package name in the "org.tartarus.snowball.ext" class hierarchy.
protected: Path of a text file containing a list of protected words, one per line. Protected words will not be
stemmed. Blank lines and lines that begin with "#" are ignored. This may be an absolute path, or a simple file name
in the Solr config directory.
Example:
Default behavior:





In: "flip flipped flipping"
Tokenizer to Filter: "flip", "flipped", "flipping"
Out: "flip", "flip", "flip"
Example:
French stemmer, English words:





In: "flip flipped flipping"
Tokenizer to Filter: "flip", "flipped", "flipping"
Out: "flip", "flipped", "flipping"
Example:
Spanish stemmer, Spanish words:





In: "cante canta"
Tokenizer to Filter: "cante", "canta"
Out: "cant", "cant"

Standard Filter
This filter removes dots from acronyms and the substring "'s" from the end of tokens. This filter depends on the
tokens being tagged with the appropriate term-type to recognize acronyms and words with apostrophes.
Factory class: solr.StandardFilterFactory

Apache Solr Reference Guide 4.10

115

Arguments: None
This filter is no longer operational in Solr when the luceneMatchVersion (in solrconfig.xml) is higher
than "3.1".

Stop Filter
This filter discards, or stops analysis of, tokens that are on the given stop words list. A standard stop words list is
included in the Solr config directory, named stopwords.txt, which is appropriate for typical English language text.
Factory class: solr.StopFilterFactory
Arguments:
words: (optional) The path to a file that contains a list of stop words, one per line. Blank lines and lines that begin
with "#" are ignored. This may be an absolute path, or path relative to the Solr config directory.
format: (optional) If the stopwords list has been formatted for Snowball, you can specify format="snowball" so
Solr can read the stopwords file.
ignoreCase: (true/false, default false) Ignore case when testing for stop words. If true, the stop list should contain
lowercase words.
As of Solr 4.4, the enablePositionIncrements argument is no longer supported.
Example:
Case-sensitive matching, capitalized words not stopped. Token positions skip stopped words.





In: "To be or what?"
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4)
Out: "To"(1), "what"(4)
Example:





In: "To be or what?"
Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4)
Out: "what"(4)

Synonym Filter

Apache Solr Reference Guide 4.10

116

This filter does synonym mapping. Each token is looked up in the list of synonyms and if a match is found, then the
synonym is emitted in place of the token. The position value of the new tokens are set such they all occur at the
same position as the original token.
Factory class: solr.SynonymFilterFactory
Arguments:
synonyms: (required) The path of a file that contains a list of synonyms, one per line. Blank lines and lines that
begin with "#" are ignored. This may be an absolute path, or path relative to the Solr config directory.There are two
ways to specify synonym mappings:
A comma-separated list of words. If the token matches any of the words, then all the words in the list are
substituted, which will include the original token.
Two comma-separated lists of words with the symbol "=>" between them. If the token matches any word on
the left, then the list on the right is substituted. The original token will not be included unless it is also in the
list on the right.
For the following examples, assume a synonyms file named mysynonyms.txt:
couch,sofa,divan
teh => the
huge,ginormous,humungous => large
small => tiny,teeny,weeny

Example:





In: "teh small couch"
Tokenizer to Filter: "teh"(1), "small"(2), "couch"(3)
Out: "the"(1), "tiny"(2), "teeny"(2), "weeny"(2), "couch"(3), "sofa"(3), "divan"(3)
Example:





In: "teh ginormous, humungous sofa"
Tokenizer to Filter: "teh"(1), "ginormous"(2), "humungous"(3), "sofa"(4)
Out: "the"(1), "large"(2), "large"(3), "couch"(4), "sofa"(4), "divan"(4)

Token Offset Payload Filter
This filter adds the numeric character offsets of the token as a payload value for that token.

Apache Solr Reference Guide 4.10

117

Factory class: solr.TokenOffsetPayloadTokenFilterFactory
Arguments: None
Example:





In: "bing bang boom"
Tokenizer to Filter: "bing", "bang", "boom"
Out: "bing"[0,4], "bang"[5,9], "boom"[10,14]

Trim Filter
This filter trims leading and/or trailing whitespace from tokens. Most tokenizers break tokens at whitespace, so this
filter is most often used for special situations.
Factory class: solr.TrimFilterFactory
Arguments: None
As of Solr 4.4, the updateOffsets argument is no longer supported.
Example:
The PatternTokenizerFactory configuration used here splits the input on simple commas, it does not remove
whitespace.





In: "one, two , three ,four "
Tokenizer to Filter: "one", " two ", " three ", "four "
Out: "one", "two", "three", "four"

Type As Payload Filter
This filter adds the token's type, as an encoded byte sequence, as its payload.
Factory class: solr.TypeAsPayloadTokenFilterFactory
Arguments: None
Example:

Apache Solr Reference Guide 4.10

118






In: "Pay Bob's I.O.U."
Tokenizer to Filter: "Pay", "Bob's", "I.O.U."
Out: "Pay"[], "Bob's"[], "I.O.U."[]

Type Token Filter
This filter blacklists or whitelists a specified list of token types, assuming the tokens have type metadata associated
with them. For example, the UAX29 URL Email Tokenizer emits "" and "" typed tokens, as well as
other types. This filter would allow you to pull out only e-mail addresses from text as tokens, if you wish.
Factory class: solr.TypeTokenFilterFactory
Arguments:
types: Defines the location of a file of types to filter.
useWhitelist: If true, the file defined in types should be used as include list. If false, or undefined, the file
defined in types is used as a blacklist.
As of Solr 4.4, the enablePositionIncrements argument is no longer supported.
Example:




Word Delimiter Filter
This filter splits tokens at word delimiters. The rules for determining delimiters are determined as follows:
A change in case within a word: "CamelCase" -> "Camel", "Case". This can be disabled by setting splitOnC
aseChange="0".
A transition from alpha to numeric characters or vice versa: "Gonzo5000" -> "Gonzo", "5000" "4500XL" -> "45
00", "XL". This can be disabled by setting splitOnNumerics="0".
Non-alphanumeric characters (discarded): "hot-spot" -> "hot", "spot"
A trailing "'s" is removed: "O'Reilly's" -> "O", "Reilly"
Any leading or trailing delimiters are discarded: "--hot-spot--" -> "hot", "spot"
Factory class: solr.WordDelimiterFilterFactory
Arguments:

Apache Solr Reference Guide 4.10

119

generateWordParts: (integer, default 1) If non-zero, splits words at delimiters. For example:"CamelCase",
"hot-spot" -> "Camel", "Case", "hot", "spot"
generateNumberParts: (integer, default 1) If non-zero, splits numeric strings at delimiters:"1947-32" ->"1947",
"32"
splitOnCaseChange: (integer, default 1) If 0, words are not split on camel-case changes:"BugBlaster-XL" -> "Bug
Blaster", "XL". Example 1 below illustrates the default (non-zero) splitting behavior.
splitOnNumerics: (integer, default 1) If 0, don't split words on transitions from alpha to numeric:"FemBot3000" ->
"Fem", "Bot3000"
catenateWords: (integer, default 0) If non-zero, maximal runs of word parts will be joined: "hot-spot-sensor's" -> "h
otspotsensor"
catenateNumbers: (integer, default 0) If non-zero, maximal runs of number parts will be joined: 1947-32" -> "1947
32"
catenateAll: (0/1, default 0) If non-zero, runs of word and number parts will be joined: "Zap-Master-9000" -> "Zap
Master9000"
preserveOriginal: (integer, default 0) If non-zero, the original token is preserved: "Zap-Master-9000" -> "Zap-Ma
ster-9000", "Zap", "Master", "9000"
protected: (optional) The pathname of a file that contains a list of protected words that should be passed through
without splitting.
stemEnglishPossessive: (integer, default 1) If 1, strips the possessive "'s" from each subword.
Example:
Default behavior. The whitespace tokenizer is used here to preserve non-alphanumeric characters.





In: "hot-spot RoboBlaster/9000 100XL"
Tokenizer to Filter: "hot-spot", "RoboBlaster/9000", "100XL"
Out: "hot", "spot", "Robo", "Blaster", "9000", "100", "XL"
Example:
Do not split on case changes, and do not generate number parts. Note that by not generating number parts, tokens
containing only numeric parts are ultimately discarded.





In: "hot-spot RoboBlaster/9000 100-42"

Apache Solr Reference Guide 4.10

120

Tokenizer to Filter: "hot-spot", "RoboBlaster/9000", "100-42"
Out: "hot", "spot", "RoboBlaster", "9000"
Example:
Concatenate word parts and number parts, but not word and number parts that occur in the same token.





In: "hot-spot 100+42 XL40"
Tokenizer to Filter: "hot-spot"(1), "100+42"(2), "XL40"(3)
Out: "hot"(1), "spot"(2), "hotspot"(2), "100"(3), "42"(4), "10042"(4), "XL"(5), "40"(6)
Example:
Concatenate all. Word and/or number parts are joined together.





In: "XL-4000/ES"
Tokenizer to Filter: "XL-4000/ES"(1)
Out: "XL"(1), "4000"(2), "ES"(3), "XL4000ES"(3)
Example:
Using a protected words list that contains "AstroBlaster" and "XL-5000" (among others).





In: "FooBar AstroBlaster XL-5000 ==ES-34-"
Tokenizer to Filter: "FooBar", "AstroBlaster", "XL-5000", "==ES-34-"
Out: "FooBar", "FooBar", "AstroBlaster", "XL-5000", "ES", "34"

Related Topics
TokenFilterFactories

CharFilterFactories
Char Filter is a component that pre-processes input
characters. Char Filters can be chained like Token Filters

Apache Solr Reference Guide 4.10

121

and placed in front of a Tokenizer. Char Filters can add,
change, or remove characters while preserving the
original character offsets to support features like
highlighting.

Topics discussed in this section:
solr.MappingCharFilterFactory
solr.HTMLStripCharFilterFactory
solr.ICUNormalizer2CharFilterFactory
solr.PatternReplaceCharFilterFactory
Related Topics

solr.MappingCharFilterFactory
This filter creates org.apache.lucene.analysis.MappingCharFilter, which can be used for changing one
character to another (for example, for normalizing é to e.).
This filter requires specifying a mapping argument, which is the path and name of a file containing the mappings to
perform.
Example:



[...]


solr.HTMLStripCharFilterFactory
This filter creates org.apache.solr.analysis.HTMLStripCharFilter. This Char Filter strips HTML

from the input stream and passes the result to another Char Filter or a Tokenizer.
This filter:
Removes HTML/XML tags while preserving other content.
Removes attributes within tags and supports optional attribute quoting.
Removes XML processing instructions, such as: 
Removes XML comments.
Removes XML elements starting with .
Removes contents of '); -->

hello

if a

hello

a


[...]


solr.PatternReplaceCharFilterFactory
This filter uses regular expressions to replace or change character patterns.
Arguments:
pattern: the regular expression pattern to apply to the incoming text.
replacement: the text to use to replace matching patterns.
You can configure this filter in schema.xml like this:

Apache Solr Reference Guide 4.10

123




[...]


The table below presents examples of regex-based pattern replacement:
Input

pattern

replacement

Output

Description

see-ing looking

(\w+)(ing)

$1

see-ing look

Removes "ing" from the end of
word.

see-ing looking

(\w+)ing

$1

see-ing look

Same as above. 2nd
parentheses can be omitted.

No.1 NO. no.
543

[nN][oO]\.\s*(\d+)

#$1

#1 NO. #543

Replace some string literals

abc=1234=5678

(\w+)=(\d+)=(\d+)

$3=$1=$2

5678=abc=1234

Change the order of the groups.

Related Topics
CharFilterFactories

Language Analysis
This section contains information about tokenizers and filters
related to character set conversion or for use with specific
languages. For the European languages, tokenization is fairly
straightforward. Tokens are delimited by white space and/or a
relatively small set of punctuation characters. In other languages
the tokenization rules are often not so simple. Some European
languages may require special tokenization rules as well, such
as rules for decompounding German words.
For information about language detection at index time, see Dete
cting Languages During Indexing.

Topics discussed in this section:
KeyWordMarkerFilterFactory
StemmerOverrideFilterFactory
Dictionary Compound Word
Token Filter
Unicode Collation
ASCII Folding Filter
Language-Specific Factories
Related Topics

KeyWordMarkerFilterFactory
Protects words from being modified by stemmers. A customized protected word list may be specified with the
"protected" attribute in the schema. Any words in the protected word list will not be modified by any stemmer in Solr.
A sample Solr protwords.txt with comments can be found in the /solr/conf/ directory:

Apache Solr Reference Guide 4.10

124









StemmerOverrideFilterFactory
Overrides stemming algorithms by applying a custom mapping, then protecting these terms from being modified by
stemmers.
A customized mapping of words to stems, in a tab-separated file, can be specified to the "dictionary" attribute in the
schema. Words in this mapping will be stemmed to the stems from the file, and will not be further changed by any
stemmer.
A sample stemdict.txt with comments can be found in the Source Repository.








Dictionary Compound Word Token Filter
This filter splits, or decompounds, compound words into individual words using a dictionary of the component words.
Each input token is passed through unchanged. If it can also be decompounded into subwords, each subword is
also added to the stream at the same logical position.
Compound words are most commonly found in Germanic languages.
Factory class: solr.DictionaryCompoundWordTokenFilterFactory
Arguments:
dictionary: (required) The path of a file that contains a list of simple words, one per line. Blank lines and lines
that begin with "#" are ignored. This path may be an absolute path, or path relative to the Solr config directory.
minWordSize: (integer, default 5) Any token shorter than this is not decompounded.
minSubwordSize: (integer, default 2) Subwords shorter than this are not emitted as tokens.
maxSubwordSize: (integer, default 15) Subwords longer than this are not emitted as tokens.
onlyLongestMatch: (true/false) If true (the default), only the longest matching subwords will generate new tokens.
Example:
Assume that germanwords.txt contains at least the following words: dumm kopf donau dampf schiff

Apache Solr Reference Guide 4.10

125






In: "Donaudampfschiff dummkopf"
Tokenizer to Filter: "Donaudampfschiff"(1), "dummkopf"(2),
Out: "Donaudampfschiff"(1), "Donau"(1), "dampf"(1), "schiff"(1), "dummkopf"(2), "dumm"(2), "kopf"(2)

Unicode Collation
Unicode Collation is a language-sensitive method of sorting text that can also be used for advanced search
purposes.
Unicode Collation in Solr is fast, because all the work is done at index time.
Rather than specifying an analyzer within , the solr.Collatio
nField and solr.ICUCollationField field type classes provide this functionality. solr.ICUCollationFiel
d, which is backed by the ICU4J library, provides more flexible configuration, has more locales, is significantly faster,
and requires less memory and less index space, since its keys are smaller than those produced by the JDK
implementation that backs solr.CollationField.
solr.ICUCollationField is included in the Solr analysis-extras contrib - see solr/contrib/analysis
-extras/README.txt for instructions on which jars you need to add to your SOLR_HOME/lib in order to use it.
CollationKeyFilterFactory and ICUCollationKeyFilterFactory are deprecated token filter
implementations of the same functionality as solr.CollationField and solr.ICUCollationField,
respectively. These classes will no longer be available in Solr 5.0.
solr.ICUCollationField and solr.CollationField fields can be created in two ways:
Based upon a system collator associated with a Locale.
Based upon a tailored RuleBasedCollator ruleset.
Arguments for solr.ICUCollationField, specified as attributes within the  element:
Using a System collator:
locale: (required) RFC 3066 locale ID. See the ICU locale explorer for a list of supported locales.
strength: Valid values are primary, secondary, tertiary, quaternary, or identical. See Comparison
Levels in ICU Collation Concepts for more information.
decomposition: Valid values are no or canonical. See Normalization in ICU Collation Concepts for more
information.
Using a Tailored ruleset:
custom: (required) Path to a UTF-8 text file containing rules supported by the ICU RuleBasedCollator
strength: Valid values are primary, secondary, tertiary, quaternary, or identical. See Comparison

Apache Solr Reference Guide 4.10

126

Levels in ICU Collation Concepts for more information.
decomposition: Valid values are no or canonical. See Normalization in ICU Collation Concepts for more
information.
Expert options:
alternate: Valid values are shifted or non-ignorable. Can be used to ignore punctuation/whitespace.
caseLevel: (true/false) If true, in combination with strength="primary", accents are ignored but case is taken
into account. The default is false. See CaseLevel in ICU Collation Concepts for more information.
caseFirst: Valid values are lower or upper. Useful to control which is sorted first when case is not ignored.
numeric: (true/false) If true, digits are sorted according to numeric value, e.g. foobar-9 sorts before foobar-10. The
default is false.
variableTop: Single character or contraction. Controls what is variable for alternate
Sorting Text for a Specific Language

In this example, text is sorted according to the default German rules provided by ICU4J.
Locales are typically defined as a combination of language and country, but you can specify just the language if you
want. For example, if you specify "de" as the language, you will get sorting that works well for the German language.
If you specify "de" as the language and "CH" as the country, you will get German sorting specifically tailored for
Switzerland.


...


...



In the example above, we defined the strength as "primary". The strength of the collation determines how strict the
sort order will be, but it also depends upon the language. For example, in English, "primary" strength ignores
differences in case and accents.
Another example:

Apache Solr Reference Guide 4.10

127


...

...

...


The type will be used for the fields where the data contains Polish text. The "secondary" strength will ignore case
differences, but, unlike "primary" strength, a letter with diacritic(s) will be sorted differently from the same base letter
without diacritics.
An example using the "city_sort" field to sort:
q=*:*&fl=city&sort=city_sort+asc

Sorting Text for Multiple Languages

There are two approaches to supporting multiple languages: if there is a small list of languages you wish to support,
consider defining collated fields for each language and using copyField. However, adding a large number of sort
fields can increase disk and indexing costs. An alternative approach is to use the Unicode default collator.
The Unicode default or ROOT locale has rules that are designed to work well for most languages. To use the defa
ult locale, simply define the locale as the empty string. This Unicode default sort is still significantly more advanced
than the standard Solr sort.


Sorting Text with Custom Rules

You can define your own set of sorting rules. It's easiest to take existing rules that are close to what you want and
customize them.
In the example below, we create a custom rule set for German called DIN 5007-2. This rule set treats umlauts in
German differently: it treats ö as equivalent to oe, ä as equivalent to ae, and ü as equivalent to ue. For more
information, see the ICU RuleBasedCollator javadocs.
This example shows how to create a custom rule set for solr.ICUCollationField and dump it to a file:

Apache Solr Reference Guide 4.10

128

// get the default rules for Germany
// these are called DIN 5007-1 sorting
RuleBasedCollator baseCollator = (RuleBasedCollator) Collator.getInstance(new
ULocale("de", "DE"));
// define some tailorings, to make it DIN 5007-2 sorting.
// For example, this makes ö equivalent to oe
String DIN5007_2_tailorings =
"& ae , a\u0308 & AE , A\u0308"+
"& oe , o\u0308 & OE , O\u0308"+
"& ue , u\u0308 & UE , u\u0308";
// concatenate the default rules to the tailorings, and dump it to a String
RuleBasedCollator tailoredCollator = new RuleBasedCollator(baseCollator.getRules() +
DIN5007_2_tailorings);
String tailoredRules = tailoredCollator.getRules();
// write these to a file, be sure to use UTF-8 encoding!!!
FileOutputStream os = new FileOutputStream(new
File("/solr_home/conf/customRules.dat"));
IOUtils.write(tailoredRules, os, "UTF-8");

This rule set can now be used for custom collation in Solr:


JDK Collation

As mentioned above, ICU Unicode Collation is better in several ways than JDK Collation, but if you cannot use
ICU4J for some reason, you can use solr.CollationField.
The principles of JDK Collation are the same as those of ICU Collation; you just specify language, country and v
ariant arguments instead of the combined locale argument.
Arguments for solr.CollationField, specified as attributes within the  element:
Using a System collator (see Oracle's list of locales supported in Java 7):
language: (required) ISO-639 language code
country: ISO-3166 country code
variant: Vendor or browser-specific code
strength: Valid values are primary, secondary, tertiary or identical. See Oracle Java 7 Collator
javadocs for more information.
decomposition: Valid values are no, canonical, or full. See Oracle Java 7 Collator javadocs for more
information.
Using a Tailored ruleset:
custom: (required) Path to a UTF-8 text file containing rules supported by the JDK RuleBasedCollator

Apache Solr Reference Guide 4.10

129

strength: Valid values are primary, secondary, tertiary or identical. See Oracle Java 7 Collator
javadocs for more information.
decomposition: Valid values are no, canonical, or full. See Oracle Java 7 Collator javadocs for more
information.
A solr.CollationField example:
 
...

...


ASCII Folding Filter
This filter converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII
characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists. Only those characters with
reasonable ASCII alternatives are converted:
This can increase recall by causing more matches. On the other hand, it can reduce precision because
language-specific character differences may be lost.
Factory class: solr.ASCIIFoldingFilterFactory
Arguments: None
Example:





In: "Björn Ångström"
Tokenizer to Filter: "Björn", "Ångström"
Out: "Bjorn", "Angstrom"

Language-Specific Factories
These factories are each designed to work with specific languages. The languages covered here are:
Arabic
Danish
Hindi
Polish
Brazilian
Dutch
Indonesian
Portuguese
Portuguese
Finnish
Italian
Romanian
Bulgarian
French
Irish
Russian
Catalan
Galician
Kuromoji
Spanish
Chinese
German
(Japanese)
Swedish
Simplified
Greek
Latvian
Thai
Chinese
Hebrew,
Norwegian
Turkish

Apache Solr Reference Guide 4.10

130

CJK
Czech

Lao,
Myanmar,
Khmer

Persian

Arabic

Solr provides support for the Light-10 (PDF) stemming algorithm, and Lucene includes an example stopword list.
This algorithm defines both character normalization and stemming, so these are split into two filters to provide more
flexibility.
Factory classes: solr.ArabicStemFilterFactory, solr.ArabicNormalizationFilterFactory
Arguments: None
Example:






Brazilian Portuguese

This is a Java filter written specifically for stemming the Brazilian dialect of the Portuguese language. It uses the
Lucene class org.apache.lucene.analysis.br.BrazilianStemmer. Although that stemmer can be
configured to use a list of protected words (which should not be stemmed), this factory does not accept any
arguments to specify such a list.
Factory class: solr.BrazilianStemFilterFactory
Arguments: None
Example:





In: "praia praias"
Tokenizer to Filter: "praia", "praias"
Out: "pra", "pra"
Bulgarian

Solr includes a light stemmer for Bulgarian, following this algorithm (PDF), and Lucene includes an example
stopword list.
Factory class: solr.BulgarianStemFilterFactory
Arguments: None
Example:

Apache Solr Reference Guide 4.10

131







Catalan

Solr can stem Catalan using the Snowball Porter Stemmer with an argument of language="Catalan". Solr
includes a set of contractions for Catalan, which can be stripped using solr.ElisionFilterFactory.
Factory class: solr.SnowballPorterFilterFactory
Arguments:
language: (required) stemmer language, "Catalan" in this case
Example:







In: "llengües llengua"
Tokenizer to Filter: "llengües"(1) "llengua"(2),
Out: "llengu"(1), "llengu"(2)
Chinese
Chinese Tokenizer

The Chinese Tokenizer is deprecated as of Solr 3.4. Use the solr.StandardTokenizerFactory instead.
Factory class: solr.ChineseTokenizerFactory
Arguments: None
Example:




Chinese Filter Factory

The Chinese Filter Factory is deprecated as of Solr 3.4. Use the solr.StopFilterFactory instead.
Factory class: solr.ChineseFilterFactory
Arguments: None

Apache Solr Reference Guide 4.10

132

Example:





Simplified Chinese

For Simplified Chinese, Solr provides support for Chinese sentence and word segmentation with the solr.HMMChi
neseTokenizerFactory in the analysis-extras contrib module. This component includes a large dictionary
and segments Chinese text into words with the Hidden Markov Model. To use this filter, see solr/contrib/anal
ysis-extras/README.txt for instructions on which jars you need to add to your solr_home/lib.
Factory class: solr.HMMChineseTokenizerFactory
Arguments: None
Examples:
To use the default setup with fallback to English Porter stemmer for English words, use:

Or to configure your own analysis setup, use the solr.HMMChineseTokenizerFactory along with your custom
filter setup.






CJK

This tokenizer breaks Chinese, Japanese and Korean language text into tokens. These are not whitespace delimited
languages. The tokens generated by this tokenizer are "doubles", overlapping pairs of CJK characters found in the
field text.
Factory class: solr.CJKTokenizerFactory
Arguments: None
Example:




Czech

Solr includes a light stemmer for Czech, following this algorithm, and Lucene includes an example stopword list.
Factory class: solr.CzechStemFilterFactory

Apache Solr Reference Guide 4.10

133

Arguments: None
Example:






In: "prezidenští, prezidenta, prezidentského"
Tokenizer to Filter: "prezidenští", "prezidenta", "prezidentského"
Out: "preziden", "preziden", "preziden"
Danish

Solr can stem Danish using the Snowball Porter Stemmer with an argument of language="Danish".
Factory class: solr.SnowballPorterFilterFactory
Arguments:
language: (required) stemmer language, "Danish" in this case
Example:






In: "undersøg undersøgelse"
Tokenizer to Filter: "undersøg"(1) "undersøgelse"(2),
Out: "undersøg"(1), "undersøg"(2)
Dutch

This is a Java filter written specifically for stemming the Dutch language. It uses the Lucene class org.apache.lu
cene.analysis.nl.DutchStemmer. Although that stemmer can be configured to use a list of protected words
(which should not be stemmed), this factory does not accept any arguments to specify such a list.
Another option for stemming Dutch words is to use the Snowball Porter Stemmer with an argument of language="
Dutch".
Factory class: solr.DutchStemFilterFactory
Arguments: None
Example:

Apache Solr Reference Guide 4.10

134






In: "kanaal kanalen"
Tokenizer to Filter: "kanaal", "kanalen"
Out: "kanal", "kanal"
Finnish

Solr includes support for stemming Finnish, and Lucene includes an example stopword list.
Factory class: solr.FinnishLightStemFilterFactory
Arguments: None
Example:





In: "kala kalat"
Tokenizer to Filter: "kala", "kalat"
Out: "kala", "kala"
French
Elision Filter

Removes article elisions from a token stream. This filter can be useful for languages such as French, Catalan,
Italian, and Irish.
Factory class: solr.ElisionFilterFactory
Arguments:
articles: The pathname of a file that contains a list of articles, one per line, to be stripped. Articles are words such
as "le", which are commonly abbreviated, such as in l'avion (the plane). This file should include the abbreviated
form, which precedes the apostrophe. In this case, simply "l". If no articles attribute is specified, a default set of
French articles is used.
ignoreCase: (boolean) If true, the filter ignores the case of words when comparing them to the common word file.
Defaults to false
Example:

Apache Solr Reference Guide 4.10

135






In: "L'histoire d'art"
Tokenizer to Filter: "L'histoire", "d'art"
Out: "histoire", "art"
French Light Stem Filter

Solr includes three stemmers for French: one in the solr.SnowballPorterFilterFactory, a lighter stemmer
called solr.FrenchLightStemFilterFactory, and an even less aggressive stemmer called solr.FrenchMi
nimalStemFilterFactory. Lucene includes an example stopword list.
Factory classes: solr.FrenchLightStemFilterFactory, solr.FrenchMinimalStemFilterFactory
Arguments: None
Examples:














In: "le chat, les chats"
Tokenizer to Filter: "le", "chat", "les", "chats"
Out: "le", "chat", "le", "chat"
Galician

Solr includes a stemmer for Galician following this algorithm, and Lucene includes an example stopword list.
Factory class: solr.GalicianStemFilterFactory
Arguments: None
Example:

Apache Solr Reference Guide 4.10

136







In: "felizmente Luzes"
Tokenizer to Filter: "felizmente", "luzes"
Out: "feliz", "luz"
German

Solr includes four stemmers for German: one in the solr.SnowballPorterFilterFactory
language="German", a stemmer called solr.GermanStemFilterFactory, a lighter stemmer called solr.Ge
rmanLightStemFilterFactory, and an even less aggressive stemmer called solr.GermanMinimalStemFil
terFactory. Lucene includes an example stopword list.
Factory classes: solr.GermanStemFilterFactory, solr.LightGermanStemFilterFactory, solr.Mini
malGermanStemFilterFactory
Arguments: None
Examples:















In: "hund hunden"
Tokenizer to Filter: "hund", "hunden"
Out: "hund", "hund"
Greek

This filter converts uppercase letters in the Greek character set to the equivalent lowercase character.
Factory class: solr.GreekLowerCaseFilterFactory
Arguments: None

Apache Solr Reference Guide 4.10

137

Use of custom charsets is not longer supported as of Solr 3.1. If you need to index text in these encodings,
please use Java's character set conversion facilities (InputStreamReader, and so on.) during I/O, so that
Lucene can analyze this text as Unicode instead.
Example:





Hindi

Solr includes support for stemming Hindi following this algorithm (PDF), support for common spelling differences
through the solr.HindiNormalizationFilterFactory, support for encoding differences through the solr.I
ndicNormalizationFilterFactory following this algorithm, and Lucene includes an example stopword list.
Factory classes: solr.IndicNormalizationFilterFactory, solr.HindiNormalizationFilterFactor
y, solr.HindiStemFilterFactory
Arguments: None
Example:







Indonesian

Solr includes support for stemming Indonesian (Bahasa Indonesia) following this algorithm (PDF), and Lucene
includes an example stopword list.
Factory class: solr.IndonesianStemFilterFactory
Arguments: None
Example:






In: "sebagai sebagainya"
Tokenizer to Filter: "sebagai", "sebagainya"
Out: "bagai", "bagai"

Apache Solr Reference Guide 4.10

138

Italian

Solr includes two stemmers for Italian: one in the solr.SnowballPorterFilterFactory
language="Italian", and a lighter stemmer called solr.ItalianLightStemFilterFactory. Lucene
includes an example stopword list.
Factory class: solr.ItalianStemFilterFactory
Arguments: None
Example:







In: "propaga propagare propagamento"
Tokenizer to Filter: "propaga", "propagare", "propagamento"
Out: "propag", "propag", "propag"
Irish

Solr can stem Irish using the Snowball Porter Stemmer with an argument of language="Irish". Solr includes so
lr.IrishLowerCaseFilter, which can handle Irish-specific constructs. Solr also includes a set of contractions
for Irish which can be stripped using solr.ElisionFilterFactory.
Factory class: solr.SnowballPorterFilterFactory
Arguments:
language: (required) stemmer language, "Irish" in this case
Example:







In: "siopadóireacht síceapatacha b'fhearr m'athair"
Tokenizer to Filter: "siopadóireacht", "síceapatacha", "b'fhearr", "m'athair"
Out: "siopadóir", "síceapaite", "fearr", "athair"
Kuromoji (Japanese)

Solr includes support for stemming Kuromoji (Japanese), and Lucene includes an example stopword list. Kuromoji

Apache Solr Reference Guide 4.10

139

has a search mode (default) that does segmentation useful for search. A heuristic is used to segment compounds
into its parts and the compound itself is kept as a synonym.
With Solr 4, the JapaneseIterationMarkCharFilterFactory now is included to normalize Japanese iteration
marks.
You can also make discarding punctuation configurable in the JapaneseTokenizerFactory, by setting discard
Punctuation to false (to show punctuation) or true (to discard punctuation).
Factory class: solr.KuromojiStemFilterFactory
Arguments:
mode: Use search-mode to get a noun-decompounding effect useful for search. Search mode improves
segmentation for search at the expense of part-of-speech accuracy. Valid values for mode are:
normal: default segmentation
search: segmentation useful for search (extra compound splitting)
extended: search mode with unigramming of unknown words (experimental)
For some applications it might be good to use search mode for indexing and normal mode for queries to reduce
recall and prevent parts of compounds from being matched and highlighted.
Kuromoji also has a convenient user dictionary feature that allows overriding the statistical model with your own
entries for segmentation, part-of-speech tags and readings without a need to specify weights. Note that user
dictionaries have not been subject to extensive testing. User dictionary attributes are:
userDictionary: user dictionary filename
userDictionaryEncoding: user dictionary encoding (default is UTF-8)
See lang/userdict_ja.txt for a sample user dictionary file.
Punctuation characters are discarded by default. Use discardPunctuation="false" to keep them.
Example:












Hebrew, Lao, Myanmar, Khmer

Lucene provides support, in addition to UAX#29 word break rules, for Hebrew's use of the double and single quote
characters, and for segmenting Lao, Myanmar, and Khmer into syllables with the solr.ICUTokenizerFactory in

Apache Solr Reference Guide 4.10

140

the analysis-extras contrib module. To use this tokenizer, see solr/contrib/analysis-extras/README.
txt for instructions on which jars you need to add to your solr_home/lib.
See the ICUTokenizer for more information.
Latvian

Solr includes support for stemming Latvian, and Lucene includes an example stopword list.
Factory class: solr.LatvianStemFilterFactory
Arguments: None
Example:








In: "tirgiem tirgus"
Tokenizer to Filter: "tirgiem", "tirgus"
Out: "tirg", "tirg"
Norwegian

Solr includes two classes for stemming Norwegian, NorwegianLightStemFilterFactory and NorwegianMini
malStemFilterFactory. Lucene includes an example stopword list.
Another option is to use the Snowball Porter Stemmer with an argument of language="Norwegian".
Norwegian Light Stemmer

The NorwegianLightStemFilterFactory requires a "two-pass" sort for the -dom and -het endings. This means
that in the first pass the word "kristendom" is stemmed to "kristen", and then all the general rules apply so it will be
further stemmed to "krist". The effect of this is that "kristen," "kristendom," "kristendommen," and "kristendommens"
will all be stemmed to "krist."
The second pass is to pick up -dom and -het endings. Consider this example:
One pass

Two passes

Before

After

Before

After

forlegen

forleg

forlegen

forleg

forlegenhet

forlegen

forlegenhet

forleg

forlegenheten

forlegen

forlegenheten

forleg

forlegenhetens

forlegen

forlegenhetens

forleg

Apache Solr Reference Guide 4.10

141

firkantet

firkant

firkantet

firkant

firkantethet

firkantet

firkantethet

firkant

firkantetheten

firkantet

firkantetheten

firkant

Factory class: solr.NorwegianLightStemFilterFactory
Arguments: variant: Choose the Norwegian language variant to use. Valid values are:
nb: Bokmål (default)
nn: Nynorsk
no: both

Example:









In: "Forelskelsen"
Tokenizer to Filter: "forelskelsen"
Out: "forelske"
Norwegian Minimal Stemmer

The NorwegianMinimalStemFilterFactory stems plural forms of Norwegian nouns only.
Factory class: solr.NorwegianMinimalStemFilterFactory
Arguments: variant: Choose the Norwegian language variant to use. Valid values are:
nb: Bokmål (default)
nn: Nynorsk
no: both

Example:









Apache Solr Reference Guide 4.10

142

In: "Bilens"
Tokenizer to Filter: "bilens"
Out: "bil"
Persian
Persian Filter Factories

Solr includes support for normalizing Persian, and Lucene includes an example stopword list.
Factory class: solr.PersianNormalizationFilterFactory
Arguments: None
Example:






Polish

Solr provides support for Polish stemming with the solr.StempelPolishStemFilterFactory, and solr.Morp
hologikFilterFactory for lemmatization, in the contrib/analysis-extras module. The solr.StempelP
olishStemFilterFactory component includes an algorithmic stemmer with tables for Polish. To use either of
these filters, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to add
to your solr_home/lib.
Factory class: solr.StempelPolishStemFilterFactory and solr.MorfologikFilterFactory
Arguments: None
Example:












In: ""studenta studenci"
Tokenizer to Filter: "studenta", "studenci"
Out: "student", "student"

Apache Solr Reference Guide 4.10

143

More information about the Stempel stemmer is available in the Lucene javadocs.
The Morfologik dictionary-resource param value is a constant specifying which dictionary to choose.
The dictionary resource must be named morfologik/dictionaries/{dictionaryResource}.dict and
have an associated .info metadata file. See the Morfologik project for details.

Portuguese

Solr includes four stemmers for Portuguese: one in the solr.SnowballPorterFilterFactory, an alternative
stemmer called solr.PortugueseStemFilterFactory, a lighter stemmer called solr.PortugueseLightSt
emFilterFactory, and an even less aggressive stemmer called solr.PortugueseMinimalStemFilterFact
ory. Lucene includes an example stopword list.
Factory classes: solr.PortugueseStemFilterFactory, solr.PortugueseLightStemFilterFactory, s
olr.PortugueseMinimalStemFilterFactory
Arguments: None
Example:


















In: "praia praias"
Tokenizer to Filter: "praia", "praias"
Out: "pra", "pra"
Romanian

Solr can stem Romanian using the Snowball Porter Stemmer with an argument of language="Romanian".
Factory class: solr.SnowballPorterFilterFactory
Arguments:
language: (required) stemmer language, "Romanian" in this case

Apache Solr Reference Guide 4.10

144

Example:






Russian
Russian Stem Filter

Solr includes two stemmers for Russian: one in the solr.SnowballPorterFilterFactory
language="Russian", and a lighter stemmer called solr.RussianLightStemFilterFactory. Lucene
includes an example stopword list.
Factory class: solr.RussianLightStemFilterFactory
Arguments: None
Use of custom charsets is no longer supported as of Solr 3.4. If you need to index text in these encodings,
please use Java's character set conversion facilities (InputStreamReader, and so on.) during I/O, so that
Lucene can analyze this text as Unicode instead.
Example:






Spanish

Solr includes two stemmers for Spanish: one in the solr.SnowballPorterFilterFactory
language="Spanish", and a lighter stemmer called solr.SpanishLightStemFilterFactory. Lucene
includes an example stopword list.
Factory class: solr.SpanishStemFilterFactory
Arguments: None
Example:






In: "torear toreara torearlo"
Tokenizer to Filter: "torear", "toreara", "torearlo"

Apache Solr Reference Guide 4.10

145

Out: "tor", "tor", "tor"
Swedish
Swedish Stem Filter

Solr includes two stemmers for Swedish: one in the solr.SnowballPorterFilterFactory
language="Swedish", and a lighter stemmer called solr.SwedishLightStemFilterFactory. Lucene
includes an example stopword list.
Factory class: solr.SwedishStemFilterFactory
Arguments: None
Example:






In: "kloke klokhet klokheten"
Tokenizer to Filter: "kloke", "klokhet", "klokheten"
Out: "klok", "klok", "klok"
Thai

This filter converts sequences of Thai characters into individual Thai words. Unlike European languages, Thai does
not use whitespace to delimit words.
Factory class: solr.ThaiTokenizerFactory
Arguments: None
Example:





Turkish

Solr includes support for stemming Turkish through the solr.SnowballPorterFilterFactory; support for
case-insensitive search through the solr.TurkishLowerCaseFilterFactory; support for stripping
apostrophes and following suffixes through solr.ApostropheFilterFactory (see Role of Apostrophes in
Turkish Information Retrieval); support for a form of stemming that truncating tokens at a configurable maximum
length through the solr.TruncateTokenFilterFactory (see Information Retrieval on Turkish Texts); and Lucene
includes an example stopword list.
Factory class: solr.TurkishLowerCaseFilterFactory
Arguments: None

Apache Solr Reference Guide 4.10

146

Example:







Another example, illustrating diacritics-insensitive search:










Related Topics
LanguageAnalysis

Phonetic Matching
Introduced with Solr v3.6, Beider-Morse Phonetic Matching (BMPM) is a "soundalike" tool that lets you search using
a new phonetic matching system. BMPM helps you search for personal names (or just surnames) in a Solr/Lucene
index, and is far superior to the existing phonetic codecs, such as regular soundex, metaphone, caverphone, etc.
In general, phonetic matching lets you search a name list for names that are phonetically equivalent to the desired
name. BMPM is similar to a soundex search in that an exact spelling is not required. Unlike soundex, it does not
generate a large quantity of false hits.
From the spelling of the name, BMPM attempts to determine the language. It then applies phonetic rules for that
particular language to transliterate the name into a phonetic alphabet. If it is not possible to determine the language
with a fair degree of certainty, it uses generic phonetic instead. Finally, it applies language-independent rules
regarding such things as voiced and unvoiced consonants and vowels to further insure the reliability of the matches.
For example, assume that the matches found when searching for Stephen in a database are "Stefan", "Steph",
"Stephen", "Steve", "Steven", "Stove", and "Stuffin". "Stefan", "Stephen", and "Steven" are probably relevant, and
are names that you want to see. "Stuffin", however, is probably not relevant. Also rejected were "Steph", "Steve",
and "Stove". Of those, "Stove" is probably not one that we would have wanted. But "Steph" and "Steve" are possibly
ones that you might be interested in.
For Solr, BMPM searching is available for the following languages:
English
Lithuanian and Latvian
French
Polish
German
Romanian
Greek
Russian written in Cyrillic letters
Hebrew written in Hebrew letters
Russian transliterated into English letters
Hungarian
Spanish
Italian
Turkish

Apache Solr Reference Guide 4.10

147

The name matching is also applicable to non-Jewish surnames from the countries in which those languages are
spoken.
For more information, see here: http://stevemorse.org/phoneticinfo.htm and http://stevemorse.org/phonetics/bmpm.h
tm.

Running Your Analyzer
Once you've defined a field type in schema.xml and specified the analysis steps that you want applied to it, you
should test it out to make sure that it behaves the way you expect it to. Luckily, there is a very handy page in the
Solr admin interface that lets you do just that. You can invoke the analyzer for any text field, provide sample input,
and display the resulting token stream.
For example, assume that the following field type definition has been added to schema.xml:












The objective here (during indexing) is to reconstruct hyphenated words, which may have been split across lines in
the text, then to set all words to lowercase. For queries, you want to skip the de-hyphenation step.
To test this out, point your browser at the Analysis Screen of the Solr Admin Web interface. By default, this will be at
the following URL (adjust the hostname and/or port to match your configuration): http://localhost:8983/solr/#/collectio
n1/analysis. You should see a page like this.

Empty Analysis screen
We want to test the field type definition for "mytextfield", defined above. The drop-down labeled "Analyse

Apache Solr Reference Guide 4.10

148

Fieldname/FieldType" allows choosing the field or field type to use for the analysis.
There are two "Field Value" boxes, one for how text will be analyzed during indexing and a second for how text will
be analyzed for query processing. In the "Field Value (Index)" box enter some sample text "Super-computer" in this
example) to be processed by the analyzer. We will leave the query field value empty for now.
The result we expect is that HyphenatedWordsFilter will join the hyphenated pair "Super-" and "computer" into
the single word "Supercomputer", and then LowerCaseFilter will set it to "supercomputer". Let's see what
happens:

Running index-time analyzer, verbose output.
The result is two distinct tokens rather than the one we expected. What went wrong? Looking at the first token that
came out of StandardTokenizer, we can see the trailing hyphen has been stripped off of "Super-". Checking the
documentation for StandardTokenizer, we see that it treats all punctuation characters as delimiters and discards
them. What we really want in this case is a whitespace tokenizer that will preserve the hyphen character when it
breaks the text into tokens.
Let's make this change and try again:












Re-submitting the form by clicking "Analyse Values" again, we see the result in the screen shot below.

Apache Solr Reference Guide 4.10

149

Using WhitespaceTokenizer, expected results.
That's more like it. Because the whitespace tokenizer preserved the trailing hyphen on the first token, Hyphenated
WordsFilter was able to reconstruct the hyphenated word, which then passed it on to LowerCaseFilter, where
capital letters are set to lowercase.
Now let's see what happens when invoking the analyzer for query processing. For query terms, we don't want to do
de-hyphenation and we do want to discard punctuation, so let's try the same input on it. We'll copy the same text to
the "Field Value (Query)" box and clear the one for index analysis. We'll also include the full, unhyphenated word as
another term to make sure it is processed to lower case as we expect. Submitting again yields these results:

Query-time analyzer, good results.
We can see that for queries the analyzer behaves the way we want it to. Punctuation is stripped out, HyphenatedW
ordsFilter doesn't run, and we wind up with the three tokens we expected.

Apache Solr Reference Guide 4.10

150

Indexing and Basic Data Operations
This section describes how Solr adds data to its index. It covers the following topics:
Introduction to Solr Indexing: An overview of
Updating Parts of Documents: Information about
Solr's indexing process.
how to use atomic updates and optimistic
concurrency with Solr.
Simple Post Tool: Information about using post.
Detecting Languages During Indexing:
jar to quickly upload some content to your
Information about using language identification
system.
during the indexing process.
Uploading Data with Index Handlers: Information
De-Duplication: Information about configuring Solr
about using Solr's Index Handlers to upload
to mark duplicate documents as they are indexed.
XML/XSLT, JSON and CSV data.
Uploading Data with Solr Cell using Apache
Tika: Information about using the Solr Cell
framework to upload data for indexing.
Uploading Structured Data Store Data with the
Data Import Handler: Information about uploading
and indexing data from a structured data store.

Content Streams: Information about streaming
content to Solr Request Handlers.
UIMA Integration: Information about integrating
Solr with Apache's Unstructured Information
Management Architecture (UIMA). UIMA lets you
define custom pipelines of Analysis Engines that
incrementally add metadata to your documents as
annotations.

Indexing Using Client APIs
Using client APIs, such as SolrJ, from your applications is an important option for updating Solr indexes. See the Cli
ent APIs section for more information.

Introduction to Solr Indexing
This section describes the process of indexing: adding content to a Solr index and, if necessary, modifying that
content or deleting it. By adding content to an index, we make it searchable by Solr.
A Solr index can accept data from many different sources, including XML files, comma-separated value (CSV) files,
data extracted from tables in a database, and files in common file formats such as Microsoft Word or PDF.
Here are the three most common ways of loading data into a Solr index:
Using the Solr Cell framework built on Apache Tika for ingesting binary files or structured files such as Office,
Word, PDF, and other proprietary formats.
Uploading XML files by sending HTTP requests to the Solr server from any environment where such requests
can be generated.
Writing a custom Java application to ingest data through Solr's Java Client API (which is described in more
detail in Client APIs. Using the Java API may be the best choice if you're working with an application, such as
a Content Management System (CMS), that offers a Java API.
Regardless of the method used to ingest data, there is a common basic data structure for data being fed into a Solr
index: a document containing multiple fields, each with a name and containing content, which may be empty. One of
the fields is usually designated as a unique ID field (analogous to a primary key in a database), although the use of
a unique ID field is not strictly required by Solr.

Apache Solr Reference Guide 4.10

151

If the field name is defined in the schema.xml file that is associated with the index, then the analysis steps
associated with that field will be applied to its content when the content is tokenized. Fields that are not explicitly
defined in the schema will either be ignored or mapped to a dynamic field definition (see Documents, Fields, and
Schema Design), if one matching the field name exists.
For more information on indexing in Solr, see the Solr Wiki.

The Solr Example Directory
The example/ directory includes a sample Solr implementation, along with sample documents for uploading into an
index. You will find the example docs in $SOLR_HOME/example/exampledocs.

The curl Utility for Transferring Files
Many of the instructions and examples in this section make use of the curl utility for transferring content through a
URL. curl posts and retrieves data over HTTP, FTP, and many other protocols. Most Linux distributions include a
copy of curl. You'll find curl downloads for Linux, Windows, and many other operating systems at http://curl.haxx.s
e/download.html. Documentation for curl is available here: http://curl.haxx.se/docs/manpage.html.
Using curl or other command line tools for posting data is just fine for examples or tests, but it's not the
recommended method for achieving the best performance for updates in production environments. You will
achieve better performance with Solr Cell or the other methods described in this section.
Instead of curl, you can use utilities such as GNU wget (http://www.gnu.org/software/wget/) or manage
GETs and POSTS with Perl, although the command line options will differ.

Simple Post Tool
Solr includes a simple command line tool for POSTing raw XML to a Solr port. XML data can be read from files
specified as command line arguments, as raw commandline argument strings, or via STDIN.
The tool is called post.jar and is found in the 'exampledocs' directory: $SOLR/example/exampledocs/post.
jar includes a cross-platform Java tool for POST-ing XML documents.
To run it, open a window and enter:
java -jar post.jar 

By default, this will contact the server at localhost:8983. The '-help' (or simply '-h' option will output information
on its usage (i.e., java -jar post.jar -help.

Using the Simple Post Tool
Options controlled by System Properties include the Solr URL to post to, the Content-Type of the data, whether a
commit or optimize should be executed, and whether the response should be written to STDOUT. You may override
any other request parameter through the -Dparams property.
This table lists the supported system properties and their defaults:
Parameter

Values

Apache Solr Reference Guide 4.10

Default

Description

152

-Ddata

args, stdin, files, web

files

Use args to pass
arguments along the
command line (such as a
command to delete a
document). Use files to
pass a filename or regex
pattern indicating paths
and filenames. Use stdin
to use standard input.
Use web for a very
simple web crawler
(arguments for this would
be the URL to crawl).

-Dtype



application/xml

Defines the content-type,
if -Dauto is not used.

-Durl



http://localhost:8983/solr/update

The Solr URL to send
the updates to.

-Dauto

yes, no

no

If yes, the tool will guess
the file type from file
name suffix, and set type
and url accordingly. It
also sets the ID and file
name automatically.

-Drecursive

yes, no

no

Will recurse into
sub-folders and index all
files.

-Dfiletypes

[,,..]

xml, json, csv, pdf, doc, docx,
ppt, pptx, xls, xlsx, odt, odp,
ods, rtf, htm, html

Specifies the file types to
consider when indexing
folders.

-Dparams

"=[&=...]"

none

HTTP GET params to
add to the request, so
you don't need to write
the whole URL again.
Values must be
URL-encoded.

-Dcommit

yes, no

yes

Perform a commit after
adding the documents.

-Doptimize

yes, no

no

Perform an optimize after
adding the documents.

-Dout

yes, no

no

Write the response to an
output file.

Examples

Apache Solr Reference Guide 4.10

153

There are several ways to use post.jar. Here are a few examples:
Add all documents with file extension .xml.
java -jar post.jar *.xml

Send XML arguments to delete a document from the index.
java -Ddata=args -jar post.jar '42'

Index all CSV files.
java -Dtype=text/csv -jar post.jar *.csv

Index all JSON files.
java -Dtype=application/json -jar post.jar *.json

Use the extracting request handler to index a PDF file.
java -Durl=[http://localhost:8983/solr/update/extract] -Dparams=literal.id=a
-Dtype=application/pdf -jar post.jar a.pdf

Automatically detect the content type based on the file extension.
java -Dauto=yes -jar post.jar a.pdf

Automatically detect content types in a folder, and recursively scan it for documents.
java -Dauto=yes -Drecursive=yes -jar post.jar afolder

Automatically detect content types in a folder, but limit it to PPT and HTML files.
java -Dauto=yes -Dfiletypes=ppt,html -jar post.jar afolder

Uploading Data with Index Handlers
Index Handlers are Request Handlers designed to add, delete and
update documents to the index. In addition to having plugins for
importing rich documents using Tika or from structured data sources
using the Data Import Handler, Solr natively supports indexing
structured documents in XML, CSV and JSON.
The recommended way to configure & use request handlers is with path
based names, that map to paths in the request url - but request
handlers can also be specified with the qt(query type) parameter if the
requestDispatcher is apprpriately configured.
The example URLs given here reflect the handler configuration in the

Apache Solr Reference Guide 4.10

154

supplied solrconfig.xml. If the name associated with the handler is
changed then the URLs will need to be modified. It is possible to access
the same handler using more than one name, which can be useful if you
wish to specify different sets of default options.

Topics covered in this section:
UpdateRequestHandler
Configuration
XML Formatted Index
Updates
JSON Formatted Index
Updates
CSV Formatted Index
Updates
Nested Child
Documents

The Combined UpdateRequestHandler
Prior to Solr 4, uploading content with an update request handler required declaring a unique request handler for the
format of the content in the request. Now, there is a unified update request handler that supports XML, CSV, JSON,
and javabin update requests, delegating to the appropriate ContentStreamLoader based on the Content-Type
of the ContentStream.
UpdateRequestHandler Configuration

The default configuration file has the update request handler configured by default.


XML Formatted Index Updates

Index update commands can be sent as XML message to the update handler using Content-type:
application/xml or Content-type: text/xml.
Adding Documents

The XML schema recognized by the update handler for adding documents is very straightforward:
The  element introduces one more documents to be added.
The  element introduces the fields making up a document.
The  element presents the content for a specific field.
For example:

Apache Solr Reference Guide 4.10

155



Patrick Eagar
Sports
796.35
128

12.40
Summer of the all-rounder: Test and championship
cricket in England 1982
0002166313
1982
Collins


...



Each element has certain optional attributes which may be specified.
Command

Optional
Parameter

Parameter Description



commitWithin=
number

Add the document within the specified number of milliseconds



overwrite=bool
ean

Default is true. Indicates if the unique key constraints should be checked to
overwrite previous versions of the same document (see below)



boost=float

Default is 1.0. Sets a boost value for the document.To learn more about boosting,
see Searching.



boost=float

Default is 1.0. Sets a boost value for the field.

If the document schema defines a unique key, then by default an /update operation to add a document will
overwrite (ie: replace) any document in the index with the same unique key. If no unique key has been defined,
indexing performance is somewhat faster, as no check has to be made for an existing documents to replace.
If you have a unique key field, but you feel confident that you can safely bypass the uniqueness check (eg: you build
your indexes in batch, and your indexing code guarantees it never adds the same document more then once) you
can specify the {{overwrite="false"} option when adding your documents.
Commit and Optimize Operations

The  operation writes all documents loaded since the last commit to one or more segment files on the
disk. Before a commit has been issued, newly indexed content is not visible to searches. The commit operation
opens a new searcher, and triggers any event listeners that have been configured.
Commits may be issued explicitly with a  message, and can also be triggered from  par
ameters in solrconfig.xml.
The  operation requests Solr to merge internal data structures in order to improve search performance.
For a large index, optimization will take some time to complete, but by merging many small segment files into a

Apache Solr Reference Guide 4.10

156

larger one, search performance will improve. If you are using Solr's replication mechanism to distribute searches
across many systems, be aware that after an optimize, a complete index will need to be transferred. In contrast,
post-commit transfers are usually much smaller.
The  and  elements accept these optional attributes:
Optional
Attribute

Description

waitSearcher

Default is true. Blocks until a new searcher is opened and registered as the main query
searcher, making the changes visible.

expungeDeletes

(commit only) Default is false. Merges segments that have more than 10% deleted docs,
expunging them in the process.

maxSegments

(optimize only) Default is 1. Merges the segments down to no more than this number of
segments.

Here are examples of  and  using optional attributes:




Delete Operations

Documents can be deleted from the index in two ways. "Delete by ID" deletes the document with the specified ID,
and can be used only if a UniqueID field has been defined in the schema. "Delete by Query" deletes all documents
matching a specified query, although commitWithin is ignored for a Delete by Query. A single delete message can
contain multiple delete operations.

0002166313
0031745983
subject:sport
publisher:penguin


Rollback Operations

The rollback command rolls back all add and deletes made to the index since the last commit. It neither calls any
event listeners nor creates a new searcher. Its syntax is simple: .
Using curl to Perform Updates with the Update Request Handler.

You can use the curl utility to perform any of the above commands, using its --data-binary option to append
the XML message to the curl command, and generating a HTTP POST request. For example:

Apache Solr Reference Guide 4.10

157

curl http://localhost:8983/update -H "Content-Type: text/xml" --data-binary '


Patrick Eagar
Sports
796.35
0002166313
1982
Collins

'

For posting XML messages contained in a file, you can use the alternative form:
curl http://localhost:8983/update -H "Content-Type: text/xml" --data-binary
@myfile.xml

Short requests can also be sent using a HTTP GET command, URL-encoding the request, as in the following. Note
the escaping of "<" and ">":
curl http://localhost:8983/update?stream.body=%3Ccommit/%3E

Responses from Solr take the form shown here:



0
127



The status field will be non-zero in case of failure. The servlet container will generate an appropriate
HTML-formatted message in the case of an error at the HTTP layer.
Using XSLT to Transform XML Index Updates

The UpdateRequestHandler allows you to index any arbitrary XML using the  parameter to apply an XSL
transformation. You must have an XSLT stylesheet in the solr/conf/xslt directory that can transform the incoming
data to the expected  format, and use the tr parameter to specify the name of that
stylesheet.
Here is an example XSLT stylesheet:

Apache Solr Reference Guide 4.10

158






















This stylesheet transforms Solr's XML search result format into Solr's Update XML syntax. One example is to copy a
Solr1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file
(provided that all fields are stored):
http://localhost:8983/solr/select?q=*:*&wt=xslt&tr=updateXml.xsl&rows=1000

You can also use the stylesheet in XsltUpdateRequestHandler to transform an index when updating:
curl "http://localhost:8983/solr/update?commit=true&tr=updateXml.xsl" -H
"Content-Type: text/xml" --data-binary @myexporteddata.xml

For more information about the XML Update Request Handler, see https://wiki.apache.org/solr/UpdateXmlMessages
.
JSON Formatted Index Updates

JSON formatted update requests may be sent to Solr's /update handler using the Content-Type "application
/json" or "text/json".
JSON formatted updates can take 3 basic forms, described in depth below:
A sequence of update commands, expressed as a top level JSON Object (aka: Map)
A list of documents to add, expressed as a top level JSON Array containing a JSON Object per document
A single document to add, expressed as a top level JSON Object – to differentiate this from a set of
commands, the json.command=false request parameter is required.
Adding a Single JSON Document

The simplest way to add Documents via JSON is to send each document individually as a JSON Object, using the j

Apache Solr Reference Guide 4.10

159

son.command=false request parameter:
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/collection1/update?json.command=false' --data-binary '
{
"id": "1",
"title": "Doc 1"
}'

Adding Multiple JSON Documents

Adding multiple documents at one time via JSON can be done via a JSON Array of JSON Objects, where each
object represents a document:
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/collection1/update' --data-binary '
[
{
"id": "1",
"title": "Doc 1"
},
{
"id": "2",
"title": "Doc 2"
}
]'

A sample JSON file is provided at example/exampledocs/books.json that you can use to add some
documents to the Solr example server using an Array of objects:
cd example/exampledocs
curl 'http://localhost:8983/solr/collection1/update?commit=true'
--data-binary @books.json -H 'Content-type:application/json'

Sending Arbitrary JSON Update Commands

In general, the JSON update syntax supports accepts all of the update commands that the XML update handler
supports, through a straightforward mapping. Multiple commands, adding and deleting documents, may be
contained in one message:

Apache Solr Reference Guide 4.10

160

curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/collection1/update' --data-binary '
{
"add": {
"doc": {
"id": "DOC1",
"my_boosted_field": {
/* use a map with boost/value for a boosted field
*/
"boost": 2.3,
"value": "test"
},
"my_multivalued_field": [ "aaa", "bbb" ]
/* Can use an array for a
multi-valued field */
}
},
"add": {
"commitWithin": 5000,
/* commit this document within 5 seconds */
"overwrite": false,
/* don't check for existing documents with the same
uniqueKey */
"boost": 3.45,
/* a document boost */
"doc": {
"f1": "v1",
/* Can use repeated keys for a multi-valued field
*/
"f1": "v2"
}
},
"commit": {},
"optimize": { "waitSearcher":false },
"delete": { "id":"ID" },
"delete": { "query":"QUERY" }

/* delete by ID */
/* delete by query */

}'

Comments are not allowed in JSON, but duplicate names are.
The comments in the above example are for illustrative purposes only, and can not be included in actual
commands sent to Solr.
As with other update handlers, parameters such as commit, commitWithin, optimize, and overwrite may be
specified in the URL instead of in the body of the message.
The JSON update format allows for a simple delete-by-id. The value of a delete can be an array which contains a
list of zero or more specific document id's (not a range) to be deleted. For example:
{ "delete":"myid" }

{ "delete":["id1","id2"] }

The value of a "delete" can be an array which contains a list of zero or more id's to be deleted. It is not a range (start
and end).
You can also specify _version_ with each "delete":

Apache Solr Reference Guide 4.10

161

{
"delete":"id":50,
"_version_":12345
}

You can specify the version of deletes in the body of the update request as well.
JSON Update Convenience Paths

In addition to the /update handler, there are a few additional JSON specific request handler paths available by
default in Solr, that implicitly override the behavior of some request parameters:
Path

Default Parameters

/update/json

stream.contentType=application/json

/update/json/docs

stream.contentType=application/json
json.command=false

The /update/json path may be useful for clients sending in JSON formatted update commands from applications
where setting the Content-Type proves difficult, while the /update/json/docs path can be particularly convenient
for clients that always want to send in documents – either individually or as a list – with out needing to worry about
the full JSON command syntax.
For more information about the JSON Update Request Handler, see https://wiki.apache.org/solr/UpdateJSON.
CSV Formatted Index Updates

CSV formatted update requests may be sent to Solr's /update handler using Content-Type "application/cs
v" or "text/csv".
A sample CSV file is provided at example/exampledocs/books.csv that you can use to add some documents
to the Solr example server:
cd example/exampledocs
curl 'http://localhost:8983/solr/collection1/update?commit=true'
--data-binary @books.csv -H 'Content-type:application/csv'

CSV Update Parameters

The CSV handler allows the specification of many parameters in the URL in the form: f.parameter.optional_f
ieldname=value.
The table below describes the parameters for the update handler.
Parameter

Usage

Apache Solr Reference Guide 4.10

Global
(g) or
Per
Field
(f)

Example

162

separator

Character used as field separator; default is ","

g,(f:
see
split)

separator=%

trim

If true, remove leading and trailing whitespace from
values. Default=false.

g,f

f.isbn.trim=true
trim=false

header

Set to true if first line of input contains field names.
These will be used if the field_name parameter is
absent.

g

field_name

Comma separated list of field names to use when
adding documents.

g

field_name=isbn,price,title

literal.

Comma separated list of field names to use when
processing literal values.

g

literal.color=red,blue,black

skip

Comma separated list of field names to skip.

g

skip=uninteresting,shoesize

skipLines

Number of lines to discard in the input stream
before the CSV data starts, including the header, if
present. Default=0.

g

skipLines=5

encapsulator

The character optionally used to surround values to
preserve characters such as the CSV separator or
whitespace. This standard CSV format handles the
encapsulator itself appearing in an encapsulated
value by doubling the encapsulator.

g,(f:
see
split)

encapsulator="

escape

The character used for escaping CSV separators or
other reserved characters. If an escape is
specified, the encapsulator is not used unless also
explicitly specified since most formats use either
encapsulation or escaping, not both

g

escape=\

keepEmpty

Keep and index zero length (empty) fields.
Default=false.

g,f

f.price.keepEmpty=true

map

Map one value to another. Format is
value:replacement (which can be empty.)

g,f

map=left:right
f.subject.map=history:bunk

split

If true, split a field into multiple values by a
separate parser.

f

overwrite

If true (the default), check for and overwrite
duplicate documents, based on the uniqueKey field
declared in the Solr schema. If you know the
documents you are indexing do not contain any
duplicates then you may see a considerable speed
up setting this to false.

g

commit

Issues a commit after the data has been ingested.

g

Apache Solr Reference Guide 4.10

163

commitWithin

Add the document within the specified number of
milliseconds.

g

commitWithin=10000

rowid

Map the rowid (line number) to a field specified by
the value of the parameter, for instance if your CSV
doesn't have a unique key and you want to use the
row id as such.

g

rowid=id

rowidOffset

Add the given offset (as an int) to the rowid before
adding it to the document. Default is 0

g

rowidOffset=10

CSV Update Convenience Paths

In addition to the /update handler, there is an additional CSV specific request handler path available by default in
Solr, that implicitly override the behavior of some request parameters:
Path

Default Parameters

/update/csv

stream.contentType=application/csv

The /update/csv path may be useful for clients sending in CSV formatted update commands from applications
where setting the Content-Type proves difficult.
For more information on the CSV Update Request Handler, see https://wiki.apache.org/solr/UpdateCSV.
Nested Child Documents

Solr nested documents using a "Block Join" when indexing as a way to model documents containing other
documents, such as a blog post parent document and comments as child documents -- or products as parent
documents and sizes, colors, or other variations as child documents. At query time, the Block Join Query Parsers c
an be used search against these relationships. In terms of performance, indexing the relationships between
documents may be more efficient than attempting to do joins only at query time, since the relationships are already
stored in the index and do not need to be computed.
Nested documents may be indexed via either the XML or JSON data syntax (or using SolrJ) - but regardless of
syntax, you must include a field that identifies the parent document as a parent; it can be any field that suits this
purpose, and it will be used as input for the block join query parsers.
XML Examples

For example, here are two documents and their child documents:

Apache Solr Reference Guide 4.10

164



1
Solr adds block join support
parentDocument

2
SolrCloud supports it too!



3
Lucene and Solr 4.5 is out
parentDocument

4
Lots of new features




In this example, we have indexed the parent documents with the field content_type, which has the value
"parentDocument". We could have also used a boolean field, such as isParent, with a value of "true", or any other
similar approach.
JSON Examples

This example is equivalent to the XML example above, note the special _childDocuments_ key need to indicate
the nested documents in JSON.
[
{
"id": "1",
"title": "Solr adds block join support",
"content_type": "parentDocument",
"_childDocuments_": [
{
"id": "2",
"comments": "SolrCloud supports it too!"
}
]
},
{
"id": "3",
"title": "Lucene and Solr 4.5 is out",
"content_type": "parentDocument",
"_childDocuments_": [
{
"id": "4",
"comments": "Lots of new features"
}
]
}
]

Apache Solr Reference Guide 4.10

165

Transforming and Indexing custom JSON data

This helps index JSON into a valid Solr document according to the users configuration. It lets the user to split a
single JSON file into 1 or more Solr documents. The final indexed document can be controlled using the mapping
passed along the request . One or more valid JSON documents can be sent to the /update/json/docs path with the
configuration params.
Mapping params

split : This parameter is required if you wish to transform the input JSON . This is the path at which the JSON
must be split . If the entire JSON makes a single solr document , the path must be “/” .
f : This is a multivalued mapping parameter . At least one field mapping must be provided . The format of the
parameter is {target-field-name}:{json-path} . The ‘json-path’ is a required part . target-field-name is the name
of the field in the input Solr document. It is optional and it is automatically derived from the input json
echo : This is for debugging purpose only . set it to true , if you want the docs to be returned as a response.
Nothing will be indexed

example 1:

curl 'http://localhost:8983/solr/collection1/update/json/docs'\
'?split=/exams'\
'&f=first:/first'\
'&f=last:/last'\
'&f=grade:/grade'\
'&f=subject:/exams/subject'\
'&f=test:/exams/test'\
'&f=marks:/exams/marks'\
-H 'Content-type:application/json' -d '
{
"first": "John",
"last": "Doe",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test"
: "term1",
"marks" : 90},
{
"subject": "Biology",
"test"
: "term1",
"marks" : 86}
]
}'

This indexes the following two docs

Apache Solr Reference Guide 4.10

166

{
"first":"John",
"last":"Doe",
"marks":90,
"test":"term1",
"subject":"Maths",
"grade":8
}
{
"first":"John",
"last":"Doe",
"marks":86,
"test":"term1",
"subject":"Biology",
"grade":8
}

As the final field names are the same as the input document fields, the request can be simplified as,
example 2 :
curl 'http://localhost:8983/solr/collection1/update/json/docs'\
'?split=/exams'\
'&f=/first'\
'&f=/last'\
'&f=/grade'\
'&f=/exams/subject'\
'&f=/exams/test'\
'&f=/exams/marks'\
-H 'Content-type:application/json' -d '
{
"first": "John",
"last": "Doe",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test"
: "term1",
"marks" : 90},
{
"subject": "Biology",
"test"
: "term1",
"marks" : 86}
]
}'

Wildcards
Instead of specifying all the field names explicitly, it is possible to specify wildcards to map fields automatically.
There are two restrictions: wildcards can only be used at the end of the json-path; and the split path cannot use
wildcards. A single asterisk "*" maps only to direct children, and a double asterisk "**" maps recursively to all
descendants. The following are example wildcard path mappings:

Apache Solr Reference Guide 4.10

167

f=/docs/* : maps all the fields under docs and in the name as given in json
f=/docs/** : maps all the fields under docs and its children in the name as given in json
f=searchField:/docs/* : maps all fields under /docs to a single field called ‘searchField’
f=searchField:/docs/** : maps all fields under /docs and its children to searchField
With wildcards we can simplify our previous example as follows
example 3:
curl 'http://localhost:8983/solr/collection1/update/json/docs'\
'?split=/exams'\
'&f=/**'\
-H 'Content-type:application/json' -d '
{
"first": "John",
"last": "Doe",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test"
: "term1",
"marks" : 90},
{
"subject": "Biology",
"test"
: "term1",
"marks" : 86}
]
}'

It is also possible to send all the values to a single field and do a full text search on that . This is a good
option to blindly index and query JSON documents without worrying about fields and schema
example 4 :
curl 'http://localhost:8983/solr/collection1/update/json/docs'\
'?split=/'\
'&f=txt:/**'\
-H 'Content-type:application/json' -d '
{
"first": "John",
"last": "Doe",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test"
: "term1",
"marks" : 90},
{
"subject": "Biology",
"test"
: "term1",
"marks" : 86}
]
}'

Apache Solr Reference Guide 4.10

168

Uploading Data with Solr Cell using Apache Tika
Solr uses code from the Apache Tika project to provide a
framework for incorporating many different file-format parsers
such as Apache PDFBox and Apache POI into Solr itself.
Working with this framework, Solr's ExtractingRequestHa
ndler can use Tika to support uploading binary files,
including files in popular formats such as Word and PDF, for
data extraction and indexing.
As of version 4.8, Solr uses Apache Tika v1.5.
When this framework was under development, it was called
the Solr Content Extraction Library or CEL; from that
abbreviation came this framework's name: Solr Cell.
If you want to supply your own ContentHandler for Solr to
use, you can extend the ExtractingRequestHandler and
override the createFactory() method. This factory is
responsible for constructing the SolrContentHandler that

Topics covered in this section:
Key Concepts
Trying out Tika with the Solr
Example Directory
Input Parameters
Order of Operations
Configuring the Solr
ExtractingRequestHandler
Indexing Encrypted Documents
with the
ExtractingUpdateRequestHandler
Examples
Sending Documents to Solr with
a POST
Sending Documents to Solr with
Solr Cell and SolrJ
Related Topics

interacts with Tika, and allows literals to override Tika-parsed
values. Set the parameter literalsOverride, which
normally defaults to *true, to *false to append Tika-parsed
values to literal values.
For more information on Solr's Extracting Request Handler,
see https://wiki.apache.org/solr/ExtractingRequestHandler.

Key Concepts
When using the Solr Cell framework, it is helpful to keep the following in mind:
Tika will automatically attempt to determine the input document type (Word, PDF, HTML) and extract the
content appropriately. If you like, you can explicitly specify a MIME type for Tika with the stream.type para
meter.
Tika works by producing an XHTML stream that it feeds to a SAX ContentHandler. SAX is a common
interface implemented for many different XML parsers. For more information, see http://www.saxproject.org/q
uickstart.html.
Solr then responds to Tika's SAX events and creates the fields to index.
Tika produces metadata such as Title, Subject, and Author according to specifications such as the
DublinCore. See http://tika.apache.org/1.5/formats.html for the file types supported.
Tika adds all the extracted text to the content field. This field is defined as "stored" in schema.xml. It is
also copied to the text field with a copyField rule.
You can map Tika's metadata fields to Solr fields. You can also boost these fields.
You can pass in literals for field values. Literals will override Tika-parsed values, including fields in the Tika
metadata object, the Tika content field, and any "captured content" fields.
You can apply an XPath expression to the Tika XHTML to restrict the content that is produced.

Apache Solr Reference Guide 4.10

169

While Apache Tika is quite powerful, it is not perfect and fails on some files. PDF files are particularly
problematic, mostly due to the PDF format itself. In case of a failure processing any file, the ExtractingRe
questHandler does not have a secondary mechanism to try to extract some text from the file; it will throw
an exception and fail.

Trying out Tika with the Solr Example Directory
You can try out the Tika framework using the example application included in Solr.
Start the Solr example server:
cd example -jar start.jar

In a separate window go to the docs/ directory (which contains some nice example docs), or the site directory if
you built Solr from source, and send Solr a file via HTTP POST:
curl 'http://localhost:8983/solr/update/extract?literal.id=doc1&commit=true' -F
"myfile=@tutorial.html"

The URL above calls the Extraction Request Handler, uploads the file tutorial.html and assigns it the unique ID
doc1. Here's a closer look at the components of this command:
The literal.id=doc1 parameter provides the necessary unique ID for the document being indexed.
The commit=true parameter causes Solr to perform a commit after indexing the document, making it
immediately searchable. For optimum performance when loading many documents, don't call the commit
command until you are done.
The -F flag instructs curl to POST data using the Content-Type multipart/form-data and supports the
uploading of binary files. The @ symbol instructs curl to upload the attached file.
The argument myfile=@tutorial.html needs a valid path, which can be absolute or relative (for
example, myfile=@../../site/tutorial.html if you are still in exampledocs directory).
Now you should be able to execute a query and find that document (open the following link in your browser): http://lo
calhost:8983/solr/select?q=tutorial.
You may notice that although you can search on any of the text in the sample document, you may not be able to see
that text when the document is retrieved. This is simply because the "content" field generated by Tika is mapped to
the Solr field called text, which is indexed but not stored. This operation is controlled by default map rule in the /u
pdate/extract handler in solrconfig.xml, and its behavior can be easily changed or overridden. For
example, to store and see all metadata and content, execute the following:
curl
'http://localhost:8983/solr/update/extract?literal.id=doc1&uprefix=attr_&fmap.content=
attr_content&commit=true' -F "myfile=@tutorial.html"

In this command, the uprefix=attr_ parameter causes all generated fields that aren't defined in the schema to
be prefixed with attr_, which is a dynamic field that is stored.
The fmap.content=attr_content parameter overrides the default fmap.content=text causing the content
to be added to the attr_content field instead.

Apache Solr Reference Guide 4.10

170

Then run this command to query the document: http://localhost:8983/solr/select?q=attr_content:tutorial

Input Parameters
The table below describes the parameters accepted by the Extraction Request Handler.
Parameter

Description

boost.

Boosts the specified field by the defined float amount. (Boosting a field alters its
importance in a query response. To learn about boosting fields, see Searching.)

capture

Captures XHTML elements with the specified name for a supplementary addition to
the Solr document. This parameter can be useful for copying chunks of the XHTML
into a separate field. For instance, it could be used to grab paragraphs ( 
) and
index them into a separate field. Note that content is still also captured into the overall
"content" field.

captureAttr

Indexes attributes of the Tika XHTML elements into separate fields, named after the
element. If set to true, for example, when extracting from HTML, Tika can return the
href attributes in  tags as fields named "a". See the examples below.

commitWithin

Add the document within the specified number of milliseconds.

date.formats

Defines the date format patterns to identify in the documents.

defaultField

If the uprefix parameter (see below) is not specified and a field cannot be determined,
the default field will be used.

extractOnly

Default is false. If true, returns the extracted content from Tika without indexing the
document. This literally includes the extracted XHTML as a string in the response.
When viewing manually, it may be useful to use a response format other than XML to
aid in viewing the embedded XHTML tags.For an example, see http://wiki.apache.org/
solr/TikaExtractOnlyExampleOutput.

extractFormat

Default is "xml", but the other option is "text". Controls the serialization format of the
extract content. The xml format is actually XHTML, the same format that results from
passing the -x command to the Tika command line application, while the text format
is like that produced by Tika's -t command. This parameter is valid only if extractO
nly is set to true.

fmap.

Maps (moves) one field name to another. The source_field must be a field in
incoming documents, and the value is the Solr field to map to. Example: fmap.cont
ent=text causes the data in the content field generated by Tika to be moved to
the Solr's text field.

literal.

Populates a field with the name supplied with the specified value for each document.
The data can be multivalued if the field is multivalued.

literalsOverride

If true (the default), literal field values will override other values with the same field
name. If false, literal values defined with literal. will be appended
to data already in the fields extracted from Tika. If setting literalsOverride to
"false", the field must be multivalued.

Apache Solr Reference Guide 4.10

171

lowernames

Values are "true" or "false". If true, all field names will be mapped to lowercase with
underscores, if needed. For example, "Content-Type" would be mapped to
"content_type."

multipartUploadLimitInKB

Useful if uploading very large documents, this defines the KB size of documents to
allow.

passwordsFile

Defines a file path and name for a file of file name to password mappings.

resource.name

Specifies the optional name of the file. Tika can use it as a hint for detecting a file's
MIME type.

resource.password

Defines a password to use for a password-protected PDF or OOXML file

tika.config

Defines a file path and name to a customized Tika configuration file. This is only
required if you have customized your Tika implementation.

uprefix

Prefixes all fields that are not defined in the schema with the given prefix. This is very
useful when combined with dynamic field definitions. Example: uprefix=ignored_
would effectively ignore all unknown fields generated by Tika given the example
schema contains 

xpath

When extracting, only return Tika XHTML content that satisfies the given XPath
expression. See http://tika.apache.org/1.5/index.html for details on the format of Tika
XHTML. See also http://wiki.apache.org/solr/TikaExtractOnlyExampleOutput.

Order of Operations
Here is the order in which the Solr Cell framework, using the Extraction Request Handler and Tika, processes its
input.
1. Tika generates fields or passes them in as literals specified by literal.=. If liter
alsOverride=false, literals will be appended as multi-value to the Tika-generated field.
2. If lowernames=true, Tika maps fields to lowercase.
3. Tika applies the mapping rules specified by fmap. source = target parameters.
4. If uprefix is specified, any unknown field names are prefixed with that value, else if defaultField is
specified, any unknown fields are copied to the default field.

Configuring the Solr ExtractingRequestHandler
If you are not working in the supplied example/solr directory, you must copy all libraries from example/solr/l
ibs into a libs directory within your own solr directory or to a directory you've specified in solrconfig.xml using
the new libs directive. The ExtractingRequestHandler is not incorporated into the Solr WAR file, so you have
to install it separately.
Here is an example of configuring the ExtractingRequestHandler in solrconfig.xml.

Apache Solr Reference Guide 4.10

172



last_modified
ignored_


/my/path/to/tika.config


yyyy-MM-dd



In the defaults section, we are mapping Tika's Last-Modified Metadata attribute to a field named last_modified.
We are also telling it to ignore undeclared fields. These are all overridden parameters.
The tika.config entry points to a file containing a Tika configuration. The date.formats allows you to specify
various java.text.SimpleDateFormats date formats for working with transforming extracted input to a Date.
Solr comes configured with the following date formats (see the DateUtil in Solr):
yyyy-MM-dd'T'HH:mm:ss'Z'
yyyy-MM-dd'T'HH:mm:ss
yyyy-MM-dd
yyyy-MM-dd hh:mm:ss
yyyy-MM-dd HH:mm:ss
EEE MMM d hh:mm:ss z yyyy
EEE, dd MMM yyyy HH:mm:ss zzz
EEEE, dd-MMM-yy HH:mm:ss zzz
EEE MMM d HH:mm:ss yyyy
You may also need to adjust the multipartUploadLimitInKB attribute as follows if you are submitting very large
documents.


...

Multi-Core Configuration

For a multi-core configuration, specify sharedLib='lib' in the  section of solr.xml in order for Solr to
find the JAR files in example/solr/lib.
For more information about Solr cores, see The Well-Configured Solr Instance.

Indexing Encrypted Documents with the ExtractingUpdateRequestHandler
The ExtractingRequestHandler will decrypt encrypted files and index their content if you supply a password in either
resource.password on the request, or in a passwordsFile file.

Apache Solr Reference Guide 4.10

173

In the case of passwordsFile, the file supplied must be formatted so there is one line per rule. Each rule contains
a file name regular expression, followed by "=", then the password in clear-text. Because the passwords are in
clear-text, the file should have strict access restrictions.
# This is a comment
myFileName = myPassword
.*\.docx$ = myWordPassword
.*\.pdf$ = myPdfPassword

Examples
Metadata

As mentioned before, Tika produces metadata about the document. Metadata describes different aspects of a
document, such as the author's name, the number of pages, the file size, and so on. The metadata produced
depends on the type of document submitted. For instance, PDFs have different metadata than Word documents do.
In addition to Tika's metadata, Solr adds the following metadata (defined in ExtractingMetadataConstants):
Solr Metadata

Description

stream_name

The name of the Content Stream as uploaded to Solr. Depending on how the file is
uploaded, this may or may not be set

stream_source_info

Any source info about the stream. (See the section on Content Streams later in this
section.)

stream_size

The size of the stream in bytes.

stream_content_type

The content type of the stream, if available.

We recommend that you try using the extractOnly option to discover which values Solr is setting for
these metadata elements.
Examples of Uploads Using the Extraction Request Handler
Capture and Mapping

The command below captures 
 tags separately, and then maps all the instances of that field to a dynamic
field named foo_t.
curl
"http://localhost:8983/solr/update/extract?literal.id=doc2&captureAttr=true&defaultFie
ld=text&fmap.div=foo_t&capture=div" -F "tutorial=@tutorial.pdf"

Capture, Mapping, and Boosting

The command below captures 
 tags separately, maps the field to a dynamic field named foo_t, then boosts
foo_t by 3.

Apache Solr Reference Guide 4.10

174

curl
"http://localhost:8983/solr/update/extract?literal.id=doc3&captureAttr=true&defaultFie
ld=text&capture=div&fmap.div=foo_t&boost.foo_t=3" -F "tutorial=@tutorial.pdf"

Using Literals to Define Your Own Metadata

To add in your own metadata, pass in the literal parameter along with the file:
curl
"http://localhost:8983/solr/update/extract?literal.id=doc4&captureAttr=true&defaultFie
ld=text&capture=div&fmap.div=foo_t&boost.foo_t=3&literal.blah_s=Bah" -F
"tutorial=@tutorial.pdf"

XPath

The example below passes in an XPath expression to restrict the XHTML returned by Tika:
curl
"http://localhost:8983/solr/update/extract?literal.id=doc5&captureAttr=true&defaultFie
ld=text&capture=div&fmap.div=foo_t&boost.foo_t=3&literal.id=id&xpath=/xhtml:html/xhtml
:body/xhtml:div/descendant:node()" -F "tutorial=@tutorial.pdf"

Extracting Data without Indexing It

Solr allows you to extract data without indexing. You might want to do this if you're using Solr solely as an extraction
server or if you're interested in testing Solr extraction.
The example below sets the extractOnly=true parameter to extract data without indexing it.
curl "http://localhost:8983/solr/update/extract?&extractOnly=true" --data-binary
@tutorial.html -H 'Content-type:text/html'

The output includes XML generated by Tika (and further escaped by Solr's XML) using a different output format to
make it more readable:
curl "http://localhost:8983/solr/update/extract?&extractOnly=true&wt=ruby&indent=true"
--data-binary @tutorial.html -H 'Content-type:text/html'

Sending Documents to Solr with a POST
The example below streams the file as the body of the POST, which does not, then, provide information to Solr
about the name of the file.
curl "http://localhost:8983/solr/update/extract?literal.id=doc5&defaultField=text"
--data-binary @tutorial.html -H 'Content-type:text/html'

Sending Documents to Solr with Solr Cell and SolrJ
SolrJ is a Java client that you can use to add documents to the index, update the index, or query the index. You'll
find more information on SolrJ in Client APIs.

Apache Solr Reference Guide 4.10

175

Here's an example of using Solr Cell and SolrJ to add documents to a Solr index.
First, let's use SolrJ to create a new SolrServer, then we'll construct a request containing a ContentStream
(essentially a wrapper around a file) and sent it to Solr:
public class SolrCellRequestDemo {
public static void main (String[] args) throws IOException, SolrServerException {
SolrServer server = new HttpSolrServer("http://localhost:8983/solr");
ContentStreamUpdateRequest req = new
ContentStreamUpdateRequest("/update/extract");
req.addFile(new File("apache-solr/site/features.pdf"));
req.setParam(ExtractingParams.EXTRACT_ONLY, "true");
NamedList