Starting Solr In Debug mode

Make sure you imported your project in eclipse, consisting the solr folder and java project. You can maintain the folder structure as per your project requirement.

Create a bat file consisting of the following Contents in to it. And name it as solr-start.bat

cd "<Solr-Installed-Dir>\solr-6.4.2\bin"

solr.cmd start -a "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=18983"

once if you execute the above bat file then the solr server is started in the debug mode .

Configure the Below Setting in the Eclipse

Host:localhost

Port: 18983

As like below.

Make sure that you have Included all the libraries as below.

Put debug points and execute some query through the browser then you will get the debug in your eclipse .

Happy Learning!!!!! 

Delta Updates in Solr

Hi Guys,

We have Learned the ways to do indexing from the mysql database those who missed can read from here, Now it’s the time to learn the Delta Imports in Solr .

edit db-data-config.xml from the conf folder ,

<?xml version="1.0" encoding="UTF-8" ?>

<dataConfig>

<dataSource type="JdbcDataSource"

            driver="com.mysql.jdbc.Driver"

            url="jdbc:mysql://localhost:3306/classicmodels"

            user="root"

            password="root"/>

<document name="classicmodels">

   <entity name="products" query="select * from products" deltaImportQuery="select * from products"

   deltaQuery="select * from products where last_modified > '${dataimporter.last_index_time}'">

     <field column="productCode" name="id"/>

     <field column="productName" name="name"/>

     <field column="productDescription" name="description"/>      

  </entity>

</document>

</dataConfig>

We have already dealt with the  query, now it’s the time to deal with the delta queries now introduce two queries .

deltaImportQuery :this query is used for the importing the data while performing the Delta import, make sure you include all the Fields in deltaImportQuery as like the full import query. If missed any fields solr will through the Run Time Exception.

deltaQuery : this is the query which identifies the Delta Changes . ${dataimporter.last_index_time} will give you te last index time.

Once this changes are done, then it’s the time to index the Delta changes.

http://localhost:8983/solr/refrence/dataimport?command=delta-import

Once you hit the above url , then delta import will be done and indexed. You can see the 

status of the delta update by the Following url.

http://localhost:8983/solr/refrence/dataimport

So it’s the time to see the Delta Changes in the response. Happy Learning !!!!

Integrating Solr 6 with My Sql

Hi Readers,

Good Day, I was doing some couple of tasks for identifying Integration of solr with mysql, the strange part is that solr has changed its schema structure when compared to the older version of solr .I am Writing this tutorial for solr 6 , the output of this is to make the readers to connect to mysql and index and query for the Results .

 Pre-requisites

      

     MySql - any version

     MySql – JDBC Connector

     Sample DatBase Containing any data

Step:1

Create a Core in Solr called Refrence you can see my blog here.

Step:2

Once you create the Core then the Following folder will be Created in the Server

Ie <Solr Installed Dir>solr-6.4.2\server

Navigate to  <Solr Installed Dir>solr-6.4.2\server\solr\refrence

There will be two folders called conf and data created Default, if not create it.

Conf – this folder is used for the Defining the Configs for solr , example what to index and what should take as part of the query.

Data- this folder is used for the Indexing stuff.

Step:3

Edit the File solrconfig.xml  and if this file is not avalaible in the Conf Directory copy from some other examples of solr and paste it .

Add the Following handler in the request handler

  <requestHandler name="/dataimport" class="solr.DataImportHandler">

    <lst name="defaults">

      <str name="config">db-data-config.xml</str>

    </lst>

  </requestHandler>

By Defaut solr does not comes with the dataimport handler , you need to add it externally.

You also need to add the dependency jars in that xml.

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

<lib dir="="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" />

Make sure you copy the mysql jdbc jar in that path eg(cotrib /lib) , otherwise you can create a lib folder inside the refrence and paste the jar.

Step:4

As defined in the conig , you need to have the following file db-data-config.xml  in the conf directory.

Create a file called db-data-config.xml and edit with the below contents.

<?xml version="1.0" encoding="UTF-8" ?>

<dataConfig>

<dataSource type="JdbcDataSource"  driver="com.mysql.jdbc.Driver"  url="jdbc:mysql://localhost:3306/classicmodels"

            user="root"

            password="root"/>

<document name="classicmodels">

   <entity name="products" query="select * from products">

     <field column="productCode" name="id"/>

     <field column="productName" name="name"/>

     <field column="productDescription" name="description"/>      

  </entity>

</document>

</dataConfig>

This config file has the cofigurations for the Database I hope you are familiar with the Database Connections  for mysql

Document name is the DataBase Name .

Entity name is the Table Name.

Query is how to fetch the Data From the Table.

Name is the identification in the Solr.

Once you create mapping and changes done , save it and you need to define this fields , this is the place where the mapping of the database names is done with the field names .

This Steps Differs from the Previous versions of the Solr.

Step:5

You have to declare these Fields to the solr , how you can do it ?

Navigate to managed-schema   File in <SOLR-INSTALLED_DIR>\solr-6.4.2\server\solr\refrence\conf

Check for the field name tag and declare your defined fields also here.

    <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

    <field name="name" type="string" indexed="true" stored="true" required="true" multiValued="false" />

    

    <field name="description" type="string" indexed="true" stored="true" required="true" multiValued="false" />

Once you declare here, then solr understands your Fields .

then you need to copy your fields or link your fields as like below


    <copyField source="id" dest="refrence"/>
    <copyField source="name" dest="refrence"/>
    <copyField source="description" dest="refrence"/> 

where refrence can be defined as below 


    <field name="refrence" type="text_general" indexed="true" stored="false" multiValued="true"/>

it is of type "text_general" you can use this filed type for copying of the fields 

Step:6

Now we have declared the Fields, we have to define the default Fields to be searched .

We can achieve by two ways, one by defining in the solrconfig and other is by passing through the query.

We can see how to define in the solrconfig.xml

Navigate to solrconfig.xml in conf directory and search for the request handler select and add the following fields to it like below

<requestHandler name="/select" class="solr.SearchHandler">

     <lst name="defaults">

       <str name="echoParams">explicit</str>

       <int name="rows">10</int>

                   <str name="df">refrence</str>

     </lst>

</ requestHandler>

Once if you define like this, it will be treated as the search fields and no need to define in the URL.

Step:7 you need to import and index the Data.

Restart the solr server after the changes in the above Files .

Step:8

Navigate to http://localhost:8983/solr/refrence/dataimport?command=full-import

Once if you do this , it will import and index it.

You can see this by the below

http://localhost:8983/solr/refrence/dataimport

this gives the details about the processed skipped etc

Step:9

Now this Is the time to query for the Data you have Fetched and Indexed.

http://localhost:8983/solr/refrence/select?q=harley&wt=json

this will give the below response

Once if you see the below response, congragulations you have connected your database and indexed it as per your requirement .

Happy Learning . Stay Tuned for more solr Tutorials .

Getting Started with Solr

Hi Friends my first post will be, Installing the Solr and  how to index the Sample File Given in the Examples.

Step:1

Download the Apache Solr Latest Version , I downloaded the Version 6.4.2

Step:2

Before Starting the Apache Solr make sure that you have the Java 1.8 or Higher Version.

Step:3

Start the Server using the Command solr.cmd start

once if the server is started then you can access by the url

http://localhost:8983/solr/#

once you start the server then you will get the below console 

Step:4 
In Solr the term core is used to refer to a single index, 

So before staring the search create the Core by the following way.

Navigate to the solr-6.2.0\bin folder in command prompt

And run the below command

solr create –c refrence

Step:5

Once the Server is started you have index the Examples by the below Method

<Solr-ExtractedPath>/

And Execute java -Dc=refrence -Dauto -jar example\exampledocs\post.jar -c .\docs

Once if you execute the above script it will index the files inside the docs.

Challenges :

bin/post exists currently only as a Unix shell script, however it delegates its work to a cross-platform capable Java program.  The SimplePostTool can be run directly in supported environments, including Windows. By the above Script.

Once you execute the above link SimplePostTool can be used to index .

Errors:

Execute the post command Directly you will get the below error.

'post' is not recognized as an internal or external command,

operable program or batch file.

So use the steps I have given to resolve

SimplePostTool version 5.0.0

Posting files to [base] url http://localhost:8983/solr/gettingstarted/update using content-type application/xml...

SimplePostTool: WARNING: No files or directories matching -c

SimplePostTool: WARNING: No files or directories matching refrence

POSTing file films.json to [base]

SimplePostTool: WARNING: Solr returned an error #404 (Not Found) for url: http://localhost:8983/solr/gettingstarted/update

SimplePostTool: WARNING: Response: <html>

<head>

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>

<title>Error 404 Not Found</title>

</head>

<body><h2>HTTP ERROR 404</h2>

<p>Problem accessing /solr/gettingstarted/update. Reason:

<pre>    Not Found</pre></p>

</body>

</html>

SimplePostTool: WARNING: IOException while reading response: java.io.FileNotFoundException: http://localhost:8983/solr/gettingstarted/update

1 files indexed.

COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...

SimplePostTool: WARNING: Solr returned an error #404 (Not Found) for url: http://localhost:8983/solr/gettingstarted/update?commit=true

SimplePostTool: WARNING: Response: <html>

<head>

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>

<title>Error 404 Not Found</title>

</head>

<body><h2>HTTP ERROR 404</h2>

<p>Problem accessing /solr/gettingstarted/update. Reason:

<pre>    Not Found</pre></p>

</body>

</html>

Time spent: 0:00:00.589

if the above error comes make sure that you are passing the Correct arguments .

java -Dc=refrence -Dauto -jar example\exampledocs\post.jar -c example\exampledocs

     Once if you do this it will index the Files as below

SimplePostTool version 5.0.0

Posting files to [base] url http://localhost:8983/solr/refrence/update...

Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log

SimplePostTool: WARNING: No files or directories matching -c

Indexing directory example\exampledocs (19 files, depth=0)

POSTing file books.csv (text/csv) to [base]

POSTing file books.json (application/json) to [base]/json/docs

POSTing file gb18030-example.xml (application/xml) to [base]

POSTing file hd.xml (application/xml) to [base]

POSTing file ipod_other.xml (application/xml) to [base]

POSTing file ipod_video.xml (application/xml) to [base]

POSTing file manufacturers.xml (application/xml) to [base]

POSTing file mem.xml (application/xml) to [base]

POSTing file money.xml (application/xml) to [base]

POSTing file monitor.xml (application/xml) to [base]

POSTing file monitor2.xml (application/xml) to [base]

POSTing file more_books.jsonl (application/json) to [base]/json/docs

POSTing file mp500.xml (application/xml) to [base]

POSTing file sample.html (text/html) to [base]/extract

POSTing file sd500.xml (application/xml) to [base]

POSTing file solr-word.pdf (application/pdf) to [base]/extract

POSTing file solr.xml (application/xml) to [base]

POSTing file utf8-example.xml (application/xml) to [base]

POSTing file vidcard.xml (application/xml) to [base]

19 files indexed.

COMMITting Solr index changes to http://localhost:8983/solr/refrence/update...

Time spent: 0:00:12.995

Once Indexing is done then you can access by the Followin url or search by the following url

Access:

http://localhost:8983/solr/refrence/select?q=book&wt=xml