FAST Forum

Hi,

we have scheduled filetraverser to run automatically for every one minute. It is started with the set of 1.3 million documents(including below and above 5 MB documents). It was started without any issues, but after indexing is completed for one lakh documents we faced the below issue, and none of the files are indexed after that. As a temporary fix we restarted FAST service. We faced the same issue many times in the middle of indexing the remaining documents but we did a temporary fix(FAST restart) and indexed all the documents(1.3 million).
Every day we are getting around 500 files to index(including below and above 5 MB documents), but sometimes in the middle of the indexing it get jammed.Please let me know that if anyone have any suggestion to resolve this.

Issue:
[2011-09-22 14:32:42.926] VERBOSE LNCapmon ESP batch delivered in 12.93 seconds
[2011-09-22 14:32:42.926] VERBOSE systemmsg ESP content batch ID=45-3-176787 has completed processing
[2011-09-22 14:32:42.926] VERBOSE systemmsg Re-queuing document for re-submit (try 1) to the ESP content distributor. URI=file:////LNGSYDAPPP065/pdf/pressrel/gn111216.pdf
[2011-09-22 14:32:42.926] VERBOSE LNCapmon Sending 1 documents in collection LNCapmon to ESP content distributor
[2011-09-22 14:32:49.894] VERBOSE LNCapmon Content feeding batch timed out (first: 176795, last: 176795), generating callback
[2011-09-22 14:32:49.894] VERBOSE LNCapmon Content feeding batch timed out (first: 176803, last: 176803), generating callback
[2011-09-22 14:32:49.894] VERBOSE LNCapmon Content feeding batch timed out (first: 176791, last: 176791), generating callback
[2011-09-22 14:32:49.895] VERBOSE LNCapmon Content feeding batch timed out (first: 176799, last: 176799), generating callback
[2011-09-22 14:32:49.895] VERBOSE LNCapmon Content feeding batch timed out (first: 176796, last: 176796), generating callback
[2011-09-22 14:32:49.895] VERBOSE LNCapmon Content feeding batch timed out (first: 176792, last: 176792), generating callback
[2011-09-22 14:32:49.895] VERBOSE LNCapmon Content feeding batch timed out (first: 176806, last: 176806), generating callback
[2011-09-22 14:32:49.895] VERBOSE LNCapmon Content feeding batch timed out (first: 176805, last: 176805), generating callback
[2011-09-22 14:32:49.896] VERBOSE LNCapmon Content feeding batch timed out (first: 176788, last: 176788), generating callback
[2011-09-22 14:32:49.896] VERBOSE LNCapmon Content feeding batch timed out (first: 176802, last: 176802), generating callback
[2011-09-22 14:32:49.896] VERBOSE LNCapmon Content feeding batch timed out (first: 176801, last: 176801), generating callback
[2011-09-22 14:32:49.896] VERBOSE LNCapmon Content feeding batch timed out (first: 176797, last: 176797), generating callback
[2011-09-22 14:32:49.896] VERBOSE LNCapmon Content feeding batch timed out (first: 176793, last: 176793), generating callback
[2011-09-22 14:32:49.897] VERBOSE LNCapmon Content feeding batch timed out (first: 176789, last: 176789), generating callback
[2011-09-22 14:32:49.897] VERBOSE LNCapmon Content feeding batch timed out (first: 176804, last: 176804), generating callback
[2011-09-22 14:32:49.897] VERBOSE LNCapmon Content feeding batch timed out (first: 176794, last: 176794), generating callback
[2011-09-22 14:32:49.897] VERBOSE LNCapmon Content feeding batch timed out (first: 176790, last: 176790), generating callback
[2011-09-22 14:32:49.897] VERBOSE LNCapmon Content feeding batch timed out (first: 176798, last: 176798), generating callback
[2011-09-22 14:32:49.898] VERBOSE LNCapmon Content feeding batch timed out (first: 176800, last: 176800), generating callback
[2011-09-22 14:32:49.952] VERBOSE LNCapmon ESP batch delivered in 7.03 seconds
[2011-09-22 14:32:49.952] VERBOSE systemmsg ESP content batch ID=45-3-176795 has completed processing
[2011-09-22 14:32:49.952] VERBOSE systemmsg Re-queuing document for re-submit (try 1) to the ESP content distributor. URI=file:////LNGSYDAPPP065/pdf/pressrel/gn11020188.pdf
[2011-09-22 14:32:49.952] VERBOSE systemmsg ESP content batch ID=45-3-176803 has completed processing

Thanks in advance.

-Gomathi.

Varies things that can be tried are, minimizing the batch size, changing the callback status etc.,

Basically the filetraverser feeds in a much faster rate that your indexer can handle. Look at it that way and you will get your answer.

_________________
Freddie Maize

..A story with Glory is History.
Doesn’t matter whether Glory rest in the world of Demon or God.
Lets create History..