The JSpider engine has quite a complex start-up process that uses a range of property files, static factory methods, abstractions and reflective invocation. These notes are to clarify the origin of runtime properties, configuration and storage facilities. Most class references refer to JSpider library classes.
main entry point
The primary JSpider class takes two arguments: a base URL to start spidering, and a configuration directory reference. If no configuration is specified, a directory called "default" is checked for configuration property files.
The JSpider class completes the following steps:
- Loads the relevant configuration using the static
- Creates a new
- Calls the
startmethod on the
ConfigurationFactory methods that may be called by
JSpider create a singleton
JSpiderConfiguration instance from a new
PropertiesConfiguration. Methods also exist to assign a
JSpiderConfiguration directly and "clean" the configuration by assigning a
null. The properties configuration assembles JSpider, plugin and site configuration settings from directories and property files under the command line configuration directory.
The JSpider constructor instantiates a
SpiderNest object; gets a
SpiderContext from the static
SpiderContextFactory.createContext(URL) method; and passes the context to the nest to get a
Spider instance. The spider nest gets number of spider threads and thinker threads from the context and passes these to the spider implementation.
The spider implementation contains two
WorkerThreadPools; one for the spiders and one for the thinkers. It also creates thread pool monitors to periodically report the status of the pools. The spider's
crawl method gets an
SpiderContext and dispatches a
SpideringStartedEvent with the base URL that was passed on the command line. This is followed by the creation of new
DispatchThinkerTasks instances, which are assigned to the worker thread pools.
crawl method returns, the spidering process is complete, but before it shuts down the spider context, it dispatches a
SpideringSummaryEvent and a
Copyright MKDoc Ltd. and others.
The Free Documentation License http://www.gnu.org/copyleft/fdl.html