================================================ Agents for Job Execution and Communication Tasks ================================================ Agents do some work specified by jobs, the main task being to collect information objects from the local machine or some external source and transfer them e.g. to a loops server on the same machine or another. ($Id$) This package does not depend on Zope but represents a standalone application. Sub-Packages ============ Top-level Generic interfaces, ``commponents``: adapter registry, ``tests`` module, this ``README.txt`` file. base Base and sample classes. core Agent and scheduling implementations. control Communication with an external agent control and job database application. crawl Scanning/crawling some system, e.g. the database of an email application, the local file system, or an external document source. transport Transfer of information objects to agents on another machine or to an information management application (e.g. loops). util Various utility modules, e.g. a backport of the ``twisted.internet.task.coiterate()`` function from Twisted 2.5 so that we can use the Twisted version coming with Zope 3.3.1 for cybertools.agent. All sub-packages except ``base`` depend on Twisted. Object Structure and Control Flow ================================= :: ---------- ------------ -------- | |<---1| | | config |--> | master |<---2| controller | -------- /| |4--->| | / ---------- ------------ / 1 ^ 2 / | | \ ----------- / | | `---->| scheduler | v v 4 ----------- ----- --------- 3 | log |<--| agent |<----------ยด ----- --------- (1) Agent specifications control creation and setup of agents (2) Job specifications control scheduling of jobs (3) Scheduler triggers job execution by agent (4) Results are recorded, possibly triggering further actions Basic Stuff =========== While the real application is based on the asynchronous communication framework Twisted there is some basic stuff (mainly interfaces and base classes with basic, sample, or dummy implementations) that is independent of Twisted. The code for this resides in in the ``base`` sub-package. Master Agent and Configuration ------------------------------ All activity is controlled by the master agent. The master agent is set up according to the specifications in a configuration file. The configuration typically provides only basic informations, e.g. about the controller(s) and logger(s) to be used; details on jobs and agent configuration are provided by the controller. >>> from cybertools.agent.tests import baseDir >>> import os >>> configFile = open(os.path.join(baseDir, 'base', 'sample.cfg')) So we are now ready to create a master agent and configure it by supplying the path to the configuration file. >>> from cybertools.agent.main import setup >>> master = setup(configFile) Starting agent application... Using controllers base.sample. >>> configFile.close() >>> master.config controller.names = ['base.sample'] logger.name = 'default' logger.standard = 30 scheduler.name = 'sample' Controllers ----------- Creation of agents and scheduling of jobs is controlled by controller objects. These are typically associated with a sort of control storage that provides agent and job specifications and receives the results of job execution. >>> master.controllers [] We make the contollers provide the specifications via the master agent's ``setup()`` method. >>> master.setup() Starting agent application... Using controllers base.sample. Other Agents ------------ The above ``setup()`` call has triggered the creation of one child agent - that is all the sample controller provides. >>> master.children {'sample01': } Let's check a few attributes of the newly created agent. >>> agent01 = master.children['sample01'] >>> agent01.master is master True >>> agent01.config is master.config True Job Scheduling and Execution ---------------------------- A scheduler is responsible for triggering the execution of a job at the appropriate time. The master agent schedules the jobs based upon the information (job specifications) it gets from the controller. There is just one scheduler associated with the master agent. >>> master.scheduler We schedule a sample job by calling an internal method of the agent's controller. In addition to the output of the job execution itself we also get a note from the controller about the feedback it received about the outcome of the job execution. >>> master.controllers[0].enterJob('sample', 'sample01') Job 00001 on agent sample01 has been executed. Job 00001 completed; result: None; Logging ------- >>> master.logger >>> agent01.logger is master.logger True >>> master.config.logger.standard = 20 >>> master.logger.setup() >>> master.controllers[0].enterJob('sample', 'sample01') Job 00002 on agent sample01 has been executed. 2... agent:sample01 job:00002 message:job execution result:OK Job 00002 completed; result: None; >>> for r in master.logger.records: ... print r 2... agent:sample01 job:00001 message:job execution result:OK 2... agent:sample01 job:00002 message:job execution result:OK Using the Twisted-based Scheduler ================================= By specifying the core scheduler in the agent's configuration this will be used automatically for scheduling. In addition, we use another sample controller, now also the twisted-based from the core package. This will in turn set up a queueable agent from the core package so that now everything is running under the control of the twisted reactor. >>> config = ''' ... controller(names=['core.sample']) ... scheduler(name='core') ... logger(name='default', standard=30) ... ''' >>> master = setup(config) Starting agent application... Using controllers core.sample. >>> master.scheduler We enter the same job specification as above. >>> master.controllers[0].enterJob('sample', 'sample01') Now the job is not executed immediately - we have to hand over control to the twisted reactor first. The running of the reactor is simulated by the ``iterate()`` method provided for testing. >>> from cybertools.agent.tests import tester >>> tester.iterate() Job 00001 on agent sample01 has been executed. Job 00001 completed; result: Done;