项目编号 : 563

项目预算 : $1,000-5,000

开发周期 : 7 天

类别 : 应用软件开发 - 其它软件开发

发布日期 : 2008-08-27

描述

<h1>Introduction</h1>  The Bidder shall develop a web crawling and reporting tool using Nutch (http://lucene.apache.org/nutch/) distributed over a series of Linux workstations. The tool shall index and search the World Wide Web to identify sites that contain certain keywords and URLs (via regular expressions). These sites shall be cross referenced with information from a database and a public web service to join in additional site information such as traffic rank. The database shall be provided by the Requester. The tool shall generate a report displaying a categorized list of all sites containing specified keywords/URLs sorted by traffic rank or other attributes assigned to the identified sites. The keywords and URLs to be displayed in the report shall be entered into a configuration file using regular expressions.  The tool shall run continuously, producing a new report daily based on the information most recently generated by the tool.  The report shall be of the following example structure:  ------------------------------------------------------------------------------------------------------------------------------------------  Site Rank Other Attribute #1 Other Attribute #2  ------------------------------------------------------------------------------------------------------------------------------------------  “auto|car manufacturers”   <a href="http://www.site1.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site1.com/', 250)" onmouseout="hideddrivetip()">www.site1.com</a> 500 103 +7  <a href="http://www.site2.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site2.com/', 250)" onmouseout="hideddrivetip()">www.site2.com</a> 400 214 +9  <a href="http://www.site3.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site3.com/', 250)" onmouseout="hideddrivetip()">www.site3.com</a> 300 120 +77  <a href="http://www.site4.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site4.com/', 250)" onmouseout="hideddrivetip()">www.site4.com</a> 200 121 +13  <a href="http://www.site5.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site5.com/', 250)" onmouseout="hideddrivetip()">www.site5.com</a> 100 210 -8  “fast cars”   <a href="http://www.site4.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site4.com/', 250)" onmouseout="hideddrivetip()">www.site4.com</a> 700 20 +4  <a href="http://www.site7.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site7.com/', 250)" onmouseout="hideddrivetip()">www.site7.com</a> 600 53 -10  <a href="http://www.site8.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site8.com/', 250)" onmouseout="hideddrivetip()">www.site8.com</a> 500 11 -8  <a href="http://www.site9.com/" class="extlink" rel="nofollow" onmouseover="ddrivetip('You are about to go to a URL outside odesk.com - http://www.site9.com/', 250)" onmouseout="hideddrivetip()">www.site9.com</a> 400 4 +5  www.site10.com 300 52 +1   ------------------------------------------------------------------------------------------------------------------------------------------  Total 998 908 90  ------------------------------------------------------------------------------------------------------------------------------------------  Source: Fictitious data, for illustration purposes only  Figure 1 – Example Report  The Bidder shall work at least four hours during U.S. Eastern Standard Time to facilitate communication with the Requester.  The system shall be delivered in four phases, described below:  <h1>Phase 1 – Proof of Concept</h1>  The Phase 1 system shall operate on a single Linux workstation and will be limited to crawling and searching for specified keyword sets. The report generated shall resemble that shown in Figure 1, except that sites will be organized according to rank based on link counts rather than attributes selected from the database. Phase 1 shall be designed to be scalable to multiple servers, per Phase 3 below.  <h1>Phase 2 – Incorporate Site Attributes from Database and Web Service</h1>  In Phase 2 the attributes from the provided MySQL database shall be incorporated into the report, along with attributes from Amazon’s Alexa web service. The account to access the Alexa web service shall be provided by the Requester. Attributes shall include Alexa rank and Nielson volume estimate.  <h1>Phase 3 – Distributed Search and Algorithmic Tests</h1>  In Phase 3, the system shall be distributed across a family of servers, which will provide for much more extensive search and reporting capabilities. Phase 3 also includes certain tests on pages that match the search criteria, for example, testing of certain URLs and scripts embedded in the page match algorithmic criteria. These tests shall be specified as configuration parameters, for example, by using a scripting language such as Groovy, Jython, or similar (to be specified by Requester). The test results (e.g., pass/fail) shall be displayed in the report.  <h1>Phase 4 – Site Characterization</h1>  In Phase 4, Site Characterization Information (SCI) shall be included for each site. Initially, SCI shall be the top twenty index words (based on word frequency in the site, not including words specified in an index exclusion configuration file. The index exclusion configuration file shall be a line oriented text file, where each line is a regular expression.  <h1>Feature Summary (End of Phase 4)</h1>  · Crawls the WWW and captures sites containing keywords/URLs specified using regular expressions in a text-based configuration file.  · Adds Site Characterization Information, i.e., metadata, for each site matching the keyword criteria. Initially, the SCI shall be the top twenty index words (by frequency), found in the site, excluding words or phrases that match regular expressions in a configurable exclusion file.  · Runs continuously, generating a daily report based on the recent information captured by the crawler. The report and system shall run continuously and reliably, without requiring any manual maintenance activities on the part of the Requester.  · Adds information to the site report captured via:  o MySQL database provided by the Requester  o Amazon Alexa web service (bidder shall implement interface)  o Algorithmic tests running against pages in the sites that match the search criteria. The algorithmic tests shall be specific using a scripting language in a configuration file.  · Generates reports similar in format to that shown in Figure 1. The report shall be in ASCII text format.  · Supplementary tools to aid in maintenance and monitoring of the system shall be provided by the Bidder. However, the system shall operate reliably 24x7 without requiring manual maintenance activities.  <h1>Deliverables</h1>  1. Source code, libraries, and scripts need to run and administer the application  2. Demonstration of the system running continuously, generating reports as specified.  3. Documentation describing the use and maintenance of the system, including examples of the reports generated.  <h1>Development Environment</h1>  The development environment is:  · Linux OS, multiple workstations, accessed via ssh  · Java Language (Nutch is written in Java)  · Scripting via Bourne shell and Perl  · MySQL database, loaded via csv files  · Scripting language (Groovy, Jython, or similar)

竞标

请您先登录，然后提交此项目的竞标方案。

还不是智城用户? 智城期待您的加入，请注册成为我们的一员吧！

Sr. Java Developer

描述

竞标