core/src/main/java/com/digitalpebble/stormcrawler/protocol/RobotRulesParser.java
public BaseRobotRules parseRules(String url, byte[] content,
            String contentType, String robotName) {
        return robotParser.parseContent(url, content, contentType, robotName);
    }
public BaseRobotRules parseRules(String url, byte[] content,
            String contentType, String robotName) {
        return robotParser.parseContent(url, content, contentType, robotName);
    }
public BaseRobotRules parseRules ( String url , byte [ ] content , String contentType , String robotName ) { return robotParser . parseContent ( url , content , contentType , robotName ) ; }
Parses the robots content using the {@link SimpleRobotRulesParser} from
crawler commons

@param url
A string containing url
@param content
Contents of the robots file in a byte array
@param contentType
The
@param robotName
A string containing value of
@return BaseRobotRules object
Parses the robots content using the { @link SimpleRobotRulesParser } from crawler commons