JUnit testing for HTML parsing

Question

I'm trying to set up unit tests on a web crawler and am rather confused as to how I would test them. (I've only done unit testing once and it was on a calculator program.)

Here are two example methods from the program:

protected static void HttpURLConnection(String URL) throws IOException {

    try {
        URL pageURL = new URL(URL);

        HttpURLConnection connection = (HttpURLConnection) pageURL
                .openConnection();
        stCode = connection.getResponseCode();
        System.out.println("HTTP Status code: " + stCode);

        // append to CVS string
        CvsString.append(stCode);
        CvsString.append("\n");

        // retrieve URL
        siteURL = connection.getURL();
        System.out.println(siteURL + " = URL");

        CvsString.append(siteURL);
        CvsString.append(",");
    } catch (MalformedURLException e) {
        e.printStackTrace();
    }
}

and:

public static void HtmlParse(String line) throws IOException {

    // create new string reader object
    aReader = new StringReader(line);

    // create HTML parser object
    HTMLEditorKit.Parser parser = new ParserDelegator();

    // parse A anchor tags whilst handling start tag
    parser.parse(aReader, new HTMLEditorKit.ParserCallback() {
        // method to handle start tags
        public void handleStartTag(HTML.Tag t, MutableAttributeSet a,
                int pos) {
            // check if A tag
            if (t == HTML.Tag.A) {
                Object link = a.getAttribute(HTML.Attribute.HREF);
                if (link != null) {
                    links.add(String.valueOf(link));

                    // cast to string and pass to methods to get title,
                    // status
                    String pageURL = link.toString();
                    try {
                        parsePage(pageURL); // Title - To print URL, HTML
                        // page title, and HTTP status
                        HttpURLConnection(pageURL); // Status
                        // pause for half a second between pages
                        Thread.sleep(500);

                    } catch (IOException e) {
                        e.printStackTrace();
                    } catch (BadLocationException e) {
                        e.printStackTrace();
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }
        }
    }, true);
    aReader.close();
}

I've set up a test class in Eclipse and have outline test methods along these lines:

@Test
public void testHttpURLConnection() throws IOException {
    classToTest.HttpURLConnection( ? );
    assertEquals("Result", ? ? )
}

I don't really know where to go from here. I'm not even sure whether I should be testing live URLs or local files. I found this question here: https://stackoverflow.com/questions/5555024/junit-testing-httpurlconnection but I couldn't really follow it and I'm not sure it was solved anyway. Any pointers appreciated.

Tomer · Accepted Answer · 2012-07-16 12:29:30Z

1

There is no one conclusive answer to your question, what you test depends on what your code does and how deep you want to test it.

So if you have a parse method that takes an HTML and returns the string: "this is a parsed html" (obviously not very usefull, but just making a point), you'll test it like:

@Test
public void testHtmlParseSuccess() throws IOException {        
    assertEquals("this is a parsed html", classToTest.parse(html) ) //will return true, test will pass
}

@Test
    public void testHtmlParseSuccess() throws IOException {        
        assertEquals("this is a wrong answer", classToTest.parse(html) ) //will return false, test will fail
    }

There are a lot more methods besides assertEquals() so you should look here.

eventually it is up to you to decide what parts to test and how to test them.

answered Jul 16, 2012 at 12:29

Tomer

17.9k16 gold badges82 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Robert Over a year ago

Thanks - I'm off to check out that cookbook.

lost · Accepted Answer · 2012-07-16 12:47:43Z

Think about what effects your methods should have. In the first case the expected thing that should happen when HttpURLConnection(url) is called seems to be that the status code and url are appended to something called CvsString. You will have to implement something in CvsString so that you can inspect if that what you expected did actually happen.

However: Looking at your code I would suggest you consult a book about unit testing and how to refactor code so that it becomes well testable. In your code snippets I see a lot of reasons why unit testing your code is difficult if not impossible, e. g. overall use of static methods, methods with side effects, very little separation of concerns etc. Because of this it is impossible to answer your question fully in this context.

Don't get me wrong, this isn't meant in an offending way. It is well worth learning these things it will improve your coding abilities a lot.

Collectives™ on Stack Overflow

JUnit testing for HTML parsing

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related