Creating a Java program that can interact with a web page

Hey, I know this is a broad question, I don't really expect a complete answer, but a point in the direction of a good resource would be really helpful.

I'm a college student with a few semesters of java programming under my belt. I recently saw a java program that logs onto an account of an online game, and actually plays some aspects of the game for you.

This intrigued me, because in my classes I've never learned how java programs can interact with web pages or php forms, and I set off to learn how it works, and if I could even replicate some of its uses.

The problem I ran into, however, was that I did't have the correct vocabulary to describe what I want to do in a google search. I didn't know enough to learn what I needed to know! All I ended up learning was how to open a webpage through internet explorer, and how to display a webpage's html in a console, neither very useful for the task.

So I guess my question is, where should I start reading if I want to learn how to create a program that can interact with javascript/php/html forms on a webpage, navigate through links, and get info about a web page and store to a text file.


If you think this is easier than I'm making it, and could answer concisely, I'd be happy to post more info about my specific needs. If this is far harder than I'm making it, help me figure out what I need to know before I can do this! My question most likely reveals some ignorance about how this kind of interaction works, if so please point it out to me.

Comments

  • I made some similar Java applications like:
    - a limited web server,
    - search engine response downloader. It simulated Firefox because Google blocked responses for GET requests sent from URL's openStream() method.
    - website link checker, so I can tell you quite precisely what to do.


    To make a program that interacts with a website, you should learn about the HTTP protocol. You have to learn it because the interaction with a website would involve sending and receiving HTTP messages.

    If you just want to download content from a web page at a certain URL, you can use the java.net.URL class and call openStream() to get a stream for reading content. This is done by implicitly send an HTTP GET request so you don't have to worry about the details of HTTP for this.

    For simulating form submission, GET and more commonly POST methods from HTTP are used. If a form submits using GET, you can submit by knowing how to encode parameters into the URL. Since it is GET, a URL instance can do the job. For POST, you'll have to use java.net.Socket and encode the parameters properly into the request body properly. You'll also have to know enough about HTTP to write the HTTP headers.

    You don't have to know about PHP specifically. Extensions like php, asp, aspx, pl, cgi... all mean the same thing as far as your program would be concerned.

    I wrote some tasks at [link=http://www.hightechdreams.com/weaver.php?topic=http_tasks]hightechdreams website[/link] that should guide you through the process.
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Categories

In this Discussion