Getting a Web Page

Now that we have so many functions we can tackle the two standard tasks in using the TCP stack - getting and sending data as a client and as a server. 

First we tackle the problem of acting as a client.

This isn't as common a requirement as you might expect because most of the time devices like the Pi are used to supply data to other servers, not the other way round. However, it is worth seeing how it is done. 

It doesn't matter if you are implementing a client or a server, either way you make use of sockets which represent the basic TCP connection. What you do with this connection is up to you. For example, if you send HTTP headers on an appropriate port then you can fetch or deliver a web page, i.e. HTTP over TCP. However, what data you actually send and receive over a socket connection is up to you and/or the protocol you are trying to use.  For more information on socket programming see: 

C Sockets - No Need For A Web Server!

The first thing we have to do is set up a socket connection between the client, i.e. the Pi,  and the server.

int getWebPageWiFi(char URL[], char page[]) {   dprintf(sfd, "AT+CIPSTART=\"TCP\",\"%s\",80\r\n", URL);   if (getBlocks(10, "OK") < 0) return -1;

You pass the URL to the function as an IP address or as a full URL, but the device looks up domain names using a fixed set of DNS servers. It is recommended that you use an IP address, especially when testing. The CIPSTART command opens a socket to the specified IP address and port. You can also specify a TCP or UDP connection:

AT+CIPSTART=type, IP, port

In this case we open port 80 on the specified IP address or URL. If it works you will get back a message something like:

Connect
AT+CIPSTART="TCP","192.168.253.23",80
CONNECT
OK

Now we have a socket open we can send some data to the server and wait for some data to be sent back to us. This can be a problem as you can't anticipate the amount of data you get back from the web server. 

For this example the web page is served by a small sensor that returns a JSON temperature and Humidity reading - yes it is another Pi and the web server described in the next section. 

To send data over a socket you use CIPSEND, which will send any data you specify to the server. As already made clear, what you send is a matter of what protocol you are using over the socket. In this case it is HTTP and we are going to send headers corresponding to a GET request for index.html

char http[150];
sprintf(http, "GET %s HTTP/1.1\r\nHost:%s\r\n\r\n",
                                  page, URL);

This time we can't just send the string using dprintf because we need to include the length of the string in the header we send to the server. The headers we are sending are:

GET /index.html HTTP/1.0
Host:192.168.253.23

An HTTP request always ends with a blank line.

To send this request we use the CIPSEND command which specifies the number of characters that are to follow:

dprintf(sfd, "AT+CIPSEND=%d\r\n", strlen(http));

Now we have to send the number of bytes/ characters that we specified in the CIPSEND. but first we wait for a ">" to indicate that the device is ready to receive the data:

if (getBlocks(10, ">") < 0) return -1;
dprintf(sfd, "%s", http);
return getBlocks(10, "</html>");
}

What happens next depends on the server. As a result of the HTTP GET the server will now send data over the WiFi link and the device will send this over the serial connection as soon as it gets it. 

Notice that this data is not a direct response to a command and so the device prefixes it with: 

+IPD,len:

The +IPD makes it clear to the client that is a packet of data sent from the server. The len value gives the number of characters sent after the colon. You could use this to work out when to stop reading data, but for simplicity we scan for "</html>" which is usually, but not always, the final tag at the end of a web page. 

In principle what your program should do next is sit in a polling loop looking for +IPD. It should then read the digits between the comma and the colon and convert this to an integer. Finally it should then read exactly that number of characters from the serial port. 

+IPD,17:HTTP/1.0 200 OK
+IPD,99:Server: BaseHTTP/0.6 Python/3.2.3
Date: Thu, 14 Jul 2016 16:42:37 GMT
Content-type: text/html
+IPD,127:<html><head><title>Temperature</title></head>

<body><p>{"humidity":0,"airtemperature":0}

</p></body></html>CLOSED

You can see that there are three "packets" of data - 17 characters, 99 characters and finally 127 characters. In principle you could process the +IPD characters as they come in and work out how many characters to read. However, you still wouldn't know how many packets to expect. 

The complete function is:

int getWebPageWiFi(char URL[], char page[]) {
 dprintf(sfd, "AT+CIPSTART=\"TCP\",\"%s\",80\r\n", URL);
 if (getBlocks(10, "OK") < 0) return -1;
 char http[150];
 sprintf(http, "GET %s HTTP/1.1\r\nHost:%s\r\n\r\n",
                                          page, URL);
 dprintf(sfd, "AT+CIPSEND=%d\r\n", strlen(http));
 if (getBlocks(10, ">") < 0) return -1;
 dprintf(sfd, "%s", http);
 return getBlocks(10, "</html>");
}