CMPUT 404

Web Applications and Architecture

Part 05: More HTTP

Created by
Abram Hindle (abram.hindle@ualberta.ca)
and Hazel Campbell (hazel.campbell@ualberta.ca).
Copyright 2014-2019.

HTTP PUT

• Like HTTP POST except the URI does not handle the request, it is the request.

  The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server MUST NOT attempt to apply the request to some other resource. Fielding, et al. ,RFC2616: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6

POST vs PUT

POST vs PUT

HTTP Delete

• Like HTTP POST except the URI does not handle the request, it is the request, a request to delete the entity at that URI.

The DELETE method requests that the origin server delete the resource identified by the Request-URI. This method MAY be overridden by human intervention (or other means) on the origin server. The client cannot be guaranteed that the operation has been carried out, even if the status code returned from the origin server indicates that the action has been completed successfully. However, the server SHOULD NOT indicate success unless, at the time the response is given, it intends to delete the resource or move it to an inaccessible location. A successful response SHOULD be 200 (OK) if the response includes an entity describing the status, 202 (Accepted) if the action has not yet been enacted, or 204 (No Content) if the action has been enacted but the response does not include an entity. Fielding, et al. ,RFC2616: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.7

POST vs DELETE

POST vs DELETE

HTTP PUT/GET/DELETE/GET

In the following example we use ElasticSearch and we PUT, GET, DELETE and GET a single URI.

The URI represents an entry in the ElasticSearch index, a NoSQL database, not a file. The entries for ElasticSearch are formatted in JSON.

• PUT stores the entity
• GET retrieves it
• DELETE deletes it

Step 1: PUT the entry in ElasticSearch

curl -v --trace-ascii /dev/stdout -h 'content-type: application/json' -x put http://cmput301.softwareprocess.es:8080/testing/junk/1 -d '{"name":"one"}'

put /testing/junk/1 http/1.1
user-agent: curl/7.29.0
host: cmput301.softwareprocess.es:8080
accept: */*
content-type: application/json
content-length: 14

{"name":"one"}

Server response:

http/1.1 200 ok
content-type: application/json; charset=utf-8
content-length: 68

{"ok":true,"_index":"testing","_type":"junk","_id":"1","_version":5}

Step 2: GET the entry from ElasticSearch

curl -v --trace-ascii /dev/stdout -h 'accept: application/json' -x get http://cmput301.softwareprocess.es:8080/testing/junk/1

get /testing/junk/1 http/1.1
user-agent: curl/7.29.0
host: cmput301.softwareprocess.es:8080
accept: application/json

Server response:

http/1.1 200 ok
content-type: application/json; charset=utf-8
content-length: 100

{"_index":"testing","_type":"junk","_id":"1","_version":5,"exists":true, "_source" :{"name":"one"}}

Step 3: DELETE the entry from ElasticSearch

curl -v --trace-ascii /dev/stdout -x delete http://cmput301.softwareprocess.es:8080/testing/junk/1

delete /testing/junk/1 http/1.1
user-agent: curl/7.29.0
host: cmput301.softwareprocess.es:8080

Server response:

http/1.1 200 ok
content-type: application/json; charset=utf-8
content-length: 81

{"ok":true,"found":true,"_index":"testing","_type":"junk","_id": "1","_version":6}

Step 4: GET the entry from ElasticSearch

curl -v --trace-ascii /dev/stdout -h 'accept: application/json' -x get http://cmput301.softwareprocess.es:8080/testing/junk/1

get /testing/junk/1 http/1.1
user-agent: curl/7.29.0
host: cmput301.softwareprocess.es:8080
accept: application/json

Server response:

http/1.1 404 not found
content-type: application/json; charset=utf-8
content-length: 60

{"_index":"testing","_type":"junk","_id":"1","exists":false}

It's gone, we deleted it.

WebDAV

• RFC: http://tools.ietf.org/html/rfc4918
• Like FTP but for the web!
  • Let's you create and upload to a URI using HTTP PUT
  • Download from a URI using HTTP GET
  • Delete an entity at a URI (and the URI) using HTTP DELETE
  • Make directories/folders using a new HTTP command: HTTP MKCOL
    • MaKe COLlection

Why PUT/DELETE and WebDAV?

• Why would we bother with HTTP PUT and DELETE when we have POST which can do both?

HTTP User Agent

• A client/browser
  • In RFCs it usually means the HTTP client and often means a browser

  Flynn: Who's that guy?
  Program: That's Tron. He fights for the Users. Tron, 1982

Example User-Agents

• Chrome 71.0 on Windows 10 on a PC: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
• Firefox 64.0 on Windows 10 on a PC: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0
• Safari 12 on OSX 10.14: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15
• MS Edge 17: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134

Put the word Mobile in there if its coming from a mobile device, iPhone for an iPhone, Android for an Android device, etc.

• Samsung Galaxy S8: Mozilla/5.0 (Linux; Android 7.0; SM-G892A Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Mobile Safari/537.36
• Apple iPhone X: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1
• Nintendo Switch: Mozilla/5.0 (New Nintendo Switch like iPhone) AppleWebKit/536.30 (KHTML, like Gecko) NX/3.0.0.5.20 Mobile NintendoBrowser/1.9.10160.US

Search engines gathering webpages so you can search them:

• Google's web crawler: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

What?

user-agent: mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/64.0.3282.140 safari/537.36 edge/17.17134

Microsoft Edge version 17
which is based on Chrome 64
which is based on WebKit 537 (part of Safari)
(WebKit itself was forked from/based on KHTML, KDE's HTML rendering engine, but it claims to work like Gecko, Mozilla's old rendering engine)
which is like Mozilla/5.0 (Netscape Navigator version 5.0)
running on Windows NT 10.0 on a 64-bit Intel x86 CPU.

So some code from KDE (a linux Desktop Environment) is now in Microsft's latest browser, Edge.

2019: year of the Linux desktop on the Windows desktop

HTTP Status Codes

In HTTP/1.1 200 OK, 200 is the status code.

• 1XX — Informational codes, ex: 100 Continue
• 2XX — Success codes, ex: 200 OK
• 3XX — Redirection codes, ex: 301 Moved Permanently
• 4XX — Client Error codes, ex: 404 Not Found
• 5XX — Server Error codes, ex: 500 Internal Server Error

Informational Status Codes: 1XX

• HTTP/1.1 100 Continue
  • Used in multipart and uploads
  • Tells the client to send the request body/data
  • The server had a choice to accept the request or not and it has decided to accept it

• HTTP/1.1 101 Switching Protocols (rare)
  • Upgrade: header specifies the new protocol
  • Protocol switches immediately after the blank line at the end of headers
  • Switch from HTTP/1.1 to HTTP/2 on unencrypted connections:
    • but browsers don't support HTTP/2 over unencrypted connections
    • so it's basically never used
• Browsers select HTTP/1.1 or HTTP/2 during the TLS handshake for https URLS
  • TLS Application-Layer Protocol Negotiation
  • Occurs at the same as cipher suite negotiation
  • No additional round-trip latency
  • RFC 7301: https://tools.ietf.org/html/rfc7301

Successful Status Codes: 2XX

• HTTP/1.1 200 OK
  • Request succeeded, depends on the client request
  • GET: sends an entity for to the requested URI
  • HEAD: sends headers for the entity for the requested URI
  • POST: sends an entity describing the result of the POST
  • TRACE: sends back the entity it received

• HTTP/1.1 201 Created
  • Request succeeded and a new entity was created and exists (e.g. PUT)
• HTTP/1.1 202 Accepted
  • Like 200 OK but the server's not done with it yet. For example, you asked the server to perform a calculation and its working on it.
• HTTP/1.1 203 Non-Authoritative Information (rare)
  • Used with proxies.

• HTTP/1.1 204 No Content
  • Request succeeded but the server is only sending headers and no entity or response body
• HTTP/1.1 205 Reset Content (rare)
  • Like 204 No Content but the browser should clear the form/page

• HTTP/1.1 206 Partial Content
  • Used to resume downloads:
    1. Client does HTTP GET with a Range header to continue a big download that was interrupted
    2. Server responds with 206 Partial Content and a Content-Range indicating what part its sending

Redirect Status Codes: 3XX

There's always a reason why you want to redirect something

• HTTP allows for redirection
• Redirection is cheap abstraction
• Work-around for browser/protocol issues
• Redirect HTTP requests to HTTPS
• Redirect requests to old URIs to their new URIs
• Load balancing
• Separate dynamic and static content onto two different servers
• One host in URIs but use more than one server behind the scenes
• URL shorteners
• Reorganized website but want to keep old URIs working (fighting link-rot)

• HTTP/1.1 300 Multiple Choices (rare)
  • Responds with a list of choices (e.g. same page in different languages) and the user or the browser chooses one
• HTTP/1.1 301 Moved Permanently
  • Go to the URI mentioned in the Location header, and don't ask me again!
  • URI in the location bar automatically changes
• HTTP/1.1 302 Found
  • Temporary redirect. Client should GET the URI mentioned in the Location header and display that response instead
  • URI in the location bar stays the same (invisible to the user)
  • Useful for load-balancing

• HTTP/1.1 303 See Other
  • Exists to solve the forum-POSTing problem
  • GET the URI in the Location header
  • Don't save the redirect in your cache, you can keep making POSTs to the URI that gave you a 303
  • URI in the location bar changes

• HTTP/1.1 304 Not Modified
  • Browser can make a conditional GETs request for URIs that it has cached, asking the server to send the entity only if it's changed since the time it was cached or a specific version
  • If-None-Match: followed by a list of etags (like git tags, named versions) will get a 200 OK only if it has a new version not in the list
  • If-Modified-Since: followed by a date and time will get a 200 OK only if it has a newer version since that time
  • No response body/entity

• HTTP/1.1 307 Temporary Redirect
  • Go to the URI mentioned in the Location header
  • Keep making requests to the URI you originally requested in case the server needs to redirect you somewhere else next time
  • Cache the redirection using standard caching headers and rules
  • URI in the location bar is updated

• HTTP/1.1 308 Permanent Redirect
  • Go to the URI mentioned in the Location header
  • Similar to 301 Moved Permanently
  • Client must repeat the same request for the new location, unlike 301

Client Error Status Codes: 4XX

It's your fault! (or the User Agent's)

• You're not allowed
• You're wrong
• You owe us money
• You can't handle it
• You're taking too long
• You're changing things at the same time as someone else
• It ain't here and it ain't never coming back
• You're not making any sense
• You're asking too much

• HTTP/1.1 400 Bad Request
  • Hey buddy, I can't read this garbage. Don't send it again.
• HTTP/1.1 401 Unauthorized
  • You have to send authentication information to see this URI.
  • Headers and entity (response body) explains to the browser and user how to log in.
  • Mostly useful for HTTP Authorization: header authentication

• HTTP/1.1 402 Payment Required (rare)
  • Pay up, buttercup!
  • Supposedly reserved, but some services use it anyway, e.g. MobileMe used it (the predecessor to iCloud)
  • Google APIs use it
  • YouTube will use it to force you to solve a CAPTCHA

• HTTP/1.1 403 Forbidden
  • The web server will never respond to this request, no matter who you log in as
  • Maybe it could answer your request but an administrator disabled that ability.
• HTTP/1.1 404 Not Found
  • You've got the wrong resource or path. Can't find what you're looking for. Droids? What droids?
• HTTP/1.1 405 Method not allowed
  • Whatever method you used (GET/HEAD/POST/PUT/DELETE/...) doesn't work on this URI

• HTTP/1.1 406 Not Acceptable
  • The sever cannot respond in way that matches your request's accept header line.
  • Example: You asked for JSON and we can only serve XML, but it's your fault.
• HTTP/1.1 407 Proxy Authentication Required
  • We're not going to proxy your request till you authenticate.
• HTTP/1.1 408 Request time out
  • You took too long to send your request, we're not going to service you. Try again but faster next time.
  • Example: the Slowloris attack

• HTTP/1.1 409 Conflict
  • The request is in conflict. Often used with PUT requests.
  • Example: Two people trying to PUT a new version for the same URI at the same time
• HTTP/1.1 410 Gone
  • Yeah it was here, but it ain't coming back. Don't even try again.
• HTTP/1.1 411 Length Required
  • I can't service a request (ex: POST) without a Content-Length: header

• HTTP/1.1 412 Precondition Failed
  • Header information wouldn't be what you wanted it to be so I won't process the request
  • Example: modified too recently, so don't allow PUT to succeed to modify it again
• HTTP/1.1 413 Request Entity Too Large
  • Sending an entity (POST, PUT, ...) that's bigger than the server can handle
• HTTP/1.1 414 Request-URI Too Long
  • Webservers will only handle URLs up to a certain length
  • Example: Apache webserver is limited to 4000-8192 by default (depending on version)
  • Example: nginx webserver places a limit on total length of HTTP headers + requested URI

• HTTP/1.1 415 Unsupported Media Type
  • Uploading (POST, PUT, ...) using a format the server doesn't understand.
  • Example: posting pictures formatted in JPEG2000
• HTTP/1.1 416 Request Range Not Satisfiable
  • You sent a Range: header to get just part of a file but the part you asked for doesn't make sense
  • Example: You ask to resume a download that was interrupted of a 1MiB file at 1.1Mib

• HTTP/1.1 417 Expectation Failed
  • The server cannot meet the Expect: header
  • Example: client sent Expect: 100 Continue while POSTing multipart/form-data, but the server can't do that

• HTTP/1.1 418 I'm a teapot
  • Indicates that the server refuses to brew coffee because it is a teapot.
• HTTP/1.1 422 Unprocessable Entity
  • Indicates that the server understood the Content-Type and the syntax of the entity (request body) is correct but that it was unable to process it.
• HTTP/1.1 426 Upgrade Required
  • Indicates that the server requires use of HTTP 2 or later.

• HTTP/1.1 428 Precondition Required
  • Indicates that the client needs to send a request that is conditional, i.e. has an If- header.
• HTTP/1.1 429 Too Many Requests
  • Indicates that the client has sent too many requests in a short period of time.
• HTTP/1.1 431 Request Header Fields Too Large
  • Indicates that the client has sent request headers that are too long.

• HTTP/1.1 451 Unavailable For Legal Reasons
  • Indicates that the server could service the request... if it wasn't illegal.
  • Example: Server used to host content that was later taken down for copyright reasons.

This request may not be serviced in the Roman Province of Judea due to the Lex Julia Majestatis, which disallows access to resources hosted on servers deemed to be operated by the People's Front of Judea. — RFC 7725

Server Error Status Codes: 5XX

It's the server's fault!

• My script crashed
• My system is misconfigured
• I broke something
• We didn't implement that
• The server is down
• I don't support that version of HTTP

• HTTP/1.1 500 Internal Server Error
  • Server side software encountered some kind of error
  • Example: your Python Django code crashed
  • Example: couldn't connect to the database
• HTTP/1.1 501 Not Implemented
  • The server can't fulfill that request (such as an HTTP PUT) because it doesn't even know what HTTP PUT is

• HTTP/1.1 502 Bad Gateway
  • The server talks to another HTTP server to fulfill this request and that other server isn't working.

• Bad gateway example: Website using a reverse-proxy to reduce load, balance load, or provide redundancy on their own server
• Your webbrowser connects to somehost, but that actually resolves to a CDN: a webserver near you that acts a cache that's closer to you than the actual webservers of the website you want to visit
• Normally this speeds things up for you while reducing traffic to the actual webservers
• You don't see a difference, but only requests for dynamic content actually make it back to the actual webservers, whereas most content (e.g. images, css, javascript, ...) is served from the CDN caching reverse-proxy
• When you make a request for dynamic content the CDN caching reverse-proxy forwards the request to the actual website on your behalf
• But when the original webservers go down you get 502 Bad Gateway

• HTTP/1.1 502 Bad Gateway

• HTTP/1.1 503 Service Unavailable
  • The service is temporarily down. Something's broken and we'll bring it back up eventually.
  • Also used when servers are undergoing maintenance
• HTTP/1.1 504 Gateway Timeout
  • The server talks to another process to fulfill this request and that other process isn't responding fast enough. Very common when a webapp is overloaded.
  • Similar to 502, except in this case the packets between the reverse proxy and the origin webserver are just vanishing...

• HTTP/1.1 505 HTTP Version Not Supported
  • Your request used the wrong HTTP version. A version the server no longer supports.
  • Example: Twitter doesn't let you do HTTP/1.0 requests anymore

• HTTP/1.1 511 Network Authentication Required
  • Not used by webservers, but rather by captive portals to tell the web browser that it should show the user a login page for the network.

• Example: You connect to a guest WiFi
  1. You try to browse to some random webpage
  2. Captive portal intercepts your request and sends a 511 response back (the request never makes it onto the internet)
  3. Your web browser tells you you need to log on to the network
  4. You log on to the network using a webpage on the captive portal
  5. You retry your original request, this time it's not intercepted and proceeds to the server it was meant for

HTTP Errors: Client, Server or Application?

• If there's a problem in your web application how should you respond?
  • 4xx code?
  • 5xx code?
  • HTML page explaining to the user what happened?
  • 4xx/5xx + HTML page explaining to the user?

• If there's a problem in your web application how should you respond?
• Some suggest that business logic errors (software error reporting) should be done with the client without codes
• Some suggest that you should use HTTP status codes.
• Decide if the application is user facing and how you should handle it for your audience.
  • Do you need to talk to the User Agent (browser, software) or to the user (human)?
  • Example: many websites let you log in with cookie-based authentication. There's no standards-compliant way to use a response code, e.g. 401 Unauthorized to tell the user to log in this way. So it may be best to use a 302 or 307 redirect to send the user to the login page.
• Warning: Old versions of IE only show a few kb of HTML for 404 pages.

HTTP Headers

• There are 47+ standard HTTP headers just for version 1...
• These headers have an effect on:
  • Authentication
  • Caching
  • Encoding
  • Partial downloading
  • Content type
  • More...
• There are so many in the RFC that I won't go over all of them
• http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

• MDN has a great reference list for all the http headers supported by browsers:
  • https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

• OWASP has a great reference list for all the http headers you should use to help secure your app/server/site:
  • https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html

• HTTP header demo:
  • http://buttercup.cs.ualberta.ca:9000/
  • http://buttercup.cs.ualberta.ca/ (Reverse Proxy)
  • https://buttercup.cs.ualberta.ca/ (Reverse Proxy + HTTPS or /2 or /3)
  • Node.js server source code

HTTP Request Headers

• Accept: */*
  • Example: Accept: video/ogg,video/*
  • Specifies the kind of media the client can handle
• Accept-Charset: UTF-8
  • Deprecated -> Forbidden! Use path instead.
• Accept-Encoding: gzip,compress,deflate,br
  • Specifies compression formats the client can handle

HTTP Request Headers

• Accept-Language: en-CA
  • Creates privacy problems. Use path instead.

HTTP Request Headers

• Access-Control-Request-Headers: Content-Type
  • Lets the browser ask the server if JS is allowed to make requests with those headers
• Access-Control-Request-Method: POST
  • Lets the browser ask the server if JS is allowed to make e.g. POST requests

HTTP Request Headers

• Authorization: Basic YWxhZGRpbjpvcGVuc2VzYW1l
  • The user agent is sending a username and password or other kind of credentials to the server (rare, usually cookies are used instead)
• Cache-Control: max-age=60
  • Asks the server/proxy not to send data thats been sitting in its cache too long
• Content-Disposition: form-data; name="fieldName"; filename="filename.jpg"
  • Tells the server what the name/filename of the form data being uploaded when POSTing multipart/form-data

HTTP Request Headers

• Cookie: session=1nhbn4b123h4; csrftoken=1n1b4b1j2j3j2
  • The user agent is sending cookies (stored key-value pairs) relevant to the server
  • The cookies were previously sent to the user agent to store by the server or JS
• DNT: 1
  • The user prefers not to be tracked over receiving personalized content
  • Browsers continue to add support, servers continue to ignore it

HTTP Request Headers

• Expect: 100-continue
  • The user agent expects the server to respond with 100 Continue
• Forwarded: for=192.0.2.60;proto=http;host=example.com
  • Used by (reverse) proxies to tell the server who made the original request, over what protocol, and what the original Host header was
  • X-Forwarded-For is usually used instead.
• From: hazel.campbell@ualberta.ca
  • Email address of the person making the requests
  • Example: bot owner, so people can contact them about their bot if it misbehaves

HTTP Request Headers

• Host: ualberta.ca
  • The hostname (and sometimes port) of the website the user agent is trying to connect to
  • When a single server or proxy is handling requests for many different websites, it needs to know which site the request was made to
  • Otherwise it only can differentiate by IP address, but server/proxy usually has only one public IP address
  • Required in HTTP/1.1 and later for all requests
  • If Host: is missing the server may respond with 400 Bad Request

HTTP Request Headers

• If-Match: "705a092f59b73758dbb458f6e654a077d81c938e"
  • Asks the server to send the content only if the ETag matches the specified string
• If-None-Match: "705a092f59b73758dbb458f6e654a077d81c938e"
  • Asks the server to send the content only if the ETag doesn't match the specified string
• If-Modified-Since: Tue, 22 Jan 2019 23:15:50 GMT
  • Asks the server to send the content only if it's changed recently
• If-Unmodified-Since: Tue, 22 Jan 2019 23:15:50 GMT
  • Asks the server to accept the request only if it hasn't changed recently
• If-Range: Tue, 22 Jan 2019 23:15:50 GMT
  • Only resume download if I'm still going to download the same version, otherwise start over

HTTP Request Headers

• If-Range: Tue, 22 Jan 2019 23:15:50 GMT
  • Only resume download if I'm still going to download the same version, otherwise start over
• Origin: https://ualberta.ca
  • Tells the server where the JS code making this request came from
  • Broken until Firefox 65...
• Proxy-Authorization
  • Log into forward proxy server

HTTP Request Headers

• Range: bytes=10485760-
  • Resume download at the 10MiB point
• Referer: https://mysecretblog.com/
  • Tells the server what URI you were viewing that caused you to make the current request
  • Shows what page had the link you clicked to get to the current page
  • Destroys privacy
  • Lets server admins know, for example, that people are finding their page on Google, or Twitter, etc.
  • Misspelling of "referrer"

HTTP Request Headers

• Sec-Fetch-Dest: image
  Sec-Fetch-Mode: same-origin
  Sec-Fetch-Site: same-origin
  Sec-Fetch-User: ?1
  Sec-Purpose: prefetch
  • Web browser informs server why its getting something and for what page/site

HTTP Request Headers

• TE: gzip
  • Like Accept-Encoding but used with a proxy
• Upgrade-Insecure-Requests: 1
  • Asks the server to send a redirect to the HTTPS version of the page
• User-Agent:
  • Tells the server what version of browser/client is making the request

HTTP Server Headers

• Accept-Patch: application/merge-patch+json
  • Used with HTTP PATCH only.
• Accept-Post: image/webp
  • Used with HTTP POST only.
• Accept-Ranges: bytes
  • Used with resumable downloads to specify the units. Useless: they're always in bytes.

HTTP Server Headers

• Access-Control-Allow-Credentials: true
  • Whether to allow JS code running in the browser to make requests with cookies
• Access-Control-Allow-Headers: true
  • Whether to allow JS code running in the browser to make requests with extra headers
• Access-Control-Allow-Methods: GET, POST, OPTIONS
  • What methods the server will allow JS code running in the browser to make

HTTP Server Headers

• Access-Control-Allow-Origin: https://developer.mozilla.org
  • Only allow JS that came from a certain host to make requests to this server
• Access-Control-Expose-Headers: Content-Length
  • Whether to allow JS code running in the browser to see headers
• Access-Control-Max-Age: 600
  • How long the browser can remember the other access-control headers

HTTP Server Headers

• Age: 24
  • Tells the browser the content came from a (reverse) proxy cache that's e.g. 24 seconds old
• Allow: GET, POST, HEAD
  • Tells the user agent what HTTP methods the server supports, use with 405 Method Not Allowed.
• Connection: close
  • Tells the server to close the connection after it's done sending the content

HTTP Server Headers

• Content-Disposition: attachment; filename="filename.jpg"
  • Tells the browser to prompt the user to save the content with a default filename, instead of displaying the content
• Content-Encoding: gzip
  • Tells the browser to decompress the content before using it and what format its compressed in
• Content-Language: fr-CA
  • Tells the browser what natural language (e.g. Canadian French) the content its sending is in

HTTP Server Headers

• Content-Length: 500
  • Tells the client how many bytes of content to expect
• Content-Location: http://myawesomewebsite.com/fr/
  • Tells the client where it can find the content that matches the Accept, Accept-* headers it sent
  • Tells the client where it can find the content that it created using, e.g. POST
• Content-Range: bytes 10485760-11534336/20971520
  • Tells the client what bytes of the requested content the server is sending, out of the total length in bytes
  • Example: 1MiB starting at the 10MiB of a 20Mib file

HTTP Server Headers

• Content-Security-Policy: default-src https: 'unsafe-eval' 'unsafe-inline'; object-src 'none'
  • Used to combat XSS attacks
  • Example: Restrict the places that the User Agent should fetch and run JS from
• Content-Security-Policy-Report-Only: default-src https:; report-uri /csp-violation-report-endpoint/
  • Used to debug Content-Security-Policy

HTTP Server Headers

• Content-Type: text/html; charset=utf-8
  • Tells the user agent what kind of media the server's sending
• ETag: "705a092f59b73758dbb458f6e654a077d81c938e"
  • Version number/name of the content. If it changes it means the content has changed
  • Example: a SHA1 hash of the content
• Expect-CT: max-age=86400, enforce, report-uri="https://foo.example/report"
  • Tells the web browser to double-check the TLS certificate with a public certificate log service

HTTP Server Headers

• Date: Tue, 22 Jan 2019 23:15:50 GMT
  • The date and time the server sent the content
• Expires: Tue, 23 Jan 2019 23:15:50 GMT
  • Tells the user agent or caching proxy to cache a response only until the specified date and time
• Last-Modified: Tue, 23 Jan 2019 23:15:50 GMT
  • Tells the user agent or caching proxy when the content last actually changed

HTTP Server Headers

• Keep-Alive: timeout=5, max=1000
  • Tells the client not to send more than 1000 requests on this connection or to let it idle for more than 5 seconds
  • HTTP/1 only, forbidden (and useless) in HTTP/2 and HTTP/3
• Location: http://ualberta.ca/
  • Tells the client where to redirect to for 3XX redirects
• Proxy-Authenticate, Proxy-Authorization
  • Ask client to log into proxy server

HTTP Server Headers

• Public-Key-Pins: pin-sha256="cUPcTAZWKaASuYWhhneDttWpY3oBAkE3h2+soZS7sWs="; pin-sha256= "M8HztCzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGPWE="; max-age=5184000; includeSubDomains; report-uri="https://www.example.org/hpkp-report"
  • Tells the client not to only accept one of the two specified certificates from the webserver for the next 60 days
  • Dangerous: can lock people out of being able to view your site for a long time
  • Disabled in Chrome

HTTP Server Headers

• Referrer-Policy: no-referrer-when-downgrade
  • Webserver tells the client not to send as much referrer information to the next webserver
• Retry-After: 60
  • Webserver tells the client to wait 1 minute before retrying after a 503 Service Unavailable, 429 Too Many Requests or how long to delay before following a 3xx redirect

HTTP Server Headers

• Server: mycoolwebserveriwroteincmput404/0.1 (Unix)
  • Tells the client the server software and version, like User Agent but for servers
• Set-Cookie: session=1nhbn4b123h4; Secure; HttpOnly; SameSite=Strict; MaxAge=86400
  • Webserver tells the client to store a cookie (key-value pair) in its cookie jar
  • The cookie will be stored and sent back to the webserver every time the client makes a request, using the Cookie: header
  • But only if the security restrictions are met: only over HTTPS, not with requests made by JS, not with requests initiated by code that came from a different website, delete the cookie after a day...

HTTP Server Headers

• Transfer-Encoding: gzip
  • Like Content-Encoding but used by a proxy
• Transfer-Encoding: chunked
  • Data will be sent in chunks. Replaced by HTTP/2.

HTTP Server Headers

• Strict-Transport-Security: max-age=31536000; includeSubDomains aka HSTS
  • Don't contact this server or any subdomains without TLS for the next year
• Trailer: Expires
  • Tells the client there will be additional headers after the content is sent
  • HTTP/1.1 requires Transfer-Encoding: Chunked for this to work
• Vary: Accept-Language
  • Tells a cache that this page is one of multiple versions of the page based on the HTTP request headers listed

HTTP Server Headers

• Via: HTTP/1.1 fancyproxy.example
  • Records the proxy chain a request or response went through
• WWW-Authenticate: Basic realm="What is the password", charset="UTF-8"
  • Sent with 401 Unauthorized to tell the user agent/user how to authenticate
  • Browser will pop up a dialog box with "What is the password" on it

HTTP Server Headers

• Warning: 110 fancyproxy/1.3.37 "Response is Stale"
  • Warns the client that the content was in the cache for too long
• Warning: 111 fancyproxy/1.3.37 "Revalidation Failed"
  • Warns the client that the content was in the cache but the original server is down
• Warning: 112 fancyproxy/1.3.37 "Disconnected Operation"
  • Warns the client that the content was in the cache but the network is down

HTTP Server Headers

• Warning: 113 fancyproxy/1.3.37 "Heuristic Expiration"
  • Warns the client that the content was in the cache over 24 hours
• Warning: 199 fancyproxy/1.3.37 "Miscellaneous Warning"
  • Warns the client about something
• Warning: 214 fancyproxy/1.3.37 "Transformation Applied"
  • Warns the client that the content was changed in some way by the proxy
• Warning: 299 fancyproxy/1.3.37 "Persistent Warning"
  • Warns the client about something the proxy doesn't expect to change anytime soon

HTTP Server Headers

• X-Content-Type-Options: nosniff
  • Tells the client not to try to figure out the Content Type (MIME Type) of the content on its own, e.g. based on contents
• X-DNS-Prefetch-Control: off
  • Tells the browser not to resolve hostnames in links before the user clicks on them
• X-Frame-Options: deny
  • Tells the browser not to show the page in a <frame>, <iframe>, <embed> or <object>

Custom HTTP Headers

• Used to have use X- headers, but X- headers are now deprecated
• You can add whatever headers you want as long as they don't conflict with standard headers

License

Copyright 2014-2023 ⓒ Abram Hindle

Copyright 2019-2023 ⓒ Hazel Victoria Campbell and contributors

Other images used under fair use and copyright their copyright holders.

License

Copyright (C) 2019-2023 Hazel Victoria Campbell
Copyright (C) 2014-2023 Abram Hindle and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN.

01234567890123456789012345678901234567890123456789012345678901234567890123456789