HTTP.request(method, url [, headers [, body]]; <keyword arguments>]) -> HTTP.Response

Send a HTTP Request Message and recieve a HTTP Response Message.

e.g.

r = HTTP.request("GET", "http://httpbin.org/ip")
println(r.status)
println(String(r.body))

headers can be any collection where [string(k) => string(v) for (k,v) in headers] yields Vector{Pair}. e.g. a Dict(), a Vector{Tuple}, a Vector{Pair} or an iterator.

body can take a number of forms:

• a String, a Vector{UInt8} or any T accepted by write(::IO, ::T)

• a collection of String or AbstractVector{UInt8} or IO streams or items of any type T accepted by write(::IO, ::T...)

• a readable IO stream or any IO-like type T for which eof(T) and readavailable(T) are defined.

The HTTP.Response struct contains:

• status::Int16 e.g. 200

• headers::Vector{Pair{String,String}} e.g. ["Server" => "Apache", "Content-Type" => "text/html"]

• body::Vector{UInt8}, the Response Body bytes (empty if a response_stream was specified in the request).

Functions HTTP.get, HTTP.put, HTTP.post and HTTP.head are defined as shorthand for HTTP.request("GET", ...), etc.

HTTP.request and HTTP.open also accept optional keyword parameters.

e.g.

HTTP.request("GET", "http://httpbin.org/ip"; retries=4, cookies=true)

HTTP.get("http://s3.us-east-1.amazonaws.com/"; aws_authorization=true)

conf = (readtimeout = 10,
        pipeline_limit = 4,
        retry = false,
        redirect = false)

HTTP.get("http://httpbin.org/ip"; conf..)
HTTP.put("http://httpbin.org/put", [], "Hello"; conf..)

URL options

• query = nothing, replaces the query part of url.

Streaming options

• response_stream = nothing, a writeable IO stream or any IO-like type T for which write(T, AbstractVector{UInt8}) is defined.

• verbose = 0, set to 1 or 2 for extra message logging.

Connection Pool options

• connection_limit = 8, number of concurrent connections to each host:port.

• pipeline_limit = 16, number of concurrent requests per connection.

• reuse_limit = nolimit, number of times a connection is reused after the first request.

• socket_type = TCPSocket

Timeout options

• readtimeout = 60, close the connection if no data is recieved for this many seconds. Use readtimeout = 0 to disable.

Retry options

• retry = true, retry idempotent requests in case of error.

• retries = 4, number of times to retry.

• retry_non_idempotent = false, retry non-idempotent requests too. e.g. POST.

Redirect options

• redirect = true, follow 3xx redirect responses.

• redirect_limit = 3, number of times to redirect.

• forwardheaders = true, forward original headers on redirect.

Status Exception options

• status_exception = true, throw HTTP.StatusError for response status >= 300.

SSLContext options

• require_ssl_verification = false, pass MBEDTLS_SSL_VERIFY_REQUIRED to the mbed TLS library. "... peer must present a valid certificate, handshake is aborted if verification failed."

• sslconfig = SSLConfig(require_ssl_verification)

Basic Authenticaiton options

• basic_authorization=false, add Authorization: Basic header using credentials from url userinfo.

AWS Authenticaiton options

• aws_authorization = false, enable AWS4 Authentication.

• aws_service = split(url.host, ".")[1]

• aws_region = split(url.host, ".")[2]

• aws_access_key_id = ENV["AWS_ACCESS_KEY_ID"]

• aws_secret_access_key = ENV["AWS_SECRET_ACCESS_KEY"]

• aws_session_token = get(ENV, "AWS_SESSION_TOKEN", "")

• body_sha256 = digest(MD_SHA256, body),

• body_md5 = digest(MD_MD5, body),

Cookie options

• cookies = false, enable cookies.

• cookiejar::Dict{String, Set{Cookie}}=default_cookiejar

Cananoincalization options

• canonicalize_headers = false, rewrite request and response headers in Canonical-Camel-Dash-Format.

Request Body Examples

String body:

HTTP.request("POST", "http://httpbin.org/post", [], "post body data")

Stream body from file:

io = open("post_data.txt", "r")
HTTP.request("POST", "http://httpbin.org/post", [], io)

Generator body:

chunks = ("chunk$i" for i in 1:1000)
HTTP.request("POST", "http://httpbin.org/post", [], chunks)

Collection body:

chunks = [preamble_chunk, data_chunk, checksum(data_chunk)]
HTTP.request("POST", "http://httpbin.org/post", [], chunks)

open() do io body:

HTTP.open("POST", "http://httpbin.org/post") do io
    write(io, preamble_chunk)
    write(io, data_chunk)
    write(io, checksum(data_chunk))
end

Response Body Examples

String body:

r = HTTP.request("GET", "http://httpbin.org/get")
println(String(r.body))

Stream body to file:

io = open("get_data.txt", "w")
r = HTTP.request("GET", "http://httpbin.org/get", response_stream=io)
close(io)
println(read("get_data.txt"))

Stream body through buffer:

io = Base.BufferStream()
@async while !eof(io)
    bytes = readavailable(io))
    println("GET data: $bytes")
end
r = HTTP.request("GET", "http://httpbin.org/get", response_stream=io)
close(io)

Stream body through open() do io:

r = HTTP.open("GET", "http://httpbin.org/stream/10") do io
   while !eof(io)
       println(String(readavailable(io)))
   end
end

using HTTP.IOExtras

HTTP.open("GET", "https://tinyurl.com/bach-cello-suite-1-ogg") do http
    n = 0
    r = startread(http)
    l = parse(Int, header(r, "Content-Length"))
    open(`vlc -q --play-and-exit --intf dummy -`, "w") do vlc
        while !eof(http)
            bytes = readavailable(http)
            write(vlc, bytes)
            n += length(bytes)
            println("streamed $n-bytes $((100*n)÷l)%\u1b[1A")
        end
    end
end

Request and Response Body Examples

String bodies:

r = HTTP.request("POST", "http://httpbin.org/post", [], "post body data")
println(String(r.body))

Stream bodies from and to files:

in = open("foo.png", "r")
out = open("foo.jpg", "w")
HTTP.request("POST", "http://convert.com/png2jpg", [], in, response_stream=out)

Stream bodies through: open() do io:

using HTTP.IOExtras

HTTP.open("POST", "http://music.com/play") do io
    write(io, JSON.json([
        "auth" => "12345XXXX",
        "song_id" => 7,
    ]))
    r = startread(io)
    @show r.status
    while !eof(io)
        bytes = readavailable(io))
        play_audio(bytes)
    end
end

HTTP.open(method, url, [,headers]) do io
    write(io, body)
    [startread(io) -> HTTP.Response]
    while !eof(io)
        readavailable(io) -> AbstractVector{UInt8}
    end
end -> HTTP.Response

The HTTP.open API allows the Request Body to be written to (and/or the Response Body to be read from) an IO stream.

e.g. Streaming an audio file to the vlc player:

HTTP.open("GET", "https://tinyurl.com/bach-cello-suite-1-ogg") do http
    open(`vlc -q --play-and-exit --intf dummy -`, "w") do vlc
        write(vlc, http)
    end
end

HTTP.get(url [, headers]; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("GET", ...). See HTTP.request.

HTTP.put(url, headers, body; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("PUT", ...). See HTTP.request.

HTTP.post(url, headers, body; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("POST", ...). See HTTP.request.

HTTP.head(url; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("HEAD", ...). See HTTP.request.

The Response has a 4xx, 5xx or unrecognised status code.

Fields:

• status::Int16, the response status code.

• response the HTTP.Response

Parser input was invalid.

Fields:

• code, error code

• bytes, the offending input.

The request terminated with due to an IO-related error.

Fields:

• e, the error.

HTTP.listen([host=Sockets.localhost[, port=8081]]; <keyword arguments>) do http
    ...
end

Listen for HTTP connections and execute the do function for each request.

Optional keyword arguments:

• ssl::Bool = false, use https.

• require_ssl_verification = true, pass MBEDTLS_SSL_VERIFY_REQUIRED to the mbed TLS library. "... peer must present a valid certificate, handshake is aborted if verification failed."

• sslconfig = SSLConfig(require_ssl_verification)

• pipeline_limit = 16, number of concurrent requests per connection.

• reuse_limit = nolimit, number of times a connection is allowed to be reused after the first request.

• tcpisvalid::Function (::TCPSocket) -> Bool, check accepted connection before processing requests. e.g. to implement source IP filtering, rate-limiting, etc.

• tcpref::Ref{Base.IOServer}, this reference is set to the underlying IOServer. e.g. to allow closing the server.

e.g.

    HTTP.listen() do http::HTTP.Stream
        @show http.message
        @show HTTP.header(http, "Content-Type")
        while !eof(http)
            println("body data: ", String(readavailable(http)))
        end
        setstatus(http, 404)
        setheader(http, "Foo-Header" => "bar")
        startwrite(http)
        write(http, "response body")
        write(http, "more response body")
    end

    HTTP.listen() do request::HTTP.Request
        @show HTTP.header(request, "Content-Type")
        @show HTTP.payload(request)
        return HTTP.Response(404)
    end

HTTP.serve([server,] host::Union{IPAddr, String}, port::Integer; verbose::Bool=true, kwargs...)
HTTP.serve([server,] host::InetAddr; verbose::Bool=true, kwargs...)
HTTP.serve([server,] host::String; verbose::Bool=true, kwargs...)

Start a server listening on the provided host:port. verbose indicates whether server activity should be logged. Optional keyword arguments allow construction of Server on the fly if the server argument isn't provided directly. See ?HTTP.Server for more details on server construction and supported keyword arguments. By default, HTTP.serve aims to "never die", catching and recovering from all internal errors. Two methods for stopping HTTP.serve include interrupting (ctrl/cmd+c) if blocking on the main task, or sending the kill signal via the server's in channel (put!(server.in, HTTP.Servers.KILL)).

Server(handler, logger::IO=stdout; kwargs...)

An http/https server. Supports listening on a host and port via the HTTP.serve(server, host, port) function. handler is a function of the form f(::Request, ::Response) -> HTTP.Response, i.e. it takes both a Request and pre-built Response objects as inputs and returns the, potentially modified, Response. logger indicates where logging output should be directed. When HTTP.serve is called, it aims to "never die", catching and recovering from all internal errors. To forcefully stop, one can obviously kill the julia process, interrupt (ctrl/cmd+c) if main task, or send the kill signal over a server in channel like: put!(server.in, HTTP.Servers.KILL).

Supported keyword arguments include:

• cert: if https, the cert file to use, as passed to HTTP.MbedTLS.SSLConfig(cert, key)

• key: if https, the key file to use, as passed to HTTP.MbedTLS.SSLConfig(cert, key)

• sslconfig: pass in an already-constructed HTTP.MbedTLS.SSLConfig instance

• readtimeout: how long a client connection will be left open without receiving any bytes

• ratelimit: a Rational{Int} of the form 5//1 indicating how many messages//second should be allowed per client IP address; requests exceeding the rate limit will be dropped

• support100continue: a Bool indicating whether Expect: 100-continue headers should be supported for delayed request body sending; default = true

• logbody: whether the Response body should be logged when verbose=true logging is enabled; default = true

Abstract type representing an object that knows how to "handle" a server request.

Types of handlers include HandlerFunction (a julia function of the form f(request) and Router (which pattern matches request url paths to other specific Handler types).

HandlerFunction(f::Function)

A Function-wrapper type that is a subtype of Handler. Takes a single Function as an argument. The provided argument should be of the form f(request) => Response, i.e. it accepts a Request returns a Response.

Router(h::Handler) Router(f::Function) Router()

An HTTP.Handler type that supports mapping request url paths to other HTTP.Handler types. Can accept a default Handler or Function that will be used in case no other handlers match; by default, a 404 response handler is used. Paths can be mapped to a handler via HTTP.register!(r::Router, path, handler), see ?HTTP.register! for more details.

HTTP.register!(r::Router, url, handler) HTTP.register!(r::Router, m::String, url, handler)

Function to map request urls matching url and an optional method m to another handler::HTTP.Handler. URLs are registered one at a time, and multiple urls can map to the same handler. The URL can be passed as a String or HTTP.URI object directly. Requests can be routed based on: method, scheme, hostname, or path. The following examples show how various urls will direct how a request is routed by a server:

• "http://*": match all HTTP requests, regardless of path

• "https://*": match all HTTPS requests, regardless of path

• "google": regardless of scheme, match requests to the hostname "google"

• "google/gmail": match requests to hostname "google", and path starting with "gmail"

• "/gmail": regardless of scheme or host, match any request with a path starting with "gmail"

• "/gmail/userId/*/inbox: match any request matching the path pattern, "*" is used as a wildcard that matches any value between the two "/"

HTTP.URI(; scheme="", host="", port="", etc...)
HTTP.URI(str) = parse(HTTP.URI, str::String)

A type representing a valid uri. Can be constructed from distinct parts using the various supported keyword arguments. With a raw, already-encoded uri string, use parse(HTTP.URI, str) to parse the HTTP.URI directly. The HTTP.URI constructors will automatically escape any provided query arguments, typically provided as "key"=>"value"::Pair or Dict("key"=>"value"). Note that multiple values for a single query key can provided like Dict("key"=>["value1", "value2"]).

The URI struct stores the compelte URI in the uri::String field and the component parts in the following SubString fields:

• scheme, e.g. "http" or "https"

• userinfo, e.g. "username:password"

• host e.g. "julialang.org"

• port e.g. "80" or ""

• path e.g "/"

• query e.g. "Foo=1&Bar=2"

• fragment

The HTTP.resource(::URI) function returns a target-resource string for the URI RFC7230 5.3. e.g. "$path?$query#$fragment".

The HTTP.queryparams(::URI) function returns a Dict containing the query.

percent-encode a string, dict, or pair for a uri

unescape a percent-encoded uri/url

Splits the path into components See: http://tools.ietf.org/html/rfc3986#section-3.3

checks if a HTTP.URI is valid

Cookie()
Cookie(; kwargs...)
Cookie(name, value; kwargs...)

A Cookie represents an HTTP cookie as sent in the Set-Cookie header of an HTTP response or the Cookie header of an HTTP request. Supported fields (which can be set using keyword arguments) include:

• name: name of the cookie

• value: value of the cookie

• path: applicable path for the cookie

• domain: applicable domain for the cookie

• expires: a Dates.DateTime representing when the cookie should expire

• maxage: maxage == 0 means no max age, maxage < 0 means delete cookie now, max age > 0 means the # of seconds until expiration

• secure::Bool: secure cookie attribute

• httponly::Bool: httponly cookie attribute

• hostonly::Bool: hostonly cookie attribute

See http:#tools.ietf.org/html/rfc6265 for details.

HTTP.sniff(content::Union{Vector{UInt8}, String, IO}) => String (mimetype)

HTTP.sniff will look at the first 512 bytes of content to try and determine a valid mimetype. If a mimetype can't be determined appropriately, "application/octet-stream" is returned.

Supports JSON detection through the HTTP.isjson(content) function.

escapeHTML(i::String)

Returns a string with special HTML characters escaped: &, <, >, ", '

Request Execution Stack

The Request Execution Stack is separated into composable layers.

Each layer is defined by a nested type Layer{Next} where the Next parameter defines the next layer in the stack. The request method for each layer takes a Layer{Next} type as its first argument and dispatches the request to the next layer using request(Next, ...).

The example below defines three layers and three stacks each with a different combination of layers.

abstract type Layer end
abstract type Layer1{Next <: Layer} <: Layer end
abstract type Layer2{Next <: Layer} <: Layer end
abstract type Layer3 <: Layer end

request(::Type{Layer1{Next}}, data) where Next = "L1", request(Next, data)
request(::Type{Layer2{Next}}, data) where Next = "L2", request(Next, data)
request(::Type{Layer3}, data) = "L3", data

const stack1 = Layer1{Layer2{Layer3}}
const stack2 = Layer2{Layer1{Layer3}}
const stack3 = Layer1{Layer3}

julia> request(stack1, "foo")
("L1", ("L2", ("L3", "foo")))

julia> request(stack2, "bar")
("L2", ("L1", ("L3", "bar")))

julia> request(stack3, "boo")
("L1", ("L3", "boo"))

This stack definition pattern gives the user flexibility in how layers are combined but still allows Julia to do whole-stack comiple time optimistations.

e.g. the request(stack1, "foo") call above is optimised down to a single function:

julia> code_typed(request, (Type{stack1}, String))[1].first
CodeInfo(:(begin
    return (Core.tuple)("L1", (Core.tuple)("L2", (Core.tuple)("L3", data)))
end))

The stack() function returns the default HTTP Layer-stack type. This type is passed as the first parameter to the HTTP.request function.

stack() accepts optional keyword arguments to enable/disable specific layers in the stack: request(method, args...; kw...) request(stack(;kw...), args...; kw...)

The minimal request execution stack is:

stack = MessageLayer{ConnectionPoolLayer{StreamLayer}}

The figure below illustrates the full request exection stack and its relationship with HTTP.Response, HTTP.Parsers, HTTP.Stream and the HTTP.ConnectionPool.

 ┌────────────────────────────────────────────────────────────────────────────┐
 │                                            ┌───────────────────┐           │
 │  HTTP.jl Request Execution Stack           │ HTTP.ParsingError ├ ─ ─ ─ ─ ┐ │
 │                                            └───────────────────┘           │
 │                                            ┌───────────────────┐         │ │
 │                                            │ HTTP.IOError      ├ ─ ─ ─     │
 │                                            └───────────────────┘      │  │ │
 │                                            ┌───────────────────┐           │
 │                                            │ HTTP.StatusError  │─ ─   │  │ │
 │                                            └───────────────────┘   │       │
 │                                            ┌───────────────────┐      │  │ │
 │     request(method, url, headers, body) -> │ HTTP.Response     │   │       │
 │             ──────────────────────────     └─────────▲─────────┘      │  │ │
 │                           ║                          ║             │       │
 │   ┌────────────────────────────────────────────────────────────┐      │  │ │
 │   │ request(RedirectLayer,     method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(BasicAuthLayer,    method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(CookieLayer,       method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(CanonicalizeLayer, method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(MessageLayer,      method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(AWS4AuthLayer,             ::URI, ::Request, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(RetryLayer,                ::URI, ::Request, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(ExceptionLayer,            ::URI, ::Request, body) ├ ─ ┘       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
┌┼───┤ request(ConnectionPoolLayer,       ::URI, ::Request, body) ├ ─ ─ ─     │
││   ├────────────────────────────────────────────────────────────┤         │ │
││   │ request(DebugLayer,                ::IO,  ::Request, body) │           │
││   ├────────────────────────────────────────────────────────────┤         │ │
││   │ request(TimeoutLayer,              ::IO,  ::Request, body) │           │
││   ├────────────────────────────────────────────────────────────┤         │ │
││   │ request(StreamLayer,               ::IO,  ::Request, body) │           │
││   └──────────────┬───────────────────┬─────────────────────────┘         │ │
│└──────────────────┼────────║──────────┼───────────────║─────────────────────┘
│                   │        ║          │               ║                   │  
│┌──────────────────▼───────────────┐   │  ┌──────────────────────────────────┐
││ HTTP.Request                     │   │  │ HTTP.Response                  │ │
││                                  │   │  │                                  │
││ method::String                   ◀───┼──▶ status::Int                    │ │
││ target::String                   │   │  │ headers::Vector{Pair}            │
││ headers::Vector{Pair}            │   │  │ body::Vector{UInt8}            │ │
││ body::Vector{UInt8}              │   │  │                                  │
│└──────────────────▲───────────────┘   │  └───────────────▲────────────────┼─┘
│┌──────────────────┴────────║──────────▼───────────────║──┴──────────────────┐
││ HTTP.Stream <:IO          ║           ╔══════╗       ║                   │ │
││   ┌───────────────────────────┐       ║   ┌──▼─────────────────────────┐   │
││   │ startwrite(::Stream)      │       ║   │ startread(::Stream)        │ │ │
││   │ write(::Stream, body)     │       ║   │ read(::Stream) -> body     │   │
││   │ ...                       │       ║   │ ...                        │ │ │
││   │ closewrite(::Stream)      │       ║   │ closeread(::Stream)        │   │
││   └───────────────────────────┘       ║   └────────────────────────────┘ │ │
│└───────────────────────────║────────┬──║──────║───────║──┬──────────────────┘
│┌──────────────────────────────────┐ │  ║ ┌────▼───────║──▼────────────────┴─┐
││ HTTP.Messages                    │ │  ║ │ HTTP.Parsers                     │
││                                  │ │  ║ │                                  │
││ writestartline(::IO, ::Request)  │ │  ║ │ parse_status_line(bytes, ::Req') │
││ writeheaders(::IO, ::Request)    │ │  ║ │ parse_header_field(bytes, ::Req')│
│└──────────────────────────────────┘ │  ║ └──────────────────────────────────┘
│                            ║        │  ║                                     
│┌───────────────────────────║────────┼──║────────────────────────────────────┐
└▶ HTTP.ConnectionPool       ║        │  ║                                    │
 │                     ┌──────────────▼────────┐ ┌───────────────────────┐    │
 │ getconnection() ->  │ HTTP.Transaction <:IO │ │ HTTP.Transaction <:IO │    │
 │                     └───────────────────────┘ └───────────────────────┘    │
 │                           ║    ╲│╱    ║                  ╲│╱               │
 │                           ║     │     ║                   │                │
 │                     ┌───────────▼───────────┐ ┌───────────▼───────────┐    │
 │              pool: [│ HTTP.Connection       │,│ HTTP.Connection       │...]│
 │                     └───────────┬───────────┘ └───────────┬───────────┘    │
 │                           ║     │     ║                   │                │
 │                     ┌───────────▼───────────┐ ┌───────────▼───────────┐    │
 │                     │ Base.TCPSocket <:IO   │ │MbedTLS.SSLContext <:IO│    │
 │                     └───────────────────────┘ └───────────┬───────────┘    │
 │                           ║           ║                   │                │
 │                           ║           ║       ┌───────────▼───────────┐    │
 │                           ║           ║       │ Base.TCPSocket <:IO   │    │
 │                           ║           ║       └───────────────────────┘    │
 └───────────────────────────║───────────║────────────────────────────────────┘
                             ║           ║                                     
 ┌───────────────────────────║───────────║──────────────┐  ┏━━━━━━━━━━━━━━━━━━┓
 │ HTTP Server               ▼                          │  ┃ data flow: ════▶ ┃
 │                        Request     Response          │  ┃ reference: ────▶ ┃
 └──────────────────────────────────────────────────────┘  ┗━━━━━━━━━━━━━━━━━━┛

See docs/src/layers.monopic.

request(RedirectLayer, method, ::URI, headers, body) -> HTTP.Response

Redirects the request in the case of 3xx response status.

request(BasicAuthLayer, method, ::URI, headers, body) -> HTTP.Response

Add Authorization: Basic header using credentials from url userinfo.

request(CookieLayer, method, ::URI, headers, body) -> HTTP.Response

Add locally stored Cookies to the request headers. Store new Cookies found in the response headers.

request(CanonicalizeLayer, method, ::URI, headers, body) -> HTTP.Response

Rewrite request and response headers in Canonical-Camel-Dash-Format.

request(MessageLayer, method, ::URI, headers, body) -> HTTP.Response

Construct a Request object and set mandatory headers.

request(AWS4AuthLayer, ::URI, ::Request, body) -> HTTP.Response

Add a AWS Signature Version 4 Authorization header to a Request.

Credentials are read from environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.

request(RetryLayer, ::URI, ::Request, body) -> HTTP.Response

Retry the request if it throws a recoverable exception.

Base.retry and Base.ExponentialBackOff implement a randomised exponentially increasing delay is introduced between attempts to avoid exacerbating network congestion.

Methods of isrecoverable(e) define which exception types lead to a retry. e.g. HTTP.IOError, Sockets.DNSError, Base.EOFError and HTTP.StatusError (if status is `5xx).

request(ExceptionLayer, ::URI, ::Request, body) -> HTTP.Response

Throw a StatusError if the request returns an error response status.

request(ConnectionPoolLayer, ::URI, ::Request, body) -> HTTP.Response

Retrieve an IO connection from the ConnectionPool.

Close the connection if the request throws an exception. Otherwise leave it open so that it can be reused.

IO related exceptions from Base are wrapped in HTTP.IOError. See isioerror.

request(TimeoutLayer, ::IO, ::Request, body) -> HTTP.Response

Close IO if no data has been received for timeout seconds.

request(StreamLayer, ::IO, ::Request, body) -> HTTP.Response

Create a Stream to send a Request and body to an IO stream and read the response.

Send the Request body in a background task and begins reading the response immediately so that the transmission can be aborted if the Response status indicates that the server does not wish to receive the message body. RFC7230 6.5.

The parser separates a raw HTTP Message into its component parts.

If the input data is invalid the Parser throws a HTTP.ParseError.

The parse_* functions processes a single element of a HTTP Message at a time and return a SubString containing the unused portion of the input.

The Parser does not interpret the Message Headers. It is beyond the scope of the Parser to deal with repeated header fields, multi-line values, cookies or case normalization.

The Parser has no knowledge of the high-level Request and Response structs defined in Messages.jl. However, the Request and Response structs must have field names compatible with those expected by the parse_status_line! and parse_request_line! functions.

The Messages module defines structs that represent HTTP.Request and HTTP.Response Messages.

The Response struct has a request field that points to the corresponding Request; and the Request struct has a response field. The Request struct also has a parent field that points to a Response in the case of HTTP Redirect.

The Messages module defines IO read and write methods for Messages but it does not deal with URIs, creating connections, or executing requests.

The read methods throw EOFError exceptions if input data is incomplete. and call parser functions that may throw HTTP.ParsingError exceptions. The read and write methods may also result in low level IO exceptions.

Sending Messages

Messages are formatted and written to an IO stream by Base.write(::IO,::HTTP.Messages.Message) and or HTTP.Messages.writeheaders.

Receiving Messages

Messages are parsed from IO stream data by HTTP.Messages.readheaders. This function calls HTTP.Parsers.parse_header_field and passes each header-field to HTTP.Messages.appendheader.

readheaders relies on HTTP.IOExtras.unread! to push excess data back to the input stream.

Headers

Headers are represented by Vector{Pair{String,String}}. As compared to Dict{String,String} this allows repeated header fields and preservation of order.

Header values can be accessed by name using HTTP.Messages.header and HTTP.Messages.setheader (case-insensitive).

The HTTP.Messages.appendheader function handles combining multi-line values, repeated header fields and special handling of multiple Set-Cookie headers.

Bodies

The HTTP.Message structs represent the Message Body as Vector{UInt8}.

Streaming of request and response bodies is handled by the HTTP.StreamLayer and the HTTP.Stream <: IO stream.

Stream(::IO, ::Request)

Creates a HTTP.Stream that wraps an existing IO stream.

• startwrite(::Stream) sends the Request headers to the IO stream.

• write(::Stream, body) sends the body (or a chunk of the body).

• closewrite(::Stream) sends the final 0 chunk (if needed) and calls closewrite on the IO stream. When the IO stream is a HTTP.ConnectionPool.Transaction, calling closewrite releases the HTTP.ConnectionPool.Connection back into the pool for use by the next pipelined request.

• startread(::Stream) calls startread on the IO stream then reads and parses the Response headers. When the IO stream is a HTTP.ConnectionPool.Transaction, calling startread waits for other pipelined responses to be read from the HTTP.ConnectionPool.Connection.

• eof(::Stream) and readavailable(::Stream) parse the body from the IO stream.

• closeread(::Stream) reads the trailers and calls closeread on the IO stream. When the IO stream is a HTTP.ConnectionPool.Transaction, calling closeread releases the readlock and allows the next pipelined response to be read by another Stream that is waiting in startread. If a complete response has not been recieved, closeread throws EOFError.

This module provides the getconnection function with support for:

• Opening TCP and SSL connections.

• Reusing connections for multiple Request/Response Messages,

• Pipelining Request/Response Messages. i.e. allowing a new Request to be sent before previous Responses have been read.

This module defines a Connection struct to manage pipelining and connection reuse and a Transaction<: IO struct to manage a single pipelined request. Methods are provided for eof, readavailable, unsafe_write and close. This allows the Transaction object to act as a proxy for the TCPSocket or SSLContext that it wraps.

The pool is a collection of open Connections. The request function calls getconnection to retrieve a connection from the pool. When the request function has written a Request Message it calls closewrite to signal that the Connection can be reused for writing (to send the next Request). When the request function has read the Response Message it calls closeread to signal that the Connection can be reused for reading.

Request <: Message

Represents a HTTP Request Message.

• method::String RFC7230 3.1.1

• target::String RFC7230 5.3

• version::VersionNumber RFC7230 2.6

• headers::Vector{Pair{String,String}} RFC7230 3.2

• body::Vector{UInt8} RFC7230 3.3

• response, the Response to this Request

• txcount, number of times this Request has been sent (see RetryRequest.jl).

• parent, the Response (if any) that led to this request (e.g. in the case of a redirect). RFC7230 6.4

Response <: Message

Represents a HTTP Response Message.

• version::VersionNumber RFC7230 2.6

• status::Int16 RFC7230 3.1.2 RFC7231 6

• headers::Vector{Pair{String,String}} RFC7230 3.2

• body::Vector{UInt8} RFC7230 3.3

• request, the Request that yielded this Response.

iserror(::Response)

Does this Response have an error status?

isredirect(::Response)

Does this Response have a redirect status?

ischunked(::Message)

Does the Message have a "Transfer-Encoding: chunked" header?

issafe(::Request)

https://tools.ietf.org/html/rfc7231#section-4.2.1

isidempotent(::Request)

https://tools.ietf.org/html/rfc7231#section-4.2.2

header(::Message, key [, default=""]) -> String

Get header value for key (case-insensitive).

hasheader(::Message, key) -> Bool

Does header value for key exist (case-insensitive)?

hasheader(::Message, key, value) -> Bool

Does header for key match value (both case-insensitive)?

setheader(::Message, key => value)

Set header value for key (case-insensitive).

defaultheader(::Message, key => value)

Set header value for key if it is not already set.

appendheader(::Message, key => value)

Append a header value to message.headers.

If key is "" the value is appended to the value of the previous header.

If key is the same as the previous header, the value is appended to the value of the previous header with a comma delimiter

Set-Cookie headers are not comma-combined because cookies often contain internal commas.

readheaders(::IO, ::Message)

Read headers (and startline) from an IO stream into a Message struct. Throw EOFError if input is incomplete.

Set the default User-Agent string to be used in each HTTP request; when set, can be manually overridden by passing in an explicit User-Agent header

Read chunk-size from an IO stream. After the final zero size chunk, read trailers into a Message struct.

headerscomplete(::Message)

Have the headers been read into this Message?

writestartline(::IO, ::Message)

e.g. "GET /path HTTP/1.1\r\n" or "HTTP/1.1 200 OK\r\n"

writeheaders(::IO, ::Message)

Write Message start line and a line for each "name: value" pair and a trailing blank line.

write(::IO, ::Message)

Write start line, headers and body of HTTP Message.

closebody(::Stream)

Write the final 0 chunk if needed.

isaborted(::Stream{Response})

Has the server signaled that it does not wish to receive the message body?

"If [the response] indicates the server does not wish to receive the message body and is closing the connection, the client SHOULD immediately cease transmitting the body and close the connection." RFC7230, 6.5

Connection{T <: IO}

A TCPSocket or SSLContext connection to a HTTP host and port.

Fields:

• host::String

• port::String, exactly as specified in the URI (i.e. may be empty).

• pipeline_limit, number of requests to send before waiting for responses.

• idle_timeout, No. of sconds to maintain connection after last transaction.

• peerport, remote TCP port number (used for debug messages).

• localport, local TCP port number (used for debug messages).

• io::T, the TCPSocket or `SSLContext.

• excess::ByteView, left over bytes read from the connection after the end of a response message. These bytes are probably the start of the next response message.

• sequence, number of most recent Transaction.

• writecount, number of Messages that have been written.

• writedone, signal that writecount was incremented.

• readcount, number of Messages that have been read.

• readdone, signal that readcount was incremented.

• timestamp, time data was last recieved.

A single pipelined HTTP Request/Response transaction`.

Fields:

• c, the shared Connection used for this Transaction.

• sequence::Int, identifies this Transaction among the others that share c.

The pool is a collection of open Connections. The request function calls getconnection to retrieve a connection from the pool. When the request function has written a Request Message it calls closewrite to signal that the Connection can be reused for writing (to send the next Request). When the request function has read the Response Message it calls closeread to signal that the Connection can be reused for reading.

getconnection(type, host, port) -> Connection

Find a reusable Connection in the pool, or create a new Connection if required.

unread!(::Transaction, bytes)

Push bytes back into a connection's excess buffer (to be returned by the next read).

startwrite(::Transaction)

Wait for prior pending writes to complete.

closewrite(::Transaction)

Signal that an entire Request Message has been written to the Transaction.

startread(::Transaction)

Wait for prior pending reads to complete.

closeread(::Transaction)

Signal that an entire Response Message has been read from the Transaction.

Increment readcount and wake up tasks waiting in startread.