Edit on GitHub

HTTP.jl Documentation

Overview

HTTP.jl provides both client and server functionality for the http and websocket protocols. As a client, it provides the ability to make a wide range of requests, including GET, POST, websocket upgrades, form data, multipart, chunking, and cookie handling. There is also advanced functionality to provide client-side middleware and generate your own customized HTTP client. On the server side, it provides the ability to listen, accept, and route http requests, with middleware and handler interfaces to provide flexibility in processing responses.

Quickstart

Making requests (client)

HTTP.request sends an http request and returns a response.

# make a GET request, both forms are equivalent
resp = HTTP.request("GET", "http://httpbin.org/ip")
resp = HTTP.get("http://httpbin.org/ip")
println(resp.status)
println(String(resp.body))

# make a POST request, sending data via `body` keyword argument
resp = HTTP.post("http://httpbin.org/body"; body="request body")

# make a POST request, sending form-urlencoded body
resp = HTTP.post("http://httpbin.org/body"; body=Dict("nm" => "val"))

# include query parameters in a request
# and turn on verbose logging of the request/response process
resp = HTTP.get("http://httpbin.org/anything"; query=["hello" => "world"], verbose=2)

# simple websocket client
WebSockets.open("ws://websocket.org") do ws
    # we can iterate the websocket
    # where each iteration yields a received message
    # iteration finishes when the websocket is closed
    for msg in ws
        # do stuff with msg
        # send back message as String, Vector{UInt8}, or iterable of either
        send(ws, resp)
    end
end

Handling requests (server)

HTTP.serve allows specifying middleware + handlers for how incoming requests should be processed.

# authentication middleware to ensure property security
function auth(handler)
    return function(req)
        ident = parse_auth(req)
        if ident === nothing
            # failed to security authentication
            return HTTP.Response(401, "unauthorized")
        else
            # store parsed identity in request context for handler usage
            req.context[:auth] = ident
            # pass request on to handler function for further processing
            return handler(req)
        end
    end
end

# handler function to return specific user's data
function handler(req)
    ident = req.context[:auth]
    return HTTP.Response(200, get_user_data(ident))
end

# start a server listening on port 8081 (default port) for localhost (default host)
# requests will first be handled by teh auth middleware before being passed to the `handler`
# request handler function
HTTP.serve(auth(handler))

# websocket server is very similar to client usage
WebSockets.listen("0.0.0.0", 8080) do ws
    for msg in ws
        # simple echo server
        send(ws, msg)
    end
end

Further Documentation

Check out the client, server, and websocket-specific documentation pages for more in-depth discussions and examples for the many configurations available.

Migrating Legacy Code to 1.0

The 1.0 release is finally here! It's been a lot of work over the course of about 9 months combing through every part of the codebase to try and modernize APIs, fix long-standing issues, and bring the level of functionality up to par with other language http implementations. Along the way, some breaking changes were made, but with the aim that the package will now be committed to current published APIs for a long time to come. With the amount of increased functionality and fixes, we hope it provides enough incentive to make the update; as always, if you run into issues upgrading or feel something didn't get polished or fixed quite right, don't hesitate to open an issue so we can help.

The sections below outline a mix of breaking changes that were made, in addition to some of the new features in 1.0 with the aim to help those updating legacy codebases.

Struct Changes

  • The HTTP.Request and HTTP.Response body fields are not restricted to Vector{UInt8}; if a response_stream is passed to HTTP.request, it will be set as the resp.body (previously the body was an empty UInt8[]). This simplified many codepaths so these "other body object types" didn't have to be held in some other state, but could be stored in the Request/Response directly. It also opens up the possibility, (as shown in the Cors Server example), where middleware can serialize/deserialize to/from the body field directly.
  • In related news, a Request body can now be passed as a Dict or NamedTuple to have the key-value pairs serialized in the appliction/x-www-form-urlencoded Content-Type matching many other libraries functionality
  • Responses with the Transfer-Encoding: gzip header will now also be automatically decompressed, and this behavior is configurable via the decompress::Bool keyword argument for HTTP.request
  • If a response_stream is provided for streaming a request's response body, HTTP.request will not call close before returning, leaving that up to the caller.

In addition, in the face of redirects or retried requests, note the response_stream will not be written to until the final response is received.

  • If a streaming request body is provided, it should support the mark/reset methods in case the request needs to be retried.
  • Users are encouraged to access the publicly documented fields of Request/Response instead of the previously documented "accessor" functions; these fields are now committed as the public API, so feel free to do resp.body instead of HTTP.body(resp). The accessor methods are still defined for backwards compat.
  • The Request object now stores the original url argument provided to HTTP.request as a parsed URIs.URI object, and accessed via the req.url field. This is commonly desired in handlers/middleware, so convenient to keep it around.
  • The Request object also has a new req.context field of type Dict{Symbol, Any} for storing/sharing state between handler/middleware layers. For example, the HTTP.Router now parses and stores named path parameters with the :params key in the context for handlers to access. Another HTTP.cookie_middleware will parse and store any request Cookie header in the :cookies context key.
  • HTTP.request now throws more consistent and predictable error types, including (and restricted to): HTTP.ConnectError, HTTP.StatusError, HTTP.TimeoutError, and HTTP.RequestError. See the Request exceptions section for more details on each exception type.
  • Cookie persistence used to use a Dict per thread to store domain-specific cookie sessions. A new threadsafe CookieJar struct now globally manages cookie persistence by default. Users can still construct and pass their own cookiejar keyword argument to HTTP.request if desired.

Keyword Argument Changes

  • The pipeline_limit keyword argument (and support for it) were removed in HTTP.request; the implementation was poor and it drastically complicated the request internal implementation. In addition, it's not commonly supported in other modern http implementations, which encourage use of HTTP/2 for better designed functionality.
  • reuse_limit support was removed in both HTTP.request and HTTP.listen; another feature that complicated code more than it was actually useful and hence removed.
  • aws_authentication and its related keyword arguments have been removed in favor of using the AWS.jl package
  • A new redirect_method keyword argument exists and supports finer-grained control over which method to use in the case of a request redirect

Other Largish Changes

"Handlers" framework overhaul

The server-side Handlers framework has been changed to a more modern and flexible framework, including the Handler and Middleware interfaces. It's similar in ways to the old interfaces, but in our opinion, simpler and more straightforward with the clear distinction/pattern between what a Handler does vs. a Middlware.

In that vein, HTTP.Handlers.handle has been removed. HTTP.serve expects a single request or stream Handler function, which should be of the form f(::Request)::Response for the request case, or f(::Stream)::Nothing for streams.

There are also plans to either include some common useful middleware functions in HTTP.jl directly, or a sister package specifically for collecting useful middlewares people can reuse.

WebSockets overhaul

The WebSockets code was some of the oldest and least maintained code in HTTP.jl. It was debated removing it entirely, but there aren't really other modern implementations that are well-maintained. So the WebSockets code was overhauled, modernized, and is now tested against the industry standard autobahn test suite (yay for 3rd party verification!). The API changed as well; while WebSockets.open and WebSockets.listen have stayed the same, the WebSocket object itself now doesn't subtype IO and has a restricted interface like:

  • ws.id access a unique generated UUID id for this websocket connection
  • receive(ws) receive a single non-control message on a websocket, returning a String or Vector{UInt8} depending on whether the message was sent as TEXT or BINARY
  • send(ws, msg) send a message; supports TEXT and BINARY messages, and can provide an iterable for msg to send fragmented messages
  • close(ws) close a websocket connection
  • For convenience, a WebSocket object can be iterated, where each iteration yields a non-control message and iteration terminates when the connection is closed

HTTP.Router reimplementation

While clever, the old HTTP.Router implementation relied on having routes registered "statically", which can be really inconvenient for any cases where the routes are generated programmatically or need to be set/updated dynamically.

The new HTTP.Router implementation uses a text-matching based trie data structure on incoming request path segments to find the right matching handler to process the request. It also supports parsing and storing path variables, like /api/{id} or double wildcards for matching trailing path segments, like /api/**.

HTTP.Router now also supports complete unrestricted route registration via HTTP.register!.

Internal client-side layers overhaul

While grandiose in vision, the old type-based "layers" framework relied heavily on type parameter abuse for generating a large "stack" of layers to handle different parts of each HTTP.request. The new framework actually matches very closely with the server-side Handler and Middleware interfaces, and can be found in more detail under the Client-side Middleware (Layers) section of the docs. The new implementation, while hopefully bringing greater consistency between client-side and server-side frameworks, is much simpler and forced a large cleanup of state-handling in the HTTP.request process for the better.

In addition to the changing of all the client-side layer definitions, HTTP.stack now behaves slightly different in returning the new "layer" chain for HTTP.request, while also accepting custom request/stream layers is provided. A new HTTP.@client macro is provided for convenience in the case that users want to write a custom client-side middleware/layer and wrap its usage in an HTTP.jl-like client.

There also existed a few internal methods previously for manipulating the global stack of client-side layers (insert, insert_default!, etc.). These have been removed and replaced with a more formal (and documented) API via HTTP.pushlayer! and HTTP.poplayer!. These can be used to globally manipulate the client-side stack of layers for any HTTP.request that is made.