diff --git a/.gitignore b/.gitignore index c33693b..bee000d 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ esp-wifi -./vscode +/.vscode +/local *.bin *.elf build - +*.log \ No newline at end of file diff --git a/README.md b/README.md index bb3e05d..ef2b9a6 100644 --- a/README.md +++ b/README.md @@ -12,81 +12,74 @@ Currently supports WiFi on the following processors: Bluetooth is in progress, along with more processors. -### Features +## Features - WiFi station (STA) and soft-AP modes - WiFi scanning +- ESP-NOW peer-to-peer messaging, no access point or TCP/IP needed - Go stdlib `net` support using the TinyGo `netdev`/`netlink` interface - Pure-Go TCP/IP stack with DHCP, DNS, and NTP (uses [`lneto`](https://github.com/soypat/lneto)) +- Heapless HTTP/1.1 server that allocates nothing per request (uses [`httphi`](https://github.com/soypat/lneto/tree/main/http/httphi)) - TCP and UDP Berkeley sockets API - Raw Ethernet frame send/receive - MQTT client support (uses [`natiu-mqtt`](https://github.com/soypat/natiu-mqtt)) - QEMU simulation target for ESP32-C3 -## How to use - -This code starts a basic webserver running on a [Seeed Studio XIAO-ESP32C3](https://www.seeedstudio.com/Seeed-XIAO-ESP32C3-p-5431.html) using `espradio` along with the Go stdlib `net/http` package: +## Getting started with HTTP `hello` example +We can start a basic HTTP webserver on the [Seeed Studio XIAO-ESP32C3](https://www.seeedstudio.com/Seeed-XIAO-ESP32C3-p-5431.html) using espradio along with the lightweight [`httphi`](https://github.com/soypat/lneto/tree/main/http/httphi) library using the code below. See [the example for all the code](./examples/http-hello/main.go): ```go -package main - -import ( - "io" - "log" - "net/http" - "time" - - "tinygo.org/x/drivers/netdev" - nl "tinygo.org/x/drivers/netlink" - link "tinygo.org/x/espradio/netlink" -) - -var ( - ssid string - password string - port string = ":80" -) - func main() { // use ESP32 radio - link := link.Esplink{} - netdev.UseNetdev(&link) + link := &link.Esplink{} + netdev.UseNetdev(link) println("Connecting to WiFi...") - err := link.NetConnect(&nl.ConnectParams{ + failIfErr("connect", link.NetConnect(&nl.ConnectParams{ Ssid: ssid, Passphrase: password, - }) - if err != nil { - log.Fatal(err) - } + })) println("Connected to WiFi.") - - // now setup the web server using the Go "net/http" package: - http.HandleFunc("/", hello) - - h, _ := link.Addr() - host := h.String() - println("HTTP server listening on http://" + host + port) - err = http.ListenAndServe(host+port, nil) - for err != nil { - println("error:", err.Error()) - time.Sleep(5 * time.Second) + var http httphi.MuxSlice + http.Handle("/", func(exch *httphi.Exchange) { + exch.RespondString(200, "application/json", `{"message":"hello"}`) + }) + var router httphi.Router + failIfErr("configuring Router", router.Configure(httphi.RouterConfig{ + FixedNumGoroutines: 4, + RequestHeaderBufferSize: 1024, // Google chrome requests are around 700 bytes in header size. + ResponseHeaderMinBufferSize: 128, // We won't be writing too much to headers. Unused request memory is reused on top of this. + RequestNumHeaderKVCap: 16, // Max number of headers we can expect. + Mux: &http, + })) + defer router.Shutdown() // Despawns goroutines. + listener, err := Listen(link, port) + failIfErr("listening to port", err) + defer listener.Close() + host, _ := link.Addr() + println("HTTP server listening on http://" + host.String() + port) + for { + conn, err := listener.Accept() + failIfErr("accepting conn", err) + err = router.Handle(conn) + if err != nil { + conn.Close() + println("failed to handle connection: ", err.Error()) + } } } -func hello(w http.ResponseWriter, r *http.Request) { - println(r.Method, r.URL.Path) - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "hello") -} ``` -Flash it using TinyGo like this: +Upload your program to the ESP32 using TinyGo as follows. You'll need to change the command below for the next steps: +- Set you wifi SSID and password +- Set the correct `target` depending on your Xiao Model! TinyGo cannot distinguish from a S3 or C3: + - `-target xiao-esp32c3` for C3 + - `-target xiao-esp32s3` for S3 ``` -$ tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/hello +$ tinygo flash -target xiao-esp32x3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/http-hello code data bss | flash ram 860073 33552 345392 | 893625 378944 Connecting to /dev/ttyACM0... @@ -94,18 +87,16 @@ Connected. Detected chip: ESP32-S3 ... ``` +If all went well and your ESP32 connected to the router you should see it print out your ESP32's address like so : `HTTP server listening on http://192.168.1.241:80` -Then you can test it by using `curl`: +You can now test it by using `curl` or directly accessing the ESP32 via browser at the printed address! -``` + +```sh $ curl -w "\n" http://192.168.1.241/ -hello +{"message":"hello"} ``` -## Features - - - ## How it works `espradio` uses the binary blobs provided by Espressif and calls them directly using TinyGo's built-in CGo support. This allows them to be fast and utilize the well-tested existing binaries for low level radio communication. @@ -114,252 +105,10 @@ On top of that `espradio` then uses the [`lneto`](https://github.com/soypat/lnet See the [architecture diagram](#architecture) for more details. -## Examples - `net` package calling `lneto` - -### hello - -Runs a minimal webserver using the Go `net/http` package using the `netlink` interface. - -``` -tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/hello -``` - -### mqtt - -Uses the MQTT machine to machine protocol to publish and subscribe to messages with the `broker.hivemq.com` test server. Uses the Go stdlib and the [`natiu-mqtt`](github.com/soypat/natiu-mqtt) package with the `netlink` interface. - -``` -$ tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/mqtt/ - code data bss | flash ram - 672497 22956 284464 | 695453 307420 -Connecting to /dev/ttyACM0... -Connected. -Detected chip: ESP32-S3 -Loading stub loader... -Stub running. -Erasing entire flash... -Flash erased. -Attaching SPI flash... -Configuring flash size... -Auto-detected flash size: 8MB -Flash params set to 0x023F -SHA digest in image updated -Attaching SPI flash... -Compressed 695552 bytes to 472382 (68%) -Flash begin: 472382 bytes at 0x00000000 (29 compressed blocks) -[##################################################] 100.0% -Flash complete. Verifying... -MD5 verified: 468b20c64e138c8786f8b0b669a5d717 - -Device reset. -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -load:0x4202db20,len:0x7c1b8 -SHA-256 comparison failed: -Calculated: 716b35e13bffed22f2ce7fc865c81242c3bcf494fb3481bab12494585e0cfac9 -Expected: 09eb25d79a2297f382faa0f8b842998881cd4a2474e9e80eb215799c7ddef1a6 -Attempting to boot anyway... -entry 0x40386c8c -Connecting to WiFi... -Connected to WiFi. -ClientId: tinygo-client-WYVKRWBJRP -Connecting to MQTT broker at broker.hivemq.com:1883 -TCP connected to 3.122.68.120:1883 -Sending MQTT CONNECT... -MQTT CONNECT succeeded -Subscribed to topic cpu/usage -Message Random value: 34 received on topic cpu/usage -Message Random value: 8 received on topic cpu/usage -Message Random value: 32 received on topic cpu/usage -... -``` - -### webserver - -![webserver image](./images/webserver.png) - -Runs a webserver using the Go `net/http` package using the `netlink` interface: - -``` -$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/webserver/ - code data bss | flash ram - 932780 34484 353934 | 967264 388418 -Connecting to /dev/ttyACM0... -Connected. -Detected chip: ESP32-C3 -Loading stub loader... -Stub running. -Erasing entire flash... -Flash erased. -Attaching SPI flash... -Configuring flash size... -Auto-detected flash size: 4MB -Flash params set to 0x022F -SHA digest in image updated -Attaching SPI flash... -Compressed 967360 bytes to 624318 (65%) -Flash begin: 624318 bytes at 0x00000000 (39 compressed blocks) -[##################################################] 100.0% -Flash complete. Verifying... -MD5 verified: 530da6d941a937d248fce4dbb88d420b - -Device reset. -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -load:0x3fc8a7b0,len:0x86b4 -load:0x40392e64,len:0xa474 -load:0x4203905c,len:0xb3240 -SHA-256 comparison failed: -Calculated: 698a9fd323776b909347315bd8e6dec519318ba17be4685ce367bea2464e5b27 -Expected: a8fdd97c926ffa40023343a82e4366b1d7103db2214b1607bd34dac4225c78f7 -Attempting to boot anyway... -entry 0x4039d294 -Connecting to WiFi... -HTTP server listening on http://192.168.1.46:80 -``` - -## Examples - `lneto` package only - -### ap - -Shows how to set up a WiFi access point with a DHCP server using the low-level lneto interface. - -``` -$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=tinygoap -X main.password=YourPasswordHere" -monitor ./examples/ap - code data bss | flash ram - 582004 21988 257010 | 603992 278998 -Connected to ESP32-C3 -Flashing: 604096/604096 bytes (100%) -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -SHA-256 comparison failed: -Calculated: 8d1512da059d76b08bbfe99c14ab43d792466ee8fd85a365b338a6ad17aa1e2a -Expected: 06fd1290b7002c716ccf8a9df263fb33d5e22b6892e08038cb6a8e8f6b04be5f -Attempting to boot anyway... -entry 0x40398b3c -ap: enabling radio... -ap: starting AP... -ap: starting L2 netdev (AP)... -ap: creating lneto stack... -ap: configuring DHCP server... -ap: AP is running on 192.168.4.1 - connect to tinygo-ap -ap: rx_cb= 0 rx_drop= 0 -ap: rx_cb= 0 rx_drop= 0 -ap: rx_cb= 0 rx_drop= 0 -... -``` - -### connect-and-dhcp - -Connects to a Wi-Fi network and gets an IP address with DHCP using the low-level lneto interface. - -``` -$ tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=YourPasswordHere" -monitor ./examples/connect-and-dhcp - code data bss | flash ram - 574233 22304 259184 | 596537 281488 -Connected to ESP32-S3 -Flashing: 596640/596640 bytes (100%) -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -SHA-256 comparison failed: -Calculated: 84318f99e1f97b458c8a4afc3237908a2d5be760b42c66b39c03367a96e2ae32 -Expected: c3a26bf70f12f0ffc686e708a86b10548920a08e166a0b14d9e2a8f80e662d2e -Attempting to boot anyway... -entry 0x4038688c -initializing radio... -starting radio... -connecting to rems ... -connected to rems ! -starting L2 netdev... -creating lneto stack... -starting DHCP... -got IP: 192.168.1.241 -gateway: 192.168.1.1 -DNS: 192.168.1.1 -done! -alive -alive -... -``` - -### http-app - -Connects to a WiFi access point, calls NTP to obtain the current date/time, then serves a tiny web application using the low-level lneto interface. - -``` -$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=YourPasswordHere" -monitor ./examples/http-app/ -Connected to ESP32-C3 -Flashing: 652848/652848 bytes (100%) -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -SHA-256 comparison failed: -Calculated: 5520848628102c249831cc101bbd042d6311260e697288fb5bf082ad4c912b32 -Expected: 8b36e3d705b1ed66d74082f300d35a796d6153344e7e8b39eccdbaa2f0bff23f -Attempting to boot anyway... -entry 0x40395708 -initializing radio... -starting radio... -connecting to rems ... -connected to rems ! -starting L2 netdev... -creating lneto stack... -starting DHCP... -got IP: 192.168.1.46 -resolving ntp host: pool.ntp.org -NTP success: 2026-03-21 08:51:31.136908291 +0000 UTC m=+3.079340401 -listening on http://192.168.1.46:80 -... -``` - -### http-static - -Minimal HTTP server that serves a static webpage using the low-level lneto interface. - -``` -$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=YourPasswordHere" -monitor ./examples/http-static/ -Connected to ESP32-C3 -Flashing: 627504/627504 bytes (100%) -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -SHA-256 comparison failed: -Calculated: 468629c0cff3cf13345660532bcc748a1a93df46c197d51727d75863bd985195 -Expected: 785df5a5bb20c057bcc0580b2870982aa779faeb1f946a5d3c371ad36da077f0 -Attempting to boot anyway... -entry 0x40395350 -initializing radio... -starting radio... -connecting to rems ... -connected to rems ! -starting L2 netdev... -creating lneto stack... -listening on http://192.168.1.46:80 -incoming connection: 192.168.1.223 from port 53636 -incoming connection: 192.168.1.223 from port 53640 -Got webpage request! -``` - -### scan - -Scans for WiFi access points. - -``` -$ tinygo flash -target xiao-esp32c3 -monitor ./examples/scan -Connected to ESP32-C3 -Flashing: 442736/442736 bytes (100%) -Connected to /dev/ttyACM0. Press Ctrl-C to exit. -SHA-256 comparison failed: -Calculated: 0045ab8467d485eb94005908a8e7f9dd7baf4dfa610f20ed67884c8ae5e98737 -Expected: 6be1722a792dec8849a2a9fb26c8faf50ba48294ea6617e125472469e56ea719 -Attempting to boot anyway... -entry 0x4038e3d0 -initializing radio... -starting radio... -scanning WiFi... -AP: rems RSSI -59 -AP: rems RSSI -78 - -scanning WiFi... -AP: rems RSSI -59 -AP: rems RSSI -79 -``` -### starting +## [Examples](./examples) +The [`examples`](./examples) directory contains examples and a README that describes each one with examples of expected output. -Starts the ESP32 radio. ## Architecture @@ -373,10 +122,10 @@ flowchart TD F["Espressif Binary Blobs"] G["ESP32 Radio Hardware"] - A --Go net/http or lneto API--> B --Esplink netdev interface--> C --lneto TCP/IP + DHCP/DNS/NTP--> D --EthernetDevice send/recv--> E --radio.c / lib.c / isr.c--> F --WiFi + PHY libs--> G + A --Go net/http, httphi or lneto API--> B --Esplink netdev interface--> C --lneto TCP/IP + DHCP/DNS/NTP--> D --EthernetDevice send/recv--> E --radio.c / lib.c / isr.c--> F --WiFi + PHY libs--> G ``` -- **User Application** - your code, using Go `net/http` or the `lneto` API directly. +- **User Application** - your code, using Go `net/http`, the heapless `httphi` router, or the `lneto` API directly. - **espradio/netlink** - implements the TinyGo `netdev`/`netlink` interface so the Go stdlib `net` package works. - **espradio Stack** - wraps the [`lneto`](https://github.com/soypat/lneto) pure-Go TCP/IP stack with DHCP, DNS, and NTP support. - **espradio NetDev** - L2 Ethernet device that sends/receives raw frames to and from the radio. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..4c562cb --- /dev/null +++ b/examples/README.md @@ -0,0 +1,185 @@ + +## Examples + +### mqtt + +Uses the MQTT machine to machine protocol to publish and subscribe to messages with the `broker.hivemq.com` test server. Uses the Go stdlib and the [`natiu-mqtt`](github.com/soypat/natiu-mqtt) package with the `netlink` interface. + +``` +$ tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/mqtt/ + code data bss | flash ram + 672497 22956 284464 | 695453 307420 +... +Connecting to WiFi... +Connected to WiFi. +ClientId: tinygo-client-WYVKRWBJRP +Connecting to MQTT broker at broker.hivemq.com:1883 +TCP connected to 3.122.68.120:1883 +Sending MQTT CONNECT... +MQTT CONNECT succeeded +Subscribed to topic cpu/usage +Message Random value: 34 received on topic cpu/usage +Message Random value: 8 received on topic cpu/usage +Message Random value: 32 received on topic cpu/usage +... +``` + +### apwebserver + +Brings the device up as a WiFi access point with its own DHCP server and serves +the same page as [webserver](#webserver). Clients get an address automatically; +browse to `http://192.168.4.1` once connected. + +``` +$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=YourSSID -X main.password=YourPassword" -size short -monitor ./examples/apwebserver + code data bss | flash ram + 695624 22700 90436 | 718324 113136 +... +Starting AP... +HTTP server listening on http://192.168.4.1:80 +``` + +### http-hello + +Minimal `httphi` webserver answering `{"message":"hello"}` on `/`. This is the +[getting started](#getting-started-with-http-hello-example) example. + +``` +tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/http-hello +``` + +### webserver + +![webserver image](./images/webserver.png) + +Serves an embedded page with `httphi`, driving the on-board LED and a counter +over `fetch()` from the browser. Shows routing several paths on a +`httphi.MuxSlice`, and reading a POSTed form with `httpraw.Form`: + +``` +$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/webserver/ + code data bss | flash ram + 695130 22692 90460 | 717822 113152 +... +Connecting to WiFi... +HTTP server listening on http://192.168.1.46:80 +``` + +## Examples - `espradio` and `lneto` directly + +These skip `netdev`/`netlink` and drive the radio, and where they need one the +`lneto` stack, themselves. + +### ap + +Shows how to set up a WiFi access point with a DHCP server using the low-level lneto interface. + +``` +$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=tinygoap -X main.password=YourPasswordHere" -monitor ./examples/ap + code data bss | flash ram + 582004 21988 257010 | 603992 278998 +... +ap: enabling radio... +ap: starting AP... +ap: starting L2 netdev (AP)... +ap: creating lneto stack... +ap: configuring DHCP server... +ap: AP is running on 192.168.4.1 - connect to tinygo-ap +ap: rx_cb= 0 rx_drop= 0 +ap: rx_cb= 0 rx_drop= 0 +ap: rx_cb= 0 rx_drop= 0 +... +``` + +### connect-and-dhcp + +Connects to a Wi-Fi network and gets an IP address with DHCP using the low-level lneto interface. + +``` +$ tinygo flash -target xiao-esp32s3 -ldflags="-X main.ssid=yourssid -X main.password=YourPasswordHere" -monitor ./examples/connect-and-dhcp + code data bss | flash ram + 574233 22304 259184 | 596537 281488 +... +initializing radio... +starting radio... +connecting to rems ... +connected to rems ! +starting L2 netdev... +creating lneto stack... +starting DHCP... +got IP: 192.168.1.241 +gateway: 192.168.1.1 +DNS: 192.168.1.1 +done! +alive +alive +... +``` + +### http-app + +Connects to a WiFi access point, calls NTP to obtain the current date/time, then serves a tiny web application with `httphi` on the low-level lneto interface. + +``` +$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=YourPasswordHere" -monitor ./examples/http-app/ +... +got IP: 192.168.1.46 +resolving ntp host: pool.ntp.org +NTP success: 2026-03-21 08:51:31.136908291 +0000 UTC m=+3.079340401 +listening on http://192.168.1.46:80 +... +``` + +### http-static + +Minimal HTTP server using `httphi` that serves a static embedded webpage on the low-level lneto interface. + +``` +$ tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=YourPasswordHere" -monitor ./examples/http-static/ +... +listening on http://192.168.1.46:80 +incoming connection: 192.168.1.223 from port 53636 +incoming connection: 192.168.1.223 from port 53640 +Got webpage request! +``` + +### scan + +Scans for WiFi access points. + +``` +$ tinygo flash -target xiao-esp32c3 -monitor ./examples/scan +... +scanning WiFi... +AP: rems RSSI -59 +AP: rems RSSI -78 + +scanning WiFi... +AP: rems RSSI -59 +AP: rems RSSI -79 +``` + +### espnow + +Sends and receives ESP-NOW packets, the connectionless Espressif protocol that +talks peer to peer without an access point or any TCP/IP stack. Set +`peerAddress` to the other device's ESP-NOW address, which each device prints on +boot, then flash both. + +``` +tinygo flash -target xiao-esp32c3 -monitor ./examples/espnow +``` + +### http-stdlib +Use the standard library net/http package for hosting a HTTP server. +``` +tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/http-stdlib +``` + +### http-get + +Fetches a URL with `http.Get()`. + +``` +tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=yourssid -X main.password=yourpassword" -size short -monitor ./examples/http-get +``` \ No newline at end of file diff --git a/examples/apwebserver/apwebserver.go b/examples/apwebserver/apwebserver.go index 2b377aa..e48b16f 100644 --- a/examples/apwebserver/apwebserver.go +++ b/examples/apwebserver/apwebserver.go @@ -7,18 +7,25 @@ package main import ( + "context" _ "embed" - "io" - "net/http" + "errors" + "net" "net/netip" "strconv" + "sync/atomic" "time" - "tinygo.org/x/drivers/netdev" + "github.com/soypat/lneto/http/httphi" + "github.com/soypat/lneto/http/httpraw" + "github.com/soypat/lneto/x/xnet" "tinygo.org/x/espradio" link "tinygo.org/x/espradio/netlink" ) +// Pages are embedded as bytes so serving them needs no string conversion, +// which would copy the page to the heap on every request. + //go:embed index.html var indexHTML string @@ -37,11 +44,10 @@ func main() { // wait a bit for serial time.Sleep(2 * time.Second) - lnk := link.Esplink{} - netdev.UseNetdev(&lnk) + lnk := &link.Esplink{} println("Starting AP...") - err := lnk.NetConnectAP(link.APConnectParams{ + failIfErr("starting AP", lnk.NetConnectAP(link.APConnectParams{ APConfig: espradio.APConfig{ SSID: ssid, Password: password, @@ -52,80 +58,124 @@ func main() { MaxUDPPorts: 2, MaxTCPPorts: 4, PassivePeers: 64, - }) - if err != nil { - failure("could not start AP: " + err.Error()) - } - - http.Handle("/", logRequest(root)) - http.Handle("/hello", logRequest(hello)) - http.Handle("/cnt", logRequest(cnt)) - http.Handle("/6", logRequest(sixlines)) - http.Handle("/off", logRequest(LED_OFF)) - http.Handle("/on", logRequest(LED_ON)) - + })) + + var mux httphi.MuxSlice + mux.Handle("/hello", logRequest(hello)) + mux.Handle("/cnt", logRequest(cnt)) + mux.Handle("/6", logRequest(sixlines)) + mux.Handle("/off", logRequest(LED_OFF)) + mux.Handle("/on", logRequest(LED_ON)) + // A trailing slash is an anonymous wildcard, so "/" matches every path. + // Registered last it serves what the paths above did not claim. + mux.Handle("/", logRequest(root)) + + var router httphi.Router + failIfErr("configuring Router", router.Configure(httphi.RouterConfig{ + FixedNumGoroutines: 4, + RequestHeaderBufferSize: 1024, // Google chrome requests are around 700 bytes in header size. + ResponseHeaderMinBufferSize: 128, // We won't be writing too much to headers. Unused request memory is reused on top of this. + RequestNumHeaderKVCap: 16, // Max number of headers we can expect. + Mux: &mux, + })) + defer router.Shutdown() // Despawns goroutines. + + listener, err := Listen(lnk, port) + failIfErr("listening to port", err) + defer listener.Close() println("HTTP server listening on http://" + apIP + port) - err = http.ListenAndServe(apIP+port, nil) - if err != nil { - failure("http.ListenAndServe: " + err.Error()) + for { + conn, err := listener.Accept() + failIfErr("accepting conn", err) + err = router.Handle(conn) + if err != nil { + conn.Close() + println("failed to handle connection: ", err.Error()) + } } } -func logRequest(h http.HandlerFunc) http.Handler { - return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - println(r.Method, r.URL.Path) - h(w, r) - }) +func logRequest(h httphi.HandlerFunc) httphi.HandlerFunc { + return func(exch *httphi.Exchange) { + println(exch.RequestMethod().String(), " ", exch.MuxPattern()) + h(exch) + } } -func root(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusOK) - io.WriteString(w, indexHTML) +func root(exch *httphi.Exchange) { + exch.RespondString(200, "text/html", indexHTML) } -func sixlines(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusOK) - io.WriteString(w, sixlinesHTML) +func sixlines(exch *httphi.Exchange) { + exch.RespondString(200, "text/html", sixlinesHTML) } -func LED_ON(w http.ResponseWriter, r *http.Request) { +func LED_ON(exch *httphi.Exchange) { setLED(true) - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "led.High()") + exch.RespondString(200, "text/plain; charset=UTF-8", "led.High()") } -func LED_OFF(w http.ResponseWriter, r *http.Request) { +func LED_OFF(exch *httphi.Exchange) { setLED(false) - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "led.Low()") + exch.RespondString(200, "text/plain; charset=UTF-8", "led.Low()") } -func hello(w http.ResponseWriter, r *http.Request) { - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "hello") +func hello(exch *httphi.Exchange) { + exch.RespondString(200, "text/plain; charset=UTF-8", "hello") } -var counter int - -func cnt(w http.ResponseWriter, r *http.Request) { - r.ParseForm() - if r.Method == "POST" { - c := r.Form.Get("cnt") - if c != "" { - i64, _ := strconv.ParseInt(c, 0, 0) - counter = int(i64) +// counter is read and written from the router's goroutines, so several clients +// may be inside cnt at once. +var counter atomic.Int64 + +func cnt(exch *httphi.Exchange) { + var scratch [64]byte + switch exch.RequestMethod() { + case httphi.MethPost: // POST + var form httpraw.Form + form.Reset(scratch[:], 2) // 2 query values max. + const parseURL, prioritizeURL = true, false + err := exch.RequestParseForm(&form, parseURL, prioritizeURL) + if err != nil { + exch.Respond(500, "", nil) + return + } + c := form.Get("cnt") + if len(c) > 0 { + // Parsed before the form's memory is reused for the response below. + i64, _ := strconv.ParseInt(string(c), 0, 0) + counter.Store(i64) + println("set counter", i64) } } - - w.Header().Set(`Content-Type`, `application/json`) - io.WriteString(w, `{"cnt": `) - io.WriteString(w, strconv.Itoa(counter)) - io.WriteString(w, `}`) + json := append(scratch[:0], `{"cnt": `...) + json = strconv.AppendInt(json, counter.Load(), 10) + json = append(json, '}') + exch.Respond(200, "application/json", json) } -func failure(msg string) { - for { - println("failure:", msg) +func failIfErr(action string, err error) { + for err != nil { + println("fail " + action + ": " + err.Error()) time.Sleep(1 * time.Second) } } + +func Listen(lnk *link.Esplink, port string) (net.Listener, error) { + // Listen by asking the lneto stack for a socket directly instead of going + // through stdlib net.Listen and the netdev file descriptor layer due to a bug. + stack := lnk.StackGo() + laddr, err := netip.ParseAddrPort("0.0.0.0" + port) + if err != nil { + return nil, err + } + sock, err := stack.SocketNetip(context.Background(), "tcp4", xnet.AF_INET, xnet.SOCK_STREAM, laddr, netip.AddrPort{}) + if err != nil { + return nil, err + } + listener, ok := sock.(net.Listener) + if !ok { + return nil, errors.New("stack returned non-listener socket") + } + return listener, nil +} diff --git a/examples/http-app/main-http.go b/examples/http-app/main-http.go index 4da9354..d4a0651 100644 --- a/examples/http-app/main-http.go +++ b/examples/http-app/main-http.go @@ -7,6 +7,11 @@ // navigating to the IP address assigned to the ESP32 in your browser. // Click the "Toggle LED" button to see the LED state change and the action recorded on the webpage. // +// Requests are served by a httphi.Router: it allocates its exchanges (request +// header buffer, response header buffer) and its worker goroutines once at +// Configure time, so serving requests costs no allocation and memory does not +// grow with load. Accepting connections is still this program's job. +// // tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=YourSSID -X main.password=YourPassword" -monitor ./examples/http-app package main @@ -21,7 +26,7 @@ import ( "unsafe" "github.com/soypat/lneto" - "github.com/soypat/lneto/http/httpraw" + "github.com/soypat/lneto/http/httphi" "github.com/soypat/lneto/tcp" "github.com/soypat/lneto/x/xnet" "tinygo.org/x/espradio" @@ -36,9 +41,21 @@ const ( ntpHost = "pool.ntp.org" pollTime = 5 * time.Millisecond maxConns = 4 - httpBuf = 1024 listenPort = 80 + // reqHeaderBuf bounds the request header. The router refuses to grow it, so + // a request whose header does not fit is answered 431 instead of eating memory. + reqHeaderBuf = 1024 + // respHeaderBuf is the room for staged response header fields. The status + // line does not count towards it and unused request memory is reused. + respHeaderBuf = 128 + // numHeaderFields is how many request header fields are parsed before + // answering 431. A browser sends around twenty. Each field costs 8 bytes. + numHeaderFields = 32 + // connDeadline fails the reads/writes of a peer that opens a connection and + // then stalls, instead of it holding a router goroutine forever. + connDeadline = 8 * time.Second + templateActionMarker = "" ) @@ -108,9 +125,7 @@ func main() { println("got IP:", addr.String()) lstack := espstack.LnetoStack() - rstack := lstack.StackRetrying(lneto.BackoffStrategy(func(_ uint) time.Duration { - return pollTime - })) + rstack := lstack.StackRetrying(pollBackoff) gatewayHW, err := rstack.DoResolveHardwareAddress6(dhcp.Router, 500*time.Millisecond, 4) if err != nil { failure("ARP resolve failed: " + err.Error()) @@ -135,12 +150,7 @@ func main() { RxBufSize: 1024, EstablishedTimeout: 5 * time.Second, ClosingTimeout: 5 * time.Second, - NewUserData: func() any { - cs := new(connState) - cs.hdr.Reset(cs.httpBuf[:]) - return cs - }, - NewBackoff: func() lneto.BackoffStrategy { return pollBackoff }, + NewBackoff: func() lneto.BackoffStrategy { return pollBackoff }, }) if err != nil { failure("tcppool create: " + err.Error()) @@ -157,16 +167,29 @@ func main() { failure("listener register: " + err.Error()) } + // The mux resolves method+path to the handler serving it. Paths are matched + // exactly and the query string is the handler's business, so + // "/toggle-led?callsign=pato" lands on handleToggleLED. Anything else is 404. + var mux httphi.MuxSlice + var server Server + server.InitAndRegister(&mux) + // Router allocates its exchanges and goroutines here and never again. + var router httphi.Router + err = router.Configure(httphi.RouterConfig{ + FixedNumGoroutines: maxConns, + RequestHeaderBufferSize: reqHeaderBuf, + ResponseHeaderMinBufferSize: respHeaderBuf, + RequestNumHeaderKVCap: numHeaderFields, + Mux: &mux, + }) + if err != nil { + failure("router configure: " + err.Error()) + } + defer router.Shutdown() // Despawns goroutines. + println("listening on", "http://"+listenAddr.String()) lstack.Debug("init-complete") - // Pre-allocate worker goroutines so stacks are allocated once at startup - // instead of per-connection. Maintains full concurrency up to maxConns. - jobCh := make(chan connJob, maxConns) - for range maxConns { - go connWorker(jobCh) - } - lstack.Debug("goroutines allocated") for { if listener.NumberOfReadyToAccept() == 0 { time.Sleep(pollTime) @@ -174,13 +197,21 @@ func main() { continue } - conn, userData, err := listener.TryAccept() + conn, _, err := listener.TryAccept() if err != nil { println("err listener accept:", err.Error()) time.Sleep(time.Second) continue } - jobCh <- connJob{conn: conn, cs: userData.(*connState), stack: lstack} + printIncoming(conn) + conn.SetDeadline(time.Now().Add(connDeadline)) + err = router.Handle(conn) + if err != nil { + // Every router goroutine is busy and the queue is full. Dropping the + // connection is the backpressure that keeps memory bounded. + println("dropped connection:", err.Error()) + conn.Close() + } } } @@ -188,30 +219,59 @@ var pollBackoff = lneto.BackoffStrategy(func(_ uint) time.Duration { return pollTime }) -// connState holds all per-connection buffers, pre-allocated during pool init. -// Eliminates per-connection heap escapes of local arrays (buf, dynBuf, csBuf) -// and the make([]byte, httpBuf) that exceeds TinyGo's 256-byte stack limit. -type connState struct { - hdr httpraw.Header - httpBuf [httpBuf]byte - buf [128]byte - dynBuf [256]byte - csBuf [9]byte +type Server struct { + state ServerState + scratchPool sync.Pool } -type connJob struct { - conn *tcp.Conn - cs *connState - stack *xnet.StackAsync +// scratch holds the work buffers a handler needs beyond the exchange's own +// memory. One is checked out per request instead of declaring the buffers as +// locals, which TinyGo would escape to the heap on every request. +type scratch struct { + dyn [1024]byte // Rendered action list, sized so append never grows it. } -type page uint8 +func (sv *Server) InitAndRegister(mux *httphi.MuxSlice) { + sv.scratchPool.New = func() interface{} { return &scratch{} } + mux.Handle("GET /", sv.handleLanding) + mux.Handle("GET /toggle-led", sv.handleToggleLED) +} -const ( - pageNotExists page = iota - pageLanding - pageToggleLED -) +func (sv *Server) handleLanding(exch *httphi.Exchange) { + println("Got webpage request!") + scratch := sv.scratchPool.Get().(*scratch) + defer sv.scratchPool.Put(scratch) + + dynContent := sv.state.AppendActionsHTML(scratch.dyn[:0]) + if len(dynContent) > len(scratch.dyn) { + println("[WARN] dynamic content outgrew scratch, heap allocated:", cap(dynContent)) + } + exch.StageHeader("Content-Type", "text/html") + exch.StageHeaderInt("Content-Length", int64(len(htmlTemplate)+len(dynContent))) + // The router serves one exchange per connection and then closes it. Saying so + // avoids notably slower paint times in the browser. Content-Length above is + // what keeps the browser from treating the close as a truncated page. + exch.StageHeader("Connection", "close") + exch.WriteHeader(httphi.StatusOK) + exch.WriteBody(htmlTemplate[:htmlActionIdx]) + exch.WriteBody(dynContent) + exch.WriteBody(htmlTemplate[htmlActionIdx:]) + time.Sleep(pollTime) +} + +func (sv *Server) handleToggleLED(exch *httphi.Exchange) { + println("got toggle led request") + scratch := sv.scratchPool.Get().(*scratch) + defer sv.scratchPool.Put(scratch) + + // The raw view aliases the exchange buffer, no copy needed: sanitizeCallsign + // stops at the first non-letter, so a percent escape ends the callsign. + rawCallsign, _ := exch.RequestQueryValue("callsign") + sv.state.RecordToggle(sanitizeCallsign(scratch.dyn[:0], rawCallsign)) + exch.StageHeader("Content-Length", "0") + exch.StageHeader("Connection", "close") + exch.WriteHeader(httphi.StatusOK) +} // ServerState stores the state of the HTTP server. It has a ring buffer with last 8 actions // performed. Every time a new action is performed it replaces the oldest action by advancing the ring buffer. @@ -229,8 +289,6 @@ type Action struct { TurnedLEDOn bool } -var state ServerState - func (s *ServerState) RecordToggle(callsign []byte) { s.mu.Lock() defer s.mu.Unlock() @@ -294,19 +352,6 @@ func appendDurationAgo(dst []byte, d time.Duration) []byte { return dst } -func parseCallsignValue(query []byte) []byte { - const key = "callsign=" - idx := bytes.Index(query, []byte(key)) - if idx < 0 || (idx > 0 && query[idx-1] != '&') { - return nil - } - val := query[idx+len(key):] - if end := bytes.IndexByte(val, '&'); end >= 0 { - val = val[:end] - } - return val -} - func sanitizeCallsign(dst, raw []byte) []byte { dst = dst[:0] for _, b := range raw { @@ -324,125 +369,14 @@ func sanitizeCallsign(dst, raw []byte) []byte { return dst } -func connWorker(ch <-chan connJob) { - for job := range ch { - handleConn(job.conn, job.cs, job.stack) - } -} - -func handleConn(conn *tcp.Conn, cs *connState, stack *xnet.StackAsync) { - defer conn.Close() - const AsRequest = false - hdr := &cs.hdr - hdr.Reset(nil) - buf := cs.buf[:] +// addrBuf formats remote addresses in printIncoming. Only the accept loop uses it. +var addrBuf [64]byte - stack.Debug("conn-start") - conn.SetDeadline(time.Now().Add(8 * time.Second)) +func printIncoming(conn *tcp.Conn) { remoteAddr, _ := netip.AddrFromSlice(conn.RemoteAddr()) print("incoming connection: ") - printAddr(remoteAddr, buf[:0]) + printAddr(remoteAddr, addrBuf[:0]) println(" from port", conn.RemotePort()) - stack.Debug("post-deadline+println") - - for { - n, err := conn.Read(buf) - if n > 0 { - hdr.ReadFromBytes(buf[:n]) - needMoreData, err := hdr.TryParse(AsRequest) - if err != nil && !needMoreData { - println("parsing HTTP request:", err.Error()) - return - } - if !needMoreData { - break - } - } - if err != nil { - println("read error:", err.Error()) - return - } - closed := conn.State() != tcp.StateEstablished - if closed { - break - } else if hdr.BufferReceived() >= httpBuf { - println("too much HTTP data") - return - } - } - // BEGIN PARSING REQUEST. - uri := hdr.RequestURI() - uriPath := uri - var uriQuery []byte - if qIdx := bytes.IndexByte(uri, '?'); qIdx >= 0 { - uriPath = uri[:qIdx] - uriQuery = uri[qIdx+1:] - } - - var requestedPage page - switch { - case bytesEqual(uriPath, "/"): - println("Got webpage request!") - requestedPage = pageLanding - case bytesEqual(uriPath, "/toggle-led"): - println("got toggle led request") - requestedPage = pageToggleLED - callsign := sanitizeCallsign(cs.csBuf[:0], parseCallsignValue(uriQuery)) - state.RecordToggle(callsign) - } - - stack.Debug("post-read-loop") - // BEGIN RESPONSE. - // Reuse header to write response. - hdr.Reset(nil) - stack.Debug("post-reset") - hdr.SetProtocol("HTTP/1.1") - stack.Debug("post-setproto") - if requestedPage == pageNotExists { - hdr.SetStatus("404", "Not Found") - } else { - hdr.SetStatus("200", "OK") - } - stack.Debug("post-setstatus") - // We call Close() on exiting this function. - // If we omit Connection:close in header we'll have notably slower paint times in browser. - // One thing to keep in mind when using Connection:close is using Content-Length to prevent early browser close. - hdr.Set("Connection", "close") - stack.Debug("post-set-conn-close") - switch requestedPage { - case pageLanding: - dynContent := state.AppendActionsHTML(cs.dynBuf[:0]) - contentLength := len(htmlTemplate) + len(dynContent) - buf = strconv.AppendUint(buf[:0], uint64(contentLength), 10) - stack.Debug("post-appendhtml") - hdr.Set("Content-Type", "text/html") - hdr.SetBytes("Content-Length", buf) - responseHeader, err := hdr.AppendResponse(buf[:0]) - if err != nil { - println("error appending:", err.Error()) - } - conn.Write(responseHeader) - conn.Write(htmlTemplate[:htmlActionIdx]) - conn.Write(dynContent) - conn.Write(htmlTemplate[htmlActionIdx:]) - time.Sleep(pollTime) - - case pageToggleLED: - hdr.Set("Content-Length", "0") - responseHeader, err := hdr.AppendResponse(buf[:0]) - if err != nil { - println("error appending:", err.Error()) - } - conn.Write(responseHeader) - - default: - responseHeader, err := hdr.AppendResponse(buf[:0]) - if err != nil { - println("error appending:", err.Error()) - } - conn.Write(responseHeader) - } - stack.Debug("pre-close") } // printAddr prints a netip.Addr without heap allocation by formatting into buf. @@ -451,17 +385,6 @@ func printAddr(addr netip.Addr, buf []byte) { print(unsafe.String(&buf[0], len(buf))) } -// bytesEqual compares a byte slice to a string without heap allocation. -func bytesEqual(b []byte, s string) bool { - if len(b) != len(s) { - return false - } - if len(b) == 0 { - return true - } - return unsafe.String(&b[0], len(b)) == s -} - func loopForeverStack(stack *espradio.Stack) { for { send, recv, _ := stack.RecvAndSend() diff --git a/examples/http-hello/main.go b/examples/http-hello/main.go new file mode 100644 index 0000000..0d71875 --- /dev/null +++ b/examples/http-hello/main.go @@ -0,0 +1,89 @@ +package main + +import ( + "context" + "errors" + "net" + "net/netip" + "time" + + "github.com/soypat/lneto/http/httphi" + "github.com/soypat/lneto/x/xnet" + "tinygo.org/x/drivers/netdev" + nl "tinygo.org/x/drivers/netlink" + "tinygo.org/x/espradio/netlink" + link "tinygo.org/x/espradio/netlink" +) + +var ( + ssid string + password string + port string = ":80" +) + +func main() { + // use ESP32 radio + link := &link.Esplink{} + netdev.UseNetdev(link) + + println("Connecting to WiFi...") + failIfErr("connect", link.NetConnect(&nl.ConnectParams{ + Ssid: ssid, + Passphrase: password, + })) + + println("Connected to WiFi.") + var http httphi.MuxSlice + http.Handle("/", func(exch *httphi.Exchange) { + exch.RespondString(200, "application/json", `{"message":"hello"}`) + }) + var router httphi.Router + failIfErr("configuring Router", router.Configure(httphi.RouterConfig{ + FixedNumGoroutines: 4, + RequestHeaderBufferSize: 1024, // Google chrome requests are around 700 bytes in header size. + ResponseHeaderMinBufferSize: 128, // We won't be writing too much to headers. Unused request memory is reused on top of this. + RequestNumHeaderKVCap: 16, // Max number of headers we can expect. + Mux: &http, + })) + defer router.Shutdown() // Despawns goroutines. + listener, err := Listen(link, port) + failIfErr("listening to port", err) + defer listener.Close() + host, _ := link.Addr() + println("HTTP server listening on http://" + host.String() + port) + for { + conn, err := listener.Accept() + failIfErr("accepting conn", err) + err = router.Handle(conn) + if err != nil { + conn.Close() + println("failed to handle connection: ", err.Error()) + } + } +} + +func failIfErr(action string, err error) { + for err != nil { + println("fail " + action + ": " + err.Error()) + time.Sleep(1 * time.Second) + } +} + +func Listen(link *netlink.Esplink, port string) (net.Listener, error) { + // Listen by asking the lneto stack for a socket directly instead of going + // through stdlib net.Listen and the netdev file descriptor layer due to a bug. + stack := link.StackGo() + laddr, err := netip.ParseAddrPort("0.0.0.0" + port) + if err != nil { + return nil, err + } + sock, err := stack.SocketNetip(context.Background(), "tcp4", xnet.AF_INET, xnet.SOCK_STREAM, laddr, netip.AddrPort{}) + if err != nil { + return nil, err + } + listener, ok := sock.(net.Listener) + if !ok { + return nil, errors.New("stack returned non-listener socket") + } + return listener, nil +} diff --git a/examples/http-static/main-httpstatic.go b/examples/http-static/main-httpstatic.go index b518c0c..4cc12e2 100644 --- a/examples/http-static/main-httpstatic.go +++ b/examples/http-static/main-httpstatic.go @@ -3,17 +3,22 @@ // embedded `index.html` file. You can test it by connecting to the same Wi-Fi network // as the ESP32 and navigating to the IP address assigned to the ESP32 in your browser. // +// Requests are served by a httphi.Router: it allocates its exchanges (request +// header buffer, response header buffer) and its worker goroutines once at +// Configure time, so serving requests costs no allocation and memory does not +// grow with load. Accepting connections is still this program's job. +// // tinygo flash -target xiao-esp32c3 -ldflags="-X main.ssid=YourSSID -X main.password=YourPassword" -monitor ./examples/http-static package main import ( _ "embed" - "net" "net/netip" - "strconv" + "sync" "time" - "github.com/soypat/lneto/http/httpraw" + "github.com/soypat/lneto" + "github.com/soypat/lneto/http/httphi" "github.com/soypat/lneto/tcp" "github.com/soypat/lneto/x/xnet" "tinygo.org/x/espradio" @@ -27,14 +32,25 @@ var ( const ( pollTime = 5 * time.Millisecond maxConns = 4 - httpBuf = 1024 listenPort = 80 + + // reqHeaderBuf bounds the request header. The router refuses to grow it, so + // a request whose header does not fit is answered 431 instead of eating memory. + reqHeaderBuf = 1024 + // respHeaderBuf is the room for staged response header fields. The status + // line does not count towards it and unused request memory is reused. + respHeaderBuf = 128 + // numHeaderFields is how many request header fields are parsed before + // answering 431. A browser sends around twenty. Each field costs 8 bytes. + numHeaderFields = 32 + // connDeadline fails the reads/writes of a peer that opens a connection and + // then stalls, instead of it holding a router goroutine forever. + connDeadline = 8 * time.Second ) var ( //go:embed index.html - webPage []byte - lastLEDState bool + webPage []byte ) func setLED(lightOn bool) { @@ -100,12 +116,7 @@ func main() { RxBufSize: 256, EstablishedTimeout: 5 * time.Second, ClosingTimeout: 5 * time.Second, - NewUserData: func() any { - var hdr httpraw.Header - buf := make([]byte, httpBuf) - hdr.Reset(buf) - return &hdr - }, + NewBackoff: func() lneto.BackoffStrategy { return pollBackoff }, }) if err != nil { failure("tcppool create: " + err.Error()) @@ -130,111 +141,98 @@ func main() { if err != nil { failure("listener register: " + err.Error()) } + + // The mux resolves method+path to the handler serving it, paths matched + // exactly. Anything not registered below is answered 404 by the router. + var mux httphi.MuxSlice + var server Server + server.InitAndRegister(&mux) + // Router allocates its exchanges and goroutines here and never again. + var router httphi.Router + err = router.Configure(httphi.RouterConfig{ + FixedNumGoroutines: maxConns, + RequestHeaderBufferSize: reqHeaderBuf, + ResponseHeaderMinBufferSize: respHeaderBuf, + RequestNumHeaderKVCap: numHeaderFields, + Mux: &mux, + }) + if err != nil { + failure("router configure: " + err.Error()) + } + defer router.Shutdown() // Despawns goroutines. + println("listening on", "http://"+listenAddr.String()) for { if listener.NumberOfReadyToAccept() == 0 { - time.Sleep(5 * time.Millisecond) + time.Sleep(pollTime) tcpPool.CheckTimeouts() continue } - conn, httpBuf, err := listener.TryAccept() + conn, _, err := listener.TryAccept() if err != nil { println("listener accept err", err.Error()) time.Sleep(time.Second) continue } - // Beware, this allocates a stack on the heap on - // every connection. See http-app example on - // how to allocate a pool of connections up front - // and avoid heap allocations. - go handleConn(conn, httpBuf.(*httpraw.Header)) + remoteAddr, _ := netip.AddrFromSlice(conn.RemoteAddr()) + println("incoming connection:", remoteAddr.String(), "from port", conn.RemotePort()) + conn.SetDeadline(time.Now().Add(connDeadline)) + err = router.Handle(conn) + if err != nil { + // Every router goroutine is busy and the queue is full. Dropping the + // connection is the backpressure that keeps memory bounded. + println("dropped connection:", err.Error()) + conn.Close() + } } } -type page uint8 - -const ( - pageNotExits page = iota - pageLanding // / - pageToggleLED // /toggle-led -) - -func handleConn(conn *tcp.Conn, hdr *httpraw.Header) { - defer conn.Close() - const AsRequest = false - var buf [64]byte - hdr.Reset(nil) - - remoteAddr, _ := netip.AddrFromSlice(conn.RemoteAddr()) - println("incoming connection:", remoteAddr.String(), "from port", conn.RemotePort()) +var pollBackoff = lneto.BackoffStrategy(func(_ uint) time.Duration { + return pollTime +}) + +// Server holds the state the handlers share. Handlers run on the router's +// goroutines, up to maxConns of them at a time, so state they touch is guarded. +// Both responses here are static, so unlike the http-app example this server +// needs no scratch buffer pool: nothing is rendered per request. +type Server struct { + mu sync.Mutex + ledState bool +} - for { - n, err := conn.Read(buf[:]) - if n > 0 { - hdr.ReadFromBytes(buf[:n]) - needMoreData, err := hdr.TryParse(AsRequest) - if err != nil && !needMoreData { - println("parsing HTTP request:", err.Error()) - return - } - if !needMoreData { - break - } - } - closed := err == net.ErrClosed || conn.State() != tcp.StateEstablished - if closed { - break - } else if hdr.BufferReceived() >= httpBuf { - println("too much HTTP data") - return - } - } - // Check requested requestedPage URI. - var requestedPage page - uri := hdr.RequestURI() - switch string(uri) { - case "/": - println("Got webpage request!") - requestedPage = pageLanding - case "/toggle-led": - println("got toggle led request") - requestedPage = pageToggleLED - lastLEDState = !lastLEDState - setLED(lastLEDState) - } +func (sv *Server) InitAndRegister(mux *httphi.MuxSlice) { + mux.Handle("GET /", sv.handleLanding) + mux.Handle("GET /toggle-led", sv.handleToggleLED) +} - // Prepare response with same buffer. - hdr.Reset(nil) - hdr.SetProtocol("HTTP/1.1") - if requestedPage == pageNotExits { - hdr.SetStatus("404", "Not Found") - } else { - hdr.SetStatus("200", "OK") - } - var body []byte - switch requestedPage { - case pageLanding: - body = webPage - hdr.Set("Content-Type", "text/html") - } - if len(body) > 0 { - hdr.Set("Content-Length", strconv.Itoa(len(body))) - } - responseHeader, err := hdr.AppendResponse(buf[:0]) +func (sv *Server) handleLanding(exch *httphi.Exchange) { + println("Got webpage request!") + exch.StageHeader("Content-Type", "text/html") + exch.StageHeaderInt("Content-Length", int64(len(webPage))) + // The router serves one exchange per connection and then closes it. Saying so + // avoids notably slower paint times in the browser. Content-Length above is + // what keeps the browser from treating the close as a truncated page. + exch.StageHeader("Connection", "close") + exch.WriteHeader(httphi.StatusOK) + _, err := exch.WriteBody(webPage) if err != nil { - println("error appending:", err.Error()) - } - conn.Write(responseHeader) - if len(body) > 0 { - _, err := conn.Write(body) - if err != nil { - println("writing body:", err.Error()) - } - time.Sleep(pollTime) + println("writing body:", err.Error()) } - // connection closed automatically by defer. + time.Sleep(pollTime) +} + +func (sv *Server) handleToggleLED(exch *httphi.Exchange) { + println("got toggle led request") + sv.mu.Lock() + sv.ledState = !sv.ledState + setLED(sv.ledState) + sv.mu.Unlock() + + exch.StageHeader("Content-Length", "0") + exch.StageHeader("Connection", "close") + exch.WriteHeader(httphi.StatusOK) } func loopForeverStack(stack *espradio.Stack) { diff --git a/examples/hello/main.go b/examples/http-stdlib/http-stdlib.go similarity index 100% rename from examples/hello/main.go rename to examples/http-stdlib/http-stdlib.go diff --git a/examples/webserver/webserver.go b/examples/webserver/webserver.go index 8962445..18e30aa 100644 --- a/examples/webserver/webserver.go +++ b/examples/webserver/webserver.go @@ -5,35 +5,51 @@ package main import ( + "context" _ "embed" - "io" - "net/http" + "errors" + "net" + "net/netip" "strconv" + "sync" "time" - "tinygo.org/x/drivers/netdev" + "github.com/soypat/lneto/http/httphi" + "github.com/soypat/lneto/http/httpraw" + "github.com/soypat/lneto/x/xnet" nl "tinygo.org/x/drivers/netlink" + "tinygo.org/x/espradio/netlink" link "tinygo.org/x/espradio/netlink" ) +// Pages are embedded as bytes so serving them needs no string conversion, +// which would copy the page to the heap on every request. + //go:embed index.html var indexHTML string //go:embed sixlines.html var sixlinesHTML string +// Stores []byte buffers to avoid allocations in HTTP handlers. +var scratchPool sync.Pool + +const scratchSize = 128 + +const ( + port = ":80" +) + var ( ssid string password string - port string = ":80" ) func main() { // wait a bit for serial time.Sleep(2 * time.Second) - link := link.Esplink{} - netdev.UseNetdev(&link) + link := &link.Esplink{} println("Connecting to WiFi...") err := link.NetConnect(&nl.ConnectParams{ @@ -43,7 +59,8 @@ func main() { if err != nil { failure("could not connect to WiFi: " + err.Error()) } - + scratchPool.New = func() interface{} { return make([]byte, scratchSize) } + var http httphi.MuxSlice http.Handle("/", logRequest(root)) http.Handle("/hello", logRequest(hello)) http.Handle("/cnt", logRequest(cnt)) @@ -51,65 +68,93 @@ func main() { http.Handle("/off", logRequest(LED_OFF)) http.Handle("/on", logRequest(LED_ON)) + var router httphi.Router + err = router.Configure(httphi.RouterConfig{ + FixedNumGoroutines: 4, + RequestHeaderBufferSize: 1024, // Google chrome requests are around 700 bytes in header size. + ResponseHeaderMinBufferSize: 128, // We won't be writing too much to headers. Unused request memory is reused on top of this. + RequestNumHeaderKVCap: 16, // Max number of headers we can expect. + Mux: &http, + }) + if err != nil { + failure("configure Router: " + err.Error()) + } + defer router.Shutdown() // Despawns goroutines. + + listener, err := Listen(link, port) + defer listener.Close() h, _ := link.Addr() host := h.String() println("HTTP server listening on http://" + host + port) - err = http.ListenAndServe(host+port, nil) - if err != nil { - failure("http.ListenAndServe: " + err.Error()) + for { + conn, err := listener.Accept() + if err != nil { + failure("failed to accept conn: " + err.Error()) + } + // conn.SetDeadline(time.Now().Add(time.Second)) + err = router.Handle(conn) + if err != nil { + conn.Close() + println("failed to handle connection: ", err.Error()) + } } } -func logRequest(h http.HandlerFunc) http.Handler { - return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - println(r.Method, r.URL.Path) - h(w, r) - }) +func logRequest(h httphi.HandlerFunc) httphi.HandlerFunc { + return func(exch *httphi.Exchange) { + println(exch.RequestMethod().String(), " ", exch.MuxPattern()) + h(exch) + } } -func root(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusOK) - io.WriteString(w, indexHTML) +func root(exch *httphi.Exchange) { + exch.RespondString(200, "text/html", indexHTML) } -func sixlines(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusOK) - io.WriteString(w, sixlinesHTML) +func sixlines(exch *httphi.Exchange) { + exch.RespondString(200, "text/html", indexHTML) } -func LED_ON(w http.ResponseWriter, r *http.Request) { +func LED_ON(exch *httphi.Exchange) { setLED(true) - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "led.High()") + exch.RespondString(200, "text/plain; charset=UTF-8", "led.High()") } -func LED_OFF(w http.ResponseWriter, r *http.Request) { +func LED_OFF(exch *httphi.Exchange) { setLED(false) - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "led.Low()") + exch.RespondString(200, "text/plain; charset=UTF-8", "led.Low()") } -func hello(w http.ResponseWriter, r *http.Request) { - w.Header().Set(`Content-Type`, `text/plain; charset=UTF-8`) - io.WriteString(w, "hello") +func hello(exch *httphi.Exchange) { + exch.RespondString(200, "text/plain; charset=UTF-8", "hello") } var counter int -func cnt(w http.ResponseWriter, r *http.Request) { - r.ParseForm() - if r.Method == "POST" { - c := r.Form.Get("cnt") - if c != "" { - i64, _ := strconv.ParseInt(c, 0, 0) +func cnt(exch *httphi.Exchange) { + scratch := scratchPool.Get().([]byte) + defer scratchPool.Put(scratch) + switch exch.RequestMethod() { + case httphi.MethPost: // POST + var form httpraw.Form + form.Reset(scratch, 2) // 2 query values max. + const parseURL, prioritizeURL = true, false + err := exch.RequestParseForm(&form, parseURL, prioritizeURL) + if err != nil { + exch.Respond(500, "", nil) + return + } + c := form.Get("cnt") + if len(c) > 0 { + i64, _ := strconv.ParseInt(string(c), 0, 0) counter = int(i64) + println("set counter", counter) } } - - w.Header().Set(`Content-Type`, `application/json`) - io.WriteString(w, `{"cnt": `) - io.WriteString(w, strconv.Itoa(counter)) - io.WriteString(w, `}`) + json := append(scratch[:0], `{"cnt": `...) + json = strconv.AppendInt(json, int64(counter), 10) + json = append(json, '}') + exch.Respond(200, "application/json", json) } func failure(msg string) { @@ -118,3 +163,22 @@ func failure(msg string) { time.Sleep(1 * time.Second) } } + +func Listen(link *netlink.Esplink, port string) (net.Listener, error) { + // Listen by asking the lneto stack for a socket directly instead of going + // through stdlib net.Listen and the netdev file descriptor layer due to a bug. + stack := link.StackGo() + laddr, err := netip.ParseAddrPort("0.0.0.0" + port) + if err != nil { + return nil, err + } + sock, err := stack.SocketNetip(context.Background(), "tcp4", xnet.AF_INET, xnet.SOCK_STREAM, laddr, netip.AddrPort{}) + if err != nil { + return nil, err + } + listener, ok := sock.(net.Listener) + if !ok { + return nil, errors.New("stack returned non-listener socket") + } + return listener, nil +} diff --git a/go.mod b/go.mod index db2cf6f..0c2a633 100644 --- a/go.mod +++ b/go.mod @@ -6,4 +6,4 @@ require tinygo.org/x/drivers v0.35.1-0.20260604174950-1d695a231aef require github.com/soypat/natiu-mqtt v0.7.0 -require github.com/soypat/lneto v0.2.1 +require github.com/soypat/lneto v0.3.0 diff --git a/go.sum b/go.sum index b0e538c..11f2d94 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,5 @@ -github.com/soypat/lneto v0.2.1 h1:o07lXylFCkRs4djA9PYrIf0oW+tMLjxO2/JbHv2XoK0= -github.com/soypat/lneto v0.2.1/go.mod h1:Be5PjwoYukvHFiUXxpYi8+ppH2F/gw/vjGBvFdv+Ti8= +github.com/soypat/lneto v0.3.0 h1:P9PPYejdkuvcG9z7HE47zDtEEXrWXDnSkxOn1hEPmEc= +github.com/soypat/lneto v0.3.0/go.mod h1:Be5PjwoYukvHFiUXxpYi8+ppH2F/gw/vjGBvFdv+Ti8= github.com/soypat/natiu-mqtt v0.7.0 h1:94nWBMt2Pxu+8uEXvOIUXLp6hNWv5912D1qPH6wbGAg= github.com/soypat/natiu-mqtt v0.7.0/go.mod h1:xEta+cwop9izVCW7xOx2W+ct9PRMqr0gNVkvBPnQTc4= tinygo.org/x/drivers v0.35.1-0.20260604174950-1d695a231aef h1:nG/qd6hSQonHse2l8DYUrCuJSiCfcFD9n8YMze8sVo0= diff --git a/netlink/netlink.go b/netlink/netlink.go index 1fb4d2f..c446fed 100644 --- a/netlink/netlink.go +++ b/netlink/netlink.go @@ -27,6 +27,7 @@ type Esplink struct { notifyCb func(nl.Event) netstack *espradio.Stack + gostack xnet.StackGo berkeley xnet.StackBerkeley stackloop sync.Once @@ -122,9 +123,9 @@ func (n *Esplink) NetConnect(params *nl.ConnectParams) error { } n.stackloop.Do(func() { // Start stack goroutine once. - gostack := n.netstack.LnetoStack().StackGo(pollBackoff, xnet.StackGoConfig{ + n.gostack = n.netstack.LnetoStack().StackGo(pollBackoff, xnet.StackGoConfig{ ListenerPoolConfig: xnet.TCPPoolConfig{ - PoolSize: 2, + PoolSize: 4, QueueSize: 4, TxBufSize: 4096, RxBufSize: 1024, @@ -134,7 +135,7 @@ func (n *Esplink) NetConnect(params *nl.ConnectParams) error { }, }) - n.berkeley = *xnet.NewBerkeleyStack(gostack.Socket) + n.berkeley = *xnet.NewBerkeleyStack(n.gostack.Socket) go handleStack(espstack) }) _, err = n.netstack.SetupWithDHCP(espradio.DHCPConfig{}) @@ -147,6 +148,8 @@ func (n *Esplink) NetConnect(params *nl.ConnectParams) error { return nil } +func (n *Esplink) StackGo() xnet.StackGo { return n.gostack } + // NetDisconnect device from network func (n *Esplink) NetDisconnect() { // TODO: implement this. For now, just do nothing and let the connection time out.