Proposal: live EC2 port publishing with security-group enforcement

[CoreOxide]

This proposes a separate design discussion for host port publishing while EC2 security-group enforcement is enabled.

Related work:

• Original request: [FEAT] EC2/ECS: enforce security group rules on container traffic #2564
• Implementation and review context: feat(ec2)!: complete security group enforcement #3771
• Hector's request to separate port publishing from the ENI and IPv6 work: feat(ec2)!: complete security group enforcement #3771 (comment)

Problem

The current socat publisher supports opening application ports on an already-running EC2 instance. That behavior matters: a security-group rule can authorize a new port after launch.

The launch-time Docker binding approach explored in #3771 does not preserve that behavior. Docker port bindings are fixed when a container is created, so putting those bindings on the protected namespace at startup leaves later-authorized ports unpublished. Recreating the workload to add a port would be a regression.

A plain TCP relay also creates a new connection to the application. Filtering that connection by the relay's address cannot establish whether the original peer was permitted.

Proposed direction

Keep the existing publisher when enforcement is disabled. Enforcement remains opt-in through FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_ENABLED=false by default.

For enforcement mode, investigate a separately managed, replaceable publishing helper for each published port, reusing existing port allocation and lifecycle patterns. Adding or removing a publication should affect that helper, not restart the EC2 workload or its protected network namespace.

The helper would use packet forwarding rather than terminate and recreate application connections. Every forwarded connection must cross the protected endpoint's security-group policy before reaching the application. Docker would still own host port bindings; Floci would manage only its own helper namespaces, rules, containers, and labels.

This is a direction to prototype, not a claim that a simple DNAT rule is sufficient. The prototype must establish the return path, source-address visibility, and required namespace connections on both rootful Linux Docker and Linux containers on Docker Desktop. If preserving the necessary identity requires a different topology, that result should come back to this discussion before implementation expands.

Required behavior

• Authorizing a new application port after launch can publish it without restarting the instance. A published listener is only transport availability; ingress rules must still decide which sources can connect.
• Managed peers are evaluated using logical ENI addresses and security-group membership, then translated to transport addresses. A publishing helper must not collapse distinct managed peers into one trusted identity or provide a bypass.
• New connections require the applicable ingress and egress permissions. Policy updates must not accidentally discard permitted tracked return traffic.
• Revocation immediately prevents newly forbidden connections. Listener removal and helper cleanup must account explicitly for established flows, rather than silently breaking connection tracking.
• A failed helper setup or policy update must not expose an unrestricted path. Install policy before making a publication reachable, and reconcile or quarantine failures.
• Stop/start, reboot, termination, restart recovery, and port-allocation cleanup must preserve the intended mapping without leaking helpers or reserving ports indefinitely.
• Workload containers receive no networking privileges. No changes to Docker-owned firewall rules.

Boundaries and alternatives

The initial scope is EC2 application host publishing. ECS bridge/host networking and new ECS publishing behavior are outside this proposal. No VPC routing, NACLs, NAT gateways, or peering are introduced.

Host-published ingress can only filter the source visible inside Docker. Original external-client IP fidelity on Docker Desktop is excluded and must be documented. Managed-peer identity preservation is a separate requirement and must be demonstrated.

Launch-only bindings do not meet the live-addition requirement. Recreating the workload to change bindings also does not meet it. Keeping an ordinary socat relay in the enforced path would require a separate, demonstrated way to enforce original-peer policy before identity is lost.

Until a design is agreed and validated, document that application host publishing, including ports authorized after launch, is unavailable with enforcement enabled. Preserve existing publishing behavior with enforcement disabled.

Validation before landing

Use SDK-driven management operations and real traffic on Linux and Docker Desktop to prove live additions, wrong-CIDR denial followed by correct-CIDR success, independent ingress/egress decisions, SG references, and IPv4/IPv6 TCP and UDP behavior. Exercise tracked connections during revocation, restart recovery, cleanup, and injected helper/policy failures. Verify that both direct and host-published access cannot bypass filtering.

The main design question is whether replaceable packet-forwarding helpers can meet these identity and lifecycle requirements with acceptable complexity. Feedback on that topology, return-path handling, and established-flow behavior would help determine the smallest viable implementation.

[CoreOxide]

@hectorvent is there a "waiting for community review" tag or something? 😆

[upwardsolutions-inc]

Nice writeup. Since socat and the firewall from #3681 are already in, this feels more like reworking how publishing works under enforcement than building something new.

I'd prototype the tricky bit first: can forwarding actually keep the real peer address on Docker Desktop, not just on Linux? If that doesn't hold up, the rest of the design probably changes, so it seems worth knowing early.

Meanwhile, the doc note saying publishing is off when enforcement is on could probably go in right away. It's small and doesn't depend on anything else.

I'd also start with plain IPv4 TCP on Linux and grow from there. One open question: if a rule gets revoked, should existing connections stay up like they do in AWS, or get cut?

Happy to test on Docker Desktop once there's something to try.

[CoreOxide]

Thanks @upwardsolutions-inc. I also feel that this would require more testing that work. Even with AI, there are a lot of gaps here that could be easily overlooked. Lets wait a few more days, see if anyone else has a comment. Then get to POC.

[CoreOxide]

@upwardsolutions-inc I followed up with a local IPv4 TCP prototype on Windows Docker Desktop with its WSL2 backend, running the Java harness in Ubuntu. Management operations used the AWS SDK.

Tested against upstream main a707cb326 (Floci 2.4.0), Docker Engine 29.8.2, Docker Desktop CLI 4.95.0, and WSL 2.7.14.0 in NAT mode.

The transport was a separate per-port helper doing DNAT without SNAT or a TCP relay, forwarding into the existing protected namespace and an unprivileged echo workload.

Two blockers showed up:

1. Managed identity is lost on Desktop's host-published path. Direct connections to the helper preserved the two peers' transport addresses, 172.17.0.3 and 172.17.0.4, and logical-CIDR/SG-reference filtering worked. Through host.docker.internal:30000 (IPv4 192.168.65.254), both peers reached the helper and target as 172.17.0.1. Windows and Ubuntu localhost connections also appeared as that address. Allowing its /32 for host access then admitted the otherwise-forbidden managed peer. The identity loss happens before our helper, so DNAT alone cannot recover it.

2. Per-source static return routes do not support concurrent helpers for the same source. Adding a second helper left the first tracked connection intact, but the second published port could not return traffic through the existing route. This rejects the bounded routing approach, rather than proving packet-forwarding helpers impossible.

Other checks passed: independent sender-egress denial, fresh connections denied after narrow-rule revocation while the tracked connection continued, unchanged workload ID/start time/namespace, and default-deny behavior after injected policy-installation failure. All seven experiment containers were removed and allocated ports released.

The opted-in test reports four failed acceptance assertions, with zero errors or skips. Checkstyle passes; without opt-in the test is skipped. The harness and report are still local, not published yet.

The documentation note disabling application publishing under enforcement is already present on current main. I'd keep that restriction. Before production integration, we need a topology that preserves managed identity or enforces the applicable peer policy before identity is lost, plus a validated return path for concurrent publications.

These results cover this Windows/Desktop/WSL2 configuration only. Native Linux Docker, IPv6, UDP, restart recovery, and AWS's untracked-flow exceptions were not validated.

[CoreOxide]

The prototype is now pushed to a side branch: CoreOxide/floci: codex/ec2-live-publishing-poc, commit 3a6671369.

• Results, tested environment, topology limits, and reproduction instructions
• Opt-in integration test

From Ubuntu WSL2 with Java 25 and Docker Desktop WSL integration enabled:

FLOCI_EC2_PUBLISHING_POC=true sh ./mvnw \
  -Dtest=Ec2LivePublishingPocIntegrationTest test

On the tested Desktop/WSL2 NAT configuration, the expected experimental outcome is a failed test with four acceptance failures: both managed-peer identity checks, denial of the forbidden peer after host access is granted, and the second publication's return path. The report explains the passing checks as well.

This branch adds only the test and report. Production publishing stays disabled under enforcement. Checkstyle passes, and the test is skipped unless explicitly opted in.

[CoreOxide]

@hectorvent would love your input here. This is a substantial change.