I am failing to make port forwarding from KVM host to KVM guest work

[TM]

Dear All,

I have tried hard, and finally failed to get port forwarding working so that other clients on the external laptop host KVM network access the KVM guest Apache httpd.
So I come here for guidance and/or advices.
In the following lines, and I do apologize in advance if they are long, I expose what I have done.

Thank you in advance.

I have just installed Fedora 42 LXQt on an HP EliteBook 840 G6 laptop.

I then have upgraded it with the below command.

dnf -y upgrade

So after I rebooted it, it has all the latest and required packages.

Futhermore, I have installed KVM, configured an AlmaLinux 9 KVM guest machine.
I omit the commands that did the above. I can put them if required. Just let me know.
The almalinux KVM guest machine have then been upgraded with the below command.

dnf -y upgrade

Finally I have installed, enabled, and allowed Apache httpd on it with the below commands.

dnf -y httpd
systemctl enable --now httpd
firewall-cmd --permanent --add-service=http
firewall-cmd --reload
echo 'hello from almalinux' > /var/www/html/index.html

So now I have fully upgraded and working systems as below.
1) Fedora 42 LXQt HP laptop (Linux kernel 6.15.5-200.fc42.x86_64, NetworkManager-1.52.1-1.fc42.x86_64, libvirt-libs-11.0.0-3.fc42.x86_64, firewalld-2.3.0-5.fc42.noarch), with the IP 192.168.1.100 from my Mikrotik access point router and the IP 192.168.122.1 in KVM libvirt network
2) AlmaLinux 9 KVM guest (Linux kernel 5.14.0-570.37.1.el9_6.x86_64, NetworkManager-1.52.0-5.el9_6.x86_64, firewalld-1.3.4-9.el9_5.noarch, httpd-2.4.62-4.el9.x86_64), with the IP 192.168.122.180 in KVM libvirt network

Obviously the Apache httpd is working and responding to clients. The below output from the KVM host proves it.h

curl http://192.168.122.180/
hello from almalinux

I did not want to follow my old fix for this purpose, as it relied on iptables that is now fully deprecated but still officially referenced here.

I have try to go the new way with firewalld libvirt zone and customed policy, and followed this link.

I have read various sources official and not such as this and this...

But still, it does not work.

All responses, guidance and advices, or requests for more details are welcome.

Best regards,

Tshimanga

[Chetan_Tiwary_]

@TM did you try changing the libvirt's default NAT mode to routed mode and see if it helped ? I mean chnge the mode and then restart your network and then enable masquerading via firewall rich rule.

[TM]

@Chetan_Tiwary_,

No, I did not modify the default NAT mode.
And I don't want to use routed mode, that will require specific route to also be defined in the external network.

I am looking for the smallest set of modifications to be done from default settings.

Regards,

[TM · Accepted solution]

Dear All,

I have reached this solution after reading several documentations, trying and testing several commands and configuration. I worked on it during the whole week with several hours daily. And it might not be the best solution, but at least it fulfills my need.
So I shared it here without any warranty.

The detailed solution with nice formating is available on LinuxQuestions where I first started the discussion.

I will have wanted to put back the whole solution here, but without being able to format code in reply, that is not actually possible. Hopefully this issue will be addressed as it has already been raised here.

The KVM host runs Fedora 42. So it has the below software and versions.
Linux kernel 6.15.5-200.fc42.x86_64
firewalld-2.3.0-5.fc42.noarch
NetworkManager-1.52.1-1.fc42.x86_64
libvirt-libs-11.0.0-3.fc42.x86_64

Within that KVM host runs an AlmaLinux 9 KVM guest that has a reachable Apache httpd server listening on IP address 192.168.122.100 and tcp port 80.
So what I want to achieve is having the traffic coming on tcp port 30080 on the KVM host external IP address (what it really is is actually irrelevant for my use case) be forwarded to the KVM guest IP address of 192.168.122.100 on tcp port 80.

With default configuration and options, Fedora 42 enables and runs firewalld.service, but not nftables.service.
firewalld.service will add in the nft ruleset the table inet firewalld. That table adds all its chains with the priority offset of + 10.
libvirt when started will independently add in the nft ruleset the table ip libvirt_network and table ip6 libvirt_network. Those tables have their chains with no priority offset. That means they will be executed before the ones added by firewalld.service for the same hook.

Basically to allow the port forwarding few main setup configuration is required.
1) The first one is intercept the traffic coming at the tcp port 30080 and forward it to IP address 192.168.122.100 on tcp port 80.
2) The second one is allow that traffic to go through.
3) Additionally a third one is allow responses in the reverse path order to reach the original client.

All the commands below are executed as root and with root privileges.
Being comfortable working from bash command line, and commands such firewall-cmd and nft is required.

3) By design (and by default), libvirt does masquerade traffic initiated from the KVM guests that needs to go elsewhere (not the KVM host, neither another KVM guest). That takes care of the third additional required setup configuration. So nothing to do.

1) For the first setup configuration, we just need to execute an --add-forward-port firewall-cmd command.
For our case, it is as below. Remark that without mentioning the zone it will affect the default zone, which is FedoraServer in my Fedora 42 case.
# firewall-cmd --add-forward-port=port=30080:proto=tcp:toport=80:toaddr=192.168.122.100
success

2) The tricky parts are all related to the second setup configuration.

Normally as far as firewalld.service is concerned the above should have been enough. Because firewalld.service puts in its filter forward hooked chain a rule (see below with handle 200) that allows the forwarded traffic to go through.
# nft -a list chain inet firewalld filter_FORWARD
table inet firewalld {
chain filter_FORWARD { # handle 186
type filter hook forward priority filter + 10; policy accept;
ct state { established, related } accept # handle 199
ct status dnat accept # handle 200
iifname "lo" accept # handle 201
ct state invalid log prefix "STATE_INVALID_DROP: " # handle 202
ct state invalid drop # handle 203
ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 addr-unreachable # handle 220
jump filter_FORWARD_POLICIES # handle 205
log prefix "FINAL_REJECT: " # handle 206
reject with icmpx admin-prohibited # handle 207
}
}
With nft syntax: dnat means destination network address translation, and ct status dnat identifies exactly the connection tracking status of a packet belonging to a forwarded traffic.

But that rule (with handle 200 above) is never reached (because of its chain priority being filter + 10), and instead the rule below (with handle 11) is executed (because of its upper chain forward priority being filter) and that rule rejects the traffic. That executed rule is added by libvirt. libvirt acts and adds its nft tables, chains (with no priority offset) and rules without taking care of firewalld.service.
# nft -a list table ip libvirt_network
table ip libvirt_network { # handle 5
chain forward { # handle 1
type filter hook forward priority filter; policy accept;
counter packets 102 bytes 7752 jump guest_cross # handle 7
counter packets 102 bytes 7752 jump guest_input # handle 5
counter packets 51 bytes 3876 jump guest_output # handle 3
}
chain guest_output { # handle 2
ip saddr 192.168.122.0/24 iif "virbr0" counter packets 51 bytes 3876 accept # handle 13
iif "virbr0" counter packets 0 bytes 0 reject # handle 10
}
chain guest_input { # handle 4
oif "virbr0" ip daddr 192.168.122.0/24 ct state established,related counter packets 51 bytes 3876 accept # handle 14
oif "virbr0" counter packets 0 bytes 0 reject # handle 11
}
chain guest_cross { # handle 6
iif "virbr0" oif "virbr0" counter packets 0 bytes 0 accept # handle 12
}
chain guest_nat { # handle 8
type nat hook postrouting priority srcnat; policy accept;
ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 counter packets 2 bytes 170 return # handle 21
ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return # handle 20
meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 # handle 19
meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 51 bytes 3876 masquerade to :1024-65535 # handle 18
ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade # handle 17
}
}

So to allow the forwarded traffic to go through we need to insert a rule (see below with handle 22) that accepts the forwarded traffic before the rule that will reject it.
# nft insert rule ip libvirt_network guest_input handle 11 ct status dnat accept
# nft -a list chain ip libvirt_network guest_input
table ip libvirt_network {
chain guest_input { # handle 4
oif "virbr0" ip daddr 192.168.122.0/24 ct state established,related counter packets 60 bytes 4560 accept # handle 14
ct status dnat accept # handle 22
oif "virbr0" counter packets 0 bytes 0 reject # handle 11
}
}
With that in place, the rule (with handle 22) will allow the forwarded traffic to go through.

Et voilà !

Best regards,

Tshimanga

[Chetan_Tiwary_]

@TM Thanks for letting us know here , totally forgot about this.