A helm deployment procedure for free5gc
[Ubuntu 20.04 LTS Server] (https://www.releases.ubuntu.com/focal/)
4 GPU, 10GB RAM, 25GB virtual hard disk, two NAT network adapters
uname -r
output:
5.4.0-163-generic
port forwarding:
Host Address:
Host Port: 8022
Guest Address: [address for enp0s3/eth0/ens3]
Guest Port: 22
configure another NAT network interface
ssh into Ubuntu Server:
ssh -p 8022 [email protected]
check VM networks:
ip a
output:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:df:4e:37 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 86364sec preferred_lft 86364sec
inet6 fe80::a00:27ff:fedf:4e37/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 08:00:27:3d:1c:53 brd ff:ff:ff:ff:ff:ff
configure netplan:
sudo vim /etc/netplan/00-installer-config.yaml
network:
ethernets:
enp0s3:
dhcp4: true
enp0s8:
dhcp4: true
version: 2
sudo netplan try
check configurations:
ip a
output:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:df:4e:37 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 86396sec preferred_lft 86396sec
inet6 fe80::a00:27ff:fedf:4e37/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:3d:1c:53 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.15/24 brd 10.0.3.255 scope global dynamic enp0s8
valid_lft 86396sec preferred_lft 86396sec
inet6 fe80::a00:27ff:fe3d:1c53/64 scope link
valid_lft forever preferred_lft forever
now we have two VM network interfaces belong to two different subnets.
install Microk8s
sudo snap install microk8s --classic --channel=1.28
enable kube-ovn:
sudo microk8s enable kube-ovn --force
here I use kube-ovn CNI to avoid the ipv4_forwarding problem in the calico CNI
join Microk8s cluster (make sure to join the cluster after enable kube-ove otherwise you may lose control of the cluster):
sudo usermod -a -G microk8s $USER\
&& sudo chown -f -R $USER ~/.kube
su - $USER
enable multus:
microk8s enable community\
&& microk8s enable multus
(re)enable dns:
microk8s disable dns\
&& microk8s enable dns
when Microk8s is ready, type microk8s status to check, it should have these addons enabled:
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
multus # (community) Multus CNI enables attaching multiple network interfaces to pods
community # (core) The community addons repository
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
kube-ovn # (core) An advanced network fabric for Kubernetes
disabled:
...
install kubectl:
sudo snap install kubectl --classic
install helm:
sudo snap install helm --classic
enable microk8s to work with VM's kubectl:
cd $HOME
cd .kube
microk8s config > config
install gtp5g v8.0.1 (need to be compatible with kernel version)
sudo apt install gcc
sudo apt install make
git clone -b v0.8.1 https://github.com/free5gc/gtp5g.git
cd gtp5g
make
sudo make install
cd --
see also install-gtp5g
download free5gc helm charts
git clone https://github.com/Orange-OpenSource/towards5gs-helm.git
check the gateway of VM network interfeces:
ip r
output:
default via 10.0.3.2 dev enp0s8 proto dhcp src 10.0.3.15 metric 100
default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
10.0.3.0/24 dev enp0s8 proto kernel scope link src 10.0.3.15
10.0.3.2 dev enp0s8 proto dhcp scope link src 10.0.3.15 metric 100
blackhole 10.1.214.128/26 proto 80
10.1.214.135 dev cali032701d8fbd scope link
10.1.214.136 dev calicd7aa8cce63 scope link
configure n6 network for upf DN connection:
cd --\
&& vim towards5gs-helm/charts/free5gc/values.yaml
n6network:
enabled: true
name: n6network
type: ipvlan
masterIf: enp0s8
subnetIP: 10.0.3.0
cidr: 24
gatewayIP: 10.0.3.2
excludeIP: 10.0.3.254
masterIf should be the name of the second interface, subnetIP should be the subnet directly connected to the second interfece, gatewayIP shoud be the gateway IP of the second interface (not the ip of the second interface).
configure the name of the default interface (first interface) as the masterIf for other(n2, n3, n4 and n9):
:%s/eth0/enp0s3/gc
for now I leave the default IP configurations (10.100.50.0/24) of n2, n3, n4 and n9 unchanged because I deploy all 5g core and gNB functions on one kubernetes node/VM. This may change if you need to deploy them over a kubernetes cluster.
also remember to configure ip for n6 interface, it needs to be an ip address in the n6 network/subnet.
vim towards5gs-helm/charts/free5gc/charts/free5gc-upf/values.yaml
n3if: # GTP-U
ipAddress: 10.100.50.233
n4if: # PFCP
ipAddress: 10.100.50.241
n6if: # DN
ipAddress: 10.0.3.12
configure 8G of persistent volume with this file
cd --
vim persistentvolume-definition.yml
persistentvolume-definition.yml:
apiVersion: v1
kind: PersistentVolume
metadata:
name: [persistent volume name]
labels:
project: free5gc
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
local:
path: /home/[username]/kubedata
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- [node name]
in my case, [persistent volume name] is free5gc-pv0, replace [username] with your username and [node name] with the name of your kubernetes worker node.
deployment and test with ueransim
now, deploy free5gc:
cd --
mkdir kubedata \
&& kubectl create ns free5gc \
&& kubectl apply -f persistentvolume-definition.yml \
&& cd towards5gs-helm/charts/ \
&& helm -n free5gc install free5gc-v1 ./free5gc/ \
&& watch kubectl get pods -n free5gc
if successful:
Every 2.0s: kubectl get pods -n free5gc free5gc: Wed Sep 27 11:28:47 2023
NAME READY STATUS RESTARTS AGE
free5gc-v1-free5gc-upf-upf-555cfb4d7f-k9lxq 1/1 Running 0 64s
mongodb-0 1/1 Running 0 64s
free5gc-v1-free5gc-nrf-nrf-6699c64fd7-8xt9g 1/1 Running 0 64s
free5gc-v1-free5gc-dbpython-dbpython-5d46c79f86-vjpjm 1/1 Running 0 64s
free5gc-v1-free5gc-webui-webui-54f45bb69f-wvsjg 1/1 Running 0 64s
free5gc-v1-free5gc-pcf-pcf-5fc9b5f98f-5pgxh 1/1 Running 0 64s
free5gc-v1-free5gc-udm-udm-5d8d96c57b-qkcvh 1/1 Running 0 64s
free5gc-v1-free5gc-amf-amf-6fc4bf584-pbrf6 1/1 Running 0 64s
free5gc-v1-free5gc-udr-udr-6dfc5f6f74-9bwch 1/1 Running 0 64s
free5gc-v1-free5gc-smf-smf-58b7874499-72hb8 1/1 Running 0 64s
free5gc-v1-free5gc-nssf-nssf-58c6cddc69-bxq9r 1/1 Running 0 64s
free5gc-v1-free5gc-ausf-ausf-75586797cf-wpg57 1/1 Running 0 64s
if not, check the configrations or post an issue.
check upf interfces:
kubectl exec -it -n free5gc [upf pod id] -- ip a
it should give something like this (with different ip addresses):
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if166: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 2e:83:7a:d0:2c:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.214.169/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::2c83:7aff:fed0:2cc5/64 scope link
valid_lft forever preferred_lft forever
4: n3@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 08:00:27:df:4e:37 brd ff:ff:ff:ff:ff:ff
inet 10.100.50.233/29 brd 10.100.50.239 scope global n3
valid_lft forever preferred_lft forever
inet6 fe80::800:2700:3df:4e37/64 scope link
valid_lft forever preferred_lft forever
5: n6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 08:00:27:3d:1c:53 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.12/24 brd 10.0.3.255 scope global n6
valid_lft forever preferred_lft forever
inet6 fe80::800:2700:13d:1c53/64 scope link
valid_lft forever preferred_lft forever
6: n4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 08:00:27:df:4e:37 brd ff:ff:ff:ff:ff:ff
inet 10.100.50.241/29 brd 10.100.50.247 scope global n4
valid_lft forever preferred_lft forever
inet6 fe80::800:2700:4df:4e37/64 scope link
valid_lft forever preferred_lft forever
7: upfgtp: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1464 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet6 fe80::8ec6:e4da:9c15:9866/64 scope link stable-privacy
valid_lft forever preferred_lft forever
check n6 connection to the DN:
kubectl exec -it -n free5gc [upf pod id] -- ping -I n6 8.8.8.8
it should work:
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=112 time=10.474 ms
64 bytes from 8.8.8.8: seq=1 ttl=112 time=17.193 ms
...
to test uesimtun0 form ue, add ue profile to amf with webui service.
forward webui service to host machine
kubectl port-forward --namespace free5gc svc/webui-service 5000:5000
listen to port 5000 from host:
ssh -L localhost:5000:localhost:5000 -p 8022 [email protected]
access webui form host's broswer:
http://localhost:5000
login and add a new subscriber.
configure ueransim (n2, n3 interface):
vim towards5gs-helm/charts/ueransim/values.yaml
n2network:
enabled: true
name: n2network
type: ipvlan
masterIf: enp0s3
subnetIP: 10.100.50.248
cidr: 29
gatewayIP: 10.100.50.254
excludeIP: 10.100.50.254
n3network:
enabled: true
name: n3network
type: ipvlan
masterIf: enp0s3
subnetIP: 10.100.50.232
cidr: 29
gatewayIP: 10.100.50.238
excludeIP: 10.100.50.238
deploy ueransim:
cd --\
&& cd towards5gs-helm/charts/ \
&& helm -n free5gc install ueransim-v1 ./ueransim/
if successful:
Every 2.0s: kubectl get pods -n free5gc free5gc: Wed Sep 27 11:40:41 2023
NAME READY STATUS RESTARTS AGE
free5gc-v1-free5gc-upf-upf-555cfb4d7f-fk7xw 1/1 Running 0 33s
mongodb-0 1/1 Running 0 33s
free5gc-v1-free5gc-webui-webui-54f45bb69f-wg5rn 1/1 Running 0 33s
free5gc-v1-free5gc-dbpython-dbpython-5d46c79f86-m5lnz 1/1 Running 0 33s
free5gc-v1-free5gc-nrf-nrf-6699c64fd7-vs8b9 1/1 Running 0 33s
free5gc-v1-free5gc-pcf-pcf-5fc9b5f98f-vkhbx 1/1 Running 0 33s
free5gc-v1-free5gc-nssf-nssf-58c6cddc69-7l7j5 1/1 Running 0 33s
free5gc-v1-free5gc-ausf-ausf-75586797cf-r8xb6 1/1 Running 0 33s
free5gc-v1-free5gc-udr-udr-6dfc5f6f74-thqs7 1/1 Running 0 33s
free5gc-v1-free5gc-udm-udm-5d8d96c57b-jz6cm 1/1 Running 0 33s
free5gc-v1-free5gc-smf-smf-58b7874499-4pfc2 1/1 Running 0 33s
ueransim-v1-ue-65985c8bfd-4x9dq 1/1 Running 0 32s
free5gc-v1-free5gc-amf-amf-6fc4bf584-nwt4r 1/1 Running 0 33s
ueransim-v1-gnb-7ccf5f7bf9-xx2rw 1/1 Running 0 32s
if not, check the configrations or post an issue.
kubectl exec -it -n [upf pod id] -- cat /proc/sys/net/ipv4/ip_forward
it should give:
1
otherwise, check document here and solution here or use microk8s kube-ovn as CNI. check uesimtun0:
kubectl exec -it -n [ue pod id] -- ip a
output should have uesimtun0 in the interfeaces test uesimtun0:
kubectl exec -it -n free5gc ueransim-v1-ue-65985c8bfd-8w5x5 -- ping -I uesimtun0 8.8.8.8
and it should work:
PING 8.8.8.8 (8.8.8.8) from 10.1.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=12.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=10.7 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=111 time=12.2 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=111 time=13.3 ms
...
to stop free5gc for this deployment, run:
kubectl delete -n free5gc pod,svc --all \
&& kubectl delete -n free5gc deployment --all \
&& kubectl delete ns free5gc \
&& kubectl delete PersistentVolume free5gc-pv0 \
&& cd -- \
&& sudo rm -rf kubedata/
helm -n free5gc uninstall free5gc-v1
kubectl delete ns free5gc
cd --
sudo rm -rf kubedata/
sudo ip link add name f5gcn6 link ens5 type ipvlan mode l2
sudo ip link set f5gcn6 up
sudo ip addr add 10.128.2.15/16 dev f5gcn6
sudo ip route add default via 10.128.0.1 dev f5gcn6
ping 8.8.8.8 -I f5gcn6