Part 3 of 4:
A virtualized network with OpenBSD's vmm
UPDATE: As I already suggested in the outlook of my last post, I wanted to exchange nginx with relayd for redirection. I did this now, thanks to an answered question on stackoverflow. See the new section: Configuring relayd on the host-vm.
Hello dear reader,
in my last post I described the setup of a http server which is reachable through a public domain on OpenBSD. This time I would like to go a little bit further and extend the server with a network of virtual machines, where each machine can be reached by the name of the subdomain it should represent.
Intro
For this setup to work, I needed to download two files from the OpenBSD site:
- The installation file of the most recent OpenBSD system (at the time it was 6.1)
- The boot file (bsd.rd)
Afterwards I had to install the VM, configure it, duplicate it for the virtual network.
Then I had to configure the main server and everything worked like a charm in the end.
For better explenation I will include a similar figure as show in my last two posts.
You can see that there are serveral components to be configured for the virtual network to work as desired in the end:
I need to…:
- …configure
pffor redirection and NAT - …configure
unboundfor acting as a DNS server - …create three different VMs (host-vm, foo-vm, bar-vm)
- …connect the VMs to a virtual switch called “localnet”
- …configure
relaydon the host-vm for redirection of subdomain requests
Outline
Here a short outline to jump back and forth (turn off noscript for a second to see it :-) ):
Create the VMs
This section will deal about creating a raw disk image and installing OpenBSD on it via the OpenBSD included vmd framework. This includes downloading some necessary resources and creating and installing a VM with the application called vmctl.
Afterwards the created installation will be duplicated and set up for the different VMs.
Create the Host VM
Choose my mirror to download from
There are many mirrors that I could choose from: listed here on the OpenBSD website.
Download the files
At first I located the necessary files in a webbrowser on my system. I chose the mirror in Frankfurt, so the main url was https://ftp.hostserver.de/pub/OpenBSD/.
There I chose the most recent OpenBSD version 6.1 and navigated to the directory for amd64 ending here: https://ftp.hostserver.de/pub/OpenBSD/6.1/amd64/.
The necessary files are:
install61.fsthe OpenBSD installation image, that I already downloaded when I created a bootable USB Stick for my OpenBSD Installation.bsd.rdthe boot image of OpenBSD.
I opened an ssh-session to the server and downloaded those files via ftp (I’m using ftp now and not wget, as it is in base, thank you for the comment Raf):
cd ~
mkdir vm-network
cd vm-network
ftp https://ftp.hostserver.de/pub/OpenBSD/6.1/amd64/install61.fs
ftp https://ftp.hostserver.de/pub/OpenBSD/6.1/amd64/bsd.rdAs PengouinBSD correctly suggested in the comments, I should also check the downloaded files:
cd vm-network
ftp https://ftp.hostserver.de/pub/OpenBSD/6.1/amd64/SHA256.sig
signify -Cp /etc/signify/openbsd-61-base.pub -x SHA256.sig install61.fs bsd.rdFrom now on I assume you also have a folder called vm-network in your user’s home directory.
Create a raw disk file and start the installation
I’m located in my dedicated folder vm-network, where the previously downloaded files install61.fs and bsd.rd are located.
cd ~/vm-networkWithin this folder I created a raw disk image called host.drive with 8GB:
doas vmctl create host.drive -s 8GAnd started installing on this disk interactively with a console:
doas vmctl start "host-vm" -c -b bsd.rd -m 2G -i 1 -d host.drive -d install61.fsThe options that are provided to vmctl mean the following:
start "host-vm": start the virtual machine, naming it “host-vm”-c: immediately connect the console of the vm into the current shell-b bsd.rd: choose the specified image as boot kernel-m 2G: start the virtual machine with two gigabytes of memory-i 1: create a virtual network interface connected to that virtual machine-d host.drive: attach the raw disk file where to install OpenBSD on-d install61.fs: attach the image with the OpenBSD 6.1 installation files
| A TIP TO PREVENT FUTURE HEADACHES |
|---|
If you ever run into the problem of this annoying Error when running the command then you’re probably missing the device file for the taps in …and so on for each new tap device you need. I took this tip from here when I ran into a simmilar problem. |
Guided Installation of OpenBSD for the host-vm
In Table 1 below I show the whole installation process and what I did including some comments.
| Terminal Output | My Input | Comments |
|---|---|---|
|
|
I want to install OpenBSD |
|
return |
|
|
|
Call it host, because it will be the host for the vm net. |
|
return |
Yes, configure vio0. |
|
|
ip address of my vm in the virtual network |
|
return |
|
|
return |
Don’t need an IPv6 address. |
|
return |
I’m done. |
|
|
The address of the router that manages my virtual network. I chose 192.168.30.1 for the subnet 192.168.30.0/24. |
|
||
|
|
The root domain (In the end it will resolve to the hostname |
|
|
My DNS nameserver should be serving on the virtual NIC with that IP. |
|
2x |
I did choose another password of course.. :D |
|
return |
Yes I want to connect via ssh later on. |
|
return |
sure… |
|
return |
definitely… |
|
|
I want the user called “host” |
|
return |
Don’t need a full name -.- |
|
2x |
Another safe password… |
|
return |
I don’t want to support a direct root login via ssh. |
|
|
List me the disks! |
|
Ok it seems the sd0 with 8G is the one I want. | |
|
return |
Yes, take sd0. |
|
return |
Use the whole disk! |
|
return |
Auto layout worked for me.. |
|
return |
Don’t need another disk. |
|
|
Ok the sets are on the disk with “install61.fs” on. |
|
return |
No the other disk is not mounted yet. |
|
return |
Ok sd0 was the 8G so sd1 should be the one I want. |
|
return |
mmh, yes seems as if partition |
|
return |
That pathname sounds good to me |
|
|
I want them all!! |
|
return |
Ok, now you can go on. |
|
|
There is no way around that.. |
|
return |
That’s it with the sets. |
|
|
Aha… ok I guess there is a Berlin in Europe… |
|
|
…so I take that. |
|
||
|
Made it! |
Ok now he is telling me to reboot the machine… But I can easily just choose to exit the vm via this key-sequence:
~^DSo first enter the tilde symbol and afterwards Ctrl-D.
The tilde is not supposed to apear in the terminal input.
Thereafter I stopped the created virtual machine:
doas vmctl stop "host-vm"And made a copy of the installation:
doas cp host.drive host.drive.bakI also checked if everything installed fine:
doas vmctl start "host-vm" -c -i 1 -d host.driveIt should boot into the installed system now asking you for login and password in the end.
I configured doas and added the /etc/installurl file as shown in my last post.
Duplicate the Host VM for other VMs I want
Now I could easily duplicate the created disk file for other needed VMs:
doas cp host.drive foo.drive
doas cp host.drive bar.driveAfterwards I started each VM and did some adjustments (I will only show what I did for the “foo” vm as the “bar” vm will be configured analogously:
Starting the VM as usual:
doas vmctl start "foo-vm" -c -i 1 -d foo.driveAdd the foo user:
doas adduserI copied my terminal output:
Couldn't find /etc/adduser.conf: creating a new adduser configuration file
Reading /etc/shells
Enter your default shell: csh ksh nologin sh [ksh]:
Your default shell is: ksh -> /bin/ksh
Default login class: authpf bgpd daemon default pbuild staff unbound
[default]:
Enter your default HOME partition: [/home]:
Copy dotfiles from: /etc/skel no [/etc/skel]:
Send welcome message?: /path/file default no [no]:
Do not send message(s)
Prompt for passwords by default (y/n) [y]:
Default encryption method for passwords: auto blowfish [auto]:
Use option ``-silent'' if you don't want to see all warnings and questions.
Reading /etc/shells
Check /etc/master.passwd
Check /etc/group
Ok, let's go.
Don't worry about mistakes. There will be a chance later to correct any input.
Enter username []: foo
Enter full name []:
Enter shell csh ksh nologin sh [ksh]:
Uid [1001]:
Login group foo [foo]:
Login group is ``foo''. Invite foo into other groups: guest no
[no]: wheel
Login class authpf bgpd daemon default pbuild staff unbound
[default]:
Enter password []:
Enter password again []:
Name: foo
Password: ******
Fullname: foo
Uid: 1001
Gid: 1001 (foo)
Groups: foo wheel
Login Class: default
HOME: /home/foo
Shell: /bin/ksh
OK? (y/n) [y]:
Added user ``foo''
Copy files from /etc/skel to /home/foo
Add another user? (y/n) [y]: n
Goodbye!Deleted the host user:
doas userdel host
doas rm -rf /home/hostConfigured the new hostname:
## /etc/hosts
127.0.0.1 localhost
::1 localhost
192.168.30.10 foo.hermes-technology.de foo## /etc/myname
foo.hermes-technology.deAlter the /etc/hostname.vio0 for the correct ip (it will be 192.168.30.4 for the bar-vm):
## /etc/hostname.vio0
inet 192.168.30.3 255.255.255.0Then I enabled httpd to show a “Hello from foo” html page.
Refer to my last post to see how to setup httpd for this task:
I exited the VM.
I did the above for each VM that I added to the virtual network.
Back on the main server
Ok now that I had the host-vm, foo-vm and bar-vm successfully set up I had to do some configuration on the main server.
At first I created a virtual NIC that acts as a Router and DNS server to the virtual-machine-network.
Then I configured the internal firewall pf and the DNS service via unbound.
At the end I configured the virtual machines for vmd in /etc/vm.conf.
The virtual network interface card (NIC)
Creating a virtual network device was very easy in OpenBSD.
I just had to create a file called /etc/hostname.vether0 and insert the desired configuration:
doas -s
touch /etc/hostname.vether0
echo "inet 192.168.30.1 255.255.255.0 NONE" > /etc/hostname.vether0Now on next reboot or after restarting the networking service I had the vether0 device at 192.168.30.1.
PF - the OpenBSD firewall
There is a configuration file for the firewall at /etc/pf.conf, I configured it like this:
# $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf
int_if="{ vether0 em0 }"
##set skip on lo
##block return # block stateless traffic
##pass # establish keep-state
# By default, do not permit remote connections to X11
##block return in on ! lo0 proto tcp to port 6000:6010
set block-policy drop
set loginterface egress
set skip on lo0
## act as a nat
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)
## allow all outgoing
pass out quick inet
pass in on $int_if inet
## redirect http and https to the host of the vms
pass in on egress inet proto tcp from any to (egress) port { 80 443 } rdr-to 192.168.30.2Now incoming http traffic will be redirected to the host-vm as soon as it is up and running.
To apply the configuration changes I executed doas pfctl -f /etc/pf.conf.
And verified those changes via doas pfctl -sr.
unbound - my DNS name server
I enabled unbound:
doas rcctl enable unboundAnd configured it the way I needed it:
## /var/unbound/etc/unbound.conf
# $OpenBSD: unbound.conf,v 1.7 2016/03/30 01:41:25 sthen Exp $
## I definately want the vether0 device act as a DNS name server for the virtual network
server:
interface: 192.168.30.1
interface: 127.0.0.1
interface: ::1
#do-ip6: no
# override the default "any" address to send queries; if multiple
# addresses are available, they are used randomly to counter spoofing
#outgoing-interface: 192.0.2.1
#outgoing-interface: 2001:db8::53
access-control: 0.0.0.0/0 refuse
access-control: 127.0.0.0/8 allow
access-control: ::0/0 refuse
access-control: ::1 allow
access-control: 192.168.1.0/24 allow
access-control: 192.168.30.0/24 allow ## I want the virtual network to be allowed for access
do-not-query-localhost: no
hide-identity: yes
hide-version: yes
# Uncomment to enable qname minimisation.
# https://tools.ietf.org/html/draft-ietf-dnsop-qname-minimisation-08
#
# qname-minimisation: yes
# Uncomment to enable DNSSEC validation.
#
#auto-trust-anchor-file: "/var/unbound/db/root.key"
# Serve zones authoritatively from Unbound to resolver clients.
# Not for external service.
#
#local-zone: "local." static
#local-data: "mycomputer.local. IN A 192.0.2.51"
#local-zone: "2.0.192.in-addr.arpa." static
#local-data-ptr: "192.0.2.51 mycomputer.local"
# UDP EDNS reassembly buffer advertised to peers. Default 4096.
# May need lowering on broken networks with fragmentation/MTU issues,
# particularly if validating DNSSEC.
#
#edns-buffer-size: 1480
# Use TCP for "forward-zone" requests. Useful if you are making
# DNS requests over an SSH port forwarding.
#
#tcp-upstream: yes
# DNS64 options, synthesizes AAAA records for hosts that don't have
# them. For use with NAT64 (PF "af-to").
#
#module-config: "dns64 validator iterator"
#dns64-prefix: 64:ff9b::/96 # well-known prefix (default)
#dns64-synthall: no
remote-control:
control-enable: yes
control-use-cert: no
control-interface: /var/run/unbound.sock
forward-zone:
name: "."
forward-addr: 192.168.1.1 ## Forward the DNS Requests to my local real router
forward-addr: 74.82.42.42 # he.net
forward-addr: 2001:470:20::2 # he.net v6
forward-addr: 8.8.8.8 # google.com
forward-addr: 2001:4860:4860::8888 # google.com v6
forward-addr: 208.67.222.222 # opendns.com
forward-first: yes # try direct if forwarder failsNow DNS requests from the 192.168.30.0/24 subnet will be answered by the DNS server at 192.168.30.1.
vmd - What goes into /etc/vm.conf
Now my next task was to create the correct vm configuration, so that each vm would be connected to a local virtual switch managing the virtual network.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | |
So on lines 4, 10 and 16 I configured the specific VMs and on line 22 I configured the virtual switch that the VMs network interfaces are attached to.
Then I enabled vmd:
doas rcctl enable vmdAnd restarted the server:
doas rebootConfiguring relayd on the host-vm
The last task was the configuration of relayd on the host-vm.
For this I logged into the host-vm, which should have been started automatically now at reboot:
doas vmctl console host-vmThen I configured relayd to redirect requests to my subdomains:
# /etc/relayd.conf
## MACROS
host_vm=192.168.30.2
foo_vm=192.168.30.3
bar_vm=192.168.30.4
## TABLES
table <foo_service> { $foo_vm }
table <bar_service> { $bar_vm }
## PROTOCOLS
# define a new http protocol called "reverseproxy"
http protocol "reverseproxy" {
# match a request that has a Host value with "foo.hermes-technology.de"
# and forward that to the foo_service
# "quick" means that it will be handled as the last rule if it matches... no other will be processed
match request quick header "Host" value "foo.hermes-technology.de" \
forward to <foo_service>
match request quick header "Host" value "www.hermes-technology.de" \
forward to <foo_service>
match request quick header "Host" value "bar.hermes-technology.de" \
forward to <bar_service>
}
## RELAYS
# create a new relay called "proxy"
# relays are for layer 7 (e.g. http)
relay "proxy" {
# the relay should listen on the host ip on port 80
listen on $host_vm port 80
# use the protocol "reverseproxy" we defined above
protocol "reverseproxy"
# forwarding is enabled for the specified services
forward to <foo_service> port 80
forward to <bar_service> port 80
}The last task was to enable and start relayd:
doas rcctl enable relayd
doas rcctl start relaydThis does work now
Because I bought the domain hermes-technology.de and registered blog.hermes-technology.de as CNAME entries, the following does work:
- Enter blog.hermes-technology.de anywhere into a webbrowser.
- The request lands at my main server’s ip (hermes-technology.de).
- PF redirects any http requests to the host-vm on 192.168.30.2.
relaydon the host-vm redirects the “blog” subdomain request to the ip 192.168.30.3 of the blog-vm.- The blog-vm and its httpd-service sends the response to its gateway 192.168.30.1
- The response on the vether0 gets a Network Address Translation (NAT) by
pfto the router. - The router sends the response to the client.
Outlook
In the next post I will explain how I set up a proper ssh connection to my server with a public key. Which is much safer than using a password. Also I will explain how to configure ssh-tunneling so that you login to your virtual servers from the workstation where you’re maintaining your server from. At last I will explain how I configured pf to do an ssh protection like fail2ban does.