Setting up a home server

[mfenn]

Personal preference I guess. If it was mine, at home, i'd want to be able to run more then just ZFS. Performance wouldn't take that big of a hit.

Web portals are nice, but command line is where it's at!

I think that you're under the impression that the OP is running FreeNAS. He's not, he's running Ubuntu Server and ZoL. Linux is a perfectly capable hypervisor with KVM.

That's not really true - if the hardware is enterprisey enough, it'll support PCI passthrough - just drop in a SATA card, pass it through the the VM, and it'll address the drives directly as though it was a hardware box, then create iSCSI LUNs on the ZFS server and pass them back to ESX as datastores.

Now, that said, it's not recommended since you can end up with chicken/egg problems if you put the SAN server datastore on a LUN it manages.

Yes, you could do some "cool virtualization tricks" to make it work. But just because you can doesn't meant that it's a good idea. Which is why I didn't mention it at all. In other words, "cool virtualization tricks" are a really bad idea for a system that is primarily supposed to serve storage with no fuss.

 

[dave_the_nerd]

Yes, you could do some "cool virtualization tricks" to make it work. But just because you can doesn't meant that it's a good idea. Which is why I didn't mention it at all. In other words, "cool virtualization tricks" are a really bad idea for a system that is primarily supposed to serve storage with no fuss.

How very old-fashioned of you. It's neither cool nor tricky - you just have to know where your vmdks are and in what order to turn things on.

 

[Red Squirrel]

I would personally keep storage as a dedicated machine. Preferably something with dual power supplies. Where do you even store the vmdk for the SAN if the SAN is virtualized? Sounds like a chicken and egg problem.

Idealy hardware raid 1 for the OS then software raid (ZFS, mdraid etc) for the data. Though personally I never bother with hardware raid for the OS, I just put a SSD and figure the odds of it crapping out are probably so slim I don't worry about it. I also usually keep an image of the OS handy so if it does crap out at least I don't have to rebuild the OS from scratch.

 

[dave_the_nerd]

I would personally keep storage as a dedicated machine. Preferably something with dual power supplies.

I thought we were talking about OP's home server/learning lab? If you have budget/space for a single box, you have a single box. Installing ESXi doesn't get you SAN capability (or much in the way of storage management at all), and installing anything else doesn't get you ESXi.

Where do you even store the vmdk for the SAN if the SAN is virtualized? Sounds like a chicken and egg problem.

When you install ESX, it automatically configures unused boot disk as a datastore. Have the SAN OS's VM files on a small datastore on the hypervisor's boot drive.

Most of the SAN controllers I work with store their OS on internal flash. Enterprise-grade SD cards or something, I guess. As far as reliability goes, a datastore on non-redundant flash isn't more or less susceptible to hardware failure than an OS on non-redundant flash.

I suppose it would be better just to use a hardware RAID card.

But it's not like nobody has done this before, at least for non-production stuff.

 

[mfenn]

How very old-fashioned of you. It's neither cool nor tricky - you just have to know where your vmdks are and in what order to turn things on.

I'm not sure if you're joking or not, but that is pretty much the definition of a "too clever and tricky" system.

 

[mfenn]

I suppose it would be better just to use a hardware RAID card.

At which point you're adding expense to a machine which is ostensibly a ZFS storage box which should not need or want hardware RAID.

But it's not like nobody has done this before, at least for non-production stuff.

The way I read the OP is that he's looking for something that works as a storage server first and foremost and is looking at virtualization as a means to an end, and not trying to build a home VM lab.

 

[EtherealDragon]

I want to start off by thanking everyone for the assistance through this project, it has been GREAT!!!


Work and family life has been busy, so I haven't had much time to tinker on the server. Things have been better this week, so yesterday I took the time to format the OS drive on the server, and got it up and working the way I wanted in about 2 hours. I have 3 live virtual machines running, and so far all is well. Plex works, and I was able to get it access to the ZFS pool via NFS for the media. I sent all night getting my Standard Def movies and music copied to the server, currently working on the High Def movies, will take a day or two probably. The Minecraft server works, I was able to connect to it from a local laptop, haven't tried over the internet yet, but it should work once I get the port forwarding set up. I also have a test server in there so that any changes I want to make I can try in there before possibly breaking production.

I still have more to do, but I am just happy this is up and working. I have a backups folder in the zpool which all of the virtual guests have access to. Right now they all have access to the full folder, but I will eventually make it so that each guest has its own folder and can only save data to that folder, unable to access anything else. I also want to figure out how to get Samba to require logging in from a Windows client, so that everyone doesn't have wide-open access to the shares. I tried before, could SEE the shares, but couldn't access them, or mount them as network drives from the windows hosts. This backups folder will be for the virtual guests, our laptops, and my desktop. I would EVENTUALLY like to figure out how to get phones to back up to it as well (create and app, hit sync or something, and all of the photos in the albums get copied to the back up folder for instance...) As noted though, there is more reading and testing to do. I am not sure what exactly on each of the virtual guests would need to be back up, and not sure how to accomplish that. Eventually down the road, that backups folder would then automatically back up to a second network device, a D-Link DNS-323 with two 3TB drives in it. I have it sitting here, just not plugged in or connected to the network at the moment. I migrated its duties to my desktop which has 3x 2TB drives in it for just about the same capacity as the network drive.


If anyone else is looking for help in getting things up and running, here are the commands I used to get this working...

[EDIT]
Plenty more room to stretch my legs, AND, 6 empty drives bays too!!

NAME              USED  AVAIL  REFER  MOUNTPOINT
Storage          1.97T  12.3T   192K  /Storage
Storage/Backups   192K  12.3T   192K  /mnt/Storage/Backups
Storage/Media    1.97T  12.3T  1.97T  /mnt/Storage/Media

[/EDIT]

1. Install Ubuntu Server, with only open-ssh
2. Root Password
3. Install other packages
4. Updates
5. network
6. ZFS 
7. KVM



=====================================================================

1.
Install Ubunty server with only open ssh package
verify device IP, connect with putty from network host


=====================================================================

2.
Root Password

sudo passwd root


=====================================================================

3.
Install other packages


sudo add-apt-repository ppa:zfs-native/stable;
sudo apt-get update
sudo apt-get install bridge-utils htop samba qemu-kvm libvirt-bin ubuntu-vm-builder stress nfs-kernel-server nfs-common ubuntu-zfs

 	
=====================================================================

4.
Network:


sudo vi etc/network/interfaces

"
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

# The secondary network interface
auto eth1
iface eth1 inet manual

# The bridge network interface
auto br0
 iface br0 inet static
  address 172.16.5.25
  netmask 255.255.255.0
  network 172.16.5.0
  gateway 172.16.5.1
  dns-nameservers 8.8.8.8 8.8.4.4
  bridge_ports eth0 eth1
  bridge_stp off
  bridge_fd 0
  bridge_maxwait 0
"
sudo reboot now


=====================================================================

5. 
Updates

sudo apt-get update
sudo apt-get upgrade


=====================================================================

6.
ZFS and shares


sudo zpool create Storage raidz2 -f /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1304884 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1344552 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1344610 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1324763 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1380342 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4EFUL6XA9 log mirror ata-TOSHIBA_THNSNH256GBST_83IS10JLTE8Y-part1 ata-TOSHIBA_THNSNH256GBST_83JS106NTE8Y-part1 cache ata-TOSHIBA_THNSNH256GBST_83IS10JLTE8Y-part2 ata-TOSHIBA_THNSNH256GBST_83JS106NTE8Y-part2
sudo zpool status
sudo zfs create -o mountpoint=/mnt/Storage/Backups Storage/Backups
sudo zfs create -o mountpoint=/mnt/Storage/Media Storage/Media
sudo chown nobody:nogroup /mnt/Storage/Media/
sudo chown nobody:nogroup /mnt/Storage/Backups/
sudo zfs list


Go to root mode, issue - fixes reboot zpool loss issue
echo manual >/etc/init/zpool-import.override


[Only if zpool disappears]
zpool import -d /dev/disk/by-id




sudo vi /etc/samba/smb.conf

"
[Media]
path = /mnt/Storage/Media
available = yes
read only = no
browseable = yes
public = yes
writable = yes
create mask = 0755


[Backup]
path = /mnt/Storage/Backups
available = yes
read only = no
browseable = yes
public = yes
writable = yes
create mask = 0755
"



sudo vi /etc/exports
"
/mnt/Storage/Media 172.16.5.26(rw,sync,no_subtree_check)
/mnt/Storage/Backups 172.16.5.10(rw,sync,no_subtree_check) 172.16.5.26(rw,sync,no_subtree_check) 172.16.5.27(rw,sync,no_subtree_check)
"
sudo exportfs -ra
sudo service nfs-kernel-server start
showmount -e

=====================================================================

7.
KVM: 


Plex

sudo ubuntu-vm-builder kvm trusty \
 --domain * \
 --dest VG-PlexMediaServer \
 --hostname VG-PlexMediaServer \
 --arch amd64 \
 --mem 8192 \
 --cpus 4 \
 --user *** \
 --pass *** \
 --bridge br0 \
 --ip 172.16.5.26 \
 --mask 255.255.255.0 \
 --net 172.16.5.0 \
 --bcast 172.16.5.255 \
 --gw 172.16.5.1 \
 --dns 8.8.8.8 8.8.4.4 \
 --components main,universe \
 --addpkg acpid \
 --addpkg openssh-server \
 --addpkg nfs-common \
 --addpkg linux-image-generic \
 --addpkg htop \
 --addpkg stress \
 --libvirt qemu:///system ;

virsh
autostart VG-PlexMediaServer
start VG-PlexMediaServer


ON GUEST
  wget downloads.plex.tv/plex-media-server/0.9.12.1.1079-b655370/plexmediaserver_0.9.12.1.1079-b655370_amd64.deb
  sudo dpkg -i plexmediaserver_0.9.12.1.1079-b655370_amd64.deb
  mkdir Backups
  mkdir Media
  sudo vi /etc/fstab

  "
  172.16.5.25:/mnt/Storage/Backups                /home/padmin/Backups nfs  auto         0       0
  172.16.5.25:/mnt/Storage/Media                  /home/padmin/Media   nfs  auto         0       0
  "
  sudo mount -a





----------

Minecraft

sudo ubuntu-vm-builder kvm trusty \
 --domain * \
 --dest VG-MineCraftServer \
 --hostname VG-MineCraftServer \
 --arch amd64 \
 --mem 8192 \
 --cpus 4 \
 --user *** \
 --pass *** \
 --bridge br0 \
 --ip 172.16.5.27 \
 --mask 255.255.255.0 \
 --net 172.16.5.0 \
 --bcast 172.16.5.255 \
 --gw 172.16.5.1 \
 --dns 8.8.8.8 8.8.4.4 \
 --components main,universe \
 --addpkg acpid \
 --addpkg openssh-server \
 --addpkg nfs-common \
 --addpkg linux-image-generic \
 --addpkg htop \
 --addpkg stress \
 --addpkg default-jdk \
 --libvirt qemu:///system ;

virsh
autostart VG-MineCraftServer
start VG-MineCraftServer


ON GUEST
  mkdir Backups
  sudo vi /etc/fstab

  "
  172.16.5.25:/mnt/Storage/Backups                /home/madmin/Backups nfs  auto          0       0
  "
  sudo mount -a
  mkdir MCServers
  cd MCServers
  mkdir CreativeWorld1
  mkdir SurvivalWorld1
  cd SurvivalWorld1
  wget https://s3.amazonaws.com/Minecraft.Download/versions/1.8.4/minecraft_server.1.8.4.jar
  sudo vi eula.txt
  "
  eula=true
  "
  java -Xmx4096M -Xms8192M -jar minecraft_server.1.8.4.jar nogui

----------

sudo ubuntu-vm-builder kvm trusty \
 --domain * \
 --dest VG-TestServer \
 --hostname VG-TestServer \
 --arch amd64 \
 --mem 1024 \
 --cpus 1 \
 --user *** \
 --pass *** \
 --bridge br0 \
 --ip 172.16.5.30 \
 --mask 255.255.255.0 \
 --net 172.16.5.0 \
 --bcast 172.16.5.255 \
 --gw 172.16.5.1 \
 --dns 8.8.8.8 8.8.4.4 \
 --components main,universe \
 --addpkg acpid \
 --addpkg openssh-server \
 --addpkg linux-image-generic \
 --addpkg htop \
 --addpkg stress \
 --libvirt qemu:///system ;

virsh
autostart VG-TestServer
start VG-TestServer

----------

sudo ubuntu-vm-builder kvm trusty \
 --domain *** \
 --dest VG-<NAME> \
 --hostname <NAME> \
 --arch amd64 \
 --mem <MEM M-BYTES> \
 --cpus <#> \
 --user <USER> \
 --pass <PASS> \
 --bridge  br0 \
 --ip 172.16.5.<##> \
 --mask 255.255.255.0 \
 --net 172.16.5.0 \
 --bcast 172.16.5.255 \
 --gw 172.16.5.1 \
 --dns 8.8.8.8 8.8.4.4 \
 --components main,universe \
 --addpkg acpid \
 --addpkg openssh-server \
 --addpkg linux-image-generic \
 --libvirt qemu:///system ;

=====================================================================

 

[mfenn]

Something to be aware of is that your VM images are sitting in /var/lib/libvirt/images, which is not part of your ZFS filesystem. If I were you, I would probably do the following:

1. Create a ZFS filesystem for your VMs
2. Shut down all VMs.
3. mv /var/lib/libvirt/images /zfs/<vmfilesystem>/images; ln -s /zfs/<vmfilesystem>/images /var/lib/libvirt/images
4. Start VMs

Then your VMs are on your big, fast storage and can be protected with ZFS snapshots.

 

[EtherealDragon]

I have been trying to copy the VG systems from the guest itself to the /mnt/Storage/Backups ZFS system, but I am getting permissions errors. Needs more reading to get that sorted out. This would be to save Minecraft world save data frequently to the ZFS backup. I hadn't thought of running the images FROM the ZFS pool though. I could be misunderstanding, but the only place I am seeing the images is inside my home directory, inside of a folder named for each of the VG's. The libvirt directory is empty, even after the VG is shutdown. Would I need to do anything with the monitor files in the /libvirt/qemu directory?


Also, with the size of the VG images, 1.2 Gig, and 4.0 Gig, is this a snapshot containing the entire system with all of its directories and files and everything?? This is what I am thinking, just not sure.

Also, I ordered more memory a few days ago, because htop is showing that 22 of 24 gigs is in use, and I want to give more memory to my Plex and Minecraft servers. I doubled the Plex memory from 4 to 8 gigs, and some buffering issue went away, but I would like to try bumping that up a bit more perhaps.

ziggidy@VH-Ziggidy:~$ sudo zfs list
[sudo] password for ziggidy:
NAME              USED  AVAIL  REFER  MOUNTPOINT
Storage          4.13T  10.1T   192K  /Storage
Storage/Backups   192K  10.1T   192K  /mnt/Storage/Backups
Storage/Media    4.13T  10.1T  4.13T  /mnt/Storage/Media
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ pwd
/home/ziggidy
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ ls -l
total 12
drwxr-xr-x 2 ziggidy ziggidy 4096 May  8 20:06 VG-MineCraftServer
drwxr-xr-x 2 ziggidy ziggidy 4096 May  8 19:53 VG-PlexMediaServer
drwxr-xr-x 2 ziggidy ziggidy 4096 May  8 20:56 VG-TestServer
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ ls -l VG-MineCraftServer
total 1216712
-rw-r--r-- 1 libvirt-qemu kvm 1245970432 May 11 08:17 tmpzz69vK.qcow2
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ ls -l VG-PlexMediaServer
total 3966856
-rw-r--r-- 1 libvirt-qemu kvm 4062117888 May 11 08:17 tmpVQE9OH.qcow2
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ ls -l VG-TestServer
total 781576
-rw-r--r-- 1 libvirt-qemu kvm 800391168 May 11 08:17 tmpDZketi.qcow2
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ sudo ls -l /var/lib/libvirt/images
[sudo] password for ziggidy:
total 0
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ sudo ls -l /var/lib/libvirt/qemu
total 12
srwxr-xr-x 1 libvirt-qemu kvm     0 May  8 21:10 capabilities.monitor.sock
drwxr-xr-x 2 root         root 4096 May  8 19:17 dump
drwxr-xr-x 2 libvirt-qemu kvm  4096 May  8 19:17 save
drwxr-xr-x 2 libvirt-qemu kvm  4096 May  8 19:17 snapshot
srwxr-xr-x 1 libvirt-qemu kvm     0 May 10 11:10 VG-MineCraftServer.monitor
srwxr-xr-x 1 libvirt-qemu kvm     0 May 10 12:02 VG-PlexMediaServer.monitor
srwxr-xr-x 1 libvirt-qemu kvm     0 May  8 21:10 VG-TestServer.monitor
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # destroy VG-TestServer
Domain VG-TestServer destroyed

virsh #
virsh # list --all
 Id    Name                           State
----------------------------------------------------
 5     VG-MineCraftServer             running
 6     VG-PlexMediaServer             running
 -     VG-TestServer                  shut off

virsh #
virsh # exit

ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ sudo ls -l /var/lib/libvirt/images
total 0
ziggidy@VH-Ziggidy:~$

 

[EtherealDragon]

I am not 100% sure if I did this correctly, but I BELIEVE I have my test guest system is now running from the ZFS pool. I need to read up more on hardlink and symbolic links to full grasp it though...

ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ sudo zfs create -o mountpoint=/mnt/Storage/Virtual_Machines Storage/Virtual_Machines
[sudo] password for ziggidy:
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ sudo chown nobody:nogroup /mnt/Storage/Virtual_Machines/
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ mv /home/ziggidy/VG-TestServer /mnt/Storage/Virtual_Machines/VG-TestServer; ln -s /mnt/Storage/Virtual_Machines/VG-TestServer /home/ziggidy/VG-TestServer
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ sudo zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
Storage                   4.13T  10.1T   192K  /Storage
Storage/Backups            192K  10.1T   192K  /mnt/Storage/Backups
Storage/Media             4.13T  10.1T  4.13T  /mnt/Storage/Media
Storage/Virtual_Machines   729M  10.1T   729M  /mnt/Storage/Virtual_Machines
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ cd /mnt/Storage/Virtual_Machines/VG-TestServer
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines/VG-TestServer$ ls -l
total 782468
-rw-r--r-- 1 root    root    800391168 May 11 08:17 tmpDZketi.qcow2
lrwxrwxrwx 1 ziggidy ziggidy        43 May 11 10:04 VG-TestServer -> /mnt/Storage/Virtual_Machines/VG-TestServer
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines/VG-TestServer$
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines/VG-TestServer$
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines/VG-TestServer$ cd ..
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines$ ls -l
total 1
drwxr-xr-x 2 ziggidy ziggidy 4 May 11 10:04 VG-TestServer
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines$
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines$
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines$
ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines$ cd /home/ziggidy
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$ ls -l
total 8
drwxr-xr-x 2 ziggidy ziggidy 4096 May  8 20:06 VG-MineCraftServer
drwxr-xr-x 2 ziggidy ziggidy 4096 May  8 19:53 VG-PlexMediaServer
lrwxrwxrwx 1 ziggidy ziggidy   43 May 11 10:04 VG-TestServer -> /mnt/Storage/Virtual_Machines/VG-TestServer
ziggidy@VH-Ziggidy:~$
ziggidy@VH-Ziggidy:~$

ziggidy@VH-Ziggidy:/mnt/Storage/Virtual_Machines$ virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # start VG-TestServer
Domain VG-TestServer started

virsh # list
 Id    Name                           State
----------------------------------------------------
 5     VG-MineCraftServer             running
 6     VG-PlexMediaServer             running
 7     VG-TestServer                  running

virsh #

 

[mfenn]

I guess ubuntu-vm-builder puts the VM images in whatever (potentially relative) path you specify instead a path relative to the standard libvirt directory. The sysadmin in me cringes at the idea of having symlinks from your home directory into a data directory, but what you have will work. The "virsh" way to copy the VM storage to a new location would be something along the lines of:

virsh blockcopy --domain VG-TestServer vda /mnt/Storage/Virtual_Machines/VG-TestServer --wait --verbose --pivot

Use domblklist to get the block device name ("vda" above).

 

[EtherealDragon]

Well, I found out yesterday that the Plex Media server has totally flatlined on me... It is running, has resources, shows the library, but will NOT play any files at all. I checked the logs, found failures to write temp file errors in it. I tried rebooting the virtual guest, even the host machine completely. I had to shutdown overnight last night.

This morning, I remembered that it was good before moving the virtual machines onto the ZFS pool. It must be some kind of permissions issue, but I haven't the foggiest on where to even start. May do more reading on Tuesday after work.

 

[mfenn]

Well, I found out yesterday that the Plex Media server has totally flatlined on me... It is running, has resources, shows the library, but will NOT play any files at all. I checked the logs, found failures to write temp file errors in it. I tried rebooting the virtual guest, even the host machine completely. I had to shutdown overnight last night.

This morning, I remembered that it was good before moving the virtual machines onto the ZFS pool. It must be some kind of permissions issue, but I haven't the foggiest on where to even start. May do more reading on Tuesday after work.

Moving the image file which contains the VM wouldn't have any effect on permissions within the VM. Can you post the relevant log entries?

 

[EtherealDragon]

This isn't the entire log, but it is mostly all the same error, with random other info that isn't related to the issue. For some reason, the guest isn't able to write anymore data. I tried to manually create a directory, and get an error that the device is full.

padmin@VG-PlexMediaServer:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs$ ls -l
total 2128
-rw-r--r-- 1 plex plex      0 May 18 19:59 Plex DLNA Server.log
-rw-r--r-- 1 plex plex  61440 May 18 16:08 Plex DLNA Server.log.1
-rw-r--r-- 1 plex plex  36864 May 17 20:41 Plex DLNA Server.log.2
-rw-r--r-- 1 plex plex   5294 May 16 21:06 Plex DLNA Server.log.3
-rw-r--r-- 1 plex plex   1574 May  9 02:22 Plex DLNA Server.log.4
-rw-r--r-- 1 plex plex   3238 May  9 01:52 Plex DLNA Server.log.5
-rw-r--r-- 1 plex plex      0 May 17 15:10 Plex DLNA Server Neptune.log
-rw-r--r-- 1 plex plex      0 May 16 03:38 Plex DLNA Server Neptune.old.log
-rw-r--r-- 1 plex plex   4096 May 15 19:02 Plex Media Scanner Analysis.log
-rw-r--r-- 1 plex plex   6840 May 15 19:02 Plex Media Scanner Analysis.log.1
-rw-r--r-- 1 plex plex   4096 May 15 19:02 Plex Media Scanner Analysis.log.2
-rw-r--r-- 1 plex plex      0 May 15 19:02 Plex Media Scanner Analysis.log.3
-rw-r--r-- 1 plex plex   6159 May 10 05:12 Plex Media Scanner Analysis.log.4
-rw-r--r-- 1 plex plex   5505 May 10 05:12 Plex Media Scanner Analysis.log.5
-rw-r--r-- 1 plex plex   1459 May 16 02:56 Plex Media Scanner.log
-rw-r--r-- 1 plex plex   3672 May 16 02:56 Plex Media Scanner.log.1
-rw-r--r-- 1 plex plex 276034 May 16 02:56 Plex Media Scanner.log.2
-rw-r--r-- 1 plex plex 439270 May 16 02:56 Plex Media Scanner.log.3
-rw-r--r-- 1 plex plex  43585 May 16 02:56 Plex Media Scanner.log.4
-rw-r--r-- 1 plex plex  19269 May 16 02:56 Plex Media Scanner.log.5
-rw-r--r-- 1 plex plex 249856 May 18 20:00 Plex Media Server.log
-rw-r--r-- 1 plex plex 410756 May 17 14:10 Plex Media Server.log.1
-rw-r--r-- 1 plex plex 315392 May 16 20:23 Plex Media Server.log.2
-rw-r--r-- 1 plex plex 245760 May 16 20:23 Plex Media Server.log.3
-rw-r--r-- 1 plex plex   4096 May 16 20:23 Plex Media Server.log.4
-rw-r--r-- 1 plex plex      0 May 16 20:23 Plex Media Server.log.5
drwxr-xr-x 2 plex plex   4096 May 17 15:13 PMS Plugin Logs
padmin@VG-PlexMediaServer:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs$
padmin@VG-PlexMediaServer:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs$ more "Plex Media Server.log.1"
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp
May 16, 2015 20:23:50 [0x7eff783fd700] ERROR - Failed to write chunk to file /tmp/plex-transcode-427bqz274l9grpb9-fba6c2a9-7c8d-48df-8737-727b65414dc6/chunk-temp


padmin@VG-PlexMediaServer:
padmin@VG-PlexMediaServer:
padmin@VG-PlexMediaServer:/$ cd tmp
padmin@VG-PlexMediaServer:/tmp$ ls -l
total 0
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$ mkdir test
mkdir: cannot create directory ‘test’: No space left on device
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$ pwd
/tmp
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$
padmin@VG-PlexMediaServer:/tmp$ mkdir test
mkdir: cannot create directory ‘test’: No space left on device
padmin@VG-PlexMediaServer:/tmp$

 

[mfenn]

OK, so that's the filesystem inside the VM that is full, which has nothing really to do with where the image is sitting. Do you have /tmp on a filesystem or as a ramdisk? Do

df -h
cat /proc/mounts

inside the VM and post the output if you're not sure.

 

[EtherealDragon]

Here is the output:

padmin@VG-PlexMediaServer:~$ df -h
Filesystem                                           Size  Used Avail Use% Mounted on
/dev/sda1                                            3.7G  3.5G     0 100% /
none                                                 4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                                 3.8G  4.0K  3.8G   1% /dev
tmpfs                                                780M  296K  780M   1% /run
none                                                 5.0M     0  5.0M   0% /run/lock
none                                                 3.9G  4.0K  3.9G   1% /run/shm
none                                                 100M     0  100M   0% /run/user
172.16.5.25:/mnt/Storage/Media                        15T  4.5T  9.8T  32% /home/padmin/Media
172.16.5.25:/mnt/Storage/Backups/VG-PlexMediaServer  9.8T     0  9.8T   0% /home/padmin/Backups
padmin@VG-PlexMediaServer:~$
padmin@VG-PlexMediaServer:~$
padmin@VG-PlexMediaServer:~$ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=3980360k,nr_inodes=995090,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=798208k,mode=755 0 0
/dev/disk/by-uuid/fbbd0f14-d11d-4c7f-bb76-b0cc5912cc53 / ext4 rw,relatime,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
172.16.5.25:/mnt/Storage/Media /home/padmin/Media nfs4 rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.5.26,local_lock=none,addr=172.16.5.25 0 0
172.16.5.25:/mnt/Storage/Backups/VG-PlexMediaServer /home/padmin/Backups nfs4 rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.5.26,local_lock=none,addr=172.16.5.25 0 0
padmin@VG-PlexMediaServer:~$

 

[XavierMace]

I'm not that great with Linux, but looks to me like this is your problem:

/dev/sda1 3.7G 3.5G 0 100% /

 

[mfenn]

I'm not that great with Linux, but looks to me like this is your problem:

:thumbsup: 100% correct.

The root filesystem is full, which means that nothing else can be written. You can use du to find where the files are. For example:

cd /
du -m --max-depth=1 . | sort -n

The du command will print a sorted list of the biggest subdirectories of / (size in megabytes). Then cd into the biggest one and repeat the du command until you find where all the space is being used. From there it kind of depends on what you find, but I'm betting it'll be some Plex temporary files that you can delete.

To add some more space, you'll either need to add another NFS mount or use virsh to add another disk to the VM. If all else fails, you could rebuild the VM with ubuntu-vm-builder and pass the --rootsize=NNNN option to give it a bigger disk, where NNNN is a disk size in megabytes.

 

[EtherealDragon]

mfenn,

I can't even put into WORDS my appreciation for you assisting me with this build, and my trials along the way. Your help, and that of others has been truly invaluable!!

That said, I was able to kill 2 birds with one stone. I was able to increase the file size of the image, but try as I did, I couldn't figure out how to get that space into the guest itself. So, I killed the guest, and wiped it out totally, deleted the image (all of them actually), removed the sim links... I didn't actually check to see what was using up all of the space, but I believe that it was actually the media meta data (I think thats what it is called), which includes the ratings info, movie summary, movie poster, and all of the same for a music library, consisting of over 450 movies, and not sure yet how may albums.


I am now happy to say that I have only rebuilt ONE of the 3 guests, but it is looking good, and I am about 99% certain that it is properly set up on the ZFS pool, with NO simlinks.

sudo ubuntu-vm-builder kvm trusty \
 --domain ### \
 --dest /mnt/Storage/Virtual_Machines/VG-PlexMediaServer/VG-PlexMediaServer.qcow2 \
 --hostname VG-PlexMediaServer \
 --arch amd64 \
 --mem 8192 \
 --cpus 8 \
 --user padmin \
 --pass ### \
 --bridge br0 \
 --ip 172.16.5.26 \
 --mask 255.255.255.0 \
 --net 172.16.5.0 \
 --bcast 172.16.5.255 \
 --gw 172.16.5.1 \
 --dns 8.8.8.8 8.8.4.4 \
 --components main,universe \
 --addpkg acpid \
 --addpkg openssh-server \
 --addpkg nfs-common \
 --addpkg linux-image-generic \
 --addpkg htop \
 --addpkg stress \
 --rootsize=100000 \
 --libvirt qemu:///system ;

Seeing that the guest has over 85 gigs of usable space, so that is good.

padmin@VG-PlexMediaServer:~$ df -h
Filesystem                                           Size  Used Avail Use% Mounted on
/dev/sda1                                             92G  1.3G   86G   2% /
none                                                 4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                                 3.9G  4.0K  3.9G   1% /dev
tmpfs                                                799M  296K  799M   1% /run
none                                                 5.0M     0  5.0M   0% /run/lock
none                                                 3.9G  4.0K  3.9G   1% /run/shm
none                                                 100M     0  100M   0% /run/user
172.16.5.25:/mnt/Storage/Backups/VG-PlexMediaServer  9.8T     0  9.8T   0% /home/padmin/Backups
172.16.5.25:/mnt/Storage/Media                        15T  4.5T  9.8T  32% /home/padmin/Media
padmin@VG-PlexMediaServer:~$

 

[mfenn]

mfenn,

I can't even put into WORDS my appreciation for you assisting me with this build, and my trials along the way. Your help, and that of others has been truly invaluable!!

That said, I was able to kill 2 birds with one stone. I was able to increase the file size of the image, but try as I did, I couldn't figure out how to get that space into the guest itself. So, I killed the guest, and wiped it out totally, deleted the image (all of them actually), removed the sim links... I didn't actually check to see what was using up all of the space, but I believe that it was actually the media meta data (I think thats what it is called), which includes the ratings info, movie summary, movie poster, and all of the same for a music library, consisting of over 450 movies, and not sure yet how may albums.


I am now happy to say that I have only rebuilt ONE of the 3 guests, but it is looking good, and I am about 99% certain that it is properly set up on the ZFS pool, with NO simlinks.

sudo ubuntu-vm-builder kvm trusty \
 --domain ### \
 --dest /mnt/Storage/Virtual_Machines/VG-PlexMediaServer/VG-PlexMediaServer.qcow2 \
 --hostname VG-PlexMediaServer \
 --arch amd64 \
 --mem 8192 \
 --cpus 8 \
 --user padmin \
 --pass ### \
 --bridge br0 \
 --ip 172.16.5.26 \
 --mask 255.255.255.0 \
 --net 172.16.5.0 \
 --bcast 172.16.5.255 \
 --gw 172.16.5.1 \
 --dns 8.8.8.8 8.8.4.4 \
 --components main,universe \
 --addpkg acpid \
 --addpkg openssh-server \
 --addpkg nfs-common \
 --addpkg linux-image-generic \
 --addpkg htop \
 --addpkg stress \
 --rootsize=100000 \
 --libvirt qemu:///system ;

Seeing that the guest has over 85 gigs of usable space, so that is good.

padmin@VG-PlexMediaServer:~$ df -h
Filesystem                                           Size  Used Avail Use% Mounted on
/dev/sda1                                             92G  1.3G   86G   2% /
none                                                 4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                                 3.9G  4.0K  3.9G   1% /dev
tmpfs                                                799M  296K  799M   1% /run
none                                                 5.0M     0  5.0M   0% /run/lock
none                                                 3.9G  4.0K  3.9G   1% /run/shm
none                                                 100M     0  100M   0% /run/user
172.16.5.25:/mnt/Storage/Backups/VG-PlexMediaServer  9.8T     0  9.8T   0% /home/padmin/Backups
172.16.5.25:/mnt/Storage/Media                        15T  4.5T  9.8T  32% /home/padmin/Media
padmin@VG-PlexMediaServer:~$

:thumbsup: Glad you got it sorted. Redoing the VM makes sense since you hadn't built up a lot of history in it yet.

The reason you weren't able to see the space in the guest after resizing the image is that while the disk was bigger, the partition table in the guest was still referencing the old size. You would have needed to modify that partition to reflect the new size (using a partition editor like fdisk, parted, or gparted) and then run resize2fs to extend the filesystem itself.