View unanswered posts | View active topics It is currently Thu Nov 27, 2014 11:56 pm



Reply to topic  [ 11 posts ] 
A guide to managing system load with ZoneMinder 
Author Message

Joined: Wed Jan 11, 2006 1:19 pm
Posts: 442
Post A guide to managing system load with ZoneMinder
A guide to managing system load with ZoneMinder
(Written with IP Cameras in mind)


Zoneminder is a superb application in every way, but it does a job that needs a lot of horsepower especially when using multiple IP cameras. IP Cams require an extra level of processing to analogue cards as the jpg or mjpeg images need to be decoded before analysing. This needs grunt. If you have lots of cameras, you need lots of grunt.


Why do ZM need so much grunt?
Think what Zoneminder is actually doing. In modect mode ZM is:
1. Fetching a jpeg from the camera. (Either in single part or multipart stream)
2. Decoding the jpeg image.
3. Comparing the zoned selections to the previous image or images and applying rules.
4. If in alarm state, writing that image to the disk and updating the mysql database.

If you're capturing at five frames per second, the above is repeated five times every second, multiplied by the number of cameras. Decoding the images is what takes the real power from the processor and this is the main reason why analogue cameras which present an image ready-decoded in memory take less work.


How do I know if my computer is overloaded?
If your CPU is running at 100% all the time, it's probably overloaded (or running at exact optimisation). If the load is consistently high (over 10.0 for a single processor) then Bad Things happen - like lost frames, unrecorded events etc. Occasional peaks are fine, normal and nothing to worry about.

Zoneminder runs on Linux, Linux measures system load using "load", which is complicated but gives a rough guide on what the computer is doing at any given time. Zoneminder shows Load on the main page (top right) as well as disk space. Typing "uptime" on the command line will give a similar guide, but with three figures to give a fuller measure of what's happening over a period of time but for the best guide to see what's happening, install "htop" - which gives easy to read graphs for load, memory and cpu usage.

A load of 1.0 means the processor has "just enough to do right now". Also worth noting that a load of 4.0 means exactly the same for a quad processor machine - each number equals a single processor's workload. A very high load can be fine on a computer that has a stacked workload - such as a machine sending out bulk emails, or working its way through a knotty problem; it'll just keep churning away until it's done. However - Zoneminder needs to process information in real time so it can't afford to stack its jobs, it needs to deal with them right away.

For a better and full explanation of Load: http://en.wikipedia.org/wiki/Load_%28computing%29


My load is too high, how can I reduce it?
Zoneminder is /very/ tweakable and it's possible to tune it to compromise. The following are good things to try, in no particular order;

Change the jpeg libraries. In most distributions Linux uses standard jpeg libraries which although fine for most things, don't use the MMX functions in nearly all modern processors. Check whether your cpu supports mmx by running "cpuid |grep MMX" which should give you a line or two along the lines of "MMX instructions". If so, give the libs a try. Most people report their load halves simply by using these libs. http://www.zoneminder.com/forums/viewtopic.php?t=6419 gives more info. Nobody's posted there to say it broke their system... Yet.

If your camera allows you to change image size, think whether you can get away with smaller images. Smaller pics = less load. 320x240 is usually ok for close-up corridor shots.

Go Black and White. Colour pictures use twice to three times the CPU, memory and diskspace but give little benefit to identification.

Reduce frames per second. Halve the fps, halve the workload. If your camera supports fps throttling (Axis do), try that - saves ZM having to drop frames from a stream. 2-5 fps seems to be widely used.

Experiment with using jpeg instead of mjpeg. Some users have reported it gives better performance, but YMMV.

Tweak the zones. Keep them as small and as few as possible. Stick to one zone unless you really need more.

Schedule. If you are running a linux system at near capacity, you'll need to think carefully about things like backups and scheduled tasks. updatedb - the process which maintains a file database so that 'locate' works quickly, is normally scheduled to run once a day and if on a busy system can create a heavy increase on the load. The same is true for scheduled backups, especially those which compress the files. Re-schedule these tasks to a time when the cpu is less likely to be busy, if possible - and also use the "nice" command to reduce their priority. (crontab and /etc/cron.daily/ are good places to start)


More expensive options:

Increase RAM. If your system is having to use disk swap it will HUGELY impact performance in all areas. Again, htop is a good monitor - but first you need to understand that because Linux is using all the memory, it doesn't mean it needs it all - linux handles ram very differently to Windows/DOS and caches stuff. htop will show cached ram as a different colour in the memory graph. Also check that you're actually using a high memory capable kernel - many kernels don't enable high memory by default.

Faster CPU. Simple but effective. Zoneminder also works very well with multiple processor systems out of the box (if SMP is enabled in your kernel). The load of different cameras is spread across the processors.


What about disks and bandwidth?

In most modern pc-based servers, disk I/O is more than adequate for the speeds involved in capturing from multiple cameras in most scenarios.

A typical 100mbit LAN will cope with most setups easily. If you're feeding from cameras over smaller or internet links, obviously fps will be much lower.

Disk and Bandwidth calculators are referenced on the Zoneminder wiki here: http://www.zoneminder.com/wiki/index.ph ... _for_ZM.3F


Fri Oct 27, 2006 9:23 pm
Profile
User avatar

Joined: Fri Mar 05, 2004 5:47 pm
Posts: 5218
Location: /USA/Washington/Seattle
Post 
Thanks for this Flash_ :D
And Thank you for taking the time to post it.

Cheers,
Corey


Sat Oct 28, 2006 12:11 am
Profile WWW

Joined: Wed Jun 08, 2005 9:07 pm
Posts: 5110
Location: Midlands UK
Post 
yes very useful. perhaps you would like it on the wiki!

Great advice there

_________________
James Wilson

Disclaimer: The above is pure theory and may work on a good day with the wind behind it. etc etc.
http://www.securitywarehouse.co.uk


Sat Oct 28, 2006 12:36 am
Profile WWW

Joined: Wed Jan 11, 2006 1:19 pm
Posts: 442
Post 
Thanks and thanks. And it's wiki'd already - my first wiki!


Sat Oct 28, 2006 8:26 am
Profile

Joined: Mon Mar 17, 2008 10:27 pm
Posts: 32
Post Disk IO
Quote:
In most modern pc-based servers, disk I/O is more than adequate for the speeds involved in capturing from multiple cameras in most scenarios.


Disk IO has been my main bottleneck. In the test server I have running, anytime I go to delete a number of events, the deletion causes the system load to increase tremendously. During normal recording conditions the RAM is underutilized, CPU is underutilized, and the recording works smoothly. So far so good. But when anywhere from 5 to 50 GB worth of events are deleted, the system load jumps from say .5 up to 4. At that point Apache drops requests and I can no longer view the main ZM console window. The Apache problem is likely due to this being a test system and using old hardware.

The disk IO performance when deleting and recording at the same time is causing a significant spike in system load. This isn't a surprise, but I'm wondering if other people have problems like this as well? It's to be expected when you have all your events on a single IDE drive. When I move this to production, the production server will most likely use RAID 0 (or 5) or JBOD which will be of great help.


Mon Mar 17, 2008 10:49 pm
Profile

Joined: Wed Jan 30, 2008 6:53 pm
Posts: 519
Location: St. Louis, MO, USA
Post Disk speed
I'm pretty new at the ZoneMinder game, but in my non-scientific experience so far, RAID 0 is worlds better than one drive. My initial tests set to record all on one drive took ages to delete old events, and pegged the CPU. Reformatted the system (2 320gb drives) into 1 RAID 0 640GB, and it's running like a champ, deleting piles of dirs (more than rm -r can usually handle in one swipe) fairly instantly.

May or may not have something to do with it now being Reiserfs instead of ext3.


Tue Mar 18, 2008 1:23 am
Profile

Joined: Mon Mar 17, 2008 10:27 pm
Posts: 32
Post 
Thanks for the update Coke. Like I said, the performance problem is to be expected given my hardware setup. It has nothing to do with ZoneMinder. And as for deleting all those files, after a certain point, you can't even use "rm -rf" anymore. You have to use "find" and pipe the output to "rm". Fortunately, ZM takes care of this all.

Either way, I'd never use a single drive in a production system unless it only had a couple of cameras.

In addition to JBOD for disk spanning, I'm considering switching from Ubuntu to Fedora for the built in LVM support. With LVM you can take multiple disks of different size and group them together into 1 logical drive. This would make life a lot easier for those of us with handfuls of spare 120 and 160 gig drives.


Tue Mar 18, 2008 7:11 am
Profile
Site Admin
User avatar

Joined: Wed Jul 09, 2003 3:07 pm
Posts: 5221
Location: Bristol, UK
Post 
If you use the original flat filesystem then deleting files from directories with tens of thousands of events in will be slow. The deep filesystem available in 1.23.x is much quicker as no directory has more than a handful of other directories in.

_________________
Phil


Tue Mar 18, 2008 9:55 pm
Profile ICQ YIM WWW

Joined: Mon Mar 17, 2008 10:27 pm
Posts: 32
Post 
zoneminder wrote:
If you use the original flat filesystem then deleting files from directories with tens of thousands of events in will be slow. The deep filesystem available in 1.23.x is much quicker as no directory has more than a handful of other directories in.


When I first read about that feature I was very excited. However, the documentation states that this feature is not ready for production. My test server is running ZM 1.23.1 and in the configuration window, the help for that feature states "Be warned: deep storage should be considered in beta for now, if you value your data do not select it.". Maybe 1.23.2 has fixed this?

If it is in fact a stable feature, then I'll switch to that and compare the performance of a single IDE drive using deep directories VS RAID 0 drives using the flat file structure.

Either way, RAID 0 should eliminate any serious disk IO issues.


Thu Mar 20, 2008 3:58 am
Profile

Joined: Sat Mar 31, 2007 10:18 pm
Posts: 1059
Location: Houston, TX
Post 
Consider the filesystem as well. The Live CDs for ZM and most other distributions use ext3 by default. While robust, ext3 is NOT a performance filesystem. I have been thinking a while about benchmarking a few filesystems with ZM to see the difference. Most of them have better delete performance than ext3.


Sun Mar 30, 2008 7:35 pm
Profile

Joined: Tue Jul 03, 2007 7:46 am
Posts: 4
Post Large scale deployment
I have been using zoneminder for a few months on test basis, and it works perfectly for my requirements, including motion detection, recording etc. Currently, we have a requirement to implement large scale surveillance, covering around 70 IP cameras, each at 640x480 resolution (some PTZ, some fixed). Is it advisable to use zm for this situation? I feel everything ok, but the shared memory required may be a big issue.......

Has anyone tested zm on 50+ ip cameras? Or can any one give me tips on how to go about it?


Thu Apr 24, 2008 4:29 am
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 11 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group