Bluetrait

For some reason I decided to download Opera 8 beta today. I'm liking it. It has some nice styling, although maybe a bit out there. It seems to render most stuff very well, and maybe even better than Firefox.

The only problem I've had so far is that my admin panel which uses fckeditor doesn't work. I think I might need to search for a new wysiwyg editor, because it doesn't work in safari either (my browser of choice on the mac). Anyone know of a good one?

Anyway back on topic; Opera is cool. Go download and try it.

I received an email earlier today from a reader who has a better way to manage and run backups. Here is a slightly edited version of the email.

I tackled the same problem at my own company and I think my solutions a little nicer than yours so I thought I'd share it with you :)

Ok so this is how I do backup for the local server at my company:

General Steps:

1. Take a snapshot of everything I want to backup (every night)
2. Archive and compress that snapshot (once a week)
3. Upload that snapshot to a remote location (once a week)
4. Leave the compressed archive lying around in a samba share so I can  burn it to DVD from my desktop machine whenever I get around to it.

The process is very similar to yours, the main difference between our implementations is that I use rsync to speed everything up and my scripts log their actions. I also wrote a couple of tests scripts to automate testing and prove everything works. The whole thing is a little hacky and I'd like to make the format of the log files look nicer but it does work :) It's also a really really fast way to do backups. The technical details are:

1. Use rsync to copy your data from it's current folder to the snapshot folder. This is really fast as rsync only copies the differences not the entire tree. Logs on my server shows it usually takes about 3 mins. I call this script 'snapshot.sh'
2. Use tar to create an archive.
3. Gzip the archive with a version of gzip that has the '--rsyncable' patch applied[1]. I call the script that does 2 and 3 'archive.sh'
4. Upload the compressed archive to the remote server using rsync. I call this script 'upload_archive.sh' (I'm really creative)

I wrap these scripts in two scripts called 'everyday.sh' and 'weekly.sh' depending on what needs to happen on each of those days. There also a little timer script that writes the time each command takes.

For connecting to the remote server I use ssh with keys so I don't need to worry about storing passwords in any of the scripts.

The really cool part of this whole thing is in step 4 where rsync will only copy the different bits in the compressed archive. I don't know exactly how this brilliant piece of magic works[2] but it's save me and my clients quite a bit of time and money.

[1] The patch is already in redhat's gzip so hopefully it's in the default freebsd one. Nothing in the man page about it yet though. [2] I first read about it here where the guy provides some explanation of what its doing: http://lists.ubuntu.com/archives/ubuntu-devel/2005-January/003327.html

--

Myles Byrne Web Architect

I have setup an IPv6 tunnel through AARNET

E:\>tracert vee-six.telstra.net

Tracing route to vee-six.telstra.net [2001:360::3] over a maximum of 30 hops:

1 22 ms 19 ms 18 ms 2001:388:f000::246
2 39 ms 53 ms 40 ms gigether0-2-0.bb1.a.syd.aarnet.net.au [2001:388:1:5001:204:e0ff:fe00:1022]
3 19 ms 17 ms 19 ms gigabitethernet2.7304.syd.aarnet.net.au [2001:388:1:5006:20f:23ff:fea3:ef02]
4 26 ms 26 ms 39 ms 2001:388:200:4::2
5 26 ms 24 ms 26 ms vee-six.telstra.net [2001:360::3]

Trace complete.

I'll have a play around with it some more later. :)

Earlier this morning I made the jump to m0n0wall from Smoothwall. Both these programs are small operating systems that provide services such as NAT/Firewalls etc.

There were a few reasons I decided to move. I’ve been talking about it for a while and finally got the time to do it over this long weekend.

Smoothwall had done pretty well. It was hacked at a bit to provide better uptime but it was really starting to get dodgy.

Smoothwall has a built in intrusion detection system called Snort. It could be useful. It logs all possible intrusions, more than a basic firewall that just blocks stuff. But it had gotten to the point where it was filling up the hard drive EACH DAY with log files. So the system would start having issues as there was no drive space to do anything. The basic routing was fine because that was loaded into Ram, but everything else stopped. Smoothwall just decided to turn services off when it got to this point. Great.

There was no way to clean out logs via the webadmin and I was getting sick of SSH’ing into it all the time.

So time to give m0n0wall a run. The install process was very easy. Copy the .img file to the Smoothwall box via FTP over SSH. And run the following command:

There went Smoothwall. The funny thing was the box was still routing. It wasn’t until I rebooted and m0n0wall started did I loose internet connectivity.

I was greeted with a few options. So I configure the LAN interface. After that was setup I had access to the webadmin. Which is very nice. So from then on everything was done via a web interface.

I setup PPPoE and save the setting. Mmmm did that work? Tried whirlpool and it was going! Cool. The basic box was up in 5 minutes. This includes killing the old OS.

I then decided to setup incoming connections. I went to the firewall rules and setup a few. It didn’t work. The problem was that m0n0wall has two areas to handle connections. The firewall and the NAT section. Both can have different settings, which is cool I just didn’t realize that I needed to setup both. Anyway I did that and incoming connections were allowed through.

All I can say is that I am very happy with m0n0wall. There are so many options. The thing is very flexible.
It isn’t as easy to use as Smoothwall but it feels more like a real router.

The following is from the m0n0wall site:

m0n0wall is probably the first UNIX system that has its boot-time configuration done with PHP, rather than the usual shell scripts, and that has the entire system configuration stored in XML format.

Very cool. Here are a few more screen shots:

On another note, I am slowing syncing the web servers time so that it isn't 10 minutes wrong. Since I have the security level so high I can only change the time 1 second at a time. And I don't feel like rebooting it. ;) Give it a few more hours and it should be right.

So right now the time is 8:20am and not 9:20am EST aka GMT10+. It is also Easter Sunday. And my servers clock is about 8-10minutes fast. 

UTF-8 is an encoding method that allows you to mix languages and scripts within a single document without needing to switch between different character sets.
Everything is moving away from the old standard ISO-8859-1 to UTF-8.

UTF-8 is great because it allows you to use a wider range of characters. For example Greek:

Τη γλώσσα μου έδωσαν ελληνική
το σπίτι φτωχικό στις αμμουδιές του Ομήρου.
Μονάχη έγνοια η γλώσσα μου στις αμμουδιές του Ομήρου.
από το Άξιον Εστί
του Οδυσσέα Ελύτη

But sometimes there can be problems in the transition from ISO-8859-1 to UTF-8.

There are a whole range of examples:

• Inputting UTF-8 data into a Mysql 4.0 database while the html form is set to use ISO-8859-1.
• Reading UTF-8 data out of a database onto a ISO-8859-1 html page.
• Sending an email where the inputted data is using UTF-8 but is transmitted as ISO-8859-1.

The list could go on.

If you're planning on using UTF-8 (which you should) there is a simple way to set your website to this character set.
Using php:
<?php header('Content-Type: text/html; charset=UTF-8'); ?>

Now I could go into the Content-Type and using application/xhtml+xml if you're using XHTML 1.1 or higher. But I won't because IE is crap and doesn't support it.
I might talk about it later, because it isn't really related to the character set.

One thing to note. Mysql 4.0 doesn't *really* support UTF-8. So that Greek mightn't work. That is one of the reasons I am looking to moving to Mysql 4.1 (and also for sub-query support).

I’ve got a Generation 4 20gb iPod that I’ve had since late last year. It is a pretty cool device and is built well. But I always had the problem of it crashing on me once or twice a week. Worse than my Windows box!

Now I also used to charge it via Firewire on either the PC or the Mac laptop. But anyway a few months ago I decided to setup my dock for it properly. I didn’t want to have to turn the computer on to listen to music. So I decided to play everything through my iPod.

The dock is connect via the lineout to my amplifier and also connected to the 240v AC to Firewire power converter.

Since I’ve been charging my iPod through the mains power it hasn’t crashed since. It’s the same Firewire cable as I used to charge with on the computer. I also charged the thing on two different computers.

So for some very odd reason, the iPod crashes every now and then if you don’t charge it at least once a week through mains power.

So now my iPod is as stable as my Mac. Cool.

I've just been looking at my inbox and it is the biggest mess. I have about 24,000 emails dating back to early 2002 sorted into about 20-30 folders, but it’s just a nightmare! I don't properly sort my email. Most of it goes straight into my inbox and doesn't leave.

I have a few folders for certain people depending on how much email I get from them; I have a few for mailing lists and some folders for daily server reports. But it still isn't good enough. It not the easiest place to find things.

The only program that has a half decent search function (like I can search the body of emails) is Mail.app on the Mac. But I don't use that program because of the lack of Exchange support. So finding stuff isn't always easy.

I don't know, anyone have some ideas? How do you sort your emails?

After a long time in development we are finally happy with the finished product.

Our corporate website is now online!

 
base10solutions.com.au

As most of you know I changed over to FreeBSD for my web server a few days ago. Before that I was running Windows 2003. To backup my data I was simply using the built in "NT backup" in Windows with a scheduled task to run once a day. This backup would copy all the important data and replace yesterday’s backup (done nightly). Then once a month I would copy that backup to CD.

Now that worked well for about a year but there were two main problems

1. The backup would overwrite the previous backup. So this protected me against hardware failure but not an attack where someone deleted all the files.

2. The file that NT backup was making started to get far too big. Just before I moved to FreeBSD I was backing up just under 3 GB every night.

The other problem is simply that FreeBSD doesn't have "NT backup" :)

So I decided to rethink the way I do backups.

There are three main folders that store users data on my FreeBSD server (plus a few more for configuration, but I am just looking at backing up user data today)

/usr/home/
/var/db/mysql
/var/mail

The first is the user’s home directory. This contains their main website, log files and maybe a few other things.

The second directory is all the mysql databases. If a user have a mysql database it is stored here.

The last directory is mail that the user hasn’t moved to there home directory

Now instead of dumping all these directories to a single backup file nightly I decided to split the backup procedure.

I now only backup mail and mysql nightly, while I backup the home directory weekly.

The backup files are stored in /usr/backup and are also uploaded via ftp to my other server.

So let’s get down to setting this up.

1. First we’ll create the .tar.gz backup file
2. Then we’ll automatically upload it via ftp
3. Finally we’ll setup crontab to do it for us.

So to create the home directory backup we use this command:

tar cvvpzf /usr/backup/home-`date +'%Y%m%d'`.tar.gz /usr/home

(we will also be using:
tar cvvpzf /usr/backup/mysql-`date +'%Y%m%d'`.tar.gz /var/db/mysql
tar cvvpzf /usr/backup/mail-`date +'%Y%m%d'`.tar.gz /var/mail)

Let me explain what is does.

Tar: which really stands for tape archiver, but it will also write normal files.

cvvpzf = create file, verbose, verbose, preserve-permissions, gzip it, write to file

/usr/backup/home-`date +'%Y%m%d'`.tar.gz

This shows where to save the file to. Notice that we add the date to the end.

An example is:
home-20050319.tar.gz

/usr/home

Shows what to backup.

Okay so why don’t you run that script and make sure it works. You’ll need to have permissions to access each folder (root does). Or if you’re just backing up your personal home directory that will work too.

On a side note: You might like to add --ignore-failed-read to that command. This means the backup continues even if it cannot read a file.

Now we want to automatically upload this to our offsite ftp server. First we need to create a file called .netrc in our home directory.

.netrc is a script that is run when ever you connect to an ftp server.

*make sure the permissions for this are chmod: 600, otherwise it will not work*

So an example of .netrc

You need a blank line at the end of the file.

at the top we have machine ftp.example.com. This means the script will run when you connect to that server. You can also have an IP address there.

Login is your ftp login name, and password is the password for it.

Now we have two scripts here. A daily one and a weekly one. I’ll show how you call these later.

First you need: macdef. This is the command to start the script. Straight after this we have a command name “daily”.

Lcd /usr/backup (lcd stands for local change directory, we have already connected to the ftp server so cd will change the directory on the ftp connection)

put mysql-$1.tar.gz
put mail-$1.tar.gz

This will upload our backup files. Notice the $1? We’ll pass the date of the file to that later. i.e mysql-20050319.tar.gz

Okay once that is setup, save and close it. Remember the permissions for this file.

Once this is done let us setup a script that does it all for us. Create daily.sh and give it execute permissions.

First we have the standard #!/bin/sh

Then we have our tar commands to make the archive (note we’re not doing the home directory in this one)

Then we have: echo "\$ daily `date +'%Y%m%d'`" | ftp ftp.example.com

This echoes out $ daily ‘some date here’ to the ftp server at ftp.example.com.

The .netrc file picks up this command. It will run the daily script. And $1 is assigned the correct date.

When .netrc runs it will upload the correct mysql and mail backup. 

Now create a weekly.sh and put in:

So we have our two scripts. daily.sh and weekly.sh

Go can run ./daily.sh and watched it gzip the files and then upload them. If there are any errors, go back and look over your scripts.

Now the last thing to do is setup crontab to make this happen daily and weekly.

Lets create a file called crontab.

touch crontab.

Edit crontab

Now paste this in

Make sure you edit path-to-file to show where your script is. If it is in your home folder then. ~/daily.sh will do.

The first command runs daily at 3.01am

The second command runs on the 5th day of the week, at 4:15am

Save and close this file.

Now type:

crontab name-of-crontab-file (i.e crontab)

This copies that information into your cron setup. Now you should have two backups, one daily and one weekly.

That should be it! I might post a cut down version for people who want to backup their information off a server that they have shell access to.

A received an email that has some good tips on this