The Guide
Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI. There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires exactly version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM’s or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.
Imapsync may run faster and consume less CPU with larger I/O buffers, but no benchmarking of this has been performed. The default buffer size is 4KB; increasing it by one or two orders of magnitude is not uncommon for typical Zimbra-ready configurations. To increase the buffersize to 8MB, add the arguments “–buffersize 8192000” when running imapsync.
For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.
imapsync --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \ --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword
Of course the complete command belongs on one line (signified by the backslashes: \).
A slightly more secure method is to write each password into a separate file, and then use the –passfile{1|2} options intead of the –password{1|2} options:
imapsync --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \ --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile
If your old IMAP server doesen’t support NAMESPACE you have to also add appropriate options for example on iMail 6.0 the following are required:
--sep1 . --prefix1 INBOX
Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.
You may interrupt imapsync at any time with CTRL-C. Simply restart the same command again to resume the migration. You can also run it more than once to sync changes for a staged migration. Imapsync will not copy the same data twice, so you do not have to be concerned about duplicate messages.
I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The ReadMe lists a number of other IMAP servers compatible with imapsync.
See http://imapsync.lamiral.info/INSTALL for more information on installing imapsync.
Note: see Talk:User_Migration if you’re having problems with the imapsync source.
IMAPSync with TLS Connections (STARTTLS)
Since Zimbra default settings prohibit plaintext logins we use the following method to just test if IMAPSync would actually work:
$ imapsync --nosyncacls --subscribe --syncinternaldates --fast --dry \ --host1 zimbra.firshost.com --user1 [email protected] --password1 TotallySecret123 --tls1 \ --host2 zimbra.secondhost.com --user2 [email protected] --password2 TotallySecret456 --tls2
This will do nothing nothing but show if it would work. If you really want to migrate the data just remove the “–dry” option from above example.
Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10MB)
The default Zimbra IMAP maximum email size is 10MB. This will cause errors to be raised during the imapsync for emails which are larger than 10MB. Errors such as the following will be displayed in the imapsync log.
+ Copying msg #131:11690588 to folder Folders/Projects/Consulting flags from : [\Seen]["16-Jul-2007 14:57:35 +1000"] Couldn't append msg #131 (Subject:[RE: Consultancy Approval]) to folder Folders/Projects/Consulting: 5160 BAD parse error: request too long
OR
+ Copying msg #87598:54935762 to folder INBOX flags from : [\Seen]["08-Feb-2008 14:09:05 -0500"] Couldn't append msg #87598 (Subject:[Re: photos]) to folder INBOX: 894 NO [TOOBIG] request too long
The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10MB if it’s not set. At this time the only way to set it is via the command line.
Use the following command as the zimbra user to increase to 50MB:
zmprov mcf zimbraMtaMaxMessageSize 52428800
imapsync over SSL
You can use SSL connections with imapsync providing “–ssl1” and/or “–ssl2” (and the “–port1 and/or –port2” if using nonstandard ports) for each of the accounts you wish to migrate.
For example:
$ imapsync \ --nosyncacls --subscribe --syncinternaldates \ --host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \ --host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2
NOTE: imapsync will not be able to authenticate to ZCS until you “Enable clear text login” for the IMAP service. You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.
Example
Remove the comments (lines starting with ‘#’) in order to use this command:
$ imapsync \
# imapsync with options that may help to optimize and modify disallowed folder names
--maxsize 52428800 --buffersize 8192000 \
--nofoldersizes --nosyncacls --subscribe --syncinternaldates \
--authmech2 PLAIN \
# do not migrate folders that looks like (ignore case) junk|spam|trash
--exclude '(?i)\b(Junk|Spam|Trash)\b' \
# avoid all X- headers when comparing messages
'--skipheader 'X-*' \
# avoid all nonstandard system flags (system flags start with '\')
# RFC 3501: \Answered, \Flagged, \Deleted, \Seen, \Recent, \Draft
# NOTE: need \\\\ because we want s/\\(?...) and Perl makes \\ => \
--regexflag 's/\\\\(?!Answered|Flagged|Deleted|Seen|Recent|Draft)[^\s]*\s*//ig' \
# translate folder names for characters ZCS does not allow
# * colon (:) to hyphen (-)
--regextrans2 's,:,-,g' \
# double quote (") to single quote (')
--regextrans2 's,\",'\'',g'
# Remove spaces before separators and at the end of the folder name
--regextrans2 's,\s+(?=/|$),,g'
# change reserved folder names to something that is allowed
--regextrans2 's,^(Briefcase|Calendar|Contacts|Emailed Contacts|Notebook|Tasks)(?=/|$), $1 Folder,ig' \
### site-variable options
--host1 "$host1" --host2 "$host2" \
--user1 "$username" --user2 "$username"
--password1 "$pass1" --authuser2 "$auth2" --password2 "$pass2" \
--regextrans2 's,\",-,g' \ # change quotes to dashes
--regextrans2 's,&AAo-|&AA0ACg-|&AA0ACgANAAo-(?=/|$),,g' \
--ssl1 --authmech1 PLAIN --maxcommandlength1 16384 \
# --delete2 \ # be careful - this can be destructible if not used carefully
# --dry --debug --debugimap \ # debug options
Migrating large Mailboxes and avoid timeouts
If you want to migrate large mailboxes (hundreds of folders and sub-folders containing several thousand messages) the default behaviour of imapsync may cause timeouts on the ZCS side. By default imapsync computes a lot of numbers for the sake of statistics only. By switching this off using
--nofoldersizes --skipsize --fast
in addition to above options the sync will be much faster and timeouts due to the time needed for computation of statistics will not accour anymore.
Info on migrating from courier-imap and imapsync: http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords[1]
Migrating without knowing the passwords
In the case your authentication backend stores user password in encrypted format and you don’t know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.
During a cyrus imap to zimbra migration, here’s how it worked :
- temporary openldap installed on third server, provisioned with the list of accounts to be migrated
- zimbra’s external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
- cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn’t used)
- if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed…
With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts shown below.
Batch Processing
You can automate migrating multiple users by using a script like the following:
#!/bin/bash
logfile="sinklog.txt"
host1=123.123.123.123 # host1 is Source
host2=321.321.321.321 # host2 is Dest
domain=xyz.com # domain is where email account is
# everything after @ symbol
###### Do not modify past here ######
date=`date +%X_-_%x`
echo "" >> $logfile
echo "------------------------------------" >> $logfile
echo "IMAPSync started.. $date" >> $logfile
echo "" >> $logfile
{ while IFS=';' read u1 p1; do
user=$u1"@"$domain
echo "Syncing User $user"
date=`date +%X_-_%x`
echo "Start Syncing User $u1"
echo "Starting $u1 $date" >> $logfile
imapsync --nosyncacls --syncinternaldates --host1 $host1 --user1 "$user" --password1 \
"$p1" --host2 $host2 --user2 "$user" --password2 "$p1"
date=`date +%X_-_%x`
echo "User $user done"
echo "Finished $user $date" >> $logfile
echo "" >> $logfile
done ; } < userlist.txt
date=`date +%X_-_%x`
echo "" >> $logfile
echo "IMAPSync Finished.. $date" >> $logfile
echo "------------------------------------" >> $logfile
Create a file userlist.txt with the user names and passwords in the following format:
user;password
user2;password2
Here is a second batch script that will allow different usernames on the Source and Destination servers. Name this file imapsyncbatch then chmod to 755 execute with ./imapsyncbatch can be used with the –dry option for a dry run.
#!/bin/bash
##Modified by Steve Fink stevef-at-ublug.org
##This IMAPSync Batch Script is used when you have different
##usernames on the Source and Destination servers
##(kinda what IMAPSync was intended for)
##the format for the user-list.csv file is
##sourceusername,sourcepassword,destinationusername,destinationpassword
## Get the info
while [ -z $infile ]; do
echo "What is the path to the input file?"
read infile
done
while [ -z $host1 ]; do
echo "What is the Source Host? (mail1.domain.com)"
read host1
done
while [ -z $host2 ]; do
echo "What is the Destination Host? (mail2.domain.com)"
read host2
done
while [ -z $domain ]; do
echo "What is the Domain? (domain.com)"
read domain
done
while [ -z $logfile ]; do
echo "Where would you like the log? (synclog.txt)"
read logfile
done
if [ ! -f $infile ]; then
echo "The input file does not exist!"
echo ""
echo "What is the path to the input file?"
read infile
fi
INPUTFILE=$infile
clear
echo "IMAPSync is about to begin using:"
echo "Input File $INPUTFILE"
echo "Source Host $host1"
echo "Destination Host $host2"
echo "Domain $domain"
echo "Log File $logfile"
echo ""
echo "Is this information correct?"
echo "Press Enter to continue or"
echo "Hit CTRL+C to start over"
read wait
## Begin IMAPSync
date=`date +%X_-_%x`
echo "IMAPSync Logfile started @ $logfile"
echo "" >> $logfile
echo "------------------------------------" >> $logfile
echo "IMAPSync started.. $date" >> $logfile
echo "" >> $logfile
#Get rid of the commas
tr "," " " <$INPUTFILE | while read u1 p1 u2 p2
do
user=$u1"@"$domain
user2=$u2"@"$domain
echo "Syncing User $user to $user2"
date=`date +%X_-_%x`
echo "Start Syncing User $user to $user2"
echo "Starting $u1 $date" >> $logfile
imapsync $1 --nosyncacls --syncinternaldates \
--exclude "#KnownSpam" --exclude "#FalsePositives" \
--exclude "Trash" --exclude "Deleted Items" \
--exclude "Deleted Messages" --exclude "Deleted" \
--exclude "Sent" --exclude "Sent Items" \
--exclude "Sent Messages" \
--host1 $host1 --user1 "$user" --password1 "$p1" \
--host2 $host2 --user2 "$user2" --password2 "$p2"
date=`date +%X_-_%x`
echo "User $user to $user2 done"
echo "Finished $user to $user2 $date" >> $logfile
echo "" >> $logfile
done
date=`date +%X_-_%x`
echo "" >> $logfile
echo "IMAPSync Finished.. $date" >> $logfile
echo "------------------------------------" >> $logfile
The format for the .csv file for this script is: sourceusername,sourcepassword,destinationusername,destinationpassword
