FTP Download Script

Posted on 2013 年 05 月 22 日 by cowman.chiang

<

pre lang=”bash”>

!/bin/sh

程式說明：每日將 Mail Log 匯入 Splunk 之 Script

版本時間：2013/05/30 10:40

開發維護：cowman.chiang@udngroup.com

檢查檔案是否存在，存在則移除檔案

避免ftp下載時無法覆蓋檔案以致無法取得新版檔案

function checkfile(){
if test -f “${local_dir}/${1}”
then
echo “File ${1} exist, remove!”
rm “${local_dir}/${1}”
else
echo “File ${1} not exist”
a=1
fi
}

設定 Mail Log 主機資訊

HOST1={IP_address}
USER1={User}
PASS1={Pass}

設定 imss Log 主機資訊

HOST2={IP_address}
USER2={User}
PASS2={Pass}

設定各項日期、程式參數

year_d1=$(date –date=”1 days ago” +”%Y”)
year_d2=$(date –date=”2 days ago” +”%Y”)
year_d3=$(date –date=”3 days ago” +”%Y”)
month_d1=$(date –date=”1 days ago” +”%m”)
month_d2=$(date –date=”2 days ago” +”%m”)
month_d3=$(date –date=”3 days ago” +”%m”)
ext_d1=$(date –date=”1 days ago” +”%y%m%d”)
ext_d2=$(date –date=”2 days ago” +”%y%m%d”)
local_path=”/home/splunk_input_data”
local_dir=”${local_path}/all”
declare -i date_dem=date --date="20130501" +%s
declare -i date_d2=date --date="2 days ago" +%s
declare -i date_total_s=$(($date_d2 – $date_dem))
declare -i date_diff=$(($date_total_s/60/60/24))
date_diff_d3=$((($date_diff – 1) * 86400 + 1367366400 ))
date_diff_d2=$(($date_diff * 86400 + 1367366400 ))
date_diff_d1=$((($date_diff + 1) * 86400 + 1367366400 ))
date_d1_ext=$(date –date=”1 days ago” +”%Y%m%d”)
date_d2_ext=$(date –date=”2 days ago” +”%Y%m%d”)
date_d3_ext=$(date –date=”3 days ago” +”%Y%m%d”)

依據 Log 命名原則進行設定

pop3d_log=”pop3d.log.${ext_d1}”
pop3d_log_m2=”${pop3d_log}_m2″
smtpd_log=”smtpd.log.${ext_d1}”
smtpd_log_m2=”${smtpd_log}_m2″
smtpd_2log=”smtpd_2nd.log.${ext_d1}”
smtpd_2log_m2=”${smtpd_2log}_m2″
login_log=”login.log.${ext_d1}”
login_log_m2=”${login_log}_m2″
imap4d_log=”imap4d.log.${ext_d1}”
imap4d_log_m2=”${imap4d_log}_m2″
access_log=”access_log.${date_diff_d1}.${date_d1_ext}”
access_log_m2=”${access_log}_m2″
access_2log=”access_log.${date_diff_d2}.${date_d2_ext}”
access_2log_m2=”${access_2log}_m2″
access_3log=”access_log.${date_diff_d3}.${date_d3_ext}”
access_3log_m2=”${access_3log}_m2″
imss_log=”log.imss.${date_d1_ext}.0001″
imss_log2=”log.imss.${date_d1_ext}.0002″
imss_2log=”log.imss.${date_d2_ext}.0001″
imss_2log2=”log.imss.${date_d2_ext}.0002″
mailerd_log=”mailerd.log.${ext_d1}”
mailerd_log2=”${mailerd_log}_m2″
mailerd_2log=”mailerd_2nd.log.${ext_d1}”
mailerd_2log2=”${mailerd_2log}_m2″

檢查欲處理之檔案是否已經存在於暫存目錄

checkfile ${pop3d_log}
checkfile ${pop3d_log_m2}
checkfile ${smtpd_log}
checkfile ${smtpd_log_m2}
checkfile ${smtpd_2log}
checkfile ${smtpd_2log_m2}
checkfile ${login_log}
checkfile ${login_log_m2}
checkfile ${imap4d_log}
checkfile ${imap4d_log_m2}
checkfile ${access_log}
checkfile ${access_log_m2}
checkfile ${access_2log}
checkfile ${access_2log_m2}
checkfile ${access_3log}
checkfile ${access_3log_m2}
checkfile ${imss_log}
checkfile ${imss_log2}
checkfile ${imss_2log}
checkfile ${imss_2log2}
checkfile ${mailerd_log}
checkfile ${mailerd_log2}
checkfile ${mailerd_2log}
checkfile ${mailerd_2log2}

輸出本次日期

echo date +%Y%m%d

當發現硬碟使用率大於80%時，通知維護人員

if [ /bin/df -h | awk 'NR==4 {print $5}' | sed 's/%//' -ge 80 ]
then
(
echo To: cowman.chiang@udngroup.com
echo From: Splunk@udnlap01
echo Subject: Splunk Server disk usage > 80%
echo
) | /usr/sbin/sendmail -t
fi

FTP抓取第一階段

echo “sftping file … (1/2)”

lftp -u ${USER1},${PASS1} sftp://${HOST1} <<EOF
lcd ${local_dir}
cd /webmail/usr/share/log_archive/${year_d1}${month_d1}
get ${pop3d_log}
get ${pop3d_log_m2}
get ${smtpd_log}
get ${smtpd_log_m2}
get ${smtpd_2log}
get ${smtpd_2log_m2}
get ${login_log}
get ${login_log_m2}
get ${imap4d_log}
get ${imap4d_log_m2}
get ${access_log}
get ${access_log_m2}
get ${mailerd_log}
get ${mailerd_log2}
get ${mailerd_2log}
get ${mailerd_2log2}
cd /webmail/usr/share/log_archive/${year_d2}${month_d2}
get ${access_2log}
get ${access_2log_m2}
cd /webmail/usr/share/log_archive/${year_d3}_${month_d3}
get ${access_3log}
get ${access_3log_m2}
bye
EOF

FTP抓取第二階段

echo “sftping file … (2/2)”

lftp -u ${USER2},${PASS2} sftp://${HOST2} <<EOF
lcd ${local_dir}
cd /opt/trend/imss/log
get ${imss_log}
get ${imss_log2}
get ${imss_2log}
get ${imss_2log2}
bye
EOF

輸出本次抓取資料量

echo “/usr/bin/du -m ${local_dir} | awk '{print $1}'MB”

假設抓取資料量小於450MB則進行

但有可能誤判，因access.log產生時間不確定，故在執行上會從三天前之檔案進行抓取

以至於有可能抓取到重複的檔案，導致抓取資料量遠大於將匯入Splunk處理的資料量

if [ /usr/bin/du -m ${local_dir} | awk '{print $1}' -lt 450 ]
then

假設換年度了，則創建新的年度目錄，並通知維護人員換年度了，需要上來檢查一下

if [ ${year_d1} -gt ${year_d2} ]
then
mkdir ${local_path}/pop3d/${year_d1}/
mkdir ${local_path}/smtpd/${year_d1}/
mkdir ${local_path}/login/${year_d1}/
mkdir ${local_path}/imap4d/${year_d1}/
mkdir ${local_path}/access/${year_d1}/
mkdir ${local_path}/imss/${year_d1}/
mkdir ${local_path}/mailerd/${year_d1}/
(
echo To: cowman.chiang@udngroup.com
echo From: Splunk@udnlap01
echo Subject: Splunk Change Year
echo
) | /usr/sbin/sendmail -t
fi

搬移檔案到指定目錄

mv ${local_dir}/${pop3d_log} ${local_path}/pop3d/${year_d1}/
mv ${local_dir}/${pop3d_log_m2} ${local_path}/pop3d/${year_d1}/
mv ${local_dir}/${smtpd_log} ${local_path}/smtpd/${year_d1}/
mv ${local_dir}/${smtpd_log_m2} ${local_path}/smtpd/${year_d1}/
mv ${local_dir}/${smtpd_2log} ${local_path}/smtpd/${year_d1}/
mv ${local_dir}/${smtpd_2log_m2} ${local_path}/smtpd/${year_d1}/
mv ${local_dir}/${login_log} ${local_path}/login/${year_d1}/
mv ${local_dir}/${login_log_m2} ${local_path}/login/${year_d1}/
mv ${local_dir}/${imap4d_log} ${local_path}/imap4d/${year_d1}/
mv ${local_dir}/${imap4d_log_m2} ${local_path}/imap4d/${year_d1}/
mv ${local_dir}/${access_log} ${local_path}/access/${year_d1}/
mv ${local_dir}/${access_log_m2} ${local_path}/access/${year_d1}/
mv ${local_dir}/${access_2log} ${local_path}/access/${year_d2}/
mv ${local_dir}/${access_2log_m2} ${local_path}/access/${year_d2}/
mv ${local_dir}/${access_3log} ${local_path}/access/${year_d3}/
mv ${local_dir}/${access_3log_m2} ${local_path}/access/${year_d3}/
mv ${local_dir}/${imss_log} ${local_path}/imss/${year_d1}/
mv ${local_dir}/${imss_log2} ${local_path}/imss/${year_d1}/
mv ${local_dir}/${imss_2log} ${local_path}/imss/${year_d2}/
mv ${local_dir}/${imss_2log2} ${local_path}/imss/${year_d2}/
mv ${local_dir}/${mailerd_log} ${local_path}/mailerd/${year_d1}/
mv ${local_dir}/${mailerd_log2} ${local_path}/mailerd/${year_d1}/
mv ${local_dir}/${mailerd_2log} ${local_path}/mailerd/${year_d1}/
mv ${local_dir}/${mailerd_2log2} ${local_path}/mailerd/${year_d1}/
echo “move ok”

搬移完成，通知維護人員本日匯入成功

mail -s “Splunk input data ok” cowman.chiang@udngroup.com < /home/splunk_input_data/ok.txt

(
echo To: cowman.chiang@udngroup.com
echo From: Splunk@udnlap01
echo Subject: Splunk Input Data ok
echo
cat ${local_path}/ok.txt
) | /usr/sbin/sendmail -t

else

處理失敗，通知維護人員本日因資料量大於450mb，需要手動進行操作

mail -s “Splunk input data no ok” cowman.chiang@udngroup.com < /home/splunk_input_data/nok.txt

(
echo To: cowman.chiang@udngroup.com
echo From: Splunk@udnlap01
echo Subject: Splunk Input Data No ok
echo
cat ${local_path}/nok.txt
) | /usr/sbin/sendmail -t
fi

echo “done”

This entry was posted in Linux, Splunk. Bookmark the permalink.