Jerry Normandin
Current Employer: Welch's, Concord MA
Position: Sr. Linux Systems Specialist / Architect
Current Assignment: ERP / Linux Support. Sometimes a project sounds so interesting that you just have to drop what your doing and get on board! Welch's is currently redesigning their ERP infrastructure. The new ERP is based on a Linux/Oracle platform, what's unique to this deployment is that they are using RAC. Yes.. A Linux Cluster! We have currently 8 nodes on PROD, 6 nodes on QA, and for now development is a collapsed tier. Soon development will evolve into a 5 node cluster.
I'm the only Linux Systems Admin so I've written many scripts to manage the cluster. I'm currently modifying the "Distribulator" to fit our needs at Welchs. With the Distribulator you can easily configure groups of servers in XML. I've got global, prod, qa, dev, dbservers, appservers, and cmserver groups configured. This utility is extremely handy. Say for instance you have a kernel tuning parameter you wish to apply to only the database servers, all you need to do is configure your system.conf locally, select your dbserver group, and issue the copy command to /etc. The sysconf file will get copied to all your database servers. When complete issue the run "/sbin/sysctl -p command, All you have to do is issue this once and all your dbservers will get updated. I've written scripts that parse Welch's printer assignment spreadsheets and generate a xml printer configuration file, this file gets fed to Redhat's Printconf tool via the "Distribualtor" to globally update the printer assignments. I'm also designing a bare metal recovery system based on "monodarchive". The difference is I am backing up a os install complete with a Veritas client install. When complete Veritas will take over and restore oracle and rac. On July 13th, 2004 we survived our first Major power hit. A spike hit Welch's and the Best UPS blew a hole through one of the batteries, this led to the SAN coming down without our Oracle Cluster coming down cleanly. When the UPS was repaired and power was restored, the Linux Oracle cluster came up cleanly! We only lost one drive on the SAN (a hot-spare took over) and one drive on prdapp3 (The Drive was mirrored). Note: Dell had a drive shipped to me in 4hrs! The server was up with no downtime. Dell Poweredge is hot swappable! I had the mirror rebuilt in three minutes!
A full 8hr day at Welch's feels like a 3hr Day. Time flies!