Snapvault

As I wrote before, we’re using SnapManager (for SQL/Oracle) to create consistent snapshots. However my database guys don’t want to name their snapshots daily. (which I can understand), as once you archive those snapshots to a secondary (and tertiary) system, the names become junk. So, they’re naming the snapshots like snap__vcsrv_29_12_2012-10.00.01. Sadly, when it comes to SnapVault, it expects the names in form of daily. otherwise you won’t be able to transfer the snapshots with the CLI (none that I have found anyway). But we didn’t want to move away from naming the snapshots the way they are, so I ended up writing a PowerShell script, that once triggered archives the Snapshots needed for a set of databases. It took me a while to figure a bunch of stuff out, but in the end I think I have a working way of archiving custom-named snapshots.

Well, I recently had the pleasant task of implementing SnapVault backups, that are being shipped to an offsite location with SnapMirror. That in itself isn’t the bad thing, however we decided against Protection Manager (since it was a charged product back when we decided on this). So I basically had the three tasks: Actually implement the SnapVault stuff (and learn my way around it and also document it) Write a bunch of scripts, that help us in creating scheduled backups of our databases Create a monitoring script, that’ll fit into our Nagios environment already in place Well, two months later (sadly it still has some kinks - I can’t figure out this one bug though for the life of it) and a few hundred hours of working on/with it and out came four things: ...

I’ve been spending a lot of my time the last week on getting SnapVault with out FAS-filers to work. Out came a script, which does this for a given volume (and of course SnapVault Primary and Secondary). The script expects, that SSH public key authentification has been set up.

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me. Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl. Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice. So far the plugin supports the following things: Monitoring FlexVolumes (simply watching the free space) Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs) Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots) Monitoring SnapVault relations (and their age) Monitoring SnapMirror relations (and their age) The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor - which sadly needs to have the admin role.

Well, for the past two months I had a case open with NetApp to figure out this SnapVault replication issue we were seeing. The initial transfer of the SnapVault relation would complete with a hick up, manual snapshot transfers also work - just the scheduled, auto-created Snapshots won’t replicate. At first I (and the NetApp support) thought this was an issue with SnapVault itself, however after being away for the last four weeks I looked at the issue with fresh eyes. After a short peek into the logs, I found what I had found back when I first looked into this. ...

Well, the name says it pretty much. Once you rename the snapshot on the SnapVault destination from daily.0 to something else, the whole builtin SnapVault snapshot retention isn’t gonna work anymore. Back when I started all the code-writing, I wasn’t aware of this. One of my co-worker complained to me about it on Wednesday that there are an assfull of snapshots on the SnapVault destination (one snapshot each day since the end of October, meaning more than 50 snapshots per volume, in a total of 12 or so FlexVolumes, making the total about 500 snapshots). So I took the time to write this little Bash script (yeah, I know I’m mixing a bunch of languages - I really like the KISS principle), which will get the necessary information from the filer (snapvault snap sched needs to be set) and then deletes the over-aged snapshots.

The MSSQL admins decided to dump the SMSQL Snapinfo stuff on a separate volume, that SMSQL also snapshots. Same as before, I need a PowerShell script that’ll archive the snapshot and rename it.

And here’s the script for SMO. However, since different people administrate the Oracle Databases, they don’t want me to tinker with the database like the MSSQL admins. They give me a CSV-list of volumes, that should be backed up and I work with that.