I defragged two of the four information store in my exchange server on Friday night. I started at 10pm and I went to bed at 4:30am Saturday morning.
My four mailbox stores are over 100BG. It's was already over the 100GB partition that I had them on. So months ago I had to more one to the same partition that our public store is on. The plan was to defrag all the stores getting back all the white space and moving the store on the public store partition back with the rest of the mailbox stores. Well that didn't happen.
All the stores are about 25GB and their streaming databases are about 6BG each. So you can see how that is well over 100GB. I did them one at a time. I first did a database move to our recovery storage partition since it's apart of our clusters mount points. This move took 45mins for both the .edb and the .stm databases. Once
moved I ran the
eseutil /d :drive\location\"database name b/c I have spaces"
This took some 3 hours. About 9GB her hours is about right according to MS. When done I mounted the store from the defrag location and did a database move back to the original location. This took another 45mins to copy back. I started the second store at about 3:30am. After that one copied to the defrag location (45min copy) and ran the eseutil and WENT TO BED. I woke up at 6:30ish am and found it was done. I copied it back and that was then of that.
The scheduled downtime was from 10pm Friday night to 10am Saturday morning. So attempting to do the last two stores would have gone well over the time I allotted for maintenance.
What was i doing why I waited for the copy and defrag to finish? While the wife and baby was asleep I took the time to play some WOW ;)
Tuesday, November 13, 2007
Tuesday, October 30, 2007
Job is relocating
My job is moving a couple a blocks. From an old historic building to another old build in NYC. At least this new old building is a lot better.
My responsibility as the MIS is move or build a new IT infrastructure. I think we are going to do a little bit of both. The problem is is that we are not all moving in at once. We are moving at about 100 a time over a 2 year period. So that means I can't just pull up the infrastructure in one location and move it over a weekend. There can't be any downtime (you think this was a financial firm). So I tasked with setting up two networks that will talk to each other so when people move from the old office to the new office everything works exactly the same. Here is what is involved on the IT side of things to get this to work.
- A solid WAN connection
- WAN accelerators Riverbed devices
- Cisco IPT phone system
- SAN and VMware
- Switches for new space
- lots of cabling
- security
- A/V
- Wireless
- new workstations
- metro card
This is pretty much the basics. We have all of this stuff now but we may need to get a second of everything. The bottom line is that there can be ZERO downtime. I think this will be a piece of cake. My boss the Director of IT seems to be stressing a bit. Hey you can only play the cards you are dealt.
My responsibility as the MIS is move or build a new IT infrastructure. I think we are going to do a little bit of both. The problem is is that we are not all moving in at once. We are moving at about 100 a time over a 2 year period. So that means I can't just pull up the infrastructure in one location and move it over a weekend. There can't be any downtime (you think this was a financial firm). So I tasked with setting up two networks that will talk to each other so when people move from the old office to the new office everything works exactly the same. Here is what is involved on the IT side of things to get this to work.
- A solid WAN connection
- WAN accelerators Riverbed devices
- Cisco IPT phone system
- SAN and VMware
- Switches for new space
- lots of cabling
- security
- A/V
- Wireless
- new workstations
- metro card
This is pretty much the basics. We have all of this stuff now but we may need to get a second of everything. The bottom line is that there can be ZERO downtime. I think this will be a piece of cake. My boss the Director of IT seems to be stressing a bit. Hey you can only play the cards you are dealt.
It's been a while..
Didn't I already use this title?
Anyway. I've done a lot since my last post. Lets see;
- I got married in June. 7/7/07 :D <--this will NEVER be any form of password LOL!
- Went on a 2 week honeymoon. Can't beat a Carribean Cruise
- Started playing WOW again
- Did a crazy setup in my house 5 boxing WOW
- Planning a physical relocation at my job for the IT infrastructure.
- Also consolidating Windows 2003 domains into a single domain.
That wasn't too much was it?
Oh and I got an iPhone too. How could I forget that. THEE BEST PHONE EVER! I can't believe I left it at home today too F%$@!
Anyway. I've done a lot since my last post. Lets see;
- I got married in June. 7/7/07 :D <--this will NEVER be any form of password LOL!
- Went on a 2 week honeymoon. Can't beat a Carribean Cruise
- Started playing WOW again
- Did a crazy setup in my house 5 boxing WOW
- Planning a physical relocation at my job for the IT infrastructure.
- Also consolidating Windows 2003 domains into a single domain.
That wasn't too much was it?
Oh and I got an iPhone too. How could I forget that. THEE BEST PHONE EVER! I can't believe I left it at home today too F%$@!
Wednesday, May 16, 2007
VMware and EMC news
I was back to focusing on some VMware last week and found that the VM's that I created were not showing up anymore. Hmm wonder why that is all of a sudden. I checked the zoning and it was all right, I checked the fiber cables they were lit. I even went as far as swapping the connection on the HBA's that didn't work as my paths the the storage under storage adapter in the VMware Infrastructure Client vanished. So I out it back. I then went into Navisphere to see if the host was showing up and it wasn't. Not even the IP was coming up. WTH! I then start browsing around the VMware forums and did a search for Clarion and ESX. I didn't find anything concrete. I'm not a member of the forum yet either so I didn't bother asking a question. I figured that this was basic and was probably asking to some degree in the past.
Anyway the problem was that this server was originally a windows 2003 server with all the emc software (SAN surfer and powerpath) installed so when the server was on it registered with CX300 automatically. So under the host tab in Navishpere the server would be right there and you can assign LUNs and away you go. Since the ESX server I didn't have the software installed it wasn't showing up in navi. I was rattling my brain trying to get this working again b/c it worked before. The VMware forums lead me to a post where a guy mentioned just adding the WWN of the server to navi and thats it. But it didn't mention exactly how. So this is where I figured out what was going on. was right clicking on everything in navi trying to find where to add a host or WWN name. I finally came across connectivity Status window. Here is where all the host and WWN names are associated. I noticed that the names (it was renamed two times) of what this server was as a windows 2003 box was still in there and associating itself with the WWN's of the HBA's. Being that the new ESX host did not have the updating software the CX300 didn't know about this server even while zoning was still in place and the hardware was the same. So I had to deregister the WWN name from all the old host names and register the WWN name to the new host which is the ESX server and new IP. Then everything started to work.
That took about a day to figure out but it felt good figuring it out none-the-less. So if anyone runs into this sort of thing who is not a VMware/EMC expert check the Connectivity Status from right clicking the Storage System in Navishpere to make sure your connection are all up to date and old connection aren't lingering around.
Anyway the problem was that this server was originally a windows 2003 server with all the emc software (SAN surfer and powerpath) installed so when the server was on it registered with CX300 automatically. So under the host tab in Navishpere the server would be right there and you can assign LUNs and away you go. Since the ESX server I didn't have the software installed it wasn't showing up in navi. I was rattling my brain trying to get this working again b/c it worked before. The VMware forums lead me to a post where a guy mentioned just adding the WWN of the server to navi and thats it. But it didn't mention exactly how. So this is where I figured out what was going on. was right clicking on everything in navi trying to find where to add a host or WWN name. I finally came across connectivity Status window. Here is where all the host and WWN names are associated. I noticed that the names (it was renamed two times) of what this server was as a windows 2003 box was still in there and associating itself with the WWN's of the HBA's. Being that the new ESX host did not have the updating software the CX300 didn't know about this server even while zoning was still in place and the hardware was the same. So I had to deregister the WWN name from all the old host names and register the WWN name to the new host which is the ESX server and new IP. Then everything started to work.
That took about a day to figure out but it felt good figuring it out none-the-less. So if anyone runs into this sort of thing who is not a VMware/EMC expert check the Connectivity Status from right clicking the Storage System in Navishpere to make sure your connection are all up to date and old connection aren't lingering around.
Friday, May 11, 2007
Exchange Cluster issue
About two weeks ago I was pretty much alone running the NY side of things. My boss the Director was out in our other office in London then Shanghai and my Admin was on vacation. So I was left to handle the back-end and make decisions on my own, AGAIN!
It was a dark and stormy Tuesday night...(it was just dark) the phone rang right as my wife tells me that my baby girl has a fever of 100+ degree's yikes! It's my boss on the phone and he says he can't connect to the exchange server from Shanghai. We are on an MPLS so everything should work. So I dig up my laptop and have a million things running through my head as I am most worried about why my daughter has such a high fever. I boot up and VPN into the office to check things out. At first glance everything looks fine. I am in my Outlook and I can OWA in as well. So what is he talking about. I VNC ALLLLLLLLLLLLLLLL the way to the shanghai server and see if I can do anything from there and I can. So what is the deal here. He tell me he keeps getting an error when trying to open outlook and OWA. So I try to login from there and I can OWA fine. I try to use his credentials from the same box and I get the error. I use his credentials on my box in the NY office (remote desktop + VPN is great) and I get the error too. So what the hell I say.
I start snooping around the exchange server manager to see if I can see anything abnormal nothing. Nothing b/c the damn thing gives no errors and the app does not refresh so I didn't know there was a problem until later. I start checking the event log, mind you it is going on 11pm and I am getting sleepy and worried about my daughter and this damn problem here at the same time. The event log was saying that the mailbox store was having problems wiring the the disk and was stopping I think it said. But that didn't register b/c I wasn't focused on this problem my daughter was boiling up and I was scared to shit. I'm still on the phone with my boss and he tells me he has to go to a meeting over there and will call me back.
I'm off the phone worried about two things. My work and my daughter. Well my daughter has went to bed and fever came down and my work was really starting to get to me. It was about 12am now I am just realizing what is happening. I at first thought my transaction logs filled up so I checked the space and it was fine then I reread the event error and was like hmm. Then is dawned on me the store can't write b/c the drive is FULL. I check and sure enough the 100% full. Then I really lost it b/c all what was going on had me not thinking logical. At that point I thought new information was over writing existing information (why? like I said I was worried about my daughter all night and not thinking straight) So I look back in the exchange system manger and refresh the mailbox stores and mailbox store #4 was down. I nearly had a heart attack. In that second I though my bosses mailbox and others were completely gone and I got up from the dinning room table walked into the living room and collapsed on the floor. It felt like all the blood drained from my head and extremities and pooled up in my stomach. Did I have an anxiety attack or pannick attack or both? After a few minutes on the floor I got up and regaining my composure. I was able to analyze what had happened and came up with a game plan to resolve the issue. I needed to move one of the mailbox stores to another partition to free up space in this one so that all store can come back online. I went to bed and got up at 3am drove into work and moved the mailbox store. it took about 15 minutes to move to 16GB store. But that did the trick.
What I need to do next (still) is shrink the database with the esutil tool to reclaim the white space. In all I should get back about 25GB. What caused all of this all of a sudden was when we moved to the cluster. The limits in the stores were not put back allowing the users to fill up their mailboxes in a matter of a month. We are back on track now and all is good again. For now!
It was a dark and stormy Tuesday night...(it was just dark) the phone rang right as my wife tells me that my baby girl has a fever of 100+ degree's yikes! It's my boss on the phone and he says he can't connect to the exchange server from Shanghai. We are on an MPLS so everything should work. So I dig up my laptop and have a million things running through my head as I am most worried about why my daughter has such a high fever. I boot up and VPN into the office to check things out. At first glance everything looks fine. I am in my Outlook and I can OWA in as well. So what is he talking about. I VNC ALLLLLLLLLLLLLLLL the way to the shanghai server and see if I can do anything from there and I can. So what is the deal here. He tell me he keeps getting an error when trying to open outlook and OWA. So I try to login from there and I can OWA fine. I try to use his credentials from the same box and I get the error. I use his credentials on my box in the NY office (remote desktop + VPN is great) and I get the error too. So what the hell I say.
I start snooping around the exchange server manager to see if I can see anything abnormal nothing. Nothing b/c the damn thing gives no errors and the app does not refresh so I didn't know there was a problem until later. I start checking the event log, mind you it is going on 11pm and I am getting sleepy and worried about my daughter and this damn problem here at the same time. The event log was saying that the mailbox store was having problems wiring the the disk and was stopping I think it said. But that didn't register b/c I wasn't focused on this problem my daughter was boiling up and I was scared to shit. I'm still on the phone with my boss and he tells me he has to go to a meeting over there and will call me back.
I'm off the phone worried about two things. My work and my daughter. Well my daughter has went to bed and fever came down and my work was really starting to get to me. It was about 12am now I am just realizing what is happening. I at first thought my transaction logs filled up so I checked the space and it was fine then I reread the event error and was like hmm. Then is dawned on me the store can't write b/c the drive is FULL. I check and sure enough the 100% full. Then I really lost it b/c all what was going on had me not thinking logical. At that point I thought new information was over writing existing information (why? like I said I was worried about my daughter all night and not thinking straight) So I look back in the exchange system manger and refresh the mailbox stores and mailbox store #4 was down. I nearly had a heart attack. In that second I though my bosses mailbox and others were completely gone and I got up from the dinning room table walked into the living room and collapsed on the floor. It felt like all the blood drained from my head and extremities and pooled up in my stomach. Did I have an anxiety attack or pannick attack or both? After a few minutes on the floor I got up and regaining my composure. I was able to analyze what had happened and came up with a game plan to resolve the issue. I needed to move one of the mailbox stores to another partition to free up space in this one so that all store can come back online. I went to bed and got up at 3am drove into work and moved the mailbox store. it took about 15 minutes to move to 16GB store. But that did the trick.
What I need to do next (still) is shrink the database with the esutil tool to reclaim the white space. In all I should get back about 25GB. What caused all of this all of a sudden was when we moved to the cluster. The limits in the stores were not put back allowing the users to fill up their mailboxes in a matter of a month. We are back on track now and all is good again. For now!
Management duties
My management duties have consumed most of my time since my last post. I also have an exchange information store shutdown. NOT a crash a shutdown. I'll make a new post about that soon. I've been dealing with meeting, talking with vendors and making sure a lot of things get done. BORING stuff. Still a lot of thinking required. The most frustrating is the unorganization of the office environment, from a business standpoint I am speaking of. Yeah I new a few things about how a business should run enough to hold a conversation ;)
Wednesday, April 18, 2007
Multi-tasking at it's finest
Multi-tasking baby. I'm talking about me here not computers. I've been swamped today trying to get VMware ESX 3 going configuring the Juniper SSL VPN box and making sure our over seas users have the proper access to their resources. Basically I am locking them down and forcing them to use the Juniper SSL VPN as their entry point. Yes I'm doing this all at the same time.
VMware ESX 3 has it's own learning curve. I've been rattling my brain just trying to install my first guest OS. I've got the VM's installed that's the easy part. For some reason the VM's won't boot from the CD-ROM. I've tried the ESX host machine and I've tried my workstation. Nadda! I've been beating the boards all morning. I ended up creating an ISO of my Windows 2003 server CD using the dd command on the ESX host.
dd if=/dev/cdrom of=/vmimages/myISO.iso bs=32k
What it does exactly I have no clue just yet. I'm joking, it's copying the files from the CD to the location it's the bs=32k that's got me. But this is the learning process anyway. I will be reinstalling once I get a handle on what exactly is happening. Also b/c I am using my only 100GB of my SAN to install all these VM's. Each VM I am giving 10GB. Eventually I will figure out best practice on the installation and how to manage LUNs off the SAN. I did a typical install :p call me a noob I don't care, two VM's with guest OS's installed more to go ;)
And as for the Juniper you can say I had a crash course in configuring that too. Under pressure it's amazing what you can do. Thats if you know what you doing.
All in all the Juniper device is great. So great in fact I ordered it today.
VMware ESX 3 has it's own learning curve. I've been rattling my brain just trying to install my first guest OS. I've got the VM's installed that's the easy part. For some reason the VM's won't boot from the CD-ROM. I've tried the ESX host machine and I've tried my workstation. Nadda! I've been beating the boards all morning. I ended up creating an ISO of my Windows 2003 server CD using the dd command on the ESX host.
dd if=/dev/cdrom of=/vmimages/myISO.iso bs=32k
What it does exactly I have no clue just yet. I'm joking, it's copying the files from the CD to the location it's the bs=32k that's got me. But this is the learning process anyway. I will be reinstalling once I get a handle on what exactly is happening. Also b/c I am using my only 100GB of my SAN to install all these VM's. Each VM I am giving 10GB. Eventually I will figure out best practice on the installation and how to manage LUNs off the SAN. I did a typical install :p call me a noob I don't care, two VM's with guest OS's installed more to go ;)
And as for the Juniper you can say I had a crash course in configuring that too. Under pressure it's amazing what you can do. Thats if you know what you doing.
All in all the Juniper device is great. So great in fact I ordered it today.
Tuesday, April 17, 2007
VMware ESX 3
Finally I can get around to installing this thing and trying it out. I'll update more once it's setup.
Wednesday, April 04, 2007
Juniper SSL VPN appliance
We are testing a Juniper SSL VPN SA-2000 appliance for 30 days. It was installed on Monday and I am impressed with what it can do. We are looking for a better VPN solution than our current. Right now we use Checkpoints secure client and we have to install that on all our remote users laptops. This limits who can VPN into the office to only those with company laptops. With the SA-2000 we can have anyone VPN into the office.
Based on the flexibility of the device we can setup policies to allow different levels of access. I can set the device to do a hardware check, user check or any combinations of checks. Example, I have hardware check on that scans the registry for company name machines based on our naming convention. If that checks out fine network connect will install. Network connect is pretty much a VPN java applet that create an SSL tunnel over http giving you an IP from a pool allowing you to have full network access. Now if the hardware check scans the registry and see that you are not a company machine you will not get network connect and will only get browser access to resources. All authentication is done via Active Directory which is nice.
If you are a certain user that is not on a company machine you have more resources published to the SSL VPN home page. Example, if I log in I will get my intranet, terminal service, all mapped drives, meeting (like webEx) and whatever other internal links that I want to add. If a regular user logs in I can have them only get Outlook Web Access and/or whatever resource they are working on internally.
I really like this solution as it pretty much the one stop shot for remote access. And the level of flexibility is great. It runs a hardened verion of LINUX not sure which distro but I can get into that some other time.
Based on the flexibility of the device we can setup policies to allow different levels of access. I can set the device to do a hardware check, user check or any combinations of checks. Example, I have hardware check on that scans the registry for company name machines based on our naming convention. If that checks out fine network connect will install. Network connect is pretty much a VPN java applet that create an SSL tunnel over http giving you an IP from a pool allowing you to have full network access. Now if the hardware check scans the registry and see that you are not a company machine you will not get network connect and will only get browser access to resources. All authentication is done via Active Directory which is nice.
If you are a certain user that is not on a company machine you have more resources published to the SSL VPN home page. Example, if I log in I will get my intranet, terminal service, all mapped drives, meeting (like webEx) and whatever other internal links that I want to add. If a regular user logs in I can have them only get Outlook Web Access and/or whatever resource they are working on internally.
I really like this solution as it pretty much the one stop shot for remote access. And the level of flexibility is great. It runs a hardened verion of LINUX not sure which distro but I can get into that some other time.
Exchange Cluster complete
So we've finished our cluster install on Friday. All mailboxes moved accept one 11GB culprit. We got Trend ScanMail installed and running. That 11GB culprit we finally got him down to 3 GB last night and moved his mailbox over. Now we can properly decommision that exchange server and reclaim the box.
Thursday, March 29, 2007
Exchange Clustering Day 5.6.7.8.9 something
In the last few days we have been moving mailboxes and trying to iron out some issues that have come up.
One issue that came up was RUS (Recipient Update Service)service was pointing to an old domain controller that was decommissioned a LONG time ago. Any way we reconfigured that. The issues that came up were when a mailbox was moved it took forever in a day for the outlook client to reconnect back. And it should reconnect back in seconds. FIXED!
Another issue we were having is when moving the mailbox that had blackberry's associated with them the BB would not be able to send emails. I think this was b/c RUS was all eF'ed up too. FIXED!
We were also having Symantec Enterprise Vault issues when we had to reset up the services to archive the public folders. The application had to associate the service to the system mailbox and it could not see ANY mailboxes. So we rebooted the EV server and we where able to see all the mailboxes and chose the system mailbox for the EV service. FIXED!
Repathing SMTP. All of our incoming and outgoing emails go to MessageLabs. You can pretty much say they have an MX record for us. They only send emails to SMTP.domain.com and only recieve emails from A.B.C.D IP's we give them. If my current single exchange server is already working can't I just swap IP's. Well yes I can't and no it will not work 100% of the time. Here is why. My firewall objects points to my exchange server, switching the internal IP on the object will work for incoming emails. SMTP.domain.com points to a public IP that is NAT'ed to an internal IP. BUT... in a cluster what we have come to find is that even though the virtual exchange server that now SMTP.domain.com will get all emails going out is totally different. Which ever node in the cluster is the active one that's the IP that will be attached to the email header. So Messagelabs is seeing this new IP from the active node trying to send emails and is rejecting it even though the emails are coming from the virtual cluster with is registered with the correct external IP. It's the active node's IP that the emails hang on to. So I had to call Messagelabs and get them to add both external IPs of the active node that I created. Well it takes 4-6 hours to propagate. We send emails to a Messagelabs cluster so the changes I've added have to hit all server in thier cluster. I was able to get emails to go out b/c some towers in the cluster had the changes and some didn't. I'll just wait until they all have the changes to switch IP's later on tonight. So now all my emails are still flowing through the single (non-clustered) exchange server. Something to watch out for if you are exchange clustering. FIXED!(in a few hours)
One issue that came up was RUS (Recipient Update Service)service was pointing to an old domain controller that was decommissioned a LONG time ago. Any way we reconfigured that. The issues that came up were when a mailbox was moved it took forever in a day for the outlook client to reconnect back. And it should reconnect back in seconds. FIXED!
Another issue we were having is when moving the mailbox that had blackberry's associated with them the BB would not be able to send emails. I think this was b/c RUS was all eF'ed up too. FIXED!
We were also having Symantec Enterprise Vault issues when we had to reset up the services to archive the public folders. The application had to associate the service to the system mailbox and it could not see ANY mailboxes. So we rebooted the EV server and we where able to see all the mailboxes and chose the system mailbox for the EV service. FIXED!
Repathing SMTP. All of our incoming and outgoing emails go to MessageLabs. You can pretty much say they have an MX record for us. They only send emails to SMTP.domain.com and only recieve emails from A.B.C.D IP's we give them. If my current single exchange server is already working can't I just swap IP's. Well yes I can't and no it will not work 100% of the time. Here is why. My firewall objects points to my exchange server, switching the internal IP on the object will work for incoming emails. SMTP.domain.com points to a public IP that is NAT'ed to an internal IP. BUT... in a cluster what we have come to find is that even though the virtual exchange server that now SMTP.domain.com will get all emails going out is totally different. Which ever node in the cluster is the active one that's the IP that will be attached to the email header. So Messagelabs is seeing this new IP from the active node trying to send emails and is rejecting it even though the emails are coming from the virtual cluster with is registered with the correct external IP. It's the active node's IP that the emails hang on to. So I had to call Messagelabs and get them to add both external IPs of the active node that I created. Well it takes 4-6 hours to propagate. We send emails to a Messagelabs cluster so the changes I've added have to hit all server in thier cluster. I was able to get emails to go out b/c some towers in the cluster had the changes and some didn't. I'll just wait until they all have the changes to switch IP's later on tonight. So now all my emails are still flowing through the single (non-clustered) exchange server. Something to watch out for if you are exchange clustering. FIXED!(in a few hours)
Friday, March 23, 2007
Exchange Clustering Day 4
We've tested mailbox moves and all works well a very small mailbox. I am in the process of moving my mailbox (700mb. I estimate it will take 40 minutes to move b/c I'm doing this in the middle of the day and the server is busy.
One thing I have to keep in mind is that the current production Exchnage server is the only server that can send emails through the firewall. Also it's the only server that can send emails to Messagelabs. So what I will have to do is change the firewall object to point to the clusters internal virtual IP. This should allow the cluster to send and recieve emails without sending on behalf of the current exchange server like it is doing now for testing.
I am also testing our blackberry functionality with the mailbox move to a new cluster as well.
One thing I have to keep in mind is that the current production Exchnage server is the only server that can send emails through the firewall. Also it's the only server that can send emails to Messagelabs. So what I will have to do is change the firewall object to point to the clusters internal virtual IP. This should allow the cluster to send and recieve emails without sending on behalf of the current exchange server like it is doing now for testing.
I am also testing our blackberry functionality with the mailbox move to a new cluster as well.
Thursday, March 22, 2007
Exchange Clustering Day 3
Today we are tweking the cluster and configuring replication on the public folders since that will take forever in a day. DAMN a day if will take a few days. We have a 100GB public information store. Yeah we really use our public folders. Hopefully today we can move a test mailbox over to the cluster and see the results.
Exchange Clustering Day 2
Day 2 was actually yesterday. It was a bit busy and frustrating at times. One of both servers were acting real funny right from the very beginning. On one the OS service pack 2 wasn't showing up in Add remove programs but it was installed. I even went ahead and installed it again three times. We went ahead and installed exchange on both servers and then created the exchange virtual server. The exchange virtual server was created but creating an IP Address Resource a Network Name Resource and a Physical Disk Resource then the System Attendant resource which allowed the cluster to show up in the Organization.
After that the failover from server A to B and back was taking enirely too long. Then Exchange SP2 wouldn't install on server A. The MSDTC service would always fail to start. So we followed some of the articles to remove and re-add it but the service never added back for some reason. So at the very end of the day I said made the decision to REFORMAT both servers and reinstall.
We've decided to look for some newer hardware drivers and firmware updates and found some and installed those. Then we replaces the heartbeat cross-over cable. Then reinstalled. Everything works so much better and the service pack was installed before the cluster was brought up this time, LOL!
That was yesterday.
After that the failover from server A to B and back was taking enirely too long. Then Exchange SP2 wouldn't install on server A. The MSDTC service would always fail to start. So we followed some of the articles to remove and re-add it but the service never added back for some reason. So at the very end of the day I said made the decision to REFORMAT both servers and reinstall.
We've decided to look for some newer hardware drivers and firmware updates and found some and installed those. Then we replaces the heartbeat cross-over cable. Then reinstalled. Everything works so much better and the service pack was installed before the cluster was brought up this time, LOL!
That was yesterday.
Tuesday, March 20, 2007
Exchange Clustering Day 1
After getting the RAM and HBA's into the server, it racked, heartbeat connected and LAN connection we installed the OS and patched them. We've named them A and B and got the carving work all set on the SAN. We've carved up 64GB for logs, 100GB for private store and 150GB for public store. We've also carved us 500mb for the Quorum drive and 5GB for the exchange mounts. The exchange LUN is to minimize the amount of drive letters that will show up in the server. There will only be C: E: and Q: NO D: F: G: H: Why? Here is why. In the drive labled exchange there will be mount points to the transaction logs, the private information store and public information store. Normally these mounts would have been drive letters in my computer.
Then we'll turn off one of the servers. In this case B. We'll assign the the Quorum and exchnage LUN to server A and run diskpart to offset the disk for performance.
diskpart
select disk #
create partition primary align=64
Do this for each LUN as per EMC's best practice.
Shut down server A and bring up server B. On server B we'll just assign the LUNs that we've just assigned to A. (NOTE you technically are not suppose to assign a sign LUN to two servers the acception is in a cluster environment which we are implementing. This is why one server is turned off). Once assigned we can run cluster manager. It does not matter what server we are on as long as one of them is turned off.
On cluster administrator click open and create new cluster. Add the current server to the cluster. This server will be in the cluster alone for now and most importantly LOCK the shared LUN's so the other server cannot write to it when it's turned back on. Add the name of the other server in the wizard. Once done turn the other server back on. Open cluster administrator on that server ( the one just turned on) and run the wizard but select add node to cluster.
The cluster should be all set up now. You should see your heartbeat and LAN connections under networks. You will have to setup disks in the Cluster group for all your LUN's even the LUN's that are mount point in the Exchnage LUN created eariler. You should also see who the server owner is for the disk are at that given time. There can only be one server owner for the disks. You can change owners which will shift the disks over to the other server by right clicking on Cluster group and move group. This will move the disks manually over to the other server in the node. This will automatically happen in the event that something happens to the active server. I am setting up an active/passive cluster BTW.
Exchange install tomorrow.
Then we'll turn off one of the servers. In this case B. We'll assign the the Quorum and exchnage LUN to server A and run diskpart to offset the disk for performance.
diskpart
select disk #
create partition primary align=64
Do this for each LUN as per EMC's best practice.
Shut down server A and bring up server B. On server B we'll just assign the LUNs that we've just assigned to A. (NOTE you technically are not suppose to assign a sign LUN to two servers the acception is in a cluster environment which we are implementing. This is why one server is turned off). Once assigned we can run cluster manager. It does not matter what server we are on as long as one of them is turned off.
On cluster administrator click open and create new cluster. Add the current server to the cluster. This server will be in the cluster alone for now and most importantly LOCK the shared LUN's so the other server cannot write to it when it's turned back on. Add the name of the other server in the wizard. Once done turn the other server back on. Open cluster administrator on that server ( the one just turned on) and run the wizard but select add node to cluster.
The cluster should be all set up now. You should see your heartbeat and LAN connections under networks. You will have to setup disks in the Cluster group for all your LUN's even the LUN's that are mount point in the Exchnage LUN created eariler. You should also see who the server owner is for the disk are at that given time. There can only be one server owner for the disks. You can change owners which will shift the disks over to the other server by right clicking on Cluster group and move group. This will move the disks manually over to the other server in the node. This will automatically happen in the event that something happens to the active server. I am setting up an active/passive cluster BTW.
Exchange install tomorrow.
VMware consolidation project, answer
To answer my own question of how a VMware server with 5 host and 4 HBA's (2 going to the SAN and 2 going to the XServeRAID) will share resourse?
I'll need ESX server and the ESX server will find all the hardware. It will act as the sole server connected to the SAN and XServeRAID. The five virtual server will no nothing of these storage devices. The ESX server will have the two paths to the SAN and two paths the the XServeRAID with two of each HBA. Once setup the ESX server will have the drives needed for each server before the actual virtual servers are installed. Then I'll install the OS's and assign the disks to each server.
I can't wait to tackle this project.
I'll need ESX server and the ESX server will find all the hardware. It will act as the sole server connected to the SAN and XServeRAID. The five virtual server will no nothing of these storage devices. The ESX server will have the two paths to the SAN and two paths the the XServeRAID with two of each HBA. Once setup the ESX server will have the drives needed for each server before the actual virtual servers are installed. Then I'll install the OS's and assign the disks to each server.
I can't wait to tackle this project.
Friday, March 16, 2007
VMware consolidation project
As I am starting my installation for my Exchange cluster project at the same time I am thinking about future projects. What came to mind was to consolidate five servers into one VMware box. Sounds easy enought but I have questions that I am uncertain about. We have;
Intranet server
OWA server
file server (Lib)
file server (home directorys)
file server (img)
I wan't my IIS servers to pull their data from my SAN (separate LUNS). I want my File server (lib) to pull from the SAN also (separate LUN) that's the easy part. The other two server File server (home directory) and (img) I'd like them to pull data from an XServeRAID. Being that the XServeRAID is not a SAN (in our environment) I cannot give them their own LUNS. They would be sharing the same volume if I set it up in it's current state and that is not good practice. So I'd ether have to put one or the other on the SAN. Anyway the real question is if I have these five servers on one physical box with four HBA's (2 to the SAN and 2 to the XServeRAID). How would five servers share four HBA's using VMware? Are virtual HBA's setup? I'll have to find out the answer to that.
VMware vendors don't call me, I'll call you.
Intranet server
OWA server
file server (Lib)
file server (home directorys)
file server (img)
I wan't my IIS servers to pull their data from my SAN (separate LUNS). I want my File server (lib) to pull from the SAN also (separate LUN) that's the easy part. The other two server File server (home directory) and (img) I'd like them to pull data from an XServeRAID. Being that the XServeRAID is not a SAN (in our environment) I cannot give them their own LUNS. They would be sharing the same volume if I set it up in it's current state and that is not good practice. So I'd ether have to put one or the other on the SAN. Anyway the real question is if I have these five servers on one physical box with four HBA's (2 to the SAN and 2 to the XServeRAID). How would five servers share four HBA's using VMware? Are virtual HBA's setup? I'll have to find out the answer to that.
VMware vendors don't call me, I'll call you.
Thursday, March 15, 2007
Update 4 External file hosting
I've also been trying to identify an external file hosting solution. I won't get into names as i don't want to directly give away the industry I work in. But some that can relate can figure that out. Or you can just ask me via email. Anyway for 200GB and 1000 users they want to charge us 90K a year to host some files for us. I can do that for a fraction of that as Verizon FIOS is available in my neighborhood now :D They've made the same joke and said it's not the storage alone we are paying but the server and the redundancy and X and Y and Z. I think it's the name that associated with them that make the price to high. Afterall they do make the applications we use the generate our large files.
Update 3 Symantec FSA
We've got Sysmantec's FSA (file system archive) installed and running. We are using this to clean up old files of the production file server. We have 1.2TB of Adobe PSD files on the file server older that 60 days. Some of you have a total of that for all your storage. Well that is a fraction of our storage and it's easily noticable via running tools to identiy files by extension and size. We are removing these files and leaving pointers so that we can get back space for our Exchange Cluster project that start next week.
Update 2 Cisco IPT
I've been dealing with our Cisco rep and our IPT implementer trying to figure out why the deployment of key features that we've liked to roll out the the user base is so painful.
The installation of our IPT system went perfectly. The migration the the new system went well also. It's been up and running for a few months now and we are satisfied with the phone system itself. What we are not satisfied with is that features of the system require different password. Pretty much every feature of Cisco Unity Connection has a separate password with different credentials. What I mean is when we setup user template originally with Unity Connection the password is 8 characters. So we went with that not knowing that when we've setup other features these features required passwords that required 6 characters. HUH! So change one set to match everything, easy fix right? WRONG! When setting up the original users on the system before rolling out these additional features you can not change the template without wiping out the user. The Template is set in stone. So as a result the users had a voice mail password of 8 characters and a PCA (Personal Communications Assistant) password of 6 characters. Not to mention a Windows password that expires every 90 days. Also we wanted to roll out the IMAP feature that allows your voice mail to show up in you outlook in a different mailbox that also has a password. You see what I'm getting at. Too many DAMN password using Unity Connection and nobody told us this when we were buying it.
These little things are easily overlooked of too stupid to even ask if you are buying a 250K phone system. You would thing Cisco would streamline some of these features and the dumbest thing an ordinary person can think of would be there. Sadly that is not the case.
Another gripe with Cisco's IPT is that we went with Unity Connection b/c we did NOT want our voice mails stored on our exchange server which is what Unified Messaging does. Unified Messaging has all these feature single password (I think) but the VM's are stored inside the Exchange server. That's a NO NO on so many levels. Some companies have very strict email policies and emeil are deleted every X days. If VM's are in there they are automatically treated like emails and wiped out. Financial firms come to mind. Law firms also. So this is why Unity Connection is there. VM's are not stored on the Exchange server. They are kept in the Unity server. But Unity Connection has all the crap I discussed up top. So why don't they have the best of both worlds? Who knows! The ideal product will be Unified Messaging with the ability to pick where you wanted to store your voice mails. If I wanted them in my Exchange server that would be the default. If I wanted them on another server server with links to exchange that should be an option. It can be done this is America and we are in 2007 anything can be done. We use Symantic Enterprise Vault to archive emails that pulls emails out of the Exchange server and stores them in a database on another server/storage device but leaves a pointer to those archives. One click and it's back in seconds. The same thing can happen with VM too Cisco. Wake up!
The installation of our IPT system went perfectly. The migration the the new system went well also. It's been up and running for a few months now and we are satisfied with the phone system itself. What we are not satisfied with is that features of the system require different password. Pretty much every feature of Cisco Unity Connection has a separate password with different credentials. What I mean is when we setup user template originally with Unity Connection the password is 8 characters. So we went with that not knowing that when we've setup other features these features required passwords that required 6 characters. HUH! So change one set to match everything, easy fix right? WRONG! When setting up the original users on the system before rolling out these additional features you can not change the template without wiping out the user. The Template is set in stone. So as a result the users had a voice mail password of 8 characters and a PCA (Personal Communications Assistant) password of 6 characters. Not to mention a Windows password that expires every 90 days. Also we wanted to roll out the IMAP feature that allows your voice mail to show up in you outlook in a different mailbox that also has a password. You see what I'm getting at. Too many DAMN password using Unity Connection and nobody told us this when we were buying it.
These little things are easily overlooked of too stupid to even ask if you are buying a 250K phone system. You would thing Cisco would streamline some of these features and the dumbest thing an ordinary person can think of would be there. Sadly that is not the case.
Another gripe with Cisco's IPT is that we went with Unity Connection b/c we did NOT want our voice mails stored on our exchange server which is what Unified Messaging does. Unified Messaging has all these feature single password (I think) but the VM's are stored inside the Exchange server. That's a NO NO on so many levels. Some companies have very strict email policies and emeil are deleted every X days. If VM's are in there they are automatically treated like emails and wiped out. Financial firms come to mind. Law firms also. So this is why Unity Connection is there. VM's are not stored on the Exchange server. They are kept in the Unity server. But Unity Connection has all the crap I discussed up top. So why don't they have the best of both worlds? Who knows! The ideal product will be Unified Messaging with the ability to pick where you wanted to store your voice mails. If I wanted them in my Exchange server that would be the default. If I wanted them on another server server with links to exchange that should be an option. It can be done this is America and we are in 2007 anything can be done. We use Symantic Enterprise Vault to archive emails that pulls emails out of the Exchange server and stores them in a database on another server/storage device but leaves a pointer to those archives. One click and it's back in seconds. The same thing can happen with VM too Cisco. Wake up!
Update 1 Exchange
So what has been going on since my last post that was on.......January 4th? Well back in January One of our exchange servers had a hardware failure. This server was in our remote office in London. The server was down for a day or two b/c there was no hardware 4 hours turn around time waranty on parts. As a result of that some very important people could not get email on that end. This important person wanted this to never happen again. So I come up with a solution for both offices. (oh just to be clear I do not administer the london office, my counter part does that. My servers have 4 hour turn around time on parts.) With that said I've come up with a solution for both offices.
My solution is to cluster the exchange servers in both offices. Increasing the rundancy of the server. Both offices currtly have one exchange 2003 server. All email flows come into the NY office and goes to the LD office across our private line. Next week we will start this project in both offices. My solution requires two new exchnage server running on MS Server 2003 Enterprise Edition. The information stores will be held on our SAN and in the SAN in the LD office. This is going to set ourselves up to start cloning the databases for backups and do away with tape, sort of. Our current exchnage 2003 server will act as a restore server. It will be hanging off the cluster so to speak and in the event of restoring that server will be the one mounting the database (information store).
My solution is to cluster the exchange servers in both offices. Increasing the rundancy of the server. Both offices currtly have one exchange 2003 server. All email flows come into the NY office and goes to the LD office across our private line. Next week we will start this project in both offices. My solution requires two new exchnage server running on MS Server 2003 Enterprise Edition. The information stores will be held on our SAN and in the SAN in the LD office. This is going to set ourselves up to start cloning the databases for backups and do away with tape, sort of. Our current exchnage 2003 server will act as a restore server. It will be hanging off the cluster so to speak and in the event of restoring that server will be the one mounting the database (information store).
Thursday, January 04, 2007
Exchange GAL updating
I've been beating myself over the head for a while now trying to figure out why my GAL (Global Address List) wasn't updating in Outlook when I make changes to a user account or AD object. When you create users in AD even if they are test users they will show up in the GAL if you are not copying a preexisting configured template. We try to keep our GAL clean and free of all the crap that we test but some times if we don't delete the accounts or can't for that matter they show up in the GAL. So I would go in and hide the user from the GAL. Exchange/Outlook is funny where you have to either close outlook and reopen it for the changes to show or restart the Information store to force the change. Well recently I remember one of my colleagues saying it was best to have Outlook cache mode on. I sort of disagreed but I turning it on anyway and forgot about it. So I get to request to clean up the GAL. I proceed to do so and low and behold the names in GAL still show up. I remember this happening in the past a restart of the Information Store always fixed it. Not this time. Then I go and check Outlook to see if it was caching and yes as I mentioned before I forgot that I turned caching on and it was keeping all of the information. Don't know why cache insists on keeping the information even after restarting the Information Store and restarting outlook. A safety net perhaps?
Subscribe to:
Posts (Atom)