Out of inodes is like running out of gas. You need gas to drive your vehicle. In a similar way you need more inodes to create files in a file system. This out of inodes typically pops up when the file systems contains lots of small files. To give a little background about this, An inode is a data structure that defines a file, except for the filename which is stored in the directory entry. Note that a directory is just another file. Inodes point to blocks that make up a file, and inodes also contain the metadata of the file.

NetApp allocates one inode per 32KB of data in a volume by default. This value can be increased to as high as 1 inode per 4KB of data via the maxfiles command.

You can use the following command to find out how many are used & how many are available.

[root@cerridwen ~]# rsh sundog df -ih layout_rm2700_002
Filesystem iused ifree %iused Mounted on
/vol/layout_rm2700_002/ 40887 9702674 0% /vol/layout_rm2700_002/
[root@cerridwen ~]#

Corrective actions:

1. Move files to another volume or delete files if possible. Ask our end users to clean up.

Note: Snapshots lock files (and corresponding inodes). If you delete the file from the active file system and have snapshots of those deleted files, the inodes are not removed until the snapshots are deleted or rolled off.

2. If the above option is not possible (like bby1bakfil01 where we can’t delete stuff).

Increase the max inode value on the volume. Use the maxfiles command to increase the number of files at any time. My preference would be increasing in small amount like 1-2 %. Once we increase, there is no way to decrease. Increasing in small amounts would be good way to do. Please find the example below

[root@cerridwen ~]# rsh bby1bakfil01 df -ih RVL_binford_1
Filesystem iused ifree %iused Mounted on
/vol/RVL_binford_1/ 32285584 111860 100% /vol/RVL_binford_1/
[root@cerridwen ~]#

[root@cerridwen ~]# rsh bby1bakfil01 maxfiles RVL_binford_1 32485590

The new maximum number of files will be rounded to 32485583.

[root@cerridwen ~]#
[root@cerridwen ~]# rsh bby1bakfil01 df -ih RVL_binford_1
Filesystem iused ifree %iused Mounted on
/vol/RVL_binford_1/ 32285619 199964 99% /vol/RVL_binford_1/
[root@cerridwen ~]#

Thx, Satheesh

It’s been a while since I posted/updated this high-end technical blog . My apologies. I got caught up with the work and forgot this. Giving back to community is an important thing and I shouldn’t have neglected. From here now, I’ll dedicated 2-3 hours on saturday/sunday to write about the technical stuff.

This time I’m going to write more hands on stuff. Please watch out ..

Satheesh

All of a sudden (and I’m sure plenty of people will claim it isn’t sudden) solid state disks are all the rage. For servers, I can see the logic. It’s another step in keeping the power and cooling demands of servers down; it also extends primary memory further and will definitely increase performance.

But what about storage arrays? I can see the benefit of putting a tier of SSD drives into DMX arrays, especially in the way EMC have chosen to implement it. It allows those targeted applications to get the performance they require at a manageable price point without a drastic reconfiguration of the array. But an entire array of SSD? That’s just the same as existing products like Tera-RamSan.

If SSD prices are driven down, then surely so will the price of standard hard drives. HDD manufacturers aren’t going to lie down and let solid state take away their business. We’ve seen their response already with Seagate taking STEC to court over patent infringements.

So where will it end? Well, tape didn’t go away as many forecast it would. I don’t see spinning drives going away any time soon either. What I’d like to see is the rise of intelligent storage systems that learn the busy and quiet blocks and move the data between SSD and HDD to keep optimal performance. Meantime, HDD prices will continue to fall and the battle will be between cheap (but fast) HDDs and balancing their cost against the power/cooling they need.

But I’d love to see the rise of SSD’s as it’s a transforming technology in drive business. Let’s see how will be the scenario in 2 yrs down the line

What’s cloud computing? WTF is happening ?.  On one fine day, I started to think what is happening around me. What is cloud computing & how does it impact my job/nature of work. I surfed over internet & though of writing my analysis it in my blog.  Well It’ll have major impact on all the Storage Administrators all the world. Currently clients manage thier IT infrastructure by recruiting their Admins. But if this Cloud Computing catches, all these jobs will be gone as clients to choose to go for managed services from the vendor

It’s a broad and somewhat nebulous term that refers to services and software that run on computers that you don’t need to purchase or operate yourself – they’re “out there,” in the clouds. Essentially, it’s a new twist on outsourcing. Cloud computing users pay only for the services they need – much the same way you pay your water bill – which offers the ability to increase or decrease capacity as needed.

If you use Google’s Gmail service for e-mail, that’s a type of cloud computing. But much of the cloud action happening locally is around enabling businesses to outsource tasks such as archiving e-mails or running analyses of their latest sales numbers, without having to buy the hardware or software themselves.

Cloud computing will change how we do IT, end-to-end, over the next five years, It’s like in the early 20th century, if you were a manufacturer, you had to build your own power plant. But eventually, you had the option to buy your electricity from the grid, and let someone else worry about how it was generated. Corporate data centers are going to have that kind of choice, too.

I heard that EMC has formed a new business unit called the Cloud Infrastructure Group, run by an executive in Seattle who also oversees Mozy, the company’s data back-up service for consumers and small businesses. But the company isn’t yet selling cloud services to its meat-and-potatoes corporate customers.

Last month, Iron Mountain, a  company based in Boston, unveiled its own cloud service called Virtual File Store. Customers can use it to archive data to which they don’t need frequent access, paying a monthly fee rather than buying their own storage devices (which often come from a company such as EMC).

It seems small and midsize businesses will be the first to integrate cloud-based services, while bigger companies will take longer to figure out how these services relate to their existing operations, whether they’re secure and reliable enough, and whether they comply with all sorts of regulatory requirements. But the appeal is that cloud-based services can cut the costs of all sorts of IT operations like storage and number-crunching and payment processing, while giving companies the flexibility to simply pay a higher monthly fee as they grow – or pay less when they shrink.

I personally know some of the clients using the Cloud Computing Data centre in Texas, Singapore & Scotland. Currently Tiny city countries  like Singapore & Hongkong are most preferred locations in Asia for building Cloud Computing DC’s.

Since I can’t sleep on airplanes I decided to read up on Service Oriented Architectures, I need not have bothered since I’ve already studied object oriented programming. (Smalltalk…. So I learned OOP the way Alan Kay wanted it taught.) If you take the concepts of Abstraction, Encapsulation, Inheritance, and Polymorphism then map them to the design of reusable components aligned against business processes, as you throw in the network as your message passing transport, you’re pretty much in business.

What’s interesting is from what I read they’re all pretty much in agreement that Windows is the wrong foundation on which to base your SOA due to it’s incredibly poor resource utilization, being a UNIX guy I tend to agree with anything which points out that Windows is a prolific platform for gaming & malware but near worthless for everything else

I’ll admit that I’m bitter as I’ve had to work with Win2K3 a lot recently and I’ve found it to be a pain in the testicles.

Going from one waste of money to another, a significant amount of SOA literature advocates allowing the Mainframe weirdos back into your data center. Of course the literature in question is written by IBM’rs who are looking to ensure that the Mainframe pension plan continues for the another 50 years or until the last Mainframe dude shuffles off the mortal coil. Mainframe operators are the only people I’ve ever met who’d happily be entombed in their Z-series when they expire.

One of the big things which gets me about SOA is how messy backup & recovery becomes. Security you can do if you design it in from the start & not try and bolt it on afterwards, while backup is one thing lets say you’re recovering information to a service on which numerous other services have a dependency, and since it’s a SOA there should be a lot of other services which have dependencies, who knows what chaos a restore could cause when the information set you’re restoring might touch so many different services or end user mashups?

The vision is great, but the implementation and ongoing management looks as if it could be horrific. It’s layer upon layer of complexity which typically does nothing but add fragility

He has a small environment of about 15 servers with local storage.  Customer has a requirements for another 40 + servers, but with strict budget requirements, they are looking towards  me for cost effective, performance effective solutions.

I proposed VMWare ESX server , 1:5 physical to virutal ratio, all booting from iSCSI SAN ( NetApp FAS 270 Cluster) . Databases, Sharedrives & retention data on FAS 270c. This is the Best part I like about NetApp. You buy one Box & it you can use it for DB’s, Sharedrives & Retention data, Backup snaphsots etc & etc. That’s  what I call as Unified Storage.  SMB customers like this kind of design. This makes their IT environment to be simple, rather than EMC Design where you end up buying boxes for each & every feature.

Now that we’ve established our  stage, let’s take a look at the redundancy.  We’ve 15 physcal servers & 75 VM’s. Booting from iSCSI SAN and databases also using iSCSI from NetApp FAS 270c

The question I get asked most is won’t my virtual machines be slow if the storage is via iSCSI? My response is show me a server that uses more then 1Gbps of storage bandwidth and then maybe it might be slow. The reality is that you don’t even use 100Mbps 95% of the time, even during boot up which is the most disk intensive operation.

So, in any VMware implementation, Fibre Channel provided the best fail over. It was instant. Network failover was pretty good, you’d look 1-3 packets depending on the switches you used, but FC was flawless everytime.

So now you have a smaller site with a smaller budget and they want the benefits of virtualization and the ever important VMotion. So you buy a couple of servers for ESX, a virtual center server, and nice little Netapp, say a FAS250 or a 270. Could actually be some other iSCSI provider, but we’ll stick to Netapp.

The key requirement is that you need to provide redundancy so that the storage for your ESX servers doesn’t disappear and all your servers crash. In this post we’ll talk about protecting against port failure and switch failure.

From a Netapp perspective the key to failure protection is to ensure that you have two network ports available for iSCSI traffic. If possible, these ports should only do iSCSI, but on the smaller devices you only have two ports per head so you don’t have this option. For this I would recommend a non-routable VLAN, one that doesn’t have anything else communicating on it.

When you have the two network ports, define a Single Mode Virtual Interace (VIF). This means that of the two ports, only one communicates at a time and requires no configuration from a switch perspective. Otherwise known as Network Failover. As a result, you can connect each port to a different switch and protect against both switch failover and port failover.

Another option is that if you have four ports available, you can create two Multimode VIFs that trunks/aggregates the two ports together for Network Load Balancing. With Multimode VIFs, you can then create a Single Mode VIF that spans them so that you have Network Failover with Network Load Balancing. Of the four ports, only two will be active at a time. The two ports in a Mutlimode VIF must be connected to the same switch and you’ll have to enable LACP on those two ports. However, the other Multimode VIF can be connected to another network switch so you are still protecting against a switch failure. Hope that makes sense, its easier then it sounds.

Now most people would say that I have a 2Gbps interface into my Netapp, this is great. However, keep in mind that between two devices, the fastest you can communicate is 1Gbps even if both stations have multiple network cards trunked together. Where the benefit lies is when multiple servers communicate to one Netapp consuming more then 1Gbps of traffic. However, you’ll rarely see it and if you do then you probably already have Fibre Channel anyways.

Our network redundancy is now complete from a Netapp perspective, so we will now switch to the ESX server.

ESX 3.0 allows you to do something similar to the Netapp in that you can configure two ports to either operate together in Network Load Balancing (requiring switch configuration) or in Network Failover allowing you to span the connection across multiple switches.

In the configuration of the ESX server, define a new virtual switch and assign two adapters to it. For a connection that will use the ESX iSCSI software intiator, you need a service console port and a VMkernel port assigned to the virtual switch. I haven’t really figured out why yet, but to make it work both of these are required. Ensure as well that one of the network adapters that you assigned to the virtual switch is set to standby mode. This sets it up as Network Failover.

Also, you need to enable iSCSI Client traffic from the ESX Server by modifying the Security Profile of the ESX Server.

You now have network failover from a Netapp perspective and an ESX Server.

However….you do not have head failover if you are running clustering on your Netapp. This is because the ESX iSCSI software initiator does not provide mutli-pathing Input Output (MPIO). When connecting to the Netapp with iSCSI you should always use MPIO by providing the IP addresses associated with both Netapp heads. So that if the primary head fails it uses the IP address of the second head to get to the disks.

I hope this helps. There is a few gotchas as mentioned, but the value of iSCSI is definately worth trying it out.

Cheers

There are couple of ways we can provide “multipathing” of iSCSI targets to a Solaris system, none of which are rocket science. Because we’re using ethernet, we can use Solaris IP Multipathing (IPMP) to provide active-failover ethernet interfaces, or we can use the new link-aggregation capabilities in Solaris 10 Update 1 (and OpenSolaris) to create trunked connections, or we can use the traditional Solaris MPXIO to provide multipathing in situations where one path is via Fibre Channel and the other is via iSCSI. How cool is that! Options, Options… options are a good thing. Viva Sun!

The best and easiest solution for Solaris when used with a NetApp Filer is to use the active-failover method on gigabit ethernet and ensuring that each path is through a seperate switch, like this:

So this is pretty straight forward. On Solaris we use IPMP on 2 or more interfaces to allow for failover in the case that either the switch or the NIC fails. On the NetApp side we do the same thing, except they call it VIF’s, Virtual Interfaces.

The easiest way to setup VIF’s on a NetApp Filer is to modify your /etc/rc file which sets up the Filer enviroment at boot. Here is an example of three lines that need to be present for VIFs to work:

hostname toaster
vif create single Failover0 e8 e7
ifconfig Failover0 `hostname`-Failover0 netmask 255.255.252.0 broadcast 10.10.3.255 mtusize 1500

So here we’re creating a failover interface named “failover0″ using interfaces e8 and e7 (these numbers correspond the the PCI slot in which the interface exists). The ifconfig command used here requires that “hostname-Failover0″ is listed in your /etc/hosts as the IP which will failover between interfaces.

Once you have that setup properly, you can reboot the Filer and check the setup:

toaster> vif status
default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’
Failover0: 1 link, transmit ‘none’, VIF Type ‘single_mode’ fail ‘default’
VIF Status Up Addr_set
up:
e7: state up, since 28Jun2004 15:20:05 (13+23:36:21)
mediatype: auto-1000sx-fd-up
flags: enabled
input packets 4742755, input bytes 602715164
output packets 36385, output bytes 3460991
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: up at boot
consecutive 1208171, transitions 1
down:
e8: state down, since 28Jun2004 15:20:05 (13+23:36:21)
mediatype: auto-1000sx-fd-up
flags: enabled
input packets 4988564, input bytes 635539753
output packets 0, output bytes 0
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: up at boot
consecutive 1208171, transitions 1

On the Solaris side we use IPMP, IP Multipathing. For coverage of that I’ll point you to a more complete blog entry on the subject that I wrote some time ago: Solaris Tips & Tricks: IP Multipathing.

With VIF’s in place on the Filer and IPMP on the system side we can survive a link failure caused by interfaces on either end or failure of a switch and keep pushing data. If you wanted to go even further we could use link aggregation (aka: trunking), but thats beyond the scope of this entry.

Many of you know that I used live in Singapore. Singapore is country transformed from a mosquito-infested swamp full of poor people into a vibrant developed nation of prosperous people in a brief span of 40 years is exhilarating. Comparing Singapore to India from an Indian’s perspective is depressing: how did we–given all the advantages we had in 1950 compared to Singapore–squander it all and end up being a poor misgoverned over-populated country? That is the depressing bit.

There are lessons by the score that one can learn from the Singapore experiment; lessons that could be arrived at through simple logical reasoning in the abstract but made all the more compelling to see it actually work out in practice. The fundamental lesson to my mind is this: policies — well thought out, rigorously implemented, and single-mindedly enforced — have the power to transform.

Lee Kuan Yew is one of the most intelligent leaders in contemporary history. The man is a practical genius. The people of Singapore got lucky when in the random draw from which leaders are drawn, they drew Lee Kuan Yew. India, I cannot but note with sadness and grief, drew from the same random draw and came up with Jawaharlal Nehru. Both are leaders, but one a practical genius and the other . . . well, the less said the better.

There are deep contrasts between India and Singapore. Take for instance the degree of corruption that permeates both public and private sectors. According to Transparency International, India ranks 90th (in the company of such nations as Malawi, Mozambique, Nepal, Russia, and Tanzania) while Singapore ranks 5th (led by Finland, New Zealand, Demark, and Iceland) least corrupt country.

The corrosive impact of corruption on economic development and growth is not a mystery, nor was it unknown fifty years ago. Lee Kuan Yew decided on a zero-tolerance policy on corruption. Corruption at all levels of society had to go. The task was to re-invent the whole culture so that corruption had no place in it. That was the first bit: deciding that corruption was history. The next bit is implementation and enforcement.

This is what I heard. A certain minister, very close to Lee Kuan Yew, in charge of housing (or some such) was involved in some kick-backs. The word went around that the guy will surely get off easy since he was in the inside circle. Lee asked the minister to see him. The meeting was brief. Two days later the minister blew his brains out. The message was clear: zero tolerance.

In India we hear of some high-level bureaucrat or politician robbing the public purse blind with sickening regularity. But we have never heard of even one high-ranking corrupt public official or politician ever being punished for his misdeeds. We have a free press of sorts and people get to know about how the most corrupt get away with murder. The notion that it is OK to be corrupt is internalized and soon enough we justify our own petty corruption by referring it back to those high and mighty whose corruption is legendary and who are never punished. We grow cynical and the society suffers as a whole. Our culture erodes and standards of probity and justice fall until we are a nation of petty thieves ruled by mega-robbers.

To re-iterate once again (as they say in the Department of Redundancy Department), you have to have intelligent policy, rigorous implementation and no-exception enforcement to bring about a radical change. Most policies in India don’t meet the intelligence criterion, and those that do suffer from indifferent implementation and half-hearted enforcement.

Crimes other than corruption are also a brake on economic growth. Singapore controls these without a too visible police force. I only saw a couple of cops during my three-day visit. One of the most impressive people I met while in Singapore (who is an alien in Singapore but runs a very successful business) told me of his informal theory about how they keep crime low. He said that he imagines that in the police headquarters they have a huge wall chart where each crime has a schedule of enforcement. So, for instance, “vandalism” may be scheduled for the week of 15th of August. That week they go out and catch a vandal, prosecute him to the utmost, and plaster his picture on the papers and in the write-up use the word “shame” a dozen times.

Prospective vandals, however irregular they may be in keeping up with current affairs, get to learn about the punishment and decide to curb their impulses. But public memory fades with time. So after a suitable span of time, the police will once again catch a vandal and make an example of him. They repeat this same formula with other routine crimes.

The important bit is that you don’t have to have zillions of cops watching every corner for vandal all round the year. You just catch the one every now and then to put the fear of god into the others and thus prevent vandalism from happening in the first place.

Lee Kuan Yew (I like using his full name because there is a certain something, a rhythm to it) must be a remarkable man. He is someone I would have liked to meet him and bow deep as a sign of my respect for what he did for Singapore.

Apparently little things, things that one may not consider very important or significant in the grand scheme of things, they too have a transformational impact on the society. Litter and garbage on the streets depresses the spirit and instills a sense of hopelessness and helplessness in the society. Lee Kuan Yew fined people who littered so vigorously that Singapore became clean but earned the reputation of being a “Fine City.”

Freedom of religion is guaranteed in Singapore but freedom to proselytize is not. Proselytizing essentially says that my religion is better than your religion and that if you don’t accept my god as the One True Savior(TM), you will rot in hell that my god has specially prepared for you. This sows seeds of discord in society and soon the newly converted start asking for special treatment and handouts and in the limiting case, when the bunch grows sufficiently large, ask for a separate state of their own because they cannot bear to live with the other people who are destined to go to hell.

So Singapore is strict about proselytizing. In keeping with their policy of discouraging that anti-social behavior, they caught a meek little Catholic lady who was going door to door peddling her religion and threw her into jail after she was found guilty by the courts. Then they publicized the event. This sent the message to all religious bigots who follow the dictates of their own hearts that bigotry is not ok.

No such luck in India, of course. We have Christian missionaries from all over the world having a grand old time converting heathens and soon enough you have the neo-converts pissing on Ganesh idols to show their new-found faith. News gets around and finally out of desperation and plain old brutality, a few missionaries get roasted and this gives the country an ill-deserved reputation of being intolerant. Madrassas funded by Saudi money flourish by the thousands where apparently the mullahs teach the young that killing kuffars is a pretty practical way of arranging society.

In reaction to this ocassionally, a few of the normally tolerant Hindus band together and retaliate. This hits the international press and India is tarred as a society full of murdering morons.

As I was saying, Singapore does not have those problems because they have the enlightened policy of making proselytizing a crime and then enforce it. Lacking the essential bit that leads to religious disharmony, they avoid the entire series of unwelcome consequences.

The essential important faculty that gives rise to good policy — which our Indian leaders lack — is imagination.

Humans, I imagine, are different cognitively from other things in the universe in their capacity to imagine. We can ask “What if” and think through the consequences of a set of actions that are not yet set into motion. We have to be able to foresee the consequence of our present actions to reach a desired future state. Or by backward induction, we can start at a future desired state and work our way back to what we should be doing today to obtain the future state.

Every chronic persistent shortage you see around you in India is the result of a failure of imagination. (I think that this statement should be elevated to the status of a principle. Here is the one of the first axioms, then.)

In Bangalore, we have power cuts for about 4 hours a day on average. Pune is a city with a population the size of New Zealand’s population — four million people. It is certainly not an obscure little village in the middle of some god-forsaken forest. Power is not a new-fangled fad whose demand could not be foreseen. The growth of the size of the city and the consequent demand for power could have been easily foreseen and actions taken. Power generation is not an esoteric undertaking which the private sector is incapable of doing. Yet there is a shortage and the economy suffers because some idiot in charge did not have the imagination to realize that more power is needed.

Not so the Singaporeans under Lee Kuan Yew. They learnt to use their imagination. They build capacity before they hit shortage. For example, they have started building the third terminal at the airport even though the second one is not even up to full capacity. Most of my fellow Singaporeans say “Singapore Govt thinks 20 Years ahead & designs the polcies”. That should be the vision & imagination a Government should possess.

Compare that to India. First a road gets choked with 10 times the number of vehicles than it was designed to handle. Then the realization dawns on people that the capacity has to be increased. On an already congested road, they start making some changes — for instance a bridge. This take about four years to complete (whereas the same work in a different place would have taken four months). By the time the capacity is in place, the traffic has also increased so that once again it is 10 times what the road can handle.

This reminds me of my email inbox. For the last year or so, I am constantly falling behind — the number of messages sitting there increases monotonically. I am forever trying to catch up.

But enough of my woes. I was going on about how smart Lee Kuan Yew was. He has the best imagination of them all, I guess. Take for example his insistence on air-conditioning. Singapore is a hot and extremely humid place around the year. Without AC, you are bound to be less productive than with it. Air-conditioning makes sense if the cost is lower than the increased income from a more productive workforce. He saw the benefits of AC and implemented it.

I don’t know why but some people just draw good cards from the random draw that is life. Singaporeans are lucky. I am sure there are those who will immediately retort that the Singaporeans don’t have the freedoms that are normally associated with a liberal democracy. And I am also sure that the person making that statement is sitting comfortably well-fed in his nice office or home accessing the world wide web for knowledge and entertainment. For the average schmuck in a third world country, he would any day trade in his imaginary freedoms for a decent shot at a full stomach, a roof over his head, and a chance to get his children educated. After the average schmuck has achieved those basic necessities, he would ask for all sorts of goodies that a liberal democracy provides. And that is when the society should become a liberal democracy.

A year back, I happen to visit this site freenas & I went on reading about it. The guy is developing a NAS software code which can be deployed on a work station/server. I felt like trying it once, but I couldn’t find time to experiment on this.

Last week, I took a 5 day break from my work to spend time in photography. But it was raining all the time, so I didn’t do much photography. I thought about spending time on deploying this software in my Old computer ( This is very old & it’s unused for a while). I got some hardware to test it.

I downloaded & deployed the Free NAS server software. Initially it was throwing problems for so many reasons. After 3 days of continuous debugging, Finally I made it up. It was worth spending the time.
Now my Old computer which I was about to throw, is being used as NAS Server.

I Can access the data on Hard drive ( It’s 500 Gb) through CIFS & NFS. I Can do a backup to tape. I can connect the NAS Server to internet as well.

I strongly believe that this Free NAS Server is gonna give NetApp run for their money. NetApp sells their products not less than 50 K USD. When compared with Money, Free NAS server does deserve appaulds. I strongly recommend people with unused computers to try deploying this software

If you come across any problems during deployment, Just shoot a email to me at satheesh.mc2@gmail.com. I’ll try to help you out. To know more about me, click here