Prevent Disaster in the Data Center

By George Spafford

July 26, 2005: CEOs ignore data-center environmentals at their own peril.

There are many reasons to create segregated physical locations for servers and other critical infrastructure equipment.

First, access is controlled, thus limiting security threats. Second, the controlled access limits human error arising from accidents and “curiosity.” Third, the concentration allows for efficient oversight and administration. Fourth, and the focus of this article, the relative consolidation of assets enables a controlled environment to better manage the risks associated with air-conditioning, fire and flooding.

Air-Conditioning/System Cooling

Today’s IT systems generate a tremendous amount of heat and need dedicated air-conditioning systems to be properly cooled. Some systems are even implementing cooling systems that dissipate heat via mechanisms other than the traditional muffin fans and are reminiscent of the days of mainframes and water cooling (at least to those of us who have been around long enough to have seen the water pipes and raised floors).

Systems that are running hotter than recommended are more likely to have component failure than ones in a cooler setting. Some years ago, I was involved with a small server room that didn’t have a dedicated AC unit, but did have a dedicated duct. It worked great during the week when people were present to cause the AC unit to run because the thermostat wasn’t in the server room. On weekends, the office area would cool off quickly and shut down while the server room baked. We knew something odd was going on when RAID drives and other components started failing far too often.

The climax came when a Dell-hosted clustered SQL Server system announced at the console that it had reached a critical internal temperature and was shutting down immediately to protect itself. This made several production departments grind to a complete halt. The first step was to put in a temperature probe that had an IP address that could be SNMP-polled every few minutes. The data was logged, trended graphically and the resulting report to senior management with graphics resulted in a dedicated AC unit getting capital approval and installed in record time.

A second benefit of air-conditioning relates to filtered air. Manufacturing environments are often very dusty places. Systems with cooling fans that either draw or push air through a cabinet to cool actually wind up coating all components with dust over time in uncontrolled environments. Depending on the thickness and type of dust, overheating and/or short circuits can happen. Air conditioning feeds to data centers should have the dust removed and ensure that humidity is at proper levels.

When planning for cooling systems in a data center, take power failure into consideration. Frequently, groups plan to keep the equipment and lights on, but overlook cooling. In the event of power failure, air-conditioning (or whatever the cooling system is) may very well be needed to protect sensitive electronics.

Lastly, don’t guess on the requirements. Consultants and vendors have formulas to determine the size of cooling systems based on current needs as well as future growth.

Conditioned Power

IT systems need stable, reliable power. It is not cost-effective to buy dozens of good UPSes. All too often, IT buys dozens of cheap systems to protect distributed systems. It is more economical to buy several good systems that can protect dozens, if not hundreds, of devices than buying one-off power fixes.

First, lightning strikes need to be dealt with. Second, fluctuations in voltage, harmonics, EMI/RFI and other problems need to be removed. Third, in the event of an outage, there must be a solution that allows for the systems to stay on-line the necessary amount of time for a controlled shutdown and this may mean UPSes or a mixture of UPSes and generators. These types of solutions are very economical when applied to a large collection of systems, but less so when applied to fewer and fewer systems.

Moreover, all these systems need maintenance and the fewer the better. Monitoring and swapping batteries in a handful of enterprise UPSes is better than trying to keep track of dozens of small UPSes spread all over. In the end, business needs and associated risks must drive the solution and thus the investment. IT must architect with centralization and/or consolidation in mind.

Fire Management

The best way to deal with a fire in a data center is when it is just starting. There are fire detection systems that are so sensitive they can detect the increase in particulates and temperature as a group moves through a data center. These sensors go far beyond the traditional smoke detectors and can send alerts via the network as well as backup means. These systems can be deployed in a controlled environment such as a data center with much success. The whole idea is to detect a problem and react before the fire becomes significant and is manageable.

By layering early detection with a corrective control, namely suppression, the risks of damage from fire can be further mitigated. Take the time to investigate fire suppression technologies -- including Inergen, which is a combination of gasses, and Sapphire, which is a very interesting liquid that changes to a gas with very little additional energy -- that can put out fires without damaging electronics and leaving particulates. There are many options and the trick is to pick the one best suited to your needs and expert guidance should be sought.

Using the threat of fire as an example, always think about how to compensate in layers. How can the risk be prevented? How can it be detected early on when the impact is minimal? How can the problem be corrected? Most times, a layered approach is more effective and reliable than any single method.

Water

For some data centers, flooding is a very real concern. In dedicated data centers, it is possible to elevate equipment, re-route water pipes, disconnect water sprinklers and use alternative fire suppression systems, protect key wiring, install sump pumps, alarms and so on all aimed at reducing the risk of damage due to water in a particular location.

Summary

Environmental issues need to be addressed to ensure availability. The mixture of elements to consider depends on the data center, geographic location and so on. It is not the intent of this article to argue for total centralization, but rather pragmatic consolidation. Some systems must be located relatively near the user community and need to be protected regardless. In all cases, a balance must be struck between costs, risks and benefits.

In the end, its all about meeting the needs of the business. Today, when IT systems fail for whatever reason, it’s not just old-fashioned report printing that stops -- it is the business that stops.

George Spafford is an IT consultant and a long-time IT professional. He focuses on compliance, management and process improvement.

Courtesy: Sysman Computers P. Ltd, Mumbai, email dated August 1, 2005

Go Top

Fraud Investigation - Investigating Source of Malicious Chain E-Mails

By Sanjay Sharma. CFE, CPO, Country Security Manager DHL Express India

The Computer Crime and Abuse Report (India) 2001 – 2002, published in 2003 by the Asian School of Cyber Laws, has come out with startling data related to computer crimes. The report analyses 6,266 incidents of computer crime and abuse that affected 600 organizations spanning IT, manufacturing financial services, education, telecommunications, healthcare and other services sectors in India during this period. Some of the findings of the report:

• A disgruntled former employee is more likely to commit a computer crime than a business rival.

• Overall, 21% of the reported incidents were traced back to employees, while 31% were traced to former employees of the victim organization.

• Another interesting fact is that more than half the incidents (52%) are attributable to employees (current as well as former).

• 60% of the incidents of email abuse related to obscene emails. Out of these obscene emails, almost all (97%) were sent to women employees. 25% of the incidents of email abuse related to threatening emails. Most of these were targeted towards the top management of the victim organization. The balance incidents (15%) related to emails that sought to defame employees of the victim organization.

• Over 60% of the victims did not report the incidents because of the fear of negative publicity. 23% did not know whether the police in their area were technically equipped to handle computer crime cases.

The Indian Law

The passage of the Information Technology Act, 2000 followed by the subsequent amendment to the Indian Penal Code and the Evidence Act, amongst other laws, have paved the way for stringent penalties for computer crimes.

Sending pornographic or obscene emails are punishable under Section 67 of the IT Act. An offence under this section is punishable on first conviction with imprisonment for a term, which may extend to five years and with fine, which may extend to one lakh rupees.

In the event of a second or subsequent conviction, the recommended punishment is imprisonment for a term, which may extend to ten years and also with fine which may extend to two lakh rupees.

Emails that are defamatory in nature are punishable under section 500 of the Indian Penal Code (IPC), which recommends an imprisonment of up to two years or a fine or both.

Threatening emails are punishable under the provisions of IPC pertaining to criminal intimidation, insult and annoyance.

However, the number of computer crime and abuse incidents that are not reported to the law enforcement authorities are staggering. At a low cost and with virtually no technical know how, anyone can misuse the Internet to send malicious anonymous e-mails.

Case: This case (based on a real investigation with details changed) illustrates how e-mail fraud has become potential security vulnerability.

The senior management as well as various employees of a company had been getting anonymous chain e-mails since long. The number of anonymous communications sent was very large and they ranged in importance all the way from obscene jokes to threats. The sender of these mails had also indulged in character assassination of the employees mainly at a particular facility of the company.

When I was asked to investigate the source of these malicious e- mails the situation had worsened as these mails were being circulated to various outside agencies including newspapers, prompting the company to issue denials. This put the jobs of many senior managers on the firing line and the credibility and reputation of the company was in jeopardy. Apart from this much mental distress was caused by these communications and wholly unwarranted accusations had led to much sorrow.

Investigation Methodology

This type of forensic tracing of e-mail is similar to traditional investigate on work. By verifying each point through which e-mail passed the investigator can work step by step back to the originating computer and the perpetrator.

Anonymous chain e-mails receive little attention initially, with most of them being deleted. But most of the times such communications are important links in chains of evidence pointing to the perpetrator. It is usually advisable to preserve them for a while, as such mails may continue their unwelcome visits and in an investigation it is important to have in hand all e-mails, especially the first. In this instance, many of the mails were deleted on account of their obscene nature. Therefore, the first step was to collect as many such mails as were available with people who had preserved it.

I decided to follow the following approach in this particular anonymous mail investigation. The first of these steps is a study of the content in the message and the second step is a careful listing of all the probable and possible writers, and the third step is to take the help of outside agencies such as Documents experts, ISP’s, law enforcement.

Examining content and establishing the motive

A hard copy of most of these mails was collected to determine the purpose and motive from the content of the mails themselves. The idea being to determine the provocation for the mails with a view to narrow the field of possible writers. Caution may be exercised, as the real provocation may be unknown for the reason that it may be some trivial thing that is unknown to the victims of the mails.

The most common motives of such mails which indulge in character assassination are: seeking of money by blackmailing, unrequited love, jealousy, envy, actual or imagined slight, slow promotion, discharge or insane desire to cause suffering. Finally it must be said that such mails sometimes contain truth telling warnings and accusations of actual moral or other lapses on the part of recipient.

Apart from analyzing the contents of the mail, interviews were also conducted with most of the connected people with a view to calmly, thoroughly, and frankly consider all the various motives so that the identity of the sender could be recalled or correctly guessed. It is a good practice to write down names of all possible writers. If the whole subject and all the circumstances are carefully considered the trail of the writer often can be found. This is true because from the very nature of the problem, the number of possible writers of the series of anonymous mails cannot be very great. Obviously it was not necessary to consider every employee in the subject company as a possible writer. The motive, the opportunity, the location and the extent of circle of acquaintance, all limit the number of possible writers. Then the facts and information in the mails themselves narrowed the circle until the group became very small.

The initial mails suggested that the writer was out to discredit a woman employee and alleged that she was having a relationship with a senior manager. The intention seemed to be to persuade another male colleague not to marry this woman employee; alternatively, if married the marriage would result in a disaster. These mails made the male colleague an obvious victim. These mails also spoke of certain incidents at another location, when purportedly this manager was seen with the woman employee. One of the mails specifically mentioned an ex- employee. Many of these mails had Hindi phrases written in English.

The educational and cultural qualities shown in the mails also restricted the number of possible writers.

One of the mails stated “ I received a mail fro this woman employee’s brother which is quite surprising, how did he get my email address and he is asking for those photographs”. According to this woman employee her brother had written to this ex- employee.

The mails seemed to have increased in frequency and vehemence after four months of the first mail, maybe until then there was no indication that they had attracted positive attention and caused pain, annoyance or fear. These mails although sticking to the same subject, the allegation of a relationship between manager and woman employee became more pronounced. In fact it was justified by raising issues such as her promotion, the company’s HR policy, incidents at other locations and that there were witnesses. Even male colleagues depiction as a victim was highlighted. However the basic content remained the same as in the earlier mails.

After the above, during the end of the year, the mails for the first time started going to senior management of the company and increased in vehemence, probably the author realizing that some attention was being given but was not enough. The basic content of the mails remained the same, only the corruption angle was added. Also around this time the conduct of the mail colleague became unusual. After few such mails the author openly accused the male colleague of writing these mails.

These mails had been coming in such large numbers and over a very long period of time, it is usual that the disguise employed is apt to be partly disregarded, or almost entirely forgotten, after all these mails which may be prepared with great care and initially very effectively disguised.

It also transpired during investigation that when the services of another employee were terminated, his PC was examined and reportedly it contained lot of pornographic material apart from mails. Unfortunately none of these had been preserved. However one such mail copy could be retrieved which was written by this person to another friend. The contents of this mail not only mentioned the incident but also talked about similar circumstances. Reportedly, the person to whom this mail was sent happened to be a close friend the ex- employee.

A typed letter was also received at the office on the similar lines as the mails. The envelope contained a word in the handwriting of the sender that would have been good evidence to track the sender. Unfortunately the envelope had not been preserved.

As mentioned earlier, after analyzing various circumstances, contents of the mails and interviews with employees collectively led me to list the names of three possible suspects. It is also important to point out that in a large number of such cases the actual writer is one of a number who has received one or more of such mails and is supposed to be one of several victims of the work of someone else. This is a natural defense and is often employed.

Forensic analysis – next step was to use forensic tools to connect the perpetrator with these mails. Before I proceed it is necessary to understand the various web related terms such as domain names, IP (Internet Protocol) addresses, mail headers and computer logs. These terms identify distinct concepts and provide information sources about those entities, which are likely to be used by the originator of malicious e-mails.

Domain name - A domain name is a word or series of words that is registered to a particular person or organization that identifies that entities web presence. IP addresses are the Internets underlying system of numerical addresses. With www.yahoo.com, the domain name can be inferred simply by dropping www prefix, which leaves yahoo.com.

URLs identify specific resources within a particular domain name (for example, www.cnn.com/library/001465html refers to a particular document in www.cnn.com. In order to locate the resource web browsers must resolve the URL to a numerical address. This numerical address is the IP address.

To register a domain name, one must pay a fee to a registrar. One must provide the company with certain information, such as the name of the registered owner, as well as contact information including name, address, and phone numbers.

An initial inquiry about a particular domain name can be done at the web site of any registrar by using a service called Whois. The Whois search result will provide some of the registration information available including the name of a specific registrar that issued the domain name. This contact information gathered from the registry gives a lead to follow.

IP address - Whenever you get online your computer is assigned an IP address. IP address is the Internet Protocol (IP) address given to every computer connected to the Internet. An IP address is needed to route information much like a street address or PO box is needed to receive regular mail. . An IP address specifies a connection to the Internet and identifies the computer that is using the connection. IP addresses are four numbers between 0 and 255, separated by periods.

Several freely available tools can be used to resolve a URL into its IP address. The most frequently used are the nslookup, Whois, ARIN/APIC. Given a URL as input the utility returns the IP addresses associated with the URL as output. With the IP address in hand an investigator can query IP registries for publicly available information. The purpose of the WHOIS data is to identify the entity (person or company) to which blocks of addresses have been delegated. It is essentially an ISP Map, rather than an IP Address Map There are services available which searches location of any IP address. The service searches the City, State, Country, longitude & latitude of where the computer with that particular IP address is located and can also Includes name and address of the owner of IP address.

As a result of these searches the ISP and the geographical location of the computer was traced. The investigation was able to trace companies that had leased these lines from the Internet service provider. It also transpired that this ex- employee had worked with all the companies and was currently working for one of them.

Mail Headers - Most tracing of external e-mail starts at the receiving PC with emails Internet message header information. A message header is text at the top of an e-mail in the “from” line, while in the “received” lines, the header lists every point the e-mail passed through on its journey, along with the date and time. It’s like having each post office that handles a letter print an identity, date and time on the envelope.

The “Received” line is the most important line in a header. They form a list of all sites through which the message traveled in order to reach you. Received lines are read from bottom to top. That is, the first received line is your own system or mail server and the last received line is where the mail originated. Many mailers have the IP address of the sending system also added.

The message header provides an audit trail of where e-mail has been. Finding the individual who sent the email is a matter of walking back up the audit trail point by point gathering evidence, that the message passed each point.

Logs - Forensic tracing of e-mail relies on computer logs. A computer log is a record of each e-mail message that passes through a computer in a network. Ideally I needed prove that e-mail traveled through a machine by looking up the message ID on a log of email transaction together with date, time and the address that was recorded.

Finding the person behind the computer becomes a matter of determining who used the machine at the time the message was sent.

Limits of private investigations – the ISP’s vary in their willingness to help private investigations. Most ISP’s refuse to give up logs without a court order or a police case. For private investigators without the backing of a police case, getting a civil court order may be difficult or impossible. To overcome this impediment to progress, investigators can work with law enforcement. If law enforcement officers contact the ISP and inform them that a certain user is being investigated, the ISP is obligated to preserve and provide any information they would normally have.

The case was registered with the local law enforcement and all the material collected thus far was handed over to them. Continuing the investigation, the police obtained the computer log from the ISP’s corresponding to the date and time on the mail headers and tracked the computer from which most of the mails were sent. This turned out to be a cyber café in the locality of the ex-employee. Who used the computer was determined from the sign on logs maintained at the cyber cafe.

Whether the case could be proved in the court is not particularly relevant, since as a result of this enquiry these malicious e-mails permanently stopped visiting the company and have not revisited since then, fulfilling the objective of the company. Few of the employees also had to exit the company. However the hardship and mental distress caused by cyber defamation to the innocent victims is irreparable. One of the important lesson learnt is that the one first suspected of writing anonymous mails is not the actual writer and frequently the actual writer is one who for some time wholly escapes even suspicion. Hence it is particularly important in such cases to protect the innocent from unwarranted accusations and not jump to conclusions with preliminary facts.

Go Top