Boost Your Employability with Hadoop Skills

The rise of Big Data technology is driving the need for Hadoop experts in India.

In the information-driven world, as colossal volumes of data stream at a breakneck speed, data management remains a major challenge. Providentially, the emergence of a new breed of technologies is redefining the way we handle the explosion of data—one of them is Hadoop. These technologies require dexterous IT professionals who have an appetite to learn and take on challenging tasks in managing Big Data. So, if you are planning to take a plunge into the Big Data realm, acquiring Hadoop skills can enhance your chances of getting hired immediately.

Why is Hadoop important in the business landscape?

So what is Hadoop all about? As Hortonworks, a global business software company dealing with the development and distribution of Hadoop, puts it, “Apache Hadoop is an open source project governed by the Apache Software Foundation (ASF) that allows you to gain insight from massive amounts of structured and unstructured data, quickly and without significant investment.”

Over the years, organisations have realised that to enhance business productivity, they need to analyse and gain insight into Big Data. However, since 80 per cent of this data is ‘unstructured’, it must be formatted (or structured) so that one can mine the data effectively and then analyse it. Hadoop, as industry experts say, is the ‘nucleus’ platform for structuring Big Data, and plays a key role in making it effective for analytics. Explaining the importance of Hadoop in the business world, Pradip Kulkarni, distinguished engineer, Symantec India, says, “Hadoop is both an ecosystem that runs analytics applications, as well as a platform upon which individuals can develop such applications. These applications highlight business insights, help develop intelligence and cull out useful information that helps enterprises make informed decisions. This helps to improve the overall business and IT processes. Hadoop, being open source, is an attractive option as a foundation to build products, technologies and solutions that cater to markets in which the need is to process Big Data and find valuable insights from it.”

Talking about the Indian business angle, Vineet Tyagi, associate VP at Impetus, shares, “Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware. Hadoop is democratising the access to large scale data processing at reasonable price points for enterprises. Indian businesses now have access to the IT frameworks that can help their decision

making be transformed from reactive to predictive, and from ‘batch’ to ‘in time’.”

How strong is the demand in India?

Industry leaders believe that there is a steady latent demand for Hadoop experts as the Indian Big Data market grows. Explains Gaurav Makkar, technical director, NetApp India, “The way you manage your data is very important in terms of the financial top line and bottom line of a company. In India, large volumes of data are being generated, thanks to the speedy digitisation of data. It is important to manage the unstructured data and this is fuelling the need for Hadoop experts. Also, there is a demand for analytics players who like to seek consequential insights from their huge volumes of data.”

Different roles available

Individuals entering this field can take up niche roles, which include data scientists, Hadoop architects, Hadoop developers, Hadoop cluster administrators, Hadoop testers, support engineers for Hadoop clusters and Hadoop ELT testers. Let’s look at some of these in detail.

Data architects: Data architects are required to have some experience in creating data models, in data warehousing, analysing data and data migration. Experience as a database administrator and, preferably, with Hadoop, is highly desired.

Data scientists: A data scientist is a role that calls for a variety of data-driven skills. Data scientists gather data, analyse it, present the data visually, and use the data to make predictions/forecasts. Data scientists are currently in high demand, and the demand will likely continue to increase.

Hadoop developers: One of the more popular careers for people with NoSQL and Hadoop skills is software development. People with these skills can get ample freelance work or can launch their own start-ups if they have the entrepreneurial spirit. In addition to database management experience, you will also need programming skills.

What it takes to get hired

So, what do hiring managers look for while recruiting candidates in this arena? According to Kulkarni, “While hiring candidates, managers look for individuals who are comfortable working with the Hadoop ecosystem, have the ability to develop map-reduce applications, write parallel applications, and work with Java and C++. Individuals interested in working in this space must be able to develop object-oriented programs and design patterns, perform statistical analysis, be knowledgeable about data science, and also possess various data warehousing techniques and skills.”

Parameswara R Ganta, CTO and head, BigData Practice, VisualPath, feels that the demand for adept Hadoop professionals is good and is growing day by day. “Most of the companies are looking for experienced IT professionals, with prior experience in business intelligence, ETL/DWH, sound knowledge on Hadoop ecosystems like Pig/Hive with NoSQL databases like HBase, MongoDB, Cassendra, along with HDFS and MapReduce,” Ganta says.

According to Tyagi, at Impetus, they look for candidates with a good understanding of the Big Data technology landscape and a thorough knowledge of the parallel distributed computing paradigm. Hands-on experience with Java and writing MapReduce code is preferred.

Get the right skill sets

With the technology being so advanced, it is a challenge for organisations to find Hadoop experts who are well versed with the entire Hadoop stack and ecosystem. So what are the skills one must have to excel in this terrain? “Individuals are expected to have skill sets that will enable them to work on distributed and parallel computing, writing distributed applications, working on Java and C++, object-oriented concepts, statistics, analytics and warehousing skills,” says Kulkarni.

Ganta, too, comes up with a list of skill sets. According to him, one must have a core understanding of HDFS and MapReduce, NoSQL, Pig, Hive, Sqoop, Flume, Oozie, Linux, Hadoop on Windows, Hadoop on Java, Cloudera Apache Hadoop, Hortonworks Hadoop, IBM BigInsights, Sears MetaScale, Pivotal HD, Pentaho, Talend, KarmaSphere, TabLeau, QlikView, Micrpsoft Azure, Hadoop testing on the cloud, and more. Gaurav Makkar feels that professionals in India have an edge over their counterparts in other parts of the world as they are good in math and statistics.

Course curriculum and certification

Is the course curriculum on par with the industry standards? “I don’t think the academics have a Big Data course to cope up with this domain. There is an opportunity to do that,” feels Tyagi. Ganta feels that most of the training and courses are of the ‘boot camp’ variety, and not very hands-on. “Hadoop training camps should include physical Hadoop clusters with 50 to 1000 nodes, with huge volumes of structured and unstructured data sets, along with Hadoop commercial distributions like Cloudera, HortonWorks and MapR,” says Ganta.

Makkar feels that Tier-2 colleges do focus on the practical aspects of Hadoop and the awareness level is slowly catching up across India. “I am sure every IT company must have used Hadoop in a small way. If you talk about the awareness levels, it is progressively happening in the form of meet-ups,” he says. And what kinds of certifications are available in this domain? “Cloudera Certified Developer/Admin/Data Scientist, Hortonworks Certified Developer and Administrator, MapR Certified Administrator, NoSQL database certifications like HBase from Cloudera, MongoDB and Cassendra, etc, are some of the certifications,” says Ganta.

By Priyanka Sarkar

The author is a member of the editorial team. She loves to weave in and out the little nuances of life and scribble her thoughts and experiences in her personal blog.