81 results found for ""

  • Short Service Commission in Indian Army

    In Indian army, Short Service Commission (SSC) recruitment grants qualified candidates the option of joining Indian Army as a "Short Service Commissioned Officer" for a duration of 10/14 years. At the end 10 years you have 3 options. Convert a Short Service Commission to Permanent Commission (PC). You can opt for 4 years extension. You can resign at any time during this period of 4 yrs extension. Or simply opt out! Short Service Commission empowers you with analytical thinking, planning skills, administrative and organizational abilities. Qualities which will make you an invaluable asset for any organization that you join after the Army and there are opportunities to side step to an alternate career for which Army will also help you. Officers Training Academy, Chennai What happens after SSC selection? Once selected for Short Service Commission, you go to the Officers Training Academy at Chennai. Selected candidates will be detailed for training at Officers Training Academy, Chennai according to your position in the final order of merit, up to the number of vacancies available at the time. SSC Training Duration of training is 49 weeks (approximately 11 months) training at OTA, Chennai, on successful completion of which candidates will be granted commission in the rank of Lieutenant. Ante date seniority will be granted only to Technical Graduates joining through Technical Entry. Training at OTA is at Govt. expense, all candidates who successfully complete Pre-Commission training at Officers Training Academy, Chennai will be awarded "Post Graduate Diploma in Defence Management and Strategic Studies” by University of Madras. OTA training provides you with opportunities to broaden your perspective and widen your horizon. Whether it's skeet-shooting, golf, riding or angling, at OTA you can do it all. Officers granted Short Service Commission are liable to serve anywhere in India and abroad. The main entries for Men & Women both are as under: 1. Short Service Commission (Non Technical) Men & Women Officers recruitment through this entry is advertised through the Employment News and all important National and Regional Newspapers / dailies in the months of Jul and Nov. Mode of Application is Online only through official website of UPSC. Selection for short service commission is through a written exam conducted by the UPSC twice every year in the month Sept. and Feb, followed by an interview by the Service Selection Board (SSB). Courses are conducted at OTA, Chennai twice every year in the Months of April and October. 2. Short Service Commission (Technical) Men & Women Officers Recruitment through this entry is advertised through the Employment News and All important National and Regional Newspapers / dailies in the months of January and July. Mode of Application is Online only through official website of Recruitment Directorate. Selection for short service commission is through the Services Selection Board (SSB) and there is no written exam for this entry as on date. The candidates are shortlisted for SSB interview based on the basis of cut off percentage applied on marks obtained by the candidates’ up to 6th Semester. Courses are conducted at OTA, Chennai twice every year in the Months of April and October. 3. Short Service Commission (NCC) Men & Women Officers Recruitment through this entry is advertised through the Employment News and All important National and Regional Newspapers / dailies in the months of Jun and Dec. If you have done NCC Senior Division (Army) and obtained 'C' certificate with minimum 'B' grade, you can apply through your NCC Battalion from where NCC 'C' certificate has been issued to you. 4. Short Service Commission (JAG) Men & Women To apply for JAG Entry, you have to be a Law Graduate with minimum 55% aggregate marks in LLB Degree (Three years professional after graduation or five years after 10+2 examination). The candidates should be eligible for registration with Bar Council of India/State. Candidate should be from a College/University recognized by Bar Council of India. Officers Recruitment through this entry is advertised through the Employment News and All important National and Regional Newspapers / dailies in the months of June and December. Thank you. If you have any question please don't hesitate to ask in SSB group discussion forum or simply comment below. #SSBTips #IndianArmy

  • Apache Kafka and Zookeeper Installation & Sample Pub-Sub Model

    There are many technologies available today which provides real time data ingestion (refer my previous blog). Apache Kafka is one of my favorites because of its distributed streaming platform. What exactly does that mean? Basically it can act as "publisher-subscriber" data streaming platform. It can process data streams as they occur. Lastly, it can store data stream in a fault-tolerant durable way. Kafka can run as a cluster on one or more servers that can span multiple data-centers. The Kafka cluster stores streams of records in categories called "topics". For instance, if I have data streams coming from Twitter (refer blog), you can name this topic as "Tweets". We will learn how to define these topics and how you can access pub-sub model. Kafka has basically four core APIs. Each record consists of a key, a value, and a timestamp. Producer API Consumer API Streams API & Connector API Whats the need of Zookeeper with Kafka? As each data stream in Kafka consist of a key, a value and a timestamp, we need someone to manage it's key-value pair and synchronicity. Zookeeper is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide configuration information, naming, providing synchronization service, and providing group services. Apache Kafka package installer comes with inbuilt Zookeeper but in production environment where we have multiple nodes, people usually install Zookeeper separately. I will explain you both the ways to run Kafka: one with inbuilt Zookeeper and another with separately installed Zookeeper. Kafka Installation Before we start Kafka installation. I hope you all have Java installed on your machine, if not please refer to my previous blog. Once you have successfully installed Java, go to the this link and download latest Kafka release. I have downloaded Scala 2.11 - Kafka_2.11-1.1.0.tgz (asc, sha512) to explain this blog. Once you download it on your local machine, move it to your Linux environment where you want to run Kafka. I use MobaXterm (open source tool) to transfer the file from my windows machine to Linux environment (Red Hat Linux client without GUI in this case). Navigate to the directory where you transferred .tgz file and untar the file. For instance I have kept it in /var folder: Now remove the .tgz file, not a necessary step but in order to save some space. cd /var rm /kafka_2.11-1.1.0.tgz Rename the folder for your convenience. mv kafka_2.11-1.1.0 kafka Zookeeper Installation Follow similar steps to download Zookeeper latest release from this link. Move the .gz file to your Linux machine (/var folder in my case) and perform the below steps: Run the below commands to untar the file and configure the .conf file. tar -xvf zookeeper-3.4.11.tar rm zookeeper-3.4.11.tar mv zookeeper-3.4.11 zookeeper cd zookeeper Now let's setup the configuration file. cd conf cp zoo_sample.cfg zoo.cfg Your configuration file will look something like below. You can change dataDir, if you don't want to depend upon your /tmp directory. If server reboots due to some infrastructure issues you might lose /tmp data, hence people usually don't rely on this. Or you can simply leave it as it is. At this point you are done with Zookeeper setup. Now we will start the Zookeeper as it is needed for Kafka. There are basically 2 ways to do this. Start Zookeeper First, like I said Kafka comes with inbuilt Zookeeper program (you can find Zookeeper files under /kafka/bin directory). So either you can start the Zookeeper which comes with Kafka or you can run Zookeeper separately. For this you can navigate to your Kafka directory and run the below command. cd /var/kafka bin/zookeeper-server-start.sh config/zookeeper.properties Or you can start Zookeeper which you just installed by running below command in /var/zookeeper directory. bin/zkServer.sh start You will get output something like this. Start Kafka Go to your Kafka directory and execute the below command: cd /var/kafka bin/kafka-server-start.sh config/server.properties You will find lots of events being generated and screen getting stuck at one point. Keep this terminal open and open a new terminal to verify if Zookeeper and Kafka services are running fine. Type the jps command to check active java process status. QuorunPeerMain basically shows our Zookeeper process 12050 & 11021 is our Kafka process. Process id might vary for you. Once you close that terminal Kafka service will stop. Another way is to run these services in background with "nohup", like; nohup bin/zookeeper-server-start.sh config/zookeeper.properties nohup bin/kafka-server-start.sh config/server.properties Create Kafka Topic As discussed earlier, Kafka cluster stores streams of records in categories called "topics". Lets create a topic called "Tweets". To do this run the below command in your Kafka directory. cd /var/kafka bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Tweets You will get a prompt saying - Created topic "Tweets". You can also see the list of topics by running below command. bin/kafka-topics.sh --list --zookeeper localhost:2181 Running Kafka Producer and Consumer Run the below command in order to start Producer API. You need to tell Kafka which topic you want to start and on which port. Check /config/server.properties in Kafka directory for details. For example I am running it for Tweets on port 9092: cd /var/kafka bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Tweets Now open a "new terminal" and lets run the Consumer API. Make sure you are entering the correct topic name. Port will be same where your Zookeeper is running. bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Tweets --from-beginning Now go back to Producer API terminal and type anything, hit enter. Same message will be shown on your Consumer API like below; Producer Terminal Consumer Terminal Thank you!! If you have any question please write in comments section below. #DataIngestion #Kafka #Zookeeper #ApacheKafkaandZookeeperInstallation #kafkapubsubmodel Learn Apache Spark in 7 days, start today! Main menu: Spark Scala Tutorial 1. Apache Spark and Scala Installation 1.1 Spark installation on Windows​ 1.2 Spark installation on Mac 2. Getting Familiar with Scala IDE 2.1 Hello World with Scala IDE​ 3. Spark data structure basics 3.1 Spark RDD Transformations and Actions example 4. Spark Shell 4.1 Starting Spark shell with SparkContext example​ 5. Reading data files in Spark 5.1 SparkContext Parallelize and read textFile method 5.2 Loading JSON file using Spark Scala 5.3 Loading TEXT file using Spark Scala 5.4 How to convert RDD to dataframe? 6. Writing data files in Spark ​6.1 How to write single CSV file in Spark 7. Spark streaming 7.1 Word count example Scala 7.2 Analyzing Twitter texts 8. Sample Big Data Architecture with Apache Spark 9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science? 10. Spark Interview Questions and Answers

  • Apache Spark Tutorial Scala: A Beginners Guide to Apache Spark Programming

    Learn Apache Spark: Tutorial for Beginners - This Apache Spark tutorial documentation will introduce you to Apache Spark programming in Scala. You will learn about Scala programming, dataframe, RDD, Spark SQL, Spark Streaming with examples and finally prepare yourself for Spark interview questions and answers. What is Apache Spark? Apache Spark is an analytics engine for big data processing. It runs 100 times faster than Hadoop and gives you full freedom to process large scale data in real-time, run analytics and apply machine learning algorithms. Navigation menu ​ 1. Apache Spark and Scala Installation 1.1 Spark installation on Windows​ 1.2 Spark installation on Mac 2. Getting Familiar with Scala IDE 2.1 Hello World with Scala IDE​ 3. Spark data structure basics 3.1 Spark RDD Transformations and Actions example 4. Spark Shell 4.1 Starting Spark shell with SparkContext example​ 5. Reading data files in Spark 5.1 SparkContext Parallelize and read textFile method 5.2 Loading JSON file using Spark Scala 5.3 Loading TEXT file using Spark Scala 5.4 How to convert RDD to dataframe? 6. Writing data files in Spark ​6.1 How to write single CSV file in Spark 7. Spark streaming 7.1 Word count example Scala 7.2 Analyzing Twitter texts 8. Sample Big Data Architecture with Apache Spark 9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science? 10. Spark Interview Questions and Answers Next: Apache Spark Installation (Windows | Mac)

  • "Hello World" with Scala IDE

    Main menu: Spark Scala Tutorial In this Apache Spark Scala tutorial you will learn how to create, "Hello World" Scala application with Eclipse Scala IDE. Scala application, project, package, objects, run configuration and debug the application. Motive is to get you guys familiar with Scala IDE. I assume that you have already installed Eclipse Scala IDE, if not please refer my previous post for installation (Windows | Mac users). Now, open Eclipse Scala IDE. 1. Create a new Scala project "hellooWorld" Go to File → New → Project and enter hellooWorld in project name field and click finish. 2. Create a new Scala Package "hellooWorld" Right click on the hellooWorld project in the Package Explorer panel → New → Package and enter name hellooWorld and finish. 3. Create a Scala object "hello": Expand the hellooWorld project tree and right click on the hellooWorld package → New → Scala Object → enter hello in the Object name field and press finish. 4. Write the program to print Hello World message. package hellooWorld object hello { def main (args: Array[String]) { println("Hello World") } } 5. Create a Run configuration for the Scala application: Right click on hello.scala in package explorer → Run As → Scala Application. Select the first matching item, the HellooWorld class and click the Ok. OR you can also define configuration manually and run it. Just mention the project and class name. That's it. You can see the output Hello World in output console. That's all guys! If you face any problem while running the program please mention in the comments section below. Thank you. Next: Spark Shell Navigation menu ​ 1. Apache Spark and Scala Installation 1.1 Spark installation on Windows​ 1.2 Spark installation on Mac 2. Getting Familiar with Scala IDE 2.1 Hello World with Scala IDE​ 3. Spark data structure basics 3.1 Spark RDD Transformations and Actions example 4. Spark Shell 4.1 Starting Spark shell with SparkContext example​ 5. Reading data files in Spark 5.1 SparkContext Parallelize and read textFile method 5.2 Loading JSON file using Spark Scala 5.3 Loading TEXT file using Spark Scala 5.4 How to convert RDD to dataframe? 6. Writing data files in Spark ​6.1 How to write single CSV file in Spark 7. Spark streaming 7.1 Word count example Scala 7.2 Analyzing Twitter texts 8. Sample Big Data Architecture with Apache Spark 9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science? 10. Spark Interview Questions and Answers

  • What programming language is famous amongst Russians?

    According to Google Trends it seems like Javascript. However, I didn’t except Python as least popular language amongst Russians. First of all, it’s little difficult to cover all the programming languages used by Russians (or used in Russia). But we can find what are the most popular programming languages in Russia and then drill down geographically. Javascript, Java, Python, PHP and C++ are the most popular languages according to Google Trends. I didn’t include Swift, Ruby, Rust, Scala, PERL etc because they are comparatively less popular to show on charts. Geographically, it looks like Javascript is most popular programming language across Russia. Did you notice the blue color in above chart? I didn’t expect this for Python. Interesting. Drilling down just for Javascript “Saint Petersburg” is the most popular location with Javascript in Russia. I haven’t heard this place before, sorry my geography is not very good.

  • Tangy Creamy Tomato Carrot Soup Recipe

    When it's cold outside, nothing hits the spot quite like a creamy tomato carrot soup. This healthy, quick and easy recipe makes use of real cream, for a thick and satisfying soup with fabulous flavor that's ready in less than an hour. What could be better for dinner that a nice hot bowl of soup along with your favorite salad. Well this is what I was craving for one evening and turned out I had no vegetables in my refrigerator except for 3 carrots and few tomatoes. So a mashup of the left over ingredients in my fridge turned into a nice soothing bowl of carrot tomato soup topped with croutons and dollops of cottage cheese (current favorite at my place). Below are the details of the recipe. Do try it out and I would love to hear how it turned out. Let me know if anyone comes up with interesting variations as well. Preparation time: 30-40 min, Serves: 3-4 Ingredients 3 Carrots 2 Tomatoes 1/2 inch Ginger 3-4 Garlic Cloves 1 Onion 2-3 Green Chilies (as per spice preference) 1-2 tsp of cottage cheese ( as per preference) Salt to taste Pepper Oregano/Cilantro/Parsley ( as per preference) Heavy cream ( optional) Preparation Steps Heat Oil in a pan. Add chopped onions, garlic, ginger, green chilies. Saute till the onions are translucent. Next, add chopped carrots and saute for another 3 minutes. Next add the chopped tomatoes and continue to saute till the tomatoes are mushy and carrots are soft. I prefer to saute till the veggies are nicely roasted and the tomatoes have dissolved. Now blend this to a smooth paste. Pour the paste into the same pan and add 2 cups of hot water while stirring continuously. Add Salt per taste. Cook till it boils rapidly. Adjust the consistency as per your preference. Finally add salt to taste , fresh cracked pepper and some oregano or any herb of your choice. Add 2 teaspoons of cottage cheese to each serving and top it with some croutons. Hope you all like this recipe and can enjoy it on a cozy winter evening!

  • Kafka Producer and Consumer example (in Java)

    In this Kafka pub sub example you will learn, Kafka producer components (producer api, serializer and partition strategy) Kafka producer architecture Kafka producer send method (fire and forget, sync and async types) Kafka producer config (connection properties) example Kafka producer example Kafka consumer example Prerequisite - refer my previous post on Apache Kafka Overview (Windows). Apache Kafka is one of the client for Kafka broker. It publishes message to kafka topic. Messages are serialized before transferring it over the network. Kafka Producer Components Producers API Kafka provides a collection of producer APIs to publish the message to the topic. The messaging can be optimize by setting different parameters. Developer can decide to publish the message on a particular partition by providing custom key for the message. Serializer Serializer serializes the message to pass over the network. Default or custom serializer can be set by developer to serialize the message. Below are the String Serializer. value.serializer=org.apache.kafka.common.serialization.StringSerializer key.serializer=org.apache.kafka.common.serialization.StringSerializer The serialize are set for the value for the key and value both. Partitioner This component apply the hashing algorithm and finds the partition for the message, if keys are provided. If Key for the message is not provided by developer then it uses the round-robin algorithm to assign the the topic for the message. Kafka Producer Send Methods Fire and Forget Producer does not care for the message arrives at destination or not. ProducerRecord data = new ProducerRecord ("topic", key, message ); producer.send(data); Synchronous Send Send() method returns future object and developer can use the get() method on future object to know the status of message. ProducerRecord data = new ProducerRecord ("topic", key, message ); producer.send(data).get(); Asynchronous Send Developers can use the send() with a callback function which is called once broker send the response back to the producer. TestCallback callback = new TestCallback(); ProducerRecord data = new ProducerRecord ("topic", key, message ); producer.send(data, callback); private static class TestCallback implements Callback { public void onCompletion(RecordMetadata recordMetadata, Exception e) { if (e != null) { System.out.println("Error while producing message to topic :" + recordMetadata); e.printStackTrace(); } else { String message = String.format("sent message to topic:%s partition:%s offset:%s", recordMetadata.topic(), recordMetadata.partition(), recordMetadata.offset()); System.out.println(message); } } } Producer Configuration bootstrap.servers=localhost:9092 acks=all ProducerConfig.RETRIES_CONFIG=0 value.serializer=org.apache.kafka.common.serialization.StringSerializer key.serializer=org.apache.kafka.common.serialization.StringSerializer retries=2 batch.size=32768 linger.ms=5 buffer.memory=33554432 max.block.ms=60000 Kafka Producer Example Step-1: Start Zookeeper Step-2: Start Kafka broker and create a topic TEST.TOPIC Step-3: Create a Java project Step-4: Create a properties file - kconnection.properties bootstrap.servers=localhost:9092 acks=all ProducerConfig.RETRIES_CONFIG=0 value.serializer=org.apache.kafka.common.serialization.StringSerializer key.serializer=org.apache.kafka.common.serialization.StringSerializer retries=2 TOPIC_NAME=TEST.TOPIC batch.size=32768 linger.ms=5 buffer.memory=33554432 max.block.ms=60000 Step-5: KafkaConnection.java package com.demo.twitter.util; import java.io.InputStream; import java.util.HashMap; import java.util.Map; import java.util.Properties; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.Producer; public class KafkaConnection { static Properties props=null; private static Properties loadConPropsFromClasspath() throws Exception { if(props==null){ InputStream stream = KafkaConnection.class.getResourceAsStream("kconnection.properties"); props = new Properties(); props.load(stream); stream.close(); System.out.println("Configuration "+props); } return props; } public static Producer getKafkaConnection()throws Exception{ Properties props=loadConPropsFromClasspath(); Producer producer = new KafkaProducer(props); return producer; } public static String getTopicName() throws Exception{ if(props!=null){ return props.getProperty(IKafkaSourceConstant.TOPIC_NAME); }else{ return null; } } } Step-6: KafkaProducerClient.java package com.demo.client.producer; import org.apache.kafka.clients.producer.Callback; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.RecordMetadata; import com.demo.twitter.util.KafkaConnection; public class KafkaProducerClient { static KafkaProducer producer=null; /** * @param args * @throws Exception */ public static void main(String[] args) throws Exception { try{ KafkaProducerClient pclient=new KafkaProducerClient(); long i = 1; for (; i <=10 ; i++) { KafkaProducerClient.sendMessage(""+i, "Hello This is test message ..Demo"+i); } System.out.println("Number of message sent "+(i-1)); pclient.closeProducer(); }catch(Exception e){ e.printStackTrace(); } } public static void sendMessage(String key,String message)throws Exception{ try{ if(producer==null){ producer =(KafkaProducer) KafkaConnection.getKafkaConnection(); System.out.println("Kafka Connection created for topic.. demo"+KafkaConnection.getTopicName()); } TestCallback callback = new TestCallback(); long startTime=System.currentTimeMillis(); ProducerRecord data = new ProducerRecord(KafkaConnection.getTopicName(), key, message ); producer.send(data, callback); System.out.println("Total Time:---> "+Long.valueOf(System.currentTimeMillis()-startTime)); }catch(Exception e){ e.printStackTrace(); producer.close(); } } public void closeProducer(){ try{ producer.close(); }catch(Exception e){ e.printStackTrace(); } } private static class TestCallback implements Callback { public void onCompletion(RecordMetadata recordMetadata, Exception e) { if (e != null) { System.out.println("Error while producing message to topic :" + recordMetadata); e.printStackTrace(); } else { String message = String.format("sent message to topic:%s partition:%s offset:%s", recordMetadata.topic(), recordMetadata.partition(), recordMetadata.offset()); System.out.println(message); } } } } Apache Kafka Consumer Example Continue in the same project. Step-1: Create a properties file: kconsumer.properties with below contents bootstrap.servers=localhost:9092 acks=all ProducerConfig.RETRIES_CONFIG=0 value.deserializer=org.apache.kafka.common.serialization.StringDeserializer key.deserializer=org.apache.kafka.common.serialization.StringDeserializer retries=0 group.id=group1 TOPIC_NAME=TEST.TOPIC CONSUMER_TIMEOUT=1000 worker.thread.count=5 counsumer.count=3 Step-2: Create KafkaConsumerClient.java package com.demo.kafka.consumer; import java.io.InputStream; import java.util.Collections; import java.util.Properties; import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; import com.demo.twitter.util.KafkaConnection; public class KafkaConsumerClient { Properties props=null; KafkaConsumer consumer =null; public static void main(String[] args) { KafkaConsumerClient conClient=new KafkaConsumerClient(); try { conClient.subscribeMessage("kconsumer.properties"); } catch (Exception e) { e.printStackTrace(); } } public synchronized void subscribeMessage(String configPropsFile)throws Exception{ try{ //Common for below two approach if(consumer==null){ consumer =(KafkaConsumer) getKafkaConnection(configPropsFile); } consumer.subscribe(Collections.singletonList(getTopicName())); while (true) { ConsumerRecords records = consumer.poll(1000L); for (ConsumerRecord record : records) { System.out.printf("Received Message topic =%s, partition =%s, offset = %d, key = %s, value = %s\n", record.topic(), record.partition(), record.offset(), record.key(), record.value()); } consumer.commitSync(); } }catch(Exception e){ e.printStackTrace(); consumer.close(); } } public KafkaConsumer getKafkaConnection(String fileName)throws Exception{ if(props==null){ props=loadConPropsFromClasspath(fileName); System.out.println(props); } KafkaConsumer consumer = new KafkaConsumer(props); return consumer; } private Properties loadConPropsFromClasspath(String fileName) throws Exception { if(props==null){ InputStream stream = KafkaConnection.class.getResourceAsStream(fileName); props = new Properties(); props.load(stream); stream.close(); System.out.println("Configuration "+props); } return props; } public String getTopicName() throws Exception{ if(props!=null){ return props.getProperty("TOPIC_NAME"); }else{ return null; } } } Thank you. If you have any question please write in comments section below. [09/09/2019 10:38 PM CST - Reviewed by: PriSin]

  • A Day in the Life of a Computer Programmer

    As a computer programmer, my daily life is actually kind of weird. I did my undergrad in computer science and worked with Microsoft for 4 years. I'm self taught and have spent a far greater amount of hours learning how to code. I work on US based client projects. Due to different time zone, my hours are not fully standard. Usually I try to work 7–9 hours a day, however sometimes it can be as much as 10–12 hours. Here is my routine that I follow roughly everyday, 4:50 am : Alarm beeps, Snooze 1 .. 5:05 am : Snooze 2 .. 5:20 am : Snooze 3 .. 5:35 am : Snooze n, Rolling in bed .. 6:00 am : Semi awake 6:10 am : Check Facebook, Instagram, Whatsapp, Robinhood, 9gag 6:15 am : Wake-up, Workout (pushups, gym, meditation, yoga, I am lying) 6:30 am : Shower, Dress up 7:30 am : Leave for work, Mustang, Daily traffic, Pandora (free subscription) 8:15 am : Arrive at work, Parking, Swipe access card at the entrance 8:20 am : Checking email, Service Now 8:30 am : Offshore-onshore call, Status updates, Discuss target for the day 9:00 am : Breakfast (uncertain) 9:20 am : Code, debug, code, code, debug 10:00 am : Error, error, error, error 11:00 am : Coffee, "Smoking kills" 11:30 am : Code, code, debug, code 12:00 pm - 2:00 pm : Global variable "Lunch" 2:00 pm : Code, debug, code, debug 3:00 pm : Code, code, code, code 3:30 pm : Check on the entire team 4:30 pm : Wrap up, Leave for home, Drive, Traffic, Pandora 5:15 pm : Arrive home, Change dress, Chillax 5:45 pm : Jog for 4-5 miles, Shower 6:30 pm : Facebook, Youtube, News, Blogs 8:00 pm - 9:00 pm : Dinner, Eat 24, Cooking (rare element) 9:30 pm : Offshore-onshore call (sometimes free), Otherwise Netflix 10:00 pm : Netflix, Youtube, Chit-chat with girlfriend 10:30 pm - 11:00 pm : Shut down, Sleep 4:50 am : Alarm beeps, Snooze 1, 2, 3, n .. Thanks for reading, hit like and share if you enjoyed the post!

  • Just enough Scala for Spark

    In this tutorial you will learn just enough Scala for Spark, it's like a quick guide for Scala basics needed for Spark programming, Scala syntax and few Scala examples. Well, you can't become Scala expert in a day but after reading this post you will be able to write Spark programs. I will be using Spark-shell to run Scala commands, so no installation needed if you have Spark shell running on your machine. I would encourage you to run these commands side-by-side on your machine. Staring with printing "Hello World", for example, scala> println("Hello World") Hello World For comments you can use double forward slash, or for multiline comments you can use similar syntax like Java. For example, ignore the pipe character it's because I am using spark-shell. scala> // Hello Data Nebulae - This is single line comment scala> /* Hello World | This is multi-line comment | Data Nebulae | */ We have two types of variables in Scala - mutable and immutable variables. Mutable variable are defined with var keyword and immutable variable with val keyword. You can't re-assign immutable variables. For example, scala> val myNumber :Int = 7 myNumber: Int = 7 scala> var myWord :String = "Hello" myWord: String = Hello Because myNumber is immutable variable so re-assignment failed scala> myNumber = 10 :25: error: reassignment to val myNumber = 10 scala> myWord = "Dataneb" myWord: String = Dataneb You can specify datatype (Int, Double, Boolean, String) in front of variable name, if not Scala compiler will automatically assign the type (called variable type inference). scala> val myNumber :Int = 10 myNumber: Int = 10 scala> val myFlag = true myFlag: Boolean = true You can also assign variables in pairs, basically tuples similar to Python, scala> val (x, y) = (1, 5) x: Int = 1 y: Int = 5 keep going.. scala> var (x, y, z) = (1, 2, "Hello") x: Int = 1 y: Int = 2 z: String = Hello You can pass these variables to println function scala> println (x) 1 String interpolation, like you do in other languages s with double quotes; scala> println(s"Value of x is: $x") Value of x is: 1 Similar to other languages, you can create a range with step-size and print for each element. scala> (1 to 5).foreach(println) 1 2 3 4 5 scala> (5 to 1 by -1) res144: scala.collection.immutable.Range = Range(5, 4, 3, 2, 1) scala> (5 to 1 by -2) res145: scala.collection.immutable.Range = Range(5, 3, 1) Strings are surrounded by double quotes and characters with single quotes, for example, scala> "Hello Word" res111: String = Hello Word scala> 'H' res112: Char = H scala> :type ('H') Char You can apply similar methods like other languages, length, substring, replace etc, for example scala> "Hello World".length res113: Int = 11 scala> "Hello World".size res1: Int = 11 scala> "Hello World".toUpperCase res2: String = HELLO WORLD scala> "Hello World".contains('H') res5: Boolean = true scala> 19.toHexString res4: String = 13 scala> "Hello World".take(3) res114: String = Hel scala> "Hello World".drop(3) res115: String = lo World scala> "Hello World".substring(3,6) res116: String = "lo " scala> "Hello World".replace("H","3") res123: String = 3ello World scala> "Hello".map(x=>(x,1)) res7: scala.collection.immutable.IndexedSeq[(Char, Int)] = Vector((H,1), (e,1), (l,1), (l,1), (o,1)) Array, List, Map, Set - behaves similarly like other languages data structures scala> val a = Array("Hello", "World", "Scala", "Spark") a: Array[String] = Array(Hello, World, Scala, Spark) // you can access the elements with index positions scala> a(0) res159: String = Hello scala> (a(0),a(3)) res160: (String, String) = (Hello,Spark) Similarly List.. // List of Integers scala> val l = List(1, 2, 3, 4, 5) l: List[Int] = List(1, 2, 3, 4, 5) // List of strings scala> val strings = List("Hello", "World", "Dataneb", "Spark") strings: List[String] = List(Hello, World, Dataneb, Spark) // List of List scala> val listOfList = List(List(1,2,3), List(2,6,7), List(2,5,3)) listOfList: List[List[Int]] = List(List(1, 2, 3), List(2, 6, 7), List(2, 5, 3)) scala> val emptyList = List() emptyList: List[Nothing] = List() Similarly Map.. scala> val m = Map("one" -> 1, "two" -> 2 ) m: scala.collection.immutable.Map[String,Int] = Map(one -> 1, two -> 2) scala> m("two") res163: Int = 2 Set, returns boolean scala> val s = Set("Apple", "Orange", "Banana") s: scala.collection.immutable.Set[String] = Set(Apple, Orange, Banana) scala> s("Apple") res164: Boolean = true scala> s("Grapes") res165: Boolean = false Arithmetic operations + (adds), -(subtracts), *(multiply), / (divide), %(remainder) for example, scala> val (x, y) = (5, 8) x: Int = 5 y: Int = 8 scala> y%x res95: Int = 3 scala> res95 + 7 res110: Int = 10 scala> "Hello" + " World" res0: String = Hello World Relational operators ==, !=, <, >, >=, <= for example, scala> y > x res96: Boolean = true Logical operators &&, ||, ! for example, scala> !(y>x && x>y) res98: Boolean = true Assignment operators =, +=, %= etc for example, like other languages x+=y is same as x=x+y; scala> var (x, y) = (5, 8) x: Int = 5 y: Int = 8 scala> x+=y scala> x res102: Int = 13 Array of integers, with println and index scala> val a = Array(1, 2, 3) a: Array[Int] = Array(1, 2, 3) scala> println(s"Sum is ${a(0) + a(1) + a(2)}") Sum is 6 Defining function has also similar syntax (ignore | character), (Int, Int) => (Int, Int) means function takes two integer argument and returns two integers. scala> def squareOfNumbers(x: Int, y: Int): (Int,Int) = {(x*x, y*y) | // for multiline you have to use curly {} brackets | } squareOfNumbers: (x: Int, y: Int)(Int, Int) scala> squareOfNumbers(2,3) res131: (Int, Int) = (4,9) Lambda function, if you will not mention datatype, Scala compiler will automatically decide it (inference). scala> (x:Int) => x+x res132: Int => Int = Int => Int means function takes integer return integer scala> val func: Int => Int = x => x + x func: Int => Int = scala> func(3) res133: Int = 6 Takes two integer and returns one integer, first _ for first input and so on.. scala> val underscoreFunc: (Int, Int) => Int = (_ * 3 + _ * 2) underscoreFunc: (Int, Int) => Int = scala> underscoreFunc(7, 5) res134: Int = 31 if-else statements, for example scala> x res139: Int = 5 scala> if (x==5) { println("five") } // curly braces not needed here but in case of multiline program five scala> println(if (x==4) println("Hello") else "Bye") Bye Loops, while, do-while and for loop scala> while (i<5) {println(i); i+=1} 0 1 2 3 4 scala> do {println(i); i-=1} while (i>0) 5 4 3 2 1 In Scala, <- is like a generator, read like x in range(1 to 5) similar to Python scala> for (x <- 1 to 5) println(x) 1 2 3 4 5 Pattern matching, for example; scala> def patternMatch (x: Int) :String = x match { | case 1 => "one" | case 2 => "two" | case _ => "unknown" | } patternMatch: (x: Int)String scala> patternMatch(2) res40: String = two scala> patternMatch(4) res41: String = unknown Classes can be defined like other languages, for example scala> class Dog(breed: String){ | var br: String = breed | def bark = "Woof woof!" | private def eat(food: String) = | println(s"I am eating $food") | } defined class Dog scala> val myDog = new Dog("pitbull") myDog: Dog = Dog@62882596 scala> myDog.br res155: String = pitbull scala> myDog.bark res156: String = Woof woof! Case classes, these will be useful while performing data operations, for example scala> case class Order(orderNum: Int, orderItem: String) defined class Order scala> val myOrder = Order(123, "iPhone") myOrder: Order = Order(123,iPhone) scala> val anotherOrder = Order(124, "macBook") anotherOrder: Order = Order(124, macBook) scala> myOrder.orderItem res158: String = iPhone Exercise For Spark, most of the time you will be writing lambda functions. I have hardly seen complex functions written to transform the data in Spark. Spark has built-in transformations which takes care of complex transformations which you will learn soon. For practice, try these examples. Example 1: Area of Circle scala> def areaCircle(radius:Double ) : Double = 3.14 * radius * radius areaCircle: (radius: Double)Double scala> areaCircle(5) res17: Double = 78.5 Example 2: Sum of Squares of input numbers scala> def sumOfSquares(x: Int, y:Int) : Int = x*x + y*y sumOfSquares: (x: Int, y: Int)Int scala> sumOfSquares(2,3) res18: Int = 13 Example 3: Reverse the Sign of input number scala> def reverseTheSign (x: Int) : Int = if (x>0) -x else -x reverseTheSign: (x: Int)Int scala> reverseTheSign(-6) res23: Int = 6 scala> reverseTheSign(6) res24: Int = -6 Example 4: Factorial of a number (to explain recursion), note how we are calling func within func; scala> def factorial (x: Int) :Int = if (x==1) x else factorial(x-1)*x factorial: (x: Int)Int scala> factorial(4) res26: Int = 24 Example 5: Defining objects and methods, you can define it like (ignore |) scala> object MyObject{ | val MyVal = 1 | def MyMethod = "Hello" | } defined object MyObject scala> MyObject.MyMethod res30: String = Hello for example; scala> object Foo {val x = 1} defined object Foo scala> object Bar {val x = 2} defined object Bar scala> object fooBar { | val y = Bar.x | } defined object fooBar scala> fooBar.y res31: Int = 2 Example 6: Sum of Squares using Lambda or anonymous func scala> val z = (x:Int, y:Int) => x*x + y*y z: (Int, Int) => Int = scala> z(2,3) res34: Int = 13 Example 7: Filtering the list with anonymous func scala> List(1,2,3,4,5,6).filter(x => x % 2 == 0) res39: List[Int] = List(2, 4, 6) Example 8: For loops with yield scala> for (x <- 1 to 5) yield x res42: scala.collection.immutable.IndexedSeq[Int] = Vector(1, 2, 3, 4, 5) scala> for (x <- 1 to 3; y <- Array("Hello","World")) yield (x, y) res47: scala.collection.immutable.IndexedSeq[(Int, String)] = Vector((1,Hello), (1,World), (2,Hello), (2,World), (3,Hello), (3,World)) That's all guys! If you have any question please mention in the comments section below. Thank you! Next: Hello with Eclipse Scala IDE Navigation menu 1. Apache Spark and Scala Installation 1.1 Spark installation on Windows​ 1.2 Spark installation on Mac 2. Getting Familiar with Scala IDE 2.1 Hello World with Scala IDE​ 3. Spark data structure basics 3.1 Spark RDD Transformations and Actions example 4. Spark Shell 4.1 Starting Spark shell with SparkContext example​ 5. Reading data files in Spark 5.1 SparkContext Parallelize and read textFile method 5.2 Loading JSON file using Spark Scala 5.3 Loading TEXT file using Spark Scala 5.4 How to convert RDD to dataframe? 6. Writing data files in Spark ​6.1 How to write single CSV file in Spark 7. Spark streaming 7.1 Word count example Scala 7.2 Analyzing Twitter texts 8. Sample Big Data Architecture with Apache Spark 9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science? 10. Spark Interview Questions and Answers

  • How to pull data from Oracle IDCS (Identity Cloud Services) rest API

    Oracle IDCS has various rest apis which can be used to pull data and you can utilize it further for data analytics. Lets see how we can pull data using simple shell scripts. Step1: Create a parameter file "param.txt" which will contain Customer ID, Customer secret and organization url (all in new line). You can leave the environment name as it is. Please note below values are just dummy values to showcase how you need to create param file. Validate using postman if your keys are working properly before running the script. scripts> pwd /home/hadoop/scripts scripts> more param.txt CID= 61rgrjk5869bjrvrb9999rbre20 CSEC= 01rgt-atbt-4956-9e77-15rjb74756nr64 ORG= https://idcs-9bbrtj756bjer8gbk753gbvj8f7eh3.identity.oraclecloud.com ENV= QUAL Step2: At the same path create the script for pulling data in JSON format, brief description is given before each step. #Check if parameter file exists and its parameters #!/bin/bash [ -f ./param.txt ] && echo "Parameter file is present" || echo "Parameter file not found!! Create param.txt with CID,CSEC,ORG,ENV details." ENV=`head -8 ./param.txt | tail -1` [ -z "$ENV" ] && echo "Environment variable is empty" || echo "Environment variable looks good" case $ENV in DEV) echo "Welcome to DEV environment!" ;; QUAL) echo "Welcome to QUAL environment!" CID=`head -2 ./param.txt | tail -1` CSEC=`head -4 ./param.txt | tail -1` ORG=`head -6 ./param.txt | tail -1` sleep 1;; PL) echo "Welcome to ProdLike environment!" ;; PT) echo "Welcome to ProdTest environment!" ;; PROD) echo "Welcome to PROD environment!" ;; *) echo "Invalid environment selection!" ;; esac # Create basic_token base64, basic_token generated at https://www.base64encode.org/ basic_token=`echo -n $CID:$CSEC | base64 -w 0` # Function to regenerate token if token test fails or expires regenToken() { curl -X POST \ "$ORG/oauth2/v1/token" \ -H "Authorization: Basic $basic_token" \ -H "Cache-Control: no-cache" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials&scope=urn%3Aopc%3Aidm%3A__myscopes__" | awk -F[":"] '{print$2}' | awk -F[","] '{print$1}' | awk '{print substr($0, 2, length($0) - 2)}' > access_token.tmp echo "New token is generated.. access_token refreshed!!" } # Testing if token is valid, if invalid generate the new token access_token=`more access_token.tmp` tokenTest=`curl -X POST "$ORG/oauth2/v1/introspect" -H "Authorization: Basic $basic_token" -H "Cache-Control: no-cache" -H "Content-Type: application/x-www-form-urlencoded" -d token=$access_token | awk -F"," '{print$1}' | awk -F":" '{print$2}' | sed 's/[{}]//g'` if [ "$tokenTest" = "true" ]; then echo "Token is valid.."; else echo "Invalid token! Invoking func to pull new token.." regenToken access_token=`more access_token.tmp` fi # Remove all the previous files. Script can be modified later to pull delta records only rm -f auditevents.idcs* # Pull totalResults count totalResults=`curl -X GET "$ORG/admin/v1/AuditEvents?&count=0" -H "Authorization: Bearer $access_token" -H "Cache-Control: no-cache" | awk -F"\"totalResults\"\:" '{print$2}' | awk -F"," '{print$1}'` echo "Total number of qualified records: $totalResults" sleep 5 # Loop to pull the records with maximum limitation of 1000 items per page itemsPerPage=1000 startIndex=1 while [ $startIndex -le $totalResults ] do echo "startIndex: $startIndex" curl -X GET \ "$ORG/admin/v1/AuditEvents?&startIndex=$startIndex&count=$itemsPerPage" \ -H "Authorization: Bearer $access_token" \ -H "Cache-Control: no-cache" | awk -F"Resources" '{print$2}' | awk -F"startIndex" '{print$1}' | cut -c 4- | rev | cut -c 4- | rev > auditevents.idcs.json # Formatting the JSON output and writing into a file with date time PAT=]},{\"idcsCreatedBy REP_PAT=]}'\n'{\"idcsCreatedBy sed "s/$PAT/$REP_PAT/g" auditevents.idcs.json > auditevents.idcs.json_`date +"%Y%m%d_%H%M%S%N"` startIndex=`expr $startIndex + 1000` done # Remove access token temp file rm -f access_token.tmp #END See also - How to setup Postman client Thank you!! If you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on Google or Facebook. Refer the links below. Also click on "Subscribe" button on top right corner to stay updated with latest posts. Your opinion matters a lot please comment if you have any suggestion for me. #howto #PULL #DATA #oracle #idcs #howtopulldatafromoracleidcs #identitycloudservices

Home   |   Contact Us

©2020 by Data Nebulae