Search this site

This post include Big Data Spark Interview Questions and Answers for experienced and beginners. If you are a beginner don't worry, answers are explained in detail. These are very frequently asked Data Engineer Interview Questions which will help you to crack big data job interview. What is Apache Spark? According to Spark documentation, Apache Spark is a fast and general-purpose in-memory cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. In simple terms, Spark is a distributed data processing engine which supports programming language like Java, Scala, Python and R. In core, Spark engine has four built-in libraries which supports Spark SQL, Machine Learning, Spark Streaming and GraphX. What is Apache Spark used for? Apache Spark is used for real time data processing. Implementing Extract, Transform, Load (ETL) processes. Implementing machine learning algorithms and create interactive dashboards for data analytics. Apache Spark is also used to store petabytes of data with data distributed over cluster with thousands of nodes. How does Apache Spark work? Spark uses master-slave architecture to distribute data across worker nodes and process them in parallel. Just like mapreduce, Spark has a central coordinator called driver and rest worker nodes as executors. Driver communicates with the executors to process the data. Why is Spark faster than Hadoop mapreduce? One of the drawbacks of Hadoop mapreduce is that it holds full data into HDFS after running each mapper and reducer job. This is very expensive because it consumes lot of disk I/O and network I/O. While in Spark, there are two processes transformations and actions. Spark doesn't write or hold the data in memory until an action is called. Thus, it reduces disk I/O and network I/O. Another innovation is in-memory caching where you can instruct Spark to hold input data in-memory so that program doesn't have to read data again from disk, thus reducing disk I/O. Is Hadoop required for spark? No, Hadoop file system is not required for Spark. However for better performance, Spark can use HDFS-YARN if required. Is Spark built on top of Hadoop? No. Spark is totally independent of Hadoop. What is Spark API? Apache Spark has basically three sets of APIs (Application Program Interface) - RDDs, Datasets and DataFrames that allow developers to access the data and run various functions across four different languages - Java, Scala, Python and R. What is Spark RDD? Resilient Distributed Datasets (RDDs) are basically an immutable collection of elements which is used as fundamental data structure in Apache Spark. These are logically partitioned data across thousands of nodes in your cluster that can be accessed and computed in parallel. RDD was the primary Spark API since Apache Spark foundation. Which are the methods to create RDD in spark? There are mainly two methods to create RDD. Parallelizing - sc.parallelize() Reference external dataset - sc.textFile() Read - Spark context parallelize and reference external dataset example. When would you use Spark RDD? RDDs are used for unstructured data like streams of media texts, when schema and columnar format of data is not mandatory requirement like accessing data by column name and any other tabular attributes. Secondly, RRDs are used when you want full control over physical distribution of data. What is SparkContext, SQLContext, SparkSession and SparkConf? SparkContext tells Spark driver application whether to access the cluster through a resource manager or to run locally in standalone mode. The resource manager can be YARN, or Spark's cluster manager. SparkConf stores configuration parameters that Spark driver application passes to SparkContext. These parameters define properties of Spark driver application which is used by Spark to allocate resources on the cluster. Such as the number, memory size and cores used by the executors running on the worker nodes. SQLContext is a class which is used to implement Spark SQL. You need to create SparkConf and SparkContext first in order to implement SQLContext. It is basically used for structured data when you want to implement schema and run SQL. All three - SparkContext, SparkConf and SQLContext are encapsulated within SparkSession. In newer version you can directly implement Spark SQL with SparkSession. What is Spark checkpointing? Spark checkpointing is a process that saves the actual intermediate RDD data to a reliable distributed file system. It's the process of saving intermediate stage of a RDD lineage. You can do it by calling checkpoint, RDD.checkpoint() while developing the Spark driver application. You need to set up checkpoint directory where Spark can store these intermediate RDDs by calling RDD.setCheckpointDir(). What is an action in Spark and what happens when it's executed? Action triggers execution of RDD lineage graph, loads original data from disk to create intermediate RDDs, performs all transformations and returns the final output to the Spark driver program or writes the data to file system (based on the type of action). According to Spark documentation, following are the list of actions. What is Spark Streaming? Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams. Reference: Apache Spark documentation Login to see more; What is difference between RDDs, DataFrame and dataset? Why is spark RDD immutable? Are spark DataFrames immutable? Are spark DataFrames distributed? What is Spark stage? How does SQL spark work? What is spark executor and how does it work? How will you improve Apache Spark performance? What is spark SQL Warehouse Dir? What is Spark shell? How would you open and close it? How will you clear the screen on spark shell? What is parallelize in spark? Does spark SQL require hive? What is Metastore in hive? What does repartition and coalesce do in spark? What is spark mapPartitions? What is difference between MAP and flatMap in spark? What is spark reduceByKey? What is lazy evaluation in spark? What is accumulator in spark? Can RDD be shared between SparkContexts? What happens if RDD partition is lost due to worker node failure? Which serialization libraries are supported in spark? What is cluster manager in spark? Questions? Feel free to write in comments section below. Thank you.

In this post, you will learn how to download and install SQL Server 2014 from an ISO image. I will also download the AdventureWorks sample database and load it into our SQL server 2014. Downloading ISO image 1. You can download the SQL Server from this link. But in case of the link changes, you can Google - "download SQL server enterprise evaluation" and it will show you the results below. Related: How to file B1/B2 visa for parents? 2. Open the link highlighted above and you will find all the latest SQL Servers available. I am installing SQL Server 2014 for this post. As its old version, it is very stable. 3. Click on the little plus sign and select the iso image to download. 4. Fill out these details and hit continue. 5. Select your platform based on your system properties whether it's 32-bit or 64-bit (Go to your control panel and look for system properties). The download will take approximately 10 minutes based on your network. It's a ~3 GB ISO file. Make sure you have enough space on your computer to perform this installation. Run ISO image 1. Once the download is complete, double-click on the ISO file and it will automatically mount on your drive as shown below. If it doesn't mount automatically you can use WinRAR to unzip the ISO file. 2. Open the drive to run the setup. SQL Server 2014 Installation 1. Setup will take some time (approx. a minute) to open. You will get this screen. 2. Go to the Installation tab as shown above and click on New SQL Server stand-alone installation. 3. Click next, you can select check for updates or leave it as default. 4. This step will take some time (approx. 5 mins) when it will install setup files. 5. Finally, you will get this screen. Click Next. 6. I have SQL Server Express edition running on my machine so we are seeing this option. We will just continue with the new installation default option. 7. Keep this edition as Evaluation and hit Next. 8. Check the box to accept license terms and hit Next. 9. Select SQL Server Feature Installation and hit Next. 10. Select all the features and hit Next as shown below. 11. Enter the name of the instance, I have named it "SQL2014" and hit Next. 12. Leave it as default and hit Next as shown below. 13. Just leave the default as Windows authentication for simplicity and add the current user. Windows will automatically add the current user when you click the Add current user button. Hit Next. Leave data directories and file streams as default for simplicity. Usually, in a production environment, we choose different drives for logs and data (like in case the server crashes). 14. Leave analysis services mode as default and add the current user. Hit Next. 15. Leave the default options as shown below. 16. Click add current user and hit Next. 17. Give Controller Name, I have named it "Dataneb" and hit Next. 18. Hit the Install button. 19. Installation will take some time (took 1 hour for me). So sit back and relax. Completion After 1-hour the installation was completed for me. This might take the same or less time for you depending upon your system configuration. Now, you can open the SQL Server configuration manager and see if SQL Server 2014 is running or not. I have SQL Server Express edition also running on this machine, ignore that. AdventureWorks Database I am loading the AdventureWorks database for example purposes here. 1. In order to load AdventureWorks, open SSMS (SQL Server Management Studio) and connect to SQL2014 which we just installed. 2. Now download a sample AdventureWorks 2014 database from the Microsoft site. You can go to this link, and you can choose anyone. I am downloading AdventureWorks 2014. 3. Once the download completes, move the AdventureWorks2014.bak file to the MSSQL server 2014 backup location. It will be C:\Program Files\Microsoft SQL Server\MSSQL12.SQL2014\MSSQL\Backup 4. Now open SSMS > right click on Databases > Restore Database > select Device > click on three Dots > Add 5. Add the AdventureWorks2014 backup file and hit OK. 6. Once the backup is loaded, you will be able to see AdventureWorks2014 loaded under your database panel, if not right click and refresh. That's all! If you have any questions please mention them in the comments section below. Next: How to create an SSIS package in Visual Studio 2017

One of the best method is to use Sieve of Eratosthenes 0 → False 1 → False 2 → True and so on.. Python program is shown below, def primes(n): flag = [True] * n flag[0] = flag[1] = False for (index, prime) in enumerate(flag): if prime: yield index for i in range(index*index, n, index): flag[i] = False Print the last item from generator, p = None for p in is_prime(1000000): pass print(p) >>> 999983 Regular way To make the program efficient you need to: Cross out 1 as it's not prime. Cross out all of the multiples of 2 if input number is odd. Cross out all of the multiples of 3 if input number is even. Most importantly, for number N, you only need to test until you get to a prime till √N. If a number N has a prime factor larger than √N , then it surely has a prime factor smaller than √N. So it's sufficient to search for prime factors in the range [1,√N]. If no prime factors exist in the range [1,√N], then N itself is prime and there is no need to continue searching beyond that range. You can do it like this, def is_prime(num): if num<2: return False else: return all(num%i for i in range(2, int(num ** 0.5)+1)) Checking if 1471 is prime, is_prime(1471) True Let's say you want to check all the prime numbers up-to 2 million, prime_list = [i for i in range(2000000) if is_prime(i)] I have seen similar type of problems in project Euler like sum of primes or total number of primes below (a very large number). If the number is small you can simply divide by all the numbers till (n-1) and check if it’s prime or not, but that program (shown below) performs very bad if the number is big. def is_prime(num): if num < 2: return False for i in range(2, num): if num%i==0: return False else: return True For example: is_prime(47) True >>> %timeit ("is_prime(47)") 10.9 ns ± 0.0779 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each) Above program performs very bad if the number is big. Better solution is, instead of dividing by all the numbers up-to (n-1), you just need to check if number is divisible till sqrt(num). def is_prime(x): '''Checking if number is prime, run until sqrt(number)''' sqrt = round(x ** 0.5) if x < 2: return False else: for i in range(2, sqrt + 1): if x%i == 0: return False else: return True OR, def is_prime(num): if num<2: return False else: return all(num%i for i in range(2, int(num ** 0.5)+1)) That’s it, now use list comprehension to get prime_list and print whatever you need. >>> prime_list=[i for i in range(3000000) if is_prime(i)] >>> print(f"Number of primes below 3 million: {len(prime_list)}") >>> print(f"Sum of primes below 3 million: {sum(prime_list)}") >>> print(prime_list) Number of primes below 3 million: 216816 Sum of primes below 3 million: 312471072265 [2, 3, 5, 7, 11, 13, 17, 19....... ]

Referenced from www.java.com (added few additional steps in order to make installation process more perfect) Java for Linux Platforms 1. First check if Java is already installed on your machine, Type java -version, or simply run this command on your terminal: which java 2. If Java is not present, your terminal will not understand this command and it will say command not found. 3. Now to install Java, change to the directory in which you want to install. Type: cd directory_path_name For example, to install the software in the /usr/java/ directory, Type: cd /usr/java/ 4. Download the tarball Java file from www.java.com (snippet shown above). ​ 5. Get 32 bit or 64 bit tarball file depending upon on your Linux machine configuration. 6. Move (sftp) the .tar.gz archive binary to the current directory /usr/java/. 7. Unpack the tarball and install Java tar zxvf jre-8u73-linux-i586.tar.gz In this example, it is installed in the /usr/java/jre1.8.0_73 directory. You can remove the version detail and rename the file according to your convenience. 8. Delete .tar.gz file if you want to save some disk space. 9. Setup .bashrc file. Type: vi ~/.bashrc and enter these two lines in the file; export JAVA_HOME=/usr/java/jre1.8.0_73 export PATH=$PATH:$JAVA_HOME/bin 12. Now, run source ~/.bashrc Now type command: java -version in order to see if java is successfully installed or not. If it's not running find bin directory where you unzipped Java and run: /path_to_your_Java/bin/java -version Java for RPM based Linux Platforms Become root by running su and entering the super-user password. Uninstall any earlier installations of the Java packages. rpm -e package_name Change to the directory in which you want to install. Type: cd directory_path_name For example, to install the software in the /usr/java/ directory, Type: cd /usr/java Install the package. rpm -ivh jre-8u73-linux-i586.rpm To upgrade a package, Type: rpm -Uvh jre-8u73-linux-i586.rpm Exit the root shell. No need to reboot. Delete the .rpm file if you want to save disk space. If you have any question, please write in comments section below. Thank you! #Javainstallation #OracleLinux #OEL #OL

J2 work permit (EAD) application processing time can take between 3-5 months. I am sharing our journey about J2 visa work permit and I think most of you will find this article helpful. My spouse worked with a J2 EAD for approximately 3 years until he got the green card. Before J2 EAD, my husband was working on H1B for 4 years and it was such a nightmare working on H1B, I remember we use to watch the news regarding H1B visas during the Trump administration period from January 2017 to January 2021. Every other day there was some sizzling news on H1B. During that time period, H1B processing time went high, extensions started getting rejected, premium processing fee was increased, some companies stopped sponsoring H1B visa holders, etc. There were so many negative things going on with H1B. Many of my friends had to go back to India as their visa extension got rejected. All different kinds of reasons to deny an extension, this is when my husband planned to switch from H1B to J2 EAD, followed by some personal reason and this was a wise decision for us. So, why J2 EAD? A J1 visa is granted under a cultural exchange program and its dependent (J2 visa holder) can get EAD within ~3-5 months and work part-time or full-time without any restriction. In our case, we were out of the danger ⚠️ zone of H1B. As you know every year ”highly skilled“ people are hired on H1B via a lottery system. H1B workers constitute the biggest visa group among all the other work visa holders, so H1B is the first target when it comes to securing a job for US citizens. Kick out H1B visa holders and most US people will get jobs. This is not my thought but that’s how it’s portrayed to the whole system. Google Trends The biggest advantage, you don’t need any sponsorship when you have J2 EAD, you can work anywhere anytime you want. You can even work extra hours and have 2 jobs. You get a lot of freedom with J2 EAD. You would have noticed that when you apply for a job in the US, usually there is a question if you need sponsorship. You will choose yes if you are an H1B visa holder and most probably due to this reason your resume will be trashed. For J2 EAD, you don’t need any kind of sponsorship. J1 visa holders are eligible to file for the green card themselves or by hiring a lawyer, unlike H1B where companies have to file on your behalf. You can’t do it by yourself. So, another advantage of being J2 dependent is that you automatically get a green card when the primary visa holder (J1) gets the green card. J2 work permit (EAD) When you apply for J2 EAD you will get EAD for the same period as your DS-2019 validity. If your DS-2019 is valid for a year, you will get EAD for a year. If your DS-2019 is valid for more, your EAD will have more time period. In most cases, the DS-2019 duration depends upon how much funding your professor has for your research work and how the University visa and immigration department grants you the DS-2019. Initially, when I came to the US, I had straight 5 years of DS-2019. You can even negotiate with your professor to show more funding and get maximum period on DS-2019 so that your dependent can get more time on EAD. Similary it helps you getting longer Drivers License, State ID etc.. The maximum you can work on J2 EAD is 5 years, the same as the primary visa holder. Later J1 can apply for a green card. Related post: J1 visa holders can apply for green card, see how? J2 EAD extension is a pretty simple process and usually, you can get it extended in 3 months, the same as you can get a new EAD. You can apply for extension 6 months before your EAD expiry so you have sufficient time. We applied for a J2 EAD extension 3 times and it was never rejected or never got RFE, etc. Please refer to my next blog on how you can apply for J2 EAD. Please comment if you have any questions or doubts. I will try my best to share my experience so far. Next: Filing J2 EAD Related Posts ✔ Go to Main Menu ✔ How long can I stay after my J1 visa expires ✔ J1 visa waiver application process end-to-end ✔ How to convert from a J1 visa to H1B ✔ How to apply for J2 EAD ✔ Indian Passport Renewal ✔ How to apply for an OCI card Related Topics

Apple's iPhone has so many features that it's impossible to use all of them but here's few hidden iOS tips and tricks you probably didn't know even existed. 1. Checkout Your iPhone Battery Health For iPhone 6 and later, iOS 11.3 and later - New feature has been added to display battery health status and recommend if a battery needs to be replaced. These can be found in Settings > Battery > Battery Health Click on Battery Health to check your battery status. Maximum battery capacity measures the device battery capacity relative to when it was new. Initially it will be 100% and it decreases with time as battery health degrades. Checkout this for complete details. 2. Apple's iPhone Battery Replacement - Just $29 Apple is offering discounted battery replacement (just $29) for all eligible iPhone 6 and later model. Refer following chart. Visit Apple store near you or book an online appointment here to replace your battery if required. 3. Guided Access It keeps the iPhone in a single app mode, and allows you to control which features are available to the secondary user. This feature is very useful when someone like your little cousin borrows your iPhone just to play game. In that case you can grant access to the game app only. Open the settings app and tap General > Accessibility > Guided Access as shown below. Tap Guided Access so its slider turns green. Turn on Accessibility Shortcut as well. To start Guided Access, triple-click the Home button in the app which you want to give access. 4. Live iPhone Screen Recording You can start live screen recording of your iPhone screen on a single tap. First enable it - Open settings app > Control Center > Customize Controls > add Screen Recording Now, turn on screen recording by taping below button. 5. Smart Invert Colors (while using iPhone late at night) This features allows you to invert the colors (Black to White and vice-versa), mostly used at the time of reading or writing. Open the settings app and tap General > Accessibility > Display Accommodations > Invert Colors. Tap Smart Invert Colors so its slider turns green. It will appear like below. Before (Invert Colors : OFF) After (Invert Colors : ON) Invert Colors Shortcut You can turn on Invert Colors simply by triple clicking home button. To setup open the settings app and tap General > Accessibility > Accessibility Shortcut and check Smart Invert Colors as shown below. Now just triple click your home button to turn ON/OFF Invert Colors. 6. Hide Your Photos Open Photos app, select all the photos you want to hide, tap share and select hide. 7. Emergency SOS (highly recommended) Rapidly click the sleep/wake button five times to quickly call emergency services - health issues, theft or threat. Open the settings app > Emergency SOS (turn slider green) 8. Automatically Offload Unused Applications Apple automatically uninstalls the apps which you don't use while keeping the app data in case you want to re-install the app. This feature is very useful when you are running short on iPhone memory. Go to settings app > General > iPhone Storage > Offload Unused App (enable this by turning slider green) 9. Automatically Turn On "Do Not Disturb" While Driving Go to settings app > Do Not Disturb > Do Not Disturb While Driving (enable this by turning slider green) 10. Turn Your Live Photo Into a GIF Open Photos app and choose any live photo. Now simply slide up to reveal hidden edit features like - Bounce, Loop & Long Exposure. Thank you!! If you enjoyed this post, I’d be very grateful if you’d help it spread by sharing it to a friend, or sharing it on Google or Facebook.

A mathematical joke is a form of humor which relies on aspects of mathematics or a stereotype of mathematicians to derive humor. The humor may come from a pun, or from a double meaning of a mathematical term, or from a lay person's misunderstanding of a mathematical concept. Instead of good-bye we say Calc-U-later Why should you not mix alcohol and calculus? Because you should never drink and derive. Write the expression for the volume of a thick crust pizza with height "a" and radius "z". The formula for volume is π·(radius)**2·(height). In this case, pi·z·z·a. How do you make seven even? Just remove the “s.” Q: What is a proof? A: One-half percent of alcohol. Q: What is gray and huge and has integer coefficients? A: An elephantine equation. Q: Why do truncated Maclaurin series fit the original function so well? A: Because they are “Taylor” made. Q: What is gray and huge and has integer coefficients? A: An elephantine equation. Q: What’s a polar bear? A: A rectangular bear after a coordinate transform. Q: What do you get if you cross a mosquito with a mountain climber? A: You can’t cross a vector with a scalar. Theorem. 3=4. Proof. Suppose a + b = c This can also be written as: 4a − 3a + 4b − 3b = 4c − 3c After reorganizing: 4a + 4b − 4c = 3a + 3b − 3c Take the constants out of the brackets: 4(a + b − c) = 3(a + b − c) Remove the same term left and right: 4=3 A mathematician and an engineer are on a desert island. They find two palm trees with one coconut each. The engineer shinnies up one tree, gets the coconut, and eats it. The mathematician shinnies up the other tree, gets the coconut, climbs the other tree and puts it there. “Now we’ve reduced it to a problem we know how to solve.” There are a mathematician and a physicist and a burning building with people inside. There are a fire hydrant and a hose on the sidewalk. The physicist has to put the fire out…so, he attaches the hose to the hydrant, puts the fire out, and saves the house and the family. Then they put the people back in the house, set it on fire, and ask the mathematician to solve the problem. So, he takes the hose off the hydrant and lays it on the sidewalk. “Now I’ve reduced it to a previously solved problem” and walks away. Three men are in a hot-air balloon. Soon, they find themselves lost in a canyon somewhere. One of the three men says, “I’ve got an idea. We can call for help in this canyon and the echo will carry our voices far.” So he leans over the basket and yells out, “Helloooooo! Where are we?” (They hear the echo several times.) Fifteen minutes later, they hear this echoing voice: “Hellooooo! You’re lost!!” One of the men says, “That must have been a mathematician.” Puzzled, one of the other men asks, “Why do you say that?” The reply: “For three reasons: (1) He took a long time to answer, (2) he was absolutely correct, and (3) his answer was absolutely useless.” Infinitely many mathematicians walk into a bar. The first says, "I'll have a beer." The second says, "I'll have half a beer." The third says, "I'll have a quarter of a beer." Before anyone else can speak, the barman fills up exactly two glasses of beer and serves them. "Come on, now,” he says to the group, “You guys have got to learn your limits.” Scientists caught a physicist and a mathematician and locked them in separate rooms so both could not interact with each other. They started studying their behavior. The two were assigned a task to remove a hammered nail from inside the wall. The only tools they had were a hammer and a nail-drawer. After some muscular effort, both solved the tasks similarly by using the nail-drawer. Then there was a second task, to remove the nail that was barely touching the wall with its sharp end. The physicist simply took the nail with his hand. The mathematician hammered the nail inside the wall with full force and proudly announced: the problem has been reduced to the previous one! A mathematician organizes a raffle in which the prize is an infinite amount of money paid over an infinite amount of time. Of course, with the promise of such a prize, his tickets sell like hot cake. When the winning ticket is drawn, and the jubilant winner comes to claim his prize, the mathematician explains the mode of payment: "1 dollar now, 1/2 dollar next week, 1/3 dollar the week after that..." Sherlock Holmes and Watson travel on a balloon. They were hidden in clouds, so they didn’t know which country they flew above. Finally they saw a guy below between clouds, so they asked. “Hey, you know where we are?” “Yes” “Where?” “In a balloon”. And the guy was hidden by clouds again. Watson:”Goddamn, what a stupid idiot!” Holmes:”No my friend, he’s a mathematician”. Watson:”How can you know that, Holmes?” Holmes:”Elementary, my dear Watson. He responded with an absolutely correct and absolutely useless answer”. My girlfriend is the square root of -100. She’s a perfect 10, but purely imaginary. How do mathematicians scold their children? "If I've told you n times, I've told you n+1 times..." What’s the best way to woo a math teacher? Use acute angle. What do you call a number that can't keep still? A roamin' numeral. Take a positive integer N. No wait, N is too big; take a positive integer k. A farmer counted 196 cows in 
the field. But when he rounded them up, he had 200. Why should you never argue with decimals? Because decimals always have a point. When someone once asked Professor Eilenberg if he could eat Chinese food with three chopsticks, he answered, "Of course," according to Professor Morgan. How are you going to do it? I'll take the three chopsticks, I'll put one of them aside on the table, and I'll use the other two. A statistics professor is going through security at the airport when they discover a bomb in his carry-on. The TSA officer is livid. "I don't understand why you'd want to kill so many innocent people!" The professor laughs and explains that he never wanted to blow up the plane; in fact, he was trying to save them all. "So then why did you bring a bomb?!" The professor explains that the probability of a bomb being on an airplane is 1/1000, which is quite high if you think about it, and statistically relevant enough to prevent him from being able to fly stress-free. "So what does that have to do with you packing a bomb?" the TSA officer wants to know, so the professor explains. "You see, if there's 1/1000 probability of a bomb being on my plane, the chance that there are two bombs is 1/1000000. So if I bring a bomb, the chance there is another bomb is only 1/1000000, so we are all much safer." The great probabilist Mark Kac (1914-1984) once gave a lecture at Caltech, with Feynman in the audience. When Kac finished, Feynman stood up and loudly proclaimed, "If all mathematics disappeared, it would set physics back precisely one week." To that outrageous comment, Kac shot back with that yes, he knew of that week; it was "Precisely the week in which God created the world." An experimental physicist meets a mathematician in a bar and they start talking. The physicict asks, "What kind of math do you do?" to which the mathematician replies, "Knot theory." The physicist says, "Me neither!" A poet, a priest, and a mathematician are discussing whether it's better to have a wife or a mistress. The poet argues that it's better to have a mistress because love should be free and spontaneous. The priest argues that it's better to have a wife because love should be sanctified by God. The mathematician says, "I think it's better to have both. That way, when each of them thinks you're with the other, you can do some mathematics." Three mathematicians walk into a bar. Bartender asks:”Will all of you guys have beer?” The first mathematician: “I don’t know”. The second mathematician: “I don’t know”. The third one: ”Yes”. A mathematician is attending a conference in another country and is sleeping at a hotel. Suddenly, there is a fire alarm and he rushes out in panic. He also notices some smoke coming from one end of the corridor. As he is running, he spots a fire extinguisher. “Ah!”, he exclaims, “A solution exists!” and comes back to his room and sleeps peacefully. Two statisticians go to hunt a bear. After roaming the woods for a while, they spot a lone grizzly. The first statistician takes aim and shoots, but it hits three feet in front of the bear. The second one shoots next, and it hits three feet behind the bear. They both agree that they have shot the bear and go to retrieve it.. Parallel lines have so much in common. It’s a shame they’ll never meet. I just saw my math teacher with a piece of graph paper. I think he must be plotting something. Are monsters good at math? No, unless you Count Dracula. My girlfriend is the square root of -100. She's a perfect 10, but purely imaginary. Q: Why is a math book depressed? A: Because it has so many problems. How do you stay warm in an empty room? Go into the corner where it is always 90 degrees. There are three kinds of people in the world: those who can count and those who can't. Q: Why did I divide sin by tan? A: Just cos. Q: Where's the only place you can buy 64 watermelons and nobody wonders why? A: In an elementary school math class. 60 out of 50 people have trouble with fractions. But why did 7 eat 9? Because you’re supposed to eat 3 squared meals a day. Q: Why is the obtuse triangle depressed? A: Because it is never right. Q: Why did the 30-60-90 degree triangle marry the 45-45-90 degree triangle? A: Because they were right for each other. Q: Why didn't the Romans find algebra very challenging? A: Because they always knew X was 10. Two statisticians went out hunting and they found a deer. The first one overshoots by 5 meters. The second one undershoots by 5 meters. They both hug each other and shout out “We Got It!” An astronomer, a physicist and a mathematician are on a train traveling from England to Scotland. It is the first time for each of them. Some time after the train crosses the border, the three of them notice a sheep in a field. “Amazing!” says the astronomer. “All the sheep in Scotland are black!”. “No, no” responds the physicist. “Some sheep in Scotland are black!” The mathematician closes his eyes pityingly, and intones: “In Scotland, there is at least one field, containing at least one sheep, at least one side of which is black.” An engineer, a physicist and a mathematician go to a hotel. The boiler malfunctions in the middle of the night and the radiators in each room set the curtains on fire. The engineer sees the fire, sees there is a bucket in the bathroom, fills the bucket with water and throws it over the fire. The physicist sees the fire, sees the bucket, fills the bucket to the top of his mentally calculated error margin and throws it over the fire. The mathematician sees the fire, sees the bucket, see the solution and goes back to sleep. #MathJokes #FunnyMath #MathPuns #ShortMathJoke

Refer to my previous blogs (Linux | Mac users) to install the ELK stack on your machine. Once installation is done, there are couple of ways to load CSV files into Elasticsearch which I am aware of (one via Logstash and another with filebeat). In this blog, we will be using Logstash to load the file. I am using sample Squid access logs (comma-separated CSV file) in order to explain this blog. You can find the file format details at this link. Sample Data Copy and paste these records to create an access_log.csv file. I randomly selected this format because it's a CSV file which is a widely used format in the production environment and we have various types of attributes (data types) in the file. $ more /Volumes/MYLAB/testdata/access_log.csv Time,Duration,Client_address,Result_code,Bytes,Request_method,Requested_URL,User,Hierarchy_code,Type 1121587707.473,60439,219.138.188.61,TCP_MISS/503,0,CONNECT,203.84.194.44:25,-,DIRECT/203.84.194.44,- 1121587709.448,61427,219.138.188.61,TCP_MISS/503,0,CONNECT,203.84.194.50:25,-,DIRECT/203.84.194.50,- 1121587709.448,61276,219.138.188.61,TCP_MISS/503,0,CONNECT,67.28.114.36:25,-,DIRECT/67.28.114.36,- 1121587709.449,60148,219.138.188.61,TCP_MISS/503,0,CONNECT,4.79.181.12:25,-,DIRECT/4.79.181.12,- 1121587710.889,60778,219.138.188.61,TCP_MISS/503,0,CONNECT,203.84.194.39:25,-,DIRECT/203.84.194.39,- 1121587714.803,60248,219.138.188.61,TCP_MISS/503,0,CONNECT,203.84.194.50:25,-,DIRECT/203.84.194.50,- 1121587714.803,59866,219.138.188.61,TCP_MISS/503,0,CONNECT,203.84.194.43:25,-,DIRECT/203.84.194.43,- 1121587719.834,60068,219.138.188.61,TCP_MISS/503,0,CONNECT,203.84.194.45:25,-,DIRECT/203.84.194.45,- 1121587728.564,59642,219.138.188.55,TCP_MISS/503,0,CONNECT,168.95.5.45:25,-,DIRECT/168.95.5.45,- My file looks something like this, Start Elastics Now start Elasticsearch and Kibana (if you don't remember how to start them, refer to my previous blogs Linux | Mac users). Don't kill these processes, both are required by logstash to load the data. $ elasticsearch $ kibana Logstash Configuration In order to read the CSV files with Logstash, you need to create a configuration file that will have all the configuration details for accessing the log file like input, filter & output. In short, The input tag contains details like filename, location, start position, etc. Filter tag contains file type, separator, column details, transformations, etc. Output tag contains host detail where the file will be written, index name (should be in lower case), document type, etc. These tags look like JSON but actually, these are not JSON. You can say these formats are specific to Logstash. I have created a config file under the config directory in logstash as shown below. $ more /usr/local/var/homebrew/linked/logstash-full/libexec/config/logstash_accesslog.config input { file { path => "/Volumes/MYLAB/testdata/access_log.csv" start_position => "beginning" sincedb_path => "/Volumes/MYLAB/testdata/logstash.txt" } } filter { csv { separator => "," columns => [ "Time" , "Duration" , "Client_address" , "Result_code" , "Bytes" , "Request_method" , "Requested_URL" , "User" , "Hierarchy_code" , "Type" ] } date { match => [ "Time" , "UNIX" ] target => "EventTime" } mutate {convert => ["Duration", "integer"]} mutate {convert => ["Bytes", "integer"]} } output { elasticsearch { hosts => "localhost" index => "logstash-accesslog" } stdout {} } Explanation! These are very basic tags and straightforward. You use the columns tag to define the list of fields within quotes (if you face issues use single quotes instead). "mutate" is basically doing minor datatype conversion. Also, "match" is used to convert the UNIX timestamps to human-readable time format. Further, a logstash-access log is the index name that I am using for Squid access logs. Still, if there is any question regarding any of the tags please comment (in the comment section below) and I will get back to you as soon as possible. Also, you can change sincedb_path to /dev/null if you don't want to keep the track of loaded files. If you want to reload the same file again make sure you delete the entry from the sincedb_path file (logstash.txt in this case). Here is my config file snapshot, zoom it a little bit to see the content :) Run Logstash & Load data You are all set to start Logstash now with the configuration file which we just created. Follow the below steps in order to run Logstash with the config file. It will take a few seconds to index all the records. Change your logstash home location accordingly, mine is homebrew as I am using Mac. $ /usr/local/var/homebrew/linked/logstash-full/bin/logstash -f /usr/local/var/homebrew/linked/logstash-full/libexec/config/logstash_accesslog.config Make sure you are not getting any Error while loading the file in logstash output, otherwise the file will not load and there will be no index creation. Now open Kibana and run the below command in the "Dev Tools" tab in order to see how many records got loaded. I loaded 10 records just for demonstration Kibana Index Creation Now go to the "Management" tab in Kibana and click on Index Patterns => Create Index Pattern. Create an Index pattern with the same name which we mentioned in the configuration file logstash-accesslog. Hit "Next step" and select time filter => I don't want to use time filter. Now hit "Create index pattern". Now go to the "Discover" tab and you will be able to view the indexed data which we just loaded. Kibana Dashboard I don't have much data to create a Kibana dashboard here, but just for demonstration purposes, let's say you want to see the number of events that occurred/milliseconds. It's an impractical example, as you will never have a such a use case. Go to the "Visualize" tab and hit Create new. Select a pie chart for example, Now select logstash-accesslog index and apply changes in buckets as highlighted below, That's all, you can see a number of events that occurred each millisecond. If you have any questions please write in the comments section below. Thank you. Next: Create Kibana Dashboard Example Navigation Menu: Introduction to ELK Stack Installation Loading data into Elasticsearch with Logstash Create Kibana Dashboard Example Kibana GeoIP Dashboard Example Loading data into Elasticsearch using Apache Spark

01. Sign Up First sign up, after this, you should automatically receive writer's privilege within 24 hours. If you don't get writer's privilege, please email us. 02. Start Writing Go to your profile > Create a post. Before you start, please read blogging guidelines carefully. Blogging Guidelines These rules are meant to keep a quality blogging environment at Dataneb. Blog Uniqueness Blogs should be unique. Dataneb does not accept syndicated/unoriginal posts, research papers, duplicate posts, copying of others' content/articles, etc... NOTE: Violation to this guidelines will result into direct loss of writers privilege. Blog Length Blogs should have a minimum length of 2000 characters, there is no upper limit. You will find the total number of characters in the top left corner of the editor while drafting blogs. Blogs not fulfilling this criterion will be automatically moved to draft status. Image Requirement You can add images (but they should not be copyrighted images). Or, you can leave it to us. One of our moderators will handle image requirements. Back-links Backlinks are allowed (maximum 5 & sometimes more) as far as the intention is clear. Make sure you are not linking any blacklisted websites. Miscellaneous Moderators have the authority to add keywords and modify texts, images, etc so that your blog can get a higher Google ranking. This will help your blog to get more organic views. You can delete your post anytime, but Dataneb has full rights to re-publish that content again. Wait! There is an Easier Way to Publish Your Blog We understand that you don't want to publish your blog without review. Don't worry! Just draft the blog and save it. Email us when your blog is ready to publish and one of our moderators will review and publish it for you. If you are just a member and don't want to become a writer. You can also write your post in a word document and email us for submission. What's next? Share your blog post on Facebook, Twitter, etc to get more views, earn badges and invite others. Sharing blogs on social media is the easiest and fastest way to earn views. We value your words, please don't hurt others' feelings while commenting on blog posts, and maintain a quality environment at Dataneb. Email us if you have any queries.

Learn Apache Spark: Tutorial for Beginners - This Apache Spark tutorial documentation will introduce you to Apache Spark programming in Scala. You will learn about Scala programming, dataframe, RDD, Spark SQL, and Spark Streaming with examples and finally prepare yourself for Spark interview questions and answers. What is Apache Spark? Apache Spark is an analytics engine for big data processing. It runs 100 times faster than Hadoop and gives you full freedom to process large-scale data in real time, run analytics and apply machine learning algorithms. Navigation menu ​ 1. Apache Spark and Scala Installation 1.1 Spark installation on Windows​ 1.2 Spark installation on Mac 2. Getting Familiar with Scala IDE 2.1 Hello World with Scala IDE​ 3. Spark data structure basics 3.1 Spark RDD Transformations and Actions example 4. Spark Shell 4.1 Starting Spark shell with SparkContext example​ 5. Reading data files in Spark 5.1 SparkContext Parallelize and read textFile method 5.2 Loading JSON file using Spark Scala 5.3 Loading TEXT file using Spark Scala 5.4 How to convert RDD to dataframe? 6. Writing data files in Spark ​6.1 How to write single CSV file in Spark 7. Spark streaming 7.1 Word count example Scala 7.2 Analyzing Twitter texts 8. Sample Big Data Architecture with Apache Spark 9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science? 10. Spark Interview Questions and Answers Next: Apache Spark Installation ( Windows | Mac )