81 results found for ""

  • J1 Visa Waiver Application Process and Timeline (Stage 3)

    I am post-doctoral research scientist with University of California, Riverside and I have recently gone through J1 visa waiver process. I am sharing my experience with everyone which I believe would help others to obtain J1 waiver easily. I referred several blogs and videos which was time consuming. I have divided the whole process into 4 sections: Introduction Stage 1 Stage 2 Stage 3 Previous Page: Stage 2 Stage 3 After receiving clearance form all the three (or two) office as mentioned in Stage 2, the applicant needs to complete an online application for DS-3035 with Department of State. Please see the link and site will guide you through the information needed. After you finish online application for DS-3035, a case number will be generated at the end along with a PDF package (the PDF will consists of two sets). Now, applicant need to make two separate packages: Package 1 Set one generated with the DS-3035 package along with supporting documents (mentioned below). Statement of reason (page generated with DS-3035 package) Passport copy of exchange visitors (first and last page) Visa and I-94 copy Copies of all DS-2019 issued Two self addressed stamped envelop Cashier's check of $120 (will be mentioned on the DS3035 PDF package) Send the Packet to following address: Postal Service Department of State J-1Waiver P.O. Box 979037 St. Louis, MO 63197-9000 Courier Service Department of State J-1 Waiver P.O. Box 979037 1005 Convention Plaza St. Louis, MO 63101-1200 Package 2 Set 2; i.e. third copy barcode page Cashier check or money order of $25 + S2 (for using miscellaneous service). You can make for the total amount or separate. One self addressed envelop; to receive the notice that your clearance is sent to the embassy. Copies of passport front and back page, I-94, visa, driving license ( proof of my current address) and the cover letter received during stage 1. It was not mention for CGI-SF but I have included it with my package. Send the package to your respective consulate, in my case it was CGI-SF. Consulate General Of India 540 Arguello Blvd, San Francisco, CA 94118 Please write your case number on the packages. Note: Please ensure to check your respective consulate pages for Stage 3 documents since it varies between all the consulates. What will happen next after applicant has posted both the packages: The applicant will receive a copy recommendation of waiver addressed to Indian embassy in Washington (only if you have provided a self addressed stamped envelop). The Indian embassy, Washington will forward your No objection statement to the waiver review division. After this you can check your status on DOS website using your case number. The status mostly shows pending, and the review process starts when following documents are received (see below). It will appear online step by step on what documents they have received so please wait patiently. No objection from Indian embassy Fee DS-3035 Form Copies of DS-2019 Statement of reason Passport pages It will take up to 4-8 weeks and the status will change to favorable recommendation (most of the cases), and a recommendation will be sent to USCIS. After that, probably in 14 days you will receive I-797 notice of action from USCIS. Mostly after this within a week or two the applicant will receive an official I-797 approval notice from USCIS. Congratulations! You have received your waiver. I hope this blog will be helpful for people going through the process. Please drop me message in the the comment section below if you have any queries. I will get back to you ASAP.

  • J1 Visa Waiver Application Process and Timeline (Stage 2)

    I am post-doctoral research scientist with University of California, Riverside and I have recently gone through J1 visa waiver process. I am sharing my experience with everyone which I believe would help others to obtain J1 waiver easily. I referred several blogs and videos which was time consuming. I have divided the whole process into 4 sections: Introduction Stage 1 Stage 2 Stage 3 Previous Page: Stage 1 Stage 2 Your cover letter obtained in Stage 1 would say - NORI certificate will be issued after you obtain clearance from the following authorities. The Ministry of Human Resource and Development, New Delhi (MHRD): This is online process now, please follow this link and upload all required documents. Department of Home State in India: The state where you belong (address on passport). Regional passport office (RPO): The office from where your first passport was issued. These days RPO clearance is not required and your cover letter should say it. In my case, it was not required. MHRD process is very easy and straight forward. Just follow the link and create your login ID, upload all the necessary document and site will guide you through. Once you submit everything it will take around 3-6 weeks to get the clearance. You will get this certificate by e-mail so please enter the correct email address. Please remember they do it in batches like 1-20 person in one batch so it might take some time. You can always check your status by login to the same site. Department of Home State is where you live in India, in my case it was Uttar Pradesh and this step is the tedious one. Document required: Cover Letter Attested statement and Affidavit (bio-data and affidavit) Self-Attested copies of Passport & Visa Self-Attested copies of Certificates (PhD, M.Sc., B.Sc., and High School) DS-2019 Resume The document list might change based on Home Department of State but in general it should be more or less same. So, have all your supporting documents ready. You can either mail all your documents to the secretariat (where your home department is) or go there physically. If you have any family relatives/parents you can ask them to go there with your documents and submit it. It will take some time because you need to find the right person who handles NORI department since secretariat office have several departments. I would recommend you to go their physically or ask someone to visit on your behalf. Once your documents is accepted by secretariat, administration will forward it for police verification and District Magistrate (DM) office clearance. There will be police inquiry held to verify your address/documents. The police might pay a physical visit to your home address or call the phone number provided by you in bio-data/affidavit, or might ask your parents to visit police station, in my case it was my dad. Then your parents have to submit an affidavit on your behalf stating that you do not have any loan/criminal activity against you and they (your parents) have no objection if you stay in US along with some other documents (ID proof etc., the cop will guide you through what documents are needed, this might vary depending upon State). Also, a separate file will be sent out to your respective District Magistrate (DM), they need to sign it and send it to secretariat office. Please follow up with both police office and DM office, it took 3-4 months for me end-to-end to get this clearance. Once secretariat will receive no objection from DM and Police then they will e-mail you your clearance certificate and simultaneously post it to Indian government, New Delhi. The Indian government will send one physical copy of certificate to your physical address in USA and another copy to respective consulate office in US. Receiving physical copy might take 1-2 months, so be patient. In case you don't receive the physical copy then contact you consulate in USA (via email) to inquire if the email copy will work or not. One of my friend had the same issue (she never received physical copy) and CGI-SF confirmed that email will also work. Follow up at this step every time otherwise it will be much slower for you or they might forget your case. If possible take someone phone number at NORI office so that you can frequently contact them for the update. In my case a gentleman at Secretariat office was kind enough to provide me his number, further he guided me through all the process. Next Page: Stage 3

  • J1 Visa Waiver Application Process and Timeline (Stage 1)

    I am post-doctoral research scientist with University of California, Riverside and I have recently gone through J1 visa waiver process. I am sharing my experience with everyone which I believe would help others to obtain J1 waiver easily. I referred several blogs and videos which was time consuming. I have divided the whole process into 4 sections: Introduction Stage 1 Stage 2 Stage 3 Previous Page: Introduction Stage 1 Following are the steps for stage 1, it would max take 3-5 weeks based on your location and consulate. Download miscellaneous service form from your respective Consulate General of India website (links given above). You need one copy of this form and it should be hand-filled, paste a recent photograph on it. I have attached miscellaneous form which I used for CGI-SF. Download Waiver/NORI (No Objection Return to India) Certificate from your respective CGI website. It contains two sections - Bio-data and Affidavit. You need four copies of this form and it should be hand-filled. Also, make sure to use the latest forms, it might change in future. You can use this link to download the form for CGI-SF. I have attached it below also. Once hand-filled, NORI (bio-data & affidavit) should be notarized. You can get it notarized from bank or UPS. Check with your bank, they might do it for free. If you choose to go to UPS store you have to pay 25$ per document. I was not aware at that time and I ended up paying few hundred bucks to UPS store. Next, make two more Xerox copies of notarized NORI form. Now, you will have total 6 notarized NORI forms (4 original and 2 Xerox copies). A non-refundable fee of 66$ is required for Stage 1 and the fee should be paid through money order or cashiers check drawn in the favor of consulate general of India, San Francisco (in my case). Some CGI accepts cash too but that was not the case with CGI-SF. Additionally, ICWF charges (2$) will be applied for using miscellaneous service. So, you can prepare money order/cashiers check of 68$ or make it separate 66$ and 2$. In my case I prepared two cashiers check worth 66$ and 2$ both payable to drawn in the favor of consulate general of India, San Francisco. The fee might change depending on CGI location, so please double check before proceeding. Next, you need all the supporting documents (listed below) along with these forms and you have following two options. You can mail all the documents to your respective consulate general of India along with your original passport. Mailing address is available on CGI websites. If you want you can physically visit CGI office as well, in my case I went to CGI-SF since I was planning to travel to India and it was not possible for me to send my original passport by mail. You do not need any appointment for CGI visit. In both the options (1 & 2) you have to provide them a return envelop with your address on it. It will take at-least 5-6 weeks to get the documents signed by Indian counselor officer and then it's posted to you using the return envelop you provided. However, this varies location to location. One of my friend went to CGI-NY and the Indian counselor officer signed bio-data and affidavit on the same day and she came back home with all the documents. I thought the same for CGI-SF but they took 5 weeks. Supporting documents: 1. Current Indian passport in original and photocopy of the first five pages and last two pages of the current passports. 2. Proof of your US Visa Status:(copy of any one of the following) Photocopy of the page containing visa on passport (H1-B, H-4 etc.), the copy of I-94 and photocopy of all DS-2019. Clear Photocopy of Green Card Employment Authorization Document (Work Permit) I-797, I-140 or I-20 (If approval copy of these notices are pending, also attach a handwritten note detailing the efforts being taken to regularize status) 3. Proof of current US residence address: (copy of any one of the following) U.S Driving license PG&E, Water or landline telephone bill displaying applicant’s address House Lease Agreement State Identification Card Note: Bank/credit card/mobile phone statements are not accepted as residence proof. Congratulations Stage 1 is done! Along with all the signed documents you will also receive a cover letter from Indian Consulate officer, which will indicate from which offices in India you need to obtain clearance certificates. Next Page: Stage 2

  • J1 Visa Waiver Application Process and Timeline (Intro)

    I am post-doctoral research scientist with University of California, Riverside and I have recently gone through J1 visa waiver process. I am sharing my experience with everyone which I believe would help others to obtain J1 waiver easily. I referred several blogs and videos which was time consuming. To reduce the complexity, I have divided the whole process into 4 sections: Introduction Stage 1 Stage 2 Stage 3 Before I begin I would like to clarify that these steps might change in future or slightly differ for you based on your location within United States. I applied it from Consulate Journal of India, San Francisco, California (CGI-SF) as I live in Riverside, CA. However, overall process should not vary much. What is J1 waiver? J1 visa is a non-immigrant visa granted to individuals who wants to participate in exchange visitors program in United States. Some individuals will be subject to 2 year home country physical presence requirement under which applicant has to go back to home country and serve for minimum 2 years. To waive off this two year rule, you need to obtain J1 waiver. How to know if you are subjected to 2 year home residency 212(e) rule? Please check your J1 Visa or DS 2019, anyone of them could state 212(e) rule. 1. Look at the bottom of your visa, it would say "Bearer is subject to section 212(e), Two year home residency rule does apply". 2. If you don't have it on your visa, please check your DS 2019. I don't have it on DS 2019 since it was on my visa. Refer this screenshot, you might see a check mark on option 2. When should you apply for J1 visa waiver? You can apply for waiver anytime, there is no restriction as such. But it's recommended to apply once you have received DS 2019 extension for the entire possible term (i.e. 5 years). There are basically 3 stages to get J1 waiver and end-to-end it takes 1-2 years to complete. I would suggest you to complete first 2 stages when you have completed 3 years on your J1 visa and apply for third/final stage once you receive full 5 years of extension on J1 visa. I am suggesting that because once you apply for third/final stage of waiver and lets say it gets approved, you will not be able to extend J1 anymore. So apply for third stage only when you have received full 5 years of extension on your visa. Few universities/institutes provide first DS 2019 with complete 5 years of period, but few of them provide yearly extension, and few of them provide two/three years extension. So you have to decide based on your situation. You can post your query in comment section below If you have still doubts, I will help you with this. Where should you apply? Please refer this link to see which Consulate General of India (CGI) office covers your state. Washington DC (details) Chicago (details) New York (details) San Francisco (details) Houston (details) Atlanta (details) I went in person to Consulate General of India, San Francisco. However, you can apply in person or send via mail. Next Page : Stage 1

  • How Families Can Keep Their Home Ready For Showings

    How Families Can Keep Their Home Ready For Showings Getting your house ready to go up on the market is always a big job. Add kids to the picture, however, and that job gets much bigger. You want to make a great presentation, which means keeping up with chaos that can go hand-in-hand with kids. Here’s a look at how families can keep their home looking great until the right buyers come along: Have A Plan It’s important to get everyone in the house on the same page when it comes to staging your home. This includes any children old enough to take on some tidying tasks. For the time your house is up for sale, upgrade everyone’s chore charts to reflect a few items off your staging checklist. This way you’re constantly keeping your home ready for buyers. This is super useful, since it allows you to host agents and house hunters at the drop of a hat. For practical purposes, keep the checklist handy and make sure everyone knows where it is. You can even share your list electronically with other adults in the household and kids who are old enough to use phones or tablets. If your agent lets you know they’re swinging by soon, anyone old enough can ensure that surfaces are wiped, shades are open, and personal items are safely stashed away. Use The Best Tools Keeping your house tidy all the time is a big task, but it can be made substantially easier with the right tools. For example, a good set of microfiber cloths can make quickly wiping up surfaces a breeze. A stick vacuum is another tool you’ll want at your disposal. Since these are more versatile and lighter than traditional vacuums, they make spot cleaning fast and easy. Go through all of your cleaning supplies and try to identify which are most useful for a quick, efficient clean up, and assemble a cleaning caddy so you can grab everything at once when you’re on the run. Remove Personal Items One of the most important things your family can do when it comes to staging your home is taking down décor that makes it look too lived in. Per Creative Home Stagers, this includes family photos, bold color schemes, and especially stylistic wall art or furniture. These personal touches may make you feel at home, but they’ll make potential buyers feel like they’re in someone else’s home. On one hand, this is true, but on the other, it can be a problem. Even if sellers aren’t thinking of it consciously, they’re trying to picture themselves in the space. Pictures of your family holiday party or child’s first steps will make it harder for them to imagine their life in the house. Plan Fun Outings – And A Backup – For Open Houses Although it may be tempting to try and scope out interested buyers, sellers should never be at an open house. In addition to being an even starker reminder that the home belongs to someone else, The Balance points out that your presence will put uncomfortable pressure on the buyers and make it harder for them to pay attention to the property. Instead, plan a fun outing with your family during the scheduled open houses. Head to a park, playground, or museum to pass a little time. If you have younger children, it might also be wise to find a friend or family member who will be willing to host you if your plans go south. You don’t want to show up to an open house at all, much less with a screaming toddler. Keeping a house market-ready with kids can be a challenge, but don’t be intimidated. Prep your home and family, and make arrangements for showings and open house events. With a plan under your belt, there’s nothing stopping you from keeping your house buyer-ready until that magic day it’s sold! Photo Credit: Pexels

  • 6 Budget-Friendly Ways to Prepare for Your Pregnancy (checklist)

    Every pregnancy is different, and that is true even in the same person. Your first pregnancy might have been plagued with morning sickness, high blood pressure and lower back pain, while in your second pregnancy you hardly felt a thing. That can make pregnancy preparation tricky — not knowing what to expect can be hard on your mood and your finances. Many pregnant women enjoy feeling their new child growing and developing, but in those times of discomfort, it’s important to have a plan to manage physical and mental stress. Here are a few budget-friendly tips to help you with sound and solid pregnancy prep. 1. Before and after clothes When you think about buying maternity clothes do you just cringe at the cost, knowing you’ll only have to wear this size for a short period of time? There are actually ways to cut costs when it comes to pregnancy wear. First, consider buying a belly band so you can transform the pants you currently wear into pregnancy and postpartum pants. Second, look into comfortable nursing pajamas (you can find a pair for $33.99) that you can fit into now and after the baby comes. The more cozy and flowy they are, the more comfortable you’ll be during some of those long, late night nursing marathons. 2. Amazon’s “Subscribe and Save” You should have bought stock in antacids with the kind of heartburn you are experiencing. Now it’s 3 am and you can’t sleep and you are out of Tums. You can save time and money by subscribing to items you use a lot. Not only will these be automatically delivered to your home so you never have to experience late night heartburn unaided again, but the cost per item is often reduced when you subscribe. You can do this with other items, like foods you have been craving, shea butter to help reduce stretch marks or hemorrhoid cream for sore bottoms. 3. Putting together a nursery Putting together a warm and comfortable nursery is important for mother and baby. Since you and your newborn will spend a lot of time there, you want it to be as nurturing as possible. And while you might be tempted to go overboard with the decor, it’s important to focus only on the basics so you can stay within budget. Also, while you might be tempted to do everything yourself, don’t tackle any projects that you feel are out of your wheelhouse. Fortunately, in Minneapolis, you can hire a handyman for an assortment of small jobs for an average of $403 per project, depending on the size of the project. And although that might sound like a lot of money, you’ll rest assured knowing that the tasks were completed by a professional. 4. Children’s consignment stores While primarily an ideal spot to find good deals on gently used clothes, toys, furniture and bedding, you can also find steep discounts on used maternity and postpartum accessories. You can find breast pumps and parts, breastfeeding pillows and other nursing items. And the e-commerce boom has also helped increase access to quality used pre- and postpartum clothes. You can even rent high end used maternity and nursing clothes. Browse online and have them delivered right to your door. 5. Explore Coupons and Groupons The big box retailers love a pregnant woman — families are very profitable to stores that sell food, clothing, home goods and furniture. They will be looking to entice you into the store by offering coupons and discounts on maternity and baby items. Take advantage of these discounts! And don’t just look there; websites that offers discounts, like Groupon, also often have a section with items to help you plan and prepare for a baby. And don’t forget about stores like Sam’s Club and Costco. After you pay their membership fee, you get access to bulk and wholesale items with steep discounts. In fact, consider adding a membership to one of those stores to your baby registry. 6. Facebook groups for new moms Social media is a place where we can build community. Of course, anyone watching the news knows social media has a dark side, but there are also opportunities to find and make real connections. Look for mom groups out there in your area. There are often breastfeeding groups, buy-sale-trade groups, baby-wearing groups and other mom-themed groups in many cities. More importantly than being able to purchase used items, you are able to ask questions, get advice and provide — and receive— support. Pregnancy is going to be a time of discovery, even for those on their second child or beyond. Give yourself space to breathe easier by setting a budget and staying within that budget. And don’t forget to lean on your community as much as you can for support.

  • Apache Avro Schema Example (in Java)

    Introduction Avro provides data serialization based on JSON Schema. It is language neutral data serialization system, means a language A can serialize and languages B can de-serialize and use it. Avro supports both dynamic and static types as per requirement. It supports many languages like Java,C, C++, C#, Python and Ruby. Benefits Producers and consumers are decoupled from their change in application. Schemas help future proof your data and make it more robust. Supports and used in all use cases in streaming specially in Kafka. Avro are compact and fast for streaming. Supports for schema registry in case of Kafka. Steps to Serialize Object Create JSON schema. Compile the schema in the application. Populate the schema with data. Serialize data using Avro serializer. Steps to Deserialize Object Use Apache Avro api to read the serialized file. Populate the schema from file. Use the object for application. Sample Example for Avro (in Java) Step-1: Create a Java project and add the dependencies as below. Step-2: Create a Schema file as below: Customer_v0.avsc { "namespace": "com.demo.avro", "type": "record", "name": "Customer", "fields": [ { "name": "id", "type": "int" }, { "name": "name", "type": "string" }, { "name": "faxNumber", "type": [ "null", "string" ], "default": "null" } ] } Step-3: Compile the schema. java -jar lib\avro-tools-1.8.1.jar compile schema schema\Customer_v0.avsc schema Step-4: Put the java generated file to the source directory of the project as shown in project structure. Step-5: Create the Producer.java package com.demo.producer; import java.io.File; import java.io.IOException; import org.apache.avro.file.DataFileWriter; import org.apache.avro.io.DatumWriter; import org.apache.avro.specific.SpecificDatumWriter; import com.demo.avro.Customer; public class Producer { public static void main(String[] args)throws IOException { serailizeMessage(); } public static void serailizeMessage()throws IOException{ DatumWriter datumWriter = new SpecificDatumWriter(Customer.class); DataFileWriter dataFileWriter = new DataFileWriter(datumWriter); File file = new File("customer.avro"); Customer customer=new Customer(); dataFileWriter.create(customer.getSchema(), file); customer.setId(1001); customer.setName("Customer -1"); customer.setFaxNumber("284747384343333".subSequence(0, 10)); dataFileWriter.append(customer); customer=new Customer(); customer.setId(1002); customer.setName("Customer -2"); customer.setFaxNumber("45454747384343333".subSequence(0, 10)); dataFileWriter.append(customer); dataFileWriter.close(); } } Step-6: Create the Consumer.java package com.demo.consumer; import java.io.File; import java.io.IOException; import org.apache.avro.file.DataFileReader; import org.apache.avro.io.DatumReader; import org.apache.avro.specific.SpecificDatumReader; import com.demo.avro.Customer; public class Consumer { public static void main(String[] args)throws IOException { deSerailizeMessage(); } public static void deSerailizeMessage()throws IOException{ File file = new File("customer.avro"); DatumReader datumReader = new SpecificDatumReader(Customer.class); DataFileReader dataFileReader= new DataFileReader(file,datumReader); Customer customer=null; while(dataFileReader.hasNext()){ customer=dataFileReader.next(customer); System.out.println(customer); } } } Step-7: Run Producer.java It creates customer.avro file and puts the customer in Avro format. Step-8: Run Consumer.java It reads the customer.avro file and get the customer records. Thank you! If you have any question please mention in comments section below. [12/09/2019 10:38 PM CST - Reviewed by: PriSin]

  • StreamingContext: Spark streaming word count example Scala

    Main menu: Spark Scala Tutorial In this tutorial you will learn, How to stream data in real time using Spark streaming? Spark streaming is basically used for near real-time data processing. Why I said "near" real-time? Because data processing takes some time, few milliseconds. This lag can be reduced but obviously it can't be reduced to zero. This lag is so minute that we end up calling it real-time processing. Streaming word seems very cool but honestly speaking most of you have already implemented this in the form of "batch mode". Only difference is the time window. Everyone is aware of batch mode where you pull the data on hourly, daily, weekly or monthly basis and process it to fulfill your business requirements. What if you start pulling data every second and simultaneously you made your code so efficient that it can process the data in milliseconds. It would be automatically near real time processing, right? Spark streaming basically provides you ability to define sliding time windows where you can define the batch interval. After that, Spark automatically breaks the data into smaller batches for real-time processing. It basically uses RDD's (resilient distributed datasets) to perform the computation on unstructured dataset. RDD's are nothing but references to the actual data which are distributed across multiple nodes with some replication factor which reveal their values only when you perform an action (like collect) on top of it, called lazy evaluation. If you haven't installed Apache Spark, please refer this (Windows | Mac users). Screen 1 Open Scala IDE, create package com.dataneb.spark and define Scala object SparkStreaming. Check this link how you can create packages and objects using Scala IDE. Copy paste the below code on your Scala IDE and let the program run. package com.dataneb.spark /** Libraries needed for streaming the data. */ import org.apache.spark._ import org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ object SparkStreaming { /** Reducing the logging level to print just ERROR. */ def setupLogging() = { import org.apache.log4j.{Level, Logger} val rootLogger = Logger.getRootLogger() rootLogger.setLevel(Level.ERROR) } def main(args: Array[String]) { /** Defining Spark configuration to utilize all the resources and * setting application name as TerminalWordCount*/ val conf = new SparkConf().setMaster("local[*]").setAppName("TerminalWordCount") /** Calling logging function */ setupLogging() /** Defining spark streaming context with above configuration and batch interval as 1*/ val ssc = new StreamingContext(conf, Seconds(1)) /** Terminal 9999 where we will entering real time messages */ val lines = ssc.socketTextStream("localhost", 9999) /** Flat map to split the words with spaces and reduce by key pair to perform count */ val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) // Print the first ten elements of each RDD wordCounts.print() ssc.start() // Start the computation ssc.awaitTermination() // Wait for the computation to terminate } } Screen 2 Now open your local terminal and type "nc -lk 9999" , hit enter. What we did just now? We just made a network connection (nc or NetCat) to local port 9999. Now type some random inputs for Spark processing as shown below, Screen 1 Now go back to Scala IDE to see the processed records, you need to swap the screens quickly to see the results as Spark will process these lines within seconds. Just kidding, you can simply pin the console and scroll back to see the results. You can see the output below. That's it. It's so simple. In real life scenario you can stream the Kafka producer to local terminal from where Spark can pick up for processing. Or you can also configure Spark to communicate with your application directly. Thank you!! If you have any question write in comments section below. #spark #scala #example #StreamingContext #streaming #word #count Next: Analyzing Twitter text with Spark Streaming Navigation menu ​ 1. Apache Spark and Scala Installation 1.1 Spark installation on Windows​ 1.2 Spark installation on Mac 2. Getting Familiar with Scala IDE 2.1 Hello World with Scala IDE​ 3. Spark data structure basics 3.1 Spark RDD Transformations and Actions example 4. Spark Shell 4.1 Starting Spark shell with SparkContext example​ 5. Reading data files in Spark 5.1 SparkContext Parallelize and read textFile method 5.2 Loading JSON file using Spark Scala 5.3 Loading TEXT file using Spark Scala 5.4 How to convert RDD to dataframe? 6. Writing data files in Spark ​6.1 How to write single CSV file in Spark 7. Spark streaming 7.1 Word count example Scala 7.2 Analyzing Twitter texts 8. Sample Big Data Architecture with Apache Spark 9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science? 10. Spark Interview Questions and Answers

  • Kibana GeoIP example: How to index geographical location of IP addresses into Elasticsearch

    The relation between your IP address and geolocation is very simple. There are numerous websites available as of today like Maxmind, IP2Location, IPstack , Software77 etc where you can track the geolocation of an IP address. What's the benefit? It's very simple, it gives you another dimension to analyze your data. Let's say my data predicts that most of the users traffic is coming from 96.67.149.166. It doesn't make complete sense until I say most of the traffic is coming from New Jersey. When I say geolocation it includes multiple attributes like city, state, country, continent, region, currency, country flag, country language, latitude, longitude etc. Most of the websites which provide geolocation are paid sites. But there are few like IPstack which provides you free access token to make calls to their rest API's. Still there are limitations like how many rest API calls you can make per day and also how many types of attributes you can pull. Suppose I want to showcase specific city in the report and API provides limited access to country and continent only, then obviously that data is useless for me. Now the best part is Elastic stack provides you free plugin called "GeoIP" which grants you access to lookup millions of IP addresses. You would be thinking from where it gets the location details? The answer is Maxmind which I referred earlier. GeoIP plugin internally does a lookup from stored copy of Maxmind database which keeps on updating and creates number of extra fields with geo coordinates (longitude & latitude). These geo coordinates can be used to plot maps in Kibana. ELK Stack Installation I am installing ELK stack on Mac OS, for installation on Linux machine refer this. ELK installation is very easy on Mac with Homebrew. It's hardly few minutes task if done properly. 1. Homebrew Installation Run this command on your terminal. If you have already installed Homebrew move to the next step, or if this command doesn't work - copy it from here. $ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 2. Java Installation Check if java is installed on your machine. $ java -version java version "9.0.1" If java is not installed, run following steps to install java. $ brew tap caskroom/cask $ brew cask install java $ brew cask info java 3. Elasticsearch Installation $ brew tap elastic/tap $ brew install elastic/tap/elasticsearch-full $ elasticsearch If you see all INFO without any error, that means installation went fine. Let this run, don't kill the process. Now, simply open localhost:9200 in your local browser. You will see elasticsearch version. [TIP] You might face permission issue if you are not logged in with root user. To enable root user on Mac you can follow this. It's due to security reasons that root user is disabled by default on Mac. However another solution is to change folder permission itself. Run these commands if you want to change folder permissions, $ sudo chown -R $(whoami) /usr/local/include /usr/local/lib/pkgconfig $ chmod u+w /usr/local/include /usr/local/lib/pkgconfig Install xcode if it's missing, $ xcode-select --install 4. Kibana Installation $ brew install elastic/tap/kibana-full $ kibana Let this process run, don't kill. Now, open localhost:5601 in your local browser to check if kibana is running properly, 5. Logstash Installation $ brew install elastic/tap/logstash-full Configuring Logstash for GeoIP Let's begin with few sample IP addresses as listed below. I generated this sample data from browserling.com so please ignore if there is some known ip address in this list. Honestly speaking even I don't know where these IP addresses will point to when we generate the maps. Sample Data 1. Copy paste these records into a flat file with "ipaddress" header (sampleip.csv). ipaddress 0.42.56.104 82.67.74.30 55.159.212.43 108.218.89.226 189.65.42.171 62.218.183.66 210.116.94.157 80.243.180.223 169.44.232.173 232.117.72.103 242.14.158.127 14.209.62.41 4.110.11.42 135.235.149.26 93.60.177.34 145.121.235.122 170.68.154.171 206.234.141.195 179.22.18.176 178.35.233.119 145.156.239.238 192.114.2.154 212.36.131.210 252.185.209.0 238.49.69.205 2. Make sure your Elasticsearch and Kibana services are up and running. If not, please refer my previous blog - how to restart them. 3. [Update 9/Aug/2019: Not mandatory step now] Install GeoIP plugin for Elasticsearch. Run the below command in your Elasticsearch home directory. Once GeoIP plugin is installed successfully, you will be able to find plugin details under elasticsearch home plugin directory "/elasticsearch/plugins". You need to run installation command on each node if you are working in a clustered environment and then restart the services. /elasticsearch/bin/elasticsearch-plugin install ingest-geoip New version of elastics has built in GeoIP module, so you don't need to install it separately. Configure Logstash Configure logstash config file to create "logstash-iplocation" index. Please note your index name should start with logstash-name otherwise your attributes will not be mapped properly as geo_points datatype. This is because the default index name in logstash template is declared as logstash-* , you can change it if you want but as of now lets move ahead with logstash-iplocation. Below is the sample input, filter and output configuration. input { file { path => "/Volumes/MYLAB/testdata/sampleip.csv" start_position => "beginning" sincedb_path => "/Volumes/MYLAB/testdata/logstash.txt" } } filter { csv { columns => "ipaddress" } geoip { source => "message" } } output { elasticsearch { hosts => "localhost" index => "logstash-iplocation" } stdout{ codec => rubydebug } } My configuration file looks something like this: Important Notes Your index name should be in lower caps, starting with logstash- for example logstash-abcd Also, sincedb path is created once per file input, so if you want to reload the same file make sure you delete the sincedb file entry. It looks like this, You invoke geoip plugin from filter configuration, it has no relation with input/output. Run Logstash Load the data into elasticsearch by running below command (it's a single line command). Now wait, it will take few seconds to load. Change your home location accordingly, for me its homebrew linked as shown below. /usr/local/var/homebrew/linked/logstash-full/bin/logstash -f /usr/local/var/homebrew/linked/logstash-full/libexec/config/logstash_ip.config Sample output Important Notes See if filters geoip is invoked when you load the data into elasticsearch. Also, the datatype of location should be geo_point, otherwise there is some issue with your configuration. Latitude and longitude datatype should be float. These datatypes are like confirmation that logstash loaded this data as expected. Kibana Dashboard Creation 1. Once data is loaded into Elasticsearch, open Kibana UI and go to Management tab => Kibana Index pattern. 2. Create Kibana index with "logstash-iplocation" pattern and hit Next. 3. Select timestamp if you want to show it with your index and hit create index pattern. 4. Now go to Discover tab and select "logstash-iplocation" to see the data which we just loaded. You can expand the fields and see geoip.location has datatype as geo_point. You can verify this by "globe" sign which you will find just before geoip.location field. If it's not there then you have done some mistake and datatype mapping is incorrect. 5. Now go to Visualize tab and select coordinate map from the types of visualization and index name as "logstash-iplocation". 6. Apply the filters (Buckets: Geo coordinates, Aggregation: Geohash & Field: geoip.location) as shown below and hit the "Play" button. That's it !! You have located all the ip addresses. Thank you!! If you have any question please comment. Next: Loading data into Elasticsearch using Apache Spark Navigation Menu: Introduction to ELK Stack Installation Loading data into Elasticsearch with Logstash Create Kibana Dashboard Example Kibana GeoIP Dashboard Example Loading data into Elasticsearch using Apache Spark

  • Enable Root User on Mac

    By default root user is disabled on Mac, you need to follow below steps in order to enable/disable or change password for root user on Mac. 1. From top left hand side, choose Apple menu > System Preferences, then click Users & Groups (or Accounts). 2. Click the lock icon , then enter an administrator name and password. 3. After you unlock the lock. Click Login Options, right next to home icon. 4. Now Click Join (or Edit), right next to Network Account Server. Now Click Open Directory Utility. 5. Click lock icon in the Directory Utility window, then enter an administrator name and password. 6. From the menu bar in Directory Utility: Choose Edit > Enable Root User, then enter the password that you want to use for the root user. You can enable/disable/change password for root user from here. 7. Now go to Terminal and switch user to root and test. Rajas-MacBook-Pro: Rajput$ su root Password: Thank you!! If you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on Google or Facebook. Refer the links below. Also click on "Subscribe" button on top right corner to stay updated with latest posts. Your opinion matters a lot please comment if you have any suggestion for me. #enable #root #user #Mac

Home   |   Contact Us

©2020 by Data Nebulae