Tutorial by Examples: t

Creating a Scala function that receives an python RDD is easy. What you need to build is a function that get a JavaRDD[Any] import org.apache.spark.api.java.JavaRDD def doSomethingByPythonRDD(rdd :JavaRDD[Any]) = { //do something rdd.map { x => ??? } }

This part of development you should serialize the python RDD to the JVM. This process uses the main development of Spark to call the jar function. from pyspark.serializers import PickleSerializer, AutoBatchedSerializer rdd = sc.parallelize(range(10000)) reserialized_rdd = rdd._reserialize(Aut...

To call this code you should create the jar of your scala code. Than you have to call your spark submit like this: spark-submit --master yarn-client --jars ./my-scala-code.jar --driver-class-path ./my-scala-code.jar main.py This will allow you to call any kind of scala code that you need in your...

Using many ECHO commands to create a batch file: ( echo echo hi, this is the date today echo date /T echo echo created on %DATE% echo pause >nul )>hi.bat The command interpreter treats the whole section in parenthesis as a single command, then saves all the output to hi.bat. ...

Ant provides some built-in properties Property NameValuebasedirthe absolute path of the project's basedirant.filethe absolute path of the buildfile.ant.versionthe version of Antant.project.default-targetthe name of the currently executing project's default targetant.project.namename of the projecta...

There are two kind of types in Python. Immutable types and mutable types. Immutables An object of an immutable type cannot be changed. Any attempt to modify the object will result in a copy being created. This category includes: integers, floats, complex, strings, bytes, tuples, ranges and frozen...

One of the major use case when a developer needs to take mutability into account is when passing arguments to a function. This is very important, because this will determine the ability for the function to modify objects that doesn't belong to its scope, or in other words if the function has side ef...

There are four ways to work with JavaScript plugins, Ember add-on Import JavaScript plugins globally Consume named AMD plugins Via ember-browserify

First, complete the installation and setup to connect your app to Firebase. Then from anywhere in your class, you can write: // Write a message to the database FirebaseDatabase database = FirebaseDatabase.getInstance(); DatabaseReference myRef = database.getReference("message"); myRe...

This behaviour can be extended. Here is a 3-dimensional array: [[[111,112,113],[121,122,123],[131,132,133]],[[211,212,213],[221,222,223],[231,232,233]],[[311,312,313],[321,322,323],[331,332,333]]] As is probably obvious, this gets a bit hard to read. Use backslashes to break up the different dim...

Gradle (All Versions) This method works for all versions of gradle Add the buildscript code at the beginning of your build.gradle file. buildscript { repositories { maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "org.example...

Gradle (All Versions) When adding multiple third party plugins you do not need to separate them into different instances of the buildscript(All) or plugin(2.1+) code, new plug ins can be added alongside pre-existing plugins. buildscript { repositories { maven { url "https://plu...

Basic Idea The notation used when describing the speed of your Python program is called Big-O notation. Let's say you have a function: def list_check(to_check, the_list): for item in the_list: if to_check == item: return True return False This is a simple function ...

Operations : Average Case (assumes parameters are randomly generated) Append : O(1) Copy : O(n) Del slice : O(n) Delete item : O(n) Insert : O(n) Get item : O(1) Set item : O(1) Iteration : O(n) Get slice : O(k) Set slice : O(n + k) Extend : O(k) Sort : O(n log n) Multiply : O(nk) x in...

A deque is a double-ended queue. class Deque: def __init__(self): self.items = [] def isEmpty(self): return self.items == [] def addFront(self, item): self.items.append(item) def addRear(self, item): self.items.insert(0,item) def removeFront(self): return self....

Operation : Average Case (assumes parameters generated randomly) : Worst case x in s : O(1) Difference s - t : O(len(s)) Intersection s&t : O(min(len(s), len(t))) : O(len(s) * len(t) Multiple intersection s1&s2&s3&...&sn : : (n-1) * O(l) where l is max(len(s1),...,len(sn)) s...

When using d3.request() or one of the convenience constructors (d3.json, d3.csv, d3.tsv, d3.html and d3.xml) there are many sources for error. There may be problems with the issued request or its response due to network errors, or the parsing might fail if the content is not well-formed. Within the...

Like page caching, action caching caches the whole page. The difference is that the request hits the Rails stack so before filters are run before the cache is served. It's extracted from Rails to actionpack-action_caching gem. A common example is caching of an action that requires authentication: ...

Sample Data: CREATE TABLE table_name ( id, list ) AS SELECT 1, 'a,b,c,d' FROM DUAL UNION ALL -- Multiple items in the list SELECT 2, 'e' FROM DUAL UNION ALL -- Single item in the list SELECT 3, NULL FROM DUAL UNION ALL -- NULL list SELECT 4, 'f,,g' FROM DUAL; -- NULL item...

Decryption Code public static string Decrypt(string cipherText) { if (cipherText == null) return null; byte[] cipherBytes = Convert.FromBase64String(cipherText); using (Aes encryptor = Aes.Create()) { Rfc2898DeriveBytes pdb = ...