In Bluemix I have a java_buildpack application that currently picks up/consumes a message from a Message Hub topic and I now want to reformat that message and publish it to a Mqtt... moreIn Bluemix I have a java_buildpack application that currently picks up/consumes a message from a Message Hub topic and I now want to reformat that message and publish it to a Mqtt client. I have the Internet of Things Platform service bound to my java_buildpack application and I created a device type, javaDevice, and device id, javaDevice01.
Here's the environment variables for my iotf-service:
True ... it has been discussed quite a lot.
However there is a lot of ambiguity and some of the answers provided ... including duplicating jar references in the... moreTrue ... it has been discussed quite a lot.
However there is a lot of ambiguity and some of the answers provided ... including duplicating jar references in the jars/executor/driver configuration or options.
The ambiguous and/or omitted details
Following ambiguity, unclear, and/or omitted details should be clarified for each option:
How ClassPath is affected
Driver
Executor (for tasks running)
Both
not at all
Separation character: comma, colon, semicolon
If provided files are automatically distributed
for the tasks (to each executor)
for the remote Driver (if ran in cluster mode)
type of URI accepted: local file, hdfs, http, etc
If copied into a common location, where that location is (hdfs, local?)
The options to which it affects :
--jarsSparkContext.addJar(...) methodSparkContext.addFile(...) method--conf spark.driver.extraClassPath=... or --driver-class-path ...--conf spark.driver.extraLibraryPath=..., or --driver-library-path ...--conf spark.executor.extraClassPath=...--conf... less
I want to setup a hadoop-cluster in pseudo-distributed mode. I managed to perform all the setup-steps, including startuping a Namenode, Datanode, Jobtracker and a Tasktracker on... moreI want to setup a hadoop-cluster in pseudo-distributed mode. I managed to perform all the setup-steps, including startuping a Namenode, Datanode, Jobtracker and a Tasktracker on my machine.
Then I tried to run some exemplary programms and faced the java.net.ConnectException: Connection refused error. I stepped back to the very first steps of running some operations in standalone mode and faced the same problem.
I performed even triple-check of all the installation steps and have no idea how to fix it. (I am new to Hadoop and a beginner Ubuntu user thus I kindly ask you for "taking it into account" if providing any guide or tip).
This is the error output I keep receiving:
hduser@marta-komputer:/usr/local/hadoop$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs+'
15/02/22 18:23:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/02/22 18:23:04 INFO client.RMProxy:... less
I tried installing Hadoop following this http://hadoop.apache.org/common/docs/stable/single_node_setup.html document. When I tried executing this
bin/hadoop jar... moreI tried installing Hadoop following this http://hadoop.apache.org/common/docs/stable/single_node_setup.html document. When I tried executing this
bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs+'
I am getting the following Exception
java.lang.OutOfMemoryError: Java heap space
Please suggest a solution so that i can try out the example. The entire Exception is listed below. I am new to Hadoop I might have done something dumb . Any suggestion will be highly appreciated.
anuj@anuj-VPCEA13EN:~/hadoop$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs+'
11/12/11 17:38:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library
11/12/11 17:38:22 INFO mapred.FileInputFormat: Total input paths to process : 7
11/12/11 17:38:22 INFO mapred.JobClient: Running job: job_local_0001
11/12/11 17:38:22 INFO util.ProcessTree: setsid exited with exit code 0
11/12/11 17:38:22 INFO mapred.Task: Using ResourceCalculatorPlugin :... less
I am trying to run a simple NaiveBayesClassifer using hadoop, getting this error
Exception in thread "main" java.io.IOException: No FileSystem for scheme: file
at... moreI am trying to run a simple NaiveBayesClassifer using hadoop, getting this error
Exception in thread "main" java.io.IOException: No FileSystem for scheme: file
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:180)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
at org.apache.mahout.classifier.naivebayes.NaiveBayesModel.materialize(NaiveBayesModel.java:100)
Code :
Configuration configuration = new Configuration();
NaiveBayesModel model = NaiveBayesModel.materialize(new Path(modelPath), configuration);// error in this line..
modelPath is pointing to NaiveBayes.bin file, and configuration object is printing... less
I am trying to get live JSON data from RabbitMQ to Apache Spark using Java and do some realtime analytics out of it.
I am able to get the data and also do some basic SQL queries... moreI am trying to get live JSON data from RabbitMQ to Apache Spark using Java and do some realtime analytics out of it.
I am able to get the data and also do some basic SQL queries on it, but I am not able to figure out the grouping part.
Below is the JSON I have
{"DeviceId":"MAC-101","DeviceType":"Simulator-1","data":{"TimeStamp":"26-06-2017 16:43:41","FR":10,"ASSP":20,"Mode":1,"EMode":2,"ProgramNo":2,"Status":3,"Timeinmillisecs":636340922213668165}}
{"DeviceId":"MAC-101","DeviceType":"Simulator-1","data":{"TimeStamp":"26-06-2017 16:43:41","FR":10,"ASSP":20,"Mode":1,"EMode":2,"ProgramNo":2,"Status":3,"Timeinmillisecs":636340922213668165}}
{"DeviceId":"MAC-102","DeviceType":"Simulator-1","data":{"TimeStamp":"26-06-2017 16:43:41","FR":10,"ASSP":20,"Mode":1,"EMode":2,"ProgramNo":2,"Status":3,"Timeinmillisecs":636340922213668165}}
{"DeviceId":"MAC-102","DeviceType":"Simulator-1","data":{"TimeStamp":"26-06-2017... less
I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:WARN util.NativeCodeLoader: Unable to load... moreI'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableI'm running Hadoop 2.2.0.Doing a search online brought up this link: http://balanceandbreath.blogspot.ca/2013/01/utilnativecodeloader-unable-to-load.htmlHowever, the contents of /native/ directory on hadoop 2.x appear to be different so I am not sure what to do.I've also added these two environment variables in hadoop-env.sh:export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"Any ideas? less
I am incredibly new to the topic of websockets and am trying to figure out the correct way to handle the communication between a device and a server.
Here is my scenario: I have a... moreI am incredibly new to the topic of websockets and am trying to figure out the correct way to handle the communication between a device and a server.
Here is my scenario: I have a thermostat (very similar to the Nest) that needs to communicate with a web server. Every time you change the temperature on the thermostat, I need to send data to the web server to update it's "current stats" in the database. Easy, I can do that.
The part that I am confused about, and think websockets might be a use-case is when the user changes the temperature from the web interface. The thermostat has to pull in that information from the server to see "Oh, okay you want it to be 66 degrees."
I've thought of having the thermostat long-polling the server every 2-5 seconds to see what the "current stats" are in the database to change the temperature, but that seems like overkill.
Is there a way to open a connection between the thermostat and the server to listen for messages?
I began down the road of reading about... less
I've been asked to port a legacy data processing application over to Java.
The current version of the system is composed of a nubmer of (badly written) Excel sheets. The sheets... moreI've been asked to port a legacy data processing application over to Java.
The current version of the system is composed of a nubmer of (badly written) Excel sheets. The sheets implement a big loop: A number of data-sources are polled. These source are a mixture of CSV and XML-based web-servics.
The process is conceptually simple:
It's stateless, that means the calculations which run are purely dependant on the inputs. The results from the calculations are published (currently by writing a number of CSV files in some standard locations on the network).
Having published the results the polling cycle begins again.
The process will not need an admin GUI, however it would be neat if I could implemnt some kind of web-based control panel. It would be nothing pretty and purely for internal use. The control panel would do little more than dispay stats about the source feeds and possibly force refresh the input feeds in the event of a problem. This component is purely optional in the first delivery round.
A... less
I have completed my graduate public policy program but it was not at all tech heavy - some economics and econometrics but not requiring any CS knowledge. A good portion of the... moreI have completed my graduate public policy program but it was not at all tech heavy - some economics and econometrics but not requiring any CS knowledge. A good portion of the research jobs in DC require a basic level of programming knowledge. Mostly they want people who can perform advanced search and retrieval functions with large datasets and save stuff in different formats within their servers. And, they want STATA/stats knowledge, which I have some of.
My question is this: where is the best place to start learning some programming to get to this level? For instance, is Java, SQL, VBA or something else the best thing and most useful for these purposes? And, how much math do I need to write and run simple requests?
Thanks less
I am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS.
For this I am using Flume. I just started trying... moreI am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS.
For this I am using Flume. I just started trying it out. Succesfully created a node but now I am willing to setup a HTTP source and a sink that will write incoming requests over HTTP to local file.
Any suggesstions?
Thanks in Advance/
I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:WARN util.NativeCodeLoader: Unable to load... moreI'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableI'm running Hadoop 2.2.0.Doing a search online brought up this link: http://balanceandbreath.blogspot.ca/2013/01/utilnativecodeloader-unable-to-load.htmlHowever, the contents of /native/ directory on hadoop 2.x appear to be different so I am not sure what to do.I've also added these two environment variables in hadoop-env.sh:export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"Any ideas? less