For example, I'm trying to understand this code
def _get_child_candidates(self, distance, min_dist, max_dist): if self._leftchild and distance - max_dist < self._median: yield... moreFor example, I'm trying to understand this code
def _get_child_candidates(self, distance, min_dist, max_dist): if self._leftchild and distance - max_dist < self._median: yield self._leftchild if self._rightchild and distance + max_dist >= self._median: yield self._rightchild
And this is the caller:
result, candidates = , while candidates: node = candidates.pop() distance = node._get_dist(obj) if distance <= max_dist and distance >= min_dist: result.extend(node._values) candidates.extend(node._get_child_candidates(distance, min_dist, max_dist)) return result
What happens when the method _get_child_candidates is called? Is a list returned? A single element? Is it called again? When will subsequent calls stop? less
And when I check my GPU memory usage, around 90% of it gets... moreCreating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN Black, pci bus id: 0000:01:00.0)
And when I check my GPU memory usage, around 90% of it gets consumed.
Tensorflow documentation does not say anything about this. Does it take control of the gpu ? Why does it consume most of the memory ?
I'm currently trying to get an ANN to play a video game and and I was hoping to get some help from the wonderful community here.
I've settled on Diablo 2. Game play is thus in... moreI'm currently trying to get an ANN to play a video game and and I was hoping to get some help from the wonderful community here.
I've settled on Diablo 2. Game play is thus in real-time and from an isometric viewpoint, with the player controlling a single avatar whom the camera is centered on.
To make things concrete, the task is to get your character x experience points without having its health drop to 0, where experience point are gained through killing monsters. Here is an example of the gameplay:
Now, since I want the net to operate based solely on the information it gets from the pixels on the screen, it must learn a very rich representation in order to play efficiently, since this would presumably require it to know (implicitly at least) how divide the game world up into objects and how to interact with them.
And all of this information must be taught to the net somehow. I can't for the life of me think of how to train this thing. My only idea is have a separate program visually extract... less
Can docker containers be used along with UI based RPA tools like blueprism or uiPath? Blueprism recommends using virtual machines but offers no support of docker
This question pops up in my mind as in many developing countries at present internet connectivity is very poor or no connectivity at all and the customer base is very huge, in... moreThis question pops up in my mind as in many developing countries at present internet connectivity is very poor or no connectivity at all and the customer base is very huge, in this case how IoT can help in making life easy?
I installed the latest version of Python (3.6.4 64-bit) and the latest version of PyCharm (2017.3.3 64-bit). Then I installed some modules in PyCharm (Numpy, Pandas, etc), but... moreI installed the latest version of Python (3.6.4 64-bit) and the latest version of PyCharm (2017.3.3 64-bit). Then I installed some modules in PyCharm (Numpy, Pandas, etc), but when I tried installing Tensorflow it didn't install, and I got the error message:Could not find a version that satisfies the requirement TensorFlow (from versions: ) No matching distribution found for TensorFlow.Then I tried installing TensorFlow from the command prompt and I got the same error message. I did however successfully install tflearn.I also installed Python 2.7, but I got the same error message again. I googled the error and tried some of the things which were suggested to other people, but nothing worked (this included installing Flask).How can I install Tensorflow? Thanks. less
Is there an online tool or an app that can create a graph or an image of an imported database that shows the relationship between the tables inside it?
ETL on source data from Excel, Access, Sql Server '8, .txt... moreBackground:
ETL on source data from Excel, Access, Sql Server '8, .txt files.
Data Cloud is created
Dashboard is in progress
I have searched online because I remember seeing a marketting demo video by QlikView that it's possible to share the dashboard among other users. Not just a snapshot image or pdf. The real dashboard as a working file.
If client pcs receive a link to connect to the same data cloud via web - that's easy. But what I want to know, is it possible to package and "port" the entire working file with underlying data to another person? (I am not asking for zipping!) less
Nathan Marz in his book "Big Data" describes how to maintain files of data in HDFS and how to optimize files' sizes to be as near native HDFS block size as possible using... moreNathan Marz in his book "Big Data" describes how to maintain files of data in HDFS and how to optimize files' sizes to be as near native HDFS block size as possible using his Pail library running on top of Map Reduce.
Is it possible to achieve the same result in Google Cloud Storage?
Can I use Google Cloud Dataflow instead of MapReduce for this purpose?
I'm using Simba's ODBC driver for connecting Tableau to MongoDB. I connected my workbook to MongoDB, developed my dashboard and everything went pretty smooth. However, after I... moreI'm using Simba's ODBC driver for connecting Tableau to MongoDB. I connected my workbook to MongoDB, developed my dashboard and everything went pretty smooth. However, after I publish my workbook to Tableau server and try to open it, I get an error message:
"An unexpected error occurred. If you continue to receive this error please contact your Tableau Server Administrator. Tableau Server encountered an error. Contact your Tableau Server Administrator. Would you like to reset the view?"
Why can't I view this dashboard in Tableau Server? less
I have purchased Tableau Online to share my workbooks with a specific group of people without providing an option to download the actual data I used to create them. But I figured... moreI have purchased Tableau Online to share my workbooks with a specific group of people without providing an option to download the actual data I used to create them. But I figured that other users would also need a license to view/edit my workbooks..
My initial plan was to share my published workbooks and put them on my website so that others (that I specify) can VIEW the workbook without offering them a privilege of downloading the data. Any suggestion will be greatly appreciated...
First up, this is most certainly homework (so no full code samples please). That said...
I need to test an unsupervised algorithm next to a supervised algorithm, using the Neural... moreFirst up, this is most certainly homework (so no full code samples please). That said...
I need to test an unsupervised algorithm next to a supervised algorithm, using the Neural Network toolbox in Matlab. The data set is the UCI Artificial Characters Database. The problem is, I've had a good tutorial on supervised algorithms, and been left to sink on unsupervised.
So I know how to create a self organising map using selforgmap, and then I train it using train(net, trainingSet). I don't understand what to do next. I know that it's clustered the data I gave it into (hopefully) 10 clusters (one for each letter).
Two questions then:
How can I then label the clusters (given that I have a comparison pattern)?
Am I trying to turn this into a supervised learning problem when I do this?
How can I create a confusion matrix on (another) testing set to compare to the supervised algorithm?
I think I'm missing something conceptual or jargon-based here - all my searches come up with supervised learning... less
I installed Spark using the AWS EC2 guide and I can launch the program fine using the bin/pyspark script to get to the spark prompt and can also do the Quick Start quide... moreI installed Spark using the AWS EC2 guide and I can launch the program fine using the bin/pyspark script to get to the spark prompt and can also do the Quick Start quide successfully.However, I cannot for the life of me figure out how to stop all of the verbose INFO logging after each command.I have tried nearly every possible scenario in the below code (commenting out, setting to OFF) within my log4j.properties file in the conf folder in where I launch the application from as well as on each node and nothing is doing anything. I still get the logging INFO statements printing after executing each statement.I am very confused with how this is supposed to work.
#Set everything to be logged to the console log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout... less
I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:WARN util.NativeCodeLoader: Unable to load... moreI'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableI'm running Hadoop 2.2.0.Doing a search online brought up this link: http://balanceandbreath.blogspot.ca/2013/01/utilnativecodeloader-unable-to-load.htmlHowever, the contents of /native/ directory on hadoop 2.x appear to be different so I am not sure what to do.I've also added these two environment variables in hadoop-env.sh:export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"Any ideas? less
I just have a bunch of dimensions in rows, so its just a table of information and I need to hide a column of data because its confidential, but I need the data there because it... moreI just have a bunch of dimensions in rows, so its just a table of information and I need to hide a column of data because its confidential, but I need the data there because it affects the layout of the rest of the table.
I have a field such... moreI have a field such that
Color
Pink
Blue
Pink
Blue
Pink
Pink
Pink
Pink
Pink
Blue
Blue
Pink
Pink
Pink
Blue
Pink
Blue
Pink
Blue
How to create calculated field to count number of times Pink appears?
Total numbers of Blue?
Coming from a programming background where you write code, test, deploy, run.. I'm trying to wrap my head around the concept of "training a model" or a "trained model" in data... moreComing from a programming background where you write code, test, deploy, run.. I'm trying to wrap my head around the concept of "training a model" or a "trained model" in data science, and deploying that trained model.
I'm not really concerned about the deployment environment, automation, etc.. I'm trying to understand the deployment unit.. a trained model. What does a trained model look like on a file system, what does it contain?
I understand the concept of training a model, and splitting a set of data into a training set and testing set, but lets say I have a notebook (python / jupyter) and I load in some data, split between training/testing data, and run an algorithm to "train" my model. What is my deliverable under the hood? While I'm training a model I'd think there'd be a certain amount of data being stored in memory.. so how does that become part of the trained model? It obviously can't contain all the data used for training; so for instance if I'm training a chatbot agent (retrieval-based),... less
Task: We have to setup a periodic sync of records from Spanner to Big Query. Our Spanner database has a relational table hierarchy.
Option Considered I was thinking of using... moreTask: We have to setup a periodic sync of records from Spanner to Big Query. Our Spanner database has a relational table hierarchy.
Option Considered I was thinking of using Dataflow templates to setup this data pipeline.
Option1: Setup a job with Dataflow template 'Cloud Spanner to Cloud Storage Text' and then another with Dataflow template 'Cloud Storage Text to BigQuery'. Con: The first template works only on a single table and we have many tables to export.
Option2: Use 'Cloud Spanner to Cloud Storage Avro' template which exports the entire database. Con: I only need to export selected tables within a database and I don't see a template to import Avro into Big Query.
Questions: Please suggest what is the best option for setting up this pipeline less
I'm learning object oriented programing in a data science context.
I want to understand what good practice is in terms of writing methods within a class that relate to one... moreI'm learning object oriented programing in a data science context.
I want to understand what good practice is in terms of writing methods within a class that relate to one another.
When I run my code:
import pandas as pd
pd.options.mode.chained_assignment = None
class MyData:
def __init__(self, file_path):
self.file_path = file_path
def prepper_fun(self):
'''Reads in an excel sheet, gets rid of missing values and sets datatype to numerical'''
df = pd.read_excel(self.file_path)
df = df.dropna()
df = df.apply(pd.to_numeric)
self.df = df
return(df)
def quality_fun(self):
'''Checks if any value in any column is more than 10. If it is, the value is replaced with
a warning 'check the original data value'.'''
for col in self.df.columns:
for row in self.df.index:
if self.df > 10:
self.df = str('check original data value')
return(self.df)
I'm just wondering what is the difference between an RDD and DataFrame (Spark 2.0.0 DataFrame is a mere type alias for Dataset) in Apache Spark?
Can you convert one to the other?
I have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as... moreI have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as above:
So It will result a (2656, 9683) Dataframe. Thanks for any help.
Typo on image: on Dataframe 3, it will 7025, not 5668.
I am currently starting a project titled "Cloud computing for time series mining algorithms using Hadoop". The data which I have is hdf files of size over a terabyte.In hadoop as... moreI am currently starting a project titled "Cloud computing for time series mining algorithms using Hadoop". The data which I have is hdf files of size over a terabyte.In hadoop as I know that we should have text files as input for further processing (map-reduce task). So I have one option that I convert all my .hdf files to text files which is going to take a lot of time.
Or I find a way of how to use raw hdf files in map reduce programmes. So far I have not been successful in finding any java code which reads hdf files and extract data from them. If somebody has a better idea of how to work with hdf files I will really appreciate such help.
Thanks Ayush less