I used Azure IoT Hub earlier and now I found a new topic Azure IoT Central which looks as same as the Azure IoT Hub.I am confused with the difference between these IoT Services,... moreI used Azure IoT Hub earlier and now I found a new topic Azure IoT Central which looks as same as the Azure IoT Hub.I am confused with the difference between these IoT Services, Can anyone explain me the difference and which one is better between Azure IoT Central and Azure IoT Hub?Thanks in advance
I used Azure IoT Hub earlier and now I found a new topic Azure IoT Central which looks as same as the Azure IoT Hub.I am confused with the difference between these IoT Services,... moreI used Azure IoT Hub earlier and now I found a new topic Azure IoT Central which looks as same as the Azure IoT Hub.I am confused with the difference between these IoT Services, Can anyone explain me the difference and which one is better between Azure IoT Central and Azure IoT Hub?Thanks in advance
I would like to remove specific characters from strings within a vector, similar to the Find and Replace feature in Excel.
Here are the data I start with:
group <-... moreI would like to remove specific characters from strings within a vector, similar to the Find and Replace feature in Excel.
Here are the data I start with:
group <- data.frame(c("12357e", "12575e", "197e18", "e18947")
I start with just the first column; I want to produce the second column by removing the e's:
group group.no.e
12357e 12357
12575e 12575
197e18 19718
e18947 18947
I never got a chance to work on Impala. I have just started reading about Impala. But i have one basic question which i am not clear about Impala. Impala has its own demons so it... moreI never got a chance to work on Impala. I have just started reading about Impala. But i have one basic question which i am not clear about Impala. Impala has its own demons so it also has its own execution engine or it works on MapR or other execution engine. Thanks in advance
I am using spark-csv to load data into a DataFrame. I want to do a simple query and display the content:
val df =... moreI am using spark-csv to load data into a DataFrame. I want to do a simple query and display the content:
val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("my.csv")
df.registerTempTable("tasks")
results = sqlContext.sql("select col from tasks");
results.show()
I am a beginner of Tableau. I have gone through the Trusted Authentication on Tableau online help.
But I found that to add the web server IP address... moreI am a beginner of Tableau. I have gone through the Trusted Authentication on Tableau online help.
But I found that to add the web server IP address to https://tsm-computer-name:8850. And I found that Tableau Server domain and Ip address are also required.
So I just want to know the following questions.
Are Tableau Server and Tableau Service Management on different computers? If so, where should TSM be installed and how? How do I create a TSM credential (admin)?
I am trying to do some deep learning work. For this, I first installed all the packages for deep learning in my Python environment.Here is what I did.In Anaconda, I created an... moreI am trying to do some deep learning work. For this, I first installed all the packages for deep learning in my Python environment.Here is what I did.In Anaconda, I created an environment called tensorflow as follows
conda create -n tensorflow
Then installed the data science Python packages, like Pandas, NumPy, etc., inside it. I also installed TensorFlow and Keras there. Here is the list of packages in that environment
(tensorflow) SFOM00618927A:dl i854319$ conda list
# packages in environment at /Users/i854319/anaconda/envs/tensorflow:
#
appdirs 1.4.3 <pip>
appnope 0.1.0 py36_0
beautifulsoup4 4.5.3 py36_0
bleach 1.5.0 py36_0
cycler 0.10.0 py36_0
decorator 4.0.11 py36_0
entrypoints 0.2.2 py36_1
freetype 2.5.5 ... less
I'm new to TensorFlow and Data Science. I made a simple module that should figure out the relationship between input and output numbers. In this case, x and x squared. The code in... moreI'm new to TensorFlow and Data Science. I made a simple module that should figure out the relationship between input and output numbers. In this case, x and x squared. The code in Python:
import numpy as np
import tensorflow as tf
# TensorFlow only log error messages.
tf.logging.set_verbosity(tf.logging.ERROR)
model = tf.keras.Sequential([
tf.keras.layers.Dense(units = 1, input_shape = )
model.compile(loss = "mean_squared_error", optimizer = tf.keras.optimizers.Adam(0.0001))
model.fit(features, labels, epochs = 50000, verbose = False)
print(model.predict())
I tried a different number of units, and adding more layers, and even using the relu activation function, but the results were always wrong. It works with other relationships like x and 2x. What is the problem here? less
Docker data volumes is living within the host or boot2docker on the local VM(boot2docker).
Having big data from mongoDB running in a data container and mongoDB in another seems to... moreDocker data volumes is living within the host or boot2docker on the local VM(boot2docker).
Having big data from mongoDB running in a data container and mongoDB in another seems to be the way. Will this scale on Google Cloud Engine or Azure Virtuel Machines or other. I mean if all this is running within ONE Virtuel Machine, like boot2docker or other in the cloud. Normally you would scale VM's by creating new instances of VM's but how is this possible with Docker?
Sorry to ask this on StackExchange, but there is no category on dba - but StackExchange has a Category for Docker. less
I have posted this question on Data Science StackExchange site since StackOverflow does not support LaTeX. Linking it here because this site is probably more appropriate.The... moreI have posted this question on Data Science StackExchange site since StackOverflow does not support LaTeX. Linking it here because this site is probably more appropriate.The question with correctly rendered LaTeX is here: https://datascience.stackexchange.com/questions/48062/pytorch-does-not-seem-to-be-optimizing-correctlyThe idea is that I am considering sums of sine waves with different phases. The waves are sampled with some sample rate s in the interval . I need to select phases in such a way, that the sum of the waves at any sample point is minimized.Below is the Python code. Optimization does not seem to be computed correctly.
import numpy as np
import torch
I have created a data science neural network in Python anaconda Spyder. My projects have multiple .py files and I have trained this model for 10 days and weights are generated. I... moreI have created a data science neural network in Python anaconda Spyder. My projects have multiple .py files and I have trained this model for 10 days and weights are generated. I want to use this model as a service and want to deploy it to Azure for consumption. I tried following but facing difficulties -
1) I tried deploying this as "Execute Python Script" in Azure ML studio but I am not finding an option to upload the weights. I understand I can zip all the .py files but what about the trained weights and virtual environment (I am using an old version of tensorflow)?
2) I am seeing an option of creating a Jupyter notebook but my project is created in Spyder and doesn't have .ipynb files. Is there any way to convert my .py files into .ipynb files? ALso, I have created a virtual environment with the older version of tensorflow and python version? How to take care of this while deploying to azure?
3) I tried deploying this to azure as a python web app but again what shall I do with the virtual... less
I am wanting to start a data warehouse in Google Big Query but I'm not sure how to actually schedule jobs to get the data into the cloud.
To give some background. I have a MySQL... moreI am wanting to start a data warehouse in Google Big Query but I'm not sure how to actually schedule jobs to get the data into the cloud.
To give some background. I have a MySQL database hosted on-prem which I currently take a demp of each night as a backup. My idea is that I can send this dump to the Google Cloud and have it import the data into Big Query. I have thought that I could send the dump and probably use a cloud scheduler function to then run something that opens the dump and does this but I'm unsure how these services all fit together.
I'm a bit of a newby with the Google Cloud so if there is a better way to achieve this then I'm happy to change my plan of action.
Thanks in advance. less
I started learning the concepts of machine learning.
I would like to know some of the use case scenarios for... moreI started learning the concepts of machine learning.
I would like to know some of the use case scenarios for below
Can any one tell my what that part (town = thisLine)exactly do?
def get_list_of_university_towns():
'''Returns a DataFrame of towns and the states they are in from the... moreCan any one tell my what that part (town = thisLine)exactly do?
def get_list_of_university_towns():
'''Returns a DataFrame of towns and the states they are in from the
university_towns.txt list. The format of the DataFrame should be:
DataFrame( [ , ,
columns= )
The following cleaning needs to be done:
1. For "State", removing characters from "[" to the end.
2. For "RegionName", when applicable, removing every character from " (" to the end.
3. Depending on how you read the data, you may need to remove newline character '\n'. '''
data =
state = None
state_towns =
with open('university_towns.txt') as file:
for line in file:
thisLine = line
if thisLine == '':
state = thisLine
continue
if '(' in line:
town = thisLine
state_towns.append()
else:
town = thisLine
state_towns.append()
data.append(thisLine)
df = pd.DataFrame(state_towns,columns = )
return df
I'm trying to install ROracle package on a unix box. The package gets installed properly. But library(ROracle) does not work fine with the error
library(ROracle)
Error in... moreI'm trying to install ROracle package on a unix box. The package gets installed properly. But library(ROracle) does not work fine with the error
library(ROracle)
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/u01/group1/home/oracle/R/x86_64-redhat-linux-gnu-library/3.1/ROracle/libs/ROracle.so':
libclntsh.so.11.1: cannot open shared object file: No such file or directory
Error: package or namespace load failed for ‘ROracle’
The package installs fine from the command line , but just does not work in R studio. I went through lot of threads in forum and lot of them suggested to export the LD_LIBRARY_PATH and reset it.infact i went ahead and copied all the R system variables from command line into R Studio. But it still does not work out fine.
One thing i have also noticed is that the R system variables change every time i restart R studio. Can it be the problem that R studio is not taking path values correctly. less
I am learning assembly language in my spare time to become a better developer.
I understand the difference between stack-based machines and register-based machines at a conceptual... moreI am learning assembly language in my spare time to become a better developer.
I understand the difference between stack-based machines and register-based machines at a conceptual level, but I am wondering how stack-based machines are actually implemented. If a virtual machine, e.g. JVM or .NET, runs on a register-based architecture, e.g. x86 or x64, then it must use registers at the assembly level (as far as I am concerned). I am obviously missing something here. Therefore I am unsure of the distinction at assembly language.
I have read articles on here e.g. Stack-based machine depends on a register-based machine? and also on Wikipedia, but I don't believe they answer my question directly. less
Need help with tableau .tdsx usage.
Current Process: - We publish and refresh data sources (packaged .tdsx) to the tableau server. Our customers download the data sources and... moreNeed help with tableau .tdsx usage.
Current Process: - We publish and refresh data sources (packaged .tdsx) to the tableau server. Our customers download the data sources and build ad hoc reports for their business needs.
Problem: - If the customer already build some views last month, is there a way to download the latest ‘.tdsx’ file and use it in the existing workbook were they created the views.Today, customer is re-creating the views using each month updated ‘.tdsx’ files.
Can you suggest a better way to achieve this? I can provide more details if it is needed.
Note: - Publishing a packaged workbook (.twbx) with views in it is not feasible as the report requirement varies most of the months. less
I'm interesting in getting the connection from Python to machine learning part of OpenCV 2.2. OpenCV 2.2 already includes python bindings but only to the computer vision (cv) part... moreI'm interesting in getting the connection from Python to machine learning part of OpenCV 2.2. OpenCV 2.2 already includes python bindings but only to the computer vision (cv) part of it and not to the machine learning (ml) part.
Where could I get some third party bindings to also have access to the machine learning part
I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset,... moreI am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other versions of it (Flipping, Cropping...etc). But that doesn't seem like happening in PyTorch. As far as I understood from the references, when we use data.transforms in PyTorch, then it applies them one by one. So for example:
data_transforms = {
'train': transforms.Compose([
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(, ),
'val': transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(, ),
}
Here , for the training, we are first randomly cropping the image and resizing it to shape (224,224). Then we are taking these (224,224) images and horizontally flipping them. Therefore, our dataset is now... less
For now I'm using early stopping in Keras like this:
X,y= load_data('train_data') X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=12) datagen... moreFor now I'm using early stopping in Keras like this:
X,y= load_data('train_data') X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=12) datagen = ImageDataGenerator( horizontal_flip=True, vertical_flip=True) early_stopping_callback = EarlyStopping(monitor='val_loss', patience=epochs_to_wait_for_improve) history = model.fit_generator(datagen.flow(X_train, y_train, batch_size=batch_size), steps_per_epoch=len(X_train) / batch_size, validation_data=(X_test, y_test), epochs=n_epochs, callbacks=)
But at the end of model.fit_generator it will save model after epochs_to_wait_for_improve, but I want to save model with min val_loss does it make sense and is it possible? less
First up, this is most certainly homework (so no full code samples please). That said...
I need to test an unsupervised algorithm next to a supervised algorithm, using the Neural... moreFirst up, this is most certainly homework (so no full code samples please). That said...
I need to test an unsupervised algorithm next to a supervised algorithm, using the Neural Network toolbox in Matlab. The data set is the UCI Artificial Characters Database. The problem is, I've had a good tutorial on supervised algorithms, and been left to sink on unsupervised.
So I know how to create a self organising map using selforgmap, and then I train it using train(net, trainingSet). I don't understand what to do next. I know that it's clustered the data I gave it into (hopefully) 10 clusters (one for each letter).
Two questions then:
How can I then label the clusters (given that I have a comparison pattern)?
Am I trying to turn this into a supervised learning problem when I do this?
How can I create a confusion matrix on (another) testing set to compare to the supervised algorithm?
I think I'm missing something conceptual or jargon-based here - all my searches come up with supervised learning... less
Are they supposed to be equal?
but, why the "hadoop fs" commands show the hdfs files while the "hdfs dfs" commands show the local files?
here is the hadoop version... moreAre they supposed to be equal?
but, why the "hadoop fs" commands show the hdfs files while the "hdfs dfs" commands show the local files?
here is the hadoop version information:
Hadoop 2.0.0-mr1-cdh4.2.1 Subversion git://ubuntu-slave07.jenkins.cloudera.com/var/lib/jenkins/workspace/CDH4.2.1-Packaging-MR1/build/cdh4/mr1/2.0.0-mr1-cdh4.2.1/source -r Compiled by jenkins on Mon Apr 22 10:48:26 PDT 2013
1) Users are prompted to login to Tableau when viewing an embedded dashboard within a web application.
2) If they close their browser, start a different browser session, or let... more1) Users are prompted to login to Tableau when viewing an embedded dashboard within a web application.
2) If they close their browser, start a different browser session, or let the Tableau cookie expire, they will be prompted to login again.
3) Throughout the day, you could potentially be prompted to login multiple times when trying to view dashboards. This quickly becomes annoying and tiresome.
Tableau offers a solution called "Trusted Authentication" which bypasses the manual login process. After a week of debugging and troubleshooting, I was able to accomplish this. I could not find any solutions on Stackoverflow, so I wanted to share my knowledge on how I accomplished this in hope to help others. less
I need to run my python script under Azure Machine Learning, using python=3.6.8 (not the default 3.6.2). I am using the AML "PyTorch()" Estimator, setting the "conda_packages"... moreI need to run my python script under Azure Machine Learning, using python=3.6.8 (not the default 3.6.2). I am using the AML "PyTorch()" Estimator, setting the "conda_packages" arg to .
I am relying on this doc page for the PyTorch Estimator:
I expected to see python 3.6.8, since I specified that in the PyTorch Estimator's conda_packages arg.
I also tried moving the "python==3.6.8" from conda_packages to pip_packages, but received an error saying pip could not locate that package.
FYI, I have another package specified in pip_packages, and that does get installed correctly during this process. It seems like the value of the "conda_packages" arg is not being used (I can find no mention of a conda or python install error in... less