RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling... moreI keep running into this error:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
I had searched in Pytorch forum, but still can’t find out what I have done wrong in my custom loss function. My model is nn.GRU, and here is my custom loss function:
def _loss(outputs, session, items): # `items` is a dict() contains embedding of all items
def f(output, target):
pos = torch.from_numpy(np.array([items[target)).float()
neg = torch.from_numpy(np.array([items for idx in target if idx != target)).float()
if USE_CUDA:
pos, neg = pos.cuda(), neg.cuda()
pos, neg = Variable(pos), Variable(neg)
I have constructed a CLDNN (Convolutional, LSTM, Deep Neural Network) structure for raw signal classification task.
Each training epoch runs for about 90 seconds and the... moreI have constructed a CLDNN (Convolutional, LSTM, Deep Neural Network) structure for raw signal classification task.
Each training epoch runs for about 90 seconds and the hyperparameters seems to be very difficult to optimize.
I have been research various ways to optimize the hyperparameters (e.g. random or grid search) and found out about Bayesian Optimization.
Although I am still not fully understanding the optimization algorithm, I feed like it will help me greatly.
I would like to ask few questions regarding the optimization task.
How do I set up the Bayesian Optimization with regards to a deep network?(What is the cost function we are trying to optimize?)
What is the function I am trying to optimize? Is it the cost of the validation set after N epochs?
Is spearmint a good starting point for this task? Any other suggestions for this task?
I would greatly appreciate any insights into this problem. less
I am trying to understand what would be the best big data solution for reporting purposes?
Currently I narrowed it down to HBase vs Hive.
The use case is that we have hundreds of... moreI am trying to understand what would be the best big data solution for reporting purposes?
Currently I narrowed it down to HBase vs Hive.
The use case is that we have hundreds of terabytes of data with hundreds different files. The data is live and gets updated all the time. We need to provide the most efficient way to do reporting. We have dozens different reports pages where each report consist of different type of numeric and graph data. For instance:
Show all users that logged in to the system in the last hour and their origin is US.
Show a graph with the most played games to the least played games.
From all users in the system show the percentage of paying vs non paying users.
For a given user, show his entire history. How many games he played? What kind of games he played. What was his score in each and every game?
The way I see it, there are 3 solutions:
Store all data in Hadoop and do the queries in Hive. This might work but I am not sure about the performance. How will it perform when the... less
I have recently deployed Big Data cluster. In that I've used Apache Kafka and zookeeper. But still I didn't understand about its usage in cluster. When both are required and for... moreI have recently deployed Big Data cluster. In that I've used Apache Kafka and zookeeper. But still I didn't understand about its usage in cluster. When both are required and for what purpose?
I'm working on machine learning problem and want to use linear regression as learning algorithm. I have implemented 2 different methods to find parameters theta of linear... moreI'm working on machine learning problem and want to use linear regression as learning algorithm. I have implemented 2 different methods to find parameters theta of linear regression model: Gradient (steepest) descent and Normal equation. On the same data they should both give approximately equal theta vector. However they do not.
Both theta vectors are very similar on all elements but the first one. That is the one used to multiply vector of all 1 added to the data.
Here is how the thetas look like (fist column is output of Gradient descent, second output of Normal equation):
Grad desc Norm eq -237.7752 -4.6736 -5.8471 -5.8467 9.9174 9.9178 2.1135 2.1134 -1.5001 -1.5003 -37.8558 -37.8505 -1.1024 -1.1116 -19.2969 -19.2956 66.6423 66.6447 297.3666 296.7604 -741.9281 -744.1541 296.4649 296.3494 146.0304 144.4158 -2.9978 -2.9976 -0.8190 -0.8189
What can cause the difference in theta(1, 1) returned by gradient descent compared to theta(1, 1) returned by normal equation? Do I have bug in my... less
I have an Express Node.js application, but I also have a machine learning algorithm to use in Python. Is there a way I can call Python functions from my Node.js application to... moreI have an Express Node.js application, but I also have a machine learning algorithm to use in Python. Is there a way I can call Python functions from my Node.js application to make use of the power of machine learning libraries?
This may not be the type of question to ask on SO, but just wanted to hear what about other people have to say regarding what factors to consider in implementing machine-learning... moreThis may not be the type of question to ask on SO, but just wanted to hear what about other people have to say regarding what factors to consider in implementing machine-learning algorithms in a large enterprise environment.
One of my goals is to research industry machine-learning solutions that can be tailored to my company's specific needs. Being pretty much the only person who has a math background on my team and and who has done some background reading on machine-learning algorithms previously, I'm tasked with explaining/comparing machine-learning solutions in the industry. From what I've gleaned by googling around, it seems that:
a. Machine-learning and predictive analytics aren't exactly the same thing, so what's inherently different when a company offers predictive analytics software vs. machine-learning software? (e.g. IBM Predictive Analytics vs. Skytree Server)
b. A lot of popular terminology often gets entangled together, especially regarding Big Data, Hadoop, machine-learning, etc. Could... less
what is the benefit of using Gradient Descent in the linear regression space? looks like the we can solve the problem (finding theta0-n that minimum the cost func) with analytical... morewhat is the benefit of using Gradient Descent in the linear regression space? looks like the we can solve the problem (finding theta0-n that minimum the cost func) with analytical method so why we still want to use gradient descent to do the same thing?
When I trained my neural network with Theano or Tensorflow, they will report a variable called "loss" per epoch.
How should I interpret this variable? Higher loss is better or... moreWhen I trained my neural network with Theano or Tensorflow, they will report a variable called "loss" per epoch.
How should I interpret this variable? Higher loss is better or worse, or what does it mean for the final performance (accuracy) of my neural network?
I have five text files that I input to a CountVectorizer. When specifying min_df and max_df to the CountVectorizer instance what does the min/max document frequency exactly means?... moreI have five text files that I input to a CountVectorizer. When specifying min_df and max_df to the CountVectorizer instance what does the min/max document frequency exactly means? Is it the frequency of a word in its particular text file or is it the frequency of the word in the entire overall corpus (5 txt files)?
How is it different when min_df and max_df are provided as integers or as floats?
The documentation doesn't seem to provide a thorough explanation nor does it supply an example to demonstrate the use of min_df and/or max_df. Could someone provide an explanation or example demonstrating min_df or max_df. less
Can any one tell my what that part (town = thisLine)exactly do?
def get_list_of_university_towns():
'''Returns a DataFrame of towns and the states they are in from the... moreCan any one tell my what that part (town = thisLine)exactly do?
def get_list_of_university_towns():
'''Returns a DataFrame of towns and the states they are in from the
university_towns.txt list. The format of the DataFrame should be:
DataFrame( [ , ,
columns= )
The following cleaning needs to be done:
1. For "State", removing characters from "[" to the end.
2. For "RegionName", when applicable, removing every character from " (" to the end.
3. Depending on how you read the data, you may need to remove newline character '\n'. '''
data =
state = None
state_towns =
with open('university_towns.txt') as file:
for line in file:
thisLine = line
if thisLine == '':
state = thisLine
continue
if '(' in line:
town = thisLine
state_towns.append()
else:
town = thisLine
state_towns.append()
data.append(thisLine)
df = pd.DataFrame(state_towns,columns = )
return df
get_list_of_university_towns() less
I am studying R end Data Science. In a question, I need to validate if a number in an array is even.
My code:
vetor <- list(c(1, 2, 3, 4, 5, 6, 7, 8, 9,... moreI am studying R end Data Science. In a question, I need to validate if a number in an array is even.
My code:
vetor <- list(c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
for (i in vetor) {
if (i %% 2 == 0) {
print(i)
}
}
But the result is a warning message:
Warning message:
In if (i%%2 == 0) { :
a condição tem comprimento > 1 e somente o primeiro elemento será usado
Translating:
The condition has a length > 1 and only the first element will be used.
What I need, that each element in a list be verified if is even, and if true, then, print it.
In R, how can I do it? less
I am incredibly new to the topic of websockets and am trying to figure out the correct way to handle the communication between a device and a server.
Here is my scenario: I have a... moreI am incredibly new to the topic of websockets and am trying to figure out the correct way to handle the communication between a device and a server.
Here is my scenario: I have a thermostat (very similar to the Nest) that needs to communicate with a web server. Every time you change the temperature on the thermostat, I need to send data to the web server to update it's "current stats" in the database. Easy, I can do that.
The part that I am confused about, and think websockets might be a use-case is when the user changes the temperature from the web interface. The thermostat has to pull in that information from the server to see "Oh, okay you want it to be 66 degrees."
I've thought of having the thermostat long-polling the server every 2-5 seconds to see what the "current stats" are in the database to change the temperature, but that seems like overkill.
Is there a way to open a connection between the thermostat and the server to listen for messages?
I began down the road of reading about... less
Any browser that encounter the text www.domain.com or http://domain.com/etc/ in a text section of some html will automatically translate it into <a... moreAny browser that encounter the text www.domain.com or http://domain.com/etc/ in a text section of some html will automatically translate it into <a href="http://www.domain.com">www.domain.com</a> or <a href="http://domain.com/etc/">http://domain.com/etc/</a> tag. I have to clean-up and verify some texts like this and do this replacement automatically, but the problem is that i can't insert new tags into an element's string.
#!/usr/bin/python
# -*- coding: utf8
import re
from bs4 import BeautifulSoup as bs
We have been training a neural net on the AI engine with a data-set consisting of 96 000 000 data points. The neural net was trained in a distributed manner, and as customary we... moreWe have been training a neural net on the AI engine with a data-set consisting of 96 000 000 data points. The neural net was trained in a distributed manner, and as customary we used 20 % of the data-set as evaluation data. In order to train distributed we used TensorFlow estimators and the method tf.estimator.train_and_evaluate. Since our data-set is very large, our evaluation set is also quite large. Looking into the cpu usage of the master vs the workers nodes, and testing with an evaluation data-set consisting of only 100 samples, it appears as though the evaluation is not distributed and happens only on the master node. This makes the amount of ML units consumed increase by a factor of approximately 5 between having the standard size evaluation data (20 % of the total data) and only having 100 data points for evaluation, while the amount of training data is the same.We see two possible solutions to this problem:Doing also the evaluation distributed, but is that technically possible on the AI... less
I have a simple ETL process in an Azure environmentblob storage > datafactory > datalake raw > databricks > datalake curated > datwarehouse(main ETL).the datasets for this... more
I have a simple ETL process in an Azure environmentblob storage > datafactory > datalake raw > databricks > datalake curated > datwarehouse(main ETL).the datasets for this project are not very big (~1 million rows 20 columns give or take) however I would like to keep them partitioned properly in my datalake as Parquet files.currently I run some simple logic to figure where in my lake each file should sit based off business calendars.the files vaguely looks like this
Year Week Data
2019 01 XXX
2019 02 XXX
I then partition a given file into the following format replacing data that exists and creating new folders for new data.
curated ---
dataset --
Year 2019
- Week 01 - file.pq + metadata
- Week 02 - file.pq + metadata
- Week 03 - file.pq + datadata #(pre existing file)
the metadata are success and commits that are auto generated.
to this end i use the following query in... less
How to save/restore a model after training? (26... moreThis question already has answers here:
How to save/restore a model after training? (26 answers)
Closed 3 years ago.
I'm relatively new to machine learning and the Tensorflow framework. I was trying to take my trained model heavily influenced by the code presented here, using the MNIST handwritten digit dataset and perform inferences on testing examples that I have created. However, I am doing the training on a remote machine with a GPU and am trying to save the data to a directory so that I can transfer the data and inference on a local machine
It seems that I was able to save some of the model with tf.saved_model.simple_save, however, I'm unsure of how to use the saved data to do inferencing and to use the data to make a prediction given a new image. It seems like there are multiple ways to save a model, but I am unsure of what the convention or of what the "correct way" is to do it with the Tensorflow framwork.
So far, this is the line that I think I would need, but am unsure if it is... less
This is a follow up question to: ruby variable scoping across classes. The solution makes sense to me conceptually, but I can't get it to work. Thought maybe with more code... moreThis is a follow up question to: ruby variable scoping across classes. The solution makes sense to me conceptually, but I can't get it to work. Thought maybe with more code someone could help me.I have a class Login that declares a new IMAP class, authenticates, and picks a mailbox.I then am trying to create a separate class that will "do stuff" in the mailbox. For example, calculate the number of emails received. The problem is that the @imap instance of Net::IMAP doesn't pass from the Login class to the Stat class -- I'm getting no method errors for imap.search in the new class. I don't want to re-log in and re-authenticate each time I need to "do some stuff" with the mailbox. I'm trying to implement the solution in the other thread, but can't seem to get it to work.Here's the Login class:
class Login
def initialize(user, domain, pass)
@username = user
@domain = domain
@pass = pass
#check if gmail or other domain
gmail_test = @domain.include? "gmail.com"
if gmail_test ==... less
I am using Django to receive and process push notifications from the foursquare real-time api. Each checkin is pushed as a POST request to my server containing a single parameter... moreI am using Django to receive and process push notifications from the foursquare real-time api. Each checkin is pushed as a POST request to my server containing a single parameter named checkin. I am trying to grab the value of the checkin parameter and convert it to a python dict. However, calling json.loads always results in the following error:
NameError: name 'true' is not defined
I know the json is valid, so I must be doing something wrong.
The code is:
import json
def push(request):
if request.is_secure():
checkin_json = request.POST
checkin = json.load(request.POST)
The body of the post request is:
"checkin =
{
"id": "4e6fe1404b90c00032eeac34",
"createdAt": 1315955008,
"type": "checkin",
"timeZone": "America/New_York",
"user": {
"id": "1",
"firstName": "Jimmy",
"lastName": "Foursquare",
"photo": "https://foursquare.com/img/blank_boy.png",
"gender": "male",
"homeCity": "New York, NY",
"relationship": "self"
},
"venue": {
"id":... less