There are 2 ways to get the output of an intermediate area, as posted here on this... moreThere are 2 ways to get the output of an intermediate area, as posted here on this forum:https://www.cluzters.ai/forums/topic/353/keras-how-to-get-the-output-of-each-layer?c=1597You don't need (or want) both of these, either works:
get_layer_output = K.function([model.layers, [model.layers)
layer_output = get_layer_output()
or
tmp_model = Model(model.layers.input, model.layers.output)
tmp_output = tmp_model.predict(x_train)
This works well for a smaller model, but I always get OOM if my model is too large. Is there an easy way to free the GPU memory of the original model before creating tmp_model or calling get_layer_output? I can save my weights to a text file and create another program in another session, but it seems like there should be an easier way. less
I am a beginner and I want to learn computer programming. So, for now, I have started learning Python by myself with some knowledge about programming in C and Fortran.
Now, I have... moreI am a beginner and I want to learn computer programming. So, for now, I have started learning Python by myself with some knowledge about programming in C and Fortran.
Now, I have installed Python version 3.6.0 and I have struggled finding a suitable text for learning Python in this version. Even the online lecture series ask for versions 2.7 and 2.5 .
Now that I have got a book which, however, makes codes in version 2 and tries to make it as close as possible in version 3 (according to the author); the author recommends "downloading Anaconda for Windows" for installing Python.
So, my question is: What is this 'Anaconda'? I saw that it was some open data science platform. What does it mean? Is it some editor or something like Pycharm, IDLE or something?
Also, I downloaded my Python (the one that I am using right now) for Windows from Python.org and I didn't need to install any "open data science platform". So what is this happening?
Please explain in easy language. I don't have too much knowledge about these. less
Good afternoon.
I have this question I am trying to solve using "panda" statistical data structures and related syntax from the Python scripting language. I am already graduated... moreGood afternoon.
I have this question I am trying to solve using "panda" statistical data structures and related syntax from the Python scripting language. I am already graduated from a US university and employed while currently taking the Coursera.org course of "Python for Data Science" just for professional development, which is offered online at Coursera's platform by the University of Michigan. I'm not sharing answers to anyone either as I abide by Coursera's Honor Code.
First, I was given this panda dataframe chart concerning Olympic medals won by countries around the world:
# Summer Gold Silver Bronze Total # Winter Gold.1 Silver.1 Bronze.1 Total.1 # Games Gold.2 Silver.2 Bronze.2 Combined total ID
I'm trying to build a Python Lambda to send images to TensorFlow Serving for inferences. I have at least two dependencies: CV2 and tensorflow_serving.apis. I've run multiple... moreI'm trying to build a Python Lambda to send images to TensorFlow Serving for inferences. I have at least two dependencies: CV2 and tensorflow_serving.apis. I've run multiple tutorials showing it's possible to run tensorflow in a lambda, but they provide the package to install and don't explain how they got it to fit in the limit of less than 256MB unzipped.
How to Deploy ... Lambda and TensorFlow
Using TensorFlow and the Serverless Framework...
I've tried following the official instructions for packaging but just this downloads 475MB of dependencies:
$ python -m pip install tensorflow-serving-api --target .
Collecting tensorflow-serving-api
Downloading https://files.pythonhosted.org/packages/79/69/1e724c0d98f12b12f9ad583a3df7750e14ec5f06069aa4be8d75a2ab9bb8/tensorflow_serving_api-1.12.0-py2.py3-none-any.whl
...
$ du -hs .
475M .
I see that others have fought this dragon and won (1) (2) by doing contortions to rip out all unused libraries from all dependencies or compile from scratch. But... less
I am trying to perform matrix multiplication of multiple matrices in PyTorch and was wondering what is the equivalent of numpy.linalg.multi_dot() in PyTorch?
If there isn't one,... moreI am trying to perform matrix multiplication of multiple matrices in PyTorch and was wondering what is the equivalent of numpy.linalg.multi_dot() in PyTorch?
If there isn't one, what is the next best way (in terms of speed and memory) I can do this in PyTorch?
Code:
import numpy as np
import torch
A = np.random.rand(3, 3)
B = np.random.rand(3, 3)
C = np.random.rand(3, 3)
I go to the PyTorch website and select the following options
PyTorch Build: Stable (1.2)
Your OS: Windows
Package:... moreI go to the PyTorch website and select the following options
PyTorch Build: Stable (1.2)
Your OS: Windows
Package: pip
Language: Python 3.7
CUDA: None
(All of these are correct)
Than it displays a command to run
pip3 install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
I have already tried to mix around the the different options but none of them has worked.
ERROR: ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2) ERROR: No matching distribution found for torch==1.2.0+cpu
I tried to do pip install pytorch but pytorch doesn't support pypi less
traceback:
Traceback (most recent call last):
File "/Users/soubhagyapradhan/Desktop/upwork/baby/data-science/env/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/Users/soubhagyapradhan/Desktop/upwork/baby/data-science/env/lib/python3.8/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/Users/soubhagyapradhan/Desktop/upwork/baby/data-science/env/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details =... less
I often want to quickly save some Python data, but I would also like to save it in a stable file format in case the date lingers for a long time. And so I have the question, how... moreI often want to quickly save some Python data, but I would also like to save it in a stable file format in case the date lingers for a long time. And so I have the question, how can I save my data?
In data science, there are three kinds of data I want to store -- arbitrary Python objects, numpy arrays, and Pandas dataframes. -- what are the stable ways of storing these?
I am really new to Data Science/ML and have been working on Tensorflow to implement Linear Regression on California Housing Prices from Kaggle.
I tried to train a mode in two... moreI am really new to Data Science/ML and have been working on Tensorflow to implement Linear Regression on California Housing Prices from Kaggle.
I tried to train a mode in two different ways:
Using a Sequential model
Custom implementation
In both cases, the loss of the model was really high and I have not been able to understand what are the ways to improve it.
Dataset prep
df = pd.read_csv('california-housing-prices.zip')
df = df
print('Shape of dataset before removing NAs and duplicates {}'.format(df.shape))
df.dropna(inplace=True)
df.drop_duplicates(inplace=True)
input_train, input_test, target_train, target_test = train_test_split(df.values, df.values, test_size=0.2)
scaler = MinMaxScaler()
input_train = input_train.reshape(-1,1)
input_test = input_test.reshape(-1,1)
input_train = scaler.fit_transform(input_train)
input_test = scaler.fit_transform(input_test)
target_train = target_train.reshape(-1,1)
target_train = scaler.fit_transform(target_train)
target_test =... less
I just got into SQL to do some data science and was wondering why my code was running but not affecting the MySQL database in any way. I am using pycharm and the MySQLdb module.
... moreI just got into SQL to do some data science and was wondering why my code was running but not affecting the MySQL database in any way. I am using pycharm and the MySQLdb module.
import MySQLdb
db = MySQLdb.connect(host="localhost",
user="root",
passwd="********", #Password blocked
db="test")
cur = db.cursor()
cur.execute("SELECT * FROM movies")
cur.execute("Update movies set genre = 'action' where id = 1")
for row in cur.fetchall() :
print row, " ", row, " ", row
My code runs and returns no errors, but when I delete the
cur.execute("Update movies set genre = 'action' where id = 1")
line it just prints out the table the as it was before. Just for reference, here is the table:
1 Interstellar sci-fi
2 Thor: Ragnarok action
3 Thor: The Dark World action
How can I make the commands in python actually affect the table? Thank you so much for your help! less
I have checked the PyTorch tutorial and questions similar to this one on Stackoverflow.
I get confused; does the embedding in pytorch (Embedding) make the similar words closer to... moreI have checked the PyTorch tutorial and questions similar to this one on Stackoverflow.
I get confused; does the embedding in pytorch (Embedding) make the similar words closer to each other? And do I just need to give to it all the sentences? Or it is just a lookup table and I need to code the model?
__future__ frequently appears in Python modules. I do not understand what __future__ is for and how/when to use it even after reading the Python's __future__ doc.Can anyone... more__future__ frequently appears in Python modules. I do not understand what __future__ is for and how/when to use it even after reading the Python's __future__ doc.Can anyone explain with examples?A few answers regarding the basic usage of __future__ I've received seemed correct.However, I need to understand one more thing regarding how __future__ works:The most confusing concept for me is how a current python release includes features for future releases, and how a program using a feature from a future release can be be compiled successfully in the current version of Python.I am guessing that the current release is packaged with potential features for the future. However, the features are available only by using __future__ because they are not the current standard. Let me know if I am right. less
I began to fall in love with a Python Visualization library called Altair, and i use it with every small data science project that ive done.
Now, in terms of Industry use... moreI began to fall in love with a Python Visualization library called Altair, and i use it with every small data science project that ive done.
Now, in terms of Industry use cases, Does it make sense to visualize Big Data or should we just take a random sample?
I trained quora question pair detection with LSTM but training accuracy is very low and always changes when i train. I dont understand what mistake i did.
I tried changing loss... moreI trained quora question pair detection with LSTM but training accuracy is very low and always changes when i train. I dont understand what mistake i did.
I tried changing loss and optimiser and with increased epoch.
import numpy as np
from numpy import array
from keras.callbacks import ModelCheckpoint
import keras
from keras.optimizers import SGD
import tensorflow as tf
from sklearn import preprocessing
import xgboost as xgb
from keras import backend as K
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from keras.preprocessing.text import Tokenizer , text_to_word_sequence
from keras.preprocessing.sequence import pad_sequences
from keras.layers.embeddings import Embedding
from keras.models import Sequential, model_from_json, load_model
from keras.layers import LSTM, Dense, Input, concatenate, Concatenate, Activation, Flatten
from keras.models import Model
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from... less
I'm following a course on EdX on Programming with Python in Data Science. When using a given function to plot the results of my linear regression model, the graph seems very off... moreI'm following a course on EdX on Programming with Python in Data Science. When using a given function to plot the results of my linear regression model, the graph seems very off with all the scatter points clustered at the bottom and the regression line way up top.
I'm not sure if it is the defined function drawline to be incorrect or sth else is wrong with my modeling process.
here is the defined function
def drawLine(model, X_test, y_test, title, R2):
fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(X_test, y_test, c='g', marker='o')
ax.plot(X_test, model.predict(X_test), color='orange', linewidth=1, alpha=0.7)
plt.show()
here is the code I wrote
import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn import linear_model
from sklearn.model_selection import... less
How do I convert a PyTorch tensor into a python list?
My current use case is to convert a tensor of size into a list of 2048 elements.
My tensor has floating point values. Is... moreHow do I convert a PyTorch tensor into a python list?
My current use case is to convert a tensor of size into a list of 2048 elements.
My tensor has floating point values. Is there a solution which also accounts for int and possibly other data types?
I want to train a simple neural network with PyTorch on a pandas dataframe df.
One of the columns is named "Target", and it is the target variable of the network. How can I use... moreI want to train a simple neural network with PyTorch on a pandas dataframe df.
One of the columns is named "Target", and it is the target variable of the network. How can I use this dataframe as input to the PyTorch network?
I tried this, but it doesn't work:
import pandas as pd
import torch.utils.data as data_utils
I've recently started getting into data analysis and I've learned quite a bit over the last year (at the moment, pretty much exclusively using Python). I feel the next step is to... moreI've recently started getting into data analysis and I've learned quite a bit over the last year (at the moment, pretty much exclusively using Python). I feel the next step is to begin training myself in MapReduce/Hadoop. I have no formal computer science training however and so often don't quite understand the jargon that is used when people write about Hadoop, hence my question here.
What I am hoping for is a top level overview of Hadoop (unless there is something else I should be using?) and perhaps a recommendation for some sort of tutorial/text book.
If, for example, I want to parallelise a neural network which I have written in Python, where would I start? Is there a relatively standard method for implementing Hadoop with an algorithm or is each solution very problem specific?
The Apache wiki page describes Hadoop as "a framework for running applications on large cluster built of commodity hardware". But what does that mean? I've heard the term "Hadoop Cluster" and I know that Hadoop is Java... less
I copied and pasted tensorflow's official Basic classification: Classify images of clothing code https://www.tensorflow.org/tutorials/keras/classification
import tensorflow as... moreI copied and pasted tensorflow's official Basic classification: Classify images of clothing code https://www.tensorflow.org/tutorials/keras/classification
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
and ran it. Upon running it printed a load of gibberish and wouldn't stop (almost like when you accidentally put a print in a while loop):
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
so I terminated it. The above is just a VERY small portion of what printed. I ran it again, only to get an error straight away.
line 7, in <module>
(train_images, train_labels), (test_images,... less
I'm learning object oriented programing in a data science context.
I want to understand what good practice is in terms of writing methods within a class that relate to one... moreI'm learning object oriented programing in a data science context.
I want to understand what good practice is in terms of writing methods within a class that relate to one another.
When I run my code:
import pandas as pd
pd.options.mode.chained_assignment = None
class MyData:
def __init__(self, file_path):
self.file_path = file_path
def prepper_fun(self):
'''Reads in an excel sheet, gets rid of missing values and sets datatype to numerical'''
df = pd.read_excel(self.file_path)
df = df.dropna()
df = df.apply(pd.to_numeric)
self.df = df
return(df)
def quality_fun(self):
'''Checks if any value in any column is more than 10. If it is, the value is replaced with
a warning 'check the original data value'.'''
for col in self.df.columns:
for row in self.df.index:
if self.df > 10:
self.df = str('check original data value')
return(self.df)
I tried to install XGBoost package in python. I am using windows os, 64bits . I have gone through following.
The package directory states that xgboost is unstable for windows and... moreI tried to install XGBoost package in python. I am using windows os, 64bits . I have gone through following.
The package directory states that xgboost is unstable for windows and is disabled: pip installation on windows is currently disabled for further invesigation, please install from github. https://pypi.python.org/pypi/xgboost/
I am not well versed in Visual Studio, facing problem building XGBoost. I am missing opportunities to utilize xgboost package in data science.
Please guide, so that I can import the XGBoost package in python.
Thanks less
I have two question:
(1) The question about importing some subpackages inside tensorflow.keras.
(2) How to differentiate between the packages installed by 'pip install' and 'conda... moreI have two question:
(1) The question about importing some subpackages inside tensorflow.keras.
(2) How to differentiate between the packages installed by 'pip install' and 'conda install'.(under windows)
I am using anaconda with tensorflow 2.0.0. I am trying to import package like:
import tensorflow.keras.utils.np_utils
However, the error shown that:
---------------------------------------------------------------------------
> ModuleNotFoundError Traceback (most recent call
> last) <ipython-input-2-ee1bc59a14ab> in <module>
> ----> 1 import tensorflow.keras.utils.np_utils
>
> ModuleNotFoundError: No module named 'tensorflow.keras.utils.np_utils'
I am confused about why this is happening, I install the tensorflow with command:
conda install tensorflow==2.0.0
from Anaconda prompt.
Yes, I know the anaconda should have already had all the data science package inside it, the reason that I uninstall tensorflow provided by anaconda and reinstall it was before using... less
I have a list of strings containing arbitary phone numbers in python. The extension is an optional part.
st =
My objective is to segregate the phone numbers so that I can... moreI have a list of strings containing arbitary phone numbers in python. The extension is an optional part.
st =
My objective is to segregate the phone numbers so that I can isolate each individual group viz. '800', '555', '1212' and the optional '1234'.
I have tried out the following code.
p1 = re.compile(r'(\d{3}).*(\d{3}).*(\d{4}).*(\d{4})?')
step1 =
p2 = re.compile(r'(\d{3})(\d{3})(\d{4})(\d{4})?')
step2 =
p1 and p2 being the two patterns to fetch the desired output.
for i in range(len(step2)):
print step2
Since I am a newbie, I wish to get suggestions if there are better ways to tacle such problems or some best practices followed in Python community. Thanks in advance.