I am using pixellib fot training custom image instance segmentation. I have created a dataset whiche can be seen below in link.... moreI am using pixellib fot training custom image instance segmentation. I have created a dataset whiche can be seen below in link. Dataset:https://drive.google.com/drive/folders/1MjpDNZtzGRNxEtCDcTmrjUuB1ics_3Jk?usp=sharing the code which I used to make a custom model is
I am using pixellib fot training custom image instance segmentation. I have created a dataset whiche can be seen below in link. Dataset:https://drive.google.com/drive/folders/1MjpDNZtzGRNxEtCDcTmrjUuB1ics_3Jk?usp=sharing the code which I used to make a custom model is
import pixellib
from pixellib.custom_train import instance_custom_training
I have lots of excel files(xlsx format) and want to read and handle them.
For example, file names are ex201901, ex201902, 201912.
Its name is made by ex YYYYMM format.
Anyway,... moreI have lots of excel files(xlsx format) and want to read and handle them.
For example, file names are ex201901, ex201902, 201912.
Its name is made by ex YYYYMM format.
Anyway, to import these files in pandas as an usual case, it's easy.
I want to know what a learning curve in machine learning is. What is the standard way of plotting it? I mean what should be the x and y axis of my plot?
Suppose we are training a keras model on 1000 images of 3 classes and the labels list is . How can we save these labels to a file and use them again during predictions to get the... moreSuppose we are training a keras model on 1000 images of 3 classes and the labels list is . How can we save these labels to a file and use them again during predictions to get the label name from the prediction array?
When working with image or text-based data, are there any neural network architectures that can handle input with different sizes? If not, what are the best ways to handle these... moreWhen working with image or text-based data, are there any neural network architectures that can handle input with different sizes? If not, what are the best ways to handle these kinds of data?
I am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS.
For this I am using Flume. I just started trying... moreI am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS.
For this I am using Flume. I just started trying it out. Succesfully created a node but now I am willing to setup a HTTP source and a sink that will write incoming requests over HTTP to local file.
Any suggesstions?
Thanks in Advance/
I am trying to read messages on Kafka topic, but I am unable to read it. The process gets killed after sometime, without reading any messages.
Here is the rebalancing error which... moreI am trying to read messages on Kafka topic, but I am unable to read it. The process gets killed after sometime, without reading any messages.
Here is the rebalancing error which I get:
ERROR Error processing message, stopping consumer: (kafka.consumer.ConsoleConsumer$)
kafka.common.ConsumerRebalanceFailedException: topic-1395414642817-47bb4df2 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:428)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:718)
at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:752)
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:142)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
at... less
I have a field such... moreI have a field such that
Color
Pink
Blue
Pink
Blue
Pink
Pink
Pink
Pink
Pink
Blue
Blue
Pink
Pink
Pink
Blue
Pink
Blue
Pink
Blue
How to create calculated field to count number of times Pink appears?
Total numbers of Blue?
Can one write plug-ins for Tableau? Is Tableau equipped with any sort of general-purpose scripting language?
e.g., for generating visualizations that cannot be created using the... moreCan one write plug-ins for Tableau? Is Tableau equipped with any sort of general-purpose scripting language?
e.g., for generating visualizations that cannot be created using the default Tableau tools, or for doing k-means clustering on a dataset using various metrics, etc...
I have a number of columns that I would like to remove from a data frame. I know that we can delete them individually using something like:
df$x <- NULL
But I was hoping to do... moreI have a number of columns that I would like to remove from a data frame. I know that we can delete them individually using something like:
df$x <- NULL
But I was hoping to do this with fewer commands.
Also, I know that I could drop columns using integer indexing like this:
df <- df
But I am concerned that the relative position of my variables may change.
Given how powerful R is, I figured there might be a better way than dropping each column one by one.
Getting strange behavior when calling function outside of a closure:
when function is in a object everything is working
when function is in a class get... moreGetting strange behavior when calling function outside of a closure:
when function is in a object everything is working
when function is in a class get :
Task not serializable: java.io.NotSerializableException: testing
The problem is I need my code in a class and not an object. Any idea why this is happening? Is a Scala object serialized (default?)?
This is a working code example:
object working extends App {
val list = List(1,2,3)
val rddList = Spark.ctx.parallelize(list)
//calling function outside closure
val after = rddList.map(someFunc(_))
def someFunc(a:Int) = a+1
after.collect().map(println(_))
}
This is the non-working example :
object NOTworking extends App {
new testing().doIT
}
//adding extends Serializable wont help
class testing {
val list = List(1,2,3)
val rddList = Spark.ctx.parallelize(list)
def doIT = {
//again calling the fucntion someFunc
val after = rddList.map(someFunc(_))
//this will crash (spark lazy)
... less
I read Cluster Mode Overview and I still can't understand the different processes in the Spark Standalone cluster and the parallelism.
Is the worker a JVM process or not? I... moreI read Cluster Mode Overview and I still can't understand the different processes in the Spark Standalone cluster and the parallelism.
Is the worker a JVM process or not? I ran the bin\start-slave.sh and found that it spawned the worker, which is actually a JVM.
As per the above link, an executor is a process launched for an application on a worker node that runs tasks. An executor is also a JVM.
These are my questions:
Executors are per application. Then what is the role of a worker? Does it co-ordinate with the executor and communicate the result back to the driver? Or does the driver directly talks to the executor? If so, what is the worker's purpose then?
How to control the number of executors for an application?
Can the tasks be made to run in parallel inside the executor? If so, how to configure the number of threads for an executor?
What is the relation between a worker, executors and executor cores ( --total-executor-cores)?
What does it mean to have more workers per... less
Is there any way, I can add simple L1/L2 regularization in PyTorch? We can probably compute the regularized loss by simply adding the data_loss with the reg_loss but is there any... moreIs there any way, I can add simple L1/L2 regularization in PyTorch? We can probably compute the regularized loss by simply adding the data_loss with the reg_loss but is there any explicit way, any support from PyTorch library to do it more easily without doing it manually?
I am reading through the documentation of PyTorch and found an example where they write
gradients = torch.FloatTensor()
y.backward(gradients)
print(x.grad)
where x was an initial... moreI am reading through the documentation of PyTorch and found an example where they write
gradients = torch.FloatTensor()
y.backward(gradients)
print(x.grad)
where x was an initial variable, from which y was constructed (a 3-vector). The question is, what are the 0.1, 1.0 and 0.0001 arguments of the gradients tensor ? The documentation is not very clear on that.
I would like to know if pytorch is using my GPU. It's possible to detect with nvidia-smi if there is any activity from the GPU during the process, but I want something written in... moreI would like to know if pytorch is using my GPU. It's possible to detect with nvidia-smi if there is any activity from the GPU during the process, but I want something written in a python script.Is there a way to do so?
I found many options recently, and interesting in their comparisons primarely by maturity and stability.
Crunch - https://github.com/cloudera/crunch
Scrunch... moreI found many options recently, and interesting in their comparisons primarely by maturity and stability.
One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?".
I've read and watched... moreOne of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?".
I've read and watched some introductory materials. In particular, take, for example, Kafka: a Distributed Messaging System for Log Processing, which writes:
"a Topic is the container with which messages are associated"
"the smallest unit of parallelism is the partition of a topic. This implies that all messages that ... belong to a particular partition of a topic will be consumed by a consumer in a consumer group."
Knowing this, what would be a good example that illustrates how to use topics and partitions? When should something be a topic? When should something be a partition?
As an example, let's say my (Clojure) data looks like:
{:user-id 101 :viewed "/page1.html" :at #inst "2013-04-12T23:20:50.22Z"}
{:user-id 102 :viewed "/page2.html" :at #inst "2013-04-12T23:20:55.50Z"}
Should the topic be based on user-id? viewed? at? What about the... less
Is there a way to purge the topic in kafka?I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an... moreIs there a way to purge the topic in kafka?I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error:
kafka.common.InvalidMessageSizeException: invalid message size
Increasing the fetch.size is not ideal here, because I don't actually want to accept messages that big.
I'm just wondering what is the difference between an RDD and DataFrame (Spark 2.0.0 DataFrame is a mere type alias for Dataset) in Apache Spark?Can you convert one to the other?