Import Error with Databricks

I am attempting to read a table from BigQuery and write it to another database.
On my first few tries, I was able to read the table from BigQuery, but when I attempted to write said table to the database, I got the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 13.0 failed 4 times, most recent failure: Lost task 1.3 in stage 13.0 (TID 41, <IP_ADDRESS>, executor 0): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with  index 2, which isn't less than or equal to than endFileNumber 0!
Then, I checked our server, and it had actually worked! So I figured never mind, let me run the process all the way through again.
However, now I can't even read the initial table from BigQuery, with the exact same code. It returns this error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 20.0 failed 4 times, most recent failure: Lost task 0.3 in stage 20.0 (TID 53, <IP_ADDRESS>, executor 0): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 1!
Any clues? I'm using Databricks to import a table from BigQuery. The cluster I'm using is Databricks 3.3, which includes Spark 2.2.0 and Scala 2.11.
Thank you so much for your time and expertise

Hi
Seems to be a problem with the transaction log. It's hard to be sure
without looking at the full stack trace
The library is built using spark 2.1. I've been meaning to upgrade to 2.2
and I'll be doing that over the next few days
I would say try aiming it at a 2.1 cluster or build a custom package and
see how it goes
Regards
Sam
On Tue, 14 Nov 2017 at 23:50, mayankshah891<EMAIL_ADDRESS>wrote:

I am attempting to read a table from BigQuery and write it to another
database.
On my first few tries, I was able to read the table from BigQuery, but
when I attempted to write said table to the database, I got the following
error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 13.0 failed 4 times, most recent failure: Lost task 1.3 in stage
13.0 (TID 41, <IP_ADDRESS>, executor 0): java.lang.IllegalStateException:
Found known file 'data-000000000002.avro' with index 2, which
isn't less than or equal to than endFileNumber 0!
Then, I checked our server, and it had actually worked! So I figured
never mind, let me run the process all the way through again.
However, now I can't even read the initial table from BigQuery, with the
exact same code. It returns this error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 20.0 failed 4 times, most recent failure: Lost task 0.3 in stage
20.0 (TID 53, <IP_ADDRESS>, executor 0): java.lang.IllegalStateException:
Found known file 'data-000000000002.avro' with index 2, which isn't less
than or equal to than endFileNumber 1!
Any clues? I'm using Databricks to import a table from BigQuery. The
cluster I'm using is Databricks 3.3, which includes Spark 2.2.0 and Scala
2.11.
Thank you so much for your time and expertise
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/48, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AEHLm6Fir7GYmnquj-vnDufO6AT2oNxlks5s2gsZgaJpZM4QeDPk
.

Wow! thank you for the incredibly fast response! I'm launching a 2.1 cluster now, will let you know how it goes

Update:
So using a v2.1 cluster fixed the initial issue, in that I can read data from BigQuery into Spark. The table does indeed display (!)
Writing to the other database is still producing this weird error, and this time no data came through (besides the schema):

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, <IP_ADDRESS>, executor 1): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 0!

Here is the complete error:

Py4JJavaError                             Traceback (most recent call last)
 in ()
----> 1 Events_data.write.format("jdbc").option("url", "jdbc:mapd:ec2-18-220-187-39.us-east-2.compute.amazonaws.com:9091:mapd").option("driver", "com.mapd.jdbc.MapDDriver").option("dbtable", "InAppEvents3").option("user", "mapd").option("password", "i-053bdfd74c7ab374b").mode("overwrite").save()

/databricks/spark/python/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy, **options)
546             self.format(format)
547         if path is None:
--> 548             self._jwrite.save()
549         else:
550             self._jwrite.save(path)
/databricks/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in call(self, *args)
1131         answer = self.gateway_client.send_command(command)
1132         return_value = get_return_value(
-> 1133             answer, self.gateway_client, self.target_id, self.name)
1134
1135         for temp_arg in temp_args:
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
61     def deco(*a, **kw):
62         try:
---> 63             return f(*a, **kw)
64         except py4j.protocol.Py4JJavaError as e:
65             s = e.java_exception.toString()
/databricks/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
317                 raise Py4JJavaError(
318                     "An error occurred while calling {0}{1}{2}.\n".
--> 319                     format(target_id, ".", name), value)
320             else:
321                 raise Py4JError(
Py4JJavaError: An error occurred while calling o186.save.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, <IP_ADDRESS>, executor 1): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 0!
at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
at com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.setEndFileMarkerFile(DynamicFileListRecordReader.java:327)
at com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.nextKeyValue(DynamicFileListRecordReader.java:177)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:199)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:578)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1442)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1430)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1429)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1429)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:803)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:803)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:803)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1657)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1612)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1601)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1937)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1950)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1963)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1977)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply$mcV$sp(Dataset.scala:2323)
at org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply(Dataset.scala:2323)
at org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply(Dataset.scala:2323)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:87)
at org.apache.spark.sql.execution.SQLExecution$.withFileAccessAudit(SQLExecution.scala:60)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:70)
at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2777)
at org.apache.spark.sql.Dataset.foreachPartition(Dataset.scala:2322)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.saveTable(JdbcUtils.scala:670)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:90)
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:488)
at org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply$mcV$sp(DataFrameWriter.scala:217)
at org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply(DataFrameWriter.scala:209)
at org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply(DataFrameWriter.scala:209)
at org.apache.spark.sql.execution.SQLExecution$.withFileAccessAudit(SQLExecution.scala:53)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:209)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 0!
at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
at com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.setEndFileMarkerFile(DynamicFileListRecordReader.java:327)
at com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.nextKeyValue(DynamicFileListRecordReader.java:177)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:199)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:578)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more

Hey sorry I'm at a conference so won't have a chance to really look at this
until the conference
So just to clarify this is a fresh cluster with the Google creds json in it
( it must be since you wouldn't be able to read if it wasn't)
Does it fail everytime or after a while? Also is it a structured streaming
save or a batch save? I.e one off save to bigquery?
On Wed, 15 Nov 2017 at 00:08, mayankshah891<EMAIL_ADDRESS>wrote:

Update:
So using a v2.1 cluster fixed the initial issue, in that I can read data
from BigQuery into Spark. The table does indeed display (!)
Writing to the other database is still producing this weird error, and
this time no data came through (besides the schema):
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage
3.0 (TID 8, <IP_ADDRESS>, executor 1): java.lang.IllegalStateException:
Found known file 'data-000000000002.avro' with index 2, which isn't less
than or equal to than endFileNumber 0!
Here is the complete error:
Py4JJavaError Traceback (most recent call last)
in ()
----> 1 Events_data.write.format("jdbc").option("url",
"jdbc:mapd:ec2-18-220-187-39.us-east-2.compute.amazonaws.com:9091:mapd").option("driver",
"com.mapd.jdbc.MapDDriver").option("dbtable",
"InAppEvents3").option("user", "mapd").option("password",
"i-053bdfd74c7ab374b").mode("overwrite").save()
/databricks/spark/python/pyspark/sql/readwriter.py in save(self, path,
format, mode, partitionBy, **options)
546 self.format(format)
547 if path is None:
--> 548 self._jwrite.save()
549 else:
550 self._jwrite.save(path)
/databricks/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in
call(self, *args)
1131 answer = self.gateway_client.send_command(command)
1132 return_value = get_return_value(
-> 1133 answer, self.gateway_client, self.target_id, self.name)
1134
1135 for temp_arg in temp_args:
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
/databricks/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in
get_return_value(answer, gateway_client, target_id, name)
317 raise Py4JJavaError(
318 "An error occurred while calling {0}{1}{2}.\n".
--> 319 format(target_id, ".", name), value)
320 else:
321 raise Py4JError(
Py4JJavaError: An error occurred while calling o186.save.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task
1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage
3.0 (TID 8, <IP_ADDRESS>, executor 1): java.lang.IllegalStateException:
Found known file 'data-000000000002.avro' with index 2, which isn't less
than or equal to than endFileNumber 0!
at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
at
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.setEndFileMarkerFile(DynamicFileListRecordReader.java:327)
at
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.nextKeyValue(DynamicFileListRecordReader.java:177)
at
org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:199)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:578)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1442)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1430)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1429)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1429)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:803)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:803)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:803)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1657)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1612)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1601)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1937)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1950)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1963)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1977)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:926)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at
org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply$mcV$sp(Dataset.scala:2323)
at
org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply(Dataset.scala:2323)
at
org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply(Dataset.scala:2323)
at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:87)
at
org.apache.spark.sql.execution.SQLExecution$.withFileAccessAudit(SQLExecution.scala:60)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:70)
at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2777)
at org.apache.spark.sql.Dataset.foreachPartition(Dataset.scala:2322)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.saveTable(JdbcUtils.scala:670)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:90)
at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:488)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply$mcV$sp(DataFrameWriter.scala:217)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply(DataFrameWriter.scala:209)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply(DataFrameWriter.scala:209)
at
org.apache.spark.sql.execution.SQLExecution$.withFileAccessAudit(SQLExecution.scala:53)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:209)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Found known file
'data-000000000002.avro' with index 2, which isn't less than or equal to
than endFileNumber 0!
at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
at
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.setEndFileMarkerFile(DynamicFileListRecordReader.java:327)
at
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.nextKeyValue(DynamicFileListRecordReader.java:177)
at
org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:199)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:578)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:670)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1963)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/48#issuecomment-344415456,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm2hkIF7aeUYP6_9F8TO2MGdFEithks5s2g9CgaJpZM4QeDPk
.

that seems to be a problem with the data being read in, the connector doest
read or use avro.
Is the data being read avro?
On Wed, Nov 15, 2017 at 5:52 PM, mayankshah891<EMAIL_ADDRESS>wrote:

Its a fresh cluster, I have the credentials saved via json correctly (I'm
assuming bc I was able to read it at least once).
Now every time I fire up the cluster, it fails to read the table
initially. It is a batch save. I was going to run it every night.
This is the error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage
1.0 (TID 4, <IP_ADDRESS>, executor 0): java.lang.IllegalStateException:
Found known file 'data-000000000004.avro' with index 4, which isn't less
than or equal to than endFileNumber 2!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/48#issuecomment-344635883,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm6oqLg74dQH1l8L5b3S5ZLRVCStjks5s2wjFgaJpZM4QeDPk
.

How do I check if the data is avro? Its a table in BigQuery (obviously), the person who put it there says he doesn't know what avro is, so I'm assuming not? Again, it worked once and just hasn't worked since, but I still have the print statement showing the table in databricks, so I'm 100% sure it worked at least once.

Can you make sure you are using my connector and not the spotify one? I
believe they convert to avro
On Thu, Nov 16, 2017 at 5:39 PM, mayankshah891<EMAIL_ADDRESS>wrote:

How do I check if the data is avro? Its a table in BigQuery (obviously),
the person who put it there says he doesn't know what avro is, so I'm
assuming not? Again, it worked once and just hasn't worked since, but I
still have the print statement showing the table in databricks, so I'm 100%
sure it worked at least once.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/48#issuecomment-344961355,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm_A6s-CvN87mSO4mUtDCm6Qh2n66ks5s3FcsgaJpZM4QeDPk
.

actually my mistake, digging into the code I see the google hadoop
connector is using avro
https://github.com/samelamin/spark-bigquery/blob/
ed716c8cc5b2f1fdf0d57c0e8da7d7d6e1bc0925/src/main/scala/com/
samelamin/spark/bigquery/BigQuerySQLContext.scala#L89
I am not near my laptop at the moment but I will have a look tomorrow
In the meantime try restart/fresh cluster just to make sure its nothing to
do with the state, the connector def works on Databricks I am just not sure
why only you are facing the problem. My only guess is that its something
with the cluster set up
On Thu, Nov 16, 2017 at 6:16 PM, Sam Elamin<EMAIL_ADDRESS>wrote:

Can you make sure you are using my connector and not the spotify one? I
believe they convert to avro
On Thu, Nov 16, 2017 at 5:39 PM, mayankshah891<EMAIL_ADDRESS>wrote:

How do I check if the data is avro? Its a table in BigQuery (obviously),
the person who put it there says he doesn't know what avro is, so I'm
assuming not? Again, it worked once and just hasn't worked since, but I
still have the print statement showing the table in databricks, so I'm 100%
sure it worked at least once.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/48#issuecomment-344961355,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm_A6s-CvN87mSO4mUtDCm6Qh2n66ks5s3FcsgaJpZM4QeDPk
.

Okay thank you so much for your time and for building this in the first place. I'll try some different clusters and setups and we'll see. Thank you again

I've tried a bunch of different clusters, to mixed results. I can read in the data sometimes, but if I try to call .show(200) on the data, it fails. Again the error is something to do with the partitions. Databricks seems to believe it is done reading the data, but then another file pops up that is past the "endFileNumber".
Someone else was running into an error like this here, but I haven't cracked how to solve it myself yet:
https://stackoverflow.com/questions/46404081/illegalstateexception-on-google-cloud-dataproc

Sam, I think I figured out the problem. The bigquery table I'm reading from is streaming new data every few seconds, so the lazy transformations in Spark are breaking the code because the file size keeps changing! I just need to call .persist or .cache and hold the data in memory and then it works! (Well, sort of works, its not writing properly but I think that can be fixed as well)
Tentatively I think this issue can be closed. Thanks again

Hi no worries glad we got there in the end
Had no idea you were streaming!
Can you close the issue please
On Fri, 17 Nov 2017 at 19:59, mayankshah891<EMAIL_ADDRESS>wrote:

Sam, I think I figured out the problem. The bigquery table I'm reading
from is streaming new data every few seconds, so the lazy transformations
in Spark are breaking the code because the file size keeps changing! I just
need to call .persist or .cache and hold the data in memory and then it
works! (Well, sort of works, its not writing properly but I think that can
be fixed as well)
Tentatively I think this issue can be closed. Thanks again
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/48#issuecomment-345317207,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm3YmlIlyX83oYAzMczY35ZpndXWgks5s3cmEgaJpZM4QeDPk
.


Cmd + E opens up "google Search" in Adressbar

I did't not know how to open up the sidebar. When pressen "Cmd + E" " Ctrl + E" the cursor focuses on adressbar (google search).

I am same
use it in Windows and Linux

I am same
use it in Windows and Linux

I also get these results.

When pressen "Cmd + E" " Ctrl + E" the cursor focuses on adressbar (google search).

@lxieyang - need to fix that wrong instruction here...

https://github.com/lxieyang/vertical-tabs-chrome-extension

I did not know how to open up the sidebar.

--Is this not working for you?

I also get these results.

Since Vertical Tabs works for most users, I am really trying to determine why it does not work in this case.  But we cannot solve the problem until we know if this is a bug in the browser or the extension - or something else.  Can you tell me what happens when you press the toolbar button here:

it was pulled almost two months ago

This extension is still in the Chrome store, so I do not know what that means.  But I am concerned that you might have got a different version installed, since there are three extensions in the store with the same name:
https://chrome.google.com/webstore/search/verticaltabs
If the extension ID shown below does not match the one in your browser at chrome://extensions/, then you have a different extension installed.
chrome://extensions/?id=pddljdmihkpdfpkgmbhdomeeifpklgnm
https://chrome.google.com/webstore/detail/vertical-tabs/pddljdmihkpdfpkgmbhdomeeifpklgnm
Can you please be more specific about "it was pulled" -?

If I could see my entire comment as opposed to fragments, I could betteraddress what is said below.  There may also be a language problem.  Seeremarks below . . .
We are all in this together,
Jan Neff-Sinclair<PHONE_NUMBER>
@.***
On Monday, March 22, 2021, 08:14:19 PM PDT, kekukui<EMAIL_ADDRESS>wrote:

When I press CTRL-E, I get this display:Also, please note that there is not stack of sideways orange rectangles anywhere for me toclick on.
When pressen "Cmd + E" " Ctrl + E" the cursor focuses on adressbar (google search).
@lxieyang - need to fix that wrong instruction here...
https://github.com/lxieyang/vertical-tabs-chrome-extension
I did not know how to open up the sidebar.
--Is this not working for you?  I see no extension icon and never have.
I also get these results.
Since Vertical Tabs works for most users, I am really trying to determine why it does not work in this case. But we cannot solve the problem until we know if this is a bug in the browser or the extension - or something else. Can you tell me what happens when you press the toolbar button here:
I do not see this icon anywhere on my screen.
it was pulled almost two months ago
You all have a list of problems and it shows when an issue is assigned tosomeone or is "pulled" to be worked on.  After two months, if an error isn'tfixed, I doubt anyone thinks it is important enough to bother with.
This extension is still in the Chrome store, so I do not know what that means. But I am concerned that you might have got a different version installed, since there are three extensions in the store with the same name:
https://chrome.google.com/webstore/search/verticaltabs
If the extension ID shown below does not match the one in your browser at chrome://extensions/, then you have a different extension installed.  IT is the right extension down to the Scientific American videochrome://extensions/?id=pddljdmihkpdfpkgmbhdomeeifpklgnm
https://chrome.google.com/webstore/detail/vertical-tabs/pddljdmihkpdfpkgmbhdomeeifpklgnm
Can you please be more specific about "it was pulled" -?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

If I could see my entire comment as opposed to fragments...

In Write mode, you can enlarge the text entry box by dragging the corner, just like a window on the desktop.  (What would we do without JavaScript, eh?)

Then switch to Preview to see how your comments will appear when posted.

You can edit a previous post to correct any mistakes.  You can select text and then press the quote button on the Write toolbar to highlight a sentence you wish to call out.  Feel free to test this on your previous post, if you like.

I see no extension icon and never have.

Is it hidden under the active extensions menu?  --If yes, then click the pin to anchor it to the browsers tool bar.
https://askcaty.com/how-to-pin-un-pin-extensions-from-the-google-chrome-toolbar

And please let me know if this resolves the issue.

[ ]  How many of the participants here might be a new user of Chrome who switched from another browser that works differently in this regard?  If so, what did you use previously?

After two months, if an error isn't fixed, I doubt anyone thinks it is important enough to bother with.

If the issue remains open, then it has not been abandoned or rejected.  We just do not have enough time to work on all of the projects which require our attention.  Other people are welcome to fix these bugs and submit a patch.  But currently there are no outstanding patches waiting to be merged that address open issues.

when an issue is assigned to someone or is "pulled" to be worked on...

Oh, that.  Yes, a "pull request" usually means that someone has made a copy of the code, added some improvements, and now wants the changes to be merged with the original.  But in this particular situation, the dependabot is just notifying about updates to common, shared libraries which are not application-specific.  Those pull requests will not resolve any open issues.  (In some cases, it could even break stuff.)
I am sure the lead developer is busy with other projects, and this is why I am trying to handle some of the bug characterization tasks.  This will allow him more time to address those issues which add new features. ☺  Just imagine how much more slowly development would progress if we did not have a shared workspace and a version control system!

I was a Software/Firmware (embedded) Engineer for almost thirty years

Thanks so much @kekukui ! I'm indeed busy with other projects at the moment, but will definitely get back on this one in mid-April!

When Chrome implemented the Extension Toolbar Menu in version 76, the extension toolbar buttons became hidden by default.  And now I think we have some indication that this change is confusing new users of Chrome who migrated from some other browsing platform.  (Much like the concealment of page titles, this behavior obstructs the user's workflow.)  This is my hypothesis, anyway:
A lot of people are migrating to Chrome from something else.  I suspect it is probably users of the new Microsoft Edge who noticed the performance improvement of a Chromium-based browser but they switched to Chrome because Edge is unstable and has too many rendering bugs.  These new Chrome users do not see the Vertical Tabs toolbar button, so they try the keyboard shortcut (which does not work).  This creates a false perception that the extension froze, crashed, or did not launch.  In reality, it is probably just hiding.  The underlying problem is that Chrome likes to hide things too often or too quickly, which makes the interface less intuitive.
When a new extension is installed, Chrome should highlight the new toolbar button - and this deficiency is generating extra bug reports for extension developers.  The Mac OS help menu demonstrates a better way to highlight controls, which eliminates confusion.  This is the sort of thing which Chrome needs to do when a new extension is installed: the extension menu should open and the precise location of the control should be highlighted.  You would then click the indicator to dismiss it, and this would be the normal installation procedure.

▶ https://www.youtube.com/watch?v=dBcAdU0cF5k
If some information can be represented more efficiently with graphics & animation, then it should be done that way.  If we cannot convince the Chromium team to address this usability issue, I guess you could present some kind of slide show to the user the first time the extension is launched, which indicates the precise location of the toolbar button.


Fix issue when no fortran is defined

I added "no fortran" spec so this doesn't happen again
This issue was brought up by a Spack user. This is probably the second or third time not testing against a spec that didn't have fortran has come up.

Thanks @white238.
I'm assuming you're planning to add a host-config for this when it builds, right?

It just succeeded my small test. I will build just this spec and add its specific host-config

I'm running into a weird error where my compiler specs are bleeding together. I am trying something to see if it goes away.

This is ready to go.  Just so everyone knows... spack sees<EMAIL_ADDRESS>as a replacement/upgrade of<EMAIL_ADDRESS> So the former overrides the later.  I have talked to the spack team about it and its a known design problem/flaw/feature.


IdentityServerAuthenticationOptions events

Hi, I'm trying to set the Events property of the IdentityServerAuthenticationOptions to be able to add permissions when my api is called using a user token - I'm basing my solution based on the Combined_AspNetIdentity_and_EntityFrameworkStorage in the Quickstarts. I add the events as follows:
            services.AddAuthentication("Bearer")
                .AddIdentityServerAuthentication(opts =>
                {
                    opts.Authority = "http://localhost:5000";
                    opts.RequireHttpsMetadata = false;
                    opts.ApiName = "example_api";
                    opts.Events = new JwtBearerEvents()
                    {
                        OnTokenValidated = async context =>
                        {
                            var user = context.Principal;
                            if (!user.Identity.IsAuthenticated) throw new ApplicationException("User must be authenticated before calling this method");

                            ((ClaimsIdentity)user.Identity).AddClaim(new Claim("ApiClaim", "FromTheApi"));

                            await Task.FromResult(0);
                        }
                    };
                });

my IdentityServer contains the following modification:
            services.AddIdentityServer(opts =>
            {
                opts.Events.RaiseSuccessEvents = true;
                opts.Events.RaiseFailureEvents = true;
                opts.Events.RaiseErrorEvents = true;
            })

However the OnTokenValidated method is never called and my claim is never added. I have also tried with OAuthEvents OnCreatingTicket but with no luck.
Am I doing something wrong?
Thank you

I have noticed that there is an overload to allow you to send the JwtBearerOptions, however I'm not sure how to use it. I've tried the following, but I am receiving 401 errors
            services.AddAuthentication("Bearer")
                .AddIdentityServerAuthentication(
                "Bearer",
                jwtOpts =>
                {
                    jwtOpts.Authority = "http://localhost:5000";
                    jwtOpts.RequireHttpsMetadata = false;

                    jwtOpts.Events = new JwtBearerEvents()
                    {
                        OnTokenValidated = async context =>
                        {
                            var user = context.Principal;
                            if (!user.Identity.IsAuthenticated) throw new ApplicationException("User must be authenticated before calling this method");

                            ((ClaimsIdentity)user.Identity).AddClaim(new Claim("ApiClaim", "FromTheApi"));

                            await Task.FromResult(0);
                        }
                    };
                },
                oAuthOpts =>
                {
                    oAuthOpts.Authority = "http://localhost:5000";
                    oAuthOpts.ClientId = "example_api";
                });

Is it necessary to implement methods similar to the internal methods ConfigureJwtBearer and ConfigureIntrospection?
Thanks again

First of all - I personally would rather use claims transformation or just some middleware to do the claims augmentation.
The event should be called nevertheless - I will double check.

Ok, thanks. I'll change the structure of what I was trying to do


Unable to install sentry on minikube

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:
Running on OSX with minikube
$ helm version
Client: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"clean", BuildDate:"2017-12-07T18:09:07Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

$ minikube version
minikube version: v0.24.1

Which chart: stable/sentry 0.1.6
What happened:
Postgres pod won't start. Logs give the following output.
Error from server (BadRequest): container "sentry-postgresql" in pod "sentry-postgresql-55574f87c9-9w4n2" is waiting to start: CreateContainerConfigError

What you expected to happen:
Sentry should install successfully
How to reproduce it (as minimally and precisely as possible):
Try to run the install command helm install --name sentry --wait stable/sentry
Anything else we need to know:

cc: @rothgar

Do you have a default storage provider for the cluster? Minikube doesn't provide one by default.
Some info in the prerequisites https://github.com/kubernetes/charts/tree/master/stable/sentry#prerequisites
or you can disable persistence here https://github.com/kubernetes/charts/blob/master/stable/sentry/values.yaml#L69-L70 If your postgres pod is restarted all data will be lost.
You can try rook for persistent storage https://rook.github.io/docs/rook/master/helm-operator.html YMMV I couldn't get rook to install in minikube 1.7.0

@rothgar The default-storageclass and storage-provisioner minikube addons are enabled.
$ minikube addons list
- kube-dns: enabled
- heapster: disabled
- registry-creds: disabled
- default-storageclass: enabled
- storage-provisioner: enabled
- coredns: disabled
- ingress: enabled
- registry: disabled
- addon-manager: enabled
- dashboard: enabled
- efk: disabled

Does that not enable a default storage provider?
I tried to disable persistance but no luck unfortunately. Same error.

Are you able to install the postgres chart on its own? Sentry is dependent on that.
helm install stable/postgresql

I can try to reproduce the problem but I may not have time until after the holiday break.

That doesn't work either. Found the following error from a kubectl describe pod command.
Error: lstat /tmp/hostpath-provisioner/pvc-6319bf22-e4f4-11e7-8995-08002754c7c7: no such file or directory

Some googling resulted in this open github issue https://github.com/kubernetes/minikube/issues/2256


Add comments to all the functions

add the linting comments to all the exiting code that doesn't have it. so all the params and returns are explicitly typed for nicer linting

ok... i think i got all the models and modules commented... more or less...

i think i've got most all the stuff that really needs commenting commented.....


Feat: Links in PQIR's

User story
As a user I want to add links in problems, ideas, questions and risks so that the functionality is coherent throughout the application.
User input 1 (by Thorjan Knudsvik- sharepoint ID170):
Not possible to add links properly to PQIR's like the activity boxes. pasting regular links creates visualisation issues in the categorisation page.
Describe the solution you'd like
Markdown links as in the cards.
Relevant issues
#432

new design

ready for test in https://web-flyt-dev.radix.equinor.com/ @QiJin-Bouvet

ready for test in https://web-flyt-dev.radix.equinor.com/ @QiJin-Bouvet

ready for test in https://web-flyt-test.radix.equinor.com/ @QiJin-Bouvet

Tested in TEST env.

able to add links when creating a link
able to add links to an existing PQIR
Links are shown in category page


[do-not-merge] Adapt helm files for release version 1.3.2

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 64.35%. Comparing base (4dbf392) to head (255c38b).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3982   +/-   ##
=======================================
  Coverage   64.35%   64.35%
=======================================
  Files         391      391
  Lines       26131    26131
=======================================
  Hits        16816    16816
  Misses       7996     7996
  Partials     1319     1319

Flag
Coverage Δ

unittests
64.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


cal_jacobian for SingleArm

Hi jdj2261,
Thanks for developing this awesome library. I am trying to cal_jacobian for my customised SingleArm. According to your example, pekin/examples/robot_jacobian.py, it uses robot.arm_type for BimanualArm. But how should I cal_jacobian for SingleArm? where SingleArm object has no attribute 'arm_type'
Thanks for your help.
Chuan

Hello Chuan,
Thank you for your interest in our library.
In case of a single arm, you can use the code below.

import numpy as np
from pykin.kinematics.transform import Transform
from pykin.robots.single_arm import SingleArm
from pykin.kinematics import jacobian as jac
file_path = "urdf/doosan/doosan_with_robotiq140.urdf"
robot = SingleArm(
file_path,
Transform(rot=[0.0, 0.0, 0.0], pos=[0, 0, 0]),
has_gripper=True,
gripper_name="robotiq140",
)
robot.setup_link_name("base_0", "right_hand")
thetas = np.zeros(6)
fk = robot.forward_kin(thetas)
J = jac.calc_jacobian(robot.desired_frames, fk, len(np.zeros(7)))
print(J)
Thank you


Adds extra calendar-* icons

Icon Request

Icon name: extra calendar functions & operations
Use cases:

calendar-plus: creating new event, adding event to calendar
calendar-minus: deleting or unsubscribing from event
calendar-x:  removing or deleting event or marking a date as taken/busy
calendar-x-2, calendar-off:  removing or deleting event
calendar-check: subscribing to event, adding event to calendar, marking an event as complete or a date as taken etc
calendar-days: using as an alternate style for calendar, representing a month or a list of dates.
calendar-range: representing a date range, e.g. for daterange widgets.

Nice love it!


[Technical Writing] : E-commerce Webpage

Description
How to Develop a E commerce Webpage
Screenshots
No response
Checklist

[X] I have checked the existing issues.
[X] I follow Contributing Guidelines of this project.
[X] I am a GSSoC'24 contributor.
[X] I want to work on this issue.

Code of Conduct

[X] I agree to follow this project's "Code of Conduct".

@CSEdgeOff @Durgesh-Vaigandla please assign me the issue

Hello @CSEdgeOff and @Durgesh-Vaigandla , I want to work on this Task. Can you please assign me this so I can contribute to this project.

@CSEdgeOff  @Durgesh-Vaigandla please assign me the issue


Disable TestToolTests on Linux as they are macOS only

I realized this isn't correct, see mail on swift-build-dev.


telemetryLoggerMiddleware does not log Adaptive Card responses

Is your feature request related to a problem? Please describe.
I am using the Virtual Assistant template which uses the telemetryLoggerMiddleware. After using it for a while, it became apparent when Adaptive cards are being used as a reply they are not being picked up by the telemetry logger. Only text and speak are being populated into the event properties in the fillSendEventProperties method.
telemetryLoggerMiddleware.ts line 243
if (activity.text && activity.text.trim()) {
    properties[this.telemetryConstants.textProperty] = activity.text;
}

if (activity.speak && activity.speak.trim()) {
    properties[this.telemetryConstants.speakProperty] = activity.speak;
}

Describe the solution you'd like
I would like TelemetryLoggerMiddleware, or a variant of it, to log attachments like Adaptive cards. This could either be by default if detected, or by configuration.
Describe alternatives you've considered
To work around this issue, I could make my own version of the telemetryLoggerMiddleware, and may need to do so in the short term. In my option it would be better to have it as a part of the framework, as using Adaptive Cards is a documented way of sending messages. These should be captured.
[enhancement]

@nathanlindorff - I'm a little confused because you have a link that's labeled fillUpdateEventProperties but it links to fillReceiveEventProperties and then you've posted a code snippet from fillSendEventProperties. Can you explain why?

PEBKAC cut and paste error sorry. I'll fix it up. It all should point to fillSendEventProperties

What I ended up doing in my scenario was subclassing TelemetryLoggerMiddleware and overriding fillSendEventProperties.
I added a new 'attachment' property to which I wrote the stringified version of the attachment. I also did some specific work to extract the text from the Adaptive card into a readable form. The latter was more specific to my cards. In retrospect unless there is a standard way to turn cards to text this may be the better option than framework changes.
Perhaps a documentation update would be more appropriate, indicating that if Cards are used, the telemetry middleware doesn't log any details.

@nathanlindorff - It is curious that fillSendEventProperties provides an unused telemetryProperties parameter. I wonder if it's meant to be used by a subclass like yours, in which case you'd only need to override onSendActivity.

@daveta - I've assigned you to this issue so that you can shed some light on it and decide if you want to change anything. There doesn't seem to be anything for my team to do.

@v-kydela I was initially going to do something with telemetryProperties but the attachment was part of the Activity.

Adding @scheyal : not sure how attachments fits into the reporting strategy..

@garypretty Can you please confirm if this is still on the list for R9?

@garypretty can you please confirm? Thanks!

@carlosscastro @clearab This is still TBC for R9.
@scheyal consideration of adaptive cards and their inclusion in telemetry should be considered when we make the necessary changes to telemetry for adaptive (to ensure the schema is aligned for waterfall / adaptive).  Can you ensure this is tracked when the work is picked up by the adaptive team?  Happy to discuss.

@nathanlindorff This has now been addressed in JS / C# for release 4.9, with tracking items for Python / Java referenced above. Thanks again for raising the feature request.

Thanks for the update Gary!


MWPW-124571: FG Copy Action

Copy Action button added to the Floodgate Admin page
Copies the file from the source tree to the floodgate equivalent pink tree
Checks the existence of the file at the source and ignores if 404
Checks for the existence of the file at the destination.

If not available, copies the file at the destination
If available, overwrites the file using rename->copy->delete method

Creates missing folders at the destination as necessary
Once copy is completed, updates the status of the copy in the project excel sheet with the start and end timestamp and logs list of pages where copy could not be performed

Resolves: MWPW-124571
Test URLs:

Before: https://main--milo--adobecom.hlx.page/tools/floodgate/index.html

After: https://fg-copy-action--milo--sukamat.hlx.page/tools/floodgate/index.html

Loc test - copy to english langstore validated:
https://fg-copy-action--milo--sukamat.hlx.page/tools/loc/index.html

Minor UX nit: the buttons for the different floodgate actions do not provide visual feedback that these are actual clickable buttons (see screen recording):
https://user-images.githubusercontent.com/1764977/223384955-e64bf625-7d81-486b-900f-e17cdea4f9b9.mov

@sukamat , You can go ahead and merge this ticket. I will check if things work on localization once it's merged.


Spyder 3.0.0 runfile issue

Description of your problem
Getting this error
runfile('C:/Users/D/'Costa/Google Drive/Certification/Udemy/Machine Learning/ML1.py', wdir='C:/Users/D/'Costa/Google Drive/Certification/Udemy/Machine Learning')
^
SyntaxError: invalid syntax
I have an apostrophe in my name and the windows folder is structured that way, Is there any way to workaround this , and also fix this in the long term as a lot of people have the apostrophe in their names.

Please update to Spyder 3.1.3. This error is already fixed in that version.


Test Runs twice

Hello! I'm dealing with a really simple set up with Angular 1.5, webpack and karma. But dunno why is running the tests twice. The thing is that is happening just when I'm calling the service.
Here is the repo https://github.com/IngenieroLata/lata-simple-test-set
// heroService.test.js

import droidServiceModule from './index.js';

describe('droidServiceModule', () => {

    let droidService;

    beforeEach(angular.mock.module(droidServiceModule.name));

    beforeEach(inject(function($injector) {
        droidService = $injector.get('droidService');
    }));

    it('should test properly', () => {

        /*RUN TWICE*/
        // droidService.name = 'r2d2';
        // console.log(droidService.name);
        // console.log(droidService.speak());

        expect(true).to.be.true;

    });
});

I'm using karma-webpack with a configuration file './test/test_index.js';:
// test_index.js

require('angular');
require('angular-mocks');

var testsContext = require.context("../app", true, /\.test$/);
testsContext.keys().forEach(testsContext);

Any idea why is this happening?

reading this comment https://github.com/nikku/karma-browserify/issues/67#issuecomment-84281528 related with this other comment https://github.com/webpack/karma-webpack/issues/44#issuecomment-133619802
I realize that I have configured the webpack.HotModuleReplacementPlugin so, disabling it from plugins and hot option in the devServer sections in webpack.config file works sometimes for me.
Also, I try to use files like this but got the same result
files: [
  {pattern: './spec.js', watched: false, included: true, served: true}
],

This solution 1 from this comment worked for me: https://github.com/nikku/karma-browserify/issues/67#issuecomment-262128601

Using files: [{pattern: '**/*.spec.js', watched: false}]by @mattfysh

Closing due to the age of this ticket, if you're still having issues with the current build feel free to open a new ticket with the relevant information and I'll help as best I can!


Create or update certificate returns error azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Accepted'

Type of issue
Code doesn't work
Description
When using the python sdk to create a certificate, I get error: Operation returned an invalid status 'Accepted'.
Page URL
https://learn.microsoft.com/en-us/python/api/azure-mgmt-web/azure.mgmt.web.v2023_01_01.operations.certificatesoperations?view=azure-python
Content source URL
https://github.com/MicrosoftDocs/azure-docs-sdk-python/blob/main/docs-ref-autogen/azure-mgmt-web/azure.mgmt.web.v2023_01_01.operations.CertificatesOperations.yml
Document Version Independent Id
284b2cde-1df5-768a-2abf-6507c5632b26
Article author
@lmazuel
Metadata

ID: c5c81a2f-9b05-479c-e3f5-06757121cfad
Service: app-service
Sub-service: web-apps

Hi @Hasanofal - thanks for opening an issue. Tagging the right people to take a look. In the meantime, would you be able to provide the following info?

A sample to reproduce this issue.
Confirm whether you are using azure-mgmt-web v7.2.0.

Could you provide more detailed code about how to reproduce the error?

Hi, thanks for your reply

yes I'm using azure-mgmt-web 7.2.0

To reproduce the issue, first I have to add the correct A/CNAME and TXT records to my domain provider. Next, I create a host name binding. Then I create the certificate using the custom domain. The error only happens when creating a certificate and not when updating it. After getting the invalid status error, Azure still creates the certificate even though I got the error. So if I run it a second time after a few minutes, I won't get the error as the certificate will have been created already and it will only update it. Make sure all domain records are added to the domain provider so you don't get validation error.
`
from azure.identity import DefaultAzureCredential
from azure.mgmt.web import WebSiteManagementClient

credential = DefaultAzureCredential()
app_service_client = WebSiteManagementClient(
credential, subscription_id
)
app_service_client.web_apps.create_or_update_host_name_binding(
RESOURCE_GROUP_NAME,
web_app_name,
custom_domain,
{}
)
app_service_client.certificates.create_or_update(
RESOURCE_GROUP_NAME,
"certificate_name",
{
"location": "uaenorth",
"canonicalName": custom_domain,
"serverFarmId": web_app_plan_id,
},
)
`

according to the swagger definition here, response with status code except 200 would be considered as a failure, please follow this issue for updated news: https://github.com/Azure/azure-rest-api-specs/issues/27962.

Closing the issue and please use https://github.com/Azure/azure-rest-api-specs/issues/27962 to track.


Helm Values yaml documentation

@msohailhussain Am I good to merge this to main?

@msohailhussain Am I good to merge this to main?

I will review today. Please wait.

@msohailhussain I'm ready for a second code review
I was using this command to test
helm install -f values.yaml --dry-run --debug debugging-only ./

lgtm


Remove route controllers

Ingress to Route Controller
Service Ingress IP Controller

they were moved into https://github.com/openshift/route-controller-manager

/lgtm

/retest

there was a mixup in the operator PR with image names so we have to hold for now: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/265
/hold

route-controller-image has been included in the payload
/hold cancel
/retest

/label docs-approved
/label px-approved
/label qe-approved


the posix_arrange_msg function crashes.

it crashes when more than 1 message of the same priority present in the messageQ.
solution:
To verify temp_q pointer is not null before Swap the messages:
if(temp_q)
{
/* Swap the messages.  */
for ( msg = 0; msg < 4; msg++)
{
    swap = *temp_q;
    *temp_q = *Qread;
    *Qread = swap;
    temp_q++;
    Qread++;
}

}

Do you have a test case for this? I am putting two and three messages of the same priority in the queue and not seeing any crash.

@boria8 - as of release 6.2.0, this issue has been fixed: https://github.com/azure-rtos/threadx/blob/master/utility/rtos_compatibility_layers/posix/px_mq_arrange_msg.c


COMPAT: set ACLs with an empty AccessControlList

FIXES #304
I checked against AWS S3 and when:

adding multiple AccessControlList section

OR

AccessControlList section is not empty and there is no Grant section
it returns a MalformedACLError.

If AccessControlList section is empty <AccessControlList></AccessControlList>, AWS does not throw any error.

@ironman-machine please

Hello @nicolas2bert
Sorry, but this PR cannot be merged in this state, please rebase it on the master branch


Temporary file is not being used when accumulated_num option is set.

When using --accumulated_num the progress is not saved to the temporary file somebook_bilingual_temp but it's directly written to somebook_bilingual.
Additionally the --resume option does not work together with accumulated_num

its only work for txt file I wonder, will check this week thanks for the issue


Add examples for packaging and move it to the collapsed fields

Add examples for packaging and move it to the collapsed fields

or move packaging to the collapsed fields

Thanks @VaiTon for the fix. Closing the issue


"node(s) had volume node affinity conflict" when launching notebook with multiple volumes

I'm reposting an issue from upstream kubeflow/kubeflow here as it is specific to Kubeflow on AWS. All details can be found there.
https://github.com/kubeflow/kubeflow/issues/7109

Hello thanks for the issue. This is something we will look into to determine if it is a bug or a feature request.

An EBS volume and the instance to which it attaches must be in the same Availability Zone. This is a requirement from the EBS service side. From the Kubeflow side, there nothing that can be done at the moment to change this but following are a few options/recommendations:
EKS supports other storage backends that could be used zone independently, with different properties (like performance, pricing etc). For example there is AWS EFS that can be used in any AZ within an AWS region.
That being said, using EBS is a common option with Kubernetes PVs if you can:

Spin up pods/notebook instances in same AZ using using node affinity or nodeSelector
Create a custom StorageClass to specify the zones where volumes are created
Setup a nodegroup + Autoscaling for nodes in that AZ

Hope this helps. Let us know if you more information in any of the above options.


Make certbot no longer set itself up as standalone

Fixes #20593.
Testing plan: Tested upgrading a server, as well as installing a new one.  Both resulted in a authenticator = webroot certificate.

This looks correct to me; it'd be great to get a review from @andersk as well, since  this is a type of thing that would be annoying if it were buggy.

Cherry-picked to 4.x as b44a1b68f6 and 2e1e2b08f1.


CMB-1656: Add a kind for dynamic change of style sheet 'moon.Theme'

Issue:

Container has 'dark' theme by default
'Light' theme app additionally load one more css file

Fix:

Instead of load additional file, replace existing file while app loading.

DCO-1.1-Signed-Off-By: Kunmyon Choi<EMAIL_ADDRESS>

@KunmyonChoi  - should we close this since the Jira has been closed as no longer required? thanks.

Closing this PR per smartshare implemented overriding app css on that specific dark theme case.


rbd: add kmip encryption type

The Key Management Interoperability Protocol (KMIP)
is an extensible communication protocol
that defines message formats for the manipulation
of cryptographic keys on a key management server.
Ceph-CSI can now be configured to connect to
various KMS using KMIP for encrypting RBD volumes.
https://en.wikipedia.org/wiki/Key_Management_Interoperability_Protocol
Resolves: https://github.com/ceph/ceph-csi/issues/3282
Signed-off-by: Rakshith R<EMAIL_ADDRESS>

@Rakshith-R looks like ans field has to be renamed for the codespell to be happy :)
./internal/kms/kms_util.go:49: ans ==> and
./internal/kms/kms_util.go:61: ans ==> and
./internal/kms/kms_util.go:71: ans ==> and
./internal/kms/kms_util.go:81: ans ==> and

Please split the vendor part from the actual feature, that makes it much easier to review.

@Rakshith-R can you please address the comments.. I am starting with Release work of 3.7, thats why :)

mostly nits
Add a note somewhere about how it was tested?

This was tested with pykimp server instance deployed in kubernetes cluster,
I'll open an issue to track this, so we can add e2e for kmip encryption in a follow-up pr.

addressed the comments PTAL @nixpanic @humblec


Add a parse_number() function

I think it could be really useful to have a function called parse_number() that expects a written number and returns the equivalent number (Python number):
>>> parse_number("one")
1

>>> parse_number("twenty six")
26

It's easy to imagine use cases, and it will be useful to refactor the current code.
With the currently implemented logic, it shouldn't be difficult to create it by using the number_builder.
>>> number_builder(["twenty", "six"])
['26']

>>> number_builder(["one"])
['1']

We should decide what to do if we pass something else. I think we could return just None.
>>> parse_number("one second")
None

>>> parse_number("twenty six cars")
None

Of course, this will be multi-locale and it would be useful to accept a locales or locale argument:
>>> parse_number("uno", locales=["en"])
None

>>> parse_number("uno", locales=["es"])
1

Hei @arnavkapoor, I think that we could prioritize this as it will help to define a better structure for the parser.py.

By the way, I think that: parse_number("1") should return 1 too.

Sure @noviluni I will start working on this, just confirming even multiple numbers in the same string things should return None.
> parse_number("one two")
None

I will create a PR in a while, however not very sure how exactly do we use it within the main parse
function for reusability.

even multiple numbers in the same string things should return None.

I think so.

not very sure how exactly do we use it within the main parse function
I think that we could extract part of the logic inside number_builder() in another function to be reused in parse_number(). In fact, the logic would be pretty the same except for:

we will return a number instead of an str.
we shouldn't return anything if there is a token that is not part of the number.

However, I prefer you to code something and discuss it through the PR, as it will be easier for both to see how can it be applied / refactored. :smile:

Hi @noviluni sure I was wondering about the feasibility of the main  parse() function using this parse_number(). But of course, they both can use the common number_builder().
This structure is what I went with for the current PR #9.

Hi @arnavkapoor! As we have merged the PR I think you can close this issue. :muscle:


Compute plugin list in bash completion for docker daemon --authz-plugin

#18836 added the available Authorization plugins to the output of docker info.
This PR makes this information available in bash completion for docker daemon --authz-plugin instead of allowing any string.
Ref https://github.com/docker/docker/pull/18649#issuecomment-164573830

Awesome, @albers! Approach looks the same as the "volume" plugins, so
LGTM :-)
ping @tianon for review/merge

Niiice, this is so cool. :+1:
LGTM


Self compiled version of evcc

Is there a way to use a version of evcc that I compiled myself.
Thanks

Yes you can set this in config.yaml


STUB: Remove shuffling cycle from VMC

What is wrong?
STUB
How can it be fixed
STUB

@mhchia I think this issue has already been fixed via #348? If not, please reopen it.


want find dataset

where are the datasets ？ any link？

The datasets are documented on the results website and in the Python documentation.


Not working for mailables

I am using the new version 3.6.0
When returning a view in a controller:
return view (['template' => '{{$token}}'], ['token' => 'I am the child template']);
Works fine, but when trying to create a Mailable:
return $this->view (['template' => '{{$token}}'], ['token' => 'I am the child template']);
Creates an empty mail. When I set a file as template, it works fine.

I now use a workaround:
rendering the template like this:
$this->content = \View::make(['template' => '{{$token}}'], ['token' => 'I am the child template'])->render();
and injecting that rendered template as a variable in a template file with contents: "{!! $content !!}"

@martijnc94 I think that is the work around for mailable. I think something similar was mentioned in this ticket https://github.com/TerrePorter/StringBladeCompiler/issues/21

@martijnc94 if you don't want to create extra blade file you can try this way.
$html = view(['template' => '{{$token}}'], ['token' => 'I am the child template']);
return $this->html($html->render());

@kyawkyawsoezhu That's the way I eventually figured out.


403 error with provisioning request for Learning Pathways

Hi,
I've had 2 provisioning requests for the Learning Pathways solution fail, both with a 403 error. I tried before and had the "appCatalog permissions" issue, but have had an app catalog with this user having global-admin access as well as both Site Collection Admin and Site Owner permissions to the App Catalog for years already since we used this tenant for 2013-style Add-In development back before dev tenants were a thing.
Below is the content of the most recent error email I received:
Your request to provision "Microsoft 365 learning pathways"  template failed! Please  report the issue here providing the following provisioning action correlation ID: 406b1b5d-a2b3-4681-985d-4aa6edaad8be !
Please provide use feedback around your used template or the service in general by using our feedback form  . Thank you for your input advance.
Exception: The remote server returned an error: (403) Forbidden.
Provisioning action correlation ID: 406b1b5d-a2b3-4681-985d-4aa6edaad8be

Hi, Thanks for your feedback. I'm closing this item because it is now outdated.


[Bug] : Inconsistent footer between home page and other routes

Description
Hey 👋, I am a Hacktoberfest'23 contributor
What ?
I would like to fix the inconsistent footer among the pages the social icons are only appearing for home page not other. But the thing is that it should be consistent for all.
Screenshots

FrontendFreaks Page

Learn Page

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Kindly assign it to me under HacktoberFest'23
I will surely complete it ASAP

Thank You

sure @AKACHI-4 Assigned to you :)


bump isObject@v1, update check

I catch that when I start one PR where I need both isobject and get-value and notice that npm download isobject twice. In case, there's no breaking changes in isobject@v1, so no problem to bump to it.

btw sorry for the delaying utils merge, I'll do it in a bit

@jonschlinkert ping


Expose Encodings variable for ORTC

It's not possible to use a RTPReceiver without it.

I've been trying to figure out if the SSRC is actually required and I'm not sure. There doesn't seem to be a convenient way to get this information from the API like some of the other information required for signaling. Do you even need to signal the track SSRC?
c.receiver.Receive(webrtc.RTPReceiveParameters{
  Encodings: webrtc.RTPDecodingParameters{
    webrtc.RTPCodingParameters{SSRC: ssrc},
  },
})

Pull Request Test Coverage Report for Build 2211

7 of 11 (63.64%)  changed or added relevant lines in 2 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage remained the same at 79.653%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

rtpreceiver.go
5
9
55.56%

Files with Coverage Reduction
New Missed Lines
%

internal/mux/mux.go
2
95.0%

Totals

Change from base Build 2165:
0.0%

Covered Lines:
3766

Relevant Lines:
4728

💛  - Coveralls

Pull Request Test Coverage Report for Build 2211

7 of 11 (63.64%)  changed or added relevant lines in 2 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage remained the same at 79.653%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

rtpreceiver.go
5
9
55.56%

Files with Coverage Reduction
New Missed Lines
%

internal/mux/mux.go
2
95.0%

Totals

Change from base Build 2165:
0.0%

Covered Lines:
3766

Relevant Lines:
4728

💛  - Coveralls


"Error when evaluating expression validation.name.invalid"

Hi,
great work on this plugin!
When I access to the validation object, i got in the console the following message:
[Vue warn]: Error when evaluating expression "validation.name.invalid"

While this doesn't break the application I wonder if I'm missing something in the setup.
Thanks in advance!

I think that there are not problem in your setup code.
This is small problem of vue-validator.
See http://vuejs.org/guide/best-practices.html#Data_Initialization
But it does not effect in application behavior.
That's OK. :smile:
In the v2.0, I'm going to resolve it !

Ok thanks for the explanation!

I met the same problem.If the level is too much,it's not convenient to initialize your data like this instead of an empty object.I do this now:
validation&&validation.form.name.invalid

I found if I turned on Vue.config.debug, Vue will throw exception, the app will stop working.

+1

I can also verify this. My application was breaking until I set Vue.config.debug to false. I get the same warning, although that app now behaves as expected.

Sorry, I cannot find a solution to this issue yet.
This issue is difficult.
I'm asking the advise about related issue at below forum.
http://forum.vuejs.org/topic/957/how-to-avoid-evaluating-expression-warning-for-plugin-author
This issue is occuring when we using development version of vue.js
In development of app, we can avoid this issue by setting the Vue.config.warnExpressionErrors = false.
I'll try to write the README.md later. :smile_cat:


Setup architecture and add menu and slots

Resolved questions listed above

@ckeisc43rd-cjtsai Please add the prittierrc file and commands to yarn


where can set data argument

like color jit, image flip x, y , swap xy, or something

default flip x or flip x y?

Custom data argument could be added in /mmdet/datasets/extra_aug.py.
Here is an expmple for flip:
class Flip(object):
    """
    Filp img and bboxes.
    Author: Zhu Feijia<EMAIL_ADDRESS>    """

    def __init__(self, direction='r'):
        """
        :param direction:
            'h':horizontal
            'v':vertical
            'hv': h then v (180° rotated )
            'r': random choose from h,v,hv
        """
        self._directions = ['h', 'v', 'hv', 'r']
        assert direction in self._directions
        self.direction = direction
        print()

    def __call__(self, img, bboxes, labels):
        """
        Filp img and bboxes.
        :param img: :An image.
        :param bboxes(ndarray): shape (..., 4*k): (xmin,ymin,xmax,ymax)
        :param labels: labels of bboxes
        :return: flipped img, flipped bboxes, labels
        """
        if self.direction == 'h':
            img = np.flip(img, axis=1)
        elif self.direction == 'v':
            img = np.flip(img, axis=0)
        elif self.direction == 'hv':
            img = np.flip(img, axis=1)
            img = np.flip(img, axis=0)
        bboxes = self.bbox_flip(bboxes, img.shape)

        if self.direction == 'r':
            self.direction = random.choice(self._directions[:-1])
            img, bboxes, labels = self(img, bboxes, labels)
            self.direction = 'r'
        return img, bboxes, labels

    def bbox_flip(self, bboxes, img_shape, direction=None):
        """
        Flip bboxes.
        :param bboxes: ndarray, shape (..., 4*k): (xmin,ymin,xmax,ymax)
        :param img_shape: img.shape
        :param direction:
                'h':horizontal
                'v':vertical
                'hv': h then v (180° rotated )
        :return: flipped bboxes
        """

        d = self.direction if direction is None else direction
        assert bboxes.shape[-1] % 4 == 0
        h, w = img_shape[0], img_shape[1]
        flipped = bboxes.copy()

        if d == 'h':
            flipped[..., 0::4] = w - bboxes[..., 2::4] - 1
            flipped[..., 2::4] = w - bboxes[..., 0::4] - 1
        elif d == 'v':
            flipped[..., 1::4] = h - bboxes[..., 3::4] - 1
            flipped[..., 3::4] = h - bboxes[..., 1::4] - 1
        elif d == 'hv':
            flipped[..., 0::4] = w - bboxes[..., 2::4] - 1
            flipped[..., 2::4] = w - bboxes[..., 0::4] - 1
            flipped[..., 1::4] = h - bboxes[..., 3::4] - 1
            flipped[..., 3::4] = h - bboxes[..., 1::4] - 1
        return flipped

More transforms can be found in mmcv/image/transforms/

Does it support different img scale in training ? I find it seem to be scale to 1388 x 800.

Then, how to change the config file or codes to use Flip in /mmdet/datasets/extra_aug.py ?

Just add it to class ExtraAugmentation.init.
Details in the codes.

tks

Flip augmentation is actually built in and used in all configs with 50%
probability (look for flip in the config file). However, using the extra
aug is probably the best way to add more augmentation until they finish the
training pipeline configuration.
On Wed, Aug 21, 2019, 20:35 Zhou Hang<EMAIL_ADDRESS>wrote:

tks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/open-mmlab/mmdetection/issues/1225?email_source=notifications&email_token=AC7IWC26KQ4ELR7GZEZMY73QFT455A5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y4V2A#issuecomment-523356904,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC7IWC6T6NR5WJBW43C2Z6TQFT455ANCNFSM4IOALKJQ
.

Flip augmentation is actually built in and used in all configs with 50% probability (look for flip in the config file). However, using the extra aug is probably the best way to add more augmentation until they finish the training pipeline configuration.
…
On Wed, Aug 21, 2019, 20:35 Zhou Hang @.***> wrote: tks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1225?email_source=notifications&email_token=AC7IWC26KQ4ELR7GZEZMY73QFT455A5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y4V2A#issuecomment-523356904>, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7IWC6T6NR5WJBW43C2Z6TQFT455ANCNFSM4IOALKJQ .

The built-in method has some limitations, such as the inability to specify the flip direction of the bbox.
BTW  I implemented an enhance augmentation mode, that can schedule different augmentation methods according to their probability and order. Waiting for the new pipeline.

So did I, although I didn't put too much effort into it, considering it
will probably have to be re-written. Did you get noticeable improvement
from the augmentation? I found that in my case it didn't help as much as in
classification problems I worked with.
On Wed, Aug 21, 2019 at 8:54 PM zzzfinal0<EMAIL_ADDRESS>wrote:

Flip augmentation is actually built in and used in all configs with 50%
probability (look for flip in the config file). However, using the extra
aug is probably the best way to add more augmentation until they finish the
training pipeline configuration.
… <#m_-4483078431302119069_>
On Wed, Aug 21, 2019, 20:35 Zhou Hang @.***> wrote: tks — You are
receiving this because you are subscribed to this thread. Reply to this
email directly, view it on GitHub <#1225
https://github.com/open-mmlab/mmdetection/issues/1225?email_source=notifications&email_token=AC7IWC26KQ4ELR7GZEZMY73QFT455A5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y4V2A#issuecomment-523356904>,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC7IWC6T6NR5WJBW43C2Z6TQFT455ANCNFSM4IOALKJQ
.
The built-in method has some limitations, such as the inability to specify
the flip direction of the bbox.
BTW I implemented an enhance augmentation mode, that can schedule
different augmentation methods according to their probability and order.
Waiting for the new pipeline.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/open-mmlab/mmdetection/issues/1225?email_source=notifications&email_token=AC7IWCZBD6O4BIU5TBQJAH3QFT7CXA5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y6LOQ#issuecomment-523363770,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC7IWC5DHW6QINK23R7FLGTQFT7CXANCNFSM4IOALKJQ
.

So did I, although I didn't put too much effort into it, considering it will probably have to be re-written. Did you get noticeable improvement from the augmentation? I found that in my case it didn't help as much as in classification problems I worked with.
…
On Wed, Aug 21, 2019 at 8:54 PM zzzfinal0 @.> wrote: Flip augmentation is actually built in and used in all configs with 50% probability (look for flip in the config file). However, using the extra aug is probably the best way to add more augmentation until they finish the training pipeline configuration. … <#m_-4483078431302119069_> On Wed, Aug 21, 2019, 20:35 Zhou Hang @.> wrote: tks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1225 <#1225>?email_source=notifications&email_token=AC7IWC26KQ4ELR7GZEZMY73QFT455A5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y4V2A#issuecomment-523356904>, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7IWC6T6NR5WJBW43C2Z6TQFT455ANCNFSM4IOALKJQ . The built-in method has some limitations, such as the inability to specify the flip direction of the bbox. BTW I implemented an enhance augmentation mode, that can schedule different augmentation methods according to their probability and order. Waiting for the new pipeline. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1225?email_source=notifications&email_token=AC7IWCZBD6O4BIU5TBQJAH3QFT7CXA5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y6LOQ#issuecomment-523363770>, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7IWC5DHW6QINK23R7FLGTQFT7CXANCNFSM4IOALKJQ .

Yes, they change it too fast, so we have to slow down.
In my defect detection case,  maybe my images isn't enough, the final mAP +15% after augmentation. They are Pcajitter/Flip/Shift/Rotated.

So did I, although I didn't put too much effort into it, considering it will probably have to be re-written. Did you get noticeable improvement from the augmentation? I found that in my case it didn't help as much as in classification problems I worked with.
…
On Wed, Aug 21, 2019 at 8:54 PM zzzfinal0 @.**> wrote: Flip augmentation is actually built in and used in all configs with 50% probability (look for flip in the config file). However, using the extra aug is probably the best way to add more augmentation until they finish the training pipeline configuration. … <#m-4483078431302119069_> On Wed, Aug 21, 2019, 20:35 Zhou Hang @.**_> wrote: tks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1225 <#1225>?email_source=notifications&email_token=AC7IWC26KQ4ELR7GZEZMY73QFT455A5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y4V2A#issuecomment-523356904>, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7IWC6T6NR5WJBW43C2Z6TQFT455ANCNFSM4IOALKJQ . The built-in method has some limitations, such as the inability to specify the flip direction of the bbox. BTW I implemented an enhance augmentation mode, that can schedule different augmentation methods according to their probability and order. Waiting for the new pipeline. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1225?email_source=notifications&email_token=AC7IWCZBD6O4BIU5TBQJAH3QFT7CXA5CNFSM4IOALKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y6LOQ#issuecomment-523363770>, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7IWC5DHW6QINK23R7FLGTQFT7CXANCNFSM4IOALKJQ .

Yes, they change it too fast, so we have to slow down.
In my defect detection case, maybe my images are not enough, the final mAP +15% after augmentation. They are Pcajitter/Flip/Shift/Rotated.

Could you share your augmentation code of  Pcajitter/Flip/Shift/Rotated.

what's mean of shift ? could you share your code of Pcajitter/Shift

do you means shift is subpixel shift


[GOBBLIN-1739]Define Datanodes and Dataset Descriptor for Iceberg

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1739

Description

[ ] Here are some details about my PR, including screenshots (if applicable):
created iceberg datanode on top of hive datanode which specifies having a metastore uri
created dataset descriptor extending the sql descriptor

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Added unit tests for the iceberg datanode and dataset descriptor

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3596 (7a5d7c9) into master (bc222f5) will decrease coverage by 0.00%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##             master    #3596      +/-   ##
============================================
- Coverage     46.88%   46.87%   -0.01%
- Complexity    10673    10679       +6
============================================
  Files          2120     2122       +2
  Lines         83067    83085      +18
  Branches       9252     9255       +3
============================================
+ Hits          38944    38948       +4
- Misses        40555    40563       +8
- Partials       3568     3574       +6

Impacted Files
Coverage Δ

...s/flowgraph/datanodes/iceberg/IcebergDataNode.java
40.00% <40.00%> (ø)

...vice/modules/dataset/IcebergDatasetDescriptor.java
53.84% <53.84%> (ø)

...a/org/apache/gobblin/cluster/GobblinHelixTask.java
60.21% <0.00%> (-4.31%)
:arrow_down:

.../apache/gobblin/runtime/api/JobExecutionState.java
79.43% <0.00%> (-0.94%)
:arrow_down:

...odules/orchestration/InMemoryUserQuotaManager.java
70.07% <0.00%> (-0.79%)
:arrow_down:

.../org/apache/gobblin/cluster/GobblinTaskRunner.java
63.63% <0.00%> (-0.32%)
:arrow_down:

...anagement/copy/replication/ConfigBasedDataset.java
68.87% <0.00%> (ø)

...lin/util/filesystem/FileSystemInstrumentation.java
100.00% <0.00%> (+7.14%)
:arrow_up:

...a/org/apache/gobblin/util/limiter/NoopLimiter.java
60.00% <0.00%> (+20.00%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Chronos fails to start with zk auth

If we run a chronos with --zk-hosts chronos:8WDxe30zbCzcuZJc@zookeeper.service.consul:2181 param we get the following error:
May 29 09:52:09 clean-deploy-test-01 chronos[17850]: 2015-05-29 09:52:09,300:17850(0x7fd7f1374700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=chronos:8WDxe30zbCzcuZJc@z
ookeeper.service.consul:2181 sessionTimeout=10000 watcher=0x7fd7fae3c1e0 sessionId=0 sessionPasswd=<null> context=0x7fd818021d40 flags=0
May 29 09:52:09 clean-deploy-test-01 chronos[17850]: F0529 09:52:09.333401 18132 zookeeper.cpp:113] Failed to create ZooKeeper, zookeeper_init: No such file or directory [2]

host=chronos:8WDxe30zbCzcuZJc@zookeeper.service.consul:2181 -  i think this is the cause of the issue.
The marathon log looks a little different:
May 29 10:12:27 clean-deploy-test-01 marathon[20835]: 2015-05-29 10:12:27,840:20835(0x7fb639368700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=zookeeper.service.consul:
2181 sessionTimeout=10000 watcher=0x7fb642e3c1e0 sessionId=0 sessionPasswd=<null> context=0x7fb648beeb00 flags=0

i cant say if marathon just drops the auth part and this makes it start, or its an error in the chronos args parser

Is this error still outstanding? Did PR #390 fix your issue?

Closing, because Chronos is well integrated with Mantl. Reopen if you still experience issues!