java.io.IOException: Not found: Uris gs

Hi Sam,
Ive been running my streaming ETL jobs (non watermark ones) in Databricks UAT for about 1.5 weeks.
For the most part everything is working fine, however I do get the occasion IO error to GCP bucket which is odd.
java.io.IOException: Not found: Uris gs://sdv-analytics-spark-streaming/hadoop/tmp/spark-bigquery/spark-bigquery-1498172713848=666506084/*

I think this seems to be possibly because of my config / job setup rather then application/transformation logic.

So one thing to point out, in a given SS job, I have 1 streaming input DF ( reading input events off kinesis), but I stream to 3 targets (doing some transformationsin the middle) - 2 seperate BigQuery tables, 1 S3 bucket sinks - so 3 streamingQuery's in total in a single SS job.
For the 2 BQ target tables in my BQ config, I am using a shared GCP storage folder within the job for SS for intermediary staging area, NOT individual ones.
spark.sqlContext.setBigQueryGcsBucket(sdv-analytics-spark-streaming)

Im wondering if with this IOError, having 2 BQ streamingQuery's from the same single input source DF in the same job is causing this and that the gcp folder suffix part ....
spark-bigquery-1498172713848=666506084

...is wrongly shared as a staging area (as maybe its derived from the input source) ?  And  does one streamingQuery cleans this up while the other one expects it to be there?  Or maybe Im barking up the wrong tree too!!
Should each target StreamingQuery have its OWN GCP bucket rather then a shared one because they are working off the same input stream?
e.g  spark.sqlContext.setBigQueryGcsBucket(sdv-analytics-spark-streaming-)
The spark session though is unique per job and not shared, so seems I would need to break the 1 job up into 3 to do this given the shared context - 1 input to 1 sink instead of 1 input to 3?
This could be because of something else, e.g intermittent network issues for example?
When I restart the job it did pick up where it left off and didnt error so its intermittent.
Does this make sense?  Should I refactor to 1:1 input to BQ output?
Sorry its a bit long-winded!
Thanks heaps!
Main part of the trace in logs below:
java.io.IOException: Not found: Uris gs://sdv-analytics-spark-streaming/hadoop/tmp/spark-bigquery/spark-bigquery-1498172713848=666506084/*
at com.google.cloud.hadoop.io.bigquery.BigQueryUtils.waitForJobCompletion(BigQueryUtils.java:95) at com.samelamin.spark.bigquery.BigQueryClient.com$samelamin$spark$bigquery$BigQueryClient$$waitForJob(BigQueryClient.scala:143) at com.samelamin.spark.bigquery.BigQueryClient.load(BigQueryClient.scala:110) at com.samelamin.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:163) at com.samelamin.spark.bigquery.streaming.BigQuerySink.addBatch(BigQuerySink.scala:27) at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatch$1.apply$mcV$sp(StreamExecution.scala:658)

Hi Sam,
Getting a few more of these java.io.IOException: Not found: Uris gs:// excpetions occuring, do you think its worth doing as described above and isolating the storgage buckets?
I dont loose any data when the errors occur (of course), I just restart the job and it works.  The io errors appear intermittent at the moment quite hard to debug why tit cant find the gcp storage bucket for the job?  ANy ideas?
Ive had a handful of instances where I had duplicates in my SS BQ sink table too - possibly when there is a replay by spark from kinesis source due an issue somewhere and as BQ is not an SS idempotant sink so am getting duplicates appended......Ill be able to address this with the new watermarking feature anyway once you introduce it, can set drop_duplicates....that still assumes no duplicaates come after the watermark though, which the only way is to support the SS 'update' mode.  I see BQ does support SQL updates now, but those quotas look tiny!
Thanks mate

hey @kurtmaile just finding time to look through these issues
I should think that if you had seperate buckets for both checkpointing and data transfer it should resolve the issue. You can try setting a different key in the bucket for the job if you want everything in one master bucket. eg {bucket-name}/{job_name}
Give it a try and give me a shout with the results. Im having a look at the watermarking issue now

Hi Sam!
Thanks so much.
I've done as you mentioned will see how testing goes today.
And awesome news on the watermarking! Legend thanks
Kurt
On Wed, 28 Jun 2017 at 20:57, Sam Elamin<EMAIL_ADDRESS>wrote:

hey @kurtmaile https://github.com/kurtmaile just finding time to look
through these issues
I should think that if you had seperate buckets for both checkpointing and
data transfer it should resolve the issue. You can try setting a different
key in the bucket for the job if you want everything in one master bucket.
eg {bucket-name}/{job_name}
Give it a try and give me a shout with the results. Im having a look at
the watermarking issue now
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-311769806,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATIvhSHoBAkxnlHUk4x8LsUyrWaVTWhtks5sIrASgaJpZM4ODLy1
.

Hi Sam,
On this issue, I know you are still recovering so only when you get some time to look, Im oddly still getting intermittent issues with the gcp staging bucket.  I've split my jobs out so each is an isolated streaming notebook (but using a shared 'Library' - more on this later) - with its gcp bucket and s3 checkpoint in the spark session.
In databricks, I am using the 3.0 Dbricks runtime (so v2.2RC with dbricks extra bells and whistles).  The cluster by default is set to NOT share the spark session on the cluster, so each job should be isolated and have its own SS.
However......refer below

You can see that the gcpp staging bucket reported in the ERROR is not the one for this notebook job in the cell above when configured, rather, its the one for one of the others!  These are highlighted in red.  Its reporting on the inventory bucket, but its configured in the above cell to the customeraccountevents bucket (which is the config for a seperate job)?

Now I absolutely dont think dbricks would get this ss sharing wrong, so its something Im doing likely.  But I have totally isolate the jobs so a little bit stummped and there seems no way it should be looking for that staging bucket.
One thing I can say, I try keep the notebooks as light as possible, mostly just configuring the srreaing job, and they use a reusable library I created for our solution to hide some details from users.  Fairly standard pattern.  So the BQ streaming sink class below (use in the notebook you can see in the image) is as follows, pretty simple.
class BigQueryStreamingSink(spark: SparkSession = SparkSession.builder().getOrCreate(),
                            gcpKeyFilePath: String,
                            bqProjectId: String,
                            bqDataset: String,
                            bqBucket: String,
                            checkpoint: String,
                            bqRegion: String = "EU",
                            bqAllowSchemaUpdates: Boolean = true) extends Serializable {

  spark.sparkContext.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
  spark.sqlContext.setGcpJsonKeyFile(gcpKeyFilePath)
  spark.sqlContext.setGSProjectId(bqProjectId)
  spark.sqlContext.setBigQueryProjectId(bqProjectId)
  spark.sqlContext.setBigQueryGcsBucket(bqBucket)
  spark.sqlContext.setBigQueryDatasetLocation(bqRegion)

  if(bqAllowSchemaUpdates)
    spark.sqlContext.setAllowSchemaUpdates()

  def writeStreamToBigQueryTable(streamingDf: DataFrame,
                                 bqTable: String,
                                 outputMode: OutputMode = OutputMode.Append): StreamingQuery = {
    streamingDf
      .writeStream
      .format("com.samelamin.spark.bigquery")
      .option("checkpointLocation", checkpoint)
      .option("tableReferenceSink", s"$bqProjectId:$bqDataset.$bqTable")
      .outputMode(outputMode)
      .start()
  }

This is part of this reusable library I mentioned that gets loaded into dbicks fter my builds and onto the cluster.   But I cant see this being an issue, unless Iver got this wrong, as the instance of this is created in the context of the isolated jobs jvm?  Or am I missing an very obvious thing here?
The job aborts at this point, and retries again as it is set up to, and usually works first re-attempt, if not a couple.  I never loose data, but I do get dupliates as it retries the batch again from the last checkpoint.  Which itself isnt an issue, I need to accomodate them anyway, and I dont loose data.  But it does just pick back up on the next run and work.....so its intermittent, anoying, but not a show stopper.
Any of your expert help appreciated, when you are feeling up to it of course no rush please heal and get better mate.
Regards and best wishes
Kurt

Hi Kurt
I will try and replicate when I have time, can you just confirm that not
only are you using seperate buckets but also different checkpoint and
transactionLog locations
Try separating everything at all to rule out any of these issues
Regards
Sam
On Fri, Jul 7, 2017 at 5:16 PM, kurtmaile<EMAIL_ADDRESS>wrote:

Hi Sam,
On this issue, I know you are still recovering so only when you get some
time to look, Im oddly still getting intermittent issues with the gcp
staging bucket. I've split my jobs out so each is an isolated streaming
notebook (but using a shared 'Library' - more on this later) - with its gcp
bucket and s3 checkpoint in the spark session.
In databricks, I am using the 3.0 Dbricks runtime (so v2.2RC with dbricks
extra bells and whistles). The cluster by default is set to NOT share the
spark session on the cluster, so each job should be isolated and have its
own SS.
However......refer below
[image: ioerror]
https://user-images.githubusercontent.com/20066181/27960800-7ddd9634-6324-11e7-906b-0d9d49dec913.png
You can see that the gcpp staging bucket reported in the ERROR is not the
one for this notebook job in the cell above when configured, rather, its
the one for one of the others! These are highlighted in red. Its reporting
on the inventory bucket, but its configured in the above cell to the
customeraccountevents bucket (which is the config for a seperate job)?
[image: screenshot 2017-07-07 15 01 03]
https://user-images.githubusercontent.com/20066181/27960997-2ca82daa-6325-11e7-87fa-ad4124366307.png
Now I absolutely dont think dbricks would get this ss sharing wrong, so
its something Im doing likely. But I have totally isolate the jobs so a
little bit stummped and there seems no way it should be looking for that
staging bucket.
One thing I can say, I try keep the notebooks as light as possible, mostly
just configuring the srreaing job, and they use a reusable library I
created for our solution to hide some details from users. Fairly standard
pattern. So the BQ streaming sink class below (use in the notebook you can
see in the image) is as follows, pretty simple.
class BigQueryStreamingSink(spark: SparkSession = SparkSession.builder().getOrCreate(),
gcpKeyFilePath: String,
bqProjectId: String,
bqDataset: String,
bqBucket: String,
checkpoint: String,
bqRegion: String = "EU",
bqAllowSchemaUpdates: Boolean = true) extends Serializable {
spark.sparkContext.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.sqlContext.setGcpJsonKeyFile(gcpKeyFilePath)
spark.sqlContext.setGSProjectId(bqProjectId)
spark.sqlContext.setBigQueryProjectId(bqProjectId)
spark.sqlContext.setBigQueryGcsBucket(bqBucket)
spark.sqlContext.setBigQueryDatasetLocation(bqRegion)
if(bqAllowSchemaUpdates)
spark.sqlContext.setAllowSchemaUpdates()
def writeStreamToBigQueryTable(streamingDf: DataFrame,
bqTable: String,
outputMode: OutputMode = OutputMode.Append): StreamingQuery = {
streamingDf
.writeStream
.format("com.samelamin.spark.bigquery")
.option("checkpointLocation", checkpoint)
.option("tableReferenceSink", s"$bqProjectId:$bqDataset.$bqTable")
.outputMode(outputMode)
.start()
}
This is part of this reusable library I mentioned that gets loaded into
dbicks fter my builds and onto the cluster. But I cant see this being an
issue, unless Iver got this wrong, as the instance of this is created in
the context of the isolated jobs jvm? Or am I missing an very obvious thing
here?
The job aborts at this point, and retries again as it is set up to, and
usually works first re-attempt, if not a couple. I never loose data, but I
do get dupliates as it retries the batch again from the last checkpoint.
Which itself isnt an issue, I need to accomodate them anyway, and I dont
loose data. But it does just pick back up on the next run and work.....so
its intermittent, anoying, but not a show stopper.
Any of your expert help appreciated, when you are feeling up to it of
course no rush please heal and get better mate.
Regards and best wishes
Kurt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-313694685,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm5wk8ZJh5CZBWXUm1U6Rg9qWdLYxks5sLj2-gaJpZM4ODLy1
.

Hi Sam,
I can confirm this yes.

Each streaming query has its own unique gcp bucket for staging data, check,
S3 bucket / folder for the checkpoint most certainly.
Where is a transactionLog defined/configured though?
Hey thanks for even replying dont stress on the reply.
Cheers,
K
On 7 July 2017 at 15:41, Sam Elamin<EMAIL_ADDRESS>wrote:

Hi Kurt
I will try and replicate when I have time, can you just confirm that not
only are you using seperate buckets but also different checkpoint and
transactionLog locations
Try separating everything at all to rule out any of these issues
Regards
Sam
On Fri, Jul 7, 2017 at 5:16 PM, kurtmaile<EMAIL_ADDRESS>wrote:

Hi Sam,
On this issue, I know you are still recovering so only when you get some
time to look, Im oddly still getting intermittent issues with the gcp
staging bucket. I've split my jobs out so each is an isolated streaming
notebook (but using a shared 'Library' - more on this later) - with its
gcp
bucket and s3 checkpoint in the spark session.
In databricks, I am using the 3.0 Dbricks runtime (so v2.2RC with dbricks
extra bells and whistles). The cluster by default is set to NOT share the
spark session on the cluster, so each job should be isolated and have its
own SS.
However......refer below
[image: ioerror]
<https://user-images.githubusercontent.com/20066181/27960800-7ddd9634-
6324-11e7-906b-0d9d49dec913.png>
You can see that the gcpp staging bucket reported in the ERROR is not the
one for this notebook job in the cell above when configured, rather, its
the one for one of the others! These are highlighted in red. Its
reporting
on the inventory bucket, but its configured in the above cell to the
customeraccountevents bucket (which is the config for a seperate job)?
[image: screenshot 2017-07-07 15 01 03]
<https://user-images.githubusercontent.com/20066181/27960997-2ca82daa-
6325-11e7-87fa-ad4124366307.png>
Now I absolutely dont think dbricks would get this ss sharing wrong, so
its something Im doing likely. But I have totally isolate the jobs so a
little bit stummped and there seems no way it should be looking for that
staging bucket.
One thing I can say, I try keep the notebooks as light as possible,
mostly
just configuring the srreaing job, and they use a reusable library I
created for our solution to hide some details from users. Fairly standard
pattern. So the BQ streaming sink class below (use in the notebook you
can
see in the image) is as follows, pretty simple.
class BigQueryStreamingSink(spark: SparkSession = SparkSession.builder().
getOrCreate(),
gcpKeyFilePath: String,
bqProjectId: String,
bqDataset: String,
bqBucket: String,
checkpoint: String,
bqRegion: String = "EU",
bqAllowSchemaUpdates: Boolean = true) extends Serializable {
spark.sparkContext.hadoopConfiguration.set("fs.gs.impl",
"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.sqlContext.setGcpJsonKeyFile(gcpKeyFilePath)
spark.sqlContext.setGSProjectId(bqProjectId)
spark.sqlContext.setBigQueryProjectId(bqProjectId)
spark.sqlContext.setBigQueryGcsBucket(bqBucket)
spark.sqlContext.setBigQueryDatasetLocation(bqRegion)
if(bqAllowSchemaUpdates)
spark.sqlContext.setAllowSchemaUpdates()
def writeStreamToBigQueryTable(streamingDf: DataFrame,
bqTable: String,
outputMode: OutputMode = OutputMode.Append): StreamingQuery = {
streamingDf
.writeStream
.format("com.samelamin.spark.bigquery")
.option("checkpointLocation", checkpoint)
.option("tableReferenceSink", s"$bqProjectId:$bqDataset.$bqTable")
.outputMode(outputMode)
.start()
}
This is part of this reusable library I mentioned that gets loaded into
dbicks fter my builds and onto the cluster. But I cant see this being an
issue, unless Iver got this wrong, as the instance of this is created in
the context of the isolated jobs jvm? Or am I missing an very obvious
thing
here?
The job aborts at this point, and retries again as it is set up to, and
usually works first re-attempt, if not a couple. I never loose data, but
I
do get dupliates as it retries the batch again from the last checkpoint.
Which itself isnt an issue, I need to accomodate them anyway, and I dont
loose data. But it does just pick back up on the next run and work.....so
its intermittent, anoying, but not a show stopper.
Any of your expert help appreciated, when you are feeling up to it of
course no rush please heal and get better mate.
Regards and best wishes
Kurt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/samelamin/spark-bigquery/issues/26#
issuecomment-313694685>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AEHLm5wk8ZJh5CZBWXUm1U6Rg9qWdLYxks5sLj2-gaJpZM4ODLy1>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-313701524,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATIvhbtUCIjgoJFYc_wbI36miE166F77ks5sLkOzgaJpZM4ODLy1
.

note:  sometimes can take a good few hours for it to occur....
On 7 July 2017 at 16:07, Kurt Maile<EMAIL_ADDRESS>wrote:

Hi Sam,
I can confirm this yes.

Each streaming query has its own unique gcp bucket for staging data,
check, S3 bucket / folder for the checkpoint most certainly.
Where is a transactionLog defined/configured though?
Hey thanks for even replying dont stress on the reply.
Cheers,
K
On 7 July 2017 at 15:41, Sam Elamin<EMAIL_ADDRESS>wrote:

Hi Kurt
I will try and replicate when I have time, can you just confirm that not
only are you using seperate buckets but also different checkpoint and
transactionLog locations
Try separating everything at all to rule out any of these issues
Regards
Sam
On Fri, Jul 7, 2017 at 5:16 PM, kurtmaile<EMAIL_ADDRESS>wrote:

Hi Sam,
On this issue, I know you are still recovering so only when you get some
time to look, Im oddly still getting intermittent issues with the gcp
staging bucket. I've split my jobs out so each is an isolated streaming
notebook (but using a shared 'Library' - more on this later) - with its
gcp
bucket and s3 checkpoint in the spark session.
In databricks, I am using the 3.0 Dbricks runtime (so v2.2RC with
dbricks
extra bells and whistles). The cluster by default is set to NOT share
the
spark session on the cluster, so each job should be isolated and have
its
own SS.
However......refer below
[image: ioerror]
<https://user-images.githubusercontent.com/20066181/
27960800-7ddd9634-6324-11e7-906b-0d9d49dec913.png>
You can see that the gcpp staging bucket reported in the ERROR is not
the
one for this notebook job in the cell above when configured, rather, its
the one for one of the others! These are highlighted in red. Its
reporting
on the inventory bucket, but its configured in the above cell to the
customeraccountevents bucket (which is the config for a seperate job)?
[image: screenshot 2017-07-07 15 01 03]
<https://user-images.githubusercontent.com/20066181/
27960997-2ca82daa-6325-11e7-87fa-ad4124366307.png>
Now I absolutely dont think dbricks would get this ss sharing wrong, so
its something Im doing likely. But I have totally isolate the jobs so a
little bit stummped and there seems no way it should be looking for that
staging bucket.
One thing I can say, I try keep the notebooks as light as possible,
mostly
just configuring the srreaing job, and they use a reusable library I
created for our solution to hide some details from users. Fairly
standard
pattern. So the BQ streaming sink class below (use in the notebook you
can
see in the image) is as follows, pretty simple.
class BigQueryStreamingSink(spark: SparkSession =
SparkSession.builder().getOrCreate(),
gcpKeyFilePath: String,
bqProjectId: String,
bqDataset: String,
bqBucket: String,
checkpoint: String,
bqRegion: String = "EU",
bqAllowSchemaUpdates: Boolean = true) extends Serializable {
spark.sparkContext.hadoopConfiguration.set("fs.gs.impl",
"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.sqlContext.setGcpJsonKeyFile(gcpKeyFilePath)
spark.sqlContext.setGSProjectId(bqProjectId)
spark.sqlContext.setBigQueryProjectId(bqProjectId)
spark.sqlContext.setBigQueryGcsBucket(bqBucket)
spark.sqlContext.setBigQueryDatasetLocation(bqRegion)
if(bqAllowSchemaUpdates)
spark.sqlContext.setAllowSchemaUpdates()
def writeStreamToBigQueryTable(streamingDf: DataFrame,
bqTable: String,
outputMode: OutputMode = OutputMode.Append): StreamingQuery = {
streamingDf
.writeStream
.format("com.samelamin.spark.bigquery")
.option("checkpointLocation", checkpoint)
.option("tableReferenceSink", s"$bqProjectId:$bqDataset.$bqTable")
.outputMode(outputMode)
.start()
}
This is part of this reusable library I mentioned that gets loaded into
dbicks fter my builds and onto the cluster. But I cant see this being an
issue, unless Iver got this wrong, as the instance of this is created in
the context of the isolated jobs jvm? Or am I missing an very obvious
thing
here?
The job aborts at this point, and retries again as it is set up to, and
usually works first re-attempt, if not a couple. I never loose data,
but I
do get dupliates as it retries the batch again from the last checkpoint.
Which itself isnt an issue, I need to accomodate them anyway, and I dont
loose data. But it does just pick back up on the next run and
work.....so
its intermittent, anoying, but not a show stopper.
Any of your expert help appreciated, when you are feeling up to it of
course no rush please heal and get better mate.
Regards and best wishes
Kurt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/samelamin/spark-bigquery/issues/26#issue
comment-313694685>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEHLm5wk8
ZJh5CZBWXUm1U6Rg9qWdLYxks5sLj2-gaJpZM4ODLy1>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-313701524,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATIvhbtUCIjgoJFYc_wbI36miE166F77ks5sLkOzgaJpZM4ODLy1
.

Hey @kurtmaile  you can change the transaction log path by adding an option for it on the write stream, similar to what you would do with the the tableReferenceSink
     .option("tableReferenceSink",tableName2)

Hey @kurtmaile  you can change the transaction log path by adding an option for it on the write stream, similar to what you would do with the the tableReferenceSink
     .option("transactionLog",pathToLog)

Hey thanks Sam.
I've not set this (is this the WAL location)?, is this a problem do you
think?
Assume it is a local file system path?
Cheers
Kurt
On Tue, 11 Jul 2017 at 09:19, Sam Elamin<EMAIL_ADDRESS>wrote:

Hey @kurtmaile https://github.com/kurtmaile you can change the
transaction log path by adding an option for it on the write stream,
similar to what you would do with the the tableReferenceSink
.option("transactionLog",pathToLog)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-314370050,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATIvhe-RToW84jhJn2qsPIH547vTZSjBks5sMzAQgaJpZM4ODLy1
.

It is yes, but I think you can  also point it to cloud storage
Try separating the transaction log and see if it helps, I am skeptical whether it will fix it though but it is best to rule it out from the start

Hey,
I am going to implement this change today - quick question, if I explicitly set the WAL to something different will this cause any issues with existing streaming jobs WAL / metadata?  Im not doing anything stateful - just consume off kinesis stream and write to BQ.  Ill stop my running streaming jobs, make the upgrade and restart with new WAL location.
Is there any restrictions on setting the WAL - i.e existing directory needs to be in place, or has to be at a root level?  Naming conventions / extensions for the file? Does it need a 'file:' prefix or is this assumed?
Thinking '/'
Not alot on setting this, probably not very common I guess.  Sorry for the questions on it!
I think as you say this is unlikely to be it but will do anyway.
Kurt

Hey,
I could reproduce this by 'very quickly' starting two jobs (subscribed to same kinesis stream) in quick succession via the databricks jobs screen - very odd, almost race condition/threading issue.
Ill keep investigating.  The code in the shared library I load into Databricks is below FYI incase you notice anything squify.
My notebooks each instantiate an instance of the BigQueryStreamingSink, passing in their SparkContext, so seems threadsafe? (as the sparkContext is NOT shared as mentioned).
package com.xiatechs.analytics.datasource.google

import com.samelamin.spark.bigquery._
import org.apache.spark.sql.streaming.OutputMode
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.streaming.StreamingQuery
import org.apache.spark.sql.streaming.Trigger

class BigQueryStreamingSink(spark: SparkSession = SparkSession.builder().getOrCreate(),
                            gcpKeyFilePath: String,
                            bqProjectId: String,
                            bqDataset: String,
                            bqBucket: String,
                            checkpoint: String,
                            bqRegion: String = "EU",
                            bqAllowSchemaUpdates: Boolean = true) extends Serializable {

  spark.sparkContext.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
  spark.sqlContext.setGcpJsonKeyFile(gcpKeyFilePath)
  spark.sqlContext.setGSProjectId(bqProjectId)
  spark.sqlContext.setBigQueryProjectId(bqProjectId)
  spark.sqlContext.setBigQueryGcsBucket(bqBucket)
  spark.sqlContext.setBigQueryDatasetLocation(bqRegion)

  if(bqAllowSchemaUpdates)
    spark.sqlContext.setAllowSchemaUpdates()

  def writeStreamToBigQueryTable(streamingDf: DataFrame,
                                 bqTable: String,
                                 outputMode: OutputMode = OutputMode.Append): StreamingQuery = {
    streamingDf
      .writeStream
      .format("com.samelamin.spark.bigquery")
      .option("checkpointLocation", checkpoint)
      .option("tableReferenceSink", s"$bqProjectId:$bqDataset.$bqTable")
     // .option("transactionLog","/")
      .trigger(Trigger.ProcessingTime("2 minutes"))
      .outputMode(outputMode)
      .start()
  }
}

Kurt

Hi Sam,
Thanks heaps fo fixing the timestamp issue so quickly - one this topic, I had this occurr again, and interestingly the GS bucket it seems to think it cant find is the one from the LAST job started, maybe indicating in some way isolaton isnt happening?  Be interesting to see if you can reproduce yourself.
Cheers

I couldnt reproduce it but I am curious to see if it is only for kinesis streams or can be caused by any other source
I suggest streaming from local storage or S3 to rule out it being an issue from the kinesis side
Or stream into an S3 directory and write to parquet.
Also as a last attempt, try running one query on Databricks and another locally on your machine to rule out any shared context on the databricks side

Hey,
Thanks for replying - we went live last night so there is a slew of activity and events Im processing.  But, I am seeing alot more of these errors today, and in BQ during ingest there are some 409s I can see (conflicts?) in the dashboard
Dbricks use Kinesis themselves extensively and they've GA the connector, Id doubt it is that, more like something I am doing.
I am not loosing any data, but Ive had to restart the cluster twice to 'reset' the driver applications as a few jobs 'got stuck'.  Spark seems to keep the integrity of the checkpointing so I get the data ( duplicates though) for which my query in BQ filters out.  But ive had to reset the cluster which is of course not fun.
Do you think it is worth moving my BQ streaming query code out of the shared library direct into the notebook?  Its the only thing I can think of at the moment Im doing wrong, somehow its not isolated, but each job has its own executor java processes so should be isolated.
Thanks for the tips Ill keep investigating, arh.....bad for my confidence! :(
K

If you are already in production then certainly its better to move into its
own notebook. The cluster will still be running so you should be fine
Manually restarting is def not fun so I will do anything I can to keep the
lights on with no manual intervention and try to isolate the issue. Also if
you convert the notebook to a job on Databricks, cant you set it to restart
when it fails? I seem tor recall you
Have a look at this if you havent already
https://docs.databricks.com/spark/latest/structured-streaming/production.html
On Tue, Jul 18, 2017 at 3:19 PM, kurtmaile<EMAIL_ADDRESS>wrote:

Hey,
Thanks for replying - we went live last night so there is a slew of
activity and events Im processing. But, I am seeing alot more of these
errors today, and in BQ during ingest there are some 409s I can see
(conflicts?) in the dashboard
Dbricks use Kinesis themselves extensively and they've GA the connector,
Id doubt it is that, more like something I am doing.
I am not loosing any data, but Ive had to restart the cluster twice to
'reset' the driver applications as a few jobs 'got stuck'. Spark seems to
keep the integrity of the checkpointing so I get the data ( duplicates
though) for which my query in BQ filters out. But ive had to reset the
cluster which is of course not fun.
Do you think it is worth moving my BQ streaming query code out of the
shared library direct into the notebook? Its the only thing I can think of
at the moment Im doing wrong, somehow its not isolated, but each job has
its own executor java processes so should be isolated.
Thanks for the tips Ill keep investigating, arh.....bad for my confidence!
:(
K
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-316078858,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm8AYtjJ3F69rniZVaq2LpvnkKJ7Gks5sPL8PgaJpZM4ODLy1
.

Yes the notebook is running as a job in dbricks and running with retries (thanks databricks!) so a good amount of time db restarts the job automatically and it picks up again....but seems to be failing at the moment every 20 mins with its confused gs bucket problem...but then works on restart.  So yes its still a notebook, but its running as a streaming job, form which that job uses a library I create and upload via the libraries API.....the notebooks are thin top level orchestration.  But thinking I could move the BQ specific code I had in there up to the notebook level.  i tried to wrap it in some convenience method a simple 2 liner to create a streaming sink...
Thanks for taking the time to reply!
Rebooting the cluster has only happened today, because even though the retry worked, there was no data flowing through so it got stuck somehow....

....the shared library is where I set the spark.sqlContext.xxxx for all the variables for BQ SS sink, including the gs bucket, but the spark session variable should still be isolated, thats the confusing part....

Hmm it is very difficult to debug without pairing on it. It is most likely
a bug in the connector if the shared library isnt setting anything with
regards to the bucket.
Are you not able to set any BQ specific variables outside the shared
library? It might duplicate your code but atleast it will decouple it and
help narrow down the problem
On Tue, Jul 18, 2017 at 3:33 PM, kurtmaile<EMAIL_ADDRESS>wrote:

....the shared library is where I set the spark.sqlContext.xxxx for all
the variables for BQ SS sink, including the gs bucket, but the spark
session variable should still be isolated, thats the confusing part....
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-316083366,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm-lE_fGH-FTH0aS6YQf8X7Zx4BiRks5sPMJRgaJpZM4ODLy1
.

yea no problems having someone as a sounding board (even if not pairing) is nice so thank you.  I need to be able to sort my own sh!t out, so will keep trying.  My plan tonight is to move the BQ stuff up into the top level notebook, see how it goes - feels like the best next step.  Thanks a lot mate youre a star appreciate your support.

Hi Sam,
Unfortunately this hasnt worked, I pulled all the BQ code up into the top level notebooks so it isnt in the shared library at all, and still get this wrong buckets issue.  Are you running multiple SS jobs in databricks in prod at your company?  At a loss as to why I am seeing this, my jobs are so simple...
I thought about trying to set this cluster spark env var (spark.driver.allowMultipleContexts to true, its false in dbricks, but thats the sparkContext anyay so not sure it would make a difference), its the SQL context where these things get set.
Kurt
Ive logged a support ticket with databricks, but they arent that helpful as they have not 'certified' your library.

Hi Sam,
Just a thought, is it in theory possible to move the storgage bucket configuraton out of the SQL context, maybe define it as an option on the DataStreamWriter much like the other options?
e.g
val orderEventsStreamingQuery =
  orderEventDeltaStreamingOutputDF
    .writeStream
    .format("com.samelamin.spark.bigquery")
    .option("checkpointLocation", bqStreamingOrderEventQueryCheckpointS3Path)
    .option("tableReferenceSink", s"$bqProjectId:$bqSDVDataDataset.$bqOrderEventsTable")
    .option("stagingBucket", "xxxxxxxxxx") <- NEW
    .trigger(Trigger.ProcessingTime("2 minutes"))
    .start()

Would mean I could have more then 1 sink too per job, at the moment its just 1 sink per job, some of the jobs use the same source.  Just a thought, racking my brain thinking of options.

Hi @kurtmaile  Thats an idea worth trying, ill push a separate branch and give it a go
In the meantime can you stick in a stack trace. It might give me more information

cool that would be super thanks - will do

Hey @kurtmaile please try running your notebooks using the above branch and let me know. If this doesnt fix it then Ill add an override to the gcs path
I think the problem is we are storing the gcs bucket in the hadoop config and that gets overwritten
I am assuming in the screenshots you attached earlier, the first job was "customeraccountsevents" then it got overwritten in another job by "inventoryrefdata"
is that correct?

Hey yes that is correct - awesome will do that and report back! Legend
thanks so much.
On Fri, 21 Jul 2017 at 11:02, Sam Elamin<EMAIL_ADDRESS>wrote:

Hey @kurtmaile https://github.com/kurtmaile please try running your
notebooks using the above branch and let me know. If this doesnt fix it
then Ill add an override to the gcs path
I think the problem is we are storing the gcs bucket in the hadoop config
and that gets overwritten
I am assuming in the screenshots you attached earlier, the first job was
"customeraccountsevents" then it got overwritten in another job by
"inventoryrefdata"
is that correct?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/26#issuecomment-316961378,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATIvhWYzWlWNOtBM_hcOCbXQbos8s4rdks5sQHdMgaJpZM4ODLy1
.

Morning Sam,
Thanks for trying this fix branch.
I tried this in UAT this morning and got the following error:

Hey @kurtmaile try the latest branch, I can see the tables being created. But because I cant replicate your error it might be a false positive
If that is the case we will have to pass in an override every time we write

Also @kurtmaile I noticed that you are instantiating a new BigQuerySink per job. This might seem silly but its best to rule it out,  are you sure you are instantiating a new sparkSession? because the default parameter to your bigquerysink is to "SparkSession.builder().getOrCreate()" which might be the reason you are sharing contexts

Morning Sam - thanks for this.
I had mentioned that I moved the whole code from that library up into the notebook, so I dont use that bigQuerySink class of mine and the top level notebook has the code now.
Should I add the lines above then into the notebook?  Ill send you my notebook via email anyway.
Thanks

Hey @kurtmaile I set up the gcs bucket to only be written once, have a go but this should stop any overwriting happening

Hi Sam have just upgraded to the latest on the gcs_fix branch - will let it run and see what happens, hopefully (fingers crossed) fixes this pesky gcs bucket issue once and for all!  Cheers for trying multiple options and helping to try resolve.

so far so good :)

Added an override for the gcs bucket, you need to set it via the option
.option("gcsBucket", bucket)
.option("transaction_log", path)

Fixed on master 0.2.0


bug(google-maps): missing setZoom()

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
the setZoom() method is missing in the new angular google-maps component.
Reproduction
use "@angular/google-maps": "14.0.4",
Expected Behavior
being able to change the zoom of the map as follow
  @ViewChild('googleMap', { static: true }) googleMap: GoogleMap

  setZoom(zoom: number) {
     this.googleMap.setZoom(zoom)
  }

<google-map
  #googleMap
  width="100%"
  height="100%"
  [options]="options"
>
</google-map>

Actual Behavior
Property 'setZoom' does not exist on type 'GoogleMap'. Did you mean 'getZoom'?
Environment

Angular: v14.0.6
CDK/Material: 14.0.4
Browser(s): Chrome
Operating System (e.g. Windows, macOS, Ubuntu): Windows 10

Seems like we are using this function already so this seems like it won't be difficult to add: https://github.com/angular/components/blob/main/src/google-maps/google-map/google-map.ts#L290

Hey @amysorto
I would like to work on this issue if no one else is already using it.
Thanks!

There's already a zoom input that allows changing the zoom: https://github.com/angular/components/blob/main/src/google-maps/google-map/google-map.ts#L95
It seems like exposing setZoom would be duplicative. If we make this change, there are other "set" functions we would need expose too, for example setCenter.

@kgajera correct me if I'm wrong,
But the default Google map component have the zoom exposed as input and the setZoom() as function.
So this isn't really a duplicate
Additionally, since this library doesn't have any documentation other than "look what the original Google nap component does" I would expect it to do the same and not having to look myself into the code to see the reason why this isn't working (what I actually have to do..)
Or we could add a documentation that explain the difference, but I'm quite sure this make less sense .
As people mentioned they are ready to implement themselves and create a merge request, just need an approval here :)

@kgajera @amysorto @amysorto Wait a bit and still no real answer.
Since this is a really straight forward I've implemented it and made a pull-request
Hope this will help that feature get implemented

Closing since while it's confusing, it's currently working as expected. The zoom level is meant to be set through the zoom input. We'll look into making this less confusing in the future.

@crisbeto Sorry to be annoying about that matter, but why is my pull request not accepted?
This is just a nice new feature, it remove the confusion and make it more like the original google API.
So I'm a bit lost on the why this did get rejected.

There's already a way to set the zoom by assigning it like GoogleMap.zoom = <your zoom>.


image_conv2d_version

Is there a conv2d version that aims to reconstruct images like the experiment on MNIST mentioned in the paper?

Sorry. I do not have time to clarify and clean the other part of the code recently. Please drop me an email. I will try to find time to send you an uncleaned copy for your reference.

@alexzeng1206 Hi,Do you have the code of conv2d version?

Hello, can you send me the code of 2D convolution  or uncleaned copy for image detection, my email is<EMAIL_ADDRESS>thank you

Sorry. I do not have time to clarify and clean the other part of the code recently. Please drop me an email. I will try to find time to send you an uncleaned copy for your reference.

Hello, can you send me the code of 2D convolution or uncleaned copy for image detection, my email is<EMAIL_ADDRESS>thanks.


Only processes 64k-ish of long sequences??

Was trying this out, and on some shorter sequences, all good.  But when I try any longer (>100k) sequences, it stops reporting any hits at about what looks like 64k in.
Bug, or... ???
Thanks for any assistance you can provide.
Tom

Is it for any 100K+ sequence, or do you have a specific test case?

I have run several - a couple of examples:NC_001666.2 Zea mays chloroplast  (https://www.ncbi.nlm.nih.gov/nuccore/11994090?report=fasta)
KM103369.1   Oryza sativa cultivar TN1 chloroplast (https://www.ncbi.nlm.nih.gov/nuccore/743432831?report=fasta)
both are ~140k bp
running as :

qgrs -i zm_chloro_NC_001666.2.fa -o zm_chloro.out

last hits are:
...
76          63112  63121  63129  63135  2  19  CCATTTTATGGGCGAAAGGCGGGGG
77          64493  64501  64509  64521  2  18  TTTAATTTGGAATAGCGGCCAAGATCTTGG
78          64932  64937  64940  64946  2  19  CCCTCGGGGGTCGTGG
2) qgrs -i oryza_chloroplast_KM103369.1.fa -o oryza_chloro.out
last hits are:
...
110         64640  64642  64646  64649  2  19  AAGGAAGGAGG
111         65221  65229  65236  65243  2  20  GGAAAGGAGGTTCATGGCCAAGGG
112         65299  65308  65314  65319  2  18  GGCCAATGAGGAGTCGGCAGGG
From other G4 tools, there are certainly hits through toward the 3' end.
Additionally, if I try to run with larger sequences (e.g. Zea mays mitochondrials (NA/NB/CMS-S/... => 500k nt to 700k nt) it takes so much memory the process is killed.
It would be good to be able to operate on chromosomal size pieces, but it seems that isn't currently possible.
Regards,
Tom
  From: Scott Frees<EMAIL_ADDRESS>
To: freezer333/qgrs-cpp<EMAIL_ADDRESS>Cc: kushtun<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Sent: Thursday, February 9, 2017 9:59 AM
Subject: Re: [freezer333/qgrs-cpp] Only processes 64k-ish of long sequences?? (#1)
Is it for any 100K+ sequence, or do you have a specific test case?—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi Tom - OK, understood.  I will let you know as soon as I get a chance to update the code - it sounds like it's running out of memory for longer sequences, I'll need to reorganize some of the code to handle those situations better.

Hi Tom - very sorry it took so long for me to get back to this.  I make a small adjustment to the code and it can handle over 64K sequences now.  The larger mitochondrials you mentioned still consume too much memory though.  The algorithm we use deals with overlapping G4 differently than others, and it does end up being pretty memory intensive.  Hopefully I can get some time soon to redesign parts of it to allow for these larger sequences to be run.


Investigate merging EditorPane and EditorInput registration into one

Continuation of https://github.com/microsoft/vscode/issues/126603.
We have a way to register an editor pane and a way to register an editor input. It might be clever to have one registration which pipes through the other one to the proper service. I'm not sure if Panes 1:1 to inputs though.

Typical editor pane registration code is simply mapping a typed editor input class onto the pane (1 to many):
Registry.as<IEditorPaneRegistry>(EditorExtensions.EditorPane)
.registerEditorPane(
	EditorPaneDescriptor.create(
		WebviewEditor,
		WebviewEditor.ID,
		'Webview Editor',
	), [
	new SyncDescriptor(CustomEditorInput)
]);

When I register an editor, maybe I can also register the associated pane for the editors.
The only worry I have is that the code to register the pane today executes when code is loading because it is static and would load at a later point if used via service...

Given issue grooming, I tend to think this issue is out of scope. At least I am not planning to work on this anytime soon, leaving up to you if you want to look into it.

Yeah I agree. I think there's a lot more debt issues that we gain a larger value from and we could always revisit this if we find it limiting ourselves.


Update ubuntu.md

Proposed changes

Unreleased project version (optional)

Related issues (optional)

I prefer the longer form, and experts will already know they can use the shorter form. Thanks, though!

The id used is also still wrong.
Supposed to be 6D085F96

@mstanleyjones ^^

Please double-check again, I am pretty sure it's correct: https://docs.docker.com/engine/installation/linux/docker-ee/ubuntu/#install-using-the-repository
Note that the key for Docker EE is different from the one for Docker CE. It is likely that you actually have both keys installed, if you started out with Docker CE. You can use apt-key list to show all keys.


Problem with nested class

I'm working with Rails 5. My policy looks like this:
class ModelPolicy < ApplicationPolicy
  class Scope < Scope
    def resolve
      scope
    end
  end

  def index?
    ...
  end
end

And I'm getting NameError: uninitialized constant ModelPolicy::Scope

Scope should also be defined in ApplicationPolicy so you can inherit from it.
My ApplicationPolicy looks like this: (this was generated by Pundit via rails g pundit:install)
class ApplicationPolicy
  # all the other methods
  class Scope
    attr_reader :user, :scope

    def initialize(user, scope)
      @user = user
      @scope = scope
    end

    def resolve
      scope
    end
  end

You don't have to extend Scope for your custom Scope.
class ModelPolicy < ApplicationPolicy
  class Scope
  end
end

Should work.

I believe this is solved?


Add workflow to publish on the marketplace (with debug)

This PR adds a workflow allowing maintainers to push  new release directly on the marketplace as soon as the release has been created on GitHub

To be reopened when branch 2.0 has been merged into dev


Mental Model guide

TODO:

[x] Throw illustrations to design team to make them better.

Seems the preview does not reflect the current state of the PR :/

Messed up the git history. Closing this in favor of https://github.com/prisma/docs/pull/4213


customObservabilityBackend for logs

What feature do you want to see added?
Currently there is customObservabilityBackend for Traces and Metrics.  Please consider adding one for Logs.
    observabilityBackends:
    - customObservabilityBackend:
        metricsVisualizationUrlTemplate: "foo"
        traceVisualisationUrlTemplate: "http://example.com"
        name: "Custom Observability"

Upstream changes
No response

This would be a great enhancement. Out of curiosity, what would be the backend please?

We planned to use Grafana Loki.

I have started working on a Loki integration. I'll open a draft PR soon to make the progess visible and welcome contributions.

Please see:

https://github.com/jenkinsci/opentelemetry-plugin/pull/727

@chewrocca can you please test Loki storage added by https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.18.0-beta1

Grafana Loki support fixed in https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.18.0-beta1
We will reopen the topic of a generic CustomObservabilityBackend when the new use case comes so we can see if existing capabilities are generic enough.

@cyrille-leclerc These changes work great.  Thank you!
Should I open a PR to add the configuration examples for the Grafana backend?
    observabilityBackends:
    - grafana:
        grafanaBaseUrl: "https://example.grafana.net"
        tempoDataSourceIdentifier: "grafanacloud-traces"
        tempoQueryType: "traceql"
        grafanaOrgId: "1"
        name: "Grafana"
        grafanaLogsBackend:
          GrafanaLogsBackendWithoutJenkinsVisualization:
            grafanaLokiDatasourceIdentifier: "grafanacloud-logs"

What is the option for grafanaMetricsDashboard ?  I'm curious about what this configuration does.  I didn't enable it.
Is it possible to avoid confusion by ignoring the "View as plain text" under Console Output?  That link didn't do anything once the Grafana Logs were set up.

Thanks @chewrocca ,
Opening a PR would be great.

What is the option for grafanaMetricsDashboard ? I'm curious about what this configuration does. I didn't enable it.

It's the possibility to have a link from the Jenkins main navigation to a monitoring dashboard. We didn't have the time to create such a dashboard for grafana, contributions are welcome.
Produced metrics are documented here.
Here is what has been implemented a while ago for Kibana:

Is it possible to avoid confusion by ignoring the "View as plain text" under Console Output? That link didn't do anything once the Grafana Logs were set up.

Great catch, please file an enhancement request and contributions are more than welcome.


Testing & CI Refactor

Closes https://github.com/defenseunicorns/uds-package-software-factory/issues/105

/test upgrade

/test upgrade

/test upgrade


What does scripts batch Pack and Unpack do in Thorium?

Hello I would like to know what functioned performs these scripts batch in thorium operation or what can we do with them?.

https://github.com/Alex313031/thorium/tree/main/pak_src

In these files . pak all Thorium icons are stored?

@SifcaeIT Not all, but many. .png and .jpg assets are stored in it.


Fix helm lint

What this PR does / why we need it:
Fix helm lint which is currently failing with
[ERROR] /home/vpnachev/go/src/github.com/gardener/oidc-webhook-authenticator/charts/oidc-webhook-authenticator: chart metadata is missing these dependencies: application,runtime

Error: 1 chart(s) linted, 1 chart(s) failed

Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

I am OK to leave some local repository configurations in order to satisfy helm lint. Maybe we can add a lint step here so we take actual advantage of this change.

/close
in favor of #51


add new endpoints and refactor dependencies

Codecov Report

Merging #32 into master will increase coverage by 0.63%.
The diff coverage is 93.57%.

@@            Coverage Diff             @@
##           master      #32      +/-   ##
==========================================
+ Coverage   93.13%   93.76%   +0.63%
==========================================
  Files          20       20
  Lines         568      738     +170
==========================================
+ Hits          529      692     +163
- Misses         39       46       +7

Impacted Files
Coverage Δ

titiler/api/endpoints/stac.py
88.48% <86.90%> (-1.60%)
:arrow_down:

titiler/api/endpoints/cog.py
96.11% <96.25%> (+3.49%)
:arrow_up:

titiler/api/deps.py
98.88% <100.00%> (-1.12%)
:arrow_down:

titiler/ressources/responses.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 99db80a...7f38da9. Read the comment docs.


Write Readme.md

Description of Ad Network, project installation, summary of each models etc.. write them

Include reasons why I used these language and frameworks


built-in role-based access control

Hi,
The first note in the link  here : https://docs.microsoft.com/en-us/azure/postgresql/howto-hyperscale-create-users points to general RBAC documentation(So that means built in role is supported for HS(citus)? )
In security baseline documentation for HS:
https://docs.microsoft.com/en-us/azure/postgresql/postgresql-hyperscale-security-baseline#identity-and-access-control
"Hyperscale (Citus) does not support built-in role-based access control, but you can create custom roles based on specific resource provider operations."
Can I suggest the correction in the note or pointing to correct documentation?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6b6acc5d-debc-df76-4133-e62eb8e94b0b
Version Independent ID: f1de4bb8-a64b-f6a8-2dca-c1cdd8037b5a
Content: Create users - Hyperscale (Citus) - Azure Database for PostgreSQL
Content Source: articles/postgresql/howto-hyperscale-create-users.md
Service: postgresql
Sub-service: hyperscale-citus
GitHub Login: @jonels-msft
Microsoft Alias: jonels

@AbhishekShaha We are checking this and will get back to you.

@AbhishekShaha The note is as stated below:
The term "users" refers to users within a Hyperscale (Citus) server group. To learn instead about Azure subscription users and their privileges, visit the Azure role-based access control (Azure RBAC) article or review how to customize roles.
It states the users mentioned in document are only valid for Hyperscale Citus server group and doesn't correspond to RBAC roles Azure subscription can be used to create. Both the note and the second document also gives the details on creating custom roles which both mentions are applicable to Hyperscale citus.
If you still have a question, please provide more details to take this further.

@AbhishekShaha Please convey if you need any further help regarding this.

@AbhishekShaha At this time we are going ahead and closing this thread. If you have any other questions or concern please check back to us.

Hi @NavtejSaini-MSFT sorry for late response, all good for now. Thank You.


please update logzio-nodejs dependency

https://github.com/logzio/logzio-nodejs/issues/45

@adrai
Thanks for the reminder:)
I'll try to do it the latest at the beginning of next week


[ng2] Uncaught (in promise): TransitionRejection(type: 5, message: The transition was ignored., detail: "undefined")

A bug easy to reproduce with the ng2 ui router, just start the quickstart example (https://github.com/ui-router/quickstart-ng2), click on one of the state links (Foo for example), then reclick on Foo.
You get the following error in the console:
EXCEPTION: Error: Uncaught (in promise): TransitionRejection(type: 5, message: The transition was ignored., detail: "undefined")
browser_adapter.js:76 EXCEPTION: Error: Uncaught (in promise): TransitionRejection(type: 5, message: The transition was ignored., detail: "undefined")BrowserDomAdapter.logError @ browser_adapter.js:76BrowserDomAdapter.logGroup @ browser_adapter.js:86ExceptionHandler.call @ exception_handler.js:56(anonymous function) @ application_ref.js:193schedulerFn @ async.js:122SafeSubscriber.__tryOrUnsub @ Subscriber.js:166SafeSubscriber.next @ Subscriber.js:115Subscriber._next @ Subscriber.js:74Subscriber.next @ Subscriber.js:51Subject._finalNext @ Subject.js:124Subject._next @ Subject.js:116Subject.next @ Subject.js:73EventEmitter.emit @ async.js:111NgZone._zoneImpl.ng_zone_impl_1.NgZoneImpl.onError @ ng_zone.js:119NgZoneImpl.inner.inner.fork.onHandleError @ ng_zone_impl.js:65ZoneDelegate.handleError @ zone.js:310Zone.runGuarded @ zone.js:218drainMicroTaskQueue @ zone.js:469ZoneTask.invoke @ zone.js:408
browser_adapter.js:76 STACKTRACE:BrowserDomAdapter.logError @ browser_adapter.js:76ExceptionHandler.call @ exception_handler.js:58(anonymous function) @ application_ref.js:193schedulerFn @ async.js:122SafeSubscriber.__tryOrUnsub @ Subscriber.js:166SafeSubscriber.next @ Subscriber.js:115Subscriber._next @ Subscriber.js:74Subscriber.next @ Subscriber.js:51Subject._finalNext @ Subject.js:124Subject._next @ Subject.js:116Subject.next @ Subject.js:73EventEmitter.emit @ async.js:111NgZone._zoneImpl.ng_zone_impl_1.NgZoneImpl.onError @ ng_zone.js:119NgZoneImpl.inner.inner.fork.onHandleError @ ng_zone_impl.js:65ZoneDelegate.handleError @ zone.js:310Zone.runGuarded @ zone.js:218drainMicroTaskQueue @ zone.js:469ZoneTask.invoke @ zone.js:408
browser_adapter.js:76 Error: Uncaught (in promise): TransitionRejection(type: 5, message: The transition was ignored., detail: "undefined")
    at resolvePromise (zone.js:517)
    at zone.js:494
    at justjs.js:17
    at new ZoneAwarePromise (zone.js:563)
    at Function.coreservices_1.services.$q.reject (justjs.js:17)
    at RejectFactory.ignored (rejectFactory.js:51)
    at Transition.run (transition.js:285)
    at TransitionManager.runTransition (transitionManager.js:50)
    at StateService.transitionTo (stateService.js:281)
    at StateService.go (stateService.js:210)
    at UiSref.go (uiSref.js:113)
    at AbstractChangeDetector.ChangeDetector_App_0.handleEventInternal (viewFactory_App:242)
    at AbstractChangeDetector.handleEvent (abstract_change_detector.js:57)
    at AppView.triggerEventHandlers (view.js:221)
    at eval (viewFactory_App:349)
    at dom_renderer.js:282
    at dom_events.js:28
    at ZoneDelegate.invoke (zone.js:306)
    at Object.NgZoneImpl.inner.inner.fork.onInvoke (ng_zone_impl.js:44)
    at ZoneDelegate.invoke (zone.js:305)
    at Zone.runGuarded (zone.js:215)
    at NgZoneImpl.runInner (ng_zone_impl.js:75)
    at NgZone.run (ng_zone.js:220)
    at HTMLAnchorElement.outsideHandler (dom_events.js:28)
    at ZoneDelegate.invokeTask (zone.js:339)
    at Zone.runTask (zone.js:237)
    at HTMLAnchorElement.ZoneTask.invoke (zone.js:405)BrowserDomAdapter.logError @ browser_adapter.js:76ExceptionHandler.call @ exception_handler.js:59(anonymous function) @ application_ref.js:193schedulerFn @ async.js:122SafeSubscriber.__tryOrUnsub @ Subscriber.js:166SafeSubscriber.next @ Subscriber.js:115Subscriber._next @ Subscriber.js:74Subscriber.next @ Subscriber.js:51Subject._finalNext @ Subject.js:124Subject._next @ Subject.js:116Subject.next @ Subject.js:73EventEmitter.emit @ async.js:111NgZone._zoneImpl.ng_zone_impl_1.NgZoneImpl.onError @ ng_zone.js:119NgZoneImpl.inner.inner.fork.onHandleError @ ng_zone_impl.js:65ZoneDelegate.handleError @ zone.js:310Zone.runGuarded @ zone.js:218drainMicroTaskQueue @ zone.js:469ZoneTask.invoke @ zone.js:408
zone.js:443 Unhandled Promise rejection: TransitionRejection {type: 5, message: "The transition was ignored.", detail: undefined} ; Zone: angular ; Task: HTMLAnchorElement.addEventListener:click ; Value: TransitionRejection {type: 5, message: "The transition was ignored.", detail: undefined}consoleError @ zone.js:443drainMicroTaskQueue @ zone.js:472ZoneTask.invoke @ zone.js:408
zone.js:445 Error: Uncaught (in promise): TransitionRejection(type: 5, message: The transition was ignored., detail: "undefined")(…)

Can you please fix the same problem for ng1?

@adharris reports that this is still an issue in alpha.5 when a TargetState is returned from an onEnter hook.

Can you please fix the same problem for ng1?

@pdanpdan is there an issue for this for ng1? I'm not aware of the problem existing? Can you show me?

Can't reproduce the report by @adharris that TargetState returned from onEnter causes an Uncaught in promise.  Resetting milestone to alpha.5 and closing so this doesn't block the beta release.

Able to reproduce now using @adharris example: https://github.com/adharris/quickstart-ng2/tree/4e5a06cf8ce5c596ae0d15af7359ab4b8ad67d36
Will fix in beta.1

this also happens in the redirectTo hook on the state declaration.

This should be fixed. Are you using the latest version? Can you explain exactly how to reproduce it?

so, responding with anything inside the redirectTo on a state definition results in the transition superseded error message. it may be related to a different issue
https://github.com/angular-ui/ui-router/issues/2977
      views: {
        '': {
          templateUrl: 'views/main/main.html',
          controller: 'MainCtrl',
          controllerAs: 'main'
        }
      },
      redirectTo: function(trans){
// this causes an error message in the console
        return trans.getResolveValue('startRoute');
//so does
       return trans.router.stateService.target(trans.getResolveValue('startRoute'), newParams, stateOptions);
      }

Are you sure you're on the latest version (1.0.0-beta.3)?  You shouldn't get a superseded error when a redirect happens.  Is the redirect functioning properly, but you get the superseded message?

That's correct. It happens successfully but still throws an error. We
should be on latest I will have to check in the morning
On Wed, Dec 14, 2016 at 5:52 PM Chris Thielen<EMAIL_ADDRESS>wrote:

Are you sure you're on the latest version (1.0.0-beta.3)? You shouldn't
get a superseded error when a redirect happens. Is the redirect functioning
properly, but you get the superseded message?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/angular-ui/ui-router/issues/2676#issuecomment-267203915,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABp-H0vyD41dNCzcVhhGWzqP5jQPWxJqks5rII9CgaJpZM4IA4Qe
.

@timothyswt if you verify you're on the latest, can you open a new issue for this?  I can't reproduce it.

@christopherthielen yes, I can reproduce it on Beta 3

After updating to ui-router 1.0.0-rc.1 (from beta3) and to angular 1.6 (from 1.5.8), I'm now getting the "Possibly unhandled rejection" error, with "The transition has been superceded by a different transition".  Looking at the $trace output, it appears this is a result of redirectTo on my state (the initial transition is rejected and shows the "superceded" message in the trace, then successfully transitions to the state in redirectTo).
From the comments, I gather this would have been fixed?  Am I doing something wrong?

I think beta.3 and above shouldn't have this problem.
Do you happen to use the ui router visualizer too?

No

I just installed it.  The chevrons show [success] -> [redirected] -> [success], while the (massive!) error gets dumped to the console...

I was asking because I think the visualizer itself can cause this error to be logged. Thanks for following up.

I am also seeing:
angular.js:14328 Possibly unhandled rejection: {"type":2,"message":"The transition has been superseded by a different transition","detail":{"deferred":{"promise":{"$$state":
With code similar to the below:
var onBeforeStateHook = function(transition) {
    if (!user.isLoggedIn()) {
        if (transition.to().name !== 'login') {
            var $state = transition.router.stateService;
            return $state.target('login');
        } else {
            return true;
        }
    } else {
        return true;
    }
}

$transitions.onBefore({}, onBeforeStateHook);

Stack trace for unhandled rejected promise:
app.js:111111:33)\n    at new Promise (
angular.js:16605:18)\n    at new Deferred (
angular.js:16596:34)\n    at Function.defer (
angular.js:16592:12)\n    at new Transition (
angular-ui-router.js:6081:38)\n    at TransitionService.create (
angular-ui-router.js:4337:16)\n    at Transition.redirect (
angular-ui-router.js:6459:59)\n    at
angular-ui-router.js:5602:47\n    at processQueue (
angular.js:16648:37)\n    at
angular.js:16692:27","value":{"views":....

I believe this is fixed by https://github.com/ui-router/core/commit/a7464bb28466d794bc0406fec0e32f8256ced58b

@christopherthielen  Is it possible to use 1.0.0-rc1 with ui-router/core that has the fix or do we have to wait for a new release?
For some reason, I am suddenly getting the following error after I added a new route. The new route I added has a different state name and different URL but it is being "ignored" due to "All states and parameters in the To and From paths are identical"  Any idea?

  I'm closing this ticket because it's a confusion of angular 1 angular 2.  If there is still a problem, let's open a different ticket.


Testing Objects for Properties - Javascript - FCC

I just solved this myself. I am not sure, how to delete it. Sorry for the inconvinence.
Challenge Testing Objects for Properties has an issue.
User Agent is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

// Setup
var myObj = {
  gift: "pony",
  pet: "kitten",
  bed: "sleigh"
};

function checkObj(checkProp) {
  // Your Code Here
  result = myObj.hasOwnProperty(checkProp);
if (result === true)
  return myObj[checkProp];
else
  return "Not found";
}

// Test your code by modifying these values
checkObj("house");

@shauryakapoor No worries, someone will come along and remove the message shortly.
When you encounter bugs, try to bring it up in the Freecodecamp Chatrooms first and discuss it with other coders, most often than not, its a difficult challenge or project to problem solve.
Happy coding! 🎉

Ok
On Jun 17, 2017 8:16 PM, "anthonygallina1"<EMAIL_ADDRESS>wrote:

Closed #15421 https://github.com/freeCodeCamp/freeCodeCamp/issues/15421.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/freeCodeCamp/freeCodeCamp/issues/15421#event-1128005426,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASk9SOACyuMqwWxzt6sbRy23M-Ylk4wcks5sFHqAgaJpZM4N9WyN
.


Make Dolphin independent from Git tooling?

When getting the Dolphin source through without Git tools (e.g. downloading the .zip),
several files are now replaced with their Git LFS references, including DBOOT..
So it is not possible anymore to build a functioning Dolphin this way.
.
Also, building the VM depends on this Git command in Dolphin.targets to build a version header file:
"git describe --tag --always --long > "$(IntDir)version.txt"
Could the Dolphin version be stored in a source (text) file and let Git use that?
.
Besides Git independence, an additional advantage is that local playing-around copies of Dolphin to not have to drag around the sizable hidden, .git folder anymore.

I'm afraid I don't see why being independent of Git is net positive.
Git is designed for disconnected operation, and having done a clone, that's what you have subsequently. Why would you bother downloading a zip? It's not like Dolphin is a "library" you'd want to include in other builds, its an IDE.  Having an accurate git version in the build information is a benefit I wouldn't want to give up, and generating it on build is a superior solution to checking in a file with each change, or smudging one on checkout (which requires git anyway). If anything I'd like to see more git integration not less.
The Dolphin repo is not large, and the .git folder currently occupies ~120Mb on disk in a fresh clone.  The reason for adopting LFS was to stop that getting rapidly bigger due to large binary files that need to be updated fairly regularly. The Scintilla DLL is a case in point, being a 1Mb file. Earlier on this was downloaded at build time, but that causes a problem for people who want to build with no external dependencies except their git server.

Yes, for maintainers, the git integration is indeed handy. It's a choice.
It might be worth putting a note in the readme that the ZIP file does not contain buildable files,
and that Git tooling should be used to get the source.
This to guide casual visitors that are "browsing" Dolphin.
I do that myself sometimes with other repo's...

Well the readme describes the steps required to build everything already in some detail, including the need to pull stuff from LFS (although a fresh clone with later git clients seems to do that anyway). The steps are relatively involved, and you can't just "build" and be done - not unusual for more complex repos. For casual examination, the pre-built installers are much the quickest and easiest way.
If it were possible to avoid including those source archives in the releases I would, but GitHub insists on including them as far as I can tell.

 It can actually be any location but for convenience we'll call it \Dolphin. Use a Git client tool to clone, downloading the ZIP file won't work due use of Git LFS.

Please feel free to submit a PR for the readme if you wish to change it.

Well, still good change, but a PR from a forked repo for one line of text seems a bit too much overhead.
Could you update it with your next code commit, in the wording of your choice?

There is no more overhead in submitting a PR from a forked repo than directly to the repo. In any case if it is something you care about (I'd assumed it was, since you raised the issue, and suggested the change), then you should make it. It is not something that I think important. If you don't care about it enough to make the change, then the issue may as well be closed as I won't be doing it.

All right then. I'll do a PR with with the wording suggested above.

Thank you, merged.


New hackage release?

Hi,
I've written quite a lot of code that depends on the Identity and Const instances that were added in #113. I'm getting a bit tired of having to compile hashable (and everything that depends on it) manually every time to take advantage of these though.
Any chance you bump the version and upload this to Hackage?
Thanks.

<IP_ADDRESS> released.


Feature/add templating for naked yaml files in eks module

Description of changes: Add templating for naked yaml files in eks module, to have an ability to deploy eks module in isolated networks
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Primary test lead me to the below error:
Traceback (most recent call last):     File “/codebuild/output/src132313204/src/bundle/module/app.py”, line 60, in <module>     stack = Eks(     File “/root/.venv/lib/python3.10/site-packages/jsii/_runtime.py”, line 111, in __call__     inst = super().__call__(*args, **kwargs)     File “/codebuild/output/src132313204/src/bundle/module/stack.py”, line 929, in __init__     cwagentconfig_content = t.substitute(image=image)     File “/root/.pyenv/versions/3.10.10/lib/python3.10/string.py”, line 121, in substitute     return self.pattern.sub(convert, self.template)    TypeError: expected string or bytes-like object

LGTM. Tested the replication of secrets-store and it worked fine.


chg: preserve original text in GoodDataRestException

Coverage increased (+0.15%) to 79.77% when pulling ec9a206a9b214bead6eaced1ee4bd721ad43e5d5 on mcl-ex into 6227fe22ea2714adbde37d95634094837903569c on master.


Sync with json from github

Is your feature request related to a problem? Please describe.
When distributing design tokens via github, it is a potential problem if users have to download a json from a repo to update it in figma.
Describe the solution you'd like
It would be great if the plugin could sync with a json file (similar to jsonBin) from github.
E.g. just point it at a github url to the raw file, add access credentials and allow it to sync this way. (Ideally it would even grab the version from a package.json if it exists)
Describe alternatives you've considered
Using JsonBin is not an option for every org. :( It also means another source of truth that is not the same as the one the dev team uses.

Hey @lukasoppermann - thanks for the feedback, I really appreciate it! ❤
I'm pretty far along with a github sync and am currently running the beta with some teams. If you'd like to betatest that feature I've written a short guide.
The feature is working so far, I'm mainly looking for feedback regarding if that solution could work or if there's anything broken in edge-cases.
I've also setup a test repository that transforms tokens stored in Figma Token to something SD can work with (no math). That repo contains a lot of code that's not needed for demonstrative purposes but I've set it up so there's a live updating token reference page at https://tokens-example.vercel.app/ - the interesting part is the github action that uses both token-transformer (strip tokens of math, aliases and combine themes) and style-dictionary

I got the Beta version running and syncing the json to Github. Though the part after that I can't seem to get working :)
Figma Tokens .json -> Token-Transformer -> Style-dictionary
(I blame the lack of Git knowledge on my part :D

Thank you, I'll try this out!


Symbol's value as variable is void: org-mpvi-seek-map

my config
(use-package org-mpvi
  :straight
  (org-mpvi :type git :host github :repo "lorniu/org-mpvi" :files (:defaults "*"))
  :after org
  )

loading error msg:
Symbol's value as variable is void: org-mpvi-seek-map

Now, it's ok.


bazel can't build keras 2.9 on MacBook Air M1

System information.

Have I written custom code (as opposed to using a stock example script provided in Keras): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 12.3.1 (M1 ARM64 CPU)
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 2.9
Python version:  3.9.12
Bazel version (if compiling from source): 5.0.0
GPU model and memory: no GPU used, memory: 16GB
Exact command to reproduce: bazel build -c opt //keras/tools/pip_package:build_pip_package

Describe the problem.
git clone https://github.com/keras-team/keras keras-2.9.0_pip_package
cd keras-2.9.0_pip_package
git checkout r2.9
bazel build -c opt //keras/tools/pip_package:build_pip_package
Target //keras/tools/pip_package:build_pip_package failed to build
Describe the current behavior.
bazel can't build keras 2.9 on MacBook Air M1
Describe the expected behavior.
bazel can build keras 2.9 on MacBook Air M1
Standalone code to reproduce the issue.
bazel build -c opt //keras/tools/pip_package:build_pip_package
Source code / logs.
INFO: Found 1 target...
ERROR: /Users/andyueng/samba/workspace_TensorFlow2/keras-2.9.0_pip_package/keras/api/BUILD:160:19: Executing genrule //keras/api:keras_python_api_gen failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)
Traceback (most recent call last):
File "/private/var/tmp/_bazel_andyueng/4b992bed8f1471a8dd4099a36cff54c6/execroot/org_keras/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/keras/api/create_keras_api_1_keras_python_api_gen.runfiles/org_keras/keras/api/create_python_api_wrapper.py", line 26, in
import keras  # pylint: disable=unused-import
File "/private/var/tmp/_bazel_andyueng/4b992bed8f1471a8dd4099a36cff54c6/execroot/org_keras/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/keras/api/create_keras_api_1_keras_python_api_gen.runfiles/org_keras/keras/init.py", line 21, in
from tensorflow.python import tf2
ImportError: cannot import name 'tf2' from 'tensorflow.python' (unknown location)
ERROR: /Users/andyueng/samba/workspace_TensorFlow2/keras-2.9.0_pip_package/keras/api/BUILD:187:19: Executing genrule //keras/api:keras_python_api_gen_compat_v2 failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)
Traceback (most recent call last):
File "/private/var/tmp/_bazel_andyueng/4b992bed8f1471a8dd4099a36cff54c6/execroot/org_keras/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/keras/api/create_keras_api_2_keras_python_api_gen_compat_v2.runfiles/org_keras/keras/api/create_python_api_wrapper.py", line 26, in
import keras  # pylint: disable=unused-import
File "/private/var/tmp/_bazel_andyueng/4b992bed8f1471a8dd4099a36cff54c6/execroot/org_keras/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/keras/api/create_keras_api_2_keras_python_api_gen_compat_v2.runfiles/org_keras/keras/init.py", line 21, in
from tensorflow.python import tf2
ImportError: cannot import name 'tf2' from 'tensorflow.python' (unknown location)
ERROR: /Users/andyueng/samba/workspace_TensorFlow2/keras-2.9.0_pip_package/keras/api/BUILD:173:19: Executing genrule //keras/api:keras_python_api_gen_compat_v1 failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)
Traceback (most recent call last):
File "/private/var/tmp/_bazel_andyueng/4b992bed8f1471a8dd4099a36cff54c6/execroot/org_keras/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/keras/api/create_keras_api_1_keras_python_api_gen_compat_v1.runfiles/org_keras/keras/api/create_python_api_wrapper.py", line 26, in
import keras  # pylint: disable=unused-import
File "/private/var/tmp/_bazel_andyueng/4b992bed8f1471a8dd4099a36cff54c6/execroot/org_keras/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/keras/api/create_keras_api_1_keras_python_api_gen_compat_v1.runfiles/org_keras/keras/init.py", line 21, in
from tensorflow.python import tf2
ImportError: cannot import name 'tf2' from 'tensorflow.python' (unknown location)
Target //keras/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /Users/andyueng/samba/workspace_TensorFlow2/keras-2.9.0_pip_package/keras/tools/pip_package/BUILD:39:10 Middleman _middlemen/keras_Stools_Spip_Upackage_Sbuild_Upip_Upackage-runfiles failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)
INFO: Elapsed time: 49.365s, Critical Path: 6.90s
INFO: 239 processes: 24 internal, 215 local.
FAILED: Build did NOT complete successfully
andyueng@jinkangdeMacBook-Air keras-2.9.0_pip_package % python3
Python 3.9.12 (main, May  8 2022, 17:57:49)
[Clang 13.1.6 (clang-13<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

from tensorflow.python import tf2
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'tf2' from 'tensorflow.python' (unknown location)

@snowuyl ,
Hello @snowuyl ,
For Apple M1 chip, you can install it through miniforge installers by following the guide here.
Since, few of the mentioned packages in the above link are community build, could you wait for sometime to get the 2.9 dependencies.

% pip3 install tensorflow-macos==2.9.0
Successfully installed tensorflow-macos-2.9.0
% python3
Python 3.9.12 (main, May 8 2022, 17:57:49)
[Clang 13.1.6 (clang-13<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
import tensorflow as tf
print(tf.version)
2.9.0
from tensorflow.keras import layers


IntelliJ Run-> Debug menu item sometimes grayed out for target,

I've seen this issue on both 2019.2 and 2019.3. For bazel command target, the debug menu item under run is grayed out. And, the port field in the configuration does not show in this case too.

have the same issue on 2021.2.3. did you ever figure this out?

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.


Some docs not showing up

From this file: https://github.com/ALLAN-DIP/baseline_bots/blob/main/docs/source/development.rst
The following doesnt show up on the website:
If using VSCode, run `unset PROMPT_COMMAND` in the terminal or add it to .bashrc
Otherwise, you may see a lot of `__vsc_prompt_cmd_original`

@trigaten I believe we just need to change the link in the sidebar. If you look at the vercel sites generated at the latest commit, everything shows up.
These are the links:
b-henna.vercel.app
baseline-bots-allan-dip.vercel.app
baseline-bots-git-main-allan-dip.vercel.app
It seems that the sidebar instead points to baseline-bots-git-updatedocs-allan-dip.vercel.app which has old docs but I do not have permission to change it.

@kkahadze I updated the link to https://baseline-bots-allan-dip.vercel.app/
Thanks for the investigative work :)


ECMPS Resources - Responsiveness

We want to "provide a usable view of ECMPS at all resolutions above 480 px" while ensuring that priority resolutions of 1024 px and above render correctly.
Responsiveness Wireframes: https://xd.adobe.com/view/6c2fe2a3-0bfc-44cf-b3c7-5b9a53cec701-397c/

Test Case: TC379

Test Passed


TrackedSet not propagating change notifications correctly for #each template helper

I have a TrackedSet and some actions to add and remove elements. The #each template helper shows the initial contents of the TrackedSet, but does not seem to reflect any changes. Oddly enough, #each-in works great (the "key" is the index).
It's not clear to me whether or not this is something that is supposed to work, but it'd be cooler if it did. Ember 3.15.0, tracked-built-ins 0.2.0.

This has been fixed in v1.0.0 😄

Fantastic! Thank you!


Preparation for daily Elasticsearch sync

Issue number: DH-1660
Description of change
This makes a few changes in preparation for the ES sync Celery task:

Moves ES index creation and the updating of the mapping to a dedicated management command.

This stops the initialisation being done unnecessarily (e.g. for manage.py migrate), and also stops the Elasticsearch connection being shared amongst Celery workers (due to how elasticsearch-dsl manages connections, and how Celery forks).

Changes sync_es functions to use a logger.

This is so that they can be used outside of the Django management command infrastructure.

Changes the Celery worker log format to include the logger name.

Sets the Elasticsearch logger log level to WARNING for Celery tasks (to cut down on noise).

Checklist

[x] Have any relevant search models been updated?
[x] Have any relevant fixtures (fixtures/test_data.yaml) been updated?
[x] Have any relevant select-/prefetch-related field lists in the views and search apps been updated?
[x] Has the admin site been updated (for new models, fields etc.)?
[x] Has the README been updated (if needed)?

Codecov Report

Merging #926 into develop will decrease coverage by 0.07%.
The diff coverage is 66.66%.

@@             Coverage Diff             @@
##           develop     #926      +/-   ##
===========================================
- Coverage    96.65%   96.58%   -0.08%
===========================================
  Files          171      172       +1
  Lines         4876     4893      +17
  Branches       468      466       -2
===========================================
+ Hits          4713     4726      +13
- Misses         104      112       +8
+ Partials        59       55       -4

Impacted Files
Coverage Δ

datahub/search/management/commands/init_es.py
0% <0%> (ø)

datahub/search/management/commands/sync_es.py
95.45% <81.81%> (+6.32%)
:arrow_up:

datahub/search/elasticsearch.py
85.41% <92.85%> (+7.03%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f1c0b6...9976a9d. Read the comment docs.


[python package]: Dataset.subset().get_weight() returns "None"?

import lightgbm as lgb
import numpy as np

data = np.arange(6).reshape([-1,2])
w = np.array([0.1,0.2,0.3])
lgb_train = lgb.Dataset(data, weight = w)
lgb_train_sub = lgb_train.subset([0,2])
print(lgb_train_sub.get_weight())

The code above returns "None", not "[0.1, 0.3]"!

you should construct dataset first, if the dataset is constructed by the subset.
import lightgbm as lgb
import numpy as np

data = np.arange(6).reshape([-1,2])
w = np.array([0.1,0.2,0.3])
params = {'min_data':1, 'min_data_in_bin':1}
lgb_train = lgb.Dataset(data, weight = w, params=params)
lgb_train_sub = lgb_train.subset([0,2])
lgb_train_sub.construct()
print(lgb_train_sub.get_weight())

Thanks! So, is it right that i should call construct()  if i need to define  the fpreproc function in lgb.cv() ? @guolinke
def fpreproc(dtrain, dtest, param):
    dtrain = dtrain.construct()
    dtest = dtest.construct()

    w_train = dtrain.get_weight()
    # rescale weight
    ...

lgb.cv(..., fpreproc=fpreproc,...)

@qclijun if you call train or cv, the construct function will be auto called.
But if you need to reset weight, you need this fpreproc .


Implements Issue #295

Adds a numerals option using powers of 10.
This will destroy the min/max options for natural if a numerals count
is passed.

Serves me right. I can get the thing to work, but I can't get the error message to come up.
@victorquinn "Help me Obi Wan Kenobi, you're my only hope"

FYI I will manually merge this -- made a mess of things by rewriting all of the tests this weekend so now it's not an easy merge


CI: add Ruby 3.2 to the test matrix

Ruby 3.2 has been released! Let's add it to the CI build matrix and ensure the project is compatible.

Just jruby still being problematic.


Fix TarFileTest

fix TarFileTest.testDirectoryWithLongNameEndsWithSlash

@garydgregory, this fixes TarFileTest on Linux but it's not clear to me if this is the correct fix or if a deeper issue needs to be resolved. The test started failing with the PR for COMPRESS-612 and the extended logic for detecting the XSTAR format here. The logic previously evaluated the test tar files as being in POSIX_FORMAT and so appended the '/' character to the end of directory names. It now evaluates to XSTAR and so does not append the '/' character. In order to make the test pass, I changed the requested entry format in the test from BIGNUMBER_STAR to BIGNUMBER_POSIX.


🍒 Cherry pick PR #4322 to staging 🍒

🍒 Cherry pick https://github.com/Expensify/Expensify.cash/pull/4322 to staging 🍒

🚀 Deployed to staging         in version: 1.0.81-4🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production         in version: 1.0.81-4🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


docs: Rename contributing/gsoc-ideas -> contributing/gsoc.

This page contains a lot of other material related to GSoC than
just project ideas.
We would also want to add a redirect from the old URL to the new
one from the RTD admin page.
It seems redirects need to be added from the RTD admin page: https://docs.readthedocs.io/en/stable/user-defined-redirects.html

Once this has been merged, I can kick off an intro thread on CZO.

(There was a minor merge conflict, it seems. I rebased on top of latest main and pushed here. No other changes!)

Merged, thanks @abhijeetbodas2001!

I've added the necessary redirect, and will merge an additional commit to update CONTRIBUTING.md once that is confirmed working.


Feature request: Some changes for Guilds list

Reverse the order of guilds (its the opposite from order that is in default Discord's client)
Close all folders on startup

The configuration has auto_expand_folders field in theme.guilds_tree table. You can set it to false to disable expansion of folders on startup.
[theme.guilds_tree]
auto_expand_folders = false


User & Project Settings - Amazon S3 should have Connect Account link instead of Access & Secret Key input fields

Reported on 0.43 trello board by @jinluyuan
Testing card: https://trello.com/c/LITno6xe/72-user-settings-connect-amazon-s3-account#
In both User & Project settings when there is not an Amazon S3 account connected, there is no "Connect Account" link next to Amazon S3. Instead, the key input fields are by default visible. This does not match the style of our other add-on settings.
User Settings

Project Settings

Reference #3778 - I think there are anticipated reasons as to why S3 is handled differently, but moving toward consistency is worth noting.

Same issue as #3778 . Close this one in favor of that.


martin config expose only specific schema

Hello ! Thank you for the superbe work !
I am wondering if we could expose only a entire schema instead of exposing one table at a time ?
Thank you

Hi @Captain-Oski! Thanks for the idea, but I'm not sure how do you want it to be. Could you please provide an example for this?

Thank you for replying so quickly :
it could be something like this alongside the config.yaml:
schema_sources:
  public
  mySchema
  myOtherSchema

So that it would only targets all the tables and functions into a specific schema.
I founding it extremely verbose and difficult to maintain an config file when you need multiple tables and functions but need to hide from the public numerous tables.
I guess one could migrate the needed tables and functions into a new database instance but ...

I see the problem with config verbosity, but schema sources don't fit very well yet. I'll think about how to simplify config to expose schemas, thanks!

The #380 makes this possible, but would require us to think how to add it to configuration

Implemented in #510, pending a few more PR megres

added in #546


Cannot use default translation values using Next's app folder structure

Description
Hello there!
My issue is, that when I try to provide some default values for formatting the rich texts, like:
<NextIntlClientProvider
  locale={params.locale}
  messages={messages}
  defaultTranslationValues={{
    // This function breaks the app
    highlight: (text) => <strong>{text}</strong>,
  }}
>
  {children}
</NextIntlClientProvider>

then I got that error:
Error: Functions cannot be passed directly to Client Components because they're not serializable.

Is this issue might have been resolved already using the<EMAIL_ADDRESS>version?
Reproduction URL (CodeSandbox)
https://codesandbox.io/p/sandbox/next-intl-bug-template-forked-6jfg9x?file=%2Fsrc%2Fapp%2F[locale]%2Flayout.tsx&selection=[{"endColumn"%3A38%2C"endLineNumber"%3A30%2C"startColumn"%3A38%2C"startLineNumber"%3A30}]
Reproduction description

Migrate to NextJS's app folder.
Set up next-intl based on the documentation..
Create some rich text translations.
Try to pass some value to the defaultTranslationValues prop of the NextIntlClientProvider component.
The app's gonna crash

Expected behaviour
The app should not crash because of this, instead it should render the rich text message. If that's not possible because the nature of the server components, then it should provide a fallback value or remove this props and it's type declarations when using the app folder structure.

Hey @peterbrczan,
thanks for the report! That's an expected error, React can not serialize functions like defaultTranslationValues when you pass them to a Client Component.
The fix is to create a Client Component where the prop is defined. Have a look at this fixed version of your CodeSandbox.
I'm not sure yet how much information next-intl needs to provide in regards to this, as this is a general Next.js/React topic. Next.js already has a small section in regards to this.

Hi @amannn
Thanks for you quick and detailed response, and furthermore for putting effort to extend the CodeSandbox example!
You answered and helped me a lot :)


DUB: Require most recent compiler / dub, simplify

Note that LDC v1.27.0-beta1 has been released so it's not going to stay the most recent version for long.

This allowed me to find #2158 BTWm so an immediate win.

Alpine needs to be updated

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22088

Actually what we needed was https://github.com/bosagora/docker-agora-builder/pull/13 since we don't use upstream images (but it doesn't hurt to update them either).

@omerfirmak : I blame you: https://github.com/dlang/dub/issues/2142

@omerfirmak : I blame you: dlang/dub#2142

That is happening a lot these days.

💚


⚠ ClamAV has degraded performance

In 897bfdb, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 590 ms

Resolved: ClamAV performance has improved in 98f1c46 after 5 minutes.


Ability to disable printout of comma-separated timecodes if csv is output

Problem/Use Case
The comma-separated timecodes list that is output distracts the user from important information when PySceneDetect is used inside some other tool. Setting -q does stop the table from showing up but not this list.
Solutions
Make -q skip the list display as well, or add a separate parameter for this option. The progress bar is a very useful feature so muting all output is not a good workaround.

What version of PySceneDetect are you using?  In the latest version, you shouldn't see any output when using -q at all.  Do you see the same issue when you use -v none?
I can also look into adding a config file option for always showing the progress bar.  The progress bar always goes to stderr, so it might be okay to enable it regardless of the verbosity.

It's v0.6.2, installed via pip3.
Sorry, I think I should have been more clear: the issue is in the list-scenes parameter. Setting -q for pyscenedetect itself mutes everything (as does -v none).

Gotcha, thanks for clarifying.  Originally this was to allow using scenedetect output directly with mkvmerge but this ended up getting overlooked with the logging overhaul in v0.6.  It shouldn't be a problem to hide it now, thanks for the report.


pythonPackages.vdirsyncer: fix build with python 3.9

Motivation for this change
ZHF: #122042
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 123455 at c1c28ed7 run on x86_64-linux 1

4 packages built successfully:

khal
vdirsyncer (python38Packages.vdirsyncer)
python39Packages.vdirsyncer
tests.trivial


:memo: Move bionty-base docs

Fixes #74

Now referring to bionty-base as a standalone option on the Bionty landing page
Updates outdated information
Removes old or unused guides
Keeps some information in docs/guide that is not rendered anywhere.

To discuss:

The comparison content is interesting but not extremely important. Where would we render it? Rewrite it a bit and harmonize it with https://docs.lamin.ai/influences?
It's a tiny bit weird to have things like concept.md in a non-rendered guide. What do we do with those?

Next steps:

No more https://bionty-base.lamin.ai/

See also https://laminlabs.slack.com/archives/C04MU979KD3/p1722675133510609

Uhhm the README doesn't render properly. Not sure what's going on here.


Dashboard should reflect the user role

🗣 Description

Updated header to include the formatted user role
Removed old styling to rely solely on MUI
Updated RegionUsers to display  Dashboard

Global Admin Dashboard
Regional Admin Dashboard

💭 Motivation and context

🧪 Testing

📷 Screenshots (if appropriate)

✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All future TODOs are captured in issues, which are referenced
in code comments.
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[x] Tests have been added and/or modified to cover the changes in this PR.
[x] All new and existing tests pass.

✅ Pre-merge checklist

[ ] Revert dependencies to default branches.
[ ] Finalize version.

✅ Post-merge checklist

[ ] Create a release.

Works for me when I switch to global admin, regional admin, and standard user.
When I am a global view user, it says "Regional Admin" . I'm assuming it should say "Global View User". @schmelz21 ?

Resolved with recent commit.

@stewartl97 - In mobile (sm) mode are we satisfied with not displaying the UserType?

@schmelz21  Yes, we can revisit this later, but for this particular issue we can hold off on displaying usertype.


Converts clinicfinder controller to postgres

I rule and have completed some work on Case Manager that's ready for review!
I did a PG conversion! 🎉
This pull request makes the following changes:

down with nin, long live .not

This is one place where our tests don't fully exercise the code - I temporarily set my local env GOOGLE_GEO_API_KEY to the staging value and confirmed correct behavior.
(If there are changes to the views, please include a screenshot so we know what to look for!)
It relates to the following issue #s:

Fixes #2226

For reviewer:

Adjust the title to explain what it does for the notification email to the listserv.
Tag this PR:

feature if it contains a feature, fix, or similar. This is anything that contains a user-facing fix in some way, such as frontend changes, alterations to backend behavior, or bug fixes.
dependencies if it contains library upgrades or similar. This is anything that upgrades any dependency, such as a Gemfile update or npm package upgrade.

If it contains neither, no need to tag this PR.

Codecov Report

Merging #2248 (3acd8f9) into main (9cc05e5) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #2248   +/-   ##
=======================================
  Coverage   65.51%   65.51%
=======================================
  Files          93       93
  Lines        2421     2421
=======================================
  Hits         1586     1586
  Misses        835      835

Impacted Files
Coverage Δ

app/controllers/clinicfinders_controller.rb
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9cc05e5...3acd8f9. Read the comment docs.